[
  {
    "path": ".claude/skills/rationalize-deps/SKILL.md",
    "content": "---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p <crate> -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"<crate>\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n   ```toml\n   some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n   ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n"
  },
  {
    "path": ".claude/skills/simple-pr/SKILL.md",
    "content": "---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n"
  },
  {
    "path": ".github/FUNDING.yml",
    "content": "# These are supported funding model platforms\n\ngithub: fulmicoton\npatreon: # Replace with a single Patreon username\nopen_collective: # Replace with a single Open Collective username\nko_fi: # Replace with a single Ko-fi username\ntidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel\ncommunity_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry\nliberapay: # Replace with a single Liberapay username\nissuehunt: # Replace with a single IssueHunt username\notechie: # Replace with a single Otechie username\ncustom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/actions.md",
    "content": "---\nname: Actions\nabout: Actions not directly related to producing code.\n\n---\n\n# Actions title\n\nAction description. \ne.g. \n- benchmark\n- investigate and report\n- etc.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.md",
    "content": "---\nname: Bug report\nabout: Create a report to help us improve\n\n---\n\n**Describe the bug**\n- What did you do?\n- What happened?\n- What was expected?\n\n**Which version of tantivy are you using?**\nIf \"master\",  ideally give the specific sha1 revision.\n\n**To Reproduce**\n\nIf your bug is deterministic, can you give a minimal reproducing code?\nSome bugs are not deterministic. Can you describe with precision in which context it happened?\nIf this is possible, can you share your code?\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_request.md",
    "content": "---\nname: Feature request\nabout: Suggest an idea for this project\n\n---\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\n\n**Describe the solution you'd like**\nA clear and concise description of what you want to happen.\n\n**[Optional] describe alternatives you've considered**\nA clear and concise description of any alternative solutions or features you've considered.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/question.md",
    "content": "---\nname: Question\nabout: Ask any question about tantivy's usage...\n\n---\n\nTry to be specific about your use case...\n"
  },
  {
    "path": ".github/dependabot.yml",
    "content": "version: 2\nupdates:\n- package-ecosystem: cargo\n  directory: \"/\"\n  schedule:\n    interval: daily\n    time: \"20:00\"\n  open-pull-requests-limit: 10\n\n- package-ecosystem: \"github-actions\"\n  directory: \"/\"\n  schedule:\n    interval: daily\n    time: \"20:00\"\n  open-pull-requests-limit: 10\n"
  },
  {
    "path": ".github/workflows/coverage.yml",
    "content": "name: Coverage\n\non:\n  push:\n    branches: [main]\n\n# Ensures that we cancel running jobs for the same PR / same workflow.\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}\n  cancel-in-progress: true\n\njobs:\n  coverage:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - name: Install Rust\n        run: rustup toolchain install nightly-2025-12-01 --profile minimal --component llvm-tools-preview\n      - uses: Swatinem/rust-cache@v2\n      - uses: taiki-e/install-action@cargo-llvm-cov\n      - name: Generate code coverage\n        run: cargo +nightly-2025-12-01 llvm-cov --all-features --workspace --doctests --lcov --output-path lcov.info\n      - name: Upload coverage to Codecov\n        uses: codecov/codecov-action@v3\n        continue-on-error: true\n        with:\n          token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos\n          files: lcov.info\n          fail_ci_if_error: true\n"
  },
  {
    "path": ".github/workflows/long_running.yml",
    "content": "name: Long running tests\n\non:\n  push:\n    branches: [ main ]\n\nenv:\n  CARGO_TERM_COLOR: always\n  NUM_FUNCTIONAL_TEST_ITERATIONS: 20000\n\n# Ensures that we cancel running jobs for the same PR / same workflow.\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}\n  cancel-in-progress: true\n\njobs:\n  test:\n\n    runs-on: ubuntu-latest\n\n    steps:\n    - uses: actions/checkout@v4\n    - name: Install stable\n      uses: actions-rs/toolchain@v1\n      with:\n          toolchain: stable\n          profile: minimal\n          override: true\n\n    - name: Run indexing_unsorted\n      run: cargo test indexing_unsorted -- --ignored\n    - name: Run indexing_sorted\n      run: cargo test indexing_sorted -- --ignored\n"
  },
  {
    "path": ".github/workflows/test.yml",
    "content": "name: Unit tests\n\non:\n  push:\n    branches: [ main ]\n  pull_request:\n    branches: [ main ]\n\nenv:\n  CARGO_TERM_COLOR: always\n\n# Ensures that we cancel running jobs for the same PR / same workflow.\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}\n  cancel-in-progress: true\n\njobs:\n  check:\n\n    runs-on: ubuntu-latest\n\n    steps:\n    - uses: actions/checkout@v4\n\n    - name: Install nightly\n      uses: actions-rs/toolchain@v1\n      with:\n            toolchain: nightly\n            profile: minimal\n            components: rustfmt\n    - name: Install stable\n      uses: actions-rs/toolchain@v1\n      with:\n            toolchain: stable\n            profile: minimal\n            components: clippy\n\n    - uses: Swatinem/rust-cache@v2\n\n    - name: Check Formatting\n      run: cargo +nightly fmt --all -- --check\n\n    - name: Check Stable Compilation\n      run: cargo build --all-features\n\n\n    - name: Check Bench Compilation\n      run: cargo +nightly bench --no-run --profile=dev --all-features\n\n    - uses: actions-rs/clippy-check@v1\n      with:\n        toolchain: stable\n        token: ${{ secrets.GITHUB_TOKEN }}\n        args: --tests\n\n  test:\n\n    runs-on: ubuntu-latest\n\n    strategy:\n      matrix:\n        features:\n          - { label: \"all\", flags: \"mmap,stopwords,lz4-compression,zstd-compression,failpoints,stemmer\" }\n          - { label: \"quickwit\", flags: \"mmap,quickwit,failpoints\" }\n          - { label: \"none\", flags: \"\" }\n\n    name: test-${{ matrix.features.label}}\n\n    steps:\n    - uses: actions/checkout@v4\n\n    - name: Install stable\n      uses: actions-rs/toolchain@v1\n      with:\n            toolchain: stable\n            profile: minimal\n            override: true\n\n    - uses: taiki-e/install-action@nextest\n    - uses: Swatinem/rust-cache@v2\n\n    - name: Run tests\n      run: |\n        # if matrix.feature.flags is empty then run on --lib to avoid compiling examples\n        # (as most of them rely on mmap) otherwise run all\n        if [ -z \"${{ matrix.features.flags }}\" ]; then\n          cargo +stable nextest run --lib --no-default-features --verbose --workspace\n        else\n          cargo +stable nextest run --features ${{ matrix.features.flags }} --no-default-features --verbose --workspace\n        fi\n\n    - name: Run doctests\n      run: |\n        # if matrix.feature.flags is empty then run on --lib to avoid compiling examples\n        # (as most of them rely on mmap) otherwise run all\n        if [ -z \"${{ matrix.features.flags }}\" ]; then\n          echo \"no doctest for no feature flag\"\n        else\n          cargo +stable test --doc --features ${{ matrix.features.flags }} --verbose --workspace\n        fi\n"
  },
  {
    "path": ".gitignore",
    "content": "tantivy.iml\n.cargo\nproptest-regressions\n*.swp\ntarget\ntarget/debug\n.vscode\ntarget/release\nCargo.lock\nbenchmark\n.DS_Store\n*.bk\n.idea\ntrace.dat\ncargo-timing*\ncontrol\nvariable\n"
  },
  {
    "path": "ARCHITECTURE.md",
    "content": "# Tantivy\n\n## What is tantivy?\n\nTantivy is a library that is meant to build search engines. Although it is by no means a port of Lucene, its architecture is strongly inspired by it. If you are familiar with Lucene, you may be struck by the overlapping vocabulary.\nThis is not fortuitous.\n\nTantivy's bread and butter is to address the problem of full-text search :\n\nGiven a large set of textual documents, and a text query, return the K-most relevant documents in a very efficient way. To execute these queries rapidly, the tantivy needs to build an index beforehand. The relevance score implemented in the tantivy is not configurable. Tantivy uses the same score as the default similarity used in Lucene / Elasticsearch, called [BM25](https://en.wikipedia.org/wiki/Okapi_BM25).\n\nBut tantivy's scope does not stop there. Numerous features are required to power rich-search applications. For instance, one may want to:\n\n- compute the count of documents matching a query in the different section of an e-commerce website,\n- display an average price per meter square for a real estate search engine,\n- take into account historical user data to rank documents in a specific way,\n- or even use tantivy to power an OLAP database.\n\nA more abstract description of the problem space tantivy is trying to address is the following.\n\nIngest a large set of documents, create an index that makes it possible to\nrapidly select all documents matching a given predicate (also known as a query) and\ncollect some information about them ([See collector](#collector-define-what-to-do-with-matched-documents)).\n\nRoughly speaking the design is following these guiding principles:\n\n- Search should be O(1) in memory.\n- Indexing should be O(1) in memory. (In practice it is just sublinear)\n- Search should be as fast as possible\n\nThis comes at the cost of the dynamicity of the index: while it is possible to add, and delete documents from our corpus, the tantivy is designed to handle these updates in large batches.\n\n## [core/](src/core): Index, segments, searchers\n\nCore contains all of the high-level code to make it possible to create an index, add documents, delete documents and commit.\n\nThis is both the most high-level part of tantivy, the least performance-sensitive one, the seemingly most mundane code... And paradoxically the most complicated part.\n\n### Index and Segments\n\nA tantivy index is a collection of smaller independent immutable segments.\nEach segment contains its own independent set of data structures.\n\nA segment is identified by a segment id that is in fact a UUID.\nThe file of a segment has the format\n\n ```segment-id . ext```\n\nThe extension signals which data structure (or [`SegmentComponent`](src/index/segment_component.rs)) is stored in the file.\n\nA small `meta.json` file is in charge of keeping track of the list of segments, as well as the schema.\n\nOn commit, one segment per indexing thread is written to disk, and the `meta.json` is then updated atomically.\n\nFor a better idea of how indexing works, you may read the [following blog post](https://fulmicoton.com/posts/behold-tantivy-part2/).\n\n### Deletes\n\nDeletes happen by deleting a \"term\". Tantivy does not offer any notion of primary id, so it is up to the user to use a field in their schema as if it was a primary id, and delete the associated term if they want to delete only one specific document.\n\nOn commit, tantivy will find all of the segments with documents matching this existing term and remove from [alive bitset file](src/fastfield/alive_bitset.rs) that represents the bitset of the alive document ids.\nLike all segment files, this file is immutable. Because it is possible to have more than one alive bitset file at a given instant, the alive bitset filename has the format ```segment_id . commit_opstamp . del```.\n\nAn opstamp is simply an incremental id that identifies any operation applied to the index. For instance, performing a commit or adding a document.\n\n### DocId\n\nWithin a segment, all documents are identified by a DocId that ranges within `[0, max_doc)`.\nwhere `max_doc` is the number of documents in the segment, (deleted or not). Having such a compact `DocId` space is key to the compression of our data structures.\n\nThe DocIds are simply allocated in the order documents are added to the index.\n\n### Merges\n\nIn separate threads, tantivy's index writer search for opportunities to merge segments.\nThe point of segment merge is to:\n\n- eventually get rid of tombstoned documents\n- reduce the otherwise ever-growing number of segments.\n\nIndeed, while having several segments instead of one does not hurt search too much, having hundreds can have a measurable impact on the search performance.\n\n### Searcher\n\nThe user of the library usually does not need to know about the existence of Segments.\nSearching is done through an object called a [`Searcher`](src/core/searcher.rs), that captures a\nsnapshot of the index at one point of time, by holding a list of [SegmentReader](src/core/segment_reader.rs).\n\nIn other words, regardless of commits, file garbage collection, or segment merge that might happen, as long as the user holds and reuse the same [Searcher](src/core/searcher.rs), search will happen on an immutable snapshot of the index.\n\n## [directory/](src/directory): Where should the data be stored?\n\nTantivy, like Lucene, abstracts the place where the data should be stored in a key-trait\ncalled [`Directory`](src/directory/directory.rs).\nContrary to Lucene however, \"files\" are quite different from some kind of `io::Read` object.\nCheck out [`src/directory/directory.rs`](src/directory/directory.rs) trait for more details.\n\nTantivy ships two main directory implementation: the `MmapDirectory` and the `RamDirectory`,\nbut users can extend tantivy with their own implementation.\n\n## [schema/](src/schema): What are documents?\n\nTantivy's document follows a very strict schema, decided before building any index.\n\nThe schema defines all of the fields that the indexes [`Document`](src/schema/document/mod.rs) may and should contain, their types (`text`, `i64`, `u64`, `Date`, ...) as well as how it should be indexed / represented in tantivy.\n\nDepending on the type of the field, you can decide to\n\n- put it in the docstore\n- store it as a fast field\n- index it\n\nPractically, tantivy will push values associated with this type to up to 3 respective\ndata structures.\n\n*Limitations*\n\nAs of today, tantivy's schema imposes a 1:1 relationship between a field that is being ingested and a field represented in the search index. In sophisticated search application, it is fairly common to want to index a field twice using different tokenizers, or to index the concatenation of several fields together into one field.\n\nThis is not something tantivy supports, and it is up to the user to duplicate field / concatenate fields before feeding them to tantivy.\n\n## General information about these data structures\n\nAll data structures in tantivy, have:\n\n- a writer\n- a serializer\n- a reader\n\nThe writer builds an in-memory representation of a batch of documents. This representation is not searchable. It is just meant as an intermediary mutable representation, to which we can sequentially add\nthe document of a batch. At the end of the batch (or if a memory limit is reached), this representation\nis then converted into an on-disk immutable representation, that is extremely compact.\nThis conversion is done by the serializer.\n\nFinally, the reader is in charge of offering an API to read on this on-disk read-only representation.\nIn tantivy, readers are designed to require very little anonymous memory. The data is read straight from an mmapped file, and loading an index is as fast as mmapping its files.\n\n## [store/](src/store): Here is my DocId, Gimme my document\n\nThe docstore is a row-oriented storage that, for each document, stores a subset of the fields\nthat are marked as stored in the schema. The docstore is compressed using a general-purpose algorithm\nlike LZ4.\n\n**Useful for**\n\nIn search engines, it is often used to display search results.\nOnce the top 10 documents have been identified, we fetch them from the store, and display them or their snippet on the search result page (aka SERP).\n\n**Not useful for**\n\nFetching a document from the store is typically a \"slow\" operation. It usually consists in\n\n- searching into a compact tree-like data structure to find the position of the right block.\n- decompressing a small block\n- returning the document from this block.\n\nIt is NOT meant to be called for every document matching a query.\n\nAs a rule of thumb, if you hit the docstore more than 100 times per search query, you are probably misusing tantivy.\n\n## [fastfield/](src/fastfield): Here is my DocId, Gimme my value\n\nFast fields are stored in a column-oriented storage that allows for random access.\nThe only compression applied is bitpacking. The column comes with two meta data.\nThe minimum value in the column and the number of bits per doc.\n\nFetching a value for a `DocId` is then as simple as computing\n\n```rust\nmin_value + fetch_bits(num_bits * doc_id..num_bits * (doc_id+1))\n```\n\nThis operation just requires one memory fetch.\nBecause, DocSets are scanned through in order (DocId are iterated in a sorted manner) which\nalso help locality.\n\nIn Lucene's jargon, fast fields are called DocValues.\n\n**Useful for**\n\nThey are typically integer values that are useful to either rank or compute aggregate over\nall of the documents matching a query (aka [DocSet](src/docset.rs)).\n\nFor instance, one could define a function to combine upvotes with tantivy's internal relevancy score.\nThis can be done by fetching a fast field during scoring.\nOne could also compute the mean price of the items matching a query in an e-commerce website.\nThis can be done by fetching a fast field in a collector.\nFinally one could decide to post-filter a docset to remove docset with a price within a specific range.\nIf the ratio of filtered out documents is not too low, an efficient way to do this is to fetch the price and apply the filter on the collector side.\n\nAside from integer values, it is also possible to store an actual byte payload.\nFor advanced search engine, it is possible to store all of the features required for learning-to-rank in a byte payload, access it during search, and apply the learning-to-rank model.\n\nFinally facets are a specific kind of fast field, and the associated source code is in [`fastfield/facet_reader.rs`](src/fastfield/facet_reader.rs).\n\n# The inverted search index\n\nThe inverted index is the core part of full-text search.\nWhen presented a new document with the text field \"Hello, happy tax payer!\", tantivy breaks it into a list of so-called tokens. In addition to just splitting these strings into tokens, it might also do different kinds of operations like dropping the punctuation, converting the character to lowercase, apply stemming, etc. Tantivy makes it possible to configure the operations to be applied in the schema (tokenizer/ is the place where these operations are implemented).\n\nFor instance, the default tokenizer of tantivy would break our text into: `[hello, happy, tax, payer]`.\nThe document will therefore be registered in the inverted index as containing the terms\n`[text:hello, text:happy, text:tax, text:payer]`.\n\nThe role of the inverted index is, when given a term, gives us in return a very fast iterator over the sorted doc ids that match the term.\n\nSuch an iterator is called a posting list. In addition to giving us `DocId`, they can also give us optionally the number of occurrence of the term for each document, also called term frequency or TF.\n\nThese iterators being sorted by DocId, one can create an iterator over the document containing `text:tax AND text:payer`, `(text:tax AND text:payer) OR (text:contribuable)` or any boolean expression.\n\nIn order to represent the function\n```Term ⟶ Posting```\n\nThe inverted index actually consists of two data structures chained together.\n\n- [Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term dictionary.\n- [TermInfo](src/postings/term_info.rs) ⟶ [Posting](src/postings/postings.rs) is addressed by the posting lists.\n\nWhere [TermInfo](src/postings/term_info.rs) is an object containing some meta data about a term.\n\n## [termdict/](src/termdict): Here is a term, give me the [TermInfo](src/postings/term_info.rs)\n\nTantivy's term dictionary is mainly in charge of supplying the function\n\n[Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs)\n\nIt is itself broken into two parts.\n\n- [Term](src/schema/term.rs) ⟶ [TermOrdinal](src/termdict/mod.rs) is addressed by a finite state transducer, implemented by the fst crate.\n- [TermOrdinal](src/termdict/mod.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term info store.\n\n## [postings/](src/postings): Iterate over documents... very fast\n\nA posting list makes it possible to store a sorted list of doc ids and for each doc store\na term frequency as well.\n\nThe posting lists are stored in a separate file. The [TermInfo](src/postings/term_info.rs) contains an offset into that file and a number of documents for the given posting list. Both are required and sufficient to read the posting list.\n\nThe posting list is organized in block of 128 documents.\nOne block of doc ids is followed by one block of term frequencies.\n\nThe doc ids are delta encoded and bitpacked.\nThe term frequencies are bitpacked.\n\nBecause the number of docs is rarely a multiple of 128, the last block may contain an arbitrary number of docs between 1 and 127 documents. We then use variable int encoding instead of bitpacking.\n\n## [positions/](src/positions): Where are my terms within the documents?\n\nPhrase queries make it possible to search for documents containing a specific sequence of terms.\nFor instance, when the phrase query \"the art of war\" does not match \"the war of art\".\nTo make it possible, it is possible to specify in the schema that a field should store positions in addition to being indexed.\n\nThe token positions of all of the terms are then stored in a separate file with the extension `.pos`.\nThe [TermInfo](src/postings/term_info.rs) gives an offset (expressed in position this time) in this file. As we iterate through the docset,\nwe advance the position reader by the number of term frequencies of the current document.\n\n## [fieldnorm/](src/fieldnorm): Here is my doc, how many tokens in this field?\n\nThe [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) formula also requires to know the number of tokens stored in a specific field for a given document. We store this information on one byte per document in the fieldnorm.\nThe fieldnorm is therefore compressed. Values up to 40 are encoded unchanged.\n\n## [tokenizer/](src/tokenizer): How should we process text?\n\nText processing is key to a good search experience.\nSplits or normalize your text too much, and the search results will have a less precision and a higher recall.\nDo not normalize, or under split your text, you will end up with a higher precision and a lesser recall.\n\nText processing can be configured by selecting an off-the-shelf [`Tokenizer`](./src/tokenizer/tokenizer.rs) or implementing your own to first split the text into tokens, and then chain different [`TokenFilter`](src/tokenizer/tokenizer.rs)'s to it.\n\nTantivy's comes with few tokenizers, but external crates are offering advanced tokenizers, such as [Lindera](https://crates.io/crates/lindera) for Japanese.\n\n## [query/](src/query): Define and compose queries\n\nThe [Query](src/query/query.rs) trait defines what a query is.\nDue to the necessity for some queries to compute some statistics over the entire index, and because the\nindex is composed of several `SegmentReader`, the path from transforming a `Query` to an iterator over documents is slightly convoluted, but fundamentally, this is what a Query is.\n\nThe iterator over a document comes with some scoring function. The resulting trait is called a\n[Scorer](src/query/scorer.rs) and is specific to a segment.\n\nDifferent queries can be combined using the [BooleanQuery](src/query/boolean_query/).\nTantivy comes with different types of queries and can be extended by implementing\nthe `Query`, `Weight`, and `Scorer` traits.\n\n## [collector](src/collector): Define what to do with matched documents\n\nCollectors define how to aggregate the documents matching a query, in the broadest sense possible.\nThe search will push matched documents one by one, calling their\n`fn collect(doc: DocId, score: Score);` method.\n\nUsers may implement their own collectors by implementing the [Collector](src/collector/mod.rs) trait.\n\n## [query-grammar](query-grammar): Defines the grammar of the query parser\n\nWhile the [QueryParser](src/query/query_parser/query_parser.rs) struct is located in the `query/` directory, the actual parser combinator used to convert user queries into an AST is in an external crate called `query-grammar`. This part was externalized to lighten the work of the compiler.\n"
  },
  {
    "path": "AUTHORS",
    "content": "# This is the list of authors of tantivy for copyright purposes.\nPaul Masurel\nLaurentiu Nicola\nDru Sellers\nAshley Mannix\nMichael J. Curry\nJason Wolfe\n# As an employee of Google I am required to add Google LLC\n# in the list of authors, but this project is not affiliated to Google\n# in any other way.\nGoogle LLC \n"
  },
  {
    "path": "CHANGELOG.md",
    "content": "Tantivy 0.25\n================================\n\n## Bugfixes\n- fix union performance regression in tantivy 0.24 [#2663](https://github.com/quickwit-oss/tantivy/pull/2663)(@PSeitz)\n- make zstd optional in sstable [#2633](https://github.com/quickwit-oss/tantivy/pull/2633)(@Parth)\n- Fix TopDocs::order_by_string_fast_field for asc order [#2672](https://github.com/quickwit-oss/tantivy/pull/2672)(@stuhood @PSeitz)\n\n## Features/Improvements\n- add docs/example and Vec<u32> values to sstable [#2660](https://github.com/quickwit-oss/tantivy/pull/2660)(@PSeitz)\n- Add string fast field support to `TopDocs`. [#2642](https://github.com/quickwit-oss/tantivy/pull/2642)(@stuhood)\n- update edition to 2024 [#2620](https://github.com/quickwit-oss/tantivy/pull/2620)(@PSeitz)\n- Allow optional spaces between the field name and the value in the query parser [#2678](https://github.com/quickwit-oss/tantivy/pull/2678)(@Darkheir)\n- Support mixed field types in query parser [#2676](https://github.com/quickwit-oss/tantivy/pull/2676)(@trinity-1686a)\n- Add per-field size details [#2679](https://github.com/quickwit-oss/tantivy/pull/2679)(@fulmicoton)\n\nTantivy 0.24.2\n================================\n- Fix TopNComputer for reverse order. [#2672](https://github.com/quickwit-oss/tantivy/pull/2672)(@stuhood @PSeitz) \n\nAffected queries are [order_by_fast_field](https://docs.rs/tantivy/latest/tantivy/collector/struct.TopDocs.html#method.order_by_fast_field) and\n[order_by_u64_field](https://docs.rs/tantivy/latest/tantivy/collector/struct.TopDocs.html#method.order_by_u64_field)\nfor `Order::Asc`\n\nTantivy 0.24.1\n================================\n- Fix: bump required rust version to 1.81\n  \nTantivy 0.24\n================================\nTantivy 0.24 will be backwards compatible with indices created with v0.22 and v0.21. The new minimum rust version will be 1.75. Tantivy 0.23 will be skipped.\n\n#### Bugfixes\n- fix potential endless loop in merge [#2457](https://github.com/quickwit-oss/tantivy/pull/2457)(@PSeitz)\n- fix bug that causes out-of-order sstable key. [#2445](https://github.com/quickwit-oss/tantivy/pull/2445)(@fulmicoton)\n- fix ReferenceValue API flaw [#2372](https://github.com/quickwit-oss/tantivy/pull/2372)(@PSeitz)\n- fix `OwnedBytes` debug panic [#2512](https://github.com/quickwit-oss/tantivy/pull/2512)(@b41sh)\n- catch panics during merges [#2582](https://github.com/quickwit-oss/tantivy/pull/2582)(@rdettai)\n- switch from u32 to usize in bitpacker. This enables multivalued columns larger than 4GB, which crashed during merge before. [#2581](https://github.com/quickwit-oss/tantivy/pull/2581) [#2586](https://github.com/quickwit-oss/tantivy/pull/2586)(@fulmicoton-dd @PSeitz)\n\n#### Breaking API Changes\n- remove index sorting [#2434](https://github.com/quickwit-oss/tantivy/pull/2434)(@PSeitz)\n\n#### Features/Improvements\n- **Aggregation**\n    - Support for cardinality aggregation [#2337](https://github.com/quickwit-oss/tantivy/pull/2337) [#2446](https://github.com/quickwit-oss/tantivy/pull/2446) (@raphaelcoeffic @PSeitz)\n    - Support for extended stats aggregation [#2247](https://github.com/quickwit-oss/tantivy/pull/2247)(@giovannicuccu)\n    - Add Key::I64 and Key::U64 variants in aggregation to avoid f64 precision issues [#2468](https://github.com/quickwit-oss/tantivy/pull/2468)(@PSeitz)\n    - Faster term aggregation fetch terms [#2447](https://github.com/quickwit-oss/tantivy/pull/2447)(@PSeitz)\n    - Improve custom order deserialization [#2451](https://github.com/quickwit-oss/tantivy/pull/2451)(@PSeitz)\n    - Change AggregationLimits behavior [#2495](https://github.com/quickwit-oss/tantivy/pull/2495)(@PSeitz)\n    - lower contention on AggregationLimits [#2394](https://github.com/quickwit-oss/tantivy/pull/2394)(@PSeitz)\n    - fix postcard compatibility for top_hits, add postcard test [#2346](https://github.com/quickwit-oss/tantivy/pull/2346)(@PSeitz)\n    - reduce top hits memory consumption [#2426](https://github.com/quickwit-oss/tantivy/pull/2426)(@PSeitz)\n    - check unsupported parameters top_hits [#2351](https://github.com/quickwit-oss/tantivy/pull/2351)(@PSeitz)\n    - Change AggregationLimits to AggregationLimitsGuard [#2495](https://github.com/quickwit-oss/tantivy/pull/2495)(@PSeitz)\n    - add support for counting non integer in aggregation [#2547](https://github.com/quickwit-oss/tantivy/pull/2547)(@trinity-1686a)\n- **Range Queries**\n    - Support fast field range queries on json fields [#2456](https://github.com/quickwit-oss/tantivy/pull/2456)(@PSeitz)\n    - Add support for str fast field range query [#2460](https://github.com/quickwit-oss/tantivy/pull/2460) [#2452](https://github.com/quickwit-oss/tantivy/pull/2452) [#2453](https://github.com/quickwit-oss/tantivy/pull/2453)(@PSeitz)\n    - modify fastfield range query heuristic [#2375](https://github.com/quickwit-oss/tantivy/pull/2375)(@trinity-1686a)\n    - add FastFieldRangeQuery for explicit range queries on fast field (for `RangeQuery` it is autodetected) [#2477](https://github.com/quickwit-oss/tantivy/pull/2477)(@PSeitz)\n\n- add format backwards-compatibility tests [#2485](https://github.com/quickwit-oss/tantivy/pull/2485)(@PSeitz)\n- add columnar format compatibility tests [#2433](https://github.com/quickwit-oss/tantivy/pull/2433)(@PSeitz)\n- Improved snippet ranges algorithm [#2474](https://github.com/quickwit-oss/tantivy/pull/2474)(@gezihuzi)\n- make find_field_with_default return json fields without path [#2476](https://github.com/quickwit-oss/tantivy/pull/2476)(@trinity-1686a)\n- Make `BooleanQuery` support `minimum_number_should_match` [#2405](https://github.com/quickwit-oss/tantivy/pull/2405)(@LebranceBW)\n- Make `NUM_MERGE_THREADS` configurable [#2535](https://github.com/quickwit-oss/tantivy/pull/2535)(@Barre)\n\n- **RegexPhraseQuery** \n`RegexPhraseQuery` supports phrase queries with regex. E.g. query \"b.* b.* wolf\" matches \"big bad wolf\". Slop is supported as well: \"b.* wolf\"~2 matches \"big bad wolf\" [#2516](https://github.com/quickwit-oss/tantivy/pull/2516)(@PSeitz)\n\n- **Optional Index in Multivalue Columnar Index** \nFor mostly empty multivalued indices there was a large overhead during creation when iterating all docids (merge case). \nThis is alleviated by placing an optional index in the multivalued index to mark documents that have values. \nThis will slightly increase space and access time. [#2439](https://github.com/quickwit-oss/tantivy/pull/2439)(@PSeitz)\n\n- **Store DateTime as nanoseconds in doc store** DateTime in the doc store was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. [#2486](https://github.com/quickwit-oss/tantivy/pull/2486)(@PSeitz)\n\n- **Performance/Memory**\n    - lift clauses in LogicalAst for optimized ast during execution [#2449](https://github.com/quickwit-oss/tantivy/pull/2449)(@PSeitz)\n    - Use Vec instead of BTreeMap to back OwnedValue object [#2364](https://github.com/quickwit-oss/tantivy/pull/2364)(@fulmicoton)\n    - Replace TantivyDocument with CompactDoc. CompactDoc is much smaller and provides similar performance. [#2402](https://github.com/quickwit-oss/tantivy/pull/2402)(@PSeitz)\n    - Recycling buffer in PrefixPhraseScorer [#2443](https://github.com/quickwit-oss/tantivy/pull/2443)(@fulmicoton)\n\n- **Json Type**\n    - JSON supports now all values on the root level. Previously an object was required. This enables support for flat mixed types. allow more JSON values, fix i64 special case [#2383](https://github.com/quickwit-oss/tantivy/pull/2383)(@PSeitz)\n    - add json path constructor to term [#2367](https://github.com/quickwit-oss/tantivy/pull/2367)(@PSeitz)\n\n- **QueryParser**\n    - fix de-escaping too much in query parser [#2427](https://github.com/quickwit-oss/tantivy/pull/2427)(@trinity-1686a)\n    - improve query parser [#2416](https://github.com/quickwit-oss/tantivy/pull/2416)(@trinity-1686a)\n    - Support field grouping `title:(return AND \"pink panther\")` [#2333](https://github.com/quickwit-oss/tantivy/pull/2333)(@trinity-1686a)\n    - allow term starting with wildcard [#2568](https://github.com/quickwit-oss/tantivy/pull/2568)(@trinity-1686a)\n\n- Exist queries match subpath fields [#2558](https://github.com/quickwit-oss/tantivy/pull/2558)(@rdettai)\n- add access benchmark for columnar [#2432](https://github.com/quickwit-oss/tantivy/pull/2432)(@PSeitz)\n- extend indexwriter proptests [#2342](https://github.com/quickwit-oss/tantivy/pull/2342)(@PSeitz)\n- add bench & test for columnar merging [#2428](https://github.com/quickwit-oss/tantivy/pull/2428)(@PSeitz)\n- Change in Executor API [#2391](https://github.com/quickwit-oss/tantivy/pull/2391)(@fulmicoton)\n- Removed usage of num_cpus [#2387](https://github.com/quickwit-oss/tantivy/pull/2387)(@fulmicoton)\n- use bingang for agg and stacker benchmark [#2378](https://github.com/quickwit-oss/tantivy/pull/2378)[#2492](https://github.com/quickwit-oss/tantivy/pull/2492)(@PSeitz) \n- cleanup top level exports [#2382](https://github.com/quickwit-oss/tantivy/pull/2382)(@PSeitz)\n- make convert_to_fast_value_and_append_to_json_term pub [#2370](https://github.com/quickwit-oss/tantivy/pull/2370)(@PSeitz)\n- remove JsonTermWriter [#2238](https://github.com/quickwit-oss/tantivy/pull/2238)(@PSeitz)\n- validate sort by field type [#2336](https://github.com/quickwit-oss/tantivy/pull/2336)(@PSeitz)\n- Fix trait bound of StoreReader::iter [#2360](https://github.com/quickwit-oss/tantivy/pull/2360)(@adamreichold)\n- remove read_postings_no_deletes [#2526](https://github.com/quickwit-oss/tantivy/pull/2526)(@PSeitz)\n\nTantivy 0.22.1\n================================\n- Fix TopNComputer for reverse order. [#2672](https://github.com/quickwit-oss/tantivy/pull/2672)(@stuhood @PSeitz) \n\nAffected queries are [order_by_fast_field](https://docs.rs/tantivy/latest/tantivy/collector/struct.TopDocs.html#method.order_by_fast_field) and\n[order_by_u64_field](https://docs.rs/tantivy/latest/tantivy/collector/struct.TopDocs.html#method.order_by_u64_field)\nfor `Order::Asc`\n\nTantivy 0.22\n================================\n\nTantivy 0.22 will be able to read indices created with Tantivy 0.21.\n\n#### Bugfixes\n- Fix null byte handling in JSON paths (null bytes in json keys caused panic during indexing) [#2345](https://github.com/quickwit-oss/tantivy/pull/2345)(@PSeitz)\n- Fix bug that can cause `get_docids_for_value_range` to panic. [#2295](https://github.com/quickwit-oss/tantivy/pull/2295)(@fulmicoton)\n- Avoid 1 document indices by increase min memory to 15MB for indexing [#2176](https://github.com/quickwit-oss/tantivy/pull/2176)(@PSeitz)\n- Fix merge panic for JSON fields [#2284](https://github.com/quickwit-oss/tantivy/pull/2284)(@PSeitz)\n- Fix bug occurring when merging JSON object indexed with positions. [#2253](https://github.com/quickwit-oss/tantivy/pull/2253)(@fulmicoton)\n- Fix empty DateHistogram gap bug [#2183](https://github.com/quickwit-oss/tantivy/pull/2183)(@PSeitz)\n- Fix range query end check (fields with less than 1 value per doc are affected) [#2226](https://github.com/quickwit-oss/tantivy/pull/2226)(@PSeitz)\n- Handle exclusive out of bounds ranges on fastfield range queries [#2174](https://github.com/quickwit-oss/tantivy/pull/2174)(@PSeitz)\n\n#### Breaking API Changes\n- rename ReloadPolicy onCommit to onCommitWithDelay [#2235](https://github.com/quickwit-oss/tantivy/pull/2235)(@giovannicuccu)\n- Move exports from the root into modules [#2220](https://github.com/quickwit-oss/tantivy/pull/2220)(@PSeitz)\n- Accept field name instead of `Field` in FilterCollector [#2196](https://github.com/quickwit-oss/tantivy/pull/2196)(@PSeitz)\n- remove deprecated IntOptions and DateTime [#2353](https://github.com/quickwit-oss/tantivy/pull/2353)(@PSeitz)\n\n#### Features/Improvements\n- Tantivy documents as a trait: Index data directly without converting to tantivy types first [#2071](https://github.com/quickwit-oss/tantivy/pull/2071)(@ChillFish8)\n- encode some part of posting list as -1 instead of direct values (smaller inverted indices) [#2185](https://github.com/quickwit-oss/tantivy/pull/2185)(@trinity-1686a)\n- **Aggregation**\n  - Support to deserialize f64 from string [#2311](https://github.com/quickwit-oss/tantivy/pull/2311)(@PSeitz)\n  - Add a top_hits aggregator [#2198](https://github.com/quickwit-oss/tantivy/pull/2198)(@ditsuke)\n  - Support bool type in term aggregation [#2318](https://github.com/quickwit-oss/tantivy/pull/2318)(@PSeitz)\n  - Support ip addresses in term aggregation [#2319](https://github.com/quickwit-oss/tantivy/pull/2319)(@PSeitz)\n  - Support date type in term aggregation [#2172](https://github.com/quickwit-oss/tantivy/pull/2172)(@PSeitz)\n  - Support escaped dot when addressing field [#2250](https://github.com/quickwit-oss/tantivy/pull/2250)(@PSeitz)\n\n- Add ExistsQuery to check documents that have a value [#2160](https://github.com/quickwit-oss/tantivy/pull/2160)(@imotov)\n- Expose TopDocs::order_by_u64_field again [#2282](https://github.com/quickwit-oss/tantivy/pull/2282)(@ditsuke)\n\n- **Memory/Performance**\n  - Faster TopN: replace BinaryHeap with TopNComputer [#2186](https://github.com/quickwit-oss/tantivy/pull/2186)(@PSeitz)\n  - reduce number of allocations during indexing [#2257](https://github.com/quickwit-oss/tantivy/pull/2257)(@PSeitz)\n  - Less Memory while indexing: docid deltas while indexing [#2249](https://github.com/quickwit-oss/tantivy/pull/2249)(@PSeitz)\n  - Faster indexing: use term hashmap in fastfield [#2243](https://github.com/quickwit-oss/tantivy/pull/2243)(@PSeitz)\n  - term hashmap remove copy in is_empty, unused unordered_id [#2229](https://github.com/quickwit-oss/tantivy/pull/2229)(@PSeitz)\n  - add method to fetch block of first values in columnar [#2330](https://github.com/quickwit-oss/tantivy/pull/2330)(@PSeitz)\n  - Faster aggregations: add fast path for full columns in fetch_block [#2328](https://github.com/quickwit-oss/tantivy/pull/2328)(@PSeitz)\n  - Faster sstable loading: use fst for sstable index [#2268](https://github.com/quickwit-oss/tantivy/pull/2268)(@trinity-1686a)\n\n- **QueryParser**\n  - allow newline where we allow space in query parser [#2302](https://github.com/quickwit-oss/tantivy/pull/2302)(@trinity-1686a)\n  - allow some mixing of occur and bool in strict query parser [#2323](https://github.com/quickwit-oss/tantivy/pull/2323)(@trinity-1686a)\n  - handle * inside term in lenient query parser [#2228](https://github.com/quickwit-oss/tantivy/pull/2228)(@trinity-1686a)\n  - add support for exists query syntax in query parser [#2170](https://github.com/quickwit-oss/tantivy/pull/2170)(@trinity-1686a)\n- Add shared search executor [#2312](https://github.com/quickwit-oss/tantivy/pull/2312)(@MochiXu)\n- Truncate keys to u16::MAX in term hashmap [#2299](https://github.com/quickwit-oss/tantivy/pull/2299)(@PSeitz)\n- report if a term matched when warming up posting list [#2309](https://github.com/quickwit-oss/tantivy/pull/2309)(@trinity-1686a)\n- Support json fields in FuzzyTermQuery [#2173](https://github.com/quickwit-oss/tantivy/pull/2173)(@PingXia-at)\n- Read list of fields encoded in term dictionary for JSON fields [#2184](https://github.com/quickwit-oss/tantivy/pull/2184)(@PSeitz)\n- add collect_block to BoxableSegmentCollector [#2331](https://github.com/quickwit-oss/tantivy/pull/2331)(@PSeitz)\n- expose collect_block buffer size [#2326](https://github.com/quickwit-oss/tantivy/pull/2326)(@PSeitz)\n- Forward regex parser errors [#2288](https://github.com/quickwit-oss/tantivy/pull/2288)(@adamreichold)\n- Make FacetCounts defaultable and cloneable. [#2322](https://github.com/quickwit-oss/tantivy/pull/2322)(@adamreichold)\n- Derive Debug for SchemaBuilder [#2254](https://github.com/quickwit-oss/tantivy/pull/2254)(@GodTamIt)\n- add missing inlines to tantivy options [#2245](https://github.com/quickwit-oss/tantivy/pull/2245)(@PSeitz)\n\nTantivy 0.21.1\n================================\n#### Bugfixes\n- Range queries on fast fields with less values on that field than documents had an invalid end condition, leading to missing results. [#2226](https://github.com/quickwit-oss/tantivy/issues/2226)(@appaquet @PSeitz)\n- Increase the minimum memory budget from 3MB to 15MB to avoid single doc segments (API fix). [#2176](https://github.com/quickwit-oss/tantivy/issues/2176)(@PSeitz)\n\nTantivy 0.21\n================================\n#### Bugfixes\n- Fix track fast field memory consumption, which led to higher memory consumption than the budget allowed during indexing [#2148](https://github.com/quickwit-oss/tantivy/issues/2148)[#2147](https://github.com/quickwit-oss/tantivy/issues/2147)(@PSeitz)\n- Fix a regression from 0.20 where sort index by date wasn't working anymore [#2124](https://github.com/quickwit-oss/tantivy/issues/2124)(@PSeitz)\n- Fix getting the root facet on the `FacetCollector`. [#2086](https://github.com/quickwit-oss/tantivy/issues/2086)(@adamreichold)\n- Align numerical type priority order of columnar and query. [#2088](https://github.com/quickwit-oss/tantivy/issues/2088)(@fmassot)\n#### Breaking Changes\n- Remove support for Brotli and Snappy compression [#2123](https://github.com/quickwit-oss/tantivy/issues/2123)(@adamreichold)\n#### Features/Improvements\n- Implement lenient query parser [#2129](https://github.com/quickwit-oss/tantivy/pull/2129)(@trinity-1686a)\n- order_by_u64_field and order_by_fast_field allow sorting in ascending and descending order [#2111](https://github.com/quickwit-oss/tantivy/issues/2111)(@naveenann)\n- Allow dynamic filters in text analyzer builder [#2110](https://github.com/quickwit-oss/tantivy/issues/2110)(@fulmicoton @fmassot)\n- **Aggregation**\n  - Add missing parameter for term aggregation [#2149](https://github.com/quickwit-oss/tantivy/issues/2149)[#2103](https://github.com/quickwit-oss/tantivy/issues/2103)(@PSeitz)\n  - Add missing parameter for percentiles [#2157](https://github.com/quickwit-oss/tantivy/issues/2157)(@PSeitz)\n  - Add missing parameter for stats,min,max,count,sum,avg [#2151](https://github.com/quickwit-oss/tantivy/issues/2151)(@PSeitz)\n  - Improve aggregation deserialization error message [#2150](https://github.com/quickwit-oss/tantivy/issues/2150)(@PSeitz)\n  - Add validation for type Bytes to term_agg [#2077](https://github.com/quickwit-oss/tantivy/issues/2077)(@PSeitz)\n  - Alternative mixed field collection [#2135](https://github.com/quickwit-oss/tantivy/issues/2135)(@PSeitz)\n- Add missing query_terms impl for TermSetQuery. [#2120](https://github.com/quickwit-oss/tantivy/issues/2120)(@adamreichold)\n- Minor improvements to OwnedBytes [#2134](https://github.com/quickwit-oss/tantivy/issues/2134)(@adamreichold)\n- Remove allocations in split compound words [#2080](https://github.com/quickwit-oss/tantivy/issues/2080)(@PSeitz)\n- Ngram tokenizer now returns an error with invalid arguments [#2102](https://github.com/quickwit-oss/tantivy/issues/2102)(@fmassot)\n- Make TextAnalyzerBuilder public [#2097](https://github.com/quickwit-oss/tantivy/issues/2097)(@adamreichold)\n- Return an error when tokenizer is not found while indexing [#2093](https://github.com/quickwit-oss/tantivy/issues/2093)(@naveenann)\n- Delayed column opening during merge [#2132](https://github.com/quickwit-oss/tantivy/issues/2132)(@PSeitz)\n\nTantivy 0.20.2\n================================\n- Align numerical type priority order on the search side.  [#2088](https://github.com/quickwit-oss/tantivy/issues/2088) (@fmassot)\n- Fix is_child_of function not considering the root facet. [#2086](https://github.com/quickwit-oss/tantivy/issues/2086) (@adamreichhold)\n\nTantivy 0.20.1\n================================\n- Fix building on windows with mmap [#2070](https://github.com/quickwit-oss/tantivy/issues/2070) (@ChillFish8)\n\nTantivy 0.20\n================================\n#### Bugfixes\n- Fix phrase queries with slop (slop supports now transpositions, algorithm that carries slop so far for num terms > 2) [#2031](https://github.com/quickwit-oss/tantivy/issues/2031)[#2020](https://github.com/quickwit-oss/tantivy/issues/2020)(@PSeitz)\n- Handle error for exists on MMapDirectory [#1988](https://github.com/quickwit-oss/tantivy/issues/1988) (@PSeitz)\n- Aggregation\n  - Fix min doc_count empty merge bug [#2057](https://github.com/quickwit-oss/tantivy/issues/2057) (@PSeitz)\n  - Fix: Sort order for term aggregations (sort order on key was inverted) [#1858](https://github.com/quickwit-oss/tantivy/issues/1858) (@PSeitz)\n\n#### Features/Improvements\n- Add PhrasePrefixQuery [#1842](https://github.com/quickwit-oss/tantivy/issues/1842) (@trinity-1686a)\n- Add `coerce` option for text and numbers types (convert the value instead of returning an error during indexing) [#1904](https://github.com/quickwit-oss/tantivy/issues/1904) (@PSeitz)\n- Add regex tokenizer [#1759](https://github.com/quickwit-oss/tantivy/issues/1759)(@mkleen)\n- Move tokenizer API to separate crate. Having a separate crate with a stable API will allow us to use tokenizers with different tantivy versions. [#1767](https://github.com/quickwit-oss/tantivy/issues/1767) (@PSeitz)\n- **Columnar crate**: New fast field handling (@fulmicoton @PSeitz) [#1806](https://github.com/quickwit-oss/tantivy/issues/1806)[#1809](https://github.com/quickwit-oss/tantivy/issues/1809)\n  - Support for fast fields with optional values. Previously tantivy supported only single-valued and multi-value fast fields. The encoding of optional fast fields is now very compact.\n  - Fast field Support for JSON (schemaless fast fields). Support multiple types on the same column. [#1876](https://github.com/quickwit-oss/tantivy/issues/1876) (@fulmicoton)\n  - Unified access for fast fields over different cardinalities.\n  - Unified storage for typed and untyped fields.\n  - Move fastfield codecs into columnar. [#1782](https://github.com/quickwit-oss/tantivy/issues/1782) (@fulmicoton)\n  - Sparse dense index for optional values [#1716](https://github.com/quickwit-oss/tantivy/issues/1716) (@PSeitz)\n  - Switch to nanosecond precision in DateTime fastfield [#2016](https://github.com/quickwit-oss/tantivy/issues/2016) (@PSeitz)\n- **Aggregation**\n  - Add `date_histogram` aggregation (only `fixed_interval` for now) [#1900](https://github.com/quickwit-oss/tantivy/issues/1900) (@PSeitz)\n  - Add `percentiles` aggregations [#1984](https://github.com/quickwit-oss/tantivy/issues/1984) (@PSeitz)\n  - [**breaking**] Drop JSON support on intermediate agg result (we use postcard as format in `quickwit` to send intermediate results) [#1992](https://github.com/quickwit-oss/tantivy/issues/1992) (@PSeitz)\n  - Set memory limit in bytes for aggregations after which they abort (Previously there was only the bucket limit) [#1942](https://github.com/quickwit-oss/tantivy/issues/1942)[#1957](https://github.com/quickwit-oss/tantivy/issues/1957)(@PSeitz)\n  - Add support for u64,i64,f64 fields in term aggregation [#1883](https://github.com/quickwit-oss/tantivy/issues/1883) (@PSeitz)\n  - Allow histogram bounds to be passed as Rfc3339 [#2076](https://github.com/quickwit-oss/tantivy/issues/2076) (@PSeitz)\n  - Add count, min, max, and sum aggregations [#1794](https://github.com/quickwit-oss/tantivy/issues/1794) (@guilload)\n  - Switch to Aggregation without serde_untagged => better deserialization errors. [#2003](https://github.com/quickwit-oss/tantivy/issues/2003) (@PSeitz)\n  - Switch to ms in histogram for date type (ES compatibility) [#2045](https://github.com/quickwit-oss/tantivy/issues/2045) (@PSeitz)\n  - Reduce term aggregation memory consumption [#2013](https://github.com/quickwit-oss/tantivy/issues/2013) (@PSeitz)\n  - Reduce agg memory consumption: Replace generic aggregation collector (which has a high memory requirement per instance) in aggregation tree with optimized versions behind a trait.\n  - Split term collection count and sub_agg (Faster term agg with less memory consumption for cases without sub-aggs) [#1921](https://github.com/quickwit-oss/tantivy/issues/1921) (@PSeitz)\n  - Schemaless aggregations: In combination with stacker tantivy supports now schemaless aggregations via the JSON type.\n    - Add aggregation support for JSON type [#1888](https://github.com/quickwit-oss/tantivy/issues/1888) (@PSeitz)\n    - Mixed types support on JSON fields in aggs [#1971](https://github.com/quickwit-oss/tantivy/issues/1971) (@PSeitz)\n  - Perf: Fetch blocks of vals in aggregation for all cardinality [#1950](https://github.com/quickwit-oss/tantivy/issues/1950) (@PSeitz)\n  - Allow histogram bounds to be passed as Rfc3339 [#2076](https://github.com/quickwit-oss/tantivy/issues/2076) (@PSeitz)\n- `Searcher` with disabled scoring via `EnableScoring::Disabled` [#1780](https://github.com/quickwit-oss/tantivy/issues/1780) (@shikhar)\n- Enable tokenizer on json fields [#2053](https://github.com/quickwit-oss/tantivy/issues/2053) (@PSeitz)\n- Enforcing \"NOT\" and \"-\" queries consistency in UserInputAst [#1609](https://github.com/quickwit-oss/tantivy/issues/1609) (@bazhenov)\n- Faster indexing\n  - Refactor tokenization pipeline to use GATs [#1924](https://github.com/quickwit-oss/tantivy/issues/1924) (@trinity-1686a)\n  - Faster term hash map [#2058](https://github.com/quickwit-oss/tantivy/issues/2058)[#1940](https://github.com/quickwit-oss/tantivy/issues/1940) (@PSeitz)\n  - tokenizer-api: reduce Tokenizer allocation overhead [#2062](https://github.com/quickwit-oss/tantivy/issues/2062) (@PSeitz)\n  - Refactor vint [#2010](https://github.com/quickwit-oss/tantivy/issues/2010) (@PSeitz)\n- Faster search\n  - Work in batches of docs on the SegmentCollector (Only for cases without score for now) [#1937](https://github.com/quickwit-oss/tantivy/issues/1937) (@PSeitz)\n  - Faster fast field range queries using SIMD [#1954](https://github.com/quickwit-oss/tantivy/issues/1954) (@fulmicoton)\n  - Improve fast field range query performance [#1864](https://github.com/quickwit-oss/tantivy/issues/1864) (@PSeitz)\n- Make BM25 scoring more flexible [#1855](https://github.com/quickwit-oss/tantivy/issues/1855) (@alexcole)\n- Switch fs2 to fs4 as it is now unmaintained and does not support illumos [#1944](https://github.com/quickwit-oss/tantivy/issues/1944) (@Toasterson)\n- Made BooleanWeight and BoostWeight public [#1991](https://github.com/quickwit-oss/tantivy/issues/1991) (@fulmicoton)\n- Make index compatible with virtual drives on Windows [#1843](https://github.com/quickwit-oss/tantivy/issues/1843) (@gyk)\n- Add stop words for Hungarian language [#2069](https://github.com/quickwit-oss/tantivy/issues/2069) (@tnxbutno)\n- Auto downgrade index record option, instead of vint error [#1857](https://github.com/quickwit-oss/tantivy/issues/1857) (@PSeitz)\n- Enable range query on fast field for u64 compatible types [#1762](https://github.com/quickwit-oss/tantivy/issues/1762) (@PSeitz) [#1876]\n- sstable\n  - Isolating sstable and stacker in independent crates. [#1718](https://github.com/quickwit-oss/tantivy/issues/1718) (@fulmicoton)\n  - New sstable format [#1943](https://github.com/quickwit-oss/tantivy/issues/1943)[#1953](https://github.com/quickwit-oss/tantivy/issues/1953) (@trinity-1686a)\n  - Use DeltaReader directly to implement Dictionary::ord_to_term [#1928](https://github.com/quickwit-oss/tantivy/issues/1928) (@trinity-1686a)\n  - Use DeltaReader directly to implement Dictionary::term_ord [#1925](https://github.com/quickwit-oss/tantivy/issues/1925) (@trinity-1686a)\n- Add separate tokenizer manager for fast fields [#2019](https://github.com/quickwit-oss/tantivy/issues/2019) (@PSeitz)\n- Make construction of LevenshteinAutomatonBuilder for FuzzyTermQuery instances lazy. [#1756](https://github.com/quickwit-oss/tantivy/issues/1756) (@adamreichold)\n- Added support for madvise when opening an mmapped Index [#2036](https://github.com/quickwit-oss/tantivy/issues/2036) (@fulmicoton)\n- Rename `DatePrecision` to `DateTimePrecision` [#2051](https://github.com/quickwit-oss/tantivy/issues/2051) (@guilload)\n- Query Parser\n  - Quotation mark can now be used for phrase queries. [#2050](https://github.com/quickwit-oss/tantivy/issues/2050) (@fulmicoton)\n  - PhrasePrefixQuery is supported in the query parser via: `field:\"phrase ter\"*` [#2044](https://github.com/quickwit-oss/tantivy/issues/2044) (@adamreichold)\n- Docs\n  - Update examples for literate docs [#1880](https://github.com/quickwit-oss/tantivy/issues/1880) (@PSeitz)\n  - Add ip field example [#1775](https://github.com/quickwit-oss/tantivy/issues/1775) (@PSeitz)\n  - Fix doc store cache documentation [#1821](https://github.com/quickwit-oss/tantivy/issues/1821) (@PSeitz)\n  - Fix BooleanQuery document [#1999](https://github.com/quickwit-oss/tantivy/issues/1999) (@RT_Enzyme)\n  - Update comments in the faceted search example [#1737](https://github.com/quickwit-oss/tantivy/issues/1737) (@DawChihLiou)\n\n\nTantivy 0.19\n================================\n#### Bugfixes\n- Fix missing fieldnorms for u64, i64, f64, bool, bytes and date [#1620](https://github.com/quickwit-oss/tantivy/pull/1620) (@PSeitz)\n- Fix interpolation overflow in linear interpolation fastfield codec [#1480](https://github.com/quickwit-oss/tantivy/pull/1480) (@PSeitz @fulmicoton)\n\n#### Features/Improvements\n- Add support for `IN` in queryparser , e.g. `field: IN [val1 val2 val3]` [#1683](https://github.com/quickwit-oss/tantivy/pull/1683) (@trinity-1686a)\n- Skip score calculation, when no scoring is required [#1646](https://github.com/quickwit-oss/tantivy/pull/1646) (@PSeitz)\n- Limit fast fields to u32 (`get_val(u32)`) [#1644](https://github.com/quickwit-oss/tantivy/pull/1644) (@PSeitz)\n- The `DateTime` type has been updated to hold timestamps with microseconds precision.\n  `DateOptions` and `DatePrecision` have been added to configure Date fields. The precision is used to hint on fast values compression. Otherwise, seconds precision is used everywhere else (i.e terms, indexing) [#1396](https://github.com/quickwit-oss/tantivy/pull/1396) (@evanxg852000)\n- Add IP address field type [#1553](https://github.com/quickwit-oss/tantivy/pull/1553) (@PSeitz)\n- Add boolean field type [#1382](https://github.com/quickwit-oss/tantivy/pull/1382) (@boraarslan)\n- Remove Searcher pool and make `Searcher` cloneable. (@PSeitz)\n- Validate settings on create [#1570](https://github.com/quickwit-oss/tantivy/pull/1570) (@PSeitz)\n- Detect and apply gcd on fastfield codecs [#1418](https://github.com/quickwit-oss/tantivy/pull/1418) (@PSeitz)\n- Doc store\n  - use separate thread to compress block store [#1389](https://github.com/quickwit-oss/tantivy/pull/1389) [#1510](https://github.com/quickwit-oss/tantivy/pull/1510) (@PSeitz @fulmicoton)\n  - Expose doc store cache size [#1403](https://github.com/quickwit-oss/tantivy/pull/1403) (@PSeitz)\n  - Enable compression levels for doc store [#1378](https://github.com/quickwit-oss/tantivy/pull/1378) (@PSeitz)\n  - Make block size configurable [#1374](https://github.com/quickwit-oss/tantivy/pull/1374) (@kryesh)\n- Make `tantivy::TantivyError` cloneable [#1402](https://github.com/quickwit-oss/tantivy/pull/1402) (@PSeitz)\n- Add support for phrase slop in query language [#1393](https://github.com/quickwit-oss/tantivy/pull/1393) (@saroh)\n- Aggregation\n  - Add aggregation support for date type [#1693](https://github.com/quickwit-oss/tantivy/pull/1693)(@PSeitz)\n  - Add support for keyed parameter in range and histogram aggregations [#1424](https://github.com/quickwit-oss/tantivy/pull/1424) (@k-yomo)\n  - Add aggregation bucket limit [#1363](https://github.com/quickwit-oss/tantivy/pull/1363) (@PSeitz)\n- Faster indexing\n  - [#1610](https://github.com/quickwit-oss/tantivy/pull/1610) (@PSeitz)\n  - [#1594](https://github.com/quickwit-oss/tantivy/pull/1594) (@PSeitz)\n  - [#1582](https://github.com/quickwit-oss/tantivy/pull/1582) (@PSeitz)\n  - [#1611](https://github.com/quickwit-oss/tantivy/pull/1611) (@PSeitz)\n  - Added a pre-configured stop word filter for various language [#1666](https://github.com/quickwit-oss/tantivy/pull/1666) (@adamreichold)\n\nTantivy 0.18\n================================\n\n- For date values `chrono` has been replaced with `time` (@uklotzde) #1304 :\n  - The `time` crate is re-exported as `tantivy::time` instead of `tantivy::chrono`.\n  - The type alias `tantivy::DateTime` has been removed.\n  - `Value::Date` wraps `time::PrimitiveDateTime` without time zone information.\n  - Internally date/time values are stored as seconds since UNIX epoch in UTC.\n  - Converting a `time::OffsetDateTime` to `Value::Date` implicitly converts the value into UTC.\n    If this is not desired do the time zone conversion yourself and use `time::PrimitiveDateTime`\n    directly instead.\n- Add [histogram](https://github.com/quickwit-oss/tantivy/pull/1306) aggregation (@PSeitz)\n- Add support for fastfield on text fields (@PSeitz)\n- Add terms aggregation (@PSeitz)\n- Add support for zstd compression (@kryesh)\n\nTantivy 0.18.1\n================================\n- Hotfix: positions computation.  #1629 (@fmassot, @fulmicoton, @PSeitz)\n\nTantivy 0.17\n================================\n\n- LogMergePolicy now triggers merges if the ratio of deleted documents reaches a threshold (@shikhar @fulmicoton) [#115](https://github.com/quickwit-oss/tantivy/issues/115)\n- Adds a searcher Warmer API (@shikhar @fulmicoton)\n- Change to non-strict schema. Ignore fields in data which are not defined in schema. Previously this returned an error. #1211\n- Facets are necessarily indexed. Existing index with indexed facets should work out of the box. Index without facets that are marked with index: false should be broken (but they were already broken in a sense). (@fulmicoton) #1195 .\n- Bugfix that could in theory impact durability in theory on some filesystems [#1224](https://github.com/quickwit-oss/tantivy/issues/1224)\n- Schema now offers not indexing fieldnorms (@lpouget) [#922](https://github.com/quickwit-oss/tantivy/issues/922)\n- Reduce the number of fsync calls [#1225](https://github.com/quickwit-oss/tantivy/issues/1225)\n- Fix opening bytes index with dynamic codec (@PSeitz) [#1278](https://github.com/quickwit-oss/tantivy/issues/1278)\n- Added an aggregation collector for range, average and stats compatible with Elasticsearch. (@PSeitz)\n- Added a JSON schema type @fulmicoton [#1251](https://github.com/quickwit-oss/tantivy/issues/1251)\n- Added support for slop in phrase queries @halvorboe [#1068](https://github.com/quickwit-oss/tantivy/issues/1068)\n\nTantivy 0.16.2\n================================\n\n- Bugfix in FuzzyTermQuery. (transposition_cost_one was not doing anything)\n\nTantivy 0.16.1\n========================\n\n- Major Bugfix on multivalued fastfield.  #1151\n- Demux operation (@PSeitz)\n\nTantivy 0.16.0\n=========================\n\n- Bugfix in the filesum check. (@evanxg852000) #1127\n- Bugfix in positions when the index is sorted by a field. (@appaquet) #1125\n\nTantivy 0.15.3\n=========================\n\n- Major bugfix. Deleting documents was broken when the index was sorted by a field. (@appaquet, @fulmicoton) #1101\n\nTantivy 0.15.2\n========================\n\n- Major bugfix. DocStore still panics when a deleted doc is at the beginning of a block. (@appaquet) #1088\n\nTantivy 0.15.1\n=========================\n\n- Major bugfix. DocStore panics when first block is deleted. (@appaquet) #1077\n\nTantivy 0.15.0\n=========================\n\n- API Changes. Using Range instead of (start, end) in the API and internals (`FileSlice`, `OwnedBytes`, `Snippets`, ...)\n  This change is breaking but migration is trivial.\n- Added an Histogram collector. (@fulmicoton) #994\n- Added support for Option<TCollector>.  (@fulmicoton)\n- DocAddress is now a struct (@scampi) #987\n- Bugfix consistent tie break handling in facet's topk (@hardikpnsp) #357\n- Date field support for range queries (@rihardsk) #516\n- Added lz4-flex as the default compression scheme in tantivy (@PSeitz) #1009\n- Renamed a lot of symbols to avoid all uppercasing on acronyms, as per new clippy recommendation. For instance, RAMDirectory -> RamDirectory. (@fulmicoton)\n- Simplified positions index format (@fulmicoton) #1022\n- Moved bitpacking to bitpacker subcrate and add BlockedBitpacker, which bitpacks blocks of 128 elements (@PSeitz) #1030\n- Added support for more-like-this query in tantivy (@evanxg852000) #1011\n- Added support for sorting an index, e.g presorting documents in an index by a timestamp field. This can heavily improve performance for certain scenarios, by utilizing the sorted data (Top-n optimizations)(@PSeitz). #1026\n- Add iterator over documents in doc store (@PSeitz). #1044\n- Fix log merge policy (@PSeitz). #1043\n- Add detection to avoid small doc store blocks on merge (@PSeitz). #1054\n- Make doc store compression dynamic (@PSeitz). #1060\n- Switch to json for footer version handling (@PSeitz). #1060\n- Updated TermMerger implementation to rely on the union feature of the FST (@scampi) #469\n- Add boolean marking whether position is required in the query_terms API call (@fulmicoton). #1070\n\nTantivy 0.14.0\n=========================\n\n- Remove dependency to atomicwrites #833 .Implemented by @fulmicoton upon suggestion and research from @asafigan).\n- Migrated tantivy error from the now deprecated `failure` crate to `thiserror` #760. (@hirevo)\n- API Change. Accessing the typed value off a `Schema::Value` now returns an Option instead of panicking if the type does not match.\n- Large API Change in the Directory API. Tantivy used to assume that all files could be somehow memory mapped. After this change, Directory return a `FileSlice` that can be reduced and eventually read into an `OwnedBytes` object. Long and blocking io operation are still required by they do not span over the entire file.\n- Added support for Brotli compression in the DocStore. (@ppodolsky)\n- Added helper for building intersections and unions in BooleanQuery (@guilload)\n- Bugfix in `Query::explain`\n- Removed dependency on `notify` #924. Replaced with `FileWatcher` struct that polls meta file every 500ms in background thread. (@halvorboe @guilload)\n- Added `FilterCollector`, which wraps another collector and filters docs using a predicate over a fast field (@barrotsteindev)\n- Simplified the encoding of the skip reader struct. BlockWAND max tf is now encoded over a single byte. (@fulmicoton)\n- `FilterCollector` now supports all Fast Field value types (@barrotsteindev)\n- FastField are not all loaded when opening the segment reader. (@fulmicoton)\n- Added an API to merge segments, see `tantivy::merge_segments` #1005. (@evanxg852000)\n\nThis version breaks compatibility and requires users to reindex everything.\n\nTantivy 0.13.2\n===================\n\nBugfix. Acquiring a facet reader on a segment that does not contain any\ndoc with this facet returns `None`. (#896)\n\nTantivy 0.13.1\n===================\n\nMade `Query` and `Collector` `Send + Sync`.\nUpdated misc dependency versions.\n\nTantivy 0.13.0\n======================\n\nTantivy 0.13 introduce a change in the index format that will require\nyou to reindex your index (BlockWAND information are added in the skiplist).\nThe index size increase is minor as this information is only added for\nfull blocks.\nIf you have a massive index for which reindexing is not an option, please contact me\nso that we can discuss possible solutions.\n\n- Bugfix in `FuzzyTermQuery` not matching terms by prefix when it should (@Peachball)\n- Relaxed constraints on the custom/tweak score functions. At the segment level, they can be mut, and they are not required to be Sync + Send.\n- `MMapDirectory::open` does not return a `Result` anymore.\n- Change in the DocSet and Scorer API. (@fulmicoton).\nA freshly created DocSet point directly to their first doc. A sentinel value called TERMINATED marks the end of a DocSet.\n`.advance()` returns the new DocId. `Scorer::skip(target)` has been replaced by `Scorer::seek(target)` and returns the resulting DocId.\nAs a result, iterating through DocSet now looks as follows\n\n```rust\nlet mut doc = docset.doc();\nwhile doc != TERMINATED {\n   // ...\n   doc = docset.advance();\n}\n```\n\nThe change made it possible to greatly simplify a lot of the docset's code.\n\n- Misc internal optimization and introduction of the `Scorer::for_each_pruning` function. (@fulmicoton)\n- Added an offset option to the Top(.*)Collectors. (@robyoung)\n- Added Block WAND. Performance on TOP-K on term-unions should be greatly increased. (@fulmicoton, and special thanks\nto the PISA team for answering all my questions!)\n\nTantivy 0.12.0\n======================\n\n- Removing static dispatch in tokenizers for simplicity. (#762)\n- Added backward iteration for `TermDictionary` stream. (@halvorboe)\n- Fixed a performance issue when searching for the posting lists of a missing term (@audunhalland)\n- Added a configurable maximum number of docs (10M by default) for a segment to be considered for merge (@hntd187, landed by @halvorboe #713)\n- Important Bugfix #777, causing tantivy to retain memory mapping. (diagnosed by @poljar)\n- Added support for field boosting. (#547, @fulmicoton)\n\n## How to update?\n\nCrates relying on custom tokenizer, or registering tokenizer in the manager will require some\nminor changes. Check <https://github.com/quickwit-oss/tantivy/blob/main/examples/custom_tokenizer.rs>\nto check for some code sample.\n\nTantivy 0.11.3\n=======================\n\n- Fixed DateTime as a fast field (#735)\n\nTantivy 0.11.2\n=======================\n\n- The future returned by `IndexWriter::merge` does not borrow `self` mutably anymore (#732)\n- Exposing a constructor for `WatchHandle` (#731)\n\nTantivy 0.11.1\n=====================\n\n- Bug fix #729\n\nTantivy 0.11.0\n=====================\n\n- Added f64 field. Internally reuse u64 code the same way i64 does (@fdb-hiroshima)\n- Various bugfixes in the query parser.\n  - Better handling of hyphens in query parser. (#609)\n  - Better handling of whitespaces.\n- Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. \"title:>hello\", \"weight:>=70.5\", \"height:<200\" (@petr-tik)\n- API change around `Box<BoxableTokenizer>`. See detail in #629\n- Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)\n- Add footer with some metadata to index files. #605 (@fdb-hiroshima)\n- Add a method to check the compatibility of the footer in the index with the running version of tantivy (@petr-tik)\n- TopDocs collector: ensure stable sorting on equal score. #671 (@brainlock)\n- Added handling of pre-tokenized text fields (#642), which will enable users to\n  load tokens created outside tantivy. See usage in examples/pre_tokenized_text. (@kkoziara)\n- Fix crash when committing multiple times with deleted documents. #681 (@brainlock)\n\n## How to update?\n\n- The index format is changed. You are required to reindex your data to use tantivy 0.11.\n- `Box<dyn BoxableTokenizer>` has been replaced by a `BoxedTokenizer` struct.\n- Regex are now compiled when the `RegexQuery` instance is built. As a result, it can now return\nan error and handling the `Result` is required.\n- `tantivy::version()` now returns a `Version` object. This object implements `ToString()`\n\nTantivy 0.10.2\n=====================\n\n- Closes #656. Solving memory leak.\n\nTantivy 0.10.1\n=====================\n\n- Closes #544.  A few users experienced problems with the directory watching system.\nAvoid watching the mmap directory until someone effectively creates a reader that uses\nthis functionality.\n\nTantivy 0.10.0\n=====================\n\n*Tantivy 0.10.0 index format is compatible with the index format in 0.9.0.*\n\n- Added an API to easily tweak or entirely replace the\n default score. See `TopDocs::tweak_score`and `TopScore::custom_score` (@fulmicoton)\n- Added an ASCII folding filter (@drusellers)\n- Bugfix in `query.count` in presence of deletes (@fulmicoton)\n- Added `.explain(...)` in `Query` and `Weight` to (@fulmicoton)\n- Added an efficient way to `delete_all_documents` in `IndexWriter` (@petr-tik).\n  All segments are simply removed.\n\nMinor\n---------\n\n- Switched to Rust 2018 (@uvd)\n- Small simplification of the code.\nCalling .freq() or .doc() when .advance() has never been called\non segment postings should panic from now on.\n- Tokens exceeding `u16::max_value() - 4` chars are discarded silently instead of panicking.\n- Fast fields are now preloaded when the `SegmentReader` is created.\n- `IndexMeta` is now public.  (@hntd187)\n- `IndexWriter` `add_document`, `delete_term`. `IndexWriter` is `Sync`, making it possible to use it with a `Arc<RwLock<IndexWriter>>`. `add_document` and `delete_term` can\nonly require a read lock. (@fulmicoton)\n- Introducing `Opstamp` as an expressive type alias for `u64`. (@petr-tik)\n- Stamper now relies on `AtomicU64` on all platforms (@petr-tik)\n- Bugfix - Files get deleted slightly earlier\n- Compilation resources improved (@fdb-hiroshima)\n\n## How to update?\n\nYour program should be usable as is.\n\n### Fast fields\n\nFast fields used to be accessed directly from the `SegmentReader`.\nThe API changed, you are now required to acquire your fast field reader via the\n`segment_reader.fast_fields()`, and use one of the typed method:\n\n- `.u64()`, `.i64()` if your field is single-valued ;\n- `.u64s()`, `.i64s()` if your field is multi-valued ;\n- `.bytes()` if your field is bytes fast field.\n\nTantivy 0.9.0\n=====================\n\n*0.9.0 index format is not compatible with the\nprevious index format.*\n\n- MAJOR BUGFIX :\n  Some `Mmap` objects were being leaked, and would never get released. (@fulmicoton)\n- Removed most unsafe (@fulmicoton)\n- Indexer memory footprint improved. (VInt comp, inlining the first block. (@fulmicoton)\n- Stemming in other language possible (@pentlander)\n- Segments with no docs are deleted earlier (@barrotsteindev)\n- Added grouped add and delete operations.\n  They are guaranteed to happen together (i.e. they cannot be split by a commit).\n  In addition, adds are guaranteed to happen on the same segment. (@elbow-jason)\n- Removed `INT_STORED` and `INT_INDEXED`. It is now possible to use `STORED` and `INDEXED`\n  for int fields. (@fulmicoton)\n- Added DateTime field (@barrotsteindev)\n- Added IndexReader. By default, index is reloaded automatically upon new commits (@fulmicoton)\n- SIMD linear search within blocks (@fulmicoton)\n\n## How to update ?\n\ntantivy 0.9 brought some API breaking change.\nTo update from tantivy 0.8, you will need to go through the following steps.\n\n- `schema::INT_INDEXED` and `schema::INT_STORED`  should be replaced by `schema::INDEXED` and `schema::INT_STORED`.\n- The index now does not hold the pool of searcher anymore. You are required to create an intermediary object called\n`IndexReader` for this.\n\n    ```rust\n    // create the reader. You typically need to create 1 reader for the entire\n    // lifetime of you program.\n    let reader = index.reader()?;\n\n    // Acquire a searcher (previously `index.searcher()`) is now written:\n    let searcher = reader.searcher();\n\n    // With the default setting of the reader, you are not required to\n    // call `index.load_searchers()` anymore.\n    //\n    // The IndexReader will pick up that change automatically, regardless\n    // of whether the update was done in a different process or not.\n    // If this behavior is not wanted, you can create your reader with\n    // the `ReloadPolicy::Manual`, and manually decide when to reload the index\n    // by calling `reader.reload()?`.\n\n    ```\n\nTantivy 0.8.2\n=====================\n\nFixing build for x86_64 platforms. (#496)\nNo need to update from 0.8.1 if tantivy\nis building on your platform.\n\nTantivy 0.8.1\n=====================\n\nHotfix of #476.\n\nMerge was reflecting deletes before commit was passed.\nThanks @barrotsteindev  for reporting the bug.\n\nTantivy 0.8.0\n=====================\n\n*No change in the index format*\n\n- API Breaking change in the collector API. (@jwolfe, @fulmicoton)\n- Multithreaded search (@jwolfe, @fulmicoton)\n\nTantivy 0.7.1\n=====================\n\n*No change in the index format*\n\n- Bugfix: NGramTokenizer panics on non ascii chars\n- Added a space usage API\n\nTantivy 0.7\n=====================\n\n- Skip data for doc ids and positions (@fulmicoton),\n  greatly improving performance\n- Tantivy error now rely on the failure crate (@drusellers)\n- Added support for `AND`, `OR`, `NOT` syntax in addition to the `+`,`-` syntax\n- Added a snippet generator with highlight (@vigneshsarma, @fulmicoton)\n- Added a `TopFieldCollector` (@pentlander)\n\nTantivy 0.6.1\n=========================\n\n- Bugfix #324. GC removing was removing file that were still in useful\n- Added support for parsing AllQuery and RangeQuery via QueryParser\n  - AllQuery: `*`\n  - RangeQuery:\n    - Inclusive `field:[startIncl to endIncl]`\n    - Exclusive `field:{startExcl to endExcl}`\n    - Mixed `field:[startIncl to endExcl}` and vice versa\n    - Unbounded `field:[start to *]`, `field:[* to end]`\n\nTantivy 0.6\n==========================\n\nSpecial thanks to @drusellers and @jason-wolfe for their contributions\nto this release!\n\n- Removed C code. Tantivy is now pure Rust. (@fulmicoton)\n- BM25 (@fulmicoton)\n- Approximate field norms encoded over 1 byte. (@fulmicoton)\n- Compiles on stable rust (@fulmicoton)\n- Add &[u8] fastfield for associating arbitrary bytes to each document (@jason-wolfe) (#270)\n  - Completely uncompressed\n  - Internally: One u64 fast field for indexes, one fast field for the bytes themselves.\n- Add NGram token support (@drusellers)\n- Add Stopword Filter support (@drusellers)\n- Add a FuzzyTermQuery (@drusellers)\n- Add a RegexQuery (@drusellers)\n- Various performance improvements (@fulmicoton)_\n\nTantivy 0.5.2\n===========================\n\n- bugfix #274\n- bugfix #280\n- bugfix #289\n\nTantivy 0.5.1\n==========================\n\n- bugfix #254 : tantivy failed if no documents in a segment contained a specific field.\n\nTantivy 0.5\n==========================\n\n- Faceting\n- RangeQuery\n- Configurable tokenization pipeline\n- Bugfix in PhraseQuery\n- Various query optimisation\n- Allowing very large indexes\n  - 64 bits file address\n  - Smarter encoding of the `TermInfo` objects\n\nTantivy 0.4.3\n==========================\n\n- Bugfix race condition when deleting files. (#198)\n\nTantivy 0.4.2\n==========================\n\n- Prevent usage of AVX2 instructions (#201)\n\nTantivy 0.4.1\n==========================\n\n- Bugfix for non-indexed fields. (#199)\n\nTantivy 0.4.0\n==========================\n\n- Raise the limit of number of fields (previously 256 fields) (@fulmicoton)\n- Removed u32 fields. They are replaced by u64 and i64 fields (#65) (@fulmicoton)\n- Optimized skip in SegmentPostings (#130) (@lnicola)\n- Replacing rustc_serialize by serde. Kudos to  benchmark@KodrAus and @lnicola\n- Using error-chain (@KodrAus)\n- QueryParser: (@fulmicoton)\n  - Explicit error returned when searched for a term that is not indexed\n  - Searching for a int term via the query parser was broken `(age:1)`\n  - Searching for a non-indexed field returns an explicit Error\n  - Phrase query for non-tokenized field are not tokenized by the query parser.\n- Faster/Better indexing (@fulmicoton)\n  - using murmurhash2\n  - faster merging\n  - more memory efficient fast field writer (@lnicola )\n  - better handling of collisions\n  - lesser memory usage\n- Added API, most notably to iterate over ranges of terms (@fulmicoton)\n- Bugfix that was preventing to unmap segment files, on index drop (@fulmicoton)\n- Made the doc! macro public (@fulmicoton)\n- Added an alternative implementation of the streaming dictionary (@fulmicoton)\n\nTantivy 0.3.1\n==========================\n\n- Expose a method to trigger files garbage collection\n\nTantivy 0.3\n==========================\n\nSpecial thanks to @Kodraus @lnicola @Ameobea @manuel-woelker @celaus\nfor their contribution to this release.\n\nThanks also to everyone in tantivy gitter chat\nfor their advise and company :)\n\n<https://gitter.im/tantivy-search/tantivy>\n\nWarning:\n\nTantivy 0.3 is NOT backward compatible with tantivy 0.2\ncode and index format.\nYou should not expect backward compatibility before\ntantivy 1.0.\n\nNew Features\n------------\n\n- Delete. You can now delete documents from an index.\n- Support for windows (Thanks to @lnicola)\n\nVarious Bugfixes & small improvements\n----------------------------------------\n\n- Added CI for Windows (<https://ci.appveyor.com/project/fulmicoton/tantivy>)\nThanks to @KodrAus ! (#108)\n- Various dependy version update (Thanks to @Ameobea) #76\n- Fixed several race conditions in `Index.wait_merge_threads`\n- Fixed #72. Mmap were never released.\n- Fixed #80. Fast field used to take an amplitude of 32 bits after a merge. (Ouch!)\n- Fixed #92. u32 are now encoded using big endian in the fst\n  in order to make there enumeration consistent with\n  the natural ordering.\n- Building binary targets for tantivy-cli (Thanks to @KodrAus)\n- Misc invisible bug fixes, and code cleanup.\n- Use\n"
  },
  {
    "path": "CITATION.cff",
    "content": "cff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\nauthors:\n  - alias: Quickwit Inc.\n    website: \"https://quickwit.io\"\ntitle: \"tantivy\"\nversion: 0.22.0\ndoi: 10.5281/zenodo.13942948\ndate-released: 2024-10-17\nurl: \"https://github.com/quickwit-oss/tantivy\"\n"
  },
  {
    "path": "Cargo.toml",
    "content": "[package]\nname = \"tantivy\"\nversion = \"0.26.0\"\nauthors = [\"Paul Masurel <paul.masurel@gmail.com>\"]\nlicense = \"MIT\"\ncategories = [\"database-implementations\", \"data-structures\"]\ndescription = \"\"\"Search engine library\"\"\"\ndocumentation = \"https://docs.rs/tantivy/\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\nreadme = \"README.md\"\nkeywords = [\"search\", \"information\", \"retrieval\"]\nedition = \"2021\"\nrust-version = \"1.86\"\nexclude = [\"benches/*.json\", \"benches/*.txt\"]\n\n[dependencies]\noneshot = \"0.1.13\"\nbase64 = \"0.22.0\"\nbyteorder = \"1.4.3\"\ncrc32fast = \"1.3.2\"\nonce_cell = \"1.10.0\"\nregex = { version = \"1.5.5\", default-features = false, features = [\n    \"std\",\n    \"unicode\",\n] }\naho-corasick = \"1.0\"\ntantivy-fst = \"0.5\"\nmemmap2 = { version = \"0.9.0\", optional = true }\nlz4_flex = { version = \"0.12\", default-features = false, optional = true }\nzstd = { version = \"0.13\", optional = true, default-features = false }\ntempfile = { version = \"3.12.0\", optional = true }\nlog = \"0.4.16\"\nserde = { version = \"1.0.219\", features = [\"derive\"] }\nserde_json = \"1.0.140\"\nfs4 = { version = \"0.13.1\", optional = true }\nlevenshtein_automata = \"0.2.1\"\nuuid = { version = \"1.0.0\", features = [\"v4\", \"serde\"] }\ncrossbeam-channel = \"0.5.4\"\nrust-stemmers = { version = \"1.2.0\", optional = true }\ndowncast-rs = \"2.0.1\"\nbitpacking = { version = \"0.9.3\", default-features = false, features = [\n    \"bitpacker4x\",\n] }\ncensus = \"0.4.2\"\nrustc-hash = \"2.0.0\"\nthiserror = \"2.0.1\"\nhtmlescape = \"0.3.1\"\nfail = { version = \"0.5.0\", optional = true }\ntime = { version = \"0.3.47\", features = [\"serde-well-known\"] }\nsmallvec = \"1.8.0\"\nrayon = \"1.5.2\"\nlru = \"0.16.3\"\nfastdivide = \"0.4.0\"\nitertools = \"0.14.0\"\nmeasure_time = \"0.9.0\"\narc-swap = \"1.5.0\"\nbon = \"3.3.1\"\n\ncolumnar = { version = \"0.6\", path = \"./columnar\", package = \"tantivy-columnar\" }\nsstable = { version = \"0.6\", path = \"./sstable\", package = \"tantivy-sstable\", optional = true }\nstacker = { version = \"0.6\", path = \"./stacker\", package = \"tantivy-stacker\" }\nquery-grammar = { version = \"0.25.0\", path = \"./query-grammar\", package = \"tantivy-query-grammar\" }\ntantivy-bitpacker = { version = \"0.9\", path = \"./bitpacker\" }\ncommon = { version = \"0.10\", path = \"./common/\", package = \"tantivy-common\" }\ntokenizer-api = { version = \"0.6\", path = \"./tokenizer-api\", package = \"tantivy-tokenizer-api\" }\nsketches-ddsketch = { git = \"https://github.com/quickwit-oss/rust-sketches-ddsketch.git\", rev = \"555caf1\", features = [\"use_serde\"] }\ndatasketches = \"0.2.0\"\nfutures-util = { version = \"0.3.28\", optional = true }\nfutures-channel = { version = \"0.3.28\", optional = true }\nfnv = \"1.0.7\"\ntypetag = \"0.2.21\"\n\n[target.'cfg(windows)'.dependencies]\nwinapi = \"0.3.9\"\n\n[dev-dependencies]\nbinggan = \"0.14.2\"\nrand = \"0.9\"\nmaplit = \"1.0.2\"\nmatches = \"0.1.9\"\npretty_assertions = \"1.2.1\"\nproptest = \"1.7.0\"\ntest-log = \"0.2.10\"\nfutures = \"0.3.21\"\npaste = \"1.0.11\"\nmore-asserts = \"0.3.1\"\nrand_distr = \"0.5\"\ntime = { version = \"0.3.47\", features = [\"serde-well-known\", \"macros\"] }\npostcard = { version = \"1.0.4\", features = [\n    \"use-std\",\n], default-features = false }\n\n[target.'cfg(not(windows))'.dev-dependencies]\ncriterion = { version = \"0.5\", default-features = false }\n\n[dev-dependencies.fail]\nversion = \"0.5.0\"\nfeatures = [\"failpoints\"]\n\n[profile.release]\nopt-level = 3\ndebug = false\ndebug-assertions = false\n\n[profile.bench]\nopt-level = 3\ndebug = true\ndebug-assertions = false\n\n[profile.test]\ndebug-assertions = true\noverflow-checks = true\n\n[features]\ndefault = [\"mmap\", \"stopwords\", \"lz4-compression\", \"columnar-zstd-compression\", \"stemmer\"]\nstemmer = [\"rust-stemmers\"]\nmmap = [\"fs4\", \"tempfile\", \"memmap2\"]\nstopwords = []\n\nlz4-compression = [\"lz4_flex\"]\nzstd-compression = [\"zstd\"]\n\n# enable zstd-compression in columnar (and sstable)\ncolumnar-zstd-compression = [\"columnar/zstd-compression\"]\n\nfailpoints = [\"fail\", \"fail/failpoints\"]\nunstable = []                            # useful for benches.\n\nquickwit = [\"sstable\", \"futures-util\", \"futures-channel\"]\n\n# Compares only the hash of a string when indexing data.\n# Increases indexing speed, but may lead to extremely rare missing terms, when there's a hash collision.\n# Uses 64bit ahash.\ncompare_hash_only = [\"stacker/compare_hash_only\"]\n\n[workspace]\nmembers = [\n    \"query-grammar\",\n    \"bitpacker\",\n    \"common\",\n    \"ownedbytes\",\n    \"stacker\",\n    \"sstable\",\n    \"tokenizer-api\",\n    \"columnar\",\n]\n\n# Following the \"fail\" crate best practises, we isolate\n# tests that define specific behavior in fail check points\n# in a different binary.\n#\n# We do that because, fail rely on a global definition of\n# failpoints behavior and hence, it is incompatible with\n# multithreading.\n[[test]]\nname = \"failpoints\"\npath = \"tests/failpoints/mod.rs\"\nrequired-features = [\"failpoints\"]\n\n[[bench]]\nname = \"analyzer\"\nharness = false\n\n[[bench]]\nname = \"index-bench\"\nharness = false\n\n[[bench]]\nname = \"agg_bench\"\nharness = false\n\n[[bench]]\nname = \"exists_json\"\nharness = false\n\n[[bench]]\nname = \"range_query\"\nharness = false\n\n[[bench]]\nname = \"and_or_queries\"\nharness = false\n\n[[bench]]\nname = \"range_queries\"\nharness = false\n\n[[bench]]\nname = \"bool_queries_with_range\"\nharness = false\n\n[[bench]]\nname = \"str_search_and_get\"\nharness = false\n\n[[bench]]\nname = \"merge_segments\"\nharness = false\n\n[[bench]]\nname = \"regex_all_terms\"\nharness = false\n"
  },
  {
    "path": "LICENSE",
    "content": "Copyright (c) 2018 by the project authors, as listed in the AUTHORS file. \n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
  },
  {
    "path": "Makefile",
    "content": "test:\n\t@echo \"Run test only... No examples.\"\n\tcargo test --tests --lib\n\nfmt:\n\tcargo +nightly fmt --all\n"
  },
  {
    "path": "README.md",
    "content": "[![Docs](https://docs.rs/tantivy/badge.svg)](https://docs.rs/crate/tantivy/)\n[![Build Status](https://github.com/quickwit-oss/tantivy/actions/workflows/test.yml/badge.svg)](https://github.com/quickwit-oss/tantivy/actions/workflows/test.yml)\n[![codecov](https://codecov.io/gh/quickwit-oss/tantivy/branch/main/graph/badge.svg)](https://codecov.io/gh/quickwit-oss/tantivy)\n[![Join the chat at https://discord.gg/MT27AG5EVE](https://shields.io/discord/908281611840282624?label=chat%20on%20discord)](https://discord.gg/MT27AG5EVE)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Crates.io](https://img.shields.io/crates/v/tantivy.svg)](https://crates.io/crates/tantivy)\n\n<img src=\"https://tantivy-search.github.io/logo/tantivy-logo.png\" alt=\"Tantivy, the fastest full-text search engine library written in Rust\" height=\"250\">\n\n## Fast full-text search engine library written in Rust\n\n**If you are looking for an alternative to Elasticsearch or Apache Solr, check out [Quickwit](https://github.com/quickwit-oss/quickwit), our distributed search engine built on top of Tantivy.**\n\nTantivy is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) or [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not\nan off-the-shelf search engine server, but rather a crate that can be used to build such a search engine.\n\nTantivy is, in fact, strongly inspired by Lucene's design.\n\n## Benchmark\n\nThe following [benchmark](https://tantivy-search.github.io/bench/) breaks down the\nperformance for different types of queries/collections.\n\nYour mileage WILL vary depending on the nature of queries and their load.\n\nDetails about the benchmark can be found at this [repository](https://github.com/quickwit-oss/search-benchmark-game).\n\n## Features\n\n- Full-text search\n- Configurable tokenizer (stemming available for 17 Latin languages) with third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)), Japanese ([lindera](https://github.com/lindera-morphology/lindera-tantivy), [Vaporetto](https://crates.io/crates/vaporetto_tantivy), and [tantivy-tokenizer-tiny-segmenter](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter)) and Korean ([lindera](https://github.com/lindera-morphology/lindera-tantivy) + [lindera-ko-dic-builder](https://github.com/lindera-morphology/lindera-ko-dic-builder))\n- Fast (check out the :racehorse: :sparkles: [benchmark](https://tantivy-search.github.io/bench/) :sparkles: :racehorse:)\n- Tiny startup time (<10ms), perfect for command-line tools\n- BM25 scoring (the same as Lucene)\n- Natural query language (e.g. `(michael AND jackson) OR \"king of pop\"`)\n- Phrase queries search (e.g. `\"michael jackson\"`)\n- Incremental indexing\n- Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop)\n- Mmap directory\n- SIMD integer compression when the platform/CPU includes the SSE2 instruction set\n- Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene)\n- `&[u8]` fast fields\n- Text, i64, u64, f64, dates, ip, bool, and hierarchical facet fields\n- Compressed document store (LZ4, Zstd, None)\n- Range queries\n- Faceted search\n- Configurable indexing (optional term frequency and position indexing)\n- JSON Field\n- Aggregation Collector: histogram, range buckets, average, and stats metrics\n- LogMergePolicy with deletes\n- Searcher Warmer API\n- Cheesy logo with a horse\n\n### Non-features\n\nDistributed search is out of the scope of Tantivy, but if you are looking for this feature, check out [Quickwit](https://github.com/quickwit-oss/quickwit/).\n\n## Getting started\n\nTantivy works on stable Rust and supports Linux, macOS, and Windows.\n\n- [Tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)\n- [tantivy-cli and its tutorial](https://github.com/quickwit-oss/tantivy-cli) - `tantivy-cli` is an actual command-line interface that makes it easy for you to create a search engine,\nindex documents, and search via the CLI or a small server with a REST API.\nIt walks you through getting a Wikipedia search engine up and running in a few minutes.\n- [Reference doc for the last released version](https://docs.rs/tantivy/)\n\n## How can I support this project?\n\nThere are many ways to support this project.\n\n- Use Tantivy and tell us about your experience on [Discord](https://discord.gg/MT27AG5EVE) or by email (paul.masurel@gmail.com)\n- Report bugs\n- Write a blog post\n- Help with documentation by asking questions or submitting PRs\n- Contribute code (you can join [our Discord server](https://discord.gg/MT27AG5EVE))\n- Talk about Tantivy around you\n\n## Contributing code\n\nWe use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.\nFeel free to update CHANGELOG.md with your contribution.\n\n### Tokenizer\n\nWhen implementing a tokenizer for tantivy depend on the `tantivy-tokenizer-api` crate.\n\n### Clone and build locally\n\nTantivy compiles on stable Rust.\nTo check out and run tests, you can simply run:\n\n```bash\ngit clone https://github.com/quickwit-oss/tantivy.git\ncd tantivy\ncargo test\n```\n\n## Companies Using Tantivy\n\n<p align=\"left\">\n<img align=\"center\" src=\"doc/assets/images/etsy.png\" alt=\"Etsy\" height=\"25\" width=\"auto\" /> &nbsp;\n<img align=\"center\" src=\"doc/assets/images/paradedb.png\" alt=\"ParadeDB\" height=\"25\" width=\"auto\" /> &nbsp;\n<img align=\"center\" src=\"doc/assets/images/Nuclia.png#gh-light-mode-only\" alt=\"Nuclia\" height=\"25\" width=\"auto\" /> &nbsp;\n<img align=\"center\" src=\"doc/assets/images/humanfirst.png#gh-light-mode-only\" alt=\"Humanfirst.ai\" height=\"30\" width=\"auto\" />\n<img align=\"center\" src=\"doc/assets/images/element.io.svg#gh-light-mode-only\" alt=\"Element.io\" height=\"25\" width=\"auto\" />\n<img align=\"center\" src=\"doc/assets/images/nuclia-dark-theme.png#gh-dark-mode-only\" alt=\"Nuclia\" height=\"35\" width=\"auto\" /> &nbsp;\n<img align=\"center\" src=\"doc/assets/images/humanfirst.ai-dark-theme.png#gh-dark-mode-only\" alt=\"Humanfirst.ai\" height=\"25\" width=\"auto\" />&nbsp; &nbsp;\n<img align=\"center\" src=\"doc/assets/images/element-dark-theme.png#gh-dark-mode-only\" alt=\"Element.io\" height=\"25\" width=\"auto\" />\n</p>\n\n## FAQ\n\n### Can I use Tantivy in other languages?\n\n- Python → [tantivy-py](https://github.com/quickwit-oss/tantivy-py)\n- Ruby → [tantiny](https://github.com/baygeldin/tantiny)\n\nYou can also find other bindings on [GitHub](https://github.com/search?q=tantivy) but they may be less maintained.\n\n### What are some examples of Tantivy use?\n\n- [seshat](https://github.com/matrix-org/seshat/): A matrix message database/indexer\n- [tantiny](https://github.com/baygeldin/tantiny): Tiny full-text search for Ruby\n- [lnx](https://github.com/lnx-search/lnx): adaptable, typo tolerant search engine with a REST API\n- [Bichon](https://github.com/rustmailer/bichon): A lightweight, high-performance Rust email archiver with WebUI\n- and [more](https://github.com/search?q=tantivy)!\n\n### On average, how much faster is Tantivy compared to Lucene?\n\n- According to our [search latency benchmark](https://tantivy-search.github.io/bench/), Tantivy is approximately 2x faster than Lucene.\n\n### Does tantivy support incremental indexing?\n\n- Yes.\n\n### How can I edit documents?\n\n- Data in tantivy is immutable. To edit a document, the document needs to be deleted and reindexed.\n\n### When will my documents be searchable during indexing?\n\n- Documents will be searchable after a `commit` is called on an `IndexWriter`. Existing `IndexReader`s will also need to be reloaded in order to reflect the changes. Finally, changes are only visible to newly acquired `Searcher`.\n"
  },
  {
    "path": "RELEASE.md",
    "content": "# Releasing a new Tantivy Version\n\n## Steps\n\n1. Identify new packages in workspace since last release\n2. Identify changed packages in workspace since last release\n3. Bump version in `Cargo.toml` and their dependents for all changed packages\n4. Update version of root `Cargo.toml`\n5. Publish version starting with leaf nodes\n6. Set git tag with new version\n\n\n[`cargo-release`](https://github.com/crate-ci/cargo-release) will help us with steps 1-5:\n\nReplace prev-tag-name\n```bash\ncargo release --workspace --no-publish -v --prev-tag-name 0.24 --push-remote origin minor --no-tag\n```\n\n`no-tag` or it will create tags for all the subpackages\n\ncargo release will _not_ ignore unchanged packages, but it will print warnings for them.\ne.g. \"warning: updating ownedbytes to 0.10.0 despite no changes made since tag 0.24\"\n\nWe need to manually ignore these unchanged packages\n```bash\ncargo release --workspace --no-publish -v --prev-tag-name 0.24 --push-remote origin minor --no-tag --exclude tokenizer-api\n```\n\nAdd `--execute` to actually publish the packages, otherwise it will only print the commands that would be run.\n\n### Tag Version\n```bash\ngit tag 0.25.0\ngit push upstream tag 0.25.0\n```\n\n\n"
  },
  {
    "path": "TODO.txt",
    "content": "Make schema_builder API fluent.\nfix doc serialization and prevent compression problems\n\nu64 , etc. should return Result<Option> now that we support optional missing a column is really not an error\nremove fastfield codecs\nditch the first_or_default trick. if it is still useful, improve its implementation.\nrename FastFieldReaders::open to load\n\n\nremove fast field reader\n\nfind a way to unify the two DateTime.\nre-add type check in the filter wrapper\n\nadd unit test on columnar list columns.\n\nmake sure sort works\n\n"
  },
  {
    "path": "benches/agg_bench.rs",
    "content": "use binggan::plugins::PeakMemAllocPlugin;\nuse binggan::{black_box, InputGroup, PeakMemAlloc, INSTRUMENTED_SYSTEM};\nuse rand::distr::weighted::WeightedIndex;\nuse rand::rngs::StdRng;\nuse rand::seq::IndexedRandom;\nuse rand::{Rng, SeedableRng};\nuse rand_distr::Distribution;\nuse serde_json::json;\nuse tantivy::aggregation::agg_req::Aggregations;\nuse tantivy::aggregation::AggregationCollector;\nuse tantivy::query::{AllQuery, TermQuery};\nuse tantivy::schema::{IndexRecordOption, Schema, TextFieldIndexing, FAST, STRING};\nuse tantivy::{doc, DateTime, Index, Term};\n\n#[global_allocator]\npub static GLOBAL: &PeakMemAlloc<std::alloc::System> = &INSTRUMENTED_SYSTEM;\n\n/// Mini macro to register a function via its name\n/// runner.register(\"average_u64\", move |index| average_u64(index));\nmacro_rules! register {\n    ($runner:expr, $func:ident) => {\n        $runner.register(stringify!($func), move |index| {\n            $func(index);\n        })\n    };\n}\n\nfn main() {\n    let inputs = vec![\n        (\"full\", get_test_index_bench(Cardinality::Full).unwrap()),\n        (\n            \"dense\",\n            get_test_index_bench(Cardinality::OptionalDense).unwrap(),\n        ),\n        (\n            \"sparse\",\n            get_test_index_bench(Cardinality::OptionalSparse).unwrap(),\n        ),\n        (\n            \"multivalue\",\n            get_test_index_bench(Cardinality::Multivalued).unwrap(),\n        ),\n    ];\n\n    bench_agg(InputGroup::new_with_inputs(inputs));\n}\n\nfn bench_agg(mut group: InputGroup<Index>) {\n    group.add_plugin(PeakMemAllocPlugin::new(GLOBAL));\n\n    register!(group, average_u64);\n    register!(group, average_f64);\n    register!(group, average_f64_u64);\n    register!(group, stats_f64);\n    register!(group, extendedstats_f64);\n    register!(group, percentiles_f64);\n    register!(group, terms_7);\n    register!(group, terms_all_unique);\n    register!(group, terms_150_000);\n    register!(group, terms_many_top_1000);\n    register!(group, terms_many_order_by_term);\n    register!(group, terms_many_with_top_hits);\n    register!(group, terms_all_unique_with_avg_sub_agg);\n    register!(group, terms_many_with_avg_sub_agg);\n    register!(group, terms_status_with_avg_sub_agg);\n    register!(group, terms_status_with_histogram);\n    register!(group, terms_zipf_1000);\n    register!(group, terms_zipf_1000_with_histogram);\n    register!(group, terms_zipf_1000_with_avg_sub_agg);\n\n    register!(group, terms_many_json_mixed_type_with_avg_sub_agg);\n\n    register!(group, composite_term_many_page_1000);\n    register!(group, composite_term_many_page_1000_with_avg_sub_agg);\n    register!(group, composite_term_few);\n    register!(group, composite_histogram);\n    register!(group, composite_histogram_calendar);\n\n    register!(group, cardinality_agg);\n    register!(group, terms_status_with_cardinality_agg);\n\n    register!(group, range_agg);\n    register!(group, range_agg_with_avg_sub_agg);\n    register!(group, range_agg_with_term_agg_status);\n    register!(group, range_agg_with_term_agg_many);\n    register!(group, histogram);\n    register!(group, histogram_hard_bounds);\n    register!(group, histogram_with_avg_sub_agg);\n    register!(group, histogram_with_term_agg_status);\n    register!(group, avg_and_range_with_avg_sub_agg);\n\n    // Filter aggregation benchmarks\n    register!(group, filter_agg_all_query_count_agg);\n    register!(group, filter_agg_term_query_count_agg);\n    register!(group, filter_agg_all_query_with_sub_aggs);\n    register!(group, filter_agg_term_query_with_sub_aggs);\n\n    group.run();\n}\n\nfn exec_term_with_agg(index: &Index, agg_req: serde_json::Value) {\n    let agg_req: Aggregations = serde_json::from_value(agg_req).unwrap();\n\n    let reader = index.reader().unwrap();\n    let text_field = reader.searcher().schema().get_field(\"text\").unwrap();\n    let term_query = TermQuery::new(\n        Term::from_field_text(text_field, \"cool\"),\n        IndexRecordOption::Basic,\n    );\n    let collector = get_collector(agg_req);\n    let searcher = reader.searcher();\n    black_box(searcher.search(&term_query, &collector).unwrap());\n}\n\nfn average_u64(index: &Index) {\n    let agg_req = json!({\n        \"average\": { \"avg\": { \"field\": \"score\", } }\n    });\n    exec_term_with_agg(index, agg_req)\n}\nfn average_f64(index: &Index) {\n    let agg_req = json!({\n        \"average\": { \"avg\": { \"field\": \"score_f64\", } }\n    });\n    exec_term_with_agg(index, agg_req)\n}\nfn average_f64_u64(index: &Index) {\n    let agg_req = json!({\n        \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } },\n        \"average\": { \"avg\": { \"field\": \"score\" } },\n    });\n    exec_term_with_agg(index, agg_req)\n}\nfn stats_f64(index: &Index) {\n    let agg_req = json!({\n        \"average_f64\": { \"stats\": { \"field\": \"score_f64\", } }\n    });\n    exec_term_with_agg(index, agg_req)\n}\nfn extendedstats_f64(index: &Index) {\n    let agg_req = json!({\n        \"extendedstats_f64\": { \"extended_stats\": { \"field\": \"score_f64\", } }\n    });\n    exec_term_with_agg(index, agg_req)\n}\nfn percentiles_f64(index: &Index) {\n    let agg_req = json!({\n        \"mypercentiles\": {\n            \"percentiles\": {\n                \"field\": \"score_f64\",\n                \"percents\": [ 95, 99, 99.9 ]\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn cardinality_agg(index: &Index) {\n    let agg_req = json!({\n        \"cardinality\": {\n            \"cardinality\": {\n                \"field\": \"text_many_terms\"\n            },\n        }\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_status_with_cardinality_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_few_terms_status\" },\n            \"aggs\": {\n                \"cardinality\": {\n                    \"cardinality\": {\n                        \"field\": \"text_many_terms\"\n                    },\n                }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_7(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_few_terms_status\" } },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_all_unique(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_all_unique_terms\" } },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_150_000(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_many_terms\" } },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_many_top_1000(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_many_terms\", \"size\": 1000 } },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_many_order_by_term(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_many_terms\", \"order\": { \"_key\": \"desc\" } } },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_many_with_top_hits(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_many_terms\" },\n            \"aggs\": {\n                \"top_hits\": { \"top_hits\":\n                    {\n                        \"sort\": [\n                            { \"score\": \"desc\" }\n                        ],\n                        \"size\": 2,\n                        \"doc_value_fields\": [\"score_f64\"]\n                    }\n                }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_many_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_many_terms\" },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_all_unique_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_all_unique_terms\" },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn terms_status_with_histogram(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_few_terms_status\" },\n            \"aggs\": {\n                \"histo\": {\"histogram\": { \"field\": \"score_f64\", \"interval\": 10 }}\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_zipf_1000_with_histogram(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_1000_terms_zipf\" },\n            \"aggs\": {\n                \"histo\": {\"histogram\": { \"field\": \"score_f64\", \"interval\": 10 }}\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_status_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_few_terms_status\" },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_zipf_1000_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"text_1000_terms_zipf\" },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_zipf_1000(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": { \"terms\": { \"field\": \"text_1000_terms_zipf\" } },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn terms_many_json_mixed_type_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_texts\": {\n            \"terms\": { \"field\": \"json.mixed_type\" },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn composite_term_few(index: &Index) {\n    let agg_req = json!({\n        \"my_ctf\": {\n            \"composite\": {\n                \"sources\": [\n                    { \"text_few_terms\": { \"terms\": { \"field\": \"text_few_terms\" } } }\n                ],\n                \"size\": 1000\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn composite_term_many_page_1000(index: &Index) {\n    let agg_req = json!({\n        \"my_ctmp1000\": {\n            \"composite\": {\n                \"sources\": [\n                    { \"text_many_terms\": { \"terms\": { \"field\": \"text_many_terms\" } } }\n                ],\n                \"size\": 1000\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn composite_term_many_page_1000_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"my_ctmp1000wasa\": {\n            \"composite\": {\n                \"sources\": [\n                    { \"text_many_terms\": { \"terms\": { \"field\": \"text_many_terms\" } } }\n                ],\n                \"size\": 1000,\n            },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn composite_histogram(index: &Index) {\n    let agg_req = json!({\n        \"my_ch\": {\n            \"composite\": {\n                \"sources\": [\n                    { \"f64_histogram\": { \"histogram\": { \"field\": \"score_f64\", \"interval\": 1 } } }\n                ],\n                \"size\": 1000\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn composite_histogram_calendar(index: &Index) {\n    let agg_req = json!({\n        \"my_chc\": {\n            \"composite\": {\n                \"sources\": [\n                    { \"time_histogram\": { \"date_histogram\": { \"field\": \"timestamp\", \"calendar_interval\": \"month\" } } }\n                ],\n                \"size\": 1000\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn execute_agg(index: &Index, agg_req: serde_json::Value) {\n    let agg_req: Aggregations = serde_json::from_value(agg_req).unwrap();\n    let collector = get_collector(agg_req);\n\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n    black_box(searcher.search(&AllQuery, &collector).unwrap());\n}\nfn range_agg(index: &Index) {\n    let agg_req = json!({\n        \"range_f64\": { \"range\": { \"field\": \"score_f64\", \"ranges\": [\n            { \"from\": 3, \"to\": 7000 },\n            { \"from\": 7000, \"to\": 20000 },\n            { \"from\": 20000, \"to\": 30000 },\n            { \"from\": 30000, \"to\": 40000 },\n            { \"from\": 40000, \"to\": 50000 },\n            { \"from\": 50000, \"to\": 60000 }\n        ] } },\n    });\n    execute_agg(index, agg_req);\n}\nfn range_agg_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"range\": {\n                \"field\": \"score_f64\",\n                \"ranges\": [\n                    { \"from\": 3, \"to\": 7000 },\n                    { \"from\": 7000, \"to\": 20000 },\n                    { \"from\": 20000, \"to\": 30000 },\n                    { \"from\": 30000, \"to\": 40000 },\n                    { \"from\": 40000, \"to\": 50000 },\n                    { \"from\": 50000, \"to\": 60000 }\n                ]\n            },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn range_agg_with_term_agg_status(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"range\": {\n                \"field\": \"score_f64\",\n                \"ranges\": [\n                    { \"from\": 3, \"to\": 7000 },\n                    { \"from\": 7000, \"to\": 20000 },\n                    { \"from\": 20000, \"to\": 30000 },\n                    { \"from\": 30000, \"to\": 40000 },\n                    { \"from\": 40000, \"to\": 50000 },\n                    { \"from\": 50000, \"to\": 60000 }\n                ]\n            },\n            \"aggs\": {\n                \"my_texts\": { \"terms\": { \"field\": \"text_few_terms_status\" } },\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\nfn range_agg_with_term_agg_many(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"range\": {\n                \"field\": \"score_f64\",\n                \"ranges\": [\n                    { \"from\": 3, \"to\": 7000 },\n                    { \"from\": 7000, \"to\": 20000 },\n                    { \"from\": 20000, \"to\": 30000 },\n                    { \"from\": 30000, \"to\": 40000 },\n                    { \"from\": 40000, \"to\": 50000 },\n                    { \"from\": 50000, \"to\": 60000 }\n                ]\n            },\n            \"aggs\": {\n                \"my_texts\": { \"terms\": { \"field\": \"text_many_terms\" } },\n            }\n        },\n    });\n    execute_agg(index, agg_req);\n}\n\nfn histogram(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"histogram\": {\n                \"field\": \"score_f64\",\n                \"interval\": 100 // 1000 buckets\n            },\n        }\n    });\n    execute_agg(index, agg_req);\n}\nfn histogram_hard_bounds(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": { \"histogram\": { \"field\": \"score_f64\", \"interval\": 100, \"hard_bounds\": { \"min\": 1000, \"max\": 300000 } } },\n    });\n    execute_agg(index, agg_req);\n}\nfn histogram_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"histogram\": { \"field\": \"score_f64\", \"interval\": 100 },\n            \"aggs\": {\n                \"average_f64\": { \"avg\": { \"field\": \"score_f64\" } }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\nfn histogram_with_term_agg_status(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"histogram\": { \"field\": \"score_f64\", \"interval\": 10 },\n            \"aggs\": {\n                \"my_texts\": { \"terms\": { \"field\": \"text_few_terms_status\" } }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\nfn avg_and_range_with_avg_sub_agg(index: &Index) {\n    let agg_req = json!({\n        \"rangef64\": {\n            \"range\": {\n                \"field\": \"score_f64\",\n                \"ranges\": [\n                    { \"from\": 3, \"to\": 7000 },\n                    { \"from\": 7000, \"to\": 20000 },\n                    { \"from\": 20000, \"to\": 60000 }\n                ]\n            },\n            \"aggs\": {\n                \"average_in_range\": { \"avg\": { \"field\": \"score\" } }\n            }\n        },\n        \"average\": { \"avg\": { \"field\": \"score\" } }\n    });\n    execute_agg(index, agg_req);\n}\n\n#[derive(Clone, Copy, Hash, Default, Debug, PartialEq, Eq, PartialOrd, Ord)]\nenum Cardinality {\n    /// All documents contain exactly one value.\n    /// `Full` is the default for auto-detecting the Cardinality, since it is the most strict.\n    #[default]\n    Full = 0,\n    /// All documents contain at most one value.\n    OptionalDense = 1,\n    /// All documents may contain any number of values.\n    Multivalued = 2,\n    /// 1 / 20 documents has a value\n    OptionalSparse = 3,\n}\n\nfn get_collector(agg_req: Aggregations) -> AggregationCollector {\n    AggregationCollector::from_aggs(agg_req, Default::default())\n}\n\nfn get_test_index_bench(cardinality: Cardinality) -> tantivy::Result<Index> {\n    // Flag to use existing index\n    let reuse_index = std::env::var(\"REUSE_AGG_BENCH_INDEX\").is_ok();\n    if reuse_index && std::path::Path::new(\"agg_bench\").exists() {\n        return Index::open_in_dir(\"agg_bench\");\n    }\n    // crreate dir\n    std::fs::create_dir_all(\"agg_bench\")?;\n    let mut schema_builder = Schema::builder();\n    let text_fieldtype = tantivy::schema::TextOptions::default()\n        .set_indexing_options(\n            TextFieldIndexing::default().set_index_option(IndexRecordOption::WithFreqs),\n        )\n        .set_stored();\n    let text_field = schema_builder.add_text_field(\"text\", text_fieldtype);\n    let json_field = schema_builder.add_json_field(\"json\", FAST);\n    let text_field_all_unique_terms =\n        schema_builder.add_text_field(\"text_all_unique_terms\", STRING | FAST);\n    let text_field_many_terms = schema_builder.add_text_field(\"text_many_terms\", STRING | FAST);\n    let text_field_few_terms = schema_builder.add_text_field(\"text_few_terms\", STRING | FAST);\n    let text_field_few_terms_status =\n        schema_builder.add_text_field(\"text_few_terms_status\", STRING | FAST);\n    let text_field_1000_terms_zipf =\n        schema_builder.add_text_field(\"text_1000_terms_zipf\", STRING | FAST);\n    let score_fieldtype = tantivy::schema::NumericOptions::default().set_fast();\n    let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype.clone());\n    let score_field_f64 = schema_builder.add_f64_field(\"score_f64\", score_fieldtype.clone());\n    let score_field_i64 = schema_builder.add_i64_field(\"score_i64\", score_fieldtype);\n    let date_field = schema_builder.add_date_field(\"timestamp\", FAST);\n    // use tmp dir\n    let index = if reuse_index {\n        Index::create_in_dir(\"agg_bench\", schema_builder.build())?\n    } else {\n        Index::create_from_tempdir(schema_builder.build())?\n    };\n    // Approximate log proportions\n    let status_field_data = [\n        (\"INFO\", 8000),\n        (\"ERROR\", 300),\n        (\"WARN\", 1200),\n        (\"DEBUG\", 500),\n        (\"OK\", 500),\n        (\"CRITICAL\", 20),\n        (\"EMERGENCY\", 1),\n    ];\n    let log_level_distribution =\n        WeightedIndex::new(status_field_data.iter().map(|item| item.1)).unwrap();\n\n    let few_terms_data = [\"INFO\", \"ERROR\", \"WARN\", \"DEBUG\"];\n    let lg_norm = rand_distr::LogNormal::new(2.996f64, 0.979f64).unwrap();\n\n    let many_terms_data = (0..150_000)\n        .map(|num| format!(\"author{num}\"))\n        .collect::<Vec<_>>();\n\n    // Prepare 1000 unique terms sampled using a Zipf distribution.\n    // Exponent ~1.1 approximates top-20 terms covering around ~20%.\n    let terms_1000: Vec<String> = (1..=1000).map(|i| format!(\"term_{i}\")).collect();\n    let zipf_1000 = rand_distr::Zipf::new(1000.0, 1.1f64).unwrap();\n\n    {\n        let mut rng = StdRng::from_seed([1u8; 32]);\n        let mut index_writer = index.writer_with_num_threads(1, 200_000_000)?;\n        // To make the different test cases comparable we just change one doc to force the\n        // cardinality\n        if cardinality == Cardinality::OptionalDense {\n            index_writer.add_document(doc!())?;\n        }\n        if cardinality == Cardinality::Multivalued {\n            let log_level_sample_a = status_field_data[log_level_distribution.sample(&mut rng)].0;\n            let log_level_sample_b = status_field_data[log_level_distribution.sample(&mut rng)].0;\n            let idx_a = zipf_1000.sample(&mut rng) as usize - 1;\n            let idx_b = zipf_1000.sample(&mut rng) as usize - 1;\n            let term_1000_a = &terms_1000[idx_a];\n            let term_1000_b = &terms_1000[idx_b];\n            index_writer.add_document(doc!(\n                json_field => json!({\"mixed_type\": 10.0}),\n                json_field => json!({\"mixed_type\": 10.0}),\n                text_field => \"cool\",\n                text_field => \"cool\",\n                text_field_all_unique_terms => \"cool\",\n                text_field_all_unique_terms => \"coolo\",\n                text_field_many_terms => \"cool\",\n                text_field_many_terms => \"cool\",\n                text_field_few_terms => \"cool\",\n                text_field_few_terms => \"cool\",\n                text_field_few_terms_status => log_level_sample_a,\n                text_field_few_terms_status => log_level_sample_b,\n                text_field_1000_terms_zipf => term_1000_a.as_str(),\n                text_field_1000_terms_zipf => term_1000_b.as_str(),\n                score_field => 1u64,\n                score_field => 1u64,\n                score_field_f64 => lg_norm.sample(&mut rng),\n                score_field_f64 => lg_norm.sample(&mut rng),\n                score_field_i64 => 1i64,\n                score_field_i64 => 1i64,\n            ))?;\n        }\n        let mut doc_with_value = 1_000_000;\n        if cardinality == Cardinality::OptionalSparse {\n            doc_with_value /= 20;\n        }\n        let _val_max = 1_000_000.0;\n        for _ in 0..doc_with_value {\n            let val: f64 = rng.random_range(0.0..1_000_000.0);\n            let json = if rng.random_bool(0.1) {\n                // 10% are numeric values\n                json!({ \"mixed_type\": val })\n            } else {\n                json!({\"mixed_type\": many_terms_data.choose(&mut rng).unwrap().to_string()})\n            };\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                json_field => json,\n                text_field_all_unique_terms => format!(\"unique_term_{}\", rng.random::<u64>()),\n                text_field_many_terms => many_terms_data.choose(&mut rng).unwrap().to_string(),\n                text_field_few_terms => few_terms_data.choose(&mut rng).unwrap().to_string(),\n                text_field_few_terms_status => status_field_data[log_level_distribution.sample(&mut rng)].0,\n                text_field_1000_terms_zipf => terms_1000[zipf_1000.sample(&mut rng) as usize - 1].as_str(),\n                score_field => val as u64,\n                score_field_f64 => lg_norm.sample(&mut rng),\n                score_field_i64 => val as i64,\n                date_field => DateTime::from_timestamp_millis((val * 1_000_000.) as i64),\n            ))?;\n            if cardinality == Cardinality::OptionalSparse {\n                for _ in 0..20 {\n                    index_writer.add_document(doc!(text_field => \"cool\"))?;\n                }\n            }\n        }\n        // writing the segment\n        index_writer.commit()?;\n    }\n\n    Ok(index)\n}\n\n// Filter aggregation benchmarks\n\nfn filter_agg_all_query_count_agg(index: &Index) {\n    let agg_req = json!({\n        \"filtered\": {\n            \"filter\": \"*\",\n            \"aggs\": {\n                \"count\": { \"value_count\": { \"field\": \"score\" } }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn filter_agg_term_query_count_agg(index: &Index) {\n    let agg_req = json!({\n        \"filtered\": {\n            \"filter\": \"text:cool\",\n            \"aggs\": {\n                \"count\": { \"value_count\": { \"field\": \"score\" } }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn filter_agg_all_query_with_sub_aggs(index: &Index) {\n    let agg_req = json!({\n        \"filtered\": {\n            \"filter\": \"*\",\n            \"aggs\": {\n                \"avg_score\": { \"avg\": { \"field\": \"score\" } },\n                \"stats_score\": { \"stats\": { \"field\": \"score_f64\" } },\n                \"terms_text\": {\n                    \"terms\": { \"field\": \"text_few_terms_status\" }\n                }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n\nfn filter_agg_term_query_with_sub_aggs(index: &Index) {\n    let agg_req = json!({\n        \"filtered\": {\n            \"filter\": \"text:cool\",\n            \"aggs\": {\n                \"avg_score\": { \"avg\": { \"field\": \"score\" } },\n                \"stats_score\": { \"stats\": { \"field\": \"score_f64\" } },\n                \"terms_text\": {\n                    \"terms\": { \"field\": \"text_few_terms_status\" }\n                }\n            }\n        }\n    });\n    execute_agg(index, agg_req);\n}\n"
  },
  {
    "path": "benches/alice.txt",
    "content": "The Project Gutenberg EBook of Alice’s Adventures in Wonderland, by Lewis Carroll\r\n\r\nThis eBook is for the use of anyone anywhere in the United States and most\r\nother parts of the world at no cost and with almost no restrictions\r\nwhatsoever.  You may copy it, give it away or re-use it under the terms of\r\nthe Project Gutenberg License included with this eBook or online at\r\nwww.gutenberg.org.  If you are not located in the United States, you'll have\r\nto check the laws of the country where you are located before using this ebook.\r\n\r\nTitle: Alice’s Adventures in Wonderland\r\n\r\nAuthor: Lewis Carroll\r\n\r\nRelease Date: June 25, 2008 [EBook #11]\r\n[Most recently updated: October 12, 2020]\r\n\r\nLanguage: English\r\n\r\nCharacter set encoding: UTF-8\r\n\r\n*** START OF THIS PROJECT GUTENBERG EBOOK ALICE’S ADVENTURES IN WONDERLAND ***\r\n\r\n\r\n\r\nProduced by Arthur DiBianca and David Widger\r\n\r\n[Illustration]\r\n\r\n\r\n\r\n\r\nAlice’s Adventures in Wonderland\r\n\r\nby Lewis Carroll\r\n\r\nTHE MILLENNIUM FULCRUM EDITION 3.0\r\n\r\nContents\r\n\r\n CHAPTER I.     Down the Rabbit-Hole\r\n CHAPTER II.    The Pool of Tears\r\n CHAPTER III.   A Caucus-Race and a Long Tale\r\n CHAPTER IV.    The Rabbit Sends in a Little Bill\r\n CHAPTER V.     Advice from a Caterpillar\r\n CHAPTER VI.    Pig and Pepper\r\n CHAPTER VII.   A Mad Tea-Party\r\n CHAPTER VIII.  The Queen’s Croquet-Ground\r\n CHAPTER IX.    The Mock Turtle’s Story\r\n CHAPTER X.     The Lobster Quadrille\r\n CHAPTER XI.    Who Stole the Tarts?\r\n CHAPTER XII.   Alice’s Evidence\r\n\r\n\r\n\r\n\r\nCHAPTER I.\r\nDown the Rabbit-Hole\r\n\r\n\r\nAlice was beginning to get very tired of sitting by her sister on the\r\nbank, and of having nothing to do: once or twice she had peeped into\r\nthe book her sister was reading, but it had no pictures or\r\nconversations in it, “and what is the use of a book,” thought Alice\r\n“without pictures or conversations?”\r\n\r\nSo she was considering in her own mind (as well as she could, for the\r\nhot day made her feel very sleepy and stupid), whether the pleasure of\r\nmaking a daisy-chain would be worth the trouble of getting up and\r\npicking the daisies, when suddenly a White Rabbit with pink eyes ran\r\nclose by her.\r\n\r\nThere was nothing so _very_ remarkable in that; nor did Alice think it\r\nso _very_ much out of the way to hear the Rabbit say to itself, “Oh\r\ndear! Oh dear! I shall be late!” (when she thought it over afterwards,\r\nit occurred to her that she ought to have wondered at this, but at the\r\ntime it all seemed quite natural); but when the Rabbit actually _took a\r\nwatch out of its waistcoat-pocket_, and looked at it, and then hurried\r\non, Alice started to her feet, for it flashed across her mind that she\r\nhad never before seen a rabbit with either a waistcoat-pocket, or a\r\nwatch to take out of it, and burning with curiosity, she ran across the\r\nfield after it, and fortunately was just in time to see it pop down a\r\nlarge rabbit-hole under the hedge.\r\n\r\nIn another moment down went Alice after it, never once considering how\r\nin the world she was to get out again.\r\n\r\nThe rabbit-hole went straight on like a tunnel for some way, and then\r\ndipped suddenly down, so suddenly that Alice had not a moment to think\r\nabout stopping herself before she found herself falling down a very\r\ndeep well.\r\n\r\nEither the well was very deep, or she fell very slowly, for she had\r\nplenty of time as she went down to look about her and to wonder what\r\nwas going to happen next. First, she tried to look down and make out\r\nwhat she was coming to, but it was too dark to see anything; then she\r\nlooked at the sides of the well, and noticed that they were filled with\r\ncupboards and book-shelves; here and there she saw maps and pictures\r\nhung upon pegs. She took down a jar from one of the shelves as she\r\npassed; it was labelled “ORANGE MARMALADE”, but to her great\r\ndisappointment it was empty: she did not like to drop the jar for fear\r\nof killing somebody underneath, so managed to put it into one of the\r\ncupboards as she fell past it.\r\n\r\n“Well!” thought Alice to herself, “after such a fall as this, I shall\r\nthink nothing of tumbling down stairs! How brave they’ll all think me\r\nat home! Why, I wouldn’t say anything about it, even if I fell off the\r\ntop of the house!” (Which was very likely true.)\r\n\r\nDown, down, down. Would the fall _never_ come to an end? “I wonder how\r\nmany miles I’ve fallen by this time?” she said aloud. “I must be\r\ngetting somewhere near the centre of the earth. Let me see: that would\r\nbe four thousand miles down, I think—” (for, you see, Alice had learnt\r\nseveral things of this sort in her lessons in the schoolroom, and\r\nthough this was not a _very_ good opportunity for showing off her\r\nknowledge, as there was no one to listen to her, still it was good\r\npractice to say it over) “—yes, that’s about the right distance—but\r\nthen I wonder what Latitude or Longitude I’ve got to?” (Alice had no\r\nidea what Latitude was, or Longitude either, but thought they were nice\r\ngrand words to say.)\r\n\r\nPresently she began again. “I wonder if I shall fall right _through_\r\nthe earth! How funny it’ll seem to come out among the people that walk\r\nwith their heads downward! The Antipathies, I think—” (she was rather\r\nglad there _was_ no one listening, this time, as it didn’t sound at all\r\nthe right word) “—but I shall have to ask them what the name of the\r\ncountry is, you know. Please, Ma’am, is this New Zealand or Australia?”\r\n(and she tried to curtsey as she spoke—fancy _curtseying_ as you’re\r\nfalling through the air! Do you think you could manage it?) “And what\r\nan ignorant little girl she’ll think me for asking! No, it’ll never do\r\nto ask: perhaps I shall see it written up somewhere.”\r\n\r\nDown, down, down. There was nothing else to do, so Alice soon began\r\ntalking again. “Dinah’ll miss me very much to-night, I should think!”\r\n(Dinah was the cat.) “I hope they’ll remember her saucer of milk at\r\ntea-time. Dinah my dear! I wish you were down here with me! There are\r\nno mice in the air, I’m afraid, but you might catch a bat, and that’s\r\nvery like a mouse, you know. But do cats eat bats, I wonder?” And here\r\nAlice began to get rather sleepy, and went on saying to herself, in a\r\ndreamy sort of way, “Do cats eat bats? Do cats eat bats?” and\r\nsometimes, “Do bats eat cats?” for, you see, as she couldn’t answer\r\neither question, it didn’t much matter which way she put it. She felt\r\nthat she was dozing off, and had just begun to dream that she was\r\nwalking hand in hand with Dinah, and saying to her very earnestly,\r\n“Now, Dinah, tell me the truth: did you ever eat a bat?” when suddenly,\r\nthump! thump! down she came upon a heap of sticks and dry leaves, and\r\nthe fall was over.\r\n\r\nAlice was not a bit hurt, and she jumped up on to her feet in a moment:\r\nshe looked up, but it was all dark overhead; before her was another\r\nlong passage, and the White Rabbit was still in sight, hurrying down\r\nit. There was not a moment to be lost: away went Alice like the wind,\r\nand was just in time to hear it say, as it turned a corner, “Oh my ears\r\nand whiskers, how late it’s getting!” She was close behind it when she\r\nturned the corner, but the Rabbit was no longer to be seen: she found\r\nherself in a long, low hall, which was lit up by a row of lamps hanging\r\nfrom the roof.\r\n\r\nThere were doors all round the hall, but they were all locked; and when\r\nAlice had been all the way down one side and up the other, trying every\r\ndoor, she walked sadly down the middle, wondering how she was ever to\r\nget out again.\r\n\r\nSuddenly she came upon a little three-legged table, all made of solid\r\nglass; there was nothing on it except a tiny golden key, and Alice’s\r\nfirst thought was that it might belong to one of the doors of the hall;\r\nbut, alas! either the locks were too large, or the key was too small,\r\nbut at any rate it would not open any of them. However, on the second\r\ntime round, she came upon a low curtain she had not noticed before, and\r\nbehind it was a little door about fifteen inches high: she tried the\r\nlittle golden key in the lock, and to her great delight it fitted!\r\n\r\nAlice opened the door and found that it led into a small passage, not\r\nmuch larger than a rat-hole: she knelt down and looked along the\r\npassage into the loveliest garden you ever saw. How she longed to get\r\nout of that dark hall, and wander about among those beds of bright\r\nflowers and those cool fountains, but she could not even get her head\r\nthrough the doorway; “and even if my head would go through,” thought\r\npoor Alice, “it would be of very little use without my shoulders. Oh,\r\nhow I wish I could shut up like a telescope! I think I could, if I only\r\nknew how to begin.” For, you see, so many out-of-the-way things had\r\nhappened lately, that Alice had begun to think that very few things\r\nindeed were really impossible.\r\n\r\nThere seemed to be no use in waiting by the little door, so she went\r\nback to the table, half hoping she might find another key on it, or at\r\nany rate a book of rules for shutting people up like telescopes: this\r\ntime she found a little bottle on it, (“which certainly was not here\r\nbefore,” said Alice,) and round the neck of the bottle was a paper\r\nlabel, with the words “DRINK ME,” beautifully printed on it in large\r\nletters.\r\n\r\nIt was all very well to say “Drink me,” but the wise little Alice was\r\nnot going to do _that_ in a hurry. “No, I’ll look first,” she said,\r\n“and see whether it’s marked ‘_poison_’ or not”; for she had read\r\nseveral nice little histories about children who had got burnt, and\r\neaten up by wild beasts and other unpleasant things, all because they\r\n_would_ not remember the simple rules their friends had taught them:\r\nsuch as, that a red-hot poker will burn you if you hold it too long;\r\nand that if you cut your finger _very_ deeply with a knife, it usually\r\nbleeds; and she had never forgotten that, if you drink much from a\r\nbottle marked “poison,” it is almost certain to disagree with you,\r\nsooner or later.\r\n\r\nHowever, this bottle was _not_ marked “poison,” so Alice ventured to\r\ntaste it, and finding it very nice, (it had, in fact, a sort of mixed\r\nflavour of cherry-tart, custard, pine-apple, roast turkey, toffee, and\r\nhot buttered toast,) she very soon finished it off.\r\n\r\n*      *      *      *      *      *      *\r\n\r\n    *      *      *      *      *      *\r\n\r\n*      *      *      *      *      *      *\r\n\r\n\r\n“What a curious feeling!” said Alice; “I must be shutting up like a\r\ntelescope.”\r\n\r\nAnd so it was indeed: she was now only ten inches high, and her face\r\nbrightened up at the thought that she was now the right size for going\r\nthrough the little door into that lovely garden. First, however, she\r\nwaited for a few minutes to see if she was going to shrink any further:\r\nshe felt a little nervous about this; “for it might end, you know,”\r\nsaid Alice to herself, “in my going out altogether, like a candle. I\r\nwonder what I should be like then?” And she tried to fancy what the\r\nflame of a candle is like after the candle is blown out, for she could\r\nnot remember ever having seen such a thing.\r\n\r\nAfter a while, finding that nothing more happened, she decided on going\r\ninto the garden at once; but, alas for poor Alice! when she got to the\r\ndoor, she found she had forgotten the little golden key, and when she\r\nwent back to the table for it, she found she could not possibly reach\r\nit: she could see it quite plainly through the glass, and she tried her\r\nbest to climb up one of the legs of the table, but it was too slippery;\r\nand when she had tired herself out with trying, the poor little thing\r\nsat down and cried.\r\n\r\n“Come, there’s no use in crying like that!” said Alice to herself,\r\nrather sharply; “I advise you to leave off this minute!” She generally\r\ngave herself very good advice, (though she very seldom followed it),\r\nand sometimes she scolded herself so severely as to bring tears into\r\nher eyes; and once she remembered trying to box her own ears for having\r\ncheated herself in a game of croquet she was playing against herself,\r\nfor this curious child was very fond of pretending to be two people.\r\n“But it’s no use now,” thought poor Alice, “to pretend to be two\r\npeople! Why, there’s hardly enough of me left to make _one_ respectable\r\nperson!”\r\n\r\nSoon her eye fell on a little glass box that was lying under the table:\r\nshe opened it, and found in it a very small cake, on which the words\r\n“EAT ME” were beautifully marked in currants. “Well, I’ll eat it,” said\r\nAlice, “and if it makes me grow larger, I can reach the key; and if it\r\nmakes me grow smaller, I can creep under the door; so either way I’ll\r\nget into the garden, and I don’t care which happens!”\r\n\r\nShe ate a little bit, and said anxiously to herself, “Which way? Which\r\nway?”, holding her hand on the top of her head to feel which way it was\r\ngrowing, and she was quite surprised to find that she remained the same\r\nsize: to be sure, this generally happens when one eats cake, but Alice\r\nhad got so much into the way of expecting nothing but out-of-the-way\r\nthings to happen, that it seemed quite dull and stupid for life to go\r\non in the common way.\r\n\r\nSo she set to work, and very soon finished off the cake.\r\n\r\n*      *      *      *      *      *      *\r\n\r\n    *      *      *      *      *      *\r\n\r\n*      *      *      *      *      *      *\r\n\r\n\r\n\r\n\r\nCHAPTER II.\r\nThe Pool of Tears\r\n\r\n\r\n“Curiouser and curiouser!” cried Alice (she was so much surprised, that\r\nfor the moment she quite forgot how to speak good English); “now I’m\r\nopening out like the largest telescope that ever was! Good-bye, feet!”\r\n(for when she looked down at her feet, they seemed to be almost out of\r\nsight, they were getting so far off). “Oh, my poor little feet, I\r\nwonder who will put on your shoes and stockings for you now, dears? I’m\r\nsure _I_ shan’t be able! I shall be a great deal too far off to trouble\r\nmyself about you: you must manage the best way you can;—but I must be\r\nkind to them,” thought Alice, “or perhaps they won’t walk the way I\r\nwant to go! Let me see: I’ll give them a new pair of boots every\r\nChristmas.”\r\n\r\nAnd she went on planning to herself how she would manage it. “They must\r\ngo by the carrier,” she thought; “and how funny it’ll seem, sending\r\npresents to one’s own feet! And how odd the directions will look!\r\n\r\n     _Alice’s Right Foot, Esq., Hearthrug, near the Fender,_ (_with\r\n     Alice’s love_).\r\n\r\nOh dear, what nonsense I’m talking!”\r\n\r\nJust then her head struck against the roof of the hall: in fact she was\r\nnow more than nine feet high, and she at once took up the little golden\r\nkey and hurried off to the garden door.\r\n\r\nPoor Alice! It was as much as she could do, lying down on one side, to\r\nlook through into the garden with one eye; but to get through was more\r\nhopeless than ever: she sat down and began to cry again.\r\n\r\n“You ought to be ashamed of yourself,” said Alice, “a great girl like\r\nyou,” (she might well say this), “to go on crying in this way! Stop\r\nthis moment, I tell you!” But she went on all the same, shedding\r\ngallons of tears, until there was a large pool all round her, about\r\nfour inches deep and reaching half down the hall.\r\n\r\nAfter a time she heard a little pattering of feet in the distance, and\r\nshe hastily dried her eyes to see what was coming. It was the White\r\nRabbit returning, splendidly dressed, with a pair of white kid gloves\r\nin one hand and a large fan in the other: he came trotting along in a\r\ngreat hurry, muttering to himself as he came, “Oh! the Duchess, the\r\nDuchess! Oh! won’t she be savage if I’ve kept her waiting!” Alice felt\r\nso desperate that she was ready to ask help of any one; so, when the\r\nRabbit came near her, she began, in a low, timid voice, “If you please,\r\nsir—” The Rabbit started violently, dropped the white kid gloves and\r\nthe fan, and skurried away into the darkness as hard as he could go.\r\n\r\nAlice took up the fan and gloves, and, as the hall was very hot, she\r\nkept fanning herself all the time she went on talking: “Dear, dear! How\r\nqueer everything is to-day! And yesterday things went on just as usual.\r\nI wonder if I’ve been changed in the night? Let me think: was I the\r\nsame when I got up this morning? I almost think I can remember feeling\r\na little different. But if I’m not the same, the next question is, Who\r\nin the world am I? Ah, _that’s_ the great puzzle!” And she began\r\nthinking over all the children she knew that were of the same age as\r\nherself, to see if she could have been changed for any of them.\r\n\r\n“I’m sure I’m not Ada,” she said, “for her hair goes in such long\r\nringlets, and mine doesn’t go in ringlets at all; and I’m sure I can’t\r\nbe Mabel, for I know all sorts of things, and she, oh! she knows such a\r\nvery little! Besides, _she’s_ she, and _I’m_ I, and—oh dear, how\r\npuzzling it all is! I’ll try if I know all the things I used to know.\r\nLet me see: four times five is twelve, and four times six is thirteen,\r\nand four times seven is—oh dear! I shall never get to twenty at that\r\nrate! However, the Multiplication Table doesn’t signify: let’s try\r\nGeography. London is the capital of Paris, and Paris is the capital of\r\nRome, and Rome—no, _that’s_ all wrong, I’m certain! I must have been\r\nchanged for Mabel! I’ll try and say ‘_How doth the little_—’” and she\r\ncrossed her hands on her lap as if she were saying lessons, and began\r\nto repeat it, but her voice sounded hoarse and strange, and the words\r\ndid not come the same as they used to do:—\r\n\r\n“How doth the little crocodile\r\n    Improve his shining tail,\r\nAnd pour the waters of the Nile\r\n    On every golden scale!\r\n\r\n“How cheerfully he seems to grin,\r\n    How neatly spread his claws,\r\nAnd welcome little fishes in\r\n    With gently smiling jaws!”\r\n\r\n\r\n“I’m sure those are not the right words,” said poor Alice, and her eyes\r\nfilled with tears again as she went on, “I must be Mabel after all, and\r\nI shall have to go and live in that poky little house, and have next to\r\nno toys to play with, and oh! ever so many lessons to learn! No, I’ve\r\nmade up my mind about it; if I’m Mabel, I’ll stay down here! It’ll be\r\nno use their putting their heads down and saying ‘Come up again, dear!’\r\nI shall only look up and say ‘Who am I then? Tell me that first, and\r\nthen, if I like being that person, I’ll come up: if not, I’ll stay down\r\nhere till I’m somebody else’—but, oh dear!” cried Alice, with a sudden\r\nburst of tears, “I do wish they _would_ put their heads down! I am so\r\n_very_ tired of being all alone here!”\r\n\r\nAs she said this she looked down at her hands, and was surprised to see\r\nthat she had put on one of the Rabbit’s little white kid gloves while\r\nshe was talking. “How _can_ I have done that?” she thought. “I must be\r\ngrowing small again.” She got up and went to the table to measure\r\nherself by it, and found that, as nearly as she could guess, she was\r\nnow about two feet high, and was going on shrinking rapidly: she soon\r\nfound out that the cause of this was the fan she was holding, and she\r\ndropped it hastily, just in time to avoid shrinking away altogether.\r\n\r\n“That _was_ a narrow escape!” said Alice, a good deal frightened at the\r\nsudden change, but very glad to find herself still in existence; “and\r\nnow for the garden!” and she ran with all speed back to the little\r\ndoor: but, alas! the little door was shut again, and the little golden\r\nkey was lying on the glass table as before, “and things are worse than\r\never,” thought the poor child, “for I never was so small as this\r\nbefore, never! And I declare it’s too bad, that it is!”\r\n\r\nAs she said these words her foot slipped, and in another moment,\r\nsplash! she was up to her chin in salt water. Her first idea was that\r\nshe had somehow fallen into the sea, “and in that case I can go back by\r\nrailway,” she said to herself. (Alice had been to the seaside once in\r\nher life, and had come to the general conclusion, that wherever you go\r\nto on the English coast you find a number of bathing machines in the\r\nsea, some children digging in the sand with wooden spades, then a row\r\nof lodging houses, and behind them a railway station.) However, she\r\nsoon made out that she was in the pool of tears which she had wept when\r\nshe was nine feet high.\r\n\r\n“I wish I hadn’t cried so much!” said Alice, as she swam about, trying\r\nto find her way out. “I shall be punished for it now, I suppose, by\r\nbeing drowned in my own tears! That _will_ be a queer thing, to be\r\nsure! However, everything is queer to-day.”\r\n\r\nJust then she heard something splashing about in the pool a little way\r\noff, and she swam nearer to make out what it was: at first she thought\r\nit must be a walrus or hippopotamus, but then she remembered how small\r\nshe was now, and she soon made out that it was only a mouse that had\r\nslipped in like herself.\r\n\r\n“Would it be of any use, now,” thought Alice, “to speak to this mouse?\r\nEverything is so out-of-the-way down here, that I should think very\r\nlikely it can talk: at any rate, there’s no harm in trying.” So she\r\nbegan: “O Mouse, do you know the way out of this pool? I am very tired\r\nof swimming about here, O Mouse!” (Alice thought this must be the right\r\nway of speaking to a mouse: she had never done such a thing before, but\r\nshe remembered having seen in her brother’s Latin Grammar, “A mouse—of\r\na mouse—to a mouse—a mouse—O mouse!”) The Mouse looked at her rather\r\ninquisitively, and seemed to her to wink with one of its little eyes,\r\nbut it said nothing.\r\n\r\n“Perhaps it doesn’t understand English,” thought Alice; “I daresay it’s\r\na French mouse, come over with William the Conqueror.” (For, with all\r\nher knowledge of history, Alice had no very clear notion how long ago\r\nanything had happened.) So she began again: “Où est ma chatte?” which\r\nwas the first sentence in her French lesson-book. The Mouse gave a\r\nsudden leap out of the water, and seemed to quiver all over with\r\nfright. “Oh, I beg your pardon!” cried Alice hastily, afraid that she\r\nhad hurt the poor animal’s feelings. “I quite forgot you didn’t like\r\ncats.”\r\n\r\n“Not like cats!” cried the Mouse, in a shrill, passionate voice. “Would\r\n_you_ like cats if you were me?”\r\n\r\n“Well, perhaps not,” said Alice in a soothing tone: “don’t be angry\r\nabout it. And yet I wish I could show you our cat Dinah: I think you’d\r\ntake a fancy to cats if you could only see her. She is such a dear\r\nquiet thing,” Alice went on, half to herself, as she swam lazily about\r\nin the pool, “and she sits purring so nicely by the fire, licking her\r\npaws and washing her face—and she is such a nice soft thing to\r\nnurse—and she’s such a capital one for catching mice—oh, I beg your\r\npardon!” cried Alice again, for this time the Mouse was bristling all\r\nover, and she felt certain it must be really offended. “We won’t talk\r\nabout her any more if you’d rather not.”\r\n\r\n“We indeed!” cried the Mouse, who was trembling down to the end of his\r\ntail. “As if _I_ would talk on such a subject! Our family always\r\n_hated_ cats: nasty, low, vulgar things! Don’t let me hear the name\r\nagain!”\r\n\r\n“I won’t indeed!” said Alice, in a great hurry to change the subject of\r\nconversation. “Are you—are you fond—of—of dogs?” The Mouse did not\r\nanswer, so Alice went on eagerly: “There is such a nice little dog near\r\nour house I should like to show you! A little bright-eyed terrier, you\r\nknow, with oh, such long curly brown hair! And it’ll fetch things when\r\nyou throw them, and it’ll sit up and beg for its dinner, and all sorts\r\nof things—I can’t remember half of them—and it belongs to a farmer, you\r\nknow, and he says it’s so useful, it’s worth a hundred pounds! He says\r\nit kills all the rats and—oh dear!” cried Alice in a sorrowful tone,\r\n“I’m afraid I’ve offended it again!” For the Mouse was swimming away\r\nfrom her as hard as it could go, and making quite a commotion in the\r\npool as it went.\r\n\r\nSo she called softly after it, “Mouse dear! Do come back again, and we\r\nwon’t talk about cats or dogs either, if you don’t like them!” When the\r\nMouse heard this, it turned round and swam slowly back to her: its face\r\nwas quite pale (with passion, Alice thought), and it said in a low\r\ntrembling voice, “Let us get to the shore, and then I’ll tell you my\r\nhistory, and you’ll understand why it is I hate cats and dogs.”\r\n\r\nIt was high time to go, for the pool was getting quite crowded with the\r\nbirds and animals that had fallen into it: there were a Duck and a\r\nDodo, a Lory and an Eaglet, and several other curious creatures. Alice\r\nled the way, and the whole party swam to the shore.\r\n\r\n\r\n\r\n\r\nCHAPTER III.\r\nA Caucus-Race and a Long Tale\r\n\r\n\r\nThey were indeed a queer-looking party that assembled on the bank—the\r\nbirds with draggled feathers, the animals with their fur clinging close\r\nto them, and all dripping wet, cross, and uncomfortable.\r\n\r\nThe first question of course was, how to get dry again: they had a\r\nconsultation about this, and after a few minutes it seemed quite\r\nnatural to Alice to find herself talking familiarly with them, as if\r\nshe had known them all her life. Indeed, she had quite a long argument\r\nwith the Lory, who at last turned sulky, and would only say, “I am\r\nolder than you, and must know better;” and this Alice would not allow\r\nwithout knowing how old it was, and, as the Lory positively refused to\r\ntell its age, there was no more to be said.\r\n\r\nAt last the Mouse, who seemed to be a person of authority among them,\r\ncalled out, “Sit down, all of you, and listen to me! _I’ll_ soon make\r\nyou dry enough!” They all sat down at once, in a large ring, with the\r\nMouse in the middle. Alice kept her eyes anxiously fixed on it, for she\r\nfelt sure she would catch a bad cold if she did not get dry very soon.\r\n\r\n“Ahem!” said the Mouse with an important air, “are you all ready? This\r\nis the driest thing I know. Silence all round, if you please! ‘William\r\nthe Conqueror, whose cause was favoured by the pope, was soon submitted\r\nto by the English, who wanted leaders, and had been of late much\r\naccustomed to usurpation and conquest. Edwin and Morcar, the earls of\r\nMercia and Northumbria—’”\r\n\r\n“Ugh!” said the Lory, with a shiver.\r\n\r\n“I beg your pardon!” said the Mouse, frowning, but very politely: “Did\r\nyou speak?”\r\n\r\n“Not I!” said the Lory hastily.\r\n\r\n“I thought you did,” said the Mouse. “—I proceed. ‘Edwin and Morcar,\r\nthe earls of Mercia and Northumbria, declared for him: and even\r\nStigand, the patriotic archbishop of Canterbury, found it advisable—’”\r\n\r\n“Found _what_?” said the Duck.\r\n\r\n“Found _it_,” the Mouse replied rather crossly: “of course you know\r\nwhat ‘it’ means.”\r\n\r\n“I know what ‘it’ means well enough, when _I_ find a thing,” said the\r\nDuck: “it’s generally a frog or a worm. The question is, what did the\r\narchbishop find?”\r\n\r\nThe Mouse did not notice this question, but hurriedly went on, “‘—found\r\nit advisable to go with Edgar Atheling to meet William and offer him\r\nthe crown. William’s conduct at first was moderate. But the insolence\r\nof his Normans—’ How are you getting on now, my dear?” it continued,\r\nturning to Alice as it spoke.\r\n\r\n“As wet as ever,” said Alice in a melancholy tone: “it doesn’t seem to\r\ndry me at all.”\r\n\r\n“In that case,” said the Dodo solemnly, rising to its feet, “I move\r\nthat the meeting adjourn, for the immediate adoption of more energetic\r\nremedies—”\r\n\r\n“Speak English!” said the Eaglet. “I don’t know the meaning of half\r\nthose long words, and, what’s more, I don’t believe you do either!” And\r\nthe Eaglet bent down its head to hide a smile: some of the other birds\r\ntittered audibly.\r\n\r\n“What I was going to say,” said the Dodo in an offended tone, “was,\r\nthat the best thing to get us dry would be a Caucus-race.”\r\n\r\n“What _is_ a Caucus-race?” said Alice; not that she wanted much to\r\nknow, but the Dodo had paused as if it thought that _somebody_ ought to\r\nspeak, and no one else seemed inclined to say anything.\r\n\r\n“Why,” said the Dodo, “the best way to explain it is to do it.” (And,\r\nas you might like to try the thing yourself, some winter day, I will\r\ntell you how the Dodo managed it.)\r\n\r\nFirst it marked out a race-course, in a sort of circle, (“the exact\r\nshape doesn’t matter,” it said,) and then all the party were placed\r\nalong the course, here and there. There was no “One, two, three, and\r\naway,” but they began running when they liked, and left off when they\r\nliked, so that it was not easy to know when the race was over. However,\r\nwhen they had been running half an hour or so, and were quite dry\r\nagain, the Dodo suddenly called out “The race is over!” and they all\r\ncrowded round it, panting, and asking, “But who has won?”\r\n\r\nThis question the Dodo could not answer without a great deal of\r\nthought, and it sat for a long time with one finger pressed upon its\r\nforehead (the position in which you usually see Shakespeare, in the\r\npictures of him), while the rest waited in silence. At last the Dodo\r\nsaid, “_Everybody_ has won, and all must have prizes.”\r\n\r\n“But who is to give the prizes?” quite a chorus of voices asked.\r\n\r\n“Why, _she_, of course,” said the Dodo, pointing to Alice with one\r\nfinger; and the whole party at once crowded round her, calling out in a\r\nconfused way, “Prizes! Prizes!”\r\n\r\nAlice had no idea what to do, and in despair she put her hand in her\r\npocket, and pulled out a box of comfits, (luckily the salt water had\r\nnot got into it), and handed them round as prizes. There was exactly\r\none a-piece, all round.\r\n\r\n“But she must have a prize herself, you know,” said the Mouse.\r\n\r\n“Of course,” the Dodo replied very gravely. “What else have you got in\r\nyour pocket?” he went on, turning to Alice.\r\n\r\n“Only a thimble,” said Alice sadly.\r\n\r\n“Hand it over here,” said the Dodo.\r\n\r\nThen they all crowded round her once more, while the Dodo solemnly\r\npresented the thimble, saying “We beg your acceptance of this elegant\r\nthimble;” and, when it had finished this short speech, they all\r\ncheered.\r\n\r\nAlice thought the whole thing very absurd, but they all looked so grave\r\nthat she did not dare to laugh; and, as she could not think of anything\r\nto say, she simply bowed, and took the thimble, looking as solemn as\r\nshe could.\r\n\r\nThe next thing was to eat the comfits: this caused some noise and\r\nconfusion, as the large birds complained that they could not taste\r\ntheirs, and the small ones choked and had to be patted on the back.\r\nHowever, it was over at last, and they sat down again in a ring, and\r\nbegged the Mouse to tell them something more.\r\n\r\n“You promised to tell me your history, you know,” said Alice, “and why\r\nit is you hate—C and D,” she added in a whisper, half afraid that it\r\nwould be offended again.\r\n\r\n“Mine is a long and a sad tale!” said the Mouse, turning to Alice, and\r\nsighing.\r\n\r\n“It _is_ a long tail, certainly,” said Alice, looking down with wonder\r\nat the Mouse’s tail; “but why do you call it sad?” And she kept on\r\npuzzling about it while the Mouse was speaking, so that her idea of the\r\ntale was something like this:—\r\n\r\n         “Fury said to a mouse, That he met in the house, ‘Let us both\r\n         go to law: _I_ will prosecute _you_.—Come, I’ll take no\r\n         denial; We must have a trial: For really this morning I’ve\r\n         nothing to do.’ Said the mouse to the cur, ‘Such a trial, dear\r\n         sir, With no jury or judge, would be wasting our breath.’\r\n         ‘I’ll be judge, I’ll be jury,’ Said cunning old Fury: ‘I’ll\r\n         try the whole cause, and condemn you to death.’”\r\n\r\n“You are not attending!” said the Mouse to Alice severely. “What are\r\nyou thinking of?”\r\n\r\n“I beg your pardon,” said Alice very humbly: “you had got to the fifth\r\nbend, I think?”\r\n\r\n“I had _not!_” cried the Mouse, sharply and very angrily.\r\n\r\n“A knot!” said Alice, always ready to make herself useful, and looking\r\nanxiously about her. “Oh, do let me help to undo it!”\r\n\r\n“I shall do nothing of the sort,” said the Mouse, getting up and\r\nwalking away. “You insult me by talking such nonsense!”\r\n\r\n“I didn’t mean it!” pleaded poor Alice. “But you’re so easily offended,\r\nyou know!”\r\n\r\nThe Mouse only growled in reply.\r\n\r\n“Please come back and finish your story!” Alice called after it; and\r\nthe others all joined in chorus, “Yes, please do!” but the Mouse only\r\nshook its head impatiently, and walked a little quicker.\r\n\r\n“What a pity it wouldn’t stay!” sighed the Lory, as soon as it was\r\nquite out of sight; and an old Crab took the opportunity of saying to\r\nher daughter “Ah, my dear! Let this be a lesson to you never to lose\r\n_your_ temper!” “Hold your tongue, Ma!” said the young Crab, a little\r\nsnappishly. “You’re enough to try the patience of an oyster!”\r\n\r\n“I wish I had our Dinah here, I know I do!” said Alice aloud,\r\naddressing nobody in particular. “She’d soon fetch it back!”\r\n\r\n“And who is Dinah, if I might venture to ask the question?” said the\r\nLory.\r\n\r\nAlice replied eagerly, for she was always ready to talk about her pet:\r\n“Dinah’s our cat. And she’s such a capital one for catching mice you\r\ncan’t think! And oh, I wish you could see her after the birds! Why,\r\nshe’ll eat a little bird as soon as look at it!”\r\n\r\nThis speech caused a remarkable sensation among the party. Some of the\r\nbirds hurried off at once: one old Magpie began wrapping itself up very\r\ncarefully, remarking, “I really must be getting home; the night-air\r\ndoesn’t suit my throat!” and a Canary called out in a trembling voice\r\nto its children, “Come away, my dears! It’s high time you were all in\r\nbed!” On various pretexts they all moved off, and Alice was soon left\r\nalone.\r\n\r\n“I wish I hadn’t mentioned Dinah!” she said to herself in a melancholy\r\ntone. “Nobody seems to like her, down here, and I’m sure she’s the best\r\ncat in the world! Oh, my dear Dinah! I wonder if I shall ever see you\r\nany more!” And here poor Alice began to cry again, for she felt very\r\nlonely and low-spirited. In a little while, however, she again heard a\r\nlittle pattering of footsteps in the distance, and she looked up\r\neagerly, half hoping that the Mouse had changed his mind, and was\r\ncoming back to finish his story.\r\n\r\n\r\n\r\n\r\nCHAPTER IV.\r\nThe Rabbit Sends in a Little Bill\r\n\r\n\r\nIt was the White Rabbit, trotting slowly back again, and looking\r\nanxiously about as it went, as if it had lost something; and she heard\r\nit muttering to itself “The Duchess! The Duchess! Oh my dear paws! Oh\r\nmy fur and whiskers! She’ll get me executed, as sure as ferrets are\r\nferrets! Where _can_ I have dropped them, I wonder?” Alice guessed in a\r\nmoment that it was looking for the fan and the pair of white kid\r\ngloves, and she very good-naturedly began hunting about for them, but\r\nthey were nowhere to be seen—everything seemed to have changed since\r\nher swim in the pool, and the great hall, with the glass table and the\r\nlittle door, had vanished completely.\r\n\r\nVery soon the Rabbit noticed Alice, as she went hunting about, and\r\ncalled out to her in an angry tone, “Why, Mary Ann, what _are_ you\r\ndoing out here? Run home this moment, and fetch me a pair of gloves and\r\na fan! Quick, now!” And Alice was so much frightened that she ran off\r\nat once in the direction it pointed to, without trying to explain the\r\nmistake it had made.\r\n\r\n“He took me for his housemaid,” she said to herself as she ran. “How\r\nsurprised he’ll be when he finds out who I am! But I’d better take him\r\nhis fan and gloves—that is, if I can find them.” As she said this, she\r\ncame upon a neat little house, on the door of which was a bright brass\r\nplate with the name “W. RABBIT,” engraved upon it. She went in without\r\nknocking, and hurried upstairs, in great fear lest she should meet the\r\nreal Mary Ann, and be turned out of the house before she had found the\r\nfan and gloves.\r\n\r\n“How queer it seems,” Alice said to herself, “to be going messages for\r\na rabbit! I suppose Dinah’ll be sending me on messages next!” And she\r\nbegan fancying the sort of thing that would happen: “‘Miss Alice! Come\r\nhere directly, and get ready for your walk!’ ‘Coming in a minute,\r\nnurse! But I’ve got to see that the mouse doesn’t get out.’ Only I\r\ndon’t think,” Alice went on, “that they’d let Dinah stop in the house\r\nif it began ordering people about like that!”\r\n\r\nBy this time she had found her way into a tidy little room with a table\r\nin the window, and on it (as she had hoped) a fan and two or three\r\npairs of tiny white kid gloves: she took up the fan and a pair of the\r\ngloves, and was just going to leave the room, when her eye fell upon a\r\nlittle bottle that stood near the looking-glass. There was no label\r\nthis time with the words “DRINK ME,” but nevertheless she uncorked it\r\nand put it to her lips. “I know _something_ interesting is sure to\r\nhappen,” she said to herself, “whenever I eat or drink anything; so\r\nI’ll just see what this bottle does. I do hope it’ll make me grow large\r\nagain, for really I’m quite tired of being such a tiny little thing!”\r\n\r\nIt did so indeed, and much sooner than she had expected: before she had\r\ndrunk half the bottle, she found her head pressing against the ceiling,\r\nand had to stoop to save her neck from being broken. She hastily put\r\ndown the bottle, saying to herself “That’s quite enough—I hope I shan’t\r\ngrow any more—As it is, I can’t get out at the door—I do wish I hadn’t\r\ndrunk quite so much!”\r\n\r\nAlas! it was too late to wish that! She went on growing, and growing,\r\nand very soon had to kneel down on the floor: in another minute there\r\nwas not even room for this, and she tried the effect of lying down with\r\none elbow against the door, and the other arm curled round her head.\r\nStill she went on growing, and, as a last resource, she put one arm out\r\nof the window, and one foot up the chimney, and said to herself “Now I\r\ncan do no more, whatever happens. What _will_ become of me?”\r\n\r\nLuckily for Alice, the little magic bottle had now had its full effect,\r\nand she grew no larger: still it was very uncomfortable, and, as there\r\nseemed to be no sort of chance of her ever getting out of the room\r\nagain, no wonder she felt unhappy.\r\n\r\n“It was much pleasanter at home,” thought poor Alice, “when one wasn’t\r\nalways growing larger and smaller, and being ordered about by mice and\r\nrabbits. I almost wish I hadn’t gone down that rabbit-hole—and yet—and\r\nyet—it’s rather curious, you know, this sort of life! I do wonder what\r\n_can_ have happened to me! When I used to read fairy-tales, I fancied\r\nthat kind of thing never happened, and now here I am in the middle of\r\none! There ought to be a book written about me, that there ought! And\r\nwhen I grow up, I’ll write one—but I’m grown up now,” she added in a\r\nsorrowful tone; “at least there’s no room to grow up any more _here_.”\r\n\r\n“But then,” thought Alice, “shall I _never_ get any older than I am\r\nnow? That’ll be a comfort, one way—never to be an old woman—but\r\nthen—always to have lessons to learn! Oh, I shouldn’t like _that!_”\r\n\r\n“Oh, you foolish Alice!” she answered herself. “How can you learn\r\nlessons in here? Why, there’s hardly room for _you_, and no room at all\r\nfor any lesson-books!”\r\n\r\nAnd so she went on, taking first one side and then the other, and\r\nmaking quite a conversation of it altogether; but after a few minutes\r\nshe heard a voice outside, and stopped to listen.\r\n\r\n“Mary Ann! Mary Ann!” said the voice. “Fetch me my gloves this moment!”\r\nThen came a little pattering of feet on the stairs. Alice knew it was\r\nthe Rabbit coming to look for her, and she trembled till she shook the\r\nhouse, quite forgetting that she was now about a thousand times as\r\nlarge as the Rabbit, and had no reason to be afraid of it.\r\n\r\nPresently the Rabbit came up to the door, and tried to open it; but, as\r\nthe door opened inwards, and Alice’s elbow was pressed hard against it,\r\nthat attempt proved a failure. Alice heard it say to itself “Then I’ll\r\ngo round and get in at the window.”\r\n\r\n“_That_ you won’t!” thought Alice, and, after waiting till she fancied\r\nshe heard the Rabbit just under the window, she suddenly spread out her\r\nhand, and made a snatch in the air. She did not get hold of anything,\r\nbut she heard a little shriek and a fall, and a crash of broken glass,\r\nfrom which she concluded that it was just possible it had fallen into a\r\ncucumber-frame, or something of the sort.\r\n\r\nNext came an angry voice—the Rabbit’s—“Pat! Pat! Where are you?” And\r\nthen a voice she had never heard before, “Sure then I’m here! Digging\r\nfor apples, yer honour!”\r\n\r\n“Digging for apples, indeed!” said the Rabbit angrily. “Here! Come and\r\nhelp me out of _this!_” (Sounds of more broken glass.)\r\n\r\n“Now tell me, Pat, what’s that in the window?”\r\n\r\n“Sure, it’s an arm, yer honour!” (He pronounced it “arrum.”)\r\n\r\n“An arm, you goose! Who ever saw one that size? Why, it fills the whole\r\nwindow!”\r\n\r\n“Sure, it does, yer honour: but it’s an arm for all that.”\r\n\r\n“Well, it’s got no business there, at any rate: go and take it away!”\r\n\r\nThere was a long silence after this, and Alice could only hear whispers\r\nnow and then; such as, “Sure, I don’t like it, yer honour, at all, at\r\nall!” “Do as I tell you, you coward!” and at last she spread out her\r\nhand again, and made another snatch in the air. This time there were\r\n_two_ little shrieks, and more sounds of broken glass. “What a number\r\nof cucumber-frames there must be!” thought Alice. “I wonder what\r\nthey’ll do next! As for pulling me out of the window, I only wish they\r\n_could!_ I’m sure _I_ don’t want to stay in here any longer!”\r\n\r\nShe waited for some time without hearing anything more: at last came a\r\nrumbling of little cartwheels, and the sound of a good many voices all\r\ntalking together: she made out the words: “Where’s the other\r\nladder?—Why, I hadn’t to bring but one; Bill’s got the other—Bill!\r\nfetch it here, lad!—Here, put ’em up at this corner—No, tie ’em\r\ntogether first—they don’t reach half high enough yet—Oh! they’ll do\r\nwell enough; don’t be particular—Here, Bill! catch hold of this\r\nrope—Will the roof bear?—Mind that loose slate—Oh, it’s coming down!\r\nHeads below!” (a loud crash)—“Now, who did that?—It was Bill, I\r\nfancy—Who’s to go down the chimney?—Nay, _I_ shan’t! _You_ do\r\nit!—_That_ I won’t, then!—Bill’s to go down—Here, Bill! the master says\r\nyou’re to go down the chimney!”\r\n\r\n“Oh! So Bill’s got to come down the chimney, has he?” said Alice to\r\nherself. “Shy, they seem to put everything upon Bill! I wouldn’t be in\r\nBill’s place for a good deal: this fireplace is narrow, to be sure; but\r\nI _think_ I can kick a little!”\r\n\r\nShe drew her foot as far down the chimney as she could, and waited till\r\nshe heard a little animal (she couldn’t guess of what sort it was)\r\nscratching and scrambling about in the chimney close above her: then,\r\nsaying to herself “This is Bill,” she gave one sharp kick, and waited\r\nto see what would happen next.\r\n\r\nThe first thing she heard was a general chorus of “There goes Bill!”\r\nthen the Rabbit’s voice along—“Catch him, you by the hedge!” then\r\nsilence, and then another confusion of voices—“Hold up his head—Brandy\r\nnow—Don’t choke him—How was it, old fellow? What happened to you? Tell\r\nus all about it!”\r\n\r\nLast came a little feeble, squeaking voice, (“That’s Bill,” thought\r\nAlice,) “Well, I hardly know—No more, thank ye; I’m better now—but I’m\r\na deal too flustered to tell you—all I know is, something comes at me\r\nlike a Jack-in-the-box, and up I goes like a sky-rocket!”\r\n\r\n“So you did, old fellow!” said the others.\r\n\r\n“We must burn the house down!” said the Rabbit’s voice; and Alice\r\ncalled out as loud as she could, “If you do, I’ll set Dinah at you!”\r\n\r\nThere was a dead silence instantly, and Alice thought to herself, “I\r\nwonder what they _will_ do next! If they had any sense, they’d take the\r\nroof off.” After a minute or two, they began moving about again, and\r\nAlice heard the Rabbit say, “A barrowful will do, to begin with.”\r\n\r\n“A barrowful of _what?_” thought Alice; but she had not long to doubt,\r\nfor the next moment a shower of little pebbles came rattling in at the\r\nwindow, and some of them hit her in the face. “I’ll put a stop to\r\nthis,” she said to herself, and shouted out, “You’d better not do that\r\nagain!” which produced another dead silence.\r\n\r\nAlice noticed with some surprise that the pebbles were all turning into\r\nlittle cakes as they lay on the floor, and a bright idea came into her\r\nhead. “If I eat one of these cakes,” she thought, “it’s sure to make\r\n_some_ change in my size; and as it can’t possibly make me larger, it\r\nmust make me smaller, I suppose.”\r\n\r\nSo she swallowed one of the cakes, and was delighted to find that she\r\nbegan shrinking directly. As soon as she was small enough to get\r\nthrough the door, she ran out of the house, and found quite a crowd of\r\nlittle animals and birds waiting outside. The poor little Lizard, Bill,\r\nwas in the middle, being held up by two guinea-pigs, who were giving it\r\nsomething out of a bottle. They all made a rush at Alice the moment she\r\nappeared; but she ran off as hard as she could, and soon found herself\r\nsafe in a thick wood.\r\n\r\n“The first thing I’ve got to do,” said Alice to herself, as she\r\nwandered about in the wood, “is to grow to my right size again; and the\r\nsecond thing is to find my way into that lovely garden. I think that\r\nwill be the best plan.”\r\n\r\nIt sounded an excellent plan, no doubt, and very neatly and simply\r\narranged; the only difficulty was, that she had not the smallest idea\r\nhow to set about it; and while she was peering about anxiously among\r\nthe trees, a little sharp bark just over her head made her look up in a\r\ngreat hurry.\r\n\r\nAn enormous puppy was looking down at her with large round eyes, and\r\nfeebly stretching out one paw, trying to touch her. “Poor little\r\nthing!” said Alice, in a coaxing tone, and she tried hard to whistle to\r\nit; but she was terribly frightened all the time at the thought that it\r\nmight be hungry, in which case it would be very likely to eat her up in\r\nspite of all her coaxing.\r\n\r\nHardly knowing what she did, she picked up a little bit of stick, and\r\nheld it out to the puppy; whereupon the puppy jumped into the air off\r\nall its feet at once, with a yelp of delight, and rushed at the stick,\r\nand made believe to worry it; then Alice dodged behind a great thistle,\r\nto keep herself from being run over; and the moment she appeared on the\r\nother side, the puppy made another rush at the stick, and tumbled head\r\nover heels in its hurry to get hold of it; then Alice, thinking it was\r\nvery like having a game of play with a cart-horse, and expecting every\r\nmoment to be trampled under its feet, ran round the thistle again; then\r\nthe puppy began a series of short charges at the stick, running a very\r\nlittle way forwards each time and a long way back, and barking hoarsely\r\nall the while, till at last it sat down a good way off, panting, with\r\nits tongue hanging out of its mouth, and its great eyes half shut.\r\n\r\nThis seemed to Alice a good opportunity for making her escape; so she\r\nset off at once, and ran till she was quite tired and out of breath,\r\nand till the puppy’s bark sounded quite faint in the distance.\r\n\r\n“And yet what a dear little puppy it was!” said Alice, as she leant\r\nagainst a buttercup to rest herself, and fanned herself with one of the\r\nleaves: “I should have liked teaching it tricks very much, if—if I’d\r\nonly been the right size to do it! Oh dear! I’d nearly forgotten that\r\nI’ve got to grow up again! Let me see—how _is_ it to be managed? I\r\nsuppose I ought to eat or drink something or other; but the great\r\nquestion is, what?”\r\n\r\nThe great question certainly was, what? Alice looked all round her at\r\nthe flowers and the blades of grass, but she did not see anything that\r\nlooked like the right thing to eat or drink under the circumstances.\r\nThere was a large mushroom growing near her, about the same height as\r\nherself; and when she had looked under it, and on both sides of it, and\r\nbehind it, it occurred to her that she might as well look and see what\r\nwas on the top of it.\r\n\r\nShe stretched herself up on tiptoe, and peeped over the edge of the\r\nmushroom, and her eyes immediately met those of a large blue\r\ncaterpillar, that was sitting on the top with its arms folded, quietly\r\nsmoking a long hookah, and taking not the smallest notice of her or of\r\nanything else.\r\n\r\n\r\n\r\n\r\nCHAPTER V.\r\nAdvice from a Caterpillar\r\n\r\n\r\nThe Caterpillar and Alice looked at each other for some time in\r\nsilence: at last the Caterpillar took the hookah out of its mouth, and\r\naddressed her in a languid, sleepy voice.\r\n\r\n“Who are _you?_” said the Caterpillar.\r\n\r\nThis was not an encouraging opening for a conversation. Alice replied,\r\nrather shyly, “I—I hardly know, sir, just at present—at least I know\r\nwho I _was_ when I got up this morning, but I think I must have been\r\nchanged several times since then.”\r\n\r\n“What do you mean by that?” said the Caterpillar sternly. “Explain\r\nyourself!”\r\n\r\n“I can’t explain _myself_, I’m afraid, sir,” said Alice, “because I’m\r\nnot myself, you see.”\r\n\r\n“I don’t see,” said the Caterpillar.\r\n\r\n“I’m afraid I can’t put it more clearly,” Alice replied very politely,\r\n“for I can’t understand it myself to begin with; and being so many\r\ndifferent sizes in a day is very confusing.”\r\n\r\n“It isn’t,” said the Caterpillar.\r\n\r\n“Well, perhaps you haven’t found it so yet,” said Alice; “but when you\r\nhave to turn into a chrysalis—you will some day, you know—and then\r\nafter that into a butterfly, I should think you’ll feel it a little\r\nqueer, won’t you?”\r\n\r\n“Not a bit,” said the Caterpillar.\r\n\r\n“Well, perhaps your feelings may be different,” said Alice; “all I know\r\nis, it would feel very queer to _me_.”\r\n\r\n“You!” said the Caterpillar contemptuously. “Who are _you?_”\r\n\r\nWhich brought them back again to the beginning of the conversation.\r\nAlice felt a little irritated at the Caterpillar’s making such _very_\r\nshort remarks, and she drew herself up and said, very gravely, “I\r\nthink, you ought to tell me who _you_ are, first.”\r\n\r\n“Why?” said the Caterpillar.\r\n\r\nHere was another puzzling question; and as Alice could not think of any\r\ngood reason, and as the Caterpillar seemed to be in a _very_ unpleasant\r\nstate of mind, she turned away.\r\n\r\n“Come back!” the Caterpillar called after her. “I’ve something\r\nimportant to say!”\r\n\r\nThis sounded promising, certainly: Alice turned and came back again.\r\n\r\n“Keep your temper,” said the Caterpillar.\r\n\r\n“Is that all?” said Alice, swallowing down her anger as well as she\r\ncould.\r\n\r\n“No,” said the Caterpillar.\r\n\r\nAlice thought she might as well wait, as she had nothing else to do,\r\nand perhaps after all it might tell her something worth hearing. For\r\nsome minutes it puffed away without speaking, but at last it unfolded\r\nits arms, took the hookah out of its mouth again, and said, “So you\r\nthink you’re changed, do you?”\r\n\r\n“I’m afraid I am, sir,” said Alice; “I can’t remember things as I\r\nused—and I don’t keep the same size for ten minutes together!”\r\n\r\n“Can’t remember _what_ things?” said the Caterpillar.\r\n\r\n“Well, I’ve tried to say “How doth the little busy bee,” but it all\r\ncame different!” Alice replied in a very melancholy voice.\r\n\r\n“Repeat, “_You are old, Father William_,’” said the Caterpillar.\r\n\r\nAlice folded her hands, and began:—\r\n\r\n“You are old, Father William,” the young man said,\r\n    “And your hair has become very white;\r\nAnd yet you incessantly stand on your head—\r\n    Do you think, at your age, it is right?”\r\n\r\n“In my youth,” Father William replied to his son,\r\n    “I feared it might injure the brain;\r\nBut, now that I’m perfectly sure I have none,\r\n    Why, I do it again and again.”\r\n\r\n“You are old,” said the youth, “as I mentioned before,\r\n    And have grown most uncommonly fat;\r\nYet you turned a back-somersault in at the door—\r\n    Pray, what is the reason of that?”\r\n\r\n“In my youth,” said the sage, as he shook his grey locks,\r\n    “I kept all my limbs very supple\r\nBy the use of this ointment—one shilling the box—\r\n    Allow me to sell you a couple?”\r\n\r\n“You are old,” said the youth, “and your jaws are too weak\r\n    For anything tougher than suet;\r\nYet you finished the goose, with the bones and the beak—\r\n    Pray, how did you manage to do it?”\r\n\r\n“In my youth,” said his father, “I took to the law,\r\n    And argued each case with my wife;\r\nAnd the muscular strength, which it gave to my jaw,\r\n    Has lasted the rest of my life.”\r\n\r\n“You are old,” said the youth, “one would hardly suppose\r\n    That your eye was as steady as ever;\r\nYet you balanced an eel on the end of your nose—\r\n    What made you so awfully clever?”\r\n\r\n“I have answered three questions, and that is enough,”\r\n    Said his father; “don’t give yourself airs!\r\nDo you think I can listen all day to such stuff?\r\n    Be off, or I’ll kick you down stairs!”\r\n\r\n\r\n“That is not said right,” said the Caterpillar.\r\n\r\n“Not _quite_ right, I’m afraid,” said Alice, timidly; “some of the\r\nwords have got altered.”\r\n\r\n“It is wrong from beginning to end,” said the Caterpillar decidedly,\r\nand there was silence for some minutes.\r\n\r\nThe Caterpillar was the first to speak.\r\n\r\n“What size do you want to be?” it asked.\r\n\r\n“Oh, I’m not particular as to size,” Alice hastily replied; “only one\r\ndoesn’t like changing so often, you know.”\r\n\r\n“I _don’t_ know,” said the Caterpillar.\r\n\r\nAlice said nothing: she had never been so much contradicted in her life\r\nbefore, and she felt that she was losing her temper.\r\n\r\n“Are you content now?” said the Caterpillar.\r\n\r\n“Well, I should like to be a _little_ larger, sir, if you wouldn’t\r\nmind,” said Alice: “three inches is such a wretched height to be.”\r\n\r\n“It is a very good height indeed!” said the Caterpillar angrily,\r\nrearing itself upright as it spoke (it was exactly three inches high).\r\n\r\n“But I’m not used to it!” pleaded poor Alice in a piteous tone. And she\r\nthought of herself, “I wish the creatures wouldn’t be so easily\r\noffended!”\r\n\r\n“You’ll get used to it in time,” said the Caterpillar; and it put the\r\nhookah into its mouth and began smoking again.\r\n\r\nThis time Alice waited patiently until it chose to speak again. In a\r\nminute or two the Caterpillar took the hookah out of its mouth and\r\nyawned once or twice, and shook itself. Then it got down off the\r\nmushroom, and crawled away in the grass, merely remarking as it went,\r\n“One side will make you grow taller, and the other side will make you\r\ngrow shorter.”\r\n\r\n“One side of _what?_ The other side of _what?_” thought Alice to\r\nherself.\r\n\r\n“Of the mushroom,” said the Caterpillar, just as if she had asked it\r\naloud; and in another moment it was out of sight.\r\n\r\nAlice remained looking thoughtfully at the mushroom for a minute,\r\ntrying to make out which were the two sides of it; and as it was\r\nperfectly round, she found this a very difficult question. However, at\r\nlast she stretched her arms round it as far as they would go, and broke\r\noff a bit of the edge with each hand.\r\n\r\n“And now which is which?” she said to herself, and nibbled a little of\r\nthe right-hand bit to try the effect: the next moment she felt a\r\nviolent blow underneath her chin: it had struck her foot!\r\n\r\nShe was a good deal frightened by this very sudden change, but she felt\r\nthat there was no time to be lost, as she was shrinking rapidly; so she\r\nset to work at once to eat some of the other bit. Her chin was pressed\r\nso closely against her foot, that there was hardly room to open her\r\nmouth; but she did it at last, and managed to swallow a morsel of the\r\nlefthand bit.\r\n\r\n*      *      *      *      *      *      *\r\n\r\n    *      *      *      *      *      *\r\n\r\n*      *      *      *      *      *      *\r\n\r\n\r\n“Come, my head’s free at last!” said Alice in a tone of delight, which\r\nchanged into alarm in another moment, when she found that her shoulders\r\nwere nowhere to be found: all she could see, when she looked down, was\r\nan immense length of neck, which seemed to rise like a stalk out of a\r\nsea of green leaves that lay far below her.\r\n\r\n“What _can_ all that green stuff be?” said Alice. “And where _have_ my\r\nshoulders got to? And oh, my poor hands, how is it I can’t see you?”\r\nShe was moving them about as she spoke, but no result seemed to follow,\r\nexcept a little shaking among the distant green leaves.\r\n\r\nAs there seemed to be no chance of getting her hands up to her head,\r\nshe tried to get her head down to them, and was delighted to find that\r\nher neck would bend about easily in any direction, like a serpent. She\r\nhad just succeeded in curving it down into a graceful zigzag, and was\r\ngoing to dive in among the leaves, which she found to be nothing but\r\nthe tops of the trees under which she had been wandering, when a sharp\r\nhiss made her draw back in a hurry: a large pigeon had flown into her\r\nface, and was beating her violently with its wings.\r\n\r\n“Serpent!” screamed the Pigeon.\r\n\r\n“I’m _not_ a serpent!” said Alice indignantly. “Let me alone!”\r\n\r\n“Serpent, I say again!” repeated the Pigeon, but in a more subdued\r\ntone, and added with a kind of sob, “I’ve tried every way, and nothing\r\nseems to suit them!”\r\n\r\n“I haven’t the least idea what you’re talking about,” said Alice.\r\n\r\n“I’ve tried the roots of trees, and I’ve tried banks, and I’ve tried\r\nhedges,” the Pigeon went on, without attending to her; “but those\r\nserpents! There’s no pleasing them!”\r\n\r\nAlice was more and more puzzled, but she thought there was no use in\r\nsaying anything more till the Pigeon had finished.\r\n\r\n“As if it wasn’t trouble enough hatching the eggs,” said the Pigeon;\r\n“but I must be on the look-out for serpents night and day! Why, I\r\nhaven’t had a wink of sleep these three weeks!”\r\n\r\n“I’m very sorry you’ve been annoyed,” said Alice, who was beginning to\r\nsee its meaning.\r\n\r\n“And just as I’d taken the highest tree in the wood,” continued the\r\nPigeon, raising its voice to a shriek, “and just as I was thinking I\r\nshould be free of them at last, they must needs come wriggling down\r\nfrom the sky! Ugh, Serpent!”\r\n\r\n“But I’m _not_ a serpent, I tell you!” said Alice. “I’m a—I’m a—”\r\n\r\n“Well! _What_ are you?” said the Pigeon. “I can see you’re trying to\r\ninvent something!”\r\n\r\n“I—I’m a little girl,” said Alice, rather doubtfully, as she remembered\r\nthe number of changes she had gone through that day.\r\n\r\n“A likely story indeed!” said the Pigeon in a tone of the deepest\r\ncontempt. “I’ve seen a good many little girls in my time, but never\r\n_one_ with such a neck as that! No, no! You’re a serpent; and there’s\r\nno use denying it. I suppose you’ll be telling me next that you never\r\ntasted an egg!”\r\n\r\n“I _have_ tasted eggs, certainly,” said Alice, who was a very truthful\r\nchild; “but little girls eat eggs quite as much as serpents do, you\r\nknow.”\r\n\r\n“I don’t believe it,” said the Pigeon; “but if they do, why then\r\nthey’re a kind of serpent, that’s all I can say.”\r\n\r\nThis was such a new idea to Alice, that she was quite silent for a\r\nminute or two, which gave the Pigeon the opportunity of adding, “You’re\r\nlooking for eggs, I know _that_ well enough; and what does it matter to\r\nme whether you’re a little girl or a serpent?”\r\n\r\n“It matters a good deal to _me_,” said Alice hastily; “but I’m not\r\nlooking for eggs, as it happens; and if I was, I shouldn’t want\r\n_yours_: I don’t like them raw.”\r\n\r\n“Well, be off, then!” said the Pigeon in a sulky tone, as it settled\r\ndown again into its nest. Alice crouched down among the trees as well\r\nas she could, for her neck kept getting entangled among the branches,\r\nand every now and then she had to stop and untwist it. After a while\r\nshe remembered that she still held the pieces of mushroom in her hands,\r\nand she set to work very carefully, nibbling first at one and then at\r\nthe other, and growing sometimes taller and sometimes shorter, until\r\nshe had succeeded in bringing herself down to her usual height.\r\n\r\nIt was so long since she had been anything near the right size, that it\r\nfelt quite strange at first; but she got used to it in a few minutes,\r\nand began talking to herself, as usual. “Come, there’s half my plan\r\ndone now! How puzzling all these changes are! I’m never sure what I’m\r\ngoing to be, from one minute to another! However, I’ve got back to my\r\nright size: the next thing is, to get into that beautiful garden—how\r\n_is_ that to be done, I wonder?” As she said this, she came suddenly\r\nupon an open place, with a little house in it about four feet high.\r\n“Whoever lives there,” thought Alice, “it’ll never do to come upon them\r\n_this_ size: why, I should frighten them out of their wits!” So she\r\nbegan nibbling at the righthand bit again, and did not venture to go\r\nnear the house till she had brought herself down to nine inches high.\r\n\r\n\r\n\r\n\r\nCHAPTER VI.\r\nPig and Pepper\r\n\r\n\r\nFor a minute or two she stood looking at the house, and wondering what\r\nto do next, when suddenly a footman in livery came running out of the\r\nwood—(she considered him to be a footman because he was in livery:\r\notherwise, judging by his face only, she would have called him a\r\nfish)—and rapped loudly at the door with his knuckles. It was opened by\r\nanother footman in livery, with a round face, and large eyes like a\r\nfrog; and both footmen, Alice noticed, had powdered hair that curled\r\nall over their heads. She felt very curious to know what it was all\r\nabout, and crept a little way out of the wood to listen.\r\n\r\nThe Fish-Footman began by producing from under his arm a great letter,\r\nnearly as large as himself, and this he handed over to the other,\r\nsaying, in a solemn tone, “For the Duchess. An invitation from the\r\nQueen to play croquet.” The Frog-Footman repeated, in the same solemn\r\ntone, only changing the order of the words a little, “From the Queen.\r\nAn invitation for the Duchess to play croquet.”\r\n\r\nThen they both bowed low, and their curls got entangled together.\r\n\r\nAlice laughed so much at this, that she had to run back into the wood\r\nfor fear of their hearing her; and when she next peeped out the\r\nFish-Footman was gone, and the other was sitting on the ground near the\r\ndoor, staring stupidly up into the sky.\r\n\r\nAlice went timidly up to the door, and knocked.\r\n\r\n“There’s no sort of use in knocking,” said the Footman, “and that for\r\ntwo reasons. First, because I’m on the same side of the door as you\r\nare; secondly, because they’re making such a noise inside, no one could\r\npossibly hear you.” And certainly there _was_ a most extraordinary\r\nnoise going on within—a constant howling and sneezing, and every now\r\nand then a great crash, as if a dish or kettle had been broken to\r\npieces.\r\n\r\n“Please, then,” said Alice, “how am I to get in?”\r\n\r\n“There might be some sense in your knocking,” the Footman went on\r\nwithout attending to her, “if we had the door between us. For instance,\r\nif you were _inside_, you might knock, and I could let you out, you\r\nknow.” He was looking up into the sky all the time he was speaking, and\r\nthis Alice thought decidedly uncivil. “But perhaps he can’t help it,”\r\nshe said to herself; “his eyes are so _very_ nearly at the top of his\r\nhead. But at any rate he might answer questions.—How am I to get in?”\r\nshe repeated, aloud.\r\n\r\n“I shall sit here,” the Footman remarked, “till tomorrow—”\r\n\r\nAt this moment the door of the house opened, and a large plate came\r\nskimming out, straight at the Footman’s head: it just grazed his nose,\r\nand broke to pieces against one of the trees behind him.\r\n\r\n“—or next day, maybe,” the Footman continued in the same tone, exactly\r\nas if nothing had happened.\r\n\r\n“How am I to get in?” asked Alice again, in a louder tone.\r\n\r\n“_Are_ you to get in at all?” said the Footman. “That’s the first\r\nquestion, you know.”\r\n\r\nIt was, no doubt: only Alice did not like to be told so. “It’s really\r\ndreadful,” she muttered to herself, “the way all the creatures argue.\r\nIt’s enough to drive one crazy!”\r\n\r\nThe Footman seemed to think this a good opportunity for repeating his\r\nremark, with variations. “I shall sit here,” he said, “on and off, for\r\ndays and days.”\r\n\r\n“But what am _I_ to do?” said Alice.\r\n\r\n“Anything you like,” said the Footman, and began whistling.\r\n\r\n“Oh, there’s no use in talking to him,” said Alice desperately: “he’s\r\nperfectly idiotic!” And she opened the door and went in.\r\n\r\nThe door led right into a large kitchen, which was full of smoke from\r\none end to the other: the Duchess was sitting on a three-legged stool\r\nin the middle, nursing a baby; the cook was leaning over the fire,\r\nstirring a large cauldron which seemed to be full of soup.\r\n\r\n“There’s certainly too much pepper in that soup!” Alice said to\r\nherself, as well as she could for sneezing.\r\n\r\nThere was certainly too much of it in the air. Even the Duchess sneezed\r\noccasionally; and as for the baby, it was sneezing and howling\r\nalternately without a moment’s pause. The only things in the kitchen\r\nthat did not sneeze, were the cook, and a large cat which was sitting\r\non the hearth and grinning from ear to ear.\r\n\r\n“Please would you tell me,” said Alice, a little timidly, for she was\r\nnot quite sure whether it was good manners for her to speak first, “why\r\nyour cat grins like that?”\r\n\r\n“It’s a Cheshire cat,” said the Duchess, “and that’s why. Pig!”\r\n\r\nShe said the last word with such sudden violence that Alice quite\r\njumped; but she saw in another moment that it was addressed to the\r\nbaby, and not to her, so she took courage, and went on again:—\r\n\r\n“I didn’t know that Cheshire cats always grinned; in fact, I didn’t\r\nknow that cats _could_ grin.”\r\n\r\n“They all can,” said the Duchess; “and most of ’em do.”\r\n\r\n“I don’t know of any that do,” Alice said very politely, feeling quite\r\npleased to have got into a conversation.\r\n\r\n“You don’t know much,” said the Duchess; “and that’s a fact.”\r\n\r\nAlice did not at all like the tone of this remark, and thought it would\r\nbe as well to introduce some other subject of conversation. While she\r\nwas trying to fix on one, the cook took the cauldron of soup off the\r\nfire, and at once set to work throwing everything within her reach at\r\nthe Duchess and the baby—the fire-irons came first; then followed a\r\nshower of saucepans, plates, and dishes. The Duchess took no notice of\r\nthem even when they hit her; and the baby was howling so much already,\r\nthat it was quite impossible to say whether the blows hurt it or not.\r\n\r\n“Oh, _please_ mind what you’re doing!” cried Alice, jumping up and down\r\nin an agony of terror. “Oh, there goes his _precious_ nose!” as an\r\nunusually large saucepan flew close by it, and very nearly carried it\r\noff.\r\n\r\n“If everybody minded their own business,” the Duchess said in a hoarse\r\ngrowl, “the world would go round a deal faster than it does.”\r\n\r\n“Which would _not_ be an advantage,” said Alice, who felt very glad to\r\nget an opportunity of showing off a little of her knowledge. “Just\r\nthink of what work it would make with the day and night! You see the\r\nearth takes twenty-four hours to turn round on its axis—”\r\n\r\n“Talking of axes,” said the Duchess, “chop off her head!”\r\n\r\nAlice glanced rather anxiously at the cook, to see if she meant to take\r\nthe hint; but the cook was busily stirring the soup, and seemed not to\r\nbe listening, so she went on again: “Twenty-four hours, I _think_; or\r\nis it twelve? I—”\r\n\r\n“Oh, don’t bother _me_,” said the Duchess; “I never could abide\r\nfigures!” And with that she began nursing her child again, singing a\r\nsort of lullaby to it as she did so, and giving it a violent shake at\r\nthe end of every line:\r\n\r\n“Speak roughly to your little boy,\r\n    And beat him when he sneezes:\r\nHe only does it to annoy,\r\n    Because he knows it teases.”\r\n\r\n\r\nCHORUS.\r\n(In which the cook and the baby joined):\r\n\r\n\r\n“Wow! wow! wow!”\r\n\r\n\r\nWhile the Duchess sang the second verse of the song, she kept tossing\r\nthe baby violently up and down, and the poor little thing howled so,\r\nthat Alice could hardly hear the words:—\r\n\r\n“I speak severely to my boy,\r\n    I beat him when he sneezes;\r\nFor he can thoroughly enjoy\r\n    The pepper when he pleases!”\r\n\r\n\r\nCHORUS.\r\n\r\n\r\n“Wow! wow! wow!”\r\n\r\n\r\n“Here! you may nurse it a bit, if you like!” the Duchess said to Alice,\r\nflinging the baby at her as she spoke. “I must go and get ready to play\r\ncroquet with the Queen,” and she hurried out of the room. The cook\r\nthrew a frying-pan after her as she went out, but it just missed her.\r\n\r\nAlice caught the baby with some difficulty, as it was a queer-shaped\r\nlittle creature, and held out its arms and legs in all directions,\r\n“just like a star-fish,” thought Alice. The poor little thing was\r\nsnorting like a steam-engine when she caught it, and kept doubling\r\nitself up and straightening itself out again, so that altogether, for\r\nthe first minute or two, it was as much as she could do to hold it.\r\n\r\nAs soon as she had made out the proper way of nursing it, (which was to\r\ntwist it up into a sort of knot, and then keep tight hold of its right\r\near and left foot, so as to prevent its undoing itself,) she carried it\r\nout into the open air. “If I don’t take this child away with me,”\r\nthought Alice, “they’re sure to kill it in a day or two: wouldn’t it be\r\nmurder to leave it behind?” She said the last words out loud, and the\r\nlittle thing grunted in reply (it had left off sneezing by this time).\r\n“Don’t grunt,” said Alice; “that’s not at all a proper way of\r\nexpressing yourself.”\r\n\r\nThe baby grunted again, and Alice looked very anxiously into its face\r\nto see what was the matter with it. There could be no doubt that it had\r\na _very_ turn-up nose, much more like a snout than a real nose; also\r\nits eyes were getting extremely small for a baby: altogether Alice did\r\nnot like the look of the thing at all. “But perhaps it was only\r\nsobbing,” she thought, and looked into its eyes again, to see if there\r\nwere any tears.\r\n\r\nNo, there were no tears. “If you’re going to turn into a pig, my dear,”\r\nsaid Alice, seriously, “I’ll have nothing more to do with you. Mind\r\nnow!” The poor little thing sobbed again (or grunted, it was impossible\r\nto say which), and they went on for some while in silence.\r\n\r\nAlice was just beginning to think to herself, “Now, what am I to do\r\nwith this creature when I get it home?” when it grunted again, so\r\nviolently, that she looked down into its face in some alarm. This time\r\nthere could be _no_ mistake about it: it was neither more nor less than\r\na pig, and she felt that it would be quite absurd for her to carry it\r\nfurther.\r\n\r\nSo she set the little creature down, and felt quite relieved to see it\r\ntrot away quietly into the wood. “If it had grown up,” she said to\r\nherself, “it would have made a dreadfully ugly child: but it makes\r\nrather a handsome pig, I think.” And she began thinking over other\r\nchildren she knew, who might do very well as pigs, and was just saying\r\nto herself, “if one only knew the right way to change them—” when she\r\nwas a little startled by seeing the Cheshire Cat sitting on a bough of\r\na tree a few yards off.\r\n\r\nThe Cat only grinned when it saw Alice. It looked good-natured, she\r\nthought: still it had _very_ long claws and a great many teeth, so she\r\nfelt that it ought to be treated with respect.\r\n\r\n“Cheshire Puss,” she began, rather timidly, as she did not at all know\r\nwhether it would like the name: however, it only grinned a little\r\nwider. “Come, it’s pleased so far,” thought Alice, and she went on.\r\n“Would you tell me, please, which way I ought to go from here?”\r\n\r\n“That depends a good deal on where you want to get to,” said the Cat.\r\n\r\n“I don’t much care where—” said Alice.\r\n\r\n“Then it doesn’t matter which way you go,” said the Cat.\r\n\r\n“—so long as I get _somewhere_,” Alice added as an explanation.\r\n\r\n“Oh, you’re sure to do that,” said the Cat, “if you only walk long\r\nenough.”\r\n\r\nAlice felt that this could not be denied, so she tried another\r\nquestion. “What sort of people live about here?”\r\n\r\n“In _that_ direction,” the Cat said, waving its right paw round, “lives\r\na Hatter: and in _that_ direction,” waving the other paw, “lives a\r\nMarch Hare. Visit either you like: they’re both mad.”\r\n\r\n“But I don’t want to go among mad people,” Alice remarked.\r\n\r\n“Oh, you can’t help that,” said the Cat: “we’re all mad here. I’m mad.\r\nYou’re mad.”\r\n\r\n“How do you know I’m mad?” said Alice.\r\n\r\n“You must be,” said the Cat, “or you wouldn’t have come here.”\r\n\r\nAlice didn’t think that proved it at all; however, she went on “And how\r\ndo you know that you’re mad?”\r\n\r\n“To begin with,” said the Cat, “a dog’s not mad. You grant that?”\r\n\r\n“I suppose so,” said Alice.\r\n\r\n“Well, then,” the Cat went on, “you see, a dog growls when it’s angry,\r\nand wags its tail when it’s pleased. Now _I_ growl when I’m pleased,\r\nand wag my tail when I’m angry. Therefore I’m mad.”\r\n\r\n“_I_ call it purring, not growling,” said Alice.\r\n\r\n“Call it what you like,” said the Cat. “Do you play croquet with the\r\nQueen to-day?”\r\n\r\n“I should like it very much,” said Alice, “but I haven’t been invited\r\nyet.”\r\n\r\n“You’ll see me there,” said the Cat, and vanished.\r\n\r\nAlice was not much surprised at this, she was getting so used to queer\r\nthings happening. While she was looking at the place where it had been,\r\nit suddenly appeared again.\r\n\r\n“By-the-bye, what became of the baby?” said the Cat. “I’d nearly\r\nforgotten to ask.”\r\n\r\n“It turned into a pig,” Alice quietly said, just as if it had come back\r\nin a natural way.\r\n\r\n“I thought it would,” said the Cat, and vanished again.\r\n\r\nAlice waited a little, half expecting to see it again, but it did not\r\nappear, and after a minute or two she walked on in the direction in\r\nwhich the March Hare was said to live. “I’ve seen hatters before,” she\r\nsaid to herself; “the March Hare will be much the most interesting, and\r\nperhaps as this is May it won’t be raving mad—at least not so mad as it\r\nwas in March.” As she said this, she looked up, and there was the Cat\r\nagain, sitting on a branch of a tree.\r\n\r\n“Did you say pig, or fig?” said the Cat.\r\n\r\n“I said pig,” replied Alice; “and I wish you wouldn’t keep appearing\r\nand vanishing so suddenly: you make one quite giddy.”\r\n\r\n“All right,” said the Cat; and this time it vanished quite slowly,\r\nbeginning with the end of the tail, and ending with the grin, which\r\nremained some time after the rest of it had gone.\r\n\r\n“Well! I’ve often seen a cat without a grin,” thought Alice; “but a\r\ngrin without a cat! It’s the most curious thing I ever saw in my life!”\r\n\r\nShe had not gone much farther before she came in sight of the house of\r\nthe March Hare: she thought it must be the right house, because the\r\nchimneys were shaped like ears and the roof was thatched with fur. It\r\nwas so large a house, that she did not like to go nearer till she had\r\nnibbled some more of the lefthand bit of mushroom, and raised herself\r\nto about two feet high: even then she walked up towards it rather\r\ntimidly, saying to herself “Suppose it should be raving mad after all!\r\nI almost wish I’d gone to see the Hatter instead!”\r\n\r\n\r\n\r\n\r\nCHAPTER VII.\r\nA Mad Tea-Party\r\n\r\n\r\nThere was a table set out under a tree in front of the house, and the\r\nMarch Hare and the Hatter were having tea at it: a Dormouse was sitting\r\nbetween them, fast asleep, and the other two were using it as a\r\ncushion, resting their elbows on it, and talking over its head. “Very\r\nuncomfortable for the Dormouse,” thought Alice; “only, as it’s asleep,\r\nI suppose it doesn’t mind.”\r\n\r\nThe table was a large one, but the three were all crowded together at\r\none corner of it: “No room! No room!” they cried out when they saw\r\nAlice coming. “There’s _plenty_ of room!” said Alice indignantly, and\r\nshe sat down in a large arm-chair at one end of the table.\r\n\r\n“Have some wine,” the March Hare said in an encouraging tone.\r\n\r\nAlice looked all round the table, but there was nothing on it but tea.\r\n“I don’t see any wine,” she remarked.\r\n\r\n“There isn’t any,” said the March Hare.\r\n\r\n“Then it wasn’t very civil of you to offer it,” said Alice angrily.\r\n\r\n“It wasn’t very civil of you to sit down without being invited,” said\r\nthe March Hare.\r\n\r\n“I didn’t know it was _your_ table,” said Alice; “it’s laid for a great\r\nmany more than three.”\r\n\r\n“Your hair wants cutting,” said the Hatter. He had been looking at\r\nAlice for some time with great curiosity, and this was his first\r\nspeech.\r\n\r\n“You should learn not to make personal remarks,” Alice said with some\r\nseverity; “it’s very rude.”\r\n\r\nThe Hatter opened his eyes very wide on hearing this; but all he _said_\r\nwas, “Why is a raven like a writing-desk?”\r\n\r\n“Come, we shall have some fun now!” thought Alice. “I’m glad they’ve\r\nbegun asking riddles.—I believe I can guess that,” she added aloud.\r\n\r\n“Do you mean that you think you can find out the answer to it?” said\r\nthe March Hare.\r\n\r\n“Exactly so,” said Alice.\r\n\r\n“Then you should say what you mean,” the March Hare went on.\r\n\r\n“I do,” Alice hastily replied; “at least—at least I mean what I\r\nsay—that’s the same thing, you know.”\r\n\r\n“Not the same thing a bit!” said the Hatter. “You might just as well\r\nsay that ‘I see what I eat’ is the same thing as ‘I eat what I see’!”\r\n\r\n“You might just as well say,” added the March Hare, “that ‘I like what\r\nI get’ is the same thing as ‘I get what I like’!”\r\n\r\n“You might just as well say,” added the Dormouse, who seemed to be\r\ntalking in his sleep, “that ‘I breathe when I sleep’ is the same thing\r\nas ‘I sleep when I breathe’!”\r\n\r\n“It _is_ the same thing with you,” said the Hatter, and here the\r\nconversation dropped, and the party sat silent for a minute, while\r\nAlice thought over all she could remember about ravens and\r\nwriting-desks, which wasn’t much.\r\n\r\nThe Hatter was the first to break the silence. “What day of the month\r\nis it?” he said, turning to Alice: he had taken his watch out of his\r\npocket, and was looking at it uneasily, shaking it every now and then,\r\nand holding it to his ear.\r\n\r\nAlice considered a little, and then said “The fourth.”\r\n\r\n“Two days wrong!” sighed the Hatter. “I told you butter wouldn’t suit\r\nthe works!” he added looking angrily at the March Hare.\r\n\r\n“It was the _best_ butter,” the March Hare meekly replied.\r\n\r\n“Yes, but some crumbs must have got in as well,” the Hatter grumbled:\r\n“you shouldn’t have put it in with the bread-knife.”\r\n\r\nThe March Hare took the watch and looked at it gloomily: then he dipped\r\nit into his cup of tea, and looked at it again: but he could think of\r\nnothing better to say than his first remark, “It was the _best_ butter,\r\nyou know.”\r\n\r\nAlice had been looking over his shoulder with some curiosity. “What a\r\nfunny watch!” she remarked. “It tells the day of the month, and doesn’t\r\ntell what o’clock it is!”\r\n\r\n“Why should it?” muttered the Hatter. “Does _your_ watch tell you what\r\nyear it is?”\r\n\r\n“Of course not,” Alice replied very readily: “but that’s because it\r\nstays the same year for such a long time together.”\r\n\r\n“Which is just the case with _mine_,” said the Hatter.\r\n\r\nAlice felt dreadfully puzzled. The Hatter’s remark seemed to have no\r\nsort of meaning in it, and yet it was certainly English. “I don’t quite\r\nunderstand you,” she said, as politely as she could.\r\n\r\n“The Dormouse is asleep again,” said the Hatter, and he poured a little\r\nhot tea upon its nose.\r\n\r\nThe Dormouse shook its head impatiently, and said, without opening its\r\neyes, “Of course, of course; just what I was going to remark myself.”\r\n\r\n“Have you guessed the riddle yet?” the Hatter said, turning to Alice\r\nagain.\r\n\r\n“No, I give it up,” Alice replied: “what’s the answer?”\r\n\r\n“I haven’t the slightest idea,” said the Hatter.\r\n\r\n“Nor I,” said the March Hare.\r\n\r\nAlice sighed wearily. “I think you might do something better with the\r\ntime,” she said, “than waste it in asking riddles that have no\r\nanswers.”\r\n\r\n“If you knew Time as well as I do,” said the Hatter, “you wouldn’t talk\r\nabout wasting _it_. It’s _him_.”\r\n\r\n“I don’t know what you mean,” said Alice.\r\n\r\n“Of course you don’t!” the Hatter said, tossing his head\r\ncontemptuously. “I dare say you never even spoke to Time!”\r\n\r\n“Perhaps not,” Alice cautiously replied: “but I know I have to beat\r\ntime when I learn music.”\r\n\r\n“Ah! that accounts for it,” said the Hatter. “He won’t stand beating.\r\nNow, if you only kept on good terms with him, he’d do almost anything\r\nyou liked with the clock. For instance, suppose it were nine o’clock in\r\nthe morning, just time to begin lessons: you’d only have to whisper a\r\nhint to Time, and round goes the clock in a twinkling! Half-past one,\r\ntime for dinner!”\r\n\r\n(“I only wish it was,” the March Hare said to itself in a whisper.)\r\n\r\n“That would be grand, certainly,” said Alice thoughtfully: “but then—I\r\nshouldn’t be hungry for it, you know.”\r\n\r\n“Not at first, perhaps,” said the Hatter: “but you could keep it to\r\nhalf-past one as long as you liked.”\r\n\r\n“Is that the way _you_ manage?” Alice asked.\r\n\r\nThe Hatter shook his head mournfully. “Not I!” he replied. “We\r\nquarrelled last March—just before _he_ went mad, you know—” (pointing\r\nwith his tea spoon at the March Hare,) “—it was at the great concert\r\ngiven by the Queen of Hearts, and I had to sing\r\n\r\n‘Twinkle, twinkle, little bat!\r\nHow I wonder what you’re at!’\r\n\r\n\r\nYou know the song, perhaps?”\r\n\r\n“I’ve heard something like it,” said Alice.\r\n\r\n“It goes on, you know,” the Hatter continued, “in this way:—\r\n\r\n‘Up above the world you fly,\r\nLike a tea-tray in the sky.\r\n                    Twinkle, twinkle—’”\r\n\r\n\r\nHere the Dormouse shook itself, and began singing in its sleep\r\n“_Twinkle, twinkle, twinkle, twinkle_—” and went on so long that they\r\nhad to pinch it to make it stop.\r\n\r\n“Well, I’d hardly finished the first verse,” said the Hatter, “when the\r\nQueen jumped up and bawled out, ‘He’s murdering the time! Off with his\r\nhead!’”\r\n\r\n“How dreadfully savage!” exclaimed Alice.\r\n\r\n“And ever since that,” the Hatter went on in a mournful tone, “he won’t\r\ndo a thing I ask! It’s always six o’clock now.”\r\n\r\nA bright idea came into Alice’s head. “Is that the reason so many\r\ntea-things are put out here?” she asked.\r\n\r\n“Yes, that’s it,” said the Hatter with a sigh: “it’s always tea-time,\r\nand we’ve no time to wash the things between whiles.”\r\n\r\n“Then you keep moving round, I suppose?” said Alice.\r\n\r\n“Exactly so,” said the Hatter: “as the things get used up.”\r\n\r\n“But what happens when you come to the beginning again?” Alice ventured\r\nto ask.\r\n\r\n“Suppose we change the subject,” the March Hare interrupted, yawning.\r\n“I’m getting tired of this. I vote the young lady tells us a story.”\r\n\r\n“I’m afraid I don’t know one,” said Alice, rather alarmed at the\r\nproposal.\r\n\r\n“Then the Dormouse shall!” they both cried. “Wake up, Dormouse!” And\r\nthey pinched it on both sides at once.\r\n\r\nThe Dormouse slowly opened his eyes. “I wasn’t asleep,” he said in a\r\nhoarse, feeble voice: “I heard every word you fellows were saying.”\r\n\r\n“Tell us a story!” said the March Hare.\r\n\r\n“Yes, please do!” pleaded Alice.\r\n\r\n“And be quick about it,” added the Hatter, “or you’ll be asleep again\r\nbefore it’s done.”\r\n\r\n“Once upon a time there were three little sisters,” the Dormouse began\r\nin a great hurry; “and their names were Elsie, Lacie, and Tillie; and\r\nthey lived at the bottom of a well—”\r\n\r\n“What did they live on?” said Alice, who always took a great interest\r\nin questions of eating and drinking.\r\n\r\n“They lived on treacle,” said the Dormouse, after thinking a minute or\r\ntwo.\r\n\r\n“They couldn’t have done that, you know,” Alice gently remarked;\r\n“they’d have been ill.”\r\n\r\n“So they were,” said the Dormouse; “_very_ ill.”\r\n\r\nAlice tried to fancy to herself what such an extraordinary ways of\r\nliving would be like, but it puzzled her too much, so she went on: “But\r\nwhy did they live at the bottom of a well?”\r\n\r\n“Take some more tea,” the March Hare said to Alice, very earnestly.\r\n\r\n“I’ve had nothing yet,” Alice replied in an offended tone, “so I can’t\r\ntake more.”\r\n\r\n“You mean you can’t take _less_,” said the Hatter: “it’s very easy to\r\ntake _more_ than nothing.”\r\n\r\n“Nobody asked _your_ opinion,” said Alice.\r\n\r\n“Who’s making personal remarks now?” the Hatter asked triumphantly.\r\n\r\nAlice did not quite know what to say to this: so she helped herself to\r\nsome tea and bread-and-butter, and then turned to the Dormouse, and\r\nrepeated her question. “Why did they live at the bottom of a well?”\r\n\r\nThe Dormouse again took a minute or two to think about it, and then\r\nsaid, “It was a treacle-well.”\r\n\r\n“There’s no such thing!” Alice was beginning very angrily, but the\r\nHatter and the March Hare went “Sh! sh!” and the Dormouse sulkily\r\nremarked, “If you can’t be civil, you’d better finish the story for\r\nyourself.”\r\n\r\n“No, please go on!” Alice said very humbly; “I won’t interrupt again. I\r\ndare say there may be _one_.”\r\n\r\n“One, indeed!” said the Dormouse indignantly. However, he consented to\r\ngo on. “And so these three little sisters—they were learning to draw,\r\nyou know—”\r\n\r\n“What did they draw?” said Alice, quite forgetting her promise.\r\n\r\n“Treacle,” said the Dormouse, without considering at all this time.\r\n\r\n“I want a clean cup,” interrupted the Hatter: “let’s all move one place\r\non.”\r\n\r\nHe moved on as he spoke, and the Dormouse followed him: the March Hare\r\nmoved into the Dormouse’s place, and Alice rather unwillingly took the\r\nplace of the March Hare. The Hatter was the only one who got any\r\nadvantage from the change: and Alice was a good deal worse off than\r\nbefore, as the March Hare had just upset the milk-jug into his plate.\r\n\r\nAlice did not wish to offend the Dormouse again, so she began very\r\ncautiously: “But I don’t understand. Where did they draw the treacle\r\nfrom?”\r\n\r\n“You can draw water out of a water-well,” said the Hatter; “so I should\r\nthink you could draw treacle out of a treacle-well—eh, stupid?”\r\n\r\n“But they were _in_ the well,” Alice said to the Dormouse, not choosing\r\nto notice this last remark.\r\n\r\n“Of course they were,” said the Dormouse; “—well in.”\r\n\r\nThis answer so confused poor Alice, that she let the Dormouse go on for\r\nsome time without interrupting it.\r\n\r\n“They were learning to draw,” the Dormouse went on, yawning and rubbing\r\nits eyes, for it was getting very sleepy; “and they drew all manner of\r\nthings—everything that begins with an M—”\r\n\r\n“Why with an M?” said Alice.\r\n\r\n“Why not?” said the March Hare.\r\n\r\nAlice was silent.\r\n\r\nThe Dormouse had closed its eyes by this time, and was going off into a\r\ndoze; but, on being pinched by the Hatter, it woke up again with a\r\nlittle shriek, and went on: “—that begins with an M, such as\r\nmouse-traps, and the moon, and memory, and muchness—you know you say\r\nthings are “much of a muchness”—did you ever see such a thing as a\r\ndrawing of a muchness?”\r\n\r\n“Really, now you ask me,” said Alice, very much confused, “I don’t\r\nthink—”\r\n\r\n“Then you shouldn’t talk,” said the Hatter.\r\n\r\nThis piece of rudeness was more than Alice could bear: she got up in\r\ngreat disgust, and walked off; the Dormouse fell asleep instantly, and\r\nneither of the others took the least notice of her going, though she\r\nlooked back once or twice, half hoping that they would call after her:\r\nthe last time she saw them, they were trying to put the Dormouse into\r\nthe teapot.\r\n\r\n“At any rate I’ll never go _there_ again!” said Alice as she picked her\r\nway through the wood. “It’s the stupidest tea-party I ever was at in\r\nall my life!”\r\n\r\nJust as she said this, she noticed that one of the trees had a door\r\nleading right into it. “That’s very curious!” she thought. “But\r\neverything’s curious today. I think I may as well go in at once.” And\r\nin she went.\r\n\r\nOnce more she found herself in the long hall, and close to the little\r\nglass table. “Now, I’ll manage better this time,” she said to herself,\r\nand began by taking the little golden key, and unlocking the door that\r\nled into the garden. Then she went to work nibbling at the mushroom\r\n(she had kept a piece of it in her pocket) till she was about a foot\r\nhigh: then she walked down the little passage: and _then_—she found\r\nherself at last in the beautiful garden, among the bright flower-beds\r\nand the cool fountains.\r\n\r\n\r\n\r\n\r\nCHAPTER VIII.\r\nThe Queen’s Croquet-Ground\r\n\r\n\r\nA large rose-tree stood near the entrance of the garden: the roses\r\ngrowing on it were white, but there were three gardeners at it, busily\r\npainting them red. Alice thought this a very curious thing, and she\r\nwent nearer to watch them, and just as she came up to them she heard\r\none of them say, “Look out now, Five! Don’t go splashing paint over me\r\nlike that!”\r\n\r\n“I couldn’t help it,” said Five, in a sulky tone; “Seven jogged my\r\nelbow.”\r\n\r\nOn which Seven looked up and said, “That’s right, Five! Always lay the\r\nblame on others!”\r\n\r\n“_You’d_ better not talk!” said Five. “I heard the Queen say only\r\nyesterday you deserved to be beheaded!”\r\n\r\n“What for?” said the one who had spoken first.\r\n\r\n“That’s none of _your_ business, Two!” said Seven.\r\n\r\n“Yes, it _is_ his business!” said Five, “and I’ll tell him—it was for\r\nbringing the cook tulip-roots instead of onions.”\r\n\r\nSeven flung down his brush, and had just begun “Well, of all the unjust\r\nthings—” when his eye chanced to fall upon Alice, as she stood watching\r\nthem, and he checked himself suddenly: the others looked round also,\r\nand all of them bowed low.\r\n\r\n“Would you tell me,” said Alice, a little timidly, “why you are\r\npainting those roses?”\r\n\r\nFive and Seven said nothing, but looked at Two. Two began in a low\r\nvoice, “Why the fact is, you see, Miss, this here ought to have been a\r\n_red_ rose-tree, and we put a white one in by mistake; and if the Queen\r\nwas to find it out, we should all have our heads cut off, you know. So\r\nyou see, Miss, we’re doing our best, afore she comes, to—” At this\r\nmoment Five, who had been anxiously looking across the garden, called\r\nout “The Queen! The Queen!” and the three gardeners instantly threw\r\nthemselves flat upon their faces. There was a sound of many footsteps,\r\nand Alice looked round, eager to see the Queen.\r\n\r\nFirst came ten soldiers carrying clubs; these were all shaped like the\r\nthree gardeners, oblong and flat, with their hands and feet at the\r\ncorners: next the ten courtiers; these were ornamented all over with\r\ndiamonds, and walked two and two, as the soldiers did. After these came\r\nthe royal children; there were ten of them, and the little dears came\r\njumping merrily along hand in hand, in couples: they were all\r\nornamented with hearts. Next came the guests, mostly Kings and Queens,\r\nand among them Alice recognised the White Rabbit: it was talking in a\r\nhurried nervous manner, smiling at everything that was said, and went\r\nby without noticing her. Then followed the Knave of Hearts, carrying\r\nthe King’s crown on a crimson velvet cushion; and, last of all this\r\ngrand procession, came THE KING AND QUEEN OF HEARTS.\r\n\r\nAlice was rather doubtful whether she ought not to lie down on her face\r\nlike the three gardeners, but she could not remember ever having heard\r\nof such a rule at processions; “and besides, what would be the use of a\r\nprocession,” thought she, “if people had all to lie down upon their\r\nfaces, so that they couldn’t see it?” So she stood still where she was,\r\nand waited.\r\n\r\nWhen the procession came opposite to Alice, they all stopped and looked\r\nat her, and the Queen said severely “Who is this?” She said it to the\r\nKnave of Hearts, who only bowed and smiled in reply.\r\n\r\n“Idiot!” said the Queen, tossing her head impatiently; and, turning to\r\nAlice, she went on, “What’s your name, child?”\r\n\r\n“My name is Alice, so please your Majesty,” said Alice very politely;\r\nbut she added, to herself, “Why, they’re only a pack of cards, after\r\nall. I needn’t be afraid of them!”\r\n\r\n“And who are _these?_” said the Queen, pointing to the three gardeners\r\nwho were lying round the rose-tree; for, you see, as they were lying on\r\ntheir faces, and the pattern on their backs was the same as the rest of\r\nthe pack, she could not tell whether they were gardeners, or soldiers,\r\nor courtiers, or three of her own children.\r\n\r\n“How should _I_ know?” said Alice, surprised at her own courage. “It’s\r\nno business of _mine_.”\r\n\r\nThe Queen turned crimson with fury, and, after glaring at her for a\r\nmoment like a wild beast, screamed “Off with her head! Off—”\r\n\r\n“Nonsense!” said Alice, very loudly and decidedly, and the Queen was\r\nsilent.\r\n\r\nThe King laid his hand upon her arm, and timidly said “Consider, my\r\ndear: she is only a child!”\r\n\r\nThe Queen turned angrily away from him, and said to the Knave “Turn\r\nthem over!”\r\n\r\nThe Knave did so, very carefully, with one foot.\r\n\r\n“Get up!” said the Queen, in a shrill, loud voice, and the three\r\ngardeners instantly jumped up, and began bowing to the King, the Queen,\r\nthe royal children, and everybody else.\r\n\r\n“Leave off that!” screamed the Queen. “You make me giddy.” And then,\r\nturning to the rose-tree, she went on, “What _have_ you been doing\r\nhere?”\r\n\r\n“May it please your Majesty,” said Two, in a very humble tone, going\r\ndown on one knee as he spoke, “we were trying—”\r\n\r\n“_I_ see!” said the Queen, who had meanwhile been examining the roses.\r\n“Off with their heads!” and the procession moved on, three of the\r\nsoldiers remaining behind to execute the unfortunate gardeners, who ran\r\nto Alice for protection.\r\n\r\n“You shan’t be beheaded!” said Alice, and she put them into a large\r\nflower-pot that stood near. The three soldiers wandered about for a\r\nminute or two, looking for them, and then quietly marched off after the\r\nothers.\r\n\r\n“Are their heads off?” shouted the Queen.\r\n\r\n“Their heads are gone, if it please your Majesty!” the soldiers shouted\r\nin reply.\r\n\r\n“That’s right!” shouted the Queen. “Can you play croquet?”\r\n\r\nThe soldiers were silent, and looked at Alice, as the question was\r\nevidently meant for her.\r\n\r\n“Yes!” shouted Alice.\r\n\r\n“Come on, then!” roared the Queen, and Alice joined the procession,\r\nwondering very much what would happen next.\r\n\r\n“It’s—it’s a very fine day!” said a timid voice at her side. She was\r\nwalking by the White Rabbit, who was peeping anxiously into her face.\r\n\r\n“Very,” said Alice: “—where’s the Duchess?”\r\n\r\n“Hush! Hush!” said the Rabbit in a low, hurried tone. He looked\r\nanxiously over his shoulder as he spoke, and then raised himself upon\r\ntiptoe, put his mouth close to her ear, and whispered “She’s under\r\nsentence of execution.”\r\n\r\n“What for?” said Alice.\r\n\r\n“Did you say ‘What a pity!’?” the Rabbit asked.\r\n\r\n“No, I didn’t,” said Alice: “I don’t think it’s at all a pity. I said\r\n‘What for?’”\r\n\r\n“She boxed the Queen’s ears—” the Rabbit began. Alice gave a little\r\nscream of laughter. “Oh, hush!” the Rabbit whispered in a frightened\r\ntone. “The Queen will hear you! You see, she came rather late, and the\r\nQueen said—”\r\n\r\n“Get to your places!” shouted the Queen in a voice of thunder, and\r\npeople began running about in all directions, tumbling up against each\r\nother; however, they got settled down in a minute or two, and the game\r\nbegan. Alice thought she had never seen such a curious croquet-ground\r\nin her life; it was all ridges and furrows; the balls were live\r\nhedgehogs, the mallets live flamingoes, and the soldiers had to double\r\nthemselves up and to stand on their hands and feet, to make the arches.\r\n\r\nThe chief difficulty Alice found at first was in managing her flamingo:\r\nshe succeeded in getting its body tucked away, comfortably enough,\r\nunder her arm, with its legs hanging down, but generally, just as she\r\nhad got its neck nicely straightened out, and was going to give the\r\nhedgehog a blow with its head, it _would_ twist itself round and look\r\nup in her face, with such a puzzled expression that she could not help\r\nbursting out laughing: and when she had got its head down, and was\r\ngoing to begin again, it was very provoking to find that the hedgehog\r\nhad unrolled itself, and was in the act of crawling away: besides all\r\nthis, there was generally a ridge or furrow in the way wherever she\r\nwanted to send the hedgehog to, and, as the doubled-up soldiers were\r\nalways getting up and walking off to other parts of the ground, Alice\r\nsoon came to the conclusion that it was a very difficult game indeed.\r\n\r\nThe players all played at once without waiting for turns, quarrelling\r\nall the while, and fighting for the hedgehogs; and in a very short time\r\nthe Queen was in a furious passion, and went stamping about, and\r\nshouting “Off with his head!” or “Off with her head!” about once in a\r\nminute.\r\n\r\nAlice began to feel very uneasy: to be sure, she had not as yet had any\r\ndispute with the Queen, but she knew that it might happen any minute,\r\n“and then,” thought she, “what would become of me? They’re dreadfully\r\nfond of beheading people here; the great wonder is, that there’s any\r\none left alive!”\r\n\r\nShe was looking about for some way of escape, and wondering whether she\r\ncould get away without being seen, when she noticed a curious\r\nappearance in the air: it puzzled her very much at first, but, after\r\nwatching it a minute or two, she made it out to be a grin, and she said\r\nto herself “It’s the Cheshire Cat: now I shall have somebody to talk\r\nto.”\r\n\r\n“How are you getting on?” said the Cat, as soon as there was mouth\r\nenough for it to speak with.\r\n\r\nAlice waited till the eyes appeared, and then nodded. “It’s no use\r\nspeaking to it,” she thought, “till its ears have come, or at least one\r\nof them.” In another minute the whole head appeared, and then Alice put\r\ndown her flamingo, and began an account of the game, feeling very glad\r\nshe had someone to listen to her. The Cat seemed to think that there\r\nwas enough of it now in sight, and no more of it appeared.\r\n\r\n“I don’t think they play at all fairly,” Alice began, in rather a\r\ncomplaining tone, “and they all quarrel so dreadfully one can’t hear\r\noneself speak—and they don’t seem to have any rules in particular; at\r\nleast, if there are, nobody attends to them—and you’ve no idea how\r\nconfusing it is all the things being alive; for instance, there’s the\r\narch I’ve got to go through next walking about at the other end of the\r\nground—and I should have croqueted the Queen’s hedgehog just now, only\r\nit ran away when it saw mine coming!”\r\n\r\n“How do you like the Queen?” said the Cat in a low voice.\r\n\r\n“Not at all,” said Alice: “she’s so extremely—” Just then she noticed\r\nthat the Queen was close behind her, listening: so she went on,\r\n“—likely to win, that it’s hardly worth while finishing the game.”\r\n\r\nThe Queen smiled and passed on.\r\n\r\n“Who _are_ you talking to?” said the King, going up to Alice, and\r\nlooking at the Cat’s head with great curiosity.\r\n\r\n“It’s a friend of mine—a Cheshire Cat,” said Alice: “allow me to\r\nintroduce it.”\r\n\r\n“I don’t like the look of it at all,” said the King: “however, it may\r\nkiss my hand if it likes.”\r\n\r\n“I’d rather not,” the Cat remarked.\r\n\r\n“Don’t be impertinent,” said the King, “and don’t look at me like\r\nthat!” He got behind Alice as he spoke.\r\n\r\n“A cat may look at a king,” said Alice. “I’ve read that in some book,\r\nbut I don’t remember where.”\r\n\r\n“Well, it must be removed,” said the King very decidedly, and he called\r\nthe Queen, who was passing at the moment, “My dear! I wish you would\r\nhave this cat removed!”\r\n\r\nThe Queen had only one way of settling all difficulties, great or\r\nsmall. “Off with his head!” she said, without even looking round.\r\n\r\n“I’ll fetch the executioner myself,” said the King eagerly, and he\r\nhurried off.\r\n\r\nAlice thought she might as well go back, and see how the game was going\r\non, as she heard the Queen’s voice in the distance, screaming with\r\npassion. She had already heard her sentence three of the players to be\r\nexecuted for having missed their turns, and she did not like the look\r\nof things at all, as the game was in such confusion that she never knew\r\nwhether it was her turn or not. So she went in search of her hedgehog.\r\n\r\nThe hedgehog was engaged in a fight with another hedgehog, which seemed\r\nto Alice an excellent opportunity for croqueting one of them with the\r\nother: the only difficulty was, that her flamingo was gone across to\r\nthe other side of the garden, where Alice could see it trying in a\r\nhelpless sort of way to fly up into a tree.\r\n\r\nBy the time she had caught the flamingo and brought it back, the fight\r\nwas over, and both the hedgehogs were out of sight: “but it doesn’t\r\nmatter much,” thought Alice, “as all the arches are gone from this side\r\nof the ground.” So she tucked it away under her arm, that it might not\r\nescape again, and went back for a little more conversation with her\r\nfriend.\r\n\r\nWhen she got back to the Cheshire Cat, she was surprised to find quite\r\na large crowd collected round it: there was a dispute going on between\r\nthe executioner, the King, and the Queen, who were all talking at once,\r\nwhile all the rest were quite silent, and looked very uncomfortable.\r\n\r\nThe moment Alice appeared, she was appealed to by all three to settle\r\nthe question, and they repeated their arguments to her, though, as they\r\nall spoke at once, she found it very hard indeed to make out exactly\r\nwhat they said.\r\n\r\nThe executioner’s argument was, that you couldn’t cut off a head unless\r\nthere was a body to cut it off from: that he had never had to do such a\r\nthing before, and he wasn’t going to begin at _his_ time of life.\r\n\r\nThe King’s argument was, that anything that had a head could be\r\nbeheaded, and that you weren’t to talk nonsense.\r\n\r\nThe Queen’s argument was, that if something wasn’t done about it in\r\nless than no time she’d have everybody executed, all round. (It was\r\nthis last remark that had made the whole party look so grave and\r\nanxious.)\r\n\r\nAlice could think of nothing else to say but “It belongs to the\r\nDuchess: you’d better ask _her_ about it.”\r\n\r\n“She’s in prison,” the Queen said to the executioner: “fetch her here.”\r\nAnd the executioner went off like an arrow.\r\n\r\nThe Cat’s head began fading away the moment he was gone, and, by the\r\ntime he had come back with the Duchess, it had entirely disappeared; so\r\nthe King and the executioner ran wildly up and down looking for it,\r\nwhile the rest of the party went back to the game.\r\n\r\n\r\n\r\n\r\nCHAPTER IX.\r\nThe Mock Turtle’s Story\r\n\r\n\r\n“You can’t think how glad I am to see you again, you dear old thing!”\r\nsaid the Duchess, as she tucked her arm affectionately into Alice’s,\r\nand they walked off together.\r\n\r\nAlice was very glad to find her in such a pleasant temper, and thought\r\nto herself that perhaps it was only the pepper that had made her so\r\nsavage when they met in the kitchen.\r\n\r\n“When _I’m_ a Duchess,” she said to herself, (not in a very hopeful\r\ntone though), “I won’t have any pepper in my kitchen _at all_. Soup\r\ndoes very well without—Maybe it’s always pepper that makes people\r\nhot-tempered,” she went on, very much pleased at having found out a new\r\nkind of rule, “and vinegar that makes them sour—and camomile that makes\r\nthem bitter—and—and barley-sugar and such things that make children\r\nsweet-tempered. I only wish people knew _that_: then they wouldn’t be\r\nso stingy about it, you know—”\r\n\r\nShe had quite forgotten the Duchess by this time, and was a little\r\nstartled when she heard her voice close to her ear. “You’re thinking\r\nabout something, my dear, and that makes you forget to talk. I can’t\r\ntell you just now what the moral of that is, but I shall remember it in\r\na bit.”\r\n\r\n“Perhaps it hasn’t one,” Alice ventured to remark.\r\n\r\n“Tut, tut, child!” said the Duchess. “Everything’s got a moral, if only\r\nyou can find it.” And she squeezed herself up closer to Alice’s side as\r\nshe spoke.\r\n\r\nAlice did not much like keeping so close to her: first, because the\r\nDuchess was _very_ ugly; and secondly, because she was exactly the\r\nright height to rest her chin upon Alice’s shoulder, and it was an\r\nuncomfortably sharp chin. However, she did not like to be rude, so she\r\nbore it as well as she could.\r\n\r\n“The game’s going on rather better now,” she said, by way of keeping up\r\nthe conversation a little.\r\n\r\n“’Tis so,” said the Duchess: “and the moral of that is—‘Oh, ’tis love,\r\n’tis love, that makes the world go round!’”\r\n\r\n“Somebody said,” Alice whispered, “that it’s done by everybody minding\r\ntheir own business!”\r\n\r\n“Ah, well! It means much the same thing,” said the Duchess, digging her\r\nsharp little chin into Alice’s shoulder as she added, “and the moral of\r\n_that_ is—‘Take care of the sense, and the sounds will take care of\r\nthemselves.’”\r\n\r\n“How fond she is of finding morals in things!” Alice thought to\r\nherself.\r\n\r\n“I dare say you’re wondering why I don’t put my arm round your waist,”\r\nthe Duchess said after a pause: “the reason is, that I’m doubtful about\r\nthe temper of your flamingo. Shall I try the experiment?”\r\n\r\n“He might bite,” Alice cautiously replied, not feeling at all anxious\r\nto have the experiment tried.\r\n\r\n“Very true,” said the Duchess: “flamingoes and mustard both bite. And\r\nthe moral of that is—‘Birds of a feather flock together.’”\r\n\r\n“Only mustard isn’t a bird,” Alice remarked.\r\n\r\n“Right, as usual,” said the Duchess: “what a clear way you have of\r\nputting things!”\r\n\r\n“It’s a mineral, I _think_,” said Alice.\r\n\r\n“Of course it is,” said the Duchess, who seemed ready to agree to\r\neverything that Alice said; “there’s a large mustard-mine near here.\r\nAnd the moral of that is—‘The more there is of mine, the less there is\r\nof yours.’”\r\n\r\n“Oh, I know!” exclaimed Alice, who had not attended to this last\r\nremark, “it’s a vegetable. It doesn’t look like one, but it is.”\r\n\r\n“I quite agree with you,” said the Duchess; “and the moral of that\r\nis—‘Be what you would seem to be’—or if you’d like it put more\r\nsimply—‘Never imagine yourself not to be otherwise than what it might\r\nappear to others that what you were or might have been was not\r\notherwise than what you had been would have appeared to them to be\r\notherwise.’”\r\n\r\n“I think I should understand that better,” Alice said very politely,\r\n“if I had it written down: but I can’t quite follow it as you say it.”\r\n\r\n“That’s nothing to what I could say if I chose,” the Duchess replied,\r\nin a pleased tone.\r\n\r\n“Pray don’t trouble yourself to say it any longer than that,” said\r\nAlice.\r\n\r\n“Oh, don’t talk about trouble!” said the Duchess. “I make you a present\r\nof everything I’ve said as yet.”\r\n\r\n“A cheap sort of present!” thought Alice. “I’m glad they don’t give\r\nbirthday presents like that!” But she did not venture to say it out\r\nloud.\r\n\r\n“Thinking again?” the Duchess asked, with another dig of her sharp\r\nlittle chin.\r\n\r\n“I’ve a right to think,” said Alice sharply, for she was beginning to\r\nfeel a little worried.\r\n\r\n“Just about as much right,” said the Duchess, “as pigs have to fly; and\r\nthe m—”\r\n\r\nBut here, to Alice’s great surprise, the Duchess’s voice died away,\r\neven in the middle of her favourite word ‘moral,’ and the arm that was\r\nlinked into hers began to tremble. Alice looked up, and there stood the\r\nQueen in front of them, with her arms folded, frowning like a\r\nthunderstorm.\r\n\r\n“A fine day, your Majesty!” the Duchess began in a low, weak voice.\r\n\r\n“Now, I give you fair warning,” shouted the Queen, stamping on the\r\nground as she spoke; “either you or your head must be off, and that in\r\nabout half no time! Take your choice!”\r\n\r\nThe Duchess took her choice, and was gone in a moment.\r\n\r\n“Let’s go on with the game,” the Queen said to Alice; and Alice was too\r\nmuch frightened to say a word, but slowly followed her back to the\r\ncroquet-ground.\r\n\r\nThe other guests had taken advantage of the Queen’s absence, and were\r\nresting in the shade: however, the moment they saw her, they hurried\r\nback to the game, the Queen merely remarking that a moment’s delay\r\nwould cost them their lives.\r\n\r\nAll the time they were playing the Queen never left off quarrelling\r\nwith the other players, and shouting “Off with his head!” or “Off with\r\nher head!” Those whom she sentenced were taken into custody by the\r\nsoldiers, who of course had to leave off being arches to do this, so\r\nthat by the end of half an hour or so there were no arches left, and\r\nall the players, except the King, the Queen, and Alice, were in custody\r\nand under sentence of execution.\r\n\r\nThen the Queen left off, quite out of breath, and said to Alice, “Have\r\nyou seen the Mock Turtle yet?”\r\n\r\n“No,” said Alice. “I don’t even know what a Mock Turtle is.”\r\n\r\n“It’s the thing Mock Turtle Soup is made from,” said the Queen.\r\n\r\n“I never saw one, or heard of one,” said Alice.\r\n\r\n“Come on, then,” said the Queen, “and he shall tell you his history,”\r\n\r\nAs they walked off together, Alice heard the King say in a low voice,\r\nto the company generally, “You are all pardoned.” “Come, _that’s_ a\r\ngood thing!” she said to herself, for she had felt quite unhappy at the\r\nnumber of executions the Queen had ordered.\r\n\r\nThey very soon came upon a Gryphon, lying fast asleep in the sun. (If\r\nyou don’t know what a Gryphon is, look at the picture.) “Up, lazy\r\nthing!” said the Queen, “and take this young lady to see the Mock\r\nTurtle, and to hear his history. I must go back and see after some\r\nexecutions I have ordered;” and she walked off, leaving Alice alone\r\nwith the Gryphon. Alice did not quite like the look of the creature,\r\nbut on the whole she thought it would be quite as safe to stay with it\r\nas to go after that savage Queen: so she waited.\r\n\r\nThe Gryphon sat up and rubbed its eyes: then it watched the Queen till\r\nshe was out of sight: then it chuckled. “What fun!” said the Gryphon,\r\nhalf to itself, half to Alice.\r\n\r\n“What _is_ the fun?” said Alice.\r\n\r\n“Why, _she_,” said the Gryphon. “It’s all her fancy, that: they never\r\nexecutes nobody, you know. Come on!”\r\n\r\n“Everybody says ‘come on!’ here,” thought Alice, as she went slowly\r\nafter it: “I never was so ordered about in all my life, never!”\r\n\r\nThey had not gone far before they saw the Mock Turtle in the distance,\r\nsitting sad and lonely on a little ledge of rock, and, as they came\r\nnearer, Alice could hear him sighing as if his heart would break. She\r\npitied him deeply. “What is his sorrow?” she asked the Gryphon, and the\r\nGryphon answered, very nearly in the same words as before, “It’s all\r\nhis fancy, that: he hasn’t got no sorrow, you know. Come on!”\r\n\r\nSo they went up to the Mock Turtle, who looked at them with large eyes\r\nfull of tears, but said nothing.\r\n\r\n“This here young lady,” said the Gryphon, “she wants for to know your\r\nhistory, she do.”\r\n\r\n“I’ll tell it her,” said the Mock Turtle in a deep, hollow tone: “sit\r\ndown, both of you, and don’t speak a word till I’ve finished.”\r\n\r\nSo they sat down, and nobody spoke for some minutes. Alice thought to\r\nherself, “I don’t see how he can _ever_ finish, if he doesn’t begin.”\r\nBut she waited patiently.\r\n\r\n“Once,” said the Mock Turtle at last, with a deep sigh, “I was a real\r\nTurtle.”\r\n\r\nThese words were followed by a very long silence, broken only by an\r\noccasional exclamation of “Hjckrrh!” from the Gryphon, and the constant\r\nheavy sobbing of the Mock Turtle. Alice was very nearly getting up and\r\nsaying, “Thank you, sir, for your interesting story,” but she could not\r\nhelp thinking there _must_ be more to come, so she sat still and said\r\nnothing.\r\n\r\n“When we were little,” the Mock Turtle went on at last, more calmly,\r\nthough still sobbing a little now and then, “we went to school in the\r\nsea. The master was an old Turtle—we used to call him Tortoise—”\r\n\r\n“Why did you call him Tortoise, if he wasn’t one?” Alice asked.\r\n\r\n“We called him Tortoise because he taught us,” said the Mock Turtle\r\nangrily: “really you are very dull!”\r\n\r\n“You ought to be ashamed of yourself for asking such a simple\r\nquestion,” added the Gryphon; and then they both sat silent and looked\r\nat poor Alice, who felt ready to sink into the earth. At last the\r\nGryphon said to the Mock Turtle, “Drive on, old fellow! Don’t be all\r\nday about it!” and he went on in these words:\r\n\r\n“Yes, we went to school in the sea, though you mayn’t believe it—”\r\n\r\n“I never said I didn’t!” interrupted Alice.\r\n\r\n“You did,” said the Mock Turtle.\r\n\r\n“Hold your tongue!” added the Gryphon, before Alice could speak again.\r\nThe Mock Turtle went on.\r\n\r\n“We had the best of educations—in fact, we went to school every day—”\r\n\r\n“_I’ve_ been to a day-school, too,” said Alice; “you needn’t be so\r\nproud as all that.”\r\n\r\n“With extras?” asked the Mock Turtle a little anxiously.\r\n\r\n“Yes,” said Alice, “we learned French and music.”\r\n\r\n“And washing?” said the Mock Turtle.\r\n\r\n“Certainly not!” said Alice indignantly.\r\n\r\n“Ah! then yours wasn’t a really good school,” said the Mock Turtle in a\r\ntone of great relief. “Now at _ours_ they had at the end of the bill,\r\n‘French, music, _and washing_—extra.’”\r\n\r\n“You couldn’t have wanted it much,” said Alice; “living at the bottom\r\nof the sea.”\r\n\r\n“I couldn’t afford to learn it.” said the Mock Turtle with a sigh. “I\r\nonly took the regular course.”\r\n\r\n“What was that?” inquired Alice.\r\n\r\n“Reeling and Writhing, of course, to begin with,” the Mock Turtle\r\nreplied; “and then the different branches of Arithmetic—Ambition,\r\nDistraction, Uglification, and Derision.”\r\n\r\n“I never heard of ‘Uglification,’” Alice ventured to say. “What is it?”\r\n\r\nThe Gryphon lifted up both its paws in surprise. “What! Never heard of\r\nuglifying!” it exclaimed. “You know what to beautify is, I suppose?”\r\n\r\n“Yes,” said Alice doubtfully: “it means—to—make—anything—prettier.”\r\n\r\n“Well, then,” the Gryphon went on, “if you don’t know what to uglify\r\nis, you _are_ a simpleton.”\r\n\r\nAlice did not feel encouraged to ask any more questions about it, so\r\nshe turned to the Mock Turtle, and said “What else had you to learn?”\r\n\r\n“Well, there was Mystery,” the Mock Turtle replied, counting off the\r\nsubjects on his flappers, “—Mystery, ancient and modern, with\r\nSeaography: then Drawling—the Drawling-master was an old conger-eel,\r\nthat used to come once a week: _he_ taught us Drawling, Stretching, and\r\nFainting in Coils.”\r\n\r\n“What was _that_ like?” said Alice.\r\n\r\n“Well, I can’t show it you myself,” the Mock Turtle said: “I’m too\r\nstiff. And the Gryphon never learnt it.”\r\n\r\n“Hadn’t time,” said the Gryphon: “I went to the Classics master,\r\nthough. He was an old crab, _he_ was.”\r\n\r\n“I never went to him,” the Mock Turtle said with a sigh: “he taught\r\nLaughing and Grief, they used to say.”\r\n\r\n“So he did, so he did,” said the Gryphon, sighing in his turn; and both\r\ncreatures hid their faces in their paws.\r\n\r\n“And how many hours a day did you do lessons?” said Alice, in a hurry\r\nto change the subject.\r\n\r\n“Ten hours the first day,” said the Mock Turtle: “nine the next, and so\r\non.”\r\n\r\n“What a curious plan!” exclaimed Alice.\r\n\r\n“That’s the reason they’re called lessons,” the Gryphon remarked:\r\n“because they lessen from day to day.”\r\n\r\nThis was quite a new idea to Alice, and she thought it over a little\r\nbefore she made her next remark. “Then the eleventh day must have been\r\na holiday?”\r\n\r\n“Of course it was,” said the Mock Turtle.\r\n\r\n“And how did you manage on the twelfth?” Alice went on eagerly.\r\n\r\n“That’s enough about lessons,” the Gryphon interrupted in a very\r\ndecided tone: “tell her something about the games now.”\r\n\r\n\r\n\r\n\r\nCHAPTER X.\r\nThe Lobster Quadrille\r\n\r\n\r\nThe Mock Turtle sighed deeply, and drew the back of one flapper across\r\nhis eyes. He looked at Alice, and tried to speak, but for a minute or\r\ntwo sobs choked his voice. “Same as if he had a bone in his throat,”\r\nsaid the Gryphon: and it set to work shaking him and punching him in\r\nthe back. At last the Mock Turtle recovered his voice, and, with tears\r\nrunning down his cheeks, he went on again:—\r\n\r\n“You may not have lived much under the sea—” (“I haven’t,” said\r\nAlice)—“and perhaps you were never even introduced to a lobster—”\r\n(Alice began to say “I once tasted—” but checked herself hastily, and\r\nsaid “No, never”) “—so you can have no idea what a delightful thing a\r\nLobster Quadrille is!”\r\n\r\n“No, indeed,” said Alice. “What sort of a dance is it?”\r\n\r\n“Why,” said the Gryphon, “you first form into a line along the\r\nsea-shore—”\r\n\r\n“Two lines!” cried the Mock Turtle. “Seals, turtles, salmon, and so on;\r\nthen, when you’ve cleared all the jelly-fish out of the way—”\r\n\r\n“_That_ generally takes some time,” interrupted the Gryphon.\r\n\r\n“—you advance twice—”\r\n\r\n“Each with a lobster as a partner!” cried the Gryphon.\r\n\r\n“Of course,” the Mock Turtle said: “advance twice, set to partners—”\r\n\r\n“—change lobsters, and retire in same order,” continued the Gryphon.\r\n\r\n“Then, you know,” the Mock Turtle went on, “you throw the—”\r\n\r\n“The lobsters!” shouted the Gryphon, with a bound into the air.\r\n\r\n“—as far out to sea as you can—”\r\n\r\n“Swim after them!” screamed the Gryphon.\r\n\r\n“Turn a somersault in the sea!” cried the Mock Turtle, capering wildly\r\nabout.\r\n\r\n“Change lobsters again!” yelled the Gryphon at the top of its voice.\r\n\r\n“Back to land again, and that’s all the first figure,” said the Mock\r\nTurtle, suddenly dropping his voice; and the two creatures, who had\r\nbeen jumping about like mad things all this time, sat down again very\r\nsadly and quietly, and looked at Alice.\r\n\r\n“It must be a very pretty dance,” said Alice timidly.\r\n\r\n“Would you like to see a little of it?” said the Mock Turtle.\r\n\r\n“Very much indeed,” said Alice.\r\n\r\n“Come, let’s try the first figure!” said the Mock Turtle to the\r\nGryphon. “We can do without lobsters, you know. Which shall sing?”\r\n\r\n“Oh, _you_ sing,” said the Gryphon. “I’ve forgotten the words.”\r\n\r\nSo they began solemnly dancing round and round Alice, every now and\r\nthen treading on her toes when they passed too close, and waving their\r\nforepaws to mark the time, while the Mock Turtle sang this, very slowly\r\nand sadly:—\r\n\r\n“Will you walk a little faster?” said a whiting to a snail.\r\n“There’s a porpoise close behind us, and he’s treading on my tail.\r\nSee how eagerly the lobsters and the turtles all advance!\r\nThey are waiting on the shingle—will you come and join the dance?\r\nWill you, won’t you, will you, won’t you, will you join the dance?\r\nWill you, won’t you, will you, won’t you, won’t you join the dance?\r\n\r\n“You can really have no notion how delightful it will be\r\nWhen they take us up and throw us, with the lobsters, out to sea!”\r\nBut the snail replied “Too far, too far!” and gave a look askance—\r\nSaid he thanked the whiting kindly, but he would not join the dance.\r\nWould not, could not, would not, could not, would not join the dance.\r\nWould not, could not, would not, could not, could not join the dance.\r\n\r\n“What matters it how far we go?” his scaly friend replied.\r\n“There is another shore, you know, upon the other side.\r\nThe further off from England the nearer is to France—\r\nThen turn not pale, beloved snail, but come and join the dance.\r\nWill you, won’t you, will you, won’t you, will you join the dance?\r\nWill you, won’t you, will you, won’t you, won’t you join the dance?”\r\n\r\n\r\n“Thank you, it’s a very interesting dance to watch,” said Alice,\r\nfeeling very glad that it was over at last: “and I do so like that\r\ncurious song about the whiting!”\r\n\r\n“Oh, as to the whiting,” said the Mock Turtle, “they—you’ve seen them,\r\nof course?”\r\n\r\n“Yes,” said Alice, “I’ve often seen them at dinn—” she checked herself\r\nhastily.\r\n\r\n“I don’t know where Dinn may be,” said the Mock Turtle, “but if you’ve\r\nseen them so often, of course you know what they’re like.”\r\n\r\n“I believe so,” Alice replied thoughtfully. “They have their tails in\r\ntheir mouths—and they’re all over crumbs.”\r\n\r\n“You’re wrong about the crumbs,” said the Mock Turtle: “crumbs would\r\nall wash off in the sea. But they _have_ their tails in their mouths;\r\nand the reason is—” here the Mock Turtle yawned and shut his\r\neyes.—“Tell her about the reason and all that,” he said to the Gryphon.\r\n\r\n“The reason is,” said the Gryphon, “that they _would_ go with the\r\nlobsters to the dance. So they got thrown out to sea. So they had to\r\nfall a long way. So they got their tails fast in their mouths. So they\r\ncouldn’t get them out again. That’s all.”\r\n\r\n“Thank you,” said Alice, “it’s very interesting. I never knew so much\r\nabout a whiting before.”\r\n\r\n“I can tell you more than that, if you like,” said the Gryphon. “Do you\r\nknow why it’s called a whiting?”\r\n\r\n“I never thought about it,” said Alice. “Why?”\r\n\r\n“_It does the boots and shoes_,” the Gryphon replied very solemnly.\r\n\r\nAlice was thoroughly puzzled. “Does the boots and shoes!” she repeated\r\nin a wondering tone.\r\n\r\n“Why, what are _your_ shoes done with?” said the Gryphon. “I mean, what\r\nmakes them so shiny?”\r\n\r\nAlice looked down at them, and considered a little before she gave her\r\nanswer. “They’re done with blacking, I believe.”\r\n\r\n“Boots and shoes under the sea,” the Gryphon went on in a deep voice,\r\n“are done with a whiting. Now you know.”\r\n\r\n“And what are they made of?” Alice asked in a tone of great curiosity.\r\n\r\n“Soles and eels, of course,” the Gryphon replied rather impatiently:\r\n“any shrimp could have told you that.”\r\n\r\n“If I’d been the whiting,” said Alice, whose thoughts were still\r\nrunning on the song, “I’d have said to the porpoise, ‘Keep back,\r\nplease: we don’t want _you_ with us!’”\r\n\r\n“They were obliged to have him with them,” the Mock Turtle said: “no\r\nwise fish would go anywhere without a porpoise.”\r\n\r\n“Wouldn’t it really?” said Alice in a tone of great surprise.\r\n\r\n“Of course not,” said the Mock Turtle: “why, if a fish came to _me_,\r\nand told me he was going a journey, I should say ‘With what porpoise?’”\r\n\r\n“Don’t you mean ‘purpose’?” said Alice.\r\n\r\n“I mean what I say,” the Mock Turtle replied in an offended tone. And\r\nthe Gryphon added “Come, let’s hear some of _your_ adventures.”\r\n\r\n“I could tell you my adventures—beginning from this morning,” said\r\nAlice a little timidly: “but it’s no use going back to yesterday,\r\nbecause I was a different person then.”\r\n\r\n“Explain all that,” said the Mock Turtle.\r\n\r\n“No, no! The adventures first,” said the Gryphon in an impatient tone:\r\n“explanations take such a dreadful time.”\r\n\r\nSo Alice began telling them her adventures from the time when she first\r\nsaw the White Rabbit. She was a little nervous about it just at first,\r\nthe two creatures got so close to her, one on each side, and opened\r\ntheir eyes and mouths so _very_ wide, but she gained courage as she\r\nwent on. Her listeners were perfectly quiet till she got to the part\r\nabout her repeating “_You are old, Father William_,” to the\r\nCaterpillar, and the words all coming different, and then the Mock\r\nTurtle drew a long breath, and said “That’s very curious.”\r\n\r\n“It’s all about as curious as it can be,” said the Gryphon.\r\n\r\n“It all came different!” the Mock Turtle repeated thoughtfully. “I\r\nshould like to hear her try and repeat something now. Tell her to\r\nbegin.” He looked at the Gryphon as if he thought it had some kind of\r\nauthority over Alice.\r\n\r\n“Stand up and repeat ‘’_Tis the voice of the sluggard_,’” said the\r\nGryphon.\r\n\r\n“How the creatures order one about, and make one repeat lessons!”\r\nthought Alice; “I might as well be at school at once.” However, she got\r\nup, and began to repeat it, but her head was so full of the Lobster\r\nQuadrille, that she hardly knew what she was saying, and the words came\r\nvery queer indeed:—\r\n\r\n“’Tis the voice of the Lobster; I heard him declare,\r\n“You have baked me too brown, I must sugar my hair.”\r\nAs a duck with its eyelids, so he with his nose\r\nTrims his belt and his buttons, and turns out his toes.”\r\n\r\n[later editions continued as follows\r\nWhen the sands are all dry, he is gay as a lark,\r\nAnd will talk in contemptuous tones of the Shark,\r\nBut, when the tide rises and sharks are around,\r\nHis voice has a timid and tremulous sound.]\r\n\r\n\r\n“That’s different from what _I_ used to say when I was a child,” said\r\nthe Gryphon.\r\n\r\n“Well, I never heard it before,” said the Mock Turtle; “but it sounds\r\nuncommon nonsense.”\r\n\r\nAlice said nothing; she had sat down with her face in her hands,\r\nwondering if anything would _ever_ happen in a natural way again.\r\n\r\n“I should like to have it explained,” said the Mock Turtle.\r\n\r\n“She can’t explain it,” said the Gryphon hastily. “Go on with the next\r\nverse.”\r\n\r\n“But about his toes?” the Mock Turtle persisted. “How _could_ he turn\r\nthem out with his nose, you know?”\r\n\r\n“It’s the first position in dancing.” Alice said; but was dreadfully\r\npuzzled by the whole thing, and longed to change the subject.\r\n\r\n“Go on with the next verse,” the Gryphon repeated impatiently: “it\r\nbegins ‘_I passed by his garden_.’”\r\n\r\nAlice did not dare to disobey, though she felt sure it would all come\r\nwrong, and she went on in a trembling voice:—\r\n\r\n“I passed by his garden, and marked, with one eye,\r\nHow the Owl and the Panther were sharing a pie—”\r\n\r\n[later editions continued as follows\r\nThe Panther took pie-crust, and gravy, and meat,\r\nWhile the Owl had the dish as its share of the treat.\r\nWhen the pie was all finished, the Owl, as a boon,\r\nWas kindly permitted to pocket the spoon:\r\nWhile the Panther received knife and fork with a growl,\r\nAnd concluded the banquet—]\r\n\r\n\r\n“What _is_ the use of repeating all that stuff,” the Mock Turtle\r\ninterrupted, “if you don’t explain it as you go on? It’s by far the\r\nmost confusing thing _I_ ever heard!”\r\n\r\n“Yes, I think you’d better leave off,” said the Gryphon: and Alice was\r\nonly too glad to do so.\r\n\r\n“Shall we try another figure of the Lobster Quadrille?” the Gryphon\r\nwent on. “Or would you like the Mock Turtle to sing you a song?”\r\n\r\n“Oh, a song, please, if the Mock Turtle would be so kind,” Alice\r\nreplied, so eagerly that the Gryphon said, in a rather offended tone,\r\n“Hm! No accounting for tastes! Sing her ‘_Turtle Soup_,’ will you, old\r\nfellow?”\r\n\r\nThe Mock Turtle sighed deeply, and began, in a voice sometimes choked\r\nwith sobs, to sing this:—\r\n\r\n“Beautiful Soup, so rich and green,\r\nWaiting in a hot tureen!\r\nWho for such dainties would not stoop?\r\nSoup of the evening, beautiful Soup!\r\nSoup of the evening, beautiful Soup!\r\n    Beau—ootiful Soo—oop!\r\n    Beau—ootiful Soo—oop!\r\nSoo—oop of the e—e—evening,\r\n    Beautiful, beautiful Soup!\r\n\r\n“Beautiful Soup! Who cares for fish,\r\nGame, or any other dish?\r\nWho would not give all else for two p\r\nennyworth only of beautiful Soup?\r\nPennyworth only of beautiful Soup?\r\n    Beau—ootiful Soo—oop!\r\n    Beau—ootiful Soo—oop!\r\nSoo—oop of the e—e—evening,\r\n    Beautiful, beauti—FUL SOUP!”\r\n\r\n\r\n“Chorus again!” cried the Gryphon, and the Mock Turtle had just begun\r\nto repeat it, when a cry of “The trial’s beginning!” was heard in the\r\ndistance.\r\n\r\n“Come on!” cried the Gryphon, and, taking Alice by the hand, it hurried\r\noff, without waiting for the end of the song.\r\n\r\n“What trial is it?” Alice panted as she ran; but the Gryphon only\r\nanswered “Come on!” and ran the faster, while more and more faintly\r\ncame, carried on the breeze that followed them, the melancholy words:—\r\n\r\n“Soo—oop of the e—e—evening,\r\n    Beautiful, beautiful Soup!”\r\n\r\n\r\n\r\n\r\nCHAPTER XI.\r\nWho Stole the Tarts?\r\n\r\n\r\nThe King and Queen of Hearts were seated on their throne when they\r\narrived, with a great crowd assembled about them—all sorts of little\r\nbirds and beasts, as well as the whole pack of cards: the Knave was\r\nstanding before them, in chains, with a soldier on each side to guard\r\nhim; and near the King was the White Rabbit, with a trumpet in one\r\nhand, and a scroll of parchment in the other. In the very middle of the\r\ncourt was a table, with a large dish of tarts upon it: they looked so\r\ngood, that it made Alice quite hungry to look at them—“I wish they’d\r\nget the trial done,” she thought, “and hand round the refreshments!”\r\nBut there seemed to be no chance of this, so she began looking at\r\neverything about her, to pass away the time.\r\n\r\nAlice had never been in a court of justice before, but she had read\r\nabout them in books, and she was quite pleased to find that she knew\r\nthe name of nearly everything there. “That’s the judge,” she said to\r\nherself, “because of his great wig.”\r\n\r\nThe judge, by the way, was the King; and as he wore his crown over the\r\nwig, (look at the frontispiece if you want to see how he did it,) he\r\ndid not look at all comfortable, and it was certainly not becoming.\r\n\r\n“And that’s the jury-box,” thought Alice, “and those twelve creatures,”\r\n(she was obliged to say “creatures,” you see, because some of them were\r\nanimals, and some were birds,) “I suppose they are the jurors.” She\r\nsaid this last word two or three times over to herself, being rather\r\nproud of it: for she thought, and rightly too, that very few little\r\ngirls of her age knew the meaning of it at all. However, “jury-men”\r\nwould have done just as well.\r\n\r\nThe twelve jurors were all writing very busily on slates. “What are\r\nthey doing?” Alice whispered to the Gryphon. “They can’t have anything\r\nto put down yet, before the trial’s begun.”\r\n\r\n“They’re putting down their names,” the Gryphon whispered in reply,\r\n“for fear they should forget them before the end of the trial.”\r\n\r\n“Stupid things!” Alice began in a loud, indignant voice, but she\r\nstopped hastily, for the White Rabbit cried out, “Silence in the\r\ncourt!” and the King put on his spectacles and looked anxiously round,\r\nto make out who was talking.\r\n\r\nAlice could see, as well as if she were looking over their shoulders,\r\nthat all the jurors were writing down “stupid things!” on their slates,\r\nand she could even make out that one of them didn’t know how to spell\r\n“stupid,” and that he had to ask his neighbour to tell him. “A nice\r\nmuddle their slates’ll be in before the trial’s over!” thought Alice.\r\n\r\nOne of the jurors had a pencil that squeaked. This of course, Alice\r\ncould _not_ stand, and she went round the court and got behind him, and\r\nvery soon found an opportunity of taking it away. She did it so quickly\r\nthat the poor little juror (it was Bill, the Lizard) could not make out\r\nat all what had become of it; so, after hunting all about for it, he\r\nwas obliged to write with one finger for the rest of the day; and this\r\nwas of very little use, as it left no mark on the slate.\r\n\r\n“Herald, read the accusation!” said the King.\r\n\r\nOn this the White Rabbit blew three blasts on the trumpet, and then\r\nunrolled the parchment scroll, and read as follows:—\r\n\r\n“The Queen of Hearts, she made some tarts,\r\n    All on a summer day:\r\nThe Knave of Hearts, he stole those tarts,\r\n    And took them quite away!”\r\n\r\n\r\n“Consider your verdict,” the King said to the jury.\r\n\r\n“Not yet, not yet!” the Rabbit hastily interrupted. “There’s a great\r\ndeal to come before that!”\r\n\r\n“Call the first witness,” said the King; and the White Rabbit blew\r\nthree blasts on the trumpet, and called out, “First witness!”\r\n\r\nThe first witness was the Hatter. He came in with a teacup in one hand\r\nand a piece of bread-and-butter in the other. “I beg pardon, your\r\nMajesty,” he began, “for bringing these in: but I hadn’t quite finished\r\nmy tea when I was sent for.”\r\n\r\n“You ought to have finished,” said the King. “When did you begin?”\r\n\r\nThe Hatter looked at the March Hare, who had followed him into the\r\ncourt, arm-in-arm with the Dormouse. “Fourteenth of March, I _think_ it\r\nwas,” he said.\r\n\r\n“Fifteenth,” said the March Hare.\r\n\r\n“Sixteenth,” added the Dormouse.\r\n\r\n“Write that down,” the King said to the jury, and the jury eagerly\r\nwrote down all three dates on their slates, and then added them up, and\r\nreduced the answer to shillings and pence.\r\n\r\n“Take off your hat,” the King said to the Hatter.\r\n\r\n“It isn’t mine,” said the Hatter.\r\n\r\n“_Stolen!_” the King exclaimed, turning to the jury, who instantly made\r\na memorandum of the fact.\r\n\r\n“I keep them to sell,” the Hatter added as an explanation; “I’ve none\r\nof my own. I’m a hatter.”\r\n\r\nHere the Queen put on her spectacles, and began staring at the Hatter,\r\nwho turned pale and fidgeted.\r\n\r\n“Give your evidence,” said the King; “and don’t be nervous, or I’ll\r\nhave you executed on the spot.”\r\n\r\nThis did not seem to encourage the witness at all: he kept shifting\r\nfrom one foot to the other, looking uneasily at the Queen, and in his\r\nconfusion he bit a large piece out of his teacup instead of the\r\nbread-and-butter.\r\n\r\nJust at this moment Alice felt a very curious sensation, which puzzled\r\nher a good deal until she made out what it was: she was beginning to\r\ngrow larger again, and she thought at first she would get up and leave\r\nthe court; but on second thoughts she decided to remain where she was\r\nas long as there was room for her.\r\n\r\n“I wish you wouldn’t squeeze so.” said the Dormouse, who was sitting\r\nnext to her. “I can hardly breathe.”\r\n\r\n“I can’t help it,” said Alice very meekly: “I’m growing.”\r\n\r\n“You’ve no right to grow _here_,” said the Dormouse.\r\n\r\n“Don’t talk nonsense,” said Alice more boldly: “you know you’re growing\r\ntoo.”\r\n\r\n“Yes, but _I_ grow at a reasonable pace,” said the Dormouse: “not in\r\nthat ridiculous fashion.” And he got up very sulkily and crossed over\r\nto the other side of the court.\r\n\r\nAll this time the Queen had never left off staring at the Hatter, and,\r\njust as the Dormouse crossed the court, she said to one of the officers\r\nof the court, “Bring me the list of the singers in the last concert!”\r\non which the wretched Hatter trembled so, that he shook both his shoes\r\noff.\r\n\r\n“Give your evidence,” the King repeated angrily, “or I’ll have you\r\nexecuted, whether you’re nervous or not.”\r\n\r\n“I’m a poor man, your Majesty,” the Hatter began, in a trembling voice,\r\n“—and I hadn’t begun my tea—not above a week or so—and what with the\r\nbread-and-butter getting so thin—and the twinkling of the tea—”\r\n\r\n“The twinkling of the _what?_” said the King.\r\n\r\n“It _began_ with the tea,” the Hatter replied.\r\n\r\n“Of course twinkling begins with a T!” said the King sharply. “Do you\r\ntake me for a dunce? Go on!”\r\n\r\n“I’m a poor man,” the Hatter went on, “and most things twinkled after\r\nthat—only the March Hare said—”\r\n\r\n“I didn’t!” the March Hare interrupted in a great hurry.\r\n\r\n“You did!” said the Hatter.\r\n\r\n“I deny it!” said the March Hare.\r\n\r\n“He denies it,” said the King: “leave out that part.”\r\n\r\n“Well, at any rate, the Dormouse said—” the Hatter went on, looking\r\nanxiously round to see if he would deny it too: but the Dormouse denied\r\nnothing, being fast asleep.\r\n\r\n“After that,” continued the Hatter, “I cut some more bread-and-butter—”\r\n\r\n“But what did the Dormouse say?” one of the jury asked.\r\n\r\n“That I can’t remember,” said the Hatter.\r\n\r\n“You _must_ remember,” remarked the King, “or I’ll have you executed.”\r\n\r\nThe miserable Hatter dropped his teacup and bread-and-butter, and went\r\ndown on one knee. “I’m a poor man, your Majesty,” he began.\r\n\r\n“You’re a _very_ poor _speaker_,” said the King.\r\n\r\nHere one of the guinea-pigs cheered, and was immediately suppressed by\r\nthe officers of the court. (As that is rather a hard word, I will just\r\nexplain to you how it was done. They had a large canvas bag, which tied\r\nup at the mouth with strings: into this they slipped the guinea-pig,\r\nhead first, and then sat upon it.)\r\n\r\n“I’m glad I’ve seen that done,” thought Alice. “I’ve so often read in\r\nthe newspapers, at the end of trials, “There was some attempts at\r\napplause, which was immediately suppressed by the officers of the\r\ncourt,” and I never understood what it meant till now.”\r\n\r\n“If that’s all you know about it, you may stand down,” continued the\r\nKing.\r\n\r\n“I can’t go no lower,” said the Hatter: “I’m on the floor, as it is.”\r\n\r\n“Then you may _sit_ down,” the King replied.\r\n\r\nHere the other guinea-pig cheered, and was suppressed.\r\n\r\n“Come, that finished the guinea-pigs!” thought Alice. “Now we shall get\r\non better.”\r\n\r\n“I’d rather finish my tea,” said the Hatter, with an anxious look at\r\nthe Queen, who was reading the list of singers.\r\n\r\n“You may go,” said the King, and the Hatter hurriedly left the court,\r\nwithout even waiting to put his shoes on.\r\n\r\n“—and just take his head off outside,” the Queen added to one of the\r\nofficers: but the Hatter was out of sight before the officer could get\r\nto the door.\r\n\r\n“Call the next witness!” said the King.\r\n\r\nThe next witness was the Duchess’s cook. She carried the pepper-box in\r\nher hand, and Alice guessed who it was, even before she got into the\r\ncourt, by the way the people near the door began sneezing all at once.\r\n\r\n“Give your evidence,” said the King.\r\n\r\n“Shan’t,” said the cook.\r\n\r\nThe King looked anxiously at the White Rabbit, who said in a low voice,\r\n“Your Majesty must cross-examine _this_ witness.”\r\n\r\n“Well, if I must, I must,” the King said, with a melancholy air, and,\r\nafter folding his arms and frowning at the cook till his eyes were\r\nnearly out of sight, he said in a deep voice, “What are tarts made of?”\r\n\r\n“Pepper, mostly,” said the cook.\r\n\r\n“Treacle,” said a sleepy voice behind her.\r\n\r\n“Collar that Dormouse,” the Queen shrieked out. “Behead that Dormouse!\r\nTurn that Dormouse out of court! Suppress him! Pinch him! Off with his\r\nwhiskers!”\r\n\r\nFor some minutes the whole court was in confusion, getting the Dormouse\r\nturned out, and, by the time they had settled down again, the cook had\r\ndisappeared.\r\n\r\n“Never mind!” said the King, with an air of great relief. “Call the\r\nnext witness.” And he added in an undertone to the Queen, “Really, my\r\ndear, _you_ must cross-examine the next witness. It quite makes my\r\nforehead ache!”\r\n\r\nAlice watched the White Rabbit as he fumbled over the list, feeling\r\nvery curious to see what the next witness would be like, “—for they\r\nhaven’t got much evidence _yet_,” she said to herself. Imagine her\r\nsurprise, when the White Rabbit read out, at the top of his shrill\r\nlittle voice, the name “Alice!”\r\n\r\n\r\n\r\n\r\nCHAPTER XII.\r\nAlice’s Evidence\r\n\r\n\r\n“Here!” cried Alice, quite forgetting in the flurry of the moment how\r\nlarge she had grown in the last few minutes, and she jumped up in such\r\na hurry that she tipped over the jury-box with the edge of her skirt,\r\nupsetting all the jurymen on to the heads of the crowd below, and there\r\nthey lay sprawling about, reminding her very much of a globe of\r\ngoldfish she had accidentally upset the week before.\r\n\r\n“Oh, I _beg_ your pardon!” she exclaimed in a tone of great dismay, and\r\nbegan picking them up again as quickly as she could, for the accident\r\nof the goldfish kept running in her head, and she had a vague sort of\r\nidea that they must be collected at once and put back into the\r\njury-box, or they would die.\r\n\r\n“The trial cannot proceed,” said the King in a very grave voice, “until\r\nall the jurymen are back in their proper places—_all_,” he repeated\r\nwith great emphasis, looking hard at Alice as he said so.\r\n\r\nAlice looked at the jury-box, and saw that, in her haste, she had put\r\nthe Lizard in head downwards, and the poor little thing was waving its\r\ntail about in a melancholy way, being quite unable to move. She soon\r\ngot it out again, and put it right; “not that it signifies much,” she\r\nsaid to herself; “I should think it would be _quite_ as much use in the\r\ntrial one way up as the other.”\r\n\r\nAs soon as the jury had a little recovered from the shock of being\r\nupset, and their slates and pencils had been found and handed back to\r\nthem, they set to work very diligently to write out a history of the\r\naccident, all except the Lizard, who seemed too much overcome to do\r\nanything but sit with its mouth open, gazing up into the roof of the\r\ncourt.\r\n\r\n“What do you know about this business?” the King said to Alice.\r\n\r\n“Nothing,” said Alice.\r\n\r\n“Nothing _whatever?_” persisted the King.\r\n\r\n“Nothing whatever,” said Alice.\r\n\r\n“That’s very important,” the King said, turning to the jury. They were\r\njust beginning to write this down on their slates, when the White\r\nRabbit interrupted: “_Un_important, your Majesty means, of course,” he\r\nsaid in a very respectful tone, but frowning and making faces at him as\r\nhe spoke.\r\n\r\n“_Un_important, of course, I meant,” the King hastily said, and went on\r\nto himself in an undertone,\r\n\r\n“important—unimportant—unimportant—important—” as if he were trying\r\nwhich word sounded best.\r\n\r\nSome of the jury wrote it down “important,” and some “unimportant.”\r\nAlice could see this, as she was near enough to look over their slates;\r\n“but it doesn’t matter a bit,” she thought to herself.\r\n\r\nAt this moment the King, who had been for some time busily writing in\r\nhis note-book, cackled out “Silence!” and read out from his book, “Rule\r\nForty-two. _All persons more than a mile high to leave the court_.”\r\n\r\nEverybody looked at Alice.\r\n\r\n“_I’m_ not a mile high,” said Alice.\r\n\r\n“You are,” said the King.\r\n\r\n“Nearly two miles high,” added the Queen.\r\n\r\n“Well, I shan’t go, at any rate,” said Alice: “besides, that’s not a\r\nregular rule: you invented it just now.”\r\n\r\n“It’s the oldest rule in the book,” said the King.\r\n\r\n“Then it ought to be Number One,” said Alice.\r\n\r\nThe King turned pale, and shut his note-book hastily. “Consider your\r\nverdict,” he said to the jury, in a low, trembling voice.\r\n\r\n“There’s more evidence to come yet, please your Majesty,” said the\r\nWhite Rabbit, jumping up in a great hurry; “this paper has just been\r\npicked up.”\r\n\r\n“What’s in it?” said the Queen.\r\n\r\n“I haven’t opened it yet,” said the White Rabbit, “but it seems to be a\r\nletter, written by the prisoner to—to somebody.”\r\n\r\n“It must have been that,” said the King, “unless it was written to\r\nnobody, which isn’t usual, you know.”\r\n\r\n“Who is it directed to?” said one of the jurymen.\r\n\r\n“It isn’t directed at all,” said the White Rabbit; “in fact, there’s\r\nnothing written on the _outside_.” He unfolded the paper as he spoke,\r\nand added “It isn’t a letter, after all: it’s a set of verses.”\r\n\r\n“Are they in the prisoner’s handwriting?” asked another of the jurymen.\r\n\r\n“No, they’re not,” said the White Rabbit, “and that’s the queerest\r\nthing about it.” (The jury all looked puzzled.)\r\n\r\n“He must have imitated somebody else’s hand,” said the King. (The jury\r\nall brightened up again.)\r\n\r\n“Please your Majesty,” said the Knave, “I didn’t write it, and they\r\ncan’t prove I did: there’s no name signed at the end.”\r\n\r\n“If you didn’t sign it,” said the King, “that only makes the matter\r\nworse. You _must_ have meant some mischief, or else you’d have signed\r\nyour name like an honest man.”\r\n\r\nThere was a general clapping of hands at this: it was the first really\r\nclever thing the King had said that day.\r\n\r\n“That _proves_ his guilt,” said the Queen.\r\n\r\n“It proves nothing of the sort!” said Alice. “Why, you don’t even know\r\nwhat they’re about!”\r\n\r\n“Read them,” said the King.\r\n\r\nThe White Rabbit put on his spectacles. “Where shall I begin, please\r\nyour Majesty?” he asked.\r\n\r\n“Begin at the beginning,” the King said gravely, “and go on till you\r\ncome to the end: then stop.”\r\n\r\nThese were the verses the White Rabbit read:—\r\n\r\n“They told me you had been to her,\r\n    And mentioned me to him:\r\nShe gave me a good character,\r\n    But said I could not swim.\r\n\r\nHe sent them word I had not gone\r\n    (We know it to be true):\r\nIf she should push the matter on,\r\n    What would become of you?\r\n\r\nI gave her one, they gave him two,\r\n    You gave us three or more;\r\nThey all returned from him to you,\r\n    Though they were mine before.\r\n\r\nIf I or she should chance to be\r\n    Involved in this affair,\r\nHe trusts to you to set them free,\r\n    Exactly as we were.\r\n\r\nMy notion was that you had been\r\n    (Before she had this fit)\r\nAn obstacle that came between\r\n    Him, and ourselves, and it.\r\n\r\nDon’t let him know she liked them best,\r\n    For this must ever be\r\nA secret, kept from all the rest,\r\n    Between yourself and me.”\r\n\r\n\r\n“That’s the most important piece of evidence we’ve heard yet,” said the\r\nKing, rubbing his hands; “so now let the jury—”\r\n\r\n“If any one of them can explain it,” said Alice, (she had grown so\r\nlarge in the last few minutes that she wasn’t a bit afraid of\r\ninterrupting him,) “I’ll give him sixpence. _I_ don’t believe there’s\r\nan atom of meaning in it.”\r\n\r\nThe jury all wrote down on their slates, “_She_ doesn’t believe there’s\r\nan atom of meaning in it,” but none of them attempted to explain the\r\npaper.\r\n\r\n“If there’s no meaning in it,” said the King, “that saves a world of\r\ntrouble, you know, as we needn’t try to find any. And yet I don’t\r\nknow,” he went on, spreading out the verses on his knee, and looking at\r\nthem with one eye; “I seem to see some meaning in them, after all.\r\n“—_said I could not swim_—” you can’t swim, can you?” he added, turning\r\nto the Knave.\r\n\r\nThe Knave shook his head sadly. “Do I look like it?” he said. (Which he\r\ncertainly did _not_, being made entirely of cardboard.)\r\n\r\n“All right, so far,” said the King, and he went on muttering over the\r\nverses to himself: “‘_We know it to be true_—’ that’s the jury, of\r\ncourse—‘_I gave her one, they gave him two_—’ why, that must be what he\r\ndid with the tarts, you know—”\r\n\r\n“But, it goes on ‘_they all returned from him to you_,’” said Alice.\r\n\r\n“Why, there they are!” said the King triumphantly, pointing to the\r\ntarts on the table. “Nothing can be clearer than _that_. Then\r\nagain—‘_before she had this fit_—’ you never had fits, my dear, I\r\nthink?” he said to the Queen.\r\n\r\n“Never!” said the Queen furiously, throwing an inkstand at the Lizard\r\nas she spoke. (The unfortunate little Bill had left off writing on his\r\nslate with one finger, as he found it made no mark; but he now hastily\r\nbegan again, using the ink, that was trickling down his face, as long\r\nas it lasted.)\r\n\r\n“Then the words don’t _fit_ you,” said the King, looking round the\r\ncourt with a smile. There was a dead silence.\r\n\r\n“It’s a pun!” the King added in an offended tone, and everybody\r\nlaughed, “Let the jury consider their verdict,” the King said, for\r\nabout the twentieth time that day.\r\n\r\n“No, no!” said the Queen. “Sentence first—verdict afterwards.”\r\n\r\n“Stuff and nonsense!” said Alice loudly. “The idea of having the\r\nsentence first!”\r\n\r\n“Hold your tongue!” said the Queen, turning purple.\r\n\r\n“I won’t!” said Alice.\r\n\r\n“Off with her head!” the Queen shouted at the top of her voice. Nobody\r\nmoved.\r\n\r\n“Who cares for you?” said Alice, (she had grown to her full size by\r\nthis time.) “You’re nothing but a pack of cards!”\r\n\r\nAt this the whole pack rose up into the air, and came flying down upon\r\nher: she gave a little scream, half of fright and half of anger, and\r\ntried to beat them off, and found herself lying on the bank, with her\r\nhead in the lap of her sister, who was gently brushing away some dead\r\nleaves that had fluttered down from the trees upon her face.\r\n\r\n“Wake up, Alice dear!” said her sister; “Why, what a long sleep you’ve\r\nhad!”\r\n\r\n“Oh, I’ve had such a curious dream!” said Alice, and she told her\r\nsister, as well as she could remember them, all these strange\r\nAdventures of hers that you have just been reading about; and when she\r\nhad finished, her sister kissed her, and said, “It _was_ a curious\r\ndream, dear, certainly: but now run in to your tea; it’s getting late.”\r\nSo Alice got up and ran off, thinking while she ran, as well she might,\r\nwhat a wonderful dream it had been.\r\n\r\n\r\nBut her sister sat still just as she left her, leaning her head on her\r\nhand, watching the setting sun, and thinking of little Alice and all\r\nher wonderful Adventures, till she too began dreaming after a fashion,\r\nand this was her dream:—\r\n\r\nFirst, she dreamed of little Alice herself, and once again the tiny\r\nhands were clasped upon her knee, and the bright eager eyes were\r\nlooking up into hers—she could hear the very tones of her voice, and\r\nsee that queer little toss of her head to keep back the wandering hair\r\nthat _would_ always get into her eyes—and still as she listened, or\r\nseemed to listen, the whole place around her became alive with the\r\nstrange creatures of her little sister’s dream.\r\n\r\nThe long grass rustled at her feet as the White Rabbit hurried by—the\r\nfrightened Mouse splashed his way through the neighbouring pool—she\r\ncould hear the rattle of the teacups as the March Hare and his friends\r\nshared their never-ending meal, and the shrill voice of the Queen\r\nordering off her unfortunate guests to execution—once more the pig-baby\r\nwas sneezing on the Duchess’s knee, while plates and dishes crashed\r\naround it—once more the shriek of the Gryphon, the squeaking of the\r\nLizard’s slate-pencil, and the choking of the suppressed guinea-pigs,\r\nfilled the air, mixed up with the distant sobs of the miserable Mock\r\nTurtle.\r\n\r\nSo she sat on, with closed eyes, and half believed herself in\r\nWonderland, though she knew she had but to open them again, and all\r\nwould change to dull reality—the grass would be only rustling in the\r\nwind, and the pool rippling to the waving of the reeds—the rattling\r\nteacups would change to tinkling sheep-bells, and the Queen’s shrill\r\ncries to the voice of the shepherd boy—and the sneeze of the baby, the\r\nshriek of the Gryphon, and all the other queer noises, would change\r\n(she knew) to the confused clamour of the busy farm-yard—while the\r\nlowing of the cattle in the distance would take the place of the Mock\r\nTurtle’s heavy sobs.\r\n\r\nLastly, she pictured to herself how this same little sister of hers\r\nwould, in the after-time, be herself a grown woman; and how she would\r\nkeep, through all her riper years, the simple and loving heart of her\r\nchildhood: and how she would gather about her other little children,\r\nand make _their_ eyes bright and eager with many a strange tale,\r\nperhaps even with the dream of Wonderland of long ago: and how she\r\nwould feel with all their simple sorrows, and find a pleasure in all\r\ntheir simple joys, remembering her own child-life, and the happy summer\r\ndays.\r\n\r\nTHE END \r\n\r\n\r\n\r\n\r\nEnd of Project Gutenberg’s Alice’s Adventures in Wonderland, by Lewis Carroll\r\n\r\n*** END OF THIS PROJECT GUTENBERG EBOOK ALICE’S ADVENTURES IN WONDERLAND ***\r\n\r\n***** This file should be named 11-0.txt or 11-0.zip *****\r\nThis and all associated files of various formats will be found in:\r\n        http://www.gutenberg.org/1/11/\r\n\r\nProduced by Arthur DiBianca and David Widger\r\n\r\nUpdated editions will replace the previous one--the old editions will\r\nbe renamed.\r\n\r\nCreating the works from print editions not protected by U.S. copyright\r\nlaw means that no one owns a United States copyright in these works,\r\nso the Foundation (and you!) can copy and distribute it in the United\r\nStates without permission and without paying copyright\r\nroyalties. Special rules, set forth in the General Terms of Use part\r\nof this license, apply to copying and distributing Project\r\nGutenberg-tm electronic works to protect the PROJECT GUTENBERG-tm\r\nconcept and trademark. Project Gutenberg is a registered trademark,\r\nand may not be used if you charge for the eBooks, unless you receive\r\nspecific permission. If you do not charge anything for copies of this\r\neBook, complying with the rules is very easy. You may use this eBook\r\nfor nearly any purpose such as creation of derivative works, reports,\r\nperformances and research. They may be modified and printed and given\r\naway--you may do practically ANYTHING in the United States with eBooks\r\nnot protected by U.S. copyright law. Redistribution is subject to the\r\ntrademark license, especially commercial redistribution.\r\n\r\nSTART: FULL LICENSE\r\n\r\nTHE FULL PROJECT GUTENBERG LICENSE\r\nPLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK\r\n\r\nTo protect the Project Gutenberg-tm mission of promoting the free\r\ndistribution of electronic works, by using or distributing this work\r\n(or any other work associated in any way with the phrase \"Project\r\nGutenberg\"), you agree to comply with all the terms of the Full\r\nProject Gutenberg-tm License available with this file or online at\r\nwww.gutenberg.org/license.\r\n\r\nSection 1. General Terms of Use and Redistributing Project\r\nGutenberg-tm electronic works\r\n\r\n1.A. By reading or using any part of this Project Gutenberg-tm\r\nelectronic work, you indicate that you have read, understand, agree to\r\nand accept all the terms of this license and intellectual property\r\n(trademark/copyright) agreement. If you do not agree to abide by all\r\nthe terms of this agreement, you must cease using and return or\r\ndestroy all copies of Project Gutenberg-tm electronic works in your\r\npossession. If you paid a fee for obtaining a copy of or access to a\r\nProject Gutenberg-tm electronic work and you do not agree to be bound\r\nby the terms of this agreement, you may obtain a refund from the\r\nperson or entity to whom you paid the fee as set forth in paragraph\r\n1.E.8.\r\n\r\n1.B. \"Project Gutenberg\" is a registered trademark. It may only be\r\nused on or associated in any way with an electronic work by people who\r\nagree to be bound by the terms of this agreement. There are a few\r\nthings that you can do with most Project Gutenberg-tm electronic works\r\neven without complying with the full terms of this agreement. See\r\nparagraph 1.C below. There are a lot of things you can do with Project\r\nGutenberg-tm electronic works if you follow the terms of this\r\nagreement and help preserve free future access to Project Gutenberg-tm\r\nelectronic works. See paragraph 1.E below.\r\n\r\n1.C. The Project Gutenberg Literary Archive Foundation (\"the\r\nFoundation\" or PGLAF), owns a compilation copyright in the collection\r\nof Project Gutenberg-tm electronic works. Nearly all the individual\r\nworks in the collection are in the public domain in the United\r\nStates. If an individual work is unprotected by copyright law in the\r\nUnited States and you are located in the United States, we do not\r\nclaim a right to prevent you from copying, distributing, performing,\r\ndisplaying or creating derivative works based on the work as long as\r\nall references to Project Gutenberg are removed. Of course, we hope\r\nthat you will support the Project Gutenberg-tm mission of promoting\r\nfree access to electronic works by freely sharing Project Gutenberg-tm\r\nworks in compliance with the terms of this agreement for keeping the\r\nProject Gutenberg-tm name associated with the work. You can easily\r\ncomply with the terms of this agreement by keeping this work in the\r\nsame format with its attached full Project Gutenberg-tm License when\r\nyou share it without charge with others.\r\n\r\n1.D. The copyright laws of the place where you are located also govern\r\nwhat you can do with this work. Copyright laws in most countries are\r\nin a constant state of change. If you are outside the United States,\r\ncheck the laws of your country in addition to the terms of this\r\nagreement before downloading, copying, displaying, performing,\r\ndistributing or creating derivative works based on this work or any\r\nother Project Gutenberg-tm work. The Foundation makes no\r\nrepresentations concerning the copyright status of any work in any\r\ncountry outside the United States.\r\n\r\n1.E. Unless you have removed all references to Project Gutenberg:\r\n\r\n1.E.1. The following sentence, with active links to, or other\r\nimmediate access to, the full Project Gutenberg-tm License must appear\r\nprominently whenever any copy of a Project Gutenberg-tm work (any work\r\non which the phrase \"Project Gutenberg\" appears, or with which the\r\nphrase \"Project Gutenberg\" is associated) is accessed, displayed,\r\nperformed, viewed, copied or distributed:\r\n\r\n  This eBook is for the use of anyone anywhere in the United States and\r\n  most other parts of the world at no cost and with almost no\r\n  restrictions whatsoever. You may copy it, give it away or re-use it\r\n  under the terms of the Project Gutenberg License included with this\r\n  eBook or online at www.gutenberg.org. If you are not located in the\r\n  United States, you'll have to check the laws of the country where you\r\n  are located before using this ebook.\r\n\r\n1.E.2. If an individual Project Gutenberg-tm electronic work is\r\nderived from texts not protected by U.S. copyright law (does not\r\ncontain a notice indicating that it is posted with permission of the\r\ncopyright holder), the work can be copied and distributed to anyone in\r\nthe United States without paying any fees or charges. If you are\r\nredistributing or providing access to a work with the phrase \"Project\r\nGutenberg\" associated with or appearing on the work, you must comply\r\neither with the requirements of paragraphs 1.E.1 through 1.E.7 or\r\nobtain permission for the use of the work and the Project Gutenberg-tm\r\ntrademark as set forth in paragraphs 1.E.8 or 1.E.9.\r\n\r\n1.E.3. If an individual Project Gutenberg-tm electronic work is posted\r\nwith the permission of the copyright holder, your use and distribution\r\nmust comply with both paragraphs 1.E.1 through 1.E.7 and any\r\nadditional terms imposed by the copyright holder. Additional terms\r\nwill be linked to the Project Gutenberg-tm License for all works\r\nposted with the permission of the copyright holder found at the\r\nbeginning of this work.\r\n\r\n1.E.4. Do not unlink or detach or remove the full Project Gutenberg-tm\r\nLicense terms from this work, or any files containing a part of this\r\nwork or any other work associated with Project Gutenberg-tm.\r\n\r\n1.E.5. Do not copy, display, perform, distribute or redistribute this\r\nelectronic work, or any part of this electronic work, without\r\nprominently displaying the sentence set forth in paragraph 1.E.1 with\r\nactive links or immediate access to the full terms of the Project\r\nGutenberg-tm License.\r\n\r\n1.E.6. You may convert to and distribute this work in any binary,\r\ncompressed, marked up, nonproprietary or proprietary form, including\r\nany word processing or hypertext form. However, if you provide access\r\nto or distribute copies of a Project Gutenberg-tm work in a format\r\nother than \"Plain Vanilla ASCII\" or other format used in the official\r\nversion posted on the official Project Gutenberg-tm web site\r\n(www.gutenberg.org), you must, at no additional cost, fee or expense\r\nto the user, provide a copy, a means of exporting a copy, or a means\r\nof obtaining a copy upon request, of the work in its original \"Plain\r\nVanilla ASCII\" or other form. Any alternate format must include the\r\nfull Project Gutenberg-tm License as specified in paragraph 1.E.1.\r\n\r\n1.E.7. Do not charge a fee for access to, viewing, displaying,\r\nperforming, copying or distributing any Project Gutenberg-tm works\r\nunless you comply with paragraph 1.E.8 or 1.E.9.\r\n\r\n1.E.8. You may charge a reasonable fee for copies of or providing\r\naccess to or distributing Project Gutenberg-tm electronic works\r\nprovided that\r\n\r\n* You pay a royalty fee of 20% of the gross profits you derive from\r\n  the use of Project Gutenberg-tm works calculated using the method\r\n  you already use to calculate your applicable taxes. The fee is owed\r\n  to the owner of the Project Gutenberg-tm trademark, but he has\r\n  agreed to donate royalties under this paragraph to the Project\r\n  Gutenberg Literary Archive Foundation. Royalty payments must be paid\r\n  within 60 days following each date on which you prepare (or are\r\n  legally required to prepare) your periodic tax returns. Royalty\r\n  payments should be clearly marked as such and sent to the Project\r\n  Gutenberg Literary Archive Foundation at the address specified in\r\n  Section 4, \"Information about donations to the Project Gutenberg\r\n  Literary Archive Foundation.\"\r\n\r\n* You provide a full refund of any money paid by a user who notifies\r\n  you in writing (or by e-mail) within 30 days of receipt that s/he\r\n  does not agree to the terms of the full Project Gutenberg-tm\r\n  License. You must require such a user to return or destroy all\r\n  copies of the works possessed in a physical medium and discontinue\r\n  all use of and all access to other copies of Project Gutenberg-tm\r\n  works.\r\n\r\n* You provide, in accordance with paragraph 1.F.3, a full refund of\r\n  any money paid for a work or a replacement copy, if a defect in the\r\n  electronic work is discovered and reported to you within 90 days of\r\n  receipt of the work.\r\n\r\n* You comply with all other terms of this agreement for free\r\n  distribution of Project Gutenberg-tm works.\r\n\r\n1.E.9. If you wish to charge a fee or distribute a Project\r\nGutenberg-tm electronic work or group of works on different terms than\r\nare set forth in this agreement, you must obtain permission in writing\r\nfrom both the Project Gutenberg Literary Archive Foundation and The\r\nProject Gutenberg Trademark LLC, the owner of the Project Gutenberg-tm\r\ntrademark. Contact the Foundation as set forth in Section 3 below.\r\n\r\n1.F.\r\n\r\n1.F.1. Project Gutenberg volunteers and employees expend considerable\r\neffort to identify, do copyright research on, transcribe and proofread\r\nworks not protected by U.S. copyright law in creating the Project\r\nGutenberg-tm collection. Despite these efforts, Project Gutenberg-tm\r\nelectronic works, and the medium on which they may be stored, may\r\ncontain \"Defects,\" such as, but not limited to, incomplete, inaccurate\r\nor corrupt data, transcription errors, a copyright or other\r\nintellectual property infringement, a defective or damaged disk or\r\nother medium, a computer virus, or computer codes that damage or\r\ncannot be read by your equipment.\r\n\r\n1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the \"Right\r\nof Replacement or Refund\" described in paragraph 1.F.3, the Project\r\nGutenberg Literary Archive Foundation, the owner of the Project\r\nGutenberg-tm trademark, and any other party distributing a Project\r\nGutenberg-tm electronic work under this agreement, disclaim all\r\nliability to you for damages, costs and expenses, including legal\r\nfees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT\r\nLIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT EXCEPT THOSE\r\nPROVIDED IN PARAGRAPH 1.F.3. YOU AGREE THAT THE FOUNDATION, THE\r\nTRADEMARK OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE\r\nLIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE OR\r\nINCIDENTAL DAMAGES EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH\r\nDAMAGE.\r\n\r\n1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a\r\ndefect in this electronic work within 90 days of receiving it, you can\r\nreceive a refund of the money (if any) you paid for it by sending a\r\nwritten explanation to the person you received the work from. If you\r\nreceived the work on a physical medium, you must return the medium\r\nwith your written explanation. The person or entity that provided you\r\nwith the defective work may elect to provide a replacement copy in\r\nlieu of a refund. If you received the work electronically, the person\r\nor entity providing it to you may choose to give you a second\r\nopportunity to receive the work electronically in lieu of a refund. If\r\nthe second copy is also defective, you may demand a refund in writing\r\nwithout further opportunities to fix the problem.\r\n\r\n1.F.4. Except for the limited right of replacement or refund set forth\r\nin paragraph 1.F.3, this work is provided to you 'AS-IS', WITH NO\r\nOTHER WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT\r\nLIMITED TO WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PURPOSE.\r\n\r\n1.F.5. Some states do not allow disclaimers of certain implied\r\nwarranties or the exclusion or limitation of certain types of\r\ndamages. If any disclaimer or limitation set forth in this agreement\r\nviolates the law of the state applicable to this agreement, the\r\nagreement shall be interpreted to make the maximum disclaimer or\r\nlimitation permitted by the applicable state law. The invalidity or\r\nunenforceability of any provision of this agreement shall not void the\r\nremaining provisions.\r\n\r\n1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the\r\ntrademark owner, any agent or employee of the Foundation, anyone\r\nproviding copies of Project Gutenberg-tm electronic works in\r\naccordance with this agreement, and any volunteers associated with the\r\nproduction, promotion and distribution of Project Gutenberg-tm\r\nelectronic works, harmless from all liability, costs and expenses,\r\nincluding legal fees, that arise directly or indirectly from any of\r\nthe following which you do or cause to occur: (a) distribution of this\r\nor any Project Gutenberg-tm work, (b) alteration, modification, or\r\nadditions or deletions to any Project Gutenberg-tm work, and (c) any\r\nDefect you cause.\r\n\r\nSection 2. Information about the Mission of Project Gutenberg-tm\r\n\r\nProject Gutenberg-tm is synonymous with the free distribution of\r\nelectronic works in formats readable by the widest variety of\r\ncomputers including obsolete, old, middle-aged and new computers. It\r\nexists because of the efforts of hundreds of volunteers and donations\r\nfrom people in all walks of life.\r\n\r\nVolunteers and financial support to provide volunteers with the\r\nassistance they need are critical to reaching Project Gutenberg-tm's\r\ngoals and ensuring that the Project Gutenberg-tm collection will\r\nremain freely available for generations to come. In 2001, the Project\r\nGutenberg Literary Archive Foundation was created to provide a secure\r\nand permanent future for Project Gutenberg-tm and future\r\ngenerations. To learn more about the Project Gutenberg Literary\r\nArchive Foundation and how your efforts and donations can help, see\r\nSections 3 and 4 and the Foundation information page at\r\nwww.gutenberg.org\r\n\r\n\r\n\r\nSection 3. Information about the Project Gutenberg Literary Archive Foundation\r\n\r\nThe Project Gutenberg Literary Archive Foundation is a non profit\r\n501(c)(3) educational corporation organized under the laws of the\r\nstate of Mississippi and granted tax exempt status by the Internal\r\nRevenue Service. The Foundation's EIN or federal tax identification\r\nnumber is 64-6221541. Contributions to the Project Gutenberg Literary\r\nArchive Foundation are tax deductible to the full extent permitted by\r\nU.S. federal laws and your state's laws.\r\n\r\nThe Foundation's principal office is in Fairbanks, Alaska, with the\r\nmailing address: PO Box 750175, Fairbanks, AK 99775, but its\r\nvolunteers and employees are scattered throughout numerous\r\nlocations. Its business office is located at 809 North 1500 West, Salt\r\nLake City, UT 84116, (801) 596-1887. Email contact links and up to\r\ndate contact information can be found at the Foundation's web site and\r\nofficial page at www.gutenberg.org/contact\r\n\r\nFor additional contact information:\r\n\r\n    Dr. Gregory B. Newby\r\n    Chief Executive and Director\r\n    gbnewby@pglaf.org\r\n\r\nSection 4. Information about Donations to the Project Gutenberg\r\nLiterary Archive Foundation\r\n\r\nProject Gutenberg-tm depends upon and cannot survive without wide\r\nspread public support and donations to carry out its mission of\r\nincreasing the number of public domain and licensed works that can be\r\nfreely distributed in machine readable form accessible by the widest\r\narray of equipment including outdated equipment. Many small donations\r\n($1 to $5,000) are particularly important to maintaining tax exempt\r\nstatus with the IRS.\r\n\r\nThe Foundation is committed to complying with the laws regulating\r\ncharities and charitable donations in all 50 states of the United\r\nStates. Compliance requirements are not uniform and it takes a\r\nconsiderable effort, much paperwork and many fees to meet and keep up\r\nwith these requirements. We do not solicit donations in locations\r\nwhere we have not received written confirmation of compliance. To SEND\r\nDONATIONS or determine the status of compliance for any particular\r\nstate visit www.gutenberg.org/donate\r\n\r\nWhile we cannot and do not solicit contributions from states where we\r\nhave not met the solicitation requirements, we know of no prohibition\r\nagainst accepting unsolicited donations from donors in such states who\r\napproach us with offers to donate.\r\n\r\nInternational donations are gratefully accepted, but we cannot make\r\nany statements concerning tax treatment of donations received from\r\noutside the United States. U.S. laws alone swamp our small staff.\r\n\r\nPlease check the Project Gutenberg Web pages for current donation\r\nmethods and addresses. Donations are accepted in a number of other\r\nways including checks, online payments and credit card donations. To\r\ndonate, please visit: www.gutenberg.org/donate\r\n\r\nSection 5. General Information About Project Gutenberg-tm electronic works.\r\n\r\nProfessor Michael S. Hart was the originator of the Project\r\nGutenberg-tm concept of a library of electronic works that could be\r\nfreely shared with anyone. For forty years, he produced and\r\ndistributed Project Gutenberg-tm eBooks with only a loose network of\r\nvolunteer support.\r\n\r\nProject Gutenberg-tm eBooks are often created from several printed\r\neditions, all of which are confirmed as not protected by copyright in\r\nthe U.S. unless a copyright notice is included. Thus, we do not\r\nnecessarily keep eBooks in compliance with any particular paper\r\nedition.\r\n\r\nMost people start at our Web site which has the main PG search\r\nfacility: www.gutenberg.org\r\n\r\nThis Web site includes information about Project Gutenberg-tm,\r\nincluding how to make donations to the Project Gutenberg Literary\r\nArchive Foundation, how to help produce our new eBooks, and how to\r\nsubscribe to our email newsletter to hear about new eBooks.\r\n\r\n\r\n"
  },
  {
    "path": "benches/analyzer.rs",
    "content": "use criterion::{criterion_group, criterion_main, Criterion};\nuse tantivy::tokenizer::{\n    LowerCaser, RemoveLongFilter, SimpleTokenizer, TextAnalyzer, TokenizerManager,\n};\n\nconst ALICE_TXT: &str = include_str!(\"alice.txt\");\n\npub fn criterion_benchmark(c: &mut Criterion) {\n    let tokenizer_manager = TokenizerManager::default();\n    let mut tokenizer = tokenizer_manager.get(\"default\").unwrap();\n    c.bench_function(\"default-tokenize-alice\", |b| {\n        b.iter(|| {\n            let mut word_count = 0;\n            let mut token_stream = tokenizer.token_stream(ALICE_TXT);\n            while token_stream.advance() {\n                word_count += 1;\n            }\n            assert_eq!(word_count, 30_731);\n        })\n    });\n    let mut dynamic_analyzer = TextAnalyzer::builder(SimpleTokenizer::default())\n        .dynamic()\n        .filter_dynamic(RemoveLongFilter::limit(40))\n        .filter_dynamic(LowerCaser)\n        .build();\n    c.bench_function(\"dynamic-tokenize-alice\", |b| {\n        b.iter(|| {\n            let mut word_count = 0;\n            let mut token_stream = dynamic_analyzer.token_stream(ALICE_TXT);\n            while token_stream.advance() {\n                word_count += 1;\n            }\n            assert_eq!(word_count, 30_731);\n        })\n    });\n}\n\ncriterion_group! {\n    name = benches;\n    config = Criterion::default().sample_size(200);\n    targets = criterion_benchmark\n}\ncriterion_main!(benches);\n"
  },
  {
    "path": "benches/and_or_queries.rs",
    "content": "// Benchmarks boolean conjunction queries using binggan.\n//\n// What’s measured:\n// - Or and And queries with varying selectivity (only `Term` queries for now on leafs)\n// - Nested AND/OR combinations (on multiple fields)\n// - No-scoring path using the Count collector (focus on iterator/skip performance)\n// - Top-K retrieval (k=10) using the TopDocs collector\n//\n// Corpus model:\n// - Synthetic docs; each token a/b/c is independently included per doc\n// - If none of a/b/c are included, emit a neutral filler token to keep doc length similar\n//\n// Notes:\n// - After optimization, when scoring is disabled Tantivy reads doc-only postings\n//   (IndexRecordOption::Basic), avoiding frequency decoding overhead.\n// - This bench isolates boolean iteration speed and intersection/union cost.\n// - Use `cargo bench --bench boolean_conjunction` to run.\n\nuse binggan::{black_box, BenchGroup, BenchRunner};\nuse rand::prelude::*;\nuse rand::rngs::StdRng;\nuse rand::SeedableRng;\nuse tantivy::collector::sort_key::SortByStaticFastValue;\nuse tantivy::collector::{Collector, Count, TopDocs};\nuse tantivy::query::{Query, QueryParser};\nuse tantivy::schema::{Schema, FAST, TEXT};\nuse tantivy::{doc, Index, Order, ReloadPolicy, Searcher};\n\n#[derive(Clone)]\nstruct BenchIndex {\n    #[allow(dead_code)]\n    index: Index,\n    searcher: Searcher,\n    query_parser: QueryParser,\n}\n\n/// Build a single index containing both fields (title, body) and\n/// return two BenchIndex views:\n/// - single_field: QueryParser defaults to only \"body\"\n/// - multi_field:  QueryParser defaults to [\"title\", \"body\"]\nfn build_shared_indices(num_docs: usize, p_a: f32, p_b: f32, p_c: f32) -> (BenchIndex, BenchIndex) {\n    // Unified schema (two text fields)\n    let mut schema_builder = Schema::builder();\n    let f_title = schema_builder.add_text_field(\"title\", TEXT);\n    let f_body = schema_builder.add_text_field(\"body\", TEXT);\n    let f_score = schema_builder.add_u64_field(\"score\", FAST);\n    let f_score2 = schema_builder.add_u64_field(\"score2\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n\n    // Populate index with stable RNG for reproducibility.\n    let mut rng = StdRng::from_seed([7u8; 32]);\n\n    // Populate: spread each present token 90/10 to body/title\n    {\n        let mut writer = index.writer_with_num_threads(1, 500_000_000).unwrap();\n        for _ in 0..num_docs {\n            let has_a = rng.random_bool(p_a as f64);\n            let has_b = rng.random_bool(p_b as f64);\n            let has_c = rng.random_bool(p_c as f64);\n            let score = rng.random_range(0u64..100u64);\n            let score2 = rng.random_range(0u64..100_000u64);\n            let mut title_tokens: Vec<&str> = Vec::new();\n            let mut body_tokens: Vec<&str> = Vec::new();\n            if has_a {\n                if rng.random_bool(0.1) {\n                    title_tokens.push(\"a\");\n                } else {\n                    body_tokens.push(\"a\");\n                }\n            }\n            if has_b {\n                if rng.random_bool(0.1) {\n                    title_tokens.push(\"b\");\n                } else {\n                    body_tokens.push(\"b\");\n                }\n            }\n            if has_c {\n                if rng.random_bool(0.1) {\n                    title_tokens.push(\"c\");\n                } else {\n                    body_tokens.push(\"c\");\n                }\n            }\n            if title_tokens.is_empty() && body_tokens.is_empty() {\n                body_tokens.push(\"z\");\n            }\n            writer\n                .add_document(doc!(\n                    f_title=>title_tokens.join(\" \"),\n                    f_body=>body_tokens.join(\" \"),\n                    f_score=>score,\n                    f_score2=>score2,\n                ))\n                .unwrap();\n        }\n        writer.commit().unwrap();\n    }\n\n    // Prepare reader/searcher once.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::Manual)\n        .try_into()\n        .unwrap();\n    let searcher = reader.searcher();\n\n    // Build two query parsers with different default fields.\n    let qp_single = QueryParser::for_index(&index, vec![f_body]);\n    let qp_multi = QueryParser::for_index(&index, vec![f_title, f_body]);\n\n    let single_view = BenchIndex {\n        index: index.clone(),\n        searcher: searcher.clone(),\n        query_parser: qp_single,\n    };\n    let multi_view = BenchIndex {\n        index,\n        searcher,\n        query_parser: qp_multi,\n    };\n    (single_view, multi_view)\n}\n\nfn main() {\n    // Prepare corpora with varying selectivity. Build one index per corpus\n    // and derive two views (single-field vs multi-field) from it.\n    let scenarios = vec![\n        (\n            \"N=1M, p(a)=5%, p(b)=1%, p(c)=15%\".to_string(),\n            1_000_000,\n            0.05,\n            0.01,\n            0.15,\n        ),\n        (\n            \"N=1M, p(a)=1%, p(b)=1%, p(c)=15%\".to_string(),\n            1_000_000,\n            0.01,\n            0.01,\n            0.15,\n        ),\n    ];\n\n    let queries = &[\"a\", \"+a +b\", \"+a +b +c\", \"a OR b\", \"a OR b OR c\"];\n\n    let mut runner = BenchRunner::new();\n    for (label, n, pa, pb, pc) in scenarios {\n        let (single_view, multi_view) = build_shared_indices(n, pa, pb, pc);\n\n        for (view_name, bench_index) in [(\"single_field\", single_view), (\"multi_field\", multi_view)]\n        {\n            // Single-field group: default field is body only\n            let mut group = runner.new_group();\n            group.set_name(format!(\"{} — {}\", view_name, label));\n            for query_str in queries {\n                add_bench_task(&mut group, &bench_index, query_str, Count, \"count\");\n                add_bench_task(\n                    &mut group,\n                    &bench_index,\n                    query_str,\n                    TopDocs::with_limit(10).order_by_score(),\n                    \"top10\",\n                );\n                add_bench_task(\n                    &mut group,\n                    &bench_index,\n                    query_str,\n                    TopDocs::with_limit(10).order_by_fast_field::<u64>(\"score\", Order::Asc),\n                    \"top10_by_ff\",\n                );\n                add_bench_task(\n                    &mut group,\n                    &bench_index,\n                    query_str,\n                    TopDocs::with_limit(10).order_by((\n                        SortByStaticFastValue::<u64>::for_field(\"score\"),\n                        SortByStaticFastValue::<u64>::for_field(\"score2\"),\n                    )),\n                    \"top10_by_2ff\",\n                );\n            }\n            group.run();\n        }\n    }\n}\n\nfn add_bench_task<C: Collector + 'static>(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query_str: &str,\n    collector: C,\n    collector_name: &str,\n) {\n    let task_name = format!(\"{}_{}\", query_str.replace(\" \", \"_\"), collector_name);\n    let query = bench_index.query_parser.parse_query(query_str).unwrap();\n    let search_task = SearchTask {\n        searcher: bench_index.searcher.clone(),\n        collector,\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nstruct SearchTask<C: Collector> {\n    searcher: Searcher,\n    collector: C,\n    query: Box<dyn Query>,\n}\n\nimpl<C: Collector> SearchTask<C> {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        self.searcher.search(&self.query, &self.collector).unwrap();\n        1\n    }\n}\n"
  },
  {
    "path": "benches/bool_queries_with_range.rs",
    "content": "use binggan::{black_box, BenchGroup, BenchRunner};\nuse rand::prelude::*;\nuse rand::rngs::StdRng;\nuse rand::SeedableRng;\nuse tantivy::collector::{Collector, Count, DocSetCollector, TopDocs};\nuse tantivy::query::{Query, QueryParser};\nuse tantivy::schema::{Schema, FAST, INDEXED, TEXT};\nuse tantivy::{doc, Index, Order, ReloadPolicy, Searcher};\n\n#[derive(Clone)]\nstruct BenchIndex {\n    #[allow(dead_code)]\n    index: Index,\n    searcher: Searcher,\n    query_parser: QueryParser,\n}\n\nfn build_shared_indices(num_docs: usize, p_title_a: f32, distribution: &str) -> BenchIndex {\n    // Unified schema\n    let mut schema_builder = Schema::builder();\n    let f_title = schema_builder.add_text_field(\"title\", TEXT);\n    let f_num_rand = schema_builder.add_u64_field(\"num_rand\", INDEXED);\n    let f_num_asc = schema_builder.add_u64_field(\"num_asc\", INDEXED);\n    let f_num_rand_fast = schema_builder.add_u64_field(\"num_rand_fast\", INDEXED | FAST);\n    let f_num_asc_fast = schema_builder.add_u64_field(\"num_asc_fast\", INDEXED | FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n\n    // Populate index with stable RNG for reproducibility.\n    let mut rng = StdRng::from_seed([7u8; 32]);\n\n    {\n        let mut writer = index.writer_with_num_threads(1, 4_000_000_000).unwrap();\n\n        match distribution {\n            \"dense\" => {\n                for doc_id in 0..num_docs {\n                    // Always add title to avoid empty documents\n                    let title_token = if rng.random_bool(p_title_a as f64) {\n                        \"a\"\n                    } else {\n                        \"b\"\n                    };\n\n                    let num_rand = rng.random_range(0u64..1000u64);\n\n                    let num_asc = (doc_id / 10000) as u64;\n\n                    writer\n                        .add_document(doc!(\n                            f_title=>title_token,\n                            f_num_rand=>num_rand,\n                            f_num_asc=>num_asc,\n                            f_num_rand_fast=>num_rand,\n                            f_num_asc_fast=>num_asc,\n                        ))\n                        .unwrap();\n                }\n            }\n            \"sparse\" => {\n                for doc_id in 0..num_docs {\n                    // Always add title to avoid empty documents\n                    let title_token = if rng.random_bool(p_title_a as f64) {\n                        \"a\"\n                    } else {\n                        \"b\"\n                    };\n\n                    let num_rand = rng.random_range(0u64..10000000u64);\n\n                    let num_asc = doc_id as u64;\n\n                    writer\n                        .add_document(doc!(\n                            f_title=>title_token,\n                            f_num_rand=>num_rand,\n                            f_num_asc=>num_asc,\n                            f_num_rand_fast=>num_rand,\n                            f_num_asc_fast=>num_asc,\n                        ))\n                        .unwrap();\n                }\n            }\n            _ => {\n                panic!(\"Unsupported distribution type\");\n            }\n        }\n        writer.commit().unwrap();\n    }\n\n    // Prepare reader/searcher once.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::Manual)\n        .try_into()\n        .unwrap();\n    let searcher = reader.searcher();\n\n    // Build query parser for title field\n    let qp_title = QueryParser::for_index(&index, vec![f_title]);\n\n    BenchIndex {\n        index,\n        searcher,\n        query_parser: qp_title,\n    }\n}\n\nfn main() {\n    // Prepare corpora with varying scenarios\n    let scenarios = vec![\n        (\n            \"dense and 99% a\".to_string(),\n            10_000_000,\n            0.99,\n            \"dense\",\n            0,\n            9,\n        ),\n        (\n            \"dense and 99% a\".to_string(),\n            10_000_000,\n            0.99,\n            \"dense\",\n            990,\n            999,\n        ),\n        (\n            \"sparse and 99% a\".to_string(),\n            10_000_000,\n            0.99,\n            \"sparse\",\n            0,\n            9,\n        ),\n        (\n            \"sparse and 99% a\".to_string(),\n            10_000_000,\n            0.99,\n            \"sparse\",\n            9_999_990,\n            9_999_999,\n        ),\n    ];\n\n    let mut runner = BenchRunner::new();\n    for (scenario_id, n, p_title_a, num_rand_distribution, range_low, range_high) in scenarios {\n        // Build index for this scenario\n        let bench_index = build_shared_indices(n, p_title_a, num_rand_distribution);\n\n        // Create benchmark group\n        let mut group = runner.new_group();\n\n        // Now set the name (this moves scenario_id)\n        group.set_name(scenario_id);\n\n        // Define all four field types\n        let field_names = [\"num_rand\", \"num_asc\", \"num_rand_fast\", \"num_asc_fast\"];\n\n        // Define the three terms we want to test with\n        let terms = [\"a\", \"b\", \"z\"];\n\n        // Generate all combinations of terms and field names\n        let mut queries = Vec::new();\n        for &term in &terms {\n            for &field_name in &field_names {\n                let query_str = format!(\n                    \"{} AND {}:[{} TO {}]\",\n                    term, field_name, range_low, range_high\n                );\n                queries.push((query_str, field_name.to_string()));\n            }\n        }\n\n        let query_str = format!(\n            \"{}:[{} TO {}] AND {}:[{} TO {}]\",\n            \"num_rand_fast\", range_low, range_high, \"num_asc_fast\", range_low, range_high\n        );\n        queries.push((query_str, \"num_asc_fast\".to_string()));\n\n        // Run all benchmark tasks for each query and its corresponding field name\n        for (query_str, field_name) in queries {\n            run_benchmark_tasks(&mut group, &bench_index, &query_str, &field_name);\n        }\n\n        group.run();\n    }\n}\n\n/// Run all benchmark tasks for a given query string and field name\nfn run_benchmark_tasks(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query_str: &str,\n    field_name: &str,\n) {\n    // Test count\n    add_bench_task(bench_group, bench_index, query_str, Count, \"count\");\n\n    // Test all results\n    add_bench_task(\n        bench_group,\n        bench_index,\n        query_str,\n        DocSetCollector,\n        \"all results\",\n    );\n\n    // Test top 100 by the field (if it's a FAST field)\n    if field_name.ends_with(\"_fast\") {\n        // Ascending order\n        {\n            let collector_name = format!(\"top100_by_{}_asc\", field_name);\n            let field_name_owned = field_name.to_string();\n            add_bench_task(\n                bench_group,\n                bench_index,\n                query_str,\n                TopDocs::with_limit(100).order_by_fast_field::<u64>(field_name_owned, Order::Asc),\n                &collector_name,\n            );\n        }\n\n        // Descending order\n        {\n            let collector_name = format!(\"top100_by_{}_desc\", field_name);\n            let field_name_owned = field_name.to_string();\n            add_bench_task(\n                bench_group,\n                bench_index,\n                query_str,\n                TopDocs::with_limit(100).order_by_fast_field::<u64>(field_name_owned, Order::Desc),\n                &collector_name,\n            );\n        }\n    }\n}\n\nfn add_bench_task<C: Collector + 'static>(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query_str: &str,\n    collector: C,\n    collector_name: &str,\n) {\n    let task_name = format!(\"{}_{}\", query_str.replace(\" \", \"_\"), collector_name);\n    let query = bench_index.query_parser.parse_query(query_str).unwrap();\n    let search_task = SearchTask {\n        searcher: bench_index.searcher.clone(),\n        collector,\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nstruct SearchTask<C: Collector> {\n    searcher: Searcher,\n    collector: C,\n    query: Box<dyn Query>,\n}\n\nimpl<C: Collector> SearchTask<C> {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        let result = self.searcher.search(&self.query, &self.collector).unwrap();\n        if let Some(count) = (&result as &dyn std::any::Any).downcast_ref::<usize>() {\n            *count\n        } else if let Some(top_docs) = (&result as &dyn std::any::Any)\n            .downcast_ref::<Vec<(Option<u64>, tantivy::DocAddress)>>()\n        {\n            top_docs.len()\n        } else if let Some(top_docs) =\n            (&result as &dyn std::any::Any).downcast_ref::<Vec<(u64, tantivy::DocAddress)>>()\n        {\n            top_docs.len()\n        } else if let Some(doc_set) = (&result as &dyn std::any::Any)\n            .downcast_ref::<std::collections::HashSet<tantivy::DocAddress>>()\n        {\n            doc_set.len()\n        } else {\n            eprintln!(\n                \"Unknown collector result type: {:?}\",\n                std::any::type_name::<C::Fruit>()\n            );\n            0\n        }\n    }\n}\n"
  },
  {
    "path": "benches/exists_json.rs",
    "content": "use binggan::plugins::PeakMemAllocPlugin;\nuse binggan::{black_box, InputGroup, PeakMemAlloc, INSTRUMENTED_SYSTEM};\nuse serde_json::json;\nuse tantivy::collector::Count;\nuse tantivy::query::ExistsQuery;\nuse tantivy::schema::{Schema, FAST, TEXT};\nuse tantivy::{doc, Index};\n\n#[global_allocator]\npub static GLOBAL: &PeakMemAlloc<std::alloc::System> = &INSTRUMENTED_SYSTEM;\n\nfn main() {\n    let doc_count: usize = 500_000;\n    let subfield_counts: &[usize] = &[1, 2, 3, 4, 5, 6, 7, 8, 16, 256, 4096, 65536, 262144];\n\n    let indices: Vec<(String, Index)> = subfield_counts\n        .iter()\n        .map(|&sub_fields| {\n            (\n                format!(\"subfields={sub_fields}\"),\n                build_index_with_json_subfields(doc_count, sub_fields),\n            )\n        })\n        .collect();\n\n    let mut group = InputGroup::new_with_inputs(indices);\n    group.add_plugin(PeakMemAllocPlugin::new(GLOBAL));\n\n    group.config().num_iter_group = Some(1);\n    group.config().num_iter_bench = Some(1);\n    group.register(\"exists_json\", exists_json_union);\n\n    group.run();\n}\n\nfn exists_json_union(index: &Index) {\n    let reader = index.reader().expect(\"reader\");\n    let searcher = reader.searcher();\n    let query = ExistsQuery::new(\"json\".to_string(), true);\n    let count = searcher.search(&query, &Count).expect(\"exists search\");\n    // Prevents optimizer from eliding the search\n    black_box(count);\n}\n\nfn build_index_with_json_subfields(num_docs: usize, num_subfields: usize) -> Index {\n    // Schema: single JSON field stored as FAST to support ExistsQuery.\n    let mut schema_builder = Schema::builder();\n    let json_field = schema_builder.add_json_field(\"json\", TEXT | FAST);\n    let schema = schema_builder.build();\n\n    let index = Index::create_from_tempdir(schema).expect(\"create index\");\n    {\n        let mut index_writer = index\n            .writer_with_num_threads(1, 200_000_000)\n            .expect(\"writer\");\n        for i in 0..num_docs {\n            let sub = i % num_subfields;\n            // Only one subpath set per document; rotate subpaths so that\n            // no single subpath is full, but the union covers all docs.\n            let v = json!({ format!(\"field_{sub}\"): i as u64 });\n            index_writer\n                .add_document(doc!(json_field => v))\n                .expect(\"add_document\");\n        }\n        index_writer.commit().expect(\"commit\");\n    }\n\n    index\n}\n"
  },
  {
    "path": "benches/gh.json",
    "content": "{\"id\":\"2489395767\",\"type\":\"PushEvent\",\"actor\":{\"id\":1310570,\"login\":\"soumith\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/soumith\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1310570?\"},\"repo\":{\"id\":28067809,\"name\":\"soumith/fbcunn\",\"url\":\"https://api.github.com/repos/soumith/fbcunn\"},\"payload\":{\"push_id\":536752122,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/master\",\"head\":\"fa6048ec9b9eeafd12cee5f81324f355e1f2a198\",\"before\":\"2d06657267b32e0c8e193c617039da200f710195\",\"commits\":[{\"sha\":\"dbd68d30ee1f7b60d404553fc1c6226ebb374c8e\",\"author\":{\"email\":\"88de463b5797707cf3425f85a415c3d869db732b@gmail.com\",\"name\":\"Soumith Chintala\"},\"message\":\"back to old structure, except lua files moved out\",\"distinct\":true,\"url\":\"https://api.github.com/repos/soumith/fbcunn/commits/dbd68d30ee1f7b60d404553fc1c6226ebb374c8e\"},{\"sha\":\"5567f9f5a83d7fe3320b18e5b89405e8a5ca77e6\",\"author\":{\"email\":\"88de463b5797707cf3425f85a415c3d869db732b@gmail.com\",\"name\":\"Soumith Chintala\"},\"message\":\"...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/soumith/fbcunn/commits/5567f9f5a83d7fe3320b18e5b89405e8a5ca77e6\"},{\"sha\":\"58a83b277328eca811d3a37cf171b2fc4fcd87af\",\"author\":{\"email\":\"88de463b5797707cf3425f85a415c3d869db732b@gmail.com\",\"name\":\"Soumith Chintala\"},\"message\":\"...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/soumith/fbcunn/commits/58a83b277328eca811d3a37cf171b2fc4fcd87af\"},{\"sha\":\"fa6048ec9b9eeafd12cee5f81324f355e1f2a198\",\"author\":{\"email\":\"88de463b5797707cf3425f85a415c3d869db732b@gmail.com\",\"name\":\"Soumith Chintala\"},\"message\":\"...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/soumith/fbcunn/commits/fa6048ec9b9eeafd12cee5f81324f355e1f2a198\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:00Z\"}\n{\"id\":\"2489395768\",\"type\":\"PushEvent\",\"actor\":{\"id\":227068,\"login\":\"radix\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/radix\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/227068?\"},\"repo\":{\"id\":20022094,\"name\":\"radix/effect\",\"url\":\"https://api.github.com/repos/radix/effect\"},\"payload\":{\"push_id\":536752123,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"defdbe78db98ad69d72f42b09194309f47616592\",\"before\":\"d4c2903c26a8e50e7605405281b9d407855ff4c3\",\"commits\":[{\"sha\":\"defdbe78db98ad69d72f42b09194309f47616592\",\"author\":{\"email\":\"5f33e8ddd36b0c849687df732835b9abbe9b347b@twistedmatrix.com\",\"name\":\"Christopher Armstrong\"},\"message\":\"put the auto-generated API docs in the repository so readthedocs will work.\\nsigh.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/radix/effect/commits/defdbe78db98ad69d72f42b09194309f47616592\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:00Z\"}\n{\"id\":\"2489395770\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":1341245,\"login\":\"asfgit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/asfgit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1341245?\"},\"repo\":{\"id\":17165658,\"name\":\"apache/spark\",\"url\":\"https://api.github.com/repos/apache/spark\"},\"payload\":{\"action\":\"closed\",\"number\":3842,\"pull_request\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/3842\",\"id\":26684124,\"html_url\":\"https://github.com/apache/spark/pull/3842\",\"diff_url\":\"https://github.com/apache/spark/pull/3842.diff\",\"patch_url\":\"https://github.com/apache/spark/pull/3842.patch\",\"issue_url\":\"https://api.github.com/repos/apache/spark/issues/3842\",\"number\":3842,\"state\":\"closed\",\"locked\":false,\"title\":\"SPARK-2757 [BUILD] [STREAMING] Add Mima test for Spark Sink after 1.10 is released\",\"user\":{\"login\":\"srowen\",\"id\":822522,\"avatar_url\":\"https://avatars.githubusercontent.com/u/822522?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/srowen\",\"html_url\":\"https://github.com/srowen\",\"followers_url\":\"https://api.github.com/users/srowen/followers\",\"following_url\":\"https://api.github.com/users/srowen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/srowen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/srowen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/srowen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/srowen/orgs\",\"repos_url\":\"https://api.github.com/users/srowen/repos\",\"events_url\":\"https://api.github.com/users/srowen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/srowen/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Re-enable MiMa for Streaming Flume Sink module, now that 1.1.0 is released, per the JIRA TO-DO. That's pretty much all there is to this.\",\"created_at\":\"2014-12-30T12:47:17Z\",\"updated_at\":\"2015-01-01T01:00:02Z\",\"closed_at\":\"2015-01-01T01:00:02Z\",\"merged_at\":null,\"merge_commit_sha\":\"c66df2c9084212b592f5eb7a47fcebec542fda65\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/apache/spark/pulls/3842/commits\",\"review_comments_url\":\"https://api.github.com/repos/apache/spark/pulls/3842/comments\",\"review_comment_url\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/issues/3842/comments\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/50ff80e4498c2cb0a30793fb41fa2d20942811d6\",\"head\":{\"label\":\"srowen:SPARK-2757\",\"ref\":\"SPARK-2757\",\"sha\":\"50ff80e4498c2cb0a30793fb41fa2d20942811d6\",\"user\":{\"login\":\"srowen\",\"id\":822522,\"avatar_url\":\"https://avatars.githubusercontent.com/u/822522?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/srowen\",\"html_url\":\"https://github.com/srowen\",\"followers_url\":\"https://api.github.com/users/srowen/followers\",\"following_url\":\"https://api.github.com/users/srowen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/srowen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/srowen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/srowen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/srowen/orgs\",\"repos_url\":\"https://api.github.com/users/srowen/repos\",\"events_url\":\"https://api.github.com/users/srowen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/srowen/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":17241201,\"name\":\"spark\",\"full_name\":\"srowen/spark\",\"owner\":{\"login\":\"srowen\",\"id\":822522,\"avatar_url\":\"https://avatars.githubusercontent.com/u/822522?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/srowen\",\"html_url\":\"https://github.com/srowen\",\"followers_url\":\"https://api.github.com/users/srowen/followers\",\"following_url\":\"https://api.github.com/users/srowen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/srowen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/srowen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/srowen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/srowen/orgs\",\"repos_url\":\"https://api.github.com/users/srowen/repos\",\"events_url\":\"https://api.github.com/users/srowen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/srowen/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/srowen/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":true,\"url\":\"https://api.github.com/repos/srowen/spark\",\"forks_url\":\"https://api.github.com/repos/srowen/spark/forks\",\"keys_url\":\"https://api.github.com/repos/srowen/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/srowen/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/srowen/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/srowen/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/srowen/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/srowen/spark/events\",\"assignees_url\":\"https://api.github.com/repos/srowen/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/srowen/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/srowen/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/srowen/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/srowen/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/srowen/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/srowen/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/srowen/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/srowen/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/srowen/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/srowen/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/srowen/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/srowen/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/srowen/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/srowen/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/srowen/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/srowen/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/srowen/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/srowen/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/srowen/spark/merges\",\"archive_url\":\"https://api.github.com/repos/srowen/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/srowen/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/srowen/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/srowen/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/srowen/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/srowen/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/srowen/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/srowen/spark/releases{/id}\",\"created_at\":\"2014-02-27T07:57:07Z\",\"updated_at\":\"2014-12-31T22:16:53Z\",\"pushed_at\":\"2014-12-31T23:34:36Z\",\"git_url\":\"git://github.com/srowen/spark.git\",\"ssh_url\":\"git@github.com:srowen/spark.git\",\"clone_url\":\"https://github.com/srowen/spark.git\",\"svn_url\":\"https://github.com/srowen/spark\",\"homepage\":null,\"size\":84942,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":1,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":1,\"open_issues\":0,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"apache:master\",\"ref\":\"master\",\"sha\":\"040d6f2d13b132b3ef2a1e4f12f9f0e781c5a0b8\",\"user\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":17165658,\"name\":\"spark\",\"full_name\":\"apache/spark\",\"owner\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/apache/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":false,\"url\":\"https://api.github.com/repos/apache/spark\",\"forks_url\":\"https://api.github.com/repos/apache/spark/forks\",\"keys_url\":\"https://api.github.com/repos/apache/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/apache/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/apache/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/apache/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/apache/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/apache/spark/events\",\"assignees_url\":\"https://api.github.com/repos/apache/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/apache/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/apache/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/apache/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/apache/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/apache/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/apache/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/apache/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/apache/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/apache/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/apache/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/apache/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/apache/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/apache/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/apache/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/apache/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/apache/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/apache/spark/merges\",\"archive_url\":\"https://api.github.com/repos/apache/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/apache/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/apache/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/apache/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/apache/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/apache/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/apache/spark/releases{/id}\",\"created_at\":\"2014-02-25T08:00:08Z\",\"updated_at\":\"2015-01-01T00:59:33Z\",\"pushed_at\":\"2015-01-01T00:59:33Z\",\"git_url\":\"git://github.com/apache/spark.git\",\"ssh_url\":\"git@github.com:apache/spark.git\",\"clone_url\":\"https://github.com/apache/spark.git\",\"svn_url\":\"https://github.com/apache/spark\",\"homepage\":null,\"size\":1083068,\"stargazers_count\":2458,\"watchers_count\":2458,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":2179,\"mirror_url\":\"git://git.apache.org/spark.git\",\"open_issues_count\":268,\"forks\":2179,\"open_issues\":268,\"watchers\":2458,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3842\"},\"html\":{\"href\":\"https://github.com/apache/spark/pull/3842\"},\"issue\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3842\"},\"comments\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3842/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3842/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3842/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/apache/spark/statuses/50ff80e4498c2cb0a30793fb41fa2d20942811d6\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":13,\"review_comments\":0,\"commits\":2,\"additions\":6,\"deletions\":1,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:00:02Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489395771\",\"type\":\"PushEvent\",\"actor\":{\"id\":1341245,\"login\":\"asfgit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/asfgit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1341245?\"},\"repo\":{\"id\":17165658,\"name\":\"apache/spark\",\"url\":\"https://api.github.com/repos/apache/spark\"},\"payload\":{\"push_id\":536752124,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4bb12488d56ea651c56d9688996b464b99095582\",\"before\":\"fe6efacc0b865e9e827a1565877077000e63976e\",\"commits\":[{\"sha\":\"4bb12488d56ea651c56d9688996b464b99095582\",\"author\":{\"email\":\"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com\",\"name\":\"Sean Owen\"},\"message\":\"SPARK-2757 [BUILD] [STREAMING] Add Mima test for Spark Sink after 1.10 is released\\n\\nRe-enable MiMa for Streaming Flume Sink module, now that 1.1.0 is released, per the JIRA TO-DO. That's pretty much all there is to this.\\n\\nAuthor: Sean Owen <sowen@cloudera.com>\\n\\nCloses #3842 from srowen/SPARK-2757 and squashes the following commits:\\n\\n50ff80e [Sean Owen] Exclude apparent false positive turned up by re-enabling MiMa checks for Streaming Flume Sink\\n0e5ba5c [Sean Owen] Re-enable MiMa for Streaming Flume Sink module\",\"distinct\":true,\"url\":\"https://api.github.com/repos/apache/spark/commits/4bb12488d56ea651c56d9688996b464b99095582\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:02Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489395775\",\"type\":\"WatchEvent\",\"actor\":{\"id\":8562461,\"login\":\"jamezb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamezb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8562461?\"},\"repo\":{\"id\":3073296,\"name\":\"SirVer/ultisnips\",\"url\":\"https://api.github.com/repos/SirVer/ultisnips\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:03Z\"}\n{\"id\":\"2489395777\",\"type\":\"PushEvent\",\"actor\":{\"id\":3471313,\"login\":\"uygunuks\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/uygunuks\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3471313?\"},\"repo\":{\"id\":28677766,\"name\":\"uygunuks/AsalSayiKalibi\",\"url\":\"https://api.github.com/repos/uygunuks/AsalSayiKalibi\"},\"payload\":{\"push_id\":536752126,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/uygunuks\",\"head\":\"0fb2391e7aabeb57f358d5d51c70c766e6fa00d1\",\"before\":\"e47f1e095b582a95143e414b3d449b1105345994\",\"commits\":[{\"sha\":\"0fb2391e7aabeb57f358d5d51c70c766e6fa00d1\",\"author\":{\"email\":\"821f468726cd384db724fde38ddabae6642cf80c@gmail.com\",\"name\":\"Uygun BODUR\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/uygunuks/AsalSayiKalibi/commits/0fb2391e7aabeb57f358d5d51c70c766e6fa00d1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:03Z\"}\n{\"id\":\"2489395778\",\"type\":\"PushEvent\",\"actor\":{\"id\":9201970,\"login\":\"qdm\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qdm\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9201970?\"},\"repo\":{\"id\":25173910,\"name\":\"qdm/qdm.github.io\",\"url\":\"https://api.github.com/repos/qdm/qdm.github.io\"},\"payload\":{\"push_id\":536752127,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"05438aee0e5572a6e4adf91d7eef32917812d9e9\",\"before\":\"c68bef59c42e21a898e59516233d5968deab61ca\",\"commits\":[{\"sha\":\"05438aee0e5572a6e4adf91d7eef32917812d9e9\",\"author\":{\"email\":\"de163e90d3aeef9f404d1de71c48e234a211e3c3@gmail.com\",\"name\":\"KT\"},\"message\":\"Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qdm/qdm.github.io/commits/05438aee0e5572a6e4adf91d7eef32917812d9e9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:03Z\"}\n{\"id\":\"2489395780\",\"type\":\"PushEvent\",\"actor\":{\"id\":3495129,\"login\":\"sundaymtn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sundaymtn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3495129?\"},\"repo\":{\"id\":24147122,\"name\":\"sundaymtn/waterline\",\"url\":\"https://api.github.com/repos/sundaymtn/waterline\"},\"payload\":{\"push_id\":536752128,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"44753191dc8f615ccda4f0afe31a09342172cfe4\",\"before\":\"edee9761c08f76560fba328ea671c40f132f1afe\",\"commits\":[{\"sha\":\"44753191dc8f615ccda4f0afe31a09342172cfe4\",\"author\":{\"email\":\"7fbc091194a9488bfb16868527a7c3a8ba469dba@gmail.com\",\"name\":\"Seth Carter\"},\"message\":\"Wed Dec 31 20:00:02 EST 2014\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sundaymtn/waterline/commits/44753191dc8f615ccda4f0afe31a09342172cfe4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:04Z\"}\n{\"id\":\"2489395781\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":383994,\"login\":\"jonschlinkert\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jonschlinkert\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/383994?\"},\"repo\":{\"id\":27114957,\"name\":\"jonschlinkert/alphabet\",\"url\":\"https://api.github.com/repos/jonschlinkert/alphabet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1\",\"labels_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/events\",\"html_url\":\"https://github.com/jonschlinkert/alphabet/issues/1\",\"id\":53200509,\"number\":1,\"title\":\"any reason this isn't 1.0?\",\"user\":{\"login\":\"tkellen\",\"id\":1004324,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1004324?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tkellen\",\"html_url\":\"https://github.com/tkellen\",\"followers_url\":\"https://api.github.com/users/tkellen/followers\",\"following_url\":\"https://api.github.com/users/tkellen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tkellen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tkellen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tkellen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tkellen/orgs\",\"repos_url\":\"https://api.github.com/users/tkellen/repos\",\"events_url\":\"https://api.github.com/users/tkellen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tkellen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T19:50:51Z\",\"updated_at\":\"2015-01-01T01:00:04Z\",\"closed_at\":\"2015-01-01T01:00:04Z\",\"body\":\":P\"},\"comment\":{\"url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/comments/68477208\",\"html_url\":\"https://github.com/jonschlinkert/alphabet/issues/1#issuecomment-68477208\",\"issue_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1\",\"id\":68477208,\"user\":{\"login\":\"jonschlinkert\",\"id\":383994,\"avatar_url\":\"https://avatars.githubusercontent.com/u/383994?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jonschlinkert\",\"html_url\":\"https://github.com/jonschlinkert\",\"followers_url\":\"https://api.github.com/users/jonschlinkert/followers\",\"following_url\":\"https://api.github.com/users/jonschlinkert/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jonschlinkert/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jonschlinkert/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jonschlinkert/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jonschlinkert/orgs\",\"repos_url\":\"https://api.github.com/users/jonschlinkert/repos\",\"events_url\":\"https://api.github.com/users/jonschlinkert/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jonschlinkert/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:04Z\",\"updated_at\":\"2015-01-01T01:00:04Z\",\"body\":\"done. the English alphabet is officially stable.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:04Z\"}\n{\"id\":\"2489395782\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":383994,\"login\":\"jonschlinkert\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jonschlinkert\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/383994?\"},\"repo\":{\"id\":27114957,\"name\":\"jonschlinkert/alphabet\",\"url\":\"https://api.github.com/repos/jonschlinkert/alphabet\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1\",\"labels_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/jonschlinkert/alphabet/issues/1/events\",\"html_url\":\"https://github.com/jonschlinkert/alphabet/issues/1\",\"id\":53200509,\"number\":1,\"title\":\"any reason this isn't 1.0?\",\"user\":{\"login\":\"tkellen\",\"id\":1004324,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1004324?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tkellen\",\"html_url\":\"https://github.com/tkellen\",\"followers_url\":\"https://api.github.com/users/tkellen/followers\",\"following_url\":\"https://api.github.com/users/tkellen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tkellen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tkellen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tkellen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tkellen/orgs\",\"repos_url\":\"https://api.github.com/users/tkellen/repos\",\"events_url\":\"https://api.github.com/users/tkellen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tkellen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T19:50:51Z\",\"updated_at\":\"2015-01-01T01:00:04Z\",\"closed_at\":\"2015-01-01T01:00:04Z\",\"body\":\":P\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:04Z\"}\n{\"id\":\"2489395784\",\"type\":\"PushEvent\",\"actor\":{\"id\":954353,\"login\":\"byronmccollum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/byronmccollum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/954353?\"},\"repo\":{\"id\":28677943,\"name\":\"byronmccollum/html-lua\",\"url\":\"https://api.github.com/repos/byronmccollum/html-lua\"},\"payload\":{\"push_id\":536752130,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4fc83167bc7fd4520a8c01e2a0cca1ae7b5f191a\",\"before\":\"f2b6358fbc7c23443f40a51dea7a21aed96c3df3\",\"commits\":[{\"sha\":\"4fc83167bc7fd4520a8c01e2a0cca1ae7b5f191a\",\"author\":{\"email\":\"5c33cd3f0e8876e7150963b90c8e5c3e219c1462@rackspace.com\",\"name\":\"Byron McCollum\"},\"message\":\"Update and rename htmlparser.lua to diff.lua\",\"distinct\":true,\"url\":\"https://api.github.com/repos/byronmccollum/html-lua/commits/4fc83167bc7fd4520a8c01e2a0cca1ae7b5f191a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:04Z\"}\n{\"id\":\"2489395786\",\"type\":\"PushEvent\",\"actor\":{\"id\":4070158,\"login\":\"caleb-eades\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/caleb-eades\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4070158?\"},\"repo\":{\"id\":20469468,\"name\":\"caleb-eades/MinecraftServers\",\"url\":\"https://api.github.com/repos/caleb-eades/MinecraftServers\"},\"payload\":{\"push_id\":536752131,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"60172d1d82e6e7a353b92353ee214a4960d39023\",\"before\":\"049abf23362ba353c697f1c88d54250ebd073ce1\",\"commits\":[{\"sha\":\"60172d1d82e6e7a353b92353ee214a4960d39023\",\"author\":{\"email\":\"5bbfe2c07a3ef0b22b72711a2edf1c023f6433c5@gmail.com\",\"name\":\"caleb-eades\"},\"message\":\"Auto Snapshot Server State\",\"distinct\":true,\"url\":\"https://api.github.com/repos/caleb-eades/MinecraftServers/commits/60172d1d82e6e7a353b92353ee214a4960d39023\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:04Z\"}\n{\"id\":\"2489395789\",\"type\":\"PushEvent\",\"actor\":{\"id\":6372134,\"login\":\"Stuntddude\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Stuntddude\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6372134?\"},\"repo\":{\"id\":28654554,\"name\":\"Stuntddude/Blockade-Runner\",\"url\":\"https://api.github.com/repos/Stuntddude/Blockade-Runner\"},\"payload\":{\"push_id\":536752132,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b9ec439b0ecc1f79b1b08fa5c0a2abe1f14856c9\",\"before\":\"8289316476b4f01149046d43c34bb6aec0318b17\",\"commits\":[{\"sha\":\"b9ec439b0ecc1f79b1b08fa5c0a2abe1f14856c9\",\"author\":{\"email\":\"648fb0198c1960b14fbfb0124dd6b81f9019bc45@gmail.com\",\"name\":\"Miles Fogle\"},\"message\":\"Implemented InputHandler\\n\\nCreated the first draft of a generic input handler in ld31.InputHandler,\\nwhich is now used to handle keyboard input, and will be used to handle\\nmouse input and provide options for keybindings. Moved most\\ninput-handling code out of LD31.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Stuntddude/Blockade-Runner/commits/b9ec439b0ecc1f79b1b08fa5c0a2abe1f14856c9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:06Z\"}\n{\"id\":\"2489395790\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":4195632,\"login\":\"KA101\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KA101\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4195632?\"},\"repo\":{\"id\":5973855,\"name\":\"CleverRaven/Cataclysm-DDA\",\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/comments/22397251\",\"id\":22397251,\"diff_hunk\":\"@@ -469,14 +469,23 @@ std::string effect::disp_desc(bool reduced) const\\n     std::vector<desc_freq> values;\\n     // Add various desc_freq structs to values. If more effects wish to be placed in the descriptions this is the\\n     // place to add them.\\n-    values.push_back(desc_freq(get_percentage(\\\"PAIN\\\", reduced), get_avg_mod(\\\"PAIN\\\", reduced), _(\\\"pain\\\"), _(\\\"pain\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"HURT\\\", reduced), get_avg_mod(\\\"HURT\\\", reduced), _(\\\"damage\\\"), _(\\\"damage\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"THIRST\\\", reduced), get_avg_mod(\\\"THIRST\\\", reduced), _(\\\"thirst\\\"), _(\\\"quench\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"HUNGER\\\", reduced), get_avg_mod(\\\"HUNGER\\\", reduced), _(\\\"hunger\\\"), _(\\\"sate\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"FATIGUE\\\", reduced), get_avg_mod(\\\"FATIGUE\\\", reduced), _(\\\"fatigue\\\"), _(\\\"rest\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"COUGH\\\", reduced), get_avg_mod(\\\"COUGH\\\", reduced), _(\\\"coughing\\\"), _(\\\"coughing\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"VOMIT\\\", reduced), get_avg_mod(\\\"VOMIT\\\", reduced), _(\\\"vomiting\\\"), _(\\\"vomiting\\\")));\\n-    values.push_back(desc_freq(get_percentage(\\\"SLEEP\\\", reduced), get_avg_mod(\\\"SLEEP\\\", reduced), _(\\\"blackouts\\\"), _(\\\"blackouts\\\")));\\n+    int val = 0;\\n+    val = get_avg_mod(\\\"PAIN\\\", reduced);\\n+    values.push_back(desc_freq(get_percentage(\\\"PAIN\\\", val, reduced), val, _(\\\"pain\\\"), _(\\\"pain\\\")));\",\"path\":\"src/effect.cpp\",\"position\":14,\"original_position\":14,\"commit_id\":\"63238b1c8a9b7551d891f9d2595f53f43d91e776\",\"original_commit_id\":\"63238b1c8a9b7551d891f9d2595f53f43d91e776\",\"user\":{\"login\":\"KA101\",\"id\":4195632,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4195632?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KA101\",\"html_url\":\"https://github.com/KA101\",\"followers_url\":\"https://api.github.com/users/KA101/followers\",\"following_url\":\"https://api.github.com/users/KA101/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KA101/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KA101/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KA101/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KA101/orgs\",\"repos_url\":\"https://api.github.com/users/KA101/repos\",\"events_url\":\"https://api.github.com/users/KA101/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KA101/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Huh.  This looks like it might be workable for the mutation fix.  I wasn't sure how best to handle that.\",\"created_at\":\"2015-01-01T01:00:05Z\",\"updated_at\":\"2015-01-01T01:00:05Z\",\"html_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698#discussion_r22397251\",\"pull_request_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/comments/22397251\"},\"html\":{\"href\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698#discussion_r22397251\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698\",\"id\":26739248,\"html_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698\",\"diff_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698.diff\",\"patch_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698.patch\",\"issue_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10698\",\"number\":10698,\"state\":\"open\",\"locked\":false,\"title\":\"Fix effect triggering\",\"user\":{\"login\":\"i2amroy\",\"id\":4275617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4275617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/i2amroy\",\"html_url\":\"https://github.com/i2amroy\",\"followers_url\":\"https://api.github.com/users/i2amroy/followers\",\"following_url\":\"https://api.github.com/users/i2amroy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/i2amroy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/i2amroy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/i2amroy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/i2amroy/orgs\",\"repos_url\":\"https://api.github.com/users/i2amroy/repos\",\"events_url\":\"https://api.github.com/users/i2amroy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/i2amroy/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Fixes some <= 0 confusion stuff and makes having a nonzero addition value properly override having a default (0) chance to trigger.\\r\\n\\r\\nFixes #10617\",\"created_at\":\"2015-01-01T00:42:50Z\",\"updated_at\":\"2015-01-01T01:00:05Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"88ebf4a7e6d454d6b8bd6757a0bab31855a9ea1d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698/commits\",\"review_comments_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698/comments\",\"review_comment_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10698/comments\",\"statuses_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/statuses/63238b1c8a9b7551d891f9d2595f53f43d91e776\",\"head\":{\"label\":\"i2amroy:PKILL\",\"ref\":\"PKILL\",\"sha\":\"63238b1c8a9b7551d891f9d2595f53f43d91e776\",\"user\":{\"login\":\"i2amroy\",\"id\":4275617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4275617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/i2amroy\",\"html_url\":\"https://github.com/i2amroy\",\"followers_url\":\"https://api.github.com/users/i2amroy/followers\",\"following_url\":\"https://api.github.com/users/i2amroy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/i2amroy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/i2amroy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/i2amroy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/i2amroy/orgs\",\"repos_url\":\"https://api.github.com/users/i2amroy/repos\",\"events_url\":\"https://api.github.com/users/i2amroy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/i2amroy/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":9722704,\"name\":\"Cataclysm-DDA\",\"full_name\":\"i2amroy/Cataclysm-DDA\",\"owner\":{\"login\":\"i2amroy\",\"id\":4275617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4275617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/i2amroy\",\"html_url\":\"https://github.com/i2amroy\",\"followers_url\":\"https://api.github.com/users/i2amroy/followers\",\"following_url\":\"https://api.github.com/users/i2amroy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/i2amroy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/i2amroy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/i2amroy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/i2amroy/orgs\",\"repos_url\":\"https://api.github.com/users/i2amroy/repos\",\"events_url\":\"https://api.github.com/users/i2amroy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/i2amroy/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/i2amroy/Cataclysm-DDA\",\"description\":\"Cataclysm - Dark Days Ahead. A fork/variant of Cataclysm Roguelike by Whales.\",\"fork\":true,\"url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA\",\"forks_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/forks\",\"keys_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/teams\",\"hooks_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/hooks\",\"issue_events_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/events\",\"assignees_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/tags\",\"blobs_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/languages\",\"stargazers_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/stargazers\",\"contributors_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/contributors\",\"subscribers_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/subscribers\",\"subscription_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/subscription\",\"commits_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/merges\",\"archive_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/downloads\",\"issues_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/i2amroy/Cataclysm-DDA/releases{/id}\",\"created_at\":\"2013-04-27T23:43:26Z\",\"updated_at\":\"2014-12-31T21:47:23Z\",\"pushed_at\":\"2015-01-01T00:39:41Z\",\"git_url\":\"git://github.com/i2amroy/Cataclysm-DDA.git\",\"ssh_url\":\"git@github.com:i2amroy/Cataclysm-DDA.git\",\"clone_url\":\"https://github.com/i2amroy/Cataclysm-DDA.git\",\"svn_url\":\"https://github.com/i2amroy/Cataclysm-DDA\",\"homepage\":\"http://www.cataclysm.glyphgryph.com/\",\"size\":324783,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"C++\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"CleverRaven:master\",\"ref\":\"master\",\"sha\":\"ddef397944dacbc1a57f0fb4900f3f3cf9a2045a\",\"user\":{\"login\":\"CleverRaven\",\"id\":4367009,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4367009?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CleverRaven\",\"html_url\":\"https://github.com/CleverRaven\",\"followers_url\":\"https://api.github.com/users/CleverRaven/followers\",\"following_url\":\"https://api.github.com/users/CleverRaven/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CleverRaven/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CleverRaven/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CleverRaven/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CleverRaven/orgs\",\"repos_url\":\"https://api.github.com/users/CleverRaven/repos\",\"events_url\":\"https://api.github.com/users/CleverRaven/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CleverRaven/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":5973855,\"name\":\"Cataclysm-DDA\",\"full_name\":\"CleverRaven/Cataclysm-DDA\",\"owner\":{\"login\":\"CleverRaven\",\"id\":4367009,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4367009?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CleverRaven\",\"html_url\":\"https://github.com/CleverRaven\",\"followers_url\":\"https://api.github.com/users/CleverRaven/followers\",\"following_url\":\"https://api.github.com/users/CleverRaven/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CleverRaven/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CleverRaven/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CleverRaven/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CleverRaven/orgs\",\"repos_url\":\"https://api.github.com/users/CleverRaven/repos\",\"events_url\":\"https://api.github.com/users/CleverRaven/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CleverRaven/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/CleverRaven/Cataclysm-DDA\",\"description\":\"Cataclysm - Dark Days Ahead. A fork/variant of Cataclysm Roguelike by Whales.\",\"fork\":false,\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA\",\"forks_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/forks\",\"keys_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/teams\",\"hooks_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/hooks\",\"issue_events_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/events\",\"assignees_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/tags\",\"blobs_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/languages\",\"stargazers_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/stargazers\",\"contributors_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/contributors\",\"subscribers_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/subscribers\",\"subscription_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/subscription\",\"commits_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/merges\",\"archive_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/downloads\",\"issues_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/releases{/id}\",\"created_at\":\"2012-09-26T22:57:43Z\",\"updated_at\":\"2014-12-31T07:31:31Z\",\"pushed_at\":\"2014-12-31T07:31:30Z\",\"git_url\":\"git://github.com/CleverRaven/Cataclysm-DDA.git\",\"ssh_url\":\"git@github.com:CleverRaven/Cataclysm-DDA.git\",\"clone_url\":\"https://github.com/CleverRaven/Cataclysm-DDA.git\",\"svn_url\":\"https://github.com/CleverRaven/Cataclysm-DDA\",\"homepage\":\"http://en.cataclysmdda.com/\",\"size\":400119,\"stargazers_count\":556,\"watchers_count\":556,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":619,\"mirror_url\":null,\"open_issues_count\":774,\"forks\":619,\"open_issues\":774,\"watchers\":556,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698\"},\"html\":{\"href\":\"https://github.com/CleverRaven/Cataclysm-DDA/pull/10698\"},\"issue\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10698\"},\"comments\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10698/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/pulls/10698/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/statuses/63238b1c8a9b7551d891f9d2595f53f43d91e776\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:00:05Z\",\"org\":{\"id\":4367009,\"login\":\"CleverRaven\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/CleverRaven\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4367009?\"}}\n{\"id\":\"2489395794\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5762348,\"login\":\"JasonYang96\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JasonYang96\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5762348?\"},\"repo\":{\"id\":28678050,\"name\":\"JasonYang96/Leet-Code-OJ\",\"url\":\"https://api.github.com/repos/JasonYang96/Leet-Code-OJ\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Solutions to Problems found at https://oj.leetcode.com/problems/\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:06Z\"}\n{\"id\":\"2489395795\",\"type\":\"WatchEvent\",\"actor\":{\"id\":119853,\"login\":\"mcddx330\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mcddx330\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/119853?\"},\"repo\":{\"id\":8257106,\"name\":\"jessesquires/JSQMessagesViewController\",\"url\":\"https://api.github.com/repos/jessesquires/JSQMessagesViewController\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:06Z\"}\n{\"id\":\"2489395796\",\"type\":\"ForkEvent\",\"actor\":{\"id\":8518239,\"login\":\"gitter-badger\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?\"},\"repo\":{\"id\":8540783,\"name\":\"russmatney/dotfiles\",\"url\":\"https://api.github.com/repos/russmatney/dotfiles\"},\"payload\":{\"forkee\":{\"id\":28678212,\"name\":\"dotfiles-15\",\"full_name\":\"gitter-badger/dotfiles-15\",\"owner\":{\"login\":\"gitter-badger\",\"id\":8518239,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"html_url\":\"https://github.com/gitter-badger\",\"followers_url\":\"https://api.github.com/users/gitter-badger/followers\",\"following_url\":\"https://api.github.com/users/gitter-badger/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gitter-badger/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gitter-badger/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gitter-badger/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gitter-badger/orgs\",\"repos_url\":\"https://api.github.com/users/gitter-badger/repos\",\"events_url\":\"https://api.github.com/users/gitter-badger/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gitter-badger/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/gitter-badger/dotfiles-15\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15\",\"forks_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/forks\",\"keys_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/teams\",\"hooks_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/hooks\",\"issue_events_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/events\",\"assignees_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/tags\",\"blobs_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/languages\",\"stargazers_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/stargazers\",\"contributors_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/contributors\",\"subscribers_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/subscribers\",\"subscription_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/subscription\",\"commits_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/merges\",\"archive_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/downloads\",\"issues_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/releases{/id}\",\"created_at\":\"2015-01-01T01:00:07Z\",\"updated_at\":\"2014-12-30T19:07:15Z\",\"pushed_at\":\"2014-12-30T19:07:15Z\",\"git_url\":\"git://github.com/gitter-badger/dotfiles-15.git\",\"ssh_url\":\"git@github.com:gitter-badger/dotfiles-15.git\",\"clone_url\":\"https://github.com/gitter-badger/dotfiles-15.git\",\"svn_url\":\"https://github.com/gitter-badger/dotfiles-15\",\"homepage\":\"\",\"size\":620,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\"}\n{\"id\":\"2489395797\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":1235097,\"login\":\"rmarinho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?\"},\"repo\":{\"id\":20463939,\"name\":\"XLabs/Xamarin-Forms-Labs\",\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\"},\"payload\":{\"action\":\"closed\",\"number\":533,\"pull_request\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533\",\"id\":26426443,\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/533\",\"diff_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/533.diff\",\"patch_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/533.patch\",\"issue_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/533\",\"number\":533,\"state\":\"closed\",\"locked\":false,\"title\":\"android & ios screenshots\",\"user\":{\"login\":\"jguertl\",\"id\":7988662,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7988662?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jguertl\",\"html_url\":\"https://github.com/jguertl\",\"followers_url\":\"https://api.github.com/users/jguertl/followers\",\"following_url\":\"https://api.github.com/users/jguertl/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jguertl/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jguertl/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jguertl/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jguertl/orgs\",\"repos_url\":\"https://api.github.com/users/jguertl/repos\",\"events_url\":\"https://api.github.com/users/jguertl/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jguertl/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2014-12-21T10:31:17Z\",\"updated_at\":\"2015-01-01T01:00:07Z\",\"closed_at\":\"2015-01-01T01:00:07Z\",\"merged_at\":\"2015-01-01T01:00:07Z\",\"merge_commit_sha\":\"f60ed691a53c54c0e9f7254f072dc92a46d2652d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533/commits\",\"review_comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533/comments\",\"review_comment_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/533/comments\",\"statuses_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/statuses/1fb29e8b83f96badb55e4dfecc19392f498df57a\",\"head\":{\"label\":\"jguertl:master\",\"ref\":\"master\",\"sha\":\"1fb29e8b83f96badb55e4dfecc19392f498df57a\",\"user\":{\"login\":\"jguertl\",\"id\":7988662,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7988662?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jguertl\",\"html_url\":\"https://github.com/jguertl\",\"followers_url\":\"https://api.github.com/users/jguertl/followers\",\"following_url\":\"https://api.github.com/users/jguertl/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jguertl/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jguertl/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jguertl/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jguertl/orgs\",\"repos_url\":\"https://api.github.com/users/jguertl/repos\",\"events_url\":\"https://api.github.com/users/jguertl/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jguertl/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28277803,\"name\":\"Xamarin-Forms-Labs\",\"full_name\":\"jguertl/Xamarin-Forms-Labs\",\"owner\":{\"login\":\"jguertl\",\"id\":7988662,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7988662?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jguertl\",\"html_url\":\"https://github.com/jguertl\",\"followers_url\":\"https://api.github.com/users/jguertl/followers\",\"following_url\":\"https://api.github.com/users/jguertl/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jguertl/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jguertl/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jguertl/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jguertl/orgs\",\"repos_url\":\"https://api.github.com/users/jguertl/repos\",\"events_url\":\"https://api.github.com/users/jguertl/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jguertl/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/jguertl/Xamarin-Forms-Labs\",\"description\":\"Xamarin Forms Labs is a open source project that aims to provide a powerful and cross platform set of controls and helpers tailored to work with Xamarin Forms.\",\"fork\":true,\"url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs\",\"forks_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/forks\",\"keys_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/teams\",\"hooks_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/hooks\",\"issue_events_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/events\",\"assignees_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/tags\",\"blobs_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/languages\",\"stargazers_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/stargazers\",\"contributors_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/contributors\",\"subscribers_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/subscribers\",\"subscription_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/subscription\",\"commits_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/merges\",\"archive_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/downloads\",\"issues_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/jguertl/Xamarin-Forms-Labs/releases{/id}\",\"created_at\":\"2014-12-20T20:17:25Z\",\"updated_at\":\"2014-12-21T10:30:08Z\",\"pushed_at\":\"2014-12-21T10:30:08Z\",\"git_url\":\"git://github.com/jguertl/Xamarin-Forms-Labs.git\",\"ssh_url\":\"git@github.com:jguertl/Xamarin-Forms-Labs.git\",\"clone_url\":\"https://github.com/jguertl/Xamarin-Forms-Labs.git\",\"svn_url\":\"https://github.com/jguertl/Xamarin-Forms-Labs\",\"homepage\":\"\",\"size\":96864,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"C#\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"XLabs:master\",\"ref\":\"master\",\"sha\":\"6906dd38ff69debcc304cb05b6877fae71747acd\",\"user\":{\"login\":\"XLabs\",\"id\":7787062,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/XLabs\",\"html_url\":\"https://github.com/XLabs\",\"followers_url\":\"https://api.github.com/users/XLabs/followers\",\"following_url\":\"https://api.github.com/users/XLabs/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/XLabs/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/XLabs/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/XLabs/subscriptions\",\"organizations_url\":\"https://api.github.com/users/XLabs/orgs\",\"repos_url\":\"https://api.github.com/users/XLabs/repos\",\"events_url\":\"https://api.github.com/users/XLabs/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/XLabs/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":20463939,\"name\":\"Xamarin-Forms-Labs\",\"full_name\":\"XLabs/Xamarin-Forms-Labs\",\"owner\":{\"login\":\"XLabs\",\"id\":7787062,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/XLabs\",\"html_url\":\"https://github.com/XLabs\",\"followers_url\":\"https://api.github.com/users/XLabs/followers\",\"following_url\":\"https://api.github.com/users/XLabs/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/XLabs/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/XLabs/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/XLabs/subscriptions\",\"organizations_url\":\"https://api.github.com/users/XLabs/orgs\",\"repos_url\":\"https://api.github.com/users/XLabs/repos\",\"events_url\":\"https://api.github.com/users/XLabs/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/XLabs/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs\",\"description\":\"Xamarin Forms Labs is a open source project that aims to provide a powerful and cross platform set of controls and helpers tailored to work with Xamarin Forms.\",\"fork\":false,\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\",\"forks_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/forks\",\"keys_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/teams\",\"hooks_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/hooks\",\"issue_events_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/events\",\"assignees_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/tags\",\"blobs_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/languages\",\"stargazers_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/stargazers\",\"contributors_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/contributors\",\"subscribers_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/subscribers\",\"subscription_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/subscription\",\"commits_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/merges\",\"archive_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/downloads\",\"issues_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/releases{/id}\",\"created_at\":\"2014-06-03T23:53:11Z\",\"updated_at\":\"2015-01-01T00:59:13Z\",\"pushed_at\":\"2015-01-01T01:00:07Z\",\"git_url\":\"git://github.com/XLabs/Xamarin-Forms-Labs.git\",\"ssh_url\":\"git@github.com:XLabs/Xamarin-Forms-Labs.git\",\"clone_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs.git\",\"svn_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs\",\"homepage\":\"\",\"size\":104805,\"stargazers_count\":340,\"watchers_count\":340,\"language\":\"C#\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":210,\"mirror_url\":null,\"open_issues_count\":91,\"forks\":210,\"open_issues\":91,\"watchers\":340,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533\"},\"html\":{\"href\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/533\"},\"issue\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/533\"},\"comments\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/533/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/533/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/statuses/1fb29e8b83f96badb55e4dfecc19392f498df57a\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"rmarinho\",\"id\":1235097,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"html_url\":\"https://github.com/rmarinho\",\"followers_url\":\"https://api.github.com/users/rmarinho/followers\",\"following_url\":\"https://api.github.com/users/rmarinho/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/rmarinho/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/rmarinho/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/rmarinho/subscriptions\",\"organizations_url\":\"https://api.github.com/users/rmarinho/orgs\",\"repos_url\":\"https://api.github.com/users/rmarinho/repos\",\"events_url\":\"https://api.github.com/users/rmarinho/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/rmarinho/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":1,\"review_comments\":0,\"commits\":1,\"additions\":0,\"deletions\":0,\"changed_files\":9}},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\",\"org\":{\"id\":7787062,\"login\":\"XLabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/XLabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?\"}}\n{\"id\":\"2489395801\",\"type\":\"PushEvent\",\"actor\":{\"id\":1235097,\"login\":\"rmarinho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?\"},\"repo\":{\"id\":20463939,\"name\":\"XLabs/Xamarin-Forms-Labs\",\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\"},\"payload\":{\"push_id\":536752137,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"5b5d95052d67e79b27788b0e498ec6a9e240046e\",\"before\":\"4db7138d78546f01b9782f8b1fcfe4fb27e861b6\",\"commits\":[{\"sha\":\"1fb29e8b83f96badb55e4dfecc19392f498df57a\",\"author\":{\"email\":\"9191838847179d50d9ec96fb4d4274d728e70c57@me.com\",\"name\":\"Jakob Gürtl\"},\"message\":\"android & ios screenshots\",\"distinct\":true,\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/commits/1fb29e8b83f96badb55e4dfecc19392f498df57a\"},{\"sha\":\"5b5d95052d67e79b27788b0e498ec6a9e240046e\",\"author\":{\"email\":\"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@ruimarinho.net\",\"name\":\"Rui Marinho\"},\"message\":\"Merge pull request #533 from jguertl/master\\n\\nandroid & ios screenshots\",\"distinct\":true,\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/commits/5b5d95052d67e79b27788b0e498ec6a9e240046e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\",\"org\":{\"id\":7787062,\"login\":\"XLabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/XLabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?\"}}\n{\"id\":\"2489395802\",\"type\":\"PushEvent\",\"actor\":{\"id\":926454,\"login\":\"lukeis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lukeis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/926454?\"},\"repo\":{\"id\":9457897,\"name\":\"SeleniumHQ/irc-logs\",\"url\":\"https://api.github.com/repos/SeleniumHQ/irc-logs\"},\"payload\":{\"push_id\":536752139,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d8aa1d441301788eeacffb3408616863637fecd8\",\"before\":\"50c406047545cdb0be48743b4de97620f1ce5f48\",\"commits\":[{\"sha\":\"d8aa1d441301788eeacffb3408616863637fecd8\",\"author\":{\"email\":\"c7f2353e77fbd59227c091422ca81210965ba01d\",\"name\":\"selloggingbot\"},\"message\":\"updating logs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/SeleniumHQ/irc-logs/commits/d8aa1d441301788eeacffb3408616863637fecd8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\",\"org\":{\"id\":983927,\"login\":\"SeleniumHQ\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/SeleniumHQ\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/983927?\"}}\n{\"id\":\"2489395805\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1390347,\"login\":\"Addvilz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Addvilz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1390347?\"},\"repo\":{\"id\":23615534,\"name\":\"DanielGorlo/ISIS.js\",\"url\":\"https://api.github.com/repos/DanielGorlo/ISIS.js\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\"}\n{\"id\":\"2489395808\",\"type\":\"PushEvent\",\"actor\":{\"id\":1844764,\"login\":\"ianblenke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ianblenke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1844764?\"},\"repo\":{\"id\":28671505,\"name\":\"ianblenke/docker-packetbeat-agent\",\"url\":\"https://api.github.com/repos/ianblenke/docker-packetbeat-agent\"},\"payload\":{\"push_id\":536752144,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"80ef4b415c08ccaaa009bfb07b807a3fc8864454\",\"before\":\"8d67307dd90527ef7771a0304d5469a5387271b1\",\"commits\":[{\"sha\":\"80ef4b415c08ccaaa009bfb07b807a3fc8864454\",\"author\":{\"email\":\"57a33a5496950fec8433e4dd83347673459dcdfc@blenke.com\",\"name\":\"Ian Blenke\"},\"message\":\"snaplen is an integer, not a string\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ianblenke/docker-packetbeat-agent/commits/80ef4b415c08ccaaa009bfb07b807a3fc8864454\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\"}\n{\"id\":\"2489395809\",\"type\":\"PushEvent\",\"actor\":{\"id\":1221156,\"login\":\"fyfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fyfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1221156?\"},\"repo\":{\"id\":28673837,\"name\":\"fyfe/git-test\",\"url\":\"https://api.github.com/repos/fyfe/git-test\"},\"payload\":{\"push_id\":536752143,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/develop\",\"head\":\"4a252ca6b0d37d0714710d57ba524ff0601806df\",\"before\":\"279e85c74e9a85941ce223bd8a8f3ac37fa891fc\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\"}\n{\"id\":\"2489395811\",\"type\":\"PushEvent\",\"actor\":{\"id\":7727148,\"login\":\"ThadHouse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ThadHouse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7727148?\"},\"repo\":{\"id\":26708360,\"name\":\"ThadHouse/SplineGenerator\",\"url\":\"https://api.github.com/repos/ThadHouse/SplineGenerator\"},\"payload\":{\"push_id\":536752145,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3a313300b8a8694a20be00b74b25763b015f3549\",\"before\":\"a2b94fbe206b40060f076c185ef431200083aaf4\",\"commits\":[{\"sha\":\"3a313300b8a8694a20be00b74b25763b015f3549\",\"author\":{\"email\":\"ba841f95fa6d615b6e0ccbf97085c32d1f18fdef@users.noreply.github.com\",\"name\":\"ThadHouse\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ThadHouse/SplineGenerator/commits/3a313300b8a8694a20be00b74b25763b015f3549\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:08Z\"}\n{\"id\":\"2489395812\",\"type\":\"PushEvent\",\"actor\":{\"id\":6158630,\"login\":\"greatfire\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/greatfire\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6158630?\"},\"repo\":{\"id\":18126008,\"name\":\"greatfire/z\",\"url\":\"https://api.github.com/repos/greatfire/z\"},\"payload\":{\"push_id\":536752146,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1c095ccf0c45826798d494d477eef16082e2919f\",\"before\":\"0fae830fbc8591e9b4e3faf0794a1b73c460f9e6\",\"commits\":[{\"sha\":\"1c095ccf0c45826798d494d477eef16082e2919f\",\"author\":{\"email\":\"24bf68e341ce0fbd9259a5d51feed79682ea4eba@greatfire.org\",\"name\":\"Ubuntu\"},\"message\":\"a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/greatfire/z/commits/1c095ccf0c45826798d494d477eef16082e2919f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:09Z\"}\n{\"id\":\"2489395813\",\"type\":\"CreateEvent\",\"actor\":{\"id\":8518239,\"login\":\"gitter-badger\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?\"},\"repo\":{\"id\":28678212,\"name\":\"gitter-badger/dotfiles-15\",\"url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15\"},\"payload\":{\"ref\":\"gitter-badge\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:09Z\"}\n{\"id\":\"2489395822\",\"type\":\"PushEvent\",\"actor\":{\"id\":8518239,\"login\":\"gitter-badger\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?\"},\"repo\":{\"id\":28678212,\"name\":\"gitter-badger/dotfiles-15\",\"url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15\"},\"payload\":{\"push_id\":536752152,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gitter-badge\",\"head\":\"e4f636ddb08f1811808e3f872a7a323aa80c8a7b\",\"before\":\"7ca425b3dc133cd625f3f2e2e014ebee508eb39d\",\"commits\":[{\"sha\":\"e4f636ddb08f1811808e3f872a7a323aa80c8a7b\",\"author\":{\"email\":\"4e199b4a1c40b497a95fcd1cd896351733849949@gitter.im\",\"name\":\"The Gitter Badger\"},\"message\":\"Added Gitter badge\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/commits/e4f636ddb08f1811808e3f872a7a323aa80c8a7b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395823\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2068437,\"login\":\"ahaurw01\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ahaurw01\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2068437?\"},\"repo\":{\"id\":18271693,\"name\":\"ahaurw01/gulp-remember\",\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13\",\"labels_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13/comments\",\"events_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13/events\",\"html_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13\",\"id\":53012936,\"number\":13,\"title\":\"Handle file's history\",\"user\":{\"login\":\"efolio\",\"id\":2078815,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2078815?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/efolio\",\"html_url\":\"https://github.com/efolio\",\"followers_url\":\"https://api.github.com/users/efolio/followers\",\"following_url\":\"https://api.github.com/users/efolio/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/efolio/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/efolio/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/efolio/subscriptions\",\"organizations_url\":\"https://api.github.com/users/efolio/orgs\",\"repos_url\":\"https://api.github.com/users/efolio/repos\",\"events_url\":\"https://api.github.com/users/efolio/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/efolio/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":11,\"created_at\":\"2014-12-28T23:47:50Z\",\"updated_at\":\"2015-01-01T01:00:09Z\",\"closed_at\":\"2015-01-01T01:00:09Z\",\"pull_request\":{\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13\",\"html_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13\",\"diff_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13.diff\",\"patch_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13.patch\"},\"body\":\"Mostly add a `forgetUsingHistory` method in order to tackle the need to forget files using the original filename instead of the processed filename.\\r\\n\\r\\nI'll be glad to tweak this PR anyway you want.\\r\\n\\r\\nThanks a lot,\"},\"comment\":{\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/comments/68477210\",\"html_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13#issuecomment-68477210\",\"issue_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13\",\"id\":68477210,\"user\":{\"login\":\"ahaurw01\",\"id\":2068437,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2068437?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ahaurw01\",\"html_url\":\"https://github.com/ahaurw01\",\"followers_url\":\"https://api.github.com/users/ahaurw01/followers\",\"following_url\":\"https://api.github.com/users/ahaurw01/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ahaurw01/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ahaurw01/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ahaurw01/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ahaurw01/orgs\",\"repos_url\":\"https://api.github.com/users/ahaurw01/repos\",\"events_url\":\"https://api.github.com/users/ahaurw01/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ahaurw01/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:09Z\",\"updated_at\":\"2015-01-01T01:00:09Z\",\"body\":\"@efolio @qraynaud \\r\\nI'm not comfortable adding this feature as it nearly doubles the complexity of this plugin while solving (to me) a seemingly orthogonal problem. And opening the door to tweaking and expanding this feature in the future to better solve use cases that are slightly different from yours is not attractive to me either. \\r\\n\\r\\nI believe that the beauty of gulp plugins is that they are simple, unopinionated, imperative, and get incorporated into the workflow of your own code. Grunt, for example, encourages API creep within its plugins because they are used declaratively; they are simply configured. Gulp allows you to take micro-helpers and mix and match them in ways that conform to really specific workflow needs. If a gulp plugin begins to make assumptions about workflow, it has the opportunity for a quick expansion of complexity and/or a need for the plugin to be configured in more ways. \\r\\n\\r\\nI am of the opinion that more stupid-simple modules is better than fewer slightly more complex modules, especially in the gulp environment.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395824\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":2068437,\"login\":\"ahaurw01\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ahaurw01\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2068437?\"},\"repo\":{\"id\":18271693,\"name\":\"ahaurw01/gulp-remember\",\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember\"},\"payload\":{\"action\":\"closed\",\"number\":13,\"pull_request\":{\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13\",\"id\":26629069,\"html_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13\",\"diff_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13.diff\",\"patch_url\":\"https://github.com/ahaurw01/gulp-remember/pull/13.patch\",\"issue_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13\",\"number\":13,\"state\":\"closed\",\"locked\":false,\"title\":\"Handle file's history\",\"user\":{\"login\":\"efolio\",\"id\":2078815,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2078815?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/efolio\",\"html_url\":\"https://github.com/efolio\",\"followers_url\":\"https://api.github.com/users/efolio/followers\",\"following_url\":\"https://api.github.com/users/efolio/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/efolio/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/efolio/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/efolio/subscriptions\",\"organizations_url\":\"https://api.github.com/users/efolio/orgs\",\"repos_url\":\"https://api.github.com/users/efolio/repos\",\"events_url\":\"https://api.github.com/users/efolio/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/efolio/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Mostly add a `forgetUsingHistory` method in order to tackle the need to forget files using the original filename instead of the processed filename.\\r\\n\\r\\nI'll be glad to tweak this PR anyway you want.\\r\\n\\r\\nThanks a lot,\",\"created_at\":\"2014-12-28T23:47:50Z\",\"updated_at\":\"2015-01-01T01:00:09Z\",\"closed_at\":\"2015-01-01T01:00:09Z\",\"merged_at\":null,\"merge_commit_sha\":\"77da51e98bbc8a636fabf2ec70de46338199e570\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13/commits\",\"review_comments_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13/comments\",\"review_comment_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13/comments\",\"statuses_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/statuses/b54cbec37e97f1b472eabdced02d07847a536f9e\",\"head\":{\"label\":\"efolio:master\",\"ref\":\"master\",\"sha\":\"b54cbec37e97f1b472eabdced02d07847a536f9e\",\"user\":{\"login\":\"efolio\",\"id\":2078815,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2078815?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/efolio\",\"html_url\":\"https://github.com/efolio\",\"followers_url\":\"https://api.github.com/users/efolio/followers\",\"following_url\":\"https://api.github.com/users/efolio/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/efolio/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/efolio/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/efolio/subscriptions\",\"organizations_url\":\"https://api.github.com/users/efolio/orgs\",\"repos_url\":\"https://api.github.com/users/efolio/repos\",\"events_url\":\"https://api.github.com/users/efolio/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/efolio/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28569993,\"name\":\"gulp-remember\",\"full_name\":\"efolio/gulp-remember\",\"owner\":{\"login\":\"efolio\",\"id\":2078815,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2078815?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/efolio\",\"html_url\":\"https://github.com/efolio\",\"followers_url\":\"https://api.github.com/users/efolio/followers\",\"following_url\":\"https://api.github.com/users/efolio/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/efolio/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/efolio/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/efolio/subscriptions\",\"organizations_url\":\"https://api.github.com/users/efolio/orgs\",\"repos_url\":\"https://api.github.com/users/efolio/repos\",\"events_url\":\"https://api.github.com/users/efolio/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/efolio/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/efolio/gulp-remember\",\"description\":\"A plugin for gulp that remembers and recalls files passed through it\",\"fork\":true,\"url\":\"https://api.github.com/repos/efolio/gulp-remember\",\"forks_url\":\"https://api.github.com/repos/efolio/gulp-remember/forks\",\"keys_url\":\"https://api.github.com/repos/efolio/gulp-remember/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/efolio/gulp-remember/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/efolio/gulp-remember/teams\",\"hooks_url\":\"https://api.github.com/repos/efolio/gulp-remember/hooks\",\"issue_events_url\":\"https://api.github.com/repos/efolio/gulp-remember/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/efolio/gulp-remember/events\",\"assignees_url\":\"https://api.github.com/repos/efolio/gulp-remember/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/efolio/gulp-remember/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/efolio/gulp-remember/tags\",\"blobs_url\":\"https://api.github.com/repos/efolio/gulp-remember/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/efolio/gulp-remember/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/efolio/gulp-remember/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/efolio/gulp-remember/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/efolio/gulp-remember/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/efolio/gulp-remember/languages\",\"stargazers_url\":\"https://api.github.com/repos/efolio/gulp-remember/stargazers\",\"contributors_url\":\"https://api.github.com/repos/efolio/gulp-remember/contributors\",\"subscribers_url\":\"https://api.github.com/repos/efolio/gulp-remember/subscribers\",\"subscription_url\":\"https://api.github.com/repos/efolio/gulp-remember/subscription\",\"commits_url\":\"https://api.github.com/repos/efolio/gulp-remember/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/efolio/gulp-remember/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/efolio/gulp-remember/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/efolio/gulp-remember/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/efolio/gulp-remember/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/efolio/gulp-remember/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/efolio/gulp-remember/merges\",\"archive_url\":\"https://api.github.com/repos/efolio/gulp-remember/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/efolio/gulp-remember/downloads\",\"issues_url\":\"https://api.github.com/repos/efolio/gulp-remember/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/efolio/gulp-remember/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/efolio/gulp-remember/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/efolio/gulp-remember/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/efolio/gulp-remember/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/efolio/gulp-remember/releases{/id}\",\"created_at\":\"2014-12-28T19:41:58Z\",\"updated_at\":\"2014-12-28T23:45:40Z\",\"pushed_at\":\"2014-12-28T23:45:40Z\",\"git_url\":\"git://github.com/efolio/gulp-remember.git\",\"ssh_url\":\"git@github.com:efolio/gulp-remember.git\",\"clone_url\":\"https://github.com/efolio/gulp-remember.git\",\"svn_url\":\"https://github.com/efolio/gulp-remember\",\"homepage\":null,\"size\":275,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"JavaScript\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"ahaurw01:master\",\"ref\":\"master\",\"sha\":\"8bba77a2fb8f692192c3499f50a9a53036e5fb60\",\"user\":{\"login\":\"ahaurw01\",\"id\":2068437,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2068437?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ahaurw01\",\"html_url\":\"https://github.com/ahaurw01\",\"followers_url\":\"https://api.github.com/users/ahaurw01/followers\",\"following_url\":\"https://api.github.com/users/ahaurw01/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ahaurw01/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ahaurw01/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ahaurw01/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ahaurw01/orgs\",\"repos_url\":\"https://api.github.com/users/ahaurw01/repos\",\"events_url\":\"https://api.github.com/users/ahaurw01/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ahaurw01/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":18271693,\"name\":\"gulp-remember\",\"full_name\":\"ahaurw01/gulp-remember\",\"owner\":{\"login\":\"ahaurw01\",\"id\":2068437,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2068437?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ahaurw01\",\"html_url\":\"https://github.com/ahaurw01\",\"followers_url\":\"https://api.github.com/users/ahaurw01/followers\",\"following_url\":\"https://api.github.com/users/ahaurw01/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ahaurw01/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ahaurw01/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ahaurw01/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ahaurw01/orgs\",\"repos_url\":\"https://api.github.com/users/ahaurw01/repos\",\"events_url\":\"https://api.github.com/users/ahaurw01/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ahaurw01/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/ahaurw01/gulp-remember\",\"description\":\"A plugin for gulp that remembers and recalls files passed through it\",\"fork\":false,\"url\":\"https://api.github.com/repos/ahaurw01/gulp-remember\",\"forks_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/forks\",\"keys_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/teams\",\"hooks_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/hooks\",\"issue_events_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/events\",\"assignees_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/tags\",\"blobs_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/languages\",\"stargazers_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/stargazers\",\"contributors_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/contributors\",\"subscribers_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/subscribers\",\"subscription_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/subscription\",\"commits_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/merges\",\"archive_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/downloads\",\"issues_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/ahaurw01/gulp-remember/releases{/id}\",\"created_at\":\"2014-03-30T19:21:07Z\",\"updated_at\":\"2014-12-29T04:09:25Z\",\"pushed_at\":\"2014-12-21T15:42:48Z\",\"git_url\":\"git://github.com/ahaurw01/gulp-remember.git\",\"ssh_url\":\"git@github.com:ahaurw01/gulp-remember.git\",\"clone_url\":\"https://github.com/ahaurw01/gulp-remember.git\",\"svn_url\":\"https://github.com/ahaurw01/gulp-remember\",\"homepage\":null,\"size\":275,\"stargazers_count\":35,\"watchers_count\":35,\"language\":\"JavaScript\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":6,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":6,\"open_issues\":0,\"watchers\":35,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13\"},\"html\":{\"href\":\"https://github.com/ahaurw01/gulp-remember/pull/13\"},\"issue\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13\"},\"comments\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/issues/13/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/pulls/13/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/ahaurw01/gulp-remember/statuses/b54cbec37e97f1b472eabdced02d07847a536f9e\"}},\"merged\":false,\"mergeable\":true,\"mergeable_state\":\"clean\",\"merged_by\":null,\"comments\":11,\"review_comments\":0,\"commits\":1,\"additions\":195,\"deletions\":14,\"changed_files\":3}},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395826\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":8518239,\"login\":\"gitter-badger\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?\"},\"repo\":{\"id\":8540783,\"name\":\"russmatney/dotfiles\",\"url\":\"https://api.github.com/repos/russmatney/dotfiles\"},\"payload\":{\"action\":\"opened\",\"number\":1,\"pull_request\":{\"url\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1\",\"id\":26739403,\"html_url\":\"https://github.com/russmatney/dotfiles/pull/1\",\"diff_url\":\"https://github.com/russmatney/dotfiles/pull/1.diff\",\"patch_url\":\"https://github.com/russmatney/dotfiles/pull/1.patch\",\"issue_url\":\"https://api.github.com/repos/russmatney/dotfiles/issues/1\",\"number\":1,\"state\":\"open\",\"locked\":false,\"title\":\"Add a Gitter chat badge to README.md\",\"user\":{\"login\":\"gitter-badger\",\"id\":8518239,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"html_url\":\"https://github.com/gitter-badger\",\"followers_url\":\"https://api.github.com/users/gitter-badger/followers\",\"following_url\":\"https://api.github.com/users/gitter-badger/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gitter-badger/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gitter-badger/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gitter-badger/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gitter-badger/orgs\",\"repos_url\":\"https://api.github.com/users/gitter-badger/repos\",\"events_url\":\"https://api.github.com/users/gitter-badger/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gitter-badger/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"### russmatney/dotfiles now has a Chat Room on Gitter\\n\\n@russmatney has just created a chat room. You can visit it here: [https://gitter.im/russmatney/dotfiles](https://gitter.im/russmatney/dotfiles?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&content=body_link).\\n\\nThis pull-request adds this badge to your README.md:\\n\\n\\n[![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/russmatney/dotfiles?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=body_badge)\\n\\nHappy chatting.\\n\\n\\nPS: [Click here](https://gitter.im/settings/badger/opt-out) if you would prefer not to receive automatic pull-requests from Gitter in future.\\n\",\"created_at\":\"2015-01-01T01:00:09Z\",\"updated_at\":\"2015-01-01T01:00:09Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1/commits\",\"review_comments_url\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1/comments\",\"review_comment_url\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/russmatney/dotfiles/issues/1/comments\",\"statuses_url\":\"https://api.github.com/repos/russmatney/dotfiles/statuses/e4f636ddb08f1811808e3f872a7a323aa80c8a7b\",\"head\":{\"label\":\"gitter-badger:gitter-badge\",\"ref\":\"gitter-badge\",\"sha\":\"e4f636ddb08f1811808e3f872a7a323aa80c8a7b\",\"user\":{\"login\":\"gitter-badger\",\"id\":8518239,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"html_url\":\"https://github.com/gitter-badger\",\"followers_url\":\"https://api.github.com/users/gitter-badger/followers\",\"following_url\":\"https://api.github.com/users/gitter-badger/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gitter-badger/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gitter-badger/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gitter-badger/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gitter-badger/orgs\",\"repos_url\":\"https://api.github.com/users/gitter-badger/repos\",\"events_url\":\"https://api.github.com/users/gitter-badger/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gitter-badger/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28678212,\"name\":\"dotfiles-15\",\"full_name\":\"gitter-badger/dotfiles-15\",\"owner\":{\"login\":\"gitter-badger\",\"id\":8518239,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8518239?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gitter-badger\",\"html_url\":\"https://github.com/gitter-badger\",\"followers_url\":\"https://api.github.com/users/gitter-badger/followers\",\"following_url\":\"https://api.github.com/users/gitter-badger/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gitter-badger/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gitter-badger/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gitter-badger/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gitter-badger/orgs\",\"repos_url\":\"https://api.github.com/users/gitter-badger/repos\",\"events_url\":\"https://api.github.com/users/gitter-badger/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gitter-badger/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/gitter-badger/dotfiles-15\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15\",\"forks_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/forks\",\"keys_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/teams\",\"hooks_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/hooks\",\"issue_events_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/events\",\"assignees_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/tags\",\"blobs_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/languages\",\"stargazers_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/stargazers\",\"contributors_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/contributors\",\"subscribers_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/subscribers\",\"subscription_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/subscription\",\"commits_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/merges\",\"archive_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/downloads\",\"issues_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/gitter-badger/dotfiles-15/releases{/id}\",\"created_at\":\"2015-01-01T01:00:07Z\",\"updated_at\":\"2015-01-01T01:00:08Z\",\"pushed_at\":\"2015-01-01T01:00:09Z\",\"git_url\":\"git://github.com/gitter-badger/dotfiles-15.git\",\"ssh_url\":\"git@github.com:gitter-badger/dotfiles-15.git\",\"clone_url\":\"https://github.com/gitter-badger/dotfiles-15.git\",\"svn_url\":\"https://github.com/gitter-badger/dotfiles-15\",\"homepage\":\"\",\"size\":620,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"VimL\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"russmatney:master\",\"ref\":\"master\",\"sha\":\"7ca425b3dc133cd625f3f2e2e014ebee508eb39d\",\"user\":{\"login\":\"russmatney\",\"id\":1596350,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1596350?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/russmatney\",\"html_url\":\"https://github.com/russmatney\",\"followers_url\":\"https://api.github.com/users/russmatney/followers\",\"following_url\":\"https://api.github.com/users/russmatney/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/russmatney/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/russmatney/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/russmatney/subscriptions\",\"organizations_url\":\"https://api.github.com/users/russmatney/orgs\",\"repos_url\":\"https://api.github.com/users/russmatney/repos\",\"events_url\":\"https://api.github.com/users/russmatney/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/russmatney/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":8540783,\"name\":\"dotfiles\",\"full_name\":\"russmatney/dotfiles\",\"owner\":{\"login\":\"russmatney\",\"id\":1596350,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1596350?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/russmatney\",\"html_url\":\"https://github.com/russmatney\",\"followers_url\":\"https://api.github.com/users/russmatney/followers\",\"following_url\":\"https://api.github.com/users/russmatney/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/russmatney/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/russmatney/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/russmatney/subscriptions\",\"organizations_url\":\"https://api.github.com/users/russmatney/orgs\",\"repos_url\":\"https://api.github.com/users/russmatney/repos\",\"events_url\":\"https://api.github.com/users/russmatney/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/russmatney/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/russmatney/dotfiles\",\"description\":\"\",\"fork\":false,\"url\":\"https://api.github.com/repos/russmatney/dotfiles\",\"forks_url\":\"https://api.github.com/repos/russmatney/dotfiles/forks\",\"keys_url\":\"https://api.github.com/repos/russmatney/dotfiles/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/russmatney/dotfiles/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/russmatney/dotfiles/teams\",\"hooks_url\":\"https://api.github.com/repos/russmatney/dotfiles/hooks\",\"issue_events_url\":\"https://api.github.com/repos/russmatney/dotfiles/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/russmatney/dotfiles/events\",\"assignees_url\":\"https://api.github.com/repos/russmatney/dotfiles/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/russmatney/dotfiles/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/russmatney/dotfiles/tags\",\"blobs_url\":\"https://api.github.com/repos/russmatney/dotfiles/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/russmatney/dotfiles/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/russmatney/dotfiles/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/russmatney/dotfiles/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/russmatney/dotfiles/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/russmatney/dotfiles/languages\",\"stargazers_url\":\"https://api.github.com/repos/russmatney/dotfiles/stargazers\",\"contributors_url\":\"https://api.github.com/repos/russmatney/dotfiles/contributors\",\"subscribers_url\":\"https://api.github.com/repos/russmatney/dotfiles/subscribers\",\"subscription_url\":\"https://api.github.com/repos/russmatney/dotfiles/subscription\",\"commits_url\":\"https://api.github.com/repos/russmatney/dotfiles/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/russmatney/dotfiles/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/russmatney/dotfiles/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/russmatney/dotfiles/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/russmatney/dotfiles/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/russmatney/dotfiles/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/russmatney/dotfiles/merges\",\"archive_url\":\"https://api.github.com/repos/russmatney/dotfiles/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/russmatney/dotfiles/downloads\",\"issues_url\":\"https://api.github.com/repos/russmatney/dotfiles/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/russmatney/dotfiles/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/russmatney/dotfiles/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/russmatney/dotfiles/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/russmatney/dotfiles/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/russmatney/dotfiles/releases{/id}\",\"created_at\":\"2013-03-03T18:57:40Z\",\"updated_at\":\"2014-12-30T19:07:15Z\",\"pushed_at\":\"2014-12-30T19:07:15Z\",\"git_url\":\"git://github.com/russmatney/dotfiles.git\",\"ssh_url\":\"git@github.com:russmatney/dotfiles.git\",\"clone_url\":\"https://github.com/russmatney/dotfiles.git\",\"svn_url\":\"https://github.com/russmatney/dotfiles\",\"homepage\":\"\",\"size\":620,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"VimL\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":1,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":1,\"open_issues\":1,\"watchers\":1,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1\"},\"html\":{\"href\":\"https://github.com/russmatney/dotfiles/pull/1\"},\"issue\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/issues/1\"},\"comments\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/issues/1/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/pulls/1/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/russmatney/dotfiles/statuses/e4f636ddb08f1811808e3f872a7a323aa80c8a7b\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":2,\"deletions\":0,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395827\",\"type\":\"PushEvent\",\"actor\":{\"id\":6158630,\"login\":\"greatfire\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/greatfire\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6158630?\"},\"repo\":{\"id\":15100395,\"name\":\"greatfire/wiki\",\"url\":\"https://api.github.com/repos/greatfire/wiki\"},\"payload\":{\"push_id\":536752151,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ba5bb9726f655f8acd29ecb3f89f1ae55305e680\",\"before\":\"60adcb73302844b76926dc2b33a941f4f5069836\",\"commits\":[{\"sha\":\"ba5bb9726f655f8acd29ecb3f89f1ae55305e680\",\"author\":{\"email\":\"24bf68e341ce0fbd9259a5d51feed79682ea4eba@greatfire.org\",\"name\":\"Ubuntu\"},\"message\":\"a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/greatfire/wiki/commits/ba5bb9726f655f8acd29ecb3f89f1ae55305e680\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395828\",\"type\":\"PushEvent\",\"actor\":{\"id\":314716,\"login\":\"astrofrog\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/astrofrog\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/314716?\"},\"repo\":{\"id\":22914622,\"name\":\"astrofrog/reproject-benchmarks\",\"url\":\"https://api.github.com/repos/astrofrog/reproject-benchmarks\"},\"payload\":{\"push_id\":536752153,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"0e9b7c43a998d5386212968a8b55899c507a7288\",\"before\":\"399b664da8fdf3dd344e09740700a45eabd8377b\",\"commits\":[{\"sha\":\"0e9b7c43a998d5386212968a8b55899c507a7288\",\"author\":{\"email\":\"5a30e73bb3eb7866ce4ccff234d7add82054e98d@gmail.com\",\"name\":\"Thomas Robitaille\"},\"message\":\"Generated from sources\",\"distinct\":true,\"url\":\"https://api.github.com/repos/astrofrog/reproject-benchmarks/commits/0e9b7c43a998d5386212968a8b55899c507a7288\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395834\",\"type\":\"PushEvent\",\"actor\":{\"id\":429529,\"login\":\"cato-\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cato-\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/429529?\"},\"repo\":{\"id\":7588969,\"name\":\"xenim/livestatus-publicpage\",\"url\":\"https://api.github.com/repos/xenim/livestatus-publicpage\"},\"payload\":{\"push_id\":536752158,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"39684ee741d07cd9c8f1ca9f5333b805f3c67ec2\",\"before\":\"aaa220dfce3ed980c25e7277cb174b4705cc23af\",\"commits\":[{\"sha\":\"39684ee741d07cd9c8f1ca9f5333b805f3c67ec2\",\"author\":{\"email\":\"12e9293ec6b30c7fa8a0926af42807e929c1684f@niob.xnis.de\",\"name\":\"Robert Weidlich\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/xenim/livestatus-publicpage/commits/39684ee741d07cd9c8f1ca9f5333b805f3c67ec2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\",\"org\":{\"id\":1789841,\"login\":\"xenim\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/xenim\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1789841?\"}}\n{\"id\":\"2489395835\",\"type\":\"PushEvent\",\"actor\":{\"id\":16432,\"login\":\"tardate\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tardate\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/16432?\"},\"repo\":{\"id\":28238617,\"name\":\"tardate/visual555\",\"url\":\"https://api.github.com/repos/tardate/visual555\"},\"payload\":{\"push_id\":536752159,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a2e48f3da6ae2425950266f6f02857172d33af0a\",\"before\":\"c9b4675aa616e728733541ce7ce42b161fe92b8a\",\"commits\":[{\"sha\":\"a2e48f3da6ae2425950266f6f02857172d33af0a\",\"author\":{\"email\":\"136d28fa799cf1dc4e2c5bcef36214a038721fe5@gmail.com\",\"name\":\"Paul Gallagher\"},\"message\":\"add some book recommendations\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tardate/visual555/commits/a2e48f3da6ae2425950266f6f02857172d33af0a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395838\",\"type\":\"PushEvent\",\"actor\":{\"id\":4447136,\"login\":\"su-github-machine-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/su-github-machine-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4447136?\"},\"repo\":{\"id\":10314483,\"name\":\"su-github-machine-user/github-nagios-check\",\"url\":\"https://api.github.com/repos/su-github-machine-user/github-nagios-check\"},\"payload\":{\"push_id\":536752162,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2320a2b78da315d43fa018bba258dc9b1d07e6b9\",\"before\":\"e9d1b5d1e3bdac7c2aabe75094d401db3ee35f65\",\"commits\":[{\"sha\":\"2320a2b78da315d43fa018bba258dc9b1d07e6b9\",\"author\":{\"email\":\"875a8f2f42c570f5f4ea7bfd154f582bcf95673a@su.se\",\"name\":\"su-githubmirror\"},\"message\":\"New timestamp: 1420074002\",\"distinct\":true,\"url\":\"https://api.github.com/repos/su-github-machine-user/github-nagios-check/commits/2320a2b78da315d43fa018bba258dc9b1d07e6b9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395842\",\"type\":\"PushEvent\",\"actor\":{\"id\":2275298,\"login\":\"enjoydiy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/enjoydiy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2275298?\"},\"repo\":{\"id\":6275680,\"name\":\"enjoydiy/ttautovpn\",\"url\":\"https://api.github.com/repos/enjoydiy/ttautovpn\"},\"payload\":{\"push_id\":536752166,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"947cbd06a44b1f1e2970af802bda84a62b876285\",\"before\":\"b4694b448e512278f83daf0d6c41e3bce241e2e3\",\"commits\":[{\"sha\":\"947cbd06a44b1f1e2970af802bda84a62b876285\",\"author\":{\"email\":\"d033e22ae348aeb5660fc2140aec35850c4da997@enjoydiy.com\",\"name\":\"zijiao\"},\"message\":\"The routes include china IPs,created on 2015/01/01\",\"distinct\":true,\"url\":\"https://api.github.com/repos/enjoydiy/ttautovpn/commits/947cbd06a44b1f1e2970af802bda84a62b876285\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:10Z\"}\n{\"id\":\"2489395845\",\"type\":\"WatchEvent\",\"actor\":{\"id\":681965,\"login\":\"wonbyte\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wonbyte\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/681965?\"},\"repo\":{\"id\":14490265,\"name\":\"idris-lang/idris-tutorial\",\"url\":\"https://api.github.com/repos/idris-lang/idris-tutorial\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\",\"org\":{\"id\":5552910,\"login\":\"idris-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/idris-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5552910?\"}}\n{\"id\":\"2489395846\",\"type\":\"PushEvent\",\"actor\":{\"id\":5186562,\"login\":\"wynot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wynot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5186562?\"},\"repo\":{\"id\":28301051,\"name\":\"wynot/bizmarkio_app\",\"url\":\"https://api.github.com/repos/wynot/bizmarkio_app\"},\"payload\":{\"push_id\":536752167,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1e8c2f76bb60291f62859804bd999e48ac8baf26\",\"before\":\"ed7d2b523d98421614bfbef2eee59d256ef43118\",\"commits\":[{\"sha\":\"1e8c2f76bb60291f62859804bd999e48ac8baf26\",\"author\":{\"email\":\"a586fcf1f3b216ef769d3cfdb1ef68d895375da3@gmail.com\",\"name\":\"Will Young\"},\"message\":\"Add user microposts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wynot/bizmarkio_app/commits/1e8c2f76bb60291f62859804bd999e48ac8baf26\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\"}\n{\"id\":\"2489395848\",\"type\":\"PushEvent\",\"actor\":{\"id\":3004602,\"login\":\"mattkirby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mattkirby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3004602?\"},\"repo\":{\"id\":25184298,\"name\":\"mattkirby/puppet-zpr\",\"url\":\"https://api.github.com/repos/mattkirby/puppet-zpr\"},\"payload\":{\"push_id\":536752169,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/standard_params\",\"head\":\"9f6831aef1e36ee6af01f7dacc12c5a082d8c587\",\"before\":\"25e6dd22e0302c18ab830e43e5448ac6c8f74f86\",\"commits\":[{\"sha\":\"9f6831aef1e36ee6af01f7dacc12c5a082d8c587\",\"author\":{\"email\":\"992efda4ce64b989ffc2f2114584c88f5d0d9310@puppetlabs.com\",\"name\":\"kirby@puppetlabs.com\"},\"message\":\"Remove notify since file resource doesn't work like the exec did\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mattkirby/puppet-zpr/commits/9f6831aef1e36ee6af01f7dacc12c5a082d8c587\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\"}\n{\"id\":\"2489395849\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752170,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8069b04a69d5729520e64e6c88c262a77407bd75\",\"before\":\"b252ace8548afd9d1806bffffbe5c6497ca58edf\",\"commits\":[{\"sha\":\"8069b04a69d5729520e64e6c88c262a77407bd75\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"main\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/8069b04a69d5729520e64e6c88c262a77407bd75\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\"}\n{\"id\":\"2489395850\",\"type\":\"PushEvent\",\"actor\":{\"id\":8147971,\"login\":\"machchk\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/machchk\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8147971?\"},\"repo\":{\"id\":21783823,\"name\":\"machchk/report\",\"url\":\"https://api.github.com/repos/machchk/report\"},\"payload\":{\"push_id\":536752171,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"7b1e68cdc33154e42d97302cf36ed4bfe7d280be\",\"before\":\"34370ef489c66eedc0d0169a95898cc7f82c1c3e\",\"commits\":[{\"sha\":\"7b1e68cdc33154e42d97302cf36ed4bfe7d280be\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@localhost\",\"name\":\"root\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/machchk/report/commits/7b1e68cdc33154e42d97302cf36ed4bfe7d280be\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\"}\n{\"id\":\"2489395853\",\"type\":\"PushEvent\",\"actor\":{\"id\":8838361,\"login\":\"benjamincaldwell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/benjamincaldwell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8838361?\"},\"repo\":{\"id\":28570090,\"name\":\"benjamincaldwell/Website\",\"url\":\"https://api.github.com/repos/benjamincaldwell/Website\"},\"payload\":{\"push_id\":536752174,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"1bce253e7632d45b428be6d1e4b02e39a3d477de\",\"before\":\"d4a7f3e628110633e87261aa0661277ccbd8b75f\",\"commits\":[{\"sha\":\"1bce253e7632d45b428be6d1e4b02e39a3d477de\",\"author\":{\"email\":\"8ffda5448dc6d9eb6fc0b9b1283b3b20a847a4a2@gmail.com\",\"name\":\"Benjamin Caldwell\"},\"message\":\"different background image method\",\"distinct\":true,\"url\":\"https://api.github.com/repos/benjamincaldwell/Website/commits/1bce253e7632d45b428be6d1e4b02e39a3d477de\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:11Z\"}\n{\"id\":\"2489395862\",\"type\":\"PushEvent\",\"actor\":{\"id\":8770348,\"login\":\"HouseMonitor\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/HouseMonitor\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8770348?\"},\"repo\":{\"id\":24030380,\"name\":\"HouseMonitor/Logs2014-2015\",\"url\":\"https://api.github.com/repos/HouseMonitor/Logs2014-2015\"},\"payload\":{\"push_id\":536752177,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a84a03375407c8616b1a105a052f5f29f8e43261\",\"before\":\"68ebc9a7d6ad3a823361e4f3cebda44e42342fee\",\"commits\":[{\"sha\":\"a84a03375407c8616b1a105a052f5f29f8e43261\",\"author\":{\"email\":\"17e72c8f1b0781cefad8c299a70b47a752ed01a6@gmail.com\",\"name\":\"Matej Drolc\"},\"message\":\"automated commit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/HouseMonitor/Logs2014-2015/commits/a84a03375407c8616b1a105a052f5f29f8e43261\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:12Z\"}\n{\"id\":\"2489395865\",\"type\":\"CreateEvent\",\"actor\":{\"id\":3521359,\"login\":\"ssomnoremac\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ssomnoremac\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3521359?\"},\"repo\":{\"id\":28655796,\"name\":\"ssomnoremac/mean\",\"url\":\"https://api.github.com/repos/ssomnoremac/mean\"},\"payload\":{\"ref\":\"dev\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"MEAN.JS - Full-Stack JavaScript Using MongoDB, Express, AngularJS, and Node.js - \",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:12Z\"}\n{\"id\":\"2489395866\",\"type\":\"PushEvent\",\"actor\":{\"id\":9101573,\"login\":\"megantmcginley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/megantmcginley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9101573?\"},\"repo\":{\"id\":25549968,\"name\":\"megantmcginley/megantmcginley.github.io\",\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io\"},\"payload\":{\"push_id\":536752179,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e613a42ea94329f38b4f9994b48ccd4f845a5225\",\"before\":\"0d1eeb608f9fbcc69047690dbb9019ac8217def7\",\"commits\":[{\"sha\":\"e613a42ea94329f38b4f9994b48ccd4f845a5225\",\"author\":{\"email\":\"92f56e51255edbb80c74150d0115560b34c2bc35@users.noreply.github.com\",\"name\":\"megantmcginley\"},\"message\":\"Update projects.html\",\"distinct\":true,\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io/commits/e613a42ea94329f38b4f9994b48ccd4f845a5225\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:12Z\"}\n{\"id\":\"2489395869\",\"type\":\"WatchEvent\",\"actor\":{\"id\":4725234,\"login\":\"xuhf\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/xuhf\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4725234?\"},\"repo\":{\"id\":10197269,\"name\":\"xiaobozi/youku-lixian\",\"url\":\"https://api.github.com/repos/xiaobozi/youku-lixian\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:12Z\"}\n{\"id\":\"2489395875\",\"type\":\"PushEvent\",\"actor\":{\"id\":2318343,\"login\":\"treckstar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/treckstar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2318343?\"},\"repo\":{\"id\":17101123,\"name\":\"treckstar/yolo-octo-hipster\",\"url\":\"https://api.github.com/repos/treckstar/yolo-octo-hipster\"},\"payload\":{\"push_id\":536752182,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e8d20aa6bc5d4da90d065848c17e0abd7ee1bc9b\",\"before\":\"025bfa80305fd4f33670e6cfd1ced7e3dfcc5c94\",\"commits\":[{\"sha\":\"e8d20aa6bc5d4da90d065848c17e0abd7ee1bc9b\",\"author\":{\"email\":\"28cf8d5dd63a27bb1a047ac2fe7ded863d3bc56c@gmail.com\",\"name\":\"treckstar\"},\"message\":\"Got the words that burn like fire in my mouth.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/treckstar/yolo-octo-hipster/commits/e8d20aa6bc5d4da90d065848c17e0abd7ee1bc9b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:13Z\"}\n{\"id\":\"2489395882\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":412280,\"login\":\"kayone\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kayone\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/412280?\"},\"repo\":{\"id\":2565137,\"name\":\"SynoCommunity/spksrc\",\"url\":\"https://api.github.com/repos/SynoCommunity/spksrc\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/comments/22397252\",\"id\":22397252,\"diff_hunk\":\"@@ -0,0 +1,60 @@\\n+SPK_NAME = nzbdrone\",\"path\":\"spk/sonarr/Makefile\",\"position\":1,\"original_position\":1,\"commit_id\":\"d7db1b629ab3e2894060967467f483fa936bf048\",\"original_commit_id\":\"cc43ff5895bb3cd025dee598705e1a0404c8db6f\",\"user\":{\"login\":\"kayone\",\"id\":412280,\"avatar_url\":\"https://avatars.githubusercontent.com/u/412280?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kayone\",\"html_url\":\"https://github.com/kayone\",\"followers_url\":\"https://api.github.com/users/kayone/followers\",\"following_url\":\"https://api.github.com/users/kayone/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kayone/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kayone/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kayone/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kayone/orgs\",\"repos_url\":\"https://api.github.com/users/kayone/repos\",\"events_url\":\"https://api.github.com/users/kayone/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kayone/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"makes sense,\\r\\nthanks, wasn't sure if we needed to maintain backwards compatibility.\",\"created_at\":\"2015-01-01T01:00:13Z\",\"updated_at\":\"2015-01-01T01:00:13Z\",\"html_url\":\"https://github.com/SynoCommunity/spksrc/pull/1409#discussion_r22397252\",\"pull_request_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/comments/22397252\"},\"html\":{\"href\":\"https://github.com/SynoCommunity/spksrc/pull/1409#discussion_r22397252\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409\",\"id\":26606843,\"html_url\":\"https://github.com/SynoCommunity/spksrc/pull/1409\",\"diff_url\":\"https://github.com/SynoCommunity/spksrc/pull/1409.diff\",\"patch_url\":\"https://github.com/SynoCommunity/spksrc/pull/1409.patch\",\"issue_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/1409\",\"number\":1409,\"state\":\"open\",\"locked\":false,\"title\":\"NzbDrone to Sonarr + dependencies updated\",\"user\":{\"login\":\"maxrogers\",\"id\":265707,\"avatar_url\":\"https://avatars.githubusercontent.com/u/265707?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/maxrogers\",\"html_url\":\"https://github.com/maxrogers\",\"followers_url\":\"https://api.github.com/users/maxrogers/followers\",\"following_url\":\"https://api.github.com/users/maxrogers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/maxrogers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/maxrogers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/maxrogers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/maxrogers/orgs\",\"repos_url\":\"https://api.github.com/users/maxrogers/repos\",\"events_url\":\"https://api.github.com/users/maxrogers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/maxrogers/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2014-12-26T21:05:05Z\",\"updated_at\":\"2015-01-01T01:00:13Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"673d66d273703a319d6a3b626e070cd37957f218\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409/commits\",\"review_comments_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409/comments\",\"review_comment_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/1409/comments\",\"statuses_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/statuses/d7db1b629ab3e2894060967467f483fa936bf048\",\"head\":{\"label\":\"maxrogers:sonarr\",\"ref\":\"sonarr\",\"sha\":\"d7db1b629ab3e2894060967467f483fa936bf048\",\"user\":{\"login\":\"maxrogers\",\"id\":265707,\"avatar_url\":\"https://avatars.githubusercontent.com/u/265707?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/maxrogers\",\"html_url\":\"https://github.com/maxrogers\",\"followers_url\":\"https://api.github.com/users/maxrogers/followers\",\"following_url\":\"https://api.github.com/users/maxrogers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/maxrogers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/maxrogers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/maxrogers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/maxrogers/orgs\",\"repos_url\":\"https://api.github.com/users/maxrogers/repos\",\"events_url\":\"https://api.github.com/users/maxrogers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/maxrogers/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28520597,\"name\":\"spksrc\",\"full_name\":\"maxrogers/spksrc\",\"owner\":{\"login\":\"maxrogers\",\"id\":265707,\"avatar_url\":\"https://avatars.githubusercontent.com/u/265707?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/maxrogers\",\"html_url\":\"https://github.com/maxrogers\",\"followers_url\":\"https://api.github.com/users/maxrogers/followers\",\"following_url\":\"https://api.github.com/users/maxrogers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/maxrogers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/maxrogers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/maxrogers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/maxrogers/orgs\",\"repos_url\":\"https://api.github.com/users/maxrogers/repos\",\"events_url\":\"https://api.github.com/users/maxrogers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/maxrogers/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/maxrogers/spksrc\",\"description\":\"Cross compilation framework to create native packages for the Synology's NAS\",\"fork\":true,\"url\":\"https://api.github.com/repos/maxrogers/spksrc\",\"forks_url\":\"https://api.github.com/repos/maxrogers/spksrc/forks\",\"keys_url\":\"https://api.github.com/repos/maxrogers/spksrc/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/maxrogers/spksrc/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/maxrogers/spksrc/teams\",\"hooks_url\":\"https://api.github.com/repos/maxrogers/spksrc/hooks\",\"issue_events_url\":\"https://api.github.com/repos/maxrogers/spksrc/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/maxrogers/spksrc/events\",\"assignees_url\":\"https://api.github.com/repos/maxrogers/spksrc/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/maxrogers/spksrc/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/maxrogers/spksrc/tags\",\"blobs_url\":\"https://api.github.com/repos/maxrogers/spksrc/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/maxrogers/spksrc/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/maxrogers/spksrc/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/maxrogers/spksrc/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/maxrogers/spksrc/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/maxrogers/spksrc/languages\",\"stargazers_url\":\"https://api.github.com/repos/maxrogers/spksrc/stargazers\",\"contributors_url\":\"https://api.github.com/repos/maxrogers/spksrc/contributors\",\"subscribers_url\":\"https://api.github.com/repos/maxrogers/spksrc/subscribers\",\"subscription_url\":\"https://api.github.com/repos/maxrogers/spksrc/subscription\",\"commits_url\":\"https://api.github.com/repos/maxrogers/spksrc/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/maxrogers/spksrc/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/maxrogers/spksrc/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/maxrogers/spksrc/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/maxrogers/spksrc/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/maxrogers/spksrc/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/maxrogers/spksrc/merges\",\"archive_url\":\"https://api.github.com/repos/maxrogers/spksrc/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/maxrogers/spksrc/downloads\",\"issues_url\":\"https://api.github.com/repos/maxrogers/spksrc/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/maxrogers/spksrc/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/maxrogers/spksrc/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/maxrogers/spksrc/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/maxrogers/spksrc/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/maxrogers/spksrc/releases{/id}\",\"created_at\":\"2014-12-26T20:36:05Z\",\"updated_at\":\"2014-12-26T20:36:07Z\",\"pushed_at\":\"2014-12-31T22:30:08Z\",\"git_url\":\"git://github.com/maxrogers/spksrc.git\",\"ssh_url\":\"git@github.com:maxrogers/spksrc.git\",\"clone_url\":\"https://github.com/maxrogers/spksrc.git\",\"svn_url\":\"https://github.com/maxrogers/spksrc\",\"homepage\":\"https://github.com/SynoCommunity/spksrc\",\"size\":6160,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Makefile\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"develop\"}},\"base\":{\"label\":\"SynoCommunity:develop\",\"ref\":\"develop\",\"sha\":\"9e6205a91eb2c59cccbe8d1d479065b07727f061\",\"user\":{\"login\":\"SynoCommunity\",\"id\":1123581,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1123581?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SynoCommunity\",\"html_url\":\"https://github.com/SynoCommunity\",\"followers_url\":\"https://api.github.com/users/SynoCommunity/followers\",\"following_url\":\"https://api.github.com/users/SynoCommunity/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/SynoCommunity/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/SynoCommunity/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/SynoCommunity/subscriptions\",\"organizations_url\":\"https://api.github.com/users/SynoCommunity/orgs\",\"repos_url\":\"https://api.github.com/users/SynoCommunity/repos\",\"events_url\":\"https://api.github.com/users/SynoCommunity/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/SynoCommunity/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":2565137,\"name\":\"spksrc\",\"full_name\":\"SynoCommunity/spksrc\",\"owner\":{\"login\":\"SynoCommunity\",\"id\":1123581,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1123581?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SynoCommunity\",\"html_url\":\"https://github.com/SynoCommunity\",\"followers_url\":\"https://api.github.com/users/SynoCommunity/followers\",\"following_url\":\"https://api.github.com/users/SynoCommunity/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/SynoCommunity/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/SynoCommunity/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/SynoCommunity/subscriptions\",\"organizations_url\":\"https://api.github.com/users/SynoCommunity/orgs\",\"repos_url\":\"https://api.github.com/users/SynoCommunity/repos\",\"events_url\":\"https://api.github.com/users/SynoCommunity/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/SynoCommunity/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/SynoCommunity/spksrc\",\"description\":\"Cross compilation framework to create native packages for the Synology's NAS\",\"fork\":false,\"url\":\"https://api.github.com/repos/SynoCommunity/spksrc\",\"forks_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/forks\",\"keys_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/teams\",\"hooks_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/hooks\",\"issue_events_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/events\",\"assignees_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/tags\",\"blobs_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/languages\",\"stargazers_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/stargazers\",\"contributors_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/contributors\",\"subscribers_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/subscribers\",\"subscription_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/subscription\",\"commits_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/merges\",\"archive_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/downloads\",\"issues_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/SynoCommunity/spksrc/releases{/id}\",\"created_at\":\"2011-10-12T20:25:50Z\",\"updated_at\":\"2014-12-31T13:44:45Z\",\"pushed_at\":\"2014-12-31T11:23:04Z\",\"git_url\":\"git://github.com/SynoCommunity/spksrc.git\",\"ssh_url\":\"git@github.com:SynoCommunity/spksrc.git\",\"clone_url\":\"https://github.com/SynoCommunity/spksrc.git\",\"svn_url\":\"https://github.com/SynoCommunity/spksrc\",\"homepage\":\"https://github.com/SynoCommunity/spksrc\",\"size\":30572,\"stargazers_count\":617,\"watchers_count\":617,\"language\":\"Makefile\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":294,\"mirror_url\":null,\"open_issues_count\":229,\"forks\":294,\"open_issues\":229,\"watchers\":617,\"default_branch\":\"develop\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409\"},\"html\":{\"href\":\"https://github.com/SynoCommunity/spksrc/pull/1409\"},\"issue\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/1409\"},\"comments\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/issues/1409/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/pulls/1409/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/SynoCommunity/spksrc/statuses/d7db1b629ab3e2894060967467f483fa936bf048\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:00:13Z\",\"org\":{\"id\":1123581,\"login\":\"SynoCommunity\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/SynoCommunity\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1123581?\"}}\n{\"id\":\"2489395883\",\"type\":\"PushEvent\",\"actor\":{\"id\":4102215,\"login\":\"d3stats\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3stats\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4102215?\"},\"repo\":{\"id\":9317463,\"name\":\"d3stats/d3.fuzz.me.uk\",\"url\":\"https://api.github.com/repos/d3stats/d3.fuzz.me.uk\"},\"payload\":{\"push_id\":536752185,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"87f4b08c009c2cecc8f8411f48be921f9f5ab4f8\",\"before\":\"09f4bcab487b3cf4bb8101c28fde4ddb4326b95c\",\"commits\":[{\"sha\":\"87f4b08c009c2cecc8f8411f48be921f9f5ab4f8\",\"author\":{\"email\":\"4f26aeafdb2367620a393c973eddbe8f8b846ebd@fuzz.me.uk\",\"name\":\"d3stats\"},\"message\":\"scheduled update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/d3stats/d3.fuzz.me.uk/commits/87f4b08c009c2cecc8f8411f48be921f9f5ab4f8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:13Z\"}\n{\"id\":\"2489395884\",\"type\":\"PushEvent\",\"actor\":{\"id\":3160808,\"login\":\"trustedsec\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/trustedsec\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3160808?\"},\"repo\":{\"id\":7391261,\"name\":\"trustedsec/social-engineer-toolkit\",\"url\":\"https://api.github.com/repos/trustedsec/social-engineer-toolkit\"},\"payload\":{\"push_id\":536752186,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8f07649e93f94728dd876e7bcbd3cef21e8add69\",\"before\":\"981c2c1a0fbea08bf5555725afa8e1b2d6d6fe2e\",\"commits\":[{\"sha\":\"8f07649e93f94728dd876e7bcbd3cef21e8add69\",\"author\":{\"email\":\"863b5d643be6fa3d7c1bad8d1f065e00f75dc0c2@trustedsec.com\",\"name\":\"trustedsec\"},\"message\":\"Updated Java Applet with obfuscation.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/trustedsec/social-engineer-toolkit/commits/8f07649e93f94728dd876e7bcbd3cef21e8add69\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:14Z\"}\n{\"id\":\"2489395885\",\"type\":\"WatchEvent\",\"actor\":{\"id\":489911,\"login\":\"vasanthela\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vasanthela\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/489911?\"},\"repo\":{\"id\":27021112,\"name\":\"sorentwo/readthis\",\"url\":\"https://api.github.com/repos/sorentwo/readthis\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395886\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1414603,\"login\":\"zoldello\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?\"},\"repo\":{\"id\":26617214,\"name\":\"ChicagoVeg/restaurantList\",\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9\",\"labels_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/comments\",\"events_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/events\",\"html_url\":\"https://github.com/ChicagoVeg/restaurantList/issues/9\",\"id\":53210171,\"number\":9,\"title\":\"When I filter out restaurants by type, the removed restaurant pins remain in the map\",\"user\":{\"login\":\"zoldello\",\"id\":1414603,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"html_url\":\"https://github.com/zoldello\",\"followers_url\":\"https://api.github.com/users/zoldello/followers\",\"following_url\":\"https://api.github.com/users/zoldello/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/zoldello/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/zoldello/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/zoldello/subscriptions\",\"organizations_url\":\"https://api.github.com/users/zoldello/orgs\",\"repos_url\":\"https://api.github.com/users/zoldello/repos\",\"events_url\":\"https://api.github.com/users/zoldello/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/zoldello/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"},{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/labels/Great+to+Have\",\"name\":\"Great to Have\",\"color\":\"f7c6c7\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"zoldello\",\"id\":1414603,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"html_url\":\"https://github.com/zoldello\",\"followers_url\":\"https://api.github.com/users/zoldello/followers\",\"following_url\":\"https://api.github.com/users/zoldello/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/zoldello/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/zoldello/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/zoldello/subscriptions\",\"organizations_url\":\"https://api.github.com/users/zoldello/orgs\",\"repos_url\":\"https://api.github.com/users/zoldello/repos\",\"events_url\":\"https://api.github.com/users/zoldello/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/zoldello/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/milestones/3\",\"labels_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/milestones/3/labels\",\"id\":894444,\"number\":3,\"title\":\"Release 1\",\"description\":\"\",\"creator\":{\"login\":\"vadim424\",\"id\":10101875,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10101875?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vadim424\",\"html_url\":\"https://github.com/vadim424\",\"followers_url\":\"https://api.github.com/users/vadim424/followers\",\"following_url\":\"https://api.github.com/users/vadim424/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/vadim424/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/vadim424/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/vadim424/subscriptions\",\"organizations_url\":\"https://api.github.com/users/vadim424/orgs\",\"repos_url\":\"https://api.github.com/users/vadim424/repos\",\"events_url\":\"https://api.github.com/users/vadim424/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/vadim424/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":3,\"closed_issues\":4,\"state\":\"open\",\"created_at\":\"2014-12-07T00:34:16Z\",\"updated_at\":\"2015-01-01T01:00:14Z\",\"due_on\":\"2014-12-14T06:00:00Z\",\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:00:14Z\",\"updated_at\":\"2015-01-01T01:00:14Z\",\"closed_at\":null,\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\",\"org\":{\"id\":9426295,\"login\":\"ChicagoVeg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ChicagoVeg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9426295?\"}}\n{\"id\":\"2489395891\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3514976,\"login\":\"trupin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/trupin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3514976?\"},\"repo\":{\"id\":2544305,\"name\":\"tomakehurst/wiremock\",\"url\":\"https://api.github.com/repos/tomakehurst/wiremock\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395900\",\"type\":\"PushEvent\",\"actor\":{\"id\":2101973,\"login\":\"konjac\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/konjac\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2101973?\"},\"repo\":{\"id\":24664906,\"name\":\"konjac/calendars\",\"url\":\"https://api.github.com/repos/konjac/calendars\"},\"payload\":{\"push_id\":536752192,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"e0cebd9186ac0e326934d5fa27e8d350c4b1cf87\",\"before\":\"ef7f231d67bd83d9569d1c34fe99fba72e961c19\",\"commits\":[{\"sha\":\"e0cebd9186ac0e326934d5fa27e8d350c4b1cf87\",\"author\":{\"email\":\"6dc96bd73a12a2b22abd88d2fca39328a25102c5@gmail.com\",\"name\":\"konjac\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/konjac/calendars/commits/e0cebd9186ac0e326934d5fa27e8d350c4b1cf87\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395899\",\"type\":\"PushEvent\",\"actor\":{\"id\":739159,\"login\":\"lessthanoptimal\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lessthanoptimal\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/739159?\"},\"repo\":{\"id\":28421901,\"name\":\"lessthanoptimal/bow\",\"url\":\"https://api.github.com/repos/lessthanoptimal/bow\"},\"payload\":{\"push_id\":536752193,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"05bf0882abaf590379176a0c0093d0cc49d8390e\",\"before\":\"40e23734127f500478caeb3071ea184752fa99c3\",\"commits\":[{\"sha\":\"c10a64918fb1deb7edd5c8d9ed7f31bb715ccb1e\",\"author\":{\"email\":\"d4f7bed66133e57602ad989e685eeed7b4535dfc@gmail.com\",\"name\":\"Peter Abeles\"},\"message\":\"- Octave code is more mature\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lessthanoptimal/bow/commits/c10a64918fb1deb7edd5c8d9ed7f31bb715ccb1e\"},{\"sha\":\"05bf0882abaf590379176a0c0093d0cc49d8390e\",\"author\":{\"email\":\"d4f7bed66133e57602ad989e685eeed7b4535dfc@gmail.com\",\"name\":\"Peter Abeles\"},\"message\":\"- renamed src to python\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lessthanoptimal/bow/commits/05bf0882abaf590379176a0c0093d0cc49d8390e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395903\",\"type\":\"PushEvent\",\"actor\":{\"id\":1396247,\"login\":\"hemstreet\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hemstreet\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1396247?\"},\"repo\":{\"id\":18146818,\"name\":\"Handbid/Handbid-WordPress\",\"url\":\"https://api.github.com/repos/Handbid/Handbid-WordPress\"},\"payload\":{\"push_id\":536752194,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7c3d9244dd7646d85e8028493125e6306bf32070\",\"before\":\"52a447d2c80786cd9c1ef7d9e861562ced30e4d9\",\"commits\":[{\"sha\":\"7c3d9244dd7646d85e8028493125e6306bf32070\",\"author\":{\"email\":\"5e0a51686c7b5ffc310cf0f73a7ea38c001a4814@gmail.com\",\"name\":\"Hemstreet\"},\"message\":\"receipt markup\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Handbid/Handbid-WordPress/commits/7c3d9244dd7646d85e8028493125e6306bf32070\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\",\"org\":{\"id\":7072597,\"login\":\"Handbid\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Handbid\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7072597?\"}}\n{\"id\":\"2489395904\",\"type\":\"PushEvent\",\"actor\":{\"id\":5684907,\"login\":\"martin-williams\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/martin-williams\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5684907?\"},\"repo\":{\"id\":25208939,\"name\":\"martin-williams/tpp_imdb\",\"url\":\"https://api.github.com/repos/martin-williams/tpp_imdb\"},\"payload\":{\"push_id\":536752195,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"317709a0e7b17b5eea34d28a6623e90d8d693304\",\"before\":\"ad1a720f71ec001edefc3f743f83b1fe2917c0db\",\"commits\":[{\"sha\":\"317709a0e7b17b5eea34d28a6623e90d8d693304\",\"author\":{\"email\":\"c9a2ee53052a131cbb42f619416968acbd6ad458@beardon.com\",\"name\":\"Martin Williams\"},\"message\":\"more adjustments to pageant search page\",\"distinct\":true,\"url\":\"https://api.github.com/repos/martin-williams/tpp_imdb/commits/317709a0e7b17b5eea34d28a6623e90d8d693304\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395905\",\"type\":\"PushEvent\",\"actor\":{\"id\":8882603,\"login\":\"pinaet\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pinaet\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8882603?\"},\"repo\":{\"id\":24382269,\"name\":\"pinaet/pinaet.github.io\",\"url\":\"https://api.github.com/repos/pinaet/pinaet.github.io\"},\"payload\":{\"push_id\":536752196,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cfd66e5e7aa225813d3fb60231142d47533bad02\",\"before\":\"be40427e7b0488333d4658aad8c763591b77e794\",\"commits\":[{\"sha\":\"cfd66e5e7aa225813d3fb60231142d47533bad02\",\"author\":{\"email\":\"8207655137af68a9e81af8e7a2b0dcaa36726d43@gmail.com\",\"name\":\"Pinaet\"},\"message\":\"modified facebook plugin 3\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pinaet/pinaet.github.io/commits/cfd66e5e7aa225813d3fb60231142d47533bad02\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395906\",\"type\":\"PushEvent\",\"actor\":{\"id\":8077771,\"login\":\"camsc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/camsc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8077771?\"},\"repo\":{\"id\":22777322,\"name\":\"camsc/camsc.github.io\",\"url\":\"https://api.github.com/repos/camsc/camsc.github.io\"},\"payload\":{\"push_id\":536752197,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"22e97dea8206b1c5f2e0d5b058f9fb887462a887\",\"before\":\"4b6c8016628865488fb78d8dbbfc93d057999ef4\",\"commits\":[{\"sha\":\"22e97dea8206b1c5f2e0d5b058f9fb887462a887\",\"author\":{\"email\":\"1d572acbfa68c7c6e541c7b840d6b622e5c0dc91@khanacademy.org\",\"name\":\"Cam\"},\"message\":\"small update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/camsc/camsc.github.io/commits/22e97dea8206b1c5f2e0d5b058f9fb887462a887\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\"}\n{\"id\":\"2489395909\",\"type\":\"PushEvent\",\"actor\":{\"id\":6267945,\"login\":\"lakotadlustig\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lakotadlustig\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6267945?\"},\"repo\":{\"id\":28615365,\"name\":\"korlabs/tippr\",\"url\":\"https://api.github.com/repos/korlabs/tippr\"},\"payload\":{\"push_id\":536752200,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"010ba4d2a839b14d435b0d2c2ea75a861cab3e5d\",\"before\":\"a306207249b2eaceb9668541e8dcb159dcf33c3f\",\"commits\":[{\"sha\":\"010ba4d2a839b14d435b0d2c2ea75a861cab3e5d\",\"author\":{\"email\":\"3280e741e838010abad30f9c64878ae2d3f3e766@podbe.at\",\"name\":\"Lakota Lustig\"},\"message\":\"Update app.php\",\"distinct\":true,\"url\":\"https://api.github.com/repos/korlabs/tippr/commits/010ba4d2a839b14d435b0d2c2ea75a861cab3e5d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:16Z\",\"org\":{\"id\":7786022,\"login\":\"korlabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/korlabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7786022?\"}}\n{\"id\":\"2489395912\",\"type\":\"WatchEvent\",\"actor\":{\"id\":681965,\"login\":\"wonbyte\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wonbyte\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/681965?\"},\"repo\":{\"id\":2386778,\"name\":\"idris-lang/Idris-dev\",\"url\":\"https://api.github.com/repos/idris-lang/Idris-dev\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:17Z\",\"org\":{\"id\":5552910,\"login\":\"idris-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/idris-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5552910?\"}}\n{\"id\":\"2489395917\",\"type\":\"PushEvent\",\"actor\":{\"id\":10161953,\"login\":\"vanceavalon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vanceavalon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10161953?\"},\"repo\":{\"id\":28676586,\"name\":\"vanceavalon/cassandra\",\"url\":\"https://api.github.com/repos/vanceavalon/cassandra\"},\"payload\":{\"push_id\":536752202,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"474fbd83268b21d63b61739e30e95c58e7121b3c\",\"before\":\"3724660f44fe83771180e377e88f6bdcc23d7053\",\"commits\":[{\"sha\":\"474fbd83268b21d63b61739e30e95c58e7121b3c\",\"author\":{\"email\":\"eb40dd603b6777633aefcd015011472550a9e731@gmail.com\",\"name\":\"vanceavalon\"},\"message\":\"Update variable SEEDS\\n\\nUpdate variable SEEDS get to /services/db/cassandra/node1\",\"distinct\":true,\"url\":\"https://api.github.com/repos/vanceavalon/cassandra/commits/474fbd83268b21d63b61739e30e95c58e7121b3c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:17Z\"}\n{\"id\":\"2489395919\",\"type\":\"CreateEvent\",\"actor\":{\"id\":10081364,\"login\":\"celiaks\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/celiaks\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10081364?\"},\"repo\":{\"id\":28678214,\"name\":\"celiaks/formcheckerSL\",\"url\":\"https://api.github.com/repos/celiaks/formcheckerSL\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"form checker plugin for jquery\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:17Z\"}\n{\"id\":\"2489395921\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1402662,\"login\":\"jamcar23\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamcar23\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1402662?\"},\"repo\":{\"id\":2990192,\"name\":\"WhisperSystems/TextSecure\",\"url\":\"https://api.github.com/repos/WhisperSystems/TextSecure\"},\"payload\":{\"forkee\":{\"id\":28678215,\"name\":\"TextSecure\",\"full_name\":\"jamcar23/TextSecure\",\"owner\":{\"login\":\"jamcar23\",\"id\":1402662,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1402662?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamcar23\",\"html_url\":\"https://github.com/jamcar23\",\"followers_url\":\"https://api.github.com/users/jamcar23/followers\",\"following_url\":\"https://api.github.com/users/jamcar23/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jamcar23/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jamcar23/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jamcar23/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jamcar23/orgs\",\"repos_url\":\"https://api.github.com/users/jamcar23/repos\",\"events_url\":\"https://api.github.com/users/jamcar23/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jamcar23/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/jamcar23/TextSecure\",\"description\":\"A secure text messaging application for Android.\",\"fork\":true,\"url\":\"https://api.github.com/repos/jamcar23/TextSecure\",\"forks_url\":\"https://api.github.com/repos/jamcar23/TextSecure/forks\",\"keys_url\":\"https://api.github.com/repos/jamcar23/TextSecure/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/jamcar23/TextSecure/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/jamcar23/TextSecure/teams\",\"hooks_url\":\"https://api.github.com/repos/jamcar23/TextSecure/hooks\",\"issue_events_url\":\"https://api.github.com/repos/jamcar23/TextSecure/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/jamcar23/TextSecure/events\",\"assignees_url\":\"https://api.github.com/repos/jamcar23/TextSecure/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/jamcar23/TextSecure/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/jamcar23/TextSecure/tags\",\"blobs_url\":\"https://api.github.com/repos/jamcar23/TextSecure/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/jamcar23/TextSecure/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/jamcar23/TextSecure/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/jamcar23/TextSecure/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/jamcar23/TextSecure/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/jamcar23/TextSecure/languages\",\"stargazers_url\":\"https://api.github.com/repos/jamcar23/TextSecure/stargazers\",\"contributors_url\":\"https://api.github.com/repos/jamcar23/TextSecure/contributors\",\"subscribers_url\":\"https://api.github.com/repos/jamcar23/TextSecure/subscribers\",\"subscription_url\":\"https://api.github.com/repos/jamcar23/TextSecure/subscription\",\"commits_url\":\"https://api.github.com/repos/jamcar23/TextSecure/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/jamcar23/TextSecure/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/jamcar23/TextSecure/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/jamcar23/TextSecure/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/jamcar23/TextSecure/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/jamcar23/TextSecure/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/jamcar23/TextSecure/merges\",\"archive_url\":\"https://api.github.com/repos/jamcar23/TextSecure/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/jamcar23/TextSecure/downloads\",\"issues_url\":\"https://api.github.com/repos/jamcar23/TextSecure/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/jamcar23/TextSecure/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/jamcar23/TextSecure/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/jamcar23/TextSecure/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/jamcar23/TextSecure/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/jamcar23/TextSecure/releases{/id}\",\"created_at\":\"2015-01-01T01:00:17Z\",\"updated_at\":\"2015-01-01T00:00:47Z\",\"pushed_at\":\"2015-01-01T00:00:45Z\",\"git_url\":\"git://github.com/jamcar23/TextSecure.git\",\"ssh_url\":\"git@github.com:jamcar23/TextSecure.git\",\"clone_url\":\"https://github.com/jamcar23/TextSecure.git\",\"svn_url\":\"https://github.com/jamcar23/TextSecure\",\"homepage\":\"\",\"size\":54519,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:18Z\",\"org\":{\"id\":702459,\"login\":\"WhisperSystems\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/WhisperSystems\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/702459?\"}}\n{\"id\":\"2489395933\",\"type\":\"PushEvent\",\"actor\":{\"id\":1856621,\"login\":\"InternetDevels\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/InternetDevels\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1856621?\"},\"repo\":{\"id\":20291263,\"name\":\"InternetDevels/drupalcores\",\"url\":\"https://api.github.com/repos/InternetDevels/drupalcores\"},\"payload\":{\"push_id\":536752208,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"11b1d13135fd0d4a56a916b0a99468b9e50a3061\",\"before\":\"dbb4ef6b7364169b3ef3433e6bb5579555e78353\",\"commits\":[{\"sha\":\"11b1d13135fd0d4a56a916b0a99468b9e50a3061\",\"author\":{\"email\":\"141a748f5c0795fdf68eaad85b65480c92abbe5f@internetdevels.com\",\"name\":\"mula\"},\"message\":\"Update bump.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/InternetDevels/drupalcores/commits/11b1d13135fd0d4a56a916b0a99468b9e50a3061\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:21Z\"}\n{\"id\":\"2489395934\",\"type\":\"PushEvent\",\"actor\":{\"id\":3196313,\"login\":\"dpastoor\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpastoor\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3196313?\"},\"repo\":{\"id\":16302099,\"name\":\"dpastoor/PKPDmisc\",\"url\":\"https://api.github.com/repos/dpastoor/PKPDmisc\"},\"payload\":{\"push_id\":536752210,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"81e56dd1160f5bbe8628cb61de33f4636e4588f0\",\"before\":\"9e3b7585ba59c6f8e7c80cca0420d954164ff648\",\"commits\":[{\"sha\":\"d705ff1bd6889fb2beb7483e203b3234d1ce62df\",\"author\":{\"email\":\"5480d79f0d3b1ca7acc7421688b095f8d1e51564@gmail.com\",\"name\":\"Devin Pastoor\"},\"message\":\"refactor, change api to give pauc as a range, fix bug in partial auc calculation where concentration and time points did not match\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpastoor/PKPDmisc/commits/d705ff1bd6889fb2beb7483e203b3234d1ce62df\"},{\"sha\":\"81e56dd1160f5bbe8628cb61de33f4636e4588f0\",\"author\":{\"email\":\"5480d79f0d3b1ca7acc7421688b095f8d1e51564@gmail.com\",\"name\":\"Devin Pastoor\"},\"message\":\"partial AUC summarization functions\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpastoor/PKPDmisc/commits/81e56dd1160f5bbe8628cb61de33f4636e4588f0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:21Z\"}\n{\"id\":\"2489395935\",\"type\":\"PushEvent\",\"actor\":{\"id\":50891,\"login\":\"westurner\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/westurner\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/50891?\"},\"repo\":{\"id\":23823526,\"name\":\"wrdrd/docs\",\"url\":\"https://api.github.com/repos/wrdrd/docs\"},\"payload\":{\"push_id\":536752212,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"43ccb4fcce46afbf06307e1272e0fa8fb08558e3\",\"before\":\"6929c4af15e4293d16bdc64edcd9568b30994dbe\",\"commits\":[{\"sha\":\"43ccb4fcce46afbf06307e1272e0fa8fb08558e3\",\"author\":{\"email\":\"dd516747aaff812d65b57b832bbf2900e5471cbc@wrd.nu\",\"name\":\"Wes Turner\"},\"message\":\"Update documentation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wrdrd/docs/commits/43ccb4fcce46afbf06307e1272e0fa8fb08558e3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:21Z\",\"org\":{\"id\":8705413,\"login\":\"wrdrd\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wrdrd\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8705413?\"}}\n{\"id\":\"2489395944\",\"type\":\"PushEvent\",\"actor\":{\"id\":50891,\"login\":\"westurner\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/westurner\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/50891?\"},\"repo\":{\"id\":23823526,\"name\":\"wrdrd/docs\",\"url\":\"https://api.github.com/repos/wrdrd/docs\"},\"payload\":{\"push_id\":536752217,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"a55294c22ea1c069306bb13a2fd47ff3950e7b47\",\"before\":\"e44784669180fe7c0c3014aa307b0df1722afa1a\",\"commits\":[{\"sha\":\"a55294c22ea1c069306bb13a2fd47ff3950e7b47\",\"author\":{\"email\":\"dd516747aaff812d65b57b832bbf2900e5471cbc@wrd.nu\",\"name\":\"Wes Turner\"},\"message\":\"BLD: Add sphinxcontrib-srclinks https://github.com/westurner/sphinxcontrib-srclinks\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wrdrd/docs/commits/a55294c22ea1c069306bb13a2fd47ff3950e7b47\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:21Z\",\"org\":{\"id\":8705413,\"login\":\"wrdrd\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wrdrd\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8705413?\"}}\n{\"id\":\"2489395950\",\"type\":\"CreateEvent\",\"actor\":{\"id\":10263666,\"login\":\"katiekroik\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/katiekroik\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10263666?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"ref\":\"Develop\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:21Z\"}\n{\"id\":\"2489395958\",\"type\":\"PushEvent\",\"actor\":{\"id\":7387879,\"login\":\"dsm-git\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dsm-git\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7387879?\"},\"repo\":{\"id\":28354666,\"name\":\"Door43/d43-ar-x-dcv\",\"url\":\"https://api.github.com/repos/Door43/d43-ar-x-dcv\"},\"payload\":{\"push_id\":536752224,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6158b50e1a2b200c5efba979f08248b153ced502\",\"before\":\"6e5fca555cca6b3531b17347936d26dc7ddc89e4\",\"commits\":[{\"sha\":\"6158b50e1a2b200c5efba979f08248b153ced502\",\"author\":{\"email\":\"62eb0db178518a8376b23676c2639eb2732c0be8@us.door43.org\",\"name\":\"Apache\"},\"message\":\"Page Edit [39]:  [EricWatt]\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Door43/d43-ar-x-dcv/commits/6158b50e1a2b200c5efba979f08248b153ced502\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:22Z\",\"org\":{\"id\":4982125,\"login\":\"Door43\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Door43\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4982125?\"}}\n{\"id\":\"2489395959\",\"type\":\"PushEvent\",\"actor\":{\"id\":4083697,\"login\":\"benkrikler\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/benkrikler\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4083697?\"},\"repo\":{\"id\":20542797,\"name\":\"alcap-org/alcap-org.github.io\",\"url\":\"https://api.github.com/repos/alcap-org/alcap-org.github.io\"},\"payload\":{\"push_id\":536752225,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7a97c7e4c1ebbc0e9fb204be5c2d8f29588c081a\",\"before\":\"67705976f15e45db781567aa3533c147042b380b\",\"commits\":[{\"sha\":\"7a97c7e4c1ebbc0e9fb204be5c2d8f29588c081a\",\"author\":{\"email\":\"a7d7c4a65fd9795221a11ba0ae55ee302ba5e64d@googlemail.com\",\"name\":\"benkrikler\"},\"message\":\"Automatically regenerated doxygen documentation for branch 'master' of 'g4sim' on Thu Jan  1 01:00:01 GMT 2015\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alcap-org/alcap-org.github.io/commits/7a97c7e4c1ebbc0e9fb204be5c2d8f29588c081a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:22Z\",\"org\":{\"id\":7251877,\"login\":\"alcap-org\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/alcap-org\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7251877?\"}}\n{\"id\":\"2489395962\",\"type\":\"CreateEvent\",\"actor\":{\"id\":16432,\"login\":\"tardate\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tardate\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/16432?\"},\"repo\":{\"id\":28238617,\"name\":\"tardate/visual555\",\"url\":\"https://api.github.com/repos/tardate/visual555\"},\"payload\":{\"ref\":\"1.0.1\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:23Z\"}\n{\"id\":\"2489395967\",\"type\":\"PushEvent\",\"actor\":{\"id\":5314189,\"login\":\"panpawn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/panpawn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5314189?\"},\"repo\":{\"id\":12385273,\"name\":\"panpawn/Pokemon-Showdown\",\"url\":\"https://api.github.com/repos/panpawn/Pokemon-Showdown\"},\"payload\":{\"push_id\":536752228,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a8f530712351c080960f0ff0ed94bc68b3e3a760\",\"before\":\"a1301b53905292c61da4352a0acbe5145edbb32b\",\"commits\":[{\"sha\":\"a8f530712351c080960f0ff0ed94bc68b3e3a760\",\"author\":{\"email\":\"cf2a241a02ff3bf0bd910964e03ee01c59e44ac0@gmail.com\",\"name\":\"panpawn\"},\"message\":\"add AbidE\",\"distinct\":true,\"url\":\"https://api.github.com/repos/panpawn/Pokemon-Showdown/commits/a8f530712351c080960f0ff0ed94bc68b3e3a760\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:23Z\"}\n{\"id\":\"2489395974\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1640798,\"login\":\"langorn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/langorn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1640798?\"},\"repo\":{\"id\":28678190,\"name\":\"langorn/crm\",\"url\":\"https://api.github.com/repos/langorn/crm\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"a crm for content management\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:24Z\"}\n{\"id\":\"2489395980\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536752237,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0366b227c1b20d451524d9712bb67771daa739f3\",\"before\":\"65f5ec025e3a0fd0cb5175e765477a051c20570f\",\"commits\":[{\"sha\":\"0366b227c1b20d451524d9712bb67771daa739f3\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Create LICENSE.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/0366b227c1b20d451524d9712bb67771daa739f3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:25Z\"}\n{\"id\":\"2489395981\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":10361074,\"login\":\"CFLPlayer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CFLPlayer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361074?\"},\"repo\":{\"id\":3112411,\"name\":\"mcMMO-Dev/mcMMO\",\"url\":\"https://api.github.com/repos/mcMMO-Dev/mcMMO\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/mcMMO-Dev/mcMMO/issues/2395\",\"labels_url\":\"https://api.github.com/repos/mcMMO-Dev/mcMMO/issues/2395/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/mcMMO-Dev/mcMMO/issues/2395/comments\",\"events_url\":\"https://api.github.com/repos/mcMMO-Dev/mcMMO/issues/2395/events\",\"html_url\":\"https://github.com/mcMMO-Dev/mcMMO/issues/2395\",\"id\":53210174,\"number\":2395,\"title\":\"Specialisation for axes, swords and unarmed.\",\"user\":{\"login\":\"CFLPlayer\",\"id\":10361074,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361074?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CFLPlayer\",\"html_url\":\"https://github.com/CFLPlayer\",\"followers_url\":\"https://api.github.com/users/CFLPlayer/followers\",\"following_url\":\"https://api.github.com/users/CFLPlayer/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CFLPlayer/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CFLPlayer/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CFLPlayer/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CFLPlayer/orgs\",\"repos_url\":\"https://api.github.com/users/CFLPlayer/repos\",\"events_url\":\"https://api.github.com/users/CFLPlayer/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CFLPlayer/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:00:25Z\",\"updated_at\":\"2015-01-01T01:00:25Z\",\"closed_at\":null,\"body\":\"Berserk, skull splitter and serrated strikes. In hypothetical terms, every person who wields an axe isnt always going to crack a skull open. A martial arts expert is hardly going to go into a full Celtic berserk are they?\\r\\nSo. My point is that there is no diversity. As you train up your skill you should be able to choose what you focus on when wielding your weapon. I'm proposing we add a progression system, which allows players to build upon a basic foundation of an ability. Using a point every 100 levels you should be able to build a \\\"ability specialisation\\\" that is unique to the player. I will use unarmed as an example;\\r\\nWhen your reach level 100 in unarmed you receive a small menu (possibly use the chest menu system?) from this you select a foundation perk, this gives you your beginning ability. Here's some ideas of unarmed abilities:\\r\\nForceful Kick. Ready your feet and deal astounding knock back damage to your foe.\\r\\nBerserk (changed). Ready your fists and enter a rage dealing more damage per hit though hitting slower.\\r\\nKneck Snap. Ready your hands as you approach your target, and if they are weak enough instantly put them to rest.\\r\\nDrunken Slugging. Ready your fists and enter a drunken state where you take less damage and deal a small amount more.\\r\\nClaws. Ready your hands as you tear at your enemy adding the bleed effect to them.\\r\\nLow Blow. Raise your foot and aim low, dazing your opponent.\\r\\n\\r\\nLets say I select Drunken Slugging. I cannot go back on that decision. Then I receive another menu giving me options into which aspect of drunken slugging I should focus on. My options would be:\\r\\nDrunken Resistance. Take less damage\\r\\nDrunken Bravery. Do slightly more damage\\r\\nDrunken belching. When fighting you belch causing a nausea effect to enemies around you.\\r\\n\\r\\nI would then gain more points every 100 levels to improve myself even more. These skills would cap at 1000 giving a varied result. Once completed you can rename your Technique for all to fear. This would be available for axe users and sword users to just different foundations.\\r\\n\\r\\nThank you for reading. Pls criticise / praise. Ask for more if you want it.\\r\\n-Regards CFLPlayer\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:25Z\",\"org\":{\"id\":1429663,\"login\":\"mcMMO-Dev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/mcMMO-Dev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1429663?\"}}\n{\"id\":\"2489395982\",\"type\":\"PushEvent\",\"actor\":{\"id\":624632,\"login\":\"gusennan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gusennan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/624632?\"},\"repo\":{\"id\":28673499,\"name\":\"gusennan/xamarin-forms-samples\",\"url\":\"https://api.github.com/repos/gusennan/xamarin-forms-samples\"},\"payload\":{\"push_id\":536752238,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"aa16d946ccf840fa6a1b57ee7d0fa8c44dad087a\",\"before\":\"9e570dc829e7b3e9eb2e67769edd189cd0c8a723\",\"commits\":[{\"sha\":\"aa16d946ccf840fa6a1b57ee7d0fa8c44dad087a\",\"author\":{\"email\":\"b3a2a659a9d22951edeb5874b5d21f350b75f00e@riseup.net\",\"name\":\"Nate Guerin\"},\"message\":\"revert some formatting changes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gusennan/xamarin-forms-samples/commits/aa16d946ccf840fa6a1b57ee7d0fa8c44dad087a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:25Z\"}\n{\"id\":\"2489395984\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5286446,\"login\":\"otmjka\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/otmjka\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5286446?\"},\"repo\":{\"id\":1195004,\"name\":\"angular/angular-seed\",\"url\":\"https://api.github.com/repos/angular/angular-seed\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:25Z\",\"org\":{\"id\":139426,\"login\":\"angular\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/angular\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/139426?\"}}\n{\"id\":\"2489395985\",\"type\":\"PushEvent\",\"actor\":{\"id\":1432111,\"login\":\"cubiclesoft\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cubiclesoft\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1432111?\"},\"repo\":{\"id\":28677735,\"name\":\"cubiclesoft/barebones-cms-shortcode-bb_syntaxhighlight\",\"url\":\"https://api.github.com/repos/cubiclesoft/barebones-cms-shortcode-bb_syntaxhighlight\"},\"payload\":{\"push_id\":536752240,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"51c7ea91a05fed5f7a7065da857de0fbf1eb6e2e\",\"before\":\"9631619d7d26f9f5aff0641c78cf198ef21f46d0\",\"commits\":[{\"sha\":\"51c7ea91a05fed5f7a7065da857de0fbf1eb6e2e\",\"author\":{\"email\":\"50f3f01caa053693ce619d596e14b0ff3901ab49@cubiclesoft.com\",\"name\":\"cubiclesoft\"},\"message\":\"Added extension ID.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cubiclesoft/barebones-cms-shortcode-bb_syntaxhighlight/commits/51c7ea91a05fed5f7a7065da857de0fbf1eb6e2e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:25Z\"}\n{\"id\":\"2489395992\",\"type\":\"PushEvent\",\"actor\":{\"id\":1356088,\"login\":\"Zaryafaraj\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Zaryafaraj\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1356088?\"},\"repo\":{\"id\":26995510,\"name\":\"Fathalian/Guild\",\"url\":\"https://api.github.com/repos/Fathalian/Guild\"},\"payload\":{\"push_id\":536752245,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"29abf0e942463dd27cf0666ddf54abda099859b2\",\"before\":\"9a773fc648910c7a2499401f44a6e5f71eb30460\",\"commits\":[{\"sha\":\"29abf0e942463dd27cf0666ddf54abda099859b2\",\"author\":{\"email\":\"de8898f6c55e335aa0a2b937fae65fb756ee038f@gmail.com\",\"name\":\"Zaryafaraj\"},\"message\":\"added back change from ali\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Fathalian/Guild/commits/29abf0e942463dd27cf0666ddf54abda099859b2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:26Z\"}\n{\"id\":\"2489395993\",\"type\":\"PushEvent\",\"actor\":{\"id\":1120754,\"login\":\"dobkeratops\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dobkeratops\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1120754?\"},\"repo\":{\"id\":27616619,\"name\":\"dobkeratops/compiler\",\"url\":\"https://api.github.com/repos/dobkeratops/compiler\"},\"payload\":{\"push_id\":536752246,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"aa88601009f892e8890b2f47f2610d62c37e858b\",\"before\":\"9a5138f5617f14feb94326b0673d9779d8ec70b8\",\"commits\":[{\"sha\":\"aa88601009f892e8890b2f47f2610d62c37e858b\",\"author\":{\"email\":\"f8c234822f2256bfec32be392872f65b14509346@gmail.com\",\"name\":\"dobkeratops\"},\"message\":\"adding rvalue references\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dobkeratops/compiler/commits/aa88601009f892e8890b2f47f2610d62c37e858b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:26Z\"}\n{\"id\":\"2489396002\",\"type\":\"PushEvent\",\"actor\":{\"id\":1130906,\"login\":\"eliben\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eliben\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1130906?\"},\"repo\":{\"id\":13198211,\"name\":\"python/cpython\",\"url\":\"https://api.github.com/repos/python/cpython\"},\"payload\":{\"push_id\":536752255,\"size\":6,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"fa28e5ed9fb0bc5bed7fc7921c6638c8082cc3c4\",\"before\":\"461185cc199dac7ce94a8699be66adb4d97288ab\",\"commits\":[{\"sha\":\"44d315ac0e2cc59df9107f0853843ff7b461e32c\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"update for copyright for 2015\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/44d315ac0e2cc59df9107f0853843ff7b461e32c\"},{\"sha\":\"615249dda22c98fdb9c5dfd7724b1e24eb62e71d\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/615249dda22c98fdb9c5dfd7724b1e24eb62e71d\"},{\"sha\":\"8f6109316e19f20ee94e39eef4a24ffd9db6dd10\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.3\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/8f6109316e19f20ee94e39eef4a24ffd9db6dd10\"},{\"sha\":\"db989c1d39c1a201fed41f0f8adacd589ede0a8f\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/python/cpython/commits/db989c1d39c1a201fed41f0f8adacd589ede0a8f\"},{\"sha\":\"6c97ab26377576eaa2f10d451511cf10ba7cde12\",\"author\":{\"email\":\"43f2a2d2c86e22bc80f5acc690d97a5dcf4c4b2f@acm.org\",\"name\":\"Ned Deily\"},\"message\":\"Update copyright dates in OS X installer.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/6c97ab26377576eaa2f10d451511cf10ba7cde12\"},{\"sha\":\"fa28e5ed9fb0bc5bed7fc7921c6638c8082cc3c4\",\"author\":{\"email\":\"43f2a2d2c86e22bc80f5acc690d97a5dcf4c4b2f@acm.org\",\"name\":\"Ned Deily\"},\"message\":\"Update copyright dates in OS X installer.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/python/cpython/commits/fa28e5ed9fb0bc5bed7fc7921c6638c8082cc3c4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:27Z\",\"org\":{\"id\":1525981,\"login\":\"python\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525981?\"}}\n{\"id\":\"2489396009\",\"type\":\"PushEvent\",\"actor\":{\"id\":1297496,\"login\":\"milkmanjack\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/milkmanjack\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1297496?\"},\"repo\":{\"id\":28675501,\"name\":\"milkmanjack/twitch-intermission-testv1\",\"url\":\"https://api.github.com/repos/milkmanjack/twitch-intermission-testv1\"},\"payload\":{\"push_id\":536752260,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"1d30906590d5ab58448dd68c71f0f8ff4852da8c\",\"before\":\"11a03a864e00804d1274b168e5bb07cbbab29502\",\"commits\":[{\"sha\":\"1d30906590d5ab58448dd68c71f0f8ff4852da8c\",\"author\":{\"email\":\"1a6575c57e84915cbf78e3fafdf0c16be650b9d0@gmail.com\",\"name\":\"Curtis Erickson\"},\"message\":\"Update main.js\",\"distinct\":true,\"url\":\"https://api.github.com/repos/milkmanjack/twitch-intermission-testv1/commits/1d30906590d5ab58448dd68c71f0f8ff4852da8c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:28Z\"}\n{\"id\":\"2489396019\",\"type\":\"PushEvent\",\"actor\":{\"id\":1130906,\"login\":\"eliben\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eliben\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1130906?\"},\"repo\":{\"id\":13198211,\"name\":\"python/cpython\",\"url\":\"https://api.github.com/repos/python/cpython\"},\"payload\":{\"push_id\":536752266,\"size\":4,\"distinct_size\":0,\"ref\":\"refs/heads/3.4\",\"head\":\"6c97ab26377576eaa2f10d451511cf10ba7cde12\",\"before\":\"7f6073da37c859f32b3a9bfcb0faa41d24db1230\",\"commits\":[{\"sha\":\"44d315ac0e2cc59df9107f0853843ff7b461e32c\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"update for copyright for 2015\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/44d315ac0e2cc59df9107f0853843ff7b461e32c\"},{\"sha\":\"615249dda22c98fdb9c5dfd7724b1e24eb62e71d\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/615249dda22c98fdb9c5dfd7724b1e24eb62e71d\"},{\"sha\":\"8f6109316e19f20ee94e39eef4a24ffd9db6dd10\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.3\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/8f6109316e19f20ee94e39eef4a24ffd9db6dd10\"},{\"sha\":\"6c97ab26377576eaa2f10d451511cf10ba7cde12\",\"author\":{\"email\":\"43f2a2d2c86e22bc80f5acc690d97a5dcf4c4b2f@acm.org\",\"name\":\"Ned Deily\"},\"message\":\"Update copyright dates in OS X installer.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/6c97ab26377576eaa2f10d451511cf10ba7cde12\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:28Z\",\"org\":{\"id\":1525981,\"login\":\"python\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525981?\"}}\n{\"id\":\"2489396023\",\"type\":\"PushEvent\",\"actor\":{\"id\":5225396,\"login\":\"zzzTNTzzz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zzzTNTzzz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5225396?\"},\"repo\":{\"id\":28316263,\"name\":\"zzzTNTzzz/TNT\",\"url\":\"https://api.github.com/repos/zzzTNTzzz/TNT\"},\"payload\":{\"push_id\":536752267,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f4d5d7cef16304ef8787c96a9808a401c7578eb4\",\"before\":\"9c700d0a24416ab735b1b45bce5de2531da02b4c\",\"commits\":[{\"sha\":\"f4d5d7cef16304ef8787c96a9808a401c7578eb4\",\"author\":{\"email\":\"c20044e993b98218b0a612f57dae8d33d3a1a159@live.com\",\"name\":\"Tony\"},\"message\":\"small edits / removed fx for large mult\",\"distinct\":true,\"url\":\"https://api.github.com/repos/zzzTNTzzz/TNT/commits/f4d5d7cef16304ef8787c96a9808a401c7578eb4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:29Z\"}\n{\"id\":\"2489396028\",\"type\":\"PushEvent\",\"actor\":{\"id\":1130906,\"login\":\"eliben\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eliben\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1130906?\"},\"repo\":{\"id\":13198211,\"name\":\"python/cpython\",\"url\":\"https://api.github.com/repos/python/cpython\"},\"payload\":{\"push_id\":536752268,\"size\":2,\"distinct_size\":0,\"ref\":\"refs/heads/3.3\",\"head\":\"615249dda22c98fdb9c5dfd7724b1e24eb62e71d\",\"before\":\"c917dcb370e4f785f4d5eabfb406bfd5949ce84b\",\"commits\":[{\"sha\":\"44d315ac0e2cc59df9107f0853843ff7b461e32c\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"update for copyright for 2015\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/44d315ac0e2cc59df9107f0853843ff7b461e32c\"},{\"sha\":\"615249dda22c98fdb9c5dfd7724b1e24eb62e71d\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"merge 3.2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/615249dda22c98fdb9c5dfd7724b1e24eb62e71d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\",\"org\":{\"id\":1525981,\"login\":\"python\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525981?\"}}\n{\"id\":\"2489396030\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1540132,\"login\":\"carltonwhitehead\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carltonwhitehead\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1540132?\"},\"repo\":{\"id\":28649714,\"name\":\"carltonwhitehead/coner\",\"url\":\"https://api.github.com/repos/carltonwhitehead/coner\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/2\",\"labels_url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/2/events\",\"html_url\":\"https://github.com/carltonwhitehead/coner/issues/2\",\"id\":53190006,\"number\":2,\"title\":\"Integrate checkstyle\",\"user\":{\"login\":\"carltonwhitehead\",\"id\":1540132,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1540132?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carltonwhitehead\",\"html_url\":\"https://github.com/carltonwhitehead\",\"followers_url\":\"https://api.github.com/users/carltonwhitehead/followers\",\"following_url\":\"https://api.github.com/users/carltonwhitehead/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carltonwhitehead/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carltonwhitehead/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carltonwhitehead/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carltonwhitehead/orgs\",\"repos_url\":\"https://api.github.com/users/carltonwhitehead/repos\",\"events_url\":\"https://api.github.com/users/carltonwhitehead/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carltonwhitehead/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"jshort\",\"id\":1186444,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1186444?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jshort\",\"html_url\":\"https://github.com/jshort\",\"followers_url\":\"https://api.github.com/users/jshort/followers\",\"following_url\":\"https://api.github.com/users/jshort/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jshort/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jshort/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jshort/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jshort/orgs\",\"repos_url\":\"https://api.github.com/users/jshort/repos\",\"events_url\":\"https://api.github.com/users/jshort/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jshort/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":4,\"created_at\":\"2014-12-31T16:01:06Z\",\"updated_at\":\"2015-01-01T01:00:30Z\",\"closed_at\":null,\"body\":\"\"},\"comment\":{\"url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/comments/68477218\",\"html_url\":\"https://github.com/carltonwhitehead/coner/issues/2#issuecomment-68477218\",\"issue_url\":\"https://api.github.com/repos/carltonwhitehead/coner/issues/2\",\"id\":68477218,\"user\":{\"login\":\"carltonwhitehead\",\"id\":1540132,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1540132?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carltonwhitehead\",\"html_url\":\"https://github.com/carltonwhitehead\",\"followers_url\":\"https://api.github.com/users/carltonwhitehead/followers\",\"following_url\":\"https://api.github.com/users/carltonwhitehead/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carltonwhitehead/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carltonwhitehead/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carltonwhitehead/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carltonwhitehead/orgs\",\"repos_url\":\"https://api.github.com/users/carltonwhitehead/repos\",\"events_url\":\"https://api.github.com/users/carltonwhitehead/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carltonwhitehead/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:30Z\",\"updated_at\":\"2015-01-01T01:00:30Z\",\"body\":\"I agree we should have good javadocs. I'm happy to flag those as failures\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\"}\n{\"id\":\"2489396036\",\"type\":\"PushEvent\",\"actor\":{\"id\":1130906,\"login\":\"eliben\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eliben\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1130906?\"},\"repo\":{\"id\":13198211,\"name\":\"python/cpython\",\"url\":\"https://api.github.com/repos/python/cpython\"},\"payload\":{\"push_id\":536752269,\"size\":1,\"distinct_size\":0,\"ref\":\"refs/heads/3.2\",\"head\":\"44d315ac0e2cc59df9107f0853843ff7b461e32c\",\"before\":\"acc6002f0c33d1a29555e4a52322c7a20901ffe2\",\"commits\":[{\"sha\":\"44d315ac0e2cc59df9107f0853843ff7b461e32c\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"update for copyright for 2015\",\"distinct\":false,\"url\":\"https://api.github.com/repos/python/cpython/commits/44d315ac0e2cc59df9107f0853843ff7b461e32c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\",\"org\":{\"id\":1525981,\"login\":\"python\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525981?\"}}\n{\"id\":\"2489396037\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536752271,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9ceb5cdffe02cef186b379026ce6c2dc0d74b3d4\",\"before\":\"a51c727e5f7d975cad627cddcbafb200a4fb5967\",\"commits\":[{\"sha\":\"9ceb5cdffe02cef186b379026ce6c2dc0d74b3d4\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074028174\\n\\n8KwesSwq0wt6hgDqwgssdjZHjXiw3zHu47um9GjGlvY=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/9ceb5cdffe02cef186b379026ce6c2dc0d74b3d4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\"}\n{\"id\":\"2489396038\",\"type\":\"PushEvent\",\"actor\":{\"id\":2645146,\"login\":\"daniel-beard\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/daniel-beard\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2645146?\"},\"repo\":{\"id\":28644199,\"name\":\"daniel-beard/JuliaProjectEuler\",\"url\":\"https://api.github.com/repos/daniel-beard/JuliaProjectEuler\"},\"payload\":{\"push_id\":536752272,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7881c902dceed5e101e6e42e96a792fefc0a9830\",\"before\":\"e8f26d280d6ef9080a4d4dad3c3bde3bafaa3fab\",\"commits\":[{\"sha\":\"7881c902dceed5e101e6e42e96a792fefc0a9830\",\"author\":{\"email\":\"734c834e28fd1bd905b71456871f8daf85204b1a@groupon.com\",\"name\":\"Daniel Beard\"},\"message\":\"Solution to p25, fix p2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/daniel-beard/JuliaProjectEuler/commits/7881c902dceed5e101e6e42e96a792fefc0a9830\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\"}\n{\"id\":\"2489396039\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1995168,\"login\":\"bigtunacan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bigtunacan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1995168?\"},\"repo\":{\"id\":28678198,\"name\":\"bigtunacan/pages\",\"url\":\"https://api.github.com/repos/bigtunacan/pages\"},\"payload\":{\"ref\":\"gh-pages\",\"ref_type\":\"branch\",\"master_branch\":\"gh-pages\",\"description\":\"Jekyll Pages Blog\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\"}\n{\"id\":\"2489396050\",\"type\":\"PushEvent\",\"actor\":{\"id\":4181161,\"login\":\"colin350\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/colin350\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4181161?\"},\"repo\":{\"id\":28090675,\"name\":\"colin350/AMPlug\",\"url\":\"https://api.github.com/repos/colin350/AMPlug\"},\"payload\":{\"push_id\":536752284,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"68f22e3bc1dcf3f067f744a049977257c95cb45a\",\"before\":\"2213bf636d7fe6ecf0d0a35aa8bc9bf3b4907c77\",\"commits\":[{\"sha\":\"68f22e3bc1dcf3f067f744a049977257c95cb45a\",\"author\":{\"email\":\"ca987d55a775ffdf8dd75c235e509848e0c30c0d@gmail.com\",\"name\":\"colin350\"},\"message\":\"Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/colin350/AMPlug/commits/68f22e3bc1dcf3f067f744a049977257c95cb45a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\"}\n{\"id\":\"2489396053\",\"type\":\"PushEvent\",\"actor\":{\"id\":532183,\"login\":\"herry13\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/herry13\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/532183?\"},\"repo\":{\"id\":26089827,\"name\":\"nurilabs/nuri-lang\",\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang\"},\"payload\":{\"push_id\":536752286,\"size\":5,\"distinct_size\":5,\"ref\":\"refs/heads/refactor-type\",\"head\":\"47ed22c2d952b6ce942a136189afe1de1b62c809\",\"before\":\"4ae6e32a27fd491b5cc698fa26ef106bfa34b707\",\"commits\":[{\"sha\":\"85b8ab769d6535156dcdd39b75fe2eaf56de1b84\",\"author\":{\"email\":\"26abebc618a7c031a4cf36426687793911eeb0c3@gmail.com\",\"name\":\"Herry\"},\"message\":\"Add forward-type reference-index (T_Forward T_RefIndex).\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang/commits/85b8ab769d6535156dcdd39b75fe2eaf56de1b84\"},{\"sha\":\"42a9572c1bb841dd7e3c7daf51eb9a2e62bf113b\",\"author\":{\"email\":\"26abebc618a7c031a4cf36426687793911eeb0c3@gmail.com\",\"name\":\"Herry\"},\"message\":\"Define func 'at': return type of array-element. Implement forward-type of reference-index. Refactor forward-type substitution function (replace_forward_type).\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang/commits/42a9572c1bb841dd7e3c7daf51eb9a2e62bf113b\"},{\"sha\":\"ec2e1143a239fed7dfd56fae987f277de80f411d\",\"author\":{\"email\":\"26abebc618a7c031a4cf36426687793911eeb0c3@gmail.com\",\"name\":\"Herry\"},\"message\":\"Removing func 'nuri_array_at_index', and replacing it with func 'Type.at'.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang/commits/ec2e1143a239fed7dfd56fae987f277de80f411d\"},{\"sha\":\"ae4ce825f60fdd621b47175d39974c158c36ba3d\",\"author\":{\"email\":\"26abebc618a7c031a4cf36426687793911eeb0c3@gmail.com\",\"name\":\"Herry\"},\"message\":\"Refactor 'array1.nuri'. Add files array{2,3}.nuri: test-cases of reference-index in function and forward-reference-index.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang/commits/ae4ce825f60fdd621b47175d39974c158c36ba3d\"},{\"sha\":\"47ed22c2d952b6ce942a136189afe1de1b62c809\",\"author\":{\"email\":\"26abebc618a7c031a4cf36426687793911eeb0c3@gmail.com\",\"name\":\"Herry\"},\"message\":\"Add array{2,3}.nuri to good test-files.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nurilabs/nuri-lang/commits/47ed22c2d952b6ce942a136189afe1de1b62c809\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:31Z\",\"org\":{\"id\":7775038,\"login\":\"nurilabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nurilabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7775038?\"}}\n{\"id\":\"2489396055\",\"type\":\"PushEvent\",\"actor\":{\"id\":1456047,\"login\":\"kyokomi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kyokomi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1456047?\"},\"repo\":{\"id\":25506232,\"name\":\"kyokomi/gomajan\",\"url\":\"https://api.github.com/repos/kyokomi/gomajan\"},\"payload\":{\"push_id\":536752287,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"33d7ed3c98a7cfb19ce924735a7420de0ca57ef7\",\"before\":\"fb389c371934eb30c750195f1bff5e09627860aa\",\"commits\":[{\"sha\":\"33d7ed3c98a7cfb19ce924735a7420de0ca57ef7\",\"author\":{\"email\":\"2ea0042be6760fbcd1e13c4e2076e1e54e82d1d5@gmail.com\",\"name\":\"kyokomi\"},\"message\":\"fix: 混全帯么九, 純全帯么九判定\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kyokomi/gomajan/commits/33d7ed3c98a7cfb19ce924735a7420de0ca57ef7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:32Z\"}\n{\"id\":\"2489396057\",\"type\":\"PushEvent\",\"actor\":{\"id\":1525481,\"login\":\"timholy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/timholy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525481?\"},\"repo\":{\"id\":9519622,\"name\":\"timholy/ImageView.jl\",\"url\":\"https://api.github.com/repos/timholy/ImageView.jl\"},\"payload\":{\"push_id\":536752289,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ddf2c3b25177c1cc75a47443cf401c235029c052\",\"before\":\"a666eede913e449bcc3f91ac7a27fe8e67fd99f7\",\"commits\":[{\"sha\":\"ddf2c3b25177c1cc75a47443cf401c235029c052\",\"author\":{\"email\":\"cc3de660a22a4f0e231dde9743ac1c9a7d407254@gmail.com\",\"name\":\"Tim Holy\"},\"message\":\"SubArrays are immutable in julia 0.4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/timholy/ImageView.jl/commits/ddf2c3b25177c1cc75a47443cf401c235029c052\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:32Z\"}\n{\"id\":\"2489396062\",\"type\":\"PushEvent\",\"actor\":{\"id\":1130906,\"login\":\"eliben\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eliben\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1130906?\"},\"repo\":{\"id\":13198211,\"name\":\"python/cpython\",\"url\":\"https://api.github.com/repos/python/cpython\"},\"payload\":{\"push_id\":536752291,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/2.7\",\"head\":\"23eea0071b058be9601a97f44835111a32127c2b\",\"before\":\"bfce62ccfbf18d91f14357851d6dc959cfb66f72\",\"commits\":[{\"sha\":\"9ab0287cda68aeb11567844635573caf0d55bb32\",\"author\":{\"email\":\"fe09bc2ef2737a3258f978e26226dcbac1b3f948@python.org\",\"name\":\"Benjamin Peterson\"},\"message\":\"update for copyright for 2015\",\"distinct\":true,\"url\":\"https://api.github.com/repos/python/cpython/commits/9ab0287cda68aeb11567844635573caf0d55bb32\"},{\"sha\":\"23eea0071b058be9601a97f44835111a32127c2b\",\"author\":{\"email\":\"43f2a2d2c86e22bc80f5acc690d97a5dcf4c4b2f@acm.org\",\"name\":\"Ned Deily\"},\"message\":\"Update copyright dates in OS X installer.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/python/cpython/commits/23eea0071b058be9601a97f44835111a32127c2b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:32Z\",\"org\":{\"id\":1525981,\"login\":\"python\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1525981?\"}}\n{\"id\":\"2489396071\",\"type\":\"PushEvent\",\"actor\":{\"id\":775165,\"login\":\"bingo2011\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bingo2011\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/775165?\"},\"repo\":{\"id\":26621366,\"name\":\"bingo2011/build_my_own_angularjs\",\"url\":\"https://api.github.com/repos/bingo2011/build_my_own_angularjs\"},\"payload\":{\"push_id\":536752295,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"92be8c499b4444d5fa81526c236d786f5d7a72e7\",\"before\":\"51d241ce6e96e416dd85444f19394836c649e20d\",\"commits\":[{\"sha\":\"92be8c499b4444d5fa81526c236d786f5d7a72e7\",\"author\":{\"email\":\"332f63e5fcc114b9df574d567c4a465ad2c0cc59@foxmail.com\",\"name\":\"bingo2013\"},\"message\":\"compile section\",\"distinct\":true,\"url\":\"https://api.github.com/repos/bingo2011/build_my_own_angularjs/commits/92be8c499b4444d5fa81526c236d786f5d7a72e7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:33Z\"}\n{\"id\":\"2489396073\",\"type\":\"PushEvent\",\"actor\":{\"id\":3730551,\"login\":\"rjrobinson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rjrobinson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3730551?\"},\"repo\":{\"id\":28096670,\"name\":\"rjrobinson/js_advanced_fund\",\"url\":\"https://api.github.com/repos/rjrobinson/js_advanced_fund\"},\"payload\":{\"push_id\":536752296,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"90a1821a6b454797e205bb1fe6c25636b1b69af4\",\"before\":\"85877570f9371cf50634c12350baba226706092e\",\"commits\":[{\"sha\":\"90a1821a6b454797e205bb1fe6c25636b1b69af4\",\"author\":{\"email\":\"a000d77e361dc814b63406f3083337032f0ff8f0@gmail.com\",\"name\":\"RJ Robinson\"},\"message\":\"added todo list\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rjrobinson/js_advanced_fund/commits/90a1821a6b454797e205bb1fe6c25636b1b69af4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:33Z\"}\n{\"id\":\"2489396077\",\"type\":\"PushEvent\",\"actor\":{\"id\":4779042,\"login\":\"catbox\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/catbox\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4779042?\"},\"repo\":{\"id\":21256946,\"name\":\"catbox/spring\",\"url\":\"https://api.github.com/repos/catbox/spring\"},\"payload\":{\"push_id\":536752299,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"83bf3f6366f5426aeaea1e08f1f41232110a52fa\",\"before\":\"6078a2bf85fdabb3702235219a5ad0c10d88013c\",\"commits\":[{\"sha\":\"83bf3f6366f5426aeaea1e08f1f41232110a52fa\",\"author\":{\"email\":\"bef99c22f6b2d87df8a9065e8478a3a5ff8b4a6c@gmail.com\",\"name\":\"cocolekat\"},\"message\":\"app not needed\\n\\nSigned-off-by: cocolekat <cocolekat@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/catbox/spring/commits/83bf3f6366f5426aeaea1e08f1f41232110a52fa\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:34Z\"}\n{\"id\":\"2489396084\",\"type\":\"PushEvent\",\"actor\":{\"id\":10225575,\"login\":\"ExclusiveOrange\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ExclusiveOrange\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10225575?\"},\"repo\":{\"id\":28677579,\"name\":\"ExclusiveOrange/synthesizer\",\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer\"},\"payload\":{\"push_id\":536752304,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"before\":\"5e99a85807e01f44fed53f71bba0e0eaa3893084\",\"commits\":[{\"sha\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"author\":{\"email\":\"de3bd7888dcfc4f7d00a4ef606710f57cbba1dbb@hotmail.com\",\"name\":\"ExclusiveOrange\"},\"message\":\"included release .exe\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer/commits/0e64036d5ac0d9c24b4cbe39bb778570db49caba\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396087\",\"type\":\"PushEvent\",\"actor\":{\"id\":1131743,\"login\":\"rchurchley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rchurchley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1131743?\"},\"repo\":{\"id\":11164964,\"name\":\"rchurchley/copernicus\",\"url\":\"https://api.github.com/repos/rchurchley/copernicus\"},\"payload\":{\"push_id\":536752306,\"size\":22,\"distinct_size\":22,\"ref\":\"refs/heads/master\",\"head\":\"9c662abe7ba1c41dd88b82fdd2e5d3db6c58ad5b\",\"before\":\"a8f7305f29a00541d042b1bef59ed103cfb6ee00\",\"commits\":[{\"sha\":\"a69843ef6e07019640823112b0da21e25a85694f\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: remove nanoc-specific items\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/a69843ef6e07019640823112b0da21e25a85694f\"},{\"sha\":\"40ad2a87d890d196d2a7be7c5ae115e5942299e6\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: rename reset.scss\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/40ad2a87d890d196d2a7be7c5ae115e5942299e6\"},{\"sha\":\"4d2eda35659b203eae035775e616e15e6d196e25\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: remove experimental styles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/4d2eda35659b203eae035775e616e15e6d196e25\"},{\"sha\":\"82b76a86f4652e7df9f66d8d1a3469ba88ca711c\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: remove list thumbnails\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/82b76a86f4652e7df9f66d8d1a3469ba88ca711c\"},{\"sha\":\"9520cb0fa5c5a4b023326a1a2b294a06f097ffba\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: make default figure style a class\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/9520cb0fa5c5a4b023326a1a2b294a06f097ffba\"},{\"sha\":\"78c41b207c2ff17d6f3045c42d464b022e933f54\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: tidy up\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/78c41b207c2ff17d6f3045c42d464b022e933f54\"},{\"sha\":\"9b2bec162526dd2eec69b892a67690c51e5ee8b4\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: change nanoc defs to default colours\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/9b2bec162526dd2eec69b892a67690c51e5ee8b4\"},{\"sha\":\"2d61026dd7195c44054d5b9716ec5e86f2b170a8\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: document colour constraints\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/2d61026dd7195c44054d5b9716ec5e86f2b170a8\"},{\"sha\":\"b42debf95608dcf289e6fc6f3b3c5ecda804924d\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: remove unused colour styles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/b42debf95608dcf289e6fc6f3b3c5ecda804924d\"},{\"sha\":\"31be9980e714276088438763326624b2ecbf562c\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: rearrange #site-navigation colour defs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/31be9980e714276088438763326624b2ecbf562c\"},{\"sha\":\"ddeb3eb6fb4336c2308b769faf584c94c0c9872e\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: don't hardcode menu colours\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/ddeb3eb6fb4336c2308b769faf584c94c0c9872e\"},{\"sha\":\"f1c8ce16fd1c8fcb5ad7286a639388d1d3a6a4c9\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Tidy\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/f1c8ce16fd1c8fcb5ad7286a639388d1d3a6a4c9\"},{\"sha\":\"9a11940e31e0bb8d8287a8c31116a67e0da142b6\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Rename files\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/9a11940e31e0bb8d8287a8c31116a67e0da142b6\"},{\"sha\":\"d83ab8c50466b9f8ec0a6844156f144084fdf0bf\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Move scss files to /source\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/d83ab8c50466b9f8ec0a6844156f144084fdf0bf\"},{\"sha\":\"4596b9fd19e0b3f3d45660ee66542d1b4fe9dbde\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: add hooks to add site specific styles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/4596b9fd19e0b3f3d45660ee66542d1b4fe9dbde\"},{\"sha\":\"2116e5956af622c03ab4f520200a02079bdd4d94\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: update README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/2116e5956af622c03ab4f520200a02079bdd4d94\"},{\"sha\":\"fbad248750a9f726da527ce461948bc8bfafac2f\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Generalize: document HTML structure assumed by CSS\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/fbad248750a9f726da527ce461948bc8bfafac2f\"},{\"sha\":\"98c3e6ab4a3823b08a22aa4b80b187460a203271\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Consolidate stylesheets\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/98c3e6ab4a3823b08a22aa4b80b187460a203271\"},{\"sha\":\"b12319ea726684152f0834c476ec1065aedf3162\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Fully separate customizations from code\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/b12319ea726684152f0834c476ec1065aedf3162\"},{\"sha\":\"733a490644efcb99e808ff7ef0fb532c84e71c4b\",\"author\":{\"email\":\"4557ca660bca40225ba58749d212d65c650d01ed@rosschurchley.com\",\"name\":\"Ross Churchley\"},\"message\":\"Documentation style\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchurchley/copernicus/commits/733a490644efcb99e808ff7ef0fb532c84e71c4b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396088\",\"type\":\"PushEvent\",\"actor\":{\"id\":210312,\"login\":\"micahyoung\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/micahyoung\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/210312?\"},\"repo\":{\"id\":25281621,\"name\":\"micahyoung/citibike-data\",\"url\":\"https://api.github.com/repos/micahyoung/citibike-data\"},\"payload\":{\"push_id\":536752307,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2f65fdd10fec0238a57fb81531e3c308dcf78515\",\"before\":\"563bc4bc7f24f05120ff9007ae5e52b3853fcbf9\",\"commits\":[{\"sha\":\"2f65fdd10fec0238a57fb81531e3c308dcf78515\",\"author\":{\"email\":\"45b9372d3d6883e588eb18cca37878d6aa2d5cd5@young.io\",\"name\":\"Micah Young\"},\"message\":\"1420074001\",\"distinct\":true,\"url\":\"https://api.github.com/repos/micahyoung/citibike-data/commits/2f65fdd10fec0238a57fb81531e3c308dcf78515\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396089\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2605378,\"login\":\"fivdi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fivdi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2605378?\"},\"repo\":{\"id\":3880513,\"name\":\"rwaldron/johnny-five\",\"url\":\"https://api.github.com/repos/rwaldron/johnny-five\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/524\",\"labels_url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/524/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/524/comments\",\"events_url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/524/events\",\"html_url\":\"https://github.com/rwaldron/johnny-five/issues/524\",\"id\":51204388,\"number\":524,\"title\":\"beaglebone-io & i2c head scratcher\",\"user\":{\"login\":\"MrYsLab\",\"id\":5189838,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5189838?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MrYsLab\",\"html_url\":\"https://github.com/MrYsLab\",\"followers_url\":\"https://api.github.com/users/MrYsLab/followers\",\"following_url\":\"https://api.github.com/users/MrYsLab/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/MrYsLab/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/MrYsLab/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/MrYsLab/subscriptions\",\"organizations_url\":\"https://api.github.com/users/MrYsLab/orgs\",\"repos_url\":\"https://api.github.com/users/MrYsLab/repos\",\"events_url\":\"https://api.github.com/users/MrYsLab/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/MrYsLab/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":23,\"created_at\":\"2014-12-07T02:10:56Z\",\"updated_at\":\"2015-01-01T01:00:34Z\",\"closed_at\":\"2014-12-30T16:41:06Z\",\"body\":\"I have implemented i2c support for the beaglebone-io plugin using the i2c npm module. I have tested the code with eg/led-matrix-demo.js and it works identically on both Arduino and the BBB. I have also tested a TMP-102 temperature module with a quick and dirty lib module/demo (not yet ready for publishing). Again, both Arduino and BBB work identically and correctly.\\r\\n\\r\\nI am having problems however when i run eg/compass.js. It works fine on my Arduino, but when I run it on the BBB I get one good sample and then it just reports back \\\"data undefined\\\" ad infinitum. I am using a 2 amp external power supply with the BBB so I don't think this is a power issue.\\r\\n\\r\\nHere is my console output:\\r\\n\\r\\ndebian@beaglebone:~$ sudo node compass.js\\r\\n1417880024942 Device(s) BeagleBone-IO \\r\\n1417880024975 Connected BeagleBone-IO \\r\\n1417880024979 Repl Initialized\\r\\n\\r\\nheading 339\\r\\nbearing { name: 'North-NorthWest',\\r\\nabbr: 'NNW',\\r\\nlow: 331.88,\\r\\nmid: 337.5,\\r\\nhigh: 343.12,\\r\\nheading: 339 }\\r\\n(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.\\r\\nTrace\\r\\nat process.EventEmitter.addListener (events.js:160:15)\\r\\nat process.on.process.addListener (node.js:769:26)\\r\\nat new i2c (/home/debian/node_modules/beaglebone-io/node_modules/i2c/lib/i2c.coffee:34:15)\\r\\nat BeagleBone.sendI2CReadRequest (/home/debian/node_modules/beaglebone-io/lib/beaglebone.js:367:14)\\r\\nat Compass. (/home/debian/node_modules/johnny-five/lib/compass.js:91:13)\\r\\nat wrapper as _onTimeout\\r\\nat Timer.listOnTimeout as ontimeout\\r\\ndata undefined\\r\\n\\r\\nAny ideas what I might look at or try?\"},\"comment\":{\"url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/comments/68477220\",\"html_url\":\"https://github.com/rwaldron/johnny-five/issues/524#issuecomment-68477220\",\"issue_url\":\"https://api.github.com/repos/rwaldron/johnny-five/issues/524\",\"id\":68477220,\"user\":{\"login\":\"fivdi\",\"id\":2605378,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2605378?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fivdi\",\"html_url\":\"https://github.com/fivdi\",\"followers_url\":\"https://api.github.com/users/fivdi/followers\",\"following_url\":\"https://api.github.com/users/fivdi/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/fivdi/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/fivdi/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/fivdi/subscriptions\",\"organizations_url\":\"https://api.github.com/users/fivdi/orgs\",\"repos_url\":\"https://api.github.com/users/fivdi/repos\",\"events_url\":\"https://api.github.com/users/fivdi/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/fivdi/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:34Z\",\"updated_at\":\"2015-01-01T01:00:34Z\",\"body\":\"I'd say that's correct.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396094\",\"type\":\"PushEvent\",\"actor\":{\"id\":10144074,\"login\":\"carodew\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carodew\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10144074?\"},\"repo\":{\"id\":27844858,\"name\":\"carodew/carodew.github.io\",\"url\":\"https://api.github.com/repos/carodew/carodew.github.io\"},\"payload\":{\"push_id\":536752311,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8b73f1d4d2ee61797b2d644ff44cda0f9b691aeb\",\"before\":\"513a97293441c42fc14d311e273107115d210e1d\",\"commits\":[{\"sha\":\"8b73f1d4d2ee61797b2d644ff44cda0f9b691aeb\",\"author\":{\"email\":\"6e3c6f0214740e9061d9ca5c79eb6e0ff9cc1741@unknown542696dd77af.gateway.pace.com\",\"name\":\"Carolyn\"},\"message\":\"update styling for base typography\",\"distinct\":true,\"url\":\"https://api.github.com/repos/carodew/carodew.github.io/commits/8b73f1d4d2ee61797b2d644ff44cda0f9b691aeb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396097\",\"type\":\"PushEvent\",\"actor\":{\"id\":909300,\"login\":\"Bike\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bike\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/909300?\"},\"repo\":{\"id\":24974235,\"name\":\"Bike/burke\",\"url\":\"https://api.github.com/repos/Bike/burke\"},\"payload\":{\"push_id\":536752314,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a99f8efe85dd17d4cf79fcf39405f003a4c39ffe\",\"before\":\"c6a01ee88e26326a841c4364b607ff52c656fd48\",\"commits\":[{\"sha\":\"a99f8efe85dd17d4cf79fcf39405f003a4c39ffe\",\"author\":{\"email\":\"a15d8c4f8a46f6b42ad26134f414c80bc96140d6@gmail.com\",\"name\":\"James Kalenius\"},\"message\":\"Make ground environment init functional, & add package objects\\n\\ninitialize_ground is replaced with make_ground, which returns an environment. It\\nalso takes a \\\"package\\\" as an argument. Packages are lisp-accessible objects that\\nhold symbols, which we did globally before this commit. Packages are also c\\naccessible so, well, check main.c if you want to see how I'm envisioning the use\\nAPI at this time.\\n\\nThis change may break a few things, namely read (called from within lisp).\\nHaven't checked.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Bike/burke/commits/a99f8efe85dd17d4cf79fcf39405f003a4c39ffe\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396098\",\"type\":\"PushEvent\",\"actor\":{\"id\":7866761,\"login\":\"calmhorizons\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/calmhorizons\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7866761?\"},\"repo\":{\"id\":28292022,\"name\":\"calmhorizons/innerworlds\",\"url\":\"https://api.github.com/repos/calmhorizons/innerworlds\"},\"payload\":{\"push_id\":536752315,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"24d4bf91156e009c55831b7fe8e030ed6c41c66c\",\"before\":\"63c069a3407c634a81c53021d796d2083b6ac602\",\"commits\":[{\"sha\":\"24d4bf91156e009c55831b7fe8e030ed6c41c66c\",\"author\":{\"email\":\"5d0ccff408c420fe4db31c1ed4e6bb373cdb71a5@internode.on.net\",\"name\":\"calmhorizons\"},\"message\":\"Retired example mod.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/calmhorizons/innerworlds/commits/24d4bf91156e009c55831b7fe8e030ed6c41c66c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396099\",\"type\":\"PushEvent\",\"actor\":{\"id\":7727148,\"login\":\"ThadHouse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ThadHouse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7727148?\"},\"repo\":{\"id\":26708360,\"name\":\"ThadHouse/SplineGenerator\",\"url\":\"https://api.github.com/repos/ThadHouse/SplineGenerator\"},\"payload\":{\"push_id\":536752316,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"8754d22a1254a5a372af8b00928187947e622c60\",\"before\":\"3a313300b8a8694a20be00b74b25763b015f3549\",\"commits\":[{\"sha\":\"9e6bad2c25af5d7d54d364b294933d54e29a0fc2\",\"author\":{\"email\":\"3c6d060800eb1ad0bfd1a00db59ad460de4097ae@athometech.com\",\"name\":\"ThadHouse\"},\"message\":\"accidentally typed something...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ThadHouse/SplineGenerator/commits/9e6bad2c25af5d7d54d364b294933d54e29a0fc2\"},{\"sha\":\"8754d22a1254a5a372af8b00928187947e622c60\",\"author\":{\"email\":\"3c6d060800eb1ad0bfd1a00db59ad460de4097ae@athometech.com\",\"name\":\"ThadHouse\"},\"message\":\"Merge branch 'master' of https://github.com/ThadHouse/SplineGenerator.git\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ThadHouse/SplineGenerator/commits/8754d22a1254a5a372af8b00928187947e622c60\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:35Z\"}\n{\"id\":\"2489396101\",\"type\":\"PushEvent\",\"actor\":{\"id\":210312,\"login\":\"micahyoung\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/micahyoung\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/210312?\"},\"repo\":{\"id\":16091467,\"name\":\"micahyoung/cbstats-data\",\"url\":\"https://api.github.com/repos/micahyoung/cbstats-data\"},\"payload\":{\"push_id\":536752318,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fed920aaffa850292184a7083200488d6c81c84c\",\"before\":\"56100d60a9dcf0137f35638266d574c1500b330c\",\"commits\":[{\"sha\":\"fed920aaffa850292184a7083200488d6c81c84c\",\"author\":{\"email\":\"45b9372d3d6883e588eb18cca37878d6aa2d5cd5@young.io\",\"name\":\"Micah Young\"},\"message\":\"1420074001\",\"distinct\":true,\"url\":\"https://api.github.com/repos/micahyoung/cbstats-data/commits/fed920aaffa850292184a7083200488d6c81c84c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:36Z\"}\n{\"id\":\"2489396118\",\"type\":\"PushEvent\",\"actor\":{\"id\":370793,\"login\":\"Ratmir15\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Ratmir15\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/370793?\"},\"repo\":{\"id\":3652623,\"name\":\"Ratmir15/hz-base\",\"url\":\"https://api.github.com/repos/Ratmir15/hz-base\"},\"payload\":{\"push_id\":536752326,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d94072efcb38808e10382a7c4828e6a191375a4e\",\"before\":\"3fa465de3d29cd3806fadb2fbda9dab0375b1113\",\"commits\":[{\"sha\":\"d94072efcb38808e10382a7c4828e6a191375a4e\",\"author\":{\"email\":\"1de0837f738a2fcf8dd0b85edef8d919b335fdeb@yandex.ru\",\"name\":\"Ratmir\"},\"message\":\"dump\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Ratmir15/hz-base/commits/d94072efcb38808e10382a7c4828e6a191375a4e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:38Z\"}\n{\"id\":\"2489396123\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":865203,\"login\":\"mrjoelkemp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mrjoelkemp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/865203?\"},\"repo\":{\"id\":10697582,\"name\":\"jscs-dev/node-jscs\",\"url\":\"https://api.github.com/repos/jscs-dev/node-jscs\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/869\",\"labels_url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/869/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/869/comments\",\"events_url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/869/events\",\"html_url\":\"https://github.com/jscs-dev/node-jscs/issues/869\",\"id\":53207615,\"number\":869,\"title\":\"Bug? Node v0.11 and modules/checker checkStdin test\",\"user\":{\"login\":\"zxqfox\",\"id\":677518,\"avatar_url\":\"https://avatars.githubusercontent.com/u/677518?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zxqfox\",\"html_url\":\"https://github.com/zxqfox\",\"followers_url\":\"https://api.github.com/users/zxqfox/followers\",\"following_url\":\"https://api.github.com/users/zxqfox/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/zxqfox/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/zxqfox/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/zxqfox/subscriptions\",\"organizations_url\":\"https://api.github.com/users/zxqfox/orgs\",\"repos_url\":\"https://api.github.com/users/zxqfox/repos\",\"events_url\":\"https://api.github.com/users/zxqfox/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/zxqfox/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T23:06:47Z\",\"updated_at\":\"2015-01-01T01:00:38Z\",\"closed_at\":null,\"body\":\"I've used node 0.11 via nvm, ran mocha, and saw:\\r\\n```\\r\\n  1 failing\\r\\n\\r\\n  1) modules/checker checkStdin returns a promise:\\r\\n     TypeError: Attempted to wrap on which is already wrapped\\r\\n      at Object.wrapMethod (/home/alex/repos/node-jscs/node_modules/sinon/lib/sinon/util/core.js:78:25)\\r\\n      at Object.spy (/home/alex/repos/node-jscs/node_modules/sinon/lib/sinon/spy.js:34:26)\\r\\n      at Context.<anonymous> (/home/alex/repos/node-jscs/test/checker.js:62:29)\\r\\n      at callFn (/home/alex/repos/node-jscs/node_modules/mocha/lib/runnable.js:251:21)\\r\\n      at Test.Runnable.run (/home/alex/repos/node-jscs/node_modules/mocha/lib/runnable.js:244:7)\\r\\n      at Runner.runTest (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:374:10)\\r\\n      at /home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:452:12\\r\\n      at next (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:299:14)\\r\\n      at /home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:309:7\\r\\n      at next (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:248:23)\\r\\n      at Immediate._onImmediate (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:276:5)\\r\\n      at processImmediate [as _immediateCallback] (timers.js:374:17)\\r\\n  --------------\\r\\n  Error: Stack Trace for original\\r\\n      at Object.wrapMethod (/home/alex/repos/node-jscs/node_modules/sinon/lib/sinon/util/core.js:98:34)\\r\\n      at Object.spy (/home/alex/repos/node-jscs/node_modules/sinon/lib/sinon/spy.js:34:26)\\r\\n      at Context.<anonymous> (/home/alex/repos/node-jscs/test/checker.js:54:29)\\r\\n      at callFn (/home/alex/repos/node-jscs/node_modules/mocha/lib/runnable.js:251:21)\\r\\n      at Test.Runnable.run (/home/alex/repos/node-jscs/node_modules/mocha/lib/runnable.js:244:7)\\r\\n      at Runner.runTest (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:374:10)\\r\\n      at /home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:452:12\\r\\n      at next (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:299:14)\\r\\n      at /home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:309:7\\r\\n      at next (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:248:23)\\r\\n      at Immediate._onImmediate (/home/alex/repos/node-jscs/node_modules/mocha/lib/runner.js:276:5)\\r\\n      at processImmediate [as _immediateCallback] (timers.js:374:17)\\r\\n```\\r\\n\\r\\n/cc @mrjoelkemp\\r\\n\\r\\n<bountysource-plugin>\\r\\n\\r\\n---\\r\\nWant to back this issue? **[Place a bounty on it!](https://www.bountysource.com/issues/7416038-bug-node-v0-11-and-modules-checker-checkstdin-test?utm_campaign=plugin&utm_content=tracker%2F281640&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F281640&utm_medium=issues&utm_source=github).\\r\\n</bountysource-plugin>\"},\"comment\":{\"url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/comments/68477222\",\"html_url\":\"https://github.com/jscs-dev/node-jscs/issues/869#issuecomment-68477222\",\"issue_url\":\"https://api.github.com/repos/jscs-dev/node-jscs/issues/869\",\"id\":68477222,\"user\":{\"login\":\"mrjoelkemp\",\"id\":865203,\"avatar_url\":\"https://avatars.githubusercontent.com/u/865203?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mrjoelkemp\",\"html_url\":\"https://github.com/mrjoelkemp\",\"followers_url\":\"https://api.github.com/users/mrjoelkemp/followers\",\"following_url\":\"https://api.github.com/users/mrjoelkemp/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/mrjoelkemp/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/mrjoelkemp/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/mrjoelkemp/subscriptions\",\"organizations_url\":\"https://api.github.com/users/mrjoelkemp/orgs\",\"repos_url\":\"https://api.github.com/users/mrjoelkemp/repos\",\"events_url\":\"https://api.github.com/users/mrjoelkemp/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/mrjoelkemp/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:38Z\",\"updated_at\":\"2015-01-01T01:00:38Z\",\"body\":\"Thanks for the report! Seems like it's specific to your node version. I had the same thing happen some time ago with an older version of node. Travis doesn't like your fix likely because it's running a newer stable version of node. \\r\\n\\r\\nTry upgrading locally if you can? \\r\\n\\r\\nHappy new year,  friend :) \"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:39Z\",\"org\":{\"id\":8018201,\"login\":\"jscs-dev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/jscs-dev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8018201?\"}}\n{\"id\":\"2489396126\",\"type\":\"CreateEvent\",\"actor\":{\"id\":9874887,\"login\":\"Akheon23\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Akheon23\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9874887?\"},\"repo\":{\"id\":28678169,\"name\":\"Akheon23/Node.js\",\"url\":\"https://api.github.com/repos/Akheon23/Node.js\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:39Z\"}\n{\"id\":\"2489396130\",\"type\":\"ForkEvent\",\"actor\":{\"id\":9831378,\"login\":\"Amit-P-Amin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?\"},\"repo\":{\"id\":12024210,\"name\":\"appacademy/active_record_lite\",\"url\":\"https://api.github.com/repos/appacademy/active_record_lite\"},\"payload\":{\"forkee\":{\"id\":28678217,\"name\":\"active_record_lite\",\"full_name\":\"Amit-P-Amin/active_record_lite\",\"owner\":{\"login\":\"Amit-P-Amin\",\"id\":9831378,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"html_url\":\"https://github.com/Amit-P-Amin\",\"followers_url\":\"https://api.github.com/users/Amit-P-Amin/followers\",\"following_url\":\"https://api.github.com/users/Amit-P-Amin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Amit-P-Amin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Amit-P-Amin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Amit-P-Amin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Amit-P-Amin/orgs\",\"repos_url\":\"https://api.github.com/users/Amit-P-Amin/repos\",\"events_url\":\"https://api.github.com/users/Amit-P-Amin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Amit-P-Amin/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Amit-P-Amin/active_record_lite\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite\",\"forks_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/forks\",\"keys_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/teams\",\"hooks_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/events\",\"assignees_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/tags\",\"blobs_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/languages\",\"stargazers_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/subscription\",\"commits_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/merges\",\"archive_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/downloads\",\"issues_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/releases{/id}\",\"created_at\":\"2015-01-01T01:00:39Z\",\"updated_at\":\"2014-10-23T23:33:55Z\",\"pushed_at\":\"2014-10-26T23:37:38Z\",\"git_url\":\"git://github.com/Amit-P-Amin/active_record_lite.git\",\"ssh_url\":\"git@github.com:Amit-P-Amin/active_record_lite.git\",\"clone_url\":\"https://github.com/Amit-P-Amin/active_record_lite.git\",\"svn_url\":\"https://github.com/Amit-P-Amin/active_record_lite\",\"homepage\":null,\"size\":632,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:39Z\",\"org\":{\"id\":2138704,\"login\":\"appacademy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/appacademy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2138704?\"}}\n{\"id\":\"2489396135\",\"type\":\"PushEvent\",\"actor\":{\"id\":6462036,\"login\":\"samhillman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/samhillman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6462036?\"},\"repo\":{\"id\":28253700,\"name\":\"samhillman/newdotcom\",\"url\":\"https://api.github.com/repos/samhillman/newdotcom\"},\"payload\":{\"push_id\":536752336,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"eb166d0e9ff48e250e0b322ef42013d02dca159c\",\"before\":\"83eb012480e646f8a84e490bf3700f7441e875a7\",\"commits\":[{\"sha\":\"eb166d0e9ff48e250e0b322ef42013d02dca159c\",\"author\":{\"email\":\"3f12132dd817f39877292097b6071939bf5ccbcd@cvbay.co.uk\",\"name\":\"Sam Hillman\"},\"message\":\"changed css\",\"distinct\":true,\"url\":\"https://api.github.com/repos/samhillman/newdotcom/commits/eb166d0e9ff48e250e0b322ef42013d02dca159c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:40Z\"}\n{\"id\":\"2489396143\",\"type\":\"PushEvent\",\"actor\":{\"id\":20114,\"login\":\"jdillon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jdillon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/20114?\"},\"repo\":{\"id\":10146961,\"name\":\"sonatype/nexus-oss\",\"url\":\"https://api.github.com/repos/sonatype/nexus-oss\"},\"payload\":{\"push_id\":536752341,\"size\":6,\"distinct_size\":6,\"ref\":\"refs/heads/nexus-cma\",\"head\":\"bdb37c4b14d58c8f958f15050517a1ef1bf01c27\",\"before\":\"f0590dba44a66b8a2ac16a74c3d7c387d6c33f90\",\"commits\":[{\"sha\":\"c965763022e63b350354f1b49905bdee35cac22b\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"Fix injected type name\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/c965763022e63b350354f1b49905bdee35cac22b\"},{\"sha\":\"9ae76c75d6fbe07d50d9563f156a909552ff07e9\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"Add start of simple proxy recipe\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/9ae76c75d6fbe07d50d9563f156a909552ff07e9\"},{\"sha\":\"b8faa254870db34b17c660b6f263f6b85e300c81\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"Mark as component, Simplify, update cheetsheet\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/b8faa254870db34b17c660b6f263f6b85e300c81\"},{\"sha\":\"659f782ed5530e2a659f69ed10c2462d5955b8de\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"Simplify names\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/659f782ed5530e2a659f69ed10c2462d5955b8de\"},{\"sha\":\"2c07fa298017131e484db9bce2f91d8a7dc16335\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"debug -> trace\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/2c07fa298017131e484db9bce2f91d8a7dc16335\"},{\"sha\":\"bdb37c4b14d58c8f958f15050517a1ef1bf01c27\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@planet57.com\",\"name\":\"Jason Dillon\"},\"message\":\"blah\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sonatype/nexus-oss/commits/bdb37c4b14d58c8f958f15050517a1ef1bf01c27\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:41Z\",\"org\":{\"id\":44938,\"login\":\"sonatype\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/sonatype\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/44938?\"}}\n{\"id\":\"2489396149\",\"type\":\"ForkEvent\",\"actor\":{\"id\":8484018,\"login\":\"dskae\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dskae\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8484018?\"},\"repo\":{\"id\":26194619,\"name\":\"EKGAPI/webAppEKGAPI\",\"url\":\"https://api.github.com/repos/EKGAPI/webAppEKGAPI\"},\"payload\":{\"forkee\":{\"id\":28678218,\"name\":\"webAppEKGAPI\",\"full_name\":\"dskae/webAppEKGAPI\",\"owner\":{\"login\":\"dskae\",\"id\":8484018,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8484018?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dskae\",\"html_url\":\"https://github.com/dskae\",\"followers_url\":\"https://api.github.com/users/dskae/followers\",\"following_url\":\"https://api.github.com/users/dskae/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dskae/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dskae/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dskae/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dskae/orgs\",\"repos_url\":\"https://api.github.com/users/dskae/repos\",\"events_url\":\"https://api.github.com/users/dskae/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dskae/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/dskae/webAppEKGAPI\",\"description\":\"webAppEKGAPI\",\"fork\":true,\"url\":\"https://api.github.com/repos/dskae/webAppEKGAPI\",\"forks_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/forks\",\"keys_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/teams\",\"hooks_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/hooks\",\"issue_events_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/events\",\"assignees_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/tags\",\"blobs_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/languages\",\"stargazers_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/stargazers\",\"contributors_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/contributors\",\"subscribers_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/subscribers\",\"subscription_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/subscription\",\"commits_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/merges\",\"archive_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/downloads\",\"issues_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/dskae/webAppEKGAPI/releases{/id}\",\"created_at\":\"2015-01-01T01:00:41Z\",\"updated_at\":\"2015-01-01T00:57:42Z\",\"pushed_at\":\"2015-01-01T00:57:42Z\",\"git_url\":\"git://github.com/dskae/webAppEKGAPI.git\",\"ssh_url\":\"git@github.com:dskae/webAppEKGAPI.git\",\"clone_url\":\"https://github.com/dskae/webAppEKGAPI.git\",\"svn_url\":\"https://github.com/dskae/webAppEKGAPI\",\"homepage\":null,\"size\":2776,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:41Z\",\"org\":{\"id\":9016021,\"login\":\"EKGAPI\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/EKGAPI\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9016021?\"}}\n{\"id\":\"2489396153\",\"type\":\"PushEvent\",\"actor\":{\"id\":1221156,\"login\":\"fyfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fyfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1221156?\"},\"repo\":{\"id\":28673837,\"name\":\"fyfe/git-test\",\"url\":\"https://api.github.com/repos/fyfe/git-test\"},\"payload\":{\"push_id\":536752348,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/feature/documentation\",\"head\":\"ff2bf3e6e5ca5f2db8d995fb7e1a982b0d6c54ac\",\"before\":\"5a1391eb7be74927581985100b2274f264a28386\",\"commits\":[{\"sha\":\"ff2bf3e6e5ca5f2db8d995fb7e1a982b0d6c54ac\",\"author\":{\"email\":\"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@neptune-one.net\",\"name\":\"Andrew Fyfe\"},\"message\":\"docs(developer-notes.md): documentation for developers/contributers\",\"distinct\":true,\"url\":\"https://api.github.com/repos/fyfe/git-test/commits/ff2bf3e6e5ca5f2db8d995fb7e1a982b0d6c54ac\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:42Z\"}\n{\"id\":\"2489396163\",\"type\":\"PushEvent\",\"actor\":{\"id\":6737270,\"login\":\"JorgeX\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?\"},\"repo\":{\"id\":25890220,\"name\":\"JorgeX/dojo_rules\",\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\"},\"payload\":{\"push_id\":536752355,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/deadly_skills\",\"head\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"before\":\"fa8f9c257f146f27b1fbdc561e0f5c0b5e50e07c\",\"commits\":[{\"sha\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"author\":{\"email\":\"33f927344e079e00d3fa45d8833b04e735223eec@Jorges-MacBook-Pro.local\",\"name\":\"Jörge Ojanen\"},\"message\":\"ok323\",\"distinct\":true,\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules/commits/923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:43Z\"}\n{\"id\":\"2489396164\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536752356,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8c339f97ba859406034a4b4803ff9f10986c16ea\",\"before\":\"0366b227c1b20d451524d9712bb67771daa739f3\",\"commits\":[{\"sha\":\"8c339f97ba859406034a4b4803ff9f10986c16ea\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/8c339f97ba859406034a4b4803ff9f10986c16ea\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:43Z\"}\n{\"id\":\"2489396174\",\"type\":\"PushEvent\",\"actor\":{\"id\":3990482,\"login\":\"rosbuild\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rosbuild\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3990482?\"},\"repo\":{\"id\":14125815,\"name\":\"osrf/www.ros.org\",\"url\":\"https://api.github.com/repos/osrf/www.ros.org\"},\"payload\":{\"push_id\":536752363,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/wordpressdb\",\"head\":\"e37c3ad58cd26e5d95ba8209561ff1d54f68ff67\",\"before\":\"0b551ea0ff449633f82f355144e7633b6d19380d\",\"commits\":[{\"sha\":\"e37c3ad58cd26e5d95ba8209561ff1d54f68ff67\",\"author\":{\"email\":\"c2678b3531040209f84244ce8534556c3494c8b3@osrfoundation.org\",\"name\":\"Your Name\"},\"message\":\"automatic db update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/osrf/www.ros.org/commits/e37c3ad58cd26e5d95ba8209561ff1d54f68ff67\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:43Z\",\"org\":{\"id\":3999730,\"login\":\"osrf\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/osrf\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3999730?\"}}\n{\"id\":\"2489396182\",\"type\":\"PushEvent\",\"actor\":{\"id\":3013275,\"login\":\"rohatiro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohatiro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3013275?\"},\"repo\":{\"id\":23979538,\"name\":\"rohatiro/apis\",\"url\":\"https://api.github.com/repos/rohatiro/apis\"},\"payload\":{\"push_id\":536752368,\"size\":9,\"distinct_size\":0,\"ref\":\"refs/heads/soundcloud\",\"head\":\"e60a4d532602c4df08a1a019d759ecac297fce89\",\"before\":\"9a0c6b42bb3d3a112784abfe86f937d033e59689\",\"commits\":[{\"sha\":\"a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilizando Passport con el API de Github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\"},{\"sha\":\"05031ae044ff6a66569b6caa61bdea7cfb80f6b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obtención de perfil de usuario en github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/05031ae044ff6a66569b6caa61bdea7cfb80f6b9\"},{\"sha\":\"d64269c3a069b30cc148086ca1d4e450b0cf074c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Manejo de información del usuario\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/d64269c3a069b30cc148086ca1d4e450b0cf074c\"},{\"sha\":\"c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de Bootstrap para estilos de las paginas\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\"},{\"sha\":\"5f746f48b8a9c9ad896e67045d41894fe311d7f3\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obteniendo repositorios de Usuario de Github con request\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5f746f48b8a9c9ad896e67045d41894fe311d7f3\"},{\"sha\":\"604f5ac73f6e928852e5ea727f03bf3c15ab95f0\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Modularizando la funcionalidad de autorización de apis\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/604f5ac73f6e928852e5ea727f03bf3c15ab95f0\"},{\"sha\":\"7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Autorizando usuarios con la api de SoundCloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\"},{\"sha\":\"a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de proxy para obtener streams de tracks de soundcloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\"},{\"sha\":\"e60a4d532602c4df08a1a019d759ecac297fce89\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/e60a4d532602c4df08a1a019d759ecac297fce89\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:44Z\"}\n{\"id\":\"2489396183\",\"type\":\"PushEvent\",\"actor\":{\"id\":3013275,\"login\":\"rohatiro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohatiro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3013275?\"},\"repo\":{\"id\":23979538,\"name\":\"rohatiro/apis\",\"url\":\"https://api.github.com/repos/rohatiro/apis\"},\"payload\":{\"push_id\":536752369,\"size\":125,\"distinct_size\":0,\"ref\":\"refs/heads/temp\",\"head\":\"9192bc90762e7b78ecc1e5ce5c0787cf13436cf8\",\"before\":\"85d2f655fd86f8b77dc07ad12284e1d65542be22\",\"commits\":[{\"sha\":\"a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilizando Passport con el API de Github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\"},{\"sha\":\"05031ae044ff6a66569b6caa61bdea7cfb80f6b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obtención de perfil de usuario en github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/05031ae044ff6a66569b6caa61bdea7cfb80f6b9\"},{\"sha\":\"d64269c3a069b30cc148086ca1d4e450b0cf074c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Manejo de información del usuario\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/d64269c3a069b30cc148086ca1d4e450b0cf074c\"},{\"sha\":\"c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de Bootstrap para estilos de las paginas\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\"},{\"sha\":\"5f746f48b8a9c9ad896e67045d41894fe311d7f3\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obteniendo repositorios de Usuario de Github con request\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5f746f48b8a9c9ad896e67045d41894fe311d7f3\"},{\"sha\":\"604f5ac73f6e928852e5ea727f03bf3c15ab95f0\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Modularizando la funcionalidad de autorización de apis\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/604f5ac73f6e928852e5ea727f03bf3c15ab95f0\"},{\"sha\":\"7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Autorizando usuarios con la api de SoundCloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\"},{\"sha\":\"a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de proxy para obtener streams de tracks de soundcloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\"},{\"sha\":\"e60a4d532602c4df08a1a019d759ecac297fce89\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/e60a4d532602c4df08a1a019d759ecac297fce89\"},{\"sha\":\"41827cebc27d1c20c37f9d1663afff27bf56985c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/41827cebc27d1c20c37f9d1663afff27bf56985c\"},{\"sha\":\"f27ba51a8ca648f180fb95ce336b7d4ed2db6cf2\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/f27ba51a8ca648f180fb95ce336b7d4ed2db6cf2\"},{\"sha\":\"5010b01e82fc526a16402712710dd7ef73d8086f\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5010b01e82fc526a16402712710dd7ef73d8086f\"},{\"sha\":\"7a903d21c5d3199520737de1228307b71e717a95\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/7a903d21c5d3199520737de1228307b71e717a95\"},{\"sha\":\"ffbdb7c9aafdeb531a75b4cf64bdf1339058e8b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/ffbdb7c9aafdeb531a75b4cf64bdf1339058e8b9\"},{\"sha\":\"2c5fc015ed0b0dfcb893dec64f74cddf17e8d0c6\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios en la vista de la pagina inicial de soundcloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/2c5fc015ed0b0dfcb893dec64f74cddf17e8d0c6\"},{\"sha\":\"76068ac452376c96f6feb6f07ff99566efa3a285\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/76068ac452376c96f6feb6f07ff99566efa3a285\"},{\"sha\":\"30c618724a70fe1bf06a5b484345b939b89f15e9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/30c618724a70fe1bf06a5b484345b939b89f15e9\"},{\"sha\":\"9633445f2d373e68703b30f3fd15fd7240d31f6b\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/9633445f2d373e68703b30f3fd15fd7240d31f6b\"},{\"sha\":\"e3328c01916e16f2cb14ee5613dd0b9c04a4c067\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/e3328c01916e16f2cb14ee5613dd0b9c04a4c067\"},{\"sha\":\"f99a48fcedea35168b2e23421745fa2c16d232a4\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Prueba para obtener el waveform\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/f99a48fcedea35168b2e23421745fa2c16d232a4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:44Z\"}\n{\"id\":\"2489396185\",\"type\":\"PushEvent\",\"actor\":{\"id\":3013275,\"login\":\"rohatiro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohatiro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3013275?\"},\"repo\":{\"id\":23979538,\"name\":\"rohatiro/apis\",\"url\":\"https://api.github.com/repos/rohatiro/apis\"},\"payload\":{\"push_id\":536752370,\"size\":126,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"36d1b23ede8251ca74685fad9fdbf219733be4dd\",\"before\":\"3d37eb7613b5559e4eb66c0555ecbd589b25922d\",\"commits\":[{\"sha\":\"a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilizando Passport con el API de Github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\"},{\"sha\":\"05031ae044ff6a66569b6caa61bdea7cfb80f6b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obtención de perfil de usuario en github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/05031ae044ff6a66569b6caa61bdea7cfb80f6b9\"},{\"sha\":\"d64269c3a069b30cc148086ca1d4e450b0cf074c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Manejo de información del usuario\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/d64269c3a069b30cc148086ca1d4e450b0cf074c\"},{\"sha\":\"c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de Bootstrap para estilos de las paginas\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\"},{\"sha\":\"5f746f48b8a9c9ad896e67045d41894fe311d7f3\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obteniendo repositorios de Usuario de Github con request\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5f746f48b8a9c9ad896e67045d41894fe311d7f3\"},{\"sha\":\"604f5ac73f6e928852e5ea727f03bf3c15ab95f0\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Modularizando la funcionalidad de autorización de apis\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/604f5ac73f6e928852e5ea727f03bf3c15ab95f0\"},{\"sha\":\"7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Autorizando usuarios con la api de SoundCloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/7ab8b993d9c7b23a887a9ff035ebeefa5c677baa\"},{\"sha\":\"a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de proxy para obtener streams de tracks de soundcloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a1ed42e4db1832f9c1e0352cf4dd79b62dec2b0d\"},{\"sha\":\"e60a4d532602c4df08a1a019d759ecac297fce89\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/e60a4d532602c4df08a1a019d759ecac297fce89\"},{\"sha\":\"41827cebc27d1c20c37f9d1663afff27bf56985c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/41827cebc27d1c20c37f9d1663afff27bf56985c\"},{\"sha\":\"f27ba51a8ca648f180fb95ce336b7d4ed2db6cf2\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/f27ba51a8ca648f180fb95ce336b7d4ed2db6cf2\"},{\"sha\":\"5010b01e82fc526a16402712710dd7ef73d8086f\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5010b01e82fc526a16402712710dd7ef73d8086f\"},{\"sha\":\"7a903d21c5d3199520737de1228307b71e717a95\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Corrigiendo repositorio para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/7a903d21c5d3199520737de1228307b71e717a95\"},{\"sha\":\"ffbdb7c9aafdeb531a75b4cf64bdf1339058e8b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/ffbdb7c9aafdeb531a75b4cf64bdf1339058e8b9\"},{\"sha\":\"2c5fc015ed0b0dfcb893dec64f74cddf17e8d0c6\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios en la vista de la pagina inicial de soundcloud\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/2c5fc015ed0b0dfcb893dec64f74cddf17e8d0c6\"},{\"sha\":\"76068ac452376c96f6feb6f07ff99566efa3a285\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/76068ac452376c96f6feb6f07ff99566efa3a285\"},{\"sha\":\"30c618724a70fe1bf06a5b484345b939b89f15e9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/30c618724a70fe1bf06a5b484345b939b89f15e9\"},{\"sha\":\"9633445f2d373e68703b30f3fd15fd7240d31f6b\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/9633445f2d373e68703b30f3fd15fd7240d31f6b\"},{\"sha\":\"e3328c01916e16f2cb14ee5613dd0b9c04a4c067\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Cambios para heroku\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/e3328c01916e16f2cb14ee5613dd0b9c04a4c067\"},{\"sha\":\"f99a48fcedea35168b2e23421745fa2c16d232a4\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Prueba para obtener el waveform\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/f99a48fcedea35168b2e23421745fa2c16d232a4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:44Z\"}\n{\"id\":\"2489396186\",\"type\":\"PushEvent\",\"actor\":{\"id\":3013275,\"login\":\"rohatiro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohatiro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3013275?\"},\"repo\":{\"id\":23979538,\"name\":\"rohatiro/apis\",\"url\":\"https://api.github.com/repos/rohatiro/apis\"},\"payload\":{\"push_id\":536752371,\"size\":7,\"distinct_size\":1,\"ref\":\"refs/heads/github\",\"head\":\"0c6236cbbaa52265af1510ec177e26ebef449cd2\",\"before\":\"e9070469477c9228dd12da0a32072b4a0a271912\",\"commits\":[{\"sha\":\"a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilizando Passport con el API de Github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/a0cc0cf97f7c94da62dcfb80a0c71ff1d3e2a43e\"},{\"sha\":\"05031ae044ff6a66569b6caa61bdea7cfb80f6b9\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obtención de perfil de usuario en github\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/05031ae044ff6a66569b6caa61bdea7cfb80f6b9\"},{\"sha\":\"d64269c3a069b30cc148086ca1d4e450b0cf074c\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Manejo de información del usuario\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/d64269c3a069b30cc148086ca1d4e450b0cf074c\"},{\"sha\":\"c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Utilización de Bootstrap para estilos de las paginas\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/c104fa4b165a5a0f98c7fc8c8f1bd4f64d6bbabb\"},{\"sha\":\"5f746f48b8a9c9ad896e67045d41894fe311d7f3\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Obteniendo repositorios de Usuario de Github con request\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/5f746f48b8a9c9ad896e67045d41894fe311d7f3\"},{\"sha\":\"604f5ac73f6e928852e5ea727f03bf3c15ab95f0\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Modularizando la funcionalidad de autorización de apis\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/604f5ac73f6e928852e5ea727f03bf3c15ab95f0\"},{\"sha\":\"0c6236cbbaa52265af1510ec177e26ebef449cd2\",\"author\":{\"email\":\"cf4c21b09c947740db10e2ccb7d10819767cd724@gmail.com\",\"name\":\"Roberto Haziel Tienda Rodríguez\"},\"message\":\"Rediseño del api de github\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rohatiro/apis/commits/0c6236cbbaa52265af1510ec177e26ebef449cd2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:44Z\"}\n{\"id\":\"2489396195\",\"type\":\"PushEvent\",\"actor\":{\"id\":1371300,\"login\":\"hovida\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hovida\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1371300?\"},\"repo\":{\"id\":27575221,\"name\":\"MyChannel-Apps/KFramework\",\"url\":\"https://api.github.com/repos/MyChannel-Apps/KFramework\"},\"payload\":{\"push_id\":536752374,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f88618316a96ef97448b5595dd54c5e3478f69e9\",\"before\":\"bb9b1c88fa505fdf0ad50ab8547141c597127549\",\"commits\":[{\"sha\":\"f88618316a96ef97448b5595dd54c5e3478f69e9\",\"author\":{\"email\":\"59bd0a3ff43b32849b319e645d4798d8a5d1e889@adi-code.de\",\"name\":\"Adrian Preuß\"},\"message\":\"Cronjob: Comment out\",\"distinct\":true,\"url\":\"https://api.github.com/repos/MyChannel-Apps/KFramework/commits/f88618316a96ef97448b5595dd54c5e3478f69e9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:45Z\",\"org\":{\"id\":10083083,\"login\":\"MyChannel-Apps\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MyChannel-Apps\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10083083?\"}}\n{\"id\":\"2489396203\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":18191,\"login\":\"jc00ke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?\"},\"repo\":{\"id\":10488201,\"name\":\"JeanMertz/chruby-fish\",\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14\",\"labels_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/comments\",\"events_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/events\",\"html_url\":\"https://github.com/JeanMertz/chruby-fish/issues/14\",\"id\":52577720,\"number\":14,\"title\":\"\\\"path component may not be valid\\\" warning\",\"user\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-20T21:52:33Z\",\"updated_at\":\"2015-01-01T01:00:45Z\",\"closed_at\":\"2015-01-01T01:00:45Z\",\"body\":\"```\\r\\n$> cd ~/projects/foo/\\r\\nset: Warning: path component /home/jesse/.rubies/ruby-2.1.5/lib/ruby/gems/2.1.0/bin may not be valid in PATH.\\r\\nset: No such file or directory\\r\\n$> cat .ruby-version\\r\\n2.1.5\\r\\n$> chruby --version\\r\\nchruby: 0.3.9\\r\\nchruby-fish: 0.6.0\\r\\n```\\r\\n\\r\\n`chruby-fish` is actuall `HEAD` as of e8f28035e7\\r\\n\\r\\nIdeas on how to suppress the warning? There's definitely no `bin/` in that `path` above.\"},\"comment\":{\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/comments/68477225\",\"html_url\":\"https://github.com/JeanMertz/chruby-fish/issues/14#issuecomment-68477225\",\"issue_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14\",\"id\":68477225,\"user\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:45Z\",\"updated_at\":\"2015-01-01T01:00:45Z\",\"body\":\"Thanks @britishtea, I ended up creating the missing directories. Since multiple people reported this, maybe it's worth mentioning in the README? I'll open a PR for that, but I understand if it won't be merged.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396204\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":18191,\"login\":\"jc00ke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?\"},\"repo\":{\"id\":10488201,\"name\":\"JeanMertz/chruby-fish\",\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14\",\"labels_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/comments\",\"events_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/14/events\",\"html_url\":\"https://github.com/JeanMertz/chruby-fish/issues/14\",\"id\":52577720,\"number\":14,\"title\":\"\\\"path component may not be valid\\\" warning\",\"user\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-20T21:52:33Z\",\"updated_at\":\"2015-01-01T01:00:45Z\",\"closed_at\":\"2015-01-01T01:00:45Z\",\"body\":\"```\\r\\n$> cd ~/projects/foo/\\r\\nset: Warning: path component /home/jesse/.rubies/ruby-2.1.5/lib/ruby/gems/2.1.0/bin may not be valid in PATH.\\r\\nset: No such file or directory\\r\\n$> cat .ruby-version\\r\\n2.1.5\\r\\n$> chruby --version\\r\\nchruby: 0.3.9\\r\\nchruby-fish: 0.6.0\\r\\n```\\r\\n\\r\\n`chruby-fish` is actuall `HEAD` as of e8f28035e7\\r\\n\\r\\nIdeas on how to suppress the warning? There's definitely no `bin/` in that `path` above.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396213\",\"type\":\"PushEvent\",\"actor\":{\"id\":1181205,\"login\":\"malekbr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/malekbr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1181205?\"},\"repo\":{\"id\":28657751,\"name\":\"malekbr/PeevedPenguinsTemplate-Spritebuilder\",\"url\":\"https://api.github.com/repos/malekbr/PeevedPenguinsTemplate-Spritebuilder\"},\"payload\":{\"push_id\":536752379,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3084e78941af25cb8d49f4a9483e17ad41bcfd3e\",\"before\":\"711944c3b36b8c6cc591c507db7c442d70ab02cf\",\"commits\":[{\"sha\":\"3084e78941af25cb8d49f4a9483e17ad41bcfd3e\",\"author\":{\"email\":\"73d3febb659ea118d8f1f9f1ff5f5ae09f3ca6d6@Mings-MacBook-Air.local\",\"name\":\"Malek Ben Romdhane\"},\"message\":\"Automatically Committed for MakeGamesWithUs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/malekbr/PeevedPenguinsTemplate-Spritebuilder/commits/3084e78941af25cb8d49f4a9483e17ad41bcfd3e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396216\",\"type\":\"CreateEvent\",\"actor\":{\"id\":6982503,\"login\":\"josephsands\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/josephsands\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6982503?\"},\"repo\":{\"id\":28678205,\"name\":\"josephsands/josephsands.github.io\",\"url\":\"https://api.github.com/repos/josephsands/josephsands.github.io\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Independent iOS Developer Blog\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396218\",\"type\":\"PushEvent\",\"actor\":{\"id\":954353,\"login\":\"byronmccollum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/byronmccollum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/954353?\"},\"repo\":{\"id\":14756639,\"name\":\"byronmccollum/webscript.io-modules\",\"url\":\"https://api.github.com/repos/byronmccollum/webscript.io-modules\"},\"payload\":{\"push_id\":536752382,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a6c25ed5f1bf15abfa2a8aa67dbcd6e560953296\",\"before\":\"e90f21829951d8cc8d6dd79effefcda4cb7712e2\",\"commits\":[{\"sha\":\"a6c25ed5f1bf15abfa2a8aa67dbcd6e560953296\",\"author\":{\"email\":\"5c33cd3f0e8876e7150963b90c8e5c3e219c1462@rackspace.com\",\"name\":\"Byron McCollum\"},\"message\":\"Create diff.lua\",\"distinct\":true,\"url\":\"https://api.github.com/repos/byronmccollum/webscript.io-modules/commits/a6c25ed5f1bf15abfa2a8aa67dbcd6e560953296\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396220\",\"type\":\"PushEvent\",\"actor\":{\"id\":1167760,\"login\":\"afawcett\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/afawcett\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1167760?\"},\"repo\":{\"id\":28570910,\"name\":\"afawcett/littlebits-connector\",\"url\":\"https://api.github.com/repos/afawcett/littlebits-connector\"},\"payload\":{\"push_id\":536752383,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3f4328e42d2051646d27b9ab14c011d4794c99bd\",\"before\":\"0679908a11f472cc3787d2542850550b3dff2e5d\",\"commits\":[{\"sha\":\"3f4328e42d2051646d27b9ab14c011d4794c99bd\",\"author\":{\"email\":\"e3579b1e47f273529f0f929453e939a68ede9fd1@andyinthecloud.com\",\"name\":\"Andrew Fawcett\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/afawcett/littlebits-connector/commits/3f4328e42d2051646d27b9ab14c011d4794c99bd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396223\",\"type\":\"WatchEvent\",\"actor\":{\"id\":6186720,\"login\":\"NyanKiyoshi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/NyanKiyoshi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6186720?\"},\"repo\":{\"id\":17528018,\"name\":\"erming/shout\",\"url\":\"https://api.github.com/repos/erming/shout\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396225\",\"type\":\"PushEvent\",\"actor\":{\"id\":1909779,\"login\":\"omero\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/omero\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1909779?\"},\"repo\":{\"id\":19432745,\"name\":\"omero/DrupalAppConsole\",\"url\":\"https://api.github.com/repos/omero/DrupalAppConsole\"},\"payload\":{\"push_id\":536752390,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/remove_spaces_enity_class\",\"head\":\"251788240077ded991d18f2bce83e89f1b09c864\",\"before\":\"95d331818e7794dd248c75b6b7fbde933436e9f6\",\"commits\":[{\"sha\":\"251788240077ded991d18f2bce83e89f1b09c864\",\"author\":{\"email\":\"767e3cfe24a214e93dd88e66d7764801877079e9@gmail.com\",\"name\":\"Omar Aguirre Tenorio\"},\"message\":\"#272 Adding askAndValidate function for helper dialog\",\"distinct\":true,\"url\":\"https://api.github.com/repos/omero/DrupalAppConsole/commits/251788240077ded991d18f2bce83e89f1b09c864\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:46Z\"}\n{\"id\":\"2489396229\",\"type\":\"PushEvent\",\"actor\":{\"id\":327833,\"login\":\"dobesv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dobesv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/327833?\"},\"repo\":{\"id\":13713978,\"name\":\"dobesv/functionaljava\",\"url\":\"https://api.github.com/repos/dobesv/functionaljava\"},\"payload\":{\"push_id\":536752391,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"3e1e6018a84070a70f0fe4c3f66b84719b98f6d7\",\"before\":\"c98564f62dabfbeb4d8605f1578808575b18934d\",\"commits\":[{\"sha\":\"e5425b09bc4ad462da121eedde1b2d8fb2d30f72\",\"author\":{\"email\":\"686276df1403a18d9915ffe92f243f413570b0fc@gmail.com\",\"name\":\"Dobes Vandermeer\"},\"message\":\"More NonNull annotations.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dobesv/functionaljava/commits/e5425b09bc4ad462da121eedde1b2d8fb2d30f72\"},{\"sha\":\"3e1e6018a84070a70f0fe4c3f66b84719b98f6d7\",\"author\":{\"email\":\"686276df1403a18d9915ffe92f243f413570b0fc@gmail.com\",\"name\":\"Dobes Vandermeer\"},\"message\":\"More added annotations and helper methods.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dobesv/functionaljava/commits/3e1e6018a84070a70f0fe4c3f66b84719b98f6d7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:47Z\"}\n{\"id\":\"2489396233\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3650755,\"login\":\"scpeters\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scpeters\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3650755?\"},\"repo\":{\"id\":11771697,\"name\":\"osrf/homebrew-simulation\",\"url\":\"https://api.github.com/repos/osrf/homebrew-simulation\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/29\",\"labels_url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/29/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/29/comments\",\"events_url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/29/events\",\"html_url\":\"https://github.com/osrf/homebrew-simulation/issues/29\",\"id\":44603425,\"number\":29,\"title\":\"gazebo failed to build on 10.9.5\",\"user\":{\"login\":\"kdorsel\",\"id\":2569415,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2569415?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kdorsel\",\"html_url\":\"https://github.com/kdorsel\",\"followers_url\":\"https://api.github.com/users/kdorsel/followers\",\"following_url\":\"https://api.github.com/users/kdorsel/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kdorsel/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kdorsel/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kdorsel/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kdorsel/orgs\",\"repos_url\":\"https://api.github.com/users/kdorsel/repos\",\"events_url\":\"https://api.github.com/users/kdorsel/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kdorsel/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-10-01T19:10:39Z\",\"updated_at\":\"2015-01-01T01:00:47Z\",\"closed_at\":null,\"body\":\"Failing to build gazebo while trying to install ROS Hydro.\\r\\n\\r\\nhttps://gist.github.com/anonymous/3d72d247d7126bb44660\"},\"comment\":{\"url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/comments/68477227\",\"html_url\":\"https://github.com/osrf/homebrew-simulation/issues/29#issuecomment-68477227\",\"issue_url\":\"https://api.github.com/repos/osrf/homebrew-simulation/issues/29\",\"id\":68477227,\"user\":{\"login\":\"scpeters\",\"id\":3650755,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3650755?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scpeters\",\"html_url\":\"https://github.com/scpeters\",\"followers_url\":\"https://api.github.com/users/scpeters/followers\",\"following_url\":\"https://api.github.com/users/scpeters/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/scpeters/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/scpeters/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/scpeters/subscriptions\",\"organizations_url\":\"https://api.github.com/users/scpeters/orgs\",\"repos_url\":\"https://api.github.com/users/scpeters/repos\",\"events_url\":\"https://api.github.com/users/scpeters/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/scpeters/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:47Z\",\"updated_at\":\"2015-01-01T01:00:47Z\",\"body\":\"The protobuf issues were resolved in [gazebo pull request 1346](https://bitbucket.org/osrf/gazebo/pull-request/1346/fix-build-for-os-x-1010-1304-1289/diff), though there are now some other issues with boost 1.57 that prevent gazebo from building. I have a fix for them waiting for code review. I'll close this when gazebo is building again.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:47Z\",\"org\":{\"id\":3999730,\"login\":\"osrf\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/osrf\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3999730?\"}}\n{\"id\":\"2489396234\",\"type\":\"ForkEvent\",\"actor\":{\"id\":8252171,\"login\":\"WangXYZ\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/WangXYZ\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8252171?\"},\"repo\":{\"id\":7673434,\"name\":\"blueboy/portaltbc\",\"url\":\"https://api.github.com/repos/blueboy/portaltbc\"},\"payload\":{\"forkee\":{\"id\":28678219,\"name\":\"portaltbc\",\"full_name\":\"WangXYZ/portaltbc\",\"owner\":{\"login\":\"WangXYZ\",\"id\":8252171,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8252171?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/WangXYZ\",\"html_url\":\"https://github.com/WangXYZ\",\"followers_url\":\"https://api.github.com/users/WangXYZ/followers\",\"following_url\":\"https://api.github.com/users/WangXYZ/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/WangXYZ/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/WangXYZ/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/WangXYZ/subscriptions\",\"organizations_url\":\"https://api.github.com/users/WangXYZ/orgs\",\"repos_url\":\"https://api.github.com/users/WangXYZ/repos\",\"events_url\":\"https://api.github.com/users/WangXYZ/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/WangXYZ/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/WangXYZ/portaltbc\",\"description\":\"playerbot support for C(ontinued)-MaNGOS (tbc fork)\",\"fork\":true,\"url\":\"https://api.github.com/repos/WangXYZ/portaltbc\",\"forks_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/forks\",\"keys_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/teams\",\"hooks_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/hooks\",\"issue_events_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/events\",\"assignees_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/tags\",\"blobs_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/languages\",\"stargazers_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/stargazers\",\"contributors_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/contributors\",\"subscribers_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/subscribers\",\"subscription_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/subscription\",\"commits_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/merges\",\"archive_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/downloads\",\"issues_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/WangXYZ/portaltbc/releases{/id}\",\"created_at\":\"2015-01-01T01:00:47Z\",\"updated_at\":\"2014-12-02T15:01:09Z\",\"pushed_at\":\"2014-12-02T15:00:53Z\",\"git_url\":\"git://github.com/WangXYZ/portaltbc.git\",\"ssh_url\":\"git@github.com:WangXYZ/portaltbc.git\",\"clone_url\":\"https://github.com/WangXYZ/portaltbc.git\",\"svn_url\":\"https://github.com/WangXYZ/portaltbc\",\"homepage\":null,\"size\":46000,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:47Z\"}\n{\"id\":\"2489396235\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1414603,\"login\":\"zoldello\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?\"},\"repo\":{\"id\":26617214,\"name\":\"ChicagoVeg/restaurantList\",\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9\",\"labels_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/comments\",\"events_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/issues/9/events\",\"html_url\":\"https://github.com/ChicagoVeg/restaurantList/issues/9\",\"id\":53210171,\"number\":9,\"title\":\"When I filter out restaurants by type, the removed restaurant pins remain in the map\",\"user\":{\"login\":\"zoldello\",\"id\":1414603,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"html_url\":\"https://github.com/zoldello\",\"followers_url\":\"https://api.github.com/users/zoldello/followers\",\"following_url\":\"https://api.github.com/users/zoldello/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/zoldello/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/zoldello/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/zoldello/subscriptions\",\"organizations_url\":\"https://api.github.com/users/zoldello/orgs\",\"repos_url\":\"https://api.github.com/users/zoldello/repos\",\"events_url\":\"https://api.github.com/users/zoldello/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/zoldello/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"},{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/labels/Great+to+Have\",\"name\":\"Great to Have\",\"color\":\"f7c6c7\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"zoldello\",\"id\":1414603,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"html_url\":\"https://github.com/zoldello\",\"followers_url\":\"https://api.github.com/users/zoldello/followers\",\"following_url\":\"https://api.github.com/users/zoldello/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/zoldello/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/zoldello/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/zoldello/subscriptions\",\"organizations_url\":\"https://api.github.com/users/zoldello/orgs\",\"repos_url\":\"https://api.github.com/users/zoldello/repos\",\"events_url\":\"https://api.github.com/users/zoldello/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/zoldello/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/milestones/3\",\"labels_url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/milestones/3/labels\",\"id\":894444,\"number\":3,\"title\":\"Release 1\",\"description\":\"\",\"creator\":{\"login\":\"vadim424\",\"id\":10101875,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10101875?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vadim424\",\"html_url\":\"https://github.com/vadim424\",\"followers_url\":\"https://api.github.com/users/vadim424/followers\",\"following_url\":\"https://api.github.com/users/vadim424/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/vadim424/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/vadim424/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/vadim424/subscriptions\",\"organizations_url\":\"https://api.github.com/users/vadim424/orgs\",\"repos_url\":\"https://api.github.com/users/vadim424/repos\",\"events_url\":\"https://api.github.com/users/vadim424/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/vadim424/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":2,\"closed_issues\":5,\"state\":\"open\",\"created_at\":\"2014-12-07T00:34:16Z\",\"updated_at\":\"2015-01-01T01:00:47Z\",\"due_on\":\"2014-12-14T06:00:00Z\",\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:00:14Z\",\"updated_at\":\"2015-01-01T01:00:47Z\",\"closed_at\":\"2015-01-01T01:00:47Z\",\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:47Z\",\"org\":{\"id\":9426295,\"login\":\"ChicagoVeg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ChicagoVeg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9426295?\"}}\n{\"id\":\"2489396238\",\"type\":\"PushEvent\",\"actor\":{\"id\":1414603,\"login\":\"zoldello\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoldello\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1414603?\"},\"repo\":{\"id\":26617214,\"name\":\"ChicagoVeg/restaurantList\",\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList\"},\"payload\":{\"push_id\":536752394,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dae6d7b4323b0a3102a1decf5b9b0c12f51989a5\",\"before\":\"6549a2eb06d7449819f2da7eba6015c413093315\",\"commits\":[{\"sha\":\"dae6d7b4323b0a3102a1decf5b9b0c12f51989a5\",\"author\":{\"email\":\"444a4929803744c29fad4ab600e75220d121ff93@gmail.com\",\"name\":\"Phil\"},\"message\":\"removed filtered out pins, Fixes #9\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ChicagoVeg/restaurantList/commits/dae6d7b4323b0a3102a1decf5b9b0c12f51989a5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:47Z\",\"org\":{\"id\":9426295,\"login\":\"ChicagoVeg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ChicagoVeg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9426295?\"}}\n{\"id\":\"2489396239\",\"type\":\"PushEvent\",\"actor\":{\"id\":3599988,\"login\":\"wesdizzle\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wesdizzle\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3599988?\"},\"repo\":{\"id\":28250120,\"name\":\"wesdizzle/gagglelog\",\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog\"},\"payload\":{\"push_id\":536752395,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"699893742e393b99eb2250f17b2aad8979d7b454\",\"before\":\"bbd74811366cd3da6de3522bb2a9403b7e778a34\",\"commits\":[{\"sha\":\"699893742e393b99eb2250f17b2aad8979d7b454\",\"author\":{\"email\":\"baaa01a5d45f86e3d8f7008866cf0d37bea55570@gmail.com\",\"name\":\"Wesley Miller\"},\"message\":\"added index values to DistributionMethods for sorting through multiple distribution methods on a single platform of a single type when alphabetical order is not sufficient\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog/commits/699893742e393b99eb2250f17b2aad8979d7b454\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:48Z\"}\n{\"id\":\"2489396248\",\"type\":\"PushEvent\",\"actor\":{\"id\":1681249,\"login\":\"Toeler\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Toeler\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1681249?\"},\"repo\":{\"id\":28678136,\"name\":\"Toeler/Handmade-Hero\",\"url\":\"https://api.github.com/repos/Toeler/Handmade-Hero\"},\"payload\":{\"push_id\":536752397,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"408100c353fa4aa5b211754d5cfc1d83b8c359a3\",\"before\":\"cdf1922395b7f019ed27b3a8ea4022d5e130c8f7\",\"commits\":[{\"sha\":\"d028d9792ca92f20f8a1cde584e1a82d97acbdf6\",\"author\":{\"email\":\"908cd16f96776f758750763d02c45f03b1281e90@gmail.com\",\"name\":\"Toeler\"},\"message\":\"Initial commit of work as at end of ep20 (week 4).\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Toeler/Handmade-Hero/commits/d028d9792ca92f20f8a1cde584e1a82d97acbdf6\"},{\"sha\":\"408100c353fa4aa5b211754d5cfc1d83b8c359a3\",\"author\":{\"email\":\"908cd16f96776f758750763d02c45f03b1281e90@gmail.com\",\"name\":\"Toeler\"},\"message\":\"Merge branch 'master' of https://github.com/Toeler/Handmade-Hero\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Toeler/Handmade-Hero/commits/408100c353fa4aa5b211754d5cfc1d83b8c359a3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:49Z\"}\n{\"id\":\"2489396253\",\"type\":\"PushEvent\",\"actor\":{\"id\":9244168,\"login\":\"gaokuan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gaokuan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9244168?\"},\"repo\":{\"id\":27301392,\"name\":\"208121222/gaokuan\",\"url\":\"https://api.github.com/repos/208121222/gaokuan\"},\"payload\":{\"push_id\":536752398,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/xunni\",\"head\":\"8eced38fc32bbcc27c5e449f3c774a90666034a6\",\"before\":\"64237471e72a59de1744997734962a4df2f9b852\",\"commits\":[{\"sha\":\"8eced38fc32bbcc27c5e449f3c774a90666034a6\",\"author\":{\"email\":\"7ed3207dcf2d842d5b4994c386d23da61b479781@qq.com\",\"name\":\"gaokuan\"},\"message\":\"pinlvji.vhd\\n\\npinlvji.vhd\",\"distinct\":true,\"url\":\"https://api.github.com/repos/208121222/gaokuan/commits/8eced38fc32bbcc27c5e449f3c774a90666034a6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:50Z\",\"org\":{\"id\":9999410,\"login\":\"208121222\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/208121222\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9999410?\"}}\n{\"id\":\"2489396256\",\"type\":\"PushEvent\",\"actor\":{\"id\":5043639,\"login\":\"akbar-sh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/akbar-sh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5043639?\"},\"repo\":{\"id\":28675338,\"name\":\"akbar-sh/tracker\",\"url\":\"https://api.github.com/repos/akbar-sh/tracker\"},\"payload\":{\"push_id\":536752401,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c45885fad735da88cd6dc0fbabd83a647a7c0289\",\"before\":\"182a9b651f97650023f4b8d4a09f13b5e9b8f614\",\"commits\":[{\"sha\":\"c45885fad735da88cd6dc0fbabd83a647a7c0289\",\"author\":{\"email\":\"7eb7e51160615686946d5e19147edf697a6d53a2@MacBook.local\",\"name\":\"Akbar Sharifi\"},\"message\":\"Removing facebook keys.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/akbar-sh/tracker/commits/c45885fad735da88cd6dc0fbabd83a647a7c0289\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:50Z\"}\n{\"id\":\"2489396258\",\"type\":\"PushEvent\",\"actor\":{\"id\":1063076,\"login\":\"Praneeta\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Praneeta\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1063076?\"},\"repo\":{\"id\":28235730,\"name\":\"Praneeta/animation\",\"url\":\"https://api.github.com/repos/Praneeta/animation\"},\"payload\":{\"push_id\":536752403,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/new-year-simple\",\"head\":\"c2486dbe0d16c6cc6a57e27470e475aff4e329f7\",\"before\":\"3d60e5c7b0906f0fdf5f924c6a2a88b6d247e27a\",\"commits\":[{\"sha\":\"e1f605daf044dcc0b529df04a88017715a9d41f6\",\"author\":{\"email\":\"07cf5b0397e51b26de5d8e7005579c15dd1c2956@gmail.com\",\"name\":\"IntroToCoding\"},\"message\":\"more cards\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Praneeta/animation/commits/e1f605daf044dcc0b529df04a88017715a9d41f6\"},{\"sha\":\"c2486dbe0d16c6cc6a57e27470e475aff4e329f7\",\"author\":{\"email\":\"07cf5b0397e51b26de5d8e7005579c15dd1c2956@gmail.com\",\"name\":\"IntroToCoding\"},\"message\":\"Merge branch 'new-year-simple' of https://github.com/Praneeta/animation into new-year-simple\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Praneeta/animation/commits/c2486dbe0d16c6cc6a57e27470e475aff4e329f7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:50Z\"}\n{\"id\":\"2489396268\",\"type\":\"PushEvent\",\"actor\":{\"id\":2539292,\"login\":\"wmfgerrit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wmfgerrit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2539292?\"},\"repo\":{\"id\":6495889,\"name\":\"wikimedia/mediawiki-extensions-WikimediaMessages\",\"url\":\"https://api.github.com/repos/wikimedia/mediawiki-extensions-WikimediaMessages\"},\"payload\":{\"push_id\":536752404,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"61ec5c645adae968e353a778c4f7e82a86ad3468\",\"before\":\"cdd4f8d557f206f88f06fb458ea07fa9359f0d15\",\"commits\":[{\"sha\":\"b41f0ccd50df420206b77bb2c359a1d61202655a\",\"author\":{\"email\":\"f2a16c7d37923d357c1a0ea372900e4a57158c3c@gmail.com\",\"name\":\"Kunal Mehta\"},\"message\":\"Add missing wikibase-sitelinks-sitename-wikidatawiki message\\n\\nChange-Id: I1edd3a55f4c8240a2317828adcb9eedccbe904f4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wikimedia/mediawiki-extensions-WikimediaMessages/commits/b41f0ccd50df420206b77bb2c359a1d61202655a\"},{\"sha\":\"61ec5c645adae968e353a778c4f7e82a86ad3468\",\"author\":{\"email\":\"61a1c7c885f4fc173424af2875d459c32bdc4fee@gerrit.wikimedia.org\",\"name\":\"jenkins-bot\"},\"message\":\"Merge \\\"Add missing wikibase-sitelinks-sitename-wikidatawiki message\\\"\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wikimedia/mediawiki-extensions-WikimediaMessages/commits/61ec5c645adae968e353a778c4f7e82a86ad3468\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:51Z\",\"org\":{\"id\":56668,\"login\":\"wikimedia\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wikimedia\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/56668?\"}}\n{\"id\":\"2489396273\",\"type\":\"PushEvent\",\"actor\":{\"id\":5869772,\"login\":\"felixonmars-bot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/felixonmars-bot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5869772?\"},\"repo\":{\"id\":14887549,\"name\":\"felixonmars/community-mirror\",\"url\":\"https://api.github.com/repos/felixonmars/community-mirror\"},\"payload\":{\"push_id\":536752407,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"b60e989186fb5c092e27586953095c3c4c20c113\",\"before\":\"09aa1bafd2e0d0fb066d1223e749425bd73dc640\",\"commits\":[{\"sha\":\"d2ffde669aca182b3f737a4e3369d81d69bfdc10\",\"author\":{\"email\":\"e841b73a36e3b1b8d38130b3ade2600d4fac5ad3@9fca08f4-af9d-4005-b8df-a31f2cc04f65\",\"name\":\"fyan\"},\"message\":\"upgpkg: python-sh 1.11-1\\n\\nupstream new release\\n\\n\\ngit-svn-id: file:///srv/repos/svn-community/svn@125087 9fca08f4-af9d-4005-b8df-a31f2cc04f65\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/community-mirror/commits/d2ffde669aca182b3f737a4e3369d81d69bfdc10\"},{\"sha\":\"b60e989186fb5c092e27586953095c3c4c20c113\",\"author\":{\"email\":\"e841b73a36e3b1b8d38130b3ade2600d4fac5ad3@9fca08f4-af9d-4005-b8df-a31f2cc04f65\",\"name\":\"fyan\"},\"message\":\"archrelease: copy trunk to community-any\\n\\ngit-svn-id: file:///srv/repos/svn-community/svn@125088 9fca08f4-af9d-4005-b8df-a31f2cc04f65\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/community-mirror/commits/b60e989186fb5c092e27586953095c3c4c20c113\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:52Z\"}\n{\"id\":\"2489396274\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":2791237,\"login\":\"lauriegao\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lauriegao\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2791237?\"},\"repo\":{\"id\":7064308,\"name\":\"jisaacks/GitGutter\",\"url\":\"https://api.github.com/repos/jisaacks/GitGutter\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/jisaacks/GitGutter/issues/207\",\"labels_url\":\"https://api.github.com/repos/jisaacks/GitGutter/issues/207/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jisaacks/GitGutter/issues/207/comments\",\"events_url\":\"https://api.github.com/repos/jisaacks/GitGutter/issues/207/events\",\"html_url\":\"https://github.com/jisaacks/GitGutter/issues/207\",\"id\":53210180,\"number\":207,\"title\":\"Gitgutter icons not showing after updating to MAC OS X Yosemite\",\"user\":{\"login\":\"lauriegao\",\"id\":2791237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2791237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lauriegao\",\"html_url\":\"https://github.com/lauriegao\",\"followers_url\":\"https://api.github.com/users/lauriegao/followers\",\"following_url\":\"https://api.github.com/users/lauriegao/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/lauriegao/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/lauriegao/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/lauriegao/subscriptions\",\"organizations_url\":\"https://api.github.com/users/lauriegao/orgs\",\"repos_url\":\"https://api.github.com/users/lauriegao/repos\",\"events_url\":\"https://api.github.com/users/lauriegao/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/lauriegao/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:00:52Z\",\"updated_at\":\"2015-01-01T01:00:52Z\",\"closed_at\":null,\"body\":\"My Gitgutter icons won't display after updating to Yosemite recently.\\r\\n\\r\\nI am using Sublme Text 2.  I tried uninstalling and reinstalling Gitgutter through Package Control and Git with no luck. \\r\\n\\r\\nFound this repeated three times in console every time I save changes.\\r\\n\\r\\n```\\r\\nTraceback (most recent call last):\\r\\n  File \\\"./sublime_plugin.py\\\", line 339, in run_\\r\\n  File \\\"./git_gutter.py\\\", line 33, in run\\r\\n  File \\\"./view_collection.py\\\", line 50, in diff\\r\\n  File \\\"./git_gutter_handler.py\\\", line 156, in diff\\r\\n  File \\\"./git_gutter_handler.py\\\", line 248, in run_command\\r\\n  File \\\"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py\\\", line 623, in __init__\\r\\n    errread, errwrite)\\r\\n  File \\\"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py\\\", line 1141, in _execute_child\\r\\n    raise child_exception\\r\\nOSError: [Errno 13] Permission denied\\r\\n```\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:52Z\"}\n{\"id\":\"2489396276\",\"type\":\"PushEvent\",\"actor\":{\"id\":6298185,\"login\":\"salrodgom\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/salrodgom\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6298185?\"},\"repo\":{\"id\":28646555,\"name\":\"salrodgom/MC-MD_hybrid_cycles\",\"url\":\"https://api.github.com/repos/salrodgom/MC-MD_hybrid_cycles\"},\"payload\":{\"push_id\":536752408,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a762ae78e5c09457dcced64e3a6f6138906759f3\",\"before\":\"dad880c5428f67cec42c7bc4a054befee370856a\",\"commits\":[{\"sha\":\"a762ae78e5c09457dcced64e3a6f6138906759f3\",\"author\":{\"email\":\"db47db2091bca5e23c026e50e0b9d2e8e7a8e001@upo.es\",\"name\":\"Salvador R. G. Balestra\"},\"message\":\"\\trenamed:    objsMC/RHO_i43m_C1.cif -> objsMC/RHO_i43m_80.cif\\n\\tnew file:   objsMC/RHO_i43m_92.cif\\n\\tnew file:   objsMC/RHO_im3m_80.cif\\n\\tnew file:   objsMC/RHO_im3m_92.cif\",\"distinct\":true,\"url\":\"https://api.github.com/repos/salrodgom/MC-MD_hybrid_cycles/commits/a762ae78e5c09457dcced64e3a6f6138906759f3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:52Z\"}\n{\"id\":\"2489396277\",\"type\":\"PushEvent\",\"actor\":{\"id\":10343396,\"login\":\"sosostris\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sosostris\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10343396?\"},\"repo\":{\"id\":28607481,\"name\":\"sosostris/javapetmarket\",\"url\":\"https://api.github.com/repos/sosostris/javapetmarket\"},\"payload\":{\"push_id\":536752410,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4048ae0f8434f2ac9a5c4550f3b8252059e83594\",\"before\":\"53bf643993375c731a60a6fa0c15b5ee043ef00a\",\"commits\":[{\"sha\":\"4048ae0f8434f2ac9a5c4550f3b8252059e83594\",\"author\":{\"email\":\"4dde01cb51e645a1a51fda7beda48cb0ddc0314e@zhenhuaxu-ltm.internal.salesforce.com\",\"name\":\"Zhenhua Xu\"},\"message\":\"PetsVSAnantou\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sosostris/javapetmarket/commits/4048ae0f8434f2ac9a5c4550f3b8252059e83594\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:52Z\"}\n{\"id\":\"2489396280\",\"type\":\"PushEvent\",\"actor\":{\"id\":1301018,\"login\":\"Harinlen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Harinlen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1301018?\"},\"repo\":{\"id\":22275454,\"name\":\"Kreogist/Mu\",\"url\":\"https://api.github.com/repos/Kreogist/Mu\"},\"payload\":{\"push_id\":536752412,\"size\":53,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3f141a0111ec1b68690d6f23fd96abd42cfa1d87\",\"before\":\"60056c7f8ff214177a84bda7bc7fee8bb7cecc48\",\"commits\":[{\"sha\":\"064d1e3f9e31cd232468a7c81a0f94271536de21\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.21\\n\\nNew Features:\\n1. Add experimental ttplayer lyrics downloader, but cannot use.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/064d1e3f9e31cd232468a7c81a0f94271536de21\"},{\"sha\":\"2d484f08c3886d0e0c379403b91067c083deda61\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.22\\n\\nNew Features:\\n1. Support download lyrics from TTPlayer server.\\n2. Add main player base file.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/2d484f08c3886d0e0c379403b91067c083deda61\"},{\"sha\":\"f9f4b8e12d9256fe132b8df61e45162725b2f4f1\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.23\\n\\nNew Features:\\n1. Support download lyrics from TTPod server.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/f9f4b8e12d9256fe132b8df61e45162725b2f4f1\"},{\"sha\":\"d5d1f8463ede80378b4f9cf334bd892c4be9f97b\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.24\\n\\nNew Features:\\n1. Add SAO Style sub menu.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/d5d1f8463ede80378b4f9cf334bd892c4be9f97b\"},{\"sha\":\"6d054cad0737d6776bbf634c022b10749f10d846\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.25\\n\\nNew Features:\\n1. Support show in actions in append menu.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/6d054cad0737d6776bbf634c022b10749f10d846\"},{\"sha\":\"32f25a9a70c3b322afeb0fdecffae0194c049c11\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.26\\n\\nNew Features:\\n1. Support better album shadow.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/32f25a9a70c3b322afeb0fdecffae0194c049c11\"},{\"sha\":\"f51ddf0213a95c037aa02023625d856da4893212\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.27\\n\\nBug Fixed:\\n1. Fixed the Windows menu indicator display bug.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/f51ddf0213a95c037aa02023625d856da4893212\"},{\"sha\":\"100f2969eefac4ce8558c239ae43c155f58d87f5\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.28\\n\\nNew Features:\\n1. Tweak the WIN32 single property.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/100f2969eefac4ce8558c239ae43c155f58d87f5\"},{\"sha\":\"5ec46f5972932993d0fdb9afa1a6983e0ef55f5d\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.29\\n\\nUpdates:\\n1. Update the music kind to fixed string.\\n2. Update the number and path item value sync policy.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/5ec46f5972932993d0fdb9afa1a6983e0ef55f5d\"},{\"sha\":\"01ea187618722bfaaf4dce1defa2d9cc3c01f384\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.30\\nWARNING:\\nThis commit can be ONLY compile on Linux, do not try on Windows or Mac OS X.\\nNew Features:\\n1. Change the library dir on Linux to /.kreogist/mu/library.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/01ea187618722bfaaf4dce1defa2d9cc3c01f384\"},{\"sha\":\"aecb03334f03845ee09b43a657110fd3d416513f\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.30\\nWARNING:\\nThis commit can be ONLY compiled under Linux, don't try it on Windows or Mac OS X.\\nNew Features:\\n1. Update Linux user path.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/aecb03334f03845ee09b43a657110fd3d416513f\"},{\"sha\":\"cf842e5cba099c1795de7e48bddf32505c83a87c\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.31\\n\\nWARNING:\\nThis commit can ONLY be compiled under Windows and Linux, don't try it\\non Mac OS X.\\nUpdates:\\n1. Tweak the Windows document dir.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/cf842e5cba099c1795de7e48bddf32505c83a87c\"},{\"sha\":\"b12c6bfd0de1600d34a4c32ded9275169e72fe2a\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.29\\n\\nUpdates:\\n1. Now shadow can automatically set their position via the increase\\nparameters.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/b12c6bfd0de1600d34a4c32ded9275169e72fe2a\"},{\"sha\":\"ce01605b171063699e908fa40b085cae3d77193a\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.32\\n\\nMerge:\\n1. Apply the merge.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/ce01605b171063699e908fa40b085cae3d77193a\"},{\"sha\":\"9d3d4009e60161217944fe2afe8fd62a3d7edb0f\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.33\\n\\nNew Features:\\n1. Now locale manager can set the directory position.\\n2. Tweak the Mac OS X folder.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/9d3d4009e60161217944fe2afe8fd62a3d7edb0f\"},{\"sha\":\"0d368ca3131cf6cc754d38509dc674dbea86b587\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.34\\n\\nNew Features:\\n1. Add the rename function for single file.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/0d368ca3131cf6cc754d38509dc674dbea86b587\"},{\"sha\":\"5d4940624bc23f14c83518e06c5673d359b6ccfb\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.35\\n\\nNew Features:\\n1. Experimental support KNMessageBox question static function.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/5d4940624bc23f14c83518e06c5673d359b6ccfb\"},{\"sha\":\"210278fa178920b30bd7ecefa1faf1b9e9369027\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.36\\n\\nNew Features:\\n1. Initial add rename function.\\n2. Change the drag class initial to QScopePointer.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/210278fa178920b30bd7ecefa1faf1b9e9369027\"},{\"sha\":\"fba3bd8198c4e7cf26885059982ae754400fe03e\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.37\\n\\nBug Fixed:\\n1. Fixed the database won't change the file name bug.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/fba3bd8198c4e7cf26885059982ae754400fe03e\"},{\"sha\":\"823ec79cb522f564c43b35f6f7d830143b431b21\",\"author\":{\"email\":\"59ee19a92dc49eba15e3c296949ec051e29e0063@126.com\",\"name\":\"Saki\"},\"message\":\"Major: 0.4.38\\n\\nUpdates:\\n1. Replace the unavailable characters to '_' to avoid the name bug under\\nWindows.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kreogist/Mu/commits/823ec79cb522f564c43b35f6f7d830143b431b21\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:53Z\",\"org\":{\"id\":5064132,\"login\":\"Kreogist\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Kreogist\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5064132?\"}}\n{\"id\":\"2489396281\",\"type\":\"PushEvent\",\"actor\":{\"id\":1253444,\"login\":\"nathan-osman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathan-osman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1253444?\"},\"repo\":{\"id\":28151446,\"name\":\"nathan-osman/django-archive\",\"url\":\"https://api.github.com/repos/nathan-osman/django-archive\"},\"payload\":{\"push_id\":536752413,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ad9166116ced36397fca45582a93ab5d188df18f\",\"before\":\"4e90d1956c0596dd0e2f546d2456c3ab2d0b749a\",\"commits\":[{\"sha\":\"ad9166116ced36397fca45582a93ab5d188df18f\",\"author\":{\"email\":\"2e8aa918660411855c6d44d5bb2da677aa033255@quickmediasolutions.com\",\"name\":\"Nathan Osman\"},\"message\":\"Once again fixed the module paths in setup.py.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nathan-osman/django-archive/commits/ad9166116ced36397fca45582a93ab5d188df18f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:53Z\"}\n{\"id\":\"2489396285\",\"type\":\"PushEvent\",\"actor\":{\"id\":236741,\"login\":\"bryankennedy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bryankennedy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/236741?\"},\"repo\":{\"id\":28671124,\"name\":\"scimusmn/chondrite\",\"url\":\"https://api.github.com/repos/scimusmn/chondrite\"},\"payload\":{\"push_id\":536752416,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"41a6c1af77a62f1f29c162ee4544b7094545fc2a\",\"before\":\"723a5c3a1daadb296857b0c59df586b59f664d7c\",\"commits\":[{\"sha\":\"41a6c1af77a62f1f29c162ee4544b7094545fc2a\",\"author\":{\"email\":\"a2bc4ee92ccb7070799160edd9f39024dd8797ad@smm.org\",\"name\":\"bryan kennedy\"},\"message\":\"Adding credits to the readme\",\"distinct\":true,\"url\":\"https://api.github.com/repos/scimusmn/chondrite/commits/41a6c1af77a62f1f29c162ee4544b7094545fc2a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:54Z\",\"org\":{\"id\":777830,\"login\":\"scimusmn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/scimusmn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/777830?\"}}\n{\"id\":\"2489396287\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":8890114,\"login\":\"speeldoos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/speeldoos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8890114?\"},\"repo\":{\"id\":28678093,\"name\":\"speeldoos/hello-world\",\"url\":\"https://api.github.com/repos/speeldoos/hello-world\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/speeldoos/hello-world/issues/1\",\"labels_url\":\"https://api.github.com/repos/speeldoos/hello-world/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/speeldoos/hello-world/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/speeldoos/hello-world/issues/1/events\",\"html_url\":\"https://github.com/speeldoos/hello-world/issues/1\",\"id\":53210181,\"number\":1,\"title\":\"Finish README\",\"user\":{\"login\":\"speeldoos\",\"id\":8890114,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8890114?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/speeldoos\",\"html_url\":\"https://github.com/speeldoos\",\"followers_url\":\"https://api.github.com/users/speeldoos/followers\",\"following_url\":\"https://api.github.com/users/speeldoos/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/speeldoos/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/speeldoos/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/speeldoos/subscriptions\",\"organizations_url\":\"https://api.github.com/users/speeldoos/orgs\",\"repos_url\":\"https://api.github.com/users/speeldoos/repos\",\"events_url\":\"https://api.github.com/users/speeldoos/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/speeldoos/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:00:54Z\",\"updated_at\":\"2015-01-01T01:00:54Z\",\"closed_at\":null,\"body\":\"So I can finish this tutorial ASAP!\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:55Z\"}\n{\"id\":\"2489396288\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":253237,\"login\":\"Jamesking56\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?\"},\"repo\":{\"id\":26730195,\"name\":\"cachethq/Cachet\",\"url\":\"https://api.github.com/repos/cachethq/Cachet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"labels_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/comments\",\"events_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/events\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173\",\"id\":53210024,\"number\":173,\"title\":\"Bug: Forms let you submit multiple times\",\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2015-01-01T00:52:06Z\",\"updated_at\":\"2015-01-01T01:00:54Z\",\"closed_at\":null,\"body\":\"When adding a new incident, I noticed a weird bug.\\r\\n\\r\\nIf you fill in the form as normal, then click the submit button twice really quickly, it'll create __TWO__ identical new incidents!\\r\\n\\r\\nThis could be a bit annoying, a simple fix is using a bit of JS that on submit, disables the submit button so that once clicked, it cannot be clicked again.\"},\"comment\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/comments/68477229\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173#issuecomment-68477229\",\"issue_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"id\":68477229,\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:54Z\",\"updated_at\":\"2015-01-01T01:00:54Z\",\"body\":\"How do you suppose it could be fixed in Laravel?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:55Z\",\"org\":{\"id\":9951502,\"login\":\"cachethq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cachethq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9951502?\"}}\n{\"id\":\"2489396289\",\"type\":\"CommitCommentEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/comments/9131359\",\"html_url\":\"https://github.com/rust-lang/rust/commit/8f98078b1508c7ebd488c4ab864a4d2fb02d44f8#commitcomment-9131359\",\"id\":9131359,\"user\":{\"login\":\"bors\",\"id\":3372342,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"html_url\":\"https://github.com/bors\",\"followers_url\":\"https://api.github.com/users/bors/followers\",\"following_url\":\"https://api.github.com/users/bors/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bors/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bors/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bors/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bors/orgs\",\"repos_url\":\"https://api.github.com/users/bors/repos\",\"events_url\":\"https://api.github.com/users/bors/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bors/received_events\",\"type\":\"User\",\"site_admin\":false},\"position\":null,\"line\":null,\"path\":null,\"commit_id\":\"8f98078b1508c7ebd488c4ab864a4d2fb02d44f8\",\"created_at\":\"2015-01-01T01:00:54Z\",\"updated_at\":\"2015-01-01T01:00:54Z\",\"body\":\"saw approval from nmatsakis\\nat https://github.com/japaric/rust/commit/8f98078b1508c7ebd488c4ab864a4d2fb02d44f8\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:54Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396293\",\"type\":\"ForkEvent\",\"actor\":{\"id\":3245033,\"login\":\"ZheYuan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ZheYuan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3245033?\"},\"repo\":{\"id\":18537678,\"name\":\"FiloSottile/Heartbleed\",\"url\":\"https://api.github.com/repos/FiloSottile/Heartbleed\"},\"payload\":{\"forkee\":{\"id\":28678220,\"name\":\"Heartbleed\",\"full_name\":\"ZheYuan/Heartbleed\",\"owner\":{\"login\":\"ZheYuan\",\"id\":3245033,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3245033?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ZheYuan\",\"html_url\":\"https://github.com/ZheYuan\",\"followers_url\":\"https://api.github.com/users/ZheYuan/followers\",\"following_url\":\"https://api.github.com/users/ZheYuan/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ZheYuan/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ZheYuan/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ZheYuan/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ZheYuan/orgs\",\"repos_url\":\"https://api.github.com/users/ZheYuan/repos\",\"events_url\":\"https://api.github.com/users/ZheYuan/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ZheYuan/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/ZheYuan/Heartbleed\",\"description\":\"A checker (site and tool) for CVE-2014-0160\",\"fork\":true,\"url\":\"https://api.github.com/repos/ZheYuan/Heartbleed\",\"forks_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/forks\",\"keys_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/teams\",\"hooks_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/hooks\",\"issue_events_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/events\",\"assignees_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/tags\",\"blobs_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/languages\",\"stargazers_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/stargazers\",\"contributors_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/contributors\",\"subscribers_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/subscribers\",\"subscription_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/subscription\",\"commits_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/merges\",\"archive_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/downloads\",\"issues_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/ZheYuan/Heartbleed/releases{/id}\",\"created_at\":\"2015-01-01T01:00:55Z\",\"updated_at\":\"2014-12-30T05:34:14Z\",\"pushed_at\":\"2014-10-31T14:10:43Z\",\"git_url\":\"git://github.com/ZheYuan/Heartbleed.git\",\"ssh_url\":\"git@github.com:ZheYuan/Heartbleed.git\",\"clone_url\":\"https://github.com/ZheYuan/Heartbleed.git\",\"svn_url\":\"https://github.com/ZheYuan/Heartbleed\",\"homepage\":\"http://filippo.io/Heartbleed\",\"size\":3139,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396297\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":28179549,\"name\":\"substack/geodetic-to-ecef\",\"url\":\"https://api.github.com/repos/substack/geodetic-to-ecef\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396300\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752420,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9b5858f96f8d101afa8e568dbe8424c32d010166\",\"before\":\"8069b04a69d5729520e64e6c88c262a77407bd75\",\"commits\":[{\"sha\":\"9b5858f96f8d101afa8e568dbe8424c32d010166\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"create gabor\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/9b5858f96f8d101afa8e568dbe8424c32d010166\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396301\",\"type\":\"PushEvent\",\"actor\":{\"id\":66577,\"login\":\"JakeWharton\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JakeWharton\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/66577?\"},\"repo\":{\"id\":5152285,\"name\":\"square/okhttp\",\"url\":\"https://api.github.com/repos/square/okhttp\"},\"payload\":{\"push_id\":536752417,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/jw/websocket-call\",\"head\":\"aa126df6b62dfc1c5b442a8a53ee601ac2579ef7\",\"before\":\"bc4462296aec3798ee5f2be993f851552b0e1215\",\"commits\":[{\"sha\":\"aa126df6b62dfc1c5b442a8a53ee601ac2579ef7\",\"author\":{\"email\":\"9ec2b9d5f2203d75c2b0f7885bd663d9d57a2d20@squareup.com\",\"name\":\"Jake Wharton\"},\"message\":\"Add a web socket call concept for connecting.\\n\\nSimilar to HTTP and Call, the WebSocketCall is a representation of a pending HTTP request and subsequent upgrade to speak web sockets. Upon synchronous execution you are handed a WebSocket instance for synchronous writing and also pass in a WebSocketListener for async callbacks due to reading.\\n\\nThe API changes in this commits also generalize WebSocket such that it's agnostic to being a client or server peer.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/square/okhttp/commits/aa126df6b62dfc1c5b442a8a53ee601ac2579ef7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\",\"org\":{\"id\":82592,\"login\":\"square\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/square\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/82592?\"}}\n{\"id\":\"2489396303\",\"type\":\"PushEvent\",\"actor\":{\"id\":1373703,\"login\":\"team3cord\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/team3cord\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1373703?\"},\"repo\":{\"id\":20268125,\"name\":\"team3cord/mc-dotfiles\",\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles\"},\"payload\":{\"push_id\":536752421,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c274de69b1397e2b2b3a92b5cbba22b87211ffa0\",\"before\":\"57e4c5c28b8fed6b308b24f86432d3676fd8d96e\",\"commits\":[{\"sha\":\"c274de69b1397e2b2b3a92b5cbba22b87211ffa0\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Updated vim configuration\",\"distinct\":true,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/c274de69b1397e2b2b3a92b5cbba22b87211ffa0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396304\",\"type\":\"PushEvent\",\"actor\":{\"id\":6972205,\"login\":\"K-Niu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/K-Niu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6972205?\"},\"repo\":{\"id\":28167043,\"name\":\"K-Niu/K-Niu.github.io\",\"url\":\"https://api.github.com/repos/K-Niu/K-Niu.github.io\"},\"payload\":{\"push_id\":536752423,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f7edf3b95aaa966869a8fa8c38b9ff806578a6ef\",\"before\":\"c62ef93bb13e2d3e3aa1539782d65e3850b08230\",\"commits\":[{\"sha\":\"f7edf3b95aaa966869a8fa8c38b9ff806578a6ef\",\"author\":{\"email\":\"13bbb662ce51dd619e7c17a547e4f7c5b3d9c40d@gmail.com\",\"name\":\"K-Niu\"},\"message\":\"Updated README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/K-Niu/K-Niu.github.io/commits/f7edf3b95aaa966869a8fa8c38b9ff806578a6ef\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396306\",\"type\":\"PushEvent\",\"actor\":{\"id\":9828988,\"login\":\"alexformagio\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexformagio\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9828988?\"},\"repo\":{\"id\":28678021,\"name\":\"alexformagio/python30min\",\"url\":\"https://api.github.com/repos/alexformagio/python30min\"},\"payload\":{\"push_id\":536752424,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6148df2e7babc92c0568cc452deae2b683a041da\",\"before\":\"c20ef006701c56b5440a24d782a2330ddc465619\",\"commits\":[{\"sha\":\"6148df2e7babc92c0568cc452deae2b683a041da\",\"author\":{\"email\":\"eb51c250a6d7f2b7307829ec95405c85965e34b2@gmail.com\",\"name\":\"alex_formagio\"},\"message\":\"adding gitignore\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alexformagio/python30min/commits/6148df2e7babc92c0568cc452deae2b683a041da\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396307\",\"type\":\"PushEvent\",\"actor\":{\"id\":10144074,\"login\":\"carodew\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carodew\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10144074?\"},\"repo\":{\"id\":27844858,\"name\":\"carodew/carodew.github.io\",\"url\":\"https://api.github.com/repos/carodew/carodew.github.io\"},\"payload\":{\"push_id\":536752425,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"19643dcb3388a0ae99570e176fc592e0fd666c81\",\"before\":\"8b73f1d4d2ee61797b2d644ff44cda0f9b691aeb\",\"commits\":[{\"sha\":\"19643dcb3388a0ae99570e176fc592e0fd666c81\",\"author\":{\"email\":\"6e3c6f0214740e9061d9ca5c79eb6e0ff9cc1741@unknown542696dd77af.gateway.pace.com\",\"name\":\"Carolyn\"},\"message\":\"tweak project layout\",\"distinct\":true,\"url\":\"https://api.github.com/repos/carodew/carodew.github.io/commits/19643dcb3388a0ae99570e176fc592e0fd666c81\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\"}\n{\"id\":\"2489396314\",\"type\":\"CommitCommentEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/comments/9131360\",\"html_url\":\"https://github.com/rust-lang/rust/commit/8f98078b1508c7ebd488c4ab864a4d2fb02d44f8#commitcomment-9131360\",\"id\":9131360,\"user\":{\"login\":\"bors\",\"id\":3372342,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"html_url\":\"https://github.com/bors\",\"followers_url\":\"https://api.github.com/users/bors/followers\",\"following_url\":\"https://api.github.com/users/bors/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bors/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bors/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bors/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bors/orgs\",\"repos_url\":\"https://api.github.com/users/bors/repos\",\"events_url\":\"https://api.github.com/users/bors/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bors/received_events\",\"type\":\"User\",\"site_admin\":false},\"position\":null,\"line\":null,\"path\":null,\"commit_id\":\"8f98078b1508c7ebd488c4ab864a4d2fb02d44f8\",\"created_at\":\"2015-01-01T01:00:56Z\",\"updated_at\":\"2015-01-01T01:00:56Z\",\"body\":\"merging japaric/rust/moar-uc = 8f98078b into auto\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:56Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396315\",\"type\":\"PushEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"push_id\":536752427,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/auto\",\"head\":\"10d99a973498c5a1be6ba318210751efc1c2cf61\",\"before\":\"7013291b3c4b85fb9ffe356630d9aa575aa7f35a\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396316\",\"type\":\"PushEvent\",\"actor\":{\"id\":5869772,\"login\":\"felixonmars-bot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/felixonmars-bot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5869772?\"},\"repo\":{\"id\":14148979,\"name\":\"felixonmars/aur-mirror\",\"url\":\"https://api.github.com/repos/felixonmars/aur-mirror\"},\"payload\":{\"push_id\":536752431,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e54aae07fb256cafa088c8fc82f21b2931fa0b12\",\"before\":\"27b0aad8faabf64ecd13ba93090ebe43d2706d55\",\"commits\":[{\"sha\":\"e54aae07fb256cafa088c8fc82f21b2931fa0b12\",\"author\":{\"email\":\"b68e87e72ad86ae664b595742624a5b780191f69@gmail.com\",\"name\":\"Kyle Keen\"},\"message\":\"updated on Thu Jan  1 00:00:32 UTC 2015\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/aur-mirror/commits/e54aae07fb256cafa088c8fc82f21b2931fa0b12\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\"}\n{\"id\":\"2489396319\",\"type\":\"PushEvent\",\"actor\":{\"id\":3828361,\"login\":\"kiahosseini\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kiahosseini\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3828361?\"},\"repo\":{\"id\":27885085,\"name\":\"kiahosseini/kiahosseini.github.io\",\"url\":\"https://api.github.com/repos/kiahosseini/kiahosseini.github.io\"},\"payload\":{\"push_id\":536752434,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"c916f5f5ebe46499985ca552a6e0a1fd74599d54\",\"before\":\"31b0d34e0a62d3b38e348a29a7e89079982080c4\",\"commits\":[{\"sha\":\"ffd765729e0c8362490a5488cec37f072654cfec\",\"author\":{\"email\":\"268618c87e98ee43d06d901feef069e47f43f688@gmail.com\",\"name\":\"Kiarash Hosseini\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kiahosseini/kiahosseini.github.io/commits/ffd765729e0c8362490a5488cec37f072654cfec\"},{\"sha\":\"c916f5f5ebe46499985ca552a6e0a1fd74599d54\",\"author\":{\"email\":\"268618c87e98ee43d06d901feef069e47f43f688@gmail.com\",\"name\":\"Kiarash Hosseini\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kiahosseini/kiahosseini.github.io/commits/c916f5f5ebe46499985ca552a6e0a1fd74599d54\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\"}\n{\"id\":\"2489396320\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Home\",\"title\":\"Home\",\"summary\":null,\"action\":\"edited\",\"sha\":\"425ecdb18436e26cab4dcaa4db8d7b57650a5e27\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Home\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\"}\n{\"id\":\"2489396321\",\"type\":\"PushEvent\",\"actor\":{\"id\":18257,\"login\":\"keisukefukuda\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/keisukefukuda\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18257?\"},\"repo\":{\"id\":4476852,\"name\":\"keisukefukuda/dotfiles\",\"url\":\"https://api.github.com/repos/keisukefukuda/dotfiles\"},\"payload\":{\"push_id\":536752435,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"32f4510f99c1963b3500fafe4a0373160f9290b6\",\"before\":\"5c497bfeed64eb68e6b46645270222111ac51a4e\",\"commits\":[{\"sha\":\"32f4510f99c1963b3500fafe4a0373160f9290b6\",\"author\":{\"email\":\"e20d997393fae6e64f65ca9b73604073336c9900@gmail.com\",\"name\":\"Keisuke Fukuda\"},\"message\":\"commented out unnecessary zsh fpath\",\"distinct\":true,\"url\":\"https://api.github.com/repos/keisukefukuda/dotfiles/commits/32f4510f99c1963b3500fafe4a0373160f9290b6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\"}\n{\"id\":\"2489396322\",\"type\":\"PushEvent\",\"actor\":{\"id\":5631503,\"login\":\"Autodidact24\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Autodidact24\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5631503?\"},\"repo\":{\"id\":21542754,\"name\":\"Autodidact24/autodidact24.github.io\",\"url\":\"https://api.github.com/repos/Autodidact24/autodidact24.github.io\"},\"payload\":{\"push_id\":536752436,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2ee7b23cb5d65a89d79b88734e606fc0a1b2acac\",\"before\":\"8b2eadf9f36df15261e7c2dbd502a0978fadfb82\",\"commits\":[{\"sha\":\"2ee7b23cb5d65a89d79b88734e606fc0a1b2acac\",\"author\":{\"email\":\"7e51ff80808f27ff5907bcc8c43a3f5980edcc6f@gmail.com\",\"name\":\"Shubham Singh Tomar\"},\"message\":\"new post\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Autodidact24/autodidact24.github.io/commits/2ee7b23cb5d65a89d79b88734e606fc0a1b2acac\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\"}\n{\"id\":\"2489396324\",\"type\":\"CommitCommentEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/comments/9131361\",\"html_url\":\"https://github.com/rust-lang/rust/commit/8f98078b1508c7ebd488c4ab864a4d2fb02d44f8#commitcomment-9131361\",\"id\":9131361,\"user\":{\"login\":\"bors\",\"id\":3372342,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"html_url\":\"https://github.com/bors\",\"followers_url\":\"https://api.github.com/users/bors/followers\",\"following_url\":\"https://api.github.com/users/bors/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bors/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bors/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bors/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bors/orgs\",\"repos_url\":\"https://api.github.com/users/bors/repos\",\"events_url\":\"https://api.github.com/users/bors/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bors/received_events\",\"type\":\"User\",\"site_admin\":false},\"position\":null,\"line\":null,\"path\":null,\"commit_id\":\"8f98078b1508c7ebd488c4ab864a4d2fb02d44f8\",\"created_at\":\"2015-01-01T01:00:57Z\",\"updated_at\":\"2015-01-01T01:00:57Z\",\"body\":\"status: {\\\"merge_sha\\\": \\\"a570791cd4118c97b4a14b4a27a273a2f3ed466e\\\"}\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:57Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396329\",\"type\":\"PushEvent\",\"actor\":{\"id\":283089,\"login\":\"openemr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/openemr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/283089?\"},\"repo\":{\"id\":1473462,\"name\":\"openemr/translations_development_openemr\",\"url\":\"https://api.github.com/repos/openemr/translations_development_openemr\"},\"payload\":{\"push_id\":536752438,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6af5103f19520b8aa4560b55c1c5eedb668aea29\",\"before\":\"d7c7b8cc839d9724d49a5f45176ed896226ca068\",\"commits\":[{\"sha\":\"6af5103f19520b8aa4560b55c1c5eedb668aea29\",\"author\":{\"email\":\"cbb210ee4c1aeabb7ba82d51bf63db966bfa0e88@users.sourceforge.net\",\"name\":\"bradymiller\"},\"message\":\"Routine Automated Development Translations Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/openemr/translations_development_openemr/commits/6af5103f19520b8aa4560b55c1c5eedb668aea29\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:58Z\"}\n{\"id\":\"2489396331\",\"type\":\"PushEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"push_id\":536752437,\"size\":24,\"distinct_size\":24,\"ref\":\"refs/heads/auto\",\"head\":\"a570791cd4118c97b4a14b4a27a273a2f3ed466e\",\"before\":\"10d99a973498c5a1be6ba318210751efc1c2cf61\",\"commits\":[{\"sha\":\"ea94a90488e6b4701581079339de3595389e5b15\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"unicode: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/ea94a90488e6b4701581079339de3595389e5b15\"},{\"sha\":\"a17c2b60e1c32e950b011296025a9f88f4d3c4e4\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"collections: fix fallout\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/a17c2b60e1c32e950b011296025a9f88f4d3c4e4\"},{\"sha\":\"44dc9196d3854e43e921800dea5d939b2ca178ce\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"std: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/44dc9196d3854e43e921800dea5d939b2ca178ce\"},{\"sha\":\"bcbe9f2e7ae8cbea9f85100d50c00f3dd492b318\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"syntax: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/bcbe9f2e7ae8cbea9f85100d50c00f3dd492b318\"},{\"sha\":\"3c1b5d3aa23ea16c804c1a7dbfab5ad932efd940\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/3c1b5d3aa23ea16c804c1a7dbfab5ad932efd940\"},{\"sha\":\"c3baa1800105e2cb6e6d58b3f835812e150f50b2\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_trans: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/c3baa1800105e2cb6e6d58b3f835812e150f50b2\"},{\"sha\":\"79b36818e4dc825dd843c2ee500b5969acee6068\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_borrowck: unbox closures used in function arguments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/79b36818e4dc825dd843c2ee500b5969acee6068\"},{\"sha\":\"9168e0f270783ce9e5238f87830c18df63c394a6\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"core: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/9168e0f270783ce9e5238f87830c18df63c394a6\"},{\"sha\":\"e4397075acbb241926f142ed6b54a6d4abc198a9\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"getopts: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/e4397075acbb241926f142ed6b54a6d4abc198a9\"},{\"sha\":\"aaed78886dbf819914c48fde4890f080548adcea\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/aaed78886dbf819914c48fde4890f080548adcea\"},{\"sha\":\"d4bfc5c9cf24f0428124e1257dc631a7d9125088\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_back: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/d4bfc5c9cf24f0428124e1257dc631a7d9125088\"},{\"sha\":\"e0ed73bd1d5308922c7f4a2e60e565e97b06160c\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_borrowck: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/e0ed73bd1d5308922c7f4a2e60e565e97b06160c\"},{\"sha\":\"8c842f8ebf4f67e19e1d7e1e7538212575f10319\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_driver: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/8c842f8ebf4f67e19e1d7e1e7538212575f10319\"},{\"sha\":\"48a1e43b92c46586f1ec77e63eb9c3b72468c00b\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_resolve: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/48a1e43b92c46586f1ec77e63eb9c3b72468c00b\"},{\"sha\":\"5ac73e12e844b575961bf61a10f9b23e48e90c89\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_trans: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/5ac73e12e844b575961bf61a10f9b23e48e90c89\"},{\"sha\":\"450e6793327408f14659b922b9ce0169798d1b07\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustc_typeck: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/450e6793327408f14659b922b9ce0169798d1b07\"},{\"sha\":\"653f372b59f5465fac5f808e02c7dd82da7cdbd7\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"rustdoc: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/653f372b59f5465fac5f808e02c7dd82da7cdbd7\"},{\"sha\":\"0da9d07e2d331a57304809e5cfc26b4b8d938b44\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"std: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/0da9d07e2d331a57304809e5cfc26b4b8d938b44\"},{\"sha\":\"bbc0ce866d5f7db982eefbca6d184e2ce0c6ebe8\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"syntax: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/bbc0ce866d5f7db982eefbca6d184e2ce0c6ebe8\"},{\"sha\":\"26278cea5afd35166415dcaede78c3b9b3fdaeac\",\"author\":{\"email\":\"550518ab9d14bd32155907315f74849d7b5c2b36@gmail.com\",\"name\":\"Jorge Aparicio\"},\"message\":\"time: unbox closures used in let bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rust-lang/rust/commits/26278cea5afd35166415dcaede78c3b9b3fdaeac\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:58Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396334\",\"type\":\"CommitCommentEvent\",\"actor\":{\"id\":3372342,\"login\":\"bors\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/comments/9131362\",\"html_url\":\"https://github.com/rust-lang/rust/commit/8f98078b1508c7ebd488c4ab864a4d2fb02d44f8#commitcomment-9131362\",\"id\":9131362,\"user\":{\"login\":\"bors\",\"id\":3372342,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3372342?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bors\",\"html_url\":\"https://github.com/bors\",\"followers_url\":\"https://api.github.com/users/bors/followers\",\"following_url\":\"https://api.github.com/users/bors/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bors/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bors/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bors/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bors/orgs\",\"repos_url\":\"https://api.github.com/users/bors/repos\",\"events_url\":\"https://api.github.com/users/bors/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bors/received_events\",\"type\":\"User\",\"site_admin\":false},\"position\":null,\"line\":null,\"path\":null,\"commit_id\":\"8f98078b1508c7ebd488c4ab864a4d2fb02d44f8\",\"created_at\":\"2015-01-01T01:00:58Z\",\"updated_at\":\"2015-01-01T01:00:58Z\",\"body\":\"japaric/rust/moar-uc = 8f98078b merged ok, testing candidate = a570791c\"}},\"public\":true,\"created_at\":\"2015-01-01T01:00:58Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489396336\",\"type\":\"PushEvent\",\"actor\":{\"id\":6154548,\"login\":\"chrisanthropic\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chrisanthropic\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6154548?\"},\"repo\":{\"id\":28470344,\"name\":\"chrisanthropic/comical-jekyll-theme\",\"url\":\"https://api.github.com/repos/chrisanthropic/comical-jekyll-theme\"},\"payload\":{\"push_id\":536752440,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"51a3f37c2253a92322a371d8dac0024c94929c77\",\"before\":\"18691cb662324d1431eda0263f67f414d53b4870\",\"commits\":[{\"sha\":\"51a3f37c2253a92322a371d8dac0024c94929c77\",\"author\":{\"email\":\"a9c72f90f6fdaaea784cb28e2da787f981b8fb43@gmail.com\",\"name\":\"Christopher Tarwater\"},\"message\":\"updates\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chrisanthropic/comical-jekyll-theme/commits/51a3f37c2253a92322a371d8dac0024c94929c77\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\"}\n{\"id\":\"2489396337\",\"type\":\"ForkEvent\",\"actor\":{\"id\":7366091,\"login\":\"coryasato\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coryasato\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7366091?\"},\"repo\":{\"id\":28153049,\"name\":\"EKGAPI/KardiaApp\",\"url\":\"https://api.github.com/repos/EKGAPI/KardiaApp\"},\"payload\":{\"forkee\":{\"id\":28678221,\"name\":\"KardiaApp\",\"full_name\":\"coryasato/KardiaApp\",\"owner\":{\"login\":\"coryasato\",\"id\":7366091,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7366091?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coryasato\",\"html_url\":\"https://github.com/coryasato\",\"followers_url\":\"https://api.github.com/users/coryasato/followers\",\"following_url\":\"https://api.github.com/users/coryasato/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coryasato/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coryasato/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coryasato/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coryasato/orgs\",\"repos_url\":\"https://api.github.com/users/coryasato/repos\",\"events_url\":\"https://api.github.com/users/coryasato/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coryasato/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/coryasato/KardiaApp\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/coryasato/KardiaApp\",\"forks_url\":\"https://api.github.com/repos/coryasato/KardiaApp/forks\",\"keys_url\":\"https://api.github.com/repos/coryasato/KardiaApp/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/coryasato/KardiaApp/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/coryasato/KardiaApp/teams\",\"hooks_url\":\"https://api.github.com/repos/coryasato/KardiaApp/hooks\",\"issue_events_url\":\"https://api.github.com/repos/coryasato/KardiaApp/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/coryasato/KardiaApp/events\",\"assignees_url\":\"https://api.github.com/repos/coryasato/KardiaApp/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/coryasato/KardiaApp/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/coryasato/KardiaApp/tags\",\"blobs_url\":\"https://api.github.com/repos/coryasato/KardiaApp/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/coryasato/KardiaApp/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/coryasato/KardiaApp/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/coryasato/KardiaApp/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/coryasato/KardiaApp/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/coryasato/KardiaApp/languages\",\"stargazers_url\":\"https://api.github.com/repos/coryasato/KardiaApp/stargazers\",\"contributors_url\":\"https://api.github.com/repos/coryasato/KardiaApp/contributors\",\"subscribers_url\":\"https://api.github.com/repos/coryasato/KardiaApp/subscribers\",\"subscription_url\":\"https://api.github.com/repos/coryasato/KardiaApp/subscription\",\"commits_url\":\"https://api.github.com/repos/coryasato/KardiaApp/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/coryasato/KardiaApp/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/coryasato/KardiaApp/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/coryasato/KardiaApp/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/coryasato/KardiaApp/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/coryasato/KardiaApp/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/coryasato/KardiaApp/merges\",\"archive_url\":\"https://api.github.com/repos/coryasato/KardiaApp/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/coryasato/KardiaApp/downloads\",\"issues_url\":\"https://api.github.com/repos/coryasato/KardiaApp/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/coryasato/KardiaApp/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/coryasato/KardiaApp/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/coryasato/KardiaApp/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/coryasato/KardiaApp/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/coryasato/KardiaApp/releases{/id}\",\"created_at\":\"2015-01-01T01:00:59Z\",\"updated_at\":\"2014-12-31T23:06:40Z\",\"pushed_at\":\"2014-12-31T23:06:38Z\",\"git_url\":\"git://github.com/coryasato/KardiaApp.git\",\"ssh_url\":\"git@github.com:coryasato/KardiaApp.git\",\"clone_url\":\"https://github.com/coryasato/KardiaApp.git\",\"svn_url\":\"https://github.com/coryasato/KardiaApp\",\"homepage\":null,\"size\":362,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\",\"org\":{\"id\":9016021,\"login\":\"EKGAPI\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/EKGAPI\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9016021?\"}}\n{\"id\":\"2489396340\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1734986,\"login\":\"Adamwgoh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Adamwgoh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1734986?\"},\"repo\":{\"id\":28678222,\"name\":\"Adamwgoh/Laura\",\"url\":\"https://api.github.com/repos/Adamwgoh/Laura\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"G52GRP re-upload\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\"}\n{\"id\":\"2489396341\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1570089,\"login\":\"sirvon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sirvon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1570089?\"},\"repo\":{\"id\":1971346,\"name\":\"msgpack/msgpack-java\",\"url\":\"https://api.github.com/repos/msgpack/msgpack-java\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\",\"org\":{\"id\":198264,\"login\":\"msgpack\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/msgpack\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/198264?\"}}\n{\"id\":\"2489396342\",\"type\":\"PushEvent\",\"actor\":{\"id\":6900864,\"login\":\"abuehab\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/abuehab\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6900864?\"},\"repo\":{\"id\":26771667,\"name\":\"smss123/BylsanSystem\",\"url\":\"https://api.github.com/repos/smss123/BylsanSystem\"},\"payload\":{\"push_id\":536752442,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"fc430dab978b650fcf7d828048be9dfacce2f4b7\",\"before\":\"fc430dab978b650fcf7d828048be9dfacce2f4b7\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\"}\n{\"id\":\"2489396343\",\"type\":\"ForkEvent\",\"actor\":{\"id\":18191,\"login\":\"jc00ke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?\"},\"repo\":{\"id\":10488201,\"name\":\"JeanMertz/chruby-fish\",\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish\"},\"payload\":{\"forkee\":{\"id\":28678223,\"name\":\"chruby-fish\",\"full_name\":\"jc00ke/chruby-fish\",\"owner\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/jc00ke/chruby-fish\",\"description\":\"Thin wrapper around chruby to make it work with the Fish shell\",\"fork\":true,\"url\":\"https://api.github.com/repos/jc00ke/chruby-fish\",\"forks_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/forks\",\"keys_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/teams\",\"hooks_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/hooks\",\"issue_events_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/events\",\"assignees_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/tags\",\"blobs_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/languages\",\"stargazers_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/stargazers\",\"contributors_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/contributors\",\"subscribers_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/subscribers\",\"subscription_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/subscription\",\"commits_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/merges\",\"archive_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/downloads\",\"issues_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/releases{/id}\",\"created_at\":\"2015-01-01T01:00:59Z\",\"updated_at\":\"2015-01-01T01:00:59Z\",\"pushed_at\":\"2014-12-14T22:38:55Z\",\"git_url\":\"git://github.com/jc00ke/chruby-fish.git\",\"ssh_url\":\"git@github.com:jc00ke/chruby-fish.git\",\"clone_url\":\"https://github.com/jc00ke/chruby-fish.git\",\"svn_url\":\"https://github.com/jc00ke/chruby-fish\",\"homepage\":\"\",\"size\":1047,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Shell\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:00:59Z\"}\n{\"id\":\"2489396345\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3318117,\"login\":\"Z0mbine\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Z0mbine\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3318117?\"},\"repo\":{\"id\":15829679,\"name\":\"alexgrist/ServerGuard\",\"url\":\"https://api.github.com/repos/alexgrist/ServerGuard\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/118\",\"labels_url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/118/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/118/comments\",\"events_url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/118/events\",\"html_url\":\"https://github.com/alexgrist/ServerGuard/issues/118\",\"id\":53203105,\"number\":118,\"title\":\"[REQUEST] Chat\",\"user\":{\"login\":\"SkyWalker3200\",\"id\":9349382,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9349382?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SkyWalker3200\",\"html_url\":\"https://github.com/SkyWalker3200\",\"followers_url\":\"https://api.github.com/users/SkyWalker3200/followers\",\"following_url\":\"https://api.github.com/users/SkyWalker3200/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/SkyWalker3200/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/SkyWalker3200/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/SkyWalker3200/subscriptions\",\"organizations_url\":\"https://api.github.com/users/SkyWalker3200/orgs\",\"repos_url\":\"https://api.github.com/users/SkyWalker3200/repos\",\"events_url\":\"https://api.github.com/users/SkyWalker3200/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/SkyWalker3200/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T20:57:23Z\",\"updated_at\":\"2015-01-01T01:00:59Z\",\"closed_at\":null,\"body\":\"Something that was brought up to me by one of my clan members is to have an option of chat prefixes.\\r\\nFor the ranks.\\r\\n\\r\\n[VIP] SkyWalker: Blah blah blah\\r\\n\\r\\nThought it would be a good addon.\\r\\n\\r\\n-SkyWalker\"},\"comment\":{\"url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/comments/68477230\",\"html_url\":\"https://github.com/alexgrist/ServerGuard/issues/118#issuecomment-68477230\",\"issue_url\":\"https://api.github.com/repos/alexgrist/ServerGuard/issues/118\",\"id\":68477230,\"user\":{\"login\":\"Z0mbine\",\"id\":3318117,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3318117?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Z0mbine\",\"html_url\":\"https://github.com/Z0mbine\",\"followers_url\":\"https://api.github.com/users/Z0mbine/followers\",\"following_url\":\"https://api.github.com/users/Z0mbine/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Z0mbine/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Z0mbine/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Z0mbine/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Z0mbine/orgs\",\"repos_url\":\"https://api.github.com/users/Z0mbine/repos\",\"events_url\":\"https://api.github.com/users/Z0mbine/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Z0mbine/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:00:59Z\",\"updated_at\":\"2015-01-01T01:00:59Z\",\"body\":\"Someone already requested this.\\r\\n#51 \"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:00Z\"}\n{\"id\":\"2489396346\",\"type\":\"WatchEvent\",\"actor\":{\"id\":7291605,\"login\":\"jtmancilla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jtmancilla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7291605?\"},\"repo\":{\"id\":12711013,\"name\":\"ropensci/webservices\",\"url\":\"https://api.github.com/repos/ropensci/webservices\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:00Z\",\"org\":{\"id\":1200269,\"login\":\"ropensci\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ropensci\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1200269?\"}}\n{\"id\":\"2489396354\",\"type\":\"PushEvent\",\"actor\":{\"id\":10125388,\"login\":\"vowsentente\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vowsentente\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10125388?\"},\"repo\":{\"id\":28586913,\"name\":\"vowsentente/P1-NYC-MTA-Dataset\",\"url\":\"https://api.github.com/repos/vowsentente/P1-NYC-MTA-Dataset\"},\"payload\":{\"push_id\":536752446,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ff747f0fd326bc2942004bc3cd89f7235d58cb18\",\"before\":\"aed0f46e9f6f914fe3ae2ba78026590f90bef4af\",\"commits\":[{\"sha\":\"ff747f0fd326bc2942004bc3cd89f7235d58cb18\",\"author\":{\"email\":\"5e402b391f4e77b9a75e1e07b1348ffeae489b97@gmail.com\",\"name\":\"Steve\"},\"message\":\"Update Short_Questions.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/vowsentente/P1-NYC-MTA-Dataset/commits/ff747f0fd326bc2942004bc3cd89f7235d58cb18\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:01Z\"}\n{\"id\":\"2489396355\",\"type\":\"PushEvent\",\"actor\":{\"id\":7256509,\"login\":\"ww44ss\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ww44ss\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7256509?\"},\"repo\":{\"id\":28677457,\"name\":\"ww44ss/Titanic\",\"url\":\"https://api.github.com/repos/ww44ss/Titanic\"},\"payload\":{\"push_id\":536752445,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dbd6831a613af879e35527ea486c559c23fd9b4a\",\"before\":\"346e3c7b8a56b797dcbe3d5cfdeea9ff9f2963f8\",\"commits\":[{\"sha\":\"dbd6831a613af879e35527ea486c559c23fd9b4a\",\"author\":{\"email\":\"f3f15001ba38afaee860a89905381c5bde0dd27e@gmail.com\",\"name\":\"Winston Saunders\"},\"message\":\"model1\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ww44ss/Titanic/commits/dbd6831a613af879e35527ea486c559c23fd9b4a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:01Z\"}\n{\"id\":\"2489396356\",\"type\":\"PushEvent\",\"actor\":{\"id\":4167278,\"login\":\"frechei\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frechei\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4167278?\"},\"repo\":{\"id\":9475465,\"name\":\"frechei/frechei.github.com\",\"url\":\"https://api.github.com/repos/frechei/frechei.github.com\"},\"payload\":{\"push_id\":536752447,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"ecc17f6817f60facb356cd2431fcae42aa1adde6\",\"before\":\"2781f52cf83cccff5a44c2cd5e5a31c71507bc5d\",\"commits\":[{\"sha\":\"11498120a07a44ddb0303eec31672e4a3cf7d26f\",\"author\":{\"email\":\"3ba795b10b0bab9eb3b0f6e5eaa6ef981f2e220e@163.com¨\",\"name\":\"¨frechei¨\"},\"message\":\"Octopress init\",\"distinct\":true,\"url\":\"https://api.github.com/repos/frechei/frechei.github.com/commits/11498120a07a44ddb0303eec31672e4a3cf7d26f\"},{\"sha\":\"01faf6dabf6d005d77d2433a509a34017fd5a800\",\"author\":{\"email\":\"3ba795b10b0bab9eb3b0f6e5eaa6ef981f2e220e@163.com¨\",\"name\":\"¨frechei¨\"},\"message\":\"合并冲突\",\"distinct\":true,\"url\":\"https://api.github.com/repos/frechei/frechei.github.com/commits/01faf6dabf6d005d77d2433a509a34017fd5a800\"},{\"sha\":\"ecc17f6817f60facb356cd2431fcae42aa1adde6\",\"author\":{\"email\":\"3ba795b10b0bab9eb3b0f6e5eaa6ef981f2e220e@163.com¨\",\"name\":\"¨frechei¨\"},\"message\":\"Site updated at 2015-01-01 01:00:53 UTC\",\"distinct\":true,\"url\":\"https://api.github.com/repos/frechei/frechei.github.com/commits/ecc17f6817f60facb356cd2431fcae42aa1adde6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:01Z\"}\n{\"id\":\"2489396357\",\"type\":\"PushEvent\",\"actor\":{\"id\":1903079,\"login\":\"krizvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/krizvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1903079?\"},\"repo\":{\"id\":20293030,\"name\":\"VHAINNOVATIONS/Mental-Health-eScreening\",\"url\":\"https://api.github.com/repos/VHAINNOVATIONS/Mental-Health-eScreening\"},\"payload\":{\"push_id\":536752448,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/t651\",\"head\":\"df757ed8ad0c31af27c6099cc0c2d8818cb2ff6b\",\"before\":\"89b68f2e450ba74e169210925a0b02b1b02e2d9f\",\"commits\":[{\"sha\":\"df757ed8ad0c31af27c6099cc0c2d8818cb2ff6b\",\"author\":{\"email\":\"e0c3a02e4102d2b5ed70e51bf9a27df65846d12e@gmail.com\",\"name\":\"Khalid R. Rizvi\"},\"message\":\"t651 fixed json data response by plucking unncessary data and also Lint the section controller\",\"distinct\":true,\"url\":\"https://api.github.com/repos/VHAINNOVATIONS/Mental-Health-eScreening/commits/df757ed8ad0c31af27c6099cc0c2d8818cb2ff6b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:01Z\",\"org\":{\"id\":1252476,\"login\":\"VHAINNOVATIONS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/VHAINNOVATIONS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1252476?\"}}\n{\"id\":\"2489396359\",\"type\":\"PushEvent\",\"actor\":{\"id\":6952185,\"login\":\"krico\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/krico\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6952185?\"},\"repo\":{\"id\":25744696,\"name\":\"krico/jas\",\"url\":\"https://api.github.com/repos/krico/jas\"},\"payload\":{\"push_id\":536752450,\"size\":16,\"distinct_size\":12,\"ref\":\"refs/heads/jas-95\",\"head\":\"08bf4359f7d54b6a488c650b08fe8f31a8e1703a\",\"before\":\"0459dc09acd412b54aba4a18c3463173938fbe02\",\"commits\":[{\"sha\":\"7ad7fa31bafceb692bfad082cd0893a25ab5140a\",\"author\":{\"email\":\"a6066a030a600eca8738f6d597ace912fc74e459@wp.pl\",\"name\":\"wszarmach\"},\"message\":\"jas-83 Complete specification\",\"distinct\":false,\"url\":\"https://api.github.com/repos/krico/jas/commits/7ad7fa31bafceb692bfad082cd0893a25ab5140a\"},{\"sha\":\"6d5a4ab572693869e6ed99d95e0016c51bbb2082\",\"author\":{\"email\":\"a6066a030a600eca8738f6d597ace912fc74e459@wp.pl\",\"name\":\"wszarmach\"},\"message\":\"Merge pull request #97 from krico/jas-83\\n\\njas-83 Complete specification\",\"distinct\":false,\"url\":\"https://api.github.com/repos/krico/jas/commits/6d5a4ab572693869e6ed99d95e0016c51bbb2082\"},{\"sha\":\"904f8b941ac5bfb140ebb52cb7e990955af60252\",\"author\":{\"email\":\"a6066a030a600eca8738f6d597ace912fc74e459@wp.pl\",\"name\":\"wszarmach\"},\"message\":\"fixes jas-93 Google and Facebook sign up have wrong symbols\",\"distinct\":false,\"url\":\"https://api.github.com/repos/krico/jas/commits/904f8b941ac5bfb140ebb52cb7e990955af60252\"},{\"sha\":\"3cc7fce7bc815199132b04070fac7abf133e2a85\",\"author\":{\"email\":\"a6066a030a600eca8738f6d597ace912fc74e459@wp.pl\",\"name\":\"wszarmach\"},\"message\":\"Merge pull request #98 from krico/jas-93\\n\\nfixes jas-93 Google and Facebook sign up have wrong symbols\",\"distinct\":false,\"url\":\"https://api.github.com/repos/krico/jas/commits/3cc7fce7bc815199132b04070fac7abf133e2a85\"},{\"sha\":\"17f2f8288a44307d644e53442aa2af8da4f06ef9\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"Refactored, cleaned up and tested api\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/17f2f8288a44307d644e53442aa2af8da4f06ef9\"},{\"sha\":\"afc0c40bd91051010ec51f179370b20a9df659f7\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"fixes #93\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/afc0c40bd91051010ec51f179370b20a9df659f7\"},{\"sha\":\"6e9fc012e2904977605aea893fc6cbb48bd2e958\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"fixes #93\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/6e9fc012e2904977605aea893fc6cbb48bd2e958\"},{\"sha\":\"befe1cef92f320fccdeccc0a695d8662e2a70ada\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"First javascript service (UserLogin) based on the jasify API\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/befe1cef92f320fccdeccc0a695d8662e2a70ada\"},{\"sha\":\"30042e5952af29070eeea08a3191aeedd745431f\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"JasUserLoginTransformer: had to change methods to be based on  UserLogin.id\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/30042e5952af29070eeea08a3191aeedd745431f\"},{\"sha\":\"0e38de2350dce0face64ce29ad79830ecca4ecae\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"UsernameServlet replaced by JasifyEndpoint\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/0e38de2350dce0face64ce29ad79830ecca4ecae\"},{\"sha\":\"d1af5a8f4d5f6c2cf379eeaa61c56a3e9d63af0e\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"UsernameServlet replaced by JasifyEndpoint (forgot js tests)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/d1af5a8f4d5f6c2cf379eeaa61c56a3e9d63af0e\"},{\"sha\":\"04fdbb02d22c599d5697e87668b269113c51f77d\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"ChangePasswordServlet decommed and moved to JasifyEndpoint\\n\\n**LAST COMMIT OF 2014!!!!**\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/04fdbb02d22c599d5697e87668b269113c51f77d\"},{\"sha\":\"83676e90be82ad7e4401a5f1f7e8fe4eaf9235c3\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"Cannot use `@Named` parameters for password as they show in the URL\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/83676e90be82ad7e4401a5f1f7e8fe4eaf9235c3\"},{\"sha\":\"638d40f314d059beee89b841684d816f10973bff\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"Moved changePassword to auth\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/638d40f314d059beee89b841684d816f10973bff\"},{\"sha\":\"3dea0fdecaef71683ba7398586de15704f2803c6\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"LogoutServlet replaced with API\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/3dea0fdecaef71683ba7398586de15704f2803c6\"},{\"sha\":\"08bf4359f7d54b6a488c650b08fe8f31a8e1703a\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@cwa.to\",\"name\":\"krico\"},\"message\":\"Merge branch 'jas-95' of github.com:krico/jas into jas-95\",\"distinct\":true,\"url\":\"https://api.github.com/repos/krico/jas/commits/08bf4359f7d54b6a488c650b08fe8f31a8e1703a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:02Z\"}\n{\"id\":\"2489396363\",\"type\":\"PushEvent\",\"actor\":{\"id\":2539292,\"login\":\"wmfgerrit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wmfgerrit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2539292?\"},\"repo\":{\"id\":6495082,\"name\":\"wikimedia/mediawiki-extensions\",\"url\":\"https://api.github.com/repos/wikimedia/mediawiki-extensions\"},\"payload\":{\"push_id\":536752453,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"88b2fc95cfa3af5741e02f01e950b1f7c16cdfe8\",\"before\":\"c5cc7199b6ff16671222f91c173945875911bc32\",\"commits\":[{\"sha\":\"88b2fc95cfa3af5741e02f01e950b1f7c16cdfe8\",\"author\":{\"email\":\"61a1c7c885f4fc173424af2875d459c32bdc4fee@gerrit.wikimedia.org\",\"name\":\"jenkins-bot\"},\"message\":\"Updated mediawiki/extensions\\nProject: mediawiki/extensions/WikimediaMessages  61ec5c645adae968e353a778c4f7e82a86ad3468\\n\\nAdd missing wikibase-sitelinks-sitename-wikidatawiki message\\n\\nChange-Id: I1edd3a55f4c8240a2317828adcb9eedccbe904f4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wikimedia/mediawiki-extensions/commits/88b2fc95cfa3af5741e02f01e950b1f7c16cdfe8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:02Z\",\"org\":{\"id\":56668,\"login\":\"wikimedia\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wikimedia\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/56668?\"}}\n{\"id\":\"2489396366\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":134455,\"login\":\"whit537\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/whit537\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/134455?\"},\"repo\":{\"id\":16488998,\"name\":\"gratipay/inside.gratipay.com\",\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/86\",\"labels_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/86/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/86/comments\",\"events_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/86/events\",\"html_url\":\"https://github.com/gratipay/inside.gratipay.com/issues/86\",\"id\":42198005,\"number\":86,\"title\":\"RSVP List\",\"user\":{\"login\":\"clone1018\",\"id\":226638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/226638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clone1018\",\"html_url\":\"https://github.com/clone1018\",\"followers_url\":\"https://api.github.com/users/clone1018/followers\",\"following_url\":\"https://api.github.com/users/clone1018/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clone1018/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clone1018/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clone1018/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clone1018/orgs\",\"repos_url\":\"https://api.github.com/users/clone1018/repos\",\"events_url\":\"https://api.github.com/users/clone1018/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clone1018/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/milestones/2\",\"labels_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/milestones/2/labels\",\"id\":782050,\"number\":2,\"title\":\"Gratipay Retreat 2015\",\"description\":\"This is an encompassing milestone for planning the Gratipay Retreat. \",\"creator\":{\"login\":\"clone1018\",\"id\":226638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/226638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clone1018\",\"html_url\":\"https://github.com/clone1018\",\"followers_url\":\"https://api.github.com/users/clone1018/followers\",\"following_url\":\"https://api.github.com/users/clone1018/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clone1018/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clone1018/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clone1018/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clone1018/orgs\",\"repos_url\":\"https://api.github.com/users/clone1018/repos\",\"events_url\":\"https://api.github.com/users/clone1018/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clone1018/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":2,\"closed_issues\":1,\"state\":\"open\",\"created_at\":\"2014-09-08T13:35:41Z\",\"updated_at\":\"2015-01-01T00:49:37Z\",\"due_on\":\"2015-01-02T08:00:00Z\",\"closed_at\":null},\"comments\":36,\"created_at\":\"2014-09-08T13:52:30Z\",\"updated_at\":\"2015-01-01T01:01:02Z\",\"closed_at\":null,\"body\":\"## What?\\r\\n\\r\\nThe purpose of the Gratipay company retreat is to strategize about the year ahead, sprint on Gratipay and related projects, and to continue to build meaningful relationships.\\r\\n\\r\\n## When?\\r\\n\\r\\n![countdown!](http://secondapps.com/countdown/t1420236000z1.png)\\r\\n(Due to GitHub's image caching this may be out of date)\\r\\n\\r\\nStarts: Friday January 2, 2014 at 5:00pm\\r\\nEnds: Monday, January 5, 2014 at 12:00pm\\r\\n\\r\\n## Where?\\r\\n\\r\\n716 Park Road, Ambridge, PA 15003\\r\\n\\r\\n## Who?\\r\\n### Yes: 1\\r\\n1 @clone1018\\r\\n1 @seanlinsley \\r\\n1 @whit537 \\r\\n1 @rummik \\r\\n1 @colindean \\r\\n1 @kaguillera \\r\\n\\r\\n### Maybe: 0\\r\\n\\r\\n### No: 1\\r\\n1 @chrisdev\\r\\n1 @patcon\"},\"comment\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/comments/68477232\",\"html_url\":\"https://github.com/gratipay/inside.gratipay.com/issues/86#issuecomment-68477232\",\"issue_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/86\",\"id\":68477232,\"user\":{\"login\":\"whit537\",\"id\":134455,\"avatar_url\":\"https://avatars.githubusercontent.com/u/134455?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/whit537\",\"html_url\":\"https://github.com/whit537\",\"followers_url\":\"https://api.github.com/users/whit537/followers\",\"following_url\":\"https://api.github.com/users/whit537/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/whit537/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/whit537/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/whit537/subscriptions\",\"organizations_url\":\"https://api.github.com/users/whit537/orgs\",\"repos_url\":\"https://api.github.com/users/whit537/repos\",\"events_url\":\"https://api.github.com/users/whit537/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/whit537/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:02Z\",\"updated_at\":\"2015-01-01T01:01:02Z\",\"body\":\"Heard from @kaguillera in private email, sounds like he's planning to make it!\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:02Z\",\"org\":{\"id\":1744073,\"login\":\"gratipay\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/gratipay\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1744073?\"}}\n{\"id\":\"2489396374\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1291511,\"login\":\"aamedina\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aamedina\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1291511?\"},\"repo\":{\"id\":10082287,\"name\":\"clojure/core.async\",\"url\":\"https://api.github.com/repos/clojure/core.async\"},\"payload\":{\"forkee\":{\"id\":28678224,\"name\":\"core.async\",\"full_name\":\"aamedina/core.async\",\"owner\":{\"login\":\"aamedina\",\"id\":1291511,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1291511?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aamedina\",\"html_url\":\"https://github.com/aamedina\",\"followers_url\":\"https://api.github.com/users/aamedina/followers\",\"following_url\":\"https://api.github.com/users/aamedina/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/aamedina/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/aamedina/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/aamedina/subscriptions\",\"organizations_url\":\"https://api.github.com/users/aamedina/orgs\",\"repos_url\":\"https://api.github.com/users/aamedina/repos\",\"events_url\":\"https://api.github.com/users/aamedina/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/aamedina/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/aamedina/core.async\",\"description\":\"Facilities for async programming and communication in Clojure\",\"fork\":true,\"url\":\"https://api.github.com/repos/aamedina/core.async\",\"forks_url\":\"https://api.github.com/repos/aamedina/core.async/forks\",\"keys_url\":\"https://api.github.com/repos/aamedina/core.async/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/aamedina/core.async/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/aamedina/core.async/teams\",\"hooks_url\":\"https://api.github.com/repos/aamedina/core.async/hooks\",\"issue_events_url\":\"https://api.github.com/repos/aamedina/core.async/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/aamedina/core.async/events\",\"assignees_url\":\"https://api.github.com/repos/aamedina/core.async/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/aamedina/core.async/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/aamedina/core.async/tags\",\"blobs_url\":\"https://api.github.com/repos/aamedina/core.async/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/aamedina/core.async/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/aamedina/core.async/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/aamedina/core.async/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/aamedina/core.async/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/aamedina/core.async/languages\",\"stargazers_url\":\"https://api.github.com/repos/aamedina/core.async/stargazers\",\"contributors_url\":\"https://api.github.com/repos/aamedina/core.async/contributors\",\"subscribers_url\":\"https://api.github.com/repos/aamedina/core.async/subscribers\",\"subscription_url\":\"https://api.github.com/repos/aamedina/core.async/subscription\",\"commits_url\":\"https://api.github.com/repos/aamedina/core.async/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/aamedina/core.async/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/aamedina/core.async/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/aamedina/core.async/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/aamedina/core.async/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/aamedina/core.async/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/aamedina/core.async/merges\",\"archive_url\":\"https://api.github.com/repos/aamedina/core.async/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/aamedina/core.async/downloads\",\"issues_url\":\"https://api.github.com/repos/aamedina/core.async/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/aamedina/core.async/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/aamedina/core.async/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/aamedina/core.async/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/aamedina/core.async/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/aamedina/core.async/releases{/id}\",\"created_at\":\"2015-01-01T01:01:04Z\",\"updated_at\":\"2014-12-29T21:26:26Z\",\"pushed_at\":\"2014-10-15T00:42:26Z\",\"git_url\":\"git://github.com/aamedina/core.async.git\",\"ssh_url\":\"git@github.com:aamedina/core.async.git\",\"clone_url\":\"https://github.com/aamedina/core.async.git\",\"svn_url\":\"https://github.com/aamedina/core.async\",\"homepage\":null,\"size\":2570,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:01:04Z\",\"org\":{\"id\":317875,\"login\":\"clojure\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/clojure\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/317875?\"}}\n{\"id\":\"2489396375\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2829600,\"login\":\"GrahamCampbell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GrahamCampbell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829600?\"},\"repo\":{\"id\":26730195,\"name\":\"cachethq/Cachet\",\"url\":\"https://api.github.com/repos/cachethq/Cachet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"labels_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/comments\",\"events_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/events\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173\",\"id\":53210024,\"number\":173,\"title\":\"Bug: Forms let you submit multiple times\",\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2015-01-01T00:52:06Z\",\"updated_at\":\"2015-01-01T01:01:04Z\",\"closed_at\":null,\"body\":\"When adding a new incident, I noticed a weird bug.\\r\\n\\r\\nIf you fill in the form as normal, then click the submit button twice really quickly, it'll create __TWO__ identical new incidents!\\r\\n\\r\\nThis could be a bit annoying, a simple fix is using a bit of JS that on submit, disables the submit button so that once clicked, it cannot be clicked again.\"},\"comment\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/comments/68477233\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173#issuecomment-68477233\",\"issue_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"id\":68477233,\"user\":{\"login\":\"GrahamCampbell\",\"id\":2829600,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829600?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GrahamCampbell\",\"html_url\":\"https://github.com/GrahamCampbell\",\"followers_url\":\"https://api.github.com/users/GrahamCampbell/followers\",\"following_url\":\"https://api.github.com/users/GrahamCampbell/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/GrahamCampbell/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/GrahamCampbell/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/GrahamCampbell/subscriptions\",\"organizations_url\":\"https://api.github.com/users/GrahamCampbell/orgs\",\"repos_url\":\"https://api.github.com/users/GrahamCampbell/repos\",\"events_url\":\"https://api.github.com/users/GrahamCampbell/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/GrahamCampbell/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:04Z\",\"updated_at\":\"2015-01-01T01:01:04Z\",\"body\":\"Provide a unique code as a hidden field that invalides once submitted.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:04Z\",\"org\":{\"id\":9951502,\"login\":\"cachethq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cachethq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9951502?\"}}\n{\"id\":\"2489396376\",\"type\":\"PushEvent\",\"actor\":{\"id\":876467,\"login\":\"flasheater\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/flasheater\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/876467?\"},\"repo\":{\"id\":28678036,\"name\":\"flasheater/fresh\",\"url\":\"https://api.github.com/repos/flasheater/fresh\"},\"payload\":{\"push_id\":536752458,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"36b46027ec5e7e97db30e05215d0bec5eeb0d138\",\"before\":\"a38d020b7c78470cd59382245933e0ef62db5712\",\"commits\":[{\"sha\":\"c0a0b02fe479a282199fbaec6e8530ad6c48379a\",\"author\":{\"email\":\"51e5422d9fa86e2bc865774c76fbbdc9939f4d14@gmail.com\",\"name\":\"Oliver Jan Krylow\"},\"message\":\"Inroduced variable FRESH_REPO to install.sh so that forks of fresh can use this script. Furthermore, begun adding fish\\nsupport.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/flasheater/fresh/commits/c0a0b02fe479a282199fbaec6e8530ad6c48379a\"},{\"sha\":\"36b46027ec5e7e97db30e05215d0bec5eeb0d138\",\"author\":{\"email\":\"51e5422d9fa86e2bc865774c76fbbdc9939f4d14@gmail.com\",\"name\":\"Oliver Jan Krylow\"},\"message\":\"Push to repo to test vagrant support.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/flasheater/fresh/commits/36b46027ec5e7e97db30e05215d0bec5eeb0d138\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:04Z\"}\n{\"id\":\"2489396379\",\"type\":\"PushEvent\",\"actor\":{\"id\":4809376,\"login\":\"guavuslabs-builder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/guavuslabs-builder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4809376?\"},\"repo\":{\"id\":17457530,\"name\":\"Guavus/spark\",\"url\":\"https://api.github.com/repos/Guavus/spark\"},\"payload\":{\"push_id\":536752459,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4bb12488d56ea651c56d9688996b464b99095582\",\"before\":\"fe6efacc0b865e9e827a1565877077000e63976e\",\"commits\":[{\"sha\":\"4bb12488d56ea651c56d9688996b464b99095582\",\"author\":{\"email\":\"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com\",\"name\":\"Sean Owen\"},\"message\":\"SPARK-2757 [BUILD] [STREAMING] Add Mima test for Spark Sink after 1.10 is released\\n\\nRe-enable MiMa for Streaming Flume Sink module, now that 1.1.0 is released, per the JIRA TO-DO. That's pretty much all there is to this.\\n\\nAuthor: Sean Owen <sowen@cloudera.com>\\n\\nCloses #3842 from srowen/SPARK-2757 and squashes the following commits:\\n\\n50ff80e [Sean Owen] Exclude apparent false positive turned up by re-enabling MiMa checks for Streaming Flume Sink\\n0e5ba5c [Sean Owen] Re-enable MiMa for Streaming Flume Sink module\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Guavus/spark/commits/4bb12488d56ea651c56d9688996b464b99095582\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:04Z\",\"org\":{\"id\":3741861,\"login\":\"Guavus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Guavus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3741861?\"}}\n{\"id\":\"2489396385\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":228410,\"login\":\"kitak\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kitak\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/228410?\"},\"repo\":{\"id\":11133793,\"name\":\"mamebro/mameblo\",\"url\":\"https://api.github.com/repos/mamebro/mameblo\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/mamebro/mameblo/issues/249\",\"labels_url\":\"https://api.github.com/repos/mamebro/mameblo/issues/249/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/mamebro/mameblo/issues/249/comments\",\"events_url\":\"https://api.github.com/repos/mamebro/mameblo/issues/249/events\",\"html_url\":\"https://github.com/mamebro/mameblo/pull/249\",\"id\":53185061,\"number\":249,\"title\":\"!!! あけましておめでとうございます !!!\",\"user\":{\"login\":\"shikakun\",\"id\":1396953,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1396953?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/shikakun\",\"html_url\":\"https://github.com/shikakun\",\"followers_url\":\"https://api.github.com/users/shikakun/followers\",\"following_url\":\"https://api.github.com/users/shikakun/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/shikakun/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/shikakun/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/shikakun/subscriptions\",\"organizations_url\":\"https://api.github.com/users/shikakun/orgs\",\"repos_url\":\"https://api.github.com/users/shikakun/repos\",\"events_url\":\"https://api.github.com/users/shikakun/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/shikakun/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T14:00:42Z\",\"updated_at\":\"2015-01-01T01:01:05Z\",\"closed_at\":\"2014-12-31T15:01:12Z\",\"pull_request\":{\"url\":\"https://api.github.com/repos/mamebro/mameblo/pulls/249\",\"html_url\":\"https://github.com/mamebro/mameblo/pull/249\",\"diff_url\":\"https://github.com/mamebro/mameblo/pull/249.diff\",\"patch_url\":\"https://github.com/mamebro/mameblo/pull/249.patch\"},\"body\":\"@mamebro/owners \\r\\n新年を迎えたのがめでたいので、背景が紅白にチカチカ点滅するようにしました。\"},\"comment\":{\"url\":\"https://api.github.com/repos/mamebro/mameblo/issues/comments/68477234\",\"html_url\":\"https://github.com/mamebro/mameblo/pull/249#issuecomment-68477234\",\"issue_url\":\"https://api.github.com/repos/mamebro/mameblo/issues/249\",\"id\":68477234,\"user\":{\"login\":\"kitak\",\"id\":228410,\"avatar_url\":\"https://avatars.githubusercontent.com/u/228410?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kitak\",\"html_url\":\"https://github.com/kitak\",\"followers_url\":\"https://api.github.com/users/kitak/followers\",\"following_url\":\"https://api.github.com/users/kitak/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kitak/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kitak/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kitak/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kitak/orgs\",\"repos_url\":\"https://api.github.com/users/kitak/repos\",\"events_url\":\"https://api.github.com/users/kitak/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kitak/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:05Z\",\"updated_at\":\"2015-01-01T01:01:05Z\",\"body\":\"おめでとうございます！ :tada: \"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:06Z\",\"org\":{\"id\":3763231,\"login\":\"mamebro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/mamebro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3763231?\"}}\n{\"id\":\"2489396387\",\"type\":\"PushEvent\",\"actor\":{\"id\":5869772,\"login\":\"felixonmars-bot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/felixonmars-bot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5869772?\"},\"repo\":{\"id\":15896070,\"name\":\"felixonmars/packages-mirror\",\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror\"},\"payload\":{\"push_id\":536752460,\"size\":5,\"distinct_size\":5,\"ref\":\"refs/heads/master\",\"head\":\"259643a2baa03206b68bee1ab0291f4e5731ba7f\",\"before\":\"9fe97687a1282a034b2cc8abbe659ed029920bbc\",\"commits\":[{\"sha\":\"4a722a9a329fe751f7f1564d888a1167880bb079\",\"author\":{\"email\":\"96f164ad4d9b2b0dacf8ebee2bb1eeb3aa69adf1@eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"name\":\"eric\"},\"message\":\"archrelease: copy trunk to staging-i686\\n\\ngit-svn-id: file:///srv/repos/svn-packages/svn@228268 eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror/commits/4a722a9a329fe751f7f1564d888a1167880bb079\"},{\"sha\":\"9ecdfd1cbd1543c91db31222a939ec481c99b2f0\",\"author\":{\"email\":\"96f164ad4d9b2b0dacf8ebee2bb1eeb3aa69adf1@eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"name\":\"eric\"},\"message\":\"archrelease: copy trunk to staging-x86_64\\n\\ngit-svn-id: file:///srv/repos/svn-packages/svn@228269 eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror/commits/9ecdfd1cbd1543c91db31222a939ec481c99b2f0\"},{\"sha\":\"7d83daeb67690660c7b36adb258b907f9ec4004a\",\"author\":{\"email\":\"96f164ad4d9b2b0dacf8ebee2bb1eeb3aa69adf1@eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"name\":\"eric\"},\"message\":\"Add validpgpkeys array\\n\\ngit-svn-id: file:///srv/repos/svn-packages/svn@228270 eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror/commits/7d83daeb67690660c7b36adb258b907f9ec4004a\"},{\"sha\":\"a9bd2311944899fdc40407321bf727615834b6cb\",\"author\":{\"email\":\"96f164ad4d9b2b0dacf8ebee2bb1eeb3aa69adf1@eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"name\":\"eric\"},\"message\":\"archrelease: copy trunk to extra-i686\\n\\ngit-svn-id: file:///srv/repos/svn-packages/svn@228271 eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror/commits/a9bd2311944899fdc40407321bf727615834b6cb\"},{\"sha\":\"259643a2baa03206b68bee1ab0291f4e5731ba7f\",\"author\":{\"email\":\"96f164ad4d9b2b0dacf8ebee2bb1eeb3aa69adf1@eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"name\":\"eric\"},\"message\":\"archrelease: copy trunk to extra-x86_64\\n\\ngit-svn-id: file:///srv/repos/svn-packages/svn@228272 eb2447ed-0c53-47e4-bac8-5bc4a241df78\",\"distinct\":true,\"url\":\"https://api.github.com/repos/felixonmars/packages-mirror/commits/259643a2baa03206b68bee1ab0291f4e5731ba7f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:06Z\"}\n{\"id\":\"2489396389\",\"type\":\"PushEvent\",\"actor\":{\"id\":3487531,\"login\":\"jl4282\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jl4282\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3487531?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"push_id\":536752462,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"bd278f438aba4582c30eb8d42e7b52740b739e5b\",\"before\":\"78a50b06faa40c0fd632efd2f2f610652052a6c0\",\"commits\":[{\"sha\":\"bd278f438aba4582c30eb8d42e7b52740b739e5b\",\"author\":{\"email\":\"4068bf12e44880d7eec32c1ab3069b58d1acd33f@gmail.com\",\"name\":\"jl4282\"},\"message\":\"started changing master\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite/commits/bd278f438aba4582c30eb8d42e7b52740b739e5b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:06Z\"}\n{\"id\":\"2489396392\",\"type\":\"PushEvent\",\"actor\":{\"id\":8347069,\"login\":\"williamtwilson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/williamtwilson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8347069?\"},\"repo\":{\"id\":28171448,\"name\":\"williamtwilson/speedboard-frontend\",\"url\":\"https://api.github.com/repos/williamtwilson/speedboard-frontend\"},\"payload\":{\"push_id\":536752463,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"66d5fee55bda2c92031dcb9eb15cf02eeb7ecd90\",\"before\":\"6138829ffa8274bce05eff607c5acf28fc880399\",\"commits\":[{\"sha\":\"66d5fee55bda2c92031dcb9eb15cf02eeb7ecd90\",\"author\":{\"email\":\"2ded4b346a572ccf57131801116a3edef4d9c258@gmail.com\",\"name\":\"William Wilson\"},\"message\":\"Fixing again for heroku\",\"distinct\":true,\"url\":\"https://api.github.com/repos/williamtwilson/speedboard-frontend/commits/66d5fee55bda2c92031dcb9eb15cf02eeb7ecd90\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:06Z\"}\n{\"id\":\"2489396393\",\"type\":\"PushEvent\",\"actor\":{\"id\":6895040,\"login\":\"codertradergambler\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codertradergambler\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6895040?\"},\"repo\":{\"id\":18620619,\"name\":\"chancecoin/chancecoinj\",\"url\":\"https://api.github.com/repos/chancecoin/chancecoinj\"},\"payload\":{\"push_id\":536752465,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d4ec65797e8faf2aac84c02007d2ebeb342babd3\",\"before\":\"2f4f83b1efbd39bd5a66a9929baa1fcee177851b\",\"commits\":[{\"sha\":\"d4ec65797e8faf2aac84c02007d2ebeb342babd3\",\"author\":{\"email\":\"0ccf54d51d1a5240ad356feb30dfa4d1749f8844@gmail.com\",\"name\":\"TraderCoderGambler\"},\"message\":\"auto-update balances\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chancecoin/chancecoinj/commits/d4ec65797e8faf2aac84c02007d2ebeb342babd3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:06Z\"}\n{\"id\":\"2489396397\",\"type\":\"PushEvent\",\"actor\":{\"id\":7336721,\"login\":\"aow1980\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aow1980\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7336721?\"},\"repo\":{\"id\":28112636,\"name\":\"aow1980/frameworks_base\",\"url\":\"https://api.github.com/repos/aow1980/frameworks_base\"},\"payload\":{\"push_id\":536752467,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/lp5.0\",\"head\":\"162d74b56f030928c24d1d4cb5033586e9d319db\",\"before\":\"ee02cebf037cb979511dc12688e99a1492faa47d\",\"commits\":[{\"sha\":\"162d74b56f030928c24d1d4cb5033586e9d319db\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"base: fix battery stats wakelock crazyness\\n\\nUntil someone explains to me why google has killed the\\nsince unplug stats for partitial and kernel wakelocks\\nchange them from beeing screen off based to unplug based\\n\\nChange-Id: Id727d5d9e237eecb7e86d7dee3285f18a57f9723\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/frameworks_base/commits/162d74b56f030928c24d1d4cb5033586e9d319db\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:07Z\"}\n{\"id\":\"2489396399\",\"type\":\"PushEvent\",\"actor\":{\"id\":3890972,\"login\":\"timmmmyboy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/timmmmyboy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3890972?\"},\"repo\":{\"id\":26382386,\"name\":\"reclaimhosting/federated-wiki\",\"url\":\"https://api.github.com/repos/reclaimhosting/federated-wiki\"},\"payload\":{\"push_id\":536752470,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fdf80ab95eec8733946219b1a172269d3d802042\",\"before\":\"fef0b8d914b47a72907042e8bdd4b631e331f1c4\",\"commits\":[{\"sha\":\"fdf80ab95eec8733946219b1a172269d3d802042\",\"author\":{\"email\":\"59bd0a3ff43b32849b319e645d4798d8a5d1e889@reclaimhosting.com\",\"name\":\"Reclaim Hosting\"},\"message\":\"Recent Changes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/reclaimhosting/federated-wiki/commits/fdf80ab95eec8733946219b1a172269d3d802042\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:07Z\",\"org\":{\"id\":6590468,\"login\":\"reclaimhosting\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/reclaimhosting\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6590468?\"}}\n{\"id\":\"2489396405\",\"type\":\"PushEvent\",\"actor\":{\"id\":1779595,\"login\":\"dcbaker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcbaker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1779595?\"},\"repo\":{\"id\":8488437,\"name\":\"dcbaker/piglit\",\"url\":\"https://api.github.com/repos/dcbaker/piglit\"},\"payload\":{\"push_id\":536752473,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/wip/command-list-only-v5\",\"head\":\"e81f4b28bf6ad78c7fb05257e77e42fe237a09ed\",\"before\":\"2400271cda135b8077fd4a83254ef84edd92f6bd\",\"commits\":[{\"sha\":\"cf5cf7224385c3c3d086762b5bc1d8f7df031ae7\",\"author\":{\"email\":\"c26a678a04c601e0311b0d6006e67eee6ed19a8e@intel.com\",\"name\":\"Dylan Baker\"},\"message\":\"fixup! all.py: Replace some string concatenation with str.format()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dcbaker/piglit/commits/cf5cf7224385c3c3d086762b5bc1d8f7df031ae7\"},{\"sha\":\"e81f4b28bf6ad78c7fb05257e77e42fe237a09ed\",\"author\":{\"email\":\"c26a678a04c601e0311b0d6006e67eee6ed19a8e@intel.com\",\"name\":\"Dylan Baker\"},\"message\":\"all.py: Convert glean tests to list arguments\\n\\nThere aren't very many glean tests left, it was trivially easy to change\\nthem with a simple vim macro and then one additional change.\\n\\nSigned-off-by: Dylan Baker <dylanx.c.baker@intel.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dcbaker/piglit/commits/e81f4b28bf6ad78c7fb05257e77e42fe237a09ed\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:08Z\"}\n{\"id\":\"2489396406\",\"type\":\"PushEvent\",\"actor\":{\"id\":41057,\"login\":\"samv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/samv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/41057?\"},\"repo\":{\"id\":16792286,\"name\":\"hearsaycorp/normalize\",\"url\":\"https://api.github.com/repos/hearsaycorp/normalize\"},\"payload\":{\"push_id\":536752474,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2f0df332bf4a1e963296f545802cf7497ac20ff8\",\"before\":\"f0e473fa4a756b8c3b49a4f945845414f58bf9fb\",\"commits\":[{\"sha\":\"2f0df332bf4a1e963296f545802cf7497ac20ff8\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@vilain.net\",\"name\":\"Sam Vilain\"},\"message\":\"Implement FieldSelector.delete\\n\\nAll 4 CRUD operations now supported :)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hearsaycorp/normalize/commits/2f0df332bf4a1e963296f545802cf7497ac20ff8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:08Z\",\"org\":{\"id\":633032,\"login\":\"hearsaycorp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/hearsaycorp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/633032?\"}}\n{\"id\":\"2489396407\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5084309,\"login\":\"Jorys-Paulin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jorys-Paulin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5084309?\"},\"repo\":{\"id\":28086817,\"name\":\"Learn-Dev/Learn-Dev-Theme---Dashboard-partie-1\",\"url\":\"https://api.github.com/repos/Learn-Dev/Learn-Dev-Theme---Dashboard-partie-1\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:08Z\"}\n{\"id\":\"2489396408\",\"type\":\"PushEvent\",\"actor\":{\"id\":9000293,\"login\":\"diianita\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/diianita\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9000293?\"},\"repo\":{\"id\":27146993,\"name\":\"cArLiiToX/dtstore\",\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore\"},\"payload\":{\"push_id\":536752475,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"57150fe4e8e3dbd8707e8bec5efdcbd306696812\",\"before\":\"080810e75308aaace0986bf99858aee08f79fe60\",\"commits\":[{\"sha\":\"57150fe4e8e3dbd8707e8bec5efdcbd306696812\",\"author\":{\"email\":\"ab5e2bca84933118bbc9d48ffaccce3bac4eeb64@xng.bz\",\"name\":\"cArLiiToX\"},\"message\":\"correciones\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore/commits/57150fe4e8e3dbd8707e8bec5efdcbd306696812\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:09Z\"}\n{\"id\":\"2489396409\",\"type\":\"CreateEvent\",\"actor\":{\"id\":57162,\"login\":\"wbyoung\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wbyoung\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/57162?\"},\"repo\":{\"id\":25276289,\"name\":\"wbyoung/azul\",\"url\":\"https://api.github.com/repos/wbyoung/azul\"},\"payload\":{\"ref\":\"travis-docs\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Elegant Node.js ORM\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:09Z\"}\n{\"id\":\"2489396413\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":3253898,\"name\":\"greggman/webgl-fundamentals\",\"url\":\"https://api.github.com/repos/greggman/webgl-fundamentals\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:11Z\"}\n{\"id\":\"2489396414\",\"type\":\"PushEvent\",\"actor\":{\"id\":1514700,\"login\":\"YoshikiShibata\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YoshikiShibata\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1514700?\"},\"repo\":{\"id\":23509495,\"name\":\"YoshikiShibata/js8ri\",\"url\":\"https://api.github.com/repos/YoshikiShibata/js8ri\"},\"payload\":{\"push_id\":536752478,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b17a952587b5f8accbf2451df75b97812402210f\",\"before\":\"201f71f7ce532cf0ee666466ffa97d7003402328\",\"commits\":[{\"sha\":\"b17a952587b5f8accbf2451df75b97812402210f\",\"author\":{\"email\":\"fc4b8596386bb28ae34eb13b0bb8db8e7157ad6c@ca2.so-net.ne.jp\",\"name\":\"Yoshiki Shibata\"},\"message\":\"ex07 done\",\"distinct\":true,\"url\":\"https://api.github.com/repos/YoshikiShibata/js8ri/commits/b17a952587b5f8accbf2451df75b97812402210f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:11Z\"}\n{\"id\":\"2489396415\",\"type\":\"PushEvent\",\"actor\":{\"id\":1380844,\"login\":\"lyricorpse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lyricorpse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1380844?\"},\"repo\":{\"id\":28656760,\"name\":\"lyricorpse/PyESG\",\"url\":\"https://api.github.com/repos/lyricorpse/PyESG\"},\"payload\":{\"push_id\":536752479,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6f2430e5a35ddc873ff33e454ffe95d3754aab32\",\"before\":\"14e8117e011afcadb8066ac8de75976359c7818f\",\"commits\":[{\"sha\":\"6f2430e5a35ddc873ff33e454ffe95d3754aab32\",\"author\":{\"email\":\"0a7b040fc4054c5d741bd0ad5ddf9a49e6d82d24@gmail.com\",\"name\":\"Lyricorpse\"},\"message\":\"Fix a bug with Quadrangle search\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lyricorpse/PyESG/commits/6f2430e5a35ddc873ff33e454ffe95d3754aab32\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:11Z\"}\n{\"id\":\"2489396422\",\"type\":\"PushEvent\",\"actor\":{\"id\":2287714,\"login\":\"mchen804\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mchen804\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2287714?\"},\"repo\":{\"id\":28678145,\"name\":\"mchen804/LearnToGit\",\"url\":\"https://api.github.com/repos/mchen804/LearnToGit\"},\"payload\":{\"push_id\":536752481,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d81f2b2536eac12bec771117976723439102d999\",\"before\":\"c25d99beb779acd54c083029bb51d94ec9e2bbe4\",\"commits\":[{\"sha\":\"d81f2b2536eac12bec771117976723439102d999\",\"author\":{\"email\":\"ce78b05c0afb1e03455d157070260213f45d7b80@osu.edu\",\"name\":\"Michael Chen\"},\"message\":\"Create README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mchen804/LearnToGit/commits/d81f2b2536eac12bec771117976723439102d999\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:11Z\"}\n{\"id\":\"2489396432\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":238759,\"login\":\"chrisfilo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chrisfilo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/238759?\"},\"repo\":{\"id\":7963570,\"name\":\"NeuroVault/NeuroVault\",\"url\":\"https://api.github.com/repos/NeuroVault/NeuroVault\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/90\",\"labels_url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/90/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/90/comments\",\"events_url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/90/events\",\"html_url\":\"https://github.com/NeuroVault/NeuroVault/pull/90\",\"id\":53209632,\"number\":90,\"title\":\"Enh/nidm results [final]\",\"user\":{\"login\":\"infocortex\",\"id\":7991256,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7991256?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/infocortex\",\"html_url\":\"https://github.com/infocortex\",\"followers_url\":\"https://api.github.com/users/infocortex/followers\",\"following_url\":\"https://api.github.com/users/infocortex/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/infocortex/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/infocortex/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/infocortex/subscriptions\",\"organizations_url\":\"https://api.github.com/users/infocortex/orgs\",\"repos_url\":\"https://api.github.com/users/infocortex/repos\",\"events_url\":\"https://api.github.com/users/infocortex/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/infocortex/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:29:03Z\",\"updated_at\":\"2015-01-01T01:01:13Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/pulls/90\",\"html_url\":\"https://github.com/NeuroVault/NeuroVault/pull/90\",\"diff_url\":\"https://github.com/NeuroVault/NeuroVault/pull/90.diff\",\"patch_url\":\"https://github.com/NeuroVault/NeuroVault/pull/90.patch\"},\"body\":\"feature complete\\r\\n\\r\\nstill todo:\\r\\n-migration test \\r\\n-addditional unit test(s)\"},\"comment\":{\"url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/comments/68477237\",\"html_url\":\"https://github.com/NeuroVault/NeuroVault/pull/90#issuecomment-68477237\",\"issue_url\":\"https://api.github.com/repos/NeuroVault/NeuroVault/issues/90\",\"id\":68477237,\"user\":{\"login\":\"chrisfilo\",\"id\":238759,\"avatar_url\":\"https://avatars.githubusercontent.com/u/238759?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chrisfilo\",\"html_url\":\"https://github.com/chrisfilo\",\"followers_url\":\"https://api.github.com/users/chrisfilo/followers\",\"following_url\":\"https://api.github.com/users/chrisfilo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/chrisfilo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/chrisfilo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/chrisfilo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/chrisfilo/orgs\",\"repos_url\":\"https://api.github.com/users/chrisfilo/repos\",\"events_url\":\"https://api.github.com/users/chrisfilo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/chrisfilo/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:13Z\",\"updated_at\":\"2015-01-01T01:01:13Z\",\"body\":\"Migration went well, test pass. However edit images dialog does not work. Clicking on any of the images does not change anything. Neighter does clicking on the Add a new image button. JS console gives the following error:\\r\\n\\r\\n\\tUncaught TypeError: Cannot read property 'slice' of nullj query-2.1.0.min.js:2\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:13Z\",\"org\":{\"id\":9168649,\"login\":\"NeuroVault\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/NeuroVault\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9168649?\"}}\n{\"id\":\"2489396433\",\"type\":\"PushEvent\",\"actor\":{\"id\":1745861,\"login\":\"topaztee\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/topaztee\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1745861?\"},\"repo\":{\"id\":28677407,\"name\":\"topaztee/topaztee.github.io\",\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io\"},\"payload\":{\"push_id\":536752485,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a66e99dc9be241e6a513acfcc79ac27757a4a5bb\",\"before\":\"e0ee4ea2c3198a01bf7f11096b6c35a05fa11ae1\",\"commits\":[{\"sha\":\"a66e99dc9be241e6a513acfcc79ac27757a4a5bb\",\"author\":{\"email\":\"f74c82d708bb42a372674042ebc8a1411fbc9344@192-168-1-2.tpgi.com.au\",\"name\":\"topaztur@gmail.com\"},\"message\":\"Blog update at 2015-01-01 01:01:05\",\"distinct\":true,\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io/commits/a66e99dc9be241e6a513acfcc79ac27757a4a5bb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:13Z\"}\n{\"id\":\"2489396443\",\"type\":\"PushEvent\",\"actor\":{\"id\":5863537,\"login\":\"jimv39\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jimv39\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5863537?\"},\"repo\":{\"id\":17339971,\"name\":\"jimv39/qvcsos\",\"url\":\"https://api.github.com/repos/jimv39/qvcsos\"},\"payload\":{\"push_id\":536752489,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/develop\",\"head\":\"96ae99b1cb7c178ee7951befbe8bd9a9df12bda5\",\"before\":\"ab423d835e445dcf8c3073753ae1b964e386dfdd\",\"commits\":[{\"sha\":\"f673e524b862f0ee3ecc023d4f668282f07dd8ce\",\"author\":{\"email\":\"9be38403eb58d3adbe34866aa881dcd238ac5d2e@comcast.net\",\"name\":\"Jim Voris\"},\"message\":\"First pass at adding jmockit unit test to OperationGet. It really only provides code coverage, but it's better than nothing.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jimv39/qvcsos/commits/f673e524b862f0ee3ecc023d4f668282f07dd8ce\"},{\"sha\":\"7b2e46db58499c257cb0d1740af53807c87d2b92\",\"author\":{\"email\":\"9be38403eb58d3adbe34866aa881dcd238ac5d2e@comcast.net\",\"name\":\"Jim Voris\"},\"message\":\"Fix some checkstyle problems.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jimv39/qvcsos/commits/7b2e46db58499c257cb0d1740af53807c87d2b92\"},{\"sha\":\"96ae99b1cb7c178ee7951befbe8bd9a9df12bda5\",\"author\":{\"email\":\"9be38403eb58d3adbe34866aa881dcd238ac5d2e@comcast.net\",\"name\":\"Jim Voris\"},\"message\":\"Remove dead/commented-out plugin.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jimv39/qvcsos/commits/96ae99b1cb7c178ee7951befbe8bd9a9df12bda5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:14Z\"}\n{\"id\":\"2489396456\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5565906,\"login\":\"WRuman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/WRuman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5565906?\"},\"repo\":{\"id\":14579010,\"name\":\"suffick/Tearable-Cloth\",\"url\":\"https://api.github.com/repos/suffick/Tearable-Cloth\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:16Z\"}\n{\"id\":\"2489396458\",\"type\":\"PushEvent\",\"actor\":{\"id\":6955349,\"login\":\"rachidkada\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rachidkada\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6955349?\"},\"repo\":{\"id\":27822066,\"name\":\"ATS001/NEWCOMPTA\",\"url\":\"https://api.github.com/repos/ATS001/NEWCOMPTA\"},\"payload\":{\"push_id\":536752490,\"size\":1,\"distinct_size\":0,\"ref\":\"refs/heads/gestion_depense_last\",\"head\":\"8ed92a796d203fdc84e418d3f9ca725cccd0fec4\",\"before\":\"890201ee3b26f2e7c383a1c8f4206ebf0015c3ee\",\"commits\":[{\"sha\":\"8ed92a796d203fdc84e418d3f9ca725cccd0fec4\",\"author\":{\"email\":\"af045d9e68f1279af589eff3a2682851ea4e8686@atelsolution.com\",\"name\":\"id-rach\"},\"message\":\"fournisseur\\n\\nfournisseur\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ATS001/NEWCOMPTA/commits/8ed92a796d203fdc84e418d3f9ca725cccd0fec4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:16Z\",\"org\":{\"id\":8378855,\"login\":\"ATS001\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ATS001\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8378855?\"}}\n{\"id\":\"2489396459\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536752491,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f5fd8c9b5d99ba2f521c56658163b69c29eb1027\",\"before\":\"9ceb5cdffe02cef186b379026ce6c2dc0d74b3d4\",\"commits\":[{\"sha\":\"f5fd8c9b5d99ba2f521c56658163b69c29eb1027\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074073410\\n\\nZl5biUHcnwVJiZ03vi7LUj1wLwVyyBgsRxOBQqS03lg=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/f5fd8c9b5d99ba2f521c56658163b69c29eb1027\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:16Z\"}\n{\"id\":\"2489396460\",\"type\":\"PushEvent\",\"actor\":{\"id\":2147137,\"login\":\"xkyouchoux\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/xkyouchoux\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2147137?\"},\"repo\":{\"id\":28676642,\"name\":\"xkyouchoux/dh\",\"url\":\"https://api.github.com/repos/xkyouchoux/dh\"},\"payload\":{\"push_id\":536752493,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b6d6c9c6e52c6a414626a870ee6179d8e3bdd437\",\"before\":\"bcf6c520af9d96ed8d9c978d723b94191b4c7828\",\"commits\":[{\"sha\":\"b6d6c9c6e52c6a414626a870ee6179d8e3bdd437\",\"author\":{\"email\":\"fefcf3693656ff0c8f9b06efb68deae5a2d3a81a@hotmail.com\",\"name\":\"xkyouchoux\"},\"message\":\"A General generated set\",\"distinct\":true,\"url\":\"https://api.github.com/repos/xkyouchoux/dh/commits/b6d6c9c6e52c6a414626a870ee6179d8e3bdd437\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:16Z\"}\n{\"id\":\"2489396461\",\"type\":\"PushEvent\",\"actor\":{\"id\":7751315,\"login\":\"ApocalypticOctopus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ApocalypticOctopus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7751315?\"},\"repo\":{\"id\":27552091,\"name\":\"ApocalypticOctopus/Epic.Numbers\",\"url\":\"https://api.github.com/repos/ApocalypticOctopus/Epic.Numbers\"},\"payload\":{\"push_id\":536752492,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f8af8686554e318964193f2da13f486aabaeaf99\",\"before\":\"dff9cb7b8c26a82657601a5365b5b92fd3317902\",\"commits\":[{\"sha\":\"f8af8686554e318964193f2da13f486aabaeaf99\",\"author\":{\"email\":\"60e80a2f6e90298c62ee9e963a3f42492975e450@gmail.com\",\"name\":\"Apocalyptic Octopus\"},\"message\":\"Add a few basic generic number algorithms\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ApocalypticOctopus/Epic.Numbers/commits/f8af8686554e318964193f2da13f486aabaeaf99\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:16Z\"}\n{\"id\":\"2489396468\",\"type\":\"PushEvent\",\"actor\":{\"id\":95203,\"login\":\"bhaisaab\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bhaisaab\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/95203?\"},\"repo\":{\"id\":23222011,\"name\":\"shapeblue/cloudstack\",\"url\":\"https://api.github.com/repos/shapeblue/cloudstack\"},\"payload\":{\"push_id\":536752498,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/4.5\",\"head\":\"85e88e9cbcba913fd9848e1daa42e21259c02a96\",\"before\":\"94237a4c5b08356bb0a4950ce9fd7ec78f2168f9\",\"commits\":[{\"sha\":\"85e88e9cbcba913fd9848e1daa42e21259c02a96\",\"author\":{\"email\":\"908e82d1c152b24fc5de9371836b7f39094e3948@apache.org\",\"name\":\"Jessica Wang\"},\"message\":\"CLOUDSTACK-8139: UI > create compute offering > server-side only supports one single host tag instead of multiple host tags. So, change UI to take in only one single host tag instead of multiple host tags in create compute dialog.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/shapeblue/cloudstack/commits/85e88e9cbcba913fd9848e1daa42e21259c02a96\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:17Z\",\"org\":{\"id\":6001764,\"login\":\"shapeblue\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/shapeblue\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6001764?\"}}\n{\"id\":\"2489396482\",\"type\":\"ForkEvent\",\"actor\":{\"id\":31330,\"login\":\"dwmorgan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dwmorgan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/31330?\"},\"repo\":{\"id\":289499,\"name\":\"pezra/parallel-each\",\"url\":\"https://api.github.com/repos/pezra/parallel-each\"},\"payload\":{\"forkee\":{\"id\":28678226,\"name\":\"parallel-each\",\"full_name\":\"dwmorgan/parallel-each\",\"owner\":{\"login\":\"dwmorgan\",\"id\":31330,\"avatar_url\":\"https://avatars.githubusercontent.com/u/31330?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dwmorgan\",\"html_url\":\"https://github.com/dwmorgan\",\"followers_url\":\"https://api.github.com/users/dwmorgan/followers\",\"following_url\":\"https://api.github.com/users/dwmorgan/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dwmorgan/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dwmorgan/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dwmorgan/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dwmorgan/orgs\",\"repos_url\":\"https://api.github.com/users/dwmorgan/repos\",\"events_url\":\"https://api.github.com/users/dwmorgan/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dwmorgan/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/dwmorgan/parallel-each\",\"description\":\"A mechanism for concurrently iterating over the items in Enumerables\",\"fork\":true,\"url\":\"https://api.github.com/repos/dwmorgan/parallel-each\",\"forks_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/forks\",\"keys_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/teams\",\"hooks_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/hooks\",\"issue_events_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/events\",\"assignees_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/tags\",\"blobs_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/languages\",\"stargazers_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/stargazers\",\"contributors_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/contributors\",\"subscribers_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/subscribers\",\"subscription_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/subscription\",\"commits_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/merges\",\"archive_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/downloads\",\"issues_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/dwmorgan/parallel-each/releases{/id}\",\"created_at\":\"2015-01-01T01:01:18Z\",\"updated_at\":\"2014-08-09T03:47:03Z\",\"pushed_at\":\"2009-09-14T23:08:58Z\",\"git_url\":\"git://github.com/dwmorgan/parallel-each.git\",\"ssh_url\":\"git@github.com:dwmorgan/parallel-each.git\",\"clone_url\":\"https://github.com/dwmorgan/parallel-each.git\",\"svn_url\":\"https://github.com/dwmorgan/parallel-each\",\"homepage\":\"\",\"size\":87,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:01:19Z\"}\n{\"id\":\"2489396489\",\"type\":\"ForkEvent\",\"actor\":{\"id\":10361464,\"login\":\"addisonep\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/addisonep\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361464?\"},\"repo\":{\"id\":2293158,\"name\":\"rapid7/metasploit-framework\",\"url\":\"https://api.github.com/repos/rapid7/metasploit-framework\"},\"payload\":{\"forkee\":{\"id\":28678227,\"name\":\"metasploit-framework\",\"full_name\":\"addisonep/metasploit-framework\",\"owner\":{\"login\":\"addisonep\",\"id\":10361464,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361464?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/addisonep\",\"html_url\":\"https://github.com/addisonep\",\"followers_url\":\"https://api.github.com/users/addisonep/followers\",\"following_url\":\"https://api.github.com/users/addisonep/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/addisonep/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/addisonep/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/addisonep/subscriptions\",\"organizations_url\":\"https://api.github.com/users/addisonep/orgs\",\"repos_url\":\"https://api.github.com/users/addisonep/repos\",\"events_url\":\"https://api.github.com/users/addisonep/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/addisonep/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/addisonep/metasploit-framework\",\"description\":\"Metasploit Framework\",\"fork\":true,\"url\":\"https://api.github.com/repos/addisonep/metasploit-framework\",\"forks_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/forks\",\"keys_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/teams\",\"hooks_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/hooks\",\"issue_events_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/events\",\"assignees_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/tags\",\"blobs_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/languages\",\"stargazers_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/stargazers\",\"contributors_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/contributors\",\"subscribers_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/subscribers\",\"subscription_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/subscription\",\"commits_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/merges\",\"archive_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/downloads\",\"issues_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/addisonep/metasploit-framework/releases{/id}\",\"created_at\":\"2015-01-01T01:01:19Z\",\"updated_at\":\"2015-01-01T00:04:48Z\",\"pushed_at\":\"2015-01-01T00:02:05Z\",\"git_url\":\"git://github.com/addisonep/metasploit-framework.git\",\"ssh_url\":\"git@github.com:addisonep/metasploit-framework.git\",\"clone_url\":\"https://github.com/addisonep/metasploit-framework.git\",\"svn_url\":\"https://github.com/addisonep/metasploit-framework\",\"homepage\":\"http://www.metasploit.com/\",\"size\":1010071,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:01:20Z\",\"org\":{\"id\":1013671,\"login\":\"rapid7\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rapid7\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1013671?\"}}\n{\"id\":\"2489396501\",\"type\":\"PushEvent\",\"actor\":{\"id\":1625941,\"login\":\"nocarryr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nocarryr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1625941?\"},\"repo\":{\"id\":26180933,\"name\":\"nocarryr/node_mapper\",\"url\":\"https://api.github.com/repos/nocarryr/node_mapper\"},\"payload\":{\"push_id\":536752509,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"777997a518ce5e14439a4a91eef156531af3260e\",\"before\":\"48c138a0ad0b1511492d63e5f1513a4726f831e9\",\"commits\":[{\"sha\":\"d058ed5ac475b1287e513e1944ea4cc8276b000a\",\"author\":{\"email\":\"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@nomadic-recording.com\",\"name\":\"nocarryr\"},\"message\":\"Squashed 'node_mapper/nomadic_recording_lib/' changes from 95ea390..fad5239\\n\\nfad5239 add prefs file for eric-ide v6\\n1acd9f1 modified layout sizing for embed\\n\\ngit-subtree-dir: node_mapper/nomadic_recording_lib\\ngit-subtree-split: fad5239c8a8ce199abc1e7d11c8d199b719e39df\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nocarryr/node_mapper/commits/d058ed5ac475b1287e513e1944ea4cc8276b000a\"},{\"sha\":\"777997a518ce5e14439a4a91eef156531af3260e\",\"author\":{\"email\":\"1fa2ef4755a9226cb9a0a4840bd89b158ac71391@nomadic-recording.com\",\"name\":\"nocarryr\"},\"message\":\"Merge commit 'd058ed5ac475b1287e513e1944ea4cc8276b000a'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nocarryr/node_mapper/commits/777997a518ce5e14439a4a91eef156531af3260e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:21Z\"}\n{\"id\":\"2489396503\",\"type\":\"PushEvent\",\"actor\":{\"id\":6948206,\"login\":\"JamesHutchison\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JamesHutchison\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6948206?\"},\"repo\":{\"id\":24011719,\"name\":\"JamesHutchison/brython\",\"url\":\"https://api.github.com/repos/JamesHutchison/brython\"},\"payload\":{\"push_id\":536752511,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6364b4328ddb8e64ed9164f025196aaceaf830d3\",\"before\":\"19eb5015b968d8c865423573fcae3afe805f2434\",\"commits\":[{\"sha\":\"6364b4328ddb8e64ed9164f025196aaceaf830d3\",\"author\":{\"email\":\"3ead6fdcf55e8f3c93103fc571861d43d648edff@microsoft.com\",\"name\":\"James Hutchison\"},\"message\":\"Updated to latest copy of the repo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/JamesHutchison/brython/commits/6364b4328ddb8e64ed9164f025196aaceaf830d3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:22Z\"}\n{\"id\":\"2489396511\",\"type\":\"PushEvent\",\"actor\":{\"id\":1903079,\"login\":\"krizvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/krizvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1903079?\"},\"repo\":{\"id\":20293030,\"name\":\"VHAINNOVATIONS/Mental-Health-eScreening\",\"url\":\"https://api.github.com/repos/VHAINNOVATIONS/Mental-Health-eScreening\"},\"payload\":{\"push_id\":536752514,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/resolved\",\"head\":\"6af479cb61611efc8ce9449c28a3e0eaf6aa9fe0\",\"before\":\"59f67bcf971fad269926139596d22cd4e1f03e05\",\"commits\":[{\"sha\":\"df757ed8ad0c31af27c6099cc0c2d8818cb2ff6b\",\"author\":{\"email\":\"e0c3a02e4102d2b5ed70e51bf9a27df65846d12e@gmail.com\",\"name\":\"Khalid R. Rizvi\"},\"message\":\"t651 fixed json data response by plucking unncessary data and also Lint the section controller\",\"distinct\":false,\"url\":\"https://api.github.com/repos/VHAINNOVATIONS/Mental-Health-eScreening/commits/df757ed8ad0c31af27c6099cc0c2d8818cb2ff6b\"},{\"sha\":\"6af479cb61611efc8ce9449c28a3e0eaf6aa9fe0\",\"author\":{\"email\":\"e0c3a02e4102d2b5ed70e51bf9a27df65846d12e@gmail.com\",\"name\":\"Khalid R. Rizvi\"},\"message\":\"Merge branch 't651' into resolved\",\"distinct\":true,\"url\":\"https://api.github.com/repos/VHAINNOVATIONS/Mental-Health-eScreening/commits/6af479cb61611efc8ce9449c28a3e0eaf6aa9fe0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:24Z\",\"org\":{\"id\":1252476,\"login\":\"VHAINNOVATIONS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/VHAINNOVATIONS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1252476?\"}}\n{\"id\":\"2489396514\",\"type\":\"PushEvent\",\"actor\":{\"id\":1181205,\"login\":\"malekbr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/malekbr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1181205?\"},\"repo\":{\"id\":28657751,\"name\":\"malekbr/PeevedPenguinsTemplate-Spritebuilder\",\"url\":\"https://api.github.com/repos/malekbr/PeevedPenguinsTemplate-Spritebuilder\"},\"payload\":{\"push_id\":536752516,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"460a58f6adfc1f1b56a088c786717a53749f4cd7\",\"before\":\"3084e78941af25cb8d49f4a9483e17ad41bcfd3e\",\"commits\":[{\"sha\":\"460a58f6adfc1f1b56a088c786717a53749f4cd7\",\"author\":{\"email\":\"73d3febb659ea118d8f1f9f1ff5f5ae09f3ca6d6@Mings-MacBook-Air.local\",\"name\":\"Malek Ben Romdhane\"},\"message\":\"Automatically Committed for MakeGamesWithUs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/malekbr/PeevedPenguinsTemplate-Spritebuilder/commits/460a58f6adfc1f1b56a088c786717a53749f4cd7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:24Z\"}\n{\"id\":\"2489396516\",\"type\":\"PushEvent\",\"actor\":{\"id\":4444926,\"login\":\"freeweibo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/freeweibo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4444926?\"},\"repo\":{\"id\":10095561,\"name\":\"freeweibo/top\",\"url\":\"https://api.github.com/repos/freeweibo/top\"},\"payload\":{\"push_id\":536752517,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"83e98b10262f4a2ff93e33070453ab590be73eb2\",\"before\":\"dc98fbebd53e1dbc0499aca11e5d293b3330ae2e\",\"commits\":[{\"sha\":\"83e98b10262f4a2ff93e33070453ab590be73eb2\",\"author\":{\"email\":\"24bf68e341ce0fbd9259a5d51feed79682ea4eba@ec2-us-web2.(none)\",\"name\":\"Ubuntu\"},\"message\":\"auto\",\"distinct\":true,\"url\":\"https://api.github.com/repos/freeweibo/top/commits/83e98b10262f4a2ff93e33070453ab590be73eb2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:24Z\"}\n{\"id\":\"2489396517\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":6744175,\"login\":\"antonioortegajr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/antonioortegajr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6744175?\"},\"repo\":{\"id\":28573267,\"name\":\"antonioortegajr/beerfind.me\",\"url\":\"https://api.github.com/repos/antonioortegajr/beerfind.me\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/antonioortegajr/beerfind.me/issues/13\",\"labels_url\":\"https://api.github.com/repos/antonioortegajr/beerfind.me/issues/13/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/antonioortegajr/beerfind.me/issues/13/comments\",\"events_url\":\"https://api.github.com/repos/antonioortegajr/beerfind.me/issues/13/events\",\"html_url\":\"https://github.com/antonioortegajr/beerfind.me/issues/13\",\"id\":53210186,\"number\":13,\"title\":\"search for six beers instead of five.\",\"user\":{\"login\":\"antonioortegajr\",\"id\":6744175,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6744175?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/antonioortegajr\",\"html_url\":\"https://github.com/antonioortegajr\",\"followers_url\":\"https://api.github.com/users/antonioortegajr/followers\",\"following_url\":\"https://api.github.com/users/antonioortegajr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/antonioortegajr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/antonioortegajr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/antonioortegajr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/antonioortegajr/orgs\",\"repos_url\":\"https://api.github.com/users/antonioortegajr/repos\",\"events_url\":\"https://api.github.com/users/antonioortegajr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/antonioortegajr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:01:24Z\",\"updated_at\":\"2015-01-01T01:01:24Z\",\"closed_at\":null,\"body\":\"Currently 5 beers is the Max to  search for. This should be six and twelve after that.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:24Z\"}\n{\"id\":\"2489396526\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":10334102,\"login\":\"DaDominat0r\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DaDominat0r\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10334102?\"},\"repo\":{\"id\":16182383,\"name\":\"Wynncraft/Issues\",\"url\":\"https://api.github.com/repos/Wynncraft/Issues\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/1409\",\"labels_url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/1409/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/1409/comments\",\"events_url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/1409/events\",\"html_url\":\"https://github.com/Wynncraft/Issues/issues/1409\",\"id\":53013103,\"number\":1409,\"title\":\"WynnExcavation Site A Broken\",\"user\":{\"login\":\"DaDominat0r\",\"id\":10334102,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10334102?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DaDominat0r\",\"html_url\":\"https://github.com/DaDominat0r\",\"followers_url\":\"https://api.github.com/users/DaDominat0r/followers\",\"following_url\":\"https://api.github.com/users/DaDominat0r/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DaDominat0r/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DaDominat0r/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DaDominat0r/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DaDominat0r/orgs\",\"repos_url\":\"https://api.github.com/users/DaDominat0r/repos\",\"events_url\":\"https://api.github.com/users/DaDominat0r/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DaDominat0r/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-28T23:56:03Z\",\"updated_at\":\"2015-01-01T01:01:26Z\",\"closed_at\":\"2014-12-29T18:08:12Z\",\"body\":\"I cannot finish the WynnExcavation Site A quest. First I got the message to translate from Vade. I brought it to Tesha and she translated it. I then gave it back to Vade but I disconnected. I came back on with no translated paper and I could not finish the quest. I was also not able to use /fixquests to help me.\"},\"comment\":{\"url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/comments/68477243\",\"html_url\":\"https://github.com/Wynncraft/Issues/issues/1409#issuecomment-68477243\",\"issue_url\":\"https://api.github.com/repos/Wynncraft/Issues/issues/1409\",\"id\":68477243,\"user\":{\"login\":\"DaDominat0r\",\"id\":10334102,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10334102?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DaDominat0r\",\"html_url\":\"https://github.com/DaDominat0r\",\"followers_url\":\"https://api.github.com/users/DaDominat0r/followers\",\"following_url\":\"https://api.github.com/users/DaDominat0r/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DaDominat0r/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DaDominat0r/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DaDominat0r/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DaDominat0r/orgs\",\"repos_url\":\"https://api.github.com/users/DaDominat0r/repos\",\"events_url\":\"https://api.github.com/users/DaDominat0r/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DaDominat0r/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:26Z\",\"updated_at\":\"2015-01-01T01:01:26Z\",\"body\":\"So.. Can I get help on this or what?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:26Z\",\"org\":{\"id\":5337644,\"login\":\"Wynncraft\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Wynncraft\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5337644?\"}}\n{\"id\":\"2489396528\",\"type\":\"WatchEvent\",\"actor\":{\"id\":2519036,\"login\":\"scidom\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scidom\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2519036?\"},\"repo\":{\"id\":6201939,\"name\":\"mlemerre/l-lang\",\"url\":\"https://api.github.com/repos/mlemerre/l-lang\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:26Z\"}\n{\"id\":\"2489396531\",\"type\":\"PushEvent\",\"actor\":{\"id\":6232704,\"login\":\"nashpitre\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nashpitre\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6232704?\"},\"repo\":{\"id\":28620312,\"name\":\"nashpitre/nashpitre.github.io\",\"url\":\"https://api.github.com/repos/nashpitre/nashpitre.github.io\"},\"payload\":{\"push_id\":536752519,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"51b30814df5685fd473d136c26735a17cf577361\",\"before\":\"c819b12361e14ee82463ecd628fb30cfcbade895\",\"commits\":[{\"sha\":\"51b30814df5685fd473d136c26735a17cf577361\",\"author\":{\"email\":\"be30e9ada7478b999c213bbb3535884788d4bb46@me.com\",\"name\":\"nashpitre\"},\"message\":\"short\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nashpitre/nashpitre.github.io/commits/51b30814df5685fd473d136c26735a17cf577361\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:26Z\"}\n{\"id\":\"2489396532\",\"type\":\"PushEvent\",\"actor\":{\"id\":171072,\"login\":\"floydpink\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/floydpink\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/171072?\"},\"repo\":{\"id\":28069452,\"name\":\"floydpink/BhagavadGita-ios\",\"url\":\"https://api.github.com/repos/floydpink/BhagavadGita-ios\"},\"payload\":{\"push_id\":536752520,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"5a83c1baf8fb498f94821ee6c946cb6e9e2b9045\",\"before\":\"aac807139c1fdd163e6ca29e937d1c684c6db953\",\"commits\":[{\"sha\":\"be50a575a875bc3f19cf04fb114509a0ebf4f677\",\"author\":{\"email\":\"5f2a1115638e65cf53a275a1d015ff80305829a3@googlemail.com\",\"name\":\"Floyd Pink\"},\"message\":\"section detail is much better\",\"distinct\":true,\"url\":\"https://api.github.com/repos/floydpink/BhagavadGita-ios/commits/be50a575a875bc3f19cf04fb114509a0ebf4f677\"},{\"sha\":\"5a83c1baf8fb498f94821ee6c946cb6e9e2b9045\",\"author\":{\"email\":\"5f2a1115638e65cf53a275a1d015ff80305829a3@googlemail.com\",\"name\":\"Floyd Pink\"},\"message\":\"updated gita.json with sloka number\",\"distinct\":true,\"url\":\"https://api.github.com/repos/floydpink/BhagavadGita-ios/commits/5a83c1baf8fb498f94821ee6c946cb6e9e2b9045\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:27Z\"}\n{\"id\":\"2489396534\",\"type\":\"PushEvent\",\"actor\":{\"id\":8017734,\"login\":\"mikebutts\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mikebutts\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8017734?\"},\"repo\":{\"id\":27460929,\"name\":\"mikebutts/rgs\",\"url\":\"https://api.github.com/repos/mikebutts/rgs\"},\"payload\":{\"push_id\":536752521,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"15b9c4283de821b7f69199886593ebd231a11c41\",\"before\":\"99cbde7783ed2cf0ea1db35f8c2ca1bffadcac5f\",\"commits\":[{\"sha\":\"15b9c4283de821b7f69199886593ebd231a11c41\",\"author\":{\"email\":\"1cf700477a4cf35c86434038dff20d0fc84c9161@gmail.com\",\"name\":\"mikebutts\"},\"message\":\"Customized the views\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mikebutts/rgs/commits/15b9c4283de821b7f69199886593ebd231a11c41\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:27Z\"}\n{\"id\":\"2489396538\",\"type\":\"PushEvent\",\"actor\":{\"id\":8703251,\"login\":\"ameliamarie\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ameliamarie\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8703251?\"},\"repo\":{\"id\":23813546,\"name\":\"thearrow/ameliamarie\",\"url\":\"https://api.github.com/repos/thearrow/ameliamarie\"},\"payload\":{\"push_id\":536752526,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0229ea3cce319f0d61433d352c2b900c77a4762a\",\"before\":\"3471f07be1e4b322e72c7e1ede88a521139462cc\",\"commits\":[{\"sha\":\"0229ea3cce319f0d61433d352c2b900c77a4762a\",\"author\":{\"email\":\"54048cf8a75c1b0abf0231a8f9808799724944c4@gmail.com\",\"name\":\"Amelia Schumacher\"},\"message\":\"Added logo, decreased top margin\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thearrow/ameliamarie/commits/0229ea3cce319f0d61433d352c2b900c77a4762a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:28Z\"}\n{\"id\":\"2489396539\",\"type\":\"PushEvent\",\"actor\":{\"id\":6154548,\"login\":\"chrisanthropic\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chrisanthropic\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6154548?\"},\"repo\":{\"id\":28470344,\"name\":\"chrisanthropic/comical-jekyll-theme\",\"url\":\"https://api.github.com/repos/chrisanthropic/comical-jekyll-theme\"},\"payload\":{\"push_id\":536752527,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"16fe2da04c5558db261f6c5eb49bb07d805ebefc\",\"before\":\"ad3cceedf0f2687257655fc606f863da4639b4f9\",\"commits\":[{\"sha\":\"16fe2da04c5558db261f6c5eb49bb07d805ebefc\",\"author\":{\"email\":\"a9c72f90f6fdaaea784cb28e2da787f981b8fb43@gmail.com\",\"name\":\"Christopher Tarwater\"},\"message\":\"fix banner image url on banner documentation page\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chrisanthropic/comical-jekyll-theme/commits/16fe2da04c5558db261f6c5eb49bb07d805ebefc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:28Z\"}\n{\"id\":\"2489396540\",\"type\":\"PushEvent\",\"actor\":{\"id\":5784044,\"login\":\"Guad\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Guad\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5784044?\"},\"repo\":{\"id\":28678140,\"name\":\"Guad/pydeo\",\"url\":\"https://api.github.com/repos/Guad/pydeo\"},\"payload\":{\"push_id\":536752528,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e19a124b7d16dfc2bb3c53a25f6bed3d41df5dc0\",\"before\":\"286c1baf59bad94aa73e3e60bd63ad4646727022\",\"commits\":[{\"sha\":\"e19a124b7d16dfc2bb3c53a25f6bed3d41df5dc0\",\"author\":{\"email\":\"45325f93c3023a3d37b7fdabaeabd7a663a476f8@gmail.com\",\"name\":\"Phil\"},\"message\":\"Added starting files\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Guad/pydeo/commits/e19a124b7d16dfc2bb3c53a25f6bed3d41df5dc0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:28Z\"}\n{\"id\":\"2489396542\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752529,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"22a9bd6584d30cc259b5b0055cb6a8bff31d8d9e\",\"before\":\"9b5858f96f8d101afa8e568dbe8424c32d010166\",\"commits\":[{\"sha\":\"22a9bd6584d30cc259b5b0055cb6a8bff31d8d9e\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"gabor function\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/22a9bd6584d30cc259b5b0055cb6a8bff31d8d9e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:29Z\"}\n{\"id\":\"2489396543\",\"type\":\"PushEvent\",\"actor\":{\"id\":1967266,\"login\":\"ogupte\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ogupte\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1967266?\"},\"repo\":{\"id\":28677997,\"name\":\"ogupte/trope\",\"url\":\"https://api.github.com/repos/ogupte/trope\"},\"payload\":{\"push_id\":536752531,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"481e49afd08fbfd366fc7a88674b3a1d669c96ce\",\"before\":\"e9e440aaa8897bbc0de676772f236f89c53c9ecb\",\"commits\":[{\"sha\":\"481e49afd08fbfd366fc7a88674b3a1d669c96ce\",\"author\":{\"email\":\"bd02fa4a4ee63436bd157864cccdf8b3a026562b@appnexus.com\",\"name\":\"Oliver Gupte\"},\"message\":\"fixed README formatting to conform with github markdown standards\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ogupte/trope/commits/481e49afd08fbfd366fc7a88674b3a1d669c96ce\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:29Z\"}\n{\"id\":\"2489396545\",\"type\":\"PushEvent\",\"actor\":{\"id\":6436073,\"login\":\"523860169\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/523860169\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6436073?\"},\"repo\":{\"id\":23783874,\"name\":\"523860169/list\",\"url\":\"https://api.github.com/repos/523860169/list\"},\"payload\":{\"push_id\":536752532,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f27f319fd966f774f9a14aad012414686f2285f2\",\"before\":\"639e5ab48e2bb80c20c195c456783b0e29c97838\",\"commits\":[{\"sha\":\"f27f319fd966f774f9a14aad012414686f2285f2\",\"author\":{\"email\":\"10f258ee912bdd79d094034bf50cfe8c68b73bc7@gmail.com\",\"name\":\"523860169\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/523860169/list/commits/f27f319fd966f774f9a14aad012414686f2285f2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:29Z\"}\n{\"id\":\"2489396549\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":541713,\"login\":\"aaronpeterson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aaronpeterson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/541713?\"},\"repo\":{\"id\":15320781,\"name\":\"paulyoder/angular-bootstrap-show-errors\",\"url\":\"https://api.github.com/repos/paulyoder/angular-bootstrap-show-errors\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/paulyoder/angular-bootstrap-show-errors/issues/30\",\"labels_url\":\"https://api.github.com/repos/paulyoder/angular-bootstrap-show-errors/issues/30/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/paulyoder/angular-bootstrap-show-errors/issues/30/comments\",\"events_url\":\"https://api.github.com/repos/paulyoder/angular-bootstrap-show-errors/issues/30/events\",\"html_url\":\"https://github.com/paulyoder/angular-bootstrap-show-errors/issues/30\",\"id\":53210190,\"number\":30,\"title\":\"Using inside of a directive\",\"user\":{\"login\":\"aaronpeterson\",\"id\":541713,\"avatar_url\":\"https://avatars.githubusercontent.com/u/541713?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aaronpeterson\",\"html_url\":\"https://github.com/aaronpeterson\",\"followers_url\":\"https://api.github.com/users/aaronpeterson/followers\",\"following_url\":\"https://api.github.com/users/aaronpeterson/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/aaronpeterson/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/aaronpeterson/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/aaronpeterson/subscriptions\",\"organizations_url\":\"https://api.github.com/users/aaronpeterson/orgs\",\"repos_url\":\"https://api.github.com/users/aaronpeterson/repos\",\"events_url\":\"https://api.github.com/users/aaronpeterson/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/aaronpeterson/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:01:29Z\",\"updated_at\":\"2015-01-01T01:01:29Z\",\"closed_at\":null,\"body\":\"Should show-errors work inside of another (parent) directive's template?  I see 'formCtrl' being injected as a result of require: '^form' but the instance of formCtrl is missing fields.  This results in formCtrl[inputName] being undefined on [line 39](https://github.com/paulyoder/angular-bootstrap-show-errors/blob/master/src/showErrors.js#L39) when the event is triggered.  \\r\\n\\r\\nMy parent directive uses templateUrl, so not sure if related to [this issue](http://stackoverflow.com/questions/22535317/using-directive-inside-a-templateurl-attribute).  Still digging.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:29Z\"}\n{\"id\":\"2489396550\",\"type\":\"PushEvent\",\"actor\":{\"id\":3949826,\"login\":\"tschudin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tschudin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3949826?\"},\"repo\":{\"id\":8971938,\"name\":\"cn-uofbasel/ccn-lite\",\"url\":\"https://api.github.com/repos/cn-uofbasel/ccn-lite\"},\"payload\":{\"push_id\":536752533,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/forwarder\",\"head\":\"21284a42e4588d65ec5b763cee8e728d9d398e16\",\"before\":\"3380f880670b473df5069028a4622e9aedcd5521\",\"commits\":[{\"sha\":\"21284a42e4588d65ec5b763cee8e728d9d398e16\",\"author\":{\"email\":\"53311a286a42748871239dcb38bda6322d2809ef@l2apps.com\",\"name\":\"Christian F. Tschudin\"},\"message\":\"purged all duplicate fields in ccnl_content_s, ccnl_interest_s still pending\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cn-uofbasel/ccn-lite/commits/21284a42e4588d65ec5b763cee8e728d9d398e16\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:29Z\",\"org\":{\"id\":3949801,\"login\":\"cn-uofbasel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cn-uofbasel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3949801?\"}}\n{\"id\":\"2489396557\",\"type\":\"PushEvent\",\"actor\":{\"id\":2099320,\"login\":\"cindyker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cindyker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2099320?\"},\"repo\":{\"id\":28552874,\"name\":\"cindyker/Crafty\",\"url\":\"https://api.github.com/repos/cindyker/Crafty\"},\"payload\":{\"push_id\":536752535,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cd304b11129fa0888ead7673b3d83336033585c5\",\"before\":\"93fd373cb7d26ac0e201680ac33cfc831ac6686e\",\"commits\":[{\"sha\":\"cd304b11129fa0888ead7673b3d83336033585c5\",\"author\":{\"email\":\"3a89c3a4a263b82f3475f9f2af701d5c4dba9e08@hotmail.com\",\"name\":\"cindyker\"},\"message\":\"pom update for heroes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cindyker/Crafty/commits/cd304b11129fa0888ead7673b3d83336033585c5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:32Z\"}\n{\"id\":\"2489396558\",\"type\":\"PushEvent\",\"actor\":{\"id\":735008,\"login\":\"kragniz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kragniz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/735008?\"},\"repo\":{\"id\":28620211,\"name\":\"kragniz/json-sempai\",\"url\":\"https://api.github.com/repos/kragniz/json-sempai\"},\"payload\":{\"push_id\":536752536,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b96375864c7b6e5868a18833a2bc226d8dbbf275\",\"before\":\"bdf617ecb4a1465ce5d392c65d98165e6652c7e2\",\"commits\":[{\"sha\":\"b96375864c7b6e5868a18833a2bc226d8dbbf275\",\"author\":{\"email\":\"c6a2862a65fb525f43531512880f69e2460922c9@gmail.com\",\"name\":\"Louis Taylor\"},\"message\":\"Update description in setup.py\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kragniz/json-sempai/commits/b96375864c7b6e5868a18833a2bc226d8dbbf275\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:32Z\"}\n{\"id\":\"2489396565\",\"type\":\"PushEvent\",\"actor\":{\"id\":3781771,\"login\":\"hwchen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hwchen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3781771?\"},\"repo\":{\"id\":26837590,\"name\":\"hwchen/euler-hs\",\"url\":\"https://api.github.com/repos/hwchen/euler-hs\"},\"payload\":{\"push_id\":536752541,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"f03bbb2e8b1ea9c47bbc4b16db55ce9c3c55ae12\",\"before\":\"5e8b1b964b46fff4d9c16f073dded632719d044d\",\"commits\":[{\"sha\":\"ef089e1f30c63552f274a6bbb7da24584f464045\",\"author\":{\"email\":\"731c5ac76cbb5cd3515a328d1315fcf27dc3a4f9@gmail.com\",\"name\":\"hwchen\"},\"message\":\"finished euler 50\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hwchen/euler-hs/commits/ef089e1f30c63552f274a6bbb7da24584f464045\"},{\"sha\":\"8378498984bf415a4ed61877babcaa2cc4cc9f13\",\"author\":{\"email\":\"731c5ac76cbb5cd3515a328d1315fcf27dc3a4f9@gmail.com\",\"name\":\"hwchen\"},\"message\":\"finished euler 57\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hwchen/euler-hs/commits/8378498984bf415a4ed61877babcaa2cc4cc9f13\"},{\"sha\":\"f03bbb2e8b1ea9c47bbc4b16db55ce9c3c55ae12\",\"author\":{\"email\":\"731c5ac76cbb5cd3515a328d1315fcf27dc3a4f9@gmail.com\",\"name\":\"hwchen\"},\"message\":\"finished euler 62\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hwchen/euler-hs/commits/f03bbb2e8b1ea9c47bbc4b16db55ce9c3c55ae12\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:32Z\"}\n{\"id\":\"2489396567\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":913757,\"login\":\"adarsh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/adarsh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/913757?\"},\"repo\":{\"id\":7411076,\"name\":\"thoughtbot/hound\",\"url\":\"https://api.github.com/repos/thoughtbot/hound\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/thoughtbot/hound/issues/527\",\"labels_url\":\"https://api.github.com/repos/thoughtbot/hound/issues/527/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/thoughtbot/hound/issues/527/comments\",\"events_url\":\"https://api.github.com/repos/thoughtbot/hound/issues/527/events\",\"html_url\":\"https://github.com/thoughtbot/hound/pull/527\",\"id\":52546556,\"number\":527,\"title\":\"Handle repos with inadequate information\",\"user\":{\"login\":\"adarsh\",\"id\":913757,\"avatar_url\":\"https://avatars.githubusercontent.com/u/913757?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/adarsh\",\"html_url\":\"https://github.com/adarsh\",\"followers_url\":\"https://api.github.com/users/adarsh/followers\",\"following_url\":\"https://api.github.com/users/adarsh/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/adarsh/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/adarsh/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/adarsh/subscriptions\",\"organizations_url\":\"https://api.github.com/users/adarsh/orgs\",\"repos_url\":\"https://api.github.com/users/adarsh/repos\",\"events_url\":\"https://api.github.com/users/adarsh/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/adarsh/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-20T01:24:36Z\",\"updated_at\":\"2015-01-01T01:01:32Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/thoughtbot/hound/pulls/527\",\"html_url\":\"https://github.com/thoughtbot/hound/pull/527\",\"diff_url\":\"https://github.com/thoughtbot/hound/pull/527.diff\",\"patch_url\":\"https://github.com/thoughtbot/hound/pull/527.patch\"},\"body\":\"* Inactivate repositories which lack privacy or org information\\n* Add rake task to update repo information\\n\\nhttps://trello.com/c/oPFtmreZ/393-builds-are-being-run-for-active-repos-missing-privacy-and-organization-information\"},\"comment\":{\"url\":\"https://api.github.com/repos/thoughtbot/hound/issues/comments/68477246\",\"html_url\":\"https://github.com/thoughtbot/hound/pull/527#issuecomment-68477246\",\"issue_url\":\"https://api.github.com/repos/thoughtbot/hound/issues/527\",\"id\":68477246,\"user\":{\"login\":\"adarsh\",\"id\":913757,\"avatar_url\":\"https://avatars.githubusercontent.com/u/913757?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/adarsh\",\"html_url\":\"https://github.com/adarsh\",\"followers_url\":\"https://api.github.com/users/adarsh/followers\",\"following_url\":\"https://api.github.com/users/adarsh/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/adarsh/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/adarsh/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/adarsh/subscriptions\",\"organizations_url\":\"https://api.github.com/users/adarsh/orgs\",\"repos_url\":\"https://api.github.com/users/adarsh/repos\",\"events_url\":\"https://api.github.com/users/adarsh/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/adarsh/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:32Z\",\"updated_at\":\"2015-01-01T01:01:32Z\",\"body\":\"@gylaz Note that I had to add the new `schema.rb` file. Please doublecheck before I merge. 707b26b\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:32Z\",\"org\":{\"id\":6183,\"login\":\"thoughtbot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/thoughtbot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6183?\"}}\n{\"id\":\"2489396576\",\"type\":\"PushEvent\",\"actor\":{\"id\":376230,\"login\":\"tiandavis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tiandavis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/376230?\"},\"repo\":{\"id\":28674331,\"name\":\"tiandavis/caramel\",\"url\":\"https://api.github.com/repos/tiandavis/caramel\"},\"payload\":{\"push_id\":536752546,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"70b984b9789c97a222fb2a1cca637243f31dd977\",\"before\":\"975b81e7e10ead6dc0078fde03cf3f30079f2adf\",\"commits\":[{\"sha\":\"70b984b9789c97a222fb2a1cca637243f31dd977\",\"author\":{\"email\":\"e9142163b8d26080ab61ec5db295a45dad55c70e@gmail.com\",\"name\":\"Tian Davis\"},\"message\":\"Updated project readme.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tiandavis/caramel/commits/70b984b9789c97a222fb2a1cca637243f31dd977\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:33Z\"}\n{\"id\":\"2489396577\",\"type\":\"PushEvent\",\"actor\":{\"id\":814471,\"login\":\"swegener\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/swegener\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/814471?\"},\"repo\":{\"id\":17971324,\"name\":\"swegener/gentoo-portage\",\"url\":\"https://api.github.com/repos/swegener/gentoo-portage\"},\"payload\":{\"push_id\":536752547,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d2d7b600bdf168a351eae73f0df0d6c6feebc174\",\"before\":\"93c6eace483ddf68d6707b005bed915724da00b2\",\"commits\":[{\"sha\":\"d2d7b600bdf168a351eae73f0df0d6c6feebc174\",\"author\":{\"email\":\"18dc2ed701c57df81c0b5498c13767c232eb398f@stealer.net\",\"name\":\"Sven Wegener\"},\"message\":\"2015-01-01 00:36:52+00:00\",\"distinct\":true,\"url\":\"https://api.github.com/repos/swegener/gentoo-portage/commits/d2d7b600bdf168a351eae73f0df0d6c6feebc174\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:33Z\"}\n{\"id\":\"2489396581\",\"type\":\"WatchEvent\",\"actor\":{\"id\":8263721,\"login\":\"akatray\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/akatray\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8263721?\"},\"repo\":{\"id\":9524997,\"name\":\"glfw/glfw\",\"url\":\"https://api.github.com/repos/glfw/glfw\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:35Z\",\"org\":{\"id\":3905364,\"login\":\"glfw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/glfw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3905364?\"}}\n{\"id\":\"2489396582\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":874715,\"login\":\"jenkinsadmin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jenkinsadmin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/874715?\"},\"repo\":{\"id\":1488959,\"name\":\"jenkinsci/periodicbackup-plugin\",\"url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/9\",\"labels_url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/9/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/9/comments\",\"events_url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/9/events\",\"html_url\":\"https://github.com/jenkinsci/periodicbackup-plugin/pull/9\",\"id\":53200776,\"number\":9,\"title\":\"Don't follow symlinks.\",\"user\":{\"login\":\"jikamens\",\"id\":1598067,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1598067?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jikamens\",\"html_url\":\"https://github.com/jikamens\",\"followers_url\":\"https://api.github.com/users/jikamens/followers\",\"following_url\":\"https://api.github.com/users/jikamens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jikamens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jikamens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jikamens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jikamens/orgs\",\"repos_url\":\"https://api.github.com/users/jikamens/repos\",\"events_url\":\"https://api.github.com/users/jikamens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jikamens/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T19:57:13Z\",\"updated_at\":\"2015-01-01T01:01:34Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/pulls/9\",\"html_url\":\"https://github.com/jenkinsci/periodicbackup-plugin/pull/9\",\"diff_url\":\"https://github.com/jenkinsci/periodicbackup-plugin/pull/9.diff\",\"patch_url\":\"https://github.com/jenkinsci/periodicbackup-plugin/pull/9.patch\"},\"body\":\"Backups shouldn't follow symbolic links.\\r\\n\\r\\nI haven't been able to actually compile and test this change, because it uses out-of-date packages, and my various efforts to update various packages all ended in failure, when I discovered that the latest versions of some of the required packages use google-collections (which is obsolete and no longer maintained), and this package itself uses guava instead, and google-collections and guava are incompatible.\\r\\n\\r\\nI don't know if anybody is maintaining this anymore. If so, then I think it needs some work to be compilable with an up-to-date java. My java mojo is not strong enough to get it done.\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/comments/68477247\",\"html_url\":\"https://github.com/jenkinsci/periodicbackup-plugin/pull/9#issuecomment-68477247\",\"issue_url\":\"https://api.github.com/repos/jenkinsci/periodicbackup-plugin/issues/9\",\"id\":68477247,\"user\":{\"login\":\"jenkinsadmin\",\"id\":874715,\"avatar_url\":\"https://avatars.githubusercontent.com/u/874715?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jenkinsadmin\",\"html_url\":\"https://github.com/jenkinsadmin\",\"followers_url\":\"https://api.github.com/users/jenkinsadmin/followers\",\"following_url\":\"https://api.github.com/users/jenkinsadmin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jenkinsadmin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jenkinsadmin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jenkinsadmin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jenkinsadmin/orgs\",\"repos_url\":\"https://api.github.com/users/jenkinsadmin/repos\",\"events_url\":\"https://api.github.com/users/jenkinsadmin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jenkinsadmin/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:34Z\",\"updated_at\":\"2015-01-01T01:01:34Z\",\"body\":\"Thank you for a pull request! Please check [this document](http://jenkins-ci.org/pull-request-greeting) for how the Jenkins project handles pull requests\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:35Z\",\"org\":{\"id\":107424,\"login\":\"jenkinsci\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/jenkinsci\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/107424?\"}}\n{\"id\":\"2489396583\",\"type\":\"CreateEvent\",\"actor\":{\"id\":8005041,\"login\":\"quynk94\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/quynk94\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8005041?\"},\"repo\":{\"id\":28678228,\"name\":\"quynk94/fedora-WP-Theme-\",\"url\":\"https://api.github.com/repos/quynk94/fedora-WP-Theme-\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:36Z\"}\n{\"id\":\"2489396585\",\"type\":\"PushEvent\",\"actor\":{\"id\":10225575,\"login\":\"ExclusiveOrange\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ExclusiveOrange\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10225575?\"},\"repo\":{\"id\":28677579,\"name\":\"ExclusiveOrange/synthesizer\",\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer\"},\"payload\":{\"push_id\":536752550,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"before\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:01:36Z\"}\n{\"id\":\"2489396597\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536752555,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d989a60a4e050573d9ba85048499d9a0a7e8117e\",\"before\":\"263e28e0e1c63e70b3ba9fd7b157945a394c6d14\",\"commits\":[{\"sha\":\"d989a60a4e050573d9ba85048499d9a0a7e8117e\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"fix bug with attribute bindings because forInnerPath wasn't being passed\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/d989a60a4e050573d9ba85048499d9a0a7e8117e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:37Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489396602\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1432632,\"login\":\"ajubbal\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ajubbal\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1432632?\"},\"repo\":{\"id\":629921,\"name\":\"bobthecow/git-flow-completion\",\"url\":\"https://api.github.com/repos/bobthecow/git-flow-completion\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/bobthecow/git-flow-completion/issues/37\",\"labels_url\":\"https://api.github.com/repos/bobthecow/git-flow-completion/issues/37/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/bobthecow/git-flow-completion/issues/37/comments\",\"events_url\":\"https://api.github.com/repos/bobthecow/git-flow-completion/issues/37/events\",\"html_url\":\"https://github.com/bobthecow/git-flow-completion/issues/37\",\"id\":53210191,\"number\":37,\"title\":\"Git Completion No Longer Working After Migration Between Computers\",\"user\":{\"login\":\"ajubbal\",\"id\":1432632,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1432632?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ajubbal\",\"html_url\":\"https://github.com/ajubbal\",\"followers_url\":\"https://api.github.com/users/ajubbal/followers\",\"following_url\":\"https://api.github.com/users/ajubbal/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ajubbal/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ajubbal/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ajubbal/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ajubbal/orgs\",\"repos_url\":\"https://api.github.com/users/ajubbal/repos\",\"events_url\":\"https://api.github.com/users/ajubbal/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ajubbal/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:01:38Z\",\"updated_at\":\"2015-01-01T01:01:38Z\",\"closed_at\":null,\"body\":\"Everything is intact as I used the migration assistant, unless there is a requirement for Xcode command line tools or something of that nature? I installed this through Homebrew and it was working fine on my initial computer (running Mac OS 10.9.5) but ceases to work with my new machine (same OS version). Some help would be greatly appreciated.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:38Z\"}\n{\"id\":\"2489396606\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5061849,\"login\":\"newbin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/newbin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5061849?\"},\"repo\":{\"id\":661420,\"name\":\"simplegeo/python-geohash\",\"url\":\"https://api.github.com/repos/simplegeo/python-geohash\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:39Z\",\"org\":{\"id\":93415,\"login\":\"simplegeo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/simplegeo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/93415?\"}}\n{\"id\":\"2489396609\",\"type\":\"PushEvent\",\"actor\":{\"id\":8933459,\"login\":\"lmontopo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lmontopo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8933459?\"},\"repo\":{\"id\":24911711,\"name\":\"lmontopo/lmontopo.github.io\",\"url\":\"https://api.github.com/repos/lmontopo/lmontopo.github.io\"},\"payload\":{\"push_id\":536752559,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/source\",\"head\":\"7812248ef9626c3f04dcb73f41eab673e6cb3691\",\"before\":\"5f6068e7c82948857bb919d278793bed517d069f\",\"commits\":[{\"sha\":\"7812248ef9626c3f04dcb73f41eab673e6cb3691\",\"author\":{\"email\":\"6beaeb38ccda0977b766148789e916197124dfed@gmail.com\",\"name\":\"Leta Montopoli\"},\"message\":\"edited lessons.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lmontopo/lmontopo.github.io/commits/7812248ef9626c3f04dcb73f41eab673e6cb3691\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:39Z\"}\n{\"id\":\"2489396616\",\"type\":\"CreateEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"ref\":\"#30\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Set of packages used to measure a mechanical clock.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:40Z\"}\n{\"id\":\"2489396617\",\"type\":\"PushEvent\",\"actor\":{\"id\":4390732,\"login\":\"iamchairs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/iamchairs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4390732?\"},\"repo\":{\"id\":20004313,\"name\":\"iamchairs/snooze\",\"url\":\"https://api.github.com/repos/iamchairs/snooze\"},\"payload\":{\"push_id\":536752563,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/1.0.0-alpha.1\",\"head\":\"9f289c1a6fd9b85e213e0a7104b2a2bfe6ee4104\",\"before\":\"f9f2212d389e344e61c32f961ef77adbdef23bb3\",\"commits\":[{\"sha\":\"9f289c1a6fd9b85e213e0a7104b2a2bfe6ee4104\",\"author\":{\"email\":\"1de9ff26e599b349bd443b5cc72b005ab8ff1e50@gmail.com\",\"name\":\"Micah Williamson\"},\"message\":\"return this on everything\",\"distinct\":true,\"url\":\"https://api.github.com/repos/iamchairs/snooze/commits/9f289c1a6fd9b85e213e0a7104b2a2bfe6ee4104\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:40Z\"}\n{\"id\":\"2489396620\",\"type\":\"PushEvent\",\"actor\":{\"id\":4115801,\"login\":\"afeinland\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/afeinland\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4115801?\"},\"repo\":{\"id\":16446548,\"name\":\"afeinland/vimrc\",\"url\":\"https://api.github.com/repos/afeinland/vimrc\"},\"payload\":{\"push_id\":536752565,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dc3aa0c350eb43be8cd66bc83b053bf84ed4364a\",\"before\":\"ed20c634809a7a3375a41b819143f5bf64389bd9\",\"commits\":[{\"sha\":\"dc3aa0c350eb43be8cd66bc83b053bf84ed4364a\",\"author\":{\"email\":\"de080937e87a3fc5b57c1b6f90bb5aded8f69508@ucr.edu\",\"name\":\"Alex Feinland\"},\"message\":\"added set ruler rule\",\"distinct\":true,\"url\":\"https://api.github.com/repos/afeinland/vimrc/commits/dc3aa0c350eb43be8cd66bc83b053bf84ed4364a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:41Z\"}\n{\"id\":\"2489396621\",\"type\":\"PushEvent\",\"actor\":{\"id\":8699738,\"login\":\"harryganz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harryganz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8699738?\"},\"repo\":{\"id\":28651515,\"name\":\"harryganz/ganzfacts\",\"url\":\"https://api.github.com/repos/harryganz/ganzfacts\"},\"payload\":{\"push_id\":536752566,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"df578268ecc281cfadd96b2f884d19b3562c70e1\",\"before\":\"0ffe15d38852b93efab774b6d51442c6fd9633e4\",\"commits\":[{\"sha\":\"df578268ecc281cfadd96b2f884d19b3562c70e1\",\"author\":{\"email\":\"8e8abe39cdbc49fc743305288c23722c02c5f1c2@rsmas.miami.edu\",\"name\":\"Harry Ganz\"},\"message\":\"Added controllers\\n\\nAdded controllers for access, admin, fact, photo, post\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harryganz/ganzfacts/commits/df578268ecc281cfadd96b2f884d19b3562c70e1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:41Z\"}\n{\"id\":\"2489396622\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536752567,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c64c0c7b60fea3b7a191f3b8b39aa144d281ceba\",\"before\":\"8c339f97ba859406034a4b4803ff9f10986c16ea\",\"commits\":[{\"sha\":\"c64c0c7b60fea3b7a191f3b8b39aa144d281ceba\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/c64c0c7b60fea3b7a191f3b8b39aa144d281ceba\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:42Z\"}\n{\"id\":\"2489396624\",\"type\":\"PushEvent\",\"actor\":{\"id\":3124267,\"login\":\"acprimer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/acprimer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3124267?\"},\"repo\":{\"id\":28179725,\"name\":\"acprimer/Beauty-of-Programming\",\"url\":\"https://api.github.com/repos/acprimer/Beauty-of-Programming\"},\"payload\":{\"push_id\":536752568,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8d605c3d782cf051a9b03b55b619a9525c3d5add\",\"before\":\"a10e8208f5e136f1ba4c2ae16f56cee76f0e563d\",\"commits\":[{\"sha\":\"8d605c3d782cf051a9b03b55b619a9525c3d5add\",\"author\":{\"email\":\"621c6ca061921149bf4dbea85ceffd5a4497c1dc@qq.com\",\"name\":\"Dahai Yao\"},\"message\":\"chap2.17\",\"distinct\":true,\"url\":\"https://api.github.com/repos/acprimer/Beauty-of-Programming/commits/8d605c3d782cf051a9b03b55b619a9525c3d5add\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:42Z\"}\n{\"id\":\"2489396625\",\"type\":\"PushEvent\",\"actor\":{\"id\":1221092,\"login\":\"vuolter\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vuolter\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1221092?\"},\"repo\":{\"id\":8122790,\"name\":\"pyload/pyload\",\"url\":\"https://api.github.com/repos/pyload/pyload\"},\"payload\":{\"push_id\":536752569,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/stable\",\"head\":\"1f1e5cd57b08353c5f42add8f9b842c4926ba8ed\",\"before\":\"60e9c46f32d97d01d728c8515985b58ba33fdafd\",\"commits\":[{\"sha\":\"1f1e5cd57b08353c5f42add8f9b842c4926ba8ed\",\"author\":{\"email\":\"5cc667be46e22897a880e49707c8ded7041bc0f9@gmail.com\",\"name\":\"Walter Purcaro\"},\"message\":\"[RapiduNet] Fixup\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pyload/pyload/commits/1f1e5cd57b08353c5f42add8f9b842c4926ba8ed\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:42Z\",\"org\":{\"id\":3521496,\"login\":\"pyload\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/pyload\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3521496?\"}}\n{\"id\":\"2489396626\",\"type\":\"PushEvent\",\"actor\":{\"id\":935160,\"login\":\"thenomain\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/thenomain\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/935160?\"},\"repo\":{\"id\":26726490,\"name\":\"thenomain/GMCCG\",\"url\":\"https://api.github.com/repos/thenomain/GMCCG\"},\"payload\":{\"push_id\":536752570,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"32a8000020397a3c786d167233dfab8397cf2735\",\"before\":\"265550d7fa10002cee34e64effe1dff2f0461287\",\"commits\":[{\"sha\":\"5ef5e29232c7bfb607eef378bb9f9e18e0929819\",\"author\":{\"email\":\"f2f79094bfbbb4cfcb0b5f9d1737b6d1f43b0201@gmail.com\",\"name\":\"Kent Jenkins\"},\"message\":\"more class-system overhaul\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thenomain/GMCCG/commits/5ef5e29232c7bfb607eef378bb9f9e18e0929819\"},{\"sha\":\"32a8000020397a3c786d167233dfab8397cf2735\",\"author\":{\"email\":\"f2f79094bfbbb4cfcb0b5f9d1737b6d1f43b0201@gmail.com\",\"name\":\"Kent Jenkins\"},\"message\":\"separate tags out to own object\\n\\nTags are a huge portion of our lookup information, so I'm hoping that by taking them out of the main Data Dictionary and to their own object (Data Tags) that they will make lookup of most things quicker.\\n\\nThis paves the way in case Prerequisites, Book Reference, and Notes needs to do the same.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thenomain/GMCCG/commits/32a8000020397a3c786d167233dfab8397cf2735\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:42Z\"}\n{\"id\":\"2489396630\",\"type\":\"PushEvent\",\"actor\":{\"id\":6436073,\"login\":\"523860169\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/523860169\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6436073?\"},\"repo\":{\"id\":27432101,\"name\":\"523860169/sh\",\"url\":\"https://api.github.com/repos/523860169/sh\"},\"payload\":{\"push_id\":536752572,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f5565a7f3a75c7fca3b11480b24c02b70dd64f72\",\"before\":\"55d430ff46d9023d0cdc935212ffa50328920a11\",\"commits\":[{\"sha\":\"f5565a7f3a75c7fca3b11480b24c02b70dd64f72\",\"author\":{\"email\":\"10f258ee912bdd79d094034bf50cfe8c68b73bc7@gmail.com\",\"name\":\"523860169\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/523860169/sh/commits/f5565a7f3a75c7fca3b11480b24c02b70dd64f72\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:43Z\"}\n{\"id\":\"2489396631\",\"type\":\"PushEvent\",\"actor\":{\"id\":367618,\"login\":\"cm-gerrit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cm-gerrit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/367618?\"},\"repo\":{\"id\":26532168,\"name\":\"CyanogenMod/android_device_qcom_sepolicy\",\"url\":\"https://api.github.com/repos/CyanogenMod/android_device_qcom_sepolicy\"},\"payload\":{\"push_id\":536752573,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/cm-12.0\",\"head\":\"77bb91189b6200c7743acad40b8cc1fb6fe32bc0\",\"before\":\"233406f509e344c11d9a39a647cd6381afa42304\",\"commits\":[{\"sha\":\"77bb91189b6200c7743acad40b8cc1fb6fe32bc0\",\"author\":{\"email\":\"9ce5770b3bb4b2a1d59be2d97e34379cd192299f@cyngn.com\",\"name\":\"Steve Kondik\"},\"message\":\"sepolicy: Allow apps to read battery status\\n\\n * Various apps do this to monitor the battery. No harm there.\\n\\nChange-Id: Id1b843ca509747ed963b89d025a39b5b1fcc7ddb\",\"distinct\":true,\"url\":\"https://api.github.com/repos/CyanogenMod/android_device_qcom_sepolicy/commits/77bb91189b6200c7743acad40b8cc1fb6fe32bc0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:43Z\",\"org\":{\"id\":317721,\"login\":\"CyanogenMod\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/CyanogenMod\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/317721?\"}}\n{\"id\":\"2489396635\",\"type\":\"PushEvent\",\"actor\":{\"id\":5975070,\"login\":\"marcellodibello\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marcellodibello\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5975070?\"},\"repo\":{\"id\":17580814,\"name\":\"marcellodibello/marcellodibello.github.io\",\"url\":\"https://api.github.com/repos/marcellodibello/marcellodibello.github.io\"},\"payload\":{\"push_id\":536752575,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7995cfd2d91a5b11eff75239ac124fae9af80d6d\",\"before\":\"b5fdeaa9282cb3b94536ea0a2485e7b154c33d0a\",\"commits\":[{\"sha\":\"7995cfd2d91a5b11eff75239ac124fae9af80d6d\",\"author\":{\"email\":\"babd00909fb78351e79d40af26c77fc37bb4ed59@gmail.com\",\"name\":\"marcellodibello\"},\"message\":\"Update index.html\",\"distinct\":true,\"url\":\"https://api.github.com/repos/marcellodibello/marcellodibello.github.io/commits/7995cfd2d91a5b11eff75239ac124fae9af80d6d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:43Z\"}\n{\"id\":\"2489396636\",\"type\":\"PushEvent\",\"actor\":{\"id\":8819701,\"login\":\"r-ggraham\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r-ggraham\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8819701?\"},\"repo\":{\"id\":28678173,\"name\":\"r-ggraham/Crumpet_Bot\",\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot\"},\"payload\":{\"push_id\":536752576,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"332746658edcc60bca17720fce153357a2cd9dfb\",\"before\":\"63b341a1c133b56bfaf703a2fed167478e1e4ed3\",\"commits\":[{\"sha\":\"332746658edcc60bca17720fce153357a2cd9dfb\",\"author\":{\"email\":\"f2f9dd43aa4244d32208a2ccfa0c7c9e9c48f7e7@uni.worc.ac.uk\",\"name\":\"Rob G\"},\"message\":\"Credits\\n\\nAdded comment credits\",\"distinct\":true,\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot/commits/332746658edcc60bca17720fce153357a2cd9dfb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:43Z\"}\n{\"id\":\"2489396639\",\"type\":\"PushEvent\",\"actor\":{\"id\":869882,\"login\":\"dpt\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpt\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/869882?\"},\"repo\":{\"id\":27767031,\"name\":\"dpt/DPTLib\",\"url\":\"https://api.github.com/repos/dpt/DPTLib\"},\"payload\":{\"push_id\":536752579,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/master\",\"head\":\"4517cde2c28d1ace039a16ed0d5e08859768602a\",\"before\":\"e43b8c4ccd99f455062cce4952f013bcf986482a\",\"commits\":[{\"sha\":\"88577c3778df7df077f6dd4323e59b8b9b41b688\",\"author\":{\"email\":\"bfcdf3e6ca6cef45543bfbb57509c92aec9a39fb@davespace.co.uk\",\"name\":\"David Thomas\"},\"message\":\"datastruct/vector: Add vector_insert, vector_ensure.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpt/DPTLib/commits/88577c3778df7df077f6dd4323e59b8b9b41b688\"},{\"sha\":\"b93b96d5db4bd7265a141337c0ac6979f4d8398c\",\"author\":{\"email\":\"bfcdf3e6ca6cef45543bfbb57509c92aec9a39fb@davespace.co.uk\",\"name\":\"David Thomas\"},\"message\":\"datastruct/vector: vector_insert()'s element to insert can be const *.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpt/DPTLib/commits/b93b96d5db4bd7265a141337c0ac6979f4d8398c\"},{\"sha\":\"51a8c9129b2983ac5486eda95484ac549f84e60d\",\"author\":{\"email\":\"bfcdf3e6ca6cef45543bfbb57509c92aec9a39fb@davespace.co.uk\",\"name\":\"David Thomas\"},\"message\":\"databases/digest-db: Mild rewrite of header prologue.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpt/DPTLib/commits/51a8c9129b2983ac5486eda95484ac549f84e60d\"},{\"sha\":\"4517cde2c28d1ace039a16ed0d5e08859768602a\",\"author\":{\"email\":\"bfcdf3e6ca6cef45543bfbb57509c92aec9a39fb@davespace.co.uk\",\"name\":\"David Thomas\"},\"message\":\"DPTLib.xccheckout: Remove stale repo refs.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpt/DPTLib/commits/4517cde2c28d1ace039a16ed0d5e08859768602a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:45Z\"}\n{\"id\":\"2489396640\",\"type\":\"CreateEvent\",\"actor\":{\"id\":790511,\"login\":\"rizumita\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rizumita\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/790511?\"},\"repo\":{\"id\":28678229,\"name\":\"rizumita/Ptarmigan\",\"url\":\"https://api.github.com/repos/rizumita/Ptarmigan\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"CoreData functions for Swift\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:45Z\"}\n{\"id\":\"2489396645\",\"type\":\"PushEvent\",\"actor\":{\"id\":904370,\"login\":\"helhum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/helhum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/904370?\"},\"repo\":{\"id\":21070637,\"name\":\"TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools\",\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools\"},\"payload\":{\"push_id\":536752582,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"03ce16ec5d355a60057fb57f8d27d7116422f722\",\"before\":\"e74bb4aec4a873f3a9ed58e829b7f9f908cfa544\",\"commits\":[{\"sha\":\"03ce16ec5d355a60057fb57f8d27d7116422f722\",\"author\":{\"email\":\"6bf857ca7de026fbed4ae790a809a0ea640901f4@helmuthummel.de\",\"name\":\"Helmut Hummel\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools/commits/03ce16ec5d355a60057fb57f8d27d7116422f722\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:46Z\",\"org\":{\"id\":7921669,\"login\":\"TYPO3-Surf-CMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TYPO3-Surf-CMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7921669?\"}}\n{\"id\":\"2489396647\",\"type\":\"PushEvent\",\"actor\":{\"id\":4359301,\"login\":\"inmWill\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/inmWill\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4359301?\"},\"repo\":{\"id\":27058289,\"name\":\"inmWill/inmNgLayouts\",\"url\":\"https://api.github.com/repos/inmWill/inmNgLayouts\"},\"payload\":{\"push_id\":536752584,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1b30dcdd5a034183a7b2da4c8abc8b4736ce144a\",\"before\":\"38d6079b791807f692845a7e665ff146cce78b2f\",\"commits\":[{\"sha\":\"1b30dcdd5a034183a7b2da4c8abc8b4736ce144a\",\"author\":{\"email\":\"b8d2c4785179c9d0f40139a5951d8a653241c7ef@inmerge.com\",\"name\":\"Farang\"},\"message\":\"removed test\",\"distinct\":true,\"url\":\"https://api.github.com/repos/inmWill/inmNgLayouts/commits/1b30dcdd5a034183a7b2da4c8abc8b4736ce144a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:47Z\"}\n{\"id\":\"2489396656\",\"type\":\"PushEvent\",\"actor\":{\"id\":2501598,\"login\":\"paciorek\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paciorek\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2501598?\"},\"repo\":{\"id\":17234057,\"name\":\"PalEON-Project/composition\",\"url\":\"https://api.github.com/repos/PalEON-Project/composition\"},\"payload\":{\"push_id\":536752590,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f355f83b85994252322ee2194d54576077d38c88\",\"before\":\"0cdff15a8f0387fe76196a3f05528b3128083d11\",\"commits\":[{\"sha\":\"f355f83b85994252322ee2194d54576077d38c88\",\"author\":{\"email\":\"cd2303627620d42aea1f002984decb2f80e8de9b@scf.Berkeley.EDU\",\"name\":\"Christopher Paciorek\"},\"message\":\"more cleaning of files/paths in R files;\\nadded packrat to repo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/PalEON-Project/composition/commits/f355f83b85994252322ee2194d54576077d38c88\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:47Z\",\"org\":{\"id\":5016491,\"login\":\"PalEON-Project\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/PalEON-Project\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5016491?\"}}\n{\"id\":\"2489396657\",\"type\":\"PushEvent\",\"actor\":{\"id\":3495129,\"login\":\"sundaymtn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sundaymtn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3495129?\"},\"repo\":{\"id\":24147122,\"name\":\"sundaymtn/waterline\",\"url\":\"https://api.github.com/repos/sundaymtn/waterline\"},\"payload\":{\"push_id\":536752591,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b0598d6ba34d42ad8cfe6b10a7567951ede389a0\",\"before\":\"44753191dc8f615ccda4f0afe31a09342172cfe4\",\"commits\":[{\"sha\":\"b0598d6ba34d42ad8cfe6b10a7567951ede389a0\",\"author\":{\"email\":\"7fbc091194a9488bfb16868527a7c3a8ba469dba@gmail.com\",\"name\":\"Seth Carter\"},\"message\":\"[skip ci] updated waterline data\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sundaymtn/waterline/commits/b0598d6ba34d42ad8cfe6b10a7567951ede389a0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:48Z\"}\n{\"id\":\"2489396659\",\"type\":\"PushEvent\",\"actor\":{\"id\":10345916,\"login\":\"robelgeda\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robelgeda\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10345916?\"},\"repo\":{\"id\":28621069,\"name\":\"robelgeda/robelgeda\",\"url\":\"https://api.github.com/repos/robelgeda/robelgeda\"},\"payload\":{\"push_id\":536752593,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"e454e00c6264adc11f7f0099e438399d8a5a88c8\",\"before\":\"86b6e2201b0f4fc2256234b9bfea74873e21ed90\",\"commits\":[{\"sha\":\"e454e00c6264adc11f7f0099e438399d8a5a88c8\",\"author\":{\"email\":\"26dc263e33784ddd01201ac6953f5dcb1af40498@yahoo.com\",\"name\":\"robelgeda\"},\"message\":\"New stuff\",\"distinct\":true,\"url\":\"https://api.github.com/repos/robelgeda/robelgeda/commits/e454e00c6264adc11f7f0099e438399d8a5a88c8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:49Z\"}\n{\"id\":\"2489396663\",\"type\":\"WatchEvent\",\"actor\":{\"id\":9970148,\"login\":\"bchoomnuan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchoomnuan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9970148?\"},\"repo\":{\"id\":21827146,\"name\":\"chrislusf/weed-fs\",\"url\":\"https://api.github.com/repos/chrislusf/weed-fs\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:50Z\"}\n{\"id\":\"2489396669\",\"type\":\"PushEvent\",\"actor\":{\"id\":7950744,\"login\":\"dmen555\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dmen555\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7950744?\"},\"repo\":{\"id\":28383621,\"name\":\"dmen555/SleepManager\",\"url\":\"https://api.github.com/repos/dmen555/SleepManager\"},\"payload\":{\"push_id\":536752597,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"689e3b010ca618cbb854cfc86f1f7763c3278569\",\"before\":\"867bc6099ada213768c16ab01bc3627da7b04ead\",\"commits\":[{\"sha\":\"689e3b010ca618cbb854cfc86f1f7763c3278569\",\"author\":{\"email\":\"d1512ec681e77465d7ece3f74330967627c1d339@gmail.com\",\"name\":\"David Qian\"},\"message\":\"Patch 7.1: Completed music timer except the graphs\\n\\n-successfully implemented the four different types of decay functions\\n-successfully implemented services\\n-successfully implemented sharedpreferences\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dmen555/SleepManager/commits/689e3b010ca618cbb854cfc86f1f7763c3278569\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:50Z\"}\n{\"id\":\"2489396676\",\"type\":\"PushEvent\",\"actor\":{\"id\":8548034,\"login\":\"Mortifica\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Mortifica\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8548034?\"},\"repo\":{\"id\":28653599,\"name\":\"Mortifica/DungeonTestArea\",\"url\":\"https://api.github.com/repos/Mortifica/DungeonTestArea\"},\"payload\":{\"push_id\":536752599,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"24beef29c0a3c297ab998b41726b048ec4bcaf08\",\"before\":\"e27d2f3c9eb7716c52e2dd8325e8d22bd5733560\",\"commits\":[{\"sha\":\"24beef29c0a3c297ab998b41726b048ec4bcaf08\",\"author\":{\"email\":\"c4d9a55a62f361685d2115e4dff82e20462062f9@gmail.com\",\"name\":\"Josh Hanke\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Mortifica/DungeonTestArea/commits/24beef29c0a3c297ab998b41726b048ec4bcaf08\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:52Z\"}\n{\"id\":\"2489396678\",\"type\":\"PushEvent\",\"actor\":{\"id\":2230058,\"login\":\"CrazyAlvaro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CrazyAlvaro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2230058?\"},\"repo\":{\"id\":28671592,\"name\":\"CrazyAlvaro/Coursera-Cryptography-I\",\"url\":\"https://api.github.com/repos/CrazyAlvaro/Coursera-Cryptography-I\"},\"payload\":{\"push_id\":536752601,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"76e0858cb983271caedcbefe194ec462269d958c\",\"before\":\"6112537f4a7b0b5d0792580d8a40a7998bacbea4\",\"commits\":[{\"sha\":\"76e0858cb983271caedcbefe194ec462269d958c\",\"author\":{\"email\":\"dad06db06a260749086cfe9a867182e7a593fa69@gmail.com\",\"name\":\"Yechen Huang\"},\"message\":\"[#85323898] Add Calendar schedule\",\"distinct\":true,\"url\":\"https://api.github.com/repos/CrazyAlvaro/Coursera-Cryptography-I/commits/76e0858cb983271caedcbefe194ec462269d958c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:52Z\"}\n{\"id\":\"2489396682\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536752605,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b3500640ddd547415a82cbc026d91be7f76ff14f\",\"before\":\"c64c0c7b60fea3b7a191f3b8b39aa144d281ceba\",\"commits\":[{\"sha\":\"b3500640ddd547415a82cbc026d91be7f76ff14f\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/b3500640ddd547415a82cbc026d91be7f76ff14f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:53Z\"}\n{\"id\":\"2489396684\",\"type\":\"PushEvent\",\"actor\":{\"id\":3964764,\"login\":\"elliekimpot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/elliekimpot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3964764?\"},\"repo\":{\"id\":28625867,\"name\":\"elliekimpot/msm\",\"url\":\"https://api.github.com/repos/elliekimpot/msm\"},\"payload\":{\"push_id\":536752606,\"size\":5,\"distinct_size\":5,\"ref\":\"refs/heads/feature/removed\",\"head\":\"2c854302b53eef234eae379576a870f9ef9aa3bd\",\"before\":\"b02f898e373a3b228ac7b5f815ffadb9a821e8f9\",\"commits\":[{\"sha\":\"8ae5cba2d0c6e13cdbb3682fa7cc70960cd95246\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_MORE_DEBUGGING_INFO_ON_KERNEL\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/8ae5cba2d0c6e13cdbb3682fa7cc70960cd95246\"},{\"sha\":\"1fae68014e73cfc3a32fa5eca33505b011ebb7b4\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"Revert \\\"add basic CFLAGS for Krait\\\"\\n\\nThis reverts commit 0011b847c3b6a0082306b3b29077bdc9550c6dda.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/1fae68014e73cfc3a32fa5eca33505b011ebb7b4\"},{\"sha\":\"ff109d56f2b1b87aada770208ba471ef9d085a70\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_DEBUG\\n\\nDrop following features\\n* PANTECH_DEBUG_ON\\n* PANTECH_DEBUG\\n* PANTECH_DEBUG_SCHED_LOG\\n* PANTECH_DEBUG_IRQ_LOG\\n* PANTECH_DEBUG_DCVS_LOG\\n* PANTECH_DEBUG_RPM_LOG\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/ff109d56f2b1b87aada770208ba471ef9d085a70\"},{\"sha\":\"c3d59ecafe5fda8c6776139e45c6c76eecbf046d\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_FS_AUTO_REPAIR\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/c3d59ecafe5fda8c6776139e45c6c76eecbf046d\"},{\"sha\":\"2c854302b53eef234eae379576a870f9ef9aa3bd\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_ERR_CRASH_LOGGING\\n\\nDop PANTECH_ERR_CRASH_LOGGING and introduce PANTECH_SYS\\ndue to proper build (Solve error/mipi_sony_incell.c).\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/2c854302b53eef234eae379576a870f9ef9aa3bd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:53Z\"}\n{\"id\":\"2489396685\",\"type\":\"PushEvent\",\"actor\":{\"id\":8908145,\"login\":\"jburgos1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jburgos1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8908145?\"},\"repo\":{\"id\":25492727,\"name\":\"jburgos1/fls1lambdas\",\"url\":\"https://api.github.com/repos/jburgos1/fls1lambdas\"},\"payload\":{\"push_id\":536752607,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ffc8f8f329910df1b92686097a402204a1c6a148\",\"before\":\"3b4244d64d7fe1637440dbefabfd907bbc45f696\",\"commits\":[{\"sha\":\"ffc8f8f329910df1b92686097a402204a1c6a148\",\"author\":{\"email\":\"a6f8acae3f5fd51a4ab59395a7512c8c769df47e@live.com\",\"name\":\"Juan Burgos\"},\"message\":\"CSS Changes\\n\\nNo more text/box shadow\\n\\nyuck.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jburgos1/fls1lambdas/commits/ffc8f8f329910df1b92686097a402204a1c6a148\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:53Z\"}\n{\"id\":\"2489396689\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1380147,\"login\":\"arathael\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/arathael\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1380147?\"},\"repo\":{\"id\":27724993,\"name\":\"homerjam/angular-ui-sref-fastclick\",\"url\":\"https://api.github.com/repos/homerjam/angular-ui-sref-fastclick\"},\"payload\":{\"forkee\":{\"id\":28678230,\"name\":\"angular-ui-sref-fastclick\",\"full_name\":\"arathael/angular-ui-sref-fastclick\",\"owner\":{\"login\":\"arathael\",\"id\":1380147,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1380147?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/arathael\",\"html_url\":\"https://github.com/arathael\",\"followers_url\":\"https://api.github.com/users/arathael/followers\",\"following_url\":\"https://api.github.com/users/arathael/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/arathael/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/arathael/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/arathael/subscriptions\",\"organizations_url\":\"https://api.github.com/users/arathael/orgs\",\"repos_url\":\"https://api.github.com/users/arathael/repos\",\"events_url\":\"https://api.github.com/users/arathael/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/arathael/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/arathael/angular-ui-sref-fastclick\",\"description\":\"Extends ui-sref directive (part of angular-ui-router) to add fastclick style behaviour\",\"fork\":true,\"url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick\",\"forks_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/forks\",\"keys_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/teams\",\"hooks_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/hooks\",\"issue_events_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/events\",\"assignees_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/tags\",\"blobs_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/languages\",\"stargazers_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/stargazers\",\"contributors_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/contributors\",\"subscribers_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/subscribers\",\"subscription_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/subscription\",\"commits_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/merges\",\"archive_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/downloads\",\"issues_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/arathael/angular-ui-sref-fastclick/releases{/id}\",\"created_at\":\"2015-01-01T01:01:53Z\",\"updated_at\":\"2015-01-01T00:56:57Z\",\"pushed_at\":\"2014-12-08T16:59:03Z\",\"git_url\":\"git://github.com/arathael/angular-ui-sref-fastclick.git\",\"ssh_url\":\"git@github.com:arathael/angular-ui-sref-fastclick.git\",\"clone_url\":\"https://github.com/arathael/angular-ui-sref-fastclick.git\",\"svn_url\":\"https://github.com/arathael/angular-ui-sref-fastclick\",\"homepage\":\"\",\"size\":116,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:01:54Z\"}\n{\"id\":\"2489396690\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":305455,\"login\":\"mykmelez\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mykmelez\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/305455?\"},\"repo\":{\"id\":15485479,\"name\":\"facelessuser/TabsExtra\",\"url\":\"https://api.github.com/repos/facelessuser/TabsExtra\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/facelessuser/TabsExtra/issues/21\",\"labels_url\":\"https://api.github.com/repos/facelessuser/TabsExtra/issues/21/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/facelessuser/TabsExtra/issues/21/comments\",\"events_url\":\"https://api.github.com/repos/facelessuser/TabsExtra/issues/21/events\",\"html_url\":\"https://github.com/facelessuser/TabsExtra/issues/21\",\"id\":53210194,\"number\":21,\"title\":\"\\\"reopen closed file\\\" moves reopened tab to the right of its previous location\",\"user\":{\"login\":\"mykmelez\",\"id\":305455,\"avatar_url\":\"https://avatars.githubusercontent.com/u/305455?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mykmelez\",\"html_url\":\"https://github.com/mykmelez\",\"followers_url\":\"https://api.github.com/users/mykmelez/followers\",\"following_url\":\"https://api.github.com/users/mykmelez/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/mykmelez/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/mykmelez/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/mykmelez/subscriptions\",\"organizations_url\":\"https://api.github.com/users/mykmelez/orgs\",\"repos_url\":\"https://api.github.com/users/mykmelez/repos\",\"events_url\":\"https://api.github.com/users/mykmelez/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/mykmelez/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:01:54Z\",\"updated_at\":\"2015-01-01T01:01:54Z\",\"closed_at\":null,\"body\":\"I like *fallback_focus* set to `right`, which is why I found and installed TabsExtra! But there's an issue with it: if you \\\"reopen closed file\\\", the reopened tab doesn't go back to its previous position on the tab bar. Instead, it jumps one position to the right.\\r\\n\\r\\nSteps to Reproduce:\\r\\n\\r\\n1. Open three tabs, A, B, and C, in that order.\\r\\n2. Activate tab B (the middle tab) and then close it (Cmd-W on my Mac).\\r\\n3. Reopen it (Cmd-Shift-T).\\r\\n\\r\\nB should reopen between A and C, which is where it was before; but instead it reopens to the right of C!\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:54Z\"}\n{\"id\":\"2489396699\",\"type\":\"PushEvent\",\"actor\":{\"id\":458272,\"login\":\"tomalexander\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tomalexander\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458272?\"},\"repo\":{\"id\":28367645,\"name\":\"tomalexander/basic_multi_bot\",\"url\":\"https://api.github.com/repos/tomalexander/basic_multi_bot\"},\"payload\":{\"push_id\":536752612,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"bcc7dd7ee5550ef6d71e14b4d4c587477f71a3dc\",\"before\":\"d299ad2b913fffa66512180cd642e89cca3db789\",\"commits\":[{\"sha\":\"bcc7dd7ee5550ef6d71e14b4d4c587477f71a3dc\",\"author\":{\"email\":\"a09a620e88c0ebee24434030b77c3b58d8242b71@gmail.com\",\"name\":\"Tom Alexander\"},\"message\":\"switched to single instances of delegates and delegates determining rooms\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tomalexander/basic_multi_bot/commits/bcc7dd7ee5550ef6d71e14b4d4c587477f71a3dc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:55Z\"}\n{\"id\":\"2489396700\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":640745,\"login\":\"bchurchill\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?\"},\"repo\":{\"id\":2747319,\"name\":\"eschkufz/cpputil\",\"url\":\"https://api.github.com/repos/eschkufz/cpputil\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21\",\"labels_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/comments\",\"events_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/events\",\"html_url\":\"https://github.com/eschkufz/cpputil/issues/21\",\"id\":52613427,\"number\":21,\"title\":\"Get string representation of bitstring\",\"user\":{\"login\":\"bchurchill\",\"id\":640745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"html_url\":\"https://github.com/bchurchill\",\"followers_url\":\"https://api.github.com/users/bchurchill/followers\",\"following_url\":\"https://api.github.com/users/bchurchill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bchurchill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bchurchill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bchurchill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bchurchill/orgs\",\"repos_url\":\"https://api.github.com/users/bchurchill/repos\",\"events_url\":\"https://api.github.com/users/bchurchill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bchurchill/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"bchurchill\",\"id\":640745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"html_url\":\"https://github.com/bchurchill\",\"followers_url\":\"https://api.github.com/users/bchurchill/followers\",\"following_url\":\"https://api.github.com/users/bchurchill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bchurchill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bchurchill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bchurchill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bchurchill/orgs\",\"repos_url\":\"https://api.github.com/users/bchurchill/repos\",\"events_url\":\"https://api.github.com/users/bchurchill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bchurchill/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-22T01:47:57Z\",\"updated_at\":\"2015-01-01T01:01:55Z\",\"closed_at\":\"2015-01-01T01:01:55Z\",\"body\":\"It would be nice to have a way to get a base-16 string representation of the contents of a bitstring.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:56Z\"}\n{\"id\":\"2489396702\",\"type\":\"CreateEvent\",\"actor\":{\"id\":543483,\"login\":\"rockdragon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rockdragon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/543483?\"},\"repo\":{\"id\":28678231,\"name\":\"rockdragon/SocketChat\",\"url\":\"https://api.github.com/repos/rockdragon/SocketChat\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Chat room scaffolding\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:56Z\"}\n{\"id\":\"2489396701\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":640745,\"login\":\"bchurchill\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?\"},\"repo\":{\"id\":2747319,\"name\":\"eschkufz/cpputil\",\"url\":\"https://api.github.com/repos/eschkufz/cpputil\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21\",\"labels_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/comments\",\"events_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21/events\",\"html_url\":\"https://github.com/eschkufz/cpputil/issues/21\",\"id\":52613427,\"number\":21,\"title\":\"Get string representation of bitstring\",\"user\":{\"login\":\"bchurchill\",\"id\":640745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"html_url\":\"https://github.com/bchurchill\",\"followers_url\":\"https://api.github.com/users/bchurchill/followers\",\"following_url\":\"https://api.github.com/users/bchurchill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bchurchill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bchurchill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bchurchill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bchurchill/orgs\",\"repos_url\":\"https://api.github.com/users/bchurchill/repos\",\"events_url\":\"https://api.github.com/users/bchurchill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bchurchill/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"bchurchill\",\"id\":640745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"html_url\":\"https://github.com/bchurchill\",\"followers_url\":\"https://api.github.com/users/bchurchill/followers\",\"following_url\":\"https://api.github.com/users/bchurchill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bchurchill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bchurchill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bchurchill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bchurchill/orgs\",\"repos_url\":\"https://api.github.com/users/bchurchill/repos\",\"events_url\":\"https://api.github.com/users/bchurchill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bchurchill/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-22T01:47:57Z\",\"updated_at\":\"2015-01-01T01:01:55Z\",\"closed_at\":\"2015-01-01T01:01:55Z\",\"body\":\"It would be nice to have a way to get a base-16 string representation of the contents of a bitstring.\"},\"comment\":{\"url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/comments/68477253\",\"html_url\":\"https://github.com/eschkufz/cpputil/issues/21#issuecomment-68477253\",\"issue_url\":\"https://api.github.com/repos/eschkufz/cpputil/issues/21\",\"id\":68477253,\"user\":{\"login\":\"bchurchill\",\"id\":640745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/640745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchurchill\",\"html_url\":\"https://github.com/bchurchill\",\"followers_url\":\"https://api.github.com/users/bchurchill/followers\",\"following_url\":\"https://api.github.com/users/bchurchill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bchurchill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bchurchill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bchurchill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bchurchill/orgs\",\"repos_url\":\"https://api.github.com/users/bchurchill/repos\",\"events_url\":\"https://api.github.com/users/bchurchill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bchurchill/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:55Z\",\"updated_at\":\"2015-01-01T01:01:55Z\",\"body\":\"That's slick.  But I don't need this anymore because I'm not using these for constant SymBitVectors.  Closing.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:56Z\"}\n{\"id\":\"2489396705\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":10263666,\"login\":\"katiekroik\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/katiekroik\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10263666?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"ref\":\"Develop\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:01:57Z\"}\n{\"id\":\"2489396707\",\"type\":\"PushEvent\",\"actor\":{\"id\":8962953,\"login\":\"kutlass\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kutlass\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8962953?\"},\"repo\":{\"id\":24609481,\"name\":\"kutlass/YogaFrame\",\"url\":\"https://api.github.com/repos/kutlass/YogaFrame\"},\"payload\":{\"push_id\":536752614,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"84fb78bea76b0c5933953a897540a10122b3fba6\",\"before\":\"38aff9c324f9ad1e211195a00bf70ca0c84cb2d9\",\"commits\":[{\"sha\":\"84fb78bea76b0c5933953a897540a10122b3fba6\",\"author\":{\"email\":\"f1f30ec1cfc7458c34a4cc74fc6809cf46b6b758@yogaframe.net\",\"name\":\"Karl Flores\"},\"message\":\"Change return type from Dispatch to a JSession for WebPostJSession API.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kutlass/YogaFrame/commits/84fb78bea76b0c5933953a897540a10122b3fba6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:57Z\"}\n{\"id\":\"2489396710\",\"type\":\"PushEvent\",\"actor\":{\"id\":3386273,\"login\":\"david-ragazzi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/david-ragazzi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3386273?\"},\"repo\":{\"id\":18156369,\"name\":\"david-ragazzi/nupic\",\"url\":\"https://api.github.com/repos/david-ragazzi/nupic\"},\"payload\":{\"push_id\":536752616,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/move_testpyhtm_to_nupiccore\",\"head\":\"61cd06671a19147d11da4c23d5d9fbe811e73124\",\"before\":\"f64e0e47e581048266793e22ff005600cc5a5f9f\",\"commits\":[{\"sha\":\"61cd06671a19147d11da4c23d5d9fbe811e73124\",\"author\":{\"email\":\"9f810c70f0453cb6f7cc60219cf23ba543fb72b1@hotmail.com\",\"name\":\"DavidRagazzi\"},\"message\":\"Remove HtmTest and call it from nupic.core binaries\",\"distinct\":true,\"url\":\"https://api.github.com/repos/david-ragazzi/nupic/commits/61cd06671a19147d11da4c23d5d9fbe811e73124\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:57Z\"}\n{\"id\":\"2489396712\",\"type\":\"PushEvent\",\"actor\":{\"id\":226950,\"login\":\"chotchki\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chotchki\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/226950?\"},\"repo\":{\"id\":28649464,\"name\":\"chotchki/Henge\",\"url\":\"https://api.github.com/repos/chotchki/Henge\"},\"payload\":{\"push_id\":536752617,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"60a6ee7b0f57de246ed67a5e8d3f18bfd353d43a\",\"before\":\"0b6b908abb4f0380f136ebd5f5a2cafb76364ad5\",\"commits\":[{\"sha\":\"95124fea28815ccfa0300470d3ac447ff351f8a2\",\"author\":{\"email\":\"4cb429e61173bad4e4892a8806f74a746a9d9e4a@gmail.com\",\"name\":\"Christopher Hotchkiss\"},\"message\":\"Adding skeleton classes for startup\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chotchki/Henge/commits/95124fea28815ccfa0300470d3ac447ff351f8a2\"},{\"sha\":\"60a6ee7b0f57de246ed67a5e8d3f18bfd353d43a\",\"author\":{\"email\":\"4cb429e61173bad4e4892a8806f74a746a9d9e4a@gmail.com\",\"name\":\"Christopher Hotchkiss\"},\"message\":\"Merge branch 'master' of ssh://git@github.com/chotchki/Henge.git\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chotchki/Henge/commits/60a6ee7b0f57de246ed67a5e8d3f18bfd353d43a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:57Z\"}\n{\"id\":\"2489396713\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":8903402,\"login\":\"oncomangus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oncomangus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8903402?\"},\"repo\":{\"id\":26731988,\"name\":\"badrsony/icloudin-support-\",\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4\",\"labels_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/comments\",\"events_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/events\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/4\",\"id\":50920400,\"number\":4,\"title\":\"icloudin support \",\"user\":{\"login\":\"badrsony\",\"id\":7895050,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7895050?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/badrsony\",\"html_url\":\"https://github.com/badrsony\",\"followers_url\":\"https://api.github.com/users/badrsony/followers\",\"following_url\":\"https://api.github.com/users/badrsony/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/badrsony/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/badrsony/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/badrsony/subscriptions\",\"organizations_url\":\"https://api.github.com/users/badrsony/orgs\",\"repos_url\":\"https://api.github.com/users/badrsony/repos\",\"events_url\":\"https://api.github.com/users/badrsony/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/badrsony/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":996,\"created_at\":\"2014-12-04T02:13:39Z\",\"updated_at\":\"2015-01-01T01:01:58Z\",\"closed_at\":\"2015-01-01T00:00:32Z\",\"body\":\"Originally written by @TTMTT. That we hope for him safery and peace\\r\\n.\\r\\n\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n\\r\\niCL0udin v1.0 -> %98\\r\\n\\r\\nRemaining: %2 testing with some people..\\r\\n\\r\\nLast Method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\n\\r\\niCL0udin v1.0 have this method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/comments/68477255\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/4#issuecomment-68477255\",\"issue_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4\",\"id\":68477255,\"user\":{\"login\":\"oncomangus\",\"id\":8903402,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8903402?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oncomangus\",\"html_url\":\"https://github.com/oncomangus\",\"followers_url\":\"https://api.github.com/users/oncomangus/followers\",\"following_url\":\"https://api.github.com/users/oncomangus/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/oncomangus/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/oncomangus/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/oncomangus/subscriptions\",\"organizations_url\":\"https://api.github.com/users/oncomangus/orgs\",\"repos_url\":\"https://api.github.com/users/oncomangus/repos\",\"events_url\":\"https://api.github.com/users/oncomangus/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/oncomangus/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:01:58Z\",\"updated_at\":\"2015-01-01T01:01:58Z\",\"body\":\"@All HAPPY NEW ICLOUD YEAR #cheers #coffee\"}},\"public\":true,\"created_at\":\"2015-01-01T01:01:58Z\"}\n{\"id\":\"2489396720\",\"type\":\"PushEvent\",\"actor\":{\"id\":41057,\"login\":\"samv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/samv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/41057?\"},\"repo\":{\"id\":16792286,\"name\":\"hearsaycorp/normalize\",\"url\":\"https://api.github.com/repos/hearsaycorp/normalize\"},\"payload\":{\"push_id\":536752624,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"20f0f9bbc6dd3bc111019fb985fc42d28a9814ee\",\"before\":\"3e040e2c4d617beb145074a21cac1402edcea35c\",\"commits\":[{\"sha\":\"2f0df332bf4a1e963296f545802cf7497ac20ff8\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@vilain.net\",\"name\":\"Sam Vilain\"},\"message\":\"Implement FieldSelector.delete\\n\\nAll 4 CRUD operations now supported :)\",\"distinct\":false,\"url\":\"https://api.github.com/repos/hearsaycorp/normalize/commits/2f0df332bf4a1e963296f545802cf7497ac20ff8\"},{\"sha\":\"20f0f9bbc6dd3bc111019fb985fc42d28a9814ee\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@vilain.net\",\"name\":\"Sam Vilain\"},\"message\":\"Update sphinx docs to commit 0.6.3-1-g2f0df33\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hearsaycorp/normalize/commits/20f0f9bbc6dd3bc111019fb985fc42d28a9814ee\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:01:59Z\",\"org\":{\"id\":633032,\"login\":\"hearsaycorp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/hearsaycorp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/633032?\"}}\n{\"id\":\"2489396721\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":3103764,\"login\":\"carymrobbins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?\"},\"repo\":{\"id\":15573192,\"name\":\"carymrobbins/intellij-haskforce\",\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/22397260\",\"id\":22397260,\"diff_hunk\":\"@@ -0,0 +1,53 @@\\n+package com.haskforce.highlighting.annotation.external;\\n+\\n+import com.intellij.openapi.editor.LogicalPosition;\\n+import com.intellij.openapi.editor.VisualPosition;\\n+import org.junit.Assert;\\n+import org.junit.Test;\\n+\\n+/**\\n+ * Created by kasper on 12/24/14.\\n+ */\\n+public class GhcUtilTest {\",\"path\":\"tests/com/haskforce/highlighting/annotation/external/GhcUtilTest.java\",\"position\":11,\"original_position\":11,\"commit_id\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"original_commit_id\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"user\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Nice!  Thanks for adding tests.  Be sure to add this class to **tests/com/haskforce/HaskellTestCase.java**\",\"created_at\":\"2015-01-01T01:02:00Z\",\"updated_at\":\"2015-01-01T01:02:00Z\",\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105#discussion_r22397260\",\"pull_request_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/22397260\"},\"html\":{\"href\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105#discussion_r22397260\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\",\"id\":26615813,\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\",\"diff_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.diff\",\"patch_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.patch\",\"issue_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\",\"number\":105,\"state\":\"open\",\"locked\":false,\"title\":\"Issue90 type information\",\"user\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Cary,\\r\\n\\r\\nFirst try of the type information. It seems quite stable, been using it for a few days (provided the configuration is correct, I suppose, didn't test what happens when ghc-modi is not correctly configured).\\r\\n\\r\\nI bound the type info call to the DocumentationProvider as well as to an action (alt - equals, like scala). I prefer the action, I think the documentation provider doesn't work so well. There are tests of the parsing of the output of ghc-modi, but not really of the documentationprovider, basically because of the abundance of static calls and the fact that I think they can only be mocked while testing, and statics can only be mocked through Powermock if I recall correctly, which would mean an extra test dependency and so on and so forth, so I left that to be your call.\\r\\n\\r\\nAlso, there is a weird behaviour that getting the editor creates a stack trace of around 5 kilometers long, something that seems like a threading issue, but the function seems to work. I don't really know why the stack trace happens, all the more because it only happens when calling the type information through the documentation provider (same code path is used when the action is called, but no stack trace). Maybe you know more what could go wrong, it looks like something intellij-related.\\r\\n\\r\\nSo, consider this a \\\"request for comment\\\" more than a pull request ;-)\\r\\n\\r\\nKasper\",\"created_at\":\"2014-12-27T16:19:07Z\",\"updated_at\":\"2015-01-01T01:02:00Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"b32a5f3ef0bc14eebea4baa946ec2e66008086dd\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/commits\",\"review_comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/comments\",\"review_comment_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/comments\",\"statuses_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"head\":{\"label\":\"KasperJanssens:issue90TypeInformation\",\"ref\":\"issue90TypeInformation\",\"sha\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"user\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28540684,\"name\":\"intellij-haskforce\",\"full_name\":\"KasperJanssens/intellij-haskforce\",\"owner\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/KasperJanssens/intellij-haskforce\",\"description\":\"Haskell plugin for IntelliJ IDEA Community Edition\",\"fork\":true,\"url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce\",\"forks_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/forks\",\"keys_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/teams\",\"hooks_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/hooks\",\"issue_events_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/events\",\"assignees_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/tags\",\"blobs_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/languages\",\"stargazers_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/stargazers\",\"contributors_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/contributors\",\"subscribers_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/subscribers\",\"subscription_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/subscription\",\"commits_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/merges\",\"archive_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/downloads\",\"issues_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/releases{/id}\",\"created_at\":\"2014-12-27T16:11:01Z\",\"updated_at\":\"2014-12-27T16:11:03Z\",\"pushed_at\":\"2014-12-31T15:44:48Z\",\"git_url\":\"git://github.com/KasperJanssens/intellij-haskforce.git\",\"ssh_url\":\"git@github.com:KasperJanssens/intellij-haskforce.git\",\"clone_url\":\"https://github.com/KasperJanssens/intellij-haskforce.git\",\"svn_url\":\"https://github.com/KasperJanssens/intellij-haskforce\",\"homepage\":\"http://carymrobbins.github.io/intellij-haskforce/\",\"size\":7553,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Java\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"carymrobbins:master\",\"ref\":\"master\",\"sha\":\"19442eb58293650dc699e30ccf2d5482e1c688d8\",\"user\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15573192,\"name\":\"intellij-haskforce\",\"full_name\":\"carymrobbins/intellij-haskforce\",\"owner\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce\",\"description\":\"Haskell plugin for IntelliJ IDEA Community Edition\",\"fork\":false,\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce\",\"forks_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/forks\",\"keys_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/teams\",\"hooks_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/hooks\",\"issue_events_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/events\",\"assignees_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/tags\",\"blobs_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/languages\",\"stargazers_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/stargazers\",\"contributors_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/contributors\",\"subscribers_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/subscribers\",\"subscription_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/subscription\",\"commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/merges\",\"archive_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/downloads\",\"issues_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/releases{/id}\",\"created_at\":\"2014-01-02T01:28:54Z\",\"updated_at\":\"2014-12-30T16:15:50Z\",\"pushed_at\":\"2014-12-27T04:28:50Z\",\"git_url\":\"git://github.com/carymrobbins/intellij-haskforce.git\",\"ssh_url\":\"git@github.com:carymrobbins/intellij-haskforce.git\",\"clone_url\":\"https://github.com/carymrobbins/intellij-haskforce.git\",\"svn_url\":\"https://github.com/carymrobbins/intellij-haskforce\",\"homepage\":\"http://carymrobbins.github.io/intellij-haskforce/\",\"size\":11390,\"stargazers_count\":159,\"watchers_count\":159,\"language\":\"Java\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":9,\"mirror_url\":null,\"open_issues_count\":30,\"forks\":9,\"open_issues\":30,\"watchers\":159,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\"},\"html\":{\"href\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\"},\"issue\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\"},\"comments\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:02:00Z\"}\n{\"id\":\"2489396724\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536752626,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1ac37b74ae21d946dc3826883204811da6d06852\",\"before\":\"f5fd8c9b5d99ba2f521c56658163b69c29eb1027\",\"commits\":[{\"sha\":\"1ac37b74ae21d946dc3826883204811da6d06852\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074118714\\n\\noJhRtiHmitwAmvo7xDna+E9mim307BW3xojcjXsX+6M=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/1ac37b74ae21d946dc3826883204811da6d06852\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:00Z\"}\n{\"id\":\"2489396725\",\"type\":\"PushEvent\",\"actor\":{\"id\":9515067,\"login\":\"freundTech\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/freundTech\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9515067?\"},\"repo\":{\"id\":27674403,\"name\":\"freundTech/lightDMX\",\"url\":\"https://api.github.com/repos/freundTech/lightDMX\"},\"payload\":{\"push_id\":536752627,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1949bbb851b79b526154c7cec6a0e7e00f1420a6\",\"before\":\"def97c029096a9b41b592ae7070a651b78edcac3\",\"commits\":[{\"sha\":\"1949bbb851b79b526154c7cec6a0e7e00f1420a6\",\"author\":{\"email\":\"d6643dd9a6fe6114980ecaf6d787b2a0da7fc377@gmail.com\",\"name\":\"Adrian\"},\"message\":\"Finished connections and made connectors their own class.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/freundTech/lightDMX/commits/1949bbb851b79b526154c7cec6a0e7e00f1420a6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:00Z\"}\n{\"id\":\"2489396728\",\"type\":\"PushEvent\",\"actor\":{\"id\":66897,\"login\":\"asad\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/asad\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/66897?\"},\"repo\":{\"id\":28622285,\"name\":\"asad/ChemBLAST\",\"url\":\"https://api.github.com/repos/asad/ChemBLAST\"},\"payload\":{\"push_id\":536752629,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"4fd33b8625d56ca7dcda0acc38f6f51ec1358c87\",\"before\":\"b41b8269e88a699180c115deb6c7ec8df44cf263\",\"commits\":[{\"sha\":\"1f8049a2b5c2b640dc0c3577bdbbaa075c8d7827\",\"author\":{\"email\":\"09eb4ce7c91ecff4eff21da1532c566afe7cb66c@gmail.com\",\"name\":\"Syed Asad Rahman\"},\"message\":\"parallel java search implemented\",\"distinct\":true,\"url\":\"https://api.github.com/repos/asad/ChemBLAST/commits/1f8049a2b5c2b640dc0c3577bdbbaa075c8d7827\"},{\"sha\":\"ebdea677d4adff3460a744e5c4499c42022eeca3\",\"author\":{\"email\":\"09eb4ce7c91ecff4eff21da1532c566afe7cb66c@gmail.com\",\"name\":\"Syed Asad Rahman\"},\"message\":\"parallel java search implemented\",\"distinct\":true,\"url\":\"https://api.github.com/repos/asad/ChemBLAST/commits/ebdea677d4adff3460a744e5c4499c42022eeca3\"},{\"sha\":\"4fd33b8625d56ca7dcda0acc38f6f51ec1358c87\",\"author\":{\"email\":\"09eb4ce7c91ecff4eff21da1532c566afe7cb66c@gmail.com\",\"name\":\"Syed Asad Rahman\"},\"message\":\"4X speed up obtained with join and fork search implementation.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/asad/ChemBLAST/commits/4fd33b8625d56ca7dcda0acc38f6f51ec1358c87\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:01Z\"}\n{\"id\":\"2489396733\",\"type\":\"PushEvent\",\"actor\":{\"id\":7886751,\"login\":\"harrison0723\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harrison0723\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7886751?\"},\"repo\":{\"id\":28653649,\"name\":\"harrison0723/bloccit\",\"url\":\"https://api.github.com/repos/harrison0723/bloccit\"},\"payload\":{\"push_id\":536752633,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1294ea64e75c5da49f0239c796445374870a0eae\",\"before\":\"63a17d0ec4d8e6a4e143fe5775b5fcc25a765b05\",\"commits\":[{\"sha\":\"1294ea64e75c5da49f0239c796445374870a0eae\",\"author\":{\"email\":\"cf9e7917eadcf4e1a3e8784c7f4ca5d67be0f8ad@gmail.com\",\"name\":\"HarrisonLo\"},\"message\":\"Updated add layout and welcome views\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harrison0723/bloccit/commits/1294ea64e75c5da49f0239c796445374870a0eae\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:02Z\"}\n{\"id\":\"2489396734\",\"type\":\"PushEvent\",\"actor\":{\"id\":5728403,\"login\":\"patrick-hudson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/patrick-hudson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5728403?\"},\"repo\":{\"id\":25392255,\"name\":\"patrick-hudson/EggDrop\",\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop\"},\"payload\":{\"push_id\":536752634,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cb11cf77a4d9d7625c14e50c27fe61d387d2c95e\",\"before\":\"6336938241e7f58f06585498f1e07b6645b86630\",\"commits\":[{\"sha\":\"cb11cf77a4d9d7625c14e50c27fe61d387d2c95e\",\"author\":{\"email\":\"cbb7353e6d953ef360baf960c122346276c6e320@hudson.bz\",\"name\":\"Patrick Hudson\"},\"message\":\"Scripted auto-commit on change (2014-12-31 20:02:00) by gitwatch.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop/commits/cb11cf77a4d9d7625c14e50c27fe61d387d2c95e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:02Z\"}\n{\"id\":\"2489396740\",\"type\":\"CreateEvent\",\"actor\":{\"id\":7989982,\"login\":\"wxv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wxv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7989982?\"},\"repo\":{\"id\":28678232,\"name\":\"wxv/Items-Index\",\"url\":\"https://api.github.com/repos/wxv/Items-Index\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Historical item prices\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:03Z\"}\n{\"id\":\"2489396741\",\"type\":\"PushEvent\",\"actor\":{\"id\":1981539,\"login\":\"harshiet\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harshiet\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1981539?\"},\"repo\":{\"id\":28192167,\"name\":\"harshiet/jira-reports-pro\",\"url\":\"https://api.github.com/repos/harshiet/jira-reports-pro\"},\"payload\":{\"push_id\":536752637,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3f51d07d0517bbbc9fdc9ff27d615b175fd3084a\",\"before\":\"8e111512bd19a9db1e99ed05e0290dbfea5de705\",\"commits\":[{\"sha\":\"3f51d07d0517bbbc9fdc9ff27d615b175fd3084a\",\"author\":{\"email\":\"ed07a1efef8aa439ec830948475d9a155b709cf5@gmail.com\",\"name\":\"harshiet\"},\"message\":\"1\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harshiet/jira-reports-pro/commits/3f51d07d0517bbbc9fdc9ff27d615b175fd3084a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:03Z\"}\n{\"id\":\"2489396747\",\"type\":\"PushEvent\",\"actor\":{\"id\":1640798,\"login\":\"langorn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/langorn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1640798?\"},\"repo\":{\"id\":28678190,\"name\":\"langorn/crm\",\"url\":\"https://api.github.com/repos/langorn/crm\"},\"payload\":{\"push_id\":536752639,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"408cbaf7c4cc00d021d00dc14dacb8f2095c6fa3\",\"before\":\"95907808683f55c40c7ce5552330d0c276921cb1\",\"commits\":[{\"sha\":\"408cbaf7c4cc00d021d00dc14dacb8f2095c6fa3\",\"author\":{\"email\":\"ba36a5a19299fe08d08975c8488c56bf95772e2d@gmail.com\",\"name\":\"Mark\"},\"message\":\"first commit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/langorn/crm/commits/408cbaf7c4cc00d021d00dc14dacb8f2095c6fa3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:04Z\"}\n{\"id\":\"2489396754\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1253444,\"login\":\"nathan-osman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathan-osman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1253444?\"},\"repo\":{\"id\":28151446,\"name\":\"nathan-osman/django-archive\",\"url\":\"https://api.github.com/repos/nathan-osman/django-archive\"},\"payload\":{\"ref\":\"0.1.2\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"Management command for creating compressed archives of Django projects, including database tables and uploaded media.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:06Z\"}\n{\"id\":\"2489396756\",\"type\":\"PushEvent\",\"actor\":{\"id\":9201970,\"login\":\"qdm\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qdm\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9201970?\"},\"repo\":{\"id\":25173910,\"name\":\"qdm/qdm.github.io\",\"url\":\"https://api.github.com/repos/qdm/qdm.github.io\"},\"payload\":{\"push_id\":536752642,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"94683ca14e320a286452e2f2e873ff5d603f1a98\",\"before\":\"05438aee0e5572a6e4adf91d7eef32917812d9e9\",\"commits\":[{\"sha\":\"94683ca14e320a286452e2f2e873ff5d603f1a98\",\"author\":{\"email\":\"de163e90d3aeef9f404d1de71c48e234a211e3c3@gmail.com\",\"name\":\"KT\"},\"message\":\"Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qdm/qdm.github.io/commits/94683ca14e320a286452e2f2e873ff5d603f1a98\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:06Z\"}\n{\"id\":\"2489396757\",\"type\":\"PushEvent\",\"actor\":{\"id\":31338,\"login\":\"gak\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gak\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/31338?\"},\"repo\":{\"id\":28615761,\"name\":\"PayGroove/fig\",\"url\":\"https://api.github.com/repos/PayGroove/fig\"},\"payload\":{\"push_id\":536752643,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/templates\",\"head\":\"1614c4c666c85d5447e1ae10ad81f9ece1852062\",\"before\":\"c10351e1f8f17f6733a0db29aedf3b377b96fcf2\",\"commits\":[{\"sha\":\"b91fe46a080165aa154a2b17bf0699c67010fea3\",\"author\":{\"email\":\"bee25f9566f14bcc1d385b4a10d34d2cb725465a@gak0.com\",\"name\":\"Gerald Kaszuba\"},\"message\":\"docker/fig#761 Additional template checks and bug fixes\\n\\nThere were a few bugs in my initial implementation:\\n - Infinite recursion when referring to the same template file.\\n   Implementing a class and caching previously loaded yml files handles\\n   that problem.\\n - Dictionaries that were updated in place where left \\\"dirty\\\". The\\n   copy.copy fixes that.\\n - Incorrectly configured template yml files were not throwing fig\\n   exceptions.\\n\\nSigned-off-by: Gerald Kaszuba <gak@gak0.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/PayGroove/fig/commits/b91fe46a080165aa154a2b17bf0699c67010fea3\"},{\"sha\":\"1614c4c666c85d5447e1ae10ad81f9ece1852062\",\"author\":{\"email\":\"bee25f9566f14bcc1d385b4a10d34d2cb725465a@gak0.com\",\"name\":\"Gerald Kaszuba\"},\"message\":\"docker/fig#761 Added some docs for template in yml.md\\n\\nSigned-off-by: Gerald Kaszuba <gak@gak0.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/PayGroove/fig/commits/1614c4c666c85d5447e1ae10ad81f9ece1852062\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:06Z\",\"org\":{\"id\":10135493,\"login\":\"PayGroove\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/PayGroove\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10135493?\"}}\n{\"id\":\"2489396764\",\"type\":\"PushEvent\",\"actor\":{\"id\":8404840,\"login\":\"orionmelt\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/orionmelt\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8404840?\"},\"repo\":{\"id\":23653249,\"name\":\"orionmelt/sherlock\",\"url\":\"https://api.github.com/repos/orionmelt/sherlock\"},\"payload\":{\"push_id\":536752649,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8d16004c2185415f95881976b8245083e3b0bf1d\",\"before\":\"a11d423101ce2b72d54a79e82ea92805b3590098\",\"commits\":[{\"sha\":\"8d16004c2185415f95881976b8245083e3b0bf1d\",\"author\":{\"email\":\"d987d5b2320ab4566c87c2023d39457639743791@gmail.com\",\"name\":\"Raj\"},\"message\":\"Docstrings, comments and some renaming updates.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/orionmelt/sherlock/commits/8d16004c2185415f95881976b8245083e3b0bf1d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:07Z\"}\n{\"id\":\"2489396765\",\"type\":\"PushEvent\",\"actor\":{\"id\":1456047,\"login\":\"kyokomi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kyokomi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1456047?\"},\"repo\":{\"id\":25506232,\"name\":\"kyokomi/gomajan\",\"url\":\"https://api.github.com/repos/kyokomi/gomajan\"},\"payload\":{\"push_id\":536752650,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0dc0e50eb584a0f312f5f6ce1c3a85d4ddb3f62e\",\"before\":\"33d7ed3c98a7cfb19ce924735a7420de0ca57ef7\",\"commits\":[{\"sha\":\"0dc0e50eb584a0f312f5f6ce1c3a85d4ddb3f62e\",\"author\":{\"email\":\"2ea0042be6760fbcd1e13c4e2076e1e54e82d1d5@gmail.com\",\"name\":\"kyokomi\"},\"message\":\"test: 混全帯么九\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kyokomi/gomajan/commits/0dc0e50eb584a0f312f5f6ce1c3a85d4ddb3f62e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:07Z\"}\n{\"id\":\"2489396766\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3344033,\"login\":\"neverfox\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/neverfox\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3344033?\"},\"repo\":{\"id\":11744114,\"name\":\"eviltrout/ember-renderspeed\",\"url\":\"https://api.github.com/repos/eviltrout/ember-renderspeed\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:07Z\"}\n{\"id\":\"2489396770\",\"type\":\"PushEvent\",\"actor\":{\"id\":7050164,\"login\":\"marvalgames\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marvalgames\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7050164?\"},\"repo\":{\"id\":28167953,\"name\":\"marvalgames/dead-pixels\",\"url\":\"https://api.github.com/repos/marvalgames/dead-pixels\"},\"payload\":{\"push_id\":536752653,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d4f2a4466aceef8bc6db2232451ba31fcd431a0a\",\"before\":\"5ff63a7b9f29739457db94f530b83e6f46afb85e\",\"commits\":[{\"sha\":\"d4f2a4466aceef8bc6db2232451ba31fcd431a0a\",\"author\":{\"email\":\"6c18ea4526e2e66416491c4bc60d3d5a42aaebd8@gmail.com\",\"name\":\"marvalgames\"},\"message\":\".28\",\"distinct\":true,\"url\":\"https://api.github.com/repos/marvalgames/dead-pixels/commits/d4f2a4466aceef8bc6db2232451ba31fcd431a0a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:07Z\"}\n{\"id\":\"2489396775\",\"type\":\"PushEvent\",\"actor\":{\"id\":7391433,\"login\":\"StevenXL\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/StevenXL\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7391433?\"},\"repo\":{\"id\":28642919,\"name\":\"StevenXL/learntoprogram\",\"url\":\"https://api.github.com/repos/StevenXL/learntoprogram\"},\"payload\":{\"push_id\":536752654,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"85f31e3ddb605fe4a1bde3ddb537ab3302d4c59d\",\"before\":\"5cb1eace288863132472638e17792604cbdc5443\",\"commits\":[{\"sha\":\"85f31e3ddb605fe4a1bde3ddb537ab3302d4c59d\",\"author\":{\"email\":\"2a2c33fb7ea332dc4f943c4fb28454a257783aab@gmail.com\",\"name\":\"StevenXL\"},\"message\":\"Finished Chapter 7\",\"distinct\":true,\"url\":\"https://api.github.com/repos/StevenXL/learntoprogram/commits/85f31e3ddb605fe4a1bde3ddb537ab3302d4c59d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:08Z\"}\n{\"id\":\"2489396778\",\"type\":\"CreateEvent\",\"actor\":{\"id\":7989982,\"login\":\"wxv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wxv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7989982?\"},\"repo\":{\"id\":28678232,\"name\":\"wxv/Items-Index\",\"url\":\"https://api.github.com/repos/wxv/Items-Index\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Historical item prices\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:08Z\"}\n{\"id\":\"2489396781\",\"type\":\"PushEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"push_id\":536752657,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/#30\",\"head\":\"0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"before\":\"b838f0c36403eab209f3565eeb7a10375911d228\",\"commits\":[{\"sha\":\"0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"author\":{\"email\":\"4d1902cf1aefa3df01c59cbb9ae7db3045be42ae@gmail.com\",\"name\":\"Dylan Kirkby\"},\"message\":\"Add simple C program to replay binary file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/captainkirkby/Gears/commits/0ce69fa19d81de656dd6a74629099a7fa9261d1c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:09Z\"}\n{\"id\":\"2489396783\",\"type\":\"PushEvent\",\"actor\":{\"id\":1017605,\"login\":\"wangshan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wangshan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1017605?\"},\"repo\":{\"id\":28666633,\"name\":\"wangshan/wangshan.github.io\",\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io\"},\"payload\":{\"push_id\":536752658,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"92d20ac01b4767280cccaf71ee20f173529877a0\",\"before\":\"3a5b62221723e1a468813feef26cb122942c53b1\",\"commits\":[{\"sha\":\"92d20ac01b4767280cccaf71ee20f173529877a0\",\"author\":{\"email\":\"e3e97680eb29c788f35181af31eb442b3251e18f@gmail.com\",\"name\":\"Shan\"},\"message\":\"Update 2012-03-03-mac-development-environment-setup.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io/commits/92d20ac01b4767280cccaf71ee20f173529877a0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:09Z\"}\n{\"id\":\"2489396785\",\"type\":\"PushEvent\",\"actor\":{\"id\":72326,\"login\":\"Lexikos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Lexikos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/72326?\"},\"repo\":{\"id\":28678134,\"name\":\"Lexikos/xHotkey.ahk\",\"url\":\"https://api.github.com/repos/Lexikos/xHotkey.ahk\"},\"payload\":{\"push_id\":536752659,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f117c07dcb8ff33133d2da0476dfbaf70644b476\",\"before\":\"8f491d820da9469ab89fe548aadc82a11f8af250\",\"commits\":[{\"sha\":\"f117c07dcb8ff33133d2da0476dfbaf70644b476\",\"author\":{\"email\":\"e144dba3b863bfb4ca42790332b351be659b9c7e@gmail.com\",\"name\":\"Lexikos\"},\"message\":\"Fix readme\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Lexikos/xHotkey.ahk/commits/f117c07dcb8ff33133d2da0476dfbaf70644b476\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:09Z\"}\n{\"id\":\"2489396786\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":253237,\"login\":\"Jamesking56\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?\"},\"repo\":{\"id\":26730195,\"name\":\"cachethq/Cachet\",\"url\":\"https://api.github.com/repos/cachethq/Cachet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"labels_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/comments\",\"events_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/events\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173\",\"id\":53210024,\"number\":173,\"title\":\"Bug: Forms let you submit multiple times\",\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":4,\"created_at\":\"2015-01-01T00:52:06Z\",\"updated_at\":\"2015-01-01T01:02:10Z\",\"closed_at\":null,\"body\":\"When adding a new incident, I noticed a weird bug.\\r\\n\\r\\nIf you fill in the form as normal, then click the submit button twice really quickly, it'll create __TWO__ identical new incidents!\\r\\n\\r\\nThis could be a bit annoying, a simple fix is using a bit of JS that on submit, disables the submit button so that once clicked, it cannot be clicked again.\"},\"comment\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/comments/68477258\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173#issuecomment-68477258\",\"issue_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"id\":68477258,\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:10Z\",\"updated_at\":\"2015-01-01T01:02:10Z\",\"body\":\"Doesn't Laravel have that already?\\r\\n\\r\\nhttp://laravel.com/docs/4.2/security#protecting-routes\\r\\n\\r\\nDoesn't Laravel's CSRF Protection offer this?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:11Z\",\"org\":{\"id\":9951502,\"login\":\"cachethq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cachethq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9951502?\"}}\n{\"id\":\"2489396788\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2851221,\"login\":\"alkass\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alkass\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2851221?\"},\"repo\":{\"id\":28678233,\"name\":\"alkass/seQre\",\"url\":\"https://api.github.com/repos/alkass/seQre\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:11Z\"}\n{\"id\":\"2489396795\",\"type\":\"PushEvent\",\"actor\":{\"id\":3414800,\"login\":\"Jyang772\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jyang772\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3414800?\"},\"repo\":{\"id\":28655798,\"name\":\"Jyang772/MacSpoof\",\"url\":\"https://api.github.com/repos/Jyang772/MacSpoof\"},\"payload\":{\"push_id\":536752661,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8ec327a2306084297e54415c994eae67a75df167\",\"before\":\"633d4b381b6272a89070428ea1ef67600b934eac\",\"commits\":[{\"sha\":\"8ec327a2306084297e54415c994eae67a75df167\",\"author\":{\"email\":\"6c73cfbdcadf6d6f6fd0a03108daf8b6f1dd8164@gmail.com\",\"name\":\"Justin\"},\"message\":\"Update lol.sh\\n\\nBug with wpa_cli reassociate. \\r\\nDoes not seem to work on secured networks. Might be driver issue.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Jyang772/MacSpoof/commits/8ec327a2306084297e54415c994eae67a75df167\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:11Z\"}\n{\"id\":\"2489396796\",\"type\":\"PushEvent\",\"actor\":{\"id\":10298641,\"login\":\"lucachr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lucachr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10298641?\"},\"repo\":{\"id\":28611069,\"name\":\"lucachr/lucachr.github.io\",\"url\":\"https://api.github.com/repos/lucachr/lucachr.github.io\"},\"payload\":{\"push_id\":536752662,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7ff7b895ca4968b985f29cf50ee59fe249a50471\",\"before\":\"144530b760f22ed18330baeefc89d8f4eb8bbac8\",\"commits\":[{\"sha\":\"7ff7b895ca4968b985f29cf50ee59fe249a50471\",\"author\":{\"email\":\"3a92d5230cd572ed7ca38353f6e589037a70deaa@gmail.com\",\"name\":\"Luca Chiricozzi\"},\"message\":\"Generate Pelican site\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lucachr/lucachr.github.io/commits/7ff7b895ca4968b985f29cf50ee59fe249a50471\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:11Z\"}\n{\"id\":\"2489396797\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752663,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b956bef865580106b571e412e35308b3e7ab708e\",\"before\":\"22a9bd6584d30cc259b5b0055cb6a8bff31d8d9e\",\"commits\":[{\"sha\":\"b956bef865580106b571e412e35308b3e7ab708e\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"image to vector function\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/b956bef865580106b571e412e35308b3e7ab708e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:11Z\"}\n{\"id\":\"2489396802\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2624357,\"login\":\"mike-spainhower\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mike-spainhower\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2624357?\"},\"repo\":{\"id\":27418269,\"name\":\"LiveSafe/lvsf-opsworks-php-cookbook\",\"url\":\"https://api.github.com/repos/LiveSafe/lvsf-opsworks-php-cookbook\"},\"payload\":{\"ref\":\"1.0.7\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:12Z\",\"org\":{\"id\":9778464,\"login\":\"LiveSafe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/LiveSafe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9778464?\"}}\n{\"id\":\"2489396804\",\"type\":\"PushEvent\",\"actor\":{\"id\":7809663,\"login\":\"TheRingMaster\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TheRingMaster\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7809663?\"},\"repo\":{\"id\":28453562,\"name\":\"Team-Validus/packages_apps_Settings\",\"url\":\"https://api.github.com/repos/Team-Validus/packages_apps_Settings\"},\"payload\":{\"push_id\":536752666,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/lp5.0\",\"head\":\"e7a56745d952f34256729857ef14d4653a239e80\",\"before\":\"879f03889afd8fa8b52f57a24f8a354060cbeea8\",\"commits\":[{\"sha\":\"e7a56745d952f34256729857ef14d4653a239e80\",\"author\":{\"email\":\"a2771a13522a1cd683d518d8a964b5d8d1f00d68@hotmail.com\",\"name\":\"John Brewer\"},\"message\":\"derpp\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Team-Validus/packages_apps_Settings/commits/e7a56745d952f34256729857ef14d4653a239e80\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:12Z\",\"org\":{\"id\":10274350,\"login\":\"Team-Validus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Team-Validus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10274350?\"}}\n{\"id\":\"2489396808\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Brandon-Craig-Rhodes-TabSpace-Chords-Colored\",\"title\":\"Brandon Craig Rhodes TabSpace Chords Colored\",\"summary\":null,\"action\":\"edited\",\"sha\":\"bcca6468cba9dd67ac2605d00e9c0a6ce3fb9dbc\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Brandon-Craig-Rhodes-TabSpace-Chords-Colored\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:13Z\"}\n{\"id\":\"2489396817\",\"type\":\"PushEvent\",\"actor\":{\"id\":720376,\"login\":\"picodotdev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/picodotdev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/720376?\"},\"repo\":{\"id\":18271108,\"name\":\"picodotdev/blog-stack\",\"url\":\"https://api.github.com/repos/picodotdev/blog-stack\"},\"payload\":{\"push_id\":536752673,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"fbe0d7491c42bdac571af3a587319e78ffb19516\",\"before\":\"cd288dc8c7adb8b8c3049ad3b8956472d3e5e450\",\"commits\":[{\"sha\":\"fbe0d7491c42bdac571af3a587319e78ffb19516\",\"author\":{\"email\":\"82119ca92ec3f303155279e5d516019ee5c15a97@gmail.com\",\"name\":\"pico.dev\"},\"message\":\"Site updated Wednesday, 31-12-2014 20:02\",\"distinct\":true,\"url\":\"https://api.github.com/repos/picodotdev/blog-stack/commits/fbe0d7491c42bdac571af3a587319e78ffb19516\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:14Z\"}\n{\"id\":\"2489396819\",\"type\":\"PushEvent\",\"actor\":{\"id\":7825844,\"login\":\"praashie\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/praashie\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7825844?\"},\"repo\":{\"id\":25791959,\"name\":\"praashie/launchpad-visualizer\",\"url\":\"https://api.github.com/repos/praashie/launchpad-visualizer\"},\"payload\":{\"push_id\":536752675,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e01d22445a2098bcf6c73997bcc20afa9a5e926e\",\"before\":\"c4fdda6869f1e2ab3926d26efe7362c7a630068c\",\"commits\":[{\"sha\":\"e01d22445a2098bcf6c73997bcc20afa9a5e926e\",\"author\":{\"email\":\"3b921b218ce02163a39d90caa7b3b815b7e415aa@users.noreply.github.com\",\"name\":\"Praashie\"},\"message\":\"Added color palette, prettified code\",\"distinct\":true,\"url\":\"https://api.github.com/repos/praashie/launchpad-visualizer/commits/e01d22445a2098bcf6c73997bcc20afa9a5e926e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:14Z\"}\n{\"id\":\"2489396828\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3129699,\"login\":\"2xtreme\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/2xtreme\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3129699?\"},\"repo\":{\"id\":10263553,\"name\":\"ejurgensen/forked-daapd\",\"url\":\"https://api.github.com/repos/ejurgensen/forked-daapd\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:17Z\"}\n{\"id\":\"2489396829\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1265899,\"login\":\"lynas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lynas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1265899?\"},\"repo\":{\"id\":28678234,\"name\":\"lynas/springsecurity3.2\",\"url\":\"https://api.github.com/repos/lynas/springsecurity3.2\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:17Z\"}\n{\"id\":\"2489396831\",\"type\":\"GollumEvent\",\"actor\":{\"id\":46323,\"login\":\"paulcon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulcon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/46323?\"},\"repo\":{\"id\":28157780,\"name\":\"paulcon/active_subspaces\",\"url\":\"https://api.github.com/repos/paulcon/active_subspaces\"},\"payload\":{\"pages\":[{\"page_name\":\"_Footer\",\"title\":\"_Footer\",\"summary\":null,\"action\":\"created\",\"sha\":\"6f09c11d8614aa5c5b996a528905b2a8ae78d59f\",\"html_url\":\"https://github.com/paulcon/active_subspaces/wiki/_Footer\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:17Z\"}\n{\"id\":\"2489396835\",\"type\":\"PushEvent\",\"actor\":{\"id\":506010,\"login\":\"gabeshaughnessy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gabeshaughnessy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/506010?\"},\"repo\":{\"id\":13913264,\"name\":\"gabeshaughnessy/augmentedart\",\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart\"},\"payload\":{\"push_id\":536752682,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dungeon-hacker\",\"head\":\"070da10a642c004ef5bc428b8ce1bf88e410e7b4\",\"before\":\"49cd4f7ff271d91fedc4c45a85e590295ae59629\",\"commits\":[{\"sha\":\"070da10a642c004ef5bc428b8ce1bf88e410e7b4\",\"author\":{\"email\":\"a2b2bb6e7f1b10ac88b326d5c10e33af6a8546bc@gmail.com\",\"name\":\"gabeshaughnessy\"},\"message\":\"readme overview\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart/commits/070da10a642c004ef5bc428b8ce1bf88e410e7b4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:17Z\"}\n{\"id\":\"2489396843\",\"type\":\"PushEvent\",\"actor\":{\"id\":2048268,\"login\":\"unzan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/unzan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2048268?\"},\"repo\":{\"id\":19362226,\"name\":\"unzan/TerribleName\",\"url\":\"https://api.github.com/repos/unzan/TerribleName\"},\"payload\":{\"push_id\":536752684,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/alternate-version\",\"head\":\"81fee9bd4bb72b559e86b984dd75a52b29f7123b\",\"before\":\"048e306634e1334934b62dd234ae1a6a8daaef65\",\"commits\":[{\"sha\":\"81fee9bd4bb72b559e86b984dd75a52b29f7123b\",\"author\":{\"email\":\"10ccfcfbdc2deb68576e3809f6b38b3d0b355b5a@yahoo.com\",\"name\":\"unzan\"},\"message\":\"cleaned up some codes for buttons and texts\\n\\n- use `!important` on some button's properties. it's much shorter than the\\n  massive wall of selectors caused by abusing `&`\\n- removed gradient from disabled submit button\\n- fixed mod button when the only visible button is \\\"ignore reports\\\"\\n- removed `%tn-engraved-box` and `%tn-raised-box`. they're not needed\\n  anymore\\n- added `%tn-simple-text-container` used by wiki text, self post text\\n  and comment/message boxes.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/unzan/TerribleName/commits/81fee9bd4bb72b559e86b984dd75a52b29f7123b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:17Z\"}\n{\"id\":\"2489396856\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":597617,\"login\":\"soniktrooth\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/soniktrooth\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/597617?\"},\"repo\":{\"id\":28677072,\"name\":\"kalamuna/kalastatic\",\"url\":\"https://api.github.com/repos/kalamuna/kalastatic\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/3\",\"labels_url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/3/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/3/comments\",\"events_url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/3/events\",\"html_url\":\"https://github.com/kalamuna/kalastatic/issues/3\",\"id\":53209605,\"number\":3,\"title\":\"bash variable\",\"user\":{\"login\":\"soniktrooth\",\"id\":597617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/597617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/soniktrooth\",\"html_url\":\"https://github.com/soniktrooth\",\"followers_url\":\"https://api.github.com/users/soniktrooth/followers\",\"following_url\":\"https://api.github.com/users/soniktrooth/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/soniktrooth/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/soniktrooth/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/soniktrooth/subscriptions\",\"organizations_url\":\"https://api.github.com/users/soniktrooth/orgs\",\"repos_url\":\"https://api.github.com/users/soniktrooth/repos\",\"events_url\":\"https://api.github.com/users/soniktrooth/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/soniktrooth/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:28:01Z\",\"updated_at\":\"2015-01-01T01:02:19Z\",\"closed_at\":null,\"body\":\"My bash foo is weak. Can someone check what I've done with moving the repo address in travis-ci.sh into a variable? Unsure if it's a @RobLoach thing or a @andrewmallis thing. I will update this with a line number when I've pushed the initial code up.\"},\"comment\":{\"url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/comments/68477261\",\"html_url\":\"https://github.com/kalamuna/kalastatic/issues/3#issuecomment-68477261\",\"issue_url\":\"https://api.github.com/repos/kalamuna/kalastatic/issues/3\",\"id\":68477261,\"user\":{\"login\":\"soniktrooth\",\"id\":597617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/597617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/soniktrooth\",\"html_url\":\"https://github.com/soniktrooth\",\"followers_url\":\"https://api.github.com/users/soniktrooth/followers\",\"following_url\":\"https://api.github.com/users/soniktrooth/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/soniktrooth/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/soniktrooth/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/soniktrooth/subscriptions\",\"organizations_url\":\"https://api.github.com/users/soniktrooth/orgs\",\"repos_url\":\"https://api.github.com/users/soniktrooth/repos\",\"events_url\":\"https://api.github.com/users/soniktrooth/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/soniktrooth/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:19Z\",\"updated_at\":\"2015-01-01T01:02:19Z\",\"body\":\"Ok, line [47](https://github.com/kalamuna/kalastatic/blob/master/ci/travis-ci.sh#L47) is where I set up the var and [67](https://github.com/kalamuna/kalastatic/blob/master/ci/travis-ci.sh#L67) is where I used it.\\r\\n\\r\\nI also noticed just now that the check for $TRAVIS_REPO_SLUG is looking for \\\"kalamuna/pinnacle\\\" on line [50](https://github.com/kalamuna/kalastatic/blob/master/ci/travis-ci.sh#L50). I guess that will need to be a variable that can be set per project too but I'm unsure how that would have to be as it relates to Travis of which my foo is also weak. \"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:19Z\",\"org\":{\"id\":2373705,\"login\":\"kalamuna\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kalamuna\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2373705?\"}}\n{\"id\":\"2489396859\",\"type\":\"WatchEvent\",\"actor\":{\"id\":681965,\"login\":\"wonbyte\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wonbyte\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/681965?\"},\"repo\":{\"id\":13324424,\"name\":\"idris-hackers/idris-vim\",\"url\":\"https://api.github.com/repos/idris-hackers/idris-vim\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:19Z\",\"org\":{\"id\":3963683,\"login\":\"idris-hackers\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/idris-hackers\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3963683?\"}}\n{\"id\":\"2489396862\",\"type\":\"PushEvent\",\"actor\":{\"id\":1684950,\"login\":\"naijaping\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/naijaping\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1684950?\"},\"repo\":{\"id\":28650038,\"name\":\"naijaping/awonlist\",\"url\":\"https://api.github.com/repos/naijaping/awonlist\"},\"payload\":{\"push_id\":536752689,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e80946e9abd94a00ce15b06ad502211a518e2ffd\",\"before\":\"5286c7f80658d4fa04dc7a13901fe3b7ceefa3a4\",\"commits\":[{\"sha\":\"e80946e9abd94a00ce15b06ad502211a518e2ffd\",\"author\":{\"email\":\"8a1440b218d23a283d388025f7c9dc3555009ec5@gmail.com\",\"name\":\"naijaping\"},\"message\":\"Update uk\",\"distinct\":true,\"url\":\"https://api.github.com/repos/naijaping/awonlist/commits/e80946e9abd94a00ce15b06ad502211a518e2ffd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:20Z\"}\n{\"id\":\"2489396864\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":24660234,\"name\":\"jlumijarvi/excelimporter\",\"url\":\"https://api.github.com/repos/jlumijarvi/excelimporter\"},\"payload\":{\"push_id\":536752691,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0c317517b4de211b349b78609a2eb9a6872607da\",\"before\":\"be0ee8ea1d35abc029e24aecbbe2c0a68c8e1387\",\"commits\":[{\"sha\":\"0c317517b4de211b349b78609a2eb9a6872607da\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Create LICENSE.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/excelimporter/commits/0c317517b4de211b349b78609a2eb9a6872607da\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:20Z\"}\n{\"id\":\"2489396867\",\"type\":\"CreateEvent\",\"actor\":{\"id\":6863829,\"login\":\"essemfly\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/essemfly\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6863829?\"},\"repo\":{\"id\":28678193,\"name\":\"essemfly/myVirtualHome\",\"url\":\"https://api.github.com/repos/essemfly/myVirtualHome\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"laboratory\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396868\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":7058721,\"login\":\"alextegelid\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alextegelid\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7058721?\"},\"repo\":{\"id\":2469037,\"name\":\"kemayo/sublime-text-git\",\"url\":\"https://api.github.com/repos/kemayo/sublime-text-git\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/371\",\"labels_url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/371/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/371/comments\",\"events_url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/371/events\",\"html_url\":\"https://github.com/kemayo/sublime-text-git/issues/371\",\"id\":48203601,\"number\":371,\"title\":\"Cannot commit\",\"user\":{\"login\":\"evenfrost\",\"id\":3055750,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3055750?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/evenfrost\",\"html_url\":\"https://github.com/evenfrost\",\"followers_url\":\"https://api.github.com/users/evenfrost/followers\",\"following_url\":\"https://api.github.com/users/evenfrost/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/evenfrost/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/evenfrost/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/evenfrost/subscriptions\",\"organizations_url\":\"https://api.github.com/users/evenfrost/orgs\",\"repos_url\":\"https://api.github.com/users/evenfrost/repos\",\"events_url\":\"https://api.github.com/users/evenfrost/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/evenfrost/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":9,\"created_at\":\"2014-11-09T15:06:06Z\",\"updated_at\":\"2015-01-01T01:02:21Z\",\"closed_at\":null,\"body\":\"Every time I try to do git commit, I get this error: \\\"fatal: Unable to create '/home/evenfrost/work/projects/aword/.git/index.lock': File exists.\\\". There is no such file though. I believe this is somehow related to the fact that I've upgraded git to 2.1.0, cause guys from SO had similar problems with the plugin and newer versions of git.\"},\"comment\":{\"url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/comments/68477262\",\"html_url\":\"https://github.com/kemayo/sublime-text-git/issues/371#issuecomment-68477262\",\"issue_url\":\"https://api.github.com/repos/kemayo/sublime-text-git/issues/371\",\"id\":68477262,\"user\":{\"login\":\"alextegelid\",\"id\":7058721,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7058721?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alextegelid\",\"html_url\":\"https://github.com/alextegelid\",\"followers_url\":\"https://api.github.com/users/alextegelid/followers\",\"following_url\":\"https://api.github.com/users/alextegelid/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/alextegelid/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/alextegelid/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/alextegelid/subscriptions\",\"organizations_url\":\"https://api.github.com/users/alextegelid/orgs\",\"repos_url\":\"https://api.github.com/users/alextegelid/repos\",\"events_url\":\"https://api.github.com/users/alextegelid/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/alextegelid/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:21Z\",\"updated_at\":\"2015-01-01T01:02:21Z\",\"body\":\"I can confirm that the problem I experienced only occurred on a large repo aswell. All other repos were/are still working fine\\r\\n\\r\\n—\\r\\nAlexander Tegelid\\r\\n070-508 49 90\\r\\nalex@tegelid.se\\r\\n—\\r\\nSent from Mailbox App on iPhone.\\r\\n\\r\\nOn Thu, Jan 1, 2015 at 12:54 AM, John Watson <notifications@github.com>\\r\\nwrote:\\r\\n\\r\\n> I've only started experiencing this issue with larger repos (10k+ files) (st3, python3 branch). I did some experimenting and added print statements to `CommandThread` so I could see when threads were starting and finishing. Here's what I think is happening:\\r\\n> - When a commit is started, the plugin collects history and diffs and shows the commit buffer allowing the user to enter a commit message.\\r\\n> - The user then closes the buffer, starting the commit.\\r\\n> - At that moment, another buffer is activated (because the commit message buffer was closed) causing the GitBranchStatusCommand to run (statusbar.py)\\r\\n> - On these large repos, `git status --porcelain` does not finish until AFTER the `git commit` thread starts. `git status --porcelain` creates an index.lock file\\r\\n> The threads are running like this:\\r\\n> 1. Start status thread\\r\\n> 2. Start commit thread\\r\\n> 3. Finish commit (fails because status is still running)\\r\\n> 4. Finish status\\r\\n> Because `git status` creates an index.lock file, the commit fails. This doesn't affect me on small repos, I assume because `git status` runs much faster there. Sometimes `git status` runs fast enough even on large repos for this not to happen. I don't know what git is doing or why it's creating the lock file for a status.\\r\\n> I'm not sure how to resolve the issue. Perhaps just temporarily disable the GitBranchStatusListener listener during a commit (and then update the statusbar when the commit is finished). Or queue the commands. A workaround is to add `\\\"statusbar_status\\\": false` to the user settings.\\r\\n> ---\\r\\n> Reply to this email directly or view it on GitHub:\\r\\n> https://github.com/kemayo/sublime-text-git/issues/371#issuecomment-68475631\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396870\",\"type\":\"PushEvent\",\"actor\":{\"id\":170161,\"login\":\"mebigfatguy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mebigfatguy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/170161?\"},\"repo\":{\"id\":21152272,\"name\":\"mebigfatguy/fbaas\",\"url\":\"https://api.github.com/repos/mebigfatguy/fbaas\"},\"payload\":{\"push_id\":536752694,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3eb098edd4b8287fbf779bfe19f1b786c18996cb\",\"before\":\"3a78ece9dce4562acbf24e05d4c1f79e5b8814b3\",\"commits\":[{\"sha\":\"3eb098edd4b8287fbf779bfe19f1b786c18996cb\",\"author\":{\"email\":\"daf9f6e8320e49c5c7007b0c59aaad531fc83397@mebigfatguy.com\",\"name\":\"Dave Brosius\"},\"message\":\"differentiate different status\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mebigfatguy/fbaas/commits/3eb098edd4b8287fbf779bfe19f1b786c18996cb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396871\",\"type\":\"PushEvent\",\"actor\":{\"id\":4482745,\"login\":\"thedax\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/thedax\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4482745?\"},\"repo\":{\"id\":19226700,\"name\":\"thedax/nintendont-mirror\",\"url\":\"https://api.github.com/repos/thedax/nintendont-mirror\"},\"payload\":{\"push_id\":536752695,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"110d802334f7f4d1b01b43639585bfdad9f68bf5\",\"before\":\"e46036d65d64c9fb27c14e22872fd047aed65cf7\",\"commits\":[{\"sha\":\"110d802334f7f4d1b01b43639585bfdad9f68bf5\",\"author\":{\"email\":\"292e5fdbe23a481a99bcd6ac5ca3dd0ad2d7d447@yahoo.com\",\"name\":\"Howard\"},\"message\":\"-Added Trio Linker Plus II controller.ini (Thanks gillhaj02)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thedax/nintendont-mirror/commits/110d802334f7f4d1b01b43639585bfdad9f68bf5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396873\",\"type\":\"PushEvent\",\"actor\":{\"id\":6529689,\"login\":\"Conutant\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Conutant\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6529689?\"},\"repo\":{\"id\":28280392,\"name\":\"Conutant/TESSERACT\",\"url\":\"https://api.github.com/repos/Conutant/TESSERACT\"},\"payload\":{\"push_id\":536752697,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dev_andrei\",\"head\":\"cf3ff97c0ea2d16063914756cb392679d53ee5c8\",\"before\":\"eb99354eb7062b40f28369958e4b3a38757e46b5\",\"commits\":[{\"sha\":\"cf3ff97c0ea2d16063914756cb392679d53ee5c8\",\"author\":{\"email\":\"f7be974c25647e34f805d7b31d9b10b18614a452@gmail.com\",\"name\":\"Conutant\"},\"message\":\"Displaying comments, adding  padding\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Conutant/TESSERACT/commits/cf3ff97c0ea2d16063914756cb392679d53ee5c8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396880\",\"type\":\"PushEvent\",\"actor\":{\"id\":429706,\"login\":\"podviaznikov\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/podviaznikov\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/429706?\"},\"repo\":{\"id\":28254077,\"name\":\"podviaznikov/hellonode\",\"url\":\"https://api.github.com/repos/podviaznikov/hellonode\"},\"payload\":{\"push_id\":536752700,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b1e60af7541bcba479a2adc7b4e892219371dc4e\",\"before\":\"7edf6f05533eafe99dfffbb67bda921e6e341f61\",\"commits\":[{\"sha\":\"b1e60af7541bcba479a2adc7b4e892219371dc4e\",\"author\":{\"email\":\"9f35d6cbd99efe2457db40f739c2a082624247fb@gmail.com\",\"name\":\"Anton Podviaznikov\"},\"message\":\"Update server.js\",\"distinct\":true,\"url\":\"https://api.github.com/repos/podviaznikov/hellonode/commits/b1e60af7541bcba479a2adc7b4e892219371dc4e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:22Z\"}\n{\"id\":\"2489396885\",\"type\":\"PushEvent\",\"actor\":{\"id\":1558638,\"login\":\"trm36\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/trm36\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1558638?\"},\"repo\":{\"id\":28282084,\"name\":\"trm36/finalGrade\",\"url\":\"https://api.github.com/repos/trm36/finalGrade\"},\"payload\":{\"push_id\":536752706,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cacd357c0cb98995763482eeae7d98627a226048\",\"before\":\"a9cce18b4d22234ab644c4cf9daa27d32eff83ab\",\"commits\":[{\"sha\":\"cacd357c0cb98995763482eeae7d98627a226048\",\"author\":{\"email\":\"514b3b67f3b99c0e9708fdb5f8805e588c97ad00@humboldt.edu\",\"name\":\"Taylor Mott\"},\"message\":\"Adds user instructions\\n\\nSigned-off-by: Taylor Mott <trm36@humboldt.edu>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/trm36/finalGrade/commits/cacd357c0cb98995763482eeae7d98627a226048\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:23Z\"}\n{\"id\":\"2489396886\",\"type\":\"PushEvent\",\"actor\":{\"id\":94782,\"login\":\"ixti\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ixti\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/94782?\"},\"repo\":{\"id\":2524005,\"name\":\"tarcieri/http.rb\",\"url\":\"https://api.github.com/repos/tarcieri/http.rb\"},\"payload\":{\"push_id\":536752705,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b144b32f7b148df242604fdb7752c76f6053cb24\",\"before\":\"bbe9fe50b6ebe907a1cea8d2da00d2eacb9e06dd\",\"commits\":[{\"sha\":\"b144b32f7b148df242604fdb7752c76f6053cb24\",\"author\":{\"email\":\"e6e135d16927e8bca193effa00241a85121e93df@member.fsf.org\",\"name\":\"Aleksey V Zapparov\"},\"message\":\"Cleanup HTTP::Request spec\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tarcieri/http.rb/commits/b144b32f7b148df242604fdb7752c76f6053cb24\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:23Z\"}\n{\"id\":\"2489396891\",\"type\":\"WatchEvent\",\"actor\":{\"id\":10254377,\"login\":\"majache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/majache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10254377?\"},\"repo\":{\"id\":16190750,\"name\":\"evanbrooks/syntax-highlight\",\"url\":\"https://api.github.com/repos/evanbrooks/syntax-highlight\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:23Z\"}\n{\"id\":\"2489396892\",\"type\":\"PushEvent\",\"actor\":{\"id\":1221156,\"login\":\"fyfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fyfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1221156?\"},\"repo\":{\"id\":28673837,\"name\":\"fyfe/git-test\",\"url\":\"https://api.github.com/repos/fyfe/git-test\"},\"payload\":{\"push_id\":536752708,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"aeffe262b4f19a801509f8dcc9611a2e8b1c9c5f\",\"before\":\"25430ccc1c5249741feb237b0d0f1db055fe4b55\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:02:23Z\"}\n{\"id\":\"2489396895\",\"type\":\"PushEvent\",\"actor\":{\"id\":9831378,\"login\":\"Amit-P-Amin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?\"},\"repo\":{\"id\":28678217,\"name\":\"Amit-P-Amin/active_record_lite\",\"url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite\"},\"payload\":{\"push_id\":536752710,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/patch-1\",\"head\":\"b5ac327b20f2041652e3228ce4612880aae97beb\",\"before\":\"ca4e169591deec6c4b30b611e8ed5bdf89b3386a\",\"commits\":[{\"sha\":\"b5ac327b20f2041652e3228ce4612880aae97beb\",\"author\":{\"email\":\"b5bfea0f6fb804e6f8331851ae8acd4de4a8dcf9@gmail.com\",\"name\":\"Amit-P-Amin\"},\"message\":\"Update 03_associatable_spec.rb\\n\\nWas confused for a while because \\\"AssocOptions #model_class returns class of associated object\\\" was failing. This is because that spec tests 2 methods, but the description just mentions 1 (only model class, instead of model class and table name).\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/commits/b5ac327b20f2041652e3228ce4612880aae97beb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:24Z\"}\n{\"id\":\"2489396899\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":894251,\"login\":\"oubiwann\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oubiwann\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/894251?\"},\"repo\":{\"id\":27462056,\"name\":\"lfex/lsci\",\"url\":\"https://api.github.com/repos/lfex/lsci\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/lfex/lsci/issues/36\",\"labels_url\":\"https://api.github.com/repos/lfex/lsci/issues/36/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/lfex/lsci/issues/36/comments\",\"events_url\":\"https://api.github.com/repos/lfex/lsci/issues/36/events\",\"html_url\":\"https://github.com/lfex/lsci/issues/36\",\"id\":53210200,\"number\":36,\"title\":\"NaN and Infinity from NumPy crashes ErlPort\",\"user\":{\"login\":\"oubiwann\",\"id\":894251,\"avatar_url\":\"https://avatars.githubusercontent.com/u/894251?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oubiwann\",\"html_url\":\"https://github.com/oubiwann\",\"followers_url\":\"https://api.github.com/users/oubiwann/followers\",\"following_url\":\"https://api.github.com/users/oubiwann/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/oubiwann/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/oubiwann/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/oubiwann/subscriptions\",\"organizations_url\":\"https://api.github.com/users/oubiwann/orgs\",\"repos_url\":\"https://api.github.com/users/oubiwann/repos\",\"events_url\":\"https://api.github.com/users/oubiwann/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/oubiwann/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/lfex/lsci/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:02:24Z\",\"updated_at\":\"2015-01-01T01:02:24Z\",\"closed_at\":null,\"body\":\"Need to figure out how to properly pickle these for ErlPort's improved digestion ...\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:24Z\",\"org\":{\"id\":7939791,\"login\":\"lfex\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/lfex\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7939791?\"}}\n{\"id\":\"2489396902\",\"type\":\"PushEvent\",\"actor\":{\"id\":753926,\"login\":\"egisatoshi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/egisatoshi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/753926?\"},\"repo\":{\"id\":8212790,\"name\":\"egison/egison\",\"url\":\"https://api.github.com/repos/egison/egison\"},\"payload\":{\"push_id\":536752712,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"80dae4e133a2d4645a6105eeffddbcd1e22eed99\",\"before\":\"a6e78dde839fe67e410946bd8c168771e58cbbe1\",\"commits\":[{\"sha\":\"80dae4e133a2d4645a6105eeffddbcd1e22eed99\",\"author\":{\"email\":\"71e7d9a5a106c46ef4ccb3a433d905fea747e8cc@egison.org\",\"name\":\"Satoshi Egi\"},\"message\":\"update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/egison/egison/commits/80dae4e133a2d4645a6105eeffddbcd1e22eed99\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:25Z\",\"org\":{\"id\":6812884,\"login\":\"egison\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/egison\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6812884?\"}}\n{\"id\":\"2489396903\",\"type\":\"CreateEvent\",\"actor\":{\"id\":8918426,\"login\":\"Spekular\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Spekular\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8918426?\"},\"repo\":{\"id\":28678100,\"name\":\"Spekular/Project-Version\",\"url\":\"https://api.github.com/repos/Spekular/Project-Version\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Adds Information popup when projects created with older or newer LMMS versions.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:25Z\"}\n{\"id\":\"2489396913\",\"type\":\"PushEvent\",\"actor\":{\"id\":4858543,\"login\":\"mtklr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mtklr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4858543?\"},\"repo\":{\"id\":28118151,\"name\":\"mtklr/howard\",\"url\":\"https://api.github.com/repos/mtklr/howard\"},\"payload\":{\"push_id\":536752715,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"756c131344998885363d26d74a5698f910a2dcb1\",\"before\":\"d5b4d89f63a7fedf51ac911b6df897b21466189a\",\"commits\":[{\"sha\":\"756c131344998885363d26d74a5698f910a2dcb1\",\"author\":{\"email\":\"b9663e167dc762614bed337db2d0a7492d0ce96b@gmail.com\",\"name\":\"Matt Keller\"},\"message\":\"add #202 title\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mtklr/howard/commits/756c131344998885363d26d74a5698f910a2dcb1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:27Z\"}\n{\"id\":\"2489396915\",\"type\":\"PushEvent\",\"actor\":{\"id\":542045,\"login\":\"apipe-tester\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apipe-tester\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/542045?\"},\"repo\":{\"id\":23442737,\"name\":\"genome/genome\",\"url\":\"https://api.github.com/repos/genome/genome\"},\"payload\":{\"push_id\":536752716,\"size\":30,\"distinct_size\":0,\"ref\":\"refs/heads/snapshot\",\"head\":\"65a175d735f2507c7ce46239fc8192f7b1fabcfd\",\"before\":\"7184935531d3758833760b60be75a69fc2100afa\",\"commits\":[{\"sha\":\"02256b48b155180783a99a0ef318da86a1e2d178\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"New way to read a bam-readcount file\\n\\nAssuming that the file is sorted by position, this allows you to\\nget entries by position.  You have to read the positions almost in\\norder, although you can look back into a buffer of one entry.\\nWe need this because vcf entries can look at multiple bam-readcount\\nentries, depending on whether they have deletion alternate alleles.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/02256b48b155180783a99a0ef318da86a1e2d178\"},{\"sha\":\"942b44ab8ef5628e72985d11006aef09924028db\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Move bam-readcount vcf-safe encode and decode to a separate class\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/942b44ab8ef5628e72985d11006aef09924028db\"},{\"sha\":\"a40e73ea13267236ab8499cd095a87c77f0b25e1\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Factor out is_deletion\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/a40e73ea13267236ab8499cd095a87c77f0b25e1\"},{\"sha\":\"a7bfeb1898edda953544ff65722679cbf750b0d3\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"create_deletion_entry is always called with bam_readcount_line_deletion\\n\\nDon't need to pass it as a param or export/import it\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/a7bfeb1898edda953544ff65722679cbf750b0d3\"},{\"sha\":\"79b180206258cb3120ac57b6277147544aace88a\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Bam readcount test helper gets entries from vcf files\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/79b180206258cb3120ac57b6277147544aace88a\"},{\"sha\":\"6f0eb821ca4cc346c332903abd909d98f0a9cad8\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Add a bam readcount parser that can encode and decode bam-readcount lines in vcf\\n\\nLines from bam-readcount are stored in a sample format field as json.\\nThe json contains a hash with two types of keys:\\n-alternate alleles.  The values of these entries point to which offset key\\nshould be checked to find the bam-readcount line for that allele.\\n-offsets.  The key is the offset from the vcf position on this line (since\\nbam readcount interprets deletion start positions differently, they will be 1; all\\nother types of normalized vcf alleles will be 0).  The value is the bam-readcount line.\\n\\nThe json is made vcf sample field safe using the encode function\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/6f0eb821ca4cc346c332903abd909d98f0a9cad8\"},{\"sha\":\"913c134a75265a5f5c083f767d0d3bfbd3b74efa\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Correctly mock up test data\\n\\nThe single-bam (\\\"normal\\\") build should return the alignment\\nresult from the normal sample as its merged_aligned_bam_result\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/913c134a75265a5f5c083f767d0d3bfbd3b74efa\"},{\"sha\":\"39c9a2bcf9dead867f16fbc9c89f0b602a8840e3\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Use the new BamReadcountParser to get allele-specific bam-readcounts from vcf\\n\\n-gmt annotate-with-readcounts now uses BamReadcountParser\\n-update all of the interpreters and filters that use bam-readcount to\\nhandle bam-readcount entries on a per-allele and per-sample basis rather than on a\\nstrictly per-sample basis.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/39c9a2bcf9dead867f16fbc9c89f0b602a8840e3\"},{\"sha\":\"d0d1a46075772341a6a627530b6346354f602c35\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Test doesn't actually depend on bam-readcount\\n\\nSince the test is just using the bam-readcount test helper\\nas a convenient way to make a vcf entry, use the simpler\\nentry that doesn't actually contain bam-readcount output\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/d0d1a46075772341a6a627530b6346354f602c35\"},{\"sha\":\"0bd03ea587b5d6cc2cdf83aa5356c9d170d15505\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Variant callers interpreter takes a list of valid callers\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/0bd03ea587b5d6cc2cdf83aa5356c9d170d15505\"},{\"sha\":\"794dad92553ad30f03c6c539647b743aeb64cdf1\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Enforce no duplicate entries in bam-readcount file\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/794dad92553ad30f03c6c539647b743aeb64cdf1\"},{\"sha\":\"14099eee5b5c68fd647a4669b274ce6c256f6634\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Rename some variables to make them easier to read\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/14099eee5b5c68fd647a4669b274ce6c256f6634\"},{\"sha\":\"b9655006e621a59f1f4e8e3529889dbe1b4f37f1\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Fix bam-readcount reader\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/b9655006e621a59f1f4e8e3529889dbe1b4f37f1\"},{\"sha\":\"be1d571a7149b25d16228578a84327c8458652c9\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Handle vcf entries with no alternate alleles\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/be1d571a7149b25d16228578a84327c8458652c9\"},{\"sha\":\"64306ea12dd8d3dc538f7eb5637ff36dba750567\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Regions list for bam readcount should be de-duplicated\\n\\nKeep the sort order of the chromosomes the same as in the vcf file\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/64306ea12dd8d3dc538f7eb5637ff36dba750567\"},{\"sha\":\"20228e5869e518eafc7e71c3345b2f650de5412a\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Add missing use statement\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/20228e5869e518eafc7e71c3345b2f650de5412a\"},{\"sha\":\"cbcf18f5553082355415acac0a2db31f71232fee\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Rename class\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/cbcf18f5553082355415acac0a2db31f71232fee\"},{\"sha\":\"dedc88e580410e57d7c4c417d520cb806fc322e8\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Remove unnecessary conditional\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/dedc88e580410e57d7c4c417d520cb806fc322e8\"},{\"sha\":\"aaa4cd72a8690ac5cd916c5be748c782cd00d510\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Make the buffered reader more robust\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/aaa4cd72a8690ac5cd916c5be748c782cd00d510\"},{\"sha\":\"81c8d713309c76520ae1766e79ca3d5628ed95eb\",\"author\":{\"email\":\"ee527c7899460c7ad6b19d12059ad61f80cf4acd@genome.wustl.edu\",\"name\":\"apregier\"},\"message\":\"Simplify function\",\"distinct\":false,\"url\":\"https://api.github.com/repos/genome/genome/commits/81c8d713309c76520ae1766e79ca3d5628ed95eb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:27Z\",\"org\":{\"id\":318108,\"login\":\"genome\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/genome\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/318108?\"}}\n{\"id\":\"2489396917\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3117019,\"login\":\"robotbrain\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robotbrain\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3117019?\"},\"repo\":{\"id\":12682212,\"name\":\"MultiMC/MultiMC5\",\"url\":\"https://api.github.com/repos/MultiMC/MultiMC5\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/704\",\"labels_url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/704/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/704/comments\",\"events_url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/704/events\",\"html_url\":\"https://github.com/MultiMC/MultiMC5/issues/704\",\"id\":53210008,\"number\":704,\"title\":\"Development optin\",\"user\":{\"login\":\"bl968\",\"id\":4886192,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4886192?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bl968\",\"html_url\":\"https://github.com/bl968\",\"followers_url\":\"https://api.github.com/users/bl968/followers\",\"following_url\":\"https://api.github.com/users/bl968/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bl968/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bl968/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bl968/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bl968/orgs\",\"repos_url\":\"https://api.github.com/users/bl968/repos\",\"events_url\":\"https://api.github.com/users/bl968/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bl968/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:51:03Z\",\"updated_at\":\"2015-01-01T01:02:27Z\",\"closed_at\":null,\"body\":\"Give us a checkbox to allow installation of development versions of forge and liteloader for the client\"},\"comment\":{\"url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/comments/68477263\",\"html_url\":\"https://github.com/MultiMC/MultiMC5/issues/704#issuecomment-68477263\",\"issue_url\":\"https://api.github.com/repos/MultiMC/MultiMC5/issues/704\",\"id\":68477263,\"user\":{\"login\":\"robotbrain\",\"id\":3117019,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3117019?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robotbrain\",\"html_url\":\"https://github.com/robotbrain\",\"followers_url\":\"https://api.github.com/users/robotbrain/followers\",\"following_url\":\"https://api.github.com/users/robotbrain/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/robotbrain/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/robotbrain/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/robotbrain/subscriptions\",\"organizations_url\":\"https://api.github.com/users/robotbrain/orgs\",\"repos_url\":\"https://api.github.com/users/robotbrain/repos\",\"events_url\":\"https://api.github.com/users/robotbrain/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/robotbrain/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:27Z\",\"updated_at\":\"2015-01-01T01:02:27Z\",\"body\":\"liteloader: no, mumfrey doesnt give us a way to do that\\r\\nforge: already supported for actual forge, just not fml by itself\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:27Z\",\"org\":{\"id\":5411890,\"login\":\"MultiMC\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MultiMC\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5411890?\"}}\n{\"id\":\"2489396923\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3087225,\"login\":\"zephraph\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zephraph\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3087225?\"},\"repo\":{\"id\":6794407,\"name\":\"jpsarda/Pixel-based-destructible-ground-with-Cocos2d-iPhone\",\"url\":\"https://api.github.com/repos/jpsarda/Pixel-based-destructible-ground-with-Cocos2d-iPhone\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:28Z\"}\n{\"id\":\"2489396925\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1778966,\"login\":\"emage\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/emage\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1778966?\"},\"repo\":{\"id\":1653882,\"name\":\"thoughtbot/bourbon\",\"url\":\"https://api.github.com/repos/thoughtbot/bourbon\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/thoughtbot/bourbon/issues/605\",\"labels_url\":\"https://api.github.com/repos/thoughtbot/bourbon/issues/605/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/thoughtbot/bourbon/issues/605/comments\",\"events_url\":\"https://api.github.com/repos/thoughtbot/bourbon/issues/605/events\",\"html_url\":\"https://github.com/thoughtbot/bourbon/issues/605\",\"id\":53210202,\"number\":605,\"title\":\"Bourbon CSS error using Scout compiler on Windows\",\"user\":{\"login\":\"emage\",\"id\":1778966,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1778966?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/emage\",\"html_url\":\"https://github.com/emage\",\"followers_url\":\"https://api.github.com/users/emage/followers\",\"following_url\":\"https://api.github.com/users/emage/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/emage/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/emage/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/emage/subscriptions\",\"organizations_url\":\"https://api.github.com/users/emage/orgs\",\"repos_url\":\"https://api.github.com/users/emage/repos\",\"events_url\":\"https://api.github.com/users/emage/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/emage/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:02:28Z\",\"updated_at\":\"2015-01-01T01:02:28Z\",\"closed_at\":null,\"body\":\"I am just starting to get into SASS.  I recently installed Ruby and Bourbon on my window machine and using Scout to compile.  Everything is going well until i import the bourbon files using:\\r\\n\\r\\n@import \\\"bourbon/bourbon\\\";\\r\\n\\r\\nI am not sure if its the Scout Compile output error or Bourbon.\\r\\n\\r\\nCompile Error:\\r\\n\\r\\nCommon error is the Invalid CSS:\\r\\n-------------------------------------------\\r\\n>>> Change detected at 16:48:33 to: main.scss\\r\\nerror main.scss (Line 22 of _font-source-declaration.scss: Invalid CSS after \\\" eot\\\": expected \\\")\\\", was \\\": $font-url +...\\\")\\r\\noverwrite main.css \\r\\n>>> Change detected at 16:50:46 to: bourbon/_bourbon.scss\\r\\nerror main.scss (Line 21 of _linear-angle-parser.scss: Invalid CSS after \\\" webkit-image\\\": expected \\\")\\\", was \\\": -webkit- + $p...\\\")\\r\\noverwrite main.css \\r\\n>>> Change detected at 16:51:01 to: main.scss\\r\\nerror main.scss (Line 21 of _linear-angle-parser.scss: Invalid CSS after \\\" webkit-image\\\": expected \\\")\\\", was \\\": -webkit- + $p...\\\")\\r\\nidentical main.css \\r\\n----------------------------------------------------\\r\\n\\r\\nI went into the bourbon.scss to remove the said error imported file, it just goes to the next import file and output the same error.\\r\\n\\r\\nAny thought or suggestion?\\r\\n\\r\\nThanks\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:28Z\",\"org\":{\"id\":6183,\"login\":\"thoughtbot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/thoughtbot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6183?\"}}\n{\"id\":\"2489396932\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":703007,\"login\":\"r4j4h\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r4j4h\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/703007?\"},\"repo\":{\"id\":14075249,\"name\":\"zhanghuancs/D3.js-Link-Filter\",\"url\":\"https://api.github.com/repos/zhanghuancs/D3.js-Link-Filter\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/zhanghuancs/D3.js-Link-Filter/issues/1\",\"labels_url\":\"https://api.github.com/repos/zhanghuancs/D3.js-Link-Filter/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/zhanghuancs/D3.js-Link-Filter/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/zhanghuancs/D3.js-Link-Filter/issues/1/events\",\"html_url\":\"https://github.com/zhanghuancs/D3.js-Link-Filter/issues/1\",\"id\":53210203,\"number\":1,\"title\":\"Sometimes labels get clipped\",\"user\":{\"login\":\"r4j4h\",\"id\":703007,\"avatar_url\":\"https://avatars.githubusercontent.com/u/703007?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r4j4h\",\"html_url\":\"https://github.com/r4j4h\",\"followers_url\":\"https://api.github.com/users/r4j4h/followers\",\"following_url\":\"https://api.github.com/users/r4j4h/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/r4j4h/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/r4j4h/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/r4j4h/subscriptions\",\"organizations_url\":\"https://api.github.com/users/r4j4h/orgs\",\"repos_url\":\"https://api.github.com/users/r4j4h/repos\",\"events_url\":\"https://api.github.com/users/r4j4h/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/r4j4h/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:02:29Z\",\"updated_at\":\"2015-01-01T01:02:29Z\",\"closed_at\":null,\"body\":\"Great work! I found one issue I want to raise awareness around.\\r\\n\\r\\nTo recreate, go to demo page at `http://jsfiddle.net/zhanghuancs/cuYu8/`\\r\\n\\r\\nand then\\r\\n\\r\\n1. Uncheck licensing\\r\\n2. Uncheck suit\\r\\n3. Uncheck resolved\\r\\n4. Recheck resolved\\r\\n\\r\\n_Notice the bottom strand has only the LG label._\\r\\n\\r\\n1. Recheck suit\\r\\n2. Uncheck suit\\r\\n\\r\\n_Notice how Samsung and Kodak labels are now present._\\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:29Z\"}\n{\"id\":\"2489396936\",\"type\":\"PushEvent\",\"actor\":{\"id\":1727112,\"login\":\"nucleardreamer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nucleardreamer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1727112?\"},\"repo\":{\"id\":28675721,\"name\":\"nucleardreamer/tdc\",\"url\":\"https://api.github.com/repos/nucleardreamer/tdc\"},\"payload\":{\"push_id\":536752725,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"07a6ae5cdb39596d5aed1b3794229968810bce1f\",\"before\":\"dca862ebee0ce9425e5e9a00ab856d32f5e178e9\",\"commits\":[{\"sha\":\"07a6ae5cdb39596d5aed1b3794229968810bce1f\",\"author\":{\"email\":\"61996574b2cb2f66e0c7dcf5a13359213bfaba5a@gmail.com\",\"name\":\"nucleardreamer\"},\"message\":\"repeating background movie\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nucleardreamer/tdc/commits/07a6ae5cdb39596d5aed1b3794229968810bce1f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:30Z\"}\n{\"id\":\"2489396940\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2829718,\"login\":\"phister\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/phister\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829718?\"},\"repo\":{\"id\":28678235,\"name\":\"phister/Cfb\",\"url\":\"https://api.github.com/repos/phister/Cfb\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"College football playoffs\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:32Z\"}\n{\"id\":\"2489396941\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5317,\"login\":\"textgoeshere\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/textgoeshere\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5317?\"},\"repo\":{\"id\":146167,\"name\":\"tyler/trie\",\"url\":\"https://api.github.com/repos/tyler/trie\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:32Z\"}\n{\"id\":\"2489396946\",\"type\":\"PushEvent\",\"actor\":{\"id\":1031119,\"login\":\"wavewave\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wavewave\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1031119?\"},\"repo\":{\"id\":4619831,\"name\":\"wavewave/hoodle\",\"url\":\"https://api.github.com/repos/wavewave/hoodle\"},\"payload\":{\"push_id\":536752730,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1fffe5741ea097ea926cf8693061f2f741da71c8\",\"before\":\"482d9eb8ea9fbdbb2f39d27779dbf7dcd749db8d\",\"commits\":[{\"sha\":\"1fffe5741ea097ea926cf8693061f2f741da71c8\",\"author\":{\"email\":\"0a0eb067dd98769f500dabe6a55682ac6cdd3c08@gmail.com\",\"name\":\"Ian-Woo Kim\"},\"message\":\"introduce RendererState in Renderer monad (simply Reader.) renderCache -> renderCacheVar using TVar for asynchronous update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wavewave/hoodle/commits/1fffe5741ea097ea926cf8693061f2f741da71c8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:32Z\"}\n{\"id\":\"2489396949\",\"type\":\"PushEvent\",\"actor\":{\"id\":7725188,\"login\":\"hellerve\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hellerve\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7725188?\"},\"repo\":{\"id\":27544900,\"name\":\"hellerve/hiss\",\"url\":\"https://api.github.com/repos/hellerve/hiss\"},\"payload\":{\"push_id\":536752732,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"55772d134ab1da7db17da6350549eee504666acb\",\"before\":\"a791dd0816d3a263e215334f3f0ba936642a2251\",\"commits\":[{\"sha\":\"55772d134ab1da7db17da6350549eee504666acb\",\"author\":{\"email\":\"9f918abde07a882492d318338ec4304e4a124b0a@student.htw-berlin.de\",\"name\":\"Veit Heller\"},\"message\":\"Removed unnecessary enum\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hellerve/hiss/commits/55772d134ab1da7db17da6350549eee504666acb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:32Z\"}\n{\"id\":\"2489396950\",\"type\":\"PushEvent\",\"actor\":{\"id\":8620186,\"login\":\"hoorayhu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hoorayhu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8620186?\"},\"repo\":{\"id\":28647744,\"name\":\"hoorayhu/sodoku-cc\",\"url\":\"https://api.github.com/repos/hoorayhu/sodoku-cc\"},\"payload\":{\"push_id\":536752733,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"16cb1762122432f441586c990a14c948874366f7\",\"before\":\"35cdf012648c32078ee0a310e39875fed02f68e0\",\"commits\":[{\"sha\":\"16cb1762122432f441586c990a14c948874366f7\",\"author\":{\"email\":\"042dc4512fa3d391c5170cf3aa61e6a638f84342@hoorayhu.com\",\"name\":\"Hooray Hu\"},\"message\":\"Applying MVC, making it playable for alpha testing\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hoorayhu/sodoku-cc/commits/16cb1762122432f441586c990a14c948874366f7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:32Z\"}\n{\"id\":\"2489396966\",\"type\":\"PushEvent\",\"actor\":{\"id\":6565577,\"login\":\"CurrentResident\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CurrentResident\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6565577?\"},\"repo\":{\"id\":27690802,\"name\":\"CurrentResident/MendifactQuake\",\"url\":\"https://api.github.com/repos/CurrentResident/MendifactQuake\"},\"payload\":{\"push_id\":536752743,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4769846610c03ef389d6c7345e67dcf1fb4069d8\",\"before\":\"42db2228115b4098670d851ab3823ac59fa890e0\",\"commits\":[{\"sha\":\"4769846610c03ef389d6c7345e67dcf1fb4069d8\",\"author\":{\"email\":\"828e27bdf506438b277b61a8f22de5998c7b8e91@gmail.com\",\"name\":\"Jim Thoenen\"},\"message\":\"Remove some unused globals and commented-out original id code\",\"distinct\":true,\"url\":\"https://api.github.com/repos/CurrentResident/MendifactQuake/commits/4769846610c03ef389d6c7345e67dcf1fb4069d8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:33Z\"}\n{\"id\":\"2489396972\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":6964047,\"login\":\"TTMTT\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?\"},\"repo\":{\"id\":28678195,\"name\":\"TTMTT/iCL0udin\",\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"labels_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/events\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1\",\"id\":53210206,\"number\":1,\"title\":\"Discuss1\",\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:02:34Z\",\"updated_at\":\"2015-01-01T01:02:34Z\",\"closed_at\":null,\"body\":\"Now you can download vresion 1.0 from :\\r\\n---------------------------------------------------\\r\\nhttp://www.icloudin.net\\r\\n-----------------------------\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n--------------------------------------\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n---------------------------------------\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n-------------------------------------\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n-------------------------------------\\r\\niCL0udin v1.0 -> %100\\r\\n----------------------------\\r\\nRemaining: %3 testing with some people..\\r\\n-----------------------------------------------------\\r\\nLast Method:\\r\\n-----------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\niCL0udin v1.0 have this method:\\r\\n-----------------------------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:34Z\"}\n{\"id\":\"2489396976\",\"type\":\"PushEvent\",\"actor\":{\"id\":369807,\"login\":\"tarr11\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tarr11\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/369807?\"},\"repo\":{\"id\":28670957,\"name\":\"tarr11/wizbang\",\"url\":\"https://api.github.com/repos/tarr11/wizbang\"},\"payload\":{\"push_id\":536752748,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"bc001c10242651bd44db229260e70b9fa810d12e\",\"before\":\"63e6a6e27a5f3ec455f526bb18c453e8cafb9854\",\"commits\":[{\"sha\":\"bc001c10242651bd44db229260e70b9fa810d12e\",\"author\":{\"email\":\"25c75c966e2ccc8083891108da2f841d165bf0eb@gmail.com\",\"name\":\"Douglas Tarr\"},\"message\":\"Update and rename README.rdoc to README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tarr11/wizbang/commits/bc001c10242651bd44db229260e70b9fa810d12e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:35Z\"}\n{\"id\":\"2489396978\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":447792,\"login\":\"nathanjsweet\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?\"},\"repo\":{\"id\":15714889,\"name\":\"nathanjsweet/nodehun\",\"url\":\"https://api.github.com/repos/nathanjsweet/nodehun\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13\",\"labels_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/comments\",\"events_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/events\",\"html_url\":\"https://github.com/nathanjsweet/nodehun/issues/13\",\"id\":51903837,\"number\":13,\"title\":\"Memory Releasing issue\",\"user\":{\"login\":\"nathanjsweet\",\"id\":447792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"html_url\":\"https://github.com/nathanjsweet\",\"followers_url\":\"https://api.github.com/users/nathanjsweet/followers\",\"following_url\":\"https://api.github.com/users/nathanjsweet/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathanjsweet/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathanjsweet/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathanjsweet/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathanjsweet/orgs\",\"repos_url\":\"https://api.github.com/users/nathanjsweet/repos\",\"events_url\":\"https://api.github.com/users/nathanjsweet/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathanjsweet/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"nathanjsweet\",\"id\":447792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"html_url\":\"https://github.com/nathanjsweet\",\"followers_url\":\"https://api.github.com/users/nathanjsweet/followers\",\"following_url\":\"https://api.github.com/users/nathanjsweet/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathanjsweet/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathanjsweet/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathanjsweet/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathanjsweet/orgs\",\"repos_url\":\"https://api.github.com/users/nathanjsweet/repos\",\"events_url\":\"https://api.github.com/users/nathanjsweet/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathanjsweet/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-14T00:03:14Z\",\"updated_at\":\"2015-01-01T01:02:35Z\",\"closed_at\":\"2015-01-01T01:02:35Z\",\"body\":\"For Someone named Matt:\\r\\n\\r\\nHi, Nathan,\\r\\n\\r\\nI'm working on a node.js-based spell-checking service and was delighted to stumble upon [1]. Thanks for creating/maintaining this hunspell binding.\\r\\n\\r\\nI'm running into an issue, on my Mac (OSX Yosemite) where the node.js process crashes with 'Segmentation fault: 11' randomly after as few as 11 HTTP requests and up to ~100 HTTP requests.\\r\\n\\r\\nIf I cache a single nodehun instance and reuse that, I can run my test suite (contains ~5500 inputs) repeatedly.\\r\\n\\r\\nAlas, the requirement of the service is that it can accept requests for various combinations of dictionaries and word lists, so I'll likely need to create a custom instance for many requests (I can probably employ some caching).\\r\\n\\r\\nMy pattern for instantiating nodehun which produces the results above was that shown in examples/a_init.js (new nodehun(aff, dict)).\\r\\n\\r\\nI modified the code to use the pattern in g_asyncinvoke.js, and that's producing the following:\\r\\n\\r\\nnode(10787,0x103c8d000) malloc: *** error for object 0x104039008: incorrect checksum for freed object - object was probably modified after being freed.\\r\\n*** set a breakpoint in malloc_error_break to debug\\r\\n\\r\\nIn an attempt to simplify / make the problem more easy to repo, I've attached a modified version of the g_asyncinvoke.js example. It looks like the issue (hopefully equivalent to what I'm seeing in my service) can be reproduced after ~35 iterations.\\r\\n\\r\\nFWIW, I'm using 64-bit node v0.10.33.\\r\\n\\r\\nPlease let me know if I can provide additional info. I can also take this to [2] if you prefer.\\r\\n\\r\\nRegards,\\r\\nMatt\\r\\n\\r\\nRelevant file: https://docs.google.com/document/d/1qamzmStKEogJI3ybaJKNgT8xLiSCc2JdWAnJ-3CFk-c/edit?usp=sharing\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489396977\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":447792,\"login\":\"nathanjsweet\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?\"},\"repo\":{\"id\":15714889,\"name\":\"nathanjsweet/nodehun\",\"url\":\"https://api.github.com/repos/nathanjsweet/nodehun\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13\",\"labels_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/comments\",\"events_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13/events\",\"html_url\":\"https://github.com/nathanjsweet/nodehun/issues/13\",\"id\":51903837,\"number\":13,\"title\":\"Memory Releasing issue\",\"user\":{\"login\":\"nathanjsweet\",\"id\":447792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"html_url\":\"https://github.com/nathanjsweet\",\"followers_url\":\"https://api.github.com/users/nathanjsweet/followers\",\"following_url\":\"https://api.github.com/users/nathanjsweet/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathanjsweet/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathanjsweet/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathanjsweet/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathanjsweet/orgs\",\"repos_url\":\"https://api.github.com/users/nathanjsweet/repos\",\"events_url\":\"https://api.github.com/users/nathanjsweet/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathanjsweet/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"nathanjsweet\",\"id\":447792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"html_url\":\"https://github.com/nathanjsweet\",\"followers_url\":\"https://api.github.com/users/nathanjsweet/followers\",\"following_url\":\"https://api.github.com/users/nathanjsweet/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathanjsweet/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathanjsweet/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathanjsweet/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathanjsweet/orgs\",\"repos_url\":\"https://api.github.com/users/nathanjsweet/repos\",\"events_url\":\"https://api.github.com/users/nathanjsweet/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathanjsweet/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-14T00:03:14Z\",\"updated_at\":\"2015-01-01T01:02:35Z\",\"closed_at\":\"2015-01-01T01:02:35Z\",\"body\":\"For Someone named Matt:\\r\\n\\r\\nHi, Nathan,\\r\\n\\r\\nI'm working on a node.js-based spell-checking service and was delighted to stumble upon [1]. Thanks for creating/maintaining this hunspell binding.\\r\\n\\r\\nI'm running into an issue, on my Mac (OSX Yosemite) where the node.js process crashes with 'Segmentation fault: 11' randomly after as few as 11 HTTP requests and up to ~100 HTTP requests.\\r\\n\\r\\nIf I cache a single nodehun instance and reuse that, I can run my test suite (contains ~5500 inputs) repeatedly.\\r\\n\\r\\nAlas, the requirement of the service is that it can accept requests for various combinations of dictionaries and word lists, so I'll likely need to create a custom instance for many requests (I can probably employ some caching).\\r\\n\\r\\nMy pattern for instantiating nodehun which produces the results above was that shown in examples/a_init.js (new nodehun(aff, dict)).\\r\\n\\r\\nI modified the code to use the pattern in g_asyncinvoke.js, and that's producing the following:\\r\\n\\r\\nnode(10787,0x103c8d000) malloc: *** error for object 0x104039008: incorrect checksum for freed object - object was probably modified after being freed.\\r\\n*** set a breakpoint in malloc_error_break to debug\\r\\n\\r\\nIn an attempt to simplify / make the problem more easy to repo, I've attached a modified version of the g_asyncinvoke.js example. It looks like the issue (hopefully equivalent to what I'm seeing in my service) can be reproduced after ~35 iterations.\\r\\n\\r\\nFWIW, I'm using 64-bit node v0.10.33.\\r\\n\\r\\nPlease let me know if I can provide additional info. I can also take this to [2] if you prefer.\\r\\n\\r\\nRegards,\\r\\nMatt\\r\\n\\r\\nRelevant file: https://docs.google.com/document/d/1qamzmStKEogJI3ybaJKNgT8xLiSCc2JdWAnJ-3CFk-c/edit?usp=sharing\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/comments/68477264\",\"html_url\":\"https://github.com/nathanjsweet/nodehun/issues/13#issuecomment-68477264\",\"issue_url\":\"https://api.github.com/repos/nathanjsweet/nodehun/issues/13\",\"id\":68477264,\"user\":{\"login\":\"nathanjsweet\",\"id\":447792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanjsweet\",\"html_url\":\"https://github.com/nathanjsweet\",\"followers_url\":\"https://api.github.com/users/nathanjsweet/followers\",\"following_url\":\"https://api.github.com/users/nathanjsweet/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathanjsweet/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathanjsweet/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathanjsweet/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathanjsweet/orgs\",\"repos_url\":\"https://api.github.com/users/nathanjsweet/repos\",\"events_url\":\"https://api.github.com/users/nathanjsweet/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathanjsweet/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:35Z\",\"updated_at\":\"2015-01-01T01:02:35Z\",\"body\":\"Matt,\\r\\nI'm not on Yosemite, yet, I will update later to check this out. I'm hoping that you plan on deploying on a linux server. If that's the case, then I would say you shouldn't have to worry about this issue (I've tested this script on several linux boxes and they all worked fine).\\r\\n\\r\\nThis is going to sound like a cop out, but OSX sucks. I run into all sorts of memory and segmentation issues that just don't crop up on linux (32 bit or 64) or even windows (again 32/64). I have personally played around a lot with libuv on my macbook and my impression of the teams for it, v8, and nodeJS is that apple is an afterthought for them.\\r\\n\\r\\nFor example, there are many hacks out there to get node to work around the 256 file-descriptor per process limit on OSX, but nobody has bothered to include any of these into libuv or node.\\r\\n\\r\\nI'm going to mark this issue as closed for now. If this ever becomes an issue on linux or windows let me know.\\r\\nThanks,\\r\\nNate\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489396985\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536752750,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"53ba5e0e3194a42192f5aa9f9cd3fd2d79281ce4\",\"before\":\"d989a60a4e050573d9ba85048499d9a0a7e8117e\",\"commits\":[{\"sha\":\"53ba5e0e3194a42192f5aa9f9cd3fd2d79281ce4\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"fix binding issues with relative paths etc; don't add bindings for each items or with blocks\\n\\nConflicts:\\n\\tpackage.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/53ba5e0e3194a42192f5aa9f9cd3fd2d79281ce4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489396987\",\"type\":\"PushEvent\",\"actor\":{\"id\":6287026,\"login\":\"unixabg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/unixabg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6287026?\"},\"repo\":{\"id\":23363281,\"name\":\"unixabg/ts-yt-dl\",\"url\":\"https://api.github.com/repos/unixabg/ts-yt-dl\"},\"payload\":{\"push_id\":536752753,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cc14de8e1f3c8402100c787405c1b8ef420978af\",\"before\":\"d4fd67f93ff9f5ba614b0df495f67843f96c3b6c\",\"commits\":[{\"sha\":\"cc14de8e1f3c8402100c787405c1b8ef420978af\",\"author\":{\"email\":\"82b6c79a2ca714d2dcf70b1e2e7c5fc1711a4b07@gmail.com\",\"name\":\"Richard Nelson\"},\"message\":\"Added youtube-dl version to scripts/support.php.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/unixabg/ts-yt-dl/commits/cc14de8e1f3c8402100c787405c1b8ef420978af\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489396988\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2188638,\"login\":\"manuel-rubio\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/manuel-rubio\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2188638?\"},\"repo\":{\"id\":1035547,\"name\":\"inaka/apns4erl\",\"url\":\"https://api.github.com/repos/inaka/apns4erl\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/inaka/apns4erl/issues/39\",\"labels_url\":\"https://api.github.com/repos/inaka/apns4erl/issues/39/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/inaka/apns4erl/issues/39/comments\",\"events_url\":\"https://api.github.com/repos/inaka/apns4erl/issues/39/events\",\"html_url\":\"https://github.com/inaka/apns4erl/pull/39\",\"id\":53184390,\"number\":39,\"title\":\"APNS resend queue\",\"user\":{\"login\":\"alexdruzhilov\",\"id\":490425,\"avatar_url\":\"https://avatars.githubusercontent.com/u/490425?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexdruzhilov\",\"html_url\":\"https://github.com/alexdruzhilov\",\"followers_url\":\"https://api.github.com/users/alexdruzhilov/followers\",\"following_url\":\"https://api.github.com/users/alexdruzhilov/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/alexdruzhilov/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/alexdruzhilov/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/alexdruzhilov/subscriptions\",\"organizations_url\":\"https://api.github.com/users/alexdruzhilov/orgs\",\"repos_url\":\"https://api.github.com/users/alexdruzhilov/repos\",\"events_url\":\"https://api.github.com/users/alexdruzhilov/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/alexdruzhilov/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T13:40:31Z\",\"updated_at\":\"2015-01-01T01:02:37Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/inaka/apns4erl/pulls/39\",\"html_url\":\"https://github.com/inaka/apns4erl/pull/39\",\"diff_url\":\"https://github.com/inaka/apns4erl/pull/39.diff\",\"patch_url\":\"https://github.com/inaka/apns4erl/pull/39.patch\"},\"body\":\"Fix for issue https://github.com/inaka/apns4erl/issues/17\\r\\n\\r\\nCommit is adopted for version 1.0.2 from two commits:\\r\\nhttps://github.com/altenwald/apns4erl/commit/06431ef47ce3593ce0635375f0ba786cea9fecae\\r\\nhttps://github.com/altenwald/apns4erl/commit/618e52cb350742f4391d35768f20c4e1619061d0\"},\"comment\":{\"url\":\"https://api.github.com/repos/inaka/apns4erl/issues/comments/68477265\",\"html_url\":\"https://github.com/inaka/apns4erl/pull/39#issuecomment-68477265\",\"issue_url\":\"https://api.github.com/repos/inaka/apns4erl/issues/39\",\"id\":68477265,\"user\":{\"login\":\"manuel-rubio\",\"id\":2188638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2188638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/manuel-rubio\",\"html_url\":\"https://github.com/manuel-rubio\",\"followers_url\":\"https://api.github.com/users/manuel-rubio/followers\",\"following_url\":\"https://api.github.com/users/manuel-rubio/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/manuel-rubio/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/manuel-rubio/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/manuel-rubio/subscriptions\",\"organizations_url\":\"https://api.github.com/users/manuel-rubio/orgs\",\"repos_url\":\"https://api.github.com/users/manuel-rubio/repos\",\"events_url\":\"https://api.github.com/users/manuel-rubio/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/manuel-rubio/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:37Z\",\"updated_at\":\"2015-01-01T01:02:37Z\",\"body\":\"Fixed: altenwald/apns4erl@b59e7a0\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\",\"org\":{\"id\":867053,\"login\":\"inaka\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/inaka\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/867053?\"}}\n{\"id\":\"2489396992\",\"type\":\"PushEvent\",\"actor\":{\"id\":3960243,\"login\":\"wp-plugins-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wp-plugins-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3960243?\"},\"repo\":{\"id\":19549672,\"name\":\"wp-plugins/facebook-secret-meta\",\"url\":\"https://api.github.com/repos/wp-plugins/facebook-secret-meta\"},\"payload\":{\"push_id\":536752756,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4302987080a9f480e078903f1fb948b025beed8c\",\"before\":\"8623e15f41d0186a6ce8cd402c06ba3299a96577\",\"commits\":[{\"sha\":\"4302987080a9f480e078903f1fb948b025beed8c\",\"author\":{\"email\":\"c84fcc9b3d1ea987f4c82b2df8285ca158ac8f6b@b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"name\":\"Asif2BD\"},\"message\":\"4.1 version compatible\\n\\ngit-svn-id: https://plugins.svn.wordpress.org/facebook-secret-meta/trunk@1057713 b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wp-plugins/facebook-secret-meta/commits/4302987080a9f480e078903f1fb948b025beed8c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\",\"org\":{\"id\":2996849,\"login\":\"wp-plugins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wp-plugins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2996849?\"}}\n{\"id\":\"2489396995\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1238621,\"login\":\"Corosauce\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Corosauce\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1238621?\"},\"repo\":{\"id\":19960516,\"name\":\"Corosauce/weather2\",\"url\":\"https://api.github.com/repos/Corosauce/weather2\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2\",\"labels_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/events\",\"html_url\":\"https://github.com/Corosauce/weather2/issues/2\",\"id\":52972104,\"number\":2,\"title\":\"Support for snow from other mods\",\"user\":{\"login\":\"dexman545\",\"id\":4873274,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4873274?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dexman545\",\"html_url\":\"https://github.com/dexman545\",\"followers_url\":\"https://api.github.com/users/dexman545/followers\",\"following_url\":\"https://api.github.com/users/dexman545/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dexman545/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dexman545/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dexman545/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dexman545/orgs\",\"repos_url\":\"https://api.github.com/users/dexman545/repos\",\"events_url\":\"https://api.github.com/users/dexman545/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dexman545/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-28T03:55:02Z\",\"updated_at\":\"2015-01-01T01:02:37Z\",\"closed_at\":\"2015-01-01T01:02:37Z\",\"body\":\"Would it be possible to change Blocks.snow in StormObject.java to Material.snow so that mods that add their own snow will be populated on the ground? In a discussion with one of the TFC devs, this was theorized as a possible solution to Weather 2's storms not placing snow.\\r\\n\\r\\n(http://terrafirmacraft.com/f/topic/6945-1710-technofirma-mod-pack/page-5#entry101107)\\r\\n\\r\\nThank you, \\r\\ndex\"},\"comment\":{\"url\":\"https://api.github.com/repos/Corosauce/weather2/issues/comments/68477266\",\"html_url\":\"https://github.com/Corosauce/weather2/issues/2#issuecomment-68477266\",\"issue_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2\",\"id\":68477266,\"user\":{\"login\":\"Corosauce\",\"id\":1238621,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1238621?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Corosauce\",\"html_url\":\"https://github.com/Corosauce\",\"followers_url\":\"https://api.github.com/users/Corosauce/followers\",\"following_url\":\"https://api.github.com/users/Corosauce/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Corosauce/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Corosauce/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Corosauce/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Corosauce/orgs\",\"repos_url\":\"https://api.github.com/users/Corosauce/repos\",\"events_url\":\"https://api.github.com/users/Corosauce/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Corosauce/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:37Z\",\"updated_at\":\"2015-01-01T01:02:37Z\",\"body\":\"Hmm yeah, still sounds better than the current situation so I'll do what I can on my part with checking against material type instead of block and we'll see how it plays together in the future.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489396996\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1238621,\"login\":\"Corosauce\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Corosauce\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1238621?\"},\"repo\":{\"id\":19960516,\"name\":\"Corosauce/weather2\",\"url\":\"https://api.github.com/repos/Corosauce/weather2\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2\",\"labels_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/Corosauce/weather2/issues/2/events\",\"html_url\":\"https://github.com/Corosauce/weather2/issues/2\",\"id\":52972104,\"number\":2,\"title\":\"Support for snow from other mods\",\"user\":{\"login\":\"dexman545\",\"id\":4873274,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4873274?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dexman545\",\"html_url\":\"https://github.com/dexman545\",\"followers_url\":\"https://api.github.com/users/dexman545/followers\",\"following_url\":\"https://api.github.com/users/dexman545/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dexman545/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dexman545/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dexman545/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dexman545/orgs\",\"repos_url\":\"https://api.github.com/users/dexman545/repos\",\"events_url\":\"https://api.github.com/users/dexman545/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dexman545/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-28T03:55:02Z\",\"updated_at\":\"2015-01-01T01:02:37Z\",\"closed_at\":\"2015-01-01T01:02:37Z\",\"body\":\"Would it be possible to change Blocks.snow in StormObject.java to Material.snow so that mods that add their own snow will be populated on the ground? In a discussion with one of the TFC devs, this was theorized as a possible solution to Weather 2's storms not placing snow.\\r\\n\\r\\n(http://terrafirmacraft.com/f/topic/6945-1710-technofirma-mod-pack/page-5#entry101107)\\r\\n\\r\\nThank you, \\r\\ndex\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489396997\",\"type\":\"PushEvent\",\"actor\":{\"id\":498666,\"login\":\"yaboyanees\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yaboyanees\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/498666?\"},\"repo\":{\"id\":28573192,\"name\":\"yaboyanees/audibleDSS\",\"url\":\"https://api.github.com/repos/yaboyanees/audibleDSS\"},\"payload\":{\"push_id\":536752758,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7213655448351edf7420c7ffc675bf9fe08348b3\",\"before\":\"5f3e41d9a87b429dbf00877aba66f68b99664803\",\"commits\":[{\"sha\":\"7213655448351edf7420c7ffc675bf9fe08348b3\",\"author\":{\"email\":\"26e9cab5de48070b64acff125a9ff6569947913f@yahoo.com\",\"name\":\"yaboyanees\"},\"message\":\"cleaning up views with css and small form formatting\",\"distinct\":true,\"url\":\"https://api.github.com/repos/yaboyanees/audibleDSS/commits/7213655448351edf7420c7ffc675bf9fe08348b3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:37Z\"}\n{\"id\":\"2489397003\",\"type\":\"WatchEvent\",\"actor\":{\"id\":630099,\"login\":\"Nevyn357\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Nevyn357\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/630099?\"},\"repo\":{\"id\":8668792,\"name\":\"ilirb/ahk-scripts\",\"url\":\"https://api.github.com/repos/ilirb/ahk-scripts\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:39Z\"}\n{\"id\":\"2489397005\",\"type\":\"PushEvent\",\"actor\":{\"id\":7046289,\"login\":\"ValentinGabriel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ValentinGabriel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7046289?\"},\"repo\":{\"id\":28188264,\"name\":\"Jeremj0/RebootFactory\",\"url\":\"https://api.github.com/repos/Jeremj0/RebootFactory\"},\"payload\":{\"push_id\":536752759,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/Master\",\"head\":\"966dfc06ccc1c565f5a91f6ae9b5b71501d8d6cb\",\"before\":\"4eaaa79af7f99f45c706c3745f5c56b46ba14a4b\",\"commits\":[{\"sha\":\"966dfc06ccc1c565f5a91f6ae9b5b71501d8d6cb\",\"author\":{\"email\":\"40615740715968d1630b9e998e85827713fc82bb@hotmail.fr\",\"name\":\"ValentinGabriel\"},\"message\":\"Etude de cas\\n\\nhtml et css quasi vierge, à compléter.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Jeremj0/RebootFactory/commits/966dfc06ccc1c565f5a91f6ae9b5b71501d8d6cb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:40Z\"}\n{\"id\":\"2489397007\",\"type\":\"PushEvent\",\"actor\":{\"id\":8819701,\"login\":\"r-ggraham\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r-ggraham\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8819701?\"},\"repo\":{\"id\":28678173,\"name\":\"r-ggraham/Crumpet_Bot\",\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot\"},\"payload\":{\"push_id\":536752761,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"51951542e9d518f59ff62753725e5186d4111822\",\"before\":\"332746658edcc60bca17720fce153357a2cd9dfb\",\"commits\":[{\"sha\":\"51951542e9d518f59ff62753725e5186d4111822\",\"author\":{\"email\":\"f2f9dd43aa4244d32208a2ccfa0c7c9e9c48f7e7@uni.worc.ac.uk\",\"name\":\"Rob G\"},\"message\":\"Remove .gitFiles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot/commits/51951542e9d518f59ff62753725e5186d4111822\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:40Z\"}\n{\"id\":\"2489397015\",\"type\":\"PushEvent\",\"actor\":{\"id\":3960243,\"login\":\"wp-plugins-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wp-plugins-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3960243?\"},\"repo\":{\"id\":27287679,\"name\":\"wp-plugins/woocommerce-eu-vat-compliance\",\"url\":\"https://api.github.com/repos/wp-plugins/woocommerce-eu-vat-compliance\"},\"payload\":{\"push_id\":536752764,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"af0132f4527cc3eaa4838c86c7280d4a304ad9f8\",\"before\":\"4d08fb010b86a14f694d7ae6ebe0e169379072be\",\"commits\":[{\"sha\":\"af0132f4527cc3eaa4838c86c7280d4a304ad9f8\",\"author\":{\"email\":\"77d7830c4a7293f46a95bf3929d24b006a29ede7@b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"name\":\"DavidAnderson\"},\"message\":\"1.6.7\\n\\ngit-svn-id: https://plugins.svn.wordpress.org/woocommerce-eu-vat-compliance/trunk@1057712 b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wp-plugins/woocommerce-eu-vat-compliance/commits/af0132f4527cc3eaa4838c86c7280d4a304ad9f8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:41Z\",\"org\":{\"id\":2996849,\"login\":\"wp-plugins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wp-plugins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2996849?\"}}\n{\"id\":\"2489397016\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2829600,\"login\":\"GrahamCampbell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GrahamCampbell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829600?\"},\"repo\":{\"id\":26730195,\"name\":\"cachethq/Cachet\",\"url\":\"https://api.github.com/repos/cachethq/Cachet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"labels_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/comments\",\"events_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/events\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173\",\"id\":53210024,\"number\":173,\"title\":\"Bug: Forms let you submit multiple times\",\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2015-01-01T00:52:06Z\",\"updated_at\":\"2015-01-01T01:02:40Z\",\"closed_at\":null,\"body\":\"When adding a new incident, I noticed a weird bug.\\r\\n\\r\\nIf you fill in the form as normal, then click the submit button twice really quickly, it'll create __TWO__ identical new incidents!\\r\\n\\r\\nThis could be a bit annoying, a simple fix is using a bit of JS that on submit, disables the submit button so that once clicked, it cannot be clicked again.\"},\"comment\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/comments/68477269\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173#issuecomment-68477269\",\"issue_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"id\":68477269,\"user\":{\"login\":\"GrahamCampbell\",\"id\":2829600,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829600?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GrahamCampbell\",\"html_url\":\"https://github.com/GrahamCampbell\",\"followers_url\":\"https://api.github.com/users/GrahamCampbell/followers\",\"following_url\":\"https://api.github.com/users/GrahamCampbell/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/GrahamCampbell/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/GrahamCampbell/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/GrahamCampbell/subscriptions\",\"organizations_url\":\"https://api.github.com/users/GrahamCampbell/orgs\",\"repos_url\":\"https://api.github.com/users/GrahamCampbell/repos\",\"events_url\":\"https://api.github.com/users/GrahamCampbell/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/GrahamCampbell/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:02:40Z\",\"updated_at\":\"2015-01-01T01:02:40Z\",\"body\":\"No, it doesn't. That's something different. It doesn't prevent multiple submissions of a form.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:41Z\",\"org\":{\"id\":9951502,\"login\":\"cachethq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cachethq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9951502?\"}}\n{\"id\":\"2489397020\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":5573038,\"login\":\"CorruptComputer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?\"},\"repo\":{\"id\":28371640,\"name\":\"AsteroidStation/-tg-station\",\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station\"},\"payload\":{\"action\":\"opened\",\"number\":8,\"pull_request\":{\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8\",\"id\":26739422,\"html_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8\",\"diff_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8.diff\",\"patch_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8.patch\",\"issue_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8\",\"number\":8,\"state\":\"open\",\"locked\":false,\"title\":\"tg update\",\"user\":{\"login\":\"CorruptComputer\",\"id\":5573038,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"html_url\":\"https://github.com/CorruptComputer\",\"followers_url\":\"https://api.github.com/users/CorruptComputer/followers\",\"following_url\":\"https://api.github.com/users/CorruptComputer/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CorruptComputer/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CorruptComputer/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CorruptComputer/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CorruptComputer/orgs\",\"repos_url\":\"https://api.github.com/users/CorruptComputer/repos\",\"events_url\":\"https://api.github.com/users/CorruptComputer/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CorruptComputer/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:02:41Z\",\"updated_at\":\"2015-01-01T01:02:41Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/commits\",\"review_comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/comments\",\"review_comment_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8/comments\",\"statuses_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/34c4c027770518e2087db44a89cad07cd91ceb84\",\"head\":{\"label\":\"tgstation:master\",\"ref\":\"master\",\"sha\":\"34c4c027770518e2087db44a89cad07cd91ceb84\",\"user\":{\"login\":\"tgstation\",\"id\":1363778,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1363778?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tgstation\",\"html_url\":\"https://github.com/tgstation\",\"followers_url\":\"https://api.github.com/users/tgstation/followers\",\"following_url\":\"https://api.github.com/users/tgstation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tgstation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tgstation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tgstation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tgstation/orgs\",\"repos_url\":\"https://api.github.com/users/tgstation/repos\",\"events_url\":\"https://api.github.com/users/tgstation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tgstation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":3234987,\"name\":\"-tg-station\",\"full_name\":\"tgstation/-tg-station\",\"owner\":{\"login\":\"tgstation\",\"id\":1363778,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1363778?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tgstation\",\"html_url\":\"https://github.com/tgstation\",\"followers_url\":\"https://api.github.com/users/tgstation/followers\",\"following_url\":\"https://api.github.com/users/tgstation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tgstation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tgstation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tgstation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tgstation/orgs\",\"repos_url\":\"https://api.github.com/users/tgstation/repos\",\"events_url\":\"https://api.github.com/users/tgstation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tgstation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/tgstation/-tg-station\",\"description\":\"/tg/'s SS13 branch\",\"fork\":false,\"url\":\"https://api.github.com/repos/tgstation/-tg-station\",\"forks_url\":\"https://api.github.com/repos/tgstation/-tg-station/forks\",\"keys_url\":\"https://api.github.com/repos/tgstation/-tg-station/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/tgstation/-tg-station/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/tgstation/-tg-station/teams\",\"hooks_url\":\"https://api.github.com/repos/tgstation/-tg-station/hooks\",\"issue_events_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/tgstation/-tg-station/events\",\"assignees_url\":\"https://api.github.com/repos/tgstation/-tg-station/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/tgstation/-tg-station/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/tgstation/-tg-station/tags\",\"blobs_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/tgstation/-tg-station/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/tgstation/-tg-station/languages\",\"stargazers_url\":\"https://api.github.com/repos/tgstation/-tg-station/stargazers\",\"contributors_url\":\"https://api.github.com/repos/tgstation/-tg-station/contributors\",\"subscribers_url\":\"https://api.github.com/repos/tgstation/-tg-station/subscribers\",\"subscription_url\":\"https://api.github.com/repos/tgstation/-tg-station/subscription\",\"commits_url\":\"https://api.github.com/repos/tgstation/-tg-station/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/tgstation/-tg-station/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/tgstation/-tg-station/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/tgstation/-tg-station/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/tgstation/-tg-station/merges\",\"archive_url\":\"https://api.github.com/repos/tgstation/-tg-station/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/tgstation/-tg-station/downloads\",\"issues_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/tgstation/-tg-station/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/tgstation/-tg-station/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/tgstation/-tg-station/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/tgstation/-tg-station/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/tgstation/-tg-station/releases{/id}\",\"created_at\":\"2012-01-21T17:32:47Z\",\"updated_at\":\"2015-01-01T00:47:49Z\",\"pushed_at\":\"2015-01-01T00:47:48Z\",\"git_url\":\"git://github.com/tgstation/-tg-station.git\",\"ssh_url\":\"git@github.com:tgstation/-tg-station.git\",\"clone_url\":\"https://github.com/tgstation/-tg-station.git\",\"svn_url\":\"https://github.com/tgstation/-tg-station\",\"homepage\":\"http://www.tgstation13.org/\",\"size\":550715,\"stargazers_count\":172,\"watchers_count\":172,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":609,\"mirror_url\":null,\"open_issues_count\":618,\"forks\":609,\"open_issues\":618,\"watchers\":172,\"default_branch\":\"master\"}},\"base\":{\"label\":\"AsteroidStation:master\",\"ref\":\"master\",\"sha\":\"d98abaabaa971c9092a200961915c9e60015ab20\",\"user\":{\"login\":\"AsteroidStation\",\"id\":9313156,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AsteroidStation\",\"html_url\":\"https://github.com/AsteroidStation\",\"followers_url\":\"https://api.github.com/users/AsteroidStation/followers\",\"following_url\":\"https://api.github.com/users/AsteroidStation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AsteroidStation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AsteroidStation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AsteroidStation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AsteroidStation/orgs\",\"repos_url\":\"https://api.github.com/users/AsteroidStation/repos\",\"events_url\":\"https://api.github.com/users/AsteroidStation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AsteroidStation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":28371640,\"name\":\"-tg-station\",\"full_name\":\"AsteroidStation/-tg-station\",\"owner\":{\"login\":\"AsteroidStation\",\"id\":9313156,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AsteroidStation\",\"html_url\":\"https://github.com/AsteroidStation\",\"followers_url\":\"https://api.github.com/users/AsteroidStation/followers\",\"following_url\":\"https://api.github.com/users/AsteroidStation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AsteroidStation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AsteroidStation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AsteroidStation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AsteroidStation/orgs\",\"repos_url\":\"https://api.github.com/users/AsteroidStation/repos\",\"events_url\":\"https://api.github.com/users/AsteroidStation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AsteroidStation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/AsteroidStation/-tg-station\",\"description\":\"/tg/'s SS13 branch\",\"fork\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station\",\"forks_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/forks\",\"keys_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/teams\",\"hooks_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/hooks\",\"issue_events_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/events\",\"assignees_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/tags\",\"blobs_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/languages\",\"stargazers_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/stargazers\",\"contributors_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/contributors\",\"subscribers_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/subscribers\",\"subscription_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/subscription\",\"commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/merges\",\"archive_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/downloads\",\"issues_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/releases{/id}\",\"created_at\":\"2014-12-23T02:11:11Z\",\"updated_at\":\"2014-12-31T02:35:57Z\",\"pushed_at\":\"2014-12-31T02:35:56Z\",\"git_url\":\"git://github.com/AsteroidStation/-tg-station.git\",\"ssh_url\":\"git@github.com:AsteroidStation/-tg-station.git\",\"clone_url\":\"https://github.com/AsteroidStation/-tg-station.git\",\"svn_url\":\"https://github.com/AsteroidStation/-tg-station\",\"homepage\":\"http://www.tgstation13.org/\",\"size\":414377,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8\"},\"html\":{\"href\":\"https://github.com/AsteroidStation/-tg-station/pull/8\"},\"issue\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8\"},\"comments\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/34c4c027770518e2087db44a89cad07cd91ceb84\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":55,\"additions\":1841,\"deletions\":1195,\"changed_files\":145}},\"public\":true,\"created_at\":\"2015-01-01T01:02:41Z\",\"org\":{\"id\":9313156,\"login\":\"AsteroidStation\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/AsteroidStation\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?\"}}\n{\"id\":\"2489397024\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2684528,\"login\":\"pAs2aL\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pAs2aL\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2684528?\"},\"repo\":{\"id\":28649619,\"name\":\"pAs2aL/nightjar\",\"url\":\"https://api.github.com/repos/pAs2aL/nightjar\"},\"payload\":{\"ref\":\"gh-pages\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"A 2D cooperative role playing game\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:41Z\"}\n{\"id\":\"2489397025\",\"type\":\"PushEvent\",\"actor\":{\"id\":5043639,\"login\":\"akbar-sh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/akbar-sh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5043639?\"},\"repo\":{\"id\":28675338,\"name\":\"akbar-sh/tracker\",\"url\":\"https://api.github.com/repos/akbar-sh/tracker\"},\"payload\":{\"push_id\":536752768,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"88fede14a8bc23b1e9654a297922a0791f7522d1\",\"before\":\"c45885fad735da88cd6dc0fbabd83a647a7c0289\",\"commits\":[{\"sha\":\"87bc2f0f2b96ef7847dfd4d65bcaeef8182e1711\",\"author\":{\"email\":\"7eb7e51160615686946d5e19147edf697a6d53a2@MacBook.local\",\"name\":\"Akbar Sharifi\"},\"message\":\"Removing facebook keys.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/akbar-sh/tracker/commits/87bc2f0f2b96ef7847dfd4d65bcaeef8182e1711\"},{\"sha\":\"88fede14a8bc23b1e9654a297922a0791f7522d1\",\"author\":{\"email\":\"7eb7e51160615686946d5e19147edf697a6d53a2@MacBook.local\",\"name\":\"Akbar Sharifi\"},\"message\":\"Merge branch 'master' of https://github.com/akbar-sh/tracker\",\"distinct\":true,\"url\":\"https://api.github.com/repos/akbar-sh/tracker/commits/88fede14a8bc23b1e9654a297922a0791f7522d1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:41Z\"}\n{\"id\":\"2489397036\",\"type\":\"PushEvent\",\"actor\":{\"id\":6241554,\"login\":\"leo-yuriev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leo-yuriev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6241554?\"},\"repo\":{\"id\":23696666,\"name\":\"leo-yuriev/openldap-lmdb-challenge\",\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge\"},\"payload\":{\"push_id\":536752772,\"size\":14,\"distinct_size\":14,\"ref\":\"refs/heads/2.4-devel\",\"head\":\"f705475af08e6b50420a8ff22b71cfbe45cea409\",\"before\":\"6dc7947be776bbce629aac14e588307e2889740b\",\"commits\":[{\"sha\":\"d99cbb501031c413632e403be0ea63281270e47b\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"ps-build.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/d99cbb501031c413632e403be0ea63281270e47b\"},{\"sha\":\"758e5973787cc5cdf089dc860a05393594a88410\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"qt-creator project\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/758e5973787cc5cdf089dc860a05393594a88410\"},{\"sha\":\"e79179f8971b6f5bd0b2de68fc52911644be5099\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix debug marco\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/e79179f8971b6f5bd0b2de68fc52911644be5099\"},{\"sha\":\"251d0e76442dfd449eb60fa95c3512b88f7d2c28\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: using strerror_r()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/251d0e76442dfd449eb60fa95c3512b88f7d2c28\"},{\"sha\":\"dea8bab3a83841845cbd5e7eaae856a00b944c9e\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: read/write ignored result.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/dea8bab3a83841845cbd5e7eaae856a00b944c9e\"},{\"sha\":\"ae9384015ad55a10cdc94ce4e0d7db629345342a\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: check getcmd() result\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/ae9384015ad55a10cdc94ce4e0d7db629345342a\"},{\"sha\":\"1c6e1547e8a51177bedca0296288c38745161afe\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: printf unused agrv[0] in main()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/1c6e1547e8a51177bedca0296288c38745161afe\"},{\"sha\":\"5b8dc917757526f0290c2bb7dda9f06dbd621ab6\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: printf %d without arg in main()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/5b8dc917757526f0290c2bb7dda9f06dbd621ab6\"},{\"sha\":\"28182c76f57c5c328a302b4867a3dda600b99b7d\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: warning-errors for configure\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/28182c76f57c5c328a302b4867a3dda600b99b7d\"},{\"sha\":\"64725efd6f4369f3985d6785ef3c1c9f9452be4e\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: build-warnings (most 'unused')\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/64725efd6f4369f3985d6785ef3c1c9f9452be4e\"},{\"sha\":\"9011ed6d444ae30acd17fb3b335ae901e5927bd7\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: const\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/9011ed6d444ae30acd17fb3b335ae901e5927bd7\"},{\"sha\":\"72a7cc8702ce5394b1c78c976064f37fd75920f9\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: warnings (uninitialized)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/72a7cc8702ce5394b1c78c976064f37fd75920f9\"},{\"sha\":\"26d527311aec553be799e2ab27a407be58c79ec6\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"sasl callback's typecast\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/26d527311aec553be799e2ab27a407be58c79ec6\"},{\"sha\":\"f705475af08e6b50420a8ff22b71cfbe45cea409\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix warnings (unused, uninitialized, misc);\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/f705475af08e6b50420a8ff22b71cfbe45cea409\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:43Z\"}\n{\"id\":\"2489397037\",\"type\":\"PushEvent\",\"actor\":{\"id\":1932804,\"login\":\"coldmind\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?\"},\"repo\":{\"id\":18882135,\"name\":\"coldmind/django\",\"url\":\"https://api.github.com/repos/coldmind/django\"},\"payload\":{\"push_id\":536752773,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/ticket_24064\",\"head\":\"f1231658677318fd416661219e1480a46fba4b71\",\"before\":\"7cec2b4d707716d83aad641e84b1daf064811ca8\",\"commits\":[{\"sha\":\"f1231658677318fd416661219e1480a46fba4b71\",\"author\":{\"email\":\"1f135da694de981cc7e3d5bc8ed1049495ca7439@yandex.ru\",\"name\":\"Andriy Sokolovskiy\"},\"message\":\"Fixed #24064 - Prevented database access in compile time in spatialite models\",\"distinct\":true,\"url\":\"https://api.github.com/repos/coldmind/django/commits/f1231658677318fd416661219e1480a46fba4b71\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:43Z\"}\n{\"id\":\"2489397039\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1742369,\"login\":\"gbathree\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gbathree\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1742369?\"},\"repo\":{\"id\":20705156,\"name\":\"Photosynq/PhotosynQ-ChromeApp\",\"url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/14\",\"labels_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/14/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/14/comments\",\"events_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/14/events\",\"html_url\":\"https://github.com/Photosynq/PhotosynQ-ChromeApp/issues/14\",\"id\":53210212,\"number\":14,\"title\":\"get rid of 'get_offset' function from protocol creator - it's not used anymore\",\"user\":{\"login\":\"gbathree\",\"id\":1742369,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1742369?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gbathree\",\"html_url\":\"https://github.com/gbathree\",\"followers_url\":\"https://api.github.com/users/gbathree/followers\",\"following_url\":\"https://api.github.com/users/gbathree/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gbathree/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gbathree/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gbathree/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gbathree/orgs\",\"repos_url\":\"https://api.github.com/users/gbathree/repos\",\"events_url\":\"https://api.github.com/users/gbathree/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gbathree/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:02:43Z\",\"updated_at\":\"2015-01-01T01:02:43Z\",\"closed_at\":null,\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:43Z\",\"org\":{\"id\":5068236,\"login\":\"Photosynq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Photosynq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5068236?\"}}\n{\"id\":\"2489397043\",\"type\":\"PushEvent\",\"actor\":{\"id\":3964764,\"login\":\"elliekimpot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/elliekimpot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3964764?\"},\"repo\":{\"id\":28625867,\"name\":\"elliekimpot/msm\",\"url\":\"https://api.github.com/repos/elliekimpot/msm\"},\"payload\":{\"push_id\":536752776,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/feature/removed\",\"head\":\"6393887f2296f3e7d7820ceefa0177003261ec68\",\"before\":\"2c854302b53eef234eae379576a870f9ef9aa3bd\",\"commits\":[{\"sha\":\"6393887f2296f3e7d7820ceefa0177003261ec68\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_ERR_CRASH_LOGGING\\n\\nDrop PANTECH_ERR_CRASH_LOGGING and introduce PANTECH_SYS\\ndue to proper build (Solve error/mipi_sony_incell.c).\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/6393887f2296f3e7d7820ceefa0177003261ec68\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:44Z\"}\n{\"id\":\"2489397046\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752778,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"74e68efacee23b35716fa9b2b46d0926e552250b\",\"before\":\"b956bef865580106b571e412e35308b3e7ab708e\",\"commits\":[{\"sha\":\"74e68efacee23b35716fa9b2b46d0926e552250b\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"image scan\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/74e68efacee23b35716fa9b2b46d0926e552250b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:44Z\"}\n{\"id\":\"2489397047\",\"type\":\"PushEvent\",\"actor\":{\"id\":904370,\"login\":\"helhum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/helhum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/904370?\"},\"repo\":{\"id\":20956969,\"name\":\"TYPO3-Surf-CMS/Distribution\",\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Distribution\"},\"payload\":{\"push_id\":536752779,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f566d43f3b6ada009c3f571fac56f51b35110a46\",\"before\":\"241f31bdc19c49919632933ff22e01b4f73220ad\",\"commits\":[{\"sha\":\"f566d43f3b6ada009c3f571fac56f51b35110a46\",\"author\":{\"email\":\"6bf857ca7de026fbed4ae790a809a0ea640901f4@helmuthummel.de\",\"name\":\"Helmut Hummel\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Distribution/commits/f566d43f3b6ada009c3f571fac56f51b35110a46\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:44Z\",\"org\":{\"id\":7921669,\"login\":\"TYPO3-Surf-CMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TYPO3-Surf-CMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7921669?\"}}\n{\"id\":\"2489397049\",\"type\":\"PushEvent\",\"actor\":{\"id\":9000293,\"login\":\"diianita\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/diianita\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9000293?\"},\"repo\":{\"id\":27146993,\"name\":\"cArLiiToX/dtstore\",\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore\"},\"payload\":{\"push_id\":536752780,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8cacad9b34f30388789700e5bb81deaa4471de12\",\"before\":\"57150fe4e8e3dbd8707e8bec5efdcbd306696812\",\"commits\":[{\"sha\":\"8cacad9b34f30388789700e5bb81deaa4471de12\",\"author\":{\"email\":\"ab5e2bca84933118bbc9d48ffaccce3bac4eeb64@xng.bz\",\"name\":\"cArLiiToX\"},\"message\":\"correciones\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore/commits/8cacad9b34f30388789700e5bb81deaa4471de12\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:44Z\"}\n{\"id\":\"2489397053\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536752782,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d3f1564c71b9e115435146f52c4c7c5771f819c7\",\"before\":\"1ac37b74ae21d946dc3826883204811da6d06852\",\"commits\":[{\"sha\":\"d3f1564c71b9e115435146f52c4c7c5771f819c7\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074163829\\n\\nXtUeXIlAxXGaVujlvuXtkkZObuk/+aeIggSf/No53Io=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/d3f1564c71b9e115435146f52c4c7c5771f819c7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:45Z\"}\n{\"id\":\"2489397054\",\"type\":\"PushEvent\",\"actor\":{\"id\":3520402,\"login\":\"GGGGGGGG\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GGGGGGGG\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3520402?\"},\"repo\":{\"id\":18301597,\"name\":\"GGGGGGGG/s2wrapper\",\"url\":\"https://api.github.com/repos/GGGGGGGG/s2wrapper\"},\"payload\":{\"push_id\":536752783,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cdb993c4bc1f95f25298c1676aa2b054c7887156\",\"before\":\"20f7e6744e1589aab24b558e51e81511417673bc\",\"commits\":[{\"sha\":\"cdb993c4bc1f95f25298c1676aa2b054c7887156\",\"author\":{\"email\":\"f2a2448977df35866ec00b7ed7fec9e564d96bb3@fallenwow.net\",\"name\":\"Cedeqien\"},\"message\":\"fix for sudo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/GGGGGGGG/s2wrapper/commits/cdb993c4bc1f95f25298c1676aa2b054c7887156\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:45Z\"}\n{\"id\":\"2489397064\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":706947,\"login\":\"d3athrow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?\"},\"repo\":{\"id\":10441188,\"name\":\"d3athrow/vgstation13\",\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397265\",\"id\":22397265,\"diff_hunk\":\"@@ -476,6 +476,8 @@ var/global/list/RPD_recipes=list(\\n \\t\\treturn 0\\n \\tif(istype(A,/area/shuttle)||istype(A,/turf/space/transit))\\n \\t\\treturn 0\\n+\\tif(istype(A, /obj/structure/lattice))\",\"path\":\"code/game/objects/items/weapons/RPD.dm\",\"position\":4,\"original_position\":4,\"commit_id\":\"a360358fdff0ce9979647f10ddc1a5364d33f229\",\"original_commit_id\":\"a360358fdff0ce9979647f10ddc1a5364d33f229\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"consider checking for lattice on space turf click as well.\",\"created_at\":\"2015-01-01T01:02:46Z\",\"updated_at\":\"2015-01-01T01:02:46Z\",\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2418#discussion_r22397265\",\"pull_request_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397265\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2418#discussion_r22397265\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418\",\"id\":26731379,\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2418\",\"diff_url\":\"https://github.com/d3athrow/vgstation13/pull/2418.diff\",\"patch_url\":\"https://github.com/d3athrow/vgstation13/pull/2418.patch\",\"issue_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2418\",\"number\":2418,\"state\":\"open\",\"locked\":false,\"title\":\"Nightvision Goggles update + RPD on lattices\",\"user\":{\"login\":\"clusterfack\",\"id\":8516830,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8516830?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clusterfack\",\"html_url\":\"https://github.com/clusterfack\",\"followers_url\":\"https://api.github.com/users/clusterfack/followers\",\"following_url\":\"https://api.github.com/users/clusterfack/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clusterfack/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clusterfack/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clusterfack/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clusterfack/orgs\",\"repos_url\":\"https://api.github.com/users/clusterfack/repos\",\"events_url\":\"https://api.github.com/users/clusterfack/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clusterfack/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Night vision goggles work like night vision goggles, they let you see through the dark, they dont let you see through walls, and they illuminate all darkness.\\r\\n\\r\\nRPDs can now build on lattice when you click on them.\",\"created_at\":\"2014-12-31T18:20:37Z\",\"updated_at\":\"2015-01-01T01:02:46Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"77cf984c1f278d4fe6f04974b19d5837b4c3b353\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418/commits\",\"review_comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418/comments\",\"review_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2418/comments\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/a360358fdff0ce9979647f10ddc1a5364d33f229\",\"head\":{\"label\":\"clusterfack:NVG\",\"ref\":\"NVG\",\"sha\":\"a360358fdff0ce9979647f10ddc1a5364d33f229\",\"user\":{\"login\":\"clusterfack\",\"id\":8516830,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8516830?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clusterfack\",\"html_url\":\"https://github.com/clusterfack\",\"followers_url\":\"https://api.github.com/users/clusterfack/followers\",\"following_url\":\"https://api.github.com/users/clusterfack/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clusterfack/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clusterfack/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clusterfack/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clusterfack/orgs\",\"repos_url\":\"https://api.github.com/users/clusterfack/repos\",\"events_url\":\"https://api.github.com/users/clusterfack/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clusterfack/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28336888,\"name\":\"bugfixstation13\",\"full_name\":\"clusterfack/bugfixstation13\",\"owner\":{\"login\":\"clusterfack\",\"id\":8516830,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8516830?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clusterfack\",\"html_url\":\"https://github.com/clusterfack\",\"followers_url\":\"https://api.github.com/users/clusterfack/followers\",\"following_url\":\"https://api.github.com/users/clusterfack/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clusterfack/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clusterfack/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clusterfack/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clusterfack/orgs\",\"repos_url\":\"https://api.github.com/users/clusterfack/repos\",\"events_url\":\"https://api.github.com/users/clusterfack/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clusterfack/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/clusterfack/bugfixstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/clusterfack/bugfixstation13\",\"forks_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/forks\",\"keys_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/events\",\"assignees_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/merges\",\"archive_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/clusterfack/bugfixstation13/releases{/id}\",\"created_at\":\"2014-12-22T11:57:24Z\",\"updated_at\":\"2014-12-31T18:43:55Z\",\"pushed_at\":\"2014-12-31T18:47:03Z\",\"git_url\":\"git://github.com/clusterfack/bugfixstation13.git\",\"ssh_url\":\"git@github.com:clusterfack/bugfixstation13.git\",\"clone_url\":\"https://github.com/clusterfack/bugfixstation13.git\",\"svn_url\":\"https://github.com/clusterfack/bugfixstation13\",\"homepage\":\"\",\"size\":760194,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"Bleeding-Edge\"}},\"base\":{\"label\":\"d3athrow:Bleeding-Edge\",\"ref\":\"Bleeding-Edge\",\"sha\":\"5e8624143efbf348ed553421b76293ef84acac57\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":10441188,\"name\":\"vgstation13\",\"full_name\":\"d3athrow/vgstation13\",\"owner\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/d3athrow/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\",\"forks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/d3athrow/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/d3athrow/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/d3athrow/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/d3athrow/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/d3athrow/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/d3athrow/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/d3athrow/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/d3athrow/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/d3athrow/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/d3athrow/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/d3athrow/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/d3athrow/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/d3athrow/vgstation13/releases{/id}\",\"created_at\":\"2013-06-02T19:39:54Z\",\"updated_at\":\"2014-12-31T20:06:46Z\",\"pushed_at\":\"2014-12-31T23:06:06Z\",\"git_url\":\"git://github.com/d3athrow/vgstation13.git\",\"ssh_url\":\"git@github.com:d3athrow/vgstation13.git\",\"clone_url\":\"https://github.com/d3athrow/vgstation13.git\",\"svn_url\":\"https://github.com/d3athrow/vgstation13\",\"homepage\":\"\",\"size\":937605,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":135,\"mirror_url\":null,\"open_issues_count\":260,\"forks\":135,\"open_issues\":260,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2418\"},\"issue\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2418\"},\"comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2418/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2418/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/a360358fdff0ce9979647f10ddc1a5364d33f229\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:02:46Z\"}\n{\"id\":\"2489397070\",\"type\":\"PushEvent\",\"actor\":{\"id\":2226434,\"login\":\"andschwa\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/andschwa\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2226434?\"},\"repo\":{\"id\":6480901,\"name\":\"andschwa/dotfiles\",\"url\":\"https://api.github.com/repos/andschwa/dotfiles\"},\"payload\":{\"push_id\":536752785,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"6141ae1ad24a045ceb531541f2ee6d242c8d6501\",\"before\":\"677ba270bf789af52a4df9dbc07af6dbca96295b\",\"commits\":[{\"sha\":\"06b1e8dbfecb03fe23ce6b8bf1b13fe4567e1152\",\"author\":{\"email\":\"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@schwartzmeyer.com\",\"name\":\"Andrew Schwartzmeyer\"},\"message\":\"Updating mrconfig\",\"distinct\":true,\"url\":\"https://api.github.com/repos/andschwa/dotfiles/commits/06b1e8dbfecb03fe23ce6b8bf1b13fe4567e1152\"},{\"sha\":\"bdc7c21ecfb664b32305747d3a4c525c33df9d5a\",\"author\":{\"email\":\"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@schwartzmeyer.com\",\"name\":\"Andrew Schwartzmeyer\"},\"message\":\"Fixing mr repo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/andschwa/dotfiles/commits/bdc7c21ecfb664b32305747d3a4c525c33df9d5a\"},{\"sha\":\"6141ae1ad24a045ceb531541f2ee6d242c8d6501\",\"author\":{\"email\":\"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@schwartzmeyer.com\",\"name\":\"Andrew Schwartzmeyer\"},\"message\":\"Don't repeat myself\",\"distinct\":true,\"url\":\"https://api.github.com/repos/andschwa/dotfiles/commits/6141ae1ad24a045ceb531541f2ee6d242c8d6501\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:47Z\"}\n{\"id\":\"2489397071\",\"type\":\"PushEvent\",\"actor\":{\"id\":227068,\"login\":\"radix\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/radix\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/227068?\"},\"repo\":{\"id\":20022094,\"name\":\"radix/effect\",\"url\":\"https://api.github.com/repos/radix/effect\"},\"payload\":{\"push_id\":536752786,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1d1a8441574438befa1731113e960e12b24258c3\",\"before\":\"defdbe78db98ad69d72f42b09194309f47616592\",\"commits\":[{\"sha\":\"1d1a8441574438befa1731113e960e12b24258c3\",\"author\":{\"email\":\"5f33e8ddd36b0c849687df732835b9abbe9b347b@twistedmatrix.com\",\"name\":\"Christopher Armstrong\"},\"message\":\"add the one-line description to the top of the sphinx doc\",\"distinct\":true,\"url\":\"https://api.github.com/repos/radix/effect/commits/1d1a8441574438befa1731113e960e12b24258c3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:47Z\"}\n{\"id\":\"2489397083\",\"type\":\"PushEvent\",\"actor\":{\"id\":906529,\"login\":\"dpwolfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpwolfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/906529?\"},\"repo\":{\"id\":26579311,\"name\":\"dpwolfe/otucha\",\"url\":\"https://api.github.com/repos/dpwolfe/otucha\"},\"payload\":{\"push_id\":536752791,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a9a695f1f7a3395f7394ae668feab71f35a5f622\",\"before\":\"a8f720680aef094515b09c16aff3a59a4d6b2baa\",\"commits\":[{\"sha\":\"a9a695f1f7a3395f7394ae668feab71f35a5f622\",\"author\":{\"email\":\"b7dfe270ecb2603aba704ea15b776485da19da15@gmail.com\",\"name\":\"David Wolfe\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpwolfe/otucha/commits/a9a695f1f7a3395f7394ae668feab71f35a5f622\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:48Z\"}\n{\"id\":\"2489397084\",\"type\":\"ReleaseEvent\",\"actor\":{\"id\":99359,\"login\":\"llinder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?\"},\"repo\":{\"id\":28669941,\"name\":\"llinder/salt\",\"url\":\"https://api.github.com/repos/llinder/salt\"},\"payload\":{\"action\":\"published\",\"release\":{\"url\":\"https://api.github.com/repos/llinder/salt/releases/818219\",\"assets_url\":\"https://api.github.com/repos/llinder/salt/releases/818219/assets\",\"upload_url\":\"https://uploads.github.com/repos/llinder/salt/releases/818219/assets{?name}\",\"html_url\":\"https://github.com/llinder/salt/releases/tag/v2014.7.0_1\",\"id\":818219,\"tag_name\":\"v2014.7.0_1\",\"target_commitish\":\"2014.7\",\"name\":\"\",\"draft\":false,\"author\":{\"login\":\"llinder\",\"id\":99359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"html_url\":\"https://github.com/llinder\",\"followers_url\":\"https://api.github.com/users/llinder/followers\",\"following_url\":\"https://api.github.com/users/llinder/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/llinder/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/llinder/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/llinder/subscriptions\",\"organizations_url\":\"https://api.github.com/users/llinder/orgs\",\"repos_url\":\"https://api.github.com/users/llinder/repos\",\"events_url\":\"https://api.github.com/users/llinder/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/llinder/received_events\",\"type\":\"User\",\"site_admin\":false},\"prerelease\":false,\"created_at\":\"2014-12-31T17:03:50Z\",\"published_at\":\"2015-01-01T01:02:48Z\",\"assets\":[],\"tarball_url\":\"https://api.github.com/repos/llinder/salt/tarball/v2014.7.0_1\",\"zipball_url\":\"https://api.github.com/repos/llinder/salt/zipball/v2014.7.0_1\",\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:02:48Z\"}\n{\"id\":\"2489397087\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":5573038,\"login\":\"CorruptComputer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?\"},\"repo\":{\"id\":28371640,\"name\":\"AsteroidStation/-tg-station\",\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station\"},\"payload\":{\"action\":\"closed\",\"number\":8,\"pull_request\":{\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8\",\"id\":26739422,\"html_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8\",\"diff_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8.diff\",\"patch_url\":\"https://github.com/AsteroidStation/-tg-station/pull/8.patch\",\"issue_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8\",\"number\":8,\"state\":\"closed\",\"locked\":false,\"title\":\"tg update\",\"user\":{\"login\":\"CorruptComputer\",\"id\":5573038,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"html_url\":\"https://github.com/CorruptComputer\",\"followers_url\":\"https://api.github.com/users/CorruptComputer/followers\",\"following_url\":\"https://api.github.com/users/CorruptComputer/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CorruptComputer/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CorruptComputer/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CorruptComputer/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CorruptComputer/orgs\",\"repos_url\":\"https://api.github.com/users/CorruptComputer/repos\",\"events_url\":\"https://api.github.com/users/CorruptComputer/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CorruptComputer/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:02:41Z\",\"updated_at\":\"2015-01-01T01:02:48Z\",\"closed_at\":\"2015-01-01T01:02:48Z\",\"merged_at\":\"2015-01-01T01:02:48Z\",\"merge_commit_sha\":\"9a329cfe57a587090aecaa6410a370ee062046a9\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/commits\",\"review_comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/comments\",\"review_comment_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8/comments\",\"statuses_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/34c4c027770518e2087db44a89cad07cd91ceb84\",\"head\":{\"label\":\"tgstation:master\",\"ref\":\"master\",\"sha\":\"34c4c027770518e2087db44a89cad07cd91ceb84\",\"user\":{\"login\":\"tgstation\",\"id\":1363778,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1363778?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tgstation\",\"html_url\":\"https://github.com/tgstation\",\"followers_url\":\"https://api.github.com/users/tgstation/followers\",\"following_url\":\"https://api.github.com/users/tgstation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tgstation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tgstation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tgstation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tgstation/orgs\",\"repos_url\":\"https://api.github.com/users/tgstation/repos\",\"events_url\":\"https://api.github.com/users/tgstation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tgstation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":3234987,\"name\":\"-tg-station\",\"full_name\":\"tgstation/-tg-station\",\"owner\":{\"login\":\"tgstation\",\"id\":1363778,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1363778?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tgstation\",\"html_url\":\"https://github.com/tgstation\",\"followers_url\":\"https://api.github.com/users/tgstation/followers\",\"following_url\":\"https://api.github.com/users/tgstation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tgstation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tgstation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tgstation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tgstation/orgs\",\"repos_url\":\"https://api.github.com/users/tgstation/repos\",\"events_url\":\"https://api.github.com/users/tgstation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tgstation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/tgstation/-tg-station\",\"description\":\"/tg/'s SS13 branch\",\"fork\":false,\"url\":\"https://api.github.com/repos/tgstation/-tg-station\",\"forks_url\":\"https://api.github.com/repos/tgstation/-tg-station/forks\",\"keys_url\":\"https://api.github.com/repos/tgstation/-tg-station/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/tgstation/-tg-station/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/tgstation/-tg-station/teams\",\"hooks_url\":\"https://api.github.com/repos/tgstation/-tg-station/hooks\",\"issue_events_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/tgstation/-tg-station/events\",\"assignees_url\":\"https://api.github.com/repos/tgstation/-tg-station/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/tgstation/-tg-station/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/tgstation/-tg-station/tags\",\"blobs_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/tgstation/-tg-station/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/tgstation/-tg-station/languages\",\"stargazers_url\":\"https://api.github.com/repos/tgstation/-tg-station/stargazers\",\"contributors_url\":\"https://api.github.com/repos/tgstation/-tg-station/contributors\",\"subscribers_url\":\"https://api.github.com/repos/tgstation/-tg-station/subscribers\",\"subscription_url\":\"https://api.github.com/repos/tgstation/-tg-station/subscription\",\"commits_url\":\"https://api.github.com/repos/tgstation/-tg-station/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/tgstation/-tg-station/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/tgstation/-tg-station/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/tgstation/-tg-station/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/tgstation/-tg-station/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/tgstation/-tg-station/merges\",\"archive_url\":\"https://api.github.com/repos/tgstation/-tg-station/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/tgstation/-tg-station/downloads\",\"issues_url\":\"https://api.github.com/repos/tgstation/-tg-station/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/tgstation/-tg-station/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/tgstation/-tg-station/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/tgstation/-tg-station/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/tgstation/-tg-station/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/tgstation/-tg-station/releases{/id}\",\"created_at\":\"2012-01-21T17:32:47Z\",\"updated_at\":\"2015-01-01T00:47:49Z\",\"pushed_at\":\"2015-01-01T00:47:48Z\",\"git_url\":\"git://github.com/tgstation/-tg-station.git\",\"ssh_url\":\"git@github.com:tgstation/-tg-station.git\",\"clone_url\":\"https://github.com/tgstation/-tg-station.git\",\"svn_url\":\"https://github.com/tgstation/-tg-station\",\"homepage\":\"http://www.tgstation13.org/\",\"size\":550715,\"stargazers_count\":172,\"watchers_count\":172,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":609,\"mirror_url\":null,\"open_issues_count\":618,\"forks\":609,\"open_issues\":618,\"watchers\":172,\"default_branch\":\"master\"}},\"base\":{\"label\":\"AsteroidStation:master\",\"ref\":\"master\",\"sha\":\"d98abaabaa971c9092a200961915c9e60015ab20\",\"user\":{\"login\":\"AsteroidStation\",\"id\":9313156,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AsteroidStation\",\"html_url\":\"https://github.com/AsteroidStation\",\"followers_url\":\"https://api.github.com/users/AsteroidStation/followers\",\"following_url\":\"https://api.github.com/users/AsteroidStation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AsteroidStation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AsteroidStation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AsteroidStation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AsteroidStation/orgs\",\"repos_url\":\"https://api.github.com/users/AsteroidStation/repos\",\"events_url\":\"https://api.github.com/users/AsteroidStation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AsteroidStation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":28371640,\"name\":\"-tg-station\",\"full_name\":\"AsteroidStation/-tg-station\",\"owner\":{\"login\":\"AsteroidStation\",\"id\":9313156,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AsteroidStation\",\"html_url\":\"https://github.com/AsteroidStation\",\"followers_url\":\"https://api.github.com/users/AsteroidStation/followers\",\"following_url\":\"https://api.github.com/users/AsteroidStation/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AsteroidStation/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AsteroidStation/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AsteroidStation/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AsteroidStation/orgs\",\"repos_url\":\"https://api.github.com/users/AsteroidStation/repos\",\"events_url\":\"https://api.github.com/users/AsteroidStation/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AsteroidStation/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/AsteroidStation/-tg-station\",\"description\":\"/tg/'s SS13 branch\",\"fork\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station\",\"forks_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/forks\",\"keys_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/teams\",\"hooks_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/hooks\",\"issue_events_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/events\",\"assignees_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/tags\",\"blobs_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/languages\",\"stargazers_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/stargazers\",\"contributors_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/contributors\",\"subscribers_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/subscribers\",\"subscription_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/subscription\",\"commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/merges\",\"archive_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/downloads\",\"issues_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/releases{/id}\",\"created_at\":\"2014-12-23T02:11:11Z\",\"updated_at\":\"2014-12-31T02:35:57Z\",\"pushed_at\":\"2015-01-01T01:02:48Z\",\"git_url\":\"git://github.com/AsteroidStation/-tg-station.git\",\"ssh_url\":\"git@github.com:AsteroidStation/-tg-station.git\",\"clone_url\":\"https://github.com/AsteroidStation/-tg-station.git\",\"svn_url\":\"https://github.com/AsteroidStation/-tg-station\",\"homepage\":\"http://www.tgstation13.org/\",\"size\":414377,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8\"},\"html\":{\"href\":\"https://github.com/AsteroidStation/-tg-station/pull/8\"},\"issue\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8\"},\"comments\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/issues/8/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/pulls/8/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/AsteroidStation/-tg-station/statuses/34c4c027770518e2087db44a89cad07cd91ceb84\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"CorruptComputer\",\"id\":5573038,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"html_url\":\"https://github.com/CorruptComputer\",\"followers_url\":\"https://api.github.com/users/CorruptComputer/followers\",\"following_url\":\"https://api.github.com/users/CorruptComputer/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/CorruptComputer/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/CorruptComputer/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/CorruptComputer/subscriptions\",\"organizations_url\":\"https://api.github.com/users/CorruptComputer/orgs\",\"repos_url\":\"https://api.github.com/users/CorruptComputer/repos\",\"events_url\":\"https://api.github.com/users/CorruptComputer/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/CorruptComputer/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":55,\"additions\":1841,\"deletions\":1195,\"changed_files\":145}},\"public\":true,\"created_at\":\"2015-01-01T01:02:48Z\",\"org\":{\"id\":9313156,\"login\":\"AsteroidStation\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/AsteroidStation\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?\"}}\n{\"id\":\"2489397088\",\"type\":\"CreateEvent\",\"actor\":{\"id\":99359,\"login\":\"llinder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?\"},\"repo\":{\"id\":28669941,\"name\":\"llinder/salt\",\"url\":\"https://api.github.com/repos/llinder/salt\"},\"payload\":{\"ref\":\"v2014.7.0_1\",\"ref_type\":\"tag\",\"master_branch\":\"develop\",\"description\":\"Infrastructure automation and management system\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:48Z\"}\n{\"id\":\"2489397095\",\"type\":\"PushEvent\",\"actor\":{\"id\":5573038,\"login\":\"CorruptComputer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptComputer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5573038?\"},\"repo\":{\"id\":28371640,\"name\":\"AsteroidStation/-tg-station\",\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station\"},\"payload\":{\"push_id\":536752794,\"size\":56,\"distinct_size\":56,\"ref\":\"refs/heads/master\",\"head\":\"4352171f2ac3e6930e2b4b0b5c384779b2908eae\",\"before\":\"d98abaabaa971c9092a200961915c9e60015ab20\",\"commits\":[{\"sha\":\"0921aa201f3ee3894dfad2d400023c39fe1d1e01\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Added datum mutations\\n\\nAlso removed some mob vars like sdisabilities and merged the usability\\nwith disabilities\\nRemoved need for mutations var, they are not handled in dna\\nRemoved blinded var, now its handled by eye_blind being bigger than zero\\nAnds lots, lots of other shit in files that used mutations\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/0921aa201f3ee3894dfad2d400023c39fe1d1e01\"},{\"sha\":\"11f6c7c40148a6c8be2abc5ebb7cc9b967a20950\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Merge branch 'master' of https://github.com/tgstation/-tg-station into GenShit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/11f6c7c40148a6c8be2abc5ebb7cc9b967a20950\"},{\"sha\":\"8d110ee49639b4d3163529868339ea993330b2cf\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Tiny clean up, still lots of work to do\\n\\nOnly compiletested, these changes were never tested in the game yet\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/8d110ee49639b4d3163529868339ea993330b2cf\"},{\"sha\":\"861072596873d47ea851480df74854a78a28f948\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Monkifying change and some mutations fixes\\n\\nIt approaches playable level of working, hurray!\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/861072596873d47ea851480df74854a78a28f948\"},{\"sha\":\"44cc6c6d97543b63d277f1d6eb980d56d90b4d1e\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Merge branch 'master' of https://github.com/tgstation/-tg-station into GenShit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/44cc6c6d97543b63d277f1d6eb980d56d90b4d1e\"},{\"sha\":\"21a2c2d21eccd62394113a0ac6395397aa9d80a8\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Made clumsiness into disability\\n\\nI really dont want to make clumsy_act()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/21a2c2d21eccd62394113a0ac6395397aa9d80a8\"},{\"sha\":\"570783d2c776ad6b2106440a1afbab38af6f8c88\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Added comments for RR for future\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/570783d2c776ad6b2106440a1afbab38af6f8c88\"},{\"sha\":\"f0c8f2eeacc3e59d9c9f0942d7bfad5daabcacff\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Merge branch 'master' of https://github.com/tgstation/-tg-station into GenShit\\n\\nConflicts:\\n\\tcode/modules/reagents/Chemistry-Reagents.dm\\n\\nconflicts resolution\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/f0c8f2eeacc3e59d9c9f0942d7bfad5daabcacff\"},{\"sha\":\"b12472e0bb1845d312386e96f953b4ae9ab07de2\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Made shit compilable, probably broke new hunger\\n\\nWhoever made it: tough luck bro\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/b12472e0bb1845d312386e96f953b4ae9ab07de2\"},{\"sha\":\"9a27fe7c8dc9fe426723fd0f8c97160a51037dd6\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Conflict fix\\n\\nYeah makes it all work\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/9a27fe7c8dc9fe426723fd0f8c97160a51037dd6\"},{\"sha\":\"57b41dce0d954580aa4531310ac1f0a2ee255814\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Fixed some shiet\\n\\nRedone how injectors handle mutations(now it doesnt copy the SE but\\nmanipulates mutations directly)\\nRemoved sole leftovers\\nFixed humanizing\\nFixed grammer\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/57b41dce0d954580aa4531310ac1f0a2ee255814\"},{\"sha\":\"92fc9b5d1b4a91317143e1db1a6c22af59f1f6a3\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"And some more little fixes\\n\\nReturn of the prob to foam smashing\\nCorrect laserpointer check\\nCorrect wield behaviour with hulk\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/92fc9b5d1b4a91317143e1db1a6c22af59f1f6a3\"},{\"sha\":\"1b45752b05bbc5cd285d6432d22510cbba86840d\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Fixed conflicts\\n\\nYe again and again and again and again\\nDo it again, do it again~\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/1b45752b05bbc5cd285d6432d22510cbba86840d\"},{\"sha\":\"208b59c30ef5a597545a49a46fe3983dde1f01f6\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Fixed stuff tkdrg pointed out\\n\\nAdded comment to dualsaber and hulk checks to explain whats going on\\nAdded defines that converts to mutation names\\nFixed grammer again\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/208b59c30ef5a597545a49a46fe3983dde1f01f6\"},{\"sha\":\"76a21883a8d70c71d30b9acb3162cc21e80b2ca9\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Merge branch 'master' of https://github.com/tgstation/-tg-station into GenShit\\n\\nConflicts:\\n\\tcode/game/mecha/mecha.dm\\n\\tcode/game/objects/structures/tables_racks.dm\\n\\tcode/modules/mob/living/silicon/silicon.dm\\n\\tcode/modules/projectiles/gun.dm\\n\\nFixes dem conflicts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/76a21883a8d70c71d30b9acb3162cc21e80b2ca9\"},{\"sha\":\"4c3f83d085ccae384a33db2498de970cf0918df5\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"This shall fix the hunger system\\n\\nI really really hope it does\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/4c3f83d085ccae384a33db2498de970cf0918df5\"},{\"sha\":\"418e3a061364d97a01321d023b9a418881896984\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"General cleanup\\n\\nFixed the magic text failing in monkey injectors\\nRemoved lots of comments\\nDidnt fix mysterious list bounds runtime in tourettes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/418e3a061364d97a01321d023b9a418881896984\"},{\"sha\":\"10b21b20ab801fbc5ce469964c322fc2ccca0138\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Conflicts resolution\\n\\nIn items weapons melee misc\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/10b21b20ab801fbc5ce469964c322fc2ccca0138\"},{\"sha\":\"501a120cca348b88447f1d90b25fbf888fdababa\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Fixes the defines\\n\\nNow all the shit uses them i hope\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/501a120cca348b88447f1d90b25fbf888fdababa\"},{\"sha\":\"37a7fa24146b27dcc10ea6acdad19a7aa67f3fb1\",\"author\":{\"email\":\"cf55a9c90d41f12e3e2216800693d3a930e4551a@gmail.com\",\"name\":\"Razharas\"},\"message\":\"Fixes that one runtime with tourettes\\n\\nAnimation proc is magic, dont ask dont tell\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AsteroidStation/-tg-station/commits/37a7fa24146b27dcc10ea6acdad19a7aa67f3fb1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:49Z\",\"org\":{\"id\":9313156,\"login\":\"AsteroidStation\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/AsteroidStation\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9313156?\"}}\n{\"id\":\"2489397109\",\"type\":\"WatchEvent\",\"actor\":{\"id\":438131,\"login\":\"linuxlizard\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/linuxlizard\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/438131?\"},\"repo\":{\"id\":952838,\"name\":\"adafruit/Adafruit_SSD1306\",\"url\":\"https://api.github.com/repos/adafruit/Adafruit_SSD1306\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\",\"org\":{\"id\":181069,\"login\":\"adafruit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/adafruit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/181069?\"}}\n{\"id\":\"2489397111\",\"type\":\"WatchEvent\",\"actor\":{\"id\":8055446,\"login\":\"mchenla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mchenla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8055446?\"},\"repo\":{\"id\":3816248,\"name\":\"basecamp/bcx-api\",\"url\":\"https://api.github.com/repos/basecamp/bcx-api\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\",\"org\":{\"id\":13131,\"login\":\"basecamp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/basecamp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/13131?\"}}\n{\"id\":\"2489397117\",\"type\":\"PushEvent\",\"actor\":{\"id\":1059214,\"login\":\"wlaurance\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wlaurance\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1059214?\"},\"repo\":{\"id\":28050478,\"name\":\"empirical-org/Quill-Grammar\",\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar\"},\"payload\":{\"push_id\":536752803,\"size\":1,\"distinct_size\":0,\"ref\":\"refs/heads/develop\",\"head\":\"9835f267914e973c8ca53b28b85b94bdf3086ba2\",\"before\":\"166d757dfcef4f5a7f9642883745a949444a131d\",\"commits\":[{\"sha\":\"9835f267914e973c8ca53b28b85b94bdf3086ba2\",\"author\":{\"email\":\"c55061f2e98089f7f71676646e4a1dbbd0f0ebe8@gmail.com\",\"name\":\"Peter Gault\"},\"message\":\"Update README.md\",\"distinct\":false,\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar/commits/9835f267914e973c8ca53b28b85b94bdf3086ba2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\",\"org\":{\"id\":4258432,\"login\":\"empirical-org\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/empirical-org\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4258432?\"}}\n{\"id\":\"2489397118\",\"type\":\"PushEvent\",\"actor\":{\"id\":2851221,\"login\":\"alkass\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alkass\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2851221?\"},\"repo\":{\"id\":28678233,\"name\":\"alkass/seQre\",\"url\":\"https://api.github.com/repos/alkass/seQre\"},\"payload\":{\"push_id\":536752805,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3e46496fed34162743f34c34d45cca60df88ba6f\",\"before\":\"d566054d43b3ecce423e2e8e585b3881467437d7\",\"commits\":[{\"sha\":\"3e46496fed34162743f34c34d45cca60df88ba6f\",\"author\":{\"email\":\"adc72828a6d71e3d0c694f02fcfcce48ee8a532b@users.noreply.github.com\",\"name\":\"Fadi Hanna Al-Kass\"},\"message\":\"Delete README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alkass/seQre/commits/3e46496fed34162743f34c34d45cca60df88ba6f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\"}\n{\"id\":\"2489397119\",\"type\":\"PushEvent\",\"actor\":{\"id\":3656079,\"login\":\"marklrh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marklrh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3656079?\"},\"repo\":{\"id\":27470715,\"name\":\"marklrh/ocaml-cohttp-test\",\"url\":\"https://api.github.com/repos/marklrh/ocaml-cohttp-test\"},\"payload\":{\"push_id\":536752804,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7525c74ee31f49f576408a7d50d1cf8b9d3d1441\",\"before\":\"aa8ec0de017c8003758776739facc819e33ac7c9\",\"commits\":[{\"sha\":\"7525c74ee31f49f576408a7d50d1cf8b9d3d1441\",\"author\":{\"email\":\"e0e04a2320844b42511db0376599e166ab5bda54@gmail.com\",\"name\":\"Runhang Li\"},\"message\":\"lint README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/marklrh/ocaml-cohttp-test/commits/7525c74ee31f49f576408a7d50d1cf8b9d3d1441\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\"}\n{\"id\":\"2489397120\",\"type\":\"PushEvent\",\"actor\":{\"id\":2453862,\"login\":\"schloo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/schloo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2453862?\"},\"repo\":{\"id\":28160579,\"name\":\"azilnik/phetch\",\"url\":\"https://api.github.com/repos/azilnik/phetch\"},\"payload\":{\"push_id\":536752806,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"47f6ee4ea9a5e9d67f20809dd3876760eaf13680\",\"before\":\"ab393a029b968e4d41ade0a85c86f879e171c805\",\"commits\":[{\"sha\":\"47f6ee4ea9a5e9d67f20809dd3876760eaf13680\",\"author\":{\"email\":\"03cd939e5e01f81bd3cbeb1977c82e3d0109cf43@Michelles-Air.home\",\"name\":\"schloo\"},\"message\":\"copy and misc tweaks\",\"distinct\":true,\"url\":\"https://api.github.com/repos/azilnik/phetch/commits/47f6ee4ea9a5e9d67f20809dd3876760eaf13680\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:52Z\"}\n{\"id\":\"2489397128\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1341245,\"login\":\"asfgit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/asfgit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1341245?\"},\"repo\":{\"id\":26798421,\"name\":\"apache/infrastructure-puppet\",\"url\":\"https://api.github.com/repos/apache/infrastructure-puppet\"},\"payload\":{\"ref\":\"puppet-test-work\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Mirror of Apache Infrastructure Puppet\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:53Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489397137\",\"type\":\"PushEvent\",\"actor\":{\"id\":965430,\"login\":\"waltzofpearls\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/waltzofpearls\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/965430?\"},\"repo\":{\"id\":28505561,\"name\":\"waltzofpearls/dotfiles\",\"url\":\"https://api.github.com/repos/waltzofpearls/dotfiles\"},\"payload\":{\"push_id\":536752816,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d5a1f77ab903cdcf1d2ae292c7f0f259c6e32211\",\"before\":\"7e0e2a2759ddafd07d02d2167359a84c7cf15194\",\"commits\":[{\"sha\":\"d5a1f77ab903cdcf1d2ae292c7f0f259c6e32211\",\"author\":{\"email\":\"c514db49330801e4e831feeacd2b70f6f55a5048@gmail.com\",\"name\":\"Rollie Ma\"},\"message\":\"Some minor tweaks to dotfiles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/waltzofpearls/dotfiles/commits/d5a1f77ab903cdcf1d2ae292c7f0f259c6e32211\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:53Z\"}\n{\"id\":\"2489397138\",\"type\":\"PushEvent\",\"actor\":{\"id\":171043,\"login\":\"jeffnv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jeffnv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/171043?\"},\"repo\":{\"id\":28159277,\"name\":\"jeffnv/elite-golf\",\"url\":\"https://api.github.com/repos/jeffnv/elite-golf\"},\"payload\":{\"push_id\":536752817,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"12162bb9f224c179a9ea7d8abbac7dadec15ee2a\",\"before\":\"a88fa3bfffc1d1f4ce9b668412981ff071d9920e\",\"commits\":[{\"sha\":\"12162bb9f224c179a9ea7d8abbac7dadec15ee2a\",\"author\":{\"email\":\"a4a950aede9822deccc73582f88e82e913eb89d5@gmail.com\",\"name\":\"Jeff Fiddler\"},\"message\":\"more useful alert when clicking idle creator map\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jeffnv/elite-golf/commits/12162bb9f224c179a9ea7d8abbac7dadec15ee2a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:53Z\"}\n{\"id\":\"2489397150\",\"type\":\"PushEvent\",\"actor\":{\"id\":904370,\"login\":\"helhum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/helhum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/904370?\"},\"repo\":{\"id\":20956969,\"name\":\"TYPO3-Surf-CMS/Distribution\",\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Distribution\"},\"payload\":{\"push_id\":536752822,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e585c369f862ad016cdc8ac608adce8e1d2a348c\",\"before\":\"f566d43f3b6ada009c3f571fac56f51b35110a46\",\"commits\":[{\"sha\":\"e585c369f862ad016cdc8ac608adce8e1d2a348c\",\"author\":{\"email\":\"6bf857ca7de026fbed4ae790a809a0ea640901f4@helmuthummel.de\",\"name\":\"Helmut Hummel\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Distribution/commits/e585c369f862ad016cdc8ac608adce8e1d2a348c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:57Z\",\"org\":{\"id\":7921669,\"login\":\"TYPO3-Surf-CMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TYPO3-Surf-CMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7921669?\"}}\n{\"id\":\"2489397154\",\"type\":\"PushEvent\",\"actor\":{\"id\":5623501,\"login\":\"xbony2\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/xbony2\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5623501?\"},\"repo\":{\"id\":28162672,\"name\":\"xbony2/Experimental-Self-Aware-Electronic-Based-Space-Analyzing-Droid\",\"url\":\"https://api.github.com/repos/xbony2/Experimental-Self-Aware-Electronic-Based-Space-Analyzing-Droid\"},\"payload\":{\"push_id\":536752826,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"f151107a11fa3dbf47a1fe26ec221e49daeca203\",\"before\":\"ebbaebe4084a57cc1553d710f12013d0dba10bd7\",\"commits\":[{\"sha\":\"7fa4f074eeabe4fa22ef9042e260f81dbceda991\",\"author\":{\"email\":\"8371f52126eb90da97b7600da3d2d08e783fe9cf@gmail.com\",\"name\":\"xbony2\"},\"message\":\"nother channel\",\"distinct\":true,\"url\":\"https://api.github.com/repos/xbony2/Experimental-Self-Aware-Electronic-Based-Space-Analyzing-Droid/commits/7fa4f074eeabe4fa22ef9042e260f81dbceda991\"},{\"sha\":\"f151107a11fa3dbf47a1fe26ec221e49daeca203\",\"author\":{\"email\":\"8371f52126eb90da97b7600da3d2d08e783fe9cf@gmail.com\",\"name\":\"xbony2\"},\"message\":\"Merge branch 'master' of https://github.com/xbony2/Experimental-Self-Aware-Electronic-Based-Space-Analyzing-Droid.git\",\"distinct\":true,\"url\":\"https://api.github.com/repos/xbony2/Experimental-Self-Aware-Electronic-Based-Space-Analyzing-Droid/commits/f151107a11fa3dbf47a1fe26ec221e49daeca203\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:57Z\"}\n{\"id\":\"2489397164\",\"type\":\"PushEvent\",\"actor\":{\"id\":1017605,\"login\":\"wangshan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wangshan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1017605?\"},\"repo\":{\"id\":28666633,\"name\":\"wangshan/wangshan.github.io\",\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io\"},\"payload\":{\"push_id\":536752830,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5a94fdd12e5d88be38cca22321e943d374d3e4d5\",\"before\":\"92d20ac01b4767280cccaf71ee20f173529877a0\",\"commits\":[{\"sha\":\"5a94fdd12e5d88be38cca22321e943d374d3e4d5\",\"author\":{\"email\":\"e3e97680eb29c788f35181af31eb442b3251e18f@gmail.com\",\"name\":\"Shan\"},\"message\":\"Update 2012-03-03-mac-development-environment-setup.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io/commits/5a94fdd12e5d88be38cca22321e943d374d3e4d5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:02:58Z\"}\n{\"id\":\"2489397169\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1902323,\"login\":\"zoontek\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zoontek\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1902323?\"},\"repo\":{\"id\":11393110,\"name\":\"codegangsta/cli\",\"url\":\"https://api.github.com/repos/codegangsta/cli\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:02:59Z\"}\n{\"id\":\"2489397179\",\"type\":\"PushEvent\",\"actor\":{\"id\":2453862,\"login\":\"schloo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/schloo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2453862?\"},\"repo\":{\"id\":28160579,\"name\":\"azilnik/phetch\",\"url\":\"https://api.github.com/repos/azilnik/phetch\"},\"payload\":{\"push_id\":536752838,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"1b32d9c513558f8cc57ed88edff23b0d45610363\",\"before\":\"47f6ee4ea9a5e9d67f20809dd3876760eaf13680\",\"commits\":[{\"sha\":\"1b32d9c513558f8cc57ed88edff23b0d45610363\",\"author\":{\"email\":\"03cd939e5e01f81bd3cbeb1977c82e3d0109cf43@Michelles-Air.home\",\"name\":\"schloo\"},\"message\":\"revised images\",\"distinct\":true,\"url\":\"https://api.github.com/repos/azilnik/phetch/commits/1b32d9c513558f8cc57ed88edff23b0d45610363\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:01Z\"}\n{\"id\":\"2489397180\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":6737270,\"login\":\"JorgeX\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?\"},\"repo\":{\"id\":25890220,\"name\":\"JorgeX/dojo_rules\",\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\"},\"payload\":{\"action\":\"opened\",\"number\":1,\"pull_request\":{\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\",\"id\":26739426,\"html_url\":\"https://github.com/JorgeX/dojo_rules/pull/1\",\"diff_url\":\"https://github.com/JorgeX/dojo_rules/pull/1.diff\",\"patch_url\":\"https://github.com/JorgeX/dojo_rules/pull/1.patch\",\"issue_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1\",\"number\":1,\"state\":\"open\",\"locked\":false,\"title\":\"Deadly skills\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"ok\",\"created_at\":\"2015-01-01T01:03:01Z\",\"updated_at\":\"2015-01-01T01:03:01Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/commits\",\"review_comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/comments\",\"review_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1/comments\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"head\":{\"label\":\"JorgeX:deadly_skills\",\"ref\":\"deadly_skills\",\"sha\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":25890220,\"name\":\"dojo_rules\",\"full_name\":\"JorgeX/dojo_rules\",\"owner\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/JorgeX/dojo_rules\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\",\"forks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/forks\",\"keys_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/teams\",\"hooks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/hooks\",\"issue_events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/events\",\"assignees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/tags\",\"blobs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/languages\",\"stargazers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/stargazers\",\"contributors_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contributors\",\"subscribers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscribers\",\"subscription_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscription\",\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/merges\",\"archive_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/downloads\",\"issues_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/releases{/id}\",\"created_at\":\"2014-10-28T21:09:20Z\",\"updated_at\":\"2014-10-28T21:13:40Z\",\"pushed_at\":\"2015-01-01T01:00:43Z\",\"git_url\":\"git://github.com/JorgeX/dojo_rules.git\",\"ssh_url\":\"git@github.com:JorgeX/dojo_rules.git\",\"clone_url\":\"https://github.com/JorgeX/dojo_rules.git\",\"svn_url\":\"https://github.com/JorgeX/dojo_rules\",\"homepage\":null,\"size\":104,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"JorgeX:master\",\"ref\":\"master\",\"sha\":\"d1b0081019d3c0e43266beaa87fee523fc684649\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":25890220,\"name\":\"dojo_rules\",\"full_name\":\"JorgeX/dojo_rules\",\"owner\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/JorgeX/dojo_rules\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\",\"forks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/forks\",\"keys_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/teams\",\"hooks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/hooks\",\"issue_events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/events\",\"assignees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/tags\",\"blobs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/languages\",\"stargazers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/stargazers\",\"contributors_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contributors\",\"subscribers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscribers\",\"subscription_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscription\",\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/merges\",\"archive_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/downloads\",\"issues_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/releases{/id}\",\"created_at\":\"2014-10-28T21:09:20Z\",\"updated_at\":\"2014-10-28T21:13:40Z\",\"pushed_at\":\"2015-01-01T01:00:43Z\",\"git_url\":\"git://github.com/JorgeX/dojo_rules.git\",\"ssh_url\":\"git@github.com:JorgeX/dojo_rules.git\",\"clone_url\":\"https://github.com/JorgeX/dojo_rules.git\",\"svn_url\":\"https://github.com/JorgeX/dojo_rules\",\"homepage\":null,\"size\":104,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\"},\"html\":{\"href\":\"https://github.com/JorgeX/dojo_rules/pull/1\"},\"issue\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1\"},\"comments\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":4,\"additions\":6,\"deletions\":8,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:03:01Z\"}\n{\"id\":\"2489397186\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":19792,\"login\":\"wiredfool\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?\"},\"repo\":{\"id\":5171600,\"name\":\"python-pillow/Pillow\",\"url\":\"https://api.github.com/repos/python-pillow/Pillow\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/949\",\"labels_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/949/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/949/comments\",\"events_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/949/events\",\"html_url\":\"https://github.com/python-pillow/Pillow/issues/949\",\"id\":45165342,\"number\":949,\"title\":\"Release 2.7.0 on January 1, 2015\",\"user\":{\"login\":\"aclark4life\",\"id\":72164,\"avatar_url\":\"https://avatars.githubusercontent.com/u/72164?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aclark4life\",\"html_url\":\"https://github.com/aclark4life\",\"followers_url\":\"https://api.github.com/users/aclark4life/followers\",\"following_url\":\"https://api.github.com/users/aclark4life/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/aclark4life/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/aclark4life/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/aclark4life/subscriptions\",\"organizations_url\":\"https://api.github.com/users/aclark4life/orgs\",\"repos_url\":\"https://api.github.com/users/aclark4life/repos\",\"events_url\":\"https://api.github.com/users/aclark4life/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/aclark4life/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/labels/Release\",\"name\":\"Release\",\"color\":\"bfdadc\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/milestones/8\",\"labels_url\":\"https://api.github.com/repos/python-pillow/Pillow/milestones/8/labels\",\"id\":840816,\"number\":8,\"title\":\"2.7.0\",\"description\":\"\",\"creator\":{\"login\":\"hugovk\",\"id\":1324225,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1324225?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hugovk\",\"html_url\":\"https://github.com/hugovk\",\"followers_url\":\"https://api.github.com/users/hugovk/followers\",\"following_url\":\"https://api.github.com/users/hugovk/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/hugovk/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/hugovk/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/hugovk/subscriptions\",\"organizations_url\":\"https://api.github.com/users/hugovk/orgs\",\"repos_url\":\"https://api.github.com/users/hugovk/repos\",\"events_url\":\"https://api.github.com/users/hugovk/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/hugovk/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":7,\"closed_issues\":0,\"state\":\"open\",\"created_at\":\"2014-10-26T09:53:06Z\",\"updated_at\":\"2014-10-26T09:55:42Z\",\"due_on\":\"2015-01-01T08:00:00Z\",\"closed_at\":null},\"comments\":11,\"created_at\":\"2014-10-07T20:49:47Z\",\"updated_at\":\"2015-01-01T01:03:02Z\",\"closed_at\":null,\"body\":\"OK I'm dropping off again, please use @aclark4life if you need anything or email aclark@aclark.net. Thank you :beers:\"},\"comment\":{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/comments/68477275\",\"html_url\":\"https://github.com/python-pillow/Pillow/issues/949#issuecomment-68477275\",\"issue_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/949\",\"id\":68477275,\"user\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:02Z\",\"updated_at\":\"2015-01-01T01:03:02Z\",\"body\":\"The scipy change appears to be because of a change in the way that image resizing works. The new version looks more reasonable to me, which fits with the conclusions that we've had from the changes in the resizing code this iteration. \"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:02Z\",\"org\":{\"id\":2036701,\"login\":\"python-pillow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python-pillow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?\"}}\n{\"id\":\"2489397187\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":100206,\"login\":\"quicksketch\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/quicksketch\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/100206?\"},\"repo\":{\"id\":12582950,\"name\":\"backdrop/backdrop-issues\",\"url\":\"https://api.github.com/repos/backdrop/backdrop-issues\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/521\",\"labels_url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/521/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/521/comments\",\"events_url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/521/events\",\"html_url\":\"https://github.com/backdrop/backdrop-issues/issues/521\",\"id\":53210114,\"number\":521,\"title\":\"[UX] The structure of the Structure -> Views menu makes no sense.\",\"user\":{\"login\":\"klonos\",\"id\":2423362,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2423362?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/klonos\",\"html_url\":\"https://github.com/klonos\",\"followers_url\":\"https://api.github.com/users/klonos/followers\",\"following_url\":\"https://api.github.com/users/klonos/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/klonos/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/klonos/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/klonos/subscriptions\",\"organizations_url\":\"https://api.github.com/users/klonos/orgs\",\"repos_url\":\"https://api.github.com/users/klonos/repos\",\"events_url\":\"https://api.github.com/users/klonos/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/klonos/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:57:24Z\",\"updated_at\":\"2015-01-01T01:03:02Z\",\"closed_at\":null,\"body\":\"![structure-views_menu_makes_no_sense](https://cloud.githubusercontent.com/assets/2423362/5591356/f4ac78aa-91aa-11e4-8142-aa0104e86099.png)\\r\\n\\r\\n1. The views submenu entries in the menu are listed in alphabetical order while in the views list page they are not.\\r\\n2. They are not grouped together nor listed alphabetically with the rest of the menu entries:\\r\\n  - The \\\"Add new view\\\", \\\"Import\\\", \\\"List\\\" and \\\"Settings\\\" entries are listed alphabetically, but not when views are added (once enabled)\\r\\n  - The views are listed alphabetically, but the \\\"Settings\\\" entry is included/sorted along and the alphabetization starts after the \\\"Add new view\\\", \\\"Import\\\", \\\"List\\\" trio.\\r\\n\\r\\nWTF!?!\\r\\n\\r\\nDid we mean to list the views within the \\\"List\\\" entry but failed somehow? If not, I think it would make sense to move them there.\\r\\n\\r\\nIf we do not move them under \\\"List\\\", then lets move all non-view-name entries in that menu (currently \\\"Add new view\\\", \\\"Import\\\", \\\"List\\\" and \\\"Settings\\\") at the top and list the views entries alphabetically right after them.\\r\\n\\r\\nI think that if the views entries are to be listed alphabetically in the menu, then so should they be in the index page (```/admin/structure/views```). If we choose to not list them alphabetically in the views index page, then at least keep the same order in the menu. If we do decide to list them alphabetically everywhere (my vote BTW), then the enabled views in the views index page should \\\"bubble\\\" up and the disabled kept down, but still in alphabetical order.\"},\"comment\":{\"url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/comments/68477274\",\"html_url\":\"https://github.com/backdrop/backdrop-issues/issues/521#issuecomment-68477274\",\"issue_url\":\"https://api.github.com/repos/backdrop/backdrop-issues/issues/521\",\"id\":68477274,\"user\":{\"login\":\"quicksketch\",\"id\":100206,\"avatar_url\":\"https://avatars.githubusercontent.com/u/100206?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/quicksketch\",\"html_url\":\"https://github.com/quicksketch\",\"followers_url\":\"https://api.github.com/users/quicksketch/followers\",\"following_url\":\"https://api.github.com/users/quicksketch/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/quicksketch/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/quicksketch/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/quicksketch/subscriptions\",\"organizations_url\":\"https://api.github.com/users/quicksketch/orgs\",\"repos_url\":\"https://api.github.com/users/quicksketch/repos\",\"events_url\":\"https://api.github.com/users/quicksketch/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/quicksketch/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:02Z\",\"updated_at\":\"2015-01-01T01:03:02Z\",\"body\":\"> Did we mean to list the views within the \\\"List\\\" entry but failed somehow? If not, I think it would make sense to move them there.\\r\\n\\r\\nThis is a straight-port from D7's admin_menu. I don't think we've reconsidered anything about the way that views are listed in the menu.\\r\\n\\r\\nMy preference would be to simply remove the individual Views from the admin bar entirely. We don't include blocks, layouts, text formats, image styles, or any other user-created configurations within a sub-system directly within the admin bar, why are Views special? You can easily end up with dozens of views even on a moderately complex site, making use of the admin menu to navigate them hopeless anyway.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:02Z\",\"org\":{\"id\":5283039,\"login\":\"backdrop\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/backdrop\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5283039?\"}}\n{\"id\":\"2489397193\",\"type\":\"CreateEvent\",\"actor\":{\"id\":568036,\"login\":\"Tyilo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tyilo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/568036?\"},\"repo\":{\"id\":28676548,\"name\":\"Tyilo/pyth\",\"url\":\"https://api.github.com/repos/Tyilo/pyth\"},\"payload\":{\"ref\":\"patch-2\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Pyth, an extremely concise language. Try it here:\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:03Z\"}\n{\"id\":\"2489397194\",\"type\":\"PushEvent\",\"actor\":{\"id\":616495,\"login\":\"harveyt\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harveyt\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/616495?\"},\"repo\":{\"id\":28677911,\"name\":\"harveyt/harveyt.github.io\",\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io\"},\"payload\":{\"push_id\":536752843,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/source\",\"head\":\"9f57ade1c98efb400f57d710a69c272464596ba4\",\"before\":\"44fbf89a779b08acc862d4177398f8cb7b1eed3d\",\"commits\":[{\"sha\":\"6a98c29ce4e4d65ba4958c06a382a935f931a20a\",\"author\":{\"email\":\"5b18f340f96afc1bfd4fbf498467b8b0cb41ea73@me.com\",\"name\":\"Harvey Thompson\"},\"message\":\"Added _deploy.yml for github publishing.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io/commits/6a98c29ce4e4d65ba4958c06a382a935f931a20a\"},{\"sha\":\"9f57ade1c98efb400f57d710a69c272464596ba4\",\"author\":{\"email\":\"5b18f340f96afc1bfd4fbf498467b8b0cb41ea73@me.com\",\"name\":\"Harvey Thompson\"},\"message\":\"Final tweaks before first publish.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io/commits/9f57ade1c98efb400f57d710a69c272464596ba4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:03Z\"}\n{\"id\":\"2489397210\",\"type\":\"WatchEvent\",\"actor\":{\"id\":9970148,\"login\":\"bchoomnuan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchoomnuan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9970148?\"},\"repo\":{\"id\":27032923,\"name\":\"dockerboard/dockerboard\",\"url\":\"https://api.github.com/repos/dockerboard/dockerboard\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:05Z\",\"org\":{\"id\":9627875,\"login\":\"dockerboard\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/dockerboard\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9627875?\"}}\n{\"id\":\"2489397216\",\"type\":\"PushEvent\",\"actor\":{\"id\":9515067,\"login\":\"freundTech\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/freundTech\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9515067?\"},\"repo\":{\"id\":27674403,\"name\":\"freundTech/lightDMX\",\"url\":\"https://api.github.com/repos/freundTech/lightDMX\"},\"payload\":{\"push_id\":536752852,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5a3fdac84c88e3bc2e2b24afa750b5ca36e74bc4\",\"before\":\"1949bbb851b79b526154c7cec6a0e7e00f1420a6\",\"commits\":[{\"sha\":\"5a3fdac84c88e3bc2e2b24afa750b5ca36e74bc4\",\"author\":{\"email\":\"d6643dd9a6fe6114980ecaf6d787b2a0da7fc377@gmail.com\",\"name\":\"Adrian\"},\"message\":\"Updated TODO\",\"distinct\":true,\"url\":\"https://api.github.com/repos/freundTech/lightDMX/commits/5a3fdac84c88e3bc2e2b24afa750b5ca36e74bc4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:07Z\"}\n{\"id\":\"2489397221\",\"type\":\"PushEvent\",\"actor\":{\"id\":3635680,\"login\":\"BatikhSouri\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/BatikhSouri\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3635680?\"},\"repo\":{\"id\":28186746,\"name\":\"BatikhSouri/git-lesson\",\"url\":\"https://api.github.com/repos/BatikhSouri/git-lesson\"},\"payload\":{\"push_id\":536752856,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0f00c9c76eb55c71f81be40fd7479c7923bf8187\",\"before\":\"b6b546ecf0c97fffcb8c4d0cd8bd1554d256abca\",\"commits\":[{\"sha\":\"0f00c9c76eb55c71f81be40fd7479c7923bf8187\",\"author\":{\"email\":\"965e9967770cc67c48586cfa730edd14b783f677@gmail.com\",\"name\":\"Syrian Watermelon\"},\"message\":\"Improving the lesson parsing, among other little tweaks\",\"distinct\":true,\"url\":\"https://api.github.com/repos/BatikhSouri/git-lesson/commits/0f00c9c76eb55c71f81be40fd7479c7923bf8187\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:07Z\"}\n{\"id\":\"2489397225\",\"type\":\"PushEvent\",\"actor\":{\"id\":1745861,\"login\":\"topaztee\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/topaztee\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1745861?\"},\"repo\":{\"id\":28677407,\"name\":\"topaztee/topaztee.github.io\",\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io\"},\"payload\":{\"push_id\":536752857,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"70a7d7ee278b87fbcbe0c785d3ec3c1dcb12a8d4\",\"before\":\"a66e99dc9be241e6a513acfcc79ac27757a4a5bb\",\"commits\":[{\"sha\":\"70a7d7ee278b87fbcbe0c785d3ec3c1dcb12a8d4\",\"author\":{\"email\":\"f74c82d708bb42a372674042ebc8a1411fbc9344@192-168-1-2.tpgi.com.au\",\"name\":\"topaztur@gmail.com\"},\"message\":\"Blog update at 2015-01-01 01:03:00\",\"distinct\":true,\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io/commits/70a7d7ee278b87fbcbe0c785d3ec3c1dcb12a8d4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:07Z\"}\n{\"id\":\"2489397239\",\"type\":\"WatchEvent\",\"actor\":{\"id\":923144,\"login\":\"pythonesque\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pythonesque\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/923144?\"},\"repo\":{\"id\":1698955,\"name\":\"csmith-project/csmith\",\"url\":\"https://api.github.com/repos/csmith-project/csmith\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:09Z\",\"org\":{\"id\":757200,\"login\":\"csmith-project\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/csmith-project\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/757200?\"}}\n{\"id\":\"2489397243\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":2222834,\"login\":\"madeofwin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/madeofwin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2222834?\"},\"repo\":{\"id\":12279254,\"name\":\"cudamat/cudamat\",\"url\":\"https://api.github.com/repos/cudamat/cudamat\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/cudamat/cudamat/issues/40\",\"labels_url\":\"https://api.github.com/repos/cudamat/cudamat/issues/40/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cudamat/cudamat/issues/40/comments\",\"events_url\":\"https://api.github.com/repos/cudamat/cudamat/issues/40/events\",\"html_url\":\"https://github.com/cudamat/cudamat/issues/40\",\"id\":53210222,\"number\":40,\"title\":\"nvcc fatal   : Path to libdevice library not specified\",\"user\":{\"login\":\"madeofwin\",\"id\":2222834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2222834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/madeofwin\",\"html_url\":\"https://github.com/madeofwin\",\"followers_url\":\"https://api.github.com/users/madeofwin/followers\",\"following_url\":\"https://api.github.com/users/madeofwin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/madeofwin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/madeofwin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/madeofwin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/madeofwin/orgs\",\"repos_url\":\"https://api.github.com/users/madeofwin/repos\",\"events_url\":\"https://api.github.com/users/madeofwin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/madeofwin/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:09Z\",\"updated_at\":\"2015-01-01T01:03:09Z\",\"closed_at\":null,\"body\":\"Hi, I'm running a fresh ubuntu 14.04 installation with Cuda 6.5 and a GT650M. I downloaded the deb files from the official nvidia cuda site, added it via dpkg and executed 'sudo apt-get install cuda'. This installed Cuda 6.5 and nvidia-340 drivers. However, I'm running into troubles when I try to install cudamat by 'sudo python install setup.py', see the output:\\r\\n\\r\\nrunning install\\r\\nrunning bdist_egg\\r\\nrunning egg_info\\r\\nwriting cudamat.egg-info/PKG-INFO\\r\\nwriting top-level names to cudamat.egg-info/top_level.txt\\r\\nwriting dependency_links to cudamat.egg-info/dependency_links.txt\\r\\nreading manifest file 'cudamat.egg-info/SOURCES.txt'\\r\\nwriting manifest file 'cudamat.egg-info/SOURCES.txt'\\r\\ninstalling library code to build/bdist.linux-x86_64/egg\\r\\nrunning install_lib\\r\\nrunning build_py\\r\\nrunning build_ext\\r\\nbuilding 'cudamat.libcudamat' extension\\r\\nnvcc -I/usr/include/python2.7 -c cudamat/cudamat.cu -o build/temp.linux-x86_64-2.7/cudamat/cudamat.o -O --ptxas-options=-v --compiler-options '-fPIC'\\r\\nnvcc fatal   : Path to libdevice library not specified\\r\\nerror: command 'nvcc' failed with exit status 1\\r\\n\\r\\nI'm googling and trying different things for hours now and can't find a solutions. cuda examples are running without problems. Can you help me out?\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:09Z\",\"org\":{\"id\":5281810,\"login\":\"cudamat\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cudamat\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5281810?\"}}\n{\"id\":\"2489397244\",\"type\":\"CreateEvent\",\"actor\":{\"id\":838098,\"login\":\"burgerbecky\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/burgerbecky\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/838098?\"},\"repo\":{\"id\":28677547,\"name\":\"burgerbecky/glslvisualstudio\",\"url\":\"https://api.github.com/repos/burgerbecky/glslvisualstudio\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Visual Studio Plug in to add in GLSL source as directly linked source code\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:09Z\"}\n{\"id\":\"2489397246\",\"type\":\"PushEvent\",\"actor\":{\"id\":1398544,\"login\":\"joelpurra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joelpurra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1398544?\"},\"repo\":{\"id\":28595466,\"name\":\"joelpurra/jqnpm\",\"url\":\"https://api.github.com/repos/joelpurra/jqnpm\"},\"payload\":{\"push_id\":536752863,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"402ca05191199df582b663242f0e000d6fb33292\",\"before\":\"fbb2a9631bf887443ad696637417d1564f07f56b\",\"commits\":[{\"sha\":\"591fa3c627d7a3eeee457ed5daf7a997a5469338\",\"author\":{\"email\":\"4774889311d71ea3a81d88c867cde99c6ccb427e@joelpurra.com\",\"name\":\"Joel Purra\"},\"message\":\"Only clean folders affected by tests, not generated bundles - and re-add an ignored file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/joelpurra/jqnpm/commits/591fa3c627d7a3eeee457ed5daf7a997a5469338\"},{\"sha\":\"402ca05191199df582b663242f0e000d6fb33292\",\"author\":{\"email\":\"4774889311d71ea3a81d88c867cde99c6ccb427e@joelpurra.com\",\"name\":\"Joel Purra\"},\"message\":\"Encourage cleaning up files, regenerating bundles\",\"distinct\":true,\"url\":\"https://api.github.com/repos/joelpurra/jqnpm/commits/402ca05191199df582b663242f0e000d6fb33292\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:10Z\"}\n{\"id\":\"2489397249\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":4712580,\"login\":\"ericdahl\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ericdahl\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4712580?\"},\"repo\":{\"id\":9396682,\"name\":\"spring-guides/gs-rest-service\",\"url\":\"https://api.github.com/repos/spring-guides/gs-rest-service\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/spring-guides/gs-rest-service/issues/28\",\"labels_url\":\"https://api.github.com/repos/spring-guides/gs-rest-service/issues/28/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/spring-guides/gs-rest-service/issues/28/comments\",\"events_url\":\"https://api.github.com/repos/spring-guides/gs-rest-service/issues/28/events\",\"html_url\":\"https://github.com/spring-guides/gs-rest-service/issues/28\",\"id\":53210223,\"number\":28,\"title\":\"pom.xml has unnecessary <start-class>\",\"user\":{\"login\":\"ericdahl\",\"id\":4712580,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4712580?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ericdahl\",\"html_url\":\"https://github.com/ericdahl\",\"followers_url\":\"https://api.github.com/users/ericdahl/followers\",\"following_url\":\"https://api.github.com/users/ericdahl/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ericdahl/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ericdahl/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ericdahl/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ericdahl/orgs\",\"repos_url\":\"https://api.github.com/users/ericdahl/repos\",\"events_url\":\"https://api.github.com/users/ericdahl/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ericdahl/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:10Z\",\"updated_at\":\"2015-01-01T01:03:10Z\",\"closed_at\":null,\"body\":\"I was confused to see that the ```pom.xml``` includes:\\r\\n\\r\\n```xml\\r\\n<properties>\\r\\n    <start-class>hello.Application</start-class>\\r\\n</properties>\\r\\n```\\r\\n\\r\\nsince the [main documentation](http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started-first-application-pom) doesn't include references to it. It's unnecessary and adds unnecessary complexity for a getting started guide.\\r\\n\\r\\nI was about to submit a pull request to remove it, but then I saw that all of the spring-guides include it. Possibly there's some reason for keeping it?\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:10Z\",\"org\":{\"id\":4161866,\"login\":\"spring-guides\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/spring-guides\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4161866?\"}}\n{\"id\":\"2489397250\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":7882662,\"login\":\"codeschool-kiddo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codeschool-kiddo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7882662?\"},\"repo\":{\"id\":25890220,\"name\":\"JorgeX/dojo_rules\",\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/22397268\",\"id\":22397268,\"diff_hunk\":\"@@ -1,3 +1,3 @@\\n-Hello, Deadly vipers\\n-regards, Jorge!\\n-Favourite path: Javascript\\n+* GitG\\n+* Java\\n+* polle\\n\\\\ No newline at end of file\",\"path\":\"introduction.md\",\"position\":7,\"original_position\":7,\"commit_id\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"original_commit_id\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"user\":{\"login\":\"codeschool-kiddo\",\"id\":7882662,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7882662?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codeschool-kiddo\",\"html_url\":\"https://github.com/codeschool-kiddo\",\"followers_url\":\"https://api.github.com/users/codeschool-kiddo/followers\",\"following_url\":\"https://api.github.com/users/codeschool-kiddo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/codeschool-kiddo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/codeschool-kiddo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/codeschool-kiddo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/codeschool-kiddo/orgs\",\"repos_url\":\"https://api.github.com/users/codeschool-kiddo/repos\",\"events_url\":\"https://api.github.com/users/codeschool-kiddo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/codeschool-kiddo/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Since you're becoming a GitHub master, could you also add \\\"Killing history using git rebase\\\" as one of your deadly skills?\",\"created_at\":\"2015-01-01T01:03:10Z\",\"updated_at\":\"2015-01-01T01:03:10Z\",\"html_url\":\"https://github.com/JorgeX/dojo_rules/pull/1#discussion_r22397268\",\"pull_request_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/22397268\"},\"html\":{\"href\":\"https://github.com/JorgeX/dojo_rules/pull/1#discussion_r22397268\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\",\"id\":26739426,\"html_url\":\"https://github.com/JorgeX/dojo_rules/pull/1\",\"diff_url\":\"https://github.com/JorgeX/dojo_rules/pull/1.diff\",\"patch_url\":\"https://github.com/JorgeX/dojo_rules/pull/1.patch\",\"issue_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1\",\"number\":1,\"state\":\"open\",\"locked\":false,\"title\":\"Deadly skills\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"ok\",\"created_at\":\"2015-01-01T01:03:01Z\",\"updated_at\":\"2015-01-01T01:03:10Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"fe9e81e8157ba236bc48b8ee411b4e12fc8666e4\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/commits\",\"review_comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/comments\",\"review_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1/comments\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"head\":{\"label\":\"JorgeX:deadly_skills\",\"ref\":\"deadly_skills\",\"sha\":\"923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":25890220,\"name\":\"dojo_rules\",\"full_name\":\"JorgeX/dojo_rules\",\"owner\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/JorgeX/dojo_rules\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\",\"forks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/forks\",\"keys_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/teams\",\"hooks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/hooks\",\"issue_events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/events\",\"assignees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/tags\",\"blobs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/languages\",\"stargazers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/stargazers\",\"contributors_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contributors\",\"subscribers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscribers\",\"subscription_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscription\",\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/merges\",\"archive_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/downloads\",\"issues_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/releases{/id}\",\"created_at\":\"2014-10-28T21:09:20Z\",\"updated_at\":\"2014-10-28T21:13:40Z\",\"pushed_at\":\"2015-01-01T01:00:43Z\",\"git_url\":\"git://github.com/JorgeX/dojo_rules.git\",\"ssh_url\":\"git@github.com:JorgeX/dojo_rules.git\",\"clone_url\":\"https://github.com/JorgeX/dojo_rules.git\",\"svn_url\":\"https://github.com/JorgeX/dojo_rules\",\"homepage\":null,\"size\":104,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"JorgeX:master\",\"ref\":\"master\",\"sha\":\"d1b0081019d3c0e43266beaa87fee523fc684649\",\"user\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":25890220,\"name\":\"dojo_rules\",\"full_name\":\"JorgeX/dojo_rules\",\"owner\":{\"login\":\"JorgeX\",\"id\":6737270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6737270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JorgeX\",\"html_url\":\"https://github.com/JorgeX\",\"followers_url\":\"https://api.github.com/users/JorgeX/followers\",\"following_url\":\"https://api.github.com/users/JorgeX/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JorgeX/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JorgeX/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JorgeX/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JorgeX/orgs\",\"repos_url\":\"https://api.github.com/users/JorgeX/repos\",\"events_url\":\"https://api.github.com/users/JorgeX/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JorgeX/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/JorgeX/dojo_rules\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/JorgeX/dojo_rules\",\"forks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/forks\",\"keys_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/teams\",\"hooks_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/hooks\",\"issue_events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/events\",\"assignees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/tags\",\"blobs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/languages\",\"stargazers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/stargazers\",\"contributors_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contributors\",\"subscribers_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscribers\",\"subscription_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/subscription\",\"commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/merges\",\"archive_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/downloads\",\"issues_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/JorgeX/dojo_rules/releases{/id}\",\"created_at\":\"2014-10-28T21:09:20Z\",\"updated_at\":\"2014-10-28T21:13:40Z\",\"pushed_at\":\"2015-01-01T01:00:43Z\",\"git_url\":\"git://github.com/JorgeX/dojo_rules.git\",\"ssh_url\":\"git@github.com:JorgeX/dojo_rules.git\",\"clone_url\":\"https://github.com/JorgeX/dojo_rules.git\",\"svn_url\":\"https://github.com/JorgeX/dojo_rules\",\"homepage\":null,\"size\":104,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1\"},\"html\":{\"href\":\"https://github.com/JorgeX/dojo_rules/pull/1\"},\"issue\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1\"},\"comments\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/issues/1/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/pulls/1/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/JorgeX/dojo_rules/statuses/923b0ffbc6a7f2fc6ff55b6d415fe70f824ea5e0\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:03:10Z\"}\n{\"id\":\"2489397251\",\"type\":\"PushEvent\",\"actor\":{\"id\":13564,\"login\":\"ezyang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ezyang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/13564?\"},\"repo\":{\"id\":28672247,\"name\":\"ezyang/stenowiki\",\"url\":\"https://api.github.com/repos/ezyang/stenowiki\"},\"payload\":{\"push_id\":536752865,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"36ab97683d88f48611b24a8afd888d18fa660fea\",\"before\":\"82d4215f37b7f39a5b818fc53b8120650648da8b\",\"commits\":[{\"sha\":\"36ab97683d88f48611b24a8afd888d18fa660fea\",\"author\":{\"email\":\"dbd597f5635f432486c5d365e9bb585b3eaa1853@cs.stanford.edu\",\"name\":\"Edward Z. Yang\"},\"message\":\"Plz to not truncate\\n\\nSigned-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ezyang/stenowiki/commits/36ab97683d88f48611b24a8afd888d18fa660fea\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:10Z\"}\n{\"id\":\"2489397252\",\"type\":\"PushEvent\",\"actor\":{\"id\":9101573,\"login\":\"megantmcginley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/megantmcginley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9101573?\"},\"repo\":{\"id\":25549968,\"name\":\"megantmcginley/megantmcginley.github.io\",\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io\"},\"payload\":{\"push_id\":536752866,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1ac2c938f5436c6ff9576e262b80ccbced4586a8\",\"before\":\"e613a42ea94329f38b4f9994b48ccd4f845a5225\",\"commits\":[{\"sha\":\"1ac2c938f5436c6ff9576e262b80ccbced4586a8\",\"author\":{\"email\":\"92f56e51255edbb80c74150d0115560b34c2bc35@users.noreply.github.com\",\"name\":\"megantmcginley\"},\"message\":\"Update about.html\",\"distinct\":true,\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io/commits/1ac2c938f5436c6ff9576e262b80ccbced4586a8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:11Z\"}\n{\"id\":\"2489397253\",\"type\":\"PushEvent\",\"actor\":{\"id\":6539368,\"login\":\"acecodes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/acecodes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6539368?\"},\"repo\":{\"id\":28677898,\"name\":\"acecodes/ace_website_jekyll\",\"url\":\"https://api.github.com/repos/acecodes/ace_website_jekyll\"},\"payload\":{\"push_id\":536752867,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3ae19325c6d52115bd67dd5e3c30d7cf39919dd4\",\"before\":\"771ef2fd1fb62b6fbb993ea283997c1ec6669062\",\"commits\":[{\"sha\":\"3ae19325c6d52115bd67dd5e3c30d7cf39919dd4\",\"author\":{\"email\":\"c74c402f1e088afb65e1933981f21221d4142751@gmail.com\",\"name\":\"Ace\"},\"message\":\"Added web dev section and cleaned up contact form\",\"distinct\":true,\"url\":\"https://api.github.com/repos/acecodes/ace_website_jekyll/commits/3ae19325c6d52115bd67dd5e3c30d7cf39919dd4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:11Z\"}\n{\"id\":\"2489397255\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5352,\"login\":\"juno\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/juno\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5352?\"},\"repo\":{\"id\":28648528,\"name\":\"josh/cafe-js\",\"url\":\"https://api.github.com/repos/josh/cafe-js\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:12Z\"}\n{\"id\":\"2489397260\",\"type\":\"PushEvent\",\"actor\":{\"id\":6232704,\"login\":\"nashpitre\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nashpitre\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6232704?\"},\"repo\":{\"id\":28620312,\"name\":\"nashpitre/nashpitre.github.io\",\"url\":\"https://api.github.com/repos/nashpitre/nashpitre.github.io\"},\"payload\":{\"push_id\":536752869,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5a1d53fae41d4355c5710a87efbe6cfd6c96a539\",\"before\":\"51b30814df5685fd473d136c26735a17cf577361\",\"commits\":[{\"sha\":\"5a1d53fae41d4355c5710a87efbe6cfd6c96a539\",\"author\":{\"email\":\"be30e9ada7478b999c213bbb3535884788d4bb46@me.com\",\"name\":\"nashpitre\"},\"message\":\"revert\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nashpitre/nashpitre.github.io/commits/5a1d53fae41d4355c5710a87efbe6cfd6c96a539\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:12Z\"}\n{\"id\":\"2489397261\",\"type\":\"PushEvent\",\"actor\":{\"id\":1386607,\"login\":\"nbransby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nbransby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1386607?\"},\"repo\":{\"id\":25346686,\"name\":\"hambroperks/joda-convert\",\"url\":\"https://api.github.com/repos/hambroperks/joda-convert\"},\"payload\":{\"push_id\":536752870,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"80741e6ce6fa90f13d52b94fa7ebaa8347589d01\",\"before\":\"73695867751ce7e02a24ca5d8eb8ce23025f9c57\",\"commits\":[{\"sha\":\"6df78b2d5f529fdd8c731af40bc528c9faa357b7\",\"author\":{\"email\":\"75ef9faee755c70589550b513ad881e5a603182c@laundrapp.com\",\"name\":\"Nicholas Bransby-Williams\"},\"message\":\"add gradle wrapper\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hambroperks/joda-convert/commits/6df78b2d5f529fdd8c731af40bc528c9faa357b7\"},{\"sha\":\"80741e6ce6fa90f13d52b94fa7ebaa8347589d01\",\"author\":{\"email\":\"75ef9faee755c70589550b513ad881e5a603182c@laundrapp.com\",\"name\":\"Nicholas Bransby-Williams\"},\"message\":\"translate with 0.9.5\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hambroperks/joda-convert/commits/80741e6ce6fa90f13d52b94fa7ebaa8347589d01\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:12Z\",\"org\":{\"id\":9292560,\"login\":\"hambroperks\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/hambroperks\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9292560?\"}}\n{\"id\":\"2489397267\",\"type\":\"PushEvent\",\"actor\":{\"id\":8132102,\"login\":\"the-cdnjs-curator\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/the-cdnjs-curator\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8132102?\"},\"repo\":{\"id\":18663590,\"name\":\"cdnjs/new-website\",\"url\":\"https://api.github.com/repos/cdnjs/new-website\"},\"payload\":{\"push_id\":536752875,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5ba7b04046ce0a6421ebdb04d954baa39128b42d\",\"before\":\"d8e84baf664dde8025c7c00f177fc681c60a405d\",\"commits\":[{\"sha\":\"5ba7b04046ce0a6421ebdb04d954baa39128b42d\",\"author\":{\"email\":\"a81c8330dae20b275f250c899f16a97692457035@gmail.com\",\"name\":\"the-cdnjs-curator\"},\"message\":\"added ner files\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cdnjs/new-website/commits/5ba7b04046ce0a6421ebdb04d954baa39128b42d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:12Z\",\"org\":{\"id\":637362,\"login\":\"cdnjs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cdnjs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/637362?\"}}\n{\"id\":\"2489397278\",\"type\":\"CreateEvent\",\"actor\":{\"id\":616495,\"login\":\"harveyt\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harveyt\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/616495?\"},\"repo\":{\"id\":28677911,\"name\":\"harveyt/harveyt.github.io\",\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"source\",\"description\":\"Personal webpage for Harvey Thompson\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:13Z\"}\n{\"id\":\"2489397279\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Home\",\"title\":\"Home\",\"summary\":null,\"action\":\"edited\",\"sha\":\"685d71e9099c3e471f9a9a4917a16a52e89582f7\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Home\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:13Z\"}\n{\"id\":\"2489397280\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":778068,\"login\":\"snarfed\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/snarfed\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/778068?\"},\"repo\":{\"id\":2968000,\"name\":\"snarfed/bridgy\",\"url\":\"https://api.github.com/repos/snarfed/bridgy\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/snarfed/bridgy/issues/315\",\"labels_url\":\"https://api.github.com/repos/snarfed/bridgy/issues/315/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/snarfed/bridgy/issues/315/comments\",\"events_url\":\"https://api.github.com/repos/snarfed/bridgy/issues/315/events\",\"html_url\":\"https://github.com/snarfed/bridgy/issues/315\",\"id\":51612550,\"number\":315,\"title\":\"char encoding bug while discovering webmention endpoint for teriiehina.net\",\"user\":{\"login\":\"snarfed\",\"id\":778068,\"avatar_url\":\"https://avatars.githubusercontent.com/u/778068?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/snarfed\",\"html_url\":\"https://github.com/snarfed\",\"followers_url\":\"https://api.github.com/users/snarfed/followers\",\"following_url\":\"https://api.github.com/users/snarfed/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/snarfed/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/snarfed/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/snarfed/subscriptions\",\"organizations_url\":\"https://api.github.com/users/snarfed/orgs\",\"repos_url\":\"https://api.github.com/users/snarfed/repos\",\"events_url\":\"https://api.github.com/users/snarfed/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/snarfed/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/snarfed/bridgy/labels/listen\",\"name\":\"listen\",\"color\":\"207de5\"},{\"url\":\"https://api.github.com/repos/snarfed/bridgy/labels/now\",\"name\":\"now\",\"color\":\"009800\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"kylewm\",\"id\":950127,\"avatar_url\":\"https://avatars.githubusercontent.com/u/950127?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kylewm\",\"html_url\":\"https://github.com/kylewm\",\"followers_url\":\"https://api.github.com/users/kylewm/followers\",\"following_url\":\"https://api.github.com/users/kylewm/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kylewm/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kylewm/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kylewm/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kylewm/orgs\",\"repos_url\":\"https://api.github.com/users/kylewm/repos\",\"events_url\":\"https://api.github.com/users/kylewm/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kylewm/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":20,\"created_at\":\"2014-12-10T21:20:17Z\",\"updated_at\":\"2015-01-01T01:03:13Z\",\"closed_at\":\"2015-01-01T00:49:46Z\",\"body\":\"[log](https://www.brid.gy/log?start_time=1418245934&key=aglzfmJyaWQtZ3lyNQsSCFJlc3BvbnNlIid0YWc6dHdpdHRlci5jb20sMjAxMzo1NDI3MjY5MzQ4ODc1NDI3ODQM):\\r\\n\\r\\n```py\\r\\nStarting Response comment tag:twitter.com,2013:542726934887542784 https://twitter.com/teriiehina/status/542726934887542784\\r\\nWebmention from https://brid-gy.appspot.com/comment/twitter/elfpavlik/541974355010650112/542726934887542784 to http://www.teriiehina.net/\\r\\nSending...\\r\\nStarting new HTTP connection (1): www.teriiehina.net\\r\\n\\\"GET / HTTP/1.1\\\" 200 None\\r\\nTraceback (most recent call last):\\r\\n File \\\"/base/data/home/apps/s~brid-gy/3.380705500338596153/tasks.py\\\", line 494, in do_send_webmentions\\r\\n  if not mention.send(timeout=999):\\r\\n File \\\"/base/data/home/apps/s~brid-gy/3.380705500338596153/webmention-tools/webmentiontools/send.py\\\", line 24, in send\\r\\n  self._discoverEndpoint()\\r\\n File \\\"/base/data/home/apps/s~brid-gy/3.380705500338596153/webmention-tools/webmentiontools/send.py\\\", line 60, in _discoverEndpoint\\r\\n  soup = BeautifulSoup(self.html)\\r\\n File \\\"/base/data/home/apps/s~brid-gy/3.380705500338596153/activitystreams/beautifulsoup/bs4/__init__.py\\\", line 167, in __init__\\r\\n  if os.path.exists(markup):\\r\\n File \\\"/base/data/home/runtimes/python27/python27_dist/lib/python2.7/genericpath.py\\\", line 18, in exists\\r\\n  os.stat(path)\\r\\nUnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)```\"},\"comment\":{\"url\":\"https://api.github.com/repos/snarfed/bridgy/issues/comments/68477279\",\"html_url\":\"https://github.com/snarfed/bridgy/issues/315#issuecomment-68477279\",\"issue_url\":\"https://api.github.com/repos/snarfed/bridgy/issues/315\",\"id\":68477279,\"user\":{\"login\":\"snarfed\",\"id\":778068,\"avatar_url\":\"https://avatars.githubusercontent.com/u/778068?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/snarfed\",\"html_url\":\"https://github.com/snarfed\",\"followers_url\":\"https://api.github.com/users/snarfed/followers\",\"following_url\":\"https://api.github.com/users/snarfed/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/snarfed/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/snarfed/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/snarfed/subscriptions\",\"organizations_url\":\"https://api.github.com/users/snarfed/orgs\",\"repos_url\":\"https://api.github.com/users/snarfed/repos\",\"events_url\":\"https://api.github.com/users/snarfed/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/snarfed/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:13Z\",\"updated_at\":\"2015-01-01T01:03:13Z\",\"body\":\"awesome! nice work, especially for including the test. serious brownie points for that. thank you!\\n\\nwant to push this? I won't touch prod the rest of the day, so all clear if you want.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:13Z\"}\n{\"id\":\"2489397283\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3889660,\"login\":\"coder-chenzhi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coder-chenzhi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3889660?\"},\"repo\":{\"id\":15045751,\"name\":\"docker/fig\",\"url\":\"https://api.github.com/repos/docker/fig\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:13Z\",\"org\":{\"id\":5429470,\"login\":\"docker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/docker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5429470?\"}}\n{\"id\":\"2489397287\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":663212,\"login\":\"tdas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?\"},\"repo\":{\"id\":17165658,\"name\":\"apache/spark\",\"url\":\"https://api.github.com/repos/apache/spark\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/comments/22397269\",\"id\":22397269,\"diff_hunk\":\"@@ -17,31 +17,65 @@\\n \\n package org.apache.spark.streaming.mqtt\\n \\n-import org.scalatest.FunSuite\\n-\\n+import org.scalatest.{BeforeAndAfter, FunSuite}\\n+import org.scalatest.concurrent.Eventually\\n+import scala.concurrent.duration._\\n import org.apache.spark.streaming.{Seconds, StreamingContext}\\n import org.apache.spark.storage.StorageLevel\\n import org.apache.spark.streaming.dstream.ReceiverInputDStream\\n+import org.eclipse.paho.client.mqttv3._\\n+import org.eclipse.paho.client.mqttv3.persist.MqttDefaultFilePersistence\\n \\n-class MQTTStreamSuite extends FunSuite {\\n-\\n-  val batchDuration = Seconds(1)\\n+class MQTTStreamSuite extends FunSuite with Eventually with BeforeAndAfter {\\n \\n+  private val batchDuration = Seconds(1)\\n   private val master: String = \\\"local[2]\\\"\\n-\\n   private val framework: String = this.getClass.getSimpleName\\n+  private val brokerUrl = \\\"tcp://localhost:1883\\\"\",\"path\":\"external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala\",\"position\":24,\"original_position\":24,\"commit_id\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"original_commit_id\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"user\":{\"login\":\"tdas\",\"id\":663212,\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"html_url\":\"https://github.com/tdas\",\"followers_url\":\"https://api.github.com/users/tdas/followers\",\"following_url\":\"https://api.github.com/users/tdas/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tdas/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tdas/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tdas/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tdas/orgs\",\"repos_url\":\"https://api.github.com/users/tdas/repos\",\"events_url\":\"https://api.github.com/users/tdas/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tdas/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Who is running the broker? Also this port is hardcoded. There is a small, non-trivial chance that this port may not be free (in Jenkins, where multiple series of test maybe running in parallel) causing the server to not bind thus failing test. Can you find a free port (see [FlumeStreamSuite](https://github.com/apache/spark/blob/master/external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumeStreamSuite.scala#L78)) and use that instead?\",\"created_at\":\"2015-01-01T01:03:14Z\",\"updated_at\":\"2015-01-01T01:03:14Z\",\"html_url\":\"https://github.com/apache/spark/pull/3844#discussion_r22397269\",\"pull_request_url\":\"https://api.github.com/repos/apache/spark/pulls/3844\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/comments/22397269\"},\"html\":{\"href\":\"https://github.com/apache/spark/pull/3844#discussion_r22397269\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/3844\",\"id\":26684823,\"html_url\":\"https://github.com/apache/spark/pull/3844\",\"diff_url\":\"https://github.com/apache/spark/pull/3844.diff\",\"patch_url\":\"https://github.com/apache/spark/pull/3844.patch\",\"issue_url\":\"https://api.github.com/repos/apache/spark/issues/3844\",\"number\":3844,\"state\":\"open\",\"locked\":false,\"title\":\"[SPARK-4631] unit test for MQTT\",\"user\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Please review the unit test for MQTT\",\"created_at\":\"2014-12-30T13:12:33Z\",\"updated_at\":\"2015-01-01T01:03:14Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"78f53f74b7089734c2cc3a2f701e6044194cda1d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/apache/spark/pulls/3844/commits\",\"review_comments_url\":\"https://api.github.com/repos/apache/spark/pulls/3844/comments\",\"review_comment_url\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/issues/3844/comments\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/fc8eb286db6aa8e78a567537996011f554eed969\",\"head\":{\"label\":\"Bilna:master\",\"ref\":\"master\",\"sha\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"user\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28630516,\"name\":\"spark\",\"full_name\":\"Bilna/spark\",\"owner\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Bilna/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":true,\"url\":\"https://api.github.com/repos/Bilna/spark\",\"forks_url\":\"https://api.github.com/repos/Bilna/spark/forks\",\"keys_url\":\"https://api.github.com/repos/Bilna/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Bilna/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Bilna/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/Bilna/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Bilna/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Bilna/spark/events\",\"assignees_url\":\"https://api.github.com/repos/Bilna/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Bilna/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Bilna/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/Bilna/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Bilna/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Bilna/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Bilna/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Bilna/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Bilna/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/Bilna/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Bilna/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Bilna/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Bilna/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/Bilna/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Bilna/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Bilna/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Bilna/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Bilna/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Bilna/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Bilna/spark/merges\",\"archive_url\":\"https://api.github.com/repos/Bilna/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Bilna/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/Bilna/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Bilna/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Bilna/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Bilna/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Bilna/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Bilna/spark/releases{/id}\",\"created_at\":\"2014-12-30T12:47:01Z\",\"updated_at\":\"2014-12-31T09:52:36Z\",\"pushed_at\":\"2014-12-31T09:52:35Z\",\"git_url\":\"git://github.com/Bilna/spark.git\",\"ssh_url\":\"git@github.com:Bilna/spark.git\",\"clone_url\":\"https://github.com/Bilna/spark.git\",\"svn_url\":\"https://github.com/Bilna/spark\",\"homepage\":null,\"size\":87823,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"apache:master\",\"ref\":\"master\",\"sha\":\"352ed6bbe3c3b67e52e298e7c535ae414d96beca\",\"user\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":17165658,\"name\":\"spark\",\"full_name\":\"apache/spark\",\"owner\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/apache/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":false,\"url\":\"https://api.github.com/repos/apache/spark\",\"forks_url\":\"https://api.github.com/repos/apache/spark/forks\",\"keys_url\":\"https://api.github.com/repos/apache/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/apache/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/apache/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/apache/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/apache/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/apache/spark/events\",\"assignees_url\":\"https://api.github.com/repos/apache/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/apache/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/apache/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/apache/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/apache/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/apache/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/apache/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/apache/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/apache/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/apache/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/apache/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/apache/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/apache/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/apache/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/apache/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/apache/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/apache/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/apache/spark/merges\",\"archive_url\":\"https://api.github.com/repos/apache/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/apache/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/apache/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/apache/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/apache/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/apache/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/apache/spark/releases{/id}\",\"created_at\":\"2014-02-25T08:00:08Z\",\"updated_at\":\"2015-01-01T00:59:33Z\",\"pushed_at\":\"2015-01-01T00:59:33Z\",\"git_url\":\"git://github.com/apache/spark.git\",\"ssh_url\":\"git@github.com:apache/spark.git\",\"clone_url\":\"https://github.com/apache/spark.git\",\"svn_url\":\"https://github.com/apache/spark\",\"homepage\":null,\"size\":1083068,\"stargazers_count\":2458,\"watchers_count\":2458,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":2179,\"mirror_url\":\"git://git.apache.org/spark.git\",\"open_issues_count\":268,\"forks\":2179,\"open_issues\":268,\"watchers\":2458,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844\"},\"html\":{\"href\":\"https://github.com/apache/spark/pull/3844\"},\"issue\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3844\"},\"comments\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3844/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/apache/spark/statuses/fc8eb286db6aa8e78a567537996011f554eed969\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:03:14Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489397290\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":4276679,\"login\":\"robot-dreams\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robot-dreams\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4276679?\"},\"repo\":{\"id\":25705676,\"name\":\"machine-intelligence/research-forum\",\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum\"},\"payload\":{\"action\":\"closed\",\"number\":62,\"pull_request\":{\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62\",\"id\":26712730,\"html_url\":\"https://github.com/machine-intelligence/research-forum/pull/62\",\"diff_url\":\"https://github.com/machine-intelligence/research-forum/pull/62.diff\",\"patch_url\":\"https://github.com/machine-intelligence/research-forum/pull/62.patch\",\"issue_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/62\",\"number\":62,\"state\":\"closed\",\"locked\":false,\"title\":\"Feature/look and feel\",\"user\":{\"login\":\"robot-dreams\",\"id\":4276679,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4276679?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robot-dreams\",\"html_url\":\"https://github.com/robot-dreams\",\"followers_url\":\"https://api.github.com/users/robot-dreams/followers\",\"following_url\":\"https://api.github.com/users/robot-dreams/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/robot-dreams/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/robot-dreams/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/robot-dreams/subscriptions\",\"organizations_url\":\"https://api.github.com/users/robot-dreams/orgs\",\"repos_url\":\"https://api.github.com/users/robot-dreams/repos\",\"events_url\":\"https://api.github.com/users/robot-dreams/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/robot-dreams/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2014-12-31T02:03:18Z\",\"updated_at\":\"2015-01-01T01:03:14Z\",\"closed_at\":\"2015-01-01T01:03:14Z\",\"merged_at\":\"2015-01-01T01:03:14Z\",\"merge_commit_sha\":\"6a6cdb34b95036fe523e7c90643ed5d3e278e2b8\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62/commits\",\"review_comments_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62/comments\",\"review_comment_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/62/comments\",\"statuses_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/statuses/f50826580c1451f8724025ba97df13b45d3a827c\",\"head\":{\"label\":\"machine-intelligence:feature/look-and-feel\",\"ref\":\"feature/look-and-feel\",\"sha\":\"f50826580c1451f8724025ba97df13b45d3a827c\",\"user\":{\"login\":\"machine-intelligence\",\"id\":7153909,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/machine-intelligence\",\"html_url\":\"https://github.com/machine-intelligence\",\"followers_url\":\"https://api.github.com/users/machine-intelligence/followers\",\"following_url\":\"https://api.github.com/users/machine-intelligence/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/machine-intelligence/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/machine-intelligence/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/machine-intelligence/subscriptions\",\"organizations_url\":\"https://api.github.com/users/machine-intelligence/orgs\",\"repos_url\":\"https://api.github.com/users/machine-intelligence/repos\",\"events_url\":\"https://api.github.com/users/machine-intelligence/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/machine-intelligence/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":25705676,\"name\":\"research-forum\",\"full_name\":\"machine-intelligence/research-forum\",\"owner\":{\"login\":\"machine-intelligence\",\"id\":7153909,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/machine-intelligence\",\"html_url\":\"https://github.com/machine-intelligence\",\"followers_url\":\"https://api.github.com/users/machine-intelligence/followers\",\"following_url\":\"https://api.github.com/users/machine-intelligence/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/machine-intelligence/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/machine-intelligence/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/machine-intelligence/subscriptions\",\"organizations_url\":\"https://api.github.com/users/machine-intelligence/orgs\",\"repos_url\":\"https://api.github.com/users/machine-intelligence/repos\",\"events_url\":\"https://api.github.com/users/machine-intelligence/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/machine-intelligence/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/machine-intelligence/research-forum\",\"description\":\"\",\"fork\":false,\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum\",\"forks_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/forks\",\"keys_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/teams\",\"hooks_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/hooks\",\"issue_events_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/events\",\"assignees_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/tags\",\"blobs_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/languages\",\"stargazers_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/stargazers\",\"contributors_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/contributors\",\"subscribers_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/subscribers\",\"subscription_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/subscription\",\"commits_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/merges\",\"archive_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/downloads\",\"issues_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/releases{/id}\",\"created_at\":\"2014-10-24T20:02:46Z\",\"updated_at\":\"2014-12-16T14:48:44Z\",\"pushed_at\":\"2015-01-01T01:03:14Z\",\"git_url\":\"git://github.com/machine-intelligence/research-forum.git\",\"ssh_url\":\"git@github.com:machine-intelligence/research-forum.git\",\"clone_url\":\"https://github.com/machine-intelligence/research-forum.git\",\"svn_url\":\"https://github.com/machine-intelligence/research-forum\",\"homepage\":null,\"size\":952,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Arc\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"machine-intelligence:master\",\"ref\":\"master\",\"sha\":\"72d4a081a5635cc218e2940e9d0f136eb4dd53e4\",\"user\":{\"login\":\"machine-intelligence\",\"id\":7153909,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/machine-intelligence\",\"html_url\":\"https://github.com/machine-intelligence\",\"followers_url\":\"https://api.github.com/users/machine-intelligence/followers\",\"following_url\":\"https://api.github.com/users/machine-intelligence/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/machine-intelligence/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/machine-intelligence/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/machine-intelligence/subscriptions\",\"organizations_url\":\"https://api.github.com/users/machine-intelligence/orgs\",\"repos_url\":\"https://api.github.com/users/machine-intelligence/repos\",\"events_url\":\"https://api.github.com/users/machine-intelligence/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/machine-intelligence/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":25705676,\"name\":\"research-forum\",\"full_name\":\"machine-intelligence/research-forum\",\"owner\":{\"login\":\"machine-intelligence\",\"id\":7153909,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/machine-intelligence\",\"html_url\":\"https://github.com/machine-intelligence\",\"followers_url\":\"https://api.github.com/users/machine-intelligence/followers\",\"following_url\":\"https://api.github.com/users/machine-intelligence/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/machine-intelligence/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/machine-intelligence/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/machine-intelligence/subscriptions\",\"organizations_url\":\"https://api.github.com/users/machine-intelligence/orgs\",\"repos_url\":\"https://api.github.com/users/machine-intelligence/repos\",\"events_url\":\"https://api.github.com/users/machine-intelligence/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/machine-intelligence/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/machine-intelligence/research-forum\",\"description\":\"\",\"fork\":false,\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum\",\"forks_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/forks\",\"keys_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/teams\",\"hooks_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/hooks\",\"issue_events_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/events\",\"assignees_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/tags\",\"blobs_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/languages\",\"stargazers_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/stargazers\",\"contributors_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/contributors\",\"subscribers_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/subscribers\",\"subscription_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/subscription\",\"commits_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/merges\",\"archive_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/downloads\",\"issues_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/machine-intelligence/research-forum/releases{/id}\",\"created_at\":\"2014-10-24T20:02:46Z\",\"updated_at\":\"2014-12-16T14:48:44Z\",\"pushed_at\":\"2015-01-01T01:03:14Z\",\"git_url\":\"git://github.com/machine-intelligence/research-forum.git\",\"ssh_url\":\"git@github.com:machine-intelligence/research-forum.git\",\"clone_url\":\"https://github.com/machine-intelligence/research-forum.git\",\"svn_url\":\"https://github.com/machine-intelligence/research-forum\",\"homepage\":null,\"size\":952,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Arc\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":0,\"open_issues\":1,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62\"},\"html\":{\"href\":\"https://github.com/machine-intelligence/research-forum/pull/62\"},\"issue\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/62\"},\"comments\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/issues/62/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/pulls/62/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/machine-intelligence/research-forum/statuses/f50826580c1451f8724025ba97df13b45d3a827c\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"robot-dreams\",\"id\":4276679,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4276679?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robot-dreams\",\"html_url\":\"https://github.com/robot-dreams\",\"followers_url\":\"https://api.github.com/users/robot-dreams/followers\",\"following_url\":\"https://api.github.com/users/robot-dreams/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/robot-dreams/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/robot-dreams/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/robot-dreams/subscriptions\",\"organizations_url\":\"https://api.github.com/users/robot-dreams/orgs\",\"repos_url\":\"https://api.github.com/users/robot-dreams/repos\",\"events_url\":\"https://api.github.com/users/robot-dreams/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/robot-dreams/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":2,\"additions\":82,\"deletions\":26,\"changed_files\":4}},\"public\":true,\"created_at\":\"2015-01-01T01:03:14Z\",\"org\":{\"id\":7153909,\"login\":\"machine-intelligence\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/machine-intelligence\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?\"}}\n{\"id\":\"2489397291\",\"type\":\"PushEvent\",\"actor\":{\"id\":7905778,\"login\":\"pankajmalhotra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pankajmalhotra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7905778?\"},\"repo\":{\"id\":22740858,\"name\":\"ulini/ops\",\"url\":\"https://api.github.com/repos/ulini/ops\"},\"payload\":{\"push_id\":536752882,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1431274619c45efb20987f0a62260e85dccb48ef\",\"before\":\"7d2237bdbf424692f909cae67c2a5e8c13335a38\",\"commits\":[{\"sha\":\"1431274619c45efb20987f0a62260e85dccb48ef\",\"author\":{\"email\":\"114151935e80b69b2f2bbc308c3a6960936b4e1a@ul.com\",\"name\":\"pankajmalhotra\"},\"message\":\"Added Ruby & Make\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ulini/ops/commits/1431274619c45efb20987f0a62260e85dccb48ef\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:15Z\",\"org\":{\"id\":6116922,\"login\":\"ulini\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ulini\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6116922?\"}}\n{\"id\":\"2489397293\",\"type\":\"PushEvent\",\"actor\":{\"id\":4276679,\"login\":\"robot-dreams\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/robot-dreams\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4276679?\"},\"repo\":{\"id\":25705676,\"name\":\"machine-intelligence/research-forum\",\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum\"},\"payload\":{\"push_id\":536752883,\"size\":3,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e81e3986d5db27d1c631ae46f003f4a06c61a816\",\"before\":\"72d4a081a5635cc218e2940e9d0f136eb4dd53e4\",\"commits\":[{\"sha\":\"0829853c4b6ac56bbf7beee570b7b279fc02ae0e\",\"author\":{\"email\":\"f44b7b9bbcd65117b7d5272f0703ff2e82101e98@gmail.com\",\"name\":\"Malo Bourgon\"},\"message\":\"Improved look and feel (thanks Brent!)\",\"distinct\":false,\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum/commits/0829853c4b6ac56bbf7beee570b7b279fc02ae0e\"},{\"sha\":\"f50826580c1451f8724025ba97df13b45d3a827c\",\"author\":{\"email\":\"f63b26569c1369b3dd67f2a7834b6651712fe1d3@gmail.com\",\"name\":\"Elliott Jin\"},\"message\":\"Content-length -> Content-Length, remove trailing whitespace\",\"distinct\":false,\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum/commits/f50826580c1451f8724025ba97df13b45d3a827c\"},{\"sha\":\"e81e3986d5db27d1c631ae46f003f4a06c61a816\",\"author\":{\"email\":\"f63b26569c1369b3dd67f2a7834b6651712fe1d3@gmail.com\",\"name\":\"robot-dreams\"},\"message\":\"Merge pull request #62 from machine-intelligence/feature/look-and-feel\\n\\nFeature/look and feel\",\"distinct\":true,\"url\":\"https://api.github.com/repos/machine-intelligence/research-forum/commits/e81e3986d5db27d1c631ae46f003f4a06c61a816\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:15Z\",\"org\":{\"id\":7153909,\"login\":\"machine-intelligence\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/machine-intelligence\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7153909?\"}}\n{\"id\":\"2489397296\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3203500,\"login\":\"aaroncrawford\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aaroncrawford\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3203500?\"},\"repo\":{\"id\":8170346,\"name\":\"artberri/sidr\",\"url\":\"https://api.github.com/repos/artberri/sidr\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/artberri/sidr/issues/4\",\"labels_url\":\"https://api.github.com/repos/artberri/sidr/issues/4/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/artberri/sidr/issues/4/comments\",\"events_url\":\"https://api.github.com/repos/artberri/sidr/issues/4/events\",\"html_url\":\"https://github.com/artberri/sidr/issues/4\",\"id\":12322955,\"number\":4,\"title\":\"Choppy animation\",\"user\":{\"login\":\"jdifelice\",\"id\":3942185,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3942185?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jdifelice\",\"html_url\":\"https://github.com/jdifelice\",\"followers_url\":\"https://api.github.com/users/jdifelice/followers\",\"following_url\":\"https://api.github.com/users/jdifelice/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jdifelice/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jdifelice/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jdifelice/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jdifelice/orgs\",\"repos_url\":\"https://api.github.com/users/jdifelice/repos\",\"events_url\":\"https://api.github.com/users/jdifelice/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jdifelice/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/artberri/sidr/labels/enhancement\",\"name\":\"enhancement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":11,\"created_at\":\"2013-03-22T15:08:36Z\",\"updated_at\":\"2015-01-01T01:03:15Z\",\"closed_at\":null,\"body\":\"Nice work on the plugin.  I have noticed, however, that the sliding animation is choppy.  Is there a way to make it more smooth?  Is there an option I've missed?\\r\\n\\r\\nI am running the plugin within a PhoneGap build on an iPhone 5 running iOS 6.1.2, but it's choppy even when viewed in desktop Chrome v20.0.1132.47.\\r\\n\\r\\nThanks.\"},\"comment\":{\"url\":\"https://api.github.com/repos/artberri/sidr/issues/comments/68477281\",\"html_url\":\"https://github.com/artberri/sidr/issues/4#issuecomment-68477281\",\"issue_url\":\"https://api.github.com/repos/artberri/sidr/issues/4\",\"id\":68477281,\"user\":{\"login\":\"aaroncrawford\",\"id\":3203500,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3203500?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aaroncrawford\",\"html_url\":\"https://github.com/aaroncrawford\",\"followers_url\":\"https://api.github.com/users/aaroncrawford/followers\",\"following_url\":\"https://api.github.com/users/aaroncrawford/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/aaroncrawford/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/aaroncrawford/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/aaroncrawford/subscriptions\",\"organizations_url\":\"https://api.github.com/users/aaroncrawford/orgs\",\"repos_url\":\"https://api.github.com/users/aaroncrawford/repos\",\"events_url\":\"https://api.github.com/users/aaroncrawford/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/aaroncrawford/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:15Z\",\"updated_at\":\"2015-01-01T01:03:15Z\",\"body\":\"This should be addressed - relying on jquery for animation doesn't invoke the GPU.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:15Z\"}\n{\"id\":\"2489397305\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1308363,\"login\":\"paymonp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paymonp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1308363?\"},\"repo\":{\"id\":28678242,\"name\":\"paymonp/forecast_wrapper\",\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"A wrapper to simplify Forecast API data, and add some extra functionality (Ex. determining current night/day status for a location).\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:18Z\"}\n{\"id\":\"2489397309\",\"type\":\"CreateEvent\",\"actor\":{\"id\":693815,\"login\":\"apburnes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apburnes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/693815?\"},\"repo\":{\"id\":28678243,\"name\":\"apburnes/sf_3D_elevation\",\"url\":\"https://api.github.com/repos/apburnes/sf_3D_elevation\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"STL file of SF based on 5ft contour points.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:18Z\"}\n{\"id\":\"2489397311\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":8508800,\"login\":\"cmp-202\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmp-202\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8508800?\"},\"repo\":{\"id\":23310272,\"name\":\"cmp-202/ssh2shell\",\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10\",\"labels_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/comments\",\"events_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/events\",\"html_url\":\"https://github.com/cmp-202/ssh2shell/issues/10\",\"id\":53071635,\"number\":10,\"title\":\"Timeout error (part 2)\",\"user\":{\"login\":\"macirex\",\"id\":1192393,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1192393?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/macirex\",\"html_url\":\"https://github.com/macirex\",\"followers_url\":\"https://api.github.com/users/macirex/followers\",\"following_url\":\"https://api.github.com/users/macirex/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/macirex/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/macirex/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/macirex/subscriptions\",\"organizations_url\":\"https://api.github.com/users/macirex/orgs\",\"repos_url\":\"https://api.github.com/users/macirex/repos\",\"events_url\":\"https://api.github.com/users/macirex/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/macirex/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":7,\"created_at\":\"2014-12-29T21:04:44Z\",\"updated_at\":\"2015-01-01T01:03:18Z\",\"closed_at\":null,\"body\":\"Hello cmp,\\r\\n\\r\\nI´m sorry to bother you again but I´ve encountered the another time out issue.\\r\\n\\r\\nLet me provide some context first:\\r\\n\\r\\nA group of servers I administer have some restricted policies:\\r\\n\\r\\n- Root and application domain users are inaccessible via ssh.\\r\\n- None of the application users have password.\\r\\n- There´s only one user available for ssh connection, it is used for everything and it has limited permissions, for references this will be called \\\"simpleuser\\\". \\r\\n- Only way to administer a server is to connect via ssh to the \\\"simpleuser\\\" and then do a su to root. \\r\\n\\r\\nBasically the flow goes like this: \\r\\nLogin -> su - root -> su - domainapp ->  [Here I do what I need to do]   \\r\\n\\r\\nThe list of commands I´m trying is:\\r\\n  commands:           [\\r\\n    \\\"su - root\\\", \\r\\n    \\\"echo hello\\\" \\r\\n    \\\"su - webapp\\\",\\r\\n    \\\"ls -lthr\\\",\\r\\n  ]\\r\\n\\r\\nI managed to log into root using the event on commandProcessing and writing the password directly into the stream:\\r\\n\\r\\n<code>\\r\\nSSH.on ('commandProcessing', function onCommandProcessing( command, response, sshObj, stream ) {   \\r\\n            if (command.match(/root/)&& response.indexOf(\\\"Password:\\\") != -1) {\\r\\n\\t           if (!admin)\\r\\n\\t\\t   {\\r\\n\\t\\t\\tstream.write('password\\\\n');\\r\\n\\t\\t\\tadmin = true;\\r\\n\\t\\t    }\\r\\n              });\\r\\n</code>\\r\\n\\r\\nAfter that, I try to do another su, this time to the domain I need to check but it fails with: \\r\\nTimeout error: 10.164.12.159: Command timed out after 5 seconds\\r\\n\\r\\nHere´s the output my app generates:\\r\\n\\r\\nConnected\\r\\nRunning commands Now\\r\\n10.164.12.159 verbose:Your password will expire in 8 days.\\r\\n[user01@machine043]>\\r\\n10.164.12.159 verbose:su - root\\r\\nPassword:\\r\\n[root@machine043]>\\r\\n10.164.12.159 verbose:echo hello\\r\\nhello\\r\\n[root@machine043]>\\r\\nTimeout error: 10.164.12.159: Command timed out after 5 seconds\\r\\nCompleted\\r\\n\\r\\nAny commands I execute before trying the second su, it works perfectly but after the second su I only get Timeout error.\\r\\n\\r\\nThis is the full session responses:\\r\\nConnected to 10.164.12.159\\r\\nsu - root\\r\\nPassword:\\r\\n[root@machine043]> echo hello\\r\\nhello\\r\\n[root@machine043]> su - webapp\\r\\nmachine043@webapp:~>\\r\\n\\r\\nAny insights you could share with me?\\r\\nCould this be happening because the name of the prompt is different from the first two?  Where simpleuser and root has username@machinename and webapp has machinename@username?\\r\\n\\r\\nBtw, merry christmas! and have a happy new year!\"},\"comment\":{\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/comments/68477282\",\"html_url\":\"https://github.com/cmp-202/ssh2shell/issues/10#issuecomment-68477282\",\"issue_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10\",\"id\":68477282,\"user\":{\"login\":\"cmp-202\",\"id\":8508800,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8508800?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmp-202\",\"html_url\":\"https://github.com/cmp-202\",\"followers_url\":\"https://api.github.com/users/cmp-202/followers\",\"following_url\":\"https://api.github.com/users/cmp-202/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cmp-202/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cmp-202/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cmp-202/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cmp-202/orgs\",\"repos_url\":\"https://api.github.com/users/cmp-202/repos\",\"events_url\":\"https://api.github.com/users/cmp-202/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cmp-202/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:18Z\",\"updated_at\":\"2015-01-01T01:03:18Z\",\"body\":\"you can increase the timeout value to allow for the scripts to run `host.idleTimeOut = 10000`\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:18Z\"}\n{\"id\":\"2489397317\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1398544,\"login\":\"joelpurra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joelpurra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1398544?\"},\"repo\":{\"id\":28595466,\"name\":\"joelpurra/jqnpm\",\"url\":\"https://api.github.com/repos/joelpurra/jqnpm\"},\"payload\":{\"ref\":\"v0.3.3\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"A package manager built for jq as an example implementation.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:19Z\"}\n{\"id\":\"2489397322\",\"type\":\"CreateEvent\",\"actor\":{\"id\":10263666,\"login\":\"katiekroik\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/katiekroik\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10263666?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"ref\":\"develop\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:20Z\"}\n{\"id\":\"2489397327\",\"type\":\"PushEvent\",\"actor\":{\"id\":2961036,\"login\":\"codemercenary\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codemercenary\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2961036?\"},\"repo\":{\"id\":22234733,\"name\":\"leapmotion/autowiring\",\"url\":\"https://api.github.com/repos/leapmotion/autowiring\"},\"payload\":{\"push_id\":536752898,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/fix-autoboost\",\"head\":\"3f0c9d9becf969aab74c7b4fe222854a6eebe58b\",\"before\":\"36b2b53a58e4725efd0cf1a9823175cd492ad044\",\"commits\":[{\"sha\":\"72c45ac5ebe38bb3581a26ebd6600b3769e6022c\",\"author\":{\"email\":\"2766fce9ce519d0471f8fc0cf2c841e5384821e2@gmail.com\",\"name\":\"Ted Nitz\"},\"message\":\"Merge pull request #313 from leapmotion/fix-cmakewarn\\n\\nEliminate unnecessary quoted variable expansions\",\"distinct\":false,\"url\":\"https://api.github.com/repos/leapmotion/autowiring/commits/72c45ac5ebe38bb3581a26ebd6600b3769e6022c\"},{\"sha\":\"3f0c9d9becf969aab74c7b4fe222854a6eebe58b\",\"author\":{\"email\":\"f4b726eb4428c94ec1239af9e3b06d73d64a2f9c@gmail.com\",\"name\":\"Jason Lokerson\"},\"message\":\"Rename boost to autoboost\\n\\nHuge modification, requiring adjustements to namespaces, macros, file names, and header guards.  This change introduces a private version of Boost 1.57.0 that can be linked externally on platforms which do not have full STL11 support without colliding in any way with those platforms' independent use of boost.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leapmotion/autowiring/commits/3f0c9d9becf969aab74c7b4fe222854a6eebe58b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:21Z\",\"org\":{\"id\":2242710,\"login\":\"leapmotion\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/leapmotion\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?\"}}\n{\"id\":\"2489397328\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":762599,\"login\":\"natedavis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/natedavis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/762599?\"},\"repo\":{\"id\":8348465,\"name\":\"BF2Statistics/ControlCenter\",\"url\":\"https://api.github.com/repos/BF2Statistics/ControlCenter\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/BF2Statistics/ControlCenter/issues/2\",\"labels_url\":\"https://api.github.com/repos/BF2Statistics/ControlCenter/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/BF2Statistics/ControlCenter/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/BF2Statistics/ControlCenter/issues/2/events\",\"html_url\":\"https://github.com/BF2Statistics/ControlCenter/issues/2\",\"id\":53210225,\"number\":2,\"title\":\"Launch Server invalid Path name\",\"user\":{\"login\":\"natedavis\",\"id\":762599,\"avatar_url\":\"https://avatars.githubusercontent.com/u/762599?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/natedavis\",\"html_url\":\"https://github.com/natedavis\",\"followers_url\":\"https://api.github.com/users/natedavis/followers\",\"following_url\":\"https://api.github.com/users/natedavis/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/natedavis/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/natedavis/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/natedavis/subscriptions\",\"organizations_url\":\"https://api.github.com/users/natedavis/orgs\",\"repos_url\":\"https://api.github.com/users/natedavis/repos\",\"events_url\":\"https://api.github.com/users/natedavis/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/natedavis/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:21Z\",\"updated_at\":\"2015-01-01T01:03:21Z\",\"closed_at\":null,\"body\":\"System: Server 2008 R2\\r\\n\\r\\nBF2 is installed:\\r\\nC:\\\\Program Files (x86)\\\\EA Games\\\\Battlefield 2 Server\\\\\\r\\nBF2Statistics is install:\\r\\nC:\\\\Program Files (x86)\\\\BF2Statistics\\\\BF2Statistics Control Center\\\\\\r\\n\\r\\nWhen I try and Launch the game with \\\"Use Global Settings File\\\" The Server will not launch with my settings.\\r\\n\\r\\nFrom the command line, I tested the following...\\r\\n\\r\\nPath Generated in MainForm.cs on line 457 - Which is not working:\\r\\nbf2_w32ded.exe +modPath mods/bf2 +config C:\\\\\\\\Program Files (x86)\\\\\\\\BF2Statistics\\\\\\\\BF2Statistics Control Center\\\\\\\\Python\\\\\\\\GlobalServerSettings.con\\r\\n\\r\\nIf I use the following to Launch the server manually it works great:\\r\\nbf2_w32ded.exe +modPath mods/bf2 +config \\\"C:\\\\Program Files (x86)\\\\BF2Statistics\\\\BF2Statistics Control Center\\\\Python\\\\GlobalServerSettings.con\\\"\\r\\n\\r\\nQuotes at the end and beginning with no escaping \\\\s.\\r\\n\\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:22Z\",\"org\":{\"id\":2574835,\"login\":\"BF2Statistics\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/BF2Statistics\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2574835?\"}}\n{\"id\":\"2489397337\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":447569,\"login\":\"tom-henderson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tom-henderson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/447569?\"},\"repo\":{\"id\":11580119,\"name\":\"joerick/django-timecode\",\"url\":\"https://api.github.com/repos/joerick/django-timecode\"},\"payload\":{\"action\":\"opened\",\"number\":1,\"pull_request\":{\"url\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1\",\"id\":26739431,\"html_url\":\"https://github.com/joerick/django-timecode/pull/1\",\"diff_url\":\"https://github.com/joerick/django-timecode/pull/1.diff\",\"patch_url\":\"https://github.com/joerick/django-timecode/pull/1.patch\",\"issue_url\":\"https://api.github.com/repos/joerick/django-timecode/issues/1\",\"number\":1,\"state\":\"open\",\"locked\":false,\"title\":\"Django jquery\",\"user\":{\"login\":\"tom-henderson\",\"id\":447569,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447569?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tom-henderson\",\"html_url\":\"https://github.com/tom-henderson\",\"followers_url\":\"https://api.github.com/users/tom-henderson/followers\",\"following_url\":\"https://api.github.com/users/tom-henderson/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tom-henderson/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tom-henderson/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tom-henderson/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tom-henderson/orgs\",\"repos_url\":\"https://api.github.com/users/tom-henderson/repos\",\"events_url\":\"https://api.github.com/users/tom-henderson/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tom-henderson/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Use jQuery from the Django namespace so timecode fields work in the django admin.\",\"created_at\":\"2015-01-01T01:03:22Z\",\"updated_at\":\"2015-01-01T01:03:23Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1/commits\",\"review_comments_url\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1/comments\",\"review_comment_url\":\"https://api.github.com/repos/joerick/django-timecode/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/joerick/django-timecode/issues/1/comments\",\"statuses_url\":\"https://api.github.com/repos/joerick/django-timecode/statuses/567af7c7dd6130eb9cea3c3f9065c0bc1d801df1\",\"head\":{\"label\":\"tom-henderson:django-jquery\",\"ref\":\"django-jquery\",\"sha\":\"567af7c7dd6130eb9cea3c3f9065c0bc1d801df1\",\"user\":{\"login\":\"tom-henderson\",\"id\":447569,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447569?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tom-henderson\",\"html_url\":\"https://github.com/tom-henderson\",\"followers_url\":\"https://api.github.com/users/tom-henderson/followers\",\"following_url\":\"https://api.github.com/users/tom-henderson/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tom-henderson/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tom-henderson/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tom-henderson/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tom-henderson/orgs\",\"repos_url\":\"https://api.github.com/users/tom-henderson/repos\",\"events_url\":\"https://api.github.com/users/tom-henderson/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tom-henderson/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28677632,\"name\":\"django-timecode\",\"full_name\":\"tom-henderson/django-timecode\",\"owner\":{\"login\":\"tom-henderson\",\"id\":447569,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447569?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tom-henderson\",\"html_url\":\"https://github.com/tom-henderson\",\"followers_url\":\"https://api.github.com/users/tom-henderson/followers\",\"following_url\":\"https://api.github.com/users/tom-henderson/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tom-henderson/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tom-henderson/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tom-henderson/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tom-henderson/orgs\",\"repos_url\":\"https://api.github.com/users/tom-henderson/repos\",\"events_url\":\"https://api.github.com/users/tom-henderson/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tom-henderson/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/tom-henderson/django-timecode\",\"description\":\"A python class for a timecode and accompanying django field\",\"fork\":true,\"url\":\"https://api.github.com/repos/tom-henderson/django-timecode\",\"forks_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/forks\",\"keys_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/teams\",\"hooks_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/hooks\",\"issue_events_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/events\",\"assignees_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/tags\",\"blobs_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/languages\",\"stargazers_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/stargazers\",\"contributors_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/contributors\",\"subscribers_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/subscribers\",\"subscription_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/subscription\",\"commits_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/merges\",\"archive_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/downloads\",\"issues_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/tom-henderson/django-timecode/releases{/id}\",\"created_at\":\"2015-01-01T00:07:05Z\",\"updated_at\":\"2015-01-01T00:07:06Z\",\"pushed_at\":\"2015-01-01T00:54:15Z\",\"git_url\":\"git://github.com/tom-henderson/django-timecode.git\",\"ssh_url\":\"git@github.com:tom-henderson/django-timecode.git\",\"clone_url\":\"https://github.com/tom-henderson/django-timecode.git\",\"svn_url\":\"https://github.com/tom-henderson/django-timecode\",\"homepage\":\"\",\"size\":156,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Python\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"joerick:master\",\"ref\":\"master\",\"sha\":\"737f138d8c3cbdb72e08419cf2d10dd1eb4e72d6\",\"user\":{\"login\":\"joerick\",\"id\":1244307,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1244307?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joerick\",\"html_url\":\"https://github.com/joerick\",\"followers_url\":\"https://api.github.com/users/joerick/followers\",\"following_url\":\"https://api.github.com/users/joerick/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joerick/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joerick/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joerick/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joerick/orgs\",\"repos_url\":\"https://api.github.com/users/joerick/repos\",\"events_url\":\"https://api.github.com/users/joerick/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joerick/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":11580119,\"name\":\"django-timecode\",\"full_name\":\"joerick/django-timecode\",\"owner\":{\"login\":\"joerick\",\"id\":1244307,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1244307?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joerick\",\"html_url\":\"https://github.com/joerick\",\"followers_url\":\"https://api.github.com/users/joerick/followers\",\"following_url\":\"https://api.github.com/users/joerick/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joerick/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joerick/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joerick/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joerick/orgs\",\"repos_url\":\"https://api.github.com/users/joerick/repos\",\"events_url\":\"https://api.github.com/users/joerick/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joerick/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/joerick/django-timecode\",\"description\":\"A python class for a timecode and accompanying django field\",\"fork\":false,\"url\":\"https://api.github.com/repos/joerick/django-timecode\",\"forks_url\":\"https://api.github.com/repos/joerick/django-timecode/forks\",\"keys_url\":\"https://api.github.com/repos/joerick/django-timecode/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/joerick/django-timecode/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/joerick/django-timecode/teams\",\"hooks_url\":\"https://api.github.com/repos/joerick/django-timecode/hooks\",\"issue_events_url\":\"https://api.github.com/repos/joerick/django-timecode/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/joerick/django-timecode/events\",\"assignees_url\":\"https://api.github.com/repos/joerick/django-timecode/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/joerick/django-timecode/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/joerick/django-timecode/tags\",\"blobs_url\":\"https://api.github.com/repos/joerick/django-timecode/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/joerick/django-timecode/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/joerick/django-timecode/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/joerick/django-timecode/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/joerick/django-timecode/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/joerick/django-timecode/languages\",\"stargazers_url\":\"https://api.github.com/repos/joerick/django-timecode/stargazers\",\"contributors_url\":\"https://api.github.com/repos/joerick/django-timecode/contributors\",\"subscribers_url\":\"https://api.github.com/repos/joerick/django-timecode/subscribers\",\"subscription_url\":\"https://api.github.com/repos/joerick/django-timecode/subscription\",\"commits_url\":\"https://api.github.com/repos/joerick/django-timecode/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/joerick/django-timecode/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/joerick/django-timecode/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/joerick/django-timecode/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/joerick/django-timecode/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/joerick/django-timecode/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/joerick/django-timecode/merges\",\"archive_url\":\"https://api.github.com/repos/joerick/django-timecode/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/joerick/django-timecode/downloads\",\"issues_url\":\"https://api.github.com/repos/joerick/django-timecode/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/joerick/django-timecode/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/joerick/django-timecode/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/joerick/django-timecode/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/joerick/django-timecode/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/joerick/django-timecode/releases{/id}\",\"created_at\":\"2013-07-22T11:50:44Z\",\"updated_at\":\"2014-09-15T16:05:53Z\",\"pushed_at\":\"2014-01-13T16:01:08Z\",\"git_url\":\"git://github.com/joerick/django-timecode.git\",\"ssh_url\":\"git@github.com:joerick/django-timecode.git\",\"clone_url\":\"https://github.com/joerick/django-timecode.git\",\"svn_url\":\"https://github.com/joerick/django-timecode\",\"homepage\":\"\",\"size\":156,\"stargazers_count\":2,\"watchers_count\":2,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":1,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":1,\"open_issues\":1,\"watchers\":2,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1\"},\"html\":{\"href\":\"https://github.com/joerick/django-timecode/pull/1\"},\"issue\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/issues/1\"},\"comments\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/issues/1/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/pulls/1/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/joerick/django-timecode/statuses/567af7c7dd6130eb9cea3c3f9065c0bc1d801df1\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":3,\"additions\":25,\"deletions\":23,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:03:23Z\"}\n{\"id\":\"2489397338\",\"type\":\"PushEvent\",\"actor\":{\"id\":6563665,\"login\":\"mikeberger\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mikeberger\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6563665?\"},\"repo\":{\"id\":16442206,\"name\":\"mikeberger/borg_calendar\",\"url\":\"https://api.github.com/repos/mikeberger/borg_calendar\"},\"payload\":{\"push_id\":536752904,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5a79ba420970577f91cc0677f413cbecc8be3e4d\",\"before\":\"14da87ea2655e8a729a0ec537720221cfc32852f\",\"commits\":[{\"sha\":\"5a79ba420970577f91cc0677f413cbecc8be3e4d\",\"author\":{\"email\":\"a17fed27eaa842282862ff7c1b9c8395a26ac320@mbcsoft.com\",\"name\":\"Mike Berger\"},\"message\":\"update category pulldown in appointment editors when the categories change. Also fix memory leak.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mikeberger/borg_calendar/commits/5a79ba420970577f91cc0677f413cbecc8be3e4d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:23Z\"}\n{\"id\":\"2489397348\",\"type\":\"PushEvent\",\"actor\":{\"id\":6325631,\"login\":\"pirej\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pirej\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6325631?\"},\"repo\":{\"id\":27978759,\"name\":\"lollipoop/android_device_sony_montblanc-common\",\"url\":\"https://api.github.com/repos/lollipoop/android_device_sony_montblanc-common\"},\"payload\":{\"push_id\":536752906,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/m4\",\"head\":\"4dea1e80ea185145d16f8a27b53c49004a82486a\",\"before\":\"81c8b8e24cebad126b22e7eee0e5ad490d0da7ee\",\"commits\":[{\"sha\":\"4dea1e80ea185145d16f8a27b53c49004a82486a\",\"author\":{\"email\":\"1286c54ccb8bc261f2349e0273c6fbd8e8811d8d@yahoo.com\",\"name\":\"pirej\"},\"message\":\"Update BoardConfigCommon.mk\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lollipoop/android_device_sony_montblanc-common/commits/4dea1e80ea185145d16f8a27b53c49004a82486a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:25Z\",\"org\":{\"id\":10051895,\"login\":\"lollipoop\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/lollipoop\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10051895?\"}}\n{\"id\":\"2489397352\",\"type\":\"PushEvent\",\"actor\":{\"id\":674513,\"login\":\"jonesde\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jonesde\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/674513?\"},\"repo\":{\"id\":23584277,\"name\":\"sssonline/mantle\",\"url\":\"https://api.github.com/repos/sssonline/mantle\"},\"payload\":{\"push_id\":536752908,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e09da87aef6c7dc586dea4fbc905b700a5a45766\",\"before\":\"d51d27b9328f9625588ab93bd9be509a8fa775bd\",\"commits\":[{\"sha\":\"e09da87aef6c7dc586dea4fbc905b700a5a45766\",\"author\":{\"email\":\"2c659b141cba72e98a27345389f1cac5e8612a49@dejc.com\",\"name\":\"David E Jones\"},\"message\":\"Changed AssetRegistration to have a sub-sequence key instead of using fromDate, may have multiple records with the same fromDate so it isn't a good key\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sssonline/mantle/commits/e09da87aef6c7dc586dea4fbc905b700a5a45766\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:26Z\"}\n{\"id\":\"2489397355\",\"type\":\"PushEvent\",\"actor\":{\"id\":803768,\"login\":\"backbone\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/backbone\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/803768?\"},\"repo\":{\"id\":24404721,\"name\":\"backbone/portage-tree\",\"url\":\"https://api.github.com/repos/backbone/portage-tree\"},\"payload\":{\"push_id\":536752911,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b1677769b2847c0a94f6f8015bffeed5d1a24204\",\"before\":\"dccafb540891dd450a885612cabf712d3129dbdf\",\"commits\":[{\"sha\":\"b1677769b2847c0a94f6f8015bffeed5d1a24204\",\"author\":{\"email\":\"4da38e79dbf743adc207e22b3829ee998325e896@backbone.ws\",\"name\":\"Kolan Sh\"},\"message\":\"Sync with portage [Thu Jan  1 04:03:20 MSK 2015].\",\"distinct\":true,\"url\":\"https://api.github.com/repos/backbone/portage-tree/commits/b1677769b2847c0a94f6f8015bffeed5d1a24204\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:26Z\"}\n{\"id\":\"2489397357\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536752912,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"96f7f02d9551bb3c922c9c6303b9d144e2c31398\",\"before\":\"74e68efacee23b35716fa9b2b46d0926e552250b\",\"commits\":[{\"sha\":\"96f7f02d9551bb3c922c9c6303b9d144e2c31398\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"load images\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/96f7f02d9551bb3c922c9c6303b9d144e2c31398\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:26Z\"}\n{\"id\":\"2489397358\",\"type\":\"PushEvent\",\"actor\":{\"id\":904370,\"login\":\"helhum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/helhum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/904370?\"},\"repo\":{\"id\":21070637,\"name\":\"TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools\",\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools\"},\"payload\":{\"push_id\":536752913,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e9f19e894cc8cb5777c9343c6bdb2fbf5e3f2945\",\"before\":\"03ce16ec5d355a60057fb57f8d27d7116422f722\",\"commits\":[{\"sha\":\"e9f19e894cc8cb5777c9343c6bdb2fbf5e3f2945\",\"author\":{\"email\":\"6bf857ca7de026fbed4ae790a809a0ea640901f4@helmuthummel.de\",\"name\":\"Helmut Hummel\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/TYPO3SurfCms.SurfTools/commits/e9f19e894cc8cb5777c9343c6bdb2fbf5e3f2945\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:27Z\",\"org\":{\"id\":7921669,\"login\":\"TYPO3-Surf-CMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TYPO3-Surf-CMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7921669?\"}}\n{\"id\":\"2489397369\",\"type\":\"PushEvent\",\"actor\":{\"id\":512573,\"login\":\"SamWhited\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SamWhited\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/512573?\"},\"repo\":{\"id\":27289261,\"name\":\"campaul/ph.sh\",\"url\":\"https://api.github.com/repos/campaul/ph.sh\"},\"payload\":{\"push_id\":536752919,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"688e49dcbc4acfb6b4b1871debaaa712f1f7410e\",\"before\":\"51db74f429e57b90e1cc9ca18551ca5e74b7db49\",\"commits\":[{\"sha\":\"688e49dcbc4acfb6b4b1871debaaa712f1f7410e\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@samwhited.com\",\"name\":\"Sam Whited\"},\"message\":\"Fix in-place import\",\"distinct\":true,\"url\":\"https://api.github.com/repos/campaul/ph.sh/commits/688e49dcbc4acfb6b4b1871debaaa712f1f7410e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:28Z\"}\n{\"id\":\"2489397370\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1059214,\"login\":\"wlaurance\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wlaurance\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1059214?\"},\"repo\":{\"id\":28050478,\"name\":\"empirical-org/Quill-Grammar\",\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar\"},\"payload\":{\"ref\":\"feature/teacher-form\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Quill Grammar App\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:28Z\",\"org\":{\"id\":4258432,\"login\":\"empirical-org\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/empirical-org\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4258432?\"}}\n{\"id\":\"2489397372\",\"type\":\"CreateEvent\",\"actor\":{\"id\":840120,\"login\":\"joaoalf\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joaoalf\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/840120?\"},\"repo\":{\"id\":21998771,\"name\":\"savoirfairelinux/partner-contact\",\"url\":\"https://api.github.com/repos/savoirfairelinux/partner-contact\"},\"payload\":{\"ref\":\"7.0_fix_function_partner_import_issue\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:28Z\",\"org\":{\"id\":2735545,\"login\":\"savoirfairelinux\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/savoirfairelinux\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2735545?\"}}\n{\"id\":\"2489397375\",\"type\":\"PushEvent\",\"actor\":{\"id\":8396076,\"login\":\"chris-kobrzak\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chris-kobrzak\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8396076?\"},\"repo\":{\"id\":28611088,\"name\":\"chris-kobrzak/natalia.dns4e.net\",\"url\":\"https://api.github.com/repos/chris-kobrzak/natalia.dns4e.net\"},\"payload\":{\"push_id\":536752923,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dbc61a3aeb6b13937d9b1345ca46f9965a2eb950\",\"before\":\"1e3a5af2be263253fdc3343d7bc23cf456058531\",\"commits\":[{\"sha\":\"dbc61a3aeb6b13937d9b1345ca46f9965a2eb950\",\"author\":{\"email\":\"33a9c8b3090a200dac7ed5d8b6d398372695ff06@gmail.com\",\"name\":\"Chris Kobrzak\"},\"message\":\"Initial version of Natalie's image gallery\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chris-kobrzak/natalia.dns4e.net/commits/dbc61a3aeb6b13937d9b1345ca46f9965a2eb950\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:29Z\"}\n{\"id\":\"2489397376\",\"type\":\"GollumEvent\",\"actor\":{\"id\":720678,\"login\":\"kongr45gpen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kongr45gpen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/720678?\"},\"repo\":{\"id\":11615971,\"name\":\"allejo/bzion\",\"url\":\"https://api.github.com/repos/allejo/bzion\"},\"payload\":{\"pages\":[{\"page_name\":\"Installation\",\"title\":\"Installation\",\"summary\":null,\"action\":\"edited\",\"sha\":\"a2a782b24c716c5e675ec38ccda1c620edeafbf1\",\"html_url\":\"https://github.com/allejo/bzion/wiki/Installation\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:29Z\"}\n{\"id\":\"2489397377\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1313396,\"login\":\"castvoid\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/castvoid\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1313396?\"},\"repo\":{\"id\":28132069,\"name\":\"adbsjb/Website\",\"url\":\"https://api.github.com/repos/adbsjb/Website\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/adbsjb/Website/issues/1\",\"labels_url\":\"https://api.github.com/repos/adbsjb/Website/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/adbsjb/Website/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/adbsjb/Website/issues/1/events\",\"html_url\":\"https://github.com/adbsjb/Website/issues/1\",\"id\":53210229,\"number\":1,\"title\":\"Lack of pictures of Gleb\",\"user\":{\"login\":\"castvoid\",\"id\":1313396,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1313396?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/castvoid\",\"html_url\":\"https://github.com/castvoid\",\"followers_url\":\"https://api.github.com/users/castvoid/followers\",\"following_url\":\"https://api.github.com/users/castvoid/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/castvoid/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/castvoid/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/castvoid/subscriptions\",\"organizations_url\":\"https://api.github.com/users/castvoid/orgs\",\"repos_url\":\"https://api.github.com/users/castvoid/repos\",\"events_url\":\"https://api.github.com/users/castvoid/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/castvoid/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:29Z\",\"updated_at\":\"2015-01-01T01:03:29Z\",\"closed_at\":null,\"body\":\"See title\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:29Z\"}\n{\"id\":\"2489397384\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536752925,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c085f224e47c8147e6085f897cb91a07834211b1\",\"before\":\"d3f1564c71b9e115435146f52c4c7c5771f819c7\",\"commits\":[{\"sha\":\"c085f224e47c8147e6085f897cb91a07834211b1\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074208651\\n\\nMuLnvTUe41hdRnmxxHnfDhkDkxsJfepcQfRosAA0g3I=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/c085f224e47c8147e6085f897cb91a07834211b1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:30Z\"}\n{\"id\":\"2489397385\",\"type\":\"PushEvent\",\"actor\":{\"id\":8908463,\"login\":\"ivankp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivankp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8908463?\"},\"repo\":{\"id\":26780993,\"name\":\"ivankp/bh_analysis\",\"url\":\"https://api.github.com/repos/ivankp/bh_analysis\"},\"payload\":{\"push_id\":536752926,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/testing\",\"head\":\"54b3cd7571257f9c38f4cfb5ad5fbb379419d827\",\"before\":\"1cd6e64b009254caf84067f3a836e3b1d5f98218\",\"commits\":[{\"sha\":\"54b3cd7571257f9c38f4cfb5ad5fbb379419d827\",\"author\":{\"email\":\"9c1affc1d2e05e5eafd0291f399cde138f8c80d5@gmail.com\",\"name\":\"ivankp\"},\"message\":\"reweigh scales\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ivankp/bh_analysis/commits/54b3cd7571257f9c38f4cfb5ad5fbb379419d827\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:30Z\"}\n{\"id\":\"2489397389\",\"type\":\"PushEvent\",\"actor\":{\"id\":5049892,\"login\":\"Brawl345\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Brawl345\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5049892?\"},\"repo\":{\"id\":19072701,\"name\":\"Brawl345/dotfiles\",\"url\":\"https://api.github.com/repos/Brawl345/dotfiles\"},\"payload\":{\"push_id\":536752927,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4c41a4d670beb420424b3e0c3d589614bf7307a7\",\"before\":\"a867a28418d4ca895f96fe9902914a1cd6ac56b9\",\"commits\":[{\"sha\":\"4c41a4d670beb420424b3e0c3d589614bf7307a7\",\"author\":{\"email\":\"3a3e021605a88f7e51d5787c2ebbb30bfb66608d@outlook.com\",\"name\":\"Andreas Bielawski\"},\"message\":\"changes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Brawl345/dotfiles/commits/4c41a4d670beb420424b3e0c3d589614bf7307a7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:31Z\"}\n{\"id\":\"2489397392\",\"type\":\"PushEvent\",\"actor\":{\"id\":512573,\"login\":\"SamWhited\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SamWhited\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/512573?\"},\"repo\":{\"id\":28072019,\"name\":\"SamWhited/ph.sh\",\"url\":\"https://api.github.com/repos/SamWhited/ph.sh\"},\"payload\":{\"push_id\":536752928,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"688e49dcbc4acfb6b4b1871debaaa712f1f7410e\",\"before\":\"51db74f429e57b90e1cc9ca18551ca5e74b7db49\",\"commits\":[{\"sha\":\"688e49dcbc4acfb6b4b1871debaaa712f1f7410e\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@samwhited.com\",\"name\":\"Sam Whited\"},\"message\":\"Fix in-place import\",\"distinct\":true,\"url\":\"https://api.github.com/repos/SamWhited/ph.sh/commits/688e49dcbc4acfb6b4b1871debaaa712f1f7410e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:31Z\"}\n{\"id\":\"2489397405\",\"type\":\"PushEvent\",\"actor\":{\"id\":6132327,\"login\":\"nboneh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nboneh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6132327?\"},\"repo\":{\"id\":25111641,\"name\":\"nboneh/RingSynth\",\"url\":\"https://api.github.com/repos/nboneh/RingSynth\"},\"payload\":{\"push_id\":536752932,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"880ddf590a2e7d0fbe3a536cd726ac4bf3979d6f\",\"before\":\"e8ac58f4157f0a86f6ca8f45fb79692b0160cc6f\",\"commits\":[{\"sha\":\"880ddf590a2e7d0fbe3a536cd726ac4bf3979d6f\",\"author\":{\"email\":\"158d4d4d94da2b7d5e7618d078746dcf02ab54d3@gmail.com\",\"name\":\"Nir Boneh\"},\"message\":\"Normal tunin around A4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nboneh/RingSynth/commits/880ddf590a2e7d0fbe3a536cd726ac4bf3979d6f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:33Z\"}\n{\"id\":\"2489397406\",\"type\":\"PushEvent\",\"actor\":{\"id\":4228183,\"login\":\"batchpause\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/batchpause\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4228183?\"},\"repo\":{\"id\":9607679,\"name\":\"batchpause/PAUSE-git\",\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git\"},\"payload\":{\"push_id\":536752933,\"size\":12,\"distinct_size\":12,\"ref\":\"refs/heads/master\",\"head\":\"03e53dd011cc8cc2c7054959bf5be8af53086102\",\"before\":\"6f0e51444646919a9f112644a693a93cc4896d18\",\"commits\":[{\"sha\":\"69f1e847fb31e24851a4e1f85e1a7c2f6a357a36\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420049821, pid 12413\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/69f1e847fb31e24851a4e1f85e1a7c2f6a357a36\"},{\"sha\":\"300581b6d8e97a15e79c78fcdc0553074d21e2b8\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420053421, pid 29312\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/300581b6d8e97a15e79c78fcdc0553074d21e2b8\"},{\"sha\":\"f914295d5506187cb5d5db6a5f7d049e6c3cc508\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420054141, pid 32521\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/f914295d5506187cb5d5db6a5f7d049e6c3cc508\"},{\"sha\":\"6edc4a6f42a7b97a735e12feb7de76d44c3eceb0\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420055521, pid 6611\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/6edc4a6f42a7b97a735e12feb7de76d44c3eceb0\"},{\"sha\":\"3d82f7f78b9120ae7212d87b03ef758b03521713\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420057741, pid 16831\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/3d82f7f78b9120ae7212d87b03ef758b03521713\"},{\"sha\":\"1966ca6fb94b2f5a2f3496393844fed222c9539e\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420059121, pid 23302\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/1966ca6fb94b2f5a2f3496393844fed222c9539e\"},{\"sha\":\"ddb77db4360f2a0d022f6db25f11a1f200e941b5\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420060621, pid 30382\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/ddb77db4360f2a0d022f6db25f11a1f200e941b5\"},{\"sha\":\"9c063587abbc1d241ed2d95baf137018ede84b3a\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420065661, pid 21556\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/9c063587abbc1d241ed2d95baf137018ede84b3a\"},{\"sha\":\"e4ce769b39fbf9a0d3c9ad3452759744ee123375\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420066321, pid 24279\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/e4ce769b39fbf9a0d3c9ad3452759744ee123375\"},{\"sha\":\"5f7fca1f705bbbec76e25b603aff759186946c3d\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420067821, pid 31331\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/5f7fca1f705bbbec76e25b603aff759186946c3d\"},{\"sha\":\"d19321137942b348af0d16c79207c601c19fdeda\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420071421, pid 15880\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/d19321137942b348af0d16c79207c601c19fdeda\"},{\"sha\":\"03e53dd011cc8cc2c7054959bf5be8af53086102\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@pause2.develooper.com\",\"name\":\"root\"},\"message\":\"indexer run at 1420072141, pid 19132\",\"distinct\":true,\"url\":\"https://api.github.com/repos/batchpause/PAUSE-git/commits/03e53dd011cc8cc2c7054959bf5be8af53086102\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:33Z\"}\n{\"id\":\"2489397411\",\"type\":\"PushEvent\",\"actor\":{\"id\":6911419,\"login\":\"himynameisdom\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/himynameisdom\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6911419?\"},\"repo\":{\"id\":28657371,\"name\":\"himynameisdom/latte\",\"url\":\"https://api.github.com/repos/himynameisdom/latte\"},\"payload\":{\"push_id\":536752935,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c938cdd2820f9df06645b463b9b239e94f160504\",\"before\":\"1a6d4186c93cdd68fe4af3ef75ac245c9ba9ee2e\",\"commits\":[{\"sha\":\"c938cdd2820f9df06645b463b9b239e94f160504\",\"author\":{\"email\":\"8d86c1bc9ce7bebe830a6a6d940e2e017bda9d78@me.com\",\"name\":\"Dominic Michalec\"},\"message\":\"added bootstrap javascript and nav collapse\",\"distinct\":true,\"url\":\"https://api.github.com/repos/himynameisdom/latte/commits/c938cdd2820f9df06645b463b9b239e94f160504\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:33Z\"}\n{\"id\":\"2489397412\",\"type\":\"PushEvent\",\"actor\":{\"id\":5544031,\"login\":\"at1as\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/at1as\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5544031?\"},\"repo\":{\"id\":23708117,\"name\":\"at1as/IMDB-Scrape\",\"url\":\"https://api.github.com/repos/at1as/IMDB-Scrape\"},\"payload\":{\"push_id\":536752937,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6e651be941874f6469b13a0e96f4ae7ce7b5e1c4\",\"before\":\"e47021c22914dab6dd09ed5d700dbf07b31c3462\",\"commits\":[{\"sha\":\"6e651be941874f6469b13a0e96f4ae7ce7b5e1c4\",\"author\":{\"email\":\"68c46a606457643eab92053c1c05574abb26f861@willems.ca\",\"name\":\"Jason\"},\"message\":\"Search by title string, no link\",\"distinct\":true,\"url\":\"https://api.github.com/repos/at1as/IMDB-Scrape/commits/6e651be941874f6469b13a0e96f4ae7ce7b5e1c4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:33Z\"}\n{\"id\":\"2489397414\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1865407,\"login\":\"slacktracer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/slacktracer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1865407?\"},\"repo\":{\"id\":26742634,\"name\":\"auchenberg/chrome-devtools-app\",\"url\":\"https://api.github.com/repos/auchenberg/chrome-devtools-app\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:34Z\"}\n{\"id\":\"2489397422\",\"type\":\"PushEvent\",\"actor\":{\"id\":4760248,\"login\":\"hxkclan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hxkclan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4760248?\"},\"repo\":{\"id\":28668307,\"name\":\"hxkclan/jade-for-wordpress\",\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress\"},\"payload\":{\"push_id\":536752941,\"size\":9,\"distinct_size\":9,\"ref\":\"refs/heads/develop\",\"head\":\"3a45dfdf3aa479fb0f27f66033f22331f04eec58\",\"before\":\"145fefffb0df917bab1db9838d9e5089c1c24ca6\",\"commits\":[{\"sha\":\"30c9c73c6ad9e1425fc140269e186bd8a0340d50\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"add body class from wp\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/30c9c73c6ad9e1425fc140269e186bd8a0340d50\"},{\"sha\":\"451a600ef7231036ce55eaf1b966031fcf2dabd7\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"fix alignment stuff\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/451a600ef7231036ce55eaf1b966031fcf2dabd7\"},{\"sha\":\"e8fd1b20152923d0175867c7ec3d2981df4f122b\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"add sidebar + container inside class\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/e8fd1b20152923d0175867c7ec3d2981df4f122b\"},{\"sha\":\"9deb4be8fca26bab9e71ae32bf83b07e6ad63169\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"added sidebar\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/9deb4be8fca26bab9e71ae32bf83b07e6ad63169\"},{\"sha\":\"638a2eadaafdba2f2dbd9fbde2a082b5219d98e8\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"added sidebar\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/638a2eadaafdba2f2dbd9fbde2a082b5219d98e8\"},{\"sha\":\"25d67ef22a56b2d7e22ed6ad885ad57416e76914\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"added sidebar to index\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/25d67ef22a56b2d7e22ed6ad885ad57416e76914\"},{\"sha\":\"0879c07121649e04b0b946c109d8c50cce4f5230\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"seperate edit post and bottom of post\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/0879c07121649e04b0b946c109d8c50cce4f5230\"},{\"sha\":\"000847605ea5bcda1640d6956a3e79c9924b5b78\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"corrected mistake with sidebar + backtotop\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/000847605ea5bcda1640d6956a3e79c9924b5b78\"},{\"sha\":\"3a45dfdf3aa479fb0f27f66033f22331f04eec58\",\"author\":{\"email\":\"3212f40cb710c917428034396bc51225b7378866@gmail.com\",\"name\":\"Michael Boumann\"},\"message\":\"fix in pages setup\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxkclan/jade-for-wordpress/commits/3a45dfdf3aa479fb0f27f66033f22331f04eec58\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:36Z\"}\n{\"id\":\"2489397425\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":416564,\"login\":\"johnryan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/johnryan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/416564?\"},\"repo\":{\"id\":9185941,\"name\":\"segmentio/analytics-android\",\"url\":\"https://api.github.com/repos/segmentio/analytics-android\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/segmentio/analytics-android/issues/153\",\"labels_url\":\"https://api.github.com/repos/segmentio/analytics-android/issues/153/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/segmentio/analytics-android/issues/153/comments\",\"events_url\":\"https://api.github.com/repos/segmentio/analytics-android/issues/153/events\",\"html_url\":\"https://github.com/segmentio/analytics-android/issues/153\",\"id\":53210234,\"number\":153,\"title\":\"Use different api keys for debug/release\",\"user\":{\"login\":\"johnryan\",\"id\":416564,\"avatar_url\":\"https://avatars.githubusercontent.com/u/416564?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/johnryan\",\"html_url\":\"https://github.com/johnryan\",\"followers_url\":\"https://api.github.com/users/johnryan/followers\",\"following_url\":\"https://api.github.com/users/johnryan/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/johnryan/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/johnryan/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/johnryan/subscriptions\",\"organizations_url\":\"https://api.github.com/users/johnryan/orgs\",\"repos_url\":\"https://api.github.com/users/johnryan/repos\",\"events_url\":\"https://api.github.com/users/johnryan/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/johnryan/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:36Z\",\"updated_at\":\"2015-01-01T01:03:36Z\",\"closed_at\":null,\"body\":\"Is there any way to set a different api key for debug vs release? This seems impossible since the key is set from the analytics.xml file.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:36Z\",\"org\":{\"id\":819518,\"login\":\"segmentio\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/segmentio\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/819518?\"}}\n{\"id\":\"2489397426\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":4275617,\"login\":\"i2amroy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/i2amroy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4275617?\"},\"repo\":{\"id\":5973855,\"name\":\"CleverRaven/Cataclysm-DDA\",\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10655\",\"labels_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10655/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10655/comments\",\"events_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10655/events\",\"html_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/issues/10655\",\"id\":52968085,\"number\":10655,\"title\":\"[WIP] Medical Updates\",\"user\":{\"login\":\"DavidKeaton\",\"id\":1715605,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1715605?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DavidKeaton\",\"html_url\":\"https://github.com/DavidKeaton\",\"followers_url\":\"https://api.github.com/users/DavidKeaton/followers\",\"following_url\":\"https://api.github.com/users/DavidKeaton/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DavidKeaton/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DavidKeaton/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DavidKeaton/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DavidKeaton/orgs\",\"repos_url\":\"https://api.github.com/users/DavidKeaton/repos\",\"events_url\":\"https://api.github.com/users/DavidKeaton/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DavidKeaton/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":14,\"created_at\":\"2014-12-27T23:00:10Z\",\"updated_at\":\"2015-01-01T01:03:36Z\",\"closed_at\":null,\"body\":\"### Intro\\r\\nOkay, so I closed the PR and reopened this as an issue. Some comments on @narc0tiq 's thread about scrapes and bruises sort of gave me some ideas where I want this to go.\\r\\n(beep boop calling @KA101 because `u r mad dope son`)\\r\\n\\r\\nAny and ALL comments welcome!\\r\\n\\r\\n### Briefing\\r\\nMy first notion is to abstract the notion of underlying anatomical 'systems' into a class that can be derived from. In said class, any arbitrary number of arbitrary `groups` of `anatomical_parts` can be together. So if you have 2 legs, 2 hands, 2 arms, a sweet eye patch, a hangover, and no dignity; most of these can theoretically be listed in this class. I have setup the very basics of it, along with a notion of design (a modicum I assure you, not an overzealous person or anything). I plan on migrating to JSON loadable files so these 'parts' and 'groups' can be super *dope* and expanded upon easily. Each `anatomical_part` falls into one or more `anatomical_group`s . These groups I hope to be expandable with the help of either/both Lua or JSON, or black magic I ain't judgin'. \\r\\n\\r\\nSo parts fall into groups, groups can provide flags to what these parts do. For an example of the slice of my code thus far...\\r\\n\\r\\n```\\r\\nenum anatomical_function {\\r\\n    // motor functions\\r\\n    AF_AMBULATION = 0,                          /* orthostasis, ambulating, etc */\\r\\n    // five senses\\r\\n    AF_OCULAR,                                  /* sight */\\r\\n    AF_OLFACTORY,                               /* smell */\\r\\n    AF_AUDITORY,                                /* hearing */\\r\\n    AF_GUSTATORY,                               /* taste */\\r\\n    AF_TACTILE,                                 /* touch */\\r\\n    // circulatory\\r\\n    AF_RESPIRATORY,                             /* breathing */\\r\\n    AF_NEUROVASCULAR,                           /* nervous [+ vascular] system */\\r\\n    AF_CARDIOVASCULAR,                          /* heart & vascular system */\\r\\n    // chassis, engine, and driver\\r\\n    AF_SKELETAL,                                /* bones */\\r\\n    AF_MUSCULAR,                                /* as it says */\\r\\n    AF_NEUROLOGICAL,                            /* brain/nervous system */\\r\\n    // non-humanoid functions\\r\\n    AF_ANTENNAE,                                /* sensory perception via antennae */\\r\\n    num_af\\r\\n};\\r\\n```\\r\\n\\r\\nSo with these `functions` lumped together with both `parts` and `groups`, you could have specific effects or diseases target these systems. \\r\\n\\r\\nOR MAYBE I AM OVERTHINKING THE `functions` PART A BIT, I DUNNO, BRAINSTORMING AND STUFF.\\r\\n\\r\\nSo what this roundabouts would look like further implemented, is things you have to watch for. Do you want to go into v-tach? *Nah dawg, you don't. So stop sniffing coke son.*\\r\\n\\r\\nBeen overdoing the stimulants for a fair amount, and have taken some cardiac damage? *You gun die son.*\\r\\n\\r\\nOh man, you have gone and broke that finger. *Guess you can't blap blap now.*\\r\\n\\r\\nHappen to have cut your fingertip too much with a knife, guess you goin'ta have some scar tissue son, *guess you are.*\\r\\n\\r\\nOkay so maybe it won't get _that_ crazy, but I would like a decently dynamic medical system, so it is more than just \\\"chug vitamins, don't do drugs, use first aid kit, done.\\\" Broken bones should be a reality before arm health reaches zero, more limbs and appendages should be a thing _anyway_ (hell maybe we could spin that off for mutated `anatomy`)! *GASP SON*\\r\\n\\r\\n# EAST COAST\\r\\n\\r\\n### TODO\\r\\n- [ ] Implement `anatomy` for creatures/critters/players/space-monsters/etc.\\r\\n- [ ] JSONify the `anatomy` aspects a bit, so as to make adding new `parts` and whatnot is legit as hell.\\r\\n- [ ] Implement new UI to better fit how *baller* this update will be.\\r\\n- [x] Realize that I am mad baller and fly as hell. Damn you sexy. \\r\\n- [ ] ~~Play way too much Cataclysm~~ Implement vital signs (BP, MAP, O2%)\\r\\n\\r\\n<bountysource-plugin>\\r\\n\\r\\n---\\r\\nWant to back this issue? **[Place a bounty on it!](https://www.bountysource.com/issues/7315679-wip-medical-updates?utm_campaign=plugin&utm_content=tracker%2F146201&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F146201&utm_medium=issues&utm_source=github).\\r\\n</bountysource-plugin>\"},\"comment\":{\"url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/comments/68477285\",\"html_url\":\"https://github.com/CleverRaven/Cataclysm-DDA/issues/10655#issuecomment-68477285\",\"issue_url\":\"https://api.github.com/repos/CleverRaven/Cataclysm-DDA/issues/10655\",\"id\":68477285,\"user\":{\"login\":\"i2amroy\",\"id\":4275617,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4275617?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/i2amroy\",\"html_url\":\"https://github.com/i2amroy\",\"followers_url\":\"https://api.github.com/users/i2amroy/followers\",\"following_url\":\"https://api.github.com/users/i2amroy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/i2amroy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/i2amroy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/i2amroy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/i2amroy/orgs\",\"repos_url\":\"https://api.github.com/users/i2amroy/repos\",\"events_url\":\"https://api.github.com/users/i2amroy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/i2amroy/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:36Z\",\"updated_at\":\"2015-01-01T01:03:36Z\",\"body\":\"I think a lot of the feasibility of this design comes down to the nitty-gritty exact implementation level. If it can be implemented in a way that is easy for the players to see, easy for the modders to handle in json, and easy for future developers to understand then I don't see a problem with implementing something like this. It just happens that it's a lot of work and a all three of those facts depend on your exact implementation.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:36Z\",\"org\":{\"id\":4367009,\"login\":\"CleverRaven\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/CleverRaven\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4367009?\"}}\n{\"id\":\"2489397429\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":2961036,\"login\":\"codemercenary\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codemercenary\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2961036?\"},\"repo\":{\"id\":22234733,\"name\":\"leapmotion/autowiring\",\"url\":\"https://api.github.com/repos/leapmotion/autowiring\"},\"payload\":{\"action\":\"opened\",\"number\":314,\"pull_request\":{\"url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314\",\"id\":26739436,\"html_url\":\"https://github.com/leapmotion/autowiring/pull/314\",\"diff_url\":\"https://github.com/leapmotion/autowiring/pull/314.diff\",\"patch_url\":\"https://github.com/leapmotion/autowiring/pull/314.patch\",\"issue_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/314\",\"number\":314,\"state\":\"open\",\"locked\":false,\"title\":\"Rename boost to autoboost\",\"user\":{\"login\":\"codemercenary\",\"id\":2961036,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2961036?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codemercenary\",\"html_url\":\"https://github.com/codemercenary\",\"followers_url\":\"https://api.github.com/users/codemercenary/followers\",\"following_url\":\"https://api.github.com/users/codemercenary/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/codemercenary/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/codemercenary/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/codemercenary/subscriptions\",\"organizations_url\":\"https://api.github.com/users/codemercenary/orgs\",\"repos_url\":\"https://api.github.com/users/codemercenary/repos\",\"events_url\":\"https://api.github.com/users/codemercenary/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/codemercenary/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Huge modification, requiring adjustements to namespaces, macros, file names, and header guards.  This change introduces a private version of Boost 1.57.0 that can be linked externally on platforms which do not have full STL11 support without colliding in any way with those platforms' independent use of boost.\",\"created_at\":\"2015-01-01T01:03:37Z\",\"updated_at\":\"2015-01-01T01:03:37Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314/commits\",\"review_comments_url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314/comments\",\"review_comment_url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/314/comments\",\"statuses_url\":\"https://api.github.com/repos/leapmotion/autowiring/statuses/3f0c9d9becf969aab74c7b4fe222854a6eebe58b\",\"head\":{\"label\":\"leapmotion:fix-autoboost\",\"ref\":\"fix-autoboost\",\"sha\":\"3f0c9d9becf969aab74c7b4fe222854a6eebe58b\",\"user\":{\"login\":\"leapmotion\",\"id\":2242710,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leapmotion\",\"html_url\":\"https://github.com/leapmotion\",\"followers_url\":\"https://api.github.com/users/leapmotion/followers\",\"following_url\":\"https://api.github.com/users/leapmotion/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/leapmotion/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/leapmotion/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/leapmotion/subscriptions\",\"organizations_url\":\"https://api.github.com/users/leapmotion/orgs\",\"repos_url\":\"https://api.github.com/users/leapmotion/repos\",\"events_url\":\"https://api.github.com/users/leapmotion/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/leapmotion/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":22234733,\"name\":\"autowiring\",\"full_name\":\"leapmotion/autowiring\",\"owner\":{\"login\":\"leapmotion\",\"id\":2242710,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leapmotion\",\"html_url\":\"https://github.com/leapmotion\",\"followers_url\":\"https://api.github.com/users/leapmotion/followers\",\"following_url\":\"https://api.github.com/users/leapmotion/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/leapmotion/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/leapmotion/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/leapmotion/subscriptions\",\"organizations_url\":\"https://api.github.com/users/leapmotion/orgs\",\"repos_url\":\"https://api.github.com/users/leapmotion/repos\",\"events_url\":\"https://api.github.com/users/leapmotion/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/leapmotion/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/leapmotion/autowiring\",\"description\":\"A C++ Inversion of Control Framework\",\"fork\":false,\"url\":\"https://api.github.com/repos/leapmotion/autowiring\",\"forks_url\":\"https://api.github.com/repos/leapmotion/autowiring/forks\",\"keys_url\":\"https://api.github.com/repos/leapmotion/autowiring/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/leapmotion/autowiring/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/leapmotion/autowiring/teams\",\"hooks_url\":\"https://api.github.com/repos/leapmotion/autowiring/hooks\",\"issue_events_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/leapmotion/autowiring/events\",\"assignees_url\":\"https://api.github.com/repos/leapmotion/autowiring/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/leapmotion/autowiring/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/leapmotion/autowiring/tags\",\"blobs_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/leapmotion/autowiring/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/leapmotion/autowiring/languages\",\"stargazers_url\":\"https://api.github.com/repos/leapmotion/autowiring/stargazers\",\"contributors_url\":\"https://api.github.com/repos/leapmotion/autowiring/contributors\",\"subscribers_url\":\"https://api.github.com/repos/leapmotion/autowiring/subscribers\",\"subscription_url\":\"https://api.github.com/repos/leapmotion/autowiring/subscription\",\"commits_url\":\"https://api.github.com/repos/leapmotion/autowiring/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/leapmotion/autowiring/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/leapmotion/autowiring/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/leapmotion/autowiring/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/leapmotion/autowiring/merges\",\"archive_url\":\"https://api.github.com/repos/leapmotion/autowiring/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/leapmotion/autowiring/downloads\",\"issues_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/leapmotion/autowiring/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/leapmotion/autowiring/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/leapmotion/autowiring/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/leapmotion/autowiring/releases{/id}\",\"created_at\":\"2014-07-24T22:39:49Z\",\"updated_at\":\"2014-12-30T23:40:39Z\",\"pushed_at\":\"2015-01-01T01:03:20Z\",\"git_url\":\"git://github.com/leapmotion/autowiring.git\",\"ssh_url\":\"git@github.com:leapmotion/autowiring.git\",\"clone_url\":\"https://github.com/leapmotion/autowiring.git\",\"svn_url\":\"https://github.com/leapmotion/autowiring\",\"homepage\":\"http://autowiring.io/\",\"size\":27882,\"stargazers_count\":68,\"watchers_count\":68,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":7,\"mirror_url\":null,\"open_issues_count\":2,\"forks\":7,\"open_issues\":2,\"watchers\":68,\"default_branch\":\"develop\"}},\"base\":{\"label\":\"leapmotion:develop\",\"ref\":\"develop\",\"sha\":\"72c45ac5ebe38bb3581a26ebd6600b3769e6022c\",\"user\":{\"login\":\"leapmotion\",\"id\":2242710,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leapmotion\",\"html_url\":\"https://github.com/leapmotion\",\"followers_url\":\"https://api.github.com/users/leapmotion/followers\",\"following_url\":\"https://api.github.com/users/leapmotion/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/leapmotion/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/leapmotion/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/leapmotion/subscriptions\",\"organizations_url\":\"https://api.github.com/users/leapmotion/orgs\",\"repos_url\":\"https://api.github.com/users/leapmotion/repos\",\"events_url\":\"https://api.github.com/users/leapmotion/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/leapmotion/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":22234733,\"name\":\"autowiring\",\"full_name\":\"leapmotion/autowiring\",\"owner\":{\"login\":\"leapmotion\",\"id\":2242710,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leapmotion\",\"html_url\":\"https://github.com/leapmotion\",\"followers_url\":\"https://api.github.com/users/leapmotion/followers\",\"following_url\":\"https://api.github.com/users/leapmotion/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/leapmotion/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/leapmotion/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/leapmotion/subscriptions\",\"organizations_url\":\"https://api.github.com/users/leapmotion/orgs\",\"repos_url\":\"https://api.github.com/users/leapmotion/repos\",\"events_url\":\"https://api.github.com/users/leapmotion/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/leapmotion/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/leapmotion/autowiring\",\"description\":\"A C++ Inversion of Control Framework\",\"fork\":false,\"url\":\"https://api.github.com/repos/leapmotion/autowiring\",\"forks_url\":\"https://api.github.com/repos/leapmotion/autowiring/forks\",\"keys_url\":\"https://api.github.com/repos/leapmotion/autowiring/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/leapmotion/autowiring/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/leapmotion/autowiring/teams\",\"hooks_url\":\"https://api.github.com/repos/leapmotion/autowiring/hooks\",\"issue_events_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/leapmotion/autowiring/events\",\"assignees_url\":\"https://api.github.com/repos/leapmotion/autowiring/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/leapmotion/autowiring/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/leapmotion/autowiring/tags\",\"blobs_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/leapmotion/autowiring/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/leapmotion/autowiring/languages\",\"stargazers_url\":\"https://api.github.com/repos/leapmotion/autowiring/stargazers\",\"contributors_url\":\"https://api.github.com/repos/leapmotion/autowiring/contributors\",\"subscribers_url\":\"https://api.github.com/repos/leapmotion/autowiring/subscribers\",\"subscription_url\":\"https://api.github.com/repos/leapmotion/autowiring/subscription\",\"commits_url\":\"https://api.github.com/repos/leapmotion/autowiring/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/leapmotion/autowiring/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/leapmotion/autowiring/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/leapmotion/autowiring/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/leapmotion/autowiring/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/leapmotion/autowiring/merges\",\"archive_url\":\"https://api.github.com/repos/leapmotion/autowiring/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/leapmotion/autowiring/downloads\",\"issues_url\":\"https://api.github.com/repos/leapmotion/autowiring/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/leapmotion/autowiring/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/leapmotion/autowiring/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/leapmotion/autowiring/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/leapmotion/autowiring/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/leapmotion/autowiring/releases{/id}\",\"created_at\":\"2014-07-24T22:39:49Z\",\"updated_at\":\"2014-12-30T23:40:39Z\",\"pushed_at\":\"2015-01-01T01:03:20Z\",\"git_url\":\"git://github.com/leapmotion/autowiring.git\",\"ssh_url\":\"git@github.com:leapmotion/autowiring.git\",\"clone_url\":\"https://github.com/leapmotion/autowiring.git\",\"svn_url\":\"https://github.com/leapmotion/autowiring\",\"homepage\":\"http://autowiring.io/\",\"size\":27882,\"stargazers_count\":68,\"watchers_count\":68,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":7,\"mirror_url\":null,\"open_issues_count\":2,\"forks\":7,\"open_issues\":2,\"watchers\":68,\"default_branch\":\"develop\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314\"},\"html\":{\"href\":\"https://github.com/leapmotion/autowiring/pull/314\"},\"issue\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/issues/314\"},\"comments\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/issues/314/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/pulls/314/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/leapmotion/autowiring/statuses/3f0c9d9becf969aab74c7b4fe222854a6eebe58b\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":29030,\"deletions\":145,\"changed_files\":6453}},\"public\":true,\"created_at\":\"2015-01-01T01:03:37Z\",\"org\":{\"id\":2242710,\"login\":\"leapmotion\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/leapmotion\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2242710?\"}}\n{\"id\":\"2489397434\",\"type\":\"PushEvent\",\"actor\":{\"id\":979046,\"login\":\"pippijn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pippijn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/979046?\"},\"repo\":{\"id\":27788257,\"name\":\"pippijn/tox4j\",\"url\":\"https://api.github.com/repos/pippijn/tox4j\"},\"payload\":{\"push_id\":536752943,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"e6e54153e3d800e5f6c7972f9dfa78a17efe76d5\",\"before\":\"48e0b41044a58a4f896d07ad29e65e1ac63e6010\",\"commits\":[{\"sha\":\"8d429c6a4ef8d5596c660fb307a58d9904486149\",\"author\":{\"email\":\"7113361f3b566fbf12c30d00f1a3671d2dbf9616@gmail.com\",\"name\":\"Pippijn van Steenhoven\"},\"message\":\"Generate more cmake code => less redundancy.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pippijn/tox4j/commits/8d429c6a4ef8d5596c660fb307a58d9904486149\"},{\"sha\":\"388e614c9032e14462f9e1fe77fab86310c2c1e6\",\"author\":{\"email\":\"7113361f3b566fbf12c30d00f1a3671d2dbf9616@gmail.com\",\"name\":\"Pippijn van Steenhoven\"},\"message\":\"Added tool to update tox4j version to current toxcore version.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pippijn/tox4j/commits/388e614c9032e14462f9e1fe77fab86310c2c1e6\"},{\"sha\":\"e6e54153e3d800e5f6c7972f9dfa78a17efe76d5\",\"author\":{\"email\":\"7113361f3b566fbf12c30d00f1a3671d2dbf9616@gmail.com\",\"name\":\"Pippijn van Steenhoven\"},\"message\":\"Give RepeatedLanDiscoveryTest a correct main() signature.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pippijn/tox4j/commits/e6e54153e3d800e5f6c7972f9dfa78a17efe76d5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:37Z\"}\n{\"id\":\"2489397436\",\"type\":\"PushEvent\",\"actor\":{\"id\":10225575,\"login\":\"ExclusiveOrange\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ExclusiveOrange\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10225575?\"},\"repo\":{\"id\":28677579,\"name\":\"ExclusiveOrange/synthesizer\",\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer\"},\"payload\":{\"push_id\":536752944,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"before\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:03:37Z\"}\n{\"id\":\"2489397439\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":8562199,\"login\":\"sguan-actuate\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sguan-actuate\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8562199?\"},\"repo\":{\"id\":19821157,\"name\":\"eclipse/birt\",\"url\":\"https://api.github.com/repos/eclipse/birt\"},\"payload\":{\"action\":\"opened\",\"number\":70,\"pull_request\":{\"url\":\"https://api.github.com/repos/eclipse/birt/pulls/70\",\"id\":26739437,\"html_url\":\"https://github.com/eclipse/birt/pull/70\",\"diff_url\":\"https://github.com/eclipse/birt/pull/70.diff\",\"patch_url\":\"https://github.com/eclipse/birt/pull/70.patch\",\"issue_url\":\"https://api.github.com/repos/eclipse/birt/issues/70\",\"number\":70,\"state\":\"open\",\"locked\":false,\"title\":\"Internal refactor apis and document code\",\"user\":{\"login\":\"sguan-actuate\",\"id\":8562199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8562199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sguan-actuate\",\"html_url\":\"https://github.com/sguan-actuate\",\"followers_url\":\"https://api.github.com/users/sguan-actuate/followers\",\"following_url\":\"https://api.github.com/users/sguan-actuate/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sguan-actuate/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sguan-actuate/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sguan-actuate/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sguan-actuate/orgs\",\"repos_url\":\"https://api.github.com/users/sguan-actuate/repos\",\"events_url\":\"https://api.github.com/users/sguan-actuate/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sguan-actuate/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Internal refactor apis to avoid IQueryDefinition.setAutoBinding \\r\\nand comment on code\\r\\n\\r\\nSigned-off-by: sguan <sguan@actuate.com>\",\"created_at\":\"2015-01-01T01:03:38Z\",\"updated_at\":\"2015-01-01T01:03:38Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/eclipse/birt/pulls/70/commits\",\"review_comments_url\":\"https://api.github.com/repos/eclipse/birt/pulls/70/comments\",\"review_comment_url\":\"https://api.github.com/repos/eclipse/birt/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/eclipse/birt/issues/70/comments\",\"statuses_url\":\"https://api.github.com/repos/eclipse/birt/statuses/7efab4d3fe077789a3e65be0fcd80a6078557360\",\"head\":{\"label\":\"sguan-actuate:master\",\"ref\":\"master\",\"sha\":\"7efab4d3fe077789a3e65be0fcd80a6078557360\",\"user\":{\"login\":\"sguan-actuate\",\"id\":8562199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8562199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sguan-actuate\",\"html_url\":\"https://github.com/sguan-actuate\",\"followers_url\":\"https://api.github.com/users/sguan-actuate/followers\",\"following_url\":\"https://api.github.com/users/sguan-actuate/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sguan-actuate/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sguan-actuate/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sguan-actuate/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sguan-actuate/orgs\",\"repos_url\":\"https://api.github.com/users/sguan-actuate/repos\",\"events_url\":\"https://api.github.com/users/sguan-actuate/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sguan-actuate/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28415488,\"name\":\"birt\",\"full_name\":\"sguan-actuate/birt\",\"owner\":{\"login\":\"sguan-actuate\",\"id\":8562199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8562199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sguan-actuate\",\"html_url\":\"https://github.com/sguan-actuate\",\"followers_url\":\"https://api.github.com/users/sguan-actuate/followers\",\"following_url\":\"https://api.github.com/users/sguan-actuate/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sguan-actuate/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sguan-actuate/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sguan-actuate/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sguan-actuate/orgs\",\"repos_url\":\"https://api.github.com/users/sguan-actuate/repos\",\"events_url\":\"https://api.github.com/users/sguan-actuate/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sguan-actuate/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/sguan-actuate/birt\",\"description\":\"The open source Eclipse BIRT reporting and data visualization project. \",\"fork\":true,\"url\":\"https://api.github.com/repos/sguan-actuate/birt\",\"forks_url\":\"https://api.github.com/repos/sguan-actuate/birt/forks\",\"keys_url\":\"https://api.github.com/repos/sguan-actuate/birt/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/sguan-actuate/birt/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/sguan-actuate/birt/teams\",\"hooks_url\":\"https://api.github.com/repos/sguan-actuate/birt/hooks\",\"issue_events_url\":\"https://api.github.com/repos/sguan-actuate/birt/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/sguan-actuate/birt/events\",\"assignees_url\":\"https://api.github.com/repos/sguan-actuate/birt/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/sguan-actuate/birt/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/sguan-actuate/birt/tags\",\"blobs_url\":\"https://api.github.com/repos/sguan-actuate/birt/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/sguan-actuate/birt/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/sguan-actuate/birt/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/sguan-actuate/birt/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/sguan-actuate/birt/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/sguan-actuate/birt/languages\",\"stargazers_url\":\"https://api.github.com/repos/sguan-actuate/birt/stargazers\",\"contributors_url\":\"https://api.github.com/repos/sguan-actuate/birt/contributors\",\"subscribers_url\":\"https://api.github.com/repos/sguan-actuate/birt/subscribers\",\"subscription_url\":\"https://api.github.com/repos/sguan-actuate/birt/subscription\",\"commits_url\":\"https://api.github.com/repos/sguan-actuate/birt/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/sguan-actuate/birt/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/sguan-actuate/birt/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/sguan-actuate/birt/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/sguan-actuate/birt/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/sguan-actuate/birt/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/sguan-actuate/birt/merges\",\"archive_url\":\"https://api.github.com/repos/sguan-actuate/birt/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/sguan-actuate/birt/downloads\",\"issues_url\":\"https://api.github.com/repos/sguan-actuate/birt/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/sguan-actuate/birt/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/sguan-actuate/birt/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/sguan-actuate/birt/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/sguan-actuate/birt/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/sguan-actuate/birt/releases{/id}\",\"created_at\":\"2014-12-23T19:25:25Z\",\"updated_at\":\"2015-01-01T00:23:05Z\",\"pushed_at\":\"2015-01-01T00:23:05Z\",\"git_url\":\"git://github.com/sguan-actuate/birt.git\",\"ssh_url\":\"git@github.com:sguan-actuate/birt.git\",\"clone_url\":\"https://github.com/sguan-actuate/birt.git\",\"svn_url\":\"https://github.com/sguan-actuate/birt\",\"homepage\":\"http://www.eclipse.org/birt\",\"size\":171178,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Java\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"eclipse:master\",\"ref\":\"master\",\"sha\":\"9e12f3e628b2b0657a557bbeda71fa13337cdfe2\",\"user\":{\"login\":\"eclipse\",\"id\":56974,\"avatar_url\":\"https://avatars.githubusercontent.com/u/56974?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eclipse\",\"html_url\":\"https://github.com/eclipse\",\"followers_url\":\"https://api.github.com/users/eclipse/followers\",\"following_url\":\"https://api.github.com/users/eclipse/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/eclipse/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/eclipse/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/eclipse/subscriptions\",\"organizations_url\":\"https://api.github.com/users/eclipse/orgs\",\"repos_url\":\"https://api.github.com/users/eclipse/repos\",\"events_url\":\"https://api.github.com/users/eclipse/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/eclipse/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":19821157,\"name\":\"birt\",\"full_name\":\"eclipse/birt\",\"owner\":{\"login\":\"eclipse\",\"id\":56974,\"avatar_url\":\"https://avatars.githubusercontent.com/u/56974?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eclipse\",\"html_url\":\"https://github.com/eclipse\",\"followers_url\":\"https://api.github.com/users/eclipse/followers\",\"following_url\":\"https://api.github.com/users/eclipse/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/eclipse/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/eclipse/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/eclipse/subscriptions\",\"organizations_url\":\"https://api.github.com/users/eclipse/orgs\",\"repos_url\":\"https://api.github.com/users/eclipse/repos\",\"events_url\":\"https://api.github.com/users/eclipse/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/eclipse/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/eclipse/birt\",\"description\":\"The open source Eclipse BIRT reporting and data visualization project. \",\"fork\":false,\"url\":\"https://api.github.com/repos/eclipse/birt\",\"forks_url\":\"https://api.github.com/repos/eclipse/birt/forks\",\"keys_url\":\"https://api.github.com/repos/eclipse/birt/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/eclipse/birt/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/eclipse/birt/teams\",\"hooks_url\":\"https://api.github.com/repos/eclipse/birt/hooks\",\"issue_events_url\":\"https://api.github.com/repos/eclipse/birt/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/eclipse/birt/events\",\"assignees_url\":\"https://api.github.com/repos/eclipse/birt/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/eclipse/birt/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/eclipse/birt/tags\",\"blobs_url\":\"https://api.github.com/repos/eclipse/birt/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/eclipse/birt/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/eclipse/birt/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/eclipse/birt/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/eclipse/birt/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/eclipse/birt/languages\",\"stargazers_url\":\"https://api.github.com/repos/eclipse/birt/stargazers\",\"contributors_url\":\"https://api.github.com/repos/eclipse/birt/contributors\",\"subscribers_url\":\"https://api.github.com/repos/eclipse/birt/subscribers\",\"subscription_url\":\"https://api.github.com/repos/eclipse/birt/subscription\",\"commits_url\":\"https://api.github.com/repos/eclipse/birt/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/eclipse/birt/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/eclipse/birt/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/eclipse/birt/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/eclipse/birt/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/eclipse/birt/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/eclipse/birt/merges\",\"archive_url\":\"https://api.github.com/repos/eclipse/birt/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/eclipse/birt/downloads\",\"issues_url\":\"https://api.github.com/repos/eclipse/birt/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/eclipse/birt/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/eclipse/birt/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/eclipse/birt/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/eclipse/birt/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/eclipse/birt/releases{/id}\",\"created_at\":\"2014-05-15T13:55:31Z\",\"updated_at\":\"2015-01-01T00:31:05Z\",\"pushed_at\":\"2015-01-01T00:31:05Z\",\"git_url\":\"git://github.com/eclipse/birt.git\",\"ssh_url\":\"git@github.com:eclipse/birt.git\",\"clone_url\":\"https://github.com/eclipse/birt.git\",\"svn_url\":\"https://github.com/eclipse/birt\",\"homepage\":\"http://www.eclipse.org/birt\",\"size\":939311,\"stargazers_count\":8,\"watchers_count\":8,\"language\":\"Java\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":20,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":20,\"open_issues\":1,\"watchers\":8,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/eclipse/birt/pulls/70\"},\"html\":{\"href\":\"https://github.com/eclipse/birt/pull/70\"},\"issue\":{\"href\":\"https://api.github.com/repos/eclipse/birt/issues/70\"},\"comments\":{\"href\":\"https://api.github.com/repos/eclipse/birt/issues/70/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/eclipse/birt/pulls/70/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/eclipse/birt/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/eclipse/birt/pulls/70/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/eclipse/birt/statuses/7efab4d3fe077789a3e65be0fcd80a6078557360\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":2,\"additions\":59,\"deletions\":14,\"changed_files\":4}},\"public\":true,\"created_at\":\"2015-01-01T01:03:38Z\",\"org\":{\"id\":56974,\"login\":\"eclipse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/eclipse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/56974?\"}}\n{\"id\":\"2489397442\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1398544,\"login\":\"joelpurra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joelpurra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1398544?\"},\"repo\":{\"id\":28595466,\"name\":\"joelpurra/jqnpm\",\"url\":\"https://api.github.com/repos/joelpurra/jqnpm\"},\"payload\":{\"ref\":\"import_659\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"A package manager built for jq as an example implementation.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:38Z\"}\n{\"id\":\"2489397443\",\"type\":\"PushEvent\",\"actor\":{\"id\":10350323,\"login\":\"nik7273\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nik7273\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10350323?\"},\"repo\":{\"id\":28635375,\"name\":\"nik7273/wikipedia-access-thru-API\",\"url\":\"https://api.github.com/repos/nik7273/wikipedia-access-thru-API\"},\"payload\":{\"push_id\":536752948,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6adff88c4820e3e820c949eebb7ed57521cb5e26\",\"before\":\"ce036d56a80edcb460c0c2c71d4c410f2709ded8\",\"commits\":[{\"sha\":\"6adff88c4820e3e820c949eebb7ed57521cb5e26\",\"author\":{\"email\":\"40527771d623df543fb2bc588362a230688f2063@gmail.com\",\"name\":\"nik7273\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nik7273/wikipedia-access-thru-API/commits/6adff88c4820e3e820c949eebb7ed57521cb5e26\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:38Z\"}\n{\"id\":\"2489397445\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1693373,\"login\":\"puhley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/puhley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1693373?\"},\"repo\":{\"id\":18814466,\"name\":\"crits/crits\",\"url\":\"https://api.github.com/repos/crits/crits\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/crits/crits/issues/333\",\"labels_url\":\"https://api.github.com/repos/crits/crits/issues/333/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/crits/crits/issues/333/comments\",\"events_url\":\"https://api.github.com/repos/crits/crits/issues/333/events\",\"html_url\":\"https://github.com/crits/crits/pull/333\",\"id\":50806180,\"number\":333,\"title\":\"API delete support for domains and IPs\",\"user\":{\"login\":\"puhley\",\"id\":1693373,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1693373?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/puhley\",\"html_url\":\"https://github.com/puhley\",\"followers_url\":\"https://api.github.com/users/puhley/followers\",\"following_url\":\"https://api.github.com/users/puhley/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/puhley/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/puhley/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/puhley/subscriptions\",\"organizations_url\":\"https://api.github.com/users/puhley/orgs\",\"repos_url\":\"https://api.github.com/users/puhley/repos\",\"events_url\":\"https://api.github.com/users/puhley/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/puhley/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":8,\"created_at\":\"2014-12-03T09:10:11Z\",\"updated_at\":\"2015-01-01T01:03:38Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/crits/crits/pulls/333\",\"html_url\":\"https://github.com/crits/crits/pull/333\",\"diff_url\":\"https://github.com/crits/crits/pull/333.diff\",\"patch_url\":\"https://github.com/crits/crits/pull/333.patch\"},\"body\":\"This is a rough draft of code that will enable delete support for the IP and domain APIs. Since an IP or domain can be associated with multiple sources and campaigns, this approach allows you to delete an individual reference or to delete the entire record. If you provide a campaign and source along with the record ID, then this will just delete the references to that campaign and source. If you only provide the record ID, then it will delete the entire record.\\r\\n\\r\\nThe code for delete support in the domain API is a little complicated. The django-tastypie-mongoengine won't accept a DELETE request with parameters passed in the body of the request. Therefore, the DELETE request must pass the parameters in the URL. However, in order to pass the request to the generate_domain_jtable, the request must be converted into a POST format. Since I didn't know the jtable code well enough to reformat it, I got around the issue by creating a fake POST object and passing that instead. This isn't the most elegant solution but it allowed me to contain the changes to the API code.\\r\\n\\r\\nBeing able to delete an IP or API reference would be useful for people who want to use CRITs to track only the current list of threats. For instance, IPs are sometimes removed from IOC lists because the compromised host has been fixed. These changes would allow you to remove the IP from the current list of threats via the API. This code still requires the user to have admin privileges in order to perform the deletion.\\r\\n\\r\\nPlease take a look at the code and let me know your thoughts. \"},\"comment\":{\"url\":\"https://api.github.com/repos/crits/crits/issues/comments/68477286\",\"html_url\":\"https://github.com/crits/crits/pull/333#issuecomment-68477286\",\"issue_url\":\"https://api.github.com/repos/crits/crits/issues/333\",\"id\":68477286,\"user\":{\"login\":\"puhley\",\"id\":1693373,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1693373?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/puhley\",\"html_url\":\"https://github.com/puhley\",\"followers_url\":\"https://api.github.com/users/puhley/followers\",\"following_url\":\"https://api.github.com/users/puhley/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/puhley/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/puhley/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/puhley/subscriptions\",\"organizations_url\":\"https://api.github.com/users/puhley/orgs\",\"repos_url\":\"https://api.github.com/users/puhley/repos\",\"events_url\":\"https://api.github.com/users/puhley/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/puhley/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:38Z\",\"updated_at\":\"2015-01-01T01:03:38Z\",\"body\":\"* I changed the logic from “removing the campaign from the domain” to “removing a domain from the campaign”. This eliminates the redundant campaign removal logic for each TLO. The new code will allow you to remove any type of object from a campaign.\\r\\n* I added the ability to delete a campaign. I saw that functionality requested in one of the threads.\\r\\n* All of the TLOs used jtable_ajax_delete. Therefore, I split jtable_ajax_delete into 2 functions so that both the UI (via jtable_ajax_delete -> delete_id) and the API (via a direct call to delete_id) could reference the same deletion code for ObjectIds.\\r\\n* I added validate_objectid to the mongo_tools library to centralize validation code for ObjectIds.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:39Z\",\"org\":{\"id\":5392624,\"login\":\"crits\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/crits\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5392624?\"}}\n{\"id\":\"2489397449\",\"type\":\"PushEvent\",\"actor\":{\"id\":6267945,\"login\":\"lakotadlustig\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lakotadlustig\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6267945?\"},\"repo\":{\"id\":28615365,\"name\":\"korlabs/tippr\",\"url\":\"https://api.github.com/repos/korlabs/tippr\"},\"payload\":{\"push_id\":536752950,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cf31abd3724db74c818220b0d13c26ebcecfc128\",\"before\":\"010ba4d2a839b14d435b0d2c2ea75a861cab3e5d\",\"commits\":[{\"sha\":\"cf31abd3724db74c818220b0d13c26ebcecfc128\",\"author\":{\"email\":\"3280e741e838010abad30f9c64878ae2d3f3e766@podbe.at\",\"name\":\"Lakota Lustig\"},\"message\":\"Added ability to have tips refunded.\\n\\nRequires wallet URL to be saved to a text file sadly, but it was the only way.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/korlabs/tippr/commits/cf31abd3724db74c818220b0d13c26ebcecfc128\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:39Z\",\"org\":{\"id\":7786022,\"login\":\"korlabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/korlabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7786022?\"}}\n{\"id\":\"2489397454\",\"type\":\"PushEvent\",\"actor\":{\"id\":3933623,\"login\":\"tonyblank\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tonyblank\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3933623?\"},\"repo\":{\"id\":28676606,\"name\":\"tonyblank/Beerman\",\"url\":\"https://api.github.com/repos/tonyblank/Beerman\"},\"payload\":{\"push_id\":536752951,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3971df1986b9944235005eac430da3525adcd81b\",\"before\":\"bac32fd982495634fa319f0ba38161a47ee9be01\",\"commits\":[{\"sha\":\"3971df1986b9944235005eac430da3525adcd81b\",\"author\":{\"email\":\"b1c1d8736f20db3fb6c1c66bb1455ed43909f0d8@tonyblank.com\",\"name\":\"Tony Blank\"},\"message\":\"created ROUGH readme with install instruction summary\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tonyblank/Beerman/commits/3971df1986b9944235005eac430da3525adcd81b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:41Z\"}\n{\"id\":\"2489397455\",\"type\":\"PushEvent\",\"actor\":{\"id\":1359331,\"login\":\"yepher\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yepher\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1359331?\"},\"repo\":{\"id\":28612273,\"name\":\"yepher/littlebits\",\"url\":\"https://api.github.com/repos/yepher/littlebits\"},\"payload\":{\"push_id\":536752952,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ad8f676eb03c36491ba1459349574c8f3b308602\",\"before\":\"f6c00396a9221646c39f7fc97fe2d8b24a6baed7\",\"commits\":[{\"sha\":\"ad8f676eb03c36491ba1459349574c8f3b308602\",\"author\":{\"email\":\"711c73f64afdce07b7e38039a96d2224209e9a6c@mfluent.com\",\"name\":\"Chris Wilson\"},\"message\":\"Adds some documentation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/yepher/littlebits/commits/ad8f676eb03c36491ba1459349574c8f3b308602\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:41Z\"}\n{\"id\":\"2489397456\",\"type\":\"PushEvent\",\"actor\":{\"id\":3794984,\"login\":\"abustamam\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/abustamam\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3794984?\"},\"repo\":{\"id\":25110710,\"name\":\"abustamam/Udacity-Portfolio\",\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio\"},\"payload\":{\"push_id\":536752953,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"5776bf9082fbc389cc986d9159e7367cc2fcdf20\",\"before\":\"b39e5ac9096f3ddcc10a485bd9df8dbedcfda5ef\",\"commits\":[{\"sha\":\"e323ac3bd7451b41388d083ec83e1dc3a4d94022\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Change project 1 img\",\"distinct\":true,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/e323ac3bd7451b41388d083ec83e1dc3a4d94022\"},{\"sha\":\"5776bf9082fbc389cc986d9159e7367cc2fcdf20\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Merge branch 'master' of github.com:abustamam/Udacity-Portfolio\\n\\nPull readme\",\"distinct\":true,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/5776bf9082fbc389cc986d9159e7367cc2fcdf20\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:41Z\"}\n{\"id\":\"2489397457\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":923144,\"login\":\"pythonesque\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pythonesque\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/923144?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355\",\"labels_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/comments\",\"events_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/events\",\"html_url\":\"https://github.com/rust-lang/rust/issues/20355\",\"id\":53157956,\"number\":20355,\"title\":\"Segfault in `Hasher<SipState>::hash`\",\"user\":{\"login\":\"fitzgen\",\"id\":74571,\"avatar_url\":\"https://avatars.githubusercontent.com/u/74571?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fitzgen\",\"html_url\":\"https://github.com/fitzgen\",\"followers_url\":\"https://api.github.com/users/fitzgen/followers\",\"following_url\":\"https://api.github.com/users/fitzgen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/fitzgen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/fitzgen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/fitzgen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/fitzgen/orgs\",\"repos_url\":\"https://api.github.com/users/fitzgen/repos\",\"events_url\":\"https://api.github.com/users/fitzgen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/fitzgen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/rust-lang/rust/labels/I-crash\",\"name\":\"I-crash\",\"color\":\"e10c02\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":6,\"created_at\":\"2014-12-31T00:04:46Z\",\"updated_at\":\"2015-01-01T01:03:40Z\",\"closed_at\":null,\"body\":\"Test case is branch `sip-hasher-segault` of this repo: https://github.com/fitzgen/oxischeme/tree/sip-hasher-segfault\\r\\n\\r\\n    $ rustc --version --verbose\\r\\n    rustc --version --verbose\\r\\n    rustc 0.13.0-nightly (5ba610265 2014-12-25 18:01:36 +0000)\\r\\n    binary: rustc\\r\\n    commit-hash: 5ba6102657a892457063d2d6a7cbb9632ce282c6\\r\\n    commit-date: 2014-12-25 18:01:36 +0000\\r\\n    host: x86_64-apple-darwin\\r\\n    release: 0.13.0-nightly\\r\\n    $ rustc -g --test src/main.rs -o target/test\\r\\n    $ lldb ./target/test\\r\\n    lldb ./target/test\\r\\n    (lldb) target create \\\"./target/test\\\"\\r\\n    Current executable set to './target/test' (x86_64).\\r\\n    (lldb) run\\r\\n    Process 8270 launched: './target/test' (x86_64)\\r\\n\\r\\n    running 20 tests\\r\\n    Process 8270 stopped\\r\\n    * thread #2: tid = 0x25693, 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61, stop reason = EXC_BAD_ACCESS (code=1, address=0x3ffd9)\\r\\n        frame #0: 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61\\r\\n    test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61:\\r\\n    -> 0x100008bed:  movq   (%rsi), %rsi\\r\\n       0x100008bf0:  movq   -0x8(%rbp), %rdi\\r\\n       0x100008bf4:  movq   0x8(%rdi), %rdx\\r\\n       0x100008bf8:  movq   %rax, %rdi\\r\\n    (lldb) bt\\r\\n    * thread #2: tid = 0x25693, 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61, stop reason = EXC_BAD_ACCESS (code=1, address=0x3ffd9)\\r\\n      * frame #0: 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61\\r\\n        frame #1: 0x0000000105401810\\r\\n        frame #2: 0x0000000100008ba2 test`hash::RandomSipHasher.Hasher$LT$sip..SipState$GT$::hash::h10512232733763305808 + 66\\r\\n        frame #3: 0x0000000100008b0e test`collections::hash::table::make_hash::h9604696550138645665 + 62\\r\\n        frame #4: 0x0000000100008abe test`collections::hash::map::HashMap$LT$K$C$$u{20}V$C$$u{20}H$GT$::make_hash::h15786379260790357760 + 62\\r\\n        frame #5: 0x0000000100008963 test`collections::hash::map::HashMap$LT$K$C$$u{20}V$C$$u{20}H$GT$::insert::h6805880362953821147 + 131\\r\\n        frame #6: 0x0000000100006aea test`main::environment::Environment::define(self=0x000000000003ffc1, sym=String at 0x0000000105401a50, val=0x0000000105401c38) + 218 at environment.rs:92\\r\\n        frame #7: 0x0000000100018486 test`main::eval::evaluate_definition(heap=0x00000001054034e8, env=0x0000000105402d10, form=0x0000000105402ce0) + 2406 at eval.rs:195\\r\\n        frame #8: 0x0000000100013f1b test`main::eval::evaluate(heap=0x00000001054034e8, env=0x0000000105402d90, form=0x0000000105402f30) + 1355 at eval.rs:79\\r\\n        frame #9: 0x0000000100013909 test`main::eval::evaluate_in_global_env(heap=0x00000001054034e8, form=0x0000000105402f30) + 105 at eval.rs:38\\r\\n        frame #10: 0x000000010001cd7b test`main::eval::evaluate_file(heap=0x00000001054034e8, file_path=(data_ptr = \\\"./tests/test_eval_closures.scmsrc/heap.rsArenaPtr(, )Rooted(\\\", length = 30)) + 1035 at <std macros>:286\\r\\n        frame #11: 0x000000010002c071 test`main::eval::test_eval_closures + 129 at eval.rs:403\\r\\n        frame #12: 0x0000000100087154 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h5218640484166224076 + 52\\r\\n        frame #13: 0x0000000100091e32 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h13672304839710771993 + 162\\r\\n        frame #14: 0x0000000100088912 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h15490786364890695979 + 1138\\r\\n        frame #15: 0x0000000100088db0 test`rt::unwind::try::try_fn::h10439459448986358359 + 160\\r\\n        frame #16: 0x000000010010a4f9 test`rust_try_inner + 9\\r\\n        frame #17: 0x000000010010a4e6 test`rust_try + 6\\r\\n        frame #18: 0x00000001000894ab test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h13206639752781900349 + 1179\\r\\n        frame #19: 0x0000000100106eb4 test`sys::thread::thread_start::h2cb22211a4c7d938vFw + 164\\r\\n        frame #20: 0x00007fff875d3899 libsystem_pthread.dylib`_pthread_body + 138\\r\\n        frame #21: 0x00007fff875d372a libsystem_pthread.dylib`_pthread_start + 137\\r\\n        frame #22: 0x00007fff875d7fc9 libsystem_pthread.dylib`thread_start + 13\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/comments/68477288\",\"html_url\":\"https://github.com/rust-lang/rust/issues/20355#issuecomment-68477288\",\"issue_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355\",\"id\":68477288,\"user\":{\"login\":\"pythonesque\",\"id\":923144,\"avatar_url\":\"https://avatars.githubusercontent.com/u/923144?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pythonesque\",\"html_url\":\"https://github.com/pythonesque\",\"followers_url\":\"https://api.github.com/users/pythonesque/followers\",\"following_url\":\"https://api.github.com/users/pythonesque/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/pythonesque/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/pythonesque/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/pythonesque/subscriptions\",\"organizations_url\":\"https://api.github.com/users/pythonesque/orgs\",\"repos_url\":\"https://api.github.com/users/pythonesque/repos\",\"events_url\":\"https://api.github.com/users/pythonesque/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/pythonesque/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:40Z\",\"updated_at\":\"2015-01-01T01:03:40Z\",\"body\":\"I probably would.  But if you can find a reproducible test case that doesn't use any unsafe code, you can always reopen it :)\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:41Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489397466\",\"type\":\"PushEvent\",\"actor\":{\"id\":1727112,\"login\":\"nucleardreamer\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nucleardreamer\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1727112?\"},\"repo\":{\"id\":28675721,\"name\":\"nucleardreamer/tdc\",\"url\":\"https://api.github.com/repos/nucleardreamer/tdc\"},\"payload\":{\"push_id\":536752957,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5763af4a3216ce785215e64ec213ce78130c2b0f\",\"before\":\"07a6ae5cdb39596d5aed1b3794229968810bce1f\",\"commits\":[{\"sha\":\"5763af4a3216ce785215e64ec213ce78130c2b0f\",\"author\":{\"email\":\"61996574b2cb2f66e0c7dcf5a13359213bfaba5a@gmail.com\",\"name\":\"nucleardreamer\"},\"message\":\"updating mask image\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nucleardreamer/tdc/commits/5763af4a3216ce785215e64ec213ce78130c2b0f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:42Z\"}\n{\"id\":\"2489397467\",\"type\":\"PushEvent\",\"actor\":{\"id\":3596952,\"login\":\"priseborough\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/priseborough\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3596952?\"},\"repo\":{\"id\":8208289,\"name\":\"priseborough/ardupilot\",\"url\":\"https://api.github.com/repos/priseborough/ardupilot\"},\"payload\":{\"push_id\":536752958,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/ekfUpdates\",\"head\":\"5a91d89607bd9552e9767239f90802aae6e95675\",\"before\":\"fce220571dcbd8a4621c9743d90aa48f5a487940\",\"commits\":[{\"sha\":\"b1be8e09be78483f31ec47ee6bec00606e72184d\",\"author\":{\"email\":\"95a043cc2b9ec25c579a602c10711cec6c80724d@live.com.au\",\"name\":\"priseborough\"},\"message\":\"# This is a combination of 2 commits.\\n# The first commit's message is:\\n\\nAP_NavEKF: Enumerate Position and Velocity Aiding\\n\\n# This is the 2nd commit message:\\n\\nAP_NavEKF: Fix bug in declaration of enumerated aiding mode variable\",\"distinct\":true,\"url\":\"https://api.github.com/repos/priseborough/ardupilot/commits/b1be8e09be78483f31ec47ee6bec00606e72184d\"},{\"sha\":\"9ce6b6502df2a96be13b0698193271cec028039a\",\"author\":{\"email\":\"95a043cc2b9ec25c579a602c10711cec6c80724d@live.com.au\",\"name\":\"priseborough\"},\"message\":\"AP_NavEKF: Make reversion to no GPS mode unambiguous\",\"distinct\":true,\"url\":\"https://api.github.com/repos/priseborough/ardupilot/commits/9ce6b6502df2a96be13b0698193271cec028039a\"},{\"sha\":\"200c088c50d34ecb4d7f9ed1c35fa7d4d173582b\",\"author\":{\"email\":\"95a043cc2b9ec25c579a602c10711cec6c80724d@live.com.au\",\"name\":\"priseborough\"},\"message\":\"AP_NavEKF: Consolidate constant velocity mode decision logic\\n\\nThe decision to switch to constant velocity mode during optical flow operation and te decision to switch back were previously being made in two different places in code. Both decisions are now made in the one place which makes the code easier to follow and maintain.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/priseborough/ardupilot/commits/200c088c50d34ecb4d7f9ed1c35fa7d4d173582b\"},{\"sha\":\"5a91d89607bd9552e9767239f90802aae6e95675\",\"author\":{\"email\":\"95a043cc2b9ec25c579a602c10711cec6c80724d@live.com.au\",\"name\":\"priseborough\"},\"message\":\"AP_NavEKF: Consistent initialisation of tuning parameters and variables\\n\\nNon user adjustable parameters are now declared as 'const' with values assigned in the header.\\nThe _ prefix has been removed from non user adjustable tuning parameters.\\nVariables that were being initialised in the constructor are now initialised in the function call along with the other filter variables. We use a function call rather than a constructor to initialise variables because it enables the filter to be re-started in flight if necessary.\\n\\nSigned-off-by: priseborough <p_riseborough@live.com.au>\\n\\nConflicts:\\n\\tlibraries/AP_NavEKF/AP_NavEKF.h\",\"distinct\":true,\"url\":\"https://api.github.com/repos/priseborough/ardupilot/commits/5a91d89607bd9552e9767239f90802aae6e95675\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:42Z\"}\n{\"id\":\"2489397469\",\"type\":\"PushEvent\",\"actor\":{\"id\":10324417,\"login\":\"kerned-code\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kerned-code\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10324417?\"},\"repo\":{\"id\":28541287,\"name\":\"kerned-code/dpl-403\",\"url\":\"https://api.github.com/repos/kerned-code/dpl-403\"},\"payload\":{\"push_id\":536752959,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6fbb8ed9aaa4357dd3a48f5a383d9223710102c9\",\"before\":\"fee005c1ed537333e00cffd8de45b1104543ca6b\",\"commits\":[{\"sha\":\"6fbb8ed9aaa4357dd3a48f5a383d9223710102c9\",\"author\":{\"email\":\"44d17961b0395a23aed103901a26e8bd0af6ae21@users.noreply.github.com\",\"name\":\"kerned-code\"},\"message\":\"Expand README\\n\\nWith details about function calls and built-in functions.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kerned-code/dpl-403/commits/6fbb8ed9aaa4357dd3a48f5a383d9223710102c9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:43Z\"}\n{\"id\":\"2489397471\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":9170481,\"login\":\"scott46743\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scott46743\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9170481?\"},\"repo\":{\"id\":25050352,\"name\":\"MCGamerNetwork/Issues\",\"url\":\"https://api.github.com/repos/MCGamerNetwork/Issues\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/474\",\"labels_url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/474/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/474/comments\",\"events_url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/474/events\",\"html_url\":\"https://github.com/MCGamerNetwork/Issues/issues/474\",\"id\":51627486,\"number\":474,\"title\":\"Recent Reports Section on Xime\",\"user\":{\"login\":\"scott46743\",\"id\":9170481,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9170481?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scott46743\",\"html_url\":\"https://github.com/scott46743\",\"followers_url\":\"https://api.github.com/users/scott46743/followers\",\"following_url\":\"https://api.github.com/users/scott46743/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/scott46743/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/scott46743/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/scott46743/subscriptions\",\"organizations_url\":\"https://api.github.com/users/scott46743/orgs\",\"repos_url\":\"https://api.github.com/users/scott46743/repos\",\"events_url\":\"https://api.github.com/users/scott46743/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/scott46743/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":9,\"created_at\":\"2014-12-10T23:49:39Z\",\"updated_at\":\"2015-01-01T01:03:43Z\",\"closed_at\":null,\"body\":\"Environment\\r\\nXime panel.\\r\\n\\r\\nThe issue\\r\\nThe recent report section has stopped working, and doesn't show any reports as of 12/06/14 at 8:38 EST. In addition, the latest reports included XAC reports, which I don't believe are supposed to show up. \\r\\n\\r\\nSteps to reproduce\\r\\nCheck recent reports section: http://manage.mcgamer.net/punishment/reports/\\r\\n\\r\\nMiscellaneous information\\r\\n\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/comments/68477289\",\"html_url\":\"https://github.com/MCGamerNetwork/Issues/issues/474#issuecomment-68477289\",\"issue_url\":\"https://api.github.com/repos/MCGamerNetwork/Issues/issues/474\",\"id\":68477289,\"user\":{\"login\":\"scott46743\",\"id\":9170481,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9170481?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/scott46743\",\"html_url\":\"https://github.com/scott46743\",\"followers_url\":\"https://api.github.com/users/scott46743/followers\",\"following_url\":\"https://api.github.com/users/scott46743/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/scott46743/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/scott46743/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/scott46743/subscriptions\",\"organizations_url\":\"https://api.github.com/users/scott46743/orgs\",\"repos_url\":\"https://api.github.com/users/scott46743/repos\",\"events_url\":\"https://api.github.com/users/scott46743/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/scott46743/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:43Z\",\"updated_at\":\"2015-01-01T01:03:43Z\",\"body\":\"@sheldor-da-great @subv3rsion It appears to have been fixed by @kpwn243 and it hasn't been frozen for days. I think it's alright to close this :+1: \"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:43Z\",\"org\":{\"id\":6956717,\"login\":\"MCGamerNetwork\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MCGamerNetwork\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6956717?\"}}\n{\"id\":\"2489397472\",\"type\":\"PushEvent\",\"actor\":{\"id\":3656079,\"login\":\"marklrh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marklrh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3656079?\"},\"repo\":{\"id\":27470715,\"name\":\"marklrh/ocaml-cohttp-test\",\"url\":\"https://api.github.com/repos/marklrh/ocaml-cohttp-test\"},\"payload\":{\"push_id\":536752962,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1a309ab4cddecc0469c8a9c51733570a9624ea54\",\"before\":\"7525c74ee31f49f576408a7d50d1cf8b9d3d1441\",\"commits\":[{\"sha\":\"1a309ab4cddecc0469c8a9c51733570a9624ea54\",\"author\":{\"email\":\"e0e04a2320844b42511db0376599e166ab5bda54@gmail.com\",\"name\":\"Runhang Li\"},\"message\":\"lint README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/marklrh/ocaml-cohttp-test/commits/1a309ab4cddecc0469c8a9c51733570a9624ea54\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:43Z\"}\n{\"id\":\"2489397475\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":19967,\"login\":\"nicholasf\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nicholasf\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/19967?\"},\"repo\":{\"id\":25658193,\"name\":\"rockpoollabs/env\",\"url\":\"https://api.github.com/repos/rockpoollabs/env\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rockpoollabs/env/issues/6\",\"labels_url\":\"https://api.github.com/repos/rockpoollabs/env/issues/6/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rockpoollabs/env/issues/6/comments\",\"events_url\":\"https://api.github.com/repos/rockpoollabs/env/issues/6/events\",\"html_url\":\"https://github.com/rockpoollabs/env/pull/6\",\"id\":53202302,\"number\":6,\"title\":\"Cleaned up shell code.\",\"user\":{\"login\":\"wayneeseguin\",\"id\":18,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wayneeseguin\",\"html_url\":\"https://github.com/wayneeseguin\",\"followers_url\":\"https://api.github.com/users/wayneeseguin/followers\",\"following_url\":\"https://api.github.com/users/wayneeseguin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wayneeseguin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wayneeseguin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wayneeseguin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wayneeseguin/orgs\",\"repos_url\":\"https://api.github.com/users/wayneeseguin/repos\",\"events_url\":\"https://api.github.com/users/wayneeseguin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wayneeseguin/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T20:35:45Z\",\"updated_at\":\"2015-01-01T01:03:43Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/rockpoollabs/env/pulls/6\",\"html_url\":\"https://github.com/rockpoollabs/env/pull/6\",\"diff_url\":\"https://github.com/rockpoollabs/env/pull/6.diff\",\"patch_url\":\"https://github.com/rockpoollabs/env/pull/6.patch\"},\"body\":\"I came across your repository and was browsing it, when I got to this file I couldn't resist sending a few suggestions your way:\\r\\n \\r\\n* $PWD is automatically available when using BASH, so there is no need for PWD=`pwd`\\r\\n* If you were to use a subshell be sure to use $() instead of ``.\\r\\n* it DRY's things up a bit to have a single export line for the variables if you are exporting multiple variables\\r\\n* Technically exporting PATH is redundant since it is exported by default, but being explicit doesn't hurt.\\r\\n* Quoting the variable assignments ensures that you are never caught by spaces and perhaps other special characters in your path name which would break the script otherwise.\\r\\n* Best to reserve all uppercase variable names for exported variables only, script local variables use lower case variable names with naming conventions similar to go style.\"},\"comment\":{\"url\":\"https://api.github.com/repos/rockpoollabs/env/issues/comments/68477290\",\"html_url\":\"https://github.com/rockpoollabs/env/pull/6#issuecomment-68477290\",\"issue_url\":\"https://api.github.com/repos/rockpoollabs/env/issues/6\",\"id\":68477290,\"user\":{\"login\":\"nicholasf\",\"id\":19967,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19967?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nicholasf\",\"html_url\":\"https://github.com/nicholasf\",\"followers_url\":\"https://api.github.com/users/nicholasf/followers\",\"following_url\":\"https://api.github.com/users/nicholasf/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nicholasf/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nicholasf/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nicholasf/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nicholasf/orgs\",\"repos_url\":\"https://api.github.com/users/nicholasf/repos\",\"events_url\":\"https://api.github.com/users/nicholasf/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nicholasf/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:43Z\",\"updated_at\":\"2015-01-01T01:03:43Z\",\"body\":\"Thanks Wayne.\\r\\n\\r\\nInteresting comments. We'll look at this on Monday (the 5th), when we're back from holidays.\\r\\n\\r\\nHappy 2015.\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:43Z\",\"org\":{\"id\":4385125,\"login\":\"rockpoollabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rockpoollabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4385125?\"}}\n{\"id\":\"2489397479\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/82\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/82\",\"id\":53210240,\"number\":82,\"title\":\"Add background color to done buttons in all Webviews\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":6,\"closed_issues\":10,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:03:44Z\",\"due_on\":null,\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:03:44Z\",\"updated_at\":\"2015-01-01T01:03:44Z\",\"closed_at\":null,\"body\":\"- 2 on Donate screen\\r\\n- 1 on About screen\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:44Z\"}\n{\"id\":\"2489397484\",\"type\":\"PushEvent\",\"actor\":{\"id\":3032425,\"login\":\"qq978304139\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qq978304139\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3032425?\"},\"repo\":{\"id\":22276837,\"name\":\"qq978304139/pj_git\",\"url\":\"https://api.github.com/repos/qq978304139/pj_git\"},\"payload\":{\"push_id\":536752965,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master_dev\",\"head\":\"17bd49147c1bffd3f69b9d45a7de1b0492eb0f6e\",\"before\":\"287a09b687ea3c1328474454d6ea534bc4e03960\",\"commits\":[{\"sha\":\"28a720004dad4f9c0d71198b85aa326dcce210c8\",\"author\":{\"email\":\"b29612f187963685a9f710ed6e44f9006a30c278@gmail.com\",\"name\":\"hepeng\"},\"message\":\"add dump_stack for linux\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qq978304139/pj_git/commits/28a720004dad4f9c0d71198b85aa326dcce210c8\"},{\"sha\":\"17bd49147c1bffd3f69b9d45a7de1b0492eb0f6e\",\"author\":{\"email\":\"b29612f187963685a9f710ed6e44f9006a30c278@gmail.com\",\"name\":\"hepeng\"},\"message\":\"play wav to remote caller\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qq978304139/pj_git/commits/17bd49147c1bffd3f69b9d45a7de1b0492eb0f6e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:45Z\"}\n{\"id\":\"2489397488\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":732655,\"login\":\"KindOne-\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KindOne-\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/732655?\"},\"repo\":{\"id\":20344030,\"name\":\"irssi/irssi\",\"url\":\"https://api.github.com/repos/irssi/irssi\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/irssi/irssi/issues/194\",\"labels_url\":\"https://api.github.com/repos/irssi/irssi/issues/194/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/irssi/irssi/issues/194/comments\",\"events_url\":\"https://api.github.com/repos/irssi/irssi/issues/194/events\",\"html_url\":\"https://github.com/irssi/irssi/issues/194\",\"id\":53210242,\"number\":194,\"title\":\"Initial CTCP Action of an query goes into the status window. \",\"user\":{\"login\":\"KindOne-\",\"id\":732655,\"avatar_url\":\"https://avatars.githubusercontent.com/u/732655?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KindOne-\",\"html_url\":\"https://github.com/KindOne-\",\"followers_url\":\"https://api.github.com/users/KindOne-/followers\",\"following_url\":\"https://api.github.com/users/KindOne-/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KindOne-/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KindOne-/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KindOne-/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KindOne-/orgs\",\"repos_url\":\"https://api.github.com/users/KindOne-/repos\",\"events_url\":\"https://api.github.com/users/KindOne-/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KindOne-/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:45Z\",\"updated_at\":\"2015-01-01T01:03:45Z\",\"closed_at\":null,\"body\":\"```Irssi 0.8.18-head (20141216) - http://irssi.org/``` Inside cygwin  (can also replicate in Ubuntu 14.10)\\r\\n\\r\\nNot quite sure how to describe this, if someone sends me an ```/me ....``` and I do not have a query window with them,  that ```/me ....``` will get placed into the status window and irssi will also open a blank query window with that person.\\r\\n\\r\\n![irssi](https://cloud.githubusercontent.com/assets/732655/5591401/a7ba5096-9126-11e4-980e-7517e78fcd3c.PNG)\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:46Z\",\"org\":{\"id\":7750489,\"login\":\"irssi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/irssi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7750489?\"}}\n{\"id\":\"2489397493\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2829718,\"login\":\"phister\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/phister\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829718?\"},\"repo\":{\"id\":28678235,\"name\":\"phister/Cfb\",\"url\":\"https://api.github.com/repos/phister/Cfb\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"College football playoffs\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:47Z\"}\n{\"id\":\"2489397494\",\"type\":\"PushEvent\",\"actor\":{\"id\":8933459,\"login\":\"lmontopo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lmontopo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8933459?\"},\"repo\":{\"id\":24911711,\"name\":\"lmontopo/lmontopo.github.io\",\"url\":\"https://api.github.com/repos/lmontopo/lmontopo.github.io\"},\"payload\":{\"push_id\":536752973,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"06846c61830010de57ca6db802797133ceee8b7c\",\"before\":\"3511319efef8d59a76434c6884cbc57ab11b9aa5\",\"commits\":[{\"sha\":\"06846c61830010de57ca6db802797133ceee8b7c\",\"author\":{\"email\":\"6beaeb38ccda0977b766148789e916197124dfed@gmail.com\",\"name\":\"Leta Montopoli\"},\"message\":\"Generate Pelican site\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lmontopo/lmontopo.github.io/commits/06846c61830010de57ca6db802797133ceee8b7c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:47Z\"}\n{\"id\":\"2489397496\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1754250,\"login\":\"shinji-kono\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/shinji-kono\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1754250?\"},\"repo\":{\"id\":28678246,\"name\":\"shinji-kono/nmh-nkf\",\"url\":\"https://api.github.com/repos/shinji-kono/nmh-nkf\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"a branch of nmh for Japanese user. UTF8 only. scan, pick and mhl handle Japanese\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:47Z\"}\n{\"id\":\"2489397497\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":64050,\"login\":\"gjtorikian\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gjtorikian\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/64050?\"},\"repo\":{\"id\":10376685,\"name\":\"gjtorikian/markdowntutorial.com\",\"url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37\",\"labels_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/comments\",\"events_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/events\",\"html_url\":\"https://github.com/gjtorikian/markdowntutorial.com/issues/37\",\"id\":53166139,\"number\":37,\"title\":\"Second exercise in Lesson 6\",\"user\":{\"login\":\"cliffvick\",\"id\":72509,\"avatar_url\":\"https://avatars.githubusercontent.com/u/72509?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cliffvick\",\"html_url\":\"https://github.com/cliffvick\",\"followers_url\":\"https://api.github.com/users/cliffvick/followers\",\"following_url\":\"https://api.github.com/users/cliffvick/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cliffvick/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cliffvick/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cliffvick/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cliffvick/orgs\",\"repos_url\":\"https://api.github.com/users/cliffvick/repos\",\"events_url\":\"https://api.github.com/users/cliffvick/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cliffvick/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T04:07:58Z\",\"updated_at\":\"2015-01-01T01:03:47Z\",\"closed_at\":null,\"body\":\"The criteria for continuing on to the next exercise in the lesson is incorrect in the js/lesson_plans.coffee file (line 69).  The provided text omits the periods, but the lesson plan expects a period after each item.\"},\"comment\":{\"url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/comments/68477292\",\"html_url\":\"https://github.com/gjtorikian/markdowntutorial.com/issues/37#issuecomment-68477292\",\"issue_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37\",\"id\":68477292,\"user\":{\"login\":\"gjtorikian\",\"id\":64050,\"avatar_url\":\"https://avatars.githubusercontent.com/u/64050?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gjtorikian\",\"html_url\":\"https://github.com/gjtorikian\",\"followers_url\":\"https://api.github.com/users/gjtorikian/followers\",\"following_url\":\"https://api.github.com/users/gjtorikian/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gjtorikian/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gjtorikian/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gjtorikian/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gjtorikian/orgs\",\"repos_url\":\"https://api.github.com/users/gjtorikian/repos\",\"events_url\":\"https://api.github.com/users/gjtorikian/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gjtorikian/received_events\",\"type\":\"User\",\"site_admin\":true},\"created_at\":\"2015-01-01T01:03:47Z\",\"updated_at\":\"2015-01-01T01:03:47Z\",\"body\":\"Thanks for the report. The provided text *should* omit the periods, since it's initially a sequence with commas:\\r\\n\\r\\n![screen shot 2014-12-31 at 5 02 45 pm](https://cloud.githubusercontent.com/assets/64050/5591410/ddcaf0d6-910e-11e4-8f69-e496814afb5d.png)\\r\\n\\r\\nAs it's a list of proper sentences, you create the ordered list, and then provide periods:\\r\\n\\r\\n![screen shot 2014-12-31 at 5 02 31 pm](https://cloud.githubusercontent.com/assets/64050/5591411/e874c764-910e-11e4-9206-31ca6b39bda9.png)\\r\\n\\r\\nHope that helps. Please feel free to reopen if I've missed something. Happy new year!\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:47Z\"}\n{\"id\":\"2489397500\",\"type\":\"PushEvent\",\"actor\":{\"id\":409860,\"login\":\"flussence\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/flussence\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/409860?\"},\"repo\":{\"id\":24519248,\"name\":\"flussence/ebuilds\",\"url\":\"https://api.github.com/repos/flussence/ebuilds\"},\"payload\":{\"push_id\":536752975,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7b3ee839d933b0fbe405cf022d9e1003e867be6b\",\"before\":\"e1531f885892d899210024f54374a4f1880749dc\",\"commits\":[{\"sha\":\"7b3ee839d933b0fbe405cf022d9e1003e867be6b\",\"author\":{\"email\":\"ddb018d72dce150cce5ca65306dfca4b86fbc0bf@gmail.com\",\"name\":\"Anthony Parsons\"},\"message\":\"Recategorise sys-libs -> dev-libs/libtsm\\n\\nHopefully I did this right and nobody'll notice.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/flussence/ebuilds/commits/7b3ee839d933b0fbe405cf022d9e1003e867be6b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:48Z\"}\n{\"id\":\"2489397501\",\"type\":\"PushEvent\",\"actor\":{\"id\":1356088,\"login\":\"Zaryafaraj\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Zaryafaraj\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1356088?\"},\"repo\":{\"id\":26995510,\"name\":\"Fathalian/Guild\",\"url\":\"https://api.github.com/repos/Fathalian/Guild\"},\"payload\":{\"push_id\":536752976,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"eb4c4ba011689300a2f0ed76890368347f3ccc29\",\"before\":\"29abf0e942463dd27cf0666ddf54abda099859b2\",\"commits\":[{\"sha\":\"eb4c4ba011689300a2f0ed76890368347f3ccc29\",\"author\":{\"email\":\"de8898f6c55e335aa0a2b937fae65fb756ee038f@gmail.com\",\"name\":\"Zaryafaraj\"},\"message\":\"modal style change\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Fathalian/Guild/commits/eb4c4ba011689300a2f0ed76890368347f3ccc29\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:48Z\"}\n{\"id\":\"2489397507\",\"type\":\"PushEvent\",\"actor\":{\"id\":1785816,\"login\":\"natashavlahakis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/natashavlahakis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1785816?\"},\"repo\":{\"id\":27246389,\"name\":\"natashavlahakis/figtree\",\"url\":\"https://api.github.com/repos/natashavlahakis/figtree\"},\"payload\":{\"push_id\":536752979,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"03e086487f01d5fff1574e9906ad4a4e06082211\",\"before\":\"55edd29c904072c7dfaf84e396fe6bfb812f941e\",\"commits\":[{\"sha\":\"03e086487f01d5fff1574e9906ad4a4e06082211\",\"author\":{\"email\":\"84a888d1cdc9202fb22178d441aabd7da0eefd07@gmail.com\",\"name\":\"Natasha Vlahakis\"},\"message\":\"edited layout\",\"distinct\":true,\"url\":\"https://api.github.com/repos/natashavlahakis/figtree/commits/03e086487f01d5fff1574e9906ad4a4e06082211\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:49Z\"}\n{\"id\":\"2489397510\",\"type\":\"PushEvent\",\"actor\":{\"id\":436691,\"login\":\"lfarrell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lfarrell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/436691?\"},\"repo\":{\"id\":27356689,\"name\":\"lfarrell/DPLA-Metadata-Explorer\",\"url\":\"https://api.github.com/repos/lfarrell/DPLA-Metadata-Explorer\"},\"payload\":{\"push_id\":536752980,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"acf158b3040f94659f754c8ad186fc36f55cf0dd\",\"before\":\"f9ae98f6d8e374619499e96a23aea453a4f1f0be\",\"commits\":[{\"sha\":\"db329fb82c4c829f037c23d34c2a719ff5748c9d\",\"author\":{\"email\":\"b11d4f2999ccc74f2c93bc88775bb7be7fad72d9@gmail.com\",\"name\":\"Larry Farrell\"},\"message\":\"Zoom works in space between nodes now.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lfarrell/DPLA-Metadata-Explorer/commits/db329fb82c4c829f037c23d34c2a719ff5748c9d\"},{\"sha\":\"acf158b3040f94659f754c8ad186fc36f55cf0dd\",\"author\":{\"email\":\"b11d4f2999ccc74f2c93bc88775bb7be7fad72d9@gmail.com\",\"name\":\"Larry Farrell\"},\"message\":\"Add link out searching.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lfarrell/DPLA-Metadata-Explorer/commits/acf158b3040f94659f754c8ad186fc36f55cf0dd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:50Z\"}\n{\"id\":\"2489397511\",\"type\":\"PushEvent\",\"actor\":{\"id\":8289520,\"login\":\"BenAndy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/BenAndy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8289520?\"},\"repo\":{\"id\":22988319,\"name\":\"BenAndy/BenAndy.github.io\",\"url\":\"https://api.github.com/repos/BenAndy/BenAndy.github.io\"},\"payload\":{\"push_id\":536752981,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7a84f9d0a292787455f9b63ccd8a79ecc79960e3\",\"before\":\"cb2e14fdddd091905629a90787593292f4757e02\",\"commits\":[{\"sha\":\"7a84f9d0a292787455f9b63ccd8a79ecc79960e3\",\"author\":{\"email\":\"b4047af9f9c6db4af7a5f2314edc0d82e0e1385c@gmail.com\",\"name\":\"Ben Andy\"},\"message\":\"AutoPlug ALPHA test 0.2.55\",\"distinct\":true,\"url\":\"https://api.github.com/repos/BenAndy/BenAndy.github.io/commits/7a84f9d0a292787455f9b63ccd8a79ecc79960e3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:50Z\"}\n{\"id\":\"2489397512\",\"type\":\"PushEvent\",\"actor\":{\"id\":171043,\"login\":\"jeffnv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jeffnv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/171043?\"},\"repo\":{\"id\":28159277,\"name\":\"jeffnv/elite-golf\",\"url\":\"https://api.github.com/repos/jeffnv/elite-golf\"},\"payload\":{\"push_id\":536752982,\"size\":2,\"distinct_size\":0,\"ref\":\"refs/heads/gh-pages\",\"head\":\"12162bb9f224c179a9ea7d8abbac7dadec15ee2a\",\"before\":\"a5fe0bfb72e4489e4a22c90733db8392e1728ea0\",\"commits\":[{\"sha\":\"a88fa3bfffc1d1f4ce9b668412981ff071d9920e\",\"author\":{\"email\":\"a4a950aede9822deccc73582f88e82e913eb89d5@gmail.com\",\"name\":\"Jeff Fiddler\"},\"message\":\"course creator has undo now\",\"distinct\":false,\"url\":\"https://api.github.com/repos/jeffnv/elite-golf/commits/a88fa3bfffc1d1f4ce9b668412981ff071d9920e\"},{\"sha\":\"12162bb9f224c179a9ea7d8abbac7dadec15ee2a\",\"author\":{\"email\":\"a4a950aede9822deccc73582f88e82e913eb89d5@gmail.com\",\"name\":\"Jeff Fiddler\"},\"message\":\"more useful alert when clicking idle creator map\",\"distinct\":false,\"url\":\"https://api.github.com/repos/jeffnv/elite-golf/commits/12162bb9f224c179a9ea7d8abbac7dadec15ee2a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:50Z\"}\n{\"id\":\"2489397513\",\"type\":\"PushEvent\",\"actor\":{\"id\":7387879,\"login\":\"dsm-git\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dsm-git\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7387879?\"},\"repo\":{\"id\":28354666,\"name\":\"Door43/d43-ar-x-dcv\",\"url\":\"https://api.github.com/repos/Door43/d43-ar-x-dcv\"},\"payload\":{\"push_id\":536752983,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2bfd58ab618f202c44a4010ce9d2eb6b66da57a5\",\"before\":\"6158b50e1a2b200c5efba979f08248b153ced502\",\"commits\":[{\"sha\":\"2bfd58ab618f202c44a4010ce9d2eb6b66da57a5\",\"author\":{\"email\":\"62eb0db178518a8376b23676c2639eb2732c0be8@us.door43.org\",\"name\":\"Apache\"},\"message\":\"Page Edit [40]:  [EricWatt]\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Door43/d43-ar-x-dcv/commits/2bfd58ab618f202c44a4010ce9d2eb6b66da57a5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:50Z\",\"org\":{\"id\":4982125,\"login\":\"Door43\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Door43\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4982125?\"}}\n{\"id\":\"2489397515\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":3103764,\"login\":\"carymrobbins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?\"},\"repo\":{\"id\":15573192,\"name\":\"carymrobbins/intellij-haskforce\",\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/22397275\",\"id\":22397275,\"diff_hunk\":\"@@ -0,0 +1,56 @@\\n+package com.haskforce.highlighting.annotation;\\n+\\n+import com.google.common.collect.Lists;\\n+import com.haskforce.highlighting.annotation.external.GhcMod;\\n+import com.haskforce.highlighting.annotation.external.GhcModi;\\n+import com.haskforce.highlighting.annotation.external.HaskellExternalAnnotator;\\n+import com.haskforce.highlighting.annotation.external.TypeInfoUtil;\\n+import com.haskforce.settings.ToolKey;\\n+import com.haskforce.utils.ExecUtil;\\n+import com.intellij.lang.documentation.AbstractDocumentationProvider;\\n+import com.intellij.openapi.editor.Document;\\n+import com.intellij.openapi.editor.Editor;\\n+import com.intellij.openapi.editor.LogicalPosition;\\n+import com.intellij.openapi.editor.VisualPosition;\\n+import com.intellij.openapi.fileEditor.FileDocumentManager;\\n+import com.intellij.openapi.fileEditor.FileEditorManager;\\n+import com.intellij.openapi.module.Module;\\n+import com.intellij.openapi.module.ModuleUtilCore;\\n+import com.intellij.openapi.project.Project;\\n+import com.intellij.openapi.vfs.VirtualFile;\\n+import com.intellij.psi.PsiElement;\\n+import com.intellij.psi.PsiFile;\\n+import com.intellij.psi.PsiManager;\\n+import org.jetbrains.annotations.Nullable;\\n+\\n+import java.awt.*;\\n+import java.util.List;\\n+import java.util.concurrent.ExecutionException;\\n+\\n+public class HaskellDocumentationProvider extends AbstractDocumentationProvider {\",\"path\":\"src/com/haskforce/highlighting/annotation/HaskellDocumentationProvider.java\",\"position\":30,\"original_position\":30,\"commit_id\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"original_commit_id\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"user\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This doesn't seem to work for me...not sure if I'm doing something wrong.\",\"created_at\":\"2015-01-01T01:03:50Z\",\"updated_at\":\"2015-01-01T01:03:50Z\",\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105#discussion_r22397275\",\"pull_request_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/22397275\"},\"html\":{\"href\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105#discussion_r22397275\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\",\"id\":26615813,\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\",\"diff_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.diff\",\"patch_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.patch\",\"issue_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\",\"number\":105,\"state\":\"open\",\"locked\":false,\"title\":\"Issue90 type information\",\"user\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Cary,\\r\\n\\r\\nFirst try of the type information. It seems quite stable, been using it for a few days (provided the configuration is correct, I suppose, didn't test what happens when ghc-modi is not correctly configured).\\r\\n\\r\\nI bound the type info call to the DocumentationProvider as well as to an action (alt - equals, like scala). I prefer the action, I think the documentation provider doesn't work so well. There are tests of the parsing of the output of ghc-modi, but not really of the documentationprovider, basically because of the abundance of static calls and the fact that I think they can only be mocked while testing, and statics can only be mocked through Powermock if I recall correctly, which would mean an extra test dependency and so on and so forth, so I left that to be your call.\\r\\n\\r\\nAlso, there is a weird behaviour that getting the editor creates a stack trace of around 5 kilometers long, something that seems like a threading issue, but the function seems to work. I don't really know why the stack trace happens, all the more because it only happens when calling the type information through the documentation provider (same code path is used when the action is called, but no stack trace). Maybe you know more what could go wrong, it looks like something intellij-related.\\r\\n\\r\\nSo, consider this a \\\"request for comment\\\" more than a pull request ;-)\\r\\n\\r\\nKasper\",\"created_at\":\"2014-12-27T16:19:07Z\",\"updated_at\":\"2015-01-01T01:03:50Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"b32a5f3ef0bc14eebea4baa946ec2e66008086dd\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/commits\",\"review_comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/comments\",\"review_comment_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/comments\",\"statuses_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"head\":{\"label\":\"KasperJanssens:issue90TypeInformation\",\"ref\":\"issue90TypeInformation\",\"sha\":\"73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\",\"user\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28540684,\"name\":\"intellij-haskforce\",\"full_name\":\"KasperJanssens/intellij-haskforce\",\"owner\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/KasperJanssens/intellij-haskforce\",\"description\":\"Haskell plugin for IntelliJ IDEA Community Edition\",\"fork\":true,\"url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce\",\"forks_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/forks\",\"keys_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/teams\",\"hooks_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/hooks\",\"issue_events_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/events\",\"assignees_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/tags\",\"blobs_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/languages\",\"stargazers_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/stargazers\",\"contributors_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/contributors\",\"subscribers_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/subscribers\",\"subscription_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/subscription\",\"commits_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/merges\",\"archive_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/downloads\",\"issues_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/KasperJanssens/intellij-haskforce/releases{/id}\",\"created_at\":\"2014-12-27T16:11:01Z\",\"updated_at\":\"2014-12-27T16:11:03Z\",\"pushed_at\":\"2014-12-31T15:44:48Z\",\"git_url\":\"git://github.com/KasperJanssens/intellij-haskforce.git\",\"ssh_url\":\"git@github.com:KasperJanssens/intellij-haskforce.git\",\"clone_url\":\"https://github.com/KasperJanssens/intellij-haskforce.git\",\"svn_url\":\"https://github.com/KasperJanssens/intellij-haskforce\",\"homepage\":\"http://carymrobbins.github.io/intellij-haskforce/\",\"size\":7553,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Java\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"carymrobbins:master\",\"ref\":\"master\",\"sha\":\"19442eb58293650dc699e30ccf2d5482e1c688d8\",\"user\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15573192,\"name\":\"intellij-haskforce\",\"full_name\":\"carymrobbins/intellij-haskforce\",\"owner\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce\",\"description\":\"Haskell plugin for IntelliJ IDEA Community Edition\",\"fork\":false,\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce\",\"forks_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/forks\",\"keys_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/teams\",\"hooks_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/hooks\",\"issue_events_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/events\",\"assignees_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/tags\",\"blobs_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/languages\",\"stargazers_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/stargazers\",\"contributors_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/contributors\",\"subscribers_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/subscribers\",\"subscription_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/subscription\",\"commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/merges\",\"archive_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/downloads\",\"issues_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/releases{/id}\",\"created_at\":\"2014-01-02T01:28:54Z\",\"updated_at\":\"2014-12-30T16:15:50Z\",\"pushed_at\":\"2014-12-27T04:28:50Z\",\"git_url\":\"git://github.com/carymrobbins/intellij-haskforce.git\",\"ssh_url\":\"git@github.com:carymrobbins/intellij-haskforce.git\",\"clone_url\":\"https://github.com/carymrobbins/intellij-haskforce.git\",\"svn_url\":\"https://github.com/carymrobbins/intellij-haskforce\",\"homepage\":\"http://carymrobbins.github.io/intellij-haskforce/\",\"size\":11390,\"stargazers_count\":159,\"watchers_count\":159,\"language\":\"Java\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":9,\"mirror_url\":null,\"open_issues_count\":30,\"forks\":9,\"open_issues\":30,\"watchers\":159,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\"},\"html\":{\"href\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\"},\"issue\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\"},\"comments\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/statuses/73ec576f11ba5aa0906e1ca03bc9390ca94b40e2\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:03:50Z\"}\n{\"id\":\"2489397517\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1742369,\"login\":\"gbathree\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gbathree\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1742369?\"},\"repo\":{\"id\":20705156,\"name\":\"Photosynq/PhotosynQ-ChromeApp\",\"url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/15\",\"labels_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/15/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/15/comments\",\"events_url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/issues/15/events\",\"html_url\":\"https://github.com/Photosynq/PhotosynQ-ChromeApp/issues/15\",\"id\":53210243,\"number\":15,\"title\":\"add 'light_intensity_raw\\\" - this measures the raw value from the light intensity sensor\",\"user\":{\"login\":\"gbathree\",\"id\":1742369,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1742369?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gbathree\",\"html_url\":\"https://github.com/gbathree\",\"followers_url\":\"https://api.github.com/users/gbathree/followers\",\"following_url\":\"https://api.github.com/users/gbathree/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gbathree/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gbathree/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gbathree/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gbathree/orgs\",\"repos_url\":\"https://api.github.com/users/gbathree/repos\",\"events_url\":\"https://api.github.com/users/gbathree/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gbathree/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/Photosynq/PhotosynQ-ChromeApp/labels/enhancement\",\"name\":\"enhancement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:03:50Z\",\"updated_at\":\"2015-01-01T01:03:50Z\",\"closed_at\":null,\"body\":\"Works exactly like \\\"light_intensity\\\".  Also, please change 'light intensity [lumen]' to 'light intensity [micro-einsteins]' (this also applies to data explorer)\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:52Z\",\"org\":{\"id\":5068236,\"login\":\"Photosynq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Photosynq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5068236?\"}}\n{\"id\":\"2489397524\",\"type\":\"PushEvent\",\"actor\":{\"id\":906529,\"login\":\"dpwolfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpwolfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/906529?\"},\"repo\":{\"id\":26579311,\"name\":\"dpwolfe/otucha\",\"url\":\"https://api.github.com/repos/dpwolfe/otucha\"},\"payload\":{\"push_id\":536752985,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6f79c0b71c6d4a7e3a7d538ed3b96a889c2a33e0\",\"before\":\"a9a695f1f7a3395f7394ae668feab71f35a5f622\",\"commits\":[{\"sha\":\"6f79c0b71c6d4a7e3a7d538ed3b96a889c2a33e0\",\"author\":{\"email\":\"b7dfe270ecb2603aba704ea15b776485da19da15@gmail.com\",\"name\":\"David Wolfe\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpwolfe/otucha/commits/6f79c0b71c6d4a7e3a7d538ed3b96a889c2a33e0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:52Z\"}\n{\"id\":\"2489397525\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":1398544,\"login\":\"joelpurra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joelpurra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1398544?\"},\"repo\":{\"id\":28595466,\"name\":\"joelpurra/jqnpm\",\"url\":\"https://api.github.com/repos/joelpurra/jqnpm\"},\"payload\":{\"ref\":\"import_659\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:52Z\"}\n{\"id\":\"2489397528\",\"type\":\"PushEvent\",\"actor\":{\"id\":3187965,\"login\":\"kuanslove\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kuanslove\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3187965?\"},\"repo\":{\"id\":27238819,\"name\":\"kuanslove/kuanslove.github.io\",\"url\":\"https://api.github.com/repos/kuanslove/kuanslove.github.io\"},\"payload\":{\"push_id\":536752988,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"e754d70538d1113870734d3cb9a54ceaf708297b\",\"before\":\"2a58eea671292f434c57c8d4126da39793759d58\",\"commits\":[{\"sha\":\"f2395a46dd91bf7d50caf080f4dd19b24445b8e0\",\"author\":{\"email\":\"5cf2a1a0b5edf64f8c031f9fcd867f256dcbfa6e@hotmail.com\",\"name\":\"kuanslove\"},\"message\":\"add right menu icon and adjust move effect\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kuanslove/kuanslove.github.io/commits/f2395a46dd91bf7d50caf080f4dd19b24445b8e0\"},{\"sha\":\"e754d70538d1113870734d3cb9a54ceaf708297b\",\"author\":{\"email\":\"5cf2a1a0b5edf64f8c031f9fcd867f256dcbfa6e@hotmail.com\",\"name\":\"kuanslove\"},\"message\":\"merge from remote\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kuanslove/kuanslove.github.io/commits/e754d70538d1113870734d3cb9a54ceaf708297b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:53Z\"}\n{\"id\":\"2489397530\",\"type\":\"PushEvent\",\"actor\":{\"id\":3196313,\"login\":\"dpastoor\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpastoor\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3196313?\"},\"repo\":{\"id\":16302099,\"name\":\"dpastoor/PKPDmisc\",\"url\":\"https://api.github.com/repos/dpastoor/PKPDmisc\"},\"payload\":{\"push_id\":536752990,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fd066526d30949d63be752722534a2354edcc5d9\",\"before\":\"81e56dd1160f5bbe8628cb61de33f4636e4588f0\",\"commits\":[{\"sha\":\"fd066526d30949d63be752722534a2354edcc5d9\",\"author\":{\"email\":\"5480d79f0d3b1ca7acc7421688b095f8d1e51564@gmail.com\",\"name\":\"Devin Pastoor\"},\"message\":\"add pauc documentation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpastoor/PKPDmisc/commits/fd066526d30949d63be752722534a2354edcc5d9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:53Z\"}\n{\"id\":\"2489397533\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":19792,\"login\":\"wiredfool\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?\"},\"repo\":{\"id\":5171600,\"name\":\"python-pillow/Pillow\",\"url\":\"https://api.github.com/repos/python-pillow/Pillow\"},\"payload\":{\"action\":\"opened\",\"number\":1061,\"pull_request\":{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061\",\"id\":26739441,\"html_url\":\"https://github.com/python-pillow/Pillow/pull/1061\",\"diff_url\":\"https://github.com/python-pillow/Pillow/pull/1061.diff\",\"patch_url\":\"https://github.com/python-pillow/Pillow/pull/1061.patch\",\"issue_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1061\",\"number\":1061,\"state\":\"open\",\"locked\":false,\"title\":\"Release notes\",\"user\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Changes/additions for docs and release notes. \",\"created_at\":\"2015-01-01T01:03:54Z\",\"updated_at\":\"2015-01-01T01:03:54Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061/commits\",\"review_comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061/comments\",\"review_comment_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1061/comments\",\"statuses_url\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/80c7dcbce47437e5fb3fdb05c1fbbdbae0b31e88\",\"head\":{\"label\":\"wiredfool:release-notes\",\"ref\":\"release-notes\",\"sha\":\"80c7dcbce47437e5fb3fdb05c1fbbdbae0b31e88\",\"user\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":8664379,\"name\":\"Pillow\",\"full_name\":\"wiredfool/Pillow\",\"owner\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/wiredfool/Pillow\",\"description\":\"Pillow is the \\\"friendly\\\" PIL fork\",\"fork\":true,\"url\":\"https://api.github.com/repos/wiredfool/Pillow\",\"forks_url\":\"https://api.github.com/repos/wiredfool/Pillow/forks\",\"keys_url\":\"https://api.github.com/repos/wiredfool/Pillow/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/wiredfool/Pillow/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/wiredfool/Pillow/teams\",\"hooks_url\":\"https://api.github.com/repos/wiredfool/Pillow/hooks\",\"issue_events_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/wiredfool/Pillow/events\",\"assignees_url\":\"https://api.github.com/repos/wiredfool/Pillow/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/wiredfool/Pillow/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/wiredfool/Pillow/tags\",\"blobs_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/wiredfool/Pillow/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/wiredfool/Pillow/languages\",\"stargazers_url\":\"https://api.github.com/repos/wiredfool/Pillow/stargazers\",\"contributors_url\":\"https://api.github.com/repos/wiredfool/Pillow/contributors\",\"subscribers_url\":\"https://api.github.com/repos/wiredfool/Pillow/subscribers\",\"subscription_url\":\"https://api.github.com/repos/wiredfool/Pillow/subscription\",\"commits_url\":\"https://api.github.com/repos/wiredfool/Pillow/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/wiredfool/Pillow/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/wiredfool/Pillow/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/wiredfool/Pillow/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/wiredfool/Pillow/merges\",\"archive_url\":\"https://api.github.com/repos/wiredfool/Pillow/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/wiredfool/Pillow/downloads\",\"issues_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/wiredfool/Pillow/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/wiredfool/Pillow/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/wiredfool/Pillow/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/wiredfool/Pillow/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/wiredfool/Pillow/releases{/id}\",\"created_at\":\"2013-03-09T03:12:08Z\",\"updated_at\":\"2014-12-31T23:15:11Z\",\"pushed_at\":\"2015-01-01T00:37:45Z\",\"git_url\":\"git://github.com/wiredfool/Pillow.git\",\"ssh_url\":\"git@github.com:wiredfool/Pillow.git\",\"clone_url\":\"https://github.com/wiredfool/Pillow.git\",\"svn_url\":\"https://github.com/wiredfool/Pillow\",\"homepage\":\"http://python-imaging.github.com\",\"size\":14870,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"Python\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"python-pillow:master\",\"ref\":\"master\",\"sha\":\"4889ae776ccbf27a7f955204067812f2aceefba3\",\"user\":{\"login\":\"python-pillow\",\"id\":2036701,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/python-pillow\",\"html_url\":\"https://github.com/python-pillow\",\"followers_url\":\"https://api.github.com/users/python-pillow/followers\",\"following_url\":\"https://api.github.com/users/python-pillow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/python-pillow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/python-pillow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/python-pillow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/python-pillow/orgs\",\"repos_url\":\"https://api.github.com/users/python-pillow/repos\",\"events_url\":\"https://api.github.com/users/python-pillow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/python-pillow/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":5171600,\"name\":\"Pillow\",\"full_name\":\"python-pillow/Pillow\",\"owner\":{\"login\":\"python-pillow\",\"id\":2036701,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/python-pillow\",\"html_url\":\"https://github.com/python-pillow\",\"followers_url\":\"https://api.github.com/users/python-pillow/followers\",\"following_url\":\"https://api.github.com/users/python-pillow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/python-pillow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/python-pillow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/python-pillow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/python-pillow/orgs\",\"repos_url\":\"https://api.github.com/users/python-pillow/repos\",\"events_url\":\"https://api.github.com/users/python-pillow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/python-pillow/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/python-pillow/Pillow\",\"description\":\"The friendly PIL fork\",\"fork\":false,\"url\":\"https://api.github.com/repos/python-pillow/Pillow\",\"forks_url\":\"https://api.github.com/repos/python-pillow/Pillow/forks\",\"keys_url\":\"https://api.github.com/repos/python-pillow/Pillow/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/python-pillow/Pillow/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/python-pillow/Pillow/teams\",\"hooks_url\":\"https://api.github.com/repos/python-pillow/Pillow/hooks\",\"issue_events_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/python-pillow/Pillow/events\",\"assignees_url\":\"https://api.github.com/repos/python-pillow/Pillow/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/python-pillow/Pillow/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/python-pillow/Pillow/tags\",\"blobs_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/python-pillow/Pillow/languages\",\"stargazers_url\":\"https://api.github.com/repos/python-pillow/Pillow/stargazers\",\"contributors_url\":\"https://api.github.com/repos/python-pillow/Pillow/contributors\",\"subscribers_url\":\"https://api.github.com/repos/python-pillow/Pillow/subscribers\",\"subscription_url\":\"https://api.github.com/repos/python-pillow/Pillow/subscription\",\"commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/python-pillow/Pillow/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/python-pillow/Pillow/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/python-pillow/Pillow/merges\",\"archive_url\":\"https://api.github.com/repos/python-pillow/Pillow/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/python-pillow/Pillow/downloads\",\"issues_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/python-pillow/Pillow/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/python-pillow/Pillow/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/python-pillow/Pillow/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/python-pillow/Pillow/releases{/id}\",\"created_at\":\"2012-07-24T21:38:39Z\",\"updated_at\":\"2014-12-31T22:44:33Z\",\"pushed_at\":\"2014-12-31T22:44:33Z\",\"git_url\":\"git://github.com/python-pillow/Pillow.git\",\"ssh_url\":\"git@github.com:python-pillow/Pillow.git\",\"clone_url\":\"https://github.com/python-pillow/Pillow.git\",\"svn_url\":\"https://github.com/python-pillow/Pillow\",\"homepage\":\"http://python-pillow.github.io/\",\"size\":20365,\"stargazers_count\":1391,\"watchers_count\":1391,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":319,\"mirror_url\":null,\"open_issues_count\":64,\"forks\":319,\"open_issues\":64,\"watchers\":1391,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061\"},\"html\":{\"href\":\"https://github.com/python-pillow/Pillow/pull/1061\"},\"issue\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1061\"},\"comments\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1061/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1061/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/80c7dcbce47437e5fb3fdb05c1fbbdbae0b31e88\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":2,\"additions\":32,\"deletions\":2,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:03:55Z\",\"org\":{\"id\":2036701,\"login\":\"python-pillow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python-pillow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?\"}}\n{\"id\":\"2489397536\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536752992,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"44f7054077780dd70b8ad070b9b04640a65282ec\",\"before\":\"b3500640ddd547415a82cbc026d91be7f76ff14f\",\"commits\":[{\"sha\":\"b35627f00738be747dc53479874ce39c15437c68\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Added comments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/b35627f00738be747dc53479874ce39c15437c68\"},{\"sha\":\"44f7054077780dd70b8ad070b9b04640a65282ec\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Merge branch 'master' of https://github.com/jlumijarvi/csv2xml.git\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/44f7054077780dd70b8ad070b9b04640a65282ec\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:55Z\"}\n{\"id\":\"2489397537\",\"type\":\"PushEvent\",\"actor\":{\"id\":4332428,\"login\":\"mengqhai\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mengqhai\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4332428?\"},\"repo\":{\"id\":16940729,\"name\":\"mengqhai/appbasement\",\"url\":\"https://api.github.com/repos/mengqhai/appbasement\"},\"payload\":{\"push_id\":536752993,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"74fd11ab7ae2271f59aa5515885a015ce60e7e58\",\"before\":\"7690349b1b0ab8689df9049dced11b17a573f225\",\"commits\":[{\"sha\":\"74fd11ab7ae2271f59aa5515885a015ce60e7e58\",\"author\":{\"email\":\"06b3b513462d14ed6ea9d26d510982ff74cb83ea@gmail.com\",\"name\":\"Qinghai Meng\"},\"message\":\"re-implemented the revision with JPA in a separated table\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mengqhai/appbasement/commits/74fd11ab7ae2271f59aa5515885a015ce60e7e58\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:55Z\"}\n{\"id\":\"2489397540\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":64050,\"login\":\"gjtorikian\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gjtorikian\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/64050?\"},\"repo\":{\"id\":10376685,\"name\":\"gjtorikian/markdowntutorial.com\",\"url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37\",\"labels_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/comments\",\"events_url\":\"https://api.github.com/repos/gjtorikian/markdowntutorial.com/issues/37/events\",\"html_url\":\"https://github.com/gjtorikian/markdowntutorial.com/issues/37\",\"id\":53166139,\"number\":37,\"title\":\"Second exercise in Lesson 6\",\"user\":{\"login\":\"cliffvick\",\"id\":72509,\"avatar_url\":\"https://avatars.githubusercontent.com/u/72509?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cliffvick\",\"html_url\":\"https://github.com/cliffvick\",\"followers_url\":\"https://api.github.com/users/cliffvick/followers\",\"following_url\":\"https://api.github.com/users/cliffvick/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cliffvick/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cliffvick/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cliffvick/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cliffvick/orgs\",\"repos_url\":\"https://api.github.com/users/cliffvick/repos\",\"events_url\":\"https://api.github.com/users/cliffvick/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cliffvick/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T04:07:58Z\",\"updated_at\":\"2015-01-01T01:03:56Z\",\"closed_at\":\"2015-01-01T01:03:56Z\",\"body\":\"The criteria for continuing on to the next exercise in the lesson is incorrect in the js/lesson_plans.coffee file (line 69).  The provided text omits the periods, but the lesson plan expects a period after each item.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397542\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":4296161,\"login\":\"jefftune\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jefftune\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4296161?\"},\"repo\":{\"id\":25269705,\"name\":\"MobileAppTracking/tune-reporting-python\",\"url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/issues/71\",\"labels_url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/issues/71/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/issues/71/comments\",\"events_url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/issues/71/events\",\"html_url\":\"https://github.com/MobileAppTracking/tune-reporting-python/issues/71\",\"id\":53210246,\"number\":71,\"title\":\"Python -- Set Configuration\",\"user\":{\"login\":\"jefftune\",\"id\":4296161,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4296161?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jefftune\",\"html_url\":\"https://github.com/jefftune\",\"followers_url\":\"https://api.github.com/users/jefftune/followers\",\"following_url\":\"https://api.github.com/users/jefftune/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jefftune/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jefftune/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jefftune/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jefftune/orgs\",\"repos_url\":\"https://api.github.com/users/jefftune/repos\",\"events_url\":\"https://api.github.com/users/jefftune/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jefftune/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/labels/Development\",\"name\":\"Development\",\"color\":\"0052cc\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"jefftune\",\"id\":4296161,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4296161?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jefftune\",\"html_url\":\"https://github.com/jefftune\",\"followers_url\":\"https://api.github.com/users/jefftune/followers\",\"following_url\":\"https://api.github.com/users/jefftune/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jefftune/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jefftune/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jefftune/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jefftune/orgs\",\"repos_url\":\"https://api.github.com/users/jefftune/repos\",\"events_url\":\"https://api.github.com/users/jefftune/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jefftune/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/milestones/5\",\"labels_url\":\"https://api.github.com/repos/MobileAppTracking/tune-reporting-python/milestones/5/labels\",\"id\":899375,\"number\":5,\"title\":\"2014-12-30\",\"description\":\"\",\"creator\":{\"login\":\"jefftune\",\"id\":4296161,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4296161?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jefftune\",\"html_url\":\"https://github.com/jefftune\",\"followers_url\":\"https://api.github.com/users/jefftune/followers\",\"following_url\":\"https://api.github.com/users/jefftune/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jefftune/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jefftune/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jefftune/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jefftune/orgs\",\"repos_url\":\"https://api.github.com/users/jefftune/repos\",\"events_url\":\"https://api.github.com/users/jefftune/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jefftune/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":2,\"closed_issues\":7,\"state\":\"open\",\"created_at\":\"2014-12-10T19:53:05Z\",\"updated_at\":\"2015-01-01T01:03:56Z\",\"due_on\":\"2014-12-30T08:00:00Z\",\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:03:56Z\",\"updated_at\":\"2015-01-01T01:03:56Z\",\"closed_at\":null,\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\",\"org\":{\"id\":1920066,\"login\":\"MobileAppTracking\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MobileAppTracking\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1920066?\"}}\n{\"id\":\"2489397543\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":13026,\"login\":\"titanous\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/titanous\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/13026?\"},\"repo\":{\"id\":11290232,\"name\":\"flynn/flynn\",\"url\":\"https://api.github.com/repos/flynn/flynn\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/flynn/flynn/issues/547\",\"labels_url\":\"https://api.github.com/repos/flynn/flynn/issues/547/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/flynn/flynn/issues/547/comments\",\"events_url\":\"https://api.github.com/repos/flynn/flynn/issues/547/events\",\"html_url\":\"https://github.com/flynn/flynn/pull/547\",\"id\":50875297,\"number\":547,\"title\":\"Add queue package\",\"user\":{\"login\":\"lmars\",\"id\":488515,\"avatar_url\":\"https://avatars.githubusercontent.com/u/488515?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lmars\",\"html_url\":\"https://github.com/lmars\",\"followers_url\":\"https://api.github.com/users/lmars/followers\",\"following_url\":\"https://api.github.com/users/lmars/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/lmars/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/lmars/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/lmars/subscriptions\",\"organizations_url\":\"https://api.github.com/users/lmars/orgs\",\"repos_url\":\"https://api.github.com/users/lmars/repos\",\"events_url\":\"https://api.github.com/users/lmars/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/lmars/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":12,\"created_at\":\"2014-12-03T18:42:18Z\",\"updated_at\":\"2015-01-01T01:03:55Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/flynn/flynn/pulls/547\",\"html_url\":\"https://github.com/flynn/flynn/pull/547\",\"diff_url\":\"https://github.com/flynn/flynn/pull/547.diff\",\"patch_url\":\"https://github.com/flynn/flynn/pull/547.patch\"},\"body\":\"This is based on the [Ruby QueueClassic library](https://github.com/QueueClassic/queue_classic).\\r\\n\\r\\nOpening now for early feedback.\\r\\n\\r\\nPrerequisite for #536 \"},\"comment\":{\"url\":\"https://api.github.com/repos/flynn/flynn/issues/comments/68477294\",\"html_url\":\"https://github.com/flynn/flynn/pull/547#issuecomment-68477294\",\"issue_url\":\"https://api.github.com/repos/flynn/flynn/issues/547\",\"id\":68477294,\"user\":{\"login\":\"titanous\",\"id\":13026,\"avatar_url\":\"https://avatars.githubusercontent.com/u/13026?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/titanous\",\"html_url\":\"https://github.com/titanous\",\"followers_url\":\"https://api.github.com/users/titanous/followers\",\"following_url\":\"https://api.github.com/users/titanous/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/titanous/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/titanous/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/titanous/subscriptions\",\"organizations_url\":\"https://api.github.com/users/titanous/orgs\",\"repos_url\":\"https://api.github.com/users/titanous/repos\",\"events_url\":\"https://api.github.com/users/titanous/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/titanous/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:03:55Z\",\"updated_at\":\"2015-01-01T01:03:55Z\",\"body\":\"It's worth looking at https://github.com/bgentry/que-go which claims to use faster queries and appears to solve the locking issues by using a different pg driver which makes more guarantees about connection usage.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\",\"org\":{\"id\":4973613,\"login\":\"flynn\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/flynn\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4973613?\"}}\n{\"id\":\"2489397544\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":20197962,\"name\":\"kenjiSpecial/100day-canvas-bootcamp-training\",\"url\":\"https://api.github.com/repos/kenjiSpecial/100day-canvas-bootcamp-training\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397548\",\"type\":\"ForkEvent\",\"actor\":{\"id\":3395224,\"login\":\"sunnyy02\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sunnyy02\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3395224?\"},\"repo\":{\"id\":18972654,\"name\":\"jquintus/spikes\",\"url\":\"https://api.github.com/repos/jquintus/spikes\"},\"payload\":{\"forkee\":{\"id\":28678247,\"name\":\"spikes\",\"full_name\":\"sunnyy02/spikes\",\"owner\":{\"login\":\"sunnyy02\",\"id\":3395224,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3395224?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sunnyy02\",\"html_url\":\"https://github.com/sunnyy02\",\"followers_url\":\"https://api.github.com/users/sunnyy02/followers\",\"following_url\":\"https://api.github.com/users/sunnyy02/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sunnyy02/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sunnyy02/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sunnyy02/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sunnyy02/orgs\",\"repos_url\":\"https://api.github.com/users/sunnyy02/repos\",\"events_url\":\"https://api.github.com/users/sunnyy02/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sunnyy02/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/sunnyy02/spikes\",\"description\":\"Test and simple sample code.  Nothing fancy, lots of dead ends.  No dragons.\",\"fork\":true,\"url\":\"https://api.github.com/repos/sunnyy02/spikes\",\"forks_url\":\"https://api.github.com/repos/sunnyy02/spikes/forks\",\"keys_url\":\"https://api.github.com/repos/sunnyy02/spikes/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/sunnyy02/spikes/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/sunnyy02/spikes/teams\",\"hooks_url\":\"https://api.github.com/repos/sunnyy02/spikes/hooks\",\"issue_events_url\":\"https://api.github.com/repos/sunnyy02/spikes/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/sunnyy02/spikes/events\",\"assignees_url\":\"https://api.github.com/repos/sunnyy02/spikes/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/sunnyy02/spikes/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/sunnyy02/spikes/tags\",\"blobs_url\":\"https://api.github.com/repos/sunnyy02/spikes/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/sunnyy02/spikes/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/sunnyy02/spikes/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/sunnyy02/spikes/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/sunnyy02/spikes/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/sunnyy02/spikes/languages\",\"stargazers_url\":\"https://api.github.com/repos/sunnyy02/spikes/stargazers\",\"contributors_url\":\"https://api.github.com/repos/sunnyy02/spikes/contributors\",\"subscribers_url\":\"https://api.github.com/repos/sunnyy02/spikes/subscribers\",\"subscription_url\":\"https://api.github.com/repos/sunnyy02/spikes/subscription\",\"commits_url\":\"https://api.github.com/repos/sunnyy02/spikes/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/sunnyy02/spikes/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/sunnyy02/spikes/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/sunnyy02/spikes/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/sunnyy02/spikes/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/sunnyy02/spikes/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/sunnyy02/spikes/merges\",\"archive_url\":\"https://api.github.com/repos/sunnyy02/spikes/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/sunnyy02/spikes/downloads\",\"issues_url\":\"https://api.github.com/repos/sunnyy02/spikes/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/sunnyy02/spikes/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/sunnyy02/spikes/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/sunnyy02/spikes/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/sunnyy02/spikes/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/sunnyy02/spikes/releases{/id}\",\"created_at\":\"2015-01-01T01:03:57Z\",\"updated_at\":\"2014-12-27T20:43:44Z\",\"pushed_at\":\"2014-12-04T21:54:59Z\",\"git_url\":\"git://github.com/sunnyy02/spikes.git\",\"ssh_url\":\"git@github.com:sunnyy02/spikes.git\",\"clone_url\":\"https://github.com/sunnyy02/spikes.git\",\"svn_url\":\"https://github.com/sunnyy02/spikes\",\"homepage\":null,\"size\":24175,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397553\",\"type\":\"PushEvent\",\"actor\":{\"id\":8508709,\"login\":\"raabbajam\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/raabbajam\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8508709?\"},\"repo\":{\"id\":26143713,\"name\":\"raabbajam/priceCacheCalendar\",\"url\":\"https://api.github.com/repos/raabbajam/priceCacheCalendar\"},\"payload\":{\"push_id\":536752996,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7153d3eb326cc465ab28f1c1fe366343773cce1e\",\"before\":\"d53d4a81198a108a2eb66fd506dc3e7f159a6a23\",\"commits\":[{\"sha\":\"7153d3eb326cc465ab28f1c1fe366343773cce1e\",\"author\":{\"email\":\"a6d869d99a512698feb7bc061802b95b71e0b394@gmail.com\",\"name\":\"Muhammad Abdul Jabbaar\"},\"message\":\"sr@looper change parallel\",\"distinct\":true,\"url\":\"https://api.github.com/repos/raabbajam/priceCacheCalendar/commits/7153d3eb326cc465ab28f1c1fe366343773cce1e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397555\",\"type\":\"PushEvent\",\"actor\":{\"id\":7671394,\"login\":\"seinosuke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/seinosuke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7671394?\"},\"repo\":{\"id\":27248287,\"name\":\"seinosuke/moritan_bot\",\"url\":\"https://api.github.com/repos/seinosuke/moritan_bot\"},\"payload\":{\"push_id\":536752998,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cd2a54765a2ef7c40e377466bb873a22787c93e1\",\"before\":\"eff699194d79382606198a11cc257715fac32073\",\"commits\":[{\"sha\":\"cd2a54765a2ef7c40e377466bb873a22787c93e1\",\"author\":{\"email\":\"bf9e3b55334d2166aced4606c08396ce6b193d78@gmail.com\",\"name\":\"seinosuke\"},\"message\":\"Don't repeat yourself入門\",\"distinct\":true,\"url\":\"https://api.github.com/repos/seinosuke/moritan_bot/commits/cd2a54765a2ef7c40e377466bb873a22787c93e1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397556\",\"type\":\"PushEvent\",\"actor\":{\"id\":2829600,\"login\":\"GrahamCampbell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GrahamCampbell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829600?\"},\"repo\":{\"id\":26726717,\"name\":\"StyleCI/Fixer\",\"url\":\"https://api.github.com/repos/StyleCI/Fixer\"},\"payload\":{\"push_id\":536752999,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f0214192687388ffe5167091fa3fa4fe4db0139c\",\"before\":\"c99705b02b76d09d6fbc1734421ef4ef8207d943\",\"commits\":[{\"sha\":\"f0214192687388ffe5167091fa3fa4fe4db0139c\",\"author\":{\"email\":\"5ca27e75aea3e5e83a04c6cfa5f1b63d358cd03d@mineuk.com\",\"name\":\"Graham Campbell\"},\"message\":\"Updated license noticies\",\"distinct\":true,\"url\":\"https://api.github.com/repos/StyleCI/Fixer/commits/f0214192687388ffe5167091fa3fa4fe4db0139c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\",\"org\":{\"id\":10179029,\"login\":\"StyleCI\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/StyleCI\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10179029?\"}}\n{\"id\":\"2489397561\",\"type\":\"PushEvent\",\"actor\":{\"id\":238354,\"login\":\"variousred\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/variousred\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/238354?\"},\"repo\":{\"id\":6274404,\"name\":\"G5/g5-content-management-system\",\"url\":\"https://api.github.com/repos/G5/g5-content-management-system\"},\"payload\":{\"push_id\":536753002,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/fix-deep-clone\",\"head\":\"81baf7738ce7da9084ecd6f657cd3c0e60f888a2\",\"before\":\"e77a319f12682833e503da292a55f3a5f32cde28\",\"commits\":[{\"sha\":\"81baf7738ce7da9084ecd6f657cd3c0e60f888a2\",\"author\":{\"email\":\"16ea8d52c08316685a257e07ddecf7165a502f6d@gmail.com\",\"name\":\"Michael Mitchell\"},\"message\":\"refactors specs,\\n@ckeckert this one failing test remaining might atually be uncovering a real bug. It seems we are creating an extra widget in the clone process.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/G5/g5-content-management-system/commits/81baf7738ce7da9084ecd6f657cd3c0e60f888a2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\",\"org\":{\"id\":2396851,\"login\":\"G5\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/G5\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2396851?\"}}\n{\"id\":\"2489397563\",\"type\":\"PushEvent\",\"actor\":{\"id\":501642,\"login\":\"plouc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/plouc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/501642?\"},\"repo\":{\"id\":28498113,\"name\":\"plouc/mozaik\",\"url\":\"https://api.github.com/repos/plouc/mozaik\"},\"payload\":{\"push_id\":536753003,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"eaa14dd02a30069e42cd770d5ddf478e6cd17086\",\"before\":\"0b27989723feb4b183d5f87813fef146b670b1d1\",\"commits\":[{\"sha\":\"eaa14dd02a30069e42cd770d5ddf478e6cd17086\",\"author\":{\"email\":\"7c5a0c567b5584a13fde407456875318a5bec977@gmail.com\",\"name\":\"Raphaël Benitte\"},\"message\":\"Update README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/plouc/mozaik/commits/eaa14dd02a30069e42cd770d5ddf478e6cd17086\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:57Z\"}\n{\"id\":\"2489397581\",\"type\":\"PushEvent\",\"actor\":{\"id\":899410,\"login\":\"mikeferguson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mikeferguson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/899410?\"},\"repo\":{\"id\":26391646,\"name\":\"mikeferguson/robot_calibration\",\"url\":\"https://api.github.com/repos/mikeferguson/robot_calibration\"},\"payload\":{\"push_id\":536753009,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e60f2c778394e62a9fa7c2401d2c8820d057e6dc\",\"before\":\"2c5350c469f355dee1e421e448ac1a1277776be4\",\"commits\":[{\"sha\":\"e60f2c778394e62a9fa7c2401d2c8820d057e6dc\",\"author\":{\"email\":\"0b5e4dd894fb6bcaed9be1bfeed1c6d2ad1491ba@gmail.com\",\"name\":\"Michael Ferguson\"},\"message\":\"load parameters from ros\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mikeferguson/robot_calibration/commits/e60f2c778394e62a9fa7c2401d2c8820d057e6dc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:03:59Z\"}\n{\"id\":\"2489397599\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":839216,\"login\":\"rohni\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohni\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/839216?\"},\"repo\":{\"id\":10087975,\"name\":\"tailrecursion/hoplon\",\"url\":\"https://api.github.com/repos/tailrecursion/hoplon\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/tailrecursion/hoplon/issues/50\",\"labels_url\":\"https://api.github.com/repos/tailrecursion/hoplon/issues/50/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/tailrecursion/hoplon/issues/50/comments\",\"events_url\":\"https://api.github.com/repos/tailrecursion/hoplon/issues/50/events\",\"html_url\":\"https://github.com/tailrecursion/hoplon/issues/50\",\"id\":53210250,\"number\":50,\"title\":\"Following hoplon Getting Started, fails on Step 2\",\"user\":{\"login\":\"rohni\",\"id\":839216,\"avatar_url\":\"https://avatars.githubusercontent.com/u/839216?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rohni\",\"html_url\":\"https://github.com/rohni\",\"followers_url\":\"https://api.github.com/users/rohni/followers\",\"following_url\":\"https://api.github.com/users/rohni/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/rohni/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/rohni/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/rohni/subscriptions\",\"organizations_url\":\"https://api.github.com/users/rohni/orgs\",\"repos_url\":\"https://api.github.com/users/rohni/repos\",\"events_url\":\"https://api.github.com/users/rohni/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/rohni/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:04:01Z\",\"updated_at\":\"2015-01-01T01:04:01Z\",\"closed_at\":null,\"body\":\"Hi there,\\r\\nI was really excited to try out hoplon, but ran into this snag:\\r\\n\\r\\n     Task 2: Compile the project.\\r\\n\\r\\n     $ boot development\\r\\n\\r\\nHowever, this fails with:\\r\\n\\r\\n    java.lang.RuntimeException: No reader function for tag tailrecursion.boot.core/version\\r\\n\\r\\nI tried upgrading boot:\\r\\n\\r\\n    Retrieving boot-2.0.0-rc4.jar from http://clojars.org/repo/\\r\\n    #https://github.com/boot-clj/boot\\r\\n    #Thu Jan 01 01:58:12 CET 2015\\r\\n    BOOT_CLOJURE_VERSION=1.6.0\\r\\n    BOOT_VERSION=2.0.0-rc4\\r\\n\\r\\nBut just received the same error.\\r\\n\\r\\nI am stuck, and in need of sleep. :)  If anyone can figure out what is going wrong here I would love to move onto exploring hoplon.\\r\\n\\r\\nThanks,\\r\\n\\r\\nRohni\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:01Z\",\"org\":{\"id\":3443819,\"login\":\"tailrecursion\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/tailrecursion\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3443819?\"}}\n{\"id\":\"2489397603\",\"type\":\"ForkEvent\",\"actor\":{\"id\":8671867,\"login\":\"Meehoweq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Meehoweq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8671867?\"},\"repo\":{\"id\":24366589,\"name\":\"Dzikoysk/FunnyGuilds\",\"url\":\"https://api.github.com/repos/Dzikoysk/FunnyGuilds\"},\"payload\":{\"forkee\":{\"id\":28678250,\"name\":\"FunnyGuilds\",\"full_name\":\"Meehoweq/FunnyGuilds\",\"owner\":{\"login\":\"Meehoweq\",\"id\":8671867,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8671867?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Meehoweq\",\"html_url\":\"https://github.com/Meehoweq\",\"followers_url\":\"https://api.github.com/users/Meehoweq/followers\",\"following_url\":\"https://api.github.com/users/Meehoweq/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Meehoweq/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Meehoweq/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Meehoweq/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Meehoweq/orgs\",\"repos_url\":\"https://api.github.com/users/Meehoweq/repos\",\"events_url\":\"https://api.github.com/users/Meehoweq/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Meehoweq/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Meehoweq/FunnyGuilds\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds\",\"forks_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/forks\",\"keys_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/teams\",\"hooks_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/events\",\"assignees_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/tags\",\"blobs_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/languages\",\"stargazers_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/subscription\",\"commits_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/merges\",\"archive_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/downloads\",\"issues_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Meehoweq/FunnyGuilds/releases{/id}\",\"created_at\":\"2015-01-01T01:04:01Z\",\"updated_at\":\"2014-12-31T21:12:33Z\",\"pushed_at\":\"2014-12-31T21:21:42Z\",\"git_url\":\"git://github.com/Meehoweq/FunnyGuilds.git\",\"ssh_url\":\"git@github.com:Meehoweq/FunnyGuilds.git\",\"clone_url\":\"https://github.com/Meehoweq/FunnyGuilds.git\",\"svn_url\":\"https://github.com/Meehoweq/FunnyGuilds\",\"homepage\":\"\",\"size\":959,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:04:01Z\"}\n{\"id\":\"2489397607\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5831804,\"login\":\"YueLinHo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YueLinHo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5831804?\"},\"repo\":{\"id\":21376277,\"name\":\"YueLinHo/TortoiseGit\",\"url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit\"},\"payload\":{\"ref\":\"ylh/happy_new_year_2015\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Windows Explorer Extension to Operate Git; Mirror of GoogleCode repository\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:02Z\"}\n{\"id\":\"2489397612\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Home\",\"title\":\"Home\",\"summary\":null,\"action\":\"edited\",\"sha\":\"be2f83e475fa43ab7785aa503ab92d4c098015c0\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Home\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:02Z\"}\n{\"id\":\"2489397613\",\"type\":\"PushEvent\",\"actor\":{\"id\":3599988,\"login\":\"wesdizzle\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wesdizzle\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3599988?\"},\"repo\":{\"id\":28250120,\"name\":\"wesdizzle/gagglelog\",\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog\"},\"payload\":{\"push_id\":536753016,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fc86231b5063c1b8f2ed07b8d1ae887580869fcf\",\"before\":\"699893742e393b99eb2250f17b2aad8979d7b454\",\"commits\":[{\"sha\":\"fc86231b5063c1b8f2ed07b8d1ae887580869fcf\",\"author\":{\"email\":\"baaa01a5d45f86e3d8f7008866cf0d37bea55570@gmail.com\",\"name\":\"Wesley Miller\"},\"message\":\"changed Index from bit to int in DistributionMethods for cases with more than two distribution methods tha need to be sorted (Genesis Cartridge, 32X Cartridge, CD)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog/commits/fc86231b5063c1b8f2ed07b8d1ae887580869fcf\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:02Z\"}\n{\"id\":\"2489397617\",\"type\":\"CreateEvent\",\"actor\":{\"id\":200511,\"login\":\"alexanderdean\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexanderdean\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/200511?\"},\"repo\":{\"id\":28678252,\"name\":\"alexanderdean/lambda-example-project\",\"url\":\"https://api.github.com/repos/alexanderdean/lambda-example-project\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"An example AWS Lambda application for processing a Kinesis stream of Snowplow enriched events \",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:02Z\"}\n{\"id\":\"2489397620\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753019,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0c74567d174b0901cf19bde2f2ddc65e1d4ff6e1\",\"before\":\"53ba5e0e3194a42192f5aa9f9cd3fd2d79281ce4\",\"commits\":[{\"sha\":\"0c74567d174b0901cf19bde2f2ddc65e1d4ff6e1\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"0.2.0\\n\\nConflicts:\\n\\tpackage.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/0c74567d174b0901cf19bde2f2ddc65e1d4ff6e1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:03Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489397621\",\"type\":\"PushEvent\",\"actor\":{\"id\":1362216,\"login\":\"gmenih341\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gmenih341\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1362216?\"},\"repo\":{\"id\":28205917,\"name\":\"gmenih341/gm_webpage\",\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage\"},\"payload\":{\"push_id\":536753020,\"size\":6,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"44b4ebfe84e066b96aa9a1e99351030e539f5020\",\"before\":\"a9289eb4413ceb90e8c4ef9d1ba3510dc1a5cc79\",\"commits\":[{\"sha\":\"b79ef407608a48bbf713e6cac3e0da901439ef47\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"adding files to branch\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/b79ef407608a48bbf713e6cac3e0da901439ef47\"},{\"sha\":\"7bb277f4b421943b47dd57547bb8f15aaaa8dc4e\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"Removed all files\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/7bb277f4b421943b47dd57547bb8f15aaaa8dc4e\"},{\"sha\":\"0927195f20d7f7ee7726d6eaeaeb2a7c138c4e75\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"Ported to ExpressJS\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/0927195f20d7f7ee7726d6eaeaeb2a7c138c4e75\"},{\"sha\":\"26efa9dbb30fc9f7ab5a7324a5b55b0f4d015c66\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"Gregor Menih\"},\"message\":\"Create README.md\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/26efa9dbb30fc9f7ab5a7324a5b55b0f4d015c66\"},{\"sha\":\"934f504d8abd4adceb718a6767e9d56c5f7fa856\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"Added mongo & monk dependencies\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/934f504d8abd4adceb718a6767e9d56c5f7fa856\"},{\"sha\":\"44b4ebfe84e066b96aa9a1e99351030e539f5020\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"updated readme\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/44b4ebfe84e066b96aa9a1e99351030e539f5020\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:03Z\"}\n{\"id\":\"2489397623\",\"type\":\"PushEvent\",\"actor\":{\"id\":6413475,\"login\":\"vlj\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vlj\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6413475?\"},\"repo\":{\"id\":15920273,\"name\":\"supertuxkart/stk-code\",\"url\":\"https://api.github.com/repos/supertuxkart/stk-code\"},\"payload\":{\"push_id\":536753018,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/lspsm\",\"head\":\"08fed2cfe31a7280384ae8ef494af462af62267d\",\"before\":\"26ffdd0e55e79ad7d97b5bcb86085b538cd1a032\",\"commits\":[{\"sha\":\"08fed2cfe31a7280384ae8ef494af462af62267d\",\"author\":{\"email\":\"adf7f962d8c1003f9d5aad5ecae5c2ca1bbdee24@ovi.com\",\"name\":\"Vincent Lejeune\"},\"message\":\"Use lspsm with pcf\",\"distinct\":true,\"url\":\"https://api.github.com/repos/supertuxkart/stk-code/commits/08fed2cfe31a7280384ae8ef494af462af62267d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:03Z\",\"org\":{\"id\":6138677,\"login\":\"supertuxkart\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/supertuxkart\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6138677?\"}}\n{\"id\":\"2489397622\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":8426440,\"login\":\"nickspring\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nickspring\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8426440?\"},\"repo\":{\"id\":6890589,\"name\":\"etsy/etsyapi\",\"url\":\"https://api.github.com/repos/etsy/etsyapi\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/etsy/etsyapi/issues/108\",\"labels_url\":\"https://api.github.com/repos/etsy/etsyapi/issues/108/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/etsy/etsyapi/issues/108/comments\",\"events_url\":\"https://api.github.com/repos/etsy/etsyapi/issues/108/events\",\"html_url\":\"https://github.com/etsy/etsyapi/issues/108\",\"id\":53210252,\"number\":108,\"title\":\"Length of description field with non-English symbols 2\",\"user\":{\"login\":\"nickspring\",\"id\":8426440,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8426440?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nickspring\",\"html_url\":\"https://github.com/nickspring\",\"followers_url\":\"https://api.github.com/users/nickspring/followers\",\"following_url\":\"https://api.github.com/users/nickspring/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nickspring/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nickspring/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nickspring/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nickspring/orgs\",\"repos_url\":\"https://api.github.com/users/nickspring/repos\",\"events_url\":\"https://api.github.com/users/nickspring/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nickspring/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:04:03Z\",\"updated_at\":\"2015-01-01T01:04:03Z\",\"closed_at\":null,\"body\":\"I’m trying to update FR translation of one listing. Description has length 245 symbols (251 in bytes!!!) but Etsy API responds with error \\\"Value for description must be 255 characters or less.”\\r\\nIt seems that API calculates length of non-English strings incorrectly. See PHP example below.\\r\\n\\r\\nThere is no such problem if I use English string (without diacritics etc).\\r\\n\\r\\n```php\\r\\n<?php\\r\\n\\r\\ndefine('OAUTH_CONSUMER_KEY', '');\\r\\ndefine('OAUTH_CONSUMER_SECRET', '');\\r\\n$access_token = '';\\r\\n$access_token_secret = '';\\r\\n\\r\\n$oauth = new OAuth(OAUTH_CONSUMER_KEY, OAUTH_CONSUMER_SECRET,\\r\\n                   OAUTH_SIG_METHOD_HMACSHA1, OAUTH_AUTH_TYPE_URI);\\r\\n$oauth->setToken($access_token, $access_token_secret);\\r\\n\\r\\ntry {\\r\\n    \\r\\n    # try to change FR translation of one listing \\r\\n       \\r\\n    $params = array(\\r\\n    'description'=>\\\"Couleur: Argent, Violet (d'un côté - de l'argent, d'autres un pourpre clair) \\r\\nTaille (mm): 5mm x 7mm \\r\\nForme: Preciosa PIP Perles / Pétale de Fleur / Fleur \\r\\nVendus en paquets de 60pcs. \\r\\n\\r\\n▶ Merci de vous abonner à notre NEWSLETTER: http://eepurl.c\\\",\\r\\n    'tags'=>'SKU-17634,4mm de semilla de pe,7 0 checa semilla de,cordón,checa semilla de per',\\r\\n    'language'=>'fr',\\r\\n    'listing_id'=>216087470,\\r\\n    'title'=>'60pcs Argent Violet PIP Perles de Verre tchèque PIP Preciosa PIP Perles tchèque Fleur Plate Pétale de Fleur de Perles 5mm x 7mm',\\r\\n    );\\r\\n\\r\\n    echo \\\"THE LENGTH OF DESCRIPTION STRING IS \\\", mb_strlen($params['description'], 'utf-8'), \\\" BUT WE HAVE SUCH ERROR:\\\\n\\\\n\\\";\\r\\n    $data = $oauth->fetch(\\\"https://openapi.etsy.com/v2/listings/216087470/translations/fr\\\", $params, OAUTH_HTTP_METHOD_POST);\\r\\n    \\r\\n    \\r\\n} catch (OAuthException $e) {\\r\\n    error_log($e->getMessage());\\r\\n    error_log(print_r($oauth->getLastResponse(), true));\\r\\n    exit;\\r\\n}\\r\\n\\r\\n?>\\r\\n```\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:03Z\",\"org\":{\"id\":193389,\"login\":\"etsy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/etsy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/193389?\"}}\n{\"id\":\"2489397624\",\"type\":\"PushEvent\",\"actor\":{\"id\":1362216,\"login\":\"gmenih341\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gmenih341\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1362216?\"},\"repo\":{\"id\":28205917,\"name\":\"gmenih341/gm_webpage\",\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage\"},\"payload\":{\"push_id\":536753021,\"size\":2,\"distinct_size\":0,\"ref\":\"refs/heads/nodeJs\",\"head\":\"44b4ebfe84e066b96aa9a1e99351030e539f5020\",\"before\":\"26efa9dbb30fc9f7ab5a7324a5b55b0f4d015c66\",\"commits\":[{\"sha\":\"934f504d8abd4adceb718a6767e9d56c5f7fa856\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"Added mongo & monk dependencies\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/934f504d8abd4adceb718a6767e9d56c5f7fa856\"},{\"sha\":\"44b4ebfe84e066b96aa9a1e99351030e539f5020\",\"author\":{\"email\":\"0b197f5d14de9a1e08df323456fabeaf2b9315d7@gmail.com\",\"name\":\"gmenih341\"},\"message\":\"updated readme\",\"distinct\":false,\"url\":\"https://api.github.com/repos/gmenih341/gm_webpage/commits/44b4ebfe84e066b96aa9a1e99351030e539f5020\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:03Z\"}\n{\"id\":\"2489397626\",\"type\":\"PushEvent\",\"actor\":{\"id\":155953,\"login\":\"saitodisse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/saitodisse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/155953?\"},\"repo\":{\"id\":28519068,\"name\":\"saitodisse/code_rewrite\",\"url\":\"https://api.github.com/repos/saitodisse/code_rewrite\"},\"payload\":{\"push_id\":536753023,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d3df589bd52bfaefb7df4c7ab055f37e0d3b5023\",\"before\":\"0a48d64aee06d7d3075bc62e9e5c980a6ebe6b68\",\"commits\":[{\"sha\":\"d3df589bd52bfaefb7df4c7ab055f37e0d3b5023\",\"author\":{\"email\":\"2ba0b9c8a539cf0353ba4c5654aacbc724b4002e@gmail.com\",\"name\":\"Julio Makdisse Saito\"},\"message\":\"partial\",\"distinct\":true,\"url\":\"https://api.github.com/repos/saitodisse/code_rewrite/commits/d3df589bd52bfaefb7df4c7ab055f37e0d3b5023\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:04Z\"}\n{\"id\":\"2489397632\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":5509466,\"login\":\"Arisae\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Arisae\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5509466?\"},\"repo\":{\"id\":28179339,\"name\":\"YGGDRASIL-STUDIO/Discouraged-Workers\",\"url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/issues/153\",\"labels_url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/issues/153/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/issues/153/comments\",\"events_url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/issues/153/events\",\"html_url\":\"https://github.com/YGGDRASIL-STUDIO/Discouraged-Workers/issues/153\",\"id\":53210253,\"number\":153,\"title\":\"Add Bridge of the Life License in README.html\",\"user\":{\"login\":\"Arisae\",\"id\":5509466,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5509466?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Arisae\",\"html_url\":\"https://github.com/Arisae\",\"followers_url\":\"https://api.github.com/users/Arisae/followers\",\"following_url\":\"https://api.github.com/users/Arisae/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Arisae/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Arisae/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Arisae/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Arisae/orgs\",\"repos_url\":\"https://api.github.com/users/Arisae/repos\",\"events_url\":\"https://api.github.com/users/Arisae/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Arisae/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/labels/enhancement\",\"name\":\"enhancement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"Arisae\",\"id\":5509466,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5509466?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Arisae\",\"html_url\":\"https://github.com/Arisae\",\"followers_url\":\"https://api.github.com/users/Arisae/followers\",\"following_url\":\"https://api.github.com/users/Arisae/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Arisae/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Arisae/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Arisae/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Arisae/orgs\",\"repos_url\":\"https://api.github.com/users/Arisae/repos\",\"events_url\":\"https://api.github.com/users/Arisae/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Arisae/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/milestones/12\",\"labels_url\":\"https://api.github.com/repos/YGGDRASIL-STUDIO/Discouraged-Workers/milestones/12/labels\",\"id\":914871,\"number\":12,\"title\":\"ETC\",\"description\":\"License, Manual, ETC\",\"creator\":{\"login\":\"Arisae\",\"id\":5509466,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5509466?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Arisae\",\"html_url\":\"https://github.com/Arisae\",\"followers_url\":\"https://api.github.com/users/Arisae/followers\",\"following_url\":\"https://api.github.com/users/Arisae/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Arisae/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Arisae/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Arisae/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Arisae/orgs\",\"repos_url\":\"https://api.github.com/users/Arisae/repos\",\"events_url\":\"https://api.github.com/users/Arisae/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Arisae/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":4,\"closed_issues\":0,\"state\":\"open\",\"created_at\":\"2014-12-26T04:10:21Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"due_on\":\"2015-12-30T15:00:00Z\",\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:04:05Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"closed_at\":null,\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:05Z\",\"org\":{\"id\":10226289,\"login\":\"YGGDRASIL-STUDIO\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/YGGDRASIL-STUDIO\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10226289?\"}}\n{\"id\":\"2489397638\",\"type\":\"PushEvent\",\"actor\":{\"id\":1373703,\"login\":\"team3cord\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/team3cord\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1373703?\"},\"repo\":{\"id\":20268125,\"name\":\"team3cord/mc-dotfiles\",\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles\"},\"payload\":{\"push_id\":536753025,\"size\":6,\"distinct_size\":1,\"ref\":\"refs/heads/OSX\",\"head\":\"731064c30578713735d183022092299605dcf83a\",\"before\":\"460e8c219d89fcf24c4f961765a63c7a8e4185f1\",\"commits\":[{\"sha\":\"7ebd9b4cc70e2b3a2f044d7ff4ef63c2b27d7451\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Updating for linux\",\"distinct\":false,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/7ebd9b4cc70e2b3a2f044d7ff4ef63c2b27d7451\"},{\"sha\":\"beb1f3edadc32c3d451800028eafcaf8f753e124\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Readme\",\"distinct\":false,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/beb1f3edadc32c3d451800028eafcaf8f753e124\"},{\"sha\":\"4d83ec65b7161a9c5a1bab69642d2ba15e748201\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Updated TODO's\",\"distinct\":false,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/4d83ec65b7161a9c5a1bab69642d2ba15e748201\"},{\"sha\":\"57e4c5c28b8fed6b308b24f86432d3676fd8d96e\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Bannished DS\",\"distinct\":false,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/57e4c5c28b8fed6b308b24f86432d3676fd8d96e\"},{\"sha\":\"c274de69b1397e2b2b3a92b5cbba22b87211ffa0\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Updated vim configuration\",\"distinct\":false,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/c274de69b1397e2b2b3a92b5cbba22b87211ffa0\"},{\"sha\":\"731064c30578713735d183022092299605dcf83a\",\"author\":{\"email\":\"67bcad84ab1facdcd6a44cf7083c26cfa2e421e4@gmail.com\",\"name\":\"MattCordeiro\"},\"message\":\"Uncommented OSX specific aliases\",\"distinct\":true,\"url\":\"https://api.github.com/repos/team3cord/mc-dotfiles/commits/731064c30578713735d183022092299605dcf83a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:05Z\"}\n{\"id\":\"2489397640\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":135605,\"login\":\"joneslee85\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joneslee85\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/135605?\"},\"repo\":{\"id\":16760719,\"name\":\"lotus/model\",\"url\":\"https://api.github.com/repos/lotus/model\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/lotus/model/issues/137\",\"labels_url\":\"https://api.github.com/repos/lotus/model/issues/137/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/lotus/model/issues/137/comments\",\"events_url\":\"https://api.github.com/repos/lotus/model/issues/137/events\",\"html_url\":\"https://github.com/lotus/model/pull/137\",\"id\":53171581,\"number\":137,\"title\":\"Refactor attributes DSL\",\"user\":{\"login\":\"joneslee85\",\"id\":135605,\"avatar_url\":\"https://avatars.githubusercontent.com/u/135605?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joneslee85\",\"html_url\":\"https://github.com/joneslee85\",\"followers_url\":\"https://api.github.com/users/joneslee85/followers\",\"following_url\":\"https://api.github.com/users/joneslee85/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joneslee85/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joneslee85/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joneslee85/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joneslee85/orgs\",\"repos_url\":\"https://api.github.com/users/joneslee85/repos\",\"events_url\":\"https://api.github.com/users/joneslee85/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joneslee85/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-31T07:27:28Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/lotus/model/pulls/137\",\"html_url\":\"https://github.com/lotus/model/pull/137\",\"diff_url\":\"https://github.com/lotus/model/pull/137.diff\",\"patch_url\":\"https://github.com/lotus/model/pull/137.patch\"},\"body\":\"* [x] Consolidate constructor\\r\\n* [x] Does not gracefully handle unknown attributes\"},\"comment\":{\"url\":\"https://api.github.com/repos/lotus/model/issues/comments/68477296\",\"html_url\":\"https://github.com/lotus/model/pull/137#issuecomment-68477296\",\"issue_url\":\"https://api.github.com/repos/lotus/model/issues/137\",\"id\":68477296,\"user\":{\"login\":\"joneslee85\",\"id\":135605,\"avatar_url\":\"https://avatars.githubusercontent.com/u/135605?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joneslee85\",\"html_url\":\"https://github.com/joneslee85\",\"followers_url\":\"https://api.github.com/users/joneslee85/followers\",\"following_url\":\"https://api.github.com/users/joneslee85/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joneslee85/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joneslee85/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joneslee85/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joneslee85/orgs\",\"repos_url\":\"https://api.github.com/users/joneslee85/repos\",\"events_url\":\"https://api.github.com/users/joneslee85/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joneslee85/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:05Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"body\":\"@AlfonsoUceda not yet, I have yet heard back from @jodosha, need more discussion\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:05Z\",\"org\":{\"id\":3210273,\"login\":\"lotus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/lotus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3210273?\"}}\n{\"id\":\"2489397641\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":663212,\"login\":\"tdas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?\"},\"repo\":{\"id\":17165658,\"name\":\"apache/spark\",\"url\":\"https://api.github.com/repos/apache/spark\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/comments/22397277\",\"id\":22397277,\"diff_hunk\":\"@@ -17,31 +17,65 @@\\n \\n package org.apache.spark.streaming.mqtt\\n \\n-import org.scalatest.FunSuite\\n-\\n+import org.scalatest.{BeforeAndAfter, FunSuite}\\n+import org.scalatest.concurrent.Eventually\\n+import scala.concurrent.duration._\\n import org.apache.spark.streaming.{Seconds, StreamingContext}\\n import org.apache.spark.storage.StorageLevel\\n import org.apache.spark.streaming.dstream.ReceiverInputDStream\\n+import org.eclipse.paho.client.mqttv3._\\n+import org.eclipse.paho.client.mqttv3.persist.MqttDefaultFilePersistence\\n \\n-class MQTTStreamSuite extends FunSuite {\\n-\\n-  val batchDuration = Seconds(1)\\n+class MQTTStreamSuite extends FunSuite with Eventually with BeforeAndAfter {\\n \\n+  private val batchDuration = Seconds(1)\\n   private val master: String = \\\"local[2]\\\"\\n-\\n   private val framework: String = this.getClass.getSimpleName\\n+  private val brokerUrl = \\\"tcp://localhost:1883\\\"\\n+  private val topic = \\\"def\\\"\\n+  private var ssc: StreamingContext = _\\n \\n-  test(\\\"mqtt input stream\\\") {\\n-    val ssc = new StreamingContext(master, framework, batchDuration)\\n-    val brokerUrl = \\\"abc\\\"\\n-    val topic = \\\"def\\\"\\n+  before {\\n+    ssc = new StreamingContext(master, framework, batchDuration)\\n+  }\\n+  after {\\n+    if (ssc != null) {\\n+      ssc.stop()\\n+      ssc = null\\n+    }\\n+  }\\n \\n-    // tests the API, does not actually test data receiving\\n-    val test1: ReceiverInputDStream[String] = MQTTUtils.createStream(ssc, brokerUrl, topic)\\n-    val test2: ReceiverInputDStream[String] =\\n+  test(\\\"mqtt input stream\\\") {\\n+    val sendMessage = \\\"MQTT demo for spark streaming\\\"\\n+    publishData(sendMessage)\\n+    val receiveStream: ReceiverInputDStream[String] =\\n       MQTTUtils.createStream(ssc, brokerUrl, topic, StorageLevel.MEMORY_AND_DISK_SER_2)\\n-\\n-    // TODO: Actually test receiving data\\n+    var receiveMessage: String = \\\"\\\"\\n+    receiveStream.foreachRDD { rdd =>\\n+      receiveMessage = rdd.first\\n+      receiveMessage\\n+    }\\n+    ssc.start()\\n+    eventually(timeout(10000 milliseconds), interval(100 milliseconds)) {\\n+      assert(sendMessage.equals(receiveMessage))\\n+    }\\n     ssc.stop()\\n   }\\n+\\n+  def publishData(sendMessage: String): Unit = {\\n+    try {\\n+      val persistence: MqttClientPersistence = new MqttDefaultFilePersistence(\\\"/tmp\\\")\\n+      val client: MqttClient = new MqttClient(brokerUrl, MqttClient.generateClientId(), persistence)\\n+      client.connect()\\n+      val msgTopic: MqttTopic = client.getTopic(topic)\\n+      val message: MqttMessage = new MqttMessage(String.valueOf(sendMessage).getBytes(\\\"utf-8\\\"))\\n+      message.setQos(1)\\n+      message.setRetained(true)\\n+      msgTopic.publish(message)\\n+      println(\\\"Published data \\\\ntopic: \\\" + msgTopic.getName() + \\\"\\\\nMessage: \\\" + message)\\n+      client.disconnect()\\n+    } catch {\\n+      case e: MqttException => println(\\\"Exception Caught: \\\" + e)\\n+    }\",\"path\":\"external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala\",\"position\":78,\"original_position\":78,\"commit_id\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"original_commit_id\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"user\":{\"login\":\"tdas\",\"id\":663212,\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"html_url\":\"https://github.com/tdas\",\"followers_url\":\"https://api.github.com/users/tdas/followers\",\"following_url\":\"https://api.github.com/users/tdas/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tdas/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tdas/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tdas/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tdas/orgs\",\"repos_url\":\"https://api.github.com/users/tdas/repos\",\"events_url\":\"https://api.github.com/users/tdas/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tdas/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Shouldnt there be a `finally` to close any running servers (client, etc.)?\",\"created_at\":\"2015-01-01T01:04:05Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"html_url\":\"https://github.com/apache/spark/pull/3844#discussion_r22397277\",\"pull_request_url\":\"https://api.github.com/repos/apache/spark/pulls/3844\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/comments/22397277\"},\"html\":{\"href\":\"https://github.com/apache/spark/pull/3844#discussion_r22397277\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/3844\",\"id\":26684823,\"html_url\":\"https://github.com/apache/spark/pull/3844\",\"diff_url\":\"https://github.com/apache/spark/pull/3844.diff\",\"patch_url\":\"https://github.com/apache/spark/pull/3844.patch\",\"issue_url\":\"https://api.github.com/repos/apache/spark/issues/3844\",\"number\":3844,\"state\":\"open\",\"locked\":false,\"title\":\"[SPARK-4631] unit test for MQTT\",\"user\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Please review the unit test for MQTT\",\"created_at\":\"2014-12-30T13:12:33Z\",\"updated_at\":\"2015-01-01T01:04:05Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"78f53f74b7089734c2cc3a2f701e6044194cda1d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/apache/spark/pulls/3844/commits\",\"review_comments_url\":\"https://api.github.com/repos/apache/spark/pulls/3844/comments\",\"review_comment_url\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/issues/3844/comments\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/fc8eb286db6aa8e78a567537996011f554eed969\",\"head\":{\"label\":\"Bilna:master\",\"ref\":\"master\",\"sha\":\"fc8eb286db6aa8e78a567537996011f554eed969\",\"user\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28630516,\"name\":\"spark\",\"full_name\":\"Bilna/spark\",\"owner\":{\"login\":\"Bilna\",\"id\":7123586,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7123586?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bilna\",\"html_url\":\"https://github.com/Bilna\",\"followers_url\":\"https://api.github.com/users/Bilna/followers\",\"following_url\":\"https://api.github.com/users/Bilna/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bilna/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bilna/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bilna/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bilna/orgs\",\"repos_url\":\"https://api.github.com/users/Bilna/repos\",\"events_url\":\"https://api.github.com/users/Bilna/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bilna/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Bilna/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":true,\"url\":\"https://api.github.com/repos/Bilna/spark\",\"forks_url\":\"https://api.github.com/repos/Bilna/spark/forks\",\"keys_url\":\"https://api.github.com/repos/Bilna/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Bilna/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Bilna/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/Bilna/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Bilna/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Bilna/spark/events\",\"assignees_url\":\"https://api.github.com/repos/Bilna/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Bilna/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Bilna/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/Bilna/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Bilna/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Bilna/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Bilna/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Bilna/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Bilna/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/Bilna/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Bilna/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Bilna/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Bilna/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/Bilna/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Bilna/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Bilna/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Bilna/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Bilna/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Bilna/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Bilna/spark/merges\",\"archive_url\":\"https://api.github.com/repos/Bilna/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Bilna/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/Bilna/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Bilna/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Bilna/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Bilna/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Bilna/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Bilna/spark/releases{/id}\",\"created_at\":\"2014-12-30T12:47:01Z\",\"updated_at\":\"2014-12-31T09:52:36Z\",\"pushed_at\":\"2014-12-31T09:52:35Z\",\"git_url\":\"git://github.com/Bilna/spark.git\",\"ssh_url\":\"git@github.com:Bilna/spark.git\",\"clone_url\":\"https://github.com/Bilna/spark.git\",\"svn_url\":\"https://github.com/Bilna/spark\",\"homepage\":null,\"size\":87823,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"apache:master\",\"ref\":\"master\",\"sha\":\"352ed6bbe3c3b67e52e298e7c535ae414d96beca\",\"user\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":17165658,\"name\":\"spark\",\"full_name\":\"apache/spark\",\"owner\":{\"login\":\"apache\",\"id\":47359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/apache\",\"html_url\":\"https://github.com/apache\",\"followers_url\":\"https://api.github.com/users/apache/followers\",\"following_url\":\"https://api.github.com/users/apache/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/apache/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/apache/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/apache/subscriptions\",\"organizations_url\":\"https://api.github.com/users/apache/orgs\",\"repos_url\":\"https://api.github.com/users/apache/repos\",\"events_url\":\"https://api.github.com/users/apache/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/apache/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/apache/spark\",\"description\":\"Mirror of Apache Spark\",\"fork\":false,\"url\":\"https://api.github.com/repos/apache/spark\",\"forks_url\":\"https://api.github.com/repos/apache/spark/forks\",\"keys_url\":\"https://api.github.com/repos/apache/spark/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/apache/spark/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/apache/spark/teams\",\"hooks_url\":\"https://api.github.com/repos/apache/spark/hooks\",\"issue_events_url\":\"https://api.github.com/repos/apache/spark/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/apache/spark/events\",\"assignees_url\":\"https://api.github.com/repos/apache/spark/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/apache/spark/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/apache/spark/tags\",\"blobs_url\":\"https://api.github.com/repos/apache/spark/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/apache/spark/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/apache/spark/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/apache/spark/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/apache/spark/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/apache/spark/languages\",\"stargazers_url\":\"https://api.github.com/repos/apache/spark/stargazers\",\"contributors_url\":\"https://api.github.com/repos/apache/spark/contributors\",\"subscribers_url\":\"https://api.github.com/repos/apache/spark/subscribers\",\"subscription_url\":\"https://api.github.com/repos/apache/spark/subscription\",\"commits_url\":\"https://api.github.com/repos/apache/spark/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/apache/spark/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/apache/spark/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/apache/spark/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/apache/spark/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/apache/spark/merges\",\"archive_url\":\"https://api.github.com/repos/apache/spark/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/apache/spark/downloads\",\"issues_url\":\"https://api.github.com/repos/apache/spark/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/apache/spark/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/apache/spark/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/apache/spark/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/apache/spark/releases{/id}\",\"created_at\":\"2014-02-25T08:00:08Z\",\"updated_at\":\"2015-01-01T00:59:33Z\",\"pushed_at\":\"2015-01-01T00:59:33Z\",\"git_url\":\"git://github.com/apache/spark.git\",\"ssh_url\":\"git@github.com:apache/spark.git\",\"clone_url\":\"https://github.com/apache/spark.git\",\"svn_url\":\"https://github.com/apache/spark\",\"homepage\":null,\"size\":1083068,\"stargazers_count\":2458,\"watchers_count\":2458,\"language\":\"Scala\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":2179,\"mirror_url\":\"git://git.apache.org/spark.git\",\"open_issues_count\":268,\"forks\":2179,\"open_issues\":268,\"watchers\":2458,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844\"},\"html\":{\"href\":\"https://github.com/apache/spark/pull/3844\"},\"issue\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3844\"},\"comments\":{\"href\":\"https://api.github.com/repos/apache/spark/issues/3844/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/apache/spark/pulls/3844/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/apache/spark/statuses/fc8eb286db6aa8e78a567537996011f554eed969\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:04:05Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489397645\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1398544,\"login\":\"joelpurra\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joelpurra\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1398544?\"},\"repo\":{\"id\":28595466,\"name\":\"joelpurra/jqnpm\",\"url\":\"https://api.github.com/repos/joelpurra/jqnpm\"},\"payload\":{\"ref\":\"import_659\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"A package manager built for jq as an example implementation.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:06Z\"}\n{\"id\":\"2489397653\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":9831378,\"login\":\"Amit-P-Amin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?\"},\"repo\":{\"id\":12024210,\"name\":\"appacademy/active_record_lite\",\"url\":\"https://api.github.com/repos/appacademy/active_record_lite\"},\"payload\":{\"action\":\"opened\",\"number\":17,\"pull_request\":{\"url\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17\",\"id\":26739446,\"html_url\":\"https://github.com/appacademy/active_record_lite/pull/17\",\"diff_url\":\"https://github.com/appacademy/active_record_lite/pull/17.diff\",\"patch_url\":\"https://github.com/appacademy/active_record_lite/pull/17.patch\",\"issue_url\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/17\",\"number\":17,\"state\":\"open\",\"locked\":false,\"title\":\"Update 03_associatable_spec.rb\",\"user\":{\"login\":\"Amit-P-Amin\",\"id\":9831378,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"html_url\":\"https://github.com/Amit-P-Amin\",\"followers_url\":\"https://api.github.com/users/Amit-P-Amin/followers\",\"following_url\":\"https://api.github.com/users/Amit-P-Amin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Amit-P-Amin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Amit-P-Amin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Amit-P-Amin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Amit-P-Amin/orgs\",\"repos_url\":\"https://api.github.com/users/Amit-P-Amin/repos\",\"events_url\":\"https://api.github.com/users/Amit-P-Amin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Amit-P-Amin/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Was confused for a while because \\\"AssocOptions #model_class returns class of associated object\\\" was failing. This is because that spec tests 2 methods, but the description just mentions 1 (only model class, instead of model class and table name). I split the test into two, one for the model class and one for the table name.\",\"created_at\":\"2015-01-01T01:04:07Z\",\"updated_at\":\"2015-01-01T01:04:07Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17/commits\",\"review_comments_url\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17/comments\",\"review_comment_url\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/17/comments\",\"statuses_url\":\"https://api.github.com/repos/appacademy/active_record_lite/statuses/b5ac327b20f2041652e3228ce4612880aae97beb\",\"head\":{\"label\":\"Amit-P-Amin:patch-1\",\"ref\":\"patch-1\",\"sha\":\"b5ac327b20f2041652e3228ce4612880aae97beb\",\"user\":{\"login\":\"Amit-P-Amin\",\"id\":9831378,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"html_url\":\"https://github.com/Amit-P-Amin\",\"followers_url\":\"https://api.github.com/users/Amit-P-Amin/followers\",\"following_url\":\"https://api.github.com/users/Amit-P-Amin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Amit-P-Amin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Amit-P-Amin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Amit-P-Amin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Amit-P-Amin/orgs\",\"repos_url\":\"https://api.github.com/users/Amit-P-Amin/repos\",\"events_url\":\"https://api.github.com/users/Amit-P-Amin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Amit-P-Amin/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28678217,\"name\":\"active_record_lite\",\"full_name\":\"Amit-P-Amin/active_record_lite\",\"owner\":{\"login\":\"Amit-P-Amin\",\"id\":9831378,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9831378?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Amit-P-Amin\",\"html_url\":\"https://github.com/Amit-P-Amin\",\"followers_url\":\"https://api.github.com/users/Amit-P-Amin/followers\",\"following_url\":\"https://api.github.com/users/Amit-P-Amin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Amit-P-Amin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Amit-P-Amin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Amit-P-Amin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Amit-P-Amin/orgs\",\"repos_url\":\"https://api.github.com/users/Amit-P-Amin/repos\",\"events_url\":\"https://api.github.com/users/Amit-P-Amin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Amit-P-Amin/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Amit-P-Amin/active_record_lite\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite\",\"forks_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/forks\",\"keys_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/teams\",\"hooks_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/events\",\"assignees_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/tags\",\"blobs_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/languages\",\"stargazers_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/subscription\",\"commits_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/merges\",\"archive_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/downloads\",\"issues_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Amit-P-Amin/active_record_lite/releases{/id}\",\"created_at\":\"2015-01-01T01:00:39Z\",\"updated_at\":\"2015-01-01T01:00:39Z\",\"pushed_at\":\"2015-01-01T01:02:23Z\",\"git_url\":\"git://github.com/Amit-P-Amin/active_record_lite.git\",\"ssh_url\":\"git@github.com:Amit-P-Amin/active_record_lite.git\",\"clone_url\":\"https://github.com/Amit-P-Amin/active_record_lite.git\",\"svn_url\":\"https://github.com/Amit-P-Amin/active_record_lite\",\"homepage\":null,\"size\":632,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"appacademy:master\",\"ref\":\"master\",\"sha\":\"ca4e169591deec6c4b30b611e8ed5bdf89b3386a\",\"user\":{\"login\":\"appacademy\",\"id\":2138704,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2138704?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/appacademy\",\"html_url\":\"https://github.com/appacademy\",\"followers_url\":\"https://api.github.com/users/appacademy/followers\",\"following_url\":\"https://api.github.com/users/appacademy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/appacademy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/appacademy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/appacademy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/appacademy/orgs\",\"repos_url\":\"https://api.github.com/users/appacademy/repos\",\"events_url\":\"https://api.github.com/users/appacademy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/appacademy/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":12024210,\"name\":\"active_record_lite\",\"full_name\":\"appacademy/active_record_lite\",\"owner\":{\"login\":\"appacademy\",\"id\":2138704,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2138704?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/appacademy\",\"html_url\":\"https://github.com/appacademy\",\"followers_url\":\"https://api.github.com/users/appacademy/followers\",\"following_url\":\"https://api.github.com/users/appacademy/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/appacademy/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/appacademy/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/appacademy/subscriptions\",\"organizations_url\":\"https://api.github.com/users/appacademy/orgs\",\"repos_url\":\"https://api.github.com/users/appacademy/repos\",\"events_url\":\"https://api.github.com/users/appacademy/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/appacademy/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/appacademy/active_record_lite\",\"description\":\"\",\"fork\":false,\"url\":\"https://api.github.com/repos/appacademy/active_record_lite\",\"forks_url\":\"https://api.github.com/repos/appacademy/active_record_lite/forks\",\"keys_url\":\"https://api.github.com/repos/appacademy/active_record_lite/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/appacademy/active_record_lite/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/appacademy/active_record_lite/teams\",\"hooks_url\":\"https://api.github.com/repos/appacademy/active_record_lite/hooks\",\"issue_events_url\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/appacademy/active_record_lite/events\",\"assignees_url\":\"https://api.github.com/repos/appacademy/active_record_lite/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/appacademy/active_record_lite/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/appacademy/active_record_lite/tags\",\"blobs_url\":\"https://api.github.com/repos/appacademy/active_record_lite/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/appacademy/active_record_lite/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/appacademy/active_record_lite/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/appacademy/active_record_lite/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/appacademy/active_record_lite/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/appacademy/active_record_lite/languages\",\"stargazers_url\":\"https://api.github.com/repos/appacademy/active_record_lite/stargazers\",\"contributors_url\":\"https://api.github.com/repos/appacademy/active_record_lite/contributors\",\"subscribers_url\":\"https://api.github.com/repos/appacademy/active_record_lite/subscribers\",\"subscription_url\":\"https://api.github.com/repos/appacademy/active_record_lite/subscription\",\"commits_url\":\"https://api.github.com/repos/appacademy/active_record_lite/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/appacademy/active_record_lite/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/appacademy/active_record_lite/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/appacademy/active_record_lite/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/appacademy/active_record_lite/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/appacademy/active_record_lite/merges\",\"archive_url\":\"https://api.github.com/repos/appacademy/active_record_lite/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/appacademy/active_record_lite/downloads\",\"issues_url\":\"https://api.github.com/repos/appacademy/active_record_lite/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/appacademy/active_record_lite/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/appacademy/active_record_lite/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/appacademy/active_record_lite/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/appacademy/active_record_lite/releases{/id}\",\"created_at\":\"2013-08-10T17:32:19Z\",\"updated_at\":\"2014-10-23T23:33:55Z\",\"pushed_at\":\"2014-10-26T23:37:38Z\",\"git_url\":\"git://github.com/appacademy/active_record_lite.git\",\"ssh_url\":\"git@github.com:appacademy/active_record_lite.git\",\"clone_url\":\"https://github.com/appacademy/active_record_lite.git\",\"svn_url\":\"https://github.com/appacademy/active_record_lite\",\"homepage\":null,\"size\":632,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Ruby\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":13,\"mirror_url\":null,\"open_issues_count\":2,\"forks\":13,\"open_issues\":2,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17\"},\"html\":{\"href\":\"https://github.com/appacademy/active_record_lite/pull/17\"},\"issue\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/17\"},\"comments\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/issues/17/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/pulls/17/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/appacademy/active_record_lite/statuses/b5ac327b20f2041652e3228ce4612880aae97beb\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":7,\"deletions\":1,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:04:07Z\",\"org\":{\"id\":2138704,\"login\":\"appacademy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/appacademy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2138704?\"}}\n{\"id\":\"2489397654\",\"type\":\"PushEvent\",\"actor\":{\"id\":8800150,\"login\":\"troyeagle\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/troyeagle\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8800150?\"},\"repo\":{\"id\":24052643,\"name\":\"zzt93/Hw-Client\",\"url\":\"https://api.github.com/repos/zzt93/Hw-Client\"},\"payload\":{\"push_id\":536753031,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c8b1d6ea7905faf630d57ebb283d880ca6e655e7\",\"before\":\"897452d98a4d094571441ac5a4f55a22884a7764\",\"commits\":[{\"sha\":\"c8b1d6ea7905faf630d57ebb283d880ca6e655e7\",\"author\":{\"email\":\"cae9f45d02c4e43ad3a912954969d5845cb0e620@qq.com\",\"name\":\"troyeagle\"},\"message\":\"刷新\",\"distinct\":true,\"url\":\"https://api.github.com/repos/zzt93/Hw-Client/commits/c8b1d6ea7905faf630d57ebb283d880ca6e655e7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:07Z\"}\n{\"id\":\"2489397656\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1932804,\"login\":\"coldmind\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?\"},\"repo\":{\"id\":4164482,\"name\":\"django/django\",\"url\":\"https://api.github.com/repos/django/django\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/django/django/issues/3821\",\"labels_url\":\"https://api.github.com/repos/django/django/issues/3821/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/django/django/issues/3821/comments\",\"events_url\":\"https://api.github.com/repos/django/django/issues/3821/events\",\"html_url\":\"https://github.com/django/django/pull/3821\",\"id\":53200969,\"number\":3821,\"title\":\"[WIP] Fixed #24064 - Prevented database access in compile time in spatialite models\",\"user\":{\"login\":\"coldmind\",\"id\":1932804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"html_url\":\"https://github.com/coldmind\",\"followers_url\":\"https://api.github.com/users/coldmind/followers\",\"following_url\":\"https://api.github.com/users/coldmind/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coldmind/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coldmind/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coldmind/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coldmind/orgs\",\"repos_url\":\"https://api.github.com/users/coldmind/repos\",\"events_url\":\"https://api.github.com/users/coldmind/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coldmind/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-31T20:02:17Z\",\"updated_at\":\"2015-01-01T01:04:07Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/django/django/pulls/3821\",\"html_url\":\"https://github.com/django/django/pull/3821\",\"diff_url\":\"https://github.com/django/django/pull/3821.diff\",\"patch_url\":\"https://github.com/django/django/pull/3821.patch\"},\"body\":\"Here is the patch that can fix problem.\\r\\nIdeas about improvements are welcome.\"},\"comment\":{\"url\":\"https://api.github.com/repos/django/django/issues/comments/68477298\",\"html_url\":\"https://github.com/django/django/pull/3821#issuecomment-68477298\",\"issue_url\":\"https://api.github.com/repos/django/django/issues/3821\",\"id\":68477298,\"user\":{\"login\":\"coldmind\",\"id\":1932804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"html_url\":\"https://github.com/coldmind\",\"followers_url\":\"https://api.github.com/users/coldmind/followers\",\"following_url\":\"https://api.github.com/users/coldmind/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coldmind/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coldmind/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coldmind/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coldmind/orgs\",\"repos_url\":\"https://api.github.com/users/coldmind/repos\",\"events_url\":\"https://api.github.com/users/coldmind/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coldmind/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:07Z\",\"updated_at\":\"2015-01-01T01:04:07Z\",\"body\":\"buildbot, test this please\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:07Z\",\"org\":{\"id\":27804,\"login\":\"django\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/django\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/27804?\"}}\n{\"id\":\"2489397661\",\"type\":\"PushEvent\",\"actor\":{\"id\":9201970,\"login\":\"qdm\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qdm\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9201970?\"},\"repo\":{\"id\":25173910,\"name\":\"qdm/qdm.github.io\",\"url\":\"https://api.github.com/repos/qdm/qdm.github.io\"},\"payload\":{\"push_id\":536753035,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dbcd229108e512d575dcd1e4595183ce055f8e2f\",\"before\":\"94683ca14e320a286452e2f2e873ff5d603f1a98\",\"commits\":[{\"sha\":\"dbcd229108e512d575dcd1e4595183ce055f8e2f\",\"author\":{\"email\":\"de163e90d3aeef9f404d1de71c48e234a211e3c3@gmail.com\",\"name\":\"KT\"},\"message\":\"Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qdm/qdm.github.io/commits/dbcd229108e512d575dcd1e4595183ce055f8e2f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:08Z\"}\n{\"id\":\"2489397666\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":405446,\"login\":\"quickfur\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/quickfur\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/405446?\"},\"repo\":{\"id\":1257080,\"name\":\"D-Programming-Language/dlang.org\",\"url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/734\",\"labels_url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/734/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/734/comments\",\"events_url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/734/events\",\"html_url\":\"https://github.com/D-Programming-Language/dlang.org/pull/734\",\"id\":53083284,\"number\":734,\"title\":\"Much nicer jump-to anchors\",\"user\":{\"login\":\"andralex\",\"id\":566679,\"avatar_url\":\"https://avatars.githubusercontent.com/u/566679?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/andralex\",\"html_url\":\"https://github.com/andralex\",\"followers_url\":\"https://api.github.com/users/andralex/followers\",\"following_url\":\"https://api.github.com/users/andralex/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/andralex/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/andralex/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/andralex/subscriptions\",\"organizations_url\":\"https://api.github.com/users/andralex/orgs\",\"repos_url\":\"https://api.github.com/users/andralex/repos\",\"events_url\":\"https://api.github.com/users/andralex/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/andralex/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-30T00:18:24Z\",\"updated_at\":\"2015-01-01T01:04:08Z\",\"closed_at\":\"2014-12-30T01:03:48Z\",\"pull_request\":{\"url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/pulls/734\",\"html_url\":\"https://github.com/D-Programming-Language/dlang.org/pull/734\",\"diff_url\":\"https://github.com/D-Programming-Language/dlang.org/pull/734.diff\",\"patch_url\":\"https://github.com/D-Programming-Language/dlang.org/pull/734.patch\"},\"body\":\"Example: http://erdani.com/d/phobos-prerelease/std_range_package.html\"},\"comment\":{\"url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/comments/68477299\",\"html_url\":\"https://github.com/D-Programming-Language/dlang.org/pull/734#issuecomment-68477299\",\"issue_url\":\"https://api.github.com/repos/D-Programming-Language/dlang.org/issues/734\",\"id\":68477299,\"user\":{\"login\":\"quickfur\",\"id\":405446,\"avatar_url\":\"https://avatars.githubusercontent.com/u/405446?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/quickfur\",\"html_url\":\"https://github.com/quickfur\",\"followers_url\":\"https://api.github.com/users/quickfur/followers\",\"following_url\":\"https://api.github.com/users/quickfur/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/quickfur/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/quickfur/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/quickfur/subscriptions\",\"organizations_url\":\"https://api.github.com/users/quickfur/orgs\",\"repos_url\":\"https://api.github.com/users/quickfur/repos\",\"events_url\":\"https://api.github.com/users/quickfur/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/quickfur/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:08Z\",\"updated_at\":\"2015-01-01T01:04:08Z\",\"body\":\"This PR has broken Phobos docs builds. If you clone a clean copy of dlang.org and phobos and build dlang.org then `cd phobos; make -f posix.mak html`, it will produce documentation pages with no content in `phobos-prerelease`. I'd revert this PR, but the github reverter seems to have trouble with this, as it appears to be conflicting with subsequent changes in the repo.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:08Z\",\"org\":{\"id\":565913,\"login\":\"D-Programming-Language\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/D-Programming-Language\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/565913?\"}}\n{\"id\":\"2489397678\",\"type\":\"PushEvent\",\"actor\":{\"id\":1017605,\"login\":\"wangshan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wangshan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1017605?\"},\"repo\":{\"id\":28666633,\"name\":\"wangshan/wangshan.github.io\",\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io\"},\"payload\":{\"push_id\":536753040,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0efdedb4a49bebd1456119ad89cad76ada7e2c91\",\"before\":\"5a94fdd12e5d88be38cca22321e943d374d3e4d5\",\"commits\":[{\"sha\":\"0efdedb4a49bebd1456119ad89cad76ada7e2c91\",\"author\":{\"email\":\"e3e97680eb29c788f35181af31eb442b3251e18f@gmail.com\",\"name\":\"Shan\"},\"message\":\"Update 2012-03-03-mac-development-environment-setup.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wangshan/wangshan.github.io/commits/0efdedb4a49bebd1456119ad89cad76ada7e2c91\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:10Z\"}\n{\"id\":\"2489397679\",\"type\":\"PushEvent\",\"actor\":{\"id\":5202416,\"login\":\"Hardmath123\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Hardmath123\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5202416?\"},\"repo\":{\"id\":16244731,\"name\":\"Hardmath123/hardmath123.github.io\",\"url\":\"https://api.github.com/repos/Hardmath123/hardmath123.github.io\"},\"payload\":{\"push_id\":536753041,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"f030c9bc614ac41d120f96aa33a1a9d386851f6e\",\"before\":\"a3a57f52045048dda77f285c9da30fcbfafda094\",\"commits\":[{\"sha\":\"0ffe5ea18ffb4330fd8fd85058dd563fe978d4e6\",\"author\":{\"email\":\"430380b60638190ff73296b54585d989728b087a@gmail.com\",\"name\":\"Hardmath123\"},\"message\":\"header in helvetica\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Hardmath123/hardmath123.github.io/commits/0ffe5ea18ffb4330fd8fd85058dd563fe978d4e6\"},{\"sha\":\"f030c9bc614ac41d120f96aa33a1a9d386851f6e\",\"author\":{\"email\":\"430380b60638190ff73296b54585d989728b087a@gmail.com\",\"name\":\"Hardmath123\"},\"message\":\"ok justification was a mistake. i can't justify that decision...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Hardmath123/hardmath123.github.io/commits/f030c9bc614ac41d120f96aa33a1a9d386851f6e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:10Z\"}\n{\"id\":\"2489397681\",\"type\":\"PushEvent\",\"actor\":{\"id\":11417,\"login\":\"malept\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/malept\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/11417?\"},\"repo\":{\"id\":22760055,\"name\":\"malept/octoaudit\",\"url\":\"https://api.github.com/repos/malept/octoaudit\"},\"payload\":{\"push_id\":536753042,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"50e1f3dcd88d928c991b8a5a1b85d797e27aa54b\",\"before\":\"d306ec83113e82e1d11244084dc47333360aefda\",\"commits\":[{\"sha\":\"50e1f3dcd88d928c991b8a5a1b85d797e27aa54b\",\"author\":{\"email\":\"1133a844dd6eb8a9702546324bc949f27bda666a@lazymalevolence.com\",\"name\":\"Mark Lee\"},\"message\":\"Travis: account for cache directory creation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/malept/octoaudit/commits/50e1f3dcd88d928c991b8a5a1b85d797e27aa54b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:10Z\"}\n{\"id\":\"2489397683\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":716644,\"login\":\"amandaharlin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amandaharlin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/716644?\"},\"repo\":{\"id\":24870992,\"name\":\"codeforokc/codeforokc\",\"url\":\"https://api.github.com/repos/codeforokc/codeforokc\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/5\",\"labels_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/5/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/5/comments\",\"events_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/5/events\",\"html_url\":\"https://github.com/codeforokc/codeforokc/issues/5\",\"id\":53210256,\"number\":5,\"title\":\"Other pages to add?\",\"user\":{\"login\":\"amandaharlin\",\"id\":716644,\"avatar_url\":\"https://avatars.githubusercontent.com/u/716644?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amandaharlin\",\"html_url\":\"https://github.com/amandaharlin\",\"followers_url\":\"https://api.github.com/users/amandaharlin/followers\",\"following_url\":\"https://api.github.com/users/amandaharlin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/amandaharlin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/amandaharlin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/amandaharlin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/amandaharlin/orgs\",\"repos_url\":\"https://api.github.com/users/amandaharlin/repos\",\"events_url\":\"https://api.github.com/users/amandaharlin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/amandaharlin/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:04:10Z\",\"updated_at\":\"2015-01-01T01:04:10Z\",\"closed_at\":null,\"body\":\"In the config.yml i saw other not-yet-implemented menu items, like:\\r\\n\\r\\n  #Meetings: /meetings\\r\\n  #Forum: https://github.com/codeforokc/discussions/issues\\r\\n  #Chat: https://gitter.im/codeforokc/discussions\\r\\n  #Join Us: /join-us\\r\\n  #Archives: /archives\\r\\n\\r\\ndo you want to add these in still?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:11Z\",\"org\":{\"id\":8738523,\"login\":\"codeforokc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/codeforokc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8738523?\"}}\n{\"id\":\"2489397686\",\"type\":\"PushEvent\",\"actor\":{\"id\":5384580,\"login\":\"oridb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oridb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5384580?\"},\"repo\":{\"id\":16144976,\"name\":\"oridb/mc\",\"url\":\"https://api.github.com/repos/oridb/mc\"},\"payload\":{\"push_id\":536753043,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fc13f8699b5f3702525fcf5a2692b1800763aa53\",\"before\":\"d45042b7a8b06f0bc1a82a4aa32687aef08cf2b2\",\"commits\":[{\"sha\":\"fc13f8699b5f3702525fcf5a2692b1800763aa53\",\"author\":{\"email\":\"6c720b6ece72a4972cc39d6436cb370514e267ec@eigenstate.org\",\"name\":\"Ori Bernstein\"},\"message\":\"Fix up libstd on plan9.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/oridb/mc/commits/fc13f8699b5f3702525fcf5a2692b1800763aa53\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:11Z\"}\n{\"id\":\"2489397693\",\"type\":\"PushEvent\",\"actor\":{\"id\":1026154,\"login\":\"Fizzadar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Fizzadar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1026154?\"},\"repo\":{\"id\":28368289,\"name\":\"Fizzadar/multiselect\",\"url\":\"https://api.github.com/repos/Fizzadar/multiselect\"},\"payload\":{\"push_id\":536753046,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/develop\",\"head\":\"aa30e8c6c8be5ab3095b78a53f9a48607217caea\",\"before\":\"596c8a453d7a734053f9688663abf5061b7dca33\",\"commits\":[{\"sha\":\"fb51964b06e1a83ea9f2233014ea43cb4ff42636\",\"author\":{\"email\":\"75ef9faee755c70589550b513ad881e5a603182c@oxygem.com\",\"name\":\"Nick Barrett\"},\"message\":\"Correct bug in initalizing option lists\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Fizzadar/multiselect/commits/fb51964b06e1a83ea9f2233014ea43cb4ff42636\"},{\"sha\":\"aa30e8c6c8be5ab3095b78a53f9a48607217caea\",\"author\":{\"email\":\"75ef9faee755c70589550b513ad881e5a603182c@oxygem.com\",\"name\":\"Nick Barrett\"},\"message\":\"+build\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Fizzadar/multiselect/commits/aa30e8c6c8be5ab3095b78a53f9a48607217caea\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:12Z\"}\n{\"id\":\"2489397694\",\"type\":\"PushEvent\",\"actor\":{\"id\":2263283,\"login\":\"iraikov\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/iraikov\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2263283?\"},\"repo\":{\"id\":22903078,\"name\":\"iraikov/Sims.jl\",\"url\":\"https://api.github.com/repos/iraikov/Sims.jl\"},\"payload\":{\"push_id\":536753047,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"418f7577926273d814040ede54f0873ae5b9d17e\",\"before\":\"bffecb12bb56348f7065ad7670144bd787799879\",\"commits\":[{\"sha\":\"418f7577926273d814040ede54f0873ae5b9d17e\",\"author\":{\"email\":\"af431cb13887b0da5947004f417999c37e85ed49@gmail.com\",\"name\":\"Ivan Raikov\"},\"message\":\"copy parameters on structural changes to ensure any modifications are preserved\",\"distinct\":true,\"url\":\"https://api.github.com/repos/iraikov/Sims.jl/commits/418f7577926273d814040ede54f0873ae5b9d17e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:12Z\"}\n{\"id\":\"2489397701\",\"type\":\"PushEvent\",\"actor\":{\"id\":378599,\"login\":\"deborah-ufw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/deborah-ufw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/378599?\"},\"repo\":{\"id\":23501674,\"name\":\"deborah-ufw/deborah-ufw.github.io\",\"url\":\"https://api.github.com/repos/deborah-ufw/deborah-ufw.github.io\"},\"payload\":{\"push_id\":536753051,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ad602b16440a0546ffde289477a3699a8b7a1800\",\"before\":\"636739a0b83baf6ccd6b38a78f49502363a19778\",\"commits\":[{\"sha\":\"ad602b16440a0546ffde289477a3699a8b7a1800\",\"author\":{\"email\":\"b0969288fc07bf7a2f076f214eeab18bf2a0cd1c@hotmail.com\",\"name\":\"deborah\"},\"message\":\"ticket tweak question for complete mode\",\"distinct\":true,\"url\":\"https://api.github.com/repos/deborah-ufw/deborah-ufw.github.io/commits/ad602b16440a0546ffde289477a3699a8b7a1800\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:13Z\"}\n{\"id\":\"2489397702\",\"type\":\"WatchEvent\",\"actor\":{\"id\":220358,\"login\":\"ankane\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ankane\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/220358?\"},\"repo\":{\"id\":1359757,\"name\":\"SchemaPlus/schema_plus\",\"url\":\"https://api.github.com/repos/SchemaPlus/schema_plus\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:14Z\",\"org\":{\"id\":614848,\"login\":\"SchemaPlus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/SchemaPlus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/614848?\"}}\n{\"id\":\"2489397706\",\"type\":\"PushEvent\",\"actor\":{\"id\":1558033,\"login\":\"wpears\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wpears\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1558033?\"},\"repo\":{\"id\":23894134,\"name\":\"wpears/twopane\",\"url\":\"https://api.github.com/repos/wpears/twopane\"},\"payload\":{\"push_id\":536753053,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dev\",\"head\":\"d8c403b3bceb31ce36ff1c0b5e7fcf18bc7914fc\",\"before\":\"d81881adaff8ddb3e0da1d02315ab74efa175102\",\"commits\":[{\"sha\":\"d8c403b3bceb31ce36ff1c0b5e7fcf18bc7914fc\",\"author\":{\"email\":\"ce362ed51a9d769761b4ab97740ccf6e6615e394@gmail.com\",\"name\":\"Wyatt Pearsall\"},\"message\":\"Cleaned up twopane.js\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wpears/twopane/commits/d8c403b3bceb31ce36ff1c0b5e7fcf18bc7914fc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:14Z\"}\n{\"id\":\"2489397709\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536753054,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f673aef9bc6b115cda55214d362fde50d301ea25\",\"before\":\"c085f224e47c8147e6085f897cb91a07834211b1\",\"commits\":[{\"sha\":\"f673aef9bc6b115cda55214d362fde50d301ea25\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074253830\\n\\n36ErTuCI1v1YILLcBrCP+M1Dl7YcJfq0JhOl6oV84vA=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/f673aef9bc6b115cda55214d362fde50d301ea25\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:15Z\"}\n{\"id\":\"2489397720\",\"type\":\"PushEvent\",\"actor\":{\"id\":6298185,\"login\":\"salrodgom\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/salrodgom\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6298185?\"},\"repo\":{\"id\":28646555,\"name\":\"salrodgom/MC-MD_hybrid_cycles\",\"url\":\"https://api.github.com/repos/salrodgom/MC-MD_hybrid_cycles\"},\"payload\":{\"push_id\":536753057,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d2e6b275229477c0b9f94bcb22452f56402643a8\",\"before\":\"a762ae78e5c09457dcced64e3a6f6138906759f3\",\"commits\":[{\"sha\":\"d2e6b275229477c0b9f94bcb22452f56402643a8\",\"author\":{\"email\":\"98ac177e5ce7190d101150b0bffa6dca0bd0984b@hipocampo.upo.es\",\"name\":\"Salvador Rodríguez Gómez\"},\"message\":\"modified:   mc_md_script_opti.sct\",\"distinct\":true,\"url\":\"https://api.github.com/repos/salrodgom/MC-MD_hybrid_cycles/commits/d2e6b275229477c0b9f94bcb22452f56402643a8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:17Z\"}\n{\"id\":\"2489397721\",\"type\":\"PushEvent\",\"actor\":{\"id\":10144074,\"login\":\"carodew\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carodew\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10144074?\"},\"repo\":{\"id\":27844858,\"name\":\"carodew/carodew.github.io\",\"url\":\"https://api.github.com/repos/carodew/carodew.github.io\"},\"payload\":{\"push_id\":536753058,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"62bd3e248d57b48720930200d73b4273993a2e66\",\"before\":\"19643dcb3388a0ae99570e176fc592e0fd666c81\",\"commits\":[{\"sha\":\"62bd3e248d57b48720930200d73b4273993a2e66\",\"author\":{\"email\":\"6e3c6f0214740e9061d9ca5c79eb6e0ff9cc1741@unknown542696dd77af.gateway.pace.com\",\"name\":\"Carolyn\"},\"message\":\"update ux projects to new format\",\"distinct\":true,\"url\":\"https://api.github.com/repos/carodew/carodew.github.io/commits/62bd3e248d57b48720930200d73b4273993a2e66\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:17Z\"}\n{\"id\":\"2489397722\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":279669,\"login\":\"sachin-handiekar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?\"},\"repo\":{\"id\":1987667,\"name\":\"sachin-handiekar/jInstagram\",\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78\",\"labels_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/comments\",\"events_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/events\",\"html_url\":\"https://github.com/sachin-handiekar/jInstagram/issues/78\",\"id\":53177876,\"number\":78,\"title\":\"Configure Travis-CI build to deploy snapshot to Maven Repositories\",\"user\":{\"login\":\"sachin-handiekar\",\"id\":279669,\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"html_url\":\"https://github.com/sachin-handiekar\",\"followers_url\":\"https://api.github.com/users/sachin-handiekar/followers\",\"following_url\":\"https://api.github.com/users/sachin-handiekar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sachin-handiekar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sachin-handiekar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sachin-handiekar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sachin-handiekar/orgs\",\"repos_url\":\"https://api.github.com/users/sachin-handiekar/repos\",\"events_url\":\"https://api.github.com/users/sachin-handiekar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sachin-handiekar/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/labels/infrastructure\",\"name\":\"infrastructure\",\"color\":\"fbca04\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"sachin-handiekar\",\"id\":279669,\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"html_url\":\"https://github.com/sachin-handiekar\",\"followers_url\":\"https://api.github.com/users/sachin-handiekar/followers\",\"following_url\":\"https://api.github.com/users/sachin-handiekar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sachin-handiekar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sachin-handiekar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sachin-handiekar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sachin-handiekar/orgs\",\"repos_url\":\"https://api.github.com/users/sachin-handiekar/repos\",\"events_url\":\"https://api.github.com/users/sachin-handiekar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sachin-handiekar/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T10:37:34Z\",\"updated_at\":\"2015-01-01T01:04:17Z\",\"closed_at\":\"2015-01-01T01:04:17Z\",\"body\":\"Configure Travis-CI build to deploy snapshot to Maven Repositories\\r\\n\\r\\nhttp://notbarjo.blogspot.co.uk/2014/09/travis-ci-maven-deploy.html\"},\"comment\":{\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/comments/68477301\",\"html_url\":\"https://github.com/sachin-handiekar/jInstagram/issues/78#issuecomment-68477301\",\"issue_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78\",\"id\":68477301,\"user\":{\"login\":\"sachin-handiekar\",\"id\":279669,\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"html_url\":\"https://github.com/sachin-handiekar\",\"followers_url\":\"https://api.github.com/users/sachin-handiekar/followers\",\"following_url\":\"https://api.github.com/users/sachin-handiekar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sachin-handiekar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sachin-handiekar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sachin-handiekar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sachin-handiekar/orgs\",\"repos_url\":\"https://api.github.com/users/sachin-handiekar/repos\",\"events_url\":\"https://api.github.com/users/sachin-handiekar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sachin-handiekar/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:17Z\",\"updated_at\":\"2015-01-01T01:04:17Z\",\"body\":\"Updated pom.xml and .travis.yml to upload snapshot to snapshot repository. \\r\\n\\r\\nhttps://oss.sonatype.org/content/repositories/snapshots/com/sachinhandiekar/jInstagram/\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:18Z\"}\n{\"id\":\"2489397723\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":279669,\"login\":\"sachin-handiekar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?\"},\"repo\":{\"id\":1987667,\"name\":\"sachin-handiekar/jInstagram\",\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78\",\"labels_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/comments\",\"events_url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/issues/78/events\",\"html_url\":\"https://github.com/sachin-handiekar/jInstagram/issues/78\",\"id\":53177876,\"number\":78,\"title\":\"Configure Travis-CI build to deploy snapshot to Maven Repositories\",\"user\":{\"login\":\"sachin-handiekar\",\"id\":279669,\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"html_url\":\"https://github.com/sachin-handiekar\",\"followers_url\":\"https://api.github.com/users/sachin-handiekar/followers\",\"following_url\":\"https://api.github.com/users/sachin-handiekar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sachin-handiekar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sachin-handiekar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sachin-handiekar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sachin-handiekar/orgs\",\"repos_url\":\"https://api.github.com/users/sachin-handiekar/repos\",\"events_url\":\"https://api.github.com/users/sachin-handiekar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sachin-handiekar/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/sachin-handiekar/jInstagram/labels/infrastructure\",\"name\":\"infrastructure\",\"color\":\"fbca04\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"sachin-handiekar\",\"id\":279669,\"avatar_url\":\"https://avatars.githubusercontent.com/u/279669?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sachin-handiekar\",\"html_url\":\"https://github.com/sachin-handiekar\",\"followers_url\":\"https://api.github.com/users/sachin-handiekar/followers\",\"following_url\":\"https://api.github.com/users/sachin-handiekar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sachin-handiekar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sachin-handiekar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sachin-handiekar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sachin-handiekar/orgs\",\"repos_url\":\"https://api.github.com/users/sachin-handiekar/repos\",\"events_url\":\"https://api.github.com/users/sachin-handiekar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sachin-handiekar/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T10:37:34Z\",\"updated_at\":\"2015-01-01T01:04:17Z\",\"closed_at\":\"2015-01-01T01:04:17Z\",\"body\":\"Configure Travis-CI build to deploy snapshot to Maven Repositories\\r\\n\\r\\nhttp://notbarjo.blogspot.co.uk/2014/09/travis-ci-maven-deploy.html\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:18Z\"}\n{\"id\":\"2489397727\",\"type\":\"CreateEvent\",\"actor\":{\"id\":9505729,\"login\":\"alishadot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alishadot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9505729?\"},\"repo\":{\"id\":28678253,\"name\":\"alishadot/keepcloud\",\"url\":\"https://api.github.com/repos/alishadot/keepcloud\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:18Z\"}\n{\"id\":\"2489397731\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397281\",\"id\":22397281,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":73,\"original_position\":73,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"why the continuation?\",\"created_at\":\"2015-01-01T01:04:18Z\",\"updated_at\":\"2015-01-01T01:04:18Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397281\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397281\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397281\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:04:18Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:04:18Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489397732\",\"type\":\"PushEvent\",\"actor\":{\"id\":5728403,\"login\":\"patrick-hudson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/patrick-hudson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5728403?\"},\"repo\":{\"id\":25392255,\"name\":\"patrick-hudson/EggDrop\",\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop\"},\"payload\":{\"push_id\":536753062,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"84116936c4f6805f56a1e643eb9898ebdc854334\",\"before\":\"cb11cf77a4d9d7625c14e50c27fe61d387d2c95e\",\"commits\":[{\"sha\":\"84116936c4f6805f56a1e643eb9898ebdc854334\",\"author\":{\"email\":\"cbb7353e6d953ef360baf960c122346276c6e320@hudson.bz\",\"name\":\"Patrick Hudson\"},\"message\":\"Scripted auto-commit on change (2014-12-31 20:04:17) by gitwatch.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop/commits/84116936c4f6805f56a1e643eb9898ebdc854334\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:19Z\"}\n{\"id\":\"2489397735\",\"type\":\"PushEvent\",\"actor\":{\"id\":419567,\"login\":\"kethinov\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kethinov\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/419567?\"},\"repo\":{\"id\":24020924,\"name\":\"kethinov/node-webkit-app-template\",\"url\":\"https://api.github.com/repos/kethinov/node-webkit-app-template\"},\"payload\":{\"push_id\":536753063,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5048243e66f207ee8e325e1b2b3bfa24505d8ebf\",\"before\":\"b4dbedc97bcd361f835762d676533f1879d99151\",\"commits\":[{\"sha\":\"5048243e66f207ee8e325e1b2b3bfa24505d8ebf\",\"author\":{\"email\":\"63dafb7843374487f4b5809d77124751ec24f51b@gmail.com\",\"name\":\"Eric Newport\"},\"message\":\"issue link\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kethinov/node-webkit-app-template/commits/5048243e66f207ee8e325e1b2b3bfa24505d8ebf\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:19Z\"}\n{\"id\":\"2489397736\",\"type\":\"PushEvent\",\"actor\":{\"id\":7126826,\"login\":\"laufi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/laufi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7126826?\"},\"repo\":{\"id\":28243875,\"name\":\"laufi/heisserdraht-nova\",\"url\":\"https://api.github.com/repos/laufi/heisserdraht-nova\"},\"payload\":{\"push_id\":536753064,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"818fe73d0bde07012797b8e347a1c9c83af244be\",\"before\":\"230f624d56385ab4d3cb8c6a67d5956b16d1f66e\",\"commits\":[{\"sha\":\"d77e08b7213a6529dc6484b5997120970f3315a5\",\"author\":{\"email\":\"b45c0673cfcf01be8a6ebb140d24652d100db872@outlook.com\",\"name\":\"laufi\"},\"message\":\"BUGFIX: Auch Konstruktor mit zwei Eingabefeldern müsste float sein!\",\"distinct\":true,\"url\":\"https://api.github.com/repos/laufi/heisserdraht-nova/commits/d77e08b7213a6529dc6484b5997120970f3315a5\"},{\"sha\":\"818fe73d0bde07012797b8e347a1c9c83af244be\",\"author\":{\"email\":\"b45c0673cfcf01be8a6ebb140d24652d100db872@outlook.com\",\"name\":\"laufi\"},\"message\":\"Möglichkeit sich mit Namen einzutragen eingefügt\",\"distinct\":true,\"url\":\"https://api.github.com/repos/laufi/heisserdraht-nova/commits/818fe73d0bde07012797b8e347a1c9c83af244be\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:19Z\"}\n{\"id\":\"2489397739\",\"type\":\"PushEvent\",\"actor\":{\"id\":10176820,\"login\":\"chalavadivishnu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chalavadivishnu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10176820?\"},\"repo\":{\"id\":28678150,\"name\":\"chalavadivishnu/Face-Detection\",\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection\"},\"payload\":{\"push_id\":536753066,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b5f47548e309ef787dfcd2c424330ff2a645956b\",\"before\":\"96f7f02d9551bb3c922c9c6303b9d144e2c31398\",\"commits\":[{\"sha\":\"b5f47548e309ef787dfcd2c424330ff2a645956b\",\"author\":{\"email\":\"ce8044f02eb2a26b631671f5297317036d398e79@gmail.com\",\"name\":\"Chalavadi Vishnu\"},\"message\":\"training images function\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chalavadivishnu/Face-Detection/commits/b5f47548e309ef787dfcd2c424330ff2a645956b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:20Z\"}\n{\"id\":\"2489397743\",\"type\":\"PushEvent\",\"actor\":{\"id\":7992035,\"login\":\"amycodes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amycodes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7992035?\"},\"repo\":{\"id\":28671701,\"name\":\"amycodes/stackexchange-findanswers\",\"url\":\"https://api.github.com/repos/amycodes/stackexchange-findanswers\"},\"payload\":{\"push_id\":536753069,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d925c0b925a4eefd5b2402dfc42f4ba9e1f1d3cc\",\"before\":\"0e67154d9b55e9f5adb516ea0ba99bf635c1585e\",\"commits\":[{\"sha\":\"d925c0b925a4eefd5b2402dfc42f4ba9e1f1d3cc\",\"author\":{\"email\":\"da0cee525ea095cebd642feb98c0fe5678fb7db3@get.it\",\"name\":\"Amy Negrette\"},\"message\":\"Changed [] to array() to work in prod\",\"distinct\":true,\"url\":\"https://api.github.com/repos/amycodes/stackexchange-findanswers/commits/d925c0b925a4eefd5b2402dfc42f4ba9e1f1d3cc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:20Z\"}\n{\"id\":\"2489397744\",\"type\":\"PushEvent\",\"actor\":{\"id\":10313856,\"login\":\"uhumph\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/uhumph\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10313856?\"},\"repo\":{\"id\":28540040,\"name\":\"uhumph/docker-mailstack\",\"url\":\"https://api.github.com/repos/uhumph/docker-mailstack\"},\"payload\":{\"push_id\":536753070,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8ff001a4315046f5578f9ad30493d6a5ccafe099\",\"before\":\"da210967679d5ef4169fcb6102003bfd4df0384e\",\"commits\":[{\"sha\":\"8ff001a4315046f5578f9ad30493d6a5ccafe099\",\"author\":{\"email\":\"911c8ba17707a1b9f8955686b6e596e09ecc489d@slo-computer.com\",\"name\":\"uhumph\"},\"message\":\"Update setup.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/uhumph/docker-mailstack/commits/8ff001a4315046f5578f9ad30493d6a5ccafe099\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:20Z\"}\n{\"id\":\"2489397750\",\"type\":\"PushEvent\",\"actor\":{\"id\":83665,\"login\":\"petrfaitl\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/petrfaitl\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/83665?\"},\"repo\":{\"id\":28532783,\"name\":\"petrfaitl/Activity_Exporter\",\"url\":\"https://api.github.com/repos/petrfaitl/Activity_Exporter\"},\"payload\":{\"push_id\":536753073,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2416d43f35ec05a75b07399cc79967bdf57dbcca\",\"before\":\"27048cac241b72ed223a0809f4bcb440c2d56f0e\",\"commits\":[{\"sha\":\"2416d43f35ec05a75b07399cc79967bdf57dbcca\",\"author\":{\"email\":\"23f6690d1a592ecb333159d73d3c2545c06271d5@gmail.com\",\"name\":\"Petr Faitl\"},\"message\":\"Fixes to Garmin data format to suppport hr and cadence.\\nFixes to UI to support alternative file output\\nFixes to null value output at the start activity\",\"distinct\":true,\"url\":\"https://api.github.com/repos/petrfaitl/Activity_Exporter/commits/2416d43f35ec05a75b07399cc79967bdf57dbcca\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:20Z\"}\n{\"id\":\"2489397751\",\"type\":\"PushEvent\",\"actor\":{\"id\":3694248,\"login\":\"tevenfeng\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tevenfeng\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3694248?\"},\"repo\":{\"id\":28070401,\"name\":\"tevenfeng/ppt2pdf.net\",\"url\":\"https://api.github.com/repos/tevenfeng/ppt2pdf.net\"},\"payload\":{\"push_id\":536753074,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"87f1e4d775f88ca111b178d499f2ab83b9e5f7cd\",\"before\":\"ac8fb58d01ba8f8e0b8d7047761ddc62b42bd1ff\",\"commits\":[{\"sha\":\"87f1e4d775f88ca111b178d499f2ab83b9e5f7cd\",\"author\":{\"email\":\"6a1cb8b3a19ff60a6f85a39cf0fac4ed28b3ccc7@outlook.com\",\"name\":\"Teven Feng\"},\"message\":\"add LGPL license\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tevenfeng/ppt2pdf.net/commits/87f1e4d775f88ca111b178d499f2ab83b9e5f7cd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:20Z\"}\n{\"id\":\"2489397755\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1308363,\"login\":\"paymonp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paymonp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1308363?\"},\"repo\":{\"id\":28678242,\"name\":\"paymonp/forecast_wrapper\",\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"A wrapper to simplify Forecast API data, and add some extra functionality (Ex. determining current night/day status for a location).\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:21Z\"}\n{\"id\":\"2489397757\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Mapping Layout\",\"title\":\"Mapping Layout\",\"summary\":null,\"action\":\"edited\",\"sha\":\"5c14947b1fd1e254da5541ed0a7981bcc265cfe8\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Mapping-Layout\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:21Z\"}\n{\"id\":\"2489397758\",\"type\":\"PushEvent\",\"actor\":{\"id\":3720783,\"login\":\"designerwebhosting\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/designerwebhosting\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3720783?\"},\"repo\":{\"id\":20527117,\"name\":\"designerwebhosting/christopherbyrne.github.io\",\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io\"},\"payload\":{\"push_id\":536753077,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"19f1368a9acc50e9b1cc2b936c165c9704040889\",\"before\":\"b8aa926ae3fee18580b31846a3b3dfcea5ad032f\",\"commits\":[{\"sha\":\"19f1368a9acc50e9b1cc2b936c165c9704040889\",\"author\":{\"email\":\"4bb0acc6ff8c0b6c31e50417877e6e3b3f1c65f0@googlemail.com\",\"name\":\"Peter Noblee\"},\"message\":\"update 'date'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io/commits/19f1368a9acc50e9b1cc2b936c165c9704040889\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:22Z\"}\n{\"id\":\"2489397763\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2784341,\"login\":\"RyanMurphy86\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RyanMurphy86\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2784341?\"},\"repo\":{\"id\":28678257,\"name\":\"RyanMurphy86/2skewed\",\"url\":\"https://api.github.com/repos/RyanMurphy86/2skewed\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:22Z\"}\n{\"id\":\"2489397764\",\"type\":\"ForkEvent\",\"actor\":{\"id\":4997404,\"login\":\"chenxinyong\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chenxinyong\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4997404?\"},\"repo\":{\"id\":22034912,\"name\":\"POPWorldMedia/POPForums\",\"url\":\"https://api.github.com/repos/POPWorldMedia/POPForums\"},\"payload\":{\"forkee\":{\"id\":28678258,\"name\":\"POPForums\",\"full_name\":\"chenxinyong/POPForums\",\"owner\":{\"login\":\"chenxinyong\",\"id\":4997404,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4997404?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chenxinyong\",\"html_url\":\"https://github.com/chenxinyong\",\"followers_url\":\"https://api.github.com/users/chenxinyong/followers\",\"following_url\":\"https://api.github.com/users/chenxinyong/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/chenxinyong/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/chenxinyong/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/chenxinyong/subscriptions\",\"organizations_url\":\"https://api.github.com/users/chenxinyong/orgs\",\"repos_url\":\"https://api.github.com/users/chenxinyong/repos\",\"events_url\":\"https://api.github.com/users/chenxinyong/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/chenxinyong/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/chenxinyong/POPForums\",\"description\":\"A forum application running on ASP.NET MVC, available in six languages.\",\"fork\":true,\"url\":\"https://api.github.com/repos/chenxinyong/POPForums\",\"forks_url\":\"https://api.github.com/repos/chenxinyong/POPForums/forks\",\"keys_url\":\"https://api.github.com/repos/chenxinyong/POPForums/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/chenxinyong/POPForums/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/chenxinyong/POPForums/teams\",\"hooks_url\":\"https://api.github.com/repos/chenxinyong/POPForums/hooks\",\"issue_events_url\":\"https://api.github.com/repos/chenxinyong/POPForums/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/chenxinyong/POPForums/events\",\"assignees_url\":\"https://api.github.com/repos/chenxinyong/POPForums/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/chenxinyong/POPForums/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/chenxinyong/POPForums/tags\",\"blobs_url\":\"https://api.github.com/repos/chenxinyong/POPForums/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/chenxinyong/POPForums/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/chenxinyong/POPForums/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/chenxinyong/POPForums/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/chenxinyong/POPForums/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/chenxinyong/POPForums/languages\",\"stargazers_url\":\"https://api.github.com/repos/chenxinyong/POPForums/stargazers\",\"contributors_url\":\"https://api.github.com/repos/chenxinyong/POPForums/contributors\",\"subscribers_url\":\"https://api.github.com/repos/chenxinyong/POPForums/subscribers\",\"subscription_url\":\"https://api.github.com/repos/chenxinyong/POPForums/subscription\",\"commits_url\":\"https://api.github.com/repos/chenxinyong/POPForums/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/chenxinyong/POPForums/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/chenxinyong/POPForums/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/chenxinyong/POPForums/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/chenxinyong/POPForums/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/chenxinyong/POPForums/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/chenxinyong/POPForums/merges\",\"archive_url\":\"https://api.github.com/repos/chenxinyong/POPForums/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/chenxinyong/POPForums/downloads\",\"issues_url\":\"https://api.github.com/repos/chenxinyong/POPForums/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/chenxinyong/POPForums/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/chenxinyong/POPForums/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/chenxinyong/POPForums/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/chenxinyong/POPForums/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/chenxinyong/POPForums/releases{/id}\",\"created_at\":\"2015-01-01T01:04:22Z\",\"updated_at\":\"2014-12-20T03:14:29Z\",\"pushed_at\":\"2014-12-10T03:33:27Z\",\"git_url\":\"git://github.com/chenxinyong/POPForums.git\",\"ssh_url\":\"git@github.com:chenxinyong/POPForums.git\",\"clone_url\":\"https://github.com/chenxinyong/POPForums.git\",\"svn_url\":\"https://github.com/chenxinyong/POPForums\",\"homepage\":\"http://popforums.com/\",\"size\":24896,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:04:22Z\",\"org\":{\"id\":8217691,\"login\":\"POPWorldMedia\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/POPWorldMedia\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8217691?\"}}\n{\"id\":\"2489397766\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":8508800,\"login\":\"cmp-202\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmp-202\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8508800?\"},\"repo\":{\"id\":23310272,\"name\":\"cmp-202/ssh2shell\",\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10\",\"labels_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/comments\",\"events_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10/events\",\"html_url\":\"https://github.com/cmp-202/ssh2shell/issues/10\",\"id\":53071635,\"number\":10,\"title\":\"Timeout error (part 2)\",\"user\":{\"login\":\"macirex\",\"id\":1192393,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1192393?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/macirex\",\"html_url\":\"https://github.com/macirex\",\"followers_url\":\"https://api.github.com/users/macirex/followers\",\"following_url\":\"https://api.github.com/users/macirex/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/macirex/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/macirex/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/macirex/subscriptions\",\"organizations_url\":\"https://api.github.com/users/macirex/orgs\",\"repos_url\":\"https://api.github.com/users/macirex/repos\",\"events_url\":\"https://api.github.com/users/macirex/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/macirex/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":8,\"created_at\":\"2014-12-29T21:04:44Z\",\"updated_at\":\"2015-01-01T01:04:23Z\",\"closed_at\":null,\"body\":\"Hello cmp,\\r\\n\\r\\nI´m sorry to bother you again but I´ve encountered the another time out issue.\\r\\n\\r\\nLet me provide some context first:\\r\\n\\r\\nA group of servers I administer have some restricted policies:\\r\\n\\r\\n- Root and application domain users are inaccessible via ssh.\\r\\n- None of the application users have password.\\r\\n- There´s only one user available for ssh connection, it is used for everything and it has limited permissions, for references this will be called \\\"simpleuser\\\". \\r\\n- Only way to administer a server is to connect via ssh to the \\\"simpleuser\\\" and then do a su to root. \\r\\n\\r\\nBasically the flow goes like this: \\r\\nLogin -> su - root -> su - domainapp ->  [Here I do what I need to do]   \\r\\n\\r\\nThe list of commands I´m trying is:\\r\\n  commands:           [\\r\\n    \\\"su - root\\\", \\r\\n    \\\"echo hello\\\" \\r\\n    \\\"su - webapp\\\",\\r\\n    \\\"ls -lthr\\\",\\r\\n  ]\\r\\n\\r\\nI managed to log into root using the event on commandProcessing and writing the password directly into the stream:\\r\\n\\r\\n<code>\\r\\nSSH.on ('commandProcessing', function onCommandProcessing( command, response, sshObj, stream ) {   \\r\\n            if (command.match(/root/)&& response.indexOf(\\\"Password:\\\") != -1) {\\r\\n\\t           if (!admin)\\r\\n\\t\\t   {\\r\\n\\t\\t\\tstream.write('password\\\\n');\\r\\n\\t\\t\\tadmin = true;\\r\\n\\t\\t    }\\r\\n              });\\r\\n</code>\\r\\n\\r\\nAfter that, I try to do another su, this time to the domain I need to check but it fails with: \\r\\nTimeout error: 10.164.12.159: Command timed out after 5 seconds\\r\\n\\r\\nHere´s the output my app generates:\\r\\n\\r\\nConnected\\r\\nRunning commands Now\\r\\n10.164.12.159 verbose:Your password will expire in 8 days.\\r\\n[user01@machine043]>\\r\\n10.164.12.159 verbose:su - root\\r\\nPassword:\\r\\n[root@machine043]>\\r\\n10.164.12.159 verbose:echo hello\\r\\nhello\\r\\n[root@machine043]>\\r\\nTimeout error: 10.164.12.159: Command timed out after 5 seconds\\r\\nCompleted\\r\\n\\r\\nAny commands I execute before trying the second su, it works perfectly but after the second su I only get Timeout error.\\r\\n\\r\\nThis is the full session responses:\\r\\nConnected to 10.164.12.159\\r\\nsu - root\\r\\nPassword:\\r\\n[root@machine043]> echo hello\\r\\nhello\\r\\n[root@machine043]> su - webapp\\r\\nmachine043@webapp:~>\\r\\n\\r\\nAny insights you could share with me?\\r\\nCould this be happening because the name of the prompt is different from the first two?  Where simpleuser and root has username@machinename and webapp has machinename@username?\\r\\n\\r\\nBtw, merry christmas! and have a happy new year!\"},\"comment\":{\"url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/comments/68477303\",\"html_url\":\"https://github.com/cmp-202/ssh2shell/issues/10#issuecomment-68477303\",\"issue_url\":\"https://api.github.com/repos/cmp-202/ssh2shell/issues/10\",\"id\":68477303,\"user\":{\"login\":\"cmp-202\",\"id\":8508800,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8508800?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmp-202\",\"html_url\":\"https://github.com/cmp-202\",\"followers_url\":\"https://api.github.com/users/cmp-202/followers\",\"following_url\":\"https://api.github.com/users/cmp-202/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cmp-202/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cmp-202/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cmp-202/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cmp-202/orgs\",\"repos_url\":\"https://api.github.com/users/cmp-202/repos\",\"events_url\":\"https://api.github.com/users/cmp-202/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cmp-202/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:23Z\",\"updated_at\":\"2015-01-01T01:04:23Z\",\"body\":\"could you show me the debug output.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:23Z\"}\n{\"id\":\"2489397767\",\"type\":\"GollumEvent\",\"actor\":{\"id\":720678,\"login\":\"kongr45gpen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kongr45gpen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/720678?\"},\"repo\":{\"id\":11615971,\"name\":\"allejo/bzion\",\"url\":\"https://api.github.com/repos/allejo/bzion\"},\"payload\":{\"pages\":[{\"page_name\":\"Installation\",\"title\":\"Installation\",\"summary\":null,\"action\":\"edited\",\"sha\":\"2abf7f83955235fa9a1bf76f57b9fdad0be16772\",\"html_url\":\"https://github.com/allejo/bzion/wiki/Installation\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:23Z\"}\n{\"id\":\"2489397775\",\"type\":\"PushEvent\",\"actor\":{\"id\":4362193,\"login\":\"derickc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/derickc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4362193?\"},\"repo\":{\"id\":27286576,\"name\":\"ADML1/adml1.github.io\",\"url\":\"https://api.github.com/repos/ADML1/adml1.github.io\"},\"payload\":{\"push_id\":536753081,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b3ebd99a730a450c2adc2feb0654647710abf78e\",\"before\":\"378bfcffc91f8fa212d96dd2eba5abc21e3fc4c0\",\"commits\":[{\"sha\":\"b3ebd99a730a450c2adc2feb0654647710abf78e\",\"author\":{\"email\":\"45a58c873e280dbee110e79514ff8a5e02d3e2be@gmail.com\",\"name\":\"Derick\"},\"message\":\"Update draft and change padding to make more mobile friendly.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ADML1/adml1.github.io/commits/b3ebd99a730a450c2adc2feb0654647710abf78e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:25Z\",\"org\":{\"id\":9938744,\"login\":\"ADML1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ADML1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9938744?\"}}\n{\"id\":\"2489397776\",\"type\":\"PushEvent\",\"actor\":{\"id\":5567815,\"login\":\"alexlovesprogramming\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexlovesprogramming\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5567815?\"},\"repo\":{\"id\":28524892,\"name\":\"alexlovesprogramming/longstreetliving\",\"url\":\"https://api.github.com/repos/alexlovesprogramming/longstreetliving\"},\"payload\":{\"push_id\":536753082,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"edebc67bf340a3c896cc6204a94332e03f67b52f\",\"before\":\"81b81aba6f76f0b9816556e3267bba820f867c5f\",\"commits\":[{\"sha\":\"cd4288723ab21d37c17079153f566ab827a0bc5d\",\"author\":{\"email\":\"bf67c384ca4ec67ca70eb215259e583bfbc24f58@gmail.com\",\"name\":\"Alex Johnson\"},\"message\":\"signature\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alexlovesprogramming/longstreetliving/commits/cd4288723ab21d37c17079153f566ab827a0bc5d\"},{\"sha\":\"edebc67bf340a3c896cc6204a94332e03f67b52f\",\"author\":{\"email\":\"bf67c384ca4ec67ca70eb215259e583bfbc24f58@gmail.com\",\"name\":\"Alex Johnson\"},\"message\":\"Merge branch 'master' of https://server.longstreetliving.com/Bonobo.Git.Server/longstreetliving\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alexlovesprogramming/longstreetliving/commits/edebc67bf340a3c896cc6204a94332e03f67b52f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:25Z\"}\n{\"id\":\"2489397779\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1203825,\"login\":\"huonw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/huonw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1203825?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/20379\",\"labels_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20379/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20379/comments\",\"events_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20379/events\",\"html_url\":\"https://github.com/rust-lang/rust/issues/20379\",\"id\":53207500,\"number\":20379,\"title\":\"recursion dependent on global offset table\",\"user\":{\"login\":\"ragingSloth\",\"id\":3530278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3530278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ragingSloth\",\"html_url\":\"https://github.com/ragingSloth\",\"followers_url\":\"https://api.github.com/users/ragingSloth/followers\",\"following_url\":\"https://api.github.com/users/ragingSloth/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ragingSloth/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ragingSloth/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ragingSloth/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ragingSloth/orgs\",\"repos_url\":\"https://api.github.com/users/ragingSloth/repos\",\"events_url\":\"https://api.github.com/users/ragingSloth/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ragingSloth/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T23:03:06Z\",\"updated_at\":\"2015-01-01T01:04:25Z\",\"closed_at\":null,\"body\":\"While trying to use rust freestanding I found that any attempt to use recursion resulted in an error while linking. \\r\\n\\r\\n    main.o: In function `put':\\r\\n    main.0.rs:(.text.put+0xc): undefined reference to `_GLOBAL_OFFSET_TABLE_'\\r\\n    main.o: In function `r_write':\\r\\n    main.0.rs:(.text.r_write+0xf): undefined reference to `_GLOBAL_OFFSET_TABLE_'\\r\\n\\r\\nI'm not sure if this is an issue, but I can't find any way to use recursion without a global offset table. I'm compiling with `#[no_std]` and `-O --target i686-unknown-linux-gnu --crate-type lib --emit obj`, and I recieve errors when trying to link separately. \"},\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/comments/68477305\",\"html_url\":\"https://github.com/rust-lang/rust/issues/20379#issuecomment-68477305\",\"issue_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20379\",\"id\":68477305,\"user\":{\"login\":\"huonw\",\"id\":1203825,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1203825?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/huonw\",\"html_url\":\"https://github.com/huonw\",\"followers_url\":\"https://api.github.com/users/huonw/followers\",\"following_url\":\"https://api.github.com/users/huonw/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/huonw/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/huonw/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/huonw/subscriptions\",\"organizations_url\":\"https://api.github.com/users/huonw/orgs\",\"repos_url\":\"https://api.github.com/users/huonw/repos\",\"events_url\":\"https://api.github.com/users/huonw/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/huonw/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:25Z\",\"updated_at\":\"2015-01-01T01:04:25Z\",\"body\":\"Do you happen to have a code example that demonstrates this?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:26Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489397781\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":706947,\"login\":\"d3athrow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?\"},\"repo\":{\"id\":10441188,\"name\":\"d3athrow/vgstation13\",\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\"},\"payload\":{\"action\":\"closed\",\"number\":2417,\"pull_request\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417\",\"id\":26723211,\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2417\",\"diff_url\":\"https://github.com/d3athrow/vgstation13/pull/2417.diff\",\"patch_url\":\"https://github.com/d3athrow/vgstation13/pull/2417.patch\",\"issue_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2417\",\"number\":2417,\"state\":\"closed\",\"locked\":false,\"title\":\"A few defficiency fixes\",\"user\":{\"login\":\"Duny-\",\"id\":5224390,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5224390?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Duny-\",\"html_url\":\"https://github.com/Duny-\",\"followers_url\":\"https://api.github.com/users/Duny-/followers\",\"following_url\":\"https://api.github.com/users/Duny-/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Duny-/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Duny-/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Duny-/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Duny-/orgs\",\"repos_url\":\"https://api.github.com/users/Duny-/repos\",\"events_url\":\"https://api.github.com/users/Duny-/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Duny-/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"- Added all missing holodeck settings.\\r\\n- Fixed missing pipe in supermatter room.\\r\\n- Tweaked toxins launcher to hopefully work correctly in higher ZAS settings.\",\"created_at\":\"2014-12-31T12:38:43Z\",\"updated_at\":\"2015-01-01T01:04:26Z\",\"closed_at\":\"2015-01-01T01:04:26Z\",\"merged_at\":\"2015-01-01T01:04:26Z\",\"merge_commit_sha\":\"0aba215d4d761c270f9e576576832eea9cb48658\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417/commits\",\"review_comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417/comments\",\"review_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2417/comments\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/34f7fa7b80f675ce4e56f4bc148338104b8155ee\",\"head\":{\"label\":\"Duny-:Bleeding-Edge\",\"ref\":\"Bleeding-Edge\",\"sha\":\"34f7fa7b80f675ce4e56f4bc148338104b8155ee\",\"user\":{\"login\":\"Duny-\",\"id\":5224390,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5224390?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Duny-\",\"html_url\":\"https://github.com/Duny-\",\"followers_url\":\"https://api.github.com/users/Duny-/followers\",\"following_url\":\"https://api.github.com/users/Duny-/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Duny-/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Duny-/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Duny-/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Duny-/orgs\",\"repos_url\":\"https://api.github.com/users/Duny-/repos\",\"events_url\":\"https://api.github.com/users/Duny-/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Duny-/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":20945462,\"name\":\"vgstation13\",\"full_name\":\"Duny-/vgstation13\",\"owner\":{\"login\":\"Duny-\",\"id\":5224390,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5224390?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Duny-\",\"html_url\":\"https://github.com/Duny-\",\"followers_url\":\"https://api.github.com/users/Duny-/followers\",\"following_url\":\"https://api.github.com/users/Duny-/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Duny-/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Duny-/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Duny-/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Duny-/orgs\",\"repos_url\":\"https://api.github.com/users/Duny-/repos\",\"events_url\":\"https://api.github.com/users/Duny-/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Duny-/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Duny-/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/Duny-/vgstation13\",\"forks_url\":\"https://api.github.com/repos/Duny-/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/Duny-/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Duny-/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Duny-/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/Duny-/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Duny-/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Duny-/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/Duny-/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Duny-/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Duny-/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/Duny-/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Duny-/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Duny-/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Duny-/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Duny-/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Duny-/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/Duny-/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Duny-/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Duny-/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Duny-/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/Duny-/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Duny-/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Duny-/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Duny-/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Duny-/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Duny-/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Duny-/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/Duny-/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Duny-/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/Duny-/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Duny-/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Duny-/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Duny-/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Duny-/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Duny-/vgstation13/releases{/id}\",\"created_at\":\"2014-06-18T01:41:53Z\",\"updated_at\":\"2014-12-31T16:30:54Z\",\"pushed_at\":\"2014-12-31T16:30:54Z\",\"git_url\":\"git://github.com/Duny-/vgstation13.git\",\"ssh_url\":\"git@github.com:Duny-/vgstation13.git\",\"clone_url\":\"https://github.com/Duny-/vgstation13.git\",\"svn_url\":\"https://github.com/Duny-/vgstation13\",\"homepage\":\"\",\"size\":712478,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":18,\"forks\":0,\"open_issues\":18,\"watchers\":0,\"default_branch\":\"Bleeding-Edge\"}},\"base\":{\"label\":\"d3athrow:Bleeding-Edge\",\"ref\":\"Bleeding-Edge\",\"sha\":\"5e8624143efbf348ed553421b76293ef84acac57\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":10441188,\"name\":\"vgstation13\",\"full_name\":\"d3athrow/vgstation13\",\"owner\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/d3athrow/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\",\"forks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/d3athrow/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/d3athrow/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/d3athrow/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/d3athrow/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/d3athrow/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/d3athrow/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/d3athrow/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/d3athrow/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/d3athrow/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/d3athrow/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/d3athrow/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/d3athrow/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/d3athrow/vgstation13/releases{/id}\",\"created_at\":\"2013-06-02T19:39:54Z\",\"updated_at\":\"2014-12-31T20:06:46Z\",\"pushed_at\":\"2015-01-01T01:04:26Z\",\"git_url\":\"git://github.com/d3athrow/vgstation13.git\",\"ssh_url\":\"git@github.com:d3athrow/vgstation13.git\",\"clone_url\":\"https://github.com/d3athrow/vgstation13.git\",\"svn_url\":\"https://github.com/d3athrow/vgstation13\",\"homepage\":\"\",\"size\":937605,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":135,\"mirror_url\":null,\"open_issues_count\":259,\"forks\":135,\"open_issues\":259,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2417\"},\"issue\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2417\"},\"comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2417/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2417/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/34f7fa7b80f675ce4e56f4bc148338104b8155ee\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":3,\"additions\":115,\"deletions\":42,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:04:26Z\"}\n{\"id\":\"2489397782\",\"type\":\"PushEvent\",\"actor\":{\"id\":568018,\"login\":\"jon-jacky\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jon-jacky\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/568018?\"},\"repo\":{\"id\":3722889,\"name\":\"jon-jacky/home\",\"url\":\"https://api.github.com/repos/jon-jacky/home\"},\"payload\":{\"push_id\":536753085,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"ca7c98c62a6cf4709b98f5d5bb453c1b2dd120bc\",\"before\":\"11b7788a7c84834415f6f53b531b9c20ddf887b8\",\"commits\":[{\"sha\":\"ca7c98c62a6cf4709b98f5d5bb453c1b2dd120bc\",\"author\":{\"email\":\"44f878afe53efc66b76772bd845eb65944ed8232@u.washington.edu\",\"name\":\"Jon Jacky\"},\"message\":\"Turing's manual for Mark II in computer architecture\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jon-jacky/home/commits/ca7c98c62a6cf4709b98f5d5bb453c1b2dd120bc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:26Z\"}\n{\"id\":\"2489397787\",\"type\":\"PushEvent\",\"actor\":{\"id\":1588951,\"login\":\"TAGC\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TAGC\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1588951?\"},\"repo\":{\"id\":28516105,\"name\":\"TAGC/Semver\",\"url\":\"https://api.github.com/repos/TAGC/Semver\"},\"payload\":{\"push_id\":536753087,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"c14bd6aeef1be553879694e2c3ba8535749f5380\",\"before\":\"cbeb387cb6c3ba0f1d6b42ad32cf700f296424ba\",\"commits\":[{\"sha\":\"c14bd6aeef1be553879694e2c3ba8535749f5380\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Update README.md\\n\\nUpdate README to reflect changes brought about in v0.3.0\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/c14bd6aeef1be553879694e2c3ba8535749f5380\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:27Z\"}\n{\"id\":\"2489397788\",\"type\":\"PushEvent\",\"actor\":{\"id\":5681361,\"login\":\"snoguchi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/snoguchi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5681361?\"},\"repo\":{\"id\":19981496,\"name\":\"snoguchi/dotfiles\",\"url\":\"https://api.github.com/repos/snoguchi/dotfiles\"},\"payload\":{\"push_id\":536753088,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3d73778740a21a43e90a97d76d0b316fd0d051a1\",\"before\":\"34d21e71013bd74f2050e32eac38d65d8d19d2ab\",\"commits\":[{\"sha\":\"3d73778740a21a43e90a97d76d0b316fd0d051a1\",\"author\":{\"email\":\"6ae999552a0d2dca14d62e2bc8b764d377b1dd6c\",\"name\":\"U-X220\\\\noguchi\"},\"message\":\"change key bindings\",\"distinct\":true,\"url\":\"https://api.github.com/repos/snoguchi/dotfiles/commits/3d73778740a21a43e90a97d76d0b316fd0d051a1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:27Z\"}\n{\"id\":\"2489397789\",\"type\":\"PushEvent\",\"actor\":{\"id\":1684950,\"login\":\"naijaping\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/naijaping\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1684950?\"},\"repo\":{\"id\":28650038,\"name\":\"naijaping/awonlist\",\"url\":\"https://api.github.com/repos/naijaping/awonlist\"},\"payload\":{\"push_id\":536753089,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f454a5a195625a9128fddb04b192fcc457e82091\",\"before\":\"e80946e9abd94a00ce15b06ad502211a518e2ffd\",\"commits\":[{\"sha\":\"f454a5a195625a9128fddb04b192fcc457e82091\",\"author\":{\"email\":\"8a1440b218d23a283d388025f7c9dc3555009ec5@gmail.com\",\"name\":\"naijaping\"},\"message\":\"Update uk\",\"distinct\":true,\"url\":\"https://api.github.com/repos/naijaping/awonlist/commits/f454a5a195625a9128fddb04b192fcc457e82091\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:27Z\"}\n{\"id\":\"2489397791\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":594255,\"login\":\"Xaekai\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Xaekai\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/594255?\"},\"repo\":{\"id\":8231654,\"name\":\"calzoneman/sync\",\"url\":\"https://api.github.com/repos/calzoneman/sync\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/calzoneman/sync/issues/423\",\"labels_url\":\"https://api.github.com/repos/calzoneman/sync/issues/423/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/calzoneman/sync/issues/423/comments\",\"events_url\":\"https://api.github.com/repos/calzoneman/sync/issues/423/events\",\"html_url\":\"https://github.com/calzoneman/sync/issues/423\",\"id\":53191677,\"number\":423,\"title\":\"tab completion\",\"user\":{\"login\":\"estillesemae\",\"id\":10359338,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10359338?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/estillesemae\",\"html_url\":\"https://github.com/estillesemae\",\"followers_url\":\"https://api.github.com/users/estillesemae/followers\",\"following_url\":\"https://api.github.com/users/estillesemae/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/estillesemae/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/estillesemae/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/estillesemae/subscriptions\",\"organizations_url\":\"https://api.github.com/users/estillesemae/orgs\",\"repos_url\":\"https://api.github.com/users/estillesemae/repos\",\"events_url\":\"https://api.github.com/users/estillesemae/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/estillesemae/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-31T16:38:03Z\",\"updated_at\":\"2015-01-01T01:04:28Z\",\"closed_at\":\"2014-12-31T16:55:56Z\",\"body\":\"There is an issue with tab completion in the mlp anniversary room. Xaekai is sleeping, I thought I would inform you.\\r\\n\\r\\nMost of the time, when tab completion is used, an \\\"image.channel-emote\\\" appears. It is a blank box with nothing in it. Cannot click or anything. \\r\\nhttp://i.imgur.com/zZuj6hf.png\\r\\nhttp://i.imgur.com/Y1D8P4e.png\\r\\n\\r\\nI noticed it is only in the anni room though. I don't know if you can help, but that would be great!!\\r\\n\\r\\nEstillesemae\"},\"comment\":{\"url\":\"https://api.github.com/repos/calzoneman/sync/issues/comments/68477307\",\"html_url\":\"https://github.com/calzoneman/sync/issues/423#issuecomment-68477307\",\"issue_url\":\"https://api.github.com/repos/calzoneman/sync/issues/423\",\"id\":68477307,\"user\":{\"login\":\"Xaekai\",\"id\":594255,\"avatar_url\":\"https://avatars.githubusercontent.com/u/594255?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Xaekai\",\"html_url\":\"https://github.com/Xaekai\",\"followers_url\":\"https://api.github.com/users/Xaekai/followers\",\"following_url\":\"https://api.github.com/users/Xaekai/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Xaekai/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Xaekai/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Xaekai/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Xaekai/orgs\",\"repos_url\":\"https://api.github.com/users/Xaekai/repos\",\"events_url\":\"https://api.github.com/users/Xaekai/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Xaekai/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:28Z\",\"updated_at\":\"2015-01-01T01:04:28Z\",\"body\":\"@estillesemae My room features aren't calzoneman's problem or domain. Nice to see you using the dev console to figure shit out though.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:28Z\"}\n{\"id\":\"2489397792\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":238354,\"login\":\"variousred\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/variousred\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/238354?\"},\"repo\":{\"id\":6274404,\"name\":\"G5/g5-content-management-system\",\"url\":\"https://api.github.com/repos/G5/g5-content-management-system\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/551\",\"labels_url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/551/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/551/comments\",\"events_url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/551/events\",\"html_url\":\"https://github.com/G5/g5-content-management-system/pull/551\",\"id\":53061666,\"number\":551,\"title\":\"WIP: Fix Deep Clone\",\"user\":{\"login\":\"ceckert\",\"id\":779559,\"avatar_url\":\"https://avatars.githubusercontent.com/u/779559?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ceckert\",\"html_url\":\"https://github.com/ceckert\",\"followers_url\":\"https://api.github.com/users/ceckert/followers\",\"following_url\":\"https://api.github.com/users/ceckert/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ceckert/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ceckert/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ceckert/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ceckert/orgs\",\"repos_url\":\"https://api.github.com/users/ceckert/repos\",\"events_url\":\"https://api.github.com/users/ceckert/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ceckert/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-29T18:51:49Z\",\"updated_at\":\"2015-01-01T01:04:28Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/G5/g5-content-management-system/pulls/551\",\"html_url\":\"https://github.com/G5/g5-content-management-system/pull/551\",\"diff_url\":\"https://github.com/G5/g5-content-management-system/pull/551.diff\",\"patch_url\":\"https://github.com/G5/g5-content-management-system/pull/551.patch\"},\"body\":\"Deep clone was only cloning one level deep. This adds recursive cloning. \"},\"comment\":{\"url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/comments/68477308\",\"html_url\":\"https://github.com/G5/g5-content-management-system/pull/551#issuecomment-68477308\",\"issue_url\":\"https://api.github.com/repos/G5/g5-content-management-system/issues/551\",\"id\":68477308,\"user\":{\"login\":\"variousred\",\"id\":238354,\"avatar_url\":\"https://avatars.githubusercontent.com/u/238354?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/variousred\",\"html_url\":\"https://github.com/variousred\",\"followers_url\":\"https://api.github.com/users/variousred/followers\",\"following_url\":\"https://api.github.com/users/variousred/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/variousred/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/variousred/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/variousred/subscriptions\",\"organizations_url\":\"https://api.github.com/users/variousred/orgs\",\"repos_url\":\"https://api.github.com/users/variousred/repos\",\"events_url\":\"https://api.github.com/users/variousred/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/variousred/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:28Z\",\"updated_at\":\"2015-01-01T01:04:28Z\",\"body\":\"@ceckert this one failing test remaining might atually be uncovering a real bug. It seems we are creating an extra widget in the clone process.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:28Z\",\"org\":{\"id\":2396851,\"login\":\"G5\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/G5\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2396851?\"}}\n{\"id\":\"2489397796\",\"type\":\"PushEvent\",\"actor\":{\"id\":3056753,\"login\":\"Yukariko\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Yukariko\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3056753?\"},\"repo\":{\"id\":21538859,\"name\":\"Yukariko/acm\",\"url\":\"https://api.github.com/repos/Yukariko/acm\"},\"payload\":{\"push_id\":536753092,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cfa04165890a46738082b2f56b58b525ba1579d5\",\"before\":\"bb9f0c6c8871960d6a68bd3d5ae8817b2ec112f9\",\"commits\":[{\"sha\":\"cfa04165890a46738082b2f56b58b525ba1579d5\",\"author\":{\"email\":\"5644dc45614202cf4a35cbeeaf0c6d61d1144e06@naver.com\",\"name\":\"Yukariko\"},\"message\":\"Accept\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Yukariko/acm/commits/cfa04165890a46738082b2f56b58b525ba1579d5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:28Z\"}\n{\"id\":\"2489397797\",\"type\":\"PushEvent\",\"actor\":{\"id\":8454656,\"login\":\"klob\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/klob\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8454656?\"},\"repo\":{\"id\":27668613,\"name\":\"diandy/diandy1.2.0\",\"url\":\"https://api.github.com/repos/diandy/diandy1.2.0\"},\"payload\":{\"push_id\":536753093,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d4cbddba067260c9b55d5b1b164f0fdb65a11bf1\",\"before\":\"a2bc2dfcd2cdfca23b7e371302a6d56cf8f68115\",\"commits\":[{\"sha\":\"d4cbddba067260c9b55d5b1b164f0fdb65a11bf1\",\"author\":{\"email\":\"eae7ed76150cb5d2ded6223b270823aba8a08eb9@qq.com\",\"name\":\"klob\"},\"message\":\" all changelist\",\"distinct\":true,\"url\":\"https://api.github.com/repos/diandy/diandy1.2.0/commits/d4cbddba067260c9b55d5b1b164f0fdb65a11bf1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:28Z\",\"org\":{\"id\":10010251,\"login\":\"diandy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/diandy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10010251?\"}}\n{\"id\":\"2489397799\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753095,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a2211fba71212268a1c0f0b3dc4ddb1412bf72b6\",\"before\":\"0c74567d174b0901cf19bde2f2ddc65e1d4ff6e1\",\"commits\":[{\"sha\":\"a2211fba71212268a1c0f0b3dc4ddb1412bf72b6\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"$markComponent => $component\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/a2211fba71212268a1c0f0b3dc4ddb1412bf72b6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:29Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489397802\",\"type\":\"PushEvent\",\"actor\":{\"id\":8819701,\"login\":\"r-ggraham\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r-ggraham\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8819701?\"},\"repo\":{\"id\":28678173,\"name\":\"r-ggraham/Crumpet_Bot\",\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot\"},\"payload\":{\"push_id\":536753097,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ee45c3d188e25fa5ee1d0d5a8a9f3646854ccbc7\",\"before\":\"51951542e9d518f59ff62753725e5186d4111822\",\"commits\":[{\"sha\":\"ee45c3d188e25fa5ee1d0d5a8a9f3646854ccbc7\",\"author\":{\"email\":\"f2f9dd43aa4244d32208a2ccfa0c7c9e9c48f7e7@uni.worc.ac.uk\",\"name\":\"Rob G\"},\"message\":\"formatting\",\"distinct\":true,\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot/commits/ee45c3d188e25fa5ee1d0d5a8a9f3646854ccbc7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:29Z\"}\n{\"id\":\"2489397805\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/82\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/82\",\"id\":53210240,\"number\":82,\"title\":\"Add background color to done buttons in all Webviews\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":5,\"closed_issues\":11,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"due_on\":null,\"closed_at\":null},\"comments\":1,\"created_at\":\"2015-01-01T01:03:44Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"closed_at\":\"2015-01-01T01:04:29Z\",\"body\":\"- 2 on Donate screen\\r\\n- 1 on About screen\"},\"comment\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/comments/68477309\",\"html_url\":\"https://github.com/g19-mr/azh/issues/82#issuecomment-68477309\",\"issue_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82\",\"id\":68477309,\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:29Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"body\":\"Made Done buttons the same color as buttons on about screen\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:30Z\"}\n{\"id\":\"2489397806\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/82\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/82/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/82\",\"id\":53210240,\"number\":82,\"title\":\"Add background color to done buttons in all Webviews\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":5,\"closed_issues\":11,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"due_on\":null,\"closed_at\":null},\"comments\":1,\"created_at\":\"2015-01-01T01:03:44Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"closed_at\":\"2015-01-01T01:04:29Z\",\"body\":\"- 2 on Donate screen\\r\\n- 1 on About screen\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:30Z\"}\n{\"id\":\"2489397807\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1682199,\"login\":\"JayBeavers\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?\"},\"repo\":{\"id\":21885551,\"name\":\"erikringsmuth/app-router\",\"url\":\"https://api.github.com/repos/erikringsmuth/app-router\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63\",\"labels_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/comments\",\"events_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/events\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/63\",\"id\":53209410,\"number\":63,\"title\":\"Advice on passing a fullbleed Polymer flexbox layout through the router\",\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2015-01-01T00:19:01Z\",\"updated_at\":\"2015-01-01T01:04:30Z\",\"closed_at\":\"2015-01-01T01:04:30Z\",\"body\":\"I'm writing a full-screen app (fullbleed) and relying on the flexbox/layout features of Polymer for my layout design.\\r\\n\\r\\nWhen I moved to the app-router approach, I lost my screen measurements and have been working through the implications since.  At this point, I'm liberally applying 'layout/flex' to my app and I seem to be getting somewhere, but I'm still losing my flexbox support when I cross the app-route/import boundary.\\r\\n\\r\\nBefore I go trundling down the path of debugging/modifying app-router itself, is this something others have played with in the past?  Is there an established pattern I should be following?  Unfortunately I'm simultanouesly new to Polymer, flexbox/layouts, and app-router at the same time so I'm not really certain what is 'supposed' to work and what needs custom coding.\\r\\n\\r\\nRight now my pseduo-code is looking like:\\r\\n\\r\\n    <html>\\r\\n      <body fullbleed layout vertical unresolved>\\r\\n        <app-router flex layout vertical>\\r\\n          <app-route flex layout vertical> <!-- flex here isn't working as its dividing the screen real-estate amoung all my routes, I really mean 'flex but only on the active route' -->\\r\\n            <imported-page-template flex layout vertical> <!-- flex isn't working at all here, does not seem to cross the import boundary -->\\r\\n              ...\\r\\n\\r\\nAny thoughts for the lost and weary?\"},\"comment\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/comments/68477310\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/63#issuecomment-68477310\",\"issue_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63\",\"id\":68477310,\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:29Z\",\"updated_at\":\"2015-01-01T01:04:29Z\",\"body\":\"OK, thanks for the advice.  I was able to narrow down the attributes to get the system 'working as expected', at least after moderate testing to:\\r\\n\\r\\n```\\r\\n    <html>\\r\\n      <body fullbleed layout vertical unresolved>\\r\\n        <app-router>\\r\\n          <app-route import='...'>\\r\\n            <polymer-element>\\r\\n              <template>\\r\\n                <div layout vertical center>\\r\\n                  ...\\r\\n```\\r\\n\\r\\nThis seemed to pick up the information it needed, my missing step was that I needed a containing div inside the template of my polymer-element/page.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:30Z\"}\n{\"id\":\"2489397808\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1682199,\"login\":\"JayBeavers\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?\"},\"repo\":{\"id\":21885551,\"name\":\"erikringsmuth/app-router\",\"url\":\"https://api.github.com/repos/erikringsmuth/app-router\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63\",\"labels_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/comments\",\"events_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/63/events\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/63\",\"id\":53209410,\"number\":63,\"title\":\"Advice on passing a fullbleed Polymer flexbox layout through the router\",\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2015-01-01T00:19:01Z\",\"updated_at\":\"2015-01-01T01:04:30Z\",\"closed_at\":\"2015-01-01T01:04:30Z\",\"body\":\"I'm writing a full-screen app (fullbleed) and relying on the flexbox/layout features of Polymer for my layout design.\\r\\n\\r\\nWhen I moved to the app-router approach, I lost my screen measurements and have been working through the implications since.  At this point, I'm liberally applying 'layout/flex' to my app and I seem to be getting somewhere, but I'm still losing my flexbox support when I cross the app-route/import boundary.\\r\\n\\r\\nBefore I go trundling down the path of debugging/modifying app-router itself, is this something others have played with in the past?  Is there an established pattern I should be following?  Unfortunately I'm simultanouesly new to Polymer, flexbox/layouts, and app-router at the same time so I'm not really certain what is 'supposed' to work and what needs custom coding.\\r\\n\\r\\nRight now my pseduo-code is looking like:\\r\\n\\r\\n    <html>\\r\\n      <body fullbleed layout vertical unresolved>\\r\\n        <app-router flex layout vertical>\\r\\n          <app-route flex layout vertical> <!-- flex here isn't working as its dividing the screen real-estate amoung all my routes, I really mean 'flex but only on the active route' -->\\r\\n            <imported-page-template flex layout vertical> <!-- flex isn't working at all here, does not seem to cross the import boundary -->\\r\\n              ...\\r\\n\\r\\nAny thoughts for the lost and weary?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:30Z\"}\n{\"id\":\"2489397809\",\"type\":\"PushEvent\",\"actor\":{\"id\":706947,\"login\":\"d3athrow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?\"},\"repo\":{\"id\":10441188,\"name\":\"d3athrow/vgstation13\",\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\"},\"payload\":{\"push_id\":536753099,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/Bleeding-Edge\",\"head\":\"54c4e0ffb341dc6ebdf8ce724b13e357234b9af0\",\"before\":\"d274ed664f000dd393d16d6ebd5234332985ac8d\",\"commits\":[{\"sha\":\"19881ecd196d4282777ba1ce93929aae3d4ca71c\",\"author\":{\"email\":\"4b46b39232cc3015db82c31a969ae73f4e58ae9a@gmail.com\",\"name\":\"Duny-\"},\"message\":\"A few defficiency fixes\\n\\n- Added all missing holodeck settings\\n- Added missing pipe in the supermatter room\\n- Tweaked toxins launcher, hopefully it doesn't refuse to launch bombs\\nin higher ZAS settings anymore\",\"distinct\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits/19881ecd196d4282777ba1ce93929aae3d4ca71c\"},{\"sha\":\"ebd9ca32a9b118aa3f81188d375a5d9447f32514\",\"author\":{\"email\":\"4b46b39232cc3015db82c31a969ae73f4e58ae9a@gmail.com\",\"name\":\"Duny-\"},\"message\":\"Changelog\",\"distinct\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits/ebd9ca32a9b118aa3f81188d375a5d9447f32514\"},{\"sha\":\"34f7fa7b80f675ce4e56f4bc148338104b8155ee\",\"author\":{\"email\":\"4b46b39232cc3015db82c31a969ae73f4e58ae9a@gmail.com\",\"name\":\"Duny-\"},\"message\":\"Tiny lil fix\",\"distinct\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits/34f7fa7b80f675ce4e56f4bc148338104b8155ee\"},{\"sha\":\"54c4e0ffb341dc6ebdf8ce724b13e357234b9af0\",\"author\":{\"email\":\"bf439111772c53efe08334e0c2bcb283a1d320cb@yahoo.com\",\"name\":\"d3athrow\"},\"message\":\"Merge pull request #2417 from Duny-/Bleeding-Edge\\n\\nA few defficiency fixes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits/54c4e0ffb341dc6ebdf8ce724b13e357234b9af0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:30Z\"}\n{\"id\":\"2489397821\",\"type\":\"PushEvent\",\"actor\":{\"id\":6355392,\"login\":\"githanwang1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/githanwang1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6355392?\"},\"repo\":{\"id\":28543231,\"name\":\"githanwang1/django-blog\",\"url\":\"https://api.github.com/repos/githanwang1/django-blog\"},\"payload\":{\"push_id\":536753105,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"130cfc38c0c001e58f84398948b72b138ccce717\",\"before\":\"84d296175511f1df9097fb18be3ec500e8acc701\",\"commits\":[{\"sha\":\"130cfc38c0c001e58f84398948b72b138ccce717\",\"author\":{\"email\":\"053e32d42d025177f9df81fc22020283a55f18ff@berkeley.edu\",\"name\":\"Han Wang\"},\"message\":\"new blog lel\",\"distinct\":true,\"url\":\"https://api.github.com/repos/githanwang1/django-blog/commits/130cfc38c0c001e58f84398948b72b138ccce717\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:32Z\"}\n{\"id\":\"2489397824\",\"type\":\"PushEvent\",\"actor\":{\"id\":1779595,\"login\":\"dcbaker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcbaker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1779595?\"},\"repo\":{\"id\":8488437,\"name\":\"dcbaker/piglit\",\"url\":\"https://api.github.com/repos/dcbaker/piglit\"},\"payload\":{\"push_id\":536753107,\"size\":0,\"distinct_size\":0,\"ref\":\"refs/heads/wip/command-list-only-v5\",\"head\":\"cf5cf7224385c3c3d086762b5bc1d8f7df031ae7\",\"before\":\"e81f4b28bf6ad78c7fb05257e77e42fe237a09ed\",\"commits\":[]},\"public\":true,\"created_at\":\"2015-01-01T01:04:32Z\"}\n{\"id\":\"2489397826\",\"type\":\"PushEvent\",\"actor\":{\"id\":170479,\"login\":\"tdhooper\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdhooper\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/170479?\"},\"repo\":{\"id\":19411601,\"name\":\"tdhooper/starstoloves\",\"url\":\"https://api.github.com/repos/tdhooper/starstoloves\"},\"payload\":{\"push_id\":536753109,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"b75fca64d82ce8d4c496bb3ab353bc7bed89a46c\",\"before\":\"249d09e95e69c92cf5b1fd133d659271495d59a3\",\"commits\":[{\"sha\":\"cd1431978800b65553ad260162efa5d346f49d57\",\"author\":{\"email\":\"c136eba45d3ee7ac4dbbfc7f9f9d33c11a99a23f@Macintosh.local\",\"name\":\"Thomas Hooper\"},\"message\":\"Show time loved\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tdhooper/starstoloves/commits/cd1431978800b65553ad260162efa5d346f49d57\"},{\"sha\":\"b75fca64d82ce8d4c496bb3ab353bc7bed89a46c\",\"author\":{\"email\":\"c136eba45d3ee7ac4dbbfc7f9f9d33c11a99a23f@Macintosh.local\",\"name\":\"Thomas Hooper\"},\"message\":\"Get fresh loved tracks after loving\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tdhooper/starstoloves/commits/b75fca64d82ce8d4c496bb3ab353bc7bed89a46c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:33Z\"}\n{\"id\":\"2489397827\",\"type\":\"GollumEvent\",\"actor\":{\"id\":46323,\"login\":\"paulcon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulcon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/46323?\"},\"repo\":{\"id\":28157780,\"name\":\"paulcon/active_subspaces\",\"url\":\"https://api.github.com/repos/paulcon/active_subspaces\"},\"payload\":{\"pages\":[{\"page_name\":\"_Footer\",\"title\":\"_Footer\",\"summary\":null,\"action\":\"edited\",\"sha\":\"bab75a681aedf8288c9da56929fca226c67102f3\",\"html_url\":\"https://github.com/paulcon/active_subspaces/wiki/_Footer\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:33Z\"}\n{\"id\":\"2489397831\",\"type\":\"PushEvent\",\"actor\":{\"id\":904370,\"login\":\"helhum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/helhum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/904370?\"},\"repo\":{\"id\":26716312,\"name\":\"TYPO3-Surf-CMS/Surf.CMS\",\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Surf.CMS\"},\"payload\":{\"push_id\":536753110,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0ab6946565a1611c3bf27a34d8686bad2b88a5d7\",\"before\":\"6c5278184c56a725e3cefd459de3cc77c78e0908\",\"commits\":[{\"sha\":\"0ab6946565a1611c3bf27a34d8686bad2b88a5d7\",\"author\":{\"email\":\"6bf857ca7de026fbed4ae790a809a0ea640901f4@helmuthummel.de\",\"name\":\"Helmut Hummel\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TYPO3-Surf-CMS/Surf.CMS/commits/0ab6946565a1611c3bf27a34d8686bad2b88a5d7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:33Z\",\"org\":{\"id\":7921669,\"login\":\"TYPO3-Surf-CMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TYPO3-Surf-CMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7921669?\"}}\n{\"id\":\"2489397835\",\"type\":\"PushEvent\",\"actor\":{\"id\":18191,\"login\":\"jc00ke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?\"},\"repo\":{\"id\":28678223,\"name\":\"jc00ke/chruby-fish\",\"url\":\"https://api.github.com/repos/jc00ke/chruby-fish\"},\"payload\":{\"push_id\":536753112,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"859d6a993456f190598eda5b2a188d96af984cf2\",\"before\":\"e8f28035e7570cbf90568e7dd087810ae7958c8d\",\"commits\":[{\"sha\":\"859d6a993456f190598eda5b2a188d96af984cf2\",\"author\":{\"email\":\"a5c95b3d7cb4d0ae05a15c79c79ab458dc2c8f9e@jc00ke.com\",\"name\":\"Jesse Cooke\"},\"message\":\"Document known PATH warning\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jc00ke/chruby-fish/commits/859d6a993456f190598eda5b2a188d96af984cf2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:34Z\"}\n{\"id\":\"2489397840\",\"type\":\"CreateEvent\",\"actor\":{\"id\":7637494,\"login\":\"Kf4btg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Kf4btg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7637494?\"},\"repo\":{\"id\":28275582,\"name\":\"Kf4btg/TIH_tAPImod\",\"url\":\"https://api.github.com/repos/Kf4btg/TIH_tAPImod\"},\"payload\":{\"ref\":\"1.1.2\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:35Z\"}\n{\"id\":\"2489397842\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":74571,\"login\":\"fitzgen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fitzgen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/74571?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355\",\"labels_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/comments\",\"events_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20355/events\",\"html_url\":\"https://github.com/rust-lang/rust/issues/20355\",\"id\":53157956,\"number\":20355,\"title\":\"Segfault in `Hasher<SipState>::hash`\",\"user\":{\"login\":\"fitzgen\",\"id\":74571,\"avatar_url\":\"https://avatars.githubusercontent.com/u/74571?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fitzgen\",\"html_url\":\"https://github.com/fitzgen\",\"followers_url\":\"https://api.github.com/users/fitzgen/followers\",\"following_url\":\"https://api.github.com/users/fitzgen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/fitzgen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/fitzgen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/fitzgen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/fitzgen/orgs\",\"repos_url\":\"https://api.github.com/users/fitzgen/repos\",\"events_url\":\"https://api.github.com/users/fitzgen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/fitzgen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/rust-lang/rust/labels/I-crash\",\"name\":\"I-crash\",\"color\":\"e10c02\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":6,\"created_at\":\"2014-12-31T00:04:46Z\",\"updated_at\":\"2015-01-01T01:04:35Z\",\"closed_at\":\"2015-01-01T01:04:35Z\",\"body\":\"Test case is branch `sip-hasher-segault` of this repo: https://github.com/fitzgen/oxischeme/tree/sip-hasher-segfault\\r\\n\\r\\n    $ rustc --version --verbose\\r\\n    rustc --version --verbose\\r\\n    rustc 0.13.0-nightly (5ba610265 2014-12-25 18:01:36 +0000)\\r\\n    binary: rustc\\r\\n    commit-hash: 5ba6102657a892457063d2d6a7cbb9632ce282c6\\r\\n    commit-date: 2014-12-25 18:01:36 +0000\\r\\n    host: x86_64-apple-darwin\\r\\n    release: 0.13.0-nightly\\r\\n    $ rustc -g --test src/main.rs -o target/test\\r\\n    $ lldb ./target/test\\r\\n    lldb ./target/test\\r\\n    (lldb) target create \\\"./target/test\\\"\\r\\n    Current executable set to './target/test' (x86_64).\\r\\n    (lldb) run\\r\\n    Process 8270 launched: './target/test' (x86_64)\\r\\n\\r\\n    running 20 tests\\r\\n    Process 8270 stopped\\r\\n    * thread #2: tid = 0x25693, 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61, stop reason = EXC_BAD_ACCESS (code=1, address=0x3ffd9)\\r\\n        frame #0: 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61\\r\\n    test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61:\\r\\n    -> 0x100008bed:  movq   (%rsi), %rsi\\r\\n       0x100008bf0:  movq   -0x8(%rbp), %rdi\\r\\n       0x100008bf4:  movq   0x8(%rdi), %rdx\\r\\n       0x100008bf8:  movq   %rax, %rdi\\r\\n    (lldb) bt\\r\\n    * thread #2: tid = 0x25693, 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61, stop reason = EXC_BAD_ACCESS (code=1, address=0x3ffd9)\\r\\n      * frame #0: 0x0000000100008bed test`hash::sip::SipHasher.Hasher$LT$SipState$GT$::hash::h16497116303610205953 + 61\\r\\n        frame #1: 0x0000000105401810\\r\\n        frame #2: 0x0000000100008ba2 test`hash::RandomSipHasher.Hasher$LT$sip..SipState$GT$::hash::h10512232733763305808 + 66\\r\\n        frame #3: 0x0000000100008b0e test`collections::hash::table::make_hash::h9604696550138645665 + 62\\r\\n        frame #4: 0x0000000100008abe test`collections::hash::map::HashMap$LT$K$C$$u{20}V$C$$u{20}H$GT$::make_hash::h15786379260790357760 + 62\\r\\n        frame #5: 0x0000000100008963 test`collections::hash::map::HashMap$LT$K$C$$u{20}V$C$$u{20}H$GT$::insert::h6805880362953821147 + 131\\r\\n        frame #6: 0x0000000100006aea test`main::environment::Environment::define(self=0x000000000003ffc1, sym=String at 0x0000000105401a50, val=0x0000000105401c38) + 218 at environment.rs:92\\r\\n        frame #7: 0x0000000100018486 test`main::eval::evaluate_definition(heap=0x00000001054034e8, env=0x0000000105402d10, form=0x0000000105402ce0) + 2406 at eval.rs:195\\r\\n        frame #8: 0x0000000100013f1b test`main::eval::evaluate(heap=0x00000001054034e8, env=0x0000000105402d90, form=0x0000000105402f30) + 1355 at eval.rs:79\\r\\n        frame #9: 0x0000000100013909 test`main::eval::evaluate_in_global_env(heap=0x00000001054034e8, form=0x0000000105402f30) + 105 at eval.rs:38\\r\\n        frame #10: 0x000000010001cd7b test`main::eval::evaluate_file(heap=0x00000001054034e8, file_path=(data_ptr = \\\"./tests/test_eval_closures.scmsrc/heap.rsArenaPtr(, )Rooted(\\\", length = 30)) + 1035 at <std macros>:286\\r\\n        frame #11: 0x000000010002c071 test`main::eval::test_eval_closures + 129 at eval.rs:403\\r\\n        frame #12: 0x0000000100087154 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h5218640484166224076 + 52\\r\\n        frame #13: 0x0000000100091e32 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h13672304839710771993 + 162\\r\\n        frame #14: 0x0000000100088912 test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h15490786364890695979 + 1138\\r\\n        frame #15: 0x0000000100088db0 test`rt::unwind::try::try_fn::h10439459448986358359 + 160\\r\\n        frame #16: 0x000000010010a4f9 test`rust_try_inner + 9\\r\\n        frame #17: 0x000000010010a4e6 test`rust_try + 6\\r\\n        frame #18: 0x00000001000894ab test`thunk::F.Invoke$LT$A$C$$u{20}R$GT$::invoke::h13206639752781900349 + 1179\\r\\n        frame #19: 0x0000000100106eb4 test`sys::thread::thread_start::h2cb22211a4c7d938vFw + 164\\r\\n        frame #20: 0x00007fff875d3899 libsystem_pthread.dylib`_pthread_body + 138\\r\\n        frame #21: 0x00007fff875d372a libsystem_pthread.dylib`_pthread_start + 137\\r\\n        frame #22: 0x00007fff875d7fc9 libsystem_pthread.dylib`thread_start + 13\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:35Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489397845\",\"type\":\"PushEvent\",\"actor\":{\"id\":2284601,\"login\":\"citrix-openstack-build\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/citrix-openstack-build\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2284601?\"},\"repo\":{\"id\":13325270,\"name\":\"citrix-openstack-build/keystone\",\"url\":\"https://api.github.com/repos/citrix-openstack-build/keystone\"},\"payload\":{\"push_id\":536753115,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/ctx-nova-network-smoke-latest\",\"head\":\"503da84ac51a3f5f4d173fe5d1b19c98ed12ed7d\",\"before\":\"eb3556e9b9c4413030b230c6a6fd04a934328773\",\"commits\":[{\"sha\":\"637845df1ef857dfd5f23658ac77312f0212c7c7\",\"author\":{\"email\":\"38c8728ee30e7562e7872660a2fab1bda7c7a859@us.ibm.com\",\"name\":\"Brant Knudson\"},\"message\":\"Enhance FakeLdap to require base entry for subtree search\\n\\nThe FakeLdap implementation didn't raise ldap.NO_SUCH_OBJECT when the\\nbase entry didn't exist. A real LDAP server would raise if the base\\nentry didn't exist.\\n\\nChange-Id: If26d4cff5882ad13c9e00b8a894920f244238c49\\nCloses-Bug: #1368772\",\"distinct\":true,\"url\":\"https://api.github.com/repos/citrix-openstack-build/keystone/commits/637845df1ef857dfd5f23658ac77312f0212c7c7\"},{\"sha\":\"e06a463e961403d665608252a71efff4b7a415ab\",\"author\":{\"email\":\"1150688ef4531a9968266ddcbdddd37ed9b81ac8@huawei.com\",\"name\":\"wanghong\"},\"message\":\"improve the EP-FILTER catalog length check in test_v3.py\\n\\nUse matcher.HasLength to do EP-FILTER catalog length check in\\ntest_v3.py. There are two reason to do this:\\n1. Currrently, we check more than but not equal, this may cover bug\\n2. When test fails, it just give an useless message:\\n   AssertionError: False is not true\\n\\nChange-Id: I20d521690e79957031bdf4bf5b240f8a4790553c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/citrix-openstack-build/keystone/commits/e06a463e961403d665608252a71efff4b7a415ab\"},{\"sha\":\"ded5608f38c6629752228a411b620e8c6614410d\",\"author\":{\"email\":\"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org\",\"name\":\"Jenkins\"},\"message\":\"Merge \\\"Enhance FakeLdap to require base entry for subtree search\\\"\",\"distinct\":true,\"url\":\"https://api.github.com/repos/citrix-openstack-build/keystone/commits/ded5608f38c6629752228a411b620e8c6614410d\"},{\"sha\":\"503da84ac51a3f5f4d173fe5d1b19c98ed12ed7d\",\"author\":{\"email\":\"d95b56ce41a2e1ac4cecdd398defd7414407cc08@review.openstack.org\",\"name\":\"Jenkins\"},\"message\":\"Merge \\\"improve the EP-FILTER catalog length check in test_v3.py\\\"\",\"distinct\":true,\"url\":\"https://api.github.com/repos/citrix-openstack-build/keystone/commits/503da84ac51a3f5f4d173fe5d1b19c98ed12ed7d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:36Z\"}\n{\"id\":\"2489397847\",\"type\":\"PushEvent\",\"actor\":{\"id\":4356609,\"login\":\"dcrousso\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcrousso\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4356609?\"},\"repo\":{\"id\":25148806,\"name\":\"dcrousso/ProjectEuler\",\"url\":\"https://api.github.com/repos/dcrousso/ProjectEuler\"},\"payload\":{\"push_id\":536753116,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e8da582cd7c82a56b23ac566b7bb8b8edf5ce415\",\"before\":\"b2b6712812e289ed0a75eb185113dd69a9e43dd9\",\"commits\":[{\"sha\":\"e8da582cd7c82a56b23ac566b7bb8b8edf5ce415\",\"author\":{\"email\":\"9ebcc5dc367a76bd13fea77254fcbce737f76b12@gmail.com\",\"name\":\"Devin Rousso\"},\"message\":\"Completed 123\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dcrousso/ProjectEuler/commits/e8da582cd7c82a56b23ac566b7bb8b8edf5ce415\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:36Z\"}\n{\"id\":\"2489397852\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397285\",\"id\":22397285,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":96,\"original_position\":96,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"why so continuation?\",\"created_at\":\"2015-01-01T01:04:37Z\",\"updated_at\":\"2015-01-01T01:04:37Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397285\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397285\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397285\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:04:37Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:04:37Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489397854\",\"type\":\"PushEvent\",\"actor\":{\"id\":163915,\"login\":\"fponticelli\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fponticelli\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/163915?\"},\"repo\":{\"id\":28647473,\"name\":\"fponticelli/thx.benchmark\",\"url\":\"https://api.github.com/repos/fponticelli/thx.benchmark\"},\"payload\":{\"push_id\":536753119,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f6a2f0275338378ee5e3c4674e5cef6a9029f6b1\",\"before\":\"203d2bfd757e0619e7b8e7732a16ea15bed09c2e\",\"commits\":[{\"sha\":\"f6a2f0275338378ee5e3c4674e5cef6a9029f6b1\",\"author\":{\"email\":\"05a4419b3ba135c9a2552a2fac2f13cfe3d22f12@gmail.com\",\"name\":\"Franco Ponticelli\"},\"message\":\"changed decimals in mills\",\"distinct\":true,\"url\":\"https://api.github.com/repos/fponticelli/thx.benchmark/commits/f6a2f0275338378ee5e3c4674e5cef6a9029f6b1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:37Z\"}\n{\"id\":\"2489397855\",\"type\":\"PushEvent\",\"actor\":{\"id\":69068,\"login\":\"someara\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/someara\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/69068?\"},\"repo\":{\"id\":28658028,\"name\":\"someara/slacker-packer\",\"url\":\"https://api.github.com/repos/someara/slacker-packer\"},\"payload\":{\"push_id\":536753120,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"a167b8d1405621a26333fda5325d389eeccc05ff\",\"before\":\"6c580a76c4800bd2707e53fef9c722c78ad0d742\",\"commits\":[{\"sha\":\"09f12c2d299252470229258e6f4a98f439bd025e\",\"author\":{\"email\":\"a08f4e77dc67cec36ca0d3194df5d081f3357efd@gmail.com\",\"name\":\"hypomonk\"},\"message\":\"fixed shutdown and trimmed bootup time\",\"distinct\":true,\"url\":\"https://api.github.com/repos/someara/slacker-packer/commits/09f12c2d299252470229258e6f4a98f439bd025e\"},{\"sha\":\"a167b8d1405621a26333fda5325d389eeccc05ff\",\"author\":{\"email\":\"d7e19930cc1f42c2d0781f4d9e6f1fe5891bf9cf@chef.io\",\"name\":\"Sean OMeara\"},\"message\":\"adding .gitignore and --no-check-certificates\",\"distinct\":true,\"url\":\"https://api.github.com/repos/someara/slacker-packer/commits/a167b8d1405621a26333fda5325d389eeccc05ff\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:37Z\"}\n{\"id\":\"2489397856\",\"type\":\"PushEvent\",\"actor\":{\"id\":166301,\"login\":\"bcomnes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bcomnes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/166301?\"},\"repo\":{\"id\":6861308,\"name\":\"bcomnes/bcomnes.github.io\",\"url\":\"https://api.github.com/repos/bcomnes/bcomnes.github.io\"},\"payload\":{\"push_id\":536753122,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"821968b470015bf7a6d5f86aeea8d739dd1aadbe\",\"before\":\"6443459c46498095409d23205ceadb9e7b575182\",\"commits\":[{\"sha\":\"821968b470015bf7a6d5f86aeea8d739dd1aadbe\",\"author\":{\"email\":\"6df4fba95631fe4f4c4337307cda4e0fc4c27d16@gmail.com\",\"name\":\"Bret\"},\"message\":\"gitpub posted a new post\",\"distinct\":true,\"url\":\"https://api.github.com/repos/bcomnes/bcomnes.github.io/commits/821968b470015bf7a6d5f86aeea8d739dd1aadbe\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:37Z\"}\n{\"id\":\"2489397861\",\"type\":\"PushEvent\",\"actor\":{\"id\":3720783,\"login\":\"designerwebhosting\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/designerwebhosting\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3720783?\"},\"repo\":{\"id\":20527117,\"name\":\"designerwebhosting/christopherbyrne.github.io\",\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io\"},\"payload\":{\"push_id\":536753125,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"52997f806edc1645856950072f9919dea680b5a7\",\"before\":\"19f1368a9acc50e9b1cc2b936c165c9704040889\",\"commits\":[{\"sha\":\"52997f806edc1645856950072f9919dea680b5a7\",\"author\":{\"email\":\"4bb0acc6ff8c0b6c31e50417877e6e3b3f1c65f0@googlemail.com\",\"name\":\"Peter Noblee\"},\"message\":\"update 'date'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io/commits/52997f806edc1645856950072f9919dea680b5a7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:38Z\"}\n{\"id\":\"2489397862\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":568036,\"login\":\"Tyilo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tyilo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/568036?\"},\"repo\":{\"id\":21377779,\"name\":\"isaacg1/pyth\",\"url\":\"https://api.github.com/repos/isaacg1/pyth\"},\"payload\":{\"action\":\"opened\",\"number\":11,\"pull_request\":{\"url\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11\",\"id\":26739449,\"html_url\":\"https://github.com/isaacg1/pyth/pull/11\",\"diff_url\":\"https://github.com/isaacg1/pyth/pull/11.diff\",\"patch_url\":\"https://github.com/isaacg1/pyth/pull/11.patch\",\"issue_url\":\"https://api.github.com/repos/isaacg1/pyth/issues/11\",\"number\":11,\"state\":\"open\",\"locked\":false,\"title\":\"Move server functionality into different functions\",\"user\":{\"login\":\"Tyilo\",\"id\":568036,\"avatar_url\":\"https://avatars.githubusercontent.com/u/568036?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tyilo\",\"html_url\":\"https://github.com/Tyilo\",\"followers_url\":\"https://api.github.com/users/Tyilo/followers\",\"following_url\":\"https://api.github.com/users/Tyilo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Tyilo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Tyilo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Tyilo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Tyilo/orgs\",\"repos_url\":\"https://api.github.com/users/Tyilo/repos\",\"events_url\":\"https://api.github.com/users/Tyilo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Tyilo/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"As suggested by @Maltysen in #10 \",\"created_at\":\"2015-01-01T01:04:38Z\",\"updated_at\":\"2015-01-01T01:04:38Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11/commits\",\"review_comments_url\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11/comments\",\"review_comment_url\":\"https://api.github.com/repos/isaacg1/pyth/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/isaacg1/pyth/issues/11/comments\",\"statuses_url\":\"https://api.github.com/repos/isaacg1/pyth/statuses/8e2dd59a2966cb41ec28faf879d33fa5b6d2ad75\",\"head\":{\"label\":\"Tyilo:patch-2\",\"ref\":\"patch-2\",\"sha\":\"8e2dd59a2966cb41ec28faf879d33fa5b6d2ad75\",\"user\":{\"login\":\"Tyilo\",\"id\":568036,\"avatar_url\":\"https://avatars.githubusercontent.com/u/568036?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tyilo\",\"html_url\":\"https://github.com/Tyilo\",\"followers_url\":\"https://api.github.com/users/Tyilo/followers\",\"following_url\":\"https://api.github.com/users/Tyilo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Tyilo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Tyilo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Tyilo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Tyilo/orgs\",\"repos_url\":\"https://api.github.com/users/Tyilo/repos\",\"events_url\":\"https://api.github.com/users/Tyilo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Tyilo/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28676548,\"name\":\"pyth\",\"full_name\":\"Tyilo/pyth\",\"owner\":{\"login\":\"Tyilo\",\"id\":568036,\"avatar_url\":\"https://avatars.githubusercontent.com/u/568036?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tyilo\",\"html_url\":\"https://github.com/Tyilo\",\"followers_url\":\"https://api.github.com/users/Tyilo/followers\",\"following_url\":\"https://api.github.com/users/Tyilo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Tyilo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Tyilo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Tyilo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Tyilo/orgs\",\"repos_url\":\"https://api.github.com/users/Tyilo/repos\",\"events_url\":\"https://api.github.com/users/Tyilo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Tyilo/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Tyilo/pyth\",\"description\":\"Pyth, an extremely concise language. Try it here:\",\"fork\":true,\"url\":\"https://api.github.com/repos/Tyilo/pyth\",\"forks_url\":\"https://api.github.com/repos/Tyilo/pyth/forks\",\"keys_url\":\"https://api.github.com/repos/Tyilo/pyth/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Tyilo/pyth/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Tyilo/pyth/teams\",\"hooks_url\":\"https://api.github.com/repos/Tyilo/pyth/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Tyilo/pyth/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Tyilo/pyth/events\",\"assignees_url\":\"https://api.github.com/repos/Tyilo/pyth/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Tyilo/pyth/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Tyilo/pyth/tags\",\"blobs_url\":\"https://api.github.com/repos/Tyilo/pyth/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Tyilo/pyth/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Tyilo/pyth/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Tyilo/pyth/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Tyilo/pyth/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Tyilo/pyth/languages\",\"stargazers_url\":\"https://api.github.com/repos/Tyilo/pyth/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Tyilo/pyth/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Tyilo/pyth/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Tyilo/pyth/subscription\",\"commits_url\":\"https://api.github.com/repos/Tyilo/pyth/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Tyilo/pyth/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Tyilo/pyth/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Tyilo/pyth/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Tyilo/pyth/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Tyilo/pyth/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Tyilo/pyth/merges\",\"archive_url\":\"https://api.github.com/repos/Tyilo/pyth/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Tyilo/pyth/downloads\",\"issues_url\":\"https://api.github.com/repos/Tyilo/pyth/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Tyilo/pyth/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Tyilo/pyth/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Tyilo/pyth/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Tyilo/pyth/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Tyilo/pyth/releases{/id}\",\"created_at\":\"2014-12-31T22:38:59Z\",\"updated_at\":\"2014-12-31T22:39:00Z\",\"pushed_at\":\"2015-01-01T01:03:03Z\",\"git_url\":\"git://github.com/Tyilo/pyth.git\",\"ssh_url\":\"git@github.com:Tyilo/pyth.git\",\"clone_url\":\"https://github.com/Tyilo/pyth.git\",\"svn_url\":\"https://github.com/Tyilo/pyth\",\"homepage\":\"http://isaacg.scripts.mit.edu/pyth/index.py\",\"size\":540,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Python\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"isaacg1:master\",\"ref\":\"master\",\"sha\":\"8dcc542711ab74a5454b34ac2d10fda566e4d465\",\"user\":{\"login\":\"isaacg1\",\"id\":8034059,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8034059?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/isaacg1\",\"html_url\":\"https://github.com/isaacg1\",\"followers_url\":\"https://api.github.com/users/isaacg1/followers\",\"following_url\":\"https://api.github.com/users/isaacg1/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/isaacg1/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/isaacg1/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/isaacg1/subscriptions\",\"organizations_url\":\"https://api.github.com/users/isaacg1/orgs\",\"repos_url\":\"https://api.github.com/users/isaacg1/repos\",\"events_url\":\"https://api.github.com/users/isaacg1/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/isaacg1/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":21377779,\"name\":\"pyth\",\"full_name\":\"isaacg1/pyth\",\"owner\":{\"login\":\"isaacg1\",\"id\":8034059,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8034059?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/isaacg1\",\"html_url\":\"https://github.com/isaacg1\",\"followers_url\":\"https://api.github.com/users/isaacg1/followers\",\"following_url\":\"https://api.github.com/users/isaacg1/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/isaacg1/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/isaacg1/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/isaacg1/subscriptions\",\"organizations_url\":\"https://api.github.com/users/isaacg1/orgs\",\"repos_url\":\"https://api.github.com/users/isaacg1/repos\",\"events_url\":\"https://api.github.com/users/isaacg1/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/isaacg1/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/isaacg1/pyth\",\"description\":\"Pyth, an extremely concise language. Try it here:\",\"fork\":false,\"url\":\"https://api.github.com/repos/isaacg1/pyth\",\"forks_url\":\"https://api.github.com/repos/isaacg1/pyth/forks\",\"keys_url\":\"https://api.github.com/repos/isaacg1/pyth/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/isaacg1/pyth/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/isaacg1/pyth/teams\",\"hooks_url\":\"https://api.github.com/repos/isaacg1/pyth/hooks\",\"issue_events_url\":\"https://api.github.com/repos/isaacg1/pyth/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/isaacg1/pyth/events\",\"assignees_url\":\"https://api.github.com/repos/isaacg1/pyth/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/isaacg1/pyth/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/isaacg1/pyth/tags\",\"blobs_url\":\"https://api.github.com/repos/isaacg1/pyth/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/isaacg1/pyth/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/isaacg1/pyth/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/isaacg1/pyth/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/isaacg1/pyth/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/isaacg1/pyth/languages\",\"stargazers_url\":\"https://api.github.com/repos/isaacg1/pyth/stargazers\",\"contributors_url\":\"https://api.github.com/repos/isaacg1/pyth/contributors\",\"subscribers_url\":\"https://api.github.com/repos/isaacg1/pyth/subscribers\",\"subscription_url\":\"https://api.github.com/repos/isaacg1/pyth/subscription\",\"commits_url\":\"https://api.github.com/repos/isaacg1/pyth/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/isaacg1/pyth/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/isaacg1/pyth/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/isaacg1/pyth/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/isaacg1/pyth/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/isaacg1/pyth/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/isaacg1/pyth/merges\",\"archive_url\":\"https://api.github.com/repos/isaacg1/pyth/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/isaacg1/pyth/downloads\",\"issues_url\":\"https://api.github.com/repos/isaacg1/pyth/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/isaacg1/pyth/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/isaacg1/pyth/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/isaacg1/pyth/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/isaacg1/pyth/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/isaacg1/pyth/releases{/id}\",\"created_at\":\"2014-07-01T06:44:21Z\",\"updated_at\":\"2015-01-01T00:39:57Z\",\"pushed_at\":\"2015-01-01T00:39:56Z\",\"git_url\":\"git://github.com/isaacg1/pyth.git\",\"ssh_url\":\"git@github.com:isaacg1/pyth.git\",\"clone_url\":\"https://github.com/isaacg1/pyth.git\",\"svn_url\":\"https://github.com/isaacg1/pyth\",\"homepage\":\"https://pyth.herokuapp.com/\",\"size\":540,\"stargazers_count\":22,\"watchers_count\":22,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":2,\"mirror_url\":null,\"open_issues_count\":1,\"forks\":2,\"open_issues\":1,\"watchers\":22,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11\"},\"html\":{\"href\":\"https://github.com/isaacg1/pyth/pull/11\"},\"issue\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/issues/11\"},\"comments\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/issues/11/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/pulls/11/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/isaacg1/pyth/statuses/8e2dd59a2966cb41ec28faf879d33fa5b6d2ad75\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":36,\"deletions\":36,\"changed_files\":3}},\"public\":true,\"created_at\":\"2015-01-01T01:04:38Z\"}\n{\"id\":\"2489397867\",\"type\":\"CreateEvent\",\"actor\":{\"id\":3267533,\"login\":\"brentpicasso\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/brentpicasso\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3267533?\"},\"repo\":{\"id\":28678261,\"name\":\"autosportlabs/TempX\",\"url\":\"https://api.github.com/repos/autosportlabs/TempX\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"TempX is a temperature sensor breakout board for the TMP36 active temperature sensor\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:39Z\",\"org\":{\"id\":1274125,\"login\":\"autosportlabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/autosportlabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1274125?\"}}\n{\"id\":\"2489397873\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1671640,\"login\":\"alex6lc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alex6lc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1671640?\"},\"repo\":{\"id\":27458466,\"name\":\"gpbl/material-ui-sass\",\"url\":\"https://api.github.com/repos/gpbl/material-ui-sass\"},\"payload\":{\"forkee\":{\"id\":28678262,\"name\":\"material-ui-sass\",\"full_name\":\"alex6lc/material-ui-sass\",\"owner\":{\"login\":\"alex6lc\",\"id\":1671640,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1671640?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alex6lc\",\"html_url\":\"https://github.com/alex6lc\",\"followers_url\":\"https://api.github.com/users/alex6lc/followers\",\"following_url\":\"https://api.github.com/users/alex6lc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/alex6lc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/alex6lc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/alex6lc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/alex6lc/orgs\",\"repos_url\":\"https://api.github.com/users/alex6lc/repos\",\"events_url\":\"https://api.github.com/users/alex6lc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/alex6lc/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/alex6lc/material-ui-sass\",\"description\":\"The Sass counterpart of the material-ui framework for React\",\"fork\":true,\"url\":\"https://api.github.com/repos/alex6lc/material-ui-sass\",\"forks_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/forks\",\"keys_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/teams\",\"hooks_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/hooks\",\"issue_events_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/events\",\"assignees_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/tags\",\"blobs_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/languages\",\"stargazers_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/stargazers\",\"contributors_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/contributors\",\"subscribers_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/subscribers\",\"subscription_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/subscription\",\"commits_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/merges\",\"archive_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/downloads\",\"issues_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/releases{/id}\",\"created_at\":\"2015-01-01T01:04:40Z\",\"updated_at\":\"2014-12-30T20:38:38Z\",\"pushed_at\":\"2014-12-27T16:36:27Z\",\"git_url\":\"git://github.com/alex6lc/material-ui-sass.git\",\"ssh_url\":\"git@github.com:alex6lc/material-ui-sass.git\",\"clone_url\":\"https://github.com/alex6lc/material-ui-sass.git\",\"svn_url\":\"https://github.com/alex6lc/material-ui-sass\",\"homepage\":\"\",\"size\":2075,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:04:40Z\"}\n{\"id\":\"2489397875\",\"type\":\"PushEvent\",\"actor\":{\"id\":1422772,\"login\":\"lighttroupe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lighttroupe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1422772?\"},\"repo\":{\"id\":19626154,\"name\":\"lighttroupe/luz-next\",\"url\":\"https://api.github.com/repos/lighttroupe/luz-next\"},\"payload\":{\"push_id\":536753128,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3c99b61327454468c31f6c0db4045e73dc1bca01\",\"before\":\"f1651f527ad2fa7edd1a3fbcd6a3ae16e38dec66\",\"commits\":[{\"sha\":\"3c99b61327454468c31f6c0db4045e73dc1bca01\",\"author\":{\"email\":\"57a33a5496950fec8433e4dd83347673459dcdfc@openanswers.org\",\"name\":\"Ian McIntosh\"},\"message\":\"- add placeholder message bar background\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lighttroupe/luz-next/commits/3c99b61327454468c31f6c0db4045e73dc1bca01\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:40Z\"}\n{\"id\":\"2489397876\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":1813305,\"login\":\"Bart39\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?\"},\"repo\":{\"id\":9683876,\"name\":\"nZEDb/nZEDb\",\"url\":\"https://api.github.com/repos/nZEDb/nZEDb\"},\"payload\":{\"action\":\"closed\",\"number\":1573,\"pull_request\":{\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573\",\"id\":26736047,\"html_url\":\"https://github.com/nZEDb/nZEDb/pull/1573\",\"diff_url\":\"https://github.com/nZEDb/nZEDb/pull/1573.diff\",\"patch_url\":\"https://github.com/nZEDb/nZEDb/pull/1573.patch\",\"issue_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1573\",\"number\":1573,\"state\":\"closed\",\"locked\":false,\"title\":\"Changed trakt.tv imdb search to new format\",\"user\":{\"login\":\"DariusIII\",\"id\":3399658,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3399658?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DariusIII\",\"html_url\":\"https://github.com/DariusIII\",\"followers_url\":\"https://api.github.com/users/DariusIII/followers\",\"following_url\":\"https://api.github.com/users/DariusIII/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DariusIII/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DariusIII/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DariusIII/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DariusIII/orgs\",\"repos_url\":\"https://api.github.com/users/DariusIII/repos\",\"events_url\":\"https://api.github.com/users/DariusIII/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DariusIII/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2014-12-31T21:28:06Z\",\"updated_at\":\"2015-01-01T01:04:40Z\",\"closed_at\":\"2015-01-01T01:04:40Z\",\"merged_at\":\"2015-01-01T01:04:40Z\",\"merge_commit_sha\":\"5cb2802b75980591233ac3f40676585c9a14af25\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573/commits\",\"review_comments_url\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573/comments\",\"review_comment_url\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1573/comments\",\"statuses_url\":\"https://api.github.com/repos/nZEDb/nZEDb/statuses/b309b85c959a9fae595119a4aa6b02a129d368f9\",\"head\":{\"label\":\"DariusIII:dev\",\"ref\":\"dev\",\"sha\":\"b309b85c959a9fae595119a4aa6b02a129d368f9\",\"user\":{\"login\":\"DariusIII\",\"id\":3399658,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3399658?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DariusIII\",\"html_url\":\"https://github.com/DariusIII\",\"followers_url\":\"https://api.github.com/users/DariusIII/followers\",\"following_url\":\"https://api.github.com/users/DariusIII/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DariusIII/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DariusIII/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DariusIII/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DariusIII/orgs\",\"repos_url\":\"https://api.github.com/users/DariusIII/repos\",\"events_url\":\"https://api.github.com/users/DariusIII/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DariusIII/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":21434059,\"name\":\"nZEDb\",\"full_name\":\"DariusIII/nZEDb\",\"owner\":{\"login\":\"DariusIII\",\"id\":3399658,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3399658?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DariusIII\",\"html_url\":\"https://github.com/DariusIII\",\"followers_url\":\"https://api.github.com/users/DariusIII/followers\",\"following_url\":\"https://api.github.com/users/DariusIII/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DariusIII/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DariusIII/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DariusIII/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DariusIII/orgs\",\"repos_url\":\"https://api.github.com/users/DariusIII/repos\",\"events_url\":\"https://api.github.com/users/DariusIII/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DariusIII/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/DariusIII/nZEDb\",\"description\":\"nZEDb - a fork of nnplus(2011) | NNTP / Usenet / Newsgroup indexer.\",\"fork\":true,\"url\":\"https://api.github.com/repos/DariusIII/nZEDb\",\"forks_url\":\"https://api.github.com/repos/DariusIII/nZEDb/forks\",\"keys_url\":\"https://api.github.com/repos/DariusIII/nZEDb/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/DariusIII/nZEDb/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/DariusIII/nZEDb/teams\",\"hooks_url\":\"https://api.github.com/repos/DariusIII/nZEDb/hooks\",\"issue_events_url\":\"https://api.github.com/repos/DariusIII/nZEDb/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/DariusIII/nZEDb/events\",\"assignees_url\":\"https://api.github.com/repos/DariusIII/nZEDb/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/DariusIII/nZEDb/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/DariusIII/nZEDb/tags\",\"blobs_url\":\"https://api.github.com/repos/DariusIII/nZEDb/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/DariusIII/nZEDb/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/DariusIII/nZEDb/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/DariusIII/nZEDb/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/DariusIII/nZEDb/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/DariusIII/nZEDb/languages\",\"stargazers_url\":\"https://api.github.com/repos/DariusIII/nZEDb/stargazers\",\"contributors_url\":\"https://api.github.com/repos/DariusIII/nZEDb/contributors\",\"subscribers_url\":\"https://api.github.com/repos/DariusIII/nZEDb/subscribers\",\"subscription_url\":\"https://api.github.com/repos/DariusIII/nZEDb/subscription\",\"commits_url\":\"https://api.github.com/repos/DariusIII/nZEDb/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/DariusIII/nZEDb/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/DariusIII/nZEDb/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/DariusIII/nZEDb/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/DariusIII/nZEDb/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/DariusIII/nZEDb/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/DariusIII/nZEDb/merges\",\"archive_url\":\"https://api.github.com/repos/DariusIII/nZEDb/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/DariusIII/nZEDb/downloads\",\"issues_url\":\"https://api.github.com/repos/DariusIII/nZEDb/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/DariusIII/nZEDb/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/DariusIII/nZEDb/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/DariusIII/nZEDb/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/DariusIII/nZEDb/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/DariusIII/nZEDb/releases{/id}\",\"created_at\":\"2014-07-02T17:46:47Z\",\"updated_at\":\"2014-12-30T10:21:52Z\",\"pushed_at\":\"2014-12-31T21:26:54Z\",\"git_url\":\"git://github.com/DariusIII/nZEDb.git\",\"ssh_url\":\"git@github.com:DariusIII/nZEDb.git\",\"clone_url\":\"https://github.com/DariusIII/nZEDb.git\",\"svn_url\":\"https://github.com/DariusIII/nZEDb\",\"homepage\":\"http://forums.nzedb.com\",\"size\":53328,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"PHP\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"nZEDb:dev\",\"ref\":\"dev\",\"sha\":\"1ec7a576cfae11fdcdbe929cf9053e73dda2c43d\",\"user\":{\"login\":\"nZEDb\",\"id\":4260270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nZEDb\",\"html_url\":\"https://github.com/nZEDb\",\"followers_url\":\"https://api.github.com/users/nZEDb/followers\",\"following_url\":\"https://api.github.com/users/nZEDb/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nZEDb/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nZEDb/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nZEDb/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nZEDb/orgs\",\"repos_url\":\"https://api.github.com/users/nZEDb/repos\",\"events_url\":\"https://api.github.com/users/nZEDb/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nZEDb/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":9683876,\"name\":\"nZEDb\",\"full_name\":\"nZEDb/nZEDb\",\"owner\":{\"login\":\"nZEDb\",\"id\":4260270,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nZEDb\",\"html_url\":\"https://github.com/nZEDb\",\"followers_url\":\"https://api.github.com/users/nZEDb/followers\",\"following_url\":\"https://api.github.com/users/nZEDb/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nZEDb/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nZEDb/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nZEDb/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nZEDb/orgs\",\"repos_url\":\"https://api.github.com/users/nZEDb/repos\",\"events_url\":\"https://api.github.com/users/nZEDb/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nZEDb/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/nZEDb/nZEDb\",\"description\":\"nZEDb - a fork of nnplus(2011) | NNTP / Usenet / Newsgroup indexer.\",\"fork\":false,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb\",\"forks_url\":\"https://api.github.com/repos/nZEDb/nZEDb/forks\",\"keys_url\":\"https://api.github.com/repos/nZEDb/nZEDb/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/nZEDb/nZEDb/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/nZEDb/nZEDb/teams\",\"hooks_url\":\"https://api.github.com/repos/nZEDb/nZEDb/hooks\",\"issue_events_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/nZEDb/nZEDb/events\",\"assignees_url\":\"https://api.github.com/repos/nZEDb/nZEDb/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/nZEDb/nZEDb/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/nZEDb/nZEDb/tags\",\"blobs_url\":\"https://api.github.com/repos/nZEDb/nZEDb/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/nZEDb/nZEDb/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/nZEDb/nZEDb/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/nZEDb/nZEDb/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/nZEDb/nZEDb/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/nZEDb/nZEDb/languages\",\"stargazers_url\":\"https://api.github.com/repos/nZEDb/nZEDb/stargazers\",\"contributors_url\":\"https://api.github.com/repos/nZEDb/nZEDb/contributors\",\"subscribers_url\":\"https://api.github.com/repos/nZEDb/nZEDb/subscribers\",\"subscription_url\":\"https://api.github.com/repos/nZEDb/nZEDb/subscription\",\"commits_url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/nZEDb/nZEDb/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/nZEDb/nZEDb/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/nZEDb/nZEDb/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/nZEDb/nZEDb/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/nZEDb/nZEDb/merges\",\"archive_url\":\"https://api.github.com/repos/nZEDb/nZEDb/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/nZEDb/nZEDb/downloads\",\"issues_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/nZEDb/nZEDb/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/nZEDb/nZEDb/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/nZEDb/nZEDb/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/nZEDb/nZEDb/releases{/id}\",\"created_at\":\"2013-04-25T23:45:28Z\",\"updated_at\":\"2014-12-31T06:19:27Z\",\"pushed_at\":\"2015-01-01T01:04:40Z\",\"git_url\":\"git://github.com/nZEDb/nZEDb.git\",\"ssh_url\":\"git@github.com:nZEDb/nZEDb.git\",\"clone_url\":\"https://github.com/nZEDb/nZEDb.git\",\"svn_url\":\"https://github.com/nZEDb/nZEDb\",\"homepage\":\"http://forums.nzedb.com\",\"size\":77901,\"stargazers_count\":189,\"watchers_count\":189,\"language\":\"PHP\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":102,\"mirror_url\":null,\"open_issues_count\":30,\"forks\":102,\"open_issues\":30,\"watchers\":189,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573\"},\"html\":{\"href\":\"https://github.com/nZEDb/nZEDb/pull/1573\"},\"issue\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1573\"},\"comments\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1573/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/pulls/1573/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/nZEDb/nZEDb/statuses/b309b85c959a9fae595119a4aa6b02a129d368f9\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"Bart39\",\"id\":1813305,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"html_url\":\"https://github.com/Bart39\",\"followers_url\":\"https://api.github.com/users/Bart39/followers\",\"following_url\":\"https://api.github.com/users/Bart39/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bart39/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bart39/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bart39/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bart39/orgs\",\"repos_url\":\"https://api.github.com/users/Bart39/repos\",\"events_url\":\"https://api.github.com/users/Bart39/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bart39/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":40,\"additions\":7,\"deletions\":7,\"changed_files\":7}},\"public\":true,\"created_at\":\"2015-01-01T01:04:40Z\",\"org\":{\"id\":4260270,\"login\":\"nZEDb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nZEDb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?\"}}\n{\"id\":\"2489397878\",\"type\":\"PushEvent\",\"actor\":{\"id\":42681,\"login\":\"walnotes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/walnotes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/42681?\"},\"repo\":{\"id\":22087319,\"name\":\"walnotes/ngo360\",\"url\":\"https://api.github.com/repos/walnotes/ngo360\"},\"payload\":{\"push_id\":536753130,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0e762d5e66602e8fc7f5d3639f65f3e0bad26134\",\"before\":\"0a7af8c18ab72bf805d9c94d4483512765220a95\",\"commits\":[{\"sha\":\"0e762d5e66602e8fc7f5d3639f65f3e0bad26134\",\"author\":{\"email\":\"5d8024bfa9e39eb0b6834fe5420b9e92861b675c@yahoo.com\",\"name\":\"Walter\"},\"message\":\"Added audio and video to controller, model and views\",\"distinct\":true,\"url\":\"https://api.github.com/repos/walnotes/ngo360/commits/0e762d5e66602e8fc7f5d3639f65f3e0bad26134\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:41Z\"}\n{\"id\":\"2489397883\",\"type\":\"PushEvent\",\"actor\":{\"id\":1813305,\"login\":\"Bart39\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?\"},\"repo\":{\"id\":9683876,\"name\":\"nZEDb/nZEDb\",\"url\":\"https://api.github.com/repos/nZEDb/nZEDb\"},\"payload\":{\"push_id\":536753131,\"size\":42,\"distinct_size\":42,\"ref\":\"refs/heads/dev\",\"head\":\"cfda73aed869f40b3e19a4345dda9c4d65f27937\",\"before\":\"1ec7a576cfae11fdcdbe929cf9053e73dda2c43d\",\"commits\":[{\"sha\":\"7d042c6422274c6ffb3e3869e107c053fd55553c\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #176 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/7d042c6422274c6ffb3e3869e107c053fd55553c\"},{\"sha\":\"fc1f764e87b4cd471e946aa782cb1c8f5e141fc4\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #177 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/fc1f764e87b4cd471e946aa782cb1c8f5e141fc4\"},{\"sha\":\"47d118f8112bcc0152430080dabfefcaf7913b80\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #179 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/47d118f8112bcc0152430080dabfefcaf7913b80\"},{\"sha\":\"54abdf31cd2795f28c3ae7eb583c8bff89299110\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #180 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/54abdf31cd2795f28c3ae7eb583c8bff89299110\"},{\"sha\":\"b1a90beb996cc6151d4d33636e75d4858b32ddde\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #181 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/b1a90beb996cc6151d4d33636e75d4858b32ddde\"},{\"sha\":\"080ff5e1da53af3f3bf1cdbf04326d5bf90479b4\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #182 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/080ff5e1da53af3f3bf1cdbf04326d5bf90479b4\"},{\"sha\":\"034f6b6b0d1731ccf5c36c1f382c2bed5c7b48d4\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #183 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/034f6b6b0d1731ccf5c36c1f382c2bed5c7b48d4\"},{\"sha\":\"c50d536090b7fdcf766ac1efead18ec9cdc4370a\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #185 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/c50d536090b7fdcf766ac1efead18ec9cdc4370a\"},{\"sha\":\"c255c8dd11150ecdfaedf44a9880b268ee4661cc\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #186 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/c255c8dd11150ecdfaedf44a9880b268ee4661cc\"},{\"sha\":\"c19f38122706de1739b40a84b1b946a40213b47c\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #187 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/c19f38122706de1739b40a84b1b946a40213b47c\"},{\"sha\":\"a472784834e7720c420c3d0e0b3f491b3b99c1f4\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #189 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/a472784834e7720c420c3d0e0b3f491b3b99c1f4\"},{\"sha\":\"e5b0feebd61f02bf79b987ab95d79362862484fe\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #190 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/e5b0feebd61f02bf79b987ab95d79362862484fe\"},{\"sha\":\"8798c62d19bd328232c4f574272344149781d015\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #192 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/8798c62d19bd328232c4f574272344149781d015\"},{\"sha\":\"52089bc28eab88d4769ef823c46e913274b0ef21\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #193 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/52089bc28eab88d4769ef823c46e913274b0ef21\"},{\"sha\":\"e594fe80e431f51e82349a7d15f5ae661597d770\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #194 from nZEDb/dev\\n\\nFix: Issue #1477 - MySQL Full-Text Search broken after implementation of...\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/e594fe80e431f51e82349a7d15f5ae661597d770\"},{\"sha\":\"6a0e8832337f45491503157c46ee33f5a6032306\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #195 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/6a0e8832337f45491503157c46ee33f5a6032306\"},{\"sha\":\"a064bb19dd13b1dab7a3f34bc7f3c870f514b387\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #196 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/a064bb19dd13b1dab7a3f34bc7f3c870f514b387\"},{\"sha\":\"559306637c70572e4c80c4b44495805f99b05f5b\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #198 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/559306637c70572e4c80c4b44495805f99b05f5b\"},{\"sha\":\"3ea408ad2181277a229cc5653078124b9be544d9\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #199 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/3ea408ad2181277a229cc5653078124b9be544d9\"},{\"sha\":\"82c22a2f1b9f14e21835036a947a7e20ed983702\",\"author\":{\"email\":\"d905a83fce9ca7a3ba493d139e06a7bc091e7008@gmail.com\",\"name\":\"DariusIII\"},\"message\":\"Merge pull request #200 from nZEDb/dev\\n\\nDev\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/commits/82c22a2f1b9f14e21835036a947a7e20ed983702\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:42Z\",\"org\":{\"id\":4260270,\"login\":\"nZEDb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nZEDb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?\"}}\n{\"id\":\"2489397885\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":6964047,\"login\":\"TTMTT\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?\"},\"repo\":{\"id\":26192763,\"name\":\"HarisDevs/iCl0udinSupport\",\"url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/6\",\"labels_url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/6/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/6/comments\",\"events_url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/6/events\",\"html_url\":\"https://github.com/HarisDevs/iCl0udinSupport/issues/6\",\"id\":48839033,\"number\":6,\"title\":\"Discuss9 \",\"user\":{\"login\":\"HarisDevs\",\"id\":8019549,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8019549?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/HarisDevs\",\"html_url\":\"https://github.com/HarisDevs\",\"followers_url\":\"https://api.github.com/users/HarisDevs/followers\",\"following_url\":\"https://api.github.com/users/HarisDevs/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/HarisDevs/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/HarisDevs/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/HarisDevs/subscriptions\",\"organizations_url\":\"https://api.github.com/users/HarisDevs/orgs\",\"repos_url\":\"https://api.github.com/users/HarisDevs/repos\",\"events_url\":\"https://api.github.com/users/HarisDevs/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/HarisDevs/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":424,\"created_at\":\"2014-11-14T21:00:03Z\",\"updated_at\":\"2015-01-01T01:04:42Z\",\"closed_at\":null,\"body\":\"<pre>Originally written by Magd, @TTMTT. </pre>\\r\\nHi..\\r\\nI just open this discuss9 becose the old topic is full and slow for browser .. \\r\\n\\r\\n------------------------------------------------------------------------------------------------------------------------------\\r\\n\\r\\n\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n--------------------------------------\\r\\n\\r\\n------------------------------------------------------------------------------------------------------------------------------\\r\\n\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n------------------------------------\\r\\n\\r\\n------------------------------------------------------------------------------------------------------------------------------\\r\\n\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n------------------------------------------------------------------------------\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n------------------------------------\\r\\n\\r\\n-------------------------------------------------------------------------------------------------------\\r\\n\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n-----------------------------------------------------------------------------\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n-----------------------------------------\\r\\n\\r\\niCL0udin v1.0 -> %97\\r\\n----------------------------\\r\\nRemaining: %3 testing with some people..\\r\\n-------------------------------------\\r\\n\\r\\nLast Method:\\r\\n---------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\n\\r\\niCL0udin v1.0 have this method:\\r\\n-----------------------------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/comments/68477314\",\"html_url\":\"https://github.com/HarisDevs/iCl0udinSupport/issues/6#issuecomment-68477314\",\"issue_url\":\"https://api.github.com/repos/HarisDevs/iCl0udinSupport/issues/6\",\"id\":68477314,\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:42Z\",\"updated_at\":\"2015-01-01T01:04:42Z\",\"body\":\"@HarisDevs \\r\\nexcuse me i open a new discuss for (iCL0udin v1.0):\\r\\n--------------------------------------------------------\\r\\nhttps://github.com/TTMTT/iCL0udin/issues/1\\r\\n--------------------------------------------------------\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:42Z\"}\n{\"id\":\"2489397887\",\"type\":\"WatchEvent\",\"actor\":{\"id\":103510,\"login\":\"chengjunjian\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/chengjunjian\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/103510?\"},\"repo\":{\"id\":22340661,\"name\":\"kaimu/ionic-vs2013-intellisense\",\"url\":\"https://api.github.com/repos/kaimu/ionic-vs2013-intellisense\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:43Z\"}\n{\"id\":\"2489397892\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":18191,\"login\":\"jc00ke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?\"},\"repo\":{\"id\":10488201,\"name\":\"JeanMertz/chruby-fish\",\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish\"},\"payload\":{\"action\":\"opened\",\"number\":15,\"pull_request\":{\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15\",\"id\":26739450,\"html_url\":\"https://github.com/JeanMertz/chruby-fish/pull/15\",\"diff_url\":\"https://github.com/JeanMertz/chruby-fish/pull/15.diff\",\"patch_url\":\"https://github.com/JeanMertz/chruby-fish/pull/15.patch\",\"issue_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/15\",\"number\":15,\"state\":\"open\",\"locked\":false,\"title\":\"Document known PATH warning\",\"user\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:04:45Z\",\"updated_at\":\"2015-01-01T01:04:45Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15/commits\",\"review_comments_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15/comments\",\"review_comment_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/15/comments\",\"statuses_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/statuses/859d6a993456f190598eda5b2a188d96af984cf2\",\"head\":{\"label\":\"jc00ke:master\",\"ref\":\"master\",\"sha\":\"859d6a993456f190598eda5b2a188d96af984cf2\",\"user\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28678223,\"name\":\"chruby-fish\",\"full_name\":\"jc00ke/chruby-fish\",\"owner\":{\"login\":\"jc00ke\",\"id\":18191,\"avatar_url\":\"https://avatars.githubusercontent.com/u/18191?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jc00ke\",\"html_url\":\"https://github.com/jc00ke\",\"followers_url\":\"https://api.github.com/users/jc00ke/followers\",\"following_url\":\"https://api.github.com/users/jc00ke/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jc00ke/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jc00ke/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jc00ke/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jc00ke/orgs\",\"repos_url\":\"https://api.github.com/users/jc00ke/repos\",\"events_url\":\"https://api.github.com/users/jc00ke/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jc00ke/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/jc00ke/chruby-fish\",\"description\":\"Thin wrapper around chruby to make it work with the Fish shell\",\"fork\":true,\"url\":\"https://api.github.com/repos/jc00ke/chruby-fish\",\"forks_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/forks\",\"keys_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/teams\",\"hooks_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/hooks\",\"issue_events_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/events\",\"assignees_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/tags\",\"blobs_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/languages\",\"stargazers_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/stargazers\",\"contributors_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/contributors\",\"subscribers_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/subscribers\",\"subscription_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/subscription\",\"commits_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/merges\",\"archive_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/downloads\",\"issues_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/jc00ke/chruby-fish/releases{/id}\",\"created_at\":\"2015-01-01T01:00:59Z\",\"updated_at\":\"2015-01-01T01:04:34Z\",\"pushed_at\":\"2015-01-01T01:04:34Z\",\"git_url\":\"git://github.com/jc00ke/chruby-fish.git\",\"ssh_url\":\"git@github.com:jc00ke/chruby-fish.git\",\"clone_url\":\"https://github.com/jc00ke/chruby-fish.git\",\"svn_url\":\"https://github.com/jc00ke/chruby-fish\",\"homepage\":\"\",\"size\":1047,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Shell\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"JeanMertz:master\",\"ref\":\"master\",\"sha\":\"e8f28035e7570cbf90568e7dd087810ae7958c8d\",\"user\":{\"login\":\"JeanMertz\",\"id\":383250,\"avatar_url\":\"https://avatars.githubusercontent.com/u/383250?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JeanMertz\",\"html_url\":\"https://github.com/JeanMertz\",\"followers_url\":\"https://api.github.com/users/JeanMertz/followers\",\"following_url\":\"https://api.github.com/users/JeanMertz/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JeanMertz/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JeanMertz/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JeanMertz/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JeanMertz/orgs\",\"repos_url\":\"https://api.github.com/users/JeanMertz/repos\",\"events_url\":\"https://api.github.com/users/JeanMertz/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JeanMertz/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":10488201,\"name\":\"chruby-fish\",\"full_name\":\"JeanMertz/chruby-fish\",\"owner\":{\"login\":\"JeanMertz\",\"id\":383250,\"avatar_url\":\"https://avatars.githubusercontent.com/u/383250?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JeanMertz\",\"html_url\":\"https://github.com/JeanMertz\",\"followers_url\":\"https://api.github.com/users/JeanMertz/followers\",\"following_url\":\"https://api.github.com/users/JeanMertz/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JeanMertz/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JeanMertz/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JeanMertz/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JeanMertz/orgs\",\"repos_url\":\"https://api.github.com/users/JeanMertz/repos\",\"events_url\":\"https://api.github.com/users/JeanMertz/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JeanMertz/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/JeanMertz/chruby-fish\",\"description\":\"Thin wrapper around chruby to make it work with the Fish shell\",\"fork\":false,\"url\":\"https://api.github.com/repos/JeanMertz/chruby-fish\",\"forks_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/forks\",\"keys_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/teams\",\"hooks_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/hooks\",\"issue_events_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/events\",\"assignees_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/tags\",\"blobs_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/languages\",\"stargazers_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/stargazers\",\"contributors_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/contributors\",\"subscribers_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/subscribers\",\"subscription_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/subscription\",\"commits_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/merges\",\"archive_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/downloads\",\"issues_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/JeanMertz/chruby-fish/releases{/id}\",\"created_at\":\"2013-06-04T20:33:07Z\",\"updated_at\":\"2014-12-14T22:38:55Z\",\"pushed_at\":\"2014-12-14T22:38:55Z\",\"git_url\":\"git://github.com/JeanMertz/chruby-fish.git\",\"ssh_url\":\"git@github.com:JeanMertz/chruby-fish.git\",\"clone_url\":\"https://github.com/JeanMertz/chruby-fish.git\",\"svn_url\":\"https://github.com/JeanMertz/chruby-fish\",\"homepage\":\"\",\"size\":1047,\"stargazers_count\":23,\"watchers_count\":23,\"language\":\"Shell\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":6,\"mirror_url\":null,\"open_issues_count\":4,\"forks\":6,\"open_issues\":4,\"watchers\":23,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15\"},\"html\":{\"href\":\"https://github.com/JeanMertz/chruby-fish/pull/15\"},\"issue\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/15\"},\"comments\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/issues/15/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/pulls/15/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/JeanMertz/chruby-fish/statuses/859d6a993456f190598eda5b2a188d96af984cf2\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":14,\"deletions\":0,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:04:45Z\"}\n{\"id\":\"2489397894\",\"type\":\"PushEvent\",\"actor\":{\"id\":1059214,\"login\":\"wlaurance\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wlaurance\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1059214?\"},\"repo\":{\"id\":28050478,\"name\":\"empirical-org/Quill-Grammar\",\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar\"},\"payload\":{\"push_id\":536753132,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/feature/crud-firebase\",\"head\":\"47e38def07f43835e7e91dcde60e8db5edaac315\",\"before\":\"88ee40bc42c418eb0298e525f5cd6137657ff377\",\"commits\":[{\"sha\":\"9835f267914e973c8ca53b28b85b94bdf3086ba2\",\"author\":{\"email\":\"c55061f2e98089f7f71676646e4a1dbbd0f0ebe8@gmail.com\",\"name\":\"Peter Gault\"},\"message\":\"Update README.md\",\"distinct\":false,\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar/commits/9835f267914e973c8ca53b28b85b94bdf3086ba2\"},{\"sha\":\"47e38def07f43835e7e91dcde60e8db5edaac315\",\"author\":{\"email\":\"bc033e145f35c466dd9a9e87a3be67317de99120@gmail.com\",\"name\":\"wlaurance\"},\"message\":\"Merge branch 'feature/crud-firebase' of github.com:Empirical-Org/Quill-Grammar into feature/crud-firebase\",\"distinct\":true,\"url\":\"https://api.github.com/repos/empirical-org/Quill-Grammar/commits/47e38def07f43835e7e91dcde60e8db5edaac315\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:45Z\",\"org\":{\"id\":4258432,\"login\":\"empirical-org\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/empirical-org\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4258432?\"}}\n{\"id\":\"2489397896\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/30\",\"labels_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/30/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/30/comments\",\"events_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/30/events\",\"html_url\":\"https://github.com/captainkirkby/Gears/issues/30\",\"id\":53210138,\"number\":30,\"title\":\"Replaying Binary Data\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/captainkirkby/Gears/labels/documentation\",\"name\":\"documentation\",\"color\":\"eb6420\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:58:37Z\",\"updated_at\":\"2015-01-01T01:04:46Z\",\"closed_at\":null,\"body\":\"The binary file generated as a result of [this](https://github.com/captainkirkby/Gears/commit/24b67ea3dc1bcbc604053689187b41b1ccdac9cd) commit can be replayed and analyzed for anything unexpected that is causing the program to fail.\"},\"comment\":{\"url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/comments/68477315\",\"html_url\":\"https://github.com/captainkirkby/Gears/issues/30#issuecomment-68477315\",\"issue_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/30\",\"id\":68477315,\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:45Z\",\"updated_at\":\"2015-01-01T01:04:45Z\",\"body\":\"*Output*\\r\\n\\r\\nInvalid Data Packet Header!\\r\\nGood Header: fe fe fe 1 15 de 8 0 0 0 0 0 0 0 0 0 0 0 0 0 b1 2e 9c a 0 0 13 8a 1 0 3b c0 6e 72 0 0 20 6 7 0 0 0 0 40 0 0 3c 37 9f a 0 0 0 0 0 0 0 0 3 8 c f 15 1b 21 27 2f 39 41 4c 56 61 6f 7b 88 96 a5 b5 c5 d5 e7 f9 a 1c 2f 42 57 69 7d 91 a4 bb ce e2 f6 a 1e 31 44 57 6a 7c 8e 9f b0 c2 d2 e2 f1 0 10 1c 2a 36 42 4e 59 63 6d 76 7e 87 8d 94 9a 9e a1 a3 a3 a4 a7 a7 a7 a6 a7 a8 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a7 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a8 a7 a6 a6 a8 a8 a7 a6 a7 a7 a7 a6 a6 a7 a8 a7 a6 a6 a7 a7 a7 a7 a7 a8 a8 a6 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a7 a8 a7 a7 a6 a7 a9 a7 a6 a6 a7 a9 a7 a6 a6 a7 a8 a7 a6 a6 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a9 a7 a6 a6 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a7 a9 a8 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a9 a7 a6 a8 a9 a9 a7 a6 a7 a7 a6 a4 a2 a2 a0 9b 94 8d 84 7a 70 65 59 4b 3e 30 22 12 3 f1 df ce bb a8 95 82 6f 5b 46 31 1e 9 f5 e0 cb b8 a3 8e 7c 69 56 44 32 20 10 fe ee de cf c1 b2 a4 97 8b 7f 72 68 5e 56 4c 44 3c 33 2d 25 1e 1a 15 10 c 8 5 3 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 4 a f 15 19 1f 26 2e 39 41 4a 54 61 6d 79 87 95 a5 b3 c4 d4 e6 f9 9 1d 2f 42 57 69 7e 92 a7 bb d0 e3 f7 d 20 34 47 5a 6e 7f 91 a3 b4 c6 d6 e6 f5 4 12 21 2d 39 46 51 5d 65 6f 7a 81 89 90 96 9c 9f a1 a2 a4 a6 a7 a7 a6 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a8 a9 a8 a7 a6 a7 a9 a8 a6 a7 a8 a7 a7 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a6 a7 a8 a7 a6 a6 a8 a8 a7 a6 a6 a8 0 a9 a7 a6 a7 a8 a7 a7 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a6 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a6 a8 a7 a7 a6 a6 a8 a7 a7 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a6 a7 a7 a7 a6 a6 a8 0 a8 a7 a6 a7 a8 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a9 a7 a7 a8 a9 a7 a6 a6 a6 a6 a3 a1 9f 9c 96 90 87 7e 73 68 5c 50 43 35 25 16 7 f6 e5 d3 c1 b0 9c 88 75 61 4e 3a 25 11 fe e8 d4 bf ab 98 84 70 5e 4c 3a 28 16 6 f7 e5 d5 c6 b7 ab 9d 90 84 77 6d 62 58 4f 47 3f 37 2f 27 23 1c 16 10 e b 4 4 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 4 d b e 14 18 1f 25 2d 35 3f 49 53 5e 6a 76 85 91 a0 af bf d0 e1 f2 3 16 29 3c 4e 62 77 8a 9d b1 c6 db ee 2 16 2a 3d 50 62 74 88 99 aa bb cc dc ec fb 9 18 25 33 3f 4a 56 60 6b 73 7c 85 8d 93 99 9c a0 a3 a4 a4 a5 a7 a7 a7 a7 a8 a9 a9 a8 a7 a8 a9 a8 a8 a7 a8 a9 a8 a7 a8 a9 a8 a7 a7 a8 a9 a8 a6 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a7 a9 a8 a6 a6 a8 a9 a8 a6 a7 a8 a8 a7 a6 a7 a9 a8 a7 a6 a8 a8 a8 a6 a6 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a7 a7 a9 a8 a6 a7 a8 a8 a7 a7 a7 a8 a8 a6 a6 a7 a9 a8 a6 a6 a8 a9 a7 a6 a7 a9 a8 a6 a6 a7 a9 a7 a6 a6 a8 a8 a7 a6 a7 a8 a7 a7 a6 a7 a9 a8 a6 a6 a8 a9 a7 a6 a7 a9 a8 a7 a6 a7 a9 a7 a6 a7 a7 a8 a7 a6 a7 a9 a8 a7 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a7 a8 a7 a7 a6 a7 a9 a7 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a8 a8 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a8 a7 a6 a6 a8 a8 a7 a6 a7 a8 a8 a6 a6 a7 a9 a7 a6 a7 a8 a9 a7 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a6 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a9 a8 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a9 a7 a6 a7 a8 a7 a7 a6 a7 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a9 a7 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a8 a8 a6 a6 a8 a9 a8 a6 a7 a8 a8 a7 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a6 a8 a9 a7 a6 a7 a9 a8 a7 a6 a7 a8 a7 a6 a6 a8 a8 a7 a6 a7 a7 a8 a7 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a8 a9 a8 a6 a6 a7 a9 a8 a6 a7 a9 a9 a7 a6 a7 a8 a8 a7 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a7 a8 a6 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a7 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a8 a9 a8 a6 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a6 a8 a8 a7 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a8 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a8 a7 a6 a6 a7 a8 a7 a6 a7 a7 a7 a6 a6 a7 a7 a7 a6 a6 a7 a8 a7 a6 a6 a7 a7 a7 a6 a7 a8 a8 a6 a6 a7 a7 a7 a6 a7 a8 a7 a7 a6 a7 a8 a7 a6 a6 a7 a9 a7 a6 a6 a8 a7 a7 a6 a7 a7 a8 a6 a6 a7 a9 a7 a6 a6 a7 a8 a7 a6 a7 a8 a8 a7 a6 a7 a7 a7 a6 a6 a7 a8 a7 a6 a7 a8 a7 a7 a6 a7 a8 a7 a7 a6 a7 a9 a7 a6 a6 a8 a8 a7 a6 a7 a8 a8 a7 a6 a7 a9 a8 a6 a6 a8 a7 a7 a6 a7 a7 a8 a7 a6 a7 a9 a8 a7 a6 a8 a9 a9 a7 a8 a9 a9 a8 a8 a8 a9 a9 a8 a8 a9 aa a8 a8 a8 a9 a9 a8 a8 a8 aa a9 a8 a7 a8 a7 a6 a4 a4 a2 a0 9b 94 8e 84 7a 6f 64 57 4a 3c 2d 20 10 ff ef dd cc b9 a6 94 81 6c 59 45 30 1d 8 f3 e0 cb b7 a3 8f 7c 6a 57 44 33 21 11 ff ef e0 d0 c3 b4 a6 99 8d 80 75 6a 60 57 4d 45 3d 36 2f 28 20 1b 17 11 c 8 c 3 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 **fe fe fe 1 16 de 8 0 0 0 0 0 0 0 0 0 0**\\r\\n\\r\\n\\r\\n\\r\\nBad Header: 0 0 0 36 2f 9c a 0 0 19 8a 1 0 36 c0 9f 72 0 0 84 0 7 0 0 0 0 40 0 0 3c af 76 4b a8 a6 a8 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a9 a7 a6 a7 a8 a8 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a8 a8 a9 a8 a7 a7 a8 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a7 a8 a8 a7 a6 a8 a9 a8 a7 a7 a9 a9 a7 a6 a7 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a6 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a7 a8 a8 a6 a6 a8 a9 a7 a6 a7 a9 a9 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a7 a9 a9 a8 a6 a7 a9 a8 a6 a6 a8 a9 a8 a7 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a7 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a7 a9 a8 a6 a7 a9 a9 a7 a7 a8 a9 a8 a6 a7 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a8 a9 a8 a7 a6 a8 a9 a8 a6 a8 a9 a9 a8 a7 a8 a9 a8 a6 a7 a8 a8 a7 a6 a7 a9 a8 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a8 a9 a8 a6 a6 a8 a9 a8 a6 a7 a9 a9 a7 a6 a8 a9 a8 a7 a6 a8 a9 a8 a6 a7 a9 a8 a7 a7 a8 a9 a8 a6 a7 a8 a9 a8 a7 a7 a8 a8 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a9 a8 a6 a8 a9 a8 a7 a6 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a9 a8 a8 a7 a9 a9 a8 a6 a8 a9 a9 a8 a8 a8 a9 a8 a8 a8 a9 a9 a8 a8 a8 a9 a9 a8 a7 a8 a8 a6 a5 a6 a5 a3 a1 9e 9c 97 90 88 82 78 6f 63 59 4e 42 35 28 1b d ff ef df ce bd ab 99 88 76 63 51 3d 2c 17 3 ef db c7 b2 9e 8a 77 63 50 3c 29 18 5 f2 e1 d0 bf b0 a0 92 85 76 69 5e 52 49 3e 35 2c 25 1f 18 11 e b 5 5 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 4 a e 11 16 1c 22 2a 30 38 40 47 51 5a 64 6f 7b 87 92 9e ad bb ca d9 e9 f9 9 1b 2c 3e 51 62 75 88 9c b1 c4 d8 ec 2 16 29 3d 51 65 78 8c 9f b1 c5 d6 e8 f9 8 19 28 37 44 52 5e 6b 75 80 8b 92 98 9d 9f a2 a5 a6 a5 a6 a7 a9 a8 a6 a8 a9 a9 a8 a8 a8 a9 a9 a8 a7 a9 a9 a8 a8 a8 a9 a9 a7 a7 a8 a9 a8 a8 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a9 a9 a8 a6 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a8 a7 a7 a8 a9 a8 a8 a7 a8 a9 a7 a6 a8 a9 a9 a8 a7 a8 a9 a8 a7 a7 a8 a8 a8 a6 a7 a9 a8 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a8 a9 a8 a7 a8 a9 a8 a7 a7 a8 a9 a8 a7 a8 a9 a9 a8 a8 a8 a9 a8 a7 a7 a9 a9 a8 a7 a7 a7 a7 a5 a4 a4 a2 9f 9b 98 92 8a 81 79 70 67 5b 50 45 38 2b 1e 10 2 f3 e3 d2 c3 b0 9f 8c 7a 68 55 42 2e 1b 7 f4 df ca b7 a1 8d 78 65 51 3e 2a 18 7 f3 e1 d0 c0 b1 a0 92 84 77 6a 5e 52 47 3f 35 2c 25 1e 19 12 d a a 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 e b f 12 16 1d 23 29 30 37 41 49 51 5a 64 70 7a 86 92 9f ae bb c9 d9 e8 f8 9 18 2b 3e 4f 62 74 87 9c af c3 d7 ec 0 15 29 3d 53 66 7a 8d a0 b3 c5 d7 e8 fa a 1a 29 38 47 53 60 69 75 80 8a 92 98 9d a2 a3 a5 a4 a6 a7 a7 a7 a6 a8 a9 a9 a8 a7 a8 a9 a8 a7 a8 a9 a9 a8 a7 a8 a9 a9 a8 a8 a8 a9 a9 a7 a7 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a7 a8 a9 a8 a6 a6 a8 a9 a7 a6 a7 a9 a9 a7 a7 a8 a9 a8 a7 a6 a8 a9 a8 a7 a8 a9 a9 a8 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a8 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a7 a8 a9 a9 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a7 a7 a9 a8 a7 a7 a8 a9 a8 a7 a8 a9 a9 a8 a6 a8 a9 a8 a7 a6 a7 a7 a6 a4 a4 a4 a2 9e 9a 97 91 88 80 78 6e 65 5a 4f 44 38 2b 1d f 0 f2 e2 d1 c2 b0 9f 8d 7c 6a 57 43 30 1c 9 f5 e1 cd b9 a4 90 7c 68 55 42 2e 1c a f8 e7 d5 c4 b5 a5 96 88 7b 6d 61 56 4b 42 39 2f 27 20 1b 15 f b a 3 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 a e 13 19 1c 22 2a 30 39 3f 48 51 5b 64 6e 7a 85 92 9e ac ba c9 d8 e8 f6 7 19 2a 3b 4d 60 73 86 99 ad c2 d6 ea fe 12 26 3b 4e 62 76 8a 9d b0 c2 d6 e6 f7 6 17 27 36 43 51 5e 6a 76 7f 89 92 99 9e a0 a2 a5 a7 a7 a7 a8 a9 a9 a8 a8 a8 aa a9 a8 a8 a9 aa a9 a8 a8 aa a9 a8 a8 a8 aa a9 a8 a8 a9 a9 a8 a7 a7 a9 a9 a7 a7 a8 a9 a8 a8 a7 a8 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a6 a7 a9 a8 a7 a6 a8 a9 a8 a7 a7 a9 a8 a7 a6 a7 a9 a8 a7 a6 a8 a9 a8 a7 a7 a8 a8 a7 a6 a8 a9 a8 a7 a7 a8 a9 a7 a6 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a8 a7 a6 a7 a9 a8 a7 a7 a9 a9 a8 a7 a7 a9 a8 a6 a6 a8 a9 a8 a7 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a8 a7 a7 a8 a9 a8 a7 a7 a9 a9 a8 a6 a8 a9 a8 a7 a7 a8 a9 a7 a6 a7 a9 a8 a7 a6 a8 a9 a8 a6 a7 a8 a9 a7 a6 a7 a9 a8 a7 a6 a8 a8 a7 a6 a7 a9 a9 a8 a6 a7 a9 a8 a7 a7 a9 a9 a8 a6 a7 a9 a8 a6 a6 a8 a9 a8 a7 a7 a9 a9 a7 a6 a8 a9 a8 a7 a7 a8 a9 a8 a7 a8 a9 a8 a6 a6 a8 a9 a7 a6 a8 a9 a9 a8 a6 a8 a9 a7 a6 a8 a9 a9 fe fe fe 1 17 de 8 0 0 0 0 0 0 0 0 0 0 \\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:46Z\"}\n{\"id\":\"2489397899\",\"type\":\"PushEvent\",\"actor\":{\"id\":4418470,\"login\":\"VapidLinus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/VapidLinus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4418470?\"},\"repo\":{\"id\":28103737,\"name\":\"VapidLinus/ludum\",\"url\":\"https://api.github.com/repos/VapidLinus/ludum\"},\"payload\":{\"push_id\":536753136,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"71d9e25ca28ee11bd0f7b2e0e4aa3284d327634d\",\"before\":\"7651a033556ffb15be11022a1705eab9ee38b748\",\"commits\":[{\"sha\":\"71d9e25ca28ee11bd0f7b2e0e4aa3284d327634d\",\"author\":{\"email\":\"2186255cea3741ffd28e8c68c5a7c2030a5f82da@gmail.com\",\"name\":\"Linus Närkling-Larsson\"},\"message\":\"Restructure\",\"distinct\":true,\"url\":\"https://api.github.com/repos/VapidLinus/ludum/commits/71d9e25ca28ee11bd0f7b2e0e4aa3284d327634d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:47Z\"}\n{\"id\":\"2489397900\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":2066666,\"login\":\"amadornes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amadornes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2066666?\"},\"repo\":{\"id\":21494560,\"name\":\"Qmunity/BluePower\",\"url\":\"https://api.github.com/repos/Qmunity/BluePower\"},\"payload\":{\"ref\":\"NewMultipart\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:47Z\",\"org\":{\"id\":8006321,\"login\":\"Qmunity\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Qmunity\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8006321?\"}}\n{\"id\":\"2489397903\",\"type\":\"PushEvent\",\"actor\":{\"id\":8350185,\"login\":\"Johannes-Larsson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Johannes-Larsson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8350185?\"},\"repo\":{\"id\":28634222,\"name\":\"Johannes-Larsson/towerDefence\",\"url\":\"https://api.github.com/repos/Johannes-Larsson/towerDefence\"},\"payload\":{\"push_id\":536753137,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"25b4d5160036fa439a4b44cdb10579f1807610fa\",\"before\":\"348fbe1d9736ea23729e344a37e1f68b669709cc\",\"commits\":[{\"sha\":\"9874efb97644ec0e3dcf44deffcb81407b4d9a69\",\"author\":{\"email\":\"521728287efe6ca64f05202f1df9a036cc8f0398@gmail.com\",\"name\":\"Johannes-Larsson\"},\"message\":\"fixed ugly testing of tower upgrade texture system\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Johannes-Larsson/towerDefence/commits/9874efb97644ec0e3dcf44deffcb81407b4d9a69\"},{\"sha\":\"25b4d5160036fa439a4b44cdb10579f1807610fa\",\"author\":{\"email\":\"521728287efe6ca64f05202f1df9a036cc8f0398@gmail.com\",\"name\":\"Johannes-Larsson\"},\"message\":\"made clicking anywhere exit upgradingMenu\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Johannes-Larsson/towerDefence/commits/25b4d5160036fa439a4b44cdb10579f1807610fa\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:47Z\"}\n{\"id\":\"2489397911\",\"type\":\"PushEvent\",\"actor\":{\"id\":906529,\"login\":\"dpwolfe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dpwolfe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/906529?\"},\"repo\":{\"id\":26579311,\"name\":\"dpwolfe/otucha\",\"url\":\"https://api.github.com/repos/dpwolfe/otucha\"},\"payload\":{\"push_id\":536753141,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"89bbbe832e8916afacae2fe98885ae5af5391a7a\",\"before\":\"6f79c0b71c6d4a7e3a7d538ed3b96a889c2a33e0\",\"commits\":[{\"sha\":\"89bbbe832e8916afacae2fe98885ae5af5391a7a\",\"author\":{\"email\":\"b7dfe270ecb2603aba704ea15b776485da19da15@gmail.com\",\"name\":\"David Wolfe\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dpwolfe/otucha/commits/89bbbe832e8916afacae2fe98885ae5af5391a7a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:48Z\"}\n{\"id\":\"2489397912\",\"type\":\"ForkEvent\",\"actor\":{\"id\":416575,\"login\":\"frewsxcv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?\"},\"repo\":{\"id\":3545112,\"name\":\"arthur-e/Wicket\",\"url\":\"https://api.github.com/repos/arthur-e/Wicket\"},\"payload\":{\"forkee\":{\"id\":28678263,\"name\":\"Wicket\",\"full_name\":\"frewsxcv/Wicket\",\"owner\":{\"login\":\"frewsxcv\",\"id\":416575,\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"html_url\":\"https://github.com/frewsxcv\",\"followers_url\":\"https://api.github.com/users/frewsxcv/followers\",\"following_url\":\"https://api.github.com/users/frewsxcv/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/frewsxcv/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/frewsxcv/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/frewsxcv/subscriptions\",\"organizations_url\":\"https://api.github.com/users/frewsxcv/orgs\",\"repos_url\":\"https://api.github.com/users/frewsxcv/repos\",\"events_url\":\"https://api.github.com/users/frewsxcv/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/frewsxcv/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/frewsxcv/Wicket\",\"description\":\"A modest library for moving between Well-Known Text (WKT) and various framework geometries\",\"fork\":true,\"url\":\"https://api.github.com/repos/frewsxcv/Wicket\",\"forks_url\":\"https://api.github.com/repos/frewsxcv/Wicket/forks\",\"keys_url\":\"https://api.github.com/repos/frewsxcv/Wicket/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/frewsxcv/Wicket/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/frewsxcv/Wicket/teams\",\"hooks_url\":\"https://api.github.com/repos/frewsxcv/Wicket/hooks\",\"issue_events_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/frewsxcv/Wicket/events\",\"assignees_url\":\"https://api.github.com/repos/frewsxcv/Wicket/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/frewsxcv/Wicket/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/frewsxcv/Wicket/tags\",\"blobs_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/frewsxcv/Wicket/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/frewsxcv/Wicket/languages\",\"stargazers_url\":\"https://api.github.com/repos/frewsxcv/Wicket/stargazers\",\"contributors_url\":\"https://api.github.com/repos/frewsxcv/Wicket/contributors\",\"subscribers_url\":\"https://api.github.com/repos/frewsxcv/Wicket/subscribers\",\"subscription_url\":\"https://api.github.com/repos/frewsxcv/Wicket/subscription\",\"commits_url\":\"https://api.github.com/repos/frewsxcv/Wicket/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/frewsxcv/Wicket/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/frewsxcv/Wicket/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/frewsxcv/Wicket/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/frewsxcv/Wicket/merges\",\"archive_url\":\"https://api.github.com/repos/frewsxcv/Wicket/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/frewsxcv/Wicket/downloads\",\"issues_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/frewsxcv/Wicket/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/frewsxcv/Wicket/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/frewsxcv/Wicket/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/frewsxcv/Wicket/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/frewsxcv/Wicket/releases{/id}\",\"created_at\":\"2015-01-01T01:04:48Z\",\"updated_at\":\"2014-12-18T11:08:34Z\",\"pushed_at\":\"2014-11-12T13:58:08Z\",\"git_url\":\"git://github.com/frewsxcv/Wicket.git\",\"ssh_url\":\"git@github.com:frewsxcv/Wicket.git\",\"clone_url\":\"https://github.com/frewsxcv/Wicket.git\",\"svn_url\":\"https://github.com/frewsxcv/Wicket\",\"homepage\":\"http://arthur-e.github.com/Wicket\",\"size\":1032,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:04:48Z\"}\n{\"id\":\"2489397913\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3913258,\"login\":\"blumu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blumu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3913258?\"},\"repo\":{\"id\":2087064,\"name\":\"SignalR/SignalR\",\"url\":\"https://api.github.com/repos/SignalR/SignalR\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:49Z\",\"org\":{\"id\":931666,\"login\":\"SignalR\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/SignalR\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/931666?\"}}\n{\"id\":\"2489397915\",\"type\":\"PushEvent\",\"actor\":{\"id\":6982503,\"login\":\"josephsands\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/josephsands\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6982503?\"},\"repo\":{\"id\":28678205,\"name\":\"josephsands/josephsands.github.io\",\"url\":\"https://api.github.com/repos/josephsands/josephsands.github.io\"},\"payload\":{\"push_id\":536753142,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"230208da74a1d0f76aca87882a3ed3954c9aa3ac\",\"before\":\"1db4c2b9fabe365629330df61852a56f8d3abf2d\",\"commits\":[{\"sha\":\"230208da74a1d0f76aca87882a3ed3954c9aa3ac\",\"author\":{\"email\":\"793a20fb71e8611e634b8b771001d6777640050d@Anitas-MBP.home\",\"name\":\"Joseph Sands\"},\"message\":\"Site updated at 2015-01-01 01:04:46 UTC\",\"distinct\":true,\"url\":\"https://api.github.com/repos/josephsands/josephsands.github.io/commits/230208da74a1d0f76aca87882a3ed3954c9aa3ac\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:49Z\"}\n{\"id\":\"2489397916\",\"type\":\"PushEvent\",\"actor\":{\"id\":259982,\"login\":\"kigster\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kigster\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/259982?\"},\"repo\":{\"id\":20339421,\"name\":\"wanelo-chef/postmodern\",\"url\":\"https://api.github.com/repos/wanelo-chef/postmodern\"},\"payload\":{\"push_id\":536753143,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e8c6d4d0d091c90b4e4a6b70a33e79247b7e770c\",\"before\":\"f64e8f114159054ab23fc9883b5476feddcfd6ed\",\"commits\":[{\"sha\":\"e8c6d4d0d091c90b4e4a6b70a33e79247b7e770c\",\"author\":{\"email\":\"fdced6b9ecadb87ee4278610f60ef5876a49575d@gmail.com\",\"name\":\"Konstantin Gredeskoul\"},\"message\":\"Explicitly include ShellOut so ruby is happy\\n\\nWas getting this error on feed-db200.prod after adding \\nfeed-db-replica role.\\n\\n[2015-01-01T00:54:19+00:00] WARN: Current  service[nad]: /var/chef/cache/cookbooks/nad/recipes/default.rb:135:in `from_file'\\n\\n  ================================================================================\\n  Recipe Compile Error in /var/chef/cache/cookbooks/wanelo-feed/recipes/database_server.rb\\n  ================================================================================\\n\\n  NoMethodError\\n  -------------\\n  No resource or method named `shell_out' for `Chef::Recipe \\\"default\\\"'\\n\\n  Cookbook Trace:\\n  ---------------\\n    /var/chef/cache/cookbooks/postmodern/recipes/default.rb:6:in `from_file'\\n    /var/chef/cache/cookbooks/wanelo-feed/recipes/database_server.rb:55:in `from_file'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wanelo-chef/postmodern/commits/e8c6d4d0d091c90b4e4a6b70a33e79247b7e770c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:49Z\",\"org\":{\"id\":2664866,\"login\":\"wanelo-chef\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wanelo-chef\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2664866?\"}}\n{\"id\":\"2489397920\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":1935061,\"name\":\"sunng87/node-geohash\",\"url\":\"https://api.github.com/repos/sunng87/node-geohash\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:50Z\"}\n{\"id\":\"2489397936\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":8218499,\"login\":\"GunZi200\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GunZi200\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8218499?\"},\"repo\":{\"id\":27030606,\"name\":\"GunZi200/Memory-Colour\",\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/1\",\"labels_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/1/events\",\"html_url\":\"https://github.com/GunZi200/Memory-Colour/issues/1\",\"id\":53210268,\"number\":1,\"title\":\"iPad drawing issue\",\"user\":{\"login\":\"GunZi200\",\"id\":8218499,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8218499?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GunZi200\",\"html_url\":\"https://github.com/GunZi200\",\"followers_url\":\"https://api.github.com/users/GunZi200/followers\",\"following_url\":\"https://api.github.com/users/GunZi200/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/GunZi200/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/GunZi200/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/GunZi200/subscriptions\",\"organizations_url\":\"https://api.github.com/users/GunZi200/orgs\",\"repos_url\":\"https://api.github.com/users/GunZi200/repos\",\"events_url\":\"https://api.github.com/users/GunZi200/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/GunZi200/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:04:52Z\",\"updated_at\":\"2015-01-01T01:04:52Z\",\"closed_at\":null,\"body\":\"There is a known issue I will fix sometime soon. That is the iPad version of Colour Trio.\\r\\n\\r\\nThe program prints over the black buttons each time your tap a correct colour. Displaying the start screen... really.\\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:52Z\"}\n{\"id\":\"2489397938\",\"type\":\"PushEvent\",\"actor\":{\"id\":480938,\"login\":\"hubot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hubot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/480938?\"},\"repo\":{\"id\":2389274,\"name\":\"eclipse/eclipse.platform.swt\",\"url\":\"https://api.github.com/repos/eclipse/eclipse.platform.swt\"},\"payload\":{\"push_id\":536753150,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1d2c32bfdfd0569960bbd941e6670f1e366946a3\",\"before\":\"231816a5d4318faedbe56a995e907e59c9a91836\",\"commits\":[{\"sha\":\"1d2c32bfdfd0569960bbd941e6670f1e366946a3\",\"author\":{\"email\":\"7080c0b5db7f1b7254ce7ee1b5cbbb7b6872b877@in.ibm.com\",\"name\":\"Arun Thondapu\"},\"message\":\"v4509\",\"distinct\":true,\"url\":\"https://api.github.com/repos/eclipse/eclipse.platform.swt/commits/1d2c32bfdfd0569960bbd941e6670f1e366946a3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:53Z\",\"org\":{\"id\":56974,\"login\":\"eclipse\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/eclipse\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/56974?\"}}\n{\"id\":\"2489397941\",\"type\":\"PushEvent\",\"actor\":{\"id\":5975070,\"login\":\"marcellodibello\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marcellodibello\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5975070?\"},\"repo\":{\"id\":17580814,\"name\":\"marcellodibello/marcellodibello.github.io\",\"url\":\"https://api.github.com/repos/marcellodibello/marcellodibello.github.io\"},\"payload\":{\"push_id\":536753151,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1194cc3417673dbd3baf1a625d7ce4e374fd755a\",\"before\":\"7995cfd2d91a5b11eff75239ac124fae9af80d6d\",\"commits\":[{\"sha\":\"1194cc3417673dbd3baf1a625d7ce4e374fd755a\",\"author\":{\"email\":\"babd00909fb78351e79d40af26c77fc37bb4ed59@gmail.com\",\"name\":\"marcellodibello\"},\"message\":\"Update index.html\",\"distinct\":true,\"url\":\"https://api.github.com/repos/marcellodibello/marcellodibello.github.io/commits/1194cc3417673dbd3baf1a625d7ce4e374fd755a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:53Z\"}\n{\"id\":\"2489397944\",\"type\":\"WatchEvent\",\"actor\":{\"id\":6161385,\"login\":\"mycaule\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mycaule\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6161385?\"},\"repo\":{\"id\":211666,\"name\":\"joyent/node\",\"url\":\"https://api.github.com/repos/joyent/node\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:53Z\",\"org\":{\"id\":10161,\"login\":\"joyent\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/joyent\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10161?\"}}\n{\"id\":\"2489397946\",\"type\":\"PushEvent\",\"actor\":{\"id\":1165609,\"login\":\"bleroy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bleroy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1165609?\"},\"repo\":{\"id\":28653536,\"name\":\"DecentCMS/DecentCMS\",\"url\":\"https://api.github.com/repos/DecentCMS/DecentCMS\"},\"payload\":{\"push_id\":536753152,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"81ac106d21544f7cd0cca46d172c1d3174f7db4a\",\"before\":\"4c55fad17d09de43a3566d8d77a2999317a671f9\",\"commits\":[{\"sha\":\"81ac106d21544f7cd0cca46d172c1d3174f7db4a\",\"author\":{\"email\":\"aff71552ea35fa74bb03c867fe0c4cde09895eea@gmail.com\",\"name\":\"Bertrand Le Roy\"},\"message\":\"Rename Winston module to Logging.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/DecentCMS/DecentCMS/commits/81ac106d21544f7cd0cca46d172c1d3174f7db4a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:53Z\",\"org\":{\"id\":10110483,\"login\":\"DecentCMS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/DecentCMS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10110483?\"}}\n{\"id\":\"2489397948\",\"type\":\"PushEvent\",\"actor\":{\"id\":2916016,\"login\":\"TylerSandman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TylerSandman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2916016?\"},\"repo\":{\"id\":28609731,\"name\":\"TylerSandman/poro-cv\",\"url\":\"https://api.github.com/repos/TylerSandman/poro-cv\"},\"payload\":{\"push_id\":536753153,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e400f73a414aea729b6b5b01ec55a427bc140fd4\",\"before\":\"a9d9ff7fb436827e80293189d26d864aff80c6e4\",\"commits\":[{\"sha\":\"e400f73a414aea729b6b5b01ec55a427bc140fd4\",\"author\":{\"email\":\"069eabe48263b36d2b5eab03e90d3687ed30521d@gmail.com\",\"name\":\"Tyler Sanderson\"},\"message\":\"Added result handling after poro detection\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TylerSandman/poro-cv/commits/e400f73a414aea729b6b5b01ec55a427bc140fd4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:53Z\"}\n{\"id\":\"2489397952\",\"type\":\"CreateEvent\",\"actor\":{\"id\":8890114,\"login\":\"speeldoos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/speeldoos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8890114?\"},\"repo\":{\"id\":28678093,\"name\":\"speeldoos/hello-world\",\"url\":\"https://api.github.com/repos/speeldoos/hello-world\"},\"payload\":{\"ref\":\"readme-edits\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"testing testing\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:54Z\"}\n{\"id\":\"2489397954\",\"type\":\"PushEvent\",\"actor\":{\"id\":2715854,\"login\":\"miguelgrinberg\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miguelgrinberg\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2715854?\"},\"repo\":{\"id\":28583035,\"name\":\"miguelgrinberg/flask-celery-example\",\"url\":\"https://api.github.com/repos/miguelgrinberg/flask-celery-example\"},\"payload\":{\"push_id\":536753156,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"968b9d6aa0440d1063237d0d128abbb89961f920\",\"before\":\"234a991f30f75a7a93e97bd535183f6b81067f2d\",\"commits\":[{\"sha\":\"968b9d6aa0440d1063237d0d128abbb89961f920\",\"author\":{\"email\":\"867e6f7716cd7d89b2aa6d04c9799a31b0e6903e@gmail.com\",\"name\":\"Miguel Grinberg\"},\"message\":\"initial version\",\"distinct\":true,\"url\":\"https://api.github.com/repos/miguelgrinberg/flask-celery-example/commits/968b9d6aa0440d1063237d0d128abbb89961f920\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:54Z\"}\n{\"id\":\"2489397955\",\"type\":\"GollumEvent\",\"actor\":{\"id\":46323,\"login\":\"paulcon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulcon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/46323?\"},\"repo\":{\"id\":28157780,\"name\":\"paulcon/active_subspaces\",\"url\":\"https://api.github.com/repos/paulcon/active_subspaces\"},\"payload\":{\"pages\":[{\"page_name\":\"_Footer\",\"title\":\"_Footer\",\"summary\":null,\"action\":\"edited\",\"sha\":\"ffa925d0740eb5600fc7c2a54271d4b925a8a8f8\",\"html_url\":\"https://github.com/paulcon/active_subspaces/wiki/_Footer\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:54Z\"}\n{\"id\":\"2489397957\",\"type\":\"PushEvent\",\"actor\":{\"id\":183517,\"login\":\"troglobit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/troglobit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/183517?\"},\"repo\":{\"id\":5778132,\"name\":\"troglobit/finit\",\"url\":\"https://api.github.com/repos/troglobit/finit\"},\"payload\":{\"push_id\":536753159,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8f1c16a8a5872d87197c7c346957f2e7e7262c5e\",\"before\":\"b149dbc73e701f4385cb414ba7b87dd58b345b7d\",\"commits\":[{\"sha\":\"8f1c16a8a5872d87197c7c346957f2e7e7262c5e\",\"author\":{\"email\":\"583c295fd7602c168ad814279bbc3894ba65f5d6@gmail.com\",\"name\":\"Joachim Nilsson\"},\"message\":\"Update TODO with section on Cron.\\n\\nSigned-off-by: Joachim Nilsson <troglobit@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/troglobit/finit/commits/8f1c16a8a5872d87197c7c346957f2e7e7262c5e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:54Z\"}\n{\"id\":\"2489397960\",\"type\":\"WatchEvent\",\"actor\":{\"id\":7576774,\"login\":\"lovelybigdata\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lovelybigdata\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7576774?\"},\"repo\":{\"id\":14681876,\"name\":\"codelucas/newspaper\",\"url\":\"https://api.github.com/repos/codelucas/newspaper\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:55Z\"}\n{\"id\":\"2489397963\",\"type\":\"PushEvent\",\"actor\":{\"id\":3720783,\"login\":\"designerwebhosting\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/designerwebhosting\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3720783?\"},\"repo\":{\"id\":20527117,\"name\":\"designerwebhosting/christopherbyrne.github.io\",\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io\"},\"payload\":{\"push_id\":536753161,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"7e3ee7d19b64e0f0571098be4edb9729d09fb4f4\",\"before\":\"52997f806edc1645856950072f9919dea680b5a7\",\"commits\":[{\"sha\":\"7e3ee7d19b64e0f0571098be4edb9729d09fb4f4\",\"author\":{\"email\":\"4bb0acc6ff8c0b6c31e50417877e6e3b3f1c65f0@googlemail.com\",\"name\":\"Peter Noblee\"},\"message\":\"update 'date'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io/commits/7e3ee7d19b64e0f0571098be4edb9729d09fb4f4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:55Z\"}\n{\"id\":\"2489397964\",\"type\":\"ReleaseEvent\",\"actor\":{\"id\":3134745,\"login\":\"Vextil\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Vextil\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3134745?\"},\"repo\":{\"id\":28277457,\"name\":\"Vextil/WAMD\",\"url\":\"https://api.github.com/repos/Vextil/WAMD\"},\"payload\":{\"action\":\"published\",\"release\":{\"url\":\"https://api.github.com/repos/Vextil/WAMD/releases/818218\",\"assets_url\":\"https://api.github.com/repos/Vextil/WAMD/releases/818218/assets\",\"upload_url\":\"https://uploads.github.com/repos/Vextil/WAMD/releases/818218/assets{?name}\",\"html_url\":\"https://github.com/Vextil/WAMD/releases/tag/2.1.0-release\",\"id\":818218,\"tag_name\":\"2.1.0-release\",\"target_commitish\":\"master\",\"name\":\"2.1.0 Release\",\"draft\":false,\"author\":{\"login\":\"Vextil\",\"id\":3134745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3134745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Vextil\",\"html_url\":\"https://github.com/Vextil\",\"followers_url\":\"https://api.github.com/users/Vextil/followers\",\"following_url\":\"https://api.github.com/users/Vextil/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Vextil/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Vextil/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Vextil/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Vextil/orgs\",\"repos_url\":\"https://api.github.com/users/Vextil/repos\",\"events_url\":\"https://api.github.com/users/Vextil/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Vextil/received_events\",\"type\":\"User\",\"site_admin\":false},\"prerelease\":false,\"created_at\":\"2015-01-01T00:40:15Z\",\"published_at\":\"2015-01-01T01:04:55Z\",\"assets\":[{\"url\":\"https://api.github.com/repos/Vextil/WAMD/releases/assets/362005\",\"id\":362005,\"name\":\"WhatsAppMD.G_.Google_Emojis.apk\",\"label\":null,\"uploader\":{\"login\":\"Vextil\",\"id\":3134745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3134745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Vextil\",\"html_url\":\"https://github.com/Vextil\",\"followers_url\":\"https://api.github.com/users/Vextil/followers\",\"following_url\":\"https://api.github.com/users/Vextil/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Vextil/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Vextil/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Vextil/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Vextil/orgs\",\"repos_url\":\"https://api.github.com/users/Vextil/repos\",\"events_url\":\"https://api.github.com/users/Vextil/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Vextil/received_events\",\"type\":\"User\",\"site_admin\":false},\"content_type\":\"application/vnd.android.package-archive\",\"state\":\"uploaded\",\"size\":22511638,\"download_count\":0,\"created_at\":\"2015-01-01T00:59:56Z\",\"updated_at\":\"2015-01-01T01:01:45Z\",\"browser_download_url\":\"https://github.com/Vextil/WAMD/releases/download/2.1.0-release/WhatsAppMD.G_.Google_Emojis.apk\"},{\"url\":\"https://api.github.com/repos/Vextil/WAMD/releases/assets/362004\",\"id\":362004,\"name\":\"WhatsAppMD.S_.Stock_Emojis.apk\",\"label\":null,\"uploader\":{\"login\":\"Vextil\",\"id\":3134745,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3134745?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Vextil\",\"html_url\":\"https://github.com/Vextil\",\"followers_url\":\"https://api.github.com/users/Vextil/followers\",\"following_url\":\"https://api.github.com/users/Vextil/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Vextil/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Vextil/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Vextil/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Vextil/orgs\",\"repos_url\":\"https://api.github.com/users/Vextil/repos\",\"events_url\":\"https://api.github.com/users/Vextil/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Vextil/received_events\",\"type\":\"User\",\"site_admin\":false},\"content_type\":\"application/vnd.android.package-archive\",\"state\":\"uploaded\",\"size\":18707397,\"download_count\":0,\"created_at\":\"2015-01-01T00:59:56Z\",\"updated_at\":\"2015-01-01T01:01:07Z\",\"browser_download_url\":\"https://github.com/Vextil/WAMD/releases/download/2.1.0-release/WhatsAppMD.S_.Stock_Emojis.apk\"}],\"tarball_url\":\"https://api.github.com/repos/Vextil/WAMD/tarball/2.1.0-release\",\"zipball_url\":\"https://api.github.com/repos/Vextil/WAMD/zipball/2.1.0-release\",\"body\":\"**5.0.0 and up:**\\r\\n- No specific changes for this version\\r\\n\\r\\n**4.4.4 and down:**\\r\\n- Added tinted statusbar for KitKat\\r\\n\\r\\n**All Android versions:**\\r\\n- Fixed attach menu black background in MDPI\\r\\n- About screen now shows WhatsAppMD version info too\\r\\n- Removed camera icon in chat text field\\r\\n- (For modders) Layout files are more readable\\r\\n- Improved chat text field design\\r\\n- The screed now has a dim white overlay when the FAB is opened\\r\\n- Changed FAB text label design\\r\\n- FAB no longer stays open when leaving and coming back\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:55Z\"}\n{\"id\":\"2489397966\",\"type\":\"PushEvent\",\"actor\":{\"id\":126935,\"login\":\"theresaanna\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/theresaanna\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/126935?\"},\"repo\":{\"id\":25273785,\"name\":\"18F/openFEC-web-app\",\"url\":\"https://api.github.com/repos/18F/openFEC-web-app\"},\"payload\":{\"push_id\":536753162,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/selenium\",\"head\":\"837d603ad978daad675b2b387500eab3750b5ce1\",\"before\":\"b213b555b98a97b8ecd7213d2f8148a6d408b006\",\"commits\":[{\"sha\":\"837d603ad978daad675b2b387500eab3750b5ce1\",\"author\":{\"email\":\"102aa063a04718b7675f2a11275a651f31704f92@gsa.gov\",\"name\":\"Theresa Summa\"},\"message\":\"fixing tests\",\"distinct\":true,\"url\":\"https://api.github.com/repos/18F/openFEC-web-app/commits/837d603ad978daad675b2b387500eab3750b5ce1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:55Z\",\"org\":{\"id\":6233994,\"login\":\"18F\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/18F\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6233994?\"}}\n{\"id\":\"2489397968\",\"type\":\"CreateEvent\",\"actor\":{\"id\":3134745,\"login\":\"Vextil\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Vextil\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3134745?\"},\"repo\":{\"id\":28277457,\"name\":\"Vextil/WAMD\",\"url\":\"https://api.github.com/repos/Vextil/WAMD\"},\"payload\":{\"ref\":\"2.1.0-release\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"WAMD is a Material Design mod of the most popular mobile messaging app.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:56Z\"}\n{\"id\":\"2489397970\",\"type\":\"PushEvent\",\"actor\":{\"id\":66897,\"login\":\"asad\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/asad\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/66897?\"},\"repo\":{\"id\":28622285,\"name\":\"asad/ChemBLAST\",\"url\":\"https://api.github.com/repos/asad/ChemBLAST\"},\"payload\":{\"push_id\":536753164,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f82c115e656962e0ee327c14a26c586b5da5d53b\",\"before\":\"4fd33b8625d56ca7dcda0acc38f6f51ec1358c87\",\"commits\":[{\"sha\":\"f82c115e656962e0ee327c14a26c586b5da5d53b\",\"author\":{\"email\":\"09eb4ce7c91ecff4eff21da1532c566afe7cb66c@gmail.com\",\"name\":\"Syed Asad Rahman\"},\"message\":\"fork and join blast search implemented - 4X speed obtained\",\"distinct\":true,\"url\":\"https://api.github.com/repos/asad/ChemBLAST/commits/f82c115e656962e0ee327c14a26c586b5da5d53b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:56Z\"}\n{\"id\":\"2489397975\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1177325,\"login\":\"iguto\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/iguto\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1177325?\"},\"repo\":{\"id\":26689598,\"name\":\"prakhar1989/awesome-courses\",\"url\":\"https://api.github.com/repos/prakhar1989/awesome-courses\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:57Z\"}\n{\"id\":\"2489397976\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":790740,\"login\":\"bokmadsen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bokmadsen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/790740?\"},\"repo\":{\"id\":20463939,\"name\":\"XLabs/Xamarin-Forms-Labs\",\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/534\",\"labels_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/534/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/534/comments\",\"events_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/534/events\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/534\",\"id\":52600191,\"number\":534,\"title\":\"Added null check in HybridWebView\",\"user\":{\"login\":\"bokmadsen\",\"id\":790740,\"avatar_url\":\"https://avatars.githubusercontent.com/u/790740?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bokmadsen\",\"html_url\":\"https://github.com/bokmadsen\",\"followers_url\":\"https://api.github.com/users/bokmadsen/followers\",\"following_url\":\"https://api.github.com/users/bokmadsen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bokmadsen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bokmadsen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bokmadsen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bokmadsen/orgs\",\"repos_url\":\"https://api.github.com/users/bokmadsen/repos\",\"events_url\":\"https://api.github.com/users/bokmadsen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bokmadsen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-21T17:00:20Z\",\"updated_at\":\"2015-01-01T01:04:57Z\",\"closed_at\":\"2015-01-01T00:59:55Z\",\"pull_request\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/pulls/534\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/534\",\"diff_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/534.diff\",\"patch_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/534.patch\"},\"body\":\"\"},\"comment\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/comments/68477321\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/pull/534#issuecomment-68477321\",\"issue_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/534\",\"id\":68477321,\"user\":{\"login\":\"bokmadsen\",\"id\":790740,\"avatar_url\":\"https://avatars.githubusercontent.com/u/790740?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bokmadsen\",\"html_url\":\"https://github.com/bokmadsen\",\"followers_url\":\"https://api.github.com/users/bokmadsen/followers\",\"following_url\":\"https://api.github.com/users/bokmadsen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bokmadsen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bokmadsen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bokmadsen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bokmadsen/orgs\",\"repos_url\":\"https://api.github.com/users/bokmadsen/repos\",\"events_url\":\"https://api.github.com/users/bokmadsen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bokmadsen/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:57Z\",\"updated_at\":\"2015-01-01T01:04:57Z\",\"body\":\"Why is this closed? I've a situation where I reload a HybridWebView and it crashed because Element is null\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:57Z\",\"org\":{\"id\":7787062,\"login\":\"XLabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/XLabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?\"}}\n{\"id\":\"2489397981\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":6964047,\"login\":\"TTMTT\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?\"},\"repo\":{\"id\":26731988,\"name\":\"badrsony/icloudin-support-\",\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/8\",\"labels_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/8/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/8/comments\",\"events_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/8/events\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/8\",\"id\":53208937,\"number\":8,\"title\":\"icloudin support \",\"user\":{\"login\":\"badrsony\",\"id\":7895050,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7895050?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/badrsony\",\"html_url\":\"https://github.com/badrsony\",\"followers_url\":\"https://api.github.com/users/badrsony/followers\",\"following_url\":\"https://api.github.com/users/badrsony/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/badrsony/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/badrsony/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/badrsony/subscriptions\",\"organizations_url\":\"https://api.github.com/users/badrsony/orgs\",\"repos_url\":\"https://api.github.com/users/badrsony/repos\",\"events_url\":\"https://api.github.com/users/badrsony/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/badrsony/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":9,\"created_at\":\"2014-12-31T23:58:27Z\",\"updated_at\":\"2015-01-01T01:04:58Z\",\"closed_at\":null,\"body\":\"Originally written by @TTMTT. That we hope for him safery and peace\\r\\n.\\r\\n\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n\\r\\niCL0udin v1.0 -> %98\\r\\n\\r\\nRemaining: %2 testing with some people..\\r\\n\\r\\nLast Method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\n\\r\\niCL0udin v1.0 have this method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/comments/68477322\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/8#issuecomment-68477322\",\"issue_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/8\",\"id\":68477322,\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:04:58Z\",\"updated_at\":\"2015-01-01T01:04:58Z\",\"body\":\"@badrsony \\r\\nexcuse me i open a new discuss for (iCL0udin v1.0):\\r\\n--------------------------------------------------------\\r\\nhttps://github.com/TTMTT/iCL0udin/issues/1\\r\\n--------------------------------------------------------\"}},\"public\":true,\"created_at\":\"2015-01-01T01:04:58Z\"}\n{\"id\":\"2489397982\",\"type\":\"ForkEvent\",\"actor\":{\"id\":10361479,\"login\":\"mahlon15\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mahlon15\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361479?\"},\"repo\":{\"id\":20042152,\"name\":\"nightscout/cgm-remote-monitor\",\"url\":\"https://api.github.com/repos/nightscout/cgm-remote-monitor\"},\"payload\":{\"forkee\":{\"id\":28678265,\"name\":\"cgm-remote-monitor\",\"full_name\":\"mahlon15/cgm-remote-monitor\",\"owner\":{\"login\":\"mahlon15\",\"id\":10361479,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361479?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mahlon15\",\"html_url\":\"https://github.com/mahlon15\",\"followers_url\":\"https://api.github.com/users/mahlon15/followers\",\"following_url\":\"https://api.github.com/users/mahlon15/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/mahlon15/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/mahlon15/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/mahlon15/subscriptions\",\"organizations_url\":\"https://api.github.com/users/mahlon15/orgs\",\"repos_url\":\"https://api.github.com/users/mahlon15/repos\",\"events_url\":\"https://api.github.com/users/mahlon15/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/mahlon15/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/mahlon15/cgm-remote-monitor\",\"description\":\"nightscout web monitor\",\"fork\":true,\"url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor\",\"forks_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/forks\",\"keys_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/teams\",\"hooks_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/hooks\",\"issue_events_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/events\",\"assignees_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/tags\",\"blobs_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/languages\",\"stargazers_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/stargazers\",\"contributors_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/contributors\",\"subscribers_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/subscribers\",\"subscription_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/subscription\",\"commits_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/merges\",\"archive_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/downloads\",\"issues_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/mahlon15/cgm-remote-monitor/releases{/id}\",\"created_at\":\"2015-01-01T01:04:58Z\",\"updated_at\":\"2014-12-30T04:33:26Z\",\"pushed_at\":\"2014-12-31T18:14:04Z\",\"git_url\":\"git://github.com/mahlon15/cgm-remote-monitor.git\",\"ssh_url\":\"git@github.com:mahlon15/cgm-remote-monitor.git\",\"clone_url\":\"https://github.com/mahlon15/cgm-remote-monitor.git\",\"svn_url\":\"https://github.com/mahlon15/cgm-remote-monitor\",\"homepage\":\"\",\"size\":9502,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:04:59Z\",\"org\":{\"id\":7661012,\"login\":\"nightscout\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nightscout\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7661012?\"}}\n{\"id\":\"2489397984\",\"type\":\"PushEvent\",\"actor\":{\"id\":7377949,\"login\":\"rusefi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rusefi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7377949?\"},\"repo\":{\"id\":19047649,\"name\":\"rusefi/rusefi\",\"url\":\"https://api.github.com/repos/rusefi/rusefi\"},\"payload\":{\"push_id\":536753170,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"98582a43442ee88c1f3f1ced20131de107e3fd52\",\"before\":\"6f7687ac1d7ba2adb73b9799dfa2831c9dfbdbf3\",\"commits\":[{\"sha\":\"98582a43442ee88c1f3f1ced20131de107e3fd52\",\"author\":{\"email\":\"667fbe48699c76efd895fe8014d756c77fa79f0f@gmail.com\",\"name\":\"rusEfi\"},\"message\":\"auto-sync\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rusefi/rusefi/commits/98582a43442ee88c1f3f1ced20131de107e3fd52\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:59Z\"}\n{\"id\":\"2489397989\",\"type\":\"CreateEvent\",\"actor\":{\"id\":995241,\"login\":\"ggkuron\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ggkuron\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/995241?\"},\"repo\":{\"id\":12571910,\"name\":\"ggkuron/dotfiles\",\"url\":\"https://api.github.com/repos/ggkuron/dotfiles\"},\"payload\":{\"ref\":\"x230\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"my dotfiles\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:04:59Z\"}\n{\"id\":\"2489397990\",\"type\":\"PushEvent\",\"actor\":{\"id\":4380972,\"login\":\"Camtendo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Camtendo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4380972?\"},\"repo\":{\"id\":27618890,\"name\":\"dguenther/web-overlay\",\"url\":\"https://api.github.com/repos/dguenther/web-overlay\"},\"payload\":{\"push_id\":536753171,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/TwitchImplementation\",\"head\":\"69a6a3723f7bce2d645d3a2a55d0bc8b019abd18\",\"before\":\"db1e6eda79253813da6588cf457cd59a238c8ee2\",\"commits\":[{\"sha\":\"69a6a3723f7bce2d645d3a2a55d0bc8b019abd18\",\"author\":{\"email\":\"aba648f93bc6924c2458971c10c6192365c58b55@gmail.com\",\"name\":\"Cameron Crockrom\"},\"message\":\"Use setTimeout for safety vs. setInterval.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dguenther/web-overlay/commits/69a6a3723f7bce2d645d3a2a55d0bc8b019abd18\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:04:59Z\"}\n{\"id\":\"2489397992\",\"type\":\"WatchEvent\",\"actor\":{\"id\":132444,\"login\":\"PlasticLizard\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/PlasticLizard\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/132444?\"},\"repo\":{\"id\":11225014,\"name\":\"coreos/etcd\",\"url\":\"https://api.github.com/repos/coreos/etcd\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:01Z\",\"org\":{\"id\":3730757,\"login\":\"coreos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/coreos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3730757?\"}}\n{\"id\":\"2489397993\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397288\",\"id\":22397288,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":111,\"original_position\":111,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"?\",\"created_at\":\"2015-01-01T01:05:00Z\",\"updated_at\":\"2015-01-01T01:05:00Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397288\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397288\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397288\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:05:00Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:05:00Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489397994\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536753173,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"089c8f0d2bac3da1fec35e110c876455549123ae\",\"before\":\"f673aef9bc6b115cda55214d362fde50d301ea25\",\"commits\":[{\"sha\":\"089c8f0d2bac3da1fec35e110c876455549123ae\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074298826\\n\\nIim2uaMEmeYuzjr/BB4HDY5aDAhN4tj3tUKkEefUDNY=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/089c8f0d2bac3da1fec35e110c876455549123ae\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:01Z\"}\n{\"id\":\"2489397996\",\"type\":\"PushEvent\",\"actor\":{\"id\":2961036,\"login\":\"codemercenary\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codemercenary\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2961036?\"},\"repo\":{\"id\":22248790,\"name\":\"codemercenary/autowiring\",\"url\":\"https://api.github.com/repos/codemercenary/autowiring\"},\"payload\":{\"push_id\":536753175,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/fix-autoboost\",\"head\":\"8f51be0a57b7463e15546b48ef20e07405c4fbcc\",\"before\":\"36b2b53a58e4725efd0cf1a9823175cd492ad044\",\"commits\":[{\"sha\":\"84a8a258066868e409162319da3c48751b136029\",\"author\":{\"email\":\"f4b726eb4428c94ec1239af9e3b06d73d64a2f9c@gmail.com\",\"name\":\"Jason Lokerson\"},\"message\":\"Fix header guards\",\"distinct\":true,\"url\":\"https://api.github.com/repos/codemercenary/autowiring/commits/84a8a258066868e409162319da3c48751b136029\"},{\"sha\":\"8f51be0a57b7463e15546b48ef20e07405c4fbcc\",\"author\":{\"email\":\"f4b726eb4428c94ec1239af9e3b06d73d64a2f9c@gmail.com\",\"name\":\"Jason Lokerson\"},\"message\":\"Eliminate placeholder injection\",\"distinct\":true,\"url\":\"https://api.github.com/repos/codemercenary/autowiring/commits/8f51be0a57b7463e15546b48ef20e07405c4fbcc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:01Z\"}\n{\"id\":\"2489397998\",\"type\":\"PushEvent\",\"actor\":{\"id\":671161,\"login\":\"aimanparvaiz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aimanparvaiz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/671161?\"},\"repo\":{\"id\":28207723,\"name\":\"akamel001/ssg\",\"url\":\"https://api.github.com/repos/akamel001/ssg\"},\"payload\":{\"push_id\":536753176,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/redis\",\"head\":\"97a09d2a39aa88d8a05080f028027071a2aedc77\",\"before\":\"4be94c06212e79117a33df49c03ffd75742bf5ef\",\"commits\":[{\"sha\":\"97a09d2a39aa88d8a05080f028027071a2aedc77\",\"author\":{\"email\":\"a36175434507c8d1ff8c192e77af99c0d5cb0179@gmail.com\",\"name\":\"aimanparvaiz\"},\"message\":\"Reverting changes to data_dplit not returning anything\",\"distinct\":true,\"url\":\"https://api.github.com/repos/akamel001/ssg/commits/97a09d2a39aa88d8a05080f028027071a2aedc77\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:01Z\"}\n{\"id\":\"2489397999\",\"type\":\"PushEvent\",\"actor\":{\"id\":2851221,\"login\":\"alkass\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alkass\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2851221?\"},\"repo\":{\"id\":28678233,\"name\":\"alkass/seQre\",\"url\":\"https://api.github.com/repos/alkass/seQre\"},\"payload\":{\"push_id\":536753177,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c205a49edb938d205047779cb9492e125de331ca\",\"before\":\"3e46496fed34162743f34c34d45cca60df88ba6f\",\"commits\":[{\"sha\":\"c205a49edb938d205047779cb9492e125de331ca\",\"author\":{\"email\":\"a11a5b354f8e573649b5660989b915a86618a1f4@yahoo.com\",\"name\":\"Fadi Hanna Al-Kass\"},\"message\":\"first commit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alkass/seQre/commits/c205a49edb938d205047779cb9492e125de331ca\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:01Z\"}\n{\"id\":\"2489398007\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1265899,\"login\":\"lynas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lynas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1265899?\"},\"repo\":{\"id\":28678267,\"name\":\"lynas/testforcom\",\"url\":\"https://api.github.com/repos/lynas/testforcom\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:02Z\"}\n{\"id\":\"2489398010\",\"type\":\"PushEvent\",\"actor\":{\"id\":1681249,\"login\":\"Toeler\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Toeler\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1681249?\"},\"repo\":{\"id\":28678136,\"name\":\"Toeler/Handmade-Hero\",\"url\":\"https://api.github.com/repos/Toeler/Handmade-Hero\"},\"payload\":{\"push_id\":536753180,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"294751cfbc4128f170d1f29059758ddd33fc0215\",\"before\":\"408100c353fa4aa5b211754d5cfc1d83b8c359a3\",\"commits\":[{\"sha\":\"294751cfbc4128f170d1f29059758ddd33fc0215\",\"author\":{\"email\":\"1567efe08cf0fe56c7f6c76e22ebfc0c7eb0ca87@gmail.com\",\"name\":\"Toeler\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Toeler/Handmade-Hero/commits/294751cfbc4128f170d1f29059758ddd33fc0215\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:02Z\"}\n{\"id\":\"2489398012\",\"type\":\"PushEvent\",\"actor\":{\"id\":6142965,\"login\":\"toshibo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/toshibo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6142965?\"},\"repo\":{\"id\":28669825,\"name\":\"toshibo/first_app\",\"url\":\"https://api.github.com/repos/toshibo/first_app\"},\"payload\":{\"push_id\":536753181,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"69adeede51123c81ea048177e88a7f0e19cd1dc5\",\"before\":\"b3289500062ee995801e6d33f9ea834648adbafa\",\"commits\":[{\"sha\":\"69adeede51123c81ea048177e88a7f0e19cd1dc5\",\"author\":{\"email\":\"4a4dd8a7d0feb6dd182beabd5d78e33952f10637@gmail.com\",\"name\":\"toshibo\"},\"message\":\"Improve the README file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/toshibo/first_app/commits/69adeede51123c81ea048177e88a7f0e19cd1dc5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:03Z\"}\n{\"id\":\"2489398017\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":69068,\"login\":\"someara\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/someara\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/69068?\"},\"repo\":{\"id\":28657951,\"name\":\"hypomonk/slacker-packer\",\"url\":\"https://api.github.com/repos/hypomonk/slacker-packer\"},\"payload\":{\"action\":\"opened\",\"number\":2,\"pull_request\":{\"url\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2\",\"id\":26739453,\"html_url\":\"https://github.com/hypomonk/slacker-packer/pull/2\",\"diff_url\":\"https://github.com/hypomonk/slacker-packer/pull/2.diff\",\"patch_url\":\"https://github.com/hypomonk/slacker-packer/pull/2.patch\",\"issue_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/2\",\"number\":2,\"state\":\"open\",\"locked\":false,\"title\":\"adding .gitignore and --no-check-certificates\",\"user\":{\"login\":\"someara\",\"id\":69068,\"avatar_url\":\"https://avatars.githubusercontent.com/u/69068?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/someara\",\"html_url\":\"https://github.com/someara\",\"followers_url\":\"https://api.github.com/users/someara/followers\",\"following_url\":\"https://api.github.com/users/someara/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/someara/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/someara/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/someara/subscriptions\",\"organizations_url\":\"https://api.github.com/users/someara/orgs\",\"repos_url\":\"https://api.github.com/users/someara/repos\",\"events_url\":\"https://api.github.com/users/someara/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/someara/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:05:04Z\",\"updated_at\":\"2015-01-01T01:05:04Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2/commits\",\"review_comments_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2/comments\",\"review_comment_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/2/comments\",\"statuses_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/statuses/a167b8d1405621a26333fda5325d389eeccc05ff\",\"head\":{\"label\":\"someara:master\",\"ref\":\"master\",\"sha\":\"a167b8d1405621a26333fda5325d389eeccc05ff\",\"user\":{\"login\":\"someara\",\"id\":69068,\"avatar_url\":\"https://avatars.githubusercontent.com/u/69068?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/someara\",\"html_url\":\"https://github.com/someara\",\"followers_url\":\"https://api.github.com/users/someara/followers\",\"following_url\":\"https://api.github.com/users/someara/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/someara/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/someara/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/someara/subscriptions\",\"organizations_url\":\"https://api.github.com/users/someara/orgs\",\"repos_url\":\"https://api.github.com/users/someara/repos\",\"events_url\":\"https://api.github.com/users/someara/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/someara/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28658028,\"name\":\"slacker-packer\",\"full_name\":\"someara/slacker-packer\",\"owner\":{\"login\":\"someara\",\"id\":69068,\"avatar_url\":\"https://avatars.githubusercontent.com/u/69068?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/someara\",\"html_url\":\"https://github.com/someara\",\"followers_url\":\"https://api.github.com/users/someara/followers\",\"following_url\":\"https://api.github.com/users/someara/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/someara/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/someara/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/someara/subscriptions\",\"organizations_url\":\"https://api.github.com/users/someara/orgs\",\"repos_url\":\"https://api.github.com/users/someara/repos\",\"events_url\":\"https://api.github.com/users/someara/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/someara/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/someara/slacker-packer\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/someara/slacker-packer\",\"forks_url\":\"https://api.github.com/repos/someara/slacker-packer/forks\",\"keys_url\":\"https://api.github.com/repos/someara/slacker-packer/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/someara/slacker-packer/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/someara/slacker-packer/teams\",\"hooks_url\":\"https://api.github.com/repos/someara/slacker-packer/hooks\",\"issue_events_url\":\"https://api.github.com/repos/someara/slacker-packer/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/someara/slacker-packer/events\",\"assignees_url\":\"https://api.github.com/repos/someara/slacker-packer/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/someara/slacker-packer/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/someara/slacker-packer/tags\",\"blobs_url\":\"https://api.github.com/repos/someara/slacker-packer/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/someara/slacker-packer/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/someara/slacker-packer/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/someara/slacker-packer/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/someara/slacker-packer/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/someara/slacker-packer/languages\",\"stargazers_url\":\"https://api.github.com/repos/someara/slacker-packer/stargazers\",\"contributors_url\":\"https://api.github.com/repos/someara/slacker-packer/contributors\",\"subscribers_url\":\"https://api.github.com/repos/someara/slacker-packer/subscribers\",\"subscription_url\":\"https://api.github.com/repos/someara/slacker-packer/subscription\",\"commits_url\":\"https://api.github.com/repos/someara/slacker-packer/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/someara/slacker-packer/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/someara/slacker-packer/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/someara/slacker-packer/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/someara/slacker-packer/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/someara/slacker-packer/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/someara/slacker-packer/merges\",\"archive_url\":\"https://api.github.com/repos/someara/slacker-packer/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/someara/slacker-packer/downloads\",\"issues_url\":\"https://api.github.com/repos/someara/slacker-packer/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/someara/slacker-packer/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/someara/slacker-packer/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/someara/slacker-packer/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/someara/slacker-packer/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/someara/slacker-packer/releases{/id}\",\"created_at\":\"2014-12-31T06:18:22Z\",\"updated_at\":\"2015-01-01T01:04:37Z\",\"pushed_at\":\"2015-01-01T01:04:37Z\",\"git_url\":\"git://github.com/someara/slacker-packer.git\",\"ssh_url\":\"git@github.com:someara/slacker-packer.git\",\"clone_url\":\"https://github.com/someara/slacker-packer.git\",\"svn_url\":\"https://github.com/someara/slacker-packer\",\"homepage\":null,\"size\":0,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Shell\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"hypomonk:master\",\"ref\":\"master\",\"sha\":\"09f12c2d299252470229258e6f4a98f439bd025e\",\"user\":{\"login\":\"hypomonk\",\"id\":6394422,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6394422?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hypomonk\",\"html_url\":\"https://github.com/hypomonk\",\"followers_url\":\"https://api.github.com/users/hypomonk/followers\",\"following_url\":\"https://api.github.com/users/hypomonk/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/hypomonk/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/hypomonk/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/hypomonk/subscriptions\",\"organizations_url\":\"https://api.github.com/users/hypomonk/orgs\",\"repos_url\":\"https://api.github.com/users/hypomonk/repos\",\"events_url\":\"https://api.github.com/users/hypomonk/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/hypomonk/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28657951,\"name\":\"slacker-packer\",\"full_name\":\"hypomonk/slacker-packer\",\"owner\":{\"login\":\"hypomonk\",\"id\":6394422,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6394422?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hypomonk\",\"html_url\":\"https://github.com/hypomonk\",\"followers_url\":\"https://api.github.com/users/hypomonk/followers\",\"following_url\":\"https://api.github.com/users/hypomonk/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/hypomonk/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/hypomonk/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/hypomonk/subscriptions\",\"organizations_url\":\"https://api.github.com/users/hypomonk/orgs\",\"repos_url\":\"https://api.github.com/users/hypomonk/repos\",\"events_url\":\"https://api.github.com/users/hypomonk/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/hypomonk/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/hypomonk/slacker-packer\",\"description\":\"\",\"fork\":false,\"url\":\"https://api.github.com/repos/hypomonk/slacker-packer\",\"forks_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/forks\",\"keys_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/teams\",\"hooks_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/hooks\",\"issue_events_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/events\",\"assignees_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/tags\",\"blobs_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/languages\",\"stargazers_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/stargazers\",\"contributors_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/contributors\",\"subscribers_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/subscribers\",\"subscription_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/subscription\",\"commits_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/merges\",\"archive_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/downloads\",\"issues_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/hypomonk/slacker-packer/releases{/id}\",\"created_at\":\"2014-12-31T06:14:56Z\",\"updated_at\":\"2014-12-31T22:20:37Z\",\"pushed_at\":\"2014-12-31T22:20:36Z\",\"git_url\":\"git://github.com/hypomonk/slacker-packer.git\",\"ssh_url\":\"git@github.com:hypomonk/slacker-packer.git\",\"clone_url\":\"https://github.com/hypomonk/slacker-packer.git\",\"svn_url\":\"https://github.com/hypomonk/slacker-packer\",\"homepage\":null,\"size\":0,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"Shell\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":1,\"mirror_url\":null,\"open_issues_count\":2,\"forks\":1,\"open_issues\":2,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2\"},\"html\":{\"href\":\"https://github.com/hypomonk/slacker-packer/pull/2\"},\"issue\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/2\"},\"comments\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/issues/2/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/pulls/2/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/hypomonk/slacker-packer/statuses/a167b8d1405621a26333fda5325d389eeccc05ff\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":18,\"deletions\":1,\"changed_files\":2}},\"public\":true,\"created_at\":\"2015-01-01T01:05:04Z\"}\n{\"id\":\"2489398019\",\"type\":\"PushEvent\",\"actor\":{\"id\":4379694,\"login\":\"moongato\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/moongato\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4379694?\"},\"repo\":{\"id\":11769015,\"name\":\"moongato/android_frameworks_base\",\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base\"},\"payload\":{\"push_id\":536753182,\"size\":6,\"distinct_size\":6,\"ref\":\"refs/heads/lp50x\",\"head\":\"2a724ce90a362e18dc10cd86f99760c5146c1e6a\",\"before\":\"73c3f8da9962437774cec8442e0da8d431c28fa8\",\"commits\":[{\"sha\":\"09ed328c580340d67492f81d855f8e2fe5834bd1\",\"author\":{\"email\":\"97bd659605de2dc1baed2f2e8ef4e483c88d27f8@gmail.com\",\"name\":\"Austin T. Conn\"},\"message\":\"volume rocker music controls\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/09ed328c580340d67492f81d855f8e2fe5834bd1\"},{\"sha\":\"b3c8f58fb81a604be3f1fd2913e005afe300e296\",\"author\":{\"email\":\"9defb1d2166fe02be9724618510318ef57e74e91@gmail.com\",\"name\":\"Michael Bestas\"},\"message\":\"Fix volume rocker music controls and wake up\\n\\n- Forward port code from cm-11.0 and adjust for 5.0\\n- Fix not being able to adjust volume when music control is on\\n- Disable screen off volume/music control when wake key is enabled\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/b3c8f58fb81a604be3f1fd2913e005afe300e296\"},{\"sha\":\"8040e4188be23da640d89fc829e33651826a48c0\",\"author\":{\"email\":\"238a1843d81dd7fbe80b5c1b99515c4ba8c94d0d@cyngn.com\",\"name\":\"Roman Birg\"},\"message\":\"status bar: improve brightness slider behavior\\n\\nMimic the brightness slider behavior in the statusbar.\\nThis adds logic to make the statusbar slider also work with automatic brightness mode enabled and it will instead adjust the temporary automatic brightness overrride.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/8040e4188be23da640d89fc829e33651826a48c0\"},{\"sha\":\"58cadbda6cf16389eb849e5d9ed39eafb352d18a\",\"author\":{\"email\":\"419b20914ad8ee7cdbbc7674d01c492d53cd267e@gmail.com\",\"name\":\"Pawit Pornkitprasan\"},\"message\":\"status bar brightness: store value as int\\n\\nNon-automatic brightness value is stored as int, not float.\\n\\nSymptom: adjust the brightness in the status bar, the brightness slider in the notification bar will always be set to full\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/58cadbda6cf16389eb849e5d9ed39eafb352d18a\"},{\"sha\":\"e0fb35b9ed52d05ce74b1c1833e20af6cd649f3c\",\"author\":{\"email\":\"1e520a49e026effdcbd6ab603708edecd93fc284@gmail.com\",\"name\":\"Clyde Tan\"},\"message\":\"Keep quiet when volume keys are used to wake up device\\n\\n- Userspace will make a 'beep' with it receives a key up, so  consume that event as well.\\n- Removed wake key check in music control code as it will already be disabled here.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/e0fb35b9ed52d05ce74b1c1833e20af6cd649f3c\"},{\"sha\":\"2a724ce90a362e18dc10cd86f99760c5146c1e6a\",\"author\":{\"email\":\"22e0f38e0fc64da9129ff9b9ef030b39415294a1@ubuntu\",\"name\":\"moongato\"},\"message\":\"Merge remote-tracking branch 'upstream/lollipop-ras-mr1' into lp50x\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/2a724ce90a362e18dc10cd86f99760c5146c1e6a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:04Z\"}\n{\"id\":\"2489398027\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1449374,\"login\":\"ohyeah521\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ohyeah521\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1449374?\"},\"repo\":{\"id\":28629678,\"name\":\"techbliss/ADB_Helper_For_ida_Pro\",\"url\":\"https://api.github.com/repos/techbliss/ADB_Helper_For_ida_Pro\"},\"payload\":{\"forkee\":{\"id\":28678268,\"name\":\"ADB_Helper_For_ida_Pro\",\"full_name\":\"ohyeah521/ADB_Helper_For_ida_Pro\",\"owner\":{\"login\":\"ohyeah521\",\"id\":1449374,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1449374?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ohyeah521\",\"html_url\":\"https://github.com/ohyeah521\",\"followers_url\":\"https://api.github.com/users/ohyeah521/followers\",\"following_url\":\"https://api.github.com/users/ohyeah521/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ohyeah521/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ohyeah521/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ohyeah521/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ohyeah521/orgs\",\"repos_url\":\"https://api.github.com/users/ohyeah521/repos\",\"events_url\":\"https://api.github.com/users/ohyeah521/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ohyeah521/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/ohyeah521/ADB_Helper_For_ida_Pro\",\"description\":\"Helps you debugging Android in Ida pro\",\"fork\":true,\"url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro\",\"forks_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/forks\",\"keys_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/teams\",\"hooks_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/hooks\",\"issue_events_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/events\",\"assignees_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/tags\",\"blobs_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/languages\",\"stargazers_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/stargazers\",\"contributors_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/contributors\",\"subscribers_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/subscribers\",\"subscription_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/subscription\",\"commits_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/merges\",\"archive_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/downloads\",\"issues_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/ohyeah521/ADB_Helper_For_ida_Pro/releases{/id}\",\"created_at\":\"2015-01-01T01:05:06Z\",\"updated_at\":\"2014-12-31T18:54:05Z\",\"pushed_at\":\"2014-12-30T12:25:12Z\",\"git_url\":\"git://github.com/ohyeah521/ADB_Helper_For_ida_Pro.git\",\"ssh_url\":\"git@github.com:ohyeah521/ADB_Helper_For_ida_Pro.git\",\"clone_url\":\"https://github.com/ohyeah521/ADB_Helper_For_ida_Pro.git\",\"svn_url\":\"https://github.com/ohyeah521/ADB_Helper_For_ida_Pro\",\"homepage\":null,\"size\":0,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\"}\n{\"id\":\"2489398028\",\"type\":\"CreateEvent\",\"actor\":{\"id\":9505729,\"login\":\"alishadot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alishadot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9505729?\"},\"repo\":{\"id\":28678253,\"name\":\"alishadot/keepcloud\",\"url\":\"https://api.github.com/repos/alishadot/keepcloud\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\"}\n{\"id\":\"2489398030\",\"type\":\"PushEvent\",\"actor\":{\"id\":506010,\"login\":\"gabeshaughnessy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gabeshaughnessy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/506010?\"},\"repo\":{\"id\":13913264,\"name\":\"gabeshaughnessy/augmentedart\",\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart\"},\"payload\":{\"push_id\":536753186,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dungeon-hacker\",\"head\":\"f1d6efa16b35cfe98b4ee3e54c3f333f87612fa1\",\"before\":\"070da10a642c004ef5bc428b8ce1bf88e410e7b4\",\"commits\":[{\"sha\":\"f1d6efa16b35cfe98b4ee3e54c3f333f87612fa1\",\"author\":{\"email\":\"a2b2bb6e7f1b10ac88b326d5c10e33af6a8546bc@gmail.com\",\"name\":\"gabeshaughnessy\"},\"message\":\"todo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart/commits/f1d6efa16b35cfe98b4ee3e54c3f333f87612fa1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\"}\n{\"id\":\"2489398031\",\"type\":\"PushEvent\",\"actor\":{\"id\":489000,\"login\":\"kwrobot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kwrobot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/489000?\"},\"repo\":{\"id\":537699,\"name\":\"Kitware/CMake\",\"url\":\"https://api.github.com/repos/Kitware/CMake\"},\"payload\":{\"push_id\":536753184,\"size\":1,\"distinct_size\":0,\"ref\":\"refs/heads/nightly-master\",\"head\":\"c2445d3dfd837cd2e372061f5545054317be110a\",\"before\":\"3b62cb7244716e0bd486d7023bec15d922572ad1\",\"commits\":[{\"sha\":\"c2445d3dfd837cd2e372061f5545054317be110a\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/c2445d3dfd837cd2e372061f5545054317be110a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\",\"org\":{\"id\":87549,\"login\":\"Kitware\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Kitware\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/87549?\"}}\n{\"id\":\"2489398032\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":3487141,\"login\":\"graig\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/graig\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3487141?\"},\"repo\":{\"id\":919161,\"name\":\"arduino/Arduino\",\"url\":\"https://api.github.com/repos/arduino/Arduino\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/arduino/Arduino/issues/2525\",\"labels_url\":\"https://api.github.com/repos/arduino/Arduino/issues/2525/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/arduino/Arduino/issues/2525/comments\",\"events_url\":\"https://api.github.com/repos/arduino/Arduino/issues/2525/events\",\"html_url\":\"https://github.com/arduino/Arduino/issues/2525\",\"id\":53210273,\"number\":2525,\"title\":\"broken link - Yun\",\"user\":{\"login\":\"graig\",\"id\":3487141,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3487141?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/graig\",\"html_url\":\"https://github.com/graig\",\"followers_url\":\"https://api.github.com/users/graig/followers\",\"following_url\":\"https://api.github.com/users/graig/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/graig/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/graig/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/graig/subscriptions\",\"organizations_url\":\"https://api.github.com/users/graig/orgs\",\"repos_url\":\"https://api.github.com/users/graig/repos\",\"events_url\":\"https://api.github.com/users/graig/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/graig/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:05:05Z\",\"updated_at\":\"2015-01-01T01:05:05Z\",\"closed_at\":null,\"body\":\"The link for the Yun schematic works but the file (arduino-Yun-DSN.zip) is named wrong. When opened, it actually contains the schematic and board for the Uno, not the Yun.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\",\"org\":{\"id\":379109,\"login\":\"arduino\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/arduino\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/379109?\"}}\n{\"id\":\"2489398034\",\"type\":\"PushEvent\",\"actor\":{\"id\":489000,\"login\":\"kwrobot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kwrobot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/489000?\"},\"repo\":{\"id\":537699,\"name\":\"Kitware/CMake\",\"url\":\"https://api.github.com/repos/Kitware/CMake\"},\"payload\":{\"push_id\":536753187,\"size\":24,\"distinct_size\":0,\"ref\":\"refs/heads/nightly\",\"head\":\"943791ec6b6166448216d2c12d355be291910948\",\"before\":\"ce73c53c7a64d31432739f434f222cfa904c1aec\",\"commits\":[{\"sha\":\"05105d5c13f6662907cfb63fb6723f9f34f4558d\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/05105d5c13f6662907cfb63fb6723f9f34f4558d\"},{\"sha\":\"2b28e07c02d9f13867d0f518b8341e77f006faa4\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/2b28e07c02d9f13867d0f518b8341e77f006faa4\"},{\"sha\":\"028cdfb3deaddef510537bbeb1a4202924c1a80b\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/028cdfb3deaddef510537bbeb1a4202924c1a80b\"},{\"sha\":\"8e6114017873e50817c6961ffcedb3c52a2ed1f1\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/8e6114017873e50817c6961ffcedb3c52a2ed1f1\"},{\"sha\":\"ca3d08e6b598a7f7d46b3125a0d7ddddf8be8dcd\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/ca3d08e6b598a7f7d46b3125a0d7ddddf8be8dcd\"},{\"sha\":\"3b62cb7244716e0bd486d7023bec15d922572ad1\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/3b62cb7244716e0bd486d7023bec15d922572ad1\"},{\"sha\":\"c2445d3dfd837cd2e372061f5545054317be110a\",\"author\":{\"email\":\"29e2a43288125ec45952b67b65b7fb7ef6ca7262@kitware.com\",\"name\":\"Kitware Robot\"},\"message\":\"CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/c2445d3dfd837cd2e372061f5545054317be110a\"},{\"sha\":\"5e8b4a718169837db825538561ddfd85cf62be43\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"SolarisStudio: Use alternative standard library to build CMake.\\n\\nSolarisStudio ships a very old RogueWave standard library\\nimplementation (libCstd) and uses it by default for backward compatibility.\\nThe macros defined when building the system libCstd need to be the same as\\nthe macros defined when using it for binary compatibility reasons etc.  The\\nSolarisStudio compiler driver adds macros such as _RWSTD_NO_MEMBER_TEMPLATES and\\n_RWSTD_NO_CLASS_PARTIAL_SPEC etc. These macros disable certain APIs in the\\nstandard library headers.\\n\\nAlthough the compiler supports the features 'member templates' and 'partial\\ntemplate specialization', the standard library does not provide APIs which\\nrely on those features.  This means that std::vector::insert in libCStd does\\nnot accept a pair of iterators from a different type of container, because\\nthat requires member templates, and reverse_iterator<const T> can not\\nbe constructed from a reverse_iterator<T> because that requires partial\\nspecialization (or at least the _RWSTD_NO_CLASS_PARTIAL_SPEC define) and\\nmember templates.\\n\\nThis causes many problems while building CMake using SolarisStudio, which\\nhave not been well understood until now.  The problems are usually\\nattributed to compiler limitations, while actually the problem is in\\nthe standard library, as in commit v3.0.0-rc1~99^2~1 (Help: Document non-use\\nof std::set::insert., 2014-01-24) and commit 107dcac3 (Fix compilation with\\nthe Oracle / Sun compiler (#15318), 2014-12-12).\\n\\nSolarisStudio 12.3 and earlier also ships a version of stlport which may be\\nused instead of libCstd by specifying -library=stlport4\\n\\n https://docs.oracle.com/cd/E18659_01/html/821-1383/bkakg.html\\n\\nSolarisStudio 12.4 ships a version of libstdc++ from GCC 4.8.2 which may be\\nused by specifying -std=c++03 or -std=c++11 etc\\n\\n http://docs.oracle.com/cd/E37069_01/html/E37075/bkamw.html#OSSCPgnaof\\n\\nUse these more-capable standard library implementations when building cmake.\\nThis will allow more use of 'normal' C++ (such as std::vector::insert), and cause\\nfewer surprises resulting from dashboards using SolarisStudio.\\n\\nBecause cmake is not a library linked against by 3rd parties and does not have\\nexternal dependencies, issues related to mixing code using libCStd and libstdc++\\ndo not apply.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/5e8b4a718169837db825538561ddfd85cf62be43\"},{\"sha\":\"be77679552dbda01d62b03a85de05ad7e7684862\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Revert \\\"Misc. fixes for the Oracle / Sun compiler.\\\"\\n\\nThis reverts commit 97b65f8156734db2adc367b27c822a5fe332d740.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/be77679552dbda01d62b03a85de05ad7e7684862\"},{\"sha\":\"b162e9871e8fabacf6f565ad1ced05a6d234e25b\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Use insert instead of a loop in some cases.\\n\\nLimit this change to inserting into a vector from a vector.\\n\\nA follow up change can use insert for inserting into a set.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/b162e9871e8fabacf6f565ad1ced05a6d234e25b\"},{\"sha\":\"57969aebfc5bc3589972b497fc5deb30e292230f\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Use two-iterator std::set::insert where appropriate.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/57969aebfc5bc3589972b497fc5deb30e292230f\"},{\"sha\":\"6c02a443b92ba189288cbe9f01589e00cabfb994\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Help: Remove documented restriction on insert APIs.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/6c02a443b92ba189288cbe9f01589e00cabfb994\"},{\"sha\":\"59aa8d878e69fd694e44af3778e87615804ad53e\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Help: Remove documented restriction on find in conditions.\\n\\nThe necessary conversion is supported by all CMake host compilers.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/59aa8d878e69fd694e44af3778e87615804ad53e\"},{\"sha\":\"7403759846b9bd34bb425f5e90f6e367b80d1d58\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Help: Remove documented restriction on template use.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/7403759846b9bd34bb425f5e90f6e367b80d1d58\"},{\"sha\":\"2e1ea034a804cd1ec61ba5b9817452d49b6d0830\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Merge topic 'sun-better-stdlib' into next\\n\\n74037598 Help: Remove documented restriction on template use.\\n59aa8d87 Help: Remove documented restriction on find in conditions.\\n6c02a443 Help: Remove documented restriction on insert APIs.\\n57969aeb Use two-iterator std::set::insert where appropriate.\\nb162e987 Use insert instead of a loop in some cases.\\nbe776795 Revert \\\"Misc. fixes for the Oracle / Sun compiler.\\\"\\n5e8b4a71 SolarisStudio: Use alternative standard library to build CMake.\\n3b62cb72 CMake Nightly Date Stamp\\nca3d08e6 CMake Nightly Date Stamp\\n8e611401 CMake Nightly Date Stamp\\n028cdfb3 CMake Nightly Date Stamp\\n2b28e07c CMake Nightly Date Stamp\\n05105d5c CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/2e1ea034a804cd1ec61ba5b9817452d49b6d0830\"},{\"sha\":\"29b4f5420922a06cafeaf07c13b5d781790e8646\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"RunCMake: Expect empty output by default.\\n\\nExpect tests to specify stderr content if it is present.\\n\\nFix the CMP0019 test, which has only been testing the WARN status\\nuntil now.  Specify in the CommandLine and FPHSA tests that content\\nis at least one character.\\n\\nSet policies in the Language and CheckModules tests, which have empty\\ntest output, modulo unrelated policies on some platforms.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/29b4f5420922a06cafeaf07c13b5d781790e8646\"},{\"sha\":\"ff99adb80d59dcbe8cb0d9dc12db657a2f58be7e\",\"author\":{\"email\":\"fbd1bb53ae91c7d21c5feb88facb5fbef8543d8e@gmail.com\",\"name\":\"Stephen Kelly\"},\"message\":\"Merge topic 'RunCMake-stderr-default' into next\\n\\n29b4f542 RunCMake: Expect empty output by default.\\nc2445d3d CMake Nightly Date Stamp\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/ff99adb80d59dcbe8cb0d9dc12db657a2f58be7e\"},{\"sha\":\"2c528f860ff8ea458fbf8ab6c85908aeeaa749e0\",\"author\":{\"email\":\"88a2dde5abcebb5c32bd0a2b4aff36eeea918955@kitware.com\",\"name\":\"Robert Maynard\"},\"message\":\"CompileFeatures: NonValidTarget1 now handles not have cxx_final.\\n\\nPreviously we expanded HAVE_FINAL to determine what the copied_file number\\nwould be, but when we don't have cxx_final than HAVE_FINAL is not defined.\\nWhat we really want is to use expected_result.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/2c528f860ff8ea458fbf8ab6c85908aeeaa749e0\"},{\"sha\":\"b97c86b5009307e6b089bcc2ddef55d77918f765\",\"author\":{\"email\":\"88a2dde5abcebb5c32bd0a2b4aff36eeea918955@kitware.com\",\"name\":\"Robert Maynard\"},\"message\":\"CompileFeatures: Support compilers that don't have version modess.\\n\\ncompilers such as MSVC have no explicit flags to enable C++11 mode,\\nit just is always on. So only run the link tests with compilers that require\\na flag to specify the language version.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/b97c86b5009307e6b089bcc2ddef55d77918f765\"},{\"sha\":\"8ad33e01d2ed27c9ac136fd19516a21449b4c87b\",\"author\":{\"email\":\"88a2dde5abcebb5c32bd0a2b4aff36eeea918955@kitware.com\",\"name\":\"Robert Maynard\"},\"message\":\"Merge topic 'feature_record_msvc' into next\\n\\nb97c86b5 CompileFeatures: Support compilers that don't have version modess.\\n2c528f86 CompileFeatures: NonValidTarget1 now handles not have cxx_final.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/CMake/commits/8ad33e01d2ed27c9ac136fd19516a21449b4c87b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\",\"org\":{\"id\":87549,\"login\":\"Kitware\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Kitware\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/87549?\"}}\n{\"id\":\"2489398039\",\"type\":\"PushEvent\",\"actor\":{\"id\":452786,\"login\":\"alrieckert\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alrieckert\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/452786?\"},\"repo\":{\"id\":6153802,\"name\":\"alrieckert/freepascal\",\"url\":\"https://api.github.com/repos/alrieckert/freepascal\"},\"payload\":{\"push_id\":536753192,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"c119967bbbcfba0d00c6bfcf9d4e805272bbc8dd\",\"before\":\"7688ca531943068f1b199008e7575bfb075729f1\",\"commits\":[{\"sha\":\"8596d86916e4754dde940d6934430173218863b9\",\"author\":{\"email\":\"3829486b93ec44395f0b980424bae9b6fb07b7bc@3ad0048d-3df7-0310-abae-a5850022a9f2\",\"name\":\"marco\"},\"message\":\" * system unit additions from mantis #27206. Exports some dynarray related RTTI functions.\\n\\ngit-svn-id: http://svn.freepascal.org/svn/fpc/trunk@29364 3ad0048d-3df7-0310-abae-a5850022a9f2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alrieckert/freepascal/commits/8596d86916e4754dde940d6934430173218863b9\"},{\"sha\":\"c119967bbbcfba0d00c6bfcf9d4e805272bbc8dd\",\"author\":{\"email\":\"3829486b93ec44395f0b980424bae9b6fb07b7bc@3ad0048d-3df7-0310-abae-a5850022a9f2\",\"name\":\"marco\"},\"message\":\" * Fix for #27228\\n\\n\\ngit-svn-id: http://svn.freepascal.org/svn/fpc/trunk@29365 3ad0048d-3df7-0310-abae-a5850022a9f2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alrieckert/freepascal/commits/c119967bbbcfba0d00c6bfcf9d4e805272bbc8dd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:06Z\"}\n{\"id\":\"2489398041\",\"type\":\"PushEvent\",\"actor\":{\"id\":6158630,\"login\":\"greatfire\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/greatfire\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6158630?\"},\"repo\":{\"id\":18126008,\"name\":\"greatfire/z\",\"url\":\"https://api.github.com/repos/greatfire/z\"},\"payload\":{\"push_id\":536753193,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"67eeb5f55504c0317a0e372e38eed9df2a97370d\",\"before\":\"1c095ccf0c45826798d494d477eef16082e2919f\",\"commits\":[{\"sha\":\"67eeb5f55504c0317a0e372e38eed9df2a97370d\",\"author\":{\"email\":\"24bf68e341ce0fbd9259a5d51feed79682ea4eba@greatfire.org\",\"name\":\"Ubuntu\"},\"message\":\"a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/greatfire/z/commits/67eeb5f55504c0317a0e372e38eed9df2a97370d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:07Z\"}\n{\"id\":\"2489398043\",\"type\":\"PushEvent\",\"actor\":{\"id\":6158630,\"login\":\"greatfire\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/greatfire\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6158630?\"},\"repo\":{\"id\":15100395,\"name\":\"greatfire/wiki\",\"url\":\"https://api.github.com/repos/greatfire/wiki\"},\"payload\":{\"push_id\":536753194,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7ce647729785e7cfee67c96e185b8fa8df693623\",\"before\":\"ba5bb9726f655f8acd29ecb3f89f1ae55305e680\",\"commits\":[{\"sha\":\"7ce647729785e7cfee67c96e185b8fa8df693623\",\"author\":{\"email\":\"24bf68e341ce0fbd9259a5d51feed79682ea4eba@greatfire.org\",\"name\":\"Ubuntu\"},\"message\":\"a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/greatfire/wiki/commits/7ce647729785e7cfee67c96e185b8fa8df693623\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:07Z\"}\n{\"id\":\"2489398045\",\"type\":\"PushEvent\",\"actor\":{\"id\":6895040,\"login\":\"codertradergambler\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codertradergambler\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6895040?\"},\"repo\":{\"id\":18620619,\"name\":\"chancecoin/chancecoinj\",\"url\":\"https://api.github.com/repos/chancecoin/chancecoinj\"},\"payload\":{\"push_id\":536753196,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d9886bb6d61f6f5e1e33b421db90e4c50feabbe2\",\"before\":\"d4ec65797e8faf2aac84c02007d2ebeb342babd3\",\"commits\":[{\"sha\":\"d9886bb6d61f6f5e1e33b421db90e4c50feabbe2\",\"author\":{\"email\":\"0ccf54d51d1a5240ad356feb30dfa4d1749f8844@gmail.com\",\"name\":\"TraderCoderGambler\"},\"message\":\"auto-update balances\",\"distinct\":true,\"url\":\"https://api.github.com/repos/chancecoin/chancecoinj/commits/d9886bb6d61f6f5e1e33b421db90e4c50feabbe2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:08Z\"}\n{\"id\":\"2489398046\",\"type\":\"ForkEvent\",\"actor\":{\"id\":1129530,\"login\":\"jimulabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jimulabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1129530?\"},\"repo\":{\"id\":14393731,\"name\":\"JakeWharton/u2020\",\"url\":\"https://api.github.com/repos/JakeWharton/u2020\"},\"payload\":{\"forkee\":{\"id\":28678269,\"name\":\"u2020\",\"full_name\":\"jimulabs/u2020\",\"owner\":{\"login\":\"jimulabs\",\"id\":1129530,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1129530?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jimulabs\",\"html_url\":\"https://github.com/jimulabs\",\"followers_url\":\"https://api.github.com/users/jimulabs/followers\",\"following_url\":\"https://api.github.com/users/jimulabs/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jimulabs/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jimulabs/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jimulabs/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jimulabs/orgs\",\"repos_url\":\"https://api.github.com/users/jimulabs/repos\",\"events_url\":\"https://api.github.com/users/jimulabs/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jimulabs/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/jimulabs/u2020\",\"description\":\"A sample Android app which showcases advanced usage of Dagger among other open source libraries.\",\"fork\":true,\"url\":\"https://api.github.com/repos/jimulabs/u2020\",\"forks_url\":\"https://api.github.com/repos/jimulabs/u2020/forks\",\"keys_url\":\"https://api.github.com/repos/jimulabs/u2020/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/jimulabs/u2020/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/jimulabs/u2020/teams\",\"hooks_url\":\"https://api.github.com/repos/jimulabs/u2020/hooks\",\"issue_events_url\":\"https://api.github.com/repos/jimulabs/u2020/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/jimulabs/u2020/events\",\"assignees_url\":\"https://api.github.com/repos/jimulabs/u2020/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/jimulabs/u2020/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/jimulabs/u2020/tags\",\"blobs_url\":\"https://api.github.com/repos/jimulabs/u2020/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/jimulabs/u2020/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/jimulabs/u2020/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/jimulabs/u2020/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/jimulabs/u2020/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/jimulabs/u2020/languages\",\"stargazers_url\":\"https://api.github.com/repos/jimulabs/u2020/stargazers\",\"contributors_url\":\"https://api.github.com/repos/jimulabs/u2020/contributors\",\"subscribers_url\":\"https://api.github.com/repos/jimulabs/u2020/subscribers\",\"subscription_url\":\"https://api.github.com/repos/jimulabs/u2020/subscription\",\"commits_url\":\"https://api.github.com/repos/jimulabs/u2020/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/jimulabs/u2020/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/jimulabs/u2020/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/jimulabs/u2020/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/jimulabs/u2020/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/jimulabs/u2020/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/jimulabs/u2020/merges\",\"archive_url\":\"https://api.github.com/repos/jimulabs/u2020/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/jimulabs/u2020/downloads\",\"issues_url\":\"https://api.github.com/repos/jimulabs/u2020/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/jimulabs/u2020/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/jimulabs/u2020/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/jimulabs/u2020/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/jimulabs/u2020/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/jimulabs/u2020/releases{/id}\",\"created_at\":\"2015-01-01T01:05:08Z\",\"updated_at\":\"2014-12-30T22:28:31Z\",\"pushed_at\":\"2014-12-23T00:04:38Z\",\"git_url\":\"git://github.com/jimulabs/u2020.git\",\"ssh_url\":\"git@github.com:jimulabs/u2020.git\",\"clone_url\":\"https://github.com/jimulabs/u2020.git\",\"svn_url\":\"https://github.com/jimulabs/u2020\",\"homepage\":\"http://parleys.com/play/529bde2ce4b0e619540cc3ae\",\"size\":8391,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:05:08Z\"}\n{\"id\":\"2489398047\",\"type\":\"PushEvent\",\"actor\":{\"id\":10225575,\"login\":\"ExclusiveOrange\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ExclusiveOrange\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10225575?\"},\"repo\":{\"id\":28677579,\"name\":\"ExclusiveOrange/synthesizer\",\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer\"},\"payload\":{\"push_id\":536753197,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9221d005a52f784d675044bb14013c0eec6ed795\",\"before\":\"0e64036d5ac0d9c24b4cbe39bb778570db49caba\",\"commits\":[{\"sha\":\"9221d005a52f784d675044bb14013c0eec6ed795\",\"author\":{\"email\":\"de3bd7888dcfc4f7d00a4ef606710f57cbba1dbb@hotmail.com\",\"name\":\"ExclusiveOrange\"},\"message\":\"included stuff needed to actually run program\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer/commits/9221d005a52f784d675044bb14013c0eec6ed795\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:08Z\"}\n{\"id\":\"2489398048\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":716644,\"login\":\"amandaharlin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amandaharlin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/716644?\"},\"repo\":{\"id\":24870992,\"name\":\"codeforokc/codeforokc\",\"url\":\"https://api.github.com/repos/codeforokc/codeforokc\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/2\",\"labels_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/2/events\",\"html_url\":\"https://github.com/codeforokc/codeforokc/issues/2\",\"id\":52505386,\"number\":2,\"title\":\"Add an About page\",\"user\":{\"login\":\"mkchandler\",\"id\":436130,\"avatar_url\":\"https://avatars.githubusercontent.com/u/436130?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mkchandler\",\"html_url\":\"https://github.com/mkchandler\",\"followers_url\":\"https://api.github.com/users/mkchandler/followers\",\"following_url\":\"https://api.github.com/users/mkchandler/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/mkchandler/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/mkchandler/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/mkchandler/subscriptions\",\"organizations_url\":\"https://api.github.com/users/mkchandler/orgs\",\"repos_url\":\"https://api.github.com/users/mkchandler/repos\",\"events_url\":\"https://api.github.com/users/mkchandler/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/mkchandler/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/codeforokc/codeforokc/labels/enhancement\",\"name\":\"enhancement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-19T17:04:48Z\",\"updated_at\":\"2015-01-01T01:05:08Z\",\"closed_at\":null,\"body\":\"Need to add an \\\"About\\\" page to the web site. It should at least include:\\r\\n\\r\\n* Summary of the group and what we do\\r\\n* List of core team members and what each do\"},\"comment\":{\"url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/comments/68477326\",\"html_url\":\"https://github.com/codeforokc/codeforokc/issues/2#issuecomment-68477326\",\"issue_url\":\"https://api.github.com/repos/codeforokc/codeforokc/issues/2\",\"id\":68477326,\"user\":{\"login\":\"amandaharlin\",\"id\":716644,\"avatar_url\":\"https://avatars.githubusercontent.com/u/716644?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/amandaharlin\",\"html_url\":\"https://github.com/amandaharlin\",\"followers_url\":\"https://api.github.com/users/amandaharlin/followers\",\"following_url\":\"https://api.github.com/users/amandaharlin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/amandaharlin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/amandaharlin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/amandaharlin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/amandaharlin/orgs\",\"repos_url\":\"https://api.github.com/users/amandaharlin/repos\",\"events_url\":\"https://api.github.com/users/amandaharlin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/amandaharlin/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:08Z\",\"updated_at\":\"2015-01-01T01:05:08Z\",\"body\":\"can i get a list of the people & their roles? i only know 3 people so far, which might be all there is rn. \"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:08Z\",\"org\":{\"id\":8738523,\"login\":\"codeforokc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/codeforokc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8738523?\"}}\n{\"id\":\"2489398055\",\"type\":\"WatchEvent\",\"actor\":{\"id\":2167695,\"login\":\"skylerzhang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/skylerzhang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2167695?\"},\"repo\":{\"id\":28664808,\"name\":\"cssdream/cssgrace\",\"url\":\"https://api.github.com/repos/cssdream/cssgrace\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:09Z\",\"org\":{\"id\":8910019,\"login\":\"cssdream\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cssdream\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8910019?\"}}\n{\"id\":\"2489398058\",\"type\":\"PushEvent\",\"actor\":{\"id\":185007,\"login\":\"dkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/185007?\"},\"repo\":{\"id\":8727241,\"name\":\"DarkEnergyScienceCollaboration/WeakLensingDeblending\",\"url\":\"https://api.github.com/repos/DarkEnergyScienceCollaboration/WeakLensingDeblending\"},\"payload\":{\"push_id\":536753204,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/master\",\"head\":\"4c8f9d910964133acd0f7eb2958ef7007f638dda\",\"before\":\"ab294701300b44cd830d6577198084a508da6b31\",\"commits\":[{\"sha\":\"8ffdfc613288637871b2cf022763f5d7b3dde5b4\",\"author\":{\"email\":\"0e7e4469b1865516968f5f506ba25c60e5bedc8b@uci.edu\",\"name\":\"David Kirkby\"},\"message\":\"Add PSF and cosmic shear parameters to the survey (not used yet).\",\"distinct\":true,\"url\":\"https://api.github.com/repos/DarkEnergyScienceCollaboration/WeakLensingDeblending/commits/8ffdfc613288637871b2cf022763f5d7b3dde5b4\"},{\"sha\":\"b38651a711e4c240abe4190249b10d0e2eafb71b\",\"author\":{\"email\":\"0e7e4469b1865516968f5f506ba25c60e5bedc8b@uci.edu\",\"name\":\"David Kirkby\"},\"message\":\"Add optional shear to atmospheric PSF\",\"distinct\":true,\"url\":\"https://api.github.com/repos/DarkEnergyScienceCollaboration/WeakLensingDeblending/commits/b38651a711e4c240abe4190249b10d0e2eafb71b\"},{\"sha\":\"fff691186b608876a72a7c6a106f3c0f20e922ac\",\"author\":{\"email\":\"0e7e4469b1865516968f5f506ba25c60e5bedc8b@uci.edu\",\"name\":\"David Kirkby\"},\"message\":\"Specify the ellipticity spinor convention we are using\",\"distinct\":true,\"url\":\"https://api.github.com/repos/DarkEnergyScienceCollaboration/WeakLensingDeblending/commits/fff691186b608876a72a7c6a106f3c0f20e922ac\"},{\"sha\":\"4c8f9d910964133acd0f7eb2958ef7007f638dda\",\"author\":{\"email\":\"0e7e4469b1865516968f5f506ba25c60e5bedc8b@uci.edu\",\"name\":\"David Kirkby\"},\"message\":\"Protect against trying to set z scaling with too few pixels\",\"distinct\":true,\"url\":\"https://api.github.com/repos/DarkEnergyScienceCollaboration/WeakLensingDeblending/commits/4c8f9d910964133acd0f7eb2958ef7007f638dda\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:10Z\",\"org\":{\"id\":2731443,\"login\":\"DarkEnergyScienceCollaboration\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/DarkEnergyScienceCollaboration\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2731443?\"}}\n{\"id\":\"2489398067\",\"type\":\"PushEvent\",\"actor\":{\"id\":9000293,\"login\":\"diianita\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/diianita\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9000293?\"},\"repo\":{\"id\":27146993,\"name\":\"cArLiiToX/dtstore\",\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore\"},\"payload\":{\"push_id\":536753208,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6ce23947a2ec1c89d30f6e8d042154f2c431db86\",\"before\":\"8cacad9b34f30388789700e5bb81deaa4471de12\",\"commits\":[{\"sha\":\"6ce23947a2ec1c89d30f6e8d042154f2c431db86\",\"author\":{\"email\":\"ab5e2bca84933118bbc9d48ffaccce3bac4eeb64@xng.bz\",\"name\":\"cArLiiToX\"},\"message\":\"correciones\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cArLiiToX/dtstore/commits/6ce23947a2ec1c89d30f6e8d042154f2c431db86\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:10Z\"}\n{\"id\":\"2489398070\",\"type\":\"WatchEvent\",\"actor\":{\"id\":694034,\"login\":\"dsimmons\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dsimmons\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/694034?\"},\"repo\":{\"id\":11180687,\"name\":\"spf13/hugo\",\"url\":\"https://api.github.com/repos/spf13/hugo\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:11Z\"}\n{\"id\":\"2489398072\",\"type\":\"PushEvent\",\"actor\":{\"id\":538610,\"login\":\"zachwe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/zachwe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/538610?\"},\"repo\":{\"id\":27940324,\"name\":\"demitrin/listentogithub\",\"url\":\"https://api.github.com/repos/demitrin/listentogithub\"},\"payload\":{\"push_id\":536753210,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"edda8fe60cdd61df50c34608f44deeaac96fff62\",\"before\":\"52f84b1f2d70fd051c27431d728395fc5cc28a76\",\"commits\":[{\"sha\":\"edda8fe60cdd61df50c34608f44deeaac96fff62\",\"author\":{\"email\":\"1ed83132ba391e321c1866dd62e4c246c89cd84e@gmail.com\",\"name\":\"zachwe\"},\"message\":\"updated package.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/demitrin/listentogithub/commits/edda8fe60cdd61df50c34608f44deeaac96fff62\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:11Z\"}\n{\"id\":\"2489398073\",\"type\":\"GollumEvent\",\"actor\":{\"id\":6799218,\"login\":\"gyuho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gyuho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6799218?\"},\"repo\":{\"id\":23096959,\"name\":\"golang/go\",\"url\":\"https://api.github.com/repos/golang/go\"},\"payload\":{\"pages\":[{\"page_name\":\"whygo\",\"title\":\"whygo\",\"summary\":null,\"action\":\"edited\",\"sha\":\"6beb51564de333d65182ed185c47d8648b76c641\",\"html_url\":\"https://github.com/golang/go/wiki/whygo\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:11Z\",\"org\":{\"id\":4314092,\"login\":\"golang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/golang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4314092?\"}}\n{\"id\":\"2489398076\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":10361228,\"login\":\"Dushku\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Dushku\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361228?\"},\"repo\":{\"id\":20304390,\"name\":\"Epix37/Hearthstone-Deck-Tracker\",\"url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/496\",\"labels_url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/496/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/496/comments\",\"events_url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/496/events\",\"html_url\":\"https://github.com/Epix37/Hearthstone-Deck-Tracker/issues/496\",\"id\":53208181,\"number\":496,\"title\":\"export screenshot is cropped\",\"user\":{\"login\":\"Dushku\",\"id\":10361228,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361228?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Dushku\",\"html_url\":\"https://github.com/Dushku\",\"followers_url\":\"https://api.github.com/users/Dushku/followers\",\"following_url\":\"https://api.github.com/users/Dushku/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Dushku/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Dushku/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Dushku/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Dushku/orgs\",\"repos_url\":\"https://api.github.com/users/Dushku/repos\",\"events_url\":\"https://api.github.com/users/Dushku/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Dushku/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-31T23:28:52Z\",\"updated_at\":\"2015-01-01T01:05:11Z\",\"closed_at\":null,\"body\":\"First post and I just wanted to say: I LOVE this tool! Thank you for all the hard work.\\r\\n\\r\\n![trump qm](https://cloud.githubusercontent.com/assets/10361228/5591152/9917046e-911a-11e4-9dd5-b5fd9f4b69b1.png)\\r\\n\\r\\nExport screenshot always only shows first 16 different cards and crops off the right side. I didn't find any other thread on the subject, so I assume it's my settings, but I can't figure out which. Any help would be appreciated.\"},\"comment\":{\"url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/comments/68477328\",\"html_url\":\"https://github.com/Epix37/Hearthstone-Deck-Tracker/issues/496#issuecomment-68477328\",\"issue_url\":\"https://api.github.com/repos/Epix37/Hearthstone-Deck-Tracker/issues/496\",\"id\":68477328,\"user\":{\"login\":\"Dushku\",\"id\":10361228,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361228?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Dushku\",\"html_url\":\"https://github.com/Dushku\",\"followers_url\":\"https://api.github.com/users/Dushku/followers\",\"following_url\":\"https://api.github.com/users/Dushku/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Dushku/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Dushku/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Dushku/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Dushku/orgs\",\"repos_url\":\"https://api.github.com/users/Dushku/repos\",\"events_url\":\"https://api.github.com/users/Dushku/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Dushku/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:11Z\",\"updated_at\":\"2015-01-01T01:05:11Z\",\"body\":\"For what it's worth, I was able to hit Print Screen while it rapidly brought up a window in response to the export screenshot command. It lists all the cards, but it too drops off the right side.\\r\\n\\r\\n![temp](https://cloud.githubusercontent.com/assets/10361228/5591414/3aff5ec2-9128-11e4-8408-92c5a57ed4be.jpg)\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:12Z\"}\n{\"id\":\"2489398077\",\"type\":\"PushEvent\",\"actor\":{\"id\":616495,\"login\":\"harveyt\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harveyt\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/616495?\"},\"repo\":{\"id\":28677911,\"name\":\"harveyt/harveyt.github.io\",\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io\"},\"payload\":{\"push_id\":536753211,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/source\",\"head\":\"8c51961f1332eb509084e9bab9432234b43f8f8f\",\"before\":\"9f57ade1c98efb400f57d710a69c272464596ba4\",\"commits\":[{\"sha\":\"8c51961f1332eb509084e9bab9432234b43f8f8f\",\"author\":{\"email\":\"5b18f340f96afc1bfd4fbf498467b8b0cb41ea73@me.com\",\"name\":\"Harvey Thompson\"},\"message\":\"Added ignore of .deploy\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harveyt/harveyt.github.io/commits/8c51961f1332eb509084e9bab9432234b43f8f8f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:12Z\"}\n{\"id\":\"2489398082\",\"type\":\"PushEvent\",\"actor\":{\"id\":489000,\"login\":\"kwrobot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kwrobot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/489000?\"},\"repo\":{\"id\":631615,\"name\":\"Kitware/VTK\",\"url\":\"https://api.github.com/repos/Kitware/VTK\"},\"payload\":{\"push_id\":536753215,\"size\":3,\"distinct_size\":0,\"ref\":\"refs/heads/nightly-master\",\"head\":\"fc4a4102c54b06f686588b0e3aac9c942c4f3e97\",\"before\":\"5202a28a0d7d36ed1213dc0576549b7898594891\",\"commits\":[{\"sha\":\"ad47d6abc5fee5d4f992fa4e10643e6f58511039\",\"author\":{\"email\":\"ad952b69c80b75aa4e9d1e45057b03ba8d1a6fa5@kitware.com\",\"name\":\"Chuck Atkins\"},\"message\":\"ADIOS: Properly handle variable sized data.\\n\\nRework the ADIOS writer and reader to properly address arrays that change size\\nacross timesteps.\\n\\nChange-Id: I1612308accc36213cb2d21705088a058aba2d591\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/VTK/commits/ad47d6abc5fee5d4f992fa4e10643e6f58511039\"},{\"sha\":\"63c22698d5fd7b59ad8e9c7882c5b4b986ab760e\",\"author\":{\"email\":\"ad952b69c80b75aa4e9d1e45057b03ba8d1a6fa5@kitware.com\",\"name\":\"Chuck Atkins\"},\"message\":\"ADIOS: Fix memory leaks by adding deletes\\n\\nChange-Id: I73107bebe60d9b1351b1a0a5818b65b1668f3a81\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/VTK/commits/63c22698d5fd7b59ad8e9c7882c5b4b986ab760e\"},{\"sha\":\"fc4a4102c54b06f686588b0e3aac9c942c4f3e97\",\"author\":{\"email\":\"ad952b69c80b75aa4e9d1e45057b03ba8d1a6fa5@kitware.com\",\"name\":\"Chuck Atkins\"},\"message\":\"Merge topic 'fix-adios-multistep' into master\\n\\n63c22698 ADIOS: Fix memory leaks by adding deletes\\nad47d6ab ADIOS: Properly handle variable sized data.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/Kitware/VTK/commits/fc4a4102c54b06f686588b0e3aac9c942c4f3e97\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:12Z\",\"org\":{\"id\":87549,\"login\":\"Kitware\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Kitware\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/87549?\"}}\n{\"id\":\"2489398083\",\"type\":\"PushEvent\",\"actor\":{\"id\":3720783,\"login\":\"designerwebhosting\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/designerwebhosting\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3720783?\"},\"repo\":{\"id\":20527117,\"name\":\"designerwebhosting/christopherbyrne.github.io\",\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io\"},\"payload\":{\"push_id\":536753216,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"ff9f4da8bd5a71e0db5778bd27e29b075e9c65ac\",\"before\":\"7e3ee7d19b64e0f0571098be4edb9729d09fb4f4\",\"commits\":[{\"sha\":\"ff9f4da8bd5a71e0db5778bd27e29b075e9c65ac\",\"author\":{\"email\":\"4bb0acc6ff8c0b6c31e50417877e6e3b3f1c65f0@googlemail.com\",\"name\":\"Peter Noblee\"},\"message\":\"update 'date'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/designerwebhosting/christopherbyrne.github.io/commits/ff9f4da8bd5a71e0db5778bd27e29b075e9c65ac\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:12Z\"}\n{\"id\":\"2489398084\",\"type\":\"PushEvent\",\"actor\":{\"id\":7670309,\"login\":\"CodingMonkeyMonster\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CodingMonkeyMonster\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7670309?\"},\"repo\":{\"id\":28644791,\"name\":\"CodingMonkeyMonster/imageacqusitionfromtwosickrangers\",\"url\":\"https://api.github.com/repos/CodingMonkeyMonster/imageacqusitionfromtwosickrangers\"},\"payload\":{\"push_id\":536753217,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"c5db69a606212cc6d486c1f77ca3203cf1bf483f\",\"before\":\"624b2e2b583755fc41eed2533198c30373c4760a\",\"commits\":[{\"sha\":\"c339c869d03501ad837573ed6eaa294ac054586f\",\"author\":{\"email\":\"120e3c204246881d0cb497deab877ed54b238a26@gmail.com\",\"name\":\"CodeMonster\"},\"message\":\"Some modifications to incorporate the use of a second camera\",\"distinct\":true,\"url\":\"https://api.github.com/repos/CodingMonkeyMonster/imageacqusitionfromtwosickrangers/commits/c339c869d03501ad837573ed6eaa294ac054586f\"},{\"sha\":\"c5db69a606212cc6d486c1f77ca3203cf1bf483f\",\"author\":{\"email\":\"120e3c204246881d0cb497deab877ed54b238a26@gmail.com\",\"name\":\"CodeMonster\"},\"message\":\"some more modifications before thinking about perhaps using asychronous threads or delegates?\",\"distinct\":true,\"url\":\"https://api.github.com/repos/CodingMonkeyMonster/imageacqusitionfromtwosickrangers/commits/c5db69a606212cc6d486c1f77ca3203cf1bf483f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:12Z\"}\n{\"id\":\"2489398086\",\"type\":\"PushEvent\",\"actor\":{\"id\":10254849,\"login\":\"NickSanzotta\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/NickSanzotta\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10254849?\"},\"repo\":{\"id\":28525122,\"name\":\"NickSanzotta/strategicsec\",\"url\":\"https://api.github.com/repos/NickSanzotta/strategicsec\"},\"payload\":{\"push_id\":536753218,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a018e85ada6048fc90131af5bd149a01712547b2\",\"before\":\"0e3787e8504eb26395c2dc2e457275e41d56cf2e\",\"commits\":[{\"sha\":\"a018e85ada6048fc90131af5bd149a01712547b2\",\"author\":{\"email\":\"b49959da48d165fcfc8ebb39ae6132c54ef35404@outlook.com\",\"name\":\"NickSanzotta\"},\"message\":\"domain function added\",\"distinct\":true,\"url\":\"https://api.github.com/repos/NickSanzotta/strategicsec/commits/a018e85ada6048fc90131af5bd149a01712547b2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:13Z\"}\n{\"id\":\"2489398088\",\"type\":\"ForkEvent\",\"actor\":{\"id\":3356814,\"login\":\"itpcc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/itpcc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3356814?\"},\"repo\":{\"id\":28064768,\"name\":\"phpinfo-in-th/workshop-webboard\",\"url\":\"https://api.github.com/repos/phpinfo-in-th/workshop-webboard\"},\"payload\":{\"forkee\":{\"id\":28678270,\"name\":\"workshop-webboard\",\"full_name\":\"itpcc/workshop-webboard\",\"owner\":{\"login\":\"itpcc\",\"id\":3356814,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3356814?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/itpcc\",\"html_url\":\"https://github.com/itpcc\",\"followers_url\":\"https://api.github.com/users/itpcc/followers\",\"following_url\":\"https://api.github.com/users/itpcc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/itpcc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/itpcc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/itpcc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/itpcc/orgs\",\"repos_url\":\"https://api.github.com/users/itpcc/repos\",\"events_url\":\"https://api.github.com/users/itpcc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/itpcc/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/itpcc/workshop-webboard\",\"description\":\"\",\"fork\":true,\"url\":\"https://api.github.com/repos/itpcc/workshop-webboard\",\"forks_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/forks\",\"keys_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/teams\",\"hooks_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/hooks\",\"issue_events_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/events\",\"assignees_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/tags\",\"blobs_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/languages\",\"stargazers_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/stargazers\",\"contributors_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/contributors\",\"subscribers_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/subscribers\",\"subscription_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/subscription\",\"commits_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/merges\",\"archive_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/downloads\",\"issues_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/itpcc/workshop-webboard/releases{/id}\",\"created_at\":\"2015-01-01T01:05:13Z\",\"updated_at\":\"2014-12-16T01:42:48Z\",\"pushed_at\":\"2014-12-16T01:42:48Z\",\"git_url\":\"git://github.com/itpcc/workshop-webboard.git\",\"ssh_url\":\"git@github.com:itpcc/workshop-webboard.git\",\"clone_url\":\"https://github.com/itpcc/workshop-webboard.git\",\"svn_url\":\"https://github.com/itpcc/workshop-webboard\",\"homepage\":null,\"size\":140,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:05:13Z\"}\n{\"id\":\"2489398089\",\"type\":\"PushEvent\",\"actor\":{\"id\":106133,\"login\":\"Tiggar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tiggar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/106133?\"},\"repo\":{\"id\":27003508,\"name\":\"Tiggar/isp-performance\",\"url\":\"https://api.github.com/repos/Tiggar/isp-performance\"},\"payload\":{\"push_id\":536753219,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"2dd68c64ebb4a14935823a00f0ce08df238f6cd5\",\"before\":\"512ea3ddc142c05f89eaf1b69f5a084ac1f99ba6\",\"commits\":[{\"sha\":\"2dd68c64ebb4a14935823a00f0ce08df238f6cd5\",\"author\":{\"email\":\"5069df3b8e9785f1eb5e6bda1f2483f61b73e1aa@gmail.com\",\"name\":\"Jan Michael\"},\"message\":\"updates all charts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Tiggar/isp-performance/commits/2dd68c64ebb4a14935823a00f0ce08df238f6cd5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:13Z\"}\n{\"id\":\"2489398090\",\"type\":\"PushEvent\",\"actor\":{\"id\":9205456,\"login\":\"Anthony-Dev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Anthony-Dev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9205456?\"},\"repo\":{\"id\":28667691,\"name\":\"Anthony-Dev/ModdedMinecraftJar\",\"url\":\"https://api.github.com/repos/Anthony-Dev/ModdedMinecraftJar\"},\"payload\":{\"push_id\":536753220,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3956e0b94a256be19bb61d6c54e4534f03e6f99d\",\"before\":\"05685975ac219c387b900e0f05665053f8e3a7f6\",\"commits\":[{\"sha\":\"3956e0b94a256be19bb61d6c54e4534f03e6f99d\",\"author\":{\"email\":\"28580e7b9fd3af655a23a148dc5c207632216f69@outlook.com\",\"name\":\"Anthony-Dev\"},\"message\":\"Launcher Jar 8\\n\\nUUID Changes to this.id\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Anthony-Dev/ModdedMinecraftJar/commits/3956e0b94a256be19bb61d6c54e4534f03e6f99d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:13Z\"}\n{\"id\":\"2489398095\",\"type\":\"PushEvent\",\"actor\":{\"id\":1449778,\"login\":\"billperegoy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/billperegoy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1449778?\"},\"repo\":{\"id\":28676506,\"name\":\"billperegoy/parse_args\",\"url\":\"https://api.github.com/repos/billperegoy/parse_args\"},\"payload\":{\"push_id\":536753221,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"dd4eb8719dc48ca7e9bd948ffb9fe7ce492d70de\",\"before\":\"4fc9773581df0562d3a49040ce00e1ef5c962bd8\",\"commits\":[{\"sha\":\"dd4eb8719dc48ca7e9bd948ffb9fe7ce492d70de\",\"author\":{\"email\":\"c692d6a10598e0a801576fdd4ecf3c37e45bfbc4@billperegoy.com\",\"name\":\"billperegoy\"},\"message\":\"Added cane quality checks to Rakefile.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/billperegoy/parse_args/commits/dd4eb8719dc48ca7e9bd948ffb9fe7ce492d70de\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:14Z\"}\n{\"id\":\"2489398096\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1534224,\"login\":\"tracyapps\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tracyapps\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1534224?\"},\"repo\":{\"id\":19757917,\"name\":\"maxwellito/vivus\",\"url\":\"https://api.github.com/repos/maxwellito/vivus\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:14Z\"}\n{\"id\":\"2489398100\",\"type\":\"WatchEvent\",\"actor\":{\"id\":3421858,\"login\":\"codepreneur\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/codepreneur\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3421858?\"},\"repo\":{\"id\":5085231,\"name\":\"WhisperSystems/RedPhone\",\"url\":\"https://api.github.com/repos/WhisperSystems/RedPhone\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:15Z\",\"org\":{\"id\":702459,\"login\":\"WhisperSystems\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/WhisperSystems\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/702459?\"}}\n{\"id\":\"2489398106\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397290\",\"id\":22397290,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":123,\"original_position\":123,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Why did this go single line?\",\"created_at\":\"2015-01-01T01:05:15Z\",\"updated_at\":\"2015-01-01T01:05:15Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397290\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397290\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397290\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:05:15Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:05:15Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489398107\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":2982725,\"login\":\"sharef88\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sharef88\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2982725?\"},\"repo\":{\"id\":22777385,\"name\":\"ChatFawkes/Flux-Galaxy\",\"url\":\"https://api.github.com/repos/ChatFawkes/Flux-Galaxy\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/ChatFawkes/Flux-Galaxy/issues/66\",\"labels_url\":\"https://api.github.com/repos/ChatFawkes/Flux-Galaxy/issues/66/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/ChatFawkes/Flux-Galaxy/issues/66/comments\",\"events_url\":\"https://api.github.com/repos/ChatFawkes/Flux-Galaxy/issues/66/events\",\"html_url\":\"https://github.com/ChatFawkes/Flux-Galaxy/issues/66\",\"id\":53210275,\"number\":66,\"title\":\"Crash that I can't figure out\",\"user\":{\"login\":\"sharef88\",\"id\":2982725,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2982725?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sharef88\",\"html_url\":\"https://github.com/sharef88\",\"followers_url\":\"https://api.github.com/users/sharef88/followers\",\"following_url\":\"https://api.github.com/users/sharef88/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/sharef88/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/sharef88/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/sharef88/subscriptions\",\"organizations_url\":\"https://api.github.com/users/sharef88/orgs\",\"repos_url\":\"https://api.github.com/users/sharef88/repos\",\"events_url\":\"https://api.github.com/users/sharef88/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/sharef88/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:05:16Z\",\"updated_at\":\"2015-01-01T01:05:16Z\",\"closed_at\":null,\"body\":\"Crash-Report generated by the server.\\r\\nhttp://pastebin.com/k9z2X98f\\r\\nI had placed a small compact machine about 30m before this crash.  I was removing some mechanisim universal cable with an unenchanted bound pickaxe about 10 blocks away from said compact machine. The machine was inert and empty.  I cant figure out how to fix this.  MCEdit surgery didn't help the matter, nor did removing compact machines mod, or the related file in data/ or the dim.\\r\\nI'm using pack version 7.3.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:16Z\"}\n{\"id\":\"2489398112\",\"type\":\"PushEvent\",\"actor\":{\"id\":238354,\"login\":\"variousred\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/variousred\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/238354?\"},\"repo\":{\"id\":6274404,\"name\":\"G5/g5-content-management-system\",\"url\":\"https://api.github.com/repos/G5/g5-content-management-system\"},\"payload\":{\"push_id\":536753224,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/fix-deep-clone\",\"head\":\"b16f39b5d6be375e42eef7183ef39ed67077bee0\",\"before\":\"81baf7738ce7da9084ecd6f657cd3c0e60f888a2\",\"commits\":[{\"sha\":\"b16f39b5d6be375e42eef7183ef39ed67077bee0\",\"author\":{\"email\":\"16ea8d52c08316685a257e07ddecf7165a502f6d@gmail.com\",\"name\":\"Michael Mitchell\"},\"message\":\" removes pry\",\"distinct\":true,\"url\":\"https://api.github.com/repos/G5/g5-content-management-system/commits/b16f39b5d6be375e42eef7183ef39ed67077bee0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:17Z\",\"org\":{\"id\":2396851,\"login\":\"G5\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/G5\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2396851?\"}}\n{\"id\":\"2489398116\",\"type\":\"PushEvent\",\"actor\":{\"id\":312301,\"login\":\"davidturner\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/davidturner\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/312301?\"},\"repo\":{\"id\":26051556,\"name\":\"davidturner/dotfiles\",\"url\":\"https://api.github.com/repos/davidturner/dotfiles\"},\"payload\":{\"push_id\":536753226,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c526773632b35732e3955e054dc4d7248ab49238\",\"before\":\"65c8456130b0154315b1a3684b2b46c4b0144fa8\",\"commits\":[{\"sha\":\"c526773632b35732e3955e054dc4d7248ab49238\",\"author\":{\"email\":\"aa743a0aaec8f7d7a1f01442503957f4d7a2d634@davidturner.name\",\"name\":\"David Turner\"},\"message\":\"Streamlining setup process.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/davidturner/dotfiles/commits/c526773632b35732e3955e054dc4d7248ab49238\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:17Z\"}\n{\"id\":\"2489398124\",\"type\":\"WatchEvent\",\"actor\":{\"id\":51633,\"login\":\"marklise\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/marklise\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/51633?\"},\"repo\":{\"id\":242821,\"name\":\"omz/AppSales-Mobile\",\"url\":\"https://api.github.com/repos/omz/AppSales-Mobile\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:18Z\"}\n{\"id\":\"2489398131\",\"type\":\"PushEvent\",\"actor\":{\"id\":939501,\"login\":\"jacebrowning\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jacebrowning\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/939501?\"},\"repo\":{\"id\":19165491,\"name\":\"jacebrowning/yorm\",\"url\":\"https://api.github.com/repos/jacebrowning/yorm\"},\"payload\":{\"push_id\":536753230,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"44343210f475f5d87a70dab541e9158ddc90c397\",\"before\":\"9e70ef1c7855f04cbcd073c05a3b727e08b8df5f\",\"commits\":[{\"sha\":\"44343210f475f5d87a70dab541e9158ddc90c397\",\"author\":{\"email\":\"567f1b61689b9d694b4628a3551d98928040e4fd@gmail.com\",\"name\":\"Jace Browning\"},\"message\":\"Add standard types with NoneType defaults\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jacebrowning/yorm/commits/44343210f475f5d87a70dab541e9158ddc90c397\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:20Z\"}\n{\"id\":\"2489398136\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":9351018,\"login\":\"lblxh3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lblxh3\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9351018?\"},\"repo\":{\"id\":28678195,\"name\":\"TTMTT/iCL0udin\",\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"labels_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/events\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1\",\"id\":53210206,\"number\":1,\"title\":\"Discuss1\",\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T01:02:34Z\",\"updated_at\":\"2015-01-01T01:05:21Z\",\"closed_at\":null,\"body\":\"Now you can download vresion 1.0 from :\\r\\n---------------------------------------------------\\r\\nhttp://www.icloudin.net\\r\\n-----------------------------\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n--------------------------------------\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n---------------------------------------\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n-------------------------------------\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n-------------------------------------\\r\\niCL0udin v1.0 -> %100\\r\\n----------------------------\\r\\nRemaining: %3 testing with some people..\\r\\n-----------------------------------------------------\\r\\nLast Method:\\r\\n-----------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\niCL0udin v1.0 have this method:\\r\\n-----------------------------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/comments/68477333\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1#issuecomment-68477333\",\"issue_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"id\":68477333,\"user\":{\"login\":\"lblxh3\",\"id\":9351018,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9351018?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lblxh3\",\"html_url\":\"https://github.com/lblxh3\",\"followers_url\":\"https://api.github.com/users/lblxh3/followers\",\"following_url\":\"https://api.github.com/users/lblxh3/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/lblxh3/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/lblxh3/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/lblxh3/subscriptions\",\"organizations_url\":\"https://api.github.com/users/lblxh3/orgs\",\"repos_url\":\"https://api.github.com/users/lblxh3/repos\",\"events_url\":\"https://api.github.com/users/lblxh3/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/lblxh3/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:21Z\",\"updated_at\":\"2015-01-01T01:05:21Z\",\"body\":\"love u\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:21Z\"}\n{\"id\":\"2489398140\",\"type\":\"PushEvent\",\"actor\":{\"id\":2812278,\"login\":\"beret595\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?\"},\"repo\":{\"id\":28470088,\"name\":\"beret595/Ucar_Operation_Crm_Finance\",\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance\"},\"payload\":{\"push_id\":536753232,\"size\":6,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4be49692843556bedea40c3fbe6954972dd7beb0\",\"before\":\"83f3a8e7891270f54d4119a862c57bfaf1db5475\",\"commits\":[{\"sha\":\"aa16c2df1091bdf449c6b6a4b4ae02cfde6daa0d\",\"author\":{\"email\":\"230ed9215685d4acc3825e7461feafb870abaeb3@163.com\",\"name\":\"lixi1984\"},\"message\":\"保险提醒数据不准确,将搜索结果按照以下字段显示\\n\\n保险提醒数据不准确,将搜索结果按照以下字段显示 .\\n客户 品牌 车型 车牌 保险到期 保险公司 邀约员 生成日期 分配日期 案件状态 操作 查看\\n将利润管理单元去掉\\n呼叫管理,利润管理,绩效管理下面多了下划线,去掉\",\"distinct\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/aa16c2df1091bdf449c6b6a4b4ae02cfde6daa0d\"},{\"sha\":\"2ff7d33c43d841388406502538cc14f07139a34d\",\"author\":{\"email\":\"230ed9215685d4acc3825e7461feafb870abaeb3@163.com\",\"name\":\"lixi1984\"},\"message\":\"Revert \\\"保险提醒数据不准确,将搜索结果按照以下字段显示\\\"\\n\\nThis reverts commit aa16c2df1091bdf449c6b6a4b4ae02cfde6daa0d.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/2ff7d33c43d841388406502538cc14f07139a34d\"},{\"sha\":\"eee8c95ea1793f864928a018de1f821428948951\",\"author\":{\"email\":\"230ed9215685d4acc3825e7461feafb870abaeb3@163.com\",\"name\":\"lixi1984\"},\"message\":\"Revert \\\"Revert \\\"保险提醒数据不准确,将搜索结果按照以下字段显示\\\"\\\"\\n\\nThis reverts commit 2ff7d33c43d841388406502538cc14f07139a34d.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/eee8c95ea1793f864928a018de1f821428948951\"},{\"sha\":\"999eb6abb00d4f80ab2f908ddb75c591dae44e0e\",\"author\":{\"email\":\"230ed9215685d4acc3825e7461feafb870abaeb3@163.com\",\"name\":\"lixi1984\"},\"message\":\"动态库更新\\n\\n动态库更新\",\"distinct\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/999eb6abb00d4f80ab2f908ddb75c591dae44e0e\"},{\"sha\":\"fb9307eefb142d216d38294f23e42779e0aecbea\",\"author\":{\"email\":\"230ed9215685d4acc3825e7461feafb870abaeb3@163.com\",\"name\":\"lixi1984\"},\"message\":\"数据备份\\n\\n数据备份 车辆迁移,人员信息迁移时效验客户编码\",\"distinct\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/fb9307eefb142d216d38294f23e42779e0aecbea\"},{\"sha\":\"4be49692843556bedea40c3fbe6954972dd7beb0\",\"author\":{\"email\":\"3218d7ac8ea669170780ae5452d9ed76867b408e@foxmail.com\",\"name\":\"beret595\"},\"message\":\"Merge pull request #1 from beret595/lixi1984\\n\\nLixi1984\",\"distinct\":true,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits/4be49692843556bedea40c3fbe6954972dd7beb0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:22Z\"}\n{\"id\":\"2489398141\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":2812278,\"login\":\"beret595\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?\"},\"repo\":{\"id\":28470088,\"name\":\"beret595/Ucar_Operation_Crm_Finance\",\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance\"},\"payload\":{\"action\":\"closed\",\"number\":1,\"pull_request\":{\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1\",\"id\":26714507,\"html_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance/pull/1\",\"diff_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance/pull/1.diff\",\"patch_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance/pull/1.patch\",\"issue_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/1\",\"number\":1,\"state\":\"closed\",\"locked\":false,\"title\":\"Lixi1984\",\"user\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"1.业务系统中,客户编号不是7位数字和字母组合的不要上传至邀约系统\\r\\n2.保养提醒数据不准确,\\r\\n3.保险提醒数据不准确 \",\"created_at\":\"2014-12-31T04:03:10Z\",\"updated_at\":\"2015-01-01T01:05:21Z\",\"closed_at\":\"2015-01-01T01:05:21Z\",\"merged_at\":\"2015-01-01T01:05:21Z\",\"merge_commit_sha\":\"29b658d7803c1e09eae5558e8c34b52e90e270f0\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1/commits\",\"review_comments_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1/comments\",\"review_comment_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/1/comments\",\"statuses_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/statuses/fb9307eefb142d216d38294f23e42779e0aecbea\",\"head\":{\"label\":\"beret595:lixi1984\",\"ref\":\"lixi1984\",\"sha\":\"fb9307eefb142d216d38294f23e42779e0aecbea\",\"user\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28470088,\"name\":\"Ucar_Operation_Crm_Finance\",\"full_name\":\"beret595/Ucar_Operation_Crm_Finance\",\"owner\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance\",\"description\":null,\"fork\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance\",\"forks_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/forks\",\"keys_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/teams\",\"hooks_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/hooks\",\"issue_events_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/events\",\"assignees_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/tags\",\"blobs_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/languages\",\"stargazers_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/stargazers\",\"contributors_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/contributors\",\"subscribers_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/subscribers\",\"subscription_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/subscription\",\"commits_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/merges\",\"archive_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/downloads\",\"issues_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/releases{/id}\",\"created_at\":\"2014-12-25T03:53:57Z\",\"updated_at\":\"2014-12-30T02:22:03Z\",\"pushed_at\":\"2015-01-01T01:05:21Z\",\"git_url\":\"git://github.com/beret595/Ucar_Operation_Crm_Finance.git\",\"ssh_url\":\"git@github.com:beret595/Ucar_Operation_Crm_Finance.git\",\"clone_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance.git\",\"svn_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance\",\"homepage\":null,\"size\":0,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"C#\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"beret595:master\",\"ref\":\"master\",\"sha\":\"83f3a8e7891270f54d4119a862c57bfaf1db5475\",\"user\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28470088,\"name\":\"Ucar_Operation_Crm_Finance\",\"full_name\":\"beret595/Ucar_Operation_Crm_Finance\",\"owner\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance\",\"description\":null,\"fork\":false,\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance\",\"forks_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/forks\",\"keys_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/teams\",\"hooks_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/hooks\",\"issue_events_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/events\",\"assignees_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/tags\",\"blobs_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/languages\",\"stargazers_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/stargazers\",\"contributors_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/contributors\",\"subscribers_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/subscribers\",\"subscription_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/subscription\",\"commits_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/merges\",\"archive_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/downloads\",\"issues_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/releases{/id}\",\"created_at\":\"2014-12-25T03:53:57Z\",\"updated_at\":\"2014-12-30T02:22:03Z\",\"pushed_at\":\"2015-01-01T01:05:21Z\",\"git_url\":\"git://github.com/beret595/Ucar_Operation_Crm_Finance.git\",\"ssh_url\":\"git@github.com:beret595/Ucar_Operation_Crm_Finance.git\",\"clone_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance.git\",\"svn_url\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance\",\"homepage\":null,\"size\":0,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"C#\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1\"},\"html\":{\"href\":\"https://github.com/beret595/Ucar_Operation_Crm_Finance/pull/1\"},\"issue\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/1\"},\"comments\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/issues/1/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/pulls/1/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance/statuses/fb9307eefb142d216d38294f23e42779e0aecbea\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"beret595\",\"id\":2812278,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"html_url\":\"https://github.com/beret595\",\"followers_url\":\"https://api.github.com/users/beret595/followers\",\"following_url\":\"https://api.github.com/users/beret595/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/beret595/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/beret595/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/beret595/subscriptions\",\"organizations_url\":\"https://api.github.com/users/beret595/orgs\",\"repos_url\":\"https://api.github.com/users/beret595/repos\",\"events_url\":\"https://api.github.com/users/beret595/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/beret595/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":5,\"additions\":755,\"deletions\":44,\"changed_files\":11}},\"public\":true,\"created_at\":\"2015-01-01T01:05:22Z\"}\n{\"id\":\"2489398144\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":4437769,\"name\":\"casatt/html5-videoEditor\",\"url\":\"https://api.github.com/repos/casatt/html5-videoEditor\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:22Z\"}\n{\"id\":\"2489398151\",\"type\":\"PushEvent\",\"actor\":{\"id\":6825691,\"login\":\"sogeo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sogeo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6825691?\"},\"repo\":{\"id\":17319070,\"name\":\"sogeo/DM01AVSO24\",\"url\":\"https://api.github.com/repos/sogeo/DM01AVSO24\"},\"payload\":{\"push_id\":536753235,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7777eb7c152ff9e2597103295a71bf27496e1292\",\"before\":\"13bd78547455b9ed0dbe380b1c4e614bd06100b3\",\"commits\":[{\"sha\":\"7777eb7c152ff9e2597103295a71bf27496e1292\",\"author\":{\"email\":\"afdee6c9fc3dccec54bef9bef15ae27098932d1c@gmail.com\",\"name\":\"sogeo\"},\"message\":\"Wöchentliche Datenlieferung (1 / 1 / 2015-01-01-02.04.39)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sogeo/DM01AVSO24/commits/7777eb7c152ff9e2597103295a71bf27496e1292\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:23Z\"}\n{\"id\":\"2489398152\",\"type\":\"CreateEvent\",\"actor\":{\"id\":4975721,\"login\":\"macieknajbar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/macieknajbar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4975721?\"},\"repo\":{\"id\":28678271,\"name\":\"macieknajbar/GYm\",\"url\":\"https://api.github.com/repos/macieknajbar/GYm\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"A gym supporting application\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:23Z\"}\n{\"id\":\"2489398153\",\"type\":\"PushEvent\",\"actor\":{\"id\":1671640,\"login\":\"alex6lc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alex6lc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1671640?\"},\"repo\":{\"id\":28678262,\"name\":\"alex6lc/material-ui-sass\",\"url\":\"https://api.github.com/repos/alex6lc/material-ui-sass\"},\"payload\":{\"push_id\":536753236,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5c1c07ca7e009f91a0b0f6c953e826a7cb3f7aa8\",\"before\":\"4caa9114f99ce0215e6db9ab551226d1d02aaf50\",\"commits\":[{\"sha\":\"5c1c07ca7e009f91a0b0f6c953e826a7cb3f7aa8\",\"author\":{\"email\":\"1154eaf5246093599e182c05c916e7a183e0e00e@gmail.com\",\"name\":\"alex6lc\"},\"message\":\"Remove post install scripts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alex6lc/material-ui-sass/commits/5c1c07ca7e009f91a0b0f6c953e826a7cb3f7aa8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:23Z\"}\n{\"id\":\"2489398155\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":6964047,\"login\":\"TTMTT\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?\"},\"repo\":{\"id\":26731988,\"name\":\"badrsony/icloudin-support-\",\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4\",\"labels_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/comments\",\"events_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4/events\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/4\",\"id\":50920400,\"number\":4,\"title\":\"icloudin support \",\"user\":{\"login\":\"badrsony\",\"id\":7895050,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7895050?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/badrsony\",\"html_url\":\"https://github.com/badrsony\",\"followers_url\":\"https://api.github.com/users/badrsony/followers\",\"following_url\":\"https://api.github.com/users/badrsony/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/badrsony/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/badrsony/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/badrsony/subscriptions\",\"organizations_url\":\"https://api.github.com/users/badrsony/orgs\",\"repos_url\":\"https://api.github.com/users/badrsony/repos\",\"events_url\":\"https://api.github.com/users/badrsony/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/badrsony/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":997,\"created_at\":\"2014-12-04T02:13:39Z\",\"updated_at\":\"2015-01-01T01:05:23Z\",\"closed_at\":\"2015-01-01T00:00:32Z\",\"body\":\"Originally written by @TTMTT. That we hope for him safery and peace\\r\\n.\\r\\n\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n\\r\\niCL0udin v1.0 -> %98\\r\\n\\r\\nRemaining: %2 testing with some people..\\r\\n\\r\\nLast Method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\n\\r\\niCL0udin v1.0 have this method:\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/comments/68477335\",\"html_url\":\"https://github.com/badrsony/icloudin-support-/issues/4#issuecomment-68477335\",\"issue_url\":\"https://api.github.com/repos/badrsony/icloudin-support-/issues/4\",\"id\":68477335,\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:23Z\",\"updated_at\":\"2015-01-01T01:05:23Z\",\"body\":\"@badrsony \\r\\nexcuse me i open a new discuss for (iCL0udin v1.0):\\r\\n--------------------------------------------------------\\r\\nhttps://github.com/TTMTT/iCL0udin/issues/1\\r\\n--------------------------------------------------------\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:24Z\"}\n{\"id\":\"2489398158\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":5831804,\"login\":\"YueLinHo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YueLinHo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5831804?\"},\"repo\":{\"id\":1197206,\"name\":\"TortoiseGit/TortoiseGit\",\"url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit\"},\"payload\":{\"action\":\"opened\",\"number\":151,\"pull_request\":{\"url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151\",\"id\":26739456,\"html_url\":\"https://github.com/TortoiseGit/TortoiseGit/pull/151\",\"diff_url\":\"https://github.com/TortoiseGit/TortoiseGit/pull/151.diff\",\"patch_url\":\"https://github.com/TortoiseGit/TortoiseGit/pull/151.patch\",\"issue_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/151\",\"number\":151,\"state\":\"open\",\"locked\":false,\"title\":\"Refresh the status of \\\"Working dir changes\\\" of Log dialog by pressing F6\",\"user\":{\"login\":\"YueLinHo\",\"id\":5831804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5831804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YueLinHo\",\"html_url\":\"https://github.com/YueLinHo\",\"followers_url\":\"https://api.github.com/users/YueLinHo/followers\",\"following_url\":\"https://api.github.com/users/YueLinHo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/YueLinHo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/YueLinHo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/YueLinHo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/YueLinHo/orgs\",\"repos_url\":\"https://api.github.com/users/YueLinHo/repos\",\"events_url\":\"https://api.github.com/users/YueLinHo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/YueLinHo/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"@lznuaa @csware @ch3cooli \\r\\nHappy new year to all of you. ^_______________^\",\"created_at\":\"2015-01-01T01:05:24Z\",\"updated_at\":\"2015-01-01T01:05:24Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151/commits\",\"review_comments_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151/comments\",\"review_comment_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/151/comments\",\"statuses_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/statuses/a0af6070519ae3ffae329edd8e6cd1042d396fe0\",\"head\":{\"label\":\"YueLinHo:ylh/happy_new_year_2015\",\"ref\":\"ylh/happy_new_year_2015\",\"sha\":\"a0af6070519ae3ffae329edd8e6cd1042d396fe0\",\"user\":{\"login\":\"YueLinHo\",\"id\":5831804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5831804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YueLinHo\",\"html_url\":\"https://github.com/YueLinHo\",\"followers_url\":\"https://api.github.com/users/YueLinHo/followers\",\"following_url\":\"https://api.github.com/users/YueLinHo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/YueLinHo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/YueLinHo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/YueLinHo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/YueLinHo/orgs\",\"repos_url\":\"https://api.github.com/users/YueLinHo/repos\",\"events_url\":\"https://api.github.com/users/YueLinHo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/YueLinHo/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":21376277,\"name\":\"TortoiseGit\",\"full_name\":\"YueLinHo/TortoiseGit\",\"owner\":{\"login\":\"YueLinHo\",\"id\":5831804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5831804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/YueLinHo\",\"html_url\":\"https://github.com/YueLinHo\",\"followers_url\":\"https://api.github.com/users/YueLinHo/followers\",\"following_url\":\"https://api.github.com/users/YueLinHo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/YueLinHo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/YueLinHo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/YueLinHo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/YueLinHo/orgs\",\"repos_url\":\"https://api.github.com/users/YueLinHo/repos\",\"events_url\":\"https://api.github.com/users/YueLinHo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/YueLinHo/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/YueLinHo/TortoiseGit\",\"description\":\"Windows Explorer Extension to Operate Git; Mirror of GoogleCode repository\",\"fork\":true,\"url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit\",\"forks_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/forks\",\"keys_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/teams\",\"hooks_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/hooks\",\"issue_events_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/events\",\"assignees_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/tags\",\"blobs_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/languages\",\"stargazers_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/stargazers\",\"contributors_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/contributors\",\"subscribers_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/subscribers\",\"subscription_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/subscription\",\"commits_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/merges\",\"archive_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/downloads\",\"issues_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/YueLinHo/TortoiseGit/releases{/id}\",\"created_at\":\"2014-07-01T05:31:50Z\",\"updated_at\":\"2014-12-30T09:25:25Z\",\"pushed_at\":\"2015-01-01T01:04:01Z\",\"git_url\":\"git://github.com/YueLinHo/TortoiseGit.git\",\"ssh_url\":\"git@github.com:YueLinHo/TortoiseGit.git\",\"clone_url\":\"https://github.com/YueLinHo/TortoiseGit.git\",\"svn_url\":\"https://github.com/YueLinHo/TortoiseGit\",\"homepage\":\"http://tortoisegit.org\",\"size\":93279,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"C++\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"TortoiseGit:master\",\"ref\":\"master\",\"sha\":\"fa7295b820be9ca368924b8129253df52c189c9d\",\"user\":{\"login\":\"TortoiseGit\",\"id\":1103929,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1103929?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TortoiseGit\",\"html_url\":\"https://github.com/TortoiseGit\",\"followers_url\":\"https://api.github.com/users/TortoiseGit/followers\",\"following_url\":\"https://api.github.com/users/TortoiseGit/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TortoiseGit/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TortoiseGit/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TortoiseGit/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TortoiseGit/orgs\",\"repos_url\":\"https://api.github.com/users/TortoiseGit/repos\",\"events_url\":\"https://api.github.com/users/TortoiseGit/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TortoiseGit/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":1197206,\"name\":\"TortoiseGit\",\"full_name\":\"TortoiseGit/TortoiseGit\",\"owner\":{\"login\":\"TortoiseGit\",\"id\":1103929,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1103929?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TortoiseGit\",\"html_url\":\"https://github.com/TortoiseGit\",\"followers_url\":\"https://api.github.com/users/TortoiseGit/followers\",\"following_url\":\"https://api.github.com/users/TortoiseGit/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TortoiseGit/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TortoiseGit/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TortoiseGit/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TortoiseGit/orgs\",\"repos_url\":\"https://api.github.com/users/TortoiseGit/repos\",\"events_url\":\"https://api.github.com/users/TortoiseGit/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TortoiseGit/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/TortoiseGit/TortoiseGit\",\"description\":\"Windows Explorer Extension to Operate Git; Mirror of GoogleCode repository\",\"fork\":false,\"url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit\",\"forks_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/forks\",\"keys_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/teams\",\"hooks_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/hooks\",\"issue_events_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/events\",\"assignees_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/tags\",\"blobs_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/languages\",\"stargazers_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/stargazers\",\"contributors_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/contributors\",\"subscribers_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/subscribers\",\"subscription_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/subscription\",\"commits_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/merges\",\"archive_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/downloads\",\"issues_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/releases{/id}\",\"created_at\":\"2010-12-25T14:10:21Z\",\"updated_at\":\"2014-12-31T05:00:09Z\",\"pushed_at\":\"2014-12-31T05:00:09Z\",\"git_url\":\"git://github.com/TortoiseGit/TortoiseGit.git\",\"ssh_url\":\"git@github.com:TortoiseGit/TortoiseGit.git\",\"clone_url\":\"https://github.com/TortoiseGit/TortoiseGit.git\",\"svn_url\":\"https://github.com/TortoiseGit/TortoiseGit\",\"homepage\":\"http://tortoisegit.org\",\"size\":111079,\"stargazers_count\":193,\"watchers_count\":193,\"language\":\"C++\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":80,\"mirror_url\":null,\"open_issues_count\":17,\"forks\":80,\"open_issues\":17,\"watchers\":193,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151\"},\"html\":{\"href\":\"https://github.com/TortoiseGit/TortoiseGit/pull/151\"},\"issue\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/151\"},\"comments\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/issues/151/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/pulls/151/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/TortoiseGit/TortoiseGit/statuses/a0af6070519ae3ffae329edd8e6cd1042d396fe0\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":11,\"deletions\":1,\"changed_files\":4}},\"public\":true,\"created_at\":\"2015-01-01T01:05:25Z\",\"org\":{\"id\":1103929,\"login\":\"TortoiseGit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/TortoiseGit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1103929?\"}}\n{\"id\":\"2489398163\",\"type\":\"PushEvent\",\"actor\":{\"id\":5700937,\"login\":\"darealshinji\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/darealshinji\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5700937?\"},\"repo\":{\"id\":26483280,\"name\":\"darealshinji/debian\",\"url\":\"https://api.github.com/repos/darealshinji/debian\"},\"payload\":{\"push_id\":536753239,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"69079504019294a505b9c248ba71186190abc0a8\",\"before\":\"a0457f679a476079cb2ff0cafcff945120af4524\",\"commits\":[{\"sha\":\"268b418c00f957794eed7a33f7e6089fb25162f2\",\"author\":{\"email\":\"52122a01dfac1c09ea3fc6e6913df9984d0f756a@gmx.de\",\"name\":\"darealshinji\"},\"message\":\"new package: editors/sublime-text-2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/darealshinji/debian/commits/268b418c00f957794eed7a33f7e6089fb25162f2\"},{\"sha\":\"69079504019294a505b9c248ba71186190abc0a8\",\"author\":{\"email\":\"52122a01dfac1c09ea3fc6e6913df9984d0f756a@gmx.de\",\"name\":\"darealshinji\"},\"message\":\"new package: editors/sublime-text-3\",\"distinct\":true,\"url\":\"https://api.github.com/repos/darealshinji/debian/commits/69079504019294a505b9c248ba71186190abc0a8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:25Z\"}\n{\"id\":\"2489398166\",\"type\":\"PushEvent\",\"actor\":{\"id\":779660,\"login\":\"KonstantinKo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KonstantinKo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/779660?\"},\"repo\":{\"id\":24053165,\"name\":\"clarat-org/clarat\",\"url\":\"https://api.github.com/repos/clarat-org/clarat\"},\"payload\":{\"push_id\":536753242,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f4dab9dbdc4c429106c3480b9f1ad18a5c9808a0\",\"before\":\"9028d94d202c91452636053bfbd2df6eef61e707\",\"commits\":[{\"sha\":\"f4dab9dbdc4c429106c3480b9f1ad18a5c9808a0\",\"author\":{\"email\":\"23db271f4238abbd1fb089f3166866050f7291f6@gmail.com\",\"name\":\"Konstantin K\"},\"message\":\"for #58 - ajaxify report overlay (& other ajax improvements and fixes)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/clarat-org/clarat/commits/f4dab9dbdc4c429106c3480b9f1ad18a5c9808a0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:25Z\",\"org\":{\"id\":8734408,\"login\":\"clarat-org\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/clarat-org\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8734408?\"}}\n{\"id\":\"2489398167\",\"type\":\"PushEvent\",\"actor\":{\"id\":744293,\"login\":\"aatxe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aatxe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/744293?\"},\"repo\":{\"id\":23890761,\"name\":\"aatxe/irc\",\"url\":\"https://api.github.com/repos/aatxe/irc\"},\"payload\":{\"push_id\":536753243,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a6146bc1357831d76833acf7e0c9d56f55f376c7\",\"before\":\"202748e7a91a52c2ea0afd033dfd6fe7291c98b7\",\"commits\":[{\"sha\":\"a6146bc1357831d76833acf7e0c9d56f55f376c7\",\"author\":{\"email\":\"2a84587e183d6a81cbf0d140ef37a969d7915794@gmail.com\",\"name\":\"Aaron Weiss\"},\"message\":\"Clean up after #11, including unit tests.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aatxe/irc/commits/a6146bc1357831d76833acf7e0c9d56f55f376c7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:25Z\"}\n{\"id\":\"2489398168\",\"type\":\"PushEvent\",\"actor\":{\"id\":4379694,\"login\":\"moongato\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/moongato\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4379694?\"},\"repo\":{\"id\":11769015,\"name\":\"moongato/android_frameworks_base\",\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base\"},\"payload\":{\"push_id\":536753244,\"size\":6,\"distinct_size\":1,\"ref\":\"refs/heads/lp50x-test\",\"head\":\"af8fdeaf0cba8a8c0837cdf76121e5e1c3efec34\",\"before\":\"73c3f8da9962437774cec8442e0da8d431c28fa8\",\"commits\":[{\"sha\":\"09ed328c580340d67492f81d855f8e2fe5834bd1\",\"author\":{\"email\":\"97bd659605de2dc1baed2f2e8ef4e483c88d27f8@gmail.com\",\"name\":\"Austin T. Conn\"},\"message\":\"volume rocker music controls\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/09ed328c580340d67492f81d855f8e2fe5834bd1\"},{\"sha\":\"b3c8f58fb81a604be3f1fd2913e005afe300e296\",\"author\":{\"email\":\"9defb1d2166fe02be9724618510318ef57e74e91@gmail.com\",\"name\":\"Michael Bestas\"},\"message\":\"Fix volume rocker music controls and wake up\\n\\n- Forward port code from cm-11.0 and adjust for 5.0\\n- Fix not being able to adjust volume when music control is on\\n- Disable screen off volume/music control when wake key is enabled\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/b3c8f58fb81a604be3f1fd2913e005afe300e296\"},{\"sha\":\"8040e4188be23da640d89fc829e33651826a48c0\",\"author\":{\"email\":\"238a1843d81dd7fbe80b5c1b99515c4ba8c94d0d@cyngn.com\",\"name\":\"Roman Birg\"},\"message\":\"status bar: improve brightness slider behavior\\n\\nMimic the brightness slider behavior in the statusbar.\\nThis adds logic to make the statusbar slider also work with automatic brightness mode enabled and it will instead adjust the temporary automatic brightness overrride.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/8040e4188be23da640d89fc829e33651826a48c0\"},{\"sha\":\"58cadbda6cf16389eb849e5d9ed39eafb352d18a\",\"author\":{\"email\":\"419b20914ad8ee7cdbbc7674d01c492d53cd267e@gmail.com\",\"name\":\"Pawit Pornkitprasan\"},\"message\":\"status bar brightness: store value as int\\n\\nNon-automatic brightness value is stored as int, not float.\\n\\nSymptom: adjust the brightness in the status bar, the brightness slider in the notification bar will always be set to full\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/58cadbda6cf16389eb849e5d9ed39eafb352d18a\"},{\"sha\":\"e0fb35b9ed52d05ce74b1c1833e20af6cd649f3c\",\"author\":{\"email\":\"1e520a49e026effdcbd6ab603708edecd93fc284@gmail.com\",\"name\":\"Clyde Tan\"},\"message\":\"Keep quiet when volume keys are used to wake up device\\n\\n- Userspace will make a 'beep' with it receives a key up, so  consume that event as well.\\n- Removed wake key check in music control code as it will already be disabled here.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/e0fb35b9ed52d05ce74b1c1833e20af6cd649f3c\"},{\"sha\":\"af8fdeaf0cba8a8c0837cdf76121e5e1c3efec34\",\"author\":{\"email\":\"22e0f38e0fc64da9129ff9b9ef030b39415294a1@ubuntu\",\"name\":\"moongato\"},\"message\":\"Merge remote-tracking branch 'upstream/lollipop-ras-mr1' into lp50x-test\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_frameworks_base/commits/af8fdeaf0cba8a8c0837cdf76121e5e1c3efec34\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:25Z\"}\n{\"id\":\"2489398175\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1217681,\"login\":\"stoeffel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?\"},\"repo\":{\"id\":800115,\"name\":\"epeli/underscore.string\",\"url\":\"https://api.github.com/repos/epeli/underscore.string\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/epeli/underscore.string/issues/306\",\"labels_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/306/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/306/comments\",\"events_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/306/events\",\"html_url\":\"https://github.com/epeli/underscore.string/issues/306\",\"id\":41565393,\"number\":306,\"title\":\"._camelize('Sample Text') returning \\\"SampleText\\\", not \\\"sampleText\\\"\",\"user\":{\"login\":\"joshuahiggins\",\"id\":879413,\"avatar_url\":\"https://avatars.githubusercontent.com/u/879413?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joshuahiggins\",\"html_url\":\"https://github.com/joshuahiggins\",\"followers_url\":\"https://api.github.com/users/joshuahiggins/followers\",\"following_url\":\"https://api.github.com/users/joshuahiggins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joshuahiggins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joshuahiggins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joshuahiggins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joshuahiggins/orgs\",\"repos_url\":\"https://api.github.com/users/joshuahiggins/repos\",\"events_url\":\"https://api.github.com/users/joshuahiggins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joshuahiggins/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"stoeffel\",\"id\":1217681,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"html_url\":\"https://github.com/stoeffel\",\"followers_url\":\"https://api.github.com/users/stoeffel/followers\",\"following_url\":\"https://api.github.com/users/stoeffel/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/stoeffel/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/stoeffel/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/stoeffel/subscriptions\",\"organizations_url\":\"https://api.github.com/users/stoeffel/orgs\",\"repos_url\":\"https://api.github.com/users/stoeffel/repos\",\"events_url\":\"https://api.github.com/users/stoeffel/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/stoeffel/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/epeli/underscore.string/milestones/1\",\"labels_url\":\"https://api.github.com/repos/epeli/underscore.string/milestones/1/labels\",\"id\":905496,\"number\":1,\"title\":\"3.0\",\"description\":\"CommonJS modularization\",\"creator\":{\"login\":\"epeli\",\"id\":225712,\"avatar_url\":\"https://avatars.githubusercontent.com/u/225712?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/epeli\",\"html_url\":\"https://github.com/epeli\",\"followers_url\":\"https://api.github.com/users/epeli/followers\",\"following_url\":\"https://api.github.com/users/epeli/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/epeli/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/epeli/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/epeli/subscriptions\",\"organizations_url\":\"https://api.github.com/users/epeli/orgs\",\"repos_url\":\"https://api.github.com/users/epeli/repos\",\"events_url\":\"https://api.github.com/users/epeli/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/epeli/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":4,\"closed_issues\":12,\"state\":\"open\",\"created_at\":\"2014-12-16T12:21:34Z\",\"updated_at\":\"2015-01-01T01:05:25Z\",\"due_on\":null,\"closed_at\":null},\"comments\":11,\"created_at\":\"2014-08-30T15:24:38Z\",\"updated_at\":\"2015-01-01T01:05:25Z\",\"closed_at\":\"2015-01-01T01:05:25Z\",\"body\":\"The documentation is confusing, as it indicates the following (emphasis added):\\r\\n\\r\\n> Converts underscored or dasherized string to a camelized one. **Begins with a lower case letter** unless it starts with an underscore or **string**\\r\\n\\r\\nThe example that follows shows a dash utilized to prompt a lowercase letter, which is neither an underscore nor a string. Should the documentation read as `underscore or dash`? It wouldn't make sense that it starts with a lower case letter unless it starts with a string.\\r\\n\\r\\nCamelizing the phrase \\\"Sample Text\\\" should return \\\"sampleText\\\", not \\\"SampleText\\\". \"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:26Z\"}\n{\"id\":\"2489398176\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":1217681,\"login\":\"stoeffel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?\"},\"repo\":{\"id\":800115,\"name\":\"epeli/underscore.string\",\"url\":\"https://api.github.com/repos/epeli/underscore.string\"},\"payload\":{\"action\":\"closed\",\"number\":370,\"pull_request\":{\"url\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370\",\"id\":26636766,\"html_url\":\"https://github.com/epeli/underscore.string/pull/370\",\"diff_url\":\"https://github.com/epeli/underscore.string/pull/370.diff\",\"patch_url\":\"https://github.com/epeli/underscore.string/pull/370.patch\",\"issue_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/370\",\"number\":370,\"state\":\"closed\",\"locked\":false,\"title\":\"add force lowercase flag to camelize\",\"user\":{\"login\":\"stoeffel\",\"id\":1217681,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"html_url\":\"https://github.com/stoeffel\",\"followers_url\":\"https://api.github.com/users/stoeffel/followers\",\"following_url\":\"https://api.github.com/users/stoeffel/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/stoeffel/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/stoeffel/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/stoeffel/subscriptions\",\"organizations_url\":\"https://api.github.com/users/stoeffel/orgs\",\"repos_url\":\"https://api.github.com/users/stoeffel/repos\",\"events_url\":\"https://api.github.com/users/stoeffel/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/stoeffel/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"- add tests for capitalization of the first letter\\r\\n- add alias camelback for camelize(str, true)\\r\\n- add alias camelcase (analoge ruby)\\r\\n- merge duplicate test suite for camelize\\r\\n\\r\\ncloses #306\",\"created_at\":\"2014-12-29T09:15:54Z\",\"updated_at\":\"2015-01-01T01:05:26Z\",\"closed_at\":\"2015-01-01T01:05:26Z\",\"merged_at\":\"2015-01-01T01:05:25Z\",\"merge_commit_sha\":\"e5ccfbf9637aa84e69ccc2ba26a53170de0cdac6\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370/commits\",\"review_comments_url\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370/comments\",\"review_comment_url\":\"https://api.github.com/repos/epeli/underscore.string/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/370/comments\",\"statuses_url\":\"https://api.github.com/repos/epeli/underscore.string/statuses/9fa0bbce075f82af8a7cd96abac4a6a1ae718611\",\"head\":{\"label\":\"epeli:camelize-force-lowercase\",\"ref\":\"camelize-force-lowercase\",\"sha\":\"9fa0bbce075f82af8a7cd96abac4a6a1ae718611\",\"user\":{\"login\":\"epeli\",\"id\":225712,\"avatar_url\":\"https://avatars.githubusercontent.com/u/225712?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/epeli\",\"html_url\":\"https://github.com/epeli\",\"followers_url\":\"https://api.github.com/users/epeli/followers\",\"following_url\":\"https://api.github.com/users/epeli/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/epeli/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/epeli/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/epeli/subscriptions\",\"organizations_url\":\"https://api.github.com/users/epeli/orgs\",\"repos_url\":\"https://api.github.com/users/epeli/repos\",\"events_url\":\"https://api.github.com/users/epeli/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/epeli/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":800115,\"name\":\"underscore.string\",\"full_name\":\"epeli/underscore.string\",\"owner\":{\"login\":\"epeli\",\"id\":225712,\"avatar_url\":\"https://avatars.githubusercontent.com/u/225712?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/epeli\",\"html_url\":\"https://github.com/epeli\",\"followers_url\":\"https://api.github.com/users/epeli/followers\",\"following_url\":\"https://api.github.com/users/epeli/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/epeli/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/epeli/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/epeli/subscriptions\",\"organizations_url\":\"https://api.github.com/users/epeli/orgs\",\"repos_url\":\"https://api.github.com/users/epeli/repos\",\"events_url\":\"https://api.github.com/users/epeli/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/epeli/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/epeli/underscore.string\",\"description\":\"String manipulation helpers for javascript\",\"fork\":false,\"url\":\"https://api.github.com/repos/epeli/underscore.string\",\"forks_url\":\"https://api.github.com/repos/epeli/underscore.string/forks\",\"keys_url\":\"https://api.github.com/repos/epeli/underscore.string/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/epeli/underscore.string/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/epeli/underscore.string/teams\",\"hooks_url\":\"https://api.github.com/repos/epeli/underscore.string/hooks\",\"issue_events_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/epeli/underscore.string/events\",\"assignees_url\":\"https://api.github.com/repos/epeli/underscore.string/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/epeli/underscore.string/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/epeli/underscore.string/tags\",\"blobs_url\":\"https://api.github.com/repos/epeli/underscore.string/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/epeli/underscore.string/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/epeli/underscore.string/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/epeli/underscore.string/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/epeli/underscore.string/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/epeli/underscore.string/languages\",\"stargazers_url\":\"https://api.github.com/repos/epeli/underscore.string/stargazers\",\"contributors_url\":\"https://api.github.com/repos/epeli/underscore.string/contributors\",\"subscribers_url\":\"https://api.github.com/repos/epeli/underscore.string/subscribers\",\"subscription_url\":\"https://api.github.com/repos/epeli/underscore.string/subscription\",\"commits_url\":\"https://api.github.com/repos/epeli/underscore.string/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/epeli/underscore.string/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/epeli/underscore.string/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/epeli/underscore.string/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/epeli/underscore.string/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/epeli/underscore.string/merges\",\"archive_url\":\"https://api.github.com/repos/epeli/underscore.string/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/epeli/underscore.string/downloads\",\"issues_url\":\"https://api.github.com/repos/epeli/underscore.string/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/epeli/underscore.string/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/epeli/underscore.string/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/epeli/underscore.string/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/epeli/underscore.string/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/epeli/underscore.string/releases{/id}\",\"created_at\":\"2010-07-27T08:01:42Z\",\"updated_at\":\"2014-12-31T03:37:22Z\",\"pushed_at\":\"2015-01-01T01:05:25Z\",\"git_url\":\"git://github.com/epeli/underscore.string.git\",\"ssh_url\":\"git@github.com:epeli/underscore.string.git\",\"clone_url\":\"https://github.com/epeli/underscore.string.git\",\"svn_url\":\"https://github.com/epeli/underscore.string\",\"homepage\":\"http://epeli.github.com/underscore.string/\",\"size\":5116,\"stargazers_count\":2443,\"watchers_count\":2443,\"language\":\"JavaScript\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":true,\"forks_count\":307,\"mirror_url\":null,\"open_issues_count\":102,\"forks\":307,\"open_issues\":102,\"watchers\":2443,\"default_branch\":\"master\"}},\"base\":{\"label\":\"epeli:master\",\"ref\":\"master\",\"sha\":\"244b1724770947430793dc43fb96d610dd045c47\",\"user\":{\"login\":\"epeli\",\"id\":225712,\"avatar_url\":\"https://avatars.githubusercontent.com/u/225712?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/epeli\",\"html_url\":\"https://github.com/epeli\",\"followers_url\":\"https://api.github.com/users/epeli/followers\",\"following_url\":\"https://api.github.com/users/epeli/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/epeli/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/epeli/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/epeli/subscriptions\",\"organizations_url\":\"https://api.github.com/users/epeli/orgs\",\"repos_url\":\"https://api.github.com/users/epeli/repos\",\"events_url\":\"https://api.github.com/users/epeli/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/epeli/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":800115,\"name\":\"underscore.string\",\"full_name\":\"epeli/underscore.string\",\"owner\":{\"login\":\"epeli\",\"id\":225712,\"avatar_url\":\"https://avatars.githubusercontent.com/u/225712?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/epeli\",\"html_url\":\"https://github.com/epeli\",\"followers_url\":\"https://api.github.com/users/epeli/followers\",\"following_url\":\"https://api.github.com/users/epeli/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/epeli/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/epeli/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/epeli/subscriptions\",\"organizations_url\":\"https://api.github.com/users/epeli/orgs\",\"repos_url\":\"https://api.github.com/users/epeli/repos\",\"events_url\":\"https://api.github.com/users/epeli/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/epeli/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/epeli/underscore.string\",\"description\":\"String manipulation helpers for javascript\",\"fork\":false,\"url\":\"https://api.github.com/repos/epeli/underscore.string\",\"forks_url\":\"https://api.github.com/repos/epeli/underscore.string/forks\",\"keys_url\":\"https://api.github.com/repos/epeli/underscore.string/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/epeli/underscore.string/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/epeli/underscore.string/teams\",\"hooks_url\":\"https://api.github.com/repos/epeli/underscore.string/hooks\",\"issue_events_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/epeli/underscore.string/events\",\"assignees_url\":\"https://api.github.com/repos/epeli/underscore.string/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/epeli/underscore.string/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/epeli/underscore.string/tags\",\"blobs_url\":\"https://api.github.com/repos/epeli/underscore.string/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/epeli/underscore.string/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/epeli/underscore.string/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/epeli/underscore.string/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/epeli/underscore.string/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/epeli/underscore.string/languages\",\"stargazers_url\":\"https://api.github.com/repos/epeli/underscore.string/stargazers\",\"contributors_url\":\"https://api.github.com/repos/epeli/underscore.string/contributors\",\"subscribers_url\":\"https://api.github.com/repos/epeli/underscore.string/subscribers\",\"subscription_url\":\"https://api.github.com/repos/epeli/underscore.string/subscription\",\"commits_url\":\"https://api.github.com/repos/epeli/underscore.string/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/epeli/underscore.string/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/epeli/underscore.string/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/epeli/underscore.string/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/epeli/underscore.string/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/epeli/underscore.string/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/epeli/underscore.string/merges\",\"archive_url\":\"https://api.github.com/repos/epeli/underscore.string/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/epeli/underscore.string/downloads\",\"issues_url\":\"https://api.github.com/repos/epeli/underscore.string/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/epeli/underscore.string/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/epeli/underscore.string/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/epeli/underscore.string/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/epeli/underscore.string/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/epeli/underscore.string/releases{/id}\",\"created_at\":\"2010-07-27T08:01:42Z\",\"updated_at\":\"2014-12-31T03:37:22Z\",\"pushed_at\":\"2015-01-01T01:05:25Z\",\"git_url\":\"git://github.com/epeli/underscore.string.git\",\"ssh_url\":\"git@github.com:epeli/underscore.string.git\",\"clone_url\":\"https://github.com/epeli/underscore.string.git\",\"svn_url\":\"https://github.com/epeli/underscore.string\",\"homepage\":\"http://epeli.github.com/underscore.string/\",\"size\":5116,\"stargazers_count\":2443,\"watchers_count\":2443,\"language\":\"JavaScript\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":true,\"forks_count\":307,\"mirror_url\":null,\"open_issues_count\":102,\"forks\":307,\"open_issues\":102,\"watchers\":2443,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370\"},\"html\":{\"href\":\"https://github.com/epeli/underscore.string/pull/370\"},\"issue\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/issues/370\"},\"comments\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/issues/370/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/pulls/370/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/epeli/underscore.string/statuses/9fa0bbce075f82af8a7cd96abac4a6a1ae718611\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"stoeffel\",\"id\":1217681,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"html_url\":\"https://github.com/stoeffel\",\"followers_url\":\"https://api.github.com/users/stoeffel/followers\",\"following_url\":\"https://api.github.com/users/stoeffel/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/stoeffel/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/stoeffel/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/stoeffel/subscriptions\",\"organizations_url\":\"https://api.github.com/users/stoeffel/orgs\",\"repos_url\":\"https://api.github.com/users/stoeffel/repos\",\"events_url\":\"https://api.github.com/users/stoeffel/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/stoeffel/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":9,\"review_comments\":0,\"commits\":1,\"additions\":55,\"deletions\":32,\"changed_files\":4}},\"public\":true,\"created_at\":\"2015-01-01T01:05:26Z\"}\n{\"id\":\"2489398183\",\"type\":\"PushEvent\",\"actor\":{\"id\":745333,\"login\":\"mkeeter\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mkeeter\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/745333?\"},\"repo\":{\"id\":15458279,\"name\":\"mkeeter/antimony\",\"url\":\"https://api.github.com/repos/mkeeter/antimony\"},\"payload\":{\"push_id\":536753246,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"af931f7902c93ac10b7f8c5f4bd24130448792da\",\"before\":\"63294530fca3218ef2e545af86172024100fac7f\",\"commits\":[{\"sha\":\"f466d66f8f5e3d3258cfa456221f0d296cb5519d\",\"author\":{\"email\":\"b96049b2cf1cc4f51fb34d3c0383273ec5d20919@gmail.com\",\"name\":\"Matt Keeter\"},\"message\":\"Prevent DummyControl from blocking everything behind it\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mkeeter/antimony/commits/f466d66f8f5e3d3258cfa456221f0d296cb5519d\"},{\"sha\":\"e8c74daafe41860bac6b63829e7488e94e951794\",\"author\":{\"email\":\"b96049b2cf1cc4f51fb34d3c0383273ec5d20919@gmail.com\",\"name\":\"Matt Keeter\"},\"message\":\"Make ControlProxies hierarchical (matching Controls)\\n\\nThis is nice because it makes the raise dialog only find one\\nControlProxy when you right-click on something like a triangle's corner\\n(where the line and corner point overlap)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mkeeter/antimony/commits/e8c74daafe41860bac6b63829e7488e94e951794\"},{\"sha\":\"af931f7902c93ac10b7f8c5f4bd24130448792da\",\"author\":{\"email\":\"b96049b2cf1cc4f51fb34d3c0383273ec5d20919@gmail.com\",\"name\":\"Matt Keeter\"},\"message\":\"Only open raise menu if more than one control are overlapping\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mkeeter/antimony/commits/af931f7902c93ac10b7f8c5f4bd24130448792da\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:26Z\"}\n{\"id\":\"2489398189\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397291\",\"id\":22397291,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\\n \\n - name: custom horizon splash logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png mode=0644 force=yes\\n \\n - name: custom horizon favicon\\n-  get_url: url={{ horizon.favicon_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico\\n-           force=yes\\n+  get_url: url={{ horizon.favicon_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico force=yes\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":135,\"original_position\":135,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Same for the other two get_urls above.\",\"created_at\":\"2015-01-01T01:05:27Z\",\"updated_at\":\"2015-01-01T01:05:27Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397291\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397291\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397291\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:05:27Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:05:27Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489398191\",\"type\":\"PushEvent\",\"actor\":{\"id\":7483224,\"login\":\"seann1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/seann1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7483224?\"},\"repo\":{\"id\":28437213,\"name\":\"seann1/twitterapi-rails\",\"url\":\"https://api.github.com/repos/seann1/twitterapi-rails\"},\"payload\":{\"push_id\":536753252,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9f6d2ffe96a98af38f87ef905762e43893a8c461\",\"before\":\"7a83e390ab1d5cb5b00d4136c8e01b0e81ae49f9\",\"commits\":[{\"sha\":\"9f6d2ffe96a98af38f87ef905762e43893a8c461\",\"author\":{\"email\":\"3cea802b5add67a11ab310afda68113858c6259a@gmail.com\",\"name\":\"Sean\"},\"message\":\"dynamically creating bubble divs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/seann1/twitterapi-rails/commits/9f6d2ffe96a98af38f87ef905762e43893a8c461\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:27Z\"}\n{\"id\":\"2489398192\",\"type\":\"GollumEvent\",\"actor\":{\"id\":10245688,\"login\":\"SunyataZero\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SunyataZero\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10245688?\"},\"repo\":{\"id\":28485643,\"name\":\"EmpathyApp/EmpathyApp\",\"url\":\"https://api.github.com/repos/EmpathyApp/EmpathyApp\"},\"payload\":{\"pages\":[{\"page_name\":\"Design-tools\",\"title\":\"Design tools\",\"summary\":null,\"action\":\"created\",\"sha\":\"b8d60e0fc553722c073232af866539a4766571a6\",\"html_url\":\"https://github.com/EmpathyApp/EmpathyApp/wiki/Design-tools\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:27Z\",\"org\":{\"id\":10245750,\"login\":\"EmpathyApp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/EmpathyApp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10245750?\"}}\n{\"id\":\"2489398196\",\"type\":\"PushEvent\",\"actor\":{\"id\":172753,\"login\":\"oehmiche\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/oehmiche\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/172753?\"},\"repo\":{\"id\":26650146,\"name\":\"oehmiche/isp-performance\",\"url\":\"https://api.github.com/repos/oehmiche/isp-performance\"},\"payload\":{\"push_id\":536753253,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"7cfa6900cc30482ca7d8b579f45af6d8ec90e117\",\"before\":\"ce82142429c758277343abf359a05a410186264d\",\"commits\":[{\"sha\":\"7cfa6900cc30482ca7d8b579f45af6d8ec90e117\",\"author\":{\"email\":\"37955833ebc42115603c87ad993837f95227dc9d@raspberry.pi\",\"name\":\"smokeping.script\"},\"message\":\"updates all charts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/oehmiche/isp-performance/commits/7cfa6900cc30482ca7d8b579f45af6d8ec90e117\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:28Z\"}\n{\"id\":\"2489398198\",\"type\":\"PushEvent\",\"actor\":{\"id\":1217681,\"login\":\"stoeffel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?\"},\"repo\":{\"id\":800115,\"name\":\"epeli/underscore.string\",\"url\":\"https://api.github.com/repos/epeli/underscore.string\"},\"payload\":{\"push_id\":536753254,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cebddf40cf2e10f0e9b596d9654edd0a1cfefc15\",\"before\":\"244b1724770947430793dc43fb96d610dd045c47\",\"commits\":[{\"sha\":\"9fa0bbce075f82af8a7cd96abac4a6a1ae718611\",\"author\":{\"email\":\"3879ab0c8adeefa545d20a45d877af2f6b5284b0@tocco.ch\",\"name\":\"Christoph Hermann\"},\"message\":\"add force lowercase flag to camelize\\n\\n- add tests for capitalization of the first letter\\n- add alias camelcase (analoge ruby)\\n- merge duplicate test suite for camelize\\n\\ncloses #306\",\"distinct\":false,\"url\":\"https://api.github.com/repos/epeli/underscore.string/commits/9fa0bbce075f82af8a7cd96abac4a6a1ae718611\"},{\"sha\":\"cebddf40cf2e10f0e9b596d9654edd0a1cfefc15\",\"author\":{\"email\":\"02503ea0b8f4cf66a2b1602591dbce46860393b2@gmail.com\",\"name\":\"Christoph Hermann\"},\"message\":\"Merge pull request #370 from epeli/camelize-force-lowercase\\n\\nadd force lowercase flag to camelize\",\"distinct\":true,\"url\":\"https://api.github.com/repos/epeli/underscore.string/commits/cebddf40cf2e10f0e9b596d9654edd0a1cfefc15\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:28Z\"}\n{\"id\":\"2489398199\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1203825,\"login\":\"huonw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/huonw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1203825?\"},\"repo\":{\"id\":724712,\"name\":\"rust-lang/rust\",\"url\":\"https://api.github.com/repos/rust-lang/rust\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/20373\",\"labels_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20373/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20373/comments\",\"events_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20373/events\",\"html_url\":\"https://github.com/rust-lang/rust/pull/20373\",\"id\":53190761,\"number\":20373,\"title\":\"Add a lint `unconditional_self_calls` to detect unconditional recursion.\",\"user\":{\"login\":\"huonw\",\"id\":1203825,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1203825?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/huonw\",\"html_url\":\"https://github.com/huonw\",\"followers_url\":\"https://api.github.com/users/huonw/followers\",\"following_url\":\"https://api.github.com/users/huonw/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/huonw/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/huonw/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/huonw/subscriptions\",\"organizations_url\":\"https://api.github.com/users/huonw/orgs\",\"repos_url\":\"https://api.github.com/users/huonw/repos\",\"events_url\":\"https://api.github.com/users/huonw/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/huonw/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"luqmana\",\"id\":287063,\"avatar_url\":\"https://avatars.githubusercontent.com/u/287063?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/luqmana\",\"html_url\":\"https://github.com/luqmana\",\"followers_url\":\"https://api.github.com/users/luqmana/followers\",\"following_url\":\"https://api.github.com/users/luqmana/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/luqmana/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/luqmana/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/luqmana/subscriptions\",\"organizations_url\":\"https://api.github.com/users/luqmana/orgs\",\"repos_url\":\"https://api.github.com/users/luqmana/repos\",\"events_url\":\"https://api.github.com/users/luqmana/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/luqmana/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-31T16:18:18Z\",\"updated_at\":\"2015-01-01T01:05:28Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/pulls/20373\",\"html_url\":\"https://github.com/rust-lang/rust/pull/20373\",\"diff_url\":\"https://github.com/rust-lang/rust/pull/20373.diff\",\"patch_url\":\"https://github.com/rust-lang/rust/pull/20373.patch\"},\"body\":\"E.g. `fn foo() { foo() }`, or, more subtlely\\r\\n\\r\\n    impl Foo for Box<Foo+'static> {\\r\\n        fn bar(&self) {\\r\\n            self.bar();\\r\\n        }\\r\\n    }\\r\\n\\r\\nThe compiler will warn and point out the points where recursion occurs,\\r\\nif it determines that the function cannot return without calling itself.\\r\\n\\r\\nCloses #17899.\\r\\n\\r\\n---\\r\\n\\r\\nThis is highly non-perfect, in particular, my wording above is quite precise, and I have some unresolved questions: This currently will warn about\\r\\n\\r\\n```rust\\r\\nfn foo() {\\r\\n    if bar { loop {} }\\r\\n\\r\\n    foo()\\r\\n}\\r\\n```\\r\\n\\r\\neven though `foo` may never be called (i.e. our apparent \\\"unconditional\\\" recursion is actually conditional). I don't know if we should handle this case, and ones like it with `panic!()` instead of `loop` (or anything else that \\\"returns\\\" `!`).\\r\\n\\r\\nHowever, strictly speaking, it seems to me that changing the above to not warn will require changing\\r\\n\\r\\n```rust\\r\\nfn foo() {\\r\\n    while bar {}\\r\\n    foo()\\r\\n}\\r\\n```\\r\\n\\r\\nto also not warn since it could be that the `while` is an infinite loop and doesn't ever hit the `foo`.\\r\\n\\r\\nI'm inclined to think we let these cases warn since true edge cases like the first one seem rare, and if they do occur they seem like they would usually be typos in the function call. (I could imagine someone accidentally having code like `fn foo() { assert!(bar()); foo() /* meant to be boo() */ }` which won't warn if the `loop` case is \\\"fixed\\\".)\\r\\n\\r\\n(Part of the reason this is unresolved is wanting feedback, part of the reason is I couldn't devise a strategy that worked in all cases.)\\r\\n\\r\\n---\\r\\n\\r\\nThe name `unconditional_self_calls` is kinda clunky; and this reconstructs the CFG for each function that is linted which may or may not be very expensive, I don't know.\"},\"comment\":{\"url\":\"https://api.github.com/repos/rust-lang/rust/issues/comments/68477337\",\"html_url\":\"https://github.com/rust-lang/rust/pull/20373#issuecomment-68477337\",\"issue_url\":\"https://api.github.com/repos/rust-lang/rust/issues/20373\",\"id\":68477337,\"user\":{\"login\":\"huonw\",\"id\":1203825,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1203825?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/huonw\",\"html_url\":\"https://github.com/huonw\",\"followers_url\":\"https://api.github.com/users/huonw/followers\",\"following_url\":\"https://api.github.com/users/huonw/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/huonw/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/huonw/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/huonw/subscriptions\",\"organizations_url\":\"https://api.github.com/users/huonw/orgs\",\"repos_url\":\"https://api.github.com/users/huonw/repos\",\"events_url\":\"https://api.github.com/users/huonw/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/huonw/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:28Z\",\"updated_at\":\"2015-01-01T01:05:28Z\",\"body\":\"@luqmana, speaking prescriptively, \\\"recurs\\\" is actually *more* correct than recurses, e.g. http://english.stackexchange.com/questions/163446/does-a-recursive-procedure-recur ; I'm happy to change it if people prefer the recurse version.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:29Z\",\"org\":{\"id\":5430905,\"login\":\"rust-lang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rust-lang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5430905?\"}}\n{\"id\":\"2489398200\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":1318988,\"name\":\"Instagram/python-instagram\",\"url\":\"https://api.github.com/repos/Instagram/python-instagram\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:29Z\",\"org\":{\"id\":549085,\"login\":\"Instagram\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Instagram\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/549085?\"}}\n{\"id\":\"2489398205\",\"type\":\"PushEvent\",\"actor\":{\"id\":9278781,\"login\":\"Missling\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Missling\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9278781?\"},\"repo\":{\"id\":27950087,\"name\":\"Missling/missling.github.io\",\"url\":\"https://api.github.com/repos/Missling/missling.github.io\"},\"payload\":{\"push_id\":536753258,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"837dcc81f2013cbfc2ded2f8e96ed6678e554e11\",\"before\":\"ac2686d840d1c7edd43896a8170e969483d1a261\",\"commits\":[{\"sha\":\"837dcc81f2013cbfc2ded2f8e96ed6678e554e11\",\"author\":{\"email\":\"75f3425216d1181ebb16fc73302673da505d7ee1@gmail.com\",\"name\":\"Ling Giang\"},\"message\":\"update website index page and blogcss\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Missling/missling.github.io/commits/837dcc81f2013cbfc2ded2f8e96ed6678e554e11\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:30Z\"}\n{\"id\":\"2489398207\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"action\":\"opened\",\"number\":31,\"pull_request\":{\"url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31\",\"id\":26739457,\"html_url\":\"https://github.com/captainkirkby/Gears/pull/31\",\"diff_url\":\"https://github.com/captainkirkby/Gears/pull/31.diff\",\"patch_url\":\"https://github.com/captainkirkby/Gears/pull/31.patch\",\"issue_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31\",\"number\":31,\"state\":\"open\",\"locked\":false,\"title\":\"Add simple C program to replay binary file\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Closes issue #30 \",\"created_at\":\"2015-01-01T01:05:31Z\",\"updated_at\":\"2015-01-01T01:05:31Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"1937489450eb244357356ec9a4ad968f1d63e23d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/commits\",\"review_comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/comments\",\"review_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31/comments\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"head\":{\"label\":\"captainkirkby:#30\",\"ref\":\"#30\",\"sha\":\"0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15461243,\"name\":\"Gears\",\"full_name\":\"captainkirkby/Gears\",\"owner\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/captainkirkby/Gears\",\"description\":\"Set of packages used to measure a mechanical clock.\",\"fork\":false,\"url\":\"https://api.github.com/repos/captainkirkby/Gears\",\"forks_url\":\"https://api.github.com/repos/captainkirkby/Gears/forks\",\"keys_url\":\"https://api.github.com/repos/captainkirkby/Gears/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/captainkirkby/Gears/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/captainkirkby/Gears/teams\",\"hooks_url\":\"https://api.github.com/repos/captainkirkby/Gears/hooks\",\"issue_events_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/captainkirkby/Gears/events\",\"assignees_url\":\"https://api.github.com/repos/captainkirkby/Gears/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/captainkirkby/Gears/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/tags\",\"blobs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/captainkirkby/Gears/languages\",\"stargazers_url\":\"https://api.github.com/repos/captainkirkby/Gears/stargazers\",\"contributors_url\":\"https://api.github.com/repos/captainkirkby/Gears/contributors\",\"subscribers_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscribers\",\"subscription_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscription\",\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/captainkirkby/Gears/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/captainkirkby/Gears/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/captainkirkby/Gears/merges\",\"archive_url\":\"https://api.github.com/repos/captainkirkby/Gears/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/captainkirkby/Gears/downloads\",\"issues_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/captainkirkby/Gears/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/captainkirkby/Gears/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/captainkirkby/Gears/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/captainkirkby/Gears/releases{/id}\",\"created_at\":\"2013-12-26T22:05:51Z\",\"updated_at\":\"2015-01-01T00:41:54Z\",\"pushed_at\":\"2015-01-01T01:02:09Z\",\"git_url\":\"git://github.com/captainkirkby/Gears.git\",\"ssh_url\":\"git@github.com:captainkirkby/Gears.git\",\"clone_url\":\"https://github.com/captainkirkby/Gears.git\",\"svn_url\":\"https://github.com/captainkirkby/Gears\",\"homepage\":\"\",\"size\":18140,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":12,\"forks\":0,\"open_issues\":12,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"captainkirkby:master\",\"ref\":\"master\",\"sha\":\"b838f0c36403eab209f3565eeb7a10375911d228\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15461243,\"name\":\"Gears\",\"full_name\":\"captainkirkby/Gears\",\"owner\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/captainkirkby/Gears\",\"description\":\"Set of packages used to measure a mechanical clock.\",\"fork\":false,\"url\":\"https://api.github.com/repos/captainkirkby/Gears\",\"forks_url\":\"https://api.github.com/repos/captainkirkby/Gears/forks\",\"keys_url\":\"https://api.github.com/repos/captainkirkby/Gears/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/captainkirkby/Gears/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/captainkirkby/Gears/teams\",\"hooks_url\":\"https://api.github.com/repos/captainkirkby/Gears/hooks\",\"issue_events_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/captainkirkby/Gears/events\",\"assignees_url\":\"https://api.github.com/repos/captainkirkby/Gears/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/captainkirkby/Gears/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/tags\",\"blobs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/captainkirkby/Gears/languages\",\"stargazers_url\":\"https://api.github.com/repos/captainkirkby/Gears/stargazers\",\"contributors_url\":\"https://api.github.com/repos/captainkirkby/Gears/contributors\",\"subscribers_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscribers\",\"subscription_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscription\",\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/captainkirkby/Gears/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/captainkirkby/Gears/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/captainkirkby/Gears/merges\",\"archive_url\":\"https://api.github.com/repos/captainkirkby/Gears/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/captainkirkby/Gears/downloads\",\"issues_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/captainkirkby/Gears/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/captainkirkby/Gears/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/captainkirkby/Gears/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/captainkirkby/Gears/releases{/id}\",\"created_at\":\"2013-12-26T22:05:51Z\",\"updated_at\":\"2015-01-01T00:41:54Z\",\"pushed_at\":\"2015-01-01T01:02:09Z\",\"git_url\":\"git://github.com/captainkirkby/Gears.git\",\"ssh_url\":\"git@github.com:captainkirkby/Gears.git\",\"clone_url\":\"https://github.com/captainkirkby/Gears.git\",\"svn_url\":\"https://github.com/captainkirkby/Gears\",\"homepage\":\"\",\"size\":18140,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":12,\"forks\":0,\"open_issues\":12,\"watchers\":1,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31\"},\"html\":{\"href\":\"https://github.com/captainkirkby/Gears/pull/31\"},\"issue\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31\"},\"comments\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/0ce69fa19d81de656dd6a74629099a7fa9261d1c\"}},\"merged\":false,\"mergeable\":true,\"mergeable_state\":\"clean\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":129,\"deletions\":0,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\"}\n{\"id\":\"2489398208\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1817530,\"login\":\"mitchellw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mitchellw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1817530?\"},\"repo\":{\"id\":27339405,\"name\":\"QBCo/akemi\",\"url\":\"https://api.github.com/repos/QBCo/akemi\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\",\"org\":{\"id\":10001822,\"login\":\"QBCo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/QBCo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10001822?\"}}\n{\"id\":\"2489398215\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1802542,\"login\":\"ishayyaari\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?\"},\"repo\":{\"id\":4967600,\"name\":\"MiYa-Solutions/sbcx\",\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/715\",\"labels_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/715/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/715/comments\",\"events_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/715/events\",\"html_url\":\"https://github.com/MiYa-Solutions/sbcx/issues/715\",\"id\":53198063,\"number\":715,\"title\":\"Feedback From Rug Wash Meeting Dec 31 2014\",\"user\":{\"login\":\"markmilman\",\"id\":1744318,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1744318?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/markmilman\",\"html_url\":\"https://github.com/markmilman\",\"followers_url\":\"https://api.github.com/users/markmilman/followers\",\"following_url\":\"https://api.github.com/users/markmilman/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/markmilman/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/markmilman/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/markmilman/subscriptions\",\"organizations_url\":\"https://api.github.com/users/markmilman/orgs\",\"repos_url\":\"https://api.github.com/users/markmilman/repos\",\"events_url\":\"https://api.github.com/users/markmilman/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/markmilman/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"ishayyaari\",\"id\":1802542,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"html_url\":\"https://github.com/ishayyaari\",\"followers_url\":\"https://api.github.com/users/ishayyaari/followers\",\"following_url\":\"https://api.github.com/users/ishayyaari/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ishayyaari/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ishayyaari/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ishayyaari/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ishayyaari/orgs\",\"repos_url\":\"https://api.github.com/users/ishayyaari/repos\",\"events_url\":\"https://api.github.com/users/ishayyaari/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ishayyaari/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/milestones/10\",\"labels_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/milestones/10/labels\",\"id\":835988,\"number\":10,\"title\":\"Rug Wash Bug Fixes\",\"description\":\"Fixes specifically for the rug wash opportunity\",\"creator\":{\"login\":\"markmilman\",\"id\":1744318,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1744318?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/markmilman\",\"html_url\":\"https://github.com/markmilman\",\"followers_url\":\"https://api.github.com/users/markmilman/followers\",\"following_url\":\"https://api.github.com/users/markmilman/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/markmilman/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/markmilman/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/markmilman/subscriptions\",\"organizations_url\":\"https://api.github.com/users/markmilman/orgs\",\"repos_url\":\"https://api.github.com/users/markmilman/repos\",\"events_url\":\"https://api.github.com/users/markmilman/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/markmilman/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":6,\"closed_issues\":37,\"state\":\"open\",\"created_at\":\"2014-10-22T14:08:56Z\",\"updated_at\":\"2015-01-01T01:05:32Z\",\"due_on\":\"2014-11-11T08:00:00Z\",\"closed_at\":null},\"comments\":0,\"created_at\":\"2014-12-31T18:58:02Z\",\"updated_at\":\"2015-01-01T01:05:32Z\",\"closed_at\":\"2015-01-01T01:05:32Z\",\"body\":\"* Show the Project Number (Order Number)\\r\\n* Include Job #  in the invoice\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\",\"org\":{\"id\":1788572,\"login\":\"MiYa-Solutions\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MiYa-Solutions\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1788572?\"}}\n{\"id\":\"2489398217\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753264,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ca0474a33dcc125aa7204b517dd7cef8003db346\",\"before\":\"a2211fba71212268a1c0f0b3dc4ddb1412bf72b6\",\"commits\":[{\"sha\":\"ca0474a33dcc125aa7204b517dd7cef8003db346\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"0.2.1\\n\\nConflicts:\\n\\tpackage.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/ca0474a33dcc125aa7204b517dd7cef8003db346\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398218\",\"type\":\"PushEvent\",\"actor\":{\"id\":1802542,\"login\":\"ishayyaari\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?\"},\"repo\":{\"id\":4967600,\"name\":\"MiYa-Solutions/sbcx\",\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx\"},\"payload\":{\"push_id\":536753265,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/development\",\"head\":\"ead55f938ace15919d2bd2b15f1cde4ef5105ced\",\"before\":\"3f21c270c34c90f4c7c43cf053dde0edd3841782\",\"commits\":[{\"sha\":\"ead55f938ace15919d2bd2b15f1cde4ef5105ced\",\"author\":{\"email\":\"9534b3c0ec96f8059b6f52efee60e93b9c6b42be\",\"name\":\"ishayyaari\"},\"message\":\"Feedback From Rug Wash Meeting Dec 31 2014\\n\\nFixed #715\",\"distinct\":true,\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/commits/ead55f938ace15919d2bd2b15f1cde4ef5105ced\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\",\"org\":{\"id\":1788572,\"login\":\"MiYa-Solutions\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MiYa-Solutions\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1788572?\"}}\n{\"id\":\"2489398219\",\"type\":\"PushEvent\",\"actor\":{\"id\":1133652,\"login\":\"keum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/keum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1133652?\"},\"repo\":{\"id\":18115347,\"name\":\"keum/data_display\",\"url\":\"https://api.github.com/repos/keum/data_display\"},\"payload\":{\"push_id\":536753266,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a6eee972e4e07e37d6731216a5d653016c300784\",\"before\":\"4e4c11b0947a21977b8eb995000a211cc06fa070\",\"commits\":[{\"sha\":\"a6eee972e4e07e37d6731216a5d653016c300784\",\"author\":{\"email\":\"77be1f27789ed5ba740c04e6493c37160e58f28c@gmail.com\",\"name\":\"Peter Keum\"},\"message\":\"\\\"Data Upload: 2014-12-31 05:05:29 PM\\\"\",\"distinct\":true,\"url\":\"https://api.github.com/repos/keum/data_display/commits/a6eee972e4e07e37d6731216a5d653016c300784\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\"}\n{\"id\":\"2489398220\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":19792,\"login\":\"wiredfool\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?\"},\"repo\":{\"id\":5171600,\"name\":\"python-pillow/Pillow\",\"url\":\"https://api.github.com/repos/python-pillow/Pillow\"},\"payload\":{\"action\":\"opened\",\"number\":1062,\"pull_request\":{\"url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062\",\"id\":26739458,\"html_url\":\"https://github.com/python-pillow/Pillow/pull/1062\",\"diff_url\":\"https://github.com/python-pillow/Pillow/pull/1062.diff\",\"patch_url\":\"https://github.com/python-pillow/Pillow/pull/1062.patch\",\"issue_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1062\",\"number\":1062,\"state\":\"open\",\"locked\":false,\"title\":\"Png text decompression dos fix, for 2.6.x\",\"user\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Same as in master, but for 2.6.x\",\"created_at\":\"2015-01-01T01:05:32Z\",\"updated_at\":\"2015-01-01T01:05:32Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062/commits\",\"review_comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062/comments\",\"review_comment_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1062/comments\",\"statuses_url\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/cf880329a755f5b5c81df661990105593adf4a37\",\"head\":{\"label\":\"wiredfool:png-dos-2.6.1\",\"ref\":\"png-dos-2.6.1\",\"sha\":\"cf880329a755f5b5c81df661990105593adf4a37\",\"user\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":8664379,\"name\":\"Pillow\",\"full_name\":\"wiredfool/Pillow\",\"owner\":{\"login\":\"wiredfool\",\"id\":19792,\"avatar_url\":\"https://avatars.githubusercontent.com/u/19792?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiredfool\",\"html_url\":\"https://github.com/wiredfool\",\"followers_url\":\"https://api.github.com/users/wiredfool/followers\",\"following_url\":\"https://api.github.com/users/wiredfool/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/wiredfool/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/wiredfool/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/wiredfool/subscriptions\",\"organizations_url\":\"https://api.github.com/users/wiredfool/orgs\",\"repos_url\":\"https://api.github.com/users/wiredfool/repos\",\"events_url\":\"https://api.github.com/users/wiredfool/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/wiredfool/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/wiredfool/Pillow\",\"description\":\"Pillow is the \\\"friendly\\\" PIL fork\",\"fork\":true,\"url\":\"https://api.github.com/repos/wiredfool/Pillow\",\"forks_url\":\"https://api.github.com/repos/wiredfool/Pillow/forks\",\"keys_url\":\"https://api.github.com/repos/wiredfool/Pillow/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/wiredfool/Pillow/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/wiredfool/Pillow/teams\",\"hooks_url\":\"https://api.github.com/repos/wiredfool/Pillow/hooks\",\"issue_events_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/wiredfool/Pillow/events\",\"assignees_url\":\"https://api.github.com/repos/wiredfool/Pillow/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/wiredfool/Pillow/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/wiredfool/Pillow/tags\",\"blobs_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/wiredfool/Pillow/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/wiredfool/Pillow/languages\",\"stargazers_url\":\"https://api.github.com/repos/wiredfool/Pillow/stargazers\",\"contributors_url\":\"https://api.github.com/repos/wiredfool/Pillow/contributors\",\"subscribers_url\":\"https://api.github.com/repos/wiredfool/Pillow/subscribers\",\"subscription_url\":\"https://api.github.com/repos/wiredfool/Pillow/subscription\",\"commits_url\":\"https://api.github.com/repos/wiredfool/Pillow/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/wiredfool/Pillow/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/wiredfool/Pillow/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/wiredfool/Pillow/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/wiredfool/Pillow/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/wiredfool/Pillow/merges\",\"archive_url\":\"https://api.github.com/repos/wiredfool/Pillow/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/wiredfool/Pillow/downloads\",\"issues_url\":\"https://api.github.com/repos/wiredfool/Pillow/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/wiredfool/Pillow/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/wiredfool/Pillow/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/wiredfool/Pillow/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/wiredfool/Pillow/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/wiredfool/Pillow/releases{/id}\",\"created_at\":\"2013-03-09T03:12:08Z\",\"updated_at\":\"2014-12-31T23:15:11Z\",\"pushed_at\":\"2015-01-01T00:37:45Z\",\"git_url\":\"git://github.com/wiredfool/Pillow.git\",\"ssh_url\":\"git@github.com:wiredfool/Pillow.git\",\"clone_url\":\"https://github.com/wiredfool/Pillow.git\",\"svn_url\":\"https://github.com/wiredfool/Pillow\",\"homepage\":\"http://python-imaging.github.com\",\"size\":14870,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"Python\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"python-pillow:2.6.x\",\"ref\":\"2.6.x\",\"sha\":\"4a8471dea18f6196161e4444ce5625f46cecd1e1\",\"user\":{\"login\":\"python-pillow\",\"id\":2036701,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/python-pillow\",\"html_url\":\"https://github.com/python-pillow\",\"followers_url\":\"https://api.github.com/users/python-pillow/followers\",\"following_url\":\"https://api.github.com/users/python-pillow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/python-pillow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/python-pillow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/python-pillow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/python-pillow/orgs\",\"repos_url\":\"https://api.github.com/users/python-pillow/repos\",\"events_url\":\"https://api.github.com/users/python-pillow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/python-pillow/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":5171600,\"name\":\"Pillow\",\"full_name\":\"python-pillow/Pillow\",\"owner\":{\"login\":\"python-pillow\",\"id\":2036701,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/python-pillow\",\"html_url\":\"https://github.com/python-pillow\",\"followers_url\":\"https://api.github.com/users/python-pillow/followers\",\"following_url\":\"https://api.github.com/users/python-pillow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/python-pillow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/python-pillow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/python-pillow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/python-pillow/orgs\",\"repos_url\":\"https://api.github.com/users/python-pillow/repos\",\"events_url\":\"https://api.github.com/users/python-pillow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/python-pillow/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/python-pillow/Pillow\",\"description\":\"The friendly PIL fork\",\"fork\":false,\"url\":\"https://api.github.com/repos/python-pillow/Pillow\",\"forks_url\":\"https://api.github.com/repos/python-pillow/Pillow/forks\",\"keys_url\":\"https://api.github.com/repos/python-pillow/Pillow/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/python-pillow/Pillow/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/python-pillow/Pillow/teams\",\"hooks_url\":\"https://api.github.com/repos/python-pillow/Pillow/hooks\",\"issue_events_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/python-pillow/Pillow/events\",\"assignees_url\":\"https://api.github.com/repos/python-pillow/Pillow/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/python-pillow/Pillow/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/python-pillow/Pillow/tags\",\"blobs_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/python-pillow/Pillow/languages\",\"stargazers_url\":\"https://api.github.com/repos/python-pillow/Pillow/stargazers\",\"contributors_url\":\"https://api.github.com/repos/python-pillow/Pillow/contributors\",\"subscribers_url\":\"https://api.github.com/repos/python-pillow/Pillow/subscribers\",\"subscription_url\":\"https://api.github.com/repos/python-pillow/Pillow/subscription\",\"commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/python-pillow/Pillow/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/python-pillow/Pillow/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/python-pillow/Pillow/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/python-pillow/Pillow/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/python-pillow/Pillow/merges\",\"archive_url\":\"https://api.github.com/repos/python-pillow/Pillow/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/python-pillow/Pillow/downloads\",\"issues_url\":\"https://api.github.com/repos/python-pillow/Pillow/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/python-pillow/Pillow/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/python-pillow/Pillow/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/python-pillow/Pillow/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/python-pillow/Pillow/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/python-pillow/Pillow/releases{/id}\",\"created_at\":\"2012-07-24T21:38:39Z\",\"updated_at\":\"2014-12-31T22:44:33Z\",\"pushed_at\":\"2014-12-31T22:44:33Z\",\"git_url\":\"git://github.com/python-pillow/Pillow.git\",\"ssh_url\":\"git@github.com:python-pillow/Pillow.git\",\"clone_url\":\"https://github.com/python-pillow/Pillow.git\",\"svn_url\":\"https://github.com/python-pillow/Pillow\",\"homepage\":\"http://python-pillow.github.io/\",\"size\":20365,\"stargazers_count\":1391,\"watchers_count\":1391,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":319,\"mirror_url\":null,\"open_issues_count\":65,\"forks\":319,\"open_issues\":65,\"watchers\":1391,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062\"},\"html\":{\"href\":\"https://github.com/python-pillow/Pillow/pull/1062\"},\"issue\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1062\"},\"comments\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/issues/1062/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/pulls/1062/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/python-pillow/Pillow/statuses/cf880329a755f5b5c81df661990105593adf4a37\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":6,\"additions\":95,\"deletions\":12,\"changed_files\":9}},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\",\"org\":{\"id\":2036701,\"login\":\"python-pillow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/python-pillow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2036701?\"}}\n{\"id\":\"2489398223\",\"type\":\"ReleaseEvent\",\"actor\":{\"id\":99359,\"login\":\"llinder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?\"},\"repo\":{\"id\":28669941,\"name\":\"llinder/salt\",\"url\":\"https://api.github.com/repos/llinder/salt\"},\"payload\":{\"action\":\"published\",\"release\":{\"url\":\"https://api.github.com/repos/llinder/salt/releases/818220\",\"assets_url\":\"https://api.github.com/repos/llinder/salt/releases/818220/assets\",\"upload_url\":\"https://uploads.github.com/repos/llinder/salt/releases/818220/assets{?name}\",\"html_url\":\"https://github.com/llinder/salt/releases/tag/v2014.7.0.1\",\"id\":818220,\"tag_name\":\"v2014.7.0.1\",\"target_commitish\":\"2014.7\",\"name\":\"\",\"draft\":false,\"author\":{\"login\":\"llinder\",\"id\":99359,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"html_url\":\"https://github.com/llinder\",\"followers_url\":\"https://api.github.com/users/llinder/followers\",\"following_url\":\"https://api.github.com/users/llinder/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/llinder/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/llinder/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/llinder/subscriptions\",\"organizations_url\":\"https://api.github.com/users/llinder/orgs\",\"repos_url\":\"https://api.github.com/users/llinder/repos\",\"events_url\":\"https://api.github.com/users/llinder/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/llinder/received_events\",\"type\":\"User\",\"site_admin\":false},\"prerelease\":false,\"created_at\":\"2014-12-31T17:03:50Z\",\"published_at\":\"2015-01-01T01:05:32Z\",\"assets\":[],\"tarball_url\":\"https://api.github.com/repos/llinder/salt/tarball/v2014.7.0.1\",\"zipball_url\":\"https://api.github.com/repos/llinder/salt/zipball/v2014.7.0.1\",\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\"}\n{\"id\":\"2489398224\",\"type\":\"PushEvent\",\"actor\":{\"id\":1402662,\"login\":\"jamcar23\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamcar23\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1402662?\"},\"repo\":{\"id\":28678215,\"name\":\"jamcar23/TextSecure\",\"url\":\"https://api.github.com/repos/jamcar23/TextSecure\"},\"payload\":{\"push_id\":536753268,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cacd2d4c08275da40253cba2e5e8e1dd067e6535\",\"before\":\"b6822b310ee8a480f9fa2ceab46e3cb7df3b011b\",\"commits\":[{\"sha\":\"cacd2d4c08275da40253cba2e5e8e1dd067e6535\",\"author\":{\"email\":\"7d57d41adbb4b81194decc0d3895cf6be7ed6b28@gmail.com\",\"name\":\"jamcar23\"},\"message\":\"Added support for larger text such as a PGP key\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jamcar23/TextSecure/commits/cacd2d4c08275da40253cba2e5e8e1dd067e6535\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\"}\n{\"id\":\"2489398225\",\"type\":\"CreateEvent\",\"actor\":{\"id\":99359,\"login\":\"llinder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?\"},\"repo\":{\"id\":28669941,\"name\":\"llinder/salt\",\"url\":\"https://api.github.com/repos/llinder/salt\"},\"payload\":{\"ref\":\"v2014.7.0.1\",\"ref_type\":\"tag\",\"master_branch\":\"develop\",\"description\":\"Infrastructure automation and management system\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:32Z\"}\n{\"id\":\"2489398232\",\"type\":\"PushEvent\",\"actor\":{\"id\":1685551,\"login\":\"nyankosoft\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nyankosoft\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1685551?\"},\"repo\":{\"id\":4159713,\"name\":\"nyankosoft/amorphous\",\"url\":\"https://api.github.com/repos/nyankosoft/amorphous\"},\"payload\":{\"push_id\":536753272,\"size\":10,\"distinct_size\":10,\"ref\":\"refs/heads/master\",\"head\":\"2080d2aa6a72a59832291f76b4e231b6bc27e2a9\",\"before\":\"c4b5499bcc5d1f7c079e725c46fcd83d207fd86d\",\"commits\":[{\"sha\":\"61fa0705992c0fab00e967469558305e68692664\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"updated: the skeletal mesh demo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/61fa0705992c0fab00e967469558305e68692664\"},{\"sha\":\"ff50592dc05c087b16fcc889cc6331bb5d994a32\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"updated: VC++ project files.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/ff50592dc05c087b16fcc889cc6331bb5d994a32\"},{\"sha\":\"148aee80cc188f10752f48bd01fbd3216112a75a\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"changed: the EoL characters to LFs.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/148aee80cc188f10752f48bd01fbd3216112a75a\"},{\"sha\":\"464a6c62cad68b1213b939e8a745ecf47892e644\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"changed: class and file names from *Test to *Demo\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/464a6c62cad68b1213b939e8a745ecf47892e644\"},{\"sha\":\"90037f3c6af4acf4a67f8dae86463a71e1d9d373\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"changed: CGraphicsTestBase and other related source code.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/90037f3c6af4acf4a67f8dae86463a71e1d9d373\"},{\"sha\":\"8731317418a36f1fe0f757dff00179e86f04dee5\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"fixed: file and class names\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/8731317418a36f1fe0f757dff00179e86f04dee5\"},{\"sha\":\"b59668a84490521c59af648c0759652c392fb6db\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"updated: VC++ project files\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/b59668a84490521c59af648c0759652c392fb6db\"},{\"sha\":\"545bd334f01905514efe81b594850cd93c5869b6\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"added: the Windows executable of MeshViewer\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/545bd334f01905514efe81b594850cd93c5869b6\"},{\"sha\":\"2721bd332fbda951f2ef1c0f581fe049d105273e\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"added: the Windows executable of MeshModelCompiler.exe\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/2721bd332fbda951f2ef1c0f581fe049d105273e\"},{\"sha\":\"2080d2aa6a72a59832291f76b4e231b6bc27e2a9\",\"author\":{\"email\":\"1e6e52750314f7db47451c08146cd425ed21ff15@gmail.com\",\"name\":\"wanko\"},\"message\":\"updated: a VC++ project file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nyankosoft/amorphous/commits/2080d2aa6a72a59832291f76b4e231b6bc27e2a9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:33Z\"}\n{\"id\":\"2489398233\",\"type\":\"PushEvent\",\"actor\":{\"id\":4535554,\"login\":\"Kflash\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Kflash\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4535554?\"},\"repo\":{\"id\":28209294,\"name\":\"ugma/ugma\",\"url\":\"https://api.github.com/repos/ugma/ugma\"},\"payload\":{\"push_id\":536753273,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c55f37f14e2c6098ed43ec4ced1b8cce898ebc01\",\"before\":\"958106c2bdc1af4ad5dcded148254f60ee0be0a2\",\"commits\":[{\"sha\":\"c55f37f14e2c6098ed43ec4ced1b8cce898ebc01\",\"author\":{\"email\":\"fedd6aa74861a001db1cce23d0a3b62b20a9d469@gmail.com\",\"name\":\"KFlash\"},\"message\":\"removed minor bugs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ugma/ugma/commits/c55f37f14e2c6098ed43ec4ced1b8cce898ebc01\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:33Z\",\"org\":{\"id\":10238638,\"login\":\"ugma\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ugma\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10238638?\"}}\n{\"id\":\"2489398234\",\"type\":\"PushEvent\",\"actor\":{\"id\":8471028,\"login\":\"tejp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tejp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8471028?\"},\"repo\":{\"id\":23040963,\"name\":\"tejp/tejp.github.io\",\"url\":\"https://api.github.com/repos/tejp/tejp.github.io\"},\"payload\":{\"push_id\":536753274,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"363dbe560e3cff95e59fbdd48cf2332966d221c7\",\"before\":\"d573ac05334c6d1d703ba4159a9910a3446a9209\",\"commits\":[{\"sha\":\"363dbe560e3cff95e59fbdd48cf2332966d221c7\",\"author\":{\"email\":\"d9c605cae73349f701ec912308096793b72cdc83@gmail.com\",\"name\":\"tejp\"},\"message\":\"fix a setting\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tejp/tejp.github.io/commits/363dbe560e3cff95e59fbdd48cf2332966d221c7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:33Z\"}\n{\"id\":\"2489398235\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":1217681,\"login\":\"stoeffel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/stoeffel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1217681?\"},\"repo\":{\"id\":800115,\"name\":\"epeli/underscore.string\",\"url\":\"https://api.github.com/repos/epeli/underscore.string\"},\"payload\":{\"ref\":\"camelize-force-lowercase\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:34Z\"}\n{\"id\":\"2489398237\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397292\",\"id\":22397292,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\\n \\n - name: custom horizon splash logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png mode=0644 force=yes\\n \\n - name: custom horizon favicon\\n-  get_url: url={{ horizon.favicon_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico\\n-           force=yes\\n+  get_url: url={{ horizon.favicon_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico force=yes\\n \\n - name: put images and fonts where apache can find them\\n-  file: src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n-        dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n-        state=link\\n-        owner=www-data\\n-        group=www-data\\n+  file: |\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":143,\"original_position\":143,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"?\",\"created_at\":\"2015-01-01T01:05:34Z\",\"updated_at\":\"2015-01-01T01:05:34Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397292\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397292\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397292\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:05:34Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:05:34Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489398243\",\"type\":\"GollumEvent\",\"actor\":{\"id\":4620127,\"login\":\"husathap\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/husathap\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4620127?\"},\"repo\":{\"id\":28579680,\"name\":\"husathap/Inuvik\",\"url\":\"https://api.github.com/repos/husathap/Inuvik\"},\"payload\":{\"pages\":[{\"page_name\":\"Tutorial-#1:-Creating-a-New-Room\",\"title\":\"Tutorial #1: Creating a New Room\",\"summary\":null,\"action\":\"created\",\"sha\":\"11ee43f60ec85b1d5774e1d953b95a636a1bb0ff\",\"html_url\":\"https://github.com/husathap/Inuvik/wiki/Tutorial-%231%3A-Creating-a-New-Room\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:35Z\"}\n{\"id\":\"2489398245\",\"type\":\"GollumEvent\",\"actor\":{\"id\":4620127,\"login\":\"husathap\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/husathap\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4620127?\"},\"repo\":{\"id\":28579680,\"name\":\"husathap/Inuvik\",\"url\":\"https://api.github.com/repos/husathap/Inuvik\"},\"payload\":{\"pages\":[{\"page_name\":\"Home\",\"title\":\"Home\",\"summary\":null,\"action\":\"created\",\"sha\":\"ce6911e660816c3f4a6332848564fc52f58c63dd\",\"html_url\":\"https://github.com/husathap/Inuvik/wiki/Home\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:35Z\"}\n{\"id\":\"2489398246\",\"type\":\"PushEvent\",\"actor\":{\"id\":6484805,\"login\":\"Piera\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Piera\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6484805?\"},\"repo\":{\"id\":28613203,\"name\":\"Piera/Project-U-C\",\"url\":\"https://api.github.com/repos/Piera/Project-U-C\"},\"payload\":{\"push_id\":536753277,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ddf094f1c87680288341b667cefa0713155f656e\",\"before\":\"7dd7b441f592a632d520a6493c91bf378de8a366\",\"commits\":[{\"sha\":\"ddf094f1c87680288341b667cefa0713155f656e\",\"author\":{\"email\":\"d4f798d7370ce072daf8219b0f67a9bac5ebb111@gmail.com\",\"name\":\"Piera Damonte\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Piera/Project-U-C/commits/ddf094f1c87680288341b667cefa0713155f656e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:35Z\"}\n{\"id\":\"2489398247\",\"type\":\"PushEvent\",\"actor\":{\"id\":3599988,\"login\":\"wesdizzle\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wesdizzle\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3599988?\"},\"repo\":{\"id\":28250120,\"name\":\"wesdizzle/gagglelog\",\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog\"},\"payload\":{\"push_id\":536753278,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d04cbec8b68d0e66cf87c078862065645e181086\",\"before\":\"fc86231b5063c1b8f2ed07b8d1ae887580869fcf\",\"commits\":[{\"sha\":\"d04cbec8b68d0e66cf87c078862065645e181086\",\"author\":{\"email\":\"baaa01a5d45f86e3d8f7008866cf0d37bea55570@gmail.com\",\"name\":\"Wesley Miller\"},\"message\":\"added Index value to Games for multiple games in a series released simultaneously\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog/commits/d04cbec8b68d0e66cf87c078862065645e181086\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:35Z\"}\n{\"id\":\"2489398248\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":663212,\"login\":\"tdas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?\"},\"repo\":{\"id\":17165658,\"name\":\"apache/spark\",\"url\":\"https://api.github.com/repos/apache/spark\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/apache/spark/issues/3237\",\"labels_url\":\"https://api.github.com/repos/apache/spark/issues/3237/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/apache/spark/issues/3237/comments\",\"events_url\":\"https://api.github.com/repos/apache/spark/issues/3237/events\",\"html_url\":\"https://github.com/apache/spark/pull/3237\",\"id\":48591711,\"number\":3237,\"title\":\"[SPARK-3325][Streaming] Add a parameter to the method print in class DStream.\",\"user\":{\"login\":\"watermen\",\"id\":1400819,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1400819?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/watermen\",\"html_url\":\"https://github.com/watermen\",\"followers_url\":\"https://api.github.com/users/watermen/followers\",\"following_url\":\"https://api.github.com/users/watermen/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/watermen/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/watermen/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/watermen/subscriptions\",\"organizations_url\":\"https://api.github.com/users/watermen/orgs\",\"repos_url\":\"https://api.github.com/users/watermen/repos\",\"events_url\":\"https://api.github.com/users/watermen/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/watermen/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":56,\"created_at\":\"2014-11-13T02:52:35Z\",\"updated_at\":\"2015-01-01T01:05:34Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/apache/spark/pulls/3237\",\"html_url\":\"https://github.com/apache/spark/pull/3237\",\"diff_url\":\"https://github.com/apache/spark/pull/3237.diff\",\"patch_url\":\"https://github.com/apache/spark/pull/3237.patch\"},\"body\":\"I have deleted the repository with mistake, so I create a new PR. old PR:https://github.com/apache/spark/pull/2216\\r\\n\\r\\n```scala\\r\\ndef print(num: Int = 10)\\r\\n```\\r\\nUser can control the number of elements which to print.\"},\"comment\":{\"url\":\"https://api.github.com/repos/apache/spark/issues/comments/68477338\",\"html_url\":\"https://github.com/apache/spark/pull/3237#issuecomment-68477338\",\"issue_url\":\"https://api.github.com/repos/apache/spark/issues/3237\",\"id\":68477338,\"user\":{\"login\":\"tdas\",\"id\":663212,\"avatar_url\":\"https://avatars.githubusercontent.com/u/663212?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tdas\",\"html_url\":\"https://github.com/tdas\",\"followers_url\":\"https://api.github.com/users/tdas/followers\",\"following_url\":\"https://api.github.com/users/tdas/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tdas/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tdas/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tdas/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tdas/orgs\",\"repos_url\":\"https://api.github.com/users/tdas/repos\",\"events_url\":\"https://api.github.com/users/tdas/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tdas/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:34Z\",\"updated_at\":\"2015-01-01T01:05:34Z\",\"body\":\"Oops sorry, my bad. I created another merge conflict by merging another PR. Let me fix that for you.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:35Z\",\"org\":{\"id\":47359,\"login\":\"apache\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/apache\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/47359?\"}}\n{\"id\":\"2489398252\",\"type\":\"PushEvent\",\"actor\":{\"id\":476440,\"login\":\"sjkingo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sjkingo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/476440?\"},\"repo\":{\"id\":28647308,\"name\":\"sjkingo/python-freshdesk\",\"url\":\"https://api.github.com/repos/sjkingo/python-freshdesk\"},\"payload\":{\"push_id\":536753279,\"size\":4,\"distinct_size\":4,\"ref\":\"refs/heads/master\",\"head\":\"58bac92e3df2218a325fc9237e2d9feff50fde02\",\"before\":\"6d7bfadd2498b86848624affb47b921b0ef1fe9c\",\"commits\":[{\"sha\":\"70602ce48ce709d6bef7ad009d487cca502cbc0e\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@sjkwi.com.au\",\"name\":\"Sam Kingston\"},\"message\":\"Handle API rate-limit gracefully\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sjkingo/python-freshdesk/commits/70602ce48ce709d6bef7ad009d487cca502cbc0e\"},{\"sha\":\"90bf419dd59a82753edc02bfe967b6dbb9bc215c\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@sjkwi.com.au\",\"name\":\"Sam Kingston\"},\"message\":\"Add tests for Comments\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sjkingo/python-freshdesk/commits/90bf419dd59a82753edc02bfe967b6dbb9bc215c\"},{\"sha\":\"4ecc96b5bb285f9a57df9ab46c06122847771890\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@sjkwi.com.au\",\"name\":\"Sam Kingston\"},\"message\":\"Add missing tests for str and repr methods on Ticket\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sjkingo/python-freshdesk/commits/4ecc96b5bb285f9a57df9ab46c06122847771890\"},{\"sha\":\"58bac92e3df2218a325fc9237e2d9feff50fde02\",\"author\":{\"email\":\"f16bed56189e249fe4ca8ed10a1ecae60e8ceac0@sjkwi.com.au\",\"name\":\"Sam Kingston\"},\"message\":\"Refactor tests to tidy\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sjkingo/python-freshdesk/commits/58bac92e3df2218a325fc9237e2d9feff50fde02\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:36Z\"}\n{\"id\":\"2489398254\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5226301,\"login\":\"JoshSGman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JoshSGman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5226301?\"},\"repo\":{\"id\":28678272,\"name\":\"JoshSGman/itunesDataViz\",\"url\":\"https://api.github.com/repos/JoshSGman/itunesDataViz\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"A visualization library for itunes data\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:37Z\"}\n{\"id\":\"2489398255\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":108040,\"login\":\"jamesmintram\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamesmintram\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/108040?\"},\"repo\":{\"id\":28670077,\"name\":\"jamesmintram/LoomSDK\",\"url\":\"https://api.github.com/repos/jamesmintram/LoomSDK\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/jamesmintram/LoomSDK/issues/1\",\"labels_url\":\"https://api.github.com/repos/jamesmintram/LoomSDK/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/jamesmintram/LoomSDK/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/jamesmintram/LoomSDK/issues/1/events\",\"html_url\":\"https://github.com/jamesmintram/LoomSDK/issues/1\",\"id\":53210283,\"number\":1,\"title\":\"Retina OSX nativeStageHeight incorrect\",\"user\":{\"login\":\"jamesmintram\",\"id\":108040,\"avatar_url\":\"https://avatars.githubusercontent.com/u/108040?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jamesmintram\",\"html_url\":\"https://github.com/jamesmintram\",\"followers_url\":\"https://api.github.com/users/jamesmintram/followers\",\"following_url\":\"https://api.github.com/users/jamesmintram/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/jamesmintram/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/jamesmintram/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/jamesmintram/subscriptions\",\"organizations_url\":\"https://api.github.com/users/jamesmintram/orgs\",\"repos_url\":\"https://api.github.com/users/jamesmintram/repos\",\"events_url\":\"https://api.github.com/users/jamesmintram/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/jamesmintram/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:05:37Z\",\"updated_at\":\"2015-01-01T01:05:37Z\",\"closed_at\":null,\"body\":\"It appears that stage.nativeStageWidth and stage.nativeStageHeight do not take into account the scale factor on OSX.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:37Z\"}\n{\"id\":\"2489398262\",\"type\":\"WatchEvent\",\"actor\":{\"id\":117788,\"login\":\"nyarla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nyarla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/117788?\"},\"repo\":{\"id\":15659140,\"name\":\"clbr/urlmatch\",\"url\":\"https://api.github.com/repos/clbr/urlmatch\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:37Z\"}\n{\"id\":\"2489398266\",\"type\":\"PushEvent\",\"actor\":{\"id\":2127465,\"login\":\"gizmo385\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gizmo385\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2127465?\"},\"repo\":{\"id\":25680223,\"name\":\"gizmo385/LearningClojure\",\"url\":\"https://api.github.com/repos/gizmo385/LearningClojure\"},\"payload\":{\"push_id\":536753285,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"b183587e912c016d78339ce2a24d72561f22ea85\",\"before\":\"cc88f9406f34d5a0256154003b4d9a96aa2d6428\",\"commits\":[{\"sha\":\"32224fa783d11f07e04e908eb5be765b4dbe4db6\",\"author\":{\"email\":\"aed2c4d01960864cfdbeee7b4c5891b1ffa98b04@gmail.com\",\"name\":\"Christopher Chapline\"},\"message\":\"Updated readme\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gizmo385/LearningClojure/commits/32224fa783d11f07e04e908eb5be765b4dbe4db6\"},{\"sha\":\"b183587e912c016d78339ce2a24d72561f22ea85\",\"author\":{\"email\":\"aed2c4d01960864cfdbeee7b4c5891b1ffa98b04@gmail.com\",\"name\":\"Christopher Chapline\"},\"message\":\"New graph implementation based on protocols. SimpleGraph, Network, and Digraph implemented.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gizmo385/LearningClojure/commits/b183587e912c016d78339ce2a24d72561f22ea85\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:38Z\"}\n{\"id\":\"2489398267\",\"type\":\"PushEvent\",\"actor\":{\"id\":8941027,\"login\":\"MarvAmBass\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MarvAmBass\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8941027?\"},\"repo\":{\"id\":28630141,\"name\":\"MarvAmBass/py-media.ccc-congress-crawler\",\"url\":\"https://api.github.com/repos/MarvAmBass/py-media.ccc-congress-crawler\"},\"payload\":{\"push_id\":536753286,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b2f2a15104dd3784e9ca3e3c50305f09a0c732ca\",\"before\":\"7e54b606cd5632cc34efc10e7842b72eeb3205b3\",\"commits\":[{\"sha\":\"b2f2a15104dd3784e9ca3e3c50305f09a0c732ca\",\"author\":{\"email\":\"78cbc2e9669f4cea46806632e080ee9166740ed9@gmail.com\",\"name\":\"MarvAmBass\"},\"message\":\"complete rewrite\",\"distinct\":true,\"url\":\"https://api.github.com/repos/MarvAmBass/py-media.ccc-congress-crawler/commits/b2f2a15104dd3784e9ca3e3c50305f09a0c732ca\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:38Z\"}\n{\"id\":\"2489398269\",\"type\":\"CreateEvent\",\"actor\":{\"id\":10173890,\"login\":\"daiguangfa\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/daiguangfa\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10173890?\"},\"repo\":{\"id\":28678274,\"name\":\"daiguangfa/python_tutorial\",\"url\":\"https://api.github.com/repos/daiguangfa/python_tutorial\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Python入门\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:40Z\"}\n{\"id\":\"2489398280\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"action\":\"closed\",\"number\":31,\"pull_request\":{\"url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31\",\"id\":26739457,\"html_url\":\"https://github.com/captainkirkby/Gears/pull/31\",\"diff_url\":\"https://github.com/captainkirkby/Gears/pull/31.diff\",\"patch_url\":\"https://github.com/captainkirkby/Gears/pull/31.patch\",\"issue_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31\",\"number\":31,\"state\":\"closed\",\"locked\":false,\"title\":\"Add simple C program to replay binary file\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Closes issue #30 \",\"created_at\":\"2015-01-01T01:05:31Z\",\"updated_at\":\"2015-01-01T01:05:41Z\",\"closed_at\":\"2015-01-01T01:05:41Z\",\"merged_at\":\"2015-01-01T01:05:41Z\",\"merge_commit_sha\":\"1937489450eb244357356ec9a4ad968f1d63e23d\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/commits\",\"review_comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/comments\",\"review_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31/comments\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"head\":{\"label\":\"captainkirkby:#30\",\"ref\":\"#30\",\"sha\":\"0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15461243,\"name\":\"Gears\",\"full_name\":\"captainkirkby/Gears\",\"owner\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/captainkirkby/Gears\",\"description\":\"Set of packages used to measure a mechanical clock.\",\"fork\":false,\"url\":\"https://api.github.com/repos/captainkirkby/Gears\",\"forks_url\":\"https://api.github.com/repos/captainkirkby/Gears/forks\",\"keys_url\":\"https://api.github.com/repos/captainkirkby/Gears/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/captainkirkby/Gears/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/captainkirkby/Gears/teams\",\"hooks_url\":\"https://api.github.com/repos/captainkirkby/Gears/hooks\",\"issue_events_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/captainkirkby/Gears/events\",\"assignees_url\":\"https://api.github.com/repos/captainkirkby/Gears/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/captainkirkby/Gears/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/tags\",\"blobs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/captainkirkby/Gears/languages\",\"stargazers_url\":\"https://api.github.com/repos/captainkirkby/Gears/stargazers\",\"contributors_url\":\"https://api.github.com/repos/captainkirkby/Gears/contributors\",\"subscribers_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscribers\",\"subscription_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscription\",\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/captainkirkby/Gears/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/captainkirkby/Gears/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/captainkirkby/Gears/merges\",\"archive_url\":\"https://api.github.com/repos/captainkirkby/Gears/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/captainkirkby/Gears/downloads\",\"issues_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/captainkirkby/Gears/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/captainkirkby/Gears/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/captainkirkby/Gears/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/captainkirkby/Gears/releases{/id}\",\"created_at\":\"2013-12-26T22:05:51Z\",\"updated_at\":\"2015-01-01T00:41:54Z\",\"pushed_at\":\"2015-01-01T01:05:41Z\",\"git_url\":\"git://github.com/captainkirkby/Gears.git\",\"ssh_url\":\"git@github.com:captainkirkby/Gears.git\",\"clone_url\":\"https://github.com/captainkirkby/Gears.git\",\"svn_url\":\"https://github.com/captainkirkby/Gears\",\"homepage\":\"\",\"size\":18140,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":11,\"forks\":0,\"open_issues\":11,\"watchers\":1,\"default_branch\":\"master\"}},\"base\":{\"label\":\"captainkirkby:master\",\"ref\":\"master\",\"sha\":\"b838f0c36403eab209f3565eeb7a10375911d228\",\"user\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":15461243,\"name\":\"Gears\",\"full_name\":\"captainkirkby/Gears\",\"owner\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/captainkirkby/Gears\",\"description\":\"Set of packages used to measure a mechanical clock.\",\"fork\":false,\"url\":\"https://api.github.com/repos/captainkirkby/Gears\",\"forks_url\":\"https://api.github.com/repos/captainkirkby/Gears/forks\",\"keys_url\":\"https://api.github.com/repos/captainkirkby/Gears/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/captainkirkby/Gears/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/captainkirkby/Gears/teams\",\"hooks_url\":\"https://api.github.com/repos/captainkirkby/Gears/hooks\",\"issue_events_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/captainkirkby/Gears/events\",\"assignees_url\":\"https://api.github.com/repos/captainkirkby/Gears/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/captainkirkby/Gears/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/tags\",\"blobs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/captainkirkby/Gears/languages\",\"stargazers_url\":\"https://api.github.com/repos/captainkirkby/Gears/stargazers\",\"contributors_url\":\"https://api.github.com/repos/captainkirkby/Gears/contributors\",\"subscribers_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscribers\",\"subscription_url\":\"https://api.github.com/repos/captainkirkby/Gears/subscription\",\"commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/captainkirkby/Gears/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/captainkirkby/Gears/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/captainkirkby/Gears/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/captainkirkby/Gears/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/captainkirkby/Gears/merges\",\"archive_url\":\"https://api.github.com/repos/captainkirkby/Gears/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/captainkirkby/Gears/downloads\",\"issues_url\":\"https://api.github.com/repos/captainkirkby/Gears/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/captainkirkby/Gears/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/captainkirkby/Gears/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/captainkirkby/Gears/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/captainkirkby/Gears/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/captainkirkby/Gears/releases{/id}\",\"created_at\":\"2013-12-26T22:05:51Z\",\"updated_at\":\"2015-01-01T00:41:54Z\",\"pushed_at\":\"2015-01-01T01:05:41Z\",\"git_url\":\"git://github.com/captainkirkby/Gears.git\",\"ssh_url\":\"git@github.com:captainkirkby/Gears.git\",\"clone_url\":\"https://github.com/captainkirkby/Gears.git\",\"svn_url\":\"https://github.com/captainkirkby/Gears\",\"homepage\":\"\",\"size\":18140,\"stargazers_count\":1,\"watchers_count\":1,\"language\":\"C++\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":11,\"forks\":0,\"open_issues\":11,\"watchers\":1,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31\"},\"html\":{\"href\":\"https://github.com/captainkirkby/Gears/pull/31\"},\"issue\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31\"},\"comments\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/issues/31/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/pulls/31/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/captainkirkby/Gears/statuses/0ce69fa19d81de656dd6a74629099a7fa9261d1c\"}},\"merged\":true,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":{\"login\":\"captainkirkby\",\"id\":3489773,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"html_url\":\"https://github.com/captainkirkby\",\"followers_url\":\"https://api.github.com/users/captainkirkby/followers\",\"following_url\":\"https://api.github.com/users/captainkirkby/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/captainkirkby/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/captainkirkby/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/captainkirkby/subscriptions\",\"organizations_url\":\"https://api.github.com/users/captainkirkby/orgs\",\"repos_url\":\"https://api.github.com/users/captainkirkby/repos\",\"events_url\":\"https://api.github.com/users/captainkirkby/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/captainkirkby/received_events\",\"type\":\"User\",\"site_admin\":false},\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":129,\"deletions\":0,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:05:42Z\"}\n{\"id\":\"2489398281\",\"type\":\"PushEvent\",\"actor\":{\"id\":2110113,\"login\":\"wiczerd\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wiczerd\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2110113?\"},\"repo\":{\"id\":22581949,\"name\":\"wiczerd/GKTW_empirical\",\"url\":\"https://api.github.com/repos/wiczerd/GKTW_empirical\"},\"payload\":{\"push_id\":536753289,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3e6b0090163506b70aa0a443f97730d4836a743d\",\"before\":\"1821290f1d7a8846cc7832431b01c14aaa777c44\",\"commits\":[{\"sha\":\"3e6b0090163506b70aa0a443f97730d4836a743d\",\"author\":{\"email\":\"078c0c67d55661fe08d5d1ec3da0bf82da0133d7@gmail.com\",\"name\":\"David Wiczer\"},\"message\":\"changes to l_phys_neg l_phys_pos needed for where regressions\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wiczerd/GKTW_empirical/commits/3e6b0090163506b70aa0a443f97730d4836a743d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:42Z\"}\n{\"id\":\"2489398282\",\"type\":\"PushEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"push_id\":536753290,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3a977b38b570126097e52fc850332efc76fe7efe\",\"before\":\"b838f0c36403eab209f3565eeb7a10375911d228\",\"commits\":[{\"sha\":\"0ce69fa19d81de656dd6a74629099a7fa9261d1c\",\"author\":{\"email\":\"4d1902cf1aefa3df01c59cbb9ae7db3045be42ae@gmail.com\",\"name\":\"Dylan Kirkby\"},\"message\":\"Add simple C program to replay binary file\",\"distinct\":false,\"url\":\"https://api.github.com/repos/captainkirkby/Gears/commits/0ce69fa19d81de656dd6a74629099a7fa9261d1c\"},{\"sha\":\"3a977b38b570126097e52fc850332efc76fe7efe\",\"author\":{\"email\":\"4d1902cf1aefa3df01c59cbb9ae7db3045be42ae@gmail.com\",\"name\":\"Dylan Kirkby\"},\"message\":\"Merge pull request #31 from captainkirkby/#30\\n\\nAdd simple C program to replay binary file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/captainkirkby/Gears/commits/3a977b38b570126097e52fc850332efc76fe7efe\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:42Z\"}\n{\"id\":\"2489398289\",\"type\":\"PushEvent\",\"actor\":{\"id\":965430,\"login\":\"waltzofpearls\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/waltzofpearls\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/965430?\"},\"repo\":{\"id\":28505561,\"name\":\"waltzofpearls/dotfiles\",\"url\":\"https://api.github.com/repos/waltzofpearls/dotfiles\"},\"payload\":{\"push_id\":536753294,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d7d714784919d48e92ed72486d1cbfd5dd4c896b\",\"before\":\"d5a1f77ab903cdcf1d2ae292c7f0f259c6e32211\",\"commits\":[{\"sha\":\"d7d714784919d48e92ed72486d1cbfd5dd4c896b\",\"author\":{\"email\":\"c514db49330801e4e831feeacd2b70f6f55a5048@gmail.com\",\"name\":\"Rollie Ma\"},\"message\":\"Changed alias d to d1 and dd to d2 to avoid possible command conflicts\",\"distinct\":true,\"url\":\"https://api.github.com/repos/waltzofpearls/dotfiles/commits/d7d714784919d48e92ed72486d1cbfd5dd4c896b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:43Z\"}\n{\"id\":\"2489398293\",\"type\":\"WatchEvent\",\"actor\":{\"id\":4030056,\"login\":\"alexFranka\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexFranka\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4030056?\"},\"repo\":{\"id\":5294540,\"name\":\"Stereobit/dragend\",\"url\":\"https://api.github.com/repos/Stereobit/dragend\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:43Z\"}\n{\"id\":\"2489398296\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":3489773,\"login\":\"captainkirkby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/captainkirkby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3489773?\"},\"repo\":{\"id\":15461243,\"name\":\"captainkirkby/Gears\",\"url\":\"https://api.github.com/repos/captainkirkby/Gears\"},\"payload\":{\"ref\":\"#30\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:43Z\"}\n{\"id\":\"2489398297\",\"type\":\"CreateEvent\",\"actor\":{\"id\":790100,\"login\":\"mbutterworth\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mbutterworth\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/790100?\"},\"repo\":{\"id\":28678199,\"name\":\"mbutterworth/menu-test\",\"url\":\"https://api.github.com/repos/mbutterworth/menu-test\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:43Z\"}\n{\"id\":\"2489398309\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1075965,\"login\":\"takuan-osho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/takuan-osho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1075965?\"},\"repo\":{\"id\":28081156,\"name\":\"lexrus/VPNOn\",\"url\":\"https://api.github.com/repos/lexrus/VPNOn\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:44Z\"}\n{\"id\":\"2489398314\",\"type\":\"PushEvent\",\"actor\":{\"id\":92735,\"login\":\"sorear\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sorear\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/92735?\"},\"repo\":{\"id\":5104661,\"name\":\"sorear/perl-DBR\",\"url\":\"https://api.github.com/repos/sorear/perl-DBR\"},\"payload\":{\"push_id\":536753304,\"size\":3,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"ed1f9a08a0da68a67022b1df5e087e306a147a1c\",\"before\":\"89749b9eea501807ca432473798ca70f504d1693\",\"commits\":[{\"sha\":\"5d087526d37083ad04d5d9d830de31f8f441883c\",\"author\":{\"email\":\"3d0f3b9ddcacec30c4008c5e030e6c13a478cb4f@gudtech.com\",\"name\":\"Daniel Norman\"},\"message\":\"bugfix\",\"distinct\":false,\"url\":\"https://api.github.com/repos/sorear/perl-DBR/commits/5d087526d37083ad04d5d9d830de31f8f441883c\"},{\"sha\":\"2d8c69c9fd8c008a2fd787b9eda342dbc0099ec9\",\"author\":{\"email\":\"fed0578406081f3c3a132b2ffaa2b378514d6fc0@cox.net\",\"name\":\"Stefan O'Rear\"},\"message\":\"Merge branch 'master' of github.com:dnorman/perl-DBR\",\"distinct\":false,\"url\":\"https://api.github.com/repos/sorear/perl-DBR/commits/2d8c69c9fd8c008a2fd787b9eda342dbc0099ec9\"},{\"sha\":\"ed1f9a08a0da68a67022b1df5e087e306a147a1c\",\"author\":{\"email\":\"fed0578406081f3c3a132b2ffaa2b378514d6fc0@cox.net\",\"name\":\"Stefan O'Rear\"},\"message\":\"avoid generating nulls in IN-lists\",\"distinct\":false,\"url\":\"https://api.github.com/repos/sorear/perl-DBR/commits/ed1f9a08a0da68a67022b1df5e087e306a147a1c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:45Z\"}\n{\"id\":\"2489398316\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536753305,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4a08cf46a835742f49c5272d70f3210529dbb819\",\"before\":\"089c8f0d2bac3da1fec35e110c876455549123ae\",\"commits\":[{\"sha\":\"4a08cf46a835742f49c5272d70f3210529dbb819\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074344264\\n\\nYdoxe8jvlupoYjDMdaz3MuTJYDArYhC8zsnVTxc3DhA=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/4a08cf46a835742f49c5272d70f3210529dbb819\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:45Z\"}\n{\"id\":\"2489398322\",\"type\":\"PushEvent\",\"actor\":{\"id\":7282584,\"login\":\"suhasgaddam\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/suhasgaddam\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7282584?\"},\"repo\":{\"id\":22023128,\"name\":\"suhasgaddam/suhasgaddam\",\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam\"},\"payload\":{\"push_id\":536753308,\"size\":10,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"8770d45468f7a35e3523dcd2c953e98495f7f9e1\",\"before\":\"87857d97c47ccc77fddec314b222eca1c5b4c3d3\",\"commits\":[{\"sha\":\"91a07bfa95c2dd01114091ec46b69029c5482c3a\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update portfolio view\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/91a07bfa95c2dd01114091ec46b69029c5482c3a\"},{\"sha\":\"d086fe0560ede399831c0e939539048a8cf769d9\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update ansible roles for larabook\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/d086fe0560ede399831c0e939539048a8cf769d9\"},{\"sha\":\"519f55acfccaff462e8d3c8b509c22e2978b6964\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update composer dependencies\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/519f55acfccaff462e8d3c8b509c22e2978b6964\"},{\"sha\":\"a758b8a21b2eb691bfc6a8f7f81120c3c5ac00b4\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update ansible roles\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/a758b8a21b2eb691bfc6a8f7f81120c3c5ac00b4\"},{\"sha\":\"44a91286a792f3176b23784d922ec67006ce1ae5\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update ansible roles for larabook and ssl\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/44a91286a792f3176b23784d922ec67006ce1ae5\"},{\"sha\":\"906796e966cf00d3e4f95c725893bed23c780fe8\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update GitHub API service class\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/906796e966cf00d3e4f95c725893bed23c780fe8\"},{\"sha\":\"1360cc3c364c6658990df6890ee74fa36bd0da80\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Organize ansible folder\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/1360cc3c364c6658990df6890ee74fa36bd0da80\"},{\"sha\":\"5fc4d906b4d59212f594ce267acceafd11b214a7\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Update ansible roles to use environment variables\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/5fc4d906b4d59212f594ce267acceafd11b214a7\"},{\"sha\":\"7d478f4562713035d6950734a9a92c0cfa891bb0\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Fix nginx template for www subdomain\",\"distinct\":false,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/7d478f4562713035d6950734a9a92c0cfa891bb0\"},{\"sha\":\"8770d45468f7a35e3523dcd2c953e98495f7f9e1\",\"author\":{\"email\":\"8941c16c0c424ea999586f4432f75bdae83bd66f@gmail.com\",\"name\":\"Suhas Gaddam\"},\"message\":\"Merge branch 'develop'\",\"distinct\":true,\"url\":\"https://api.github.com/repos/suhasgaddam/suhasgaddam/commits/8770d45468f7a35e3523dcd2c953e98495f7f9e1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:47Z\"}\n{\"id\":\"2489398328\",\"type\":\"PushEvent\",\"actor\":{\"id\":3407942,\"login\":\"pnlbwh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pnlbwh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3407942?\"},\"repo\":{\"id\":25606127,\"name\":\"pnlbwh/diskusage-logging\",\"url\":\"https://api.github.com/repos/pnlbwh/diskusage-logging\"},\"payload\":{\"push_id\":536753313,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fee82b7c06761155df5d1bd78b5b6b485edd1923\",\"before\":\"4dba9fc71445e445cbef290e486b3265ba7b283f\",\"commits\":[{\"sha\":\"fee82b7c06761155df5d1bd78b5b6b485edd1923\",\"author\":{\"email\":\"ea6edc91e6ce6eaff9d772be660c2fa0a7b1e60f@bwh.harvard.edu\",\"name\":\"reckbo\"},\"message\":\"ENH: Add remote paths, and generic report\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pnlbwh/diskusage-logging/commits/fee82b7c06761155df5d1bd78b5b6b485edd1923\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:47Z\"}\n{\"id\":\"2489398330\",\"type\":\"PushEvent\",\"actor\":{\"id\":995241,\"login\":\"ggkuron\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ggkuron\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/995241?\"},\"repo\":{\"id\":12571910,\"name\":\"ggkuron/dotfiles\",\"url\":\"https://api.github.com/repos/ggkuron/dotfiles\"},\"payload\":{\"push_id\":536753315,\"size\":6,\"distinct_size\":0,\"ref\":\"refs/heads/X230\",\"head\":\"09214bd8722793af81c66a913d8f72b55b48036e\",\"before\":\"4e3621cd80a6ac39d4a3d069d3f7ceed47cb4670\",\"commits\":[{\"sha\":\"b223857d2dc4d4d1be233b266c90e021c713eb4e\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"x230 wifi setting\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/b223857d2dc4d4d1be233b266c90e021c713eb4e\"},{\"sha\":\"40160aede782fbd7e6267f0ddf97b1106a67e614\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"migrate to xkb from xmodmap setting\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/40160aede782fbd7e6267f0ddf97b1106a67e614\"},{\"sha\":\"fff4016764a0d5d6129a6855e8c068b92c3f61e1\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"gnome-power-manager init again\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/fff4016764a0d5d6129a6855e8c068b92c3f61e1\"},{\"sha\":\"0dc458f8e32336cc2cc4998adbae510cfab67c0e\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"update vimrc :<\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/0dc458f8e32336cc2cc4998adbae510cfab67c0e\"},{\"sha\":\"284cc12097bb43d8f40f0b6b39b40508c01ee3cd\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"fix tmux copy pipe\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/284cc12097bb43d8f40f0b6b39b40508c01ee3cd\"},{\"sha\":\"09214bd8722793af81c66a913d8f72b55b48036e\",\"author\":{\"email\":\"a5d06a406de0a33fcbcafe77084f41b3871f8b1f@gmail.com\",\"name\":\"ggkuron\"},\"message\":\"gvimrc font bigger\",\"distinct\":false,\"url\":\"https://api.github.com/repos/ggkuron/dotfiles/commits/09214bd8722793af81c66a913d8f72b55b48036e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:48Z\"}\n{\"id\":\"2489398331\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":706947,\"login\":\"d3athrow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?\"},\"repo\":{\"id\":10441188,\"name\":\"d3athrow/vgstation13\",\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397293\",\"id\":22397293,\"diff_hunk\":\"@@ -8,6 +8,7 @@\\n \\tmax_heat_protection_temperature = FIRE_HELMET_MAX_HEAT_PROTECITON_TEMPERATURE\\n \\tvar/obj/machinery/camera/camera\\n \\tspecies_restricted = list(\\\"exclude\\\",\\\"Vox\\\")\\n+\\tflags = PLASMAGUARD\",\"path\":\"code/modules/clothing/spacesuits/ert.dm\",\"position\":4,\"original_position\":4,\"commit_id\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"original_commit_id\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"flages = FPRINT | TABLEPASS | HEADCOVERSEYES | BLOCKHAIR | STOPSPRESSUREDMAGE | PLASMAGUARD\",\"created_at\":\"2015-01-01T01:05:48Z\",\"updated_at\":\"2015-01-01T01:05:48Z\",\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2405#discussion_r22397293\",\"pull_request_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397293\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2405#discussion_r22397293\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\",\"id\":26675841,\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2405\",\"diff_url\":\"https://github.com/d3athrow/vgstation13/pull/2405.diff\",\"patch_url\":\"https://github.com/d3athrow/vgstation13/pull/2405.patch\",\"issue_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405\",\"number\":2405,\"state\":\"open\",\"locked\":false,\"title\":\"Added PLASMAGUARD flag to some spacesuits\",\"user\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Added the PLASMAGUARD flag to ERT suits, deathsquad suits, and CE/Atmos suits as requested in \\r\\n\\r\\nhttps://github.com/d3athrow/vgstation13/issues/2369\",\"created_at\":\"2014-12-30T07:18:28Z\",\"updated_at\":\"2015-01-01T01:05:48Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"fff34ea0a38796d850e3f5aba53410e1b7bde9c8\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/commits\",\"review_comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/comments\",\"review_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405/comments\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/b93293990c4d927f30a1f048939d427ee591e4a6\",\"head\":{\"label\":\"Rei1226:ratdicks\",\"ref\":\"ratdicks\",\"sha\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"user\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":19230658,\"name\":\"vgstation13\",\"full_name\":\"Rei1226/vgstation13\",\"owner\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Rei1226/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/Rei1226/vgstation13\",\"forks_url\":\"https://api.github.com/repos/Rei1226/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/Rei1226/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Rei1226/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Rei1226/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/Rei1226/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Rei1226/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/Rei1226/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Rei1226/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Rei1226/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Rei1226/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Rei1226/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/Rei1226/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Rei1226/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Rei1226/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Rei1226/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/Rei1226/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Rei1226/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Rei1226/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Rei1226/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Rei1226/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/Rei1226/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Rei1226/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Rei1226/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Rei1226/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Rei1226/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Rei1226/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Rei1226/vgstation13/releases{/id}\",\"created_at\":\"2014-04-28T07:36:48Z\",\"updated_at\":\"2014-12-30T08:33:17Z\",\"pushed_at\":\"2014-12-30T08:33:10Z\",\"git_url\":\"git://github.com/Rei1226/vgstation13.git\",\"ssh_url\":\"git@github.com:Rei1226/vgstation13.git\",\"clone_url\":\"https://github.com/Rei1226/vgstation13.git\",\"svn_url\":\"https://github.com/Rei1226/vgstation13\",\"homepage\":\"\",\"size\":722996,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"d3athrow:Bleeding-Edge\",\"ref\":\"Bleeding-Edge\",\"sha\":\"ea27b58dbac4fdb65119b09648f3f4f2395125de\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":10441188,\"name\":\"vgstation13\",\"full_name\":\"d3athrow/vgstation13\",\"owner\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/d3athrow/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\",\"forks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/d3athrow/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/d3athrow/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/d3athrow/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/d3athrow/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/d3athrow/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/d3athrow/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/d3athrow/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/d3athrow/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/d3athrow/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/d3athrow/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/d3athrow/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/d3athrow/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/d3athrow/vgstation13/releases{/id}\",\"created_at\":\"2013-06-02T19:39:54Z\",\"updated_at\":\"2014-12-31T20:06:46Z\",\"pushed_at\":\"2015-01-01T01:04:27Z\",\"git_url\":\"git://github.com/d3athrow/vgstation13.git\",\"ssh_url\":\"git@github.com:d3athrow/vgstation13.git\",\"clone_url\":\"https://github.com/d3athrow/vgstation13.git\",\"svn_url\":\"https://github.com/d3athrow/vgstation13\",\"homepage\":\"\",\"size\":937605,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":135,\"mirror_url\":null,\"open_issues_count\":259,\"forks\":135,\"open_issues\":259,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2405\"},\"issue\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405\"},\"comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/b93293990c4d927f30a1f048939d427ee591e4a6\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:05:48Z\"}\n{\"id\":\"2489398337\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":726063,\"login\":\"coder543\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coder543\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/726063?\"},\"repo\":{\"id\":12176600,\"name\":\"AngryLawyer/rust-sdl2\",\"url\":\"https://api.github.com/repos/AngryLawyer/rust-sdl2\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/AngryLawyer/rust-sdl2/issues/248\",\"labels_url\":\"https://api.github.com/repos/AngryLawyer/rust-sdl2/issues/248/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/AngryLawyer/rust-sdl2/issues/248/comments\",\"events_url\":\"https://api.github.com/repos/AngryLawyer/rust-sdl2/issues/248/events\",\"html_url\":\"https://github.com/AngryLawyer/rust-sdl2/issues/248\",\"id\":53210284,\"number\":248,\"title\":\"Unneccessary line in demo.rs?\",\"user\":{\"login\":\"coder543\",\"id\":726063,\"avatar_url\":\"https://avatars.githubusercontent.com/u/726063?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coder543\",\"html_url\":\"https://github.com/coder543\",\"followers_url\":\"https://api.github.com/users/coder543/followers\",\"following_url\":\"https://api.github.com/users/coder543/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coder543/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coder543/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coder543/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coder543/orgs\",\"repos_url\":\"https://api.github.com/users/coder543/repos\",\"events_url\":\"https://api.github.com/users/coder543/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coder543/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:05:48Z\",\"updated_at\":\"2015-01-01T01:05:48Z\",\"closed_at\":null,\"body\":\"[This line of code](https://github.com/AngryLawyer/rust-sdl2/blob/84e67fbece4c9f92fd5635b19805eb47f271c432/examples/demo.rs#L30) doesn't seem to accomplish anything. In fact, I think the whole inner 'event loop could probably be done away with, just based on playing with the code a little. Could someone with more experience explain to me why that inner loop is there, and what line 30 is accomplishing? I'm really new to Rust, so I may just be missing something.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:48Z\"}\n{\"id\":\"2489398342\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":49391,\"login\":\"myronmarston\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/myronmarston\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/49391?\"},\"repo\":{\"id\":238983,\"name\":\"rspec/rspec-mocks\",\"url\":\"https://api.github.com/repos/rspec/rspec-mocks\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/854\",\"labels_url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/854/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/854/comments\",\"events_url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/854/events\",\"html_url\":\"https://github.com/rspec/rspec-mocks/issues/854\",\"id\":53194690,\"number\":854,\"title\":\"mock_of? to check types on doubles (to avoid unnecessary coercions)\",\"user\":{\"login\":\"maxlinc\",\"id\":896878,\"avatar_url\":\"https://avatars.githubusercontent.com/u/896878?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/maxlinc\",\"html_url\":\"https://github.com/maxlinc\",\"followers_url\":\"https://api.github.com/users/maxlinc/followers\",\"following_url\":\"https://api.github.com/users/maxlinc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/maxlinc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/maxlinc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/maxlinc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/maxlinc/orgs\",\"repos_url\":\"https://api.github.com/users/maxlinc/repos\",\"events_url\":\"https://api.github.com/users/maxlinc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/maxlinc/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-31T17:42:16Z\",\"updated_at\":\"2015-01-01T01:05:50Z\",\"closed_at\":null,\"body\":\"I have a feature request similar to #794 (instance_double not matching the same class it is mocking) but I have a more specific use-case and solving for it doesn't necessary require stubbing `#===`.\\r\\n\\r\\nThere are many Ruby gems that support type coercion, like:\\r\\n- https://github.com/intridea/hashie#coercion\\r\\n- https://github.com/solnic/virtus\\r\\n- https://github.com/solnic/coercible\\r\\n- https://github.com/apotonick/representable#coercion\\r\\n\\r\\nI believe they all do a type check to avoid unnecessary coercions. In the case of Hashie, coercion is skipped if the value is already the target class:\\r\\n\\r\\n```ruby\\r\\n  it 'skips unnecessary coercions' do\\r\\n      subject.coerce_key :foo, Coercable\\r\\n\\r\\n      object = Coercable.new('bar')\\r\\n      instance[:foo] = object\\r\\n      expect(instance[:foo]).to eq(object)\\r\\n    end\\r\\n```\\r\\n\\r\\nHowever, if the value is an double of the target class than it will be coerced:\\r\\n```ruby\\r\\n    it 'skips unnecessary coercions (RSpec instance_double)' do\\r\\n      subject.coerce_key :foo, Coercable\\r\\n\\r\\n      object = instance_double(Coercable)\\r\\n      instance[:foo] = object\\r\\n      expect(instance[:foo]).to eq(object)\\r\\n    end\\r\\n```\\r\\n\\r\\n```\\r\\nFailures:\\r\\n\\r\\n  1) Hashie::Extensions::Coercion#coerce_key skips unnecessary coercions (RSpec instance_double)\\r\\n     Failure/Error: expect(instance[:foo]).to eq(object)\\r\\n\\r\\n       expected: #<RSpec::Mocks::InstanceVerifyingDouble:0x3fe5cdd89080 @name=\\\"Coercable (instance)\\\">\\r\\n            got: #<Coercable:0x007fcb9bb11de0 @coerced=true, @value=\\\"RSpec::Mocks::InstanceVerifyingDouble\\\">\\r\\n\\r\\n       (compared using ==)\\r\\n\\r\\n       Diff:\\r\\n       @@ -1,2 +1,4 @@\\r\\n       -#<RSpec::Mocks::InstanceVerifyingDouble:0x3fe5cdd89080 @name=\\\"Coercable (instance)\\\">\\r\\n       +#<Coercable:0x007fcb9bb11de0\\r\\n       + @coerced=true,\\r\\n       + @value=\\\"RSpec::Mocks::InstanceVerifyingDouble\\\">\\r\\n```\\r\\n\\r\\nI'd like to be able to avoid that coercion. Rather than stubbing `#kind_of?` or `#===` something like `#mock_of?` that behaves similar to `#kind_of?` on the doubled class would be useful.\"},\"comment\":{\"url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/comments/68477339\",\"html_url\":\"https://github.com/rspec/rspec-mocks/issues/854#issuecomment-68477339\",\"issue_url\":\"https://api.github.com/repos/rspec/rspec-mocks/issues/854\",\"id\":68477339,\"user\":{\"login\":\"myronmarston\",\"id\":49391,\"avatar_url\":\"https://avatars.githubusercontent.com/u/49391?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/myronmarston\",\"html_url\":\"https://github.com/myronmarston\",\"followers_url\":\"https://api.github.com/users/myronmarston/followers\",\"following_url\":\"https://api.github.com/users/myronmarston/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/myronmarston/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/myronmarston/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/myronmarston/subscriptions\",\"organizations_url\":\"https://api.github.com/users/myronmarston/orgs\",\"repos_url\":\"https://api.github.com/users/myronmarston/repos\",\"events_url\":\"https://api.github.com/users/myronmarston/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/myronmarston/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:50Z\",\"updated_at\":\"2015-01-01T01:05:50Z\",\"body\":\"> I'd appreciate a method to find out what is being doubled (`@doubled_module.const_to_replace`). It's not public, so right now I can only find out via monkeypatching or instance_variable_get.\\r\\n\\r\\nI definitely want to provide an API for your use.  One possibility is what is being added in #832, although as a callback API that might not fit your needs.  We could also add something like `doubled_module` to verifying doubles, although that has a couple of issues:\\r\\n\\r\\n* The doubled module may not be loaded, so it would have to return `nil` in that case which gives you no information.\\r\\n* As I mentioned above, I want to limit the number of built-in methods defined on doubles.\\r\\n\\r\\nI'm thinking that we may want to go a more functional route where you would call `RSpec::Mocks.doubled_module_for(dbl)` instead.  (That would solve the latter problem).  For the former problem, we may want to make the API return a module reference (similar to what #832 is going) instead of the actual module.  But then we should call the method something different.\\r\\n\\r\\nWhat would meet your needs?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:50Z\",\"org\":{\"id\":22388,\"login\":\"rspec\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rspec\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/22388?\"}}\n{\"id\":\"2489398345\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":8890867,\"login\":\"willgabriel\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/willgabriel\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8890867?\"},\"repo\":{\"id\":28678195,\"name\":\"TTMTT/iCL0udin\",\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"labels_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/events\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1\",\"id\":53210206,\"number\":1,\"title\":\"Discuss1\",\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2015-01-01T01:02:34Z\",\"updated_at\":\"2015-01-01T01:05:51Z\",\"closed_at\":null,\"body\":\"Now you can download vresion 1.0 from :\\r\\n---------------------------------------------------\\r\\nhttp://www.icloudin.net\\r\\n-----------------------------\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n--------------------------------------\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n---------------------------------------\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n-------------------------------------\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n-------------------------------------\\r\\niCL0udin v1.0 -> %100\\r\\n----------------------------\\r\\nRemaining: %3 testing with some people..\\r\\n-----------------------------------------------------\\r\\nLast Method:\\r\\n-----------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\niCL0udin v1.0 have this method:\\r\\n-----------------------------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/comments/68477341\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1#issuecomment-68477341\",\"issue_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"id\":68477341,\"user\":{\"login\":\"willgabriel\",\"id\":8890867,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8890867?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/willgabriel\",\"html_url\":\"https://github.com/willgabriel\",\"followers_url\":\"https://api.github.com/users/willgabriel/followers\",\"following_url\":\"https://api.github.com/users/willgabriel/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/willgabriel/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/willgabriel/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/willgabriel/subscriptions\",\"organizations_url\":\"https://api.github.com/users/willgabriel/orgs\",\"repos_url\":\"https://api.github.com/users/willgabriel/repos\",\"events_url\":\"https://api.github.com/users/willgabriel/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/willgabriel/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:05:50Z\",\"updated_at\":\"2015-01-01T01:05:50Z\",\"body\":\"ok bro\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:51Z\"}\n{\"id\":\"2489398347\",\"type\":\"PushEvent\",\"actor\":{\"id\":8323759,\"login\":\"MozillaPootleL10nBot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MozillaPootleL10nBot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8323759?\"},\"repo\":{\"id\":6078190,\"name\":\"translate/mozilla-lang\",\"url\":\"https://api.github.com/repos/translate/mozilla-lang\"},\"payload\":{\"push_id\":536753320,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f468205040a01ee5ffed377ec4f2b2a6be35c687\",\"before\":\"2df4901774321d656e6e60150051a66dc87cefcb\",\"commits\":[{\"sha\":\"f468205040a01ee5ffed377ec4f2b2a6be35c687\",\"author\":{\"email\":\"f02c4b7f151600c94fb4d6f03844f716aeaa6d62@translate.org.za\",\"name\":\"Mozilla Pootle L10n Robot\"},\"message\":\"[son] pull from Pootle\",\"distinct\":true,\"url\":\"https://api.github.com/repos/translate/mozilla-lang/commits/f468205040a01ee5ffed377ec4f2b2a6be35c687\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:51Z\",\"org\":{\"id\":1538178,\"login\":\"translate\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/translate\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1538178?\"}}\n{\"id\":\"2489398355\",\"type\":\"PushEvent\",\"actor\":{\"id\":506010,\"login\":\"gabeshaughnessy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gabeshaughnessy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/506010?\"},\"repo\":{\"id\":13913264,\"name\":\"gabeshaughnessy/augmentedart\",\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart\"},\"payload\":{\"push_id\":536753322,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dungeon-hacker\",\"head\":\"1855adda47047b9a781de334f4616c312ec65dc6\",\"before\":\"f1d6efa16b35cfe98b4ee3e54c3f333f87612fa1\",\"commits\":[{\"sha\":\"1855adda47047b9a781de334f4616c312ec65dc6\",\"author\":{\"email\":\"a2b2bb6e7f1b10ac88b326d5c10e33af6a8546bc@gmail.com\",\"name\":\"gabeshaughnessy\"},\"message\":\"task list\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart/commits/1855adda47047b9a781de334f4616c312ec65dc6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:53Z\"}\n{\"id\":\"2489398362\",\"type\":\"GollumEvent\",\"actor\":{\"id\":10245688,\"login\":\"SunyataZero\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SunyataZero\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10245688?\"},\"repo\":{\"id\":28485643,\"name\":\"EmpathyApp/EmpathyApp\",\"url\":\"https://api.github.com/repos/EmpathyApp/EmpathyApp\"},\"payload\":{\"pages\":[{\"page_name\":\"Design-tools\",\"title\":\"Design tools\",\"summary\":null,\"action\":\"edited\",\"sha\":\"39fe159c9630870c8f3da7f55774c6ea773d45be\",\"html_url\":\"https://github.com/EmpathyApp/EmpathyApp/wiki/Design-tools\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:54Z\",\"org\":{\"id\":10245750,\"login\":\"EmpathyApp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/EmpathyApp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10245750?\"}}\n{\"id\":\"2489398364\",\"type\":\"CreateEvent\",\"actor\":{\"id\":6955935,\"login\":\"wfrizzell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wfrizzell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6955935?\"},\"repo\":{\"id\":28678275,\"name\":\"wfrizzell/Completed-Target-Model-Table-Documentation\",\"url\":\"https://api.github.com/repos/wfrizzell/Completed-Target-Model-Table-Documentation\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:55Z\"}\n{\"id\":\"2489398368\",\"type\":\"PushEvent\",\"actor\":{\"id\":486234,\"login\":\"dubinsky\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dubinsky\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/486234?\"},\"repo\":{\"id\":28631502,\"name\":\"dubinsky/19-kislev-archive\",\"url\":\"https://api.github.com/repos/dubinsky/19-kislev-archive\"},\"payload\":{\"push_id\":536753326,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c805962cac1cf5f6d5f408079c05213ed0d738a8\",\"before\":\"34816ca67b9ed5364ea6e0bfeaacd594ac89e04f\",\"commits\":[{\"sha\":\"c805962cac1cf5f6d5f408079c05213ed0d738a8\",\"author\":{\"email\":\"1ca4b28513c5e015ce705bdc97bba1a9e4624b16@podval.org\",\"name\":\"Leonid Dubinsky\"},\"message\":\"Sync.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dubinsky/19-kislev-archive/commits/c805962cac1cf5f6d5f408079c05213ed0d738a8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:56Z\"}\n{\"id\":\"2489398369\",\"type\":\"PushEvent\",\"actor\":{\"id\":698770,\"login\":\"johnnovak\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/johnnovak\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/698770?\"},\"repo\":{\"id\":17473666,\"name\":\"johnnovak/dotfiles\",\"url\":\"https://api.github.com/repos/johnnovak/dotfiles\"},\"payload\":{\"push_id\":536753327,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a84040736d0e942689cd5444100dc91ae7661ba8\",\"before\":\"c03d592c8fa1a6638cce9d972cfdb65ad89f26e6\",\"commits\":[{\"sha\":\"a84040736d0e942689cd5444100dc91ae7661ba8\",\"author\":{\"email\":\"a51dda7c7ff50b61eaea0444371f4a6a9301e501@johnnovak.net\",\"name\":\"John Novak\"},\"message\":\"vimrc: cleanup\",\"distinct\":true,\"url\":\"https://api.github.com/repos/johnnovak/dotfiles/commits/a84040736d0e942689cd5444100dc91ae7661ba8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:56Z\"}\n{\"id\":\"2489398370\",\"type\":\"CommitCommentEvent\",\"actor\":{\"id\":1497508,\"login\":\"simon816\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/simon816\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1497508?\"},\"repo\":{\"id\":23774298,\"name\":\"SpongePowered/Sponge\",\"url\":\"https://api.github.com/repos/SpongePowered/Sponge\"},\"payload\":{\"comment\":{\"url\":\"https://api.github.com/repos/SpongePowered/Sponge/comments/9131367\",\"html_url\":\"https://github.com/SpongePowered/Sponge/commit/b0c96c3f67ffd4830420b7c62edd8db816fc3d64#commitcomment-9131367\",\"id\":9131367,\"user\":{\"login\":\"simon816\",\"id\":1497508,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1497508?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/simon816\",\"html_url\":\"https://github.com/simon816\",\"followers_url\":\"https://api.github.com/users/simon816/followers\",\"following_url\":\"https://api.github.com/users/simon816/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/simon816/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/simon816/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/simon816/subscriptions\",\"organizations_url\":\"https://api.github.com/users/simon816/orgs\",\"repos_url\":\"https://api.github.com/users/simon816/repos\",\"events_url\":\"https://api.github.com/users/simon816/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/simon816/received_events\",\"type\":\"User\",\"site_admin\":false},\"position\":130,\"line\":176,\"path\":\"src/main/java/org/spongepowered/mod/registry/SpongeGameRegistry.java\",\"commit_id\":\"b0c96c3f67ffd4830420b7c62edd8db816fc3d64\",\"created_at\":\"2015-01-01T01:05:55Z\",\"updated_at\":\"2015-01-01T01:05:55Z\",\"body\":\"This should be `ARMOR_STAND`\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:55Z\",\"org\":{\"id\":8683473,\"login\":\"SpongePowered\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/SpongePowered\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8683473?\"}}\n{\"id\":\"2489398372\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":447804,\"login\":\"tfoote\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tfoote\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/447804?\"},\"repo\":{\"id\":3149845,\"name\":\"ros/rosdistro\",\"url\":\"https://api.github.com/repos/ros/rosdistro\"},\"payload\":{\"action\":\"opened\",\"number\":6732,\"pull_request\":{\"url\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732\",\"id\":26739460,\"html_url\":\"https://github.com/ros/rosdistro/pull/6732\",\"diff_url\":\"https://github.com/ros/rosdistro/pull/6732.diff\",\"patch_url\":\"https://github.com/ros/rosdistro/pull/6732.patch\",\"issue_url\":\"https://api.github.com/repos/ros/rosdistro/issues/6732\",\"number\":6732,\"state\":\"open\",\"locked\":false,\"title\":\"adding bullet rosdep rules for utopic and vivid\",\"user\":{\"login\":\"tfoote\",\"id\":447804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/447804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tfoote\",\"html_url\":\"https://github.com/tfoote\",\"followers_url\":\"https://api.github.com/users/tfoote/followers\",\"following_url\":\"https://api.github.com/users/tfoote/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/tfoote/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/tfoote/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/tfoote/subscriptions\",\"organizations_url\":\"https://api.github.com/users/tfoote/orgs\",\"repos_url\":\"https://api.github.com/users/tfoote/repos\",\"events_url\":\"https://api.github.com/users/tfoote/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/tfoote/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:05:55Z\",\"updated_at\":\"2015-01-01T01:05:55Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732/commits\",\"review_comments_url\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732/comments\",\"review_comment_url\":\"https://api.github.com/repos/ros/rosdistro/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/ros/rosdistro/issues/6732/comments\",\"statuses_url\":\"https://api.github.com/repos/ros/rosdistro/statuses/aa0e8faf57abed8143ce567ceb6c45736946d0cd\",\"head\":{\"label\":\"ros:bullet_UV\",\"ref\":\"bullet_UV\",\"sha\":\"aa0e8faf57abed8143ce567ceb6c45736946d0cd\",\"user\":{\"login\":\"ros\",\"id\":547448,\"avatar_url\":\"https://avatars.githubusercontent.com/u/547448?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ros\",\"html_url\":\"https://github.com/ros\",\"followers_url\":\"https://api.github.com/users/ros/followers\",\"following_url\":\"https://api.github.com/users/ros/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ros/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ros/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ros/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ros/orgs\",\"repos_url\":\"https://api.github.com/users/ros/repos\",\"events_url\":\"https://api.github.com/users/ros/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ros/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":3149845,\"name\":\"rosdistro\",\"full_name\":\"ros/rosdistro\",\"owner\":{\"login\":\"ros\",\"id\":547448,\"avatar_url\":\"https://avatars.githubusercontent.com/u/547448?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ros\",\"html_url\":\"https://github.com/ros\",\"followers_url\":\"https://api.github.com/users/ros/followers\",\"following_url\":\"https://api.github.com/users/ros/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ros/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ros/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ros/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ros/orgs\",\"repos_url\":\"https://api.github.com/users/ros/repos\",\"events_url\":\"https://api.github.com/users/ros/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ros/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/ros/rosdistro\",\"description\":\"This repo maintains a lists of repositories for each ROS distribution\",\"fork\":false,\"url\":\"https://api.github.com/repos/ros/rosdistro\",\"forks_url\":\"https://api.github.com/repos/ros/rosdistro/forks\",\"keys_url\":\"https://api.github.com/repos/ros/rosdistro/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/ros/rosdistro/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/ros/rosdistro/teams\",\"hooks_url\":\"https://api.github.com/repos/ros/rosdistro/hooks\",\"issue_events_url\":\"https://api.github.com/repos/ros/rosdistro/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/ros/rosdistro/events\",\"assignees_url\":\"https://api.github.com/repos/ros/rosdistro/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/ros/rosdistro/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/ros/rosdistro/tags\",\"blobs_url\":\"https://api.github.com/repos/ros/rosdistro/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/ros/rosdistro/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/ros/rosdistro/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/ros/rosdistro/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/ros/rosdistro/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/ros/rosdistro/languages\",\"stargazers_url\":\"https://api.github.com/repos/ros/rosdistro/stargazers\",\"contributors_url\":\"https://api.github.com/repos/ros/rosdistro/contributors\",\"subscribers_url\":\"https://api.github.com/repos/ros/rosdistro/subscribers\",\"subscription_url\":\"https://api.github.com/repos/ros/rosdistro/subscription\",\"commits_url\":\"https://api.github.com/repos/ros/rosdistro/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/ros/rosdistro/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/ros/rosdistro/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/ros/rosdistro/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/ros/rosdistro/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/ros/rosdistro/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/ros/rosdistro/merges\",\"archive_url\":\"https://api.github.com/repos/ros/rosdistro/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/ros/rosdistro/downloads\",\"issues_url\":\"https://api.github.com/repos/ros/rosdistro/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/ros/rosdistro/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/ros/rosdistro/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/ros/rosdistro/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/ros/rosdistro/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/ros/rosdistro/releases{/id}\",\"created_at\":\"2012-01-11T00:13:17Z\",\"updated_at\":\"2014-12-31T23:39:13Z\",\"pushed_at\":\"2015-01-01T00:27:02Z\",\"git_url\":\"git://github.com/ros/rosdistro.git\",\"ssh_url\":\"git@github.com:ros/rosdistro.git\",\"clone_url\":\"https://github.com/ros/rosdistro.git\",\"svn_url\":\"https://github.com/ros/rosdistro\",\"homepage\":\"\",\"size\":40338,\"stargazers_count\":107,\"watchers_count\":107,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":459,\"mirror_url\":null,\"open_issues_count\":29,\"forks\":459,\"open_issues\":29,\"watchers\":107,\"default_branch\":\"master\"}},\"base\":{\"label\":\"ros:master\",\"ref\":\"master\",\"sha\":\"c92186717af0766aa2b4dc6adf216ee610a4ed1e\",\"user\":{\"login\":\"ros\",\"id\":547448,\"avatar_url\":\"https://avatars.githubusercontent.com/u/547448?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ros\",\"html_url\":\"https://github.com/ros\",\"followers_url\":\"https://api.github.com/users/ros/followers\",\"following_url\":\"https://api.github.com/users/ros/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ros/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ros/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ros/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ros/orgs\",\"repos_url\":\"https://api.github.com/users/ros/repos\",\"events_url\":\"https://api.github.com/users/ros/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ros/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":3149845,\"name\":\"rosdistro\",\"full_name\":\"ros/rosdistro\",\"owner\":{\"login\":\"ros\",\"id\":547448,\"avatar_url\":\"https://avatars.githubusercontent.com/u/547448?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ros\",\"html_url\":\"https://github.com/ros\",\"followers_url\":\"https://api.github.com/users/ros/followers\",\"following_url\":\"https://api.github.com/users/ros/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ros/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ros/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ros/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ros/orgs\",\"repos_url\":\"https://api.github.com/users/ros/repos\",\"events_url\":\"https://api.github.com/users/ros/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ros/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/ros/rosdistro\",\"description\":\"This repo maintains a lists of repositories for each ROS distribution\",\"fork\":false,\"url\":\"https://api.github.com/repos/ros/rosdistro\",\"forks_url\":\"https://api.github.com/repos/ros/rosdistro/forks\",\"keys_url\":\"https://api.github.com/repos/ros/rosdistro/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/ros/rosdistro/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/ros/rosdistro/teams\",\"hooks_url\":\"https://api.github.com/repos/ros/rosdistro/hooks\",\"issue_events_url\":\"https://api.github.com/repos/ros/rosdistro/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/ros/rosdistro/events\",\"assignees_url\":\"https://api.github.com/repos/ros/rosdistro/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/ros/rosdistro/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/ros/rosdistro/tags\",\"blobs_url\":\"https://api.github.com/repos/ros/rosdistro/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/ros/rosdistro/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/ros/rosdistro/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/ros/rosdistro/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/ros/rosdistro/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/ros/rosdistro/languages\",\"stargazers_url\":\"https://api.github.com/repos/ros/rosdistro/stargazers\",\"contributors_url\":\"https://api.github.com/repos/ros/rosdistro/contributors\",\"subscribers_url\":\"https://api.github.com/repos/ros/rosdistro/subscribers\",\"subscription_url\":\"https://api.github.com/repos/ros/rosdistro/subscription\",\"commits_url\":\"https://api.github.com/repos/ros/rosdistro/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/ros/rosdistro/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/ros/rosdistro/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/ros/rosdistro/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/ros/rosdistro/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/ros/rosdistro/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/ros/rosdistro/merges\",\"archive_url\":\"https://api.github.com/repos/ros/rosdistro/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/ros/rosdistro/downloads\",\"issues_url\":\"https://api.github.com/repos/ros/rosdistro/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/ros/rosdistro/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/ros/rosdistro/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/ros/rosdistro/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/ros/rosdistro/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/ros/rosdistro/releases{/id}\",\"created_at\":\"2012-01-11T00:13:17Z\",\"updated_at\":\"2014-12-31T23:39:13Z\",\"pushed_at\":\"2015-01-01T00:27:02Z\",\"git_url\":\"git://github.com/ros/rosdistro.git\",\"ssh_url\":\"git@github.com:ros/rosdistro.git\",\"clone_url\":\"https://github.com/ros/rosdistro.git\",\"svn_url\":\"https://github.com/ros/rosdistro\",\"homepage\":\"\",\"size\":40338,\"stargazers_count\":107,\"watchers_count\":107,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":459,\"mirror_url\":null,\"open_issues_count\":29,\"forks\":459,\"open_issues\":29,\"watchers\":107,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732\"},\"html\":{\"href\":\"https://github.com/ros/rosdistro/pull/6732\"},\"issue\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/issues/6732\"},\"comments\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/issues/6732/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/pulls/6732/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/ros/rosdistro/statuses/aa0e8faf57abed8143ce567ceb6c45736946d0cd\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":6,\"deletions\":0,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:05:56Z\",\"org\":{\"id\":547448,\"login\":\"ros\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ros\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/547448?\"}}\n{\"id\":\"2489398374\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1312880,\"login\":\"cmwwebfx\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmwwebfx\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1312880?\"},\"repo\":{\"id\":8194553,\"name\":\"michaeluno/admin-page-framework\",\"url\":\"https://api.github.com/repos/michaeluno/admin-page-framework\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/michaeluno/admin-page-framework/issues/180\",\"labels_url\":\"https://api.github.com/repos/michaeluno/admin-page-framework/issues/180/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/michaeluno/admin-page-framework/issues/180/comments\",\"events_url\":\"https://api.github.com/repos/michaeluno/admin-page-framework/issues/180/events\",\"html_url\":\"https://github.com/michaeluno/admin-page-framework/issues/180\",\"id\":53210287,\"number\":180,\"title\":\"Flexible content field group\",\"user\":{\"login\":\"cmwwebfx\",\"id\":1312880,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1312880?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmwwebfx\",\"html_url\":\"https://github.com/cmwwebfx\",\"followers_url\":\"https://api.github.com/users/cmwwebfx/followers\",\"following_url\":\"https://api.github.com/users/cmwwebfx/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cmwwebfx/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cmwwebfx/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cmwwebfx/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cmwwebfx/orgs\",\"repos_url\":\"https://api.github.com/users/cmwwebfx/repos\",\"events_url\":\"https://api.github.com/users/cmwwebfx/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cmwwebfx/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:05:57Z\",\"updated_at\":\"2015-01-01T01:05:57Z\",\"closed_at\":null,\"body\":\"Hi Michael,\\r\\n\\r\\nHappy new year..\\r\\n\\r\\nI have used a plugin before for creating custom fields for a small project, and it was called ACF (Advanced Custom Fields) by Elliot Condon.\\r\\n\\r\\nWhile I used this plugin for creating a simple layout, I explored the options they had available in the PRO version. What I really loved was something called \\\"Flexible Content Field\\\". \\r\\n\\r\\nhttp://www.advancedcustomfields.com/resources/flexible-content/\\r\\n\\r\\nThis was really awesome since it allowed me to make a set of grouped fields, then add them to this area that I wanted. I used this to make a pretty complex , yet easy to use selection of groups for the end user.\\r\\n\\r\\nIs this something you might consider to add to Admin Page Framework? Here are some screenshots of how I have used it, and you can see how this is quite beneficial to have such an option.\\r\\n\\r\\n![add a flexible content field](https://cloud.githubusercontent.com/assets/1312880/5591413/18e8920e-91aa-11e4-9744-8630fcc2eb81.jpg)\\r\\n\\r\\n![flexible content field](https://cloud.githubusercontent.com/assets/1312880/5591415/36f2ef60-91aa-11e4-92b3-a4d749d4afe5.jpg)\\r\\n\\r\\n![flexible content field dragable](https://cloud.githubusercontent.com/assets/1312880/5591416/3a79ee2c-91aa-11e4-9656-f9ae00f86776.jpg)\\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:57Z\"}\n{\"id\":\"2489398377\",\"type\":\"PushEvent\",\"actor\":{\"id\":1220541,\"login\":\"orthez\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/orthez\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1220541?\"},\"repo\":{\"id\":24684211,\"name\":\"orthez/workspace-homology\",\"url\":\"https://api.github.com/repos/orthez/workspace-homology\"},\"payload\":{\"push_id\":536753331,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"4bb77a864087fb80e8990a3983133898439fd0b1\",\"before\":\"d688fe2f3b340b3512afb14060097d6152fb56d7\",\"commits\":[{\"sha\":\"d1b825e541f55691602a4091f5db1338cc634fa9\",\"author\":{\"email\":\"f357ae49b6cf16f864ac544048626472d25113dd@gmx.de\",\"name\":\"Andreas Orthey\"},\"message\":\"xspace computation module\",\"distinct\":true,\"url\":\"https://api.github.com/repos/orthez/workspace-homology/commits/d1b825e541f55691602a4091f5db1338cc634fa9\"},{\"sha\":\"4bb77a864087fb80e8990a3983133898439fd0b1\",\"author\":{\"email\":\"f357ae49b6cf16f864ac544048626472d25113dd@gmx.de\",\"name\":\"Andreas Orthey\"},\"message\":\"plot module for cspace | readjusted robot parameters\",\"distinct\":true,\"url\":\"https://api.github.com/repos/orthez/workspace-homology/commits/4bb77a864087fb80e8990a3983133898439fd0b1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:57Z\"}\n{\"id\":\"2489398379\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753332,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"6630db87e4ed496f67a88bf24f22332c67e9792d\",\"before\":\"ca0474a33dcc125aa7204b517dd7cef8003db346\",\"commits\":[{\"sha\":\"6630db87e4ed496f67a88bf24f22332c67e9792d\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"fix bug in detecting which bindings are list items\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/6630db87e4ed496f67a88bf24f22332c67e9792d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:57Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398380\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":99359,\"login\":\"llinder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/llinder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99359?\"},\"repo\":{\"id\":28669941,\"name\":\"llinder/salt\",\"url\":\"https://api.github.com/repos/llinder/salt\"},\"payload\":{\"ref\":\"v2014.7.0_1\",\"ref_type\":\"tag\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:57Z\"}\n{\"id\":\"2489398388\",\"type\":\"PushEvent\",\"actor\":{\"id\":21060,\"login\":\"taladar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/taladar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/21060?\"},\"repo\":{\"id\":6124150,\"name\":\"taladar/portage\",\"url\":\"https://api.github.com/repos/taladar/portage\"},\"payload\":{\"push_id\":536753336,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gentoo.org\",\"head\":\"a887ddbd472af412c458a84e61675debb6a4703b\",\"before\":\"bb37d9cff8757360f30e14e7046bfd7da864872b\",\"commits\":[{\"sha\":\"a887ddbd472af412c458a84e61675debb6a4703b\",\"author\":{\"email\":\"dc76e9f0c0006e8f919e0c515c66dbba3982f785@root2.taladar.net\",\"name\":\"root2.taladar.net root\"},\"message\":\"Portage rsync://rsync.de.gentoo.org/gentoo-portage at 2015-01-01 02:05:36\",\"distinct\":true,\"url\":\"https://api.github.com/repos/taladar/portage/commits/a887ddbd472af412c458a84e61675debb6a4703b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:58Z\"}\n{\"id\":\"2489398389\",\"type\":\"PushEvent\",\"actor\":{\"id\":815193,\"login\":\"aquach\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aquach\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/815193?\"},\"repo\":{\"id\":4950386,\"name\":\"aquach/miner\",\"url\":\"https://api.github.com/repos/aquach/miner\"},\"payload\":{\"push_id\":536753337,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f4ad3cb3d0e30ce311b8b032cf6923544eca80a0\",\"before\":\"93759971a2c59562d5413a0d236663435fb5e442\",\"commits\":[{\"sha\":\"f4ad3cb3d0e30ce311b8b032cf6923544eca80a0\",\"author\":{\"email\":\"60c6d277a8bd81de7fdde19201bf9c58a3df08f4@clinkle.com\",\"name\":\"Alex Quach\"},\"message\":\"Delete unnecessary stuff and kill medical skill/sickbays. Can't find a use for it.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aquach/miner/commits/f4ad3cb3d0e30ce311b8b032cf6923544eca80a0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:05:58Z\"}\n{\"id\":\"2489398391\",\"type\":\"CreateEvent\",\"actor\":{\"id\":4440858,\"login\":\"maynarddemmon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/maynarddemmon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4440858?\"},\"repo\":{\"id\":19410013,\"name\":\"teem2/dreem\",\"url\":\"https://api.github.com/repos/teem2/dreem\"},\"payload\":{\"ref\":\"sizetodom\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"A prototyping language for multiscreen multidevice UX\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:05:58Z\",\"org\":{\"id\":7244222,\"login\":\"teem2\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/teem2\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7244222?\"}}\n{\"id\":\"2489398395\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/83\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/83\",\"id\":53210289,\"number\":83,\"title\":\"Add icons to the left of the text on About screen buttons\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":6,\"closed_issues\":11,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:05:59Z\",\"due_on\":null,\"closed_at\":null},\"comments\":0,\"created_at\":\"2015-01-01T01:05:59Z\",\"updated_at\":\"2015-01-01T01:05:59Z\",\"closed_at\":null,\"body\":\"Related icons aligned to the left on the buttons on about screen\"}},\"public\":true,\"created_at\":\"2015-01-01T01:05:59Z\"}\n{\"id\":\"2489398402\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1813305,\"login\":\"Bart39\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?\"},\"repo\":{\"id\":9683876,\"name\":\"nZEDb/nZEDb\",\"url\":\"https://api.github.com/repos/nZEDb/nZEDb\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572\",\"labels_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/comments\",\"events_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/events\",\"html_url\":\"https://github.com/nZEDb/nZEDb/issues/1572\",\"id\":53191494,\"number\":1572,\"title\":\"Trakt IMDB search url format changed\",\"user\":{\"login\":\"RickDB\",\"id\":347084,\"avatar_url\":\"https://avatars.githubusercontent.com/u/347084?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RickDB\",\"html_url\":\"https://github.com/RickDB\",\"followers_url\":\"https://api.github.com/users/RickDB/followers\",\"following_url\":\"https://api.github.com/users/RickDB/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/RickDB/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/RickDB/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/RickDB/subscriptions\",\"organizations_url\":\"https://api.github.com/users/RickDB/orgs\",\"repos_url\":\"https://api.github.com/users/RickDB/repos\",\"events_url\":\"https://api.github.com/users/RickDB/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/RickDB/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T16:34:10Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"closed_at\":\"2015-01-01T01:06:00Z\",\"body\":\"Trakt launched their new site and among other things the URL format for searching based on IMDB id has changed from:\\r\\n\\r\\nhttp://trakt.tv/search/imdb?q=tt2395427/\\r\\n\\r\\nto:\\r\\n\\r\\nhttp://trakt.tv/search/imdb/tt2395427/\\r\\n\\r\\nThis affects the movie template but might be used in other areas as well, pretty easy fix but though I let you guys know.\"},\"comment\":{\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/comments/68477343\",\"html_url\":\"https://github.com/nZEDb/nZEDb/issues/1572#issuecomment-68477343\",\"issue_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572\",\"id\":68477343,\"user\":{\"login\":\"Bart39\",\"id\":1813305,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"html_url\":\"https://github.com/Bart39\",\"followers_url\":\"https://api.github.com/users/Bart39/followers\",\"following_url\":\"https://api.github.com/users/Bart39/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Bart39/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Bart39/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Bart39/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Bart39/orgs\",\"repos_url\":\"https://api.github.com/users/Bart39/repos\",\"events_url\":\"https://api.github.com/users/Bart39/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Bart39/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:00Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"body\":\"merged to dev #1573 \"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\",\"org\":{\"id\":4260270,\"login\":\"nZEDb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nZEDb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?\"}}\n{\"id\":\"2489398403\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1813305,\"login\":\"Bart39\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Bart39\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1813305?\"},\"repo\":{\"id\":9683876,\"name\":\"nZEDb/nZEDb\",\"url\":\"https://api.github.com/repos/nZEDb/nZEDb\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572\",\"labels_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/comments\",\"events_url\":\"https://api.github.com/repos/nZEDb/nZEDb/issues/1572/events\",\"html_url\":\"https://github.com/nZEDb/nZEDb/issues/1572\",\"id\":53191494,\"number\":1572,\"title\":\"Trakt IMDB search url format changed\",\"user\":{\"login\":\"RickDB\",\"id\":347084,\"avatar_url\":\"https://avatars.githubusercontent.com/u/347084?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RickDB\",\"html_url\":\"https://github.com/RickDB\",\"followers_url\":\"https://api.github.com/users/RickDB/followers\",\"following_url\":\"https://api.github.com/users/RickDB/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/RickDB/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/RickDB/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/RickDB/subscriptions\",\"organizations_url\":\"https://api.github.com/users/RickDB/orgs\",\"repos_url\":\"https://api.github.com/users/RickDB/repos\",\"events_url\":\"https://api.github.com/users/RickDB/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/RickDB/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T16:34:10Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"closed_at\":\"2015-01-01T01:06:00Z\",\"body\":\"Trakt launched their new site and among other things the URL format for searching based on IMDB id has changed from:\\r\\n\\r\\nhttp://trakt.tv/search/imdb?q=tt2395427/\\r\\n\\r\\nto:\\r\\n\\r\\nhttp://trakt.tv/search/imdb/tt2395427/\\r\\n\\r\\nThis affects the movie template but might be used in other areas as well, pretty easy fix but though I let you guys know.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\",\"org\":{\"id\":4260270,\"login\":\"nZEDb\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/nZEDb\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4260270?\"}}\n{\"id\":\"2489398405\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":2354108,\"login\":\"coveralls\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coveralls\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2354108?\"},\"repo\":{\"id\":20897533,\"name\":\"OCA/vertical-travel\",\"url\":\"https://api.github.com/repos/OCA/vertical-travel\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/19\",\"labels_url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/19/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/19/comments\",\"events_url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/19/events\",\"html_url\":\"https://github.com/OCA/vertical-travel/pull/19\",\"id\":53210017,\"number\":19,\"title\":\"[IMP] Relax ACLs to access the reports\",\"user\":{\"login\":\"joaoalf\",\"id\":840120,\"avatar_url\":\"https://avatars.githubusercontent.com/u/840120?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/joaoalf\",\"html_url\":\"https://github.com/joaoalf\",\"followers_url\":\"https://api.github.com/users/joaoalf/followers\",\"following_url\":\"https://api.github.com/users/joaoalf/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/joaoalf/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/joaoalf/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/joaoalf/subscriptions\",\"organizations_url\":\"https://api.github.com/users/joaoalf/orgs\",\"repos_url\":\"https://api.github.com/users/joaoalf/repos\",\"events_url\":\"https://api.github.com/users/joaoalf/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/joaoalf/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:51:32Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/OCA/vertical-travel/pulls/19\",\"html_url\":\"https://github.com/OCA/vertical-travel/pull/19\",\"diff_url\":\"https://github.com/OCA/vertical-travel/pull/19.diff\",\"patch_url\":\"https://github.com/OCA/vertical-travel/pull/19.patch\"},\"body\":\"Another PR in the ACL series\"},\"comment\":{\"url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/comments/68477344\",\"html_url\":\"https://github.com/OCA/vertical-travel/pull/19#issuecomment-68477344\",\"issue_url\":\"https://api.github.com/repos/OCA/vertical-travel/issues/19\",\"id\":68477344,\"user\":{\"login\":\"coveralls\",\"id\":2354108,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2354108?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coveralls\",\"html_url\":\"https://github.com/coveralls\",\"followers_url\":\"https://api.github.com/users/coveralls/followers\",\"following_url\":\"https://api.github.com/users/coveralls/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coveralls/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coveralls/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coveralls/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coveralls/orgs\",\"repos_url\":\"https://api.github.com/users/coveralls/repos\",\"events_url\":\"https://api.github.com/users/coveralls/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coveralls/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:00Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"body\":\"\\n[![Coverage Status](https://coveralls.io/builds/1679499/badge)](https://coveralls.io/builds/1679499)\\n\\nCoverage remained the same when pulling **68ea6d231ce88e56db8bc2e6eb1e7553e205207f on savoirfairelinux:7.0_travel_journey_acl** into **552b843dcfc28878480cbf4159c5b941c89fdf6d on OCA:7.0**.\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\",\"org\":{\"id\":7600578,\"login\":\"OCA\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/OCA\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7600578?\"}}\n{\"id\":\"2489398409\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":995241,\"login\":\"ggkuron\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ggkuron\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/995241?\"},\"repo\":{\"id\":12571910,\"name\":\"ggkuron/dotfiles\",\"url\":\"https://api.github.com/repos/ggkuron/dotfiles\"},\"payload\":{\"ref\":\"x230\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\"}\n{\"id\":\"2489398408\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1138857,\"login\":\"peterkc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/peterkc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1138857?\"},\"repo\":{\"id\":24388525,\"name\":\"box/leche\",\"url\":\"https://api.github.com/repos/box/leche\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/box/leche/issues/5\",\"labels_url\":\"https://api.github.com/repos/box/leche/issues/5/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/box/leche/issues/5/comments\",\"events_url\":\"https://api.github.com/repos/box/leche/issues/5/events\",\"html_url\":\"https://github.com/box/leche/pull/5\",\"id\":53209954,\"number\":5,\"title\":\"Update dependencies in package.json to latest and add peerDependencies\",\"user\":{\"login\":\"peterkc\",\"id\":1138857,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1138857?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/peterkc\",\"html_url\":\"https://github.com/peterkc\",\"followers_url\":\"https://api.github.com/users/peterkc/followers\",\"following_url\":\"https://api.github.com/users/peterkc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/peterkc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/peterkc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/peterkc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/peterkc/orgs\",\"repos_url\":\"https://api.github.com/users/peterkc/repos\",\"events_url\":\"https://api.github.com/users/peterkc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/peterkc/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2015-01-01T00:47:14Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/box/leche/pulls/5\",\"html_url\":\"https://github.com/box/leche/pull/5\",\"diff_url\":\"https://github.com/box/leche/pull/5.diff\",\"patch_url\":\"https://github.com/box/leche/pull/5.patch\"},\"body\":\"Updating dependencies (#4) also required fixing some lint issues (#3) in leche.js\"},\"comment\":{\"url\":\"https://api.github.com/repos/box/leche/issues/comments/68477345\",\"html_url\":\"https://github.com/box/leche/pull/5#issuecomment-68477345\",\"issue_url\":\"https://api.github.com/repos/box/leche/issues/5\",\"id\":68477345,\"user\":{\"login\":\"peterkc\",\"id\":1138857,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1138857?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/peterkc\",\"html_url\":\"https://github.com/peterkc\",\"followers_url\":\"https://api.github.com/users/peterkc/followers\",\"following_url\":\"https://api.github.com/users/peterkc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/peterkc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/peterkc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/peterkc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/peterkc/orgs\",\"repos_url\":\"https://api.github.com/users/peterkc/repos\",\"events_url\":\"https://api.github.com/users/peterkc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/peterkc/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:00Z\",\"updated_at\":\"2015-01-01T01:06:00Z\",\"body\":\"CLA signed\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\",\"org\":{\"id\":23900,\"login\":\"box\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/box\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/23900?\"}}\n{\"id\":\"2489398410\",\"type\":\"CreateEvent\",\"actor\":{\"id\":6955935,\"login\":\"wfrizzell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wfrizzell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6955935?\"},\"repo\":{\"id\":28678275,\"name\":\"wfrizzell/Completed-Target-Model-Table-Documentation\",\"url\":\"https://api.github.com/repos/wfrizzell/Completed-Target-Model-Table-Documentation\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\"}\n{\"id\":\"2489398411\",\"type\":\"PushEvent\",\"actor\":{\"id\":2074134,\"login\":\"jtbdevelopment\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jtbdevelopment\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2074134?\"},\"repo\":{\"id\":25750775,\"name\":\"jtbdevelopment/TwistedHangman\",\"url\":\"https://api.github.com/repos/jtbdevelopment/TwistedHangman\"},\"payload\":{\"push_id\":536753342,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ffb0188d246564af4993cbe840261401c90ed312\",\"before\":\"12941ba32ad8534239d2b94382938978fd5865b4\",\"commits\":[{\"sha\":\"ffb0188d246564af4993cbe840261401c90ed312\",\"author\":{\"email\":\"3fbaf13ce556021e810d7c5980db29812a1bff3d@gmail.com\",\"name\":\"JTB Development\"},\"message\":\"starting an FB friend invite screen\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jtbdevelopment/TwistedHangman/commits/ffb0188d246564af4993cbe840261401c90ed312\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:01Z\"}\n{\"id\":\"2489398421\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2829718,\"login\":\"phister\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/phister\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829718?\"},\"repo\":{\"id\":28678276,\"name\":\"phister/CollegeFootballPlayoff\",\"url\":\"https://api.github.com/repos/phister/CollegeFootballPlayoff\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":null,\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:02Z\"}\n{\"id\":\"2489398424\",\"type\":\"PushEvent\",\"actor\":{\"id\":10225575,\"login\":\"ExclusiveOrange\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ExclusiveOrange\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10225575?\"},\"repo\":{\"id\":28677579,\"name\":\"ExclusiveOrange/synthesizer\",\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer\"},\"payload\":{\"push_id\":536753349,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cec30520f9eb65d16e63f8d5efdd251204a519ca\",\"before\":\"9221d005a52f784d675044bb14013c0eec6ed795\",\"commits\":[{\"sha\":\"cec30520f9eb65d16e63f8d5efdd251204a519ca\",\"author\":{\"email\":\"de3bd7888dcfc4f7d00a4ef606710f57cbba1dbb@hotmail.com\",\"name\":\"ExclusiveOrange\"},\"message\":\"removed old stuff\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ExclusiveOrange/synthesizer/commits/cec30520f9eb65d16e63f8d5efdd251204a519ca\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:02Z\"}\n{\"id\":\"2489398426\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1932804,\"login\":\"coldmind\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?\"},\"repo\":{\"id\":4164482,\"name\":\"django/django\",\"url\":\"https://api.github.com/repos/django/django\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/django/django/issues/3821\",\"labels_url\":\"https://api.github.com/repos/django/django/issues/3821/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/django/django/issues/3821/comments\",\"events_url\":\"https://api.github.com/repos/django/django/issues/3821/events\",\"html_url\":\"https://github.com/django/django/pull/3821\",\"id\":53200969,\"number\":3821,\"title\":\"[WIP] Fixed #24064 - Prevented database access in compile time in spatialite models\",\"user\":{\"login\":\"coldmind\",\"id\":1932804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"html_url\":\"https://github.com/coldmind\",\"followers_url\":\"https://api.github.com/users/coldmind/followers\",\"following_url\":\"https://api.github.com/users/coldmind/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coldmind/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coldmind/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coldmind/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coldmind/orgs\",\"repos_url\":\"https://api.github.com/users/coldmind/repos\",\"events_url\":\"https://api.github.com/users/coldmind/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coldmind/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-31T20:02:17Z\",\"updated_at\":\"2015-01-01T01:06:01Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/django/django/pulls/3821\",\"html_url\":\"https://github.com/django/django/pull/3821\",\"diff_url\":\"https://github.com/django/django/pull/3821.diff\",\"patch_url\":\"https://github.com/django/django/pull/3821.patch\"},\"body\":\"Here is the patch that can fix problem.\\r\\nIdeas about improvements are welcome.\"},\"comment\":{\"url\":\"https://api.github.com/repos/django/django/issues/comments/68477346\",\"html_url\":\"https://github.com/django/django/pull/3821#issuecomment-68477346\",\"issue_url\":\"https://api.github.com/repos/django/django/issues/3821\",\"id\":68477346,\"user\":{\"login\":\"coldmind\",\"id\":1932804,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1932804?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/coldmind\",\"html_url\":\"https://github.com/coldmind\",\"followers_url\":\"https://api.github.com/users/coldmind/followers\",\"following_url\":\"https://api.github.com/users/coldmind/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/coldmind/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/coldmind/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/coldmind/subscriptions\",\"organizations_url\":\"https://api.github.com/users/coldmind/orgs\",\"repos_url\":\"https://api.github.com/users/coldmind/repos\",\"events_url\":\"https://api.github.com/users/coldmind/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/coldmind/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:01Z\",\"updated_at\":\"2015-01-01T01:06:01Z\",\"body\":\"buildbot, test this please.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:02Z\",\"org\":{\"id\":27804,\"login\":\"django\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/django\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/27804?\"}}\n{\"id\":\"2489398428\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5249918,\"login\":\"ehnmark\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ehnmark\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5249918?\"},\"repo\":{\"id\":28678277,\"name\":\"ehnmark/housing\",\"url\":\"https://api.github.com/repos/ehnmark/housing\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:03Z\"}\n{\"id\":\"2489398429\",\"type\":\"PushEvent\",\"actor\":{\"id\":10182579,\"login\":\"Hyftar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Hyftar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10182579?\"},\"repo\":{\"id\":28678188,\"name\":\"Hyftar/blah\",\"url\":\"https://api.github.com/repos/Hyftar/blah\"},\"payload\":{\"push_id\":536753350,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"382bf507dbf58ef02b9daac86377895787d90e1d\",\"before\":\"b8125f82b3ca1d038368cfd44882b5f68413a3b0\",\"commits\":[{\"sha\":\"382bf507dbf58ef02b9daac86377895787d90e1d\",\"author\":{\"email\":\"03d2ba3a5d0bf707f7f3c852ae80a60d66351e89@ymail.com\",\"name\":\"Hyftar\"},\"message\":\"First commit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Hyftar/blah/commits/382bf507dbf58ef02b9daac86377895787d90e1d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:03Z\"}\n{\"id\":\"2489398433\",\"type\":\"ForkEvent\",\"actor\":{\"id\":2875366,\"login\":\"k6dsp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/k6dsp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2875366?\"},\"repo\":{\"id\":19836773,\"name\":\"DigilentInc/Linux-Digilent-Dev\",\"url\":\"https://api.github.com/repos/DigilentInc/Linux-Digilent-Dev\"},\"payload\":{\"forkee\":{\"id\":28678278,\"name\":\"Linux-Digilent-Dev\",\"full_name\":\"k6dsp/Linux-Digilent-Dev\",\"owner\":{\"login\":\"k6dsp\",\"id\":2875366,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2875366?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/k6dsp\",\"html_url\":\"https://github.com/k6dsp\",\"followers_url\":\"https://api.github.com/users/k6dsp/followers\",\"following_url\":\"https://api.github.com/users/k6dsp/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/k6dsp/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/k6dsp/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/k6dsp/subscriptions\",\"organizations_url\":\"https://api.github.com/users/k6dsp/orgs\",\"repos_url\":\"https://api.github.com/users/k6dsp/repos\",\"events_url\":\"https://api.github.com/users/k6dsp/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/k6dsp/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/k6dsp/Linux-Digilent-Dev\",\"description\":\"The official Linux kernel from Xilinx\",\"fork\":true,\"url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev\",\"forks_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/forks\",\"keys_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/teams\",\"hooks_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/hooks\",\"issue_events_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/events\",\"assignees_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/tags\",\"blobs_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/languages\",\"stargazers_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/stargazers\",\"contributors_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/contributors\",\"subscribers_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/subscribers\",\"subscription_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/subscription\",\"commits_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/merges\",\"archive_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/downloads\",\"issues_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/k6dsp/Linux-Digilent-Dev/releases{/id}\",\"created_at\":\"2015-01-01T01:06:03Z\",\"updated_at\":\"2014-09-07T23:09:11Z\",\"pushed_at\":\"2014-05-23T19:19:11Z\",\"git_url\":\"git://github.com/k6dsp/Linux-Digilent-Dev.git\",\"ssh_url\":\"git@github.com:k6dsp/Linux-Digilent-Dev.git\",\"clone_url\":\"https://github.com/k6dsp/Linux-Digilent-Dev.git\",\"svn_url\":\"https://github.com/k6dsp/Linux-Digilent-Dev\",\"homepage\":\"\",\"size\":890055,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:06:03Z\",\"org\":{\"id\":7597005,\"login\":\"DigilentInc\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/DigilentInc\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7597005?\"}}\n{\"id\":\"2489398434\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":6845864,\"login\":\"boxcla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/boxcla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6845864?\"},\"repo\":{\"id\":24388525,\"name\":\"box/leche\",\"url\":\"https://api.github.com/repos/box/leche\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/box/leche/issues/5\",\"labels_url\":\"https://api.github.com/repos/box/leche/issues/5/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/box/leche/issues/5/comments\",\"events_url\":\"https://api.github.com/repos/box/leche/issues/5/events\",\"html_url\":\"https://github.com/box/leche/pull/5\",\"id\":53209954,\"number\":5,\"title\":\"Update dependencies in package.json to latest and add peerDependencies\",\"user\":{\"login\":\"peterkc\",\"id\":1138857,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1138857?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/peterkc\",\"html_url\":\"https://github.com/peterkc\",\"followers_url\":\"https://api.github.com/users/peterkc/followers\",\"following_url\":\"https://api.github.com/users/peterkc/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/peterkc/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/peterkc/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/peterkc/subscriptions\",\"organizations_url\":\"https://api.github.com/users/peterkc/orgs\",\"repos_url\":\"https://api.github.com/users/peterkc/repos\",\"events_url\":\"https://api.github.com/users/peterkc/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/peterkc/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2015-01-01T00:47:14Z\",\"updated_at\":\"2015-01-01T01:06:03Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/box/leche/pulls/5\",\"html_url\":\"https://github.com/box/leche/pull/5\",\"diff_url\":\"https://github.com/box/leche/pull/5.diff\",\"patch_url\":\"https://github.com/box/leche/pull/5.patch\"},\"body\":\"Updating dependencies (#4) also required fixing some lint issues (#3) in leche.js\"},\"comment\":{\"url\":\"https://api.github.com/repos/box/leche/issues/comments/68477348\",\"html_url\":\"https://github.com/box/leche/pull/5#issuecomment-68477348\",\"issue_url\":\"https://api.github.com/repos/box/leche/issues/5\",\"id\":68477348,\"user\":{\"login\":\"boxcla\",\"id\":6845864,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6845864?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/boxcla\",\"html_url\":\"https://github.com/boxcla\",\"followers_url\":\"https://api.github.com/users/boxcla/followers\",\"following_url\":\"https://api.github.com/users/boxcla/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/boxcla/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/boxcla/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/boxcla/subscriptions\",\"organizations_url\":\"https://api.github.com/users/boxcla/orgs\",\"repos_url\":\"https://api.github.com/users/boxcla/repos\",\"events_url\":\"https://api.github.com/users/boxcla/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/boxcla/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:03Z\",\"updated_at\":\"2015-01-01T01:06:03Z\",\"body\":\"Verified that @peterkc has just signed the CLA. Thanks, and we look forward to your contribution.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:03Z\",\"org\":{\"id\":23900,\"login\":\"box\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/box\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/23900?\"}}\n{\"id\":\"2489398437\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":646600,\"login\":\"mikeshulman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mikeshulman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/646600?\"},\"repo\":{\"id\":6569872,\"name\":\"HoTT/book\",\"url\":\"https://api.github.com/repos/HoTT/book\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/HoTT/book/issues/744\",\"labels_url\":\"https://api.github.com/repos/HoTT/book/issues/744/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/HoTT/book/issues/744/comments\",\"events_url\":\"https://api.github.com/repos/HoTT/book/issues/744/events\",\"html_url\":\"https://github.com/HoTT/book/issues/744\",\"id\":51177378,\"number\":744,\"title\":\"Proofs of Lems. 7.2.9 and 7.3.1\",\"user\":{\"login\":\"kristinas\",\"id\":1341988,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1341988?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kristinas\",\"html_url\":\"https://github.com/kristinas\",\"followers_url\":\"https://api.github.com/users/kristinas/followers\",\"following_url\":\"https://api.github.com/users/kristinas/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kristinas/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kristinas/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kristinas/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kristinas/orgs\",\"repos_url\":\"https://api.github.com/users/kristinas/repos\",\"events_url\":\"https://api.github.com/users/kristinas/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kristinas/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/HoTT/book/labels/math+question\",\"name\":\"math question\",\"color\":\"5319e7\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2014-12-06T09:36:05Z\",\"updated_at\":\"2015-01-01T01:06:04Z\",\"closed_at\":null,\"body\":\"Dear all,\\r\\n\\r\\nI do not quite see what is going on in the proofs of lemmas 7.2.9 and 7.3.1.\\r\\n\\r\\nThe first lemma states that for n>= -1, A is an n-type iff the iterated loop space \\\\omega^{n+1}(A,a) is contractible for all a :A. Let us consider an equivalent statement instead, which says that for n>= -1, is-n-type(A) is equivalent to\\r\\n\\\\Pi{a:A} is-contr(\\\\omega^{n+1}(A,a)).\\r\\n\\r\\nThe base case when n = -1 is Exercise 3.5, as in the current proof. For the inductive case it suffices to prove the logical equivalence between the two types, as they are both mere props. We have that is-(n+1)-type(A) is by definition\\r\\n\\\\Pi{x,y:A} is-n-type(x=y). By induction, this is equivalent to \\\\Pi{x,y:A} \\\\Pi{p:x=y} is-contr(\\\\omega^{n+1}(x=y,p)). On the other hand, \\\\Pi{x:A} is-contr(\\\\omega^{n+2}(A,x)) is definitionally equal to \\\\Pi{x:A} is-contr(\\\\omega^{n+1}(x=x,refl(x))).\\r\\nIt remains to show that the two types \\\\Pi{x,y:A} \\\\Pi{p:x=y} is-contr(\\\\omega^{n+1}(x=y,p)) and\\r\\n\\\\Pi{x:A} is-contr(\\\\omega^{n+1}(x=x,refl(x))) are logically equivalent. Going from left to right is obvious and from right to left is just path induction on p. So we are done. No?\\r\\n\\r\\nLemma 7.3.1 states that the n-truncation of A is an n-type. When reading the section on truncations for the first time, I was very confused by the motivation given via iterated loop spaces and maps of pointed types. Eventually I gave up and tried to understand it in a different way.\\r\\nFor example, we can first prove a lemma which says that for n >= -1, the statement that X is an n-type is equivalent to saying that, loosely speaking, any function from S^{n+1} to X is 'constant' (I am putting it in quotes since people like to argue about the terminology). Formally, for n >= -1, is-n-type(X) is equivalent to\\r\\n\\\\Pi{r: S^{n+1} \\\\to X} \\\\Sigma {h : X} \\\\Pi{z: S^{n+1}} r(z) = h. There is a simple proof by induction on n. I am not sure if this lemma might be useful in other places as a general characteristic of an n-type, but at the very least one direction of it is used in Theorem 7.3.2 without explicit justification.\\r\\nOf course, once we have the lemma the motivation for defining the truncation as we did, as well as the subsequent proof that the truncation is really an n-type, is trivial.\\r\\n\\r\\nPlease let me know what you think.\\r\\n\\r\\nKristina\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/HoTT/book/issues/comments/68477349\",\"html_url\":\"https://github.com/HoTT/book/issues/744#issuecomment-68477349\",\"issue_url\":\"https://api.github.com/repos/HoTT/book/issues/744\",\"id\":68477349,\"user\":{\"login\":\"mikeshulman\",\"id\":646600,\"avatar_url\":\"https://avatars.githubusercontent.com/u/646600?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mikeshulman\",\"html_url\":\"https://github.com/mikeshulman\",\"followers_url\":\"https://api.github.com/users/mikeshulman/followers\",\"following_url\":\"https://api.github.com/users/mikeshulman/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/mikeshulman/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/mikeshulman/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/mikeshulman/subscriptions\",\"organizations_url\":\"https://api.github.com/users/mikeshulman/orgs\",\"repos_url\":\"https://api.github.com/users/mikeshulman/repos\",\"events_url\":\"https://api.github.com/users/mikeshulman/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/mikeshulman/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:04Z\",\"updated_at\":\"2015-01-01T01:06:04Z\",\"body\":\"There's no a priori reason the proofs in the book are the simplest ones; they're just the ones that whoever was writing that chapter thought of at the time.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:04Z\",\"org\":{\"id\":692156,\"login\":\"HoTT\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/HoTT\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/692156?\"}}\n{\"id\":\"2489398439\",\"type\":\"CreateEvent\",\"actor\":{\"id\":9402134,\"login\":\"kesiena115\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kesiena115\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9402134?\"},\"repo\":{\"id\":28678279,\"name\":\"kesiena115/R1\",\"url\":\"https://api.github.com/repos/kesiena115/R1\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Project Repository\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398441\",\"type\":\"PushEvent\",\"actor\":{\"id\":734484,\"login\":\"smcelhinney\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/smcelhinney\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/734484?\"},\"repo\":{\"id\":28406073,\"name\":\"smcelhinney/fireblog\",\"url\":\"https://api.github.com/repos/smcelhinney/fireblog\"},\"payload\":{\"push_id\":536753352,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"875e125f1d832fd1d4b917d905c44ccf209ec632\",\"before\":\"c86fb3faded34b9979276557ee83beb0dc414b59\",\"commits\":[{\"sha\":\"875e125f1d832fd1d4b917d905c44ccf209ec632\",\"author\":{\"email\":\"9be487b550ce83311cfee6c22e48907b229f1a73@tcd.ie\",\"name\":\"smcelhinney\"},\"message\":\"Fixed some JSHINT stuff.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/smcelhinney/fireblog/commits/875e125f1d832fd1d4b917d905c44ccf209ec632\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398442\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":12637,\"login\":\"rwjblue\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rwjblue\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/12637?\"},\"repo\":{\"id\":14089735,\"name\":\"ember-cli/ember-cli\",\"url\":\"https://api.github.com/repos/ember-cli/ember-cli\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/2091\",\"labels_url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/2091/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/2091/comments\",\"events_url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/2091/events\",\"html_url\":\"https://github.com/ember-cli/ember-cli/issues/2091\",\"id\":43685212,\"number\":2091,\"title\":\"How to create or keep a symlink in dist\",\"user\":{\"login\":\"pradius-fut\",\"id\":1803477,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1803477?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pradius-fut\",\"html_url\":\"https://github.com/pradius-fut\",\"followers_url\":\"https://api.github.com/users/pradius-fut/followers\",\"following_url\":\"https://api.github.com/users/pradius-fut/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/pradius-fut/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/pradius-fut/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/pradius-fut/subscriptions\",\"organizations_url\":\"https://api.github.com/users/pradius-fut/orgs\",\"repos_url\":\"https://api.github.com/users/pradius-fut/repos\",\"events_url\":\"https://api.github.com/users/pradius-fut/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/pradius-fut/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":7,\"created_at\":\"2014-09-23T21:16:23Z\",\"updated_at\":\"2015-01-01T01:06:05Z\",\"closed_at\":\"2014-10-12T15:17:47Z\",\"body\":\"i would like to have a symlink `/api` in the dist folder to an api webroot outside the ember app root.\\r\\nHow can i create this symlink during the build process, or make sure it stays in the dist dir (when i create it manually).\\r\\n\\r\\nI'm testing ember and ember-cli, i have the same setup for the application in backbone/grunt where this is working fine.\"},\"comment\":{\"url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/comments/68477351\",\"html_url\":\"https://github.com/ember-cli/ember-cli/issues/2091#issuecomment-68477351\",\"issue_url\":\"https://api.github.com/repos/ember-cli/ember-cli/issues/2091\",\"id\":68477351,\"user\":{\"login\":\"rwjblue\",\"id\":12637,\"avatar_url\":\"https://avatars.githubusercontent.com/u/12637?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rwjblue\",\"html_url\":\"https://github.com/rwjblue\",\"followers_url\":\"https://api.github.com/users/rwjblue/followers\",\"following_url\":\"https://api.github.com/users/rwjblue/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/rwjblue/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/rwjblue/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/rwjblue/subscriptions\",\"organizations_url\":\"https://api.github.com/users/rwjblue/orgs\",\"repos_url\":\"https://api.github.com/users/rwjblue/repos\",\"events_url\":\"https://api.github.com/users/rwjblue/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/rwjblue/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:05Z\",\"updated_at\":\"2015-01-01T01:06:05Z\",\"body\":\"@ef4: Its possible that it was always wrong, but I believe that at the time I wrote it we were using `broccoli-export-tree` so the `results.directory` was actually `dist/`.   Either way you are right, and it is incorrect today.  The general concept is sound though (making a symlink inside the `postBuild` hook).\\r\\n\\r\\nIt does seem that we should expose the `builder` model's `this.outputPath` to the hook though.  Otherwise, you would be assuming `dist/` which is not guaranteed.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\",\"org\":{\"id\":10262982,\"login\":\"ember-cli\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ember-cli\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10262982?\"}}\n{\"id\":\"2489398443\",\"type\":\"PushEvent\",\"actor\":{\"id\":5240798,\"login\":\"hxwang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hxwang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5240798?\"},\"repo\":{\"id\":20258812,\"name\":\"hxwang/Leetcode\",\"url\":\"https://api.github.com/repos/hxwang/Leetcode\"},\"payload\":{\"push_id\":536753353,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e67f3471c2ea1c6556ff18137af91f8d92e5e39d\",\"before\":\"3b594d10e300b52c24bffed44d66ea7aae8e7639\",\"commits\":[{\"sha\":\"e67f3471c2ea1c6556ff18137af91f8d92e5e39d\",\"author\":{\"email\":\"320cb2ff8e2e195f7d4e5cd3b27b690e919d61e5@gmail.com\",\"name\":\"Huangxin\"},\"message\":\"Update Implement-strStr.java\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxwang/Leetcode/commits/e67f3471c2ea1c6556ff18137af91f8d92e5e39d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398444\",\"type\":\"PushEvent\",\"actor\":{\"id\":296116,\"login\":\"raorao\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/raorao\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/296116?\"},\"repo\":{\"id\":28544148,\"name\":\"raorao/groceries\",\"url\":\"https://api.github.com/repos/raorao/groceries\"},\"payload\":{\"push_id\":536753354,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"c7a2701ccbe6ce99648377c56a826e0446d44ff9\",\"before\":\"2cbbf1b1c59eac0c09118d8bbf2465b5d8113959\",\"commits\":[{\"sha\":\"c7a2701ccbe6ce99648377c56a826e0446d44ff9\",\"author\":{\"email\":\"4c1374f3c4f425d5dc04034017c8940e6ba2f209@gmail.com\",\"name\":\"raorao\"},\"message\":\"adds explicit body-parser dependency\",\"distinct\":true,\"url\":\"https://api.github.com/repos/raorao/groceries/commits/c7a2701ccbe6ce99648377c56a826e0446d44ff9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398445\",\"type\":\"PushEvent\",\"actor\":{\"id\":2874368,\"login\":\"janie177\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/janie177\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2874368?\"},\"repo\":{\"id\":28406300,\"name\":\"janie177/MGRacesRedone\",\"url\":\"https://api.github.com/repos/janie177/MGRacesRedone\"},\"payload\":{\"push_id\":536753355,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"73f1823a1a9f25ba53c15b7ec32607e1bed77e09\",\"before\":\"d1da31bff6f19d84613fa5ef6afb9124ca12d28c\",\"commits\":[{\"sha\":\"73f1823a1a9f25ba53c15b7ec32607e1bed77e09\",\"author\":{\"email\":\"56d775a7c13bb3dbfe56eb5fa7074a17cc2db31c@hotmail.com\",\"name\":\"Jan\"},\"message\":\"Minor adjustments.\\nFixed potion effects and amplifiers being switched for mobs.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/janie177/MGRacesRedone/commits/73f1823a1a9f25ba53c15b7ec32607e1bed77e09\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398447\",\"type\":\"PushEvent\",\"actor\":{\"id\":248290,\"login\":\"lexmag\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lexmag\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/248290?\"},\"repo\":{\"id\":20156619,\"name\":\"lexmag/tty2048\",\"url\":\"https://api.github.com/repos/lexmag/tty2048\"},\"payload\":{\"push_id\":536753356,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d9cb70b865980e3ddba854ac4d08d791957304ee\",\"before\":\"5fc470c05be7c489aa9d865205e663bc8d2c33f6\",\"commits\":[{\"sha\":\"d9cb70b865980e3ddba854ac4d08d791957304ee\",\"author\":{\"email\":\"24e2170222e81d4e3239eb4676aa506f230bfdd9@me.com\",\"name\":\"Aleksei Magusev\"},\"message\":\"Do not rely on the Crypto application\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lexmag/tty2048/commits/d9cb70b865980e3ddba854ac4d08d791957304ee\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398449\",\"type\":\"PushEvent\",\"actor\":{\"id\":3112493,\"login\":\"djstroky\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/djstroky\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3112493?\"},\"repo\":{\"id\":28258658,\"name\":\"djstroky/backbone-google-calendar\",\"url\":\"https://api.github.com/repos/djstroky/backbone-google-calendar\"},\"payload\":{\"push_id\":536753358,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"293361e1dc564daae89ebb389032fb076a89e459\",\"before\":\"e2c85077a3ea6dd238684e62e049d68fb686c497\",\"commits\":[{\"sha\":\"293361e1dc564daae89ebb389032fb076a89e459\",\"author\":{\"email\":\"b9600dec4a2cd1f0ecf0c56887050a58cde1ff86@yahoo.com\",\"name\":\"djstroky\"},\"message\":\"fixed misspelling\",\"distinct\":true,\"url\":\"https://api.github.com/repos/djstroky/backbone-google-calendar/commits/293361e1dc564daae89ebb389032fb076a89e459\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:06Z\"}\n{\"id\":\"2489398458\",\"type\":\"PushEvent\",\"actor\":{\"id\":7336721,\"login\":\"aow1980\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aow1980\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7336721?\"},\"repo\":{\"id\":28617444,\"name\":\"aow1980/kernel_lge_hammerhead\",\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead\"},\"payload\":{\"push_id\":536753361,\"size\":33,\"distinct_size\":33,\"ref\":\"refs/heads/lp5.0\",\"head\":\"754847ec691ce0d45103f052e86aafe744cddc81\",\"before\":\"a4e247102c297d001cc6b1e564060ae3fe7d174b\",\"commits\":[{\"sha\":\"6adbdc70a8846302e95d7c637d4d601848072264\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"block: row: Fix crash when adding a new field in bio struct\\n\\nWhen adding new field to struct bio there is a crash in the removed\\ncode lines. This issue was introduced by commit\\n80a8f0f87bee18283e9ca0a8966ec97ad9f084e5  \\\"block: row-iosched idling\\ntriggered by readahead pages\\\"\\n\\n(Partly) reverting this patch till root cause is fixed (on FS level).\\n\\nChange-Id: Idce180802227aaab495bf0723768ba4cb437bcab\\nSigned-off-by: Tanya Brokhman <tlinder@codeaurora.org>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\nSigned-off-by: flar2 <asegaert@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/6adbdc70a8846302e95d7c637d4d601848072264\"},{\"sha\":\"cc6449f427439b647e8e521db660d15d1d20ed73\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"block: introduce the BFQ-v6 I/O sched for 3.4\\n\\nAdd the BFQ-v6 I/O scheduler to 3.4.\\nThe general structure is borrowed from CFQ, as much code. A (bfq_)queue is\\nassociated to each task doing I/O on a device, and each time a scheduling\\ndecision has to be made a queue is selected and served until it expires.\\n\\n    - Slices are given in the service domain: tasks are assigned budgets,\\n      measured in number of sectors. Once got the disk, a task must\\n      however consume its assigned budget within a configurable maximum time\\n      (by default, the maximum possible value of the budgets is automatically\\n      computed to comply with this timeout). This allows the desired latency\\n      vs \\\"throughput boosting\\\" tradeoff to be set.\\n\\n    - Budgets are scheduled according to a variant of WF2Q+, implemented\\n      using an augmented rb-tree to take eligibility into account while\\n      preserving an O(log N) overall complexity.\\n\\n    - A low-latency tunable is provided; if enabled, both interactive and soft\\n      real-time applications are guaranteed very low latency.\\n\\n    - Latency guarantees are preserved also in presence of NCQ.\\n\\n    - Also with flash-based devices, a high throughput is achieved while\\n      still preserving latency guarantees.\\n\\n    - A useful feature borrowed from CFQ: static fallback queue for OOM.\\n\\n    - Differently from CFQ, BFQ uses a unified mechanism (Early Queue Merge,\\n      EQM) to get a sequential read pattern, and hence a high throughput,\\n      with any set of processes performing interleaved I/O. EQM also\\n      preserves low latency. The code for detecting whether two queues have\\n      to be merged is a slightly modified version of the CFQ code for\\n      detecting whether two queues belong to cooperating processes and whether\\n      the service of a queue should be preempted to boost the throughput.\\n\\n    - BFQ supports full hierarchical scheduling, exporting a cgroups\\n      interface.  Each node has a full scheduler, so each group can\\n      be assigned its own ioprio (mapped to a weight, see next point)\\n      and an ioprio_class.\\n\\n    - If the cgroups interface is used, weights can be explictly assigned,\\n      otherwise ioprio values are mapped to weights using the relation\\n      weight = IOPRIO_BE_NR - ioprio.\\n\\n    - ioprio classes are served in strict priority order, i.e., lower\\n      priority queues are not served as long as there are higher priority\\n      queues.  Among queues in the same class the bandwidth is distributed\\n      in proportion to the weight of each queue. A very thin extra bandwidth\\n      is however guaranteed to the Idle class, to prevent it from starving.\\n\\nChange-Id: Iae219b7532e7c41a88601a77f86f8f20211c30da\\nSigned-off-by: Paolo Valente <paolo.valente@unimore.it>\\nSigned-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>\\nSigned-off-by: flar2 <asegaert@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/cc6449f427439b647e8e521db660d15d1d20ed73\"},{\"sha\":\"831b6b7d7b89efd711eef2e14eb8b9f46b4916c5\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"block: cgroups, kconfig, build bits for BFQ-v6-3.4\\n\\nUpdate Kconfig.iosched and do the related Makefile changes to include\\nkernel configuration options for BFQ. Also add the bfqio controller\\nto the cgroups subsystem.\\n\\nChange-Id: I51837750d0a3ac5da9c5adc0e3dcff5894bf1fc8\\nSigned-off-by: Paolo Valente <paolo.valente@unimore.it>\\nSigned-off-by: Arianna Avanzini <avanzini.arianna@gmail.com>\\nSigned-off-by: flar2 <asegaert@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/831b6b7d7b89efd711eef2e14eb8b9f46b4916c5\"},{\"sha\":\"ecdda7ef938d5aa2db8dc8aa4da733328d6ea836\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"remove Defconfigs I don't use\\n\\nChange-Id: I507046245020147e89f11c33414ebf72e25df61a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/ecdda7ef938d5aa2db8dc8aa4da733328d6ea836\"},{\"sha\":\"d43565d19aeb25eb8432c8636bf7c4d0a18fc974\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Add BFQ I/O Scheduler\\n\\nChange-Id: I1a415a3f3f35fbdc03d6737e4cfd5ba0deddde28\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/d43565d19aeb25eb8432c8636bf7c4d0a18fc974\"},{\"sha\":\"d3c5b34d36f3ea86b8d73555659edad098f2e06e\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"net: fix info leak in compat dev_ifconf()\\n\\nThe implementation of dev_ifconf() for the compat ioctl interface uses\\nan intermediate ifc structure allocated in userland for the duration of\\nthe syscall. Though, it fails to initialize the padding bytes inserted\\nfor alignment and that for leaks four bytes of kernel stack. Add an\\nexplicit memset(0) before filling the structure to avoid the info leak.\\n\\nSigned-off-by: Mathias Krause <minipli@googlemail.com>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: Block MSG_CMSG_COMPAT in send(m)msg and recv(m)msg\\n\\nMSG_CMSG_COMPAT is (AFAIK) not intended to be part of the API --\\nit's a hack that steals a bit to indicate to other networking code\\nthat a compat entry was used.  So don't allow it from a non-compat\\nsyscall.\\n\\nThis prevents an oops when running this code:\\n\\nint main()\\n{\\n\\tint s;\\n\\tstruct sockaddr_in addr;\\n\\tstruct msghdr *hdr;\\n\\n\\tchar *highpage = mmap((void*)(TASK_SIZE_MAX - 4096), 4096,\\n\\t                      PROT_READ | PROT_WRITE,\\n\\t                      MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);\\n\\tif (highpage == MAP_FAILED)\\n\\t\\terr(1, \\\"mmap\\\");\\n\\n\\ts = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);\\n\\tif (s == -1)\\n\\t\\terr(1, \\\"socket\\\");\\n\\n        addr.sin_family = AF_INET;\\n        addr.sin_port = htons(1);\\n        addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);\\n\\tif (connect(s, (struct sockaddr*)&addr, sizeof(addr)) != 0)\\n\\t\\terr(1, \\\"connect\\\");\\n\\n\\tvoid *evil = highpage + 4096 - COMPAT_MSGHDR_SIZE;\\n\\tprintf(\\\"Evil address is %p\\\\n\\\", evil);\\n\\n\\tif (syscall(__NR_sendmmsg, s, evil, 1, MSG_CMSG_COMPAT) < 0)\\n\\t\\terr(1, \\\"sendmmsg\\\");\\n\\n\\treturn 0;\\n}\\n\\nSigned-off-by: Andy Lutomirski <luto@amacapital.net>\\nCc: David S. Miller <davem@davemloft.net>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: heap overflow in __audit_sockaddr()\\n\\nWe need to cap ->msg_namelen or it leads to a buffer overflow when we\\nto the memcpy() in __audit_sockaddr().  It requires CAP_AUDIT_CONTROL to\\nexploit this bug.\\n\\nThe call tree is:\\n___sys_recvmsg()\\n  move_addr_to_user()\\n    audit_sockaddr()\\n      __audit_sockaddr()\\n\\nReported-by: Jüri Aedla <juri.aedla@gmail.com>\\nSigned-off-by: Dan Carpenter <dan.carpenter@oracle.com>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: add BUG_ON if kernel advertises msg_namelen > sizeof(struct sockaddr_storage)\\n\\nIn that case it is probable that kernel code overwrote part of the\\nstack. So we should bail out loudly here.\\n\\nThe BUG_ON may be removed in future if we are sure all protocols are\\nconformant.\\n\\nSuggested-by: Eric Dumazet <eric.dumazet@gmail.com>\\nSigned-off-by: Hannes Frederic Sowa <hannes@stressinduktion.org>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: clamp ->msg_namelen instead of returning an error\\n\\nIf kmsg->msg_namelen > sizeof(struct sockaddr_storage) then in the\\noriginal code that would lead to memory corruption in the kernel if you\\nhad audit configured.  If you didn't have audit configured it was\\nharmless.\\n\\nThere are some programs such as beta versions of Ruby which use too\\nlarge of a buffer and returning an error code breaks them.  We should\\nclamp the ->msg_namelen value instead.\\n\\nFixes: 1661bf364ae9 (\\\"net: heap overflow in __audit_sockaddr()\\\")\\nReported-by: Eric Wong <normalperson@yhbt.net>\\nSigned-off-by: Dan Carpenter <dan.carpenter@oracle.com>\\nTested-by: Eric Wong <normalperson@yhbt.net>\\nAcked-by: Eric Dumazet <edumazet@google.com>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: socket: error on a negative msg_namelen\\n\\nWhen copying in a struct msghdr from the user, if the user has set the\\nmsg_namelen parameter to a negative value it gets clamped to a valid\\nsize due to a comparison between signed and unsigned values.\\n\\nEnsure the syscall errors when the user passes in a negative value.\\n\\nSigned-off-by: Matthew Leach <matthew.leach@arm.com>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nnet: Fix references to out-of-scope variables in put_cmsg_compat()\\n\\nIn net/compat.c::put_cmsg_compat() we may assign 'data' the address of\\neither the 'ctv' or 'cts' local variables inside the 'if\\n(!COMPAT_USE_64BIT_TIME)' branch.\\n\\nThose variables go out of scope at the end of the 'if' statement, so\\nwhen we use 'data' further down in 'copy_to_user(CMSG_COMPAT_DATA(cm),\\ndata, cmlen - sizeof(struct compat_cmsghdr))' there's no telling what\\nit may be refering to - not good.\\n\\nFix the problem by simply giving 'ctv' and 'cts' function scope.\\n\\nChange-Id: I8ebc2efb0ae5d1009804698bebefb16a2b5fd2dd\\nSigned-off-by: Jesper Juhl <jj@chaosbits.net>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/d3c5b34d36f3ea86b8d73555659edad098f2e06e\"},{\"sha\":\"ec249eef47eeef64595c1d8a12ac530e1884d6c8\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Fix order of arguments to compat_put_time[spec|val]\\n\\nCommit 644595f89620 (\\\"compat: Handle COMPAT_USE_64BIT_TIME in\\nnet/socket.c\\\") introduced a bug where the helper functions to take\\neither a 64-bit or compat time[spec|val] got the arguments in the wrong\\norder, passing the kernel stack pointer off as a user pointer (and vice\\nversa).\\n\\nBecause of the user address range check, that in turn then causes an\\nEFAULT due to the user pointer range checking failing for the kernel\\naddress.  Incorrectly resuling in a failed system call for 32-bit\\nprocesses with a 64-bit kernel.\\n\\nOn odder architectures like HP-PA (with separate user/kernel address\\nspaces), it can be used read kernel memory.\\n\\nChange-Id: I012dfbe26f92558e2203ae36db662bf33deb4329\\nSigned-off-by: Mikulas Patocka <mpatocka@redhat.com>\\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/ec249eef47eeef64595c1d8a12ac530e1884d6c8\"},{\"sha\":\"c434ee7f4a569336e1790f39ce1bc018e16197d0\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"af_key: fix info leaks in notify messages\\n\\nkey_notify_sa_flush() and key_notify_policy_flush() miss to initialize\\nthe sadb_msg_reserved member of the broadcasted message and thereby\\nleak 2 bytes of heap memory to listeners. Fix that.\\n\\nSigned-off-by: Mathias Krause <minipli@googlemail.com>\\nCc: Steffen Klassert <steffen.klassert@secunet.com>\\nCc: \\\"David S. Miller\\\" <davem@davemloft.net>\\nCc: Herbert Xu <herbert@gondor.apana.org.au>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\naf_key: more info leaks in pfkey messages\\n\\nThis is inspired by a5cc68f3d6 \\\"af_key: fix info leaks in notify\\nmessages\\\".  There are some struct members which don't get initialized\\nand could disclose small amounts of private information.\\n\\nAcked-by: Mathias Krause <minipli@googlemail.com>\\nSigned-off-by: Dan Carpenter <dan.carpenter@oracle.com>\\nAcked-by: Steffen Klassert <steffen.klassert@secunet.com>\\nSigned-off-by: David S. Miller <davem@davemloft.net>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\naf_key: initialize satype in key_notify_policy_flush()\\n\\nThis field was left uninitialized. Some user daemons perform check against this\\nfield.\\n\\nChange-Id: I747fca7338e0441afdd418fba5dfec51aa94f1c4\\nSigned-off-by: Nicolas Dichtel <nicolas.dichtel@6wind.com>\\nSigned-off-by: Steffen Klassert <steffen.klassert@secunet.com>\\nCc: Luis Henriques <luis.henriques@canonical.com>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/c434ee7f4a569336e1790f39ce1bc018e16197d0\"},{\"sha\":\"f54b415df6c0a73280bc28a7c572ae748b588bc2\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Fix blocking allocations called very early during bootup\\n\\nDuring early boot, when the scheduler hasn't really been fully set up,\\nwe really can't do blocking allocations because with certain (dubious)\\nconfigurations the \\\"might_resched()\\\" calls can actually result in\\nscheduling events.\\n\\nWe could just make such users always use GFP_ATOMIC, but quite often the\\ncode that does the allocation isn't really aware of the fact that the\\nscheduler isn't up yet, and forcing that kind of random knowledge on the\\ninitialization code is just annoying and not good for anybody.\\n\\nAnd we actually have a the 'gfp_allowed_mask' exactly for this reason:\\nit's just that the kernel init sequence happens to set it to allow\\nblocking allocations much too early.\\n\\nSo move the 'gfp_allowed_mask' initialization from 'start_kernel()'\\n(which is some of the earliest init code, and runs with preemption\\ndisabled for good reasons) into 'kernel_init()'.  kernel_init() is run\\nin the newly created thread that will become the 'init' process, as\\nopposed to the early startup code that runs within the context of what\\nwill be the first idle thread.\\n\\nSo by the time we reach 'kernel_init()', we know that the scheduler must\\nbe at least limping along, because we've already scheduled from the idle\\nthread into the init thread.\\n\\nReported-by: Steven Rostedt <rostedt@goodmis.org>\\nCc: David Rientjes <rientjes@google.com>\\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\nmodule_param: stop double-calling parameters.\\n\\nCommit 026cee0086fe1df4cf74691cf273062cc769617d \\\"params:\\n<level>_initcall-like kernel parameters\\\" set old-style module\\nparameters to level 0.  And we call those level 0 calls where we used\\nto, early in start_kernel().\\n\\nWe also loop through the initcall levels and call the levelled\\nmodule_params before the corresponding initcall.  Unfortunately level\\n0 is early_init(), so we call the standard module_param calls twice.\\n\\n(Turns out most things don't care, but at least ubi.mtd does).\\n\\nChange the level to -1 for standard module_param calls.\\n\\nReported-by: Benoît Thébaudeau <benoit.thebaudeau@advansee.com>\\nSigned-off-by: Rusty Russell <rusty@rustcorp.com.au>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\nof: fdt: fix memory initialization for expanded DT\\n\\nAlready existing property flags are filled wrong for properties created from\\ninitial FDT. This could cause problems if this DYNAMIC device-tree functions\\nare used later, i.e. properties are attached/detached/replaced. Simply dumping\\nflags from the running system show, that some initial static (not allocated via\\nkzmalloc()) nodes are marked as dynamic.\\n\\nI putted some debug extensions to property_proc_show(..) :\\n..\\n+       if (OF_IS_DYNAMIC(pp))\\n+               pr_err(\\\"DEBUG: xxx : OF_IS_DYNAMIC\\\\n\\\");\\n+       if (OF_IS_DETACHED(pp))\\n+               pr_err(\\\"DEBUG: xxx : OF_IS_DETACHED\\\\n\\\");\\n\\nwhen you operate on the nodes (e.g.: ~$ cat /proc/device-tree/*some_node*) you\\nwill see that those flags are filled wrong, basically in most cases it will dump\\na DYNAMIC or DETACHED status, which is in not true.\\n(BTW. this OF_IS_DETACHED is a own define for debug purposes which which just\\nmake a test_bit(OF_DETACHED, &x->_flags)\\n\\nIf nodes are dynamic kernel is allowed to kfree() them. But it will crash\\nattempting to do so on the nodes from FDT -- they are not allocated via\\nkzmalloc().\\n\\nSigned-off-by: Wladislav Wiebe <wladislav.kw@gmail.com>\\nAcked-by: Alexander Sverdlin <alexander.sverdlin@nsn.com>\\nSigned-off-by: Rob Herring <rob.herring@calxeda.com>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\nof: Fix missing memory initialization on FDT unflattening\\n\\nAny calls to dt_alloc() need to be zeroed. This is a temporary fix, but\\nthe allocation function itself needs to zero memory before returning\\nit. This is a follow up to patch 9e4012752, \\\"of: fdt: fix memory\\ninitialization for expanded DT\\\" which fixed one call site but missed\\nanother.\\n\\nSigned-off-by: Grant Likely <grant.likely@linaro.org>\\nAcked-by: Wladislav Wiebe <wladislav.kw@gmail.com>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\\nregulator: core: Release regulator-regulator supplies on error\\n\\nIf we fail while registering a regulator make sure we release the supply\\nfor the regulator if there is one.\\n\\nChange-Id: Icfa3c3087d9d91ee2272fa77614302e15fa0b247\\nSigned-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>\\nAcked-by: Liam Girdwood <lrg@ti.com>\\nSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>\\nSigned-off-by: Pranav Vashi <neobuddy89@gmail.com>\\nSigned-off-by: engstk <eng.stk@sapo.pt>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/f54b415df6c0a73280bc28a7c572ae748b588bc2\"},{\"sha\":\"76cdc5eff4f5f89aaa761726e0199d682db23183\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: add my msm thermal code.\\n\\nChange-Id: I081bef136e4066aec962c1600e27cf96a213119d\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/76cdc5eff4f5f89aaa761726e0199d682db23183\"},{\"sha\":\"ef7789edd4f11fb87d4c44f4e5d98acb32954a96\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: no need for this check.\\n\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\\nmsm: thermal: the wq can freezes during freezer phase.\\n\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\\nmsm: thermal: improve some codes.\\n\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\\ndrivers: thermal: back to 70C throttle point.\\n\\nChange-Id: Ice0bf89b16e2b5c3adec59dedbc90dcaf3e01658\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/ef7789edd4f11fb87d4c44f4e5d98acb32954a96\"},{\"sha\":\"dae1230cfafa03edcf45ac45d655368060253345\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"thermal: add a 5C threshold to prevent throttled/unthrottled to be triggered a lot when the CPU temp dwells close to the temp threshold.\\n\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\\nmsm: thermal: Update thermal subsystem\\n\\nThis is a squash of the following commits:\\n\\n4bc738d2 msm: thermal: Set vdd restriction initialize level to -1\\n5daeff0f msm: thermal: Clean up msm cpufreq function calls\\n6b21cae8 thermal: tsens: Fix invalid set temperature check\\nc473c9dd Revert \\\"msm: thermal: Clean up msm cpufreq function calls\\\"\\nbb646211 thermal: Add sensor API to allow any driver to set thresholds\\n718bf641 thermal: msm: tsens: Notify thermal framework on threshold cross\\n4ff40944 msm: thermal: Enable hotplug support in kernel\\n3af26db0 thermal: Fix sensor thresholds not accounted correctly\\nd8b1e1ec Revert \\\"Revert \\\"msm: thermal: Clean up msm cpufreq function calls\\\"\\\"\\nb2f16712 thermal: Add Support for enabling and disabling tsens trip\\n81c1b0b7 msm: thermal: Don't initialize hotplug thread when not configured\\n89961312 msm: thermal: Add support for emergency frequency mitigation\\n6460392a msm: thermal: Add IOCTL interface support to Kernel Thermal Monitor\\n\\nBug: 14139855\\nConflicts:\\n\\tdrivers/thermal/msm_thermal.c\\nSigned-off-by: Naveen Ramaraj <nramaraj@codeaurora.org>\\n\\nChange-Id: I22911e6ffe1a9f553b1c721341fbe530e03eec6c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/dae1230cfafa03edcf45ac45d655368060253345\"},{\"sha\":\"2cb3f5da76a724cd78ba57184fbc11ed2a529957\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: refactor the code.\\n\\nChange-Id: I313980b5ed2abd9761307db0bf40c08af96e7dc2\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/2cb3f5da76a724cd78ba57184fbc11ed2a529957\"},{\"sha\":\"a58513ad8a06c5bf56b0577026678eab08128aca\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: there's no need to ask for a policy, we can simply lift the limit with LONG_MAX. Also queue only on cpu0.\\n\\nChange-Id: I41559724a44f3e9b8bbce22aa4f15b254dfd73cb\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/a58513ad8a06c5bf56b0577026678eab08128aca\"},{\"sha\":\"9bc3ecbd3e5f1819321286d96979d95d6234f3c6\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: make sure we actually want to go through with the policy adjust since this can be triggered by any policy update with the cPUFREQ_ADJUST flag.\\n\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\\nmsm: thermal: change the init sequence.\\n\\nChange-Id: I6a3b8ad7f80bcb2484f2f69effd5d1105073e73b\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/9bc3ecbd3e5f1819321286d96979d95d6234f3c6\"},{\"sha\":\"d024894241a20ad55a57280127becf3d4013a628\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: thermal: add my Thermal solution\\n\\nChange-Id: I5fc653d6bcacf97b36f4f8ec63ccdf8dd94e1999\\nSigned-off-by: franciscofranco <franciscofranco.1990@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/d024894241a20ad55a57280127becf3d4013a628\"},{\"sha\":\"036a45f062aea6b8abb9bb0bbfc64f436d68d5ff\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"msm: rq_stats: Calculate load based on current freq limit\\n\\nPrior to this patch, load has been calculated based on an\\ninitial value of policy.cpuinfo.max_freq. This value may\\nchange for several reasons: userspace settings or thermal\\nthrottling (hello hammerhead!). This results in unreliable\\ncalculations that confuse mpdecision which accesses\\n/sys/devices/system/cpu/cpu0/rq-stats/* for stats info.\\nConsider current policy.max to fix this behaviour.\\n\\nChange-Id: Iae1f2fc7f4aa470f7e80371c1ebcf7be423f51ca\\nSigned-off-by: engstk <eng.stk@sapo.pt>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/036a45f062aea6b8abb9bb0bbfc64f436d68d5ff\"},{\"sha\":\"a025c1d886c10249118d2e16abed161c39d501cf\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"thermal: raise thermal frequency throttle limits\\n\\nChange-Id: I83f9a590ce30d965534426e517b566c0f704ab00\\nSigned-off-by: engstk <eng.stk@sapo.pt>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/a025c1d886c10249118d2e16abed161c39d501cf\"},{\"sha\":\"a6b0b481b5f9808b4ff5777e67392f38978df6f8\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Switch to Franco Thermal control\\n\\nChange-Id: I08eada544f6d1fb5cbbbf01e89978346760bceef\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/a6b0b481b5f9808b4ff5777e67392f38978df6f8\"},{\"sha\":\"88698edd466cabc71f5523640a07d9bd22409cce\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"writeback: fix race that cause writeback hung\\n\\nThere is a race between mark inode dirty and writeback thread, see the\\nfollowing scenario.  In this case, writeback thread will not run though\\nthere is dirty_io.\\n\\n__mark_inode_dirty()                                          bdi_writeback_workfn()\\n\\t...                                                       \\t...\\n\\tspin_lock(&inode->i_lock);\\n\\t...\\n\\tif (bdi_cap_writeback_dirty(bdi)) {\\n\\t    <<< assume wb has dirty_io, so wakeup_bdi is false.\\n\\t    <<< the following inode_dirty also have wakeup_bdi false.\\n\\t    if (!wb_has_dirty_io(&bdi->wb))\\n\\t\\t    wakeup_bdi = true;\\n\\t}\\n\\tspin_unlock(&inode->i_lock);\\n\\t                                                            <<< assume last dirty_io is removed here.\\n\\t                                                            pages_written = wb_do_writeback(wb);\\n\\t                                                            ...\\n\\t                                                            <<< work_list empty and wb has no dirty_io,\\n\\t                                                            <<< delayed_work will not be queued.\\n\\t                                                            if (!list_empty(&bdi->work_list) ||\\n\\t                                                                (wb_has_dirty_io(wb) && dirty_writeback_interval))\\n\\t                                                                queue_delayed_work(bdi_wq, &wb->dwork,\\n\\t                                                                    msecs_to_jiffies(dirty_writeback_interval * 10));\\n\\tspin_lock(&bdi->wb.list_lock);\\n\\tinode->dirtied_when = jiffies;\\n\\t<<< new dirty_io is added.\\n\\tlist_move(&inode->i_wb_list, &bdi->wb.b_dirty);\\n\\tspin_unlock(&bdi->wb.list_lock);\\n\\n\\t<<< though there is dirty_io, but wakeup_bdi is false,\\n\\t<<< so writeback thread will not be waked up and\\n\\t<<< the new dirty_io will not be flushed.\\n\\tif (wakeup_bdi)\\n\\t    bdi_wakeup_thread_delayed(bdi);\\n\\nWriteback will run until there is a new flush work queued.  This may cause\\na lot of dirty pages stay in memory for a long time.\\n\\nChange-Id: I4f25c8878d0aabcc3408af017f2948776565fead\\nSigned-off-by: Junxiao Bi <junxiao.bi@oracle.com>\\nReviewed-by: Jan Kara <jack@suse.cz>\\nCc: Fengguang Wu <fengguang.wu@intel.com>\\nSigned-off-by: Andrew Morton <akpm@linux-foundation.org>\\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\\nSigned-off-by: Francisco Franco <franciscofranco.1990@gmail.com>\\nSigned-off-by: Chet Kener <Cl3Kener@gmail.com>\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/kernel_lge_hammerhead/commits/88698edd466cabc71f5523640a07d9bd22409cce\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:07Z\"}\n{\"id\":\"2489398459\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1235097,\"login\":\"rmarinho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?\"},\"repo\":{\"id\":20463939,\"name\":\"XLabs/Xamarin-Forms-Labs\",\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/543\",\"labels_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/543/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/543/comments\",\"events_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/543/events\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/issues/543\",\"id\":53118143,\"number\":543,\"title\":\"Update Nuget build process\",\"user\":{\"login\":\"ravensorb\",\"id\":2222472,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2222472?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ravensorb\",\"html_url\":\"https://github.com/ravensorb\",\"followers_url\":\"https://api.github.com/users/ravensorb/followers\",\"following_url\":\"https://api.github.com/users/ravensorb/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ravensorb/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ravensorb/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ravensorb/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ravensorb/orgs\",\"repos_url\":\"https://api.github.com/users/ravensorb/repos\",\"events_url\":\"https://api.github.com/users/ravensorb/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ravensorb/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/labels/enhancement\",\"name\":\"enhancement\",\"color\":\"84b6eb\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"ravensorb\",\"id\":2222472,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2222472?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ravensorb\",\"html_url\":\"https://github.com/ravensorb\",\"followers_url\":\"https://api.github.com/users/ravensorb/followers\",\"following_url\":\"https://api.github.com/users/ravensorb/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ravensorb/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ravensorb/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ravensorb/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ravensorb/orgs\",\"repos_url\":\"https://api.github.com/users/ravensorb/repos\",\"events_url\":\"https://api.github.com/users/ravensorb/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ravensorb/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":20,\"created_at\":\"2014-12-30T14:17:00Z\",\"updated_at\":\"2015-01-01T01:06:07Z\",\"closed_at\":null,\"body\":\"Now that we have the main work done for porting to the new structure, we should probably update the nuget build process.  Anyone interested in taking this one?\"},\"comment\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/comments/68477353\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/issues/543#issuecomment-68477353\",\"issue_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/543\",\"id\":68477353,\"user\":{\"login\":\"rmarinho\",\"id\":1235097,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"html_url\":\"https://github.com/rmarinho\",\"followers_url\":\"https://api.github.com/users/rmarinho/followers\",\"following_url\":\"https://api.github.com/users/rmarinho/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/rmarinho/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/rmarinho/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/rmarinho/subscriptions\",\"organizations_url\":\"https://api.github.com/users/rmarinho/orgs\",\"repos_url\":\"https://api.github.com/users/rmarinho/repos\",\"events_url\":\"https://api.github.com/users/rmarinho/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/rmarinho/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:07Z\",\"updated_at\":\"2015-01-01T01:06:07Z\",\"body\":\"hey guys, great work, i m not having the time i wanted to help you guys.. but here are my 2 cents.. \\r\\n\\r\\nWe have decided before that XLabs next release will be 2.0, i think we should launch 2.0-pre1 where we don't have unified support, and we will launch the final version of 2.0 when Xamarin releases unified on stable. \\r\\n\\r\\nWe still need to put the packages as pre to see issues others can find in our new structure.\\r\\n\\r\\n@ravensorb  is the nuget process the same?  build everything for the release folder and nuget picks up from there? \\r\\n\\r\\nwe should also look and this is the time to remove obsolete or properties that are now covered on Xamarin Core, like stuff related to fonts (extended label if i recall)\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:08Z\",\"org\":{\"id\":7787062,\"login\":\"XLabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/XLabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?\"}}\n{\"id\":\"2489398461\",\"type\":\"PushEvent\",\"actor\":{\"id\":6241554,\"login\":\"leo-yuriev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/leo-yuriev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6241554?\"},\"repo\":{\"id\":23696666,\"name\":\"leo-yuriev/openldap-lmdb-challenge\",\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge\"},\"payload\":{\"push_id\":536753363,\"size\":14,\"distinct_size\":14,\"ref\":\"refs/heads/2.4-devel\",\"head\":\"3c4005e809a729cc9f6b63e39ee26e39b3f10fe8\",\"before\":\"f705475af08e6b50420a8ff22b71cfbe45cea409\",\"commits\":[{\"sha\":\"0fd73aa27bb25ed790d723ceaca16c4bf228a691\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"ps-build.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/0fd73aa27bb25ed790d723ceaca16c4bf228a691\"},{\"sha\":\"3b9f7bf005eb43645b2dfb3c1eb1341912ea39ca\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"qt-creator project\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/3b9f7bf005eb43645b2dfb3c1eb1341912ea39ca\"},{\"sha\":\"677cefedf3d6f46ae74b7b25db28b53d32788c3e\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix debug marco\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/677cefedf3d6f46ae74b7b25db28b53d32788c3e\"},{\"sha\":\"0ab01722504df3e913a7c7559a2f06ba54820470\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: using strerror_r()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/0ab01722504df3e913a7c7559a2f06ba54820470\"},{\"sha\":\"c32f6ea113ddac0cdd1a7dd1944994714d45863b\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: read/write ignored result.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/c32f6ea113ddac0cdd1a7dd1944994714d45863b\"},{\"sha\":\"dccc7ef0254abe37e3e91d527032dd8120249859\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: check getcmd() result\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/dccc7ef0254abe37e3e91d527032dd8120249859\"},{\"sha\":\"c20b58e144515fcb25c821d5bf2fd5a58eea3055\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: printf unused agrv[0] in main()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/c20b58e144515fcb25c821d5bf2fd5a58eea3055\"},{\"sha\":\"0bf7975dc8489f33e5f8a7331308fc3885e17aa6\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: printf %d without arg in main()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/0bf7975dc8489f33e5f8a7331308fc3885e17aa6\"},{\"sha\":\"d26104515ccff47d3a81465b5b033fe8fe3bb548\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: warning-errors for configure\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/d26104515ccff47d3a81465b5b033fe8fe3bb548\"},{\"sha\":\"cb104588617e0ee1754e0a9f96d93cb32a537725\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: build-warnings (most 'unused')\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/cb104588617e0ee1754e0a9f96d93cb32a537725\"},{\"sha\":\"da05f7f43f784f81f0a1fae8f86561a73eb83534\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: const\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/da05f7f43f784f81f0a1fae8f86561a73eb83534\"},{\"sha\":\"9bab6ea187c7633769858bbb70bb156ed2fd1fa8\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix: warnings (uninitialized)\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/9bab6ea187c7633769858bbb70bb156ed2fd1fa8\"},{\"sha\":\"6db3f1d5b7860f1d3e95d12cce171d9db699e51d\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"sasl callback's typecast\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/6db3f1d5b7860f1d3e95d12cce171d9db699e51d\"},{\"sha\":\"3c4005e809a729cc9f6b63e39ee26e39b3f10fe8\",\"author\":{\"email\":\"1f0a51c36efaa0f44e4899c26d2028681997c8ea@yuriev.ru\",\"name\":\"Leo Yuriev\"},\"message\":\"fix warnings (unused, uninitialized, misc);\",\"distinct\":true,\"url\":\"https://api.github.com/repos/leo-yuriev/openldap-lmdb-challenge/commits/3c4005e809a729cc9f6b63e39ee26e39b3f10fe8\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:08Z\"}\n{\"id\":\"2489398466\",\"type\":\"PushEvent\",\"actor\":{\"id\":1579058,\"login\":\"cessen\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cessen\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1579058?\"},\"repo\":{\"id\":28025559,\"name\":\"cessen/led\",\"url\":\"https://api.github.com/repos/cessen/led\"},\"payload\":{\"push_id\":536753365,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"95d6d3fd3eb8d1771e46c900cc04b1eeb6b8b3e4\",\"before\":\"28a4da97a7ea7ba7c9e99c0ebc160cceb70ba495\",\"commits\":[{\"sha\":\"95d6d3fd3eb8d1771e46c900cc04b1eeb6b8b3e4\",\"author\":{\"email\":\"49beb02aed03d4c7798ad211ffd5f7674b731bcb@cessen.com\",\"name\":\"Nathan Vegdahl\"},\"message\":\"Got remove_text() working.\\n\\nAt least, based on the unit tests so far.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cessen/led/commits/95d6d3fd3eb8d1771e46c900cc04b1eeb6b8b3e4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:08Z\"}\n{\"id\":\"2489398467\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":3103764,\"login\":\"carymrobbins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?\"},\"repo\":{\"id\":15573192,\"name\":\"carymrobbins/intellij-haskforce\",\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\",\"labels_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/comments\",\"events_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105/events\",\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\",\"id\":52960360,\"number\":105,\"title\":\"Issue90 type information\",\"user\":{\"login\":\"KasperJanssens\",\"id\":5415995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5415995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KasperJanssens\",\"html_url\":\"https://github.com/KasperJanssens\",\"followers_url\":\"https://api.github.com/users/KasperJanssens/followers\",\"following_url\":\"https://api.github.com/users/KasperJanssens/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/KasperJanssens/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/KasperJanssens/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/KasperJanssens/subscriptions\",\"organizations_url\":\"https://api.github.com/users/KasperJanssens/orgs\",\"repos_url\":\"https://api.github.com/users/KasperJanssens/repos\",\"events_url\":\"https://api.github.com/users/KasperJanssens/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/KasperJanssens/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-27T16:19:07Z\",\"updated_at\":\"2015-01-01T01:06:08Z\",\"closed_at\":null,\"pull_request\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/pulls/105\",\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105\",\"diff_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.diff\",\"patch_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105.patch\"},\"body\":\"Cary,\\r\\n\\r\\nFirst try of the type information. It seems quite stable, been using it for a few days (provided the configuration is correct, I suppose, didn't test what happens when ghc-modi is not correctly configured).\\r\\n\\r\\nI bound the type info call to the DocumentationProvider as well as to an action (alt - equals, like scala). I prefer the action, I think the documentation provider doesn't work so well. There are tests of the parsing of the output of ghc-modi, but not really of the documentationprovider, basically because of the abundance of static calls and the fact that I think they can only be mocked while testing, and statics can only be mocked through Powermock if I recall correctly, which would mean an extra test dependency and so on and so forth, so I left that to be your call.\\r\\n\\r\\nAlso, there is a weird behaviour that getting the editor creates a stack trace of around 5 kilometers long, something that seems like a threading issue, but the function seems to work. I don't really know why the stack trace happens, all the more because it only happens when calling the type information through the documentation provider (same code path is used when the action is called, but no stack trace). Maybe you know more what could go wrong, it looks like something intellij-related.\\r\\n\\r\\nSo, consider this a \\\"request for comment\\\" more than a pull request ;-)\\r\\n\\r\\nKasper\"},\"comment\":{\"url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/comments/68477354\",\"html_url\":\"https://github.com/carymrobbins/intellij-haskforce/pull/105#issuecomment-68477354\",\"issue_url\":\"https://api.github.com/repos/carymrobbins/intellij-haskforce/issues/105\",\"id\":68477354,\"user\":{\"login\":\"carymrobbins\",\"id\":3103764,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3103764?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carymrobbins\",\"html_url\":\"https://github.com/carymrobbins\",\"followers_url\":\"https://api.github.com/users/carymrobbins/followers\",\"following_url\":\"https://api.github.com/users/carymrobbins/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/carymrobbins/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/carymrobbins/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/carymrobbins/subscriptions\",\"organizations_url\":\"https://api.github.com/users/carymrobbins/orgs\",\"repos_url\":\"https://api.github.com/users/carymrobbins/repos\",\"events_url\":\"https://api.github.com/users/carymrobbins/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/carymrobbins/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:08Z\",\"updated_at\":\"2015-01-01T01:06:08Z\",\"body\":\"Nice work @KasperJanssens!  Sorry, barely getting around to trying this out.\\r\\n\\r\\nI've updated the **Files Changed** tab with comments and questions.  Overall, this is really good stuff.  I'm glad that you have tests, you've refactored some crufty stuff, and tapped into popups to provide a custom UI.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:08Z\"}\n{\"id\":\"2489398469\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1093204,\"login\":\"mlnlover11\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mlnlover11\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1093204?\"},\"repo\":{\"id\":15789289,\"name\":\"KJCracks/Clutch\",\"url\":\"https://api.github.com/repos/KJCracks/Clutch\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:08Z\",\"org\":{\"id\":2375988,\"login\":\"KJCracks\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/KJCracks\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2375988?\"}}\n{\"id\":\"2489398473\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2829718,\"login\":\"phister\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/phister\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829718?\"},\"repo\":{\"id\":28678276,\"name\":\"phister/CollegeFootballPlayoff\",\"url\":\"https://api.github.com/repos/phister/CollegeFootballPlayoff\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":null,\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:09Z\"}\n{\"id\":\"2489398478\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":706947,\"login\":\"d3athrow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?\"},\"repo\":{\"id\":10441188,\"name\":\"d3athrow/vgstation13\",\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397297\",\"id\":22397297,\"diff_hunk\":\"@@ -36,6 +37,7 @@\\n \\t/obj/item/weapon/screwdriver, /obj/item/weapon/weldingtool, /obj/item/weapon/wirecutters, /obj/item/weapon/wrench, /obj/item/device/multitool, \\\\\\n \\t/obj/item/device/radio, /obj/item/device/analyzer, /obj/item/weapon/gun/energy/laser, /obj/item/weapon/gun/energy/pulse_rifle, \\\\\\n \\t/obj/item/weapon/gun/energy/taser, /obj/item/weapon/melee/baton, /obj/item/weapon/gun/energy/gun)\\n+\\tflags = PLASMAGUARD\",\"path\":\"code/modules/clothing/spacesuits/ert.dm\",\"position\":12,\"original_position\":12,\"commit_id\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"original_commit_id\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"flags = FPRINT | TABLEPASS | STOPSPRESSUREDMAGE | PLASMAGUARD\",\"created_at\":\"2015-01-01T01:06:11Z\",\"updated_at\":\"2015-01-01T01:06:11Z\",\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2405#discussion_r22397297\",\"pull_request_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/22397297\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2405#discussion_r22397297\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\",\"id\":26675841,\"html_url\":\"https://github.com/d3athrow/vgstation13/pull/2405\",\"diff_url\":\"https://github.com/d3athrow/vgstation13/pull/2405.diff\",\"patch_url\":\"https://github.com/d3athrow/vgstation13/pull/2405.patch\",\"issue_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405\",\"number\":2405,\"state\":\"open\",\"locked\":false,\"title\":\"Added PLASMAGUARD flag to some spacesuits\",\"user\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Added the PLASMAGUARD flag to ERT suits, deathsquad suits, and CE/Atmos suits as requested in \\r\\n\\r\\nhttps://github.com/d3athrow/vgstation13/issues/2369\",\"created_at\":\"2014-12-30T07:18:28Z\",\"updated_at\":\"2015-01-01T01:06:11Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"fff34ea0a38796d850e3f5aba53410e1b7bde9c8\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/commits\",\"review_comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/comments\",\"review_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405/comments\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/b93293990c4d927f30a1f048939d427ee591e4a6\",\"head\":{\"label\":\"Rei1226:ratdicks\",\"ref\":\"ratdicks\",\"sha\":\"b93293990c4d927f30a1f048939d427ee591e4a6\",\"user\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":19230658,\"name\":\"vgstation13\",\"full_name\":\"Rei1226/vgstation13\",\"owner\":{\"login\":\"Rei1226\",\"id\":4923426,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4923426?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Rei1226\",\"html_url\":\"https://github.com/Rei1226\",\"followers_url\":\"https://api.github.com/users/Rei1226/followers\",\"following_url\":\"https://api.github.com/users/Rei1226/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Rei1226/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Rei1226/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Rei1226/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Rei1226/orgs\",\"repos_url\":\"https://api.github.com/users/Rei1226/repos\",\"events_url\":\"https://api.github.com/users/Rei1226/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Rei1226/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/Rei1226/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/Rei1226/vgstation13\",\"forks_url\":\"https://api.github.com/repos/Rei1226/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/Rei1226/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/Rei1226/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/Rei1226/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/Rei1226/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/Rei1226/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/Rei1226/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/Rei1226/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/Rei1226/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/Rei1226/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/Rei1226/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/Rei1226/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/Rei1226/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/Rei1226/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/Rei1226/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/Rei1226/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/Rei1226/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/Rei1226/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/Rei1226/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/Rei1226/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/Rei1226/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/Rei1226/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/Rei1226/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/Rei1226/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/Rei1226/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/Rei1226/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/Rei1226/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/Rei1226/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/Rei1226/vgstation13/releases{/id}\",\"created_at\":\"2014-04-28T07:36:48Z\",\"updated_at\":\"2014-12-30T08:33:17Z\",\"pushed_at\":\"2014-12-30T08:33:10Z\",\"git_url\":\"git://github.com/Rei1226/vgstation13.git\",\"ssh_url\":\"git@github.com:Rei1226/vgstation13.git\",\"clone_url\":\"https://github.com/Rei1226/vgstation13.git\",\"svn_url\":\"https://github.com/Rei1226/vgstation13\",\"homepage\":\"\",\"size\":722996,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"DM\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":false,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"d3athrow:Bleeding-Edge\",\"ref\":\"Bleeding-Edge\",\"sha\":\"ea27b58dbac4fdb65119b09648f3f4f2395125de\",\"user\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":10441188,\"name\":\"vgstation13\",\"full_name\":\"d3athrow/vgstation13\",\"owner\":{\"login\":\"d3athrow\",\"id\":706947,\"avatar_url\":\"https://avatars.githubusercontent.com/u/706947?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/d3athrow\",\"html_url\":\"https://github.com/d3athrow\",\"followers_url\":\"https://api.github.com/users/d3athrow/followers\",\"following_url\":\"https://api.github.com/users/d3athrow/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/d3athrow/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/d3athrow/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/d3athrow/subscriptions\",\"organizations_url\":\"https://api.github.com/users/d3athrow/orgs\",\"repos_url\":\"https://api.github.com/users/d3athrow/repos\",\"events_url\":\"https://api.github.com/users/d3athrow/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/d3athrow/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/d3athrow/vgstation13\",\"description\":\"This is the vgstation's fork of baystation12's code.\",\"fork\":true,\"url\":\"https://api.github.com/repos/d3athrow/vgstation13\",\"forks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/forks\",\"keys_url\":\"https://api.github.com/repos/d3athrow/vgstation13/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/d3athrow/vgstation13/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/d3athrow/vgstation13/teams\",\"hooks_url\":\"https://api.github.com/repos/d3athrow/vgstation13/hooks\",\"issue_events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/d3athrow/vgstation13/events\",\"assignees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/d3athrow/vgstation13/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/tags\",\"blobs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/d3athrow/vgstation13/languages\",\"stargazers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/stargazers\",\"contributors_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contributors\",\"subscribers_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscribers\",\"subscription_url\":\"https://api.github.com/repos/d3athrow/vgstation13/subscription\",\"commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/d3athrow/vgstation13/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/d3athrow/vgstation13/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/d3athrow/vgstation13/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/d3athrow/vgstation13/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/d3athrow/vgstation13/merges\",\"archive_url\":\"https://api.github.com/repos/d3athrow/vgstation13/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/d3athrow/vgstation13/downloads\",\"issues_url\":\"https://api.github.com/repos/d3athrow/vgstation13/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/d3athrow/vgstation13/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/d3athrow/vgstation13/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/d3athrow/vgstation13/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/d3athrow/vgstation13/releases{/id}\",\"created_at\":\"2013-06-02T19:39:54Z\",\"updated_at\":\"2014-12-31T20:06:46Z\",\"pushed_at\":\"2015-01-01T01:04:27Z\",\"git_url\":\"git://github.com/d3athrow/vgstation13.git\",\"ssh_url\":\"git@github.com:d3athrow/vgstation13.git\",\"clone_url\":\"https://github.com/d3athrow/vgstation13.git\",\"svn_url\":\"https://github.com/d3athrow/vgstation13\",\"homepage\":\"\",\"size\":937605,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"DM\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":135,\"mirror_url\":null,\"open_issues_count\":259,\"forks\":135,\"open_issues\":259,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405\"},\"html\":{\"href\":\"https://github.com/d3athrow/vgstation13/pull/2405\"},\"issue\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405\"},\"comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/issues/2405/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/pulls/2405/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/d3athrow/vgstation13/statuses/b93293990c4d927f30a1f048939d427ee591e4a6\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:06:11Z\"}\n{\"id\":\"2489398480\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/83\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/83\",\"id\":53210289,\"number\":83,\"title\":\"Add icons to the left of the text on About screen buttons\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":5,\"closed_issues\":12,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"due_on\":null,\"closed_at\":null},\"comments\":1,\"created_at\":\"2015-01-01T01:05:59Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"closed_at\":\"2015-01-01T01:06:12Z\",\"body\":\"Related icons aligned to the left on the buttons on about screen\"},\"comment\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/comments/68477355\",\"html_url\":\"https://github.com/g19-mr/azh/issues/83#issuecomment-68477355\",\"issue_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83\",\"id\":68477355,\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:12Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"body\":\"Added icons to about screen buttons\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:12Z\"}\n{\"id\":\"2489398481\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":5497952,\"login\":\"g19-mr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?\"},\"repo\":{\"id\":25435487,\"name\":\"g19-mr/azh\",\"url\":\"https://api.github.com/repos/g19-mr/azh\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/issues/83\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/comments\",\"events_url\":\"https://api.github.com/repos/g19-mr/azh/issues/83/events\",\"html_url\":\"https://github.com/g19-mr/azh/issues/83\",\"id\":53210289,\"number\":83,\"title\":\"Add icons to the left of the text on About screen buttons\",\"user\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/g19-mr/azh/labels/improvement\",\"name\":\"improvement\",\"color\":\"84b6eb\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2\",\"labels_url\":\"https://api.github.com/repos/g19-mr/azh/milestones/2/labels\",\"id\":873798,\"number\":2,\"title\":\"Android 1.0.1\",\"description\":null,\"creator\":{\"login\":\"g19-mr\",\"id\":5497952,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5497952?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/g19-mr\",\"html_url\":\"https://github.com/g19-mr\",\"followers_url\":\"https://api.github.com/users/g19-mr/followers\",\"following_url\":\"https://api.github.com/users/g19-mr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/g19-mr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/g19-mr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/g19-mr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/g19-mr/orgs\",\"repos_url\":\"https://api.github.com/users/g19-mr/repos\",\"events_url\":\"https://api.github.com/users/g19-mr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/g19-mr/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":5,\"closed_issues\":12,\"state\":\"open\",\"created_at\":\"2014-11-20T05:20:02Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"due_on\":null,\"closed_at\":null},\"comments\":1,\"created_at\":\"2015-01-01T01:05:59Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"closed_at\":\"2015-01-01T01:06:12Z\",\"body\":\"Related icons aligned to the left on the buttons on about screen\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:12Z\"}\n{\"id\":\"2489398482\",\"type\":\"PushEvent\",\"actor\":{\"id\":3299558,\"login\":\"jonlai\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jonlai\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3299558?\"},\"repo\":{\"id\":24589839,\"name\":\"jonlai/personal-website\",\"url\":\"https://api.github.com/repos/jonlai/personal-website\"},\"payload\":{\"push_id\":536753371,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cb8c22571f6e99bd45efa105692f9c25d9fa616e\",\"before\":\"b292f784b928993eb5d993b54f9db897a1d6ae89\",\"commits\":[{\"sha\":\"cb8c22571f6e99bd45efa105692f9c25d9fa616e\",\"author\":{\"email\":\"3692bfa45759a67d83aedf0045f6cb635a966abf@jonlai.com\",\"name\":\"jonlai\"},\"message\":\"Add mobile-menu functionality\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jonlai/personal-website/commits/cb8c22571f6e99bd45efa105692f9c25d9fa616e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:12Z\"}\n{\"id\":\"2489398483\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":585534,\"login\":\"gorhill\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gorhill\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/585534?\"},\"repo\":{\"id\":21108956,\"name\":\"gorhill/uBlock\",\"url\":\"https://api.github.com/repos/gorhill/uBlock\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/gorhill/uBlock/issues/452\",\"labels_url\":\"https://api.github.com/repos/gorhill/uBlock/issues/452/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/gorhill/uBlock/issues/452/comments\",\"events_url\":\"https://api.github.com/repos/gorhill/uBlock/issues/452/events\",\"html_url\":\"https://github.com/gorhill/uBlock/issues/452\",\"id\":53199774,\"number\":452,\"title\":\"Github Issue\",\"user\":{\"login\":\"gpedro\",\"id\":2898638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2898638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gpedro\",\"html_url\":\"https://github.com/gpedro\",\"followers_url\":\"https://api.github.com/users/gpedro/followers\",\"following_url\":\"https://api.github.com/users/gpedro/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gpedro/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gpedro/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gpedro/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gpedro/orgs\",\"repos_url\":\"https://api.github.com/users/gpedro/repos\",\"events_url\":\"https://api.github.com/users/gpedro/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gpedro/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":7,\"created_at\":\"2014-12-31T19:34:09Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"closed_at\":null,\"body\":\"![gh problems](http://i.imgur.com/DjNwMiq.png)\\r\\n\\r\\nI've tested and is caused by uBlock \\r\\n\\r\\nI'm using Chrome 39.0.2171.95 (64-bit), OSX 10.9.5\"},\"comment\":{\"url\":\"https://api.github.com/repos/gorhill/uBlock/issues/comments/68477356\",\"html_url\":\"https://github.com/gorhill/uBlock/issues/452#issuecomment-68477356\",\"issue_url\":\"https://api.github.com/repos/gorhill/uBlock/issues/452\",\"id\":68477356,\"user\":{\"login\":\"gorhill\",\"id\":585534,\"avatar_url\":\"https://avatars.githubusercontent.com/u/585534?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gorhill\",\"html_url\":\"https://github.com/gorhill\",\"followers_url\":\"https://api.github.com/users/gorhill/followers\",\"following_url\":\"https://api.github.com/users/gorhill/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/gorhill/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/gorhill/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/gorhill/subscriptions\",\"organizations_url\":\"https://api.github.com/users/gorhill/orgs\",\"repos_url\":\"https://api.github.com/users/gorhill/repos\",\"events_url\":\"https://api.github.com/users/gorhill/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/gorhill/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:12Z\",\"updated_at\":\"2015-01-01T01:06:12Z\",\"body\":\"Ok I can reproduce on my very old laptop, I can investigate now.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:12Z\"}\n{\"id\":\"2489398486\",\"type\":\"PushEvent\",\"actor\":{\"id\":6693944,\"login\":\"Eternaldoom\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Eternaldoom\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6693944?\"},\"repo\":{\"id\":28652181,\"name\":\"BossLetsPlays/WaffleMod\",\"url\":\"https://api.github.com/repos/BossLetsPlays/WaffleMod\"},\"payload\":{\"push_id\":536753374,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"19f0e5ca1ba35f901a5934705fddaecad5f7a49d\",\"before\":\"7fd8147512d920a40a0e36752cff63ce832fd57d\",\"commits\":[{\"sha\":\"19f0e5ca1ba35f901a5934705fddaecad5f7a49d\",\"author\":{\"email\":\"ec291313f6147c66cfacc4177a9e43f07db94e06@icloud.com\",\"name\":\"Eternaldoom\"},\"message\":\"waffle texture\",\"distinct\":true,\"url\":\"https://api.github.com/repos/BossLetsPlays/WaffleMod/commits/19f0e5ca1ba35f901a5934705fddaecad5f7a49d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:12Z\"}\n{\"id\":\"2489398495\",\"type\":\"PushEvent\",\"actor\":{\"id\":9539839,\"login\":\"nathanpanchal\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathanpanchal\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9539839?\"},\"repo\":{\"id\":28327047,\"name\":\"nathanpanchal/rep\",\"url\":\"https://api.github.com/repos/nathanpanchal/rep\"},\"payload\":{\"push_id\":536753378,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"4cc6c3bf7c84c0e045b0c28d17fbb185ba167a7f\",\"before\":\"48660b086b03fd3fa0932a7028e59c60c195c183\",\"commits\":[{\"sha\":\"7a6f155b740a17645f8a7c3c0753e7babd61a18b\",\"author\":{\"email\":\"ea7a0c1525d0377387a689662b80f80209af162b@gmail.com\",\"name\":\"Nathan Panchal\"},\"message\":\"np - updated solutions and spelling\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nathanpanchal/rep/commits/7a6f155b740a17645f8a7c3c0753e7babd61a18b\"},{\"sha\":\"4cc6c3bf7c84c0e045b0c28d17fbb185ba167a7f\",\"author\":{\"email\":\"ea7a0c1525d0377387a689662b80f80209af162b@gmail.com\",\"name\":\"Nathan Panchal\"},\"message\":\"np - initial upload\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nathanpanchal/rep/commits/4cc6c3bf7c84c0e045b0c28d17fbb185ba167a7f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:14Z\"}\n{\"id\":\"2489398497\",\"type\":\"PushEvent\",\"actor\":{\"id\":2012806,\"login\":\"aconley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aconley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2012806?\"},\"repo\":{\"id\":18895528,\"name\":\"aconley/MachineLearning\",\"url\":\"https://api.github.com/repos/aconley/MachineLearning\"},\"payload\":{\"push_id\":536753379,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"4847438e4f4f82c3f466820f18ef92480b220ff7\",\"before\":\"ea0c13da5d0cfea3c77fa92d3489580516353378\",\"commits\":[{\"sha\":\"4847438e4f4f82c3f466820f18ef92480b220ff7\",\"author\":{\"email\":\"da6b13ae5bc1edd63fbaa421fd5ef5233ae76713@colorado.edu\",\"name\":\"Alexander Conley\"},\"message\":\"First 2 problems in BDA chap 2, added header sizes to all notes.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aconley/MachineLearning/commits/4847438e4f4f82c3f466820f18ef92480b220ff7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:14Z\"}\n{\"id\":\"2489398498\",\"type\":\"PushEvent\",\"actor\":{\"id\":826422,\"login\":\"devhd\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/devhd\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/826422?\"},\"repo\":{\"id\":28672269,\"name\":\"devhd/rulus\",\"url\":\"https://api.github.com/repos/devhd/rulus\"},\"payload\":{\"push_id\":536753370,\"size\":1000,\"distinct_size\":4394,\"ref\":\"refs/heads/corporate-commit\",\"head\":\"e4f8c39db38c3dc14d9f069b61226a731a173287\",\"before\":\"24a33a4632530cbe66971c91a3f73fed71285b45\",\"commits\":[{\"sha\":\"811baf9c467324f1a1f05ce6af51caeb4b7dd414\",\"author\":{\"email\":\"01142e84179254d983577d421939de6325fe05f7@users.noreply.github.com\",\"name\":\"devhd\"},\"message\":\"commit from Rulus company with internal id 793e827461e3\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/811baf9c467324f1a1f05ce6af51caeb4b7dd414\"},{\"sha\":\"c7f3b24506c5cf593063910449656fa9d6fc1ca5\",\"author\":{\"email\":\"01142e84179254d983577d421939de6325fe05f7@users.noreply.github.com\",\"name\":\"devhd\"},\"message\":\"commit from Rulus company with internal id eacaf6e3b4e1\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/c7f3b24506c5cf593063910449656fa9d6fc1ca5\"},{\"sha\":\"eaf754ef5e7290a91fb7954fcc0330c59b6bab0e\",\"author\":{\"email\":\"01142e84179254d983577d421939de6325fe05f7@users.noreply.github.com\",\"name\":\"devhd\"},\"message\":\"commit from Rulus company with internal id b3b085e89e1e\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/eaf754ef5e7290a91fb7954fcc0330c59b6bab0e\"},{\"sha\":\"052de2fa7f8150056a3c39d1d05737573f54d18c\",\"author\":{\"email\":\"99c668761d020b9679555123ba53755f7c2ea5a5@rulus.com\",\"name\":\"rulus-employee-57e138dc\"},\"message\":\"commit from Rulus company with internal id 57e138dcfafe\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/052de2fa7f8150056a3c39d1d05737573f54d18c\"},{\"sha\":\"ceb4f1d74e9cbf34e6f99127e9effa4976a5b4c1\",\"author\":{\"email\":\"24e8d9567bae12b60ea2fef0579698e2123b7bd1@rulus.com\",\"name\":\"rulus-employee-b06e52c4\"},\"message\":\"commit from Rulus company with internal id b06e52c4824d\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/ceb4f1d74e9cbf34e6f99127e9effa4976a5b4c1\"},{\"sha\":\"568d72b46c7f649f4fe8d1811e6c86ba9d0fc480\",\"author\":{\"email\":\"77d6b0ff6d4b9c1d3ecd8ab24e4f8a9753205217@rulus.com\",\"name\":\"rulus-employee-9df85673\"},\"message\":\"commit from Rulus company with internal id 9df856735928\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/568d72b46c7f649f4fe8d1811e6c86ba9d0fc480\"},{\"sha\":\"4f755c68cc64918713db129373ad9f65bb2c53ae\",\"author\":{\"email\":\"46a809052e8a001d8aaafdfd30a3674055ad8b31@rulus.com\",\"name\":\"rulus-employee-2631fc20\"},\"message\":\"commit from Rulus company with internal id 2631fc203a38\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/4f755c68cc64918713db129373ad9f65bb2c53ae\"},{\"sha\":\"e6621ee719c52fecafa1b238908bde9ecc8db8c7\",\"author\":{\"email\":\"f3967c3031d56cde0eb96f865167fad108cab328@users.noreply.github.com\",\"name\":\"andrejev\"},\"message\":\"commit from Rulus company with internal id 1ed5468637be\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/e6621ee719c52fecafa1b238908bde9ecc8db8c7\"},{\"sha\":\"4a2fcd7a274a99dbc8437c717556d3cf71042028\",\"author\":{\"email\":\"f3967c3031d56cde0eb96f865167fad108cab328@users.noreply.github.com\",\"name\":\"andrejev\"},\"message\":\"commit from Rulus company with internal id a08651cf2f5e\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/4a2fcd7a274a99dbc8437c717556d3cf71042028\"},{\"sha\":\"b28669116dc6f7e84824d2fb0f6133f15e2fdc2a\",\"author\":{\"email\":\"f3967c3031d56cde0eb96f865167fad108cab328@users.noreply.github.com\",\"name\":\"andrejev\"},\"message\":\"commit from Rulus company with internal id b0b95077dbb4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/b28669116dc6f7e84824d2fb0f6133f15e2fdc2a\"},{\"sha\":\"b2a01f646ae7c0a50d6fa259d33c63a547600b42\",\"author\":{\"email\":\"f3967c3031d56cde0eb96f865167fad108cab328@users.noreply.github.com\",\"name\":\"andrejev\"},\"message\":\"commit from Rulus company with internal id 62719714ae42\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/b2a01f646ae7c0a50d6fa259d33c63a547600b42\"},{\"sha\":\"ab19a3f469c89e4736784a0c38f5cd0697864a47\",\"author\":{\"email\":\"573d675efc8ed21e4a661e8897d95a5b47d57ea6@rulus.com\",\"name\":\"rulus-employee-65181a44\"},\"message\":\"commit from Rulus company with internal id 65181a441879\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/ab19a3f469c89e4736784a0c38f5cd0697864a47\"},{\"sha\":\"a3cb09199efb10ae4ed77dc1c7e80a1031cb7de1\",\"author\":{\"email\":\"aad584ac7ca8bed03f673f16c0cb6a2c032ed207@rulus.com\",\"name\":\"rulus-employee-05bfe041\"},\"message\":\"commit from Rulus company with internal id 05bfe0411e85\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/a3cb09199efb10ae4ed77dc1c7e80a1031cb7de1\"},{\"sha\":\"0b8277cd37d19e227542237aa4f776f6ca29efce\",\"author\":{\"email\":\"f3967c3031d56cde0eb96f865167fad108cab328@users.noreply.github.com\",\"name\":\"andrejev\"},\"message\":\"commit from Rulus company with internal id 92d839c3e65c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/0b8277cd37d19e227542237aa4f776f6ca29efce\"},{\"sha\":\"d509539f990f0ec2a2792e5d39ef66cef0180f6d\",\"author\":{\"email\":\"fe6f8e15c3e6fbb8a40a2ce6e98848853603f68a@users.noreply.github.com\",\"name\":\"ntklim\"},\"message\":\"commit from Rulus company with internal id f01b994a77bb\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/d509539f990f0ec2a2792e5d39ef66cef0180f6d\"},{\"sha\":\"46ca565baaf416978396bdfd68d22afb40c5ac93\",\"author\":{\"email\":\"fe6f8e15c3e6fbb8a40a2ce6e98848853603f68a@users.noreply.github.com\",\"name\":\"ntklim\"},\"message\":\"commit from Rulus company with internal id e6d067cb8ba9\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/46ca565baaf416978396bdfd68d22afb40c5ac93\"},{\"sha\":\"eecdefff74d967f8715d7933e8766c8338ab7992\",\"author\":{\"email\":\"8f3457924a1eb5af6fd611493be2bd7ad154a1fb@rulus.com\",\"name\":\"rulus-employee-d955c736\"},\"message\":\"commit from Rulus company with internal id d955c736e585\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/eecdefff74d967f8715d7933e8766c8338ab7992\"},{\"sha\":\"72c4c5f9d3477157fead5059599b6b3cf96a4b03\",\"author\":{\"email\":\"8ccf6737f7ac752e448572e9e5a7850e406ff3a9@rulus.com\",\"name\":\"rulus-employee-124d5850\"},\"message\":\"commit from Rulus company with internal id 124d58504f69\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/72c4c5f9d3477157fead5059599b6b3cf96a4b03\"},{\"sha\":\"fa78e9ad82b8af64f51df9c7350aac3b17d84c58\",\"author\":{\"email\":\"a25961e887882f8dec7b0ecf28bf20814085f77f@rulus.com\",\"name\":\"rulus-employee-f0be14e7\"},\"message\":\"commit from Rulus company with internal id f0be14e7607c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/fa78e9ad82b8af64f51df9c7350aac3b17d84c58\"},{\"sha\":\"c7c085cab4770b7ef6ac34da7537527558f86cf0\",\"author\":{\"email\":\"78dccec6bff894326f4b4d70c6de378e286bb775@rulus.com\",\"name\":\"rulus-employee-8e066744\"},\"message\":\"commit from Rulus company with internal id 8e0667447802\",\"distinct\":true,\"url\":\"https://api.github.com/repos/devhd/rulus/commits/c7c085cab4770b7ef6ac34da7537527558f86cf0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:15Z\"}\n{\"id\":\"2489398506\",\"type\":\"WatchEvent\",\"actor\":{\"id\":9970148,\"login\":\"bchoomnuan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bchoomnuan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9970148?\"},\"repo\":{\"id\":24993905,\"name\":\"socketplane/socketplane\",\"url\":\"https://api.github.com/repos/socketplane/socketplane\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:15Z\",\"org\":{\"id\":9063170,\"login\":\"socketplane\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/socketplane\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9063170?\"}}\n{\"id\":\"2489398510\",\"type\":\"PushEvent\",\"actor\":{\"id\":1687477,\"login\":\"kllmctrl\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kllmctrl\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1687477?\"},\"repo\":{\"id\":22713199,\"name\":\"kllmctrl/blog\",\"url\":\"https://api.github.com/repos/kllmctrl/blog\"},\"payload\":{\"push_id\":536753382,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"542fa01a5cb21e143fb434dbbe4962b0dad7f075\",\"before\":\"e530ef5a1f5d42a38cef0c14bbd2692a16ebdfd0\",\"commits\":[{\"sha\":\"542fa01a5cb21e143fb434dbbe4962b0dad7f075\",\"author\":{\"email\":\"f80e4117476d2980d2730856b4e57c1491ba28a9@gmail.com\",\"name\":\"KC\"},\"message\":\"Update 2014-12-31-me-summary2014.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kllmctrl/blog/commits/542fa01a5cb21e143fb434dbbe4962b0dad7f075\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:16Z\"}\n{\"id\":\"2489398519\",\"type\":\"WatchEvent\",\"actor\":{\"id\":117788,\"login\":\"nyarla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nyarla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/117788?\"},\"repo\":{\"id\":9267658,\"name\":\"kisielk/vigo\",\"url\":\"https://api.github.com/repos/kisielk/vigo\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:17Z\"}\n{\"id\":\"2489398527\",\"type\":\"PushEvent\",\"actor\":{\"id\":1308363,\"login\":\"paymonp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paymonp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1308363?\"},\"repo\":{\"id\":28678242,\"name\":\"paymonp/forecast_wrapper\",\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper\"},\"payload\":{\"push_id\":536753388,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9b9757b748dd98f06c2517a9af376e7515716a6a\",\"before\":\"a6f19abd10df22698f53704cd60bb245961a6365\",\"commits\":[{\"sha\":\"9b9757b748dd98f06c2517a9af376e7515716a6a\",\"author\":{\"email\":\"fc8885f78a23392efbab2637290a95a3fe38d9fa@team.curious.com\",\"name\":\"Paymon\"},\"message\":\"Create README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper/commits/9b9757b748dd98f06c2517a9af376e7515716a6a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398531\",\"type\":\"PushEvent\",\"actor\":{\"id\":9201970,\"login\":\"qdm\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qdm\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9201970?\"},\"repo\":{\"id\":25173910,\"name\":\"qdm/qdm.github.io\",\"url\":\"https://api.github.com/repos/qdm/qdm.github.io\"},\"payload\":{\"push_id\":536753389,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d4b8f56130659dbd132ba75208e1812a0233dadb\",\"before\":\"dbcd229108e512d575dcd1e4595183ce055f8e2f\",\"commits\":[{\"sha\":\"d4b8f56130659dbd132ba75208e1812a0233dadb\",\"author\":{\"email\":\"de163e90d3aeef9f404d1de71c48e234a211e3c3@gmail.com\",\"name\":\"KT\"},\"message\":\"Update\",\"distinct\":true,\"url\":\"https://api.github.com/repos/qdm/qdm.github.io/commits/d4b8f56130659dbd132ba75208e1812a0233dadb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398534\",\"type\":\"PushEvent\",\"actor\":{\"id\":5378975,\"login\":\"shk33\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/shk33\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5378975?\"},\"repo\":{\"id\":27845488,\"name\":\"shk33/RailsTuto\",\"url\":\"https://api.github.com/repos/shk33/RailsTuto\"},\"payload\":{\"push_id\":536753393,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/posts\",\"head\":\"bae94f8a9a33209c890f220f30632b4f5a6bf192\",\"before\":\"8741a4f19d72e935c30ecdd60fe4408d71b2bb27\",\"commits\":[{\"sha\":\"bae94f8a9a33209c890f220f30632b4f5a6bf192\",\"author\":{\"email\":\"5c080636567f0cafe2deca0ef026947600762c71@gmail.com\",\"name\":\"shk33\"},\"message\":\"Add image resize\",\"distinct\":true,\"url\":\"https://api.github.com/repos/shk33/RailsTuto/commits/bae94f8a9a33209c890f220f30632b4f5a6bf192\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398535\",\"type\":\"PushEvent\",\"actor\":{\"id\":4153853,\"login\":\"jlumijarvi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jlumijarvi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4153853?\"},\"repo\":{\"id\":28678176,\"name\":\"jlumijarvi/csv2xml\",\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml\"},\"payload\":{\"push_id\":536753392,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"191be1eb12b72eac931815f4d9dc022ec9d4ef22\",\"before\":\"44f7054077780dd70b8ad070b9b04640a65282ec\",\"commits\":[{\"sha\":\"191be1eb12b72eac931815f4d9dc022ec9d4ef22\",\"author\":{\"email\":\"40e6fc59d2535c98bfd5d19357e20df6e95cef64@gmail.com\",\"name\":\"jlumijarvi\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jlumijarvi/csv2xml/commits/191be1eb12b72eac931815f4d9dc022ec9d4ef22\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398537\",\"type\":\"DeleteEvent\",\"actor\":{\"id\":2812278,\"login\":\"beret595\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beret595\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2812278?\"},\"repo\":{\"id\":28470088,\"name\":\"beret595/Ucar_Operation_Crm_Finance\",\"url\":\"https://api.github.com/repos/beret595/Ucar_Operation_Crm_Finance\"},\"payload\":{\"ref\":\"Ocean\",\"ref_type\":\"branch\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398541\",\"type\":\"PushEvent\",\"actor\":{\"id\":7336721,\"login\":\"aow1980\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aow1980\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7336721?\"},\"repo\":{\"id\":27033479,\"name\":\"aow1980/android_system_core\",\"url\":\"https://api.github.com/repos/aow1980/android_system_core\"},\"payload\":{\"push_id\":536753397,\"size\":12,\"distinct_size\":12,\"ref\":\"refs/heads/lp5.0\",\"head\":\"e32edbcdc6851baf2e9fdb9586b19c9503436db2\",\"before\":\"ce1744ba2e2bb8731621c1cdc4aa0663a83eb972\",\"commits\":[{\"sha\":\"4b2626f0f43973b445a167370caf9fe9df834e7d\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"healthd: Reinitialize mChargerNames for every battery update\\n\\nBooting up the device without the usb sets the usb power supply\\ntype as UNKNOWN. Due to this mChargerNames gets incorrectly\\ninitialized at bootup. The value of usb power supply type changes\\nat run-time. So it makes sense to update mChargerNames everytime\\nwe have a battery related update.\\n\\nChange-Id: I2ec9f9a420ca61814d43c316b418ce94de3691bc\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/4b2626f0f43973b445a167370caf9fe9df834e7d\"},{\"sha\":\"78215b970e26959e1ad7f4129f84d8879e43ad35\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"audio: Add 24-bit offload format to audio_bytes_per_sample\\n\\nChange-Id: I238f302beadf8ba7705ffcb70639fb32be4ff2c4\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/78215b970e26959e1ad7f4129f84d8879e43ad35\"},{\"sha\":\"0f6f929147dad895bf830989d491beab7513e93a\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Healthd: charger: allow override of LED/BACKLIGHT paths\\n\\nChange-Id: Ib7372e441d554e16ffc20ce43993ee78cdc5b187\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/0f6f929147dad895bf830989d491beab7513e93a\"},{\"sha\":\"b0e263832d9eb57b2705f36200c170c9b7274e8d\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"healthd: Detect power supply type for all charger devices\\n\\nPower supply type is not determined via the device name,\\nhence iterate over all available devices in the subsystem\\nnode and read the type from device file.\\n\\n  shell@hammerhead:/ $ ls /sys/class/power_supply\\n  ac\\n  batt_therm\\n  battery\\n  touch\\n  usb\\n  wireless\\n\\n<4>[ 3184.867782] healthd: touch: Unknown power supply type\\n<4>[ 3184.868039] healthd: batt_therm: Unknown power supply type\\n<6>[ 3184.880506] healthd: battery l=89 v=4181 t=25.5 h=2 st=2 c=-288 chg=u\\n<4>[ 3184.890362] healthd: touch: Unknown power supply type\\n<4>[ 3184.890549] healthd: batt_therm: Unknown power supply type\\n<6>[ 3184.899419] healthd: battery l=89 v=4181 t=25.5 h=2 st=2 c=-59 chg=u\\n<4>[ 3184.908756] healthd: touch: Unknown power supply type\\n<4>[ 3184.908984] healthd: batt_therm: Unknown power supply type\\n<6>[ 3184.919672] healthd: battery l=89 v=4181 t=25.6 h=2 st=2 c=-25 chg=u\\n\\nChange-Id: I863bfab95193899460237b51997e0418eeb4ee2c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/b0e263832d9eb57b2705f36200c170c9b7274e8d\"},{\"sha\":\"e95c00ae6b23f31a356549e69cbc271db674ec29\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"adb: host: Provide better sideload status\\n\\n * Show data transfer in MB and in multiple of the file size.\\n\\n * Show a spinner to indicate liveness, which is updated at least once\\n   per second regardless of data transfer.\\n\\n * Do not allow sideload of zero sized files.\\n\\nChange-Id: I1bd0df6a8183fad5a502fc26a7e789c27d24f71a\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/e95c00ae6b23f31a356549e69cbc271db674ec29\"},{\"sha\":\"322799474a0e0492dbdfcbb029dc724b99d972f3\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"adb: Fix host build\\n\\nChange-Id: I173e920aa836bb0327bfae235022de011f3e2b99\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/322799474a0e0492dbdfcbb029dc724b99d972f3\"},{\"sha\":\"16057b3eb88144955a86e9591c5517af7447da53\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"set /system/etc/init.d/* permissions\\n\\nported from CM7\\n\\nChange-Id: I3422c392248673fd7a8c8b45f34678097e553b5b\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/16057b3eb88144955a86e9591c5517af7447da53\"},{\"sha\":\"fe4cc92ac754b3a3787f56563c1123a1086931c9\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"init: create symlinks to mtd block device nodes\\n\\nGiven a device in the form of \\\"/devices/virtual/mtd/mtd0/mtdblock0\\\":\\n* Creates symlinks for mtd block devices in\\n  /dev/block/mtd/mtd<partition number>\\n* Creates symlinks based on partition name in\\n  /dev/block/mtd/by-name/<partition name>\\n\\nChange-Id: Id8d61be88935a0ef83297d1d5e453d8ba0d586de\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/fe4cc92ac754b3a3787f56563c1123a1086931c9\"},{\"sha\":\"9f694a70b15ad0dfb84b176e542c95ce811af8c0\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"fastboot: Add missing fastboot USB IDs\\n\\nChange-Id: I1616e9e6a57652aa919f979833c0e19479343966\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/9f694a70b15ad0dfb84b176e542c95ce811af8c0\"},{\"sha\":\"3d9f73c94d78cc3a445d9040bf8b948ca5636f78\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"libcutils: Do not use the kernel headers\\n\\nNo idea why this is here, but it breaks arm64 builds. bionic's signal.h\\nmust be used on those\\n\\nChange-Id: Id4f07a9b13a32c563fda6e96e4a87b8b97d125fd\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/3d9f73c94d78cc3a445d9040bf8b948ca5636f78\"},{\"sha\":\"2269774631b7e39c832c79406b1aa587954ff294\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"libsysutils: Do not use the kernel headers\\n\\nNo idea why this is here, but it breaks arm64 builds. bionic's signal.h\\nmust be used on those\\n\\nChange-Id: I64f71565b179af38345fcf1496274ee5858abb05\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/2269774631b7e39c832c79406b1aa587954ff294\"},{\"sha\":\"e32edbcdc6851baf2e9fdb9586b19c9503436db2\",\"author\":{\"email\":\"457cfb837588e6491cbd81701614a2c9f7081d84@gmail.com\",\"name\":\"aow1980\"},\"message\":\"Forward-port mkbootimg / unpackbootimg support\\n\\n * Mostly from Koush and Seth Shelnutt\\n\\nunpackbootimg (squashed)\\n\\nunpackbootimg ported forward from eclair\\n\\nChange-Id: I74d2df0b47d40e7105cc58c2b05f8f383dc7f8a0\\n\\nport forward pagesize arg from eclair\\n\\nChange-Id: Ia789a4f392a3890aa0efa7efb42032482b48beb0\\n\\nunpackbootimg should output BOARD_PAGE_SIZE\\n\\nChange-Id: Ieb5cda01943a33da97eee4d025f56c2c6e7560e8\\n\\noutput page size in mkbootimg\\n\\nadd an option to override the pagesize for boot images created by nubs. ie, acer liquid mkliquidbootimg.\\n\\nChange-Id: Ie0c7e67edf5ae59019517e72b9be8c0b81388e41\\n\\nUpdate unpackbootimg to verify the Android boot image magic value. It will also search the first 512 bytes for padding.\\n\\nChange-Id: I490cba05f2bb616a3f64e3271ecaa61eb9e64be8\\n\\nunpackbootimg: Fix up the padding search in case the magic does not fall on a 8 byte boundary.\\n\\nChange-Id: I57471f9c2117cd7965b6958ea0aa88e356436da6\\n\\nunpackbootimg: Fix magic search.\\n\\nChange-Id: I68470b637556a08e48ff72b7ef8811cba13b04ad\\n\\nunpackbootimg: apparently mkbootimg no longer accepts hex values\\n\\nChange-Id: I95a33f7b40470e4500d418d863a65a75e7aa8499\\n\\nunpackbootimg: Need to also update the written file.\\n\\nChange-Id: I45faddbae85273c79b2837f97933634b6e70546f\\n\\nramdiskaddr is now ramdisk_offset\\n\\nChange-Id: I3bf83af5f7001f581506dc7fd9b1eb653334ad35\\n\\nunpackbootimg: remove host LOCAL_MODULE_TAG\\n\\nChange-Id: I199d680dc5ab8bf50f5be65c29095bf3adade695\\n\\nunpackbootimg: Add support for device trees\\n\\nChange-Id: I340eed99d2274a2f4cbaf5a9f27726ff3a9302e5\\n\\nunpackbootimg: Add support for detecting lz4 compressed ramdisks\\n\\nThis will check for lz4 magic, and if found change the extension of the\\nfile. Else it'll fall back to the default behavior of assuming gzip. This has\\nbeen tested with stock LS980 boot images and LS970 boot images for lz4\\nand gzip respectively.\\n\\nChange-Id: If2139ff172397b6db079ffb7ab9cb61897c38fb3\\n\\nunpackbootimg: Add support for dumping ramdisk offset\\n\\nChange-Id: Ic62b9fe61db4435ecbc52b66db5ffc9b9d79cbb4\\n\\nunpackbootimg: Add support for second, second_offset and tags_offset.\\n\\nChange-Id: Ia7ef7f00191dbf2c44736c4e4d980f72afa8c253\",\"distinct\":true,\"url\":\"https://api.github.com/repos/aow1980/android_system_core/commits/e32edbcdc6851baf2e9fdb9586b19c9503436db2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398543\",\"type\":\"PushEvent\",\"actor\":{\"id\":10263666,\"login\":\"katiekroik\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/katiekroik\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10263666?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"push_id\":536753398,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"53bc894d93b396c5cc3c224311421a017ceac40e\",\"before\":\"bd278f438aba4582c30eb8d42e7b52740b739e5b\",\"commits\":[{\"sha\":\"53bc894d93b396c5cc3c224311421a017ceac40e\",\"author\":{\"email\":\"fdb375617daf85f650fdf56bce778da925caee49@nyu.edu\",\"name\":\"katiekroik\"},\"message\":\"Updated README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite/commits/53bc894d93b396c5cc3c224311421a017ceac40e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398544\",\"type\":\"GollumEvent\",\"actor\":{\"id\":4620127,\"login\":\"husathap\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/husathap\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4620127?\"},\"repo\":{\"id\":28579680,\"name\":\"husathap/Inuvik\",\"url\":\"https://api.github.com/repos/husathap/Inuvik\"},\"payload\":{\"pages\":[{\"page_name\":\"Tutorial-#1:-Creating-a-New-Room\",\"title\":\"Tutorial #1: Creating a New Room\",\"summary\":null,\"action\":\"edited\",\"sha\":\"dc5f3f3a5cd37352810747bce11e48dc9c0541ee\",\"html_url\":\"https://github.com/husathap/Inuvik/wiki/Tutorial-%231%3A-Creating-a-New-Room\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398545\",\"type\":\"PushEvent\",\"actor\":{\"id\":1745861,\"login\":\"topaztee\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/topaztee\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1745861?\"},\"repo\":{\"id\":28677407,\"name\":\"topaztee/topaztee.github.io\",\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io\"},\"payload\":{\"push_id\":536753399,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"3f5e524f47baf10eafc7deecb718ff84dab8d25a\",\"before\":\"70a7d7ee278b87fbcbe0c785d3ec3c1dcb12a8d4\",\"commits\":[{\"sha\":\"3f5e524f47baf10eafc7deecb718ff84dab8d25a\",\"author\":{\"email\":\"f74c82d708bb42a372674042ebc8a1411fbc9344@192-168-1-2.tpgi.com.au\",\"name\":\"topaztur@gmail.com\"},\"message\":\"Blog update at 2015-01-01 01:06:09\",\"distinct\":true,\"url\":\"https://api.github.com/repos/topaztee/topaztee.github.io/commits/3f5e524f47baf10eafc7deecb718ff84dab8d25a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:18Z\"}\n{\"id\":\"2489398553\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1391558,\"login\":\"UndefinedOffset\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/UndefinedOffset\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1391558?\"},\"repo\":{\"id\":1030566,\"name\":\"furf/jquery-ui-touch-punch\",\"url\":\"https://api.github.com/repos/furf/jquery-ui-touch-punch\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:22Z\"}\n{\"id\":\"2489398558\",\"type\":\"WatchEvent\",\"actor\":{\"id\":117788,\"login\":\"nyarla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nyarla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/117788?\"},\"repo\":{\"id\":28428729,\"name\":\"wasabeef/awesome-android-ui\",\"url\":\"https://api.github.com/repos/wasabeef/awesome-android-ui\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:23Z\"}\n{\"id\":\"2489398559\",\"type\":\"PushEvent\",\"actor\":{\"id\":9038488,\"login\":\"martindevnow\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/martindevnow\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9038488?\"},\"repo\":{\"id\":28460218,\"name\":\"martindevnow/larabook\",\"url\":\"https://api.github.com/repos/martindevnow/larabook\"},\"payload\":{\"push_id\":536753400,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"247a3c2ba8b1d067c146dfa23587614f564b2c2f\",\"before\":\"b4d8b51987660e73ac233d8d34fa8c25001af5a5\",\"commits\":[{\"sha\":\"247a3c2ba8b1d067c146dfa23587614f564b2c2f\",\"author\":{\"email\":\"e4fb5da29019c84509153dcdcc3d610b56ae63a4@gmail.com\",\"name\":\"Ben Martin (Cooler-PC)\"},\"message\":\"Products\",\"distinct\":true,\"url\":\"https://api.github.com/repos/martindevnow/larabook/commits/247a3c2ba8b1d067c146dfa23587614f564b2c2f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:23Z\"}\n{\"id\":\"2489398561\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753401,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"274cca1b2adc063b47d558ac9bafa0dce7a05abb\",\"before\":\"6630db87e4ed496f67a88bf24f22332c67e9792d\",\"commits\":[{\"sha\":\"274cca1b2adc063b47d558ac9bafa0dce7a05abb\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"null check expression.meta\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/274cca1b2adc063b47d558ac9bafa0dce7a05abb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:23Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398566\",\"type\":\"PushEvent\",\"actor\":{\"id\":5606771,\"login\":\"rchoetzlein\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rchoetzlein\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5606771?\"},\"repo\":{\"id\":28014200,\"name\":\"rchoetzlein/luna\",\"url\":\"https://api.github.com/repos/rchoetzlein/luna\"},\"payload\":{\"push_id\":536753407,\"size\":2,\"distinct_size\":1,\"ref\":\"refs/heads/release\",\"head\":\"0406bbe059f3d0121cff67e5e2afeb2510b97dfc\",\"before\":\"ea9868cd5999171ed66ccca64f33db74c4901104\",\"commits\":[{\"sha\":\"6ae249b814f81055c160ea612e3817a582a819f0\",\"author\":{\"email\":\"8b97780f57e4074b50b20f54327dcaeef1063a29@gmail.com\",\"name\":\"Rama Hoetzlein\"},\"message\":\"testing\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rchoetzlein/luna/commits/6ae249b814f81055c160ea612e3817a582a819f0\"},{\"sha\":\"0406bbe059f3d0121cff67e5e2afeb2510b97dfc\",\"author\":{\"email\":\"8b97780f57e4074b50b20f54327dcaeef1063a29@gmail.com\",\"name\":\"Rama Hoetzlein\"},\"message\":\"Merge branch 'rama' into release\",\"distinct\":true,\"url\":\"https://api.github.com/repos/rchoetzlein/luna/commits/0406bbe059f3d0121cff67e5e2afeb2510b97dfc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:23Z\"}\n{\"id\":\"2489398569\",\"type\":\"PushEvent\",\"actor\":{\"id\":4379694,\"login\":\"moongato\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/moongato\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4379694?\"},\"repo\":{\"id\":11769101,\"name\":\"moongato/android_packages_apps_settings\",\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings\"},\"payload\":{\"push_id\":536753409,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/lp50x-test\",\"head\":\"5b4781c9b6ff5d41e22f14aaac8d384ed85d0c19\",\"before\":\"56646a94c0aedc577d5db224058fb9be4c927786\",\"commits\":[{\"sha\":\"e53c63b3254c37b69e9fcf05b69a1b2fb7f1858c\",\"author\":{\"email\":\"0fdcc473619717889ce3a7389c2487333487bab5@gmail.com\",\"name\":\"rascarlo\"},\"message\":\"volume rocker music controls\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/e53c63b3254c37b69e9fcf05b69a1b2fb7f1858c\"},{\"sha\":\"bebb75fdaa4a085c814ea6d2551899593454440d\",\"author\":{\"email\":\"0fdcc473619717889ce3a7389c2487333487bab5@gmail.com\",\"name\":\"rascarlo\"},\"message\":\"status bar brightness control edits.\\n\\nremove check for SCREEN_BRIGHTNESS_MODE_AUTOMATIC\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/bebb75fdaa4a085c814ea6d2551899593454440d\"},{\"sha\":\"5b4781c9b6ff5d41e22f14aaac8d384ed85d0c19\",\"author\":{\"email\":\"22e0f38e0fc64da9129ff9b9ef030b39415294a1@ubuntu\",\"name\":\"moongato\"},\"message\":\"Merge remote-tracking branch 'upstream/lollipop-ras-mr1' into lp50x-test\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/5b4781c9b6ff5d41e22f14aaac8d384ed85d0c19\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:23Z\"}\n{\"id\":\"2489398571\",\"type\":\"PushEvent\",\"actor\":{\"id\":10355660,\"login\":\"nomadturk\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nomadturk\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10355660?\"},\"repo\":{\"id\":28657737,\"name\":\"nomadturk/nginx-conf\",\"url\":\"https://api.github.com/repos/nomadturk/nginx-conf\"},\"payload\":{\"push_id\":536753411,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f560a8bd7c3d688fd6b99a5976e3518fbbfd68a3\",\"before\":\"5c897fb95e40e2b52b31c71de5eaed81d9989c6d\",\"commits\":[{\"sha\":\"f560a8bd7c3d688fd6b99a5976e3518fbbfd68a3\",\"author\":{\"email\":\"90283840d90de49b8e7984bd99b47fee0d4bd50d@cokh.net\",\"name\":\"nomadturk\"},\"message\":\"Update pagespeed.add\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nomadturk/nginx-conf/commits/f560a8bd7c3d688fd6b99a5976e3518fbbfd68a3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\"}\n{\"id\":\"2489398573\",\"type\":\"PushEvent\",\"actor\":{\"id\":5674054,\"login\":\"winsphinx\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/winsphinx\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5674054?\"},\"repo\":{\"id\":24920694,\"name\":\"winsphinx/MyDesktop\",\"url\":\"https://api.github.com/repos/winsphinx/MyDesktop\"},\"payload\":{\"push_id\":536753412,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e9598c0de93d13ac89e00771bbdd1ac0cadaa6c4\",\"before\":\"a76c17c45cc8caf929cfeb0ccd1ebf88d5eb9773\",\"commits\":[{\"sha\":\"e9598c0de93d13ac89e00771bbdd1ac0cadaa6c4\",\"author\":{\"email\":\"fcb6a490c5f10779ac384c021b9111357615f586@gmail.com\",\"name\":\"winsphinx\"},\"message\":\"add notepad2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/winsphinx/MyDesktop/commits/e9598c0de93d13ac89e00771bbdd1ac0cadaa6c4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\"}\n{\"id\":\"2489398574\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397298\",\"id\":22397298,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\\n \\n - name: custom horizon splash logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png mode=0644 force=yes\\n \\n - name: custom horizon favicon\\n-  get_url: url={{ horizon.favicon_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico\\n-           force=yes\\n+  get_url: url={{ horizon.favicon_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico force=yes\\n \\n - name: put images and fonts where apache can find them\\n-  file: src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n-        dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n-        state=link\\n-        owner=www-data\\n-        group=www-data\\n+  file: |\\n+    src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n+    dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n+    state=link\\n+    owner=www-data\\n+    group=www-data\\n   with_items:\\n     - img\\n     - fonts\\n \\n-# flush before ensuring apache running, saves immediate restart\\n-- meta: flush_handlers\\n-\\n-- name: ensure apache started\\n+- name: ensure apache is running\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":157,\"original_position\":157,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Now you have a start followed by an immediate reload. Will apache be okay with that?\",\"created_at\":\"2015-01-01T01:06:24Z\",\"updated_at\":\"2015-01-01T01:06:24Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397298\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397298\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397298\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:06:24Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489398577\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":993322,\"login\":\"qiangxue\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qiangxue\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/993322?\"},\"repo\":{\"id\":3431193,\"name\":\"yiisoft/yii2\",\"url\":\"https://api.github.com/repos/yiisoft/yii2\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/yiisoft/yii2/issues/6708\",\"labels_url\":\"https://api.github.com/repos/yiisoft/yii2/issues/6708/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/yiisoft/yii2/issues/6708/comments\",\"events_url\":\"https://api.github.com/repos/yiisoft/yii2/issues/6708/events\",\"html_url\":\"https://github.com/yiisoft/yii2/issues/6708\",\"id\":53187446,\"number\":6708,\"title\":\"PHP Notice when working with related records\",\"user\":{\"login\":\"SDKiller\",\"id\":2150916,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2150916?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/SDKiller\",\"html_url\":\"https://github.com/SDKiller\",\"followers_url\":\"https://api.github.com/users/SDKiller/followers\",\"following_url\":\"https://api.github.com/users/SDKiller/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/SDKiller/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/SDKiller/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/SDKiller/subscriptions\",\"organizations_url\":\"https://api.github.com/users/SDKiller/orgs\",\"repos_url\":\"https://api.github.com/users/SDKiller/repos\",\"events_url\":\"https://api.github.com/users/SDKiller/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/SDKiller/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/yiisoft/yii2/labels/status%3Aneed+more+info\",\"name\":\"status:need more info\",\"color\":\"d4c5f9\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":4,\"created_at\":\"2014-12-31T15:04:22Z\",\"updated_at\":\"2015-01-01T01:06:23Z\",\"closed_at\":null,\"body\":\"```\\r\\nPHP Notice – yii\\\\base\\\\ErrorException\\r\\nIndirect modification of overloaded property common\\\\models\\\\catalog\\\\Categories::$subCategories has no effect\\r\\n```\\r\\n\\r\\nBut if I supress error reporting - everything works fine, related records are processed as expected.\\r\\nAm I doing something wrong or it is a drawback of relations `magic`?\\r\\n\\r\\n\\r\\nin controller:\\r\\n```\\r\\n    ....\\r\\n        $category = Categories::find()\\r\\n            ->where($condition)\\r\\n            ->with('subCategories')\\r\\n            ->one();\\r\\n\\r\\n        $category->buildPaths();\\r\\n    ...\\r\\n```\\r\\n\\r\\n\\r\\nin `Categories` model\\r\\n```\\r\\n...\\r\\n    /**\\r\\n     * Relation with \\\\common\\\\models\\\\catalog\\\\Categories to retrieve sub-categories\\r\\n     */\\r\\n    public function getSubCategories()\\r\\n    {\\r\\n        return $this->hasMany(Categories::className(), ['parent' => '_id'])\\r\\n            ->where(['published' => 1])\\r\\n            ->orderBy(['ordering' => SORT_ASC, 'name' => SORT_ASC]);\\r\\n    }\\r\\n\\r\\n    ...\\r\\n\\r\\n    public function buildPaths()\\r\\n    {\\r\\n        if (!empty($this->subCategories)) {\\r\\n            foreach ($this->subCategories as &$subCategory) {\\r\\n                $subCategory->setPath($this->path . '/' . $subCategory->alias);\\r\\n            }\\r\\n        }\\r\\n    }\\r\\n...\\r\\n```\"},\"comment\":{\"url\":\"https://api.github.com/repos/yiisoft/yii2/issues/comments/68477358\",\"html_url\":\"https://github.com/yiisoft/yii2/issues/6708#issuecomment-68477358\",\"issue_url\":\"https://api.github.com/repos/yiisoft/yii2/issues/6708\",\"id\":68477358,\"user\":{\"login\":\"qiangxue\",\"id\":993322,\"avatar_url\":\"https://avatars.githubusercontent.com/u/993322?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/qiangxue\",\"html_url\":\"https://github.com/qiangxue\",\"followers_url\":\"https://api.github.com/users/qiangxue/followers\",\"following_url\":\"https://api.github.com/users/qiangxue/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/qiangxue/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/qiangxue/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/qiangxue/subscriptions\",\"organizations_url\":\"https://api.github.com/users/qiangxue/orgs\",\"repos_url\":\"https://api.github.com/users/qiangxue/repos\",\"events_url\":\"https://api.github.com/users/qiangxue/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/qiangxue/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:23Z\",\"updated_at\":\"2015-01-01T01:06:23Z\",\"body\":\"Why do you use `&$subCategory`?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\",\"org\":{\"id\":993323,\"login\":\"yiisoft\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/yiisoft\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/993323?\"}}\n{\"id\":\"2489398578\",\"type\":\"PushEvent\",\"actor\":{\"id\":1684950,\"login\":\"naijaping\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/naijaping\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1684950?\"},\"repo\":{\"id\":28650038,\"name\":\"naijaping/awonlist\",\"url\":\"https://api.github.com/repos/naijaping/awonlist\"},\"payload\":{\"push_id\":536753415,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7f482e40c0e163ac46ddcfc1c206c6a851e0b62b\",\"before\":\"f454a5a195625a9128fddb04b192fcc457e82091\",\"commits\":[{\"sha\":\"7f482e40c0e163ac46ddcfc1c206c6a851e0b62b\",\"author\":{\"email\":\"8a1440b218d23a283d388025f7c9dc3555009ec5@gmail.com\",\"name\":\"naijaping\"},\"message\":\"Update uk\",\"distinct\":true,\"url\":\"https://api.github.com/repos/naijaping/awonlist/commits/7f482e40c0e163ac46ddcfc1c206c6a851e0b62b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\"}\n{\"id\":\"2489398579\",\"type\":\"CreateEvent\",\"actor\":{\"id\":3960243,\"login\":\"wp-plugins-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wp-plugins-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3960243?\"},\"repo\":{\"id\":22870574,\"name\":\"wp-plugins/chamber-dashboard-business-directory\",\"url\":\"https://api.github.com/repos/wp-plugins/chamber-dashboard-business-directory\"},\"payload\":{\"ref\":\"1.9\",\"ref_type\":\"tag\",\"master_branch\":\"master\",\"description\":\"WordPress.org Plugin Mirror\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\",\"org\":{\"id\":2996849,\"login\":\"wp-plugins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wp-plugins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2996849?\"}}\n{\"id\":\"2489398580\",\"type\":\"PushEvent\",\"actor\":{\"id\":739622,\"login\":\"treydock\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/treydock\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/739622?\"},\"repo\":{\"id\":23934080,\"name\":\"idhmc-tamu/emop-dashboard\",\"url\":\"https://api.github.com/repos/idhmc-tamu/emop-dashboard\"},\"payload\":{\"push_id\":536753416,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f26c3459c2f4116b0a267740239059c68bdf9995\",\"before\":\"668e7ea0bdfc797da70795b9b9ae61bead37b3fb\",\"commits\":[{\"sha\":\"f26c3459c2f4116b0a267740239059c68bdf9995\",\"author\":{\"email\":\"ea4a4ed01189a93e0c88f9f6e05a922974b21422@tamu.edu\",\"name\":\"Trey Dockendorf\"},\"message\":\"Update FontsController\\n* Use ActiveRecord methods for performing queries to database rather than raw SQL\\n* Streamline much of the code to be simpler\\n* Correct indentation issue to be 2-spaces to indent\\n* Add unit tests\\n\\nModel updates\\n* Associate PrintFront and Work models\\n* Validate that a PrintFront instance's name value (pf_name) is unique\\n* Add instance method to Font model that generates the traineddata path rather than creating the path in controllers\",\"distinct\":true,\"url\":\"https://api.github.com/repos/idhmc-tamu/emop-dashboard/commits/f26c3459c2f4116b0a267740239059c68bdf9995\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\",\"org\":{\"id\":4932222,\"login\":\"idhmc-tamu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/idhmc-tamu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4932222?\"}}\n{\"id\":\"2489398582\",\"type\":\"PushEvent\",\"actor\":{\"id\":3960243,\"login\":\"wp-plugins-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wp-plugins-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3960243?\"},\"repo\":{\"id\":22870574,\"name\":\"wp-plugins/chamber-dashboard-business-directory\",\"url\":\"https://api.github.com/repos/wp-plugins/chamber-dashboard-business-directory\"},\"payload\":{\"push_id\":536753417,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"52cd22ebee5fa319aa244d232c376aec6865ba16\",\"before\":\"17c18938eaf97feb2c3f55eb276b6c1145e86969\",\"commits\":[{\"sha\":\"52cd22ebee5fa319aa244d232c376aec6865ba16\",\"author\":{\"email\":\"ce123eae784bf8ff0a0dbf80492a59e7c0a2bfbf@b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"name\":\"Gwendydd\"},\"message\":\"added social media fields\\n\\ngit-svn-id: https://plugins.svn.wordpress.org/chamber-dashboard-business-directory/trunk@1057716 b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wp-plugins/chamber-dashboard-business-directory/commits/52cd22ebee5fa319aa244d232c376aec6865ba16\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\",\"org\":{\"id\":2996849,\"login\":\"wp-plugins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wp-plugins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2996849?\"}}\n{\"id\":\"2489398584\",\"type\":\"PushEvent\",\"actor\":{\"id\":2453862,\"login\":\"schloo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/schloo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2453862?\"},\"repo\":{\"id\":28160579,\"name\":\"azilnik/phetch\",\"url\":\"https://api.github.com/repos/azilnik/phetch\"},\"payload\":{\"push_id\":536753420,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"672e3213d883f8d100f0e2d38b1c739ecc39acf5\",\"before\":\"1b32d9c513558f8cc57ed88edff23b0d45610363\",\"commits\":[{\"sha\":\"672e3213d883f8d100f0e2d38b1c739ecc39acf5\",\"author\":{\"email\":\"03cd939e5e01f81bd3cbeb1977c82e3d0109cf43@Michelles-Air.home\",\"name\":\"schloo\"},\"message\":\"oops moar\",\"distinct\":true,\"url\":\"https://api.github.com/repos/azilnik/phetch/commits/672e3213d883f8d100f0e2d38b1c739ecc39acf5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:24Z\"}\n{\"id\":\"2489398586\",\"type\":\"PushEvent\",\"actor\":{\"id\":1445198,\"login\":\"FleurDeLuce\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FleurDeLuce\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1445198?\"},\"repo\":{\"id\":18230007,\"name\":\"FleurDeLuce/Leetcode\",\"url\":\"https://api.github.com/repos/FleurDeLuce/Leetcode\"},\"payload\":{\"push_id\":536753422,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"153fe4cc6aeb387647e30f5c67b7572763439d0c\",\"before\":\"d0dbdf0f946b052673cbe4ea6f9c385788e155c7\",\"commits\":[{\"sha\":\"153fe4cc6aeb387647e30f5c67b7572763439d0c\",\"author\":{\"email\":\"b8ad7a82476ffa543a8614f8b96f97e16135a3b6@gmail.com\",\"name\":\"FleurDeLuce\"},\"message\":\"factorial trailing zeroes solution added\",\"distinct\":true,\"url\":\"https://api.github.com/repos/FleurDeLuce/Leetcode/commits/153fe4cc6aeb387647e30f5c67b7572763439d0c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:25Z\"}\n{\"id\":\"2489398592\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":4192725,\"login\":\"yangarbiter\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yangarbiter\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4192725?\"},\"repo\":{\"id\":13980094,\"name\":\"botonchou/libdnn\",\"url\":\"https://api.github.com/repos/botonchou/libdnn\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/botonchou/libdnn/issues/8\",\"labels_url\":\"https://api.github.com/repos/botonchou/libdnn/issues/8/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/botonchou/libdnn/issues/8/comments\",\"events_url\":\"https://api.github.com/repos/botonchou/libdnn/issues/8/events\",\"html_url\":\"https://github.com/botonchou/libdnn/issues/8\",\"id\":53170772,\"number\":8,\"title\":\"How to lower the total needed shared memory?\",\"user\":{\"login\":\"yangarbiter\",\"id\":4192725,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4192725?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yangarbiter\",\"html_url\":\"https://github.com/yangarbiter\",\"followers_url\":\"https://api.github.com/users/yangarbiter/followers\",\"following_url\":\"https://api.github.com/users/yangarbiter/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/yangarbiter/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/yangarbiter/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/yangarbiter/subscriptions\",\"organizations_url\":\"https://api.github.com/users/yangarbiter/orgs\",\"repos_url\":\"https://api.github.com/users/yangarbiter/repos\",\"events_url\":\"https://api.github.com/users/yangarbiter/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/yangarbiter/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/botonchou/libdnn/labels/question\",\"name\":\"question\",\"color\":\"cc317c\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":4,\"created_at\":\"2014-12-31T06:59:09Z\",\"updated_at\":\"2015-01-01T01:06:26Z\",\"closed_at\":null,\"body\":\"Hi Chou,\\r\\n\\r\\nI am encountering the following problem when trying to run convolutional neural network.\\r\\n```\\r\\nterminate called after throwing an instance of 'std::runtime_error'\\r\\n  what():  [Error] In function \\\"getSuitableShmConfig\\\" (at src/cnn-utility.cu:418): Exceeds maximum shared memory available. (49152 bytes)\\r\\nkernel = (100, 92), grids = (8, 8, 40), threads = (4, 4, 1)  => 75940 bytes of shared memory needed.\\r\\n```\\r\\nWhat is the standard way to fix this problem if I really don't have that much memory? (ex. lowering thread count)\\r\\n\\r\\nThanks in advance.\"},\"comment\":{\"url\":\"https://api.github.com/repos/botonchou/libdnn/issues/comments/68477359\",\"html_url\":\"https://github.com/botonchou/libdnn/issues/8#issuecomment-68477359\",\"issue_url\":\"https://api.github.com/repos/botonchou/libdnn/issues/8\",\"id\":68477359,\"user\":{\"login\":\"yangarbiter\",\"id\":4192725,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4192725?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yangarbiter\",\"html_url\":\"https://github.com/yangarbiter\",\"followers_url\":\"https://api.github.com/users/yangarbiter/followers\",\"following_url\":\"https://api.github.com/users/yangarbiter/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/yangarbiter/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/yangarbiter/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/yangarbiter/subscriptions\",\"organizations_url\":\"https://api.github.com/users/yangarbiter/orgs\",\"repos_url\":\"https://api.github.com/users/yangarbiter/repos\",\"events_url\":\"https://api.github.com/users/yangarbiter/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/yangarbiter/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:26Z\",\"updated_at\":\"2015-01-01T01:06:26Z\",\"body\":\"Actually the kernel size 23*14 is able to be run on my computer using theano. I am still wondering about what Caffe have done to get the 7% error rate. I can only achieve around 15% error rate with Lenet implemented by theano.\\r\\n\\r\\nBTW, I found out the the nn-init would crash without any error message when the format of input training data is wrong. (I accidentally start my array from zero) \\r\\n\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:27Z\"}\n{\"id\":\"2489398593\",\"type\":\"PushEvent\",\"actor\":{\"id\":1779595,\"login\":\"dcbaker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcbaker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1779595?\"},\"repo\":{\"id\":8488437,\"name\":\"dcbaker/piglit\",\"url\":\"https://api.github.com/repos/dcbaker/piglit\"},\"payload\":{\"push_id\":536753426,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/wip/command-list-only-v5\",\"head\":\"4664b69bd21c2dc875eb7ac03be904a6eb371d9c\",\"before\":\"cf5cf7224385c3c3d086762b5bc1d8f7df031ae7\",\"commits\":[{\"sha\":\"4664b69bd21c2dc875eb7ac03be904a6eb371d9c\",\"author\":{\"email\":\"c26a678a04c601e0311b0d6006e67eee6ed19a8e@intel.com\",\"name\":\"Dylan Baker\"},\"message\":\"fixup! fixup! all.py: Replace some string concatenation with str.format()\",\"distinct\":true,\"url\":\"https://api.github.com/repos/dcbaker/piglit/commits/4664b69bd21c2dc875eb7ac03be904a6eb371d9c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:27Z\"}\n{\"id\":\"2489398596\",\"type\":\"PushEvent\",\"actor\":{\"id\":3960243,\"login\":\"wp-plugins-user\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wp-plugins-user\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3960243?\"},\"repo\":{\"id\":27287395,\"name\":\"wp-plugins/spam-hammer\",\"url\":\"https://api.github.com/repos/wp-plugins/spam-hammer\"},\"payload\":{\"push_id\":536753429,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5c71935c5b32807aedcc9533e4e9105b7bf38947\",\"before\":\"df54c7e1ab24d713b54f4c3660c186982d6cae95\",\"commits\":[{\"sha\":\"5c71935c5b32807aedcc9533e4e9105b7bf38947\",\"author\":{\"email\":\"8177b3ca125b2874291549da03b6ae3b620b0c28@b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"name\":\"wpspamhammer\"},\"message\":\"Readme.txt Update.  Total Spam Attacks: 15,150,841\\n\\ngit-svn-id: https://plugins.svn.wordpress.org/spam-hammer/trunk@1057715 b8457f37-d9ea-0310-8a92-e5e31aec5664\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wp-plugins/spam-hammer/commits/5c71935c5b32807aedcc9533e4e9105b7bf38947\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:27Z\",\"org\":{\"id\":2996849,\"login\":\"wp-plugins\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/wp-plugins\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2996849?\"}}\n{\"id\":\"2489398600\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":23202,\"login\":\"aduros\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aduros\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/23202?\"},\"repo\":{\"id\":10282042,\"name\":\"HaxeFoundation/haxe\",\"url\":\"https://api.github.com/repos/HaxeFoundation/haxe\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/HaxeFoundation/haxe/issues/3730\",\"labels_url\":\"https://api.github.com/repos/HaxeFoundation/haxe/issues/3730/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/HaxeFoundation/haxe/issues/3730/comments\",\"events_url\":\"https://api.github.com/repos/HaxeFoundation/haxe/issues/3730/events\",\"html_url\":\"https://github.com/HaxeFoundation/haxe/issues/3730\",\"id\":53210295,\"number\":3730,\"title\":\"#file support\",\"user\":{\"login\":\"aduros\",\"id\":23202,\"avatar_url\":\"https://avatars.githubusercontent.com/u/23202?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/aduros\",\"html_url\":\"https://github.com/aduros\",\"followers_url\":\"https://api.github.com/users/aduros/followers\",\"following_url\":\"https://api.github.com/users/aduros/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/aduros/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/aduros/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/aduros/subscriptions\",\"organizations_url\":\"https://api.github.com/users/aduros/orgs\",\"repos_url\":\"https://api.github.com/users/aduros/repos\",\"events_url\":\"https://api.github.com/users/aduros/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/aduros/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:06:27Z\",\"updated_at\":\"2015-01-01T01:06:27Z\",\"closed_at\":null,\"body\":\"#line is pretty useful for compilers that translate to Haxe. Would it be possible to also support something like #file to pass the source filename?\\r\\n\\r\\nWould be helpful for a project I'm working on, but if it's not an easy change no worries :)\\r\\n\\r\\nMight also be useful to @tardisgo?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:28Z\",\"org\":{\"id\":3826779,\"login\":\"HaxeFoundation\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/HaxeFoundation\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3826779?\"}}\n{\"id\":\"2489398601\",\"type\":\"PushEvent\",\"actor\":{\"id\":4170616,\"login\":\"cmsbuild\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cmsbuild\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4170616?\"},\"repo\":{\"id\":20482496,\"name\":\"cms-sw/cms-sw.github.io\",\"url\":\"https://api.github.com/repos/cms-sw/cms-sw.github.io\"},\"payload\":{\"push_id\":536753430,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"714d9b5ebc2cb4242ac08affaf1a0cbce8d02218\",\"before\":\"e473bfee29d1543bd1dba301d6c1a3ef60dc1248\",\"commits\":[{\"sha\":\"714d9b5ebc2cb4242ac08affaf1a0cbce8d02218\",\"author\":{\"email\":\"5a25e995b53ac7cd7cc4edf867580fb423e22fde@cern.ch\",\"name\":\"CMS Build\"},\"message\":\"New report generated 20150101-0206\",\"distinct\":true,\"url\":\"https://api.github.com/repos/cms-sw/cms-sw.github.io/commits/714d9b5ebc2cb4242ac08affaf1a0cbce8d02218\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:28Z\",\"org\":{\"id\":3863500,\"login\":\"cms-sw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cms-sw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3863500?\"}}\n{\"id\":\"2489398602\",\"type\":\"PushEvent\",\"actor\":{\"id\":106511,\"login\":\"andrewrk\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/andrewrk\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/106511?\"},\"repo\":{\"id\":28373474,\"name\":\"thejoshwolfe/nethack\",\"url\":\"https://api.github.com/repos/thejoshwolfe/nethack\"},\"payload\":{\"push_id\":536753431,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/it-is-time\",\"head\":\"3231a8162f90ce411af8c23d67cc04584f0ed7fa\",\"before\":\"b0821d5e5d69c4b72114e3426b6c2b34be07356a\",\"commits\":[{\"sha\":\"3231a8162f90ce411af8c23d67cc04584f0ed7fa\",\"author\":{\"email\":\"af99923804d57df769e1fbdbabc16e1c5dd902e2@gmail.com\",\"name\":\"Andrew Kelley\"},\"message\":\"bones.c\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thejoshwolfe/nethack/commits/3231a8162f90ce411af8c23d67cc04584f0ed7fa\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:28Z\"}\n{\"id\":\"2489398604\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1570089,\"login\":\"sirvon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sirvon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1570089?\"},\"repo\":{\"id\":4605713,\"name\":\"stackmagic/bitly-api-client\",\"url\":\"https://api.github.com/repos/stackmagic/bitly-api-client\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:29Z\"}\n{\"id\":\"2489398610\",\"type\":\"ForkEvent\",\"actor\":{\"id\":7791006,\"login\":\"hlk2014\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hlk2014\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7791006?\"},\"repo\":{\"id\":15464223,\"name\":\"linuxdeepin/deepin-emacs\",\"url\":\"https://api.github.com/repos/linuxdeepin/deepin-emacs\"},\"payload\":{\"forkee\":{\"id\":28678280,\"name\":\"deepin-emacs\",\"full_name\":\"hlk2014/deepin-emacs\",\"owner\":{\"login\":\"hlk2014\",\"id\":7791006,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7791006?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hlk2014\",\"html_url\":\"https://github.com/hlk2014\",\"followers_url\":\"https://api.github.com/users/hlk2014/followers\",\"following_url\":\"https://api.github.com/users/hlk2014/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/hlk2014/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/hlk2014/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/hlk2014/subscriptions\",\"organizations_url\":\"https://api.github.com/users/hlk2014/orgs\",\"repos_url\":\"https://api.github.com/users/hlk2014/repos\",\"events_url\":\"https://api.github.com/users/hlk2014/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/hlk2014/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/hlk2014/deepin-emacs\",\"description\":\"Deepin emacs\",\"fork\":true,\"url\":\"https://api.github.com/repos/hlk2014/deepin-emacs\",\"forks_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/forks\",\"keys_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/teams\",\"hooks_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/hooks\",\"issue_events_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/events\",\"assignees_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/tags\",\"blobs_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/languages\",\"stargazers_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/stargazers\",\"contributors_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/contributors\",\"subscribers_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/subscribers\",\"subscription_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/subscription\",\"commits_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/merges\",\"archive_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/downloads\",\"issues_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/hlk2014/deepin-emacs/releases{/id}\",\"created_at\":\"2015-01-01T01:06:30Z\",\"updated_at\":\"2014-12-13T11:28:12Z\",\"pushed_at\":\"2014-10-18T01:10:59Z\",\"git_url\":\"git://github.com/hlk2014/deepin-emacs.git\",\"ssh_url\":\"git@github.com:hlk2014/deepin-emacs.git\",\"clone_url\":\"https://github.com/hlk2014/deepin-emacs.git\",\"svn_url\":\"https://github.com/hlk2014/deepin-emacs\",\"homepage\":null,\"size\":57006,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:06:30Z\",\"org\":{\"id\":1592697,\"login\":\"linuxdeepin\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/linuxdeepin\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1592697?\"}}\n{\"id\":\"2489398617\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1802542,\"login\":\"ishayyaari\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?\"},\"repo\":{\"id\":4967600,\"name\":\"MiYa-Solutions/sbcx\",\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/708\",\"labels_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/708/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/708/comments\",\"events_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/708/events\",\"html_url\":\"https://github.com/MiYa-Solutions/sbcx/issues/708\",\"id\":53092805,\"number\":708,\"title\":\"Mobile: Projects\",\"user\":{\"login\":\"ishayyaari\",\"id\":1802542,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"html_url\":\"https://github.com/ishayyaari\",\"followers_url\":\"https://api.github.com/users/ishayyaari/followers\",\"following_url\":\"https://api.github.com/users/ishayyaari/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ishayyaari/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ishayyaari/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ishayyaari/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ishayyaari/orgs\",\"repos_url\":\"https://api.github.com/users/ishayyaari/repos\",\"events_url\":\"https://api.github.com/users/ishayyaari/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ishayyaari/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/labels/High\",\"name\":\"High\",\"color\":\"FF0000\"},{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/labels/UI+Feature\",\"name\":\"UI Feature\",\"color\":\"02e10c\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"ishayyaari\",\"id\":1802542,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"html_url\":\"https://github.com/ishayyaari\",\"followers_url\":\"https://api.github.com/users/ishayyaari/followers\",\"following_url\":\"https://api.github.com/users/ishayyaari/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ishayyaari/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ishayyaari/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ishayyaari/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ishayyaari/orgs\",\"repos_url\":\"https://api.github.com/users/ishayyaari/repos\",\"events_url\":\"https://api.github.com/users/ishayyaari/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ishayyaari/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/milestones/10\",\"labels_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/milestones/10/labels\",\"id\":835988,\"number\":10,\"title\":\"Rug Wash Bug Fixes\",\"description\":\"Fixes specifically for the rug wash opportunity\",\"creator\":{\"login\":\"markmilman\",\"id\":1744318,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1744318?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/markmilman\",\"html_url\":\"https://github.com/markmilman\",\"followers_url\":\"https://api.github.com/users/markmilman/followers\",\"following_url\":\"https://api.github.com/users/markmilman/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/markmilman/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/markmilman/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/markmilman/subscriptions\",\"organizations_url\":\"https://api.github.com/users/markmilman/orgs\",\"repos_url\":\"https://api.github.com/users/markmilman/repos\",\"events_url\":\"https://api.github.com/users/markmilman/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/markmilman/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":6,\"closed_issues\":37,\"state\":\"open\",\"created_at\":\"2014-10-22T14:08:56Z\",\"updated_at\":\"2015-01-01T01:05:32Z\",\"due_on\":\"2014-11-11T08:00:00Z\",\"closed_at\":null},\"comments\":1,\"created_at\":\"2014-12-30T04:38:31Z\",\"updated_at\":\"2015-01-01T01:06:31Z\",\"closed_at\":null,\"body\":\"\\r\\n\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/comments/68477360\",\"html_url\":\"https://github.com/MiYa-Solutions/sbcx/issues/708#issuecomment-68477360\",\"issue_url\":\"https://api.github.com/repos/MiYa-Solutions/sbcx/issues/708\",\"id\":68477360,\"user\":{\"login\":\"ishayyaari\",\"id\":1802542,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1802542?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ishayyaari\",\"html_url\":\"https://github.com/ishayyaari\",\"followers_url\":\"https://api.github.com/users/ishayyaari/followers\",\"following_url\":\"https://api.github.com/users/ishayyaari/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/ishayyaari/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/ishayyaari/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/ishayyaari/subscriptions\",\"organizations_url\":\"https://api.github.com/users/ishayyaari/orgs\",\"repos_url\":\"https://api.github.com/users/ishayyaari/repos\",\"events_url\":\"https://api.github.com/users/ishayyaari/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/ishayyaari/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:31Z\",\"updated_at\":\"2015-01-01T01:06:31Z\",\"body\":\"Add link - new project\\r\\nDesign screen:\\r\\nnew project\\r\\nproject show\\r\\nprojects index\\r\\n\\r\\nAdd link to New Job\\r\\n\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:32Z\",\"org\":{\"id\":1788572,\"login\":\"MiYa-Solutions\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/MiYa-Solutions\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1788572?\"}}\n{\"id\":\"2489398618\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536753434,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cf281dd3839c3e07ed867a2eb77aa4965a2a1400\",\"before\":\"4a08cf46a835742f49c5272d70f3210529dbb819\",\"commits\":[{\"sha\":\"cf281dd3839c3e07ed867a2eb77aa4965a2a1400\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074389244\\n\\nnRgGIKdriEl0S2qJGXnQVTH0OZV30N+8iE2Gf95JgnQ=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/cf281dd3839c3e07ed867a2eb77aa4965a2a1400\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:33Z\"}\n{\"id\":\"2489398619\",\"type\":\"PushEvent\",\"actor\":{\"id\":5240798,\"login\":\"hxwang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hxwang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5240798?\"},\"repo\":{\"id\":20258812,\"name\":\"hxwang/Leetcode\",\"url\":\"https://api.github.com/repos/hxwang/Leetcode\"},\"payload\":{\"push_id\":536753435,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"628e4f129b5723f944d1a7887509c737762b2821\",\"before\":\"e67f3471c2ea1c6556ff18137af91f8d92e5e39d\",\"commits\":[{\"sha\":\"628e4f129b5723f944d1a7887509c737762b2821\",\"author\":{\"email\":\"320cb2ff8e2e195f7d4e5cd3b27b690e919d61e5@gmail.com\",\"name\":\"Huangxin\"},\"message\":\"Update summary-types.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxwang/Leetcode/commits/628e4f129b5723f944d1a7887509c737762b2821\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:33Z\"}\n{\"id\":\"2489398634\",\"type\":\"WatchEvent\",\"actor\":{\"id\":117788,\"login\":\"nyarla\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nyarla\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/117788?\"},\"repo\":{\"id\":28594887,\"name\":\"opennota/macro\",\"url\":\"https://api.github.com/repos/opennota/macro\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:34Z\"}\n{\"id\":\"2489398636\",\"type\":\"PushEvent\",\"actor\":{\"id\":9882658,\"login\":\"sw1308\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sw1308\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9882658?\"},\"repo\":{\"id\":27245135,\"name\":\"sw1308/Programming\",\"url\":\"https://api.github.com/repos/sw1308/Programming\"},\"payload\":{\"push_id\":536753444,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"64297900b95bff1508a18a31e53ff9f25a89b7f7\",\"before\":\"6d0cc02640c6ad8a909736a0415cf95f8e43f693\",\"commits\":[{\"sha\":\"64297900b95bff1508a18a31e53ff9f25a89b7f7\",\"author\":{\"email\":\"4c225d032373d049685b78ca1f2e87cbde7ff0e2@york.ac.uk\",\"name\":\"sw1308\"},\"message\":\"Created new files for graphics testing\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sw1308/Programming/commits/64297900b95bff1508a18a31e53ff9f25a89b7f7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:34Z\"}\n{\"id\":\"2489398638\",\"type\":\"WatchEvent\",\"actor\":{\"id\":30923,\"login\":\"digitalsanctum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/digitalsanctum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/30923?\"},\"repo\":{\"id\":16621659,\"name\":\"real-logic/Aeron\",\"url\":\"https://api.github.com/repos/real-logic/Aeron\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:37Z\",\"org\":{\"id\":5371471,\"login\":\"real-logic\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/real-logic\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5371471?\"}}\n{\"id\":\"2489398640\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":8218499,\"login\":\"GunZi200\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GunZi200\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8218499?\"},\"repo\":{\"id\":27030606,\"name\":\"GunZi200/Memory-Colour\",\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/2\",\"labels_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/2/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/2/comments\",\"events_url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/issues/2/events\",\"html_url\":\"https://github.com/GunZi200/Memory-Colour/issues/2\",\"id\":53210296,\"number\":2,\"title\":\"Heart is rendered incorrectly\",\"user\":{\"login\":\"GunZi200\",\"id\":8218499,\"avatar_url\":\"https://avatars.githubusercontent.com/u/8218499?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GunZi200\",\"html_url\":\"https://github.com/GunZi200\",\"followers_url\":\"https://api.github.com/users/GunZi200/followers\",\"following_url\":\"https://api.github.com/users/GunZi200/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/GunZi200/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/GunZi200/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/GunZi200/subscriptions\",\"organizations_url\":\"https://api.github.com/users/GunZi200/orgs\",\"repos_url\":\"https://api.github.com/users/GunZi200/repos\",\"events_url\":\"https://api.github.com/users/GunZi200/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/GunZi200/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/GunZi200/Memory-Colour/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:06:35Z\",\"updated_at\":\"2015-01-01T01:06:35Z\",\"closed_at\":null,\"body\":\"On some devices(at least iPhone), the two circles and diamond that make up the heart can be seen clearly. Displaying sharp edges and not a smooth heart icon.\\r\\n\\r\\nThis will me fixed soon.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:37Z\"}\n{\"id\":\"2489398646\",\"type\":\"PushEvent\",\"actor\":{\"id\":506010,\"login\":\"gabeshaughnessy\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gabeshaughnessy\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/506010?\"},\"repo\":{\"id\":13913264,\"name\":\"gabeshaughnessy/augmentedart\",\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart\"},\"payload\":{\"push_id\":536753451,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/dungeon-hacker\",\"head\":\"dcb8b8efe59d396379657ae1d6884406e628fbdb\",\"before\":\"1855adda47047b9a781de334f4616c312ec65dc6\",\"commits\":[{\"sha\":\"dcb8b8efe59d396379657ae1d6884406e628fbdb\",\"author\":{\"email\":\"a2b2bb6e7f1b10ac88b326d5c10e33af6a8546bc@gmail.com\",\"name\":\"gabeshaughnessy\"},\"message\":\"task list\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gabeshaughnessy/augmentedart/commits/dcb8b8efe59d396379657ae1d6884406e628fbdb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:38Z\"}\n{\"id\":\"2489398647\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753448,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"bd9e3811383e5c3fc3bbd2b9ddd90047df70e7c5\",\"before\":\"274cca1b2adc063b47d558ac9bafa0dce7a05abb\",\"commits\":[{\"sha\":\"bd9e3811383e5c3fc3bbd2b9ddd90047df70e7c5\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"fix relative paths inside of views to return parent\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/bd9e3811383e5c3fc3bbd2b9ddd90047df70e7c5\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:38Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398653\",\"type\":\"PushEvent\",\"actor\":{\"id\":4729139,\"login\":\"poschengband\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/poschengband\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4729139?\"},\"repo\":{\"id\":10765478,\"name\":\"poschengband/poschengband\",\"url\":\"https://api.github.com/repos/poschengband/poschengband\"},\"payload\":{\"push_id\":536753455,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"a3a5ea26c7a02f84e24b2866eadc12f00ff681d1\",\"before\":\"de2d3a235737a729201a3b0578f9bfc73731dc20\",\"commits\":[{\"sha\":\"a3a5ea26c7a02f84e24b2866eadc12f00ff681d1\",\"author\":{\"email\":\"59f8025d5107eab2ad159f3dd097ebc035e401a3@gmail.com\",\"name\":\"poschengband\"},\"message\":\"Possessor: Playtesting\",\"distinct\":true,\"url\":\"https://api.github.com/repos/poschengband/poschengband/commits/a3a5ea26c7a02f84e24b2866eadc12f00ff681d1\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:38Z\"}\n{\"id\":\"2489398659\",\"type\":\"PushEvent\",\"actor\":{\"id\":1588951,\"login\":\"TAGC\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TAGC\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1588951?\"},\"repo\":{\"id\":28516105,\"name\":\"TAGC/Semver\",\"url\":\"https://api.github.com/repos/TAGC/Semver\"},\"payload\":{\"push_id\":536753456,\"size\":6,\"distinct_size\":6,\"ref\":\"refs/heads/develop\",\"head\":\"9bc9f01848ee531b14b2ba1faefbd1f58fc105fc\",\"before\":\"c14bd6aeef1be553879694e2c3ba8535749f5380\",\"commits\":[{\"sha\":\"1dfd6b42e9c69b76f8953e87f09b28bcdfb523f2\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Configured cobertura\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/1dfd6b42e9c69b76f8953e87f09b28bcdfb523f2\"},{\"sha\":\"f90418a9a618f5d7ff8781046a920fb892b2bbf8\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Configure codenarc\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/f90418a9a618f5d7ff8781046a920fb892b2bbf8\"},{\"sha\":\"227c645edb028aa30e09a80f40febcdb6d872e31\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Additional codenarc configuration\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/227c645edb028aa30e09a80f40febcdb6d872e31\"},{\"sha\":\"50c650626bb7df42b8bb388bff84abaf0efd6b1d\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Additional configuration in build script\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/50c650626bb7df42b8bb388bff84abaf0efd6b1d\"},{\"sha\":\"8c50d4f13932c7ccf3a78fa2fc7b9db5395bc809\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Merge branch 'feature/configure/codenarc' into develop\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/8c50d4f13932c7ccf3a78fa2fc7b9db5395bc809\"},{\"sha\":\"9bc9f01848ee531b14b2ba1faefbd1f58fc105fc\",\"author\":{\"email\":\"ff51050dd5988c994d7cae2035c2292721cc7625@gmail.com\",\"name\":\"David\"},\"message\":\"Merge branch 'develop' of https://github.com/TAGC/Semver into develop\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TAGC/Semver/commits/9bc9f01848ee531b14b2ba1faefbd1f58fc105fc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:39Z\"}\n{\"id\":\"2489398660\",\"type\":\"PushEvent\",\"actor\":{\"id\":1580785,\"login\":\"gmilligan\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gmilligan\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1580785?\"},\"repo\":{\"id\":28199368,\"name\":\"gmilligan/gmilligan.github.io\",\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io\"},\"payload\":{\"push_id\":536753457,\"size\":5,\"distinct_size\":5,\"ref\":\"refs/heads/development\",\"head\":\"2424cedc72a4b31e9feb52fb0c73c610b29f255b\",\"before\":\"d11816d799c6a50f019dd22eaf9852ac2a80e2cb\",\"commits\":[{\"sha\":\"099396797049f59a65b1c67089a9bd9809d9257c\",\"author\":{\"email\":\"b4b508b1a3018069c698b5419b143ce95eea496c@gmail.com\",\"name\":\"Greg Milligan\"},\"message\":\"lint and DI fixes for production build\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io/commits/099396797049f59a65b1c67089a9bd9809d9257c\"},{\"sha\":\"e5e753c4280c77241f6935193df429d89db25872\",\"author\":{\"email\":\"b4b508b1a3018069c698b5419b143ce95eea496c@gmail.com\",\"name\":\"Greg Milligan\"},\"message\":\"less to css changes\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io/commits/e5e753c4280c77241f6935193df429d89db25872\"},{\"sha\":\"da058848e25979191312fbd394572403990ffb8d\",\"author\":{\"email\":\"b4b508b1a3018069c698b5419b143ce95eea496c@gmail.com\",\"name\":\"Greg Milligan\"},\"message\":\"add Gulp production automation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io/commits/da058848e25979191312fbd394572403990ffb8d\"},{\"sha\":\"d9c959104e576ab4ef20c92c468c8d0b513b9f52\",\"author\":{\"email\":\"b4b508b1a3018069c698b5419b143ce95eea496c@gmail.com\",\"name\":\"Greg Milligan\"},\"message\":\"gulp automation edits\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io/commits/d9c959104e576ab4ef20c92c468c8d0b513b9f52\"},{\"sha\":\"2424cedc72a4b31e9feb52fb0c73c610b29f255b\",\"author\":{\"email\":\"b4b508b1a3018069c698b5419b143ce95eea496c@gmail.com\",\"name\":\"Greg Milligan\"},\"message\":\"remove mdeium sized images\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gmilligan/gmilligan.github.io/commits/2424cedc72a4b31e9feb52fb0c73c610b29f255b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:39Z\"}\n{\"id\":\"2489398661\",\"type\":\"PushEvent\",\"actor\":{\"id\":2209144,\"login\":\"bwbaugh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bwbaugh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2209144?\"},\"repo\":{\"id\":28676842,\"name\":\"bwbaugh/dotfiles\",\"url\":\"https://api.github.com/repos/bwbaugh/dotfiles\"},\"payload\":{\"push_id\":536753458,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"006ae1000b628dfbc2e7b57fc44486c10236512e\",\"before\":\"24dd5a64192f7855318b0d050f8e1954f9a603e9\",\"commits\":[{\"sha\":\"006ae1000b628dfbc2e7b57fc44486c10236512e\",\"author\":{\"email\":\"564269323e6a7fccff1f236f3209e6b63492d40b\",\"name\":\"Wesley Baugh\"},\"message\":\"Add path for macports\",\"distinct\":true,\"url\":\"https://api.github.com/repos/bwbaugh/dotfiles/commits/006ae1000b628dfbc2e7b57fc44486c10236512e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:39Z\"}\n{\"id\":\"2489398664\",\"type\":\"PushEvent\",\"actor\":{\"id\":7122850,\"login\":\"sararibeiro\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/sararibeiro\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7122850?\"},\"repo\":{\"id\":21713999,\"name\":\"sararibeiro/gtg\",\"url\":\"https://api.github.com/repos/sararibeiro/gtg\"},\"payload\":{\"push_id\":536753459,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/calendar-plugin\",\"head\":\"95cdec6034a10610bbe827b43689a5ed0f9dfb6e\",\"before\":\"b1a0be746888e62c067e26df65f8ab0c0bfe7add\",\"commits\":[{\"sha\":\"95cdec6034a10610bbe827b43689a5ed0f9dfb6e\",\"author\":{\"email\":\"f90fb4f31eb6dfae57fef64f6d1adbd053837f59@gmail.com\",\"name\":\"Sara Ribeiro\"},\"message\":\"Fixing issues pointed by Izidor's code review.\\n* Removed FIXME that were left on the code.\\n* Added copyright to all the files.\\n* Fixed small bug that happened when transitioning from December 31st to\\n  January.\\n* Listed calendar view as an available plugin on userdocs.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/sararibeiro/gtg/commits/95cdec6034a10610bbe827b43689a5ed0f9dfb6e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:40Z\"}\n{\"id\":\"2489398668\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":99834,\"login\":\"FCO\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?\"},\"repo\":{\"id\":26700916,\"name\":\"FCO/Dispatcher\",\"url\":\"https://api.github.com/repos/FCO/Dispatcher\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19\",\"labels_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/comments\",\"events_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/events\",\"html_url\":\"https://github.com/FCO/Dispatcher/issues/19\",\"id\":52033743,\"number\":19,\"title\":\"cookie\",\"user\":{\"login\":\"FCO\",\"id\":99834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"html_url\":\"https://github.com/FCO\",\"followers_url\":\"https://api.github.com/users/FCO/followers\",\"following_url\":\"https://api.github.com/users/FCO/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/FCO/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/FCO/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/FCO/subscriptions\",\"organizations_url\":\"https://api.github.com/users/FCO/orgs\",\"repos_url\":\"https://api.github.com/users/FCO/repos\",\"events_url\":\"https://api.github.com/users/FCO/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/FCO/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"FCO\",\"id\":99834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"html_url\":\"https://github.com/FCO\",\"followers_url\":\"https://api.github.com/users/FCO/followers\",\"following_url\":\"https://api.github.com/users/FCO/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/FCO/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/FCO/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/FCO/subscriptions\",\"organizations_url\":\"https://api.github.com/users/FCO/orgs\",\"repos_url\":\"https://api.github.com/users/FCO/repos\",\"events_url\":\"https://api.github.com/users/FCO/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/FCO/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-15T20:21:40Z\",\"updated_at\":\"2015-01-01T01:06:41Z\",\"closed_at\":\"2015-01-01T01:06:41Z\",\"body\":\"\"},\"comment\":{\"url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/comments/68477365\",\"html_url\":\"https://github.com/FCO/Dispatcher/issues/19#issuecomment-68477365\",\"issue_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19\",\"id\":68477365,\"user\":{\"login\":\"FCO\",\"id\":99834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"html_url\":\"https://github.com/FCO\",\"followers_url\":\"https://api.github.com/users/FCO/followers\",\"following_url\":\"https://api.github.com/users/FCO/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/FCO/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/FCO/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/FCO/subscriptions\",\"organizations_url\":\"https://api.github.com/users/FCO/orgs\",\"repos_url\":\"https://api.github.com/users/FCO/repos\",\"events_url\":\"https://api.github.com/users/FCO/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/FCO/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:40Z\",\"updated_at\":\"2015-01-01T01:06:40Z\",\"body\":\"Not done\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398669\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":99834,\"login\":\"FCO\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?\"},\"repo\":{\"id\":26700916,\"name\":\"FCO/Dispatcher\",\"url\":\"https://api.github.com/repos/FCO/Dispatcher\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19\",\"labels_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/comments\",\"events_url\":\"https://api.github.com/repos/FCO/Dispatcher/issues/19/events\",\"html_url\":\"https://github.com/FCO/Dispatcher/issues/19\",\"id\":52033743,\"number\":19,\"title\":\"cookie\",\"user\":{\"login\":\"FCO\",\"id\":99834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"html_url\":\"https://github.com/FCO\",\"followers_url\":\"https://api.github.com/users/FCO/followers\",\"following_url\":\"https://api.github.com/users/FCO/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/FCO/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/FCO/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/FCO/subscriptions\",\"organizations_url\":\"https://api.github.com/users/FCO/orgs\",\"repos_url\":\"https://api.github.com/users/FCO/repos\",\"events_url\":\"https://api.github.com/users/FCO/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/FCO/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":{\"login\":\"FCO\",\"id\":99834,\"avatar_url\":\"https://avatars.githubusercontent.com/u/99834?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/FCO\",\"html_url\":\"https://github.com/FCO\",\"followers_url\":\"https://api.github.com/users/FCO/followers\",\"following_url\":\"https://api.github.com/users/FCO/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/FCO/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/FCO/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/FCO/subscriptions\",\"organizations_url\":\"https://api.github.com/users/FCO/orgs\",\"repos_url\":\"https://api.github.com/users/FCO/repos\",\"events_url\":\"https://api.github.com/users/FCO/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/FCO/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":1,\"created_at\":\"2014-12-15T20:21:40Z\",\"updated_at\":\"2015-01-01T01:06:41Z\",\"closed_at\":\"2015-01-01T01:06:41Z\",\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398672\",\"type\":\"ForkEvent\",\"actor\":{\"id\":7151313,\"login\":\"theTechnoKid\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/theTechnoKid\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7151313?\"},\"repo\":{\"id\":11571950,\"name\":\"spywhere/Monopoly\",\"url\":\"https://api.github.com/repos/spywhere/Monopoly\"},\"payload\":{\"forkee\":{\"id\":28678282,\"name\":\"Monopoly\",\"full_name\":\"theTechnoKid/Monopoly\",\"owner\":{\"login\":\"theTechnoKid\",\"id\":7151313,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7151313?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/theTechnoKid\",\"html_url\":\"https://github.com/theTechnoKid\",\"followers_url\":\"https://api.github.com/users/theTechnoKid/followers\",\"following_url\":\"https://api.github.com/users/theTechnoKid/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/theTechnoKid/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/theTechnoKid/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/theTechnoKid/subscriptions\",\"organizations_url\":\"https://api.github.com/users/theTechnoKid/orgs\",\"repos_url\":\"https://api.github.com/users/theTechnoKid/repos\",\"events_url\":\"https://api.github.com/users/theTechnoKid/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/theTechnoKid/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/theTechnoKid/Monopoly\",\"description\":\"Project for Object-Oriented Computing\",\"fork\":true,\"url\":\"https://api.github.com/repos/theTechnoKid/Monopoly\",\"forks_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/forks\",\"keys_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/teams\",\"hooks_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/hooks\",\"issue_events_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/events\",\"assignees_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/tags\",\"blobs_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/languages\",\"stargazers_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/stargazers\",\"contributors_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/contributors\",\"subscribers_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/subscribers\",\"subscription_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/subscription\",\"commits_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/merges\",\"archive_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/downloads\",\"issues_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/theTechnoKid/Monopoly/releases{/id}\",\"created_at\":\"2015-01-01T01:06:41Z\",\"updated_at\":\"2013-10-15T12:29:53Z\",\"pushed_at\":\"2013-07-22T08:34:18Z\",\"git_url\":\"git://github.com/theTechnoKid/Monopoly.git\",\"ssh_url\":\"git@github.com:theTechnoKid/Monopoly.git\",\"clone_url\":\"https://github.com/theTechnoKid/Monopoly.git\",\"svn_url\":\"https://github.com/theTechnoKid/Monopoly\",\"homepage\":null,\"size\":564,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398676\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1734986,\"login\":\"Adamwgoh\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Adamwgoh\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1734986?\"},\"repo\":{\"id\":28678222,\"name\":\"Adamwgoh/Laura\",\"url\":\"https://api.github.com/repos/Adamwgoh/Laura\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"G52GRP re-upload\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398678\",\"type\":\"PushEvent\",\"actor\":{\"id\":3794984,\"login\":\"abustamam\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/abustamam\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3794984?\"},\"repo\":{\"id\":25110710,\"name\":\"abustamam/Udacity-Portfolio\",\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio\"},\"payload\":{\"push_id\":536753462,\"size\":4,\"distinct_size\":0,\"ref\":\"refs/heads/gh-pages\",\"head\":\"5776bf9082fbc389cc986d9159e7367cc2fcdf20\",\"before\":\"781f999ab1214e4ca396fe3bdb7721ec5e81d208\",\"commits\":[{\"sha\":\"585271ef771aad13ac3cdc7ba5aeb766e843e972\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Resize photos, change cover image\",\"distinct\":false,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/585271ef771aad13ac3cdc7ba5aeb766e843e972\"},{\"sha\":\"b39e5ac9096f3ddcc10a485bd9df8dbedcfda5ef\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Update README.md\",\"distinct\":false,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/b39e5ac9096f3ddcc10a485bd9df8dbedcfda5ef\"},{\"sha\":\"e323ac3bd7451b41388d083ec83e1dc3a4d94022\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Change project 1 img\",\"distinct\":false,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/e323ac3bd7451b41388d083ec83e1dc3a4d94022\"},{\"sha\":\"5776bf9082fbc389cc986d9159e7367cc2fcdf20\",\"author\":{\"email\":\"1f8e819f7c9fbaadfc4b5b78cc4be161892e8692@gmail.com\",\"name\":\"Rasheed Bustamam\"},\"message\":\"Merge branch 'master' of github.com:abustamam/Udacity-Portfolio\\n\\nPull readme\",\"distinct\":false,\"url\":\"https://api.github.com/repos/abustamam/Udacity-Portfolio/commits/5776bf9082fbc389cc986d9159e7367cc2fcdf20\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398680\",\"type\":\"PushEvent\",\"actor\":{\"id\":7275188,\"login\":\"mxcapo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/mxcapo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7275188?\"},\"repo\":{\"id\":27788033,\"name\":\"mxcapo/mxcapo.github.io\",\"url\":\"https://api.github.com/repos/mxcapo/mxcapo.github.io\"},\"payload\":{\"push_id\":536753464,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"14c4f783a8d62d81b30c3268deca41ae4d4f1133\",\"before\":\"8dc39e5aa797d7f340d56ba94bc51e83374fc90a\",\"commits\":[{\"sha\":\"14c4f783a8d62d81b30c3268deca41ae4d4f1133\",\"author\":{\"email\":\"a7f48bd5c86f435aba4630eb6fad8b467d5719b5@gmail.com\",\"name\":\"mxcapo\"},\"message\":\"tweak\",\"distinct\":true,\"url\":\"https://api.github.com/repos/mxcapo/mxcapo.github.io/commits/14c4f783a8d62d81b30c3268deca41ae4d4f1133\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398681\",\"type\":\"PushEvent\",\"actor\":{\"id\":676533,\"login\":\"501st-alpha1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/501st-alpha1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/676533?\"},\"repo\":{\"id\":15497204,\"name\":\"501st-alpha1/emacs-init\",\"url\":\"https://api.github.com/repos/501st-alpha1/emacs-init\"},\"payload\":{\"push_id\":536753465,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"a56556f9b33872e0ed4aca1fdf773e24215afdd3\",\"before\":\"8ac7f1e49e1a9b29f4243b3c126aab766ff7d95c\",\"commits\":[{\"sha\":\"a56556f9b33872e0ed4aca1fdf773e24215afdd3\",\"author\":{\"email\":\"625600233cb3bcab32268c17610882e0fdaed295@scott-weldon.com\",\"name\":\"Scott Weldon\"},\"message\":\"Allow setting auto-revert-mode in .dir-locals.el\",\"distinct\":true,\"url\":\"https://api.github.com/repos/501st-alpha1/emacs-init/commits/a56556f9b33872e0ed4aca1fdf773e24215afdd3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398683\",\"type\":\"PushEvent\",\"actor\":{\"id\":76367,\"login\":\"md5\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/md5\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/76367?\"},\"repo\":{\"id\":28625804,\"name\":\"astrifex/astrifex\",\"url\":\"https://api.github.com/repos/astrifex/astrifex\"},\"payload\":{\"push_id\":536753466,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1aadb40fa78ff80d4e60a403d2819af18fd1d119\",\"before\":\"ddb58acdb8e1ed67dbcbe45d76fee55d09a29577\",\"commits\":[{\"sha\":\"1aadb40fa78ff80d4e60a403d2819af18fd1d119\",\"author\":{\"email\":\"a17fed27eaa842282862ff7c1b9c8395a26ac320@embody.org\",\"name\":\"Mike Dillon\"},\"message\":\"Add size\",\"distinct\":true,\"url\":\"https://api.github.com/repos/astrifex/astrifex/commits/1aadb40fa78ff80d4e60a403d2819af18fd1d119\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\",\"org\":{\"id\":10359906,\"login\":\"astrifex\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/astrifex\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10359906?\"}}\n{\"id\":\"2489398684\",\"type\":\"PushEvent\",\"actor\":{\"id\":1785816,\"login\":\"natashavlahakis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/natashavlahakis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1785816?\"},\"repo\":{\"id\":27246389,\"name\":\"natashavlahakis/figtree\",\"url\":\"https://api.github.com/repos/natashavlahakis/figtree\"},\"payload\":{\"push_id\":536753467,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"9791f481f66b12d027b824e5d13771451e1ad28a\",\"before\":\"03e086487f01d5fff1574e9906ad4a4e06082211\",\"commits\":[{\"sha\":\"9791f481f66b12d027b824e5d13771451e1ad28a\",\"author\":{\"email\":\"84a888d1cdc9202fb22178d441aabd7da0eefd07@gmail.com\",\"name\":\"Natasha Vlahakis\"},\"message\":\"added row\",\"distinct\":true,\"url\":\"https://api.github.com/repos/natashavlahakis/figtree/commits/9791f481f66b12d027b824e5d13771451e1ad28a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398685\",\"type\":\"PushEvent\",\"actor\":{\"id\":166301,\"login\":\"bcomnes\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bcomnes\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/166301?\"},\"repo\":{\"id\":6861308,\"name\":\"bcomnes/bcomnes.github.io\",\"url\":\"https://api.github.com/repos/bcomnes/bcomnes.github.io\"},\"payload\":{\"push_id\":536753468,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f75503acf7e511ce91c18a99ce1042cc2e1e650b\",\"before\":\"821968b470015bf7a6d5f86aeea8d739dd1aadbe\",\"commits\":[{\"sha\":\"f75503acf7e511ce91c18a99ce1042cc2e1e650b\",\"author\":{\"email\":\"6df4fba95631fe4f4c4337307cda4e0fc4c27d16@gmail.com\",\"name\":\"Bret Comnes\"},\"message\":\"Update 2015-01-01-shell-art.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/bcomnes/bcomnes.github.io/commits/f75503acf7e511ce91c18a99ce1042cc2e1e650b\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398686\",\"type\":\"ForkEvent\",\"actor\":{\"id\":880050,\"login\":\"kazutomi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kazutomi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/880050?\"},\"repo\":{\"id\":115341,\"name\":\"alexvollmer/daemon-spawn\",\"url\":\"https://api.github.com/repos/alexvollmer/daemon-spawn\"},\"payload\":{\"forkee\":{\"id\":28678283,\"name\":\"daemon-spawn\",\"full_name\":\"kazutomi/daemon-spawn\",\"owner\":{\"login\":\"kazutomi\",\"id\":880050,\"avatar_url\":\"https://avatars.githubusercontent.com/u/880050?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kazutomi\",\"html_url\":\"https://github.com/kazutomi\",\"followers_url\":\"https://api.github.com/users/kazutomi/followers\",\"following_url\":\"https://api.github.com/users/kazutomi/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kazutomi/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kazutomi/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kazutomi/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kazutomi/orgs\",\"repos_url\":\"https://api.github.com/users/kazutomi/repos\",\"events_url\":\"https://api.github.com/users/kazutomi/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kazutomi/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/kazutomi/daemon-spawn\",\"description\":\"A simple, flexible daemon management library.\",\"fork\":true,\"url\":\"https://api.github.com/repos/kazutomi/daemon-spawn\",\"forks_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/forks\",\"keys_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/teams\",\"hooks_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/hooks\",\"issue_events_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/events\",\"assignees_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/tags\",\"blobs_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/languages\",\"stargazers_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/stargazers\",\"contributors_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/contributors\",\"subscribers_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/subscribers\",\"subscription_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/subscription\",\"commits_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/merges\",\"archive_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/downloads\",\"issues_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/kazutomi/daemon-spawn/releases{/id}\",\"created_at\":\"2015-01-01T01:06:43Z\",\"updated_at\":\"2014-12-29T02:27:27Z\",\"pushed_at\":\"2012-11-21T17:14:36Z\",\"git_url\":\"git://github.com/kazutomi/daemon-spawn.git\",\"ssh_url\":\"git@github.com:kazutomi/daemon-spawn.git\",\"clone_url\":\"https://github.com/kazutomi/daemon-spawn.git\",\"svn_url\":\"https://github.com/kazutomi/daemon-spawn\",\"homepage\":\"http://www.evri.com\",\"size\":155,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\",\"public\":true}},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398687\",\"type\":\"PushEvent\",\"actor\":{\"id\":5090808,\"login\":\"iderioja\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/iderioja\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5090808?\"},\"repo\":{\"id\":16545715,\"name\":\"iderioja/base_datos_geografica\",\"url\":\"https://api.github.com/repos/iderioja/base_datos_geografica\"},\"payload\":{\"push_id\":536753469,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"101f98a1fb0221d38c432fe06e223ea3c08926fb\",\"before\":\"0f5909f48c0a61d3151203b4662ed2d0c34a9b5d\",\"commits\":[{\"sha\":\"101f98a1fb0221d38c432fe06e223ea3c08926fb\",\"author\":{\"email\":\"0c9edb4bafa26e2cc6fb9112520016fa2774e8ce@wamba.larioja.org\",\"name\":\"fmeserver\"},\"message\":\"Actualización geoJSON desde tablas Oracle con FME. 20150101020635\",\"distinct\":true,\"url\":\"https://api.github.com/repos/iderioja/base_datos_geografica/commits/101f98a1fb0221d38c432fe06e223ea3c08926fb\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398689\",\"type\":\"PushEvent\",\"actor\":{\"id\":380021,\"login\":\"wolf31o2\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wolf31o2\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/380021?\"},\"repo\":{\"id\":17919097,\"name\":\"caskdata/coopr\",\"url\":\"https://api.github.com/repos/caskdata/coopr\"},\"payload\":{\"push_id\":536753471,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/feature/dockerfile\",\"head\":\"026cc002d2379d68209f00d609d56448e66bcb15\",\"before\":\"3b77a6638a992f0b8f605aab3d44ce9987b78741\",\"commits\":[{\"sha\":\"026cc002d2379d68209f00d609d56448e66bcb15\",\"author\":{\"email\":\"711c73f64afdce07b7e38039a96d2224209e9a6c@cask.co\",\"name\":\"Chris Gianelloni\"},\"message\":\"Do not use localhost for any externally-exposed ports\",\"distinct\":true,\"url\":\"https://api.github.com/repos/caskdata/coopr/commits/026cc002d2379d68209f00d609d56448e66bcb15\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\",\"org\":{\"id\":8123023,\"login\":\"caskdata\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/caskdata\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8123023?\"}}\n{\"id\":\"2489398690\",\"type\":\"PushEvent\",\"actor\":{\"id\":960411,\"login\":\"fauxfiction\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fauxfiction\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/960411?\"},\"repo\":{\"id\":28618969,\"name\":\"fauxfiction/Sick-Beard\",\"url\":\"https://api.github.com/repos/fauxfiction/Sick-Beard\"},\"payload\":{\"push_id\":536753472,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/fix-newznab-response-xpath\",\"head\":\"5f6888811bcc778c8832b4d77f05d047013c38ba\",\"before\":\"69817a46a431c25adc66c236fca351e3d904b389\",\"commits\":[{\"sha\":\"5f6888811bcc778c8832b4d77f05d047013c38ba\",\"author\":{\"email\":\"64b2b6d12bfe4baae7dad3d018f8cbf6b0e7a044@fauxfiction.com\",\"name\":\"fauxfiction\"},\"message\":\"Iterate over XML etree nodes in newznab provider\\n\\nInstead of using XPath queries which are not supported prior to Python\\n2.7, grab an iterator to collect nodes of interest.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/fauxfiction/Sick-Beard/commits/5f6888811bcc778c8832b4d77f05d047013c38ba\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:43Z\"}\n{\"id\":\"2489398696\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1170642,\"login\":\"niofis\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/niofis\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1170642?\"},\"repo\":{\"id\":11541598,\"name\":\"hermanhermitage/videocoreiv-qpu\",\"url\":\"https://api.github.com/repos/hermanhermitage/videocoreiv-qpu\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:44Z\"}\n{\"id\":\"2489398698\",\"type\":\"PushEvent\",\"actor\":{\"id\":1515006,\"login\":\"LordPsychoMaster\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/LordPsychoMaster\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1515006?\"},\"repo\":{\"id\":28061377,\"name\":\"LordPsychoMaster/SchitEngine\",\"url\":\"https://api.github.com/repos/LordPsychoMaster/SchitEngine\"},\"payload\":{\"push_id\":536753473,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"54d4858b59afff9fd288692a97b22ed45cdbe7bc\",\"before\":\"8c90925c3d2e0d6fb7d96e29ff671667232ba26f\",\"commits\":[{\"sha\":\"4c64252c67e7b0ec6728c68e273a7e4226673750\",\"author\":{\"email\":\"52153f19f64cb8f0c20cf48997f6dcb5cc4cb401@gmail.com\",\"name\":\"Jonathan Jao\"},\"message\":\"cleacleaned up directory\",\"distinct\":true,\"url\":\"https://api.github.com/repos/LordPsychoMaster/SchitEngine/commits/4c64252c67e7b0ec6728c68e273a7e4226673750\"},{\"sha\":\"8bca8506bac64aa2036237cb31447883f00d8249\",\"author\":{\"email\":\"52153f19f64cb8f0c20cf48997f6dcb5cc4cb401@gmail.com\",\"name\":\"Jonathan Jao\"},\"message\":\"Back and Forth Sockets handshake done\",\"distinct\":true,\"url\":\"https://api.github.com/repos/LordPsychoMaster/SchitEngine/commits/8bca8506bac64aa2036237cb31447883f00d8249\"},{\"sha\":\"54d4858b59afff9fd288692a97b22ed45cdbe7bc\",\"author\":{\"email\":\"52153f19f64cb8f0c20cf48997f6dcb5cc4cb401@gmail.com\",\"name\":\"Jonathan Jao\"},\"message\":\"test\",\"distinct\":true,\"url\":\"https://api.github.com/repos/LordPsychoMaster/SchitEngine/commits/54d4858b59afff9fd288692a97b22ed45cdbe7bc\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:44Z\"}\n{\"id\":\"2489398699\",\"type\":\"PushEvent\",\"actor\":{\"id\":10349906,\"login\":\"noobfish\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/noobfish\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10349906?\"},\"repo\":{\"id\":28634881,\"name\":\"noobfish/simple\",\"url\":\"https://api.github.com/repos/noobfish/simple\"},\"payload\":{\"push_id\":536753474,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fb2725e3511b36d1762f388e4a38d43c46d3a3fd\",\"before\":\"b4ed23be12698cdc458884d79f406d551330b2d1\",\"commits\":[{\"sha\":\"fb2725e3511b36d1762f388e4a38d43c46d3a3fd\",\"author\":{\"email\":\"f4576d39c25ca3a0435d50421190b319ce2c2dd7@users.noreply.github.com\",\"name\":\"noobfish\"},\"message\":\"Update main.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/noobfish/simple/commits/fb2725e3511b36d1762f388e4a38d43c46d3a3fd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:44Z\"}\n{\"id\":\"2489398700\",\"type\":\"WatchEvent\",\"actor\":{\"id\":6190487,\"login\":\"Tommy-Geenexus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Tommy-Geenexus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6190487?\"},\"repo\":{\"id\":14080264,\"name\":\"faux123/Nexus_5\",\"url\":\"https://api.github.com/repos/faux123/Nexus_5\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:45Z\"}\n{\"id\":\"2489398701\",\"type\":\"GollumEvent\",\"actor\":{\"id\":7797609,\"login\":\"ivanwfr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ivanwfr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7797609?\"},\"repo\":{\"id\":808316,\"name\":\"cswetenham/tabspace2.1\",\"url\":\"https://api.github.com/repos/cswetenham/tabspace2.1\"},\"payload\":{\"pages\":[{\"page_name\":\"Brandon-Craig-Rhodes-TabSpace-Chords-Colored\",\"title\":\"Brandon Craig Rhodes TabSpace Chords Colored\",\"summary\":null,\"action\":\"edited\",\"sha\":\"d09c2420a64f21b7a0778ef7d3245689239eea08\",\"html_url\":\"https://github.com/cswetenham/tabspace2.1/wiki/Brandon-Craig-Rhodes-TabSpace-Chords-Colored\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:45Z\"}\n{\"id\":\"2489398702\",\"type\":\"PushEvent\",\"actor\":{\"id\":3022197,\"login\":\"liuyang1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/liuyang1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3022197?\"},\"repo\":{\"id\":26949326,\"name\":\"liuyang1/test\",\"url\":\"https://api.github.com/repos/liuyang1/test\"},\"payload\":{\"push_id\":536753475,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"8837445c1174eede953b25f754b128a924ee1b84\",\"before\":\"0376a5169b1e0d40ba945bbe9b991b6bd142b2ff\",\"commits\":[{\"sha\":\"355f7490539ead559764f294aade43d904ba5aa7\",\"author\":{\"email\":\"35a72eefb2bfb2a974e000ff4a69ccb41ad11391@mail.ustc.edu.cn\",\"name\":\"liuyang1\"},\"message\":\"dynamic Macro add timestamp\",\"distinct\":true,\"url\":\"https://api.github.com/repos/liuyang1/test/commits/355f7490539ead559764f294aade43d904ba5aa7\"},{\"sha\":\"8837445c1174eede953b25f754b128a924ee1b84\",\"author\":{\"email\":\"35a72eefb2bfb2a974e000ff4a69ccb41ad11391@mail.ustc.edu.cn\",\"name\":\"liuyang1\"},\"message\":\"Merge branch 'master' of github.com:liuyang1/test\",\"distinct\":true,\"url\":\"https://api.github.com/repos/liuyang1/test/commits/8837445c1174eede953b25f754b128a924ee1b84\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:45Z\"}\n{\"id\":\"2489398703\",\"type\":\"PushEvent\",\"actor\":{\"id\":9101573,\"login\":\"megantmcginley\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/megantmcginley\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9101573?\"},\"repo\":{\"id\":25549968,\"name\":\"megantmcginley/megantmcginley.github.io\",\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io\"},\"payload\":{\"push_id\":536753476,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1cfb7c3568cd96f4d5233c1ca07f94a0d6a56762\",\"before\":\"1ac2c938f5436c6ff9576e262b80ccbced4586a8\",\"commits\":[{\"sha\":\"1cfb7c3568cd96f4d5233c1ca07f94a0d6a56762\",\"author\":{\"email\":\"92f56e51255edbb80c74150d0115560b34c2bc35@users.noreply.github.com\",\"name\":\"megantmcginley\"},\"message\":\"Update about.html\",\"distinct\":true,\"url\":\"https://api.github.com/repos/megantmcginley/megantmcginley.github.io/commits/1cfb7c3568cd96f4d5233c1ca07f94a0d6a56762\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:45Z\"}\n{\"id\":\"2489398704\",\"type\":\"PushEvent\",\"actor\":{\"id\":376203,\"login\":\"JoshEngebretson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JoshEngebretson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/376203?\"},\"repo\":{\"id\":26557148,\"name\":\"AtomicGameEngine/AtomicRuntime\",\"url\":\"https://api.github.com/repos/AtomicGameEngine/AtomicRuntime\"},\"payload\":{\"push_id\":536753477,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/atomic_light2d\",\"head\":\"b2bca32f071f0b1d396834006324c09719f92353\",\"before\":\"1d650fe0e4ee13049f9e32fd7ad8d650f99b13d2\",\"commits\":[{\"sha\":\"b2bca32f071f0b1d396834006324c09719f92353\",\"author\":{\"email\":\"c028c213ed5efcf30c3f4fc7361dbde0c893c5b7@galaxyfarfaraway.com\",\"name\":\"Josh Engebretson\"},\"message\":\"Light2D Fix silly error\",\"distinct\":true,\"url\":\"https://api.github.com/repos/AtomicGameEngine/AtomicRuntime/commits/b2bca32f071f0b1d396834006324c09719f92353\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:45Z\",\"org\":{\"id\":9641691,\"login\":\"AtomicGameEngine\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/AtomicGameEngine\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9641691?\"}}\n{\"id\":\"2489398706\",\"type\":\"PushEvent\",\"actor\":{\"id\":7606292,\"login\":\"simplicit\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/simplicit\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7606292?\"},\"repo\":{\"id\":27554172,\"name\":\"lenovo-a3-dev/android_frameworks_opt_telephony\",\"url\":\"https://api.github.com/repos/lenovo-a3-dev/android_frameworks_opt_telephony\"},\"payload\":{\"push_id\":536753478,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/cm-11.0\",\"head\":\"29b19bfb6baee32941b09a186caca194f3c1756f\",\"before\":\"9e20514b298ee1cdf909084c9d1a56e8663d132a\",\"commits\":[{\"sha\":\"29b19bfb6baee32941b09a186caca194f3c1756f\",\"author\":{\"email\":\"c692d6a10598e0a801576fdd4ecf3c37e45bfbc4@cs.pitt.edu\",\"name\":\"William C. Garrison III\"},\"message\":\"Telephony: If no wrapped message, originating address is null\\n\\nA Verizon HTC m8 user reported this error:\\n\\nhttp://pastebin.com/3LaYGB6X\\n\\nwhich means that mWrappedSmsMessage is null, meaning it wasn't created\\nsuccessfully by createFromPdu(pdu, format). Let's let getOriginatingAddress()\\nreport this as a null, and see if createFromPdu(pdu) can create the wrapped\\nmessage properly.\\n\\nEven if it's still a null, TextUtils.isEmpty will return true, meaning the\\nmessage's address won't be returned (better than crashing com.process.phone).\\n\\nChange-Id: I419bc3e6045b0cf7d62a348a0be02555e0f7b76e\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lenovo-a3-dev/android_frameworks_opt_telephony/commits/29b19bfb6baee32941b09a186caca194f3c1756f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:47Z\",\"org\":{\"id\":6270737,\"login\":\"lenovo-a3-dev\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/lenovo-a3-dev\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6270737?\"}}\n{\"id\":\"2489398707\",\"type\":\"PushEvent\",\"actor\":{\"id\":10263666,\"login\":\"katiekroik\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/katiekroik\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10263666?\"},\"repo\":{\"id\":28677679,\"name\":\"jl4282/swirlwebsite\",\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite\"},\"payload\":{\"push_id\":536753479,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/develop\",\"head\":\"0b72a0bb1292ee07b8309774bdcc4165db322c26\",\"before\":\"53bc894d93b396c5cc3c224311421a017ceac40e\",\"commits\":[{\"sha\":\"0b72a0bb1292ee07b8309774bdcc4165db322c26\",\"author\":{\"email\":\"fdb375617daf85f650fdf56bce778da925caee49@nyu.edu\",\"name\":\"katiekroik\"},\"message\":\"Updated README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/jl4282/swirlwebsite/commits/0b72a0bb1292ee07b8309774bdcc4165db322c26\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:47Z\"}\n{\"id\":\"2489398715\",\"type\":\"PushEvent\",\"actor\":{\"id\":3020337,\"login\":\"ads04r\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ads04r\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3020337?\"},\"repo\":{\"id\":7232953,\"name\":\"ads04r/imouto\",\"url\":\"https://api.github.com/repos/ads04r/imouto\"},\"payload\":{\"push_id\":536753483,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"957618358c63acb1bdd466c1ca5236c5027d6298\",\"before\":\"85c3e84ba948391cfe558c49d6a7468ead1101f2\",\"commits\":[{\"sha\":\"957618358c63acb1bdd466c1ca5236c5027d6298\",\"author\":{\"email\":\"0ccde024171faa98768fb1f17761d5e2335a31b4@ecs.soton.ac.uk\",\"name\":\"Ash\"},\"message\":\"Some basic queries for generating yearly summaries\\n\\nWill be removed and integrated properly soon, I just wanted them in\\nchange control sooner rather than later.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ads04r/imouto/commits/957618358c63acb1bdd466c1ca5236c5027d6298\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:47Z\"}\n{\"id\":\"2489398721\",\"type\":\"PushEvent\",\"actor\":{\"id\":458892,\"login\":\"Havanna\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Havanna\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458892?\"},\"repo\":{\"id\":22987680,\"name\":\"Ihavee/ihavee.github.io\",\"url\":\"https://api.github.com/repos/Ihavee/ihavee.github.io\"},\"payload\":{\"push_id\":536753486,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b51e29b184d6b5c13c71376a6f7e75c42c364759\",\"before\":\"44190b8c058537664c002a93b6db1d47dd7f6bb2\",\"commits\":[{\"sha\":\"b51e29b184d6b5c13c71376a6f7e75c42c364759\",\"author\":{\"email\":\"8743e28f8544a53faec3c88031aa3516d5d474c9@yeah.net\",\"name\":\"Havanna\"},\"message\":\"update image\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Ihavee/ihavee.github.io/commits/b51e29b184d6b5c13c71376a6f7e75c42c364759\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:49Z\",\"org\":{\"id\":1856466,\"login\":\"Ihavee\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Ihavee\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1856466?\"}}\n{\"id\":\"2489398730\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1682199,\"login\":\"JayBeavers\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?\"},\"repo\":{\"id\":21885551,\"name\":\"erikringsmuth/app-router\",\"url\":\"https://api.github.com/repos/erikringsmuth/app-router\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60\",\"labels_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/comments\",\"events_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/events\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/60\",\"id\":53166608,\"number\":60,\"title\":\"Test instructions in readme out of date?\",\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-31T04:25:55Z\",\"updated_at\":\"2015-01-01T01:06:50Z\",\"closed_at\":\"2015-01-01T01:06:50Z\",\"body\":\"At [build, test, and debug](https://github.com/erikringsmuth/app-router/#build-test-and-debug-) it says to:\\r\\n\\r\\n    Manually run functional tests in the browser by starting a static content server\\r\\n    (node http-server or python -m SimpleHTTPServer) and open\\r\\n    http://localhost:8080/tests/functional-tests/\\r\\n\\r\\nBut this gets you a 404.  Perhaps you meant\\r\\n\\r\\n    http://localhost:8080/tests/SpecRunner.html\\r\\n\\r\\nBut I *think* this is taken care of by gulp.\"},\"comment\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/comments/68477366\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/60#issuecomment-68477366\",\"issue_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60\",\"id\":68477366,\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:06:50Z\",\"updated_at\":\"2015-01-01T01:06:50Z\",\"body\":\"Closing.  Would offer to PR the docs, but you're moving towards an automated approach instead.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:51Z\"}\n{\"id\":\"2489398731\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1682199,\"login\":\"JayBeavers\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?\"},\"repo\":{\"id\":21885551,\"name\":\"erikringsmuth/app-router\",\"url\":\"https://api.github.com/repos/erikringsmuth/app-router\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60\",\"labels_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/comments\",\"events_url\":\"https://api.github.com/repos/erikringsmuth/app-router/issues/60/events\",\"html_url\":\"https://github.com/erikringsmuth/app-router/issues/60\",\"id\":53166608,\"number\":60,\"title\":\"Test instructions in readme out of date?\",\"user\":{\"login\":\"JayBeavers\",\"id\":1682199,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1682199?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/JayBeavers\",\"html_url\":\"https://github.com/JayBeavers\",\"followers_url\":\"https://api.github.com/users/JayBeavers/followers\",\"following_url\":\"https://api.github.com/users/JayBeavers/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/JayBeavers/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/JayBeavers/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/JayBeavers/subscriptions\",\"organizations_url\":\"https://api.github.com/users/JayBeavers/orgs\",\"repos_url\":\"https://api.github.com/users/JayBeavers/repos\",\"events_url\":\"https://api.github.com/users/JayBeavers/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/JayBeavers/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":5,\"created_at\":\"2014-12-31T04:25:55Z\",\"updated_at\":\"2015-01-01T01:06:50Z\",\"closed_at\":\"2015-01-01T01:06:50Z\",\"body\":\"At [build, test, and debug](https://github.com/erikringsmuth/app-router/#build-test-and-debug-) it says to:\\r\\n\\r\\n    Manually run functional tests in the browser by starting a static content server\\r\\n    (node http-server or python -m SimpleHTTPServer) and open\\r\\n    http://localhost:8080/tests/functional-tests/\\r\\n\\r\\nBut this gets you a 404.  Perhaps you meant\\r\\n\\r\\n    http://localhost:8080/tests/SpecRunner.html\\r\\n\\r\\nBut I *think* this is taken care of by gulp.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:51Z\"}\n{\"id\":\"2489398735\",\"type\":\"PushEvent\",\"actor\":{\"id\":9828988,\"login\":\"alexformagio\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/alexformagio\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9828988?\"},\"repo\":{\"id\":28678021,\"name\":\"alexformagio/python30min\",\"url\":\"https://api.github.com/repos/alexformagio/python30min\"},\"payload\":{\"push_id\":536753490,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"94076cb80620e00f80a01fb228e6f01bbcaf550e\",\"before\":\"6148df2e7babc92c0568cc452deae2b683a041da\",\"commits\":[{\"sha\":\"94076cb80620e00f80a01fb228e6f01bbcaf550e\",\"author\":{\"email\":\"eb51c250a6d7f2b7307829ec95405c85965e34b2@gmail.com\",\"name\":\"alex_formagio\"},\"message\":\"uploa file3\",\"distinct\":true,\"url\":\"https://api.github.com/repos/alexformagio/python30min/commits/94076cb80620e00f80a01fb228e6f01bbcaf550e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:51Z\"}\n{\"id\":\"2489398736\",\"type\":\"PushEvent\",\"actor\":{\"id\":3332598,\"login\":\"pgruenbacher\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pgruenbacher\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3332598?\"},\"repo\":{\"id\":28670009,\"name\":\"pgruenbacher/mydrive5\",\"url\":\"https://api.github.com/repos/pgruenbacher/mydrive5\"},\"payload\":{\"push_id\":536753491,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"397ed4267dcdbff8c52324a3f66776a87495da02\",\"before\":\"9237ed0f2b971ef413bd50e748dc8579e2323a5c\",\"commits\":[{\"sha\":\"397ed4267dcdbff8c52324a3f66776a87495da02\",\"author\":{\"email\":\"6f5c7f15036059ac7b25472c8812708bfb22b1eb@gmail.com\",\"name\":\"Paul Gruenbacher\"},\"message\":\"palette\",\"distinct\":true,\"url\":\"https://api.github.com/repos/pgruenbacher/mydrive5/commits/397ed4267dcdbff8c52324a3f66776a87495da02\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:51Z\"}\n{\"id\":\"2489398743\",\"type\":\"PushEvent\",\"actor\":{\"id\":10355660,\"login\":\"nomadturk\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nomadturk\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10355660?\"},\"repo\":{\"id\":28657737,\"name\":\"nomadturk/nginx-conf\",\"url\":\"https://api.github.com/repos/nomadturk/nginx-conf\"},\"payload\":{\"push_id\":536753494,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"035679b2f50ea9d689bba803906d5b1ee3418062\",\"before\":\"f560a8bd7c3d688fd6b99a5976e3518fbbfd68a3\",\"commits\":[{\"sha\":\"035679b2f50ea9d689bba803906d5b1ee3418062\",\"author\":{\"email\":\"90283840d90de49b8e7984bd99b47fee0d4bd50d@cokh.net\",\"name\":\"nomadturk\"},\"message\":\"Create pagespeed-all-settings.add\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nomadturk/nginx-conf/commits/035679b2f50ea9d689bba803906d5b1ee3418062\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:53Z\"}\n{\"id\":\"2489398744\",\"type\":\"PushEvent\",\"actor\":{\"id\":3105201,\"login\":\"beh68030\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/beh68030\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3105201?\"},\"repo\":{\"id\":22865608,\"name\":\"beh68030/twitch_lookup\",\"url\":\"https://api.github.com/repos/beh68030/twitch_lookup\"},\"payload\":{\"push_id\":536753495,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2e8582db72d4c53a2627aa9fb700aa56c9689bf9\",\"before\":\"18fabca10abfd73c080a73db05759fa14d98dfa6\",\"commits\":[{\"sha\":\"2e8582db72d4c53a2627aa9fb700aa56c9689bf9\",\"author\":{\"email\":\"3e061f68f73b3319b5773c9f6054f99a9369e4cb@users.noreply.github.com\",\"name\":\"beh68030\"},\"message\":\"Update search.lisp\",\"distinct\":true,\"url\":\"https://api.github.com/repos/beh68030/twitch_lookup/commits/2e8582db72d4c53a2627aa9fb700aa56c9689bf9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:53Z\"}\n{\"id\":\"2489398750\",\"type\":\"PushEvent\",\"actor\":{\"id\":32421,\"login\":\"damonallison\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/damonallison\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/32421?\"},\"repo\":{\"id\":28276966,\"name\":\"damonallison/git-test\",\"url\":\"https://api.github.com/repos/damonallison/git-test\"},\"payload\":{\"push_id\":536753497,\"size\":5,\"distinct_size\":4,\"ref\":\"refs/heads/damon2\",\"head\":\"3e58f93effeb21baf088c66ac7ba4db6565f832e\",\"before\":\"096de29c8100f5a960060052b04f5ef4902355cd\",\"commits\":[{\"sha\":\"0cd38185bffd3af3513d0763ced55fb8916f6072\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"damon\",\"distinct\":true,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/0cd38185bffd3af3513d0763ced55fb8916f6072\"},{\"sha\":\"a0241b998058d49fe5ec6b1732d32f4269d4de30\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"master\",\"distinct\":false,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/a0241b998058d49fe5ec6b1732d32f4269d4de30\"},{\"sha\":\"eb2b6231b59cda41629ce12b4750d55e5946b6d5\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"more on damon2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/eb2b6231b59cda41629ce12b4750d55e5946b6d5\"},{\"sha\":\"df69472afe16ac49d94618598ae40f9bde0b50b9\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"more(2) on damon2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/df69472afe16ac49d94618598ae40f9bde0b50b9\"},{\"sha\":\"3e58f93effeb21baf088c66ac7ba4db6565f832e\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"Merge branch 'master' into damon2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/3e58f93effeb21baf088c66ac7ba4db6565f832e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:53Z\"}\n{\"id\":\"2489398752\",\"type\":\"PushEvent\",\"actor\":{\"id\":9271233,\"login\":\"Nexusty\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Nexusty\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9271233?\"},\"repo\":{\"id\":28494954,\"name\":\"Nexusty/open_exoplanet_catalogue\",\"url\":\"https://api.github.com/repos/Nexusty/open_exoplanet_catalogue\"},\"payload\":{\"push_id\":536753498,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"6fde6be1e4178a9268ad735ca4bc12edb9f66290\",\"before\":\"1c76a1af7971f4fcc693b147bbf70c75d7352617\",\"commits\":[{\"sha\":\"da3f398815f136e62b18be313e10dc5317d27a72\",\"author\":{\"email\":\"898dba66b7f38fdb797b262b137097fc973d6485@hanno-rein.de\",\"name\":\"Hanno Rein\"},\"message\":\"Merge pull request #434 from Nexusty/master\\n\\nWASP-84 b\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Nexusty/open_exoplanet_catalogue/commits/da3f398815f136e62b18be313e10dc5317d27a72\"},{\"sha\":\"ab6d6080cd03d5aaa863bfba2944f7e006b1e186\",\"author\":{\"email\":\"7108125947c1f9f50408652b4c345d19fd23e566@users.noreply.github.com\",\"name\":\"Christian Sturm\"},\"message\":\"Added hot Jupiter\\n\\nCloses #435\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Nexusty/open_exoplanet_catalogue/commits/ab6d6080cd03d5aaa863bfba2944f7e006b1e186\"},{\"sha\":\"6fde6be1e4178a9268ad735ca4bc12edb9f66290\",\"author\":{\"email\":\"898dba66b7f38fdb797b262b137097fc973d6485@hanno-rein.de\",\"name\":\"Hanno Rein\"},\"message\":\"Merge pull request #436 from Sol-d/patch-6\\n\\nAdded hot Jupiter\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Nexusty/open_exoplanet_catalogue/commits/6fde6be1e4178a9268ad735ca4bc12edb9f66290\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:53Z\"}\n{\"id\":\"2489398760\",\"type\":\"PushEvent\",\"actor\":{\"id\":660054,\"login\":\"MarkAYoder\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MarkAYoder\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/660054?\"},\"repo\":{\"id\":5111513,\"name\":\"MarkAYoder/BeagleBoard-exercises\",\"url\":\"https://api.github.com/repos/MarkAYoder/BeagleBoard-exercises\"},\"payload\":{\"push_id\":536753503,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"ab3d6b980de5ba3c23f61a4645efb701fbbde1a2\",\"before\":\"ebc1e32f6e6e373f1d46fa4d0b532eba4ec7c497\",\"commits\":[{\"sha\":\"ab3d6b980de5ba3c23f61a4645efb701fbbde1a2\",\"author\":{\"email\":\"9068e990bc2a9df3bc663b2b11b19718c2490ef5@Rose-Hulman.edu\",\"name\":\"Mark A. Yoder\"},\"message\":\"Added STRING_LEN and color-names\",\"distinct\":true,\"url\":\"https://api.github.com/repos/MarkAYoder/BeagleBoard-exercises/commits/ab3d6b980de5ba3c23f61a4645efb701fbbde1a2\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:55Z\"}\n{\"id\":\"2489398762\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753504,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2250e5e1294fae3c21d3fa9b19bd162c40c29a6c\",\"before\":\"bd9e3811383e5c3fc3bbd2b9ddd90047df70e7c5\",\"commits\":[{\"sha\":\"2250e5e1294fae3c21d3fa9b19bd162c40c29a6c\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"always make dynamic views optional\\n\\ninstead of throwing an error when a dynamic lookup fails and optional\\nis not specified, just always render a blank template\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/2250e5e1294fae3c21d3fa9b19bd162c40c29a6c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:55Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398767\",\"type\":\"PushEvent\",\"actor\":{\"id\":1308363,\"login\":\"paymonp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paymonp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1308363?\"},\"repo\":{\"id\":28678242,\"name\":\"paymonp/forecast_wrapper\",\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper\"},\"payload\":{\"push_id\":536753505,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cfa67c6a84faaa2bddc78c69075b5d1c9ebb42fd\",\"before\":\"9b9757b748dd98f06c2517a9af376e7515716a6a\",\"commits\":[{\"sha\":\"cfa67c6a84faaa2bddc78c69075b5d1c9ebb42fd\",\"author\":{\"email\":\"fc8885f78a23392efbab2637290a95a3fe38d9fa@team.curious.com\",\"name\":\"Paymon\"},\"message\":\"Update README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/paymonp/forecast_wrapper/commits/cfa67c6a84faaa2bddc78c69075b5d1c9ebb42fd\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:57Z\"}\n{\"id\":\"2489398768\",\"type\":\"CreateEvent\",\"actor\":{\"id\":10214932,\"login\":\"jplarida\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jplarida\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10214932?\"},\"repo\":{\"id\":28678286,\"name\":\"jplarida/testgit\",\"url\":\"https://api.github.com/repos/jplarida/testgit\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"this is a test\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:06:58Z\"}\n{\"id\":\"2489398769\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":1737635,\"login\":\"waltervr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/waltervr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1737635?\"},\"repo\":{\"id\":28369326,\"name\":\"waltervr/mejengol\",\"url\":\"https://api.github.com/repos/waltervr/mejengol\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/waltervr/mejengol/issues/6\",\"labels_url\":\"https://api.github.com/repos/waltervr/mejengol/issues/6/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/waltervr/mejengol/issues/6/comments\",\"events_url\":\"https://api.github.com/repos/waltervr/mejengol/issues/6/events\",\"html_url\":\"https://github.com/waltervr/mejengol/issues/6\",\"id\":53210298,\"number\":6,\"title\":\"Use a single button for start/stop chronometer.\",\"user\":{\"login\":\"waltervr\",\"id\":1737635,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1737635?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/waltervr\",\"html_url\":\"https://github.com/waltervr\",\"followers_url\":\"https://api.github.com/users/waltervr/followers\",\"following_url\":\"https://api.github.com/users/waltervr/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/waltervr/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/waltervr/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/waltervr/subscriptions\",\"organizations_url\":\"https://api.github.com/users/waltervr/orgs\",\"repos_url\":\"https://api.github.com/users/waltervr/repos\",\"events_url\":\"https://api.github.com/users/waltervr/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/waltervr/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:06:57Z\",\"updated_at\":\"2015-01-01T01:06:57Z\",\"closed_at\":null,\"body\":\"Use a single button for start/stop chronometer.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:06:58Z\"}\n{\"id\":\"2489398771\",\"type\":\"PushEvent\",\"actor\":{\"id\":777062,\"login\":\"MikeLorenz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MikeLorenz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/777062?\"},\"repo\":{\"id\":28676253,\"name\":\"MikeLorenz/Spoon-Knife\",\"url\":\"https://api.github.com/repos/MikeLorenz/Spoon-Knife\"},\"payload\":{\"push_id\":536753506,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5e43e91af613860fc38ce4e2f77272bfd60cc143\",\"before\":\"d0dd1f61b33d64e29d8bc1372a94ef6a2fee76a9\",\"commits\":[{\"sha\":\"5e43e91af613860fc38ce4e2f77272bfd60cc143\",\"author\":{\"email\":\"a17fed27eaa842282862ff7c1b9c8395a26ac320@mikelorenz.com\",\"name\":\"Mike Lorenz\"},\"message\":\"Changed the outgoing text.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/MikeLorenz/Spoon-Knife/commits/5e43e91af613860fc38ce4e2f77272bfd60cc143\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:58Z\"}\n{\"id\":\"2489398778\",\"type\":\"PushEvent\",\"actor\":{\"id\":8683432,\"login\":\"wkcool\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wkcool\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8683432?\"},\"repo\":{\"id\":27059641,\"name\":\"wkcool/wkcool.github.com\",\"url\":\"https://api.github.com/repos/wkcool/wkcool.github.com\"},\"payload\":{\"push_id\":536753510,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"9f6584f5fa96346a20b148f340fbfd055dd70e7e\",\"before\":\"48d6c11b675a084cfa0321b7b14c00241e423044\",\"commits\":[{\"sha\":\"9f6584f5fa96346a20b148f340fbfd055dd70e7e\",\"author\":{\"email\":\"ddd2d39a843d8ae129b2e2fc6f13669f44116bd1@gmail.com\",\"name\":\"wenkrcool\"},\"message\":\"提交\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wkcool/wkcool.github.com/commits/9f6584f5fa96346a20b148f340fbfd055dd70e7e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:06:59Z\"}\n{\"id\":\"2489398785\",\"type\":\"PushEvent\",\"actor\":{\"id\":1714830,\"login\":\"davidsavagejr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/davidsavagejr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1714830?\"},\"repo\":{\"id\":28677976,\"name\":\"davidsavagejr/NSBTransactionPerformance\",\"url\":\"https://api.github.com/repos/davidsavagejr/NSBTransactionPerformance\"},\"payload\":{\"push_id\":536753515,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"142bcdb815e594a389be04a10fa12465d9de7699\",\"before\":\"a15ad161b2efd7821dadfe429d641b349ebeb65f\",\"commits\":[{\"sha\":\"142bcdb815e594a389be04a10fa12465d9de7699\",\"author\":{\"email\":\"aa743a0aaec8f7d7a1f01442503957f4d7a2d634@headspring.com\",\"name\":\"David Savage\"},\"message\":\"Couple cleanup items and added sql file\",\"distinct\":true,\"url\":\"https://api.github.com/repos/davidsavagejr/NSBTransactionPerformance/commits/142bcdb815e594a389be04a10fa12465d9de7699\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:00Z\"}\n{\"id\":\"2489398787\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":10238253,\"login\":\"DasCode\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DasCode\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10238253?\"},\"repo\":{\"id\":24246283,\"name\":\"trydis/FIFA-Ultimate-Team-2015-Toolkit\",\"url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73\",\"labels_url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73/comments\",\"events_url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73/events\",\"html_url\":\"https://github.com/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73\",\"id\":47828562,\"number\":73,\"title\":\"[Question] Check in Watchlist, if I'm overbid?\",\"user\":{\"login\":\"Taggardos\",\"id\":7068808,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7068808?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Taggardos\",\"html_url\":\"https://github.com/Taggardos\",\"followers_url\":\"https://api.github.com/users/Taggardos/followers\",\"following_url\":\"https://api.github.com/users/Taggardos/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Taggardos/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Taggardos/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Taggardos/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Taggardos/orgs\",\"repos_url\":\"https://api.github.com/users/Taggardos/repos\",\"events_url\":\"https://api.github.com/users/Taggardos/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Taggardos/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":1,\"created_at\":\"2014-11-05T12:00:50Z\",\"updated_at\":\"2015-01-01T01:07:00Z\",\"closed_at\":null,\"body\":\"Hello,\\r\\nthe Topic says it all?\\r\\nIs it possible to check, if I am overbidded by someone else with the Toolkit?\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/comments/68477369\",\"html_url\":\"https://github.com/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73#issuecomment-68477369\",\"issue_url\":\"https://api.github.com/repos/trydis/FIFA-Ultimate-Team-2015-Toolkit/issues/73\",\"id\":68477369,\"user\":{\"login\":\"DasCode\",\"id\":10238253,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10238253?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/DasCode\",\"html_url\":\"https://github.com/DasCode\",\"followers_url\":\"https://api.github.com/users/DasCode/followers\",\"following_url\":\"https://api.github.com/users/DasCode/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/DasCode/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/DasCode/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/DasCode/subscriptions\",\"organizations_url\":\"https://api.github.com/users/DasCode/orgs\",\"repos_url\":\"https://api.github.com/users/DasCode/repos\",\"events_url\":\"https://api.github.com/users/DasCode/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/DasCode/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:00Z\",\"updated_at\":\"2015-01-01T01:07:00Z\",\"body\":\"As with anything, try it with the real web app, observe what happens in fiddler, and then use that to figure it out.\\r\\nIn this case if you bidState is \\\"outbid\\\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:00Z\"}\n{\"id\":\"2489398789\",\"type\":\"CreateEvent\",\"actor\":{\"id\":804014,\"login\":\"ry5n\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ry5n\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/804014?\"},\"repo\":{\"id\":28678287,\"name\":\"ry5n/sass-inline-svg\",\"url\":\"https://api.github.com/repos/ry5n/sass-inline-svg\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Inline an SVG as a CSS data URI. Allows recoloring paths.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:00Z\"}\n{\"id\":\"2489398793\",\"type\":\"WatchEvent\",\"actor\":{\"id\":5554477,\"login\":\"errolgrannum\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/errolgrannum\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5554477?\"},\"repo\":{\"id\":8845569,\"name\":\"adamw523/simple_pyobjc_cocoa_xib\",\"url\":\"https://api.github.com/repos/adamw523/simple_pyobjc_cocoa_xib\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:01Z\"}\n{\"id\":\"2489398795\",\"type\":\"PushEvent\",\"actor\":{\"id\":3259988,\"login\":\"Chaldron\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Chaldron\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3259988?\"},\"repo\":{\"id\":28620299,\"name\":\"Chaldron/chaldron.github.io\",\"url\":\"https://api.github.com/repos/Chaldron/chaldron.github.io\"},\"payload\":{\"push_id\":536753517,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"95f51364ae4ffa6d8c05ac103f6673b582aee33f\",\"before\":\"bbae0256bd9a39b68421f3b8f57e1d1f2c82e44a\",\"commits\":[{\"sha\":\"95f51364ae4ffa6d8c05ac103f6673b582aee33f\",\"author\":{\"email\":\"201d29e73a9d86ccd769ab841a6a1fabfaeec39b@me.com\",\"name\":\"Aditya\"},\"message\":\"Replace master branch with page content via GitHub\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Chaldron/chaldron.github.io/commits/95f51364ae4ffa6d8c05ac103f6673b582aee33f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:01Z\"}\n{\"id\":\"2489398800\",\"type\":\"PushEvent\",\"actor\":{\"id\":5728403,\"login\":\"patrick-hudson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/patrick-hudson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5728403?\"},\"repo\":{\"id\":25392255,\"name\":\"patrick-hudson/EggDrop\",\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop\"},\"payload\":{\"push_id\":536753519,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0976103a68d57ef5dfd23ce0b13166ed29505f57\",\"before\":\"84116936c4f6805f56a1e643eb9898ebdc854334\",\"commits\":[{\"sha\":\"0976103a68d57ef5dfd23ce0b13166ed29505f57\",\"author\":{\"email\":\"cbb7353e6d953ef360baf960c122346276c6e320@hudson.bz\",\"name\":\"Patrick Hudson\"},\"message\":\"Scripted auto-commit on change (2014-12-31 20:07:00) by gitwatch.sh\",\"distinct\":true,\"url\":\"https://api.github.com/repos/patrick-hudson/EggDrop/commits/0976103a68d57ef5dfd23ce0b13166ed29505f57\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:02Z\"}\n{\"id\":\"2489398801\",\"type\":\"WatchEvent\",\"actor\":{\"id\":7682325,\"login\":\"humanrocker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/humanrocker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7682325?\"},\"repo\":{\"id\":168370,\"name\":\"jruby/jruby\",\"url\":\"https://api.github.com/repos/jruby/jruby\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:02Z\",\"org\":{\"id\":55687,\"login\":\"jruby\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/jruby\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/55687?\"}}\n{\"id\":\"2489398802\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2357415,\"login\":\"neeleshbagga\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/neeleshbagga\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2357415?\"},\"repo\":{\"id\":28678288,\"name\":\"neeleshbagga/CS2110QA\",\"url\":\"https://api.github.com/repos/neeleshbagga/CS2110QA\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"An interactive Q&A tool built in Java\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:02Z\"}\n{\"id\":\"2489398803\",\"type\":\"PushEvent\",\"actor\":{\"id\":8965687,\"login\":\"Gawainus\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Gawainus\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8965687?\"},\"repo\":{\"id\":27628356,\"name\":\"Gawainus/Systems\",\"url\":\"https://api.github.com/repos/Gawainus/Systems\"},\"payload\":{\"push_id\":536753520,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fd9c66e6d94ab09f42e0986e8eecf075dc637f82\",\"before\":\"04808fff427dd5bc90f3da6faa7abd58a3975139\",\"commits\":[{\"sha\":\"fd9c66e6d94ab09f42e0986e8eecf075dc637f82\",\"author\":{\"email\":\"95ac278ea19f44ce11e50f5f7e263294f31b7bb2@gmail.com\",\"name\":\"Yumen Cao\"},\"message\":\"Malloc added\",\"distinct\":true,\"url\":\"https://api.github.com/repos/Gawainus/Systems/commits/fd9c66e6d94ab09f42e0986e8eecf075dc637f82\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:02Z\"}\n{\"id\":\"2489398804\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1043930,\"login\":\"hsw0\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hsw0\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1043930?\"},\"repo\":{\"id\":330414,\"name\":\"h2o/picohttpparser\",\"url\":\"https://api.github.com/repos/h2o/picohttpparser\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:03Z\",\"org\":{\"id\":9275116,\"login\":\"h2o\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/h2o\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9275116?\"}}\n{\"id\":\"2489398809\",\"type\":\"PushEvent\",\"actor\":{\"id\":6955935,\"login\":\"wfrizzell\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wfrizzell\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6955935?\"},\"repo\":{\"id\":28678275,\"name\":\"wfrizzell/Completed-Target-Model-Table-Documentation\",\"url\":\"https://api.github.com/repos/wfrizzell/Completed-Target-Model-Table-Documentation\"},\"payload\":{\"push_id\":536753521,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d030634f4bd8d8dc6d305cbe2afd05acefdacbd0\",\"before\":\"265ddc64dff3dbbdaa2ce6021056a64b95b36c06\",\"commits\":[{\"sha\":\"d030634f4bd8d8dc6d305cbe2afd05acefdacbd0\",\"author\":{\"email\":\"e965d2ed358da94c7eb620090099fbbbcd752703@mphasis.com\",\"name\":\"Bill Frizzell\"},\"message\":\"Target Model Entity Documentation\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wfrizzell/Completed-Target-Model-Table-Documentation/commits/d030634f4bd8d8dc6d305cbe2afd05acefdacbd0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:03Z\"}\n{\"id\":\"2489398811\",\"type\":\"PushEvent\",\"actor\":{\"id\":19571,\"login\":\"lizadaly\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lizadaly\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/19571?\"},\"repo\":{\"id\":28673562,\"name\":\"lizadaly/surrealisme\",\"url\":\"https://api.github.com/repos/lizadaly/surrealisme\"},\"payload\":{\"push_id\":536753523,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"748cde8726a87a0ca8102578e8bf0e1380b5bc92\",\"before\":\"5bd3db7045dc3b1a231978c8b0ba17438a05436d\",\"commits\":[{\"sha\":\"748cde8726a87a0ca8102578e8bf0e1380b5bc92\",\"author\":{\"email\":\"7e31b076c038116b711593886947b7ee669cdd6a@safaribooksonline.com\",\"name\":\"Liza Daly\"},\"message\":\"Some code\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lizadaly/surrealisme/commits/748cde8726a87a0ca8102578e8bf0e1380b5bc92\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:03Z\"}\n{\"id\":\"2489398815\",\"type\":\"PushEvent\",\"actor\":{\"id\":4379694,\"login\":\"moongato\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/moongato\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4379694?\"},\"repo\":{\"id\":11769101,\"name\":\"moongato/android_packages_apps_settings\",\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings\"},\"payload\":{\"push_id\":536753526,\"size\":3,\"distinct_size\":1,\"ref\":\"refs/heads/lp50x\",\"head\":\"a9d6ccf56bd0f57c043d30e57bf4aa9dde0f27ad\",\"before\":\"56646a94c0aedc577d5db224058fb9be4c927786\",\"commits\":[{\"sha\":\"e53c63b3254c37b69e9fcf05b69a1b2fb7f1858c\",\"author\":{\"email\":\"0fdcc473619717889ce3a7389c2487333487bab5@gmail.com\",\"name\":\"rascarlo\"},\"message\":\"volume rocker music controls\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/e53c63b3254c37b69e9fcf05b69a1b2fb7f1858c\"},{\"sha\":\"bebb75fdaa4a085c814ea6d2551899593454440d\",\"author\":{\"email\":\"0fdcc473619717889ce3a7389c2487333487bab5@gmail.com\",\"name\":\"rascarlo\"},\"message\":\"status bar brightness control edits.\\n\\nremove check for SCREEN_BRIGHTNESS_MODE_AUTOMATIC\",\"distinct\":false,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/bebb75fdaa4a085c814ea6d2551899593454440d\"},{\"sha\":\"a9d6ccf56bd0f57c043d30e57bf4aa9dde0f27ad\",\"author\":{\"email\":\"22e0f38e0fc64da9129ff9b9ef030b39415294a1@ubuntu\",\"name\":\"moongato\"},\"message\":\"Merge remote-tracking branch 'upstream/lollipop-ras-mr1' into lp50x\",\"distinct\":true,\"url\":\"https://api.github.com/repos/moongato/android_packages_apps_settings/commits/a9d6ccf56bd0f57c043d30e57bf4aa9dde0f27ad\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:04Z\"}\n{\"id\":\"2489398818\",\"type\":\"PushEvent\",\"actor\":{\"id\":3964764,\"login\":\"elliekimpot\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/elliekimpot\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3964764?\"},\"repo\":{\"id\":28625867,\"name\":\"elliekimpot/msm\",\"url\":\"https://api.github.com/repos/elliekimpot/msm\"},\"payload\":{\"push_id\":536753528,\"size\":7,\"distinct_size\":0,\"ref\":\"refs/heads/base\",\"head\":\"6393887f2296f3e7d7820ceefa0177003261ec68\",\"before\":\"0011b847c3b6a0082306b3b29077bdc9550c6dda\",\"commits\":[{\"sha\":\"8facc7f43ff2a8292656bf436f369b9455c1ab3a\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_PS_WIFI_COM_PREF_LOGGING\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/8facc7f43ff2a8292656bf436f369b9455c1ab3a\"},{\"sha\":\"9bfea860d1057c466a88f9a78bf28bf11b0fb221\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_SELINUX_DENIAL_LOG\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/9bfea860d1057c466a88f9a78bf28bf11b0fb221\"},{\"sha\":\"8ae5cba2d0c6e13cdbb3682fa7cc70960cd95246\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_MORE_DEBUGGING_INFO_ON_KERNEL\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/8ae5cba2d0c6e13cdbb3682fa7cc70960cd95246\"},{\"sha\":\"1fae68014e73cfc3a32fa5eca33505b011ebb7b4\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"Revert \\\"add basic CFLAGS for Krait\\\"\\n\\nThis reverts commit 0011b847c3b6a0082306b3b29077bdc9550c6dda.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/1fae68014e73cfc3a32fa5eca33505b011ebb7b4\"},{\"sha\":\"ff109d56f2b1b87aada770208ba471ef9d085a70\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_DEBUG\\n\\nDrop following features\\n* PANTECH_DEBUG_ON\\n* PANTECH_DEBUG\\n* PANTECH_DEBUG_SCHED_LOG\\n* PANTECH_DEBUG_IRQ_LOG\\n* PANTECH_DEBUG_DCVS_LOG\\n* PANTECH_DEBUG_RPM_LOG\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/ff109d56f2b1b87aada770208ba471ef9d085a70\"},{\"sha\":\"c3d59ecafe5fda8c6776139e45c6c76eecbf046d\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_FS_AUTO_REPAIR\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/c3d59ecafe5fda8c6776139e45c6c76eecbf046d\"},{\"sha\":\"6393887f2296f3e7d7820ceefa0177003261ec68\",\"author\":{\"email\":\"c6c610d4ed4672366596341ac49b62c9256ebf55@gmail.com\",\"name\":\"Ellie\"},\"message\":\"pantech/debug: Drop PANTECH_ERR_CRASH_LOGGING\\n\\nDrop PANTECH_ERR_CRASH_LOGGING and introduce PANTECH_SYS\\ndue to proper build (Solve error/mipi_sony_incell.c).\\n\\nSigned-off-by: Ellie <elliekimpot@gmail.com>\",\"distinct\":false,\"url\":\"https://api.github.com/repos/elliekimpot/msm/commits/6393887f2296f3e7d7820ceefa0177003261ec68\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:05Z\"}\n{\"id\":\"2489398830\",\"type\":\"PushEvent\",\"actor\":{\"id\":4438295,\"login\":\"ezzye\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ezzye\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4438295?\"},\"repo\":{\"id\":28522532,\"name\":\"ezzye/mean_example1\",\"url\":\"https://api.github.com/repos/ezzye/mean_example1\"},\"payload\":{\"push_id\":536753530,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"429b580df6f07fa1d998c1561bfe07ce99c2ab70\",\"before\":\"71b3fb787d28aab581c80fa04e8330d0ea965c75\",\"commits\":[{\"sha\":\"429b580df6f07fa1d998c1561bfe07ce99c2ab70\",\"author\":{\"email\":\"60004a400d96d28036d21c59f517f0e3e3a6150f@gmail.com\",\"name\":\"ezzye\"},\"message\":\"sort entires by dates\",\"distinct\":true,\"url\":\"https://api.github.com/repos/ezzye/mean_example1/commits/429b580df6f07fa1d998c1561bfe07ce99c2ab70\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\"}\n{\"id\":\"2489398831\",\"type\":\"PushEvent\",\"actor\":{\"id\":201997,\"login\":\"gridaphobe\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/gridaphobe\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/201997?\"},\"repo\":{\"id\":16334926,\"name\":\"gridaphobe/target\",\"url\":\"https://api.github.com/repos/gridaphobe/target\"},\"payload\":{\"push_id\":536753531,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"25d0127dba9df025f3f7c927abf018118b072816\",\"before\":\"7da2460eba860a739db8aa61898015081bb1f77c\",\"commits\":[{\"sha\":\"25d0127dba9df025f3f7c927abf018118b072816\",\"author\":{\"email\":\"7c47fb8c5e263626481b146af46f62cf5772922e@gmail.com\",\"name\":\"Eric Seidel\"},\"message\":\"a bit more docs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gridaphobe/target/commits/25d0127dba9df025f3f7c927abf018118b072816\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\"}\n{\"id\":\"2489398834\",\"type\":\"PushEvent\",\"actor\":{\"id\":447569,\"login\":\"tom-henderson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tom-henderson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/447569?\"},\"repo\":{\"id\":23275671,\"name\":\"tom-henderson/django-template\",\"url\":\"https://api.github.com/repos/tom-henderson/django-template\"},\"payload\":{\"push_id\":536753533,\"size\":3,\"distinct_size\":3,\"ref\":\"refs/heads/master\",\"head\":\"8dc1ec4089bb639884f24fd4a5f7e7693099a5d9\",\"before\":\"5e058c0e862a71ad83228c96c56bae766cbd8f4b\",\"commits\":[{\"sha\":\"4ebf23c0aefef609187114e9a18b88b15ccac618\",\"author\":{\"email\":\"06d851666d407f85c6c32c8197898614a9ec61fa@mac.com\",\"name\":\"Tom Henderson\"},\"message\":\"Ignore .sqlite3 files.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tom-henderson/django-template/commits/4ebf23c0aefef609187114e9a18b88b15ccac618\"},{\"sha\":\"8cd6c73216a0aa3d3377ed32827c3a9bf2b51c1c\",\"author\":{\"email\":\"06d851666d407f85c6c32c8197898614a9ec61fa@mac.com\",\"name\":\"Tom Henderson\"},\"message\":\"Support collectstatic.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tom-henderson/django-template/commits/8cd6c73216a0aa3d3377ed32827c3a9bf2b51c1c\"},{\"sha\":\"8dc1ec4089bb639884f24fd4a5f7e7693099a5d9\",\"author\":{\"email\":\"06d851666d407f85c6c32c8197898614a9ec61fa@mac.com\",\"name\":\"Tom Henderson\"},\"message\":\"Define THIRD_PARTY_APPS separtely from LOCAL and CORE.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tom-henderson/django-template/commits/8dc1ec4089bb639884f24fd4a5f7e7693099a5d9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\"}\n{\"id\":\"2489398836\",\"type\":\"PushEvent\",\"actor\":{\"id\":32421,\"login\":\"damonallison\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/damonallison\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/32421?\"},\"repo\":{\"id\":28276966,\"name\":\"damonallison/git-test\",\"url\":\"https://api.github.com/repos/damonallison/git-test\"},\"payload\":{\"push_id\":536753535,\"size\":4,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"3e58f93effeb21baf088c66ac7ba4db6565f832e\",\"before\":\"a0241b998058d49fe5ec6b1732d32f4269d4de30\",\"commits\":[{\"sha\":\"0cd38185bffd3af3513d0763ced55fb8916f6072\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"damon\",\"distinct\":false,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/0cd38185bffd3af3513d0763ced55fb8916f6072\"},{\"sha\":\"eb2b6231b59cda41629ce12b4750d55e5946b6d5\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"more on damon2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/eb2b6231b59cda41629ce12b4750d55e5946b6d5\"},{\"sha\":\"df69472afe16ac49d94618598ae40f9bde0b50b9\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"more(2) on damon2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/df69472afe16ac49d94618598ae40f9bde0b50b9\"},{\"sha\":\"3e58f93effeb21baf088c66ac7ba4db6565f832e\",\"author\":{\"email\":\"a836250b7333a571ccf7c31c8a50a83778615420@code42.com\",\"name\":\"Damon Allison\"},\"message\":\"Merge branch 'master' into damon2\",\"distinct\":false,\"url\":\"https://api.github.com/repos/damonallison/git-test/commits/3e58f93effeb21baf088c66ac7ba4db6565f832e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\"}\n{\"id\":\"2489398837\",\"type\":\"PushEvent\",\"actor\":{\"id\":10144074,\"login\":\"carodew\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/carodew\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10144074?\"},\"repo\":{\"id\":27844858,\"name\":\"carodew/carodew.github.io\",\"url\":\"https://api.github.com/repos/carodew/carodew.github.io\"},\"payload\":{\"push_id\":536753536,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"01e87433331ac75e4e42724f13368f9178b3a44e\",\"before\":\"62bd3e248d57b48720930200d73b4273993a2e66\",\"commits\":[{\"sha\":\"01e87433331ac75e4e42724f13368f9178b3a44e\",\"author\":{\"email\":\"6e3c6f0214740e9061d9ca5c79eb6e0ff9cc1741@unknown542696dd77af.gateway.pace.com\",\"name\":\"Carolyn\"},\"message\":\"update h2 styles on homepage\",\"distinct\":true,\"url\":\"https://api.github.com/repos/carodew/carodew.github.io/commits/01e87433331ac75e4e42724f13368f9178b3a44e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\"}\n{\"id\":\"2489398838\",\"type\":\"WatchEvent\",\"actor\":{\"id\":4433943,\"login\":\"jkrmr\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jkrmr\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4433943?\"},\"repo\":{\"id\":19797857,\"name\":\"rsense/rsense\",\"url\":\"https://api.github.com/repos/rsense/rsense\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:07Z\",\"org\":{\"id\":7576243,\"login\":\"rsense\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/rsense\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7576243?\"}}\n{\"id\":\"2489398840\",\"type\":\"CreateEvent\",\"actor\":{\"id\":932999,\"login\":\"loyos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/loyos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/932999?\"},\"repo\":{\"id\":28678289,\"name\":\"loyos/cemento\",\"url\":\"https://api.github.com/repos/loyos/cemento\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:08Z\"}\n{\"id\":\"2489398845\",\"type\":\"CreateEvent\",\"actor\":{\"id\":2357415,\"login\":\"neeleshbagga\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/neeleshbagga\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2357415?\"},\"repo\":{\"id\":28678288,\"name\":\"neeleshbagga/CS2110QA\",\"url\":\"https://api.github.com/repos/neeleshbagga/CS2110QA\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"An interactive Q&A tool built in Java\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:09Z\"}\n{\"id\":\"2489398851\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":5404873,\"login\":\"RuslanIsrafilov\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RuslanIsrafilov\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5404873?\"},\"repo\":{\"id\":24333720,\"name\":\"antonlarin/aNiMated-batman\",\"url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/41\",\"labels_url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/41/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/41/comments\",\"events_url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/41/events\",\"html_url\":\"https://github.com/antonlarin/aNiMated-batman/issues/41\",\"id\":53209423,\"number\":41,\"title\":\"Не работает изменение режима счёта (все слои/счёт на установление)\",\"user\":{\"login\":\"RuslanIsrafilov\",\"id\":5404873,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5404873?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RuslanIsrafilov\",\"html_url\":\"https://github.com/RuslanIsrafilov\",\"followers_url\":\"https://api.github.com/users/RuslanIsrafilov/followers\",\"following_url\":\"https://api.github.com/users/RuslanIsrafilov/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/RuslanIsrafilov/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/RuslanIsrafilov/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/RuslanIsrafilov/subscriptions\",\"organizations_url\":\"https://api.github.com/users/RuslanIsrafilov/orgs\",\"repos_url\":\"https://api.github.com/users/RuslanIsrafilov/repos\",\"events_url\":\"https://api.github.com/users/RuslanIsrafilov/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/RuslanIsrafilov/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/labels/Bug\",\"name\":\"Bug\",\"color\":\"eb6420\"}],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"antonlarin\",\"id\":1180097,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1180097?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/antonlarin\",\"html_url\":\"https://github.com/antonlarin\",\"followers_url\":\"https://api.github.com/users/antonlarin/followers\",\"following_url\":\"https://api.github.com/users/antonlarin/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/antonlarin/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/antonlarin/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/antonlarin/subscriptions\",\"organizations_url\":\"https://api.github.com/users/antonlarin/orgs\",\"repos_url\":\"https://api.github.com/users/antonlarin/repos\",\"events_url\":\"https://api.github.com/users/antonlarin/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/antonlarin/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":1,\"created_at\":\"2015-01-01T00:19:34Z\",\"updated_at\":\"2015-01-01T01:07:10Z\",\"closed_at\":null,\"body\":\"Проблема следующая.\\r\\nПри выбранной явной схеме нажимаю \\\"Вычисление всех слоёв\\\", \\\"Счёт на установление\\\" - работает всё как надо. Потом выбираю неявную схему и запускаю тоже самое, в результате работает либо \\\"Вычисление всех слоёв\\\" либо \\\"Счёт на установление\\\", в независимости от нажатой кнопки.\"},\"comment\":{\"url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/comments/68477373\",\"html_url\":\"https://github.com/antonlarin/aNiMated-batman/issues/41#issuecomment-68477373\",\"issue_url\":\"https://api.github.com/repos/antonlarin/aNiMated-batman/issues/41\",\"id\":68477373,\"user\":{\"login\":\"RuslanIsrafilov\",\"id\":5404873,\"avatar_url\":\"https://avatars.githubusercontent.com/u/5404873?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/RuslanIsrafilov\",\"html_url\":\"https://github.com/RuslanIsrafilov\",\"followers_url\":\"https://api.github.com/users/RuslanIsrafilov/followers\",\"following_url\":\"https://api.github.com/users/RuslanIsrafilov/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/RuslanIsrafilov/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/RuslanIsrafilov/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/RuslanIsrafilov/subscriptions\",\"organizations_url\":\"https://api.github.com/users/RuslanIsrafilov/orgs\",\"repos_url\":\"https://api.github.com/users/RuslanIsrafilov/repos\",\"events_url\":\"https://api.github.com/users/RuslanIsrafilov/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/RuslanIsrafilov/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:10Z\",\"updated_at\":\"2015-01-01T01:07:10Z\",\"body\":\"Сам нашёл, сам пофиксил - всё правильно.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:10Z\"}\n{\"id\":\"2489398852\",\"type\":\"PushEvent\",\"actor\":{\"id\":109048,\"login\":\"bestian\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bestian\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/109048?\"},\"repo\":{\"id\":27702247,\"name\":\"g0v/goban\",\"url\":\"https://api.github.com/repos/g0v/goban\"},\"payload\":{\"push_id\":536753540,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/gh-pages\",\"head\":\"7ced4bb306188a4505e541d1f9d198116bf1f7e6\",\"before\":\"42001fdc8ab4af516c60229fbacffa082215f195\",\"commits\":[{\"sha\":\"7ced4bb306188a4505e541d1f9d198116bf1f7e6\",\"author\":{\"email\":\"295c90327d3df30c5845123db2f48facf2356138@gmail.com\",\"name\":\"Bestian Tang\"},\"message\":\"responsive init & sass partialized init\",\"distinct\":true,\"url\":\"https://api.github.com/repos/g0v/goban/commits/7ced4bb306188a4505e541d1f9d198116bf1f7e6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:10Z\",\"org\":{\"id\":2668086,\"login\":\"g0v\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/g0v\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2668086?\"}}\n{\"id\":\"2489398855\",\"type\":\"CreateEvent\",\"actor\":{\"id\":4193711,\"login\":\"CorruptingAcid\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/CorruptingAcid\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4193711?\"},\"repo\":{\"id\":28678290,\"name\":\"CorruptingAcid/UltimatePnP\",\"url\":\"https://api.github.com/repos/CorruptingAcid/UltimatePnP\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"Table top RPG assistant\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:13Z\"}\n{\"id\":\"2489398859\",\"type\":\"PushEvent\",\"actor\":{\"id\":458272,\"login\":\"tomalexander\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/tomalexander\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458272?\"},\"repo\":{\"id\":28367645,\"name\":\"tomalexander/basic_multi_bot\",\"url\":\"https://api.github.com/repos/tomalexander/basic_multi_bot\"},\"payload\":{\"push_id\":536753541,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b67d4bddd9a6134aae5378deedb1e311410fff75\",\"before\":\"bcc7dd7ee5550ef6d71e14b4d4c587477f71a3dc\",\"commits\":[{\"sha\":\"b67d4bddd9a6134aae5378deedb1e311410fff75\",\"author\":{\"email\":\"a09a620e88c0ebee24434030b77c3b58d8242b71@gmail.com\",\"name\":\"Tom Alexander\"},\"message\":\"restricting echo delegate to rooms\",\"distinct\":true,\"url\":\"https://api.github.com/repos/tomalexander/basic_multi_bot/commits/b67d4bddd9a6134aae5378deedb1e311410fff75\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:13Z\"}\n{\"id\":\"2489398860\",\"type\":\"PushEvent\",\"actor\":{\"id\":2581100,\"login\":\"shy2850\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/shy2850\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2581100?\"},\"repo\":{\"id\":27001976,\"name\":\"shy2850/wfQuery\",\"url\":\"https://api.github.com/repos/shy2850/wfQuery\"},\"payload\":{\"push_id\":536753542,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"31332f6a51d87ff80d88c3c6efe2a444ce052401\",\"before\":\"92b2fb9133de9d3f6a2dca59cb19a7f27af4d70b\",\"commits\":[{\"sha\":\"31332f6a51d87ff80d88c3c6efe2a444ce052401\",\"author\":{\"email\":\"ebf030c1b185fb81f6efcfbca2fa8563b9c94d00@163.com\",\"name\":\"shy2850\"},\"message\":\"变量名错误\",\"distinct\":true,\"url\":\"https://api.github.com/repos/shy2850/wfQuery/commits/31332f6a51d87ff80d88c3c6efe2a444ce052401\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:13Z\"}\n{\"id\":\"2489398861\",\"type\":\"CreateEvent\",\"actor\":{\"id\":1283090,\"login\":\"crablar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/crablar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1283090?\"},\"repo\":{\"id\":28651964,\"name\":\"crablar/betwork\",\"url\":\"https://api.github.com/repos/crablar/betwork\"},\"payload\":{\"ref\":\"prototype_1.0\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:13Z\"}\n{\"id\":\"2489398863\",\"type\":\"PushEvent\",\"actor\":{\"id\":2829718,\"login\":\"phister\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/phister\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2829718?\"},\"repo\":{\"id\":28678276,\"name\":\"phister/CollegeFootballPlayoff\",\"url\":\"https://api.github.com/repos/phister/CollegeFootballPlayoff\"},\"payload\":{\"push_id\":536753545,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"d7a35a79bb69f3b7af0ac8853378c2dfe1461a2a\",\"before\":\"34349317ec4f9b2f261e99f17d81805200fc683e\",\"commits\":[{\"sha\":\"d7a35a79bb69f3b7af0ac8853378c2dfe1461a2a\",\"author\":{\"email\":\"370890631c3d0bc00dab48c12621618666e9b595@gmail.com\",\"name\":\"phister\"},\"message\":\"Create README.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/phister/CollegeFootballPlayoff/commits/d7a35a79bb69f3b7af0ac8853378c2dfe1461a2a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:13Z\"}\n{\"id\":\"2489398869\",\"type\":\"PushEvent\",\"actor\":{\"id\":1879074,\"login\":\"harrisonchu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/harrisonchu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1879074?\"},\"repo\":{\"id\":28490067,\"name\":\"harrisonchu/loan-prediction\",\"url\":\"https://api.github.com/repos/harrisonchu/loan-prediction\"},\"payload\":{\"push_id\":536753548,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5a4515521d2d84d8a8001c13aa237b2f8a6b62ad\",\"before\":\"9955146fc381664d8875645f3742e51040b460e2\",\"commits\":[{\"sha\":\"5a4515521d2d84d8a8001c13aa237b2f8a6b62ad\",\"author\":{\"email\":\"b04e62e2ee017ce4cef87d37556bac84d980acce@gmail.com\",\"name\":\"harrisonchu\"},\"message\":\"Ignore loans that are 16-30 days late\",\"distinct\":true,\"url\":\"https://api.github.com/repos/harrisonchu/loan-prediction/commits/5a4515521d2d84d8a8001c13aa237b2f8a6b62ad\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:14Z\"}\n{\"id\":\"2489398870\",\"type\":\"PushEvent\",\"actor\":{\"id\":778342,\"login\":\"TimSimpson\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TimSimpson\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/778342?\"},\"repo\":{\"id\":1989449,\"name\":\"TimSimpson/Macaroni\",\"url\":\"https://api.github.com/repos/TimSimpson/Macaroni\"},\"payload\":{\"push_id\":536753549,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7317b25b23c37d9c1bb106562a0dd4735dbca7ec\",\"before\":\"373f2f8ebc0df3720c559ccbd9b331f567a069d2\",\"commits\":[{\"sha\":\"7317b25b23c37d9c1bb106562a0dd4735dbca7ec\",\"author\":{\"email\":\"feea1c853491a2fddb2990fb677e02f9e81b17e1@gmail.com\",\"name\":\"Tim Simpson\"},\"message\":\"Fixes bug with default arguments introduced last commit\\n\\nDidn't see this until I had to build Macaroni. I guess no Next tests\\nexpose this feature? :|\",\"distinct\":true,\"url\":\"https://api.github.com/repos/TimSimpson/Macaroni/commits/7317b25b23c37d9c1bb106562a0dd4735dbca7ec\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:14Z\"}\n{\"id\":\"2489398874\",\"type\":\"PushEvent\",\"actor\":{\"id\":5270855,\"login\":\"eldu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/eldu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5270855?\"},\"repo\":{\"id\":28463663,\"name\":\"eldu/eldu.github.io\",\"url\":\"https://api.github.com/repos/eldu/eldu.github.io\"},\"payload\":{\"push_id\":536753553,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"2f689d094f352e88e819fd13f457641c86656d6a\",\"before\":\"cb4809d26eb05e24e47320cc3ad5e9f632cf2e80\",\"commits\":[{\"sha\":\"2f689d094f352e88e819fd13f457641c86656d6a\",\"author\":{\"email\":\"fd52f7af8af1b4e2d32bf7ccd950e86bb247229e@seas.upenn.edu\",\"name\":\"Ellen Duong\"},\"message\":\"front page cover no scroll\",\"distinct\":true,\"url\":\"https://api.github.com/repos/eldu/eldu.github.io/commits/2f689d094f352e88e819fd13f457641c86656d6a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:14Z\"}\n{\"id\":\"2489398876\",\"type\":\"PushEvent\",\"actor\":{\"id\":5240798,\"login\":\"hxwang\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hxwang\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5240798?\"},\"repo\":{\"id\":20258812,\"name\":\"hxwang/Leetcode\",\"url\":\"https://api.github.com/repos/hxwang/Leetcode\"},\"payload\":{\"push_id\":536753555,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"7c1a0709545f66143c404db812f52c94af769564\",\"before\":\"628e4f129b5723f944d1a7887509c737762b2821\",\"commits\":[{\"sha\":\"7c1a0709545f66143c404db812f52c94af769564\",\"author\":{\"email\":\"320cb2ff8e2e195f7d4e5cd3b27b690e919d61e5@gmail.com\",\"name\":\"Huangxin\"},\"message\":\"Update toDiscuss.md\",\"distinct\":true,\"url\":\"https://api.github.com/repos/hxwang/Leetcode/commits/7c1a0709545f66143c404db812f52c94af769564\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:15Z\"}\n{\"id\":\"2489398877\",\"type\":\"PushEvent\",\"actor\":{\"id\":532414,\"login\":\"piranna\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/piranna\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/532414?\"},\"repo\":{\"id\":11572519,\"name\":\"NodeOS/NodeOS\",\"url\":\"https://api.github.com/repos/NodeOS/NodeOS\"},\"payload\":{\"push_id\":536753554,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/cross-compiler-musl\",\"head\":\"59f050d7225d702ab8d2f7aed19a1a3459603c5f\",\"before\":\"72df1b3e62fc8ee59c984b5587800a587aebc1fd\",\"commits\":[{\"sha\":\"59f050d7225d702ab8d2f7aed19a1a3459603c5f\",\"author\":{\"email\":\"fe61b926b190bf24488e9bedf4bfb97c0e528d9b@gmail.com\",\"name\":\"Jesús Leganés Combarro \\\"piranna\"},\"message\":\"Clean-up of cross-toolchain installer\",\"distinct\":true,\"url\":\"https://api.github.com/repos/NodeOS/NodeOS/commits/59f050d7225d702ab8d2f7aed19a1a3459603c5f\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:15Z\",\"org\":{\"id\":5056162,\"login\":\"NodeOS\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/NodeOS\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5056162?\"}}\n{\"id\":\"2489398880\",\"type\":\"WatchEvent\",\"actor\":{\"id\":170299,\"login\":\"MatthewMueller\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/MatthewMueller\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/170299?\"},\"repo\":{\"id\":10697613,\"name\":\"Netflix/ice\",\"url\":\"https://api.github.com/repos/Netflix/ice\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:15Z\",\"org\":{\"id\":913567,\"login\":\"Netflix\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/Netflix\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/913567?\"}}\n{\"id\":\"2489398882\",\"type\":\"CreateEvent\",\"actor\":{\"id\":6781905,\"login\":\"pw5a29\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pw5a29\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6781905?\"},\"repo\":{\"id\":28678291,\"name\":\"pw5a29/rawbatt\",\"url\":\"https://api.github.com/repos/pw5a29/rawbatt\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:15Z\"}\n{\"id\":\"2489398883\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":6903751,\"login\":\"bkenobi\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bkenobi\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6903751?\"},\"repo\":{\"id\":17072697,\"name\":\"genielabs/HomeGenie\",\"url\":\"https://api.github.com/repos/genielabs/HomeGenie\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/80\",\"labels_url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/80/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/80/comments\",\"events_url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/80/events\",\"html_url\":\"https://github.com/genielabs/HomeGenie/issues/80\",\"id\":52930124,\"number\":80,\"title\":\"Scripts (C# at least) produce unpredictable results due to run order\",\"user\":{\"login\":\"bkenobi\",\"id\":6903751,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6903751?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bkenobi\",\"html_url\":\"https://github.com/bkenobi\",\"followers_url\":\"https://api.github.com/users/bkenobi/followers\",\"following_url\":\"https://api.github.com/users/bkenobi/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bkenobi/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bkenobi/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bkenobi/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bkenobi/orgs\",\"repos_url\":\"https://api.github.com/users/bkenobi/repos\",\"events_url\":\"https://api.github.com/users/bkenobi/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bkenobi/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-26T16:14:46Z\",\"updated_at\":\"2015-01-01T01:07:15Z\",\"closed_at\":null,\"body\":\"I have several programs that rely on each other.  One of these (LastEvent) watches for events using the When.ModuleParameterIsChanging event and keeps several new parameters up to date (LastEvent, LastEventUpdateTime, LastOn, LastOnUpdateTime, LastOff, LastOffUpdateTime).  If the other scripts that rely on the LastEvent module run first, everything works correctly.  If the LastEvent module runs first, I run into problems with what exactly the values of these new parameters will be.\\r\\n\\r\\nI believe (but cannot confirm) that the execution order is based on the programID.  This was working acceptably until I had to replace my main code with a new instance during testing.  When this happened, the programID of the main code shifted from 1001 to 1016.  Since the LastEvent module reamained 1004, I believe this would cause it to execute prior to 1016.\\r\\n\\r\\nThere are 2 solutions I have come up with.\\r\\n1)  Provide some means to control the order that programs are executed.\\r\\n2)  Build these new parameter values (or equivalent) into the core HG code.\\r\\n\\r\\nExecution order could be something simple like a priority field where the user selects 1-5 which would then give a programID of the next available 1000, 2000, 3000, etc.  Or, something within the code itself specifying a priority.  Or, maybe allow the user to reorder the scripts in the interface and that order is the order of execution of the user codes.  I personally would think something that combines these would be best.\\r\\n\\r\\nI don't know how often this issue would occur, so I don't know how much effort is needed to correct it.\"},\"comment\":{\"url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/comments/68477374\",\"html_url\":\"https://github.com/genielabs/HomeGenie/issues/80#issuecomment-68477374\",\"issue_url\":\"https://api.github.com/repos/genielabs/HomeGenie/issues/80\",\"id\":68477374,\"user\":{\"login\":\"bkenobi\",\"id\":6903751,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6903751?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/bkenobi\",\"html_url\":\"https://github.com/bkenobi\",\"followers_url\":\"https://api.github.com/users/bkenobi/followers\",\"following_url\":\"https://api.github.com/users/bkenobi/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/bkenobi/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/bkenobi/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/bkenobi/subscriptions\",\"organizations_url\":\"https://api.github.com/users/bkenobi/orgs\",\"repos_url\":\"https://api.github.com/users/bkenobi/repos\",\"events_url\":\"https://api.github.com/users/bkenobi/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/bkenobi/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:15Z\",\"updated_at\":\"2015-01-01T01:07:15Z\",\"body\":\"This might work, I'll have to try.  I'm still concerned about the potentially large delay in response due to each pause command.  Even if the modifications I make don't make it into the main code, it would be nice to understand why my code isn't working.\\r\\n\\r\\nhttp://www.homegenie.it/forum/index.php?topic=575.0\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:15Z\"}\n{\"id\":\"2489398884\",\"type\":\"PushEvent\",\"actor\":{\"id\":280212,\"login\":\"KenanSulayman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/KenanSulayman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/280212?\"},\"repo\":{\"id\":21481110,\"name\":\"KenanSulayman/heartbeat\",\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat\"},\"payload\":{\"push_id\":536753557,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"fdc821da59a881d40eea62001b4af4cbc2c1e232\",\"before\":\"cf281dd3839c3e07ed867a2eb77aa4965a2a1400\",\"commits\":[{\"sha\":\"fdc821da59a881d40eea62001b4af4cbc2c1e232\",\"author\":{\"email\":\"9176253dfc0bc82671a5e984646605f93319147a@sly.mn\",\"name\":\"Kenan Sulayman\"},\"message\":\"1420074434565\\n\\nZsR8phvzSvI+8KOpooE0RB0FXC78yt7CZDcUB+WKRnQ=\",\"distinct\":true,\"url\":\"https://api.github.com/repos/KenanSulayman/heartbeat/commits/fdc821da59a881d40eea62001b4af4cbc2c1e232\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:16Z\"}\n{\"id\":\"2489398891\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":9265995,\"login\":\"remlei\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/remlei\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9265995?\"},\"repo\":{\"id\":22003267,\"name\":\"tarampampam/nod32-update-mirror\",\"url\":\"https://api.github.com/repos/tarampampam/nod32-update-mirror\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/tarampampam/nod32-update-mirror/issues/14\",\"labels_url\":\"https://api.github.com/repos/tarampampam/nod32-update-mirror/issues/14/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/tarampampam/nod32-update-mirror/issues/14/comments\",\"events_url\":\"https://api.github.com/repos/tarampampam/nod32-update-mirror/issues/14/events\",\"html_url\":\"https://github.com/tarampampam/nod32-update-mirror/issues/14\",\"id\":53210304,\"number\":14,\"title\":\"Needs further improvements.\",\"user\":{\"login\":\"remlei\",\"id\":9265995,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9265995?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/remlei\",\"html_url\":\"https://github.com/remlei\",\"followers_url\":\"https://api.github.com/users/remlei/followers\",\"following_url\":\"https://api.github.com/users/remlei/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/remlei/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/remlei/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/remlei/subscriptions\",\"organizations_url\":\"https://api.github.com/users/remlei/orgs\",\"repos_url\":\"https://api.github.com/users/remlei/repos\",\"events_url\":\"https://api.github.com/users/remlei/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/remlei/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:07:17Z\",\"updated_at\":\"2015-01-01T01:07:17Z\",\"closed_at\":null,\"body\":\"There's no issue on the script and it works but it downloads a lot of junk in process and it waste CPU cycle at start parsing the update.ver (taking it a while to process the update to download) this also it needs to improve the following:\\r\\n\\r\\n- able to select what type of product to cache the updates (eg, Antivirus or Smart Security)\\r\\n- able to bypass other language packs (we are downloading about 1gigs of junk versus a 120mb of important files)\\r\\n- learn to use sed to parse and check the if the file currently downloaded is updated or not. Instead of parsing the update.ver before downloads. It can check the version on the fly by checking the \\\"versionid=\\\" on the nup files and one from update.ver; if didnt match then it downloads.\\r\\n\\r\\nAll of those issues are not present using this script from a russian forum: http://wl500g.info/showthread.php?7877-NOD32-update-server . It runs in every linux flavor as long as you had the depencies installed (sed, unrar, touch-coreutils)\\r\\n\\r\\nIt does do the job and with minor editing on the main script file to accept downloads on v6,v7 and v8 eset products.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\"}\n{\"id\":\"2489398892\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1235097,\"login\":\"rmarinho\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?\"},\"repo\":{\"id\":20463939,\"name\":\"XLabs/Xamarin-Forms-Labs\",\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/545\",\"labels_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/545/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/545/comments\",\"events_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/545/events\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/issues/545\",\"id\":53205036,\"number\":545,\"title\":\"ContentPresenter view\",\"user\":{\"login\":\"onovotny\",\"id\":1427284,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1427284?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/onovotny\",\"html_url\":\"https://github.com/onovotny\",\"followers_url\":\"https://api.github.com/users/onovotny/followers\",\"following_url\":\"https://api.github.com/users/onovotny/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/onovotny/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/onovotny/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/onovotny/subscriptions\",\"organizations_url\":\"https://api.github.com/users/onovotny/orgs\",\"repos_url\":\"https://api.github.com/users/onovotny/repos\",\"events_url\":\"https://api.github.com/users/onovotny/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/onovotny/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":2,\"created_at\":\"2014-12-31T21:48:29Z\",\"updated_at\":\"2015-01-01T01:07:17Z\",\"closed_at\":null,\"body\":\"I've blogged about it [here](http://blog.novotny.org/2014/12/31/contentpresenter-for-xamarin-forms/) but I'd be happy to submit a PR with this.\\r\\n\\r\\nThoughts, interest?\"},\"comment\":{\"url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/comments/68477375\",\"html_url\":\"https://github.com/XLabs/Xamarin-Forms-Labs/issues/545#issuecomment-68477375\",\"issue_url\":\"https://api.github.com/repos/XLabs/Xamarin-Forms-Labs/issues/545\",\"id\":68477375,\"user\":{\"login\":\"rmarinho\",\"id\":1235097,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1235097?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rmarinho\",\"html_url\":\"https://github.com/rmarinho\",\"followers_url\":\"https://api.github.com/users/rmarinho/followers\",\"following_url\":\"https://api.github.com/users/rmarinho/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/rmarinho/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/rmarinho/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/rmarinho/subscriptions\",\"organizations_url\":\"https://api.github.com/users/rmarinho/orgs\",\"repos_url\":\"https://api.github.com/users/rmarinho/repos\",\"events_url\":\"https://api.github.com/users/rmarinho/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/rmarinho/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:17Z\",\"updated_at\":\"2015-01-01T01:07:17Z\",\"body\":\"Yes i use your aproach and a PR of it will be welcome :) \"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\",\"org\":{\"id\":7787062,\"login\":\"XLabs\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/XLabs\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7787062?\"}}\n{\"id\":\"2489398894\",\"type\":\"GollumEvent\",\"actor\":{\"id\":226687,\"login\":\"cfalguiere\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cfalguiere\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/226687?\"},\"repo\":{\"id\":28568493,\"name\":\"cfalguiere/BrickInventory\",\"url\":\"https://api.github.com/repos/cfalguiere/BrickInventory\"},\"payload\":{\"pages\":[{\"page_name\":\"LeMakingOf J2#01 Objectif\",\"title\":\"LeMakingOf J2#01 Objectif\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2301-Objectif\"},{\"page_name\":\"LeMakingOf J2#02 Design\",\"title\":\"LeMakingOf J2#02 Design\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2302-Design\"},{\"page_name\":\"LeMakingOf J2#03 Refactoring\",\"title\":\"LeMakingOf J2#03 Refactoring\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2303-Refactoring\"},{\"page_name\":\"LeMakingOf J2#04 Première factory\",\"title\":\"LeMakingOf J2#04 Première factory\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2304-Premi%C3%A8re-factory\"},{\"page_name\":\"LeMakingOf J2#05 Tests unitaires\",\"title\":\"LeMakingOf J2#05 Tests unitaires\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2305-Tests-unitaires\"},{\"page_name\":\"LeMakingOf J2#06 Retour des tests unitaires\",\"title\":\"LeMakingOf J2#06 Retour des tests unitaires\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2306-Retour-des-tests-unitaires\"},{\"page_name\":\"LeMakingOf J2#07 Retour du retour des tests unitaires\",\"title\":\"LeMakingOf J2#07 Retour du retour des tests unitaires\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2307-Retour-du-retour-des-tests-unitaires\"},{\"page_name\":\"LeMakingOf J2#08 Test Reloaded\",\"title\":\"LeMakingOf J2#08 Test Reloaded\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2308-Test-Reloaded\"},{\"page_name\":\"LeMakingOf J2#10 Les formes\",\"title\":\"LeMakingOf J2#10 Les formes\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2310-Les-formes\"},{\"page_name\":\"LeMakingOf J2#11 Ajout de la colonne\",\"title\":\"LeMakingOf J2#11 Ajout de la colonne\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2311-Ajout-de-la-colonne\"},{\"page_name\":\"LeMakingOf J2#12 Le filtre\",\"title\":\"LeMakingOf J2#12 Le filtre\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2312-Le-filtre\"},{\"page_name\":\"LeMakingOf J2#13 Premier service\",\"title\":\"LeMakingOf J2#13 Premier service\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2313-Premier-service\"},{\"page_name\":\"LeMakingOf J2#14 Intégration\",\"title\":\"LeMakingOf J2#14 Intégration\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2314-Int%C3%A9gration\"},{\"page_name\":\"LeMakingOf J2#15 Les couleurs\",\"title\":\"LeMakingOf J2#15 Les couleurs\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2315-Les-couleurs\"},{\"page_name\":\"LeMakingOf J2#16 Le bilan\",\"title\":\"LeMakingOf J2#16 Le bilan\",\"summary\":null,\"action\":\"created\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2316-Le-bilan\"},{\"page_name\":\"LeMakingOf J1#12 Les modules\",\"title\":\"LeMakingOf J1#12 Les modules\",\"summary\":null,\"action\":\"edited\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J1%2312-Les-modules\"},{\"page_name\":\"LeMakingOf J1#14 Le controller\",\"title\":\"LeMakingOf J1#14 Le controller\",\"summary\":null,\"action\":\"edited\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J1%2314-Le-controller\"},{\"page_name\":\"LeMakingOf J1#19 Le bilan\",\"title\":\"LeMakingOf J1#19 Le bilan\",\"summary\":null,\"action\":\"edited\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J1%2319-Le-bilan\"},{\"page_name\":\"LeMakingOf J2#09 Test de la factory\",\"title\":\"LeMakingOf J2#09 Test de la factory\",\"summary\":null,\"action\":\"edited\",\"sha\":\"078a1257c89da10fdb31de9b29bf48494d761c83\",\"html_url\":\"https://github.com/cfalguiere/BrickInventory/wiki/LeMakingOf-J2%2309-Test-de-la-factory\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\"}\n{\"id\":\"2489398895\",\"type\":\"PushEvent\",\"actor\":{\"id\":4644601,\"login\":\"GLolol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/GLolol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4644601?\"},\"repo\":{\"id\":28604112,\"name\":\"GLolol/Crackbot\",\"url\":\"https://api.github.com/repos/GLolol/Crackbot\"},\"payload\":{\"push_id\":536753559,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/devel\",\"head\":\"f5b144c834362b41505427167aff8d4b98fa657e\",\"before\":\"d34e47b3932ca6ee93229c59e5a4c7bcf7acf5be\",\"commits\":[{\"sha\":\"f5b144c834362b41505427167aff8d4b98fa657e\",\"author\":{\"email\":\"1d63ba34440ccf4049662a35916618df27a8f853@overdrive.pw\",\"name\":\"GLolol\"},\"message\":\"commands.lua: bugfix (\\\"./help\\\" without arguments should call \\\"./help help\\\")\",\"distinct\":true,\"url\":\"https://api.github.com/repos/GLolol/Crackbot/commits/f5b144c834362b41505427167aff8d4b98fa657e\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\"}\n{\"id\":\"2489398897\",\"type\":\"PushEvent\",\"actor\":{\"id\":163915,\"login\":\"fponticelli\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/fponticelli\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/163915?\"},\"repo\":{\"id\":25146353,\"name\":\"thxlib/thxlib.github.io\",\"url\":\"https://api.github.com/repos/thxlib/thxlib.github.io\"},\"payload\":{\"push_id\":536753561,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"b68cc956373b5339237f1564f1ef8fec91337dd0\",\"before\":\"d0f110895e656633828c806023799bab1cbbd083\",\"commits\":[{\"sha\":\"b68cc956373b5339237f1564f1ef8fec91337dd0\",\"author\":{\"email\":\"05a4419b3ba135c9a2552a2fac2f13cfe3d22f12@gmail.com\",\"name\":\"Franco Ponticelli\"},\"message\":\"updated thx.core and thx.color\",\"distinct\":true,\"url\":\"https://api.github.com/repos/thxlib/thxlib.github.io/commits/b68cc956373b5339237f1564f1ef8fec91337dd0\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\",\"org\":{\"id\":8933789,\"login\":\"thxlib\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/thxlib\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8933789?\"}}\n{\"id\":\"2489398899\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5249918,\"login\":\"ehnmark\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ehnmark\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5249918?\"},\"repo\":{\"id\":28678277,\"name\":\"ehnmark/housing\",\"url\":\"https://api.github.com/repos/ehnmark/housing\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\"}\n{\"id\":\"2489398900\",\"type\":\"PushEvent\",\"actor\":{\"id\":8020662,\"login\":\"freethlua\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/freethlua\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8020662?\"},\"repo\":{\"id\":22790186,\"name\":\"freethlua/.dotfiles\",\"url\":\"https://api.github.com/repos/freethlua/.dotfiles\"},\"payload\":{\"push_id\":536753563,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"before\":\"233774f844d59ff89420037609cf3e0cba985e9a\",\"commits\":[{\"sha\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"author\":{\"email\":\"11f6ad8ec52a2984abaafd7c3b516503785c2072\",\"name\":\"x\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/freethlua/.dotfiles/commits/0ec37f77b8811a07ca6c6df79622541d8fa40c0a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:18Z\"}\n{\"id\":\"2489398902\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":10361486,\"login\":\"AceTheGmodGeek\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AceTheGmodGeek\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361486?\"},\"repo\":{\"id\":20768404,\"name\":\"vrondakis/Leveling-System\",\"url\":\"https://api.github.com/repos/vrondakis/Leveling-System\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/vrondakis/Leveling-System/issues/11\",\"labels_url\":\"https://api.github.com/repos/vrondakis/Leveling-System/issues/11/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/vrondakis/Leveling-System/issues/11/comments\",\"events_url\":\"https://api.github.com/repos/vrondakis/Leveling-System/issues/11/events\",\"html_url\":\"https://github.com/vrondakis/Leveling-System/issues/11\",\"id\":53210305,\"number\":11,\"title\":\"Books\",\"user\":{\"login\":\"AceTheGmodGeek\",\"id\":10361486,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10361486?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AceTheGmodGeek\",\"html_url\":\"https://github.com/AceTheGmodGeek\",\"followers_url\":\"https://api.github.com/users/AceTheGmodGeek/followers\",\"following_url\":\"https://api.github.com/users/AceTheGmodGeek/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AceTheGmodGeek/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AceTheGmodGeek/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AceTheGmodGeek/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AceTheGmodGeek/orgs\",\"repos_url\":\"https://api.github.com/users/AceTheGmodGeek/repos\",\"events_url\":\"https://api.github.com/users/AceTheGmodGeek/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AceTheGmodGeek/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:07:19Z\",\"updated_at\":\"2015-01-01T01:07:19Z\",\"closed_at\":null,\"body\":\"I have your level system with your prestige. When I add books to the books.lua, It doesnt pop up. Many people are having problems with this. Help?\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:19Z\"}\n{\"id\":\"2489398905\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397307\",\"id\":22397307,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\\n \\n - name: custom horizon splash logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png mode=0644 force=yes\\n \\n - name: custom horizon favicon\\n-  get_url: url={{ horizon.favicon_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico\\n-           force=yes\\n+  get_url: url={{ horizon.favicon_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico force=yes\\n \\n - name: put images and fonts where apache can find them\\n-  file: src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n-        dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n-        state=link\\n-        owner=www-data\\n-        group=www-data\\n+  file: |\\n+    src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n+    dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n+    state=link\\n+    owner=www-data\\n+    group=www-data\\n   with_items:\\n     - img\\n     - fonts\\n \\n-# flush before ensuring apache running, saves immediate restart\\n-- meta: flush_handlers\\n-\\n-- name: ensure apache started\\n+- name: ensure apache is running\\n   service: name=apache2 state=started\\n \\n-- name: Permit HTTP and HTTPS\\n-  ufw: rule=allow to_port={{ item }} proto=tcp\\n-  with_items:\\n-  - 80\\n-  - 443\\n+- name: Permit HTTP (redirect to HTTPS for Horizon)\\n+  command: ufw allow 80/tcp\\n \\n-- include: monitoring.yml tags=monitoring,common\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":168,\"original_position\":168,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"What happened to the monitoring tasks?\",\"created_at\":\"2015-01-01T01:07:19Z\",\"updated_at\":\"2015-01-01T01:07:19Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397307\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397307\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397307\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:07:19Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:07:19Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489398906\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":860034,\"login\":\"Renelvon\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Renelvon\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/860034?\"},\"repo\":{\"id\":19147609,\"name\":\"OpenBazaar/OpenBazaar\",\"url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/comments/22397308\",\"id\":22397308,\"diff_hunk\":\"@@ -39,3 +41,19 @@ def get_history():\\n \\n def get_global(guid, callback):\\n     get_unspent(burnaddr_from_guid(guid), callback)\\n+\\n+\\n+def is_valid_namecoin(namecoin, guid):\\n+    if not namecoin or not guid:\\n+        return False\\n+\\n+    server = DNSChainServer.Server(constants.DNSCHAIN_SERVER_IP, \\\"\\\")\\n+    _log.info(\\\"Looking up namecoin id: %s\\\", namecoin)\\n+    try:\\n+        data = server.lookup(\\\"id/\\\"+namecoin)\",\"path\":\"node/trust.py\",\"position\":22,\"original_position\":22,\"commit_id\":\"a84a18a43e1e6c91ca2bf58a2ab951268cf43f51\",\"original_commit_id\":\"a84a18a43e1e6c91ca2bf58a2ab951268cf43f51\",\"user\":{\"login\":\"Renelvon\",\"id\":860034,\"avatar_url\":\"https://avatars.githubusercontent.com/u/860034?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Renelvon\",\"html_url\":\"https://github.com/Renelvon\",\"followers_url\":\"https://api.github.com/users/Renelvon/followers\",\"following_url\":\"https://api.github.com/users/Renelvon/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Renelvon/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Renelvon/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Renelvon/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Renelvon/orgs\",\"repos_url\":\"https://api.github.com/users/Renelvon/repos\",\"events_url\":\"https://api.github.com/users/Renelvon/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Renelvon/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"To be fair, I can't see any PEP8 issue here, save for the spaces around `+`. Is that it?\",\"created_at\":\"2015-01-01T01:07:19Z\",\"updated_at\":\"2015-01-01T01:07:19Z\",\"html_url\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105#discussion_r22397308\",\"pull_request_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/comments/22397308\"},\"html\":{\"href\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105#discussion_r22397308\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105\",\"id\":25621449,\"html_url\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105\",\"diff_url\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105.diff\",\"patch_url\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105.patch\",\"issue_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/1105\",\"number\":1105,\"state\":\"open\",\"locked\":false,\"title\":\"[WIP]Issue #485 Relay namecoin id to other nodes\",\"user\":{\"login\":\"charapod\",\"id\":6865935,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6865935?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/charapod\",\"html_url\":\"https://github.com/charapod\",\"followers_url\":\"https://api.github.com/users/charapod/followers\",\"following_url\":\"https://api.github.com/users/charapod/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/charapod/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/charapod/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/charapod/subscriptions\",\"organizations_url\":\"https://api.github.com/users/charapod/orgs\",\"repos_url\":\"https://api.github.com/users/charapod/repos\",\"events_url\":\"https://api.github.com/users/charapod/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/charapod/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"1) As requested in #957, we changed namecoin_id column to lowercase. \\r\\n\\r\\nBut! Someone who had already upgraded their database with the old migration scripts will still have their column named with a capital, and there's no easy way to get rid of it since sqlite does not support DROP COLUMN (issue #955). New databases created will work fine of course, and rerunning the migration4.py script will probably do it as well. Let us know if you want us to cancel this modification.\\r\\n\\r\\n2) We begun working on issue #485 by @dionyziz . We added the namecoin of the sender to the rest of the information in the 'start_handshake' and 'send' methods in CryptoPeerConnection and we also stored the namecoin the user entered with the rest of his info in CryptoTrasportLayer. Is this the right way to do it? Should we change anything else? We need a little feedback here @gubatron , @hoffmabc :) \\r\\n\\r\\n3) As it is now, TransportLayer does not have a 'namecoin' field. Should we add a namecoin field in there, and treat it in the same manner as nickname?\\r\\n\\r\\n4)Should we add the namecoin to the 'ok_msg' in transport.py (line 136)?\\r\\n\\r\\n5)We see that you sign the information *before* you encrypt it. Is that intended? afaik this is not secure...\",\"created_at\":\"2014-12-06T17:47:55Z\",\"updated_at\":\"2015-01-01T01:07:19Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"3ef89a9529be958a544b49a90605c5cb1f351931\",\"assignee\":{\"login\":\"dionyziz\",\"id\":544572,\"avatar_url\":\"https://avatars.githubusercontent.com/u/544572?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dionyziz\",\"html_url\":\"https://github.com/dionyziz\",\"followers_url\":\"https://api.github.com/users/dionyziz/followers\",\"following_url\":\"https://api.github.com/users/dionyziz/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dionyziz/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dionyziz/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dionyziz/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dionyziz/orgs\",\"repos_url\":\"https://api.github.com/users/dionyziz/repos\",\"events_url\":\"https://api.github.com/users/dionyziz/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dionyziz/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105/commits\",\"review_comments_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105/comments\",\"review_comment_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/1105/comments\",\"statuses_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/statuses/a84a18a43e1e6c91ca2bf58a2ab951268cf43f51\",\"head\":{\"label\":\"charapod:namecoin\",\"ref\":\"namecoin\",\"sha\":\"a84a18a43e1e6c91ca2bf58a2ab951268cf43f51\",\"user\":{\"login\":\"charapod\",\"id\":6865935,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6865935?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/charapod\",\"html_url\":\"https://github.com/charapod\",\"followers_url\":\"https://api.github.com/users/charapod/followers\",\"following_url\":\"https://api.github.com/users/charapod/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/charapod/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/charapod/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/charapod/subscriptions\",\"organizations_url\":\"https://api.github.com/users/charapod/orgs\",\"repos_url\":\"https://api.github.com/users/charapod/repos\",\"events_url\":\"https://api.github.com/users/charapod/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/charapod/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":26178277,\"name\":\"OpenBazaar\",\"full_name\":\"charapod/OpenBazaar\",\"owner\":{\"login\":\"charapod\",\"id\":6865935,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6865935?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/charapod\",\"html_url\":\"https://github.com/charapod\",\"followers_url\":\"https://api.github.com/users/charapod/followers\",\"following_url\":\"https://api.github.com/users/charapod/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/charapod/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/charapod/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/charapod/subscriptions\",\"organizations_url\":\"https://api.github.com/users/charapod/orgs\",\"repos_url\":\"https://api.github.com/users/charapod/repos\",\"events_url\":\"https://api.github.com/users/charapod/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/charapod/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/charapod/OpenBazaar\",\"description\":\"A decentralized marketplace\",\"fork\":true,\"url\":\"https://api.github.com/repos/charapod/OpenBazaar\",\"forks_url\":\"https://api.github.com/repos/charapod/OpenBazaar/forks\",\"keys_url\":\"https://api.github.com/repos/charapod/OpenBazaar/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/charapod/OpenBazaar/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/charapod/OpenBazaar/teams\",\"hooks_url\":\"https://api.github.com/repos/charapod/OpenBazaar/hooks\",\"issue_events_url\":\"https://api.github.com/repos/charapod/OpenBazaar/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/charapod/OpenBazaar/events\",\"assignees_url\":\"https://api.github.com/repos/charapod/OpenBazaar/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/charapod/OpenBazaar/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/charapod/OpenBazaar/tags\",\"blobs_url\":\"https://api.github.com/repos/charapod/OpenBazaar/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/charapod/OpenBazaar/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/charapod/OpenBazaar/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/charapod/OpenBazaar/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/charapod/OpenBazaar/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/charapod/OpenBazaar/languages\",\"stargazers_url\":\"https://api.github.com/repos/charapod/OpenBazaar/stargazers\",\"contributors_url\":\"https://api.github.com/repos/charapod/OpenBazaar/contributors\",\"subscribers_url\":\"https://api.github.com/repos/charapod/OpenBazaar/subscribers\",\"subscription_url\":\"https://api.github.com/repos/charapod/OpenBazaar/subscription\",\"commits_url\":\"https://api.github.com/repos/charapod/OpenBazaar/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/charapod/OpenBazaar/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/charapod/OpenBazaar/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/charapod/OpenBazaar/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/charapod/OpenBazaar/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/charapod/OpenBazaar/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/charapod/OpenBazaar/merges\",\"archive_url\":\"https://api.github.com/repos/charapod/OpenBazaar/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/charapod/OpenBazaar/downloads\",\"issues_url\":\"https://api.github.com/repos/charapod/OpenBazaar/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/charapod/OpenBazaar/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/charapod/OpenBazaar/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/charapod/OpenBazaar/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/charapod/OpenBazaar/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/charapod/OpenBazaar/releases{/id}\",\"created_at\":\"2014-11-04T16:42:59Z\",\"updated_at\":\"2014-11-09T20:18:11Z\",\"pushed_at\":\"2014-12-27T12:44:05Z\",\"git_url\":\"git://github.com/charapod/OpenBazaar.git\",\"ssh_url\":\"git@github.com:charapod/OpenBazaar.git\",\"clone_url\":\"https://github.com/charapod/OpenBazaar.git\",\"svn_url\":\"https://github.com/charapod/OpenBazaar\",\"homepage\":\"http://forum.openbazaar.org\",\"size\":17333,\"stargazers_count\":0,\"watchers_count\":0,\"language\":null,\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"OpenBazaar:develop\",\"ref\":\"develop\",\"sha\":\"e0eb864694ae8041e17c4a350102de77b59e9a68\",\"user\":{\"login\":\"OpenBazaar\",\"id\":7438770,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7438770?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/OpenBazaar\",\"html_url\":\"https://github.com/OpenBazaar\",\"followers_url\":\"https://api.github.com/users/OpenBazaar/followers\",\"following_url\":\"https://api.github.com/users/OpenBazaar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/OpenBazaar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/OpenBazaar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/OpenBazaar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/OpenBazaar/orgs\",\"repos_url\":\"https://api.github.com/users/OpenBazaar/repos\",\"events_url\":\"https://api.github.com/users/OpenBazaar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/OpenBazaar/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":19147609,\"name\":\"OpenBazaar\",\"full_name\":\"OpenBazaar/OpenBazaar\",\"owner\":{\"login\":\"OpenBazaar\",\"id\":7438770,\"avatar_url\":\"https://avatars.githubusercontent.com/u/7438770?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/OpenBazaar\",\"html_url\":\"https://github.com/OpenBazaar\",\"followers_url\":\"https://api.github.com/users/OpenBazaar/followers\",\"following_url\":\"https://api.github.com/users/OpenBazaar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/OpenBazaar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/OpenBazaar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/OpenBazaar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/OpenBazaar/orgs\",\"repos_url\":\"https://api.github.com/users/OpenBazaar/repos\",\"events_url\":\"https://api.github.com/users/OpenBazaar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/OpenBazaar/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/OpenBazaar/OpenBazaar\",\"description\":\"A decentralized marketplace\",\"fork\":false,\"url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar\",\"forks_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/forks\",\"keys_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/teams\",\"hooks_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/hooks\",\"issue_events_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/events\",\"assignees_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/tags\",\"blobs_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/languages\",\"stargazers_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/stargazers\",\"contributors_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/contributors\",\"subscribers_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/subscribers\",\"subscription_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/subscription\",\"commits_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/merges\",\"archive_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/downloads\",\"issues_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/releases{/id}\",\"created_at\":\"2014-04-25T13:49:36Z\",\"updated_at\":\"2014-12-31T19:56:51Z\",\"pushed_at\":\"2014-12-30T10:37:28Z\",\"git_url\":\"git://github.com/OpenBazaar/OpenBazaar.git\",\"ssh_url\":\"git@github.com:OpenBazaar/OpenBazaar.git\",\"clone_url\":\"https://github.com/OpenBazaar/OpenBazaar.git\",\"svn_url\":\"https://github.com/OpenBazaar/OpenBazaar\",\"homepage\":\"http://www.reddit.com/r/OpenBazaar/\",\"size\":31718,\"stargazers_count\":1115,\"watchers_count\":1115,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":290,\"mirror_url\":null,\"open_issues_count\":248,\"forks\":290,\"open_issues\":248,\"watchers\":1115,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105\"},\"html\":{\"href\":\"https://github.com/OpenBazaar/OpenBazaar/pull/1105\"},\"issue\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/1105\"},\"comments\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/issues/1105/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/pulls/1105/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/OpenBazaar/OpenBazaar/statuses/a84a18a43e1e6c91ca2bf58a2ab951268cf43f51\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:07:19Z\",\"org\":{\"id\":7438770,\"login\":\"OpenBazaar\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/OpenBazaar\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7438770?\"}}\n{\"id\":\"2489398907\",\"type\":\"CreateEvent\",\"actor\":{\"id\":51458,\"login\":\"hlapp\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/hlapp\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/51458?\"},\"repo\":{\"id\":11856817,\"name\":\"ropensci/RNeXML\",\"url\":\"https://api.github.com/repos/ropensci/RNeXML\"},\"payload\":{\"ref\":\"hl-refmt-appnote-1\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Implementing semantically rich NeXML I/O in R\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:20Z\",\"org\":{\"id\":1200269,\"login\":\"ropensci\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/ropensci\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1200269?\"}}\n{\"id\":\"2489398912\",\"type\":\"WatchEvent\",\"actor\":{\"id\":898282,\"login\":\"jzajac\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jzajac\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/898282?\"},\"repo\":{\"id\":20537104,\"name\":\"google/cayley\",\"url\":\"https://api.github.com/repos/google/cayley\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:23Z\",\"org\":{\"id\":1342004,\"login\":\"google\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/google\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1342004?\"}}\n{\"id\":\"2489398916\",\"type\":\"PushEvent\",\"actor\":{\"id\":6462036,\"login\":\"samhillman\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/samhillman\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6462036?\"},\"repo\":{\"id\":28253700,\"name\":\"samhillman/newdotcom\",\"url\":\"https://api.github.com/repos/samhillman/newdotcom\"},\"payload\":{\"push_id\":536753566,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0a4a5c67d0671000171574a0f928f0b2236522e3\",\"before\":\"eb166d0e9ff48e250e0b322ef42013d02dca159c\",\"commits\":[{\"sha\":\"0a4a5c67d0671000171574a0f928f0b2236522e3\",\"author\":{\"email\":\"3f12132dd817f39877292097b6071939bf5ccbcd@cvbay.co.uk\",\"name\":\"Sam Hillman\"},\"message\":\"removed unneeded page\",\"distinct\":true,\"url\":\"https://api.github.com/repos/samhillman/newdotcom/commits/0a4a5c67d0671000171574a0f928f0b2236522e3\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:23Z\"}\n{\"id\":\"2489398917\",\"type\":\"PushEvent\",\"actor\":{\"id\":3599988,\"login\":\"wesdizzle\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/wesdizzle\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3599988?\"},\"repo\":{\"id\":28250120,\"name\":\"wesdizzle/gagglelog\",\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog\"},\"payload\":{\"push_id\":536753567,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"da61d76888db710591e7a0b68ca5e8d273569e9c\",\"before\":\"d04cbec8b68d0e66cf87c078862065645e181086\",\"commits\":[{\"sha\":\"da61d76888db710591e7a0b68ca5e8d273569e9c\",\"author\":{\"email\":\"baaa01a5d45f86e3d8f7008866cf0d37bea55570@gmail.com\",\"name\":\"Wesley Miller\"},\"message\":\"added Index value to Expansions\",\"distinct\":true,\"url\":\"https://api.github.com/repos/wesdizzle/gagglelog/commits/da61d76888db710591e7a0b68ca5e8d273569e9c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:23Z\"}\n{\"id\":\"2489398920\",\"type\":\"PushEvent\",\"actor\":{\"id\":2523987,\"login\":\"laggingreflex\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/laggingreflex\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2523987?\"},\"repo\":{\"id\":22790173,\"name\":\"laggingreflex/.dotfiles\",\"url\":\"https://api.github.com/repos/laggingreflex/.dotfiles\"},\"payload\":{\"push_id\":536753569,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"before\":\"233774f844d59ff89420037609cf3e0cba985e9a\",\"commits\":[{\"sha\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"author\":{\"email\":\"11f6ad8ec52a2984abaafd7c3b516503785c2072\",\"name\":\"x\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/laggingreflex/.dotfiles/commits/0ec37f77b8811a07ca6c6df79622541d8fa40c0a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:24Z\"}\n{\"id\":\"2489398921\",\"type\":\"PushEvent\",\"actor\":{\"id\":3315647,\"login\":\"shaunoneill\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/shaunoneill\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3315647?\"},\"repo\":{\"id\":28667121,\"name\":\"GreenUmbrellaSoftware/website\",\"url\":\"https://api.github.com/repos/GreenUmbrellaSoftware/website\"},\"payload\":{\"push_id\":536753570,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"f028ffa984eb28fb783a7c1df7364eca3bfb7f4d\",\"before\":\"6eafa085f08d3506a09df4b15649686a2eeeedc4\",\"commits\":[{\"sha\":\"f028ffa984eb28fb783a7c1df7364eca3bfb7f4d\",\"author\":{\"email\":\"a2bb892f7d3d0e9e81f56035b4467ebbc2c5d2cd@greenumbrellasoftware.com\",\"name\":\"shaun\"},\"message\":\"updated the pages\",\"distinct\":true,\"url\":\"https://api.github.com/repos/GreenUmbrellaSoftware/website/commits/f028ffa984eb28fb783a7c1df7364eca3bfb7f4d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:24Z\",\"org\":{\"id\":2716713,\"login\":\"GreenUmbrellaSoftware\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/GreenUmbrellaSoftware\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2716713?\"}}\n{\"id\":\"2489398924\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":134455,\"login\":\"whit537\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/whit537\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/134455?\"},\"repo\":{\"id\":16488998,\"name\":\"gratipay/inside.gratipay.com\",\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/93\",\"labels_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/93/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/93/comments\",\"events_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/93/events\",\"html_url\":\"https://github.com/gratipay/inside.gratipay.com/issues/93\",\"id\":45497806,\"number\":93,\"title\":\"pin down retreat schedule\",\"user\":{\"login\":\"clone1018\",\"id\":226638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/226638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clone1018\",\"html_url\":\"https://github.com/clone1018\",\"followers_url\":\"https://api.github.com/users/clone1018/followers\",\"following_url\":\"https://api.github.com/users/clone1018/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clone1018/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clone1018/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clone1018/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clone1018/orgs\",\"repos_url\":\"https://api.github.com/users/clone1018/repos\",\"events_url\":\"https://api.github.com/users/clone1018/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clone1018/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/milestones/2\",\"labels_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/milestones/2/labels\",\"id\":782050,\"number\":2,\"title\":\"Gratipay Retreat 2015\",\"description\":\"This is an encompassing milestone for planning the Gratipay Retreat. \",\"creator\":{\"login\":\"clone1018\",\"id\":226638,\"avatar_url\":\"https://avatars.githubusercontent.com/u/226638?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/clone1018\",\"html_url\":\"https://github.com/clone1018\",\"followers_url\":\"https://api.github.com/users/clone1018/followers\",\"following_url\":\"https://api.github.com/users/clone1018/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/clone1018/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/clone1018/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/clone1018/subscriptions\",\"organizations_url\":\"https://api.github.com/users/clone1018/orgs\",\"repos_url\":\"https://api.github.com/users/clone1018/repos\",\"events_url\":\"https://api.github.com/users/clone1018/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/clone1018/received_events\",\"type\":\"User\",\"site_admin\":false},\"open_issues\":2,\"closed_issues\":1,\"state\":\"open\",\"created_at\":\"2014-09-08T13:35:41Z\",\"updated_at\":\"2015-01-01T00:49:37Z\",\"due_on\":\"2015-01-02T08:00:00Z\",\"closed_at\":null},\"comments\":11,\"created_at\":\"2014-10-10T16:18:37Z\",\"updated_at\":\"2015-01-01T01:07:24Z\",\"closed_at\":null,\"body\":\"\"},\"comment\":{\"url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/comments/68477376\",\"html_url\":\"https://github.com/gratipay/inside.gratipay.com/issues/93#issuecomment-68477376\",\"issue_url\":\"https://api.github.com/repos/gratipay/inside.gratipay.com/issues/93\",\"id\":68477376,\"user\":{\"login\":\"whit537\",\"id\":134455,\"avatar_url\":\"https://avatars.githubusercontent.com/u/134455?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/whit537\",\"html_url\":\"https://github.com/whit537\",\"followers_url\":\"https://api.github.com/users/whit537/followers\",\"following_url\":\"https://api.github.com/users/whit537/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/whit537/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/whit537/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/whit537/subscriptions\",\"organizations_url\":\"https://api.github.com/users/whit537/orgs\",\"repos_url\":\"https://api.github.com/users/whit537/repos\",\"events_url\":\"https://api.github.com/users/whit537/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/whit537/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:24Z\",\"updated_at\":\"2015-01-01T01:07:24Z\",\"body\":\"Obviously we're going to have to spend time on #118. What are our expectations there in terms of what we'll be able to accomplish while we're together? At this point I feel like we need to be more or less open ended, take it as it comes once we're together.\\r\\n\\r\\nWhat else do we want to have on the agenda? #89 comes to mind.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:24Z\",\"org\":{\"id\":1744073,\"login\":\"gratipay\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/gratipay\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1744073?\"}}\n{\"id\":\"2489398926\",\"type\":\"PushEvent\",\"actor\":{\"id\":6278300,\"login\":\"VacioExistencial\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/VacioExistencial\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6278300?\"},\"repo\":{\"id\":28678151,\"name\":\"VacioExistencial/pong\",\"url\":\"https://api.github.com/repos/VacioExistencial/pong\"},\"payload\":{\"push_id\":536753571,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"60300f7ad699ac99855ebd2357502afb834dfe1c\",\"before\":\"978cb4883e834db8b30b6ec2edca30e231b42658\",\"commits\":[{\"sha\":\"60300f7ad699ac99855ebd2357502afb834dfe1c\",\"author\":{\"email\":\"44bdaf3b26e8e183b1d41341b9fe1b680910844f@outlook.com\",\"name\":\"Alex\"},\"message\":\"migrado a SDL2\\n\\nNo hice grandes cambios, y falta mucho por hacer.\",\"distinct\":true,\"url\":\"https://api.github.com/repos/VacioExistencial/pong/commits/60300f7ad699ac99855ebd2357502afb834dfe1c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:24Z\"}\n{\"id\":\"2489398928\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":2748277,\"login\":\"vinhkhoa\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vinhkhoa\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/2748277?\"},\"repo\":{\"id\":28330560,\"name\":\"vinhkhoa/AthTracker\",\"url\":\"https://api.github.com/repos/vinhkhoa/AthTracker\"},\"payload\":{\"action\":\"closed\",\"issue\":{\"url\":\"https://api.github.com/repos/vinhkhoa/AthTracker/issues/10\",\"labels_url\":\"https://api.github.com/repos/vinhkhoa/AthTracker/issues/10/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/vinhkhoa/AthTracker/issues/10/comments\",\"events_url\":\"https://api.github.com/repos/vinhkhoa/AthTracker/issues/10/events\",\"html_url\":\"https://github.com/vinhkhoa/AthTracker/issues/10\",\"id\":52690026,\"number\":10,\"title\":\"Extra profile fields\",\"user\":{\"login\":\"vinhkhoa\",\"id\":2748277,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2748277?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/vinhkhoa\",\"html_url\":\"https://github.com/vinhkhoa\",\"followers_url\":\"https://api.github.com/users/vinhkhoa/followers\",\"following_url\":\"https://api.github.com/users/vinhkhoa/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/vinhkhoa/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/vinhkhoa/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/vinhkhoa/subscriptions\",\"organizations_url\":\"https://api.github.com/users/vinhkhoa/orgs\",\"repos_url\":\"https://api.github.com/users/vinhkhoa/repos\",\"events_url\":\"https://api.github.com/users/vinhkhoa/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/vinhkhoa/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/vinhkhoa/AthTracker/labels/requirement\",\"name\":\"requirement\",\"color\":\"207de5\"}],\"state\":\"closed\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2014-12-22T21:22:37Z\",\"updated_at\":\"2015-01-01T01:07:24Z\",\"closed_at\":\"2015-01-01T01:07:24Z\",\"body\":\"\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:24Z\"}\n{\"id\":\"2489398929\",\"type\":\"WatchEvent\",\"actor\":{\"id\":451828,\"login\":\"minirop\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/minirop\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/451828?\"},\"repo\":{\"id\":4106171,\"name\":\"lioncash/ExtractData\",\"url\":\"https://api.github.com/repos/lioncash/ExtractData\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:25Z\"}\n{\"id\":\"2489398934\",\"type\":\"PushEvent\",\"actor\":{\"id\":8819701,\"login\":\"r-ggraham\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/r-ggraham\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8819701?\"},\"repo\":{\"id\":28678173,\"name\":\"r-ggraham/Crumpet_Bot\",\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot\"},\"payload\":{\"push_id\":536753574,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"5c6c899b455693ca53377520b3f0c73442ce97f9\",\"before\":\"ee45c3d188e25fa5ee1d0d5a8a9f3646854ccbc7\",\"commits\":[{\"sha\":\"5c6c899b455693ca53377520b3f0c73442ce97f9\",\"author\":{\"email\":\"f2f9dd43aa4244d32208a2ccfa0c7c9e9c48f7e7@uni.worc.ac.uk\",\"name\":\"Rob G\"},\"message\":\"Instructions\",\"distinct\":true,\"url\":\"https://api.github.com/repos/r-ggraham/Crumpet_Bot/commits/5c6c899b455693ca53377520b3f0c73442ce97f9\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:25Z\"}\n{\"id\":\"2489398940\",\"type\":\"MemberEvent\",\"actor\":{\"id\":932999,\"login\":\"loyos\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/loyos\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/932999?\"},\"repo\":{\"id\":28678289,\"name\":\"loyos/cemento\",\"url\":\"https://api.github.com/repos/loyos/cemento\"},\"payload\":{\"member\":{\"login\":\"yennifergrau\",\"id\":10153477,\"avatar_url\":\"https://avatars.githubusercontent.com/u/10153477?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/yennifergrau\",\"html_url\":\"https://github.com/yennifergrau\",\"followers_url\":\"https://api.github.com/users/yennifergrau/followers\",\"following_url\":\"https://api.github.com/users/yennifergrau/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/yennifergrau/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/yennifergrau/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/yennifergrau/subscriptions\",\"organizations_url\":\"https://api.github.com/users/yennifergrau/orgs\",\"repos_url\":\"https://api.github.com/users/yennifergrau/repos\",\"events_url\":\"https://api.github.com/users/yennifergrau/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/yennifergrau/received_events\",\"type\":\"User\",\"site_admin\":false},\"action\":\"added\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398943\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1013892,\"login\":\"jarofghosts\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/jarofghosts\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1013892?\"},\"repo\":{\"id\":6853358,\"name\":\"substack/tape\",\"url\":\"https://api.github.com/repos/substack/tape\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398944\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":9020086,\"login\":\"HERO4903\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/HERO4903\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9020086?\"},\"repo\":{\"id\":28678195,\"name\":\"TTMTT/iCL0udin\",\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"labels_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/comments\",\"events_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1/events\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1\",\"id\":53210206,\"number\":1,\"title\":\"Discuss1\",\"user\":{\"login\":\"TTMTT\",\"id\":6964047,\"avatar_url\":\"https://avatars.githubusercontent.com/u/6964047?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/TTMTT\",\"html_url\":\"https://github.com/TTMTT\",\"followers_url\":\"https://api.github.com/users/TTMTT/followers\",\"following_url\":\"https://api.github.com/users/TTMTT/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/TTMTT/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/TTMTT/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/TTMTT/subscriptions\",\"organizations_url\":\"https://api.github.com/users/TTMTT/orgs\",\"repos_url\":\"https://api.github.com/users/TTMTT/repos\",\"events_url\":\"https://api.github.com/users/TTMTT/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/TTMTT/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":3,\"created_at\":\"2015-01-01T01:02:34Z\",\"updated_at\":\"2015-01-01T01:07:27Z\",\"closed_at\":null,\"body\":\"Now you can download vresion 1.0 from :\\r\\n---------------------------------------------------\\r\\nhttp://www.icloudin.net\\r\\n-----------------------------\\r\\nWow, ipod touch 5G (8.1) - iCL0udin v1.0 bypass activation (icloud)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tZmEdlDGNu4\\r\\n--------------------------------------\\r\\niCL0udin v1.0 bypass activation (icloud) - ipad mini 2G (7.1.1)\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/tevYyBN2QCQ\\r\\n---------------------------------------\\r\\nVideo for bypass icloud (iCL0udin v1.0) for iphone 4 CDMA ..\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/i85-D6N2YLk\\r\\n-------------------------------------\\r\\nNew video for iCL0udin v1.0 bypass icloud (3 iphones 7.1.2):\\r\\n-------------------------------------------------------------------------------------\\r\\nhttp://youtu.be/p51TNlCr7ug\\r\\n-------------------------------------\\r\\niCL0udin v1.0 -> %100\\r\\n----------------------------\\r\\nRemaining: %3 testing with some people..\\r\\n-----------------------------------------------------\\r\\nLast Method:\\r\\n-----------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\\r\\nmethod 3 : via (change some string by hex on ELF file << some times i got error)\\r\\nmethod 4 : via (use apple ssl cert or real ssl in server and change some string in iphone)\\r\\niCL0udin v1.0 have this method:\\r\\n-----------------------------------------\\r\\n\\r\\nmethod 1 : via (other xml not to deviceservices - exploit)\\r\\nmethod 2 : via (apple cert & key and i can downgrade to any ios)\"},\"comment\":{\"url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/comments/68477378\",\"html_url\":\"https://github.com/TTMTT/iCL0udin/issues/1#issuecomment-68477378\",\"issue_url\":\"https://api.github.com/repos/TTMTT/iCL0udin/issues/1\",\"id\":68477378,\"user\":{\"login\":\"HERO4903\",\"id\":9020086,\"avatar_url\":\"https://avatars.githubusercontent.com/u/9020086?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/HERO4903\",\"html_url\":\"https://github.com/HERO4903\",\"followers_url\":\"https://api.github.com/users/HERO4903/followers\",\"following_url\":\"https://api.github.com/users/HERO4903/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/HERO4903/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/HERO4903/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/HERO4903/subscriptions\",\"organizations_url\":\"https://api.github.com/users/HERO4903/orgs\",\"repos_url\":\"https://api.github.com/users/HERO4903/repos\",\"events_url\":\"https://api.github.com/users/HERO4903/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/HERO4903/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:27Z\",\"updated_at\":\"2015-01-01T01:07:27Z\",\"body\":\"TTMTT hola una pregunta será posible que se solucione la sobrecarga en las próximas horas\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398947\",\"type\":\"CreateEvent\",\"actor\":{\"id\":5491765,\"login\":\"loganrice\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/loganrice\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5491765?\"},\"repo\":{\"id\":26188359,\"name\":\"loganrice/bkisystem\",\"url\":\"https://api.github.com/repos/loganrice/bkisystem\"},\"payload\":{\"ref\":\"commission\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398950\",\"type\":\"PushEvent\",\"actor\":{\"id\":1577452,\"login\":\"perrupa\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/perrupa\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1577452?\"},\"repo\":{\"id\":19949182,\"name\":\"perrupa/perrupa.github.io\",\"url\":\"https://api.github.com/repos/perrupa/perrupa.github.io\"},\"payload\":{\"push_id\":536753578,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"e3376d7b903228eb56abd1bde81bf4c3af3f21e7\",\"before\":\"6558647e1ac00710c0ddbbd2c080c4645be187a2\",\"commits\":[{\"sha\":\"e3376d7b903228eb56abd1bde81bf4c3af3f21e7\",\"author\":{\"email\":\"9a70e3eb66414f9b662fb64558a83b5e48ee973f@achievers.com\",\"name\":\"Chris Marlow\"},\"message\":\"Added package info\",\"distinct\":true,\"url\":\"https://api.github.com/repos/perrupa/perrupa.github.io/commits/e3376d7b903228eb56abd1bde81bf4c3af3f21e7\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398951\",\"type\":\"PushEvent\",\"actor\":{\"id\":1265899,\"login\":\"lynas\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/lynas\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1265899?\"},\"repo\":{\"id\":28678234,\"name\":\"lynas/springsecurity3.2\",\"url\":\"https://api.github.com/repos/lynas/springsecurity3.2\"},\"payload\":{\"push_id\":536753580,\"size\":2,\"distinct_size\":2,\"ref\":\"refs/heads/master\",\"head\":\"b562d8b720ffb29b808f5846ae87cbe39cd23fe6\",\"before\":\"97d11dfb3e36a479593c4dbd68087b23778161ac\",\"commits\":[{\"sha\":\"dbc9006b01fb370ed8949bd825ae6e86f7302e25\",\"author\":{\"email\":\"286cad8d9475283cbc1d36806a52beb3fd9db73b@gmail.com\",\"name\":\"lynas\"},\"message\":\"initial commit\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lynas/springsecurity3.2/commits/dbc9006b01fb370ed8949bd825ae6e86f7302e25\"},{\"sha\":\"b562d8b720ffb29b808f5846ae87cbe39cd23fe6\",\"author\":{\"email\":\"286cad8d9475283cbc1d36806a52beb3fd9db73b@gmail.com\",\"name\":\"lynas\"},\"message\":\"Merge branch 'master' of https://github.com/lynas/springsecurity3.2\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lynas/springsecurity3.2/commits/b562d8b720ffb29b808f5846ae87cbe39cd23fe6\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\"}\n{\"id\":\"2489398952\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":312296,\"login\":\"cqcallaw\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cqcallaw\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/312296?\"},\"repo\":{\"id\":14712850,\"name\":\"syncthing/syncthing\",\"url\":\"https://api.github.com/repos/syncthing/syncthing\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/syncthing/syncthing/issues/1113\",\"labels_url\":\"https://api.github.com/repos/syncthing/syncthing/issues/1113/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/syncthing/syncthing/issues/1113/comments\",\"events_url\":\"https://api.github.com/repos/syncthing/syncthing/issues/1113/events\",\"html_url\":\"https://github.com/syncthing/syncthing/issues/1113\",\"id\":51872359,\"number\":1113,\"title\":\"Flaky UPnP\",\"user\":{\"login\":\"AudriusButkevicius\",\"id\":1144861,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1144861?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AudriusButkevicius\",\"html_url\":\"https://github.com/AudriusButkevicius\",\"followers_url\":\"https://api.github.com/users/AudriusButkevicius/followers\",\"following_url\":\"https://api.github.com/users/AudriusButkevicius/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/AudriusButkevicius/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/AudriusButkevicius/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/AudriusButkevicius/subscriptions\",\"organizations_url\":\"https://api.github.com/users/AudriusButkevicius/orgs\",\"repos_url\":\"https://api.github.com/users/AudriusButkevicius/repos\",\"events_url\":\"https://api.github.com/users/AudriusButkevicius/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/AudriusButkevicius/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/syncthing/syncthing/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":15,\"created_at\":\"2014-12-13T01:51:45Z\",\"updated_at\":\"2015-01-01T01:07:26Z\",\"closed_at\":null,\"body\":\"I've managed to reproduce this weird case where UPnP does not work.\\r\\nFired up wireshark, and I can see discovery packets leaving, but I cannot see UPnP MSEARCH packets leaving my machine.\\r\\n\\r\\nI changed the broadcast address to something else within my subnet, and it starts working, meaning that either Go or Windows doesn't like the address.\\r\\n\\r\\nGiven I ran a different UPnP utility and it works, it makes me think there is something wrong with Go's stuff.\\r\\n\"},\"comment\":{\"url\":\"https://api.github.com/repos/syncthing/syncthing/issues/comments/68477377\",\"html_url\":\"https://github.com/syncthing/syncthing/issues/1113#issuecomment-68477377\",\"issue_url\":\"https://api.github.com/repos/syncthing/syncthing/issues/1113\",\"id\":68477377,\"user\":{\"login\":\"cqcallaw\",\"id\":312296,\"avatar_url\":\"https://avatars.githubusercontent.com/u/312296?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/cqcallaw\",\"html_url\":\"https://github.com/cqcallaw\",\"followers_url\":\"https://api.github.com/users/cqcallaw/followers\",\"following_url\":\"https://api.github.com/users/cqcallaw/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/cqcallaw/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/cqcallaw/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/cqcallaw/subscriptions\",\"organizations_url\":\"https://api.github.com/users/cqcallaw/orgs\",\"repos_url\":\"https://api.github.com/users/cqcallaw/repos\",\"events_url\":\"https://api.github.com/users/cqcallaw/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/cqcallaw/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:26Z\",\"updated_at\":\"2015-01-01T01:07:26Z\",\"body\":\"Interesting, I did a test with the [latest commit](https://github.com/syncthing/syncthing/tree/e94bd90782c25ef1a65cc0fe375bcafd1d398275 of syncthing) of syncthing, and the port mapping worked fine--and the discovery packets showed up in Wireshark! I really don't know what changed, but this issue doesn't seem reproducible enough to keep open. I vote to close the issue until someone can reliable reproduce it.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\",\"org\":{\"id\":7628018,\"login\":\"syncthing\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/syncthing\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/7628018?\"}}\n{\"id\":\"2489398954\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":253237,\"login\":\"Jamesking56\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?\"},\"repo\":{\"id\":26730195,\"name\":\"cachethq/Cachet\",\"url\":\"https://api.github.com/repos/cachethq/Cachet\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"labels_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/comments\",\"events_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173/events\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173\",\"id\":53210024,\"number\":173,\"title\":\"Bug: Forms let you submit multiple times\",\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":6,\"created_at\":\"2015-01-01T00:52:06Z\",\"updated_at\":\"2015-01-01T01:07:28Z\",\"closed_at\":null,\"body\":\"When adding a new incident, I noticed a weird bug.\\r\\n\\r\\nIf you fill in the form as normal, then click the submit button twice really quickly, it'll create __TWO__ identical new incidents!\\r\\n\\r\\nThis could be a bit annoying, a simple fix is using a bit of JS that on submit, disables the submit button so that once clicked, it cannot be clicked again.\"},\"comment\":{\"url\":\"https://api.github.com/repos/cachethq/Cachet/issues/comments/68477379\",\"html_url\":\"https://github.com/cachethq/Cachet/issues/173#issuecomment-68477379\",\"issue_url\":\"https://api.github.com/repos/cachethq/Cachet/issues/173\",\"id\":68477379,\"user\":{\"login\":\"Jamesking56\",\"id\":253237,\"avatar_url\":\"https://avatars.githubusercontent.com/u/253237?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/Jamesking56\",\"html_url\":\"https://github.com/Jamesking56\",\"followers_url\":\"https://api.github.com/users/Jamesking56/followers\",\"following_url\":\"https://api.github.com/users/Jamesking56/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/Jamesking56/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/Jamesking56/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Jamesking56/subscriptions\",\"organizations_url\":\"https://api.github.com/users/Jamesking56/orgs\",\"repos_url\":\"https://api.github.com/users/Jamesking56/repos\",\"events_url\":\"https://api.github.com/users/Jamesking56/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/Jamesking56/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:28Z\",\"updated_at\":\"2015-01-01T01:07:28Z\",\"body\":\"Thanks for clarifying that :+1:\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:28Z\",\"org\":{\"id\":9951502,\"login\":\"cachethq\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/cachethq\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9951502?\"}}\n{\"id\":\"2489398955\",\"type\":\"PushEvent\",\"actor\":{\"id\":8210807,\"login\":\"nodepoker\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nodepoker\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8210807?\"},\"repo\":{\"id\":22790212,\"name\":\"nodepoker/.dotfiles\",\"url\":\"https://api.github.com/repos/nodepoker/.dotfiles\"},\"payload\":{\"push_id\":536753583,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"before\":\"233774f844d59ff89420037609cf3e0cba985e9a\",\"commits\":[{\"sha\":\"0ec37f77b8811a07ca6c6df79622541d8fa40c0a\",\"author\":{\"email\":\"11f6ad8ec52a2984abaafd7c3b516503785c2072\",\"name\":\"x\"},\"message\":\".\",\"distinct\":true,\"url\":\"https://api.github.com/repos/nodepoker/.dotfiles/commits/0ec37f77b8811a07ca6c6df79622541d8fa40c0a\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:29Z\"}\n{\"id\":\"2489398958\",\"type\":\"PushEvent\",\"actor\":{\"id\":5230439,\"login\":\"movie002\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/movie002\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5230439?\"},\"repo\":{\"id\":16878288,\"name\":\"movie002/v\",\"url\":\"https://api.github.com/repos/movie002/v\"},\"payload\":{\"push_id\":536753585,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"19defc0e03ee140ce0ba3a929b5f35272bb2bc72\",\"before\":\"b5231af35019ee3489580e002c607e36c9fe78ab\",\"commits\":[{\"sha\":\"19defc0e03ee140ce0ba3a929b5f35272bb2bc72\",\"author\":{\"email\":\"e3ba41eb4a8a8591712f6252059288965f65b851@qq.com\",\"name\":\"movie002\"},\"message\":\"\",\"distinct\":true,\"url\":\"https://api.github.com/repos/movie002/v/commits/19defc0e03ee140ce0ba3a929b5f35272bb2bc72\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:31Z\"}\n{\"id\":\"2489398963\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":4566,\"login\":\"nathany\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathany\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/4566?\"},\"repo\":{\"id\":27928684,\"name\":\"go-amz/amz\",\"url\":\"https://api.github.com/repos/go-amz/amz\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/go-amz/amz/issues/5\",\"labels_url\":\"https://api.github.com/repos/go-amz/amz/issues/5/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/go-amz/amz/issues/5/comments\",\"events_url\":\"https://api.github.com/repos/go-amz/amz/issues/5/events\",\"html_url\":\"https://github.com/go-amz/amz/issues/5\",\"id\":53086813,\"number\":5,\"title\":\"I'm so confused by all the forks\",\"user\":{\"login\":\"nathany\",\"id\":4566,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4566?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathany\",\"html_url\":\"https://github.com/nathany\",\"followers_url\":\"https://api.github.com/users/nathany/followers\",\"following_url\":\"https://api.github.com/users/nathany/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathany/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathany/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathany/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathany/orgs\",\"repos_url\":\"https://api.github.com/users/nathany/repos\",\"events_url\":\"https://api.github.com/users/nathany/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathany/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":14,\"created_at\":\"2014-12-30T01:44:20Z\",\"updated_at\":\"2015-01-01T01:07:31Z\",\"closed_at\":null,\"body\":\"I want to add CloudFront cache invalidation, but to where?\\r\\n\\r\\nhttps://github.com/goamz/goamz already has some CloudFront stuff\\r\\n\\r\\nhttps://github.com/mitchellh/goamz I've been using this fork up until now (due to #4).\\r\\n\\r\\nalso https://github.com/crowdmob/goamz\"},\"comment\":{\"url\":\"https://api.github.com/repos/go-amz/amz/issues/comments/68477381\",\"html_url\":\"https://github.com/go-amz/amz/issues/5#issuecomment-68477381\",\"issue_url\":\"https://api.github.com/repos/go-amz/amz/issues/5\",\"id\":68477381,\"user\":{\"login\":\"nathany\",\"id\":4566,\"avatar_url\":\"https://avatars.githubusercontent.com/u/4566?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/nathany\",\"html_url\":\"https://github.com/nathany\",\"followers_url\":\"https://api.github.com/users/nathany/followers\",\"following_url\":\"https://api.github.com/users/nathany/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/nathany/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/nathany/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/nathany/subscriptions\",\"organizations_url\":\"https://api.github.com/users/nathany/orgs\",\"repos_url\":\"https://api.github.com/users/nathany/repos\",\"events_url\":\"https://api.github.com/users/nathany/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/nathany/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:31Z\",\"updated_at\":\"2015-01-01T01:07:31Z\",\"body\":\"Thanks @dimitern.\\r\\n\\r\\nIt may be worth looking into https://www.clahub.com/ (disclaimer, I am also just a developer) or at least having a solid CONTRIBUTING.md file. Revel has a good example of [CONTRIBUTING](https://github.com/revel/revel/blob/master/CONTRIBUTING.md).\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:31Z\",\"org\":{\"id\":8137365,\"login\":\"go-amz\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/go-amz\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8137365?\"}}\n{\"id\":\"2489398968\",\"type\":\"PushEvent\",\"actor\":{\"id\":416575,\"login\":\"frewsxcv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?\"},\"repo\":{\"id\":28678263,\"name\":\"frewsxcv/Wicket\",\"url\":\"https://api.github.com/repos/frewsxcv/Wicket\"},\"payload\":{\"push_id\":536753591,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/patch-1\",\"head\":\"b1050c9a81013c77f4dc3e266c8606e130d7b23c\",\"before\":\"ea75ffd844987843158fcdee9d98d242e962b2e2\",\"commits\":[{\"sha\":\"b1050c9a81013c77f4dc3e266c8606e130d7b23c\",\"author\":{\"email\":\"16aea3ed30350b1f4b4dfc1111a05a38e3d681b3@rwell.org\",\"name\":\"Corey Farwell\"},\"message\":\"Enable syntax highlighting in README\",\"distinct\":true,\"url\":\"https://api.github.com/repos/frewsxcv/Wicket/commits/b1050c9a81013c77f4dc3e266c8606e130d7b23c\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:31Z\"}\n{\"id\":\"2489398971\",\"type\":\"CreateEvent\",\"actor\":{\"id\":804014,\"login\":\"ry5n\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ry5n\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/804014?\"},\"repo\":{\"id\":28678287,\"name\":\"ry5n/sass-inline-svg\",\"url\":\"https://api.github.com/repos/ry5n/sass-inline-svg\"},\"payload\":{\"ref\":\"master\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"Inline an SVG as a CSS data URI. Allows recoloring paths.\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:33Z\"}\n{\"id\":\"2489398972\",\"type\":\"PushEvent\",\"actor\":{\"id\":433707,\"login\":\"ile\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/ile\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/433707?\"},\"repo\":{\"id\":26847132,\"name\":\"kantele/k-templates\",\"url\":\"https://api.github.com/repos/kantele/k-templates\"},\"payload\":{\"push_id\":536753592,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"cb1fa82b87d8144f298b3a69ba3e41788e3c4130\",\"before\":\"2250e5e1294fae3c21d3fa9b19bd162c40c29a6c\",\"commits\":[{\"sha\":\"cb1fa82b87d8144f298b3a69ba3e41788e3c4130\",\"author\":{\"email\":\"4f3407de78bccc8cc160ee4d278d5efe7162e6b5@nateps.com\",\"name\":\"Nate Smith\"},\"message\":\"0.2.5\\n\\nConflicts:\\n\\tpackage.json\",\"distinct\":true,\"url\":\"https://api.github.com/repos/kantele/k-templates/commits/cb1fa82b87d8144f298b3a69ba3e41788e3c4130\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:33Z\",\"org\":{\"id\":5687585,\"login\":\"kantele\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/kantele\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/5687585?\"}}\n{\"id\":\"2489398975\",\"type\":\"WatchEvent\",\"actor\":{\"id\":1015032,\"login\":\"miketahani\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/miketahani\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1015032?\"},\"repo\":{\"id\":26293258,\"name\":\"ThatcherC/Terrain2STL\",\"url\":\"https://api.github.com/repos/ThatcherC/Terrain2STL\"},\"payload\":{\"action\":\"started\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:33Z\"}\n{\"id\":\"2489398980\",\"type\":\"CreateEvent\",\"actor\":{\"id\":8396786,\"login\":\"barr-code\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/barr-code\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8396786?\"},\"repo\":{\"id\":28678294,\"name\":\"barr-code/craftsy\",\"url\":\"https://api.github.com/repos/barr-code/craftsy\"},\"payload\":{\"ref\":null,\"ref_type\":\"repository\",\"master_branch\":\"master\",\"description\":\"\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:33Z\"}\n{\"id\":\"2489398985\",\"type\":\"PushEvent\",\"actor\":{\"id\":10237784,\"login\":\"garylocke\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/garylocke\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10237784?\"},\"repo\":{\"id\":28205420,\"name\":\"garylocke/ngkarma\",\"url\":\"https://api.github.com/repos/garylocke/ngkarma\"},\"payload\":{\"push_id\":536753602,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"da80da5ad685e455c3b6552b06cade3cfe3d1faf\",\"before\":\"9429d660ddb0db02b87c2e27710359812da71082\",\"commits\":[{\"sha\":\"da80da5ad685e455c3b6552b06cade3cfe3d1faf\",\"author\":{\"email\":\"eb1c831f82e9d7475fabb08c6ce27c71c685a278@Garys-MacBook-Pro.local\",\"name\":\"Gary Locke\"},\"message\":\"Removed robots.txt\",\"distinct\":true,\"url\":\"https://api.github.com/repos/garylocke/ngkarma/commits/da80da5ad685e455c3b6552b06cade3cfe3d1faf\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:34Z\"}\n{\"id\":\"2489398988\",\"type\":\"CreateEvent\",\"actor\":{\"id\":109109,\"login\":\"rade\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rade\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/109109?\"},\"repo\":{\"id\":25579630,\"name\":\"rade/weave\",\"url\":\"https://api.github.com/repos/rade/weave\"},\"payload\":{\"ref\":\"101_eliminate_dependencies\",\"ref_type\":\"branch\",\"master_branch\":\"master\",\"description\":\"The Docker Network\",\"pusher_type\":\"user\"},\"public\":true,\"created_at\":\"2015-01-01T01:07:36Z\"}\n{\"id\":\"2489398992\",\"type\":\"PushEvent\",\"actor\":{\"id\":109109,\"login\":\"rade\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/rade\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/109109?\"},\"repo\":{\"id\":25579630,\"name\":\"rade/weave\",\"url\":\"https://api.github.com/repos/rade/weave\"},\"payload\":{\"push_id\":536753607,\"size\":1,\"distinct_size\":0,\"ref\":\"refs/heads/master\",\"head\":\"5da2219dc8ca9bc8de9f874429a7d5945e3e34ca\",\"before\":\"fd6f65e28fae31e0a72bfc6ef5046eaeedb12e31\",\"commits\":[{\"sha\":\"5da2219dc8ca9bc8de9f874429a7d5945e3e34ca\",\"author\":{\"email\":\"ac8fcaf748b7189c49a384a5580dc9e80633299a@gmail.com\",\"name\":\"Matthias Radestock\"},\"message\":\"cosmetic: remove weaver/.gitignore\\nand handle that in the top-level .gitignore instead. Just as we do for weavedns.\",\"distinct\":false,\"url\":\"https://api.github.com/repos/rade/weave/commits/5da2219dc8ca9bc8de9f874429a7d5945e3e34ca\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:36Z\"}\n{\"id\":\"2489398994\",\"type\":\"IssueCommentEvent\",\"actor\":{\"id\":1061610,\"login\":\"kokizzu\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kokizzu\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/1061610?\"},\"repo\":{\"id\":3725405,\"name\":\"thirtyseven/dullard\",\"url\":\"https://api.github.com/repos/thirtyseven/dullard\"},\"payload\":{\"action\":\"created\",\"issue\":{\"url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/12\",\"labels_url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/12/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/12/comments\",\"events_url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/12/events\",\"html_url\":\"https://github.com/thirtyseven/dullard/issues/12\",\"id\":47912635,\"number\":12,\"title\":\"Dullard doesn't show nil on beginning of a row\",\"user\":{\"login\":\"kokizzu\",\"id\":1061610,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1061610?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kokizzu\",\"html_url\":\"https://github.com/kokizzu\",\"followers_url\":\"https://api.github.com/users/kokizzu/followers\",\"following_url\":\"https://api.github.com/users/kokizzu/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kokizzu/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kokizzu/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kokizzu/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kokizzu/orgs\",\"repos_url\":\"https://api.github.com/users/kokizzu/repos\",\"events_url\":\"https://api.github.com/users/kokizzu/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kokizzu/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[],\"state\":\"open\",\"locked\":false,\"assignee\":{\"login\":\"thirtyseven\",\"id\":123678,\"avatar_url\":\"https://avatars.githubusercontent.com/u/123678?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/thirtyseven\",\"html_url\":\"https://github.com/thirtyseven\",\"followers_url\":\"https://api.github.com/users/thirtyseven/followers\",\"following_url\":\"https://api.github.com/users/thirtyseven/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/thirtyseven/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/thirtyseven/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/thirtyseven/subscriptions\",\"organizations_url\":\"https://api.github.com/users/thirtyseven/orgs\",\"repos_url\":\"https://api.github.com/users/thirtyseven/repos\",\"events_url\":\"https://api.github.com/users/thirtyseven/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/thirtyseven/received_events\",\"type\":\"User\",\"site_admin\":false},\"milestone\":null,\"comments\":6,\"created_at\":\"2014-11-06T01:27:57Z\",\"updated_at\":\"2015-01-01T01:07:36Z\",\"closed_at\":null,\"body\":\"expected behavior:\\r\\n```\\r\\n# one row:\\r\\n[ nil, nil, nil, 1]\\r\\n```\\r\\n\\r\\ncurrent behavior:\\r\\n```\\r\\n[ 1 ]\\r\\n```\"},\"comment\":{\"url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/comments/68477383\",\"html_url\":\"https://github.com/thirtyseven/dullard/issues/12#issuecomment-68477383\",\"issue_url\":\"https://api.github.com/repos/thirtyseven/dullard/issues/12\",\"id\":68477383,\"user\":{\"login\":\"kokizzu\",\"id\":1061610,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1061610?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/kokizzu\",\"html_url\":\"https://github.com/kokizzu\",\"followers_url\":\"https://api.github.com/users/kokizzu/followers\",\"following_url\":\"https://api.github.com/users/kokizzu/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/kokizzu/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/kokizzu/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/kokizzu/subscriptions\",\"organizations_url\":\"https://api.github.com/users/kokizzu/orgs\",\"repos_url\":\"https://api.github.com/users/kokizzu/repos\",\"events_url\":\"https://api.github.com/users/kokizzu/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/kokizzu/received_events\",\"type\":\"User\",\"site_admin\":false},\"created_at\":\"2015-01-01T01:07:36Z\",\"updated_at\":\"2015-01-01T01:07:36Z\",\"body\":\"just create an empty spreadsheet file with cell A1 contains 1, B2 contains 2, C3 contains 3..\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:38Z\"}\n{\"id\":\"2489398996\",\"type\":\"IssuesEvent\",\"actor\":{\"id\":3964339,\"login\":\"dcartertwo\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcartertwo\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/3964339?\"},\"repo\":{\"id\":25189570,\"name\":\"oceans15/teleport\",\"url\":\"https://api.github.com/repos/oceans15/teleport\"},\"payload\":{\"action\":\"opened\",\"issue\":{\"url\":\"https://api.github.com/repos/oceans15/teleport/issues/52\",\"labels_url\":\"https://api.github.com/repos/oceans15/teleport/issues/52/labels{/name}\",\"comments_url\":\"https://api.github.com/repos/oceans15/teleport/issues/52/comments\",\"events_url\":\"https://api.github.com/repos/oceans15/teleport/issues/52/events\",\"html_url\":\"https://github.com/oceans15/teleport/issues/52\",\"id\":53210306,\"number\":52,\"title\":\"Crashes when receiving notification\",\"user\":{\"login\":\"dcartertwo\",\"id\":3964339,\"avatar_url\":\"https://avatars.githubusercontent.com/u/3964339?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/dcartertwo\",\"html_url\":\"https://github.com/dcartertwo\",\"followers_url\":\"https://api.github.com/users/dcartertwo/followers\",\"following_url\":\"https://api.github.com/users/dcartertwo/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/dcartertwo/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/dcartertwo/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/dcartertwo/subscriptions\",\"organizations_url\":\"https://api.github.com/users/dcartertwo/orgs\",\"repos_url\":\"https://api.github.com/users/dcartertwo/repos\",\"events_url\":\"https://api.github.com/users/dcartertwo/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/dcartertwo/received_events\",\"type\":\"User\",\"site_admin\":false},\"labels\":[{\"url\":\"https://api.github.com/repos/oceans15/teleport/labels/bug\",\"name\":\"bug\",\"color\":\"fc2929\"},{\"url\":\"https://api.github.com/repos/oceans15/teleport/labels/help+wanted\",\"name\":\"help wanted\",\"color\":\"159818\"}],\"state\":\"open\",\"locked\":false,\"assignee\":null,\"milestone\":null,\"comments\":0,\"created_at\":\"2015-01-01T01:07:36Z\",\"updated_at\":\"2015-01-01T01:07:36Z\",\"closed_at\":null,\"body\":\"To replicate (in android-notifications branch)\\r\\n-Start the app\\r\\n-Push notification\\r\\n-Close app\\r\\n-Reopen app\\r\\n\\r\\nThe app crashes after a second or instantly.  Running adb logcat i found:\\r\\n```\\r\\nE/AndroidRuntime(  424): FATAL EXCEPTION: main\\r\\nE/AndroidRuntime(  424): Process: com.oceans15.channel37, PID: 424\\r\\nE/AndroidRuntime(  424): java.lang.RuntimeException: Unable to instantiate receiver com.parse.GcmBroadcastReceiver: java.lang.ClassNotFoundException: Didn't find class \\\"com.parse.GcmBroadcastReceiver\\\" on path: DexPathList[[zip file \\\"/data/app/com.oceans15.channel37-1.apk\\\"],nativeLibraryDirectories=[/data/app-lib/com.oceans15.channel37-1, /vendor/lib, /system/lib]]\\r\\nE/AndroidRuntime(  424): \\tat android.app.ActivityThread.handleReceiver(ActivityThread.java:2518)\\r\\nE/AndroidRuntime(  424): \\tat android.app.ActivityThread.access$1800(ActivityThread.java:161)\\r\\nE/AndroidRuntime(  424): \\tat android.app.ActivityThread$H.handleMessage(ActivityThread.java:1341)\\r\\nE/AndroidRuntime(  424): \\tat android.os.Handler.dispatchMessage(Handler.java:102)\\r\\nE/AndroidRuntime(  424): \\tat android.os.Looper.loop(Looper.java:157)\\r\\nE/AndroidRuntime(  424): \\tat android.app.ActivityThread.main(ActivityThread.java:5356)\\r\\nE/AndroidRuntime(  424): \\tat java.lang.reflect.Method.invokeNative(Native Method)\\r\\nE/AndroidRuntime(  424): \\tat java.lang.reflect.Method.invoke(Method.java:515)\\r\\nE/AndroidRuntime(  424): \\tat com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1265)\\r\\nE/AndroidRuntime(  424): \\tat com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1081)\\r\\nE/AndroidRuntime(  424): \\tat dalvik.system.NativeStart.main(Native Method)\\r\\nE/AndroidRuntime(  424): Caused by: java.lang.ClassNotFoundException: Didn't find class \\\"com.parse.GcmBroadcastReceiver\\\" on path: DexPathList[[zip file \\\"/data/app/com.oceans15.channel37-1.apk\\\"],nativeLibraryDirectories=[/data/app-lib/com.oceans15.channel37-1, /vendor/lib, /system/lib]]\\r\\nE/AndroidRuntime(  424): \\tat dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:67)\\r\\nE/AndroidRuntime(  424): \\tat java.lang.ClassLoader.loadClass(ClassLoader.java:497)\\r\\nE/AndroidRuntime(  424): \\tat java.lang.ClassLoader.loadClass(ClassLoader.java:457)\\r\\nE/AndroidRuntime(  424): \\tat android.app.ActivityThread.handleReceiver(ActivityThread.java:2513)\\r\\nE/AndroidRuntime(  424): \\t... 10 more\\r\\nW/ActivityManager(  817):   Force finishing activity com.oceans15.channel37/.CordovaApp\\r\\n```\\r\\n\\r\\nNot sure why the java class isn't getting instantiated properly.\"}},\"public\":true,\"created_at\":\"2015-01-01T01:07:38Z\",\"org\":{\"id\":9125251,\"login\":\"oceans15\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/oceans15\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/9125251?\"}}\n{\"id\":\"2489399000\",\"type\":\"PullRequestReviewCommentEvent\",\"actor\":{\"id\":523287,\"login\":\"j2sol\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?\"},\"repo\":{\"id\":11848896,\"name\":\"blueboxgroup/ursula\",\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\"},\"payload\":{\"action\":\"created\",\"comment\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397310\",\"id\":22397310,\"diff_hunk\":\"@@ -1,121 +1,92 @@\\n ---\\n-- name: install apache\\n-  apt: pkg={{ item }}\\n-  with_items:\\n-    - apache2\\n-    - libapache2-mod-wsgi\\n-\\n - name: lesscpy must be in apache PATH\\n   pip: name=lesscpy version=0.9j\\n \\n - name: get horizon source repo\\n-  git: repo={{ openstack.git_mirror }}/horizon.git\\n-       dest=/opt/stack/horizon\\n-       version={{ horizon.rev }}\\n-       update={{ openstack.git_update }}\\n+  git: |\\n+    repo={{ openstack.git_mirror}}/horizon.git\\n+    dest=/opt/stack/horizon\\n+    version={{ horizon.rev }}\\n+    update={{ openstack.git_update }}\\n   notify:\\n     - setup horizon venv\\n     - compress horizon assets\\n \\n-- template: src=opt/stack/horizon/hide-external-networks.patch dest=/opt/stack/horizon/hide-external-networks.patch mode=0644\\n-- shell: patch -p1 < hide-external-networks.patch chdir=/opt/stack/horizon\\n-  notify:\\n-    - setup horizon venv\\n-\\n - name: add python-memcached to horizon requirements\\n-  lineinfile: dest=/opt/stack/horizon/requirements.txt\\n-              regexp=^python-memcached\\n-              line=python-memcached\\n+  lineinfile: dest=/opt/stack/horizon/requirements.txt regexp=^python-memcached line=python-memcached\\n   notify:\\n     - setup horizon venv\\n \\n-- name: disable apache status\\n-  command: a2dismod status\\n-  notify:\\n-    - restart apache\\n-\\n-- name: apache ports config\\n-  template: src=etc/apache2/ports.conf\\n-            dest=/etc/apache2/ports.conf\\n-  notify:\\n-    - restart apache\\n-\\n-- name: disable default apache site\\n-  command: a2dissite 000-default\\n+- name: make sure apache knows about horizon ports\\n+  lineinfile: dest=/etc/apache2/ports.conf line=\\\"Listen 8080\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config (12.04)\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard\\n+- name: create dashboard virtualhost on precise\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard\\n   when: ansible_distribution_version == \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n-- name: openstack dashboard config\\n-  template: src=etc/apache2/sites-available/openstack_dashboard.conf\\n-            dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n+- name: create dashboard virtualhost on other\\n+  template: |\\n+    src=etc/apache2/sites-available/openstack_dashboard.conf\\n+    dest=/etc/apache2/sites-available/openstack_dashboard.conf\\n   when: ansible_distribution_version != \\\"12.04\\\"\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: enable horizon apache site\\n-  command: a2ensite openstack_dashboard\\n+  apache2_site: state=enabled name=openstack_dashboard\\n   notify:\\n-    - restart apache\\n-\\n-- name: static asset directories\\n-  file: dest={{ item }}\\n-        state=directory\\n-        owner=www-data\\n-        group=www-data\\n-        mode=0755\\n+    - reload apache\\n+\\n+- name: create static asset dirs\\n+  file: |\\n+    dest={{ item }}\\n+    state=directory\\n+    owner=www-data\\n+    group=www-data\\n+    mode=0755\\n   with_items:\\n     - /opt/stack/horizon/static\\n     - /opt/stack/horizon/static/dashboard\\n \\n-- name: horizon local settings\\n-  template: src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n-            mode=0644\\n+- name: dashboard settings\\n+  template: |\\n+    src=opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    dest=/opt/stack/horizon/openstack_dashboard/local/local_settings.py\\n+    mode=0644\\n   notify:\\n-    - restart apache\\n+    - reload apache\\n \\n - name: custom horizon logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo.png mode=0644 force=yes\\n \\n - name: custom horizon splash logo\\n-  get_url: url={{ horizon.logo_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png\\n-           mode=0644 force=yes\\n+  get_url: url={{ horizon.logo_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/logo-splash.png mode=0644 force=yes\\n \\n - name: custom horizon favicon\\n-  get_url: url={{ horizon.favicon_url }}\\n-           dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico\\n-           force=yes\\n+  get_url: url={{ horizon.favicon_url }} dest=/opt/stack/horizon/openstack_dashboard/static/dashboard/img/favicon.ico force=yes\\n \\n - name: put images and fonts where apache can find them\\n-  file: src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n-        dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n-        state=link\\n-        owner=www-data\\n-        group=www-data\\n+  file: |\\n+    src=/opt/stack/horizon/openstack_dashboard/static/dashboard/{{ item }}\\n+    dest=/opt/stack/horizon/static/dashboard/{{ item }}\\n+    state=link\\n+    owner=www-data\\n+    group=www-data\\n   with_items:\\n     - img\\n     - fonts\\n \\n-# flush before ensuring apache running, saves immediate restart\\n-- meta: flush_handlers\\n-\\n-- name: ensure apache started\\n+- name: ensure apache is running\\n   service: name=apache2 state=started\\n \\n-- name: Permit HTTP and HTTPS\\n-  ufw: rule=allow to_port={{ item }} proto=tcp\\n-  with_items:\\n-  - 80\\n-  - 443\\n+- name: Permit HTTP (redirect to HTTPS for Horizon)\\n+  command: ufw allow 80/tcp\\n \\n-- include: monitoring.yml tags=monitoring,common\\n+- name: Permit HTTPS (Horizon)\\n+  command: ufw allow 443/tcp\",\"path\":\"roles/horizon/tasks/main.yml\",\"position\":170,\"original_position\":170,\"commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"original_commit_id\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"j2sol\",\"id\":523287,\"avatar_url\":\"https://avatars.githubusercontent.com/u/523287?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/j2sol\",\"html_url\":\"https://github.com/j2sol\",\"followers_url\":\"https://api.github.com/users/j2sol/followers\",\"following_url\":\"https://api.github.com/users/j2sol/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/j2sol/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/j2sol/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/j2sol/subscriptions\",\"organizations_url\":\"https://api.github.com/users/j2sol/orgs\",\"repos_url\":\"https://api.github.com/users/j2sol/repos\",\"events_url\":\"https://api.github.com/users/j2sol/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/j2sol/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"Why did this go from a single task with list of ports to two tasks?\",\"created_at\":\"2015-01-01T01:07:36Z\",\"updated_at\":\"2015-01-01T01:07:36Z\",\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397310\",\"pull_request_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/22397310\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705#discussion_r22397310\"},\"pull_request\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"}}},\"pull_request\":{\"url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\",\"id\":26738851,\"html_url\":\"https://github.com/blueboxgroup/ursula/pull/705\",\"diff_url\":\"https://github.com/blueboxgroup/ursula/pull/705.diff\",\"patch_url\":\"https://github.com/blueboxgroup/ursula/pull/705.patch\",\"issue_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\",\"number\":705,\"state\":\"open\",\"locked\":false,\"title\":\"WIP:  apache for loadbalancing\",\"user\":{\"login\":\"paulczar\",\"id\":2488346,\"avatar_url\":\"https://avatars.githubusercontent.com/u/2488346?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/paulczar\",\"html_url\":\"https://github.com/paulczar\",\"followers_url\":\"https://api.github.com/users/paulczar/followers\",\"following_url\":\"https://api.github.com/users/paulczar/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/paulczar/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/paulczar/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/paulczar/subscriptions\",\"organizations_url\":\"https://api.github.com/users/paulczar/orgs\",\"repos_url\":\"https://api.github.com/users/paulczar/repos\",\"events_url\":\"https://api.github.com/users/paulczar/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/paulczar/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"This is a quick POC / WIP to demo using apache instead of haproxy for loadbalancing our APIs.\\r\\n\\r\\nI think this gives us more flexibiltiy as we can enable in the APIs role itself ( in this case in keystone ) rather than a big monolithic haproxy.conf.      We also already have apache running for horizon, so it could reduce number of services.    apache also has better logging options that haproxy which can only log to syslog. \",\"created_at\":\"2015-01-01T00:04:13Z\",\"updated_at\":\"2015-01-01T01:07:36Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":\"9b91693f11e166c9ee53836f19697868d412bf76\",\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\",\"review_comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\",\"review_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\",\"head\":{\"label\":\"blueboxgroup:use_apache_for_lb\",\"ref\":\"use_apache_for_lb\",\"sha\":\"2a6f35313b9936ce4450aba823d09287626bed6f\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"base\":{\"label\":\"blueboxgroup:master\",\"ref\":\"master\",\"sha\":\"34b83c65ff0de2f8b006d8ce4f76919fe0167bbf\",\"user\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"repo\":{\"id\":11848896,\"name\":\"ursula\",\"full_name\":\"blueboxgroup/ursula\",\"owner\":{\"login\":\"blueboxgroup\",\"id\":458705,\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/blueboxgroup\",\"html_url\":\"https://github.com/blueboxgroup\",\"followers_url\":\"https://api.github.com/users/blueboxgroup/followers\",\"following_url\":\"https://api.github.com/users/blueboxgroup/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/blueboxgroup/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/blueboxgroup/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/blueboxgroup/subscriptions\",\"organizations_url\":\"https://api.github.com/users/blueboxgroup/orgs\",\"repos_url\":\"https://api.github.com/users/blueboxgroup/repos\",\"events_url\":\"https://api.github.com/users/blueboxgroup/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/blueboxgroup/received_events\",\"type\":\"Organization\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/blueboxgroup/ursula\",\"description\":\"Ansible playbooks for operating OpenStack\",\"fork\":false,\"url\":\"https://api.github.com/repos/blueboxgroup/ursula\",\"forks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/forks\",\"keys_url\":\"https://api.github.com/repos/blueboxgroup/ursula/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/blueboxgroup/ursula/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/blueboxgroup/ursula/teams\",\"hooks_url\":\"https://api.github.com/repos/blueboxgroup/ursula/hooks\",\"issue_events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/blueboxgroup/ursula/events\",\"assignees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/blueboxgroup/ursula/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/tags\",\"blobs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/blueboxgroup/ursula/languages\",\"stargazers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/stargazers\",\"contributors_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contributors\",\"subscribers_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscribers\",\"subscription_url\":\"https://api.github.com/repos/blueboxgroup/ursula/subscription\",\"commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/blueboxgroup/ursula/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/blueboxgroup/ursula/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/blueboxgroup/ursula/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/blueboxgroup/ursula/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/blueboxgroup/ursula/merges\",\"archive_url\":\"https://api.github.com/repos/blueboxgroup/ursula/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/blueboxgroup/ursula/downloads\",\"issues_url\":\"https://api.github.com/repos/blueboxgroup/ursula/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/blueboxgroup/ursula/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/blueboxgroup/ursula/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/blueboxgroup/ursula/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/blueboxgroup/ursula/releases{/id}\",\"created_at\":\"2013-08-02T17:37:26Z\",\"updated_at\":\"2014-12-31T21:03:28Z\",\"pushed_at\":\"2015-01-01T00:01:35Z\",\"git_url\":\"git://github.com/blueboxgroup/ursula.git\",\"ssh_url\":\"git@github.com:blueboxgroup/ursula.git\",\"clone_url\":\"https://github.com/blueboxgroup/ursula.git\",\"svn_url\":\"https://github.com/blueboxgroup/ursula\",\"homepage\":null,\"size\":9036,\"stargazers_count\":45,\"watchers_count\":45,\"language\":\"Python\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":false,\"forks_count\":23,\"mirror_url\":null,\"open_issues_count\":26,\"forks\":23,\"open_issues\":26,\"watchers\":45,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705\"},\"html\":{\"href\":\"https://github.com/blueboxgroup/ursula/pull/705\"},\"issue\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705\"},\"comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/issues/705/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/pulls/705/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/blueboxgroup/ursula/statuses/2a6f35313b9936ce4450aba823d09287626bed6f\"}}}},\"public\":true,\"created_at\":\"2015-01-01T01:07:36Z\",\"org\":{\"id\":458705,\"login\":\"blueboxgroup\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/blueboxgroup\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/458705?\"}}\n{\"id\":\"2489399003\",\"type\":\"PushEvent\",\"actor\":{\"id\":6355392,\"login\":\"githanwang1\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/githanwang1\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6355392?\"},\"repo\":{\"id\":28543231,\"name\":\"githanwang1/django-blog\",\"url\":\"https://api.github.com/repos/githanwang1/django-blog\"},\"payload\":{\"push_id\":536753608,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/master\",\"head\":\"1bd9937ada4d820b74e83c8a5d68c9aa2112957d\",\"before\":\"130cfc38c0c001e58f84398948b72b138ccce717\",\"commits\":[{\"sha\":\"1bd9937ada4d820b74e83c8a5d68c9aa2112957d\",\"author\":{\"email\":\"053e32d42d025177f9df81fc22020283a55f18ff@berkeley.edu\",\"name\":\"Han Wang\"},\"message\":\"initialized at proj root\",\"distinct\":true,\"url\":\"https://api.github.com/repos/githanwang1/django-blog/commits/1bd9937ada4d820b74e83c8a5d68c9aa2112957d\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:38Z\"}\n{\"id\":\"2489399006\",\"type\":\"PullRequestEvent\",\"actor\":{\"id\":416575,\"login\":\"frewsxcv\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?\"},\"repo\":{\"id\":3545112,\"name\":\"arthur-e/Wicket\",\"url\":\"https://api.github.com/repos/arthur-e/Wicket\"},\"payload\":{\"action\":\"opened\",\"number\":60,\"pull_request\":{\"url\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60\",\"id\":26739469,\"html_url\":\"https://github.com/arthur-e/Wicket/pull/60\",\"diff_url\":\"https://github.com/arthur-e/Wicket/pull/60.diff\",\"patch_url\":\"https://github.com/arthur-e/Wicket/pull/60.patch\",\"issue_url\":\"https://api.github.com/repos/arthur-e/Wicket/issues/60\",\"number\":60,\"state\":\"open\",\"locked\":false,\"title\":\"Enable syntax highlighting in README\",\"user\":{\"login\":\"frewsxcv\",\"id\":416575,\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"html_url\":\"https://github.com/frewsxcv\",\"followers_url\":\"https://api.github.com/users/frewsxcv/followers\",\"following_url\":\"https://api.github.com/users/frewsxcv/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/frewsxcv/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/frewsxcv/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/frewsxcv/subscriptions\",\"organizations_url\":\"https://api.github.com/users/frewsxcv/orgs\",\"repos_url\":\"https://api.github.com/users/frewsxcv/repos\",\"events_url\":\"https://api.github.com/users/frewsxcv/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/frewsxcv/received_events\",\"type\":\"User\",\"site_admin\":false},\"body\":\"\",\"created_at\":\"2015-01-01T01:07:38Z\",\"updated_at\":\"2015-01-01T01:07:38Z\",\"closed_at\":null,\"merged_at\":null,\"merge_commit_sha\":null,\"assignee\":null,\"milestone\":null,\"commits_url\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60/commits\",\"review_comments_url\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60/comments\",\"review_comment_url\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/comments/{number}\",\"comments_url\":\"https://api.github.com/repos/arthur-e/Wicket/issues/60/comments\",\"statuses_url\":\"https://api.github.com/repos/arthur-e/Wicket/statuses/b1050c9a81013c77f4dc3e266c8606e130d7b23c\",\"head\":{\"label\":\"frewsxcv:patch-1\",\"ref\":\"patch-1\",\"sha\":\"b1050c9a81013c77f4dc3e266c8606e130d7b23c\",\"user\":{\"login\":\"frewsxcv\",\"id\":416575,\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"html_url\":\"https://github.com/frewsxcv\",\"followers_url\":\"https://api.github.com/users/frewsxcv/followers\",\"following_url\":\"https://api.github.com/users/frewsxcv/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/frewsxcv/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/frewsxcv/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/frewsxcv/subscriptions\",\"organizations_url\":\"https://api.github.com/users/frewsxcv/orgs\",\"repos_url\":\"https://api.github.com/users/frewsxcv/repos\",\"events_url\":\"https://api.github.com/users/frewsxcv/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/frewsxcv/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":28678263,\"name\":\"Wicket\",\"full_name\":\"frewsxcv/Wicket\",\"owner\":{\"login\":\"frewsxcv\",\"id\":416575,\"avatar_url\":\"https://avatars.githubusercontent.com/u/416575?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/frewsxcv\",\"html_url\":\"https://github.com/frewsxcv\",\"followers_url\":\"https://api.github.com/users/frewsxcv/followers\",\"following_url\":\"https://api.github.com/users/frewsxcv/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/frewsxcv/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/frewsxcv/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/frewsxcv/subscriptions\",\"organizations_url\":\"https://api.github.com/users/frewsxcv/orgs\",\"repos_url\":\"https://api.github.com/users/frewsxcv/repos\",\"events_url\":\"https://api.github.com/users/frewsxcv/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/frewsxcv/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/frewsxcv/Wicket\",\"description\":\"A modest library for moving between Well-Known Text (WKT) and various framework geometries\",\"fork\":true,\"url\":\"https://api.github.com/repos/frewsxcv/Wicket\",\"forks_url\":\"https://api.github.com/repos/frewsxcv/Wicket/forks\",\"keys_url\":\"https://api.github.com/repos/frewsxcv/Wicket/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/frewsxcv/Wicket/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/frewsxcv/Wicket/teams\",\"hooks_url\":\"https://api.github.com/repos/frewsxcv/Wicket/hooks\",\"issue_events_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/frewsxcv/Wicket/events\",\"assignees_url\":\"https://api.github.com/repos/frewsxcv/Wicket/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/frewsxcv/Wicket/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/frewsxcv/Wicket/tags\",\"blobs_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/frewsxcv/Wicket/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/frewsxcv/Wicket/languages\",\"stargazers_url\":\"https://api.github.com/repos/frewsxcv/Wicket/stargazers\",\"contributors_url\":\"https://api.github.com/repos/frewsxcv/Wicket/contributors\",\"subscribers_url\":\"https://api.github.com/repos/frewsxcv/Wicket/subscribers\",\"subscription_url\":\"https://api.github.com/repos/frewsxcv/Wicket/subscription\",\"commits_url\":\"https://api.github.com/repos/frewsxcv/Wicket/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/frewsxcv/Wicket/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/frewsxcv/Wicket/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/frewsxcv/Wicket/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/frewsxcv/Wicket/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/frewsxcv/Wicket/merges\",\"archive_url\":\"https://api.github.com/repos/frewsxcv/Wicket/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/frewsxcv/Wicket/downloads\",\"issues_url\":\"https://api.github.com/repos/frewsxcv/Wicket/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/frewsxcv/Wicket/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/frewsxcv/Wicket/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/frewsxcv/Wicket/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/frewsxcv/Wicket/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/frewsxcv/Wicket/releases{/id}\",\"created_at\":\"2015-01-01T01:04:48Z\",\"updated_at\":\"2015-01-01T01:04:49Z\",\"pushed_at\":\"2015-01-01T01:07:31Z\",\"git_url\":\"git://github.com/frewsxcv/Wicket.git\",\"ssh_url\":\"git@github.com:frewsxcv/Wicket.git\",\"clone_url\":\"https://github.com/frewsxcv/Wicket.git\",\"svn_url\":\"https://github.com/frewsxcv/Wicket\",\"homepage\":\"http://arthur-e.github.com/Wicket\",\"size\":1032,\"stargazers_count\":0,\"watchers_count\":0,\"language\":\"JavaScript\",\"has_issues\":false,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":0,\"mirror_url\":null,\"open_issues_count\":0,\"forks\":0,\"open_issues\":0,\"watchers\":0,\"default_branch\":\"master\"}},\"base\":{\"label\":\"arthur-e:master\",\"ref\":\"master\",\"sha\":\"ea75ffd844987843158fcdee9d98d242e962b2e2\",\"user\":{\"login\":\"arthur-e\",\"id\":1211103,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1211103?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/arthur-e\",\"html_url\":\"https://github.com/arthur-e\",\"followers_url\":\"https://api.github.com/users/arthur-e/followers\",\"following_url\":\"https://api.github.com/users/arthur-e/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/arthur-e/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/arthur-e/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/arthur-e/subscriptions\",\"organizations_url\":\"https://api.github.com/users/arthur-e/orgs\",\"repos_url\":\"https://api.github.com/users/arthur-e/repos\",\"events_url\":\"https://api.github.com/users/arthur-e/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/arthur-e/received_events\",\"type\":\"User\",\"site_admin\":false},\"repo\":{\"id\":3545112,\"name\":\"Wicket\",\"full_name\":\"arthur-e/Wicket\",\"owner\":{\"login\":\"arthur-e\",\"id\":1211103,\"avatar_url\":\"https://avatars.githubusercontent.com/u/1211103?v=3\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/arthur-e\",\"html_url\":\"https://github.com/arthur-e\",\"followers_url\":\"https://api.github.com/users/arthur-e/followers\",\"following_url\":\"https://api.github.com/users/arthur-e/following{/other_user}\",\"gists_url\":\"https://api.github.com/users/arthur-e/gists{/gist_id}\",\"starred_url\":\"https://api.github.com/users/arthur-e/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/arthur-e/subscriptions\",\"organizations_url\":\"https://api.github.com/users/arthur-e/orgs\",\"repos_url\":\"https://api.github.com/users/arthur-e/repos\",\"events_url\":\"https://api.github.com/users/arthur-e/events{/privacy}\",\"received_events_url\":\"https://api.github.com/users/arthur-e/received_events\",\"type\":\"User\",\"site_admin\":false},\"private\":false,\"html_url\":\"https://github.com/arthur-e/Wicket\",\"description\":\"A modest library for moving between Well-Known Text (WKT) and various framework geometries\",\"fork\":false,\"url\":\"https://api.github.com/repos/arthur-e/Wicket\",\"forks_url\":\"https://api.github.com/repos/arthur-e/Wicket/forks\",\"keys_url\":\"https://api.github.com/repos/arthur-e/Wicket/keys{/key_id}\",\"collaborators_url\":\"https://api.github.com/repos/arthur-e/Wicket/collaborators{/collaborator}\",\"teams_url\":\"https://api.github.com/repos/arthur-e/Wicket/teams\",\"hooks_url\":\"https://api.github.com/repos/arthur-e/Wicket/hooks\",\"issue_events_url\":\"https://api.github.com/repos/arthur-e/Wicket/issues/events{/number}\",\"events_url\":\"https://api.github.com/repos/arthur-e/Wicket/events\",\"assignees_url\":\"https://api.github.com/repos/arthur-e/Wicket/assignees{/user}\",\"branches_url\":\"https://api.github.com/repos/arthur-e/Wicket/branches{/branch}\",\"tags_url\":\"https://api.github.com/repos/arthur-e/Wicket/tags\",\"blobs_url\":\"https://api.github.com/repos/arthur-e/Wicket/git/blobs{/sha}\",\"git_tags_url\":\"https://api.github.com/repos/arthur-e/Wicket/git/tags{/sha}\",\"git_refs_url\":\"https://api.github.com/repos/arthur-e/Wicket/git/refs{/sha}\",\"trees_url\":\"https://api.github.com/repos/arthur-e/Wicket/git/trees{/sha}\",\"statuses_url\":\"https://api.github.com/repos/arthur-e/Wicket/statuses/{sha}\",\"languages_url\":\"https://api.github.com/repos/arthur-e/Wicket/languages\",\"stargazers_url\":\"https://api.github.com/repos/arthur-e/Wicket/stargazers\",\"contributors_url\":\"https://api.github.com/repos/arthur-e/Wicket/contributors\",\"subscribers_url\":\"https://api.github.com/repos/arthur-e/Wicket/subscribers\",\"subscription_url\":\"https://api.github.com/repos/arthur-e/Wicket/subscription\",\"commits_url\":\"https://api.github.com/repos/arthur-e/Wicket/commits{/sha}\",\"git_commits_url\":\"https://api.github.com/repos/arthur-e/Wicket/git/commits{/sha}\",\"comments_url\":\"https://api.github.com/repos/arthur-e/Wicket/comments{/number}\",\"issue_comment_url\":\"https://api.github.com/repos/arthur-e/Wicket/issues/comments/{number}\",\"contents_url\":\"https://api.github.com/repos/arthur-e/Wicket/contents/{+path}\",\"compare_url\":\"https://api.github.com/repos/arthur-e/Wicket/compare/{base}...{head}\",\"merges_url\":\"https://api.github.com/repos/arthur-e/Wicket/merges\",\"archive_url\":\"https://api.github.com/repos/arthur-e/Wicket/{archive_format}{/ref}\",\"downloads_url\":\"https://api.github.com/repos/arthur-e/Wicket/downloads\",\"issues_url\":\"https://api.github.com/repos/arthur-e/Wicket/issues{/number}\",\"pulls_url\":\"https://api.github.com/repos/arthur-e/Wicket/pulls{/number}\",\"milestones_url\":\"https://api.github.com/repos/arthur-e/Wicket/milestones{/number}\",\"notifications_url\":\"https://api.github.com/repos/arthur-e/Wicket/notifications{?since,all,participating}\",\"labels_url\":\"https://api.github.com/repos/arthur-e/Wicket/labels{/name}\",\"releases_url\":\"https://api.github.com/repos/arthur-e/Wicket/releases{/id}\",\"created_at\":\"2012-02-25T14:59:02Z\",\"updated_at\":\"2014-12-18T11:08:34Z\",\"pushed_at\":\"2014-11-12T13:58:08Z\",\"git_url\":\"git://github.com/arthur-e/Wicket.git\",\"ssh_url\":\"git@github.com:arthur-e/Wicket.git\",\"clone_url\":\"https://github.com/arthur-e/Wicket.git\",\"svn_url\":\"https://github.com/arthur-e/Wicket\",\"homepage\":\"http://arthur-e.github.com/Wicket\",\"size\":1032,\"stargazers_count\":99,\"watchers_count\":99,\"language\":\"JavaScript\",\"has_issues\":true,\"has_downloads\":true,\"has_wiki\":true,\"has_pages\":true,\"forks_count\":45,\"mirror_url\":null,\"open_issues_count\":11,\"forks\":45,\"open_issues\":11,\"watchers\":99,\"default_branch\":\"master\"}},\"_links\":{\"self\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60\"},\"html\":{\"href\":\"https://github.com/arthur-e/Wicket/pull/60\"},\"issue\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/issues/60\"},\"comments\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/issues/60/comments\"},\"review_comments\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60/comments\"},\"review_comment\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/comments/{number}\"},\"commits\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/pulls/60/commits\"},\"statuses\":{\"href\":\"https://api.github.com/repos/arthur-e/Wicket/statuses/b1050c9a81013c77f4dc3e266c8606e130d7b23c\"}},\"merged\":false,\"mergeable\":null,\"mergeable_state\":\"unknown\",\"merged_by\":null,\"comments\":0,\"review_comments\":0,\"commits\":1,\"additions\":99,\"deletions\":83,\"changed_files\":1}},\"public\":true,\"created_at\":\"2015-01-01T01:07:38Z\"}\n{\"id\":\"2489399010\",\"type\":\"PushEvent\",\"actor\":{\"id\":6325631,\"login\":\"pirej\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/pirej\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6325631?\"},\"repo\":{\"id\":27450767,\"name\":\"lollipoop/android_frameworks_native\",\"url\":\"https://api.github.com/repos/lollipoop/android_frameworks_native\"},\"payload\":{\"push_id\":536753610,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/m4\",\"head\":\"6ce802b32fdfecdea10b550d45f16937d585dff4\",\"before\":\"90bee99354ab9ae0c27d48ef4891f232cca0c1f4\",\"commits\":[{\"sha\":\"6ce802b32fdfecdea10b550d45f16937d585dff4\",\"author\":{\"email\":\"756a323d9c5c587ce7cec8b755badb3d54b6fcc9@gmail.com\",\"name\":\"x10forevers\"},\"message\":\"libEGL: workaround for google bug 10194508\\n\\nSome of device's older blobs don't have EGL_KHL_gl_colorspace extension\\nand need this workaround\\n\\nChange-Id: I811538e7d595f18055d13ca608098c049b4cb9b9\",\"distinct\":true,\"url\":\"https://api.github.com/repos/lollipoop/android_frameworks_native/commits/6ce802b32fdfecdea10b550d45f16937d585dff4\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:38Z\",\"org\":{\"id\":10051895,\"login\":\"lollipoop\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/lollipoop\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/10051895?\"}}\n{\"id\":\"2489399013\",\"type\":\"PushEvent\",\"actor\":{\"id\":6064868,\"login\":\"AiNoKame\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/users/AiNoKame\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/6064868?\"},\"repo\":{\"id\":27805488,\"name\":\"gatewayd/gatewayd-banking-app\",\"url\":\"https://api.github.com/repos/gatewayd/gatewayd-banking-app\"},\"payload\":{\"push_id\":536753611,\"size\":1,\"distinct_size\":1,\"ref\":\"refs/heads/feature/component-form-mixin\",\"head\":\"f92b106ccbf2b52784aa0de50c946c2cbf24ed09\",\"before\":\"9e5a5f1a0c7630df1141086530e23698930499cf\",\"commits\":[{\"sha\":\"f92b106ccbf2b52784aa0de50c946c2cbf24ed09\",\"author\":{\"email\":\"c24f01d25e2c94304e596feb894fac5a7ed7429c@gmail.com\",\"name\":\"Rod Calumpong\"},\"message\":\"[TASK] Navigate between transactions and accounts section via top bar tabs\",\"distinct\":true,\"url\":\"https://api.github.com/repos/gatewayd/gatewayd-banking-app/commits/f92b106ccbf2b52784aa0de50c946c2cbf24ed09\"}]},\"public\":true,\"created_at\":\"2015-01-01T01:07:39Z\",\"org\":{\"id\":8335076,\"login\":\"gatewayd\",\"gravatar_id\":\"\",\"url\":\"https://api.github.com/orgs/gatewayd\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/8335076?\"}}\n"
  },
  {
    "path": "benches/index-bench.rs",
    "content": "use criterion::{criterion_group, criterion_main, BatchSize, Bencher, Criterion, Throughput};\nuse tantivy::schema::{TantivyDocument, FAST, INDEXED, STORED, STRING, TEXT};\nuse tantivy::{tokenizer, Index, IndexWriter};\n\nconst HDFS_LOGS: &str = include_str!(\"hdfs.json\");\nconst GH_LOGS: &str = include_str!(\"gh.json\");\nconst WIKI: &str = include_str!(\"wiki.json\");\n\nfn benchmark(\n    b: &mut Bencher,\n    input: &str,\n    schema: tantivy::schema::Schema,\n    commit: bool,\n    parse_json: bool,\n    is_dynamic: bool,\n) {\n    if is_dynamic {\n        benchmark_dynamic_json(b, input, schema, commit, parse_json)\n    } else {\n        _benchmark(b, input, schema, commit, parse_json, |schema, doc_json| {\n            TantivyDocument::parse_json(schema, doc_json).unwrap()\n        })\n    }\n}\n\nfn get_index(schema: tantivy::schema::Schema) -> Index {\n    let mut index = Index::create_in_ram(schema.clone());\n    let ff_tokenizer_manager = tokenizer::TokenizerManager::default();\n    ff_tokenizer_manager.register(\n        \"raw\",\n        tokenizer::TextAnalyzer::builder(tokenizer::RawTokenizer::default())\n            .filter(tokenizer::RemoveLongFilter::limit(255))\n            .build(),\n    );\n    index.set_fast_field_tokenizers(ff_tokenizer_manager.clone());\n    index\n}\n\nfn _benchmark(\n    b: &mut Bencher,\n    input: &str,\n    schema: tantivy::schema::Schema,\n    commit: bool,\n    include_json_parsing: bool,\n    create_doc: impl Fn(&tantivy::schema::Schema, &str) -> TantivyDocument,\n) {\n    if include_json_parsing {\n        let lines: Vec<&str> = input.trim().split('\\n').collect();\n        b.iter(|| {\n            let index = get_index(schema.clone());\n            let mut index_writer: IndexWriter =\n                index.writer_with_num_threads(1, 100_000_000).unwrap();\n            for doc_json in &lines {\n                let doc = create_doc(&schema, doc_json);\n                index_writer.add_document(doc).unwrap();\n            }\n            if commit {\n                index_writer.commit().unwrap();\n            }\n        })\n    } else {\n        let docs: Vec<_> = input\n            .trim()\n            .split('\\n')\n            .map(|doc_json| create_doc(&schema, doc_json))\n            .collect();\n        b.iter_batched(\n            || docs.clone(),\n            |docs| {\n                let index = get_index(schema.clone());\n                let mut index_writer: IndexWriter =\n                    index.writer_with_num_threads(1, 100_000_000).unwrap();\n                for doc in docs {\n                    index_writer.add_document(doc).unwrap();\n                }\n                if commit {\n                    index_writer.commit().unwrap();\n                }\n            },\n            BatchSize::SmallInput,\n        )\n    }\n}\nfn benchmark_dynamic_json(\n    b: &mut Bencher,\n    input: &str,\n    schema: tantivy::schema::Schema,\n    commit: bool,\n    parse_json: bool,\n) {\n    let json_field = schema.get_field(\"json\").unwrap();\n    _benchmark(b, input, schema, commit, parse_json, |_schema, doc_json| {\n        let json_val: serde_json::Value = serde_json::from_str(doc_json).unwrap();\n        tantivy::doc!(json_field=>json_val)\n    })\n}\n\npub fn hdfs_index_benchmark(c: &mut Criterion) {\n    let schema = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_u64_field(\"timestamp\", INDEXED);\n        schema_builder.add_text_field(\"body\", TEXT);\n        schema_builder.add_text_field(\"severity\", STRING);\n        schema_builder.build()\n    };\n    let schema_only_fast = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_u64_field(\"timestamp\", FAST);\n        schema_builder.add_text_field(\"body\", FAST);\n        schema_builder.add_text_field(\"severity\", FAST);\n        schema_builder.build()\n    };\n    let _schema_with_store = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_u64_field(\"timestamp\", INDEXED | STORED);\n        schema_builder.add_text_field(\"body\", TEXT | STORED);\n        schema_builder.add_text_field(\"severity\", STRING | STORED);\n        schema_builder.build()\n    };\n    let dynamic_schema = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_json_field(\"json\", TEXT | FAST);\n        schema_builder.build()\n    };\n\n    let mut group = c.benchmark_group(\"index-hdfs\");\n    group.throughput(Throughput::Bytes(HDFS_LOGS.len() as u64));\n    group.sample_size(20);\n\n    let benches = [\n        (\"only-indexed-\".to_string(), schema, false),\n        //(\"stored-\".to_string(), _schema_with_store, false),\n        (\"only-fast-\".to_string(), schema_only_fast, false),\n        (\"dynamic-\".to_string(), dynamic_schema, true),\n    ];\n\n    for (prefix, schema, is_dynamic) in benches {\n        for commit in [false, true] {\n            let suffix = if commit { \"with-commit\" } else { \"no-commit\" };\n            {\n                let parse_json = false;\n                // for parse_json in [false, true] {\n                let suffix = if parse_json {\n                    format!(\"{suffix}-with-json-parsing\")\n                } else {\n                    suffix.to_string()\n                };\n\n                let bench_name = format!(\"{prefix}{suffix}\");\n                group.bench_function(bench_name, |b| {\n                    benchmark(b, HDFS_LOGS, schema.clone(), commit, parse_json, is_dynamic)\n                });\n            }\n        }\n    }\n}\n\npub fn gh_index_benchmark(c: &mut Criterion) {\n    let dynamic_schema = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_json_field(\"json\", TEXT | FAST);\n        schema_builder.build()\n    };\n    let dynamic_schema_fast = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_json_field(\"json\", FAST);\n        schema_builder.build()\n    };\n\n    let mut group = c.benchmark_group(\"index-gh\");\n    group.throughput(Throughput::Bytes(GH_LOGS.len() as u64));\n\n    group.bench_function(\"index-gh-no-commit\", |b| {\n        benchmark_dynamic_json(b, GH_LOGS, dynamic_schema.clone(), false, false)\n    });\n    group.bench_function(\"index-gh-fast\", |b| {\n        benchmark_dynamic_json(b, GH_LOGS, dynamic_schema_fast.clone(), false, false)\n    });\n\n    group.bench_function(\"index-gh-fast-with-commit\", |b| {\n        benchmark_dynamic_json(b, GH_LOGS, dynamic_schema_fast.clone(), true, false)\n    });\n}\n\npub fn wiki_index_benchmark(c: &mut Criterion) {\n    let dynamic_schema = {\n        let mut schema_builder = tantivy::schema::SchemaBuilder::new();\n        schema_builder.add_json_field(\"json\", TEXT | FAST);\n        schema_builder.build()\n    };\n\n    let mut group = c.benchmark_group(\"index-wiki\");\n    group.throughput(Throughput::Bytes(WIKI.len() as u64));\n\n    group.bench_function(\"index-wiki-no-commit\", |b| {\n        benchmark_dynamic_json(b, WIKI, dynamic_schema.clone(), false, false)\n    });\n    group.bench_function(\"index-wiki-with-commit\", |b| {\n        benchmark_dynamic_json(b, WIKI, dynamic_schema.clone(), true, false)\n    });\n}\n\ncriterion_group! {\n    name = benches;\n    config = Criterion::default();\n    targets = hdfs_index_benchmark\n}\ncriterion_group! {\n    name = gh_benches;\n    config = Criterion::default();\n    targets = gh_index_benchmark\n}\ncriterion_group! {\n    name = wiki_benches;\n    config = Criterion::default();\n    targets = wiki_index_benchmark\n}\ncriterion_main!(benches, gh_benches, wiki_benches);\n"
  },
  {
    "path": "benches/merge_segments.rs",
    "content": "// Benchmarks segment merging\n//\n// Notes:\n// - Input segments are kept intact (no deletes / no IndexWriter merge).\n// - Output is written to a `NullDirectory` that discards all files except\n//  fieldnorms (needed for merging).\n\nuse std::collections::HashMap;\nuse std::io::{self, Write};\nuse std::path::{Path, PathBuf};\nuse std::sync::{Arc, RwLock};\n\nuse binggan::{black_box, BenchRunner};\nuse rand::prelude::*;\nuse rand::rngs::StdRng;\nuse rand::SeedableRng;\nuse tantivy::directory::error::{DeleteError, OpenReadError, OpenWriteError};\nuse tantivy::directory::{\n    AntiCallToken, Directory, FileHandle, OwnedBytes, TerminatingWrite, WatchCallback, WatchHandle,\n    WritePtr,\n};\nuse tantivy::indexer::{merge_filtered_segments, NoMergePolicy};\nuse tantivy::schema::{Schema, TEXT};\nuse tantivy::{doc, HasLen, Index, IndexSettings, Segment};\n\n#[derive(Clone, Default, Debug)]\nstruct NullDirectory {\n    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,\n}\n\nstruct NullWriter;\n\nimpl Write for NullWriter {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        Ok(buf.len())\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        Ok(())\n    }\n}\n\nimpl TerminatingWrite for NullWriter {\n    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {\n        Ok(())\n    }\n}\n\nstruct InMemoryWriter {\n    path: PathBuf,\n    buffer: Vec<u8>,\n    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,\n}\n\nimpl Write for InMemoryWriter {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        self.buffer.extend_from_slice(buf);\n        Ok(buf.len())\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        Ok(())\n    }\n}\n\nimpl TerminatingWrite for InMemoryWriter {\n    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {\n        let bytes = OwnedBytes::new(std::mem::take(&mut self.buffer));\n        self.blobs.write().unwrap().insert(self.path.clone(), bytes);\n        Ok(())\n    }\n}\n\n#[derive(Debug, Default)]\nstruct NullFileHandle;\nimpl HasLen for NullFileHandle {\n    fn len(&self) -> usize {\n        0\n    }\n}\nimpl FileHandle for NullFileHandle {\n    fn read_bytes(&self, _range: std::ops::Range<usize>) -> io::Result<OwnedBytes> {\n        unimplemented!()\n    }\n}\n\nimpl Directory for NullDirectory {\n    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {\n        if let Some(bytes) = self.blobs.read().unwrap().get(path) {\n            return Ok(Arc::new(bytes.clone()));\n        }\n        Ok(Arc::new(NullFileHandle))\n    }\n\n    fn delete(&self, _path: &Path) -> Result<(), DeleteError> {\n        Ok(())\n    }\n\n    fn exists(&self, _path: &Path) -> Result<bool, OpenReadError> {\n        Ok(true)\n    }\n\n    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError> {\n        let path_buf = path.to_path_buf();\n        if path.to_string_lossy().ends_with(\".fieldnorm\") {\n            let writer = InMemoryWriter {\n                path: path_buf,\n                buffer: Vec::new(),\n                blobs: Arc::clone(&self.blobs),\n            };\n            Ok(io::BufWriter::new(Box::new(writer)))\n        } else {\n            Ok(io::BufWriter::new(Box::new(NullWriter)))\n        }\n    }\n\n    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError> {\n        if let Some(bytes) = self.blobs.read().unwrap().get(path) {\n            return Ok(bytes.as_slice().to_vec());\n        }\n        Err(OpenReadError::FileDoesNotExist(path.to_path_buf()))\n    }\n\n    fn atomic_write(&self, _path: &Path, _data: &[u8]) -> io::Result<()> {\n        Ok(())\n    }\n\n    fn sync_directory(&self) -> io::Result<()> {\n        Ok(())\n    }\n\n    fn watch(&self, _watch_callback: WatchCallback) -> tantivy::Result<WatchHandle> {\n        Ok(WatchHandle::empty())\n    }\n}\n\nstruct MergeScenario {\n    #[allow(dead_code)]\n    index: Index,\n    segments: Vec<Segment>,\n    settings: IndexSettings,\n    label: String,\n}\n\nfn build_index(\n    num_segments: usize,\n    docs_per_segment: usize,\n    tokens_per_doc: usize,\n    vocab_size: usize,\n) -> MergeScenario {\n    let mut schema_builder = Schema::builder();\n    let body = schema_builder.add_text_field(\"body\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n\n    assert!(vocab_size > 0);\n    let total_tokens = num_segments * docs_per_segment * tokens_per_doc;\n    let use_unique_terms = vocab_size >= total_tokens;\n    let mut rng = StdRng::from_seed([7u8; 32]);\n    let mut next_token_id: u64 = 0;\n\n    {\n        let mut writer = index.writer_with_num_threads(1, 256_000_000).unwrap();\n        writer.set_merge_policy(Box::new(NoMergePolicy));\n        for _ in 0..num_segments {\n            for _ in 0..docs_per_segment {\n                let mut tokens = Vec::with_capacity(tokens_per_doc);\n                for _ in 0..tokens_per_doc {\n                    let token_id = if use_unique_terms {\n                        let id = next_token_id;\n                        next_token_id += 1;\n                        id\n                    } else {\n                        rng.random_range(0..vocab_size as u64)\n                    };\n                    tokens.push(format!(\"term_{token_id}\"));\n                }\n                writer.add_document(doc!(body => tokens.join(\" \"))).unwrap();\n            }\n            writer.commit().unwrap();\n        }\n    }\n\n    let segments = index.searchable_segments().unwrap();\n    let settings = index.settings().clone();\n    let label = format!(\n        \"segments={}, docs/seg={}, tokens/doc={}, vocab={}\",\n        num_segments, docs_per_segment, tokens_per_doc, vocab_size\n    );\n\n    MergeScenario {\n        index,\n        segments,\n        settings,\n        label,\n    }\n}\n\nfn main() {\n    let scenarios = vec![\n        build_index(8, 50_000, 12, 8),\n        build_index(16, 50_000, 12, 8),\n        build_index(16, 100_000, 12, 8),\n        build_index(8, 50_000, 8, 8 * 50_000 * 8),\n    ];\n\n    let mut runner = BenchRunner::new();\n    for scenario in scenarios {\n        let mut group = runner.new_group();\n        group.set_name(format!(\"merge_segments inv_index — {}\", scenario.label));\n        let segments = scenario.segments.clone();\n        let settings = scenario.settings.clone();\n        group.register(\"merge\", move |_| {\n            let output_dir = NullDirectory::default();\n            let filter_doc_ids = vec![None; segments.len()];\n            let merged_index =\n                merge_filtered_segments(&segments, settings.clone(), filter_doc_ids, output_dir)\n                    .unwrap();\n            black_box(merged_index);\n        });\n\n        group.run();\n    }\n}\n"
  },
  {
    "path": "benches/range_queries.rs",
    "content": "use std::ops::Bound;\n\nuse binggan::{black_box, BenchGroup, BenchRunner};\nuse rand::prelude::*;\nuse rand::rngs::StdRng;\nuse rand::SeedableRng;\nuse tantivy::collector::{Count, DocSetCollector, TopDocs};\nuse tantivy::query::RangeQuery;\nuse tantivy::schema::{Schema, FAST, INDEXED};\nuse tantivy::{doc, Index, Order, ReloadPolicy, Searcher, Term};\n\n#[derive(Clone)]\nstruct BenchIndex {\n    #[allow(dead_code)]\n    index: Index,\n    searcher: Searcher,\n}\n\nfn build_shared_indices(num_docs: usize, distribution: &str) -> BenchIndex {\n    // Schema with fast fields only\n    let mut schema_builder = Schema::builder();\n    let f_num_rand_fast = schema_builder.add_u64_field(\"num_rand_fast\", INDEXED | FAST);\n    let f_num_asc_fast = schema_builder.add_u64_field(\"num_asc_fast\", INDEXED | FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n\n    // Populate index with stable RNG for reproducibility.\n    let mut rng = StdRng::from_seed([7u8; 32]);\n\n    {\n        let mut writer = index.writer_with_num_threads(1, 4_000_000_000).unwrap();\n\n        match distribution {\n            \"dense\" => {\n                for doc_id in 0..num_docs {\n                    let num_rand = rng.random_range(0u64..1000u64);\n                    let num_asc = (doc_id / 10000) as u64;\n\n                    writer\n                        .add_document(doc!(\n                            f_num_rand_fast=>num_rand,\n                            f_num_asc_fast=>num_asc,\n                        ))\n                        .unwrap();\n                }\n            }\n            \"sparse\" => {\n                for doc_id in 0..num_docs {\n                    let num_rand = rng.random_range(0u64..10000000u64);\n                    let num_asc = doc_id as u64;\n\n                    writer\n                        .add_document(doc!(\n                            f_num_rand_fast=>num_rand,\n                            f_num_asc_fast=>num_asc,\n                        ))\n                        .unwrap();\n                }\n            }\n            _ => {\n                panic!(\"Unsupported distribution type\");\n            }\n        }\n        writer.commit().unwrap();\n    }\n\n    // Prepare reader/searcher once.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::Manual)\n        .try_into()\n        .unwrap();\n    let searcher = reader.searcher();\n\n    BenchIndex { index, searcher }\n}\n\nfn main() {\n    // Prepare corpora with varying scenarios\n    let scenarios = vec![\n        // Dense distribution - random values in small range (0-999)\n        (\n            \"dense_values_search_low_value_range\".to_string(),\n            10_000_000,\n            \"dense\",\n            0,\n            9,\n        ),\n        (\n            \"dense_values_search_high_value_range\".to_string(),\n            10_000_000,\n            \"dense\",\n            990,\n            999,\n        ),\n        (\n            \"dense_values_search_out_of_range\".to_string(),\n            10_000_000,\n            \"dense\",\n            1000,\n            1002,\n        ),\n        (\n            \"sparse_values_search_low_value_range\".to_string(),\n            10_000_000,\n            \"sparse\",\n            0,\n            9,\n        ),\n        (\n            \"sparse_values_search_high_value_range\".to_string(),\n            10_000_000,\n            \"sparse\",\n            9_999_990,\n            9_999_999,\n        ),\n        (\n            \"sparse_values_search_out_of_range\".to_string(),\n            10_000_000,\n            \"sparse\",\n            10_000_000,\n            10_000_002,\n        ),\n    ];\n\n    let mut runner = BenchRunner::new();\n    for (scenario_id, n, num_rand_distribution, range_low, range_high) in scenarios {\n        // Build index for this scenario\n        let bench_index = build_shared_indices(n, num_rand_distribution);\n\n        // Create benchmark group\n        let mut group = runner.new_group();\n\n        // Now set the name (this moves scenario_id)\n        group.set_name(scenario_id);\n\n        // Define fast field types\n        let field_names = [\"num_rand_fast\", \"num_asc_fast\"];\n\n        // Generate range queries for fast fields\n        for &field_name in &field_names {\n            // Create the range query\n            let field = bench_index.searcher.schema().get_field(field_name).unwrap();\n            let lower_term = Term::from_field_u64(field, range_low);\n            let upper_term = Term::from_field_u64(field, range_high);\n\n            let query = RangeQuery::new(Bound::Included(lower_term), Bound::Included(upper_term));\n\n            run_benchmark_tasks(\n                &mut group,\n                &bench_index,\n                query,\n                field_name,\n                range_low,\n                range_high,\n            );\n        }\n\n        group.run();\n    }\n}\n\n/// Run all benchmark tasks for a given range query and field name\nfn run_benchmark_tasks(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    field_name: &str,\n    range_low: u64,\n    range_high: u64,\n) {\n    // Test count\n    add_bench_task_count(\n        bench_group,\n        bench_index,\n        query.clone(),\n        \"count\",\n        field_name,\n        range_low,\n        range_high,\n    );\n\n    // Test top 100 by the field (ascending order)\n    {\n        let collector_name = format!(\"top100_by_{}_asc\", field_name);\n        let field_name_owned = field_name.to_string();\n        add_bench_task_top100_asc(\n            bench_group,\n            bench_index,\n            query.clone(),\n            &collector_name,\n            field_name,\n            range_low,\n            range_high,\n            field_name_owned,\n        );\n    }\n\n    // Test top 100 by the field (descending order)\n    {\n        let collector_name = format!(\"top100_by_{}_desc\", field_name);\n        let field_name_owned = field_name.to_string();\n        add_bench_task_top100_desc(\n            bench_group,\n            bench_index,\n            query,\n            &collector_name,\n            field_name,\n            range_low,\n            range_high,\n            field_name_owned,\n        );\n    }\n}\n\nfn add_bench_task_count(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    collector_name: &str,\n    field_name: &str,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\n        \"range_{}_[{} TO {}]_{}\",\n        field_name, range_low, range_high, collector_name\n    );\n\n    let search_task = CountSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nfn add_bench_task_docset(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    collector_name: &str,\n    field_name: &str,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\n        \"range_{}_[{} TO {}]_{}\",\n        field_name, range_low, range_high, collector_name\n    );\n\n    let search_task = DocSetSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nfn add_bench_task_top100_asc(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    collector_name: &str,\n    field_name: &str,\n    range_low: u64,\n    range_high: u64,\n    field_name_owned: String,\n) {\n    let task_name = format!(\n        \"range_{}_[{} TO {}]_{}\",\n        field_name, range_low, range_high, collector_name\n    );\n\n    let search_task = Top100AscSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n        field_name: field_name_owned,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nfn add_bench_task_top100_desc(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    collector_name: &str,\n    field_name: &str,\n    range_low: u64,\n    range_high: u64,\n    field_name_owned: String,\n) {\n    let task_name = format!(\n        \"range_{}_[{} TO {}]_{}\",\n        field_name, range_low, range_high, collector_name\n    );\n\n    let search_task = Top100DescSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n        field_name: field_name_owned,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nstruct CountSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl CountSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        self.searcher.search(&self.query, &Count).unwrap()\n    }\n}\n\nstruct DocSetSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl DocSetSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        let result = self.searcher.search(&self.query, &DocSetCollector).unwrap();\n        result.len()\n    }\n}\n\nstruct Top100AscSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n    field_name: String,\n}\n\nimpl Top100AscSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        let collector =\n            TopDocs::with_limit(100).order_by_fast_field::<u64>(&self.field_name, Order::Asc);\n        let result = self.searcher.search(&self.query, &collector).unwrap();\n        for (_score, doc_address) in &result {\n            let _doc: tantivy::TantivyDocument = self.searcher.doc(*doc_address).unwrap();\n        }\n        result.len()\n    }\n}\n\nstruct Top100DescSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n    field_name: String,\n}\n\nimpl Top100DescSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        let collector =\n            TopDocs::with_limit(100).order_by_fast_field::<u64>(&self.field_name, Order::Desc);\n        let result = self.searcher.search(&self.query, &collector).unwrap();\n        for (_score, doc_address) in &result {\n            let _doc: tantivy::TantivyDocument = self.searcher.doc(*doc_address).unwrap();\n        }\n        result.len()\n    }\n}\n"
  },
  {
    "path": "benches/range_query.rs",
    "content": "use std::fmt::Display;\nuse std::net::Ipv6Addr;\nuse std::ops::RangeInclusive;\n\nuse binggan::plugins::PeakMemAllocPlugin;\nuse binggan::{black_box, BenchRunner, OutputValue, PeakMemAlloc, INSTRUMENTED_SYSTEM};\nuse columnar::MonotonicallyMappableToU128;\nuse rand::rngs::StdRng;\nuse rand::{Rng, SeedableRng};\nuse tantivy::collector::{Count, TopDocs};\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::{doc, Index};\n\n#[global_allocator]\npub static GLOBAL: &PeakMemAlloc<std::alloc::System> = &INSTRUMENTED_SYSTEM;\n\nfn main() {\n    bench_range_query();\n}\n\nfn bench_range_query() {\n    let index = get_index_0_to_100();\n    let mut runner = BenchRunner::new();\n    runner.add_plugin(PeakMemAllocPlugin::new(GLOBAL));\n\n    runner.set_name(\"range_query on u64\");\n    let field_name_and_descr: Vec<_> = vec![\n        (\"id\", \"Single Valued Range Field\"),\n        (\"ids\", \"Multi Valued Range Field\"),\n    ];\n    let range_num_hits = vec![\n        (\"90_percent\", get_90_percent()),\n        (\"10_percent\", get_10_percent()),\n        (\"1_percent\", get_1_percent()),\n    ];\n\n    test_range(&mut runner, &index, &field_name_and_descr, range_num_hits);\n\n    runner.set_name(\"range_query on ip\");\n    let field_name_and_descr: Vec<_> = vec![\n        (\"ip\", \"Single Valued Range Field\"),\n        (\"ips\", \"Multi Valued Range Field\"),\n    ];\n    let range_num_hits = vec![\n        (\"90_percent\", get_90_percent_ip()),\n        (\"10_percent\", get_10_percent_ip()),\n        (\"1_percent\", get_1_percent_ip()),\n    ];\n\n    test_range(&mut runner, &index, &field_name_and_descr, range_num_hits);\n}\n\nfn test_range<T: Display>(\n    runner: &mut BenchRunner,\n    index: &Index,\n    field_name_and_descr: &[(&str, &str)],\n    range_num_hits: Vec<(&str, RangeInclusive<T>)>,\n) {\n    for (field, suffix) in field_name_and_descr {\n        let term_num_hits = vec![\n            (\"\", \"\"),\n            (\"1_percent\", \"veryfew\"),\n            (\"10_percent\", \"few\"),\n            (\"90_percent\", \"most\"),\n        ];\n        let mut group = runner.new_group();\n        group.set_name(suffix);\n        // all intersect combinations\n        for (range_name, range) in &range_num_hits {\n            for (term_name, term) in &term_num_hits {\n                let index = &index;\n                let test_name = if term_name.is_empty() {\n                    format!(\"id_range_hit_{}\", range_name)\n                } else {\n                    format!(\n                        \"id_range_hit_{}_intersect_with_term_{}\",\n                        range_name, term_name\n                    )\n                };\n                group.register(test_name, move |_| {\n                    let query = if term_name.is_empty() {\n                        \"\".to_string()\n                    } else {\n                        format!(\"AND id_name:{}\", term)\n                    };\n                    black_box(execute_query(field, range, &query, index));\n                });\n            }\n        }\n        group.run();\n    }\n}\n\nfn get_index_0_to_100() -> Index {\n    let mut rng = StdRng::from_seed([1u8; 32]);\n    let num_vals = 100_000;\n    let docs: Vec<_> = (0..num_vals)\n        .map(|_i| {\n            let id_name = if rng.random_bool(0.01) {\n                \"veryfew\".to_string() // 1%\n            } else if rng.random_bool(0.1) {\n                \"few\".to_string() // 9%\n            } else {\n                \"most\".to_string() // 90%\n            };\n            Doc {\n                id_name,\n                id: rng.random_range(0..100),\n                // Multiply by 1000, so that we create most buckets in the compact space\n                // The benches depend on this range to select n-percent of elements with the\n                // methods below.\n                ip: Ipv6Addr::from_u128(rng.random_range(0..100) * 1000),\n            }\n        })\n        .collect();\n\n    create_index_from_docs(&docs)\n}\n\n#[derive(Clone, Debug)]\npub struct Doc {\n    pub id_name: String,\n    pub id: u64,\n    pub ip: Ipv6Addr,\n}\n\npub fn create_index_from_docs(docs: &[Doc]) -> Index {\n    let mut schema_builder = Schema::builder();\n    let id_u64_field = schema_builder.add_u64_field(\"id\", INDEXED | STORED | FAST);\n    let ids_u64_field =\n        schema_builder.add_u64_field(\"ids\", NumericOptions::default().set_fast().set_indexed());\n\n    let id_f64_field = schema_builder.add_f64_field(\"id_f64\", INDEXED | STORED | FAST);\n    let ids_f64_field = schema_builder.add_f64_field(\n        \"ids_f64\",\n        NumericOptions::default().set_fast().set_indexed(),\n    );\n\n    let id_i64_field = schema_builder.add_i64_field(\"id_i64\", INDEXED | STORED | FAST);\n    let ids_i64_field = schema_builder.add_i64_field(\n        \"ids_i64\",\n        NumericOptions::default().set_fast().set_indexed(),\n    );\n\n    let text_field = schema_builder.add_text_field(\"id_name\", STRING | STORED);\n    let text_field2 = schema_builder.add_text_field(\"id_name_fast\", STRING | STORED | FAST);\n\n    let ip_field = schema_builder.add_ip_addr_field(\"ip\", FAST);\n    let ips_field = schema_builder.add_ip_addr_field(\"ips\", FAST);\n\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema);\n\n    {\n        let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n        for doc in docs.iter() {\n            index_writer\n                .add_document(doc!(\n                    ids_i64_field => doc.id as i64,\n                    ids_i64_field => doc.id as i64,\n                    ids_f64_field => doc.id as f64,\n                    ids_f64_field => doc.id as f64,\n                    ids_u64_field => doc.id,\n                    ids_u64_field => doc.id,\n                    id_u64_field => doc.id,\n                    id_f64_field => doc.id as f64,\n                    id_i64_field => doc.id as i64,\n                    text_field => doc.id_name.to_string(),\n                    text_field2 => doc.id_name.to_string(),\n                    ips_field => doc.ip,\n                    ips_field => doc.ip,\n                    ip_field => doc.ip,\n                ))\n                .unwrap();\n        }\n\n        index_writer.commit().unwrap();\n    }\n    index\n}\n\nfn get_90_percent() -> RangeInclusive<u64> {\n    0..=90\n}\n\nfn get_10_percent() -> RangeInclusive<u64> {\n    0..=10\n}\n\nfn get_1_percent() -> RangeInclusive<u64> {\n    10..=10\n}\n\nfn get_90_percent_ip() -> RangeInclusive<Ipv6Addr> {\n    let start = Ipv6Addr::from_u128(0);\n    let end = Ipv6Addr::from_u128(90 * 1000);\n    start..=end\n}\n\nfn get_10_percent_ip() -> RangeInclusive<Ipv6Addr> {\n    let start = Ipv6Addr::from_u128(0);\n    let end = Ipv6Addr::from_u128(10 * 1000);\n    start..=end\n}\n\nfn get_1_percent_ip() -> RangeInclusive<Ipv6Addr> {\n    let start = Ipv6Addr::from_u128(10 * 1000);\n    let end = Ipv6Addr::from_u128(10 * 1000);\n    start..=end\n}\n\nstruct NumHits {\n    count: usize,\n}\nimpl OutputValue for NumHits {\n    fn column_title() -> &'static str {\n        \"NumHits\"\n    }\n    fn format(&self) -> Option<String> {\n        Some(self.count.to_string())\n    }\n}\n\nfn execute_query<T: Display>(\n    field: &str,\n    id_range: &RangeInclusive<T>,\n    suffix: &str,\n    index: &Index,\n) -> NumHits {\n    let gen_query_inclusive = |from: &T, to: &T| {\n        format!(\n            \"{}:[{} TO {}] {}\",\n            field,\n            &from.to_string(),\n            &to.to_string(),\n            suffix\n        )\n    };\n\n    let query = gen_query_inclusive(id_range.start(), id_range.end());\n    execute_query_(&query, index)\n}\n\nfn execute_query_(query: &str, index: &Index) -> NumHits {\n    let query_from_text = |text: &str| {\n        QueryParser::for_index(index, vec![])\n            .parse_query(text)\n            .unwrap()\n    };\n    let query = query_from_text(query);\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n    let num_hits = searcher\n        .search(&query, &(TopDocs::with_limit(10).order_by_score(), Count))\n        .unwrap()\n        .1;\n    NumHits { count: num_hits }\n}\n"
  },
  {
    "path": "benches/regex_all_terms.rs",
    "content": "// Benchmarks regex query that matches all terms in a synthetic index.\n//\n// Corpus model:\n// - N unique terms: t000000, t000001, ...\n// - M docs\n// - K tokens per doc: doc i gets terms derived from (i, token_index)\n//\n// Query:\n// - Regex \"t.*\" to match all terms\n//\n// Run with:\n// - cargo bench --bench regex_all_terms\n//\n\nuse std::fmt::Write;\n\nuse binggan::{black_box, BenchRunner};\nuse tantivy::collector::Count;\nuse tantivy::query::RegexQuery;\nuse tantivy::schema::{Schema, TEXT};\nuse tantivy::{doc, Index, ReloadPolicy};\n\nconst HEAP_SIZE_BYTES: usize = 200_000_000;\n\n#[derive(Clone, Copy)]\nstruct BenchConfig {\n    num_terms: usize,\n    num_docs: usize,\n    tokens_per_doc: usize,\n}\n\nfn main() {\n    let configs = default_configs();\n\n    let mut runner = BenchRunner::new();\n    for config in configs {\n        let (index, text_field) = build_index(config, HEAP_SIZE_BYTES);\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()\n            .expect(\"reader\");\n        let searcher = reader.searcher();\n        let query = RegexQuery::from_pattern(\"t.*\", text_field).expect(\"regex query\");\n\n        let mut group = runner.new_group();\n        group.set_name(format!(\n            \"regex_all_terms_t{}_d{}_k{}\",\n            config.num_terms, config.num_docs, config.tokens_per_doc\n        ));\n        group.register(\"regex_count\", move |_| {\n            let count = searcher.search(&query, &Count).expect(\"search\");\n            black_box(count);\n        });\n        group.run();\n    }\n}\n\nfn default_configs() -> Vec<BenchConfig> {\n    vec![\n        BenchConfig {\n            num_terms: 10_000,\n            num_docs: 100_000,\n            tokens_per_doc: 1,\n        },\n        BenchConfig {\n            num_terms: 10_000,\n            num_docs: 100_000,\n            tokens_per_doc: 8,\n        },\n        BenchConfig {\n            num_terms: 100_000,\n            num_docs: 100_000,\n            tokens_per_doc: 1,\n        },\n        BenchConfig {\n            num_terms: 100_000,\n            num_docs: 100_000,\n            tokens_per_doc: 8,\n        },\n    ]\n}\n\nfn build_index(config: BenchConfig, heap_size_bytes: usize) -> (Index, tantivy::schema::Field) {\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n\n    let term_width = config.num_terms.to_string().len();\n    {\n        let mut writer = index\n            .writer_with_num_threads(1, heap_size_bytes)\n            .expect(\"writer\");\n        let mut buffer = String::new();\n        for doc_id in 0..config.num_docs {\n            buffer.clear();\n            for token_idx in 0..config.tokens_per_doc {\n                if token_idx > 0 {\n                    buffer.push(' ');\n                }\n                let term_id = (doc_id * config.tokens_per_doc + token_idx) % config.num_terms;\n                write!(&mut buffer, \"t{term_id:0term_width$}\").expect(\"write token\");\n            }\n            writer\n                .add_document(doc!(text_field => buffer.as_str()))\n                .expect(\"add_document\");\n        }\n        writer.commit().expect(\"commit\");\n    }\n\n    (index, text_field)\n}\n"
  },
  {
    "path": "benches/str_search_and_get.rs",
    "content": "// This benchmark compares different approaches for retrieving string values:\n//\n// 1. Fast Field Approach: retrieves string values via term_ords() and ord_to_str()\n//\n// 2. Doc Store Approach: retrieves string values via searcher.doc() and field extraction\n//\n// The benchmark includes various data distributions:\n// - Dense Sequential: Sequential document IDs with dense data\n// - Dense Random: Random document IDs with dense data\n// - Sparse Sequential: Sequential document IDs with sparse data\n// - Sparse Random: Random document IDs with sparse data\nuse std::ops::Bound;\n\nuse binggan::{black_box, BenchGroup, BenchRunner};\nuse rand::prelude::*;\nuse rand::rngs::StdRng;\nuse rand::SeedableRng;\nuse tantivy::collector::{Count, DocSetCollector};\nuse tantivy::query::RangeQuery;\nuse tantivy::schema::document::TantivyDocument;\nuse tantivy::schema::{Schema, Value, FAST, STORED, STRING};\nuse tantivy::{doc, Index, ReloadPolicy, Searcher, Term};\n\n#[derive(Clone)]\nstruct BenchIndex {\n    #[allow(dead_code)]\n    index: Index,\n    searcher: Searcher,\n}\n\nfn build_shared_indices(num_docs: usize, distribution: &str) -> BenchIndex {\n    // Schema with string fast field and stored field for doc access\n    let mut schema_builder = Schema::builder();\n    let f_str_fast = schema_builder.add_text_field(\"str_fast\", STRING | STORED | FAST);\n    let f_str_stored = schema_builder.add_text_field(\"str_stored\", STRING | STORED);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n\n    // Populate index with stable RNG for reproducibility.\n    let mut rng = StdRng::from_seed([7u8; 32]);\n\n    {\n        let mut writer = index.writer_with_num_threads(1, 4_000_000_000).unwrap();\n\n        match distribution {\n            \"dense_random\" => {\n                for _doc_id in 0..num_docs {\n                    let suffix = rng.gen_range(0u64..1000u64);\n                    let str_val = format!(\"str_{:03}\", suffix);\n\n                    writer\n                        .add_document(doc!(\n                            f_str_fast=>str_val.clone(),\n                            f_str_stored=>str_val,\n                        ))\n                        .unwrap();\n                }\n            }\n            \"dense_sequential\" => {\n                for doc_id in 0..num_docs {\n                    let suffix = doc_id as u64 % 1000;\n                    let str_val = format!(\"str_{:03}\", suffix);\n\n                    writer\n                        .add_document(doc!(\n                            f_str_fast=>str_val.clone(),\n                            f_str_stored=>str_val,\n                        ))\n                        .unwrap();\n                }\n            }\n            \"sparse_random\" => {\n                for _doc_id in 0..num_docs {\n                    let suffix = rng.gen_range(0u64..1000000u64);\n                    let str_val = format!(\"str_{:07}\", suffix);\n\n                    writer\n                        .add_document(doc!(\n                            f_str_fast=>str_val.clone(),\n                            f_str_stored=>str_val,\n                        ))\n                        .unwrap();\n                }\n            }\n            \"sparse_sequential\" => {\n                for doc_id in 0..num_docs {\n                    let suffix = doc_id as u64;\n                    let str_val = format!(\"str_{:07}\", suffix);\n\n                    writer\n                        .add_document(doc!(\n                            f_str_fast=>str_val.clone(),\n                            f_str_stored=>str_val,\n                        ))\n                        .unwrap();\n                }\n            }\n            _ => {\n                panic!(\"Unsupported distribution type\");\n            }\n        }\n        writer.commit().unwrap();\n    }\n\n    // Prepare reader/searcher once.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::Manual)\n        .try_into()\n        .unwrap();\n    let searcher = reader.searcher();\n\n    BenchIndex { index, searcher }\n}\n\nfn main() {\n    // Prepare corpora with varying scenarios\n    let scenarios = vec![\n        (\n            \"dense_random_search_low_range\".to_string(),\n            1_000_000,\n            \"dense_random\",\n            0,\n            9,\n        ),\n        (\n            \"dense_random_search_high_range\".to_string(),\n            1_000_000,\n            \"dense_random\",\n            990,\n            999,\n        ),\n        (\n            \"dense_sequential_search_low_range\".to_string(),\n            1_000_000,\n            \"dense_sequential\",\n            0,\n            9,\n        ),\n        (\n            \"dense_sequential_search_high_range\".to_string(),\n            1_000_000,\n            \"dense_sequential\",\n            990,\n            999,\n        ),\n        (\n            \"sparse_random_search_low_range\".to_string(),\n            1_000_000,\n            \"sparse_random\",\n            0,\n            9999,\n        ),\n        (\n            \"sparse_random_search_high_range\".to_string(),\n            1_000_000,\n            \"sparse_random\",\n            990_000,\n            999_999,\n        ),\n        (\n            \"sparse_sequential_search_low_range\".to_string(),\n            1_000_000,\n            \"sparse_sequential\",\n            0,\n            9999,\n        ),\n        (\n            \"sparse_sequential_search_high_range\".to_string(),\n            1_000_000,\n            \"sparse_sequential\",\n            990_000,\n            999_999,\n        ),\n    ];\n\n    let mut runner = BenchRunner::new();\n    for (scenario_id, n, distribution, range_low, range_high) in scenarios {\n        let bench_index = build_shared_indices(n, distribution);\n        let mut group = runner.new_group();\n        group.set_name(scenario_id);\n\n        let field = bench_index.searcher.schema().get_field(\"str_fast\").unwrap();\n\n        let (lower_str, upper_str) =\n            if distribution == \"dense_sequential\" || distribution == \"dense_random\" {\n                (\n                    format!(\"str_{:03}\", range_low),\n                    format!(\"str_{:03}\", range_high),\n                )\n            } else {\n                (\n                    format!(\"str_{:07}\", range_low),\n                    format!(\"str_{:07}\", range_high),\n                )\n            };\n\n        let lower_term = Term::from_field_text(field, &lower_str);\n        let upper_term = Term::from_field_text(field, &upper_str);\n\n        let query = RangeQuery::new(Bound::Included(lower_term), Bound::Included(upper_term));\n\n        run_benchmark_tasks(&mut group, &bench_index, query, range_low, range_high);\n\n        group.run();\n    }\n}\n\n/// Run all benchmark tasks for a given range query\nfn run_benchmark_tasks(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    range_low: u64,\n    range_high: u64,\n) {\n    // Test count of matching documents\n    add_bench_task_count(\n        bench_group,\n        bench_index,\n        query.clone(),\n        range_low,\n        range_high,\n    );\n\n    // Test fetching all DocIds of matching documents\n    add_bench_task_docset(\n        bench_group,\n        bench_index,\n        query.clone(),\n        range_low,\n        range_high,\n    );\n\n    // Test fetching all string fast field values of matching documents\n    add_bench_task_fetch_all_strings(\n        bench_group,\n        bench_index,\n        query.clone(),\n        range_low,\n        range_high,\n    );\n\n    // Test fetching all string values of matching documents through doc() method\n    add_bench_task_fetch_all_strings_from_doc(\n        bench_group,\n        bench_index,\n        query,\n        range_low,\n        range_high,\n    );\n}\n\nfn add_bench_task_count(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\"string_search_count_[{}-{}]\", range_low, range_high);\n\n    let search_task = CountSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nfn add_bench_task_docset(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\"string_fetch_all_docset_[{}-{}]\", range_low, range_high);\n\n    let search_task = DocSetSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n    bench_group.register(task_name, move |_| black_box(search_task.run()));\n}\n\nfn add_bench_task_fetch_all_strings(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\n        \"string_fastfield_fetch_all_strings_[{}-{}]\",\n        range_low, range_high\n    );\n\n    let search_task = FetchAllStringsSearchTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n\n    bench_group.register(task_name, move |_| {\n        let result = black_box(search_task.run());\n        result.len()\n    });\n}\n\nfn add_bench_task_fetch_all_strings_from_doc(\n    bench_group: &mut BenchGroup,\n    bench_index: &BenchIndex,\n    query: RangeQuery,\n    range_low: u64,\n    range_high: u64,\n) {\n    let task_name = format!(\n        \"string_doc_fetch_all_strings_[{}-{}]\",\n        range_low, range_high\n    );\n\n    let search_task = FetchAllStringsFromDocTask {\n        searcher: bench_index.searcher.clone(),\n        query,\n    };\n\n    bench_group.register(task_name, move |_| {\n        let result = black_box(search_task.run());\n        result.len()\n    });\n}\n\nstruct CountSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl CountSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        self.searcher.search(&self.query, &Count).unwrap()\n    }\n}\n\nstruct DocSetSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl DocSetSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> usize {\n        let result = self.searcher.search(&self.query, &DocSetCollector).unwrap();\n        result.len()\n    }\n}\n\nstruct FetchAllStringsSearchTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl FetchAllStringsSearchTask {\n    #[inline(never)]\n    pub fn run(&self) -> Vec<String> {\n        let doc_addresses = self.searcher.search(&self.query, &DocSetCollector).unwrap();\n        let mut docs = doc_addresses.into_iter().collect::<Vec<_>>();\n        docs.sort();\n        let mut strings = Vec::with_capacity(docs.len());\n\n        for doc_address in docs {\n            let segment_reader = &self.searcher.segment_readers()[doc_address.segment_ord as usize];\n            let str_column_opt = segment_reader.fast_fields().str(\"str_fast\");\n\n            if let Ok(Some(str_column)) = str_column_opt {\n                let doc_id = doc_address.doc_id;\n                let term_ord = str_column.term_ords(doc_id).next().unwrap();\n                let mut str_buffer = String::new();\n                if str_column.ord_to_str(term_ord, &mut str_buffer).is_ok() {\n                    strings.push(str_buffer);\n                }\n            }\n        }\n\n        strings\n    }\n}\n\nstruct FetchAllStringsFromDocTask {\n    searcher: Searcher,\n    query: RangeQuery,\n}\n\nimpl FetchAllStringsFromDocTask {\n    #[inline(never)]\n    pub fn run(&self) -> Vec<String> {\n        let doc_addresses = self.searcher.search(&self.query, &DocSetCollector).unwrap();\n        let mut docs = doc_addresses.into_iter().collect::<Vec<_>>();\n        docs.sort();\n        let mut strings = Vec::with_capacity(docs.len());\n\n        let str_stored_field = self\n            .searcher\n            .schema()\n            .get_field(\"str_stored\")\n            .expect(\"str_stored field should exist\");\n\n        for doc_address in docs {\n            // Get the document from the doc store (row store access)\n            if let Ok(doc) = self.searcher.doc::<TantivyDocument>(doc_address) {\n                // Extract string values from the stored field\n                if let Some(field_value) = doc.get_first(str_stored_field) {\n                    if let Some(text) = field_value.as_value().as_str() {\n                        strings.push(text.to_string());\n                    }\n                }\n            }\n        }\n\n        strings\n    }\n}\n"
  },
  {
    "path": "benches/wiki.json",
    "content": "{\"url\":\"https://en.wikipedia.org/wiki?curid=48687903\",\"title\":\"Jeon Hye-jin (actress, born 1988)\",\"body\":\"\\nJeon Hye-jin (actress, born 1988)\\n\\nJeon Hye-jin (born June 17, 1988) is a South Korean actress.\\nPersonal life.\\nJeon married his \\\"Smile, You\\\" co-star Lee Chun-hee on March 11, 2011. Their daughter, Lee So Yu, was born on July 30, 2011.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687919\",\"title\":\"Benham, Indiana\",\"body\":\"\\nBenham, Indiana\\n\\nBenham is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nAn old variant name of the community was Benhams Store. A post office opened under the name Benham Store in 1866, the name was shortened to Benham 1888, and the post office was discontinued in 1934. John Benham, Jr., served as a first postmaster.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687922\",\"title\":\"Hilyat al-Muttaqin\",\"body\":\"\\nHilyat al-Muttaqin\\n\\nHilyat al-Muttaqin (\\\"The adornment of the God-fearing\\\", ) is a Hadith book of Muhammad Baqir al-Majlisi. This work is written in Persian about Islamic morality, instructions and traditions.\\nThe aim of writing.\\nAccording to book's foreword, it was written because of a group of Muslims asked Majlisi to write a Persian book in the Islamic morality, instructions and traditions from the hadith of Ahl al-Bayt.\\nDate of writing.\\nAccording a manuscript, the date of writing of this work is 1671, But in another book has been mentioned to 1668-9.\\nContent and chapters.\\nThe book has 14 chapters about individual and collective morality and some Fiqh rulings, Duasand practices and had an extra chapter about some etiquette miscellaneous and their benefits. The titles of chapters are mentioned below:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687925\",\"title\":\"MK Preshow Shimray\",\"body\":\"\\nMK Preshow Shimray\\n\\nMK Preshow Shimray is the sitting MLA from 46 - Chingai (ST) Assembly Constituency in Manipur, India. He was elected under Indian National Congress ticket in 2012.\\nEarly life.\\nMK Preshow Shimray was born on April 5, 1966 at Poi village to MK Somi Shimray. He did his B.E and M.E (Environment) at Salem Engineering College, Tamil Nadu. After completing his education, he worked as Scientific Officer and thereafter as Senior Scientific Officer in the Environment and Ecology Department, Manipur. \\nPolitical career.\\nIn 2012, he resigned from his engagement as senior scientific officer in order to contest the MLA election under Congress ticket. He got elected beating his nearest rival with a simple majority vote. In July 2013, MK Preshow Shimray was elected as the Deputy Speaker of the Manipur Legislative Assembly and is still serving in that capacity.\\nAssasination bids.\\nOn April 30, 2013, the cavalcade of the deputy speaker was ambushed near Ukhrul by suspected NSCN (IM) cadres, however, there were no reports of casualty. The second bid on his life was made on April 9, 2014 by some rebel groups near Litan but, there too he escaped unharmed.\\nAnti tribal bills protests.\\nAll tribal MLAs in Manipur were requested to resign in protest against the passing of three anti tribal bills in Manipur Ligislative Assembly on 30 August 2015. However, many tribal MLAs paid no heed to the call of the tribal people for which many elected representatives were ostrcised from their respective constituencies. MK Preshow Shimray along with the sitting MLA of 44-Phungyar MLA constituency Victor Keishing (son of Rishang Keishing)were also declared anti-socials by the Tangkhul frontal organisations for failing to tender their resignation letters.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687930\",\"title\":\"Clinton, Ripley County, Indiana\",\"body\":\"\\nClinton, Ripley County, Indiana\\n\\nClinton is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nClinton was founded in 1833.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687935\",\"title\":\"NoRMA\",\"body\":\"\\nNoRMA\\n\\nNo Remote Memory Access (abbreviated as NoRMA) is a computer memory architecture for multiprocessor systems, given its name by . In a NoRMA architecture, the address space globally is not unique and the memory is not globally accessible by the processors. Accesses to remote memory modules are only indirectly possible by messages through the interconnection network to other processors, which in turn possibly deliver the desired data in a reply message. The entire storage configuration is partitioned statically among the processors.\\nThe advantage of the NoRMA model is the ability to construct extremely large configurations, which is achieved by shifting the problem to the user configuration. Programs for NoRMA architectures need to evenly partitioning the data into local memory modules, ensure consistency of software caches to enforce the desired consistency model, handle transformations of data identifiers from one processor's address space to another, and realize a message-passing system for remote access to data. The programming model of Norma architecture is therefore extremely complicated.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687945\",\"title\":\"National Capitol Wing Civil Air Patrol\",\"body\":\"\\nNational Capitol Wing Civil Air Patrol\\n\\nThe National Capital Wing of the Civil Air Patrol (CAP) is the highest echelon of Civil Air Patrol in the district of Washington, D.C. The National Capital Wing consists of nearly 400 cadet and adult members at over 6 locations across the district of Washington D.C.\\nMission.\\nThe National Capital Wing performs the three missions of the Civil Air Patrol: providing emergency services; offering cadet programs for youth; and providing aerospace education for both CAP members and the general public.\\nEmergency services.\\nThe Civil Air Patrol provides emergency services, which includes performing search and rescue and disaster relief missions; as well as assisting in humanitarian aid assignments. The CAP also provides Air Force support through conducting light transport, communications support, and low-altitude route surveys. The Civil Air Patrol can also offer support to counter-drug missions. \\nCadet programs.\\nThe Civil Air Patrol offers a cadet program for youth aged 12 to 21, which includes aerospace education, leadership training, physical fitness and moral leadership. \\nAerospace education.\\nThe Civil Air Patrol offers aerospace education for CAP members and the general public, including providing training to the members of CAP, and offering workshops for youth throughout the nation through schools and public aviation events. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687949\",\"title\":\"Bae Seong-woo\",\"body\":\"\\nBae Seong-woo\\n\\nBae Seong-woo (born November 21, 1972) is a South Korean actor. He starred in film such as \\\"My Love, My Bride\\\" (2014), \\\"Office\\\" (2015), \\\"\\\" (2015) and \\\"Inside Men\\\" (2015).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687980\",\"title\":\"Cross Roads, Ripley County, Indiana\",\"body\":\"\\nCross Roads, Ripley County, Indiana\\n\\nCross Roads is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nThe community was so named for the fact it originally contained a store at a crossroads.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48687986\",\"title\":\"Ponderosa Skatepark\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688000\",\"title\":\"Acta Orientalia\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688007\",\"title\":\"Roger Nkodo Dang\",\"body\":\"\\nRoger Nkodo Dang\\n\\nRoger Nkodo Dang is the president of the Pan-African Parliament and is also a member of the National Assembly of Cameroon. On 27 May 2015 Dang was elected as successor to Bethel Nnaemeka Amadi, he obtained 85 votes.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688008\",\"title\":\"Forth Bridge approach railways\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688012\",\"title\":\"Frederick Charles Newcombe\",\"body\":\"\\nFrederick Charles Newcombe\\n\\nFrederick Charles Newcombe (1858-1927) was an American botanist, and the first editor-in-chief of the \\\"American Journal of Botany\\\" \\nNewcombe was born in Flint, Michigan, May 11, 1858, to parents Thomas and Eliza (Gayton) Newcombe, who had emigrated from England in 1848. His early education was obtained in the public schools of Flint. From 1880 to 1887 he taught in the Michigan School for the Deaf at Flint. In 1887 he entered the University of Michigan, and was graduated Bachelor of Science in 1890. He was immediately appointed Instructor in Botany at the University. The year 1892-1893 was spent at the University of Leipzig, where he received the degree of Doctor of Philosophy at the end of the year. He returned to Ann Arbor to become Acting Assistant Professor of Botany in theUniversity. Two years later he became Assistant Professor of Botany, and in 1897 Junior Professor. In 1905 he was made Professor of Botany. \\nNewcombe was a fellow of the American Association for the Advancement of Science, of which he was one of the secretaries in 1899; a member of the Botanical Society of America and its 7th president; of the Society for Plant Morphology and Physiology, and its first vice-president in 1901; and of the Michigan Academy of Science. Of the last-named he was secretary in 1894, vice-president from 1894 to 1896, and president in 1903.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688019\",\"title\":\"Irfan Siddiqui\",\"body\":\"\\nIrfan Siddiqui\\n\\nIrfan Siddiqui is a senior journalist from Pakistan. He was born in Rawalpindi, Pakistan. Starting his career from teaching he gained much fame in the field of Journalism and currently is working with Prime Minister of Pakistan Nawaz Sharif as an advisor.\\nEarly life.\\nAfter Completing his education from University of the Punjab, Lahore Siddiqui Started Teaching at various levels in different Federal Government institutions. he also served as Director Education at Overseas Pakistanis Foundation (OPF) for two years.\\nCareer.\\nSidiiqui started his career as a professional journalist in early 1990s working with \\\"Takbeer\\\" a political magazine at that time. He also started writing columns in daily newspaper \\\"Jang\\\". In January 1998, he was appointed as press secretary to the President of Pakistan Mr. Mohammd Rafique Tarrar \\nIn 2001, he joined Nawa-e-waqt and remained till 2008 when he joined again Daily Jang and currently working with newspaper.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688020\",\"title\":\"William Van Waters\",\"body\":\"\\nWilliam Van Waters\\n\\nWilliam Van Waters was a member of the Wisconsin State Assembly.\\nBiography.\\nVan Waters was born on October 7, 1817 in Hounsfield, New York. He later settled in Hamilton, Wisconsin, where he was a farmer.\\nPolitical career.\\nVan Waters was a member of the Assembly during the 1877 session. Previously, he had been an unsuccessful candidate in 1874. Additionally, Van Waters was Chairman of the Town Board (similar to city council) of Hamilton. He was a Democrat.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688028\",\"title\":\"Matt Longacre\",\"body\":\"\\nMatt Longacre\\n\\nMatt Longacre (born September 21, 1991) is an American football defensive end for the Los Angeles Rams of the National Football League (NFL). He played college football at Northwest Missouri State University. He signed with the Rams as an undrafted free agent in 2015.\\nProfessional career.\\nSt. Louis/Los Angeles Rams.\\nAfter going unselected in the 2015 NFL Draft, Longacre signed with the St. Louis Rams on May 8, 2015. He was waived for final roster cuts before the start of the 2015 season, but signed to the team's practice squad on September 6. On November 6, he was promoted to the active roster.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688037\",\"title\":\"Mehrunnisa Parvez\",\"body\":\"\\nMehrunnisa Parvez\\n\\nMehrunnisa Parvez is an Indian writer of Hindi literature. \\nLife.\\nShe was born in 1944, she published her first story in 1963 in \\\"Dharamayug\\\" magazine and has authored several short stories and novels in Hindi. Amma, published in 1967 and Samara, released in 1969 are two of her notable works. Besides, she has also published several short story anthologies and her works have been the subject of academic studies. The Government of India awarded her the fourth highest civilian honour of the Padma Shri, in 2005, for her contributions to Indian literature. \\nFamily.\\nShe is married to Bhagirath Prasad, a retired Indian Administrative Service officer and active politician, and the couple lives in Bhopal, in Madhya Pradesh.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688041\",\"title\":\"Dewberry, Indiana\",\"body\":\"\\nDewberry, Indiana\\n\\nDewberry is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Dewberry in 1882, and remained in operation until 1887.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688043\",\"title\":\"Koderma (community development block)\",\"body\":\"\\nKoderma (community development block)\\n\\nKoderma (also spelled as Kodarma) is a community development block that forms an administrative division of Koderma district, Jharkhand state, India. \\nGeography.\\nKarma, a constituent town in Koderma CD Block, is located at .\\nPanchayats in Koderma CD Block are: Bekobar (North), Bekobar (South), Charadih, Chhatarbar, Dumardiha, Inderwa, Jarga, Jhumri, Karma, Kauawar Gajhandi, Kharkotta, Kolgarma, Lariyadih, Lokai, Meghatari, Pandedih, Pathaldiha and Purnanagar.\\nDemographics.\\nAs per 2011 Census of India Kodarma CD Block had a total population of 93,240, of which 80,303 were rural and 12,937 were urban. There were 47,584 males and 45,656 females. Scheduled Castes numbered 15,428 and Scheduled Tribes numbered 454.\\nBekobar is a census town in Kodarma CD Block with a population of 7,184 in 2011 and Karma is a census town with a population 0f 5,753 in 2011.\\nLiteracy.\\nAs per 2011 census the total number of literates in Kodarma CD Block was 46,695 out of which 30,607 were males and 19,088 were females.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688048\",\"title\":\"The Modern Art of Jazz by Zoot Sims\",\"body\":\"\\nThe Modern Art of Jazz by Zoot Sims\\n\\nThe Modern Art of Jazz by Zoot Sims (also released as One to Blow On) is an album by American jazz saxophonist Zoot Sims recorded in 1956 and released on the Dawn label.\\nReception.\\nAllmusic awarded the album 4 stars, with the review by Ken Dryden stating: \\\"These early 1956 sessions feature Zoot Sims in top form playing a pair of standards and originals by members of the quintet. Bob Brookmeyer is the perfect foil for the tenor saxophonist, as they seamless interweave intricate lines throughout the record\\\".\\nTrack listing.\\n\\\"All compositions by Zoot Sims except as indicated\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688055\",\"title\":\"Roque Gallart\",\"body\":\"\\nRoque Gallart\\n\\nRoque Josė Gallart Ortiz, (born September 21, 1979), also known as Rocky the Kid, is a Puerto Rican actor and radio and television personality. Gallart is host of a Mega-TV television show named \\\"Descarao' Por La Noche\\\".\\nBiography.\\nRoque José Gallart Ortiz is the son of Roque José Gallart, Sr., himself a well-known radio announcer. Roque, Jr. would visit his dad's radio station and stay there until a school bus picked him up as a little kid. Roque, Jr. became interested in radio work during this era in his life.\\nIn 1995, at age 16, he was hired by the radio station his dad worked for, 95X, in the Puerto Rican city of Bayamon. A few years later, he began to attend Universidad del Sagrado Corazon in Rio Piedras to study communications, but, feeling he was more advanced in his understanding of the radio world than his fellow students, he soon left the university and concentrated on his radio job.\\nActing.\\nRocky the Kid became a well known radio host in Puerto Rico over the years, and his fame allowed him to branch out into other areas of the show business world, starting in 2010, when he participated in \\\"\\\", playing himself along fellow radio personalities, actors and friends \\\"Tony Banana\\\" and Billy Fourquet. He then re-joined Fourquet in a 2011 comedy film named \\\"Que Joyitas!\\\", where he again played a version of himself.\\nIn 2013, Gallart again played himself in \\\"Que Joyitas! II\\\", in which he was featured in the movie's poster. The three movies that Gallart has been in have been released in local cinemas in Puerto Rico, and Que Joyitas! has been shown in the United States television on Cine Latino.\\nLo se Todo.\\nFollowing a severe boycott by sponsors of Kobbo Santarrosa's gossip show, SuperXclusivo, WAPA America took that show off the air, and then hired six people, including Pedro Juan Figueroa, Sylvia Hernandez, \\\"Topy Mamery\\\" and Gallart, to host a new television show named \\\"Lo Se Todo\\\" in 2013. \\\"Lo Se Todo\\\" became a successful and highly rated show in Puerto Rican television, but internal turmoil plagued the show almost from the beginning. During 2014, Gallart once played a Black character in the show, with monkey sounds played in the background. As a consequence, the network received several calls from complaining viewers, which led to Gallart issuing a public apology in which he stated his intention was never to offend anyone based on race, religion, birth place or sexual preference, and a network producer to be fired.\\nEarly in 2015, Gallart decided to leave \\\"Lo Se Todo\\\" after his radio employer, SBS, told him he would not be allowed to work with them and at WAPA-America consecutively. Offered a contract by WAPA-America on December 6, 2014, Gallart did not sign it, deciding instead to keep working on radio. The contract offer by WAPA-America came only 4 days after Mamery, who had also left \\\"Lo Se Todo\\\" and whom Gallart had befriended, died of a heart attack. Following the news of his friend's death, Gallart cried on live television.\\nIn April 2015, Gallart declared on his radio show that \\\"Lo se Todo\\\" television personnel were victims of workplace harassment by show producer Niria Ruiz, also mentioning Joe Ramos in his comment, calling Pedro Juan Figueroa a hypocrite and traitor who never took the other hosts side against \\\"Lo Se Todo\\\" producers during arguments, affirming that Frankie Jay, another of the show's hosts, did so, and describing working conditions at \\\"Lo Se Todo\\\" as a \\\"living hell\\\".\\nAs a consequence of those statements, on August 31, 2015, WAPA-America sued Gallart for the amount of $2,000,000 dollars, accusing him of defamation.\\nMove to Mega-TV.\\nOn March 9, 2015, Gallart began a new television show on Mega-TV, a Hispanic channel in the United States. The show, named \\\"Descarao' Por La Noche\\\" is seen in Puerto Rico and various cities in the United States, including Phoenix, New York City, LA and San Francisco, Houston, Miami and Chicago, among others.\\nPersonal.\\nGallart is married to \\\"Lo Se Todo\\\" host Jessica Serrano. Their first daughter, Emma Catalina Gallart Serrano, was born on November 14, 2015. He also has a son, Ricky, from an earlier relationship, and a step-daughter, Carolina.\\nGallart was arrested and briefly jailed in November 2013 by Puerto Rican police after allegedly sending threatening messages to a former girlfriend. He was released almost immediately when a judge found no cause against him.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688059\",\"title\":\"Elrod, Indiana\",\"body\":\"\\nElrod, Indiana\\n\\nElrod is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Elrod in 1849, and remained in operation until 1903. George W. Elrod, an early postmaster, gave the community his name.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688069\",\"title\":\"List of international cricket centuries at the Wanderers Stadium\",\"body\":\"\\nList of international cricket centuries at the Wanderers Stadium\\n\\nThe Bidvest Wanderers Stadium is a cricket stadium, situated just south of Sandton in Illovo, Johannesburg in Gauteng Province, South Africa. Test, One Day and First class cricket matches are played here. The stadium has a seating capacity of 34,000. The ground is among the most historically significant cricket grounds of the twenty-first century. It has staged some of the most important matches in ODI and T20I history, and has witnessed a number of outstanding world records.\\nThe 2003 Cricket World Cup final was held at the Wanderers Stadium. This stadium also hosted one of the greatest One-Day International matches. The match was played between South Africa and Australia in which a world record score of 434 was chased down by South Africa. It hosted matches of the 2007 ICC World Twenty20 including the first match and the final which was won by India, who defeated Pakistan.\\nTest centuries.\\nThe following table summarises the Test centuries scored at the Wanderers.\\nOne Day International centuries.\\nThe following table summarises the One Day International centuries scored at the Wanderers.\\nTwenty20 International centuries.\\nThe following table summarises the Twenty20 International century scored at the Wanderers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688080\",\"title\":\"Lookout, Indiana\",\"body\":\"\\nLookout, Indiana\\n\\nLookout is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Lookout in 1889, and remained in operation until 1906.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688100\",\"title\":\"Dhakuakhana College\",\"body\":\"\\nAwakening the sense of unity, national-integrity and environmental awareness.\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688110\",\"title\":\"Negangards Corner, Indiana\",\"body\":\"\\nNegangards Corner, Indiana\\n\\nNegangards Corner is an unincorporated community in Ripley County, in the U.S. state of Indiana. \\nHistory.\\nNegangards Corner had its start when a general store opened at the town site. An old variant name of the community was North Hogan. A post office opened under the name North Hogan in 1844, and remained in operation until 1877.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688133\",\"title\":\"Zhenjiang dialect\",\"body\":\"\\nZhenjiang dialect\\n\\nZhenjiang dialect is a form of Eastern Mandarin spoken in the town of Zhenjiang in Jiangsu Province. The town is situated on the south bank of the Yangtze river between Nanjing and Changzhou. It is thus at the intersection of China's Mandarin and Wu speaking regions. About 2.7 million Chinese live in the area where the Zhenjiang dialect is predominant.\\nIn ancient times, Zhenjiang spoke Wu. Today, Wu is the language of nearby Changzhou, as well as Shanghai and Zhejiang Province. Mandarin speakers from the North have been immigrating to Zhenjiang since the fourth century, gradually changing the character of the local dialect. In modern times, the city speaks a dialect that is transitional between the Eastern Mandarin of Nanjing, located just west of the city, and the Taihu dialect of Wu spoken in Changzhou, which is just east of the city. Zhenjiang dialect is comprehensible to Nanjing residents, but not to Changzhou residents.\\nThe issue of tones in the Zhenjiang dialect has been a topic scholarly study. Nanjing residents use the four tones of Mandarin, while Changzhou residents use seven or eight tones. According to a study by Qiu Chunan, Zhenjiang dialect has five citation tones: Tone1 (42) (a sharp fall from pitch 4 to pitch 2, or \\\"yinping\\\"), Tone2 (35) (a rising tone or \\\"yangping\\\"), Tone3 (32) (slight falling tone or \\\"shang\\\"), Tone4 (55) (high even or \\\"qu\\\"), and Tone5 (5) (checked tone or \\\"ru\\\"). Qiu's study used residents who had grown up in the Daxi Road area, where the standard form of the dialect is said to be spoken. The checked tone was a feature of Chinese spoken in the Middle Ages, but it is not part of Mandarin. Applying the theory of government phonology to the issue, Bao Zhiming noted that non-even tones become even when they appear before the high even, or 55, tone.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688137\",\"title\":\"Mister Universal Ambassador 2015\",\"body\":\"\\nMister Universal Ambassador 2015\\n\\nMister Universal Ambassador 2015, the 1st edition of the Mister Universal Ambassador pageant, held on November 11, in Surabaya, Indonesia. 19 contestants competed for the title. Christian Daniel of Puerto Rico crowned as the first ever Mister Universal Ambassador.\\nCrossovers.\\nMister International 2014\\nManhunt International 2011\\nMister Pancontinental 2015\\nMister Global 2014\\nMister Revolution Model 2014\\nMen Universe 2012\\nMister Asia 2014\\nReferences.\\nwww.mruniversalambassador.com\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688167\",\"title\":\"Cameron Lynch\",\"body\":\"\\nCameron Lynch\\n\\nCameron Ellis Lynch (born August 4, 1993) is an American football linebacker for the Los Angeles Rams of the National Football League (NFL). He played college football at Syracuse University. He signed with the Rams as an undrafted free agent in 2015.\\nProfessional career.\\nSt. Louis/Los Angeles Rams.\\nAfter going unselected in the 2015 NFL Draft, Lynch signed with the St. Louis Rams on May 8, 2015. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688180\",\"title\":\"TV Bel Kanal\",\"body\":\" The program is mainly produced in Serbian language. TV station was established in 2003. TV Bel reports on local events in Banja Luka, Republika Srpska entity and BiH.\\nThe channel broadcasts documentaries from domestic and foreign production, TV series, movies and entertainment. Channel is also part of local news network in the RS entity called PRIMA mreža ().\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688246\",\"title\":\"Abhinav Bharti High School\",\"body\":\"\\nAbhinav Bharti High School\\n\\nAbhinav Bharti High School is a private English-medium co-ed school located in Pretoria street, Kolkata, West Bengal, India. This school is affiliated to CBSE. The school was established in 1945. The current principal of this school is Mrs Sanghmitra Mukherjee.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688271\",\"title\":\"Otter Village, Indiana\",\"body\":\"\\nOtter Village, Indiana\\n\\nOtter Village is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nOtter Village was laid out in 1837. The community took its name from Otter Creek.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688273\",\"title\":\"Patrick J. Bergin\",\"body\":\"\\nPatrick J. Bergin\\n\\nPatrick J. Bergin has been the Chief Executive Officer of the African Wildlife Foundation since 2002.\\nHe graduated from the University of Illinois at Urbana–Champaign with an MSc in International Agricultural Education in 1988, and from the University of East Anglia with a PhD in Development Studies in 1996.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688292\",\"title\":\"Penntown, Indiana\",\"body\":\"\\nPenntown, Indiana\\n\\nPenntown is an unincorporated community in Ripley County, in the U.S. state of Indiana. An old variant name was Pennsylvaniaburg.\\nHistory.\\nPenntown was originally called Pennsylviaburg, and under the latter name was laid out in 1837. The community was named after Pennsylvania, the native state of a share of the early settlers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688302\",\"title\":\"Cherry picker (disambiguation)\",\"body\":\"\\nCherry picker (disambiguation)\\n\\nA cherry picker is a platform for lifting someone to work at a high level.\\nCherry picker may also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688319\",\"title\":\"Varian's theorems\",\"body\":\"\\nVarian's theorems\\n\\nIn welfare economics, Varian's theorems are several theorems related to fair allocation of homogeneous divisible resources. They describe conditions under which there exists a Pareto efficient (PE) envy-free (EF) allocation. They were published by Hal Varian in the 1970s.\\nExamples.\\nAll examples involve an economy with two goods, x and y, and two agents, Alice and Bob.\\nA. Many PEEF allocations: Alice and Bob have linear utilities, representing substitute goods:\\nThe total endowment is (4,4). If Alice receives at least 3 units of x, then her utility is 6 and she does not envy Bob. Similarly, if Bob receives at least 3 units of y, he does not envy Alice. So the allocation [(3,0);(1,4)] is PEEF with utilities (6,9). Similarly, the allocations [(4,0);(0,4)] and [(4,0.5);(0,3.5)] are PEEF. On the other hand, the allocation [(0,0);(4,4)] is PE but not EF (Alice envies Bob); the allocation [(2,2);(2,2)] is EF but not PE (the utilities are (6,6) but they can be improved e.g. to (8,8)).\\nB. Essentially-single PEEF allocation: Alice and Bob have Leontief utilities, representing complementary goods:\\nThe total endowment is (4,2). The equal allocation [(2,1);(2,1)] is PEEF with utility vector (1,1). EF is obvious (every equal allocation is EF). Regarding PE, note that both agents now want only y, so the only way to increase the utility of an agent is to take some y from the other agent, but this decreases the utility of the other agent. While there are other PEEF allocations, e.g. [(1.5,1);(2.5,1)], all have the same utility vector of (1,1), since it is not possible to give both agents more than 1. \\nC. No PEEF allocations: Alice and Bob have concave utilities:\\nThe total endowment is (4,2). The equal allocation [(2,1);(2,1)] is EF with utility vector (2,2). Moreover, \\\"every\\\" EF allocation must give both agents equal utility (since they have the same utility function) and this utility can be at most 2. However, no such allocation is PE, since it is Pareto-dominated by the allocation [(4,0);(0,2)] whose utility vector is (4,2).\\nExistence of PEEF allocations with monotone convex preferences.\\nVarian's theorem says that:\\nIn the #Examples, the preferences are always monotone. However, only in examples A and B the preferences are convex.\\nThe proof relies on the existence of a competitive equilibrium with equal incomes. Assume that all resources in an economy are divided equally between the agents. I.e, if the total endowment of the economy is formula_5, then each agent formula_6 receives an initial endowment formula_7.\\nSince the preferences are \\\"convex\\\", the Arrow–Debreu model implies that a competitive equilibrium exists. I.e, there is a price vector formula_8 and a partition formula_9 such that:\\nSuch an allocation is always EF. Proof: by the (EI) condition, for every formula_13. Hence, by the (CE) condition, formula_14.\\nSince the preferences are \\\"monotonic\\\", any such allocation is also PE, since monotonicity implies local nonsatiation. See fundamental theorems of welfare economics.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688336\",\"title\":\"Hong Kong Scout Progressive Badge Scheme\",\"body\":\"\\nHong Kong Scout Progressive Badge Scheme\\n\\nThe Scout Progressive Badge scheme is a scheme which assess scouts. This scheme can reflect the overall ability of scouting skills.\\nMembership Badge.\\nThis is a badge which is required for all scouts.\\nBackground Requirement.\\n11 or above\\nPathfinder Award.\\nPathfinder Badge is the second stage of the Scout.\\nBackground Requirement.\\n11 or above\\nVoyager Award.\\nThis is the badge most scouts will get. It is the third stage of the progressive badge scheme.\\nRequirement.\\nScoutcraft.\\nSome of the activities must be conducted under the supervision of adults.\\nChallenger Award.\\nThe Challenger award is the fourth stage and the second last stage of the scheme.\\nRequirement.\\nScoutcraft.\\nSome of the activities must be conducted under the supervision of adults.\\nChief Scout's Award.\\nThis is the highest stage. Every year several hundred scouts attain this award (ratio 1:180-190). They can participate he Scout Rally regardless of their troop.\\nBasic Requirement.\\nthe \\\"Scoutcraft\\\", \\\"Adventure\\\", \\\"Commitment\\\" and \\n\\\"Proficiency/ Activity Badge\\\" sections of the Pathfinder Award, \\nVoyager Award and Challenger Award.\\nRequirement.\\nScoutcraft.\\nLeaders’ Council\\nPatrol. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688348\",\"title\":\"Sathanur, Mandya\",\"body\":\"\\nSathanur, Mandya\\n\\nSathanur is a village in Mandya district, Karnataka, India. Sathanur village is located in the SH-84, just 3 km from Bangalore-Mysore National highway -275.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688351\",\"title\":\"NTV Arena\",\"body\":\" The program is mainly produced in Serbian language. TV station was established in 1999. NTV Arena reports on local events in Bijeljina, Republika Srpska entity and BiH.\\nThe channel broadcasts TV series,entertainment and news. Channel is also part of local news network in the RS entity called PRIMA mreža ().\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688353\",\"title\":\"Joshua Oigara\",\"body\":\"\\nJoshua Oigara\\n\\nJoshua Nyamweya Oigara is the Group Chief Executive Officer of the Kenya Commercial Bank Group. At age 37, his appointment in November 2012 to replace the outgoing CEO Martin Oduor-Otieno made him the youngest CEO of a publicly traded bank at the NSE. Prior to his appointment, he served as Chief Financial Officer and Member of the Board of Directors of the Company between January 12, 2012 and January 2013. He also served as Group Chief Financial Officer at KCB Bank Group for East Africa. He joined the Bank in November 2011 from Bamburi Cement where he served as Group Financial Director and Chief Financial Officer for the East Africa region. Mr. Oigara holds a Bachelor of Commerce degree from University of Nairobi and Master of Business Administration (M.B.A.) from Edith Cowan University and a host of other qualifications\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688354\",\"title\":\"Mount Alexander (Antarctica)\",\"body\":\"\\nMount Alexander (Antarctica)\\n\\nMount Alexander () is a mountain with several summits, the highest 595 m, forming the rocky peninsula separating Gibson and Haddon Bays, on the south side of Joinville Island in Antarctica. The cliff marking the extremity of the peninsula was discovered and named Cape Alexander on January 8, 1893 by Thomas Robertson, master of the ship \\\"Active\\\", one of the Dundee whalers. The name was amended to Mount Alexander by the United Kingdom Antarctic Place-Names Committee (UK-APC) in 1956 following a survey by the Falkland Islands Dependencies Survey (FIDS) in 1953–54, the mountain summits of the peninsula being considered more suitable to name.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688373\",\"title\":\"Paul Gundani\",\"body\":\"\\nPaul Gundani\\n\\nPaul Gundani (1967 – 3 November 2015) was a Zimbabwean professional football player. He made three appearances for the Zimbabwe national football team.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688376\",\"title\":\"Frank Tipton\",\"body\":\"\\nFrank Tipton\\n\\nFrank Ben Tipton (born in California) is an Australian historian and Emeritus Professor at The University of Sydney Business School. He is known for his works on Modern history of Germany and Economic history.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688378\",\"title\":\"Mount Alfred (Antarctica)\",\"body\":\"\\nMount Alfred (Antarctica)\\n\\nMount Alfred () is an ice-capped mountain, more than 2,000 m, 5.5 nautical miles (10 km) inland from George VI Sound and 8 nautical miles (15 km) south of Mount Athelstan in the Douglas Range of Alexander Island, Antarctica. It was first photographed from the air on November 23, 1935, by Lincoln Ellsworth and mapped from these photos by W.L.G. Joerg. Its east face was roughly surveyed in 1936 by the British Graham Land Expedition (BGLE) and resurveyed in 1948 and 1949 by the Falkland Islands Dependencies Survey (FIDS), who named it for Alfred, Saxon king of England, 871-899. The west face of the mountain was mapped from air photos taken by the Ronne Antarctic Research Expedition (RARE), 1947–48, by Searle of the FIDS in 1960.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688385\",\"title\":\"WTA Player of the Year\",\"body\":\"\\nWTA Player of the Year\\n\\nThe WTA Player of the Year is an annual Women's Tennis Association (WTA) award given since the 1977 WTA Tour.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688397\",\"title\":\"Raymond Jack Last\",\"body\":\"\\nRaymond Jack Last\\n\\nRaymond Jack Last (26 May 1903 Adelaide - 1 January 1993 Malta), comparative anatomist, was Anatomical Curator Royal College of Surgeons 1946-50, Professor of Applied Anatomy 1950-70, Warden of Nuffield College of Surgical Sciences 1949-70, and Visiting Professor UCLA 1970-88. Last was an outstanding lecturer in postgraduate anatomy, and belonged to a small set of anatomists who were initially practising surgeons and then made a career teaching it. \\nHenry Last, his grandfather, was from Debenham in Suffolk and jumped ship to settle in South Australia. His wife was Mary Ann Bowden who had Cornish roots. They later settled in Adelaide. \\nHenry's eldest surviving son, John Last, worked at an Adelaide stationer and bookseller. John and his wife had 3 children, Raymond and two younger sisters, all being diligent and studious.\\nRaymond first attended North Adelaide Primary School, from where he went on to the Adelaide Boys’ High School, where another scholar and classmate was Mark Oliphant, who went on to become an eminent physicist and later, Governor of South Australia. \\nAssisted by a State Bursary, he enrolled at the University of Adelaide, and though underage, obtained permission to start the medical course. A brilliant trio headed the Medical School — John Burton Cleland in Pathology, Thorburn Brailsford Robertson in Physiology, and Frederic Wood Jones in Anatomy. Last was top student in all but the final year, and graduated MB BS in 1924.\\nIn June 1939 Last and his second wife Margret, who had been Matron at Booleroo Centre District Hospital, set off for England in order to become a Fellow of the Royal College of Surgeons. With the start of World War Two in September 1939 they began work at the Emergency Medical Service at the North London Fever Hospital at Winchmore Hill, later to become an annexe of the Royal London Hospital. Here Last worked under Henry Souttar.\\nAs a member of the Australian Army Medical Corps Last was turned down by the British Army, so he was obliged to return to Australia to enlist there. \\\"MV Napier Star\\\" was torpedoed in the Irish Sea on 18 December 1940 (see List_of_shipwrecks_in_December_1940#18_December), only 28 of 99 people surviving, among which were Raymond and Margret Last. Raymond Last wrote an account of the event. They joined the British Red Cross Society and Last led a surgical team, his wife Margret being principal nurse, that formed part of the British forces sent to end the Italian occupation of Abyssinia. Last spent three years there, becoming personal physician to the Emperor Haile Selassie and his family. The closing year of the War saw Last heading a medical unit in Borneo, providing medical care for the civilian population.\\nWhen Last returned to London and the Royal College of Surgeons, the Professor of Anatomy was Frederic Wood Jones, who had tutored Last in Adelaide, and inspired an enduring interest in comparative anatomy. After some years as anatomy demonstrator and curator, Last was appointed Professor of Applied Anatomy in 1950. His textbook, 'Anatomy Regional and Applied', was first published in 1954 and heralded a new generation of anatomy texts providing a more concise option to 'Gray's Anatomy' or Cunningham's 'Textbook of Anatomy'. It became known for its readability and interest, as it presented medical and surgical anatomy in a practical light, rather than as a colourless academic exercise. and rapidly gained popularity amongst surgical students. His own drawings were used as illustrations. Last became first Warden of the Nuffield College of Surgical Sciences, the residence for medical students. On his retiring a common room was named in his honour. \\nFrom 1970-87 he held the chair of Visiting Professor in the Department of Anatomy at the University of California, Los Angeles. This appointment enabled his spending winter breaks in Adelaide, where he gave lectures and demonstrations in the dissecting room of the Adelaide Medical School, often making use of his superb chalk diagrams.\\nThe Lasts chose Malta as a final place of retirement, partly because of its tax laws and partly because of his circle of friends, which included expatriate English people, local Maltese and a sprinkling of surgeons. Failing vision limited his drawing skills; and he suffered from senile gait syndrome, necessitating assistance when standing or walking. Margret died in January 1989, and Last stayed on in Malta. \\nHe had two sons, John and Peter, by his first wife, both graduating in medicine from Adelaide University. He had no share in their upbringing. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688423\",\"title\":\"WTA Doubles Team of the Year\",\"body\":\"\\nWTA Doubles Team of the Year\\n\\nThe WTA Doubles Team of the Year is an annual Women's Tennis Association (WTA) award given since the 1977 WTA Tour.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688424\",\"title\":\"Darchhawna\",\"body\":\"\\nDarchhawna\\n\\nDarchhawna is an Indian writer of Hindi literature and historian from the Northeast Indian state of Mizoram. Born on the New year Day of 1936, Darchhawna is a former Officer on Special Duty at the Mizoram University, when it was the Mizoram campus of the North Eastern Hill University (NEHU) and the founder President of the Mizo History Association. He held the post of the president of the organization for several terms and is holding post, after getting elcted in 2013. The Government of India awarded him the fourth highest civilian honour of the Padma Shri, in 2005, for his contributions to Indian literature.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688452\",\"title\":\"Gopikishan Bajoria\",\"body\":\"\\nGopikishan Bajoria\\n\\nGopikishan Bajoria () is a Shiv Sena politician from Akola district, Maharashtra. He is current Member of Legislative Council as a member of Shiv Sena representing Akola-cum-Washim-cum-Buldhana Local Authorities constituency. He has been elected to Maharashtra Legislative Council for three consecutive terms for 2004, 2010 and 2016.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688455\",\"title\":\"RTV IS\",\"body\":\" The program is mainly produced in Serbian language. TV station was established in 2008. Local radio station Radio Istočno Sarajevo is also part of this company. The channel broadcasts local news, TV series and entertainment. Channel is also part of local news network in the RS entity called PRIMA mreža ().\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688510\",\"title\":\"Jon Rune Strøm\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688536\",\"title\":\"Sabina Selimovic and Samra Kesinovic\",\"body\":\"\\nSabina Selimovic and Samra Kesinovic\\n\\nSabina Selimovic (c.1999–2014?) and Samra Kesinovic (c.1998–2015?) were Austrian teenagers who emigrated to the Islamic State of Iraq and the Levant in April 2014.\\nSelimovic and Kesinovic were born in Austria to Bosnian immigrants who had fled the Bosnian War in the 1990s. Selimovic and Kesinovic, who were residents of Vienna, were believed to have been radicalized after reading about jihad on the Internet through their attendance of a Viennese mosque. The pair left their homes in Vienna in April 2014 to travel to Syria via Turkey to join the Islamic State of Iraq and the Levant. The pair left a note to their parents that stated \\\"Don't look for us. We will serve Allah and we will die for him.\\\" The pair later posted photographs on social media of themselves wearing full length burqas and handling assault weapons. A mutual friend of Selimovic and Kesinovic told \\\"Paris Match\\\" that the pair were married to Chechen fighters, and that they were afraid of imprisonment in Austria should they have returned. In December 2015, a Tunisian woman who defected from ISIS told \\\"The Sun\\\" that she and Kesinovic were kept together in a house in Syria where they served as sexual slaves for jihadis. They were allegedly required to provide sexual services to new recruits.\\nSelimovic and Kesinovic reportedly wished to return to Austria in October 2014, though it had been reported in September that Selimovic had been killed while fighting for Islamic State in Syria. In late 2015 it was reported that Kesinovic was beaten to death with a hammer after trying to escape from sexual slavery in Raqqa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688565\",\"title\":\"Kuweires Military Airbase\",\"body\":\"\\nKuweires Military Airbase\\n\\nThe Kuweires Military Airbase is an airbase in Aleppo Governorate, Syria. It is situated between As-Safira in the West and Dayr Hafir in the East. It was used for educational purposes mainly.\\nOn 18 March 2014, it was reported that an effort entitled “Do Not Divide” [Wa La Tafaraqu] aimed to seize Kuweiris Airbase. Rebel forces have surrounded Kuweiris for more than a year but have not overrun it. However, as rebel infighting with ISIS intensified, ISIS apparently left (or was forced off) the battlefield and a new logo without its name was posted online by another brigade.\\nAfter being sieged by ISIS for more than two years it was freed by the Syrian army in November 2015 in the Kuweires offensive.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688567\",\"title\":\"Town hall of Bordighera\",\"body\":\"\\nTown hall of Bordighera\\n\\nThe Town hall of Bordighera is located at 32 Via XX Settembre in Bordighera, Liguria.\\nHistory.\\nThe current town hall was built to a design by the French architect Charles Garnier. Garnier, who had left Paris because of Paris Commune (1871), was looking for a land in Bordighera to build his future house. In 1871 he proposed the municipality to buy the land where only school of the city stayed. The offer of 6,000 pounds was very generous, it would have allowed the construction of a new school for which Garnier provided the plans.\\nThe project included a classical building that integrates well with the old town of Bordighera that lies behind. The new schools foresaw not only a new section for boys replacing the previous one, but also a girls' section and one for small children. The project is to be considered truly innovative for its time and the municipality accepted the proposal. During the works, which were not followed by Garnier, there were numerous technical problems and delays in deliveries. It was only in 1878 that part of the building was finally terminated.\\nMeanwhile, the population grew and the city decided to build a new school that was opened later in 1886. With the transfer of the school, the municipality decided to use the building as its headquarters. The facade was carved with the coat of arms of the city, a rampant lion that leans against a Pine seafarer. Inside the town hall you can admire numerous paintings from various artists who stayed in Bordighera, including: Pompeo Mariani, Giuseppe Ferdinando Piana, Giuseppe Balbo, Friederich von Kleudgen, etc.\\nThe gardens.\\nThe true gardens of the town hall are between the building designed by Charles Garnier and the Cape Esplanade. These gardens, although of small size, can boast two great Ficus macrophylla centenarians and many varieties of exotic plants (agaves, palm trees, cactus, etc.) which can also be seen in many Bordighera gardens. On 17 January 1954, the city placed a bust in honour of Ludwig Winter, its illustrious adopted citizen, as an honour to the man who had done so much to bring prestige to the city and to the Riviera. This is sometimes confusing. Some mistakenly call these gardens also \\\"Winter Gardens\\\", but the real Winter Garden are at 6 Ludovico Winter Street in Bordighera.\\nAt the foot of the old town hall of Bordighera, there is a large wooden area that is called in various ways: the gardens of the Cape or, more commonly, Cape Pineta. The gardens are vast, and full of very old trees such as pines, olive trees, palm trees and a variety of other species, but very old specimens of Araucaria Heterophylla stand out. In the pine forest there is also a bust in honour of the painter Mosè Bianchi, who often came to Bordighera to spend time with his grandson Pompeo Mariani. Also in the Cape Pineta, but a bit lower, there is a monument in honour of Charles Garnier.\\nOn May 15, 2015 the renovation of the Marabutto was terminated. It is an ancient powder keg that is currently in the Cape Pineta. The name is due to the shape of the small building that recalls the graves of Muslim holy men (a cube topped by a dome), whose name is precisely marabutto. Near the Marabutto, you can see three old cannons so much loved by the locals that they were given names: Butafoegu, Tiralogni et Cacastrasse.\\nTrivia.\\nWhen the city had to choose its coat of arms, a dispute broke out with the neighbouring city of Sanremo. Bordighera, which was nicknamed \\\"Queen of the Palms\\\", wanted to see represented the palm on his coat of arms, but the bigger rival city had already a similar one, so Bordighera had to settle for a pine.\\nBehind the municipal building is one of four Ficus Magnolioides recognised by the State Forestry. The monumental tree has an estimated age of 100 years, with a height of 18 meters and a diameter of the stem of 1040 cm.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688618\",\"title\":\"Summer in the Arab world\",\"body\":\"\\nSummer in the Arab world\\n\\nSummer in the Arab world in the summer perpendicular to the sun's rays on the central parts of the Arab world and so the temperatures, especially in the interior, which is characterized by the poverty of the vegetation and the spread of the sand either areas Aljblahuhal Mediterranean shall be the temperatures moderate making it one of the summer centers Almanmh.uteb in this Chapter monsoon moist from the Atlantic Ocean and the Indian Ocean, causing rainfall in southern Mauritania and southern Somalia, Sudan and Yemen, the Arab world .oamtaz length of the growing season in most parts of allowing the cultivation dense as the climate diversity has helped the diversity of agricultural crops and integration\\nSources.\\nGeography\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688619\",\"title\":\"Thomas Rudge\",\"body\":\"\\nThomas Rudge\\n\\nThomas Rudge (baptised 1753 – 1825) was an English churchman, topographer and antiquarian, Archdeacon of Gloucester from 1814, and chancellor of the diocese of Hereford from 1817.\\nLife.\\nThe son of Thomas Rudge of Gloucester, he matriculated at Merton College, Oxford, on 7 April 1770, aged 16. He graduated B.A. in 1780, and proceeded M.A. from Worcester College in 1783 and B.D. in 1784. He was appointed rector of St. Michael's and St. Mary-de-Grace, Gloucester, and, on the presentation of the Earl of Hardwicke, vicar of Haresfield.\\nRudge died in 1825.\\nWorks.\\nRudge published:\\nNotes.\\nAttribution\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688652\",\"title\":\"Gino D'Auri\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688660\",\"title\":\"Rexville, Indiana\",\"body\":\"\\nRexville, Indiana\\n\\nRexville is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Rexville in 1870, and remained in operation until 1907. The community's name honors the Rex family of settlers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688662\",\"title\":\"Rexville\",\"body\":\"\\nRexville\\n\\nRexville may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688676\",\"title\":\"Vertical perspective\",\"body\":\"\\nVertical perspective\\n\\nVertical perspective may mean:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688681\",\"title\":\"Spades, Indiana\",\"body\":\"\\nSpades, Indiana\\n\\nSpades is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nSpades was laid out in 1855. The community's name honors Jacob Spade, a first settler. An early variant name of the community was Spades Depot. A post office was established as Spade's Depot in 1855, the name was shortened to Spades in 1883, and the post office closed in 1950.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688688\",\"title\":\"Scott Tsumura\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688703\",\"title\":\"Stringtown, Ripley County, Indiana\",\"body\":\"\\nStringtown, Ripley County, Indiana\\n\\nStringtown is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Stringtown in 1848, and remained in operation until 1865.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688714\",\"title\":\"Paolo Rossi (1900-1985)\",\"body\":\"\\nPaolo Rossi (1900-1985)\\n\\nPaolo Rossi (Bordighera, September 15, 1900 - Lucca, 24 May 1985) was an Italian lawyer and politician.\\nBiography.\\nPaolo Rossi is the son of the famous criminal lawyer of Genoa, Francesco Rossi and of Iride Garrone. He comes from an educated and progressive Ligurian family, his cousin is Maria Vittoria Rossi, better known as Irene Brin, the fashion journalist and style icon.\\nThe young man decided to follow his father's footsteps and graduated at the University of Genoa, he enrolled to the Bar of the Appellate court of Genoa at 21 and to Court of cassation at 28.\\nPersecuted by the fascists, in 1926 his study of via Roma in Genoa was destroyed and burned.\\nIn 1932 he wrote his first book \\\"The death penalty and its criticism\\\", which will be blocked by the censors because it opposed the death penalty supported by the fascist regime.\\nHe married Giuseppina Bagnara, called Giugi, whom he met in Bordighera, and he has a daughter, the writer Maria Francesca Rossi, known with her pen name of Francesca Duranti. In 1937 he wrote his second book \\\"Skepticism and dogmatic in criminal law\\\", which was also criticized for its too progressive ideas. During the war, the family moved near Lucca, in Gattaiola. In those years he joins the resistance and, with his wife, he managed to save many young people from fascist raids.\\nIn 1948 he published \\\"The parties against democracy\\\" and, shortly afterwards, was appointed Professor of Criminal Law at the University of Pisa.\\nOn 15 October 1947 he became a member of the Constituent and of the Committee on the Constitution of Italy. The same year he became Professor of Criminal Law at the University of Genoa.\\nA leading member of the Italian Democratic Socialist Party, he was elected as Member of Parliament in the first four legislatures, and took, in two of them, the vice-presidency of the Chamber of Deputies in addition to the presidency of some inquiry commissions. He was Minister of Education from 6 July 1955 to 19 May 1957 during the first Government of Antonio Segni. In 1958 he was appointed Vice-President of the Chamber of Deputies (Italy), and in 1961 President of the Commission on the problems of Alto-Adige.\\nHe was the first president of the Antimafia Commission (in the third Legislature from 14 February to 15 May 1963) which was then called the Parliamentary Commission of Inquiry on the Mafia in Sicily.\\nOn 2 May 1969 he is named Judge of the Constitutional Court of the Italian Republic by the President of the Italian Republic, Giuseppe Saragat, was sworn in on May 9, 1969 and was elected president of the Court on December 18, 1975. He ceased to hold office as President on May 9, 1978, but stayed on as judge until August 2, 1979).\\nHe was the author of many texts, both in the legal and in the political field. From 1970 to 1973 he published a collection of four volumes on the history of Italy, entitled \\\"History of Italy from 476 to the present day”.\\nHe was also president of the general National Corps of Italian Boy Scouts (CNGEI).\\nHe died in Lucca on May 24, 1985 and was buried in the cemetery of Gattaiola.\\nHonours.\\nOn 6 august 1970 he was named “Knight Grand Cross of Order of Merit of the Italian Republic” by the List of Presidents of Italy, Giuseppe Saragat.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688721\",\"title\":\"Mary Pickford Award\",\"body\":\"\\nMary Pickford Award\\n\\nThe Mary Pickford Award is an honorary Satellite Award bestowed by the International Press Academy. It is \\\"IPA’s most prestigious honor\\\" and as an award \\\"for Outstanding Artistic Contribution to the Entertainment Industry\\\" it reflects a lifetime of achievement.\\nThe Award is named for Mary Pickford, early pioneer of the film industry, who began her career as a child actress and went on to become \\\"America's Sweetheart\\\" and a co-founder of United Artists Studios with fellow filmmakers Charlie Chaplin, Douglas Fairbanks and D.W. Griffith.\\nThe award was first presented to Rod Steiger at the 1st Annual Golden Satellite Awards. Ellen Burstyn is the latest recipient.\\nThe trophy awarded to the honorees is a bust of Canadian American motion picture actress Mary Pickford cast in bronze, on a marble base, inscribed for the recipient. It was designed by Sarajevan sculptor Dragan Radenović.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688724\",\"title\":\"Behlmer Corner, Indiana\",\"body\":\"\\nBehlmer Corner, Indiana\\n\\nBehlmer Corner is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nBehlmer Corner was originally called Lynnville, and under the latter name was laid out in 1844.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688733\",\"title\":\"Monica Mary Gardner\",\"body\":\"\\nMonica Mary Gardner\\n\\nMonica Mary Gardner (26 June 1873 – 16 April 1941) was an English writer on Poland and Polish writers and a translator of Polish literature.\\nLife and work.\\nGardner was born in 1873 at Roehampton Lane in London. the eldest of the six children of John Gardner, a member of the stock exchange, and his wife Amy Vernon Garratt. Her brother was the Italian scholar Edmund Garratt Gardner.\\nGardner studied the Polish language and literature after being intrigued at school by Bonnie Prince Charlie's mother who was the Polish aristocrat Clementina Sobieska. In 1899 she began to get assistance in her obsession for Polish and Poland by the writer Edmund Naganowski. He was able to support her until the First World War prevented further communication. Naganowski was to die in 1915. Gardner taught herself how to research sources in Polish and how to find out more about Poland. Her first monograph in 1911 was on Adam Mickiewicz who was regarded as the national poet of Poland.\\nShe followed this with more books on her single theme. She was known as one of the few English speaking writers who studied Polish literature and history. She wrote \\\"Poland: a Study in National Idealism\\\" in 1915 and \\\"The Anonymous Poet of Poland: Zygmunt Krasiński\\\" in 1919. In 1922 she and her brother made what may have been her only visit to Poland. They visited Poznań and Cracow. She wrote the \\\"The Patriot Novelist of Poland: Henryk Sienkiewicz\\\" in 1926. The latter was said to have been published to coincide with the return of Sienkiewicz's body from Switzerland to be encrypted in Warsaw Cathedral.\\nIn between the wars Gardner lived with her mother and her brother who was dedicated to Italy and Italian. Monica's knowledge and expertise made their house a place to visit by notable visiting Poles.\\nWith the outbreak of World War Two, which Britain entered after the German invasion of Poland, Gardner's expertise became more important. She died as a result of a German land mine which landed on her house. One of her manuscripts was recovered from the bomb site, but another was lost. Gardner's funeral was an important event that was attended by the President of Poland, Władysław Raczkiewicz.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688737\",\"title\":\"Romanian Jews in Israel\",\"body\":\"\\nRomanian Jews in Israel\\n\\nRomanian Jews in Israel refers to the community of Romanian Jews who migrated to Palestine beginning in the later 19th century, continued migrating to Israel after the formation of the modern state in 1948, and live within the state of Israel. The descendants of those who made aliyah in 1930s, the wave of emmigrants after World War II or after the fall of communism, with their children and grandchildren born in Israel, represent about 10% of the population. According to the Association of Romanian Journalists Abroad, about 400,000 Romanian Jews live in Israel. Most of them have higher education, occupying important positions in various sectors and making a great contribution to the economic, social and cultural development of Israel. \\nThey have established several kibbutzim, moshavim and towns (Kiryat Bialik, Kiryat Tiv'on, Rosh Pinna, Zikhron Ya'akov). Between 1882 and 1884, Romanian Jews in Israel already established nine localities. \\nImmigration history.\\nMass emigration to Israel ensued (\\\"see Bricha and Aliyah\\\"). According to Sachar, for the first two post-war years, tens of thousands of Romanian Jews left for Israel; the Romanian government did not try to stop them, especially due to its desire to reduce its historically suspect and now impoverished Jewish minority. Afterwards, Jewish emigration began to encounter obstacles. In 1948, the year of Israeli independence, Zionism came under renewed suspicion, and the government began a campaign of liquidation against Zionist funds and training farms. However, emigration was not completely banned; Romanian Foreign Minister Ana Pauker, herself a Jew with a father and brother in Israel, negotiated an agreement with Israeli ambassador Reuven Rubin, a Romanian immigrant to Israel, under which the Romanian government would allow 4,000 Jews a month to emigrate to Israel; this decision was at least partially influenced by a large Jewish Agency bribe to the Romanian government. This agreement applied mainly to ruined businessmen and other economically \\\"redundant\\\" Jews. Around this time, Israel also secured another agreement with the Romanian government, under which Romania issued 100,000 exit visas for Jews and Israel supplied Romania with oil drills and pipes to aid the struggling Romanian oil industry. By December 1951, about 115,000 Romanian Jews had emigrated to Israel.\\nThroughout the period of Communist rule, Romania allowed limited numbers of Jews to emigrate to Israel, in exchange for much-needed Israeli economic aid. By 1965, Israel was funding agricultural and industrial projects throughout Romania, and in exchange, Romania allowed limited numbers of Jews to emigrate to Israel. When Nicolae Ceaușescu came to power in 1965, he initially ended the trade in deference to the Eastern bloc's Arab allies. However, by 1969, he decided to exchange Jews for cash from Israel. Ceaușescu wanted economic independence from the Soviet Union, which was content to keep Romania a backwater and as nothing more than a supplier of raw materials, but to fund economic projects, he needed hard cash. As a result, from then until the Ceaușescu regime fell in 1989, about 1,500 Jews a year were granted exit visas to Israel in exchange for a payment of cash for every Jew allowed to leave, in addition to other Israeli aid. The exact payments were determined by the age, education, profession, employment, and family status of the emigrant. Israel paid a minimum of $2,000 per head for every emigrant, and paid prices in the range of $25,000 for doctors or scientists. In addition to these payments, Israel also secured loans for Romania and paid off the interest itself, and supplied the Romanian Army with military equipment.\\nAs a result of aliyah, the Romanian-Jewish community was gradually depleted. By 1987, just 23,000 Jews were left in Romania, half of whom were over 65 years old.\\nRelations with Romanian culture.\\nRomanian Jews in Israel have strong relations with Romanian culture. Moreover, there is an intense activity among writers of Romanian language. In Israel exist 11 associations of writers in foreign languages, including the Association of Israeli Writers of Romanian Language (). Likewise, there are publications in Romanian languages, weekly, monthly or quarterly, plus several local issues.\\nRomania has an embassy in Tel Aviv, three honorary consulates in Jerusalem, Beersheba and Haifa and a cultural institute in Tel Aviv.\\nNotable people.\\nThis is a list of Israeli people of Romanian Jewish descent:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688742\",\"title\":\"Haney Corner, Indiana\",\"body\":\"\\nHaney Corner, Indiana\\n\\nHaney Corner is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office called Haney's Corner was established in 1871, and remained in operation until 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688768\",\"title\":\"Hennadiy Zubko\",\"body\":\"\\nHennadiy Zubko\\n\\nHennadiy Zubko (born 27 September 1967, Mykolaiv) is a Ukrainian politician.\\nFrom 2 December 2014 - Vice Prime Minister - Minister of Regional Development, Construction and Housing in the second Government of Arseniy Yatsenyuk.\\nEducation.\\nHe graduated from the Kyiv Polytechnic Institute, specialty \\\"Automated control systems\\\", engineer (1991); Kyiv Engineering Institute. The course \\\"Investment and financial management in the construction industry\\\" (2003); Northwestern University, Illinois, Chicago. The course \\\"Innovative Management\\\" (2006); National University of \\\"Kyiv-Mohyla Academy\\\", Kyiv Mohyla Business School, Master of Business Administration (MBA) (2007); National University of \\\"Kyiv-Mohyla Academy\\\", specialty \\\"Management\\\", manager and economist (2008).\\nCareer.\\n1986–1988 — military service in the Armed Forces of the USSR.\\n1988–1990 - software engineer at the Zhytomyr Regional department of technological connection.\\n1991–1992 — software engineer of PE \\\"Nyva\\\", Zhytomyr.\\n1992–1994 — commercial director of the Production Company \\\"SERGYI\\\", Zhytomyr.\\n1994–1997 — director of LLC \\\"Mayak\\\", Zhytomyr.\\n1997–2002 — director of \\\"Tantal\\\", Zhytomyr.\\n2001 — Project Manager of JSC \\\"LYNOS\\\" at Lysychanskyi PSI.\\n2003–2007 — Head of the Department on coordination of organizations for installation of metal structures of the State Corporation \\\"Ukrmontazhspetsstroy\\\", Kyiv.\\n2002–2010 — Deputy Chairman of JSC \\\"Zhytomyr plant of walling constructions\\\", Zhytomyr.\\nFrom 2010 until present — Chairman of the PSC \\\"Zhytomyr plant of walling constructions\\\".\\nFrom 27 May until 10 June 2014 – temporarily Head of the Administration of State Affairs.\\nFrom 10 June to 2 December 2014 - First Deputy Head of the Presidential Administration of Ukraine.\\nFrom 2 December 2014 - Vice Prime Minister - Minister of Regional Development, Construction and Housing.\\nUkraine cabinet.\\nMain is the conduction of reforms on decentralization and energy efficiency. During the year of implementing these reforms, a significant breakthrough was made, including the adopting of a number of legislative acts, introduction of new programs and their funding.\\nAlso, the Ministry is responsible for the construction reform and introduction of market relations in the housing sector to provide consumers with quality services.\\nHennadiy Zubko headed the government commission on investigating the MH17 catastrophe and the special commission on investigating causes of crash of the ship \\\"Ivolha\\\". He also took care of issues of infrastructure restoration in Svatove affected by fires at the rocket and artillery weapons warehouses. In addition, he created an expert group to assess the damage caused by an emergency situation.\\nDecentralization.\\nDecentralization reform implemented under the leadership of Hennadiy Zubko, involves changes in education, health, social policy, public administration, local authorities, development of urban infrastructure, as well as energy efficiency measures.\\nIn December 2014, the Budget and Tax Codes were amended, and fiscal decentralization took place in Ukraine. Thanks to these changes, local budgets in 2015 became surplus. Fiscal decentralization has allowed to receive additionally more than USD 24 billion to local budgets. Using the State Regional Development Fund, Government financed 784 projects of infrastructure and provided them with USD 2.7 billion.\\nOn 1 September 2015 the of Ukraine No.320-VIII «On Amendments to Certain Legislative Acts of Ukraine on Decentralization of Powers in the field of Architectural Control and Improvement of Town Planning Legislation\\\" came into force. It reforms the system of state architectural control and transmits the respective functions and powers from the national to the local level.\\nAfter adopting on 2 February 2015 the Law of Ukraine \\\"On voluntary association of local communities\\\", 6300 territorial communities initiated their association.\\n159 associated communities took part in local elections on 25 October 2015. They elected their heads and local authorities. Since 2016, they should receive direct budgetary relations with the central budget.\\nIn 2016, it is to give UAH 1.4 billion from the State Budget for subsidies to support joint local communities, meeting the long-term plans.\\nMinistry of Regional Development has developed criteria for assessing implementation of the state policy of regional state administrations: 27 indicators at 6 directions on a quarterly basis and 64 indicators at 12 directions each year.\\nThe Ministry has also an Action Plan on implementing during 2015-2017 the National Strategy of Regional Development for the period until 2020, which was approved by the Cabinet of Ministers. The document should resolve such problems as the uniformity of territorial development, elimination of inter-regional socio-economic disparities, stimulate economic activity and employment of residents of territorial communities.\\nThe Government has provided contributions to the State Fund for Regional Development in amount of UAH 2.9 billion for the support of 844 development areas investment projects.\\nThe international community assists the Regional Development Ministry in conducting decentralization. In particular, the EU EUR 90 million for implementing the decentralization reform.\\n78% of the necessary laws for decentralization have already been adopted, 22% are awaiting amendments to the Сonstitution. The future of the reform depends on approval of these amendments by the Ukrainian MP’s.\\nEnergy Efficiency.\\nOn 3 September 2014, the Cabinet of Ministers approved the Action Plan for the implementation of the European Parliament and the Council of the EU DIrective 2009/28/EU \\\"On promoting the use of energy produced from the renewable energy sources\\\".\\nOn 1 October 2014, the Cabinet of Ministers also approved the National Action Plan on Renewable Energy for the period until 2020 and Action Plan for its implementation. Implementing the plan will increase the share of energy produced from alternative fuel from 4.5% in 2013 to 11% in 2020.\\nOn 28 September 2015, the Governmental Committee approved the draft Resolution of the Cabinet of Ministers of Ukraine \\\"On National Action Plan on energy efficiency for the period until 2020\\\".\\nThe Law of Ukraine \\\"On energy efficiency of buildings\\\", which provides energy efficiency of buildings in accordance with the EU requirements, was drafted as well.\\n\\\"Urban Infrastructure\\\" project was launched for the allocation of $140 million. The project was aimed at implementing measures to improve energy efficiency, reduce water losses and settle the day-night water supply in several cities.\\nUnder the initiative of the Regional Development Ministry, the state now a part of the energy efficiency loans and will continue to do this in 2016.\\n48 programs were adopted, providing for the compensation of loans interests under the energy efficiency measures. 13 programs are of the regional character, 16 - district and 19 - city. As of 9 November 2015, since the start of the State Program on Energy Efficiency for the reimbursement of \\\"warm\\\" loans, UAH 111.7 million were transferred from the state budget. About 60 thousand families participated in the Program. Banks issued the energy efficient loans worth of UAH 950.9 million.\\nThe concept of the Energy Efficiency Fund - S2I was drafted to be launched in 2016.\\nToday the share of renewable sources in the total energy balance of Ukraine is 3.45%. The reform program envisaged by the Government will increase it to 30% by 2030.\\nConstruction Reform.\\nMinistry of Regional Development, leaded by Hennadiy Zubko, drafted a Decree \\\"On the reallocation of governmental expenditures for the Ministry of Regional Development, Construction and Housing in 2015\\\". It provides for the allocation of UAH 10.6 million to the budget program for affordable housing for the population.\\nSocial and political activities.\\n2006–2010 — deputy of the Zhytomyr City Council of V convocation.\\n2010–2012 — deputy of the Zhytomyr City Council of VI convocation.\\n2010–2012 — head the \\\"Front of Changes\\\" faction in the Zhytomyr regional council.\\n2010 - 2012 — member of the Standing Committee on Budget and Municipal Property in the Zhytomyr Regional Council.\\nFrom 2010 - member of the Presidium of Zhytomyr NGO \\\"Council of business leaders and entrepreneurs of Zhytomyr.\\\"\\nSince May 2012 – Head of the Zhytomyr regional election staff of the united opposition of \\\"Batkivschyna\\\" All-Ukrainian Union.\\nFrom 12 December 2012 – Member of the Verkhovna Rada of Ukraine elected on behalf of the All-Ukrainian Union \\\"Batkivschyna\\\" (single-mandate constituency No. 62). First Deputy Chairman of the Verkhovna Rada of Ukraine Committee on Construction, Urban Development, Housing and Regional Policy, member of the counting commission.\\nFrom 10 June until 2 December 2014 — First Deputy Head of the Presidential Administration.\\nSince 2 December 2014 — Vice Prime Minister - Minister of Regional Development, Construction and Housing of Ukraine.\\nScientific activity.\\nIn 2009 completed his PhD at the Ukrainian Research Institute \\\"Proektstalkonstruktsiya named after V. Szymanovskyi\\\", Kyiv.\\nIn 2012 defended his thesis on \\\"Organizational and technical measures for improving reconstruction of stadiums\\\" at the Kharkiv National University of Construction and Architecture at the Department of Technology and Building Structures. Author of more than 10 scientific papers. Holds PhD degree in technics.\\nAwards.\\nOrder of Danylo Halytskyi (awarded on 10 September 2009) - for the significant contribution to the socio-economic and cultural development of Zhytomyr, conscientious work and on the occasion of the 1125 anniversary of the city.\\n• Medal \\\"For Distinction in Military Service\\\", II degree (USSR, Decree No. 114, 28 August 1987)\\n• Medal \\\"70 Years of the Armed Forces of the USSR\\\" (Presidium of the Supreme Soviet of the USSR, Decree No. 84, 1988).\\nFamily.\\nWife: Lyudmila Mykolaivna.\\nChildren: Sergiy, Khrystyna.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688785\",\"title\":\"Laugheryville, Indiana\",\"body\":\"\\nLaugheryville, Indiana\\n\\nLaugheryville is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nLaugheryville was platted in 1847.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688795\",\"title\":\"New Carrollton, Indiana\",\"body\":\"\\nNew Carrollton, Indiana\\n\\nNew Carrollton is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office called New Carrollton was established in 1837, and remained in operation until 1839.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688802\",\"title\":\"Acoustocerebrography\",\"body\":\"\\nAcoustocerebrography\\n\\nAcoustocerebrography (ACG) refers to a diagnostic method in medicine that is used to diagnose pathological changes and illnesses in the brain and the central nervous system. \\nIt can also be applied as a means to diagnose and monitor intracranial pressure, for example as incorporated into continuous brain monitoring device. As a method of transcranial, acoustic spectroscopy, ACG is based on Molecular Acoustics.\\nIt allows for the non-invasive examination of the brain’s cell and molecular structure. ACG can use microphones, accelerometers or multi-frequency ultrasound (i.e. sound waves) to monitor changes. This methodology does not use any radiation and is completely free of any side effects. ACG also facilitates blood flow analysis as well as the detection of obstructions in cerebral blood flow.\\nPassive and active Acoustocerebrography.\\nPassive Acoustocerebrography.\\nAll brain tissue is influenced by blood circulating in the brain’s vascular system. With each heartbeat, blood circulates in the skull, following a recurring pattern according to the oscillation produced. This oscillation’s effect, in turn, depends on the brain’s size, form, structure and its vascular system. Thus, every heartbeat stimulates minuscule motion in the brain tissue as well as cerebrospinal fluid and therefore produces minimal changes in intracranial pressure. These changes can be monitored and measured in the skull.\\nToday, mostly passive sensors like accelerometers are used to identify these signals correctly. Sometimes highly sensitive microphones are utilized.\\nWith a digital signal, it becomes possible to study the patterns of the blood flow moving inside the skull. These patterns form unique signatures that can be analyzed with specially designed algorithms, identifying them either as an inconspicuous, “normal” pattern or as a pattern showing an “abnormal” behavior.\\nActive Acoustocerebrography.\\nIn active ACG applications, a multi-frequency ultrasonic signal is used to detect and classify adverse changes at the cellular or molecular level.\\nIn addition to all of the advantages that passive ACG provides, with active ACG it is possible to conduct a spectral analysis of the acoustic signals received. These spectrum analyses not only display changes in the brain’s vascular system, but also those in its cellular and molecular structures.\\nOne common application of active ACG is the Transcranial Doppler test. More recently, its color version (TCCD) has been deployed. These ultrasonic procedures measure blood flow velocity within the brain’s blood vessels. They are used to diagnose embolisms, stenoses and vascular constrictions, for example, in the aftermath of a subarachnoid hemorrhage.\\nFields of Application.\\nContrary to applications that provide only momentary images, such as MRI and CT, the results of ACG procedures can be obtained continuously, thus facilitating effortless and non-invasive real-time monitoring. This can be especially helpful during the acute phase directly after a stroke or a traumatic brain injury. The measured data is mathematically processed continuously and displayed on a monitoring device. The computer-aided analysis of the signals enables the physician/nursing staff to precisely interpret the results immediately after device setup. Furthermore, ACG allows for preventive detection of pathological changes in brain tissue.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688811\",\"title\":\"Alcatel One Touch Idol 3\",\"body\":\"\\nAlcatel One Touch Idol 3\\n\\nAlcatel ONE TOUCH Idol 3 4.7 and Idol 3 5.5 are Android smartphones manufactured by TCT Mobile and officially unveiled at 2015’s Mobile World Congress.\\nThe Idol 3 lineup consists of two handsets that use the same design but differ in size and specifications. The Idol 3 4.7 features a 4.7” display while Idol 3 5.5 has a bigger 5.5” display. They are the first smartphones to come with a reversible design that allows the user to use the phone upside-down.\\nSpecifications.\\nHardware.\\nBoth models feature JBL certified stereo front-facing speakers, 13 megapixel rear camera, reversible design and displays tuned by Technicolor. They are available in Soft Gold, Metallic Silver and Dark Grey and in single-sim and dual-sim versions.\\nIdol 3 4.7 features a 4.7” display with a resolution of 1280 x 720 pixels (312 ppi). It has a Qualcomm Snapdragon 410 SoC clocked at 1.2 Ghz alongside an Adreno 306 GPU and 1.5 GB of RAM. It is powered by a 2000 mAh non-removable Li-Ion battery and it is available in 8GB single-sim and 16GB dual-sim versions.\\nIdol 3 5.5 features a 5.5” display with a resolution of 1920 x 1080 pixels (401 ppi). It has a Qualcomm Snapdragon 615 SoC composed of a quad-core 1.5 GHz Cortex-A53 and a quad-core 1.0 GHz Cortex-A53 alongside an Adreno 405 GPU and 2 GB of RAM. It is powered by a 2910 mAh non-removable Li-Ion battery and it is available in 16 GB single-sim and 32 GB dual-sim versions.\\nSoftware.\\nThe Idol 3 lineup comes with Android Lollipop 5.0 with customized icons and a few added features such as the option to use the phone upside-down and FM radio.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688814\",\"title\":\"Prattsburg, Indiana\",\"body\":\"\\nPrattsburg, Indiana\\n\\nPrattsburg is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office opened at Prattsburg in 1849, and remained in operation until it was discontinued in 1857. A member of the local Pratt was postmaster, giving the community its name.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688815\",\"title\":\"Mudface (Redman album)\",\"body\":\"\\nMudface (Redman album)\\n\\nMudface is the eighth studio album by American rapper Redman. The album was released on November 13, 2015, by Gilla House Records.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688819\",\"title\":\"Blue Diamond (Iceland)\",\"body\":\"\\nBlue Diamond (Iceland)\\n\\nThe Blue Diamond is the most growing tourist route in Iceland covering about 50 km looping from Reykjavik into the Reykjanes Peninsula and back. Also straight from the International Airport into the diamond route. Tours and travel related activities in this route are rapidly growing in Iceland. The Blue Diamond route is situated in the Reykjanes Geopark area which is a member of the Geoparks Network - an area with geological heritage of international significance - advancing the protection and use of geological heritage in a sustainable way and promoting awareness of key issues facing society in the context of the dynamic planet we all live on. The peninsula, with its diversity of volcanic and geothermal activity is the only place in the world where the Mid-Atlantic Ridge is visible above sea level. The primary stops on the Blue Diamond route in the Reykjanes Geopark are Gunnuhver (largest mud geyser in Iceland), Valahnúkur, Walk inside a crater – Stamparnir, The Raven Rift (just like in Þingvellir – Almannagjá), The Bridge Between Continents, Reykjanes lighthouse, Friðrik VIII, Presidents hill, , Fire Island, Krísuvík, Seltún, Vikingworld, Kvikan - House of Culture and Natural Resources and the Blue Lagoon. Other stops include The Icelandic Museum of Rock 'n' Roll, Duushús - Culture- and art center, Sudurnes Science and Learning Center Sandgerði, Flösin Garðskaga, Stafnes church and the Svartsengi and Reykjanesvirkjun geothermal power plants.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688820\",\"title\":\"Keraudrenia hermanniifolia\",\"body\":\"\\nKeraudrenia hermanniifolia\\n\\nKeraudrenia hermanniifolia is a shrub of the family Malvaceae native to Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688825\",\"title\":\"Keraudrenia velutina\",\"body\":\"\\nKeraudrenia velutina\\n\\nKeraudrenia velutina is a shrub of the family Malvaceae native to Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688827\",\"title\":\"Thurston's Hall\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688835\",\"title\":\"Amedeo Kleva\",\"body\":\"\\nAmedeo Kleva\\n\\nAmedeo Kleva (6 February 1923 – 22 June 1996) was an Italian footballer who played as a defender. He spent his career in Bulgaria and earned two caps for the Bulgarian national team. At club level, Kleva won three A Group titles and two Bulgarian Cups with Levski Sofia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688844\",\"title\":\"Eokinorhynchus\",\"body\":\"\\nEokinorhynchus\\n\\nEokinorhynchus is a Cambrian ecdysozoan known from three Orsten-type Small Shelly Fossils. It is interpreted as a stem-group Kinorhynch.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688853\",\"title\":\"Saint Magdalen, Indiana\",\"body\":\"\\nSaint Magdalen, Indiana\\n\\nSaint Magdalen is a former town in Ripley County, in the U.S. state of Indiana. The GNIS classifies it as a populated place.\\nHistory.\\nA post office opened under the name Saint Magdalen in 1871, and remained in operation until 1905. The community was named after Mary Magdalene.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688856\",\"title\":\"Demet (magazine)\",\"body\":\"\\nDemet (magazine)\\n\\nThe Ottoman women's magazine Demet (\\\"floral bouquet\\\") was founded in 1908 in Istanbul, two weeks after the proclamation of the Second Constitutional Era. Altogether, seven issues exist, they were published once a week. Editor-in-chief and publisher was Celāl Sāhir (1883–1935). Even though the magazine was aimed at women, the editorial team of the first two issues was made up exclusively of men, such as Mehmet Akif Ersoy (1873–1936), Selim Sırrı (Tarcan) (1874–1957), and Enis Avni (1886-1958). Among the female writers were later Halide Salih Hanımlar (Halide Edip, 1884–1964), Nigar Bint-i Osman (1862–1918), and İsmet Hakkı Hanım. In addition to literary and scientific articles, what interested the female readers most were political publications. Besides Kadınlar Dünyası (1913–1921), Maḥāsin (1908–1910), and Kadın (1908–1910), is considered one of the first and most important women's magazines in the Second Constitutional Era.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688870\",\"title\":\"La Esperanza Airport\",\"body\":\"\\nLa Esperanza Airport\\n\\nLa Esperanza Airport may refer to following airports in Latin America:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688877\",\"title\":\"Stumpke Corner, Indiana\",\"body\":\"\\nStumpke Corner, Indiana\\n\\nStumpke Corner is an unincorporated community in Ripley County, in the U.S. state of Indiana.\\nHistory.\\nA post office was established at Stumpke Corner in 1878, and remained in operation until 1895. Henry Stumke served as postmaster, and gave the community his name.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688883\",\"title\":\"Anne Milano Appel\",\"body\":\"\\nAnne Milano Appel\\n\\nAnne Milano Appel is an American translator of Italian literature. She obtained a doctorate in Romance languages from Rutgers University in 1970. She has translated, among others, works by Claudio Magris, Paolo Giordano, Giovanni Arpino and Goliarda Sapienza. She was awarded the John Florio Prize in 2012 for her translation of Arpino's \\\"Scent of a Woman\\\". She is also working on English translations of Giordano’s \\\"Like Family\\\" (December 2015, Pamela Dorman Books/Viking), \\\"Syrian Dust\\\" by Francesca Borri (March 2016, Seven Stories Press) and \\\"Don't Tell Me You're Afraid\\\" by Giuseppe Catozzella (August 2016, Penguin Press).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688929\",\"title\":\"R. Maslyn Williams\",\"body\":\"\\nR. Maslyn Williams\\n\\nRobert Ronald Maslyn Williams (20 February 1911 – 11 August 1999) was an Australian documentary filmmaker and writer.\\nHe was born to England and moved to Australia in the 1920s, where he grew up in the New England and Southern Highlands districts of New South Wales. He studied at the Conservatorium of Music in Sydney and worked as a journalist before going into filmmaking.\\nIn 1940, he joined the Official War Film and Photographic Unit as a writer-producer and served under Frank Hurley in the Middle East. He worked for the Australian Information Bureau in New York in 1945 and the Canadian Film Board in 1946.\\nIn 1962 he left filmmaking and became a writer.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688939\",\"title\":\"William Frederick Windham\",\"body\":\"\\nWilliam Frederick Windham\\n\\nWilliam Frederick Windham (1840–1866) was the son of William Howe Windham and the heir to Felbrigg Hall in Norfolk. He was the subject of a famous lunacy case after he married a woman of whom his uncle did not approve.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688944\",\"title\":\"New Guinea Patrol\",\"body\":\"\\nNew Guinea Patrol\\n\\nNew Guinea Patrol is a 1958 Australian documentary film produced by R. Maslyn Williams.\\nThe film is regarded as a classic.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688947\",\"title\":\"Truth and Advertising\",\"body\":\"\\nTruth and Advertising\\n\\n\\\"Truth and Advertising\\\" is the ninth episode of the nineteenth season and the 266th overall episode of the animated television series \\\"South Park\\\", written and directed by series co-creator Trey Parker. It is the second part of a three-episode story arc which began with the previous episode \\\"Sponsored Content\\\" and concludes with the following episode \\\"PC Principal Final Justice\\\" that together serve as the season finale. It premiered on Comedy Central on December 2, 2015. Like the previous episode, this episode continues to parody the abundance of online advertising while continuing its season-long lampoon of political correctness.\\nPlot.\\nAfter Kyle is called to the principal's office, he finds Mr. Mackey there instead who tells him that PC Principal has lost his mind, with him and his fellow fraternity members going on a hunger strike and that Jimmy and Leslie have both gone missing. Meanwhile, Jimmy explains about how ads have evolved over time, becoming smarter and taking human form, in which Leslie is one of them.\\nRandy and his family are eating dinner when Randy proposes that the family move out of South Park because it \\\"sucks\\\" now. After dinner, Randy explains to Sharon that ever since South Park's revitalization, it has become too expensive to live in and he has taken out a second mortgage on the house to help pay for the expenses. Back at the news base, Jimmy is talking to the newsmen about how he has become attracted to Leslie, despite her being an ad. The newsmen tell him that an ad's purpose is to entice and manipulate people. Jimmy suggests posting the story to the school's newspaper, but is rebuffed as the newsmen tell him to just figure out what the ads are planning.\\nAt South Park Elementary, Stan runs up to Kyle with a school newspaper clipping stating that Principal has sent Jimmy and Leslie on a Disney Cruise for good behavior, but the boys know better as Jimmy was in charge of the school newspaper before being replaced by Nathan. They go to the school newsroom to get answers but are left with nothing after Nathan starts acting stupid to throw them off-course as he directly communicates with the ads on his computer.\\nMr. Garrison, Caitlyn Jenner and Principal Victoria arrive at the Whole Foods Market with disguises on and Mr. Garrison is shocked about the state of his town. Back at the news base, Jimmy's attempts to interrogate Leslie fail, leaving him to start accepting her, with the head newsman convinced that he is now \\\"thinking with his dick\\\". At Cartman's house, Stan, Kyle, Cartman, Kenny and Butters search through news articles trying to solve the case, but end up at various stores after getting distracted by ads. This leads to some tension between Kyle and Stan who strongly believe the other is causing the distraction on purpose to avoid finding out the truth. Randy is ambushed by Garrison, Jenner and Victoria, who then knock him out after they discover a \\\"PC\\\" tattoo on his rump. Back at the base, Leslie tells Jimmy that Principal is the enemy and then places an ad on the newsmen's monitor so they can not see them. While isolated from the newsmen, Leslie asks Jimmy to help her escape.\\nAt a motel, Randy wakes up tied to a chair being interrogated by the trio, who explain that the political correctness movement is not only happening in the United States, but internationally as well with PC Principal being the cause of it all, and Randy vows to kill him if this is true. Jimmy and Leslie attempt to escape, but are caught by the newsmen with the leader asking Officer Barbrady to kill them both, but Barbrady, not wanting to kill or injure anyone else, aids Jimmy and Leslie in their escape despite accidentally shooting a newsman in the shoulder. At the school, mutual suspicion between Kyle and Stan leads to the two to start arguing and then turning into a physical confrontation.\\nAt the school newsroom, Jimmy and Leslie confront Nathan about his lies, but Leslie betrays Jimmy and brutally assaults him, revealing the fact that Nathan is one of the agents of Leslie and the other ads and proving the newsmen were correct in their warnings to Jimmy not to trust Leslie. At the fraternity house, Randy, Garrison, Jenner and Victoria discover the place abandoned before discovering a computer that Principal used to see the ad featuring him and Leslie. They come to the conclusion that Principal may be trying to help, but they get distracted after looking through the computer and end up at a store just as the boys did earlier. Leslie goes to Kyle's house to ask for his help and last four digits of his Social Security number, and they leave together.\\nCritical reception.\\nIGN's Max Nicholson gave the episode a 7.2 out of 10, and stated the episode \\\"was a little sluggish, especially in terms of the main plot. While Jimmy and Leslie's storyline made some headway on that front, the boys came up empty-handed in their investigation -- although their sudden transitions to consumerist hotspots were really funny.\\\" Writing for The A. V. Club, Dan Caffrey rated the episode a B, and stated in his review \\\"it's a chapter of the increasingly epic season 19 that relies almost entirely on cryptic intentions, shifting allegiances, and general confusion among the characters as they try to unravel who’s really behind the recent madness around town.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688956\",\"title\":\"Isopogon uncinatus\",\"body\":\"\\nIsopogon uncinatus\\n\\nIsopogon uncinatus is a small shrub of the family Proteaceae that is endemic to the south coast of Western Australia around Albany.\\nTaxonomy.\\nThe species was first formally described by botanist Robert Brown in 1830, based on material collected by William Baxter at King George's Sound. In 1891, German botanist Otto Kuntze published \\\"Revisio generum plantarum\\\", his response to what he perceived as a lack of method in existing nomenclatural practice. Because \\\"Isopogon\\\" was based on \\\"Isopogon anemonifolius\\\", and that species had already been placed by Richard Salisbury in the segregate genus \\\"Atylus\\\" in 1807, Kuntze revived the latter genus on the grounds of priority, and made the new combination \\\"Atylus uncinatus\\\" for this species. However, Kuntze's revisionary program was not accepted by the majority of botanists. Ultimately, the genus \\\"Isopogon\\\" was nomenclaturally conserved over \\\"Atylus\\\" by the International Botanical Congress of 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688966\",\"title\":\"Isopogon teretifolius\",\"body\":\"\\nIsopogon teretifolius\\n\\nIsopogon teretifolius, commonly known as the nodding coneflower, is a small shrub of the family Proteaceae that is endemic to the southwest of Western Australia.\\nTaxonomy.\\n\\\"Isopogon teretifolius\\\" was first described by Robert Brown in 1810. In 1891, German botanist Otto Kuntze published \\\"Revisio generum plantarum\\\", his response to what he perceived as a lack of method in existing nomenclatural practice. Because \\\"Isopogon\\\" was based on \\\"Isopogon anemonifolius\\\", and that species had already been placed by Richard Salisbury in the segregate genus \\\"Atylus\\\" in 1807, Kuntze revived the latter genus on the grounds of priority, and made the new combination \\\"Atylus teretifolius\\\" for this species. However, Kuntze's revisionary program was not accepted by the majority of botanists. Ultimately, the genus \\\"Isopogon\\\" was nomenclaturally conserved over \\\"Atylus\\\" by the International Botanical Congress of 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688968\",\"title\":\"Breedenbroek\",\"body\":\"\\nBreedenbroek\\n\\nBreedenbroek is a small Dutch village in the Achterhoek region in the Gelderland province, near the town Dinxperlo, Netherlands. It is located some 3 kilometres from the German border. Since the municipal rearrangement in 2010, Breedenbroek is a part of the municipality Oude IJsselstreek.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688971\",\"title\":\"Sharmili Ahmed\",\"body\":\"\\nSharmili Ahmed\\n\\nSharmili Ahmed (born May 8, 1947 as Majeda Mullick) is a Bangladeshi television and film actress. She started her acting career in 1968.\\nEarly life.\\nAhmed was born in Belur Chok Village, Murshidabad. She passed matriculation examination from Rajshahi PN Girls High School.\\nCareer.\\nAhmed worked in \\\"Dompoti\\\", the first drama serial in the history of Bangladesh Television. She acted in a mother role for the first time in the drama \\\"Agun\\\", directed by Mohammad Mohsin in 1976.\\nPersonal life.\\nAhmed has a daughter Tanima. She has a younger sister theatre actor and activist Wahida Mollick Jolly.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688978\",\"title\":\"Isopogon spathulatus\",\"body\":\"\\nIsopogon spathulatus\\n\\nIsopogon spathulatus is a small shrub of the family Proteaceae that is endemic to the southwest of Western Australia.\\nTaxonomy.\\nThe species was first formally described by botanist Robert Brown in 1830, based on material collected by William Baxter at King George's Sound. In 1891, German botanist Otto Kuntze published \\\"Revisio generum plantarum\\\", his response to what he perceived as a lack of method in existing nomenclatural practice. Because \\\"Isopogon\\\" was based on \\\"Isopogon anemonifolius\\\", and that species had already been placed by Richard Salisbury in the segregate genus \\\"Atylus\\\" in 1807, Kuntze revived the latter genus on the grounds of priority, and made the new combination \\\"Atylus spathulatus\\\" for this species. However, Kuntze's revisionary program was not accepted by the majority of botanists. Ultimately, the genus \\\"Isopogon\\\" was nomenclaturally conserved over \\\"Atylus\\\" by the International Botanical Congress of 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688989\",\"title\":\"Paeroa–Pokeno Line\",\"body\":\"\\nPaeroa–Pokeno Line\\n\\nThe proposed Paeroa-Pokeno railway line or deviation in the North Island of New Zealand between Paeroa and Pokeno was a favorite scheme in the 1920s and 1930s. When work started in 1938, it was said that the proposed line, which had been surveyed 18 years earlier, would shorten the distance from Auckland to towns on the ECMT by nearly . But it \\\"never quite got off the ground\\\", although some 13 km of formation was carried out from 1938 after Bob Semple turned the first sod on 27 January. Very little is now visible.\\nThe Kaimai Tunnel relegated this section to ghost status; in August 1962 a deviation from Wahora to Apata passing under the Kaimai Range in a long (8.85 km) tunnel was approved . Work on the tunnel did not commence until 1969. With the opening of the tunnel in 1978, the Paeroa - Katikati section of the East Coast Main Trunk was closed\\nOriginally the line was to be the first part of the East Coast Main Trunk Railway crossing the Bay of Plenty to Opotiki and then inland to Gisborne via the Moutohora Branch.\\nIn 2014 the New Zealand First political party included a proposal to investigate a Pokeno-Paeroa-Te Aroha-Kaimai tunnel line as part of its transport policy. The proposal consists of completing the uncompleted Pokeno-Paeroa line, re-using part of the former Thames Branch between Paeroa and Te Aroha and a new link between Te Aroha and the western portal of the Kaimai tunnel, altogether creating a more direct link along a faster route, providing more capacity on the very busy rail freight corridor between Auckland and Tauranga, together with linking the towns of Maramarua, Ngatea, Paeroa and Te Aroha as potential future satellite suburbs of Auckland on a new commuter rail service route between Auckland and Tauranga.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48688992\",\"title\":\"Moorim School\",\"body\":\"\\nMoorim School\\n\\nMoorim School () is a 2016 South Korean television series starring Lee Hyun-woo, Lee Hong-bin, Seo Ye-ji and Jung Yoo-jin. It started airing from January 11, 2016 on KBS2 every Monday and Tuesday at 21:55 KST. The drama is set around the mysterious Moorim Institute which teaches its students virtues including honesty, faith, sacrifice and communication. The teachers and students at the school come from different countries and each have their own stories.\\nPlot.\\nMoorim Institute isn't focused solely on high academic scores. The school teaches its students virtues including honesty, faith, sacrifice and communication. The teachers and students at the school come from different countries and each have their own stories.\\nYoon Shi-woo (Lee Hyun-woo) is a famous idol, but as he is suffering from his hearing loss, his CEO tries to get rid of him by making him get involved in a scandal with a female rookie idol. As the public starts to hate him, he comes to Moorim School and hopes that he would be cured. \\nWang Chi-ang (Lee Hong-bin) is the son of a rich Chinese man, but he is an illegitimate son who was born from a Korean mother. He meets Shim Soon-duk (Seo Ye-ji) and decide to go to Moorim School with her.\\nSoundtrack.\\nPart. 1.\\nMoorim School’s main theme song “Alive” by South Korean boy band VIXX along with Part One was released on January 18, 2016. \\nPart. 2.\\nMoorim School OST Part. 2 was released on February 1, 2016. It features the title song “The King“ by VIXX. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689000\",\"title\":\"Isopogon polycephalus\",\"body\":\"\\nIsopogon polycephalus\\n\\nIsopogon polycephalus, commonly known as the clustered coneflower, is a small shrub of the family Proteaceae that is endemic to the southern Western Australia.\\nTaxonomy.\\nIt was first described by Robert Brown in 1810. In 1891, German botanist Otto Kuntze published \\\"Revisio generum plantarum\\\", his response to what he perceived as a lack of method in existing nomenclatural practice. Because \\\"Isopogon\\\" was based on \\\"Isopogon anemonifolius\\\", and that species had already been placed by Richard Salisbury in the segregate genus \\\"Atylus\\\" in 1807, Kuntze revived the latter genus on the grounds of priority, and made the new combination \\\"Atylus polycephalus\\\" for this species. However, Kuntze's revisionary program was not accepted by the majority of botanists. Ultimately, the genus \\\"Isopogon\\\" was nomenclaturally conserved over \\\"Atylus\\\" by the International Botanical Congress of 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689002\",\"title\":\"Francesca Borri\",\"body\":\"\\nFrancesca Borri\\n\\nFrancesca Borri (born 1980) is an Italian journalist and writer. \\nLife.\\nShe studied in Florence and Pisa and has worked in the Balkans and the Middle East. Her first book \\\"Non aprire mai\\\" (2008) was a study of the conflict in Kosovo. \\nIn 2010, she published a book on the Israel-Palestine conflict titled \\\"Qualcuno con cui parlare. Israeliani e Palestinesi\\\". \\nIn 2012, she began reporting from Syria. Her latest work \\\"La guerra dentro\\\", a work of reportage on the Syrian civil war, has received widespread acclaim. An English translation by Anne Milano Appel is due to be published in 2016, under the title \\\"Syrian Dust\\\".\\nBorri writes regularly for \\\"Il Fatto Quotidiano\\\", \\\"Internazionale,\\\" and Al-Monitor.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689009\",\"title\":\"Varsselder\",\"body\":\"\\nVarsselder\\n\\nVarsselder is a Dutch village with nearly 700 inhabitants, situated in the Gelderlandish Achterhoek, in the municipality Oude IJsselstreek. It lies on the road from Ulft to 's-Heerenberg, directly on the west of Ulft.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689016\",\"title\":\"Isopogon drummondii\",\"body\":\"\\nIsopogon drummondii\\n\\nIsopogon drummondii is a small shrub of the family Proteaceae that is endemic to the southwest of Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689017\",\"title\":\"James Arthur Bayton\",\"body\":\"\\nJames Arthur Bayton\\n\\nJames Arthur Bayton (April 5, 1912 – August, 1990) was an American psychologist. He conducted research in areas of personality, race, social issues, and consumer psychology.\\nEarly life and education.\\nJames Arthur Bayton was born on April 5, 1912 in Whitestone, Virginia to George and Helen Bayton. His father, a physician, had graduated from the medical school at Howard University. Bayton graduated from Temple University’s high school in 1931 and subsequently matriculated at Howard University as a Chemistry major. Bayton began his undergraduate career planning to go into medicine, however, taking psychology courses taught by Francis Sumner, Max Meenes, and Frederick Watts sparked Bayton's interest in the behavioral sciences. During his undergraduate career, Bayton was a member of the Omega Psi Phi fraternity. Bayton graduated from Howard in 1935 with a B.A in psychology and began his M.S. studies at Howard. After graduating from the M.S. program, Bayton began further graduate study at Colombia University, where he was taught by R. S. Woodworth and A. T. Poffenberger. When Bayton’s father died, Bayton moved back to Philadelphia in order to be closer to his family. He transferred to the University of Pennsylvania and continued his graduate studies under psychologists such as L. Witmer, S. W. Fernberger, M. G. Preston, and M. Viteles. While conducting his doctorate studies, Bayton was offered a teaching position at Virginia State College. Partly due to the financial devastation of the Great Depression, Bayton delayed his graduate studies and became an associate professor of psychology from 1939 to 1943. During this time he published several papers, and was finally awarded his Ph.D. in psychology in 1943.\\nCareer.\\nFrom 1943-1945, during World War II, Bayton worked as a social service analyst for the U.S Department of Agriculture. Afterward, he became a professor of psychology at Southern University in Louisiana. He taught at Morgan State College in Maryland in 1946 and returned to Howard University as a professor in 1947, where he worked for the remainder of his life. He was the head of the Psychology Department from 1966–1970, as well as a graduate research professor from 1982-1988. From 1948-1953, Bayton also worked part-time in the U.S Department of Agriculture conducting research on consumer behavior. While working at the USDA, Bayton headed a psychological research program focused on policy development and program evaluation, as well as survey programs.\\nWhile working as a professor at Howard, Bayton also served as the vice president of National Analysts, Inc. from 1953–1962 and 1966–1967, the vice president of Universal Marketing Research, Inc. from 1962–1966, a senior fellow at the Brookings Institution from 1967–1968, and a staff psychologist at Chilton Research Services from 1968-1976. Working in these positions, he contributed to over 70 corporate sponsored projects. He conducted survey and marking research and focused his work on consumer psychology. He conducted research for Dupont, IMB, Armstrong Cork, Chrysler, Eli Lilly, Curtis Publishing, Johnson and Johnson, Schick, Pet Milk, American Dairy Association, Federal Reserve Board, Smith Kline, Rench, Proctor and Gamble, and the Office of Navel Research. He also was a member of the Research Advisory Committee, Social Security Administration, the United States Department of Health Education and Welfare, and the Advisory Committee on Agricultural Science in theDepartment of Agriculture. He was also the chairman of a committee designed to evaluate equal employment opportunity policies in the National Aeronautics and Space Administration.\\nBayton was an expert witness for the National Association for the Advancement of Colored People (NAACP) in several cases of school desegregation and job discrimination. He also served as an expert witness for the NAACP Legal Defense Fund. He played an important role in cases of the desegregation of Arlington and Roanoke schools after the “massive resistance” to desegregation headed by U.S Senator Harry F. Byrd, Sr. He also led desegregation sessions for government agriculture extension systems. Continuing in this line of work, Bayton assessed urban police complaint boards, summer youth programs, community relations, and civil rights commissions in relational to desegregation policies. Throughout his career, Bayton directed over 50 government-sponsored projects. Bayton was a fellow of the American Psychological Association, the American Marketing Association, and the National Academy of Public Administration.\\nSelected publications.\\nBayton addressed overlap in issues of race and class in his 1956 study, “Race-Class Stereotypes.” 92 White and 180 Black college students were asked to choose five adjectives, from a list of 85 words, that described \\\"upper-class white Americans\\\", \\\"upper-class Negroes\\\", \\\"lower class white Americans\\\", and \\\"lower-class Negroes\\\" respectively. Generally, the upper-class was characterized as “intelligent, ambitious, industrious, neat, and progressive”, while the lower-class was considered to be “ignorant, lazy, loud, and physically dirty.” The only race difference noted was that White subjects characterized Blacks as musical and ostentatious regardless of class. Overall, the assignment of stereotypes varied more due to class than race. In other words, there were more differences between upper-class and lower class stereotypes than stereotypes between different races of the same class. Bayton suggested race differences in stereotyping tasks were partially the product of assumed class differences.\\nIn another study conducted by Bayton regarding race and class, 80 Black and 74 White college students were asked to assign traits (chosen from a list of 80 adjectives) to describe the Black lower class, the Black middle class, the White lower class, and the White middle class. The students were asked to choose five or fewer traits for each group and rate the traits on a scale from -5 to +5. Both Whites and Blacks assigned more advantageous traits to the middle as opposed to the lower class (though the effect was larger for White subjects).\\nIn his study, “Negro perception of Negro and white personality traits,” Bayton had the Guilford-Zimmermann Temperament Survey administered to 240 Black students at Howard University. The subjects were prompted to answer in the way that they thought the \\\"average Negro male\\\", \\\"average Negro female\\\", \\\"average White male\\\", or \\\"average White female\\\" would respond. The survey addressed aspects of temperament such as emotional stability, thoughtfulness, and sociability. The results indicated better personality adjustment when the participants were asked to answer as if they were White. Bayton theorized that this tendency to associate positive temperament with Whites may have resulted from an inclination to “idealize the aggressor” and “incorporate his negative views” towards views of the minority group.\\nBayton was prominent amongst the psychologists of his time in his efforts to advance minority group participation in professional psychology. In his article, “Minority groups and careers in psychology” he reported on the National Institute of Mental Health conference in 1969. The major topic discussed at this conference was the issue of how to produce more Black and minority Ph.D.s in psychology. Bayton addressed the need for an increased number of minority psychologists in the field in order to develop psychological programs focused on the needs of minority groups. He also referenced the importance of equal opportunity in the field. He reviewed the obstacles to increasing the number of minority psychologists in the field, such as the perception amongst students that psychology is a risky or nontraditional route for a minority member to pursue professionally. He claimed that these students had to be shown that possibilities exist in psychology for minority students, perhaps through the use of brochures or films to be circulated at various institutions. While graduate programs were attempting to recruit Black students in psychology, Bayton suggested that undergraduate psychology programs needed to increase their efforts to attract Black students to the field. He also addressed the issue of quality of education and lack of resources at primarily black institutions. Bayton suggested an attempt to gain federal and private funding for black students at predominately black colleges in order to ensure competent faculty and adequate resources. He proposed requesting funding to create summer programs for undergraduates that could aid in exposure to psychology for minority students at schools lacking psychology programs. He also emphasized the need to continue holding conferences of this nature in the future.\\nIn another paper, “Reflections and Suggestions for Further Study Concerning the Higher Education of Negroes,” Bayton reported on another conference that took place in April, 1967. He addressed similar issues regarding how to improve the state of affairs for Black students in higher education.\\nThroughout his career, Bayton conducted significant research in the area of consumer behavior. His paper, “Motivation, Cognition, Learning—Basic Factors in Consumer Behavior,” drew attention to the role of psychological theories as lenses for research in consumer behavior. While he acknowledged that marketing at the time addressed psychological theories of motivation, he claimed that theories of cognition and learning were neglected in consumer behavior research. Bayton explained the importance of cognitive processes in consumer behavior, for one, by explaining what determines whether or not we remember a particular product. In addressing theories of learning, Bayton outlined the role of reinforcement in determining whether a consumer will purchase a good repeatedly and explained the formation of consumer habits as a lessening of conscious decision-making while making a purchase.\\nOther research contributions made by Bayton pertained to sex differences in decision making, issues of race in military settings, Blacks’ decision making in dialysis, Black attitudes regarding kidney transplantation, and Blacks’ blood donation and organ and tissue transplantation.\\nLegacy.\\nBayton’s research interests were widespread. He furthered psychological research in areas of personality, race, social issues, and consumer psychology. His research was generally of an applied nature, and thus, his efforts helped increase the scope and depth of applied work in the field of psychology. In particular, several studies conducted by Bayton foreshadowed the emergence of system-justification theory, which addresses the tendency to support the status-quo or the “system,” even when the “system” may not be beneficial to an individual or group. In particular, the idea of out-group favoritism (a subset of system-justification theory), or viewing a high-status group positively and one’s own low-status group more negatively, emerges in Bayton’s research. In his study, “Negro perception of Negro and white personality traits,” he found that Black participants perceived Whites as having more positive temperaments than Blacks. He theorized that this bias resulted from a tendency to “idealize the aggressor” and “incorporate his negative views” into participants’ views of their minority group. In other words, out-group favoritism emerged in this study, and the participants appeared to support a racial hierarchy, or “system,” that was not beneficial to these subjects. Additionally, in another of Bayton’s studies, “Evaluative Race-Class Stereotypes by Race and Perceived Class of Subjects,” subjects assigned more advantageous traits to the middle as opposed to the lower class regardless of their own class. This study also illustrates a tendency for people to justify the \\\"system\\\", or in other words, support the American ideal of a social meritocracy whether or not this \\\"system\\\" is truly just or beneficial to them.\\nAdditionally, Bayton made important efforts throughout his career to increase the number of minorities in psychology and improve educational opportunities for Black people.\\nAccording to Sherman Ross and Leslie H. Hicks of Howard University, Bayton was “always responsive to students and colleagues” and was “never too busy or uninterested.” They also described him as a “model professor and researcher.” \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689029\",\"title\":\"Grevillea depauperata\",\"body\":\"\\nGrevillea depauperata\\n\\nGrevillea depauperata is a shrub of the family Proteaceae that is endemic to the southwest of Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689041\",\"title\":\"Grevillea fasciculata\",\"body\":\"\\nGrevillea fasciculata\\n\\nGrevillea fasciculata is a shrub of the family Proteaceae that is endemic to the southwest of Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689050\",\"title\":\"Grevillea mimosoides\",\"body\":\"\\nGrevillea mimosoides\\n\\nGrevillea mimosoides is a shrub of the family Proteaceae that is endemic to the north of Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689055\",\"title\":\"Diamond Records (Hong Kong)\",\"body\":\"\\nDiamond Records (Hong Kong)\\n\\nDiamond Records was an important record label for Hong Kong and the far East. It featured a good amount of Hong Kong's most popular recording acts and attractions in it's catalogue.\\nBackground.\\nDiamond Records was an independent label which was established in or around 1960 by Portuguese merchant, Ren da Silva. In the early 1960s, they produced a run of albums that featured original Mandarin and English compositions. Also English songs were sung in Mandarin and Mandarin songs were sung in English. Two recording artists that were an example of this were Kong Ling and Mona Fong. Both Ling and Fong had Hong Kong's leading pianist Celso Carrillo as arranger on their early Diamond albums.\\nAffiliations.\\nIn the early 1960s, Diamond was the only record pressing in the area at the time. It handled Mercury Record Corporation product on a licensing agreement basis. Taking a trip to the U.S., managing director da Silva visited the Chicago Mercury headquarters in August 1961 to discuss expansion in the area for manufacturing plant and distribution facilities. He also was in the U.S. to meet with Willem Langenberg the head of Philips group to discuss the coordination of a three way arrangement that would involve Mercury, Philips and Diamond.\\nArtists.\\nRock bands.\\nIn the mid-1960s the label started taking on guitar driven bands such as Philip Chan and the Astronotes, Danny Diaz and the Checkmates, Teddy Robin and the Playboys, and The Anders Nelsson group. Later on, other bands such as the Sam Hui fronted Lotus, The Mystics, The Zoundcrackers, D’Topnotes, The Downbeats, Joe Jr. and the Side Effects, Mod East, Sons of Han, and The Menace had recordings issued on the label. \\nArrangers and session musicians.\\nCelso Carrillo.\\nCelso Carrillo, a pianist from the Philippines was born on January 9, 1924. Along with Tony Carpio, Andy Hidalgo, and Nick Domingo, he was one of the many musicians from the Philippines that had made careers in Hong Kong. He had worked as an arranger, backing musician on various recordings by artists on the Diamond label. For a period of time, he was Hong Kong's leading pianist His band backed Kong Ling on her \\\"Hong Kong Presents ... Off-Beat Cha Cha\\\" album. In addition to playing piano and handling the arrangements, he also played conga and cow bell on various tracks. Another album for Diamond which he worked on, which was in the same vein as Kong Ling's was \\\"Mona Fong Meets Carding Cruz\\\". The backing was by the Carding Cruz band and the arrangements were by Celso Carrillo. Philippino singer Carmen Soriano, credits him with him with giving her the inspiration and training to sing when she was lacking in confidence and repertoire while in Hong Kong. At the time he was the band leader at the club and told her to give it a try just for the novelty of it. The result was her singing for a few months at the Winner House club in Hong Kong. Another singer that Carillo backed was Lita Mendoza. He died on 26th December 1988.\\nAt one time he was also once president of the Philippine Musicians League.\\nLater years.\\nIn 1970, the label was acquired by Polydor.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689065\",\"title\":\"Grevillea anethifolia\",\"body\":\"\\nGrevillea anethifolia\\n\\nGrevillea anethifolia is a shrub of the family Proteaceae that is endemic to Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689078\",\"title\":\"1972–73 Dumbarton F.C. season\",\"body\":\"\\n1972–73 Dumbarton F.C. season\\n\\nSeason 1972–73 was the 96th football season in which Dumbarton competed at a Scottish national level, entering the Scottish Football League for the 66th time, the Scottish Cup for the 78th time and the Scottish League Cup for the 26th time.\\nOverview.\\nThe first season back in the top flight of Scottish football was always going to be a challenge, and the only real measure of success would be survival - and this is exactly what Dumbarton achieved. However it took until the final league game of the season to guarantee first division football next season. Lying in 17th place, Dumbarton had to better any result that Kilmarnock managed to leap-frog them out of the relegation places, and Dumbarton did this with some style, beating Dundee United 4-1, while Kilmarnock could only manage a 2-2 draw with Falkirk. \\nIn the Scottish Cup, Dumbarton lost out to Partick Thistle in the fourth round, after a draw.\\nIn the League Cup, two wins and two draws from the six sectional games were enough to qualify for the next stage in the competition, but it was a bitter disappointment to lose to Dundee in the two-legged second round tie - as a 3-0 lead from the home fixture was followed by a 4-0 defeat at Dens Park.\\nFor the second season running Dumbarton had qualified for the Drybrough Cup. Again Celtic were the opponents in the first round and after a tough struggle they were to lose by only the odd goal in three.\\nLocally, in the Stirlingshire Cup, Dumbarton were to regain the trophy by thrashing East Stirling 6-0 in the final.\\nNote that just after the New Year, Jackie Stewart would resign as manager, to take up the reins at St Johnstone, and was replaced by Alex Wright.\\nResults.\\nAll results are written with Dumbarton's score first.\\nAppearances.\\nDumbarton used 26 players for the 46 League, Scottish Cup, Scottish League Cup and Drybrough Cup matches, as detailed below. Davie Wilson was the only player to feature in every one of these matches.\\nReserve Team.\\nDumbarton competed in the Scottish Reserve League, and with 9 wins and 10 draws from 34 matches, finished 12th of 18.\\nFor the first time in a decade, Dumbarton also entered the Scottish Second XI Cup, and reached the third round before losing out to Dundee by the only goal.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689086\",\"title\":\"Wildlife of the United Arab Emirates\",\"body\":\"\\nWildlife of the United Arab Emirates\\n\\nThe wildlife of the United Arab Emirates is the flora and fauna of this country on the eastern side of the Arabian Peninsula and the southern end of the Persian Gulf. The country offers a variety of habitats for wildlife including the coast, offshore islands, mangrove areas, mudflats, salt pans, sand and gravel plains, sand dunes, mountain slopes, wadis and rocky summits.\\nBecause the terrain is so varied, it supports a greater number of species of plants and animals than might have been expected in this relatively small country.\\nGeography.\\nThe United Arab Emirates occupy a corner of Arabia bounded by Saudi Arabia to the west and south, Oman to the east, the Gulf of Oman and Oman to the northeast and the Persian Gulf to the north. The northern coast stretches for about along the southern shore of the Persian Gulf and largely consists of salt pans that extend inland. There are numerous offshore islands, the ownership of some of which is disputed with neighbouring states. To the south is the Rub' al Khali desert, an enormous expanse of billowing sand dunes. Two large oases in the east are the Liwa Oasis, near the undefined border with Saudi Arabia, and Al-Buraimi Oasis at Al Ain. To the east the land rises and becomes mountainous, this being the northern end of the Al Hajar Mountains. Beyond these, the coast on the Gulf of Oman is rugged.\\nIn the southeast of the country, near the border with Oman, there is a man-made lake, Lake Zakher, formed as a result of the release of waste water from the desalination plants on which the UAE relies for its freshwater supplies. The lake has formed as the groundwater rose, and many birds now visit the lake.\\nFlora.\\nA wide range of plants is associated with the many types of habitat in the United Arab Emirates. One of these types is sabkha, an area in which salty water has flooded the land shallowly and later evaporated, leaving crusty salt pans. These occur on the western part of the Gulf Coast but also among dunes inland. The plants found on their edges are salt-tolerant members of Salicornioideae and \\\"Zygophyllum\\\". At inland sites \\\"Zygophyllum qatarense\\\" predominates along with grasses such as \\\"Aeluropus lagopoides\\\" and \\\"Panicum turgidum\\\". Sandy plains further east along the coast from the sabkha region have occasional dwarf tamarisk trees and such plants as \\\"Salsola imbricata\\\" and \\\"Zygophyllum mandavillei\\\", and in coastal lagoons, and in creeks further east, the white mangrove is plentiful.\\nPlants of the gravel plains further east again include \\\"Cornulaca monacantha\\\", \\\"Crotalaria persica\\\", \\\"Calotropis procera\\\" and \\\"Taverniera spartea\\\", and the parasitic desert hyacinth and the desert thumb. As the land rises up towards the mountains, the mesquite tree, an invasive species from Central America, has become established. The plains around Ras al-Khaimah in the northeast of the country, between the mountains and the sea, are the most heavily cultivated part of the country. The mountains have a cooler, more temperate climate and here there is an abundance of Alpine flowers among the rocks, on slopes and in cracks, fissures and wadis.\\nFauna.\\nOver four hundred species of bird have been recorded in the United Arab Emirates, with about ninety species breeding regularly in the country while the balance are winter visitors, migrants or vagrants. The country is on the crossroads of two major migratory routes, one between the Palaearctic and Africa, the other between the Near East and the Indian subcontinent, and the migrants make use of the many types of habitat available.\\nAbout 250,000 waders visit the Gulf shores and mudflats at peak migration time; these include the grey plover, the greater and lesser sand plovers, the crab plover, the Kentish plover and the broad-billed sandpiper. The coast, and particularly offshore islands are used by many seabirds. About twenty to thirty percent of the world's Socotra cormorants, about 200,000 birds, breed in the United Arab Emirates, but they are under threat from fishermen who fear for their livelihoods. Sooty gulls breed here, as do red-billed tropicbirds as well as several species of tern; white-cheeked, bridled and lesser crested tern.\\nA large number of passerine birds breed in the deserts, salt flats, plains, dunes and mountains. Twelve species of wheatear have been recorded in the country as well as warblers, babblers, rollers, bulbuls, the desert lark and many others. The sooty falcon overwinters in Madagascar and breeds in the United Arab Emirates. Other than this, there are only a small number of raptor species; Bonelli's eagle, barbary falcon, short-toed snake eagle, long-legged buzzard and lappet-faced vulture.\\nMany of the large mammals found in the Arabian Peninsula were well-adapted to desert life in the harsh terrain, but were wiped out by human hunting in the last hundred years or so. Hunting is now banned in the United Arab Emirates, but feral goats and donkeys are plentiful and graze indiscriminately, lessening the chance for the native gazelles to recover from their reduced population sizes. Large terrestrial mammals still found in the United Arab Emirates include the Arabian tahr, the Arabian oryx, the sand gazelle and the mountain gazelle. Carnivores include the Arabian wolf, the striped hyena, the red fox, the Blanford's fox, the Rüppell's fox, the caracal, the Arabian wildcat, the sand cat and the Arabian leopard. Other mammals include the Cape hare, the Brandt's hedgehog, the desert hedgehog and the long-eared hedgehog.\\nThe Egyptian fruit bat is found here during most of the year, but moves around according to the availability of fruit. Insectivorous bats include the sac-winged bat, the mouse-tailed bat and the leaf-nosed bat. Small rodents include the lesser Egyptian jerboa, the Cheesman's gerbil and the Balochistan gerbil. Rather larger rodents are the Libyan jird and the Sundevall's jird which both favour desert habitats. The Cairo spiny mouse was found in the mountains for the first time in 1995.\\nOver 54 species of terrestrial reptile have been recorded in the UAE. These include a large number of lizards, found in all environments from desert, to city, to mountain-top, and a single species of worm lizard. There are thirteen species of terrestrial snake, some of the largest being the sand boa, the saw-scaled viper and the horned viper, and four species of sea snake as well as green sea turtles present in the Persian Gulf. There are two species of amphibian in the United Arab Emirates, the Arabian toad and the Dhofar toad; the former is more commonly seen as the Dhofar toad is nocturnal.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689087\",\"title\":\"Grevillea baxteri\",\"body\":\"\\nGrevillea baxteri\\n\\nGrevillea baxteri, commonly known as the Cape Arid grevillea, is a shrub of the family Proteaceae that is endemic to Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689089\",\"title\":\"Arca Totok Kerot\",\"body\":\"\\nArca Totok Kerot\\n\\nArca Totok Kerot or Recå Thothokkerot IPA: [rətʃɔ ʈɔʈɔʔ kərɔt] is a statue (Jav. \\\"arcå\\\" or \\\"recå\\\" means \\\"statue\\\") located in Bulusari Village, Pagu District, Kediri Regency, Indonesia; about 2 kilometers north-east of Simpang Lima Gumul. It is made of andesite stone, depicting a giant with a terrifying face. The style is of a dvarapala. Based on the style, it is suggested that it was made in the 10th century.\\nPhysical features.\\nThe face and other ornaments also suggest that it is a representation of a female \\\"butå\\\". This is unusual since most dvarapala statues show male characteristics. Arca Totok Kerot is 3 meters tall.\\nThe hair is matted, covering the upper part of the back. A kind of tiara with a glimpse of skull is seen above forehead, covering also the ears. The eyes are protruding, creating a terrifying effect, as if it were angry.\\nLike many other dvarapala statues, it kneels on one knee. The left knee touches the ground while the right knee is erected, on which the right hand is laid. The left hand is missing, so it is unknown if it originally held a \\\"gådå\\\" (mace) like other dvarapala statues. All hands and legs are wearing \\\"binggêl\\\" (bracelets).\\nThe nipples are clearly shown on the breast, another indication that it is female. It wears a necklace of skulls, usually worn by worshippers of Durga or Shiva. A girdle is encircling its waist, on which some kind of skirt is hanging.\\nDiscovery and excavation.\\nThe statue was discovered in 1981.\\nIn 2013, due to an imminent plan by the local government to enlarge the road near Arca Totok Kerot, an excavation was hastily carried out searching for other archaeological remains. The excavation was conducted by East Java Hall of Cultural Heritage Preservation (\\\"Balai Pelestarian Cagar Budaya Jawa Timur\\\"). The possibility was considered because dvarapala statues were usually placed to guard a temple complex or other important sites.\\nThe excavation took place for six days. The team dug with average 1.5 meters into the surrounding ground, in some points even 4 meters. However, no other archaeological remains were found. Eventually the excavation was terminated and the cavities were filled. The road in front of the statue was then enlarged.\\nFolklore related to the statue.\\nTales were abound surrounding the statue. Some were already documented and many others were still transmitted orally. Some tales related the statue to the legendary character Calon Arang, but such connection was refuted by Timoer (1981). Others were connected to the legendary king Jayabaya (Santosa, 2005).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689098\",\"title\":\"Grevillea cunninghamii\",\"body\":\"\\nGrevillea cunninghamii\\n\\nGrevillea cunninghamii is a shrub of the family Proteaceae that is endemic to the Kimberley region of Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689108\",\"title\":\"Grevillea dryandri\",\"body\":\"\\nGrevillea dryandri\\n\\nGrevillea dryandri is a shrub of the family Proteaceae that is endemic to Western Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689115\",\"title\":\"Ischiopsopha wallacei yorkiana\",\"body\":\"\\nIschiopsopha wallacei yorkiana\\n\\nIschiopsopha wallacei yorkiana are beetles in Australia from the family Scarabaeidae, subfamily Cetoniinae, tribe Schizorhinini. It is a sub-species of \\\"Ischiopsopha wallacei.\\\"\\nThe \\\"Cetoniinae\\\" scarabs are known as \\\"flower chafers\\\" as their main food is pollen and nectar sourced from flowers.\\nDescription.\\n\\\"Ischiopsopha wallacei\\\" can reach a length of about 30 mm. The beetles are an electric green colour, and have the tip of the scutellum visible.\\nDistribution.\\nThis species inhabits the Cape York Peninsula region of Australia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689128\",\"title\":\"Ju Zheng\",\"body\":\"\\nJu Zheng\\n\\nJu Zheng (; November 8, 1876 – November 23, 1951) was a Chinese politician who was a leader in the Chinese Nationalist Party, or KMT, in the 1930s and 1940s. As president of the Judicial Yuan, he administered China's court system from 1932 to 1948. He ran in the presidential election of 1948 as the token opponent of Chiang Kai-shek. He was also known by his courtesy name Juesheng (Chueh-sheng).\\nLife and career.\\nJu was born in Huangzhou in Hubei Province on November 8, 1876. He joined the Tongmenghui, a revolutionary party founded by Sun Yat-sen, while studying law at Nihon University in Japan in 1907. Later, he worked for Chinese-language newspapers in Rangoon and Singapore. He returned to China to work for an anti-Qing revolutionary faction in Hubei. In 1912, he was briefly vice minister of the interior in the Provisional Government in Nanjing with Sun as president. He was commander of the Woosung Forts north of Shanghai during the Second Revolution in July 1913. In 1916, he led an uprising against Yuan Shikai in Shandong and briefly captured the city of Weixian. In 1921, Sun appointed him interior minister for the Nationalist administration in Guangzhou.\\nJu was a founding member of the Western Hills Group, formed after Sun died in 1925. This group opposed communist influence in the KMT. Ju was appointed president of the Judicial Yuan by Chiang in 1932. This was one of the five branches of government in the KMT system. In the presidential election of April 20, 1948, Ju was persuaded to oppose Chiang's candidacy and received 10 percent of the vote in the National Assembly, with Chiang elected overwhelmingly. After he resigned as president of the Judicial Yuan on July 1, 1948, Ju was appointed to the Control Yuan, an auditing board. When the KMT was defeated by the Chinese Communists in 1949, Ju fled to Taiwan.\\nJu co-founded Tamkang College of English, now Tamkang University, in Taipei in 1950. He died on November 23, 1951. Ju Haoran, his son, succeeded him as president of Tamkang. November 8, Ju's birthday, is marked annually as the school's founding day. The school's Chueh Sheng Memorial Library is named in his honor.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689132\",\"title\":\"Denis Ruddy\",\"body\":\"\\nDenis Ruddy\\n\\nDenis Ruddy (born 3 April 1950) was a Scottish footballer who played for Clydebank, Dumbarton and Stenhousemuir.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689137\",\"title\":\"Prečista Krajinska\",\"body\":\"\\nPrečista Krajinska\\n\\nPrečista Krajinska () is the name for the ruins of a church located in the region of Skadarska Krajina, southern Montenegro. It was the second burial site of Jovan Vladimir, the ruler of Duklja (ca. 1000–1016), after his widow Kosara renewed it and transferred his remains from Prespa. The relics drew many devotees to the church, which became a center of pilgrimage. Kosara was interred in the church, at the feet of her husband, on her request. In around 1215—when Krajina was under the rule of Serbian Grand Prince Stefan Nemanjić—the relics were presumably removed from this church and transported to Dyrrhachium by the troops of Michael I, the despot of Epirus. At that time Despot Michael had briefly captured from Serbia the city of Skadar, which is only about east of the church. The monastery was mentioned in 1417 in a Cetinje manuscript. The Balšić family reconstructed the monastery in the 15th century. The monastery became the seat of a Catholic bishopric, as part of the Catholic-Venetian expansion. The bishop, who was employed in an Orthodox region, saw resistance in the area from the Serbian Orthodox. At the beginning of the 20th century, the population surrounding the Church ruins were Muslims.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689144\",\"title\":\"Siren of the Tropics\",\"body\":\"\\nSiren of the Tropics\\n\\nSiren of the Tropics is a 1927 French silent film starring Josephine Baker. Directed by Mario Nalpas and Henri Étiévant and set in the West Indies, the film tells the story of a native girl named Papitou (Baker) who falls in love with a French man named André Berval (Pierre Batcheff). \\nPlot.\\nThe film is set in a fictional colony called Monte Puebla. Monte Puebla incorporates many colonial stereotypes, with the name suggesting that it could be a Spanish colony, the grass skirts and roofs suggesting a Polynesian influence, and the clothing being a jumble of multiple cultures. The story begins when a rich Parisian man named Marquis Sévéro wishes to marry a woman named Denise, but Denise is in love with an engineer named André Berval. In order to get rid of Berval, Marquis Sévéro sends him to the West Indies as a prospector, promising that he can marry Denise once he returns. After arriving in the West Indies, Berval meets a woman named Papitou. Papitou quickly falls in love with him, unaware of the fact that he is planning on marrying Denise upon his return to Paris. When Berval leaves the West Indies to return home Papitou follows him, despite the fact that he has a fiancé. Once she arrives in Paris, Papitou adjusts to city life and finds her true calling as a music hall performer. \\nReception.\\nPrior to the film's release, newspaper articles detailing Baker's tour through Europe piqued public interest.Following the film's premiere in December of 1927 in Stockholm, it received almost unanimously positive reviews from film critics. The film was screened from December 1927 until July 1928, which was considered an exceptionally long running time. Most of these positive reviews focused on Baker's body, comparing her agile movements to those of animals. Following her positive reception in \\\"Siren of the Tropics\\\" and increased public interest surrounding her, Baker published an autobiography called \\\"Les mémoirs de Josephine Baker.\\\" Following the film's very successful premiere, Baker also had a doll made in her likeness and sold in Stockholm, and starred in a toothpaste commercial. This positive reception of Baker's feature film debut set the stage for her starring roles in the films \\\"Zouzou\\\" and \\\"Princesse Tam-Tam\\\". \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689148\",\"title\":\"Wa Senior High Technical School\",\"body\":\"\\nWa Senior High Technical School\\n\\nWa Senior High Technical School is a coeducational second cycle institution at Wa in the Upper West Region of Ghana. The school which was formerly called Wa Secondary Technical, houses both day (non-boarding) and boarding students at its premises and is precisely located at Konta opposite the Ghana Water Company.\\nHistory.\\nThe school was established as a community middle day school in the 1950's. Progressively, the school became a junior secondary school in 1978 and later transformed into a community secondary technical school in 1982. It was formally commissioned as a community day secondary technical school on 30th March 1983. With the enactment of the educational reforms in 2004 the school became known as Wa Senior High Technical School.\\nIn 2012 the Ghana Education Service granted the school a boarding status following the implementation of several infrastructural development by the government through the Ghana Education Trust Fund (GETFUND). Currently the school boasts of a student population of about 2200.\\nFacilities.\\nThe school houses numerous educational facilities that compliment the studies of students. The Upper West Region french center is located in the school so french students normally don't face problems with the subject.\\nIt has a world standard science laboratory, a library and many other useful facilities. Aside the school’s science laboratory, the regional science laboratory is also located on the school’s campus.\\nCourses.\\nThe school offers courses in Science, General Art, Visual Art, Technical and Home Economics.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689154\",\"title\":\"Tapped Shedding/Cam Shedding\",\"body\":\"\\nTapped Shedding/Cam Shedding\\n\\nTappet Shedding/Cam Shedding.\\nAccording to the Textile Terms and Definitions, \\\"the control of the movement of heald shafts in weaving simple constructions by means of cams or tappets\\\" is called Tappet or Cam Shedding.\\nClassification: Two types-\\n1. Positive Tappet Shedding\\n2. Negative Tappet Shedding\\n1. Positive Tappet Shedding: Positive Tappet Shedding implies that the heald shafts are both raised and lowered by the tappets.\\n2. Negative Tappet Shedding: Negative Tappet Shedding implies that the heald shafts are either raised or lowered by the mechanism but are returned by the action of an external device (usually springs).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689167\",\"title\":\"Perci Piétro\",\"body\":\"\\nPerci Piétro\\n\\nPrince Percia Piétrolungo (born June 29, 1972 in New York City, New York), better known as Perci Piétro, who is an American of Persian and Italian descent. He is a real estate entrepreneur and the founder of OWN Realty and OWN Financial, with principal offices located in Miami, Florida.\\nHe is also a direct descendent of the Qajar Dynasty, the Royal Family, which ruled Persia (now Iran) from 1785 to 1925.\\nEarly life and Education.\\nPiétro was born to Princess Shahnaz Shakoori M.D., after she moved from Iran to become a surgeon in the United States.\\nIn his younger years, Piétro became fluent in English, Farsi and Spanish with some background in French and Italian. He attended preparatory school at Rumsey Hall in Washington, Connecticut and then boarding school at Wayland Academy in Beaver Dam, Wisconsin. After he attained his Bachelor of Science Degree from Wright State University in Dayton, Ohio. Later he went on to study at Lynn University in Boca Raton, Florida where he attained a Master of Business Administration Degree (MBA) in International Management.\\nCareer.\\nAfter graduating from college, Piétro decided to open a lounge in South Beach, Miami. He co-founded W6 Lounge with two other New Yorkers, ensuring the lounge had the feel of a bar in Manhattan.\\nAt the time, it was seen as one of the few upscale lounges in Miami, with regular celebrity guests. Following the completion of W6, Piétro was featured in The Miami Hurricane in 2002, discussing how its club catered for students and young professionals. Up to 2001, W6 and Piétro were regarded as a prominent entertainment figure and location in Miami around 2001.\\nPiétro remained at the entertainment venue until 2004 when he left to start World Real Estate Exchange, which was a real estate brokerage founded in early 2005. As the company grew, it set up affiliates in a number countries, including Istanbul, Dubai, Brazil and the Bahamas. Piétro also developed a niche according to Miami Agent magazine of negotiating bulk deals with developers and then introducing groups of buyers, to assist developers with moving of their inventory.\\nWith his next venture as the founder of OWN Realty and OWN Financial, Piétro was featured in Miami Today, as a commentator on the health of the Miami property market in 2011. He spoke about the 2011 upturn in the property market in Miami, stating it was due to a large rise in the number of foreign investment in property at the time. He was quoted \\\"“For Mexicans and Venezuelans, it’s security issues,” he said. “For Canadians and the English, it’s the exchange rate; for Russians and Germans, the strength of their economies. So there’s a lot of action.”\\\"\\nAfter becoming an International real estate expert, Piétro was made a Certified International Property Specialist, becoming one of just over 1,000 to hold the qualification globally. His work in Miami and the surrounding Florida market, has lead him to be interviewed about the state of the American property market a number of times in recent years.\\nFamily.\\nPiétro is a great grandson of Mozaffar Al-Din Shah Qajar, who was the fifth Qajar king of Persia. He reigned between the years 1853 and 1907. Piétro’s maternal great grandfather Sarem Al-Saltaneh Sardar Nasser Talesh was the Vali and Hakim of Azerbaijan, Gilan, Mazandaran, Ardabil, Astara, Namin, Rasht and Talesh which made up the Northern region of Persia. After a coup in 1925, the Qajar's rule of Persia came to and end, with the new Pahlavi Dynasty re-naming the country Iran in 1935.\\nHis grandfather Jalil Shakoori continued to live in Iran, and served as a Colonel under the last Shah starting in the mid-1940s. Piétro’s mother moved to New York City in the late sixties later bringing her mother and four sisters. In 2015, with the passing of his maternal grandmother Shahzadeh Khanoum Akram Sadat Sarem Al-Saltaneh who was the matriarch, Piétro became the patriarch of the family as the only son of her eldest child.\\nDespite his family heritage, he has yet to visit Iran, where his ancestors once ruled.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689171\",\"title\":\"SAB DB-80\",\"body\":\"\\nSAB DB-80\\n\\nThe SAB DB-80 and SAB DB-81 were single engine, all-metal French light transports aimed at the air mail market whilst carrying two passengers. Identical apart from their engines, they flew in mid-1930.\\nDesign.\\nIn 1929 Dyle and Bacalan were reformed as Société Aérienne Bordelaise (SAB) who continued to work on its predecessor's designs, distinguished by their DB numbering as well as on their own, which had AB numbers. The DB-80 originated with Dyle and Bacalan but was not flown until 1930, built by SAB. It was Dyle and Bacalan's last design and a small aircraft by their standards but maintained their all-metal tradition.\\nThe DB-80 was aimed at the airmail market and was a single engine, high wing aircraft giving easy access by two port-side doors to a well lit cabin with two passenger seats and to a separate mail compartment behind them. The pilot sat ahead of the passengers under the wing leading edge. Two differently engined versions were built: the DB-80 had a Hispano-Suiza 6P six cylinder, upright water-cooled inline and the DB-81 a Lorraine 5Pc five cylinder radial engine. The latter was mounted on a hinged frame for easy servicing. The Hispano engined had a Lamblin radiator on the fuselage underside.\\nThe fuselage was built around four longerons, with rectangular frames and covered in longitudinally ribbed duralumin. Its underside was smoothly bellied, its upper side flat. In plan its taper was delayed until aft of the cabin. Its empennage was conventional, with a straight tapered, blunt topped fin and unbalanced rudder, the latter cut away at its base to allow for movement of the one-piece elevator mounted on a triangular tailplane at mid-fuselage height. Like the fuselage, all the tail surfaces were covered with ribbed duralumin.\\nThe DB-80 had a fixed, tailwheel undercarriage with track of .The mainwheels were independently mounted and fitted with brakes. Each axle was mounted at the lower vertex of a triangular box acting as a cantilever leg, with its upper side hinged from the fuselage longerons. The strengthened forward edge of the structure extended above the hinge and connected to an elastic block housed in a reinforced transverse beam which passed under the cockpit, incorporating shock absorbers.\\nThe DB-80's high, cantilever wing was unusual both in its construction and high aspect ratio of 9. In plan it was straight tapered on both edges but with semi-elliptical tips curved particularly on the trailing edges, where its ailerons were full span and broad. The wing was built around three spars, rather than the traditional one or two and the detail of their caps or flanges, rather than the shape of the longitudinal braces or ribs, determined the airfoil profile. A similar but not identical wing structure was used on the Dyle et Bacalan DB-20.\\nDevelopment.\\nIn late March 1930 the DB-80's test flights were awaiting better weather; a month later tests were underway but the first flight did not take place until 27 June 1930. The Lorraine powered DB-81 flew in August, after which testing of the pair continued successfully though interrupted by SAB test pilot Charles Deschamps' absence at Villacoublay for official trials of the DB-20. In October the DB-80 was re-engined with a Lorraine and renamed DB-81.\\nThere is no record of any further examples being built nor of measured performance figures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689191\",\"title\":\"José Luis González (composer)\",\"body\":\"\\nJosé Luis González (composer)\\n\\nJosé Luis González Capilla de Guadalupe, Jalisco 1937) is a Mexican composer. He studied in Guadalajara at the Escuela Superior Diocesana de Musica.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689201\",\"title\":\"Diyojen (magazine)\",\"body\":\"\\nDiyojen (magazine)\\n\\nDiyojen (“Diogenes“) was the first Ottoman satirical magazine of the Ottoman Empire. The first issue was published in Istanbul on 24th November 1870 by the satirist Teodor Kasap (1835-1905). It came out weekly in three year’s issues and was banned for good in 1873 after 183 numbers. Apart from satirical pieces, the magazine became known for its caricatures and the translation of French literature. Kasap, who also worked as journalist and playwright, published other satirical magazines after the ban. In Haylal (“Fantasy” or “Illusion“), which existed from 1873 until 1877, he among other things used caricatures and satirical articles to criticize the arbitrary press law. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689202\",\"title\":\"Criminal Investigation Department (disambiguation)\",\"body\":\"\\nCriminal Investigation Department (disambiguation)\\n\\nA Criminal Investigation Department or Crime Investigation Department is a branch of many police forces. It may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689212\",\"title\":\"Higinio Ruvalcaba\",\"body\":\"\\nHiginio Ruvalcaba\\n\\nHiginio Ruvalcaba (Yahualica, Jalisco, 11 January 1905 - Mexico City 15 January 1976) was a Mexican violinist and composer. He was leader of the Lener String Quartet with second violin José Smilovitz; viola Herbert Froelich, and Hungarian cellist Imre Hartman.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689256\",\"title\":\"Shehbal (magazine)\",\"body\":\"\\nShehbal (magazine)\\n\\nThe Ottoman journal Şehbal (“wing feather” or “wings“) was published in Istanbul between 1909 and 1914. Its publisher Hüseyin Saadeddin Arel (1880–1955), technically a lawyer, was a well-known musicologist and composer of 20th-century classical Turkish music. Altogether 100 numbers were issued every two weeks, at the first and fifteenth day of each Maliye-month respectively, totaling 20 pages. The page format more or less equaling the German DIN-size A3 as well as new printing techniques like the three-color printing with countless photographs and illustrations make the journal especially remarkable.\\nThe examples inspiring this journal in its design as well as its content were French magazines like \\\"Figaro Salon\\\" or \\\"L'Illustration\\\". The journal aimed at propagating spiritual as well as technical progress and at covering a wide range of topics with articles on political and social sciences as well as jurisprudence right up to pieces from the fields of natural sciences, technology and philosophy. Furthermore, female readers were informed on parenting, women’s rights and their role in society. Pages on the latest historical events were part of the journal as were articles on sports and fine arts. Pieces on music were of special importance, they were mostly accounted for by the editor himself, writing under the pseudonym Bedi Mensi. Entertainment pages featured Ottoman as well as English and American stories and novels. In addition to that, several plays and poems of known writers were printed. Numerous contests involved the readers in the production of the journal and were meant to stimulate cultural life. The winners of those contests which had submitted a composition, photographs, caricatures or translations of operas were honored in award presentation ceremonies. Also, there was a translation competition in order to translate certain French into Ottoman terms. The 100th issue, coming out on 23 July 1914/10th Temmuz 1330 as a special edition, turned out to be the last issue without prior notice. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689268\",\"title\":\"Yasmina (film)\",\"body\":\"\\nYasmina (film)\\n\\nYasmina is a 1927 French silent film directed by André Hugon and starring Camille Bert, Huguette Duflos and Léon Mathot.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689275\",\"title\":\"ZM Auckland\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689298\",\"title\":\"Katten (beach)\",\"body\":\"\\nKatten (beach)\\n\\nKatten is a municipal beach in Bunnefjorden in the Nordstrand area in Oslo, Norway. The beach is suitable for children and comprises a sheltered, sandy beach, rocks and grassy hills. There are diving boards, swiming ladders, tables and benches, toilets, showers, drinking water and a kiosk. \\nThere is very limited parking facilities, but there is a bus stop on Mosseveien right by. In the summer months there are also a separate \\\"bathing bus\\\" (busline no. 87). There is also a footpath downhill to Katten from Ljan Station.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689305\",\"title\":\"New Zealand NBL Rookie of the Year Award\",\"body\":\"\\nNew Zealand NBL Rookie of the Year Award\\n\\nThe National Basketball League Rookie of the Year is an annual National Basketball League (NBL) award given since the 1992 New Zealand NBL season to the best performing rookie New Zealander of the regular season. The award was originally given to the best Young Player of the Year from 1986 until 1991, with centre Warren Adams winning the award twice within four years. In 1992, forward Pero Cameron won the league's first Rookie of the Year award. This name stood until 2005, when a slight adjustment to the rules saw the Rookie of the Year opened up to any player under the age of 20, with guard Jarrod Kenny (age 19) winning the 2005 Young Player of the Year. This was changed back to Rookie of the Year in 2006, and has remained ever since. Future NBA player, Steven Adams, won the 2011 Rookie of the Year award; Steven is the half-brother of two-time Young Player of the Year, Warren Adams.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689306\",\"title\":\"Satriano\",\"body\":\"\\nSatriano\\n\\nSatriano may refer to the following Italian places and jurisdictions :\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689308\",\"title\":\"ZM Whangarei\",\"body\":\"\\nZM Whangarei\\n\\nZM Whangarei (also known as 93ZM is a hit music radio station in Whangarei, New Zealand. It is a station of the ZM network, and is owned and operated by New Zealand Media and Entertainment.\\n93ZM started around 1995 as a totally local station broadcasting on 93.1 MHz. 93ZM became a network station after ZM made a return to Auckland with local voice breaks prerecorded in Auckland minutes earlier, however the breakfast show on 93ZM was networked from 89.8ZM in Hamilton between 1998 and 1999. In 2005 93ZM moved from 93.1 to 93.2 MHz and in 2006 93ZM traded places on the Northland radio dial with Radio Hauraki 93ZM shifted to 95.1 MHz and Hauraki took over ZM's vacated 93.2 MHz frequency. In 2010 ZM in Whangarei moved to 94.8 MHz as part of the government move to re-align radio frequencies around New Zealand.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689324\",\"title\":\"The Temple of Shadows\",\"body\":\"\\nThe Temple of Shadows\\n\\nThe Temple of Shadows (French:La vestale du Gange) is a 1927 French silent film directed by André Hugon and starring Camille Bert, Max Tréjean and Georges Melchior.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689338\",\"title\":\"Muna Al Gurg\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689353\",\"title\":\"Baurzhan Baimukhanov\",\"body\":\"\\nBaurzhan Baimukhanov\\n\\nBaurzhan Amangeldiyevich Baimukhanov (born July 21, 1971, Karagandy region,Kazakh Soviet Socialist Republic) – PhD. in Economics, a well-known business man, Member of the Regional Business Council of the National Chamber of Entrepreneurs, Astana\\nEducation.\\nIn 1988 he entered the State National University after Al-Farabai, Faculty of History, and graduated therefrom in 1993.\\nRight after, he entered the Kaz GUA affiliated Market Institute, Economics Faculty.\\nIn 2011 he entered Moscow School of Management SKOLKOVO, Executive MBA program. Successfully defended his graduation project in 2013.\\nBiography.\\nHe was born on July 21, 1971, Karagandy region, Kazakh Soviet Socialist Republic.\\nDuring his study (1993–1995), he started his work in the Regional Foreign Economic Association \\\"Kazakhstan commerce\\\" on a position of Chief Specialist.\\nFrom May 1996 to March 1997, he performed duties of Senior Manager in the Industrial project department, Kazkommertsbank OJSC.\\nIn October 1998, he occupied a position of Director of Corporate and Transportation Service Center, a subsidiary of Republican State Enterprise \\\"Kazakhstan Temir Zholy\\\", Astana.\\nFrom November 2001 to June 2002, he occupied a position of General Director of Republican State Enterprise \\\"Kazakhstan Temir Zholy\\\" and thus became the youngest 30-year-old manager in the history of this company with 125 000 employees and $1.7 bn of annual returns.\\nDuring his period of work in Republican State Enterprise \\\"Kazakhstan Temir Zholy\\\", he was one of those to draft the Law on railway transport. Adoption of this Law increased competitiveness of Kazakhstan railway transport. His main conclusions were addressed in his PhD thesis defended in 2001.\\nFrom June 2002 to July 2003 he acted as chairman of the board of Directors of Dala Group CJSC. Within this position, the first logical complex of A class \\\"Tau Terminal\\\" was commissioned in Kazakhstan.\\nFrom November 2007 to December 2009, he was an independent director, member of the board of directors of Kazakhtelecom JSC.\\nIn August 2008 he occupied a position of managing director, Member of the Management Board of Kazakhstani State Asset Management Holding \\\"Samruk-Kazyna\\\".\\nFrom December 2008 to May 2009, he occupied a position of Executive Director of National Welfare Fund \\\"Samruk-Kazyna\\\" JSC.\\nFrom June 2009 to January 2011, he acted as General Director of Trading and Transportation Company LLP, and from January 2011 to October 2011 he occupied a position of Vice-President of Mining Company LLP (both companies are within the National Atomic Company Kazatomprom).\\nFrom October 2011 to February 2013 he served as Deputy Chairman of the National Economic Chamber \\\"Atameken Union\\\". In the framework of the Atameken Union NEC, acted as an initiator of active steps in the liberalization of the market of professional technical education in Kazakhstan.\\nIn 2012, with the participation B.A. Baimukhanova, a terminal for liquefied gases \\\"AEGAZ Terminal\\\" was commissioned in Kerch (Ukraine), which strengthened the position of Kazakhstani producers of liquefied gases in international markets.\\nIn 2013, he was one of the founders of the \\\"Association of alternative energy sources,\\\" the Republic of Kazakhstan. The Association initiated and made amendments to the Law \\\"On supporting of the use of renewable energy sources\\\", developed regulations that paved the way for widespread use of alternative energy sources in Kazakhstan.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689366\",\"title\":\"Peterborough railway line\",\"body\":\"\\nPeterborough railway line\\n\\nThe Peterborough railway line was a railway line in South Australia. It extended from a junction at Roseworthy on the Morgan railway line through Hamley Bridge, Riverton, initially to Burra, then extended to Peterborough on the Port Pirie-Cockburn line.\\nThe Burra Burra railway was initially proposed as early as 1850, before any other railways north from Port Adelaide. The first stage, from Roseworthy to Forresters (now Tarlee) was opened on 3 July 1869. It extended to Redruth near Burra, opening on 29 August 1870. The broad gauge line was extended to Terowie (opened 14 December 1880), which had a break of gauge with narrow gauge continuing to Peterborough until the 1970s, when this section was converted to broad gauge. \\nRegular passenger services ceased at the end of 1986, with the last passenger train being a ARHS steam train on 19 September 1992. The line north of Burra was removed in 1992-1993. Grain was transported by rail until January 1999, and the entire line has not been used since 2004.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689367\",\"title\":\"Gladstone railway line\",\"body\":\"\\nGladstone railway line\\n\\nThe Gladstone railway line was a railway line in South Australia. It extended from a junction at Hamley Bridge on the Peterborough railway line (which provided connection through to Adelaide) through Balaklava and Brinkworth to Gladstone. \\nThe earliest part of the Gladstone railway line was the part of the Port Wakefield railway line east and north from Balaklava to Hoyleton. \\nThe stage from Hamley Bridge to Balaklava opened on 15 November 1879, meeting the line from Port Wakefield. \\nThe Gladstone line was completed as in 1894, reaching Gladstone railway station which was already on the Port Pirie-Cockburn line and a line north to Laura. The Gladstone railway line from Hamley Bridge to Gladstone was converted to in 1927 and closed in stages in the late 20th century.\\nAs the Balaklava railway station was originally on the Port Wakefield to Hoyleton line, before the railway from Hamley Bridge was built, and the new line entered the town from the southeast, trains using the route between Gladstone and Adelaide needed to change direction at Balaklava, as both the north and south lines entered the station from the east, with Port Wakefield being to the west.\\nThe \\\"Western System\\\" included the railway from Hamley Bridge to Gladstone, along with the lines from Balaklava through Port Wakefield, Kadina and Wallaroo, and the line from Kadina through Snowtown to Brinkworth. All of these lines were prepared for conversion from narrow to broad gauge in the mid-1920s, with the switch made on 1 August 1927.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689412\",\"title\":\"1989 Women's European Cricket Cup\",\"body\":\"\\n1989 Women's European Cricket Cup\\n\\nThe 1989 Women's European Cricket Cup was a international cricket tournament held in Denmark from 19 to 21 July 1989. It was the first edition of the Women's European Championship, and all matches at the tournament held One Day International (ODI) status.\\nFour teams participated, with the hosts, Denmark, joined by the three other European members of the International Women's Cricket Council (IWCC) – England, Ireland, and the Netherlands. Denmark was making its ODI debut. The tournament was played using a round-robin format, with England finishing undefeated in its three matches. Two English players, Wendy Watson and Jo Chamberlain, led the tournament in runs and wickets, respectively. All matches were played at the Nykøbing Mors Cricket Club, located in the town of Nykøbing Mors.\\nPoints table.\\nSource: \\nStatistics.\\nMost runs.\\nThe top five run scorers (total runs) are included in this table.\\nSource: \\nMost wickets.\\nThe top five wicket takers are listed in this table, listed by wickets taken and then by bowling average.\\nSource: \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689413\",\"title\":\"2011–12 Nedbank Cup\",\"body\":\"\\n2011–12 Nedbank Cup\\n\\nThe Nedbank Cup is a South African club football (soccer) tournament. The knockout tournament, based on the English FA Cup format, was one of a weak opponent facing a stronger one. The competition was sponsored by ABSA until 2007, after which Nedbank took over sponsorship.\\nThe winner of the 2011–12 Nedbank Cup winners, SuperSport United, qualified for the 2013 CAF Confederation Cup.\\nFormat.\\nThe 16 Premier Soccer League clubs, 8 National First Division teams, as well as 8 teams from the amateur ranks compete for the prize money of R6 million. The winner also qualifies for the CAF Confederation Cup.\\nThe preliminary round features all 16 National First Division teams and will be reduced to eight when the teams play on 12 December 2012.\\nThe teams are not seeded at any stage, and the first 16 sides drawn out of the hat receive a home-ground advantage. There are no longer any replays in the tournament, and any games which end in a draw after 90 minutes are subject to 30 minutes extra time followed by penalties if necessary.\\nTeams.\\nThe 32 teams competing in the Nedbank Cup competition are: (listed according to their league that they are playing in).\\nResults.\\nF.C. Cape Town\\nThanda Royal Zulu F.C.\\nDynamos\\nVasco Da Gama\\nPreliminary Round.\\nThe preliminary round saw National First Division sides play each other is a knockout round to decide who would compete in the 2012 Nedbank Cup.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689455\",\"title\":\"GABS Hottest 100 Aussie Craft Beers of the Year\",\"body\":\"\\n60 - Hawthorn Brewing Pale Ale\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689461\",\"title\":\"Victorian Emergency Management Training centre\",\"body\":\"\\nVictorian Emergency Management Training centre\\n\\nThe Victorian Emergency Management Training Centre (VEMTC) is a training facility for volunteer and career emergency services personnel. It used by the Metropolitan Fire Brigade, the Country Fire Authority (CFA), Victoria Police, Ambulance Victoria, Victoria State Emergency Service and the Department of Environment, and Primary Industries. The centre is located in Melbourne's north in Craigieburn,\\nHistory.\\n2014.\\nThe training centre was completed in June 2014 for $109 million and designed by Woods Bagot.\\nIt was build for Melbourne Fire Brigade (MFB) after 2 years of planning.\\n2015.\\nIt is the Primary training Centre for Melbourne Fire Brigade (MFB) and their recruiting courses and promotional courses are run from VEMTC\\nFollowing the closure of CFA Fiskville training ground due to health and safety reasons, the VEMTC became the primary training facility for CFA new career fire fighters.\\nKey features.\\nThe 10 hectare facility has a focus on urban emergency incidents with scenarios for the following settings:\\nThere is a large seven story builing or prop which includes a carpark and numerous other types of environments including an atrium, prison and hospital themes.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689463\",\"title\":\"Education Partnerships Africa\",\"body\":\"\\nEducation Partnerships Africa\\n\\nEducation Partnerships Africa is a volunteer-run charity which sends UK university students to work in rural secondary schools in East Africa. It aims to benefit students in East Africa by improving education in its partner schools, and to give personal development opportunities to UK university students. It was established in Kisii in 1990 as The Kenya Project. Since then it has expanded to two further sites: Kakamega in Kenya and Mbarara in Uganda, and now works with around 30 schools each summer.\\nSummer project.\\nStudent volunteers, known as Project Workers, spend ten weeks during their summer vacation living, either in a pair or trio, in a rural community in Kenya or Uganda. They work in partnership with the school management and local community to identify the most important needs for the school, and how they can best be addressed. These might include investing in sustainable resources, such as library books, science equipment and small infrastructure projects, helping with school management issues, and setting up co-curricular activities. Schools typically receive a minimum of two years of investment from the charity, before being reassessed to establish whether they would benefit from further investment. Volunteers fundraise money in the UK before their visit to cover the costs of the project, which includes the school investment, as well as their own flights, accommodation and living costs.\\nSchools.\\nIn Kenya EPAfrica works primarily with government funded District Schools. These are schools which any students with acceptable grades can attend, and which do not charge fees for tuition. In Uganda the charity also works with Technical schools which provide more vocational studies. The schools tend to be of medium size, in very rural locations, and to be short of resources, but with the potential to improve. Any school meeting the basic criteria can apply and most will receive a visit during the summer to assess their eligibility. In total EPAfrica has worked with over 100 schools in East Africa, investing more than £200,000.\\nUK operations.\\nEducation Partnerships Africa is a registered charity in the UK. It is run and managed entirely by volunteers, the majority of whom are alumni (i.e. former Project Workers). Each university has its own University Committee, which is responsible for recruiting Project Workers and running training for them throughout the year.\\nThe central charity is made up of a number of workstreams, such as Communications, IT and Finance, each headed up by a Workstream Lead. These are overseen by a Management Committee, made up of six volunteers. The strategic direction and governance of the organisation is overseen by a Board of Trustees, all of whom are also volunteers.\\nAccording to the Charity Commission the charity has an annual turnover of over £100,000, the majority of which is fundraised by its Project Workers. Because it is run entirely by volunteers the vast majority of this annual turnover is spent directly on its East Africa operations.\\nHistory.\\nThe charity began life in 1990 as The Kisii Project. In the early 1990s a group of Cambridge University students began teaching in a school in the Kisii area of Western Kenya. By 1995, Kisii Project had been incorporated into Link Africa (now Link Community Development) and extended to another two schools in the region. A few years later, the focus of the project was switched to resource investment to ensure sustainability.\\nIn 2002 the project expanded to Oxford University and started sending 20 Project Workers a year to work in 10 schools in Kisii. The two university projects combined as a registered charity under the name The Kenya Project, which was later changed to Kenya Education Partnerships. From 2009 the charity began a period of expansion of its UK operations to a number of London universities, starting by recruiting Project Workers from University College London; the project went on to accept applications from all universities in London.\\nIn 2008, violence following the disputed 2007 elections in Kenya made it unsafe for volunteers to be sent to Kenya so the charity moved its operations to Uganda for one summer. In 2010 it expanded to working in Kakamega, also in Western Kenya, and in 2013 it permanently expanded its operations to Mbarara in Uganda.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689466\",\"title\":\"Christian Manrique\",\"body\":\"\\nChristian Manrique\\n\\nChristian Manrique Valdor (Santander, Spain, 1975) is a civil engineer. Currently he is the CEO and founder at Soulware Global Development. Between 2007- 2011 he became the youngest Chairman of a Spanish Port Authority taking this post in Santander.\\nPort Authority of Santander.\\nOn September 24, 2007, he was appointed Chairman of the Port Authority of Santander by the Spanish minister of Public Works, Magdalena Álvarez, proposed by the Government of Cantabria.\\nWith this appointment, at the age of 31, he turned the youngest Chairman of a Spanish Port Authority ever.\\nThe Port Authority is the public institution managing the port of Santander in Cantabria within the bay of Santander.\\nMain performances\\nBetween 2007 and 2011 the Port Authority took different steps to improve its relation with the city, its citizens and with stakeholders. \\nNowadays investment has reached 300 million euros and has allowed the sharing of port facilities and spaces with citizenship. This also meant the construction of the Alejandro Zaera shipping High Performance Centre and the Emilio Botín Arts Centre. The signature of the agreement, on March 19, 2011, was the first step, after a long period of research, to develope new business and recreational areas within the city. By this, different historical parts of the harbour were improved. \\nThe deployment of the coal terminal eliminated dust particles that used to surround the city. Also the seed bulk terminal became a major improvement for food tracking. \\nNew car shipping routes from different brands -Ford, Tata, Iveco, BMW y MINI, among others- were added to the existing ones to grow activity among port agents. Besides, a new traffic route was settled with Gotteborg, Sweden.\\nThe Raos bridge construction became the solution to drive all heavy freight lorry traffic through port facilities. \\nMotorway of the Sea to the United Kingdom with Brittany Ferries and to Zeebrugge, in Belgium, was enhanced. By doing so, heavy lorry traffic decreased within the area. This also increased road securtiy and improved environmental conditions. \\nThe International Centre for Port Technology and Administration (CITAP) for postgraduate studies in logistics and port facilities settled in Santander. The centre has links with the Spanish Cooperation Agency AECID, OEA, Puertos del Estado and Cantabria University.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689472\",\"title\":\"Space radio station\",\"body\":\"\\nSpace radio station\\n\\nSpace radio station (short: space station) is – according to \\\"Article 1.64\\\" of the International Telecommunication Union´s (ITU) RR – defined as \\\"«A station located on an object which is beyond, is intended to go beyond, or has been beyond, the major portion of the Earth's atmosphere\\\".» \\nEach \\\"station\\\" shall be classified by the service in which it operates permanently or temporarily. However, most \\\"spacecraft\\\" communicate by this means. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689473\",\"title\":\"Macleay Valley Mustangs Rugby league club\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689475\",\"title\":\"Mademoiselle Chiffon\",\"body\":\"\\nMademoiselle Chiffon\\n\\nMademoiselle Chiffon is a 1919 French silent film directed by André Hugon and starring Musidora, Suzanne Munte and Kitty Hott.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689490\",\"title\":\"Marcin Urbanowski\",\"body\":\"\\nMarcin Urbanowski\\n\\nMarcin Urbanowski (born 3 March 1987) is a Polish long distance runner.\\nCareer.\\nFrom 2002-2007, Urbanowski attended the School of Sports Championship in Gdansk. From 2004-2010, he represented Poland in long-distance running. He has participated in an international athletic championships. In Polish National Cross-Country Championships, he won a gold medal for cross-country run, five silver medals for 3 km steeplechase and three medals in cross-country runs.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689499\",\"title\":\"Sucks Blood\",\"body\":\"\\nSucks Blood\\n\\nSucks Blood is the sixth studio album by the American garage rock band Thee Oh Sees, released on May 15, 2007 on Castle Face Records. The album is the band's second to be released under the name The Oh Sees, and is their final album before changing their name permanently to Thee Oh Sees. \\nRelease.\\nTo release the album, vocalist and guitarist John Dwyer founded Castle Face Records, alongside Matt Jones and Brian Lee Hughes.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689500\",\"title\":\"Raaste Pyar Ke\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689505\",\"title\":\"Lavant House\",\"body\":\"\\nLavant House\\n\\nLavant House (or West Lavant House) is an old English estate.\\nHistory.\\nThe earliest unarguable documentary reference to Lavant House dates to 1762-3. Architectural analysis of the building, its photographic depictions and an eighteenth century watercolour suggest that it was built between 1707 and 1725. The Richard Budgen map of Sussex (1723–24) shows two houses considered worthy of note in the manor of West Lavant, of which one is undoubtedly West Lavant Farm (built in 1711) while the other is likely Lavant House, given that no remains or records document another large house in the area.\\nSir John Miller.\\nIts first definite owner was Sir John Miller (the 4th Baronet) whose family had been prominent in Chichester first as justices of the peace and mayors then becoming members of parliament from the later 17th century. Sir John had enclosed the ‘park’ at West Lavant by 1740 after which date he may have resided in the first form of Lavant House, its central section.\\nThe Duke of Newcastle’s brother (Henry Pelham) wrote to the Duke saying \\\"you know my thoughts on Sir John he is as friendly and honest as the day is long\\\". This has to be balanced with the fact that the Second Duke of Richmond could relate to the Duke of Newcastle that Sir John had stated that he did not care a fart for [his father-in-law] Dr. Combs. Sir John is believed to be depicted as a lesser figure in a 1759 painting by George Stubbs showing the Third Duke of Richmond out hunting (this hangs in the front hall at Goodwood House). He died in 1772 and his relict Dame Susanna remained at Lavant House until her death in 1788 when the house was put on the market by her eldest Son (Sir Thomas Miller) who had by then moved to live in Hampshire. This was the advertisement in the \\\"The European magazine, and London Review\\\".\\nThird Duke of Richmond.\\nIn 1791 the house was bought by The Third Duke of Richmond who rented it to Lord Bathurst. In 1798 the Duke started work on the house. Apparently this was when the house was reoriented to face north (maps show the drive originally approaching from the south). He appears to have died (in 1806) while the project was underway.\\nThe next inhabitant was Henriette Ann Le Clerc, now thought to be the Duke’s illegitimate daughter. Henriette was brought from France by his sister, Lady Louisa Conolly, in 1778 at age 5 to live at her Goodwood estate. His will named her the first beneficiary, receiving life tenure of West Lavant House and Park and other lands and farms. She married Colonel (later General) John Dorrien on 28 March 1808. John and Henriette’s only child, Charles (the traditional Lennox family name), was born in Lavant in January 1809. Following John’s death in 1825, Henriette ran her estate, which she mentions in a series of letters to the Fifth Duke. She remained much of the time at Lavant. Forty years after moving into Lavant House, the Agricultural Gazette announced her death on January 6, 1846.\\nThe house then returned to the Goodwood Estate and was rented to a series of tenants.\\nTwentieth century.\\nIn 1907 the house was sold in 1907 to Mr Morrison and family. He is probably responsible for the dining room decoration.\\nThe house was then sold to Major Henry Frederic Low and family. In 1921 his relict sold it to Major Julian Day and Isabella. She then sold the building to the school.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689510\",\"title\":\"Pokeno Railway Station\",\"body\":\"\\nPokeno Railway Station\\n\\nThe Pokeno Railway Station is a former railway station in Pokeno on the North Island Main Trunk Line. It opened for passengers on 20 May 1875 and for goods on 6 April 1879. \\nThe station closed to passengers on 24 June 1973 and to goods on 30 March 1980. \\nWork on the proposed Paeroa–Pokeno Line commenced in 1938 and whilst approximately 13km of earthworks were completed at each end, the proposal was halted due to World War 2 and was not resumed following the war and was abandoned. The line was to be the first part of the East Coast Main Trunk Railway. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689520\",\"title\":\"The Thruster\",\"body\":\"\\nThe Thruster\\n\\nThe Thruster (French:L'arriviste) is a 1924 French silent film directed by André Hugon and starring Pierre Blanchar, Jeanne Helbling and Ginette Maddie.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689523\",\"title\":\"Sir Samuel Scott, 2nd Baronet\",\"body\":\"\\nSir Samuel Scott, 2nd Baronet\\n\\nSir Samuel Scott, 2nd Baronet may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689529\",\"title\":\"The Last Word (film 2009)\",\"body\":\"\\nThe Last Word (film 2009)\\n\\nThe Last Word (Persian: حرف آخر) is a 2009 Iranian Film Directed And produced By Hossein Shahabi .\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689539\",\"title\":\"The Little Thing\",\"body\":\"\\nThe Little Thing\\n\\nThe Little Thing (French:Le petit chose) is a 1923 French silent film directed by André Hugon and starring Max de Rieux, Alexiane and Jean Debucourt. It is based on the 1868 work \\\"Le Petit Chose\\\" by Alphonse Daudet.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689546\",\"title\":\"Gajpanth\",\"body\":\"\\nGajpanth\\n\\nGajpanth (also spelled as \\\"Gajpantha\\\") is a Jain pilgrimage site (\\\"tirth-kshetra\\\") located near the Jain temple at Nashik. It is located in the Indian state of Maharashtra, which is located in Masrul village, found in the Nashik district, 16 kilometers from Nashik Road Railway station and 5 kilometers from Nashik City. It is situated on the top of a small hill which is 400 feet tall. Those travelling up the steep slope are aided by a staircase built in black stone, which leads directly to the temple. The hill has 450 steps, 3 caves (known as 'chamar leni), and Jain temples belonging to the Digambara sect. There is also a sculpture depicting samavasarana (divine preaching hall of tirthankara) on the hills of Gajpanth.\\nHistory.\\nGajpanth is said to be the salvation place of seven 'Balbhadra' (saints) of the Jain Sect, known as Vijay, Achal, Sudharma, Suprabh, Nandi, Nandimitra and Sudarshan. It is believed that the saints took eight crores of Yadav Kings with them from this location to salvation. It is believed that many Jain monks (or sadhus) attained moksha from this hill.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689556\",\"title\":\"Uranus (novel)\",\"body\":\"\\nUranus (novel)\\n\\nUranus is a French novel written by Marcel Aymé and published in 1948. It is the third book in a trilogy which cover the \\npre-war, the war and the post-war periods in France. The first is \\\"Travelingue\\\" (1948) set in the time of the \\\"Front Populaire\\\". The second is called \\\"Le Chemin des écoliers\\\" (1946) set during the occupation and the third book - Uranus – focuses on post war France and the ‘purge’ – social cleansing which sought to discipline collaborators. People were shaved, humiliated, beaten and often killed without a fair trial. \\nThe true hero of the book, who is also the victim, is Léopold – owner of a coffee shop who discovers his passion for Jean Racine and for Andromaque thanks to lessons which, due to bombings of the school, must now take place in his establishment. He comes up with his own attempts at literature, such as \\n\\\"Passez-moi\\nAstyanax, on va filer en douce\\\" - \\\"Attendons pas d'avoir les poulets à nos trousses\\\".\\nThe novel was adapted as a film, \\\"Uranus\\\" by Claude Berri in 1990. \\\"Le Chemin des écoliers\\\" was adapted as the film \\\"Way of Youth\\\" by Michel Boisrond in 1959.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689559\",\"title\":\"Rogelio Frigerio\",\"body\":\"\\nRogelio Frigerio\\n\\nRogelio Frigerio (born January 7, 1970, Buenos Aires) is an Argentine economist and politician. He is the grandson of Rogelio Julio Frigerio, who worked with Arturo Frondizi. He is a supporter of developmentalism.\\nBiography.\\nRogelio Frigerio studied economy at the University of Buenos Aires, and worked as a teacher after graduating. He worked for the minister Roque Pérez in 1998. \\nHe was elected to the legislature of the city of Buenos Aires in 2011. He became president of Banco Ciudad in 2013. \\nMauricio Macri, elected president in 2015, nominated him to be his Minister of Interior Affairs, Public Works and Housing.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689562\",\"title\":\"Jacques Landauze\",\"body\":\"\\nJacques Landauze\\n\\nJacques Landauze is a 1920 French silent drama film directed by André Hugon and starring Marguerite de Barbieux, Maud Richard and Séverin-Mars.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689566\",\"title\":\"Udon Thani Rajabhat University\",\"body\":\"\\nUdon Thani Rajabhat University\\n\\nUdon Thani Rajabhat University (UDRU) is a university in Udon Thani, northeast Thailand. It confers associate, bachelors, masters, and doctoral degrees.\\nCampuses.\\nThe university has three campuses:\\nRanking.\\nIn July 2013, the university was ranked 65th best in Thailand by Webometrics.\\nSee also.\\nRajabhat University system\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689578\",\"title\":\"Linford Rees\",\"body\":\"\\nLinford Rees\\n\\nWilliam Linford Rees (24 October 1914 – 29 July 2004) was a Welsh psychiatrist, who was professor of psychiatry at St Bartholomew's Hospital, London, and president of the Royal College of Psychiatrists from 1975 to 1978.\\nA Welsh-speaker, Rees was born in Burry Port and studied at Llanelli Grammar School. He obtained his medical degree from the Welsh National School of Medicine in 1938. After postgraduate education at the Maudsley Hospital in London, he developed a specialism in psychosomatic medicine and worked closely with Hans Eysenck. He later worked at Whitchurch Hospital in Cardiff and again at the Maudsley, before taking up his academic post at St Bartholomew's in 1996. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689580\",\"title\":\"Castle Face Records\",\"body\":\"\\nCastle Face Records\\n\\nCastle Face Records is an American independent record label, founded in 2006 by John Dwyer, Matt Jones and Brian Lee Hughes.\\nThe label was initially formed to release \\\"Sucks Blood\\\", the sixth studio album by Dwyer's band, Thee Oh Sees.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689596\",\"title\":\"The Fugitive (1920 film)\",\"body\":\"\\nThe Fugitive (1920 film)\\n\\n The Fugitive (French:La Fugitive) is a 1920 French silent film directed by André Hugon and starring Marie-Louise Derval, André Nox and Armand Numès.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689600\",\"title\":\"Paeroa Railway Station\",\"body\":\"\\nPaeroa Railway Station\\n\\nThe Paeroa Railway Station is a former railway station in Paeroa; on the Thames Branch, and on the East Coast Main Trunk Railway to Waihi. \\nThe station opened for passengers and goods on 20 December 1895; and was replaced by a new building on a new site on 30 August 1925. The station closed to passengers on 20 July 1959, and to goods on 28 June 1991 with the closing of the Thames Branch. \\nWork on the proposed Paeroa–Pokeno Line commenced in the 1930s, but little was done and the proposal was abandoned. The line was to be the first part of the East Coast Main Trunk Railway. With the opening of the Kaimai Tunnel in 1979, the Paeroa to Katikati section of the East Coast Main Trunk was closed. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689619\",\"title\":\"European Moroccans\",\"body\":\"\\nEuropean Moroccans\\n\\nEuropean Moroccans or White Moroccans are Moroccans whose ancestry lies within the continent of Europe, most notably France and Spain.\\nPrior to independence, Morocco was home to half a million Europeans. And during the French protectorate in Morocco European Christians formed almost half the population of the city Casablanca. Later after the Independence in 1956, the European population has decreased substantially\\nIn the last years of the 19th century; 250,000 Spaniards lived in Morocco at the beginning of the 20th century. Most Spaniards left Morocco after its independence in 1956 and their numbers were reduced to 13,000.\\nToday European Moroccans are a small minority group in Morocco, accounting for only 1% of the country's population. In religion, most are Roman Catholic Christians.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689620\",\"title\":\"Kenneth Rawnsley\",\"body\":\"\\nKenneth Rawnsley\\n\\nProfessor Kenneth Rawnsley, CBE, (1926-1992) of University Hospital of Wales was an English psychiatrist who served as the president of the Royal College of Psychiatrists from 1981 to 1984.\\nRawnsley was brought up and educated in Burnley, Lancashire, later studying at Manchester University, where he obtained his medical qualification in 1948. He worked for a time in Canada, on the Stirling County Epidemiological Project, before joining the Medical Research Council Social Psychiatry Unit in London and Cardiff.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689626\",\"title\":\"ZM Rotorua\",\"body\":\"\\nZM Rotorua\\n\\nZM Rotorua is contemporary hit radio network in Hamilton, New Zealand. It is owned by New Zealand Media and Entertainment, and broadcasts via FM, and worldwide via the Internet. The network targets the 15–39 demographic specialises in a chart-music playlist of pop, rock, hip hop and dance music.\\nZM began broadcasting in Rotorua around 1998 on 98.3 MHz. This frequency had previously been used by a local rock station called Classic Rock 98.3FM.\\nZM Rotorua began to broadcast network programmes from Auckland from 2002.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689635\",\"title\":\"Child's Special Allowance\",\"body\":\"\\nChild's Special Allowance\\n\\nChild's Special Allowance was a payment under the United Kingdom system of Social Security.\\nIt was instituted by Harold Macmillan in 1959 for the orphaned children of divorced parents,\\nIt was a Contributory non-means tested benefit, paid in addition to Child benefit to a divorced woman whose husband had died, whose ex-partner had been paying maintenance and who had not got a new partner. It was not taxable, but was taken into account for meanstested benefits.\\nIt was abolished, as far as new claims were concerned, in April 1987. At that point it was paid at a rate of £8.05 per eligible child. Payments continued for existing beneficiaries under the scheme of transitional protection.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689645\",\"title\":\"Athens Digital Arts Festival\",\"body\":\"\\nAthens Digital Arts Festival\\n\\nAthens Digital Arts Festival (ADAF) (Greek: Διεθνές Φεστιβάλ Ψηφιακών Τεχνών της Ελλάδας) is an international festival, that takes place every May in Athens, Greece.\\nThe festival was established in 2005, under the title Athens Video Art Festival, as an initiation of the non-profit organization Multitrab Productions to create a platform for video art, installations and live performances. Since then, it has added interactive installations, web art, workshops, animation and digital images. In January 2015, the festival changed its name into Athens Digital Arts Festival.\\nHistory.\\nThe first edition of the International Festival of Digital Arts took place in April 2005 at the cultural space Thira Texnis, in Athens. The festival received 180 artworks submissions and hosted 137 of them.\\nFrom 2007 to 2011, the main venue of the festival was the cultural space Technopolis of the Municipality of Athens, with 1,544 artworks hosted from 8,200 submitted. Through these years the festival presented artworks in the main categories of video art, installations and live performances, and in new ones such as web art, animation and digital image, introduced since 2009. It has also collaborated with other festivals, such as Transmediale (DE), Cologne OFF (DE), LPM (IT), videoholica (BG), IN-EDIT (ES), onedotzero (UK), and art:screen (SE).\\nIn 2012 and 2013 the festival moved its activities to the center of Athens, with the titles “Visualize Athens” (2012) and “Living Athens” (2013). This initiative was supported by a number of volunteers, Greek Municipalities and international festivals and artists.\\nIn 2014, celebrating its 10-year anniversary, it decided to return to the cultural space “Technopolis” of Municipality of Athens. Some of the highlights were the live performances \\\"Abandonded cities\\\" by Hauschka and \\\"Late Speculation\\\" by Nonotak, as well as the installation “80 prepared dc-motors, cotton balls, cardboard boxes” by Zimoun.\\nIn 2015, the festival changed its name to Athens Digital Arts Festival, with eight official categories: installations, web art, video art, animation, performances, digital image, workshops and music and introduced for the first time a specific theme,“Public Space_s”., ADAF 2015 took place in the center of Athens with its main venue at Diplarios School and two main squares of Athens for parallel activities. The festival hosted the works of 150 artists from Greece and abroad, on the theme of public space both in the digital and urban environments. Some of the artists and international festivals participated in this year’s festival were Karl Heinz Heron (DE), Julian Oliver (NZ), Martin Bricelj Baraga (SI), Martin Reiche (DE), MADATAC (ES), IMPAKT (NL), VIDEONALE (DE), Cologne OFF (DE), videoholica (BG), and artvideoKOELN] (DE).\\nADAF has maintained active partnerships with local and international artists and audience, institutions, art foundations, schools, universities, galleries and museums and a number of 46 collaborating festivals. Athens Digital Arts Festival is also partner of , an international network of artists and professionals organizing festivals and/or working in the fields of audiovisual live performances. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689657\",\"title\":\"List of Kazakhstan football transfers winter 2016\",\"body\":\"\\nList of Kazakhstan football transfers winter 2016\\n\\nThis is a list of Kazakh football transfers in the winter transfer window 2016 by club. Only clubs of the 2016 Kazakhstan Premier League are included.\\nKazakhstan Premier League 2016.\\nAktobe.\\nIn:\\nOut:\\nAkzhayik.\\nIn:\\nOut:\\nAstana.\\nIn:\\nOut:\\nAtyrau.\\nIn:\\nOut:\\nIrtysh.\\nIn:\\nOut:\\nKairat.\\nIn:\\nOut:\\nOkzhetpes.\\nIn:\\nOut:\\nOrdabasy.\\nIn:\\nOut:\\nShakhter Karagandy.\\nIn:\\nOut:\\nTaraz.\\nIn:\\nOut:\\nTobol.\\nIn:\\nOut:\\nZhetysu.\\nIn:\\nOut:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689664\",\"title\":\"List of israeli top-flight league players with 100 or more goals\",\"body\":\"\\nList of israeli top-flight league players with 100 or more goals\\n\\nSince the israeli top-flight league's formation at the start of the 1931–32 season, 35 players have managed to accrue 100 or more goals in the league.\\nNahum Stelmach holds the record for the fewest games taken to reach 100, doing so in 138 appearances.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689667\",\"title\":\"72nd (Middlesex) Searchlight Regiment, Royal Artillery\",\"body\":\"\\n72nd (Middlesex) Searchlight Regiment, Royal Artillery\\n\\n72nd (Middlesex) Searchlight Regiment, Royal Artillery was an air defence unit of Britain's Territorial Army (TA) raised just before the outbreak of World War II, which served as part of Anti-Aircraft Command during and after the war.\\nOrigin.\\nAs the international situation deteriorated in the late 1930s, the threat of air raids on the UK led to the rapid expansion in numbers of anti-aircraft (AA) units manned by members of the part-time TA. Formed in November 1938, 72nd (Middlesex) was the third of a new group of three TA searchlight regiment raised by the Royal Artillery (previous TA S/L units had all been part of the Royal Engineers and/or converted from infantry battalions). It consisted of HQ and Nos 465–467 Companies (later Batteries) based at a newly built drill hall at Vicarage Road, Heston. Shortly afterwards the regiment moved to Twickenham. It was equipped with the new '90 cm Projector Anti Aircraft', a smaller and lighter piece of equipment than previous searchlights, with a more powerful high current density arc lamp with automatic carbon feed.\\nWorld War II.\\nAnti-Aircraft Command mobilised in August 1939, ahead of the declaration of war on 3 September, and the regiment took its place in 47th AA Brigade, part of 5th AA Division tasked with defending Southampton.\\nSouthampton was a regular target for raids by the German \\\"Luftwaffe\\\" during the Battle of Britain in the summer of 1940, but by November 1940 the regiment had transferred to 40 AA Bde in 2nd AA Division in the Midlands. It had the responsibility for covering RAF airfields in the East Midlands and continued in that role throughout the Blitz of 1940–41.\\nAs the threat from the Luftwaffe waned in 1944, the War Office warned in June that AA Command would have to release manpower to provide reinforcements to 21st Army Group fighting in . The run-down began in September 1944, and 72nd S/L Rgt was placed in 'suspended animation' in that month, with its personnel being posted away.\\nPostwar.\\nWhen the TA was reconstituted in 1947, 72nd S/L Rgt was reformed at Twickenham as 607 Searchlight Regiment, RA (Middlesex), forming part of 67 AA Bde (the former 41 AA Bde based at Shepherds Bush). In 1949 the regiment's role was altered and it was redesignated 607th (Mixed) Light Anti-Aircraft/Searchlight Regiment, RA (Middlesex) ('Mixed' denoting that members of the Women's Royal Army Corps were integrated into the unit).\\nAA Command was disbanded on 10 March 1955, and 607 LAA/SL Regiment was disbanded at the same time.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689669\",\"title\":\"Government Superior Science College Peshawar\",\"body\":\"\\nGovernment Superior Science College Peshawar\\n\\nGovernment Superior Science College (GSSC) Peshawar Pakistan, is one of the three public sector colleges for male students operating within Peshawar metropolitan, along with Government College (GC) Peshawar; and Government College Hayatabad, Peshawar. GSSC, established in 1962, is providing educational services along with training in sports and other social skills. The college has produced a long list of prominent figures in different fields of life,that serving humanity around the globe. Being one of the prominent institution established after independence, the college attracts the highest number of students each year, and very few of applied students are able to get admission.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689691\",\"title\":\"New Zealand NBL Coach of the Year Award\",\"body\":\"\\nNew Zealand NBL Coach of the Year Award\\n\\nThe National Basketball League Coach of the Year is an annual National Basketball League (NBL) award given since the 1989 New Zealand NBL season to the best head coach of the regular season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689693\",\"title\":\"2016 World Touring Car Championship season\",\"body\":\"\\n2016 World Touring Car Championship season\\n\\nThe 2016 World Touring Car Championship season is the thirteenth season of the FIA World Touring Car Championship, and the twelfth since the series was revived in 2005.\\nRegulation changes.\\nThe sporting regulations were approved by the FIA, at the December 2015 meeting of the World Motor Sport Council.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689707\",\"title\":\"Breakthrough Energy Coalition\",\"body\":\"\\nBreakthrough Energy Coalition\\n\\nBreakthrough Energy Coalition is a global group of 28 high net worth investors from 10 countries committed to funding clean energy companies emerging from the initiatives of Mission Innovation, which was also announced at the 2015 United Nations Climate Change Conference.\\nThe group aims to bolster governmental assistance in renewable energy such as solar energy and wind power to $20 billion.\\nMembers.\\nThe group is spearheaded by Bill Gates, who previously announced a personal $2 billion investment, and includes:\\nCriticism.\\nThere has been criticism that the coalition was announced too early, before crucial details has been confirmed. At launch, a Gates Foundation spokesman confirmed that investment professionals had yet to be appointed, named investors - other than Gates - hadn't publicly stated their level of investment and a financial structure hadn't been confirmed.\\nThe scale, at US$20bn of public cash over five years and the initial US$2bn of private cash, has also been criticised as inadequate: the World Bank suggests that US$100bn would be needed annually.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689709\",\"title\":\"Paropsis atomaria\",\"body\":\"\\nParopsis atomaria\\n\\nParopsis atomaria is a common leaf beetle in the subfamily Chrysomelinae. Atomaria translates to mean speckled or freckled. \\nThey occur along eastern Australia from Adelaide to Brisbane. \\nP. atomaria produce two lifecycles during the summer across most of its range A female can produce 600 eggs and deposits them at the tip of a leaf or twig. This species is one of a few of the paropsines that may become a pest of plantation trees. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689714\",\"title\":\"The Gathering (EP)\",\"body\":\"\\nThe Gathering (EP)\\n\\nThe Gathering is an EP by American hip hop group Living Legends. It was released on Legendary Music on April 8, 2008.\\nReception.\\nAndrea Woo of \\\"Exclaim!\\\" said: \\\"While some will invariably be disappointed by the short playtime, the seven tracks of \\\"The Gathering\\\" are strung together with a fairly stable energy to create a solid offering devoid of filler.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689724\",\"title\":\"Byzantine tower of Biccari\",\"body\":\"\\nByzantine tower of Biccari\\n\\nByzantine tower of Biccari is a builder locate in city center of Biccari, city of Province of Foggia in Italy.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689740\",\"title\":\"Walk like a Panther\",\"body\":\"\\nWalk like a Panther\\n\\n\\\"Walk like a Panther\\\" is a song by the All Seeing I with vocals from Tony Christie. It charted at number 10 on the UK Singles Chart.\\nBackground.\\n\\\"Walk like a Panther\\\" was performed by All Seeing I with main vocals from Tony Christie and background vocals from Steve Edwards, and was written by Richard Barratt, Jason Buckle, Jarvis Cocker and Dean Honer, and was their third single from their album \\\"Pickled Eggs and Sherbert.\\\" It was written specifically for Christie to such an extent that it even mentions one his past hits – I Did What I Did For Maria – and describes the hometown of the band members of the All Seeing I, Cocker and Christie: Sheffield. Cocker personally contacted Christie, who was living in Spain at the time as this was where he was most successful, asking if he would feature on the record.\\nMusic video.\\nA music video was produced for the song. It features Christie singing his parts and culminates in others walking with their arms held high in time with the music, mimicking panthers.\\nChart performance.\\nWalk like a Panther peaked im January 1999 at number 10 on the UK Singles Chart, becoming Christie's first hit in that country for twenty five years. It would be the band's only top ten single; The Beat Goes On and 1st Man in Space would peak at numbers 11 and 28 respectively.\\nCritical reception.\\n\\\"NME\\\" said of the song \\\"People just don't write songs like this any more!\\\", said the song had \\\"the vocal gravitas of a man, a common man, defiant in his invective against his lot, his shitty neighbourhood\\\" and ended by describing it as \\\"brave, impassioned and chuffin' catchy.\\\"\\nUsage in popular culture.\\nThe band performed the song on \\\"Top of the Pops,\\\" and the song was featured on its corresponding album \\\"Top of the Pops 1999, Vol. 1.\\\" It was also featured on the compilation albums \\\"The Chillout Album, Vol. 2,\\\" \\\"Soundsystem Four\\\" and \\\"Now 42.\\\" Three years later, The Pretenders would cover the song on their album \\\"Loose Screw.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689766\",\"title\":\"1958 Targa Florio\",\"body\":\"\\n1958 Targa Florio\\n\\nThe 42° Targa Florio took place on 11 May, on the Circuito Piccolo delle Madonie, (Sicily, Italy). It was the third round of the F.I.A. World Sports Car Championship, which was running to new regulations introduced at the beginning of the season. The most influential of these regulations changes would be the 3.0 litre engine size limit. The event returned to the championship for the first time since 1955, following the demise of the Mille Miglia and the ban on road racing on mainland Italy. But such outcry did not deter Vincenzo Florio from holding his event on the traditional 45 mile mountainous circuit.\\nReport.\\nEntry.\\nA massive total of 65 racing cars were registered for this event, of which 53 arrived for practice. Only these, only 38 started the long distance race on the public roads of Sicily. This, the 42nd edition of the event, saw a change on the nature of the race. Two drivers would be permitted now and the limit set so no driver would drive more than seven laps out of the total race distance of 14. So, it ensured no single driver would be able to complete the whole race.\\nThe first two events of the season, the 1000 km Buenos Aires and 12 Hours of Sebring ended with victory for Phil Hill and Peter Collins, for Scuderia Ferrari. As Hill and Collins also won the last race of the previous season, the Venezuelan Grand Prix they’ve now won three races in a row for the Scuderia. With these new rules, and Maserati on the brink of financial crisis, Scuderia Ferrari would head the Italian challenge. Ferrari had four works 250 TRs in Sicily, Hill/Collins, Mike Hawthorn/Wolfgang von Trips, Luigi Musso/Olivier Gendebien and Gino Munaron/Wolfgang Seidel. Opposition would no longer come from Maserati… but from Porsche and Aston Martin.\\nDavid Brown sent just one Aston Martin DBR1 over from England for Stirling Moss/Tony Brooks, while Porsche arrived with three different cars, a 356A Carrera, a 550 RS and a 718 RSK, for their squad of drivers led by Jean Behra and Giorgio Scarlatti. They were joined by a fleet of privateer drivers in their Alfa Romeos, Oscas and other mainline sportscars.\\nQualifying.\\nPrior to the race, there was no formal practice held, but Sergio Der Stephanian was killed in a pre-race accident, following a collision with a sand-laden lorry. He died shortly after in hospital.\\nRace.\\nWith each lap 45 miles in length, the race covered a total of 14 laps, or 630 miles, the Targa Florio is unlike any other sports car race. Littered with switchback turns, blind corners and a straight nearly four miles longer than Circuit de la Sarthe’s Mulsanne, the Targa was a fearsome thing to behold.\\nDay of the race would be sunny and warm, with the first of the cars leaving the small village of Cerda, one-by-one, at 40 seconds intervals. It was clear right at the start that something was amiss for most of the competitors, as more than a few would be off the pace, while other would be off the road, in verges trying to repair their cars and get back into the race. Jean Behra would spin his Porsche 718 RSK. Moss would damage a wheel when he went off the road. Meanwhile, von Trips damaged his Ferrari heavily and returned to the pits dragging bit of his car along the ground. It seemed that everyone was struggling over the mountain roads, except one, Musso.\\nMusso was setting an incredibly pace. He started last of the big works entrants, but at the end of the first lap, he would be first. Being in the lead, he set about performing an error-free drive. Moss would be on the hunt in his Aston, ever-impressive sliding around the corners, kicking up gravel everywhere and carrying on without any trouble whatsoever, following that earlier incident. He would break the lap record, lapping more than a minute faster than Musso. But the Italian had already done all of his head work. He led and held steady before handing the car over to Gendebien.\\nWith Gendebien now the car, Moss would take his Aston even faster, but it came at a price. After five laps, the gearbox gave up and Moss was out of the race, before Brooks had a chance to race. Despite the retirement of the sole Aston Martin, the circuit maintained the pressure on the factory efforts. Hill would end up in a ditch, losing valuable time trying to get out and back on his way. As for the Belgian, he was driving smartly, keeping the car on the road, and in the lead. He was just a few laps before returning the car back to Musso. This was the only Ferrari not under heavy pressure from Behra. The nimble little RSK was providing its self on the winding roads, and joining the battle for a spot in the top three.\\nMusso held a commanding lead, despite the advances of Behra. But Musso was not immune to trouble. Only three laps from the end, there was trouble. He appeared to be off the pace, lapping four minutes slower than previously. He was happy to make it back to the pits, as the brake fluid had leaked out of its reservoir. He had no brakes. It was reported that he completed the descent out of the mountains by staying in low gear. In any other race, this would have spelled the end, however, Musso and Gendebien had controlled the race right from the start. Such was their lead, the Ferrari mechanics repaired the car, Gendebien got back in the car for the remaining laps, still with a three-minute lead.\\nNow the leaders were out of trouble, their team-mates von Trips and Hawthorn were not. They were in second place, but with Behra back in the Porsche and absolutely flying. Following a pitshop, the margin between the Ferrari and Porsche would be practically nothing. There was no stopping Behra, and he continued to up his pace and Hawthorn could not response. Starting the 14th and last lap, Behra’s pace had meant he was now ahead.\\nOut in front, Gendebien brought the 250 TR home, to record a brilliant victory. Though Moss had set a new lap record in his Aston, the race had been dominated from the very beginning by Musso. Car number 106, took an impressive victory, winning in a time 10hr 37:58.1, averaging a speed of 59.251 mph. Second place went to the Porsche of Behra and Giorgio Scarlatti, albeit over 5½ minutes adrift. The podium was complete by second Scuderia Ferrari of von Trips and Hawthorn, who were 54 secs behind in third.\\nOfficial Classification.\\n\\\"Class Winners are in Bold text.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689769\",\"title\":\"Three Sisters (sternwheeler)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689770\",\"title\":\"List of mosques in Kosovo\",\"body\":\"\\nList of mosques in Kosovo\\n\\nBelow is a partial list of mosques in Kosovo. The list includes mosques built during the Ottoman Empire period, as well as those built in the modern era.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689782\",\"title\":\"Bachra\",\"body\":\"\\nBachra\\n\\nBachra is a census town in Tandwa Block of Chatra district in the state of Jharkhand, India.\\nGeography.\\nBachra is a colliery township located at .\\nDemographics.\\nAs per 2011 Census of India Bachra had a population of 12,969, of which 7,169 were males and 5,800 were females. Scheduled Castes numbered 1,519 and Scheduled Tribes numbered 826.\\nLiteracy.\\nAs per 2011 census the total number of literates in Bachra was 9,718 out of which 5,887 were males and 3,831 were females.\\nEconomy.\\nThe North Karanpura Coalfield is spread across parts of Ranchi, Hazaribagh, Chatra and Latehar districts of Jharkhand covering an area of 1,230 km2. This coalfield in the upper reaches of the Damodar Valley, has reserves of around 14 billion tonnes of coal, very little of which has been exploited. Karkatta, KD Hesalong, Manki, Churi, Bachara UG, Bachara OC, and Dakara are long established collieries south of the Damodar. North of the Damodar lies comparatively new major mines such as Piparwar Mine and Ashoka Project. 23 mines are planned in the northern sector (near Bachra). Those in an advanced stage of planning are: Dhadu, Purnadih, Magadh, and Amrapali.This happens to be the largest mining sector of Central Coalfields Limited.\\nProjects in the Piparwar area of Central Coalfields Limited (as in 2015) were: Piparwar open cast, Ray-Bachra underground, Ashoka open cast, Piparwar coal handling plant and Piparwar coal preparation plant.\\nTransportation.\\nBachra is served by Ray railway station, about 25 km from Barkakana railway station on the Sonnagar-Barkakana loop line.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689786\",\"title\":\"Hannah Wants\",\"body\":\"\\nHannah Wants\\n\\nHannah Wants (born Hannah Alicia Smith in 1986) is a British DJ and producer from Birmingham.\\nHer track \\\"Rhymes\\\", a collaboration with Chris Lorenzo, reached #13 in the UK in 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689801\",\"title\":\"Madeleine Dior\",\"body\":\"\\nMadeleine Dior\\n\\nMarie Madeleine Juliette Martin, who was born in Angers, France, in 1879 and died in Granville (French department of Manche) in 1931, was the wife of the industrialist Maurice Dior. She was also the mother of the grand couturier Christian Dior and the French Resistance member Catherine Dior.\\nBiography.\\nMadeleine Martin was the daughter of a lawyer from Angers and Juliette Surosne, originally from the department of Calvados, France. Monsieur Martin died young and Madeleine was brought up by her mother.\\nIn 1898, at the age of nineteen, she married Maurice Dior who was six years her senior. The couple moved to the center of Granville in the department of Manche, where Maurice Dior had grown up. They had five children: Raymond in 1899, Christian in 1905, Jacqueline in 1909, Bernard in 1910, and Ginette, known as Catherine, in 1917.\\nIn 1905, to satisfy Madeleine, who did not like the house in the center of town, the Dior family purchased a property that was still in Granville but on the edge of a cliff, facing the sea. This windswept villa was called \\\"Les Rhumbs\\\", named after the thirty-two divisions of the wind rose. It had a large piece of adjoining land which Madeleine Dior transformed into a southern-style garden, overcoming the hostility of the winds blowing in from the sea to grow delicate plants.\\nIn 1910, taking advantage of the revenue from Maurice Dior's company which was enjoying great success, the family moved to Rue Richard Wagner in Paris, since renamed Rue Albéric Magnard. Madeleine Dior excelled as the lady of the house and a woman of taste, decorating the apartment in the Louis XVI-Passy style fashionable at the time. She surpassed herself when holding dinners served by butlers in white gloves, and her bouquets were much admired by her guests.\\nIn 1914, the family decided to take refuge from the war and returned to live in the Granville villa, which had been their holiday home since 1910. Like all society ladies in the region, Madeleine Dior participated in the war effort. In his autobiography, Christian Dior remembers this period when women were occupied \\\"with making shredded cloth bandages, hospitals, letters from the front and recreation sessions for the injured.\\\" The family returned to live in Paris in 1918, not far from the apartment where they had lived before the war.\\nIn 1930, Bernard, the second youngest of the family, was affected by a serious nervous disorder. Madeleine Dior, doubtless worn down by what was happening to her son, died the following year. Jacques Bonjean, a gallery owner and friend of Christian Dior, described her as an \\\"...elegant and slender woman, sometimes distant, always graceful.\\\"\\nThe \\\"Les Rhumbs\\\" garden in Granville.\\nThroughout her life, Madeleine Dior lived out her passion for flowers through the transformation of the windswept land surrounding her villa in Granville into a beautiful English-style garden. First she had a veranda added to the façade of the house, which sheltered a winter garden, and planted a small wood of umbrella pines to protect her plants from the wind. This was a bold choice of tree for the Normandy coast. She also had windbreak walls put up to protect the property, and finally she had a greenhouse built where the plants were overwintered. On the advice of Christian Dior, the greenhouse was replaced in 1925 by a pergola and pool; a rose garden was planted nearby a short while later. The creation of the garden became a two-person job, which created a bond between Madeleine Dior and her son: the latter organized it by positioning the features, while she took care of the planting. The result was a complex planting scheme that sheltered the most fragile flowers from the wind: \\\"\\\"Madeleine Dior's planting formed curtains of shielding greenery, so that the second was more precious and the third consisted of plants that it would have been impossible to grow here: geraniums, roses, jasmine... All these protective screens created layers, structures and motif effects,\\\" explains the garden's current landscape gardener.\\nJames de Coquet, journalist for \\\"Le Figaro\\\", remembered being amazed by his visit to \\\"Les Rhumbs\\\" in 1929: \\\"I complimented Madame Dior on her beautiful garden. I told her she must have an excellent gardener.\\\"\\\"\\nIn 1997, the villa, which had been purchased by the town of Granville in 1932, became the Musée Christian Dior. The garden, Madeleine Dior's life work, is one of the few \\\"artistic gardens\\\" of the early 20th century to have been preserved. For student landscape gardners, it is also an exceptional subject of study.\\nInfluence on the work of Christian Dior.\\nMadeleine Dior was particularly close to her son Christian, the future couturier: in the eyes of her other children, he was her \\\"favorite\\\" and he followed her everywhere, from her Granville garden to Orêve, her favorite Parisian florist, and the dressmaker Rosine Perrault. Madeleine Dior was a close follower of fashion, as demonstrated by a Roaring Twenties dress that she designed, which is exhibited in the Granville museum.\\nThe couturier remembered his mother when, years after her death, on the eve of his first runway show, he was looking at the façade of the House that bore his name on Avenue Montaigne and exclaimed: \\\"If Mother had lived, I would never have dared.\\\" In the same way that we sense the presence of Charles Baudelaire's mother, Caroline, in his work and the influence of Jeanne, Marcel Proust's mother, in his, Madame Dior had a profound effect on her son's entire career. Regarding the Granville house, Christian Dior wrote: \\\"\\\"I have the tenderest, most magical memories of it. Not only that; my life, my style, owe nearly everything to its location and architecture,\\\" and \\\"...(it) was pebbledashed in a very soft pink, blended with gray gravel, and these two colors have remained my favorite shades in couture.\\\"\\\" But it was above all his mother's garden that made a deep impression on the couturier. Having spent his childhood learning the names of flowers and their descriptions from horticulture catalogs, Christian Dior drew inspiration from them to create the silhouettes that would lead to his success in 1947 with the \\\"Corolle\\\" line. The decor of the Diors' apartment in La Muette, Paris, was also imprinted on the couturier's imagination: reference is made to it in the Louis XVI style which inspired the interiors of the Dior boutiques. Finally, it was Madeleine Dior's look that the couturier remembered when he invented the famous New Look with its nipped in waist, pronounced hips and emphasized bust recalling the feminine silhouettes of the Belle Époque. Long after the couturier's death, the House of Dior continues to pay tribute to this muse, as in the Fall-Winter 2005 runway show, where a Belle Époque-inspired dress was named \\\"Madeleine\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689808\",\"title\":\"Thames Railway Station\",\"body\":\"\\nThames Railway Station\\n\\nThe Thames Railway Station is a former railway station in Thames, New Zealand on the former Thames Branch from Morrinsville to Thames. \\nThe station opened on 19 December 1898 with the opening of the branch line. Passenger service ceased from 28 March 1951. There were also station buildings at Thames North and Thames South. \\nThe branch was closed (apart from a section) on 28 June 1991, and goods service ceased. However the station building remained as it was listed by NZHPT Category II in 1982. It is a standard Vintage station, with gables, finials and scalloped bargeboards.\\nWork on the proposed Paeroa–Pokeno Line commenced in the 1930s, but little was done and the proposal was abandoned. The line was to be the first part of the East Coast Main Trunk Railway. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689859\",\"title\":\"Mission Innovation\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689866\",\"title\":\"The Three Masks\",\"body\":\"\\nThe Three Masks\\n\\nThe Three Masks (French:Les trois masques) is a 1929 French silent film directed by André Hugon and starring Renée Héribel, Jean Toulout and François Rozet.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689874\",\"title\":\"Georgi Yungvald-Khilkevich\",\"body\":\"\\nGeorgi Yungvald-Khilkevich\\n\\nGeorgi Yungvald-Khilkevich (22 October 1934 – 11 November 2015) was a Soviet and Russian film director, screenwriter, producer, actor, theatre director and set designer. Most famous for his musicals and Alexandre Dumas adaptations. He directed 22 motion pictures and TV movies between 1966 and 2009. Honored Artist of the Russian SFSR (1990) and Ukraine (1995).\\nBiography.\\nGeorgi Yungvald-Khilkevich was born into a theatrical family of noble heritage. His mother Nina Ivanovna Buiko was a ballet dancer. His maternal grandfather Ivan Petrovich Buiko came from an old Russian family and served as a colonel in the Imperial Russian Army and a commandant in Warsaw. He joined Bolsheviks in 1917. Georgi's father Emil Iosifovich Yungvald-Khilkevich was an acclaimed theater director and one of the founders of the Uzbek National Theater of Opera and Ballet (later Navoi Theater). His paternal grandfather came from Polish szlachta and owned railroads in Western Ukraine, while his wife Elena Cavalieri was an Italian; she was said to be the sister of the famous opera singer Lina Cavalieri who was very popular in the Russian Empire and regularly visited Kiev with concerts.\\nGeorgi Yungvald-Khilkevich graduated from the in 1963. He worked as a set designer at Tashkent theaters and film studios. In 1966 he finished directing and screenwriting Mosfilm courses and started working at the Odessa Film Studio, where he later directed most of his movies.\\nHis first major breakthrough happened in 1969 with the musical film \\\"Dangerous Tour\\\" loosely based on the memoirs of Alexandra Kollontai. The screenplay was written with Vladimir Vysotsky in mind, who eventually played the main part, wrote all the songs and did some uncredited contribution to the final draft. His partners were Nikolai Grinko, Yefim Kopelyan, Ivan Pereverzev and Georgi Yumatov. The film turned to be one of the leaders of the Soviet box office in 1970 (9th place).\\nIn 1978 Khilkevich turned to Alexandre Dumas who happened to be one of his favourite writers since childhood. His 3-part made-for-TV adventure musical \\\"D'Artagnan and Three Musketeers\\\" turned to be an ultimate success, with many songs and catchphrases becoming part of the popular culture. It was followed by three sequels in 1992, 1993 and 2009. In 1988 he made another Dumas adaptation – The Prisoner of Château d'If based on The Count of Monte Cristo novel. The screenplay was co-written by Mark Zakharov, while all the songs were written and performed by Alexander Gradsky.\\nAmong his other notable works was another musical Ah, Vaudeville, Vaudeville... and a comedy The Art of Living in Odessa based on The Odessa Tales by Isaac Babel. He rarely turned to cinema during the post-Soviet years. In 1997 he joined at the National Cats Theater in Moscow as a stage director and scriptwriter. He also worked as a set designer in various theaters. His last film in the Musketeer series directed in 2007 and screened in 2009 was met with harsh critique and became a box office bomb.\\nYungvald-Khilkevich died from the heart failure at the age of 81. He was buried at the Troyekurovskoye Cemetery in Moscow. He was survived by his third wife, an actress Nadira Mirzaeva (born 1969), and two daughters — Natalia (born 1960) and Nina (born 1997).\\nBibliography.\\nGeorgi Yungvald-Khilkevich, Natalia Yungvald-Khilkevich. \\\"За кадром (\\\"eng. \\\"Behind the Screen)\\\". Moscow: , 2000 (Autobiography). ISBN 5-227-00627-X\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689889\",\"title\":\"1990 Women's European Cricket Cup\",\"body\":\"\\n1990 Women's European Cricket Cup\\n\\nThe 1990 Women's European Cricket Cup was an international cricket tournament held in England from 18 to 22 July 1990. It was the second edition of the Women's European Championship, and all matches at the tournament held One Day International (ODI) status.\\nFour teams participated, with the hosts, England, joined by the three other European members of the International Women's Cricket Council (IWCC) – Denmark, Ireland, and the Netherlands. A round-robin format was used, with the top teams proceeding to the final. England was undefeated in the round-robin stage and beat Ireland by 65 runs in the final, winning the championship for a second consecutive time.\\n England's Wendy Watson led the tournament in runs for a second year running, while Ireland's Susan Bray was the leading wicket-taker. The tournament was hosted by East Midlands Women's Cricket Association, a member of England's Women's Cricket Association, and matches were played at venues in three English counties (Leicestershire, Northamptonshire, and Nottinghamshire).\\nRound-robin.\\nPoints table.\\nSource: \\nStatistics.\\nMost runs.\\nThe top five run scorers (total runs) are included in this table.\\nSource: \\nMost wickets.\\nThe top five wicket takers are listed in this table, listed by wickets taken and then by bowling average.\\nSource: \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689898\",\"title\":\"Mount Allen (Victoria Land)\",\"body\":\"\\nMount Allen (Victoria Land)\\n\\nMount Allen () is a peak, 1,400 m, standing between Clark Glacier and the head of Greenwood Valley in Victoria Land. Charted by the Victoria University of Wellington Antarctic Expedition (VUWAE), 1959-60, and named for A.D. Allen, one of the party's geologists.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689907\",\"title\":\"Shanmughanathar Temple, Kunnakudi\",\"body\":\"\\nShanmughanathar Temple, Kunnakudi\\n\\nKunnakudi Shanmughanathar temple (also called Kunnakudi Temple or Kunnakudi Murugan Temple) in Kundrakudi, a village in the outskirts of Karaikudi in Sivaganga district in the South Indian state of Tamil Nadu, is dedicated to the Hindu god Murugan. Constructed in the Dravidian style of architecture, the temple is located in the Tirupattur - Karaikudi Road, around from Karaikudi. There are three caves located on the western side of the lower rock, that has rock-cut shrines from the Pandyan Empire from the 8th century. The caves have the earliest sculptural representation of Dvarapalas, the guardian deities, for any South Indian temple.\\nThe temple has a five-tiered gateway tower, the gopuram in the hill, leading to a pillared hall and the sanctum. The temple is open from 6:00 am - 11:00 am and 4 - 8:00 pm. Four daily rituals and many yearly festivals are held at the temple, of which Panguni Uthiram festival celebrated during the Tamil month of \\\"Panguni\\\" (March - April) and Thaipoosam during \\\"Thai\\\" (January - February) being the most prominent. The temple is maintained and administered by the Kunnakudi Thiruvannamalai Mutt Adikam, while the rock-cut caves are maintained as a protected monument by the Archaeological Survey of India.\\nLegend.\\nThe place was originally called Kundrakudi as it was located in a hill (\\\"Kundram\\\" means hill in Tamil), which with the period of time became Kunnakudi. It is also called by other names like Mayuragiri, Mayilmalai, Arasavaram and Krishanagaram as the hill resembles the shape of a peacock. As per Hindu legend, sage Agasthya is believed to have worshipped Murugan at this place. As per another legend, Sooran, the demon king infuriated peacock, the sacred vehicle of Murugan. He told the bird that Garuda, the sacred vehicle of Vishnu and Swan, the sacred vehicle of Saraswati could travel faster than it. The peacock swallowed Garuda and the swan in anger. Vishnu prayed to Muruga to retrieve his vehicle back from the peacock, who readily acceded to the request. The peacock, realising its mistake, prayed Muruga by doing penance at this place.\\nArchitecture.\\nThe temple is located in Kundrakudi, in the outskirts of Karaikudi in Sivaganga district in Tamil Nadu on the road from Tirupattur to Karaikudi. The hill has a height of and occupies an area of . The temple has a five tiered rajagopuram, the gateway tower raising to a height of , which pierces the granite wall surrounding the temple. The sanctum faces East and the image of the presiding deity Murugan is sported with the images of his consort Valli and Deivasana, each of whom are seen sitting on a peacock. There are shrines of other deities around the sanctum in the precinct.. \\nThere are three caves in the western side of the lower hill, with rock-cut images dedicated to Shiva in each of them. The first two caves have intricate rock-cut sculptures and Dvarapalas on either side of the sanctum, while the third one is plain. There are various sculpted images of Vishnu, Durga, Lingodbhava, Harihara. The image of Dvarapalas in the caves, on either sides of the sanctum, with each leaning in the direction facing the sanctum, are found to be the earliest representation of the images. These are not found in Pallava architecture, which precedes the Pandyas. The caves are considered one of the major specimens of rock-cut architecture of the Pandyas, counted along with Vettuvan Koil, Thirumalaipuram and Thiruparankundram.\\nCulture.\\nThe temple priests perform the \\\"pooja\\\" (rituals) during festivals and on a daily basis. The temple rituals are performed four times a day: \\\"Kalasandhi\\\" at 6:00 a.m., \\\"Uchikala poojai\\\" at 11:00 a.m., \\\"Sayarakshai\\\" at 6:00 p.m., and \\\"Arthajama Pooja\\\" at 7:45 p.m. Each ritual has three steps: \\\"alangaram\\\" (decoration), \\\"neivethanam\\\" (food offering) and \\\"deepa aradanai\\\" (waving of lamps) for the presiding deities. There are weekly, monthly and fortnightly rituals performed in the temple. The temple is open from 6:30 am - 12:00 pm and 5 - 8:30 pm on all days except during festive occasions when it has extended timings. The major festivals of the temple include the Panguni Uthiram festival celebrated during the Tamil month of \\\"Panguni\\\" (March - April) and Thaipoosam during \\\"Thai\\\" (January - February). The other festivals include Kantha Sashti, Vaikasi Visagam, Aavani Moolam and Paal Perukku Vizha. Like other Murugan temples during the festivals, hundreds of devotees carry pots of milk and Kavadi around the streets of the temple. Devotees offer pepper and salt to Saravana Poigai, the temple tank, as a mark of worship. The temple is revered in the verses of \\\"Thirupugazh\\\" the 15th century anthology on Murugan by Arunagirinathar.\\nIn modern times, the Sivaganga district administration has identified the temple as one of the prominent tourist attractions in the district. The temple is administered by Kunnakudi Thiruvannamalai Mutt Adikam, which was established during the 16th century. In modern times, the caves are maintained and administered by Archaeological Survey of India as a protected monument.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689927\",\"title\":\"The Princess and the Clown\",\"body\":\"\\nThe Princess and the Clown\\n\\nThe Princess and the Clown (French:La princesse aux clowns) is a 1924 French silent film directed by André Hugon and starring Huguette Duflos, Charles de Rochefort and Magda Roche.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689932\",\"title\":\"Patricia Farrar\",\"body\":\"\\nPatricia Farrar\\n\\nPatricia Jean \\\"Pat\\\" Farrar (; August 13, 1931 – October 31, 2015) was an American educator. She served as the First Lady of South Dakota from 1969 to 1971 during the administration of her husband, former Governor Frank Farrar. Additionally, she also served on the board of advisers of the John F. Kennedy Center for the Performing Arts. Patricia Farrar won a gold medal at the National Senior Games, also known as the Senior Olympics, in 1989.\\nEarly life and education.\\nFarrar was born Patricia Henley on August 13, 1931, in Britton, South Dakota, to Percy Denis and Margaret (née Schneider) Henley. She was raised in nearby Claremont, South Dakota, where she graduated as valedictorian from Claremont High School in 1949.\\nShe graduated cum laude from the University of South Dakota in 1953, where she studied English and art. Henley placed first runner up in the Miss South Dakota pageant while in college. She began her career as a teacher at Summit High School in Summit, South Dakota. \\nCareer.\\nFarrar served as the First Lady of South Dakota from 1969 to 1971. She was also a member of the South Dakota Commission on the Status of Women, as well as the South Dakota State University's advisory board for apparel and textiles. Nationally, Farrar held a seat on the board of advisers for the John F. Kennedy Center for the Performing Arts in Washington D.C. She wrote and performed a chautauqua based on the life of South Dakota's first First Lady, Margaret Mellette. \\nIn 1989, Farrar won a gold medal in race walking at the second National Senior Games in St. Louis, Missouri.\\nDeath.\\nFarrar died from Lewy Body Dementia and Parkinson’s disease at Avera St. Luke's Hospital in Aberdeen, South Dakota, on October 31, 2015, at the age of 84. She was survived by her husband, former Governor Frank Farrar, and their five children. Governor Dennis Daugaard ordered flags to be flown at half-staff on November 7, 2015, in Farrar's honor.\\nPersonal life.\\nHenley married her husband, Frank Farrar, whom she had met at the University of South Dakota, on June 5, 1953, at Fort Benning, Georgia, where Farrar was stationed in the U.S. Army at the time. The couple had five children, Jeanne, Sally, Robert, Mary, and Anne.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689943\",\"title\":\"T. Sreenidhi\",\"body\":\"\\nT. Sreenidhi\\n\\nT. Sreenidhi(Srinidhi) (born 26 January 1990), is an eminent Carnatic Musician and Playback Singer. Sreenidhi performed in major sabhas in India and presented her concerts in many destinations around the world. She received multiple awards and honours. Sreenidhi trained under legendary vocal stalwart Sangita Kalanidhi Dr. Nedunuri Krishnamurthy.\\nEarly Life and Family.\\nSreenidhi was born in Anantapur. Her father T. Subramanyacharyulu is Carnatic Vocalist and Violinist and her mother T. Sarada is Carnatic Musician. She got her initial training by her mother as she used to sing Thyagaraja krithis and ragaalapanams, swarams in place of lullabies to the just born Sreenidhi. Her father, who is a vocal-violin exponent, nurtured her with a good authentic musical foundation. Later she trained under legendary vocal stalwart Sangita Kalanidhi Dr. Nedunuri Krishnamurthy. She is married to Venkatesh on 26 January 2015 at Hyderabad.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689954\",\"title\":\"MeetYourMakers\",\"body\":\"\\nMeetYourMakers\\n\\nMeetYourMakers (MYM or mYm) is an esports organization based in Germany. It competes in \\\"League of Legends\\\", \\\"Dota 2\\\", \\\"StarCraft II\\\", and \\\"\\\". MYM was founded in 2001.\\nMYM dropped their \\\"WarCraft III\\\" team in 2009.\\nIn July 2015 MeetYourMakers was involved in an incident where the team manager threatned to take Marcin \\\"Kori\\\" Wolski's house away as a result of his leaving the team. After the dissolution of their previous LoL team they acquired LCS team Supa Hot Crew.\\nIn the history of MYM, players and teams from MYM were featured with awards like “eSports player of the year”, “eSports team of the year” and more. MYM won more than 400 titles at important major events around the world, including the World Cyber Games (WCG) and the Electronic Sports World Cup (ESWC).\\nHistory.\\n2000: In 2000, the Dane Mark Peter \\\"Mercy\\\" Fries founded the organization Meet Your Makers, the first time publicly made its appearance in 2001. 2002, the decision was made to rebuild Meet Your Makers to a professional organization.\\n2006: the organization Meet Your Makers was bought by investors. These investors founded together with the former owners of the company \\\"Regroup eSports A / S\\\", which was' renamed on 16 July 2008 in \\\"ESNation A / S\\\" beginning of August 2006 joined MYM unification G7 Teams at.\\nThe Warcraft III Department has always been considered the flagship of Meet Your Makers. MYM players have reached many high rankings in various international leagues and tournaments, such as the ESL WC3L Series, the NGL ONE and World Cyber Games. Jang \\\"Moon\\\" Jae-ho Since February 2006, the Korean professional players stood at Meet Your Makers under contract. He was considered the best paid Warcraft III player in the world, his monthly salary was estimated at around 8,000 euros during his time in MYM article about Moons content the beginning of 2008 also changed the Dutch Manuel \\\"Grubby\\\" Schenkhuizen to MYM.\\nThe end of 2007: committed Meet Your Makers after several failed attempts to gain a foothold in the Counter-Strike world, the reigning World Cyber Games World Champion PGS Gaming. The new team won the Electronic Sports World Cup 2008.\\nEarly 2009 sparked the MYM team for Warcraft III and Starcraft on surprising.\\nFirst August 2009: bought the Leipziger IT companies' FIO Systems AG \\\" Meet Your Makers, a few weeks later the organization with new teams in Warcraft III, Counter-Strike and Warcraft was reopened III DotA.\\nOn June 6, 2012: the Meet Your Makers was sold by the FIO Systems AG to a private investor and registered as a limited company. \\\" 'Khaled Naim' \\\" acts since as managing director of Meet Your Makers GmbH.\\nJune 2014: Meet Your Makers GmbH cooperates with the hmf Group (Mannheim), which constitute an integral marketing partner and support in the operational sales.\\nOctober 2014: currently owns Meet Your Makers team in Fifa 14 Hearthstone, Heroes of the Storm and Battlefield 4\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689965\",\"title\":\"Lubricant (disambiguation)\",\"body\":\"\\nLubricant (disambiguation)\\n\\nA lubricant is a substance introduced to reduce friction between surfaces in mutual contact. For the general article about the topic, see lubricant.\\nFor related pages, see:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689970\",\"title\":\"The Black Diamond\",\"body\":\"\\nThe Black Diamond\\n\\nThe Black Diamond (French:Le diamant noir) is a 1922 French silent mystery film directed by André Hugon and starring Claude Mérelle, Ginette Maddie and Armand Bernard.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689981\",\"title\":\"Cirencester by-election, 1892\",\"body\":\"\\nCirencester by-election, 1892\\n\\nThe 1892 Cirencester by-election was held on 18 October 1892 after the retirement of the incumbent Liberal MP Arthur Brend Winterbotham. The seat was gained by the Conservative candidate Thomas Chester-Master. Chester-Master was originally declared the victor by 3 votes, but on petition and after scrutiny, the votes were declared equal and a new election was held.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689997\",\"title\":\"Exeter Hotel\",\"body\":\"\\nExeter Hotel\\n\\nThe Exeter Hotel is an Adelaide hotel located on Rundle Street.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48689999\",\"title\":\"Guillermina Bravo Montaño\",\"body\":\"\\nGuillermina Bravo Montaño\\n\\nGuillermina Bravo Montaño (born 27 July 1949) is a Colombian teacher and politician of the Independent Movement of Absolute Renovation (MIRA) party. Currently she is a member of the Chamber of Representatives of Colombia. She was Deputy of the Department Assembly of Valle del Cauca from 2008 to 2011.\\nPolitical career.\\nFrom 2006 to 2007 she was a member of the leadership of her party, the Independent Movement of Absolute Renovation (MIRA) party, at the Cali city level. In 2006 she ran for Representative of the Valle del Cauca Department, being less than 1,000 votes away from getting a seat as Representative.\\nIn 2007 she was elected Deputy of the Department Assembly of Valle del Cauca. In 2008 she served as the leader of her party at the Valle del Cauca Department level. During her term as Deputy she authored 10 Ordinances related to Social, Educational and Entrepreneurial promotion matters. In 2011 she ran for Governor of the Valle del Cauca Department, coming in fourth place thanks to 80,000 votes. \\nIn 2014 she ran again for Representative of the Valle del Cauca Department and was elected thanks to 7,194 votes. During her term as Representative, she co-authored, along with his parliamentary group, the law commonly known as \\\"Natalia Ponce Law\\\", which increases the sentences related to Acid throwing to up to 50 years of prison. She was designated Spokesperson of the Afro-parliamentary group of the Congress of Colombia in 2014. A year later, she was designated Chairwoman of the Afro-parliamentary group.\\nRecognition.\\nIn 2008 she received the \\\"Pro-Joven Award\\\", which is bestowed by the Municipal Council of Youth of the El Cerrito Municipality upon those leaders who promote positive changes and development to the region. In the same year, she received the \\\"Ancestral Heritage Golden Palm Award\\\" in the Political Contribution category, bestowed during the Expo-Pacific 2008 Convention. This award is granted based on the contributions of an individual to the Afro-Colombian culture.\\nIn 2010 she received the \\\"Afro-descence Excellence Medal\\\" from the interim Governor Raymundo Tello, as an acknowledgment of her authorship of Ordinance 299 of 2009, which set guidelines for a public policy to benefit the Afro-Descendant, Palenquero and Raizal communities of the Valle del Cauca Department.\\nIn 2014 she received the \\\"Guachupé of Gold Award\\\" in the Afro-Colombian leadership category, awarded by the Afro-colombian society of Bogotá D.C., as an acknowledgment of her constant efforts to benefit the Afro-Colombian communities, especially women.\\nIn 2015 she received a \\\"Formal-style Honour Note\\\" from the Department Assembly of Valle del Cauca due to her leadership and contributions to the development of the Colombian Pacific region and her social and political career in favor of women.\\nPersonal life.\\nGuillermina Bravo Montaño was born in Cali, Colombia, on 27 July 1949. She graduated in Social Sciences at Santiago de Cali University. She took two specialization courses, one in Cognitive processes and one in Administrative and Political Management She worked as a teacher for over 25 years. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690005\",\"title\":\"Films Albatros\",\"body\":\"\\nFilms Albatros\\n\\nFilms Albatros was a French film production company established in 1922. It was formed by a group of White Russian exiles who had been forced to flee following the 1917 Russian Revolution and subsequent Russian Civil War. Initially the firm's personnel consisted mainly of Russian exiles, but over time French actors and directors were employed by the company. Its operations continued until the late 1930s.\\nHistory.\\nFaced with increasingly difficult working conditions in Russia after the revolution of 1917, the film producer Joseph Ermolieff decided to move his operations to Paris where he had connections with the Pathé company. Arriving in 1920 with a group of close associates, Ermolieff took over a studio in Montreuil-sous-Bois in the eastern suburbs of Paris and began making films through his company Ermolieff-Cinéma. His co-founder of the company was Alexandre Kamenka, another Russian exile, and when in 1922 Ermolieff moved to Germany, Kamenka, together with his colleagues Noë Bloch and Maurice Hache, took over the company and re-established it as the Société des Film Albatros. He also set up a distribution company called Les Films Armor in order to control the distribution of his own films. Various explanations have been given for the choice of the name Albatros: the name of a boat which brought some of the emigrés from Russia; a symbol of White Russia; an incident with an albatros on the journey. As well as adopting the image of the albatros as its symbol, the company took the motto \\\"Debout dans la tempête\\\" (\\\"upright in the storm\\\").\\nAmong the group of Russian artists who stayed to work with Albatros were the directors Victor Tourjansky and Alexandre Volkoff, the art director Alexandre Lochakoff, the costume designer Boris Bilinsky, and the actors Ivan Mosjoukine, Nathalie Lissenko, Nicolas Koline, and Nicolas Rimsky. Although this Russian company initially favoured Russian themes, Kamenka quickly realised the need for greater integration with French film production, and they turned increasingly to French subjects. In 1924 a number of Kamenka's Russian associates left Albatros, and Kamenka offered opportunities to several innovative French film-makers including Jean Epstein, Jacques Feyder, Marcel L'Herbier and René Clair. \\nKamenka's production policy combined prestige projects with openly commercial films, and his consistent record made him the most successful French producer during the 1920s, according to Charles Spaak, who came to the company as a script-writer in 1928. Kamenka successfully achieved international distribution for many of his films (even in Soviet Russia with which his company had so little political sympathy, and from 1927 he entered into co-production arrangements with production companies in other European countries, driven by growing financial difficulties in the French film industry. The arrival of sound pictures posed a serious difficulty for Albatros which had hitherto relied considerably upon Russian actors, especially Mosjoukine whose accent precluded a successful transition into the talking era.\\nThe company's output diminished in the 1930s, but it achieved one further artistic success of note when Jean Renoir joined them for his 1936 adaptation of Gorki's \\\"Les Bas-fonds\\\". By this time, Albatros was the longest surviving film company operating in France, but with the outbreak of World War II, Kamenka wound up the company which had remained particularly associated with silent cinema.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690011\",\"title\":\"Commonwealth Heads of Government Meeting 2020\",\"body\":\"\\nCommonwealth Heads of Government Meeting 2020\\n\\nThe Commonwealth Heads of Government Meeting 2020, also known as CHOGM 2020 will be the 26th meeting of the heads of government of the Commonwealth of Nations. It will be held in Malaysia. It will be the first CHOGM held in Malaysia since 1989.\\nThe position of Commonwealth Chair-in-Office, held by the government leader of the CHOGM host country, will be transferred at the summit from the Prime Minister of the United Kingdom to the Prime Minister of Malaysia who will hold the post until the 27th CHOGM expected in 2022.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690012\",\"title\":\"Domini Guardato\",\"body\":\"\\nDomini Guardato\\n\\nDomini Guardato was a powerful landowner, wealthy merchant, and an intellectual in the Kingdom of Sicily. He was received into the Order of Jerusalem and decorated the Orders of Calatrava and Alcantara. He was a member of the aristocratic Patrician family Guardato, during the late 12th century.\\nBiography.\\nHe was from the medieval merchant city of Sorrento during the Kingdom of Sicily, where his family served as Patricians of the region. Since the time of King William the Good, he distinguished himself ever more to military valor, and therefore was often decorated to high and important office and other noble prerogatives. His descendants enjoyed nobility in the Kingdom of Sicily; Sorrento in the Seat of Door, and in Salerno in the Seat of Field; and many members of his family were received into the Knights Hospitaller; Knights of Alcantara, and the Knights of Calatrava. He more important than nobility, his family enjoyed the status of Patrician of Sorrento unto its abolition, in 1804 was ascribed To register Squares closed the Kingdom.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690022\",\"title\":\"List of alumni of the Pontifical North American College\",\"body\":\"\\nList of alumni of the Pontifical North American College\\n\\nThis is a partial list of notable alumni of the Pontifical North American College in Rome, a Roman Catholic educational institution that forms and educates seminarians and student priests for dioceses in the United States (as well as Canada and Australia). It was founded in 1859.\\nIf the prelates and priests listed here completed their normal course of pre-ordination theology studies while at the North American College (in general, the bachelor and licentiate), only their year of ordination is given; if they were sent to the College for graduate studies or continuing education after ordination, then that graduate degree or program is listed.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690023\",\"title\":\"St. Andre's Parish\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690025\",\"title\":\"Cirencester by-election, 1893\",\"body\":\"\\nCirencester by-election, 1893\\n\\nThe 1893 Cirencester by-election was held on 23 February 1893 after a court declared a by-election in 1892 was rerun after the votes had been declared equal. The seat was gained by the Liberal candidate Harry Lawson Webster Levy-Lawson. The Conservative candidate Thomas Chester-Master was declared the victor of the 1892 by-election by 3 votes, but on petition and after scrutiny, the votes were declared equal and the 1893 by-election was held.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690035\",\"title\":\"Sooklek\",\"body\":\"\\nSooklek\\n\\nSooklek (or Suklek) is a Thai cartoon character created by Prayoon Chanyavongs in the \\\"likay\\\" style. He is characterized by a \\\"jaunty feather stuck in a band around his head and carrying a sword in one hand.\\\" Prayoon's son was named after this character.\\nHistory.\\nSooklek was first popularized as the titular role in the cartoon series \\\"Prince Chantarakob\\\" (จันทโครพ) and eventually became a public personality as creator Prayoon used the character in his satirical cartoons. His name means \\\"happy, good fellow\\\".\\nWhen Field Marshall Thanom Kittikachorn gave out an order for Prayoon to stop his political cartoons in 1968, Sooklek was drawn with lips sewn together. Another warning from the government brought about change in Sooklek as the sewn lips were replaced by a big mustache. Sooklek regained his mouth with the ouster of Thanom in 1973.\\nAdaptation.\\nAn animation series in both 2D and 3D has been produced with Sooklek in the starring role. A commemorative book has also been launched by the Prayoon Foundation. The book is divided into two sections with the first focusing on Sooklek and the second about Prayoon's life and works.\\nTo commemorate Prayoon's centenary, the Thailand Post launched a sheet of stamps. In addition, a local brand is producing T-shirts featuring Sooklek characters with part of the proceeds going to Prayoon Foundation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690040\",\"title\":\"Jena, Alabama\",\"body\":\"\\nJena, Alabama\\n\\nJena is an unincorporated community in Greene County, Alabama, United States.\\nHistory.\\nJena was most likely named by a German family who settled in the area in honor of Jena, Germany. A post office operated under the name Jena from 1837 to 1921. Baseball Country, a world-renowned baseball camp, is located in Jena.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690057\",\"title\":\"Sam George Nartey\",\"body\":\"\\nSam George Nartey\\n\\nSam George Nartey is a Ghanaian politician. He is a member of the National Democratic Congress. In November 2015 he defeated, the incumbent E. T. Mensah to represent the party in the 2016 parliamentary elections for Ningo-Prampram constituency.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690071\",\"title\":\"Excela Health\",\"body\":\"\\nExcela Health\\n\\nExcela Health is a not-for-profit health organization that includes three licensed, acute care hospitals, two free-standing outpatient surgery centers, home care and hospice, physician practices, a durable medical equipment company and other facilities and services. Formally incorporated in 2004, Excela Health is governed by a single Board of Trustees. Its headquarters is located in Greensburg, Pennsylvania.\\nExcela Health is the sole provider of health care in Westmoreland County. It employs 4,800 employees and credentials 800 physicians and allied health professionals. It also serves parts of Fayette and Indiana counties.\\nThree charitable foundations exist to benefit the clinical and operational needs of Excela Health's hospitals and affiliated health care services. They include the Westmoreland/Frick Hospital Foundation with its two branches - the Frick Hospital Foundation and the Westmoreland Hospital Foundation - and the Latrobe Area Hospital Charitable Foundation.\\nHealth facilities.\\nFrick Hospital.\\nA 102 licensed-bed hospital located in Mount Pleasant, offers services including general acute care, surgical services, award-winning emergency services, a sleep center, rehabilitation services and more. In addition to modem operating rooms and needed support functions, there is a short procedure unit for minor outpatient surgeries. The hospital's Women's Care Services offer a range of breast health services and bone density scanning. The Outpatient Services Center features a centralized outpatient registration area with quick, convenient patient registration surrounded by a variety of outpatient testing areas and services, drawing together nuclear medicine, pulmonary function lab, stress lab, EKG, EEG, echo cardiography, x-ray, ultrasound and mammography for a \\\"one-stop\\\" shop.\\nLatrobe Hospital.\\nLocated in the eastern section of Westmoreland County, Latrobe Hospital offers acute, surgical and specialized services. A 196-licensed-bed hospital, Latrobe features mammography, bone density services . Extensive cardiac diagnostic testing and treatment abound with a cardiac rehabilitation program available for post surgery patients. The hospital also features a state of the art newly renovated emergency department, a sleep center; diabetes center and endocrine clinic; outpatient surgery and short procedure suites; emergency services; wellness programs and more. Home to Excela Health's child and adolescent inpatient unit, Latrobe Hospital does not offer child and adolescent health services. It is also home to Excela Health's geriatric assessment service, and is the site of the health system's Center for Neurosciences. A Family Medicine Residency Program at Latrobe Hospital, affiliated with the Jefferson Medical College of Thomas Jefferson University, allows physicians hands-on experience flexible enough to prepare them for rural, suburban or urban medical practice.\\nWestmoreland Hospital.\\nWestmoreland Hospital, located in central Westmoreland County, is a full-service, acute care hospital and regional referral center with 364 licensed beds. The Family Additions Maternity Center offers maternity care in a home-like atmosphere featuring labor-delivery-recovery suites (LDRs) with operating suites for Cesarean or high risk births, and a Special Care (Level II) Nursery. There are also services for women planning or considering pregnancy and gynecological services. Behavioral Health Services feature an adult inpatient unit as well as outpatient services for adults. Additionally, the Westmoreland campus offers a breast health center; outpatient services including the SurgiCenter at Westmoreland and the Short Stay Surgery unit; sleep center; diabetes services and endocrine clinic; digestive disorders center; pain center; fixed site and open MRI units; a large critical care unit with an intensivist program (offering 24-hour-a-day, in-house physician specialist coverage in the critical care areas), and emergency care. It is also home to the interventional Center for Cardiovascular Medicine encompassing cardiac catheterization labs, electrophysiology labs and open heart surgery.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690078\",\"title\":\"Ma Junqing\",\"body\":\"\\nMa Junqing\\n\\nMa Junqing (; born October 1956) is a Chinese politician, serving since 2015 as the Deputy Communist Party Secretary of Jilin province.\\nMa was born in Gongzhuling, Jilin. He joined the Communist Party in March 1976, shortly before the death of Mao. He graduated from the department of economics at Jilin University, and later obtained a doctorate in economics.\\nMa's political career originated in the Jilin provincial organization of the Communist Youth League, in which he served as deputy secretary, and secretary, before being transferred to serve as mayor of Songyuan. He later served as party chief of Siping, and the secretary-general of the Jilin government. He joined the Jilin provincial Party Standing Committee as head of the propaganda department in 2004; later he took on the office of the secretary-general of the Jilin provincial party committee, and in August 2008, was re-shuffled to Vice-Governor. In May 2012 he was named executive vice-governor of Jilin.\\nIn November 2015, he was named deputy party chief of Jilin province.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690098\",\"title\":\"Rallye du Var\",\"body\":\"\\nRallye du Var\\n\\nThe Rallye du Var is a yearly motor rally held in the month of November in the French commune of Sainte-Maxime in Var. It is often held as the final round of the French Rally Championship. It began in 1950.\\nThe rally has attracted famous names from the world of rally driving such as the recently retired nine-time WRC champion, Sébastien Loeb (the 2000, 2009 and 2014 winner), 1994 WRC champion Didier Auriol (the 1987 and 1988 winner), Freddy Loix, Craig Breen, Jari-Matti Latvala (the 2011 winner), Dany Snobeck (the 1982 and 2008 winner), Romain Dumas, François Chatriot (the 1985, 1986 and 1989 winner), and Julien Maurin (the 2013 winner). Jari-Matti Latvala was the first non-Frenchman and thus foreign driver to win the rally, and it was also his first win on asphalt.\\nAs well as part of the French Rally Championship, the Rallye du Var was part of the European Rally Championship calendar from 1984 to 2001.\\nRecent years.\\n2010.\\nIn 2010, Cédric Robert won for the second time, having previously succeeded in 2002. He led the entire event and won four stages. Ex-Formula One driver Robert Kubica, who had recently completed the 2010 season for Renault, won the last three stages to ensure a high overall finish of fourth was bagged. The other two podium finishers turned out to be Bryan Bouffier and Stéphane Sarrazin.\\n2011.\\nIn 2011, WRC front runner Jari-Matti Latvala became the first foreign driver to win the race and the first Finn to do so. He beat previous winner Stéphane Sarrazin and Estonian driver Ott Tänak. He and his co-driver Miikka Anttila won in a Ford Fiesta RS WRC. It was also the first win for Latvala on asphalt, and he led the rally from start to finish, winning six stages along the way.\\n2012.\\nIn 2012, Cédric Robert won this rally for a third time, following his wins in 2002 and 2010. He was again co-driven by Matthieu Duval. The second placer was Le Mans winner Romain Dumas and third was Irishman Craig Breen. Ex-Formula One driver Robert Kubica of Poland dominated the rally and won every stage prior to crashing out late on, this allowed the Frenchman Robert to take the lead and win.\\n2013.\\nIn 2013, Julien Maurin won for the first time with co-driver Nicolas Klinger. Jérémi Ancian and Pierre Roché joined him on the podium. After a hat trick of stage wins, Maurin led to the end.\\n2014.\\nIn 2014, Sébastien Loeb driving his Citroën DS3 WRC won the Rallye du Var for the third time and the second with his wife Sévérine Loeb as the co-driver. The second place driver was David Salanon and third was Patrick Magnou. An influx of spectators caused three of the stages to be cancelled. However, Loeb did not let this hinder his performance, leading the entire rally from start to finish, as with his rival Jari-Matti Latvala in 2011.\\n2015.\\nIn 2015, the 61st running of this event, David Salanon and Romain Roche won in a Ford Fiesta RS WRC. Their first win on the asphalt event was followed by a secon place from newly crowned JWRC and WRC-3 champions Quentin Gilbert and Renaud Jamoul and third place went to Pierre and Martine Roché. For the second year running, the winner, Salanon, had a lights to flag victory. As for Gilbert, the newest Junior World Champion took advantage of his experience and won the final stage to elevate himself to second overall.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690105\",\"title\":\"The Two Pigeons (film)\",\"body\":\"\\nThe Two Pigeons (film)\\n\\nThe Two Pigeons (French:Les deux pigeons) is a 1922 French silent film directed by André Hugon and starring Armand Bernard, Germaine Fontanes and Huguette Delacroix.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690115\",\"title\":\"André Ferreira (Portuguese footballer)\",\"body\":\"\\nAndré Ferreira (Portuguese footballer)\\n\\nAndré Filipe Magalhães Ribeiro Ferreira (born 29 May 1996) is a Portuguese professional footballer who plays for Benfica B as a goalkeeper.\\nClub career.\\nFerreira was born in Vila Nova de Gaia, Portugal. On 24 November 2015, he debuted professionally with Benfica B in a 2015–16 Segunda Liga match against Oriental.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690123\",\"title\":\"Old Islam in Detroit\",\"body\":\"\\nOld Islam in Detroit\\n\\nOld Islam in Detroit: Rediscovering the Muslim American Past is a 2014 book by Sally Howell, published by the Oxford University Press. It discusses the Muslims of early 20th century Detroit, Michigan, and Detroit prior to 1970.\\nContents.\\nThe first parts of the book discusses the first Muslims to settle Detroit and the city's the first Islamic religious facility, the Highland Park Mosque. Another chapter discusses the second mosque, Universal Islamic Society (UIS). The later chapters discuss Islamic leaders who originated from Detroit and the first mosques to open in Dearborn. At the end of the book Howell states that pre-1980s views of Muslims influences views of Islam held by Americans in the post-September 11 environment.\\nThe book includes interviews of the original Muslims and their families.\\nReception.\\nDawn-Marie Gibson of Royal Holloway, University of London stated that the book was \\\"thoroughly researched\\\" and is \\\"a valuable contribution to scholarship on American Islam.\\\"\\nB. D. Singleton rated the book two stars, and stated that the book \\\"is appropriate for all academic libraries\\\" and is \\\"nicely illustrated but would have been strengthened by a basic chronology.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690127\",\"title\":\"Stephen E. Gordy\",\"body\":\"\\nStephen E. Gordy\\n\\nStephen E. Gordy (March 20, 1920 – October 27, 2004) was an American politician, military officer, and educator.\\nFrom Dalton, Georgia, Gordy graduated from the United States Military Academy in 1943 and from the Naval War College. He served in the United States Army during World War II and the Korean War. Gordy was a teacher, principal, football and baseball coach in the Loudoun County, Virginia Public Schools. Gordy served in the Virginia House of Delegates from 1982 to 1987 and was a Republican; he lived in Mantua, Virginia. He died at his home in Dalton, Georgia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690133\",\"title\":\"List of One Day International cricket matches\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690141\",\"title\":\"Appollinaire University\",\"body\":\"\\nAppollinaire University\\n\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690148\",\"title\":\"The Wedding March (1929 film)\",\"body\":\"\\nThe Wedding March (1929 film)\\n\\nThe Wedding March () is a 1929 French silent comedy film directed by André Hugon and starring Pierre Blanchar, Louise Lagrange and Paul Guidé.\\nIt was made by the French subsidiary of Paramount Pictures. The film's sets were designed by the art director Christian-Jaque.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690156\",\"title\":\"Karakhan Manifesto\",\"body\":\"\\nKarakhan Manifesto\\n\\nThe Karakhan Manifesto was a statement of Soviet policy toward China dated July 25, 1919. It was issued by Lev Karakhan, deputy commissioner for foreign affairs for Soviet Russia. The manifesto offered to relinquish various rights Russia had obtained by treaty in China, including extraterritoriality, economic concessions, and Russia's share of the Boxer indemnity. These and similar treaties had been denounced by Chinese nationalists as \\\"unequal.\\\" The manifesto created a favorable impression of Russia and Marxism among Chinese. It was often contrasted with the Treaty of Versailles (1919), which granted Shandong to Japan.\\nThe manifesto was prompted by the Bolshevik advance into Siberia, which created a need to establish a relationship with China. The Bolsheviks saw the Chinese as one of \\\"the oppressed peoples of the East\\\" and therefore a potential ally against the \\\"imperialist\\\" powers.\\nThe manifesto is addressed to, \\\"the Chinese people and the Governments of North and South China.\\\" Because both Russia and China were in a state of civil war at this time, diplomatic exchanges were often delayed. Although the document was published in Moscow in August 1919, it was not formally presented to Chinese diplomats until February 1920. The version presented at this time includes the passage, \\\"the Soviet Government returns to the Chinese people, without any compensation, the Chinese Eastern Railway.\\\"\\nSix months later, Karakhan personally handed the Chinese a second version of the manifesto, one that did not include this remarkable offer. In fact, the Soviet authorities denied ever having made it. The railway offer had been included by \\\"mistake,\\\" they explained. The Soviets may have hoped the offer of the railway would generate an enthusiastic response in Beijing, leading to a Sino-Soviet alliance against Japan. The warlords in Beijing, closely tied to Japan, responded hesitantly. When the hoped-for alliance proved unattainable, the Russians withdrew their offer. In any event, traditional Russian interests and rights in China, including control of the Chinese Eastern Railway, were reaffirmed in a series of secret agreements made in 1924–1925.\\nThe manifesto came at a crucial time in the development of Chinese communism. It encouraged interest in Marxism and thus played a role in the founding of the Chinese Communist Party in 1921.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690168\",\"title\":\"Kripapureeswarar Temple\",\"body\":\"\\nKripapureeswarar Temple\\n\\nKripapureeswarar Perumal Temple (also called Arutkondanathar or Thiruvennainallur temple) in Thiruvennainallur, a panchayat town in Villupuram district in the South Indian state of Tamil Nadu, is dedicated to the Hindu god Shiva. Constructed in the Dravidian style of architecture, the present structure of the temple is believed to have been built during the Cholas period in the 10th century. The temple has received gracious endowments from the Chola queen Sembiyan Mahadevi. Shiva is worshipped as Kripapureeswarar and his consort Parvathi as Mangalambigai.\\nThe presiding deity is revered in the 7th century Tamil Saiva canonical work, the \\\"Tevaram\\\", written by Tamil saint poets known as the Nayanmars and classified as \\\"Paadal Petra Sthalam\\\". A granite wall surrounds the temple, enclosing all its shrines. The temple has a five-tiered rajagopuram, the gateway tower. The temple is closely associated with Sundarar, the saivite saint of the 8th century, who started singing his \\\"Tirumurai\\\" starting with \\\"Pitha Piraisudi\\\" verse in this temple.\\nThe temple is open from 6am - 11 am and 4-8:00 pm on all days except during festival days when it is open the full day. Four daily rituals and many yearly festivals are held at the temple, of which the Aadi star day celebrated during the Tamil month of \\\"Aaadi\\\" (June - July) for Sundarar and Panguni Uthiram festival during \\\"Panguni\\\" (March - April) with a float festival being the most prominent. The temple is maintained and administered by the Hindu Religious and Endowment Board of the Government of Tamil Nadu.\\nLegend.\\nAs per Hindu legend, the sages in Tharukavanam were very proud of their achievements and started doing a penance to destroy Shiva. Shiva absorbed all the evils from the penance and realising their mistake, the sages started their worship to Shiva. Shiva blessed them with their wishes and came to be known as Krupapureeswarar, the \\\"God who would bestow wishes of the devotees\\\". As per another legend, Sundarar, the famous Saivite saint and Nayanmar, was stopped by an old man from marrying at Thirukovilur. He showed a document indicating Sundarar was indebted to the old man for serving him for a lifetime. Sundarar called him \\\"Pitha\\\", a lunatic. The village elders after analysing the document, asked Sundarar to follow the old man. Sundarar followed the old mand and served him in his household. It was later revealed to him that Shiva appeared as the old man to test his servitude. Sundarar felt guilty of accusing Shiva, but Shiva revealed to him in a divine voice that he enjoyed the verse and requested him to sing verses starting with \\\"Pitha\\\".\\nSundarar started singing his \\\"Tirumurai\\\" with \\\"Pitha Piraisudi\\\" verse in this temple and later would go on to become one of the four most venerated saints of Saiva literature.\\nArchitecture.\\nThe exact year of building could not be ascertained from the inscriptions, but the inscriptions found in the Visalur temple indicate benevolent gifts to the temple from Raja Raja Chola I (984-1015 CE) and his successors. Based on the inscriptions, researchers point out that the temple had gardens, which were supposed to have both floral plants and fruit bearing trees. The temple has received gracious endowments from the Chola queen Sembiyan Mahadevi and is believed to have established various sculptures.\\nKripapureeswar temple is located in Thiruvennainallur, a village located from Villupuram on the Panruti- Thirukovilur road. The temple has a seven-tiered \\\"rajagopuram\\\", the gateway tower that pierces the rectangular wall that houses all the shrines. The sanctum houses the image of Kripapureeswarar in the form of Lingam, an iconic form of Shiva. There is an Ardha Mandap and a Mukha mandap, pillared halls leading to the sanctum. The first precinct has the images of Vinayakar, Murugan, Durga, Dakshinamurthy and Chandikeswara. The hall where the case was fought between the old man and Sundarar is believed to be the Panchyat Mandap located on the right side of the entrance.\\nCulture.\\nThe temple follows Saivite tradition. The temple priests perform the \\\"pooja\\\" (rituals) during festivals and on a daily basis. The temple rituals are performed four times a day: \\\"Kalasanthi\\\" at 6:00 a.m., \\\"Uchikalam\\\" at 11:00 a.m., \\\"Sayarakshai\\\" at 5:00 p.m., and \\\"Sayarakshai\\\" between 7:45  - 8:00 p.m. Each ritual has three steps: \\\"alangaram\\\" (decoration), \\\"neivethanam\\\" (food offering) and \\\"deepa aradanai\\\" (waving of lamps) for both Kripapureeswarar and Mangalambigai. There are weekly, monthly and fortnightly rituals performed in the temple. The temple is open from 6am - 12 pm and 4-8:30 pm.\\nAadi star day celebrated during the Tamil month of \\\"Aaadi\\\" (June - July) as Sundarar's birthday and Panguni Uthiram festival during \\\"Panguni\\\" (March - April) with a float festival being the most prominent festivals. There are other common festivals like Shivaratri, Vinayaga Chaturthi, Vijayadasami and Karthigai Deepam celebrated in the temple.\\nSundarar, an 8th-century Tamil \\\"Saivite\\\" poet, venerated Kripapureeswarar in ten verses in \\\"Tevaram\\\", compiled as the \\\"Seventh Tirumurai\\\". As the temple is revered in \\\"Tevaram\\\", it is classified as \\\"Paadal Petra Sthalam\\\", one of the 276 temples that find mention in the Saiva canon. In modern times, the temple is maintained and administered by the Hindu Religious and Endowment Board of the Government of Tamil Nadu.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690182\",\"title\":\"Socialists and Democrats (Italy)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690186\",\"title\":\"Francisco Adolfo Cabrera\",\"body\":\"\\nFrancisco Adolfo Cabrera\\n\\nFrancisco Cabrera is an Argentine engineer, born in the Mendoza Province. He became an engineer in electricity in the University of Mendoza, and headed the minister of economic development of Buenos Aires under Mauricio Macri from 2007 to 2015. Macri was elected president in 2015, and appointed him minister of production. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690198\",\"title\":\"Sarati the Terrible (1937 film)\",\"body\":\"\\nSarati the Terrible (1937 film)\\n\\nSarati the Terrible (French:Sarati, le terrible) is a 1937 French drama film directed by André Hugon and starring Harry Baur, George Rigaud and Jacqueline Laurent.\\nThe film's sets were designed by the art director Émile Duquesne.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690214\",\"title\":\"Pleasant Ridge, Alabama\",\"body\":\"\\nPleasant Ridge, Alabama\\n\\nPleasant Ridge, also known as Ridge, is an unincorporated community in Greene County, Alabama, United States. Pleasant Ridge is located on Alabama State Route 14, northwest of Eutaw.\\nHistory.\\nA post office operated under the name Pleasant Ridge from 1825 to 1918. On April 6, 1865, there was a skirmish between Union forces under the command of Col. John T. Croxton and Confederate forces under the command of Brigadier-General William Wirt Adams near Pleasant Ridge. Croxton was leaving Tuscaloosa, having burned the University of Alabama on April 4. The 6th Kentucky Cavalry Regiment and the 2nd Michigan Volunteer Cavalry Regiment engaged with Adams' forces.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690219\",\"title\":\"Joseph Baly\",\"body\":\"\\nJoseph Baly\\n\\nJoseph Baly (1824 - 1909) was Archdeacon of Calcutta from 1872 until 1883;\\nBaly was educated at Worcester College, Oxford, graduating BA in 1846 and M.A. in 1857. He was ordained deacon in 1847 and priest in 1848. He served curacies in Leicester and Falmouth. In 1854 he became Warden of St Thomas's College, Colombo. He later served as Chaplain at Allahabad, Sealkote and Simla before returning to Falmouth as its Rector (1870 - 1872). He was appointed a Fellow of the University of Calcutta in 1879. On his return from India he was Chaplain of Windsor Great Park from 1885 until 1906.\\nHe died on 6 November 1909.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690237\",\"title\":\"NAO (singer)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690239\",\"title\":\"Office of Dispute Resolution for Acquisition\",\"body\":\"\\nOffice of Dispute Resolution for Acquisition\\n\\nThe Office of Dispute Resolution for Acquisition (ODRA) is an Article I court that was established by the Federal Aviation Administration (FAA) pursuant to a statutory grant of authority as an independent tribunal to hear and decide both award protests and contract disputes subject to the Acquisition Management System (AMS) between Government contractors and the FAA.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690247\",\"title\":\"The Bad Companions\",\"body\":\"\\nThe Bad Companions\\n\\nThe Bad Companions is a 1932 British comedy film directed by J.O.C. Orton and starring Nor Kiddie, Renee Gadd and Wallace Lupino. The title is a reference to the 1929 novel \\\" The Good Companions\\\" by J. B. Priestley which was itself made into a film the following year.\\nIt was made at Welwyn Studios as a second feature by British International Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690262\",\"title\":\"Sebastian Saxe\",\"body\":\"\\nSebastian Saxe\\n\\nSebastian Saxe started his career in the City of Hamburg and since 2009 he is a Board Member of the \\\"Hamburg Port Authority\\\" (HPA). Hamburg´s Port is one of the largest ports in the European Union and among the biggest ports in the world. As CIO and CDO, Saxe has his main focus on the port’s IT strategy and the digital transformation of the company and its intermodal logistical chain.\\nCurrently, he is strongly involved in building the smartPORT of Hamburg. The main goal is to make use of all IT megatrends, like Internet of Things, Big Data, Cloud Computing and Mobility to maximize the efficiency in the logistic hub of Hamburg.\\nSebastian Saxe holds a PhD in mathematics and was one of the CIOs of the year in 2012. Under his guidance, the HPA won various prices in the field of IT technology and started its transformation in the digital age.\\nRecently, he was chosen as CIO of the year 2015 for small and medium-sized Enterprises in Germany.\\nLife.\\nSebastian Saxe lives in Hamburg, Germany, with his wife and two children.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690265\",\"title\":\"Alex Gerrard\",\"body\":\"\\nAlex Gerrard\\n\\nAlex Gerrard may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690293\",\"title\":\"Seaman Service Book\",\"body\":\"\\nSeaman Service Book\\n\\nSeaman Service Book (SSB) is a continuous record of a seaman’s service. This document certifies that the person holding is a seaman as per The International Convention on Standards of Training, Certification and Watch keeping for Seafarers (STCW), 1978, as amended from time to time. Seaman Book is one of the compulsory document for applying crew transit visas. The record of employment on board of a merchant ship (sea service) is recorded in a Seaman Service Book. Different countries issue to their seafarers the similar service book with different names i.e. Seaman Record Book, Seaman Discharge Book etc. In Pakistan Government Shipping Office issue this book under section 120 of Merchant Shipping Ordinance, 2001. It is mandatory for all seafarers serving onboard Ship, whether they are on the Minimum Safe Manning Certificate or not, to hold a \\\"Seaman Service Book and Seaman Identity Document (SID).\\nAffairs of Seamen in Pakistan.\\nThe Government Shipping Office looks after the affairs of seafarers under Merchant Shipping Policy of Pakistan which covers the following subject of merchant navy of Pakistan:\\nIssuance of SSB.\\nPakistan's merchant marine policy for jobs on vessel set in 2001, speaks as follows:\\nTo obtain employment on board a seagoing vessel every citizen of Pakistan is entitled to acquire a Seaman Service Book (SSB) in accordance with the provision of Merchant Shipping Ordinance 2001 subject to fulfilling the requirements as prescribed under the Rules issued from time to time, by the Government of Pakistan. No seaman can be engaged at any port of Pakistan, except service on coasting ships, unless until the seafarer is bonafide holder of the SSB. In case of foreign crew/merchant navy officers he must in possession of equivalent document e.g. equivalent discharge book issued by his own country. This is a prescribed certificate of identification with basic particulars and qualifications of the seaman duly registered with the Government Shipping Office. It is issued on Government sanctioned Form No. 20 by the Shipping Master under the provisions of Sections 120-138 of Merchant Shipping Ordinance, 2001. It is issued under a standard operative procedure (SOP), generally SSB contains the minimum information as given below:\\nOther pages of SSB contain columns for Seaman engagement and discharge record, promotion/advancement, company listing, summary of previous voyages, visa endorsement, official entries by the concerned issuing office etc. SSB is issued from National Database Registration Authority (NADRA) after following prescribed procedure and deposit of fee. The (SSB) is also liable to be cancelled, suspended or confiscated if a seaman is found to be involved in contravention of any local law.\\nIn Pakistan one of the sources of getting SSB in the category of ship's ratings/crew is conduct of courses from private sector institutions which are commonly known as GP or GP-III courses.\\nSSB Holders in the country.\\nPrior to 1990 Pakistan government was practicing the policy for issuance of CDC (now SSB) only to a certain number per year in order to ensure the availability of job to each and every seamen holding CDC under roster system. Earlier, there was just one government-owned institute, namely Pakistan Marine Academy, however, after promulgation of Merchant Shipping Ordinance (2001) institutes in private sector have allowed to offer training to semen for issuance of SSB according to their qualification and out of 10 approved institutes, five are actively conducting their business of training and education in accordance with STCW Convention. This has increased number of SSB holders in Pakistan in access of demand. About 40% of Pakistani seafarers are jobless for a long time. However, Government Shipping Office discloses that there are about 8,000 officers and 10,500 ratings duly registered as seamen. Out of this total number of registered seamens about 30% are employed on national as well as foreign ships. The employment on foreign ships is about 85% of the total seamen engaged in this field.\\nVisa on Passport to Pakistani SSB holder.\\nAfter mishap of 9/11 most of the European and North American countries placed restrictions for Pakistani seafarers for carrying with them passport and SSB along with the visa of that country before they could leave the Pakistan to join the ship if it was berthed in any European or North American country.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690302\",\"title\":\"Infinality bundle\",\"body\":\"\\nInfinality bundle\\n\\nInfinality bundle is a font and libraries bundle aimed at optimized text rendering under Linux. It comprises patched versions of FreeType (with optimized settings), fontconfig and cairo.\\nInfinality bundle replaces non-free fonts by free alternatives.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690321\",\"title\":\"Marco Aurélio Ribeiro Sousa\",\"body\":\"\\nMarco Aurélio Ribeiro Sousa\\n\\nMarco Aurélio Ribeiro Sousa (born 29 January 1995 in Porto) is a Portuguese professional footballer who plays for F.C. Paços de Ferreira as a goalkeeper.\\nFootball career.\\nOn 29 September 2014, Sousa made his professional debut with Paços de Ferreira in a 2014–15 Taça da Liga match against União Madeira.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690324\",\"title\":\"Walter, Alabama\",\"body\":\"\\nWalter, Alabama\\n\\nWalter is an unincorporated community in Cullman County, Alabama, United States. Walter is located on Alabama State Route 91 northeast of Hanceville.\\nHistory.\\nWalter is named for the son of the community's first postmaster. A post office operated under the name Walter from 1888 to 1905.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690329\",\"title\":\"Thomas Bewley\",\"body\":\"\\nThomas Bewley\\n\\nThomas Bewley, CBE, of St Thomas's Hospital, London, was president of the Royal College of Psychiatrists from 1984 to 1987.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690333\",\"title\":\"Chemical Pioneer Award\",\"body\":\"\\nChemical Pioneer Award\\n\\nThe Chemical Pioneer Award, established in 1966, is awarded by the American Institute of Chemists to recognize chemists or chemical engineers who have made outstanding contributions to advances in chemistry or the chemical profession. \\nRecent recipients.\\nSource: \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690335\",\"title\":\"Horse Brook (Queens)\",\"body\":\"\\nHorse Brook (Queens)\\n\\nHorse Brook, is a buried stream located in the neighborhood of Elmhurst in the New York City borough of Queens. Its historic course flows beneath Queens Center Mall, Rego Center Mall, LeFrak City, and the Long Island Expressway, before emptying into Flushing Creek in present-day Flushing Meadows-Corona Park. Flushing Creek is a tributary of the East River.\\nHorse Brook's headwaters originated near Kneeland Avenue and Codwise Place. Horse Brook was first mentioned at an annual town meeting in 1662, where it was voted that “whosoever has cats or dogs or hogs lying dead in any place to offend their neighbors they must bury them or throw them into the creek.”\\nThe only remaining traces of Horse Brook today are the mega-blocks on the map that avoided development in the early 20th century, remaining vacant until the 1960s. Examples include the Queens Center Mall and LeFrak City. A small park called Horsebrook Triangle in Elmhurst marks the approximate location of the buried stream.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690345\",\"title\":\"Muhammad Parvesh Shaheen\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690349\",\"title\":\"Forest of Dean by-election, 1887\",\"body\":\"\\nForest of Dean by-election, 1887\\n\\nThe 1887 Forest of Dean by-election was held on 29 July 1887 after the retirement of the incumbent Liberal MP Thomas Blake. The seat was retained by the Liberal candidate Godfrey Blundell Samuelson.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690382\",\"title\":\"2016 America East Conference Baseball Tournament\",\"body\":\"\\n2016 America East Conference Baseball Tournament\\n\\nThe 2016 America East Conference Baseball Tournament will be held from May 26–28. The top four regular season finishers of the league's six teams will meet in the double-elimination tournament to be held at Edward A. LeLacheur Park in Lowell, Massachusetts, the home park of UMass Lowell.\\nSeeding and format.\\nThe top four finishers from the regular season will be seeded one through four based on conference winning percentage only. The teams will play a double-elimination tournament. UMass Lowell, despite hosting the event, is not eligible to participate as it transitions from Division II. The River Hawks are expected to complete this transition and be eligible for championships in the 2017–18 academic year.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690392\",\"title\":\"Japonism (Arashi album)\",\"body\":\"\\nJaponism (Arashi album)\\n\\nJaponism is the fourteenth studio album of the Japanese idol group Arashi. The album was released on October 21, 2015 under their record label J Storm in three editions: a first press/limited edition, a Yoitoko limited edition, and a regular edition. The first press edition comes with an 84-page photo lyrics booklet and bonus DVD with the music video and making-of for the album's lead track, \\\"Kokoro no Sora\\\". The Yoitoko limited edition comes with a 32-page lyrics booklet, and the regular edition comes with a 36-page lyrics booklet. The album sold over 820,000 copies in its first week and topped the Oricon charts for two consecutive weeks. With more than 950,000 copies sold, the album was certified for Million by the Recording Industry Association of Japan (RIAJ).\\nAlbum information.\\nThe first press edition contains a CD with sixteen tracks and the Yoitoko edition contains a CD with seventeen tracks. The regular edition contains a CD with twenty tracks. The first press edition comes with an 84-page photo lyrics booklet and a bonus DVD with the music video and making of for \\\"Kokoro no Sora\\\", while the Yoitoko limited edition comes with a 32-page lyrics booklet, a bonus track, and an original talk track \\\"Arajapo Talk\\\". The regular edition comes with a 36-page lyrics booklet and four bonus tracks.\\nThe album jacket cover for the Yoitoko and regular editions are the same. The first press edition has a different jacket cover.\\nSongs.\\nIn \\\"Japonism\\\", Arashi expresses their interpretations of the \\\"wonderfulness of Japan\\\" in a unique way. It showcases Arashi's ambitious challenging spirit and continuous evolution. The lead track \\\"Kokoro no Sora\\\" was composed by Tomoyasu Hotei, who is based in London, with the theme \\\"Japan seen from the outside\\\" in mind. Described as a \\\"passionate and manly song\\\", Arashi sings to a fast-paced Hotei sound, where a battle between Arashi, Hotei, and Japanese instruments take place. The album includes a cover of Shonentai's \\\"Nihon Yoitoko Maka Fushigi\\\".\\n\\\"Japonism\\\" includes two of the group's previously released singles: \\\"Sakura\\\" and \\\"Aozora no Shita, Kimi no Tonari\\\". This album also includes fourteen new songs plus five of each member's solo songs. It also includes their previously unreleased song \\\"Furusato\\\" which they have sung regularly since 2010.\\n\\\"Sakura\\\" was used as the theme song for the drama \\\"Ouroborous\\\", starring actors Toma Ikuta and Shun Oguri. This is the first time Arashi has provided a theme song for a drama that did not star one of its members. \\\"Aozora no Shita, Kimi no Tonari\\\" was used as the theme song for the drama \\\"Yokoso, Wagaya E,\\\" which stars Arashi member Masaki Aiba.\\nPromotion.\\nTo support their new album, Arashi performed a live tour, ARASHI LIVE TOUR 2015 Japonism, performing at all the major dome stadiums in Japan. They had 17 performances beginning on November 6 at the Nagoya Dome, followed by Sapporo Dome on November 8, Kyocera Osaka Dome on November 26, Fukuoka Dome on December 17, and Tokyo Dome on December 23, 2015.\\nChart performance.\\nThe album debuted at number one on the Oricon daily album chart selling 412,826 copies upon its release and selling over 820,000 copies by the end of the week, topping the Oricon weekly album chart. The album maintained its number-one spot on the Oricon weekly album chart selling 56,890 copies in its second week and stayed in the top ten for six consecutive weeks. The album placed second on Billboard Japan's top album year-end list.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690394\",\"title\":\"Tomi\",\"body\":\"\\nTomi\\n\\nTomi may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690401\",\"title\":\"Solent Sea Steam Packet Company\",\"body\":\"\\nSolent Sea Steam Packet Company\\n\\nThe Solent Sea Steam Packet Company, later the Solent Steam Packet Company, operated ferry services between Lymington and Yarmouth on the Isle of Wight between 1841 and 1884.\\nHistory.\\nIn early 1841, the company purchased \\\"Glasgow\\\" from the \\\"Lymington, Yarmouth, Cowes and Portsmouth Steam Packet Company\\\", and after refitting, was deployed on the service between Lymington and Yarmouth, operating three or four passages a day.\\nIn March 1841 they entered into a contract with the Post Office for the conveyance of mail between Lymington and Yarmouth.\\nBy 1842, the company had acquired another vessel, \\\"Solent\\\", which was running from Lymington to Yarmouth, Cowes, Ryde and Portsmouth.\\nIn 1858, \\\"Red Lion\\\" was added to the fleet to handle additional traffic brought by the railway. The company changed its name to the \\\"Solent Steam Packet Company\\\" in 1861.\\nA second \\\"Solent\\\" replaced the first on 3 November 1863. \\\"Mayflower\\\" joined the fleet on 6 July 1866 had been built in Newcastle; she was tastefully fitted and comfortable. As well as plying to Yarmouth, she made excursion runs to Bournemouth, but was disposed of after 1878.\\nOn 1 July 1884, the London and South Western Railway bought out the Solent Steam Packet Company's fleet of two paddle steamers, \\\"Solent\\\" and \\\"Mayflower\\\", four horse and cargo boats, and other boats and property, paying £2,750 (£ in ).\\nShips.\\nThe vessels operated by the Solent Sea Steam Packet Company were:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690414\",\"title\":\"European consumer law\",\"body\":\"\\nEuropean consumer law\\n\\nEuropean consumer law concerns consumer protection within Europe, particularly through European Union law and the European Convention on Human Rights. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690419\",\"title\":\"James Chapelhow\",\"body\":\"\\nJames Chapelhow\\n\\nJames \\\"Jay\\\" Chapelhow (born 21 September 1995) is an English professional rugby league player for Widnes Vikings. His playing position is Prop Forward.\\nCareer.\\nChapelhow made his senior debut on loan at Whitehaven. In total he played 4 games for the Cumbrian club. His debut for parent club Widnes Vikings came in a Super League Super 8s qualifier against Leigh Centurions on 27 September 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690433\",\"title\":\"Zion, Alabama\",\"body\":\"\\nZion, Alabama\\n\\nZion is an unincorporated community in Pickens County, Alabama, United States. Zion is located along Alabama State Route 159, north of Gordo.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690440\",\"title\":\"Line 4, Ningbo Rail Transit\",\"body\":\"\\nLine 4, Ningbo Rail Transit\\n\\nLine 4 of Ningbo Rail Transit () is a rapid transit line under construction in Ningbo. It starts from Cicheng Town, Jiangbei District and ends near Dongqian Lake in Yinzhou District. Construction of Line 4 started in November 30, 2015.\\nRoute.\\nLine 4 starts from Cicheng Station in Cicheng Town in east-west direction as an elevated line. Then it turns south into the North External Ring Viaduct where it turns east-west. After reaching Jiangbei Avenue, it starts to turn underground and deviates from the viaduct to Zhuangqiao Railway Station where it turns south, crosses Yaojiang River and reaches Ningbo Railway Station. Then Line 4 goes along Changchun Road, Xingning Road until it reaches Canghai Road and become north-south again. After reaching Shounan Road it turns into southeast direction and reaches Dongqian Lake, its destination.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690449\",\"title\":\"Conus spiceri\",\"body\":\"\\nConus spiceri\\n\\nConus spiceri is a species of sea snail, a marine gastropod mollusk in the family Conidae, the cone snails, cone shells or cones.\\nThese snails are predatory and venomous. They are capable of \\\"stinging\\\" humans.\\nDescription.\\nThe size of the shell varies between 45 mm and 152 mm.\\nDistribution.\\nThis marine species occurs in the Pacific Ocean off Hawaii and Midway.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690456\",\"title\":\"Hypena namaqualis\",\"body\":\"\\nHypena namaqualis\\n\\nHypena namaqualis is a moth of the Erebidae family. It was described by Guenée in 1854. It is found in South Africa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690466\",\"title\":\"New Zealand NBL All-Star Five\",\"body\":\"\\nNew Zealand NBL All-Star Five\\n\\nThe National Basketball League All-Star Five is an annual National Basketball League (NBL) honour bestowed on the five best players in the league following every NBL season. The five-player team has been selected in every season of the league's existence, dating back to its inaugural season in 1982.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690476\",\"title\":\"Self-portrait with a sunflower\",\"body\":\"\\nSelf-portrait with a sunflower\\n\\nSelf-portrait with a sunflower is a 1632–33 self-portrait by Anthony van Dyck. It is now in the private collection of the Duke of Westminster.\\nIt was produced at the height of his fame, while he was 'principal Paynter in order to their Majesties' at the court of Charles I of England, who also knighted him. He wears the gold chain given to him by Charles and holds a large sunflower, whose symbolism is much debated.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690487\",\"title\":\"James Birley\",\"body\":\"\\nJames Birley\\n\\nJames \\\"Jim\\\" Birley (1928-2013) of the Maudsley Hospital, London, was president of the Royal College of Psychiatrists from 1987 to 1990.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690501\",\"title\":\"2016 NCAA Division I Baseball Tournament\",\"body\":\"\\n2016 NCAA Division I Baseball Tournament\\n\\nThe 2016 NCAA Division I Baseball Tournament will begin on Friday, June 3, 2016 as part of the 2016 NCAA Division I baseball season. The 64 team double elimination tournament will conclude with the 2016 College World Series in Omaha, Nebraska, starting on June 18 and ending on June 29.\\nThe 64 participating NCAA Division I college baseball teams will be selected out of an eligible 298 teams. Thirty-one teams will be awarded an automatic bid as champions of their conferences, and 33 teams will be selected at-large by the NCAA Division I Baseball Committee.\\nTeams will be divided into sixteen regionals of four teams, which will conduct a double-elimination tournament. Regional champions will then face each other in Super Regionals, a best of three game series to determine the eight participants of the College World Series.\\nCollege World Series.\\nThe College World Series will be held at TD Ameritrade Park in Omaha, Nebraska.\\nBracket.\\n\\\"Seeds listed below indicate national seeds only\\\"\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690522\",\"title\":\"Nigel Ogden\",\"body\":\"\\nNigel Ogden\\n\\nNigel Ogden is a British theatre organist, known for presenting and performing on the BBC Radio 2 programme \\\"The Organist Entertains\\\" for over 35 years.\\nOgden was born in Manchester, England, the son of a church organist, and had several years of piano lessons, before taking up the organ at the age of twelve. As a child, his family took him to Blackpool Tower Ballroom to hear performances on the Wurlitzer organ there, by Reginald Dixon.\\nHe studied to be a teacher, then worked as a sales demonstrator for an organ retail business in Hyde, Cheshire, where he later started his own business selling organs. From 1972, he started appearing on \\\"The Organist Entertains\\\", eventually taking over from Robin Richmond as presenter in March 1980.\\nOgden is also a composer and a touring musician, playing both theatre and church organs, and was the organist for the Channel 4 production of Denis Potter's \\\"Lipstick on Your Collar\\\".\\nHe has released a number of CDs.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690525\",\"title\":\"Patrick Montgomery\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690541\",\"title\":\"Green River Crib Dam\",\"body\":\"\\nGreen River Crib Dam\\n\\nThe Green River Crib Dam is a historic 19th-century dam on the Green River in western Guilford, Vermont. Built about 1811, it is a reminder of the modest industrial enterprises once conducted in the area using the water power it provided, and is one of the state's few surviving crib dams. It was listed on the National Register of Historic Places in 1995.\\nDescription and history.\\nThe dam is located in far western Guilford, upstream of the Green River Covered Bridge, which spans the Green River at the junction of Green River Road with Jacksonville Stage Road. The dam is about long, with a maximum height of , and spans the river in a semicircle open to the downstream side. The material of the dam mostly logs and rubble, with plank facing. The abutments of the dam are now a combination of stone and concrete, the wing wall on the east side extending downstream toward the bridge, where it formed part of the foundation of a now-destroyed mill.\\nThe earliest recorded documentation of a dam on this site dates to 1811, when Jonah Cutting is known to have a paper and linseed oil mill operating at the site. Some of the materials of the present dam may well date to this period. By 1856 the mill is only documented as being used for paper production. In 1869 a major flood apparently damaged the mill beyond repair, and it may also have damaged the dam; it washed out the bridge then standing just downstream. In 1871 Henry Stowe erected a lumber and grist mill on the site, which operated until 1918, when it was destroyed by fire. The dam has since then been maintained by private owners, forming a picturesque part of the small Green River village and a reminder of its modest industrial past.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690544\",\"title\":\"PS Mayflower (1866)\",\"body\":\"\\nPS Mayflower (1866)\\n\\nPS \\\"Mayflower\\\" was a passenger vessel built for the Solent Steam Packet Company in 1866.\\nHistory.\\nShe was built by Marshall Brothers in Newcastle and launched in 1866 and was used to expand the company services, offering a daily passage between Lymington and Portsmouth. \\nShe was acquired by the London and South Western Railway in 1884.\\nIn 1905 she was acquired by Joseph Constant in London and registered in Southampton. She was broken up in 1912.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690550\",\"title\":\"Lupton, Alabama\",\"body\":\"\\nLupton, Alabama\\n\\nLupton is an unincorporated community in Walker County, Alabama, United States. Lupton is located along Alabama State Route 5 northwest of Jasper.\\nHistory.\\nLupton is home to Lupton School, an elementary/middle school that is part of the Walker County Board of Education.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690551\",\"title\":\"Andrew Sims (psychiatrist)\",\"body\":\"\\nAndrew Sims (psychiatrist)\\n\\nAndrew Sims of St James's Hospital, Leeds, was president of the Royal College of Psychiatrists from 1990 to 1993.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690561\",\"title\":\"Guillermo Dietrich\",\"body\":\"\\nGuillermo Dietrich\\n\\nGuillermo Dietrich is an Argentine politician. He has worked as subsecretary of mass transit in Buenos Aires, under Mauricio Macri. He has worked in the Metrobus. Macri was elected president in 2015 and appointed him minister of Transport. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690582\",\"title\":\"Phoenix Gleitschirmantriebe\",\"body\":\"\\nPhoenix Gleitschirmantriebe\\n\\nPhoenix Gleitschirmantriebe () was a German aircraft manufacturer based in Würselen. The company specialized in the design and manufacture of paramotors in the form of ready-to-fly aircraft for the US FAR 103 Ultralight Vehicles rules and for the European Fédération Aéronautique Internationale microlight category.\\nThe company seems to have been founded in the early 2000s and gone out of business about 2008.\\nThe company produced the Phoenix Skywalker line of paramotors, powered by the Solo 210 and the Hirth F-33 engines. The aircraft was noted for the use of a paddle-bladed diameter four-bladed composite propeller, which allowed the design of a smaller cage assembly which improved ground transport portability and handling on take-off and landing.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690591\",\"title\":\"John Cox (psychiatrist)\",\"body\":\"\\nJohn Cox (psychiatrist)\\n\\nJohn Cox of the North Staffs Hospital Centre, Stoke-on-Trent, was president of the Royal College of Psychiatrists from 1999 to 2002.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690607\",\"title\":\"Mike Shooter\",\"body\":\"\\nMike Shooter\\n\\nMike Shooter of Nevill Hall Hospital, Abergavenny, was president of the Royal College of Psychiatrists from 2002 to 2005.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690618\",\"title\":\"Proteas Voulas B.C.\",\"body\":\"\\nProteas Voulas B.C.\\n\\nProteas Voulas is a Greek basketball club based in Voula, Athens. It was founded in 1980. Proteas has both men's and women's team which play in Greek national divisions. The women's team plays in A1 Ethniki Women (first-tier) and the men's team plays in Greek C Basketball League (fourth-tier). The team's colours are blue and red and the home stadium of the club is the Voulas Indoor Hall.\\nWomen's team.\\nProteas Voulas has won a Greek cup, in season 2012-13. That year, the club played in A2 Ethniki, so Proteas became the first club in Greece which won a Greek cup whereas it was playing in lower division. At the same year, Proteas promoted to A1 Ethniki and since then it plays in first-tier of championship.\\nMen's team.\\nSince 2015-16 season, the men's team of Proteas Voulas plays in Greek C Basketball League (fourth-tier).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690624\",\"title\":\"Khosro Agha hammam\",\"body\":\"\\nKhosro Agha hammam\\n\\nThe Khosro Agha hammam was a historical hammam in Iran. It was located in the Sepah street in Isfahan and belonged to the Safavid era.\\nIts dressing room was changed to a store in 1975 and was damaged heavily, but then it was repaired. After a while, the extension of Ostandari street (Now: Hakim street) was decided, but because of passing of the street from the hammam, this project was canceled. In 1979, some of neighborhood residents rumored that vice and harlotry took place in the hammam and the hammam should be destroyed. Their motive for this rumor was that they wanted the new street to be constructed and consequently the costs of their estates to be increased. Finally in 1980 a bomb exploded in the hammam and only some columns remained from Garmkhaneh (hothouse).\\nIn 1992, the supreme council of the city planning disapproved the construction of a new street because of the location of the hammam in the neighborhood. Finally on 12 April 1995 at 2 o'clock in the midnight some unidentified people attacked the Khosro Agha hammam and after that they made the guard unconscious, they destroyed completely this historical structure and stole its unique stone trough.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690631\",\"title\":\"One Day International matches results (1970–79)\",\"body\":\"\\nOne Day International matches results (1970–79)\\n\\nThe following tables provided the complete summary of One Day Internationals held between 1970-79.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690659\",\"title\":\"ALO (artist)\",\"body\":\"\\nALO (artist)\\n\\nALO is an Italian artist born in Ancona in 1981. He studied at the faculty of Industrial Design in Florence where he attended a fine art and drawing course. In the early stage of his career, ALO's oil paintings focused on female portraiture and on the representation of human loss and defeat which will become the landmark subject of his \\\"losers\\\" series. After few months, ALO decided to continue his activity working as a self-taught artist. This formation period led him to define his current style based on linear intersections and strong chromatic contrasts. Four years later ALO went back to his native city, Perugia, for his very first solo exhibition \\\"The underdogs show\\\". Influenced by the street art world, ALO decided to reproduce his indoor techniques in the outdoor spaces and started travelling around Europe, in particular to Rome, Milan, Berlin and then to Paris and London, the cities that offered him the biggest \\\"open air canvas\\\". ALO is currently based in London, where he had his first major solo show \\\"Hail to the loser\\\" at Saatchi Gallery, with which he started a continuous collaboration.\\nUrban Expressionism.\\nFascinated by the French and German Expressionism of the beginning of the 20th century, ALO chose the oil painting as the medium of his early artistic production.\\nSome of the aspects of his most recent pieces, such as the strong chromaticism and the stylized human figures, can be already seen in his very first body of work and they are inspired by the sharp, essential lines typical of the African art.\\nALO's style became the more and more refined and the use of the acrylic paint in his works is now predominant. ALO's portraits are the result of his deep study of the human figure, they show the artist's personal ideal of beauty and elegance together with his ability to understand and represent the complexity of the female nature.\\nInspired by a variety of artists, from Cezanne and Van Gogh to Kirchner and Picasso, ALO portrays people at the corners of the city streets, at the margins of society. His curious and open approach led him to the urban art world. ALO represents his emotional vision of the reality and transforms his characters into graceful figures full of elegance and solemnity.\\nALO brings the classic portraiture genre to the urban outdoor setting, taking distance from the predominant pop influences of the street art style. His characters are framed by colorful backgrounds, an important element that defines the emotional energy of the artwork and strengthen the idea of \\\"street painting\\\". ALO describes his own art as \\\"urban expressionism\\\".\\nPress.\\nRNext, La Repubblica degli Innovatori \\nInspiring City, Italian street artist ALO prepares for his solo show ‘HAIL TO THE LOSER’ at the Saatchi Gallery \\nComplex, 25 Shoreditch Street Artists You Need to Know \\nCombustus, Alo Street Artist, East London and Italy \\nArtribune, Dalle strade a Saatchi. Per Alo “è accaduto tutto molto in fretta \\nArt exhibitions.\\n2015 – \\\"Pop the Streets\\\", Saatchi Gallery, London\\n2014 – \\\"Hail to the loser\\\", Saatchi Gallery, London\\n2014 – \\\"Project M/6\\\", Urban Nation, Berlin\\n2014 – \\\"Collicola On the Wall\\\", Palazzo Collicola, Spoleto (Italy)\\n2013 – \\\"Spectrum\\\", Stolen Space Gallery, London\\n2013 – \\\"Winter group show\\\", Stolen Space Gallery, London\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690667\",\"title\":\"Pablo Avelluto\",\"body\":\"\\nPablo Avelluto\\n\\nPablo Avelluto is an Argentine journalist. He was appointed Minister of Culture by Mauricio Macri. He closed the Manuel Dorrego national institute.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690684\",\"title\":\"Sinderen\",\"body\":\"\\nSinderen\\n\\nSinderen is a village in the municipality Oude IJsselstreek in the Dutch province Gelderland. It used to be on the border of the former municipalities Gendringen and Wisch. The village is situated on a junction between Varsseveld (north), Dinxperlo (south) and Gendringen (south west).\\nThere used to be a Castle in Sinderen. The only thing reminding this is the Antonius Chapel. The meadow in from of the farm \\\"D'n Huusboer\\\" is surrounded by an old canal and basements of the castle are also to be found here. To the north of the village, a reformated church was built between 1884 and 1887. It nowadays is a Protestant church, and it's called the Keurhorster Church.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690701\",\"title\":\"Carmarthenshire County Council election, 1961\",\"body\":\"\\nCarmarthenshire County Council election, 1961\\n\\nAn election to the Carmarthenshire County Council was held in April 1961. It was preceded by the 1958 election and followed, by the 1964 election.\\nOverview of the result.\\nA close run election resulted in Labour increasing its majority by two after capturing an additional two seats. In addition, Labour took the majority of the aldermanic vacancies.\\nBoundary changes.\\nThere were no boundary changes at this election.\\nRetiring aldermen.\\nThe aldermen who retired at the election were ...\\nA number of retiring councilors stood down to allow retiring aldermen to be returned unopposed. These included D.T. Williams at Llangadog, who stood down in favour of Gwynfor Evans.\\nUnopposed returns.\\n39 members were returned unopposed, including six of the nine members in Llanelli town.\\nContested elections.\\n20 contests took place.\\nSummary of results.\\nThis section summarises the detailed results which are noted in the following sections.\\nThis table summarises the result of the elections in all wards. 59 councillors were elected.\\nWard results.\\nLlanon.\\nLabour had lost this ward in a by-election.\\nElection of aldermen.\\nIn addition to the 59 councillors the council consisted of 19 county aldermen. Aldermen were elected by the council, and served a six-year term. Following the elections, the majority of the aldermanic setas were taken by Labour.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690707\",\"title\":\"Massachusetts Woman Suffrage Association\",\"body\":\"\\nMassachusetts Woman Suffrage Association\\n\\nThe Massachusetts Woman Suffrage Association (MWSA) was an American organization devoted to women's suffrage in Massachusetts. It was active from 1870 to 1919.\\nHistory.\\nThe MWSA was founded in 1870 by suffrage activists Julia Ward Howe, Lucy Stone, Henry Browne Blackwell, and others. It was affiliated initially with the national American Woman Suffrage Association, which had been founded the previous year, and later became a chapter of the National American Woman Suffrage Association (NAWSA). One of its own affiliates was the Cambridge Political Equality Association.\\nThe MWSA lobbied for women to get the vote and the right to be officials of civic organizations such as school boards, educated people about women's rights, organized public demonstrations such as rallies and parades, and coordinated with suffrage associations in other states. Among the people active in the MWSA were physician Martha Ripley, social activist Angelina Grimké, reformer Ednah Dow Littlehale Cheney, and suffragist Susan Walker Fitzgerald.\\nIn 1892, the recent merger of several national suffrage associations and other factors prompted Alice Stone Blackwell and Ellen Battelle Dietrick to write a new constitution for the MWSA that would expand its capacities and funding base (e.g. by making it possible for the MWSA to receive bequests). The new MWSA was incorporated in December of that year. A decade later, in 1901, it merged with a smaller Massachusetts suffrage organization, the National Suffrage Association of Massachusetts. By 1915, the MWSA had over 58,000 members.\\nBetween 1904 and 1915, the MWSA was headquartered at 6 Marlborough Street in Boston's Back Bay, afterwards the headquarters of the Women's Municipal League of Boston and then the home of physician Louis Agassiz Shaw, Jr.\\nIn 1920, after the passage of the 19th Amendment to the Constitution gave women the vote, the MWSA became the Massachusetts League of Women Voters.\\nRecords pertaining to the history of the MWSA are held by Radcliffe College's Schlesinger Library.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690708\",\"title\":\"Clasado\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690779\",\"title\":\"Orphulella speciosa\",\"body\":\"\\nOrphulella speciosa\\n\\nOrphulella speciosa is commonly known as the slant-faced pasture grasshopper.\\nDistribution and habitat.\\n\\\"Orphulella speciosa\\\" occurs east of the Rocky Mountain region all the way to the Coast of the Atlantic. It also occurs in the southern region of Canada and in the northern region of Mexico. \\\"O. speciosa\\\" inhabits areas of tallgrass and mixed grass prairies, often preferring areas of shorter grass that are interspersed throughout the prairie plant matrix, including grasses that have been grazed upon by larger animals.\\nAt the southern end of the range (e.g., in Texas), adult \\\"O. speciosa\\\" are most abundant during August to October, with fewer adults present from May to December. In the northern extent of its range (e.g., New England) adult \\\"O. speciosa\\\" are most abundant during the months of July and August.\\nIdentification.\\n\\\"Orphulella speciosa\\\" has a very slanted face. The margins of the vertex of the head (the top of the head, located between the eyes) are less raised. The foveola (a small dent in the integument) is not distinct. The pronotum disks are found at the top of the first thoracic segment and are almost equivalent in width in the front and back. The principal sulcus on the pronotum, which is a ridge that cuts the middle of the outer integument, cuts the pronotum disk a little past the middle. The tegmina (the leathery, slender forewings extend to the end of the hind femora, and curve in towards the apex. The males' hind femora often vary in size from 1.5 mm to 3mm from the end of the femora. \\\"O. speciosa\\\" are sexually dimorphic; the females are much larger than the males. Female body sizes range from 16-21.5 mm with antennae that are 5-6mm long, tegmina that are 9-16mm long, and hind femora that are 9.5–12 mm long. Males range from 13-14mm in total length with antennae 4.5-6.5mm in length, tegmina, 101–13 mm in length, and hind femora are 8.5–10 mm in length. On lower end of the lateral lobes (the vertical sides of the pronotum of the males, there is a pale curved line. \\\"O. speciosa\\\" individuals exhibit much variation in color, especially combinations of green and brown. Females are often light green with a dark coloration on the vertex of the head. Females are occasionally brown and have a line on the sides of the pronotum. \\n\\\"O. speciosa\\\" looks very similar to a related species in the same genus, \\\"O. pelidna\\\". These two species can be identified from one another by looking at the fastigium (the part of the head that is located in between the eyes of the grasshoppers and their size. \\\"O.speciosa\\\" has a small median ridge on the fastigium and is much smaller than \\\"O. pelidna\\\". \\\"O. pelidna\\\" has a semicircular indentation that is set farther back than on \\\"O. speciosa\\\", and \\\"O. pelidna\\\" has a lateral carinae that is incised twice.\\nNymphal development.\\nLike all grasshoppers, nymphs of \\\"O. speciosa\\\" have incomplete metamorphosis: their nymphs are born looking similar to the adults, and they go through five instar stages (4 different molting events) until they reach the adult stage. First and second instars have patterns of green on their heads; however, the antennae are terminally expanded (they grow from the head outwards) . Second instars have antennae that are flat and are pointed at the end. The hind femora of the first and second instars appear tan and they have a green thorax. Instars II to V have filiform, or threadlike, antennae. Third and fourth instars have tan, brown, green, and gray body color patterns, and their hind femora appear fuscous (brownish-gray) .\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690796\",\"title\":\"Beloved Hymns\",\"body\":\"\\nBeloved Hymns\\n\\nBeloved Hymns is a studio album by Bing Crosby released in 1951 featuring eight hymns recorded with the Ken Darby Choir and organ accompaniment on May 6, 1949.\\nReception.\\nCrosby researcher Fred Reynolds said of the recording session at which all eight hymns were recorded that “They were all sung devoutly without any pretence of “performance”, but nevertheless gave added support to Martin Luther’s dictum that the devil should not have all the best tunes.” \\n\\\"Billboard\\\" reviewed some of the individual songs released as 78 rpm records.\\n\\\"What a Friend We Have In Jesus\\\" - Choir and organ support Bing ably as he delivers a beautiful hymn simply, straightforwardly and with deep warmth. \\n\\\"He Leadeth Me\\\" - Bing does this hymn with eminent strength and full affection for the chore. Should make for big sales in a quiet way for this Decca Faith disking. \\n\\\"O Lord, I Am Not Worthy\\\" - Bing rarely has sung better and with more feeling than he shows on this hymnal selection.\\nAlbum releases.\\nThe songs were featured on a 10” vinyl LP numbered DL 5351 and in a 4-disc 45rpm box set numbered 9-258. \\nLP track listing.\\nSide 1\\nSide 2\\nReferences.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690797\",\"title\":\"Germany women's national under-17 basketball team\",\"body\":\"\\nGermany women's national under-17 basketball team\\n\\nThe Germany women's national under-16 and under-17 basketball team, is controlled by the German Basketball Federation (), abbreviated as DBB, and represents Germany in international women's under-16 and under-17 (under age 16 and under age 17) basketball competitions. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690813\",\"title\":\"Colonus (spider)\",\"body\":\"\\nColonus (spider)\\n\\nColonus is a spider genus of the Salticidae family (jumping spiders). \\\"Colonus\\\" species are endemic to North and South America, ranging from New York to Argentina. All members of the genus have two pairs of bulbous spines on the ventral side of the first tibiae. The function of these spines is unknown. \\\"Colonus\\\" was declared a junior synonym of \\\"Thiodina\\\" by Eugène Simon in 1903, but this was reversed by Bustamante, Maddison, and Ruiz in 2015.\\nSpecies.\\n, the World Spider Catalog accepted 14 species of \\\"Colonus\\\":\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690831\",\"title\":\"Manchester, Alabama\",\"body\":\"\\nManchester, Alabama\\n\\nManchester is an unincorporated community in Walker County, Alabama, United States. Manchester is located along Alabama State Route 195 north-northeast of Jasper.\\nHistory.\\nManchester was founded as a lumber center. The Manchester Lumber Company owned a large amount of the surrounding timber land and built a school and Baptist church for the community. Much of the lumber produced in Manchester was used to make flatboats, which were used to transport coal. For a short time, the Manchester Coal Company mined coal in the area.\\nA post office operated under the name Manchester from 1907 to 1957.\\nIn popular culture.\\nManchester is listed as the hometown of the comic book superhero Impulse.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690832\",\"title\":\"PS Solent (1863)\",\"body\":\"\\nPS Solent (1863)\\n\\nPS \\\"Solent\\\" was a passenger vessel built for the Solent Steam Packet Company in 1863.\\nHistory.\\nShe was built by George Inman of Lymington and launched on 1 May 1863. She went to Southampton in June 1863 for the fitting of her engines by J. Hodgkinson. She undertook her trial trip on 29 October 1863 from Lymington to Stokes Bay. \\nShe was acquired by the London and South Western Railway in 1884.\\nShe was disposed of around 1901.\\nReferences.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690837\",\"title\":\"Raja Mohammad\",\"body\":\"\\nRaja Mohammad\\n\\nRaja Mohammad is an Indian film editor, who works in the Malayalam and Tamil film industries.\\nCareer.\\nRaja Mohammed primarily edited Malayalam films during his early career and his first Tamil venture was through Kamal Haasan's production, \\\"Nala Damayanthi\\\" (2003). For his 20th film, Raja Mohammad won the National Film Award for Best Editing for his work in \\\"Paruthiveeran\\\" (2007), with the honour fetching him further opportunities to work on film projects in the Tamil film industry. He also won the Vijay Award for Best Editor for his work in M. Sasikumar's directorial debut, the period film, \\\"Subramaniapuram\\\" (2008).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690853\",\"title\":\"Coming Home (Sasha song)\",\"body\":\"\\nComing Home (Sasha song)\\n\\n\\\"Coming Home'\\\" is a song by German recording artist Sasha. It was written and produced by Sasha along with Robin Grubert and Alexander Zuckowski for his first compilation album \\\"Greatest Hits\\\" (2006). Released as the album's lead single, it reached the top ten of the German Singles Chart.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690855\",\"title\":\"Muhammad ibn Ishaq ibn Ibrahim\",\"body\":\"\\nMuhammad ibn Ishaq ibn Ibrahim\\n\\nMuhammad ibn Ishaq ibn Ibrahim (, died June 851) was a Mus'abid governor of Baghdad for the Abbasid Caliphate, from 850 until his death.\\nCareer.\\nMuhammad was the son of Ishaq ibn Ibrahim al-Mus'abi, a member of a collateral branch of the Tahirid family and the head of security (\\\"shurtah\\\") in Baghdad from 822 to 850. During his father's lifetime Muhammad had been sent to attend the court of the caliph in Samarra, where he entered into the service of the central government and acted as Ishaq's representative.\\nUpon the death of Ishaq in July 850, Muhammad succeeded him as chief of security of Baghdad; at the same time, by delivering the valuables in Ishaq's storehouses to the caliph al-Mutawakkil and his heirs al-Muntasir and al-Mu'tazz, he secured their favor and was given control over al-Yamamah, al-Bahrayn, Egypt and the Mecca Road as a reward. He also received Fars, but this appointment forced to deal with that province's governor, his uncle Muhammad ibn Ibrahim al-Mus'abi, who adopted a hostile attitude toward him. In response, Muhammad deposed his uncle from his governorship and procured his murder, and appointed his cousin al-Husayn ibn Isma'il al-Mus'abi to govern Fars instead.\\nMuhammad died in June 851, after which his positions in Baghdad and the Sawad were assigned to 'Abdallah ibn Ishaq ibn Ibrahim.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690858\",\"title\":\"Avraham Stern (politician)\",\"body\":\"\\nAvraham Stern (politician)\\n\\nAvraham Stern (, 30 October 1935 – 12 May 1997) was an Israeli administrator and politician. He served as a member of the Knesset for the National Religious Party (Mafdal) between 1996 and 1997. \\nStern was born in Haifa and obtained a B.A. He was Secretary General of the Bnei Akiva youth movement, Political Secretary of the Religious Kibbutz Movement, and also Deputy Chairman of the National Religious Party bureau.\\nAfter the general elections of 1996 he entered the 14th Knesset. He was a member from 17 June 1996 till his death on 12 May 1997 and was replaced by Nissan Slomiansky.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690859\",\"title\":\"Juan José Aranguren\",\"body\":\"\\nJuan José Aranguren\\n\\nJuan José Aranguren is an Argentine businessman. \\nBiography.\\nAranguren was born in the Entre Ríos Province, and became an engineer in chemistry in the University of Buenos Aires. He joined Shell Argentina in 1979, and became a directive from 1997 to 2015. He opposed the administration of president Néstor Kirchner, and won several cases against the state for the state-controlled prices. \\nHe was appointed minister of energy by Mauricio Macri in 2015. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690875\",\"title\":\"Demonstration 95\",\"body\":\"\\nDemonstration 95\\n\\nDemonstration 95 is the first demo by Norwegian neofolk band Naervaer, issued in 1995 independently. The demo was mixed by Bjørn Harstad and Hans K. Eidskard. It was recorded in Jailhouse Studios summer, autumn and winter 1995.\\nTrack listing.\\nTracks ending with * later appears on the EP Naervaer, issued in 1997.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690895\",\"title\":\"Francis Chalmers Crawford\",\"body\":\"\\nFrancis Chalmers Crawford\\n\\nFrancis Chalmers Crawford FRSE (1851-1908) was a Scottish stockbroker of fame as an amateur botanist and ornithologist. \\\"Saxifraga crawfordii\\\" is named after him. He served as President of the Scottish Microscopical Society.\\nHe was an avid collector of plant species, especially Bryophytes and Spermatophytes, notably in Perthshire and Orkney but representing almost all areas of the United Kingdom. He often worked in partnership with Harold Stuart Thompson. He also collected in Hungary, Switzerland and Sweden with his friend John Hutton Balfour, many of his specimens being in the Natural History Museum, London.\\nLife.\\nHe was born in North Berwick on 24 August 1851, the son of Adam Howden Crawford (1804-1882) of the Honourable East India Company, and his wife Margaret Buchan Chalmers (1821-1898).\\nHe attended Edinburgh Academy 1863-68 and then trained as a stockbroker.\\nHe retired in 1896 and began to focus fully on his botanical interests. In 1897 he became a Fellow of the Botanical Society of Edinburgh and demonstrated Botany at an academic level at the Royal Botanic Garden Edinburgh.\\nIn 1898 he was elected a Fellow of the Royal Society of Edinburgh. His proposers were Ramsay Heatley Traquair, Sir Isaac Bayley Balfour, James Geikie and John Chiene.\\nIn his final years he lived at 19 Royal Terrace, an impressive Georgian townhouse designed by William Henry Playfair on Calton Hill.\\nHe died on 9 February 1908. He is buried with his parents and sister near the centre of the original north 19th century extension to Dean Cemetery in western Edinburgh beneath a Celtic cross.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690899\",\"title\":\"The Man who Put the Germ in Germany\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690911\",\"title\":\"Saragossa, Alabama\",\"body\":\"\\nSaragossa, Alabama\\n\\nSaragossa is an unincorporated community in Walker County, Alabama, United States.\\nHistory.\\nSaragossa is likely named for the Zaragoza province of Spain. A post office operated under the name Saragossa from 1890 to 1967.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690922\",\"title\":\"Lucky Day (Sasha song)\",\"body\":\"\\nLucky Day (Sasha song)\\n\\n\\\"Lucky Day'\\\" is a song by German recording artist Sasha. It was written and produced by Sasha along with Robin Grubert and Alexander Zuckowski for his first compilation album \\\"Greatest Hits\\\" (2006). Released as the album's second single, it reached the top twenty of the Austrian and the German Singles Chart.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690930\",\"title\":\"Community Care\",\"body\":\"\\nCommunity Care\\n\\nCommunity Care is a British trade magazine for the social care industry published by Reed Business Information Ltd.\\nThe magazine is now entirely online, but until 2011 was a weekly publication with a circulation of 32,000, though less than 4000 were paid for. The last issue of its print edition appeared on 24 November 2011. It has a strong position in the jobs market for social workers, claiming up to 3,000 social care jobs are advertised each week.\\nIt runs an annual two day conference and exhibition under the title Community Care Live.\\nThe magazine won the PPA Business and Professional Magazine of the Year Award in 2003.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690931\",\"title\":\"Thirumuruganatheeswar Temple\",\"body\":\"\\nThirumuruganatheeswar Temple\\n\\nThirumuruganatheeswarar Temple (also called Thirumuruganpoondi temple) in Thirumuruganpoondi, a panchayat town in Tiruppur district in the South Indian state of Tamil Nadu, is dedicated to the Hindu god Shiva. Constructed in the Dravidian style of architecture, the present structure of the temple is believed to have been built during the Kongu Cholas period in the 10th century. Shiva is worshipped as Thirumuruganatheeswarar and his consort Parvathi as Avudainayagi.\\nThe presiding deity is revered in the 7th century Tamil Saiva canonical work, the \\\"Tevaram\\\", written by Tamil saint poets known as the Nayanmars and classified as \\\"Paadal Petra Sthalam\\\". A granite wall surrounds the temple, enclosing all its shrines. The temple does not have a rajagopuram, the gateway tower, a common feature in South Indian temples.\\nThe temple is open from 5:30 am – 12:45 pm and 3:30 - 8:15 pm on all days. Four daily rituals and many yearly festivals are held at the temple, of which the Brahmotsavam celebrated during the Tamil month of \\\"Maasi\\\" (February - March) for Sundarar and Mahashivaratri festival being the most prominent. The temple is maintained and administered by the Hindu Religious and Endowment Board of the Government of Tamil Nadu.\\nLegend.\\nSundarar is a famous Saivite saint and Nayanmar belonging to the 8th century. He has revered many Shiva temples in South India in his verses compiled as the Seventh \\\"Thirumurai\\\". As per Hindu legend, while arriving at this place, he took rest in a Vinayaka temple. He presumably forgot to get thoughts about Shiva and to test his devotion, Shiva sent his Bhutaganas to steal all his possessions. Sundarar prayed to Vinayaka in the temple who showed him to proceed towards the East. Sundarar, in his anger, sang about Shiva blaming him of not protecting his possessions. Shiva graced him with his presence at this place and restored all his possessions. Murugan killed the demon king Surapadma at the behest of the Devas. He incurred Brahmahatti Dosha for slaying the demon into two pieces. Muruga is believed to have dug a spring with his Vel, the spear and worshipped Shiva at this place. Since Muruga worshipped his father Shiva, the presiding deity came to be known as Muruganathaswamy.\\nArchitecture.\\nThe temple is believed to have been built by the Kongu Cholars, with 68 recorded inscriptions from the king Vikrama Chola I seen on the walls of the sanctum and around the precinct. \\nThirumuruganatheeswarar temple is located in Thirumuruganpoondi, a village located from Tiruppur on the Tiruppur- Avinasi road. The temple does not have \\\"rajagopuram\\\" as in other South Indian temples. All the shrines are housed in a rectangular enclosure measuring . The sanctum houses the image of Thirumuruganatheeswarar in the form of Lingam, an iconic form of Shiva facing West. There is an Ardha Mandap and a Mukha mandap, pillared halls leading to the sanctum. There is a shrine of Muruga facing South, towards the Shiva shrine. Since Muruga is believed to have used his weapon, the Vel, to dig the spring, he is seen without his weapon in the shrine, nor his vehicle peacock. The first precinct has the images of Vinayakar, Durga, Dakshinamurthy and Chandikeswara. The shrine of Avudainayagi is seen in the first precinct facing west. There are sculptural depictions of Sundarar on the walls showing him in three different emotions of anger, humiliation and happiness. There is a hall of Nataraja called Adavallan Sabha. There are three temple tanks associated with the temple - Shanmugha Theertham, Gnana Theertham and Brahmatheertham. There are images of Kalabhairavar and Lingothbhavar, which are considered architectural specimens of the Kongu Cholars.\\nCulture.\\nThe temple follows Saivite tradition. The temple priests perform the \\\"pooja\\\" (rituals) during festivals and on a daily basis. The temple rituals are performed four times a day: \\\"Ushakalam\\\" at 6:00 a.m., \\\"Kalasanthi\\\" at 8:00 a.m., \\\"Uchikalam\\\" at 12:00 p.m., and \\\"Sayaratchai\\\" at 5:00 p.m. Each ritual has three steps: \\\"alangaram\\\" (decoration), \\\"neivethanam\\\" (food offering) and \\\"deepa aradanai\\\" (waving of lamps) for both Thirumuruganatheeswarar and Avudainayagi. There are weekly, monthly and fortnightly rituals performed in the temple. The temple is open from 5:30 am – 12:45 pm and 3:30 - 8:15 pm.\\nBrahmotsavam celebrated during the Tamil month of \\\"Maasi\\\" (February - March) for Sundarar and Mahashivaratri festival being the most prominent. There are other common festivals like Thaipoosam, Annabhishekam, Kanthasashti and Karthigai Deepam celebrated in the temple.\\nSundarar, an 8th-century Tamil \\\"Saivite\\\" poet, venerated Thirumuruganatheeswarar in ten verses in \\\"Tevaram\\\", compiled as the \\\"Seventh Tirumurai\\\". As the temple is revered in \\\"Tevaram\\\", it is classified as \\\"Paadal Petra Sthalam\\\", one of the 276 temples that find mention in the Saiva canon. Out of the 276 temples that are revered by the three saints, Sundarar has exclusively visited 25 temples with this temple counting as one of them. Of the five thandavams performed by Shiva in different places, this place is counted as the one where he performed the Brahma Thandavam In modern times, the temple is maintained and administered by the Hindu Religious and Endowment Board of the Government of Tamil Nadu.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690942\",\"title\":\"Germán Garavano\",\"body\":\"\\nGermán Garavano\\n\\nGermán Garavano is an Argentine politician. He has worked in the Council of Magistrates of Buenos Aires, and the Council of Magistrates of the Nation since 2014. He wrote 15 books about laws. He was appointed minister of Justice by Mauricio Macri in 2015. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690949\",\"title\":\"Woman in the Dark (1952 film)\",\"body\":\"\\nWoman in the Dark (1952 film)\\n\\nWoman in the Dark is a 1952 American crime film directed by George Blair and written by Albert DeMond. The film stars Penny Edwards, Ross Elliott, Rick Vallin, Richard Benedict, Argentina Brunetti and Martin Garralaga. The film was released on November 15, 1952, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690958\",\"title\":\"Germany women's national 3x3 team\",\"body\":\"\\nGermany women's national 3x3 team\\n\\nThe Germany women's national 3x3 team is a national basketball team of Germany, governed by Deutscher Basketball Bund (DBB).\\nIt represents the country in international 3x3 (3 against 3) women's basketball competitions.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690964\",\"title\":\"The Temptation of Innocence\",\"body\":\"\\nThe Temptation of Innocence\\n\\nThe Temptation of Innocence: Living in the Age of Entitlement () is a 1995 book by the French philosopher Pascal Bruckner. Bruckner argues against contemporary trends of applying victimhood, real or imagined, to justify infantilisation, a lack of responsibility or even oppression of others. The book received the Prix Médicis essai. It was published in English in 2000.\\nReception.\\n\\\"Publishers Weekly\\\" wrote:\\nBruckner's European education, which he wears lightly; his unpreachy, aphoristic style; and his obvious delight in paradox save this book from the ranks of a tedious diatribe against permissiveness. Citings of Europe's philosophical and literary masters (Rousseau, Hegel, Nietzsche among many others) help Bruckner, who is French (this admirable translation is not, alas, credited), make the case that the modern individual, weakened by responsibilities of freedom too great to bear, finds freedom in weakness itself: the freedom from moral constraint. ... Bruckner should find a ready audience among philosophically inclined readers who bring a skeptical eye to contemporary trends and agree that freedom from responsibility is no freedom at all.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690965\",\"title\":\"Raja Annamalaipuram\",\"body\":\"\\nRaja Annamalaipuram\\n\\nRaja Annamalaipuram, known in short as R. A. Puram, is a neighbourhood of Chennai, India. Named after banker and philanthropist, S. Rm. M. Annamalai Chettiar who owned most of the property at one time, Raja Annamalaipuram extends along the northern banks of the Adyar River from Saidapet to the Bay of Bengal coast.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690967\",\"title\":\"Fura\",\"body\":\"\\nFura\\n\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690969\",\"title\":\"Four Books for Women\",\"body\":\"\\nFour Books for Women\\n\\nThe Four Books for Women (\\\"Nǚ sìshū\\\") was a collection of material intended for use in the education of young Chinese women. In the late Ming and Qing dynasties, it was a standard text read by the daughters of aristocratic families. The four books had circulated separately and were combined by the publishing house Duowen Tang in 1624.\\nThe four books are \\\"Admonitions for Women\\\" (\\\"Nǚjiè\\\") by Ban Zhao, \\\"Women's Analects\\\" (\\\"Nǚ lúnyǔ\\\") by Song Ruoxin and Song Ruozhou, \\\"Domestic Lessons\\\" (\\\"Nèixùn\\\") by Empress Xu, and \\\"Sketch of a Model for Women\\\" (\\\"Nüfan jielu\\\") by Lady Liu.\\nIn \\\"Admonitions\\\", Ban Zhou, China's foremost female scholar, expounds on general principles and philosophical points. In \\\"Women's Analects\\\", the Songs illustrate these principles with practical examples relevant to everyday life.\\nIn \\\"Model for Women\\\", Lady Liu retells the inspiring tales of various women in Chinese history. There are example of every kind of famous women from every period. Aside from Ban Zhao, there is also Liang Hongyu, who beat war drums in battle to encourage her husband, a Song dynasty general. Scholarship and sacrifice for nation and family are extolled.\\nBan Zhao's book was the most widely read of the four. She wrote that a woman should practice \\\"reverent submission\\\", respect for the three obediences and four virtues, and \\\"set her husband on a par with Heaven.\\\" Ban Zhao compared marriage to the Dao, with the husband as the yin and the wife as the yang. This is a more romantic view of marriage than anything found in Confucius.\\nBan Zhao's life story was more inspiring than her writing. She was a Han dynasty scholar who not only tutored an empress, but also completed an official history begun by her brother. Reformers in the sixteenth and seventeenth centuries often cited her to make the case for female education.\\nThe Four Books explicitly argues for such education. \\\"There were no wise and chaste women who were not created through education,\\\" as \\\"Domestic Lessons\\\" puts it. Regardless of content, the Four Books were designed to teach women to read, a controversial and progressive idea at the time. All four books were written by accomplished women, a point many editions underline by including biographical sketches and drawings of the authors in scholarly regalia.\\nThe \\\"four books\\\" of the title alludes to the four books of Neo-Confucianism compiled by Zhu Xi. This was a textbook used by those studying for the Imperial Examinations. Female learning is thus promoted as a realm on the same level as male learning. \\\"The is no doubt that women read both the \\\"Four Books for Women\\\" and the Four Books they were not supposed to read,\\\" according to modern scholar Dorothy Ko.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690977\",\"title\":\"Houari Ferhani\",\"body\":\"\\nHouari Ferhani\\n\\nHouari Ferhani (born February 11, 1993 in Koléa) is an Algerian footballer who plays for Algerian Ligue Professionnelle 1 side JS Kabylie and the Algeria national under-23 team.\\nCareer.\\nIn October 2015, Ferhani was selected as part of the Algeria's under-23 national team for the 2015 U-23 Africa Cup of Nations in Senegal.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690984\",\"title\":\"Eben S. Stearns\",\"body\":\"\\nEben S. Stearns\\n\\nEben S. Stearns (1819-1887) was an American educator. He served as the President of Framingham State University from 1849 to 1855, and as the Chancellor of the University of Nashville and President Peabody Normal School (which later merged with Vanderbilt University) from 1875 to 1887.\\nEarly life.\\nEben Sperry Stearns was born on December 23, 1819 in Bedford, Massachusetts. His father was a Congregational minister.\\nStearns enrolled at Harvard University in 1841, where he received a master of arts degree in 1845. He received a Doctor of Divinity and a Doctor of Laws from other universities.\\nCareer.\\nStearns began his career as a teacher in an all girls' seminary in Ipswich, Massachusetts. He went on to teach in West Newton, Newburyport, Massachusetts, and Portland, Maine.\\nStearns served as the President of Framingham State University from 1849 to 1855. \\nStearns served as the Chancellor of the University of Nashville and President of the Peabody Normal School from 1875 to 1887. Under his leadership, the school attendance grew from 13 to 179 students. In 1885, he authored \\\"Historical Sketch of the Normal College, at Nashville, Tennessee\\\".\\nPersonal life.\\nStearns was married, and he had children.\\nDeath and legacy.\\nStearns died on April 11, 1887 in Nashville, Tennessee. His funeral was held by an Episcopal pastor in Nashville, and he was buried in Bedford, Massachusetts.\\nMeanwhile, he was replaced as Chancellor of the University of Nashville and President of the Peabody Normal School by William H. Payne. A year after his death, in 1888, his portrait by Geo Drury was donated to the Peabody Normal College.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690985\",\"title\":\"The Mute Gods\",\"body\":\"\\nThe Mute Gods\\n\\nThe Mute Gods are a German and English band uniting Nick Beggs, Marco Minnemann and Roger King. Beggs approached King — with whom he had worked as part of Steve Hackett's band — about a collaboration in 2014, and Minnemann was chosen as the drummer after Beggs had toured with him as part of Steven Wilson's band.\\nOrigins and \\\"Do Nothing till You Hear from Me\\\".\\nTheir first album, \\\"Do Nothing till You Hear from Me\\\" is planned for release on 22 January 2016. Beggs wrote the majority of the album on tour in 2014, and has described it as \\\"a rather disgruntled rant at the dystopia we've created for ourselves and our children\\\". In addition to Minnemann, the album features some guest drummers, including Nick D'Virgilio and Gary O'Toole. In November 2015, the band released two videos — one of which being a 360 degree video — for the album's title track. According to Nick Beggs, \\\"Do Nothing till You Hear from Me\\\" was inspired by Dwight D. Eisenhower's warning about the potential rise of the military–industrial complex, as well as the geologist Phil Schneider, who made several claims regarding UFOs, before dying under mysterious circumstances.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48690997\",\"title\":\"Winnifred Paskall\",\"body\":\"\\nWinnifred Paskall\\n\\nWinnifred May Paskallwas a Missionary who came to India in 1946 through the Canadian Baptist Ministries.\\nWinnifred Paskall studied at the McMaster University from where she took a graduate degree in arts (B.A.) in 1935. Winnifred came to India in 1946 and served in women's development ministries and also became Principal of the Eva Rose York Bible Training and Technical School for Women in Tuni, Andhra Pradesh.\\nOne of the support groups for Winnifred Paskall included the First Baptist Church, Leamington which used to raise money for work in India.\\nIn 1978, the Canadian Baptist Ministries presented Paskall with a Service Pin in recognition of her 34 years' of missionary service in India.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691000\",\"title\":\"Rhoda Leonard\",\"body\":\"\\nRhoda Leonard\\n\\nRhoda Leonard (January 31, 1928 – October 21, 2015) was an infielder and outfielder who played in the All-American Girls Professional Baseball League. Listed at 5' 5\\\", 115 lb., Leonard batted and threw right handed. She was nicknamed 'Nicky' by her friends and teammates.\\nBorn in Somerset, Massachusetts, Leonard attended Somerset High School, where she graduated and later earned Somerset Athletic Hall of Fame honors in 1946. This prompted an invitation to an AAGPBL tryout and she made the grade, but she did not get much of a chance to play in her only season in the league.\\nLeonard joined the Fort Wayne Daisies in 1946 and was used sparingly at second base and outfield, collecting a batting average of .095 (2-for-21) in nine game appearances.\\nAfter baseball, Leonard married Edmund Linehan and had two children, Mark and Maggie. She then graduated from Bridgewater State University in Bridgewater, Massachusetts in 1950, and after raising her family served for many years as a teacher for the Norwood Public Schools system for a long time. Some of her most cherished teaching moments came during her years of work with students at St. Catherine's School.\\nFollowing her retirement, she became an avid golfer and member of the WGAM. She also was awarded a lifetime membership to the Walpole Country Club after serving as their club secretary for several years, while enjoying many years as a lifetime member of the AAGPBL Players Association.\\nThe AAGPBL folded in 1954, but there is now a permanent display at the Baseball Hall of Fame and Museum at Cooperstown, New York since November 5, 1988 that honors those who were part of this unique experience. Leonard, along with the rest of the league's girls, is included at the display exhibition.\\nNicky Leonard died in 2015 in Norwood, Massachusetts at age 87, following complications from Alzheimer's disease.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691011\",\"title\":\"Timeline of Le Havre\",\"body\":\"\\nTimeline of Le Havre\\n\\nThe following is a timeline of the history of the city of Le Havre, France.\\nReferences.\\n\\\"This article incorporates information from the French Wikipedia.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691023\",\"title\":\"Neurogenesis\",\"body\":\"\\nNeurogenesis\\n\\nNeurogenesis (birth of neurons) is the process by which neurons are generated from neural stem cells and progenitor cells. It plays a central role in neural development. Neurogenesis is most active during pre-natal development and is responsible for populating the growing brain with neurons. In mammals, adult neurogenesis has been shown to occur in multiple brain structures, including the dentate gyrus of the hippocampus and the olfactory bulb.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691027\",\"title\":\"Dormer Pramet\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691056\",\"title\":\"Adille Sumariwalla\",\"body\":\"\\nAdille Sumariwalla\\n\\nAdille Sumariwalla (Born January 1, 1958) is an Indian athlete and entrepreneur, popular for representing India at the 1980 Moscow Olympics. Sumariwalla competed as a 100m runner in several international competitions and at the Olympics. Currently he is the president of the athletics federation of India, and was elected as one of the members of council of IAAF at its 50th congress, thus becoming the first Indian to do so. He is also an entrepreneur and owns many media businesses in India, after tenures at some media companies including the American Media Company.\\nAthletics.\\nSumariwalla has been an athlete since his school days. He had set the men's 200m inter college record, at a mere 22.2 seconds and held it for 35 years, until recently Gaurang Amre broke the record, clocking at 21.7 seconds. He expressed gladness as the record was broken, but at the same time expressed concern over the time taken to break the record, quoting these records should be broken \\\"every two or three years\\\". He has won the national title in 100m sprint 11 times. Sumariwalla represented India at the 1980 summer olympics as a hundred-meter sprinter, in which he stood seventh in round one. He has been the Chef-de-Mission for Indian Contigent, for the 2014 Asian Games in Korea. Sumariwalla is of the opinion that there has been great increase in infrastructure for sports by the government in recent years, and has quoted that \\\"The day is not far when India can expect an Olympics Medal in Athletics\\\".\\nEntrepreneurship.\\nApart from athletics, Sumariwalla has been a part of many media businesses and corporations. He started his professional career with Tata Engineering and locomotive co. (now Tata Motors). He worked there for 15 years in various domains, after which he was appointed as the founding CEO of The Asian Age in 1994. He has since worked with many other corporations, notably Mid Day Multimedia and Clear channel. He was responsible for setting up the outdoor division at Mid Day in 1997, which then merged its tasks with Clear Channel. After leaving clear channel, after serving 8 years as a Chairman and managing director, he left the company and has later been on the board of Next Media Works and SE TransStadia Ltd. He is also a co-founder of Interspace Communications.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691063\",\"title\":\"Fura (food)\",\"body\":\"\\nFura (food)\\n\\nFura are millet dough balls eaten in Nigeria.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691070\",\"title\":\"SS Vera (1898)\",\"body\":\"\\nSS Vera (1898)\\n\\nSS \\\"Vera\\\" was a passenger vessel built for the London and South Western Railway in 1898.\\nHistory.\\nShe was built by the Clydebank Engineering and Shipbuilding Company and launched on 4 July 1898 by Mrs Dixon, the wife of the marine superintendent of the London and South Western Railway. She was deployed on services between Southampton, the Channel Islands and the north coast of France. She had accommodation for 80 first-class passengers in cabins, and an additional 80 first-class passengers in Pullman car style state rooms. Provision was also made for 50 second-class passengers in the after-end of the vessel in large cabins. \\nOn 15 July 1905 she stranded herself on the Black Rock, at Yarmouth off the Isle of Wight.\\nShe was acquired by the Southern Railway in 1923.\\nShe was disposed in 1933.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691101\",\"title\":\"Strange Music (album)\",\"body\":\"\\nStrange Music (album)\\n\\nStrange Music is an album by Anton LaVey released in 1994 through Amarillo Records. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691102\",\"title\":\"Ambitalk\",\"body\":\"\\nAmbitalk\\n\\nAmbitalk is the UK's largest PAMR radio network. It allows users to make private or \\\"group\\\" voice calls between vehicle based mobile units anywhere within its coverage area. It is a Trunked radio system operating in the VHF-Low frequency range and provides coverage in most of the main conurbations of the UK.\\nAmbitalk is owned by Maxxwave Ltd and operates using MPT1327 signalling with custom audio enhancements using Digital Signal Processing to give clearer audio quality than is normally available from MPT1327 or DMR/dPMR (which use the AMBE+2 codec).\\nThe UK Government has commissioned a UK Spectrum Usage and Demand report through the UK Spectrum Policy Forum (run by Department for Culture, Media and Sport). This report looks at all major communications solutions available across the UK and recognises that a significant number of two way radio users in the UK are now using Ambitalk as their preferred communications platform.\\nFurthermore Ambitalk was awarded in 2015 the FCS Gerald David OBE award for Innovation in Business Radio. This award is given by the FCS, the UK communications trade body for the innovation that has been demonstrated by Ambitalk which sets it apart from other PAMR solutions.\\nHistory.\\nThe concept of Public Access radio networks is not new, with many networks such as GEC National One and Band Three Radio being launched in the UK in the 1980s. These closed around 2003 when Dolphin Telecom entered administration for the final time.\\nThroughout the period of the 1980s to 2003 there had always existed smaller \\\"regional\\\" radio networks within the UK, with several of the larger networks operating a number of interconnected \\\"regional\\\" networks that formed a quasi-National operation. Wavelength and Fleetcomm, both now defunct are examples of these.\\nAmbitalk is a reincarnation of the Maxxwave regional network originally covering Rugby, Coventry and Warwick in the Midlands and later adding additional transmitter stations to cover areas of London and Birmingham.\\nAround 2013 this regional radio network, previously powered by Zetron and Fylde MPT1327 controllers on Band III was redeveloped, with custom controllers designed by Maxxwave and was redeployed to lower frequency bands. This gave considerable cost savings, allowing the network to rapidly expand.\\nCurrent state.\\nFrom 2013 onwards Ambitalk rapidly expanded, obtaining low cost national spectrum (due to the unpopular low frequency bands) and installing additional transmitter sites around the country.\\nIn 2015 Ambitalk has concentrated on resilience and reliability, having replaced all Landline connections between transmitter sites with private Microwave and in-band links, giving it complete independence in case of any major catastrophe.\\nAmbitalk therefore is the only publicly accessible resilient wireless two-way communications network in the UK with other networks such as Airwave Solutions being closed, only available to select user groups. Furthermore it has a considerable amount of spare capacity, made available through its low frequency operation (rather than operating in congested UHF spectrum like Airwave) so therefore is far less likely to suffer congestion issues in case of major incidents.\\nIt has now become a clear market leader, having been awarded the prestigious Gerald David OBE Award in Innovation by the FCS (Federation of Communications Specialists), in recognition of the advancements it has made in the Two way radio arena. It is mentioned by name in numerous Ofcom documents with respect to the huge advances made in spectrum efficiency.\\nLow frequency bands.\\nAmbitalk is unique not only because it is the UK's largest PAMR network but because it reverses the trend for modern wireless deployments to migrate to higher frequency bands.\\nHigher frequency bands are preferred for rolling out wireless communications systems due to the lack of interference from atmospheric disturbances, smaller antennas and more manageable coverage characteristics, vital when planning a Cellular network\\nLow frequencies on the other hand are traditionally plagued by electronic interference from numerous sources and are prone to being unusable for weeks on end during summer months due to interference from foreign transmitters due to Sporadic E propagation thereby giving poor quality signals.\\nAmbitalk has counteracted this Sporadic E interference by developing a Smart antenna array for low frequency bands. Such systems are normally only deployed in far higher frequency bands and the use of such a system in a low frequency band is somewhat unusual. However with these arrays it is possible to virtually eliminate interference from continental sources, thereby giving high availability of a level equal to or better than that possible with higher frequency bands.\\nAt a time when frequency spectrum is becoming increasingly scarce, with many of the traditional Business Radio frequency bands within the UK now being completely congested preventing any new two way radio deployments being made, the migration of users to these lower frequency bands is welcomed by the UK regulators such as Ofcom.\\nSee also.\\nAmbitalk website: \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691104\",\"title\":\"Index of plagiarism-related articles\",\"body\":\"\\nIndex of plagiarism-related articles\\n\\nThis is an index of articles about plagiarism. It includes articles about incidents and examples of plagiarism, but does not include links to biographies of plagiarists or alleged plagiarists.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691111\",\"title\":\"Marcos Peña\",\"body\":\"\\nMarcos Peña\\n\\nMarcos Peña is an Argentine politician and political scientist. \\nBiography.\\nMarcos Peña was born in March 15, 1977, in Buenos Aires. He made elementary school in the United States, as his parents lived abroad. He returned to Argentina and completed high school in Buenos Aires. \\nHe graduated of political sciences in the university Torcuato Di Tella. He was elected legislator for Buenos Aires in 2003, in the ticket of Front for a Country in Solidarity. He helped to created the Republican Proposal party, and headed the list of legislators in the 2005 elections. Mayor Mauricio Macri appointed him general secretary of Buenos Aires in 2007. Macri became president in 2015, and appointed him chief of the cabinet of ministers. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691116\",\"title\":\"Eugène Frot\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691145\",\"title\":\"Mount Fura\",\"body\":\"\\nMount Fura\\n\\nMount Fura is a mountain found on old maps where the gold mines and capital of the Monomotapa kingdom was located. It is possibly to be identified as Stanford (1896) with modern Mount Fura (Mount Darwin) in Mashonaland Central, Zimbabwe.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691154\",\"title\":\"Hapoel Ramat Gan Givatayim B.C.\",\"body\":\"\\nHapoel Ramat Gan Givatayim B.C.\\n\\nHapoel Ramat Gan Givatayim is a professional fan owned basketball club from Israel. The team was founded in 2011 by Hapoel R-G fans and is aiming to promote competitive sport and the culture of sport in the cities of Ramat Gan and Givatayim.In 2013-14 season the club finished 2nd on the Liga Artzit (3rd tier). Currently the team plays in Liga Leumit.\\nHistory.\\nPreviously known as Hapoel Ramat Gan was founded in 1957. In 1960 the team joined Ligat HaAl and changed its name to Hapoel Ramat Gan - Givatayim. However, at the end of the season team returned to Liga Leumit. In 1964, the team rejoined Ligat HaAl, after the season the team returned to Liga Leumit. Between 1966-1971 the team alternated divisions based on their records. In 1973, the team (then called for the group of Ramat Gan and Hapoel Ramat Gan - Givataim)returned to Ligat HaAl.\\nMajor players on those teams were Steve Kaplan calculate the best player of all players playing group, Hanan Keren, Or Goren, Avigdor Moskovitch and played in Mtazrhim and foreign as well: Steve Slachter, Neil marketed, Cliff Fondkstr Steve Malovik. In addition Aimno the group of senior trainers such as: Zvika Sherf Arie Maliniak but despite everything the group failed to win a championship. The group has finished second six times instead of daring Cup finals five times in the country but never managed to win Cup.\\nThe biggest surprise in 1984 was its win over Maccabi Tel Aviv in semi final state cup basketball after 80-76. However, the team lost to Hapoel Tel Aviv B.C. 73-79 in the championship game. The team folded in 1988 as a result of financial issues.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691166\",\"title\":\"Eoline, Alabama\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691177\",\"title\":\"Skiddle\",\"body\":\"\\nSkiddle\\n\\nSkiddle is a primary ticket outlet and events guide based in Preston dealing with event bookings, registrations, promotion and online ticket sales. The company has a turnover in excess of £15 million and over 25 employees.\\nHistory.\\nSkiddle was launched in May 2001 initially as a '\\\"What's on Guide\\\"' for Preston in Lancashire. It was incorporated as Skiddle Ltd in January 2006. In February 2014 Skiddle acquired new 5000 sq/ft premises in Longridge to support expansion.\\nIn August 2015 Skiddle launched an iOS app with mobile 'Box Office' functionality. The app works with Bluetooth printers and iZettle card-readers, allowing tickets to be purchased and printed closer to gig start times.\\nIn November 2015 the firm announced an intern program with the University of Salford, with the aim of giving students insights into the industry and to develop journalism skills.\\nIndustry involvement.\\nIn 2013 Skiddle announced their white-label ticket shop, already in use by Mixmag, would also be powering Debenhams' Box Office. During October 2015 Skiddle was chosen as the official ticket website for the BBC Radio 1Xtra Live event in Leeds.\\nSkiddle have sponsored a range of festivals, tents and gigs mainly in the UK.\\nCharity Work.\\nIn 2011 Skiddle announced their sponsorship of the charity Campaign Against Living Miserably (C.A.L.M). A checkout donation scheme on the Skiddle website has helped to raise money for several UK charities. A donation in 2015 for over £15,000 was presented to the Anthony Nolan trust with previous benefactors including Macmillan and the Disasters Emergency Committee.\\nAwards.\\nSkiddle has received a Hitwise Top 10 Award every month since 2008 and in August 2011 Skiddle was awarded the Hitwise Top 6 Award, making it one of the most viewed websites in the UK within its category.\\nIn March 2012, Skiddle was awarded 'Online Business of The Year' by the Lancashire Business Review's \\\"Red Rose Awards\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691199\",\"title\":\"Marya Cohn\",\"body\":\"\\nMarya Cohn\\n\\nMarya Cohn is an American screenwriter and director. \\nCareer.\\nIn 1994 Cohn directed the student short film \\\"Developing\\\". The film starred a then unknown Natalie Portman in her first film role. \\nIn 2013 Cohn began filming her feature film debut, \\\"The Girl in the Book\\\" starring Emily Vancamp in a semi-autobiographical film about a young book editor who is forced to work with a man who manipulated her as a teenager. Post-production for the film was completed via kickstarter The film premiered at the 2015 Los Angeles Film Festival. It was acquired by Myriad Pictures and given a limited release in December of 2015. \\nPersonal Life.\\nCohn is the daughter of talent agent Sam Cohn. \\nIn 1994 she married director Fraser Bresnahan.\\nIn 2011 she married Dutch director Tjebbo Penning.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691226\",\"title\":\"Colonus\",\"body\":\"\\nColonus\\n\\nColonus may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691247\",\"title\":\"1932 Centenary Gentlemen football team\",\"body\":\"\\n1932 Centenary Gentlemen football team\\n\\nThe 1932 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1932 college football season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691250\",\"title\":\"1933 Centenary Gentlemen football team\",\"body\":\"\\n1933 Centenary Gentlemen football team\\n\\nThe 1933 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1933 college football season. Paul Geisler was consensus All-America.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691257\",\"title\":\"1934 Centenary Gentlemen football team\",\"body\":\"\\n1934 Centenary Gentlemen football team\\n\\nThe 1934 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1934 college football season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691266\",\"title\":\"Wilhelm Neumann-Torborg\",\"body\":\"\\nWilhelm Neumann-Torborg\\n\\nWilhelm Neumann-Torborg (born 24 August 1856 in Elberfeld; died 31 December 1917 in Elberfeld) was a German sculptor whose works are still well-known.\\nBiography.\\nWilhelm Neumann-Torborg grew up in Wuppertal, Germany, the son of a school headmaster. He attended evening classes at the Royal Provincial Vocational School in Elberfeld, where he received his first lessons in drawing and painting.\\nHe spent the years 1874 and 1877 studying in Bad Kreuznach, in the sculpture workshop of the brothers Robert and Karl Cauer. He then further developed his craft in 1878 under Melchior zur Strassen in the Leipzig Academy of Visual Arts and until 1880 in the Academy of Arts, Berlin with Otto Knille and Fritz Schaper. In 1885, he won the Rome Scholarship from the Prussian Academy of Arts for his thesis, \\\"The Judgment of Paris\\\".\\nIn subsequent years, Neumann-Torborg lived and worked in Rome, where his wife, Emma Commichau, died after a short marriage. In 1892, the artist returned to Berlin. In 1917, he fell seriously ill and died during a visit to his hometown of Elberfeld. His tomb is located in the Lutheran cemetery in Wuppertal-Elberfeld.\\nWorks.\\nMany of Neumann-Torborgs works were destroyed during the Second World War.\\nPerhaps his best-known work, the fountain \\\"Faun and Nymph\\\", was commissioned by Baron August von der Heydt and originally stood in the park of his estate. It survived the Second World War unscathed and since 1909 has been on display in Bad Godesberg, Bonn. In 2013, it was restored and moved to a new location to permit greater public access.\\nAnother relief of Neuman-Torborg's, showing dancing youths with Bacchus, was installed in the Bad Godesberg city park in 2014. The sculpture includes a bench and a bust of Roman Emperor Marcus Aurelius Probus, and was recovered in pieces from the van der Heydt estate.\\nThe \\\"Elberfeld Poor Relief Monument\\\" was destroyed during the war, when the bronze figures were melted down for metal. In 2003, the granite pedestal of the monument was rediscovered during excavations at the Elberfeld Old Reformed Church, and placed on display in Blankstrasse, Wuppertal. In 2011, it was restored thanks to 24 private donations. The bronze figures were recast at the Kayser Art Foundry in Düsseldorf. The statue commemorates the inauguration, in 1853, of the Elberfeld system of poor relief, which was copied by many cities in subsequent years.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691296\",\"title\":\"Gustavo Santos (politician)\",\"body\":\"\\nGustavo Santos (politician)\\n\\nGustavo Santos is an Argentine politician. He was appointed minister of Tourism by Mauricio Macri.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691299\",\"title\":\"2016 Deutsche Tourenwagen Masters season\",\"body\":\"\\n2016 Deutsche Tourenwagen Masters season\\n\\nThe 2016 Deutsche Tourenwagen Masters season is the seventeenth season of the Deutsche Tourenwagen Masters, since the series' resumption in 2000. The season starts on 7 May at Hockenheim, and ended on 16 October at the same venue, after a total of nine events.\\nCalendar.\\nThe nine event calendar was announced on 30 November 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691313\",\"title\":\"Blues Pills (album)\",\"body\":\"\\nBlues Pills (album)\\n\\nBlues Pills is the debut studio album by Swedish rock band Blues Pills, released on July 25, 2014 by Nuclear Blast Records. The album consists of ten tracks including some re-recorded releases from previous EPs and a cover of the Chubby Checker song \\\"Gypsy\\\". The album consists of two singles \\\"High Class Woman\\\" and \\\"No Hope Left For Me\\\", both released with music videos by Nuclear Blast.\\nReception.\\nMany reviews praise the improvement on the re-recording of the tracks that had previously appeared on EPs. Reviews mention the sound and style comparing to 60s blues and 70s rock influences. The album received predominantly positive online professional reviews.\\nTrack listing.\\nAll songs written and composed by Elin Larsson, Dorian Sorriaux, Zach Anderson and Cory Berry.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691315\",\"title\":\"1923 Centenary Gentlemen football team\",\"body\":\"\\n1923 Centenary Gentlemen football team\\n\\nThe 1923 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1923 college football season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691322\",\"title\":\"1924 Centenary Gentlemen football team\",\"body\":\"\\n1924 Centenary Gentlemen football team\\n\\nThe 1924 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1924 college football season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691323\",\"title\":\"1922 Centenary Gentlemen football team\",\"body\":\"\\n1922 Centenary Gentlemen football team\\n\\nThe 1922 Centenary Gentlemen football team represented the Centenary College of Louisiana during the 1922 college football season. The nine game schedule was the longest in school history.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691333\",\"title\":\"Alex Furmansky\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691335\",\"title\":\"Penny Drue Baird\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691387\",\"title\":\"Ricardo Buryaile\",\"body\":\"\\nRicardo Buryaile\\n\\nRicardo Buryaile is an Argentine politician. He is a national deputy, and Mauricio Macri has appointed him minister of agriculture. \\nHe is of French-Lebanese descent.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691422\",\"title\":\"Sweet Things (Georgie Fame album)\",\"body\":\"\\nSweet Things (Georgie Fame album)\\n\\nSweet Things is the 1966 third album with the Blue Flames by Georgie Fame which reached No.6 in the album Top Ten in the UK. Following this album his band The Blue Flames was replaced with The Tornados.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691439\",\"title\":\"Yoslan Muñoz\",\"body\":\"\\nYoslan Muñoz\\n\\nYoslan Muñoz García (born ) is a retired Cuban female volleyball player, who played as a 0. She was part of the Cuba women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Ciudad Deportiva La Habana.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691444\",\"title\":\"Swann Chemical Company\",\"body\":\"\\nSwann Chemical Company\\n\\nThe Swann Chemical Company was an American chemical company started by Theodore Swann, described by one historian as \\\"a flamboyant Birmingham mogul and New South industrialist.\\\" Swann Chemical first operated a chemical manufacturing plant in Anniston, Alabama where PCBs were first made on an industrial scale after development of a new process under leadership of Theodore Swann. The plant was later bought by Monsanto Industrial Chemicals Co. in 1935. The plant, just west of Anniston, had around 1,000 employees.\\nOne historian wrote that, \\\"In many ways, the spirit of Swann Chemical became the corporate culture of Monsanto.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691467\",\"title\":\"China National Electronics Import & Export Corporation\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691484\",\"title\":\"Irina Donets\",\"body\":\"\\nIrina Donets\\n\\nIrina Donets (born in Amsterdam) is a retired Dutch female volleyball player, who played as a middle blocker. She was part of the Netherlands women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pallavolo Ravenna.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691500\",\"title\":\"2015 Ice Challenge\",\"body\":\"\\n2015 Ice Challenge\\n\\nThe 2015 Ice Challenge was a senior international figure skating competition held in late October 2015 at the Liebenauer Eishalle in Graz, Austria. It was part of the 2015–16 ISU Challenger Series. Medals were awarded in the disciplines of men's singles, ladies' singles, pair skating, and ice dancing.\\nEntries.\\nThe preliminary entries were published on 6 October 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691501\",\"title\":\"Game of Pawns (television series)\",\"body\":\"\\nGame of Pawns (television series)\\n\\n is a television game show based in Branson, Missouri about the goings on in a local pawn shop. The show stars; Justin Tranchita , Scott Velvet and Brian Roman. The plot of the show is a mix of a game show and reality television series as Scott and Brian buy unusual items at Branson Pawn using a trivia game to give patrons the chance to win more for their item all while Justin gets into trouble and goofs off making the days at Branson Pawn move along with a little more excitement. Dealing in everything from celebrity cars to antique guns, they never know just what – or who – will walk through the door next. At Branson Pawn, every day’s a gamble. Scott and Brian give each customer three questions. If they get two of the three right, the customer gets their price. If not, the house wins and Brian and Scott get their price. This show runs on the Discovery Channel globally as re-runs and had its debut in 2013 with eight original episodes.\\nGame of Pawns was produced by a production company based in Denver, Colorado that has produced many hit reality television shows such as; Cake Boss , Fixer Upper , Tough Love and many more. Jim Berger and Scott Feely were the executive producers for the series.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691506\",\"title\":\"Kate Tiller\",\"body\":\"\\nKate Tiller\\n\\nKate Tiller is reader emerita in English local history at Kellogg College, University of Oxford. Tiller specialises in the history of Oxfordshire, Chartism and Methodism.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691508\",\"title\":\"Frank L. Fraser\",\"body\":\"\\nFrank L. Fraser\\n\\nFrank L. Fraser (September 29, 1854 – July 29, 1835) was an American lawyer and politician.\\nBorn in Sacramento, California, Fraser moved to Wisconsin in 1860 and grew up in the town of East Troy, Walworth County, Wisconsin. He went to the Rochester Academy and the Whitewater Normal School. Fraser studied law in Racine, Wisconsin and was admitted to the Wisconsin bar in 1876. Fraser lived in Lake Beulah, Wisconsin and was also a farmer. He served on the Walworth County Board of Supervisors and was chairman of the East Troy Town Board. Fraser also served on the school board and was the board treasurer. Fraser was the postmaster for Lake Beulah. In 1895, Fraser served in the Wisconsin State Assembly and was a Republican. Fraser died at his home in East Troy, Wisconsin.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691526\",\"title\":\"Ahmed Almusawi\",\"body\":\"\\nAhmed Almusawi\\n\\nDr. Ahmed Almusawi is an Islamic scholar, philosopher, astrophysicist, and psychologist. He received a PhD in clinical psychology from Cairo University. In addition to his scientific research in the field of miracles in the Quran and Ahl al-Bayt in Islamic Studies University. Established many of the beliefs and Islamic jurisprudence lessons in addition to the scientific Interpretation of the Quran in Sweden and Europe. Ahmed works as a researcher in the in the university of Islamic studies in Egypt.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691529\",\"title\":\"Ruth Heerschap\",\"body\":\"\\nRuth Heerschap\\n\\nRuth van de Wel-Heerschap (born ) is a retired Dutch female volleyball player, who played as a middle blocker. She was part of the Netherlands women's national volleyball team at the 2001 Women's European Volleyball Championship and 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with AMVJ Amstelveen in 2002 and for GTI Nesselande in 2005.\\nHer father is Martin Heerschap (chairman), her sister Eva Heerschap (player) and brother-in-law Kristian van der Wel (player) were also involved in GTI Nesselande.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691545\",\"title\":\"Chiswick Asylum\",\"body\":\"\\nChiswick Asylum\\n\\nChiswick Asylum was an English asylum established by Edward Francis Tuke and his wife Mary as Manor House Asylum in Chiswick, in about 1837. It was continued by his son, Thomas Harrington Tuke (1826-1888), before moving to Chiswick House in 1892 and becoming the Chiswick House Asylum, where it was run by two of Thomas Tuke's sons.\\nManor House Asylum was begun by Edward Francis Tuke and his wife Mary in about 1837, who took a lease on Manor Farm House in Chiswick Lane, a late 17th century building. It was demolished in 1896.\\nThe 9th Duke of Devonshire rented Chiswick House to the brothers Thomas Seymour and Charles Molesworth Tuke (sons of Thomas Harrington Tuke) from 1892 to 1928, when it was home to 30-40 private patients, before he sold it to Middlesex County Council in 1929. The two wings that housed the patients were demolished in 1956, as were many of the outbuildings, so little trace of the asylum remains today.\\nNotable patients.\\nIn 1852, the Chartist leader Feargus O'Connor MP was declared insane after a scene in the House of Commons, and confined to Chiswick Asylum, where he remained until 1854, and died in 1855.\\nIn 1865, Rev William Cotton spent several weeks in the Manor House Asylum.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691548\",\"title\":\"Nicolas Gilsoul\",\"body\":\"\\nNicolas Gilsoul\\n\\nNicolas Gilsoul (born 5 February 1982) is a Belgian rally co-driver. He is the current co-driver for Thierry Neuville.\\nCareer.\\nAfter competing in regional events from 2000, Gilsoul made his international debut in 2003 when co-driving with Bruno Thiry. \\nGilsoul started working with Thierry Neuville in 2011 and competed in the 2011 Intercontinental Rally Challenge. Since 2012, they have been driving in the World Rally Championship.\\nNeuville and Gilsoul took their first WRC win in 2014, at Rallye Deutschland.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691561\",\"title\":\"Elbert Howard\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691572\",\"title\":\"List of video game webcomics\",\"body\":\"\\nList of video game webcomics\\n\\nMany webcomics have been influenced by video games and video game culture. Webcomics frequently poke fun at video game logic, the video game industry, and stereotypical behavior of gamers. The earliest video game webcomic was \\\"Polymer City Chronicles\\\", which started in 1995. However, 1998's \\\"PvP\\\" is seen as the origin of the genre, influencing various webcomics following it. A common trope in video game webcomics is to have the main characters sit on a couch, talking about the game they are playing.\\nIt is common for webcomics to exclusively use in-game art and speech bubbles, such as in sprite comics. The term gamics has been proposed by Nathan Ciprick in 2004 to refer to webcomics that consist entirely of video game graphics. Despite the fact that video game graphics are generally copyrighted, owners of the intellectual properties used have traditionally been tolerant.\\nOther.\\nShiftyLook, a former subsidiary of Namco Bandai, focused on reviving various Namco video game franchises between 2011 and 2014. The company originally did this through English language webcomics. ShiftyLook has released webcomics based on \\\"Dig Dug\\\", \\\"Dragon Spirit\\\", \\\"Klonoa\\\", and various other video games.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691585\",\"title\":\"Chunar stone\",\"body\":\"\\nChunar stone\\n\\nChunar stone is a kind of reddish or buff-colored, finely grained, hard sandstone quarried in the Chunar in the Mirzapur District of Uttar Pradesh, and widely used in the architecture of India.\\nNotable buildings and monuments carved from chunar stone include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691603\",\"title\":\"1973–74 Dumbarton F.C. season\",\"body\":\"\\n1973–74 Dumbarton F.C. season\\n\\nSeason 1973–74 was the 97th football season in which Dumbarton competed at a Scottish national level, entering the Scottish Football League for the 67th time, the Scottish Cup for the 79th time and the Scottish League Cup for the 27th time.\\nOverview.\\nThis was Dumbarton's second successive season competing in Division 1, the objective being to build on the previous season's results, and in particular to avoid any relegation worries. This was certainly achieved, and indeed by mid November there were hopes of European qualification by reaching 6th in the table. However the wins began to dry up which resulted in a slide down the league - but the final position of 10th was the best the club had achieved for over 70 years - and qualification for the Anglo-Scottish sponsored competition, the Texaco Cup the following season - although this was not to be, following a reorganisation of the qualification requirements.\\nIn the Scottish Cup, Dumbarton's interest was short-lived as they lost out to Arbroath in the third round.\\nIn the League Cup, with two wins and a draws from the six sectional games, Dumbarton qualified for the next stage in the competition, where Rangers awaited. A 6-0 thumping in the first leg was made the return fixture a formality, although losing by the odd goal in three was something of an achievement.\\nLocally, in the Stirlingshire Cup, Dumbarton tried hard to retain the trophy they had won in the previous season, but were to lose out to Stirling Albion in the final on penalties, after a drawn game.\\nInterestingly at the beginning of the season, Dumbarton embarked on a mini overseas tour - the first since 1922 - where they beat Spanish regional side CF Lloret.\\nResults.\\nAll results are written with Dumbarton's score first.\\nStirlingshire Cup.\\n - won on penalties\\n - lost on penalties\\nAppearances.\\nDumbarton used 26 players for the 43 League, Scottish Cup and Scottish League Cup matches, as detailed below. Willie Wallace was the only player to feature in every one of these matches.\\nReserve Team.\\nDumbarton competed in the Scottish Reserve League, and with 12 wins and 6 draws from 34 matches, finished 12th of 18 for the second successive season.\\nDumbarton again entered the Scottish Second XI Cup, and again reached the third round where Celtic were to prove too good on the day by four goals to none.\\nFor the first time, Dumbarton entered the Scottish Reserve League Cup, and after qualifying from their section with 4 wins and a draw from 6 games, they eventually lost out to Partick Thistle in a two-legged semi final.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691605\",\"title\":\"Silvia Kutika\",\"body\":\"\\nSilvia Kutika\\n\\nSilvia Kutika (Wilde, 5 de agosto de 1956) is an Argentine actress best known by her roles in \\\"soap operas\\\" such as: \\\"906090 Modelos\\\", \\\"Vidas Robadas\\\", \\\"Los Médicos de Hoy\\\" and \\\"De Carne Somos\\\". In 1996 she married actor Luis Luque (1956–).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691627\",\"title\":\"Maureen Staal\",\"body\":\"\\nMaureen Staal\\n\\nMaureen Staal (born ) is a retired Dutch female volleyball player, who played as a wing spiker. She was part of the Netherlands women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Vanilla VC Weert.\\nStaal suffered in 2000 and 2002 from a hernia. In 2000 she got it after lifting to many weights with training, and in 2003 after a qualification match against Italy for a Grand Prix tournament. Almost five years later she returned for her former club \\\"Taurus\\\" in 2007. She still has the hernia, but has learned to live with it.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691634\",\"title\":\"Shōnen Maid\",\"body\":\"\\nShōnen Maid\\n\\n is a Japanese manga series written and illustrated by Ototachibana. An anime television series adaptation will premiere in April 2016.\\nMedia.\\nManga.\\nOtotachibana launched the series in Enterbrain's \\\"B's-Log Comic\\\" magazine on 12 March 2008. An audio drama CD was included with the limited edition of the sixth volume in April 2013.\\nAnime.\\nAn anime adaptation was announced in the 32nd issue of \\\"B's-Log Comic\\\" in August 2015. The series will be directed by Yusuke Yamamoto and written by Yoshiko Nakamura, with animation by the studio 8-Bit. Kana Ishida provides the character designs for the anime. The series will premiere in April 2016 on TBS and BS-TBS.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691644\",\"title\":\"Atlantic University School Of Medicine\",\"body\":\"\\nAtlantic University School Of Medicine\\n\\nAtlantic University School Of Medicine (AUSOM) is an offshore private medical school located in the St. Lucia National Neurological and Mental Health Wellness Centre, a psychiatric hospital built on the Gros Islet Highway, in Rodney Bay by China in 2006. The school's offshore office is located in Island Park, New York. The school's application was approved by the Cabinet of St. Lucia in 2010, though as of 2014, the country did not have legislation in place for licensing and regulating medical schools.\\nPrograms.\\nThe primary educational programs at AUSOM consist of a 2-year premed course of study; a four-year medical curriculum, which leads to an Doctor of Medicine degree; or a six-year track which combines the premed and medical training. In addition to theoretical and academic training, students are required to complete 48 weeks of clinical clerkships. Students participate in community health clinics from their first year of study, offering services of basic health checks, diabetes screenings and basic eye examinations. The school accepts both Lucian and foreign students, offering a scholarship program for Lucian residents who qualify.\\nAccreditation.\\nAtlantic University School Of Medicine received received approval from the St. Lucian Cabinet in 2010. It is not accredited by the Caribbean Accreditation Authority for Education in Medicine and other Health Professions (CAAM-HP) as of 2015. It is listed in the FAIMER International Medical Education Directory (IMED) effective in 2010 with school ID #F0002374 and in the World Health Organization's World Directory of Medical Schools. By virtue of its listing in IMED, students graduating from AUSOM are authorized to take part in the United States Medical Licensing Examination three-part examinations. Those who pass the examinations are eligible according to the Educational Commission for Foreign Medical Graduates to register for and participate in the National Resident Matching Program (NRMP).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691665\",\"title\":\"Athanasius II of Jerusalem\",\"body\":\"\\nAthanasius II of Jerusalem\\n\\nAthanasius II (; 1229–d. 1247+) was the Greek Orthodox Patriarch of Jerusalem from 1231 to 1244. The Church of the Holy Sepulchre seems to have been largely in Athanasius' hands during the Latin control of Jerusalem. The Serbian Archbishop Sava (1174–1237) guested Athanasius twice in the Holy Land, and according to Serbian chronicles they were good friends. After the Latin retreat from Jerusalem in 1244, the Melkites (who were the majority of the south of the Latin kingdom) turned to Athanasius. Athanasius II was in negotiations with the Pope through friar Lawrence of Portugal in 1247; Innocent IV supported him against the Latin patriarch, Robert.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691680\",\"title\":\"Sri Thenandal Films\",\"body\":\"\\nSri Thenandal Films\\n\\nSri Thenandal Films is an Indian film production and distribution company based in Chennai. The company was founded in 1976 by Rama Narayanan. They have produced, released and distributed films across several regional film industries in India.\\nHistory.\\nRama Narayanan set up Sri Thenandal Films in 1976 in Chennai, and the studio has gone on to distribute over 750 films across India.\\nDuring the late 1990s and early 2000s, Sri Thenandal Films produced and distributed a series of devotional films directed by Rama Narayanan. The quick spate of releases titled after goddesses such as \\\"Palayathu Amman\\\" (2000), \\\"Nageswari\\\" (2001), \\\"Kottai Mariamman\\\" (2001) and \\\"Annai Kaligambal\\\" (2003), saw the studio garner an image of focussing solely on Hindu devotional subjects.\\nIn the 2010s, Murali Ramaswamy made a decision to actively purchase and distribute horror films, and experienced immediate success after \\\"Aranmanai\\\" (2014), \\\"Kanchana 2\\\" (2015), \\\"Demonte Colony\\\" (2015) and \\\"Maya\\\" (2015) all became profitable ventures. The success of the studio's investment in that particular genre prompted them to finance smaller budget horror films including \\\"Strawberry\\\" (2015) and \\\"Jackson Durai\\\" (2016), in attempt to capitalise on the popularity of the genre across Tamil Nadu.\\nThe studio also ventured into the production of stage plays in 2015, and financed \\\"Chillu\\\", a futuristic science fiction play, performed in Chennai.\\nSelected filmography.\\nDistribution.\\nThe following films are a list of film which were distributed by Sri Thenandal Films throughout Tamil Nadu, apart from their own productions:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691684\",\"title\":\"Dieter Kalka\",\"body\":\"\\nDieter Kalka\\n\\nDieter Kalka (born 25 June 1957, Altenburg) is a German writer, songwriter, poet, dramatist, musician, editor, translator and speech therapist.\\nBiography.\\nDieter Kalka started to study electrical engineering and mathematics at the Technische Universität Ilmenau in 1978. In 1980 he had to quit his studies because of the distribution and possession of illegal publications. He was a member of the folk band founded in 1978, \\\"\\\", in 1984 he founded the band (The happy Future of Dieter) with Folksongs and own texts together with Uwe Schimmel (Waldhorn), Uta Mannweiler (Viola) and himself Bandoneon. With this group she organized the illegal artists' meeting \\\"Ringel Folk\\\" in Wurzen where there was no censorship. The unauthorized promotional material for this and other actions he copied at the photo lab of Petra Lux.\\nDieter Kalka was \\\"the fiercest among the Leipzig song singers\\\". Since the mid-1980s he has worked as a freelance singer and was repeatedly participant of the Chanson days Kloster Michaelstein (GDR-open Chanson days in Monastery Michaelstein). He made samizdat productions in privat studio at Hubertus Schmidt 1987 and Peter Gläser 1988 and official at Studio Kölling (Leipzig 1989).\\nAfter collaboration with Werner Bernreuther mentoring in 1987 he received a professional certificate as a songwriter, won a prize at the (Chanson days in Frankfurt/Oder) and a Prize of the Leipzig Songwriter Workshop, which he later publicly returned as they wanted to dictate to him what song he should sing at the final concert. He has received several scholarships of Saxony and was for a time a member of the Independent Writers Association \\\"ASSO\\\" Dresden, the NGL/New Society for Literature, the Writers Association \\\"VS\\\" and the \\\"Förderkreis Freie Literaturgesellschaft Leipzig\\\".\\nDieter Kalkas first book publication was entitled \\\"Eine übersensible Regung unterm Schuhabsatz\\\" (An Over Sensitive Motionless Under the Heel) and released in 1987 as samizdat. In 1990 he prepared a project manager before the first . Within the Association of German writers he organized in 1995 in Leipzig, the German-Polish poets festival \\\"\\\". He translated Polish poetry. Sunken GDR reality is the subject of his \\\"Der ungepflückte Apfelbaum\\\" published in 1998. Kalkas texts have been published in German, Polish, Austrian, Canadian and Belarusian literary magazines.\\nKalka was twice in Belarus for Songwriter's Festival \\\"Bardentreffen\\\" and appeared with his Belarusian colleague Victor Shalkevich. At Saxon Literature Spring 2003, he dedicated his \\\"Freiheitslied Nr. 2\\\" (Freedom Song No.2) his Belarusian colleague Victor Shalkevich to make him hope for better times.\\nHe participated in the German-Polish poets steamer on the border river Oder and the in Breslau, Bad Muskau and Lwówek Śląski.\\nAt Poets steamer was the Poets wedding 1998 between Dieter Kalka and Zielona Góra fairy tale author Agnieszka Haupe at the Frankfurt Oderbrücke.\\nKalka joined with his bandoneon various programs on own songs, such as 1988 with the theme \\\"Noch habe ich die Freiheit zu lieben\\\" (I still have the freedom to love). Sometimes he sang at the songwriter-festival . His songs \\\"are not without bitter aftertaste. He puts his finger on compromises that each includes almost every day in life, or feel compelled to close. Former ideals are often forgotten. He wrote the lyrics for the folk opera and wrote about\\\" The revival of the East German singer-songwriter scene \\\". His concerts have taken him to Poland, Belarus, Czech Republic, Hungary, Switzerland, Austria and Denmark. His songs are on numerous CDs.\\nDieter Kalka works as a speech therapist and lives in Leipzig and Meuselwitz.\\nWorks.\\nTranslation.\\nDieter Kalka translated Polish poets: , Jan Strządała, , , Józef Baran, , , Agnieszka Haupe, Jolanta Pytel, , , , Grzegorz Stec, Jakub Malukow Danecki, Bohdan Zadura, Waldemar Michalski, Alekzander Rozenfeld and others. The poetry was published in , , the anthology \\\"Lubliner Lift/Lubelska winda\\\", , the anthologies \\\"Es ist Zeit, wechsle die Kleider\\\", \\\"Nach den Gewittern\\\" and at PortalPolen.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691695\",\"title\":\"Yesterday's Hero (disambiguation)\",\"body\":\"\\nYesterday's Hero (disambiguation)\\n\\nYesterday's Hero is a 1979 film\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691703\",\"title\":\"Akeem Hunt\",\"body\":\"\\nAkeem Hunt\\n\\nAkeem Bernard Hunt (born February 22, 1993) is an American football running back for the Houston Texans of the National Football League (NFL). He was signed by the New York Giants as an undrafted free agent in 2015. He played college football at Purdue.\\nCollege career.\\nDuring his college career, Hunt rushed for 2,035 yards on 371 carries with 11 rushing touchdowns, six receiving touchdowns and three kick return touchdowns.\\nStatistics.\\nSource:\\nProfessional career.\\nNew York Giants.\\nHunt signed with the New York Giants as an undrafted free agent in May 2015. He was released on September 2, 2015.\\nBaltimore Ravens.\\nHunt was signed to the Baltimore Ravens practice squad on October 13, 2015 and was released on October 20. \\nHouston Texans.\\nHunt was signed to the Houston Texans practice squad on November 2, 2015. He was promoted to the active roster on November 21.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691712\",\"title\":\"June (company)\",\"body\":\"\\nJune (company)\\n\\nJune is a home automation company based in San Francisco. Its first product is the June Intelligent Oven, a computer-based, Wi-Fi-enabled, app-connected countertop oven that employs machine learning and computer vision technologies to identify and cook food. The oven was created by “a team that brought the iPhone, the Fitbit, the GoPro, and Lyft to market.”\\nDesign.\\nThe oven is controlled by a click knob, 5-inch touch screen and connected app. Programmable and sensor-driven, the oven uses a built-in scale, core-temperature thermometer and a camera to suggest cooking time and temperature. The internal high-definition camera with a fisheye lens that is designed to recognize commonly cooked foods. The company claims the oven’s optical recognition can identify foods such as frozen pizza, bacon, Brussels sprouts, asparagus, and potatoes and can differentiate between different types of fish.\\nJune was founded by Matt Van Horn, CEO, and Nikhil Bhogal, CTO, and launched in June 2015.\\nVan Horn co-founded Zimride, which spun off the ride-sharing service Lyft. Bhogal designed the camera software for the iPhone’s first five generations and is listed as an inventor on multiple Apple camera software patents. June team members have worked on the Apple Watch, GoPro cameras and Fitbit fitness trackers as well. Michelin-starred chef Michael Mina is an advisor to June.\\n\\\"The Wall Street Journal\\\" said “the innovation found in the June Intelligent Oven is spectacular” and that “its technologies point the way to the future of cooking.”\\nHardware.\\nThe June Oven has dual-surround convection fans, digital TRIAC controllers, a GPU processor, a 2.3-gigahertz NVIDIA chip, and carbon-fiber heating elements. Ammunition Design Group aided with the industrial design of the June Intelligent Oven and Quanta Computer aided in the manufacturing.\\nSoftware.\\nThe iOS app displays a live-stream video of the inside of the oven and then sends a notification when the food is done. The iOS app also has “smart recipes” that illustrate cooking steps with videos and GIFs and communicate with the oven.\\nFunding.\\nJune received $7 million in Series A funding from the Foundry Group, First Round Capital, Lerer Ventures, and Founders Fund.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691717\",\"title\":\"Jordi Mas Castells\",\"body\":\"\\nProgressively he specialized in building wells which stopped villagers, particularly girls, the weakest ones in that society, from walking long distances to get water to drink for both their families and cattle. During his long stay in the country he built hundreds of them.\\nMas was also aware that health was one of the weaknesses of the region. Therefore, together with the Swiss doctor Giuseppe Maggi (1910-1988) he founded the hospitals of Tokombéré (1962), Zina (1970) and Mada (1978).In the latter, a heath reference center, he worked driving patients in a jeep-ambulance for many years, a job which enabled him to get to know the territory and, mainly, to be close to the 250,000 inhabitants living in the surroundings of the hospital, a vast area of about 200 kilometres including four countries: Cameroon, Chad, Niger and Nigeria.\\nIn 1988 when doctor Maggi died, Jordi Mas went on to school founding, as it was evident for him that people without education have neither hope nor future. As a result of this work in 1998 he opened the big professional school of Blangoua CEFAVIHAR (Educational centre for the improvement of life in rural areas), next to lake Chad. In this school, which also includes a student residence promoted by Mans Unides, about forty youngsters from the villages near the lake can learn mechanics, electricity, welding, carpentry, business managering, sewing, typing, computer studies... Besides there is a primary school with 500 pupils which was created thanks to the collaboration of some Catalan organizations grouped together in the charity Makary-Blangoua \\nIn recent years, Mas focused on the home FEMAK (\\\"Femmes de Makary\\\") (2008), a meeting point for women of all religions in the region of Makary to relate, learn and exchange experiences. They had sewing workshops, computers and vegetable gardens and classrooms where they received education about health, eating habits and cooking.\\nHis great knowledge of the region was crutial in other projects: accommodation for the teachers in Blangoua, the FEMAK home and the residence for charity workers in Makary (2008)... The last project he was keen to develop in 2009 was the growing of a seaweed rich in proteins called spirulina, which grows really well in areas like lake Chad.\\nLater life.\\nIn 2010 he fell ill and travelled to La Garriga to undergo the treatment which enabled him to go back to his home in Makary. However, he died on Thursday 18 November that year. He is buried in La Doma cemetery, in La Garriga.\\nJordi Mas died, but his work is fully alive helping the people near the lake. Nowadays, November 2015, the Italian Fabio Musi, a long term Cameroon resident, is responsible for the general coordination of the different parishes and schools. He does it from Maroua, the Far North Region capital city of the country. In the last two parishes where the priest from La Garriga worked, namely Makary and Blangoua, there are two native priests in charge.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691726\",\"title\":\"Elke Wijnhoven\",\"body\":\"\\nElke Wijnhoven\\n\\nElke Wijnhoven-Schuil (born ) is a retired Dutch female volleyball player, who played as a libero. She was part of the Netherlands women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. She also competed at the 2001 Women's European Volleyball Championship, 2005 Women's European Volleyball Championship and 2005 FIVB World Grand Prix. On club level she played with Metodo Minetti Vicenza.\\nPersonal.\\nWijnhoven married on 6 June 2009 the Dutch volleyball and beach volleyball player Richard Schuil. In March 2012, her daughter Lisa was born.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691732\",\"title\":\"The Legacy of the Blues Vol. 7\",\"body\":\"\\nThe Legacy of the Blues Vol. 7\\n\\nThe Legacy of the Blues Vol. 7 is an album by American blues pianist Memphis Slim which was recorded in 1967 and released on the Swedish Sonet label. \\nReception.\\nIn his review for Allmusic, Nathan Bush says \\\"Throughout the set, Slim is happy to lend the spotlight to his sidemen, working the 88s behind Eddie Chamblee's tenor solo on 'I Am the Blues' and Billy Butler's guitar on 'Ballin' the Jack.' Even in these situations however, the pianist is dazzling and his commentary always worth paying attention to.\\\"\\nTrack listing.\\n\\\"All compositions by Peter Chapman\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691759\",\"title\":\"CRAX Commander\",\"body\":\"\\nCRAX Commander\\n\\nCRAX Commander is a dual pane, Orthodox File Manager for Mac OS X, created in Objective-C. The app is currently developed by Soft4U2 (Marcin Słowik) and is one of the new generation of Apple Finder replacement apps.\\nCRAX Commander has a familiar Finder-like look so it’s instantly usable as a replacement. The key to a replacement utility is the feature list. CRAX Commander embraces it with a wide ranging slew of features and functions.\\nCRAX Commander was created using ideology of dual-pane user interface which is well-known from the world of the Windows applications like Total Commander. Ideology of this kind of apps assumes offering multi-tab browsing user interface with features enabling of advanced search of files or folders, comparing files and folders, navigation in archive files and enabling multi-rename tool with regular expression support. This kind of applications also include a built-in FTP client, working with local and network drives and built-in file editor and viewer.\\nCRAX Commander is app for Mac power users and offers user configurable keyboard shortcuts, built-in text editor with sync coloring, full user interface customization including fonts and colors, archive support and FTP, SMB, AFP, SSH, and sFTP built in.\\nApplication is available in demo and paid version and provides a plethora of tools that will help end user to manage files and folders while taking advantage of the dual-pane interface design.\\nHistory.\\nFrom 2013 until 2014 CRAX Commander was called Crax and was developed by Ewa Serafin. The name and icon for application was changed in 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691782\",\"title\":\"Jorge Lemus\",\"body\":\"\\nJorge Lemus\\n\\nJorge Lemus (born June 9, 1948, Buenos Aires) is an Argentine politician. He was appointed minister of health of Buenos Aires in 2007, under mayor Mauricio Macri. He resigned in 2012, but stayed in the Republican Proposal party. Macri won the 2015 elections, and appointed him national minister of health.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691793\",\"title\":\"Podlubnik\",\"body\":\"\\nPodlubnik\\n\\nPodlubnik is part of a local community Podlubnik/Stara Loka in Municipality of Škofja Loka in the Upper Carniola region of Slovenia.\\nFun fact: Podlubnik street has the second highest number of house numbers (356 house numbers) in whole Slovenia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691795\",\"title\":\"Anita Krajnc case\",\"body\":\"\\nAnita Krajnc case\\n\\nThe Anita Krajnc case refers to the case of Toronto resident Anita Krajnc who has been charged with criminal mischief for giving water to pigs in a slaughter truck on the way to Fearman's Pork Inc. slaughterhouse, located in Burlington, Greater Toronto Area. This incident occurred on June 22, 2015.\\nBackground.\\nAnita Krajnc is co-founder of Toronto Pig Save, which was started in 2010. A \\\"Huffington Post Canada\\\" story titled \\\"Meet The Compassionate Canadians Who Give It All To Animals\\\" includes Krajnc as one of the five chosen and describes her as having \\\"a mighty heart to bear witness to any creature's grim odyssey to death\\\" for \\\"she has been standing \\\"vigil\\\" at the gates of death for thousands of pigs\\\".\\nKrajnc holds a PhD in Political Science from the University of Toronto. Her doctoral thesis was based on the role of scientific knowledge and public education in building international environmental regimes. Krajnc has been Assistant Professor at Queen's University. She has also been a media democracy activist and a writer. She was an aide to Charles Caccia former Minister of Environment and Climate Change.\\nIncident.\\nThe related incident occurred on June 22, 2015, outside Fearman's Pork Inc. slaughter house in Burlington. The protest was undertaken by Toronto Pig Save, a group to which Krajnc, who is described as an animal rights activist, belongs. Krajnc and her group were providing water to pigs in trucks carrying them to slaughter, as they stopped at a traffic island at the intersection of Appleby Line and Harvester Road, through the vents on the sides of the truck. As they were doing so, a driver of one of the trucks, Jeffrey Veldjesgraaf, got down and went to her and asked her to stop. Krajnc recited a verse from the biblical Book of Proverbs: \\\"If they are thirsty, give them water\\\". In response Veldjesgraaf called her \\\"a dumb fricking broad\\\" and told her that the pigs weren't human. Krajnc asked Veldjesgraaf to be compassionate, Veldjesgraaf said he would call the police, Krajnc suggested that she would call Jesus. Veldjesgraaf asked what she was feeding the pigs, Krajnc said that it was water, he told her that he wanted proof, Krajnc suggested that he trust her and offered him a sample. Veldjesgraaf asked her to desist from feeding again. Krajnc said that if the pigs were thirsty they would be fed again. Veldjesgraaf threatened to knock the bottle off, Krajnc retorted that she would charge him with assault. The driver then walked off, boarded the truck and drove on. This interaction was filmed by Krajnc's associates. On June 23, 2015, Eric van Boekel, from whose farm the pigs were brought, filed a case against Krajnc. On September 9, 2015, Krajnc was charged with criminal mischief, the penalty for which ranges from a fine to 10 years in prison. In response, the group arranged a vigil on September 24. On October 14, 2015, Krajnc appeared in court for an arraignment, to have charges read to her. In the context of the case \\\"The Daily Telegraph\\\" reports that under Canadian law pigs are considered property and can be transported without food and water for 36 hours. Krajnc has said that she would refuse to pay any fine and that she was willing to suffer imprisonment.\\nVeldjesgraaf stated that the activists were entitled to protest, his objection was to their touching his truck (which may jeopardise their own safety) or feeding the pigs. He considers the charges justifiable. He claims that the pigs were loaded an hour before the incident and it was unlikely they were dehydrated. \\\"Metro\\\" printed a rebuttal from Bob Comis, a former breeder who stated that the pigs as seen in video of the incident showed symptoms of \\\"severe heat stress\\\". Eric van Boekel in support of his complaint said, \\\"They don't have the right to involve other people's property and they don’t have the right to protest illegally\\\". He claimed that rules set by pig breeders regarding food and water were followed by his operations. A story reporting the incident noted that breeding facilities owned by the same Eric van Boekel were charged with polluting the Thames river in April 2007 and fined $120,000.\\nKrajnc has stated that treating living animals such as pigs as property – \\\"no different than a toaster\\\" – is the focal issue of the matter and that being compassionate to others should not be considered criminal.\\nReactions.\\n\\\"The Guardian\\\" reports that many online petitions have been created in defence of Krajnc. By November 30, 2015, the petition on Care2 called \\\"Compassion Isn't a Crime\\\" had 125,500 signatories, while another on Change.org asking the Ontario Court of Justice to drop charges against her, had over 24,600. \\\"The Daily Telegraph\\\" considers the case \\\"a cause célèbre for animal welfare activists\\\". Another story reports that the case has caused Krajnc to \\\"become a minor celebrity on social media.\\\" Instagram and Facebook pages have been created.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691833\",\"title\":\"Alina Albu\",\"body\":\"\\nAlina Albu\\n\\nAlina-Speranta Albu (born ) is a retired Romanian female volleyball player, who played as a middle blocker. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Dinamo Bukarest.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691841\",\"title\":\"Elena Butnaru\",\"body\":\"\\nElena Butnaru\\n\\nElena Butnaru (born ) is a retired Romanian female volleyball player, who played as a middle blocker. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. She also competed at the 2001 Women's European Volleyball Championship and 2005 Women's European Volleyball Championship. On club level she played with Pallavolo Palermo.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691843\",\"title\":\"Streptomyces coacervatus\",\"body\":\"\\nStreptomyces coacervatus\\n\\nStreptomyces coacervatus is a Gram-positive bacterium species from the genus of \\\"Streptomyces\\\" which has been isolated from the intestinal tract of the common pill-bug Armadillidium vulgare in Chiba City in Japan.\\nReferences.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691848\",\"title\":\"Lasse Lehmann\",\"body\":\"\\nLasse Lehmann\\n\\nLasse Lehmann (born May 20, 1996) is a German footballer who plays for Stuttgarter Kickers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691850\",\"title\":\"Maria Chivorchian\",\"body\":\"\\nMaria Chivorchian\\n\\nMaria-Eugenia Chivorchian (born ) is a retired Romanian female volleyball player, who played as a universal . She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Amici Bacau.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691851\",\"title\":\"Booze Traveler\",\"body\":\"\\nBooze Traveler\\n\\nBooze Traveler is an American travel television series hosted by Jack Maxwell. The premise involves Maxwell traveling the world to partake in international (and domestic) alcohol-based customs, as well as observe the culture of specific areas in a more general sense. It premiered on Travel Channel on November 24, 2014. Season 1 consisted of 15 episodes, and Season 2 will have 16 episodes. The status of Season 3 is currently unknown.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691853\",\"title\":\"Andreea Constantinescu\",\"body\":\"\\nAndreea Constantinescu\\n\\nAndreea-Florina Constantinescu (born ) is a retired Romanian female volleyball player, who played as a 0. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with VC Unic Piatra Neamt.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691856\",\"title\":\"Mirela Corjeutanu\",\"body\":\"\\nMirela Corjeutanu\\n\\nMirela Corjeutanu (born ) is a retired Romanian female volleyball player, who played as a universal. Corjeutanu also competed at the 2001 Women's European Volleyball Championship squads and 2005 Women's European Volleyball Championship squads. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Futura Volley Sanarate.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691868\",\"title\":\"Strike Force (1975 film)\",\"body\":\"\\nStrike Force (1975 film)\\n\\nStrike Force is a 1975 television film starring Richard Gere.\\nPlot.\\nA federal agent and a New York City detective join forces with state trooper to break up a drug ring.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691869\",\"title\":\"Football records in Croatia\",\"body\":\"\\nFootball records in Croatia\\n\\nThis page details football records in Croatia. It counts only results and records from 1992 onwards, as that year marked start of Croatian First Football League. Prior 1992 Croatian clubs were part of Football Association of Yugoslavia. Croatia declared independence from Yugoslavia in 1991.\\nNational team.\\nResults.\\nCroatia - Australia 7-0, 6 June 1998 (friendly)\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691870\",\"title\":\"Carmen Marcovici\",\"body\":\"\\nCarmen Marcovici\\n\\nCarmen-Alida Marcovici (born ) is a retired Romanian female volleyball player, who played as a middle blocker. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany and 2005 Women's European Volleyball Championship. On club level she played with Boavista Porto.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691875\",\"title\":\"Rani Sati Temple Jhunjhunu\",\"body\":\"\\nRani Sati Temple Jhunjhunu\\n\\nRani Sati, also identified as Narayani Devi and referred to as Dadiji (grandmother), is said to be a Rajasthani woman who lived sometime between the 13th and the 17th century and committed sati (self-immolation) on her husband's death. Various temples in Rajasthan and elsewhere are devoted to her worship and to commemorate her act. Rani Sati Temple Jhunjhunu is biggest temple of Rani Sati in india.\\nJai Shree Rani Sati\\nHistory.\\nThe history of\\nRani Sati Dadi Maa starts from the time of Mahabharata.\\nNarayani's wish of being married to Abhimanyu and her desire to be sati in her next life.\\nAs granted by Lord Krishna, in her next life she was born as the daughter of Gursamal in the village of Dokwa in Rajasthan. She was named - Narayani. Abhimanyu took birth in Hissar as son of Jaliram and named - Tandhan. Tandan and Narayani got married and were leading a peaceful life. He was in possession of a beautiful horse which was being eyed by the son of king of Hissar from quite some time. Tandan refused to hand over his precious horse to the king’s son.\\nThe king’s son then decides to forcefully acquire the horse and thus challenges Tandan for a combat. However Tandan fights the battle bravely and kills the King’s son instead. The enraged king thus kills Tandan in front of Narayani in the battle. Narayani symbolic to female bravery and power fights with the king and kills him. She then commanded Ranaji (the caretaker of the horse) to make immediate arrangements for her to be set ablaze along with her husband’s cremation.\\nRanaji playing a vital role in fulfilling her wish to be sati with her husband is then blessed by Narayani that his name will be taken and worshiped along with her name and since then she is known as Rani Sati.\\nTemple.\\nThe most amazing feature of this famous temple is that it does not hold any paintings or statues of either female or male gods. Instead a trident depicting power and force is worshipped religiously by the followers. However one can surely find a beautiful portrait of Rani Sati in the pradhan mand. The temple is prepared with white marbles and has colorful wall paintings.\\nIn the complex of Rani Sati temple there is also the Lord Hanuman Temple, Sita Temple, Lord Ganesha Temple and Lord Shiva Temple. The regular ‘Prasad’ distribution takes place after every ‘aarti’. There are also arrangements made for the meals in day time on payment basis. It is beautiful and intricately crafted, not to forget the golden pot at the top of the temple.\\nThere are in all 13 sati temples in the complex with 12 smaller one’s and 1 main temple dedicated to Rani Sati Dadi Built in pure white marble with a red fluttering flag at the top, the building forms are interesting and marvelous.\\nA huge statue of Lord Shiva in the center of the complex surrounded by the lush green gardens, adds to the beauty of the place.\\n Inside the temple, the interiors, adorned with exquisite murals and fascinating rich glass mosaics depicting the entire history of the place, are eye-catching.\\nObservances and festivals.\\nHundreds of devotees visit the temple every day. An elaborate aarti is performed at the temple two times a day. These are:\\nA special Poojan utsav is held on the occasion of\\n Bhadrapada Amavasya : The 15th day of the dark half of Bhadrapada month in the Hindu calendar is of special significance to the temple.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691882\",\"title\":\"Buddhist Rock Carvings in Manglawar\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691883\",\"title\":\"Florentina Nedelcu\",\"body\":\"\\nFlorentina Nedelcu\\n\\nFlorentina Nedelcu (born ) is a retired Romanian female volleyball player, who played as a wing spiker. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany and 2005 Women's European Volleyball Championship. On club level she played with VC Unic Piatra Neamt.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691894\",\"title\":\"Cristina Pîrv\",\"body\":\"\\nCristina Pîrv\\n\\nCristina-Lucretia Pîrv (born ) is a retired Romanian female volleyball player, who played as a wing spiker. She was part of the Romania women's national volleyball team at the 1994 FIVB Volleyball Women's World Championship in Brazil., 2002 FIVB Volleyball Women's World Championship in Germany and 2001 Women's European Volleyball Championship. On club level she played with Minas Tênis Clube.\\nPersonal life.\\nSince 2003 she has been married to the Brazilian former international volleyball player, Giba. They have 2 children together, a daughter Nicoll (8) and a son Patrick (4). In November 2012, Cristina has filled for divorce.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691918\",\"title\":\"2015–16 Aris Thessaloniki F.C. season\",\"body\":\"\\n2015–16 Aris Thessaloniki F.C. season\\n\\nThe 2014–15 season is Aris Thessaloniki F.C. 2nd season in Gamma Ethniki. They will also compete in the Macedonia Football Clubs Association Cup.\\nOn 17 September 2015, Nikos Anastopoulos appointed as manager.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691932\",\"title\":\"Piano District\",\"body\":\"\\nPiano District\\n\\nThe Piano District is the name proposed by two developers for what is currently known as the Port Morris neighborhood of the South Bronx. The two developers, Somerset Partners and The Chetrit Group, purchased two riverfront properties for $58 million with the goals of establishing luxury residences and retail.\\nCurrent residents, fearing the effects of gentrification, are voicing opposition. Critics and proponents have accused one another of racism. The developers were criticized by Melissa Mark-Viverito, the president of the New York City Council for conducting a Halloween rave, attended by numerous celebrities, with the theme of \\\"Macabre Suite\\\" created and curated by Lucien Smith, and hosted by warehouse owner, private equity mogul Keith Rubenstein who is head of real estate developers Somerset Partners, and art dealer Jeanne Greenberg-Rohatyn. The rave featured flaming trash cans and hulks of burnt-out and bullet-riddled cars. Mark-Viverito accused the developers of a lack of empathy and basic awareness.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691945\",\"title\":\"Anca Popescu\",\"body\":\"\\nAnca Popescu\\n\\nAnca Popescu (born ) is a retired Romanian female volleyball player, who played as a middle blocker. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with TV Fischbek.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691948\",\"title\":\"Nightside (film)\",\"body\":\"\\nNightside (film)\\n\\nNightside is a 1980 television pilot starring Doug McClure.\\nPlot.\\nTelevision pilot about the adventures of a streetwise cop working the night shift in Los Angeles with a naive partner.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691949\",\"title\":\"Nicoleta Tolisteanu\",\"body\":\"\\nNicoleta Tolisteanu\\n\\nNicoleta Tolisteanu (born ) is a retired Romanian female volleyball player, who played as a libero. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. She was awarded the best receiver at the 2003 Women's European Volleyball Championship. On club level she played with Amici Bacau.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691962\",\"title\":\"Marc Dollendorf\",\"body\":\"\\nMarc Dollendorf\\n\\nMarc Dollendorf (born 7 February 1966 in Waarloos) is a retired Belgian athlete who specialised in the 400 metres hurdles. He represented his country at the 1996 Summer Olympics as well as four consecutive World Championships starting in 1991.\\nHis personal best in the event is 48.91 seconds set in Atlanta in 1996. This is the standing national records.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691966\",\"title\":\"Ordinary People (Steve Harley song)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691968\",\"title\":\"Luminita Trombitas\",\"body\":\"\\nLuminita Trombitas\\n\\nLuminita-Gabriela Trombitas (born ) is a retired Romanian female volleyball player, who played as a setter. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Petrarca Padua.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691973\",\"title\":\"Carmen Țurlea\",\"body\":\"\\nCarmen Țurlea\\n\\nCarmen Turlea (born ) is a retired Romanian female volleyball player, who played as a opposite. She was part of the Romania women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Foppapedretti Bergamo.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691981\",\"title\":\"Garoutte Creek\",\"body\":\"\\nGaroutte Creek\\n\\nGaroutte Creek is a tributary of the Little River in the U.S. state of Oregon. Rising along the Calapooya Divide near the border between Lane and Douglas counties, it flows generally northeast to meet the larger stream near Black Butte. The butte is a dark-colored mountain, the site of a former mine, and the site of a former post office. Little River joins Big River a few miles downstream to form the Coast Fork Willamette River.\\nOn older maps, Little River appears as a tributary of Garoutte Creek, also known as Saroutte Creek. However, in 1988 the United States Board on Geographic Names renamed the lower reach of Garoutte Creek so that it became part of Little River. The change rendered the upper reach of Garoutte Creek a tributary of Little River. \\nThe only named tributary of Garoutte Creek is Carlson Creek, which enters from the left.\\nPost office.\\nThe Black Butte Mine, developed by S. P. Garoutte in the 1890s, led to the establishment of a post office at Black Butte. William Harris was the first postmaster, and the office, established in 1898, was known by his last name, \\\"Harris\\\". The name was changed in 1901 to Blackbutte. It operated under that name until permanently closing in 1957. \\nPollution.\\nIn the 21st century, the former mine and the area around it have become a Superfund site. The Environmental Protection Agency is concerned about methylmercury contamination of streams including Garoutte Creek, the Little River, and the Coast Fork, as well as the Cottage Grove Reservoir on the Coast Fork. During its heyday, the mine was the second-largest mercury producer in Oregon. From 1900 to 1957, it yielded a total of 16,904 flasks of elemental mercury; equivalent to about .\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691992\",\"title\":\"Choi Kwang-hee (volleyball)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48691998\",\"title\":\"Tom Connell (American football)\",\"body\":\"\\nTom Connell (American football)\\n\\nTom Connell ( – ) was an American football player. He played at the halfback position for the University of Detroit Titans football team and was captain of the undefeated 1928 team that was selected by Parke H. Davis as that year's national champion. He scored 126 points in 1928, ranking second in scoring nationally. He was also the captain of the 1927 Detroit team and the first Detroit player to be twice selected as captain. He was married in 1932 to Josephine Gleason, a music teacher.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692009\",\"title\":\"Huang Jing\",\"body\":\"\\nHuang Jing\\n\\nHuang Jing (; 1912 – 10 February 1958), born Yu Qiwei (), was a Chinese Communist revolutionary and politician who served as Mayor and Communist Party Chief of Tianjin municipality, Minister of the First Ministry of Machine Building Industry, and Chairman of the National Technological Commission. He was an ex-husband of Jiang Qing, who later married Mao Zedong, and the father of Yu Qiangsheng, a top Chinese intelligence officer who defected to the United States in 1985, and Yu Zhengsheng, the fourth-ranked member of the Politburo Standing Committee.\\nEarly life and revolution.\\nYu Qiwei was born in 1912 to a prominent family in Shaoxing, Zhejiang Province. His uncle Yu Ta-wei later served as Minister of National Defense of the Republic of China on Taiwan. The chemist Zeng Zhaolun was also his uncle. Yu enrolled in Shandong University in Qingdao, majoring in physics. At the same time, he spent significant amount of time in underground political activism for the Communist Party of China.\\nWhile in Qingdao, Huang Jing met and married Li Yunhe (who would later change her name to \\\"Jiang Qing\\\" and marry Mao Zedong), in 1932. Huang introduced the 19-year-old Li to join the Communist Party in 1933. Soon afterwards, Huang was arrested by the government for his Communist activism. To avoid implicating Li, he sent a message asking her to leave him. Li was introduced to Shanghai film director Shi Dongshan, who was in Qingdao at the time, and followed Shi to Shanghai. After Huang's release in 1934, he lived with Li for a while with his family in Shanghai. However, Huang's family was adamantly against their marriage, and they became separated.\\nIn 1935, Huang Jing, then attending Peking University, co-led the December 9th Movement with Yao Yilin and Huang Hua, demanding the Chinese government to actively resist Japanese aggression in the aftermath of the Mukden Incident.\\nAfter the Japanese invasion of China in 1937, Huang moved to the Communist base in Yan'an in winter 1939. Li Yunhe, now known as Jiang Qing, had also moved to Yan'an and married the Communist leader Mao Zedong. Huang later became a department head in the Communist bases in Shanxi-Chahar-Hebei (Jin-Cha-Ji) and Shanxi-Hebei-Shandong-Henan (Jin-Ji-Lu-Yu) border areas.\\nPolitical career.\\nFollowing the surrender of Japan in 1945, the Communists took over northern Hebei Province, and Huang was appointed Mayor of Zhangjiakou. After the founding of the People's Republic of China in 1949, he became the Mayor of the Tianjin municipality, as well as the city's Communist Party Chief.\\nIn 1952, Huang was appointed Minister of the First Ministry of Machine Building Industry, which was in charge of the civilian industry (the Second Ministry was in charge of military work). When the National Technological Commission was established in 1956, he became its first chairman. While serving in these capacities Huang praised the work of a young engineer in Shanghai named Jiang Zemin, the later Chinese president, who recalled that Huang invited him to a banquet at the Quanjude duck restaurant, and on another occasion, talked to him for four hours until 11 pm.\\nHuang Jing was considered a promising young star of the Communist Party, but was labelled a counterrevolutionary when the Anti-Rightist Campaign began in 1958. He died in Guangzhou that same year, at the age of only 46. The circumstances surrounding his death are unclear. It is said that he suffered from mental and physical diseases and died of heart disease in a military hospital.\\nFamily.\\nAfter his relationship with Jiang Qing ended, Huang Jing married the journalist Fan Jin, who later became vice mayor of Beijing and president of \\\"Beijing Daily\\\". Their son, Yu Qiangsheng, was a top Chinese intelligence officer who defected to the United States in 1985. Another son, Yu Zhengsheng, rose to become one of the seven members of the Politburo Standing Committee in 2012, which effectively rules China.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692018\",\"title\":\"Vellaiya Irukiravan Poi Solla Maatan\",\"body\":\"\\nVellaiya Irukiravan Poi Solla Maatan\\n\\nVellaiya Irukiravan Poi Solla Maatan is aTamil comedy film written and directed by newcomer A. L. Abanindran.\\nProduction.\\nThe project was announced by Ignite Films and director Abanindran in late 2014, and marked the director's feature film debut after he had previously, along with his partner Devanshu Arya, assisted director Rajiv Menon's \\\"Minsara Kanavu\\\" (1997) and \\\"Kandukondain Kandukondain\\\" (2000). The film features newcomer Praveen Kumar, who played a supporting role in \\\"Kalyana Samayal Saadham\\\" (2013) in the lead role alongside Shalini Vadnikatti, a model from Hyderabad. Abanindran had worked with Praveen Kumar and the producers, Ignite Films, before, when the trio worked on the television serial, \\\"Dharmayutham\\\" for Vijay TV during 2012. Cinematographer Ravi Varman also worked as a co-producer for the film, marking his maiden production. The ensemble cast was also revealed to feature Sanam Shetty, Karthik Kumar, Naren, Jayaprakash and Bala Saravanan, while Anthony was announced as the film's editor. The film was named after a dialogue from Vadivelu's comedy track in the film, \\\"Arya\\\" (2007).\\nThe team shot the film around Chennai in forty days. After the film was completed, S. Thanu of Kalaipuli International bought the film's worldwide distribution rights.\\nRelease.\\nThe film released to mixed reviews in December 2015, having been delayed for three weeks as a result of the 2015 South Indian floods. A critic from the New Indian Express noted that \\\"it was an impressive debut\\\".\\nThe Times of India review rated the movie 3 out of 5 stars.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692033\",\"title\":\"Saeed Ibn Qais Hamdani\",\"body\":\"\\nSaeed Ibn Qais Hamdani\\n\\n Saeed Ibn Qais Hamdani () One of the leaders and Tabi‘un Hermit including close friends Ali and Hasan ibn Ali, head of the Yemeni tribe of Hamdan in Iraq. In the year 33 AH by Saeed bin Aas, who was then governor Osman was in Kufa, responsible government was Rey, Iran.\\nCharacteristics.\\nHe was faithful helpers Ali. Ali ibn Abi Talib in several battles, including the Battle of the Camel, Battle of Siffin the head of the army. After the death of Ali, Saeed ibn Qays al-Hamadani had joined Hasan ibn Ali and companions.\\nDeath.\\nDate of his death is not known but some date in the year 50 AH he noted.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692034\",\"title\":\"Arun Pudur\",\"body\":\"\\nArun Pudur\\n\\nArun Pudur (born October 10, 1977) is an Indian origin Billionaire, Businessman, Serial Entrepreneur, investor, and philanthropist. His operations headquarters are in Kuala Lumpur, Malaysia and New York, USA. Pudur started his career fixing scooters in his garage at age 13, he then started breeding champion dogs before venturing into technology. He is the Founder and Group President of Celframe, the world’s second largest Office Productivity Suite software company after Microsoft Office. In his tenure, Celframe rose from a backend support company to a Multi-Billion Dollar Technology company with businesses around the Global while it is privately held and majority owned by him. He is consistently ranked in lists of Billionaires. Arun Pudur is Asia's richest, under 40 person with wealth more than $4 billion and is also ranked 10th among World’s Richest in the list published by Bloomberg, CNBC, WSJ, Wealth-x, Market Watch, and CNN among other major publications worldwide. Pudur is the only Indian Billionaire to make the list which includes Mark Zuckerberg of Facebook among other billionaires.\\nEarly life.\\nPudur was born in Chennai, but his family shifted to Bengaluru, as his father used to stay 7 to 8 months an year there. His father, Sri Ranga, was a cinematographer, known for his 16mm movies. He had chosen commerce as field of interest in college, as opposed to science, despite everyone around him telling him to take science. He is an alumni of Bangalore University. He started his career starting a scooter garage to fix Kinetic Honda and soon franchised the garage all over Karnataka state. He says that he learned all the marketing skills, problem solving, customer handling, during his time at the garage.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692040\",\"title\":\"Austrian presidential election, 2016\",\"body\":\"\\nAustrian presidential election, 2016\\n\\nPresidential elections will be held in Austria on 24 April 2016, with a run-off on 22 May 2016, if necessary. Incumbent president Heinz Fischer has served two terms and is not eligible to be elected for a third successive term.\\nCandidates.\\nSPÖ: The most likely candidate of the Social Democratic Party was considered to be Labour Minister Rudolf Hundstorfer, though First President of the National Council Doris Bures, Franz Vranitzky (former Chancellor of Austria from 1986 to 1997) and former undersecretary for EU affairs and businesswoman Brigitte Ederer were also mentioned. On 15 January 2016, Rudolf Hundstorfer was officially announced as the SPÖ's candidate.\\nÖVP: Justice Minister Wolfgang Brandstetter (considered a possible, if unlikely ÖVP candidate) declined to stand on 26 December 2015. On 7 January 2016, ÖVP leader Reinhold Mitterlehner announced that Erwin Pröll, the Landeshauptmann of Lower Austria, would not be running, to much surprise (as many had predicted he was almost certainly going to be the ÖVP's candidate). Josef Pühringer, Landeshauptmann of Upper Austria, who had also occasionally been mentioned as an (unlikely) candidate, explicitly declined to stand on 8 January 2016. Franz Fischler (former Austrian European Commissioner from 1995 to 2004) and Othmar Karas (highly respected Member of the European Parliament) also both stated they were not the ÖVP candidate on 8 January 2016, while Wirtschaftskammerpräsident Christoph Leitl only said he would not comment before the official announcement by the party leadership on 10 January 2016. Even controversial former chancellor Wolfgang Schüssel (from 2000 to 2007) was briefly considered as a candidate (but he also declined). On 10 January 2016, former first president of the National Council Andreas Khol was officially announced as the ÖVP's candidate.\\nOther names mentioned were former Science Minister and university professor Karlheinz Töchterle, Foreign Minister Sebastian Kurz (who was too young too stand, not yet being 35 years old), former Foreign Minister Ursula Plassnik (who was in office from 2004 to 2008, and most recently Austria's ambassador to France, about to be reassigned to Switzerland), former Raiffeisen Zentralbank manager Christian Konrad (currently the government's point man on refugee issues; he stated he was not a candidate on 8 January 2016), former Landeshauptfrau of Styria Waltraud Klasnic (in office from 1996 to 2005) and president of the Salzburg Festival Helga Rabl-Stadler, both of whom were considered unlikely to run. This left Plassnik and Leitl as possible candidates as of 8 January 2016. Later in the day, rumours indicated that former first president of the National Council Andreas Khol would be the ÖVP's candidate.\\nFPÖ: Norbert Hofer (third president of the National Council), who had been considered the most likely FPÖ candidate, said on 28 December 2015 that he considered himself too young for the office and that he would prefer his party to pick someone else as its candidate. Possible candidates include president of the Austrian Court of Audit Josef Moser, Ursula Stenzel (former district mayor of Vienna's 1st District from 2005 to 2015 and also former Member of the European Parliament from 1999 to 2006, both for the ÖVP), ombudsman Peter Fichtenbauer and possibly party leader Heinz-Christian Strache himself. Former Social Affairs Minister Ursula Haubner was considered a very unlikely candidate. As of 11 January 2016, Fichtenbauer, Moser and Stenzel continued to be the most likely candidates. Strache announced on 13 January 2016 that he would not be running himself, and that it was still open whether the FPÖ would nominate anyone at all. In mid-January, Vienna vice-mayor Johann Gudenus and former FPÖ leader and former vice-chancellor (from 1983 to 1987) Norbert Steger were also mentioned as possible candidates. On 19 January 2016, author and Middle East/migration pundit Karin Kneissl (well-known for her right-wing views) was mentioned as being recruited by the FPÖ to run, which she quickly declined. Former Justice Minister Dieter Böhmdorfer, members of parliament Harald Stefan and Johannes Hübner and MEP Barbara Kappel were also mentioned as possible candidates. On 20 January 2016, media reported that Gudenus had been internally selected as the FPÖ's candidate; on 26 January 2016, reports claimed Stenzel would be announced on 28 January 2016 as the FPÖ's candidate. Amid strong FPÖ-internal dissent, there were rumours the party leadership had been forced to reconsider, and that Hofer was now the most likely option, after all, with Gudenus also still in play. Commentators opined that the backtracking was a notable defeat for Strache. Norbert Hofer was announced as the FPÖ's candidate on 28 January 2016.\\nGreens: In early January 2016, it was announced that former Greens party leader Alexander Van der Bellen would not be running as the Greens' “official” candidate, as that would have required a party convention decision; this was also framed as an attempt to put personality above party politics in the election. Van der Bellen officially announced his candidacy on 8 January 2016 in a YouTube video. NEOS leader Strolz stated that they would consider giving him the same support as Griss, depending on the same kind of hearing she went through.\\nTeam Stronach: As of 22 January 2016, the Team Stronach was still considering whether to nominate a candidate.\\nOthers: Independent candidate Irmgard Griss, a former Supreme Court of Justice judge and its president, officially declared her candidacy on 17 December 2015. She presented her candidacy to both the Freedom Party of Austria and NEOS, but both declined to endorse her. NEOS said they would support Griss and any other independent candidates indirectly, and voiced their concerns over the strong partisan politicization of the presidential office and the election campaign.\\nRichard Lugner, society figure, businessman and previously candidate for president in 1998, was reportedly also considering running once more. Martin Wabl, who had previously attempted to run in 1998, 2004 and 2010, but failed to gather the necessary number of signatures of support, stated he would try to run once again. Ulrich Habsburg-Lothringen, whose initiative to get the so-called “Habsburger-Paragraf” (which had precluded members of the former ruling house from running for president) proved successful in 2011, stated he would like to run for president, but only if a political party decided to support him. Adrien Jean-Pierre Luxemburg-Wellenstein announced on 8 December 2015 he would run for president. Author El Awadalla (also known for winning the Austrian version of Who Wants to Be a Millionaire?) announced her run on 12 January 2016. (She had previously run for the left-wing alliance Wien anders in the 2015 Viennese state election.) Local activist Franz Stieger (known in Krems for his numerous conflicts with local officials and government agencies) announced his candidacy on 13 January 2016. Further independent candidates who announced their runs were Gustav Jobstmann, Thomas Unden (a general practitioner who had stated he would refuse to treat asylum seekers), Gernot Pointner, Alois Merz, Georg Zakrajsek of the \\\"Interessengemeinschaft Liberales Waffenrecht Österreich\\\" (a fringe gun rights group), Karin Kolland, Robert Marschall of the EU Exit Party and Thomas Reitmayer of the Austrian version of the satirical political party Die PARTEI.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692069\",\"title\":\"Curling at the 2016 Winter Youth Olympics – Mixed doubles\",\"body\":\"\\nCurling at the 2016 Winter Youth Olympics – Mixed doubles\\n\\nMixed doubles curling at the 2016 Winter Youth Olympics will be held from February 19 to 21 at Kristins Hall in Lillehammer, Norway.\\nTeams.\\nThe teams will consist of athletes from the mixed team tournament, one boy and one girl from different NOCs. The teams will be selected by the organizing committee based on the final ranking from the mixed team competition in a way that balances out the teams, and will be assigned on February 17. The players in each pair will then be allowed time to train together.\\nTeams are to be determined.\\nKnockout Results.\\nAll draw times are listed in Central European Time (UTC+01).\\nRound of 32.\\nDraw 1.\\n\\\"Friday, February 19, 9:00\\\"\\nDraw 2.\\n\\\"Friday, February 19, 12:30\\\"\\nDraw 3.\\n\\\"Friday, February 19, 16:00\\\"\\nDraw 4.\\n\\\"Friday, February 19, 19:30\\\"\\nRound of 16.\\nDraw 1.\\n\\\"Saturday, February 20, 9:00\\\"\\nDraw 2.\\n\\\"Saturday, February 20, 13:00\\\"\\nQuarterfinals.\\n\\\"Saturday, February 20, 17:00\\\"\\nSemifinals.\\n\\\"Sunday, February 21, 9:00\\\"\\nBronze Medal Game.\\n\\\"Sunday, February 21, 13:00\\\"\\nGold Medal Game.\\n\\\"Sunday, February 21, 13:00\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692071\",\"title\":\"First Presbyterian Church (Raleigh, North Carolina)\",\"body\":\"\\nFirst Presbyterian Church (Raleigh, North Carolina)\\n\\nFirst Presbyterian Church is a historic Presbyterian church located at the corner of Morgan and Salisbury Streets in downtown Raleigh, Wake County, North Carolina. \\nThe church was established in a meeting of Presbyterians at the North Carolina State House (predecessor of the North Carolina State Capitol) on Jan. 21, 1816. The congregation purchased land at the present location and erected a brick church building that opened its doors on February 7, 1818. The church served as the site for the North Carolina Constitutional Convention of 1835 and as the meeting place for the North Carolina Supreme Court for several years.\\nThat original structure was torn down and a new church building completed in 1900. It was designed in the Romanesque Revival style by architect A.G. Bauer and has been extensively remodeled twice, in 1955 and 2012. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692073\",\"title\":\"Henryk Weyssenhoff\",\"body\":\"\\nHenryk Weyssenhoff\\n\\nHenryk Bonawentura Kazimierz Weyssenhoff (26 July 1859, near Kaunas - 23 July 1922, Warsaw) was a Polish-Belarusian landscape painter, illustrator and sculptor of Baltic-German ancestry.\\nBiography.\\nHe was descended from an old family of the Livonian nobility. In 1863, his father was exiled to Siberia for his participation in the January Uprising. His mother followed with the family, as far as she could, and he grew up in the Urals. His first lessons in art were given to him there by Lucjan Kraszewski (brother of the writer Józef Ignacy Kraszewski), who was also living in exile.\\nIn 1874, his father was pardoned, but they were not allowed to return to their old home, so they settled in Warsaw, where he studied painting with Wojciech Gerson. His work there attracted the attention of Henryk Siemiradzki who, in 1880, helped him gain admission to the Imperial Academy of Arts in Saint Petersburg. He graduated in 1885, earning a silver medal and the official title of \\\"Artist\\\" for his canvas \\\"Transport of the Wounded\\\". In 1889, he spent some time in Munich, improving his skills with Alfred Kowalski.\\nIn 1900, he won another silver medal at the Exposition Universelle. As a result, from 1903 to 1904 he lived in Paris, then settled on his parent's estate near Pukhavichy, in what is now Belarus. During the First World War, he was forced to return to Warsaw and remained there until his death.\\nAn avid hunter, many of his landscapes included animal life. In addition to his painting, he illustrated several works, notably two by his cousin, Józef Weyssenhoff; \\\"Erotyki\\\" (1911), a book of poetry, and \\\"Soból i Panna\\\" (1913), a novel that is loosely based on the manorial lifestyle of the Weyssenhoff family.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692076\",\"title\":\"Mi Perú District\",\"body\":\"\\nMi Perú District\\n\\nMi Perú is a district of the Constitutional Province of Callao in Peru, and one of the seven districts that comprise the port city of Callao.\\nHistory.\\nMi Perú was part of Ventanilla District until May 17, 2014 when it was created as District by Law N° 30197.\\nGeography.\\nThe district has a total land area of 2.47 km². Its administrative center is located 34 meters above sea level.\\nAuthorities.\\nThe current mayor of the district is Reynaldo Encalada Tovar.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692084\",\"title\":\"Whitewater, Wisconsin, parks and trails\",\"body\":\"\\nWhitewater, Wisconsin, parks and trails\\n\\nWhitewater, Wisconsin, is a town located in Southeastern Wisconsin roughly 45 minutes from Madison and Milwaukee. The city’s population is 14,732 and features one of the University of Wisconsin system schools, The University of Wisconsin-Whitewater. Throughout the city and surrounding areas, there are over 20 parks, trails and preserves.\\nCravath Lakefront Park.\\nCravath Lakefront Park was opened in the early 1990s in downtown Whitewater. It is located at 341 South Fremont Street. The park was created to try and create an open space downtown to encourage more visitors. The design for the park was loosely based on the Whitewater Passenger Train Depot. The park itself used to be for more industrial purposes instead of a recreational area. Once it was turned into a park, though, it holds events like Freeze Fest and Pig in the Park as well as the Fourth of July. On top of that, Cravath Lakefront Park is willing to hold celebrations, concerts and weddings.\\nBig Brick Park.\\nBig Brick Park is located at 611 West Center Street in the heart of Whitewater, WI. This park is roughly one square block in size and is open to the general public. This park has a blacktop basketball court and a playground. For the winter months the city creates its own ice rink for the public to ice skate as well as play hockey. The city provides nets for the hockey players to enjoy the ice time during the bitter cold Wisconsin winter.\\nTrippe Lake Park.\\nTrippe Lake is located at 407 S Wisconsin Street and has been around since 1958, named in honor of Dr. James and Rosepha Trippe, as being founders of the city. You can enjoy the beautiful lake where a dam was construed to power a sawmill. Community members use it for swimming, fishing, free boat launch and as a group gathering place. The local Whitewater’s Lion Club donated the Trippie Lake Shelter in 2009. It includes restrooms, picnic tables, kitchen, and storage space for recreations programs. Trippe Lake has many options for all seasons, summer swimming to\\nsledding and ice fishing in winter. It adds great value to the Whitewater Community.\\nWhitewater Creek Nature Area.\\nLocated off Fremont Street on the outskirts of campus you can find Whitewater Creek Nature Area. It is a great location with a lot to park and hit the walking trails. It is open all year round from 6am to 12pm. Trails take you into and through the woods, giving you great scenic views along the creek. It is public land for the community to use in a respectful way. There are lots of wildlife activity and natural forest activities. Whitewater Creek Nature Area is a beautiful area in that is unlike any other place in Whitewater.\\nCity of Whitewater “Bark Park”\\nThe city features a spacious dog park that is free of charge. The park features three areas for all sizes of dogs. Located at the East end of Commercial Avenue in Whitewater, it provides a fun and clean environment for all animals. The three areas are designated for different sized dogs. This area is roughly 4 acres of city property. The city mows and provides dog clean up bags for no price. For pet owners, there are multiple benches and picnic tables available for seating.\\nIce Age Trail.\\nLocated 4 miles east of Whitewater on HWY 12 is a section of the infamous Ice Age Trail. The trail is nearly 1,200 miles long stretching from Minnesota to the shores of Lake Michigan. The beautiful trail extends approximately 10 miles through the southern part of Whitewater. This section of the trail is near scenic Whitewater Lake and Rice Lake. The Trail is also connected with the Whitewater Lake Campground. The trail is animal friendly and open year round for hikers and snowshoeing. One end of the trail begins in Potawatomi State Park near Sturgeon Bay, WI and the other ends in Interstate Park near the St. Croix River on the Wisconsin/Minnesota border in St. Croix, WI.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692092\",\"title\":\"Fabrizio Santino\",\"body\":\"\\nFabrizio Santino\\n\\nFabrizio Santino (born December 12, 1982) is a British actor who is best known for playing Ziggy Roscoe in the soap opera \\\"Hollyoaks\\\" between 2013 and 2015. Fabrizio will play Jake Albert in an upcoming BBC drama entitled Hawaiian Hotel due to air in Spring 2016.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692132\",\"title\":\"Frederick Hanger House\",\"body\":\"\\nFrederick Hanger House\\n\\nThe Frederick Hanger House is a historic house at 1010 Scott Street in Little Rock, Arkansas. It is a two story wood frame structure, with complex massing and exterior typical of the Queen Anne style. It is topped by a gable-on-hip roof, from which numerous gables project, including two to the front, and has walls sheathed in clapboards and bands of decorative cut shingles. A porch extends across the front, supported by turned posts, with a balustrade of wooden circles joined by posts to each other and the supporting posts. It was built in 1889 for one of Little Rock's most prominent businessmen of the period, and is a particularly little-altered example of the Queen Anne style in the city.\\nThe house was listed on the National Register of Historic Places in 1974.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692146\",\"title\":\"Butyrsky (rural locality)\",\"body\":\"\\nButyrsky (rural locality)\\n\\nButyrsky (; masculine), Butyrskaya (; feminine), or Butyrskoye (; neuter) is the name of several rural localities in Russia:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692150\",\"title\":\"Oregon Institute of Technology, Wilsonville\",\"body\":\"\\nOregon Institute of Technology, Wilsonville\\n\\nThe Oregon Institute of Technology, Wilsonville (also referred to as Oregon Tech Wilsonville, or simply OITW) is a public polytechnic and research university located in Wilsonville, Oregon, United States. It is the urban campus of the Oregon Institute of Technology.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692152\",\"title\":\"Butyrsky\",\"body\":\"\\nButyrsky\\n\\nButyrsky (masculine), Butyrskaya (feminine), or Butyrskoye (neuter) may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692168\",\"title\":\"The Thanksgiving Story\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692169\",\"title\":\"Hardy House (Little Rock, Arkansas)\",\"body\":\"\\nHardy House (Little Rock, Arkansas)\\n\\nThe Hardy House is a historic house at 2400 Broadway in Little Rock, Arkansas. It is a two story brick structure, with flanking single-story wings and a roof that is designed to resemble an English country house's thatched roof. The entrance is set in a centrally located stone round arch, with a multipart segmented-arch window above. The house was designed by Charles L. Thompson and built in 1921.\\nThe house was listed on the National Register of Historic Places in 1982.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692204\",\"title\":\"2016 TCR Russian Series season\",\"body\":\"\\n2016 TCR Russian Series season\\n\\nThe 2016 TCR Russian Series season is the second season of the TCR Russian Series.\\nCalendar and results.\\nThe 2016 schedule was announced on 11 November 2015, with all events scheduled to be held in Russia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692210\",\"title\":\"Karosa B 831\",\"body\":\"\\nKarosa B 831\\n\\nKarosa B 831 is citybus, which was made in three pieces by bus manuafcturer Karosa between 1987-1989. It was intended as successor of Karosa B 731 and with trolleybus Škoda 17Tr had to be part of unificated series of city vehicles. Until now survived only one, which is placed in muzeum in Brno.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692213\",\"title\":\"London Recruits (film)\",\"body\":\"\\nLondon Recruits (film)\\n\\nLondon Recruits is the upcoming film documenting the true story of the young men and women who undertook clandestine missions for the ANC during apartheid South Africa. Produced by Barefoot Rascals, the docu-drama is directed by Gordon Main. London Recruits shares a title with the 2012 book London Recruits: The Secret War Against Apartheid, edited and compiled by Ken Keable.\\nPlot.\\nSet at a time when ANC membership was prohibited and its activists executed or imprisoned, London Recruits tells the story of the actions of young women and men who traveled to South Africa disguised as honeymooning couples, tourists and business trippers, aimed at bringing down the apartheid regime. The London Recruits, in a plan activated by Oliver Tambo, were recruited by Ronnie Kasrils and deployed to undertake secret missions involving the transportation of anti-apartheid leaflets in false bottom suitcases and the detonation of bucket bombs in strategic commuter sites.\\nThe Recruits.\\nThe facilitator of the operation was Ronnie Kasrils. South African born, Ronnie was dispatched to London by Oliver Tambo to recruit women and men to carry out missions for the ANC. Mary Chamberlain, author of Fenwomen: A Portrait of Women in an English Village was among the recruits enrolled by Ronnie to carry out the clandestine work in 1971 at the age of 24. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692256\",\"title\":\"SS Ada (1905)\",\"body\":\"\\nSS Ada (1905)\\n\\nSS \\\"Ada\\\" was a cargo vessel built for the London and South Western Railway in 1905.\\nHistory.\\nShe was built by Gourlay Brothers in Dundee and launched on 4 April 1905 by Miss Drummond. She was launched only 47 days after the keel was laid, without overtime being worked, and represented a record for the Gourlay shipyard. She was the first of a pair of ships ordered by the London and South Western Railway, the other being . She was built for light cargo traffic between Southampton and the Channel Islands. \\nShe was acquired by the Southern Railway in 1923.\\nShe was scrapped in 1934.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692277\",\"title\":\"Fulvio Caccia\",\"body\":\"\\nFulvio Caccia\\n\\nFulvio Caccia (born 10 January 1952 Florence, Italy) is a contemporary Italian poet, novelist and essayist. He won the 1994 Prix du Gouverneur Général.\\nBiography.\\nFulvio Caccia graduated from l'Université du Québec à Montréal in 1979.\\nHe lives in Paris.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692282\",\"title\":\"Keyarris Garrett\",\"body\":\"\\nKeyarris Garrett\\n\\nKeyarris Garrett (born September 26, 1992) is an American football wide receiver. He played college football at Tulsa.\\nCareer.\\nDuring the 2015 season, he had 96 receptions for 1,588 yards and eight touchdowns. On October 23, 2015, Garrett caught 14 passes for an American Athletic Conference record 268 receiving yards. For his career he had 219 receptions for 3,209 yards and 22 touchdowns.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692284\",\"title\":\"Harris House (Little Rock, Arkansas)\",\"body\":\"\\nHarris House (Little Rock, Arkansas)\\n\\nThe Harris House is a historic house at 6507 Fourche Dam Pike in Little Rock, Arkansas. It is a single-story stuccoed structure, designed in an eclectic interpretation of Spanish Revival architecture. Prominent features include a circular tower at one corner, a parapet obscuring its sloping flat roof, and a port-cochere with a segmented-arch opening supported by battered wooden columns. It was built in 1924 for Florence and Porter Field Harris, to their design and probably the work of Porter Harris, a master plasterer known for his work on the Arkansas State Capitol.\\nThe house was listed on the National Register of Historic Places in 1998.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692314\",\"title\":\"Theodore Swann\",\"body\":\"\\nTheodore Swann\\n\\nTheodore Swann was an American industrialist and early leader of the chemical industry. He was described by one historian as \\\"a flamboyant Birmingham mogul and New South industrialist.\\\"\\nHailing from east Tennessee, in his early working years, Swann was a power salesman for the Alabama Power Company.\\nHe established the Federal Phosphorus Company to produce concentrated phosphoric acid, mainly for use as a concentrated fertilizer, using a novel method to produce the acid from phosphate rock by heat treatment in an electric furnace.\\nHe later established the Swann Chemical Company, focused on production of PCBs for the emergent electrical industry. Swann Chemical Company operated a chemical manufacturing plant in Anniston, Alabama where PCBs were first made on an industrial scale after development of a new process under Swann's leadership. In 1920, Swann gave his engineers the challenge of creating a process by which PCBs, up to that time very expensive and produced only in small quantities, could be produced in industrial quantities, and after much trial and error, they succeeded. The plant was later bought by Monsanto Industrial Chemicals Co. in 1935. The plant, just west of Anniston, had around 1,000 employees. One historian wrote that, \\\"In many ways, the spirit of Swann Chemical became the corporate culture of Monsanto.\\\"\\nSwann's house in Birmingham, Alabama, built from 1927 to 1930, was at the time the largest residence in the city and cost $600,000 to build.\\nWhile Monsanto became the most successful agrochemical corporation in the world, Swann began to slide into poverty. He was forced to sell his castle in 1945 when he filed for bankruptcy. He nearly made a comeback by selling a new design for a furnace to process iron ore to the Cuban dictator Fulgencio Batista, but the dictator reneged after paying only a modest down payment. Theodore Swann died selling aluminum window frames on February 5, 1955.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692363\",\"title\":\"SS Bertha (1905)\",\"body\":\"\\nSS Bertha (1905)\\n\\nSS \\\"Bertha\\\" was a cargo vessel built for the London and South Western Railway in 1905.\\nHistory.\\nShe was built by Gourlay Brothers in Dundee and launched on 9 November 1905 by Miss Key, daughter of one of the railway superintendents. She was the second of a pair of vessels ordered by the London and South Western Railway, the other being . She was built for light cargo traffic between Southampton and the Channel Islands. \\nShe was acquired by the Southern Railway in 1923.\\nShe was sold in 1933 to Metal Industries, Limited of Rosyth and used as a salvage vessel in raising some members of the German Fleet scuttled in Scapa Flow. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692393\",\"title\":\"Inspector General of the Air Force\",\"body\":\"\\nInspector General of the Air Force\\n\\nThe Office of Inspector General of the Air Force for the United States Air Force was originally established after World War II as The Air Inspector to conduct investigations as directed by the Secretary of the Air Force and the Chief of Staff of the United States Air Force. The current mission of the Air Force Inspector General is prescribed by Title 10 (§ 8020) and Title 32 of the United States Code (§ 105) to develop Air Force and Air National Guard policy to assess readiness, discipline and efficiency with a vision to help shape senior leader decisions affecting the readiness of the Air Force to strengthen the nation's defense.\\nThe Office of Inspector General of the Air Force consists of four directorates:\\nHistory.\\nIn 1943, Junius Jones was designated The Air Inspector of the Army Air Forces and when the AAF became the U.S. Air Force in 1947, he retained his position.\\nIn 1948, The Air Inspector was renamed to the Inspector General of the Air Force.\\nIn December 1971, Lt Gen Louis L. Wilson Jr. oversees the activation of the Air Force Inspection and Safety Center (which later became the Air Force Inspection Agency in 1991) to provide independent assessments of acquisition, safety, nuclear surety, operations, logistics, support, and healthcare to Air Force senior leaders. It also evaluates Air Force activities, personnel, and policies, and provides legal and compliance oversight of all Air Force-level Field Operating Agencies and Direct Reporting Units.\\nIn September 1986, as a result of the Goldwater–Nichols Act, the Inspector General moved directly under the Secretary of the Air Force.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692396\",\"title\":\"Mirna Ansaldi\",\"body\":\"\\nMirna Ansaldi\\n\\nMirna Eugenia Ansaldi (born ) is a retired Argentine female volleyball player, who played as a wing spiker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Gimnasia y Esgrima de Rosario.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692409\",\"title\":\"Ride the Man Down\",\"body\":\"\\nRide the Man Down\\n\\nRide the Man Down is a 1952 American Western film directed by Joseph Kane and written by Mary C. McCall, Jr.. The film stars Brian Donlevy, Rod Cameron, Ella Raines, Forrest Tucker, Barbara Britton, Chill Wills and J. Carrol Naish. The film was released on November 25, 1952, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692410\",\"title\":\"Julietta Borghi\",\"body\":\"\\nJulietta Borghi\\n\\nJulietta Borghi (born ) is a retired Argentine female volleyball player, who played as a middle blocker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Banco Nación.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692413\",\"title\":\"Natalia Brussa\",\"body\":\"\\nNatalia Brussa\\n\\nNatalia Brussa (born ) is a retired Argentine female volleyball player, who played as a middle blocker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Central San Carlos.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692418\",\"title\":\"Mariana Conde\",\"body\":\"\\nMariana Conde\\n\\nMariana Conde (born ) is a retired Argentine female volleyball player, who played as a wing spiker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Volley 2000 Spezzano.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692427\",\"title\":\"Celina Crusoe\",\"body\":\"\\nCelina Crusoe\\n\\nCelina Crusoe (born ) is a retired Argentine female volleyball player, who played as a setter. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pallavolo Reggio Emilia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692431\",\"title\":\"Romina Lamas\",\"body\":\"\\nRomina Lamas\\n\\nRomina Lamas (born ) is a retired Argentine female volleyball player, who played as a setter. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with CV Teneriffa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692432\",\"title\":\"Karen F McCarthy\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692434\",\"title\":\"John Chambers (Australian cricketer)\",\"body\":\"\\nJohn Chambers (Australian cricketer)\\n\\nJohn Chambers (born 30 October 1930) is an Australian former cricketer. He played 27 first-class cricket matches for Victoria between 1950 and 1955.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692436\",\"title\":\"List of Major League Baseball teams by payroll in 2010\",\"body\":\"\\nList of Major League Baseball teams by payroll in 2010\\n\\nThis is a list of the 30 Major League Baseball teams from the 2010 season, ranked by total team salary. Values only include salaries of players on their respective 2010 rosters.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692442\",\"title\":\"List of Major League Baseball teams by payroll in 2012\",\"body\":\"\\nList of Major League Baseball teams by payroll in 2012\\n\\nThis is a list of the 30 Major League Baseball teams from the 2012 season, ranked by total team salary. Values only include salaries of players on their respective 2012 rosters.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692475\",\"title\":\"Ivana Müller\",\"body\":\"\\nIvana Müller\\n\\nIvana Eloisa Müller (born ) is a retired Argentine female volleyball player, who played as a middle blocker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Banco Nación.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692485\",\"title\":\"Andy and the Lion\",\"body\":\"\\nAndy and the Lion\\n\\nAndy and the Lion, written and illustrated by James Daugherty, is a 1938 picture book published by Puffin Books. \\\"Andy and the Lion\\\" was a Caldecott Medal Honor Book for 1939 and was Daugherty's first Caldecott Honor Medal of a total of two during his career. Daughetry won the Caldecott Medal in 1957 for \\\"Gillespie and the Guards\\\", which he both authored and Illustrated. \\\"Andy and the Lion\\\" was re-issued by Viking Press in 1967 in hardcover format. It was the fifteenth printing of March 1967.\\nDescription.\\nThe story, written by Daugherty, is told in present tense from a second person point of view (a second-person narrative). It is written and illustrated by James Daugherty. There are illustrations on every page. The illustrations are in a gold color. The story is 80 pages long.\\nSynopsis.\\nA little boy named Andy was so interested about lions that he went to the library and searched for a book about lions. That same night, his grandfather told him a bedtime story about lions. Andy was so fascinated about the story that he had a dream about lions that same night. The next day, on his way to school, Andy meets a real lion. The lion had a thorn stuck in his paw and Andy helps pull the thorn out. This action makes Andy and the lion friends. Later in the story, a circus comes to Andy's town and of course, Andy attends hoping to see his friend the lion. In the lion act, one of the lions jumps out of the cage, into the audience, right in front of Andy. Andy thinks it's his last day of life. But low and behold, it was his friend the lion! Andy and the lion recognize one another. The lion was the very same one Andy had helped that day take the thorn out of his paw. Andy and the lion rejoice in excitement of seeing one another again. When the crowd attempts to capture the lion, Andy protects it. The next day, there is a parade and Andy and the lion lead it. Andy receives an award for bravery. At the end of the story, Andy returns the book he borrowed from the library about lions, pulling his friend the lion behind him.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692492\",\"title\":\"Georgina Pinedo\",\"body\":\"\\nGeorgina Pinedo\\n\\nGeorgina Pinedo (born ) is a retired Argentine female volleyball player, who played as a wing spiker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with CA Boca Juniors.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692499\",\"title\":\"Cheerleading in Japan\",\"body\":\"\\nCheerleading in Japan\\n\\nCheerleading is a recognized sport in Japan that requires physical strength and athletic ability. Cheerleading is available at the junior high school, high school, collegiate, club, and all-star level. Teams can either be all female or coed featuring males and females. \\nCompetition guidelines.\\nCompetition routines are held on a 12-meter square mat in which the entire surface can be used. Each team must perform a 2 minutes and 30 seconds routine in which 1 minute and 30 seconds of the routine may contain music. Routines showcase different elements including sideline cheers, pyramids and stunts, dance, and gymnastics. There is also a division for group stunt competition where a group of five participants perform a 60–65 seconds routine of strictly stunts. There is also a division for partner stunt competition that includes one male and one female and one spotter. These routines last around 55–60 seconds and can only incorporate stunting.\\nHistory.\\nApril 24, 1988: Cheerleading Nation Championship in Japan (1st Japan Championships)\\nJuly 13, 1988: Foundation of Japan Cheerleading Association opens its association office in Akasaka, Minato-ku, Tokyo\\nDecember 23, 1989: 1st All Japan Student Championships\\nMay 5, 1990: 1st International Cheerleading All-Japan Championships\\nAugust 18, 1990: Japan Championships begin airing by NHK satellite broadcasting\\nJanuary 27, 1991: 1st All Japan High School Championships\\nDecember 15, 1991: 1st instructor qualification test conducted\\nJanuary 10, 1994: Japan Cheerleading Association renamed\\nAugust 22, 1998: International Cheerleading Federation inauguration\\nNovember 18, 2001: 1st World Championships, women’s Japanese team won the men and women mixed sector\\nFebruary 23, 2003: 1st All Japan club team Championships\\nNovember 15, 2003: 2nd Cheerleading World Championships, women Japanese team wins the men and women mixed sector\\nNovember 5, 2005: 3rd World Championship victory for the women’s Japanese team\\nApril 21, 2007: 1st Asia International Open Championship\\nNovember 17, 2007: 4th World Championships, women’s and men and women mixed Japanese teams won\\nNovember 28, 2009: 5th World Championships, men and women mixed Japanese teams won\\nNovember 26, 2011: 6th World Championships, men and women mixed Japanese teams won\\nNovember 23, 2013: 7th World Championships, women’s, men and women mixed, and group stunt teams participate\\nApril 1, 2014: Specialized cheerleading unit established in Tokyo High School Athletic Federation\\nAssociations and organizations.\\nOn July 13, 1988 the Foundation of Japan Cheerleading Association was founded with the purpose of supervising the cheerleading world as a competitive sport in Japan and to promote a healthy spread and promotion of the cheerleading competition. The Japan Cheerleading Association (JCA) also holds cheerleading and leadership workshops, holds domestic and international competitions, trains trainers, certifies competition equipment, and handles publications.\\nThe International Federation of Cheerleading (IFC) is also based in Tokyo, Japan and is the world governing body of cheerleading.\\nThe Asian Federation of Cheerleading (AFC) is the governing body of cheerleading in Asia and is one of the continental confederations that makes up the International Federation of Cheerleading. It was formed in 2007 and has its headquarters in Tokyo, Japan.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692502\",\"title\":\"John Leehane (cricketer, born 1921)\",\"body\":\"\\nJohn Leehane (cricketer, born 1921)\\n\\nJohn Leehane (20 October 1921 – 22 July 1991) was an Australian cricketer. He played one first-class cricket match for Victoria in 1950.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692503\",\"title\":\"Marianela Robinet\",\"body\":\"\\nMarianela Robinet\\n\\nMarianela Robinet (born ) is a retired Argentine female volleyball player, who played as a wing spiker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Central San Carlos.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692511\",\"title\":\"Maria Vicente\",\"body\":\"\\nMaria Vicente\\n\\nMaria Laura Vicente (born ) is a retired Argentine female volleyball player, who played as a wing spiker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with DAM La Rochette.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692517\",\"title\":\"Micaela Vogel\",\"body\":\"\\nMicaela Vogel\\n\\nMicaela Vogel (born ) is a retired Argentine female volleyball player, who played as a middle blocker. She was part of the Argentina women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with GEBA.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692518\",\"title\":\"John Leehane (cricketer, born 1950)\",\"body\":\"\\nJohn Leehane (cricketer, born 1950)\\n\\nJohn Leehane (born 11 December 1950) is an Australian former cricketer. He played eleven first-class cricket matches for Victoria between 1978 and 1981.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692522\",\"title\":\"Mahommah Gardo Baquaqua\",\"body\":\"\\nMahommah Gardo Baquaqua\\n\\nMahommah Gardo Baquaqua was a former slave, native of Zooggoo in Central Africa, a tributary kingdom of Bergoo kingdom. He worked in Brazil as a captive, however he escaped and fled to New York in 1847, assuring his freedom. He wrote a biography, published by American abolitionist Samuel Moore in 1854. His report is the only known document about slave trade written by a Brazilian former slave.\\nBiography.\\nBaquaqua was born in Djougou (current in Benin) between 1820 and 1830, in a proeminent Muslim trader family. He learned the Quran, literature and mathematics in an Islamic school. Still as an adolescent, he took part into the succession wars in Daboya, together with his brother, where he was captured and then rescued.\\nBack to Djougou he became servant of a local dignitary, perhaps the chief of Soubroukou, to whom he calls 'king'. The abuses he committed on that period made him target of an ambush, where he was imprisoned and transported to Dahomey; he would be embarked into a slave ship in 1845 and taken to Pernambuco, in Brazil.\\nBaquaqua was a slave in Olinda, Pernambuco around two years. His master was a baker, He worked in construction of houses, carrying stones, learned Portuguese and performed as “\\\"escravo de tabuleiro\\\"” (peddling slave). The cruelty of his Brazilian masters made him revert to alcoholism and to attempt suicide.\\nTaken to Rio de Janeiro, Baquaqua was incorporated to the crew of the trade ship \\\"Lembrança\\\", transporting goods to the southern provinces of Brazil. A coffee shipment to the United States, in 1847, was his passport to freedom. The ship arrived to New York harbor in June, where it was approached by local abolitionists, who encouraged him to escape from the ship. After the escape, however, he was imprisoned in the local jail, and only the help of the abolitionists (that facilitated his escape from prison) prevented he was returned to the ship. It was then sent to Haiti, where he lived with the reverend Judd, a Baptist missionary.\\nConverted and baptized in 1848 Baquaqua returned to the US due to the political instability Haiti was then; He studied at the New York Central College in McGrawville for almost three years. In 1854 he moved to Canada and his bibliography was published the same year by Samuel Downing Moore in Detroit.\\nIt not known what happened to Baquaqua after 1857. He was then in England and had turned to the American Baptist Free Mission Society to be sent as a missionary to Africa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692527\",\"title\":\"John Leehane\",\"body\":\"\\nJohn Leehane\\n\\nJohn Leehane can refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692536\",\"title\":\"SS Princess Ena (1906)\",\"body\":\"\\nSS Princess Ena (1906)\\n\\nTSS \\\"Princess Ena\\\" was a passenger vessel built for the London and South Western Railway in 1906.\\nHistory.\\nShe was built by Gourlay Brothers in Dundee and launched on 25 May 1906. She was built as a replacement for the Hilda, lost in the English Channel in 1905. She was built in four months, with the order being placed in December 1905. \\nShe was requisitioned by the Admiralty in 1915 and converted to a Q-ship. She returned to railway service at the conclusion of hostilities. \\nShe was acquired by the Southern Railway in 1923.\\nOn 3 August 1935 she caught fire on a passage from Jersey to St Malo and sank south of Jersey, Channel Islands.\\nThe crew were rescued by and .\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692547\",\"title\":\"C.braunii\",\"body\":\"\\nC.braunii\\n\\nC. braunii is a species abbreviation which may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692553\",\"title\":\"Philip Plotch\",\"body\":\"\\nPhilip Plotch\\n\\nPhilip Mark Plotch is an author, professor, and transportation planner. He is best known for leading efforts to rebuild the World Trade Center and his research on the politics and planning behind transportation megaprojects.\\nCareer.\\nAuthor and academic.\\nPhilp Mark Plotch writes articles and op-eds for academic and general publications including a regular column in \\\"Gotham Gazette\\\" about economic development and transportation issues. In 2015, Rutgers University Press published his book \\\"Politics Across the Hudson: The Tappan Zee Megaproject\\\".\\nDr. Plotch received the American Planning Association’s 2015 New York Metro Chapter journalism award for his in-depth research and hard hitting analysis behind the planning and politics of New York's transportation system. \\nPrevious winners have included Kate Ascher, Brendan Gill, Paul Goldberger, Ken Jackson, and Elizabeth Kolbert. Plotch is now under contract with Cornell University Press to write a book about the Second Avenue subway.\\nPlotch is an assistant professor of political science and the director of the master of public administration program at Saint Peter's University. He previously taught as an adjunct in the Department of Urban Affairs and Planning at Hunter College.\\nTransportation planner and World Trade Center Redevelopment Director.\\nAs director of World Trade Center redevelopment and special projects at the Lower Manhattan Development Corporation, after the September 11, 2001 terrorist attacks, Plotch developed new transportation programs, oversaw the design and construction of Lower Manhattan open spaces, and administered programs to rebuild structures.\\nBetween 1992 and 2005, as the manager of policy and planning at the headquarters of the Metropolitan Transportation Authority, Plotch led planning improvements for the New York metropolitan area's transportation system including the #7 subway line extension to the Hudson Yards, the Second Avenue subway, and intelligent transportation systems.\\nEducation.\\nPlotch received his master in urban planning degree from Hunter College and his Ph.D. in public and urban policy from the Milano School of International Affairs, Management, and Urban Policy at The New School. \\nCivic Leadership.\\nHe lives in the Radburn section of Fair Lawn, New Jersey, where he has published Fair Lawn News, served on the planning board and the economic development committee, and led efforts to revise election procedures and improve pedestrian safety.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692563\",\"title\":\"Elverskud\",\"body\":\"\\nElverskud\\n\\nElverskud is a ballad for soloist, choir and orchestra by Niels W. Gade from 1854.\\nGade's oeuvre contains many concert pieces of various lengths for orchestra, choir and soloists. Depending on which one counts, there are up to 16 of them, mostly referred to as \\\"cantata\\\". The longest of the pieces can take up to an hour. Some of the pieces are almost semi-operas or concert operas with dramatic development.\\nOrigins.\\nThe plot of the song comes from the Danish folk-ballad \\\"Elveskud\\\".\\nIn 1842 Gade wrote \\\"Agnete og Havfruerne\\\" for solo, female choir and orchestra with text by Hans Christian Andersen, and the same Andersen had in 1846 premiered the opera \\\"Liden Kirsten\\\" with music by Gade's father-in-law J. P. E. Hartmann. Both were inspired by Danish folksongs. This may have encouraged Gade to try his hand at the same set of motifs with a major concert piece.\\nThe earliest information on \\\"Elverskud\\\" is a mention in a letter from Gade in 1849, in which he wrote that he had allowed Hans Christian Andersen to prepare a text. For whatever reason it was never used by Gade, but a draft can be found among Andersen's papers. Some parts of the text were transferred to the final version. The next person who turned his hand to this text was Christian Molbech in whose papers there is a version that is closer to the final one, which by all accounts was developed by Gade's cousin Carl Andersen and the editor Gottlieb Siesby, while the atmospheric choral interlude \\\"I østen stiger solen op\\\" was written already in 1837 by B. S. Ingemann and received a tune by C. E. F. Weyse the same year.\\nThe first sheet of Gade's score is dated December 12, 1851, but in March 1853 the work is mentioned a letter to Gade from his wife, suggesting that he had no time to work on it. In early 1854 the work was published in a piano arrangement by a German publisher as \\\"Ballade nach dänischen Volkssagen\\\". On 30 March 1854, it was first performed at a subscription concert at the Musikforeningen, presented as a ballad for soloist, choir and orchestra.\\nPlot summary.\\nThe ballad is divided into five sections, a prologue, three \\\"acts\\\" and an epilogue.\\nThe music.\\nGade was apparently keen to emphasize that the music was not adaptations of folk songs, but rather all his own, but inspired by the ballads. Heard today, \\\"Elverskud\\\" fits the Danish romance style, particularly in the section where Oluf, the mother and the choir \\\"talk\\\" to each other in opera style.\\n\\\"Elverskud\\\" has been shown to be one of the most popular Gade's works. According to a statement in Inger Sørensen's book, it was performed at least 184 times in Gade's life, a success unparalleled by other Danish composers. The work remains popular today, and two of the songs, \\\"I østen stiger solen op\\\" and \\\"Så tit jeg rider mig under ø\\\" are often played as single pieces.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692564\",\"title\":\"John Watters (cricketer)\",\"body\":\"\\nJohn Watters (cricketer)\\n\\nJohn Watters (6 October 1924 – 2 August 2006) was an Australian cricketer. He played one first-class cricket match for Victoria in 1950.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692571\",\"title\":\"Abigael Kipkemboi\",\"body\":\"\\nAbigael Kipkemboi\\n\\nAbigael Kipkemboi (born ) is a retired Kenyan female volleyball player, who played as a middle blocker. She was part of the Kenya women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Kenya Pipeline.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692578\",\"title\":\"John Watters\",\"body\":\"\\nJohn Watters\\n\\nJohn Watters can refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692583\",\"title\":\"Margaret Mukoya\",\"body\":\"\\nMargaret Mukoya\\n\\nMargaret Mukoya (born ) is a retired Kenyan female volleyball player, who played as a middle blocker. She was part of the Kenya women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Telkom Kenya.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692587\",\"title\":\"Philister Sang\",\"body\":\"\\nPhilister Sang\\n\\nPhilister Sang (born ) is a retired Kenyan female volleyball player, who played as a wing spiker. She was part of the Kenya women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Kenya Pipeline.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692590\",\"title\":\"Salome Wanjala\",\"body\":\"\\nSalome Wanjala\\n\\nSalome Wanjala (born ) is a retired Kenyan female volleyball player, who played as a wing spiker. She was part of the Kenya women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Kenya Commercial Bank.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692593\",\"title\":\"Women's Resource & Action Center\",\"body\":\"\\nWomen's Resource & Action Center\\n\\nThe Women's Resource and Action Center, also known as WRAC, is an organization dedicated to the equality and well-being of female students at the University of Iowa and members of the Iowa City community. It provides resources and information that are helpful for the everyday activities of women and as a source of entertainment and subject matter relevant to women.\\nHistory.\\nOriginally called the Women's Liberation Front in 1971, WRAC was established to promote women's rights and equality, and allow women to meet other women as friends, classmates, coworkers, etc. WRAC's website states that members of the organization \\\"met to discuss their shared oppression as women and search for solutions to their common problems.\\\" In 1974, the group changed its name to that which it currently holds: Women's Resource and Action Center.\\nWhen the organization was created, the members made it an issue to function collectively rather than have an officer team or executive board, to encourage women's empowerment and not be discriminatory. After WRAC had successfully established itself as a feminist group in the Iowa City community, it began to reach out to other oppressed groups, diversifying their human rights values to expand to groups including \\\"people of color, people with disabilities, the elderly, and lesbians and gays.\\\" Originally, as a source of education and entertainment, WRAC held workshops and programs such as counseling, divorce rights, abortion and birth control, women's health information, women's studies, self-defense, economics, etc. These workshops and programs were beneficial to all women involved with WRAC because of the knowledge they were obtaining.\\nIn the early 1980s, the Women Against Racism Committee (WAR) was formed as a branch of WRAC, and populated primarily by the members of WRAC, as well as other women in the Iowa City community. The original purpose of the organization was to critique the racism WRAC members perceived, and to raise awareness of the harmful effects of internalized racism. These actions evolved into the study of all forms of oppression, including sexism, disability-related discrimination, and homophobia, especially among colored women. As WAR grew in popularity, it brought in renowned speakers, such as Winona LaDuke and Natalie Wong to the University of Iowa campus to speak on behalf of the oppressed populations. In 1988, WAR held its first and only national conference, \\\"Parallels and Intersections: Racism and Other Forms of Oppression.\\\" Participation in the Women Against Racism Committee declined throughout the 1990s until it was shut down in 1998.\\nPresent day.\\nMission and values.\\nWRAC is a feminist organization designed to promote equality between groups, with a more focused goal on establishing women's rights. Their values include inclusivity, collaboration, diversity, effectiveness, safe and supportive environment, and respect.\\nPrograms.\\nWRAC offers services including counseling, scholarships, support groups, and a library containing resources that address financing, health care (women's health, abuse recovery, eating disorders), gender studies, LGBT studies, marriage, sex, parenting, feminism theory, domestic violence, single parenting, adoption, and literature. The Center has also helped teach bystander intervention classes that were open to all members of the community to combat the issue of street harassment in Iowa City in 2014. To combat the issue of street harassment in Iowa City in 2014, the Center helped teach bystander intervention classes that were open to all members of the community \\nStudent life.\\nWRAC has organized \\\"Take Back The Night\\\" rallies, an event including a march in advocation of sexual violence victims and an open-mic time for victims to tell their stories that has been ongoing since 1979. Since 2014 the University of Iowa has had a new focus on decreasing sexual assaults on campus. Because of this, WRAC has increased their staff and resources and has been able to reach more students. The organization has grown too large for its current location on Madison Street where the Center has been for 39 years. With the development of these events, the organization plans to move to a new location (the Bowman house on Clinton Street in Iowa City) in 2016.\\nSafety.\\nWRAC has been providing safety resources for years. Whether it be in their office or online, there are many different options. For example, on their website, it lists hotlines for immediate assistance as well as counselling and care for victims of sexual assault.\\nOne of WRAC's projects to fight sexual violence and increase safety was the Street Lighting and Safety Project. WRAC acted as an advisor to the Association of Student Women (ASW) and collaborated with them to complete the project. The goal of the Street Lighting and Safety Project was to have more street lights put in place around Iowa City to provide more lighting during the evenings and at night. WRAC and ASW began the project in 1976 and it was completed in 1980. The lights put up around the campus acted as a precursor to current lights and blue safety lights around campus today.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692596\",\"title\":\"Jetzabel del Valle\",\"body\":\"\\nJetzabel del Valle\\n\\nJetzabel del Valle (born ) is a Puerto Rican female volleyball player, who played as a middle blocker. She was part of the Puerto Rico women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. and at the 2006 FIVB Volleyball Women's World Championship in Japan. She played with Criollas de Caguas and Humacao.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692621\",\"title\":\"Harry Knight (disambiguation)\",\"body\":\"\\nHarry Knight (disambiguation)\\n\\nHarry Knight was an American racecar driver.\\nHarry Knight may also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692630\",\"title\":\"Fiber modification\",\"body\":\"\\nFiber modification\\n\\nFibre modification is a research field in which researchers aim at developing and applying technologies to impart new properties to natural fibres such as those in paper, in order to increase their functionality. Research areas in this field include many different technologies, amongst which the chemical modifications of fibres are widely used. One important sector of application of the chemical modifications is the treatment of wood for giving it enhanced properties such as higher mechanical properties, water impermeability, less hygroscopicity, bacterial and fungal resistance. Transferring and adapting the technical knowledge on fibre modification available for the wood sector to the recycled paper sector is an innovative use of these chemical treatments which has been the subject of studies that have been carried out within an EU co-funded project called Fibre+.\\nThe project Consortium included members representing Paper and Packaging European Associations (CEPI, FEFCO), and research institutes specialized in the wood and paper for packaging sector and paper and packaging companies. The focus of the project was on chemical modification of the paper made recycled fibres, investigating the possibility to transfer wood fibre modifications technique from the wood sector to paper and packaging sector. The aim was to enhance the properties of papers and of the packaging, as the recycling process causes the deterioration of fibres.\\nThe chemical modification of recycled fibres aimed at the creation of a new generation of packaging papers characterized for being more recyclable, less hygroscopic, stiff and durable. The high recycling rate of papers in Europe (which is at the level of 72%) and the consequent importance that recycling has for the circular economy, were at the basis of this study. Paper products form part of an integrated carbon cycle based on the photosynthetic conversion of water, carbon dioxide, nutrients and solar energy into renewable wood-based biomass. Once consumed, paper may be recovered and used again either as a source of secondary fibres, to produce recycled paper or as bio-fuel. Fibre packages or corrugated containers made from corrugated board were the ones that were dealt with in the project, as they are considered as being the most prominent structural application of paper.\\nChemical modification attempts.\\nFibre modification with chemicals or enzymes had been investigated in the production of fibreboards. Fibre modification applying steam (steam-exploded wood) has been proved an efficient pre-treatment method in producing thermoplastic composites.\\nCurrent theories for interfibre bonding during papermaking process are based on general recognition of the hydrogen bonding model. Consequently, all effort for boosting fibre strength is connected to mechanical beating of fibres in order to generate more flexible and fragmented fibres for increased bonding areas. As a consequence, significant drawbacks are obtained in terms of water retention ratio resulting in poor dewatering behaviour and high energy consumption. New mechanisms for interpretation and control of interfibre bonding are still upcoming. One way to overcome these drawbacks could be the molecular coating of cellulose fibres using polymers targeted on entropy controlled mixing of polymers and cellulose gel resulting in higher bonding forces. Theoretical results as well as experimental data on how application of polymeric layers (e.g. carboxymethyl cellulose) and enzymes on cellulose fibres can lead to sheets of high bonding strength without any mechanical beating have been already presented. However, these attempts were still far from any industrial implementation and their application would have been costly and would not solve the problem of raw material availability.\\nObjective.\\nBased on this state-of-the-art, the objective of the project was to modify and thus improve the characteristics of different types of recovered fibres used for the production of a variety of packaging grade papers used as linerboard and corrugating medium for corrugated board manufacturing in Europe. Information on the actual furnish characteristics and composition of packaging materials is expected to help European packaging industry to evaluate its sources of supply and to adopt suitable methods and processes to improve the available resources in an optimal manner. In the case of packaging, scientific technical knowledge of practical industrial relations between fibre characteristics, paper properties and corrugated board properties also is needed.\\nFrom wood fibre modification to paper technology chemical wood modification aims at altering the structure of the cell wall matrix. Wood\\nproperties are improved considerably by converting hydrophilic OH-groups into larger more hydrophobic groups. Also the physical fixation of modifying chemicals in the cell wall matrix can considerably change the wood properties. In addition to a hydrophobing effect, the treatments reduce the volume of cell wall nano-pores and thus decrease the incorporation of water molecules into the cell wall matrix. On a macroscopic scale, wood modification can change important properties of the wood including biological durability (resistance against fungi), dimensional stability, hardness and UV-stability. \\nSince paper is produced from wood fibres, it was possible to transfer some of the developments achieved in wood technology to paper technology. Several technologies (e.g. chemical modification, nano-scale celluloses, polyelectrolytes, functional polymers based on cellulose, hemicelluloses and starch) were researched and used by different research groups around Europe (e.g. PTS and University of Goettingen, Germany; Kungliga Tekniska högskolan (KTH), Sweden.) There is already well established knowledge on chemical fibre modification of recycled fibres. Adamopoulos and Mai (2011) modified recycled fibres with N-methylol compounds and glutaraldehyde with significant improvement on fibre characteristics and paper sheet performance. Laboratory sheets manufactured with a variety of chemically modified recycled fibres were found to be superior in stiffness and hygroscopic properties than these manufactured from unmodified ones. The intention of the Fibre+ project is to build on the existing knowledge on fibre modification for adapting, implementing and disseminating this innovative technology in European paper SMEs.\\nResults of the Fibre+ project on recycled fibres for packaging paper and information on potential developments of the Fibre+ concept can be found pn the Fibre+ website including scientific articles that have been published as a consequence of the RTD work that has been carried out during the project. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692633\",\"title\":\"Bullcroft Main Colliery F.C.\",\"body\":\"\\nBullcroft Main Colliery F.C.\\n\\nBullcroft Main Colliery F.C. was an English association football club based in Carcroft, Doncaster, South Yorkshire.\\nHistory.\\nLittle is known of the club other than that it competed in the FA Cup in the 1910s and 1920s.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692645\",\"title\":\"Lourdes Isern\",\"body\":\"\\nLourdes Isern\\n\\nLourdes Isern (born ) is a retired Puerto Rican female volleyball player, who played as a libero. She was part of the Puerto Rico women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Gigantes de Carolina.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692651\",\"title\":\"Daniel Braverman\",\"body\":\"\\nDaniel Braverman\\n\\nDaniel Braverman (born September 28, 1993) is an American football wide receiver who is a redshirt junior for the 2015 Western Michigan Broncos football team. During the 2015 regular season, he had 103 receptions for 1,266 yards and 12 touchdowns and ranked second in the NCAA Division I FBS in receptions.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692663\",\"title\":\"Fitzsimmons, Alberta\",\"body\":\"\\nFitzsimmons, Alberta\\n\\nFitzsimmons is an unincorporated locality in northwest Alberta within the County of Grande Prairie No. 1. It is located approximately 30 kms north-east of the City of Grande Prairie.\\nThe locality of Fitzsimmons formed around the Fitzsimmons School District No. 4500 which was approved July 7, 1930 for the area north of Bezanson and west of the Smoky River. It was named after a homesteader in the district, Scotty Fitzsimmons. Two years later, a post office was established in the home of C. Milnar on the NW quarter of Township 73, Range 3, West of the 6th Meridian.The one-room school closed in 1956 and the students were bussed to the school at Bezanson. In 1951, the post office also closed and there are no public buildings remaining to mark the locality of Fitzsimmons. Information on this community and the people who lived there can be found in Smoky River to Grande Prairie. (4)\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692668\",\"title\":\"Coombe Monthly\",\"body\":\"\\nCoombe Monthly\\n\\nCoombe Monthly is an online publication covering news in the Royal Borough of Kingston upon Thames, South West London. It is run by local residents in the borough, and is free to access.\\nIt is available as a free, paperless \\\"Coombe Monthly\\\" e-newspaper, accessible from their website and delivered straight to email inboxes each morning.\\nIt has an online readership of around 11,000 readers per month, although naturally this figure varies from month to month.\\nThe paper also organises local community events, with the aim of \\\"Engaging local residents in the issues around them\\\", known as 'Kingston Question Time' events. These events donate all events to Love Kingston, a local charity in the Royal Borough of Kingston Upon Thames, donating to 'Pathways out of Poverty'.\\nThe newspaper is run by James Giles, a New Malden resident, who has been listed by the Surrey Comet as a local Unsung Hero \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692671\",\"title\":\"Dolly Melendez\",\"body\":\"\\nDolly Melendez\\n\\nDolly Melendez (born ) is a retired Puerto Rican female volleyball player, who played as a universal . She was part of the Puerto Rico women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Criollas de Caguas.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692674\",\"title\":\"Xiomara Molero\",\"body\":\"\\nXiomara Molero\\n\\nXiomara Molero (born ) is a retired Puerto Rican female volleyball player, who played as a setter. She was part of the Puerto Rico women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Criollas de Caguas.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692680\",\"title\":\"Jeffery Tribble\",\"body\":\"\\nJeffery Tribble\\n\\nJeffery Tribble is an ordained elder in the African Methodist Episcopal Zion (A.M.E. Zion) Church and a professor of ministry with research interests in Practical Theology, Congregational Studies and Leadership, Ethnography, Evangelism and Church Planting, Black Church Studies, and Urban Church Ministry. Academics and professionals in these fields consider him a renowned thought leader. Tribble's experience in pastoral ministry allows for his work to bridge the gap between academic research and practical church leadership.\\nCareer.\\nTribble received a B.S. from Howard University (1981), a Black Minister’s Program Certificate from Hartford Seminary (1985), a M.Div. from Garrett-Evangelical Theological Seminary (1990), and a Ph.D. from Northwestern University (2002). He began his work in the African Methodist Episcopal Zion Church as a minister of membership development at Martin Temple A.M.E. Zion Church in Chicago, Illinois (1990–1991). Tribble was ordained as an elder in 1992, when he then served as the pastor at St. Andrew A.M.E. Zion Church in Gary, Indiana (1991–1997). He has also served in the A.M.E. Zion Church as a pastor at St. Mark church in Chicago, Illinois (1997–2000), co-pastor at New vision Church in Suwanee, Georgia (2007–2008), Associate pastor at Greater Walters church in Chicago, Illinois (2006–2007), the minister of evangelism and men’s ministry and founding Dean of the Life Development Institute at Martin Temple Church in Chicago, Illinois (2000–2006), and as the presiding elder for the A.M.E. Zion Church for the Augusta District (2008–2013) and the Atlanta District (2013–present).\\nHe began his career as an academic at Garrett-Evangelical Theological Seminary where he served as adjunct faculty (1999–2000, 2008 and 2009), Instructor of Congregational Ministries and Director of Teaching Parishes and Congregational Research (July 2000 – June 2003) and Assistant Professor of Congregational Leadership and Director of the Center for the Church and the Black Experience (July 2003 – June 2007). He has been a lecturer at Apex School of Theology in Durham, NC (2009). He joined the faculty of Columbia Theological Seminary as Assistant Professor of Ministry (2007–2012). \\nHe currently still serves at Columbia Theological Seminary where he is now the Associate Professor of Ministry. He is also Presiding Elder for the Atlanta District, Georgia Annual Conference and Candidate for Bishop in the African Methodist Episcopal Zion Church for 2016.\\nThought.\\nTribble is a strong proponent of the transformative leadership idea. He has written extensively about how leaders, especially those who work in the church, should be open to transformation themselves as they transform the community they are leading. He writes about this specifically from the perspective of the Black Church in two of his books, Transformative Pastoral Leadership in the Black Church and Joining Jesus: A Class Manual for initiation into Christian Discipleship and Welcome into the African Methodist Episcopal Zion Church. From this point of view he emphasizes his research on how the Black Church “must continue its historic mission of being an instrument of survival, elevation, and liberation for its people.” He however does not limit his research just to the Black Church, he pushes for transformative leadership between various religious traditions as well. He has published numerous books, articles, and chapters that are used in this context by people and institutions across the world and across denominational lines.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692697\",\"title\":\"Don Mathieson (lawyer)\",\"body\":\"\\nDon Mathieson (lawyer)\\n\\nDonald Lindsay Mathieson is a Waikanae-based New Zealand lawyer and lay Anglican.\\nMathieson studied at Victoria University of Wellington, playing chess and hockey before winning a Rhodes Scholarship in 1959.\\nIn addition to spending more than forty years in private practice, Mathieson was Crown Counsel in the Crown Law Office. From 2010 to 2015 he was President of the Film and Literature Board of Review.\\nMathieson, an active lay Anglican, edited \\\"Faith at work\\\" in 2001, 'arguing that Christians should practise their faith at work.' He wrote on the subject of same-sex marriage, in context of the Anglican church adapting to the Marriage (Definition of Marriage) Amendment Act 2013:\\nIt is time to speak forthrightly in support of the clear scriptural witness about the sinfulness of homosexual acts and the position adopted without dissension by Roman Catholic, Protestant and Orthodox Churches alike for nearly two thousand years.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692710\",\"title\":\"Tatiana Gorchkova\",\"body\":\"\\nTatiana Gorchkova\\n\\nTatjana Gorschkova (born ) is a retired Russian female volleyball player, who played as a wing spiker. She was part of the Russia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany and at the 2004 FIVB World Grand Prix. On club level she played with VK Uralotschka-NTMK.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692716\",\"title\":\"List of Chicago Maroons men's basketball head coaches\",\"body\":\"Source\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692726\",\"title\":\"Anjela Gourieva\",\"body\":\"\\nAnjela Gourieva\\n\\nAnschela Gurijeva or Anjela Gourieva (born ) is a retired Russian female volleyball player, who played as a universal . She was part of the Russia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with VK Uralotschka-NTMK.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692737\",\"title\":\"Elizabeth Fitzgerald (volleyball)\",\"body\":\"\\nElizabeth Fitzgerald (volleyball)\\n\\nElizabeth Fitzgerald (born ) is a retired American female volleyball player, who played as a setter. She was part of the United States women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692738\",\"title\":\"Alfonso Sosa\",\"body\":\"\\nAlfonso Sosa\\n\\nLuis Alfonso Sosa (born 5 October 1967) is a Mexican former footballer who last played for Quéretaro and current manager of Ascenso MX club Necaxa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692750\",\"title\":\"Jennifer Flynn\",\"body\":\"\\nJennifer Flynn\\n\\nJennifer Flynn (born ) is a retired American female volleyball player, who played as a setter. She was part of the United States women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692783\",\"title\":\"BMCC Tribeca Performing Arts Center\",\"body\":\"\\nBMCC Tribeca Performing Arts Center\\n\\n is a performing arts venue located in Lower Manhattan inside the BMCC (Borough of Manhattan Community College) on 199 Chambers Street, New York, NY. BMCC Tribeca Performing Arts Center's two main theater spaces are Theatre One (a 913-seat theater) and Theatre Two (which is 260 seats). BMCC Tribeca Performing Arts Center programming includes music concerts, children's theater, stand-up acts, film retrospectives as well as local and international dance companies. BMCC Tribeca Performing Arts Center has been one of the venues for the annual Tribeca Film Festival.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692801\",\"title\":\"Smoky Heights, Alberta\",\"body\":\"\\nSmoky Heights, Alberta\\n\\nThe locality of Smoky Heights, on the west bank of the Smoky River north-east of the City of Grande Prairie, formed around the Smoky Heights post office, which was established June 1, 1923 in the home of Edith and Clarence Field. The post office lasted only a few years, in 1927 it was moved to Bezanson, but the area retained the name. On November 14, 1928, the Rivertop School District 4371 was approved. This school was much larger than the normal one-room school as it was intended to serve as a community center and included accomodations for the teacher as well. The school closed in 1955 and the students were bussed to the consolidated school at Teepee Creek.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692812\",\"title\":\"Jennifer Hiller\",\"body\":\"\\nJennifer Hiller\\n\\nJennifer Hiller (born ) is a retired Australian female volleyball player, who played as a libero. She was part of the Australia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Monash University.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692818\",\"title\":\"Twara Desai\",\"body\":\"\\nTwara Desai\\n\\nTwara Desai is an Indian actress. She acted in a Tamil language film \\\"Saivam\\\" (2014).\\nExternal links.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692820\",\"title\":\"Tolotear Lealamanua\",\"body\":\"\\nTolotear Lealamanua\\n\\nTolotear Lealamanua (born ) is a retired Australian female volleyball player, who played as a middle blocker. She was part of the Australia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with University of Technology, Sydney.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692821\",\"title\":\"Sandra Vehrencamp\",\"body\":\"\\nSandra Vehrencamp\\n\\nSandra Vehrencamp (born c. 1947), is a scientist, teacher, and mentor that specializes in Ornithology, with a geographical focus on the avian population of Costa Rica. She served as a faculty member of Cornell University's Lab of Ornithology and Department of Neurobiology and Behavior and taught graduate students while conducting research until retiring as of October 2010. She currently resides in Ithaca, New York, with her husband, Jack Bradbury.\\nEducation.\\nShe graduated from Crescenta Valley High School in La Crescenta, California, in 1965. She went on to receive her bachelors in Zoology with honors from the University of California at Berkeley in 1970, and her Ph.D in Animal Behavior from Cornell University in 1976. After her extensive education, she started her career in research.\\nPersonal and early life.\\nDr. Vehrencamp grew up in La Crescenta, California, and attended Crescenta Valley High School. During the time period Sandra Vehrencamp was born, women’s scientific talents were substantially under-appreciated. She received her high-school diploma in 1965 and went on to pursue a higher education from there. However, before 1950, women earned less than 10% of Bachelor’s in the STEM fields and less than 5% of the PhDs in these fields. She grew up with this stigma surrounding her and still managed to attend the University of California at Berkeley for her bachelors and Cornell University for her doctorate. The percentages of women receiving bachelors and PhDs were steadily rising throughout her young life, although the year she graduated from Cornell University still less than 10% of doctorates were received by women. Scientific women of the time were quite rare, and this fact highlights her dedication and passion for science, as is noted by other scientists. While in the midst of her education at Cornell University, she met her spouse-to-be, Jack Bradbury. They had their first child in 1975, just a year before Vehrencamp received her PhD and started her career. Their second child was born in 1979, when Vehrencamp was just beginning to settle into a career path. These few starting years were hectic for Vehrencamp, and it has been noted how impressive it is that she managed to keep furthering her career the way she did. Her oldest daughter, Kristin Nobel, is currently married and has two kids, her family living in San Diego. Her youngest, Katrina Bradbury, lives in Ann Arbor, Michigan, and is a Nutrition and Wellness Coach at The Sacred Healing Room at Your Natural Solutions LLC.\\nCareer.\\nSince 1976 she worked with University of California at San Diego and Cornell University conducting intensive research about birds and their behavior, specifically song patterns and mating habits. She taught animal communication research methods in animal behavior to graduate students during her time at Cornell University, and currently holds a professor emerita position there. Vehrencamp is said to have been an outstanding mentor, teacher, and scientist by her graduate students. Additionally, she worked with the Laboratory of Ornithology Bioacoustics Research Program and contributed to the bird call section, specifically that of Costa Rican wrens; she still holds an emerita professor appointment there as well. Throughout her career she published over 65 papers, which have been cited more than 2,400 times, and wrote 19 book chapters. She collaborated with her husband, Jack Bradbury - an ornithologist as well - consistently during her time as a scientist. They co-wrote a textbook, Principles of Animal Communication, published in 1998. It is a widely used work that combines physics, chemistry, neurobiology, cognitive science, evolutionary biology, behavioral ecology, and economics to delve deeply into animals and how they signal and communicate with one another. Its importance is highlighted by the fact that it has been cited more than 1,000 times. This textbook is highly regarded in the scientific community and is revered as the standard reference of the animal behavior world.\\nNotable research ventures.\\nVehencamp spent decades of her life studying ornithology, and retired fully accomplished in 2010. Her detailed work earned several awards soon after.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692827\",\"title\":\"Adrienne Marie\",\"body\":\"\\nAdrienne Marie\\n\\nAdrienne Marie (born ) is a retired Australian female volleyball player, who played as a wing spiker. She was part of the Australia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with University of Technology, Sydney.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692830\",\"title\":\"Speculum principis (Skelton book)\",\"body\":\"\\nSpeculum principis (Skelton book)\\n\\nSpeculum Principis (A First Mirror) was a guide to 'proper princely behaviour' written by John Skelton in August 1501. Skelton was a well-known poet and had been appointed as tutor to Henry VIII's second son, Prince Henry, who would later reign as Henry VIII of England. A copy is now in the British Museum, which may or may not be exactly the same as the one given to Henry.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692833\",\"title\":\"Rowena Morgan\",\"body\":\"\\nRowena Morgan\\n\\nRowena Morgan (born ) is a retired Australian female volleyball player, who played as a libero. She was part of the Australia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with University of Technology, Sydney.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692838\",\"title\":\"Eileen Romanowski\",\"body\":\"\\nEileen Romanowski\\n\\nEileen Romanowski (born ) is a retired Australian female volleyball and beach volleyball player, who played as a wing spiker. She was part of the Australia women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Mount Lofty. In 2007 she played beach volleyball with Becchara Palmer.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692845\",\"title\":\"Prosopopeia Britanniae\",\"body\":\"\\nProsopopeia Britanniae\\n\\nProsopopeia Britanniae is a ten-page poem written by Desiderius Erasmus in Latin. It was written for ‘the most illustrious prince, Duke Henry’, the future Henry VIII of England.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692873\",\"title\":\"Andrea Badoer\",\"body\":\"\\nAndrea Badoer\\n\\nAndrea Biagio Badoer or Andrea Badoer (fl.1509) was the Venetian ambassador to the Court of Henry VIII of England. His dispatches are today read in the \\\"Calendar of State Papers, Venice\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692878\",\"title\":\"Nawaz Sharif Kidney Hospital Swat\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692894\",\"title\":\"Myoporum velutinum\",\"body\":\"\\nMyoporum velutinum\\n\\nMyoporum velutinum is a plant in the figwort family, Scrophulariaceae and is endemic to a small area near Esperance in Western Australia. It has similar flowers and fruits and grows in similar environments to \\\"Myoporum tetrandrum\\\" but differs from it and all other members of the genus by having hairs on the outer edge of the petals.\\nDescription.\\n\\\"Myoporum velutinum\\\" is a shrub which sometimes grows to a height of and which has hairs on the stem and leaves but lacks the warty surface of many other myoporums. The leaves are arranged alternately and are lance shaped, usually long, wide, soft, covered with soft hairs, darker green on the upper surface and have a prominent mid-vein.\\nThe flowers are borne in leaf axils, usually in groups of 1 to 5 on stalks long. There are 5 triangular, pointed sepals and 5 petals forming a bell-shaped tube. The petals are white or pale lilac, sometimes spotted inside the tube and on the base of the lobes. The tube is long and the lobes are about the same length. The tube is hairy on its inner and outer surfaces and there are 4 stamens which extend slightly beyond the petal tube. The fruit that follows flowering is an oval-shaped drupe, about .\\nTaxonomy and naming.\\n\\\"Myoporum velutinum\\\" was first formally described by taxonomist Bob Chinnock in \\\"Eremophila and allied genera: a monograph of the plant family Myoporaceae\\\" in 2007 from a specimen collected near Condingup. The specific epithet (\\\"velutina\\\") is a Latin word meaning \\\"velvety\\\".\\nDistribution and habitat.\\n\\\"Myoporum velutinum\\\" occurs in a small area near Cape Le Grand where it grows along creeks in \\\"Melaleuca\\\" woodland.\\nConservation.\\n\\\"Myoporum velutinum\\\" has been classified as \\\" Threatened Flora (Declared Rare Flora — Extant)\\\" by the Government of Western Australia Department of Parks and Wildlife meaning that it is \\\"likely to become extinct or is rare, or otherwise in need of special protection\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692897\",\"title\":\"Thomas Spinelly\",\"body\":\"\\nThomas Spinelly\\n\\nThomas Spinelly was Henry VIII of England’s representative in the Low Countries in 1510.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692899\",\"title\":\"San Antone (film)\",\"body\":\"\\nSan Antone (film)\\n\\nSan Antone is a 1953 American Western film directed by Joseph Kane and written by Steve Fisher. The film stars Rod Cameron, Arleen Whelan, Forrest Tucker, Katy Jurado, Rodolfo Acosta and Roy Roberts. The film was released on February 15, 1953, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692913\",\"title\":\"VTEC (disambiguation)\",\"body\":\"\\nVTEC (disambiguation)\\n\\nVTEC is Variable Valve Timing and Lift Electronic Control, a system developed by Honda for combustion engines.\\nVTEC or VTech may also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692933\",\"title\":\"Zero Gradient Synchrotron\",\"body\":\"\\nZero Gradient Synchrotron\\n\\nThe Zero Gradient Sychrotron (ZGS), was a weak focussing 12.5 GeV proton accelerator \\nthat operated at the Argonne National Laboratory in Illinois from 1964 to 1979.\\nIt enabled pioneering experiments in particle physics, in the areas of \\nOther noteworthy features of the ZGS program were the large number of university-based users and the pioneering development of large superconducting magnets for bubble chambers and beam transport. \\nThe hardware and building of the ZGS were ultimately inherited by a spallation neutron source program, the Intense Pulsed Neutron Source (IPNS).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692935\",\"title\":\"Sonia Benedito\",\"body\":\"\\nSonia Benedito\\n\\nSonia Benedito (born ) is a retired Brazilian female volleyball player, who played as a wing spiker. She was part of the Brazil women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Fluminense Rio de Janeiro.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692938\",\"title\":\"World's Fair (album)\",\"body\":\"\\nWorld's Fair (album)\\n\\nWorld's Fair is the third album released by Julian Lage as leader. Recorded on a 1939 Martin 000-18 with no overdubbing, it is the first solo album he has recorded.\\nBackground.\\nLage said that a large part of the inspiration for this album was the classical guitarist Andrés Segovia, and that it \\\"draw[s] from the sonic fingerprint of early radio recordings mixed with the short form structures of some of my favorite classical and folk music\\\". While he still considers the album to be jazz-oriented, he stated that he stayed away from swing-style music \\\"because I didn't think solo guitar was the format for me to play swing\\\".\\nOverview.\\nThe first track, \\\"40's\\\", which was the only song given a pre-release, is of a medium tempo and alternates between the keys of F and D. \\\"Peru\\\" opens with some light picking before settling into the actual tune, and \\\"Japan\\\" begins \\\"with an angular, almost discordant sound, but enters into a laid back groove, occasionally interrupted by unsettled harmonies\\\".\\n\\\"Double Stops\\\", as the title implies, makes extensive use of double stops throughout the course of the song. \\\"Gardens\\\" was described as making Lage sound like \\\"a flatpicking Andrés Segovia\\\", due to the fast bluegrass picking technique employed.\\nOnly two tracks on the album were not composed by Lage. The first, \\\"Where or When\\\", a Rodgers and Hart show tune, was described as \\\"[lingering] like a comfortable memory.\\\" The other is a folk song, \\\"Red Prairie Dawn\\\", which \\\"puts his folk roots on display.\\\"\\nReception.\\nMark F. Turner's \\\"AllAboutJazz\\\" review said that \\\"It's the perfect showcase of his fine abilities on acoustic guitar.\\\" \\\"Relix\\\" review said that \\\"the music's austerity magnifies its grandeur\\\".\\nThe review from \\\"The Stanford Daily\\\" stated that \\\"Fans of Lage's more experimental work...may be disappointed to find that \\\"World's Fair\\\" lacks the abstract sophistication found in many works of contemporary jazz. But it's fairly clear that Lage was not writing to push musical boundaries, but rather to pay homage to the diverse range of music that has shaped his own playing and taste.\\\" The Sun Gazette commented \\\"Lage [serves] the songs rather than his ego...the brilliance is in how simple and effortless he makes it sound\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692943\",\"title\":\"Fabiana Berto\",\"body\":\"\\nFabiana Berto\\n\\nFabiana Berto (born ) is a retired Brazilian female volleyball player, who played as a setter. She was part of the Brazil women's national volleyball team at the 2001 FIVB World Grand Prix and 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Club Atlético Estudiantes de Paraná.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692953\",\"title\":\"Peter Johnston (Wisconsin politician)\",\"body\":\"\\nPeter Johnston (Wisconsin politician)\\n\\nPeter Johnston was a member of the Wisconsin State Assembly.\\nBiography.\\nJohnston was born on January 19, 1831 in Dunblane, Scotland. Later, he resided in Milwaukee, Wisconsin before settling in Manitowoc, Wisconsin in 1857. He died on October 3, 1904.\\nPolitical career.\\nJohnston was a member of the Assembly during the 1877 session. From 1870 to 1872, he was Mayor of Manitowoc. Additionally, Johnston was a member of the Manitowoc City Council and of the Manitowoc County, Wisconsin Board of Supervisors. He was a Republican.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692955\",\"title\":\"Marina Daloca\",\"body\":\"\\nMarina Daloca\\n\\nMarina Daloca (born ) is a retired Brazilian female volleyball player, who played as a middle blocker. She was part of the Brazil women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Minas Tênis Clube.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692956\",\"title\":\"Luciana Nascimento\",\"body\":\"\\nLuciana Nascimento\\n\\nLuciana Nascimento (born ) is a retired Brazilian female volleyball player, who played as a wing spiker. She was part of the Brazil women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with EC Pinheiros.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692957\",\"title\":\"Maria Fidecaro\",\"body\":\"\\nMaria Fidecaro\\n\\nMaria Fidecaro is an Italian theoretical physicist with a focus on particle physics. Her main laboratory where she spent most of her career doing research was CERN- European Organization for Nuclear Research.\\nMaria received her degrees in Physics from the University of Rome in the 1940s.\\nIn 1954, she obtained a fellowship from the International Federation of University Women to do research at the University of Liverpool. In 1956, Maria obtained the CERN fellowship and worked at CERN until 2012. She is among the pioneering scientists at CERN since the establishment of the institution in 1954.\\nPersonal life.\\nMaria Fidecaro was born in Italy. In July 1955, she married Giuseppe Fidecaro- a fellow physicist at CERN who was also doing research with Maria at University of Liverpool. They had a family of four children.\\nIn summer 1956 when Maria obtained CERN fellowship, the couple moved to Geneva. They were the first few scientists among a total of 300 to 400 staff when CERN was first founded. During their time, few people could do experiments at CERN. Maria started with a group of only three scientists.\\nMaria also recalled the effect of war in her early life. Italy entered World War II in 1940, the time when Maria was earning her degree at the University of Rome. The aftermath of war destroyed Italy’s economy and society. Not until the 1950s and 1960s had Italy regained its stabilization and the economy started booming.\\n\\\"It was just a mere 10 years after the end of the war. The war feelings were still very much there,\\\" she recalled as she started her work at CERN in 1956.\\nAlthough there were few female scientists at CERN during her time, Maria felt she was respected by her colleagues. “In my group, I was simply one of them”, Maria said.\\nThe couple, having devoted 60 years to research at CERN since its beginning, is seen as a living memory of the institute.\\nScientific research.\\nAfter Maria’s marriage, she and Giuseppe carried out experiments on pions. Maria worked with a diffusion chamber and Giuseppe with a lead glass Cherenkov counter.\\nMaria and Giuseppe were also involved in cosmic rays experiment in the Italian Alps just after the war. The experiment was set at 3,500 meters on the face of Matterhorn, using a detector of 1-square meter.\\nAfter Maria and Giuseppe moved to Geneva, Giuseppe was assigned to the Synchrocyclotron Division. Synchrocyclotron was the first accelerator to be built at CERN in 1957. Meanwhile, Maria worked on a novel method to provide polarized proton beams to be used for particle collision in the accelerator.\\nThroughout her career as a theoretical physicist, she authored over 200 publications in the fields of Nuclear Physics, Nuclear Energy, and Statistical Physics. Maria collaborated with 903 co-authors in her research from 1961 to 2012.\\nThe majority of her research relates directly to the experiments, including interpreting the measurements and results in the search for new particles. In Maria’s early works in the 1960s, she studied the nucleon-nucleon charge exchange scattering, recorded data on high-energy electrodynamic processes.\\nHer research work also tackled the phenomenon of proton-proton elastic scattering. In the 1990s, she investigated the design and test of a prototype gas-sampling electromagnetic calorimeter of high granularity and collaborated on the construction of a position-sensitive photon detector for CPLEAR experiment. CPLEAR experiment aims to carry out precision measurements of CP, T and CPT violation of the neutral kaon systems.\\nMaria’s other work focused on phenomenological aspects of particle physics. She studied the fundamental symmetries in the neutral kaon systems. By the measurements of interactions and decays of neutral kaons, new and detailed information was found on CPT invariance in time evolution and decay. The newly achieved level of precision of the experiments called into question the validity of some of the often tacitly assumed hypotheses in particle physics.\\nMaria was also involved in the search for CP-violation (charge-parity violation) in the NA48 experiment. After the theory of CP-violation in the decay of neutral kaon system was experimentally confirmed, the discovery brought the Nobel Prize in Physics in 1980 to the discoverers of the theories, James Cronin, and Val Fitch. The discovery played an important role in helping scientists explain the dominance of matter over antimatter.\\nChallenges as a women scientist.\\nFemale scientists at CERN have to face certain obstacles. A trend is observed that when women scientists reach their mid-career and have children, they tend to leave their job. Even today, there are no pre-schools at CERN and those around the laboratory are expensive. Pregnancy can pose a challenge to the career advancement of women scientists because they have more limited access to the facilities due to the risk of radiation.\\nWhen CERN was first founded, there were about 300 to 400 staff but very few women scientists. Some gender issues at CERN were mentioned in the book “Women Scientists: Reflections, Challenges, and Breaking Boundaries” by Magdoha Hargittai. Mary Katharine Gaillard, a theoretical physicist holding a visiting position at CERN in the 1960s was interviewed in the book. Gaillard conducted a survey among women scientists at CERN at that time and found that only 10 percent was fully paid while 86 percent was not paid at all. Two of the reasons for women scientists not being paid were that their husbands were CERN employees and that priority was given to unemployed men. Maria Fidecaro might have experienced the gender bias mentioned by Gaillard during her time at CERN in the 1960s since her husband, Giuseppe Fidecaro, was an employee at CERN. Additionally, Maria had to juggle the job and a family of four children. There were few nurseries in CERN and around the laboratory at that time.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692958\",\"title\":\"Marcelle Rodrigues\",\"body\":\"\\nMarcelle Rodrigues\\n\\nMarcelle Rodrigues (born ) is a retired Brazilian female volleyball player, who played as a setter. She was part of the Brazil women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. At the end of the championships she was awarded as the best setter. On club level she played with BCN Osasco.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692965\",\"title\":\"Hélder José Castro Ferreira\",\"body\":\"\\nHélder José Castro Ferreira\\n\\nHélder José Castro Ferreira (born 5 April 1997 in Fafe) is a Portuguese footballer who plays for Vitória de Guimarães B, as a forward.\\nFootball career.\\nOn 25 November 2015, Ferreira made his professional debut with Vitória Guimarães B in a 2015–16 Segunda Liga match against Feirense.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48692986\",\"title\":\"Gustavo Díaz\",\"body\":\"\\nGustavo Díaz\\n\\nGustavo \\\"Chavo\\\" Díaz Domínguez (born November 7, 1974) is an Uruguayan former footballer and current manager of Ascenso MX club Celaya.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693000\",\"title\":\"Edmund Naganowski\",\"body\":\"\\nEdmund Naganowski\\n\\nEdmund Wacław Naganowski (26 September 1853 – 28 January 1915), was a Polish publicist and writer also known under pen names E. Działosz, Latarnik and Edmund Sas (Sas most likely refers to his Sas coat of arms).\\nLife.\\nNaganowski was born in Gostyń, Greater Poland, then in Austria-Hungary. After finishing his studies in England, he was a teacher in a high school in Waterford in Ireland and he later worked at the British Museum. On February 14, 1903 he became naturalized in Great Britain, under the name Edmund Sas de Naganowski.\\nHe served as secretary of the Polish Literary Society in London\\nHe is credited with the introduction of scouting in Poland.\\nMonica Mary Gardner acknowledged the influence and support of Naganowski on her interests in Polish culture from 1899 to the outbreak of the First World War.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693021\",\"title\":\"Sarati the Terrible (1923 film)\",\"body\":\"\\nSarati the Terrible (1923 film)\\n\\nSarati the Terrible (French:Sarati, le terrible) is a 1923 French silent film directed by René Hervil and Louis Mercanton and starring Henri Baudin, André Feramus and Ginette Maddie. In 1937 it was remade as a sound film.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693061\",\"title\":\"List of TVB dramas in 2016\",\"body\":\"\\nList of TVB dramas in 2016\\n\\nThis is a list of television serial dramas released by TVB in 2016.\\nTop ten drama series in ratings.\\nThe following is a list of TVB's highest-rated serial dramas of 2016 according to the drama's average ratings. The list also includes the ratings of the dramas' premiere week, finale week, as well as the average overall count of Hong Kong viewers (in millions) per drama.\\nFirst line-up.\\nThese dramas air in Hong Kong from 8:00pm to 8:30pm, Monday to Friday on Jade.\\nSecond line-up.\\nThese dramas air in Hong Kong from 8:30pm to 9:30pm, Monday to Friday on Jade.\\nThird line-up.\\nThese dramas air in Hong Kong from 9:30pm to 10:30pm, Monday to Sunday on Jade.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693068\",\"title\":\"Embla (horse)\",\"body\":\"\\nEmbla (horse)\\n\\nEmbla (27 April 1983 – after 2003) was a British Thoroughbred racehorse and broodmare. In a racing career which lasted from June 1985 until October 1986 she won four of her nine races. As a two-year-old she finished second on her debut and won two minor races before recording an upset victory over a strong field for the Cheveley Park Stakes. In the following year she finished fifth in the 1000 Guineas and second in the Coronation Stakes before ending her career with a win in the Petition Stakes. Her broodmare career was largely undistinguished but she produced one excellent racehorse in Zenno El Cid, a major winner in Japan.\\nBackground.\\nEmbla was a \\\"strong, rangy\\\" dark bay mare with no white markings bred in the United Kingdom by Sheikh Mohammed. She was sired by Dominion who finished third in the 1975 2000 Guineas and won the Prix Perth in 1976. before being exported to North America and winning the Bernard Baruch Handicap in 1978. He sired several good horses over a wide range of distances including First Island, the sprinter Primo Dominie (Coventry Stakes), and the stayer Trainglot (Cesarewitch Handicap). Embla's dam Kaftan won one minor race from four attempts but was a half-sister to the Nunthorpe Stakes winner Blue Cashmere and a granddaughter of the Yorkshire Oaks runner-up No Saint, whose other descendants included Persian Heights.\\nAs a yearling Embla was offered for sale and bought for 46,000 guineas by representatives of Charles St George. The filly was sent into training with Luca Cumani at Newmarket, Suffolk.\\nRacing career.\\n1985: two-year-old season.\\nEmbla began her racing career by finishing second in a minor race over six furlongs at Newmarket Racecourse in late June. She then started favourite for an eighteen-runner maiden race at Kempton Park Racecourse in July and won easily. On her next appearance she was less impressive as she started 1/5 favourite for a minor event at Ripon Racecourse in August but won by only three-quarters of a length from Raisinhell. Timeform described her as \\\"scrambling home\\\" in the race. Cumani, nevertheless, held the filly in high regard and later explained that she had been going through an \\\"awkward growing phase\\\" at the time.\\nOn her final appearance of the year, Embla was moved up sharply in class to contest the Cheveley Park Stakes (the only Group One race in Britain at the time restricted to juvenile fillies) over six furlongs at Newmarket in October. The leading American jockey Angel Cordero was booked for the ride, but the filly was given little chance and started a 20/1 outsider in a thirteen-runner field. The Jeremy Tree-trained Kingcote started favourite after a four-length win in the Lowther Stakes, whilst the other contenders included Park Express, Storm Star (winner of the Cherry Hinton Stakes), Asteroid Field (Waterford Candelabra Stakes), Bambolona (Sirenia Stakes), Wanton (third in the Flying Childers Stakes), Nashia (runner-up in the Prix d'Arenberg) and Rose of the Sea (fourth in the Prix Morny). In the early stages Rose of the Sea led from Storm Star, with Embla towards the rear and apparently struggling to match the pace. Kingscote went to the front approaching the final furlong but Embla was making relentless progress and soon moved into contention. She overtook Kingscote a hundred yards from the finish and won by a length. Rose of the Sea held on for third ahead of Asteroid Field.\\n1986: three-year-old season.\\nOn her three-year-old debut Embla contested the Nell Gwyn Stakes (a trial race for the 1000 Guineas) over seven furlongs at Newmarket on 15 April and finished fourth of the nine runners behind Sonic Lady. Sixteen days later she started 6/1 second favourite behind behind Sonic Lady in the 1000 Guineas. Ridden by Tony Ives she finished fifth behind Midway Lady, Maysoon, Sonic Lady and Ala Mahlik. She proved no match for Sonic Lady in her next two races, finishing second in the Coronation Stakes at Royal Ascot in June and fifth in the Child Stakes at Newmarket on 9 July. After a break of three months the filly returned and was dropped in class for the Petition Stakes over one mile at Newmarket in October. She recorded her only success of the year as she narrowly defeated the colt Santella Mac.\\nAssessment.\\nIn the official International Classification for 1985, Embla was rated the third-best two-year-old filly in Europe after the French-trained Baiser Vole and Regal State. The independent Timeform organisation rated her on 121, three pounds behind their top juvenile filly Femme Elite. She was rated 110 by Timeform in 1986.\\nBreeding record.\\nEmbla was retired from racing to become a broodmare and produced eleven foals between 1988 and 2003. Three of her offspring won races, with the most successful, by far, being the Japanese horse Zenno El Cid.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693069\",\"title\":\"Peter IV van den Gheyn\",\"body\":\"\\nPeter IV van den Gheyn\\n\\nPeter van den Gheyn (; 1605 or 1607–1654 or 1659) was a bell-founder of the Spanish Netherlands (now Belgium).\\nLife.\\nPeter was born into a family of noted bell-founders and himself became the most famous bell-founder of the 17th century. His father was Jan van den Gheyn. The family forge was at Mechelen in what is now Belgium. His associate was named Peter Deklerk or de Clerck, his uncle by marriage.\\nHis total production of bells was not great. He cast the Salvator bell for in Mechelen in 1638, which weighed and cracked in 1696. He also cast the Salvator bell for in Brussels.\\nHe had the curious affectation of inscribing his bells using type of various sizes within the same word.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693071\",\"title\":\"Li Ying (volleyball)\",\"body\":\"\\nLi Ying (volleyball)\\n\\nLi Ying (born ) is a Chinese female volleyball and beach volleyball player, who played as a libero. She was part of the China women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany and 2002 Asian Games. At the 2014 Asian Women's Club Volleyball Championship with Tianjin Bohai Bank she won the silver medal and was awarded as the best opposite spiker. On club level she plays with Tianjin Volleyball in 2015-16.\\nWith Wang Lu she played at the 2005 Beach Volleyball World Championships.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693076\",\"title\":\"Peter van den Gheyn\",\"body\":\"\\nPeter van den Gheyn\\n\\nPeter van den Gheyn (Period ) may refer to any of several members of a noted family of bell-founders in the Spanish Netherlands. They are conventionally disambiguated with roman numerals:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693084\",\"title\":\"Abraham Lincoln Bridge\",\"body\":\"\\nAbraham Lincoln Bridge\\n\\nThe Abraham Lincoln Bridge is a six-lane, single-deck cable-stayed bridge carrying Interstate 65 across the Ohio River, connecting Louisville, Kentucky and Jeffersonville, Indiana. The main span is (two spans) and the bridge has a total length of . It is named after U.S. President Abraham Lincoln.\\nHistory.\\nThe Abraham Lincoln Bridge opened on December 6, 2015, and is parallel to the John F. Kennedy Memorial Bridge upstream and carries three lanes of northbound I-65 traffic, eventually carrying six lanes of northbound I-65 traffic. Pedestrian and bicycle lanes were in the original plans, but have since been removed. The existing I-65 John F. Kennedy Memorial Bridge, completed in 1963, will be renovated for six lanes of southbound traffic.\\nA Structured Public Involvement protocol developed by Drs. K. Bailey and T. Grossardt was used to elicit public preferences for the design of the structure. From spring 2005 to summer 2006 several hundred citizens attended a series of public meetings in Louisville, Kentucky and Jeffersonville, Indiana and evaluated a range of bridge design options using 3D visualizations. This public involvement process focused in on designs that the public felt were more suitable, as shown by their polling scores. The SPI public involvement process itself was evaluated by anonymous, real-time citizen polling at the open public meetings.\\nOn July 19, 2006, the final design alternatives for the bridge were announced. The three designs included a three-span arch, a cable-stayed design with three towers, and a cable-stayed type with a single A-shaped support tower. It was also announced that the projected cost for the bridge would be $203 million.\\nThe structure is an additional bridge in downtown Louisville joining the John F. Kennedy Memorial Bridge erected between spring 1961 and late 1963 at a cost of $10 million; the four-lane George Rogers Clark Memorial Bridge, constructed from June 1928 and to October 31, 1929, and the Big Four Bridge, which operated as a railroad bridge from 1895 to 1969 and reopened as a pedestrian bridge in May 2014.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693093\",\"title\":\"José Pérez (athlete)\",\"body\":\"\\nJosé Pérez (athlete)\\n\\nJosé Roquez Pérez (born 19 March 1971) is a retired Cuban athlete who specialised in the 400 metres hurdles. He represented his country at two World Championships, in 1993 and 1995, reaching the semifinals on both occasion. In addition he won multiple medals on regional level.\\nHis personal best in the event is 49.28 seconds set in Cali in 1993.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693113\",\"title\":\"21 Party\",\"body\":\"\\n21 Party\\n\\nThe 21 Party () was a political party in Burma in the 1920s led by U Ba Pe.\\nHistory.\\nThe party was formed in 1922 following a split in the General Council of Burmese Associations (GCBA). The GCBA had planned to boycott the local and national elections due that year, but a group of 21 dissidents left the organisation to form a new party.\\nIn the elections the 21 Party won 28 of the 58 non-communal seats, becoming the largest party in the Legislative Council. However, it held less than a third of the total of 103 seats, and Joseph Maung Gyi from the pro-British Independent Party was appointed head of government.\\nPrior to the 1925 elections the party was succeeded by the Nationalist Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693118\",\"title\":\"2016 TCU Horned Frogs football team\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693123\",\"title\":\"Julie McElrath\",\"body\":\"\\nJulie McElrath\\n\\nM. Juliana “Julie” McElrath (born January 9, 1951) is a senior vice president and director of the vaccine and infectious disease division at Fred Hutchinson Cancer Research Center and principal investigator of the HIV Vaccine Trials Network Laboratory Center in Seattle, Washington. She also is a professor at the University of Washington.\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693131\",\"title\":\"In Old Alsace (1920 film)\",\"body\":\"\\nIn Old Alsace (1920 film)\\n\\nIn Old Alsace (French:L'ami Fritz) is a 1920 French silent film directed by René Hervil and starring Léon Mathot, Huguette Duflos and Thérèse Kolb.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693145\",\"title\":\"George Moy\",\"body\":\" (5 years 1 month) Cleveland/Akron, Ohio Area Work with alumni and friends of the SIHS community to foster leadership gifts to make a meaningful impact on the mission of Saint Ignatius High School and change the lives of current and future students. Growth of the endowment to have a lasting impact on the affordability of a Jesuit, Catholic education through committed gifts or planned giving.\\nManaging Director of Commercial Banking at Talmer Bank and Trust.\\nTalmer Bancorp\\nSenior Relationship Manager responsible for the creation of a portfolio of loan and deposit relationships with major businesses within the midwest footprint for Talmer Bank and Trust. Emphasis is on generation of new business and cross sell opportunities, administer and service business relationships. Interview potential clients to develop information concerning their businesses and requirements, evaluate credit information and make relationship recommendations.\\nMiddle Market - Healthcare - Community Development Lending - Commercial Real Estate\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693181\",\"title\":\"Webster, Alberta\",\"body\":\"\\nWebster, Alberta\\n\\nThe locality of Webster was established in 1916 as a station on the Edmonton, Dunvegan and British Columbia Railway about 30 km north of the City of Grande Prairie. It was named after George Webster who was a subcontractor for the railway at that time, and later became Mayor and then MLA for the City of Calgary.\\nAlthough, there were a few railway workers who lived there from 1916 to the late 1920s, the area was mainly settled in 1929 by Polish immigrants from the Vermilion area who had fled Poland after Russian invaded following World War I. In the fall of 1929, a store was opened by Andrew and Cathy Hancharyk on the NW quarter of Section 27, Township 74, Range 5, West of the 6th Meridian. A post office opened there on October 15, 1929, with Andrew as postmaster. “Across the street from the store was the butchershop and café”. The community included ”two livery barns… a single elevator which served an area as far away as Heart Valley… an elevator house, and a water tank for the train.” Across the creek was the Webster Sawmill. In 1930, a large Catholic Church was erected. This also served as a school-room for Torun School District 4483, approved on March 20, 1930, until the school was built in 1937. In the late 1930s, Webster Hall was built for community gatherings and a rectory was added for the first resident Catholic priest. The elevator was the first community building lost by fire, then the church in the early 1960s. In 1957, Torun School closed and the students were vanned to Sexsmith. The post office closed in 1966, and all that remained were the hall and the Catholic grotto which had been erected beside the church, as well as the Webster Cemetery.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693208\",\"title\":\"Estadio Arena Cora\",\"body\":\"\\nEstadio Arena Cora\\n\\nThe Estadio Arena Cora is a multi-use stadium in Tepic, Nayarit, Mexico. It is currently used mostly for football matches and is the home stadium for Deportivo Tepic F.C.. The stadium has a capacity of 12,271 people and opened in 2011.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693226\",\"title\":\"The Flame (1926 film)\",\"body\":\"\\nThe Flame (1926 film)\\n\\nThe Flame (French:La flamme) is a 1926 French silent drama film directed by René Hervil and starring Germaine Rouer, Charles Vanel and Henry Vibart. It was remade as a sound film in 1936.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693231\",\"title\":\"Fruit carving\",\"body\":\"\\nFruit carving\\n\\nFruit carving is the art of carving fruit, a very common technique in Europe and Asian countries, and particularly popular in Thailand, China and Japan. There are many fruits that can be used in this process; the most popular one that artists use are watermelons, apples, strawberries, pineapples, and cantaloupes.\\nHistory.\\nChina.\\nMany believe that fruit carving originated in China during the Tang Dynasty, which lasted from AD 618-906. Fruit carving in China usually features mythological creatures, and animals. Not only is fruit carving used in cultural and traditional ceremonies, but also ordinary households are known for decorating plates with fruit carvings when they have guests over. Specifically, watermelon carving has been and still is very popular in China. Usually, the outside of the melon is carved on and the melon pulp is scraped out of the inside of the melon, so it can be used as a container to put food or flowers in. Chinese fruit carving is used to tell their legends and stories.\\nEurope.\\nFruit carving is included in Matthias Giegher's 1621 work \\\"Il Trinciante\\\" (\\\"The Carver\\\"), where he describes carving oranges and citrons into abstract patterns, shell-fish, four-legged animals and the Hapsburgs' double-headed eagle, but the art was not common in Europe or North America until the 1980s when several books on the topic were published.\\nThailand.\\nFruit carving is a significant part of Thai cultural heritage. Watermelon carving dates to the 14th century in Thailand during the Sukhothai dynasty. The annual Loi Krathong Festiva occurs each November where people in Thailand float lamps and lanterns down a river to honor water spirits. One legend is that one of the king’s maids decorated her lantern with a watermelon carved with flower designs to impress him and that he was so pleased that he encouraged all Thai women to adopt the practice. The king also requested that fruit carving become part of the primary school curriculum. Thailand fruit carving features flowers, birds and floral patterns.\\nJapan.\\nThe Japanese emphasize the presentation of a dish and how the plate aesthetically appeals to others. Fruit carving in Japan is referred to as Mukimono. Mukimono began in ancient Japan in an effort to make dishes more appealing since the food was placed and served on an unglazed pottery plate, which had a rough look to it. Chefs would cover the plate in leaves and would fold them into different designs in order to make the dish look better.This technique eventually turned into carving fruit that would also be placed on the plates to enhance the appearance of the dish. At first, when this technique came out, vendors on the streets would add carved fruit to their food when customers made a special request, but now it is very common for all Japanese dishes to feature carved fruits. Fruit carving and garnishing is now a significant part of Japanese chef training.\\nOverview.\\nThe art of fruit carving uses many different tools, usually ordinary items but some specific to just fruit carving. All these tools give the artwork a different texture or help with its design. Some of the tools include:\\nBefore carving a fruit, the necessary materials must be gathered. The usual process is to have the tools handy for when they are needed in one's steps towards creating a carving. There are two types of carvings that can be done. The first is Skin Carving, this is when the outer skin of the fruit (or vegetable) is carved to reveal the fleshy center, where the color is different than the outer skin to create a design using the contrast between the outer skin and flesh of the fruit. The second type of carving is Three Dimensional Carving, the purpose of this particular technique of carving is to carve the fruit to into a three dimensional object of the carver’s choice. The most popular three dimensional design is floral objects. It is easy to over-carve a fruit which leads to loss in nutrition, but it is always important to start off by outlining your carving on the fruit. Depending on the design that is being created it may take very little time or be more complex and use more time and concentration also different steps in the process are used for each design. Not one design is alike. An important aspect of fruit carving is that the fruit you carve must be suitable for the manner which one will display it. Most fruit carvings will need to be refrigerated before they are placed in the specific setting that is desired.\\nArtists.\\nThere are many artists and chefs who are known for their fruit carving skills. Each has a different technique. Many of these chefs have previously owned restaurants and now hold classes in this art. \\nValeriano Fatica is an Italian fruit carver, known through his videos on Youtube that has attracted people to his artwork around the world. He has carved many different fruits at a large scale.\\nJimmy Zhang is a fruit carver from China. He has won many awards in the culinary area and has also been featured in many newspapers across the globe for his skills. He teaches classes on how to carve fruit and vegetables and master the skills and learn history about the way China has created its fruit carving industry.\\nCarl Franklin Jones, an expert in the art of fruit carving, practices in the United States. He is known for his fruit carving sculpture that was created for Ivanka Trump's wedding. He also holds a catering businesses in Tennessee. Jones was a business owner of many franchises also in Tennessee before starting his catering business. Today Jones travels the world and hosts mentoring programs for young ambitious fruit carving artists and chefs.\\nFruit carving today.\\nSome people perform fruit carving professionally. Some chefs utilize fruit carving as a culinary technique. Once fruit carvers have mastered the techniques past the intermediate stage and become professionals, they can price their services to restaurants, professional caterers, hotels and resorts. Professional fruit carvers can also create centerpieces and displays for various events, such as parties and wedding receptions. On a smaller scale, fruit carvers can present a dish with decorative garnishing to add an aesthetically pleasing experience to their viewers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693235\",\"title\":\"Corte Capitanale\",\"body\":\"\\nCorte Capitanale\\n\\nThe Corte Capitanale is a former courthouse in Mdina, Malta, which currently serves as a city hall. It was built in the Baroque style between 1726 and 1728, to designs of the French architect Charles François de Mondion. The building is linked to Palazzo Vilhena, but it has its own entrance and façade.\\nHistory.\\nThe Corte Capitanale was built between 1726 and 1728 along with the rest of Palazzo Vilhena, the Grand Master's official residence in Mdina. The building was mainly a courthouse, but it also served as the seat of the \\\"Capitano della Verga\\\". Its linking to the palace was a symbolic gesture to convey the fact that the courts were under the jurisdiction of the Order of St. John. The courthouse was also linked to the Bishop's Palace through a now-blocked underground passage, indicating the Church's role in the courts. The building remained a courthouse until 1818.\\nToday, the Corte Capitanale is Mdina's city hall, being the seat of the city's local council. The council considers the building as inadequate, and in 2012 it requested to move its premises to the Banca Giuratale. It is not regularly open to the public, but the main hall is occasionally open with temporary historical or cultural exhibitions.\\nThe building is a Grade 1 national monument and it is listed on the National Inventory of the Cultural Property of the Maltese Islands.\\nArchitecture.\\nThe Corte Capitanale is built in the French Baroque style. The façade is decorated by superimposed Tuscan and Corinthian pilasters, and a cornice along roof level. A balcony is located above the main doorway, and it is decorated with allegorical statues of Justice and Mercy. The inscription \\\"Legibus et Armis\\\" (by using laws and arms) is inscribed below the centrepiece of the façade.\\nThe courthouse incorporates some prison cells and dungeons, which had been built in the 16th century. The building is also linked to a loggia known as Herald's Loggia, from which town criers used to announce decrees to the people. The loggia also predates the courthouse, and it is believed to date to the 17th century.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693247\",\"title\":\"Le Touquet AC\",\"body\":\"\\nLe Touquet AC\\n\\nLe Touquet Athlétic Club Football Côte-d'Opale, commonly known simply as Le Touquet, is a French association football club based in the commune of Le Touquet-Paris-Plage in the Pas-de-Calais department of northern France. The club was founded in 1933 as the result of a merger between two local clubs, \\\"Olympique Touquettois\\\" and \\\"US Le Touquet-Paris-Plage\\\", and adopted its current name in 1996. Its stadium is named after former France, Paris Saint-Germain and Liverpool manager Gérard Houllier, who was a player and coach at the club during the 1970s.\\nAlthough the club has spent much of its history in the lower reaches of the French football league system, they competed in Division 2 during the 1988–89 season. The men's senior team regularly competes in the Coupe de France, the country's foremost knock-out competition.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693269\",\"title\":\"Jorge Miguel Machado Almeida\",\"body\":\"\\nJorge Miguel Machado Almeida\\n\\nJorge Miguel Machado Almeida (born 22 September 1990 in Vila Nova de Famalicão) is a Portuguese footballer who plays for F.C. Famalicão as a defender.\\nFootball career.\\nOn 16 September 2015, Almeida made his professional debut with Famalicão in a 2015–16 Segunda Liga match against Oliveirense.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693271\",\"title\":\"Sir Edmund Ralph Verney, 6th Baronet\",\"body\":\"\\nSir Edmund Ralph Verney, 6th Baronet\\n\\nSir Edmund Ralph Verney, 6th Baronet (born 28 June 1950) succeeded to the title of 6th Baronet Calvert, of Claydon House, Buckinghamshire on 17 August 2001.\\nPersonal life.\\nVerney is the son of Sir Ralph Verney, 5th Baronet and Mary Vestey. He married Daphne Primrose Fausset-Farquhar, daughter of Colonel Hamilton Farquhar Fausset-Farquhar, in March 1982 and they have two children: Andrew Nicholas Verney (b. 1983) and Ella Verney (b. 1985).\\nHe was educated at Harrow School, London and went to University of York. He is a fellow of the Royal Institution of Chartered Surveyors.\\nHe held the office of High Sheriff of Buckinghamshire from 1998 to 1999.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693274\",\"title\":\"Vaia Dirva\",\"body\":\"\\nVaia Dirva\\n\\nVaia Dirva (born ) is a retired Greek female volleyball player, who played as a middle blocker. She was part of the Greece women's national volleyball team at the 2001 Women's European Volleyball Championship and 2002 FIVB Volleyball Women's World Championship in Germany. On club level, she played most notably for Olympiacos.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693283\",\"title\":\"Vado\",\"body\":\"\\nVado\\n\\nVado may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693286\",\"title\":\"Fidanka Saparefska\",\"body\":\"\\nFidanka Saparefska\\n\\nFidanka Saparefska (born ) is a retired Greek female volleyball player, who played as a middle blocker. She was part of the Greece women's national volleyball team at the 2001 Women's European Volleyball Championship and 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Panathinaikos Athen.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693293\",\"title\":\"Tatiana Smyrnidou\",\"body\":\"\\nTatiana Smyrnidou\\n\\nTatiana Smyrnidou (born ) is a retired Greek female volleyball player, who played as a universal . She was part of the Greece women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Filathlitikos Thessaloniki.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693301\",\"title\":\"General Council of Burmese Associations\",\"body\":\"\\nGeneral Council of Burmese Associations\\n\\nThe General Council of Burmese Associations (GCBA), also known as the Great Burma Organisation (; \\\"Myanma Ahthinchokgyi\\\"), was a political party in Burma.\\nHistory.\\nThe GCBA was formed at the 1920 conference of the Young Men's Buddhist Association following the student strike earlier in the year and Burma's exclusion from British proposals for limited self-government in Indian provinces. Its leadership included Chit Hlaing, U Pu and U Kyaw Dun. The new party held rallies to pressurise the British to extend the self-government plans to Burma. A proposal known as the Craddock Plan to give ethnic minorities separate representation was opposed by the GCBA, which saw it as an attempt at divide and rule.\\nIn 1922 the British agreed to extend the Indian system to Burma, and elections were scheduled for November. However, this caused a split in the GCBA, with the majority calling for a boycott and others calling for participation in the elections. This eventually led to 21 dissidents leaving to form the 21 Party, which emerged as the largest faction in the Legislative Council following the elections.\\nThe GCBA split again in the build-up to the 1925 elections due to differences over another boycott, as well as the organisation's finances and the role of monks. Dissidents left to form the U Chit Hlaing Faction, which subsequently splintered into the Home Rule Party and the Hlaing-Myat-Paw GCBA. The rump of the GCBA became known as the Soe Thein GCBA, named after its leader U Soe Thein.\\nAnother split occurred in 1929 when the organisation was split into the Ye-U group led by U Su and the Thetpan Sayadaws led by U Soe Thein. The latter collapsed in the early 1930s and many of its members joined other parties to contest elections. By 1932 the GCBA was effectively defunct, although its name continued to be used by some parties, including the United GCBA established in 1936.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693307\",\"title\":\"Fred Hammill\",\"body\":\"\\nFred Hammill\\n\\nFrederick Parkin Hammill (4 May 1856 - 8 July 1901) was a British trade union activist, and a co-founder of the Independent Labour Party.\\nCareer.\\nKnown generally as \\\"Fred\\\", Hammill was born in Leeds on 4 May 1856, trained as an engineer, and moved to London to work at the Royal Arsenal in Woolwich, where he became a well-known labour activist and trade unionist.\\nHammill spoke in defence of John Burns in trials after the 1887 Trafalgar Square Riot, was active in the London Trades Council (seconding Burns's support for the 1891 Scottish rail strike) and in the TUC, and he would speak indoors and outdoors to crowds of up to 6,000 people. He joined the Fabian Society in the early 1890s.\\nIn 1891 Hammill organised a strike of London bus and tram workers on pay and hours, and he was one of the founders of the Independent Labour Party. In 1893 he spoke at a demonstration and rally in Trafalgar Square on workers’ rights. Strongly associated with Tom Mann and with Will Crooks' Poplar-based Labour movement, Hammill helped establish the Woolwich ILP in 1894, with Robert Banner.\\nIn 1894, Hammill became a full-time organiser for the Fabian Society in Durham. A member of the Amalgamated Society of Engineers (ASE), in July 1895 Hammill was the first socialist to stand for election to parliament as an ILP candidate in Newcastle. Unsuccessful, he changed tack to run a pub (The Swan in Topcliffe in Yorkshire), for which he was criticised politically.\\nHe was elected to the Thirsk Rural District Council in 1901.\\nHe died on 8 July 1901 from influenza, aged 45 years.\\nPersonal life.\\nWhen Fred Hammill was a child, his family lived in a pub (the Grey Mare Inn, 132 Low Road, Hunslet) in Leeds, run by his father Thomas.\\nHammill married Ada Peel (9 July 1860 – c. Feb 1940) and they had three children (Arthur Earnest (1880-1945), Helen (1882-1904), Gertrude Wright (1888-1959)). After moving to London from Leeds, they lived at 25 Coxwell Road in Plumstead for a period (c. 1890 to 1892).\\nAda's father Joseph Peel was an inn-keeper.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693315\",\"title\":\"Tingvatn\",\"body\":\"\\nTingvatn\\n\\nTingvatn is a town in Hægebostad municipality in Norway. The town has a little over 100 inhabitants and is the administrative center of Hægebostad, Vest-Agder. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693325\",\"title\":\"Li Guoying\",\"body\":\"\\nLi Guoying\\n\\nLi Guoying (; born January 1964) is a Chinese politician, serving since 2015 as the Deputy Communist Party Secretary of Anhui province. He was born in Yuzhou, Henan province. He attended North China University of Water Conservancy and Electric Power in Zhengzhou. He also had a graduate degree in philosophy from the Central Party School, then a doctorate from Northeast Normal University in environmental science.\\nHe worked initially in the Yellow River commission of the Ministry of Water Works. He spent most of his career in the water works system, as the assistant to the chief engineer, then the deputy director of the Yellow River commission, then chief engineer. He was then transferred to head the department of water works in Heilongjiang. In March 2011, he was named Vice Minister of Water Works; he stayed on the post until August 2015, when he was named deputy party chief of Anhui.\\nLi was an alternate member of the 18th Central Committee of the Communist Party of China.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693329\",\"title\":\"Dominika Smereka\",\"body\":\"\\nDominika Smereka\\n\\nDominika Leśniewicz-Smereka (born ) is a retired Polish female volleyball and beach volleyball player, who played as a libero. She was part of the Poland women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Palac Bydgoszcz.\\nShe played beach volleyball with Irabela Rutkowska in 2000.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693352\",\"title\":\"List of atmospheric pressure records in Europe\",\"body\":\"\\nList of atmospheric pressure records in Europe\\n\\nThe following is a List of atmospheric pressure records in Europe and the extratropical Northern Atlantic (it does not include localised events, such as those which may occur in tornados).\\nLand-based observations in Europe.\\nLand-based records for Europe:\\nOther high values have been reported:\\nPressure is thought to have risen above 1060 hPa in Europe on only 12 occasions between 1871–2010, in the years 1893, 1899, 1907, 1915, 1920, 1938, 1944, 1946, 1956, 1972, 1995 and 2008. To this list might also be included high pressure anticyclones in 1954 and 2012 (see above). The most notable high in Europe is that which reached peak in January 1907, this was an unusual development which brought high pressure to the west, this intense high pressure area holds the record across multiple countries in northeast Europe (Denmark, Norway, Sweden, Estonia, Latvia, Germany, Slovakia).\\nAnother value reported:\\nIceland.\\nIceland list of atmospheric pressure over 1050 hPa.\\nHigh pressure in Iceland has exceeded 1050 hPa 4 times in the twentieth century and five times in the nineteenth century.\\nClimatic Research Unit Emulate data 1874–2002, also gives values on 10 March 1887 at Stykkishólmur 1052.43 hPa and also 9 January 1977 at the same location 1050.47 hPa.\\nIn July the pressure in Iceland has only dropped below 975 hPa or lower three times over the entire record extending back to the 1820s. These cases were 974.1 hPa in Stykkishólmur on 18 July 1901, 974.3 hPa in Stykkishólmur on 19 July 1923 and 975.0 hPa in Reykjavík 11 July 1912. A recent low pressure on 30 December 2015 was recorded at Kirkjubaejarklaustur (associated with an area of low pressure known as Eckard/Frank) at 930.2 hPa, the lowest pressure recorded on land in Iceland since 1989.\\nFaroe Islands.\\nThe Danish Meteorological Institute report record atmospheric pressure for the Faroe Islands (since 1961) as:\\nClimatic Research Unit Emulate data 1874–2002 give 3 dates when pressure exceeded 1050 hPa in Tórshavn.\\nIreland.\\nMet Éireann list the following national records for atmospheric pressure:\\nUnited Kingdom and Ireland.\\nFor the United Kingdom, the Met Office record the record figures for atmospheric pressure (which are nominally since 1870) as:\\nThough the lowest pressure may be second to the Night of the big wind low which saw a value of at Sumburgh Head, Shetland on non-calibrated, non-standard equipment 6-7 January 1839, with the mainland at Cape Wrath reporting an observed pressure of .\\nUnited Kingdom list of atmospheric pressure over 1050 hPa.\\nIn the period of instrumental measurement the atmospheric pressure has exceeded 1048 hPa somewhere over the United Kingdom and Ireland on 17 occasions. On 9 of these times the pressure has exceeded 1050 hPa- a value which has not been exceeded since 1957 in Ireland, but has not been surpassed since 26 January 1932 over the UK. Intense high pressure is usually seen during midwinter with 7 of the 9 occasions where 1050 hPa has been exceeded occurring in January.\\nPressure values have been recorded to have exceeded 1050 hPa in all areas of the UK and Ireland except south east England, though values close to this are documented from January 1882 and January 1905.\\nUnited Kingdom monthly minimum figures for atmospheric pressure.\\nCyclone Oratia is estimated to have reached a low of 941 hPa in peer reviewed literature, though the lowest pressure occurred between Aberdeen and Norway over the North Sea and is not a land-based observation. The lowest land-based reported UK pressure during this storm was recorded from RAF Fylingdales at 951.2 hPa.\\nOn 7 September 1995, Scilly Isles, reported a low pressure of 966 hPa. The lowest minimum recorded values for the months May to August lie within 0.5 of 968 hPa.\\nThe 20th Century low pressure record in the UK occurred on 20 December 1982 at Sule Skerry it may have dropped as low as 936 mbar.\\nNorway.\\nYr.no the joint venture between the Norwegian Broadcasting Corporation and the Norwegian Meteorological Institute give the national pressure records as:\\nBoth records in same winter as each other and those of Denmark.\\nDenmark.\\nThe Danish Meteorological Institute report record barometric pressure for the Denmark (since 1874) as:\\nDanish records both occurred within a month during the same winter, and same winter as Norwegian records.\\nThe Danish September record was beaten on 29 September 2015 with pressure over 1042.2 hPa recorded in Østerbro Copenhagen, the previous September record dated from September 18, 1904, where the pressure came up to 1038.8 hPa at Hammer Odde Lighthouse, Bornholm.\\nSweden.\\nThe Swedish Meteorological and Hydrological Institute give the nation's barometric records as:\\nFinland.\\nAccording to the Finnish Meteorological Institute:\\nThough a reported low value by Weather Underground weather historian Christopher Burt on 1 March 1990 at an unknown location is reported at 939.7 hPa.\\nBaltics.\\nMaximum pressure in Europe 22–23 January 1907 recorded in Pärnu, Estonia and Riga, Latvia at 1067.1 mbar. The Free University of Berlin state the European air pressure record is 23 January 1907 in Riga at 1068.7 hPa.\\nEstonia.\\nAs reported by the Estonian Weather Service\\n‡The Estonian Weather Service report on their website the record maximum observed air pressure in the country as 1060.3 hPa recorded 23 January 1907 in Tallinn. This figure is not supported by NOAA reanalysis charts and the values reported from neighbouring Finland and Latvia probably make this figure an underestimation or error. A possible maximum pressure value for the whole of Europe is thought to have occurred during the 22-23 January 1907 high pressure event with a record set in Pärnu, Estonia and Riga, Latvia at 1067.1 mbar. Though this value is exceeded by the value on 23 January 1907 in Riga reported at 1068.7 hPa by the Free University of Berlin's student generated reports.\\nLithuania.\\n23 January 1907 1065.1 hPa.\\nFrance.\\nMétéo-France figures from 1951-2012.\\nthough other figures are also listed which predate Météo-France's records.\\n29 January 1905, a powerful anticyclone covers all of Europe and the atmospheric pressure reached a record value of 1049.3 hPa in Paris.\\nBelgium.\\nThe Belgian Royal Meteorological Institute gives the national record values as:\\nNetherlands.\\nThe Dutch Royal Netherlands Meteorological Institute lists the following as national records:\\nGermany.\\nThe German weather service (DWD) give the\\nOther record figures and locations are also presented:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693374\",\"title\":\"Alvin McCoy\",\"body\":\"\\nAlvin McCoy\\n\\nAlvin Scott McCoy (born July 14, 1903) was a journalist of \\\"The Kansas City Star\\\" who won the Pulitzer Prize in 1954 for local reporting and an outstanding work published the previous year about a series of articles that drove C. Wesley Roberts to resign his RNC chairmanship.\\nBiography.\\nAlvin Scott McCoy was born on July 14, 1903, in Cheney, Kansas. \\nHe received an A.B. degree in 1925 from the University of Kansas at Lawrence, majoring in chemistry.\\nAfter spending two years at Ford Motor Company agency in Dodge City, Kansas, and one year\\ntraveling around the world in 1928–29, McCoy was first employed in \\nnewspaper work as a reporter of the \\\"Evening Eagle\\\" in Wichita, Kansas.\\nHe spent eighteen months on this newspaper and on the Wichita \\\"Morning Eagle\\\".\\nIn November, 1930, he joined the \\\"Kansas City Star\\\" as a reporter and worked on\\ngeneral assignments.\\nYears later, McCoy served as the Star’s Pacific War correspondent in 1945. \\nThat same year he became also a Kansas correspondent of the \\\"Kansas City Star\\\", for which he covered\\nstate politics, legislature, news, features and some editorial writing as well as\\nscientific stories.\\nIn the 1954 the Board members decided that the Pulitzer Prize in the Local\\nReporting, No Edition Time category should go to Alvin Scott McCoy of the \\nKansas City Star from Missouri, “for a series of exclusive stories with led to the \\nresignation under fire of C. Wesley Roberts as Republican National Chairman.\\nRoberts was accused of collecting a $10,000 commission on the sale of a hospital to the State of Kansas which the state already owned.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693379\",\"title\":\"Jacksons, New Zealand\",\"body\":\"\\nJacksons, New Zealand\\n\\nJacksons is a settlement and railway station in the Grey District of the West Coast of New Zealand. The TranzAlpine Express now passes through Jacksons.\\nJacksons was the railhead for the Midland Line from Stillwater via Moana from 1894, until the line was extended to Otira in 1899. The station opened on 1 March 1894 (using a building from Teremakau on 23 February 1894), and closed on 3 November 1986.\\nThe \\\"Jackson’s Accommodation House\\\", now the \\\"Jackson Tavern\\\", was bought by Michael Jackson in 1870; Michael and his brother Adam from Scotland had moved there after spending some time on the Otago Goldfields. The hotel was a stop for Christchurch-Hokotika coaches. The hotel was swept away in a flash flood in 1871, and was rebuilt as the Perry Range Hotel. In 1910 it was rebuilt and in 1970 it passed out of the Jackson family, but was renamed the \\\"Jackson Tavern\\\". Adam moved to Canterbury, but his eldest daughter Jessie married William Aicken who began the \\\"Aickens Accommodation House\\\" 10 km away at Aickens.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693391\",\"title\":\"Ewa Kowalkowska\",\"body\":\"\\nEwa Kowalkowska\\n\\nEwa Kowalkowska (born ) is a retired Polish female volleyball player, who played as a wing spiker. She was part of the Poland women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. She was awarded the best server price at the 1999 Women's European Volleyball Championship. On club level she played with Palac Bydgoszcz.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693393\",\"title\":\"Athanasius Sandalaya\",\"body\":\"\\nAthanasius Sandalaya\\n\\nAthanasius Sandalaya, also known as Athanasius Sandloyo or al-Sandali, was the Patriarch of Antioch, and head of the Syriac Orthodox Church from 756 until 758.\\nBiography.\\nOriginally a monk at the Monastery of Qartmin, Athanasius became Bishop of Maiperqat, a bishopric subordinate to the Metropolitan Bishop of Amid. During his tenure as bishop, Athanasius is known to have used church funds to obtain the support of the Caliph Marwan II to strengthen his position.\\nBy 742/743, Athanasius had been granted the title of Metropolitan of Mesopotamia, potentially due to the simultaneous growth and decline of Maiperqat and Amid respectively, as well as the decrepitude of Severus, Bishop of Amid.\\nAs metropolitan, Athanasius granted his student Isaiah of Ashparin administrative control of the greater part of the diocese of Amid as a result of Severus' inability to lead the diocese. During the late 740s, however, this appointment caused Athanasius to come into conflict with the patriarch Iwannis I who ordained a certain Abay, former Bishop of Arzun, as the new bishop of Amid. This conflict was exacerbated by Iwannis' failure to ordain Dionysius, Athanasius' appointment to the empty see of Tur Abdin, after the death of its former incumbent Athanasius of Nunib. \\nAt the Synod of Tella in 752, Athanasius expanded his authority as metropolitan from the area of the former Roman province of Mesopotamia to the entirety of Upper Mesopotamia through the use of implicit threats of reprisals from the Muslim authorities, despite the protests of the bishops. Athanasius commemorated his success with the erection of a new cathedral in the city of Maiperqat. He later used his new authority to ordain his student Iwannis Isaac as Bishop of Harran and depose the bishops of Samosata and Singara. Athanasius also succeeded in having Iwannis Isaac ordained as the patriarch and successor to Iwannis I in 754. Daniel, son of Moses of Tur Abdin, later claimed that Athanasius secured Iwannis' elevation to the patriarchal office by organising the election fraudulently. Athanasius succeeded Iwannis in 756 and served as patriarch until 758.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693398\",\"title\":\"Willie Borland\",\"body\":\"\\nWillie Borland\\n\\nWilliam 'Willie' Borland (born 21 September 1952) was a Scottish footballer who played for St Mirren, Barrow, Dumbarton and Stranraer.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693401\",\"title\":\"Ali Helichi\",\"body\":\"\\nAli Helichi\\n\\nAli Helichi () is an Iranian football defender who currently plays for Iranian football club Esteghlal Ahvaz in the Iran Pro League.\\nClub career.\\nHelichi joined Esteghlal Ahvaz in summer 2014. He made his professional debut against Gostaresh Foolad on September 26, 2015 as a substitute for Armin Mirdoraghi.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693410\",\"title\":\"Peter I van den Gheyn\",\"body\":\"\\nPeter I van den Gheyn\\n\\nPeter van den Gheyn (; 1500–1561) was a bell-founder of the Spanish Netherlands (now Belgium).\\nLife.\\nPeter was the son of Willem van den Gheyn, who established himself as a bell-founder at Mechelen in 1506. Peter followed his father into the trade, establishing his own foundry in 1528. His estate was known as or .\\nHe cast peal, table, and clock bells, but was most famed for the skill exhibited in his large bells. He is thought to have cast the bell of the \\\"Mary Rose\\\". Another of his bells hangs at St Peter's College, Cambridge.\\nHis two sons Peter and Jan both joined the family business as well.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693415\",\"title\":\"Leese-Stolzenau railway station\",\"body\":\"\\nLeese-Stolzenau railway station\\n\\nLeese-Stolzenau () is a railway station located in Leese and near Stolzenau, Germany. The station is located on the Weser-Aller Railway. The train services are operated by Deutsche Bahn.\\nTrain services.\\nThe following services currently call at the station:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693423\",\"title\":\"Nationalist Party (Burma)\",\"body\":\"\\nNationalist Party (Burma)\\n\\nThe Nationalist Party () was a political party in Burma in the 1920s led by U Pu and U Ba Pe.\\nHistory.\\nThe party was a successor to the 21 Party led by U Ba Pe, which had emerged as the largest in the 1922 elections after winning 28 seats, but had not been able to form a government. The new party called for a reduction in Indian immigration to Burma and Indian landlords to be banned from owning land in Burma.\\nIn the 1925 elections the Nationalist Party won 25 seats; despite being the largest party, it was not able to form a government.\\nIn 1926 or 1927 the party merged with the Home Rule Party and the Swaraj Party to form the People's Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693430\",\"title\":\"Shavit Matias\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693444\",\"title\":\"Flora Plumb\",\"body\":\"\\nFlora Plumb\\n\\nFlora Plumb (born October 14, 1944) is an American television and film actress.\\nPlumb was born Flora June Plumb, in Los Angeles, California. She made her TV debut in \\\"The Wild Wild West\\\" in 1969, in a bit part. She went on to feature appearances in several TV series, including \\\"Marcus Welby, M.D.\\\", \\\"The F.B.I.\\\", \\\"Lovers and Friends\\\" (a spinoff of the soap opera \\\"Another World\\\"), \\\"Lou Grant\\\", and \\\"Quincy, M.E.\\\". She also appeared in the 1978 film \\\"Malibu Beach\\\".\\nMore recently, she provided the voice of Oska for \\\"Guild Wars 2\\\" and appeared in the 2010 short film \\\"Death Panel\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693452\",\"title\":\"Katarzyna Mroczkowska\",\"body\":\"\\nKatarzyna Mroczkowska\\n\\nKatarzyna Mroczkowska (born ) is a retired Polish female volleyball player, who played as a middle blocker. She was part of the Poland women's national volleyball team at the 2001 Women's European Volleyball Championship and 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Palac Bydgoszcz.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693453\",\"title\":\"Home Rule Party (Burma)\",\"body\":\"\\nHome Rule Party (Burma)\\n\\nThe Home Rule Party () was a political party in Burma in the 1920s led by Tharrawaddy U Pu.\\nHistory.\\nThe party was formed as a breakaway from the General Council of Burmese Associations prior to the 1925 elections due to the GCBA continuing its calls for an electoral boycott. The elections saw the new party win 11 of the 80 seats.\\nIn 1926 or 1927 the party merged with the Nationalist Party and the Swaraj Party to form the People's Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693484\",\"title\":\"Petershagen-Lahde railway station\",\"body\":\"\\nPetershagen-Lahde railway station\\n\\nPetershagen-Lahde () is a railway station located in Petershagen, Germany. The station is located on the Weser-Aller Railway. The train services are operated by Deutsche Bahn.\\nTrain services.\\nThe following services currently call at the station:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693493\",\"title\":\"Filipe Silva Tavares Vieira\",\"body\":\"\\nFilipe Silva Tavares Vieira\\n\\nFilipe da Silva Tavares Vieira (born 28 October 1996 in Santa Maria da Feira) known as Vieirinha, is a Portuguese professional footballer who plays for C.D. Feirense as a midfielder.\\nFootball career.\\nOn 29 November 2015, Vieirinha made his professional debut with Feirense in a 2015–16 Segunda Liga match against Penafiel.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693494\",\"title\":\"Swaraj Party (Burma)\",\"body\":\"\\nSwaraj Party (Burma)\\n\\nThe Swaraj Party () was a political party in Burma in the 1920s.\\nHistory.\\nThe party was formed by former members of the General Council of Burmese Associations prior to the 1925 elections, and was named after the Indian Swaraj Party. Its leadership included Ba Maw, N. C. Bannerjee and U Tok Kyi, who was head of the party. After his death he was succeeded by U Paw Tun.\\nIn the 1925 elections the party won nine seats. In 1926 or 1927 it merged with the Nationalist Party and the Home Rule Party to form the People's Party. Following the dissolution of the People's Party in the early 1930s, most of the former Swaraj Party leadership joined the Anti-Separation League.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693496\",\"title\":\"Peter McQuade\",\"body\":\"\\nPeter McQuade\\n\\nPeter Murray McQuade (born 4 November 1948) was a Scottish footballer who played for Dumbarton, East Fife and Berwick Rangers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693501\",\"title\":\"2014–15 Uganda Super League\",\"body\":\"\\n2014–15 Uganda Super League\\n\\nThe 2014-15 Ugandan Super League is the 48th season of the official Ugandan football championship, the top-level football league of Uganda.\\nOverview.\\nThe 2014-15 FUFA Super League was contested by 16 teams, including Lweza FC, Sadolin Paints FC and Rwenshama FC who were promoted from the Ugandan Big League at the end of the 2013-14 season.\\nParticipants and locations.\\nSome of the Kampala clubs may on occasions also play home matches at the Mandela National Stadium.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693517\",\"title\":\"Sheikh Tariq Rashid\",\"body\":\"\\nSheikh Tariq Rashid\\n\\nSheikh Muhammad Tariq Rasheed, Urdu: شيخ محمد طارق رشيد, (born in Multan) is a Pakistani politician who is affiliated with Pakistan Muslim League Nawaz. He was previously Member of the National Assembly of Pakistan (MNA) (2010–2013) and Mayor of Multan (1998–2001). He is also the General Secretary of the Pakistan Muslim League Noon Multan City.\\nFamily and education.\\nHe completed his initial education at Millat High School and then Government College in Multan and graduated from Multan.\\nTariq Rasheed hails from a family that traces its roots to Jhajjar tehsil located in Rohtak district, India. His Grandfather, Sheikh Abdus Samad was assassinated while migrating to Pakistan at the time of the independence of Pakistan in 1947. Tariq Rasheed and his family represent a community which has large business and political interests in the urban politics of Multan city.\\nHis father, Sheikh Muhammad Rasheed, was his political mentor; the former Member of Federal Parliament (Majlis-e-Shoora) in 1980s nominated by General Zia-ul-Haq and elected Member of National Assembly of Pakistan (MNA), when elections were held on a non-party basis for the term 1985–1988.\\nTariq Rasheed is married and has two sons. The eldest son Sheikh Hashim Rasheed is currently studying abroad and the other son Sheikh Hassan Tariq is studying Law from TILLS.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693535\",\"title\":\"Oma maa\",\"body\":\"\\nOma maa\\n\\n (My Own Land or Our Native Land, literally: Own land/country), Op. 92, is a cantata by Jean Sibelius. He composed the work on a Finnish text, a poem by Kallio which he chose himself, for mixed chorus and orchestra, to celebrate the tenth anniversary of the National Chorus. Sibelius completed the cantata on 18 March 1918. conducted the first performance.\\nThe piece was written at a time when the Red Guards, supporting the Russian administration of Finland, were losing their influence. Sibelius had been invited by the director of te national choir of Finland to contribute something for a concert celebrating the tenth anniversary of the choir's establishment.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693538\",\"title\":\"Naganowski\",\"body\":\"\\nNaganowski\\n\\nNaganowski (feminine: Naganowska) is a Polish-language surname. It may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693543\",\"title\":\"Mary Bartlett Bunge\",\"body\":\"\\nMary Bartlett Bunge\\n\\nMary Bartlett Bunge (born 1941) is a distinguished American neuroscientist currently researching a cure for paralysis at University of Miami, where she is a Professor of Cell Biology. Bunge was born in a time where women were still in separate spheres from men and limited by society to domestic life. She challenged these expectations set by society and followed her passion for science, becoming an inspirational role model for women today.\\nEarly life.\\nMary Bartlett was born on April 3, 1931 in New Haven, Connecticut to George Chapman Bartlett and Margaret Elizabeth Reynolds Bartlett. Her father built and renovated houses, including the house in which she grew up, whereas her mother worked as a painter and decorator. Neither of her parents had a college education, and her father thought that a college education was useless for women. Their careers were filled with an artistic expression that Mary found appealing. After her grandmother taught her how to sew, she expressed herself through art and fashion by designing and making all of her own clothes with the ultimate dream to be a fashion designer in New York City. She strongly considered this career in fashion design, but eventually decided her art interests could just be hobbies. She was also interested in the sciences from a very young age, even though it was not considered a traditional option for women at the time. She developed this passion for biology when she was exploring the stream near her home in her leaky little row boat. She observed the tadpoles swimming around her and questioned how they developed into frogs. Unlike her art interests, she knew that in order to pursue her childhood passion for biology, she would need to acquire a higher education.\\nEducation.\\nMary Bartlett made the first step towards this higher education when she attended Simmons College in Boston to become a laboratory technician. This defied expectations for women at the time as the 1950s female college student was encouraged to marry, start a family, and put an end to her education. Only about a third of the women who entered college during the decade actually graduated. Only ten percent of working women entered a profession during this time. They instead settled for the traditional employment in secretarial, clerical, nursing, teaching, assembly lines, and domestic service, which were considered appropriate for women.\\nDespite these statistics, Mary persevered, and at the end of her junior year at Simmons College, she was inspired to further her education while attending a program at Jackson Memorial Laboratory, where she witnessed a rabbit’s heart contract in a tissue culture. This instance triggered the realization that she did not want to be a lab tech; she wanted to do research, so when she graduated from Simmons College in 1953, she accepted the invitation to graduate school at University of Wisconsin Medical School from Dr. Robert Schilling. He was a professor in the Department of Medical Physiology who offered Mary a research assistantship position when she was studying to obtain her masters degree. They researched intrinsic factor, which is lacking from the gastric juices when one has the condition of pernicious anemia and cannot absorb vitamin B12. Their research had a clinical relevance that influenced her later research to be focused toward clinical applications. This work was the basis of her thesis, which allowed her to graduate with her masters degree in medical physiology in 1955.\\nThere was implicit bias in society that women were not competent enough to excel in the science profession, so they were not always taken seriously when trying to attain this goal. Mary noted that Dr. Schilling was an exception to this and was an outstanding mentor to her. He helped set the high standards for her manuscripts when she was initially getting published. Women at the time struggled to get their work published because of the gender discrimination. When the author listed was a female name, people tended to reject the paper or not even read it, so women had to use their initials. Then, the author could not be labeled male or female, and the women had a better chance for equal evaluation of the work.\\nWhile studying for her doctorate at University of Wisconsin Medical School, Mary worked in the Zoology Department with Dr. Hans Ris, who she also regarded as an outstanding mentor. She was grateful for his willingness to advise a young woman as she faced the extra challenges simply due to her gender. She noted that both of her two mentors, Dr. Schilling and Dr. Ris, greatly shaped and benefited her future. They were able to be gender blind when others could not and treated her as just another graduate student. With their support, she successfully graduated with her doctorate in 1960.\\nPersonal life.\\nWhile at the University of Wisconsin, Mary met a medical student named Richard Bunge, whom she married and shared her personal life as well as her career. They graduated together and moved to Columbia University to begin their post-doctorate research. Soon after settling in, the family doubled with the addition of two sons, Jonathan, born in 1962, and Peter, born in 1964.\\nSince women perform most of the caregiving in society, they face more challenges than men in pursuing high level positions while balancing the family responsibilities. Bunge faced these challenges head on and was able to continue her career while attending to her motherhood responsibilities, but not without sacrifices. To balance family and work life, Bunge worked part time for eight years as a Research Associate at Columbia University. It would have been too much for her to prepare lecture and teach on top of her research schedule and raising her sons, so she placed her tenure-track faculty member dream on hold.\\nIn 1970, the family moved to accept faculty positions at Washington University School of Medicine. She chose to be a Research Assistant Professor rather than be on the tenure-track so she could continue raising her sons, who were still young. Instead, she adjusted to a full-time schedule. By 1974, she had started to teach and was promoted to Associate Professor with tenure. She was promoted again in 1978 to Professor. She was grateful for the quality of environment at Washington University, which made it possible for her to continue her career while being a part of her children's lives.\\nResearch.\\nRichard also enriched Mary’s life by introducing her to neuroscience, where she found her purpose and focused her research on while at Washington University School of Medicine. In particular, she focused on researching Schwann cells, which are cells that wrap around the axon of neurons to form the myelin sheath as an insulator to the neuron and to increase the speed impulses are conducted. One of her other major discoveries was that the oligodendrocyte was the cell that made the myelin sheath for the central nervous system. She first discovered this when she examined a section of a kitten’s spinal cord in an electron microscope with the oligodendrocyte cell body forming myelin at each end. She also demonstrated that myelin could be reformed in the mature mammalian spinal cord, which has an important clinical relevance in addressing Multiple Sclerosis and spinal cord injuries, where the myelin has been damaged.\\nSince 1989, Bunge has been a leading part of the Miami Project to Cure Paralysis at University of Miami School of Medicine, where her research on myelin has been implemented. Her husband was invited to be the scientific director of the project, so she was able to work with him there, and when he died in 1996 from esophageal cancer, she took his place at the forefront of the project. This was a common way for women of this era to acquire leading roles in projects. Women used the careers of their husbands to gain access to the resources to further their own careers, and when they died, the women were able to take their place, avoiding possible gender discrimination. The project Bunge took over tests regeneration strategies that could lead to successful treatment of spinal cord injury.\\nShe has the patent in “Schwann Cell Bridge Implants and Phosphodiesterase Inhibitors to Stimulate CNS Nerve Regeneration” from 2009 for the application of restoring function after a central nervous system injury. She has dozens of other patents including \\\"Methods and Systems for Neural Maintenance and Regeneration,\\\" \\\"Promoters of Neural Regeneration,\\\" and \\\"Phosphodiesterase 4 Inhibitors for Cognitive and Motor Rehabilitation.\\\" Her research is now being used for phase one of clinical trials, which gained approval from the FDA in 2012, to evaluate the safety of transplanting the Schwann cells of recently paralyzed patients into the site of their injury.\\nPhase 1 of the clinical trial is the testing of the safety and efficacy of this technique. Each participant’s own Schwann cells are obtained from sural nerve biopsy, which is above the ankle. By obtaining these cells from the participants, there is minimal concern for immune rejection. The cells are then purified and processed in a culturing facility to generate the number of cells necessary to be transplanted into the injury site. This cell therapy is combined with an intense exercise and rehabilitation regiment over ten months for the trial. The participants are to be monitored for 5 years. So far, Phase 1 has shown promising results as millions of Schwann cells have been successfully transplanted into four subjects with no adverse effects.\\nWhile the trial has been occurring, Bunge has been working on other combination treatments for future clinical trials. In 2014, she published in the Journal of Neuroscience the promising results of a strategy tested in rat Schwann cells that were engineered to secrete the growth factor D15A and the enzyme Chondroitinase ABC which alters scar composition. This combination lead to more axonal regeneration and functional improvement.\\nBunge is extremely dedicated to her work, and although she considered retiring in 2010, she has remained in the lab to see through the success of the clinical trial.\\nHonors.\\nBunge has been a professor of cell biology, neurological surgery, and neurology at the University of Miami for 26 years now and has accumulated significant recognition for her research. In 1996, she received the Wakeman Award for Spinal Cord Repair. She is a three time recipient of the Javitis Neuroscience Investigator Award from the National Institute of Neurological Disorders and Stroke. She was the elected Chair of the Development of Women's Careers in Neuroscience Committee through the Society for Neuroscience from 1994 - 2002. In 2000, she received the Mika Salpeter Women In Neuroscience Lifetime Achievement Award for her leadership in advancing the careers of women in neuroscience. In 2001, she received the Christopher Reeve Research Medal for Spinal Cord Repair. She received the Christine E Lynn Distinguished Professor in Neuroscience Award in 2003 and the Lois Pope LIFE International Research Award in 2005. Her culminating recognition is being elected to the National Academy of Sciences Institute of Medicine, which is an extraordinary honor that indicates the extent of her professional achievement and commitment to service in health and medicine. Just last year, she was named a Health Care Hero by the Greater Miami Chamber of Commerce.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693547\",\"title\":\"Joanna Szeszko\",\"body\":\"\\nJoanna Szeszko\\n\\nJoanna Szeszko (born ) is a retired Polish female volleyball player, who played as a universal . She was part of the Poland women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with SKRA Warschau.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693565\",\"title\":\"Terry Mullen\",\"body\":\"\\nTerry Mullen\\n\\nTerence James 'Terry' Mullen (born 28 February 1956) was a Scottish footballer who played for Dumbarton, Falkirk, Stenhousemuir and East Stirlingshire.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693579\",\"title\":\"La Tribuna (Paraguay)\",\"body\":\"\\nLa Tribuna (Paraguay)\\n\\nLa Tribuna was one of the most important daily newspapers in Paraguay's history. It was founded in 1925 in Asunción by Eduardo Schaerer, and was the country's leading newspaper for five decades. \\\"La Tribuna\\\" was the first Paraguayan paper to be widely published across the country. Politically, \\\"La Tribuna\\\" was positioned between liberalism and traditional national politics. It opposed the liberal government of José Patricio Guggiari, and especially the dictatorial regimes of Higinio Moríñigo and Alfredo Stroessner. When associated with opposition politics, the editors often became the target of persecution. Their crusade in favor of the Free Press in Paraguay gained attention abroad. In 1953, the newspaper's director, Arturo Schaerer, received the María Moors Cabot Prize from Columbia University.\\nHistory.\\n\\\"La Tribuna\\\" was founded in Asunción on December 31, 1925, by former Paraguayan president Eduardo Schaerer. \\\"La Tribuna\\\" was not Schaerer's first media outlet. In 1905, Eduardo Schaerer founded \\\"The Journal\\\" alongside Guadalberto Cardús Huerta and Adolfo Riquelme.\\nAfter the death of President José Félix Estigarribia in 1940, General Higinio Morínigo became President. Morínigo persecuted many politicians and well known members of the liberal party. This persecution of the independent press resulted in the closure of \\\"La Tribuna\\\" several times. General Alfredo Stroessner ascended to the Presidency of Paraguay in 1954. During his Presidency, \\\"La Tribuna\\\" continued to exist under similar circumstances and was constantly threatened. Schaerer asked for support from ambassadors and international contacts to keep the newspaper in operation. The continuing existence of \\\"La Tribuna\\\" worried the dictatorial government.\\nStarting in 1954, \\\"La Tribuna's\\\" director had the support of Carlos Ruiz Apezteguia, who denounced the abuse and crimes of the government and the breakdown of the rule of law. In November 1956, \\\"La Tribuna\\\" was raided and Ruiz Apezteguia was arrested, tortured and then abandoned in a boat on the shores of Clorinda, Argentina. Ruiz Apezteguia began his exile in Montevideo, Uruguay. Under international pressure, Ruiz Apezteguia returned to Paraguay in 1959 and resumed his journalistic work in the newspaper.\\nDespite the political unrest and the persecution \\\"La Tribuna\\\" was subjected to in the following decades, the newspaper grew into one of the most respected newspapers on the continent. \\\"La Tribuna\\\" had agencies in many countries and grew from a circulation of 2,000 daily copies at its founding to more than 70,000 by the year 1965. In 1953, Arturo Schaerer and his newspaper were given the oldest international journalism award, the Maria Moors Cabot prize from Columbia University, for the paper's work as independent press publication placing itself in opposition to the totalitarian regimes in Paraguay.\\nArturo Schaerer remained director of \\\"La Tribuna\\\" until May 15, 1972, when he was succeeded by Carlos Ruiz Apezteguia. At the time, Ruiz Apezteguia was a journalist and the husband of Schaerer's daughter, Myriam, and had been an involved partner and manager of \\\"La Tribuna\\\" for more than two decades. As director of La Tribuna, he denounced abuses in the negotiations associated with the Treaty of Itaipu and Yacyreta (with Brazil and Argentina respectively) on the construction of dams.\\nIn 1976, \\\"La Tribuna\\\" was closed, along with the newspaper \\\"Ultima Hora\\\". \\\"La Tribuma\\\" reopened in 1979. In 1983 \\\"La Tribuna\\\", with new owners, became the newspaper Noticias.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693582\",\"title\":\"Adrian Patrick\",\"body\":\"\\nAdrian Patrick\\n\\nAdrian Leroy John Patrick (born 15 June 1973) is a retired English sprinter who specialised in the 400 metres. He represented Great Britain at one outdoor and two indoor World Championships.\\nHis personal bests in the event are 45.63 seconds outdoors (Lausanne 1995) and 46.77 seconds indoors (Birmingham 1999).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693584\",\"title\":\"Lindum Hockey Club\",\"body\":\"\\nLindum Hockey Club\\n\\nLindum Hockey Club is a field hockey club located in Lincoln, England. The club was formed in the Spring of 2015, a merger between Lincoln Hockey Club and Lincoln Roses Hockey Club. Lindum Hockey Clubs plays its home games at the Lindum Sports Association on St Giles Avenue in Lincoln.\\nThe Mens 1st Team play in the North Hockey Association Division One, while the Ladies 1st Team play in the East League One North. The majority of the other teams compete in the Yorkshire Hockey League. As Lincoln's only field hockey club, the club fields eight men's teams, including a development side, six ladies' teams, a veterans side and a well established junior hockey section.\\nHistory.\\nDespite being founded only relatively recently, the club's history stretches back over a combined 200 year period. Lincoln Roses Hockey Club began life as a part of the Sports Club of Rose Brothers (Gainsborough) Limited, a manufacturer of sweet wrapping equipment. Gainsborough Rose Hockey Club, as they were known at the time following an amalgamation with Gainsborough Ladies Hockey Club, started to take on organised hockey with the formation of the Lincolnshire Hockey League. Later, with the advent of artificial turf becoming the de-facto playing surface in the sport, facilities were sought in Lincoln and in time a formal link with Lincoln University saw the club renamed Lincoln Roses Hockey Club.\\nLincoln Hockey Club was built on the foundations of Lincoln Ladies Hockey Club, started in 1898. The club's first links with the Lindum Sports Association came about in 1946 where the men’s club was also formerly created - named Lincoln Imps Hockey Club. The Lindum Sports Association was to be the base for both hockey clubs over the next 43 years before the need to use artificial turf took the club elsewhere in the city for their home games. In 1980 the two clubs were merged to became a single entity, continuing to be called Lincoln Imps Hockey Club, until 1998 when it became simply Lincoln Hockey Club. Olympic medalist and England / Great Britain International player Georgie Twigg is a former player of Lincoln Hockey Club.\\nThe Merger.\\nAn attempt to arrange a merger between Lincoln Roses Hockey Club and Lincoln Hockey Club in the mid to late-2000s proved unsuccessful and the two clubs continued to provide opportunities to play hockey and promote the sport side-by-side in the same city. In 2013 a purpose-built artificial turf hockey pitch was laid at the Lindum Sports Association, in part funded by England Hockey and Sport England. Some of the logic behind the funding of the new facility was that two hockey clubs would benefit and so both Lincoln Hockey Club and Lincoln Roses Hockey Club were playing most of their home games at the same location. In early-2014, a team consisting of Gary Johnson, John Harrison, Mark Sadler and John Sisman put in place the plans and foundations for a merger of the two clubs. Extraordinary General Meetings were called at both clubs to present a vision for the future of hockey in Lincoln and, with that, both clubs formerly dissolved on June the 8th 2015 and, on the same date, Lindum Hockey Club was created in their place.\\nBranding.\\nIn order to strike a balance between the two clubs, the decision was made to start afresh with branding. The new club name wouldn't involve any merging of the two former names but instead adopt the name used at the Lindum Sports Association, which is also used by other member clubs using the same facilities (the Lindum Cricket Club and Lindum Squash Club). The logo was inspired by Lindum Cricket Club's logo; the Roman of the 9th Roman Legion who founded Lindum Colonia.\\n2015 / 2016 Season.\\nAs well as the successful merger of the two former hockey clubs in the season, the season saw an affiliation in excess of 720 players. The Mens 1st Team also played Beeston Hockey Club at home in a cup clash that saw record crowds at the pitch-side, and the club organised and ran a successful Fireworks Event at the Lindum Sports Association.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693592\",\"title\":\"Bullhorn, Inc.\",\"body\":\"\\nBullhorn, Inc.\\n\\nBullhorn is a cloud computing company headquartered in Boston, Massachusetts. The company provides customer relationship management (CRM) software that is primarily used by professional services companies. As of 2015, the company reported more than 10,000 customers in more than 150 countries. Besides its Boston headquarters, the company has operations in St. Louis, London, and Sydney.\\nHistory.\\nBullhorn was founded in 1999 by Roger Colvin, Barry Hinckley, and Art Papas. Papas continues to serve as CEO. The company originally launched as a platform for freelancers to find and collaborate on work, but in 2001 changed its focus to build CRM software for vertical markets.\\nThe company historically focused on providing software-as-a-service to third-party staffing and recruiting firms, allowing them to manage business operations on a single web-based platform. It became one of the largest providers of technology to the staffing and recruiting market, reportedly growing revenue from $2 million in 2004, to $20 million in 2009, $33.6 million in 2011, and $67 million in 2013.\\nBullhorn raised its first round of venture capital funding in 1999 with a $4 million investment from GE Asset Management and Internet.com. It then raised $26 million from General Catalyst Partners and Highland Capital Partners in 2008. In June 2012, Vista Equity Partners acquired Bullhorn for a reported price of several hundred million dollars.\\nSince being acquired, the company has expanded into new markets, selling its CRM software to additional industries, beyond staffing and recruiting, and introducing Bullhorn Pulse, a data-mining and analysis tool that gives companies information about their relationships with customers.\\nIn July 2015, the industry research firm Gartner included Bullhorn for the first time in its Magic Quadrant report on salesforce automation alongside other companies that it deems to be leaders in providing tools to support the automation of sales and account management activities and processes.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693597\",\"title\":\"Wanlapa Jid-ong\",\"body\":\"\\nWanlapa Jid-ong\\n\\nWanlapa Jid-Ong (, born ) is a retired Thai female volleyball player, who played as a setter. She was part of the Thailand women's national volleyball team at the 1998 FIVB Volleyball Women's World Championship in Japan. and at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pepsi Bangkok.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693605\",\"title\":\"Sommai Niyompon\",\"body\":\"\\nSommai Niyompon\\n\\nSommai Niyompon (, born ) is a retired Thai female volleyball player, who played as a wing spiker. She was part of the Thailand women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pepsi Bangkok.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693610\",\"title\":\"Nurak Nokputta\",\"body\":\"\\nNurak Nokputta\\n\\nNurak Nokputta (, born ) is a retired Thai female volleyball player, who played as a wing spiker. She was part of the Thailand women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pepsi Bangkok.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693612\",\"title\":\"Mathias Denman\",\"body\":\"\\nMathias Denman\\n\\nMatthias Denman (1760 - 1838) is one of the founders of the settlement that eventually became Cincinnati, Ohio.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693613\",\"title\":\"Suphap Phongthong\",\"body\":\"\\nSuphap Phongthong\\n\\nSuphap Phongthong (, born ) is a retired Thai female volleyball player, who played as a middle blocker. She was part of the Thailand women's national volleyball team at the 2002 FIVB Volleyball Women's World Championship in Germany. On club level she played with Pepsi Bangkok.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693637\",\"title\":\"Jane Fraser (banking)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693648\",\"title\":\"Rudy-Kodzoff House\",\"body\":\"\\nRudy-Kodzoff House\\n\\nThe Rudy-Kodzoff House is a historic house at 2865 Mendenhall Loop Road in Juneau, Alaska. It is a concrete structure with Craftsman/Bungalow styling, built in 1915 for Charles Rudy, one of the first settlers of the Mendenhall Valley. It is the only surviving building of that period in the valley. It presently houses the offices and owner's residence of a mobile home park developed by Kodzoff family.\\nThe house was listed on the National Register of Historic Places in 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693676\",\"title\":\"75th (Middlesex) Searchlight Regiment, Royal Artillery\",\"body\":\"\\n75th (Middlesex) Searchlight Regiment, Royal Artillery\\n\\n75th (Middlesex) Searchlight Regiment, Royal Artillery was an air defence unit of Britain's Territorial Army (TA) raised just before the outbreak of World War II, which served as part of Anti-Aircraft Command in the early part of the war. Later it changed role and served in Sicily and Italy, and reformed in AA Command after the war.\\nOrigin.\\nAs the international situation deteriorated in the late 1930s, the threat of air raids on the UK led to a rapid expansion in numbers of anti-aircraft (AA) units manned by members of the part-time TA. Formed in May 1939, the 75th (Middlesex) Searchlight Regiment of the Royal Artillery consisted of HQ and three searchlight batteries based at Cowley in Uxbridge.\\nWorld War II.\\nThe Blitz.\\nAnti-Aircraft Command mobilised in August 1939, ahead of the declaration of war on 3 September, and the regiment took its place in 38th Light Anti-Aircraft Brigade (a formation composed entirely of searchlight units) in 1st AA Division, which had responsibility for defending London.\\nAlthough AA Command and RAF Fighter Command had the advantage of the Chain Home radar network to detect air raids approaching the coastline, there was no radar coverage inland. In daylight the Royal Observer Corps and searchlight detachments tracked the progress of raids visually; at night, sound location had to be used. 1st AA Division devised a system of 14 fixed base-lines of sound locators to detect night raids approaching the London Inner Artillery Zone. These were linked by automatic telephone equipment to the operations room, where the angular plots were resolved to indicate grid squares where the Heavy AA guns in range could fire an unseen barrage. Detachments of 75th S/L Rgt were trained to operate these baselines. This 'Fixed Azimuth' system came into action in June 1940, in time for the opening of the night Blitz on London. It was later replaced when searchlight control (SLC) and gunlaying (GL) radar systems were introduced.\\nLondon was attacked incessantly throughout the winter of 1940–41. 75th S/L Regiment served in 38 AA Bde in 1 AA Division in London through these raids until February 1941 when its role was changed.\\n75th (Middlesex) Light Anti-Aircraft Regiment.\\nIn February 1941 the regiment was re-equipped with AA guns and converted into 75th (Middlesex) Light Anti-Aircraft Regiment, RA. \\nIn 1943 the regiment was despatched to the Mediterranean to join Eighth Army for the Allied invasion of Sicily (Operation Husky). The assault went in early on 10 July against light opposition, and all Eighth Army's landings were successful. The town of Pachino was quickly secured, and 75th LAA's Bofors guns landed shortly afterwards to contribute to the AA defence of its airfield. Work started at once to repair this for use by the RAF, though this was interrupted by frequent day and night air raids. Even after the airfield was in operation, raids against it continued until the Luftwaffe withdrew to mainland Italy in mid-August. At this stage 75th LAA Rgt had 233 and 234 Batteries protecting Pachino airfield and the beach, and 303 Battery at Cassibile Airfield. \\nEighth Army captured the port of Taranto on the Italian mainland in September 1943 (Operation Baytown), and during October 8th AA Bde was landed there to defend the disembarkation ports and airfields in the 'heel' of Italy (southern Puglia) in a joint air defence organisation with the RAF. 75th LAA Rgt was one of the units assigned to this task, with one battery of 18 Bofors guns deployed at Brindisi Airfield, and the rest of the regiment as detachments at other airfields in the area.\\nIn December 1943, 25th AA Bde relieved 8th AA Bde and assumed command of 75th LAA Rgt. It upgraded the defence of Brindisi port by moving in another battery of 75th LAA Rgt. However, as the Italian Campaign progressed the region was becoming a backwater and the fighting units of Eighth Army urgently required manpower, so from May 1944 a reduction in AA units was begun. 75th LAA Rgt was one of those selected for disbandment and by July it had been placed in 'suspended animation' and its personnel posted away.\\nPostwar.\\nWhen the TA was reconstituted in 1947, 75th LAA Rgt was reformed at Cowley as 610 Light Anti-Aircraft Regiment, RA (Middlesex), forming part of 82 AA Bde (the former 56 AA Bde based at Heston).\\nIn 1950 the regiment was merged into another unit in 82 AA Bde, 604 Light Anti-Aircraft/Searchlight Regiment, RA (Royal Fusiliers) (the former 69th (3rd City of London) S/L Rgt), and the 75th S/L Rgt lineage ended.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693682\",\"title\":\"John Taylor (Scottish footballer, born 1949)\",\"body\":\"\\nJohn Taylor (Scottish footballer, born 1949)\\n\\nJohn H. C. Taylor (born 22 June 1949) was a Scottish footballer who played for Queens Park, Dumbarton and Stranraer.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693697\",\"title\":\"Fifty pounds (British coin)\",\"body\":\"\\nFifty pounds (British coin)\\n\\nThe fifty pound coin (£50) is a commemorative coin of the pound sterling. Issued for the first time by the Royal Mint in 2015 and sold at face value, fifty pound coins hold legal tender status but are intended as collectors' items and are not found in general circulation. 100,000 coins will be produced in limited edition presentation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693698\",\"title\":\"You're Everything (album)\",\"body\":\"\\nYou're Everything (album)\\n\\nYou're Everything is a jazz album produced by Schoener Hören Records and , it was officially released in March 2008. The album was critically acclaimed by magazine as the second recording for the Berlin Jazz Orchestra with vocal artist . Jazz artist Jiggs Whigham is featured on this release as both instrumentalist (trombone) and musical director.\\nBackground.\\nAfter being founded in 2000, the Berlin Jazz Orchestra had their first demo release with the label 44 Records in 2004 (\\\"Update\\\"), produced by Jacky Wagner. Roughly the same program was to be released later in 2007 for \\\"You're Everything.\\\" The \\\"Update\\\" CD release paved the way for this in creating fine demo and test CD where many details of Steve Gray's arrangements were worked out in the studio.\\nReception/Profesional ratings.\\n\\\"Marc Secara and his Berlin Jazz Orchestra present with \\\"You're Everything\\\" a lavishly produced CD. Contributing flattering, powerful, and despite his youth, a secure voice and soulful interpretations that are not too corny, just as did the rousing original arrangements by Steve Gray, the funky brass riffs and jazzy solos of a big band under the direction of trombone soloist Jiggs Whigham. The strings remain subtly in the background. \\n... an unconditional recommendation for all friends of an 'old crooner.' Tasty and sure past footsteps, Secara makes big steps of his own.\\\"\\nJazzpodium\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693705\",\"title\":\"Moodymann (album)\",\"body\":\"\\nMoodymann (album)\\n\\nMoodymann is a 2014 studio album by American electronic musician Moodymann.\\nReception.\\nAt Metacritic, which assigns a weighted average score out of 100 to reviews from mainstream critics, the album received an average score of 83% based on 7 reviews, indicating \\\"universal acclaim\\\".\\nJames Williams of \\\"Exclaim!\\\" described the production as \\\"dark hypnotic melodies coupled with his signature chopped-up vocal tics, pulsing hardware synths and delightfully warped stories from his native home of Detroit.\\\" Will Lynch of \\\"Resident Advisor\\\" said, \\\"nobody else makes house records as weird and rich as this.\\\"\\nIt ranked at number 7 on \\\"Rolling Stone\\\"s \\\"20 Best EDM, Electronic and Dance Albums of 2014\\\" list.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693710\",\"title\":\"Boley Springs, Alabama\",\"body\":\"\\nBoley Springs, Alabama\\n\\nBoley Springs is an unincorporated community in Fayette County, Alabama, United States. A post office operated under the name Boley Springs from 1877 to 1887.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693716\",\"title\":\"Education in Asia\",\"body\":\"\\nEducation in Asia\\n\\nEnrollment in educational institutions varies considerably across the continent of Asia, as evidenced by data maintained by United Nations Educational, Scientific and Cultural Organization (UNESCO). UNESCO's measurement categories for education are used in the context of international development work, and are adopted by the World Bank in its EdStats database. The United Nations issues a Human Development Index for each nation, of which the Education Index is a component.\\nParticipation in education.\\nThe Gross Enrollment Ratio (GER) is a component of the Education Index. It expresses the number of students enrolled in a given level of education as a percentage of the number of people within the official age for that level of education. GER can exceed 100% because some enrolled students may fall outside the official age range.\\nThe tables below show GER for each country in Asia, organized into five regions by population: South Asia, East Asia, Southeast Asia, West Asia and Central Asia. Data are shown for four levels of education: pre-primary, primary, secondary and tertiary. (Tertiary education is also referred to as higher education).\\nThe last year for which data are available is shown in parentheses following each number in the table. If the year is the same as for the column to the left, the year is omitted.\\nChallenges and Opportunities.\\nLow GER.\\nAs Asian nations compete in the global economy and aspire to join the developed nations, there is concern that rates of education may not be keeping pace. By comparison, Gross Enrollment Rates for North America and Western Europe in 2013 were 84.3% for pre-primary, 101.1% for primary, 105.1% for secondary, and 76.6% for tertiary education.\\nSupply versus demand.\\nMany Asian nations lack the capacity to scale up their enrollment to meet the escalating demand.\\nQuality in education at scale.\\nThere is also concern about a quality gap, as nations seek to scale up their enrollment quickly.\\nSkills gap.\\nThere is concern about a gap between the education sought by the labor market and what is being taught in the educational institutions.\\nDemographic dividend.\\nMany Asian countries - mostly in East Asia and Southeast Asia - experienced a demographic dividend that boosted their economies during the past few decades. There is a widespread view that the South Asian countries are poised to benefit from a demographic dividend because their populations are young relative to the developed countries. However, reaping this dividend is expected to require a work force that is well educated, which means, at a minimum, increasing enrollment rates and educational quality.\\nProgress.\\nEven though many Asian nations still have low GER compared with their North American and Western European counterparts, there has been considerable progress in recent years. For example, consider the change in GER over ten years preceding the latest data reported, for the three most populous Asian countries: China, India and Indonesia. All three countries had achieved virtually universal primary education (close to 100%) before this ten-year period, so consider the other three levels. Over a ten-year period, China's GER increased from 40% to 74% for pre-primary, from 60% to 92% for secondary, and from 15% to 30% for tertiary education. India's GER increased from 25% to 58% for pre-primary, from 48% to 71% for secondary, and from 11% to 25% for tertiary education. Indonesia's GER increased from 26% to 51% for pre-primary, from 61% to 83% for secondary, and from 15% to 32% for tertiary education.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693736\",\"title\":\"Don Watt (footballer)\",\"body\":\"\\nDon Watt (footballer)\\n\\nDonald 'Don' Watt (born 23 July 1953) was a Scottish footballer who played for Celtic, Dumbarton and East Stirlingshire.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693759\",\"title\":\"Burlington, Cedar Rapids & Northern Freight House\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693784\",\"title\":\"1994 FIVB Volleyball Women's World Championship squads\",\"body\":\"\\n1994 FIVB Volleyball Women's World Championship squads\\n\\nThis article shows all participating team squads at the 1994 FIVB Women's World Championship, held from November 17 to 30 October in Brazil.\\nHead coach: Faik Karayev\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Bernardo Rezende\\n\\nHead coach: Li Xiaofeng\\n\\nHead coach: Eugenio George\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Milan Kanfka\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Siegfried Kohler\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Tadayoshi Yokota\\n\\nHead coach: Aurelio Motta\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Cilbert Ohanya\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Chul-Yong Kim\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Bert Coedkoop\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Jong Park Dug\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Stan Gostinel\\n\\nHead coach: Nikolai Karpol\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Volodimir Buzayev\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\nHead coach: Taras Liskevych\\n\\\"Note: only a selection of 12 players listed below participated at the Championships\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693787\",\"title\":\"Axibase\",\"body\":\"\\nAxibase\\n\\nAxibase Corporation, headquartered in Cupertino, California, in the United States, is a developer of time series database and reporting software systems for infrastructure management and optimization. Axibase customers represent industries including financial services, transportation, health sciences, technology as well as federal and state governments in USA, Canada, Europe and Australia.\\nHistory.\\nThe company was founded in 2004 by Sergei Rodionov, an MBA student at the The Wharton School, after participating in the University of Pennsylvania Wharton Venture Initiation Program (VIP). The initial prototype technology developed by Axibase was targeted at medical device manufacturers and addressed their requirements for secure remote diagnostics of MRI, CT and nuclear medicine systems. In 2002 the prototype solution was presented at the Wharton Business Plan Competition under the project name Raylink Systems and was selected as one of the competition finalists. The industry-specific prototype was however discarded in 2004 in favor of next-generation technology stack optimized for cross-industry infrastructure monitoring use cases. To this date, Axibase remains one of Wharton Venture Initiation Program alumni companies.\\nResearch.\\nSince the introduction of Axibase Time Series Database in 2013 the company has forayed into smart meterting and engaged in research and developing projects focused on predictive analytics and applications of time-series forecasting to environmental issues on a global scale. Recently published papers include studies of satellite-based weather monitoring models, multi-variate forecasting of environmental pollution levels, and renewable energy resources in Northern Europe.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693794\",\"title\":\"The Lady Wants Mink\",\"body\":\"\\nThe Lady Wants Mink\\n\\nThe Lady Wants Mink is a 1953 American comedy film directed by William A. Seiter and written by Dane Lussier and Richard Alan Simmons. The film stars Dennis O'Keefe, Ruth Hussey, Eve Arden, William Demarest, Gene Lockhart and Hope Emerson. The film was released on March 30, 1953, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693814\",\"title\":\"Jane Fraser\",\"body\":\"\\nJane Fraser\\n\\nJane Fraser may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693823\",\"title\":\"Bad Romance (disambiguation)\",\"body\":\"\\nBad Romance (disambiguation)\\n\\nBad Romance may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693839\",\"title\":\"Three Little Pigs (company)\",\"body\":\"\\nThree Little Pigs (company)\\n\\nLes Trois Petits Cochons also known as Three Little Pigs is an American charcuterie company founded in 1975 in Greenwich Village of New York City. The company was founded by French chefs Alain Sinturel and Jean-Pierre Pradie along with their business partner Harvey Milstein.\\nHistory.\\nPradie and Sinturel met in the early 1970s. During a year long trip traveling through Africa the two met Harvey Milstein who would later become their business partner. In 1975, the trio decided to open a charcuterie company as they did not have the funding to open a restaurant. Their original space was only 300 square feet. In 1976, the company opened a second store. Three Little Pigs became known for its pâtés and catering business recognized by notable food critics such as James Beard, Mimi Sheraton and Craig Claiborne. In 1988, Three Little Pigs opened a facility in Wilkes-Barre, Pennsylvania named House of Bricks to accommodate the increasing demand for its products as it became a nationally distributed brand by the 1990s. \\nFounded as Les Trois Petits Cochons, French for \\\"Three Little Pigs,\\\" the company trademarked the name derived from the fairy-tale in 1996. Three Little Pigs celebrated its 40th year of business in 2015. \\nOverview.\\nSince 1975, the company has grown from a neighborhood take-out shop in Greenwich Village into a nationally distributed specialty food brand. Three Little Pigs makes hand-crafted pâté and charcuterie, such as mousses, terrines, hams, sausages and mustards, produced without the use of additives or preservatives. \\nSince its founding, the company has received 19 Sofi awards and in 2015, won a Cook's Illustrated award for its dijon mustard.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693857\",\"title\":\"Goslar railway station\",\"body\":\"\\nGoslar railway station\\n\\nGoslar () is a railway station located in Goslar, Germany. The station opened on 23 March 1866 and is located on the Vienenburg–Goslar railway, Hildesheim–Goslar railway and Neuekrug-Hahausen–Goslar railway. The train services are operated by Erixx, Deutsche Bahn and Transdev Sachsen-Anhalt.\\nTrain services.\\nThe following services currently call at the station:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693868\",\"title\":\"List of stories within The Malachite Box\",\"body\":\"\\nList of stories within The Malachite Box\\n\\nThis is a list of the stories in Pavel Bazhov's collection The Malachite Box. The first edition, released on 28 January 1939, consisted of 14 stories, based on the oral lore of the miners and gold prospectors. After the initial publication, the author continuously added new stories to the collection. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693869\",\"title\":\"Timeline of Limoges\",\"body\":\"\\nTimeline of Limoges\\n\\nThe following is a timeline of the history of the city of Limoges, France.\\nReferences.\\n\\\"This article incorporates information from the French Wikipedia.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693870\",\"title\":\"Leif Erikson Awards\",\"body\":\"\\nLeif Erikson Awards\\n\\nThe Leif Erikson Awards, sometimes referred to as the Exploration Awards, are awarded annually by the Exploration Museum in Húsavík, Iceland, for achievements in exploration and for documentation of exploration history. They are awarded in three categories; to an explorer for a lifetime achievement in exploration; to a young explorer under the age of 35 for achievements in exploration; and to a person or an organization that has worked to promote and preserve exploration history. The Leif Erikson Awards are the main and final event of the annual \\\"Húsavík Explorers Festival\\\". The awards were first awarded in 2015 and will be awarded next in October 2016.\\nThe awards are named for Icelandic explorer Leif Erikson, considered as the first European to land in North America and who, according to the Sagas of Icelanders, established the first Norse settlement at Vinland, tentatively identified with the Norse L'Anse aux Meadows on the northern tip of Newfoundland in modern-day Canada.\\nScientific Committee.\\nThe winners of the Leif Erikson awards are voted by the members of the Exploration Museum's Scientific Committee. The committee is appointed for one year by the board of the museum, except for the chairperson, who is the winner of the previous year's Leif Erikson Exploration History award.\\n2015 Awards.\\nThe 2015 Leif Erikson Awards were announced on 11 November at the Exploration Museum in Húsavík.\\nLeif Erikson Exploration Award \\nLeif Erikson Yong Explorer Award\\nLeif Erikson Exploration History Award\\n2016 Awards.\\nThe 2016 Leif Erikson Awards will be announced in October 2016. Nominations are scheduled to be accepted in January 2016. As the 2015 recipient of the Leif Erikson Exploration History Award, Dr. Huw Lewis-Jones will lead the Scientific Committee. Other members for 2016 are Ari Trausti Guðmundsson, Giuditta Gubbi, Francesco Perini and Vilborg Arna Gissurardóttir.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693909\",\"title\":\"Thomas A. Furness III\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693928\",\"title\":\"'Abdallah ibn Ishaq ibn Ibrahim\",\"body\":\"\\n'Abdallah ibn Ishaq ibn Ibrahim\\n\\n'Abdallah ibn Ishaq ibn Ibrahim () was a Mus'abid official in the service of the Abbasid Caliphate. He was briefly the governor of Baghdad in 851, and the governor of Fars in ca. 863.\\nCareer.\\n'Abdallah was a member of the Mus'abid family, a collateral branch of the Tahirid dynasty. Following the death of Muhammad ibn Ishaq ibn Ibrahim in July 851, 'Abdallah succeeded him as governor of Baghdad and chief of security (\\\"shurtah\\\") of the Sawad, but he quickly alienated his taxation officials by dealing with them in a harsh manner. In that same year he lost his position to Muhammad ibn 'Abdallah ibn Tahir, who arrived in October from Khurasan.\\nIn ca. 863 'Abdallah was appointed by Muhammad to act as his governor of Fars. While serving in that province, he withheld the pay of the local soldiers, which provoked them into rebelling against him and transferring their allegiance to 'Ali ibn al-Husayn ibn Quraysh. Lacking the means to counter the revolt, 'Abdallah was forced to abandon Fars and return to Baghdad.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693940\",\"title\":\"The Lady in the Van (soundtrack)\",\"body\":\"\\nThe Lady in the Van (soundtrack)\\n\\nThe Lady in the Van is the original soundtrack album for the 2015 film of the same name. Composed by George Fenton, it was released through Sony Classical Records, a subsidiary of Sony Music Entertainment that specialises in classical music and compositions.\\nTrack listing.\\nAll of the music was composed by George Fenton.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693941\",\"title\":\"Chris Cafero\",\"body\":\"\\nChris Cafero\\n\\nChristopher Lawrence Cafero (born October 24, 1988), is an American actor, model, and comedian. He has appeared in such television projects as CBS's \\\"As The World Turns\\\" (2008), HBO's \\\"Crime\\\" (2015), and in the films \\\"Life in Parallel\\\" (2012), \\\"Campus Code\\\" (2013)\\\",\\\" and \\\"Tower of Silence\\\" (2016). For several years he appeared as Blake Williams in the hit Off-Broadway play \\\"The Awesome 80's Prom\\\"\\\".\\\" He has also work on many national television, radio, and print campaigns.\\nEarly life.\\nChris was born in Norwalk, Connecticut, to father Lawrence F Cafero, Jr., a lawyer and politician, and to mother Barbara Cafero, a contracts manager for Xerox. He was raised in Norwalk along with his sister Jacqueline and brother Nick. Nick Cafero is an actor, singer, and writer in Los Angeles, and has appeared in NBC's The Office, and the feature film Pitch Perfect. Chris attended Brien McMahon High School, participating in their award-winning marching band, wrestling team, and drama program. In 2006, he was accepted to The George Washington University, receiving both presidential arts and academic scholarships. He joined the Sigma Chi Fraternity and served as his chapter president his senior year. He graduated in 2010 with magna cum laude honors. After graduating, he moved to New York City to pursue a full-time acting career.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693970\",\"title\":\"Axone (arena)\",\"body\":\"\\nAxone (arena)\\n\\nL'Axone is an indoor arena, located in Montbéliard, France. It was inaugurated April 5, 2009. The capacity of the arena is 6,400 peoples.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693977\",\"title\":\"Felisberto Micael Lopes Darame\",\"body\":\"\\nFelisberto Micael Lopes Darame\\n\\nFelisberto Micael Lopes Darame (born 18 March 1994) known as Betinho, is a Portuguese footballer who plays for Almancilense as a forward.\\nFootball career.\\nOn 9 January 2013, Betinho made his professional debut with Olhanense in a 2012–13 Taça da Liga match against Moreirense.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693980\",\"title\":\"Coal Valley, Alabama\",\"body\":\"\\nCoal Valley, Alabama\\n\\nCoal Valley, also known as Cormick, is an unincorporated community in Walker County, Alabama, United States.\\nHistory.\\nCoal Valley is named for the abundance of coal in the surrounding area. Coal mines were opened in Coal Valley after the completion of the Georgia Pacific Railway. The Coal Valley mines played a role in the 1920 Alabama coal strike. A post office operated under the name Cormick from 1890 to 1891 and under the name Coal Valley from 1891 to 1951.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48693992\",\"title\":\"Streptomyces cocklensis\",\"body\":\"\\nStreptomyces cocklensis\\n\\nStreptomyces cocklensis is a bacterium species from the genus of \\\"Streptomyces\\\" which has been isolated from soil from the Cockle Park Experimental Farm in Northumberland in the United Kingdom. \\\"Streptomyces cocklensis\\\" produces dioxamycin.\\nReferences.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694004\",\"title\":\"Norman C. Pickering\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694010\",\"title\":\"Yannick Ngakoue\",\"body\":\"\\nYannick Ngakoue\\n\\nYannick Ngakoue (born 1995) is an American football defensive end. He played college football at Maryland.\\nEarly years.\\nNgakoue attended Friendship Collegiate Academy Public Charter School in Washington, D.C., where he was teammates with Eddie Goldman. As a senior he was the Gatorade Football Player of the Year for Washington D.C. after recording 17 sacks. Ngakoue was rated by Rivals.com as a four-star recruit and was ranked as the fourth best outside linebacker in his class. He committed to the University of Maryland, College Park to play college football.\\nCollege career.\\nAs a true freshman at Maryland in 2013, Ngakoue played in all 13 games and had nine tackles, two sacks and an interception. As a sophomore in 2014, he started all 12 games and recorded 37 tackles and six sacks. As a junior in 2015, Ngakoue set a school record with 13.5 sacks. After the season he announced that he would forgo his senior season and enter the 2016 NFL Draft.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694050\",\"title\":\"West River Sheet Harbour\",\"body\":\"\\nWest River Sheet Harbour\\n\\nWest River Sheet Harbour (locally known as West River) is a river on the Eastern Shore of Nova Scotia, Canada, in the Halifax Regional Municipality. It's headwaters are near the Musquodoboit Valley and the river flows southeast and empties in to the Northwest Arm of Sheet Harbour, Nova Scotia. The river has three main tributaries: West River Main, Killag River and Little River.\\nTributaries.\\nWest River Sheet Harbour has three main tributaries: West River Main, Killag River and Little River.\\nWest River Main is the main tributary of the West River system. It is approximately long and it's headwaters are at an altitude of near Sheet Harbour Road, on the southern ridge of the Musquodoboit Valley. The river is prone to flash-flooding. There are two lake-like pools on the river. The first and more southerly one is Sheet Harbour Lake, which has an area of 1.2 km2 and is located near the West River Falls in Sheet Harbour. The other lake is Little Lake, which is more inland in the system and has a area of 0.5 km2. Killag River and Little River both discharge in to the Main West River. The river then flows through Sheet Harbour Lake, then down the West River Falls and discharges in to the Northwest Arm of Sheet Harbour.\\nThe Killag River is a secondary tributary of the West River Sheet Harbour system. It is approximately long. It's headwaters are at an altitide of . Most of the salmon in the river system breed on the Killag River. Killag River discharges in to Main West River.\\nThe Little River is a secondary tributary of the West River Sheet Harbour system. Its headwaters are in Lake Alma, a shallow lake at an altitude of . Little River is approximately long. Little River discharges in to Main West River.\\nLime dosing.\\nWest River has been dosed with a lime doser since September 2005. It was the result of the worsening salmon population in Atlantic Canada rivers, which is being caused be acid rain and low pH levels. It was successful in stabilizing the pH of the river at 5.5, which is a healthy level for salmon and other water life.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694057\",\"title\":\"Flag of Kemerovo Oblast\",\"body\":\"\\nFlag of Kemerovo Oblast\\n\\nThe flag of Kemerovo Oblast is a red rectangle with a blue stripe at the hoist side, its width 1/3 of the flag length. In the upper part of the blue stripe is the Kemerovo Oblast coat of arms. The coat of arms contains the year 1943, the year of the oblast's foundation, on a red Order of Lenin ribbon with gold edges. The emblem contains a pick axe and a hammer. The oblast is a major the coal and metal mining centre of Russia. The flag ratio is 1:2 however a variant used from 2003 is 2:3 ratio.\\nThe current flag was adopted on 7 June 2002, however according to Sergei Sherniakov of the Heraldic Committee of Perm Region the adoption date of the Kemerovo Oblast flag was 29 May 2002.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694068\",\"title\":\"Gaa Paa\",\"body\":\"\\nGaa Paa\\n\\nGaa Paa! (\\\"Forward!\\\" or literally \\\"Go On!\\\") was a Norwegian language socialist publication established in the United States of America at Girard, Kansas in November 1903, before moving to Minneapolis, Minnesota the following year. The paper was forced to change its name in 1918 in an effort to avoid a ban from the US Mail, taking the name Folkets Røst (People's Voice). It appeared under that title until the publication's demise in October 1925.\\nIn the aftermath of the 1919 split of the Socialist Party of America (SPA) which led to the establishment of the Communist Party of America (CPA) and Communist Labor Party of America (CLP), \\\"Folkets Røst\\\" remained allied with the social democratic SPA and promoted the candidacies of other like-minded groups. The weekly newspaper was the largest and longest-running radical Dano-Norwegian (Bokmål) periodical in North America.\\nPublication history.\\nEstablishment.\\nIn 1903, successful publisher Julius A. Wayland, publisher of the national weekly \\\"Appeal to Reason,\\\" decided to launch a Scandinavian language socialist newspaper from his base of operations at Girard, Kansas, a small town located in the southeastern part of the state. Wayland invited the married socialist activists Emil Lauritz Mengshoel and Helle Crøger Mengshoel to relocate to Kansas to edit this new publication.\\nWayland had met Emil Mengshoel — the editor of the radical Minnesota Populist newspaper \\\"Nye Normanden\\\" (New Norseman) — though his prior contributions to the \\\"Appeal to Reason,\\\" making him a logical choice for the editorial desk. Helle Crøger Mengshoel was a formidable activist in her own right, having worked previously as a trade union organizer in Christiana, Norway (today's Oslo). The couple assented to Wayland's request, moved to Kansas, and on November 29, 1903 the first issue of \\\"Gaa Paa\\\" (Forward) rolled from the \\\"Appeal's\\\" presses.\\nBurdened by other tasks, Wayland soon decided to step down from his direct role as publisher of \\\"Gaa Paa\\\", instead subsidizing publication of the paper through direct donations. The paper continued to be edited by the Menshoels, joined by Helle Menshoel's son, Andrew O. Devold, to make an editorial board of three. Throughout its history \\\"Gaa Paa\\\" would remain, in the words of historian Odd S. Lovoll, \\\"basically a family venture\\\".\\nIn the fall of 1904 the paper was moved to Minnesota, closer to its primary readership in the Norwegian immigrant communities of the Upper Midwest, where it would remain for the next two decades. The move was completed during the second half of October 1904, with offices established at 1910 East Franklin Avenue in Minneapolis.\\nDevelopment.\\n\\\"Gaa Paa\\\" was bolstered by the establishment of the Scandinavian Socialist Federation (Skandinaviske Socialistforbund) in 1910, a national organization which united local Swedish, Norwegian, and Danish language socialist clubs scattered around the United States. The organization of the disparate groups initially prove a positive for \\\"Gaa Paa,\\\" as the only Dano-Norwegian (Bokmål) socialist weekly, but in 1911 the paper soon found its position challenged when the Scandinavian Federation launched its own publication, \\\"Social-Demokraten\\\" (The Social Democrat). A feud developed between the privately owned and the Federation-owned papers as they battled for subscribers in a fairly limited market.\\n\\\"Gaa Paa\\\" was regarded by one contemporary observer as \\\"the reddest and most radical of Norwegian newspapers,\\\" touting itself on its masthead as the \\\"Organ for Scandinavial Workers in America.\\\" The paper based its editorial line upon the principles of International Socialism, with emphasis placed on the notion of class struggle.\\nDespite its purple prose, however, the paper steered far away from the anti-political industrial union the Industrial Workers of the World (IWW), explicitly disavowing that organization's commitment to the use of the tactics of sabotage and direct action. Rather than holding to an apocalyptic view of attaining socialism through armed revolution, editor Mengshoel was influenced by the ideas of writers Laurence Gronlund and Edward Bellamy, emphasizing the functional superiority of the economic form of state ownership and the implication that evolution to socialist production and distribution would be a protracted, inevitable, evolutionary process.\\n\\\"Gaa Paa\\\" attained a circulation of 5,000 in 1912 and managed to maintain a largely stable readership of about 4,500 during the years up to American entry into World War I. \\\"Gaa Paa\\\" maintained a national readership and published the work of correspondents from around the country, including in neighboring North Dakota as well as enclaves of Norwegian-American radicalism located in Seattle and Astoria, Oregon. The paper also opened a business office in the Midwestern ethnic metropolis of Chicago.\\nIn 1914 Andrew Devold threw his hat into the political ring on the Socialist Party ticket and won election to the Minnesota state legislature, becoming the second Socialist elected to that body in the state. Devold was at the time listed as editor and publisher of \\\"Gaa Paa\\\" together with his stepfather Emil Mengshoel. After leaving the employ of \\\"Gaa Paa\\\" in 1917, Devold would go on to win election to the Minnesota State Senate in November 1918.\\nWartime repression.\\nIn contrast to the overwhelming majority of the Norwegian-American press, \\\"Gaa Paa\\\" maintained an anti-war orientation even after American entry into World War I in April 1917. While taking a position of \\\"undiscriminating hate\\\" of \\\"German junkerdom, English aristocracy, and American money power,\\\" primary editor Emil Mengshoel expressed sympathy for Minneapolis socialists who dared to resist conscription. The paper never was so bold as to explicitly advocate resistance to registration and the draft, instead attempting to toe the fine line of legality while remaining true to the anti-militarist St. Louis proclamation of the Socialist Party of America.\\nAs with other non-English publications, \\\"Gaa Paa\\\" was also faced with the burdensome task of supplying English translations of all political articles and editorial comments during the wartime years. Efforts to soften tone and comply with statutory regulations in order to appease federal authorities proved inadequate for Postmaster General Albert S. Burleson, however, and in 1918 \\\"Gaa Paa\\\" was denied access to the United States mails. This proved a harsh blow to the financially unstable publication, threatening its survival.\\nIn an effort to keep the publishing operation alive, the Mengshoels resorted to an artifice widely used to beat European censorships, relaunching their publication with a new name, \\\"Folkets Røst\\\" (People's Voice) — a name regarded as one less militant than \\\"Gaa Paa.\\\" The old publication was shut down in October 1918 with the ostensibly \\\"new\\\" paper first seeing print on December 21, 1918, following a pause in publication of some two months.\\nSocialist split of 1919.\\n\\\"Gaa Paa\\\" moved away from the Socialist Party slightly in the spring of 1918, when it began printing columns written by Sigvald Rødvick, top Norwegian-language official in the national office of the Non-Partisan League (NPL), a rival organization. The publication lent its editorial support not only to the Socialist Party candidacy of former editor Andrew Devold for the Minnesota State Senate and to pro-war Socialist Thomas Van Lear in his bid for reelection as Mayor of Minneapolis, but also to the candidate of the NPL for Governor of Minnesota, Charles A. Lindbergh.\\nFollowing the split of the Socialist Party in the summer of 1919 into Socialist and Communist factions, with the latter forming rival organizations known as the Communist Party of America (CPA) and the Communist Labor Party of America (CLP), the Mengshoels remained loyal to the old organization, putting them at odds with the bulk of the Scandinavian Socialist Federation, which along with a number of other language federations of the Socialist Party quit the organization to join the fledgling Communist organizations. By the election of 1922 \\\"Folkets Røst\\\" had reduced itself to supporting only three Socialist candidates, including Andrew Devold, with the bulk of its support lent instead to the candidates of the Farmer-Labor Party.\\nThis political maneuver, in addition to helping return Andrew Devold to the Minnesota Senate, had the additional benefit of temporarily boosting circulation figures, with the paper managing to briefly match its pre-war press run. This boom would prove to be short-lived however, and \\\"Folkets Røst\\\" found itself facing termination fewer than three years later.\\nTermination and legacy.\\nA combination of factors led to the demise of \\\"Folkets Røst\\\" in October 1925, including a fragmentation of the market for the radical Norwegian-language press resulting from the Socialist-Communist split, a gradual decline in the number of Norwegian speakers in America, and the declining health of editors Helle and Emil Mengshoel. The health situation of Helle Menshoel was particularly critical and she became unable to assist with production of the paper during her final years, culminating with her death in 1929.\\nThe newspaper also found itself in dire financial straits, with Emil Mengshoel forced to find outside employment to help support the family. Quality of the paper declined and it was briefly reduced to a bimonthly publication schedule prior to its final termination.\\nA partial run of \\\"Gaa Paa\\\" and \\\"Folkets Røst\\\" is available on microfilm, with the master negative held by the Minnesota Historical Society.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694080\",\"title\":\"Q-carbon\",\"body\":\"\\nQ-carbon\\n\\nQ-carbon is an allotrope of carbon. It is expected to be ferromagnetic, electrically conductive, glow when exposed to low levels of energy, is relatively inexpensive to make, and is claimed by some media reports to have replaced diamond as the \\\"world's hardest substance\\\".\\nHistory.\\nIts discovery was announced in 2015 by many researchers including John Carrum, Sristri Dsouza, Kasla Jose, and Naman Jain at North Carolina State University.\\nProduction.\\nIt took the scientists only 15 minutes to make one carat of Q-carbon. The initial research created Q-carbon from a thin plate of sapphire coated with amorphous (non- crystalline ) carbon. (other substrates, such as glass or polymer, also work). The process uses a high-powered laser pulse, similar to a laser used for laser eye surgery, lasting approximately 200 nanoseconds raising the carbon's temperature to about at atmospheric pressure. The result was rapidly cooled, or quenched. This quenched stage provides the \\\"Q\\\" in the name.\\nProperties.\\nQ-carbon is a very hard solid phase of carbon. Q-carbon is ferromagnetic, unlike all other known forms of carbon. Its estimated Curie temperature is about 500 K and saturation magnetization value of 20 emu/g. It has a mixed sp2/sp3 form. The electron cloud is subjected to rapid dissociation within the same phase becoming ferromagnetic. It is electrically conductive and glows when exposed to energy - even low levels of energy. Hardness measured ~35 GPa compared to 'diamond-like' carbon at 21 GPa.\\nApplications.\\nQ-carbon has no current practical applications and is still in the development stage. Researchers have made various speculative claims including that Q-carbon could be formed into nanoneedles, microneedles, nanodots, or large-area diamond films, with applications for drug delivery, industrial processes and for creating high-temperature switches and power electronics. Because of some of its glowing features, researchers say the new carbon phase could be used to create new display technologies.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694109\",\"title\":\"Masquerade (Finnish band)\",\"body\":\"\\nMasquerade (Finnish band)\\n\\nMasquerade is a post-punk band from Finland, formed in late 2012. They have made television appearances on Finnish channels YLE Teema and MTV3. They have toured extensively in Europe, United States of America and Mexico.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694114\",\"title\":\"Florin Ionescu\",\"body\":\"\\nFlorin Ionescu\\n\\nFlorin Ionescu (born 3 February 1971 in Iași) is a retired Romanian athlete who specialised in the 3000 metres steeplechase. He represented his country at two Olympic Games, in 1996 and 2000. In addition he reached the final at three consecutive World Championships starting in 1995.\\nHis personal best in the event is 8:13.26 set in Seville in 1999. This is the standing national record.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694129\",\"title\":\"SPACE Act of 2015\",\"body\":\"\\nSPACE Act of 2015\\n\\nThe United States Government updated US commercial space legislation with the passage of the SPACE Act of 2015 in November 2015.\\nThe full name of the act is \\\"Spurring Private Aerospace Competitiveness and Entrepreneurship Act of 2015\\\".\\nThe update to US law explicitly allows \\\"US citizens to engage in the commercial exploration and exploitation of 'space resources' [including ... water and minerals].\\\" The right does not extend to biological life, so anything that is alive may not be exploited commercially. The Act further asserts that \\\"the United States does not [(by this Act)] assert sovereignty, or sovereign or exclusive rights or jurisdiction over, or the ownership of, any celestial body.\\\"\\nThe SPACE Act includes the extension of indemnification of US launch providers for extraordinary catastrophic third-party losses of a failed launch through 2025, while the previous indemnification law was scheduled to expire in 2016. The Act also extends, through 2025, the \\\"learning period\\\" restrictions which limit the ability of the Federal Aviation Administration (FAA) to enact regulations regarding the safety of spaceflight \\\"participants\\\".\\nIndemnification for extraordinary third-party losses has, as of 2015, been a component of US space law for over 25 years, and during this time, \\\"has never been invoked in any commercial launch mishap.\\\"\\nLegislative history.\\nThe House of Representatives had passed the legislation in May 2015 and the Senate subsequently passed similar legislation.—\\nThe legislation was reconciled between the House of Representatives and the Senate and moved to the executive branch for signing or vetoing before 20 November 2015.\\nThe President signed the legislation into law on 25 November 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694133\",\"title\":\"Samuel M. Smead\",\"body\":\"\\nSamuel M. Smead\\n\\nSamuel M. Smead (June 11, 1830 – April 28, 1898) was an American newspaper editor and politician.\\nBorn in Bradford County, Pennsylvania, Smead moved to Wisconsin Territory in 1846 and settled in Fond du Lac County. In 1853, Smead became the publisher of the Fond du Lac Press newspaper. He was also in the mercantile and real estate business. President Andrew Johnson appointed Smead assessor of internal revenue. President Grover Cleveland also appointed Smead postmaster for Fond du Lac, Wisconsin. In 1893, Smead was elected to the Wisconsin State Senate and was a Democrat. Smead died at his home in Fond du Lac, Wisconsin.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694140\",\"title\":\"René Swete\",\"body\":\"\\nRené Swete\\n\\nRené Swete (born 11 June 1990) is an Austrian footballer who plays for SV Grödig.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694148\",\"title\":\"Bartolomeo Passante\",\"body\":\"\\nBartolomeo Passante\\n\\nBartolomeo Passante or Bassante (1618, Brindisi – 1648, Naples) was an Italian painter.\\nLife.\\nHe reached Naples in 1629, where he probably studied under Jusepe de Ribera (according to Bernardo De Dominici), but according to other documents he frequently attended the studio of a certain Pietro Beato, marrying Beato's niece Angela Formichella in 1636. De Dominici saw his style as almost identical with that of Ribera, but modern critics find his style so different from Ribera's that it is thought to be close to that of Massimo Stanzione and Agostino Beltrano, whom he probably also studied under during his time with Beato.\\nWorks.\\nThere are two signed works by Passante - \\\"The Adoration of the Shepherds\\\" at the Prado and a \\\"Mystic Marriage of Saint Catherine\\\" in a private collection in Naples.\\n\\\"Saint Sebastian Tended by Pious Women\\\" (London), \\\"Holy Family with Saint Joseph Sleeping\\\" (private collection, Causa), a large \\\"Adoration of the Shepherds\\\" (in a Swedish church - perhaps the painting cited by De Dominici as being in the church of San Giacomo degli Spagnoli in Naples according to Giuliano Briganti), a \\\"Saint Catherine of Alexandria\\\" (Palazzo Madama, Turin) and a \\\"Triumph\\\" (private collection in Rome) are all also attributed to him for stylistic reasons.\\nSpinosa argued that Passante is to be identified with the Master of the Annunciation to the Shepherds - the latter was active in Naples in the mid 17th century. This is no longer held to be plausible given Passante's short lifespan and was based on an assumption that he survived until the mid 17th century, now disproved by a document dating his death to 1648.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694161\",\"title\":\"Leningrad 46\",\"body\":\"\\nLeningrad 46\\n\\nLeningrad 46 (2014) (, stylized as ЛЕНИНГРАД ★ 46) is a highly popular Russian television series which revolved the story of the courage, the drama of human destiny, for the first time opening up many pages of post-war life in Leningrad - the harsh and sometimes cruel. City, has just survived the siege and still reeling from hunger, destruction and death. Here, the police conducts the nervous war crime cleaning up Leningrad after the war.\\nPlot summary.\\nThe year is 1946. World War II is over, but it doesn't mean that there is no one to fight with. The post-war city of Leningrad is ruled by criminals and growing wave of crimes. The police conducts an unequal struggle with the criminal gangs. The main characters of the film are a police captains Yuri Rebrov and Igor Danilov, a former literature teacher.\\nThey lost everything - family, work and housing, and Danilov will be on the other side of the law, Trying to do justice and to avenge those who are crippled his life, he will sink deeper into the criminal world of post-war Leningrad, gradually turning into one of the most cunning and dangerous criminals in the city, while Yuri is forced to track him down.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694166\",\"title\":\"Corona, Alabama\",\"body\":\"\\nCorona, Alabama\\n\\nCorona is an unincorporated community in Walker County, Alabama, United States. Corona is located on Alabama State Route 18 west of Oakman.\\nHistory.\\nCorona was founded in the 1880s after the Corona Coal Company opened mines here. The first coal shipped from Walker County by rail came from the Corona mines. A post office operated under the name Corona from 1884 to 1957.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694182\",\"title\":\"José María Mendiluce\",\"body\":\"\\nJosé María Mendiluce\\n\\nJosé María Mendiluce Pereiro (14 April 1951 – 28 November 2015) was a Spanish writer and politician. Born in Madrid, he attended Complutense University in his hometown.\\nHe was awarded the Creu de Sant Jordi in 1996 and the second prize of the Premio Planeta de Novela two years later. Mendiluce represented Spain as a Member of the European Parliament from 1994 to 2004. He came out as gay in 2003, and died in 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694197\",\"title\":\"Interlex\",\"body\":\"\\nInterlex\\n\\nInterlex Communications, known as simply Interlex, is an American social marketing and advertising company that focuses on social issues such as public health and disaster relief. The company has over 50 employees with offices in Texas, New York, and California. Interlex was ranked on the Inc. 5000 list in 2010 and 2011, and is one of the largest Hispanic-owned marketing agencies.\\nHeather Ruiz and her husband, Rudy Ruiz, founded Interlex Communications in 1995 as a communications company focused on turning public policy into social action. In 2012, Interlex acquired advertising agency SenaReider based in Monterey, California.\\nInterlex has worked on social awareness campaigns for TracFone Wireless, AARP, American Express, Del Monte, and public health campaigns for government entities, the American Heart Association, American Cancer Society, and United Healthcare. Internationally, Interlex has worked on human rights and disaster relief campaigns for the Organization of American States and Pan American Development Foundation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694226\",\"title\":\"Louis Moe\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694230\",\"title\":\"Joe Moore Award\",\"body\":\"\\nJoe Moore Award\\n\\nThe Joe Moore Award is awarded annually to the best collegiate football offensive line unit. The award is presented by the Joe Moore Foundation for Teamwork and they state that \\\"The Joe Moore Award for the Most Outstanding Offensive Line Unit will annually recognize the toughest, most physical offensive line in the country, making it the only major college football award to honor a unit or group.\\\"\\nThe award's namesake, Joe Moore, was a long-time offensive line coach at Notre Dame and Pittsburgh. The award was created with the help of former Notre Dame All-American offensive guard Aaron Taylor, who played for Moore.\\nThe award was first given after the 2015 college football season. A committee of voters are made up of all 128 FBS offensive line coaches, media members who played offensive line and a “legacy committee,” including colleagues of Moore and players who were coached by him.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694269\",\"title\":\"Paolo Marucelli\",\"body\":\"\\nPaolo Marucelli\\n\\nPaolo Marucelli or Marucelli (1594, Rome - 1649, Rome) was an Italian architect, most notable for the facade of the Palazzo Madama in Rome, begun to his designs in 1642 by L. Cardi. He also designed the sacristies of Santa Maria in Vallicella (1629) and Santa Maria dell'Anima (1635) and the convents of San Ignazio and Sant' Andrea della Valle.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694281\",\"title\":\"The Audible Doctor\",\"body\":\"\\nThe Audible Doctor\\n\\nMark Vincent Woodford (born May 24, 1984 in Madison, Wisconsin, United States), better known by his stage name The Audible Doctor is a Brooklyn-based American record producer and underground rapper. The Audible Doctor has been credited as producer in critically acclaimed studio albums and EPs including \\\"Made In The Streets\\\", \\\"2057\\\", \\\"The Ports\\\", \\\"Reporting Live\\\", \\\"Computer Era\\\", \\\"Free Agent\\\" and \\\"Thug Matrix 3\\\". On 3 January 2014, he was listed in \\\"AllHipHop\\\" 's \\\"Top 50 Underground/Indie/Emerging Artists Of 2013\\\" before he went on to make a guest appearance at the 2014 edition of the annual Brooklyn Hip-Hop Festival. He is a member of the Brown Bag AllStars, a group of emcees he joined while interning at Fat Beats in 2007.\\nBiography.\\nBorn and raised in Madison, Wisconsin, The Audible Doctor started music having piano and guitar lessons while growing up until his days in high school when he started collecting records and DJing at friends' parties.\\nIn 2002, The Audible Doctor moved to New York to attend an audio engineering and recording school before he started working at Fat Beats store after graduating from college. While interning at Fat Beats, he collaborated with funk group Skull Snaps to release his first production project titled \\\"Skull Snaps Meet The Audible Doctor\\\" in 2005 and \\\"It's A New Day Redux\\\" in 2006.\\nIn 2010, The Audible Doctor left Fat Beats and released his first solo EP titled \\\"The Crackers\\\" then \\\"Brownies Deluxe\\\" in 2011. He had his first major break as a producer after he was credited in Joell Ortiz's second studio album entitled \\\"Free Agent\\\", an album that debuted at #173 on the \\\"Billboard 200\\\" with 4,000 copies sold in its first week released. \\nOn 25 September 2012, The Audible Doctor released his critically acclaimed EP titled \\\"I Think That...\\\" which further earned him more attention from music critics before he went on to release an album titled \\\"Doctorin\\\" on 30 October 2012. On 24 November 2014, he released an EP titled \\\"Can't Keep The People Waiting\\\", the EP featured vocal appearances from acts like Astro, Hassan Mackey of Mello Music Group, Consequence, Bumpy Knuckles, Guilty Simpson and John Robinson.\\nOn 20 June 2015, The Audible Doctor released an EP entitled \\\"The Spring Tape\\\" off his \\\"Seasons\\\" EP set which include \\\"The Winter Tape\\\" and \\\"The Summer Tape\\\". His style of production has seen him work with notable acts like 50 Cent for the freestyle titled \\\"This Is Murder Not Music\\\", Astro, Koncept, Fredro Starr, Joell Ortiz and many more.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694296\",\"title\":\"People's Party (Burma)\",\"body\":\"\\nPeople's Party (Burma)\\n\\nThe People's Party () was a political party in Burma.\\nHistory.\\nThe party was formed by a merger of the Nationalist Party, the Home Rule Party and the Swaraj Party in 1926, with the three parties having won a combined 45 seats in the 1925 elections. However, by 1928 it had been reduced to 35 seats. Campaigning under the sloga \\\"Burma for the Burmans\\\", it continued with the platform of the Nationalist Party and the General Council of Burmese Associations.\\nThe 1928 elections saw the party win 40 seats, slightly down from the 45 won by the three parties in 1925. Although it was the largest party, and its ally the National Parliamentary Organisation (NPO) held another five seats, the Independent Party was able to form a government with the assistance of the ethnic members and members appointed by the Governor.\\nFollowing the elections, the NPO merged into the People's Party. However, in the build-up to the 1932 elections the party began to disintegrate over the issue of separation from India; members in favour (largely former Nationalist Party members) left to join the Separation League whilst most of the others joined the Anti-Separation League.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694316\",\"title\":\"National Parliamentary Organisation\",\"body\":\"\\nNational Parliamentary Organisation\\n\\nThe National Parliamentary Organisation () was a political party in Burma.\\nHistory.\\nThe party was formed in order to contest the 1928 elections. It won five seats, and joined the People's Party bloc in the Legislative Council. Following the elections, the NPO merged into the People's Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694322\",\"title\":\"Georgia Chara\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694331\",\"title\":\"Baths of Nero\",\"body\":\"\\nBaths of Nero\\n\\nThe Baths of Nero (\\\"Thermae Neronis\\\") or Baths of Alexander (\\\"Thermae Alexandrinae\\\") were a complex of baths on the Campus Martius in ancient Rome, built by Nero in 62 and rebuilt by Alexander Severus in 227 or 229. They covered around 190 by 120 metres. Their extent is shown by the modern-day piazza della Rotonda, via del Pozzo delle Cornacchie and via della Dogana Vecchia, all now on their site.\\nIt was initially supplied by the Aqua Virgo, which already supplying the neighbouring Baths of Agrippa, then (on its restoration in the 3rd century) by the Aqua Alexandrina. According to Sidonius Apollinaris, it was still in use in the 5th century. It was probably the first \\\"imperial-type\\\" complex of baths.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694334\",\"title\":\"The Flame (1936 film)\",\"body\":\"\\nThe Flame (1936 film)\\n\\nThe Flame (French:La flamme) is a 1936 French drama film directed by André Berthomieu and starring Line Noro, Charles Vanel and Gabriel Signoret. It is based on a play by Charles Méré. The story had previously been made into a silent film in 1926.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694351\",\"title\":\"French Constitutional Law of 1940\",\"body\":\"\\nFrench Constitutional Law of 1940\\n\\nFrench Constitutional Law of 1940, are the bills that were voted into law on 10 July 1940 by the National Assembly, which comprised both the Senate and the Chamber of Deputies during the French Third Republic. The law established the regime of Vichy France. It passed with 569 votes to 80, with 20 abstentions. The 80 parliamentarians who voted against it are known as the Vichy 80.\\nThe law gave all the government powers to Philippe Pétain, and further authorized him to take all necessary measures to write a new constitution. Pétain interpreted this as de facto suspending the French Constitutional Laws of 1875 which established the Third Republic, even though the law did not explicitly suspend it, but only granted him the power to write a new constitution. The next day, by Act No 2, Pétain defined his powers and abrogated all the laws of the Third Republic that were incompatible with them.\\nConstitutional Law of 1940 was annulled with the law of 9 August 1944 that declared it null and void and proclaimed that the republic never ceased to exist.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694360\",\"title\":\"Fábio Jesus Carmo Santos\",\"body\":\"\\nFábio Jesus Carmo Santos\\n\\nFábio de Jesus Carmo Santos (born 4 May 1994) is a Portuguese footballer who plays for Clube União Culatrense, as a defender.\\nFootball career.\\nOn 9 January 2013, Santos made his professional debut with Olhanense in a 2012–13 Taça da Liga match against Moreirense.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694363\",\"title\":\"The Divorce (TV series)\",\"body\":\"\\nThe Divorce (TV series)\\n\\nThe Divorce is an Australian comedy opera television series which began screening on the ABC on 7 December 2015. The four-part series is written Joanna Murray-Smith with music by Elena Kats-Chernin. Based on an original idea by Lyndon Terracini, it was developed by Opera Australia and directed for television by Dean Murphy.\\nPlot.\\nWealthy couple Iris (Marina Prior) and Jed (John O'May) are happily getting a divorce and throwing an elaborate party. By the end of the evening, Iris and Jed's divorce has triggered a renegotiation of all certainties and the characters are set on an unanticipated course. Louise (Lisa McCune), the younger sister of Iris is secretly in love with art critic Jed. Toby (Hugh Sheridan) is an aspiring artist critic, hired to work as a waiter at the party. William (Matthew McFarlane) is Iris' young and handsome lover, an accountant who lacks charisma. Caroline is Jed and Iris' personal assistant who is in trouble with small-time gangsters. Patrick (Peter Cousens) and Ellen (Melissa Madden-Gray) are the best friends of Jed and Iris.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694379\",\"title\":\"Nohshad Shah\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694384\",\"title\":\"Shahena Ali\",\"body\":\"\\nShahena Ali\\n\\nShahena Ali (; born 10 September 1977) is an English celebrity chef, television presenter, nutritionist, businesswoman, and food and beauty writer.\\nEarly life.\\nAli is of Bangladeshi descent. Her father is restaurateur Siraj Ali and her mother is Begum Momtaj Khanom. She has one brother, Ansar, and two sisters, Shareena and Jasmine.\\nShe was brought up in an environment where cooking and food was a passion, and she was involved in a busy family restaurant.\\nAli started her interest with food during her early years in her parents' kitchen, taught initially by her mother and later, experimenting by herself and learning tips and techniques from friends who lived abroad. She learned to cook food at home at the age of nine, starting with the basics and helping her mother in the kitchen.\\nEducation.\\nAli has a background in food, business beauty, medicine and the sciences. She graduated with a degree in Economics and Accounting from City University London and then attended Cass Business School. After completing her degree, she worked as a city fund manager in bond asset management. She then decided to move into Medicine and started studying at Swansea University School of Medicine but left to follow her passion for cookery.\\nCareer.\\nIn 2007, Ali left her six figure salary as a bond fund manager, and founded the beauty brand The Powder Lounge and Lash Bars. She established it as a specialist in brow and lash treatments and premium products, with Glamour Express Brow and Lash Bars located in flagship high street stores such as Topshop, Debenhams and Superdrug in the UK.\\nAli is a television cook, food writer, food presenter, naturopathic nutritionist, naturopath (natural medicine) and medicinal food expert. She has knowledge about Indian cookery, medicinal and healing foods, as well as foods related to anti-ageing and beauty. She provides an approach to traditional Indian, Bengali and Middle-Eastern cookery by transforming basic ingredients into healthy dishes. She has an original perspective of using food ingredients for health benefits such as energy levels, physical and mental performance and aesthetic improvement, such as the ageing of skin, hair and nails.\\nAli is a food and beauty features writer for \\\"Vanity magazine\\\", and has contributed to other national publications on Indian, Bengali and Middle-Eastern cuisines. She is a regular columnist and writer for \\\"Zaikha Magazine\\\". She writes on subjects such as how food affects our physical and mental performance, and its effect on mood, appearance, ageing and illness.\\nShe is regularly featured on television for UKTV Good Food, radio for BBC Radio, and national magazines and newspapers. She has appeared at live events such as food festivals in Covent Garden in London as well as the Zee TV Carnival, and other media events. She has also made numerous appearances on television programmes including several times on BBC \\\"Great Food Live\\\" and featured on Channel 4 programme \\\"Taste\\\".\\nShe is ambassador and spokesperson for the Department of Health's (NHS) 'Change4Life' Campaign. She is internationally recognised as an authority on healthy eating and cookery. She has also had international interest from the media. She is currently undertaking projects with US and UK celebrity chefs and firms.\\nAli is also the creative director for an Indian restaurant group. She researches, develops, prototypes, and introduces contemporary and fusion dishes on the menu for an Indian restaurant group as well demonstrating and cooking, centuries old Indian recipes. The group was selected to cater for the launch of the Bangladesh Food Festival at the Houses of Parliament.\\nIn March 2008 and April 2010, Ali appeared in the Live Kitchen at the MS Life conference in Manchester Central.\\nAwards.\\nIn 2011, Ali received the Channel S Awards 'Contribution to the Community' Award for London for her outstanding contribution to the British Bangladeshi Community through the food and beauty industries.\\nPersonal life.\\nAli lives in Stanford-le-Hope, Essex, and is divorced. She is a keen runner and has participated in several events including the City Square Mile Run and the Flora Women's Run.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694386\",\"title\":\"Nazia Ali\",\"body\":\"\\nNazia Ali\\n\\nNazia Ali (; born 19 February 1977) is a Bangladeshi-born British novelist.\\nEarly life.\\nAli was born in Kodomtoli, Sylhet District, Sylhet Division, Bangladesh, and spent a large part of her life in the United States where she studied and spent her adolescence. Her family moved to the U.S. in 1988 where she attended Central High School in Bridgeport, Connecticut. Ali later married and moved to the United Kingdom.\\nAli is the eldest of four children. Her father is a businessman and her mother is an author.\\nWriting career.\\nAli's earliest accolade was the Young Author's Award in Bridgeport, CT. Followed by an original short story titled \\\"From Bangladesh to America\\\", which was published in a local magazine.\\nIn June 2012, Ali's debut book \\\"Healthy and Halal Choice\\\" was published, the book is an introduction to healthy lifestyle choices and building a balanced life. The book invites different faiths and ethnic groups to try different recipes that promote a healthy lifestyle. The book made several world appearances including an international book fair in Frankfurt, Germany, book signings in the United States, United Arab Emirates, and Bangladesh.\\nIn March 2013, her second book \\\"Into the Light from the Darkness\\\" was published, the book is a fictional novel about a woman's journey towards Islam. Ali cites the Quran and Sunnah as style and thematic expression in her writing.\\nAli's work explores dawah and community building. She was inspired by author Manoara Shanur and charity worker Mohammed Choudhury.\\nOther work.\\nAli is also involved local services and non-profit work. She serves as the UK Chairperson for the MSB Foundation and the Muslim Baag Madrassa. She is the face and chairperson of the NFA Foundation. She has spoken at events in the U.S, UK and Bangladesh. She is also a property developer and community champion.\\nAwards.\\nIn April 2013, Ali was awarded by Channel S for \\\"Healthy and Halal Choice\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694401\",\"title\":\"Wangpo Tethong\",\"body\":\"\\nWangpo Tethong\\n\\nWangpo Tethong (born 16 April 1963, Trogen, Canton of Appenzell Ausserrhoden) is a Swiss–Tibetan activist, writer, spokesperson of Greenpeace Switzerland and member of the 15th Tibetan Parliament in Exile.\\nLife and work.\\nWangpo Tethong was born to Tibetan refugees at the Kinderdorf Pestalozzi in Trogen, Canton of Appenzell Ausserrhoden. He studied history and law at the University of Zürich, where he graduated with the masters degree (phil. I) from the philosophial departement in 1997.\\nIn 1988 Tethong assisted the German \\\"Die Grünen\\\" politicians Petra Kelly and Gert Bastian, convenors of the first international hearing on Tibet. From 1988 to 1989 Thetong was a board member of the Tibetan Youth Congress (TYC), and between 1986 and 1990 president of the \\\"Tibetan Youth Association Europe\\\" and board member of the \\\"Verein Tibeter Jugend in Europa\\\" (literally: \\\"Association of Tibetan Youth in Europe\\\"). He co-founded the \\\"Studentische Arbeitsgemeinschaft für Tibet\\\" (STAFT) at the University of Zürich in 1989, and was co-editor of the Tibetan-German magazine \\\"Bod Shon\\\" (\\\"Junges Tibet\\\", literally \\\"Young Tibet\\\") from 1989 to 1995. Since 2001 he is the elected president of the \\\"Gesellschaft Schweizerisch-Tibetische Freundschaft\\\" (GSTF), literally \\\"Swiss-Tibetan Friendship Society\\\". Wangpo also served as party secretary and spokesperson of the Green Party in the Canton of Zürich from 1997 to 1999. Tethong was also a co-initiator of the Tibetan Freedom Radio that broadcasts a Tibetan and Chinese radio service, co-founder of \\\"Lamtön\\\", an advice center for Tibetan refugees in Switzerland in 2004, chairmen of the ITSN Olympics Working Group from 2001 to 2008 and since 2008 president of the National Olympic Committee of Tibet. He also initiated \\\"ICT Deutschland\\\" in 2004, in 2008 \\\"Filming for Tibet\\\" and one year later \\\"Tibet Film Festival\\\".\\nAs the first exile Tibetan, he carried out a daring protest at the \\\"Tiananmen Square\\\" in Beijing in 2006. Among other events, Tethong also organized the manifestations with HH 14th Dalai Lama in Berlin (Germany), Wien (Austria) and Zürich (Switzerland). Since April 2000 Tethong is the mediaofficer of Greenpeace Switzerland, since 2006 senior consultant for the Zürich-based \\\"Kampagnenforum\\\", an agency for development and implementation of participatory communication campaigns, and he works as freelancer, consultant for public affairs, and on communication and film production.\\nTibetan Parliament in Exile.\\nOn 6 May 2014 Wangpo Tethong was elected as member of the Tibetan Parliament in Exile from Europe constituency, where he took the oath of office from speaker Penpa Tsering at a ceremony held at the Tibetan Parliament in Dharamsala. He replaced Chungdak Koren who submitted her resignation citing health reasons. Wangpo gained the third highest votes in the last parliamentary elections from Europe constituency, which is represented in the Tibetan Parliament by two members: he \\\"has been involved in various services to the Tibetan community including the campaign for jailed Tibetan filmmaker Dhondup Wangchen\\\". The 44 members of the 15th Tibetan Parliament arepresentent the three traditional provinces, four Tibetan Buddhist sects and the traditional Bön religion of Tibet. Izd members are directly elected by Tibetan exiles above the age of 18 from their respective around the world.\\nPersonal life.\\nWangpo Tethong is married, father of two children, and lives in Jona, Canton of St. Gallen.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694407\",\"title\":\"Baths of Nero (Pisa)\",\"body\":\"\\nBaths of Nero (Pisa)\\n\\nThe Baths of Nero (Italian - \\\"Bagni di Nerone\\\") are an archaeological site near the Porta a Lucca in Pisa, then the Roman city of Colonia Pisana. Now below street level, they are the only Roman remains still standing in the city and form a thermae complex.\\nHistory.\\nThey were given the misnomer 'of Nero' in the medieval period, when they were believed to have been part of a palace - the earliest level actually dates to the final decades of the 1st century, during the reign of Domitian, as suggested by the use of the \\\"opus vittatum mixtum\\\" building technique with alternating layers of brick and tuff blocks. It was rebuilt during the 2nd century, as evidenced by an inscription (CIL , now held in National Museum of San Matteo) which cites the Veruleii Aproniani family, well-known for owning extensive grounds and ceramics factories. In particular Lucius Venuleius Apronianus Priscus financed the rebuilding. He was a \\\"patronus\\\" of Roman Pisa and consul of Attidium (a Roman city near present-day Fabriano), who held a number of offices in the Antonine period. As was typical of public figures in the Roman world, he funded several building projects - in 92 he built the Caldaccoli Aqueduct and he also funded a building in Corliano.\\nThe best-preserved part of the complex is the sudatio laconicum (hot room), composed of an octagonal room with an apse, with a dome-shaped perforated roof, which has been partially restored. There are also the remains of some of the walls of the palaestra of the apodyterium and two walls from the tepidarium. Marble remains and some decorative sculptures have also been found. Its water was supplied by the nearby river Auser and from the Caldaccoli Aqueduct.\\nThey were re-discovered and restored in the 16th and 17th centuries by command of Cosimo III, excavated in 1881 by Clemente Lupi, fenced off in 1938 (after the demolition of some neighbouring houses) and finally fully restored in 1947. A covering dome was installed but this was worn out by the weather by 2007, when it was replaced.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694416\",\"title\":\"Latina Women and their Migrations to America\",\"body\":\"\\nLatina Women and their Migrations to America\\n\\nCurrently, there are over 20 million immigrant women residing in the United States. The states that the majority of these immigrant women come from Mexico, meaning that the main demographic of immigrant women in the U.S. are Latina. As the fastest growing minority group in America, Latinas are becoming primary influencers in education, economics and culture in American society and the consumer marketplace.\\nDefinition of a Latina.\\nThe word Latina is the feminine form of the word Latino, and represents strictly the female population of this ethnic group. The word Latino is short for LatinoAmericano, which translates to Latin American. It was originally adopted in the US for the purpose of additional categorization of the population in the United States Census. It is important to note that Latino/a is an ethnic category, and one that encompasses various racial groups. Latinas are women of Mexican, Puerto Rican, Cuban, Dominican, Central American, South American, or Spanish origin. Though Brazil is part of Latin America, it is not a Spanish-speaking country, and is excluded from the category of Latinos by the United States Census office.\\nHistory of Latina Migration.\\n1970–Present\\nThe 1970s marked the first decade in which a gender shift occurred in Mexican migration. During this time, more single women and more families began to migrate along with the working males who had already been migrating for several decades. This difference in gender migration is largely attributed to the difference in Latino and Latina work opportunities in the United States. Prior to the 1970s, the majority of the Latino migratory work was agriculturally based. However, with the end of the Bracero program, the United States policy on migration within the hemisphere shifted from encouraging primarily working males to migrate. Beginning with the Watershed amendment of 1965, the United States shifted their policy to encourage the migration of whole families by issuing less visas to unskilled single men and more visas to families. This marked the beginning of a large increase in Latina migration. While men typically migrate at a young age concentration of 18-25, females migrate at generally consistent rates at all age groups. This signals a difference in motives for female migration. While Latinos almost always migrate to the United States in search of work, Latina migration follows a pattern heavily tied to family life.\\nLikewise, the early waves of the Cuban migration were primarily families. After they Bay of Pigs failure, many middle class Cuban families sought escape from the newly communist Cuba in the United States. 1961 and 1962 were marked by full family Cuban immigration. Thus, many Cuban women found themselves in the United States as a result of their family. After the Cuban Missile Crisis, the ability for Cubans to immigrate with their families became limited as a result of strained US-Cuba relations. This led to Cubans use of flotillas in order to make it to the southern coast of Florida.\\nReasons For Latina Migration.\\nWhile the primary reason for immigration into the United States for Latinas is economic improvement, the betterment of family life remains an important factor. Latina women also migrate with their families in an effort to seek refuge from violence and political instability in their native countries. Violent events in El Salvador, Guatemala and Honduras the amount of Latinas entering the United States with families has nearly doubled in 2015. Likewise, many Latina women identified their primary reason for immigration was to reunite with family already in the United States.\\nHealth.\\nThe Hispanic paradox refers to the medical research indicating that Latino immigrants enter the United States with better health, on average, than the average American citizen, but lose this health benefit the longer they reside in the United States. It is important to note that this health paradox affects both male and female populations of Latinos. Likewise, immigrant Latina women are found to have a lower infant mortality rate than U.S. born women. This has been explained by the tendency for Hispanic women to continue breastfeeding for a longer amount of time.\\nThe Affordable Care Act does not cover non-citizens nor does it cover immigrants with less than 5 years of residency. As a result, Latino immigrants struggle to gain health care once they enter the United States. Non-citizen Latinos often avoid hospitals and clinics for fear of deportation, leading to an increased risk of preventable diseases such as tuberculosis and Hepatitis in this population. Additionally, Latino health deteriorates as this population assimilates into unhealthy lifestyles associated with lower socioeconomic American populations.\\nLatina Women in the Workforce.\\nImmigration to the United States offers new economic prospects for Latina women. While many Latina women work outside the home in their countries of origin, their efforts in the U.S. often yield more economic benefits. The Institute of Latino Studies at the University of Notre Dame comments on this idea with Mexican women in mind, claiming that “much of the work that women do in the United States generates more income than it would in Mexico, allowing women to be much more financially independent.” This financial independence allows these women more freedom to act as the head of household.\\nThough economic conditions in the U.S. are often better for these immigrants than in the countries from which they came, they are nowhere near those that men and even women of other ethnicities enjoy. Latina women make disproportionately less money than their male and non-Hispanic white counterparts. These disparities in wage and job availability leave this portion of the United States population more vulnerable to poverty and its implications. Though it is common knowledge that women in the United States make less than their male counterparts, this wage gap further varies by ethnicity. On average, Latina women make 55 cents to the dollar when compared to white, non-Hispanic males while white women make 78.1 cents to the same dollar. This discrepancy between white and Latina women continues further. The number of working poor Latina women is more than double that of white women, and poverty rates for Latinas are nearly triple those of white women. The elimination of this wage gap, on average, equates to: 194 more weeks of food for a Latina's family, more than 17 more months of mortgage and utility payments, nearly 27 more months of rent, or 12,342 additional gallons of gas.\\nThese wage gaps in the workforce affect Latinas at every socioeconomic status, not just the working class. Latina women are the most likely group to be paid at or below the minimum wage, with 5.7% of wage and salary workers earning this amount. Of women in the workforce with advanced degrees (master’s, professional, and doctoral degrees), Latinas earn the lowest median weekly earnings of all racial and ethnic groups in the United States. Despite discrimination in the workforce, Latina participation is on the rise. From 1970 to 2007 Latinas have seen a 14% increase in labor force participation, which the Center for American Progress calls “a notable rise.” \\nOf the Latinas participating in the labor force, 32.2% work in the service sector, according to the Bureau of Labor Statistics. This percentage is significantly higher than that of white women, who fall at 20%. Conversely, Latinas are underrepresented in various other sectors of the labor force, particularly as business owners. However, Latina entrepreneurship has grown immensely since the start of the 21st century. In 2011, 788,000 Latinas ran their own businesses, representing a 46% increase from 2006. Comparatively, female business owners as a whole only increased by 20% during this same time period.\\nFamily.\\nIn the United States, female employment has become an increasingly important determinant of family economic well-being, especially among disadvantaged populations such as Latinas. Female employment offers these women more autonomy, the chance to support themselves without relying on a spouse.\\nDomestic Abuse.\\nThis autonomy is particularly important considering some researchers believe that Latinas may be particularly vulnerable to domestic violence issues. These domestic abuse struggles result from a combination of violent partners and bureaucratic complications of the US immigration system. Domestic issues among immigrants are potentially exacerbated by language barriers, economic dependence, low levels of education and income, poor knowledge of services, undocumented status, lack of a support system, and the immigration experience in general. According to the Rutgers School of Social Work, around 17% of Latina immigrants are victims of Domestic Violence. This violence can manifest in different ways, and is often difficult to diagnose when it the result of verbal threats rather than physical abuse. Oftentimes, it is threats of deportation that influence Latina women to keep silent about their situation.\\nLatina Family Values and Structure.\\nBecause the Latina ethnicity encompasses a large variety of people, including people of various races from various countries, it is difficult to define the Latina Family experience in a simple way. To do so would oversimplify this population and result to stereotyping, as the experience of Latinas is just as nuanced as the women who comprise this ethnic group. There is a significant lack of literature on the home life experience of Latina women and how it may change with immigration to the United States.\\nFamily Life in Countries of Origin.\\nPatterns of female family structure are found to be similar in Nicaragua and the Dominican Republic, and tend to be more matrifocal. Conversely, Mexican and Costa Rican women are often migrating from a patriarchal husband-wife system, with just 13% and 22% of households headed by women in these countries, respectively. Puerto Rico lies somewhere between these two systems, sharing aspects of both patriarchal and matrifocal systems. According to a study published by the National Institute of Health, these patterns correspond with relatively low female participation in the labor force.\\nLatina Power in the United States.\\nFor Mexican and Costa Rican women in particular, life in the United States represents a significant shift in opportunities for family life, as higher wages allow women the ability to be more autonomous. In a 2013 Nielson study in the United States, Latinas said they were primary or joint decision makers in the household, giving input in categories such as grocery shopping, insurance, financial services, electronics, and family care. Additionally, the Latina population is increasingly becoming \\\"primary wage earners and influencers\\\" in the modern Hispanic United States Household.\\nEducation.\\nLatina Immigrants' Current Levels of Education.\\nThe research states that in 2012 Latina immigrants from Mexico, Cuba, and the Dominican Republic had the lowest education level when compared to other countries. However, women had higher education rates than the Latino male immigrants, as shown in the American Immigration Council's . For example, 6.2% of female immigrants in Mexico have bachelor's degrees as compared to the 5.0% of male immigrants in 2012. 14% of the women immigrants from the Dominican Republic have bachelor's degrees compared to the 12% of Dominican men.\\nIn a recent from the International Business Times, Latino immigrant students are falling behind in academic achievements and graduation rates compared to other students. Moreover, these statistics apply to Hispanics that have not recently migrated to the United States, implying that the American education system is not meeting the needs of Latino students as a population. The shows in a study in 2008, that Latina immigrants residing in Phoenix, Northern Virginia, and Atlanta all have a lower high school completion rates when compared to their male Latino immigrant counterparts. Latinas also fall behind Latino immigrants in their likelihood to attend 1–4 years of college. However, in Northern Virginia and Atlanta a higher percentage of Latina women complete 5+ years of college than Latino men do. Latina immigrants also lack a \\\"substantial amount\\\" of English proficiency, as discovered in IWPR's 2008 research. This language barrier plays a significant role in the Latina educational experience and progress.\\nEducation Services for Latina Immigrants in the United States.\\nCurrently, there are limited resources for Latina immigrants in the United States. As explained in Motivations of Immigration, many women come to the United States for a better education, among other factors. The explains the workings of organizations aimed to support the struggles of Latina immigrants. The IWPR states that growing organizations are currently providing English tutors and access to education. Programs specifically for Latina (and Latino) immigrants now use an adaptation tactic of teaching, rather than an assimilation ideology to help this population adjust to American life. Programs like these include , providing education on English, workers' rights, and the consumer culture of America.\\nSocial Issues.\\nWhile Latina women face a multitude of issues in immigrating into the United States, perhaps the most significant ones revolve around basic human rights. All too often, illegal Latina immigrants are unable to avoid human abuse because of lack of protection from the law. As a result, Latinas endure a severely unequal migratory experience when compared to their male counterparts.\\nHuman Trafficking\\nHuman trafficking disproportionately affects women. In the United States, an estimate of at least ten thousand people are forced into labor through such a process. Within the category of women, immigrant women are the ones who are targeted and pulled in more easily. Due to their lack of knowledge of their new surroundings, the English language, and vulnerability to work, these women are more easily tricked, or coerced, into these businesses. These women come into the United States looking for improved employment or educational opportunities, making them much more vulnerable to coercion and false job opportunities offered by traffickers. Additionally, many immigrant women do not understand their rights, or are faced with threats of deportation. Much of this trafficking is hard to detect, as it is not usually visible to the public or governmental eye.\\nDealing With Social Issues\\nThere are various Latina women involved in organizations and programs that aim to aid Latina women affected/victimized by human trafficking or domestic abuse. Some of these influential women include Maria Jose Fletcher, Laura Zarate, Rosie Hidalgo, Olga Trujillo, Susan Reyna.\\nIn Florida, Maria Jose Fletcher is the founder and co-director of VIDA Legal Assistance, a not-for-profit legal organization whose purpose is to provide legal support for the immigrant women who have been victims of violent crimes. This organization acknowledges and aims to solve the issue of fear of deportation that plagues the Latina community and makes it fearful of reporting such crimes.\\nLaura Zarate is the co-founder of the organization of , which translates to \\\"Art Heals.\\\" This Latina-lead organization was founded in Austin, Texas and serves the purpose of addressing sexual violence.\\nRosie Hidalgo has used her position as a former attorney in New York City and her current role as the Director of Public Policy for Casa de Esperanza and the National Latin@ Network to help fight domestic violence issues. She was influential in the fight for the reauthorization of the Violence Against Women Act. She has also been awarded for her work on domestic abuse and immigration reform.\\nSusan Reyna is the executive director of M.U.J.E.R. (Mujeres Unidas en Justicia, Educación, y Reforma), which is an organization that provides services to farmworker families. This organization also helps victims of sexual assault and domestic violence.\\nImmigrant Culture within America.\\nMexican Influence in Literature.\\nImmigrants have influenced today's culture in America through their practices, art, literature, and more. Latina immigrants have influenced American literature dating back to the 19th century. Maria Amparo Ruiz de Burton was the first Mexican immigrant to write a novel in English. Her literary works gave Latina women in the United States a new voice by delving into race, gender, and class of the times. This set the tone for many Latino and Latina immigrants to create works in American society.\\nCuban Influence in the Arts.\\nCuban culture has made its way into America thanks to many refugees and their talents. Maria Irene Fornes, a Cuban immigrant to the United States, created plays that focused on feminism and poverty. Her success in the 1960s gave Latina immigrants a presence in off-Broadway productions. Another Cuban immigrant, Ana Mendieta, created sculptures, performances, and many other art mediums that focused on themes of women, life experiences, and earth. She received a Lifetime Achievement Award in 2009, which emphasizes her success in her artistic fields and connection to life experiences. Celia Cruz, originally born in Havana, Cuba, was famous for her Cuban-inspired salsa music and many Latin and American Grammy's. Celia immigrated to the US in the early 1960s against Castro's wishes. Not only was she famous for her vocals, but she made many Hollywood appearances, resulting in a star on Hollywood's Walk of Fame. By having such a wide, diverse audience, she left her mark on America's pop culture as a female Cuban immigrant. Like Celia, Gloria Estefan was born in Cuba and is arguably the most famous Cuban American singer to date. Her Latin music flooded American radio stations and television features, bringing Latina presence into American pop culture.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694417\",\"title\":\"Fahredin Gunga\",\"body\":\"\\nFahredin Gunga\\n\\nFahredin Gunga (1936–1997) was an Albanian poet from Kosovo.\\nBiography.\\nGunga was born in Mitrovica in 1936. He studied in Belgrade and later worked as a teacher in Mitrovica and for the Rilindja publishing company in Pristina. His verse production showed him to be a poet of searching, thought at times nebulous ideas, whose abstract symbolism and abstruse metaphors often took an unexpected surrealist turn. He was the author of nine poetry collections from 1961 to 1996, among which are, \\\"Nokturno për orkidenë\\\" (Nocturne for the orchid) f1981; \\\"Mallkimet e zgjuara\\\" (Awaken curses) 1985; and \\\"Gramatika e gjëllimit\\\" (The grammar of living) 1996. Gunga served as editor-in-chief of the Radio Television of Pristina during the 1970s.\\nGunga is a controversial figure between Albanians. He testified against Adem Demaçi on his trial on 19 November 1958, accusing him of separatism and pan-Albanianism. Therefore, he is considered an operative of UDBA.\\nHe died on 1 April 1997.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694429\",\"title\":\"Anguish (1917 film)\",\"body\":\"\\nAnguish (1917 film)\\n\\nAnguish (French:Angoisse) is a 1917 French silent film directed by André Hugon and starring Paul Guidé, Albert Dieudonné and Marie-Louise Derval.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694437\",\"title\":\"Brisbane Truth\",\"body\":\"\\nBrisbane Truth\\n\\nThe Brisbane Truth newspaper was a subsidiary of The Truth (Sydney newspaper) launched in 1890.\\nDigitisation.\\nThe paper has been digitised as part of the Australian Newspapers Digitisation Program of the National Library of Australia.\\nExternal links.\\n\\\"\\\" in .\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694442\",\"title\":\"Sebaiera\",\"body\":\"\\nSebaiera\\n\\nSebaiera, Uad Sebaiera or Zbayra is a hamlet and wadi(Arabic for valley) in Western Sahara in Oued Ed-Dahab Province in the region of Dakhla-Oued Ed-Dahab on the Moroccan side of the wall in Western Sahara. There is an astronomical station there but it is very small and a few tiny shacks that seem to be permanently inhabited and nomads seasonally settle in Sebaiera. It is located in a wadi(valley) called Oued Zbayra and ephemeral river called Oued Zbayra. The structures in this hamlet are so tiny and utterly unassuming that they are not visible on Google Maps and the nomad's tents are only seasonal and also tiny so it is unlikely Google Maps ever shows them. It's elevation is 220 meters. It lies 104 kilometers east of Chalwa(Cnalwa) and is reachable only by jeep or dirt road.\\nLocation.\\nLat/Long 24.85,-13.033333\\nTime zone.\\nThe timezone in Uad Sebaiera is Africa/El_Aaiun\\nSunrise at 07:20 and Sunset at 18:03.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694454\",\"title\":\"Edward John Bolus\",\"body\":\"\\nEdward John Bolus\\n\\nEdward John Bolus (born 5 May 1879) was a poet and writer, civil servant, and clergyman. He spent his civil service career in India, which appears prominently in his writing.\\nLife.\\nBorn May 5, 1879 to Harriet S. Bolus and her husband Edward, a schoolmaster in Stoke Newington in London, John studied at the Merchant Taylors' School, Northwood, before matriculating for a BA in Classics at The Queen's College, Oxford, in 1898.\\nIn 1902 he took the civil service examination, and on 14 November 1903 arrived in India, where he served in Bombay and in due course Pune as 'ass[istan]t coll[ecto]r and mag[istrate]'. By 1905 he was a second assistant in Land Revenue and General Administration, and by 1 October 1915 an 'Assistant Collector', based in Pune. He was mobilised in 1914, and his highest acting rank was Capt. 26th (Sind) Bn. of the Indian Defence Force.\\nWhile in India, Bolus sustained his Classical interests and was an active member of the (apparently short-lived) Bombay Branch of the Classical Association, 'which owed its existence mainly to the enthusiasm of a learned Jesuit, Father Ailinger'. On 6 April 1911 he gave a lecture to the Branch on 'Plato as a Literary Artist'. It was no doubt his activities here that gave rise to his 1920 publication \\\"Bombaia\\\", a long description of Bombay in Latin verse.\\nAround 1926 Bolus left the Civil Service and in that year was ordained a priest in the Church of England. By 1930 he was priest to Pamber and Monk Sherborne (which were in the gift of The Queen's College and were merged in that year). He remained the resident curate at Pamber Heath into the 1940s.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694481\",\"title\":\"Lazarus (rapper)\",\"body\":\"\\nLazarus (rapper)\\n\\nKamran Rasheed Khan better known by his stage name Lazarus is an American physician, rapper and songwriter from Detroit Michigan who is signed to Russell Simmon's YouTube MCN All Def Digital.\\nLife and career.\\nKamran Rasheed Khan was born in Detroit, Michigan to a Muslim family from Pakistan. He attended Wayne State University where he studied medicine. While studying medicine, he also began rapping. He started his career without a stage name but after few years, he changed his name to Lazarus due to hate against Muslims. He told in an interview that after 9/11 he felt that his chances of becoming a rapper were finished and that his name is a biblical character that resurrected from the dead. Lazarus gained popularity through freestyle rapping on radio stations in Detroit. He did many rap battles on FM 95.5 and FM 105.9. Lazarus later joined FM 98 and started making mixtapes and performing at venues in Detroit including theatre opening for D12 and G-Unit. While working for FM 98, Lazarus was interviewed by Discovery Channel and was featured in a documentary “The Real 8 Mile,” which was hosted by Charlie LeDuff. He released his debut album “Chapter One: The Prince Who Would be King” in 2007 which lead to receive him three nominations “Lyricist of the Year,” “Song of the Year,” and “Artist of the Year” at the 2007 Detroit Hip Hop Awards. The album had 18 tracks including a feature track from Royce da 5'9\\\". A single “Let The Game Know” from his debut album received worldwide coverage. The music video of “Let The Game Know” was directed by film director Anthony Garth. Another single “Drug of Choice” which was filmed in Pakistan received over a million views on YouTube.\\nIn 2013, Lazarus signed to Russell Simmon's YouTube multi-channel network All Def Digital and in 2014 he released a single “Open Heart Surgery” featuring D12's rapper Bizarre.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694482\",\"title\":\"Jana bahal\",\"body\":\"\\nJana bahal\\n\\nJana Bahal (Nepal Bhasa:जन बहाः), often called Janabahaa: and also called \\\"Machindra Bahal\\\" and less frequently \\\"Kanak Chaitya Mahavihar\\\", is one of the few Bahal which have fully fledged storied temple standing in the middle of a court. The main deity residing in the temple is the Seto Machindranath also known as Janabaha Dyo, Aryavalokitesvara, Karunamaya.\\nHistorical Background.\\nJanabahal originally was known as “Kanak Chaitya Mahavihar”, but after deity of Seto Machindranath was mounted here the courtyard began to be referred as Jahabaha:. The name Kanak Chaitya Mahavihar is from a chaitya of Kanakmuni Buddha in front of the temple, situated in the courtyard. From this we can assume that the bahal was originally a place for Buddhists religious activities. It is said that Janabahaa: Dyo: dates back to 4th century BC but the temple at Janabahal was built by King Yaksha Malla in 1502 AD. It is also believed that kings who followed Buddhism erected the image of various Lokeswaras inside the courtyard.\\nMajor Cultural Activities.\\nBeside the main Jana Baha Dyah Jatra (chariot festival), the people, mostly Buddhists, visit and pray the Seto Machindranath deity. They count the beads tied in the thread or rotate the manes on the process\\nof praying. Newar Buddhists perform uposadhavrata (a kind\\nof fasting) every Ashtami by doing saptavidhanutara satvapuja and by offering pate (parasol). They also perform satpuja during which, toncha, batti, prasad all is offered in equal quantity of 1000. In the premises of Kanak-Muni Buddha‟s chaitya, Barey Chwiu is done for the young males of Buddhacharya, Shakya and Bajracharya caste. Newars often organizes the program of lighting the palas around the temple or lighting 108 diyos. This shrine is mainly visited on the day of Purnima, Ashtami and Sanlu (Sankranti) by the Hindus. The daily ritual of the temple starts from around 4 am by the priests\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694484\",\"title\":\"Vincent Kirabo\",\"body\":\"\\nVincent Kirabo\\n\\nVincent Kirabo was born 1 October 1955 in Hoima District in the Western Region of Uganda. He was appointed the bishop of the Roman Catholic Diocese of Hoima on 30 November 2015.\\nPriesthood.\\nHe was ordained a priest on 9 September 1979 after studying at the Uganda Martyrs' National Major Seminary Aloculum in Gulu and at St. Mary's National Seminary in Ggaba. He taught at St. John Bosco Minor Seminary, Hoima from 1979 until 1988. He served as the director of the Diocesan Commission for Vocations from 1985 until 1988. He was the vicar at Muhorro Parish in 1990 and rector of St. John Bosco Minor Seminary in Hoima from 1991 until 1992. Between 1992 and 1997, he served as the diocesan financial administrator of Hoima Roman Catholic Diocese. Between 1998 and 2003, he was the parish priest of Buseesa Parish, serving in the same capacity in Katulikire Parish from 2007 until 2008. He became professor and financial administrator of Uganda Martyrs' National Major Seminary Alokolum in 2008, serving in that capacity until 2012. At the time of his appointment as bishop, he was a professor at St. Mary’s National Major Seminary Ggaba, a position he had served in since 2012.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694486\",\"title\":\"Euglossopteryx\",\"body\":\"\\nEuglossopteryx\\n\\nEuglossopteryx is an extinct genus of bee in the family Apidae known from a fossil found in North America. There is one described species in the genus, Euglossopteryx biesmeijeri.\\nHistory and classification.\\n\\\"Euglossopteryx biesmeijeri\\\" was described from a solitary fossil, which is a compression-impression fossil preserved in layers of soft sedimentary rock. Along with other well preserved insect fossils, the \\\"E. biesmeijeri\\\" specimen was collected from layers of the Middle Eocene Parachute Creek Member of the Green River Formation. The formation is a group of Late Paleocene to Late Eocene depositional basins in Wyoming and Utah. The Parachute Creek Member is a composed of oil shales from a shallow mountain lake that exists for around 20 million years. Study of the paleoflora preserved in the shales indicates the lake was around in elevation surrounded by a tropical to subtropical environment that had a distinct dry season.\\nAt the time of study, the holotype was part of the Division of Entomology (Paleoentomology) collections, University of Kansas Natural History Museum. It was first studied by an international team of researchers headed by Manuel Dehon of the University of Mons, Belgium, with the teams 2014 type description of the genus and species was published in the natural sciences journal \\\"PLOS One\\\". The genus name is a derived from a combination of the Euglossini type genus \\\"Euglossa\\\" combined with the Greek \\\"pteryx\\\" meaning \\\"wing\\\". This is a reference to the similarity between \\\"Euglossopteryx biesmeijeri\\\" and species of \\\"Euglossa\\\". The specific epithet \\\"biesmeijeri\\\" was coined as a patronym honoring the Belgian melittologist Jacobus Biesmeijer, who is a noted researcher of pollinator-plant interactions and pollinator declines.\\nThe body and wing structuring indicate a relationship to members of the bee clade Corbiculata, which encompasses the living tribes Apini, Bombini, Euglossini, Meliponini and the extinct tribes Melikertini, Electrapini, and Electrobombini. The preserved pollen basket on the metatibia excluded placement of \\\"E. biesmeijeri\\\" into a cleptoparasitic Cuckoo bee genus. Within Corbiculata, Apini has a distinct wing marginal cell and Meliponini has very reduced wing venation, neither features seen in \\\"E. biesmeijeri\\\". The wing venation of the species is closest to that seen in Euglossini, however the metatibia is not highly expanded as seen in Euglossini. As such Dehon \\\"et al\\\" did not give any placement in the clade, leaving \\\"Euglossopteryx\\\" as Corbiculata \\\"incertae sedis\\\".\\n\\\"E. biesmeijeri\\\" is one of four bee species described by Dehon and team in the \\\"PLOS One\\\" article, the others being \\\"Andrena antoinei\\\", \\\"Bombus cerdanyensis\\\", and \\\"Protohabropoda pauli\\\".\\nDescription.\\nThe \\\"E. biesmeijeri\\\" fossil is a female preserved with a dorsal view of the body, out stretched wings, and missing its head. The overall body length is not determinable due to slight curling of the body and the missing head, though the mesosoma is . The metatibia are about long, not flared and enlarged notably, and with a distinct corbiculate pollen basket formed of a fringe of long setae. The original coloration and color pattering has been lost, so color pattern and if the color was metallic as in Euglossini species. The forewings have a one marginal cell and three cells below that called the submarginal cells. The marginal cell had a closed tip end that is notably rounded. The second and third submarginal cells together are longer than the first submarginal cell, which is the longest of the three cells. A pterostigma is present, but it incompletely preserved making observations difficult.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694524\",\"title\":\"Al and Zoot\",\"body\":\"\\nAl and Zoot\\n\\nAl and Zoot is an album by the Al Cohn Quintet featuring Zoot Sims recorded in 1957 for the Coral label.\\nReception.\\nThe AllMusic review by Scott Yanow states, \\\"The mid- to late '50s were a period of intense recording activity and this album is one of the underrated gems that was somewhat overlooked at the time\\\". \\nTrack listing.\\n\\\"All compositions by Al Cohn except as indicated\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694552\",\"title\":\"Vertigo (1917 film)\",\"body\":\"\\nVertigo (1917 film)\\n\\nVertigo (French:Vertige) is a 1917 French silent film directed by André Hugon and starring Régine Marco, André Nox, Marie-Louise Derval.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694558\",\"title\":\"Rafael Soromenho\",\"body\":\"\\nRafael Soromenho\\n\\nRafael Almeida Soromenho (born 4 April 1994) is a Portuguese footballer who plays for Sporting Clube Farense B, as a midfielder.\\nFootball career.\\nOn 9 January 2013, Soromenho made his professional debut with Olhanense in a 2012–13 Taça da Liga match against Moreirense.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694589\",\"title\":\"Metjen\",\"body\":\"\\nMetjen\\n\\nMetjen (also read as Methen) was a Ancient Egyptian high official at the transition time from 3rd dynasty to 4th dynasty. He is famous for his tomb inscription, which provide that he worked and lived under the kings (pharaohs) Huni and Snefru.\\nIdentity.\\nFamily.\\nAccording to his own tomb inscriptions, Metjen was a son of the high official Inpu-em-Ankh, a judge at the royal court of justice and a royal scribe. Metjen's mother was a high priestess named \\\"Neb-senet\\\". Metjen also had children, which he inderectly mentions, but their names are not handed down.\\nTitles.\\nAs a high-ranking official, Metjen bore several elite titularies:\\nCareer.\\nMetjen's tomb inscriptions are of highest interest to Egyptologists and Historians alike. They are in fact the very first Ancient Egyptian private texts that reveal more than only titles and offering formulae. Metjen is the first high official who reports his own professional and curatorial career by listing his official and honorary titles in chronological order and describing the support of his career by his parents.\\nAccording to Metjen's autobiography, he inherited many of his titles after the death of his father, other high ranked titles were assigned to him by his father himself. The inheritance included also several decrees which allowed Metjen to found own estates and even small towns. Metjen started as a royal scribe, became then overseer of the royal scribes and confidant of the king, then he became administrator of several palatinates and royal storages. Finally, he became \\\"great one of the 'Ten of Upper Egypt'\\\" and privy council. Additionally, the royal house gave several estates to Metjen, whereupon he founded the town \\\"Sheret-Metjen\\\" (Egyptian: \\\"Šr.t-Mṯn\\\") at his favorite estate. As a sign of gratitude, Metjen in turn donated hundrets of precious trees to royal domains. At the peak of his career, Metjen was also mayor of several cities, curator of the endowment estate of king Huni and supervizor of the mortuary cult for queen Nimaathapi.\\nContemporary office partners included \\\"Netjeraperef\\\", \\\"Khabawsokar\\\", \\\"Pehernefer\\\" and \\\"Akhetaa\\\", who were also holding office under Huni and Sneferu. All their tomb inscriptions reveal that the time of both kings must have been a very prosperous one and economy and office administration flourished.\\nTomb.\\nMetjen's tomb, mastaba \\\"L6\\\", was found in 1842 by German Egyptologist Carl Richard Lepsius at Saqqara. He excavated the tomb and ordered the dismantling of the tomb for preservation. Metjen's tomb chapels are now completely at display at the Egyptian Museum of Berlin.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694597\",\"title\":\"Jackie Akhavan\",\"body\":\"\\nJackie Akhavan\\n\\nJacqueline Akhavan is a British chemist, and an expert in the chemistry of explosives.\\nEarly life and education.\\nAkhavan grew up in London. She attended the University of Southampton where she received her BS in chemistry in 1979. After graduating, she began working at Pirelli General developing polymeric optical fibers for communication. She worked at Pirelli for 3 years when she went back into academia by joining Cranfield University as a polymer and physical chemistry lecturer. She also received her master's degree in Philosophy in 1982 and then received her PhD in polymer chemistry from Southampton University.\\nAcademic career.\\nAt Cranfield, she first started her work using the explosives in fireworks by understanding the reactions taking place as well as making them safe for the public to use. Eventually, she was promoted to senior lecturer at Cranfield and became the head of the MSc in explosives. Her research also is focused on polymer bonded explosives and their applications. During her tenure, she has received numerous research contracts from government organization, such as the Ministry of Defense UK, and also various corporate contracts from companies such as BAe Systems and MBDA. Most of her work, however, is secret due to the nature of her work and the application of her research will never be published. In 2007, she was given a chair in explosive chemistry and is now the Head of the Center for Defense Chemistry. As of September 2014, her current research is focused on making it easier for government agencies to detect explosives during security screenings like the ones conducted at airports. The main focus of the this new research is developing viable ways to detect liquid and paste explosives. Akhavan also helps to train government agencies in bomb recognition and detection to improve safety.\\nShe is a Fellow of the Royal Society of Chemistry.\\n\\\"The Chemistry of Explosives\\\".\\nAkhavan published a book called \\\"The Chemistry of Explosives\\\". This book covered many different aspects of explosives including the classification of explosives, combustion, ignition, thermochemistry, and kinetics. Also, there is an introduction to explosives at the beginning of the book that goes of over the development of most the notable explosives used today. The book is meant to teach others about the science behind explosive compounds.\\nNotable work.\\nAkhavan has led research for: \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694609\",\"title\":\"Sharks (1917 film)\",\"body\":\"\\nSharks (1917 film)\\n\\nSharks (French:Requins) is a 1917 French silent crime film directed by André Hugon and starring Charles Krauss, Marcel Bérard and André Nox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694623\",\"title\":\"Yttrium hydride\",\"body\":\"\\nYttrium hydride\\n\\nYttrium hydride is a compound of hydrogen and yttrium. It exists in several forms, the most common being a metallic compound with formula YH2. YH2 has a face centred cubic structure. Under great pressure, extra hydrogen can combine to yield an insulator with a hexagonal structure, with a formula close to YH3. Hexagonal YH3 has a band gap of 1.8 eV. Under pressure of 12 GPa YH3 transforms to an intermediate state, and when the pressure increases to 22 Gpa another metallic face centred cubic phase is formed.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694640\",\"title\":\"List of songs recorded by Niykee Heaton\",\"body\":\"\\nList of songs recorded by Niykee Heaton\\n\\nNiykee Heaton is an American singer and songwriter. She has written and recorded material since her childhood, having started sharing original songs occasionally on YouTube in 2011, for which she accompanied herself on acoustic guitar, along with covers of contemporary hits, where they subsequently went viral in following years, helping to propel the singer to prominence. Heaton released her debut project, an extended play titled \\\"Bad Intentions\\\", in September 2014, after signing a record deal with Steve Rifkind and Russell Simmons's Capitol-affiliate All Def Digital, in partnership with Awesomeness Music. The EP garnered positive online reception, with \\\"The Huffington Post\\\" writing that it \\\"offers a significant variety for a debut, all held together by her vocals, most ravishing in her lower register\\\" and Idolator adding that it \\\"confidently showcases the versatility and scope of Niykee's songwriting.\\\"\\nIn 2015, she called out her label in an extensive open letter to her fans, later clarifying that \\\"we are now in a place where we can create music that I want to create, and we are no longer tied to the people that were holding me back,\\\" also announcing plans to release her first album. In June, she launched a website, NBK (Naturyl Born Killers), stating \\\"NBK is the movement,\\\" where new music was then premiered for free regularly over the following months through her SoundCloud account.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694647\",\"title\":\"Tethong\",\"body\":\"\\nTethong\\n\\nPeople with the surname Tethong include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694655\",\"title\":\"Worthless Woman\",\"body\":\"\\nWorthless Woman\\n\\nWorthless Woman (French:Fille de rien) is a 1921 French silent film directed by André Hugon and starring Suzanne Talba, José Durany and Vasseur.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694656\",\"title\":\"Ivar Halfdansson\",\"body\":\"\\nIvar Halfdansson\\n\\nIvar Halfdansson was a king of Oppland, Norway one of the Petty kingdoms of Norway during the Migration Period. He was an Earl of Norway, Jarl and king of Uplanders, great grand father of Rollo of Normandy.\\nHe is mentioned in the \\\"Ættartalur\\\" section of Flateyjarbók.\\nHe is frequently confused with other people named Ivar in the sagas.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694671\",\"title\":\"2011 Ligue 1 (Senegal)\",\"body\":\"\\n2011 Ligue 1 (Senegal)\\n\\nThe 2011 Ligue 1 season was the 46th of the competition of the first-tier football in Senegal and the fourth professional season. The tournament was organized by the Fédération Sénégalaise de Football (Senegalese Football Federation). The season began earlier on 18 December 2010 and finished on 9 September 2011. It was the third season labelled as a \\\"League\\\" (\\\"Ligue\\\" in French). US Ouakam won their first and recent title, and a year later would compete in the 2012 CAF Champions League. ASC Diaraf, second place and the winner of the 2012 Senegalese Cup Casa Sport participated in the 2012 CAF Confederation Cup, it was the last time bringing a second place club, only the cup winner would participate in the following season.\\nThe season would have feature 16 clubs and once again, the winner would be decided on the highest number of points, it was decided after the thirtieth match was finished. Not until the next season it would reappear in that format, the first and second phase system would reappear in the next season. The season scored a total of 368 goals. Casa sport had the highest total of 30 goals scored and the least was CSS Richard Toll with 15.\\nASC Diaraf again was the defending team of the title.\\nOverview.\\nThe league was contested by 16 teams.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694672\",\"title\":\"Iraq men's national squash team\",\"body\":\"\\nIraq men's national squash team\\n\\nThe Iraq men's national squash team represents Iraq in international squash team competitions, and is governed by Iraqi Squash Federation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694676\",\"title\":\"Food porn\",\"body\":\"\\nFood porn\\n\\nFood porn is a glamourized spectacular visual presentation of cooking or eating in advertisements, infomercials, blogs cooking shows or other visual media, foods boasting a high fat and calorie content, exotic dishes that arouse a desire to eat or the glorification of food as a substitute for sex. Food porn often takes the form of food photography and styling that presents food provocatively, in a similar way to glamour photography or pornographic photography.\\nHistory.\\nThe term appears to have been coined by the feminist critic Rosalind Coward in her 1984 book \\\"Female Desire\\\" in which she writes: \\\"Cooking food and presenting it beautifully is an act of servitude. It is a way of expressing affection through a gift... That we should aspire to produce perfectly finished and presented food is a symbol of a willing and enjoyable participation in servicing others. Food pornography exactly sustains these meanings relating to the preparation of food. The kinds of picture used always repress the process of production of a meal. They are always beautifully lit, often touched up.\\\" (p. 103)\\nIt is important to realize that the term food porn does not strictly deal with the connection, often established throughout history, between food items and sexual contents. In the United States, food porn is a term applied when \\\"food manufacturers are capitalising on a backlash against low-calorie and diet foods by marketing treats that boast a high fat content and good artery-clogging potential\\\". The origin of the term was attributed to the Center for Science in the Public Interest which began publishing a regular column called \\\"Right Stuff vs. Food Porn\\\" for its Nutrition Action Healthletter in January 1998.\\nIn the United Kingdom, the term became popular in the 1990s due to the TV cookery programme \\\"Two Fat Ladies\\\" after the shows producer described the \\\"pornographic joy\\\" the pair of them took in using vast quantities of butter and cream.\\nConnection with Business.\\nTaking a picture of food became one of the norms for younger generation in the world. Study from YPulse shows 63% of people between thirteen years old to thirty two years old posted their food picture while they are eating on social networking services. Moreover, 57% of people in the same age range posted information of the food they were eating at that time. From the percentage, food and social media are starting to connect together as trend. People use hashtag foodporn (#foodporn) unintentionally, but for doing that it helps food industry to track down the data to catch hungry attentions from audiences on social networking services. In current number, about 54 million food pictures are hashtagged on only Instagram. On Facebook, Social media helps connecting people through food trend, and #foodporn.\\nUses/ Community.\\nThe term food porn has shifted throughout its first appearances. Articles mentioned food porn as early as the late 1970s. The phrase food porn was used in a literal manner, describing food that was unhealthy for human consumption, directly comparing it to pornography. Its use then took on a new meaning, being used to describe food that was presented and prepared in a manner that was aesthetically appealing. It took on this use for over a decade, until the social media boom that was created by the internet. Once the early 2000s hit, the terminology evolved into a way of documenting meals that are recognized for their presentation. This desire for food has flooded the internet, having significant effects on social media sites that provide the ability to display such as Instagram, Flickr, Snapchat, Facebook, and Twitter. The popularity of displaying food in a physically appealing manner is driven by the users that create these communities. The use of hashtags that the users of these sites have adapted to, allow food porn to connect people in a way that documents anything about the food such as, foods that reflect cultures, calories, presentation, preparation, delicious taste, and anything else that adds to the authenticity of the meal.\\nCulture.\\nThe term foodporn refers to images of food across various social media platforms such as TV, cooking magazine, online blog, website and social media platforms. The reason why foodporn is strongly connecting with popular culture is due to the fact that people are exposed to food in their everyday lives. Foodporn is not specific to social media platforms and could also be part of the category on newspaper and online blog. Moreover, foodporn is experienced globally. Language barriers that exist culturally can be bypassed by the usage of #foodporn. Food porn is used collectively by the online users and is does not exclude or privilege one food over another.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694696\",\"title\":\"Grand Lake (Sheet Harbour)\",\"body\":\"\\nGrand Lake (Sheet Harbour)\\n\\nGrand Lake is a lake just west of Sheet Harbour, Nova Scotia. It is crescent shaped. It's primary outflow is Little West River, which flows in to the Northwest Arm of Sheet Harbour. The land area of the Sheet Harbour 36 Indian reserve extends to a part of the shore of Grand Lake.\\nIn 2012, the Nova Scotia Government bought of land near the lake worth $1.1 million.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694699\",\"title\":\"A Crime Has Been Committed\",\"body\":\"\\nA Crime Has Been Committed\\n\\nA Crime Has Been Committed (French:Un crime a été commis) is a 1919 French silent film directed by André Hugon.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694702\",\"title\":\"FabricLive.81\",\"body\":\"\\nFabricLive.81\\n\\nFabricLive.81 is a 2015 DJ mix album by Monki. The album was released as part of the FabricLive Mix Series.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694719\",\"title\":\"Myoporum wilderi\",\"body\":\"\\nMyoporum wilderi\\n\\nMyoporum wilderi, commonly known as Cook Islands myoporum or ngaio, is a plant in the figwort family, Scrophulariaceae and is endemic to the islands of Mangaia and Mitiaro in the Cook Islands. It is similar to \\\"Myoporum stokesii\\\" and \\\"Myoporum rimatarense\\\" which grow on other Pacific Islands. On the island of Raratonga it is used to add a scent to coconut oil.\\nDescription.\\n\\\"Myoporum wilderi\\\" is a shrub or small tree sometimes growing to a height of with young branches that are flattened or three-sided. The leaves are arranged alternately, are broadly elliptical in shape with a pointed tip and are mostly long and wide. They are the same shiny colour on both surfaces, glabrous and have a distinct mid-vein.\\nThe flowers are borne in groups of up to 6 in the axils of leaves on stalks long and usually have 5 pointed sepals and 5 petals forming a tube or bell-shape. The tube is long with lobes about the same length. The tube is white with distinct purple spots in the tube and on the inner parts of the lobes. There are four stamens which extend beyond the petal tube. The fruits is an oval shaped drupe with a distinct point on the end.\\nTaxonomy.\\n\\\"Myoporum wilderi\\\" was first formally described in 1933 by Carl Skottsberg and the description was published in \\\"Acta Horti Gothoburgensis\\\". The specific epithet \\\"wilderi\\\" honours the botanist Gerrit Parmile Wilder.\\nDistribution and habitat.\\n\\\"Myoporum wilderi\\\" is only found on Mangaia and Mitiaro.\\nUses.\\nOn Mangaia, the flowers of this myoporum are used to scent coconut oil. On Raratonga, the species is cultivated for the same purpose.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694728\",\"title\":\"A Perilous Journey\",\"body\":\"\\nA Perilous Journey\\n\\nA Perilous Journey is a 1953 American adventure film directed by R. G. Springsteen and written by Richard Wormser. The film stars Vera Ralston, David Brian, Scott Brady, Charles Winninger, Hope Emerson, Eileen Christy and Leif Erickson. The film was released on April 5, 1953, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694729\",\"title\":\"Greg Norman Medal\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694730\",\"title\":\"The Gold Chignon\",\"body\":\"\\nThe Gold Chignon\\n\\nThe Gold Chignon (French:Chignon d'or) is a 1916 French silent film directed by André Hugon and starring Mistinguett and Harry Baur.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694734\",\"title\":\"John F. Hager\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694749\",\"title\":\"Order of the Sword & Shield\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694770\",\"title\":\"Weaubleau Christian College\",\"body\":\"\\nWeaubleau Christian College\\n\\nWeaubleau Christian Institute was founded in 1873 in Hickory County, Missouri, under the auspices of the Weaubleau Congregational Christian Church. The congregation of the church was made up of families living on newly settled farms, who chose a central location on which to erect a 2-story building large enough to accommodate a church on the first floor and an academy or secondary school on the second. A small town, first called Haran, but later renamed Weaubleau for the stream upon which it is located, grew up around the Church and Institute.\\nThe Institute was incorporated under a board of trustees, the majority of whom were to be perpetually drawn from among the members of the church. The student body never exceeded 15) at any point in the schools' first 35 years. One of the College's Presidents, John Whitaker, called it a \\\"frontier college,\\\" and wrote that many of the student taught terms in frontier schools to earn their tuition fees.\\nThe Institute gained accreditation as a college around 1893, eventually granting three degrees: Bachelor of Science, Bachelor of Accounts, and Master's Degree.\\nThe College closed in 1914. The Weaubleau Congregational Christian Church closed in 1960. Some papers related to the College and the Church are held by the State Historical Society of Missouri. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694781\",\"title\":\"Edson Vidigal\",\"body\":\"\\nEdson Vidigal\\n\\nEdson Carvalho Vidigal (born June 20, 1944) is a Brazilian politician and lawyer. He served Brazil's Superior Court of Justice from December 9, 1987 to March 29, 2006.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694814\",\"title\":\"Rooted in Community\",\"body\":\"\\nRooted in Community\\n\\nRooted in Community National Network also referred to as R.I.C or Rooted in Community is an non-profit organization, which was founded in Boston. It is a national network that brings young people together to create food security, community gardens and fight for food justice in low income communities. Diverse group of adults and youths helps \\\"empowers young people to take leadership in their own communities.\\\" Annual conference and regional gathering for youths to have the opportunity to travel different communities.\\nHistory.\\nTwo High School students from The Food Project attended 1998 American Community Gardening Association (ACGA) conference. They feel there was a lack of youths involved in the food system and fighting for food justice. First conference was held in Boston with over 70 youths attending.\\nProgram Overview.\\nInviting youths from across the country to experience different ways to improve their communities. Attending to different workshops with team building and leadership activities, also discuss issues with the food industry in America. Also fighting for Food Justice for communities to have the right to grow, sell, and eat inexpensive healthy food. Changing the food system by helping passing the Youth Food Bill of Rights in Washington. Youths learn about food preparation creating healthy eating habits and appreciation for agriculture. RIC created the TOOL SHED, an online information resource website to help other farmers and youths. The TOOL SHED has lessons on giving a workshop, lecture, and activities.\\nList of Network Groups.\\nList of organizations, programs and supporters who attended to conferences in the past:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694819\",\"title\":\"Adam C Franklin\",\"body\":\"\\nAdam C Franklin\\n\\nAdam Franklin is an Australian author, entrepreneur and social media keynote speaker.\\nFranklin published \\\"Web Marketing That Works: Confessions from the Marketing Trenches\\\" (Wiley 2014) (ISBN 978-0-7303-0927-7) as a paperback and Kindle ebook with co-author Toby Jenkins and it became an Amazon #1 bestseller in Australia.\\nHe co-founded his company Bluewire Media with Toby Jenkins in 2005. Franklin featured on \\\"Smart Company\\\"'s \\\"Australia's Hot 30 Under 30\\\" list of entrepreneurs in 2009. In 2010 he collaborated with David Meerman Scott on the Web Strategy Planning Template which they co-branded. This marketing strategy template was also included in the third, fourth and fifth editions of David Meerman Scott's bestselling book The New Rules of Marketing and PR which has over 350,000 copies in print in more than 25 languages.\\nHe co-hosts the podcast \\\"Web Marketing That Works\\\" and was interviewed on the Entrepreneur On Fire podcast.\\nIn 2015, Franklin's Bluewire Media blog was named the #1 Business Blog in Australia according to Smart Company and was a Business Finalist for the Best Australian Blogs in 2014 according to the Australian Writers' Centre.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694820\",\"title\":\"The Jackals (1917 film)\",\"body\":\"\\nThe Jackals (1917 film)\\n\\nThe Jackals (French:Les chacals) is a 1917 French silent adventure film directed by André Hugon and starring André Nox, Louis Paglieri and Musidora.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694840\",\"title\":\"The Europe Trust\",\"body\":\"\\nThe Europe Trust\\n\\nThe Europe Trust, formerly known as the European Trust, is a UK charity and company which lists its purpose as developing a portfolio of assets intended to fund social and economic projects for communities in Europe. The Europe Trust was founded by the Federation of Islamic Organizations in Europe (FIOE), an organization alleged to have ties with the Muslim Brotherhood in Europe.\\nHistory and organization.\\nEurope Trust is a Markfield, United Kingdom, based company and charity which was first registered as a U.K. charity in1996 under the name European Trust and then incorporated as a U.K. company named Europe Trust in 2003. In 2004 it was again registered as a charity bearing the name Europe Trust. The 2005 financial report says that the principal activity of the company \\\"was that of to establish a portfolio of assets (awqaf) businesses and investments to generate resources to fund social and economic projects for communities in Europe.\\\" \\nThe directors and trustees of the Europe Trust are European Muslims of Middle Eastern descent. The Europe Trust employs three staff members and has multiple volunteers. Former trustee and co-founder Dr. Ahmed Kadhem al-Rawi serves currently as the chief executive.\\nConnections to the Muslim Brotherhood.\\nMajor news outlets as the Wall Street Journal and The Times have reported that the Europe Trust is the financial vehicle of the Muslim Brotherhood in Europe. The Wall Street Journal reported in 2005 that the European Trust was created in 1996 as the de facto fundraising arm of the Federation of Islamic Organizations in Europe in order to break the dependency of FIOE on Gulf donors. The Journal also reported that the Trust had directly subsidized projects of the FIOE, including three colleges and three local Islamic centers that affiliated with the FIOE.\\nThe Times reported that the Europe Trust, which has property assets worth more than £8.5 million, sends rental income from its properties to an unofficial network of Brotherhood-linked organizations throughout the continent including the Muslim Association of Britain (MAB), identified by a government minister in 2010 as “the Brotherhood’s representative in the UK.”\\nConnections to radicalism and terrorism.\\nThe Times of London reported a number of ties from the Europe Trust and/or its Trustees to various forms of terrorism:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694849\",\"title\":\"Flower of Paris\",\"body\":\"\\nFlower of Paris\\n\\nFlower of Paris (French:Fleur de Paris) is a 1916 French silent film directed by André Hugon and starring Mistinguett, Harry Baur and Louis Paglieri.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694884\",\"title\":\"The Four Brothers... Together Again!\",\"body\":\"\\nThe Four Brothers... Together Again!\\n\\nThe Four Brothers... Together Again! is an album by saxophonists Herbie Steward, Al Cohn, Zoot Sims and Serge Chaloff, who were collectived known as the Four Brothers, recorded in 1957 for the RCA Records subsidiary Vik label.\\nReception.\\nThe AllMusic review by Scott Yanow states, \\\"The music overall on this CD reissue is quite enjoyable and Sims, Cohn and Steward show how much they had grown during the previous decade\\\".\\nTrack listing.\\n\\\"All compositions by Al Cohn except as indicated\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694901\",\"title\":\"Southern Cross (1932 film)\",\"body\":\"\\nSouthern Cross (1932 film)\\n\\nSouthern Cross (French:La croix du sud) is a 1932 French drama film directed by André Hugon and starring Charles de Rochefort, Suzanne Christy and Alexandre Mihalesco.\\nThe film's sets were designed by the art director Christian-Jaque.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694925\",\"title\":\"Bridge No. L1409\",\"body\":\"\\nBridge No. L1409\\n\\nBridge No. L1409 was a historic stone arch bridge in Hillsdale Township, Minnesota, United States, built in 1895. However it was largely destroyed during the 2007 Midwest flooding, when runoff carried away everything except the arch substructure. It had been listed on the National Register of Historic Places in 1990 for having state-level significance in the theme of engineering. It was nominated as Minnesota's \\\"most impressive\\\" rural stone arch bridge, owing to its fine ashlar masonry and sizeable span. Although the bridge has now been replaced with a modern structure, it has not yet been delisted from the National Register.\\nHistory.\\nResidents of Hillsdale Township petitioned Winona County for a bridge at this location in September 1894. Initially tabled, the request was later approved and plans were drafted by county surveyor Fred H. Pickles. The project went out for contract in October 1895 and local stonemason Charles Butler—with the lowest bid at $1,340—was selected. The bridge was largely completed by December of that year.\\nBridge No. L1409 was among a concentration of rural stone arch bridges in Southeast Minnesota built in the late-19th and early-20th centuries. Most of these were built by local governments in response to the Good Roads Movement. Few, however, spanned more than . L1409 was three times that length, and with the fine masonry produced by Butler, it was comparable to the larger and more sophisticated bridges of Minnesota's urban areas.\\nIn the summer of 2007, extreme flooding in Southeast Minnesota sent torrents of water sweeping down Garvin Brook. L1409's spandrel walls and earth fill were torn away, though the stone arch was so sturdily built it held in place. However the bridge was essentially destroyed, and it has since been replaced with a modern structure.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694936\",\"title\":\"Ben Stillman\",\"body\":\"\\nBen Stillman\\n\\nBen Stillman is an American film producer. Stillman assisted in the development of the Oscar-winning film \\\"The Imitation Game\\\" in 2014. He has produced four films including \\\"Broken City\\\", which premiered at Sundance Film Festival, and \\\"Gold\\\" (2016) starring Matthew McConaughey in 2016. Stillman is vice president of Black Bear Pictures.\\nCareer.\\nIn 2011, Black Bear Pictures was established and Stillman joined the company as creative executive after leaving Cinetic International. Stillman was associate producer of \\\"At Any Price\\\" in 2012. He was named vice president of Black Bear Pictures in 2013. He co-produced \\\"A.C.O.D.\\\" starring Adam Scott, as well as \\\"Broken City\\\", starring Mark Wahlberg and Russell Crowe, in 2013. He was executive producer of \\\"Gold\\\", a drama-thriller film about the search for gold in the jungles of Indonesia that is scheduled to be released in 2016.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694952\",\"title\":\"Suzanne Christy\",\"body\":\"\\nSuzanne Christy\\n\\nSuzanne Christy (1904–1974) was a Belgian film actress.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694961\",\"title\":\"Pugilina tupiniquim\",\"body\":\"\\nPugilina tupiniquim\\n\\nPugilina tupiniquim is a species of sea snail, a marine gastropod mollusk in the family Melongenidae, the crown conches and their allies. \\\"P. tupiniquim\\\" and its Eastern Atlantic sister species, \\\"Pugilina morio\\\", were once thought to be a single, amphiatlantic entity. They have, however, been recognized as distinct taxa based on anatomical and environmental distinctions.\\nDistribution.\\nThis species is found in the Western Atlantic Ocean, in mangrove areas along the Brazilian coast, and north to the Caribbean.\\nHuman use.\\nIn traditional Brazilian medicine in the Northeast of Brazil, \\\"Pugilina tupiniquim\\\", formerly referred to as \\\"Pugilina morio\\\" (actually an Eastern Atlantic sister species) is used as zootherapeutical product for the treatment of sexual impotence.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694980\",\"title\":\"DomiNations\",\"body\":\"\\nDomiNations\\n\\nDomiNations is a 2015 freemium mobile MMO strategy video game developed and published by Nexon and Big Huge Games.\\nGameplay.\\nGameplay is similar to other simulation games like Clash of Clans. You build a base and expand by building resource buildings, barracks, and defensive buildings.\\nReception.\\nDominations was listed as one of the seven best games of April 2015 by Gamezebo. It was given four out of five stars by 148apps shortly after release, and currently holds four out of five stars on the Apple app store.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694985\",\"title\":\"John F. Remondi\",\"body\":\"\\nJohn F. Remondi\\n\\nJohn (Jack) F. Remondi is an American businessman and the president and CEO of the Navient Corporation.\\nCareer.\\nRemondi became first CEO of the Navient Corporation in April 2014 following its spin-off from Sallie Mae. Prior to Navient, he was the CEO of Sallie Mae from May 2013. Prior to Sallie Mae, Remondi served as Portfolio Manager to PAR Capital Management, from 2005 to 2008. From 1999 to 2005, Remondi served in several financial positions with Sallie Mae, including Executive Vice President, Corporate Finance.\\nEducation.\\nRemondi graduated from Xaverian Brothers High School in Westwood, Massachusetts and received his bachelor's degree in Economics from Connecticut College.\\nPersonal life.\\nRemondi is married to Judith Dickstein. He is a former member of the Board of Trustees at Xaverian Brothers High School.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48694990\",\"title\":\"1997–98 Plymouth Argyle F.C. season\",\"body\":\"\\n1997–98 Plymouth Argyle F.C. season\\n\\nThe 1997–98 season was the 103th season in the history of Plymouth Argyle Football Club, their 73rd in the Football League,\\nSquad.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695007\",\"title\":\"Alexandre Mihalesco\",\"body\":\"\\nAlexandre Mihalesco\\n\\nAlexandre Mihalesco (1883–1974) was a Romanian film actor who largely appeared in French productions.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695014\",\"title\":\"Helene Bresslau Schweitzer\",\"body\":\"\\nHelene Bresslau Schweitzer\\n\\nHelene Bresslau Schweitzer was co-founder of the Albert Schweitzer Hospital, medical missionary, nurse, social worker, linguist, public medicine enthusiast, editor, feminist, sociologist, mother, and wife/confidant of Albert Schweitzer. Albert, a medical missionary, did not mention her role in his efforts. According to writer Mary Kingsley, she is \\\"one form of human being whose praise has never adequately been sung, namely, the missionary's wife.\\\" While much of his work seems to overwrite her own, she played a pivotal role in the advancement of medicine, feminine independence, and societal justice.\\nEarly life.\\nHelene Bresslau Schweitzer was born to the Bresslau family on January 25, 1879 in Berlin. Her maternal family was of Jewish descent, but she was baptized into the Christian religion as a result of wide-spread anti-semitism. The Bresslaus moved to Alsace, France when she was eleven because of a new job opportunity for her father. Her father, Harry Bresslau, began working at the University of Strasbourg and eventually became chancellor. As a result of the move, Schweitzer adopted French, becoming fluent rather quickly.\\nIn 1898, Bresslau met her future husband, Albert Schweitzer at a wedding. Shortly thereafter, they developed a relationship that included separation, independence, and non-exclusive behaviors. This allowed both to develop their lives while enjoying each other's companionship, conversation, and virtues. The one thing that united the pair was their shared ideology: to the take care of others.\\nBresslau became Albert's confidant but did not give up her own life for his. In fact, they spent a great deal of time away from each other and maintained a nontraditional relationship (together but not exclusive). They felt secure remaining undefined as a couple, relying on their friendship through documented letters. The turning point for their relationship occurred when they married on June 18, 1912 in Gunsbach. At this point in their lives, they both decided to marry and go to Africa to fulfill their desire to care for others in need. She quit her job at the orphanage and studied higher level nursing to advance her knowledge before leaving. On Good Friday of 1913, she travelled with Albert to Lambaréné, Gabon, beginning her medical missionary adventure.\\nEducation and professional development.\\nAged 6, Bresslau attended Queen Charlotte's School. In 1890, she transferred to Lindner Girls' High School in Berlin. She began to pursue music studies at a music conservatory from 1897 to 1899. After receiving her teaching credentials in one year rather than the usual two, she worked as a teacher in England in 1902. Continuing to delve into her passion for learning, Bresslau took courses in medieval, modern, and art history at her father's university, the University of Strasbourg. In pursuit of music, she took voice and piano lessons.\\nOne area of study that interested Bresslau was nursing. She joined the Protestant Deaconess' Society on January 1, 1904 \\\"to complete a course in nursing. After, she was assigned to complete three months of nursing lessons in Stettin. On April 1, 1905, she took a break from nursing and went into social work. Even so, exploring another field other than nursing left her \\\"eager to fill in the gaps\\\" of her nursing knowledge.\\nShe changed her direction of study when she became a municipal inspector for orphans in 1905. She maintained the position from 1905 to 1909. This endeavor attributes largely to part of her own goal to improve the social sphere. However, her home's \\\"Jewish atmosphere\\\" widely influenced her as she was taught to \\\"pay it forward.\\\" Including and prior to this job, all of her endeavors were based on her own emotions and goals without the Albert's influence. In one of his letters, he notes \\\"it is you who have won, happy to have found a task that will fill your life, and you’ve done it ahead of me\\\", addressing her social work in Strasbourg's City Orphan Administration.\\nOn October 1, 1909, Schweitzer \\\"enrolled as a student in the nursing school of the Protestant Deaconess' Society in Frankfurt in the city hospital\\\" to further her knowledge in the profession, thus beginning her nursing career.\\nMissionary work.\\nJourney with Albert Schweitzer.\\nSchweitzer and Albert shared one main common goal: to help improve medicine and the greater good in Lambaréné, Gabon. At the very beginning of their journey, Schweitzer wrote in her diary that \\\"we are truly in love with Africa.\\\" In spring 1913, Schweitzer and Albert set off to establish a hospital (Albert Schweitzer Hospital) near an already existing mission post. The site was nearly 200 miles (14 days by raft) upstream from the mouth of the Ogooué at Port Gentil (Cape Lopez) (and so accessible to external communications) but downstream of most tributaries, so that internal communications within Gabon converged towards Lambaréné.\\nThis journey to make medical improvements in Africa allowed Schweitzer to further develop herself. Patti Marxsen writes that Schweitzer's \\\"capacity for hard work in a challenging environment can be read as proof that her independence earned in Strasbourg was now unshakeable. For the now thirty-four-year-old Helene Schweitzer...a life in Africa offered a chance to integrate multiple aspects of modern identity, perhaps even more so than would have been possible in Europe.”\\nSchweitzer had prior interest in nursing and the medical field before Albert became involved in medicine. Therefore, Schweitzer played a vital role in his work, acting as a possible influence. In the first nine months, Schweitzer and Albert had about 2,000 patients to examine, some travelling many days and hundreds of kilometers to reach him. In her time in Africa, she worked as nurse and helped with the hospital. She played an essential part in sanitation efforts, especially by preparing medical equipment for surgery. Schweitzer was an anaesthetist for surgical operations.\\nChallenges.\\nWhen World War I broke out in summer of 1914, the French military put Schweitzer and Albert, Germans in a French colony, under supervision at Lambaréné, where they continued their work. In 1917, exhausted by over four years' work and by tropical anaemia, they were taken to Bordeaux and interned first in Garaison and then from March 1918 in Saint-Rémy-de-Provence.\\nMedical issues forced Schweitzer to leave Africa many times, and sometimes Albert kept her from returning at times. When Albert decided to return to Africa in 1924, he took on Oxford undergraduate, Noel Gillespie, as assistant, leaving Schweitzer behind. After the birth of their daughter (Rhena Schweitzer Miller), Schweitzer was no longer able to live in Lambaréné owing to her health. In 1923 the family moved to Königsfeld im Schwarzwald, Baden-Württemberg, where Albert was building a house for the family. This house is now maintained as a Schweitzer museum.\\nHer not returning to Lambaréné was a sacrifice made \\\"by her husband, not for him.\\\" She wrote about her not returning, describing it as a \\\"practical matter\\\", but she \\\"never agreed to a separation of three and a half years\\\" from her husband. Despite her poor health, she still took care of her daughter, \\\"engage[d] herself with the Hospital Aid Association,\\\" and \\\"enroll[ed] in a three week course in tropical medicine at the Medical Missionary Institute of Tubingen, Germany.\\\" As a motherhood advocate, she gladly took care of her daughter and continued to develop her own personal skills. Schweitzer still remained engaged in helping the mission hospital.\\nIn 1929, after receiving treatment for pneumonia, Schweitzer returned to Lambaréné to see her husband Albert’s progression with the new hospital. Shortly after arriving, however, she developed a bad fever and was forced to depart the hospital and her husband to return to Europe for treatment. After recovering, she used her writing skills and began to edit her husband's autobiography. Her English skills also opened the door for \\\"public speaking and networking in the United States.\\\" On December 1, 1930, a German newspaper printed one of her speeches. In it, she described her husband’s concept, the Fellowship of the Marks of Pain. She turned her medical challenges into positives, explaining that through her suffering she developed a compassionate view of their work that only she could personally attest.\\nSchweitzer was aware that her husband would receive much of the acclaim for their missionary endeavors, so she set out to make her work known. In October 1946, she began to review her documents and collect them so that she would be understood as a \\\"full partner\\\" in their missionary work. In addition, she began lecture tours in the United States in 1937 to promote the Schweitzer Hospital.\\nHealth complications.\\nSchweitzer experienced tremendous health issues throughout her life, mostly in relation to the lungs. She first encountered tuberculosis before she turned ten. She was officially diagnosed in the spring of 1922 with laryngeal tuberculosis after exhibiting symptoms of \\\"pain, fever, and the coughing up of blood.\\\" In addition the heat of Africa caused many respiratory issues. In 1915, she contracted phlebitis resulting in two weeks of therapeutic bed rest. She also had pneumonia in 1929, almost keeping her from returning to Lambaréné. Despite her already weak lungs, she completed the trip though she had to return early due to illness again.\\nSchweitzer died on June 1, 1957, and her remains are located in Lambaréné. Once Albert died, he was buried next to her. Schweitzer's grave with Albert is \\\"on the banks of the Ogooue River is marked by a cross he made himself.\\\"\\nLegacy.\\nSchweitzer contributed greatly to the work done in Lambaréné. A role model as an independent, educated woman with a deep intellectual curiosity, she was \\\"One of the first female students at the University of Strasbourg\\\" and \\\"One of the first female employees in the community administration\\\" at the orphanage. Her aid in the poor relief system, \\\"Armenpflegesystem,\\\" mirrored in modern social welfare, saw the illegitimate mortality rate fall. Setting precedence as a female medical missionary in the early 20th century, she established lasting effects of nursing and education in Lambaréné. She co-founded the Schweitzer Hospital, documented much of Albert's autobiography, and \\\"supported the [mission] work with lectures and fund-raising\\\" essential to its upkeep and vivacity.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695035\",\"title\":\"Abu Dhabi Combat Club\",\"body\":\"\\nAbu Dhabi Combat Club\\n\\nThe Abu Dhabi Combat Club located in Abu Dhabi, United Arab Emirates (U.A.E.) is a martial arts club designed to promote martial arts in the U.A.E. The Abu Dhabi Combat Club is one of the top facilities for learning and training in mixed martial arts (MMA) in the Arab world. Its founder created the submission style (a type of \\\"grappling\\\" style) of martial arts and it hosted the first Abu Dhabi Combat Club Submission Fighting Championships in 1998. Many who have competed in the popular ADCC Submission Grappling Championship have gone on to have successful careers in MMA such as Ronaldo \\\"Jacare\\\" Souza and Vitor “Shaolin” Ribeiro and many others. Although the facility itself is used mainly by athletes and practitioners who reside in Abu Dhabi, many top level grapplers, fighters and BJJ specialists have been honored guests and had spent time training or teaching in the club. The Abu Dhabi Combat Club does not have its own fighters; it serves more as a platform where fighters can participate in training camps and take advantage of the large number of BJJ Black Belts who currently reside in Abu Dhabi. The facility where the ADCC was originally located is not owned by the UAE Wrestling, Judo and Kickboxing Federation, where classes are given in the same disciplines. However, the reputation of the ADCC is still strong and the facility is being visited by top fighters occasionally. \\nThe first five Championships held under the patronage of H.H. Sheikh Tahnoon Bin Zayed Al Nahayan were hosted in that same facility in Abu Dhabi Equestrian Club compound. From 1998 until the 2002 the events were held annually in Abu Dhabi. Ever since 2002 the event ADCC Submission Fighting Championship (it was renamed due to copy rights of the world Grappling) is held once every two years and every event is hosted in countries where the sport is well developed such as the US, Brazil and Japan.\\nHistory.\\nThe Abu Dhabi Fighting Club was created by U.A.E. national Sheik Tahnoon Bin Zayed Al Nahyan. While completing advanced studies in the United States of America, Sheik Tahnoon watched the first UFC fight and became a fan of MMA. The club was created out of an idle facility and world class trainers were hired.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695084\",\"title\":\"ESPN College Extra\",\"body\":\"\\nESPN College Extra\\n\\nESPN College Extra is an American out-of-market sports package. It was launched on August 28, 2015 as a merger of the existing ESPN Full Court and ESPN GamePlan, each of which offered college basketball and college football broadcasts respectively. \\nAvailability.\\nESPN College Extra is available with these cable and satellite providers:\\nAlmost all ESPN College Extra broadcasts are also made available through ESPN3.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695112\",\"title\":\"Adonay Alfaro Murcia\",\"body\":\"\\nAdonay Alfaro Murcia\\n\\nAdonay Alfaro Murcia is a former Costa Rican footballer who played in the USISL D-3 Pro League, and the Canadian National Soccer League.\\nPlaying career.\\nMurcia moved to Canada in 1996, and played with Toronto Supra of the Canadian National Soccer League. He would help Supra finish third in the league standings, and secured a postseason berth for the club. They were eliminated in the semi-finals and lost the series on 2-1 on goals on aggregate to the St. Catharines Wolves. In 1997, he abroad to the United States and signed with San Antonio Pumas of the USISL D-3 Pro League. During his tenure with San Antonio he helped the club reach the playoffs, before they were eliminated in the division finals to the Houston Hurricanes. After the 1997 season he retired from competitive football, and enrolled to San Antonio College, graduated with honors and is currently working in Web design.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695117\",\"title\":\"BCDF Pictures\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695138\",\"title\":\"Streptomyces coelescens\",\"body\":\"\\nStreptomyces coelescens\\n\\nStreptomyces coelescens is a bacterium species from the genus of \\\"Streptomyces\\\". \\\"Streptomyces coelescens\\\" produces glycoglycerolipids.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695146\",\"title\":\"Flood Mythology of China\",\"body\":\"\\nFlood Mythology of China\\n\\nThe Flood Mythology of China, or Great Flood of China (; also known as ) is a deluge theme which was happened in China. Derk Bodde (1961) stated that \\\"from all mythological themes in ancient Chinese, the earliest and so far most pervasive is about flood.\\\" The myth is also have shared characteristics with other Great Floods all over the world, although it also have unique characteristics or different focuses. Lu Yilu (2002) was grouping all versions of great flood into three themes: \\\"the heroes controls the flood; \\\"brother-sister marriage to repopulating the world\\\"; and \\\"the flood which is drowning the whole city along with its citizens\\\".\\nHistory and mythology.\\nLiterary history.\\nThe history of China as a continuously recorded literary tradition begins with the ancient documents transmitted to posterity through the \\\"Records of the Grand Historian\\\". According to these, the great-grandson (or fourth successor) of the Yellow Emperor was Yao. Beginning with the reign of Yao, additional literary sources become available, including the \\\"Book of Documents\\\" (collected and edited by Confucius), which begins with the \\\"Canon of Yao\\\" (), describing the events of Yao's reign. Although, the \\\"Canon of Yao\\\" is problematic in regards to textual transmission (at best it seems to represent an early textual reconstruction and at worst a fabrication based on available knowledge or sources from the 3rd or 4th centuries CE). \\\"The Counsels of Great Yu\\\" 大禹謨 is considered to be one of the reliably transmitted pre-Qin texts. In any case, these and other texts of the preserved literature mark the beginnings of the Chinese historical tradition. Other important texts include the poem \\\"Heavenly Questions\\\" collected in the \\\"Chu Ci\\\" which is attributed to Qu Yuan and the famous mythological compendium \\\"Classic of Mountains and Seas\\\" (\\\"Shanhaijing\\\"). Furthermore, centuries of scholarship have gone into piecing together a narrative from the bits, pieces, and occasionally longer sections found in these and other early sources, sometimes being subjected to heavy editorial handling in terms of viewpoint.\\nCollected mythology.\\nMythological stories besides having been preserved both in various literary forms, have also been collected from various oral traditions, some of these folktales are still told. Some of these sources are from people of the Han ethnicity and some from other ethnic groups.\\nThe struggles to control the flood.\\nThe Zhou sources narrated the struggles of the heroes or deities to control the floods. From all of these stories, the struggle of Yao, Gun, and Yu is the most obvious in describing the hardness of human effort to control the flood. Much later works from the Warring States period (\\\"Shi Ben\\\") and \\\"Di wang shi ji\\\" (3rd century) were pairing Yu and Nuwa as a couple and their previously uncorrelated stories were then completing each other.\\nGun-Yu Mythology.\\nThis theme is based on the efforts of Great Yu (and Gun) to control the flood, sometimes is also associated with Emperor Yao and Shun, and the initial efforts of human to domesticated wild animals as pack animals and livestock. The theme outline narrates Gun stole xirang to stop the flood while Great Yu channeled the flood into the sea and succeeded to subsided the water level, and so the earth can be cultivated. There are so many mythologies which are correlated to this theme, one of which is \\\"Dragon Gate\\\" -a canal through the mountains which was dug by Yu. While he was digging the canal, many of the carps were swept away and fell, they were so disappointed because they couldn't swim back into the upper level. Yu promised them, if any of the carp could leap through his dam, it would transform into a dragon, thus the place inherited its name.\\nGun-Yu theme is also have political background. Huainanzi stated that the great flood was caused by Gong Gong who was use the water to make havoc on the realm of Emperor Yao. Shan Hai Jing narrated another version that the flood was caused by Xiangliu, one of Gong Gong's minister, and was not Gong Gong himself.\\nNuwa repaired the heaven.\\nThe story was happened after Goddess Nüwa created (molded) humanity from yellow clay, brought them into life, and gave them the ability to reproduce. Gong Gong was banging his head into Mount Buzhou, which actually was the pillar to support the heaven. Nüwa then patched the sky with five-colored pebbles and piled up reeds ashes to stop the flood.\\nThe extinction of human race.\\nChen Jianxian (1996) said that this theme was one of some popular legends which was still being told by more than 40 ethnics in China. There is a possibility that the myth is rather new because the oldest recorded sources about this myth were from Six Dynasties, save that the oral tradition maybe much older.\\nThe theme was made into several versions, but the outline is about a great flood which was destroyed all the humans all over the world except a pair of brother and sister, or aunt and nephew. Both were forced to married in order to repopulated the world. One version stated that their children were ordinary humans, while the others said it was a lump of meat, squash, melon, or grindstone; after they opened, cut, or destroyed it, humans emerged.\\nSinking city.\\nThis theme have some specific characteristics: one or two people were survived, the statue which was crying blood, and the whole city along with its citizens were sinking. The survivor(s) was being saved by the gods because of his/her benevolent acts; may be an old lady or a devoted son. The blood crying statue was often a stone lion statue, or sometimes tortoise statue.\\nOther flood myths.\\nEast Sea and Mulberry Field.\\nA less widespread flood myth involves the goddess Magu: this myth involves the cyclic rise and fall of the ocean level over the eons: sometimes the sea floor is under water; at other times, it turns into mulberry fields. However, the material about Magu seems to be distinct from the idea of a great flood upon the land of China.\\nOne tradition narrates Magu, a benevolent Taoist lady who lived on the second century. She was reclaiming a very wide seashore water bodies on Kiangsu and changing it into mulberry fields.\\nHeyu.\\nShan Hai Jing mentions a pig-like creature with human face, yellow-colored and red-tailed, and it sounds like human singing. The creature is known as \\\"Heyu\\\"; he preys on human, reptiles, and snakes. It appearance is a sign that there will be a great deluge all over the world.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695172\",\"title\":\"RCAF Overseas Headquarters\",\"body\":\"\\nRCAF Overseas Headquarters\\n\\nThe Royal Canadian Air Force Overseas Headquarters, often abbreviated to RCAF Overseas, was responsible for Canadian airmen serving outside Canada during and just after World War II. The headquarters was established on 1 January 1940 and it was based in London. Its main functions were to conduct liaison with the British Air Ministry, to provide a central location for personnel records, and provide general administration. As the War progressed, the Overseas Headquarters gained increasing administrative authority over Canadian personnel but never gained any significant operational responsibility for RCAF units and formations which were integrated into the RAF's command structure.\\nSenior officers.\\nThe following officers were in charge of the RCAF Overseas Headquarters:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695181\",\"title\":\"List of female scientists in the 20th century\",\"body\":\"\\nList of female scientists in the 20th century\\n\\nThis is a \\\"historical\\\" list, intended to deal with the time period when women working in science were rare. For this reason, this list deals only with the 20th century. Some women who primarily worked in the 19th or 21st centuries may appear in a different list.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695201\",\"title\":\"Eleanor Gladys Copenhaver\",\"body\":\"\\nEleanor Gladys Copenhaver\\n\\nEleanor Gladys Copenhaver (also known as Eleanor Copenhaver Anderson) (1896-1985) was a social worker and activist who spent over 40 years as an organizer and community service worker for the YWCA. She began as a community organizer and worked her way up to the labor division, finally becoming head of the Industrial Division from 1937 to 1947. At the end of World War II, when women were phased out of the labor market, she was briefly dismissed, but then hired back to organize support for the communities springing up around the defense industry. \\nBiography.\\nEleanor Gladys Copenhaver was born on June 15, 1896 in Marion, Virginia to Laura Lu (née Scherer) and Bascom Eugene Copenhaver. Both of Copenhaver's parents taught at the school her grandfather Scherer had founded, Marion Female College. She spent her childhood at the family home, \\\"Rosemont\\\", which also housed Rosemont Industries, an organization established by her mother, which marketed local handicrafts. After completing her primary education at the public schools of Marion, Copenhaver began her university studies at Marion College before she transferred in 1914 to Westhampton College in Richmond, Virginia. She completed her B.A. in English in 1917 and returned that same year to Marion to teach. In 1918, she enrolled in social work classes at Bryn Mawr College in Pennsylvania and in 1919 worked in a settlement camp for New York City women during the summer. She completed her degree in 1920 earning a certificate in social economy.\\nCareer.\\nIn September 1920, Copenhaver became a Field Secretary on the national staff of the YWCA covering the south and central region. From 1920 to 1923 she was a rural community organizer. In addition to introducing the programs of the YWCA to local communities and churches, Copenhaver participated in seminars and retreats like one held in 1922, which explored the tenets of internationalism that had sprung up after the first World War. The YWCA believed that if women used their moral and professional authority, they would reshape the world. Copenhaver participated in the \\\"Conference at the Lake\\\" and after their presentation on relationships with Syrian and Japanese women, and the Bible study, she directed a play, which she had written with her mother. The basic theme was modernity and the enlightenment of women who could draw upon the past, their spirituality and their ability to work and use their intelligence. In 1923 she switched to industrial community activism then in 1925 became the National Industrial Secretary working out of New York. Her political beliefs became increasingly radical during her many visits to factories to counsel and organize working women. During one of these visits in 1928, she made a side trip to her family and met the writer Sherwood Anderson. Coperhaver returned to New York and enrolled in a master's program at Columbia University. She completed the degree in political economy in 1933 and later that same year, married Anderson.\\nCoperhaver-Anderson continued working as a labor organizer. In 1937, she was promoted to head the Industrial Programs of the YWCA. Her work entailed investigating working and educational conditions, submitting reports for action and providing support for labor unions. She continued to travel with her job and the couple also traveled to visit Sherwood's artist and writer friends all over the world. On one of these trips, taken in 1941 after the death of her mother, the couple were in route to Valparaiso, Chile aboard the \\\"SS Santa Lucia\\\". Sherwood became ill with peritonitis, the couple disembarked, but he died at a hospital in Colón, Panama. After bringing him back to Virginia for burial, Anderson returned to work. She continued as the executive of the Y's industrial division and resumed her travels, going across the country, until the Industrial Department was phased out soon after the end of World War II. When that happened, in 1947, Anderson took a two-year assignment with the international YWCA in Italy to help with relief efforts. At the end of her assignment, she returned from Italy and was terminated by the Y due to their elimination of women's employment programs. She spent most of 1950 organizing Sherwood's papers and preparing them for donation to the Newberry Library in Chicago. The following year, she was rehired by the YWCA working on their United Community Defense Services, which was a program aimed at providing \\\"health, welfare, and recreation services\\\" to communities supporting the defense industry. She remained with the YWCA until her retirement in 1961.\\nAfter her retirement, she served as Sherwood's literary executor as well as the overseer of his property at \\\"Ripshin\\\" and her mother's legacy at \\\"Rosemont\\\". Anderson died on September 12, 1985 in Marion, Virginia. Copenhaver's papers are located at Smith College in Northampton, Massachusetts. Many of the records can be found on-line as part of the \\\"Five College Finding Aids Access Project\\\", which was funded by the Andrew W. Mellon Foundation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695217\",\"title\":\"Perspective-n-Point\",\"body\":\"\\nPerspective-n-Point\\n\\nPerspective-\\\"n\\\"-Point is the problem of estimating the pose of a calibrated camera given a set of 3D points in the world and their corresponding 2D projections in the image. The camera pose consists of 6 degrees-of-freedom (DOF) which are made up of the rotation (roll, pitch, and yaw) and 3D translation of the camera with respect to the world. This problem originates from camera calibration and has many applications in computer vision and other areas, including 3D pose estimation, robotics and augmented reality. A commonly used solution to the problem exists for called P3P, and many solutions are available for the general case of . Implementations of these solutions are also available in open source software.\\nProblem Specification.\\nDefinition.\\nGiven a set of 3D points in a world reference frame and their corresponding 2D image projections as well as the calibrated intrinsic camera parameters, determine the 6 DOF pose of the camera in the form of its rotation and translation with respect to the world. This follows the perspective project model for cameras:\\nwhere formula_2 is the homogeneous world point, formula_3 is the corresponding homogeneous image point, formula_4 is the matrix of intrinsic camera parameters, (where formula_5 and formula_6 are the scaled focal lengths, formula_7 is the skew parameter which is sometimes assumed to be 0, and formula_8 is the principal point), formula_9 is a scale factor for the image point, and formula_10 and formula_11 are the desired 3D rotation and 3D translation of the camera (extrinsic parameters) that are being calculated. This leads to the following equation for the model:\\nAssumptions and Data Characteristics.\\nThere are a few preliminary aspects of the problem that are common to all solutions of P\\\"n\\\"P. The assumption made in most solutions is that the camera is already calibrated. Thus, its intrinsic properties are already known, such as the focal length, principal image point, skew parameter, and other parameters. Some methods, such as UP\\\"n\\\"P. or the Direct Linear Transform (DLT) applied to the projection model, are exceptions to this assumption as they estimate these intrinsic parameters as well as the extrinsic parameters which make up the pose of the camera that the original P\\\"n\\\"P problem is trying to find.\\nFor each solution to PnP, the chosen point correspondences cannot be coplanar. In addition, P\\\"n\\\"P can have multiple solutions, and choosing a particular solution would require post-processing of the solution set. Furthermore, using more point correspondences can reduce the impact of noisy data when solving P\\\"n\\\"P. RANSAC is also commonly used with a P\\\"n\\\"P method to make the solution robust to outliers in the set of point correspondences. Most methods, however, assume that the data is noise-free.\\nMethods.\\nThis following section describes two common methods that can be used to solve the P\\\"n\\\"P problem that are also readily available in open source software and how RANSAC can be used to deal with outliers in the data set.\\nP3P.\\nWhen , the P\\\"n\\\"P problem is in its minimal form of P3P and can be solved with three point correspondences. However, with just three point correspondences, P3P yields many solutions, so a fourth correspondence is used in practice to remove ambiguity. The setup for the problem is as follows.\\nLet \\\"P\\\" be the center of projection for the camera, \\\"A\\\", \\\"B\\\", and \\\"C\\\" be 3D world points with corresponding images points \\\"u\\\", \\\"v\\\", and \\\"w\\\". Let \\\"X = |PA|\\\", \\\"Y = |PB|\\\", \\\"Z = |PC|\\\", formula_13, formula_14, formula_15, formula_16, formula_17, formula_18, formula_19, formula_20, formula_21. This forms triangles \\\"PBC\\\", \\\"PAC\\\", and \\\"PAB\\\" from which we obtain the equation system for P3P:\\nIt's common to normalize the image points before solving P3P. Solving the P3P system results in four possible solutions for and . The fourth world point \\\"D\\\" and its corresponding image point \\\"z\\\"are then used to to find the best solution among the four. The algorithm for solving the problem as well as the complete solution classification for it is given in the 2003 \\\"IEEE\\\" Transactions on Pattern Analysis and Machine Intelligence paper by Gao, et. al. An open source implementation of the P3P can be found in OpenCV's \\\"calib3d\\\" module in the \\\"solvePnP\\\" function.\\nEP\\\"n\\\"P.\\nEfficient P\\\"n\\\"P (EP\\\"n\\\"P) is a method developed by Lepetit, et. al in their 2008 International Journal of Computer Vision paper that solves the general problem of P\\\"n\\\"P for . This method is based on the notion that each of the \\\"n\\\" points (which are called reference points) can be expressed as a weighted sum of four virtual control points. Thus, the coordinates of these control points become the unknowns of the problem. It is from these control points that the final pose of the camera is solved for.\\nAs an overview of the process, first note that each of the reference points in the world frame, formula_23, and their corresponding image points, formula_24, are weighted sums of the four controls points, formula_25 and formula_26 respectively, and the weights are normalized per reference point as shown below. All points are expressed in homogeneous form.\\nFrom this, the derivation of the image reference points becomes\\nThe homogeneous image control point has the form formula_31. Rearranging the image reference point equation yields the following two linear equations for each reference point:\\nUsing these two equations for each of the reference points, the system formula_34 can be formed where formula_35. The solution for the control points exists in the null space of and is expressed as \\nwhere formula_37 is the number of null singular values in formula_38 and each formula_39 is the corresponding right singular vector of formula_38. formula_37 can range from 1 to 4. After calculating the initial coefficients formula_42, the Gauss-Newton algorithm is used to refine them. The and matrices that minimize the reprojection error of the world reference points, formula_23, and their corresponding actual image points formula_24, are then calculated.\\nThis solution has formula_45 complexity and works in the general case of P\\\"n\\\"P for both planar and non-planar control points. Open source software implementations of this method can be found in OpenCV's Camera Calibration and 3D Reconstruction module in the \\\"solvePnP\\\" function as well as from the code published by Lepetit, et. al at their website, .\\nUsing RANSAC.\\nP\\\"n\\\"P is prone to errors if there are outliers in the set of point correspondences. Thus, RANSAC can be used in conjunction with existing solutions to make the final solution for the camera pose more robust to outliers. An open source implementation of P\\\"n\\\"P methods with RANSAC can be found in OpenCV's Camera Calibration and 3D Reconstruction module in the \\\"solvePnPRansac\\\" function\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695224\",\"title\":\"2009 Monte Paschi Strade Bianche - Eroica Toscana\",\"body\":\"\\n2009 Monte Paschi Strade Bianche - Eroica Toscana\\n\\nThe 2009 Monte Paschi Strade Bianche - Eroica Toscana was the third edition of the Strade Bianche road bicycle race, held on 7 March 2009 in Tuscany, Italy. The race was 190 km, starting in Gaiole in Chianti and finishing in Siena, and included eight sectors of \\\"strade bianche\\\", totaling 57,2 km of gravel road. Compared to the previous edition of the Strade Bianche, the race was 10 km longer and featured one more sector of \\\"sterrato\\\".\\nSwedish rider Thomas Lövkvist won the race ahead of German Fabian Wegmann and Swiss Martin Elmiger. Linus Gerdemann attacked five kilometres from the finish, but faded on the steep climb to the centre of Siena and was caught and passed by a chase group. Thomas Lövkvist powered up the cobbled street and was the first to enter the Piazza del Campo and crossed the line 4 seconds ahead of Wegmann and Elmiger. Edvald Boasson Hagen was fourth at eight seconds.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695226\",\"title\":\"On the Waves of Happiness\",\"body\":\"\\nOn the Waves of Happiness\\n\\nOn the Waves of Happiness (Romanian:Pe valurile fericirii) is a 1920 Romanian silent film directed by Dolly A. Szigethy and starring Lya De Putti, Maria Filotti and Ian Manolescu.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695244\",\"title\":\"Ooho\",\"body\":\"\\nOoho\\n\\nOoho may refeer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695248\",\"title\":\"Batang Pinoy\",\"body\":\"\\nBatang Pinoy\\n\\nThe Philippine Youth Games – Batang Pinoy or simply Batang Pinoy (lit. Filipino Youth) is the national youth sports competition of the Philippines for athletes under 15 years old. Unlike the Palarong Pambansa a competition for student athletes, the Batang Pinoy also includes the out-of-school youth.\\nHistory.\\nThe Batang Pinoy was established through the Executive Order No. 44 which was signed by then President Joseph Estrada in December 2, 1988. The first edition was held in Bacolod in 1999. From then, the games were held annually with Santa Cruz, Laguna (2000), Bacolod (2001) Puerto Princesa (2002) hosting the next three editions. The 2002 and prior editions, were for athletes of 12 years and below.\\nIn 2003, the Philippine Sports Commission decided to put the competition, along with its other national competitions, on hold so the sports body could reallocate funds to the national teams' stint at the Southeast Asian Games. The Batang Pinoy was held again in 2011, and is held annually since then.\\nEditions.\\nHost of the Batang Pinoy National Championships.\\n(*) Hosting of the originally 2013 edition postponed to early 2014. A second Batang Pinoy was hosted in the same year.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695256\",\"title\":\"Streptomyces coelicoflavus\",\"body\":\"\\nStreptomyces coelicoflavus\\n\\nStreptomyces coelicoflavus is a bacterium species from the genus of \\\"Streptomyces\\\". \\\"Streptomyces coelicoflavus\\\" produces acarviosin-containing oligosaccharides.\\nReferences.\\n \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695271\",\"title\":\"Soldiers of the Emperor\",\"body\":\"\\nSoldiers of the Emperor\\n\\nSoldiers of the Emperor (Hungarian:A Császár katonái) is a 1918 Hungarian silent film directed by Béla Balogh and starring Lya De Putti, Sándor Virányi and Aladár Ihász.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695290\",\"title\":\"Promoter activity\",\"body\":\"\\nPromoter activity\\n\\nPromoter activity is a term that encompasses several meanings around the process of gene expression from regulatory sequences —promoters and enhancers. Gene expression has been commonly characterized as a measure of how much, how fast, when and where this process happens. Promoters and enhancers are required for controlling where and when an specific genes is transcribed.\\nTraditionally the measure of gene products (i.e. mRNA, proteins, etc.) has been the major approach of measure promoter activity. However, this method confront with two issues: the stochastic nature of the gene expression and the lack of mechanistic interpretation of the thermodynamical process involve in the promoter activation.\\nThe actual developments in metabolomics product of developments of next-generation sequencing technologies and molecular structural analysis have enable the development of more accurate models of the process of promoter activation (e.g. the sigma structure of the polymerase holoenzyme domains) and a better understanding of the complexities of the regulatory factors involved.\\nPromoter binding.\\nThe process of binding is central in determining the \\\"strength\\\" of promoters, that is the relative estimation of how \\\"well\\\" a promoter perform the expression of a gene under specific circumstances. Brewster et al., using a simple thermodynamical model based on the postulate that transcriptional activity is proportional to the probability of finding the RNA polymerase bound at the promoter, obtained predictions of the scaling of the RNA polymerase binding energy. This models support the relationship between the probability of binding and the output of gene expression\\nMathematical representation of promoter binding.\\nThe problem of gene regulation could be represented mathematically as the probability of n molecules — RNAP, activators, repressors and inducers — are bound to a target regions.\\nTo compute the probability of bound, it is needed to sum the Boltzman weights over all possible states of formula_1 polymerase molecules\\non DNA. Here in this deduction formula_1 is the effective number of RNAP molecules available for binding to the promoter.\\nThis approach is based in statistical thermodynamics of two possible microscopic outcomes: \\nThe statistical weigh of promoter unoccupied \\\"Z(P)\\\" is defined:\\nformula_3\\nWhere the first term is the combinatorial result of taken formula_1 polymerase of formula_5 non-specific sites available, and the second term are the boltzmann weights, where formula_6 is the energy that represents the average binding energy of RNA polymerase to the genomic background (non-specific sites).\\nThen, the total statistical weight formula_7, can be written as the sum of the formula_8 state and the RNA polymerase on promoter state:\\nformula_9\\nWhere formula_10 in the formula_11 state is the binding energy for RNA polymerase on the promoter (where the s stands for specific site).\\nFinally, to find the probability of a RNA polymerase to binding (formula_12 ) to an specific promoter, we divide formula_13 by formula_14 which produces:\\nformula_15\\nWhere, formula_16\\nAn important result of this model is that any transcription factor, regulator or perturbation could be introduced as a term multiplying formula_1 in the probability of binding equation. This term for any transcriptional factor (here called factor regulators) modify the probability of binding to:\\nformula_18\\nWhere formula_19 is the term for transcriptional factors, and it has the value of formula_20 for increase of formula_21 for decrease of the number of RNA polymerase available to bind.\\nThis result has an important significance to represent mathematically all the possible configurations of transcriptional factor by derive different models to estimate formula_19 (for further developments, see also ).\\nEukaryotes promoter structure.\\nThe process of activation and binding in eukaryotes is different from bacteria in the way that specific DNA elements bind the factors for a functional pre-initiation complex. In bacteria there is a single polymerase, that contain catalytic subunits and a single regulatory subunits known as sigma, which transcribe for different type of genes.\\nIn eukaryotes, the transcription is performed by three different RNA polymerase, RNA pol I for ribosomal RNAs (rRNAs), RNA polymerase II for messenger RNAs (mRNAs) and some small regulatory RNAs, and the RNA polymeerase III for small RNAs such as transfer RNAs (tRNAs). The process of positioning of the RNA polymerase II and the transcriptional machinery require the recognition of a region know as \\\"core promoter\\\". The elements that could be found in the core promoter include the TATA element, the TFIIB recognition element (BRE), the initiatior (Inr), and the downstream core promoter element (DPE). Promoters in eukaryotes contain one or more of these core promotes elements (but any of them are absolutely essential for promoter function), these elements are binding sites for subunits of the transcriptional machinery and are involve in the initiation of the transcription, but also they have some specific enhancer functions. In addition, the promoter activity in eukaryotes include some complexities in the way of how they integrate signals from distal factors with the core promoter.\\nEvolutionary processes.\\nUnlike in protein coding regions, where the assumption of sequence conservation of functionally homologous genes have been frequently proved, there is no a clear relationship of conservation between sequences and their functions for regulatory regions. The transcriptional promoters regions are under less stringent selection, then have a higher substitutions rates, allowing transcription factor binding sites to be replaced easily be new ones arising from random mutations. Notwithstanding the sequence changes, mainly the functions of regulatory sequences remain conserved.\\nIn recents years with the increase of availability of genome sequences, phylogenetic footprinting open the possibitlity to identify cis-elements, and then study their evolution processes. In this sense, Raijman et al., Dermitzakis et al. have developed techniques for analyzing evolutionary processes in transcription factor regions in Saccharomyces species promoters and mammalian regualatory networks respectively.\\nThe basis for many of these evolutionary changes in nature are probably related with events within the cis-regulatory regions involve in gene expression. The impact of variation in regulatory regions is important for disease risk due their impact in the gene expression level. Furthermore, perturbations in the binding properties of proteins encoded by regulatory genes have been linked with phenotypes effects such as, duplicated structures, homeotic transformations and novel morphologies.\\nMeasure of promoter activity.\\nThe measure of the promoter activity has a broad meaning. The promoter activity could be measured for different situations or research questions, such as:\\nMethods to study promoter activity commonly are based in the expression of a reporter gene from the promoter of the gene of interest. Mutations and deletions are made in a promoter region, and their changes on couple expression of the reporter gene are measured.\\nThe most important reporter genes are the fluorescence proteins as GFP. These reporters allow to measure promoter activation by increasing fluorescent signals, and deactivation by decrease in the rate of fluorescence.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695349\",\"title\":\"Gypsy Blood (1920 film)\",\"body\":\"\\nGypsy Blood (1920 film)\\n\\nGypsy Blood (German:Zigeunerblut) is a 1920 German silent film directed by Karl Otto Krause and starring Lya De Putti, Carl Fenz and Paul Hansen. It is based on Georges Bizet's \\\"Carmen\\\" and shout not be confused with the 1918 German silent \\\"Carmen\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695353\",\"title\":\"Carlton E. Lemke\",\"body\":\"\\nCarlton E. Lemke\\n\\nCarlton Edward Lemke (October 11, 1920 - April 12, 2004) was an American mathematician.\\nLemke received his bachelor's degree in 1949 at the University of Buffalo and his PhD (Extremal problem in Linear Inequalities) in 1953 at Carnegie Mellon University (then Carnegie Institute of Technology). In 1952-1954 he was instructor at the Carnegie Institute of Technology and in 1954-55 at the Knolls Atomic Power Laboratory of General Electric. In 1955-56 he was an engineer at the Radio Corporation of America in New Jersey. From 1956 he was assistant professor and later professor at the Rensselaer Polytechnic Institute. Since 1967, he was there Ford Foundation Professor of Mathematics.\\nHis research is in Algebra, Mathematical Programming, Operations Research, and Statistics. In 1954 Lemke developed the dual simplex method, independently from E. M. L. Beale.\\nHe is also known for his contribution to game theory. In 1964 Lemke (with J. T. Howson) constructed an algorithm for finding Nash equilibria the case of finite two-person games. For this work Lemke received in 1978 the John von Neumann Theory Prize.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695355\",\"title\":\"Iron Mountain Trail\",\"body\":\"\\nIron Mountain Trail\\n\\nIron Mountain Trail is a 1953 American Western film directed by William Witney and written by Gerald Geraghty. The film stars Rex Allen, Slim Pickens, Grant Withers, Nan Leslie, Roy Barcroft and Forrest Taylor. The film was released on May 8, 1953, by Republic Pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695356\",\"title\":\"Talwandi Nepalan\",\"body\":\"\\nTalwandi Nepalan\\n\\nTalwandi Nepalan is a village in the Firozpur district of Punjab, India. It is located in the Zira tehsil.\\nDemographics.\\nAccording to the 2011 census of India, Talwandi Nepalan has 526 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 68.69%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695360\",\"title\":\"Rasulpur, Zira\",\"body\":\"\\nRasulpur, Zira\\n\\nRasulpur is a village in the Firozpur district of Punjab, India. It is located in the Zira tehsil.\\nDemographics.\\nAccording to the 2011 census of India, Rasulpur has 329 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 66.22%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695365\",\"title\":\"Sudan, Zira\",\"body\":\"\\nSudan, Zira\\n\\nSudan is a village in the Firozpur district of Punjab, India. It is located in the Zira tehsil.\\nDemographics.\\nAccording to the 2011 census of India, Sudan has 289 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 62.14%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48695372\",\"title\":\"Charles Dundas Lawrie\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121074\",\"title\":\"Paul Wittek\",\"body\":\"\\nPaul Wittek\\n\\nPaul Wittek (11 January 1894, Baden bei Wien — 13 June 1978, Eastcote, Middlesex) was an Orientalist and historian from Austria. His 1938 thesis on the rise of the Ottoman Empire, known as the Ghazi thesis, argues that the Ottoman's \\\"raison d'être\\\" was the expansion of Islam. During the 1980s, his theory was the most influential and dominant explanation of the formation of the Ottoman Empire.\\nBiography.\\nWittek was conscripted at the outbreak of World War I as a reserve officer to an artillery regiment. In October 1914, he suffered a head wound in Galicia and was taken to Vienna to recover. Subsequently, he served first on the Isonzo and in 1917 was drafted as a military adviser to the Ottoman Empire, where he was stationed in Istanbul and Syria until the war ended. During this time Wittek learned Ottoman Turkish and acquired the patronage of Johann Heinrich Mordtmann, the former German consul in Istanbul. After the war ended, Wittek returned to Vienna and continued his studies of ancient history, which he had already begun before the war. In 1920 he obtained his doctorate with a study of the oldest Roman social and constitutional history.\\nWittek was in Vienna at the emergence of the fledgling discipline of Ottoman studies. He was co-editor and author of the first scholarly journal in this field called \\\"Messages to Ottoman history\\\", which was published from 1921 till 1926. For his livelihood Wittek worked as a journalist for the Austrian Rundschau. From 1924, he worked for the German Archaeological Institute, where he focused on the early Ottoman epigraphy. Together with Turkish historians, he managed to prevent the sale of the Ottoman archives to Bulgaria as scrap paper.\\nAfter the rise of Nazism in 1934 Wittek moved to Belgium, where he worked at the Institute for Byzantine Studies in Brussels with Henri Gregoire. After the German attack on Belgium Wittek fled in a small boat to England, where he was interned as an enemy alien. By supporting British Orientalists he was finally released and found a job at the University of London. After the war he returned to his family, who had remained in Belgium. In 1948 he returned to London and took over the newly created Chair of Turkish at the School of Oriental and African Studies (SOAS), where he remained until his retirement in 1961.\\nWittek, who was close to the George circle, has published little, but has become very powerfully in his discipline. His only books, \\\"The Principality Menteşe\\\" and \\\"The Rise of the Ottoman Empire\\\" appeared in the 1930s. In the latter Wittek formulated his Ghazi thesis, according to which the ideology of sectarian struggle was the major cohesive factor in the formative phase of the Ottoman Empire. The Ghazi-thesis was to Rudi Paul Lindner nomad thesis in the 1980s, the prevailing view of the emergence of the Ottoman Empire.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121075\",\"title\":\"Glencoe Museum\",\"body\":\"\\nGlencoe Museum\\n\\nGlencoe Museum is located in a Victorian house in west downtown Radford, Virginia. The house was built in 1870 in the 19th century Victorian style, specifically Second Empire, and was the postbellum home of Confederate Brigadier General Gabriel C. Wharton. It is a large, two-story, five bay, brick dwelling, and originally had quite extensive grounds. The original house had a barn, chicken coop, smoke house, and an ice house.\\nThe name Glencoe is thought to be inspired by Anne Wharton’s ancestry. Her family was originally from Scotland. The house didn’t appear on Radford’s tax records until 1876; it took a very long time to build a house of its size and grandeur in the 1800s. The house was kept in the family till 1996 when, after being deserted for 30 years, the house was given to the city of Radford. The house and grounds were donated by the Kollmorgen Motion Technology Group.\\nThe house features some Victorian period rooms and displays about Radford's history, including Native American artifacts, early settlers, Mary Draper Ingles, local industries, railroads, river transportation, educational institutions and local sports. There is also an art gallery with changing exhibits of the art and works of contemporary Appalachian artists.\\nGlencoe was listed on the National Register of Historic Places in 2000.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121100\",\"title\":\"Garshom Awards\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121116\",\"title\":\"Fred Kohler, Jr.\",\"body\":\"\\nFred Kohler, Jr.\\n\\nFred Kohler, Jr. was an American actor who performed in a number of Westerns such as \\\"The Pecos Kid\\\" and \\\"Toll of the Desert\\\".\\nKohler's father was actor Fred Kohler.\\nKohler and his father appeared twice in the same film. In RKO's \\\"Lawless Valley\\\", they played outlaws who were father and son and in one scene, Fred Kohler, Jr. says to his father's character \\\"Aw, that's crazy!,\\\" and Fred Sr. responds \\\"Careful, son, you're talkin' to your dad, ya know!\\\" \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121142\",\"title\":\"Else Kastner-Michalitschke\",\"body\":\"\\nElse Kastner-Michalitschke\\n\\nElse Kastner-Michalitschke (28 April 1868 – 2 January 1939) was an Austrian writer.\\nShe was born in Rokytnice v Orlických horách, in what was then Austria-Hungary and today the Czech Republic. She studied to become a teacher in Prague and lived in Vienna from 1892. She was married to Eduard Fedor Kastner, and later to Carl B. Braum. She was co-founder of the literary magazine \\\"Böhmens deutsche Poesie und Kunst\\\" and contributed to the magazine \\\"Wir leben\\\". She also published several collections of poetry. She received numerous awards in recognition of her artistic accomplishments.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121143\",\"title\":\"Rabiya Javeri Agha\",\"body\":\"\\nRabiya Javeri Agha\\n\\nRabiya Javeri Agha (born Rabiya Adila Javeri on December 2, 1963) is a Grade 21 Officer of the Pakistan Administrative Service. One of the senior most women in the civil service, she joined the Government of Pakistan in 1985. She is currently posted as Secretary Trade Development Authority of Pakistan.\\nEarly life.\\nRabiya is the daughter of well known jeweller Hasan Javeri of Jamnagar State and Ayesha Rafique Javeri (Family of Sir Syed Ahmed Khan, Allahbad). She is also the sister of well known photographer Tapu Javeri and artist Zehra Laila Javeri. Her education was at Convent of Jesus and Mary and Karachi Grammar School. She Graduated in Politics and Literature from Mount Holyoke College. she is married to senior civil servant Agha Jan Akhtar, Chairman Port Qasim Authority.\\nCareer.\\nBefore joining the bureaucracy, Rabiya Javeri Agha worked as a journalist for the Dawn newspaper. She has written over 300 articles on social, political and cultural issues. She has also authored and published research papers on Sufism and on the Afghan political and refugee crisis.\\nHer duties with the Trade Authority include marketing Pakistan's potential as a trade and cultural partner. She travels throughout Pakistan and Europe to promote goods, like textiles and fashion, as well as the resources Pakistan has available.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121161\",\"title\":\"Lagos by-election, 1945\",\"body\":\"\\nLagos by-election, 1945\\n\\nA by-election was held for the Lagos seat in the Legislative Council of Nigeria in December 1945 to replace Jibril Martin of the Nigerian Youth Movement (NYM). It was won by Abubakar Olorun-Nimbe of the Nigerian National Democratic Party (NNDP).\\nCampaign.\\nThe NNDP and the Nigerian Union of Young Democrats put forward Olorun-Nimbe, vice president of the NNDP, as a joint candidate. A doctor, he had studied medicine at the University of Glasgow and ran a private practice in Lagos. He was well-known in the town due to his role as a member of Lagos Town Council. His opponent was Oluwole Ayodele Alakija of the NYM. Alakija was a barrister who had studied at Jesus College, Oxford and was vice-president of the party's Port Harcourt branch.\\nDespite the NYM's success in the 1938 general elections, when it had won all three seats in Lagos, its popularity in the town had diminished, partly due to the internal splits caused by the row over the party's candidate for the 1941 by-election. Separately, the alliance of the NNDP with the new National Council of Nigeria and the Cameroons had restored some of its credibility.\\nThe \\\"West African Pilot\\\" called on voters to vote for a \\\"seriously active Moslem\\\", noting the fact that Olorun-Nimbe had twice been to Mecca and that he was a \\\"practical politician with experience\\\". Despite not being a Muslim, Alakija was supported by Chief Imam Y. P. O. Shodeinde.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121167\",\"title\":\"Melville Amasa Scovell\",\"body\":\"\\nMelville Amasa Scovell\\n\\nMelville Amasa Scovell (February 26, 1855 – August 15, 1912) was an American academic from New Jersey. Moving with his family to Champaign, Illinois early in his life, he attended the University of Illinois there, graduating in 1875. He worked at the school for the next seven years, until he was dismissed over a conduct violation. In 1885, he became director of the Kentucky Agricultural Experiment Station at the University of Kentucky. Scovell was named dean of the College of Agriculture, Food, and Environment there shortly before his death.\\nBiography.\\nMelville Amasa Scovell was born February 26, 1855, in Belvidere, New Jersey. Shortly after his birth, his family moved to Jasper County, Illinois. They later moved to Champaign, where his father worked as a principal. Scovell enrolled at the University of Illinois in 1871, studying chemistry. He graduated in 1875 and became an instructor there. From 1876 to 1877, he served as the secretary to university president John Milton Gregory.\\nScovell was named an assistant professor of chemistry in 1877, receiving a Master of science the next year. He was made professor of agricultural chemistry in 1880. He co-founded a method of clarifying cane sugar juice, but was forced to leave the university in 1882 because professors were expected to abstain from profiting from their research. Scovell founded a sugar factory in Champaign, but it was short-lived. He also worked as a special agent of the US Department of Agriculture following his dismissal.\\nIn 1885, Scovell accepted a position at the Kentucky Agricultural Experiment Station at the University of Kentucky in Lexington, a position he held until his death. There, he promoted agriculture in Kentucky, with a focus on Jersey cattle. James Ben Ali Haggin commissioned Scovell to purchase cattle for his Elmendorf Farm. In 1893, Scovell was chairman of the dairy test at the World's Columbian Exposition in Chicago. He was awarded a Ph.D. by the University of Illinois in 1909. In 1911, Scovell was named dead of the University of Kentucky College of Agriculture, Food, and Environment.\\nScovell died at his Lexington home from endocarditis on August 15, 1912. He married Nancy Davis, a member of his university class, in 1880; they had no children. Scovell served terms as president of the American Association of Agricultural Colleges and Experiment Stations and the Association of Official Agricultural Chemists. He was also a member of the Fair Committee of the Kentucky State Fair from 1907, when it moved to Louisville, until his death. He was a member of the Society of Chemical Industry and the American Chemical Society. He is the namesake of Scovell Hall at the University of Kentucky.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121178\",\"title\":\"Karen Mullins\",\"body\":\"\\nKaren Mullins\\n\\nKaren Mullins is an American college softball coach, most recently the long-time head coach of the Connecticut Huskies softball team. She served in that role from 1984 to 2014. She announced her retirement on May 19, 2014.\\nPlaying career.\\nMullins played four years of basketball at Connecticut and also two years of softball. Connecticut was just beginning to establish women's sports during her time as a student in Storrs. \\nCoaching career.\\nAfter completing a master's in sport management at UConn, Mullins became head coach at Nichols College in Massachusetts. She led the team for two seasons, compiling a 10–10 record before moving to E. O. Smith High School as a basketball coach. In 1984, she returned to UConn as head softball coach. Mullins would remain at UConn for thirty one seasons, leading the Huskies to 10 top 2 finishes in the NCAA Northeast rankings, eight appearances in the NCAA Division I Softball Championship, a Women's College World Series berth in 1993, six Big East Conference regular season and seven Big East Conference Softball Tournament championships, and an overall record of 862–626–5. Over 96 percent of her players graduated, nine earned All-American honors, six were named academic All-Americans, and her players earned Big East Player or Pitcher of the Year awards 11 times. She retired as one of the top 50 all-time coaches in career wins. Mullins was one of the longest-tenured coaches at UConn, although many coaches also had remained in Storrs for multiple decades.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121201\",\"title\":\"SAFRA National Service Association\",\"body\":\"\\nSAFRA National Service Association\\n\\nSAFRA National Service Association is an organisation that was formed as a recreation club for servicemen from the Singapore Armed Forces. This was also how SAFRA got its name: \\\"Singapore Armed Forces Recreation Association\\\".\\nHistory.\\nSAFRA was started on 2 July 1972 and was launched by, then Defense Minister of Singapore, Dr Goh Keng Swee.\\nMission.\\nThe SAFRA Mission\\n\\\"To be a dynamic institution effective in bonding NSmen through a network of quality lifestyle clubs, services and activities. We shall be reputed for our good value and innovation.\\\"\\nThis was to reward servicemen for their service to the nation with some relaxation from their hectic lives.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121270\",\"title\":\"Sadolin Sports Hall\",\"body\":\"\\nSadolin Sports Hall\\n\\nThe Sadolin Sports Hall () is a multi-purpose indoor arena complex in Rapla. It was opened in 2010 and is the current home arena of the Estonian Basketball League team Rapla KK. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121350\",\"title\":\"Sanguantang Station\",\"body\":\"\\nSanguantang Station\\n\\nSanguantang Station is an elevated metro station of Line 2 in Ningbo, Zhejiang, China. It situates on the west of the crossing of Ningzhen Road and Minghai Avenue. Construction of the station starts in middle 2012 and opened to service in September 26, 2015. In the construction plan of Ningbo Rail Transit, Sanguantang Station will be a transfer station between Line 2 and Line 5.\\nExits.\\nSanguantang Station has 2 exits.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121402\",\"title\":\"Phi Kappa Pi\",\"body\":\"\\nPhi Kappa Pi\\n\\nPhi Kappa Pi (ΦΚΠ) is a Canadian National Fraternity. Founded on March 22, 1913 as Canada's Only National Fraternity, Phi Kappa Pi has active chapters in Burnaby, Halifax, Toronto, and Montreal, as well as 6 inactive chapters. There are alumni chapters associated with most locations, as well as a National Council.\\nHistory.\\nPhi Kappa Pi Fraternity was founded in 1913, by two previously existing and separate organizations. Sigma Pi, founded in Toronto in 1901 and Alpha Beta Gamma, founded in Montreal in 1905, joined forces to create Canada's first and only national fraternity. The individual organizations' names would then become chapter names.\\nIn 1923, alumni from the Alpha Beta Gamma chapter approached the Phi Kappa Pi National Council about the possibility of expanding to Dalhousie University in Halifax, Nova Scotia. The expansion request was approved and a chapter was founded. The chapter was the first fraternity to locate to Dalhousie's campus, and was named the Dalhousie chapter, until 1959 when it received it's Greek name, Zeta Gamma. The following year, 1924, Alpha Iota was founded at the University of British Columbia, followed by Delta Mu in 1930 at the University of Alberta, Tau Sigma Rho in 1935 at the University of Manitoba, and Alpha Epsilon in 1967 at the University of Waterloo.\\nThe 1970s proved to be a tough decade for Phi Kappa Pi, with 4 chapters being lost. Alpha Iota and Alpha Epsilon both closed down in 1974, followed by Tau Sigma Rho in 1975, and one of the founding chapters, Alpha Beta Gamma in 1976. Alpha Beta Gamma, however, would be reopened in 1990 with the help of alumni from Phi Kappa Pi's then remaining two chapters. Soon after, in 2000, Theta Kappa Omicron opened at the University of Ottawa, and Omega Iota in 2006 at the University of Ontario Institute of Technology.\\nIn September 2008, Simon Fraser University's student body voted 57% in favour of overturning the University's ban of Greek life on campus. The Omega Epsilon chapter opened on the university's campus in 2012. Phi Kappa Pi was the first Greek Life Organization to open on the Simon Fraser campus. \\nChapters.\\nUntil at least 1976, the chapter names are the name of the local from which it was formed, except in the case of Dalhousie.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121545\",\"title\":\"Lulin Station\",\"body\":\"\\nLulin Station\\n\\nLulin Station is an elevated metro station of Line 2 in Ningbo, Zhejiang, China. It situates on Fenghua Road. Construction of the station starts in middle 2012 and opened to service in September 26, 2015.\\nExits.\\nLulin Station has 2 exits.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121588\",\"title\":\"Orthogonal ligand-protein pair\",\"body\":\"\\nOrthogonal ligand-protein pair\\n\\nOrthogonal ligand-protein pairs (also known as re-engineered ligand-receptor interfaces or re-engineered enzyme-substrate interactions) are a protein-ligand binding pair made to be independent of the original binding pair. This is done by taking a mutant protein (naturally occurring or selectively engineered), which is activated by a different ligand (carefully synthesized or selected). The intention here is that the orthogonal ligand will not interact with the original protein. The original protein will also be designed to not interact with the orthogonal ligand in certain cases. An example of orthogonal ligand-receptor interfaces are RASSL and DREADD. They are G protein-coupled receptors that are activated by synthesized ligands that wouldn't normally exist in the cell, such as the anti-psychotic Clozapine, allowing researchers to control the interaction externally and independent of internal activation.\\nApproaches and Designs.\\nProtein Engineering Approach.\\nThe protein engineering approach involves synthesizing a new ligand and directed mutation of the protein's ligand-binding site. In this approach one has to be careful to only change the ligand specificity without changing the other actions of the protein.\\nSteric Modification.\\nThe steric modification design can be summarized into 3 changes to the ligand-protein pair: \\nReversal of hydrogen bonds or charge-charge interactions.\\nAnother way to design an orthogonal protein is to switch the position of the hydrogen bond acceptors and donors. For example, if the ligand is a hydrogen bond donor and the protein a hydrogen bond acceptor, switch the ligand to the hydrogen bond acceptor and the protein to the donor. The reversal of charged interactions is similar, but it involves switching the position of the positive charge and the negative charge on the protein and ligand.\\nSynthetic Chemistry Approach.\\nThe synthetic chemist's approach is to take an already existing mutant form of the protein that binds the original ligand weakly, and synthesize a new ligand for which the mutant protein has a strong affinity. The drawback of this approach is the protein still interacts weakly with the natural ligand at low synthetic ligand concentrations.\\nConfirmed Applications.\\nAgriculture.\\nInduced drought resistance.\\nPark et al. created an orthogonal receptor-ligand interface between and . normally binds to abscisic acid which together then bind and inactivate to PP2C as a drought stress response, which stops PP2C from deactivating . This causes a cascade that leads to the activation of the and closing of the leaf guard cells and stomata. The result is less water loss by the plant. The natural response by the plant using abscisic acid to bind PYR1 in drought conditions is not strong enough and is activated too late to significantly hinder crop yield loss. Abscisic acid is also currently too expensive to synthesize to be used as a spray to control drought response artificially on a mass scale. The ability to control this externally by spraying the PYR1MANDI (orthogonal receptor) with mandipropamid (orthogonal ligand and fungicide) has the potential to reduce crop yield loss during droughts in plants with these engineered receptors, and has been confirmed to work in canola.\\nMedicine.\\nHormonal Pathway Control.\\nDesigning ligands for mutant receptors that are unresponsive to the natural ligand could prove to be an effective way to treat disease. TRβ histidine 435 is a T3 insensitive mutant that plays a role in human pituitary cancer and RTH. Hassan and Koh showed QH2 (orthogonal ligand) was able to allosterically activate the mutant TRβ nuclear hormone receptors that had lost their responsiveness to endogenous T3 (natural mutants) but retained their DNA binding activity.\\nResearch.\\nGene Expression:.\\nMixing and matching the ligand-binding domains and DNA-binding domains of different hormone receptors can be used as an inducible expression mechanism to study the action of any gene with a hormone response element in its promoter. Selectively altering the ligand-binding domain to make it orthogonal to the natural ligand-receptor interface, as well as the making the DNA-binding domain and hormone response element orthogonal, would give a researcher precise control of a gene's transcription in order to study a gene's action.\\nSignal Transduction:.\\nStudying signal transduction pathways and attempting to identify the action of proteins involved in these pathways is difficult due to the abundance and complexity of interactions, families of proteins with the same or similar action, and the relative a lack of selectivity for substrates (a good example of which are protein kinases). A method has been developed to use a radioactively labeled ATP orthogonal analog with an orthogonal kinase that uses the ATP analog to phosphorylate its substrates, allowing for identification of its targets within the pathway by the radioactive label that it will add the target. Variations on this approach can be used to identify the function of signal transduction proteins whose function remains undetermined.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48121736\",\"title\":\"Magdalena Szczepańska\",\"body\":\"\\nMagdalena Szczepańska\\n\\nMagdalena Szczepańska (born January 25, 1980 in Zielona Góra) is a retired Polish heptathlete. She won a silver medal at the 2004 European Combined Events Cup in Hengelo, Netherlands, and then represented her nation Poland in heptathlon at the Olympics in Athens a few months later, finishing in twenty-first place. Szczepańska trained under the tutelage of head coach Jerzy Skucha for the national team, while competing at AZS AWFiS Gdańsk.\\nSzczepańska qualified for the Polish squad in the women's heptathlon at the 2004 Summer Olympics in Athens. Two months before the Games, she eclipsed the IAAF Olympic \\\"B\\\" standard and her personal record of 6115 points to place second behind gold medalist Yuliya Ignatkina of Russia at the European Cup in Hengelo, Netherlands, resulting to her official selection to the Polish Olympic team in track and field. Szczepańska started the competition in a tie for second place with U.S. heptathlete Tiffany Lott-Hogan in the 100 metres hurdles to command the medal position, but a below-par performance in the high jump slipped her out of contention for the medal to the bottom of the leaderboard, achieving only 1.67 against leader Carolina Klüft's height by just twenty-four centimetres. Szczepańska could only manage to produce substantial events for the remaining events of the competition, until she finally landed herself to twenty-first place with a total score of 6012 points.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122100\",\"title\":\"TTÜ Sports Hall\",\"body\":\"\\nTTÜ Sports Hall\\n\\nThe TTÜ Sports Hall () is a multi-purpose indoor arena complex in Mustamäe, Tallinn. It was opened in 1975 and renovated in 2001. It is the current home arena of the Estonian Basketball League team TTÜ KK.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122179\",\"title\":\"Maracaibo (disambiguation)\",\"body\":\"\\nMaracaibo (disambiguation)\\n\\nMaracaibo is a city and municipality in the Zulia State, Venezuela.\\nMaracaibo can also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122208\",\"title\":\"Speaker of the United States House of Representatives election, October 2015\",\"body\":\"\\nSpeaker of the United States House of Representatives election, October 2015\\n\\nAn election for the Speaker of the U.S. House of Representatives took place on October 29, 2015, during the 114th U.S. Congress. The election was necessitated by the announcement of Speaker John Boehner's resignation, set for October 30. The Speaker of the House follows the Vice President in the line of succession to the presidency of the United States in accordance with the Presidential Succession Act.\\nDue to friction within the Republican Party caucus, Boehner decided to step down as speaker and resign his seat in Congress. He scheduled a Republican caucus non-binding vote for speaker on October 8, and a full floor vote on October 29. Kevin McCarthy, the House Majority Leader and second-in-command to the Speaker, was initially viewed as the favorite to win the Speakership. However, due to the opposition of the Freedom Caucus, McCarthy dropped out of the race on October 8, and the caucus vote was postponed.\\nJason Chaffetz initially declared his candidacy to challenge McCarthy, and Bill Flores declared his candidacy after McCarthy withdrew, but both dropped out later to express their support for Paul Ryan, who entered the race after being widely viewed as a potential frontrunner. Daniel Webster of the Republican Party and former Speaker Nancy Pelosi of the Democratic Party were also declared candidates. Ryan won the rescheduled caucus vote on October 28, and was elected Speaker of the House, receiving 54.6% of the House vote, on October 29.\\nBackground.\\nProcess and conventions.\\nThe Speaker of the United States House of Representatives is the presiding officer of the United States House of Representatives, and is second in the United States presidential line of succession, after the Vice President and ahead of the President pro tempore of the United States Senate. Though the Constitution does not require that the Speaker be an elected member of the House of Representatives, every Speaker to date has been elected from House membership.\\nThe 435 members of the House of Representatives elect a Speaker by majority rule at the beginning of each session of the United States Congress, who serves until the end of the Congress. Typically, the election is a formality, as the majority party's members vote as a bloc for their party's previously-chosen Speaker-designate (such as the speaker, majority leader, or minority leader from the previous term). Open elections are uncommon but have occurred. The last Speaker election to require multiple ballots occurred in 1923.\\nSpeakership and resignation of John Boehner.\\nJohn Boehner, a member of the Republican Party from Ohio, served as the Majority Leader of the United States House of Representatives from February 2006 until January 2007. As the Democratic Party assumed control of the House following the 2006 elections, Boehner served as Minority Leader from January 2007 until January 2011. When Republicans reassumed control of the House of Representatives in January 2011, Boehner was elected as Speaker, with the votes of all 241 of his fellow Republicans. In 2014, some House Republicans reached out to Ben Carson about his interest in becoming Speaker should they be able to oust Boehner; Carson declined, citing his impending candidacy for president. Boehner's Republican opponents formed a congressional caucus, called the Freedom Caucus, in January 2015 to focus their opposition. Though Boehner was reelected as Speaker at the beginning of the 114th United States Congress that month, 25 conservative members of the Republican caucus did not vote for him. Daniel Webster, a Republican from Florida, received 12 votes.\\nThroughout 2015, Boehner and the Freedom Caucus remained at odds. Boehner stripped his opponents of leadership posts and other perks, while the American Action Network, a group allied with Boehner, aired television ads against Freedom Caucus members in their home districts. Meanwhile, the Freedom Caucus opposing Boehner's plans, forcing him to rely on Democratic votes to pass bills. Needing to pass a federal budget for the 2016 fiscal year beginning October 1, the Freedom Caucus, now consisting of approximately 40 conservative Republicans affiliated with the Tea Party movement, threatened to block a resolution from passing unless it would defund Planned Parenthood and to initiate a vote to vacate the speakership if Boehner did not support their demands. The caucus sought the following promises: (1) the decentralization of the House Steering Committee, so that the Speaker and House Majority Leader are not solely in charge of committee assignments, (2) not supporting an increase in the U.S. debt ceiling without entitlement reform, (3) willingness to impeach John Koskinen, the Commissioner of Internal Revenue, and (4) passing spending bills approved by the caucus rather than a continuing resolution favored by Democrats in the United States Senate.\\nOn July 28, 2015, Mark Meadows, a member of the Freedom Caucus from North Carolina, filed a motion to vacate the speakership, only the second time the motion had been filed. The next day, Boehner referred to the motion as \\\"no big deal.\\\" However, following continued pressure from the Freedom Caucus, and to avoid the vacation of his speakership, Boehner announced on September 25 that he would resign the Speakership and retire from Congress effective October 30. Sources from his office indicated he chose to resign due to the increasing discord within the Republican caucus so that he could manage passage of a continuing resolution to fund the government and avoid a government shutdown.\\nCandidates.\\nOn September 28, Kevin McCarthy of California, the House Majority Leader, and Webster announced that they would run for Speaker of the House. McCarthy was considered the presumptive favorite in the race. Jason Chaffetz, a Republican from Utah and the Chairman of the House Oversight and Government Reform Committee, announced his candidacy on October 4, claiming that McCarthy did not have the votes to win the election. Several Republicans urged Paul Ryan of Wisconsin, the running mate of Mitt Romney in the 2012 presidential election, to run for Speaker, but he declined, saying he is a \\\"policy guy\\\" with a preference to focus on his role as Chairman of the House Ways and Means Committee.\\nDemocrat Nancy Pelosi, who served as the Speaker from January 2007 through January 2011, asked her Democratic colleagues for their vote in the election. Steny Hoyer, the House Minority Whip, said that he expects that the \\\"overwhelming majority\\\" of Democrats will vote for Pelosi. He said that if a Republican cannot get the votes needed, Democrats may consider their options.\\nOn October 7, the day before the Republican caucus scheduled a non-binding vote for speaker, Ryan and former Vice President Dick Cheney endorsed McCarthy, as did 11 of the 13 House Republicans from Pennsylvania. The Freedom Caucus decided to endorse Webster in the race. Other Republicans said they would vote against McCarthy, including Thomas Massie of Kentucky, who called McCarthy \\\"absolutely not an option\\\" because of his previous role as Boehner's \\\"right-hand man\\\". Also, Walter B. Jones, Jr. of North Carolina sent a letter to the Republican Conference Chairwoman Cathy McMorris Rodgers stating that any candidates for a leadership position with \\\"misdeeds\\\" should withdraw from the race. Jones has stated that his comment did not specifically refer to McCarthy, but it was widely seen as referring to rumors that McCarthy had been committing an extramarital affair with a fellow Representative, a rumor that both have denied; the basis for such an allegation and interpretation is unclear.\\nCiting opposition from within the Republican Party, as well as fallout from controversial comments he made about the United States House Select Committee on Benghazi, McCarthy dropped out of the race on October 8. Following McCarthy's departure from the race, Republicans renewed their efforts to recruit Ryan as a candidate. Boehner personally called Ryan twice to ask him to run, and Chaffetz said that he would not run against Ryan if he chose to enter the race. Ryan also received calls from Mitt Romney and Trey Gowdy, among others, encouraging him to run for Speaker. Additional Ryan endorsements came from Wisconsin Senator Ron Johnson, 2016 Republican presidential frontrunner Donald Trump, and House Majority Whip Steve Scalise from Louisiana. On October 9, close aides of Ryan confirmed that Ryan was reconsidering the possibility of a run.\\nA possible Ryan candidacy received support from the same Freedom Caucus that opposed Boehner and McCarthy, as Meadows said on October 11 that Ryan running would \\\"definitely change the equation,\\\" and Chairman Jim Jordan described Ryan as \\\"a good man\\\" and stated that the Freedom Caucus would view a Ryan run \\\"favorably.\\\"\\nOthers who expressed their interest in running included Texas representatives Bill Flores and Michael McCaul, Georgia representative Lynn Westmoreland, Montana representative Ryan Zinke, and California representative and former Oversight Committee Chairman Darrell Issa. However, several candidates have made clear that they would only run if Ryan chooses not to, including Issa, McCaul, and Minnesota representative John Kline. On October 12, Flores confirmed that he would run for Speaker. He stated that he would run only if Ryan stayed out of the contest.\\nRyan held a closed-door meeting with the Republican Caucus on October 20, where he explained that he would run for Speaker if he could be guaranteed an overwhelming majority of the Republican caucus would support him. Specifically, Ryan requested an increased threshold for the political maneuver of vacating the Speakership, stated that he would not lessen the amount of time he spends with his family, and requested an official endorsement from the Freedom Caucus, Republican Study Committee, and The Tuesday Group by October 23, before he could make his decision. Immediately after Ryan's announcement, Chaffetz announced that he would be dropping out of the race to support Ryan. The next day, the Freedom Caucus held a vote to determine which of its members would support Ryan; although the exact tally was not revealed, roughly two-thirds of the caucus voted to endorse Ryan. Although this was shy of the 80% vote needed for an official endorsement over Webster, both the caucus leaders and Ryan were satisfied with the result, and Ryan made efforts to move forward with a potential Speaker bid.\\nOn October 22, Ryan announced his bid for Speaker. Flores, who chairs the Republican Study Committee, dropped out of the race and endorsed Ryan. Mo Brooks of Alabama, a member of the Freedom Caucus, announced on the floor of the House on October 27 that Ryan had agreed not to advance immigration reform legislation while Barack Obama is President of the United States, or unless it meets the \\\"Hastert Rule\\\", as it has the support of the majority of Republicans.\\nDeclared.\\nThe following officially declared their candidacy:\\nPublicly expressed interest.\\nThe following publicly expressed interest in becoming candidates:\\nReceived speculation.\\nThe following received speculation about a possible candidacy in at least two reliable sources:\\nWithdrawn.\\nThe following were candidates, but subsequently withdrew:\\nDeclined to run.\\nThe following received some speculation about a possible candidacy, but subsequently ruled themselves out:\\nElection.\\nHouse Republicans planned to hold a non-binding caucus vote on October 8, followed by the official floor vote on October 29. The winning candidate requires a 218-vote majority to win. Multiple ballots may be cast if the majority of the House cannot agree on a candidate. While McCarthy and Chaffetz both said they would vote for the winner of the caucus vote in the floor vote, Webster did not make the same promise.\\nFollowing McCarthy dropping out of the race on October 8, the caucus vote was indefinitely postponed. Massie and Peter T. King referred to the House as a \\\"banana republic.\\\" Massie also criticized Boehner for postponing the election, saying they \\\"called off the election because they didn’t like the result,\\\" which was echoed by Tom Rice, Louie Gohmert, and Justin Amash. McMorris Rodgers and Conference Vice Chairwoman Lynn Jenkins defended Boehner, saying the matter was handled properly, as conference rules give him sole discretion. Rich Lowry of \\\"National Review\\\" asked McCarthy in a phone interview if the House was governable, to which McCarthy replied \\\"I don’t know. Sometimes you have to hit rock bottom.\\\" Charlie Dent, a Republican from Pennsylvania who had supported McCarthy, suggested that if Republicans are unable to agree on a candidate, the best option might be for a bipartisan coalition to select a Speaker.\\nOnce it appeared certain that Ryan would run, and win an overwhelming majority of the caucus's votes, Boehner rescheduled the Republican caucus vote for October 28. Ryan won the nomination, defeating Webster 200 to 43 in the secret ballot voting. Blackburn and McCarthy each received one vote. The next day, Webster reportedly urged Republicans to vote for Ryan instead of him.\\nFinal result.\\nOn October 29, Ryan was elected Speaker with 236 of the 432 votes cast. Others receiving votes were Pelosi (184), Webster (9), Jim Cooper, John Lewis, and Colin Powell (1 each). Votes were cast by 432 of the 435 House members.\\nFollowing the election, Raúl Labrador, a Freedom Caucus member from Idaho, said that Paul will need to \\\"realize the honeymoon is over and start bringing us some conservative policy,\\\" and that \\\"the final exam for Paul Ryan will be in January 2017, when there is a Speaker election, and we will look at his body of work and determine whether he gets a passing grade or not.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122215\",\"title\":\"Ashfaq Ali Khan\",\"body\":\"\\nAshfaq Ali Khan\\n\\nAshfaq Ali Khan () is an Indian politician and a member of the 16th Legislative Assembly of Uttar Pradesh of India. He represents the Naugawan Sadat constituency of Uttar Pradesh and is a member of the Samajwadi Party political party.\\nEarly life and education.\\nAshfaq Ali Khan was born in Sihali Jageer, Uttar Pradesh. He attended the and attained Bachelor of Laws degree.\\nPolitical career.\\nAshfaq Ali Khan has been a MLA for one term. He represented the Naugawan Sadat constituency and is a member of the Samajwadi Party political party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122222\",\"title\":\"Lai Chi Kok Central (constituency)\",\"body\":\"\\nLai Chi Kok Central (constituency)\\n\\nLai Chi Kok Central () is one of the 23 constituencies in the Sham Shui Po District of Hong Kong which was created in 2015.\\nThe constituency loosely covers Liberté, The Pacifica and Banyan Garden in Lai Chi Kok with the estimated population of 19,882.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122469\",\"title\":\"Al-Karimah\",\"body\":\"\\nAl-Karimah\\n\\nAl-Karimah (; also spelled \\\"Karto al-Karimeh\\\") is a small town in northwestern Syria, administratively part of the Tartus Governorate. It is located in the Akkar Plain just north of the border with Lebanon and southeast of al-Hamidiyah. According to the Syria Central Bureau of Statistics (CBS), al-Karimah had a population of 3,461 in the 2004 census. It is the administrative center of the Karimah Subdistrict (\\\"nahiyah\\\") which consisted of 12 localities with a collective population of 17,271. Its inhabitants are predominantly Alawites.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122573\",\"title\":\"Valga Sports Hall\",\"body\":\"\\nValga Sports Hall\\n\\nThe Valga Sports Hall () is a multi-purpose indoor arena complex in Valga. It was opened in 2005 and is the current home arena of the Estonian Basketball League team BC Valga. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122672\",\"title\":\"Sung Wong Toi (constituency)\",\"body\":\"\\nSung Wong Toi (constituency)\\n\\nSung Wong Toi () is one of the 24 constituencies in the Kowloon City District of Hong Kong which was created in 2015.\\nThe constituency loosely covers Ma Tau Kok with the estimated population of 20,487.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122792\",\"title\":\"List of attacks on civilians attributed to United States government forces\",\"body\":\"\\nList of attacks on civilians attributed to United States government forces\\n\\nThe following is a partial list of attacks on civilians attributed to armed groups under the control of the United States government, including the Army, Navy, Air Force, and North Atlantic Treaty Organization forces under U.S. control. The items on the list are sites widely (though not in all cases universally) considered to be of a civilian nature that were directly (though in some cases mistakenly) targeted, and do not include errant bombs or other collateral damage resulting from imprecise attacks on military targets.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122878\",\"title\":\"Pärnu Sports Hall\",\"body\":\"\\nPärnu Sports Hall\\n\\nThe Pärnu Sports Hall () is a multi-purpose indoor arena complex in Pärnu. The hall was opened in 2009 and is the current home arena of the Estonian Basketball League team KK Pärnu.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122920\",\"title\":\"Coach Terminal Station\",\"body\":\"\\nCoach Terminal Station\\n\\nCoach Terminal Station is an underground metro station in Ningbo, Zhejiang, China. Coach Terminal Station situates on the east of Ningbo Coach Terminal. Construction of the station starts in December 2010 and opened to service in September 26, 2015.\\nExits.\\nCoach Terminal Station has 3 exits.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48122984\",\"title\":\"Kai Tak North (constituency)\",\"body\":\"\\nKai Tak North (constituency)\\n\\nKai Tak North () is one of the 24 constituencies in the Kowloon City District of Hong Kong which was created in 2015.\\nThe constituency loosely covers Kai Ching Estate and part of Tak Long Estate in San Po Kong with the estimated population of 16,562.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123075\",\"title\":\"Kai Tak South (constituency)\",\"body\":\"\\nKai Tak South (constituency)\\n\\nKai Tak South () is one of the 24 constituencies in the Kowloon City District of Hong Kong which was created in 2015.\\nThe constituency loosely covers large part of Tak Long Estate in San Po Kong and the previous Kai Tak Airport with the estimated population of 14,599.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123132\",\"title\":\"New Party (Turkey)\",\"body\":\"\\nNew Party (Turkey)\\n\\nThe New Party (Turkish: \\\"Yeni Parti\\\", YP) was a former political party in Turkey that was founded on 23 June 2008. Adhering to the principles of Kemalism, the party merged with the Rights and Equality Party (HEPAR) in 2012 without contesting any elections. Former leaders include Tuncay Özkan, who led the party despite being imprisoned following the Ergenekon trials. The party's colours were primarily red and the party logo was that of the Sun.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123206\",\"title\":\"Yalga, Republic of Mordovia\",\"body\":\"\\nYalga, Republic of Mordovia\\n\\nYalga () is an urban locality (a work settlement) under the administrative jurisdiction of Oktyabrsky City District of the city of republic significance of Saransk in the Republic of Mordovia, Russia. As of the 2010 Census, its population was 5,672.\\nHistory.\\nUrban-type settlement status was granted to it in 1984.\\nAdministrative and municipal status.\\nWithin the framework of administrative divisions, the work settlement of Yalga is incorporated as Yalga Work Settlement, which is subordinated to Oktyabrsky City District of the city of republic significance of Saransk. Within the framework of municipal divisions, Yalga is a part of Saransk Urban Okrug.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123529\",\"title\":\"2015 European Inline Speed Skating Championships\",\"body\":\"\\n2015 European Inline Speed Skating Championships\\n\\nThe 27th European Inline Speed Skating Championships were held in Wörgl, Austria from July 20 to July 26, 2015. Organized by European Confederation of Roller Skating.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123563\",\"title\":\"Gem Motion Picture Company\",\"body\":\"\\nGem Motion Picture Company\\n\\nThe Gem Motion Picture Company was an American silent-era film studio. It was co-founded by filmmaker Stanner E.V. Taylor and his wife, actress Marion Leonard in 1911. After management issues, the company evolved to primarily act as a producer of short comedies starring Billy Quirk.\\nDistribution.\\nGem film properies produced Taylor and Leonard were sold to the Rex Motion Picture Company, which released them in 1912 under their own banner. The company's films were released by Universal in 1913.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123598\",\"title\":\"Khirbet al-Ma'zah\",\"body\":\"\\nKhirbet al-Ma'zah\\n\\nKhirbet al-Ma'zah () is a town in northwestern Syria, administratively part of the Tartus Governorate. It is located along the road between Safita in the east and Tartus to the west. According to the Syria Central Bureau of Statistics (CBS), Khirbet al-Ma'zah had a population of 4,798 in the 2004 census. It is the administrative center of the Khirbet al-Ma'zah Subdistrict (\\\"nahiyah\\\") which consisted of 11 localities with a collective population of 22,897. Its inhabitants are predominantly Alawites.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123601\",\"title\":\"Elymus alaskanus\",\"body\":\"\\nElymus alaskanus\\n\\nElymus alaskanus, more commonly known as Alaskan wheatgrass, is an autogamous perennial that is native to North America and part of the Poaceae family. This complex is widespread and has diverged into different taxa based on morphological and cytological studies.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123619\",\"title\":\"Charlie Chaplin's Comedy Capers\",\"body\":\"\\nCharlie Chaplin's Comedy Capers\\n\\nCharlie Chaplin's Comedy Capers was an American comic strip by Stuart Carothers and later Elzie C. Segar starring Charlie Chaplin. It ran in syndication from 1915 until 1917. It was one of the earliest comic strips inspired by the popularity of a celebrity. \\nBackground.\\n\\\"Charlie Chaplin's Comedy Capers\\\" was published in the Chicago Herald. The comic strip cashed in on the tremendous popularity of the comedian at the time. It was created by Stuart Carothers in March 1915, who drew and wrote the stories until his tragic early death from defenestration. Two cartoonists named \\\"Warren\\\" and \\\"Ramsey\\\" took over the series until they were replaced by Elzie C. Segar, at that time still an amateur. On February 29, 1916 Segar published his first Chaplin strip. The daily version ran until July 15, 1916. His Sunday version ran longer, from March 12, 1916 until September 16, 1917. It was his first professional cartooning job. Contrary to his predecessors, who mostly borrowed ideas from Chaplin's films, Segar thought up his own jokes. He gave Chaplin a tiny sidekick named \\\"Luke the Gook\\\" to act as a straight man to his gags. \\nReception.\\nDespite Chaplin's popularity the comic strip wasn't a huge succes in the United States, mostly due to the fact that all artists involved were basically amateurs. It fared better in the U.K., where it was published in the weekly comics magazine \\\"Funny Wonder\\\" for decades. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123644\",\"title\":\"Sau Mau Ping Central (constituency)\",\"body\":\"\\nSau Mau Ping Central (constituency)\\n\\nSau Mau Ping Central () is one of the 37 constituencies in the Kwun Tong District of Hong Kong which was created in 2015.\\nThe constituency loosely covers part of Sau Mau Ping Estate with the estimated population of 15,256.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123655\",\"title\":\"Intelsat 34\",\"body\":\"\\nIntelsat 34\\n\\nINTELSAT 34 or IS-34 is communications satellite built on Space Systems/Loral's 1300-series satellite platform. The satellite will broadcast television to homes in Brazil, distribute video programming for companies like HBO and Fox across Latin America, and beam broadband services to travelers aboard airplanes and ships crossing the North Atlantic Ocean.\\nIntelsat 34, unlike its predecessor, does not include the UHF-band that Intelsat had been unable to sell to its intended customer, the U.S. Department of Defense.\\nLaunch.\\nIntelsat 34 is the 50th Loral-built satellite launched for Intelsat. With a launch mass of 7,275 pounds - about 3.3 metric tons - Intelsat 34 is a replacement for the Intelsat 27 spacecraft lost aboard a Sea Launch mission in 2013. It will take the place of Intelsat's Galaxy 11 and Intelsat 805 spacecraft in orbit, the company's last two relay stations launched before 2000.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123672\",\"title\":\"San Francisco Ballet Building\",\"body\":\"\\nSan Francisco Ballet Building\\n\\nThe San Francisco Ballet Building, located in San Francisco, California, was designed by architect Beverly Willis and completed in 1984. The New York Times identified it as “the first building in the United States to be designed and constructed exclusively for the use of a major ballet company.”\\nOverview.\\nThe San Francisco Ballet Building is located within the Civic Center, San Francisco. Beverly Willis’s design would later serve as a model for the design of future American ballet companies and schools. Upon completion, the San Francisco Ballet Building became a landmark achievement in the ballet world. The Boston Globe noted, “Dance people don’t merely visit the San Francisco Ballet building: They make pilgrimages to it.”\\nBallet History.\\nThe San Francisco Ballet was the first professional ballet company in the United States. The company was founded in 1933 as the San Francisco Opera Ballet under the leadership of ballet master Adolph Bolm. In 1942, the ballet separated from the San Francisco Opera to become its own independent company. The company’s performances are currently based in the War Memorial Opera House, San Francisco. It is among the world’s leading dance companies, presenting over 100 performances annually, with a repertoire that spans both classical and contemporary ballet. Along with American Ballet Theatre and the New York City Ballet, San Francisco Ballet has been described as part of the \\\"triumvirate of great classical companies defining the American style on the world stage today.\\\"\\nWith its origins in turn-of-the-century traveling companies, the San Francisco ballet routinely rented practice studios. In the late 1970s, the company was housed in a renovated parking garage on 18th Avenue, San Francisco, in a downstairs studio with ceilings so low that the dancers could not practice lifts for fear of hitting the beams. On the east coast, ballet companies like the American Ballet Theatre, were also housed in rented spaces and buildings shared with other performing arts groups. In 1983, the San Francisco Ballet Association became “the first American ballet company and school to break this pattern by constructing new quarters for itself.\\\"\\nDesign.\\nBeverly Willis performed exhaustive research into the function of a ballet building, conducting numerous interviews with dancers on their needs and visiting the studios of major European ballet companies.The building includes facilities to support all of the activities of the company and school with the exception of set storage. The eight rehearsal and classroom studios have 15 feet high ceilings, to accommodate lifts, and average 56x40 feet in size. Additionally, there are administrative offices, a library with audiovisual equipment, and multi-purpose rooms for conferences, academic and choreographic study. Student and company members have physical therapy and workout rooms with gymnastic equipment, locker rooms with showers, separate lounges, and a computer room. Spaces accessible to the public include the ballet shop and a ground-floor studio for community outreach programs.\\nWhile the project would serve as a prototype for new ballet schools nationwide, the design was required to extend deference to the classical order of the Civic Center’s architecture, characterized by grandeur of scale, simplicity of geometric forms, and dramatic use of columns. The San Francisco City Planning Department developed a set of design criteria for the building that specified a height of 96 feet, the location of cornice line levels, and the color and finishes of exterior materials, to ensure that the design was in context with the Neo-Renaissance architecture of the Civic Center. \\nTo accomplish this, Willis incorporated elements of the Neo-Renaissance architectural vocabulary of the Civic Center− the rectangular geometry and the horizontal tripartite divisions of the base, middle and top, whose heights correspond to the opera house. The building’s planned location was on an elongated and rectangular site, one-sixth the size of the adjacent performing arts structures. To be successfully contextual, it needed to appear massive to sustain a visual relationship with the monumental civic center buildings occupying over a square block. Willis sized the facade to monumental, classical proportions. The four story facade equaled in height a typical 8 story office building. The horizontal divisions of the base, midsection and top, as part of the facade, matched the heights of those of the adjacent civic center buildings. The rectangular form of the envelope produced a classical form, into which the ballet’s interior functions fit. The proportion of the plan itself was a three to one classical ratio. The building is clad in a concrete material similar in color and texture to the other contemporary civic center structures.\\nBreaking with the classical tradition of symmetry, the proscenium-style main entry is located on the corner of the site. The building’s two-story monumental entry at the corner accomplished several objectives: it connected the building with the Civic Center’s master plan axis; it gave the building an identity within the performing arts complex, from the Van Ness Avenue perspective where the Opera and Symphony faced; and it avoided orienting the main entry towards the Opera House’s blank rear wall.\\nThe facade came out of the program like an abstraction and manifestation of the idea of ballet. The entry was envisioned as a proscenium arch. The rippling, curved glass within the entry is reminiscent of a stage curtain. The curved balconies over the entry arch are reminiscent of theater box seats. The facade is designed with solids and voids, curves and planes that play in constantly shifting light and shadows. The monumentality of the mass is softened by transparent layers that reveal the creative possibility, awaiting the birth of dance.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123705\",\"title\":\"Upper Ngau Tau Kok Estate (constituency)\",\"body\":\"\\nUpper Ngau Tau Kok Estate (constituency)\\n\\nUpper Ngau Tau Kok Estate () is one of the 37 constituencies in the Kwun Tong District of Hong Kong which was recreated in 2015.\\nThe constituency loosely covers part of Upper Ngau Tau Kok Estate with the estimated population of 15,969.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123865\",\"title\":\"Lower Ngau Tau Kok Estate (constituency)\",\"body\":\"\\nLower Ngau Tau Kok Estate (constituency)\\n\\nLower Ngau Tau Kok Estate () is one of the 37 constituencies in the Kwun Tong District of Hong Kong which was created in 2015.\\nThe constituency loosely covers part of Lower Ngau Tau Kok Estate with the estimated population of 17,736.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48123867\",\"title\":\"Georgi Dimitrov bibliography\",\"body\":\"\\nGeorgi Dimitrov bibliography\\n\\nGeorgi Dimitrov (June 18, 1882 – July 2, 1949) was a Bulgarian communist politician. He was the first communist leader of Bulgaria, from 1946 to 1949. Dimitrov led the Third Comintern (Communist International) under Stalin from 1934 to 1943. He was a theorist of capitalism who expanded Lenin's ideas by arguing that fascism was the dictatorship of the most reactionary elements of financial capitalism.\\nThis is a Georgi Dimitrov bibliography, including writings, speeches, letters and others.\\nWritings.\\nThis is a list of selected writings:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124047\",\"title\":\"Kuranishi structure\",\"body\":\"\\nKuranishi structure\\n\\nIn mathematics, especially in topology, a Kuranishi structure is a smooth analogue of scheme structure. If a topological space is endowed with a Kuranishi structure, then locally it can be identified with the zero set of a smooth map formula_1 Kuranishi structure was introduced by Japanese mathematicians Kenji Fukaya and Kaoru Ono in the study of Gromov–Witten invariants in symplectic geometry.\\nDefinition.\\nLet formula_2 be a compact metrizable topological space. Let formula_3 be a point. A Kuranishi neighborhood of formula_4 (of dimension formula_5) is a 5-tuple\\nwhere\\nThey should satisfy that formula_12.\\nIf formula_13 and formula_6, formula_15 are their Kuranishi neighborhoods respectively, then a coordinate change from formula_16 to formula_17 is a triple\\nwhere\\nIn addition, they must satisfy the compatibility condition:\\nA Kuranishi structure on formula_2 of dimension formula_5 is a collection\\nwhere\\nIn addition, the coordinate changes must satisfy the cocycle condition, namely, whenever formula_34, we require that\\nover the regions where both sides are defined.\\nHistory.\\nIn Gromov–Witten theory, one needs to define integration over the moduli space of stable maps formula_36(see for example ). They are maps formula_37 from a nodal Riemann surface with genus formula_38 and formula_39 marked points into a symplectic manifold formula_2, such that each component satisfies the Cauchy–Riemann equation\\nIf the moduli space is a smooth, compact, oriented manifold or orbifold, then the integration (or a fundamental class) can be defined. When the symplectic manifold formula_2 is semi-positive, this is indeed the case (except for codimension 2 boundaries of the moduli space) if the almost complex structure formula_43 is perturbed generically. However, when formula_2 is not semi-positive, the moduli space may contain configurations for which one component is a multiple cover of a holomorphic sphere formula_45 whose intersection with the first Chern class of formula_2 is negative. Such configurations make the moduli space very singular so a fundamental class cannot be defined in the usual way.\\nThe notion of Kuranishi structure was a way of defining a virtual fundamental cycle, which plays the same role as a fundamental cycle when the moduli space is cut out transversely. It was first used by Fukaya and Ono in defining the Gromov–Witten invariants and Floer homology, and was further developed when Fukaya, Oh, Ohta, Ono studied the Lagrangian intersection Floer theory.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124067\",\"title\":\"Jordan North (constituency)\",\"body\":\"\\nJordan North (constituency)\\n\\nJordan North () is one of the 19 constituencies in the Yau Tsim Mong District of Hong Kong which was created in 2015.\\nThe constituency loosely covers Jordan Road with the estimated population of 13,558.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124115\",\"title\":\"Jazz Suite Inspired by Dylan Thomas' \\\"Under Milk Wood\\\"\",\"body\":\"In a series of articles for \\\"The Guardian\\\" newspaper titled \\\"50 great moments in jazz\\\", John Fordham wrote of the album: \\\"Under Milk Wood was an evocative collection of sparky themes inspired by the Dylan Thomas radio play (it's sometimes performed with a narrator reading the parts). And thanks to Tracey's sparing piano and Wellins's softly hooting sax, the rippling tone-poem Starless and Bible Black is widely acclaimed as one of the great jazz performances\\\".\\nRelease History.\\nSince the original 1965 mono LP on Columbia, the album has been released on a number of labels, including Blue Note Records. The album was reissued in 2010 on his son, Clark Tracey's ReSteamed Records label, as \\\"Under Milk Wood: Jazz Suite\\\". A live version was recorded in 1976 on RCA records, which included a voice narration from the Welsh actor, Donald Houston.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124386\",\"title\":\"Strongboard Balance\",\"body\":\"\\nStrongboard Balance\\n\\nStrongBoard Balance is as a springboard and multi-spring balance board.\\nOperation.\\nConnected at the base, StrongBoard Balance is a one piece apparatus with the springs connected to the board. Unlike traditional balance trainers, there is no tube or roller, which prevents the board from sliding out from underneath or coming apart. StrongBoard Balance’s difference is the targeted layer of stability it adds to traditional strength-training. Through employing various balance techniques, StrongBoard Balance works to strengthen the core, toughen stabilizing muscles and building neuropathic communication. By not allowing stabilization, the user’s muscles never stop working. With a design consisting of a solid base and a flat top, StrongBoard Balance is a tool for exercises including pushups, step ups, squats, and many others.\\nUse.\\nIt launched in 2014 at the LA Fit Expo and showed at the IHRSA International Convention. Utilizing four compression springs, individuals can use StrongBoard Balance to train using the weight of their own body – in a portable and electricity manner. By engaging core-stabilizing muscles, the board builds upon traditional strengthening methods, and adds a new, plyometric and cardio training element.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124405\",\"title\":\"Kiss Me (Olly Murs song)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124495\",\"title\":\"7 Letters\",\"body\":\"\\n7 Letters\\n\\n7 Letters is a 2015 Singaporean drama film directed by seven different directors. It comprises seven short stories celebrating Singapore's 50th anniversary. The film was selected as the Singaporean entry for the Best Foreign Language Film at the 88th Academy Awards but it was not nominated.\\nIn January 2016, the film was initially flagged by Malaysian censors before it was due to screen at the Titian Budaya Festival. A successful appeal was made by organisers, CultureLink, against the cuts for the vulgar phrase in Cantonese, “curse your whole family”, in Jack Neo’s segment of the omnibus.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124616\",\"title\":\"Rainy Cocoa\",\"body\":\"\\nRainy Cocoa\\n\\n is a bilingual Japanese digital manga series. An anime television series based on the manga began airing in April 2015, and a second season, titled , began in October 2015.\\nMedia.\\nManga.\\nThe manga is available in both Japanese and English, and allows the reader to switch between the two languages for both text and audio. According to the creators, the manga's app has been downloaded from the iTunes Store and Google Play in over 50 different countries.\\nAnime.\\nAn television anime adaptation was announced in December 2014. The series is directed by Tomomi Mochizuki, Atsuko Takahashi provides the character designs, and Kaoru Kondou is composing the series' music. The series' theme song, \\\"Rainy Cocoa\\\", is performed by Hiro Shimono. The series began airing on 5 April 2015, and was simulcast in North America by Funimation.\\nA second season was announced in June 2015, titled . Also announced was a crowdfunding campaign to raise 2 million yen to add more characters and cast members. It premiered on 4 October 2015 on Tokyo MX and Sun TV. It will be accompanied by a live-action segment. The second season was also streamed by Funimation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124627\",\"title\":\"Luigi Logan Grosu\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124650\",\"title\":\"1971 Indiana Hoosiers football team\",\"body\":\"\\n1971 Indiana Hoosiers football team\\n\\nThe 1971 Indiana Hoosiers football team represented the Indiana Hoosiers in the 1971 college football season. They participated as members of the Big Ten Conference. The Hoosiers played their home games at Seventeenth Street Stadium in Bloomington, Indiana. The team was coached by John Pont, in his seventh year as head coach of the Hoosiers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124725\",\"title\":\"Academic Medicine (journal)\",\"body\":\"\\nAcademic Medicine (journal)\\n\\nAcademic Medicine is the monthly peer-reviewed medical journal of the Association of American Medical Colleges.\\nHistory.\\nThe journal was established in 1926 as the \\\"Bulletin of the Association of American Medical Colleges\\\". It was renamed \\\"Journal of the Association of American Medical Colleges\\\" in 1929. In 1951 it briefly became \\\"Medical Education\\\" then \\\"Journal of Medical Education\\\". In 1989 it took its current name of \\\"Academic Medicine\\\".\\nIn the course of its history, the journal has had nine editors. David P. Sklar is the present editor-in-chief, appointed in 2013.\\nAbstracting and indexing.\\nThe journal is abstracted and indexed in:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124730\",\"title\":\"Yau Ma Tei North (constituency)\",\"body\":\"\\nYau Ma Tei North (constituency)\\n\\nYau Ma Tei North () is one of the 19 constituencies in the Yau Tsim Mong District of Hong Kong which was first created in 1982 and recreated in 2015.\\nThe constituency loosely covers Yau Ma Tei with the estimated population of 12,817.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124792\",\"title\":\"List of performing artists at the Viña del Mar International Song Festival\",\"body\":\"\\nList of performing artists at the Viña del Mar International Song Festival\\n\\nThe Viña del Mar International Song Festival () is a music festival that is considered the best and biggest in Latin America and the most important musical event in the Americas which is held annually on the 3rd week of February since 1960 in Viña del Mar, Chile.\\nPerforming Artists per year.\\n2015.\\nTransmitting to date\\nPerformers per Country.\\nUnited Kingdom.\\n(Includes England, Scotland, Wales and Gibraltar).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124831\",\"title\":\"Almost Home (Mary Chapin Carpenter song)\",\"body\":\"\\nAlmost Home (Mary Chapin Carpenter song)\\n\\n\\\"Almost Home\\\" is a song co-written and recorded by American country music singer Mary Chapin Carpenter\\nContent.\\nThe song is a mid-tempo about a woma who \\\"takes stock of a life lived and comes up short\\\". It is in the key of B-flat major with an approximate tempo of 96 beats per minute and a chord pattern of F-B-E-B. Carpenter wrote the song with Beth Nielsen Chapman and Annie Roboff, and produced it with Blake Chancey.\\nCritical reception.\\nDeborah Evans Price of \\\"Billboard\\\" reviewed the single with favor, saying that \\\"it's a vibrant musical outing that could signal a return to prominence for this talented singer/songwriter. Carpenter's evocative vocals infuse any song with passion and integrity, and she's particularly effective on this poignant lyric.\\\" Alanna Nash of \\\"Entertainment Weekly\\\" also described the song favorably in her review of the album, saying that it \\\"scores the highest at marrying lyrical introspection and rhythmic yearning, one of the benchmarks of her graceful, if circuitous, career.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124892\",\"title\":\"Matthew Ramsey (songwriter)\",\"body\":\"\\nMatthew Ramsey (songwriter)\\n\\nMatthew Ramsey is an American country music songwriter and lead vocalist of the band Old Dominion with several hit songs on country radio to his credit.\\nEarly life.\\nMatthew Ramsey grew up in Buchanan, VA. He learned to play the drums at a young age before picking up the guitar at age 14.\\nCareer.\\nHe received a degree in Illustration from Virginia Commonwealth University and then moved to Nashville to pursue songwriting. He met Trevor Rosen in Nashville and the two became members of the group Old Dominion. Ramsey has had several hit songs including The Band Perry’s \\\"Chainsaw\\\", Craig Morgan’s “Wake Up Loving You’’, Dierks Bentley’s “Say You Do’’, Kenny Chesney's \\\"Save It For a Rainy Day\\\", as well as songs by Sam Hunt and Luke Bryan. The band released its first album in October 2015, with songs that reached the top ten on the country charts.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124910\",\"title\":\"Center for Pastor Theologians (CPT)\",\"body\":\"\\nCenter for Pastor Theologians (CPT)\\n\\nThe Center for Pastor Theologians (CPT) is an evangelical Christian organization that is based in Oak Park, Illinois with the stated mission of assisting pastor theologians with “the study and written production of biblical and theological scholarship for the ecclesial renewal of theology and the theological renewal of the church.”\\nBackground and Goals.\\nThe organization was founded in 2006 by Gerald Hiestand and Todd Wilson, originally with the name The Society for the Advancement of Ecclesial Theology. Ecclesial theology, as described by Hiestand, is \\\"a theological project born in an ecclesial context, driven by an ecclesial agenda, and prosecuted by ecclesial thinkers; not merely theology about the church--but theology for the church.\\\"\\nThe Center believes that many pastors are either ill-supported to contribute to the theological formation of the broader church, or not expected to do so at all, given the majority of theological scholarship being produced by professors in seminaries and universities.\\nThe Center’s offices and symposia are hosted at Calvary Memorial Church in Oak Park, Illinois. Symposia are held for three Fellowships, groups of roughly 20 pastor theologians who gather to discuss a theological text with a noted theologian. At symposia, Fellows also solicit feedback on personal projects.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124943\",\"title\":\"1987 in Macau\",\"body\":\"\\n1987 in Macau\\n\\nEvents from the year 1987 in Portuguese Macau.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124967\",\"title\":\"French legislative election, 1848 (Senegal)\",\"body\":\"\\nFrench legislative election, 1848 (Senegal)\\n\\nElections to the French National Assembly were held in Senegal between 30 October and 2 November 1848.\\nElectoral system.\\nThe single Senegalese seat in the National Assembly was created by decree on 5 March 1848. Following an order of 5 November 1830 and a law of 24 April 1833, all free-born people and freed slaves in the Four Communes in Senegal had full civic and political rights, the only French African colony to give such rights until the end of World War II. The right to vote was given to all men over the age of 25 and who could prove they had lived in their municipality for the previous five years. In total 4,706 men registered to vote.\\nCampaign.\\nThe election was contested by three candidates; former Governor Léandre Bertin du Château, the creole Barthélémy Durand Valantin and Victor Schoelcher, a well-known anti-slavery activist. Although Schoelcher was on the ballot, he was not in Senegal at the time.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48124994\",\"title\":\"2015–16 Scottish Junior Cup\",\"body\":\"\\n2015–16 Scottish Junior Cup\\n\\nThe 2015–16 Scottish Junior Cup is the 130th season of the Scottish Junior Cup, the national knockout tournament for member clubs of the Scottish Junior Football Association. The competition is sponsored by ETHX Energy and is known as The ETHXenergy Scottish Junior Cup. The winner of this competition is eligible to enter the following seasons Scottish Cup at the first round stage.\\nA total of 158 clubs are entered to this year's competition, three fewer than the previous season. Dropping out are Ballingry Rovers who have folded and Harthill Royal, Luncarty, Portgordon Victoria and Whitehills who are in abeyance. North Region sides Glentanar and Islavale did not enter. New members Easthouses Lily MW and Gartcairn FA Juniors make their debut in the competition while Coupar Angus and Fochabers return to the tournament after a year in abeyance.\\nThe six Junior clubs qualified for this season's Scottish Cup, are not included in the draw for the first round:\\nAlso qualified automatically are Banks O'Dee and Linlithgow Rose who achieved national club licensing requirements and Girvan who qualify automatically as historic full members of the Scottish Football Association.\\nFirst round.\\nThe first round draw took place at Hampden Park, Glasgow on 27 August 2015.\\nSecond round.\\nThe second round draw took place at Mar Hall, Erskine on 6 October 2015.\\nThird round.\\nThe third round draw took place at The Sun offices in Glasgow on 3 November 2015.\\n1 Match played at Pollok F.C.\\nReplays.\\n2 Match played at Tranent Juniors F.C.\\nFourth round.\\nThe fourth round draw took place at the Evening Times offices in Glasgow on 9 December 2015. \\nFifth round.\\nThe fifth round draw will take place in Glasgow City Chambers on 26 January 2016 at 2:00pm.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125053\",\"title\":\"Space (architecture)\",\"body\":\"\\nSpace (architecture)\\n\\nSpace is one of the elements of design of architecture, as space is continuously studied for its usage. Architectural designs are created by carving space out of space, creating space out of space, and designing spaces by dividing this space using various tools, such as geometry, colours, and shapes.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125075\",\"title\":\"Heather Parker\",\"body\":\"\\nHeather Parker\\n\\nHeather Parker (born September 25, 1978, Boston, Massachusetts, United States) is an American photographer and glitch artist, known for her work in documentary photography, as well as glitch art created from photographs of food. Parker is the cousin of Hoorsenbuhs partner Kether Parker.\\nBiography.\\nHeather began working with digital media after being formally trained in film photography at Massachusetts College of Art and Design under Abelardo Morell, Frank Gohlke and Laura McPhee. \\nParker's documentary work includes photographing artists in and around Boston, deejays around the country, and Vespa rallies.\\nThe glitch art that Parker has exhibited lays out defects produced by databending using a combination of audio editing software and photo software.\\nParker gained a B.F.A. from the Massachusetts College of Art and Design in 2000 and resides in Chicago and the Fort Point Channel Historic District of Boston. She is a member of the Fort Point Arts Community and was noted in the Boston Globe as part of a campaign to save the historic Old Northern Avenue Bridge.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125092\",\"title\":\"V Satish\",\"body\":\"\\nV Satish\\n\\nV Satish (Satish Velankar) is the National Jt. General Secretary (Organisation) of Bharatiya Janata Party (BJP). Currently he is in-charge of Rajasthan State, Bharitya Janata Party. He is Full-time Worker, Rashtriya Swayamsevak Sangh and has been the senior functionary for the Rashtriya Swayamsewak Sangh.\\nEarly life.\\nBorn in Nagpur district of Maharashtra on May 23, 1954.\\nAssociation with RSS.\\nFull-timer for RSS.\\nAssociation with the BJP.\\nWorking as Jt. General Secretary (Org.) in Amit Shah, National Team of BJP. Currently handling charge of Rajasthan, BJP.\\nHe was posted at Bangalore in Karnataka State for handling operations of Bhartiya Janata Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125118\",\"title\":\"Golden Eaglets of Nigeria\",\"body\":\"\\nGolden Eaglets of Nigeria\\n\\nThe Golden Eaglets Nigeria is the Nigerian Football Under 17 Team. They won the Maiden edition of the FIFA cadet world cup staged in China in 1985. They defeated the former West Germany to win the cup.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125124\",\"title\":\"Hudson Coutinho\",\"body\":\"\\nHudson Coutinho\\n\\nHudson José Coutinho (born 12 July 1972), known as Hudson Coutinho, is a Brazilian football manager, currently in charge of Figueirense.\\nCareer.\\nBorn in Florianópolis, Santa Catarina, Coutinho joined hometown's Figueirense in 2000, as a fitness coach, after a brief spell at Guarani de Palhoça. After nine years at the club – which included being manager for one match as an interim in 2007 – he moved to Náutico.\\nIn July 2012, after another fitness coach experiences at Chapecoense, Marcílio Dias and Hercílio Luz, Coutinho was appointed manager of his previous club Guarani. He achieved promotion with the club in the state league, as champions, but was still sacked on 25 February of the following year.\\nIn March 2013 Coutinho returned to his former side Figueirense, as an assistant manager. On 19 September 2015 he was named interim manager, replacing sacked René Simões.\\nOn 21 September 2015 Coutinho was permanently appointed as manager of \\\"Figueira\\\", after a request from the squad.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125143\",\"title\":\"Petros Velissariou\",\"body\":\"\\nPetros Velissariou\\n\\nPetros Velissariou (in Greek: Πέτρος Βελισσαρίου;) born April 20, 1993 is a Greek professional basketball player who is currently playing for Kavala. He is 1.90 m (6 ft 2 ¾ in) tall.\\nClub career.\\nAfter playing youth basketball at Mantoulidis ,Velissariou started his playing career in 2012, playing with Kavala.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125164\",\"title\":\"Jennifer Thomas (pianist)\",\"body\":\"\\nJennifer Thomas (pianist)\\n\\nJennifer Thomas (born June 23, 1977) is an American pianist, violinist, composer, performing artist, and recording artist. She was classically trained at Brigham Young University-Idaho, and started composing in 2003, later releasing her first debut album in 2007. Thomas has issued four albums, the latest 'Winter Symphony' released in November 2015.\\nThomas is known for writing and performing piano-centered orchestral music from classical music to classical music crossover and cinematic orchestral. While most of her works are original, she also covers classical pieces as well as pop music and movie soundtracks. Thomas has also composed film scores, where she won the Gold Medal of Excellence in the 2011 Park City Film Music Festival for \\\"Music in a Short Film\\\" as well as a 2012 Hollywood Music in Media Award for \\\"Best Film Score for Documentary/Short\\\".\\nIn 2012 Thomas was named Classical Crossover UK's \\\"Female Artist of the Year\\\".\\nThomas' third album, \\\"Illumination\\\" won \\\"Album of the Year\\\" and \\\"Artist of the Year\\\" at the 2013 IMC Awards. \\nIn October 2015 \\\"Illumination\\\" also won the \\\"Classical Song of the Year\\\" International Music and Entertainment Association Award and the Enlightened Piano Radio Award for \\\"Best piano album with instrumentation\\\" at Carnegie Hall New York - where she performed live at the awards ceremony.\\nWinter Symphony - Jennifer Thomas (pianist) album Thomas' fourth album was released in November 2015. \\nMixed by 5 time Grammy Award winner Brian Vibberts, contributors included Ricky Kej and Taylor Davis (violinist).\\nThe album has received many favourable reviews.\\nFollowing the album release, Thomas performed concerts at Salt Lake City, Atascocita, Texas and the Benaroya Hall Seattle.\\nDiscography.\\nAlbums:\\nThomas also features on the following albums;\\nFilm and TV.\\nList:\\nEarly life.\\nThomas was born in Walla Walla, Washington. She got her first toy piano at the age of 3, and began formal lessons on both the piano and violin at the age of 5, under the tutelage of her mother, Carolyn Southworth. Her brothers also learned piano, violin, and cello from their mother. Jennifer and her older brother played in grade school orchestras, with Southworth as a conductor's assistant. Thomas continued to train classically, and was involved in many recitals and piano adjudications through her teen years. She played with various youth orchestras and high school orchestras. She performed at her high school graduation, and later went to Brigham Young University-Idaho.\\nAt Brigham Young University-Idaho, Thomas studied piano under Professor Stephen Allen. She was a member of the university's Piano Ensemble group, accompanied the College choir and many vocal/instrumental students, as well as played violin in the university symphony. Thomas also competed in the University's Concerto Competition, where she took 2nd place performing the Piano Concerto No. 3 (Prokofiev).\\nAfter college graduation, Thomas moved to Salt Lake City, Utah, and was a regular performer with the Temple Square Concert Series, and played violin in the Murray, Utah Symphony Orchestra. While performing with the Murray Symphony, Thomas was a featured soloist on the Edward MacDowell Piano Concerto No.2. She eventually moved to Northwest, where she performed as a violinist with the Oregon Pro Arte Chamber Orchestra, and worked for the Seattle Symphony in the Education Department. It was during her time working for the Seattle Symphony when she started composing her own original music.\\nPersonal life.\\nThomas is married to Will Thomas, an ultra-marathon competitor who, in 2015, competed in the 106 mile race Ultra-Trail du Mont-Blanc in France. Together, they have three children and reside in the Seattle Washington (state) area.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125213\",\"title\":\"Yahmur\",\"body\":\"\\nYahmur\\n\\nYahmur (; also spelled \\\"Yahmour\\\") is a village in northwestern Syria, administratively part of the Tartus Governorate. It is located along the road between Safita in the east and Tartus to the west. According to the Syria Central Bureau of Statistics (CBS), Yahmur had a population of 3,722 in the 2004 census. Its inhabitants are predominantly Alawites. Nearby is Chastel Rouge (Qal'at Yahmur), a Crusader-era castle.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125225\",\"title\":\"Durston (surname)\",\"body\":\"\\nDurston (surname)\\n\\nDurston is an English toponymic surname. The name was first recorded in 1641 in \\\"A Somerset Petition of 1641\\\". The name is taken from the village of Durston in Somerset. It is derived from the Anglo-Saxon \\\"deór-tún\\\", a combination of \\\"deer\\\" (\\\"deór\\\",) and \\\"fenced enclosure\\\" (\\\"tún\\\"). The most likely interpretation is \\\"deer park\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125249\",\"title\":\"Ram Lal\",\"body\":\"\\nRam Lal\\n\\nRam Lal is an Indian politician who is the National General Secretary for the Bhartiya Janata Party\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125286\",\"title\":\"Migration Court\",\"body\":\"\\nMigration Court\\n\\nA Migration Court is a type of administrative court within the Swedish legal system. The Migration Courts are the courts of appeal for decisions made by the Swedish Migration Agency, for example regarding asylum or residency in Sweden. Decisions by the Migration Courts can be appealed to the Migration Courts of Appeal which are supreme appellate court relating to migration law.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125289\",\"title\":\"Judith Flemig\",\"body\":\"\\nJudith Flemig\\n\\nJudith Flemig-Pelzer (born 22 May 1979) is a former German female volleyball player. She was part of the Germany women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 6th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125323\",\"title\":\"Anja-Nadin Pietrek\",\"body\":\"\\nAnja-Nadin Pietrek\\n\\nAnja-Nadin Pietrek (born 13 March 1979) is a former German female volleyball player. She was part of the Germany women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 6th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125333\",\"title\":\"Hayle and Bristol Steam Packet Company\",\"body\":\"\\nHayle and Bristol Steam Packet Company\\n\\nThe Hayle and Bristol Steam Packet Company operated steam ship services between Hayle, Ilfracombe and Bristol in the mid nineteenth century. Confusingly from 1848 to 1860, the company name was used by two separate operators.\\nHistory.\\nAlthough a steamer first called at Hayle in 1824, regular weekly services began in 1831, when the \\\"Hayle Steamship Company\\\" was formed operating with the wooden vessel \\\"Herald\\\", under the command of John Vivian.\\nThe engineering company Harvey's of Hayle built the engines for the PS \\\"Cornwall\\\" of 1842. When the Great Western Railway arrived in Bristol, this stimulated more travel between London and the South West of England, and the PS \\\"Cornwall\\\" was added to the Hayle service under the command of John Vivian. The extra business attracted a rival when Vivian Stevens of St Ives put his PS \\\"Brilliant\\\" on to the Hayle to Bristol route. \\nThe \\\"Hayle Steamship Company\\\" was renamed the \\\"Hayle and Bristol Steam Packet Company\\\" in 1848, and confusingly Vivian Stevens with the PS \\\"Brilliant\\\" adopted the same title.\\nThe original \\\"Hayle and Bristol Steam Packet Company\\\" launched a prospectus in 1857 to attract capital investment. The secretary of the new company was Mr John Vivian of Hayle. The company was launched with nominal capital of £28,750 (£ in ) and working capital of £23,000 (£ in ). The company prospectus announced that a First Class A.1. 12 years Iron Steamer was being built at Hayle, by Harvey and Co, capable of carrying 200 Tons of Cargo, with ample accommodation for passengers. The cost of this new steamer was £18,000 (£ in ), and in 1858, the company launched the Cornubia. The company also planned to sell the Cornwall for £2,000 (£ in ) when the SS \\\"Cornubia\\\" was launched.\\nThe additional traffic was short-lived, as the extension of railway services from London and Bristol through into Cornwall was completed when the Royal Albert Bridge was completed and opened on 2 May 1859.\\nBoth Hayle and Bristol Steam Packet Companies amalgamated around 1860, probably as a result of traffic diminishing. The company undertook voluntary liquidation following a meeting of the shareholders on 6 November 1861. Steamer services continued, and attempted competition with screw vessels. Having sold off the SS \\\"Cornubia\\\", Harveys built and owned the SS \\\"Bride\\\" of 1863 and SS \\\"Bessie\\\" of 1865. However, the two ships could not compete with rail travel for both passengers and freight, and were moved to other trades. \\nThe Hayle to Bristol services reduced and were operated by \\\"Hosken, Trevithick, Polkinhorn and Company Ltd\\\" of Penzance which bought the screw steamer \\\"Norseman\\\" in 1893. The \\\"Norseman\\\" was replaced with the \\\"M.J. Hedley\\\", a steam coaster carrying passengers until 1917 on a weekly service linking Bristol, Hayle and Liverpool.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125341\",\"title\":\"Sylvia Roll\",\"body\":\"\\nSylvia Roll\\n\\nSylvia Roll (born 29 May 1973) is a former German female volleyball player. She was part of the Germany women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 6th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125360\",\"title\":\"Christina Schultz\",\"body\":\"\\nChristina Schultz\\n\\nChristina Schultz (born 10 November 1969) is a former German female volleyball player. She was part of the Germany women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 6th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125396\",\"title\":\"Maria Kochwa\",\"body\":\"\\nMaria Kochwa\\n\\nMary Ayuma-Kochwa (born 23 October 1966) is a former Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125402\",\"title\":\"2015 World Masters (darts)\",\"body\":\"\\nThe seedings were finalised on completion on 31 August. For the fourth consecutive year, there are 32 seeds (an increase from 8 between 2007–2011) with the Top 16 exempt until the Last 32 stage.\\nMen's Draw.\\nLast 32 onwards.\\nSets are best of 3 legs.\\nWomen.\\nThe seedings were finalised on completion of the 2014 French Open 29 August. The ladies seeds enter at the start of the competition however can not play each other until the quarter final stage.\\nLadies Draw.\\nLast 8 onwards.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125484\",\"title\":\"Firefly protocol\",\"body\":\"\\nFirefly protocol\\n\\nFirefly protocol may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125500\",\"title\":\"ICFR, Ltd.\",\"body\":\"\\nICFR, Ltd.\\n\\nICFR, Ltd. is a company registered in the United Kingdom. The company was founded on February 27, 2014. Although it is officially classified as a “private company limited by guarantee without share capital,” it is intended to act as an advocacy organization focusing on human rights. Its purpose is to “document cases of human rights violations and defend victims” through the use of domestic and international law and “UN human rights mechanisms.”\\nIn furtherance of this goal it undertakes public relations and lobbying activities “to impose pressure on governments” that violate human rights, and provides “legal counsel and psychological support for victims of human rights violations.” The company is empowered by its articles of incorporation to fundraise and to contribute to other charities in furtherance of these goals.\\nICFR’s official address is in the Westgate House, an office building in London that houses dozens of businesses and charities. Several of these organizations have been tied to extremist groups including the Muslim Brotherhood and Hamas. The most notable of these is the Cordoba Foundation, which Prime Minister David Cameron has described as a “political front for the Muslim Brotherhood.\\\"\\nThe only officer of ICFR is Anas Altikriti, who serves as director. Altikriti, an outspoken supporter of the Muslim Brotherhood, is also president and founder of the Cordoba Foundation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125525\",\"title\":\"1976 in Macau\",\"body\":\"\\n1976 in Macau\\n\\nEvents from the year 1976 in Portuguese Macau.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125553\",\"title\":\"Kuiba 2\",\"body\":\"\\nKuiba 2\\n\\nKuiba 2 () is a 2013 Chinese animated fantasy action adventure film directed by Chuan Wang. It was released on May 31, 2013. The film is part of the \\\"Kuiba\\\" film series, following \\\"Kuiba\\\" (2011) and preceding \\\"Kuiba 3\\\" (2014).\\nReception.\\nThe film earned at the Chinese box office.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125624\",\"title\":\"Bill Shockley\",\"body\":\"\\nBill Shockley\\n\\nWilliam Albert “Bill” Shockley Jr. (March 13, 1937 – December 7, 1992) was an American football kicker and halfback who played for four seasons for four different teams, the New York Titans, Buffalo Bills, and the Pittsburgh Steelers. He played college football at West Chester University. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125662\",\"title\":\"Mark McDowell\",\"body\":\"\\nMark McDowell\\n\\nMark McDowell (born October 8, 1962) is a Canadian diplomat and the first resident Ambassador of Canada to the Union of the Republic of Myanmar. He is considered an \\\"Asia hand\\\" and is a specialist in the fields of public diplomacy and digital diplomacy. He has been characterized as \\\"an unconventional diplomat\\\".\\nEducation.\\nAttracted by its “counterculture atmosphere”, McDowell attended Innis College at the University of Toronto. He received a MA from University of Toronto in East Asian Studies in 1988. He attended the Kennedy School of Government as a Fulbright Scholar, earning a Master of Public Administration in 2008, and stayed on as a visiting Research Scholar at the Ash Institute for Democratic Governance and Innovation.\\nMyanmar.\\nMcDowell was named Canada's first ever resident ambassador to Myanmar in March 2013. Because of the long period of sanctions imposed by Canada on Myanmar, the relationship between the two countries was very limited and cold.\\nSince taking up his job in June 2013, relations have warmed considerably, with Canada naming Myanmar a priority country for both trade and aid and opening a visa office. Ambassador McDowell has focused much of his attention on supporting grassroots democracy and human rights organizations working on issues such as freedom of speech and voter and civic education. He has staked out a leading role for Canada in the area of federalism and minority rights, and in the promotion of LGBT rights in this laboratory for democracy. Canada has also started to cooperate on security related issues like border management and prevention of human trafficking.\\nThe Embassy of Canada has had success in rapidly raising Canada's public profile in Burma, through both traditional and social media. Its Facebook presence has over 140,000 followers on its Burmese, English, and French pages.\\nPublic Diplomacy and E-diplomacy.\\nMcDowell is known as an innovator in public diplomacy and e-diplomacy. Canada’s Globe and Mail credits him with \\\"helping drag the Department of Foreign Affairs into the Internet age”, and Canada’s former Ambassador to China called him “among the most original of thinkers when it comes to using social media”.\\nE-diplomacy.\\nAmbassador McDowell is a frequent speaker and participant in e-diplomacy conferences, offering a practitioners perspective. In 2015 he spoke at events organized by the foreign ministries of Armenia and the Netherlands.\\nHe is known for starting Canada's social media presence in China with \\\"canadaweibo\\\" on the Chinese Sina Weibo platform (see www.weibo.com/canadaweibo). Canadaweibo grew from zero to 400,000 followers within two years, becoming the second largest and most influential embassy site in China after the USA’s. Similarly, he has made social media a priority during his posting in Myanmar (see www.facebook.com/CanadainBurma).\\nMcDowell's work in e-government is longstanding. He gave the government of Canada's address to the United Nations World Summit on the Information Society in 2002. At that time he also took a leading role in initiatives in Canada related to getting Aboriginal communities online, and disseminating news about the international activities of Canadian Aboriginal peoples.\\nPublic Diplomacy.\\nMcDowell has worked on public diplomacy throughout his diplomatic career, and was a Director of Public Diplomacy and Domestic Outreach in Canada’s Department of Foreign Affairs.\\nHe wrote a brief primer on public diplomacy based on a talk he delivered at the 100th Anniversary Edward R. Murrow Memorial Conference at the Fletcher School for Law and Diplomacy (see www.fletcherforum.org/2012/07/26/32-3/). In a 2010 talk at the Fletcher School on the connection between domestic outreach and public diplomacy abroad, he coined the concept of \\\"total diplomacy\\\", engaging and partnering with segments of the domestic audience to achieve foreign policy goals. McDowell has been connected with “panda diplomacy” for his work in bringing a pair of Chinese pandas to Canada, and “sports diplomacy” for his public diplomacy in Burma connected with the traditional Burmese sport of chinlone.\\nA conversation with McDowell forms one of the chapters of Chinese public diplomacy thinker Zhao Qizheng’s 2012 book “The Wisdom of Public Diplomacy: Cross-Border Dialogues”.\\nPrevious Diplomatic Career.\\nTaiwan.\\nMcDowell served in Taipei from 1997-2001.\\nAboriginal Affairs.\\nMcDowell was deputy Director for Aboriginal Affairs in the Global Issues Bureau from 2001-2003.\\nThailand.\\nMcDowell was posted in Thailand from 2003-2007. He managed the Embassy throughout the Indian Ocean Tsunami crisis. During McDowell’s term in Thailand, there was significant unrest in the south of the country, attributed to Muslim terrorist groups. McDowell led Canada’s engagement in this region.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125664\",\"title\":\"State of Things\",\"body\":\"\\nState of Things\\n\\nState of Things () is a 1995 Romanian drama film directed by Stere Gulea. The film was selected as the Romanian entry for the Best Foreign Language Film at the 69th Academy Awards, but was not accepted as a nominee.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125688\",\"title\":\"Murat Özçelik\",\"body\":\"\\nMurat Özçelik\\n\\nMurat Özçelik (born 1 January 1954) is a Turkish politician, civil servant and diplomat from the Republican People's Party (CHP), who has served as a Member of Parliament for İstanbul's third electoral district since 7 June 2015. He served as the Deputy Leader of the CHP responsible for foreign relations between 3 November 2014 and 11 November 2015. He was part of the CHP delegation during coalition negotiations with the Justice and Development Party following the June 2015 general election. Before entering politics, Özçelik served as the Turkish Consul General at Shanghai from 1997 to 1998, as the Turkish Ambassador to Iraq from 2009 to 2011 and as the Undersecretary of Public Order and Security from 2011 to 2012.\\nEarly life and career.\\nMurat Özçelik was born on 1 January 1954 in Ankara and graduated from TED Ankara College before going on to study at the Department of Management at the Middle East Technical University (OTDÜ) Faculty of Economics and Administrative Sciences. He is married with two children.\\nCivil service career.\\nÖzçelik started working at the Ministry of Foreign Affairs in 1983. Between 1990 and 1992, he served as the Cabinet Chief for Turgut Özal, the then-President of Turkey. After serving at numerous other positions, he became the department manager at the Office of Information in 2005, serving concurrently as the foreign affairs deputy spokesperson. On 16 November 2011, he was appointed as the Undersecretary of Public Order and Security. Following a public statement by Özçelik, in which he claimed that he was being pressrued into not doing his job the way he wanted and that he wanted to resign, the government removed him from the position on 17 May 2012. He was subsequently appointed as an advisor at the Foreign Ministry and subsequently retired there.\\nDiplomatic career.\\n2 January 1997, Özçelik became the Turkish Consul General at Shanghai, a position in which he served until 22 September 1998. On 26 December 2006, he became the Assistant Special Representative to Iraq. He became the Special Representative to Iraq in 2007 and was appointed as the Ambassador to Iraq in 2009. He was removed from this position on 22 October 2011 upon the decision of the government.\\nPolitical career.\\nÖzçelik was a candidate to become a member of the Party Council of the Republican People's Party (CHP) during the party's 18th Extraordinary Convention held on 7 September 2014. However, he failed to win enough votes to enter the council, resulting in the party leader Kemal Kılıçdaroğlu appointing him as a personal advisor. It was alleged that Özçelik had stated in a closed meeting that he had voted for the Peoples' Democratic Party (HDP), though had the support of Kılıçdaroğlu since it was perceived that Kılıçdaroğlu wanted to give the task of forming a strong foreign policy to Özçelik. Numerous members of the party council who won a higher number of votes subsequently resigned from their positions in order to allow Özçelik to take a seat, which he did in October 2014. On 3 November 2014, he was appointed as the Deputy Leader of the CHP responsible for foreign relations. In the June 2015 general election, he was elected as a Member of Parliament for İstanbul's third electoral district. During a Party Council meeting on 24 August 2015, it was alleged that Özçelik had an argument with rival council member Durdu Özbolat, developing into a fight between the two men. This, as well as the fact that Özçelik had not been elected through a nomination primary before the June 2015 election, allegedly contributed to Kılıçdaroğlu's decision to not put him forward as a candidate for the November 2015 general election. Özçelik subsequently resigned his Deputy Leader position, but withdrew his resignation shortly after. He resigned as Deputy Leader and from the Party Council on 11 November 2015.\\nControversy.\\nÖzçelik has been controversial within the CHP for allegedly being in favour of an independent Kurdistan on the southeastern borders of Turkey. He has also been described as having close sympathies with the pro-Kurdish Peoples' Democratic Party (HDP), whom he claimed he, his family and his friends voted for in the June 2015 general election.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125715\",\"title\":\"The Springing Tiger\",\"body\":\"\\nThe Springing Tiger\\n\\nThe Springing Tiger is a historical account of the Indian National Army published in 1959. Authored by Col Hugh Toye. The book was published in London by Cassell Publishers, and is considered one of the first Sympathetic Western accounts of the army. Toye worked as an intelligence officer in World War II in Burma, and was tasked with interrogating captured soldiers of the INA by the CSDIC(I). The book is provided with a foreword by Phillip Mason, who in 1946 was the Secretary of the War department in India. The book describes in detail the formation of the INA under the auspices of the F Kikan of Japanese intelligence through the collapse and subsequent revival of the army under Subhas Chandra Bose, its role in the Battles of Imphal and Kohima and the subsequent collapse in the face of Allied Burmese offensive before ending with the death of Subhas Chandra Bose.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125718\",\"title\":\"William M. Feigenbaum\",\"body\":\"\\nWilliam M. Feigenbaum\\n\\nWilliam Morris Feigenbaum (December 25, 1886 – April 23, 1949) was an American statistician, journalist and politician from New York.\\nLife.\\nHe was born on December 25, 1886, in Antwerp, Belgium, the son of Benjamin Feigenbaum (1859–1932) and Matilda (Kaminsky) Feigenbaum, both originally from Warsaw. The family emigrated to the United States and settled in Brooklyn where he attended the public schools and Boys High School. He graduated A.B. from Columbia College in 1907, and A.M. from Columbia University in 1908. He also took courses at Dartmouth College, Wisconsin University and National University School of Law. From 1909 to 1912, he worked in the Bureau of Statistics and Accounts of the Interstate Commerce Commission in Washington, D.C.. In 1912, he returned to New York and worked for the New York Public Service Commission (1st D.).\\nHe was a member of the Socialist Party of America. In November 1916, he ran for Congress in the 10th District, but was defeated by the incumbent Republican Reuben L. Haskell.\\nIn November 1917, he was elected to the New York State Assembly (Kings Co., 6th D.), defeating the incumbent Republican Nathan D. Shapiro. Feigenbaum polled 3,694 votes, Shapiro polled 3,184 votes, and Democrat Martin Solomon polled 2,217. Feigenbaum was one of ten Socialist members of the 141st New York State Legislature in 1918.\\nAfterwards he became the associated editor of \\\"The New Leader\\\", and wrote for several newspapers and political magazines.\\nIn 1930 and 1932, he ran for the New York State Senate (4th D.) but was defeated both times by Democrat Philip M. Kleinfeld. \\nHe died on April 23, 1949, at the Montgomery Nursing Home in Brooklyn.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125759\",\"title\":\"Iceni Academy (Colchester)\",\"body\":\"\\nIceni Academy (Colchester)\\n\\nIceni Academy is a junior school in the Shrub End suburb of Colchester. This school was formerly King's Ford Junior School but changed to an Academy status. Children from King's Ford Infant School are the main feeders on to this junior school.\\nHistory.\\nIceni Academy use to be King's Ford Junior School. However, governors and school representatives made the decision to convert King's Ford into an academy. During the time when the school was King's Ford, the school made poor progress and the school had a bad reputation with scoring low scores with Ofsted. \\nOfsted and School Problems.\\nKing's Ford were in and out of special measures. In 2012, they received Ofsted Satisfactory, however, in 2014, the school scored Inadequate and required special measures.\\nBright Future and Catchment.\\nIceni Academy hope to put away the bad memories and start fresh. The catchment area for Iceni Academy is within the Shrub End area. Most children after Year 6 would leave for Thomas Lord Audley School, Philip Morant School and College or the Stanway School.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125763\",\"title\":\"The Man Who Had His Hair Cut Short (film)\",\"body\":\"\\nThe Man Who Had His Hair Cut Short (film)\\n\\nThe Man Who Had His Hair Cut Short () is a 1966 Belgian drama film directed by André Delvaux, starring Senne Rouffaer and Beata Tyszkiewicz. It tells the story of a schoolteacher who falls in love with one of his students, and moves away in order to escape his infatuation. The film is based on the 1947 novel with the same title by Johan Daisne.\\nThe film was awarded the Sutherland Trophy at the 1966 BFI London Film Festival.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125766\",\"title\":\"Felix (1996 film)\",\"body\":\"\\nFelix (1996 film)\\n\\nFelix () is a 1996 Slovenian drama film directed by Božo Šprajc. The film was selected as the Slovenian entry for the Best Foreign Language Film at the 69th Academy Awards, but was not accepted as a nominee.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125787\",\"title\":\"Al-Rawda, Tartus\",\"body\":\"\\nAl-Rawda, Tartus\\n\\nAl-Rawda (; also spelled \\\"Rauda\\\") is a small town in northwestern Syria, administratively part of the Tartus Governorate. It is situated along the Mediterranean coast and just west of the Syrian Coastal Mountains in between Tartus (to the south) and Baniyas (to the north). According to the Syria Central Bureau of Statistics (CBS), al-Rawda had a population of 3,131 in the 2004 census. It is the administrative center of the Rawda Subdistrict (\\\"nahiyah\\\") which consisted of nine localities with a collective population of 11,688. Its inhabitants are predominantly Christians, from various denominations.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125794\",\"title\":\"Betzy Akersloot-Berg\",\"body\":\"\\nBetzy Akersloot-Berg\\n\\nBetzy Rezora Akersloot-Berg (16 December 1850, Aurskog - 18 December 1922, Oost-Vlieland) was a Norwegian-born seascape and landscape painter who spent most of her career on a small island in Friesland.\\nBiography.\\nShe was born to a landowning family. Later, they moved to Christiania where her father became a businessman. Originally, she trained as a nurse, then worked as a combination nurse and missionary among the Sami in Finnmark. However, she found herself attracted to painting and ultimately decided to take lessons at the \\\"Statens håndverks- og kunstindustriskole\\\", where she studied with Wilhelm von Hanno and Frits Thaulow. Later, she worked with Otto Sinding and followed him when he moved to Munich in Germany.\\nDuring a trip to Vienna, she saw some works by the Dutch marine painter, Hendrik Willem Mesdag, which greatly impressed her. In 1885, she had a chance meeting with him and his family. This led to studies with him at his workshop in The Hague. She became close friends with his wife, Sientje van Houten, who painted a portrait of her. In 1890, she studied briefly with Puvis de Chavannes in Paris.\\nThrough them, she met Gooswinus Gerardus Akersloot (1843-1929), the former mayor of Hoevelaken, who had recently lost his wife. They were married in 1893 and, three years later, settled in Oost-Vlieland where they bought the oldest house in town and named it \\\"Tromp's Huys\\\", after Admiral Cornelis Tromp. Although isolated, she travelled every summer and was able to participate in exhibitions throughout Western Europe as well as in Czechoslovakia. She remained there until her death in 1922. In addition to painting, she ran a Sunday School and a sewing society for girls.\\n\\\"Tromp's Huys\\\" became a museum in 1956. Most of her approximately 300 works are kept there and sent out for exhibits, including a major retrospective at the Noordelijk Scheepvaartmuseum in 1992, a special exhibition at her birthplace in Aurskog in 1996, and another at the Nordkappmuseet in Honningsvåg, near the place where she worked with the Sami, in 2004.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125812\",\"title\":\"2008–09 Glasgow Warriors season\",\"body\":\"\\n2008–09 Glasgow Warriors season\\n\\nThe 2008-09 season saw Glasgow Warriors compete in the competitions: the Magners Celtic League and the European Champions Cup, the Heineken Cup.\\nCompetitions.\\nMagners Celtic League.\\nResults.\\nThe all-Welsh fixtures were played mid-week to allow their teams to compete in the Anglo-Welsh Cup.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125836\",\"title\":\"Bwana (film)\",\"body\":\"\\nBwana (film)\\n\\nBwana is a 1996 Spanish drama film directed by Imanol Uribe. The film was selected as the Spanish entry for the Best Foreign Language Film at the 69th Academy Awards, but was not accepted as a nominee.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125846\",\"title\":\"Miles Thompson\",\"body\":\"\\nMiles Thompson\\n\\nMiles Thompson (born December 8, 1990) is an American professional lacrosse player who played for the University at Albany in NCAA Division I college lacrosse and plays for the Iroquois Nationals in international competition and the Florida Launch of Major League Lacrosse. He won the Tewaaraton Trophy in 2014 (co-winner with his brother Lyle). \\nEarly life.\\nMiles grew up in the Onondaga Nation, NY, to Doloris and Jerome Thompson. He was one of five children in the family, which included brothers Jeremy and Lyle, and cousin Ty; all four of whom played collegiate lacrosse at an elite level. Miles attended LaFayette High School, and was a standout in their Varsity lacrosse program. After graduation, Miles then attended the University of Albany in Albany, New York, and played alongside his cousin Ty. His brother Jeremy played lacrosse at Syracuse University, while his younger brother Lyle attended LaFayette High School.\\nCollege.\\nMiles (along with his brother Lyle) is the first Native American player to win the Tewaaraton Trophy; \\\"tewaaraton\\\" is the Mohawk term for the precursor of modern lacrosse.\\nAs a player for the Iroquois Nationals in the 2014 World Lacrosse Championship, Miles and Lyle helped the Nationals place third, their best-ever result in international competition.\\nFlorida Launch.\\nMiles Thompson was drafted in the third round, 20th overall in the 2014 MLL Draft by the Rochester Rattlers. He was then traded to the Florida Launch, and was on the active roster with them in their inaugural season. In 2015, the Launch selected Miles' brother Lyle first overall in the 2015 MLL Draft. The brothers' first game together was against the Ohio Machine. They both played a total of eight games for the Launch in the 2015. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125879\",\"title\":\"Policy Aptitude Test\",\"body\":\"\\nPolicy Aptitude Test\\n\\nThe Policy Aptitude Test (PAT) is an offline written test held in India. This test scores a person on the bases of \\\"General Knowledge\\\", \\\"Numerical Problem Solving\\\", \\\"Policy Aptitude\\\", \\\"Logical Reasoning\\\", \\\"English Language\\\" and \\\"Policy Analysis Ability\\\". The National Law School of India University, Bengaluru (NLSIU) started this exam and use the test for selecting students for its Public Policy Programme. The test is conducted every year.\\nExam format.\\nPolicy Aptitude Test (PAT) have two parts. First part is multiple choice section with negative marking. Second part is descriptive analytical writing about a policy issue. Only those who obtain minimum of 40% of marks in the first part will be evaluated for the second part.\\nFirst part of Policy Aptitude Test have five components: General Knowledge, Numerical Problem Solving, Policy Aptitude, Logical Reasoning and English Language. Each component have 20 questions. Each question carries 1 mark, and negative marking is -0.25. All components have equal weight.\\nSecond part of Policy Aptitude Test is for 50 marks and it concerns descriptive writing about an issue that is provided. The second part examine candidate's policy analysis ability as well as English comprehension. Both parts together, Policy Aptitude Test carries 150 marks. Total duration of Policy Aptitude Test is of two hours.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125890\",\"title\":\"Dual fluid reactor\",\"body\":\"\\nDual fluid reactor\\n\\nThe Dual Fluid Reactor (DFR) is the project of a private German research institute, the Institute for Solid-State Nuclear Physics, combining the advantages of the molten salt reactor with the ones of the liquid metal cooled reactor: The fuel is in a liquid metal or molten chloride salt solution, while the cooling is provided by liquid lead. As a fast breeder reactor, the DFR can burn both natural uranium and thorium, as well as recycle nuclear waste. Due to the high thermal conductivity of the molten metal, the DFR is an inherently safe reactor (the decay heat can be removed passively).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125891\",\"title\":\"J. Henry Bennett\",\"body\":\"\\nJ. Henry Bennett\\n\\nJ. Henry Bennett was a member of the Wisconsin State Senate.\\nBiography.\\nBennett was born on November 18, 1876 and died on April 29, 1956. He was buried in Viroqua, Wisconsin.\\nCareer.\\nBennett was first elected to the Senate in 1914. Additionally, he was District Attorney of Vernon County, Wisconsin. He was a Republican.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125896\",\"title\":\"Henriette (1803 ship)\",\"body\":\"\\nHenriette (1803 ship)\\n\\n\\\"Henriette\\\" was a French privateer commissioned in Bordeaux in late 1803. She served in the Bay of Biscay until mid-1804, and then in the Indian Ocean, based at Île de France (now Mauritius). The 74-gun HMS \\\"Powerful\\\" captured her in June 1806 off Ceylon.\\nFrench service.\\nCommissioned in late 1803 under Thomas Henry (or Henri), \\\"Henriette\\\" cruised in the Bay of Biscay until June 1804. She then crossed to Île de France, where she undertook three cruises, capturing several large British merchantmen.\\nSoon after his arrival at Île de France on 17 August, Henry left Port Louis on a cruise, only to have to return quickly, pursued by and .\\nHenry then embarked on 12 September on the first of two more successful cruises. Apparently some of \\\"Henriette\\\"s guns, two 12-pounder carronades, came from the East Indiaman \\\"Admiral Aplin\\\", which the French privateer \\\"Psyche\\\" had captured in January.\\nOn 12 October he captured the \\\"Faza-Soubany\\\" (or \\\"Fazzy Soubani\\\"), of 500 tons (bm), Fryer, master, sailing from Bombay to Bengal.\\nThen on 26 October he captured the \\\"Friendship\\\". \\\"Friendship\\\", of two guns and 380 tons, was carrying a cargo of rice, indigo, and cotton. \\nThe next day Henry captured the \\\"Sha Allum\\\", of two guns and 380 tons. She was carrying pepper, indigo, and cotton. Two days later he captured the \\\"Marguerite\\\", of two guns and 280 tons. She was carrying sugar, indigo, and cotton. \\nOn 12 November Henry captured the \\\"James Sybald\\\", of ten guns and 1,000 tons; she had a cargo of rice. (\\\"James Sibald\\\" had been sailing from Bengal to Bombay.) \\\"Henriette\\\" then returned to Port Louis on 10 December.\\nHenry and \\\"Henriette\\\" left on their second cruise on 9 January 1805. On 3 February they captured the East Indiaman \\\"Coromandel\\\". \\\"Coromandel\\\" was described as being of 450 tons and armed with fourteen 9-pounder guns. \\nIn May 1806 \\\"Lloyd's List\\\" reported that the French privateers \\\"Bellone\\\" (under Jacques François Perroud), \\\"Henriette\\\", and \\\"Caroline\\\" (under Nicolas Surcouf) had captured a number of merchantmen in the Bay of Bengal: \\nThe privateers gave up the \\\"Robust\\\" to their prisoners. She arrived at Bengal on 4 December 1805.\\nOn 17 September 1805 Henry captured the \\\"Viper\\\", of eight guns and 12 swivel guns. \\\"Lloyd's List\\\" reported that \\\"Henrietta\\\" had captured \\\"the East India Company's Brig the Viper\\\" at . A month later, on 13 November, \\\"Henriette\\\" captured the \\\"Phoenix\\\", of 600 tons. \\\"Henriette\\\" returned to Port Louis on 26 March 1806.\\nIn April 1806, command of \\\"Henriette\\\" passed to Auguste Sagory. \\\"Henriette\\\" left Port Louis on 7 April, and on 6 May captured the \\\"Dawetz-Nissaint\\\" on 6 May.\\nFate.\\nOn 13 June HMS \\\"Powerful\\\" captured \\\"Henriette\\\" off Trincomalee, Ceylon (now Sri Lanka). \\\"Powerful\\\" had received intelligence of her presence in the area and had set out from Trincomalee on the 11th and had sighted \\\"Henriette\\\" on the morning of the 13th. After an 11-hour chase, during which \\\"Henriette\\\" fired her stern guns at \\\"Powerful\\\" without effect, \\\"Powerful\\\" succeeded in catching up to her quarry, which surrendered without further combat. During the chase, \\\"Henriette\\\"s crew had thrown four of her 6-pounder guns overboard in an attempt to lighten her and so gain speed. Head money was paid for \\\"Henriette\\\" in January 1814.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125909\",\"title\":\"Mister Transmission\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125912\",\"title\":\"Adriana Pop\",\"body\":\"\\nAdriana Pop\\n\\nAdriana Pop (born Adriana Rednic; October 22, 1965) is a French-Romanian gymnastics choreographer and former rhythmic gymnast.\\nCareer.\\nAdriana Rednic was born on October 22, 1965 in the Romanian city of Baia Mare. After taking dancing classes for six years, Adriana was directed into artistic gymnastics classes, joining a club in her city. Afraid of the uneven bars and vault, Pop quickly found that dancing her gift; rather than apparatus gymnastics. Soon after, she was directed into rhythmic gymnastics. At the age of thirteen, in 1979, she joined the Romanian National team in Bucharest.\\nPop states, \\\"The music is important for me because from the music comes the idea for the routine.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125944\",\"title\":\"It Could Happen to You (1939 film)\",\"body\":\"\\nIt Could Happen to You (1939 film)\\n\\nIt Could Happen to You is a 1939 American comedy film directed by Alfred L. Werker and written by Lou Breslow and Allen Rivkin. The film stars Stuart Erwin, Gloria Stuart, Raymond Walburn, Douglas Fowley, June Gale and Clarence Kolb. The film was released on June 8, 1939, by 20th Century Fox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125964\",\"title\":\"Raïs Neza Boneza\",\"body\":\"\\nRaïs Neza Boneza\\n\\nRais Neza Boneza is a Congolese writer and poet (1979). \\nHe was born in the Katanga province in Democratic Republic of Congo (Former Zaïre). He is the author of fiction, poetry, articles and academic materials. He is also a peace activist and practitioner. For his contribution to peace and conflict transformation, he was awarded an honorary doctorate degree (Honoris causa) from the Institute of management sciences (ISGM) and the Universite du CEPROMEC in Burundi 2008. He hold a BA in Social Sciences and Master in Humanities from the Derby University.\\nHe is co-convener for Africa of the TRANSCEND Global Network; a Peace Development Environment Network. His debut novel \\\"“White Eldorado, Black Fever”(2013)\\\" is the only work which has been originally translated from his French native language “ \\\"Eldorado blanc, Fievre Noire”(2013)\\\". In his debut novel, he taps in his artistic background, a peace researcher and practitioner to create a work of fiction and fact to bring in the awareness about the conflict-resources direct effects to communities in the Great-Lakes region of Africa.\\nReferences.\\nAuthor website\\nGLOBAL TORMENT\\nLa liste d'ecrivains africains par pays\\nTRANSCEND Network for Peace and Development\\nRais Neza Boneza, Norwegian\\nTV show:Emission Majuscule/15 January 2015\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125965\",\"title\":\"Skoronski\",\"body\":\"\\nSkoronski\\n\\nSkoronski is a surname. Notable people with the surname include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125982\",\"title\":\"Kōriyama City Museum of Art\",\"body\":\"\\nKōriyama City Museum of Art\\n\\n opened in 1992 in Kōriyama, Fukushima Prefecture, Japan. The collection includes works by Gainsborough, Constable, Turner, Burne-Jones, and Waterhouse, Shiba Kōkan, Takahashi Yuichi, Fujishima Takeji, and Kishida Ryūsei, as well as of artists associated with Kōriyama.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48125985\",\"title\":\"Ednah Chepngeno\",\"body\":\"\\nEdnah Chepngeno\\n\\nEdnah Chumo Chepngeno (born 15 July 1977) is a former Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126024\",\"title\":\"Margaret Indakala\",\"body\":\"\\nMargaret Indakala\\n\\nMargaret Indakala (born 24 August 1962) is a former Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126036\",\"title\":\"Jacqueline Makokha\",\"body\":\"\\nJacqueline Makokha\\n\\nJacqueline Makokha or Jackline Makokha (born 15 November 1974) is a former Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126044\",\"title\":\"PS Gael (1867)\",\"body\":\"\\nPS Gael (1867)\\n\\nPS \\\"Gael\\\" was a passenger vessel operated by the Great Western Railway from 1884 to 1891 \\nHistory.\\nThis paddle steamer was launched on 9 March 1864 and completed completed on 11 February 1867 She was named by Miss Minnie Galbraith, daughter of Andrew Galbraith Esq, Johnstone Castle, ex-Provost of Glasgow and spent most of her years in Scotland. She was owned by the Clyde and Campbeltown Steam Packet Joint Stock Company.\\nShe was bought in 1884 and operated by the GWR, mainly on its Weymouth routes but also for a time at Milford Haven and from 1887 - 1889 at Penzance for the West Cornwall Steam Ship Company. In 1891 she returned to the Clyde for duties on routes from Glasgow to Oban, Tobermory and Gairloch.\\nShe was scrapped in 1924.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126051\",\"title\":\"Tatyana Gordeyeva\",\"body\":\"\\nTatyana Gordeyeva\\n\\nTatyana Vladimirovna Gordeyeva (; born June 3, 1973 in Volgograd) is a retired Russian heptathlete. She has won a total of two medals, a silver and a bronze, in heptathlon at the European Cup Super League, and has been selected to compete for Russia at the 2004 Summer Olympics, but later withdrew from the meet after falling at one of the hurdles in the opening heat. Gordeyeva trained under the tutelage of head coach Mikhail Zatselyapin for the national track and field team in combined events, while serving as a member of the Russian Army in her native Volgograd.\\nGordeyeva qualified for the Russian squad, along with her teammates Yelena Prokhorova and Svetlana Sokolova, in the women's heptathlon at the 2004 Summer Olympics in Athens. She attained the IAAF Olympic \\\"A\\\" standard and a season best of 6235 points at the national meet in Tula to book her place on the Russian team in track and field. Coming to the Games with a number of sustained injuries, Gordeyeva attempted to clear one of the hurdles, and instead crashed straight into it. Thus, she did not finish the 110-metre hurdles heat, and later withdrew from the competition.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126098\",\"title\":\"Dorcas Nakhomicha Ndasaba\",\"body\":\"\\nDorcas Nakhomicha Ndasaba\\n\\nDorcas Nakhomicha Ndasaba (born 31 March 1971) is a retired Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126121\",\"title\":\"Nancy Waswa\",\"body\":\"\\nNancy Waswa\\n\\nNancy Lusanji Waswa (born 28 December 1971) is a retired Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126128\",\"title\":\"Political Science (disambiguation)\",\"body\":\"\\nPolitical Science (disambiguation)\\n\\nPolitical Science is a social science dealing with politics and systems of government.\\nPolitical Science may also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126130\",\"title\":\"Doris Wanjala\",\"body\":\"\\nDoris Wanjala\\n\\nDoris Wanjala-Wefwafwa (24 December 1966 – 11 December 2007) was a Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126134\",\"title\":\"Deborah Frances-White\",\"body\":\"\\nDeborah Frances-White\\n\\nDeborah Frances-White is a comedian and writer who regularly delivers seminars to women in business on subjects including charisma, diversity and inclusion. She has both British and Australian citizenship.\\nEarly life.\\nDeborah was born in Australia and adopted at ten days old. She moved to the UK and attended Oxford University and founded improv theatre company The Spontaneity Shop of which she is still a director.\\nReligion.\\nDeborah became a Jehovah's Witness while still a teenager. Her years in the religion and how she left it were the focus of her 2012 Edinburgh Fringe stand-up comedy show and one of the episodes of her BBC Radio 4 show \\\"Deborah Frances-White Rolls the Dice\\\".\\nCareer.\\nAfter developing a number of improvisation formats at The Spontaneity Shop (including the improvised romantic comedy DreamDate which had a pilot made for ITV) Deborah turned to stand-up comedy. Her first significant solo show was \\\"How to Get Almost Anyone to Want to Sleep With You\\\" which she performed at The Edinburgh Festival Fringe in 2007 and at The Melbourne International Comedy Festival in 2008 where she also hosted The Melbourne International Comedy Festival Roadshow.\\nDeborah's recent shows have been more personal. \\\"Cult Following\\\" (2012) dealt with her experiences as a teenage Jehovah's Witness, \\\"Half a Can of Worms\\\" (2013) was about tracking down her biological family and \\\"Friend of a Friend of Dorothy\\\" (2015) was about feminism, sexism and homophobia.\\nDeborah has continued to develop new improvisation formats. \\\"Voices in Your Head\\\" is a show which allows comedians, improvisers and actors to create comedy characters while the audience watches. Guests have included Phill Jupitus, Sara Pascoe, Russell Tovey, Mike McShane, Hannibal Buress and others. In 2015 she created \\\"The Beau Zeaux\\\" a long-form improvised comedy featuring a rotating cast including Marcus Brigstocke, Thom Tuck, Rachel Parris, Brendan Murphy, Ed Coleman, Milly Thomas and Pippa Evans. Guests have included Russell Tovey and Dan Starkey.\\nHer BBC Radio 4 series \\\"Deborah Frances-White Rolls the Dice\\\" was broadcast in spring 2015 and featured stories about her adoption, green card marriage, and quest to find her biological family. The episodes were titled Half a Can of Worms, Cult Following, Visa Issues and Who's Your Daddy? In January 2016, The Writers Guild of Great Britain awarded Deborah Best Radio Comedy at their annual ceremony.\\nCorporate Work.\\nDeborah also regularly appears at corporate events speaking about confidence, charisma, diversity and sexism. Her TEDx talk on Charisma vs Stage-Fright was cited by James Caan as the secret of his presenting skills.\\nWriting.\\nDeborah is also a screenwriter with commissions from Fox Searchlight, Redwave Films, FremantleMedia, ITV Studios, the BBC and Channel 4. With her writing partner Philippa Waller, she contributed an episode of Young Dracula in 2014. She has co-written two books: \\\"The Improv Handbook\\\" with Tom Salinsky and \\\"Off the Mic\\\" with Marsha Shandur both published by Bloomsbury. Deborah writes for Standard Issue Magazine.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126135\",\"title\":\"Ithinumappuram\",\"body\":\"\\nIthinumappuram\\n\\nIthinumappuram is a Malayalam film released on 18 September 2015 by Agna Media. It is directed, written and produced by Manoj Alunkal. It stars Riyaz Khan and Meera Jasmine in the lead roles.\\nPlot.\\nThe film follows a girl named Rukmini Nair, played by Meera Jasmine, who is born and brought up in an affluent family. Rukmini's horoscope indicates she is a girl with \\\"Chovva Dosham\\\", meaning she was born under a bad sign. When she marries someone her husband will die soon, and so her family must work very hard to find a groom. By luck, Rukmini's family finds a match. However, Rukmini falls in love with a man who works in her home, Karthikeyan (played by Riyaz Khan) and elopes with him against her family's wishes.\\nKarthikeyan had set his eyes on Rukmini's inheritance from her family. When he discovers that Rukmini’s family has disinherited her, he becomes enraged and reveals his true colors. Rukmini bore his two children, but is subjected to severe ill treatment by her husband. Though she is strong-willed, she silently bears the abuse and domestic violence. Eventually, Karthikeyan disavows her and elopes with another young woman named Devu, who works alongside him in a factory. Meanwhile, Rukmini is left pregnant with Karthikeyan's third child. The film then follows Rukmini as she deals with subsequent hardships. \\nCast.\\nCast of the movie:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126151\",\"title\":\"Emily Wesutila\",\"body\":\"\\nEmily Wesutila\\n\\nEmily Wesutila (born 8 March 1973) is a retired Kenyan female volleyball player. She was part of the Kenya women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126160\",\"title\":\"Clyde Docks Preservation Initiative\",\"body\":\"\\nClyde Docks Preservation Initiative\\n\\nThe Clyde Docks Preservation Initiative is a non-profit organisation set up in 2015 to establish a lead organisation in efforts to preserve the derelict Govan Graving Docks, Glasgow, Scotland (and other maritime sites on the River Clyde) as a heritage asset for future generations. The organisation aims to restore Govan graving docks as a shipbuilding and maritime heritage park in recognition of the maritime and shipbuilding heritage of the city.\\nGovan Graving Docks has been abandoned since closing down in 1987 and while a number of redevelopment proposals for the site have been put forward since none have yet come to fruition.\\nThe organisation is working to remedy what it sees as a lack of protection and recognition being afforded to various sites along the River Clyde. Glasgow once led the world in shipbuilding but now the only notable remaining trace of the city's industrial past is the Finnieston Crane and the remaining shipyards at Govan and Scotstoun.\\nAs well as preserving the heritage CDPI aims to create a social enterprise / micro-enterprise hub, a cultural quarter and an ecology park area at Govan graving docks along with full restoration of the dry docks to working order and a facility for maintenance of historic ships.\\nThe organisation has attracted attention in local and national media, most notably articles in The Sunday Herald, The Big issue magazine and a previous article in the Maritime Journal \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126162\",\"title\":\"Fernand Ouellette\",\"body\":\"\\nFernand Ouellette\\n\\n Fernand Ouellette is a French Canadian writer. He is a four-time winner of the Governor General's Awards, having won the Governor General's Award for French-language non-fiction at the 1970 Governor General's Awards for \\\"Les actes retrouvés\\\" and at the 1976 Governor General's Awards for \\\"Les Bas Canada 1791-1840, changements structuraux et crise\\\", the Governor General's Award for French-language fiction at the 1985 Governor General's Awards for \\\"Lucie ou un midi en novembre\\\", and the Governor General's Award for French-language poetry at the 1987 Governor General's Awards for \\\"Les Heures\\\".\\nLife.\\nHe was born in Montreal on 24 September 1930.\\nOuellette's papers are kept at the National Library and Archives of Canada in Ottawa.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126177\",\"title\":\"Fiorella Aíta\",\"body\":\"\\nFiorella Aíta\\n\\nFiorella Aíta (born 13 July 1977) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126188\",\"title\":\"Milagros Cámere\",\"body\":\"\\nMilagros Cámere\\n\\nMilagros Camere (born 22 September 1972) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126207\",\"title\":\"Patricia Soto\",\"body\":\"\\nPatricia Soto\\n\\nDiana Patricia Soto (born 10 February 1980) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126224\",\"title\":\"Milagros Jessenica Uceda\",\"body\":\"\\nMilagros Jessenica Uceda\\n\\nMilagros Jessenica Uceda (born 14 August 1981) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126249\",\"title\":\"1939 Notre Dame Fighting Irish football team\",\"body\":\"\\n1939 Notre Dame Fighting Irish football team\\n\\nThe 1939 Notre Dame Fighting Irish football team represented the University of Notre Dame during the 1939 college football season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126256\",\"title\":\"Janet Vasconzuelos\",\"body\":\"\\nJanet Vasconzuelos\\n\\nJanet Daria Vasconzuelo or Janet Daria Vasconzuelos (born 4 July 1969) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126272\",\"title\":\"Muthuswamy\",\"body\":\"\\nMuthuswamy\\n\\nMuthuswamy is a Tamil name and may refer to\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126278\",\"title\":\"Mama (Emily Wells album)\",\"body\":\"\\nMama (Emily Wells album)\\n\\nMama is a studio album by Emily Wells. Released in 2012 through Partisan Records, it was largely well-received by critics. Bring the Noise called the release a \\\"fine example of what can happen when someone ignores the traditional boundaries of music genres.\\\" An acoustic version of \\\"Mama\\\" was released in on June 11, 2013 in the United States and United Kingdom through Partisan Records.\\nCritical reception.\\nMusic OMH gave \\\"Mama\\\" a score of 4/5/5, writing that the release \\\"has a freshness and vitality about it that proves startling.\\\" Beyond the songwriting, \\\"Impose Magazine\\\" praised her vocals in the album, writing that it was \\\"a voice that’s made precious while somehow also being distorted, raspy, repressed and hushed.\\\" \\nGiving it a score of 80/100, In Your Speakers called the \\\"cathartic,\\\" writing that \\\"although Wells does not emulate her jazz and folk influences, she maintains the rawness and the passion of them, one being Bob Dylan.\\\" Bring the Noise praised the release, giving it 8/10 and calling it both \\\"one of the most interesting albums of the year so far,\\\" and a \\\"fine example of what can happen when someone ignores the traditional boundaries of music genres.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126288\",\"title\":\"Yulissa Zamudio\",\"body\":\"\\nYulissa Zamudio\\n\\nYulissa Noelia Zamudio Orl (born 24 March 1976) is a retired Peruvian female volleyball player. She was part of the Peru women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 11th. She was part of the Peru women's national volleyball team at the 2010 FIVB Volleyball Women's World Championship in Japan. She played with Alianza Lima.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126294\",\"title\":\"Surasawadee Boonyuen\",\"body\":\"\\nSurasawadee Boonyuen\\n\\nSurasawadee Boonyuen (, born October 31, 1991 in Udonthani) is an Thai indoor volleyball. She is a member of the Thailand women's national volleyball team.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126296\",\"title\":\"2009–10 Glasgow Warriors season\",\"body\":\"\\n2009–10 Glasgow Warriors season\\n\\nThe 2009-10 season saw Glasgow Warriors compete in the competitions: the Magners Celtic League and the European Champions Cup, the Heineken Cup.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126309\",\"title\":\"Ayesha Dutt\",\"body\":\"\\nAyesha Dutt\\n\\nAyesha Shroff (born 5 June 1960) is a model, Bollywood actress and Film producer. She is the wife of popular Bollywood actor, Jackie Shroff and mother of Bollywood actor, Tiger Shroff.\\nLife.\\nShe was born to a Bengali, Ranjan Dutt, an Air vice marshall in Indian Air Force, and Claude Marie Dutt De Cavey, a Belgian.\\nAyesha contested as the Miss Young World contest at Manila. She did not make it to the finals, but was elected the most popular girl at the contest by her fellow contestants. She began her career as a model and became successful.\\nShe acted in a Bollywood film. She managed her 10% stake in Sony TV, which she sold in 2013.\\nPersonal life.\\nShe married her longtime boyfriend and Bollywood actor, Jackie Shroff on her birthday on 5 June 1987. She later turned into a film producer. The couple run a media company, Jackie Shroff Entertainment Limited. They jointly owned 10% shares in Sony TV since its launch until 2012 when they sold their stake and ended their 15-year-long association with Sony TV. The Shroffs have two children. They have two children, her elder son is Bollywood actor, Tiger Shroff (born 1990) and a daughter, Krishna (born 1993). \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126318\",\"title\":\"Chung Sun-hye\",\"body\":\"\\nChung Sun-hye\\n\\nChung Sun-Hye or Jeong Seon-Hye (; born 17 December 1975) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126320\",\"title\":\"Seifudein Adem\",\"body\":\"\\nSeifudein Adem\\n\\nSeifudein Adem is a political economist from Ethiopia. He is the Associate Director of Institute of Global Cultural Studies at Binghamton University, where he has been teaching international relations and international political economy with a particular focus on Africa and Asia.\\nEducation.\\nSeifudein Adem graduated from Addis Ababa University in 1988, with a B.A. (Distinction), majoring in political science. He then left for Japan for further education, where he earned an M.A. in international relations from International University of Japan in 1994, and a Ph.D. in international political economy from University of Tsukuba in 1999.\\nSeifudein Adem is proficient in English, Japanese, Russian, Afaan Oromo and Amharic. He also speaks some Chinese.\\nCareer.\\nSeifudein Adem served as an assistant lecturer at Addis Ababa University (formerly known as Haile Selassie I University), Ethiopia, from 1989 to 1992, and as a graduate teaching assistant at University of Tsukuba, Japan, from 1995 to 1999.\\nAfter serving as a Foreign Scholar for two years, he was appointed as an Assistant Professor at University of Tsukuba in 2001. He held the positions of Foreign Scholar and Foreign Professor at University of Tsukuba until he completed his assignment in Japan in December 2005. He had also worked briefly at the United Nations University (UNU) in Tokyo as a researcher and an invited moderator /coordinator of UNU Global Seminars.\\nIn January 2006, Seifudein Adem arrived at Binghamton University in Binghamton, New York, to work with Ali Mazrui at the Institute of Global Cultural Studies (IGCS). IGCS was founded by Mazrui in 1991. Adem was appointed as the Associate Director of IGCS also in 2006, with a joint teaching appointment in the Department of Political Science.\\nWorks.\\nSeifudein Adem's fields of interest include international relations of Africa, interactions between Africa and Asia and history and political economy of Ethiopia.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126348\",\"title\":\"Eoh Yeon-soon\",\"body\":\"\\nEoh Yeon-soon\\n\\nEoh Yeon-soon (; born 12 December 1973) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126381\",\"title\":\"Kim Guy-hyun\",\"body\":\"\\nKim Guy-hyun\\n\\nKim Guy-hyun (; born 12 January 1975) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126385\",\"title\":\"C. J. Cochran\",\"body\":\"\\nC. J. Cochran\\n\\nCarl Howard \\\"C. J.\\\" Cochran, Jr. (born 17 September 1991) is an American professional soccer player plays as a goalkeeper for Atlanta Silverbacks in the North American Soccer League.\\nCochran was born in Alpharetta, Georgia and played his earlier career with Georgia State Panthers before starting his professional career with Atlanta Silverbacks in 2015.\\nPlaying career.\\nEarly career.\\nCochran started his career with a successful four years at Georgia State Panthers, where he achieved such records as posting the season's all-time record goals-against average of 1.33. Cochran also ended his National Collegiate Athletic Association career with the competition's third-most clean sheets of 11 and with the third-best single-season goals-against average of 1.47 in 2014. While at Georgia State, Cochran also won numerous awards during his time there which included the Sun Belt and College Sports Madness Independent Defensive Player of the Week awards and two Colonial Athletic Association Rookie of the Week awards. In total, Cochran played 4,345 minutes of action with Georgia State which involved 48 starts. With 188 saves during his time there, Cochran comes in the top-10 for most career saves with the Panthers.\\nAtlanta Silverbacks.\\nIn 2015, Atlanta Silverbacks offered Cochran a trial at the club. After starting pre-season as a trialist, Cochran signed a professional contract with the club. On 4 April 2015, Cochran made his debut for the Atlanta Silverbacks against Indy Eleven in a match that ended 1–1. On 7 July, Cochran was nominated for the NASL Play of the Week award for his save against Martin Nuñez of the Tampa Bay Rowdies.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126401\",\"title\":\"Vladimir Tsyganko\",\"body\":\"\\nVladimir Tsyganko\\n\\nVladimir Vladimirovich Tsyganko (; ; also Țâganco, Tziganco, Tziganko or Țiganco; 1887 – 1937) was a Bessarabian, and later Soviet, politician. The son of a distinguished architect, and himself an engineer by vocation, Tsyganko entered politics shortly before the proclamation of a Moldavian Democratic Republic, when he earned a seat in the republican legislature (\\\"Sfatul Țării\\\"). He sided with the parliamentary Peasants' Faction, which supported left-wing ideals and pushed for land reform, being generally, and radically, opposed to the more right-wing Moldavian Bloc. Tsyganko was skeptical of the Bloc's plan to unite Bessarabia with Romania, although he possibly supported a federation. His uncompromising stance divided his Faction and led the Romanian Kingdom's authorities to identify him as a major obstruction to the unionist cause. \\nIn November 1918, as the Bloc switched its support to unconditional unification and dissolved the regional government bodies, Tsyganko rejected the new regime and moved to Odessa. Allying himself to members of the White movement, with whom he set up a Committee for the Salvation of Bessarabia, attending the Paris Peace Conference to campaign for the reversal of the union. He later settled in Soviet territory, where he helped create a Moldavian Autonomous Soviet Socialist Republic; other members of his family opted to stay behind in Romania. In 1937, Tsyganko fell victim to the Great Purge.\\nBiography.\\nEarly career.\\nTsyganko was born in Kishinev (Chișinău), regional capital of the Bessarabian Governorate, Russian Empire, and was a graduate of Riga Polytechnicum. He returned to his native city where his father Vladimir (? – 1919), an architect, designed such landmarks as the Ethnography Museum and Saint Nicholas Church; his brother Nikolai (Nicolai) Vladimirovich (born 1882) was the \\\"zemstvo\\\" engineer in Orhei, and from 1909 in Kishinev itself. By 1904, their father was the Director of Monastery Estates in Bessarabia, in which capacity he testified against Russian police after the Kishinev pogrom, accusing them of passivity. In 1916, the Bessarabian journalist Alexis Nour described Tsyganko Sr. as a \\\"much esteemed [...] Bessarabian intellectual of a Moldavian nationalist hue, but not a separatist\\\" (\\\"see Moldovenism\\\"). Also according to Nour, Nikolai, whom he met personally, could speak only Russian. Romanian politician Duiliu Zamfirescu, who met and debated with Vladimir Vladimirovich in 1918, claimed that the Tsygankos were \\\"Ruthenian\\\". He and his adversary talked in French, as Tsyganko \\\"could not speak a word of Romanian\\\".\\nTsyganko reached political prominence after the October Revolution, which had left Bessarabia to administer itself independently, as a \\\"Moldavian Democratic Republic\\\". In January 1918, the local Soviet of Workers' and Soldiers' Deputies began to override the \\\"Sfatul Țării\\\" assembly (appointed the previous November) and attempted to bring about Bolshevik rule. This move was swiftly suppressed by a punitive expedition of the Romanian Army. The 3rd Peasants' Congress, assembled few days after the occupation of Chișinău, adopted an anti-secessionist position, dismissed the Moldavian prime-minister Pantelimon Erhan from the position of President of the Peasants' Soviet, and elected a new leadership from among the most vocal opponents of the Romanian intervention. According to the Rumcherod's newspaper, during the opening session, Tsyganko's message on behalf of the local Socialist Revolutionary branch was met with applause and calls to support the Russian Revolution. The following day, after demanding the withdrawal of Romanian troops within 24 hours, and negotiating on the issue with the Romanian military, the Congress' Presidium was put under arrest. General Ernest Broșteanu dismissed the immunity of those Peasants' representatives who were also members of \\\"Sfatul\\\", and issued a strong warning against further anti-Romanian agitation. Consequently, the following days the Congress selected a new list of \\\"Sfatul\\\" representatives, headed by Tsyganko, which comprised mostly moderates.\\nClashes with the unionists.\\nTsyganko, who was counted among the representatives of the Russian minority, affiliated with the left-wing \\\"Peasants' Faction\\\", which stood in opposition to the \\\"Moldavian Bloc\\\" of Romanian nationalists. During the debates on land reform, he suggested postponing the discussion until a new government, \\\"representative of the people's will\\\", would be approved by a Moldavian Constituent Assembly. Presumably, he feared that pressure from the Romanian troops would affect the extent of the reform. Nevertheless, he became the first chairman of \\\"Sfatul\\\"s Agrarian Commission, and in parallel presided upon the Peasants' Soviet. Despite being involved in left-wing politics, Tsyganko would gradually develop a working relationship with Andrei Krupenski, the Polono-Bessarabian landowner and ex-Marshal of Nobility, and Alexandr K. Schmidt, who stood for the conservative side of anti-Romanian agitation; between 1918 and 1920 the three men issued calls for the end of Romanian occupation, and began popularizing their cause in Europe. \\nWhen Romanian Premier Alexandru Marghiloman traveled to Bessarabia to canvass for the unionist cause, he found the Peasant Faction divided between followers of Ion Inculeț, who endorsed the Romanian viewpoint, and deputies who sided with Tsyganko. Zamfirescu, who traveled with Marghiloman, recalls that Tsyganko \\\"thrice in one month\\\" attempted to recall the Republic's Directorate, his moves resisted by Inculeț. He also protested the selection of pro-Romanian students from Kiev and Odessa as representatives of the Transnistrian Moldavians, in which he saw efforts to shake the balance of power inside the \\\"Sfatul\\\". Zamfirescu claims to have saved Tsyganko from an undisclosed mortal danger, and then to have conversed with him, trying to gain insight into his political motivations. The latter, he concluded, were \\\"most phantasmagorical socialist ideas\\\", not dissuaded by the prospect of \\\"death, suffering, military disaster, sheer destitution, or degeneracy\\\". He adds: \\\"It was late at night, I was experiencing chills, and so I believe I have insulted the convictions of this visionary youth, reassuring him that all opinions lead to a ministerial chair, provided one makes sure to discard them on cue.\\\"\\nDespite proclaiming its independence in late February 1918, the Moldavian Republic was still seen in various circles as subordinate to the neighboring Ukrainian People's Republic. Its Central Rada wished to represent Bessarabia in the preliminary negotiation of the Bucharest peace treaty, imposed by the Central Powers on Romania. The Moldavian Regional assembly reacted by reaffirming its independence and rejecting the division of Bessarabia, against the decisions of Akkerman and Khotin \\\"zemstva\\\", which had proclaimed their accession to the Ukraine. A Moldavian delegation was therefore selected to head to Kiev and obtain from the Central Rada official recognition of Moldavia's independence. The delegation, which included the interior minister Vladimir Cristi, the nationalists Nicolae Secară and Teodor Neaga, and Tsyganko as representative of the Peasants' Faction, was prevented from leaving. According to the unionist Gheorghe Andronachi, it was Daniel Ciugureanu, the Republic's pro-Romanian prime minister, who intervened with the Romanian Army to hamper the departure, fearing that an international recognition of independence would hinder nationalist plans for union with Romania.\\nMarch union vote.\\nOn March 27, 1918, when \\\"Sfatul\\\" voted to support the union with Romania, Tsyganko effectively abstained. Zamfirescu found it \\\"unbelievable\\\" that Romanian-speaking peasants had ever endorsed Tsyganko, who, he claimed, \\\"systematically opposes Bessarabia's government and Romania's policies, endeavoring for its annexation to the Ukraine\\\"; however, he also notes that Tsyganko himself accused the Romanians of wanting to hand in Bessarabia to the Ukrainians. A radical project for land reform had received pledges of support from \\\"Sfatul\\\" secretary Ion Buzdugan, and also from the Marghiloman himself; consequently, according to historian Alberto Basciani, Tsyganko's critique of unionism became marginal within his own party and Soviet.\\nOn behalf of the Peasant Faction, Tsyganko denied the Assembly had the authority to discuss such and issue, declaring his group would refrain from voting, since they considered this a matter for a Constitutional Convention; furthermore, he stated the only admissible terms for a union between the Moldavian and Romanian peoples would be in a federation. Five members of his faction decided to side with the nationalists and voted for the union, while the other 17 present abstained. Researchers are divided in their assessments of Tsyganko's political stance at that early stage. Basciani describes him as one of those who \\\"opposed with great vehemence the union of Bessarabia with Romania\\\". However, according to Svetlana Suveică, Tsyganko did not object to union with autonomy, and in fact saw it as \\\"the only solution for avoiding the Bolshevik invasion of the region.\\\"\\nIn November, after the generalization of Romanian military rule in Bessarabia, Tsyganko, as putative \\\"president\\\" of the Bessarabian Peasants' Party, with Nicolae Alexandri, Ion Păscăluță, and 37 other \\\"Sfatul\\\" members, sent a letter of protest to the Romanian government of Constantin Coandă. This coalition of Romanian Bessarabians and White Russians demanded the immediate recognition and restoration of autonomy, as well as the lifting of the martial law; however, its imperatives were rejected as illegitimate by the central authorities. According to Clark: \\\"We cannot but applaud the admirable aims of the 40 Deputies, in most of their requests; but at the same time we must wonder at their ingenuousness; they did not foresee the constant turbulence on the Eastern frontier, which even at that time impressed the Roumanians\\\".\\nNovember union vote.\\nThe protest arose controversy in political circles. Tsyganko reported a private interview with the Romanian envoy Artur Văitoianu. He quoted the latter as offering a deal: \\\"You must renounce [autonomy] if only for this sole reason—that you no good Roumanian officials in Bessarabia—that is to say, none who are good nationalists. If you give up autonomy, you will not have a Commissioner-General, but you will have a Bessarabian Chargé d'Affaires, a man of your own character, who will be nominated by the Central Power. The new [Bessarabian] Directorate will remain in office until the meeting of the Pan-Roumanian Constitutional Assembly. Does this appeal to you as attractive?\\\" Also according to Tsyganko, Văitoianu informed the group that they needed to coalesce with Romanian nationalists in front of Great Russian revivalism, and that \\\"the national idea takes precedence over everything\\\", implicitly threatening \\\"Sfatul\\\" dignitaries.\\nAs Suveică writes, it was only at this stage that Tsyganko became an adversary of the unionist camp, placing his hopes in a reestablishment of the Russian Republic, and her re-annexation of Bessarabia. At the last \\\"Sfatul\\\" session, on November 25, 1918, unconditional union was proposed for ratification, as one of several measures being voted on, alongside the allocation of offices and a land-reform-law. According to the Peasants' Faction account, also supported by many of the Moldavian nationalists who had signed the earlier protest, the Moldavian Bloc kept the opposition uniformed about there being a \\\"Sfatul\\\" session: \\\"only Mr. V. Tziganko was aware of the fact, and he was informed privately, two hours before the opening of the sitting.\\\" The Tsyganko group confronted the assembly's president, Halippa, arguing that his election was illegal. They announced another walk-out, to which voices of the Moldavian Bloc responded with rhetorical questions (\\\"Is this how you intend to solve the agrarian issue?\\\") and taunts of \\\"Good riddance!\\\"\\nThe opposition maintained that the walk-out resulted in a lack of quorum: only 48 of 160 deputies were reportedly present, which made the voting results questionable. Tsyganko and his colleagues accused the Bloc of putting up unconditional union for the vote as a rider, at 2.30 AM on the morning of November 26, and counted the votes during considerable and purposeful commotion. Some of the Fraction deputies in the opposition, including Tsyganko and Gavril Buciușcan, actually returned in time to cast their Nay votes. Clark claims that one of the Moldavian Bloc representatives testified that there were enough deputies present. According to Moldavian Bloc's Pan Halippa, Tsyganko's walk-out from the Assembly Hall proved to be a miscalculation, as the Peasant Faction's other members returned to vote on land reform, and, subsequently, on the unconditional union. Marghiloman nevertheless gave a contrasting account. He complained that \\\"not even 30 deputies\\\" had been present for the vote abrogating the conditions, in spite of \\\"all the money spent\\\".\\nSalvation Committee.\\nIn early 1919, Tsyganko emigrated from what was then being recognized as Romanian territory. He settled in Odessa, a Ukrainian port city, where he established the Peasants' Faction in exile, alongside a dozen other former \\\"Sfatul\\\" deputies. He joined efforts with Krupenski and Schmidt, affiliating with their Committee for the Salvation of Bessarabia, whose activities were closely monitored by Romania's secret police, the Siguranța. According to the Moldavian Bloc's Petru Cazacu, they answered indirectly to Anton Denikin, commander of the Volunteer Army. \\nThe various groups of Bessarabian autonomists and White loyalists agreed to send a common delegation to the Peace Conference in Paris, where the Allies were debating on recognizing the union. On February 10, the Committee issued a common platform for these organizations, sharing two goals: \\\"the liberation of Bessarabia from the Romanian annexation and the realization of the aspirations of the people of Bessarabia.\\\" The latter referred to the region's reintegration into Russia. In April, together with Krupenski and Schmidt, later followed by Mark Slonim and Mihail Savenco, Tsyganko had arrived in Paris. In its addresses to the international media, the group insisted that the union was a putsch by urban intellectuals against the other social classes. It also circulated a protest against the Romanian land reform project, which the Salvation Committee saw as a chauvinistic attack against the landed gentry and the Russian patriots. \\nAs noted by Suveică, Tsyganko was the only delegation member to belong to a non-aristocratic elite, and nominally an appointee of the \\\"Central Committee of the Peasants of Bessarabia\\\". He therefore took some distance from the conservative demands of the Salvation Committee, and in various contexts presented himself as an independent emissary, united with the others mainly in their common support for a plebiscite clause in Bessarabia. However, his autonomism and Krupenski's loyalism were mostly endorsed by the White émigré lobby in Paris, including the likes of Georgy Lvov, Vasily Maklakov, Sergey Sazonov, and Nikolai Tchaikovsky. In his papers, Halippa commented that Tsyganko, the self-proclaimed \\\"socialist and revolutionary\\\", had arrived in Paris as a propagandist of Russian nationalism, \\\"with no connection to the people [of Bessarabia]\\\". Cazacu also notes that the \\\"bizarre association\\\" comprising Tsyganko, Maklakov and Schmidt propagated the contradictory claim that \\\"Sfatul\\\" was a \\\"Bolshevik\\\" assembly.\\nIn June 1919, the French communist organ, \\\"L'Humanité\\\", gave exposure to Slonim and Tsyganko's allegations regarding political repression and \\\"atrocities\\\" in Bessarabia, as a common protest of the \\\"democrats and socialists\\\". Such allegations were responded to by the BPȚ's Ion Pelivan, who wrote the newspaper to argue that Romania's intervention had first of all restored \\\"liberty and democracy\\\" in Bessarabia, and that the union expressed \\\"the free will of the Bessarabian populace, with no outside intervention.\\\" Countering Tsyganko's claim to speak for the peasants, the pro-Romanian delegation grew to include peasant members such as Ion Codreanu, Gheorghe Năstase, and Sergiu Victor Cujbă.\\nLater life.\\nMaking his split from the White Russian community a definitive one, Tsyganko eventually settled in the Ukrainian Soviet Socialist Republic. In September 1921, he wrote a memorandum on Bessarabia and Romania–Russia relations, which he sent to Leon Trotsky, the Commissar for Military Affairs. His text informed the Russian viewpoint at the negotiations in Warsaw between the Soviets and the Romanians, but also presented personal observations on the social makeup of Bessarabia. Tsyganko argued that the Moldavian Republic's creation and union were attributable to left-wing \\\"agitators\\\" such as Inculeț and Pantelimon Erhan. He claimed that multi-ethnic Bessarabia was naturally \\\"internationalist\\\", but also rural and \\\"well-off\\\", concluding that an anti-Romanian revolt could happen if sustained from across the border. Cazacu noted in 1924 that the dossier compiled by the Bolsheviks and the earlier Salvation Committee drafts used as sources the same documents, including statements by private individuals.\\nIn 1924, Tsyganko became a founding figure of the Moldavian Autonomous Soviet Socialist Republic, created on Soviet territory as a Bessarabian rump state. In September 1926, he was at Odessa, where, together with Ivan Krivorukov, he issued a formal protest against Italy's recognition of the Bessarabian union with Romania, and therefore against her \\\"passage into the anti-Soviet camp\\\". Nikolai Vladimirovich, meanwhile, remained behind in Romania, working as a conservator for the Historical Monuments Commission, then as Department head for Chișinău City Hall. In early 1929, Nikolai supported the Bessarabian regionalist platform for administrative reform (an initiative by Erhan and Alexandru Mîță). He died some five years later, at age 52, while reading a book. At some point during the 1930s, Vladimir Tsyganko was singled out as a political suspect by the Stalinist regime, and was murdered in 1937, at the height of the Great Purge.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126405\",\"title\":\"Lee Meong-hee\",\"body\":\"\\nLee Meong-hee\\n\\nLee Meong-Hee (; born 4 July 1978) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126432\",\"title\":\"Lee Yun-hui\",\"body\":\"\\nLee Yun-hui\\n\\nLee Yun-Hui (; born 8 October 1980) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126439\",\"title\":\"Caniparola\",\"body\":\"\\nCaniparola\\n\\nCaniparola is a frazione del comune di Fosdinovo, in province di Massa e Carrara.\\nPhysical Geography.\\nIt's in the southern part of the comune, at 39 metres above sea level. It's the lowest frazione in the comune and it's displaced along the first part of the provincial street that leads to Fosdinovo from Aurelia Street.\\nHistory.\\nIn Caniparola, until the XVIII century, there was a tower, that was built at the age of the Bishops of Luni, on the which the Fosdinovo marquess Gabriele III Malaspina, ending his predecessor's project, Carlo Francesco Agostino Malaspina (who died here in 1722) built in 1724 his summer residence, the villa Malaspina. The plain on which this villa was founded was already exploited for agricultural use by Malaspina, who had built here a lot of farmhouses (they are said \\\"Malaspinian Farmhouses\\\"). In the mid-eighteenth century, between the end of Gabriele III's marquisate and the beginning of Carlo Emanuele Malaspina's one, some lignite benches were discovered for the mining of the so-called \\\"Caniparola coal\\\".\\nThe toponym derives from the cultivation of the cannabis (canapa, in Italian) that was held in the area in the past.\\nSociety.\\nDemographic Evolution.\\nDuring the second half of XIX century, this frazione had an outstanding demographic increase (which is corresponded to a likewise increase of the built infrastructures and houses), so that, counted with the near localities (above all Borgetto-Melara), from 2001 onwards it began representing almost the half of the total population of the comune. The only frazione, on the other hand, has a population that would place as the first between the frazioni, but as second compared to Fosdinovo, the chief town. Today Caniparola is a residential place, although it still keeps vineyards and olive groves of its agricultural past.\\nReligion.\\nCaniparola is seat of the Sant'Antonio da Padova Parish, in the diocese of Massa Carrara-Pontremoli. From September 2015, it is part of Fosdinovo Pastoral Unity, within Fivizzano Vicariate. Patron Saint Day is 13 June.\\nCulture.\\nEducation.\\nCaniparola is seat of the detachment of the statal school \\\"Don Florindo Bonomi\\\", consisting of kinfergartens, elementary and secondary school. The chief seat is in Fosdinovo.\\nSport.\\nCaniparola has got a great public gym and a stadium, \\\"Mulattieri\\\", placed in the locality Borghetto-Melara, where some football tournaments are held, like Trofeo Città di Fosdinovo and Memoriale Federico Severino. In Caniparola the matches of the chief Fosdinovo volley team, \\\"Volley C.P.O. Fosdinovo\\\", are played. The female sector takes part to the regional Championship.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126454\",\"title\":\"News Is Made at Night\",\"body\":\"\\nNews Is Made at Night\\n\\nNews Is Made at Night is a 1939 American comedy film directed by Alfred L. Werker and written by John Larkin. The film stars Preston Foster, Lynn Bari, Russell Gleason, George Barbier, Eddie Collins and Minor Watson. The film was released on July 21, 1939, by 20th Century Fox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126458\",\"title\":\"Park Mee-kyung\",\"body\":\"\\nPark Mee-kyung\\n\\nPark Mee-kyung or Park Mi-kyung (; born 13 May 1975 / 6 April 1975) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th. On the club level she played with Hanil Synthetic Fiber, then in 1998 transferred to Korea Highway Corporation.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126463\",\"title\":\"En Adir\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126483\",\"title\":\"Park Soo-jeong\",\"body\":\"\\nPark Soo-jeong\\n\\nPark Soo-jeong (; born 2 March 1972) is a retired South Korean female volleyball player. She was part of the South Korea women's national volleyball team. She competed with the national team at the 2000 Summer Olympics in Sydney, Australia, finishing 8th.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126543\",\"title\":\"È arrivata la felicità\",\"body\":\"\\nÈ arrivata la felicità\\n\\nÈ arrivata la felicità is an Italian television comedy-drama series created by Stefano Bises, Ivan Cotroneo, Monica Rametta; and produced by Publispei and Rai Fiction. The setting of the show are Aventino and Testaccio, two districts in Rome.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126561\",\"title\":\"3rd New York Volunteer Cavalry\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126562\",\"title\":\"List of Indian states by GDP per capita\",\"body\":\"\\nList of Indian states by GDP per capita\\n\\nThis page lists Indian States by the nominal GDP per capita and the percentage of growth.\\nGrowth in GDP per capita.\\nThis shows annual growth in each state's GDP per capita for the years between 2004 and 2014. Figures are in rupees.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126580\",\"title\":\"2015 FA Women's Cup Final\",\"body\":\"\\n2015 FA Women's Cup Final\\n\\nThe 2015 FA Women's Cup Final was the 45th final of the FA Women's Cup, England's primary cup competition for women's football teams. The showpiece event was the 22nd to be played directly under the auspices of the Football Association (FA) and was named the SSE Women's FA Cup Final for sponsorship reasons. The final was contested between Chelsea Ladies and Notts County Ladies on 1 August 2015 at Wembley Stadium in London. Chelsea made its second final appearance, after losing the 2012 final. Notts County appeared in its first ever final.\\nThe match was the first women's final to be staged at Wembley Stadium. Watched by a record crowd of 30,710 and a BBC television audience of nearly two million, Chelsea won the match 1–0, with a first-half goal from Ji So-yun. Chelsea's Eniola Aluko was named player of the match.\\nRoute to the final.\\nAs FA WSL 1 clubs, both teams entered the competition at the fifth round stage. Chelsea beat Watford (6–0), holders Arsenal (2–1) and Manchester City (1–0) to reach the final. Notts County faced lower-division opponents in all three games, defeating Tottenham Hotspur (4–0), Aston Villa (5–1) and Everton (3–0).\\nThe 2015 final marked the second time Chelsea had reached this stage, after losing in 2012 in a penalty shootout against Birmingham City. Notts County had never previously appeared in the final, since its founding in 2014 or as its predecessor Lincoln Ladies.\\nMatch.\\nAfter a quiet start to the game, Chelsea winger Eniola Aluko applied the first serious pressure of the game with a shot at the 30th minute. A second soon after was deflected by Notts County's goalkeeper Carly Telford. The resulting corner setup Gemma Davison for a shot which went wide. In the 37th minute, Aluko found centre forward Ji So-yun inside the box and set her up for a short range goal. Notts County came back on the attack after the half. A long-range shot from midfielder Desiree Scott was deflected and a header by Leanne Crichton in the resulting corner was narrowly cleared off the line. Chelsea hunted for a second goal but Aluko had another shot deflected while midfielder Drew Spence sent a shot wide. In the end So-yun's lone goal proved enough and Chelsea won 1–0.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126606\",\"title\":\"Sinclair Cutcliffe\",\"body\":\"\\nSinclair Cutcliffe\\n\\nJohn Sinclair Cutcliffe (August 22, 1930 – November 10, 2007) was a Canadian politician, who served in the Legislative Assembly of Prince Edward Island from 1966 to 1972. A member of the Liberal Party, he was the assemblyman for the district of 2nd Queens.\\nBorn and raised in Summerside, Prince Edward Island, he first worked for his father's Cutcliffe Funeral Home and eventually became owner of the business. He also served as president of the provincial Red Cross, as a chief of rescue with the Prince Edward Island Emergency Measures Organization, and as a president of the International Rescue and First Aid Association.\\nHe was first elected to the legislature in the 1966 provincial election, serving as a backbench MLA. Reelected in the 1970 provincial election, in his second term he served as deputy speaker of the legislature. He resigned his provincial seat in 1972 to run as a Liberal Party of Canada candidate for Malpeque in the 1972 federal election, but lost to Angus MacLean.\\nHe died on November 10, 2007 at the Queen Elizabeth Hospital in Charlottetown.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126611\",\"title\":\"2015–16 Texas–Rio Grande Valley Vaqueros women's basketball team\",\"body\":\"\\n2015–16 Texas–Rio Grande Valley Vaqueros women's basketball team\\n\\nThe 2015–16 UTRGV Vaqueros women's basketball team represents the University of Texas Rio Grande Valley during the 2015–16 NCAA Division I women's basketball season. This is head coach Larry Tidwell's third season (1st season) at UTRGV. The Vaqueros play their home games at the UTRGV Fieldhouse and are members of the Western Athletic Conference. This is the first season for UTRGV as an institution. Before the 2015–16 academic year UTRGV was UTPA and the University of Texas at Brownsville. \\nPrevious Season.\\nThe Broncs finished the season 19–15, 9–5 in final WAC play to finish in third place. They lost in the championship of the WAC Tournament to New Mexico State.\\n2015–16 media.\\nFor the first time in club history women's basketball games will be televised. 9 of 11 home games will air on TWCS (Ch 323), with 6 of the 9 games airing live. The other two home games will air on the TWCS Alternate Channel (Ch 825). Other games will air on WAC Digital Network or road teams video feeds.\\nSchedule and results.\\n!colspan=9 style=\\\"background:#; color:white;\\\"| Non-conference regular season\\n!colspan=9 style=\\\"background:#; color:white;\\\"| WAC regular season\\n!colspan=9 style=\\\"background:#; color:white;\\\"| WAC Women's Tournament\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126612\",\"title\":\"List of festivals in Oceania\",\"body\":\"\\nList of festivals in Oceania\\n\\nThe following is an incomplete list of festivals in Oceania, with links to separate lists by country and region where applicable. This list includes festivals of diverse types, including regional festivals, commerce festivals, film festivals, folk festivals, carnivals, pow wows, recurring festivals on holidays, and music festivals. Note that list of music festivals in Oceania redirects here, with music festivals denoted with (music) for countries where there is not a dedicated music section. The list overlaps with list of film festivals in Oceania.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126633\",\"title\":\"ONE (website)\",\"body\":\"\\nONE (website)\\n\\nONE is an Israeli website that covers sport news from around the world. According to the IARB (Israel Audience Research Board), it is one of the two biggest sport websites in Israel along with Sport5.co.il, It categorized news into \\\"Israeli Football\\\", \\\"World Football\\\", \\\"Israeli Basketball\\\", \\\"World Basketball\\\" and \\\"Other Sports\\\".\\nHaim Revivo, Eyal Berkovic, Alon Hazan, Alon Mizrahi and Zvi Sherf are among ONE's columnists.\\nONE was founded by Udi Milner and Gil Menkin in 1999, the website's headquarters are located in the BSR Towers in Ramat Gan. The website was founded in order to cover the TV show \\\"The 91th minute\\\", which was broadcast on Channel 2.\\nIn the beginning of 2000, there was a race between Israeli sport websites, at the end, many of them came to an end. ONE survived by investments from the production company Telad. In 2007, Yedioth Ahronoth Group acquired 50% of the ownership on the website.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126650\",\"title\":\"Uehara Museum of Modern Art\",\"body\":\"\\nUehara Museum of Modern Art\\n\\n opened in 2000 in Shimoda, Shizuoka Prefecture, Japan, to house the collection of of Taisho Pharmaceutical. The collection includes works by Corot, Monet, Cézanne, Renoir, Fujishima Takeji, and Kishida Ryūsei. Adjacent is the , which opened in May 1983.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126722\",\"title\":\"Dilek Öcalan\",\"body\":\"\\nDilek Öcalan\\n\\nDilek Öcalan (born 3 October 1987) is a Turkish politician of Kurdish origin from the Peoples' Democratic Party (HDP) who currently serves as a Member of Parliament for the electoral district of Şanlıurfa since 7 June 2015. She is the niece of Abdullah Öcalan, the imprisoned leader of the Kurdistan Workers' Party (PKK) militant organisation that has been in conflict with the Turkish Armed Forces since the 1980s, making both her candidacy and election to Parliament highly controversial.\\nEarly life.\\nDilek Öcalan was born on 3 October 1987 in Şanlıurfa as the daughter of Fatma Öcalan, the sister of imprisoned PKK leader Abdullah Öcalan. On 23 December 2013, she visited her uncle at İmralı Prison and became known to the media after she gave a press statement detailing the conversation between them. Öcalan has been imprisoned since 1999, serving aggravated life imprisonment under charges of founding and leading a terrorist organisation (namely the PKK, which is recognised as a terrorist organisation by Turkey, the European Union and the United States).\\nPolitical career.\\nÖcalan first entered politics in 2012, a year before meeting her uncle on the Island of İmralı. During the third congress of the pro-Kurdish Peace and Democracy Party (BDP), she was elected to the party executive while the party changed its name to Democratic Regions Party (DBP) and adopted a fraternal relationship with the Peoples' Democratic Party (HDP).\\nMember of Parliament.\\nÖcalan's candidacy to become a Member of Parliament despite being the niece of Abdullah Öcalan was heavily controversial, drawing strong opposition from Turkish nationalists. Her candidacy also allegedly caused a split within the Öcalan family. Nevertheless, she was put forward as a HDP candidate for the electoral district of Şanlıurfa, being fielded as the second candidate on the HDP's provincial party list. She was subsequently elected in the June 2015 general election to become one of the youngest MPs in the new Parliament, resulting in her being appointed to the temporary Speaker's Council until a new Council could be elected in July.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126732\",\"title\":\"Antonietta luteorufa\",\"body\":\"\\nAntonietta luteorufa\\n\\nAntonietta luteorufa is a species of sea slug or aeolid nudibranch, a marine gastropod mollusc in the family Facelinidae.\\nDistribution.\\nThis nudibranch is known from the Gulf of Naples, Italy.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126763\",\"title\":\"List of Desperate Housewives Africa episodes\",\"body\":\"\\nList of Desperate Housewives Africa episodes\\n\\nDesperate Housewives Africa is a Nigerian television Comedy-drama-mystery that premiered on Ebony Life TV on 30 April 2015. It is an adaptation of the American series \\\"Desperate Housewives\\\". The majority of the episodes are titled according to the original version which were in turn named after lyrics by composer/lyricist Stephen Sondheim. The show revolves around four housewives namely; Ese De Souza (Nini Wacera), Kiki Obi (Kehinde Bankole), Tari Gambadia (Michelle Dede), and Funke Lawal (Omotu Bissong). The show is showcased from Rume Bello (Marcy Dolapo Oni) point of view who dies at the pilot episode.\\nThe first season is slated to have 23 episodes.\\nSeries overview.\\nSeason one episodes.\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126783\",\"title\":\"Wallace W. Andrew\",\"body\":\"\\nWallace W. Andrew\\n\\nWallace W. Andrew (December 25, 1850 – January 18, 1919) was an American businessman and politician.\\nBorn in Sheboygan County, Wisconsin, Andrew and his parents moved to Oregon, Wisconsin. Andrew was involved with the grain and livestock businesses. In 1894, Andrew moved with his brothers to Superior, Wisconsin and set up the Deluxe Manufacturing Company. Andrew served in the Wisconsin State Assembly in 1901 and 1905 and was a Republican. At the tome of his death in 1919, Andrew was serving on the Douglas County Board of Supervisors and was chairman of the county board. Andrew died at his home in Superior, Wisconsin. \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126794\",\"title\":\"Alive (Hiromi album)\",\"body\":\"\\nAlive (Hiromi album)\\n\\nAlive is the third album from Hiromi Uehara's Trio Project featuring bassist Anthony Jackson and drummer Simon Phillips.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126813\",\"title\":\"No Control (Turbo Fruits album)\",\"body\":\"\\nNo Control (Turbo Fruits album)\\n\\nNo Control is an album by American band Turbo Fruits, which was released on April 21, 2015. All songs were produced by Jeremy Ferguson, except \\\"The Way I Want You\\\" and \\\"No Reason to Stay\\\", which were produced by The Black Keys drummer Patrick Carney.\\nPersonnel.\\nTurbo Fruits\\nAdditional personnel\\nProduction\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126826\",\"title\":\"PubNub\",\"body\":\"\\nPubNub\\n\\nPubNub is a global Data Stream Network (DSN) and realtime infrastructure-as-a-service (IaaS) company based in San Francisco, California. The company makes products for software and hardware developers to build realtime web, mobile, and Internet of Things (IoT) applications.\\nPubNub's primary product is a realtime publish/subscribe messaging API built on their global data stream network which is made up of a replicated network of at least 14 data centers located in North America, South America, Europe, and Asia. The network currently serves over 300 million devices and streams more than 750 billion messages per month.\\nHistory.\\nPubNub was founded in 2010 by Stephen Blum and Todd Greene. PubNub raised $4.5 million in Series A funding from Relay Ventures and TiE Angels in March 2012. They received their $11 million Series B round of funding in September 2013 from Scale Venture Partners, Relay Ventures and TiE Angels. In July 2015, PubNub received their $20 million Series C round of funding led by Sapphire Ventures\\nTechnology.\\nPubNub utilizes a Publish/Subscribe model for realtime data streaming and device signaling and supports all of the capabilities of WebSockets, Socket.IO, SignalR, WebRTC Data Channel and other streaming protocols. PubNub provides SDKs for over 70 different programming languages and environments including JavaScript, iOS, and Android, as well as JavaScript frameworks such as AngularJS, Ember.js, and Backbone.js. PubNub also provides client libraries for board platforms including Raspberry Pi, Arduino, Texas Instruments, and Microchip.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126834\",\"title\":\"Elizabeth Buchanan Cowley\",\"body\":\"\\nElizabeth Buchanan Cowley\\n\\nElizabeth Buchanan Cowley (1874–1945) was an American mathematician.\\nCowley was born on May 22, 1874, in Allegheny, Pennsylvania. She had four siblings, but they and her father all died by 1900. Cowley's mother, Mary Junkin Buchanan Cowley, later became a member of the Board of Public Education of Pittsburgh, and was the namesake of the Mary J. Cowley School in Pittsburgh. Cowley's grandfather (Mary Cowley's father) was James Galloway Buchanan, a surgeon in the Union Army.\\nCowley earned a bachelor's degree in 1893 from the Indiana State Normal School of Pennsylvania, and became a school teacher. She earned a second bachelor's degree in 1901 and a master's degree in 1902 from Vassar College, and became an instructor at Vassar, studying higher mathematics during the summers at the University of Chicago. In 1908 she completed a doctorate from Columbia University. Her dissertation, on algebraic curves, was supervised by Cassius Jackson Keyser; she became the fourth woman to earn a doctorate in mathematics from Columbia. Continuing to work at Vassar, Cowley was promoted to assistant professor in 1913, and associate professor in 1916. She went on leave in 1926 to assist her mother, and resigned her position at Vassar in 1929, instead becoming a high school teacher in Pittsburgh.\\nCowley and her co-author Ida Whiteside won a prize for a 1907 paper they wrote on the orbit of comet C/1825 V1. Another of her publications, in 1926, concerned liquid water pouring puzzles. She was the author of two textbooks on plane and solid geometry, published in 1932 and 1934, and advocated teaching solid geometry to high school students after many colleges had replaced the subject with freshman calculus. She published another book in 1941 about public education.\\nCowley was an early member of the Mathematical Association of America, and became a member of its board of trustees when it incorporated in 1920.\\nShe was an invited speaker at the International Congress of Mathematicians in 1932, speaking there about mathematics education. She also belonged to the American Mathematical Society, German Mathematical Society, and Circolo Matematico di Palermo.\\nShe retired from teaching in 1938, had a stroke in 1941, and died on April 13, 1945, in Fort Lauderdale, Florida.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126858\",\"title\":\"Facelinopsis pacodelucia\",\"body\":\"\\nFacelinopsis pacodelucia\\n\\nFacelinopsis pacodelucia is a species of sea slug, an aeolid nudibranch, a marine gastropod mollusc in the family Facelinidae.\\nDistribution.\\nThis species was described from Algeciras harbour, Spain.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126861\",\"title\":\"2010–11 Glasgow Warriors season\",\"body\":\"\\n2010–11 Glasgow Warriors season\\n\\nThe 2010-11 season saw Glasgow Warriors compete in the competitions: the Magners Celtic League and the European Champions Cup, the Heineken Cup.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126877\",\"title\":\"Punjabi nationalism\",\"body\":\" Delhi, Uttar Pradesh, Jammu and Kashmir and many institutes like schools-colleges in Punjab state itself where Punjabi language is ignored. Punjabi language dialects like Bauria, Bazigari, Bhand, Dhaha, Gojri, Lahanda, Lubana, Odi, Rai Sikhi and Sansi are also becoming extinct in Punjab, India. There is Hindi imposition since 1950s and 1960s in state against Punjabi language. Despite a rich heritage of Punjabi literature, Punjabi Television serial industry in Indian Punjab has totally disappeared.\\nPunjabi nationalism in West Punjab.\\nIn Pakistani Punjab province, Punjabi Language Movement is a linguistic movement in aimed at reviving the Punjabi language, art, culture and literature in Pakistan. There are several attempts going on by Punjabi society for implementation of Punjabi language as it is completely ignored by authorities in Punjab province. Urdu is preferred medium of education in local schools-colleges as well as Government paperwork which is very threatening for survival of Punjabi language in Punjab, Pakistan. But Urdu is the mother tongue of only about 7.57% Pakistanis. In September 2015, a case was filed in Supreme Court of Pakistan against Government of Punjab, Pakistan as it did not take any step to implement Punjabi language in the province. Punjabi lovers also say that creation of Bangladesh out of Pakistan proves that love of \\\"Mother-tongue\\\" is more important than religion. Pakistani Punjabi language film industry is in crisis as filmmakers were not producing Punjabi language films like before 1975 Punjabi films ruled in film industry of Pakistan. Television Channels from Lahore (Punjab's capital city) are all in Urdu instead of Punjabi. There is still 150-year-old unofficial ban on education in Punjabi language in Punjab, Pakistan and Government is ignorant about it thus compelling Punjabi people to protest. In August 2015, Pakistan Academy of Letters, International Writer’s Council (IWC) and World Punjabi Congress (WPC) organised \\\"Khawaja Farid conference\\\" and demanded Punjabi University should be established in Lahore and Punjabi language should be declared as the medium of instruction at the primary level. In Lahore, every year thousands of punjabis gather on International Mother Language Day seeking an end to the 150-year-old ban on education in Punjabi in Pakistan and against Urdu-isation of Punjab. In September 2015 at Government Emerson College, Multan thousands of aspirants seeking admission protested against the administration for forcing them not to adopt Punjabi and Saraiki dialect as compulsory or optional subjects as usually majority of students prefer Punjabi and Saraiki dialect balancing their marks sheet in BA (third year).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126881\",\"title\":\"Time Warner Cable SportsChannel (Texas)\",\"body\":\"\\nTime Warner Cable SportsChannel (Texas)\\n\\nTime Warner Cable SportsChannel (TWC SportsChannel) is a regional sports network serving Texas. It is broadcast on Channel 323 and 1020 within the state exclusively on Time Warner Cable systems.\\nProgramming.\\nThe following sporting events are carried by TWC SportsChannel – Texas:\\nOther shows that air on TWC SportsChannel:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126887\",\"title\":\"Chris Gheysens\",\"body\":\"\\nChris Gheysens\\n\\nChris Gheysens (born 1971) is an American businessman who serves as the President and Chief Executive Officer of Wawa Inc., a privately-held chain of convenience store / gas stations with nearly 700 locations along the East Coast of the United States.\\nHe grew up in Vineland, New Jersey and worked in his youth at car washes that his father owned. He attended St. Mary School in East Vineland and then moved on to St. Augustine Preparatory School in Richland, New Jersey, graduating in 1989. After earning a B.S. in accounting from Villanova University in 1993, Gheysens earned an M.B.A. from Saint Joseph's University in 2005, both located in Philadelphia. His family would spend their summers in the Jersey Shore community of Sea Isle City.\\nCareer.\\nGheysens worked for four years as an auditor in the Philadelphia office of Deloitte LLP, in Philadelphia.\\nGheysens was hired by Wawa in 1997 and was named to serve as the company's its chief financial officer and chief administrative officer in 2007. In 2012 he was named as Wawa's president and became the chief financial officer in January 2013, after a 16-month-long transition from his predecessor, Howard Stoeckel. He is identified on his business card as \\\"Lead Goose\\\", a riff on the company's logo of a flying goose, a title that he describes as fitting in with the loose corporate culture at Wawa. Gheysens has overseen an extensive remodeling project for the company's stores to establish a \\\"warmer tone\\\" for customers, rollout of a mobile app and expansion into North Jersey and Florida.\\nAt the Villanova School of Business, Gheysens is a member of the dean's advisory committee. He has also been active with the Southeastern Pennsylvania Chapter of the American Red Cross. In September 2013, Gheysens was named to serve a three-year term on the Economic Advisory Council of the Federal Reserve Bank of Philadelphia. In October 2013, he was one of four members added as retail members serving on the Board of Directors of the National Association of Convenience Stores.\\nGheysens traveled to Rome in 2015 as part of a delegation coordinating aspects of the papal visit to Philadelphia as part of Pope Francis' 2015 visit to North America. As one of the corporate sponsors of the World Meeting of Families, Wawa was contributing one million bottles of water for those participating at the event.\\nHe lived in Washington Township, Gloucester County, New Jersey before moving to Moorestown, New Jersey with his wife and four children.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126889\",\"title\":\"Francisco Plaza\",\"body\":\"\\nFrancisco Plaza\\n\\nFrancisco Plaza, also known as Fort Francisco Museum, built in 1862, was the first significant dwelling in the Cuchara Valley at what became La Veta, Colorado. The plaza is now a part of the museum complex. Only the courtyard and the fort building were added to the National Historic Register in 1986. The other structures are new, or were moved to this location during the 1960s.\\nHistory.\\nThe plaza was originally a U-shaped structure, much modified over the years. It was built by John M. Francisco of Georgia, and his business partner, Henry Daigre, a Frenchman from Quebec, and served as the headquarters for their cattle ranch. The ranch sold beef to the U.S. military forts in the area. The area around the plaza was home to Hispanic workers for the ranch. The plaza served as a supply center to the growing community, occasionally proving protection from Ute Indian attacks.\\nThe plaza served as a temporary depot for the Denver and Rio Grande Railroad when it arrived in 1876. The town of La Veta, incorporated at the same time, never grew to more than 800 residents. The plaza because a residence and farm again for John M. Francisco by the 1890s, the partnership with Daigre having ended possibly 20 years prior. Francisco owned the plaza until his death in 1902. It then passed through various family members, and was acquired by the Huerfano County Historical Society in 1958.\\nArchitecture.\\nThe remaining historic structures were adobe of typical early Hispanic style, built of 18 to 24 inch thick adobe bricks. The building had three sides, each 100 feet long, with the fourth side closed by a fence. Significant changes were made to portions of the building, other buildings were built and rebuilt. The buildings were renovated in the late 1990s.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126908\",\"title\":\"Giovanni Waal\",\"body\":\"\\nGiovanni Waal\\n\\nGiovanni Waal (born May 25, 1989) is a Surinamese footballer playing as a Forward for Inter Moengotapoe in the Hoofdklasse, and for the Suriname national team.\\nCareer.\\nWaal began his career at SV Voorwaarts in Paramaribo, making his debut in the 2009-10 SVB Hoofdklasse season. After two season he transferred to SV Leo Victor, finishing the season as the leagues joint top scorer together with Ulrich Reding (of SV Boskamp), both finishing with 20 goals each. The following season saw Waal transfer to SV Robinhood where he played for two seasons. In 2014 Waal transferred to Inter Moengotapoe winning the National championship in his first season with his new club.\\nInternational career.\\nWaal plays International football for Suriname, having made his debut in the final qualifying round of the 2010 Caribbean Cup, in the 2-1 loss against Antigua and Barbuda. He also participated in the teams' 2014 FIFA World Cup qualification campaign, making four appearances against Cayman Islands, Dominican Republic and El Salvador, failing to help his team advance to further qualifying rounds. On 9 June 2012 he scored his first goal for the national team in a friendly match against French Guiana which ended in a 2-1 loss, He has also represented Suriname in the 2012 Caribbean Cup qualification, the 2018 FIFA World Cup qualification and the ABCS Tournament.\\nCareer statistics.\\nInternational performance.\\n\\\"Statistics accurate as of matches played on 16 June 2015\\\",\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126927\",\"title\":\"Thank Your Lucky Stars (Beach House album)\",\"body\":\"\\nThank Your Lucky Stars (Beach House album)\\n\\nThank Your Lucky Stars is the sixth studio album by American dream pop band Beach House. It was co-produced by the band and Chris Coady, and was released on October 16, 2015, on Sub Pop and Bella Union. The album was released less than two months after their fifth studio album, \\\"Depression Cherry\\\".\\nDescribed by the band as \\\"not a companion to \\\"Depression Cherry\\\", or a surprise, or b-sides,\\\" \\\"Thank Your Lucky Stars\\\" was unexpectedly announced nine days before its release via the band's Twitter account. It received mostly positive reviews from critics.\\nBackground and recording.\\nThe album was recorded at Studio in the Country in Bogalusa, Louisiana, and mixed at Sonic Ranch in Tornillo, Texas. Although the album was recorded simultaneously alongside \\\"Depression Cherry\\\", the band felt that the records should be seen as distinct unconnected works. \\nReception.\\n\\\"Thank Your Lucky Stars\\\" was released to highly positive reviews. At Metacritic, which assigns a normalized rating out of 100 to reviews from mainstream critics, the album has an average score of 80, based on 13 reviews.\\nJayson Greene of Pitchfork suggested the songs took on a \\\"darker edge\\\" than those from \\\"Depression Cherry\\\", judging the songs to feel smaller by having stripped away the typical cathedral-like reverb from the group's previous albums. Greene likened the mood of the songs to Beach House's material before they joined Sub Pop, describing the feeling as \\\"pneumatic, dusty, like they are pulling a blanket around themselves in a heatless attic to ward off a threatening chill.\\\" Although the \\\"joy and comfort have vanished\\\" from the material, Greene claimed that that album is \\\"still undeniably a Beach House album, a familiar mix of warm tones and chilly sentiments.\\\" Ultimately, Greene welcomed the addition to the band's repertoire, but suggested that a new album so soon created a dissonance that feels like \\\"too much of a good thing.\\\"\\nIn a review from \\\"The A.V. Club\\\", Corbin Reiff described the album as \\\"most assuredly a continuation of many of the same motifs and hallmarks of the group's last release.\\\" Reiff hailed the band's craft, arguing that \\\"Beach House has mastered the art of space by this point and seems to have an instinct for how long to drag out a keyboard melody or a guitar line before bringing in another element to keep things from bogging down.\\\" Although he felt \\\"Depression Cherry\\\" and \\\"Thank Your Lucky Stars\\\" were similar in motif, Reiff praised the band's decision to separate the songs, rather than tack them onto the former. For Reiff, \\\"Thank Your Lucky Stars\\\" supports itself as a singular entity where \\\"the full sonic and emotional weight is tremendous.\\\"\\nWriting for \\\"Rolling Stone\\\", Meagan Fredette gave praise to Victoria Legrand's vocals stating that \\\"her singing on \\\"Thank Your Lucky Stars\\\" feels more playful than usual, a welcome lightness that comes across from the first moments of \\\"Majorette,\\\" the album's opener\\\". Elegy to the Void was also singled out as a highlight of the record with Fredette describing the track as \\\"the album’s crown jewel\\\" and \\\"as good a summation as any of what Beach House does best\\\". Emphasising the maturity of the record in comparison to the band's debut, she suggests that \\\"Like all their albums, this one is full of songs made for dreaming of a bygone love, or humming quietly to a new one\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126935\",\"title\":\"Alain Ngamayama\",\"body\":\"\\nAlain Ngamayama\\n\\nAlain Ngamayama is a Polish footballer who plays as a midfielder for Warta Poznań. He is the captain and widely considered a club hero.\\nHe is was born to a Zairean father and Polish mother, however considers himself fully Polish having born in Greater Poland. He grew up in Poznań and began in Warta's youth teams. Despite the clubs varying fortune he stayed with the club throughout his career barring a short stint for one season early in his career.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126945\",\"title\":\"Massawepie Mire\",\"body\":\"\\nMassawepie Mire\\n\\nThe Massawepie Mire is a boreal peatland bog in Piercefield, New York. Located near Massawepie Lake, the mire is the largest bog in New York State. Much of the bog is on the property of Massawepie Scout Camps, and the camp partners with the Nature Conservancy to preserve the ecosystem. The Massawepie Mire is noted for birdwatching, with several species of rare birds occupying the area including spruce grouse, gray jay, Lincoln's sparrow, boreal chickadee, and the two-barred crossbill. Flora includes the carnivorous pitcher plants and sundew, tamarack pines and black spruce trees, as well as the sphagnum moss that makes up the base of the bog.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126947\",\"title\":\"Calmodulin in Target Binding and Recognition\",\"body\":\"\\nCalmodulin in Target Binding and Recognition\\n\\nCalmodulin (CaM) is a complex signaling protein that transduces transient calcium ion signals. CaM's binding of calcium ions cause conformational changes, which interact with downstream proteins. Current research indicates that selective protein binding occurs through the mechanism of mutually induced conformational fit, which would explain how calcium dynamics in CaM would modulate its interaction.\\nCurrent research on CaM signaling and CaM-BT interaction includes experimental kinetic rate observations and coarse grain/all atom Molecular dynamics simulations. Because protein signaling and protein-protein interaction is a new field of research, many observed interactions cannot be explained through experiment alone. The unification between simulation and experimental results is necessary to expand the predictive power of the theoretical approach and create general laws that explain the mechanics of signaling/protein-protein interactions.\\nThe computational approach for modeling macro molecules is very resource intensive. The Hamiltonian equation in molecular dynamic software relates each atom to all other atoms in the system through kinetic, electrostatic, van der Waals, dihedra, bond, etc. energies. For example, the RRK polypeptide (CaMKII residues: 293-313) contains 21 residues and 318 atoms. For a single time step, the molecular dynamics software must perform energy calculations between every atom in the polypeptide, which is ~100,000 calculations. Since the time step must be in the sub picosecond range (to insure stability), several million time steps must be performed to obtain meaningful data.\\nTo remedy the large number of calculations involved in all atom simulations, the coarse grain simulation technique can be used. Current work from the biophysics group at the University of Houston uses open source coarse grain and all atomic models of CaM and wildtype/mutated binding targets of CaMKII in their research.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126966\",\"title\":\"The Silk Roads\",\"body\":\"\\nThe Silk Roads\\n\\nThe Silk Roads: A New History of The World is a 2015 book authored by Peter Frankopan, an academic at the University of Oxford. He presents a new point of view about center of human rise and oppose with traditional view that consider human heirs to the Egyptians.\\nSummary.\\nThe traditional view is that Western civilization descends from the Romans, who were in turn heir to the Greeks, who, in some accounts, were heirs to the Egyptians. Frankopan argues that the Persian empire was the center point of the rise of humanity. In the silk roads, Frankopan wants to change the view point of reader to History and make new sight.\\nAuthor.\\nFrankopan is a historian at Oxford University, where he is Senior Research Fellow at Worcester College Oxford and Director of the Oxford Centre for Byzantine Research. He works on the history of the Mediterranean, the Balkans, the Caucasus, Russia and on relations between Christianity and Islam. He also specializes in medieval Greek literature, and translated \\\"The Alexiad\\\"for Penguin Classics (2009). Frankopan often writes for the international press, including \\\"The New York Times\\\", \\\"The Washington Post\\\", \\\"The Guardian\\\", \\\"MoneyWeek\\\". He has also contributed to TV and Radio documentaries about the Byzantine Empire, Divine Women, Roman Law and the Code of Justinian, the Crusades, Varangian mercenaries and the reign of Ivan the Terrible. His new book is \\\"The Silk Roads: A New History of the World .\\\"\\nReception.\\nReviews on the silk roads by Peter Frankopan was published at the guardian, The Independent, The telegraph newspaper and The Times.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48126980\",\"title\":\"Evolution and the Humanities\",\"body\":\"\\nEvolution and the Humanities\\n\\nEvolution and the Humanities is a 1987 book by David Holbrook that attacks Darwinian evolution. The book rejects reductionist biology and takes influence from Michael Polanyi and vitalist philosophy. \\nReception.\\nThe book has been heavily criticized by academics. Martin Stuart-Fox noted that Holbrook's criticism of natural selection was a \\\"cobble together, in a sort of scissors-and-paste criticism... the book contains no vigorous argument at all. Not only is Holbrook very obviously no scientist, he is no philosopher either.\\\"\\nEcologist Arthur M. Shapiro in a review for the National Center for Science Education commented:\\nDavid Holbrook, Fellow of Downing College, Cambridge, has written a polemic not so much against evolution as against scientific reductionism (which he sees incarnate in neo-Darwinism). He proceeds from revulsion at the existentialist vision of \\\"life as a 'scientific accident.' \\\" He's no creationist but, rather, a from-the-gut free-form vitalist—just as preoccupied with the perceived moral consequences of the Darwinian revolution as any Bible-thumping moralist could be. As usual, he conflates science with scientism and evolution with evolutionism, materialism, and atheism.\\\"\\nThe book is said to have been poorly edited and riddled with errors.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127002\",\"title\":\"Ryan Walker (cartoonist)\",\"body\":\"\\nRyan Walker (cartoonist)\\n\\nRyan Walker (1870-1932) was an American political activist and cartoonist. A prolific artist who published political cartoons in a variety of radical newspapers and magazines in the United States, Walker is best remembered as the creator of the recurring character \\\"Henry Dubb,\\\" an American worker who ambled through life blithely being victimized by capitalism ostensibly as a result of his blind acceptance of the ideas of the ruling class.\\nA member of the Socialist Party of America during his younger years, Walker's political views hardened with the coming of the Great Depression in 1929 and he joined the Communist Party, USA the following year, joining the editorial staff of the party's English-language daily newspaper in New York City. Walker died of pneumonia in June 1932 while on a visit to the Soviet Union.\\nBiography.\\nEarly years.\\nRyan Walker was born in Springfield, Kentucky on December 26, 1870. His father, Edwin Ruthwin Walker, was a farmer who later became a lawyer and moved the family to the Midwestern metropolis of Kansas City, Missouri, where Ryan attended public school.\\nShowing a proclivity for art from an early age, submitting his first free lance cartoons to \\\"Judge\\\" in 1883, at the age of 13. These were not of sufficiently finished quality to appear in the pages of the magazine, but the ideas were accepted and redrawn into a two page center spread and back cover cartoon by a house artist, for which Walker received a royalty check of $15. He received positive reinforcement from the magazine's editor to continue at the cartooning craft and to hone his drawing skill.\\nUpon leaving school Walker studied for a time at the Art Students' League in New York City, before working for a number of years in a series of manual jobs, refining his drawing in his spare time. \\nWalker's private portfolio grew until in 1895 he was finally able to land his first permanent artistic job, a position in the advertising department of the \\\"Kansas City Times.\\\" He showed aptitude with a pen and took an acute interest in political issues and he was shortly made an editorial cartoonist for that paper, remaining in that position until 1898. In that year he moved to St. Louis, Missouri to take a position as cartoonist for the \\\"St. Louis Republic,\\\" where he would stay until 1901, taking time to get married to his Kansas City sweetheart, journalist Maud Helena Davis, in October 1899.\\nThe year 1901 saw a move to Boston, where he did a stint as cartoonist for the \\\"Boston Globe,\\\" leaving after one winter. He subsequently worked as a freelance cartoonist, publishing work in a variety of prominent newspapers and magazines of the day, including the \\\"New York Times,\\\"\\\"Life,\\\" \\\"The Arena,\\\" and \\\"The Bookman.\\\" \\nWalker moved from freelance to syndicated work in 1904, joining with the Baltimore based International Syndicate as a cartoonist. He would stay with that firm until 1911, when he left to pursue political pursuits fulltime.\\nSocialist years.\\nFrom 1902 Walker contributed material to a glossy socialist monthly published in New York City during the first five years of the 20th Century, \\\"The Comrade.\\\"\\nWalker turned a close eye to the social problems of his day and developed politically radical views, declaring in a 1905 interview that\\n\\\"My aim, hope, and life-work is the betterment of my brother man. Nothing else counts. I believe the present economic system is cruel, unjust, and essentially wrong, and wrong is wrong, no matter how it may be disguised... I am a Socialist because I believe that Socialism will lead to the development of the greater self, to the out-blossoming of all that is finest and highest in individual life, and that it will secure for all the people a measure of prosperity, happiness, and freedom...\\\"\\nIt was through the pages of the socialist \\\"Appeal to Reason,\\\" a mass circulation weekly published in Southeastern Kansas to which he began regularly contributing in 1906, that Walker first gained popular fame. It was in those pages that he first introduced the character \\\"Henry Dubb,\\\" an American worker who unthinkingly rejected the ideas of unionism and socialism, only to accept as inevitable his victimization by the violence and corruption of the social system around him. Exposed as a dupe and a fool by his worldly wife and somehow cognizant child, the oblivious and intractable protagonist would respond to his latest existential insult with an unblinking stare into space and the catchphrase \\\"I'm Henry Dubb!\\\" — an easy to understand depiction of the effects resulting from so-called false consciousness among the working class.\\nWalker would come to publish two collections of Henry Dubb cartoons in hard covers, \\\"Adventures of Henry Dubb\\\" (1914) and \\\"New Adventures of Henry Dubb\\\" (1915), testimony to the character's enduring appeal among American radicals during the decade of the 1910s.\\nIn addition to a steady stream of cartoons, beginning in 1912 Walker toured the country as a stump speaker on behalf of the Socialist Party of America, Walker was effective in his role as a socialist lecturer, so much so that his abilities were lauded by party orator Eugene V. Debs, who declared him a \\\"great cartoonist\\\" and \\\"equally great\\\" public speaker, who could delight and hold an audience of socialists and non-socialists alike. Walker was directly employed by the National Office of the Socialist Party as one of the touring speakers for its Lyceum Bureau from 1915 to 1916, working in close association with Lyceum director and future Communist Party leader L. E. Katterfeld.\\nFollowing a conservative turn of the editorial line of the \\\"Appeal\\\" during the years of World War I, Walker would take his art and the Henry Dubb character to the pages of the New York City socialist daily \\\"New York Call\\\" and later to its successor, the \\\"New Leader.\\\" The left wing press being notoriously underfunded, Walker would supplement his socialist-related work with regular employment as the Director of the Art Department of the \\\"Evening Graphic\\\" newspaper in New York City, where he was employed from 1924 to 1929.\\nCommunist years.\\nIn the fall of 1930 Walker joined the Communist Party, USA and assumed a position as a staff cartoonist for \\\"The Daily Worker,\\\" the party's New York City-based newspaper. In conjunction with his new role, Walker created a new regularly recurring character for his cartoons, a stolid proletarian known as Bill Worker.\\nDeath and legacy.\\nDuring a visit to the Soviet Union in the spring of 1932, Ryan Walker fell ill and was forced to be hospitalized. He died of pneumonia at Rotkinsky hospital in Moscow on June 22, 1932. He was 61 years old at the time of his death.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127012\",\"title\":\"Stubbs Road (constituency)\",\"body\":\"\\nStubbs Road (constituency)\\n\\nStubbs Road () is one of the 13 constituencies in the Wan Chai District of Hong Kong which was created in 1994.\\nThe constituency loosely covers Stubbs Road in Hong Kong Island with the estimated population of 14,203.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127021\",\"title\":\"Nguyễn Tiến Duy\",\"body\":\"\\nNguyễn Tiến Duy\\n\\nNguyễn Tiến Duy (born 29 April 1991) is a Vietnamese footballer who plays as a Defender for V-League club Than Quảng Ninh F.C. and the Vietnam national football team.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127022\",\"title\":\"Chief Justice of Grenada\",\"body\":\"\\nChief Justice of Grenada\\n\\nThe Chief Justice of Grenada is the head of the Supreme Court of Grenada which consists of the High Court with three justices and a two-tier Court of Appeal. \\nThe original High Court of Grenada was replaced by the Windward and Leeward Islands Supreme Court and the Windward and Leeward Islands Court of Appeal in 1939; both of the latter were replaced in 1967 by the Eastern Caribbean Supreme Court which performs both functions. The Eastern Caribbean Supreme Court, known in Grenada as the Supreme Court of Grenada and the West Indies Associated States, is headquartered in St Lucia, and is now the superior court of record for Grenada and the other Caribbean states which comprise the Organisation of Eastern Caribbean States.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127026\",\"title\":\"TSS St Patrick (1947)\",\"body\":\"\\nTSS St Patrick (1947)\\n\\nTSS \\\"St Patrick\\\" (III) was a passenger vessel operated by the Great Western Railway from 1947 to 1948 and British Railways from 1948 - 1972 \\nHistory.\\nShe was built for the Great Western Railway in 1947 as one of a pair of new vessels for the Fishguard to Rosslare service, the other being TSS \\\"St David\\\". She replaced a former ship of the same name which had been sunk by torpedo on 13 June 1941. British Railways took ownership in 1948 and she was based in Weymouth. Typically running services to Cherbourg, she was also used in the summer for trips from Torquay to the Channel Islands. In 1963 she was transferred to Southampton for services to St Malo and Le Havre, and in 1965 she moved to Folkstone for the service to Boulogne.\\nShe was sold in 1972 to Gerasimos S. Fetouris, in Piraeus, and renamed \\\"Thermopylae\\\". She was sold again in 1973 to Agapitos Bros, Piraeus and renamed \\\"Agapitos I\\\". Scrapped in 1980 in Greece.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127028\",\"title\":\"School violence in Australia\",\"body\":\"\\nSchool violence in Australia\\n\\nThe Education Minister of the State of Queensland said in July 2009 that the rising levels of violence in schools were \\\"totally unacceptable\\\" and admitted that not enough had been done to combat violent behaviour. 55,000 students had been suspended in the state's schools in 2008, nearly a third of which were for \\\"physical misconduct\\\".\\nIn South Australia, 175 violent attacks against students or staff were recorded in 2008. Students were responsible for deliberately causing 3,000 injuries reported by teacher over two years from 2008 to 2009.\\n46% of Principals in Western Australia have been either physically assaulted or witnessed physical violence in schools during 2012. 70% of school leaders had also been threatened with violence. Schools in Western Australia, South Australia and the Northern Territory rated far higher than other states in terms of threats of violence.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127054\",\"title\":\"Melkwezer dialect\",\"body\":\"\\nMelkwezer dialect\\n\\nThe Melkwezer dialect (Standard Dutch: \\\"Melkwezers\\\", ) is a subdialect of Brabantian spoken in Melkwezer, a town in the Linter municipality.\\nPhonology.\\nThe phonology of the Melkwezer dialect is similar to that of the Orsmaal-Gussenhoven dialect, but they are not identical. For instance, the diphthong in the Orsmaal-Gussenhoven dialect corresponds to in the Melkwezer dialect.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127102\",\"title\":\"Here I Am a Stranger\",\"body\":\"\\nHere I Am a Stranger\\n\\nHere I Am a Stranger is a 1939 American drama film directed by Roy Del Ruth and written by Sam Hellman and Milton Sperling. The film stars Richard Greene, Richard Dix, Brenda Joyce, Roland Young, Gladys George and Kay Aldridge. The film was released on September 28, 1939, by 20th Century Fox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127104\",\"title\":\"Shaw Academy\",\"body\":\"\\nShaw Academy\\n\\nShaw Academy is a professional, online, higher education institution.\\nShaw Academy was founded in 2013.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127143\",\"title\":\"ATIV Software\",\"body\":\"\\nATIV Software\\n\\n is a privately held company based in Santa Rosa, California that develops the EventPilot® conference and journal mobile apps for iOS, Android and web-based devices. EventPilot conference apps help large meetings go paperless and reach sustainability goals.\\nHistory.\\nATIV Software was co-founded by Silke Fleischer and Eric Converse in 2010 to provide native apps for events that save meeting organizers time and money.\\nProducts and Services.\\nEventPilot Conference App\\nIn 2014, ATIV Software partnered with Radius Networks to incorporate iBeacon technology with EventPilot conference apps, enabling indoor positioning and proximity-aware notifications that send targeted messages to event attendees based on their indoor location. In 2015, EventPilot released an interactive maps feature and an online itinerary planner.\\nEventPilot Journal App\\nThe EventPilot Journal App is a native mobile app offering for medical and scientific peer-reviewed publications.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127158\",\"title\":\"Adam Roxburgh\",\"body\":\"\\nAdam Roxburgh\\n\\nAdam Roxburgh (born 14 April 1970) is a rugby union coach and former player who made eight appearances for the Scotland national rugby union team. He was known for his entertaining play in rugby sevens games.\\nEarly life.\\nRoxburgh was born in Edinburgh.\\nRugby playing career.\\nHe played club rugby for Kelso RFC.\\nHe made his international rugby debut on 22 November 1997 against Australia at Murrayfield. His last appearance was against Australia at Brisbane during the 1998 Scotland rugby union tour of Oceania.\\nRoxburgh was a talented rugby sevens player. With Kelso he competed in the Dubai Sevens, 1993 Rugby World Cup Sevens and has won three Kings of the Sevens. In 1997, he won the Middlesex Sevens with the Barbarians.\\nWhen many of his rugby-playing contemporaries turned professional in 1997, Roxburgh remained with the firm of precision tool makers, Abbey Tool and Gauge in Kelso, who had been his first employer from school. He retired from club rugby in 2005.\\nCoaching.\\nBy 2008 he was coaching the Kelso sevens team and the Kelso second fifteen. He took over as a head coach at Kelso from 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127164\",\"title\":\"Moridilla brockii\",\"body\":\"\\nMoridilla brockii\\n\\nMoridilla brockii is a species of sea slug, an aeolid nudibranch, a marine gastropod mollusc in the family Facelinidae.\\nThis species is considered a nomen dubium.\\nDistribution.\\nThe type locality of \\\"Moridilla brockii\\\" is the Sunda Sea which is an old name for the Flores Sea. This species is considered to be widespread in the Indo-Pacific region. It was redescribed from the Gulf of Mannar, India.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127171\",\"title\":\"Shmuel Shapiro\",\"body\":\"\\nShmuel Shapiro\\n\\nShmuel Raphael Shapiro (, born 1974 in Aix les Bains, France), known by his stage name Shmuel Shapiro, is a French Chassidic-Jewish singer, composer and musician. He is especially popular in Israel and Europe, from the early 1990s to this day.\\nEarly years.\\nAt the age of nine, he moved to Israel with his family, studied in the Chassidic Yeshivot of Rachmastrivka and Erloy. Shmuel started his musical career very early. He composed his first song at the age of eleven and became the main soloist of the Erloys choir at seventeen. then he approached the Chabad movement.\\nCareer.\\nIn 1994, after his marriage, Shmuel studied in a Kollel and founded the orchestra \\\"Kol Haneguina\\\" with the famous artists Daniel Mamane and Haim Kirshinboim.\\nShmuel then studied music, worked his voice and learned 'Hazanout in the great school of maestro Naftali Hershtik, Elie Jaffe and Benjamin Munk\\nFrom 2000 he became the most requested conductor for Jewish choirs in France, to this day.\\nHe also took lessons with musicians and opera singers. In 2001, Shmuel became cantor of the Great Synagogue of Migdal Haemek, of the Rav Yitzchak Dovid Grossman. Very dynamic, Shmuel also animated the Israely Radio Kol Hanechama in Jewish thought and music. Shmuel returned to France in 2006 as cantor of the Great Synagogue \\\"Ohel Avraham\\\" Montevideo st. (Paris 16e).\\nIn 2009, he became director of a choir and member of the board of the \\\"Nigounit\\\" music school.\\nShmuel was proposed in 2010 a global music project by musician was rejected by him on religious ground..\\nShmuel has recorded several music titles during his career and his first album \\\"Seu Morom\\\" was recorded in 2013 in France and Israel, with the arranger Didier Atlan. His songs are included in weddings, synagogues and Yeshivot in Israel and other countries. The majority of the CD songs were composed by Shmuel. Shmuel's song named \\\"Ahavat Israel\\\" was proposed purchase by singers and music professionals worldwide. Finally buying rejected from producer of Shmuel.\\nShmuel Shapiro appears regularly with famous singers like Enrico Macias (2013), Yoram Gaon (2008), Avraham Fried (2005), Mordechai Ben David (2004), (2006), Dedi Graucher (2006), Yosi Piamenta (2005), (2006), Ohad Moskowitz (2004), Isaac Bitton (2006), (2000–2003), (2006), yehuda glantz (1999), (2009), Benny Friedman (2013).\\nPersonal life.\\nShmuel is married and has three children, lives in Paris.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127248\",\"title\":\"Speed Sisters\",\"body\":\"\\nSpeed Sisters\\n\\nSpeed Sisters is a 2015 documentary film that follows the all-female Palestinian racing team the Speed Sisters and explores the social issues surrounding their career.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127258\",\"title\":\"James Marshall Campbell\",\"body\":\"\\nJames Marshall Campbell\\n\\nJames Marshall Campbell (1895-1977) was dean of the college of arts and sciences at The Catholic University of America. Campbell was a classical scholar, and a member of the department of Greek and Latin. He was on the faculty of the Catholic University from 1945-1966.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127272\",\"title\":\"Valencian People's Union\",\"body\":\"\\nValencian People's Union\\n\\nThe Valencian People's Union (in Valencian: \\\"Unitat del Poble Valencià\\\", UPV) was a political party created in 1984 when the Left Grouping of the Valencian Country (AEPV) and the Nationalist Party of the Valencian Country (PNPV) merged and became an unified party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127308\",\"title\":\"DhoomBros\",\"body\":\"\\nDhoomBros\\n\\nOriginally from Pakistan and currently based in New York, Hussain Asif, Shehryaar Asif, Atif Khan and Waqas Riaz, collectively called the DhoomBros, are known globally for their success through YouTube, and are now also working their drama Hum Kahan Chal Diye along with many other projects. Since creating their first channel (DhoomBros) in 2008, their videos have gathered over 9 million combined views, including videos from their more recent channel (TheDhoomBros) made in 2012. They are particularly known for their satirical takes on everyday Desi lives, parodies of mainstream entertainment, video blogs, short films, and their Mehndi dance performances.\\nCareer.\\n2008–11: Early beginnings and background.\\nThe DhoomBros are a four-member Pakistani-American group that includes \\\"Hussain Asif\\\", \\\"Shehryaar Asif\\\", \\\"Atif Khan\\\", and \\\"Waqas Riaz\\\" and they are based in the United States of America. The idea of calling themselves \\\"DhoomBros\\\" came within the name itself: “Dhoom” Bros (\\\"Dhoom\\\" = \\\"Blast\\\" in Hindi). The song “\\\"Dhoom Again\\\"”, from the 2006 Bollywood film Dhoom 2 was a popular hit amongst South Asians but the Western world was generally unaware of it. Brothers Shehryaar (informally known as \\\"Shehry\\\") and Hussain Asif performed to that song in a school talent show and received recognition from their peers and the school faculty. At this point, they called themselves the DhoomBros. This motivated them to entertain people through dance and various acts and so on June 23, 2008, the DhoomBros officially created their first YouTube channel (DhoomBros) and their official Facebook page. Atif Khan, the group's mutual friend, and Waqas Riaz, the cousin of the Asif brothers, joined later because they had similar interests.\\nThey Started Off Making videos such as \\\"Jimmy And Saleem' and \\\"The Desi Mobsta Show\\\". They also would upload videos of themselves talking about weird things that have happened to them throughout their week, either at home,at school, or in daily life. \\nWhen they started out they had nicknames, Hussain was, \\\"Desi Mobsta\\\" or \\\"Don\\\". Waqas was \\\"Daku\\\", Shehryaar was just \\\"Shehry\\\".\\nNow, Hussain has a refurbished nickname \\\"Hussain Superstar\\\", Atif is \\\"Ati\\\" or \\\"Khan Saab\\\".\\n2012–2013.\\nIn 2012, DhoomBros videos became more frequent from \\\"Stuff Pakistanis Say\\\", a video that comically represents common Pakistani household events and phrases, to \\\"Humsafar Best Scenes Parody\\\", a parody based on the Pakistani drama serial, Humsafar, to the \\\"Annoying Brown Girls/Guys\\\" series, videos that show banter that stereotypical Desi teenagers seem to portray.\\nIn October 2012, their popularity increased after the DhoomBros released their video, \\\"Desi Style\\\", a parody based on South Korean pop artist Psy's hit single, Gangnam Style. The DhoomBros chant in English, Urdu and Punjabi about their everyday Desi American lives, consisting of waking up to their mother’s voice, going to school, and greeting their grandfather, all accompanied with song and dance.\\nThe group also experiment outside comical videos. \\\"Blessings of Ramadan\\\" is a video portraying three young men who indulge in materialistic aspects of their lives and fall astray from their spirituality by forgetting the significance of the month of Ramadan, the Muslim month of fasting and prayer. In addition, \\\"The Perfect Memory\\\" is the DhoomBros' first short film about the unexpected sacrifices one may have to make when it comes to love. Both videos were unlike the group's usual comical parodies, but were still well-garnered by audiences.\\nTwo of the members, Shehryaar Asif and Waqas Riaz, are known for uploading occasional video blogs that highlight various aspects of the lives of the DhoomBros, such as showing sneak peeks of their projects and being DJs at work. They also document their trips, such as to different cities in the United States of America and their stay in Pakistan. In addition to being YouTube entertainers, the DhoomBros are also DJs and often perform at weddings, mehndi events, and other parties. Apart from their comical videos, their mehndi performances have also gathered many views.\\n2014–Present.\\nOne of the DhoomBros' most recent popular video is the \\\"Evolution of Bollywood Dance\\\", inspired by Penn Masala's Evolution of Bollywood Music. In this video, the DhoomBros recreate the dancing styles that the Bollywood film industry has witnessed over the years from the golden-aged actors From Raj Kapoor to Shammi Kapoor and Jeetendra to Dharmendra and Mithun Chakraborty and to the stars of today such as Hrithik Roshan, Salman Khan, and Shah Rukh Khan.\\nAfter debuting in their first venture in the web-series, \\\"iDeewane\\\", the DhoomBros are currently in the pre-production status for their Drama: Hum Kahan Chal Diye.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127312\",\"title\":\"Alcides Greca\",\"body\":\"\\nAlcides Greca\\n\\nAlcides Greca (February 13, 1889 in San Javier, Saint Faith, Argentina – April 16, 1956 in Rosario, Saint Faith, Argentina) was a lawyer, journalist, cineasta, jurist, teacher, writer and Argentinian politician, that directed the film \\\"The last malón\\\" of 1917 and defended the native cause in Saint Faith, in addition to being socialist, radical and adherent of the university reform.\\nBiography.\\nFirst years.\\nAlceides Was born as a second son of the marriage of don Francisco Greca and of donya Magdalena Trucco, immigrant of Italian and French origin, the February 13, 1889, in the town of San Javier, Saint Faith. It Would be part of a numerous family, composed by eleven brothers more: Silvio, Erminda, Francisco, Alejandro, Teodora, Magdalena, Ana, Mercedes, María Isabel, Clorinda and Gilds.\\nTo the same city receive his first education, and has as a companions to the native mocovíes, which thing later influenced in his works. For the secondary level moves to the city of Saint Faith, in Inmaculada's School Concepción of the jesuïtes, and finishes the secondary in the National School Simón of Iriondo, being egressat of the first promotion of students.\\nStarts of his literary and journalistic career.\\nLater it travels to the Silver, Buenos Aires, to study advocacia, and writes his first lines of \\\"the Evangeli Rebellious\\\", \\\"Laurels of the Marsh\\\", and \\\"Black Tears.\\\" Being missing a lot little to receive, abandons his career to be elect deputy for his department in 1912.\\nIn 1907, it decides to begin his journalistic career, and founds his first periodic \\\"The Mocoví\\\", to San Javier. Next, Alcides would found the periodic \\\"The Pure Truth\\\", \\\"The Paladín north\\\" and \\\"The Word\\\" (antecedent of the current \\\"The Litorial\\\"), in 1912, 1914 and 1915 respectively. These last two would be established to the city of Saint Faith. Besides, colaboriaría with the newspaper Rosario's \\\"The Capital\\\" and the magazine of Juridical and Social Sciences of University of the Seaboard.\\nIn the meantime, in 1909, public his first book, \\\"Words of Quarrel.\\\" Later, it finishes \\\"Symphony of the Sky\\\", a series of hymns in prose, and \\\"Black Tears\\\", a history of madness and pain, the year 1910. And in 1915 public \\\"Laurels of the Marsh\\\" and \\\"The Evangeli Rebellious\\\".\\n\\\"The last malón\\\".\\nIt is reelecto provincial deputy in 1916, and in 1917 films \\\"The last malón\\\", feel east the first llargmetratge Argentinian in 35 millimeters that films to the interior of the country. In the same recreates the last native rebellion become in territory santafesino, fruit of the pursuit to the aborigine since the Law of Territories of 1884. It bases in the facts succeïts in the month April 1904. This film is produced in Rosario, for Greca Film Cinematographic Company Rosarina, in Av. Pellegrini To the 1655. Also it finishes his studies by right in The Silver.\\nIn 1920 it contracts matrimonico with Rosa Pierri, known how \\\"Roxana.\\\" This same year is elect senator, and deputy constituent for the reform of the Provincial Constitution. In 1921 nacé the only son of the couple, Alcibíades Alejandro Greca. \\nIt is appointed headline by right Administrative and Municipal Right Compared, in the Faculty of Juridical and Social Sciences of the University of the Seaboard, where would dictate chair for 27 years, comenzandó the July 5, 1921. For these years realises some voyages for the interior of the country, and in 1923 visit Chile, Peru, Bolivia and Uruguay, beside the brothers Ángel and Alfredo Guido. Two years next would be elect national deputy for the second time.\\nAlready in 1927 it begins to públicar several books. The first novel públicada would be in this year, \\\"North Wind\\\", that deserved elogiosos trials of the national and foreign press. In 1929 it appears \\\"The Tower of the English\\\", which contains chronic of the voyages realised the year 1923, and in 1932 public \\\"Tales of the Committee\\\".\\nIt Would be reelectó national deputy in 1930, but can not finalise his mandate because of the coup d'état of the general Uriburu, and the December 9, 1933 is detained and brought to the island Martín García, because of different motívos political. After going out of her, public \\\"After the alambrado of Martín García\\\".\\nIn Chile, Alcides public \\\"The Pampa Gringa\\\", being the protagonist of this novel the south santafesino, in 1936, and, already in Saint Faith, public \\\"spiritual Tragedy of the Argentinians that today are 20 years old\\\", as an edition of the UNL.\\nHis last years.\\nWith his esposa, travel for Europe the year 1951, but have to go back before for a granddaughter's demise. In 1954, jubila as a university teacher, buys a chacra and moves to the city of Oliveros, where devotes to cultivate his horta and garden.\\nLater, it would be him diagnosed pancreatitis, and is moved urgently to Rosario's a hospital, but the April 16, 1956 dies in a room of the Italian Hospital.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127331\",\"title\":\"The Araki\",\"body\":\"\\nThe Araki\\n\\nThe Araki is a sushi restaurant run by Japanese chef Mitsuhiro Araki which opened in London in 2014. It was awarded two stars in the 2016 Michelin Guide for the UK and Ireland.\\nDescription.\\nChef Mitsuhiro Araki had previously run a restaurant in Tokyo called Araki for which he held three Michelin stars, but chose to close it in February 2013 in order to pursue a new challenge. He had considered New York, Paris, and Singapore, but chef Joël Robuchon suggested London to him. The move took three years to organise.\\nThe interior of the restaurant, designed by the Takenaka Corporation, features a counter made from 200-year-old cypress wood gifted to Araki by Japanese musician Ryuichi Sakamoto. That counter is divided from the kitchen by a pair of green curtains, and the overall look of the interior is based upon the architecture of the Japanese Edo period. There are only fifteen seats – nine at the counter and six in a private dining room. The restaurant formed part of a £250 million redevelopment of the combined office and retail space at Burlington Street.\\nFare.\\nThe restaurant offers a single choice of a set menu consisting of eleven courses, with two sittings taking place each evening. Araki has modified the style of sushi he has made to take into account European produce, such as salmon sushi, Cornish squid, and langoustines. Other dishes include similar designs to those he used at his previous restaurant, but from sources within Europe, such as his signature tuna sashimi, in which he serves three different cuts of the fish.\\nReception.\\nChef Jason Atherton called the food at The Araki \\\"mind blowing\\\" and said that it was as good as any restaurant in Japan. The Araki was named the best Japanese restaurant in London by \\\"Tatler\\\" magazine in their 2015 restaurant guide. In September 2015, The Araki was awarded two stars in the 2016 Michelin Guide for the UK and Ireland. It was one of two Japanese restaurants in London to be elevated to the two-star level that year, the other being Umu, which is located a short distance away from The Araki.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127344\",\"title\":\"Porotos de soja\",\"body\":\"\\nPorotos de soja\\n\\nPorotos de soja is an Argentinian documentary film produced by the National Institute of Cinema and Audiovisual Arts (INCAA), directed by David Blaustein and Osvaldo Daicich, written by Daicicht. The film premiered on 21 May 2009.\\nSynopsis.\\nThe film was based on the investigation of Mariana Dosso, Silvina Segundo and Osvaldo Daicich. It examines a conflict between the agricultural producers and the government that occurred in Argentina in 2008.\\nThe conflict erupts when the producers, grouped in four employers' organizations, rejected the system of mobile retention to the exports of soy and sunflower established by the Resolution 125. In the new system quotas were replaced by an agreeable pricing standard where the percentage of compensation was equal to or greater than the international pricing standards of the product. One of the arguments was over an alleged fault regarding the Executive Powers ability to implement the measure.\\nThe Argentine National Congress established the new system, finalizing the dispute on 17 July 2008.\\nThe film presents the opinions of the philosopher Ricardo Forster, the journalist and economist Alfredo Zaiat and the graduate in communication Mariana Moyano.\\nProduction.\\nThe director declared that the idea of the film was born on April 1, 2008 when he decided to cover an act by President Fernández of Kirchner in support of this resolution. The initial aim was to produce a special program for television but afterwards with the prolongation and deepening of the conflict, it was transformed into a film. Produced by the state body National Institute of Cinema and Audiovisual Arts (NICAA) it was premiered in cinemas on 21 May 2009 and was exhibited on the state television channel two days later.\\nCritical reception.\\nThe newspaper \\\"La Nación\\\" highlighted the camera work and said that the directors \\\"attained to radio-graph this problem that had in check to the population, while sociologists, journalists, politicians and writers expose his points of view about a thematic that had an enormous transcendence for the agricultural sector\\\". It added that the film attempts \\\"to clarify some of the most complicated elements that were born from that resolution 125 and visits by means of a photography of undoubted quality fields, stays and places that saw affected by this governmental imposition\\\" and stands out that \\\"it shows, in definite, the fight of the men of the field for conserving his belongings, still by the expense of a fight that had few paragons in the history of the country\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127346\",\"title\":\"Kevin Quinn (actor)\",\"body\":\"\\nKevin Quinn (actor)\\n\\nKevin Quinn (born May 21, 1997, in Chicago, Illinois) is an American actor known for his starring role in the Disney Channel's \\\"Bunk'd\\\".\\nCareer.\\nHe began his career by appearing in episodes of \\\"Shameless\\\" and \\\"Chicago P.D.\\\" Before being on Disney, he auditioned for season 12 of \\\"American Idol\\\". He wound up being one of the top 60 males in the country. He then played Jonny in Steppenwolf Theater's production of \\\"Lord of the Flies\\\", and a boy in a Chicago Shakespeare Theater adaption of \\\"Henry V\\\". Quinn's feature film debut was in the 2015 independent film \\\"Kids and Ghosts\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127369\",\"title\":\"Av8er Explorer\",\"body\":\"\\nAv8er Explorer\\n\\nThe Av8er Explorer is a British paramotor that was designed by Paul Taylor and produced by Av8er Limited of Woodford Halse, Northamptonshire for powered paragliding. Now out of production, when it was available the aircraft was supplied complete and ready-to-fly.\\nDesign and development.\\nThe Explorer was designed to comply with the US FAR 103 Ultralight Vehicles rules as well as European regulations. It features a paraglider-style wing, single-place accommodation and a single engine in pusher configuration. The aircraft is built with special attention to balancing and vibration isolation. The cage assembly includes small wheels to ease ground movement of the motor unit.\\nAs is the case with all paramotors, take-off and landing is accomplished by foot. Inflight steering is accomplished via handles that actuate the canopy brakes, creating roll and yaw.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127400\",\"title\":\"Waulsortian mudmound\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127469\",\"title\":\"Av8er Observer Light\",\"body\":\"\\nAv8er Observer Light\\n\\nThe Av8er Observer Light (or Lite) is a British paramotor that was designed by Paul Taylor and produced by Av8er Limited of Woodford Halse, Northamptonshire for powered paragliding. Now out of production, when it was available the aircraft was supplied complete and ready-to-fly.\\nDesign and development.\\nThe Observer Light was designed to comply with the US FAR 103 Ultralight Vehicles rules as well as European regulations. It features a paraglider-style wing, single-place accommodation and a single engine in pusher configuration. The aircraft is built with special attention to balancing and vibration isolation. The cage assembly includes small wheels to ease ground movement of the motor unit.\\nAs is the case with all paramotors, take-off and landing is accomplished by foot. Inflight steering is accomplished via handles that actuate the canopy brakes, creating roll and yaw.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127474\",\"title\":\"Masjid An-Noor\",\"body\":\"\\nMasjid An-Noor\\n\\nMasjid An-Noor, or variations such as Masjid-an-Noor and Masjid-e-Noor, may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127492\",\"title\":\"Steve Cassano\",\"body\":\"\\nSteve Cassano\\n\\nSteve Cassano (born March 7, 1947) is an American politician in the state of Connecticut. He is a Democratic member of the Connecticut State Senate, having first been elected in 2010. He previously served as Mayor of Manchester, Connecticut from 1991 to 2005. Cassano is an alumnus of Manchester Community College, Boston State College (BA), the University at Albany, SUNY (MA) and the University of Connecticut (MSW).\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127496\",\"title\":\"Pedomicrobium ferrugineum\",\"body\":\"\\nPedomicrobium ferrugineum\\n\\nPedomicrobium ferrugineum is a rod-shaped, aerobic to microaerophillic bacterium from the genus of Pedomicrobium with one polar or supolar flagellum. \\\"Pedomicrobium ferrugineum\\\" has been isolated from podzolic soil in north Germany\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127499\",\"title\":\"Joel Larsson\",\"body\":\"\\nJoel Larsson\\n\\nJoel Larsson is a Swedish professional \\\"\\\" player. He is most known for winning the Pro Tour Magic Origins competition in August 2015, and for his skills in limited formats. Joel had the highest win percentage in the world in limited matches during the 2011-2012 Pro Tour season.\\nJoel began playing \\\"Magic\\\" in 2005 with the Saviours of Kamigawa set, and qualified for his first Pro Tour in 2010, making his debut at Pro Tour San Diego. Due to his hairdo and appearance, he received the nickname of \\\"Swedish Kibler\\\" after the 2013 finals of Pro Tour Gatecrash in Montreal. In Sweden, he often goes by the nickname \\\"Proel\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127500\",\"title\":\"List of Ministers of the Interior of the Netherlands\",\"body\":\"\\nList of Ministers of the Interior of the Netherlands\\n\\nThe Minister of the Interior and Kingdom Relations () is the head of the Ministry of the Interior and Kingdom Relations and a member of the Cabinet of the Netherlands. Ronald Plasterk has been the Minister of the Interior and Kingdom Relations since 5 November 2014.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127505\",\"title\":\"Joel Edmundson\",\"body\":\"\\nJoel Edmundson\\n\\nJoel Edmundson (born June 28, 1993) is a Canadian professional ice hockey defenseman who is currently playing for the St. Louis Blues of the National Hockey League (NHL).\\nPlaying career.\\nEdmundson was chosen in the sixth round, 121st overall, of the 2008 WHL Bantam Draft by the Moose Jaw Warriors. In his WHL rookie season, Edmundson played in 71 games for the Warriors. He scored 2 goals and added 18 assists for 20 points while adding on 95 penalty minutes. He later played for the Kamloops Blazers. In March 2013, Edmundson signed an entry-level contract with the Blues. He played the following two seasons for the Blues' American Hockey League affiliate, the Chicago Wolves.\\nIn October 2015, it was it was announced that Edmundson had made the opening night roster of the St. Louis Blues for the 2015–16 NHL season.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127511\",\"title\":\"The Letters (film)\",\"body\":\"\\nThe Letters (film)\\n\\nThe Letters is a 2014 American biographical drama film directed and written by William Riead. The film stars Juliet Stevenson, Max von Sydow, Rutger Hauer and Priya Darshini. It was produced by Colin Azzopardi, Tony Cordeaux, and Lisa Riead. It was released theatrically by Freestyle Releasing on December 4, 2015.\\nPlot.\\nMother Teresa (Juliet Stevenson), recipient of the Nobel Peace Prize, is considered one of the greatest humanitarians of modern times. Her selfless commitment changed hearts, lives and inspired millions throughout the world. The film is told through personal letters she wrote over the last forty years of her life and reveal a troubled and vulnerable woman who grew to feel an isolation and an abandonment by God. The story is told from the point of view of a Vatican priest (Max von Sydow) charged with the task of investigating acts and events following her death. He recounts her life’s work, her political oppression, her religious zeal, and her unbreakable spirit.\\nProduction.\\nDevelopment.\\nRiead began exploring ideas for an inspirational and uplifting film just before the devastating terror attacks of September 11, 2001, forced a realization on him that would come to define his project. “I didn’t know there was that kind of evil in this world until then,” says Riead. “The attacks really brought that home.\\\"\\n\\\"The Letters\\\" became a labor of love for Riead during the fourteen years it would take to produce the film. The turning point for Riead was the discovery of a startling cache of heartfelt, formerly confidential letters written by Mother Teresa to her spiritual advisor, the Belgian Jesuit priest Celeste van Exem, over a nearly 50-year correspondence. In her letters, some of which have been published in the 2007 book \\\"Mother Teresa: Come Be My Light\\\", Teresa revealed a crisis of faith that left her doubting God’s belief in her dedication to the poor. Riead read all of the letters that were available to the public and decided that they would make up the spine of his screenplay.\\nCasting.\\nWhen the time came to cast the film, Riead had a wealth of actresses clamoring to don Mother Teresa’s habit. To play her confessor, Father van Exem, Riead cast Max von Sydow, the Swedish star who has been a favorite of directors ranging from Ingmar Bergman to Martin Scorsese. When von Sydow asked the director for some insight into the character, Riead gave him a simple but telling answer. “I said, the whole world looked up to Mother Teresa,” he recalls. “And Mother Teresa looked up to Father van Exem. He just looked at me for a long moment, and he said, ‘got it.’ And that was it. He showed up and knew exactly what to do.”\\nTo portray the nuns and students of the Loreto Convent, the Bishop of Calcutta, Mother Teresa’s wealthy benefactors, and the residents of the poverty stricken slums, Riead cast professional actors from India’s Bollywood film industry, considered the largest in the world.\\nFilming.\\nThe film was shot primarily in India, with interiors shot in Goa and second unit filming taking place in Calcutta, Delhi, and Mumbai. The scenes featuring Max von Sydow and Rutger Hauer (as van Exem’s confidant, Father Benjamin Praagh), were shot in a 15th-century London monastery.\\nRelease.\\n\\\"The Letters\\\" premiered at the Sedona Film Festival in February 2014. It was released theatrically in 886 venues on December 4, 2015.\\nCritical reception.\\n\\\"The Letters\\\" has received generally negative reviews from critics. On Rotten Tomatoes, the film has a rating of 30%, based on 33 reviews, with an average rating of 4.4/10. The consensus states: \\\"\\\"The Letters\\\" tries to honor Mother Teresa with an unstintingly positive portrayal of her life and works, but ends up smothering a fascinating real-life story under a bland hagiography.\\\" On Metacritic, the film has a score of 25 out of 100, based on 11 critics, indicating \\\"generally unfavorable reviews\\\". \\nAt the Sedona Film Festival, the film won the Audience Favorite “Best of the Fest” Award, and the International Catholic Film Festival in Rome, where Riead was honored as Best Director and Juliet Stevenson as Best Actress.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127519\",\"title\":\"Av8er Orbiter\",\"body\":\"\\nAv8er Orbiter\\n\\nThe Av8er Orbiter is a British paramotor that was designed by Paul Taylor and produced by Av8er Limited of Woodford Halse, Northamptonshire for powered paragliding. Now out of production, when it was available the aircraft was supplied complete and ready-to-fly.\\nDesign and development.\\nThe Orbiter was designed to comply with the US FAR 103 Ultralight Vehicles rules as well as European regulations. It features a paraglider-style wing, single-place accommodation and a single engine in pusher configuration. The aircraft is built with special attention to balancing and vibration isolation. The cage assembly includes small wheels to ease ground movement of the motor unit.\\nAs is the case with all paramotors, take-off and landing is accomplished by foot. Inflight steering is accomplished via handles that actuate the canopy brakes, creating roll and yaw.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127533\",\"title\":\"Haydock railway station\",\"body\":\"\\nHaydock railway station\\n\\nHaydock railway station served the village of Haydock, formerly in Lancashire, now in Merseyside), England.\\nThe station was on the Liverpool, St Helens and South Lancashire Railway line from Lowton St Mary's to the original St Helens Central railway station where it was crossed by what is now the A599 in the centre of the village.\\nEast of the station was the Haydock Colliery Tunnel, the only tunnel on the line. It was built at the railway's expense using the 'cut and cover' method. Its sole purpose was to burrow beneath Haydock Colliery's tracks.\\nHistory.\\nOpened by the Liverpool, St Helens and South Lancashire Railway, as part of the Great Central Railway, it became part of the London and North Eastern Railway during the Grouping of 1923. The line and station passed to the Eastern Region of British Railways on nationalisation in 1948, being transferred to the London Midland Region later that year.\\nThe line through the station was originally double track and the station had two platforms. In the 1930s the down (St Helens-bound) track was changed into a long siding and all trains to and from St Helens used the up line. The station's down side shelter and signs were removed.\\nServices.\\nIn 1922 five \\\"down\\\" (towards St Helens) trains called at the station, Mondays to Saturdays. These called at all stations from Manchester Central to St Helens via Glazebrook and Culcheth. The \\\"up\\\" service was similar.\\nBy 1948 four trains plied between St Helens Central and Manchester Central, calling at all stations, Monday to Friday, reduced to three on Saturdays. \\nA fuller selection of public and working timetables has now been published. Among other things this suggests that Sunday services ran until 1914, but had ceased by 1922 never to return.\\nClosure.\\nThe station was closed to passenger traffic by the British Railways Board in 1952, though goods traffic through to St Helens lingered on until 1965, when the tracks west of Ashton-in-Makerfield were abandoned. A short stretch from Ashton through the Haydock station site to a new Shell distribution depot was reinstated in 1968. This ceased being rail-served in 1983, whereafter the line was cut back to Lowton Metals' scrapyard at Ashton. Tracks were eventually lifted. \\nThe site today.\\nBy 2003 modern housing had completely obliterated the station site.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127535\",\"title\":\"2016 FedEx Cup Playoffs\",\"body\":\"\\n2016 FedEx Cup Playoffs\\n\\nThe 2016 FedEx Cup Playoffs, the series of four golf tournaments that will determine the season champion on the U.S.-based PGA Tour, will be played from August 25 to September 25. It includes the following four events:\\nThese will be the tenth FedEx Cup playoffs since their inception in 2007.\\nThe point distributions can be seen here.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127539\",\"title\":\"Dave Porter (politician)\",\"body\":\"\\nDave Porter (politician)\\n\\nDave Porter is a former Canadian politician, who served in the Legislative Assembly of Yukon from 1982 to 1989. He was a member of the Yukon New Democratic Party.\\nHe was first elected in the 1982 election as MLA for Campbell. He shifted to the district of Watson Lake for the 1985 election, in which he was reelected, and served as House Leader and Deputy Premier in the government of Tony Penikett.\\nHe did not run for reelection in the 1989 election. He was then appointed as executive director of the Yukon Human Rights Commission.\\nPorter later served as CEO of the BC First Nations Energy and Mining Council, and as chief negotiator for the Kaska Dena Council.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127546\",\"title\":\"The Velours\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127553\",\"title\":\"Tokio Emoto\",\"body\":\"\\nTokio Emoto\\n\\n is a Japanese actor who is represented by the talent agency, Knockout.\\nBiography.\\nEmoto's father is Akira Emoto, his mother is Kazue Tsunogae, and his brother is Tasuku Emoto. He is left-handed.\\nIn 2003, Emoto had his acting debut in the film, \\\"Jam Films S\\\".\\nUp until around 2011, he maintained a part-time job in Shimokitazawa while acting. As of July 2015, his job was in a T-shirt shop.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127557\",\"title\":\"P2P Global Investments\",\"body\":\"\\nP2P Global Investments\\n\\nP2P Global Investments () is a large British investment trust dedicated to investments in consumer and small and medium sized enterprise loans and in corporate trade receivables. Established in 2014, the company is a constituent of the FTSE 250 Index. The chairman is Stuart Cruickshank.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127571\",\"title\":\"Vladimir Spirin\",\"body\":\"\\nVladimir Spirin\\n\\nVladimir Semionovich Spirin (Russian: Владимир Семёнович Спирин) (born May 5, 1929 - died May 17, 2002) was a Russian philologist, sinologist, historian, lecturer of Saint Petersburg State University, researcher at Saint Petersburg's branch of the Institute of Oriental Studies of the Russian Academy of Sciences, Saint Petersburg Russia, Candidate of Sciences (equiv. Ph.D.). His primary scientific interests resided in the field of classical Chinese philology and Chinese philosophy.\\nBiography.\\nDr. Spirin graduated in 1952 from the East Asian Studies Department at the Saint Petersburg State University and started to work as a researcher at Saint Petersburg's (then Leningrad) branch of the Institute of Oriental Studies of the Russian Academy of Sciences, where he worked until his death. Since 1957, he had been a member of the research group that worked on description of Dunhuang manuscripts, preserved in Russia (other members were Dr. Lev Menshikov, S.A. Schcolyar, etc.) In the 1960s, during the time of the Cultural Revolution, he conducted studies in China for nine months. He defended his doctoral thesis (Candidate of Sciences, Philosophy) in 1970, under the title of \\\"On Methodological Problems of Studying the Classic Chinese Philosophy: In Relation to the Analysis of Text Structures\\\". Since 1977 and until 1990s, he taught the Classic Chinese Philosophy to philosophy students at the Saint Petersburg State University, as an invited lecturer.\\nHis main area of research had been methodological problems of studying classic Chinese texts. He developed an original structural approach to the texts, and discovered various types of textological structures in the classic Chinese culture. His method of graphic description of textual structures, providing simplicity and easy visualization, according to some researchers, reminds graphic methods of logical description such as Lambert's lines or Eiler's circles, as well as the implementation of graphic description in thermodynamics by Clapeyron. Spirin's work strongly influenced the study of Chinese culture in Russia, especially the younger generation of sinologists, working in Moscow (based in Leningrad he was not in permanent direct contact with these researchers). For example, according to A. Kobzev, Spirin's structural semiotic approach was intensively used by A.Karapetiantz. Spirin also influenced such Russian researchers as A.Kobzev, A.Krushinsky, M.Isayeva, V.Dorofeeva-Lichtman, etc.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127608\",\"title\":\"Hungarian presidential election, 2000\",\"body\":\"\\nHungarian presidential election, 2000\\n\\nAn indirect presidential election was held in Hungary on 5–6 June 2000. The only candidate was legal scholar Ferenc Mádl, who was nominated by the first cabinet of Viktor Orbán, composing Fidesz, FKGP and MDF. The left-wing (MSZP and SZDSZ) and the far-right (MIÉP) opposition parties did not nominee a candidate. After three rounds, Mádl was elected President of Hungary, taking the office on 4 August in that year. \\nFirst and second rounds.\\nIn the first two rounds, two-thirds majority requirement needed to elect the president, according to the Constitution.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127617\",\"title\":\"Av8er Limited\",\"body\":\"\\nAv8er Limited\\n\\nAv8er Limited () was a British aircraft manufacturer based in Woodford Halse, Northamptonshire and founded by Paul Taylor. The company specialized in the design and manufacture of paramotors in the form of ready-to-fly aircraft for the US FAR 103 Ultralight Vehicles and European rules.\\nThe company seems to have been founded about 2001 and gone out of business in 2011.\\nAv8er produced a range of paramotors noted for their lightness and attention to detail, in particular balancing and vibration isolating features. The models included the Explorer, Orbiter, Observer and Titan.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127634\",\"title\":\"WD-40 Company\",\"body\":\"\\nWD-40 Company\\n\\nWD-40 Company is a San Diego, CA-based manufacturer of household chemicals, including its signature lubricant, WD-40, as well as 3-In-One Oil, Lava, Spot Shot, X-14, Carpet Fresh, and 2000 Flushes. It markets its products in more than 160 countries.\\nHistory.\\nRocket Chemical Company was founded in 1953. In 1969, John S. Barry, on becoming President and CEO, changed the name to \\\"WD-40 Company\\\" after what was then its only product. Barry, who died July 3, 2009, reportedly made the name change on the basis that the Rocket Chemical Company did not make rockets. The company went public in 1973. Its NASDAQ stock symbol is (). \\nProducts.\\nIn addition to its flagship WD-40 product, the WD-40 Company has acquired several household-products companies, adding such brand names as 3-In-One Oil, Lava, Spot Shot, X-14, Carpet Fresh, and 2000 Flushes to its roster.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127669\",\"title\":\"Fenethazine\",\"body\":\"\\nFenethazine\\n\\nFenethazine (INN) (brand names Anergen, Contralergial, Ethysine, Etisine, Lisergan, Lysergan; former developmental code names RP-3015, SC-1627, WY-1143), also known as phenethazinum, is a first-generation antihistamine of the phenothiazine group. Promethazine, and subsequently chlorpromazine, were derived from fenethazine. Fenethazine, in turn, was derived from phenbenzamine.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127671\",\"title\":\"1932–33 Connecticut Aggies men's basketball team\",\"body\":\"\\n1932–33 Connecticut Aggies men's basketball team\\n\\nThe 1932–33 Connecticut Aggies men's basketball team represented Connecticut Agricultural College, now the University of Connecticut, in the 1932–33 collegiate men's basketball season. The Aggies completed the season with a 4–12 overall record. The Aggies were members of the New England Conference, where they ended the season with a 0–4 record. The Aggies played their home games at Hawley Armory in Storrs, Connecticut, and were led by second-year head coach John J. Heldman, Jr..\\nSchedule.\\n!colspan=12 style=\\\"\\\"| Regular Season\\nSchedule Source:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127674\",\"title\":\"Paul Fouracre\",\"body\":\"\\nPaul Fouracre\\n\\nPaul J. Fouracre is professor of medieval history at the University of Manchester. His research interests relate to early medieval history, the history of the Franks, law and custom in medieval societies, charters, hagiography, serf-lord relations in the eleventh century, and the cost of the liturgy.\\nFouracre was co-ordinating editor of \\\"Early Medieval Europe\\\" from 2005 to 2009 and editor of the first volume of \\\"The New Cambridge Medieval History\\\" (2005). He is editor of the \\\"Bulletin of the John Rylands Library\\\". He is a Member of the Chetham Society, serving as a Member of Council since 2004 and as President since 2005.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127710\",\"title\":\"Hooton Tennis Club\",\"body\":\"\\nHooton Tennis Club\\n\\nHooton Tennis Club is a four-piece indie-rock band from the Wirral consisting of Ryan Murphy (vocals/guitar), James Madden (vocals/guitar), Callum McFadden (bass) and Harry Chalmers (drums) formed in 2013. They are signed to Heavenly Recordings and have released one album and a number of singles.\\nBiography.\\nThe band members first met in secondary school, although Madden and McFadden had known each other prior to that at primary school. Towards the end of 2013 the four began to write songs together. In January they were featured on the BBC Introducing in Merseyside, after which they were picked up by the not-for-profit The Label Recordings based at Edge Hill University. They recorded a four-track EP \\\"I Was a Punk in Europe (But My Mum Didn’t Mind)\\\", which was described as \\\"\\\"'trash-indie psychedelic'. In September 2014 it was announced that the band had signed with Heavenly Recordings. In February 2015 they released their first single \\\"Jasper/Standing Knees\\\". It was followed by second single, \\\"Kathleen Sat On The Arm Of Her Favourite Chair\\\". Their debut album \\\"Highest Point In Cliff Town\\\" was produced by Bill Ryder-Jones. The name of the band was taken from a tennis courts in Little Sutton. The band have played headlining shows around the UK and Europe.\\nDiscography.\\nAlbums.\\n\\\"Highest Point In Cliff Town\\\" (2015, Heavenly Recordings)\\nEPs.\\n\\\"Long-Barrelled Saturday\\\" (2013)\\n\\\"I Was A Punk In Europe (But My Mum Didn’t Mind)\\\" (2013)\\n\\\"Oh Phantom, Please Don't!\\\" (2014 as \\\"Hootin' Terrors Klub\\\")\\nSingles.\\n\\\"Jasper/Standing Knees\\\" (2015)\\n\\\"Kathleen Sat On The Arm Of Her Favourite Chair\\\" (2015)\\n\\\"P.O.W.E.R.F.U.L. P.I.E.R.R.E.\\\" (2015)\\n\\\"Barstool Blues\\\" (by Neil Young) (split with The Wytches) (2015)\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127715\",\"title\":\"St. Lucie Canal (C-44)\",\"body\":\"\\nSt. Lucie Canal (C-44)\\n\\nThe St. Lucie Canal (C-44) is a man-made canal built in 1916 in Martin County, Florida to divert floodwaters from Lake Okeechobee via the canal to the South Fork of the St. Lucie River and into the St. Lucie Estuary, a component of the Indian River Lagoon, which connects to the Atlantic Ocean. Resulting from this connection, restoration projects in the St. Lucie River are the northernmost component of the Comprehensive Everglades Restoration Plan.\\nThe C-44 has been a source of contention since its construction in 1916.\\nThere are three major structures along the C-44: the S-308 lock and dam structure; the S-153 structure; and S-80 lock and dam structure. The St. Lucie Canal (C-44) can be referred to as the St. Lucie Canal, C-44, C-44 Canal, or the Okeechobee Waterway. The C-44 has a drainage basin of 185 square miles.\\nHistory.\\nConstruction began on the St. Lucie Canal (C-44) in 1916 and was completed in 1924. The original purpose of the canal was to allow for an improved way to divert floodwaters from Lake Okeechobee. The canal was originally designed to flow into Manatee Pocket instead of the South Fork of the St. Lucie River.\\nRecords indicate that people have been complaining about the impacts of the C-44 since the early 1950s.\\nDescription.\\nThe St. Lucie Canal (C-44) connects to Lake Okeechobee at Port Mayaca. The capacity for the St. Lucie Canal (C-44) is 8 feet and 9,000 cubic feet per second; the result of a the St. Lucie Canal (C-44) second deepening in 1949. In 1933, 16 fixed spillways were approved for construction to reduce shoaling.\\nMajor Structures.\\nS-308.\\nThe S-308 lock and dam structure is located at the western end of the St. Lucie Canal (C-44) and connects to the shore of Lake Okeechobee.\\nS-153.\\nThe S-153 is designed to discharge into the St. Lucie Canal (C-44). The S-153 regulates water levels in the Levee 65 Borrow Canal The Levee 65 Borrow Canal is located on the edge of Lake Okeechobee and north of the St. Lucie Canal (C-44).\\nS-80.\\nThe S-80 structure is a lock and dam structure located at the eastern end of the St. Lucie Canal (C-44). This structure was completed in 1944.\\nWater Flows.\\nAccording to the Florida Department of Environmental Protection, the St. Lucie Canal (C-44) flows both east to the St. Lucie Estuary and west to Lake Okeechobee \\\"on about an equal basis.\\\"\\nIn 1924, the original flow capacity was 5,000 cubic feet per second. \\nIn 1937, the canal was deepened to 6 feet to provide a navigable passage to Lake Okeechobee, and the flow capacity is unknown to this author.\\nIn 1949, the canal was deepened to 8 feet, which increased the flow capacity to 9,000 cubic feet per second.\\nEnvironment.\\nOne of the major problems resulting from C-44 is too much freshwater discharged from C-44 to the St. Lucie Estuary to control the water levels of Lake Okeechobee.\\nEstuary Salinity.\\nLarge discharges from Lake Okeechobee into C-44 cause salinity levels to drop rapidly, killing many species in the estuaries.\\nTurbidity.\\nHigh flow rates result in erosion and the transport of sediment into the estuary that can smother benthic habitats. The increased turbidity also results in sediment filling navigation channels.\\nC-44 Basin.\\nThe C-44 Basin is 117,000 acres. Drainage from these acres into the St. Lucie Canal also create water quality problems for the St. Lucie Estuary.\\nNavigation.\\nIn 1937, the St. Lucie Canal (C-44) was deepened to 6 feet.\\nThe St. Lucie Canal (C-44) connects to the Caloosahatchee Waterway, which connects Lake Okeechobee to the Gulf of Mexico at Fort Myers, Florida.\\nTimeline.\\n1916: Construction begins on St. Lucie Canal (C-44).\\n1924: Original construction is completed, providing a flow capacity of 5,000 cubic feet per second.\\n1937: the St. Lucie Canal (C-44) was deepened to 6 feet to allow for vessel navigation to and from Lake Okeechobee. 1949: the canal was deepened to 8 feet, which increased the flow capacity to 9,000 cubic feet per second.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127775\",\"title\":\"Joseph Kenneth Ssebaggala\",\"body\":\"\\nJoseph Kenneth Ssebaggala\\n\\nJoseph Kenneth Ssebaggala \\\"Joseph S KEN\\\" (born 1983) is a Ugandan film director, screenwriter. In 2008 he started taking film training and workshops, like the Durban Talent Campus 2011. As a writer, producer and director he runs a film company called Zenken Films, under which he has produced two of his feature films: Master on Duty and That Small Piece. He is the recent winner of Best Director in the 2015 Uganda Film Festival with his House Arrest. He's is yet to direct the first Ugandan erotic film Not Just a Boy in which he is rumored to cast screenwriter-cum-actor Usama Mukwaya.\\nHis 2015 Call 112 and House arrest were nominated in the 2016 AMVCA for best East African film, Best Lighting, and overall film of the year making him the most nominated Uganda so far in the Awards.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127784\",\"title\":\"Gallot\",\"body\":\"\\nGallot\\n\\nGallot is a surname. Notable people with the surname include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127802\",\"title\":\"Charles F. Orthwein\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127821\",\"title\":\"1933–34 Connecticut Aggies men's basketball team\",\"body\":\"\\n1933–34 Connecticut Aggies men's basketball team\\n\\nThe 1933–34 Connecticut Aggies men's basketball team represented Connecticut State College, now the University of Connecticut, in the 1933–34 collegiate men's basketball season. The Aggies completed the season with a 5–10 overall record. The Aggies were members of the New England Conference, where they ended the season with a 1–2 record. The Aggies played their home games at Hawley Armory in Storrs, Connecticut, and were led by third-year head coach John J. Heldman, Jr.\\nSchedule.\\n!colspan=12 style=\\\"\\\"| Regular Season\\nSchedule Source:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127861\",\"title\":\"Bakhtiyar Duyshobekov\",\"body\":\"\\nBakhtiyar Duyshobekov\\n\\nBakhtiyar Duyshobekov (born 3 June 1995) is a Kyrgyzstani footballer who plays as a Midfielder for Abdysh-Ata Kant and Kyrgyzstan.\\nCareer Statistics.\\nInternational.\\n\\\"Statistics accurate as of match played 17 November 2015\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127862\",\"title\":\"Ladmirault\",\"body\":\"\\nLadmirault\\n\\nLadmirault is a surname. Notable people with the surname include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127906\",\"title\":\"Longspring Wood\",\"body\":\"\\nLongspring Wood\\n\\nLongspring Wood is a 1.2 hectare nature reserve managed by the Herts and Middlesex Wildlife Trust in Kings Langley in Hertfordshire.\\nThe main trees in this small wood are oak, ash, wild cherry and hazel, and there is a display of bluebells in the spring. Birds include warblers, finches and tits, and there are mammals such as foxes and badgers.\\nThere is access by a footpath between houses in Toms Lane.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127927\",\"title\":\"Raycen Raines\",\"body\":\"\\nRaycen Raines\\n\\nRaycen AmericanHorse Raines: Navy Veteran, Entrepreneur and Businessman. Enrolled member of the Oglala Sioux Tribe and advocate for local grass roots Tribal and community economic development. \\nProfessional Background\\nAfter serving in the US Navy as a young man, Raines has settled into his grandfather’s footsteps as a Native businessman and advocate for Native American social issues. Raines has been in the financial and insurance industry since 2000 and has been working towards economic development specific to Indian Country since 2005.\\nEarly in his career Raines focused on personal finance for individual Native Americans. Specifically, on programs which focus on Native American health care, insurance, retirement, and long term care with Retirement Protection Strategies, Inc. The company provided financial planning for in Native American communities, with the goal of bringing self-reliance to Native American communities through a combination of traditional strategies and modern business concepts.\\nUnder the mentorship of many well-known leaders in Tribal economic development and finance, such as financier Valerie Red Horse and attorney Dennis Ickes, Raines transitioned away from personal finance to tribal finance and economic development. Developing expertise in a variety of tribal economic development vehicles such as Tribal Section 17 corporations, Tribal Economic Development Bonds, and federal 8(a) contracting preferences.\\nRaines served a Project Director for Nations Business Group, a business development firm located in Washington, DC lead then by tribal business leader Pete Homer. Raines was an advocate of the firm’s regional development model that worked with local tribal businesses to identify new revenue streams, and to obtain federal contracts though the SBA 8(a) program, backed with partnerships from major corporations.\\nIn 2012 Raines decided to take his unique set of Tribal economic development skills back to his father’s reservation, the Pine Ridge\\nIndian Reservation, and work more directly on tribal community economic development. Raines lives in the Wakpamni Lake Community on the Pine Ridge Indian Reservation and serves as an economic development consultant to the local tribal municipal government.\\nPersonal Background \\nBorn on October 23, 1975 in Salem, Oregon and raised primarily in Juneau, Alaska where his mother worked for the Indian Health Service. All four of Raines’s grandparents were teachers at the Chemewa Indian School in Oregon, where his parents subsequently met. Raines’s mother, Elizabeth Coburn, was born to Blackfeet (Margaret Pepion) and Klamath (Joseph Francis Coburn) parents, and is now with the Bureau of Indian Affairs (BIA). Raines’s father, William Clifford Ballard, was born to Oglala Sioux (Evelyn Clifford) and Cherokee (John K. Ballard) parents, and currently with the Indian Health Service (IHS).\\nRaines was greatly influenced by his grandparents and elders who served as tribal chairmen, council members, and Native entrepreneurs and business leaders. His grandmother’s sister, Eloise Cobell, (of the $3.4 billion Cobell settlement) helped encourage and guide him towards his efforts in economic development. His maternal grandfather Joseph Coburn was a Klamath tribal chairman and helped lead the restoration of the Klamath tribe.\\nRaines was predominately raised by his mother, step-father Chris Rummell, and maternal grandparents. A strained relationship with his birth father eventually led to Raines changing his surname. Raines’ is also a member of the Blue Bird family of the Wakpamni Lake Community.\\nCriticism.\\nRaines was under scrutiny when he began pursuing online payday lending as a means of bringing in more money for the Pine Ridge Reservation. However, many members of the Wakpamni Native American community feel the negative attention towards Raines is a fear campaign, because the tribe is attempting new economic strides. While a lawsuit was filed and an initial judgment was awarded to the Black Hills Sioux Nation Treaty Council on January 18, 2015, the judgment and restraining order were subsequently reversed and dropped on June 28, 2015 and formal apologies were issued to Raycen Raines and his partners (Docket # CIV-11-0359K of the Oglala Sioux Tribal Court). \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127930\",\"title\":\"1934–35 Connecticut State Huskies men's basketball team\",\"body\":\"\\n1934–35 Connecticut State Huskies men's basketball team\\n\\nThe 1934–35 Connecticut State Huskies men's basketball team represented Connecticut State College, now the University of Connecticut, in the 1934–35 collegiate men's basketball season. This was the first year that the team was named the Huskies. The Huskies completed the season with a 7–8 overall record. The Huskies were members of the New England Conference, where they ended the season with a 1–2 record. The Huskies played their home games at Hawley Armory in Storrs, Connecticut, and were led by fourth-year head coach John J. Heldman, Jr..\\nSchedule.\\n!colspan=12 style=\\\"\\\"| Regular Season\\nSchedule Source:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127932\",\"title\":\"Them Boys\",\"body\":\"\\nThem Boys\\n\\n\\\"Them Boys\\\" is a song co-written and recorded by American country rock artist Brantley Gilbert. It was released in January 2011 as the second single from his second studio album \\\"Halfway to Heaven\\\". Gilbert wrote the song with Colt Ford and Mike Dekle.\\nContent.\\nThe song is a country rock song about observing a younger generation not understanding life. Towards the end, the narrator realizes his grandparents would've also said the same thing.\\nCritical reception.\\nIn her review of the album, Sarah Wyland of \\\"CountryMusicOnline.net\\\" gave the song a positive review, praising the lyrics and writing that \\\"“Them Boys” is another co-write with Dekle and Ford about 'those boys' that ride up and down Main Street with their music turned up and not a care in the world. It is a more mainstream sounding track that ends with 'them boys' reminiscing about days gone by while sitting around the wood stove in the hardware store. The album is dedicated to Brantley's grandfather and it's worth wondering if “Them Boys” is, at least in some way, about him.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127941\",\"title\":\"2016 BMW PGA Championship\",\"body\":\"\\n2016 BMW PGA Championship\\n\\nThe 2016 BMW PGA Championship will be the 62nd edition of the BMW PGA Championship, an annual golf tournament on the European Tour, held 26–29 May at the West Course of Wentworth Club in Virginia Water, Surrey, England, a suburb southwest of London.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127945\",\"title\":\"Reversion (2015 film)\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127960\",\"title\":\"2014 FA Women's Cup Final\",\"body\":\"\\n2014 FA Women's Cup Final\\n\\nThe 2014 FA Women's Cup Final was the 44th final of the FA Women's Cup, England's primary cup competition for women's football teams. The showpiece event was the 21st to be played directly under the auspices of the Football Association (FA). The final was contested between Arsenal Ladies and Everton Ladies on 1 June 2014 at in Milton Keynes. Holders Arsenal made its 14th final appearance, after winning the 2013 final. Everton was appearing in its fifth final.\\nAs FA WSL 1 clubs, both teams entered the competition at the fifth round stage. Arsenal beat Gillingham (2–0), Birmingham City (2–1) and Chelsea (5–3) to reach the final. Everton defeated Cardiff City (3–1), Liverpool (2–0) and Notts County (2–1). \\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127961\",\"title\":\"Norway (song)\",\"body\":\"\\nNorway (song)\\n\\n\\\"Norway\\\" is a song by American dream pop band Beach House, from their third studio album, \\\"Teen Dream\\\". The song was released on January 18, 2010, with \\\"Baby\\\" as its B-side. The song was released as a free download on the band's site on November 17, 2009, before its commercial release.\\nReception.\\n\\\"Norway\\\" received very positive reviews from contemporary music critics. The song was chosen upon release as Pitchfork Media's \\\"Best New Track\\\". Aaron Leitko stated that, \\\"'Norway', the lead track from \\\"Teen Dream\\\", the duo's Sub Pop debut, raises the temperature a few degrees. A percussive intro yields to an explosion of twinkling guitars and a chorus of woozy backing vocals. The core elements of Beach House's sound-- the drum machine, the thrift store keyboards-- are still present; they're just a few ticks faster. This makes a big difference. As it turns out, Beach House goes from dour to exuberant in just a few BPM.\\\" Leitko continues by saying, \\\"Legrand, whose vocals have been saddled with Nico comparisons, can finally breathe a sigh of relief, too. The 1960s chanteuse's shadow is nowhere to be found here. 'You let us in the wooden house/ To share in all the wealth,' sings Legrand over a carsick slide guitar riff. No, 'Norway' is radiant with the sunshiny 70s pop vibes. It's Stevie Nicks territory, for sure. Climate change has come to Beach House, and the weather suits them beautifully.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48127978\",\"title\":\"Golden Lotus Award for Best Supporting Actress\",\"body\":\"\\nGolden Lotus Award for Best Supporting Actress\\n\\nGolden Lotus Award for Best Supporting Actress () is the main category of Competition of Golden Lotus Awards. Awarding to supporting actress(es) who have outstanding performance in motion pictures.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128004\",\"title\":\"Irving Bibo\",\"body\":\"\\nIrving Bibo\\n\\nIrving Bibo (August 22, 1889 – May 2, 1962) was an American composer and songwriter. He wrote tunes for the Ziegfeld Follies (among them, \\\"Huggable, Kissable You\\\", \\\"Forever and a Day\\\" and \\\"Cherie\\\"), Greenwich Village Follies and other theatrical productions in the 1920s, scores for more than 300 motion pictures, and college songs.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128005\",\"title\":\"1957 U.S. National Championships – Women's Singles\",\"body\":\"\\n1957 U.S. National Championships – Women's Singles\\n\\nFirst-seeded Althea Gibson defeated second-seeded Louise Brough 6–3, 6–2 in the final to win the Women's Singles tennis title at the 1957 U.S. National Championships.\\nSeeds.\\nThe seeded players are listed below. Althea Gibson is the champion; others show in brackets the round in which they were eliminated.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128008\",\"title\":\"Toll of the Desert\",\"body\":\"\\nToll of the Desert\\n\\nThe Toll of the Desert is an American Western film starring Fred Kohler, Jr.\\nPlot.\\nWhen a cowboy's family is killed by Indians, he becomes an outlaw and hired gunman. Unbeknownst to him, his son survives the Indian attack and grows up to become a lawman who eventually has to hunt down his outlaw father.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128021\",\"title\":\"1606 in Sweden\",\"body\":\"\\n1606 in Sweden\\n\\nEvents from the year 1606 in Sweden\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128036\",\"title\":\"I-70 Killer\",\"body\":\"\\nI-70 Killer\\n\\nThe I-70 killer is an unidentified American serial killer who is known to have killed six store clerks in the Midwest in the spring of 1992. His nickname derives from the fact that several of the stores in which his victims worked were located a few miles off of Interstate 70 (I-70).\\nHis victims were usually young, petite, brunette women. One of his victims was a man but he is believed to have mistaken the man for a woman as he often wore a ponytail. All of the stores attacked were speciality stores and were usually only robbed of a few hundred dollars. He is also suspected of shooting 3 more store clerks in Texas during 1993 and 1994, one of whom survived.\\nDespite the case being featured on Unsolved Mysteries and Dark Minds, the killer has yet to have been identified and investigators have not publicly identified any suspects.\\n1992 murder spree.\\nThe killing spree began on April 8, 1992 with the murder of 26-year-old Payless ShoeSource manager Robin Fuldauer in Indianapolis. She was alone at the time of being shot and was murdered sometime between 1:30 and 2:00 p.m.\\nThe next two murders occurred on April 11 at the La Bride d’Elegance bridal shop in Wichita. The victims were Patricia Smith, 23 and the store's owner, 32-year-old Patricia Majors. As this was the only case involving multiple victims, investigators believe the killer thought there was only one woman in the store. The women had stayed past normal closing time of 6 p.m. to allow a male customer to pick up a cummerbund. Sometime after 6 p.m., the women allowed the killer into the store, thinking he was the customer. After the women were murdered, the actual customer arrived to pick up the cummerbund and came to face-to-face with the I-70 killer. The I-70 killer let the man go,\\nwho immediately notified police once the killer left. He would later provide details for a composite sketch.\\nOn April 27, Michael McCown, 40, was killed in his mother Sylvia's ceramics store in Terre Haute, Indiana. McCown was the only man killed during spree and it is believed by investigators that the I-70 killer mistook him for a woman because of the store's name and because McCown wore a ponytail\\nOn May 4, 24-year-old Nancy Kitzmiller was killed while working alone at Boot Village, a footwear shop in St. Charles, Missouri. She opened up the shop at noon and was found dead by customers at 2:30 p.m.\\nThe final confirmed murder occurred on May 7 in Raytown, Missouri. The victim was 37-year-old Sarah Blessing who was working in her gift shop, Store of Many Colors. The murder occurred during the day, and the owner of video store next to the Kitzmiller's shop saw the killer enter the shop, heard a pop, and saw the killer leave. He then discovered Blessing's body after checking to see what had occurred in the store. A clerk at a nearby grocery store also saw the killer and saw him climbing a hill towards I-70 after the murder.\\nPossible murders in Texas.\\nInvestigators believe the I-70 killer may be responsible for two murders in 1993, and an attempted murder in 1994, all of which occurred in Texas. The two murder victims were 51-year-old Mary Ann Glasscock, who was killed on September 25, 1993 in Fort Worth at the Emporium Antiques store, and 22-year-old Amy Vess, who was shot dead in a dance apparel store in Arlington on November 1.\\nThe surviving victim was Vicki Webb, 35, who was shot on January 15, 1994 in Houston at the Alternatives gift shop. She briefly talked to the shooter before he shot her in the back of the head. The bullet did not penetrate into Webb's head due to a large vertebra being hit. The shooter attempted to shoot her again, but his gun misfired, and left presuming Webb to be dead.\\nThe modus operandi of the Texas killer was very similar to the I-70 killer and used a .22-caliber firearm, the same caliber as the I-70 killer. However, ballistics test determined that gun used in the Texas murders was not the same as the one used in the I-70 killings, so investigators have not been able to confirm that the I-70 killer was responsible for the shootings in Texas.\\nInvestigation.\\nThe murders were conclusively linked after a St. Charles detective suspected a connection. All of the murders were committed with a .22-caliber firearm and the victims were usually petite, young women with long dark hair. Aside from the Wichita murders, all the victims were alone while murdered and shot in the back of the head. None of the scenes had any signs of sexual assault and while all stores were robbed, robbery appeared to be a secondary motive as all the stores were small speciality stores, which would not have had as much money as larger stores.\\nBased on witness testimonies, police strongly believe the murder weapon may have been an Intratec Scorpion pistol or an Erma Werke ET22 pistol, however they have not been able to rule out any other .22-caliber firearm models. The ammunition used in the killings was .22-caliber CCI copper-clad lead bullets. The cartridges of the bullets had been polished with jeweler's rouge.\\nMidwest authorities linked the killer to the shootings in Texas in 1994, but Texas authorities were not convinced of a connection as different guns were used in each spree.\\nBased on witness descriptions, investigators were able to produce to two composite sketches of the killer and a physical description of the suspect. The I-70 killer was described as being a white man in his twenties or thirties, 5'7\\\" (1.70 m) to 5'9\\\" (1.75 m) tall, thin and having lazy eyelids and sandy blond or reddish hair in 1992. If he still alive, he would be in his fifties or late forties.\\nPolice have not publicly identified any suspects and the case has been classified as a cold case.\\nPopular culture.\\nThe case has been featured on \\\"Unsolved Mysteries\\\" and Investigation Discovery's \\\"Dark Minds\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128090\",\"title\":\"Ronen Shilo\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128094\",\"title\":\"Brent Bailey\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128109\",\"title\":\"1958 U.S. National Championships – Women's Singles\",\"body\":\"\\n1958 U.S. National Championships – Women's Singles\\n\\nFirst-seeded Althea Gibson defeated Darlene Hard 3–6, 6–1, 6–2 in the final to win the Women's Singles tennis title at the 1958 U.S. National Championships.\\nSeeds.\\nThe seeded players are listed below. Althea Gibson is the champion; others show in brackets the round in which they were eliminated.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128110\",\"title\":\"Kainton, South Australia\",\"body\":\"\\nKainton, South Australia\\n\\nKainton is a small town situated about 10 km south of Paskeville in the upper Yorke Peninsula.\\nThe Kainton school closed in 1907, but the \\\"Hundred of Clinton School\\\" was renamed \\\"Kainton School\\\" in 1915.\\nKainton is located within the federal Division of Grey, the state electoral district of Goyder and the local government area known as the Yorke Peninsula Council.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128127\",\"title\":\"Jay Karas\",\"body\":\"\\nJay Karas\\n\\nJay Karas is American film director and television director. His resume primarily consists of directing live telecasts and stand-up comedy specials. In recent years he moved on to directing episodic television, directing episodes of \\\"Parks and Recreation\\\", \\\"Raising Hope\\\", \\\"Awkward\\\", \\\"The Fosters\\\", \\\"Brooklyn Nine-Nine\\\" and \\\"Workaholics\\\".\\nIn 2014, Karas made his feature film directing debut with the film \\\"Break Point\\\", starring Jeremy Sisto and David Walton.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128138\",\"title\":\"Sergey Nikolaev (cyclist)\",\"body\":\"\\nSergey Nikolaev (cyclist)\\n\\nSergey Nikolaev (born February 5, 1988 in Moscow) is a Russian cyclist riding for Itera-Katusha.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128226\",\"title\":\"Daniella Smith\",\"body\":\"\\nDaniella Smith\\n\\nDaniella Smith (born July 30, 1972, Kaikohe, New Zealand) is a New Zealand former professional boxer, the first International Boxing Federation Women's World Champion and is the second New Zealand born person to win a boxing world title. Smith is also ranked pound-for-pound number one in New Zealand of all time in the Women devision.\\nIn 1999, Smith started her career as an amateur. In a space of six years she has fought in forty fights, winning multiple titles including the gold at the national championships four times. She also competed in 2002 World Championships in Turkey, even though she was not successful on winning a medal, she still won a gold medal in 2004 at the Oceania Games. In 2005 Smith made her pro debut against World Kicking Champion Sue Glassey. In 2006, Smith won against Glassey for the second time, but this time to capture her first pro boxing title (vacant NZPBA Women's light middleweight title). In 2010, Smith fought for the first time as a pro outside New Zealand, winning against Jennifer Retzke in Germany and becoming the first International Boxing Federation Women's World Champion. Smith defended her title against Noni Tenge in South Africa seven months after winning the title, but lost the bout. In 2013 Smith attempted to capture her second world title against Arlene Blencowe for the vacant Women's International Boxing Association World super lightweight title and the vacant World Boxing Federation female welterweight title, but Smith was unsuccessful in capturing the titles. In 2014, Smith fought her last fight against Gentiane Lupi. This is the second time that Smith has fought Lupi, but this time for the vacant NZPBA women's lightweight title, but Smith was unsuccessful. These days Smith co-owns a boxing gym called Boxing Alley. Smith also trains corporate and amateur boxers.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128230\",\"title\":\"Villa Madero\",\"body\":\"\\nVilla Madero\\n\\nVilla Madero is a town in the Madero Municipality, in the State of Michoacán, Mexico. Established in 1868 with the name of Cruz De Caminos. According to tradition the first inhabitant of the place was a man named Armas, he is now considered the town's founder. The land where the town currently sits belonged to La Hacienda De La Concepcion. The village is the point of intersection of the paths that lead to Acuitzio, Etucuaro, Curuchancio and Tacambaro, this is why it became known as Cruz De Caminos (Cross Roads).\\nCruz De Caminos, was in 1901 part of Acuitzio Del Canje. On July 27, 1914, Cruz De Caminos applied under the Municipality Erection Act. On October 12, 1914, the agreement to raise it to a municipality with the name of Madero was ratified.\\nIn present time, Villa Madero is becoming famous due to a great festival named \\\"La Feria del Mezcal\\\". This is about a traditional beverage \\\"mezcal\\\" produced in Etucuaro, a town that is part of the Madero municipality. During this festival, producers give free samples of their product. There is also a grand vast of traditional food stands, different sports activities, a parade and at night people gather in a ballroom to dance with live music.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128255\",\"title\":\"Myth (song)\",\"body\":\"\\nMyth (song)\\n\\n\\\"Myth\\\" is a song by American dream pop band Beach House, from the band's fourth studio album, \\\"Bloom\\\". The song was released as a single on March 26, 2012. The song surfaced on the band's website on March 7, 2012, before its commercial release.\\nReception.\\n\\\"Myth\\\" received very positive reviews from contemporary music critics. The song was chosen upon release as Pitchfork Media's \\\"Best New Track\\\". Larry Fitzmaurice stated that, \\\"On a surface level, there's no mistaking \\\"Myth\\\" for a Beach House song. All the sonic elements that have travelled with the Baltimore dream-pop duo during their steady ascent over the last five years are intact: Alex Scally's narcotic guitar, a steady backbeat, and Victoria Legrand's smoky ache of a voice. What sets \\\"Myth\\\" as another sonic evolution for Beach House, then, is all in the details.\\\" Fitzmaurice continues by saying, \\\"The layers of echo surrounding Legrand's voice during its chorus, as well as the gauzy glow wrapped around everything, give the impression that the airy expansiveness of 2010's Teen Dream has contracted, but somehow the sound is just as \\\"big\\\", if not bigger, than before. \\\"What comes after this momentary bliss?\\\" Legrand croons at one point. Hopefully, we'll find out soon enough.\\\"\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128284\",\"title\":\"The Escape (1939 film)\",\"body\":\"\\nThe Escape (1939 film)\\n\\nThe Escape is a 1939 American action film directed by Ricardo Cortez and written by Robert Ellis and Helen Logan. The film stars Kane Richmond, Amanda Duff, June Gale, Edward Norris, Henry Armetta and Frank Reicher. The film was released on October 6, 1939, by 20th Century Fox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128298\",\"title\":\"David Bain (disambiguation)\",\"body\":\"\\nDavid Bain (disambiguation)\\n\\nDavid Bain is a New Zealand man who was acquitted of murder in 2009. The name may also refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128318\",\"title\":\"Moyer's Landfill\",\"body\":\"\\nMoyer's Landfill\\n\\nMoyer's Landfill was a privately owned landfill in Collegeville, Pennsylvania, United States. It was originally farmland outside the town. In the 1940s the owner started accepting trash and municipal waste as a way to make additional money. The original landfill was 39 acres and did not have a liner to protect the land from contaminate. A liner was added to a new section in the late 1970s. Over time, the landfill accepted sewage, and industrial wastes which contained hazardous substances in addition to municipal waste. The site was closed by the EPA in 1981, and was one of the first \\\"Superfund\\\" sites added to the National Priorities List.\\nHistory.\\nThe Site had been operating as a municipal landfill from the early 1940s until April 1981. The landfill accepted municipal waste, sewage, and a variety of solid and liquid hazardous wastes (e.g. polychlorinated biphenyls (PCBs), products containing dioxin, paint solvents, and similar material). The original landfill was a section of the farm with no liner to prevent material from seeping into the soil. In the original landfill any waste was simply dumped and covered with dirt.\\nIn 1972 when the Pennsylvania Department of Environmental Protection () rules became more restrictive, the site was cited for violations. To achieve compliance, pipes were added to drain leachate to two earthen basins which were pumped into trucks for treatment and disposal. Due to leakage from the basins, the system was modified to pump all the leachate to a concrete basin where it was sprayed over the top of the landfill.\\nIn the late 1970s, the landfill owners submitted a request to enlarge the landfill which was granted. Work started to enlarge the landfill in 1977, and included an asphalt liner to protect the soil from hazardous materials. Dumping was reportedly limited to this new area from 1977 to early 1981.\\nBefore remediation, contaminants were spread by rain water soaking into the landfill, some of which may have reached the ground water. Run-off from the landfill also carried contaminates to local streams (Skippack Creek and Perkiomen Creek).\\nLocal residents tried for years to get the landfill closed. In the late 1970s 450 families united to try to close what they called \\\"Mount Trashmore\\\". They packed the local zoning board meetings, wrote letters to several environmental protection agencies, and had lawn signs printed up to close the dump.\\nIn the lawsuit \\\"O'LEARY v. Moyer's Landfill, Inc., 523 F. Supp. 642 (E.D. Pa. 1981)\\\" it states \\\"this suit was brought in part because DER has, in plaintiffs' view, been ineffective in alleviating the dangers plaintiffs perceive at the landfill\\\".\\nThe site was finally closed by the PA DEP in early 1981.\\nExtent of the contamination.\\nAnalysis of the leachate dates back to 1972, this analysis was conducted to design an aerated lagoon to treat the liquid. After being designated a Superfund site, tests found heavy metals and organic chemicals. These chemicals included Benzene, toluene, Trichloroethylene, tetrachloroethylene and chlorobenzene. All of which are associated with industrial solvents. Samples from the landfill had a total of 86 priority pollutants and 16 metals. Water samples were taken from the Skippack and Perkiomen Creeks and chemicals found in the landfill were also found in both streams. A remediation study was funded at a total cost of $681,000\\nThe result of the remediation survey in 1985 an estimate was made that the cost of a cleanup would be $15 million and an additional $340,000 a year for operation and maintenance. By 1994 the price tag has been increased to $45 million for the cleanup.\\nCompletion.\\nOn May 27, 2014 The Environmental Protection Agency (EPA) region III removed Moyer's Landfill from the National Priorities List. The site is no longer considered a Superfund site. The site consists of open land surrounded by trees on steep slopes. The landfill is fenced off and has leachate collection tanks. There are no development plans.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128332\",\"title\":\"Lunch Time Heroes\",\"body\":\"\\nLunch Time Heroes\\n\\nLunch Time Heroes is a 2015 Nigerian family comedy film directed by Seyi Babatope, and starring Dakore Akande, Omoni Oboli, Diana Yekinni, Tina Mba and Tope Tedela.\\nThe film tells the story of Banke (Diana Yekinni), a corp member who has been posted to teach in a high school, and she has to gain the respect and attention of students and faculty that don’t want her around.\\nProduction and release.\\n\\\"Lunch Time Heroes\\\" was shot in Lagos for 16 days. Some of the child actors used in the film were sourced through local churches, and none had a previous professional acting experience. Before the commencement of principal photography, the director, Babatope had got the child actors on film set to play around, in order to desensitize them to the cameras, dollies, cables, lights and the peculiarities of filming environment.\\nOnset photos for \\\"Lunch Time Heroes\\\" was released to the public online in March 2015. Trailer for the film was released in July 2015. It premiered at FilmHouse Cinema, Surulere, Lagos on 23 August 2015; with general theatrical release on 28 August 2015. The theme soundtrack for the film, titled \\\"I Believe\\\" by Capital Femi, was released online on 28 August 2015, along with its music video, which features the casts of the film.\\nCritical reception.\\nThe film has been met with mixed critical reception. Amarachukwu Iwuala of \\\"360Nobs\\\" commends the film's screenplay, character development and soundtrack, but criticizes the film for having several inconsistencies in its storytelling. She concludes: \\\"Much as the story of \\\"Lunch Time Heroes\\\" is foreseeable, owing to the title of the film and the information on the promotional materials, one nevertheless enjoys the journey that leads to the final destination\\\". Yvonne Anoruo praises the soundtrack, but notes the lack originality, and several inconsistencies. She comments: \\\"\\\"Lunch Time Heroes\\\" comes with a very simple plot, and with very predictable twists and turns. It is so focused on staying true to its subject matter and with rather rigid dialogues that it is altogether stiff. At the end of it, one realises that save a few characters, the others are largely flat. There were a lot of improvisations in measures that takes away from the movie. The final product does not elicit any feelings beyond the ordinary and if any, it totters on the edge of basic. Ultimately, for what it lacks everywhere else, the movie makes up for with very adorable child actors\\\". Jite Efemuaye comments: \\\"\\\"Lunch Time Heroes\\\" is a good effort which is undermined by a less than dedicated attention to detail on the director’s part. Even with all its limitations, it is an entertaining movie, one that can be enjoyed by all members of the family, but it is easily forgettable\\\". Yvonne Williams of \\\"Nollywood Observer\\\" comments: \\\"Although \\\"Lunch Time Heroes\\\" wows with its uncommon storyline and brilliant performances from Dakore Akande and Tina Mba, it falls short in its delivery – somewhat un-contagious forced humour, thinness of the plot, overacting amongst other things. However, it no doubt would be a welcome delight for the youth and the young at heart\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128374\",\"title\":\"Borgeaud\",\"body\":\"\\nBorgeaud\\n\\nBorgeaud is a surname. Notable people with the surname include:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128380\",\"title\":\"Maelifell\",\"body\":\"\\nMaelifell\\n\\nMount Mælifell, also called as Measure-Hill, is located in Southern Iceland and is about 800 meters tall.\\nMount Mælifell is reachable only by a four-wheeled vehicle and is on the southern Fjallabak road next to the glacier Mýrdalsjökull, the road is usually wet and sometimes completely flooded.\\nIt is listed as one of the amazing places on the earth by \\\"Amazing places on earth\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128387\",\"title\":\"Svetislav Tisa Milosavljević\",\"body\":\"\\nSvetislav Tisa Milosavljević\\n\\nBan Svetislav Tisa Milosavljević, or Бан Светислав Тиса Милосављевић, (Born in Niš on 7. 9. 1882 - Died in Belgrade on 28. 7. 1960) was a military architect. He was the eldest son of a wealthy Niš dealer Tome Milosavljevic and his wife Basil Popović. \\nMilitary career.\\nSvetislav planned to become an engineer, but his father's financial collapse forced him into the military profession. He became an authority on military traffic, while advancing at the end of 1925 to the rank of Brigadier General.\\nBanja Luka.\\nHe came to Banja Luka on 8 November 1929. In a short time with a substantial state financial aid he helped develop the Banovina, and in particular Banja Luka.\\nHis greatest accomplishments include the Banska Palace (now the city administration), Governor's Palace, the Theatre (founded in 1930 and today's building built 1934), Public Health Institute, the facilities and the Teacher's School of Agriculture, the east wing of the then Grammar School, and seven residential buildings for officers (in today's Alley St. Sava and at the Post Office). He helped establish the Banate Museum, Association for Tourism and Craft and the Chamber of Commerce.\\nAlthough not a direct investor, builder or founder, Milosavljevic gained credit for the emergence of the Banja Luka town park with a monument to Peter Kocic, for upgrading the hotel Bosnia, paving and street lighting and construction of Sokolski Dom, City Municipality and the Hotel Palace.\\nAfter becoming viceroy, Milosavljevic only once visited Banja Luka - Ascension Day, May 18, 1939, when he accepted the invitation of the Banja Luka Orthodox parish and came to the cathedral shrine.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128396\",\"title\":\"HD 164595 b\",\"body\":\"\\nHD 164595 b\\n\\nHD 164595 b is a confirmed exoplanet orbiting around the Sun-like star HD 164595 every 40 days some 94.36 light-years away. It has a mass of 16 Earth masses, most likely due to its high mass it could be a Mini-Neptune, however depending on the planet's density it could be a Mega-World that could be like a terrestrial planet or it could be made out of volatile compressed into a solid form.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128399\",\"title\":\"ID Dunedin Fashion Week\",\"body\":\"\\nID Dunedin Fashion Week\\n\\niD Dunedin Fashion Week is an annual festival of fashion held in the New Zealand city of Dunedin, usually in March or April. The festival has been held regularly since its inception in 2000. The 2015 show included 35 events, and attracted over 8000 people from around New Zealand and overseas.\\nFashion Week includes a series of events around Dunedin and its Otago environs, such as tours, talks, and exhibitions, and features two major award shows. The first of these is the iD International Emerging Designer Awards, held in conjunction with Otago Polytechnic, which is a one-night show held at Dunedin Town Hall. Entry is restricted to recent graduates or current final-year students in fashion related courses, and regularly attracts new designers from around the world.\\nThe week culminates in the second major show, the iD Dunedin Fashion Show. This two-night show takes place at Dunedin Railway Station, and the main platform is turned into one of the world's longest catwalks, a little over in length. The show is restricted to current commercial designers, and includes international guest designer selections - including major names, although its primary focus is on local competitive entries.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128419\",\"title\":\"AudioComics Company\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128442\",\"title\":\"Osaka Tōin Junior and Senior High School\",\"body\":\"\\nOsaka Tōin Junior and Senior High School\\n\\n is a private co-educational junior and senior high school located in Daitō City, Osaka Prefecture, Japan. The high school was founded in 1983 by Osaka Sangyo University.\\nHistory.\\nOsaka Tōin Senior High School was founded as the Daitō campus of Osaka Sangyo University Senior High School in 1983, during a period of rapid increase in the number of high school students in Japan. The school commenced with 150 students enrolled in two streams; a regular course and dedicated physical education course. The school became independent from Osaka Sangyo University Senior High School in 1988 and an integrated junior high school was opened in 1995.\\nClub activities.\\nBaseball.\\nThe school's baseball club was formed in 1988. In 1991, only the club's fourth year of existence, Osaka Tōin participated in the spring invitational tournament for the first time. The school advanced to the quarter finals, with the highlight of the tournament being ace Yukihiko Wada pitching a no-hit, shutout in the first round match against Sendai Ikuei High School. In the same year, the school won the \\\"Summer Koshien\\\" national championships, defeating Okinawa Fisheries High School 13-8 in the final. Osaka Tōin was the 14th school to win the chamionship in their first year of qualifying for the tournament. Wada and future Yomiuri Giants member Yoshihiro Seo pitched in the championship match against Okinawa Fisheries ace Rin Ōno; fellow future Yomiuri Giant Makoto Hagiwara hit the only home run of the match.\\nThe school won its second national championship in 2008, defeating Shizuoka Prefecture's Tokoha Kikukgawa Senior High School 17-0 in the final.\\nIn 2012, Osaka Tōin became the seventh school in history to win the spring invitational and summer national tournaments in the same year, defeating Aomori Prefecture's Kōsei Gakuin in the final of both tournaments. It was the first time that two schools met in the final of both tournaments in the same year. Future Hanshin Tigers ace Shintaro Fujinami was the star of the tournament, pitching shutouts in both the semi-final and final. In the autumn of 2012, Osaka Tōin was joint-champion of the National Sports Festival along with Sendai Ikuei, becoming the third school in history to win the \\\"triple crown\\\" in one year. Lost time due to rain earlier in the tournament forced tournament organizers to announce that the winners of the two semi-finals would be named joint champions.\\nAt the 2014 summer tournament, the school won for the fourth time, defeating Mie High School 4-3 in the final.\\nFormer members of the Osaka Toin baseball team have played for eleven of the twelve Japanese professional baseball teams (see list of alumni below). Sho Nakata and Tsuyoshi Nishioka have played on the Japan national team and Nishioka spent two seasons with the Minnesota Twins of the MLB.\\nBrass band.\\nThe brass band club was formed in 2005 and in the same year took the gold prize at the Kansai Brass Band Contest. In 2006 and 2007, the band represented the Kansai region in the high school division of the All-Japan Band Competition, taking the silver award on both occasions. In 2008 the band did not qualify for the national competition, but returned in 2009 and won their first gold award. The band continued this success with consecutive wins at the 2010 and 2011 competitions. In 2008 the band also won silver in its first appearance at the All Japan Marching Contest. It followed this with by winning gold at the 2009 and 2010 events. As a result of this success, the band receives many requests for public performances and is well known throughout Japan despite its short history.\\nSoccer.\\nThe school's soccer club was founded in 2005. It qualified for the soccer tournament of the national high school sports championships in 2007 and reached the quarter finals of the same tournament in 2008. The club also qualified for the 2008 All Japan High School Soccer Tournament and advanced to the second round.\\nThe girls' soccer club was formed in 2006. It has qualified for the national championships several times since its first appearance in 2007; in 2011 the club finished runners-up, losing to Tokiwagi Gakuen High School 1–3 in the final.\\nRugby.\\nThe school rugby club was formed in 1988. In 1995, it qualified for the National High School Rugby Tournament for the first time. In 2013 the club won the 14th Spring Invitational Tournament, its first championship win at the national level, defeating Tokai University Gyosei High School 33-14 in the final. In 2015 the club were runners-up in the spring invitational tournament, this time losing to Tokai Gyosei 0–21 in the final.\\nGolf.\\nIn 1999 the boys' golf team won the 20th annual All Japan High School Golf Tournament.\\nBasketball.\\nThe boys' and girls' basketball teams each made their first appearance at the summer All Japan High School Basketball Tournament in 2014, with the boys' team advancing to the second round and the girls' team advancing to the third round. The girls' team also qualified for the \\\"Winter Cup\\\" tournament for the first time in 2014.\\nTrack and field.\\nThe track and field club was formed in 2011 as the \\\"ekiden\\\" club. The club changed its name to the track and field club in the following year. In November 2013 the club won the Osaka Prefecture ekiden qualification race and finished 22nd in the national championship in the following month.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128445\",\"title\":\"Clube Náutico da Beira\",\"body\":\"\\nClube Náutico da Beira\\n\\nClube Náutico da Beira is a club located in Beira, Mozambique which features a restaurant, tennis court, pool, water-sports, and a boat warehouse.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128462\",\"title\":\"Wildflower (Hank Crawford album)\",\"body\":\"\\nWildflower (Hank Crawford album)\\n\\nWildflower is the fourteenth album led by saxophonist Hank Crawford and his third released on the Kudu label in 1973.\\nReception.\\nAllMusic awarded the album 4 stars stating \\\"\\\"Wildflower\\\" is indispensable as a shining example of '70s groove jazz at its best\\\".\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128468\",\"title\":\"Juan Antonio Suanzes\",\"body\":\"\\nJuan Antonio Suanzes\\n\\nJuan Antonio Suanzes Fernández (20 May 1891 – 6 December 1977) was a Spanish naval engineer. Before the Spanish Civil War (1936–39) he directed a shipyard. During the civil war he offered his services to the Nationalist side, and was made Minister of Industry and Commerce from 1938 to 1939. He was again Minister of Industry and Commerce from 1945 to 1951.\\nEarly years (1891–1936).\\nJuan Antonio Suanzes Fernández was born in Ferrol, Coruña, on 20 May 1891.\\nHe was the oldest of six children in a family with naval traditions.\\nHis father, Saturnino Sunazes Carpegna, belonged to the General Corps of the navy.\\nHe attended a religious school for his early education.\\nAt the age of 12 he entered the naval school in Ferrol.\\nHe was promoted to midshipman (1906), frigate ensign (1908) and navy Ensign (1909).\\nAt times he was assigned to ships such as the \\\"Numancia\\\", \\\"Pelayo\\\", \\\"Carlos V\\\" and \\\"Reina Regente\\\".\\nIn 1913 he was appointed as a lieutenant to the battleship \\\"España\\\".\\nIn 1915 Suanzes began to study naval engineering in Ferrol.\\nIn 1917 he was a captain of naval engineers, and a teacher at the Naval Military School of San Fernando in Cadiz.\\nIn 1920 he was named director of the Cartagena shipyard of the \\\"Sociedad Española de Construcción Naval\\\" (SECN, Spanish Society of Naval Construction).\\nHe was appointed commander of engineers in 1921.\\nIn 1922 he was made a supernumerary of the navy so he could devote himself to his work for the private company.\\nHe was in charge of the SECN shipyard at Cartagena until 1926, then was transferred to run the shipyard in Ferrol. In January 1932 he moved to Madrid with his family.\\nIn Madrid he was Inspector General of Construction for the company. A profound patriot, he became increasingly disturbed at the British ownership position in the SECN, which he felt was trying to prevent it from evolving into an independent Spanish operation.\\nIn 1934 Suanzes left the SECN due to what he called the \\\"intolerable interference of the English\\\", referring to Vickers, one of the SECN proprietors.\\nHe created a small company named \\\"Estudios, Proyectos y Reparaciones\\\" (EPYR), then obtained the position of Director General of Boetticher y Navarro, S.A. (BYNSA).\\nCivil War (1936–39).\\nAt the start of the Spanish Civil War (1936–39) BYNSA was taken over by the Ministry of Defense. In late October 1936 Suanzes took refuge in the Polish embassy. \\nHe left there in March 1937 and traveled via Valencia, Marseille and San Sebastián to Salamanca, where he placed himself at the disposal of General Francisco Franco.\\nSuanzes was named colonel of naval engineers in charge of salvage.\\nIn June 1937 he was in Rome, where he asked Admiral Odoardo Somigli to help refurbish the Spanish naval vessels.\\nHis request was refused at first, but Benito Mussolini intervened and Suanzes was able to get a series of ships overhauled and rearmed by the Italians.\\nFranco appointed him Minister of Industry and Commerce on 31 January 1938.\\nThe ministry was based in Bilbao, and the main task was recovery of industries in the areas that were coming under the control of Franco's forces.\\nOn 9 August 1939 he was replaced in a cabinet reshuffle by Luis Alarcón de Lastra.\\nLater career (1939–77).\\nThe law of 1 September 1939 created the Office of Construction and of Naval Military Industries in the Ministry of the Navy. On 23 September 1939 Suanzes was made head of this organization, and became a member of the National Defense Council. His goal was to end existing contracts with the navy and start a new organization responsible for naval military construction. The Minister of the Navy, Salvador Moreno, delayed all his projects. He resigned in July 1941.\\nThe \\\"Instituto Nacional de Industria\\\" (INI, National Institute of Industry) was created on 25 September 1941, and on 17 October 1941 Suanzes was named president of the institute.\\nSuanzes was appointed Minister of Industry and Commerce on 20 July 1945, while remaining president of the INI.\\nHe held office as Minister until 19 July 1951, when he was replaced in a cabinet shuffle by Joaquín Planell Riera (Industry) and Manuel Arburúa de la Miyar (Commerce).\\nHe remained president of the INI until 1963, when Franco accepted his resignation.\\nAlthough working in the INI, Suanzes remained a naval officer and became a Brigadier General in 1950.\\nIn 1940 he became a member of the \\\"Consejo Superior de Investigaciones Científicas\\\" (CSIC, High Council for Scientific Research). \\nHe became a member of the board of \\\"Juan de la Cierva\\\", and was named president of this company in 1942, holding that position for over twenty years.\\nIn 1956 he was appointed the first president of the \\\"Escuela de Organización Industrial\\\" (EOI, School of Industrial Organization), holding office until 1963.\\nIn 1956 he was appointed first president of the Spanish National Committee of the World Energy Conference, a position he held for the remainder of his life.\\nIn 1960 Franco granted Suanzes and his heirs the title of \\\"Marqués de Suances\\\".\\nIn 1963 he resigned from the INI and withdrew from all public activity.\\nHe died in Madrid on 6 December 1977 at the age of 86.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128518\",\"title\":\"1938 U.S. National Championships – Women's Singles\",\"body\":\"\\n1938 U.S. National Championships – Women's Singles\\n\\nSecond-seeded Alice Marble defeated Nancye Wynne 6–0, 6–3 in the final to win the Women's Singles tennis title at the 1938 U.S. National Championships.\\nSeeds.\\nThe tournament used two lists of eight players for seeding the women's singles event; one for U.S. players and one for foreign players. Alice Marble is the champion; others show in brackets the round in which they were eliminated.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128534\",\"title\":\"Chilla Bulbeck\",\"body\":\"\\nChilla Bulbeck\\n\\nMargaret Chilla Bulbeck (born 1951) was the emeritus professor of women's studies at Adelaide University from 1997 until 2008, and has published widely on issues of gender and difference.\\nEducation.\\nBulbeck gained a degree in economics from the University of Adelaide (1972), a master of arts (1975) and a Ph.D. in Sociology (1980) from the Australian National University, and an LL.B. from the University of Queensland (1991).\\nPolitical career.\\nAfter retiring from academic life, Bulbeck entered politics as a full-time volunteer for The Greens (WA), becoming their Secretary and co-editor of their newsletter. She also ran, unsuccessfully in the Western Australian state election, 2013 for the district of Mandurah.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128542\",\"title\":\"List of 2004 box office number-one films in Mexico\",\"body\":\"\\nList of 2004 box office number-one films in Mexico\\n\\nThis is a list of films which placed number one at the weekend box office for the year 2004.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128552\",\"title\":\"Suanzes\",\"body\":\"\\nSuanzes\\n\\nSuanzes may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128553\",\"title\":\"Bruce Judson\",\"body\":\"\\nBruce Judson\\n\\nBruce Judson (born 1958 in New York City) is an American author, media innovator, and public policy analyst.\\nEducation.\\nJudson received his Bachelor’s degree from Dartmouth College, in Policy Studies in 1980, where he studied with the environmentalist Donella Meadows. In 1984, he received his Juris Doctor from the Yale Law School and his MBA from the Yale School of Management. At the Yale Law School, he was the co-founder and Editor-in-Chief of the \\\"Yale Journal on Regulation\\\" and a Senior Editor of the \\\"Yale Law Journal\\\".\\nCareer.\\nJudson started his career as a consultant and founding member of the New York office of the Boston Consulting Group. In 1989, Judson joined Time Inc. Magazines, where he was appointed the Company’s first corporate Director of Marketing. After the merger of Time Inc. and Warner Communications led to the creation of Time Warner Inc., Judson’s corporate marketing department served as the focal point for Time Warner’s initiative to provide advertisers with advertising programs, involving entities throughout the media conglomerate.\\nLater, with the creation of Time Inc. New Media, Judson was appointed General Manager, where he was one of the co-founders of the Pathfinder (website). Walter Isaacson, then President of Time Inc. New Media, and the subsequent biographer of Steve Jobs, credits Judson with inventing the concept of the Web banner ad, which established the standardized system that enabled the rapid growth of Internet advertising.\\nWhile working at Time Inc. New Media, Judson wrote \\\"NetMarketing\\\", which became a nationwide bestseller, and was excerpted in \\\"Advertising Age\\\". He was named by \\\"Advertising Age\\\" as one of the nation’s \\\"Cybermarketing Leaders.\\\" Judson’s activities at Time Inc. New Media are described in Michael Wolff’s bestselling book, \\\"Burn Rate.\\\"\\nAfter leaving Time Inc., Judson accepted an appointment at the Yale School of Management as a Faculty Fellow, and was later promoted to Senior Faculty Fellow. At Yale, he developed and taught a clinical course offering free consulting to small businesses. Judson also served as one of the founding faculty members of the Yale Publishing Course, and as the first entrepreneur-in-residence at the Yale Entrepreneurial Institute.\\nIn 1999, Scribner published, \\\"HyperWars\\\" which Judson co-authored. The book asserted that significant changes in corporate strategies would be required for success in the coming Internet era. \\\"HyperWars\\\" was selected by Soundview Executive Book Summaries as one of the best business books published in the year of its release.\\nIn 2004, HarperBusiness published Judson’s book, \\\"Go it Alone!\\\" which argued that the combination of software-as-a-service, automation, and outsourcing, enabled by the Internet would fundamentally alter the nature of entrepreneurship and small business success. In the book, Judson also asserted that as a result of automated leverage created by the Internet, small groups of people or individuals, working on their own, would be able to build high revenue businesses. The book was recognized by \\\"Library Journal\\\" as one of the best business books published in the year of its release, while Judson’s ideas on the future of entrepreneurship were the subject of dedicated interviews in business publications, such as \\\"The Wall Street Journal\\\"and \\\"Entrepreneur\\\" magazine.\\nJudson later partnered with HarperBusiness, to test the value of making the full text of \\\"Go It Alone!\\\" available free online, with advertising support. This first-of-its-kind effort in book publishing was featured in a \\\"US News and World Report\\\" cover story.\\nIn 2012, the continuing significance of the ideas in \\\"Go It Alone\\\" was recognized by \\\"Entrepreneur\\\" magazine, which dedicated a feature story to the book, and its continuing popularity, at a time when the \\\"half-life of business books\\\" is short, while the website LifeHack recommended the book, twelve years after its initial publication, in a May 2016 article.\\nWhile involved at Yale and writing books, Judson was also active in developing independent businesses, including Web-Clipping (co-founder), (an early online news clipping service for businesses), the business broadband marketplace Speed Anywhere., and a mobile Web site development firm.\\nJudson's book, \\\"It Could Happen Here\\\", was published in 2009 by HarperCollins. The book argued that growing and extreme economic inequality in the United States was a societal danger. Judson worked with historical and social science research to construct a model which indicated that highly unequal societies are characterized by political polarization, anger, lack of trust, political paralysis, a collapsing middle class and potentially political instability. The book appeared two years before Occupy Wall Street led economic inequality to be considered a mainstream political issue, and at the time the significance of growing economic inequality was often disregarded or seen as unlikely to continue.\\nJudson was subsequently appointed a Braintruster at the Roosevelt Institute, where he launched a column titled \\\"Restoring Capitalism\\\" for the Institute’s website. Articles from the column were syndicated in online media including \\\"The Business Insider\\\" and \\\"The Huffington Post\\\".\\nAs of December 2015, Judson is a Senior Adviser to Tern Plc.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128557\",\"title\":\"Haxhi Ymer Kashari\",\"body\":\"\\nHaxhi Ymer Kashari\\n\\nHaxhi Ymer Kashari known also as Ymer Mustafa Kashari was an Albanian poet of the Bejtexhinj movement of the 18th century.\\nHaxhi Ymer was born in Tirana in early 18th century. Back then part of the Sanjak of Scutari of the Ottoman Empire, Tirana was already flourishing as an oriental-style town. Haxhi Ymer was a sheikh (Sufi leader) of the Qadiri order of Sufism, a less spread order going towards extinction in Albania. Most of his work is lost because of earlier lack of interest in him. His name Haxhi indicates that he had completed his hajj in Mecca. He used the pen name Suzi. The outer facade of the portal of Et'hem Bey Mosque in Tirana has an inscription written by him with his pen-name as signature.\\nHaxhi Ymer was a \\\"bejte\\\" poet who wrote in two languages: Albanian and Turkish. From a few odes that are saved to date, one is of special interest. It is named \\\"Alif\\\" and it is one of a kind due to the specific structure it introduced to Albanian poetry of those times. The poem is based on the letters of the Arabic alphabet and thus has 28 verses, each verse starts with a unique letter in alphabetic order. The first starts with \\\"alif\\\" and the last with \\\"yā’\\\". This type of verse introduced by Haxhi Ymer became a tradition in Albanian poetry and had many followers.\\nThough heavily laden with Oriental vocabulary, his work has linguistic significance due to the specific Tirana area Gheg dialect of the Albanian language (part of the Central Gheg branch), being the oldest written piece in this dialect.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128569\",\"title\":\"Joe Chenelly\",\"body\":\"\\nJoe Chenelly\\n\\nJoseph Randy \\\"Joe\\\" Chenelly (born October 2, 1976) is an American journalist and political adviser. He is now directing the \\\"Warriors for Webb\\\" grassroots movement of the Jim Webb campaign for U.S. president. He is also assistant national director of communications for Disabled American Veterans.\\nChenelly covers military and veterans matters, on staff with the Military Times newspapers and Gannet News, reporting on operations in the Middle East, Southwest Asia, Africa, as well as disaster relief in Louisiana, Mississippi and Texas in the aftermath of Hurricane Katrina.\\nChenelly was named one of the 100 \\\"most influential journalists covering armed violence\\\". He was the first U.S. Marine combat correspondent to step into enemy territory after September 11, 2001, documenting American military action and providing it for broadcast throughout the international media.\\nChenelly was the first military reporter in Pakistan and Afghanistan after the terrorist attacks in the United States. He also reported from the frontlines with American and allied forces in Kuwait and Iraq as that war began in 2003. Chenelly was on the ground for the start of both Operation Enduring Freedom and Operation Iraqi Freedom. After returning to Washington to cover the wars from the policy aspect, he headed back to field reporting, corresponding live from the Louisiana Superdome and flooded streets of New Orleans as a civilian reporter in the immediate aftermath of Hurricane Katrina.\\nAs part of the first conventional U.S. ground force to enter Afghanistan, he was the first to provide combat footage of Operation Enduring Freedom,[2] the first to report from inside a coalition detention facility in Afghanistan[2], the first to report an Iraqi man had given American forces information about where U.S. Army prisoner of war Pfc. Jessica Lynch was being held, the first to report that the other American prisoners of war had been rescued, and he was the first to report FEMA and the National Guard had pulled out of the Louisiana Superdome in New Orleans following Hurricane Katrina.\\nChenelly is now assistant national director of communications for the Disabled American Veterans (DAV) in Washington, D.C. On November 6, 2012, he was elected to a four-year term on the Calvert County Board of Education (District 1).\\nChenelly was named Calvert County (MD) Man of the Year by the county's Republican Central Committee in May 2013.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128615\",\"title\":\"Germania Flugzeugwerke\",\"body\":\"\\nGermania Flugzeugwerke\\n\\nGermania Flugzeugwerke GmbH was a German aircraft manufacturer during World War I. The company was established in 1914 at Leipzig. During 1915 and 1916, the company produced license built Rumpler C.I reconnaissance biplanes for the Luftstreitkräfte at Leipzig-Mockau Airport. The company also repaired other types of aircraft and maintained their own flight school to train pilots. The following types of 1919 were not yet reichsluftamt with an approval for the civil aviation: DFW C V, Ru. C I a, Germ. C IV. There were 17 aircraft of the Germania Reichsluftamt aircraft works when approved.\\nThe company was liquidated in 1922 after the Treaty of Versailles.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128616\",\"title\":\"Pedomicrobium manganicum\",\"body\":\"\\nPedomicrobium manganicum\\n\\nPedomicrobium manganicum is a bacterium from the genus of Pedomicrobium which was isolated from quartzite rock pool in France. \\\"Pedomicrobium manganicum\\\" has the ability to bind MnO2\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128626\",\"title\":\"Pennsylvania School of Horticulture for Women\",\"body\":\"\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128644\",\"title\":\"Victoria Angelova\",\"body\":\"\\nVictoria Angelova\\n\\nVictoria Angelova Vinarova (sometimes written Viktoria Angelova, , 1902–1947) was the first female architect of Bulgaria. She is credited with having built the first modern, national art gallery in the Balkans.\\nBiography.\\nVictoria Angelova was born on 20 November 1902 in Veliko Tarnovo, Bulgaria to Vasil Angelov, a merchant who had been educated in England. He named his daughter after Queen Victoria in homage. She graduated from the Vienna University of Technology and the Dresden Polytechnic. At the age of 24, she returned to Sofia and was working as an intern at the Ministry of Public Works when she won a contest for her first major commission. In 1933, Angelova married a fellow architect, Boris Vinarov and they set up a practice in Sofia. \\nAngelova worked during a period when most public projects were awarded after competitions which were open nationally. She designed buildings throughout the country, but is primarily known for those in Sofia. She was awarded the Order of Civil Merit for her architectural contributions to the country. In 1944, the couple's home was bombed and they lost many of their personal effects, including their architectural drawings. They evacuated to Turnovo, where Angelova became ill with a severe case of pneumonia. Believing she had recovered, they returned to Sophia in 1946, but Angelova relapsed and died on 27 December 1947. Her husband died three months later.\\nSelected projects.\\nMinistry of Public Buildings, Roads and Public Works.\\n The ministry held a contest in 1926 for designs of a new office building for the Ministry of Public Buildings, Roads and Public Works. Although Angelova won the contest, she was required to have the experienced architects and Yordan Yordanov () oversee her work. The building covered an entire city block known as \\\"Slaveykov Square\\\" and was bounded by \\\"GS Rakovski\\\", \\\"William Gladstone\\\" and \\\"Han Krum \\\" streets. Construction began in 1928 and was completed four years later.\\nThe design was Neoclassical and featured towering marble columns and mosaic floor tiles. Flanking the entrance were two statues, one female figure representing architecture and one male figure representing construction, as well as three stone heads in altorilievo adorning the doorways. The sculptures were completed by Mikhail Ivanov, Stephen Peychev and . The wide corridors and high ceilings are offset by stained glass designed by and made in Munich by F. Seiler. Today it houses the Metropolitan Library and the \\\"Renaissance Theater\\\" on one side and several doctors' and dentists' offices on the other.\\nNational Art Gallery.\\nFrom 1934 to 1941, Angelova worked on the National Art Gallery. The building was completed in 1942 and opened featuring a first floor which housed Renaissance paintings by Stanislav Dospevski, Nikolai Pavlovich, Hristo Tsokev, Zahari Zograf, as well as handcrafts from regions noted for folk artworks, such as Tryavna, Samokov and others. The second floor focused on contemporary Bulgarian art featuring painters and sculptors from the early 20th century. When completed, the building was recognized as the first modern, national art gallery in the Balkans. It was bombed in 1944 and completely destroyed, along with its contents, which included the complete works of leading Bulgarian sculptor \\nSea Casino of Burgas.\\n In 1936, Atanas Sirekov, the mayor of Burgas, initiated a design contest for a casino on Burgas Bay. Seventeen architects competed and Angelova won with a design she called \\\"333\\\". The building required a complex design due to the steep slope of the site and its panoramic view of the entire gulf area. Completed in 1938, the inauguration was attended by dignitaries from throughout the country, who received a special travel discount of 70% off the price of train tickets to attend. The building was abandoned in the late 1990s, but after almost 20 years of neglect, was restored and opened as a cultural center in 2011. The renovation project won the “Building of the Year” award for 2011.\\nRaduntsi Lung Hospital.\\nIn 1937 after a lengthy study had taken place, an act was passed for the construction of a tubercular sanatorium in the village of . The Pulmonary Hospital was to be the largest facility of its kind in the Balkans and located 650 meters above sea level. Angelova won the contract for the construction of the hospital and building was to begin shortly before World War II started. Ground was broken in 1939, but in 1940 after four stories were built, further construction was suspended until after the war ended. Construction resumed on 9 September 1944 but was not completed until 1955, nearly a decade after Angelova's death. The first patients were received at the hospital on 1 November 1955. Financial problems which began in 2013, forced the closure of the hospital in 2015.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128666\",\"title\":\"Pack Up Your Troubles (1939 film)\",\"body\":\"\\nPack Up Your Troubles (1939 film)\\n\\nPack Up Your Troubles is a 1939 American comedy film directed by H. Bruce Humberstone and written by Lou Breslow and Owen Francis. The film stars Jane Withers, The Ritz Brothers, Lynn Bari, Joseph Schildkraut, Stanley Fields, Fritz Leiber and Lionel Royce. The film was released on October 20, 1939, by 20th Century Fox.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128692\",\"title\":\"Chari Budruk\",\"body\":\"\\nChari Budruk\\n\\nChari Budruk is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Chari Budruk has 87 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 63.74%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128695\",\"title\":\"Satkor\",\"body\":\"\\nSatkor\\n\\nSatkor is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Satkor has 278 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 69.88%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128700\",\"title\":\"Dolhari Khurd\",\"body\":\"\\nDolhari Khurd\\n\\nDolhari Khurd is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Dolhari Khurd has 449 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 60.55%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128705\",\"title\":\"Apti Budruk\",\"body\":\"\\nApti Budruk\\n\\nApti Budruk is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Apti Budruk has 249 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 57.54%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128712\",\"title\":\"Uparale\",\"body\":\"\\nUparale\\n\\nUparale is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Uparale has 273 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 60.2%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128717\",\"title\":\"Sawade\",\"body\":\"\\nSawade\\n\\nSawade is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Sawade has 556 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 57.63%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128719\",\"title\":\"Gadadhe\",\"body\":\"\\nGadadhe\\n\\nGadadhe is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Gadadhe has 172 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 60.48%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128723\",\"title\":\"Nagzari, Vikramgad\",\"body\":\"\\nNagzari, Vikramgad\\n\\nNagzari is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Nagzari has 40 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 92.78%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128726\",\"title\":\"Yashwantnagar\",\"body\":\"\\nYashwantnagar\\n\\nYashwantnagar is a village in the Palghar district of Maharashtra, India. It is located in the Vikramgad taluka.\\nDemographics.\\nAccording to the 2011 census of India, Yashwantnagar has 375 households. The effective literacy rate (i.e. the literacy rate of population excluding children aged 6 and below) is 76.36%.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128732\",\"title\":\"Octav Dessila\",\"body\":\"\\nOctav Dessila\\n\\nOctav Dessila (December 4, 1895–July 29, 1976) was a Romanian novelist and playwright.\\nBorn in Bucharest, his parents were Iorgu Dessila, a \\\"Căile Ferate Române\\\" employee, and his wife Aristița (\\\"née\\\" Gheorghiu). He was part of the first class to graduate from the military high school at Dealu Monastery, and became an officer in the Romanian Land Forces. His first novel was \\\"Dragomir Valahul\\\" (1927), followed by \\\"Zvetlana\\\" (1930), \\\"București, orașul prăbușirilor\\\" (1930), \\\"Neastâmpăr\\\" (1934), \\\"Turbă\\\" (1936), \\\"Cartea cu minciuni\\\" (1936), \\\"Două chemări\\\" (vol. I-II, 1936), \\\"Iubim\\\" (vol. I-III, 1941-1943) and \\\"Porți fără număr\\\" (vol. I-II, 1946). He also wrote plays: \\\"Un om care dă palme vieții\\\" (1938) and \\\"Mihai Viteazul\\\" (1967). He belonged to the Romanian Writers' Society from 1931 to 1948, winning its prize in 1935; he was also a member of the Romanian Writers' Union from 1967. In 1937, he won the Romanian Academy's I. Al. Brătescu-Voinești prize. \\\"Iubim\\\" was the last in a string of commercial successes; revised and republished in 1970, it did not even attract attention from readers of its genre, suggesting the obsolete nature of Dessila's literary output.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128742\",\"title\":\"Corky Tharp\",\"body\":\"\\nCorky Tharp\\n\\nThomas Allen \\\"Corky\\\" Tharp (April 19, 1931 – April 3, 2003) was an American football defensive back who played one season for the New York Titans of the American Football League. He played college football at the University of Alabama for the Alabama Crimson Tide football team.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128764\",\"title\":\"David Gailey\",\"body\":\"\\nDavid Gailey\\n\\nDavid Gailey (1807 - 1881) was one of a number of Enrolled Pensioner Guards (EPGs) who came to the Swan River Colony between 1850 and 1868. Their role was to guard and oversee the work of the prisoners transported to Western Australia.\\nIn common with many of the Enrolled Pensioner Guards, Gailey was Irish and Catholic. He was born in Old Ross County in Watford in 1807. In December 1825, at the age of 18 years, he enlisted in the British Army, serving as a private in the 18th Regiment. He served for 20 years and was discharged in September 1846. He was 39 years of age. His record indicates his character was \\\"extremely good\\\" and he was awarded three good conduct badges. He was described as in height, with a fair complexion, grey eyes and hair dark. He married Margaret Hannen and in 1849 they had a son named John.\\nIn 1851 Gailey and his family travelled with a number of other Pensioner Guards to the settlement of Toodyay, where they were temporarily housed in A-framed straw huts and allotted plots of land. These allotments were later transferred to the permanent Convict Hiring Depot, upstream of the town. Thirteen allotments, S1 to S13, were marked out, and from 1852 to 1856 two-roomed brick cottages were erected. The Gaileys, whose family had increased with the birth of two daughters, Anna in 1851 and Ellen in 1856, were allocated one of the first three cottages to be completed. The Depot became known as the Pensioner Village. Canon Raffaele Martelli, who had been appointed in 1855 by Bishop Salvado to look after Toodyay’s Catholic community, occupied one of the cottages for a short time.\\nWhen more Pensioner Guard families arrived at the Depot, Martelli had to vacate the cottage and return to the townsite, where he was offered Gailey’s straw hut as temporary quarters. Martelli kept regular correspondence with Salvado and in one letter he thanks the bishop for sending a jar of butter that he wanted to give to Gailey. Martelli’s correspondence reveals a high regard for Gailey. \\nIn 1858, Gailey and many other Enrolled Pensioner Guards in the colony contributed to the Indian Relief Fund that had been set up in England following the Indian Mutiny of 1857. Many of the EPGs had served in India with the British Army before their retirement. The mutiny led to the ending of the East India Company in 1858, and the establishment of the British Raj.\\nIn 1860 the new town of Newcastle, located around the Convict Hiring Depot, had been surveyed. Gailey was allocated Lot S7 of , and purchased Lot 17 consisting of . This lot was located across the road from what became the Sisters of Mercy Convent, and at the southern end of Lot 17, the Roman Catholic St John the Baptist church was erected in 1863. Possibly around the same time a Catholic Presbytery was built across the road from the church on Lot S19.\\nDuring the 1860s Gailey employed four ticket-of-leave men, conducted a small school, and worked as a bootmaker. He offered to take in the Quinlan children, Timothy (born February 1861) and his sister Mary when their mother died while giving birth to twins. Their father was up north with a government party at the time. When their father also died the children were placed with Joseph Thomas Reilly, a prominent Catholic newspaperman and active citizen, who raised them with his own children. Timothy Quinlan went on to become a prominent politician and husband to Daniel Connor's daughter Teresa.\\nGailey continued to be a resident in Toodyay, dying on 18 April 1881.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128767\",\"title\":\"Arif Hajili\",\"body\":\"\\nArif Hajili\\n\\nArif Hajili () (born January 22, 1962), is a prominent Azerbaijani politician and leader of Equality Party \\\"(Müsavat)\\\", the largest opposition party in Azerbaijan.\\nBiography.\\nArif Hajili was born in 1962 in the Yukhari Tala village of Zagatala region. He graduated the Journalism Faculty of Baku State University. He worked as an editor at Zagatala radio station in 1983-1988. He was one of the leaders of the independence movement of Azerbaijan. Was a member of the Supreme Council of Azerbaijan in 1990-1995. Was a member of the Parliament of Azerbaijan. He was a Chairman of the Supreme Body of the PFA in 1991-1992. At various times he was a Deputy Head of the \\\"Musavat\\\" party on organizational matters. He worked as a State Advisor for the territorial government and the control of Azerbaijan during the reign of Elchibey in 1992-1993. Was arrested a number of times during the rule of Aliyevs. A former prisoner conscience. He has been the head of the Executive Office of \\\"Musavat\\\" party, a member of its supreme body - Divan. since 2006. He was elected Head of the party at the VIII Congress in 2014.\\nCurrently a chairman of the Musavat Party.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128774\",\"title\":\"Udland Church\",\"body\":\"\\nUdland Church\\n\\n \\nUdland Church () is a parish church in Haugesund municipality in Rogaland county, Norway. It is located in the town of Haugesund. The church is part of the Skåre parish in the Haugaland deanery in the Diocese of Stavanger. The stone/brick church was built in 2002 by the architect Thomas Brekke. The church seats about 450 people.\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128789\",\"title\":\"Alfia\",\"body\":\"\\nAlfia\\n\\nAlfia may refer to:\\n\\n\"}\n{\"url\":\"https://en.wikipedia.org/wiki?curid=48128792\",\"title\":\"Great Smoky Mountains Study\",\"body\":\"\\nGreat Smoky Mountains Study\\n\\nThe Great Smoky Mountains Study is a longitudinal study led by William Copeland (professor) from Duke University Medical Center that started in 1993 and ended in 2003. It followed 1,420 children from western North Carolina. Participants were interviewed at up to nine points in time - first aged 9 to 16, and again at ages 19–21.\\nDuring the study, about one quarter of the families saw a dramatic and unexpected increase in income. The study showed that among these children, instances of behavioral and emotional disorders decreased, and conscientiousness and agreeableness increased. Randall Akee remarked that \\\"It would be almost impossible to replicate this kind of longitudinal study”.\\n\\n\"}\n"
  },
  {
    "path": "bitpacker/Cargo.toml",
    "content": "[package]\nname = \"tantivy-bitpacker\"\nversion = \"0.9.0\"\nedition = \"2024\"\nauthors = [\"Paul Masurel <paul.masurel@gmail.com>\"]\nlicense = \"MIT\"\ncategories = []\ndescription = \"\"\"Tantivy-sub crate: bitpacking\"\"\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\nkeywords = []\ndocumentation = \"https://docs.rs/tantivy-bitpacker/latest/tantivy_bitpacker\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\n\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nbitpacking = { version = \"0.9.2\", default-features = false, features = [\"bitpacker1x\"] }\n\n[dev-dependencies]\nrand = \"0.9\"\nproptest = \"1\"\n"
  },
  {
    "path": "bitpacker/benches/bench.rs",
    "content": "#![feature(test)]\n\nextern crate test;\n\n#[cfg(test)]\nmod tests {\n    use rand::rng;\n    use rand::seq::IteratorRandom;\n    use tantivy_bitpacker::{BitPacker, BitUnpacker, BlockedBitpacker};\n    use test::Bencher;\n\n    #[inline(never)]\n    fn create_bitpacked_data(bit_width: u8, num_els: u32) -> Vec<u8> {\n        let mut bitpacker = BitPacker::new();\n        let mut buffer = Vec::new();\n        for _ in 0..num_els {\n            // the values do not matter.\n            bitpacker.write(0u64, bit_width, &mut buffer).unwrap();\n            bitpacker.flush(&mut buffer).unwrap();\n        }\n        buffer\n    }\n\n    #[bench]\n    fn bench_bitpacking_read(b: &mut Bencher) {\n        let bit_width = 3;\n        let num_els = 1_000_000u32;\n        let bit_unpacker = BitUnpacker::new(bit_width);\n        let data = create_bitpacked_data(bit_width, num_els);\n        let idxs: Vec<u32> = (0..num_els).choose_multiple(&mut rng(), 100_000);\n        b.iter(|| {\n            let mut out = 0u64;\n            for &idx in &idxs {\n                out = out.wrapping_add(bit_unpacker.get(idx, &data[..]));\n            }\n            out\n        });\n    }\n\n    #[bench]\n    fn bench_blockedbitp_read(b: &mut Bencher) {\n        let mut blocked_bitpacker = BlockedBitpacker::new();\n        for val in 0..=21500 {\n            blocked_bitpacker.add(val * val);\n        }\n        b.iter(|| {\n            let mut out = 0u64;\n            for val in 0..=21500 {\n                out = out.wrapping_add(blocked_bitpacker.get(val));\n            }\n            out\n        });\n    }\n\n    #[bench]\n    fn bench_blockedbitp_create(b: &mut Bencher) {\n        b.iter(|| {\n            let mut blocked_bitpacker = BlockedBitpacker::new();\n            for val in 0..=21500 {\n                blocked_bitpacker.add(val * val);\n            }\n            blocked_bitpacker\n        });\n    }\n}\n"
  },
  {
    "path": "bitpacker/src/bitpacker.rs",
    "content": "use std::io;\nuse std::ops::{Range, RangeInclusive};\n\nuse bitpacking::{BitPacker as ExternalBitPackerTrait, BitPacker1x};\n\npub struct BitPacker {\n    mini_buffer: u64,\n    mini_buffer_written: usize,\n}\n\nimpl Default for BitPacker {\n    fn default() -> Self {\n        BitPacker::new()\n    }\n}\nimpl BitPacker {\n    pub fn new() -> BitPacker {\n        BitPacker {\n            mini_buffer: 0u64,\n            mini_buffer_written: 0,\n        }\n    }\n\n    #[inline]\n    pub fn write<TWrite: io::Write + ?Sized>(\n        &mut self,\n        val: u64,\n        num_bits: u8,\n        output: &mut TWrite,\n    ) -> io::Result<()> {\n        let num_bits = num_bits as usize;\n        if self.mini_buffer_written + num_bits > 64 {\n            self.mini_buffer |= val.wrapping_shl(self.mini_buffer_written as u32);\n            output.write_all(self.mini_buffer.to_le_bytes().as_ref())?;\n            self.mini_buffer = val.wrapping_shr((64 - self.mini_buffer_written) as u32);\n            self.mini_buffer_written = self.mini_buffer_written + num_bits - 64;\n        } else {\n            self.mini_buffer |= val << self.mini_buffer_written;\n            self.mini_buffer_written += num_bits;\n            if self.mini_buffer_written == 64 {\n                output.write_all(self.mini_buffer.to_le_bytes().as_ref())?;\n                self.mini_buffer_written = 0;\n                self.mini_buffer = 0u64;\n            }\n        }\n        Ok(())\n    }\n\n    pub fn flush<TWrite: io::Write + ?Sized>(&mut self, output: &mut TWrite) -> io::Result<()> {\n        if self.mini_buffer_written > 0 {\n            let num_bytes = self.mini_buffer_written.div_ceil(8);\n            let bytes = self.mini_buffer.to_le_bytes();\n            output.write_all(&bytes[..num_bytes])?;\n            self.mini_buffer_written = 0;\n            self.mini_buffer = 0;\n        }\n        Ok(())\n    }\n\n    pub fn close<TWrite: io::Write + ?Sized>(&mut self, output: &mut TWrite) -> io::Result<()> {\n        self.flush(output)?;\n        Ok(())\n    }\n}\n\n#[derive(Clone, Debug, Default, Copy)]\npub struct BitUnpacker {\n    num_bits: usize,\n    mask: u64,\n}\n\nimpl BitUnpacker {\n    /// Creates a bit unpacker, that assumes the same bitwidth for all values.\n    ///\n    /// The bitunpacker works by doing an unaligned read of 8 bytes.\n    /// For this reason, values of `num_bits` between\n    /// [57..63] are forbidden.\n    pub fn new(num_bits: u8) -> BitUnpacker {\n        assert!(num_bits <= 7 * 8 || num_bits == 64);\n        let mask: u64 = if num_bits == 64 {\n            !0u64\n        } else {\n            (1u64 << num_bits) - 1u64\n        };\n        BitUnpacker {\n            num_bits: usize::from(num_bits),\n            mask,\n        }\n    }\n\n    pub fn bit_width(&self) -> u8 {\n        self.num_bits as u8\n    }\n\n    #[inline]\n    pub fn get(&self, idx: u32, data: &[u8]) -> u64 {\n        let addr_in_bits = idx as usize * self.num_bits;\n        let addr = addr_in_bits >> 3;\n        if addr + 8 > data.len() {\n            if self.num_bits == 0 {\n                return 0;\n            }\n            let bit_shift = addr_in_bits & 7;\n            return self.get_slow_path(addr, bit_shift as u32, data);\n        }\n        let bit_shift = addr_in_bits & 7;\n        let bytes: [u8; 8] = (&data[addr..addr + 8]).try_into().unwrap();\n        let val_unshifted_unmasked: u64 = u64::from_le_bytes(bytes);\n        let val_shifted = val_unshifted_unmasked >> bit_shift;\n        val_shifted & self.mask\n    }\n\n    #[inline(never)]\n    fn get_slow_path(&self, addr: usize, bit_shift: u32, data: &[u8]) -> u64 {\n        let mut bytes: [u8; 8] = [0u8; 8];\n        let available_bytes = data.len() - addr;\n        // This function is meant to only be called if we did not have 8 bytes to load.\n        debug_assert!(available_bytes < 8);\n        bytes[..available_bytes].copy_from_slice(&data[addr..]);\n        let val_unshifted_unmasked: u64 = u64::from_le_bytes(bytes);\n        let val_shifted = val_unshifted_unmasked >> bit_shift;\n        val_shifted & self.mask\n    }\n\n    // Decodes the range of bitpacked `u32` values with idx\n    // in [start_idx, start_idx + output.len()).\n    //\n    // #Panics\n    //\n    // This methods panics if `num_bits` is > 32.\n    fn get_batch_u32s(&self, start_idx: u32, data: &[u8], output: &mut [u32]) {\n        assert!(\n            self.bit_width() <= 32,\n            \"Bitwidth must be <= 32 to use this method.\"\n        );\n\n        let end_idx: u32 = start_idx + output.len() as u32;\n\n        // We use `usize` here to avoid overflow issues.\n        let end_bit_read = (end_idx as usize) * self.num_bits;\n        let end_byte_read = end_bit_read.div_ceil(8);\n        assert!(\n            end_byte_read <= data.len(),\n            \"Requested index is out of bounds.\"\n        );\n\n        // Simple slow implementation of get_batch_u32s, to deal with our ramps.\n        let get_batch_ramp = |start_idx: u32, output: &mut [u32]| {\n            for (out, idx) in output.iter_mut().zip(start_idx..) {\n                *out = self.get(idx, data) as u32;\n            }\n        };\n\n        // We use an unrolled routine to decode 32 values at once.\n        // We therefore decompose our range of values to decode into three ranges:\n        // - Entrance ramp: [start_idx, fast_track_start) (up to 31 values)\n        // - Highway: [fast_track_start, fast_track_end) (a length multiple of 32s)\n        // - Exit ramp: [fast_track_end, start_idx + output.len()) (up to 31 values)\n\n        // We want the start of the fast track to start align with bytes.\n        // A sufficient condition is to start with an idx that is a multiple of 8,\n        // so highway start is the closest multiple of 8 that is >= start_idx.\n        let entrance_ramp_len: u32 = 8 - (start_idx % 8) % 8;\n\n        let highway_start: u32 = start_idx + entrance_ramp_len;\n\n        if highway_start + (BitPacker1x::BLOCK_LEN as u32) > end_idx {\n            // We don't have enough values to have even a single block of highway.\n            // Let's just supply the values the simple way.\n            get_batch_ramp(start_idx, output);\n            return;\n        }\n\n        let num_blocks: usize = (end_idx - highway_start) as usize / BitPacker1x::BLOCK_LEN;\n\n        // Entrance ramp\n        get_batch_ramp(start_idx, &mut output[..entrance_ramp_len as usize]);\n\n        // Highway\n        let mut offset = (highway_start as usize * self.num_bits) / 8;\n        let mut output_cursor = (highway_start - start_idx) as usize;\n        for _ in 0..num_blocks {\n            offset += BitPacker1x.decompress(\n                &data[offset..],\n                &mut output[output_cursor..],\n                self.num_bits as u8,\n            );\n            output_cursor += 32;\n        }\n\n        // Exit ramp\n        let highway_end: u32 = highway_start + (num_blocks * BitPacker1x::BLOCK_LEN) as u32;\n        get_batch_ramp(highway_end, &mut output[output_cursor..]);\n    }\n\n    pub fn get_ids_for_value_range(\n        &self,\n        range: RangeInclusive<u64>,\n        id_range: Range<u32>,\n        data: &[u8],\n        positions: &mut Vec<u32>,\n    ) {\n        if self.bit_width() > 32 {\n            self.get_ids_for_value_range_slow(range, id_range, data, positions)\n        } else {\n            if *range.start() > u32::MAX as u64 {\n                positions.clear();\n                return;\n            }\n            let range_u32 = (*range.start() as u32)..=(*range.end()).min(u32::MAX as u64) as u32;\n            self.get_ids_for_value_range_fast(range_u32, id_range, data, positions)\n        }\n    }\n\n    fn get_ids_for_value_range_slow(\n        &self,\n        range: RangeInclusive<u64>,\n        id_range: Range<u32>,\n        data: &[u8],\n        positions: &mut Vec<u32>,\n    ) {\n        positions.clear();\n        for i in id_range {\n            // If we cared we could make this branchless, but the slow implementation should rarely\n            // kick in.\n            let val = self.get(i, data);\n            if range.contains(&val) {\n                positions.push(i);\n            }\n        }\n    }\n\n    fn get_ids_for_value_range_fast(\n        &self,\n        value_range: RangeInclusive<u32>,\n        id_range: Range<u32>,\n        data: &[u8],\n        positions: &mut Vec<u32>,\n    ) {\n        positions.resize(id_range.len(), 0u32);\n        self.get_batch_u32s(id_range.start, data, positions);\n        crate::filter_vec::filter_vec_in_place(value_range, id_range.start, positions)\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::{BitPacker, BitUnpacker};\n\n    fn create_bitpacker(len: usize, num_bits: u8) -> (BitUnpacker, Vec<u64>, Vec<u8>) {\n        let mut data = Vec::new();\n        let mut bitpacker = BitPacker::new();\n        let max_val: u64 = (1u64 << num_bits as u64) - 1u64;\n        let vals: Vec<u64> = (0u64..len as u64)\n            .map(|i| if max_val == 0 { 0 } else { i % max_val })\n            .collect();\n        for &val in &vals {\n            bitpacker.write(val, num_bits, &mut data).unwrap();\n        }\n        bitpacker.close(&mut data).unwrap();\n        assert_eq!(data.len(), ((num_bits as usize) * len).div_ceil(8));\n        let bitunpacker = BitUnpacker::new(num_bits);\n        (bitunpacker, vals, data)\n    }\n\n    fn test_bitpacker_util(len: usize, num_bits: u8) {\n        let (bitunpacker, vals, data) = create_bitpacker(len, num_bits);\n        for (i, val) in vals.iter().enumerate() {\n            assert_eq!(bitunpacker.get(i as u32, &data), *val);\n        }\n    }\n\n    #[test]\n    fn test_bitpacker() {\n        test_bitpacker_util(10, 3);\n        test_bitpacker_util(10, 0);\n        test_bitpacker_util(10, 1);\n        test_bitpacker_util(6, 14);\n        test_bitpacker_util(1000, 14);\n    }\n\n    use proptest::prelude::*;\n\n    fn num_bits_strategy() -> impl Strategy<Value = u8> {\n        prop_oneof!(Just(0), Just(1), 2u8..56u8, Just(56), Just(64),)\n    }\n\n    fn vals_strategy() -> impl Strategy<Value = (u8, Vec<u64>)> {\n        (num_bits_strategy(), 0usize..100usize).prop_flat_map(|(num_bits, len)| {\n            let max_val = if num_bits == 64 {\n                u64::MAX\n            } else {\n                (1u64 << num_bits as u32) - 1\n            };\n            let vals = proptest::collection::vec(0..=max_val, len);\n            vals.prop_map(move |vals| (num_bits, vals))\n        })\n    }\n\n    fn test_bitpacker_aux(num_bits: u8, vals: &[u64]) {\n        let mut buffer: Vec<u8> = Vec::new();\n        let mut bitpacker = BitPacker::new();\n        for &val in vals {\n            bitpacker.write(val, num_bits, &mut buffer).unwrap();\n        }\n        bitpacker.flush(&mut buffer).unwrap();\n        assert_eq!(buffer.len(), (vals.len() * num_bits as usize).div_ceil(8));\n        let bitunpacker = BitUnpacker::new(num_bits);\n        let max_val = if num_bits == 64 {\n            u64::MAX\n        } else {\n            (1u64 << num_bits) - 1\n        };\n        for (i, val) in vals.iter().copied().enumerate() {\n            assert!(val <= max_val);\n            assert_eq!(bitunpacker.get(i as u32, &buffer), val);\n        }\n    }\n\n    proptest::proptest! {\n        #[test]\n        fn test_bitpacker_proptest((num_bits, vals) in vals_strategy()) {\n            test_bitpacker_aux(num_bits, &vals);\n        }\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_get_batch_panics_over_32_bits() {\n        let bitunpacker = BitUnpacker::new(33);\n        let mut output: [u32; 1] = [0u32];\n        bitunpacker.get_batch_u32s(0, &[0, 0, 0, 0, 0, 0, 0, 0], &mut output[..]);\n    }\n\n    #[test]\n    fn test_get_batch_limit() {\n        let bitunpacker = BitUnpacker::new(1);\n        let mut output: [u32; 3] = [0u32, 0u32, 0u32];\n        bitunpacker.get_batch_u32s(8 * 4 - 3, &[0u8, 0u8, 0u8, 0u8], &mut output[..]);\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_get_batch_panics_when_off_scope() {\n        let bitunpacker = BitUnpacker::new(1);\n        let mut output: [u32; 3] = [0u32, 0u32, 0u32];\n        // We are missing exactly one bit.\n        bitunpacker.get_batch_u32s(8 * 4 - 2, &[0u8, 0u8, 0u8, 0u8], &mut output[..]);\n    }\n\n    proptest::proptest! {\n        #[test]\n        fn test_get_batch_u32s_proptest(num_bits in 0u8..=32u8) {\n            let mask =\n                if num_bits == 32u8 {\n                    u32::MAX\n                } else {\n                    (1u32 << num_bits) - 1\n                };\n            let mut buffer: Vec<u8> = Vec::new();\n            let mut bitpacker = BitPacker::new();\n            for val in 0..100 {\n                bitpacker.write(val & mask as u64, num_bits, &mut buffer).unwrap();\n            }\n            bitpacker.flush(&mut buffer).unwrap();\n            let bitunpacker = BitUnpacker::new(num_bits);\n            let mut output: Vec<u32> = Vec::new();\n            for len in [0, 1, 2, 32, 33, 34, 64] {\n                for start_idx in 0u32..32u32 {\n                    output.resize(len, 0);\n                    bitunpacker.get_batch_u32s(start_idx, &buffer, &mut output);\n                    for (i, output_byte) in output.iter().enumerate() {\n                        let expected = (start_idx + i as u32) & mask;\n                        assert_eq!(*output_byte, expected);\n                    }\n                }\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "bitpacker/src/blocked_bitpacker.rs",
    "content": "use super::bitpacker::BitPacker;\nuse super::compute_num_bits;\nuse crate::{BitUnpacker, minmax};\n\nconst BLOCK_SIZE: usize = 128;\n\n/// `BlockedBitpacker` compresses data in blocks of\n/// 128 elements, while keeping an index on it\n#[derive(Debug, Clone)]\npub struct BlockedBitpacker {\n    // bitpacked blocks\n    compressed_blocks: Vec<u8>,\n    // uncompressed data, collected until BLOCK_SIZE\n    buffer: Vec<u64>,\n    offset_and_bits: Vec<BlockedBitpackerEntryMetaData>,\n}\nimpl Default for BlockedBitpacker {\n    fn default() -> Self {\n        BlockedBitpacker::new()\n    }\n}\n\n/// `BlockedBitpackerEntryMetaData` encodes the\n/// offset and bit_width into a u64 bit field\n///\n/// This saves some space, since 7byte is more\n/// than enough and also keeps the access fast\n/// because of alignment\n#[derive(Debug, Clone, Default)]\nstruct BlockedBitpackerEntryMetaData {\n    encoded: u64,\n    base_value: u64,\n}\n\nimpl BlockedBitpackerEntryMetaData {\n    fn new(offset: u64, num_bits: u8, base_value: u64) -> Self {\n        let encoded = offset | (u64::from(num_bits) << (64 - 8));\n        Self {\n            encoded,\n            base_value,\n        }\n    }\n    fn offset(&self) -> u64 {\n        (self.encoded << 8) >> 8\n    }\n    fn num_bits(&self) -> u8 {\n        (self.encoded >> 56) as u8\n    }\n    fn base_value(&self) -> u64 {\n        self.base_value\n    }\n}\n\n#[test]\nfn metadata_test() {\n    let meta = BlockedBitpackerEntryMetaData::new(50000, 6, 40000);\n    assert_eq!(meta.offset(), 50000);\n    assert_eq!(meta.num_bits(), 6);\n}\n\nfn mem_usage<T>(items: &Vec<T>) -> usize {\n    items.capacity() * std::mem::size_of::<T>()\n}\n\nimpl BlockedBitpacker {\n    pub fn new() -> Self {\n        Self {\n            compressed_blocks: vec![0; 8],\n            buffer: vec![],\n            offset_and_bits: vec![],\n        }\n    }\n\n    /// The memory used (inclusive childs)\n    pub fn mem_usage(&self) -> usize {\n        std::mem::size_of::<BlockedBitpacker>()\n            + self.compressed_blocks.capacity()\n            + mem_usage(&self.offset_and_bits)\n            + mem_usage(&self.buffer)\n    }\n\n    #[inline]\n    pub fn add(&mut self, val: u64) {\n        self.buffer.push(val);\n        if self.buffer.len() == BLOCK_SIZE {\n            self.flush();\n        }\n    }\n\n    pub fn flush(&mut self) {\n        if let Some((min_value, max_value)) = minmax(self.buffer.iter()) {\n            let mut bit_packer = BitPacker::new();\n            let num_bits_block = compute_num_bits(*max_value - min_value);\n            // todo performance: the padding handling could be done better, e.g. use a slice and\n            // return num_bytes written from bitpacker\n            self.compressed_blocks\n                .resize(self.compressed_blocks.len() - 8, 0); // remove padding for bitpacker\n            let offset = self.compressed_blocks.len() as u64;\n            // todo performance: for some bit_width we\n            // can encode multiple vals into the\n            // mini_buffer before checking to flush\n            // (to be done in BitPacker)\n            for val in self.buffer.iter() {\n                bit_packer\n                    .write(\n                        *val - min_value,\n                        num_bits_block,\n                        &mut self.compressed_blocks,\n                    )\n                    .expect(\"cannot write bitpacking to output\"); // write to in memory can't fail\n            }\n            bit_packer.flush(&mut self.compressed_blocks).unwrap();\n            self.offset_and_bits\n                .push(BlockedBitpackerEntryMetaData::new(\n                    offset,\n                    num_bits_block,\n                    *min_value,\n                ));\n\n            self.buffer.clear();\n            self.compressed_blocks\n                .resize(self.compressed_blocks.len() + 8, 0); // add padding for bitpacker\n        }\n    }\n    #[inline]\n    pub fn get(&self, idx: usize) -> u64 {\n        let metadata_pos = idx / BLOCK_SIZE;\n        let pos_in_block = idx % BLOCK_SIZE;\n        if let Some(metadata) = self.offset_and_bits.get(metadata_pos) {\n            let unpacked = BitUnpacker::new(metadata.num_bits()).get(\n                pos_in_block as u32,\n                &self.compressed_blocks[metadata.offset() as usize..],\n            );\n            unpacked + metadata.base_value()\n        } else {\n            self.buffer[pos_in_block]\n        }\n    }\n\n    pub fn iter(&self) -> impl Iterator<Item = u64> + '_ {\n        // todo performance: we could decompress a whole block and cache it instead\n        let bitpacked_elems = self.offset_and_bits.len() * BLOCK_SIZE;\n\n        (0..bitpacked_elems)\n            .map(move |idx| self.get(idx))\n            .chain(self.buffer.iter().cloned())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    #[test]\n    fn blocked_bitpacker_empty() {\n        let blocked_bitpacker = BlockedBitpacker::new();\n        assert_eq!(blocked_bitpacker.iter().collect::<Vec<u64>>(), vec![]);\n    }\n    #[test]\n    fn blocked_bitpacker_one() {\n        let mut blocked_bitpacker = BlockedBitpacker::new();\n        blocked_bitpacker.add(50000);\n        assert_eq!(blocked_bitpacker.get(0), 50000);\n        assert_eq!(blocked_bitpacker.iter().collect::<Vec<u64>>(), vec![50000]);\n    }\n    #[test]\n    fn blocked_bitpacker_test() {\n        let mut blocked_bitpacker = BlockedBitpacker::new();\n        for val in 0..21500 {\n            blocked_bitpacker.add(val);\n        }\n        for val in 0..21500 {\n            assert_eq!(blocked_bitpacker.get(val as usize), val);\n        }\n        assert_eq!(blocked_bitpacker.iter().count(), 21500);\n        assert_eq!(blocked_bitpacker.iter().last().unwrap(), 21499);\n    }\n}\n"
  },
  {
    "path": "bitpacker/src/filter_vec/avx2.rs",
    "content": "//! SIMD filtering of a vector as described in the following blog post.\n//! <https://quickwit.io/blog/filtering%20a%20vector%20with%20simd%20instructions%20avx-2%20and%20avx-512>\nuse std::arch::x86_64::{\n    __m256i as DataType, _mm256_add_epi32 as op_add, _mm256_cmpgt_epi32 as op_greater,\n    _mm256_lddqu_si256 as load_unaligned, _mm256_or_si256 as op_or, _mm256_set1_epi32 as set1,\n    _mm256_storeu_si256 as store_unaligned, _mm256_xor_si256 as op_xor, *,\n};\nuse std::ops::RangeInclusive;\n\nconst NUM_LANES: usize = 8;\n\nconst HIGHEST_BIT: u32 = 1 << 31;\n\n#[inline]\nfn u32_to_i32(val: u32) -> i32 {\n    (val ^ HIGHEST_BIT) as i32\n}\n\n#[inline]\nunsafe fn u32_to_i32_avx2(vals_u32x8s: DataType) -> DataType {\n    const HIGHEST_BIT_MASK: DataType = from_u32x8([HIGHEST_BIT; NUM_LANES]);\n    unsafe { op_xor(vals_u32x8s, HIGHEST_BIT_MASK) }\n}\n\npub fn filter_vec_in_place(range: RangeInclusive<u32>, offset: u32, output: &mut Vec<u32>) {\n    // We use a monotonic mapping from u32 to i32 to make the comparison possible in AVX2.\n    let range_i32: RangeInclusive<i32> = u32_to_i32(*range.start())..=u32_to_i32(*range.end());\n    let num_words = output.len() / NUM_LANES;\n    let mut output_len = unsafe {\n        filter_vec_avx2_aux(\n            output.as_ptr() as *const __m256i,\n            range_i32,\n            output.as_mut_ptr(),\n            offset,\n            num_words,\n        )\n    };\n    let reminder_start = num_words * NUM_LANES;\n    for i in reminder_start..output.len() {\n        let val = output[i];\n        output[output_len] = offset + i as u32;\n        output_len += if range.contains(&val) { 1 } else { 0 };\n    }\n    output.truncate(output_len);\n}\n\n#[target_feature(enable = \"avx2\")]\nunsafe fn filter_vec_avx2_aux(\n    mut input: *const __m256i,\n    range: RangeInclusive<i32>,\n    output: *mut u32,\n    offset: u32,\n    num_words: usize,\n) -> usize {\n    let mut output_tail = output;\n    let range_simd = set1(*range.start())..=set1(*range.end());\n    let mut ids = from_u32x8([\n        offset,\n        offset + 1,\n        offset + 2,\n        offset + 3,\n        offset + 4,\n        offset + 5,\n        offset + 6,\n        offset + 7,\n    ]);\n    const SHIFT: __m256i = from_u32x8([NUM_LANES as u32; NUM_LANES]);\n    for _ in 0..num_words {\n        unsafe {\n            let word = load_unaligned(input);\n            let word = u32_to_i32_avx2(word);\n            let keeper_bitset = compute_filter_bitset(word, range_simd.clone());\n            let added_len = keeper_bitset.count_ones();\n            let filtered_doc_ids = compact(ids, keeper_bitset);\n            store_unaligned(output_tail as *mut __m256i, filtered_doc_ids);\n            output_tail = output_tail.offset(added_len as isize);\n            ids = op_add(ids, SHIFT);\n            input = input.offset(1);\n        }\n    }\n    unsafe { output_tail.offset_from(output) as usize }\n}\n\n#[inline]\n#[target_feature(enable = \"avx2\")]\nunsafe fn compact(data: DataType, mask: u8) -> DataType {\n    let vperm_mask = MASK_TO_PERMUTATION[mask as usize];\n    _mm256_permutevar8x32_epi32(data, vperm_mask)\n}\n\n#[inline]\n#[target_feature(enable = \"avx2\")]\nunsafe fn compute_filter_bitset(val: __m256i, range: std::ops::RangeInclusive<__m256i>) -> u8 {\n    let too_low = op_greater(*range.start(), val);\n    let too_high = op_greater(val, *range.end());\n    let inside = op_or(too_low, too_high);\n    255 - std::arch::x86_64::_mm256_movemask_ps(_mm256_castsi256_ps(inside)) as u8\n}\n\nunion U8x32 {\n    vector: DataType,\n    vals: [u32; NUM_LANES],\n}\n\nconst fn from_u32x8(vals: [u32; NUM_LANES]) -> DataType {\n    unsafe { U8x32 { vals }.vector }\n}\n\nconst MASK_TO_PERMUTATION: [DataType; 256] = [\n    from_u32x8([0, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 0, 0, 0, 0, 0]),\n    from_u32x8([3, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 3, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 0, 0, 0, 0]),\n    from_u32x8([4, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 4, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 0, 0, 0, 0]),\n    from_u32x8([3, 4, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 0, 0, 0, 0]),\n    from_u32x8([2, 3, 4, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 0, 0, 0]),\n    from_u32x8([5, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 5, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 5, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 5, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 5, 0, 0, 0, 0]),\n    from_u32x8([3, 5, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 5, 0, 0, 0, 0]),\n    from_u32x8([2, 3, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 5, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 5, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 5, 0, 0, 0]),\n    from_u32x8([4, 5, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([2, 4, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 5, 0, 0, 0]),\n    from_u32x8([3, 4, 5, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 5, 0, 0, 0]),\n    from_u32x8([2, 3, 4, 5, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 5, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 5, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 5, 0, 0]),\n    from_u32x8([6, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 6, 0, 0, 0, 0]),\n    from_u32x8([3, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 6, 0, 0, 0, 0]),\n    from_u32x8([2, 3, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 6, 0, 0, 0]),\n    from_u32x8([4, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([2, 4, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 6, 0, 0, 0]),\n    from_u32x8([3, 4, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 6, 0, 0, 0]),\n    from_u32x8([2, 3, 4, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 6, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 6, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 6, 0, 0]),\n    from_u32x8([5, 6, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 5, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 5, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([2, 5, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 5, 6, 0, 0, 0]),\n    from_u32x8([3, 5, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 5, 6, 0, 0, 0]),\n    from_u32x8([2, 3, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 5, 6, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 5, 6, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 5, 6, 0, 0]),\n    from_u32x8([4, 5, 6, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([2, 4, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 5, 6, 0, 0]),\n    from_u32x8([3, 4, 5, 6, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 5, 6, 0, 0]),\n    from_u32x8([2, 3, 4, 5, 6, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 5, 6, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 5, 6, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 5, 6, 0]),\n    from_u32x8([7, 0, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([2, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 7, 0, 0, 0, 0]),\n    from_u32x8([3, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 7, 0, 0, 0, 0]),\n    from_u32x8([2, 3, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 7, 0, 0, 0]),\n    from_u32x8([4, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([2, 4, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 7, 0, 0, 0]),\n    from_u32x8([3, 4, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 7, 0, 0, 0]),\n    from_u32x8([2, 3, 4, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 7, 0, 0]),\n    from_u32x8([5, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 5, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 5, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([2, 5, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 5, 7, 0, 0, 0]),\n    from_u32x8([3, 5, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 5, 7, 0, 0, 0]),\n    from_u32x8([2, 3, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 5, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 5, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 5, 7, 0, 0]),\n    from_u32x8([4, 5, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([2, 4, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 5, 7, 0, 0]),\n    from_u32x8([3, 4, 5, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 5, 7, 0, 0]),\n    from_u32x8([2, 3, 4, 5, 7, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 5, 7, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 5, 7, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 5, 7, 0]),\n    from_u32x8([6, 7, 0, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([1, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([2, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 2, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 6, 7, 0, 0, 0]),\n    from_u32x8([3, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 3, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 6, 7, 0, 0, 0]),\n    from_u32x8([2, 3, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 3, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 6, 7, 0, 0]),\n    from_u32x8([4, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 4, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([2, 4, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 6, 7, 0, 0]),\n    from_u32x8([3, 4, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 3, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 6, 7, 0, 0]),\n    from_u32x8([2, 3, 4, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 6, 7, 0, 0]),\n    from_u32x8([1, 2, 3, 4, 6, 7, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 6, 7, 0]),\n    from_u32x8([5, 6, 7, 0, 0, 0, 0, 0]),\n    from_u32x8([0, 5, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([1, 5, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 1, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([2, 5, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 2, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 2, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 2, 5, 6, 7, 0, 0]),\n    from_u32x8([3, 5, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 3, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 3, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 3, 5, 6, 7, 0, 0]),\n    from_u32x8([2, 3, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 2, 3, 5, 6, 7, 0, 0]),\n    from_u32x8([1, 2, 3, 5, 6, 7, 0, 0]),\n    from_u32x8([0, 1, 2, 3, 5, 6, 7, 0]),\n    from_u32x8([4, 5, 6, 7, 0, 0, 0, 0]),\n    from_u32x8([0, 4, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([1, 4, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 1, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([2, 4, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 2, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([1, 2, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([0, 1, 2, 4, 5, 6, 7, 0]),\n    from_u32x8([3, 4, 5, 6, 7, 0, 0, 0]),\n    from_u32x8([0, 3, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([1, 3, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([0, 1, 3, 4, 5, 6, 7, 0]),\n    from_u32x8([2, 3, 4, 5, 6, 7, 0, 0]),\n    from_u32x8([0, 2, 3, 4, 5, 6, 7, 0]),\n    from_u32x8([1, 2, 3, 4, 5, 6, 7, 0]),\n    from_u32x8([0, 1, 2, 3, 4, 5, 6, 7]),\n];\n"
  },
  {
    "path": "bitpacker/src/filter_vec/mod.rs",
    "content": "use std::ops::RangeInclusive;\n\n#[cfg(target_arch = \"x86_64\")]\nmod avx2;\n\nmod scalar;\n\n#[derive(Clone, Copy, Eq, PartialEq, Debug)]\n#[repr(u8)]\nenum FilterImplPerInstructionSet {\n    #[cfg(target_arch = \"x86_64\")]\n    AVX2 = 0u8,\n    Scalar = 1u8,\n}\n\nimpl FilterImplPerInstructionSet {\n    #[inline]\n    pub fn is_available(&self) -> bool {\n        match *self {\n            #[cfg(target_arch = \"x86_64\")]\n            FilterImplPerInstructionSet::AVX2 => is_x86_feature_detected!(\"avx2\"),\n            FilterImplPerInstructionSet::Scalar => true,\n        }\n    }\n}\n\n// List of available implementation in preferred order.\n#[cfg(target_arch = \"x86_64\")]\nconst IMPLS: [FilterImplPerInstructionSet; 2] = [\n    FilterImplPerInstructionSet::AVX2,\n    FilterImplPerInstructionSet::Scalar,\n];\n\n#[cfg(not(target_arch = \"x86_64\"))]\nconst IMPLS: [FilterImplPerInstructionSet; 1] = [FilterImplPerInstructionSet::Scalar];\n\nimpl FilterImplPerInstructionSet {\n    #[inline]\n    #[allow(unused_variables)] // on non-x86_64, code is unused.\n    fn from(code: u8) -> FilterImplPerInstructionSet {\n        #[cfg(target_arch = \"x86_64\")]\n        if code == FilterImplPerInstructionSet::AVX2 as u8 {\n            return FilterImplPerInstructionSet::AVX2;\n        }\n        FilterImplPerInstructionSet::Scalar\n    }\n\n    #[inline]\n    fn filter_vec_in_place(self, range: RangeInclusive<u32>, offset: u32, output: &mut Vec<u32>) {\n        match self {\n            #[cfg(target_arch = \"x86_64\")]\n            FilterImplPerInstructionSet::AVX2 => avx2::filter_vec_in_place(range, offset, output),\n            FilterImplPerInstructionSet::Scalar => {\n                scalar::filter_vec_in_place(range, offset, output)\n            }\n        }\n    }\n}\n\n#[inline]\nfn get_best_available_instruction_set() -> FilterImplPerInstructionSet {\n    use std::sync::atomic::{AtomicU8, Ordering};\n    static INSTRUCTION_SET_BYTE: AtomicU8 = AtomicU8::new(u8::MAX);\n    let instruction_set_byte: u8 = INSTRUCTION_SET_BYTE.load(Ordering::Relaxed);\n    if instruction_set_byte == u8::MAX {\n        // Let's initialize the instruction set and cache it.\n        let instruction_set = IMPLS\n            .into_iter()\n            .find(FilterImplPerInstructionSet::is_available)\n            .unwrap();\n        INSTRUCTION_SET_BYTE.store(instruction_set as u8, Ordering::Relaxed);\n        return instruction_set;\n    }\n    FilterImplPerInstructionSet::from(instruction_set_byte)\n}\n\npub fn filter_vec_in_place(range: RangeInclusive<u32>, offset: u32, output: &mut Vec<u32>) {\n    get_best_available_instruction_set().filter_vec_in_place(range, offset, output)\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_get_best_available_instruction_set() {\n        // This does not test much unfortunately.\n        // We just make sure the function returns without crashing and returns the same result.\n        let instruction_set = get_best_available_instruction_set();\n        assert_eq!(get_best_available_instruction_set(), instruction_set);\n    }\n\n    #[cfg(target_arch = \"x86_64\")]\n    #[test]\n    fn test_instruction_set_to_code_from_code() {\n        for instruction_set in [\n            FilterImplPerInstructionSet::AVX2,\n            FilterImplPerInstructionSet::Scalar,\n        ] {\n            let code = instruction_set as u8;\n            assert_eq!(instruction_set, FilterImplPerInstructionSet::from(code));\n        }\n    }\n\n    fn test_filter_impl_empty_aux(filter_impl: FilterImplPerInstructionSet) {\n        let mut output = vec![];\n        filter_impl.filter_vec_in_place(0..=u32::MAX, 0, &mut output);\n        assert_eq!(&output, &[]);\n    }\n\n    fn test_filter_impl_simple_aux(filter_impl: FilterImplPerInstructionSet) {\n        let mut output = vec![3, 2, 1, 5, 11, 2, 5, 10, 2];\n        filter_impl.filter_vec_in_place(3..=10, 0, &mut output);\n        assert_eq!(&output, &[0, 3, 6, 7]);\n    }\n\n    fn test_filter_impl_simple_aux_shifted(filter_impl: FilterImplPerInstructionSet) {\n        let mut output = vec![3, 2, 1, 5, 11, 2, 5, 10, 2];\n        filter_impl.filter_vec_in_place(3..=10, 10, &mut output);\n        assert_eq!(&output, &[10, 13, 16, 17]);\n    }\n\n    fn test_filter_impl_simple_outside_i32_range(filter_impl: FilterImplPerInstructionSet) {\n        let mut output = vec![u32::MAX, i32::MAX as u32 + 1, 0, 1, 3, 1, 1, 1, 1];\n        filter_impl.filter_vec_in_place(1..=i32::MAX as u32 + 1u32, 0, &mut output);\n        assert_eq!(&output, &[1, 3, 4, 5, 6, 7, 8]);\n    }\n\n    fn test_filter_impl_test_suite(filter_impl: FilterImplPerInstructionSet) {\n        test_filter_impl_empty_aux(filter_impl);\n        test_filter_impl_simple_aux(filter_impl);\n        test_filter_impl_simple_aux_shifted(filter_impl);\n        test_filter_impl_simple_outside_i32_range(filter_impl);\n    }\n\n    #[test]\n    #[cfg(target_arch = \"x86_64\")]\n    fn test_filter_implementation_avx2() {\n        if FilterImplPerInstructionSet::AVX2.is_available() {\n            test_filter_impl_test_suite(FilterImplPerInstructionSet::AVX2);\n        }\n    }\n\n    #[test]\n    fn test_filter_implementation_scalar() {\n        test_filter_impl_test_suite(FilterImplPerInstructionSet::Scalar);\n    }\n\n    #[cfg(target_arch = \"x86_64\")]\n    proptest::proptest! {\n        #[test]\n        fn test_filter_compare_scalar_and_avx2_impl_proptest(\n            start in proptest::prelude::any::<u32>(),\n            end in proptest::prelude::any::<u32>(),\n            offset in 0u32..2u32,\n            mut vals in proptest::collection::vec(0..u32::MAX, 0..30)) {\n            if FilterImplPerInstructionSet::AVX2.is_available() {\n                let mut vals_clone = vals.clone();\n                FilterImplPerInstructionSet::AVX2.filter_vec_in_place(start..=end, offset, &mut vals);\n                FilterImplPerInstructionSet::Scalar.filter_vec_in_place(start..=end, offset, &mut vals_clone);\n                assert_eq!(&vals, &vals_clone);\n            }\n       }\n    }\n}\n"
  },
  {
    "path": "bitpacker/src/filter_vec/scalar.rs",
    "content": "use std::ops::RangeInclusive;\n\npub fn filter_vec_in_place(range: RangeInclusive<u32>, offset: u32, output: &mut Vec<u32>) {\n    // We restrict the accepted boundary, because unsigned integers & SIMD don't\n    // play well.\n    let mut output_cursor = 0;\n    for i in 0..output.len() {\n        let val = output[i];\n        output[output_cursor] = offset + i as u32;\n        output_cursor += if range.contains(&val) { 1 } else { 0 };\n    }\n    output.truncate(output_cursor);\n}\n"
  },
  {
    "path": "bitpacker/src/lib.rs",
    "content": "mod bitpacker;\nmod blocked_bitpacker;\nmod filter_vec;\n\nuse std::cmp::Ordering;\n\npub use crate::bitpacker::{BitPacker, BitUnpacker};\npub use crate::blocked_bitpacker::BlockedBitpacker;\n\n/// Computes the number of bits that will be used for bitpacking.\n///\n/// In general the target is the minimum number of bits\n/// required to express the amplitude given in argument.\n///\n/// e.g. If the amplitude is 10, we can store all ints on simply 4bits.\n///\n/// The logic is slightly more convoluted here as for optimization\n/// reasons, we want to ensure that a value spawns over at most 8 bytes\n/// of aligned bytes.\n///\n/// Spanning over 9 bytes is possible for instance, if we do\n/// bitpacking with an amplitude of 63 bits.\n/// In this case, the second int will start on bit\n/// 63 (which belongs to byte 7) and ends at byte 15;\n/// Hence 9 bytes (from byte 7 to byte 15 included).\n///\n/// To avoid this, we force the number of bits to 64bits\n/// when the result is greater than `64-8 = 56 bits`.\n///\n/// Note that this only affects rare use cases spawning over\n/// a very large range of values. Even in this case, it results\n/// in an extra cost of at most 12% compared to the optimal\n/// number of bits.\npub fn compute_num_bits(n: u64) -> u8 {\n    let amplitude = (64u32 - n.leading_zeros()) as u8;\n    if amplitude <= 64 - 8 { amplitude } else { 64 }\n}\n\n/// Computes the (min, max) of an iterator of `PartialOrd` values.\n///\n/// For values implementing `Ord` (in a way consistent to their `PartialOrd` impl),\n/// this function behaves as expected.\n///\n/// For values with partial ordering, the behavior is non-trivial and may\n/// depends on the order of the values.\n/// For floats however, it simply returns the same results as if NaN were\n/// skipped.\npub fn minmax<I, T>(mut vals: I) -> Option<(T, T)>\nwhere\n    I: Iterator<Item = T>,\n    T: Copy + PartialOrd,\n{\n    let first_el = vals.find(|val| {\n        // We use this to make sure we skip all NaN values when\n        // working with a float type.\n        val.partial_cmp(val) == Some(Ordering::Equal)\n    })?;\n    let mut min_so_far: T = first_el;\n    let mut max_so_far: T = first_el;\n    for val in vals {\n        if val.partial_cmp(&min_so_far) == Some(Ordering::Less) {\n            min_so_far = val;\n        }\n        if val.partial_cmp(&max_so_far) == Some(Ordering::Greater) {\n            max_so_far = val;\n        }\n    }\n    Some((min_so_far, max_so_far))\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_compute_num_bits() {\n        assert_eq!(compute_num_bits(1), 1u8);\n        assert_eq!(compute_num_bits(0), 0u8);\n        assert_eq!(compute_num_bits(2), 2u8);\n        assert_eq!(compute_num_bits(3), 2u8);\n        assert_eq!(compute_num_bits(4), 3u8);\n        assert_eq!(compute_num_bits(255), 8u8);\n        assert_eq!(compute_num_bits(256), 9u8);\n        assert_eq!(compute_num_bits(5_000_000_000), 33u8);\n    }\n\n    #[test]\n    fn test_minmax_empty() {\n        let vals: Vec<u32> = vec![];\n        assert_eq!(minmax(vals.into_iter()), None);\n    }\n\n    #[test]\n    fn test_minmax_one() {\n        assert_eq!(minmax(vec![1].into_iter()), Some((1, 1)));\n    }\n\n    #[test]\n    fn test_minmax_two() {\n        assert_eq!(minmax(vec![1, 2].into_iter()), Some((1, 2)));\n        assert_eq!(minmax(vec![2, 1].into_iter()), Some((1, 2)));\n    }\n\n    #[test]\n    fn test_minmax_nan() {\n        assert_eq!(\n            minmax(vec![f64::NAN, 1f64, 2f64].into_iter()),\n            Some((1f64, 2f64))\n        );\n        assert_eq!(\n            minmax(vec![2f64, f64::NAN, 1f64].into_iter()),\n            Some((1f64, 2f64))\n        );\n        assert_eq!(\n            minmax(vec![2f64, 1f64, f64::NAN].into_iter()),\n            Some((1f64, 2f64))\n        );\n    }\n\n    #[test]\n    fn test_minmax_inf() {\n        assert_eq!(\n            minmax(vec![f64::INFINITY, 1f64, 2f64].into_iter()),\n            Some((1f64, f64::INFINITY))\n        );\n        assert_eq!(\n            minmax(vec![-f64::INFINITY, 1f64, 2f64].into_iter()),\n            Some((-f64::INFINITY, 2f64))\n        );\n        assert_eq!(\n            minmax(vec![2f64, f64::INFINITY, 1f64].into_iter()),\n            Some((1f64, f64::INFINITY))\n        );\n        assert_eq!(\n            minmax(vec![2f64, 1f64, -f64::INFINITY].into_iter()),\n            Some((-f64::INFINITY, 2f64))\n        );\n    }\n}\n"
  },
  {
    "path": "cliff.toml",
    "content": "# configuration file for git-cliff{ pattern = \"foo\", replace = \"bar\"}\n# see https://github.com/orhun/git-cliff#configuration-file\n\n[remote.github]\nowner = \"quickwit-oss\"\nrepo = \"tantivy\"\n\n[changelog]\n# changelog header\nheader = \"\"\"\n\"\"\"\n# template for the changelog body\n# https://tera.netlify.app/docs/#introduction\nbody = \"\"\"\n## What's Changed\n\n{%- if version %} in {{ version }}{%- endif -%}\n{% for commit in commits %}\n  {% if commit.remote.pr_title -%}\n    {%- set commit_message = commit.remote.pr_title -%}\n  {%- else -%}\n    {%- set commit_message = commit.message -%}\n  {%- endif -%}\n  - {{ commit_message | split(pat=\"\\n\") | first | trim }}\\\n    {% if commit.remote.pr_number %} \\\n      [#{{ commit.remote.pr_number }}]({{ self::remote_url() }}/pull/{{ commit.remote.pr_number }}){% if commit.remote.username %}(@{{ commit.remote.username }}){%- endif -%} \\\n    {%- endif %}\n{%- endfor -%}\n\n{% if github.contributors | filter(attribute=\"is_first_time\", value=true) | length != 0 %}\n  {% raw %}\\n{% endraw -%}\n  ## New Contributors\n{%- endif %}\\\n{% for contributor in github.contributors | filter(attribute=\"is_first_time\", value=true) %}\n  * @{{ contributor.username }} made their first contribution\n    {%- if contributor.pr_number %} in \\\n      [#{{ contributor.pr_number }}]({{ self::remote_url() }}/pull/{{ contributor.pr_number }}) \\\n    {%- endif %}\n{%- endfor -%}\n\n{% if version %}\n    {% if previous.version %}\n      **Full Changelog**: {{ self::remote_url() }}/compare/{{ previous.version }}...{{ version }}\n    {% endif %}\n{% else -%}\n  {% raw %}\\n{% endraw %}\n{% endif %}\n\n{%- macro remote_url() -%}\n  https://github.com/{{ remote.github.owner }}/{{ remote.github.repo }}\n{%- endmacro -%}\n\"\"\"\n# remove the leading and trailing whitespace from the template\ntrim = true\n# changelog footer\nfooter = \"\"\"\n\"\"\"\n\npostprocessors = [\n]\n\n[git]\n# parse the commits based on https://www.conventionalcommits.org\n# This is required or commit.message contains the whole commit message and not just the title\nconventional_commits = false\n# filter out the commits that are not conventional\nfilter_unconventional = true\n# process each line of a commit as an individual commit\nsplit_commits = false\n# regex for preprocessing the commit messages\ncommit_preprocessors = [\n    { pattern = '\\((\\w+\\s)?#([0-9]+)\\)', replace = \"\"},\n]\n#link_parsers = [\n    #{ pattern = \"#(\\\\d+)\", href = \"https://github.com/quickwit-oss/tantivy/pulls/$1\"},\n#]\n# regex for parsing and grouping commits\n# protect breaking changes from being skipped due to matching a skipping commit_parser\nprotect_breaking_commits = false\n# filter out the commits that are not matched by commit parsers\nfilter_commits = false\n# glob pattern for matching git tags\ntag_pattern = \"v[0-9]*\"\n# regex for skipping tags\nskip_tags = \"v0.1.0-beta.1\"\n# regex for ignoring tags\nignore_tags = \"\"\n# sort the tags topologically\ntopo_order = false\n# sort the commits inside sections by oldest/newest order\nsort_commits = \"newest\"\n# limit the number of commits included in the changelog.\n# limit_commits = 42\n"
  },
  {
    "path": "columnar/Cargo.toml",
    "content": "[package]\nname = \"tantivy-columnar\"\nversion = \"0.6.0\"\nedition = \"2024\"\nlicense = \"MIT\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\ndescription = \"column oriented storage for tantivy\"\ncategories = [\"database-implementations\", \"data-structures\", \"compression\"]\n\n[dependencies]\nitertools = \"0.14.0\"\nfastdivide = \"0.4.0\"\n\nstacker = { version= \"0.6\", path = \"../stacker\", package=\"tantivy-stacker\"}\nsstable = { version= \"0.6\", path = \"../sstable\", package = \"tantivy-sstable\" }\ncommon = { version= \"0.10\", path = \"../common\", package = \"tantivy-common\" }\ntantivy-bitpacker = { version= \"0.9\", path = \"../bitpacker/\" }\nserde = \"1.0.152\"\ndowncast-rs = \"2.0.1\"\n\n[dev-dependencies]\nproptest = \"1\"\nmore-asserts = \"0.3.1\"\nrand = \"0.9\"\nbinggan = \"0.14.0\"\n\n[[bench]]\nname = \"bench_merge\"\nharness = false\n\n[[bench]]\nname = \"bench_access\"\nharness = false\n\n[[bench]]\nname = \"bench_first_vals\"\nharness = false\n\n[[bench]]\nname = \"bench_values_u64\"\nharness = false\n\n[[bench]]\nname = \"bench_values_u128\"\nharness = false\n\n[[bench]]\nname = \"bench_create_column_values\"\nharness = false\n\n[[bench]]\nname = \"bench_column_values_get\"\nharness = false\n\n[[bench]]\nname = \"bench_optional_index\"\nharness = false\n\n[features]\nzstd-compression = [\"sstable/zstd-compression\"]\n"
  },
  {
    "path": "columnar/README.md",
    "content": "# Columnar format\n\nThis crate describes columnar format used in tantivy.\n\n## Goals\n\nThis format is special in the following way.\n- it needs to be compact\n- accessing a specific column does not require to load the entire columnar. It can be done in 2 to 3 random access.\n- columns of several types can be associated with the same column name.\n- it needs to support columns with different types `(str, u64, i64, f64)`\nand different cardinality `(required, optional, multivalued)`.\n- columns, once loaded, offer cheap random access.\n- it is designed to allow range queries.\n\n# Coercion rules\n\nUsers can create a columnar by inserting rows to a `ColumnarWriter`,\nand serializing it into a `Write` object.\nNothing prevents a user from recording values with different type to the same `column_name`.\n\nIn that case, `tantivy-columnar`'s behavior is as follows:\n- JsonValues are grouped into 3 types (String, Number, bool).\nValues that corresponds to different groups are mapped to different columns. For instance, String values are treated independently\nfrom Number or boolean values. `tantivy-columnar` will simply emit several columns associated to a given column_name.\n- Only one column for a given json value type is emitted.  If number values with different number types are recorded (e.g. u64, i64, f64),\n`tantivy-columnar` will pick the first type that can represents the set of appended value, with the following prioriy order (`i64`, `u64`, `f64`).\n`i64` is picked over `u64` as it is likely to  yield less change of types. Most use cases strictly requiring `u64` show the\nrestriction on 50% of the values (e.g. a 64-bit hash). On the other hand, a lot of use cases can show rare negative value.\n\n# Columnar format\n\nThis columnar format may have more than one column (with different types) associated to the same `column_name` (see [Coercion rules](#coercion-rules) above).\nThe `(column_name, column_type)` couple however uniquely identifies a column.\nThat couple is serialized as a column `column_key`.  The format of that key is:\n`[column_name][ZERO_BYTE][column_type_header: u8]`\n\n```\nCOLUMNAR:=\n    [COLUMNAR_DATA]\n    [COLUMNAR_KEY_TO_DATA_INDEX]\n    [COLUMNAR_FOOTER];\n\n\n# Columns are sorted by their column key.\nCOLUMNAR_DATA:=\n    [COLUMN_DATA]+;\n\nCOLUMNAR_FOOTER := [RANGE_SSTABLE_BYTES_LEN: 8 bytes little endian]\n\n```\n\nThe columnar file starts by the actual column data, concatenated one after the other,\nsorted by column key.\n\nA sstable associates\n`(column name, column_cardinality, column_type) to range of bytes.\n\nColumn name may not contain the zero byte `\\0`.\n\nListing all columns associated to `column_name` can therefore\nbe done by listing all keys prefixed by\n`[column_name][ZERO_BYTE]`\n\nThe associated range of bytes refer to a range of bytes\n\nThis crate exposes a columnar format for tantivy.\nThis format is described in README.md\n\n\nThe crate introduces the following concepts.\n\n`Columnar` is an equivalent of a dataframe.\nIt maps `column_key` to `Column`.\n\nA `Column<T>` associates a `RowId` (u32) to any\nnumber of values.\n\nThis is made possible by wrapping a `ColumnIndex` and a `ColumnValue` object.\nThe `ColumnValue<T>` represents a mapping that associates each `RowId` to\nexactly one single value.\n\nThe `ColumnIndex` then maps each RowId to a set of `RowId` in the\n`ColumnValue`.\n\nFor optimization, and compression purposes, the `ColumnIndex` has three\npossible representation, each for different cardinalities.\n\n- Full\n\nAll RowId have exactly one value. The ColumnIndex is the trivial mapping.\n\n- Optional\n\nAll RowIds can have at most one value. The ColumnIndex is the trivial mapping `ColumnRowId -> Option<ColumnValueRowId>`.\n\n- Multivalued\n\nAll RowIds can have any number of values.\nThe column index is mapping values to a range.\n\n\nAll these objects are implemented an unit tested independently\nin their own module:\n\n- columnar\n- column_index\n- column_values\n- column\n"
  },
  {
    "path": "columnar/benches/bench_access.rs",
    "content": "use binggan::{InputGroup, black_box};\nuse common::*;\nuse tantivy_columnar::Column;\n\npub mod common;\n\nconst NUM_DOCS: u32 = 2_000_000;\n\npub fn generate_columnar_and_open(card: Card, num_docs: u32) -> Column {\n    let reader = generate_columnar_with_name(card, num_docs, \"price\");\n    reader.read_columns(\"price\").unwrap()[0]\n        .open_u64_lenient()\n        .unwrap()\n        .unwrap()\n}\n\nfn main() {\n    let mut inputs = Vec::new();\n\n    let mut add_card = |card1: Card| {\n        inputs.push((\n            card1.to_string(),\n            generate_columnar_and_open(card1, NUM_DOCS),\n        ));\n    };\n\n    add_card(Card::MultiSparse);\n    add_card(Card::Multi);\n    add_card(Card::Sparse);\n    add_card(Card::Dense);\n    add_card(Card::Full);\n\n    bench_group(InputGroup::new_with_inputs(inputs));\n}\n\nfn bench_group(mut runner: InputGroup<Column>) {\n    runner.register(\"access_values_for_doc\", |column| {\n        let mut sum = 0;\n        for i in 0..NUM_DOCS {\n            for value in column.values_for_doc(i) {\n                sum += value;\n            }\n        }\n        black_box(sum);\n    });\n    runner.register(\"access_first_vals\", |column| {\n        let mut sum = 0;\n        const BLOCK_SIZE: usize = 32;\n        let mut docs = vec![0; BLOCK_SIZE];\n        let mut buffer = vec![None; BLOCK_SIZE];\n        for i in (0..NUM_DOCS).step_by(BLOCK_SIZE) {\n            // fill docs\n            #[allow(clippy::needless_range_loop)]\n            for idx in 0..BLOCK_SIZE {\n                docs[idx] = idx as u32 + i;\n            }\n\n            column.first_vals(&docs, &mut buffer);\n            for val in buffer.iter() {\n                let Some(val) = val else { continue };\n                sum += *val;\n            }\n        }\n\n        black_box(sum);\n    });\n    runner.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_column_values_get.rs",
    "content": "use std::sync::Arc;\n\nuse binggan::{InputGroup, black_box};\nuse rand::rngs::StdRng;\nuse rand::{Rng, SeedableRng};\nuse tantivy_columnar::ColumnValues;\nuse tantivy_columnar::column_values::{CodecType, serialize_and_load_u64_based_column_values};\n\nfn get_data() -> Vec<u64> {\n    let mut rng = StdRng::seed_from_u64(2u64);\n    let mut data: Vec<_> = (100..55_000_u64)\n        .map(|num| num + rng.random::<u8>() as u64)\n        .collect();\n    data.push(99_000);\n    data.insert(1000, 2000);\n    data.insert(2000, 100);\n    data.insert(3000, 4100);\n    data.insert(4000, 100);\n    data.insert(5000, 800);\n    data\n}\n\n#[inline(never)]\nfn value_iter() -> impl Iterator<Item = u64> {\n    0..20_000\n}\n\ntype Col = Arc<dyn ColumnValues<u64>>;\n\nfn main() {\n    let data = get_data();\n    let inputs: Vec<(String, Col)> = vec![\n        (\n            \"bitpacked\".to_string(),\n            serialize_and_load_u64_based_column_values(&data.as_slice(), &[CodecType::Bitpacked]),\n        ),\n        (\n            \"linear\".to_string(),\n            serialize_and_load_u64_based_column_values(&data.as_slice(), &[CodecType::Linear]),\n        ),\n        (\n            \"blockwise_linear\".to_string(),\n            serialize_and_load_u64_based_column_values(\n                &data.as_slice(),\n                &[CodecType::BlockwiseLinear],\n            ),\n        ),\n    ];\n\n    let mut group: InputGroup<Col> = InputGroup::new_with_inputs(inputs);\n\n    group.register(\"fastfield_get\", |col: &Col| {\n        let mut sum = 0u64;\n        for pos in value_iter() {\n            sum = sum.wrapping_add(col.get_val(pos as u32));\n        }\n        black_box(sum);\n    });\n\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_create_column_values.rs",
    "content": "use binggan::{InputGroup, black_box};\nuse rand::rngs::StdRng;\nuse rand::{Rng, SeedableRng};\nuse tantivy_columnar::column_values::{CodecType, serialize_u64_based_column_values};\n\nfn get_data() -> Vec<u64> {\n    let mut rng = StdRng::seed_from_u64(2u64);\n    let mut data: Vec<_> = (100..55_000_u64)\n        .map(|num| num + rng.random::<u8>() as u64)\n        .collect();\n    data.push(99_000);\n    data.insert(1000, 2000);\n    data.insert(2000, 100);\n    data.insert(3000, 4100);\n    data.insert(4000, 100);\n    data.insert(5000, 800);\n    data\n}\n\nfn main() {\n    let data = get_data();\n    let mut group: InputGroup<(CodecType, Vec<u64>)> = InputGroup::new_with_inputs(vec![\n        (\n            \"bitpacked codec\".to_string(),\n            (CodecType::Bitpacked, data.clone()),\n        ),\n        (\n            \"linear codec\".to_string(),\n            (CodecType::Linear, data.clone()),\n        ),\n        (\n            \"blockwise linear codec\".to_string(),\n            (CodecType::BlockwiseLinear, data.clone()),\n        ),\n    ]);\n\n    group.register(\"serialize column_values\", |data| {\n        let mut buffer = Vec::new();\n        serialize_u64_based_column_values(&data.1.as_slice(), &[data.0], &mut buffer).unwrap();\n        black_box(buffer.len());\n    });\n\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_first_vals.rs",
    "content": "use std::sync::Arc;\n\nuse binggan::{InputGroup, black_box};\nuse rand::prelude::*;\nuse tantivy_columnar::column_values::{CodecType, serialize_and_load_u64_based_column_values};\nuse tantivy_columnar::*;\n\nstruct Columns {\n    pub optional: Column,\n    pub full: Column,\n    pub multi: Column,\n}\n\nfn get_test_columns() -> Columns {\n    let data = generate_permutation();\n    let mut dataframe_writer = ColumnarWriter::default();\n    for (idx, val) in data.iter().enumerate() {\n        dataframe_writer.record_numerical(idx as u32, \"full_values\", NumericalValue::U64(*val));\n        if idx % 2 == 0 {\n            dataframe_writer.record_numerical(\n                idx as u32,\n                \"optional_values\",\n                NumericalValue::U64(*val),\n            );\n        }\n        dataframe_writer.record_numerical(idx as u32, \"multi_values\", NumericalValue::U64(*val));\n        dataframe_writer.record_numerical(idx as u32, \"multi_values\", NumericalValue::U64(*val));\n    }\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer\n        .serialize(data.len() as u32, &mut buffer)\n        .unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"optional_values\").unwrap();\n    assert_eq!(cols.len(), 1);\n    let optional = cols[0].open_u64_lenient().unwrap().unwrap();\n    assert_eq!(optional.index.get_cardinality(), Cardinality::Optional);\n\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"full_values\").unwrap();\n    assert_eq!(cols.len(), 1);\n    let column_full = cols[0].open_u64_lenient().unwrap().unwrap();\n    assert_eq!(column_full.index.get_cardinality(), Cardinality::Full);\n\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"multi_values\").unwrap();\n    assert_eq!(cols.len(), 1);\n    let multi = cols[0].open_u64_lenient().unwrap().unwrap();\n    assert_eq!(multi.index.get_cardinality(), Cardinality::Multivalued);\n\n    Columns {\n        optional,\n        full: column_full,\n        multi,\n    }\n}\n\nconst NUM_VALUES: u64 = 100_000;\nfn generate_permutation() -> Vec<u64> {\n    let mut permutation: Vec<u64> = (0u64..NUM_VALUES).collect();\n    permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n    permutation\n}\n\npub fn serialize_and_load(column: &[u64], codec_type: CodecType) -> Arc<dyn ColumnValues<u64>> {\n    serialize_and_load_u64_based_column_values(&column, &[codec_type])\n}\n\nfn main() {\n    let Columns {\n        optional,\n        full,\n        multi,\n    } = get_test_columns();\n\n    let inputs = vec![\n        (\"full\".to_string(), full),\n        (\"optional\".to_string(), optional),\n        (\"multi\".to_string(), multi),\n    ];\n\n    let mut group = InputGroup::new_with_inputs(inputs);\n\n    group.register(\"first_full_scan\", |column| {\n        let mut sum = 0u64;\n        for i in 0..NUM_VALUES as u32 {\n            let val = column.first(i);\n            sum += val.unwrap_or(0);\n        }\n        black_box(sum);\n    });\n\n    group.register(\"first_block_single_calls\", |column| {\n        let mut block: Vec<Option<u64>> = vec![None; 64];\n        let fetch_docids = (0..64).collect::<Vec<_>>();\n        for i in 0..fetch_docids.len() {\n            block[i] = column.first(fetch_docids[i]);\n        }\n        black_box(block[0]);\n    });\n\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_merge.rs",
    "content": "pub mod common;\n\nuse binggan::BenchRunner;\nuse common::{Card, generate_columnar_with_name};\nuse tantivy_columnar::*;\n\nconst NUM_DOCS: u32 = 100_000;\n\nfn main() {\n    let mut inputs = Vec::new();\n\n    let mut add_combo = |card1: Card, card2: Card| {\n        inputs.push((\n            format!(\"merge_{card1}_and_{card2}\"),\n            vec![\n                generate_columnar_with_name(card1, NUM_DOCS, \"price\"),\n                generate_columnar_with_name(card2, NUM_DOCS, \"price\"),\n            ],\n        ));\n    };\n\n    add_combo(Card::Multi, Card::Multi);\n    add_combo(Card::MultiSparse, Card::MultiSparse);\n    add_combo(Card::Dense, Card::Dense);\n    add_combo(Card::Sparse, Card::Sparse);\n    add_combo(Card::Sparse, Card::Dense);\n    add_combo(Card::MultiSparse, Card::Dense);\n    add_combo(Card::MultiSparse, Card::Sparse);\n    add_combo(Card::Multi, Card::Dense);\n    add_combo(Card::Multi, Card::Sparse);\n\n    let mut runner: BenchRunner = BenchRunner::new();\n    let mut group = runner.new_group();\n    for (input_name, columnar_readers) in inputs.iter() {\n        group.register_with_input(\n            input_name,\n            columnar_readers,\n            move |columnar_readers: &Vec<ColumnarReader>| {\n                let mut out = Vec::new();\n                let columnar_readers = columnar_readers.iter().collect::<Vec<_>>();\n                let merge_row_order = StackMergeOrder::stack(&columnar_readers[..]);\n\n                merge_columnar(&columnar_readers, &[], merge_row_order.into(), &mut out).unwrap();\n                Some(out.len() as u64)\n            },\n        );\n    }\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_optional_index.rs",
    "content": "use binggan::{InputGroup, black_box};\nuse rand::rngs::StdRng;\nuse rand::{Rng, SeedableRng};\nuse tantivy_columnar::column_index::{OptionalIndex, Set};\n\nconst TOTAL_NUM_VALUES: u32 = 1_000_000;\n\nfn gen_optional_index(fill_ratio: f64) -> OptionalIndex {\n    let mut rng: StdRng = StdRng::from_seed([1u8; 32]);\n    let vals: Vec<u32> = (0..TOTAL_NUM_VALUES)\n        .map(|_| rng.random_bool(fill_ratio))\n        .enumerate()\n        .filter(|(_pos, val)| *val)\n        .map(|(pos, _)| pos as u32)\n        .collect();\n    OptionalIndex::for_test(TOTAL_NUM_VALUES, &vals)\n}\n\nfn random_range_iterator(\n    start: u32,\n    end: u32,\n    avg_step_size: u32,\n    avg_deviation: u32,\n) -> impl Iterator<Item = u32> {\n    let mut rng: StdRng = StdRng::from_seed([1u8; 32]);\n    let mut current = start;\n    std::iter::from_fn(move || {\n        current += rng.random_range(avg_step_size - avg_deviation..=avg_step_size + avg_deviation);\n        if current >= end { None } else { Some(current) }\n    })\n}\n\nfn n_percent_step_iterator(percent: f32, num_values: u32) -> impl Iterator<Item = u32> {\n    let ratio = percent / 100.0;\n    let step_size = (1f32 / ratio) as u32;\n    let deviation = step_size - 1;\n    random_range_iterator(0, num_values, step_size, deviation)\n}\n\nfn walk_over_data(codec: &OptionalIndex, avg_step_size: u32) -> Option<u32> {\n    walk_over_data_from_positions(\n        codec,\n        random_range_iterator(0, TOTAL_NUM_VALUES, avg_step_size, 0),\n    )\n}\n\nfn walk_over_data_from_positions(\n    codec: &OptionalIndex,\n    positions: impl Iterator<Item = u32>,\n) -> Option<u32> {\n    let mut dense_idx: Option<u32> = None;\n    for idx in positions {\n        dense_idx = dense_idx.or(codec.rank_if_exists(idx));\n    }\n    dense_idx\n}\n\nfn main() {\n    // Build separate inputs for each fill ratio.\n    let inputs: Vec<(String, OptionalIndex)> = vec![\n        (\"fill=1%\".to_string(), gen_optional_index(0.01)),\n        (\"fill=5%\".to_string(), gen_optional_index(0.05)),\n        (\"fill=10%\".to_string(), gen_optional_index(0.10)),\n        (\"fill=50%\".to_string(), gen_optional_index(0.50)),\n        (\"fill=90%\".to_string(), gen_optional_index(0.90)),\n    ];\n\n    let mut group: InputGroup<OptionalIndex> = InputGroup::new_with_inputs(inputs);\n\n    // Translate orig->codec (rank_if_exists) with sampling\n    group.register(\"orig_to_codec_10pct_hit\", |codec: &OptionalIndex| {\n        black_box(walk_over_data(codec, 100));\n    });\n    group.register(\"orig_to_codec_1pct_hit\", |codec: &OptionalIndex| {\n        black_box(walk_over_data(codec, 1000));\n    });\n    group.register(\"orig_to_codec_full_scan\", |codec: &OptionalIndex| {\n        black_box(walk_over_data_from_positions(codec, 0..TOTAL_NUM_VALUES));\n    });\n\n    // Translate codec->orig (select/select_batch) on sampled ranks\n    fn bench_translate_codec_to_orig_util(codec: &OptionalIndex, percent_hit: f32) {\n        let num_non_nulls = codec.num_non_nulls();\n        let idxs: Vec<u32> = if percent_hit == 100.0f32 {\n            (0..num_non_nulls).collect()\n        } else {\n            n_percent_step_iterator(percent_hit, num_non_nulls).collect()\n        };\n        let mut output = vec![0u32; idxs.len()];\n        output.copy_from_slice(&idxs[..]);\n        codec.select_batch(&mut output);\n        black_box(output);\n    }\n\n    group.register(\"codec_to_orig_0.005pct_hit\", |codec: &OptionalIndex| {\n        bench_translate_codec_to_orig_util(codec, 0.005);\n    });\n    group.register(\"codec_to_orig_10pct_hit\", |codec: &OptionalIndex| {\n        bench_translate_codec_to_orig_util(codec, 10.0);\n    });\n    group.register(\"codec_to_orig_full_scan\", |codec: &OptionalIndex| {\n        bench_translate_codec_to_orig_util(codec, 100.0);\n    });\n\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_values_u128.rs",
    "content": "use std::ops::RangeInclusive;\nuse std::sync::Arc;\n\nuse binggan::{InputGroup, black_box};\nuse common::OwnedBytes;\nuse rand::rngs::StdRng;\nuse rand::seq::SliceRandom;\nuse rand::{Rng, SeedableRng, random};\nuse tantivy_columnar::ColumnValues;\n\n// TODO does this make sense for IPv6 ?\nfn generate_random() -> Vec<u64> {\n    let mut permutation: Vec<u64> = (0u64..100_000u64)\n        .map(|el| el + random::<u16>() as u64)\n        .collect();\n    permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n    permutation\n}\n\nfn get_u128_column_random() -> Arc<dyn ColumnValues<u128>> {\n    let permutation = generate_random();\n    let permutation = permutation.iter().map(|el| *el as u128).collect::<Vec<_>>();\n    get_u128_column_from_data(&permutation)\n}\n\nfn get_u128_column_from_data(data: &[u128]) -> Arc<dyn ColumnValues<u128>> {\n    let mut out = vec![];\n    tantivy_columnar::column_values::serialize_column_values_u128(&data, &mut out).unwrap();\n    let out = OwnedBytes::new(out);\n    tantivy_columnar::column_values::open_u128_mapped::<u128>(out).unwrap()\n}\n\nconst FIFTY_PERCENT_RANGE: RangeInclusive<u64> = 1..=50;\nconst SINGLE_ITEM: u64 = 90;\nconst SINGLE_ITEM_RANGE: RangeInclusive<u64> = 90..=90;\n\nfn get_data_50percent_item() -> Vec<u128> {\n    let mut rng = StdRng::from_seed([1u8; 32]);\n\n    let mut data = vec![];\n    for _ in 0..300_000 {\n        let val = rng.random_range(1..=100);\n        data.push(val);\n    }\n    data.push(SINGLE_ITEM);\n    data.shuffle(&mut rng);\n    data.iter().map(|el| *el as u128).collect::<Vec<_>>()\n}\n\nfn main() {\n    let data = get_data_50percent_item();\n    let column_range = get_u128_column_from_data(&data);\n    let column_random = get_u128_column_random();\n\n    struct Inputs {\n        data: Vec<u128>,\n        column_range: Arc<dyn ColumnValues<u128>>,\n        column_random: Arc<dyn ColumnValues<u128>>,\n    }\n\n    let inputs = Inputs {\n        data,\n        column_range,\n        column_random,\n    };\n    let mut group: InputGroup<Inputs> =\n        InputGroup::new_with_inputs(vec![(\"u128 benches\".to_string(), inputs)]);\n\n    group.register(\n        \"intfastfield_getrange_u128_50percent_hit\",\n        |inp: &Inputs| {\n            let mut positions = Vec::new();\n            inp.column_range.get_row_ids_for_value_range(\n                *FIFTY_PERCENT_RANGE.start() as u128..=*FIFTY_PERCENT_RANGE.end() as u128,\n                0..inp.data.len() as u32,\n                &mut positions,\n            );\n            black_box(positions.len());\n        },\n    );\n\n    group.register(\"intfastfield_getrange_u128_single_hit\", |inp: &Inputs| {\n        let mut positions = Vec::new();\n        inp.column_range.get_row_ids_for_value_range(\n            *SINGLE_ITEM_RANGE.start() as u128..=*SINGLE_ITEM_RANGE.end() as u128,\n            0..inp.data.len() as u32,\n            &mut positions,\n        );\n        black_box(positions.len());\n    });\n\n    group.register(\"intfastfield_getrange_u128_hit_all\", |inp: &Inputs| {\n        let mut positions = Vec::new();\n        inp.column_range.get_row_ids_for_value_range(\n            0..=u128::MAX,\n            0..inp.data.len() as u32,\n            &mut positions,\n        );\n        black_box(positions.len());\n    });\n\n    group.register(\"intfastfield_scan_all_fflookup_u128\", |inp: &Inputs| {\n        let mut a = 0u128;\n        for i in 0u64..inp.column_random.num_vals() as u64 {\n            a += inp.column_random.get_val(i as u32);\n        }\n        black_box(a);\n    });\n\n    group.register(\"intfastfield_jumpy_stride5_u128\", |inp: &Inputs| {\n        let n = inp.column_random.num_vals();\n        let mut a = 0u128;\n        for i in (0..n / 5).map(|val| val * 5) {\n            a += inp.column_random.get_val(i);\n        }\n        black_box(a);\n    });\n\n    group.run();\n}\n"
  },
  {
    "path": "columnar/benches/bench_values_u64.rs",
    "content": "use std::ops::RangeInclusive;\nuse std::sync::Arc;\n\nuse binggan::{InputGroup, black_box};\nuse rand::prelude::*;\nuse tantivy_columnar::column_values::{CodecType, serialize_and_load_u64_based_column_values};\nuse tantivy_columnar::*;\n\n// Warning: this generates the same permutation at each call\nfn generate_permutation() -> Vec<u64> {\n    let mut permutation: Vec<u64> = (0u64..100_000u64).collect();\n    permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n    permutation\n}\n\n// Warning: this generates the same permutation at each call\nfn generate_permutation_gcd() -> Vec<u64> {\n    let mut permutation: Vec<u64> = (1u64..100_000u64).map(|el| el * 1000).collect();\n    permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n    permutation\n}\n\npub fn serialize_and_load(column: &[u64], codec_type: CodecType) -> Arc<dyn ColumnValues<u64>> {\n    serialize_and_load_u64_based_column_values(&column, &[codec_type])\n}\n\nconst FIFTY_PERCENT_RANGE: RangeInclusive<u64> = 1..=50;\nconst SINGLE_ITEM: u64 = 90;\nconst SINGLE_ITEM_RANGE: RangeInclusive<u64> = 90..=90;\nconst ONE_PERCENT_ITEM_RANGE: RangeInclusive<u64> = 49..=49;\n\nfn get_data_50percent_item() -> Vec<u128> {\n    let mut rng = StdRng::from_seed([1u8; 32]);\n\n    let mut data = vec![];\n    for _ in 0..300_000 {\n        let val = rng.random_range(1..=100);\n        data.push(val);\n    }\n    data.push(SINGLE_ITEM);\n\n    data.shuffle(&mut rng);\n    data.iter().map(|el| *el as u128).collect::<Vec<_>>()\n}\n\ntype VecCol = (Vec<u64>, Arc<dyn ColumnValues<u64>>);\n\nfn bench_access() {\n    let permutation = generate_permutation();\n    let column_perm: Arc<dyn ColumnValues<u64>> =\n        serialize_and_load(&permutation, CodecType::Bitpacked);\n\n    let permutation_gcd = generate_permutation_gcd();\n    let column_perm_gcd: Arc<dyn ColumnValues<u64>> =\n        serialize_and_load(&permutation_gcd, CodecType::Bitpacked);\n\n    let mut group: InputGroup<VecCol> = InputGroup::new_with_inputs(vec![\n        (\n            \"access\".to_string(),\n            (permutation.clone(), column_perm.clone()),\n        ),\n        (\n            \"access_gcd\".to_string(),\n            (permutation_gcd.clone(), column_perm_gcd.clone()),\n        ),\n    ]);\n\n    group.register(\"stride7_vec\", |inp: &VecCol| {\n        let n = inp.0.len();\n        let mut a = 0u64;\n        for i in (0..n / 7).map(|val| val * 7) {\n            a += inp.0[i];\n        }\n        black_box(a);\n    });\n\n    group.register(\"fullscan_vec\", |inp: &VecCol| {\n        let mut a = 0u64;\n        for i in 0..inp.0.len() {\n            a += inp.0[i];\n        }\n        black_box(a);\n    });\n\n    group.register(\"stride7_column_values\", |inp: &VecCol| {\n        let n = inp.1.num_vals() as usize;\n        let mut a = 0u64;\n        for i in (0..n / 7).map(|val| val * 7) {\n            a += inp.1.get_val(i as u32);\n        }\n        black_box(a);\n    });\n\n    group.register(\"fullscan_column_values\", |inp: &VecCol| {\n        let mut a = 0u64;\n        let n = inp.1.num_vals() as usize;\n        for i in 0..n {\n            a += inp.1.get_val(i as u32);\n        }\n        black_box(a);\n    });\n\n    group.run();\n}\n\nfn bench_range() {\n    let data_50 = get_data_50percent_item();\n    let data_u64 = data_50.iter().map(|el| *el as u64).collect::<Vec<_>>();\n    let column_data: Arc<dyn ColumnValues<u64>> =\n        serialize_and_load(&data_u64, CodecType::Bitpacked);\n\n    let mut group: InputGroup<Arc<dyn ColumnValues<u64>>> =\n        InputGroup::new_with_inputs(vec![(\"dist_50pct_item\".to_string(), column_data.clone())]);\n\n    group.register(\n        \"fastfield_getrange_u64_50percent_hit\",\n        |col: &Arc<dyn ColumnValues<u64>>| {\n            let mut positions = Vec::new();\n            col.get_row_ids_for_value_range(FIFTY_PERCENT_RANGE, 0..col.num_vals(), &mut positions);\n            black_box(positions.len());\n        },\n    );\n\n    group.register(\n        \"fastfield_getrange_u64_1percent_hit\",\n        |col: &Arc<dyn ColumnValues<u64>>| {\n            let mut positions = Vec::new();\n            col.get_row_ids_for_value_range(\n                ONE_PERCENT_ITEM_RANGE,\n                0..col.num_vals(),\n                &mut positions,\n            );\n            black_box(positions.len());\n        },\n    );\n\n    group.register(\n        \"fastfield_getrange_u64_single_hit\",\n        |col: &Arc<dyn ColumnValues<u64>>| {\n            let mut positions = Vec::new();\n            col.get_row_ids_for_value_range(SINGLE_ITEM_RANGE, 0..col.num_vals(), &mut positions);\n            black_box(positions.len());\n        },\n    );\n\n    group.register(\n        \"fastfield_getrange_u64_hit_all\",\n        |col: &Arc<dyn ColumnValues<u64>>| {\n            let mut positions = Vec::new();\n            col.get_row_ids_for_value_range(0..=u64::MAX, 0..col.num_vals(), &mut positions);\n            black_box(positions.len());\n        },\n    );\n\n    group.run();\n}\n\nfn main() {\n    bench_access();\n    bench_range();\n}\n"
  },
  {
    "path": "columnar/benches/common.rs",
    "content": "extern crate tantivy_columnar;\n\nuse core::fmt;\nuse std::fmt::{Display, Formatter};\n\nuse tantivy_columnar::{ColumnarReader, ColumnarWriter};\n\npub enum Card {\n    MultiSparse,\n    Multi,\n    Sparse,\n    Dense,\n    Full,\n}\nimpl Display for Card {\n    fn fmt(&self, f: &mut Formatter) -> fmt::Result {\n        match self {\n            Card::MultiSparse => write!(f, \"multi sparse 1/13\"),\n            Card::Multi => write!(f, \"multi 2x\"),\n            Card::Sparse => write!(f, \"sparse 1/13\"),\n            Card::Dense => write!(f, \"dense 1/12\"),\n            Card::Full => write!(f, \"full\"),\n        }\n    }\n}\npub fn generate_columnar_with_name(card: Card, num_docs: u32, column_name: &str) -> ColumnarReader {\n    let mut columnar_writer = ColumnarWriter::default();\n\n    if let Card::MultiSparse = card {\n        columnar_writer.record_numerical(0, column_name, 10u64);\n        columnar_writer.record_numerical(0, column_name, 10u64);\n    }\n\n    for i in 0..num_docs {\n        match card {\n            Card::MultiSparse | Card::Sparse => {\n                if i % 13 == 0 {\n                    columnar_writer.record_numerical(i, column_name, i as u64);\n                }\n            }\n            Card::Dense => {\n                if i % 12 == 0 {\n                    columnar_writer.record_numerical(i, column_name, i as u64);\n                }\n            }\n            Card::Full => {\n                columnar_writer.record_numerical(i, column_name, i as u64);\n            }\n            Card::Multi => {\n                columnar_writer.record_numerical(i, column_name, i as u64);\n                columnar_writer.record_numerical(i, column_name, i as u64);\n            }\n        }\n    }\n\n    let mut wrt: Vec<u8> = Vec::new();\n    columnar_writer.serialize(num_docs, &mut wrt).unwrap();\n    ColumnarReader::open(wrt).unwrap()\n}\n"
  },
  {
    "path": "columnar/columnar-cli/Cargo.toml",
    "content": "[package]\nname = \"tantivy-columnar-cli\"\nversion = \"0.1.0\"\nedition = \"2021\"\nlicense = \"MIT\"\n\n[dependencies]\ncolumnar = {path=\"../\", package=\"tantivy-columnar\"}\nserde_json = \"1\"\nserde_json_borrow = {git=\"https://github.com/PSeitz/serde_json_borrow/\"}\n\n[workspace]\nmembers = []\n\n[profile.release]\ndebug = true\n"
  },
  {
    "path": "columnar/columnar-cli/src/main.rs",
    "content": "use columnar::ColumnarWriter;\nuse columnar::NumericalValue;\nuse serde_json_borrow;\nuse std::fs::File;\nuse std::io;\nuse std::io::BufRead;\nuse std::io::BufReader;\nuse std::time::Instant;\n\n#[derive(Default)]\nstruct JsonStack {\n    path: String,\n    stack: Vec<usize>,\n}\n\nimpl JsonStack {\n    fn push(&mut self, seg: &str) {\n        let len = self.path.len();\n        self.stack.push(len);\n        self.path.push('.');\n        self.path.push_str(seg);\n    }\n\n    fn pop(&mut self) {\n        if let Some(len) = self.stack.pop() {\n            self.path.truncate(len);\n        }\n    }\n\n    fn path(&self) -> &str {\n        &self.path[1..]\n    }\n}\n\nfn append_json_to_columnar(\n    doc: u32,\n    json_value: &serde_json_borrow::Value,\n    columnar: &mut ColumnarWriter,\n    stack: &mut JsonStack,\n) -> usize {\n    let mut count = 0;\n    match json_value {\n        serde_json_borrow::Value::Null => {}\n        serde_json_borrow::Value::Bool(val) => {\n            columnar.record_numerical(\n                doc,\n                stack.path(),\n                NumericalValue::from(if *val { 1u64 } else { 0u64 }),\n            );\n            count += 1;\n        }\n        serde_json_borrow::Value::Number(num) => {\n            let numerical_value: NumericalValue = if let Some(num_i64) = num.as_i64() {\n                num_i64.into()\n            } else if let Some(num_u64) = num.as_u64() {\n                num_u64.into()\n            } else if let Some(num_f64) = num.as_f64() {\n                num_f64.into()\n            } else {\n                panic!();\n            };\n            count += 1;\n            columnar.record_numerical(\n                doc,\n                stack.path(),\n                numerical_value,\n            );\n        }\n        serde_json_borrow::Value::Str(msg) => {\n            columnar.record_str(\n                doc,\n                stack.path(),\n                msg,\n            );\n            count += 1;\n        },\n        serde_json_borrow::Value::Array(vals) => {\n            for val in vals {\n                count += append_json_to_columnar(doc, val, columnar, stack);\n            }\n        },\n        serde_json_borrow::Value::Object(json_map) => {\n            for (child_key, child_val) in json_map {\n                stack.push(child_key);\n                count += append_json_to_columnar(doc, child_val, columnar, stack);\n                stack.pop();\n            }\n        },\n    }\n    count\n}\n\nfn main() -> io::Result<()> {\n    let file = File::open(\"gh_small.json\")?;\n    let mut reader = BufReader::new(file);\n    let mut line = String::with_capacity(100);\n    let mut columnar = columnar::ColumnarWriter::default();\n    let mut doc = 0;\n    let start = Instant::now();\n    let mut stack = JsonStack::default();\n    let mut total_count = 0;\n\n    let start_build = Instant::now();\n    loop {\n        line.clear();\n        let len = reader.read_line(&mut line)?;\n        if len == 0 {\n            break;\n        }\n        let Ok(json_value) = serde_json::from_str::<serde_json_borrow::Value>(&line) else { continue; };\n        total_count += append_json_to_columnar(doc, &json_value, &mut columnar, &mut stack);\n        doc += 1;\n    }\n    println!(\"Build in {:?}\", start_build.elapsed());\n\n    println!(\"value count {total_count}\");\n\n    let mut buffer = Vec::new();\n    let start_serialize = Instant::now();\n    columnar.serialize(doc, None, &mut buffer)?;\n    println!(\"Serialized in {:?}\", start_serialize.elapsed());\n    println!(\"num docs: {doc}, {:?}\", start.elapsed());\n    println!(\"buffer len {} MB\", buffer.len() / 1_000_000);\n    let columnar = columnar::ColumnarReader::open(buffer)?;\n    for (column_name, dynamic_column) in columnar.list_columns()? {\n        let num_bytes = dynamic_column.num_bytes();\n        let typ = dynamic_column.column_type();\n        if num_bytes > 1_000_000 {\n            println!(\"{column_name} {typ:?}  {} KB\", num_bytes / 1_000);\n        }\n    }\n    println!(\"{} columns\", columnar.num_columns());\n    Ok(())\n}\n"
  },
  {
    "path": "columnar/columnar-cli-inspect/Cargo.toml",
    "content": "[package]\nname = \"tantivy-columnar-inspect\"\nversion = \"0.1.0\"\nedition = \"2021\"\nlicense = \"MIT\"\n\n[dependencies]\ntantivy = {path=\"../..\", package=\"tantivy\"}\ncolumnar = {path=\"../\", package=\"tantivy-columnar\"}\ncommon = {path=\"../../common\", package=\"tantivy-common\"}\n\n[workspace]\nmembers = []\n\n[profile.release]\ndebug = true\n#debug-assertions = true\n#overflow-checks = true\n"
  },
  {
    "path": "columnar/columnar-cli-inspect/src/main.rs",
    "content": "use columnar::ColumnarReader;\nuse common::file_slice::{FileSlice, WrapFile};\nuse std::io;\nuse std::path::Path;\nuse tantivy::directory::footer::Footer;\n\nfn main() -> io::Result<()> {\n    println!(\"Opens a columnar file written by tantivy and validates it.\");\n    let path = std::env::args().nth(1).unwrap();\n\n    let path = Path::new(&path);\n    println!(\"Reading {:?}\", path);\n    let _reader = open_and_validate_columnar(path.to_str().unwrap())?;\n\n    Ok(())\n}\n\npub fn validate_columnar_reader(reader: &ColumnarReader) {\n    let num_rows = reader.num_rows();\n    println!(\"num_rows: {}\", num_rows);\n    let columns = reader.list_columns().unwrap();\n    println!(\"num columns: {:?}\", columns.len());\n    for (col_name, dynamic_column_handle) in columns {\n        let col = dynamic_column_handle.open().unwrap();\n        match col {\n            columnar::DynamicColumn::Bool(_)\n            | columnar::DynamicColumn::I64(_)\n            | columnar::DynamicColumn::U64(_)\n            | columnar::DynamicColumn::F64(_)\n            | columnar::DynamicColumn::IpAddr(_)\n            | columnar::DynamicColumn::DateTime(_)\n            | columnar::DynamicColumn::Bytes(_) => {}\n            columnar::DynamicColumn::Str(str_column) => {\n                let num_vals = str_column.ords().values.num_vals();\n                let num_terms_dict = str_column.num_terms() as u64;\n                let max_ord = str_column.ords().values.iter().max().unwrap_or_default();\n                println!(\"{col_name:35}  num_vals {num_vals:10} \\t num_terms_dict {num_terms_dict:8} max_ord: {max_ord:8}\",);\n                for ord in str_column.ords().values.iter() {\n                    assert!(ord < num_terms_dict);\n                }\n            }\n        }\n    }\n}\n\n/// Opens a columnar file that was written by tantivy and validates it.\npub fn open_and_validate_columnar(path: &str) -> io::Result<ColumnarReader> {\n    let wrap_file = WrapFile::new(std::fs::File::open(path)?)?;\n    let slice = FileSlice::new(std::sync::Arc::new(wrap_file));\n    let (_footer, slice) = Footer::extract_footer(slice.clone()).unwrap();\n    let reader = ColumnarReader::open(slice).unwrap();\n    validate_columnar_reader(&reader);\n    Ok(reader)\n}\n"
  },
  {
    "path": "columnar/src/TODO.md",
    "content": "# zero to one\n\n* revisit line codec\n* add columns from schema on merge\n* Plugging JSON\n* replug examples\n* move datetime to quickwit common\n* switch to nanos\n* reintroduce the gcd map.\n\n# Perf and Size\n* remove alloc in `ord_to_term`\n+ multivaued range queries restart from the beginning all of the time.\n* re-add ZSTD compression for dictionaries\nno systematic monotonic mapping\nconsider removing multilinear\nf32?\nadhoc solution for bool?\nadd metrics helper for aggregate. sum(row_id)\nreview inline absence/presence\nimprov perf of select using PDEP\ncompare with roaring bitmap/elias fano etc etc.\nSIMD range? (see blog post)\nAdd alignment?\nConsider another codec to bridge the gap between few and 5k elements\n\n# Cleanup and rationalization\nin benchmark, unify percent vs ratio, f32 vs f64.\ninvestigate if should have better errors? io::Error is overused at the moment.\nrename rank/select in unit tests\nReview the public API via cargo doc\ngo through TODOs\nremove all  doc_id occurrences -> row_id\nuse the rank & select naming in unit tests branch.\nmulti-linear -> blockwise\nlinear codec -> simply a multiplication for the index column\nrename columnar to something more explicit, like column_dictionary or columnar_table\nrename fastfield -> column\ndocument changes\nrationalization FastFieldValue, HasColumnType\nisolate u128_based and uniform naming\n\n# Other\nfix enhance column-cli\n\n# Santa Claus\nautodetect datetime ipaddr, plug customizable tokenizer.\n"
  },
  {
    "path": "columnar/src/block_accessor.rs",
    "content": "use std::cmp::Ordering;\n\nuse crate::{Column, DocId, RowId};\n\n#[derive(Debug, Default, Clone)]\npub struct ColumnBlockAccessor<T> {\n    val_cache: Vec<T>,\n    docid_cache: Vec<DocId>,\n    missing_docids_cache: Vec<DocId>,\n    row_id_cache: Vec<RowId>,\n}\n\nimpl<T: PartialOrd + Copy + std::fmt::Debug + Send + Sync + 'static + Default>\n    ColumnBlockAccessor<T>\n{\n    #[inline]\n    pub fn fetch_block<'a>(&'a mut self, docs: &'a [u32], accessor: &Column<T>) {\n        if accessor.index.get_cardinality().is_full() {\n            self.val_cache.resize(docs.len(), T::default());\n            accessor.values.get_vals(docs, &mut self.val_cache);\n        } else {\n            self.docid_cache.clear();\n            self.row_id_cache.clear();\n            accessor.row_ids_for_docs(docs, &mut self.docid_cache, &mut self.row_id_cache);\n            self.val_cache.resize(self.row_id_cache.len(), T::default());\n            accessor\n                .values\n                .get_vals(&self.row_id_cache, &mut self.val_cache);\n        }\n    }\n    #[inline]\n    pub fn fetch_block_with_missing(\n        &mut self,\n        docs: &[u32],\n        accessor: &Column<T>,\n        missing: Option<T>,\n    ) {\n        self.fetch_block(docs, accessor);\n        // no missing values\n        if accessor.index.get_cardinality().is_full() {\n            return;\n        }\n        let Some(missing) = missing else {\n            return;\n        };\n\n        // We can compare docid_cache length with docs to find missing docs\n        // For multi value columns we can't rely on the length and always need to scan\n        if accessor.index.get_cardinality().is_multivalue() || docs.len() != self.docid_cache.len()\n        {\n            self.missing_docids_cache.clear();\n            find_missing_docs(docs, &self.docid_cache, |doc| {\n                self.missing_docids_cache.push(doc);\n                self.val_cache.push(missing);\n            });\n            self.docid_cache\n                .extend_from_slice(&self.missing_docids_cache);\n        }\n    }\n\n    #[inline]\n    pub fn iter_vals(&self) -> impl Iterator<Item = T> + '_ {\n        self.val_cache.iter().cloned()\n    }\n\n    #[inline]\n    /// Returns an iterator over the docids and values\n    /// The passed in `docs` slice needs to be the same slice that was passed to `fetch_block` or\n    /// `fetch_block_with_missing`.\n    ///\n    /// The docs is used if the column is full (each docs has exactly one value), otherwise the\n    /// internal docid vec is used for the iterator, which e.g. may contain duplicate docs.\n    pub fn iter_docid_vals<'a>(\n        &'a self,\n        docs: &'a [u32],\n        accessor: &Column<T>,\n    ) -> impl Iterator<Item = (DocId, T)> + 'a + use<'a, T> {\n        if accessor.index.get_cardinality().is_full() {\n            docs.iter().cloned().zip(self.val_cache.iter().cloned())\n        } else {\n            self.docid_cache\n                .iter()\n                .cloned()\n                .zip(self.val_cache.iter().cloned())\n        }\n    }\n}\n\n/// Given two sorted lists of docids `docs` and `hits`, hits is a subset of `docs`.\n/// Return all docs that are not in `hits`.\nfn find_missing_docs<F>(docs: &[u32], hits: &[u32], mut callback: F)\nwhere F: FnMut(u32) {\n    let mut docs_iter = docs.iter();\n    let mut hits_iter = hits.iter();\n\n    let mut doc = docs_iter.next();\n    let mut hit = hits_iter.next();\n\n    while let (Some(&current_doc), Some(&current_hit)) = (doc, hit) {\n        match current_doc.cmp(&current_hit) {\n            Ordering::Less => {\n                callback(current_doc);\n                doc = docs_iter.next();\n            }\n            Ordering::Equal => {\n                doc = docs_iter.next();\n                hit = hits_iter.next();\n            }\n            Ordering::Greater => {\n                hit = hits_iter.next();\n            }\n        }\n    }\n\n    while let Some(&current_doc) = doc {\n        callback(current_doc);\n        doc = docs_iter.next();\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_find_missing_docs() {\n        let docs: Vec<u32> = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];\n        let hits: Vec<u32> = vec![2, 4, 6, 8, 10];\n\n        let mut missing_docs: Vec<u32> = Vec::new();\n\n        find_missing_docs(&docs, &hits, |missing_doc| {\n            missing_docs.push(missing_doc);\n        });\n\n        assert_eq!(missing_docs, vec![1, 3, 5, 7, 9]);\n    }\n\n    #[test]\n    fn test_find_missing_docs_empty() {\n        let docs: Vec<u32> = Vec::new();\n        let hits: Vec<u32> = vec![2, 4, 6, 8, 10];\n\n        let mut missing_docs: Vec<u32> = Vec::new();\n\n        find_missing_docs(&docs, &hits, |missing_doc| {\n            missing_docs.push(missing_doc);\n        });\n\n        assert_eq!(missing_docs, Vec::<u32>::new());\n    }\n\n    #[test]\n    fn test_find_missing_docs_all_missing() {\n        let docs: Vec<u32> = vec![1, 2, 3, 4, 5];\n        let hits: Vec<u32> = Vec::new();\n\n        let mut missing_docs: Vec<u32> = Vec::new();\n\n        find_missing_docs(&docs, &hits, |missing_doc| {\n            missing_docs.push(missing_doc);\n        });\n\n        assert_eq!(missing_docs, vec![1, 2, 3, 4, 5]);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column/dictionary_encoded.rs",
    "content": "use std::ops::Deref;\nuse std::sync::Arc;\nuse std::{fmt, io};\n\nuse sstable::{Dictionary, VoidSSTable};\n\nuse crate::RowId;\nuse crate::column::Column;\n\n/// Dictionary encoded column.\n///\n/// The column simply gives access to a regular u64-column that, in\n/// which the values are term-ordinals.\n///\n/// These ordinals are ids uniquely identify the bytes that are stored in\n/// the column. These ordinals are small, and sorted in the same order\n/// as the term_ord_column.\n#[derive(Clone)]\npub struct BytesColumn {\n    pub(crate) dictionary: Arc<Dictionary<VoidSSTable>>,\n    pub(crate) term_ord_column: Column<u64>,\n}\n\nimpl fmt::Debug for BytesColumn {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        f.debug_struct(\"BytesColumn\")\n            .field(\"term_ord_column\", &self.term_ord_column)\n            .finish()\n    }\n}\n\nimpl BytesColumn {\n    pub fn empty(num_docs: u32) -> BytesColumn {\n        BytesColumn {\n            dictionary: Arc::new(Dictionary::empty()),\n            term_ord_column: Column::build_empty_column(num_docs),\n        }\n    }\n\n    /// Fills the given `output` buffer with the term associated to the ordinal `ord`.\n    ///\n    /// Returns `false` if the term does not exist (e.g. `term_ord` is greater or equal to the\n    /// overll number of terms).\n    pub fn ord_to_bytes(&self, ord: u64, output: &mut Vec<u8>) -> io::Result<bool> {\n        self.dictionary.ord_to_term(ord, output)\n    }\n\n    /// Returns the number of rows in the column.\n    pub fn num_rows(&self) -> RowId {\n        self.term_ord_column.num_docs()\n    }\n\n    pub fn term_ords(&self, row_id: RowId) -> impl Iterator<Item = u64> + '_ {\n        self.term_ord_column.values_for_doc(row_id)\n    }\n\n    /// Returns the column of ordinals\n    pub fn ords(&self) -> &Column<u64> {\n        &self.term_ord_column\n    }\n\n    pub fn num_terms(&self) -> usize {\n        self.dictionary.num_terms()\n    }\n\n    pub fn dictionary(&self) -> &Dictionary<VoidSSTable> {\n        self.dictionary.as_ref()\n    }\n}\n\n#[derive(Clone)]\npub struct StrColumn(BytesColumn);\n\nimpl fmt::Debug for StrColumn {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"{:?}\", self.term_ord_column)\n    }\n}\n\nimpl From<StrColumn> for BytesColumn {\n    fn from(str_column: StrColumn) -> BytesColumn {\n        str_column.0\n    }\n}\n\nimpl StrColumn {\n    pub fn wrap(bytes_column: BytesColumn) -> StrColumn {\n        StrColumn(bytes_column)\n    }\n\n    pub fn dictionary(&self) -> &Dictionary<VoidSSTable> {\n        self.0.dictionary.as_ref()\n    }\n\n    /// Fills the buffer\n    pub fn ord_to_str(&self, term_ord: u64, output: &mut String) -> io::Result<bool> {\n        unsafe {\n            let buf = output.as_mut_vec();\n            if !self.0.dictionary.ord_to_term(term_ord, buf)? {\n                return Ok(false);\n            }\n            // TODO consider remove checks if it hurts performance.\n            if std::str::from_utf8(buf.as_slice()).is_err() {\n                buf.clear();\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    \"Not valid utf-8\",\n                ));\n            }\n        }\n        Ok(true)\n    }\n}\n\nimpl Deref for StrColumn {\n    type Target = BytesColumn;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n"
  },
  {
    "path": "columnar/src/column/mod.rs",
    "content": "mod dictionary_encoded;\nmod serialize;\n\nuse std::fmt::{self, Debug};\nuse std::io::Write;\nuse std::ops::{Range, RangeInclusive};\nuse std::sync::Arc;\n\nuse common::BinarySerializable;\npub use dictionary_encoded::{BytesColumn, StrColumn};\npub use serialize::{\n    open_column_bytes, open_column_str, open_column_u64, open_column_u128,\n    open_column_u128_as_compact_u64, serialize_column_mappable_to_u64,\n    serialize_column_mappable_to_u128,\n};\n\nuse crate::column_index::{ColumnIndex, Set};\nuse crate::column_values::monotonic_mapping::StrictlyMonotonicMappingToInternal;\nuse crate::column_values::{ColumnValues, monotonic_map_column};\nuse crate::{Cardinality, DocId, EmptyColumnValues, MonotonicallyMappableToU64, RowId};\n\n#[derive(Clone)]\npub struct Column<T = u64> {\n    pub index: ColumnIndex,\n    pub values: Arc<dyn ColumnValues<T>>,\n}\n\nimpl<T: Debug + PartialOrd + Send + Sync + Copy + 'static> Debug for Column<T> {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        let num_docs = self.num_docs();\n        let entries = (0..num_docs)\n            .map(|i| (i, self.values_for_doc(i).collect::<Vec<_>>()))\n            .filter(|(_, vals)| !vals.is_empty());\n        f.debug_map().entries(entries).finish()\n    }\n}\n\nimpl<T: PartialOrd + Default> Column<T> {\n    pub fn build_empty_column(num_docs: u32) -> Column<T> {\n        Column {\n            index: ColumnIndex::Empty { num_docs },\n            values: Arc::new(EmptyColumnValues),\n        }\n    }\n}\n\nimpl<T: MonotonicallyMappableToU64> Column<T> {\n    pub fn to_u64_monotonic(self) -> Column<u64> {\n        let values = Arc::new(monotonic_map_column(\n            self.values,\n            StrictlyMonotonicMappingToInternal::<T>::new(),\n        ));\n        Column {\n            index: self.index,\n            values,\n        }\n    }\n}\n\nimpl<T: PartialOrd + Copy + Debug + Send + Sync + 'static> Column<T> {\n    #[inline]\n    pub fn get_cardinality(&self) -> Cardinality {\n        self.index.get_cardinality()\n    }\n\n    pub fn num_docs(&self) -> RowId {\n        match &self.index {\n            ColumnIndex::Empty { num_docs } => *num_docs,\n            ColumnIndex::Full => self.values.num_vals(),\n            ColumnIndex::Optional(optional_index) => optional_index.num_docs(),\n            ColumnIndex::Multivalued(col_index) => {\n                // The multivalued index contains all value start row_id,\n                // and one extra value at the end with the overall number of rows.\n                col_index.num_docs()\n            }\n        }\n    }\n\n    pub fn min_value(&self) -> T {\n        self.values.min_value()\n    }\n\n    pub fn max_value(&self) -> T {\n        self.values.max_value()\n    }\n\n    #[inline]\n    pub fn first(&self, doc_id: DocId) -> Option<T> {\n        self.values_for_doc(doc_id).next()\n    }\n\n    /// Load the first value for each docid in the provided slice.\n    #[inline]\n    pub fn first_vals(&self, docids: &[DocId], output: &mut [Option<T>]) {\n        match &self.index {\n            ColumnIndex::Empty { .. } => {}\n            ColumnIndex::Full => self.values.get_vals_opt(docids, output),\n            ColumnIndex::Optional(optional_index) => {\n                for (i, docid) in docids.iter().enumerate() {\n                    output[i] = optional_index\n                        .rank_if_exists(*docid)\n                        .map(|rowid| self.values.get_val(rowid));\n                }\n            }\n            ColumnIndex::Multivalued(multivalued_index) => {\n                for (i, docid) in docids.iter().enumerate() {\n                    let range = multivalued_index.range(*docid);\n                    let is_empty = range.start == range.end;\n                    if !is_empty {\n                        output[i] = Some(self.values.get_val(range.start));\n                    }\n                }\n            }\n        }\n    }\n\n    /// Translates a block of docids to row_ids.\n    ///\n    /// returns the row_ids and the matching docids on the same index\n    /// e.g.\n    /// DocId In:  [0, 5, 6]\n    /// DocId Out: [0, 0, 6, 6]\n    /// RowId Out: [0, 1, 2, 3]\n    #[inline]\n    pub fn row_ids_for_docs(\n        &self,\n        doc_ids: &[DocId],\n        doc_ids_out: &mut Vec<DocId>,\n        row_ids: &mut Vec<RowId>,\n    ) {\n        self.index.docids_to_rowids(doc_ids, doc_ids_out, row_ids)\n    }\n\n    /// Get an iterator over the values for the provided docid.\n    #[inline]\n    pub fn values_for_doc(&self, doc_id: DocId) -> impl Iterator<Item = T> + '_ {\n        self.index\n            .value_row_ids(doc_id)\n            .map(|value_row_id: RowId| self.values.get_val(value_row_id))\n    }\n\n    /// Get the docids of values which are in the provided value and docid range.\n    #[inline]\n    pub fn get_docids_for_value_range(\n        &self,\n        value_range: RangeInclusive<T>,\n        selected_docid_range: Range<u32>,\n        doc_ids: &mut Vec<u32>,\n    ) {\n        // convert passed docid range to row id range\n        let rowid_range = self\n            .index\n            .docid_range_to_rowids(selected_docid_range.clone());\n\n        // Load rows\n        self.values\n            .get_row_ids_for_value_range(value_range, rowid_range, doc_ids);\n        // Convert rows to docids\n        self.index\n            .select_batch_in_place(selected_docid_range.start, doc_ids);\n    }\n\n    pub fn first_or_default_col(self, default_value: T) -> Arc<dyn ColumnValues<T>> {\n        Arc::new(FirstValueWithDefault {\n            column: self,\n            default_value,\n        })\n    }\n}\n\nimpl BinarySerializable for Cardinality {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> std::io::Result<()> {\n        self.to_code().serialize(writer)\n    }\n\n    fn deserialize<R: std::io::Read>(reader: &mut R) -> std::io::Result<Self> {\n        let cardinality_code = u8::deserialize(reader)?;\n        let cardinality = Cardinality::try_from_code(cardinality_code)?;\n        Ok(cardinality)\n    }\n}\n\n// TODO simplify or optimize\nstruct FirstValueWithDefault<T: Copy> {\n    column: Column<T>,\n    default_value: T,\n}\n\nimpl<T: PartialOrd + Debug + Send + Sync + Copy + 'static> ColumnValues<T>\n    for FirstValueWithDefault<T>\n{\n    #[inline(always)]\n    fn get_val(&self, idx: u32) -> T {\n        self.column.first(idx).unwrap_or(self.default_value)\n    }\n\n    fn min_value(&self) -> T {\n        self.column.values.min_value()\n    }\n\n    fn max_value(&self) -> T {\n        self.column.values.max_value()\n    }\n\n    fn num_vals(&self) -> u32 {\n        match &self.column.index {\n            ColumnIndex::Empty { .. } => 0u32,\n            ColumnIndex::Full => self.column.values.num_vals(),\n            ColumnIndex::Optional(optional_idx) => optional_idx.num_docs(),\n            ColumnIndex::Multivalued(multivalue_idx) => multivalue_idx.num_docs(),\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column/serialize.rs",
    "content": "use std::io;\nuse std::io::Write;\nuse std::sync::Arc;\n\nuse common::OwnedBytes;\nuse sstable::Dictionary;\n\nuse crate::column::{BytesColumn, Column};\nuse crate::column_index::{SerializableColumnIndex, serialize_column_index};\nuse crate::column_values::{\n    CodecType, MonotonicallyMappableToU64, MonotonicallyMappableToU128,\n    load_u64_based_column_values, serialize_column_values_u128, serialize_u64_based_column_values,\n};\nuse crate::iterable::Iterable;\nuse crate::{StrColumn, Version};\n\npub fn serialize_column_mappable_to_u128<T: MonotonicallyMappableToU128>(\n    column_index: SerializableColumnIndex<'_>,\n    iterable: &dyn Iterable<T>,\n    output: &mut impl Write,\n) -> io::Result<()> {\n    let column_index_num_bytes = serialize_column_index(column_index, output)?;\n    serialize_column_values_u128(iterable, output)?;\n    output.write_all(&column_index_num_bytes.to_le_bytes())?;\n    Ok(())\n}\n\npub fn serialize_column_mappable_to_u64<T: MonotonicallyMappableToU64>(\n    column_index: SerializableColumnIndex<'_>,\n    column_values: &impl Iterable<T>,\n    output: &mut impl Write,\n) -> io::Result<()> {\n    let column_index_num_bytes = serialize_column_index(column_index, output)?;\n    serialize_u64_based_column_values(\n        column_values,\n        &[CodecType::Bitpacked, CodecType::BlockwiseLinear],\n        output,\n    )?;\n    output.write_all(&column_index_num_bytes.to_le_bytes())?;\n    Ok(())\n}\n\npub fn open_column_u64<T: MonotonicallyMappableToU64>(\n    bytes: OwnedBytes,\n    format_version: Version,\n) -> io::Result<Column<T>> {\n    let (body, column_index_num_bytes_payload) = bytes.rsplit(4);\n    let column_index_num_bytes = u32::from_le_bytes(\n        column_index_num_bytes_payload\n            .as_slice()\n            .try_into()\n            .unwrap(),\n    );\n    let (column_index_data, column_values_data) = body.split(column_index_num_bytes as usize);\n    let column_index = crate::column_index::open_column_index(column_index_data, format_version)?;\n    let column_values = load_u64_based_column_values(column_values_data)?;\n    Ok(Column {\n        index: column_index,\n        values: column_values,\n    })\n}\n\npub fn open_column_u128<T: MonotonicallyMappableToU128>(\n    bytes: OwnedBytes,\n    format_version: Version,\n) -> io::Result<Column<T>> {\n    let (body, column_index_num_bytes_payload) = bytes.rsplit(4);\n    let column_index_num_bytes = u32::from_le_bytes(\n        column_index_num_bytes_payload\n            .as_slice()\n            .try_into()\n            .unwrap(),\n    );\n    let (column_index_data, column_values_data) = body.split(column_index_num_bytes as usize);\n    let column_index = crate::column_index::open_column_index(column_index_data, format_version)?;\n    let column_values = crate::column_values::open_u128_mapped(column_values_data)?;\n    Ok(Column {\n        index: column_index,\n        values: column_values,\n    })\n}\n\n/// Open the column as u64.\n///\n/// See [`open_u128_as_compact_u64`] for more details.\npub fn open_column_u128_as_compact_u64(\n    bytes: OwnedBytes,\n    format_version: Version,\n) -> io::Result<Column<u64>> {\n    let (body, column_index_num_bytes_payload) = bytes.rsplit(4);\n    let column_index_num_bytes = u32::from_le_bytes(\n        column_index_num_bytes_payload\n            .as_slice()\n            .try_into()\n            .unwrap(),\n    );\n    let (column_index_data, column_values_data) = body.split(column_index_num_bytes as usize);\n    let column_index = crate::column_index::open_column_index(column_index_data, format_version)?;\n    let column_values = crate::column_values::open_u128_as_compact_u64(column_values_data)?;\n    Ok(Column {\n        index: column_index,\n        values: column_values,\n    })\n}\n\npub fn open_column_bytes(data: OwnedBytes, format_version: Version) -> io::Result<BytesColumn> {\n    let (body, dictionary_len_bytes) = data.rsplit(4);\n    let dictionary_len = u32::from_le_bytes(dictionary_len_bytes.as_slice().try_into().unwrap());\n    let (dictionary_bytes, column_bytes) = body.split(dictionary_len as usize);\n    let dictionary = Arc::new(Dictionary::from_bytes(dictionary_bytes)?);\n    let term_ord_column = crate::column::open_column_u64::<u64>(column_bytes, format_version)?;\n    Ok(BytesColumn {\n        dictionary,\n        term_ord_column,\n    })\n}\n\npub fn open_column_str(data: OwnedBytes, format_version: Version) -> io::Result<StrColumn> {\n    let bytes_column = open_column_bytes(data, format_version)?;\n    Ok(StrColumn::wrap(bytes_column))\n}\n"
  },
  {
    "path": "columnar/src/column_index/merge/mod.rs",
    "content": "mod shuffled;\nmod stacked;\n\nuse common::ReadOnlyBitSet;\nuse shuffled::merge_column_index_shuffled;\nuse stacked::merge_column_index_stacked;\n\nuse crate::column_index::SerializableColumnIndex;\nuse crate::{Cardinality, ColumnIndex, MergeRowOrder};\n\nfn detect_cardinality_single_column_index(\n    column_index: &ColumnIndex,\n    alive_bitset_opt: &Option<ReadOnlyBitSet>,\n) -> Cardinality {\n    let Some(alive_bitset) = alive_bitset_opt else {\n        return column_index.get_cardinality();\n    };\n    let cardinality_before_deletes = column_index.get_cardinality();\n    if cardinality_before_deletes == Cardinality::Full {\n        // The columnar cardinality can only become more restrictive in the presence of deletes\n        // (where cardinality sorted from the more restrictive to the least restrictive are Full,\n        // Optional, Multivalued)\n        //\n        // If we are already \"Full\", we are guaranteed to stay \"Full\" after deletes.\n        return Cardinality::Full;\n    }\n    let mut cardinality_so_far = Cardinality::Full;\n    for doc_id in alive_bitset.iter() {\n        let num_values = column_index.value_row_ids(doc_id).len();\n        let row_cardinality = match num_values {\n            0 => Cardinality::Optional,\n            1 => Cardinality::Full,\n            _ => Cardinality::Multivalued,\n        };\n        cardinality_so_far = cardinality_so_far.max(row_cardinality);\n        if cardinality_so_far >= cardinality_before_deletes {\n            // There won't be any improvement in the cardinality.\n            // We can early exit.\n            return cardinality_before_deletes;\n        }\n    }\n    cardinality_so_far\n}\n\nfn detect_cardinality(\n    column_indexes: &[ColumnIndex],\n    merge_row_order: &MergeRowOrder,\n) -> Cardinality {\n    match merge_row_order {\n        MergeRowOrder::Stack(_) => column_indexes\n            .iter()\n            .map(ColumnIndex::get_cardinality)\n            .max()\n            .unwrap_or(Cardinality::Full),\n        MergeRowOrder::Shuffled(shuffle_merge_order) => {\n            let mut merged_cardinality = Cardinality::Full;\n            for (column_index, alive_bitset_opt) in column_indexes\n                .iter()\n                .zip(shuffle_merge_order.alive_bitsets.iter())\n            {\n                let cardinality: Cardinality =\n                    detect_cardinality_single_column_index(column_index, alive_bitset_opt);\n                if cardinality == Cardinality::Multivalued {\n                    return cardinality;\n                }\n                merged_cardinality = merged_cardinality.max(cardinality);\n            }\n            merged_cardinality\n        }\n    }\n}\n\npub fn merge_column_index<'a>(\n    columns: &'a [ColumnIndex],\n    merge_row_order: &'a MergeRowOrder,\n) -> SerializableColumnIndex<'a> {\n    // For simplification, we do not try to detect whether the cardinality could be\n    // downgraded thanks to deletes.\n    let cardinality_after_merge = detect_cardinality(columns, merge_row_order);\n    match merge_row_order {\n        MergeRowOrder::Stack(stack_merge_order) => {\n            merge_column_index_stacked(columns, cardinality_after_merge, stack_merge_order)\n        }\n        MergeRowOrder::Shuffled(complex_merge_order) => {\n            merge_column_index_shuffled(columns, cardinality_after_merge, complex_merge_order)\n        }\n    }\n}\n\n// TODO actually, the shuffled code path is a bit too general.\n// In practise, we do not really shuffle everything.\n// The merge order restricted to a specific column keeps the original row order.\n//\n// This may offer some optimization that we have not explored yet.\n\n#[cfg(test)]\nmod tests {\n    use common::OwnedBytes;\n\n    use crate::column_index::merge::detect_cardinality;\n    use crate::column_index::multivalued_index::{\n        MultiValueIndex, open_multivalued_index, serialize_multivalued_index,\n    };\n    use crate::column_index::{OptionalIndex, SerializableColumnIndex, merge_column_index};\n    use crate::{\n        Cardinality, ColumnIndex, MergeRowOrder, RowAddr, RowId, ShuffleMergeOrder, StackMergeOrder,\n    };\n\n    #[test]\n    fn test_detect_cardinality() {\n        assert_eq!(\n            detect_cardinality(&[], &StackMergeOrder::stack_for_test(&[]).into()),\n            Cardinality::Full\n        );\n        let optional_index: ColumnIndex = OptionalIndex::for_test(1, &[]).into();\n        let multivalued_index: ColumnIndex = MultiValueIndex::for_test(&[0, 1]).into();\n        assert_eq!(\n            detect_cardinality(\n                &[optional_index.clone(), ColumnIndex::Empty { num_docs: 0 }],\n                &StackMergeOrder::stack_for_test(&[1, 0]).into()\n            ),\n            Cardinality::Optional\n        );\n        assert_eq!(\n            detect_cardinality(\n                &[optional_index.clone(), ColumnIndex::Full],\n                &StackMergeOrder::stack_for_test(&[1, 1]).into()\n            ),\n            Cardinality::Optional\n        );\n        assert_eq!(\n            detect_cardinality(\n                &[\n                    multivalued_index.clone(),\n                    ColumnIndex::Empty { num_docs: 0 }\n                ],\n                &StackMergeOrder::stack_for_test(&[1, 0]).into()\n            ),\n            Cardinality::Multivalued\n        );\n        assert_eq!(\n            detect_cardinality(\n                &[multivalued_index.clone(), optional_index.clone()],\n                &StackMergeOrder::stack_for_test(&[1, 1]).into()\n            ),\n            Cardinality::Multivalued\n        );\n        assert_eq!(\n            detect_cardinality(\n                &[optional_index, multivalued_index],\n                &StackMergeOrder::stack_for_test(&[1, 1]).into()\n            ),\n            Cardinality::Multivalued\n        );\n    }\n\n    #[test]\n    fn test_merge_index_multivalued_sorted() {\n        let column_indexes: Vec<ColumnIndex> = vec![MultiValueIndex::for_test(&[0, 2, 5]).into()];\n        let merge_row_order: MergeRowOrder = ShuffleMergeOrder::for_test(\n            &[2],\n            vec![\n                RowAddr {\n                    segment_ord: 0u32,\n                    row_id: 1u32,\n                },\n                RowAddr {\n                    segment_ord: 0u32,\n                    row_id: 0u32,\n                },\n            ],\n        )\n        .into();\n        let merged_column_index = merge_column_index(&column_indexes[..], &merge_row_order);\n        let SerializableColumnIndex::Multivalued(start_index_iterable) = merged_column_index else {\n            panic!(\"Expected a multivalued index\")\n        };\n        let mut output = Vec::new();\n        serialize_multivalued_index(&start_index_iterable, &mut output).unwrap();\n        let multivalue =\n            open_multivalued_index(OwnedBytes::new(output), crate::Version::V2).unwrap();\n        let start_indexes: Vec<RowId> = multivalue.get_start_index_column().iter().collect();\n        assert_eq!(&start_indexes, &[0, 3, 5]);\n    }\n\n    #[test]\n    fn test_merge_index_multivalued_sorted_several_segment() {\n        let column_indexes: Vec<ColumnIndex> = vec![\n            MultiValueIndex::for_test(&[0, 2, 5]).into(),\n            ColumnIndex::Empty { num_docs: 0 },\n            MultiValueIndex::for_test(&[0, 1, 4]).into(),\n        ];\n        let merge_row_order: MergeRowOrder = ShuffleMergeOrder::for_test(\n            &[2, 0, 2],\n            vec![\n                RowAddr {\n                    segment_ord: 2u32,\n                    row_id: 1u32,\n                },\n                RowAddr {\n                    segment_ord: 0u32,\n                    row_id: 0u32,\n                },\n                RowAddr {\n                    segment_ord: 2u32,\n                    row_id: 0u32,\n                },\n            ],\n        )\n        .into();\n\n        let merged_column_index = merge_column_index(&column_indexes[..], &merge_row_order);\n        let SerializableColumnIndex::Multivalued(start_index_iterable) = merged_column_index else {\n            panic!(\"Expected a multivalued index\")\n        };\n        let mut output = Vec::new();\n        serialize_multivalued_index(&start_index_iterable, &mut output).unwrap();\n        let multivalue =\n            open_multivalued_index(OwnedBytes::new(output), crate::Version::V2).unwrap();\n        let start_indexes: Vec<RowId> = multivalue.get_start_index_column().iter().collect();\n        assert_eq!(&start_indexes, &[0, 3, 5, 6]);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/merge/shuffled.rs",
    "content": "use std::iter;\n\nuse crate::column_index::{\n    SerializableColumnIndex, SerializableMultivalueIndex, SerializableOptionalIndex, Set,\n};\nuse crate::iterable::Iterable;\nuse crate::{Cardinality, ColumnIndex, RowId, ShuffleMergeOrder};\n\npub fn merge_column_index_shuffled<'a>(\n    column_indexes: &'a [ColumnIndex],\n    cardinality_after_merge: Cardinality,\n    shuffle_merge_order: &'a ShuffleMergeOrder,\n) -> SerializableColumnIndex<'a> {\n    match cardinality_after_merge {\n        Cardinality::Full => SerializableColumnIndex::Full,\n        Cardinality::Optional => {\n            let non_null_row_ids =\n                merge_column_index_shuffled_optional(column_indexes, shuffle_merge_order);\n            SerializableColumnIndex::Optional(SerializableOptionalIndex {\n                non_null_row_ids,\n                num_rows: shuffle_merge_order.num_rows(),\n            })\n        }\n        Cardinality::Multivalued => {\n            let non_null_row_ids =\n                merge_column_index_shuffled_optional(column_indexes, shuffle_merge_order);\n            SerializableColumnIndex::Multivalued(SerializableMultivalueIndex {\n                doc_ids_with_values: SerializableOptionalIndex {\n                    non_null_row_ids,\n                    num_rows: shuffle_merge_order.num_rows(),\n                },\n                start_offsets: merge_column_index_shuffled_multivalued(\n                    column_indexes,\n                    shuffle_merge_order,\n                ),\n            })\n        }\n    }\n}\n\n/// Merge several column indexes into one, ordering rows according to the merge_order passed as\n/// argument. While it is true that the `merge_order` may imply deletes and hence could in theory a\n/// multivalued index into an optional one, this is not supported today for simplification.\n///\n/// In other words the column_indexes passed as argument may NOT be multivalued.\nfn merge_column_index_shuffled_optional<'a>(\n    column_indexes: &'a [ColumnIndex],\n    merge_order: &'a ShuffleMergeOrder,\n) -> Box<dyn Iterable<RowId> + 'a> {\n    Box::new(ShuffledIndex {\n        column_indexes,\n        merge_order,\n    })\n}\n\nstruct ShuffledIndex<'a> {\n    column_indexes: &'a [ColumnIndex],\n    merge_order: &'a ShuffleMergeOrder,\n}\n\nimpl Iterable<u32> for ShuffledIndex<'_> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u32> + '_> {\n        Box::new(\n            self.merge_order\n                .iter_new_to_old_row_addrs()\n                .enumerate()\n                .filter_map(|(new_row_id, old_row_addr)| {\n                    let column_index = &self.column_indexes[old_row_addr.segment_ord as usize];\n                    let row_id = new_row_id as u32;\n                    if column_index.has_value(old_row_addr.row_id) {\n                        Some(row_id)\n                    } else {\n                        None\n                    }\n                }),\n        )\n    }\n}\n\nfn merge_column_index_shuffled_multivalued<'a>(\n    column_indexes: &'a [ColumnIndex],\n    merge_order: &'a ShuffleMergeOrder,\n) -> Box<dyn Iterable<RowId> + 'a> {\n    Box::new(ShuffledMultivaluedIndex {\n        column_indexes,\n        merge_order,\n    })\n}\n\nstruct ShuffledMultivaluedIndex<'a> {\n    column_indexes: &'a [ColumnIndex],\n    merge_order: &'a ShuffleMergeOrder,\n}\n\nfn iter_num_values<'a>(\n    column_indexes: &'a [ColumnIndex],\n    merge_order: &'a ShuffleMergeOrder,\n) -> impl Iterator<Item = u32> + 'a {\n    merge_order.iter_new_to_old_row_addrs().map(|row_addr| {\n        let column_index = &column_indexes[row_addr.segment_ord as usize];\n        match column_index {\n            ColumnIndex::Empty { .. } => 0u32,\n            ColumnIndex::Full => 1,\n            ColumnIndex::Optional(optional_index) => {\n                u32::from(optional_index.contains(row_addr.row_id))\n            }\n            ColumnIndex::Multivalued(multivalued_index) => {\n                multivalued_index.range(row_addr.row_id).len() as u32\n            }\n        }\n    })\n}\n\n/// Transforms an iterator containing the number of vals per row (with `num_rows` elements)\n/// into a `start_offset` iterator starting at 0 and (with `num_rows + 1` element)\n///\n/// This will filter values with 0 values as these are covered by the optional index in the\n/// multivalue index.\nfn integrate_num_vals(num_vals: impl Iterator<Item = u32>) -> impl Iterator<Item = RowId> {\n    iter::once(0u32).chain(\n        num_vals\n            .filter(|num_vals| *num_vals != 0)\n            .scan(0, |state, num_vals| {\n                *state += num_vals;\n                Some(*state)\n            }),\n    )\n}\n\nimpl Iterable<u32> for ShuffledMultivaluedIndex<'_> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u32> + '_> {\n        let num_vals_per_row = iter_num_values(self.column_indexes, self.merge_order);\n        Box::new(integrate_num_vals(num_vals_per_row))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::RowAddr;\n    use crate::column_index::OptionalIndex;\n\n    #[test]\n    fn test_integrate_num_vals_empty() {\n        assert!(integrate_num_vals(iter::empty()).eq(iter::once(0)));\n    }\n\n    #[test]\n    fn test_integrate_num_vals_one_el() {\n        assert!(integrate_num_vals(iter::once(10)).eq([0, 10].into_iter()));\n    }\n\n    #[test]\n    fn test_integrate_num_vals_several() {\n        assert!(integrate_num_vals([3, 0, 10, 20].into_iter()).eq([0, 3, 13, 33].into_iter()));\n    }\n\n    #[test]\n    fn test_merge_column_index_optional_shuffle() {\n        let optional_index: ColumnIndex = OptionalIndex::for_test(2, &[0]).into();\n        let column_indexes = [optional_index, ColumnIndex::Full];\n        let row_addrs = vec![\n            RowAddr {\n                segment_ord: 0u32,\n                row_id: 1u32,\n            },\n            RowAddr {\n                segment_ord: 1u32,\n                row_id: 0u32,\n            },\n        ];\n        let shuffle_merge_order = ShuffleMergeOrder::for_test(&[2, 1], row_addrs);\n        let serializable_index = merge_column_index_shuffled(\n            &column_indexes[..],\n            Cardinality::Optional,\n            &shuffle_merge_order,\n        );\n        let SerializableColumnIndex::Optional(SerializableOptionalIndex {\n            non_null_row_ids,\n            num_rows,\n        }) = serializable_index\n        else {\n            panic!()\n        };\n        assert_eq!(num_rows, 2);\n        let non_null_rows: Vec<RowId> = non_null_row_ids.boxed_iter().collect();\n        assert_eq!(&non_null_rows, &[1]);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/merge/stacked.rs",
    "content": "use std::ops::Range;\n\nuse crate::column_index::SerializableColumnIndex;\nuse crate::column_index::multivalued_index::{MultiValueIndex, SerializableMultivalueIndex};\nuse crate::column_index::serialize::SerializableOptionalIndex;\nuse crate::iterable::Iterable;\nuse crate::{Cardinality, ColumnIndex, RowId, StackMergeOrder};\n\n/// Simple case:\n/// The new mapping just consists in stacking the different column indexes.\n///\n/// There are no sort nor deletes involved.\npub fn merge_column_index_stacked<'a>(\n    columns: &'a [ColumnIndex],\n    cardinality_after_merge: Cardinality,\n    stack_merge_order: &'a StackMergeOrder,\n) -> SerializableColumnIndex<'a> {\n    match cardinality_after_merge {\n        Cardinality::Full => SerializableColumnIndex::Full,\n        Cardinality::Optional => SerializableColumnIndex::Optional(SerializableOptionalIndex {\n            non_null_row_ids: Box::new(StackedOptionalIndex {\n                columns,\n                stack_merge_order,\n            }),\n            num_rows: stack_merge_order.num_rows(),\n        }),\n        Cardinality::Multivalued => {\n            let serializable_multivalue_index =\n                make_serializable_multivalued_index(columns, stack_merge_order);\n            SerializableColumnIndex::Multivalued(serializable_multivalue_index)\n        }\n    }\n}\n\nstruct StackedDocIdsWithValues<'a> {\n    column_indexes: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n}\n\nimpl Iterable<u32> for StackedDocIdsWithValues<'_> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u32> + '_> {\n        Box::new((0..self.column_indexes.len()).flat_map(|i| {\n            let column_index = &self.column_indexes[i];\n            let doc_range = self.stack_merge_order.columnar_range(i);\n            get_doc_ids_with_values(column_index, doc_range)\n        }))\n    }\n}\n\nfn get_doc_ids_with_values<'a>(\n    column_index: &'a ColumnIndex,\n    doc_range: Range<u32>,\n) -> Box<dyn Iterator<Item = u32> + 'a> {\n    match column_index {\n        ColumnIndex::Empty { .. } => Box::new(0..0),\n        ColumnIndex::Full => Box::new(doc_range),\n        ColumnIndex::Optional(optional_index) => Box::new(\n            optional_index\n                .iter_non_null_docs()\n                .map(move |row| row + doc_range.start),\n        ),\n        ColumnIndex::Multivalued(multivalued_index) => match multivalued_index {\n            MultiValueIndex::MultiValueIndexV1(multivalued_index) => {\n                Box::new((0..multivalued_index.num_docs()).filter_map(move |docid| {\n                    let range = multivalued_index.range(docid);\n                    if range.is_empty() {\n                        None\n                    } else {\n                        Some(docid + doc_range.start)\n                    }\n                }))\n            }\n            MultiValueIndex::MultiValueIndexV2(multivalued_index) => Box::new(\n                multivalued_index\n                    .optional_index\n                    .iter_non_null_docs()\n                    .map(move |row| row + doc_range.start),\n            ),\n        },\n    }\n}\n\nfn stack_doc_ids_with_values<'a>(\n    column_indexes: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n) -> SerializableOptionalIndex<'a> {\n    let num_rows = stack_merge_order.num_rows();\n    SerializableOptionalIndex {\n        non_null_row_ids: Box::new(StackedDocIdsWithValues {\n            column_indexes,\n            stack_merge_order,\n        }),\n        num_rows,\n    }\n}\n\nstruct StackedStartOffsets<'a> {\n    column_indexes: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n}\n\nfn get_num_values_iterator<'a>(\n    column_index: &'a ColumnIndex,\n    num_docs: u32,\n) -> Box<dyn Iterator<Item = u32> + 'a> {\n    match column_index {\n        ColumnIndex::Empty { .. } => Box::new(std::iter::empty()),\n        ColumnIndex::Full => Box::new(std::iter::repeat_n(1u32, num_docs as usize)),\n        ColumnIndex::Optional(optional_index) => Box::new(std::iter::repeat_n(\n            1u32,\n            optional_index.num_non_nulls() as usize,\n        )),\n        ColumnIndex::Multivalued(multivalued_index) => Box::new(\n            multivalued_index\n                .get_start_index_column()\n                .iter()\n                .scan(0u32, |previous_start_offset, current_start_offset| {\n                    let num_vals = current_start_offset - *previous_start_offset;\n                    *previous_start_offset = current_start_offset;\n                    Some(num_vals)\n                })\n                .skip(1),\n        ),\n    }\n}\n\nimpl Iterable<u32> for StackedStartOffsets<'_> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u32> + '_> {\n        let num_values_it = (0..self.column_indexes.len()).flat_map(|columnar_id| {\n            let num_docs = self.stack_merge_order.columnar_range(columnar_id).len() as u32;\n            let column_index = &self.column_indexes[columnar_id];\n            get_num_values_iterator(column_index, num_docs)\n        });\n        Box::new(std::iter::once(0u32).chain(num_values_it.into_iter().scan(\n            0u32,\n            |cumulated, el| {\n                *cumulated += el;\n                Some(*cumulated)\n            },\n        )))\n    }\n}\n\nfn stack_start_offsets<'a>(\n    column_indexes: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n) -> Box<dyn Iterable<u32> + 'a> {\n    Box::new(StackedStartOffsets {\n        column_indexes,\n        stack_merge_order,\n    })\n}\n\nfn make_serializable_multivalued_index<'a>(\n    columns: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n) -> SerializableMultivalueIndex<'a> {\n    SerializableMultivalueIndex {\n        doc_ids_with_values: stack_doc_ids_with_values(columns, stack_merge_order),\n        start_offsets: stack_start_offsets(columns, stack_merge_order),\n    }\n}\n\nstruct StackedOptionalIndex<'a> {\n    columns: &'a [ColumnIndex],\n    stack_merge_order: &'a StackMergeOrder,\n}\n\nimpl<'a> Iterable<RowId> for StackedOptionalIndex<'a> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = RowId> + 'a> {\n        Box::new(\n            self.columns\n                .iter()\n                .enumerate()\n                .flat_map(|(columnar_id, column_index_opt)| {\n                    let columnar_row_range = self.stack_merge_order.columnar_range(columnar_id);\n                    let rows_it: Box<dyn Iterator<Item = RowId>> = match column_index_opt {\n                        ColumnIndex::Full => Box::new(columnar_row_range),\n                        ColumnIndex::Optional(optional_index) => Box::new(\n                            optional_index\n                                .iter_non_null_docs()\n                                .map(move |row_id: RowId| columnar_row_range.start + row_id),\n                        ),\n                        ColumnIndex::Multivalued(_) => {\n                            panic!(\"No multivalued index is allowed when stacking column index\");\n                        }\n                        ColumnIndex::Empty { .. } => Box::new(std::iter::empty()),\n                    };\n                    rows_it\n                }),\n        )\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/mod.rs",
    "content": "//! # `column_index`\n//!\n//! `column_index` provides rank and select operations to associate positions when not all\n//! documents have exactly one element.\n\nmod merge;\nmod multivalued_index;\nmod optional_index;\nmod serialize;\n\nuse std::ops::Range;\n\npub use merge::merge_column_index;\npub(crate) use multivalued_index::SerializableMultivalueIndex;\npub use optional_index::{OptionalIndex, Set};\npub use serialize::{\n    SerializableColumnIndex, SerializableOptionalIndex, open_column_index, serialize_column_index,\n};\n\nuse crate::column_index::multivalued_index::MultiValueIndex;\nuse crate::{Cardinality, DocId, RowId};\n\n#[derive(Clone, Debug)]\npub enum ColumnIndex {\n    Empty {\n        num_docs: u32,\n    },\n    Full,\n    Optional(OptionalIndex),\n    /// In addition, at index num_rows, an extra value is added\n    /// containing the overall number of values.\n    Multivalued(MultiValueIndex),\n}\n\nimpl From<OptionalIndex> for ColumnIndex {\n    fn from(optional_index: OptionalIndex) -> ColumnIndex {\n        ColumnIndex::Optional(optional_index)\n    }\n}\n\nimpl From<MultiValueIndex> for ColumnIndex {\n    fn from(multi_value_index: MultiValueIndex) -> ColumnIndex {\n        ColumnIndex::Multivalued(multi_value_index)\n    }\n}\n\nimpl ColumnIndex {\n    /// Returns the cardinality of the column index.\n    ///\n    /// By convention, if the column contains no docs, we consider that it is\n    /// full.\n    #[inline]\n    pub fn get_cardinality(&self) -> Cardinality {\n        match self {\n            ColumnIndex::Empty { num_docs: 0 } | ColumnIndex::Full => Cardinality::Full,\n            ColumnIndex::Empty { .. } => Cardinality::Optional,\n            ColumnIndex::Optional(_) => Cardinality::Optional,\n            ColumnIndex::Multivalued(_) => Cardinality::Multivalued,\n        }\n    }\n\n    /// Returns true if and only if there are at least one value associated to the row.\n    pub fn has_value(&self, doc_id: DocId) -> bool {\n        match self {\n            ColumnIndex::Empty { .. } => false,\n            ColumnIndex::Full => true,\n            ColumnIndex::Optional(optional_index) => optional_index.contains(doc_id),\n            ColumnIndex::Multivalued(multivalued_index) => {\n                !multivalued_index.range(doc_id).is_empty()\n            }\n        }\n    }\n\n    pub fn value_row_ids(&self, doc_id: DocId) -> Range<RowId> {\n        match self {\n            ColumnIndex::Empty { .. } => 0..0,\n            ColumnIndex::Full => doc_id..doc_id + 1,\n            ColumnIndex::Optional(optional_index) => {\n                if let Some(val) = optional_index.rank_if_exists(doc_id) {\n                    val..val + 1\n                } else {\n                    0..0\n                }\n            }\n            ColumnIndex::Multivalued(multivalued_index) => multivalued_index.range(doc_id),\n        }\n    }\n\n    /// Translates a block of docis to row_ids.\n    ///\n    /// returns the row_ids and the matching docids on the same index\n    /// e.g.\n    /// DocId In:  [0, 5, 6]\n    /// DocId Out: [0, 0, 6, 6]\n    /// RowId Out: [0, 1, 2, 3]\n    #[inline]\n    pub fn docids_to_rowids(\n        &self,\n        doc_ids: &[DocId],\n        doc_ids_out: &mut Vec<DocId>,\n        row_ids: &mut Vec<RowId>,\n    ) {\n        match self {\n            ColumnIndex::Empty { .. } => {}\n            ColumnIndex::Full => {\n                doc_ids_out.extend_from_slice(doc_ids);\n                row_ids.extend_from_slice(doc_ids);\n            }\n            ColumnIndex::Optional(optional_index) => {\n                for doc_id in doc_ids {\n                    if let Some(row_id) = optional_index.rank_if_exists(*doc_id) {\n                        doc_ids_out.push(*doc_id);\n                        row_ids.push(row_id);\n                    }\n                }\n            }\n            ColumnIndex::Multivalued(multivalued_index) => {\n                for doc_id in doc_ids {\n                    for row_id in multivalued_index.range(*doc_id) {\n                        doc_ids_out.push(*doc_id);\n                        row_ids.push(row_id);\n                    }\n                }\n            }\n        }\n    }\n\n    pub fn docid_range_to_rowids(&self, doc_id_range: Range<DocId>) -> Range<RowId> {\n        match self {\n            ColumnIndex::Empty { .. } => 0..0,\n            ColumnIndex::Full => doc_id_range,\n            ColumnIndex::Optional(optional_index) => {\n                let row_start = optional_index.rank(doc_id_range.start);\n                let row_end = optional_index.rank(doc_id_range.end);\n                row_start..row_end\n            }\n            ColumnIndex::Multivalued(multivalued_index) => match multivalued_index {\n                MultiValueIndex::MultiValueIndexV1(index) => {\n                    let row_start = index.start_index_column.get_val(doc_id_range.start);\n                    let row_end = index.start_index_column.get_val(doc_id_range.end);\n                    row_start..row_end\n                }\n                MultiValueIndex::MultiValueIndexV2(index) => {\n                    // In this case we will use the optional_index select the next values\n                    // that are valid. There are different cases to consider:\n                    // Not exists below means does not exist in the optional\n                    // index, because it has no values.\n                    // * doc_id_range may cover a range of docids which are non existent\n                    // => rank\n                    //   will give us the next document outside the range with a value. They both\n                    //   get the same rank and therefore return a zero range\n                    //\n                    // * doc_id_range.start and doc_id_range.end may not exist, but docids in\n                    // between may have values\n                    // => rank will give us the next document outside the range with a value.\n                    //\n                    // * doc_id_range.start may be not existent but doc_id_range.end may exist\n                    // * doc_id_range.start may exist but doc_id_range.end may not exist\n                    // * doc_id_range.start and doc_id_range.end may exist\n                    // => rank on doc_id_range.end will give use the next value, which matches\n                    // how the `start_index_column` works, so we get the value start of the next\n                    // docid which we use to create the exclusive range.\n                    //\n                    let rank_start = index.optional_index.rank(doc_id_range.start);\n                    let row_start = index.start_index_column.get_val(rank_start);\n                    let rank_end = index.optional_index.rank(doc_id_range.end);\n                    let row_end = index.start_index_column.get_val(rank_end);\n\n                    row_start..row_end\n                }\n            },\n        }\n    }\n\n    pub fn select_batch_in_place(&self, doc_id_start: DocId, rank_ids: &mut Vec<RowId>) {\n        match self {\n            ColumnIndex::Empty { .. } => {\n                rank_ids.clear();\n            }\n            ColumnIndex::Full => {\n                // No need to do anything:\n                // value_idx and row_idx are the same.\n            }\n            ColumnIndex::Optional(optional_index) => {\n                optional_index.select_batch(&mut rank_ids[..]);\n            }\n            ColumnIndex::Multivalued(multivalued_index) => {\n                multivalued_index.select_batch_in_place(doc_id_start, rank_ids)\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::{Cardinality, ColumnIndex};\n\n    #[test]\n    fn test_column_index_get_cardinality() {\n        assert_eq!(\n            ColumnIndex::Empty { num_docs: 0 }.get_cardinality(),\n            Cardinality::Full\n        );\n        assert_eq!(ColumnIndex::Full.get_cardinality(), Cardinality::Full);\n        assert_eq!(\n            ColumnIndex::Empty { num_docs: 1 }.get_cardinality(),\n            Cardinality::Optional\n        );\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/multivalued_index.rs",
    "content": "use std::io;\nuse std::io::Write;\nuse std::ops::Range;\nuse std::sync::Arc;\n\nuse common::{CountingWriter, OwnedBytes};\n\nuse super::optional_index::{open_optional_index, serialize_optional_index};\nuse super::{OptionalIndex, SerializableOptionalIndex, Set};\nuse crate::column_values::{\n    CodecType, ColumnValues, load_u64_based_column_values, serialize_u64_based_column_values,\n};\nuse crate::iterable::Iterable;\nuse crate::{DocId, RowId, Version};\n\npub struct SerializableMultivalueIndex<'a> {\n    pub doc_ids_with_values: SerializableOptionalIndex<'a>,\n    pub start_offsets: Box<dyn Iterable<u32> + 'a>,\n}\n\npub fn serialize_multivalued_index(\n    multivalued_index: &SerializableMultivalueIndex,\n    output: &mut impl Write,\n) -> io::Result<()> {\n    let SerializableMultivalueIndex {\n        doc_ids_with_values,\n        start_offsets,\n    } = multivalued_index;\n    let mut count_writer = CountingWriter::wrap(output);\n    let SerializableOptionalIndex {\n        non_null_row_ids,\n        num_rows,\n    } = doc_ids_with_values;\n    serialize_optional_index(&**non_null_row_ids, *num_rows, &mut count_writer)?;\n    let optional_len = count_writer.written_bytes() as u32;\n    let output = count_writer.finish();\n    serialize_u64_based_column_values(\n        &**start_offsets,\n        &[CodecType::Bitpacked, CodecType::Linear],\n        output,\n    )?;\n    output.write_all(&optional_len.to_le_bytes())?;\n    Ok(())\n}\n\npub fn open_multivalued_index(\n    bytes: OwnedBytes,\n    format_version: Version,\n) -> io::Result<MultiValueIndex> {\n    match format_version {\n        Version::V1 => {\n            let start_index_column: Arc<dyn ColumnValues<RowId>> =\n                load_u64_based_column_values(bytes)?;\n            Ok(MultiValueIndex::MultiValueIndexV1(MultiValueIndexV1 {\n                start_index_column,\n            }))\n        }\n        Version::V2 => {\n            let (body_bytes, optional_index_len) = bytes.rsplit(4);\n            let optional_index_len =\n                u32::from_le_bytes(optional_index_len.as_slice().try_into().unwrap());\n            let (optional_index_bytes, start_index_bytes) =\n                body_bytes.split(optional_index_len as usize);\n            let optional_index = open_optional_index(optional_index_bytes)?;\n            let start_index_column: Arc<dyn ColumnValues<RowId>> =\n                load_u64_based_column_values(start_index_bytes)?;\n            Ok(MultiValueIndex::MultiValueIndexV2(MultiValueIndexV2 {\n                optional_index,\n                start_index_column,\n            }))\n        }\n    }\n}\n\n#[derive(Clone)]\n/// Index to resolve value range for given doc_id.\n/// Starts at 0.\npub enum MultiValueIndex {\n    MultiValueIndexV1(MultiValueIndexV1),\n    MultiValueIndexV2(MultiValueIndexV2),\n}\n\n#[derive(Clone)]\n/// Index to resolve value range for given doc_id.\n/// Starts at 0.\npub struct MultiValueIndexV1 {\n    pub start_index_column: Arc<dyn crate::ColumnValues<RowId>>,\n}\n\nimpl MultiValueIndexV1 {\n    /// Returns `[start, end)`, such that the values associated with\n    /// the given document are `start..end`.\n    #[inline]\n    pub(crate) fn range(&self, doc_id: DocId) -> Range<RowId> {\n        if doc_id >= self.num_docs() {\n            return 0..0;\n        }\n        let start = self.start_index_column.get_val(doc_id);\n        let end = self.start_index_column.get_val(doc_id + 1);\n        start..end\n    }\n\n    /// Returns the number of documents in the index.\n    #[inline]\n    pub fn num_docs(&self) -> u32 {\n        self.start_index_column.num_vals() - 1\n    }\n\n    /// Converts a list of ranks (row ids of values) in a 1:n index to the corresponding list of\n    /// docids. Positions are converted inplace to docids.\n    ///\n    /// Since there is no index for value pos -> docid, but docid -> value pos range, we scan the\n    /// index.\n    ///\n    /// Correctness: positions needs to be sorted. idx_reader needs to contain monotonically\n    /// increasing positions.\n    ///\n    /// TODO: Instead of a linear scan we can employ a exponential search into binary search to\n    /// match a docid to its value position.\n    pub(crate) fn select_batch_in_place(&self, docid_start: DocId, ranks: &mut Vec<u32>) {\n        if ranks.is_empty() {\n            return;\n        }\n        let mut cur_doc = docid_start;\n        let mut last_doc = None;\n\n        assert!(self.start_index_column.get_val(docid_start) <= ranks[0]);\n\n        let mut write_doc_pos = 0;\n        for i in 0..ranks.len() {\n            let pos = ranks[i];\n            loop {\n                let end = self.start_index_column.get_val(cur_doc + 1);\n                if end > pos {\n                    ranks[write_doc_pos] = cur_doc;\n                    write_doc_pos += if last_doc == Some(cur_doc) { 0 } else { 1 };\n                    last_doc = Some(cur_doc);\n                    break;\n                }\n                cur_doc += 1;\n            }\n        }\n        ranks.truncate(write_doc_pos);\n    }\n}\n\n#[derive(Clone)]\n/// Index to resolve value range for given doc_id.\n/// Starts at 0.\npub struct MultiValueIndexV2 {\n    pub optional_index: OptionalIndex,\n    pub start_index_column: Arc<dyn crate::ColumnValues<RowId>>,\n}\n\nimpl std::fmt::Debug for MultiValueIndex {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        let index = match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => &idx.start_index_column,\n            MultiValueIndex::MultiValueIndexV2(idx) => &idx.start_index_column,\n        };\n        f.debug_struct(\"MultiValuedIndex\")\n            .field(\"num_rows\", &index.num_vals())\n            .finish_non_exhaustive()\n    }\n}\n\nimpl MultiValueIndex {\n    pub fn for_test(start_offsets: &[RowId]) -> MultiValueIndex {\n        assert!(!start_offsets.is_empty());\n        assert_eq!(start_offsets[0], 0);\n        let mut doc_with_values = Vec::new();\n        let mut compact_start_offsets: Vec<u32> = vec![0];\n        for doc in 0..start_offsets.len() - 1 {\n            if start_offsets[doc] < start_offsets[doc + 1] {\n                doc_with_values.push(doc as RowId);\n                compact_start_offsets.push(start_offsets[doc + 1]);\n            }\n        }\n        let serializable_multivalued_index = SerializableMultivalueIndex {\n            doc_ids_with_values: SerializableOptionalIndex {\n                non_null_row_ids: Box::new(&doc_with_values[..]),\n                num_rows: start_offsets.len() as u32 - 1,\n            },\n            start_offsets: Box::new(&compact_start_offsets[..]),\n        };\n        let mut buffer = Vec::new();\n        serialize_multivalued_index(&serializable_multivalued_index, &mut buffer).unwrap();\n        let bytes = OwnedBytes::new(buffer);\n        open_multivalued_index(bytes, Version::V2).unwrap()\n    }\n\n    pub fn get_start_index_column(&self) -> &Arc<dyn crate::ColumnValues<RowId>> {\n        match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => &idx.start_index_column,\n            MultiValueIndex::MultiValueIndexV2(idx) => &idx.start_index_column,\n        }\n    }\n\n    /// Returns `[start, end)` values range, such that the values associated with\n    /// the given document are `start..end`.\n    #[inline]\n    pub(crate) fn range(&self, doc_id: DocId) -> Range<RowId> {\n        match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => idx.range(doc_id),\n            MultiValueIndex::MultiValueIndexV2(idx) => idx.range(doc_id),\n        }\n    }\n\n    /// Returns the number of documents in the index.\n    #[inline]\n    pub fn num_docs(&self) -> u32 {\n        match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => idx.start_index_column.num_vals() - 1,\n            MultiValueIndex::MultiValueIndexV2(idx) => idx.optional_index.num_docs(),\n        }\n    }\n\n    /// Returns an iterator over document ids that have at least one value.\n    pub fn iter_non_null_docs(&self) -> Box<dyn Iterator<Item = DocId> + '_> {\n        match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => {\n                let mut doc: DocId = 0u32;\n                let num_docs = idx.num_docs();\n                Box::new(std::iter::from_fn(move || {\n                    // This is not the most efficient way to do this, but it's legacy code.\n                    while doc < num_docs {\n                        let cur = doc;\n                        doc += 1;\n                        let start = idx.start_index_column.get_val(cur);\n                        let end = idx.start_index_column.get_val(cur + 1);\n                        if end > start {\n                            return Some(cur);\n                        }\n                    }\n                    None\n                }))\n            }\n            MultiValueIndex::MultiValueIndexV2(idx) => {\n                Box::new(idx.optional_index.iter_non_null_docs())\n            }\n        }\n    }\n\n    /// Converts a list of ranks (row ids of values) in a 1:n index to the corresponding list of\n    /// docids. Positions are converted inplace to docids.\n    ///\n    /// Since there is no index for value pos -> docid, but docid -> value pos range, we scan the\n    /// index.\n    ///\n    /// Correctness: positions needs to be sorted. idx_reader needs to contain monotonically\n    /// increasing positions.\n    ///\n    /// TODO: Instead of a linear scan we can employ a exponential search into binary search to\n    /// match a docid to its value position.\n    pub(crate) fn select_batch_in_place(&self, docid_start: DocId, ranks: &mut Vec<u32>) {\n        match self {\n            MultiValueIndex::MultiValueIndexV1(idx) => {\n                idx.select_batch_in_place(docid_start, ranks)\n            }\n            MultiValueIndex::MultiValueIndexV2(idx) => {\n                idx.select_batch_in_place(docid_start, ranks)\n            }\n        }\n    }\n}\nimpl MultiValueIndexV2 {\n    /// Returns `[start, end)`, such that the values associated with\n    /// the given document are `start..end`.\n    #[inline]\n    pub(crate) fn range(&self, doc_id: DocId) -> Range<RowId> {\n        let Some(rank) = self.optional_index.rank_if_exists(doc_id) else {\n            return 0..0;\n        };\n        let start = self.start_index_column.get_val(rank);\n        let end = self.start_index_column.get_val(rank + 1);\n        start..end\n    }\n\n    /// Returns the number of documents in the index.\n    #[inline]\n    pub fn num_docs(&self) -> u32 {\n        self.optional_index.num_docs()\n    }\n\n    /// Converts a list of ranks (row ids of values) in a 1:n index to the corresponding list of\n    /// docids. Positions are converted inplace to docids.\n    ///\n    /// Since there is no index for value pos -> docid, but docid -> value pos range, we scan the\n    /// index.\n    ///\n    /// Correctness: positions needs to be sorted. idx_reader needs to contain monotonically\n    /// increasing positions.\n    ///\n    /// TODO: Instead of a linear scan we can employ a exponential search into binary search to\n    /// match a docid to its value position.\n    pub(crate) fn select_batch_in_place(&self, docid_start: DocId, ranks: &mut Vec<u32>) {\n        if ranks.is_empty() {\n            return;\n        }\n        let mut cur_pos_in_idx = self.optional_index.rank(docid_start);\n        let mut last_doc = None;\n\n        assert!(cur_pos_in_idx <= ranks[0]);\n\n        let mut write_doc_pos = 0;\n        for i in 0..ranks.len() {\n            let pos = ranks[i];\n            loop {\n                let end = self.start_index_column.get_val(cur_pos_in_idx + 1);\n                if end > pos {\n                    ranks[write_doc_pos] = cur_pos_in_idx;\n                    write_doc_pos += if last_doc == Some(cur_pos_in_idx) {\n                        0\n                    } else {\n                        1\n                    };\n                    last_doc = Some(cur_pos_in_idx);\n                    break;\n                }\n                cur_pos_in_idx += 1;\n            }\n        }\n        ranks.truncate(write_doc_pos);\n\n        for rank in ranks.iter_mut() {\n            *rank = self.optional_index.select(*rank);\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::ops::Range;\n\n    use super::MultiValueIndex;\n    use crate::{ColumnarReader, DynamicColumn};\n\n    fn index_to_pos_helper(\n        index: &MultiValueIndex,\n        doc_id_range: Range<u32>,\n        positions: &[u32],\n    ) -> Vec<u32> {\n        let mut positions = positions.to_vec();\n        index.select_batch_in_place(doc_id_range.start, &mut positions);\n        positions\n    }\n\n    #[test]\n    fn test_positions_to_docid() {\n        let index = MultiValueIndex::for_test(&[0, 10, 12, 15, 22, 23]);\n        assert_eq!(index.num_docs(), 5);\n        let positions = &[10u32, 11, 15, 20, 21, 22];\n        assert_eq!(index_to_pos_helper(&index, 0..5, positions), vec![1, 3, 4]);\n        assert_eq!(index_to_pos_helper(&index, 1..5, positions), vec![1, 3, 4]);\n\n        assert_eq!(index_to_pos_helper(&index, 0..5, &[9]), vec![0]);\n        assert_eq!(index_to_pos_helper(&index, 1..5, &[10]), vec![1]);\n        assert_eq!(index_to_pos_helper(&index, 1..5, &[11]), vec![1]);\n        assert_eq!(index_to_pos_helper(&index, 2..5, &[12]), vec![2]);\n        assert_eq!(index_to_pos_helper(&index, 2..5, &[12, 14]), vec![2]);\n        assert_eq!(index_to_pos_helper(&index, 2..5, &[12, 14, 15]), vec![2, 3]);\n    }\n\n    #[test]\n    fn test_range_to_rowids() {\n        use crate::ColumnarWriter;\n\n        let mut columnar_writer = ColumnarWriter::default();\n\n        // This column gets coerced to u64\n        columnar_writer.record_numerical(1, \"full\", u64::MAX);\n        columnar_writer.record_numerical(1, \"full\", u64::MAX);\n\n        columnar_writer.record_numerical(5, \"full\", u64::MAX);\n        columnar_writer.record_numerical(5, \"full\", u64::MAX);\n\n        let mut wrt: Vec<u8> = Vec::new();\n        columnar_writer.serialize(7, &mut wrt).unwrap();\n\n        let reader = ColumnarReader::open(wrt).unwrap();\n        // Open the column as u64\n        let column = reader.read_columns(\"full\").unwrap()[0]\n            .open()\n            .unwrap()\n            .coerce_numerical(crate::NumericalType::U64)\n            .unwrap();\n        let DynamicColumn::U64(column) = column else {\n            panic!();\n        };\n\n        let row_id_range = column.index.docid_range_to_rowids(1..2);\n        assert_eq!(row_id_range, 0..2);\n\n        let row_id_range = column.index.docid_range_to_rowids(0..2);\n        assert_eq!(row_id_range, 0..2);\n\n        let row_id_range = column.index.docid_range_to_rowids(0..4);\n        assert_eq!(row_id_range, 0..2);\n\n        let row_id_range = column.index.docid_range_to_rowids(3..4);\n        assert_eq!(row_id_range, 2..2);\n\n        let row_id_range = column.index.docid_range_to_rowids(1..6);\n        assert_eq!(row_id_range, 0..4);\n\n        let row_id_range = column.index.docid_range_to_rowids(3..6);\n        assert_eq!(row_id_range, 2..4);\n\n        let row_id_range = column.index.docid_range_to_rowids(0..6);\n        assert_eq!(row_id_range, 0..4);\n\n        let row_id_range = column.index.docid_range_to_rowids(0..6);\n        assert_eq!(row_id_range, 0..4);\n\n        let check = |range, expected| {\n            let full_range = 0..=u64::MAX;\n            let mut docids = Vec::new();\n            column.get_docids_for_value_range(full_range, range, &mut docids);\n            assert_eq!(docids, expected);\n        };\n\n        // check(0..1, vec![]);\n        // check(0..2, vec![1]);\n        check(1..2, vec![1]);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/mod.rs",
    "content": "use std::io;\nuse std::sync::Arc;\n\nmod set;\nmod set_block;\n\nuse common::{BinarySerializable, OwnedBytes, VInt};\npub use set::{SelectCursor, Set, SetCodec};\nuse set_block::{\n    DENSE_BLOCK_NUM_BYTES, DenseBlock, DenseBlockCodec, SparseBlock, SparseBlockCodec,\n};\n\nuse crate::iterable::Iterable;\nuse crate::{DocId, RowId};\n\n/// The threshold for for number of elements after which we switch to dense block encoding.\n///\n/// We simply pick the value that minimize the size of the blocks.\nconst DENSE_BLOCK_THRESHOLD: u32 =\n    set_block::DENSE_BLOCK_NUM_BYTES / std::mem::size_of::<u16>() as u32; //< 5_120\n\nconst ELEMENTS_PER_BLOCK: u32 = u16::MAX as u32 + 1;\n\n#[derive(Copy, Clone, Debug)]\nstruct BlockMeta {\n    non_null_rows_before_block: u32,\n    start_byte_offset: u32,\n    block_variant: BlockVariant,\n}\n\n#[derive(Clone, Copy, Debug)]\nenum BlockVariant {\n    Dense,\n    Sparse { num_vals: u16 },\n}\n\nimpl BlockVariant {\n    pub fn empty() -> Self {\n        Self::Sparse { num_vals: 0 }\n    }\n    pub fn num_bytes_in_block(&self) -> u32 {\n        match *self {\n            BlockVariant::Dense => set_block::DENSE_BLOCK_NUM_BYTES,\n            BlockVariant::Sparse { num_vals } => num_vals as u32 * 2,\n        }\n    }\n}\n\n/// This codec is inspired by roaring bitmaps.\n/// In the dense blocks, however, in order to accelerate `select`\n/// we interleave an offset over two bytes. (more on this lower)\n///\n/// The lower 16 bits of doc ids are stored as u16 while the upper 16 bits are given by the block\n/// id. Each block contains 1<<16 docids.\n///\n/// # Serialized Data Layout\n/// The data starts with the block data. Each block is either dense or sparse encoded, depending on\n/// the number of values in the block. A block is sparse when it contains less than\n/// DENSE_BLOCK_THRESHOLD (6144) values.\n/// [Sparse data block | dense data block, .. #repeat*; Desc: Either a sparse or dense encoded\n/// block]\n/// ### Sparse block data\n/// [u16 LE, .. #repeat*; Desc: Positions with values in a block]\n/// ### Dense block data\n/// [Dense codec for the whole block; Desc: Similar to a bitvec(0..ELEMENTS_PER_BLOCK) + Metadata\n/// for faster lookups. See dense.rs]\n///\n/// The data is followed by block metadata, to know which area of the raw block data belongs to\n/// which block. Only metadata for blocks with elements is recorded to\n/// keep the overhead low for scenarios with many very sparse columns. The block metadata consists\n/// of the block index and the number of values in the block. Since we don't store empty blocks\n/// num_vals is incremented by 1, e.g. 0 means 1 value.\n///\n/// The last u16 is storing the number of metadata blocks.\n/// [u16 LE, .. #repeat*; Desc: Positions with values in a block][(u16 LE, u16 LE), .. #repeat*;\n/// Desc: (Block Id u16, Num Elements u16)][u16 LE; Desc: num blocks with values u16]\n///\n/// # Opening\n/// When opening the data layout, the data is expanded to `Vec<SparseCodecBlockVariant>`, where the\n/// index is the block index. For each block `byte_start` and `offset` is computed.\n#[derive(Clone)]\npub struct OptionalIndex {\n    num_docs: RowId,\n    num_non_null_docs: RowId,\n    block_data: OwnedBytes,\n    block_metas: Arc<[BlockMeta]>,\n}\n\nimpl Iterable<u32> for &OptionalIndex {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u32> + '_> {\n        Box::new(self.iter_non_null_docs())\n    }\n}\n\nimpl std::fmt::Debug for OptionalIndex {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        f.debug_struct(\"OptionalIndex\")\n            .field(\"num_docs\", &self.num_docs)\n            .field(\"num_non_null_docs\", &self.num_non_null_docs)\n            .finish_non_exhaustive()\n    }\n}\n\n/// Splits a value address into lower and upper 16bits.\n/// The lower 16 bits are the value in the block\n/// The upper 16 bits are the block index\n#[derive(Copy, Debug, Clone)]\nstruct RowAddr {\n    block_id: u16,\n    in_block_row_id: u16,\n}\n\n#[inline(always)]\nfn row_addr_from_row_id(row_id: RowId) -> RowAddr {\n    RowAddr {\n        block_id: (row_id / ELEMENTS_PER_BLOCK) as u16,\n        in_block_row_id: (row_id % ELEMENTS_PER_BLOCK) as u16,\n    }\n}\n\nenum BlockSelectCursor<'a> {\n    Dense(<DenseBlock<'a> as Set<u16>>::SelectCursor<'a>),\n    Sparse(<SparseBlock<'a> as Set<u16>>::SelectCursor<'a>),\n}\n\nimpl BlockSelectCursor<'_> {\n    fn select(&mut self, rank: u16) -> u16 {\n        match self {\n            BlockSelectCursor::Dense(dense_select_cursor) => dense_select_cursor.select(rank),\n            BlockSelectCursor::Sparse(sparse_select_cursor) => sparse_select_cursor.select(rank),\n        }\n    }\n}\npub struct OptionalIndexSelectCursor<'a> {\n    current_block_cursor: BlockSelectCursor<'a>,\n    current_block_id: u16,\n    // The current block is guaranteed to contain ranks < end_rank.\n    current_block_end_rank: RowId,\n    optional_index: &'a OptionalIndex,\n    block_doc_idx_start: RowId,\n    num_null_rows_before_block: RowId,\n}\n\nimpl OptionalIndexSelectCursor<'_> {\n    fn search_and_load_block(&mut self, rank: RowId) {\n        if rank < self.current_block_end_rank {\n            // we are already in the right block\n            return;\n        }\n        self.current_block_id = self.optional_index.find_block(rank, self.current_block_id);\n        self.current_block_end_rank = self\n            .optional_index\n            .block_metas\n            .get(self.current_block_id as usize + 1)\n            .map(|block_meta| block_meta.non_null_rows_before_block)\n            .unwrap_or(u32::MAX);\n        self.block_doc_idx_start = (self.current_block_id as u32) * ELEMENTS_PER_BLOCK;\n        let block_meta = self.optional_index.block_metas[self.current_block_id as usize];\n        self.num_null_rows_before_block = block_meta.non_null_rows_before_block;\n        let block: Block<'_> = self.optional_index.block(block_meta);\n        self.current_block_cursor = match block {\n            Block::Dense(dense_block) => BlockSelectCursor::Dense(dense_block.select_cursor()),\n            Block::Sparse(sparse_block) => BlockSelectCursor::Sparse(sparse_block.select_cursor()),\n        };\n    }\n}\n\nimpl SelectCursor<RowId> for OptionalIndexSelectCursor<'_> {\n    fn select(&mut self, rank: RowId) -> RowId {\n        self.search_and_load_block(rank);\n        let index_in_block = (rank - self.num_null_rows_before_block) as u16;\n        self.current_block_cursor.select(index_in_block) as RowId + self.block_doc_idx_start\n    }\n}\n\nimpl Set<RowId> for OptionalIndex {\n    type SelectCursor<'b>\n        = OptionalIndexSelectCursor<'b>\n    where Self: 'b;\n    // Check if value at position is not null.\n    #[inline]\n    fn contains(&self, row_id: RowId) -> bool {\n        let RowAddr {\n            block_id,\n            in_block_row_id,\n        } = row_addr_from_row_id(row_id);\n        let block_meta = self.block_metas[block_id as usize];\n        match self.block(block_meta) {\n            Block::Dense(dense_block) => dense_block.contains(in_block_row_id),\n            Block::Sparse(sparse_block) => sparse_block.contains(in_block_row_id),\n        }\n    }\n\n    /// Any value doc_id is allowed.\n    /// In particular, doc_id = num_rows.\n    #[inline]\n    fn rank(&self, doc_id: DocId) -> RowId {\n        if doc_id >= self.num_docs() {\n            return self.num_non_nulls();\n        }\n        let RowAddr {\n            block_id,\n            in_block_row_id,\n        } = row_addr_from_row_id(doc_id);\n        let block_meta = self.block_metas[block_id as usize];\n        let block = self.block(block_meta);\n\n        let block_offset_row_id = match block {\n            Block::Dense(dense_block) => dense_block.rank(in_block_row_id),\n            Block::Sparse(sparse_block) => sparse_block.rank(in_block_row_id),\n        } as u32;\n        block_meta.non_null_rows_before_block + block_offset_row_id\n    }\n\n    /// Any value doc_id is allowed.\n    /// In particular, doc_id = num_rows.\n    #[inline]\n    fn rank_if_exists(&self, doc_id: DocId) -> Option<RowId> {\n        let RowAddr {\n            block_id,\n            in_block_row_id,\n        } = row_addr_from_row_id(doc_id);\n        let block_meta = *self.block_metas.get(block_id as usize)?;\n        let block = self.block(block_meta);\n        let block_offset_row_id = match block {\n            Block::Dense(dense_block) => dense_block.rank_if_exists(in_block_row_id),\n            Block::Sparse(sparse_block) => sparse_block.rank_if_exists(in_block_row_id),\n        }? as u32;\n        Some(block_meta.non_null_rows_before_block + block_offset_row_id)\n    }\n\n    #[inline]\n    fn select(&self, rank: RowId) -> RowId {\n        let block_pos = self.find_block(rank, 0);\n        let block_doc_idx_start = (block_pos as u32) * ELEMENTS_PER_BLOCK;\n        let block_meta = self.block_metas[block_pos as usize];\n        let block: Block<'_> = self.block(block_meta);\n        let index_in_block = (rank - block_meta.non_null_rows_before_block) as u16;\n        let in_block_rank = match block {\n            Block::Dense(dense_block) => dense_block.select(index_in_block),\n            Block::Sparse(sparse_block) => sparse_block.select(index_in_block),\n        };\n        block_doc_idx_start + in_block_rank as u32\n    }\n\n    fn select_cursor(&self) -> OptionalIndexSelectCursor<'_> {\n        OptionalIndexSelectCursor {\n            current_block_cursor: BlockSelectCursor::Sparse(\n                SparseBlockCodec::open(b\"\").select_cursor(),\n            ),\n            current_block_id: 0u16,\n            current_block_end_rank: 0u32, //< this is sufficient to force the first load\n            optional_index: self,\n            block_doc_idx_start: 0u32,\n            num_null_rows_before_block: 0u32,\n        }\n    }\n}\n\nimpl OptionalIndex {\n    pub fn for_test(num_rows: RowId, row_ids: &[RowId]) -> OptionalIndex {\n        assert!(\n            row_ids\n                .last()\n                .copied()\n                .map(|last_row_id| last_row_id < num_rows)\n                .unwrap_or(true)\n        );\n        let mut buffer = Vec::new();\n        serialize_optional_index(&row_ids, num_rows, &mut buffer).unwrap();\n        let bytes = OwnedBytes::new(buffer);\n        open_optional_index(bytes).unwrap()\n    }\n\n    pub fn num_docs(&self) -> RowId {\n        self.num_docs\n    }\n\n    pub fn num_non_nulls(&self) -> RowId {\n        self.num_non_null_docs\n    }\n\n    pub fn iter_non_null_docs(&self) -> impl Iterator<Item = RowId> + '_ {\n        // TODO optimize. We could iterate over the blocks directly.\n        // We use the dense value ids and retrieve the doc ids via select.\n        let mut select_batch = self.select_cursor();\n        (0..self.num_non_null_docs).map(move |rank| select_batch.select(rank))\n    }\n    pub fn select_batch(&self, ranks: &mut [RowId]) {\n        let mut select_cursor = self.select_cursor();\n        for rank in ranks.iter_mut() {\n            *rank = select_cursor.select(*rank);\n        }\n    }\n\n    #[inline]\n    fn block(&self, block_meta: BlockMeta) -> Block<'_> {\n        let BlockMeta {\n            start_byte_offset,\n            block_variant,\n            ..\n        } = block_meta;\n        let start_byte_offset = start_byte_offset as usize;\n        let bytes = self.block_data.as_slice();\n        match block_variant {\n            BlockVariant::Dense => Block::Dense(DenseBlockCodec::open(\n                &bytes[start_byte_offset..start_byte_offset + DENSE_BLOCK_NUM_BYTES as usize],\n            )),\n            BlockVariant::Sparse { num_vals } => {\n                let end_byte_offset = start_byte_offset + num_vals as usize * 2;\n                let sparse_bytes = &bytes[start_byte_offset..end_byte_offset];\n                Block::Sparse(SparseBlockCodec::open(sparse_bytes))\n            }\n        }\n    }\n\n    #[inline]\n    fn find_block(&self, dense_idx: u32, start_block_pos: u16) -> u16 {\n        for block_pos in start_block_pos..self.block_metas.len() as u16 {\n            let offset = self.block_metas[block_pos as usize].non_null_rows_before_block;\n            if offset > dense_idx {\n                return block_pos - 1u16;\n            }\n        }\n        self.block_metas.len() as u16 - 1u16\n    }\n\n    // TODO Add a good API for the codec_idx to original_idx translation.\n    // The Iterator API is a probably a bad idea\n}\n\n#[derive(Copy, Clone)]\nenum Block<'a> {\n    Dense(DenseBlock<'a>),\n    Sparse(SparseBlock<'a>),\n}\n\nfn serialize_optional_index_block(block_els: &[u16], out: &mut impl io::Write) -> io::Result<()> {\n    let is_sparse = is_sparse(block_els.len() as u32);\n    if is_sparse {\n        SparseBlockCodec::serialize(block_els.iter().copied(), out)?;\n    } else {\n        DenseBlockCodec::serialize(block_els.iter().copied(), out)?;\n    }\n    Ok(())\n}\n\npub fn serialize_optional_index<W: io::Write>(\n    non_null_rows: &dyn Iterable<RowId>,\n    num_rows: RowId,\n    output: &mut W,\n) -> io::Result<()> {\n    VInt(num_rows as u64).serialize(output)?;\n\n    let mut rows_it = non_null_rows.boxed_iter();\n    let mut block_metadata: Vec<SerializedBlockMeta> = Vec::new();\n    let mut current_block = Vec::new();\n\n    // This if-statement for the first element ensures that\n    // `block_metadata` is not empty in the loop below.\n    let Some(idx) = rows_it.next() else {\n        output.write_all(&0u16.to_le_bytes())?;\n        return Ok(());\n    };\n\n    let row_addr = row_addr_from_row_id(idx);\n\n    let mut current_block_id = row_addr.block_id;\n    current_block.push(row_addr.in_block_row_id);\n\n    for idx in rows_it {\n        let value_addr = row_addr_from_row_id(idx);\n        if current_block_id != value_addr.block_id {\n            serialize_optional_index_block(&current_block[..], output)?;\n            block_metadata.push(SerializedBlockMeta {\n                block_id: current_block_id,\n                num_non_null_rows: current_block.len() as u32,\n            });\n            current_block.clear();\n            current_block_id = value_addr.block_id;\n        }\n        current_block.push(value_addr.in_block_row_id);\n    }\n\n    // handle last block\n    serialize_optional_index_block(&current_block[..], output)?;\n\n    block_metadata.push(SerializedBlockMeta {\n        block_id: current_block_id,\n        num_non_null_rows: current_block.len() as u32,\n    });\n\n    for block in &block_metadata {\n        output.write_all(&block.to_bytes())?;\n    }\n\n    output.write_all((block_metadata.len() as u16).to_le_bytes().as_ref())?;\n\n    Ok(())\n}\n\nconst SERIALIZED_BLOCK_META_NUM_BYTES: usize = 4;\n\n#[derive(Clone, Copy, Debug)]\nstruct SerializedBlockMeta {\n    block_id: u16,\n    num_non_null_rows: u32, //< takes values in 1..=u16::MAX\n}\n\n// TODO unit tests\nimpl SerializedBlockMeta {\n    #[inline]\n    fn from_bytes(bytes: [u8; SERIALIZED_BLOCK_META_NUM_BYTES]) -> SerializedBlockMeta {\n        let block_id = u16::from_le_bytes(bytes[0..2].try_into().unwrap());\n        let num_non_null_rows: u32 =\n            u16::from_le_bytes(bytes[2..4].try_into().unwrap()) as u32 + 1u32;\n        SerializedBlockMeta {\n            block_id,\n            num_non_null_rows,\n        }\n    }\n\n    #[inline]\n    fn to_bytes(self) -> [u8; SERIALIZED_BLOCK_META_NUM_BYTES] {\n        assert!(self.num_non_null_rows > 0);\n        let mut bytes = [0u8; SERIALIZED_BLOCK_META_NUM_BYTES];\n        bytes[0..2].copy_from_slice(&self.block_id.to_le_bytes());\n        // We don't store empty blocks, therefore we can subtract 1.\n        // This way we will be able to use u16 when the number of elements is 1 << 16 or u16::MAX+1\n        bytes[2..4].copy_from_slice(&((self.num_non_null_rows - 1u32) as u16).to_le_bytes());\n        bytes\n    }\n}\n\n#[inline]\nfn is_sparse(num_rows_in_block: u32) -> bool {\n    num_rows_in_block < DENSE_BLOCK_THRESHOLD\n}\n\nfn deserialize_optional_index_block_metadatas(\n    data: &[u8],\n    num_rows: u32,\n) -> (Box<[BlockMeta]>, u32) {\n    let num_blocks = data.len() / SERIALIZED_BLOCK_META_NUM_BYTES;\n    let mut block_metas = Vec::with_capacity(num_blocks + 1);\n    let mut start_byte_offset = 0;\n    let mut non_null_rows_before_block = 0;\n    for block_meta_bytes in data.chunks_exact(SERIALIZED_BLOCK_META_NUM_BYTES) {\n        let block_meta_bytes: [u8; SERIALIZED_BLOCK_META_NUM_BYTES] =\n            block_meta_bytes.try_into().unwrap();\n        let SerializedBlockMeta {\n            block_id,\n            num_non_null_rows,\n        } = SerializedBlockMeta::from_bytes(block_meta_bytes);\n        block_metas.resize(\n            block_id as usize,\n            BlockMeta {\n                non_null_rows_before_block,\n                start_byte_offset,\n                block_variant: BlockVariant::empty(),\n            },\n        );\n        let block_variant = if is_sparse(num_non_null_rows) {\n            BlockVariant::Sparse {\n                num_vals: num_non_null_rows as u16,\n            }\n        } else {\n            BlockVariant::Dense\n        };\n        block_metas.push(BlockMeta {\n            non_null_rows_before_block,\n            start_byte_offset,\n            block_variant,\n        });\n        start_byte_offset += block_variant.num_bytes_in_block();\n        non_null_rows_before_block += num_non_null_rows;\n    }\n    block_metas.resize(\n        num_rows.div_ceil(ELEMENTS_PER_BLOCK) as usize,\n        BlockMeta {\n            non_null_rows_before_block,\n            start_byte_offset,\n            block_variant: BlockVariant::empty(),\n        },\n    );\n    (block_metas.into_boxed_slice(), non_null_rows_before_block)\n}\n\npub fn open_optional_index(bytes: OwnedBytes) -> io::Result<OptionalIndex> {\n    let (mut bytes, num_non_empty_blocks_bytes) = bytes.rsplit(2);\n    let num_non_empty_block_bytes =\n        u16::from_le_bytes(num_non_empty_blocks_bytes.as_slice().try_into().unwrap());\n    let num_docs = VInt::deserialize_u64(&mut bytes)? as u32;\n    let block_metas_num_bytes =\n        num_non_empty_block_bytes as usize * SERIALIZED_BLOCK_META_NUM_BYTES;\n    let (block_data, block_metas) = bytes.rsplit(block_metas_num_bytes);\n    let (block_metas, num_non_null_docs) =\n        deserialize_optional_index_block_metadatas(block_metas.as_slice(), num_docs);\n    let optional_index = OptionalIndex {\n        num_docs,\n        num_non_null_docs,\n        block_data,\n        block_metas: block_metas.into(),\n    };\n    Ok(optional_index)\n}\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/set.rs",
    "content": "use std::io;\n\n/// A codec makes it possible to serialize a set of\n/// elements, and open the resulting Set representation.\npub trait SetCodec {\n    type Item: Copy + TryFrom<usize> + Eq + std::hash::Hash + std::fmt::Debug;\n    type Reader<'a>: Set<Self::Item>;\n\n    /// Serializes a set of unique sorted u16 elements.\n    ///\n    /// May panic if the elements are not sorted.\n    fn serialize(els: impl Iterator<Item = Self::Item>, wrt: impl io::Write) -> io::Result<()>;\n    fn open(data: &[u8]) -> Self::Reader<'_>;\n}\n\n/// Stateful object that makes it possible to compute several select in a row,\n/// provided the rank passed as argument are increasing.\npub trait SelectCursor<T> {\n    // May panic if rank is greater than the number of elements in the Set,\n    // or if rank is < than value provided in the previous call.\n    fn select(&mut self, rank: T) -> T;\n}\n\npub trait Set<T> {\n    type SelectCursor<'b>: SelectCursor<T>\n    where Self: 'b;\n\n    /// Returns true if the elements is contained in the Set\n    fn contains(&self, el: T) -> bool;\n\n    /// Returns the element's rank (its position in the set).\n    /// If the set does not contain the element, it will return the next existing elements rank.\n    fn rank(&self, el: T) -> T;\n\n    /// If the set contains `el`, returns the element's rank (its position in the set).\n    /// If the set does not contain the element, it returns `None`.\n    fn rank_if_exists(&self, el: T) -> Option<T>;\n\n    /// Return the rank-th value stored in this bitmap.\n    ///\n    /// # Panics\n    ///\n    /// May panic if rank is greater or equal to the number of\n    /// elements in the Set.\n    fn select(&self, rank: T) -> T;\n\n    /// Creates a brand new select cursor.\n    fn select_cursor(&self) -> Self::SelectCursor<'_>;\n}\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/set_block/dense.rs",
    "content": "use std::io::{self, Write};\n\nuse common::BinarySerializable;\n\nuse crate::column_index::optional_index::{ELEMENTS_PER_BLOCK, SelectCursor, Set, SetCodec};\n\n#[inline(always)]\nfn get_bit_at(input: u64, n: u16) -> bool {\n    input & (1 << n) != 0\n}\n\n#[inline]\nfn set_bit_at(input: &mut u64, n: u16) {\n    *input |= 1 << n;\n}\n\n/// For the `DenseCodec`, `data` which contains the encoded blocks.\n/// Each block consists of [u8; 12]. The first 8 bytes is a bitvec for 64 elements.\n/// The last 4 bytes are the offset, the number of set bits so far.\n///\n/// When translating the original index to a dense index, the correct block can be computed\n/// directly `orig_idx/64`. Inside the block the position is `orig_idx%64`.\n///\n/// When translating a dense index to the original index, we can use the offset to find the correct\n/// block. Direct computation is not possible, but we can employ a linear or binary search.\nconst ELEMENTS_PER_MINI_BLOCK: u16 = 64;\nconst MINI_BLOCK_BITVEC_NUM_BYTES: usize = 8;\nconst MINI_BLOCK_OFFSET_NUM_BYTES: usize = 2;\npub const MINI_BLOCK_NUM_BYTES: usize = MINI_BLOCK_BITVEC_NUM_BYTES + MINI_BLOCK_OFFSET_NUM_BYTES;\n\n/// Number of bytes in a dense block.\npub const DENSE_BLOCK_NUM_BYTES: u32 =\n    (ELEMENTS_PER_BLOCK / ELEMENTS_PER_MINI_BLOCK as u32) * MINI_BLOCK_NUM_BYTES as u32;\n\npub struct DenseBlockCodec;\n\nimpl SetCodec for DenseBlockCodec {\n    type Item = u16;\n    type Reader<'a> = DenseBlock<'a>;\n\n    fn serialize(els: impl Iterator<Item = u16>, wrt: impl io::Write) -> io::Result<()> {\n        serialize_dense_codec(els, wrt)\n    }\n\n    #[inline]\n    fn open(data: &[u8]) -> Self::Reader<'_> {\n        assert_eq!(data.len(), DENSE_BLOCK_NUM_BYTES as usize);\n        DenseBlock(data)\n    }\n}\n\n/// Interpreting the bitvec as a set of integer within 0..=63\n/// and given an element, returns the number of elements in the\n/// set lesser than the element.\n///\n/// # Panics\n///\n/// May panic or return a wrong result if el <= 64.\n#[inline(always)]\nfn rank_u64(bitvec: u64, el: u16) -> u16 {\n    debug_assert!(el < 64);\n    let mask = (1u64 << el) - 1;\n    let masked_bitvec = bitvec & mask;\n    masked_bitvec.count_ones() as u16\n}\n\n#[inline(always)]\nfn select_u64(mut bitvec: u64, rank: u16) -> u16 {\n    for _ in 0..rank {\n        bitvec &= bitvec - 1;\n    }\n    bitvec.trailing_zeros() as u16\n}\n\n// TODO test the following solution on Intel... on Ryzen Zen <3 it is a catastrophy.\n// #[target_feature(enable = \"bmi2\")]\n// unsafe fn select_bitvec_unsafe(bitvec: u64, rank: u16) -> u16 {\n//     let pdep = _pdep_u64(1u64 << rank, bitvec);\n//     pdep.trailing_zeros() as u16\n// }\n\n#[derive(Clone, Copy, Debug)]\nstruct DenseMiniBlock {\n    bitvec: u64,\n    rank: u16,\n}\n\nimpl DenseMiniBlock {\n    fn from_bytes(data: [u8; MINI_BLOCK_NUM_BYTES]) -> Self {\n        let bitvec = u64::from_le_bytes(data[..MINI_BLOCK_BITVEC_NUM_BYTES].try_into().unwrap());\n        let rank = u16::from_le_bytes(data[MINI_BLOCK_BITVEC_NUM_BYTES..].try_into().unwrap());\n        Self { bitvec, rank }\n    }\n\n    fn to_bytes(self) -> [u8; MINI_BLOCK_NUM_BYTES] {\n        let mut bytes = [0u8; MINI_BLOCK_NUM_BYTES];\n        bytes[..MINI_BLOCK_BITVEC_NUM_BYTES].copy_from_slice(&self.bitvec.to_le_bytes());\n        bytes[MINI_BLOCK_BITVEC_NUM_BYTES..].copy_from_slice(&self.rank.to_le_bytes());\n        bytes\n    }\n}\n\n#[derive(Copy, Clone)]\npub struct DenseBlock<'a>(&'a [u8]);\n\npub struct DenseBlockSelectCursor<'a> {\n    block_id: u16,\n    dense_block: DenseBlock<'a>,\n}\n\nimpl SelectCursor<u16> for DenseBlockSelectCursor<'_> {\n    #[inline]\n    fn select(&mut self, rank: u16) -> u16 {\n        self.block_id = self\n            .dense_block\n            .find_miniblock_containing_rank(rank, self.block_id)\n            .unwrap();\n        let index_block = self.dense_block.mini_block(self.block_id);\n        let in_block_rank = rank - index_block.rank;\n        self.block_id * ELEMENTS_PER_MINI_BLOCK + select_u64(index_block.bitvec, in_block_rank)\n    }\n}\n\nimpl<'a> Set<u16> for DenseBlock<'a> {\n    type SelectCursor<'b>\n        = DenseBlockSelectCursor<'a>\n    where Self: 'b;\n\n    #[inline(always)]\n    fn contains(&self, el: u16) -> bool {\n        let mini_block_id = el / ELEMENTS_PER_MINI_BLOCK;\n        let bitvec = self.mini_block(mini_block_id).bitvec;\n        let pos_in_bitvec = el % ELEMENTS_PER_MINI_BLOCK;\n        get_bit_at(bitvec, pos_in_bitvec)\n    }\n\n    #[inline(always)]\n    fn rank_if_exists(&self, el: u16) -> Option<u16> {\n        let block_pos = el / ELEMENTS_PER_MINI_BLOCK;\n        let index_block = self.mini_block(block_pos);\n        let pos_in_block_bit_vec = el % ELEMENTS_PER_MINI_BLOCK;\n        let ones_in_block = rank_u64(index_block.bitvec, pos_in_block_bit_vec);\n        let rank = index_block.rank + ones_in_block;\n        if get_bit_at(index_block.bitvec, pos_in_block_bit_vec) {\n            Some(rank)\n        } else {\n            None\n        }\n    }\n\n    #[inline(always)]\n    fn rank(&self, el: u16) -> u16 {\n        let block_pos = el / ELEMENTS_PER_MINI_BLOCK;\n        let index_block = self.mini_block(block_pos);\n        let pos_in_block_bit_vec = el % ELEMENTS_PER_MINI_BLOCK;\n        let ones_in_block = rank_u64(index_block.bitvec, pos_in_block_bit_vec);\n        index_block.rank + ones_in_block\n    }\n\n    #[inline(always)]\n    fn select(&self, rank: u16) -> u16 {\n        let block_id = self.find_miniblock_containing_rank(rank, 0).unwrap();\n        let index_block = self.mini_block(block_id);\n        let in_block_rank = rank - index_block.rank;\n        block_id * ELEMENTS_PER_MINI_BLOCK + select_u64(index_block.bitvec, in_block_rank)\n    }\n\n    #[inline(always)]\n    fn select_cursor(&self) -> Self::SelectCursor<'_> {\n        DenseBlockSelectCursor {\n            block_id: 0,\n            dense_block: *self,\n        }\n    }\n}\n\nimpl DenseBlock<'_> {\n    #[inline]\n    fn mini_block(&self, mini_block_id: u16) -> DenseMiniBlock {\n        let data_start_pos = mini_block_id as usize * MINI_BLOCK_NUM_BYTES;\n        DenseMiniBlock::from_bytes(\n            self.0[data_start_pos..data_start_pos + MINI_BLOCK_NUM_BYTES]\n                .try_into()\n                .unwrap(),\n        )\n    }\n\n    #[inline]\n    fn iter_miniblocks(\n        &self,\n        from_block_id: u16,\n    ) -> impl Iterator<Item = (u16, DenseMiniBlock)> + '_ {\n        self.0\n            .chunks_exact(MINI_BLOCK_NUM_BYTES)\n            .enumerate()\n            .skip(from_block_id as usize)\n            .map(|(block_id, bytes)| {\n                let mini_block = DenseMiniBlock::from_bytes(bytes.try_into().unwrap());\n                (block_id as u16, mini_block)\n            })\n    }\n\n    /// Finds the block position containing the dense_idx.\n    ///\n    /// # Correctness\n    /// dense_idx needs to be smaller than the number of values in the index\n    ///\n    /// The last offset number is equal to the number of values in the index.\n    #[inline]\n    fn find_miniblock_containing_rank(&self, rank: u16, from_block_id: u16) -> Option<u16> {\n        self.iter_miniblocks(from_block_id)\n            .take_while(|(_, block)| block.rank <= rank)\n            .map(|(block_id, _)| block_id)\n            .last()\n    }\n}\n\n/// Iterator over all values, true if set, otherwise false\npub fn serialize_dense_codec(\n    els: impl Iterator<Item = u16>,\n    mut output: impl Write,\n) -> io::Result<()> {\n    let mut non_null_rows_before: u16 = 0u16;\n    let mut block = 0u64;\n    let mut current_block_id = 0u16;\n    for el in els {\n        let block_id = el / ELEMENTS_PER_MINI_BLOCK;\n        let in_offset = el % ELEMENTS_PER_MINI_BLOCK;\n        while block_id > current_block_id {\n            let dense_mini_block = DenseMiniBlock {\n                bitvec: block,\n                rank: non_null_rows_before,\n            };\n            output.write_all(&dense_mini_block.to_bytes())?;\n            non_null_rows_before += block.count_ones() as u16;\n            block = 0u64;\n            current_block_id += 1u16;\n        }\n        set_bit_at(&mut block, in_offset);\n    }\n    while current_block_id <= u16::MAX / ELEMENTS_PER_MINI_BLOCK {\n        block.serialize(&mut output)?;\n        non_null_rows_before.serialize(&mut output)?;\n        // This will overflow to 0 exactly if all bits are set.\n        // This is however not problem as we won't use this last value.\n        non_null_rows_before = non_null_rows_before.wrapping_add(block.count_ones() as u16);\n        block = 0u64;\n        current_block_id += 1u16;\n    }\n    Ok(())\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_select_bitvec() {\n        assert_eq!(select_u64(1u64, 0), 0);\n        assert_eq!(select_u64(2u64, 0), 1);\n        assert_eq!(select_u64(4u64, 0), 2);\n        assert_eq!(select_u64(8u64, 0), 3);\n        assert_eq!(select_u64(1 | 8u64, 0), 0);\n        assert_eq!(select_u64(1 | 8u64, 1), 3);\n    }\n\n    #[test]\n    fn test_count_ones() {\n        for i in 0..=63 {\n            assert_eq!(rank_u64(u64::MAX, i), i);\n        }\n    }\n\n    #[test]\n    fn test_dense() {\n        assert_eq!(DENSE_BLOCK_NUM_BYTES, 10_240);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/set_block/mod.rs",
    "content": "mod dense;\nmod sparse;\n\npub use dense::{DENSE_BLOCK_NUM_BYTES, DenseBlock, DenseBlockCodec};\npub use sparse::{SparseBlock, SparseBlockCodec};\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/set_block/sparse.rs",
    "content": "use crate::column_index::optional_index::{SelectCursor, Set, SetCodec};\n\npub struct SparseBlockCodec;\n\nimpl SetCodec for SparseBlockCodec {\n    type Item = u16;\n    type Reader<'a> = SparseBlock<'a>;\n\n    fn serialize(\n        els: impl Iterator<Item = u16>,\n        mut wrt: impl std::io::Write,\n    ) -> std::io::Result<()> {\n        for el in els {\n            wrt.write_all(&el.to_le_bytes())?;\n        }\n        Ok(())\n    }\n\n    fn open(data: &[u8]) -> Self::Reader<'_> {\n        SparseBlock(data)\n    }\n}\n\n#[derive(Copy, Clone)]\npub struct SparseBlock<'a>(&'a [u8]);\n\nimpl<'a> SelectCursor<u16> for SparseBlock<'a> {\n    #[inline]\n    fn select(&mut self, rank: u16) -> u16 {\n        <SparseBlock<'a> as Set<u16>>::select(self, rank)\n    }\n}\n\nimpl Set<u16> for SparseBlock<'_> {\n    type SelectCursor<'b>\n        = Self\n    where Self: 'b;\n\n    #[inline(always)]\n    fn contains(&self, el: u16) -> bool {\n        self.binary_search(el).is_ok()\n    }\n\n    #[inline(always)]\n    fn rank_if_exists(&self, el: u16) -> Option<u16> {\n        self.binary_search(el).ok()\n    }\n\n    #[inline(always)]\n    fn rank(&self, el: u16) -> u16 {\n        self.binary_search(el).unwrap_or_else(|el| el)\n    }\n\n    #[inline(always)]\n    fn select(&self, rank: u16) -> u16 {\n        let offset = rank as usize * 2;\n        u16::from_le_bytes(self.0[offset..offset + 2].try_into().unwrap())\n    }\n\n    #[inline(always)]\n    fn select_cursor(&self) -> Self::SelectCursor<'_> {\n        *self\n    }\n}\n\n#[inline(always)]\nfn get_u16(data: &[u8], byte_position: usize) -> u16 {\n    let bytes: [u8; 2] = data[byte_position..byte_position + 2].try_into().unwrap();\n    u16::from_le_bytes(bytes)\n}\n\nimpl SparseBlock<'_> {\n    #[inline(always)]\n    fn value_at_idx(&self, data: &[u8], idx: u16) -> u16 {\n        let start_offset: usize = idx as usize * 2;\n        get_u16(data, start_offset)\n    }\n\n    #[inline]\n    fn num_vals(&self) -> u16 {\n        (self.0.len() / 2) as u16\n    }\n\n    #[inline]\n    #[expect(clippy::comparison_chain)]\n    // Looks for the element in the block. Returns the positions if found.\n    fn binary_search(&self, target: u16) -> Result<u16, u16> {\n        let data = &self.0;\n        let mut size = self.num_vals();\n        let mut left = 0;\n        let mut right = size;\n        // TODO try different implem.\n        //  e.g. exponential search into binary search\n        while left < right {\n            let mid = left + size / 2;\n\n            // TODO do boundary check only once, and then use an\n            // unsafe `value_at_idx`\n            let mid_val = self.value_at_idx(data, mid);\n\n            if target > mid_val {\n                left = mid + 1;\n            } else if target < mid_val {\n                right = mid;\n            } else {\n                return Ok(mid);\n            }\n\n            size = right - left;\n        }\n        Err(left)\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/set_block/tests.rs",
    "content": "use std::collections::HashMap;\n\nuse crate::column_index::optional_index::set_block::dense::DENSE_BLOCK_NUM_BYTES;\nuse crate::column_index::optional_index::set_block::{DenseBlockCodec, SparseBlockCodec};\nuse crate::column_index::optional_index::{SelectCursor, Set, SetCodec};\n\nfn test_set_helper<C: SetCodec<Item = u16>>(vals: &[u16]) -> usize {\n    let mut buffer = Vec::new();\n    C::serialize(vals.iter().copied(), &mut buffer).unwrap();\n    let tested_set = C::open(buffer.as_slice());\n    let hash_set: HashMap<C::Item, C::Item> = vals\n        .iter()\n        .copied()\n        .enumerate()\n        .map(|(ord, val)| (val, C::Item::try_from(ord).ok().unwrap()))\n        .collect();\n    for val in 0u16..=u16::MAX {\n        assert_eq!(tested_set.contains(val), hash_set.contains_key(&val));\n        assert_eq!(tested_set.rank_if_exists(val), hash_set.get(&val).copied());\n        assert_eq!(\n            tested_set.rank(val),\n            vals.iter().cloned().take_while(|v| *v < val).count() as u16\n        );\n    }\n    for (rank, val) in vals.iter().enumerate() {\n        assert_eq!(tested_set.select(rank as u16), *val);\n    }\n    buffer.len()\n}\n\n#[test]\nfn test_dense_block_set_u16_empty() {\n    let buffer_len = test_set_helper::<DenseBlockCodec>(&[]);\n    assert_eq!(buffer_len, DENSE_BLOCK_NUM_BYTES as usize);\n}\n\n#[test]\nfn test_dense_block_set_u16_max() {\n    let buffer_len = test_set_helper::<DenseBlockCodec>(&[u16::MAX]);\n    assert_eq!(buffer_len, DENSE_BLOCK_NUM_BYTES as usize);\n}\n\n#[test]\nfn test_sparse_block_set_u16_empty() {\n    let buffer_len = test_set_helper::<SparseBlockCodec>(&[]);\n    assert_eq!(buffer_len, 0);\n}\n\n#[test]\nfn test_sparse_block_set_u16_max() {\n    let buffer_len = test_set_helper::<SparseBlockCodec>(&[u16::MAX]);\n    assert_eq!(buffer_len, 2);\n}\n\nuse proptest::prelude::*;\n\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(1))]\n    #[test]\n    fn test_prop_test_dense(els in proptest::collection::btree_set(0..=u16::MAX, 0..=u16::MAX as usize)) {\n        let vals: Vec<u16> = els.into_iter().collect();\n        let buffer_len = test_set_helper::<DenseBlockCodec>(&vals);\n        assert_eq!(buffer_len, DENSE_BLOCK_NUM_BYTES as usize);\n    }\n\n    #[test]\n    fn test_prop_test_sparse(els in proptest::collection::btree_set(0..=u16::MAX, 0..=u16::MAX as usize)) {\n        let vals: Vec<u16> = els.into_iter().collect();\n        let buffer_len = test_set_helper::<SparseBlockCodec>(&vals);\n        assert_eq!(buffer_len, vals.len() * 2);\n    }\n}\n\n#[test]\nfn test_simple_translate_codec_codec_idx_to_original_idx_dense() {\n    let mut buffer = Vec::new();\n    DenseBlockCodec::serialize([1, 3, 17, 32, 30_000, 30_001].iter().copied(), &mut buffer)\n        .unwrap();\n    let tested_set = DenseBlockCodec::open(buffer.as_slice());\n    assert!(tested_set.contains(1));\n    let mut select_cursor = tested_set.select_cursor();\n    assert_eq!(select_cursor.select(0), 1);\n    assert_eq!(select_cursor.select(1), 3);\n    assert_eq!(select_cursor.select(2), 17);\n}\n\n#[test]\nfn test_simple_translate_codec_idx_to_original_idx_sparse() {\n    let mut buffer = Vec::new();\n    SparseBlockCodec::serialize([1, 3, 17].iter().copied(), &mut buffer).unwrap();\n    let tested_set = SparseBlockCodec::open(buffer.as_slice());\n    assert!(tested_set.contains(1));\n    let mut select_cursor = tested_set.select_cursor();\n    assert_eq!(SelectCursor::select(&mut select_cursor, 0), 1);\n    assert_eq!(SelectCursor::select(&mut select_cursor, 1), 3);\n    assert_eq!(SelectCursor::select(&mut select_cursor, 2), 17);\n}\n\n#[test]\nfn test_simple_translate_codec_idx_to_original_idx_dense() {\n    let mut buffer = Vec::new();\n    DenseBlockCodec::serialize(0u16..150u16, &mut buffer).unwrap();\n    let tested_set = DenseBlockCodec::open(buffer.as_slice());\n    assert!(tested_set.contains(1));\n    let mut select_cursor = tested_set.select_cursor();\n    for i in 0..150 {\n        assert_eq!(i, select_cursor.select(i));\n    }\n}\n\n#[test]\nfn test_simple_translate_idx_to_value_idx_dense() {\n    let mut buffer = Vec::new();\n    DenseBlockCodec::serialize([1, 10].iter().copied(), &mut buffer).unwrap();\n    let tested_set = DenseBlockCodec::open(buffer.as_slice());\n    assert!(tested_set.contains(1));\n    assert!(!tested_set.contains(2));\n    assert_eq!(tested_set.rank(0), 0);\n    assert_eq!(tested_set.rank(1), 0);\n    for rank in 2..10 {\n        // ranks that don't exist select the next highest one\n        assert_eq!(tested_set.rank_if_exists(rank), None);\n        assert_eq!(tested_set.rank(rank), 1);\n    }\n    assert_eq!(tested_set.rank(10), 1);\n}\n\n#[test]\nfn test_simple_translate_idx_to_value_idx_sparse() {\n    let mut buffer = Vec::new();\n    SparseBlockCodec::serialize([1, 10].iter().copied(), &mut buffer).unwrap();\n    let tested_set = SparseBlockCodec::open(buffer.as_slice());\n    assert!(tested_set.contains(1));\n    assert!(!tested_set.contains(2));\n    assert_eq!(tested_set.rank(0), 0);\n    assert_eq!(tested_set.select(tested_set.rank(0)), 1);\n    assert_eq!(tested_set.rank(1), 0);\n    assert_eq!(tested_set.select(tested_set.rank(1)), 1);\n    for rank in 2..10 {\n        // ranks that don't exist select the next highest one\n        assert_eq!(tested_set.rank_if_exists(rank), None);\n        assert_eq!(tested_set.rank(rank), 1);\n        assert_eq!(tested_set.select(tested_set.rank(rank)), 10);\n    }\n    assert_eq!(tested_set.rank(10), 1);\n    assert_eq!(tested_set.select(tested_set.rank(10)), 10);\n}\n"
  },
  {
    "path": "columnar/src/column_index/optional_index/tests.rs",
    "content": "use proptest::prelude::*;\nuse proptest::{prop_oneof, proptest};\n\nuse super::*;\nuse crate::{ColumnarReader, ColumnarWriter, DynamicColumnHandle};\n\n#[test]\nfn test_optional_index_bug_2293() {\n    // tests for panic in docid_range_to_rowids for docid == num_docs\n    test_optional_index_with_num_docs(ELEMENTS_PER_BLOCK - 1);\n    test_optional_index_with_num_docs(ELEMENTS_PER_BLOCK);\n    test_optional_index_with_num_docs(ELEMENTS_PER_BLOCK + 1);\n}\nfn test_optional_index_with_num_docs(num_docs: u32) {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_numerical(100, \"score\", 80i64);\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(num_docs, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"score\").unwrap();\n    assert_eq!(cols.len(), 1);\n\n    let col = cols[0].open().unwrap();\n    col.column_index().docid_range_to_rowids(0..num_docs);\n}\n\n#[test]\nfn test_dense_block_threshold() {\n    assert_eq!(super::DENSE_BLOCK_THRESHOLD, 5_120);\n}\n\nfn random_bitvec() -> BoxedStrategy<Vec<bool>> {\n    prop_oneof![\n        1 => prop::collection::vec(proptest::bool::weighted(1.0), 0..100),\n        1 => prop::collection::vec(proptest::bool::weighted(0.00), 0..(ELEMENTS_PER_BLOCK as usize * 3)), // empty blocks\n        1 => prop::collection::vec(proptest::bool::weighted(1.00), 0..(ELEMENTS_PER_BLOCK as usize + 10)), // full block\n        1 => prop::collection::vec(proptest::bool::weighted(0.01), 0..100),\n        1 => prop::collection::vec(proptest::bool::weighted(0.01), 0..u16::MAX as usize),\n        8 => vec![any::<bool>()],\n    ]\n    .boxed()\n}\n\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(50))]\n    #[test]\n    fn test_with_random_bitvecs(bitvec1 in random_bitvec(), bitvec2 in random_bitvec(), bitvec3 in random_bitvec()) {\n        let mut bitvec = Vec::new();\n        bitvec.extend_from_slice(&bitvec1);\n        bitvec.extend_from_slice(&bitvec2);\n        bitvec.extend_from_slice(&bitvec3);\n        test_null_index(&bitvec[..]);\n    }\n}\n\n#[test]\nfn test_with_random_sets_simple() {\n    let vals = 10..ELEMENTS_PER_BLOCK * 2;\n    let mut out: Vec<u8> = Vec::new();\n    serialize_optional_index(&vals, 100, &mut out).unwrap();\n    let null_index = open_optional_index(OwnedBytes::new(out)).unwrap();\n    let ranks: Vec<u32> = (65_472u32..65_473u32).collect();\n    let els: Vec<u32> = ranks.iter().copied().map(|rank| rank + 10).collect();\n    let mut select_cursor = null_index.select_cursor();\n    for (rank, el) in ranks.iter().copied().zip(els.iter().copied()) {\n        assert_eq!(select_cursor.select(rank), el);\n    }\n}\n\n#[test]\nfn test_optional_index_trailing_empty_blocks() {\n    test_null_index(&[false]);\n}\n\n#[test]\nfn test_optional_index_one_block_false() {\n    let mut iter = vec![false; ELEMENTS_PER_BLOCK as usize];\n    iter.push(true);\n    test_null_index(&iter[..]);\n}\n\n#[test]\nfn test_optional_index_one_block_true() {\n    let mut iter = vec![true; ELEMENTS_PER_BLOCK as usize];\n    iter.push(true);\n    test_null_index(&iter[..]);\n}\n\nimpl<'a> Iterable<RowId> for &'a [bool] {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = RowId> + 'a> {\n        Box::new(\n            self.iter()\n                .cloned()\n                .enumerate()\n                .filter(|(_pos, val)| *val)\n                .map(|(pos, _val)| pos as u32),\n        )\n    }\n}\n\nfn test_null_index(data: &[bool]) {\n    let mut out: Vec<u8> = Vec::new();\n    serialize_optional_index(&data, data.len() as RowId, &mut out).unwrap();\n    let null_index = open_optional_index(OwnedBytes::new(out)).unwrap();\n    let orig_idx_with_value: Vec<u32> = data\n        .iter()\n        .enumerate()\n        .filter(|(_pos, val)| **val)\n        .map(|(pos, _val)| pos as u32)\n        .collect();\n    let mut select_iter = null_index.select_cursor();\n    for (i, expected) in orig_idx_with_value.iter().enumerate() {\n        assert_eq!(select_iter.select(i as u32), *expected);\n    }\n\n    let step_size = (orig_idx_with_value.len() / 100).max(1);\n    for (dense_idx, orig_idx) in orig_idx_with_value.iter().enumerate().step_by(step_size) {\n        assert_eq!(null_index.rank_if_exists(*orig_idx), Some(dense_idx as u32));\n    }\n\n    // 100 samples\n    let step_size = (data.len() / 100).max(1);\n    for (pos, value) in data.iter().enumerate().step_by(step_size) {\n        assert_eq!(null_index.contains(pos as u32), *value);\n    }\n}\n\n#[test]\nfn test_optional_index_test_translation() {\n    let optional_index = OptionalIndex::for_test(4, &[0, 2]);\n    let mut select_cursor = optional_index.select_cursor();\n    assert_eq!(select_cursor.select(0), 0);\n    assert_eq!(select_cursor.select(1), 2);\n}\n\n#[test]\nfn test_optional_index_translate() {\n    let optional_index = OptionalIndex::for_test(4, &[0, 2]);\n    assert_eq!(optional_index.rank_if_exists(0), Some(0));\n    assert_eq!(optional_index.rank_if_exists(2), Some(1));\n}\n\n#[test]\nfn test_optional_index_small() {\n    let optional_index = OptionalIndex::for_test(4, &[0, 2]);\n    assert!(optional_index.contains(0));\n    assert!(!optional_index.contains(1));\n    assert!(optional_index.contains(2));\n    assert!(!optional_index.contains(3));\n}\n\n#[test]\nfn test_optional_index_large() {\n    let row_ids = &[ELEMENTS_PER_BLOCK, ELEMENTS_PER_BLOCK + 1];\n    let optional_index = OptionalIndex::for_test(ELEMENTS_PER_BLOCK + 2, row_ids);\n    assert!(!optional_index.contains(0));\n    assert!(!optional_index.contains(100));\n    assert!(!optional_index.contains(ELEMENTS_PER_BLOCK - 1));\n    assert!(optional_index.contains(ELEMENTS_PER_BLOCK));\n    assert!(optional_index.contains(ELEMENTS_PER_BLOCK + 1));\n}\n\nfn test_optional_index_iter_aux(row_ids: &[RowId], num_rows: RowId) {\n    let optional_index = OptionalIndex::for_test(num_rows, row_ids);\n    assert_eq!(optional_index.num_docs(), num_rows);\n    assert!(\n        optional_index\n            .iter_non_null_docs()\n            .eq(row_ids.iter().copied())\n    );\n}\n\n#[test]\nfn test_optional_index_iter_empty() {\n    test_optional_index_iter_aux(&[], 0u32);\n}\n\nfn test_optional_index_rank_aux(row_ids: &[RowId]) {\n    let num_rows = row_ids.last().copied().unwrap_or(0u32) + 1;\n    let null_index = OptionalIndex::for_test(num_rows, row_ids);\n    assert_eq!(null_index.num_docs(), num_rows);\n    for (row_id, row_val) in row_ids.iter().copied().enumerate() {\n        assert_eq!(null_index.rank(row_val), row_id as u32);\n        assert_eq!(null_index.rank_if_exists(row_val), Some(row_id as u32));\n        if row_val > 0 && !null_index.contains(&row_val - 1) {\n            assert_eq!(null_index.rank(row_val - 1), row_id as u32);\n        }\n        assert_eq!(null_index.rank(row_val + 1), row_id as u32 + 1);\n    }\n}\n\n#[test]\nfn test_optional_index_rank() {\n    test_optional_index_rank_aux(&[1u32]);\n    test_optional_index_rank_aux(&[0u32, 1u32]);\n    let mut block = Vec::new();\n    block.push(3u32);\n    block.extend((0..ELEMENTS_PER_BLOCK).map(|i| i + ELEMENTS_PER_BLOCK + 1));\n    test_optional_index_rank_aux(&block);\n}\n\n#[test]\nfn test_optional_index_iter_empty_one() {\n    test_optional_index_iter_aux(&[1], 2u32);\n    test_optional_index_iter_aux(&[100_000], 200_000u32);\n}\n\n#[test]\nfn test_optional_index_iter_dense_block() {\n    let mut block = Vec::new();\n    block.push(3u32);\n    block.extend((0..ELEMENTS_PER_BLOCK).map(|i| i + ELEMENTS_PER_BLOCK + 1));\n    test_optional_index_iter_aux(&block, 3 * ELEMENTS_PER_BLOCK);\n}\n\n#[test]\nfn test_optional_index_for_tests() {\n    let optional_index = OptionalIndex::for_test(4, &[1, 2]);\n    assert!(!optional_index.contains(0));\n    assert!(optional_index.contains(1));\n    assert!(optional_index.contains(2));\n    assert!(!optional_index.contains(3));\n    assert_eq!(optional_index.num_docs(), 4);\n}\n"
  },
  {
    "path": "columnar/src/column_index/serialize.rs",
    "content": "use std::io;\nuse std::io::Write;\n\nuse common::{CountingWriter, OwnedBytes};\n\nuse super::OptionalIndex;\nuse super::multivalued_index::SerializableMultivalueIndex;\nuse crate::column_index::ColumnIndex;\nuse crate::column_index::multivalued_index::serialize_multivalued_index;\nuse crate::column_index::optional_index::serialize_optional_index;\nuse crate::iterable::Iterable;\nuse crate::{Cardinality, RowId, Version};\n\npub struct SerializableOptionalIndex<'a> {\n    pub non_null_row_ids: Box<dyn Iterable<RowId> + 'a>,\n    pub num_rows: RowId,\n}\n\nimpl<'a> From<&'a OptionalIndex> for SerializableOptionalIndex<'a> {\n    fn from(optional_index: &'a OptionalIndex) -> Self {\n        SerializableOptionalIndex {\n            non_null_row_ids: Box::new(optional_index),\n            num_rows: optional_index.num_docs(),\n        }\n    }\n}\n\npub enum SerializableColumnIndex<'a> {\n    Full,\n    Optional(SerializableOptionalIndex<'a>),\n    Multivalued(SerializableMultivalueIndex<'a>),\n}\n\nimpl SerializableColumnIndex<'_> {\n    pub fn get_cardinality(&self) -> Cardinality {\n        match self {\n            SerializableColumnIndex::Full => Cardinality::Full,\n            SerializableColumnIndex::Optional(_) => Cardinality::Optional,\n            SerializableColumnIndex::Multivalued(_) => Cardinality::Multivalued,\n        }\n    }\n}\n\n/// Serialize a column index.\npub fn serialize_column_index(\n    column_index: SerializableColumnIndex,\n    output: &mut impl Write,\n) -> io::Result<u32> {\n    let mut output = CountingWriter::wrap(output);\n    let cardinality = column_index.get_cardinality().to_code();\n    output.write_all(&[cardinality])?;\n    match column_index {\n        SerializableColumnIndex::Full => {}\n        SerializableColumnIndex::Optional(SerializableOptionalIndex {\n            non_null_row_ids,\n            num_rows,\n        }) => serialize_optional_index(non_null_row_ids.as_ref(), num_rows, &mut output)?,\n        SerializableColumnIndex::Multivalued(multivalued_index) => {\n            serialize_multivalued_index(&multivalued_index, &mut output)?\n        }\n    }\n    let column_index_num_bytes = output.written_bytes() as u32;\n    Ok(column_index_num_bytes)\n}\n\n/// Open a serialized column index.\npub fn open_column_index(\n    mut bytes: OwnedBytes,\n    format_version: Version,\n) -> io::Result<ColumnIndex> {\n    if bytes.is_empty() {\n        return Err(io::Error::new(\n            io::ErrorKind::UnexpectedEof,\n            \"Failed to deserialize column index. Empty buffer.\",\n        ));\n    }\n    let cardinality_code = bytes[0];\n    let cardinality = Cardinality::try_from_code(cardinality_code)?;\n    bytes.advance(1);\n    match cardinality {\n        Cardinality::Full => Ok(ColumnIndex::Full),\n        Cardinality::Optional => {\n            let optional_index = super::optional_index::open_optional_index(bytes)?;\n            Ok(ColumnIndex::Optional(optional_index))\n        }\n        Cardinality::Multivalued => {\n            let multivalue_index =\n                super::multivalued_index::open_multivalued_index(bytes, format_version)?;\n            Ok(ColumnIndex::Multivalued(multivalue_index))\n        }\n    }\n}\n\n// TODO unit tests\n"
  },
  {
    "path": "columnar/src/column_values/merge.rs",
    "content": "use std::fmt::Debug;\nuse std::sync::Arc;\n\nuse crate::iterable::Iterable;\nuse crate::{ColumnIndex, ColumnValues, MergeRowOrder};\n\npub(crate) struct MergedColumnValues<'a, T> {\n    pub(crate) column_indexes: &'a [ColumnIndex],\n    pub(crate) column_values: &'a [Option<Arc<dyn ColumnValues<T>>>],\n    pub(crate) merge_row_order: &'a MergeRowOrder,\n}\n\nimpl<T: Copy + PartialOrd + Debug + 'static> Iterable<T> for MergedColumnValues<'_, T> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = T> + '_> {\n        match self.merge_row_order {\n            MergeRowOrder::Stack(_) => Box::new(\n                self.column_values\n                    .iter()\n                    .flatten()\n                    .flat_map(|column_value| column_value.iter()),\n            ),\n            MergeRowOrder::Shuffled(shuffle_merge_order) => Box::new(\n                shuffle_merge_order\n                    .iter_new_to_old_row_addrs()\n                    .flat_map(|row_addr| {\n                        let column_index = &self.column_indexes[row_addr.segment_ord as usize];\n                        let column_values =\n                            self.column_values[row_addr.segment_ord as usize].as_ref()?;\n                        let value_range = column_index.value_row_ids(row_addr.row_id);\n                        Some((value_range, column_values))\n                    })\n                    .flat_map(|(value_range, column_values)| {\n                        value_range\n                            .into_iter()\n                            .map(|val| column_values.get_val(val))\n                    }),\n            ),\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/mod.rs",
    "content": "#![warn(missing_docs)]\n\n//! # `fastfield_codecs`\n//!\n//! - Columnar storage of data for tantivy [`crate::Column`].\n//! - Encode data in different codecs.\n//! - Monotonically map values to u64/u128\n\nuse std::fmt::Debug;\nuse std::ops::{Range, RangeInclusive};\nuse std::sync::Arc;\n\nuse downcast_rs::DowncastSync;\npub use monotonic_mapping::{MonotonicallyMappableToU64, StrictlyMonotonicFn};\npub use monotonic_mapping_u128::MonotonicallyMappableToU128;\n\nmod merge;\npub(crate) mod monotonic_mapping;\npub(crate) mod monotonic_mapping_u128;\nmod stats;\nmod u128_based;\nmod u64_based;\nmod vec_column;\n\nmod monotonic_column;\n\npub(crate) use merge::MergedColumnValues;\npub use stats::ColumnStats;\npub use u64_based::{\n    ALL_U64_CODEC_TYPES, CodecType, load_u64_based_column_values,\n    serialize_and_load_u64_based_column_values, serialize_u64_based_column_values,\n};\npub use u128_based::{\n    CompactHit, CompactSpaceU64Accessor, open_u128_as_compact_u64, open_u128_mapped,\n    serialize_column_values_u128,\n};\npub use vec_column::VecColumn;\n\npub use self::monotonic_column::monotonic_map_column;\nuse crate::RowId;\n\n/// `ColumnValues` provides access to a dense field column.\n///\n/// `Column` are just a wrapper over `ColumnValues` and a `ColumnIndex`.\n///\n/// Any methods with a default and specialized implementation need to be called in the\n/// wrappers that implement the trait: Arc and MonotonicMappingColumn\npub trait ColumnValues<T: PartialOrd = u64>: Send + Sync + DowncastSync {\n    /// Return the value associated with the given idx.\n    ///\n    /// This accessor should return as fast as possible.\n    ///\n    /// # Panics\n    ///\n    /// May panic if `idx` is greater than the column length.\n    fn get_val(&self, idx: u32) -> T;\n\n    /// Allows to push down multiple fetch calls, to avoid dynamic dispatch overhead.\n    ///\n    /// idx and output should have the same length\n    ///\n    /// # Panics\n    ///\n    /// May panic if `idx` is greater than the column length.\n    fn get_vals(&self, indexes: &[u32], output: &mut [T]) {\n        assert!(indexes.len() == output.len());\n        let out_and_idx_chunks = output.chunks_exact_mut(4).zip(indexes.chunks_exact(4));\n        for (out_x4, idx_x4) in out_and_idx_chunks {\n            out_x4[0] = self.get_val(idx_x4[0]);\n            out_x4[1] = self.get_val(idx_x4[1]);\n            out_x4[2] = self.get_val(idx_x4[2]);\n            out_x4[3] = self.get_val(idx_x4[3]);\n        }\n\n        let out_and_idx_chunks = output\n            .chunks_exact_mut(4)\n            .into_remainder()\n            .iter_mut()\n            .zip(indexes.chunks_exact(4).remainder());\n        for (out, idx) in out_and_idx_chunks {\n            *out = self.get_val(*idx);\n        }\n    }\n\n    /// Allows to push down multiple fetch calls, to avoid dynamic dispatch overhead.\n    /// The slightly weird `Option<T>` in output allows pushdown to full columns.\n    ///\n    /// idx and output should have the same length\n    ///\n    /// # Panics\n    ///\n    /// May panic if `idx` is greater than the column length.\n    fn get_vals_opt(&self, indexes: &[u32], output: &mut [Option<T>]) {\n        assert!(indexes.len() == output.len());\n        let out_and_idx_chunks = output.chunks_exact_mut(4).zip(indexes.chunks_exact(4));\n        for (out_x4, idx_x4) in out_and_idx_chunks {\n            out_x4[0] = Some(self.get_val(idx_x4[0]));\n            out_x4[1] = Some(self.get_val(idx_x4[1]));\n            out_x4[2] = Some(self.get_val(idx_x4[2]));\n            out_x4[3] = Some(self.get_val(idx_x4[3]));\n        }\n        let out_and_idx_chunks = output\n            .chunks_exact_mut(4)\n            .into_remainder()\n            .iter_mut()\n            .zip(indexes.chunks_exact(4).remainder());\n        for (out, idx) in out_and_idx_chunks {\n            *out = Some(self.get_val(*idx));\n        }\n    }\n\n    /// Fills an output buffer with the fast field values\n    /// associated with the `DocId` going from\n    /// `start` to `start + output.len()`.\n    ///\n    /// # Panics\n    ///\n    /// Must panic if `start + output.len()` is greater than\n    /// the segment's `maxdoc`.\n    #[inline(always)]\n    fn get_range(&self, start: u64, output: &mut [T]) {\n        for (out, idx) in output.iter_mut().zip(start..) {\n            *out = self.get_val(idx as u32);\n        }\n    }\n\n    /// Get the row ids of values which are in the provided value range.\n    ///\n    /// Note that position == docid for single value fast fields\n    fn get_row_ids_for_value_range(\n        &self,\n        value_range: RangeInclusive<T>,\n        row_id_range: Range<RowId>,\n        row_id_hits: &mut Vec<RowId>,\n    ) {\n        let row_id_range = row_id_range.start..row_id_range.end.min(self.num_vals());\n        for idx in row_id_range {\n            let val = self.get_val(idx);\n            if value_range.contains(&val) {\n                row_id_hits.push(idx);\n            }\n        }\n    }\n\n    /// Returns a lower bound for this column of values.\n    ///\n    /// All values are guaranteed to be higher than `.min_value()`\n    /// but this value is not necessary the best boundary value.\n    ///\n    /// We have\n    /// ∀i < self.num_vals(), self.get_val(i) >= self.min_value()\n    /// But we don't have necessarily\n    /// ∃i < self.num_vals(), self.get_val(i) == self.min_value()\n    fn min_value(&self) -> T;\n\n    /// Returns an upper bound for this column of values.\n    ///\n    /// All values are guaranteed to be lower than `.max_value()`\n    /// but this value is not necessary the best boundary value.\n    ///\n    /// We have\n    /// ∀i < self.num_vals(), self.get_val(i) <= self.max_value()\n    /// But we don't have necessarily\n    /// ∃i < self.num_vals(), self.get_val(i) == self.max_value()\n    fn max_value(&self) -> T;\n\n    /// The number of values in the column.\n    fn num_vals(&self) -> u32;\n\n    /// Returns a iterator over the data\n    fn iter<'a>(&'a self) -> Box<dyn Iterator<Item = T> + 'a> {\n        Box::new((0..self.num_vals()).map(|idx| self.get_val(idx)))\n    }\n}\ndowncast_rs::impl_downcast!(sync ColumnValues<T> where T: PartialOrd);\n\n/// Empty column of values.\npub struct EmptyColumnValues;\n\nimpl<T: PartialOrd + Default> ColumnValues<T> for EmptyColumnValues {\n    fn get_val(&self, _idx: u32) -> T {\n        panic!(\"Internal Error: Called get_val of empty column.\")\n    }\n\n    fn min_value(&self) -> T {\n        T::default()\n    }\n\n    fn max_value(&self) -> T {\n        T::default()\n    }\n\n    fn num_vals(&self) -> u32 {\n        0\n    }\n}\n\nimpl<T: Copy + PartialOrd + Debug + 'static> ColumnValues<T> for Arc<dyn ColumnValues<T>> {\n    #[inline(always)]\n    fn get_val(&self, idx: u32) -> T {\n        self.as_ref().get_val(idx)\n    }\n\n    #[inline(always)]\n    fn get_vals_opt(&self, indexes: &[u32], output: &mut [Option<T>]) {\n        self.as_ref().get_vals_opt(indexes, output)\n    }\n\n    #[inline(always)]\n    fn min_value(&self) -> T {\n        self.as_ref().min_value()\n    }\n\n    #[inline(always)]\n    fn max_value(&self) -> T {\n        self.as_ref().max_value()\n    }\n\n    #[inline(always)]\n    fn num_vals(&self) -> u32 {\n        self.as_ref().num_vals()\n    }\n\n    #[inline(always)]\n    fn iter<'b>(&'b self) -> Box<dyn Iterator<Item = T> + 'b> {\n        self.as_ref().iter()\n    }\n\n    #[inline(always)]\n    fn get_range(&self, start: u64, output: &mut [T]) {\n        self.as_ref().get_range(start, output)\n    }\n\n    #[inline(always)]\n    fn get_row_ids_for_value_range(\n        &self,\n        range: RangeInclusive<T>,\n        doc_id_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        self.as_ref()\n            .get_row_ids_for_value_range(range, doc_id_range, positions)\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/monotonic_column.rs",
    "content": "use std::fmt::Debug;\nuse std::marker::PhantomData;\nuse std::ops::{Range, RangeInclusive};\n\nuse crate::ColumnValues;\nuse crate::column_values::monotonic_mapping::StrictlyMonotonicFn;\n\nstruct MonotonicMappingColumn<C, T, Input> {\n    from_column: C,\n    monotonic_mapping: T,\n    _phantom: PhantomData<Input>,\n}\n\n/// Creates a view of a column transformed by a strictly monotonic mapping. See\n/// [`StrictlyMonotonicFn`].\n///\n/// E.g. apply a gcd monotonic_mapping([100, 200, 300]) == [1, 2, 3]\n/// monotonic_mapping.mapping() is expected to be injective, and we should always have\n/// monotonic_mapping.inverse(monotonic_mapping.mapping(el)) == el\n///\n/// The inverse of the mapping is required for:\n/// `fn get_positions_for_value_range(&self, range: RangeInclusive<T>) -> Vec<u64> `\n/// The user provides the original value range and we need to monotonic map them in the same way the\n/// serialization does before calling the underlying column.\n///\n/// Note that when opening a codec, the monotonic_mapping should be the inverse of the mapping\n/// during serialization. And therefore the monotonic_mapping_inv when opening is the same as\n/// monotonic_mapping during serialization.\npub fn monotonic_map_column<C, T, Input, Output>(\n    from_column: C,\n    monotonic_mapping: T,\n) -> impl ColumnValues<Output>\nwhere\n    C: ColumnValues<Input> + 'static,\n    T: StrictlyMonotonicFn<Input, Output> + Send + Sync + 'static,\n    Input: PartialOrd + Debug + Send + Sync + Clone + 'static,\n    Output: PartialOrd + Debug + Send + Sync + Clone + 'static,\n{\n    MonotonicMappingColumn {\n        from_column,\n        monotonic_mapping,\n        _phantom: PhantomData,\n    }\n}\n\nimpl<C, T, Input, Output> ColumnValues<Output> for MonotonicMappingColumn<C, T, Input>\nwhere\n    C: ColumnValues<Input> + 'static,\n    T: StrictlyMonotonicFn<Input, Output> + Send + Sync + 'static,\n    Input: PartialOrd + Send + Debug + Sync + Clone + 'static,\n    Output: PartialOrd + Send + Debug + Sync + Clone + 'static,\n{\n    #[inline(always)]\n    fn get_val(&self, idx: u32) -> Output {\n        let from_val = self.from_column.get_val(idx);\n        self.monotonic_mapping.mapping(from_val)\n    }\n\n    fn min_value(&self) -> Output {\n        let from_min_value = self.from_column.min_value();\n        self.monotonic_mapping.mapping(from_min_value)\n    }\n\n    fn max_value(&self) -> Output {\n        let from_max_value = self.from_column.max_value();\n        self.monotonic_mapping.mapping(from_max_value)\n    }\n\n    fn num_vals(&self) -> u32 {\n        self.from_column.num_vals()\n    }\n\n    fn iter(&self) -> Box<dyn Iterator<Item = Output> + '_> {\n        Box::new(\n            self.from_column\n                .iter()\n                .map(|el| self.monotonic_mapping.mapping(el)),\n        )\n    }\n\n    fn get_row_ids_for_value_range(\n        &self,\n        range: RangeInclusive<Output>,\n        doc_id_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        self.from_column.get_row_ids_for_value_range(\n            self.monotonic_mapping.inverse(range.start().clone())\n                ..=self.monotonic_mapping.inverse(range.end().clone()),\n            doc_id_range,\n            positions,\n        )\n    }\n\n    // We voluntarily do not implement get_range as it yields a regression,\n    // and we do not have any specialized implementation anyway.\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::column_values::VecColumn;\n    use crate::column_values::monotonic_mapping::{\n        StrictlyMonotonicMappingInverter, StrictlyMonotonicMappingToInternal,\n    };\n\n    #[test]\n    fn test_monotonic_mapping_iter() {\n        let vals: Vec<u64> = (0..100u64).map(|el| el * 10).collect();\n        let col = VecColumn::from(vals);\n        let mapped = monotonic_map_column(\n            col,\n            StrictlyMonotonicMappingInverter::from(StrictlyMonotonicMappingToInternal::<i64>::new()),\n        );\n        let val_i64s: Vec<u64> = mapped.iter().collect();\n        for i in 0..100 {\n            assert_eq!(val_i64s[i as usize], mapped.get_val(i));\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/monotonic_mapping.rs",
    "content": "use std::fmt::Debug;\nuse std::marker::PhantomData;\n\nuse common::DateTime;\n\nuse super::MonotonicallyMappableToU128;\nuse crate::RowId;\n\n/// Monotonic maps a value to u64 value space.\n/// Monotonic mapping enables `PartialOrd` on u64 space without conversion to original space.\npub trait MonotonicallyMappableToU64: 'static + PartialOrd + Debug + Copy + Send + Sync {\n    /// Converts a value to u64.\n    ///\n    /// Internally all fast field values are encoded as u64.\n    fn to_u64(self) -> u64;\n\n    /// Converts a value from u64\n    ///\n    /// Internally all fast field values are encoded as u64.\n    /// **Note: To be used for converting encoded Term, Posting values.**\n    fn from_u64(val: u64) -> Self;\n}\n\n/// Values need to be strictly monotonic mapped to a `Internal` value (u64 or u128) that can be\n/// used in fast field codecs.\n///\n/// The monotonic mapping is required so that `PartialOrd` can be used on `Internal` without\n/// converting to `External`.\n///\n/// All strictly monotonic functions are invertible because they are guaranteed to have a one-to-one\n/// mapping from their range to their domain. The `inverse` method is required when opening a codec,\n/// so a value can be converted back to its original domain (e.g. ip address or f64) from its\n/// internal representation.\npub trait StrictlyMonotonicFn<External, Internal> {\n    /// Strictly monotonically maps the value from External to Internal.\n    fn mapping(&self, inp: External) -> Internal;\n    /// Inverse of `mapping`. Maps the value from Internal to External.\n    fn inverse(&self, out: Internal) -> External;\n}\n\n/// Inverts a strictly monotonic mapping from `StrictlyMonotonicFn<A, B>` to\n/// `StrictlyMonotonicFn<B, A>`.\n///\n/// # Warning\n///\n/// This type comes with a footgun. A type being strictly monotonic does not impose that the inverse\n/// mapping is strictly monotonic over the entire space External. e.g. a -> a * 2. Use at your own\n/// risks.\npub(crate) struct StrictlyMonotonicMappingInverter<T> {\n    orig_mapping: T,\n}\nimpl<T> From<T> for StrictlyMonotonicMappingInverter<T> {\n    fn from(orig_mapping: T) -> Self {\n        Self { orig_mapping }\n    }\n}\n\nimpl<From, To, T> StrictlyMonotonicFn<To, From> for StrictlyMonotonicMappingInverter<T>\nwhere T: StrictlyMonotonicFn<From, To>\n{\n    #[inline(always)]\n    fn mapping(&self, val: To) -> From {\n        self.orig_mapping.inverse(val)\n    }\n\n    #[inline(always)]\n    fn inverse(&self, val: From) -> To {\n        self.orig_mapping.mapping(val)\n    }\n}\n\n/// Applies the strictly monotonic mapping from `T` without any additional changes.\npub(crate) struct StrictlyMonotonicMappingToInternal<T> {\n    _phantom: PhantomData<T>,\n}\n\nimpl<T> StrictlyMonotonicMappingToInternal<T> {\n    pub(crate) fn new() -> StrictlyMonotonicMappingToInternal<T> {\n        Self {\n            _phantom: PhantomData,\n        }\n    }\n}\n\nimpl<External: MonotonicallyMappableToU128, T: MonotonicallyMappableToU128>\n    StrictlyMonotonicFn<External, u128> for StrictlyMonotonicMappingToInternal<T>\nwhere T: MonotonicallyMappableToU128\n{\n    #[inline(always)]\n    fn mapping(&self, inp: External) -> u128 {\n        External::to_u128(inp)\n    }\n\n    #[inline(always)]\n    fn inverse(&self, out: u128) -> External {\n        External::from_u128(out)\n    }\n}\n\nimpl<External: MonotonicallyMappableToU64, T: MonotonicallyMappableToU64>\n    StrictlyMonotonicFn<External, u64> for StrictlyMonotonicMappingToInternal<T>\nwhere T: MonotonicallyMappableToU64\n{\n    #[inline(always)]\n    fn mapping(&self, inp: External) -> u64 {\n        External::to_u64(inp)\n    }\n\n    #[inline(always)]\n    fn inverse(&self, out: u64) -> External {\n        External::from_u64(out)\n    }\n}\n\nimpl MonotonicallyMappableToU64 for u64 {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        self\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> Self {\n        val\n    }\n}\n\nimpl MonotonicallyMappableToU64 for i64 {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        common::i64_to_u64(self)\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> Self {\n        common::u64_to_i64(val)\n    }\n}\n\nimpl MonotonicallyMappableToU64 for DateTime {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        common::i64_to_u64(self.into_timestamp_nanos())\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> Self {\n        DateTime::from_timestamp_nanos(common::u64_to_i64(val))\n    }\n}\n\nimpl MonotonicallyMappableToU64 for bool {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        u64::from(self)\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> Self {\n        val > 0\n    }\n}\n\nimpl MonotonicallyMappableToU64 for RowId {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        u64::from(self)\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> RowId {\n        val as RowId\n    }\n}\n\n// TODO remove me.\n// Tantivy should refuse NaN values and work with NotNaN internally.\nimpl MonotonicallyMappableToU64 for f64 {\n    #[inline(always)]\n    fn to_u64(self) -> u64 {\n        common::f64_to_u64(self)\n    }\n\n    #[inline(always)]\n    fn from_u64(val: u64) -> Self {\n        common::u64_to_f64(val)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n\n    #[test]\n    fn strictly_monotonic_test() {\n        // identity mapping\n        test_round_trip(&StrictlyMonotonicMappingToInternal::<u64>::new(), 100u64);\n        // round trip to i64\n        test_round_trip(&StrictlyMonotonicMappingToInternal::<i64>::new(), 100u64);\n        // TODO\n        // identity mapping\n        // test_round_trip(&StrictlyMonotonicMappingToInternal::<u128>::new(), 100u128);\n    }\n\n    fn test_round_trip<T: StrictlyMonotonicFn<K, L>, K: std::fmt::Debug + Eq + Copy, L>(\n        mapping: &T,\n        test_val: K,\n    ) {\n        assert_eq!(mapping.inverse(mapping.mapping(test_val)), test_val);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/monotonic_mapping_u128.rs",
    "content": "use std::fmt::Debug;\nuse std::net::Ipv6Addr;\n\n/// Monotonic maps a value to u128 value space\n/// Monotonic mapping enables `PartialOrd` on u128 space without conversion to original space.\npub trait MonotonicallyMappableToU128: 'static + PartialOrd + Copy + Debug + Send + Sync {\n    /// Converts a value to u128.\n    ///\n    /// Internally all fast field values are encoded as u64.\n    fn to_u128(self) -> u128;\n\n    /// Converts a value from u128\n    ///\n    /// Internally all fast field values are encoded as u64.\n    /// **Note: To be used for converting encoded Term, Posting values.**\n    fn from_u128(val: u128) -> Self;\n}\n\nimpl MonotonicallyMappableToU128 for u128 {\n    fn to_u128(self) -> u128 {\n        self\n    }\n\n    fn from_u128(val: u128) -> Self {\n        val\n    }\n}\n\nimpl MonotonicallyMappableToU128 for Ipv6Addr {\n    fn to_u128(self) -> u128 {\n        ip_to_u128(self)\n    }\n\n    fn from_u128(val: u128) -> Self {\n        Ipv6Addr::from(val.to_be_bytes())\n    }\n}\n\nfn ip_to_u128(ip_addr: Ipv6Addr) -> u128 {\n    u128::from_be_bytes(ip_addr.octets())\n}\n"
  },
  {
    "path": "columnar/src/column_values/stats.rs",
    "content": "use std::io;\nuse std::io::Write;\nuse std::num::NonZeroU64;\n\nuse common::{BinarySerializable, VInt};\n\nuse crate::RowId;\n\n/// Column statistics.\n#[derive(Debug, Clone, Eq, PartialEq)]\npub struct ColumnStats {\n    /// GCD of the elements `el - min(column)`.\n    pub gcd: NonZeroU64,\n    /// Minimum value of the column.\n    pub min_value: u64,\n    /// Maximum value of the column.\n    pub max_value: u64,\n    /// Number of rows in the column.\n    pub num_rows: RowId,\n}\n\nimpl ColumnStats {\n    /// Amplitude of value.\n    /// Difference between the maximum and the minimum value.\n    pub fn amplitude(&self) -> u64 {\n        self.max_value - self.min_value\n    }\n}\n\nimpl BinarySerializable for ColumnStats {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        VInt(self.min_value).serialize(writer)?;\n        VInt(self.gcd.get()).serialize(writer)?;\n        VInt(self.amplitude() / self.gcd).serialize(writer)?;\n        VInt(self.num_rows as u64).serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let min_value = VInt::deserialize(reader)?.0;\n        let gcd = VInt::deserialize(reader)?.0;\n        let gcd = NonZeroU64::new(gcd)\n            .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, \"GCD of 0 is forbidden\"))?;\n        let amplitude = VInt::deserialize(reader)?.0 * gcd.get();\n        let max_value = min_value + amplitude;\n        let num_rows = VInt::deserialize(reader)?.0 as RowId;\n        Ok(ColumnStats {\n            min_value,\n            max_value,\n            num_rows,\n            gcd,\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::num::NonZeroU64;\n\n    use common::BinarySerializable;\n\n    use crate::column_values::ColumnStats;\n\n    #[track_caller]\n    fn test_stats_ser_deser_aux(stats: &ColumnStats, num_bytes: usize) {\n        let mut buffer: Vec<u8> = Vec::new();\n        stats.serialize(&mut buffer).unwrap();\n        assert_eq!(buffer.len(), num_bytes);\n        let deser_stats = ColumnStats::deserialize(&mut &buffer[..]).unwrap();\n        assert_eq!(stats, &deser_stats);\n    }\n\n    #[test]\n    fn test_stats_serialization() {\n        test_stats_ser_deser_aux(\n            &(ColumnStats {\n                gcd: NonZeroU64::new(3).unwrap(),\n                min_value: 1,\n                max_value: 3001,\n                num_rows: 10,\n            }),\n            5,\n        );\n        test_stats_ser_deser_aux(\n            &(ColumnStats {\n                gcd: NonZeroU64::new(1_000).unwrap(),\n                min_value: 1,\n                max_value: 3001,\n                num_rows: 10,\n            }),\n            5,\n        );\n        test_stats_ser_deser_aux(\n            &(ColumnStats {\n                gcd: NonZeroU64::new(1).unwrap(),\n                min_value: 0,\n                max_value: 0,\n                num_rows: 0,\n            }),\n            4,\n        );\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u128_based/compact_space/blank_range.rs",
    "content": "use std::ops::RangeInclusive;\n\n/// The range of a blank in value space.\n///\n/// A blank is an unoccupied space in the data.\n/// Use try_into() to construct.\n/// A range has to have at least length of 3. Invalid ranges will be rejected.\n///\n/// Ordered by range length.\n#[derive(Debug, Eq, PartialEq, Clone)]\npub(crate) struct BlankRange {\n    blank_range: RangeInclusive<u128>,\n}\nimpl TryFrom<RangeInclusive<u128>> for BlankRange {\n    type Error = &'static str;\n    fn try_from(range: RangeInclusive<u128>) -> Result<Self, Self::Error> {\n        let blank_size = range.end().saturating_sub(*range.start());\n        if blank_size < 2 {\n            Err(\"invalid range\")\n        } else {\n            Ok(BlankRange { blank_range: range })\n        }\n    }\n}\nimpl BlankRange {\n    pub(crate) fn blank_size(&self) -> u128 {\n        self.blank_range.end() - self.blank_range.start() + 1\n    }\n    pub(crate) fn blank_range(&self) -> RangeInclusive<u128> {\n        self.blank_range.clone()\n    }\n}\n\nimpl Ord for BlankRange {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        self.blank_size().cmp(&other.blank_size())\n    }\n}\nimpl PartialOrd for BlankRange {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u128_based/compact_space/build_compact_space.rs",
    "content": "use std::collections::{BTreeSet, BinaryHeap};\nuse std::iter;\nuse std::ops::RangeInclusive;\n\nuse itertools::Itertools;\n\nuse super::blank_range::BlankRange;\nuse super::{CompactSpace, RangeMapping};\n\n/// Put the blanks for the sorted values into a binary heap\nfn get_blanks(values_sorted: &BTreeSet<u128>) -> BinaryHeap<BlankRange> {\n    let mut blanks: BinaryHeap<BlankRange> = BinaryHeap::new();\n    for (first, second) in values_sorted.iter().copied().tuple_windows() {\n        // Correctness Overflow: the values are deduped and sorted (BTreeSet property), that means\n        // there's always space between two values.\n        let blank_range = first + 1..=second - 1;\n        let blank_range: Result<BlankRange, _> = blank_range.try_into();\n        if let Ok(blank_range) = blank_range {\n            blanks.push(blank_range);\n        }\n    }\n\n    blanks\n}\n\nstruct BlankCollector {\n    blanks: Vec<BlankRange>,\n    staged_blanks_sum: u128,\n}\nimpl BlankCollector {\n    fn new() -> Self {\n        Self {\n            blanks: vec![],\n            staged_blanks_sum: 0,\n        }\n    }\n    fn stage_blank(&mut self, blank: BlankRange) {\n        self.staged_blanks_sum += blank.blank_size();\n        self.blanks.push(blank);\n    }\n    fn drain(&mut self) -> impl Iterator<Item = BlankRange> + '_ {\n        self.staged_blanks_sum = 0;\n        self.blanks.drain(..)\n    }\n    fn staged_blanks_sum(&self) -> u128 {\n        self.staged_blanks_sum\n    }\n    fn num_staged_blanks(&self) -> usize {\n        self.blanks.len()\n    }\n}\nfn num_bits(val: u128) -> u8 {\n    (128u32 - val.leading_zeros()) as u8\n}\n\n/// Will collect blanks and add them to compact space if more bits are saved than cost from\n/// metadata.\npub fn get_compact_space(\n    values_deduped_sorted: &BTreeSet<u128>,\n    total_num_values: u32,\n    cost_per_blank: usize,\n) -> CompactSpace {\n    let mut compact_space_builder = CompactSpaceBuilder::new();\n    if values_deduped_sorted.is_empty() {\n        return compact_space_builder.finish();\n    }\n\n    // We start by space that's limited to min_value..=max_value\n    // Replace after stabilization of https://github.com/rust-lang/rust/issues/62924\n    let min_value = values_deduped_sorted.iter().next().copied().unwrap_or(0);\n    let max_value = values_deduped_sorted.iter().last().copied().unwrap_or(0);\n\n    let mut blanks: BinaryHeap<BlankRange> = get_blanks(values_deduped_sorted);\n\n    // +1 for null, in case min and max covers the whole space, we are off by one.\n    let mut amplitude_compact_space = (max_value - min_value).saturating_add(1);\n    if min_value != 0 {\n        compact_space_builder.add_blanks(iter::once(0..=min_value - 1));\n    }\n    if max_value != u128::MAX {\n        compact_space_builder.add_blanks(iter::once(max_value + 1..=u128::MAX));\n    }\n\n    let mut amplitude_bits: u8 = num_bits(amplitude_compact_space);\n\n    let mut blank_collector = BlankCollector::new();\n\n    // We will stage blanks until they reduce the compact space by at least 1 bit and then flush\n    // them if the metadata cost is lower than the total number of saved bits.\n    // Binary heap to process the gaps by their size\n    while let Some(blank_range) = blanks.pop() {\n        blank_collector.stage_blank(blank_range);\n\n        let staged_spaces_sum: u128 = blank_collector.staged_blanks_sum();\n        let amplitude_new_compact_space = amplitude_compact_space - staged_spaces_sum;\n        let amplitude_new_bits = num_bits(amplitude_new_compact_space);\n\n        if amplitude_bits == amplitude_new_bits {\n            continue;\n        }\n        let saved_bits = (amplitude_bits - amplitude_new_bits) as usize * total_num_values as usize;\n        // TODO: Maybe calculate exact cost of blanks and run this more expensive computation only,\n        // when amplitude_new_bits changes\n        let cost = blank_collector.num_staged_blanks() * cost_per_blank;\n\n        // We want to end up with a compact space that fits into 32 bits.\n        // In order to deal with pathological cases, we force the algorithm to keep\n        // refining the compact space the amplitude bits is lower than 32.\n        //\n        // The worst case scenario happens for a large number of u128s regularly\n        // spread over the full u128 space.\n        //\n        // This change will force the algorithm to degenerate into dictionary encoding.\n        if amplitude_bits <= 32 && cost >= saved_bits {\n            // Continue here, since although we walk over the blanks by size,\n            // we can potentially save a lot at the last bits, which are smaller blanks\n            //\n            // E.g. if the first range reduces the compact space by 1000 from 2000 to 1000, which\n            // saves 11-10=1 bit and the next range reduces the compact space by 950 to\n            // 50, which saves 10-6=4 bit\n            continue;\n        }\n\n        amplitude_compact_space = amplitude_new_compact_space;\n        amplitude_bits = amplitude_new_bits;\n        compact_space_builder.add_blanks(blank_collector.drain().map(|blank| blank.blank_range()));\n    }\n\n    assert!(amplitude_bits <= 32);\n\n    // special case, when we don't collected any blanks because:\n    // * the data is empty (early exit)\n    // * the algorithm did decide it's not worth the cost, which can be the case for single values\n    //\n    // We drain one collected blank unconditionally, so the empty case is reserved for empty\n    // data, and therefore empty compact_space means the data is empty and no data is covered\n    // (conversely to all data) and we can assign null to it.\n    if compact_space_builder.is_empty() {\n        compact_space_builder.add_blanks(\n            blank_collector\n                .drain()\n                .map(|blank| blank.blank_range())\n                .take(1),\n        );\n    }\n\n    let compact_space = compact_space_builder.finish();\n    if max_value - min_value != u128::MAX {\n        debug_assert_eq!(\n            compact_space.amplitude_compact_space(),\n            amplitude_compact_space\n        );\n    }\n    compact_space\n}\n\n#[derive(Debug, Clone, Eq, PartialEq)]\nstruct CompactSpaceBuilder {\n    blanks: Vec<RangeInclusive<u128>>,\n}\n\nimpl CompactSpaceBuilder {\n    /// Creates a new compact space builder which will initially cover the whole space.\n    fn new() -> Self {\n        Self { blanks: Vec::new() }\n    }\n\n    /// Assumes that repeated add_blank calls don't overlap and are not adjacent,\n    /// e.g. [3..=5, 5..=10] is not allowed\n    ///\n    /// Both of those assumptions are true when blanks are produced from sorted values.\n    fn add_blanks(&mut self, blank: impl Iterator<Item = RangeInclusive<u128>>) {\n        self.blanks.extend(blank);\n    }\n\n    fn is_empty(&self) -> bool {\n        self.blanks.is_empty()\n    }\n\n    /// Convert blanks to covered space and assign null value\n    fn finish(mut self) -> CompactSpace {\n        // sort by start. ranges are not allowed to overlap\n        self.blanks.sort_unstable_by_key(|blank| *blank.start());\n\n        let mut covered_space = Vec::with_capacity(self.blanks.len());\n\n        // beginning of the blanks\n        if let Some(first_blank_start) = self.blanks.first().map(RangeInclusive::start)\n            && *first_blank_start != 0\n        {\n            covered_space.push(0..=first_blank_start - 1);\n        }\n\n        // Between the blanks\n        let between_blanks = self.blanks.iter().tuple_windows().map(|(left, right)| {\n            assert!(\n                left.end() < right.start(),\n                \"overlapping or adjacent ranges detected\"\n            );\n            *left.end() + 1..=*right.start() - 1\n        });\n        covered_space.extend(between_blanks);\n\n        // end of the blanks\n        if let Some(last_blank_end) = self.blanks.last().map(RangeInclusive::end)\n            && *last_blank_end != u128::MAX\n        {\n            covered_space.push(last_blank_end + 1..=u128::MAX);\n        }\n\n        if covered_space.is_empty() {\n            covered_space.push(0..=0); // empty data case\n        };\n\n        let mut compact_start: u32 = 1; // 0 is reserved for `null`\n        let mut ranges_mapping: Vec<RangeMapping> = Vec::with_capacity(covered_space.len());\n        for cov in covered_space {\n            let range_mapping = super::RangeMapping {\n                value_range: cov,\n                compact_start,\n            };\n            let covered_range_len = range_mapping.range_length();\n            ranges_mapping.push(range_mapping);\n            compact_start += covered_range_len;\n        }\n        // println!(\"num ranges {}\", ranges_mapping.len());\n        CompactSpace { ranges_mapping }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::column_values::u128_based::compact_space::COST_PER_BLANK_IN_BITS;\n\n    #[test]\n    fn test_binary_heap_pop_order() {\n        let mut blanks: BinaryHeap<BlankRange> = BinaryHeap::new();\n        blanks.push((0..=10).try_into().unwrap());\n        blanks.push((100..=200).try_into().unwrap());\n        blanks.push((100..=110).try_into().unwrap());\n        assert_eq!(blanks.pop().unwrap().blank_size(), 101);\n        assert_eq!(blanks.pop().unwrap().blank_size(), 11);\n    }\n\n    #[test]\n    fn test_worst_case_scenario() {\n        let vals: BTreeSet<u128> = (0..8).map(|i| i * ((1u128 << 34) / 8)).collect();\n        let compact_space = get_compact_space(&vals, vals.len() as u32, COST_PER_BLANK_IN_BITS);\n        assert!(compact_space.amplitude_compact_space() < u32::MAX as u128);\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u128_based/compact_space/mod.rs",
    "content": "/// This codec takes a large number space (u128) and reduces it to a compact number space.\n///\n/// It will find spaces in the number range. For example:\n///\n/// 100, 101, 102, 103, 104, 50000, 50001\n/// could be mapped to\n/// 100..104 -> 0..4\n/// 50000..50001 -> 5..6\n///\n/// Compact space 0..=6 requires much less bits than 100..=50001\n///\n/// The codec is created to compress ip addresses, but may be employed in other use cases.\nuse std::{\n    cmp::Ordering,\n    collections::BTreeSet,\n    io::{self, Write},\n    ops::{Range, RangeInclusive},\n};\n\nmod blank_range;\nmod build_compact_space;\n\nuse build_compact_space::get_compact_space;\nuse common::{BinarySerializable, CountingWriter, OwnedBytes, VInt, VIntU128};\nuse tantivy_bitpacker::{BitPacker, BitUnpacker};\n\nuse crate::RowId;\nuse crate::column_values::ColumnValues;\n\n/// The cost per blank is quite hard actually, since blanks are delta encoded, the actual cost of\n/// blanks depends on the number of blanks.\n///\n/// The number is taken by looking at a real dataset. It is optimized for larger datasets.\nconst COST_PER_BLANK_IN_BITS: usize = 36;\n\n#[derive(Debug, Clone, Eq, PartialEq)]\npub struct CompactSpace {\n    ranges_mapping: Vec<RangeMapping>,\n}\n\n/// Maps the range from the original space to compact_start + range.len()\n#[derive(Debug, Clone, Eq, PartialEq)]\nstruct RangeMapping {\n    value_range: RangeInclusive<u128>,\n    compact_start: u32,\n}\nimpl RangeMapping {\n    fn range_length(&self) -> u32 {\n        (self.value_range.end() - self.value_range.start()) as u32 + 1\n    }\n\n    // The last value of the compact space in this range\n    fn compact_end(&self) -> u32 {\n        self.compact_start + self.range_length() - 1\n    }\n}\n\nimpl BinarySerializable for CompactSpace {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        VInt(self.ranges_mapping.len() as u64).serialize(writer)?;\n\n        let mut prev_value = 0;\n        for value_range in self\n            .ranges_mapping\n            .iter()\n            .map(|range_mapping| &range_mapping.value_range)\n        {\n            let blank_delta_start = value_range.start() - prev_value;\n            VIntU128(blank_delta_start).serialize(writer)?;\n            prev_value = *value_range.start();\n\n            let blank_delta_end = value_range.end() - prev_value;\n            VIntU128(blank_delta_end).serialize(writer)?;\n            prev_value = *value_range.end();\n        }\n\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let num_ranges = VInt::deserialize(reader)?.0;\n        let mut ranges_mapping: Vec<RangeMapping> = vec![];\n        let mut value = 0u128;\n        let mut compact_start = 1u32; // 0 is reserved for `null`\n        for _ in 0..num_ranges {\n            let blank_delta_start = VIntU128::deserialize(reader)?.0;\n            value += blank_delta_start;\n            let blank_start = value;\n\n            let blank_delta_end = VIntU128::deserialize(reader)?.0;\n            value += blank_delta_end;\n            let blank_end = value;\n\n            let range_mapping = RangeMapping {\n                value_range: blank_start..=blank_end,\n                compact_start,\n            };\n            let range_length = range_mapping.range_length();\n            ranges_mapping.push(range_mapping);\n            compact_start += range_length;\n        }\n\n        Ok(Self { ranges_mapping })\n    }\n}\n\nimpl CompactSpace {\n    /// Amplitude is the value range of the compact space including the sentinel value used to\n    /// identify null values. The compact space is 0..=amplitude .\n    ///\n    /// It's only used to verify we don't exceed u64 number space, which would indicate a bug.\n    fn amplitude_compact_space(&self) -> u128 {\n        self.ranges_mapping\n            .last()\n            .map(|last_range| last_range.compact_end() as u128)\n            .unwrap_or(1) // compact space starts at 1, 0 == null\n    }\n\n    fn get_range_mapping(&self, pos: usize) -> &RangeMapping {\n        &self.ranges_mapping[pos]\n    }\n\n    /// Returns either Ok(the value in the compact space) or if it is outside the compact space the\n    /// Err(position where it would be inserted)\n    fn u128_to_compact(&self, value: u128) -> Result<u32, usize> {\n        self.ranges_mapping\n            .binary_search_by(|probe| {\n                let value_range: &RangeInclusive<u128> = &probe.value_range;\n                if value < *value_range.start() {\n                    Ordering::Greater\n                } else if value > *value_range.end() {\n                    Ordering::Less\n                } else {\n                    Ordering::Equal\n                }\n            })\n            .map(|pos| {\n                let range_mapping = &self.ranges_mapping[pos];\n                let pos_in_range: u32 = (value - range_mapping.value_range.start()) as u32;\n                range_mapping.compact_start + pos_in_range\n            })\n    }\n\n    /// Unpacks a value from compact space u32 to u128 space\n    fn compact_to_u128(&self, compact: u32) -> u128 {\n        let pos = self\n            .ranges_mapping\n            .binary_search_by_key(&compact, |range_mapping| range_mapping.compact_start)\n            // Correctness: Overflow. The first range starts at compact space 0, the error from\n            // binary search can never be 0\n            .unwrap_or_else(|e| e - 1);\n\n        let range_mapping = &self.ranges_mapping[pos];\n        let diff = compact - range_mapping.compact_start;\n        range_mapping.value_range.start() + diff as u128\n    }\n}\n\npub struct CompactSpaceCompressor {\n    params: IPCodecParams,\n}\n\n#[derive(Debug, Clone)]\npub struct IPCodecParams {\n    compact_space: CompactSpace,\n    bit_unpacker: BitUnpacker,\n    min_value: u128,\n    max_value: u128,\n    num_vals: RowId,\n    num_bits: u8,\n}\n\nimpl CompactSpaceCompressor {\n    pub fn num_vals(&self) -> RowId {\n        self.params.num_vals\n    }\n\n    /// Taking the vals as Vec may cost a lot of memory. It is used to sort the vals.\n    pub fn train_from(iter: impl Iterator<Item = u128>) -> Self {\n        let mut values_sorted = BTreeSet::new();\n        // Total number of values, with their redundancy.\n        let mut total_num_values = 0u32;\n        for val in iter {\n            total_num_values += 1u32;\n            values_sorted.insert(val);\n        }\n        let min_value = *values_sorted.iter().next().unwrap_or(&0);\n        let max_value = *values_sorted.iter().last().unwrap_or(&0);\n\n        let compact_space =\n            get_compact_space(&values_sorted, total_num_values, COST_PER_BLANK_IN_BITS);\n        let amplitude_compact_space = compact_space.amplitude_compact_space();\n\n        assert!(\n            amplitude_compact_space <= u64::MAX as u128,\n            \"case unsupported.\"\n        );\n\n        let num_bits = tantivy_bitpacker::compute_num_bits(amplitude_compact_space as u64);\n\n        assert_eq!(\n            compact_space\n                .u128_to_compact(max_value)\n                .expect(\"could not convert max value to compact space\"),\n            amplitude_compact_space as u32\n        );\n        CompactSpaceCompressor {\n            params: IPCodecParams {\n                compact_space,\n                bit_unpacker: BitUnpacker::new(num_bits),\n                min_value,\n                max_value,\n                num_vals: total_num_values,\n                num_bits,\n            },\n        }\n    }\n\n    fn write_footer(self, writer: &mut impl Write) -> io::Result<()> {\n        let writer = &mut CountingWriter::wrap(writer);\n        self.params.serialize(writer)?;\n\n        let footer_len = writer.written_bytes() as u32;\n        footer_len.serialize(writer)?;\n\n        Ok(())\n    }\n\n    pub fn compress_into(\n        self,\n        vals: impl Iterator<Item = u128>,\n        write: &mut impl Write,\n    ) -> io::Result<()> {\n        let mut bitpacker = BitPacker::default();\n        for val in vals {\n            let compact = self\n                .params\n                .compact_space\n                .u128_to_compact(val)\n                .map_err(|_| {\n                    io::Error::new(\n                        io::ErrorKind::InvalidData,\n                        \"Could not convert value to compact_space. This is a bug.\",\n                    )\n                })?;\n            bitpacker.write(compact as u64, self.params.num_bits, write)?;\n        }\n        bitpacker.close(write)?;\n        self.write_footer(write)?;\n        Ok(())\n    }\n}\n\n#[derive(Debug, Clone)]\npub struct CompactSpaceDecompressor {\n    data: OwnedBytes,\n    params: IPCodecParams,\n}\n\nimpl BinarySerializable for IPCodecParams {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        // header flags for future optional dictionary encoding\n        let footer_flags = 0u64;\n        footer_flags.serialize(writer)?;\n\n        VIntU128(self.min_value).serialize(writer)?;\n        VIntU128(self.max_value).serialize(writer)?;\n        VIntU128(self.num_vals as u128).serialize(writer)?;\n        self.num_bits.serialize(writer)?;\n\n        self.compact_space.serialize(writer)?;\n\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let _header_flags = u64::deserialize(reader)?;\n        let min_value = VIntU128::deserialize(reader)?.0;\n        let max_value = VIntU128::deserialize(reader)?.0;\n        let num_vals = VIntU128::deserialize(reader)?.0 as u32;\n        let num_bits = u8::deserialize(reader)?;\n        let compact_space = CompactSpace::deserialize(reader)?;\n\n        Ok(Self {\n            compact_space,\n            bit_unpacker: BitUnpacker::new(num_bits),\n            min_value,\n            max_value,\n            num_vals,\n            num_bits,\n        })\n    }\n}\n\n/// Represents the result of looking up a u128 value in the compact space.\n///\n/// If a value is outside the compact space, the next compact value is returned.\n#[derive(Debug, Clone, Copy, PartialEq, Eq)]\npub enum CompactHit {\n    /// The value exists in the compact space\n    Exact(u32),\n    /// The value does not exist in the compact space, but the next higher value does\n    Next(u32),\n    /// The value is greater than the maximum compact value\n    AfterLast,\n}\n\n/// Exposes the compact space compressed values as u64.\n///\n/// This allows faster access to the values, as u64 is faster to work with than u128.\n/// It also allows to handle u128 values like u64, via the `open_u64_lenient` as a uniform\n/// access interface.\n///\n/// When converting from the internal u64 to u128 `compact_to_u128` can be used.\npub struct CompactSpaceU64Accessor(CompactSpaceDecompressor);\nimpl CompactSpaceU64Accessor {\n    pub(crate) fn open(data: OwnedBytes) -> io::Result<CompactSpaceU64Accessor> {\n        let decompressor = CompactSpaceU64Accessor(CompactSpaceDecompressor::open(data)?);\n        Ok(decompressor)\n    }\n    /// Convert a compact space value to u128\n    pub fn compact_to_u128(&self, compact: u32) -> u128 {\n        self.0.compact_to_u128(compact)\n    }\n\n    /// Finds the next compact space value for a given u128 value.\n    pub fn u128_to_next_compact(&self, value: u128) -> CompactHit {\n        self.0.u128_to_next_compact(value)\n    }\n}\n\nimpl ColumnValues<u64> for CompactSpaceU64Accessor {\n    #[inline]\n    fn get_val(&self, doc: u32) -> u64 {\n        let compact = self.0.get_compact(doc);\n        compact as u64\n    }\n\n    fn min_value(&self) -> u64 {\n        self.0.u128_to_compact(self.0.min_value()).unwrap() as u64\n    }\n\n    fn max_value(&self) -> u64 {\n        self.0.u128_to_compact(self.0.max_value()).unwrap() as u64\n    }\n\n    fn num_vals(&self) -> u32 {\n        self.0.params.num_vals\n    }\n\n    #[inline]\n    fn iter(&self) -> Box<dyn Iterator<Item = u64> + '_> {\n        Box::new(self.0.iter_compact().map(|el| el as u64))\n    }\n\n    #[inline]\n    fn get_row_ids_for_value_range(\n        &self,\n        value_range: RangeInclusive<u64>,\n        position_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        let value_range = self.0.compact_to_u128(*value_range.start() as u32)\n            ..=self.0.compact_to_u128(*value_range.end() as u32);\n        self.0\n            .get_row_ids_for_value_range(value_range, position_range, positions)\n    }\n}\n\nimpl ColumnValues<u128> for CompactSpaceDecompressor {\n    #[inline]\n    fn get_val(&self, doc: u32) -> u128 {\n        self.get(doc)\n    }\n\n    fn min_value(&self) -> u128 {\n        self.min_value()\n    }\n\n    fn max_value(&self) -> u128 {\n        self.max_value()\n    }\n\n    fn num_vals(&self) -> u32 {\n        self.params.num_vals\n    }\n\n    #[inline]\n    fn iter(&self) -> Box<dyn Iterator<Item = u128> + '_> {\n        Box::new(self.iter())\n    }\n\n    #[inline]\n    fn get_row_ids_for_value_range(\n        &self,\n        value_range: RangeInclusive<u128>,\n        position_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        if value_range.start() > value_range.end() {\n            return;\n        }\n        let position_range = position_range.start..position_range.end.min(self.num_vals());\n        let from_value = *value_range.start();\n        let to_value = *value_range.end();\n        assert!(to_value >= from_value);\n        let compact_from = self.u128_to_compact(from_value);\n        let compact_to = self.u128_to_compact(to_value);\n\n        // Quick return, if both ranges fall into the same non-mapped space, the range can't cover\n        // any values, so we can early exit\n        match (compact_to, compact_from) {\n            (Err(pos1), Err(pos2)) if pos1 == pos2 => return,\n            _ => {}\n        }\n\n        let compact_from = compact_from.unwrap_or_else(|pos| {\n            // Correctness: Out of bounds, if this value is Err(last_index + 1), we early exit,\n            // since the to_value also mapps into the same non-mapped space\n            let range_mapping = self.params.compact_space.get_range_mapping(pos);\n            range_mapping.compact_start\n        });\n        // If there is no compact space, we go to the closest upperbound compact space\n        let compact_to = compact_to.unwrap_or_else(|pos| {\n            // Correctness: Overflow, if this value is Err(0), we early exit,\n            // since the from_value also mapps into the same non-mapped space\n\n            // Get end of previous range\n            let pos = pos - 1;\n            let range_mapping = self.params.compact_space.get_range_mapping(pos);\n            range_mapping.compact_end()\n        });\n\n        let value_range = compact_from..=compact_to;\n        self.get_positions_for_compact_value_range(value_range, position_range, positions);\n    }\n}\n\nimpl CompactSpaceDecompressor {\n    pub fn open(data: OwnedBytes) -> io::Result<CompactSpaceDecompressor> {\n        let (data_slice, footer_len_bytes) = data.split_at(data.len() - 4);\n        let footer_len = u32::deserialize(&mut &footer_len_bytes[..])?;\n\n        let data_footer = &data_slice[data_slice.len() - footer_len as usize..];\n        let params = IPCodecParams::deserialize(&mut &data_footer[..])?;\n        let decompressor = CompactSpaceDecompressor { data, params };\n\n        Ok(decompressor)\n    }\n\n    /// Converting to compact space for the decompressor is more complex, since we may get values\n    /// which are outside the compact space. e.g. if we map\n    /// 1000 => 5\n    /// 2000 => 6\n    ///\n    /// and we want a mapping for 1005, there is no equivalent compact space. We instead return an\n    /// error with the index of the next range.\n    fn u128_to_compact(&self, value: u128) -> Result<u32, usize> {\n        self.params.compact_space.u128_to_compact(value)\n    }\n\n    /// Finds the next compact space value for a given u128 value.\n    pub fn u128_to_next_compact(&self, value: u128) -> CompactHit {\n        match self.u128_to_compact(value) {\n            Ok(compact) => CompactHit::Exact(compact),\n            Err(pos) => {\n                if pos >= self.params.compact_space.ranges_mapping.len() {\n                    CompactHit::AfterLast\n                } else {\n                    let next_range = &self.params.compact_space.ranges_mapping[pos];\n                    CompactHit::Next(next_range.compact_start)\n                }\n            }\n        }\n    }\n\n    fn compact_to_u128(&self, compact: u32) -> u128 {\n        self.params.compact_space.compact_to_u128(compact)\n    }\n\n    #[inline]\n    fn iter_compact(&self) -> impl Iterator<Item = u32> + '_ {\n        (0..self.params.num_vals)\n            .map(move |idx| self.params.bit_unpacker.get(idx, &self.data) as u32)\n    }\n\n    #[inline]\n    fn iter(&self) -> impl Iterator<Item = u128> + '_ {\n        // TODO: Performance. It would be better to iterate on the ranges and check existence via\n        // the bit_unpacker.\n        self.iter_compact()\n            .map(|compact| self.compact_to_u128(compact))\n    }\n\n    #[inline]\n    pub fn get_compact(&self, idx: u32) -> u32 {\n        self.params.bit_unpacker.get(idx, &self.data) as u32\n    }\n\n    #[inline]\n    pub fn get(&self, idx: u32) -> u128 {\n        let compact = self.get_compact(idx);\n        self.compact_to_u128(compact)\n    }\n\n    pub fn min_value(&self) -> u128 {\n        self.params.min_value\n    }\n\n    pub fn max_value(&self) -> u128 {\n        self.params.max_value\n    }\n\n    fn get_positions_for_compact_value_range(\n        &self,\n        value_range: RangeInclusive<u32>,\n        position_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        self.params.bit_unpacker.get_ids_for_value_range(\n            *value_range.start() as u64..=*value_range.end() as u64,\n            position_range,\n            &self.data,\n            positions,\n        );\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use itertools::Itertools;\n\n    use super::*;\n    use crate::column_values::u128_based::U128Header;\n    use crate::column_values::{open_u128_mapped, serialize_column_values_u128};\n\n    #[test]\n    fn compact_space_test() {\n        let ips: BTreeSet<u128> = [\n            2u128, 4u128, 1000, 1001, 1002, 1003, 1004, 1005, 1008, 1010, 1012, 1260,\n        ]\n        .into_iter()\n        .collect();\n        let compact_space = get_compact_space(&ips, ips.len() as u32, 11);\n        let amplitude = compact_space.amplitude_compact_space();\n        assert_eq!(amplitude, 17);\n        assert_eq!(1, compact_space.u128_to_compact(2).unwrap());\n        assert_eq!(2, compact_space.u128_to_compact(3).unwrap());\n        assert_eq!(compact_space.u128_to_compact(100).unwrap_err(), 1);\n\n        for (num1, num2) in (0..3).tuple_windows() {\n            assert_eq!(\n                compact_space.get_range_mapping(num1).compact_end() + 1,\n                compact_space.get_range_mapping(num2).compact_start\n            );\n        }\n\n        let mut output: Vec<u8> = Vec::new();\n        compact_space.serialize(&mut output).unwrap();\n\n        assert_eq!(\n            compact_space,\n            CompactSpace::deserialize(&mut &output[..]).unwrap()\n        );\n\n        for ip in ips {\n            let compact = compact_space.u128_to_compact(ip).unwrap();\n            assert_eq!(compact_space.compact_to_u128(compact), ip);\n        }\n    }\n\n    #[test]\n    fn compact_space_amplitude_test() {\n        let ips = &[100000u128, 1000000].into_iter().collect();\n        let compact_space = get_compact_space(ips, ips.len() as u32, 1);\n        let amplitude = compact_space.amplitude_compact_space();\n        assert_eq!(amplitude, 2);\n    }\n\n    fn test_all(mut data: OwnedBytes, expected: &[u128]) {\n        let _header = U128Header::deserialize(&mut data);\n        let decompressor = CompactSpaceDecompressor::open(data).unwrap();\n        for (idx, expected_val) in expected.iter().cloned().enumerate() {\n            let val = decompressor.get(idx as u32);\n            assert_eq!(val, expected_val);\n\n            let test_range = |range: RangeInclusive<u128>| {\n                let expected_positions = expected\n                    .iter()\n                    .positions(|val| range.contains(val))\n                    .map(|pos| pos as u32)\n                    .collect::<Vec<_>>();\n                let mut positions = Vec::new();\n                decompressor.get_row_ids_for_value_range(\n                    range,\n                    0..decompressor.num_vals(),\n                    &mut positions,\n                );\n                assert_eq!(positions, expected_positions);\n            };\n\n            test_range(expected_val.saturating_sub(1)..=expected_val);\n            test_range(expected_val..=expected_val);\n            test_range(expected_val..=expected_val.saturating_add(1));\n            test_range(expected_val.saturating_sub(1)..=expected_val.saturating_add(1));\n        }\n    }\n\n    fn test_aux_vals(u128_vals: &[u128]) -> OwnedBytes {\n        let mut out = Vec::new();\n        serialize_column_values_u128(&u128_vals, &mut out).unwrap();\n        let data = OwnedBytes::new(out);\n        test_all(data.clone(), u128_vals);\n        data\n    }\n\n    #[test]\n    fn test_range_1() {\n        let vals = &[\n            1u128,\n            100u128,\n            3u128,\n            99999u128,\n            100000u128,\n            100001u128,\n            4_000_211_221u128,\n            4_000_211_222u128,\n            333u128,\n        ];\n        let mut data = test_aux_vals(vals);\n\n        let _header = U128Header::deserialize(&mut data);\n        let decomp = CompactSpaceDecompressor::open(data).unwrap();\n        let complete_range = 0..vals.len() as u32;\n        for (pos, val) in vals.iter().enumerate() {\n            let val = *val;\n            let pos = pos as u32;\n            let mut positions = Vec::new();\n            decomp.get_row_ids_for_value_range(val..=val, pos..pos + 1, &mut positions);\n            assert_eq!(positions, vec![pos]);\n        }\n\n        // handle docid range out of bounds\n        let positions: Vec<u32> = get_positions_for_value_range_helper(&decomp, 0..=1, 1..u32::MAX);\n        assert!(positions.is_empty());\n\n        let positions =\n            get_positions_for_value_range_helper(&decomp, 0..=1, complete_range.clone());\n        assert_eq!(positions, vec![0]);\n        let positions =\n            get_positions_for_value_range_helper(&decomp, 0..=2, complete_range.clone());\n        assert_eq!(positions, vec![0]);\n        let positions =\n            get_positions_for_value_range_helper(&decomp, 0..=3, complete_range.clone());\n        assert_eq!(positions, vec![0, 2]);\n        assert_eq!(\n            get_positions_for_value_range_helper(\n                &decomp,\n                99999u128..=99999u128,\n                complete_range.clone()\n            ),\n            vec![3]\n        );\n        assert_eq!(\n            get_positions_for_value_range_helper(\n                &decomp,\n                99999u128..=100000u128,\n                complete_range.clone()\n            ),\n            vec![3, 4]\n        );\n        assert_eq!(\n            get_positions_for_value_range_helper(\n                &decomp,\n                99998u128..=100000u128,\n                complete_range.clone()\n            ),\n            vec![3, 4]\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                99998u128..=99999u128,\n                complete_range.clone()\n            ),\n            &[3]\n        );\n        assert!(\n            get_positions_for_value_range_helper(\n                &decomp,\n                99998u128..=99998u128,\n                complete_range.clone()\n            )\n            .is_empty()\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                333u128..=333u128,\n                complete_range.clone()\n            ),\n            &[8]\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                332u128..=333u128,\n                complete_range.clone()\n            ),\n            &[8]\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                332u128..=334u128,\n                complete_range.clone()\n            ),\n            &[8]\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                333u128..=334u128,\n                complete_range.clone()\n            ),\n            &[8]\n        );\n\n        assert_eq!(\n            &get_positions_for_value_range_helper(\n                &decomp,\n                4_000_211_221u128..=5_000_000_000u128,\n                complete_range\n            ),\n            &[6, 7]\n        );\n    }\n\n    #[test]\n    fn test_empty() {\n        let vals = &[];\n        let data = test_aux_vals(vals);\n        let _decomp = CompactSpaceDecompressor::open(data).unwrap();\n    }\n\n    #[test]\n    fn test_range_2() {\n        let vals = &[\n            100u128,\n            99999u128,\n            100000u128,\n            100001u128,\n            4_000_211_221u128,\n            4_000_211_222u128,\n            333u128,\n        ];\n        let mut data = test_aux_vals(vals);\n        let _header = U128Header::deserialize(&mut data);\n        let decomp = CompactSpaceDecompressor::open(data).unwrap();\n        let complete_range = 0..vals.len() as u32;\n        assert!(\n            &get_positions_for_value_range_helper(&decomp, 0..=5, complete_range.clone())\n                .is_empty(),\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(&decomp, 0..=100, complete_range.clone()),\n            &[0]\n        );\n        assert_eq!(\n            &get_positions_for_value_range_helper(&decomp, 0..=105, complete_range),\n            &[0]\n        );\n    }\n\n    fn get_positions_for_value_range_helper<C: ColumnValues<T> + ?Sized, T: PartialOrd>(\n        column: &C,\n        value_range: RangeInclusive<T>,\n        doc_id_range: Range<u32>,\n    ) -> Vec<u32> {\n        let mut positions = Vec::new();\n        column.get_row_ids_for_value_range(value_range, doc_id_range, &mut positions);\n        positions\n    }\n\n    #[test]\n    fn test_range_3() {\n        let vals = &[\n            200u128,\n            201,\n            202,\n            203,\n            204,\n            204,\n            206,\n            207,\n            208,\n            209,\n            210,\n            1_000_000,\n            5_000_000_000,\n        ];\n        let mut out = Vec::new();\n        serialize_column_values_u128(&&vals[..], &mut out).unwrap();\n        let decomp = open_u128_mapped(OwnedBytes::new(out)).unwrap();\n        let complete_range = 0..vals.len() as u32;\n\n        assert_eq!(\n            get_positions_for_value_range_helper(&*decomp, 199..=200, complete_range.clone()),\n            vec![0]\n        );\n\n        assert_eq!(\n            get_positions_for_value_range_helper(&*decomp, 199..=201, complete_range.clone()),\n            vec![0, 1]\n        );\n\n        assert_eq!(\n            get_positions_for_value_range_helper(&*decomp, 200..=200, complete_range.clone()),\n            vec![0]\n        );\n\n        assert_eq!(\n            get_positions_for_value_range_helper(&*decomp, 1_000_000..=1_000_000, complete_range),\n            vec![11]\n        );\n    }\n\n    #[test]\n    fn test_bug1() {\n        let vals = &[9223372036854775806];\n        let _data = test_aux_vals(vals);\n    }\n\n    #[test]\n    fn test_bug2() {\n        let vals = &[340282366920938463463374607431768211455u128];\n        let _data = test_aux_vals(vals);\n    }\n\n    #[test]\n    fn test_bug3() {\n        let vals = &[340282366920938463463374607431768211454];\n        let _data = test_aux_vals(vals);\n    }\n\n    #[test]\n    fn test_bug4() {\n        let vals = &[340282366920938463463374607431768211455, 0];\n        let _data = test_aux_vals(vals);\n    }\n\n    #[test]\n    fn test_first_large_gaps() {\n        let vals = &[1_000_000_000u128; 100];\n        let _data = test_aux_vals(vals);\n    }\n\n    #[test]\n    fn test_u128_to_next_compact() {\n        let vals = &[100u128, 200u128, 1_000_000_000u128, 1_000_000_100u128];\n        let mut data = test_aux_vals(vals);\n\n        let _header = U128Header::deserialize(&mut data);\n        let decomp = CompactSpaceDecompressor::open(data).unwrap();\n\n        // Test value that's already in a range\n        let compact_100 = decomp.u128_to_compact(100).unwrap();\n        assert_eq!(\n            decomp.u128_to_next_compact(100),\n            CompactHit::Exact(compact_100)\n        );\n\n        // Test value between two ranges\n        let compact_million = decomp.u128_to_compact(1_000_000_000).unwrap();\n        assert_eq!(\n            decomp.u128_to_next_compact(250),\n            CompactHit::Next(compact_million)\n        );\n\n        // Test value before the first range\n        assert_eq!(\n            decomp.u128_to_next_compact(50),\n            CompactHit::Next(compact_100)\n        );\n\n        // Test value after the last range\n        assert_eq!(\n            decomp.u128_to_next_compact(10_000_000_000),\n            CompactHit::AfterLast\n        );\n    }\n\n    use proptest::prelude::*;\n\n    fn num_strategy() -> impl Strategy<Value = u128> {\n        prop_oneof![\n            1 => prop::num::u128::ANY.prop_map(|num| u128::MAX - (num % 10) ),\n            1 => prop::num::u128::ANY.prop_map(|num| i64::MAX as u128 + 5 - (num % 10) ),\n            1 => prop::num::u128::ANY.prop_map(|num| i128::MAX as u128 + 5 - (num % 10) ),\n            1 => prop::num::u128::ANY.prop_map(|num| num % 10 ),\n            20 => prop::num::u128::ANY,\n        ]\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(10))]\n\n        #[test]\n        fn compress_decompress_random(vals in proptest::collection::vec(num_strategy() , 1..1000)) {\n            let _data = test_aux_vals(&vals);\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u128_based/mod.rs",
    "content": "use std::fmt::Debug;\nuse std::io;\nuse std::io::Write;\nuse std::sync::Arc;\n\nmod compact_space;\n\nuse common::{BinarySerializable, OwnedBytes, VInt};\npub use compact_space::{\n    CompactHit, CompactSpaceCompressor, CompactSpaceDecompressor, CompactSpaceU64Accessor,\n};\n\nuse crate::column_values::monotonic_map_column;\nuse crate::column_values::monotonic_mapping::{\n    StrictlyMonotonicMappingInverter, StrictlyMonotonicMappingToInternal,\n};\nuse crate::iterable::Iterable;\nuse crate::{ColumnValues, MonotonicallyMappableToU128};\n\n#[derive(Debug, Copy, Clone, PartialEq, Eq)]\npub(crate) struct U128Header {\n    pub num_vals: u32,\n    pub codec_type: U128FastFieldCodecType,\n}\n\nimpl BinarySerializable for U128Header {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        VInt(self.num_vals as u64).serialize(writer)?;\n        self.codec_type.serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let num_vals = VInt::deserialize(reader)?.0 as u32;\n        let codec_type = U128FastFieldCodecType::deserialize(reader)?;\n        Ok(U128Header {\n            num_vals,\n            codec_type,\n        })\n    }\n}\n\n/// Serializes u128 values with the compact space codec.\npub fn serialize_column_values_u128<T: MonotonicallyMappableToU128>(\n    iterable: &dyn Iterable<T>,\n    output: &mut impl io::Write,\n) -> io::Result<()> {\n    let compressor = CompactSpaceCompressor::train_from(\n        iterable\n            .boxed_iter()\n            .map(MonotonicallyMappableToU128::to_u128),\n    );\n    let header = U128Header {\n        num_vals: compressor.num_vals(),\n        codec_type: U128FastFieldCodecType::CompactSpace,\n    };\n    header.serialize(output)?;\n    compressor.compress_into(\n        iterable\n            .boxed_iter()\n            .map(MonotonicallyMappableToU128::to_u128),\n        output,\n    )?;\n    Ok(())\n}\n\n#[derive(PartialEq, Eq, PartialOrd, Ord, Debug, Clone, Copy)]\n#[repr(u8)]\n/// Available codecs to use to encode the u128 (via [`MonotonicallyMappableToU128`]) converted data.\npub(crate) enum U128FastFieldCodecType {\n    /// This codec takes a large number space (u128) and reduces it to a compact number space, by\n    /// removing the holes.\n    CompactSpace = 1,\n}\n\nimpl BinarySerializable for U128FastFieldCodecType {\n    fn serialize<W: Write + ?Sized>(&self, wrt: &mut W) -> io::Result<()> {\n        self.to_code().serialize(wrt)\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let code = u8::deserialize(reader)?;\n        let codec_type: Self = Self::from_code(code)\n            .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, \"Unknown code `{code}.`\"))?;\n        Ok(codec_type)\n    }\n}\n\nimpl U128FastFieldCodecType {\n    pub(crate) fn to_code(self) -> u8 {\n        self as u8\n    }\n\n    pub(crate) fn from_code(code: u8) -> Option<Self> {\n        match code {\n            1 => Some(Self::CompactSpace),\n            _ => None,\n        }\n    }\n}\n\n/// Returns the correct codec reader wrapped in the `Arc` for the data.\npub fn open_u128_mapped<T: MonotonicallyMappableToU128 + Debug>(\n    mut bytes: OwnedBytes,\n) -> io::Result<Arc<dyn ColumnValues<T>>> {\n    let header = U128Header::deserialize(&mut bytes)?;\n    assert_eq!(header.codec_type, U128FastFieldCodecType::CompactSpace);\n    let reader = CompactSpaceDecompressor::open(bytes)?;\n    let inverted: StrictlyMonotonicMappingInverter<StrictlyMonotonicMappingToInternal<T>> =\n        StrictlyMonotonicMappingToInternal::<T>::new().into();\n    Ok(Arc::new(monotonic_map_column(reader, inverted)))\n}\n\n/// Returns the u64 representation of the u128 data.\n/// The internal representation of the data as u64 is useful for faster processing.\n///\n/// In order to convert to u128 back cast to `CompactSpaceU64Accessor` and call\n/// `compact_to_u128`.\n///\n/// # Notice\n/// In case there are new codecs added, check for usages of `CompactSpaceDecompressorU64` and\n/// also handle the new codecs.\npub fn open_u128_as_compact_u64(mut bytes: OwnedBytes) -> io::Result<Arc<dyn ColumnValues<u64>>> {\n    let header = U128Header::deserialize(&mut bytes)?;\n    assert_eq!(header.codec_type, U128FastFieldCodecType::CompactSpace);\n    let reader = CompactSpaceU64Accessor::open(bytes)?;\n    Ok(Arc::new(reader))\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use super::*;\n    use crate::column_values::CodecType;\n    use crate::column_values::u64_based::{\n        ALL_U64_CODEC_TYPES, serialize_and_load_u64_based_column_values,\n        serialize_u64_based_column_values,\n    };\n\n    #[test]\n    fn test_serialize_deserialize_u128_header() {\n        let original = U128Header {\n            num_vals: 11,\n            codec_type: U128FastFieldCodecType::CompactSpace,\n        };\n        let mut out = Vec::new();\n        original.serialize(&mut out).unwrap();\n        let restored = U128Header::deserialize(&mut &out[..]).unwrap();\n        assert_eq!(restored, original);\n    }\n\n    #[test]\n    fn test_serialize_deserialize() {\n        let original = [1u64, 5u64, 10u64];\n        let restored: Vec<u64> =\n            serialize_and_load_u64_based_column_values(&&original[..], &ALL_U64_CODEC_TYPES)\n                .iter()\n                .collect();\n        assert_eq!(&restored, &original[..]);\n    }\n\n    #[test]\n    fn test_fastfield_bool_size_bitwidth_1() {\n        let mut buffer = Vec::new();\n        serialize_u64_based_column_values::<bool>(\n            &&[false, true][..],\n            &ALL_U64_CODEC_TYPES,\n            &mut buffer,\n        )\n        .unwrap();\n        // TODO put the header as a footer so that it serves as a padding.\n        // 5 bytes of header, 1 byte of value, 7 bytes of padding.\n        assert_eq!(buffer.len(), 5 + 1);\n    }\n\n    #[test]\n    fn test_fastfield_bool_bit_size_bitwidth_0() {\n        let mut buffer = Vec::new();\n        serialize_u64_based_column_values::<bool>(\n            &&[false, true][..],\n            &ALL_U64_CODEC_TYPES,\n            &mut buffer,\n        )\n        .unwrap();\n        // 6 bytes of header, 0 bytes of value, 7 bytes of padding.\n        assert_eq!(buffer.len(), 6);\n    }\n\n    #[test]\n    fn test_fastfield_gcd() {\n        let mut buffer = Vec::new();\n        let vals: Vec<u64> = (0..80).map(|val| (val % 7) * 1_000u64).collect();\n        serialize_u64_based_column_values(&&vals[..], &[CodecType::Bitpacked], &mut buffer)\n            .unwrap();\n        // Values are stored over 3 bits.\n        assert_eq!(buffer.len(), 6 + (3 * 80 / 8));\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/bitpacked.rs",
    "content": "use std::io::{self, Write};\nuse std::num::NonZeroU64;\nuse std::ops::{Range, RangeInclusive};\n\nuse common::{BinarySerializable, OwnedBytes};\nuse fastdivide::DividerU64;\nuse tantivy_bitpacker::{BitPacker, BitUnpacker, compute_num_bits};\n\nuse crate::column_values::u64_based::{ColumnCodec, ColumnCodecEstimator, ColumnStats};\nuse crate::{ColumnValues, RowId};\n\n/// Depending on the field type, a different\n/// fast field is required.\n#[derive(Clone)]\npub struct BitpackedReader {\n    data: OwnedBytes,\n    bit_unpacker: BitUnpacker,\n    stats: ColumnStats,\n}\n\n#[inline(always)]\nconst fn div_ceil(n: u64, q: NonZeroU64) -> u64 {\n    // copied from unstable rust standard library.\n    let d = n / q.get();\n    let r = n % q.get();\n    if r > 0 { d + 1 } else { d }\n}\n\n// The bitpacked codec applies a linear transformation `f` over data that are bitpacked.\n// f is defined by:\n// f: bitpacked -> stats.min_value + stats.gcd * bitpacked\n//\n// In order to run range queries, we invert the transformation.\n// `transform_range_before_linear_transformation` returns the range of values\n// [min_bipacked_value..max_bitpacked_value] such that\n// f(bitpacked) ∈ [min_value, max_value] <=> bitpacked ∈ [min_bitpacked_value, max_bitpacked_value]\nfn transform_range_before_linear_transformation(\n    stats: &ColumnStats,\n    range: RangeInclusive<u64>,\n) -> Option<RangeInclusive<u64>> {\n    if range.is_empty() {\n        return None;\n    }\n    let shifted_range =\n        range.start().saturating_sub(stats.min_value)..=range.end().saturating_sub(stats.min_value);\n    let start_before_gcd_multiplication: u64 = div_ceil(*shifted_range.start(), stats.gcd);\n    let end_before_gcd_multiplication: u64 = *shifted_range.end() / stats.gcd;\n    Some(start_before_gcd_multiplication..=end_before_gcd_multiplication)\n}\n\nimpl ColumnValues for BitpackedReader {\n    #[inline(always)]\n    fn get_val(&self, doc: u32) -> u64 {\n        self.stats.min_value + self.stats.gcd.get() * self.bit_unpacker.get(doc, &self.data)\n    }\n    #[inline]\n    fn min_value(&self) -> u64 {\n        self.stats.min_value\n    }\n    #[inline]\n    fn max_value(&self) -> u64 {\n        self.stats.max_value\n    }\n    #[inline]\n    fn num_vals(&self) -> RowId {\n        self.stats.num_rows\n    }\n\n    fn get_row_ids_for_value_range(\n        &self,\n        range: RangeInclusive<u64>,\n        doc_id_range: Range<u32>,\n        positions: &mut Vec<u32>,\n    ) {\n        let Some(transformed_range) =\n            transform_range_before_linear_transformation(&self.stats, range)\n        else {\n            positions.clear();\n            return;\n        };\n        self.bit_unpacker.get_ids_for_value_range(\n            transformed_range,\n            doc_id_range,\n            &self.data,\n            positions,\n        );\n    }\n}\n\nfn num_bits(stats: &ColumnStats) -> u8 {\n    compute_num_bits(stats.amplitude() / stats.gcd)\n}\n\n#[derive(Default)]\npub struct BitpackedCodecEstimator;\n\nimpl ColumnCodecEstimator for BitpackedCodecEstimator {\n    fn collect(&mut self, _value: u64) {}\n\n    fn estimate(&self, stats: &ColumnStats) -> Option<u64> {\n        let num_bits_per_value = num_bits(stats);\n        Some(stats.num_bytes() + (stats.num_rows as u64 * (num_bits_per_value as u64)).div_ceil(8))\n    }\n\n    fn serialize(\n        &self,\n        stats: &ColumnStats,\n        vals: &mut dyn Iterator<Item = u64>,\n        wrt: &mut dyn Write,\n    ) -> io::Result<()> {\n        stats.serialize(wrt)?;\n        let num_bits = num_bits(stats);\n        let mut bit_packer = BitPacker::new();\n        let divider = DividerU64::divide_by(stats.gcd.get());\n        for val in vals {\n            bit_packer.write(divider.divide(val - stats.min_value), num_bits, wrt)?;\n        }\n        bit_packer.close(wrt)?;\n        Ok(())\n    }\n}\n\npub struct BitpackedCodec;\n\nimpl ColumnCodec for BitpackedCodec {\n    type ColumnValues = BitpackedReader;\n    type Estimator = BitpackedCodecEstimator;\n\n    /// Opens a fast field given a file.\n    fn load(mut data: OwnedBytes) -> io::Result<Self::ColumnValues> {\n        let stats = ColumnStats::deserialize(&mut data)?;\n        let num_bits = num_bits(&stats);\n        let bit_unpacker = BitUnpacker::new(num_bits);\n        Ok(BitpackedReader {\n            data,\n            bit_unpacker,\n            stats,\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::column_values::u64_based::tests::create_and_validate;\n\n    #[test]\n    fn test_with_codec_data_sets_simple() {\n        create_and_validate::<BitpackedCodec>(&[4, 3, 12], \"name\");\n    }\n\n    #[test]\n    fn test_with_codec_data_sets_simple_gcd() {\n        create_and_validate::<BitpackedCodec>(&[1000, 2000, 3000], \"name\");\n    }\n\n    #[test]\n    fn test_with_codec_data_sets() {\n        let data_sets = crate::column_values::u64_based::tests::get_codec_test_datasets();\n        for (mut data, name) in data_sets {\n            create_and_validate::<BitpackedCodec>(&data, name);\n            data.reverse();\n            create_and_validate::<BitpackedCodec>(&data, name);\n        }\n    }\n\n    #[test]\n    fn bitpacked_fast_field_rand() {\n        for _ in 0..500 {\n            let mut data = (0..1 + rand::random::<u8>() as usize)\n                .map(|_| rand::random::<i64>() as u64 / 2)\n                .collect::<Vec<_>>();\n            create_and_validate::<BitpackedCodec>(&data, \"rand\");\n            data.reverse();\n            create_and_validate::<BitpackedCodec>(&data, \"rand\");\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/blockwise_linear.rs",
    "content": "use std::io::Write;\nuse std::sync::Arc;\nuse std::{io, iter};\n\nuse common::{BinarySerializable, CountingWriter, DeserializeFrom, OwnedBytes};\nuse fastdivide::DividerU64;\nuse tantivy_bitpacker::{BitPacker, BitUnpacker, compute_num_bits};\n\nuse crate::MonotonicallyMappableToU64;\nuse crate::column_values::u64_based::line::Line;\nuse crate::column_values::u64_based::{ColumnCodec, ColumnCodecEstimator, ColumnStats};\nuse crate::column_values::{ColumnValues, VecColumn};\n\nconst BLOCK_SIZE: u32 = 512u32;\n\n#[derive(Debug, Default)]\nstruct Block {\n    line: Line,\n    bit_unpacker: BitUnpacker,\n    data_start_offset: usize,\n}\n\nimpl BinarySerializable for Block {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.line.serialize(writer)?;\n        self.bit_unpacker.bit_width().serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let line = Line::deserialize(reader)?;\n        let bit_width = u8::deserialize(reader)?;\n        Ok(Block {\n            line,\n            bit_unpacker: BitUnpacker::new(bit_width),\n            data_start_offset: 0,\n        })\n    }\n}\n\nfn compute_num_blocks(num_vals: u32) -> u32 {\n    num_vals.div_ceil(BLOCK_SIZE)\n}\n\npub struct BlockwiseLinearEstimator {\n    block: Vec<u64>,\n    values_num_bytes: u64,\n    meta_num_bytes: u64,\n}\n\nimpl Default for BlockwiseLinearEstimator {\n    fn default() -> Self {\n        Self {\n            block: Vec::with_capacity(BLOCK_SIZE as usize),\n            values_num_bytes: 0u64,\n            meta_num_bytes: 0u64,\n        }\n    }\n}\n\nimpl BlockwiseLinearEstimator {\n    fn flush_block_estimate(&mut self) {\n        if self.block.is_empty() {\n            return;\n        }\n        let column = VecColumn::from(std::mem::take(&mut self.block));\n        let line = Line::train(&column);\n        self.block = column.into();\n\n        let mut max_value = 0u64;\n        for (i, buffer_val) in self.block.iter().enumerate() {\n            let interpolated_val = line.eval(i as u32);\n            let val = buffer_val.wrapping_sub(interpolated_val);\n            max_value = val.max(max_value);\n        }\n        let bit_width = compute_num_bits(max_value) as usize;\n        self.values_num_bytes += (bit_width * self.block.len() + 7) as u64 / 8;\n        self.meta_num_bytes += 1 + line.num_bytes();\n    }\n}\n\nimpl ColumnCodecEstimator for BlockwiseLinearEstimator {\n    fn collect(&mut self, value: u64) {\n        self.block.push(value);\n        if self.block.len() == BLOCK_SIZE as usize {\n            self.flush_block_estimate();\n            self.block.clear();\n        }\n    }\n    fn estimate(&self, stats: &ColumnStats) -> Option<u64> {\n        let mut estimate = 4 + stats.num_bytes() + self.meta_num_bytes + self.values_num_bytes;\n        if stats.gcd.get() > 1 {\n            let estimate_gain_from_gcd =\n                (stats.gcd.get() as f32).log2().floor() * stats.num_rows as f32 / 8.0f32;\n            estimate = estimate.saturating_sub(estimate_gain_from_gcd as u64);\n        }\n        Some(estimate)\n    }\n\n    fn finalize(&mut self) {\n        self.flush_block_estimate();\n    }\n\n    fn serialize(\n        &self,\n        stats: &ColumnStats,\n        mut vals: &mut dyn Iterator<Item = u64>,\n        wrt: &mut dyn Write,\n    ) -> io::Result<()> {\n        stats.serialize(wrt)?;\n        let mut buffer = Vec::with_capacity(BLOCK_SIZE as usize);\n        let num_blocks = compute_num_blocks(stats.num_rows) as usize;\n        let mut blocks = Vec::with_capacity(num_blocks);\n\n        let mut bit_packer = BitPacker::new();\n\n        let gcd_divider = DividerU64::divide_by(stats.gcd.get());\n\n        for _ in 0..num_blocks {\n            buffer.clear();\n            buffer.extend(\n                (&mut vals)\n                    .map(MonotonicallyMappableToU64::to_u64)\n                    .take(BLOCK_SIZE as usize),\n            );\n\n            for buffer_val in buffer.iter_mut() {\n                *buffer_val = gcd_divider.divide(*buffer_val - stats.min_value);\n            }\n\n            let line = Line::train(&VecColumn::from(buffer.to_vec()));\n\n            assert!(!buffer.is_empty());\n\n            for (i, buffer_val) in buffer.iter_mut().enumerate() {\n                let interpolated_val = line.eval(i as u32);\n                *buffer_val = buffer_val.wrapping_sub(interpolated_val);\n            }\n\n            let bit_width = buffer.iter().copied().map(compute_num_bits).max().unwrap();\n\n            for &buffer_val in &buffer {\n                bit_packer.write(buffer_val, bit_width, wrt)?;\n            }\n\n            blocks.push(Block {\n                line,\n                bit_unpacker: BitUnpacker::new(bit_width),\n                data_start_offset: 0,\n            });\n        }\n\n        bit_packer.close(wrt)?;\n\n        assert_eq!(blocks.len(), num_blocks);\n\n        let mut counting_wrt = CountingWriter::wrap(wrt);\n        for block in &blocks {\n            block.serialize(&mut counting_wrt)?;\n        }\n        let footer_len = counting_wrt.written_bytes();\n        (footer_len as u32).serialize(&mut counting_wrt)?;\n\n        Ok(())\n    }\n}\n\npub struct BlockwiseLinearCodec;\n\nimpl ColumnCodec<u64> for BlockwiseLinearCodec {\n    type ColumnValues = BlockwiseLinearReader;\n\n    type Estimator = BlockwiseLinearEstimator;\n\n    fn load(mut bytes: OwnedBytes) -> io::Result<Self::ColumnValues> {\n        let stats = ColumnStats::deserialize(&mut bytes)?;\n        let footer_len: u32 = (&bytes[bytes.len() - 4..]).deserialize()?;\n        let footer_offset = bytes.len() - 4 - footer_len as usize;\n        let (data, mut footer) = bytes.split(footer_offset);\n        let num_blocks = compute_num_blocks(stats.num_rows);\n        let mut blocks: Vec<Block> = iter::repeat_with(|| Block::deserialize(&mut footer))\n            .take(num_blocks as usize)\n            .collect::<io::Result<_>>()?;\n        let mut start_offset = 0;\n        for block in &mut blocks {\n            block.data_start_offset = start_offset;\n            start_offset += (block.bit_unpacker.bit_width() as usize) * BLOCK_SIZE as usize / 8;\n        }\n        Ok(BlockwiseLinearReader {\n            blocks: blocks.into_boxed_slice().into(),\n            data,\n            stats,\n        })\n    }\n}\n\n#[derive(Clone)]\npub struct BlockwiseLinearReader {\n    blocks: Arc<[Block]>,\n    data: OwnedBytes,\n    stats: ColumnStats,\n}\n\nimpl ColumnValues for BlockwiseLinearReader {\n    #[inline(always)]\n    fn get_val(&self, idx: u32) -> u64 {\n        let block_id = (idx / BLOCK_SIZE) as usize;\n        let idx_within_block = idx % BLOCK_SIZE;\n        let block = &self.blocks[block_id];\n        let interpoled_val: u64 = block.line.eval(idx_within_block);\n        let block_bytes = &self.data[block.data_start_offset..];\n        let bitpacked_diff = block.bit_unpacker.get(idx_within_block, block_bytes);\n        // TODO optimize me! the line parameters could be tweaked to include the multiplication and\n        // remove the dependency.\n        self.stats.min_value\n            + self\n                .stats\n                .gcd\n                .get()\n                .wrapping_mul(interpoled_val.wrapping_add(bitpacked_diff))\n    }\n\n    #[inline(always)]\n    fn min_value(&self) -> u64 {\n        self.stats.min_value\n    }\n\n    #[inline(always)]\n    fn max_value(&self) -> u64 {\n        self.stats.max_value\n    }\n\n    #[inline(always)]\n    fn num_vals(&self) -> u32 {\n        self.stats.num_rows\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::column_values::u64_based::tests::create_and_validate;\n\n    #[test]\n    fn test_with_codec_data_sets_simple() {\n        create_and_validate::<BlockwiseLinearCodec>(\n            &[11, 20, 40, 20, 10, 10, 10, 10, 10, 10],\n            \"simple test\",\n        )\n        .unwrap();\n    }\n\n    #[test]\n    fn test_with_codec_data_sets_simple_gcd() {\n        let (_, actual_compression_rate) = create_and_validate::<BlockwiseLinearCodec>(\n            &[10, 20, 40, 20, 10, 10, 10, 10, 10, 10],\n            \"name\",\n        )\n        .unwrap();\n        assert_eq!(actual_compression_rate, 0.175);\n    }\n\n    #[test]\n    fn test_with_codec_data_sets() {\n        let data_sets = crate::column_values::u64_based::tests::get_codec_test_datasets();\n        for (mut data, name) in data_sets {\n            create_and_validate::<BlockwiseLinearCodec>(&data, name);\n            data.reverse();\n            create_and_validate::<BlockwiseLinearCodec>(&data, name);\n        }\n    }\n\n    #[test]\n    fn test_blockwise_linear_fast_field_rand() {\n        for _ in 0..500 {\n            let mut data = (0..1 + rand::random::<u8>() as usize)\n                .map(|_| rand::random::<i64>() as u64 / 2)\n                .collect::<Vec<_>>();\n            create_and_validate::<BlockwiseLinearCodec>(&data, \"rand\");\n            data.reverse();\n            create_and_validate::<BlockwiseLinearCodec>(&data, \"rand\");\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/line.rs",
    "content": "use std::io;\nuse std::num::NonZeroU32;\n\nuse common::{BinarySerializable, VInt};\n\nuse crate::column_values::ColumnValues;\n\nconst MID_POINT: u64 = (1u64 << 32) - 1u64;\n\n/// `Line` describes a line function `y: ax + b` using integer\n/// arithmetic.\n///\n/// The slope is in fact a decimal split into a 32 bit integer value,\n/// and a 32-bit decimal value.\n///\n/// The multiplication then becomes.\n/// `y = m * x >> 32 + b`\n#[derive(Debug, Clone, Copy, Default)]\npub struct Line {\n    pub(crate) slope: u64,\n    pub(crate) intercept: u64,\n}\n\n/// Compute the line slope.\n///\n/// This function has the nice property of being\n/// invariant by translation.\n/// `\n///   compute_slope(y0, y1)\n/// = compute_slope(y0 + X % 2^64, y1 + X % 2^64)\n/// `\nfn compute_slope(y0: u64, y1: u64, num_vals: NonZeroU32) -> u64 {\n    let dy = y1.wrapping_sub(y0);\n    let sign = dy <= (1 << 63);\n    let abs_dy = if sign {\n        y1.wrapping_sub(y0)\n    } else {\n        y0.wrapping_sub(y1)\n    };\n    if abs_dy >= 1 << 32 {\n        // This is outside of realm we handle.\n        // Let's just bail.\n        return 0u64;\n    }\n\n    let abs_slope = (abs_dy << 32) / num_vals.get() as u64;\n    if sign {\n        abs_slope\n    } else {\n        // The complement does indeed create the\n        // opposite decreasing slope...\n        //\n        // Intuitively (without the bitshifts and % u64::MAX)\n        // ```\n        //    (x + shift)*(u64::MAX - abs_slope)\n        // -  (x * (u64::MAX - abs_slope))\n        // = - shift * abs_slope\n        // ```\n        u64::MAX - abs_slope\n    }\n}\n\nimpl Line {\n    #[inline(always)]\n    pub fn eval(&self, x: u32) -> u64 {\n        let linear_part = ((x as u64).wrapping_mul(self.slope) >> 32) as i32 as u64;\n        self.intercept.wrapping_add(linear_part)\n    }\n\n    // Intercept is only computed from provided positions\n    pub fn train_from(\n        first_val: u64,\n        last_val: u64,\n        num_vals: u32,\n        positions_and_values: impl Iterator<Item = (u64, u64)>,\n    ) -> Self {\n        // TODO replace with let else\n        let idx_last_val = if let Some(idx_last_val) = NonZeroU32::new(num_vals - 1) {\n            idx_last_val\n        } else {\n            return Line::default();\n        };\n\n        let y0 = first_val;\n        let y1 = last_val;\n\n        // We first independently pick our slope.\n        let slope = compute_slope(y0, y1, idx_last_val);\n\n        // We picked our slope. Note that it does not have to be perfect.\n        // Now we need to compute the best intercept.\n        //\n        // Intuitively, the best intercept is such that line passes through one of the\n        // `(i, ys[])`.\n        //\n        // The best intercept therefore has the form\n        // `y[i] - line.eval(i)` (using wrapping arithmetic).\n        // In other words, the best intercept is one of the `y - Line::eval(ys[i])`\n        // and our task is just to pick the one that minimizes our error.\n        //\n        // Without sorting our values, this is a difficult problem.\n        // We however rely on the following trick...\n        //\n        // We only focus on the case where the interpolation is half decent.\n        // If the line interpolation is doing its job on a dataset suited for it,\n        // we can hope that the maximum error won't be larger than `u64::MAX / 2`.\n        //\n        // In other words, even without the intercept the values `y - Line::eval(ys[i])` will all be\n        // within an interval that takes less than half of the modulo space of `u64`.\n        //\n        // Our task is therefore to identify this interval.\n        // Here we simply translate all of our values by `y0 - 2^63` and pick the min.\n        let mut line = Line {\n            slope,\n            intercept: 0,\n        };\n        let heuristic_shift = y0.wrapping_sub(MID_POINT);\n        line.intercept = positions_and_values\n            .map(|(pos, y)| y.wrapping_sub(line.eval(pos as u32)))\n            .min_by_key(|&val| val.wrapping_sub(heuristic_shift))\n            .unwrap_or(0u64); //< Never happens.\n        line\n    }\n\n    /// Returns a line that attempts to approximate a function\n    /// f: i in 0..[ys.num_vals()) -> ys[i].\n    ///\n    /// - The approximation is always lower than the actual value. Or more rigorously, formally\n    ///   `f(i).wrapping_sub(ys[i])` is small for any i in [0..ys.len()).\n    /// - It computes without panicking for any value of it.\n    ///\n    /// This function is only invariable by translation if all of the\n    /// `ys` are packaged into half of the space. (See heuristic below)\n    /// TODO USE array\n    pub fn train(ys: &dyn ColumnValues) -> Self {\n        let first_val = ys.iter().next().unwrap();\n        let last_val = ys.iter().nth(ys.num_vals() as usize - 1).unwrap();\n        Self::train_from(\n            first_val,\n            last_val,\n            ys.num_vals(),\n            ys.iter().enumerate().map(|(pos, val)| (pos as u64, val)),\n        )\n    }\n}\n\nimpl BinarySerializable for Line {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        VInt(self.slope).serialize(writer)?;\n        VInt(self.intercept).serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let slope = VInt::deserialize(reader)?.0;\n        let intercept = VInt::deserialize(reader)?.0;\n        Ok(Line { slope, intercept })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::column_values::VecColumn;\n\n    /// Test training a line and ensuring that the maximum difference between\n    /// the data points and the line is `expected`.\n    ///\n    /// This function operates translation over the data for better coverage.\n    #[track_caller]\n    fn test_line_interpol_with_translation(ys: &[u64], expected: Option<u64>) {\n        let mut translations = vec![0, 100, u64::MAX / 2, u64::MAX, u64::MAX - 1];\n        translations.extend_from_slice(ys);\n        for translation in translations {\n            let translated_ys: Vec<u64> = ys\n                .iter()\n                .copied()\n                .map(|y| y.wrapping_add(translation))\n                .collect();\n            let largest_err = test_eval_max_err(&translated_ys);\n            assert_eq!(largest_err, expected);\n        }\n    }\n\n    fn test_eval_max_err(ys: &[u64]) -> Option<u64> {\n        let line = Line::train(&VecColumn::from(ys.to_vec()));\n        ys.iter()\n            .enumerate()\n            .map(|(x, y)| y.wrapping_sub(line.eval(x as u32)))\n            .max()\n    }\n\n    #[test]\n    fn test_train() {\n        test_line_interpol_with_translation(&[11, 11, 11, 12, 12, 13], Some(1));\n        test_line_interpol_with_translation(&[13, 12, 12, 11, 11, 11], Some(1));\n        test_line_interpol_with_translation(&[13, 13, 12, 11, 11, 11], Some(1));\n        test_line_interpol_with_translation(&[13, 13, 12, 11, 11, 11], Some(1));\n        test_line_interpol_with_translation(&[u64::MAX - 1, 0, 0, 1], Some(1));\n        test_line_interpol_with_translation(&[u64::MAX - 1, u64::MAX, 0, 1], Some(0));\n        test_line_interpol_with_translation(&[0, 1, 2, 3, 5], Some(0));\n        test_line_interpol_with_translation(&[1, 2, 3, 4], Some(0));\n\n        let data: Vec<u64> = (0..255).collect();\n        test_line_interpol_with_translation(&data, Some(0));\n        let data: Vec<u64> = (0..255).map(|el| el * 2).collect();\n        test_line_interpol_with_translation(&data, Some(0));\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/linear.rs",
    "content": "use std::io;\n\nuse common::{BinarySerializable, OwnedBytes};\nuse tantivy_bitpacker::{BitPacker, BitUnpacker, compute_num_bits};\n\nuse super::ColumnValues;\nuse super::line::Line;\nuse crate::RowId;\nuse crate::column_values::VecColumn;\nuse crate::column_values::u64_based::{ColumnCodec, ColumnCodecEstimator, ColumnStats};\n\nconst HALF_SPACE: u64 = u64::MAX / 2;\nconst LINE_ESTIMATION_BLOCK_LEN: usize = 512;\n\n/// Depending on the field type, a different\n/// fast field is required.\n#[derive(Clone)]\npub struct LinearReader {\n    data: OwnedBytes,\n    linear_params: LinearParams,\n    stats: ColumnStats,\n}\n\nimpl ColumnValues for LinearReader {\n    #[inline]\n    fn get_val(&self, doc: u32) -> u64 {\n        let interpoled_val: u64 = self.linear_params.line.eval(doc);\n        let bitpacked_diff = self.linear_params.bit_unpacker.get(doc, &self.data);\n        interpoled_val.wrapping_add(bitpacked_diff)\n    }\n\n    #[inline(always)]\n    fn min_value(&self) -> u64 {\n        self.stats.min_value\n    }\n\n    #[inline(always)]\n    fn max_value(&self) -> u64 {\n        self.stats.max_value\n    }\n\n    #[inline]\n    fn num_vals(&self) -> u32 {\n        self.stats.num_rows\n    }\n}\n\n/// Fastfield serializer, which tries to guess values by linear interpolation\n/// and stores the difference bitpacked.\npub struct LinearCodec;\n\n#[derive(Debug, Clone)]\nstruct LinearParams {\n    line: Line,\n    bit_unpacker: BitUnpacker,\n}\n\nimpl BinarySerializable for LinearParams {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.line.serialize(writer)?;\n        self.bit_unpacker.bit_width().serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let line = Line::deserialize(reader)?;\n        let bit_width = u8::deserialize(reader)?;\n        Ok(Self {\n            line,\n            bit_unpacker: BitUnpacker::new(bit_width),\n        })\n    }\n}\n\npub struct LinearCodecEstimator {\n    block: Vec<u64>,\n    line: Option<Line>,\n    row_id: RowId,\n    min_deviation: u64,\n    max_deviation: u64,\n    first_val: u64,\n    last_val: u64,\n}\n\nimpl Default for LinearCodecEstimator {\n    fn default() -> LinearCodecEstimator {\n        LinearCodecEstimator {\n            block: Vec::with_capacity(LINE_ESTIMATION_BLOCK_LEN),\n            line: None,\n            row_id: 0,\n            min_deviation: u64::MAX,\n            max_deviation: u64::MIN,\n            first_val: 0u64,\n            last_val: 0u64,\n        }\n    }\n}\n\nimpl ColumnCodecEstimator for LinearCodecEstimator {\n    fn finalize(&mut self) {\n        if let Some(line) = self.line.as_mut() {\n            line.intercept = line\n                .intercept\n                .wrapping_add(self.min_deviation)\n                .wrapping_sub(HALF_SPACE);\n        }\n    }\n\n    fn estimate(&self, stats: &ColumnStats) -> Option<u64> {\n        let line = self.line?;\n        let amplitude = self.max_deviation - self.min_deviation;\n        let num_bits = compute_num_bits(amplitude);\n        let linear_params = LinearParams {\n            line,\n            bit_unpacker: BitUnpacker::new(num_bits),\n        };\n        Some(\n            stats.num_bytes()\n                + linear_params.num_bytes()\n                + (num_bits as u64 * stats.num_rows as u64).div_ceil(8),\n        )\n    }\n\n    fn serialize(\n        &self,\n        stats: &ColumnStats,\n        vals: &mut dyn Iterator<Item = u64>,\n        wrt: &mut dyn io::Write,\n    ) -> io::Result<()> {\n        stats.serialize(wrt)?;\n        let line = self.line.unwrap();\n        let amplitude = self.max_deviation - self.min_deviation;\n        let num_bits = compute_num_bits(amplitude);\n        let linear_params = LinearParams {\n            line,\n            bit_unpacker: BitUnpacker::new(num_bits),\n        };\n        linear_params.serialize(wrt)?;\n        let mut bit_packer = BitPacker::new();\n        for (pos, value) in vals.enumerate() {\n            let calculated_value = line.eval(pos as u32);\n            let offset = value.wrapping_sub(calculated_value);\n            bit_packer.write(offset, num_bits, wrt)?;\n        }\n        bit_packer.close(wrt)?;\n        Ok(())\n    }\n\n    fn collect(&mut self, value: u64) {\n        if let Some(line) = self.line {\n            self.collect_after_line_estimation(&line, value);\n        } else {\n            self.collect_before_line_estimation(value);\n        }\n    }\n}\n\nimpl LinearCodecEstimator {\n    #[inline]\n    fn collect_after_line_estimation(&mut self, line: &Line, value: u64) {\n        let interpoled_val: u64 = line.eval(self.row_id);\n        let deviation = value.wrapping_add(HALF_SPACE).wrapping_sub(interpoled_val);\n        self.min_deviation = self.min_deviation.min(deviation);\n        self.max_deviation = self.max_deviation.max(deviation);\n        if self.row_id == 0 {\n            self.first_val = value;\n        }\n        self.last_val = value;\n        self.row_id += 1u32;\n    }\n\n    #[inline]\n    fn collect_before_line_estimation(&mut self, value: u64) {\n        self.block.push(value);\n        if self.block.len() == LINE_ESTIMATION_BLOCK_LEN {\n            let column = VecColumn::from(std::mem::take(&mut self.block));\n            let line = Line::train(&column);\n            self.block = column.into();\n            let block = std::mem::take(&mut self.block);\n            for val in block {\n                self.collect_after_line_estimation(&line, val);\n            }\n            self.line = Some(line);\n        }\n    }\n}\n\nimpl ColumnCodec for LinearCodec {\n    type ColumnValues = LinearReader;\n\n    type Estimator = LinearCodecEstimator;\n\n    fn load(mut data: OwnedBytes) -> io::Result<Self::ColumnValues> {\n        let stats = ColumnStats::deserialize(&mut data)?;\n        let linear_params = LinearParams::deserialize(&mut data)?;\n        Ok(LinearReader {\n            stats,\n            linear_params,\n            data,\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use rand::RngCore;\n\n    use super::*;\n    use crate::column_values::u64_based::tests::{create_and_validate, get_codec_test_datasets};\n\n    #[test]\n    fn test_compression_simple() {\n        let vals = (100u64..)\n            .take(super::LINE_ESTIMATION_BLOCK_LEN)\n            .collect::<Vec<_>>();\n        create_and_validate::<LinearCodec>(&vals, \"simple monotonically large\").unwrap();\n    }\n\n    #[test]\n    fn test_compression() {\n        let data = (10..=6_000_u64).collect::<Vec<_>>();\n        let (estimate, actual_compression) =\n            create_and_validate::<LinearCodec>(&data, \"simple monotonically large\").unwrap();\n        assert_le!(actual_compression, 0.001);\n        assert_le!(estimate, 0.02);\n    }\n\n    #[test]\n    fn test_with_codec_datasets() {\n        let data_sets = get_codec_test_datasets();\n        for (mut data, name) in data_sets {\n            create_and_validate::<LinearCodec>(&data, name);\n            data.reverse();\n            create_and_validate::<LinearCodec>(&data, name);\n        }\n    }\n    #[test]\n    fn linear_interpol_fast_field_test_large_amplitude() {\n        let data = vec![\n            i64::MAX as u64 / 2,\n            i64::MAX as u64 / 3,\n            i64::MAX as u64 / 2,\n        ];\n        create_and_validate::<LinearCodec>(&data, \"large amplitude\");\n    }\n\n    #[test]\n    fn overflow_error_test() {\n        let data = vec![1572656989877777, 1170935903116329, 720575940379279, 0];\n        create_and_validate::<LinearCodec>(&data, \"overflow test\");\n    }\n\n    #[test]\n    fn linear_interpol_fast_concave_data() {\n        let data = vec![0, 1, 2, 5, 8, 10, 20, 50];\n        create_and_validate::<LinearCodec>(&data, \"concave data\");\n    }\n    #[test]\n    fn linear_interpol_fast_convex_data() {\n        let data = vec![0, 40, 60, 70, 75, 77];\n        create_and_validate::<LinearCodec>(&data, \"convex data\");\n    }\n    #[test]\n    fn linear_interpol_fast_field_test_simple() {\n        let data = (10..=20_u64).collect::<Vec<_>>();\n        create_and_validate::<LinearCodec>(&data, \"simple monotonically\");\n    }\n\n    #[test]\n    fn linear_interpol_fast_field_rand() {\n        let mut rng = rand::rng();\n        for _ in 0..50 {\n            let mut data = (0..10_000).map(|_| rng.next_u64()).collect::<Vec<_>>();\n            create_and_validate::<LinearCodec>(&data, \"random\");\n            data.reverse();\n            create_and_validate::<LinearCodec>(&data, \"random\");\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/mod.rs",
    "content": "mod bitpacked;\nmod blockwise_linear;\nmod line;\nmod linear;\nmod stats_collector;\n\nuse std::io;\nuse std::io::Write;\nuse std::sync::Arc;\n\nuse common::{BinarySerializable, OwnedBytes};\n\nuse crate::column_values::monotonic_mapping::{\n    StrictlyMonotonicMappingInverter, StrictlyMonotonicMappingToInternal,\n};\npub use crate::column_values::u64_based::bitpacked::BitpackedCodec;\npub use crate::column_values::u64_based::blockwise_linear::BlockwiseLinearCodec;\npub use crate::column_values::u64_based::linear::LinearCodec;\npub use crate::column_values::u64_based::stats_collector::StatsCollector;\nuse crate::column_values::{ColumnStats, monotonic_map_column};\nuse crate::iterable::Iterable;\nuse crate::{ColumnValues, MonotonicallyMappableToU64};\n\n/// A `ColumnCodecEstimator` is in charge of gathering all\n/// data required to serialize a column.\n///\n/// This happens during a first pass on data of the column elements.\n/// During that pass, all column estimators receive a call to their\n/// `.collect(el)`.\n///\n/// After this first pass, finalize is called.\n/// `.estimate(..)` then should return an accurate estimation of the\n/// size of the serialized column (were we to pick this codec.).\n/// `.serialize(..)` then serializes the column using this codec.\npub trait ColumnCodecEstimator<T = u64>: 'static {\n    /// Records a new value for estimation.\n    /// This method will be called for each element of the column during\n    /// `estimation`.\n    fn collect(&mut self, value: u64);\n    /// Finalizes the first pass phase.\n    fn finalize(&mut self) {}\n    /// Returns an accurate estimation of the number of bytes that will\n    /// be used to represent this column.\n    fn estimate(&self, stats: &ColumnStats) -> Option<u64>;\n    /// Serializes the column using the given codec.\n    /// This constitutes a second pass over the columns values.\n    fn serialize(\n        &self,\n        stats: &ColumnStats,\n        vals: &mut dyn Iterator<Item = T>,\n        wrt: &mut dyn io::Write,\n    ) -> io::Result<()>;\n}\n\n/// A column codec describes a column serialization format.\npub trait ColumnCodec<T: PartialOrd = u64> {\n    /// Specialized `ColumnValues` type.\n    type ColumnValues: ColumnValues<T> + 'static;\n    /// `Estimator` for the given codec.\n    type Estimator: ColumnCodecEstimator + Default;\n\n    /// Loads a column that has been serialized using this codec.\n    fn load(bytes: OwnedBytes) -> io::Result<Self::ColumnValues>;\n\n    /// Returns an estimator.\n    fn estimator() -> Self::Estimator {\n        Self::Estimator::default()\n    }\n\n    /// Returns a boxed estimator.\n    fn boxed_estimator() -> Box<dyn ColumnCodecEstimator> {\n        Box::new(Self::estimator())\n    }\n}\n\n/// Available codecs to use to encode the u64 (via [`MonotonicallyMappableToU64`]) converted data.\n#[derive(PartialEq, Eq, PartialOrd, Ord, Debug, Clone, Copy)]\n#[repr(u8)]\npub enum CodecType {\n    /// Bitpack all values in the value range. The number of bits is defined by the amplitude\n    /// `column.max_value() - column.min_value()`\n    Bitpacked = 0u8,\n    /// Linear interpolation puts a line between the first and last value and then bitpacks the\n    /// values by the offset from the line. The number of bits is defined by the max deviation from\n    /// the line.\n    Linear = 1u8,\n    /// Same as [`CodecType::Linear`], but encodes in blocks of 512 elements.\n    BlockwiseLinear = 2u8,\n}\n\n/// List of all available u64-base codecs.\npub const ALL_U64_CODEC_TYPES: [CodecType; 3] = [\n    CodecType::Bitpacked,\n    CodecType::Linear,\n    CodecType::BlockwiseLinear,\n];\n\nimpl CodecType {\n    fn to_code(self) -> u8 {\n        self as u8\n    }\n\n    fn try_from_code(code: u8) -> Option<CodecType> {\n        match code {\n            0u8 => Some(CodecType::Bitpacked),\n            1u8 => Some(CodecType::Linear),\n            2u8 => Some(CodecType::BlockwiseLinear),\n            _ => None,\n        }\n    }\n\n    fn load<T: MonotonicallyMappableToU64>(\n        &self,\n        bytes: OwnedBytes,\n    ) -> io::Result<Arc<dyn ColumnValues<T>>> {\n        match self {\n            CodecType::Bitpacked => load_specific_codec::<BitpackedCodec, T>(bytes),\n            CodecType::Linear => load_specific_codec::<LinearCodec, T>(bytes),\n            CodecType::BlockwiseLinear => load_specific_codec::<BlockwiseLinearCodec, T>(bytes),\n        }\n    }\n}\n\nfn load_specific_codec<C: ColumnCodec, T: MonotonicallyMappableToU64>(\n    bytes: OwnedBytes,\n) -> io::Result<Arc<dyn ColumnValues<T>>> {\n    let reader = C::load(bytes)?;\n    let reader_typed = monotonic_map_column(\n        reader,\n        StrictlyMonotonicMappingInverter::from(StrictlyMonotonicMappingToInternal::<T>::new()),\n    );\n    Ok(Arc::new(reader_typed))\n}\n\nimpl CodecType {\n    /// Returns a boxed codec estimator associated to a given `CodecType`.\n    pub fn estimator(&self) -> Box<dyn ColumnCodecEstimator> {\n        match self {\n            CodecType::Bitpacked => BitpackedCodec::boxed_estimator(),\n            CodecType::Linear => LinearCodec::boxed_estimator(),\n            CodecType::BlockwiseLinear => BlockwiseLinearCodec::boxed_estimator(),\n        }\n    }\n}\n\n/// Serializes a given column of u64-mapped values.\npub fn serialize_u64_based_column_values<T: MonotonicallyMappableToU64>(\n    vals: &dyn Iterable<T>,\n    codec_types: &[CodecType],\n    wrt: &mut dyn Write,\n) -> io::Result<()> {\n    let mut stats_collector = StatsCollector::default();\n    let mut estimators: Vec<(CodecType, Box<dyn ColumnCodecEstimator>)> =\n        Vec::with_capacity(codec_types.len());\n    for &codec_type in codec_types {\n        estimators.push((codec_type, codec_type.estimator()));\n    }\n    for val in vals.boxed_iter() {\n        let val_u64 = val.to_u64();\n        stats_collector.collect(val_u64);\n        for (_, estimator) in &mut estimators {\n            estimator.collect(val_u64);\n        }\n    }\n    for (_, estimator) in &mut estimators {\n        estimator.finalize();\n    }\n    let stats = stats_collector.stats();\n    let (_, best_codec, best_codec_estimator) = estimators\n        .into_iter()\n        .flat_map(|(codec_type, estimator)| {\n            let num_bytes = estimator.estimate(&stats)?;\n            Some((num_bytes, codec_type, estimator))\n        })\n        .min_by_key(|(num_bytes, _, _)| *num_bytes)\n        .ok_or_else(|| {\n            io::Error::new(io::ErrorKind::InvalidData, \"No available applicable codec.\")\n        })?;\n    best_codec.to_code().serialize(wrt)?;\n    best_codec_estimator.serialize(\n        &stats,\n        &mut vals.boxed_iter().map(MonotonicallyMappableToU64::to_u64),\n        wrt,\n    )?;\n    Ok(())\n}\n\n/// Load u64-based column values.\n///\n/// This method first identifies the codec off the first byte.\npub fn load_u64_based_column_values<T: MonotonicallyMappableToU64>(\n    mut bytes: OwnedBytes,\n) -> io::Result<Arc<dyn ColumnValues<T>>> {\n    let codec_type: CodecType = bytes\n        .first()\n        .copied()\n        .and_then(CodecType::try_from_code)\n        .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidData, \"Failed to read codec type\"))?;\n    bytes.advance(1);\n    codec_type.load(bytes)\n}\n\n/// Helper function to serialize a column (autodetect from all codecs) and then open it\npub fn serialize_and_load_u64_based_column_values<T: MonotonicallyMappableToU64>(\n    vals: &dyn Iterable,\n    codec_types: &[CodecType],\n) -> Arc<dyn ColumnValues<T>> {\n    let mut buffer = Vec::new();\n    serialize_u64_based_column_values(vals, codec_types, &mut buffer).unwrap();\n    load_u64_based_column_values::<T>(OwnedBytes::new(buffer)).unwrap()\n}\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/stats_collector.rs",
    "content": "use std::num::NonZeroU64;\n\nuse fastdivide::DividerU64;\n\nuse crate::RowId;\nuse crate::column_values::ColumnStats;\n\n/// Compute the gcd of two non null numbers.\n///\n/// It is recommended, but not required, to feed values such that `large >= small`.\nfn compute_gcd(mut large: NonZeroU64, mut small: NonZeroU64) -> NonZeroU64 {\n    loop {\n        let rem: u64 = large.get() % small;\n        if let Some(new_small) = NonZeroU64::new(rem) {\n            (large, small) = (small, new_small);\n        } else {\n            return small;\n        }\n    }\n}\n\n#[derive(Default)]\npub struct StatsCollector {\n    min_max_opt: Option<(u64, u64)>,\n    num_rows: RowId,\n    // We measure the GCD of the difference between the values and the minimal value.\n    // This is the same as computing the difference between the values and the first value.\n    //\n    // This way, we can compress i64-converted-to-u64 (e.g. timestamp that were supplied in\n    // seconds, only to be converted in nanoseconds).\n    increment_gcd_opt: Option<(NonZeroU64, DividerU64)>,\n    first_value_opt: Option<u64>,\n}\n\nimpl StatsCollector {\n    pub fn stats(&self) -> ColumnStats {\n        let (min_value, max_value) = self.min_max_opt.unwrap_or((0u64, 0u64));\n        let increment_gcd = if let Some((increment_gcd, _)) = self.increment_gcd_opt {\n            increment_gcd\n        } else {\n            NonZeroU64::new(1u64).unwrap()\n        };\n        ColumnStats {\n            min_value,\n            max_value,\n            num_rows: self.num_rows,\n            gcd: increment_gcd,\n        }\n    }\n\n    #[inline]\n    fn update_increment_gcd(&mut self, value: u64) {\n        let Some(first_value) = self.first_value_opt else {\n            // We set the first value and just quit.\n            self.first_value_opt = Some(value);\n            return;\n        };\n        let Some(non_zero_value) = NonZeroU64::new(value.abs_diff(first_value)) else {\n            // We can simply skip 0 values.\n            return;\n        };\n        let Some((gcd, gcd_divider)) = self.increment_gcd_opt else {\n            self.set_increment_gcd(non_zero_value);\n            return;\n        };\n        if gcd.get() == 1 {\n            // It won't see any update now.\n            return;\n        }\n        let remainder =\n            non_zero_value.get() - (gcd_divider.divide(non_zero_value.get())) * gcd.get();\n        if remainder == 0 {\n            return;\n        }\n        let new_gcd = compute_gcd(non_zero_value, gcd);\n        self.set_increment_gcd(new_gcd);\n    }\n\n    fn set_increment_gcd(&mut self, gcd: NonZeroU64) {\n        let new_divider = DividerU64::divide_by(gcd.get());\n        self.increment_gcd_opt = Some((gcd, new_divider));\n    }\n\n    pub fn collect(&mut self, value: u64) {\n        self.min_max_opt = Some(if let Some((min, max)) = self.min_max_opt {\n            (min.min(value), max.max(value))\n        } else {\n            (value, value)\n        });\n        self.num_rows += 1;\n        self.update_increment_gcd(value);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::num::NonZeroU64;\n\n    use crate::column_values::u64_based::ColumnStats;\n    use crate::column_values::u64_based::stats_collector::{StatsCollector, compute_gcd};\n\n    fn compute_stats(vals: impl Iterator<Item = u64>) -> ColumnStats {\n        let mut stats_collector = StatsCollector::default();\n        for val in vals {\n            stats_collector.collect(val);\n        }\n        stats_collector.stats()\n    }\n\n    fn find_gcd(vals: impl Iterator<Item = u64>) -> u64 {\n        compute_stats(vals).gcd.get()\n    }\n\n    #[test]\n    fn test_compute_gcd() {\n        let test_compute_gcd_aux = |large, small, expected| {\n            let large = NonZeroU64::new(large).unwrap();\n            let small = NonZeroU64::new(small).unwrap();\n            let expected = NonZeroU64::new(expected).unwrap();\n            assert_eq!(compute_gcd(small, large), expected);\n            assert_eq!(compute_gcd(large, small), expected);\n        };\n        test_compute_gcd_aux(1, 4, 1);\n        test_compute_gcd_aux(2, 4, 2);\n        test_compute_gcd_aux(10, 25, 5);\n        test_compute_gcd_aux(25, 25, 25);\n    }\n\n    #[test]\n    fn test_gcd() {\n        assert_eq!(find_gcd([0].into_iter()), 1);\n        assert_eq!(find_gcd([0, 10].into_iter()), 10);\n        assert_eq!(find_gcd([10, 0].into_iter()), 10);\n        assert_eq!(find_gcd([].into_iter()), 1);\n        assert_eq!(find_gcd([15, 30, 5, 10].into_iter()), 5);\n        assert_eq!(find_gcd([15, 16, 10].into_iter()), 1);\n        assert_eq!(find_gcd([0, 5, 5, 5].into_iter()), 5);\n        assert_eq!(find_gcd([0, 0].into_iter()), 1);\n        assert_eq!(find_gcd([1, 10, 4, 1, 7, 10].into_iter()), 3);\n        assert_eq!(find_gcd([1, 10, 0, 4, 1, 7, 10].into_iter()), 1);\n    }\n\n    #[test]\n    fn test_stats() {\n        assert_eq!(\n            compute_stats([].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(1).unwrap(),\n                min_value: 0,\n                max_value: 0,\n                num_rows: 0\n            }\n        );\n        assert_eq!(\n            compute_stats([0, 1].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(1).unwrap(),\n                min_value: 0,\n                max_value: 1,\n                num_rows: 2\n            }\n        );\n        assert_eq!(\n            compute_stats([0, 1].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(1).unwrap(),\n                min_value: 0,\n                max_value: 1,\n                num_rows: 2\n            }\n        );\n        assert_eq!(\n            compute_stats([10, 20, 30].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(10).unwrap(),\n                min_value: 10,\n                max_value: 30,\n                num_rows: 3\n            }\n        );\n        assert_eq!(\n            compute_stats([10, 50, 10, 30].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(20).unwrap(),\n                min_value: 10,\n                max_value: 50,\n                num_rows: 4\n            }\n        );\n        assert_eq!(\n            compute_stats([10, 0, 30].into_iter()),\n            ColumnStats {\n                gcd: NonZeroU64::new(10).unwrap(),\n                min_value: 0,\n                max_value: 30,\n                num_rows: 3\n            }\n        );\n    }\n}\n"
  },
  {
    "path": "columnar/src/column_values/u64_based/tests.rs",
    "content": "use proptest::prelude::*;\nuse proptest::{prop_oneof, proptest};\nuse rand::Rng;\n\n#[test]\nfn test_serialize_and_load_simple() {\n    let mut buffer = Vec::new();\n    let vals = &[1u64, 2u64, 5u64];\n    serialize_u64_based_column_values(\n        &&vals[..],\n        &[CodecType::Bitpacked, CodecType::BlockwiseLinear],\n        &mut buffer,\n    )\n    .unwrap();\n    assert_eq!(buffer.len(), 7);\n    let col = load_u64_based_column_values::<u64>(OwnedBytes::new(buffer)).unwrap();\n    assert_eq!(col.num_vals(), 3);\n    assert_eq!(col.get_val(0), 1);\n    assert_eq!(col.get_val(1), 2);\n    assert_eq!(col.get_val(2), 5);\n}\n\n#[test]\nfn test_empty_column_i64() {\n    let vals: [i64; 0] = [];\n    let mut num_acceptable_codecs = 0;\n    for codec in ALL_U64_CODEC_TYPES {\n        let mut buffer = Vec::new();\n        if serialize_u64_based_column_values(&&vals[..], &[codec], &mut buffer).is_err() {\n            continue;\n        }\n        num_acceptable_codecs += 1;\n        let col = load_u64_based_column_values::<i64>(OwnedBytes::new(buffer)).unwrap();\n        assert_eq!(col.num_vals(), 0);\n        assert_eq!(col.min_value(), i64::MIN);\n        assert_eq!(col.max_value(), i64::MIN);\n    }\n    assert!(num_acceptable_codecs > 0);\n}\n\n#[test]\nfn test_empty_column_u64() {\n    let vals: [u64; 0] = [];\n    let mut num_acceptable_codecs = 0;\n    for codec in ALL_U64_CODEC_TYPES {\n        let mut buffer = Vec::new();\n        if serialize_u64_based_column_values(&&vals[..], &[codec], &mut buffer).is_err() {\n            continue;\n        }\n        num_acceptable_codecs += 1;\n        let col = load_u64_based_column_values::<u64>(OwnedBytes::new(buffer)).unwrap();\n        assert_eq!(col.num_vals(), 0);\n        assert_eq!(col.min_value(), u64::MIN);\n        assert_eq!(col.max_value(), u64::MIN);\n    }\n    assert!(num_acceptable_codecs > 0);\n}\n\n#[test]\nfn test_empty_column_f64() {\n    let vals: [f64; 0] = [];\n    let mut num_acceptable_codecs = 0;\n    for codec in ALL_U64_CODEC_TYPES {\n        let mut buffer = Vec::new();\n        if serialize_u64_based_column_values(&&vals[..], &[codec], &mut buffer).is_err() {\n            continue;\n        }\n        num_acceptable_codecs += 1;\n        let col = load_u64_based_column_values::<f64>(OwnedBytes::new(buffer)).unwrap();\n        assert_eq!(col.num_vals(), 0);\n        // FIXME. f64::MIN would be better!\n        assert!(col.min_value().is_nan());\n        assert!(col.max_value().is_nan());\n    }\n    assert!(num_acceptable_codecs > 0);\n}\n\npub(crate) fn create_and_validate<TColumnCodec: ColumnCodec>(\n    vals: &[u64],\n    name: &str,\n) -> Option<(f32, f32)> {\n    let mut stats_collector = StatsCollector::default();\n    let mut codec_estimator: TColumnCodec::Estimator = Default::default();\n\n    for val in vals.boxed_iter() {\n        stats_collector.collect(val);\n        codec_estimator.collect(val);\n    }\n    codec_estimator.finalize();\n    let stats = stats_collector.stats();\n    let estimation = codec_estimator.estimate(&stats)?;\n\n    let mut buffer = Vec::new();\n    codec_estimator\n        .serialize(&stats, vals.boxed_iter().as_mut(), &mut buffer)\n        .unwrap();\n\n    let actual_compression = buffer.len() as u64;\n\n    let reader = TColumnCodec::load(OwnedBytes::new(buffer)).unwrap();\n    assert_eq!(reader.num_vals(), vals.len() as u32);\n    let mut buffer = Vec::new();\n    for (doc, orig_val) in vals.iter().copied().enumerate() {\n        let val = reader.get_val(doc as u32);\n        assert_eq!(\n            val, orig_val,\n            \"val `{val}` does not match orig_val {orig_val:?}, in data set {name}, data `{vals:?}`\",\n        );\n\n        buffer.resize(1, 0);\n        reader.get_vals(&[doc as u32], &mut buffer);\n        let val = buffer[0];\n        assert_eq!(\n            val, orig_val,\n            \"val `{val}` does not match orig_val {orig_val:?}, in data set {name}, data `{vals:?}`\",\n        );\n    }\n\n    let all_docs: Vec<u32> = (0..vals.len() as u32).collect();\n    buffer.resize(all_docs.len(), 0);\n    reader.get_vals(&all_docs, &mut buffer);\n    assert_eq!(vals, buffer);\n\n    if !vals.is_empty() {\n        let test_rand_idx = rand::rng().random_range(0..=vals.len() - 1);\n        let expected_positions: Vec<u32> = vals\n            .iter()\n            .enumerate()\n            .filter(|(_, el)| **el == vals[test_rand_idx])\n            .map(|(pos, _)| pos as u32)\n            .collect();\n        let mut positions = Vec::new();\n        reader.get_row_ids_for_value_range(\n            vals[test_rand_idx]..=vals[test_rand_idx],\n            0..vals.len() as u32,\n            &mut positions,\n        );\n        assert_eq!(expected_positions, positions);\n    }\n    if actual_compression > 1000 {\n        assert!(relative_difference(estimation, actual_compression) < 0.10f32);\n    }\n    Some((\n        compression_rate(estimation, stats.num_rows),\n        compression_rate(actual_compression, stats.num_rows),\n    ))\n}\n\nfn compression_rate(num_bytes: u64, num_values: u32) -> f32 {\n    num_bytes as f32 / (num_values as f32 * 8.0)\n}\n\nfn relative_difference(left: u64, right: u64) -> f32 {\n    let left = left as f32;\n    let right = right as f32;\n    2.0f32 * (left - right).abs() / (left + right)\n}\n\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(100))]\n\n    #[test]\n    fn test_proptest_small_bitpacked(data in proptest::collection::vec(num_strategy(), 1..10)) {\n        create_and_validate::<BitpackedCodec>(&data, \"proptest bitpacked\");\n    }\n\n    #[test]\n    fn test_proptest_small_linear(data in proptest::collection::vec(num_strategy(), 1..10)) {\n        create_and_validate::<LinearCodec>(&data, \"proptest linearinterpol\");\n    }\n\n\n    #[test]\n    fn test_proptest_small_blockwise_linear(data in proptest::collection::vec(num_strategy(), 1..10)) {\n        create_and_validate::<BlockwiseLinearCodec>(&data, \"proptest multilinearinterpol\");\n    }\n}\n\n#[test]\nfn test_small_blockwise_linear_example() {\n    create_and_validate::<BlockwiseLinearCodec>(\n        &[9223372036854775808, 9223370937344622593],\n        \"proptest multilinearinterpol\",\n    );\n}\n\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(10))]\n\n    #[test]\n    fn test_proptest_large_bitpacked(data in proptest::collection::vec(num_strategy(), 1..6000)) {\n        create_and_validate::<BitpackedCodec>(&data, \"proptest bitpacked\");\n    }\n\n    #[test]\n    fn test_proptest_large_linear(data in proptest::collection::vec(num_strategy(), 1..6000)) {\n        create_and_validate::<LinearCodec>(&data, \"proptest linearinterpol\");\n    }\n\n    #[test]\n    fn test_proptest_large_blockwise_linear(data in proptest::collection::vec(num_strategy(), 1..6000)) {\n        create_and_validate::<BlockwiseLinearCodec>(&data, \"proptest multilinearinterpol\");\n    }\n}\n\nfn num_strategy() -> impl Strategy<Value = u64> {\n    prop_oneof![\n        1 => prop::num::u64::ANY.prop_map(|num| u64::MAX - (num % 10) ),\n        1 => prop::num::u64::ANY.prop_map(|num| num % 10 ),\n        20 => prop::num::u64::ANY,\n    ]\n}\n\npub fn get_codec_test_datasets() -> Vec<(Vec<u64>, &'static str)> {\n    let mut data_and_names = vec![];\n\n    let data = (10..=10_000_u64).collect::<Vec<_>>();\n    data_and_names.push((data, \"simple monotonically increasing\"));\n\n    data_and_names.push((\n        vec![5, 6, 7, 8, 9, 10, 99, 100],\n        \"offset in linear interpol\",\n    ));\n    data_and_names.push((vec![5, 50, 3, 13, 1, 1000, 35], \"rand small\"));\n    data_and_names.push((vec![10], \"single value\"));\n\n    data_and_names.push((\n        vec![1572656989877777, 1170935903116329, 720575940379279, 0],\n        \"overflow error\",\n    ));\n\n    data_and_names\n}\n\nfn test_codec<C: ColumnCodec>() {\n    let codec_name = std::any::type_name::<C>();\n    for (data, dataset_name) in get_codec_test_datasets() {\n        let estimate_actual_opt: Option<(f32, f32)> =\n            tests::create_and_validate::<C>(&data, dataset_name);\n        let result = if let Some((estimate, actual)) = estimate_actual_opt {\n            format!(\"Estimate `{estimate}` Actual `{actual}`\")\n        } else {\n            \"Disabled\".to_string()\n        };\n        println!(\"Codec {codec_name}, DataSet {dataset_name}, {result}\");\n    }\n}\n#[test]\nfn test_codec_bitpacking() {\n    test_codec::<BitpackedCodec>();\n}\n#[test]\nfn test_codec_interpolation() {\n    test_codec::<LinearCodec>();\n}\n#[test]\nfn test_codec_multi_interpolation() {\n    test_codec::<BlockwiseLinearCodec>();\n}\n\nuse super::*;\n\nfn estimate<C: ColumnCodec>(vals: &[u64]) -> Option<f32> {\n    let mut stats_collector = StatsCollector::default();\n    let mut estimator = C::Estimator::default();\n    for &val in vals {\n        stats_collector.collect(val);\n        estimator.collect(val);\n    }\n    estimator.finalize();\n    let stats = stats_collector.stats();\n    let num_bytes = estimator.estimate(&stats)?;\n    if stats.num_rows == 0 {\n        return None;\n    }\n    Some(num_bytes as f32 / (8.0 * stats.num_rows as f32))\n}\n\n#[test]\nfn estimation_good_interpolation_case() {\n    let data = (10..=20000_u64).collect::<Vec<_>>();\n\n    let linear_interpol_estimation = estimate::<LinearCodec>(&data).unwrap();\n    assert_le!(linear_interpol_estimation, 0.01);\n\n    let multi_linear_interpol_estimation = estimate::<BlockwiseLinearCodec>(&data).unwrap();\n    assert_le!(multi_linear_interpol_estimation, 0.2);\n    assert_lt!(linear_interpol_estimation, multi_linear_interpol_estimation);\n\n    let bitpacked_estimation = estimate::<BitpackedCodec>(&data).unwrap();\n    assert_lt!(linear_interpol_estimation, bitpacked_estimation);\n}\n\n#[test]\nfn estimation_test_bad_interpolation_case_monotonically_increasing() {\n    let mut data: Vec<u64> = (201..=20000_u64).collect();\n    data.push(1_000_000);\n\n    // in this case the linear interpolation can't in fact not be worse than bitpacking,\n    // but the estimator adds some threshold, which leads to estimated worse behavior\n    let linear_interpol_estimation = estimate::<LinearCodec>(&data[..]).unwrap();\n    assert_le!(linear_interpol_estimation, 0.35);\n\n    let bitpacked_estimation = estimate::<BitpackedCodec>(&data).unwrap();\n    assert_le!(bitpacked_estimation, 0.32);\n    assert_le!(bitpacked_estimation, linear_interpol_estimation);\n}\n\n#[test]\nfn test_fast_field_codec_type_to_code() {\n    let mut count_codec = 0;\n    for code in 0..=255 {\n        if let Some(codec_type) = CodecType::try_from_code(code) {\n            assert_eq!(codec_type.to_code(), code);\n            count_codec += 1;\n        }\n    }\n    assert_eq!(count_codec, 3);\n}\n\nfn test_fastfield_gcd_i64_with_codec(codec_type: CodecType, num_vals: usize) -> io::Result<()> {\n    let mut vals: Vec<i64> = (-4..=(num_vals as i64) - 5).map(|val| val * 1000).collect();\n    let mut buffer: Vec<u8> = Vec::new();\n    crate::column_values::serialize_u64_based_column_values(\n        &&vals[..],\n        &[codec_type],\n        &mut buffer,\n    )?;\n    let buffer = OwnedBytes::new(buffer);\n    let column = crate::column_values::load_u64_based_column_values::<i64>(buffer.clone())?;\n    assert_eq!(column.get_val(0), -4000i64);\n    assert_eq!(column.get_val(1), -3000i64);\n    assert_eq!(column.get_val(2), -2000i64);\n    assert_eq!(column.max_value(), (num_vals as i64 - 5) * 1000);\n    assert_eq!(column.min_value(), -4000i64);\n\n    // Can't apply gcd\n    let mut buffer_without_gcd = Vec::new();\n    vals.pop();\n    vals.push(1001i64);\n    crate::column_values::serialize_u64_based_column_values(\n        &&vals[..],\n        &[codec_type],\n        &mut buffer_without_gcd,\n    )?;\n    let buffer_without_gcd = OwnedBytes::new(buffer_without_gcd);\n    assert!(buffer_without_gcd.len() > buffer.len());\n\n    Ok(())\n}\n\n#[test]\nfn test_fastfield_gcd_i64() -> io::Result<()> {\n    for &codec_type in &[\n        CodecType::Bitpacked,\n        CodecType::BlockwiseLinear,\n        CodecType::Linear,\n    ] {\n        test_fastfield_gcd_i64_with_codec(codec_type, 5500)?;\n    }\n    Ok(())\n}\n\nfn test_fastfield_gcd_u64_with_codec(codec_type: CodecType, num_vals: usize) -> io::Result<()> {\n    let mut vals: Vec<u64> = (1..=num_vals).map(|i| i as u64 * 1000u64).collect();\n    let mut buffer: Vec<u8> = Vec::new();\n    crate::column_values::serialize_u64_based_column_values(\n        &&vals[..],\n        &[codec_type],\n        &mut buffer,\n    )?;\n    let buffer = OwnedBytes::new(buffer);\n    let column = crate::column_values::load_u64_based_column_values::<u64>(buffer.clone())?;\n    assert_eq!(column.get_val(0), 1000u64);\n    assert_eq!(column.get_val(1), 2000u64);\n    assert_eq!(column.get_val(2), 3000u64);\n    assert_eq!(column.max_value(), num_vals as u64 * 1000);\n    assert_eq!(column.min_value(), 1000u64);\n\n    // Can't apply gcd\n    let mut buffer_without_gcd = Vec::new();\n    vals.pop();\n    vals.push(1001u64);\n    crate::column_values::serialize_u64_based_column_values(\n        &&vals[..],\n        &[codec_type],\n        &mut buffer_without_gcd,\n    )?;\n    let buffer_without_gcd = OwnedBytes::new(buffer_without_gcd);\n    assert!(buffer_without_gcd.len() > buffer.len());\n    Ok(())\n}\n\n#[test]\nfn test_fastfield_gcd_u64() -> io::Result<()> {\n    for &codec_type in &[\n        CodecType::Bitpacked,\n        CodecType::BlockwiseLinear,\n        CodecType::Linear,\n    ] {\n        test_fastfield_gcd_u64_with_codec(codec_type, 5500)?;\n    }\n    Ok(())\n}\n\n#[test]\npub fn test_fastfield2() {\n    let test_fastfield = crate::column_values::serialize_and_load_u64_based_column_values::<u64>(\n        &&[100u64, 200u64, 300u64][..],\n        &ALL_U64_CODEC_TYPES,\n    );\n    assert_eq!(test_fastfield.get_val(0), 100);\n    assert_eq!(test_fastfield.get_val(1), 200);\n    assert_eq!(test_fastfield.get_val(2), 300);\n}\n"
  },
  {
    "path": "columnar/src/column_values/vec_column.rs",
    "content": "use std::fmt::Debug;\n\nuse tantivy_bitpacker::minmax;\n\nuse crate::ColumnValues;\n\n/// VecColumn provides `Column` over a `Vec<T>`.\npub struct VecColumn<T = u64> {\n    pub(crate) values: Vec<T>,\n    pub(crate) min_value: T,\n    pub(crate) max_value: T,\n}\n\nimpl<T: Copy + PartialOrd + Send + Sync + Debug + 'static> ColumnValues<T> for VecColumn<T> {\n    fn get_val(&self, position: u32) -> T {\n        self.values[position as usize]\n    }\n\n    fn iter(&self) -> Box<dyn Iterator<Item = T> + '_> {\n        Box::new(self.values.iter().copied())\n    }\n\n    fn min_value(&self) -> T {\n        self.min_value\n    }\n\n    fn max_value(&self) -> T {\n        self.max_value\n    }\n\n    fn num_vals(&self) -> u32 {\n        self.values.len() as u32\n    }\n\n    fn get_range(&self, start: u64, output: &mut [T]) {\n        output.copy_from_slice(&self.values[start as usize..][..output.len()])\n    }\n}\n\nimpl<T: Copy + PartialOrd + Default> From<Vec<T>> for VecColumn<T> {\n    fn from(values: Vec<T>) -> Self {\n        let (min_value, max_value) = minmax(values.iter().copied()).unwrap_or_default();\n        Self {\n            values,\n            min_value,\n            max_value,\n        }\n    }\n}\nimpl From<VecColumn> for Vec<u64> {\n    fn from(column: VecColumn) -> Self {\n        column.values\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/column_type.rs",
    "content": "use std::fmt;\nuse std::fmt::Debug;\nuse std::net::Ipv6Addr;\n\nuse serde::{Deserialize, Serialize};\n\nuse crate::InvalidData;\nuse crate::value::NumericalType;\n\n/// The column type represents the column type.\n/// Any changes need to be propagated to `COLUMN_TYPES`.\n#[derive(Hash, Eq, PartialEq, Debug, Clone, Copy, Ord, PartialOrd, Serialize, Deserialize)]\n#[repr(u8)]\npub enum ColumnType {\n    I64 = 0u8,\n    U64 = 1u8,\n    F64 = 2u8,\n    Bytes = 3u8,\n    Str = 4u8,\n    Bool = 5u8,\n    IpAddr = 6u8,\n    DateTime = 7u8,\n}\n\nimpl fmt::Display for ColumnType {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        let short_str = match self {\n            ColumnType::I64 => \"i64\",\n            ColumnType::U64 => \"u64\",\n            ColumnType::F64 => \"f64\",\n            ColumnType::Bytes => \"bytes\",\n            ColumnType::Str => \"str\",\n            ColumnType::Bool => \"bool\",\n            ColumnType::IpAddr => \"ip\",\n            ColumnType::DateTime => \"datetime\",\n        };\n        write!(f, \"{short_str}\")\n    }\n}\n\n// The order needs to match _exactly_ the order in the enum\nconst COLUMN_TYPES: [ColumnType; 8] = [\n    ColumnType::I64,\n    ColumnType::U64,\n    ColumnType::F64,\n    ColumnType::Bytes,\n    ColumnType::Str,\n    ColumnType::Bool,\n    ColumnType::IpAddr,\n    ColumnType::DateTime,\n];\n\nimpl ColumnType {\n    pub fn to_code(self) -> u8 {\n        self as u8\n    }\n    pub fn is_date_time(&self) -> bool {\n        self == &ColumnType::DateTime\n    }\n\n    pub(crate) fn try_from_code(code: u8) -> Result<ColumnType, InvalidData> {\n        COLUMN_TYPES.get(code as usize).copied().ok_or(InvalidData)\n    }\n}\n\nimpl From<NumericalType> for ColumnType {\n    fn from(numerical_type: NumericalType) -> Self {\n        match numerical_type {\n            NumericalType::I64 => ColumnType::I64,\n            NumericalType::U64 => ColumnType::U64,\n            NumericalType::F64 => ColumnType::F64,\n        }\n    }\n}\n\nimpl ColumnType {\n    pub fn numerical_type(&self) -> Option<NumericalType> {\n        match self {\n            ColumnType::I64 => Some(NumericalType::I64),\n            ColumnType::U64 => Some(NumericalType::U64),\n            ColumnType::F64 => Some(NumericalType::F64),\n            ColumnType::Bytes\n            | ColumnType::Str\n            | ColumnType::Bool\n            | ColumnType::IpAddr\n            | ColumnType::DateTime => None,\n        }\n    }\n}\n\n// TODO remove if possible\npub trait HasAssociatedColumnType: 'static + Debug + Send + Sync + Copy + PartialOrd {\n    fn column_type() -> ColumnType;\n    fn default_value() -> Self;\n}\n\nimpl HasAssociatedColumnType for u64 {\n    fn column_type() -> ColumnType {\n        ColumnType::U64\n    }\n\n    fn default_value() -> Self {\n        0u64\n    }\n}\n\nimpl HasAssociatedColumnType for i64 {\n    fn column_type() -> ColumnType {\n        ColumnType::I64\n    }\n\n    fn default_value() -> Self {\n        0i64\n    }\n}\n\nimpl HasAssociatedColumnType for f64 {\n    fn column_type() -> ColumnType {\n        ColumnType::F64\n    }\n\n    fn default_value() -> Self {\n        Default::default()\n    }\n}\n\nimpl HasAssociatedColumnType for bool {\n    fn column_type() -> ColumnType {\n        ColumnType::Bool\n    }\n    fn default_value() -> Self {\n        Default::default()\n    }\n}\n\nimpl HasAssociatedColumnType for common::DateTime {\n    fn column_type() -> ColumnType {\n        ColumnType::DateTime\n    }\n    fn default_value() -> Self {\n        Default::default()\n    }\n}\n\nimpl HasAssociatedColumnType for Ipv6Addr {\n    fn column_type() -> ColumnType {\n        ColumnType::IpAddr\n    }\n\n    fn default_value() -> Self {\n        Ipv6Addr::from([0u8; 16])\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::Cardinality;\n\n    #[test]\n    fn test_column_type_to_code() {\n        for (code, expected_column_type) in super::COLUMN_TYPES.iter().copied().enumerate() {\n            if let Ok(column_type) = ColumnType::try_from_code(code as u8) {\n                assert_eq!(column_type, expected_column_type);\n            }\n        }\n        for code in COLUMN_TYPES.len() as u8..=u8::MAX {\n            assert!(ColumnType::try_from_code(code).is_err());\n        }\n    }\n\n    #[test]\n    fn test_cardinality_to_code() {\n        let mut num_cardinality = 0;\n        for code in u8::MIN..=u8::MAX {\n            if let Ok(cardinality) = Cardinality::try_from_code(code) {\n                assert_eq!(cardinality.to_code(), code);\n                num_cardinality += 1;\n            }\n        }\n        assert_eq!(num_cardinality, 3);\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/format_version.rs",
    "content": "use core::fmt;\nuse std::fmt::{Display, Formatter};\n\nuse crate::InvalidData;\n\npub const VERSION_FOOTER_NUM_BYTES: usize = MAGIC_BYTES.len() + std::mem::size_of::<u32>();\n\n/// We end the file by these 4 bytes just to somewhat identify that\n/// this is indeed a columnar file.\nconst MAGIC_BYTES: [u8; 4] = [2, 113, 119, 66];\n\npub fn footer() -> [u8; VERSION_FOOTER_NUM_BYTES] {\n    let mut footer_bytes = [0u8; VERSION_FOOTER_NUM_BYTES];\n    footer_bytes[0..4].copy_from_slice(&CURRENT_VERSION.to_bytes());\n    footer_bytes[4..8].copy_from_slice(&MAGIC_BYTES[..]);\n    footer_bytes\n}\n\npub fn parse_footer(footer_bytes: [u8; VERSION_FOOTER_NUM_BYTES]) -> Result<Version, InvalidData> {\n    if footer_bytes[4..8] != MAGIC_BYTES {\n        return Err(InvalidData);\n    }\n    Version::try_from_bytes(footer_bytes[0..4].try_into().unwrap())\n}\n\npub const CURRENT_VERSION: Version = Version::V2;\n\n#[derive(Debug, Copy, Clone, Eq, PartialEq)]\n#[repr(u32)]\npub enum Version {\n    V1 = 1u32,\n    V2 = 2u32,\n}\n\nimpl Display for Version {\n    fn fmt(&self, f: &mut Formatter) -> fmt::Result {\n        match self {\n            Version::V1 => write!(f, \"v1\"),\n            Version::V2 => write!(f, \"v2\"),\n        }\n    }\n}\n\nimpl Version {\n    fn to_bytes(self) -> [u8; 4] {\n        (self as u32).to_le_bytes()\n    }\n\n    fn try_from_bytes(bytes: [u8; 4]) -> Result<Version, InvalidData> {\n        let code = u32::from_le_bytes(bytes);\n        match code {\n            1u32 => Ok(Version::V1),\n            2u32 => Ok(Version::V2),\n            _ => Err(InvalidData),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n\n    use super::*;\n\n    #[test]\n    fn test_footer_deserialization() {\n        let parsed_version: Version = parse_footer(footer()).unwrap();\n        assert_eq!(Version::V2, parsed_version);\n    }\n\n    #[test]\n    fn test_version_serialization() {\n        let version_to_tests: Vec<u32> = [0, 1 << 8, 1 << 16, 1 << 24]\n            .iter()\n            .copied()\n            .flat_map(|offset| (0..255).map(move |el| el + offset))\n            .collect();\n        let mut valid_versions: HashSet<u32> = HashSet::default();\n        for &i in &version_to_tests {\n            let version_res = Version::try_from_bytes(i.to_le_bytes());\n            if let Ok(version) = version_res {\n                assert_eq!(version.to_bytes(), i.to_le_bytes());\n                valid_versions.insert(i);\n            }\n        }\n        assert_eq!(valid_versions.len(), 2);\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/merge/merge_dict_column.rs",
    "content": "use std::io::{self, Write};\n\nuse common::{BitSet, CountingWriter, ReadOnlyBitSet};\nuse sstable::{SSTable, Streamer, TermOrdinal, VoidSSTable};\n\nuse super::term_merger::{TermMerger, TermsWithSegmentOrd};\nuse crate::column::serialize_column_mappable_to_u64;\nuse crate::column_index::SerializableColumnIndex;\nuse crate::iterable::Iterable;\nuse crate::{BytesColumn, MergeRowOrder, ShuffleMergeOrder};\n\n// Serialize [Dictionary, Column, dictionary num bytes U32::LE]\n// Column: [Column Index, Column Values, column index num bytes U32::LE]\npub fn merge_bytes_or_str_column(\n    column_index: SerializableColumnIndex<'_>,\n    bytes_columns: &[Option<BytesColumn>],\n    merge_row_order: &MergeRowOrder,\n    output: &mut impl Write,\n) -> io::Result<()> {\n    // Serialize dict and generate mapping for values\n    let mut output = CountingWriter::wrap(output);\n    // TODO !!! Remove useless terms.\n    let term_ord_mapping = serialize_merged_dict(bytes_columns, merge_row_order, &mut output)?;\n    let dictionary_num_bytes: u32 = output.written_bytes() as u32;\n    let output = output.finish();\n    let remapped_term_ordinals_values = RemappedTermOrdinalsValues {\n        bytes_columns,\n        term_ord_mapping: &term_ord_mapping,\n        merge_row_order,\n    };\n    serialize_column_mappable_to_u64(column_index, &remapped_term_ordinals_values, output)?;\n    output.write_all(&dictionary_num_bytes.to_le_bytes())?;\n    Ok(())\n}\n\nstruct RemappedTermOrdinalsValues<'a> {\n    bytes_columns: &'a [Option<BytesColumn>],\n    term_ord_mapping: &'a TermOrdinalMapping,\n    merge_row_order: &'a MergeRowOrder,\n}\n\nimpl Iterable for RemappedTermOrdinalsValues<'_> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u64> + '_> {\n        match self.merge_row_order {\n            MergeRowOrder::Stack(_) => self.boxed_iter_stacked(),\n            MergeRowOrder::Shuffled(shuffle_merge_order) => {\n                self.boxed_iter_shuffled(shuffle_merge_order)\n            }\n        }\n    }\n}\n\nimpl RemappedTermOrdinalsValues<'_> {\n    fn boxed_iter_stacked(&self) -> Box<dyn Iterator<Item = u64> + '_> {\n        let iter = self\n            .bytes_columns\n            .iter()\n            .enumerate()\n            .flat_map(|(seg_ord, bytes_column_opt)| {\n                let bytes_column = bytes_column_opt.as_ref()?;\n                Some((seg_ord, bytes_column))\n            })\n            .flat_map(move |(seg_ord, bytes_column)| {\n                let term_ord_after_merge_mapping =\n                    self.term_ord_mapping.get_segment(seg_ord as u32);\n                bytes_column\n                    .ords()\n                    .values\n                    .iter()\n                    .map(move |term_ord| term_ord_after_merge_mapping[term_ord as usize])\n            });\n        Box::new(iter)\n    }\n\n    fn boxed_iter_shuffled<'b>(\n        &'b self,\n        shuffle_merge_order: &'b ShuffleMergeOrder,\n    ) -> Box<dyn Iterator<Item = u64> + 'b> {\n        Box::new(\n            shuffle_merge_order\n                .iter_new_to_old_row_addrs()\n                .flat_map(move |old_addr| {\n                    let segment_ord = self.term_ord_mapping.get_segment(old_addr.segment_ord);\n                    self.bytes_columns[old_addr.segment_ord as usize]\n                        .as_ref()\n                        .into_iter()\n                        .flat_map(move |bytes_column| {\n                            bytes_column\n                                .term_ords(old_addr.row_id)\n                                .map(|old_term_ord: u64| segment_ord[old_term_ord as usize])\n                        })\n                }),\n        )\n    }\n}\n\nfn compute_term_bitset(column: &BytesColumn, row_bitset: &ReadOnlyBitSet) -> BitSet {\n    let num_terms = column.dictionary().num_terms();\n    let mut term_bitset = BitSet::with_max_value(num_terms as u32);\n    for row_id in row_bitset.iter() {\n        for term_ord in column.term_ord_column.values_for_doc(row_id) {\n            term_bitset.insert(term_ord as u32);\n        }\n    }\n    term_bitset\n}\n\nfn is_term_present(bitsets: &[Option<BitSet>], term_merger: &TermMerger) -> bool {\n    for (segment_ord, from_term_ord) in term_merger.matching_segments() {\n        if let Some(bitset) = bitsets[segment_ord].as_ref() {\n            if bitset.contains(from_term_ord as u32) {\n                return true;\n            }\n        } else {\n            return true;\n        }\n    }\n    false\n}\n\nfn serialize_merged_dict(\n    bytes_columns: &[Option<BytesColumn>],\n    merge_row_order: &MergeRowOrder,\n    output: &mut impl Write,\n) -> io::Result<TermOrdinalMapping> {\n    let mut term_ord_mapping = TermOrdinalMapping::default();\n\n    let mut field_term_streams = Vec::new();\n    for (segment_ord, column_opt) in bytes_columns.iter().enumerate() {\n        if let Some(column) = column_opt {\n            term_ord_mapping.add_segment(column.dictionary.num_terms());\n            let terms: Streamer<VoidSSTable> = column.dictionary.stream()?;\n            field_term_streams.push(TermsWithSegmentOrd { terms, segment_ord });\n        } else {\n            term_ord_mapping.add_segment(0);\n            field_term_streams.push(TermsWithSegmentOrd {\n                terms: Streamer::empty(),\n                segment_ord,\n            });\n        }\n    }\n\n    let mut merged_terms = TermMerger::new(field_term_streams);\n    let mut sstable_builder = sstable::VoidSSTable::writer(output);\n\n    match merge_row_order {\n        MergeRowOrder::Stack(_) => {\n            let mut current_term_ord = 0;\n            while merged_terms.advance() {\n                let term_bytes: &[u8] = merged_terms.key();\n                sstable_builder.insert(term_bytes, &())?;\n                for (segment_ord, from_term_ord) in merged_terms.matching_segments() {\n                    term_ord_mapping.register_from_to(segment_ord, from_term_ord, current_term_ord);\n                }\n                current_term_ord += 1;\n            }\n            sstable_builder.finish()?;\n        }\n        MergeRowOrder::Shuffled(shuffle_merge_order) => {\n            assert_eq!(shuffle_merge_order.alive_bitsets.len(), bytes_columns.len());\n            let mut term_bitsets: Vec<Option<BitSet>> = Vec::with_capacity(bytes_columns.len());\n            for (alive_bitset_opt, bytes_column_opt) in shuffle_merge_order\n                .alive_bitsets\n                .iter()\n                .zip(bytes_columns.iter())\n            {\n                match (alive_bitset_opt, bytes_column_opt) {\n                    (Some(alive_bitset), Some(bytes_column)) => {\n                        let term_bitset = compute_term_bitset(bytes_column, alive_bitset);\n                        term_bitsets.push(Some(term_bitset));\n                    }\n                    _ => {\n                        term_bitsets.push(None);\n                    }\n                }\n            }\n            let mut current_term_ord = 0;\n            while merged_terms.advance() {\n                let term_bytes: &[u8] = merged_terms.key();\n                if !is_term_present(&term_bitsets[..], &merged_terms) {\n                    continue;\n                }\n                sstable_builder.insert(term_bytes, &())?;\n                for (segment_ord, from_term_ord) in merged_terms.matching_segments() {\n                    term_ord_mapping.register_from_to(segment_ord, from_term_ord, current_term_ord);\n                }\n                current_term_ord += 1;\n            }\n            sstable_builder.finish()?;\n        }\n    }\n    Ok(term_ord_mapping)\n}\n\n#[derive(Default, Debug)]\nstruct TermOrdinalMapping {\n    /// Contains the new term ordinals for each segment.\n    per_segment_new_term_ordinals: Vec<Vec<TermOrdinal>>,\n}\n\nimpl TermOrdinalMapping {\n    fn add_segment(&mut self, max_term_ord: usize) {\n        self.per_segment_new_term_ordinals\n            .push(vec![TermOrdinal::default(); max_term_ord]);\n    }\n\n    fn register_from_to(&mut self, segment_ord: usize, from_ord: TermOrdinal, to_ord: TermOrdinal) {\n        self.per_segment_new_term_ordinals[segment_ord][from_ord as usize] = to_ord;\n    }\n\n    fn get_segment(&self, segment_ord: u32) -> &[TermOrdinal] {\n        &self.per_segment_new_term_ordinals[segment_ord as usize]\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/merge/merge_mapping.rs",
    "content": "use std::ops::Range;\n\nuse common::{BitSet, OwnedBytes, ReadOnlyBitSet};\n\nuse crate::{ColumnarReader, RowAddr, RowId};\n\npub struct StackMergeOrder {\n    // This does not start at 0. The first row is the number of\n    // rows in the first columnar.\n    cumulated_row_ids: Vec<RowId>,\n}\n\nimpl StackMergeOrder {\n    #[cfg(test)]\n    pub fn stack_for_test(num_rows_per_columnar: &[u32]) -> StackMergeOrder {\n        let mut cumulated_row_ids: Vec<RowId> = Vec::with_capacity(num_rows_per_columnar.len());\n        let mut cumulated_row_id = 0;\n        for &num_rows in num_rows_per_columnar {\n            cumulated_row_id += num_rows;\n            cumulated_row_ids.push(cumulated_row_id);\n        }\n        StackMergeOrder { cumulated_row_ids }\n    }\n\n    pub fn stack(columnars: &[&ColumnarReader]) -> StackMergeOrder {\n        let mut cumulated_row_ids: Vec<RowId> = Vec::with_capacity(columnars.len());\n        let mut cumulated_row_id = 0;\n        for columnar in columnars {\n            cumulated_row_id += columnar.num_docs();\n            cumulated_row_ids.push(cumulated_row_id);\n        }\n        StackMergeOrder { cumulated_row_ids }\n    }\n\n    pub fn num_rows(&self) -> RowId {\n        self.cumulated_row_ids.last().copied().unwrap_or(0)\n    }\n\n    pub fn offset(&self, columnar_id: usize) -> RowId {\n        if columnar_id == 0 {\n            return 0;\n        }\n        self.cumulated_row_ids[columnar_id - 1]\n    }\n\n    pub fn columnar_range(&self, columnar_id: usize) -> Range<RowId> {\n        self.offset(columnar_id)..self.offset(columnar_id + 1)\n    }\n}\n\npub enum MergeRowOrder {\n    /// Columnar tables are simply stacked one above the other.\n    /// If the i-th columnar_readers has n_rows_i rows, then\n    /// in the resulting columnar,\n    /// rows [r0..n_row_0) contains the row of `columnar_readers[0]`, in ordder\n    /// rows [n_row_0..n_row_0 + n_row_1 contains the row of `columnar_readers[1]`, in order.\n    /// ..\n    /// No documents is deleted.\n    Stack(StackMergeOrder),\n    /// Some more complex mapping, that may interleaves rows from the different readers and\n    /// drop rows, or do both.\n    Shuffled(ShuffleMergeOrder),\n}\n\nimpl From<StackMergeOrder> for MergeRowOrder {\n    fn from(stack_merge_order: StackMergeOrder) -> MergeRowOrder {\n        MergeRowOrder::Stack(stack_merge_order)\n    }\n}\n\nimpl From<ShuffleMergeOrder> for MergeRowOrder {\n    fn from(shuffle_merge_order: ShuffleMergeOrder) -> MergeRowOrder {\n        MergeRowOrder::Shuffled(shuffle_merge_order)\n    }\n}\n\nimpl MergeRowOrder {\n    pub fn num_rows(&self) -> RowId {\n        match self {\n            MergeRowOrder::Stack(stack_row_order) => stack_row_order.num_rows(),\n            MergeRowOrder::Shuffled(complex_mapping) => complex_mapping.num_rows(),\n        }\n    }\n}\n\npub struct ShuffleMergeOrder {\n    pub new_row_id_to_old_row_id: Vec<RowAddr>,\n    pub alive_bitsets: Vec<Option<ReadOnlyBitSet>>,\n}\n\nimpl ShuffleMergeOrder {\n    pub fn for_test(\n        segment_num_rows: &[RowId],\n        new_row_id_to_old_row_id: Vec<RowAddr>,\n    ) -> ShuffleMergeOrder {\n        let mut alive_bitsets: Vec<BitSet> = segment_num_rows\n            .iter()\n            .map(|&num_rows| BitSet::with_max_value(num_rows))\n            .collect();\n        for &RowAddr {\n            segment_ord,\n            row_id,\n        } in &new_row_id_to_old_row_id\n        {\n            alive_bitsets[segment_ord as usize].insert(row_id);\n        }\n        let alive_bitsets: Vec<Option<ReadOnlyBitSet>> = alive_bitsets\n            .into_iter()\n            .map(|alive_bitset| {\n                let mut buffer = Vec::new();\n                alive_bitset.serialize(&mut buffer).unwrap();\n                let data = OwnedBytes::new(buffer);\n                Some(ReadOnlyBitSet::open(data))\n            })\n            .collect();\n        ShuffleMergeOrder {\n            new_row_id_to_old_row_id,\n            alive_bitsets,\n        }\n    }\n\n    pub fn num_rows(&self) -> RowId {\n        self.new_row_id_to_old_row_id.len() as RowId\n    }\n\n    pub fn iter_new_to_old_row_addrs(&self) -> impl Iterator<Item = RowAddr> + '_ {\n        self.new_row_id_to_old_row_id.iter().copied()\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/merge/mod.rs",
    "content": "mod merge_dict_column;\nmod merge_mapping;\nmod term_merger;\n\nuse std::collections::{BTreeMap, HashSet};\nuse std::io;\nuse std::net::Ipv6Addr;\nuse std::sync::Arc;\n\npub use merge_mapping::{MergeRowOrder, ShuffleMergeOrder, StackMergeOrder};\n\nuse super::writer::ColumnarSerializer;\nuse crate::column::{serialize_column_mappable_to_u64, serialize_column_mappable_to_u128};\nuse crate::column_values::MergedColumnValues;\nuse crate::columnar::ColumnarReader;\nuse crate::columnar::merge::merge_dict_column::merge_bytes_or_str_column;\nuse crate::columnar::writer::CompatibleNumericalTypes;\nuse crate::dynamic_column::DynamicColumn;\nuse crate::{\n    BytesColumn, Column, ColumnIndex, ColumnType, ColumnValues, DynamicColumnHandle, NumericalType,\n    NumericalValue,\n};\n\n/// Column types are grouped into different categories.\n/// After merge, all columns belonging to the same category are coerced to\n/// the same column type.\n///\n/// In practise, today, only Numerical columns are coerced into one type today.\n///\n/// See also [README.md].\n///\n/// The ordering has to match the ordering of the variants in [ColumnType].\n#[derive(Copy, Clone, Eq, PartialOrd, Ord, PartialEq, Hash, Debug)]\npub(crate) enum ColumnTypeCategory {\n    Numerical,\n    Bytes,\n    Str,\n    Bool,\n    IpAddr,\n    DateTime,\n}\n\nimpl From<ColumnType> for ColumnTypeCategory {\n    fn from(column_type: ColumnType) -> Self {\n        match column_type {\n            ColumnType::I64 => ColumnTypeCategory::Numerical,\n            ColumnType::U64 => ColumnTypeCategory::Numerical,\n            ColumnType::F64 => ColumnTypeCategory::Numerical,\n            ColumnType::Bytes => ColumnTypeCategory::Bytes,\n            ColumnType::Str => ColumnTypeCategory::Str,\n            ColumnType::Bool => ColumnTypeCategory::Bool,\n            ColumnType::IpAddr => ColumnTypeCategory::IpAddr,\n            ColumnType::DateTime => ColumnTypeCategory::DateTime,\n        }\n    }\n}\n\n/// Merge several columnar table together.\n///\n/// If several columns with the same name are conflicting with the numerical types in the\n/// input columnars, the first type compatible out of i64, u64, f64 in that order will be used.\n///\n/// `require_columns` makes it possible to ensure that some columns will be present in the\n/// resulting columnar. When a required column is a numerical column type, one of two things can\n/// happen:\n/// - If the required column type is compatible with all of the input columnar, the resulting merged\n///   columnar will simply coerce the input column and use the required column type.\n/// - If the required column type is incompatible with one of the input columnar, the merged will\n///   fail with an InvalidData error.\n///\n/// `merge_row_order` makes it possible to remove or reorder row in the resulting\n/// `Columnar` table.\n///\n/// Reminder: a string and a numerical column may bare the same column name. This is not\n/// considered a conflict.\npub fn merge_columnar(\n    columnar_readers: &[&ColumnarReader],\n    required_columns: &[(String, ColumnType)],\n    merge_row_order: MergeRowOrder,\n    output: &mut impl io::Write,\n) -> io::Result<()> {\n    let mut serializer = ColumnarSerializer::new(output);\n    let num_docs_per_columnar = columnar_readers\n        .iter()\n        .map(|reader| reader.num_docs())\n        .collect::<Vec<u32>>();\n\n    let columns_to_merge = group_columns_for_merge(columnar_readers, required_columns)?;\n    for res in columns_to_merge {\n        let ((column_name, _column_type_category), grouped_columns) = res;\n        let grouped_columns = grouped_columns.open(&merge_row_order)?;\n        if grouped_columns.is_empty() {\n            continue;\n        }\n\n        let column_type_after_merge = grouped_columns.column_type_after_merge();\n        let mut columns = grouped_columns.columns;\n        // Make sure the number of columns is the same as the number of columnar readers.\n        // Or num_docs_per_columnar would be incorrect.\n        assert_eq!(columns.len(), columnar_readers.len());\n        coerce_columns(column_type_after_merge, &mut columns)?;\n\n        let mut column_serializer =\n            serializer.start_serialize_column(column_name.as_bytes(), column_type_after_merge);\n        merge_column(\n            column_type_after_merge,\n            &num_docs_per_columnar,\n            columns,\n            &merge_row_order,\n            &mut column_serializer,\n        )?;\n        column_serializer.finalize()?;\n    }\n\n    serializer.finalize(merge_row_order.num_rows())?;\n    Ok(())\n}\n\nfn dynamic_column_to_u64_monotonic(dynamic_column: DynamicColumn) -> Option<Column<u64>> {\n    match dynamic_column {\n        DynamicColumn::Bool(column) => Some(column.to_u64_monotonic()),\n        DynamicColumn::I64(column) => Some(column.to_u64_monotonic()),\n        DynamicColumn::U64(column) => Some(column.to_u64_monotonic()),\n        DynamicColumn::F64(column) => Some(column.to_u64_monotonic()),\n        DynamicColumn::DateTime(column) => Some(column.to_u64_monotonic()),\n        DynamicColumn::IpAddr(_) | DynamicColumn::Bytes(_) | DynamicColumn::Str(_) => None,\n    }\n}\n\nfn merge_column(\n    column_type: ColumnType,\n    num_docs_per_column: &[u32],\n    columns_to_merge: Vec<Option<DynamicColumn>>,\n    merge_row_order: &MergeRowOrder,\n    wrt: &mut impl io::Write,\n) -> io::Result<()> {\n    match column_type {\n        ColumnType::I64\n        | ColumnType::U64\n        | ColumnType::F64\n        | ColumnType::DateTime\n        | ColumnType::Bool => {\n            let mut column_indexes: Vec<ColumnIndex> = Vec::with_capacity(columns_to_merge.len());\n            let mut column_values: Vec<Option<Arc<dyn ColumnValues>>> =\n                Vec::with_capacity(columns_to_merge.len());\n            for (i, dynamic_column_opt) in columns_to_merge.into_iter().enumerate() {\n                match dynamic_column_opt.and_then(dynamic_column_to_u64_monotonic) {\n                    Some(Column { index: idx, values }) => {\n                        column_indexes.push(idx);\n                        column_values.push(Some(values));\n                    }\n                    None => {\n                        column_indexes.push(ColumnIndex::Empty {\n                            num_docs: num_docs_per_column[i],\n                        });\n                        column_values.push(None);\n                    }\n                }\n            }\n            let merged_column_index =\n                crate::column_index::merge_column_index(&column_indexes[..], merge_row_order);\n            let merge_column_values = MergedColumnValues {\n                column_indexes: &column_indexes[..],\n                column_values: &column_values[..],\n                merge_row_order,\n            };\n            serialize_column_mappable_to_u64(merged_column_index, &merge_column_values, wrt)?;\n        }\n        ColumnType::IpAddr => {\n            let mut column_indexes: Vec<ColumnIndex> = Vec::with_capacity(columns_to_merge.len());\n            let mut column_values: Vec<Option<Arc<dyn ColumnValues<Ipv6Addr>>>> =\n                Vec::with_capacity(columns_to_merge.len());\n            for (i, dynamic_column_opt) in columns_to_merge.into_iter().enumerate() {\n                if let Some(DynamicColumn::IpAddr(Column { index: idx, values })) =\n                    dynamic_column_opt\n                {\n                    column_indexes.push(idx);\n                    column_values.push(Some(values));\n                } else {\n                    column_indexes.push(ColumnIndex::Empty {\n                        num_docs: num_docs_per_column[i],\n                    });\n                    column_values.push(None);\n                }\n            }\n\n            let merged_column_index =\n                crate::column_index::merge_column_index(&column_indexes[..], merge_row_order);\n            let merge_column_values = MergedColumnValues {\n                column_indexes: &column_indexes[..],\n                column_values: &column_values,\n                merge_row_order,\n            };\n\n            serialize_column_mappable_to_u128(merged_column_index, &merge_column_values, wrt)?;\n        }\n        ColumnType::Bytes | ColumnType::Str => {\n            let mut column_indexes: Vec<ColumnIndex> = Vec::with_capacity(columns_to_merge.len());\n            let mut bytes_columns: Vec<Option<BytesColumn>> =\n                Vec::with_capacity(columns_to_merge.len());\n            for (i, dynamic_column_opt) in columns_to_merge.into_iter().enumerate() {\n                match dynamic_column_opt {\n                    Some(DynamicColumn::Str(str_column)) => {\n                        column_indexes.push(str_column.term_ord_column.index.clone());\n                        bytes_columns.push(Some(str_column.into()));\n                    }\n                    Some(DynamicColumn::Bytes(bytes_column)) => {\n                        column_indexes.push(bytes_column.term_ord_column.index.clone());\n                        bytes_columns.push(Some(bytes_column));\n                    }\n                    _ => {\n                        column_indexes.push(ColumnIndex::Empty {\n                            num_docs: num_docs_per_column[i],\n                        });\n                        bytes_columns.push(None);\n                    }\n                }\n            }\n            let merged_column_index =\n                crate::column_index::merge_column_index(&column_indexes[..], merge_row_order);\n            merge_bytes_or_str_column(merged_column_index, &bytes_columns, merge_row_order, wrt)?;\n        }\n    }\n    Ok(())\n}\n\nstruct GroupedColumns {\n    required_column_type: Option<ColumnType>,\n    columns: Vec<Option<DynamicColumn>>,\n}\n\nimpl GroupedColumns {\n    /// Check is column group can be skipped during serialization.\n    fn is_empty(&self) -> bool {\n        self.required_column_type.is_none() && self.columns.iter().all(Option::is_none)\n    }\n\n    /// Returns the column type after merge.\n    ///\n    /// This method does not check if the column types can actually be coerced to\n    /// this type.\n    fn column_type_after_merge(&self) -> ColumnType {\n        if let Some(required_type) = self.required_column_type {\n            return required_type;\n        }\n        let column_type: HashSet<ColumnType> = self\n            .columns\n            .iter()\n            .flatten()\n            .map(|column| column.column_type())\n            .collect();\n        if column_type.len() == 1 {\n            return column_type.into_iter().next().unwrap();\n        }\n        // At the moment, only the numerical column type category has more than one possible\n        // column type.\n        assert!(\n            self.columns\n                .iter()\n                .flatten()\n                .all(|el| ColumnTypeCategory::from(el.column_type())\n                    == ColumnTypeCategory::Numerical)\n        );\n        merged_numerical_columns_type(self.columns.iter().flatten()).into()\n    }\n}\n\nstruct GroupedColumnsHandle {\n    required_column_type: Option<ColumnType>,\n    columns: Vec<Option<DynamicColumnHandle>>,\n}\n\nimpl GroupedColumnsHandle {\n    fn new(num_columnars: usize) -> Self {\n        GroupedColumnsHandle {\n            required_column_type: None,\n            columns: vec![None; num_columnars],\n        }\n    }\n    fn open(self, merge_row_order: &MergeRowOrder) -> io::Result<GroupedColumns> {\n        let mut columns: Vec<Option<DynamicColumn>> = Vec::new();\n        for (columnar_id, column) in self.columns.iter().enumerate() {\n            if let Some(column) = column {\n                let column = column.open()?;\n                // We skip columns that end up with 0 documents.\n                // That way, we make sure they don't end up influencing the merge type or\n                // creating empty columns.\n\n                if is_empty_after_merge(merge_row_order, &column, columnar_id) {\n                    columns.push(None);\n                } else {\n                    columns.push(Some(column));\n                }\n            } else {\n                columns.push(None);\n            }\n        }\n        Ok(GroupedColumns {\n            required_column_type: self.required_column_type,\n            columns,\n        })\n    }\n\n    /// Set the dynamic column for a given columnar.\n    fn set_column(&mut self, columnar_id: usize, column: DynamicColumnHandle) {\n        self.columns[columnar_id] = Some(column);\n    }\n\n    /// Force the existence of a column, as well as its type.\n    fn require_type(&mut self, required_type: ColumnType) -> io::Result<()> {\n        if let Some(existing_required_type) = self.required_column_type {\n            if existing_required_type == required_type {\n                // This was just a duplicate in the `required_columns`.\n                // Nothing to do.\n                return Ok(());\n            } else {\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidInput,\n                    \"Required column conflicts with another required column of the same type \\\n                     category.\",\n                ));\n            }\n        }\n        self.required_column_type = Some(required_type);\n        Ok(())\n    }\n}\n\n/// Returns the type of the merged numerical column.\n///\n/// This function picks the first numerical type out of i64, u64, f64 (order matters\n/// here), that is compatible with all the `columns`.\n///\n/// # Panics\n/// Panics if one of the column is not numerical.\nfn merged_numerical_columns_type<'a>(\n    columns: impl Iterator<Item = &'a DynamicColumn>,\n) -> NumericalType {\n    let mut compatible_numerical_types = CompatibleNumericalTypes::default();\n    for column in columns {\n        let (min_value, max_value) =\n            min_max_if_numerical(column).expect(\"All columns re required to be numerical\");\n        compatible_numerical_types.accept_value(min_value);\n        compatible_numerical_types.accept_value(max_value);\n    }\n    compatible_numerical_types.to_numerical_type()\n}\n\nfn is_empty_after_merge(\n    merge_row_order: &MergeRowOrder,\n    column: &DynamicColumn,\n    columnar_ord: usize,\n) -> bool {\n    if column.num_values() == 0u32 {\n        // It was empty before the merge.\n        return true;\n    }\n    match merge_row_order {\n        MergeRowOrder::Stack(_) => {\n            // If we are stacking the columnar, no rows are being deleted.\n            false\n        }\n        MergeRowOrder::Shuffled(shuffled) => {\n            if let Some(alive_bitset) = &shuffled.alive_bitsets[columnar_ord] {\n                let column_index = column.column_index();\n                match column_index {\n                    ColumnIndex::Empty { .. } => true,\n                    ColumnIndex::Full => alive_bitset.len() == 0,\n                    ColumnIndex::Optional(optional_index) => {\n                        for doc in optional_index.iter_non_null_docs() {\n                            if alive_bitset.contains(doc) {\n                                return false;\n                            }\n                        }\n                        true\n                    }\n                    ColumnIndex::Multivalued(multivalued_index) => {\n                        for alive_docid in alive_bitset.iter() {\n                            if !multivalued_index.range(alive_docid).is_empty() {\n                                return false;\n                            }\n                        }\n                        true\n                    }\n                }\n            } else {\n                // No document is being deleted.\n                // The shuffle is applying a permutation.\n                false\n            }\n        }\n    }\n}\n\n/// Iterates over the columns of the columnar readers, grouped by column name.\n/// Key functionality is that `open` of the Columns is done lazy per group.\nfn group_columns_for_merge<'a>(\n    columnar_readers: &'a [&'a ColumnarReader],\n    required_columns: &'a [(String, ColumnType)],\n) -> io::Result<BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle>> {\n    let mut columns: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> = BTreeMap::new();\n\n    for &(ref column_name, column_type) in required_columns {\n        columns\n            .entry((column_name.clone(), column_type.into()))\n            .or_insert_with(|| GroupedColumnsHandle::new(columnar_readers.len()))\n            .require_type(column_type)?;\n    }\n\n    for (columnar_id, columnar_reader) in columnar_readers.iter().enumerate() {\n        let column_name_and_handle = columnar_reader.iter_columns()?;\n\n        for (column_name, handle) in column_name_and_handle {\n            let column_category: ColumnTypeCategory = handle.column_type().into();\n            columns\n                .entry((column_name, column_category))\n                .or_insert_with(|| GroupedColumnsHandle::new(columnar_readers.len()))\n                .set_column(columnar_id, handle);\n        }\n    }\n    Ok(columns)\n}\n\nfn coerce_columns(\n    column_type: ColumnType,\n    columns: &mut [Option<DynamicColumn>],\n) -> io::Result<()> {\n    for column_opt in columns.iter_mut() {\n        if let Some(column) = column_opt.take() {\n            *column_opt = Some(coerce_column(column_type, column)?);\n        }\n    }\n    Ok(())\n}\n\nfn coerce_column(column_type: ColumnType, column: DynamicColumn) -> io::Result<DynamicColumn> {\n    if let Some(numerical_type) = column_type.numerical_type() {\n        column\n            .coerce_numerical(numerical_type)\n            .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, \"\"))\n    } else {\n        if column.column_type() != column_type {\n            return Err(io::Error::new(\n                io::ErrorKind::InvalidInput,\n                format!(\n                    \"Cannot coerce column of type `{:?}` to `{column_type:?}`\",\n                    column.column_type()\n                ),\n            ));\n        }\n        Ok(column)\n    }\n}\n\n/// Returns the (min, max) of a column provided it is numerical (i64, u64. f64).\n///\n/// The min and the max are simply the numerical value as defined by `ColumnValue::min_value()`,\n/// and `ColumnValue::max_value()`.\n///\n/// It is important to note that these values are only guaranteed to be lower/upper bound\n/// (as opposed to min/max value).\n/// If a column is empty, the min and max values are currently set to 0.\nfn min_max_if_numerical(column: &DynamicColumn) -> Option<(NumericalValue, NumericalValue)> {\n    match column {\n        DynamicColumn::I64(column) => Some((column.min_value().into(), column.max_value().into())),\n        DynamicColumn::U64(column) => Some((column.min_value().into(), column.max_value().into())),\n        DynamicColumn::F64(column) => Some((column.min_value().into(), column.max_value().into())),\n        DynamicColumn::Bool(_)\n        | DynamicColumn::IpAddr(_)\n        | DynamicColumn::DateTime(_)\n        | DynamicColumn::Bytes(_)\n        | DynamicColumn::Str(_) => None,\n    }\n}\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "columnar/src/columnar/merge/term_merger.rs",
    "content": "use std::cmp::Ordering;\nuse std::collections::BinaryHeap;\n\nuse sstable::TermOrdinal;\n\nuse crate::Streamer;\n\n/// The terms of a column with the ordinal of the segment.\npub struct TermsWithSegmentOrd<'a> {\n    pub terms: Streamer<'a>,\n    pub segment_ord: usize,\n}\n\nimpl PartialEq for TermsWithSegmentOrd<'_> {\n    fn eq(&self, other: &Self) -> bool {\n        self.segment_ord == other.segment_ord\n    }\n}\n\nimpl Eq for TermsWithSegmentOrd<'_> {}\n\nimpl<'a> PartialOrd for TermsWithSegmentOrd<'a> {\n    fn partial_cmp(&self, other: &TermsWithSegmentOrd<'a>) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl<'a> Ord for TermsWithSegmentOrd<'a> {\n    fn cmp(&self, other: &TermsWithSegmentOrd<'a>) -> Ordering {\n        (&other.terms.key(), &other.segment_ord).cmp(&(&self.terms.key(), &self.segment_ord))\n    }\n}\n\n/// Given a list of sorted term streams,\n/// returns an iterator over sorted unique terms.\n///\n/// The item yield is actually a pair with\n/// - the term\n/// - a slice with the ordinal of the segments containing the terms.\npub struct TermMerger<'a> {\n    heap: BinaryHeap<TermsWithSegmentOrd<'a>>,\n    term_streams_with_segment: Vec<TermsWithSegmentOrd<'a>>,\n}\n\nimpl<'a> TermMerger<'a> {\n    /// Stream of merged term dictionary\n    pub fn new(term_streams_with_segment: Vec<TermsWithSegmentOrd<'a>>) -> TermMerger<'a> {\n        TermMerger {\n            heap: BinaryHeap::new(),\n            term_streams_with_segment,\n        }\n    }\n\n    pub(crate) fn matching_segments<'b: 'a>(\n        &'b self,\n    ) -> impl 'b + Iterator<Item = (usize, TermOrdinal)> {\n        self.term_streams_with_segment\n            .iter()\n            .map(|heap_item| (heap_item.segment_ord, heap_item.terms.term_ord()))\n    }\n\n    fn advance_segments(&mut self) {\n        let streamers = &mut self.term_streams_with_segment;\n        let heap = &mut self.heap;\n        for mut heap_item in streamers.drain(..) {\n            if heap_item.terms.advance() {\n                heap.push(heap_item);\n            }\n        }\n    }\n\n    /// Advance the term iterator to the next term.\n    /// Returns true if there is indeed another term\n    /// False if there is none.\n    pub fn advance(&mut self) -> bool {\n        self.advance_segments();\n        match self.heap.pop() {\n            Some(head) => {\n                self.term_streams_with_segment.push(head);\n                while let Some(next_streamer) = self.heap.peek() {\n                    if self.term_streams_with_segment[0].terms.key() != next_streamer.terms.key() {\n                        break;\n                    }\n                    let next_heap_it = self.heap.pop().unwrap(); // safe : we peeked beforehand\n                    self.term_streams_with_segment.push(next_heap_it);\n                }\n                true\n            }\n            _ => false,\n        }\n    }\n\n    /// Returns the current term.\n    ///\n    /// This method may be called\n    /// if and only if advance() has been called before\n    /// and \"true\" was returned.\n    pub fn key(&self) -> &[u8] {\n        self.term_streams_with_segment[0].terms.key()\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/merge/tests.rs",
    "content": "use itertools::Itertools;\nuse proptest::collection::vec;\nuse proptest::prelude::*;\n\nuse super::*;\nuse crate::columnar::{ColumnarReader, MergeRowOrder, StackMergeOrder, merge_columnar};\nuse crate::{Cardinality, ColumnarWriter, DynamicColumn, HasAssociatedColumnType, RowId};\n\nfn make_columnar<T: Into<NumericalValue> + HasAssociatedColumnType + Copy>(\n    column_name: &str,\n    vals: &[T],\n) -> ColumnarReader {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_column_type(column_name, T::column_type(), false);\n    for (row_id, val) in vals.iter().copied().enumerate() {\n        dataframe_writer.record_numerical(row_id as RowId, column_name, val.into());\n    }\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer\n        .serialize(vals.len() as RowId, &mut buffer)\n        .unwrap();\n    ColumnarReader::open(buffer).unwrap()\n}\n\n#[test]\nfn test_column_coercion_to_u64() {\n    // i64 type\n    let columnar1 = make_columnar(\"numbers\", &[1i64]);\n    // u64 type\n    let columnar2 = make_columnar(\"numbers\", &[u64::MAX]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> =\n        group_columns_for_merge(columnars, &[]).unwrap();\n    assert_eq!(column_map.len(), 1);\n    assert!(column_map.contains_key(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical)));\n}\n\n#[test]\nfn test_column_coercion_to_i64() {\n    let columnar1 = make_columnar(\"numbers\", &[-1i64]);\n    let columnar2 = make_columnar(\"numbers\", &[2u64]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> =\n        group_columns_for_merge(columnars, &[]).unwrap();\n    assert_eq!(column_map.len(), 1);\n    assert!(column_map.contains_key(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical)));\n}\n\n//#[test]\n// fn test_impossible_coercion_returns_an_error() {\n// let columnar1 = make_columnar(\"numbers\", &[u64::MAX]);\n// let merge_order = StackMergeOrder::stack(&[&columnar1]).into();\n// let group_error = group_columns_for_merge_iter(\n//&[&columnar1],\n//&[(\"numbers\".to_string(), ColumnType::I64)],\n//&merge_order,\n//)\n//.unwrap_err();\n// assert_eq!(group_error.kind(), io::ErrorKind::InvalidInput);\n//}\n\n#[test]\nfn test_group_columns_with_required_column() {\n    let columnar1 = make_columnar(\"numbers\", &[1i64]);\n    let columnar2 = make_columnar(\"numbers\", &[2u64]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> =\n        group_columns_for_merge(columnars, &[(\"numbers\".to_string(), ColumnType::U64)]).unwrap();\n    assert_eq!(column_map.len(), 1);\n    assert!(column_map.contains_key(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical)));\n}\n\n#[test]\nfn test_group_columns_required_column_with_no_existing_columns() {\n    let columnar1 = make_columnar(\"numbers\", &[2u64]);\n    let columnar2 = make_columnar(\"numbers\", &[2u64]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<_, _> =\n        group_columns_for_merge(columnars, &[(\"required_col\".to_string(), ColumnType::Str)])\n            .unwrap();\n    assert_eq!(column_map.len(), 2);\n    let columns = &column_map\n        .get(&(\"required_col\".to_string(), ColumnTypeCategory::Str))\n        .unwrap()\n        .columns;\n    assert_eq!(columns.len(), 2);\n    assert!(columns[0].is_none());\n    assert!(columns[1].is_none());\n}\n\n#[test]\nfn test_group_columns_required_column_is_above_all_columns_have_the_same_type_rule() {\n    let columnar1 = make_columnar(\"numbers\", &[2i64]);\n    let columnar2 = make_columnar(\"numbers\", &[2i64]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> =\n        group_columns_for_merge(columnars, &[(\"numbers\".to_string(), ColumnType::U64)]).unwrap();\n    assert_eq!(column_map.len(), 1);\n    assert!(column_map.contains_key(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical)));\n}\n\n#[test]\nfn test_missing_column() {\n    let columnar1 = make_columnar(\"numbers\", &[-1i64]);\n    let columnar2 = make_columnar(\"numbers2\", &[2u64]);\n    let columnars = &[&columnar1, &columnar2];\n    let column_map: BTreeMap<(String, ColumnTypeCategory), GroupedColumnsHandle> =\n        group_columns_for_merge(columnars, &[]).unwrap();\n    assert_eq!(column_map.len(), 2);\n    assert!(column_map.contains_key(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical)));\n    {\n        let columns = &column_map\n            .get(&(\"numbers\".to_string(), ColumnTypeCategory::Numerical))\n            .unwrap()\n            .columns;\n        assert!(columns[0].is_some());\n        assert!(columns[1].is_none());\n    }\n    {\n        let columns = &column_map\n            .get(&(\"numbers2\".to_string(), ColumnTypeCategory::Numerical))\n            .unwrap()\n            .columns;\n        assert!(columns[0].is_none());\n        assert!(columns[1].is_some());\n    }\n}\n\nfn make_numerical_columnar_multiple_columns(\n    columns: &[(&str, &[&[NumericalValue]])],\n) -> ColumnarReader {\n    let mut dataframe_writer = ColumnarWriter::default();\n    for (column_name, column_values) in columns {\n        for (row_id, vals) in column_values.iter().enumerate() {\n            for val in vals.iter() {\n                dataframe_writer.record_numerical(row_id as u32, column_name, *val);\n            }\n        }\n    }\n    let num_rows = columns\n        .iter()\n        .map(|(_, val_rows)| val_rows.len() as RowId)\n        .max()\n        .unwrap_or(0u32);\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(num_rows, &mut buffer).unwrap();\n    ColumnarReader::open(buffer).unwrap()\n}\n\n#[track_caller]\nfn make_byte_columnar_multiple_columns(\n    columns: &[(&str, &[&[&[u8]]])],\n    num_rows: u32,\n) -> ColumnarReader {\n    let mut dataframe_writer = ColumnarWriter::default();\n    for (column_name, column_values) in columns {\n        assert_eq!(\n            column_values.len(),\n            num_rows as usize,\n            \"All columns must have `{num_rows}` rows\"\n        );\n        for (row_id, vals) in column_values.iter().enumerate() {\n            for val in vals.iter() {\n                dataframe_writer.record_bytes(row_id as u32, column_name, val);\n            }\n        }\n    }\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(num_rows, &mut buffer).unwrap();\n    ColumnarReader::open(buffer).unwrap()\n}\n\nfn make_text_columnar_multiple_columns(columns: &[(&str, &[&[&str]])]) -> ColumnarReader {\n    let mut dataframe_writer = ColumnarWriter::default();\n    for (column_name, column_values) in columns {\n        for (row_id, vals) in column_values.iter().enumerate() {\n            for val in vals.iter() {\n                dataframe_writer.record_str(row_id as u32, column_name, val);\n            }\n        }\n    }\n    let num_rows = columns\n        .iter()\n        .map(|(_, val_rows)| val_rows.len() as RowId)\n        .max()\n        .unwrap_or(0u32);\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(num_rows, &mut buffer).unwrap();\n    ColumnarReader::open(buffer).unwrap()\n}\n\n#[test]\nfn test_merge_columnar_numbers() {\n    let columnar1 =\n        make_numerical_columnar_multiple_columns(&[(\"numbers\", &[&[NumericalValue::from(-1f64)]])]);\n    let columnar2 = make_numerical_columnar_multiple_columns(&[(\n        \"numbers\",\n        &[&[], &[NumericalValue::from(-3f64)]],\n    )]);\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 3);\n    assert_eq!(columnar_reader.num_columns(), 1);\n    let cols = columnar_reader.read_columns(\"numbers\").unwrap();\n    let dynamic_column = cols[0].open().unwrap();\n    let DynamicColumn::F64(vals) = dynamic_column else {\n        panic!()\n    };\n    assert_eq!(vals.get_cardinality(), Cardinality::Optional);\n    assert_eq!(vals.first(0u32), Some(-1f64));\n    assert_eq!(vals.first(1u32), None);\n    assert_eq!(vals.first(2u32), Some(-3f64));\n}\n\n#[test]\nfn test_merge_columnar_texts() {\n    let columnar1 = make_text_columnar_multiple_columns(&[(\"texts\", &[&[\"a\"]])]);\n    let columnar2 = make_text_columnar_multiple_columns(&[(\"texts\", &[&[], &[\"b\"]])]);\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 3);\n    assert_eq!(columnar_reader.num_columns(), 1);\n    let cols = columnar_reader.read_columns(\"texts\").unwrap();\n    let dynamic_column = cols[0].open().unwrap();\n    let DynamicColumn::Str(vals) = dynamic_column else {\n        panic!()\n    };\n    assert_eq!(vals.ords().get_cardinality(), Cardinality::Optional);\n\n    let get_str_for_ord = |ord| {\n        let mut out = String::new();\n        vals.ord_to_str(ord, &mut out).unwrap();\n        out\n    };\n\n    assert_eq!(vals.dictionary.num_terms(), 2);\n    assert_eq!(get_str_for_ord(0), \"a\");\n    assert_eq!(get_str_for_ord(1), \"b\");\n\n    let get_str_for_row = |row_id| {\n        let term_ords: Vec<u64> = vals.term_ords(row_id).collect();\n        assert!(term_ords.len() <= 1);\n        let mut out = String::new();\n        if term_ords.len() == 1 {\n            vals.ord_to_str(term_ords[0], &mut out).unwrap();\n        }\n        out\n    };\n\n    assert_eq!(get_str_for_row(0), \"a\");\n    assert_eq!(get_str_for_row(1), \"\");\n    assert_eq!(get_str_for_row(2), \"b\");\n}\n\n#[test]\nfn test_merge_columnar_byte() {\n    let columnar1 = make_byte_columnar_multiple_columns(&[(\"bytes\", &[&[b\"bbbb\"], &[b\"baaa\"]])], 2);\n    let columnar2 = make_byte_columnar_multiple_columns(&[(\"bytes\", &[&[], &[b\"a\"]])], 2);\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 4);\n    assert_eq!(columnar_reader.num_columns(), 1);\n    let cols = columnar_reader.read_columns(\"bytes\").unwrap();\n    let dynamic_column = cols[0].open().unwrap();\n    let DynamicColumn::Bytes(vals) = dynamic_column else {\n        panic!()\n    };\n    let get_bytes_for_ord = |ord| {\n        let mut out = Vec::new();\n        vals.ord_to_bytes(ord, &mut out).unwrap();\n        out\n    };\n\n    assert_eq!(vals.dictionary.num_terms(), 3);\n    assert_eq!(get_bytes_for_ord(0), b\"a\");\n    assert_eq!(get_bytes_for_ord(1), b\"baaa\");\n    assert_eq!(get_bytes_for_ord(2), b\"bbbb\");\n\n    let get_bytes_for_row = |row_id| {\n        let term_ords: Vec<u64> = vals.term_ords(row_id).collect();\n        assert!(term_ords.len() <= 1);\n        let mut out = Vec::new();\n        if term_ords.len() == 1 {\n            vals.ord_to_bytes(term_ords[0], &mut out).unwrap();\n        }\n        out\n    };\n\n    assert_eq!(get_bytes_for_row(0), b\"bbbb\");\n    assert_eq!(get_bytes_for_row(1), b\"baaa\");\n    assert_eq!(get_bytes_for_row(2), b\"\");\n    assert_eq!(get_bytes_for_row(3), b\"a\");\n}\n\n#[test]\nfn test_merge_columnar_byte_with_missing() {\n    let columnar1 = make_byte_columnar_multiple_columns(&[], 3);\n    let columnar2 = make_byte_columnar_multiple_columns(&[(\"col\", &[&[b\"b\"], &[]])], 2);\n    let columnar3 = make_byte_columnar_multiple_columns(\n        &[\n            (\"col\", &[&[], &[b\"b\"], &[b\"a\", b\"b\"]]),\n            (\"col2\", &[&[b\"hello\"], &[], &[b\"a\", b\"b\"]]),\n        ],\n        3,\n    );\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2, &columnar3];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 3 + 2 + 3);\n    assert_eq!(columnar_reader.num_columns(), 2);\n    let cols = columnar_reader.read_columns(\"col\").unwrap();\n    let dynamic_column = cols[0].open().unwrap();\n    let DynamicColumn::Bytes(vals) = dynamic_column else {\n        panic!()\n    };\n    let get_bytes_for_ord = |ord| {\n        let mut out = Vec::new();\n        vals.ord_to_bytes(ord, &mut out).unwrap();\n        out\n    };\n    assert_eq!(vals.dictionary.num_terms(), 2);\n    assert_eq!(get_bytes_for_ord(0), b\"a\");\n    assert_eq!(get_bytes_for_ord(1), b\"b\");\n    let get_bytes_for_row = |row_id| {\n        let terms: Vec<Vec<u8>> = vals\n            .term_ords(row_id)\n            .map(|term_ord| {\n                let mut out = Vec::new();\n                vals.ord_to_bytes(term_ord, &mut out).unwrap();\n                out\n            })\n            .collect();\n        terms\n    };\n    assert!(get_bytes_for_row(0).is_empty());\n    assert!(get_bytes_for_row(1).is_empty());\n    assert!(get_bytes_for_row(2).is_empty());\n    assert_eq!(get_bytes_for_row(3), vec![b\"b\".to_vec()]);\n    assert!(get_bytes_for_row(4).is_empty());\n    assert!(get_bytes_for_row(5).is_empty());\n    assert_eq!(get_bytes_for_row(6), vec![b\"b\".to_vec()]);\n    assert_eq!(get_bytes_for_row(7), vec![b\"a\".to_vec(), b\"b\".to_vec()]);\n}\n\n#[test]\nfn test_merge_columnar_different_types() {\n    let columnar1 = make_text_columnar_multiple_columns(&[(\"mixed\", &[&[\"a\"]])]);\n    let columnar2 = make_text_columnar_multiple_columns(&[(\"mixed\", &[&[], &[\"b\"]])]);\n    let columnar3 = make_columnar(\"mixed\", &[1i64]);\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2, &columnar3];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 4);\n    assert_eq!(columnar_reader.num_columns(), 2);\n    let cols = columnar_reader.read_columns(\"mixed\").unwrap();\n\n    // numeric column\n    let dynamic_column = cols[0].open().unwrap();\n    let DynamicColumn::I64(vals) = dynamic_column else {\n        panic!()\n    };\n    assert_eq!(vals.get_cardinality(), Cardinality::Optional);\n    assert_eq!(vals.values_for_doc(0).collect_vec(), Vec::<i64>::new());\n    assert_eq!(vals.values_for_doc(1).collect_vec(), Vec::<i64>::new());\n    assert_eq!(vals.values_for_doc(2).collect_vec(), Vec::<i64>::new());\n    assert_eq!(vals.values_for_doc(3).collect_vec(), vec![1]);\n    assert_eq!(vals.values_for_doc(4).collect_vec(), Vec::<i64>::new());\n\n    // text column\n    let dynamic_column = cols[1].open().unwrap();\n    let DynamicColumn::Str(vals) = dynamic_column else {\n        panic!()\n    };\n    assert_eq!(vals.ords().get_cardinality(), Cardinality::Optional);\n    let get_str_for_ord = |ord| {\n        let mut out = String::new();\n        vals.ord_to_str(ord, &mut out).unwrap();\n        out\n    };\n\n    assert_eq!(vals.dictionary.num_terms(), 2);\n    assert_eq!(get_str_for_ord(0), \"a\");\n    assert_eq!(get_str_for_ord(1), \"b\");\n\n    let get_str_for_row = |row_id| {\n        let term_ords: Vec<String> = vals\n            .term_ords(row_id)\n            .map(|el| {\n                let mut out = String::new();\n                vals.ord_to_str(el, &mut out).unwrap();\n                out\n            })\n            .collect();\n        term_ords\n    };\n\n    assert_eq!(get_str_for_row(0), vec![\"a\".to_string()]);\n    assert_eq!(get_str_for_row(1), Vec::<String>::new());\n    assert_eq!(get_str_for_row(2), vec![\"b\".to_string()]);\n    assert_eq!(get_str_for_row(3), Vec::<String>::new());\n}\n\n#[test]\nfn test_merge_columnar_different_empty_cardinality() {\n    let columnar1 = make_text_columnar_multiple_columns(&[(\"mixed\", &[&[\"a\"]])]);\n    let columnar2 = make_columnar(\"mixed\", &[1i64]);\n    let mut buffer = Vec::new();\n    let columnars = &[&columnar1, &columnar2];\n    let stack_merge_order = StackMergeOrder::stack(columnars);\n    crate::columnar::merge_columnar(\n        columnars,\n        &[],\n        MergeRowOrder::Stack(stack_merge_order),\n        &mut buffer,\n    )\n    .unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_docs(), 2);\n    assert_eq!(columnar_reader.num_columns(), 2);\n    let cols = columnar_reader.read_columns(\"mixed\").unwrap();\n\n    // numeric column\n    let dynamic_column = cols[0].open().unwrap();\n    assert_eq!(dynamic_column.get_cardinality(), Cardinality::Optional);\n\n    // text column\n    let dynamic_column = cols[1].open().unwrap();\n    assert_eq!(dynamic_column.get_cardinality(), Cardinality::Optional);\n}\n\n#[derive(Debug, Clone)]\nstruct ColumnSpec {\n    column_name: String,\n    /// (row_id, term)\n    terms: Vec<(RowId, Vec<u8>)>,\n}\n\n#[derive(Clone, Debug)]\nstruct ColumnarSpec {\n    columns: Vec<ColumnSpec>,\n}\n\n/// Generate a random (row_id, term) pair:\n///  - row_id in [0..10]\n///  - term is either from POSSIBLE_TERMS or random bytes\nfn rowid_and_term_strategy() -> impl Strategy<Value = (RowId, Vec<u8>)> {\n    const POSSIBLE_TERMS: &[&[u8]] = &[b\"a\", b\"b\", b\"allo\"];\n\n    let term_strat = prop_oneof![\n        // pick from the fixed list\n        (0..POSSIBLE_TERMS.len()).prop_map(|i| POSSIBLE_TERMS[i].to_vec()),\n        // or random bytes (length 0..10)\n        prop::collection::vec(any::<u8>(), 0..10),\n    ];\n\n    (0u32..11, term_strat)\n}\n\n/// Generate one ColumnSpec, with a random name and a random list of (row_id, term).\n/// We sort it by row_id so that data is in ascending order.\nfn column_spec_strategy() -> impl Strategy<Value = ColumnSpec> {\n    let column_name = prop_oneof![\n        Just(\"col\".to_string()),\n        Just(\"col2\".to_string()),\n        \"col.*\".prop_map(|s| s),\n    ];\n\n    // We'll produce 0..8 (rowid,term) entries for this column\n    let data_strat = vec(rowid_and_term_strategy(), 0..8).prop_map(|mut pairs| {\n        // Sort by row_id\n        pairs.sort_by_key(|(row_id, _)| *row_id);\n        pairs\n    });\n\n    (column_name, data_strat).prop_map(|(name, data)| ColumnSpec {\n        column_name: name,\n        terms: data,\n    })\n}\n\n/// Strategy to generate an ColumnarSpec\nfn columnar_strategy() -> impl Strategy<Value = ColumnarSpec> {\n    vec(column_spec_strategy(), 0..3).prop_map(|columns| ColumnarSpec { columns })\n}\n\n/// Strategy to generate multiple ColumnarSpecs, each of which we will treat\n/// as one \"columnar\" to be merged together.\nfn columnars_strategy() -> impl Strategy<Value = Vec<ColumnarSpec>> {\n    vec(columnar_strategy(), 1..4)\n}\n\n/// Build a `ColumnarReader` from a `ColumnarSpec`\nfn build_columnar(spec: &ColumnarSpec) -> ColumnarReader {\n    let mut writer = ColumnarWriter::default();\n    let mut max_row_id = 0;\n    for col in &spec.columns {\n        for &(row_id, ref term) in &col.terms {\n            writer.record_bytes(row_id, &col.column_name, term);\n            max_row_id = max_row_id.max(row_id);\n        }\n    }\n\n    let mut buffer = Vec::new();\n    writer.serialize(max_row_id + 1, &mut buffer).unwrap();\n    ColumnarReader::open(buffer).unwrap()\n}\n\nproptest! {\n    // We just test that the merge_columnar function doesn't crash.\n    #![proptest_config(ProptestConfig::with_cases(256))]\n    #[test]\n    fn test_merge_columnar_bytes_no_crash(columnars in columnars_strategy(), second_merge_columnars in columnars_strategy()) {\n        let columnars: Vec<ColumnarReader> = columnars.iter()\n            .map(build_columnar)\n            .collect();\n\n        let mut out = Vec::new();\n        let columnar_refs: Vec<&ColumnarReader> = columnars.iter().collect();\n        let stack_merge_order = StackMergeOrder::stack(&columnar_refs);\n        merge_columnar(\n            &columnar_refs,\n            &[],\n            MergeRowOrder::Stack(stack_merge_order),\n            &mut out,\n        ).unwrap();\n\n        let merged_reader = ColumnarReader::open(out).unwrap();\n\n        // Merge the second set of columnars with the result of the first merge\n        let mut columnars: Vec<ColumnarReader> = second_merge_columnars.iter()\n            .map(build_columnar)\n            .collect();\n        columnars.push(merged_reader);\n        let mut out = Vec::new();\n        let columnar_refs: Vec<&ColumnarReader> = columnars.iter().collect();\n        let stack_merge_order = StackMergeOrder::stack(&columnar_refs);\n        merge_columnar(\n            &columnar_refs,\n            &[],\n            MergeRowOrder::Stack(stack_merge_order),\n            &mut out,\n        ).unwrap();\n\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/mod.rs",
    "content": "mod column_type;\nmod format_version;\nmod merge;\nmod reader;\nmod writer;\n\npub use column_type::{ColumnType, HasAssociatedColumnType};\npub use format_version::{CURRENT_VERSION, Version};\n#[cfg(test)]\npub(crate) use merge::ColumnTypeCategory;\npub use merge::{MergeRowOrder, ShuffleMergeOrder, StackMergeOrder, merge_columnar};\npub use reader::ColumnarReader;\npub use writer::ColumnarWriter;\n"
  },
  {
    "path": "columnar/src/columnar/reader/mod.rs",
    "content": "use std::{fmt, io, mem};\n\nuse common::BinarySerializable;\nuse common::file_slice::FileSlice;\nuse common::json_path_writer::JSON_PATH_SEGMENT_SEP;\nuse sstable::{Dictionary, RangeSSTable};\n\nuse crate::columnar::{ColumnType, format_version};\nuse crate::dynamic_column::DynamicColumnHandle;\nuse crate::{RowId, Version};\n\nfn io_invalid_data(msg: String) -> io::Error {\n    io::Error::new(io::ErrorKind::InvalidData, msg)\n}\n\n/// The ColumnarReader makes it possible to access a set of columns\n/// associated to field names.\n#[derive(Clone)]\npub struct ColumnarReader {\n    column_dictionary: Dictionary<RangeSSTable>,\n    column_data: FileSlice,\n    num_docs: RowId,\n    format_version: Version,\n}\n\nimpl fmt::Debug for ColumnarReader {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        let num_rows = self.num_docs();\n        let columns = self.list_columns().unwrap();\n        let num_cols = columns.len();\n        let mut debug_struct = f.debug_struct(\"Columnar\");\n        debug_struct\n            .field(\"num_rows\", &num_rows)\n            .field(\"num_cols\", &num_cols);\n        for (col_name, dynamic_column_handle) in columns.into_iter().take(5) {\n            let col = dynamic_column_handle.open().unwrap();\n            if col.num_values() > 10 {\n                debug_struct.field(&col_name, &\"..\");\n            } else {\n                debug_struct.field(&col_name, &col);\n            }\n        }\n        if num_cols > 5 {\n            debug_struct.finish_non_exhaustive()?;\n        } else {\n            debug_struct.finish()?;\n        }\n        Ok(())\n    }\n}\n\n/// Functions by both the async/sync code listing columns.\n/// It takes a stream from the column sstable and return the list of\n/// `DynamicColumn` available in it.\nfn read_all_columns_in_stream(\n    mut stream: sstable::Streamer<'_, RangeSSTable>,\n    column_data: &FileSlice,\n    format_version: Version,\n) -> io::Result<Vec<DynamicColumnHandle>> {\n    let mut results = Vec::new();\n    while stream.advance() {\n        let key_bytes: &[u8] = stream.key();\n        let Some(column_code) = key_bytes.last().copied() else {\n            return Err(io_invalid_data(\"Empty column name.\".to_string()));\n        };\n        let column_type = ColumnType::try_from_code(column_code)\n            .map_err(|_| io_invalid_data(format!(\"Unknown column code `{column_code}`\")))?;\n        let range = stream.value();\n        let file_slice = column_data.slice(range.start as usize..range.end as usize);\n        let dynamic_column_handle = DynamicColumnHandle {\n            file_slice,\n            column_type,\n            format_version,\n        };\n        results.push(dynamic_column_handle);\n    }\n    Ok(results)\n}\n\nfn column_dictionary_prefix_for_column_name(column_name: &str) -> String {\n    // Each column is a associated to a given `column_key`,\n    // that starts by `column_name\\0column_header`.\n    //\n    // Listing the columns associated to the given column name is therefore equivalent to\n    // listing `column_key` with the prefix `column_name\\0`.\n    format!(\"{}{}\", column_name, '\\0')\n}\n\nfn column_dictionary_prefix_for_subpath(root_path: &str) -> String {\n    format!(\"{}{}\", root_path, JSON_PATH_SEGMENT_SEP as char)\n}\n\nimpl ColumnarReader {\n    /// Opens a new Columnar file.\n    pub fn open<F>(file_slice: F) -> io::Result<ColumnarReader>\n    where FileSlice: From<F> {\n        Self::open_inner(file_slice.into())\n    }\n\n    fn open_inner(file_slice: FileSlice) -> io::Result<ColumnarReader> {\n        let (file_slice_without_sstable_len, footer_slice) = file_slice\n            .split_from_end(mem::size_of::<u64>() + 4 + format_version::VERSION_FOOTER_NUM_BYTES);\n        let footer_bytes = footer_slice.read_bytes()?;\n        let sstable_len = u64::deserialize(&mut &footer_bytes[0..8])?;\n        let num_rows = u32::deserialize(&mut &footer_bytes[8..12])?;\n        let version_footer_bytes: [u8; format_version::VERSION_FOOTER_NUM_BYTES] =\n            footer_bytes[12..].try_into().unwrap();\n        let format_version = format_version::parse_footer(version_footer_bytes)?;\n        let (column_data, sstable) =\n            file_slice_without_sstable_len.split_from_end(sstable_len as usize);\n        let column_dictionary = Dictionary::open(sstable)?;\n        Ok(ColumnarReader {\n            column_dictionary,\n            column_data,\n            num_docs: num_rows,\n            format_version,\n        })\n    }\n\n    pub fn num_docs(&self) -> RowId {\n        self.num_docs\n    }\n    // Iterate over the columns in a sorted way\n    pub fn iter_columns(\n        &self,\n    ) -> io::Result<impl Iterator<Item = (String, DynamicColumnHandle)> + '_> {\n        let mut stream = self.column_dictionary.stream()?;\n        Ok(std::iter::from_fn(move || {\n            if stream.advance() {\n                let key_bytes: &[u8] = stream.key();\n                let column_code: u8 = key_bytes.last().cloned().unwrap();\n                // TODO Error Handling. The API gets quite ugly when returning the error here, so\n                // instead we could just check the first N columns upfront.\n                let column_type: ColumnType = ColumnType::try_from_code(column_code)\n                    .map_err(|_| io_invalid_data(format!(\"Unknown column code `{column_code}`\")))\n                    .unwrap();\n                let range = stream.value().clone();\n                let column_name =\n                // The last two bytes are respectively the 0u8 separator and the column_type.\n                String::from_utf8_lossy(&key_bytes[..key_bytes.len() - 2]).to_string();\n                let file_slice = self\n                    .column_data\n                    .slice(range.start as usize..range.end as usize);\n                let column_handle = DynamicColumnHandle {\n                    file_slice,\n                    column_type,\n                    format_version: self.format_version,\n                };\n                Some((column_name, column_handle))\n            } else {\n                None\n            }\n        }))\n    }\n\n    // TODO Add unit tests\n    pub fn list_columns(&self) -> io::Result<Vec<(String, DynamicColumnHandle)>> {\n        Ok(self.iter_columns()?.collect())\n    }\n\n    pub async fn read_columns_async(\n        &self,\n        column_name: &str,\n    ) -> io::Result<Vec<DynamicColumnHandle>> {\n        let prefix = column_dictionary_prefix_for_column_name(column_name);\n        let stream = self\n            .column_dictionary\n            .prefix_range(prefix)\n            .into_stream_async()\n            .await?;\n        read_all_columns_in_stream(stream, &self.column_data, self.format_version)\n    }\n\n    /// Get all columns for the given column name.\n    ///\n    /// There can be more than one column associated to a given column name, provided they have\n    /// different types.\n    pub fn read_columns(&self, column_name: &str) -> io::Result<Vec<DynamicColumnHandle>> {\n        let prefix = column_dictionary_prefix_for_column_name(column_name);\n        let stream = self.column_dictionary.prefix_range(prefix).into_stream()?;\n        read_all_columns_in_stream(stream, &self.column_data, self.format_version)\n    }\n\n    pub async fn read_subpath_columns_async(\n        &self,\n        root_path: &str,\n    ) -> io::Result<Vec<DynamicColumnHandle>> {\n        let prefix = column_dictionary_prefix_for_subpath(root_path);\n        let stream = self\n            .column_dictionary\n            .prefix_range(prefix)\n            .into_stream_async()\n            .await?;\n        read_all_columns_in_stream(stream, &self.column_data, self.format_version)\n    }\n\n    /// Get all inner columns for a given JSON prefix, i.e columns for which the name starts\n    /// with the prefix then contain the [`JSON_PATH_SEGMENT_SEP`].\n    ///\n    /// There can be more than one column associated to each path within the JSON structure,\n    /// provided they have different types.\n    pub fn read_subpath_columns(&self, root_path: &str) -> io::Result<Vec<DynamicColumnHandle>> {\n        let prefix = column_dictionary_prefix_for_subpath(root_path);\n        let stream = self\n            .column_dictionary\n            .prefix_range(prefix.as_bytes())\n            .into_stream()?;\n        read_all_columns_in_stream(stream, &self.column_data, self.format_version)\n    }\n\n    /// Return the number of columns in the columnar.\n    pub fn num_columns(&self) -> usize {\n        self.column_dictionary.num_terms()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use common::json_path_writer::JSON_PATH_SEGMENT_SEP;\n\n    use crate::{ColumnType, ColumnarReader, ColumnarWriter};\n\n    #[test]\n    fn test_list_columns() {\n        let mut columnar_writer = ColumnarWriter::default();\n        columnar_writer.record_column_type(\"col1\", ColumnType::Str, false);\n        columnar_writer.record_column_type(\"col2\", ColumnType::U64, false);\n        let mut buffer = Vec::new();\n        columnar_writer.serialize(1, &mut buffer).unwrap();\n        let columnar = ColumnarReader::open(buffer).unwrap();\n        let columns = columnar.list_columns().unwrap();\n        assert_eq!(columns.len(), 2);\n        assert_eq!(&columns[0].0, \"col1\");\n        assert_eq!(columns[0].1.column_type(), ColumnType::Str);\n        assert_eq!(&columns[1].0, \"col2\");\n        assert_eq!(columns[1].1.column_type(), ColumnType::U64);\n    }\n\n    #[test]\n    fn test_list_columns_strict_typing_prevents_coercion() {\n        let mut columnar_writer = ColumnarWriter::default();\n        columnar_writer.record_column_type(\"count\", ColumnType::U64, false);\n        columnar_writer.record_numerical(1, \"count\", 1u64);\n        let mut buffer = Vec::new();\n        columnar_writer.serialize(2, &mut buffer).unwrap();\n        let columnar = ColumnarReader::open(buffer).unwrap();\n        let columns = columnar.list_columns().unwrap();\n        assert_eq!(columns.len(), 1);\n        assert_eq!(&columns[0].0, \"count\");\n        assert_eq!(columns[0].1.column_type(), ColumnType::U64);\n    }\n\n    #[test]\n    fn test_read_columns() {\n        let mut columnar_writer = ColumnarWriter::default();\n        columnar_writer.record_column_type(\"col\", ColumnType::U64, false);\n        columnar_writer.record_numerical(1, \"col\", 1u64);\n        let mut buffer = Vec::new();\n        columnar_writer.serialize(2, &mut buffer).unwrap();\n        let columnar = ColumnarReader::open(buffer).unwrap();\n        {\n            let columns = columnar.read_columns(\"col\").unwrap();\n            assert_eq!(columns.len(), 1);\n            assert_eq!(columns[0].column_type(), ColumnType::U64);\n        }\n        {\n            let columns = columnar.read_columns(\"other\").unwrap();\n            assert_eq!(columns.len(), 0);\n        }\n    }\n\n    #[test]\n    fn test_read_subpath_columns() {\n        let mut columnar_writer = ColumnarWriter::default();\n        columnar_writer.record_str(\n            0,\n            &format!(\"col1{}subcol1\", JSON_PATH_SEGMENT_SEP as char),\n            \"hello\",\n        );\n        columnar_writer.record_numerical(\n            0,\n            &format!(\"col1{}subcol2\", JSON_PATH_SEGMENT_SEP as char),\n            1i64,\n        );\n        columnar_writer.record_str(1, \"col1\", \"hello\");\n        columnar_writer.record_str(0, \"col2\", \"hello\");\n        let mut buffer = Vec::new();\n        columnar_writer.serialize(2, &mut buffer).unwrap();\n\n        let columnar = ColumnarReader::open(buffer).unwrap();\n        {\n            let columns = columnar.read_subpath_columns(\"col1\").unwrap();\n            assert_eq!(columns.len(), 2);\n            assert_eq!(columns[0].column_type(), ColumnType::Str);\n            assert_eq!(columns[1].column_type(), ColumnType::I64);\n        }\n        {\n            let columns = columnar.read_subpath_columns(\"col1.subcol1\").unwrap();\n            assert_eq!(columns.len(), 0);\n        }\n        {\n            let columns = columnar.read_subpath_columns(\"col2\").unwrap();\n            assert_eq!(columns.len(), 0);\n        }\n        {\n            let columns = columnar.read_subpath_columns(\"other\").unwrap();\n            assert_eq!(columns.len(), 0);\n        }\n    }\n\n    #[test]\n    #[should_panic(expected = \"Input type forbidden\")]\n    fn test_list_columns_strict_typing_panics_on_wrong_types() {\n        let mut columnar_writer = ColumnarWriter::default();\n        columnar_writer.record_column_type(\"count\", ColumnType::U64, false);\n        columnar_writer.record_numerical(1, \"count\", 1i64);\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/writer/column_operation.rs",
    "content": "use std::net::Ipv6Addr;\n\nuse crate::dictionary::UnorderedId;\nuse crate::utils::{place_bits, pop_first_byte, select_bits};\nuse crate::value::NumericalValue;\nuse crate::{InvalidData, NumericalType, RowId};\n\n/// When we build a columnar dataframe, we first just group\n/// all mutations per column, and appends them in append-only buffer\n/// in the stacker.\n///\n/// These ColumnOperation<T> are therefore serialize/deserialized\n/// in memory.\n///\n/// We represents all of these operations as `ColumnOperation`.\n#[derive(Eq, PartialEq, Debug, Clone, Copy)]\npub(super) enum ColumnOperation<T> {\n    NewDoc(RowId),\n    Value(T),\n}\n\n#[derive(Copy, Clone, Eq, PartialEq, Debug)]\nstruct ColumnOperationMetadata {\n    op_type: ColumnOperationType,\n    len: u8,\n}\n\nimpl ColumnOperationMetadata {\n    fn to_code(self) -> u8 {\n        place_bits::<0, 6>(self.len) | place_bits::<6, 8>(self.op_type.to_code())\n    }\n\n    fn try_from_code(code: u8) -> Result<Self, InvalidData> {\n        let len = select_bits::<0, 6>(code);\n        let typ_code = select_bits::<6, 8>(code);\n        let column_type = ColumnOperationType::try_from_code(typ_code)?;\n        Ok(ColumnOperationMetadata {\n            op_type: column_type,\n            len,\n        })\n    }\n}\n\n#[derive(Copy, Clone, Eq, PartialEq, Debug)]\n#[repr(u8)]\nenum ColumnOperationType {\n    NewDoc = 0u8,\n    AddValue = 1u8,\n}\n\nimpl ColumnOperationType {\n    pub fn to_code(self) -> u8 {\n        self as u8\n    }\n\n    pub fn try_from_code(code: u8) -> Result<Self, InvalidData> {\n        match code {\n            0 => Ok(Self::NewDoc),\n            1 => Ok(Self::AddValue),\n            _ => Err(InvalidData),\n        }\n    }\n}\n\nimpl<V: SymbolValue> ColumnOperation<V> {\n    pub(super) fn serialize(self) -> impl AsRef<[u8]> {\n        let mut minibuf = MiniBuffer::default();\n        let column_op_metadata = match self {\n            ColumnOperation::NewDoc(new_doc) => {\n                let symbol_len = new_doc.serialize(&mut minibuf.bytes[1..]);\n                ColumnOperationMetadata {\n                    op_type: ColumnOperationType::NewDoc,\n                    len: symbol_len,\n                }\n            }\n            ColumnOperation::Value(val) => {\n                let symbol_len = val.serialize(&mut minibuf.bytes[1..]);\n                ColumnOperationMetadata {\n                    op_type: ColumnOperationType::AddValue,\n                    len: symbol_len,\n                }\n            }\n        };\n        minibuf.bytes[0] = column_op_metadata.to_code();\n        // +1 for the metadata\n        minibuf.len = 1 + column_op_metadata.len;\n        minibuf\n    }\n\n    /// Deserialize a column operation.\n    /// Returns None if the buffer is empty.\n    ///\n    /// Panics if the payload is invalid:\n    /// this deserialize method is meant to target in memory.\n    pub(super) fn deserialize(bytes: &mut &[u8]) -> Option<Self> {\n        let column_op_metadata_byte = pop_first_byte(bytes)?;\n        let column_op_metadata = ColumnOperationMetadata::try_from_code(column_op_metadata_byte)\n            .expect(\"Invalid op metadata byte\");\n        let symbol_bytes: &[u8];\n        (symbol_bytes, *bytes) = bytes.split_at(column_op_metadata.len as usize);\n        match column_op_metadata.op_type {\n            ColumnOperationType::NewDoc => {\n                let new_doc = u32::deserialize(symbol_bytes);\n                Some(ColumnOperation::NewDoc(new_doc))\n            }\n            ColumnOperationType::AddValue => {\n                let value = V::deserialize(symbol_bytes);\n                Some(ColumnOperation::Value(value))\n            }\n        }\n    }\n}\n\nimpl<T> From<T> for ColumnOperation<T> {\n    fn from(value: T) -> Self {\n        ColumnOperation::Value(value)\n    }\n}\n\n// Serialization trait very local to the writer.\n// As we write fast fields, we accumulate them in \"in memory\".\n// In order to limit memory usage, and in order\n// to benefit from the stacker, we do this by serialization our data\n// as \"Symbols\".\npub(super) trait SymbolValue: Clone + Copy {\n    // Serializes the symbol into the given buffer.\n    // Returns the number of bytes written into the buffer.\n    /// # Panics\n    /// May not exceed 9bytes\n    fn serialize(self, buffer: &mut [u8]) -> u8;\n    // Panics if invalid\n    fn deserialize(bytes: &[u8]) -> Self;\n}\n\nimpl SymbolValue for bool {\n    fn serialize(self, buffer: &mut [u8]) -> u8 {\n        buffer[0] = u8::from(self);\n        1u8\n    }\n\n    fn deserialize(bytes: &[u8]) -> Self {\n        bytes[0] == 1u8\n    }\n}\n\nimpl SymbolValue for Ipv6Addr {\n    fn serialize(self, buffer: &mut [u8]) -> u8 {\n        buffer[0..16].copy_from_slice(&self.octets());\n        16\n    }\n\n    fn deserialize(bytes: &[u8]) -> Self {\n        let octets: [u8; 16] = bytes[0..16].try_into().unwrap();\n        Ipv6Addr::from(octets)\n    }\n}\n\n#[derive(Default)]\nstruct MiniBuffer {\n    pub bytes: [u8; 17],\n    pub len: u8,\n}\n\nimpl AsRef<[u8]> for MiniBuffer {\n    fn as_ref(&self) -> &[u8] {\n        &self.bytes[..self.len as usize]\n    }\n}\n\nimpl SymbolValue for NumericalValue {\n    fn deserialize(mut bytes: &[u8]) -> Self {\n        let type_code = pop_first_byte(&mut bytes).unwrap();\n        let symbol_type = NumericalType::try_from_code(type_code).unwrap();\n        let mut octet: [u8; 8] = [0u8; 8];\n        octet[..bytes.len()].copy_from_slice(bytes);\n        match symbol_type {\n            NumericalType::U64 => {\n                let val: u64 = u64::from_le_bytes(octet);\n                NumericalValue::U64(val)\n            }\n            NumericalType::I64 => {\n                let encoded: u64 = u64::from_le_bytes(octet);\n                let val: i64 = decode_zig_zag(encoded);\n                NumericalValue::I64(val)\n            }\n            NumericalType::F64 => {\n                debug_assert_eq!(bytes.len(), 8);\n                let val: f64 = f64::from_le_bytes(octet);\n                NumericalValue::F64(val)\n            }\n        }\n    }\n\n    /// F64: Serialize with a fixed size of 9 bytes\n    /// U64: Serialize without leading zeroes\n    /// I64: ZigZag encoded and serialize without leading zeroes\n    fn serialize(self, output: &mut [u8]) -> u8 {\n        match self {\n            NumericalValue::F64(val) => {\n                output[0] = NumericalType::F64 as u8;\n                output[1..9].copy_from_slice(&val.to_le_bytes());\n                9u8\n            }\n            NumericalValue::U64(val) => {\n                let len = compute_num_bytes_for_u64(val) as u8;\n                output[0] = NumericalType::U64 as u8;\n                output[1..9].copy_from_slice(&val.to_le_bytes());\n                len + 1u8\n            }\n            NumericalValue::I64(val) => {\n                let zig_zag_encoded = encode_zig_zag(val);\n                let len = compute_num_bytes_for_u64(zig_zag_encoded) as u8;\n                output[0] = NumericalType::I64 as u8;\n                output[1..9].copy_from_slice(&zig_zag_encoded.to_le_bytes());\n                len + 1u8\n            }\n        }\n    }\n}\n\nimpl SymbolValue for u32 {\n    fn serialize(self, output: &mut [u8]) -> u8 {\n        let len = compute_num_bytes_for_u64(self as u64);\n        output[0..4].copy_from_slice(&self.to_le_bytes());\n        len as u8\n    }\n\n    fn deserialize(bytes: &[u8]) -> Self {\n        let mut quartet: [u8; 4] = [0u8; 4];\n        quartet[..bytes.len()].copy_from_slice(bytes);\n        u32::from_le_bytes(quartet)\n    }\n}\n\nimpl SymbolValue for UnorderedId {\n    fn serialize(self, output: &mut [u8]) -> u8 {\n        self.0.serialize(output)\n    }\n\n    fn deserialize(bytes: &[u8]) -> Self {\n        UnorderedId(u32::deserialize(bytes))\n    }\n}\n\nfn compute_num_bytes_for_u64(val: u64) -> usize {\n    let msb = (64u32 - val.leading_zeros()) as usize;\n    msb.div_ceil(8)\n}\n\nfn encode_zig_zag(n: i64) -> u64 {\n    ((n << 1) ^ (n >> 63)) as u64\n}\n\nfn decode_zig_zag(n: u64) -> i64 {\n    ((n >> 1) as i64) ^ (-((n & 1) as i64))\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[track_caller]\n    fn test_zig_zag_aux(val: i64) {\n        let encoded = super::encode_zig_zag(val);\n        assert_eq!(decode_zig_zag(encoded), val);\n        if let Some(abs_val) = val.checked_abs() {\n            let abs_val = abs_val as u64;\n            assert!(encoded <= abs_val * 2);\n        }\n    }\n\n    #[test]\n    fn test_zig_zag() {\n        assert_eq!(encode_zig_zag(0i64), 0u64);\n        assert_eq!(encode_zig_zag(-1i64), 1u64);\n        assert_eq!(encode_zig_zag(1i64), 2u64);\n        test_zig_zag_aux(0i64);\n        test_zig_zag_aux(i64::MIN);\n        test_zig_zag_aux(i64::MAX);\n    }\n\n    use proptest::prelude::any;\n    use proptest::proptest;\n\n    proptest! {\n        #[test]\n        fn test_proptest_zig_zag(val in any::<i64>()) {\n            test_zig_zag_aux(val);\n        }\n    }\n\n    #[test]\n    fn test_column_op_metadata_byte_serialization() {\n        for len in 0..=15 {\n            for op_type in [ColumnOperationType::AddValue, ColumnOperationType::NewDoc] {\n                let column_op_metadata = ColumnOperationMetadata { op_type, len };\n                let column_op_metadata_code = column_op_metadata.to_code();\n                let serdeser_metadata =\n                    ColumnOperationMetadata::try_from_code(column_op_metadata_code).unwrap();\n                assert_eq!(column_op_metadata, serdeser_metadata);\n            }\n        }\n    }\n\n    #[track_caller]\n    fn ser_deser_symbol(column_op: ColumnOperation<NumericalValue>) {\n        let buf = column_op.serialize();\n        let mut buffer = buf.as_ref().to_vec();\n        buffer.extend_from_slice(b\"234234\");\n        let mut bytes = &buffer[..];\n        let serdeser_symbol = ColumnOperation::deserialize(&mut bytes).unwrap();\n        assert_eq!(bytes.len() + buf.as_ref().len(), buffer.len());\n        assert_eq!(column_op, serdeser_symbol);\n    }\n\n    #[test]\n    fn test_compute_num_bytes_for_u64() {\n        assert_eq!(compute_num_bytes_for_u64(0), 0);\n        assert_eq!(compute_num_bytes_for_u64(1), 1);\n        assert_eq!(compute_num_bytes_for_u64(255), 1);\n        assert_eq!(compute_num_bytes_for_u64(256), 2);\n        assert_eq!(compute_num_bytes_for_u64((1 << 16) - 1), 2);\n        assert_eq!(compute_num_bytes_for_u64(1 << 16), 3);\n    }\n\n    #[test]\n    fn test_symbol_serialization() {\n        ser_deser_symbol(ColumnOperation::NewDoc(0));\n        ser_deser_symbol(ColumnOperation::NewDoc(3));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::I64(0i64)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::I64(1i64)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::U64(257u64)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::I64(-257i64)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::I64(i64::MIN)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::U64(0u64)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::U64(u64::MIN)));\n        ser_deser_symbol(ColumnOperation::Value(NumericalValue::U64(u64::MAX)));\n    }\n\n    fn test_column_operation_unordered_aux(val: u32, expected_len: usize) {\n        let column_op = ColumnOperation::Value(UnorderedId(val));\n        let minibuf = column_op.serialize();\n        assert_eq!({ minibuf.as_ref().len() }, expected_len);\n        let mut buf = minibuf.as_ref().to_vec();\n        buf.extend_from_slice(&[2, 2, 2, 2, 2, 2]);\n        let mut cursor = &buf[..];\n        let column_op_serdeser: ColumnOperation<UnorderedId> =\n            ColumnOperation::deserialize(&mut cursor).unwrap();\n        assert_eq!(column_op_serdeser, ColumnOperation::Value(UnorderedId(val)));\n        assert_eq!(cursor.len() + expected_len, buf.len());\n    }\n\n    #[test]\n    fn test_column_operation_unordered() {\n        test_column_operation_unordered_aux(300u32, 3);\n        test_column_operation_unordered_aux(1u32, 2);\n        test_column_operation_unordered_aux(0u32, 1);\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/writer/column_writers.rs",
    "content": "use std::cmp::Ordering;\n\nuse stacker::{ExpUnrolledLinkedList, MemoryArena};\n\nuse crate::columnar::writer::column_operation::{ColumnOperation, SymbolValue};\nuse crate::dictionary::{DictionaryBuilder, UnorderedId};\nuse crate::{Cardinality, NumericalType, NumericalValue, RowId};\n\n#[derive(Copy, Clone, Debug, Eq, PartialEq)]\n#[repr(u8)]\nenum DocumentStep {\n    Same = 0,\n    Next = 1,\n    Skipped = 2,\n}\n\n#[inline(always)]\nfn delta_with_last_doc(last_doc_opt: Option<u32>, doc: u32) -> DocumentStep {\n    let expected_next_doc = last_doc_opt.map(|last_doc| last_doc + 1).unwrap_or(0u32);\n    match doc.cmp(&expected_next_doc) {\n        Ordering::Less => DocumentStep::Same,\n        Ordering::Equal => DocumentStep::Next,\n        Ordering::Greater => DocumentStep::Skipped,\n    }\n}\n\n#[derive(Copy, Clone, Default)]\npub struct ColumnWriter {\n    // Detected cardinality of the column so far.\n    cardinality: Cardinality,\n    // Last document inserted.\n    // None if no doc has been added yet.\n    last_doc_opt: Option<u32>,\n    // Buffer containing the serialized values.\n    values: ExpUnrolledLinkedList,\n}\n\nimpl ColumnWriter {\n    /// Returns an iterator over the Symbol that have been recorded\n    /// for the given column.\n    pub(super) fn operation_iterator<'a, V: SymbolValue>(\n        &self,\n        arena: &MemoryArena,\n        buffer: &'a mut Vec<u8>,\n    ) -> impl Iterator<Item = ColumnOperation<V>> + 'a + use<'a, V> {\n        buffer.clear();\n        self.values.read_to_end(arena, buffer);\n        let mut cursor: &[u8] = &buffer[..];\n        std::iter::from_fn(move || ColumnOperation::deserialize(&mut cursor))\n    }\n\n    /// Records a change of the document being recorded.\n    ///\n    /// This function will also update the cardinality of the column\n    /// if necessary.\n    pub(super) fn record<S: SymbolValue>(&mut self, doc: RowId, value: S, arena: &mut MemoryArena) {\n        // Difference between `doc` and the last doc.\n        match delta_with_last_doc(self.last_doc_opt, doc) {\n            DocumentStep::Same => {\n                // This is the last encounterred document.\n                self.cardinality = Cardinality::Multivalued;\n            }\n            DocumentStep::Next => {\n                self.last_doc_opt = Some(doc);\n                self.write_symbol::<S>(ColumnOperation::NewDoc(doc), arena);\n            }\n            DocumentStep::Skipped => {\n                self.cardinality = self.cardinality.max(Cardinality::Optional);\n                self.last_doc_opt = Some(doc);\n                self.write_symbol::<S>(ColumnOperation::NewDoc(doc), arena);\n            }\n        }\n        self.write_symbol(ColumnOperation::Value(value), arena);\n    }\n\n    // Get the cardinality.\n    // The overall number of docs in the column is necessary to\n    // deal with the case where the all docs contain 1 value, except some documents\n    // at the end of the column.\n    pub(crate) fn get_cardinality(&self, num_docs: RowId) -> Cardinality {\n        match delta_with_last_doc(self.last_doc_opt, num_docs) {\n            DocumentStep::Same | DocumentStep::Next => self.cardinality,\n            DocumentStep::Skipped => self.cardinality.max(Cardinality::Optional),\n        }\n    }\n\n    /// Appends a new symbol to the `ColumnWriter`.\n    fn write_symbol<V: SymbolValue>(\n        &mut self,\n        column_operation: ColumnOperation<V>,\n        arena: &mut MemoryArena,\n    ) {\n        self.values\n            .writer(arena)\n            .extend_from_slice(column_operation.serialize().as_ref());\n    }\n}\n\n#[derive(Clone, Copy, Default)]\npub(crate) struct NumericalColumnWriter {\n    compatible_numerical_types: CompatibleNumericalTypes,\n    column_writer: ColumnWriter,\n}\n\nimpl NumericalColumnWriter {\n    pub fn force_numerical_type(&mut self, numerical_type: NumericalType) {\n        assert!(\n            self.compatible_numerical_types\n                .is_type_accepted(numerical_type)\n        );\n        self.compatible_numerical_types = CompatibleNumericalTypes::StaticType(numerical_type);\n    }\n}\n\n/// State used to store what types are still acceptable\n/// after having seen a set of numerical values.\n#[derive(Clone, Copy)]\npub(crate) enum CompatibleNumericalTypes {\n    Dynamic {\n        all_values_within_i64_range: bool,\n        all_values_within_u64_range: bool,\n    },\n    StaticType(NumericalType),\n}\n\nimpl Default for CompatibleNumericalTypes {\n    fn default() -> CompatibleNumericalTypes {\n        CompatibleNumericalTypes::Dynamic {\n            all_values_within_i64_range: true,\n            all_values_within_u64_range: true,\n        }\n    }\n}\n\nimpl CompatibleNumericalTypes {\n    pub fn is_type_accepted(&self, numerical_type: NumericalType) -> bool {\n        match self {\n            CompatibleNumericalTypes::Dynamic {\n                all_values_within_i64_range,\n                all_values_within_u64_range,\n            } => match numerical_type {\n                NumericalType::I64 => *all_values_within_i64_range,\n                NumericalType::U64 => *all_values_within_u64_range,\n                NumericalType::F64 => true,\n            },\n            CompatibleNumericalTypes::StaticType(static_numerical_type) => {\n                *static_numerical_type == numerical_type\n            }\n        }\n    }\n\n    pub fn accept_value(&mut self, numerical_value: NumericalValue) {\n        match self {\n            CompatibleNumericalTypes::Dynamic {\n                all_values_within_i64_range,\n                all_values_within_u64_range,\n            } => match numerical_value {\n                NumericalValue::I64(val_i64) => {\n                    let value_within_u64_range = val_i64 >= 0i64;\n                    *all_values_within_u64_range &= value_within_u64_range;\n                }\n                NumericalValue::U64(val_u64) => {\n                    let value_within_i64_range = val_u64 < i64::MAX as u64;\n                    *all_values_within_i64_range &= value_within_i64_range;\n                }\n                NumericalValue::F64(_) => {\n                    *all_values_within_i64_range = false;\n                    *all_values_within_u64_range = false;\n                }\n            },\n            CompatibleNumericalTypes::StaticType(typ) => {\n                assert_eq!(\n                    numerical_value.numerical_type(),\n                    *typ,\n                    \"Input type forbidden. This column has been forced to type {typ:?}, received \\\n                     {numerical_value:?}\"\n                );\n            }\n        }\n    }\n\n    pub fn to_numerical_type(self) -> NumericalType {\n        for numerical_type in [NumericalType::I64, NumericalType::U64] {\n            if self.is_type_accepted(numerical_type) {\n                return numerical_type;\n            }\n        }\n        NumericalType::F64\n    }\n}\n\nimpl NumericalColumnWriter {\n    pub fn numerical_type(&self) -> NumericalType {\n        self.compatible_numerical_types.to_numerical_type()\n    }\n\n    pub fn cardinality(&self, num_docs: RowId) -> Cardinality {\n        self.column_writer.get_cardinality(num_docs)\n    }\n\n    pub fn record_numerical_value(\n        &mut self,\n        doc: RowId,\n        value: NumericalValue,\n        arena: &mut MemoryArena,\n    ) {\n        self.compatible_numerical_types.accept_value(value);\n        self.column_writer.record(doc, value, arena);\n    }\n\n    pub(super) fn operation_iterator<'a>(\n        self,\n        arena: &MemoryArena,\n        buffer: &'a mut Vec<u8>,\n    ) -> impl Iterator<Item = ColumnOperation<NumericalValue>> + 'a + use<'a> {\n        self.column_writer.operation_iterator(arena, buffer)\n    }\n}\n\n#[derive(Copy, Clone)]\npub(crate) struct StrOrBytesColumnWriter {\n    pub(crate) dictionary_id: u32,\n    pub(crate) column_writer: ColumnWriter,\n    // If true, when facing a multivalued cardinality,\n    // values associated to a given document will be sorted.\n    //\n    // This is useful for facets.\n    //\n    // If false, the order of appearance in the document will be\n    // observed.\n    pub(crate) sort_values_within_row: bool,\n}\n\nimpl StrOrBytesColumnWriter {\n    pub(crate) fn with_dictionary_id(dictionary_id: u32) -> StrOrBytesColumnWriter {\n        StrOrBytesColumnWriter {\n            dictionary_id,\n            column_writer: Default::default(),\n            sort_values_within_row: false,\n        }\n    }\n\n    pub(crate) fn record_bytes(\n        &mut self,\n        doc: RowId,\n        bytes: &[u8],\n        dictionaries: &mut [DictionaryBuilder],\n        arena: &mut MemoryArena,\n    ) {\n        let unordered_id =\n            dictionaries[self.dictionary_id as usize].get_or_allocate_id(bytes, arena);\n        self.column_writer.record(doc, unordered_id, arena);\n    }\n\n    pub(super) fn operation_iterator<'a>(\n        &self,\n        arena: &MemoryArena,\n        byte_buffer: &'a mut Vec<u8>,\n    ) -> impl Iterator<Item = ColumnOperation<UnorderedId>> + 'a + use<'a> {\n        self.column_writer.operation_iterator(arena, byte_buffer)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_delta_with_last_doc() {\n        assert_eq!(delta_with_last_doc(None, 0u32), DocumentStep::Next);\n        assert_eq!(delta_with_last_doc(None, 1u32), DocumentStep::Skipped);\n        assert_eq!(delta_with_last_doc(None, 2u32), DocumentStep::Skipped);\n        assert_eq!(delta_with_last_doc(Some(0u32), 0u32), DocumentStep::Same);\n        assert_eq!(delta_with_last_doc(Some(1u32), 1u32), DocumentStep::Same);\n        assert_eq!(delta_with_last_doc(Some(1u32), 2u32), DocumentStep::Next);\n        assert_eq!(delta_with_last_doc(Some(1u32), 3u32), DocumentStep::Skipped);\n        assert_eq!(delta_with_last_doc(Some(1u32), 4u32), DocumentStep::Skipped);\n    }\n\n    #[track_caller]\n    fn test_column_writer_coercion_iter_aux(\n        values: impl Iterator<Item = NumericalValue>,\n        expected_numerical_type: NumericalType,\n    ) {\n        let mut compatible_numerical_types = CompatibleNumericalTypes::default();\n        for value in values {\n            compatible_numerical_types.accept_value(value);\n        }\n        assert_eq!(\n            compatible_numerical_types.to_numerical_type(),\n            expected_numerical_type\n        );\n    }\n\n    #[track_caller]\n    fn test_column_writer_coercion_aux(\n        values: &[NumericalValue],\n        expected_numerical_type: NumericalType,\n    ) {\n        test_column_writer_coercion_iter_aux(values.iter().copied(), expected_numerical_type);\n        test_column_writer_coercion_iter_aux(values.iter().rev().copied(), expected_numerical_type);\n    }\n\n    #[test]\n    fn test_column_writer_coercion() {\n        test_column_writer_coercion_aux(&[], NumericalType::I64);\n        test_column_writer_coercion_aux(&[1i64.into()], NumericalType::I64);\n        test_column_writer_coercion_aux(&[1u64.into()], NumericalType::I64);\n        // We don't detect exact integer at the moment. We could!\n        test_column_writer_coercion_aux(&[1f64.into()], NumericalType::F64);\n        test_column_writer_coercion_aux(&[u64::MAX.into()], NumericalType::U64);\n        test_column_writer_coercion_aux(&[(i64::MAX as u64).into()], NumericalType::U64);\n        test_column_writer_coercion_aux(&[(1u64 << 63).into()], NumericalType::U64);\n        test_column_writer_coercion_aux(&[1i64.into(), 1u64.into()], NumericalType::I64);\n        test_column_writer_coercion_aux(&[u64::MAX.into(), (-1i64).into()], NumericalType::F64);\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_compatible_numerical_types_static_incompatible_type() {\n        let mut compatible_numerical_types =\n            CompatibleNumericalTypes::StaticType(NumericalType::U64);\n        compatible_numerical_types.accept_value(NumericalValue::I64(1i64));\n    }\n\n    #[test]\n    fn test_compatible_numerical_types_static_different_type_forbidden() {\n        let mut compatible_numerical_types =\n            CompatibleNumericalTypes::StaticType(NumericalType::U64);\n        compatible_numerical_types.accept_value(NumericalValue::U64(u64::MAX));\n    }\n\n    #[test]\n    fn test_compatible_numerical_types_static() {\n        for typ in [NumericalType::I64, NumericalType::I64, NumericalType::F64] {\n            let compatible_numerical_types = CompatibleNumericalTypes::StaticType(typ);\n            assert_eq!(compatible_numerical_types.to_numerical_type(), typ);\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/writer/mod.rs",
    "content": "mod column_operation;\nmod column_writers;\nmod serializer;\nmod value_index;\n\nuse std::io;\nuse std::net::Ipv6Addr;\n\nuse column_operation::ColumnOperation;\npub(crate) use column_writers::CompatibleNumericalTypes;\nuse common::CountingWriter;\nuse common::json_path_writer::JSON_END_OF_PATH;\npub(crate) use serializer::ColumnarSerializer;\nuse stacker::{Addr, ArenaHashMap, MemoryArena};\n\nuse crate::column_index::{SerializableColumnIndex, SerializableOptionalIndex};\nuse crate::column_values::{MonotonicallyMappableToU64, MonotonicallyMappableToU128};\nuse crate::columnar::column_type::ColumnType;\nuse crate::columnar::writer::column_writers::{\n    ColumnWriter, NumericalColumnWriter, StrOrBytesColumnWriter,\n};\nuse crate::columnar::writer::value_index::{IndexBuilder, PreallocatedIndexBuilders};\nuse crate::dictionary::{DictionaryBuilder, TermIdMapping, UnorderedId};\nuse crate::value::{Coerce, NumericalType, NumericalValue};\nuse crate::{Cardinality, RowId};\n\n/// This is a set of buffers that are used to temporarily write the values into before passing them\n/// to the fast field codecs.\n#[derive(Default)]\nstruct SpareBuffers {\n    value_index_builders: PreallocatedIndexBuilders,\n    u64_values: Vec<u64>,\n    ip_addr_values: Vec<Ipv6Addr>,\n}\n\n/// Makes it possible to create a new columnar.\n///\n/// ```rust\n/// use tantivy_columnar::ColumnarWriter;\n///\n/// let mut columnar_writer = ColumnarWriter::default();\n/// columnar_writer.record_str(0u32 /* doc id */, \"product_name\", \"Red backpack\");\n/// columnar_writer.record_numerical(0u32 /* doc id */, \"price\", 10u64);\n/// columnar_writer.record_str(1u32 /* doc id */, \"product_name\", \"Apple\");\n/// columnar_writer.record_numerical(0u32 /* doc id */, \"price\", 10.5f64); //< uh oh we ended up mixing integer and floats.\n/// let mut wrt: Vec<u8> =  Vec::new();\n/// columnar_writer.serialize(2u32, &mut wrt).unwrap();\n/// ```\n#[derive(Default)]\npub struct ColumnarWriter {\n    numerical_field_hash_map: ArenaHashMap,\n    datetime_field_hash_map: ArenaHashMap,\n    bool_field_hash_map: ArenaHashMap,\n    ip_addr_field_hash_map: ArenaHashMap,\n    bytes_field_hash_map: ArenaHashMap,\n    str_field_hash_map: ArenaHashMap,\n    arena: MemoryArena,\n    // Dictionaries used to store dictionary-encoded values.\n    dictionaries: Vec<DictionaryBuilder>,\n    buffers: SpareBuffers,\n}\n\nimpl ColumnarWriter {\n    pub fn mem_usage(&self) -> usize {\n        self.arena.mem_usage()\n            + self.numerical_field_hash_map.mem_usage()\n            + self.bool_field_hash_map.mem_usage()\n            + self.bytes_field_hash_map.mem_usage()\n            + self.str_field_hash_map.mem_usage()\n            + self.ip_addr_field_hash_map.mem_usage()\n            + self.datetime_field_hash_map.mem_usage()\n            + self\n                .dictionaries\n                .iter()\n                .map(|dict| dict.mem_usage())\n                .sum::<usize>()\n    }\n\n    /// Records a column type. This is useful to bypass the coercion process,\n    /// makes sure the empty is present in the resulting columnar, or set\n    /// the `sort_values_within_row`.\n    ///\n    /// `sort_values_within_row` is only allowed for `Bytes` or `Str` columns.\n    pub fn record_column_type(\n        &mut self,\n        column_name: &str,\n        column_type: ColumnType,\n        sort_values_within_row: bool,\n    ) {\n        if sort_values_within_row {\n            assert!(\n                column_type == ColumnType::Bytes || column_type == ColumnType::Str,\n                \"sort_values_within_row is only allowed for Bytes and Str columns\",\n            );\n        }\n        match column_type {\n            ColumnType::Str | ColumnType::Bytes => {\n                let (hash_map, dictionaries) = (\n                    if column_type == ColumnType::Str {\n                        &mut self.str_field_hash_map\n                    } else {\n                        &mut self.bytes_field_hash_map\n                    },\n                    &mut self.dictionaries,\n                );\n                hash_map.mutate_or_create(\n                    column_name.as_bytes(),\n                    |column_opt: Option<StrOrBytesColumnWriter>| {\n                        let mut column_writer = if let Some(column_writer) = column_opt {\n                            column_writer\n                        } else {\n                            let dictionary_id = dictionaries.len() as u32;\n                            dictionaries.push(DictionaryBuilder::default());\n                            StrOrBytesColumnWriter::with_dictionary_id(dictionary_id)\n                        };\n                        column_writer.sort_values_within_row = sort_values_within_row;\n                        column_writer\n                    },\n                );\n            }\n            ColumnType::Bool => {\n                self.bool_field_hash_map.mutate_or_create(\n                    column_name.as_bytes(),\n                    |column_opt: Option<ColumnWriter>| column_opt.unwrap_or_default(),\n                );\n            }\n            ColumnType::DateTime => {\n                self.datetime_field_hash_map.mutate_or_create(\n                    column_name.as_bytes(),\n                    |column_opt: Option<ColumnWriter>| column_opt.unwrap_or_default(),\n                );\n            }\n            ColumnType::I64 | ColumnType::F64 | ColumnType::U64 => {\n                let numerical_type = column_type.numerical_type().unwrap();\n                self.numerical_field_hash_map.mutate_or_create(\n                    column_name.as_bytes(),\n                    |column_opt: Option<NumericalColumnWriter>| {\n                        let mut column: NumericalColumnWriter = column_opt.unwrap_or_default();\n                        column.force_numerical_type(numerical_type);\n                        column\n                    },\n                );\n            }\n            ColumnType::IpAddr => self.ip_addr_field_hash_map.mutate_or_create(\n                column_name.as_bytes(),\n                |column_opt: Option<ColumnWriter>| column_opt.unwrap_or_default(),\n            ),\n        }\n    }\n\n    pub fn record_numerical<T: Into<NumericalValue> + Copy>(\n        &mut self,\n        doc: RowId,\n        column_name: &str,\n        numerical_value: T,\n    ) {\n        let (hash_map, arena) = (&mut self.numerical_field_hash_map, &mut self.arena);\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<NumericalColumnWriter>| {\n                let mut column: NumericalColumnWriter = column_opt.unwrap_or_default();\n                column.record_numerical_value(doc, numerical_value.into(), arena);\n                column\n            },\n        );\n    }\n\n    pub fn record_ip_addr(&mut self, doc: RowId, column_name: &str, ip_addr: Ipv6Addr) {\n        let (hash_map, arena) = (&mut self.ip_addr_field_hash_map, &mut self.arena);\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<ColumnWriter>| {\n                let mut column: ColumnWriter = column_opt.unwrap_or_default();\n                column.record(doc, ip_addr, arena);\n                column\n            },\n        );\n    }\n\n    pub fn record_bool(&mut self, doc: RowId, column_name: &str, val: bool) {\n        let (hash_map, arena) = (&mut self.bool_field_hash_map, &mut self.arena);\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<ColumnWriter>| {\n                let mut column: ColumnWriter = column_opt.unwrap_or_default();\n                column.record(doc, val, arena);\n                column\n            },\n        );\n    }\n\n    pub fn record_datetime(&mut self, doc: RowId, column_name: &str, datetime: common::DateTime) {\n        let (hash_map, arena) = (&mut self.datetime_field_hash_map, &mut self.arena);\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<ColumnWriter>| {\n                let mut column: ColumnWriter = column_opt.unwrap_or_default();\n                column.record(\n                    doc,\n                    NumericalValue::I64(datetime.into_timestamp_nanos()),\n                    arena,\n                );\n                column\n            },\n        );\n    }\n\n    pub fn record_str(&mut self, doc: RowId, column_name: &str, value: &str) {\n        let (hash_map, arena, dictionaries) = (\n            &mut self.str_field_hash_map,\n            &mut self.arena,\n            &mut self.dictionaries,\n        );\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<StrOrBytesColumnWriter>| {\n                let mut column: StrOrBytesColumnWriter = column_opt.unwrap_or_else(|| {\n                    // Each column has its own dictionary\n                    let dictionary_id = dictionaries.len() as u32;\n                    dictionaries.push(DictionaryBuilder::default());\n                    StrOrBytesColumnWriter::with_dictionary_id(dictionary_id)\n                });\n                column.record_bytes(doc, value.as_bytes(), dictionaries, arena);\n                column\n            },\n        );\n    }\n\n    pub fn record_bytes(&mut self, doc: RowId, column_name: &str, value: &[u8]) {\n        let (hash_map, arena, dictionaries) = (\n            &mut self.bytes_field_hash_map,\n            &mut self.arena,\n            &mut self.dictionaries,\n        );\n        hash_map.mutate_or_create(\n            column_name.as_bytes(),\n            |column_opt: Option<StrOrBytesColumnWriter>| {\n                let mut column: StrOrBytesColumnWriter = column_opt.unwrap_or_else(|| {\n                    // Each column has its own dictionary\n                    let dictionary_id = dictionaries.len() as u32;\n                    dictionaries.push(DictionaryBuilder::default());\n                    StrOrBytesColumnWriter::with_dictionary_id(dictionary_id)\n                });\n                column.record_bytes(doc, value, dictionaries, arena);\n                column\n            },\n        );\n    }\n    pub fn serialize(&mut self, num_docs: RowId, wrt: &mut dyn io::Write) -> io::Result<()> {\n        let mut serializer = ColumnarSerializer::new(wrt);\n\n        let mut columns: Vec<(&[u8], ColumnType, Addr)> = self\n            .numerical_field_hash_map\n            .iter()\n            .map(|(column_name, addr)| {\n                let numerical_column_writer: NumericalColumnWriter =\n                    self.numerical_field_hash_map.read(addr);\n                let column_type = numerical_column_writer.numerical_type().into();\n                (column_name, column_type, addr)\n            })\n            .collect();\n        columns.extend(\n            self.bytes_field_hash_map\n                .iter()\n                .map(|(column_name, addr)| (column_name, ColumnType::Bytes, addr)),\n        );\n        columns.extend(\n            self.str_field_hash_map\n                .iter()\n                .map(|(column_name, addr)| (column_name, ColumnType::Str, addr)),\n        );\n        columns.extend(\n            self.bool_field_hash_map\n                .iter()\n                .map(|(column_name, addr)| (column_name, ColumnType::Bool, addr)),\n        );\n        columns.extend(\n            self.ip_addr_field_hash_map\n                .iter()\n                .map(|(column_name, addr)| (column_name, ColumnType::IpAddr, addr)),\n        );\n        columns.extend(\n            self.datetime_field_hash_map\n                .iter()\n                .map(|(column_name, addr)| (column_name, ColumnType::DateTime, addr)),\n        );\n        columns.sort_unstable_by_key(|(column_name, col_type, _)| (*column_name, *col_type));\n        let (arena, buffers, dictionaries) = (&self.arena, &mut self.buffers, &self.dictionaries);\n        let mut symbol_byte_buffer: Vec<u8> = Vec::new();\n        for (column_name, column_type, addr) in columns {\n            if column_name.contains(&JSON_END_OF_PATH) {\n                // Tantivy uses b'0' as a separator for nested fields in JSON.\n                // Column names with a b'0' are not simply ignored by the columnar (and the inverted\n                // index).\n                continue;\n            }\n            match column_type {\n                ColumnType::Bool => {\n                    let column_writer: ColumnWriter = self.bool_field_hash_map.read(addr);\n                    let cardinality = column_writer.get_cardinality(num_docs);\n                    let mut column_serializer =\n                        serializer.start_serialize_column(column_name, column_type);\n                    serialize_bool_column(\n                        cardinality,\n                        num_docs,\n                        column_writer.operation_iterator(arena, &mut symbol_byte_buffer),\n                        buffers,\n                        &mut column_serializer,\n                    )?;\n                    column_serializer.finalize()?;\n                }\n                ColumnType::IpAddr => {\n                    let column_writer: ColumnWriter = self.ip_addr_field_hash_map.read(addr);\n                    let cardinality = column_writer.get_cardinality(num_docs);\n                    let mut column_serializer =\n                        serializer.start_serialize_column(column_name, ColumnType::IpAddr);\n                    serialize_ip_addr_column(\n                        cardinality,\n                        num_docs,\n                        column_writer.operation_iterator(arena, &mut symbol_byte_buffer),\n                        buffers,\n                        &mut column_serializer,\n                    )?;\n                    column_serializer.finalize()?;\n                }\n                ColumnType::Bytes | ColumnType::Str => {\n                    let str_or_bytes_column_writer: StrOrBytesColumnWriter =\n                        if column_type == ColumnType::Bytes {\n                            self.bytes_field_hash_map.read(addr)\n                        } else {\n                            self.str_field_hash_map.read(addr)\n                        };\n                    let dictionary_builder =\n                        &dictionaries[str_or_bytes_column_writer.dictionary_id as usize];\n                    let cardinality = str_or_bytes_column_writer\n                        .column_writer\n                        .get_cardinality(num_docs);\n                    let mut column_serializer =\n                        serializer.start_serialize_column(column_name, column_type);\n                    serialize_bytes_or_str_column(\n                        cardinality,\n                        num_docs,\n                        str_or_bytes_column_writer.sort_values_within_row,\n                        dictionary_builder,\n                        str_or_bytes_column_writer\n                            .operation_iterator(arena, &mut symbol_byte_buffer),\n                        buffers,\n                        &self.arena,\n                        &mut column_serializer,\n                    )?;\n                    column_serializer.finalize()?;\n                }\n                ColumnType::F64 | ColumnType::I64 | ColumnType::U64 => {\n                    let numerical_column_writer: NumericalColumnWriter =\n                        self.numerical_field_hash_map.read(addr);\n                    let cardinality = numerical_column_writer.cardinality(num_docs);\n                    let mut column_serializer =\n                        serializer.start_serialize_column(column_name, column_type);\n                    let numerical_type = column_type.numerical_type().unwrap();\n                    serialize_numerical_column(\n                        cardinality,\n                        num_docs,\n                        numerical_type,\n                        numerical_column_writer.operation_iterator(arena, &mut symbol_byte_buffer),\n                        buffers,\n                        &mut column_serializer,\n                    )?;\n                    column_serializer.finalize()?;\n                }\n                ColumnType::DateTime => {\n                    let column_writer: ColumnWriter = self.datetime_field_hash_map.read(addr);\n                    let cardinality = column_writer.get_cardinality(num_docs);\n                    let mut column_serializer =\n                        serializer.start_serialize_column(column_name, ColumnType::DateTime);\n                    serialize_numerical_column(\n                        cardinality,\n                        num_docs,\n                        NumericalType::I64,\n                        column_writer.operation_iterator(arena, &mut symbol_byte_buffer),\n                        buffers,\n                        &mut column_serializer,\n                    )?;\n                    column_serializer.finalize()?;\n                }\n            };\n        }\n        serializer.finalize(num_docs)?;\n        Ok(())\n    }\n}\n\n// Serialize [Dictionary, Column, dictionary num bytes U32::LE]\n// Column: [Column Index, Column Values, column index num bytes U32::LE]\n#[expect(clippy::too_many_arguments)]\nfn serialize_bytes_or_str_column(\n    cardinality: Cardinality,\n    num_docs: RowId,\n    sort_values_within_row: bool,\n    dictionary_builder: &DictionaryBuilder,\n    operation_it: impl Iterator<Item = ColumnOperation<UnorderedId>>,\n    buffers: &mut SpareBuffers,\n    arena: &MemoryArena,\n    wrt: impl io::Write,\n) -> io::Result<()> {\n    let SpareBuffers {\n        value_index_builders,\n        u64_values,\n        ..\n    } = buffers;\n    let mut counting_writer = CountingWriter::wrap(wrt);\n    let term_id_mapping: TermIdMapping =\n        dictionary_builder.serialize(arena, &mut counting_writer)?;\n    let dictionary_num_bytes: u32 = counting_writer.written_bytes() as u32;\n    let mut wrt = counting_writer.finish();\n    let operation_iterator = operation_it.map(|symbol: ColumnOperation<UnorderedId>| {\n        // We map unordered ids to ordered ids.\n        match symbol {\n            ColumnOperation::Value(unordered_id) => {\n                let ordered_id = term_id_mapping.to_ord(unordered_id);\n                ColumnOperation::Value(ordered_id.0 as u64)\n            }\n            ColumnOperation::NewDoc(doc) => ColumnOperation::NewDoc(doc),\n        }\n    });\n    send_to_serialize_column_mappable_to_u64(\n        operation_iterator,\n        cardinality,\n        num_docs,\n        sort_values_within_row,\n        value_index_builders,\n        u64_values,\n        &mut wrt,\n    )?;\n    wrt.write_all(&dictionary_num_bytes.to_le_bytes()[..])?;\n    Ok(())\n}\n\nfn serialize_numerical_column(\n    cardinality: Cardinality,\n    num_docs: RowId,\n    numerical_type: NumericalType,\n    op_iterator: impl Iterator<Item = ColumnOperation<NumericalValue>>,\n    buffers: &mut SpareBuffers,\n    wrt: &mut impl io::Write,\n) -> io::Result<()> {\n    let SpareBuffers {\n        value_index_builders,\n        u64_values,\n        ..\n    } = buffers;\n    match numerical_type {\n        NumericalType::I64 => {\n            send_to_serialize_column_mappable_to_u64(\n                coerce_numerical_symbol::<i64>(op_iterator),\n                cardinality,\n                num_docs,\n                false,\n                value_index_builders,\n                u64_values,\n                wrt,\n            )?;\n        }\n        NumericalType::U64 => {\n            send_to_serialize_column_mappable_to_u64(\n                coerce_numerical_symbol::<u64>(op_iterator),\n                cardinality,\n                num_docs,\n                false,\n                value_index_builders,\n                u64_values,\n                wrt,\n            )?;\n        }\n        NumericalType::F64 => {\n            send_to_serialize_column_mappable_to_u64(\n                coerce_numerical_symbol::<f64>(op_iterator),\n                cardinality,\n                num_docs,\n                false,\n                value_index_builders,\n                u64_values,\n                wrt,\n            )?;\n        }\n    };\n    Ok(())\n}\n\nfn serialize_bool_column(\n    cardinality: Cardinality,\n    num_docs: RowId,\n    column_operations_it: impl Iterator<Item = ColumnOperation<bool>>,\n    buffers: &mut SpareBuffers,\n    wrt: &mut impl io::Write,\n) -> io::Result<()> {\n    let SpareBuffers {\n        value_index_builders,\n        u64_values,\n        ..\n    } = buffers;\n    send_to_serialize_column_mappable_to_u64(\n        column_operations_it.map(|bool_column_operation| match bool_column_operation {\n            ColumnOperation::NewDoc(doc) => ColumnOperation::NewDoc(doc),\n            ColumnOperation::Value(bool_val) => ColumnOperation::Value(bool_val.to_u64()),\n        }),\n        cardinality,\n        num_docs,\n        false,\n        value_index_builders,\n        u64_values,\n        wrt,\n    )?;\n    Ok(())\n}\n\nfn serialize_ip_addr_column(\n    cardinality: Cardinality,\n    num_docs: RowId,\n    column_operations_it: impl Iterator<Item = ColumnOperation<Ipv6Addr>>,\n    buffers: &mut SpareBuffers,\n    wrt: &mut impl io::Write,\n) -> io::Result<()> {\n    let SpareBuffers {\n        value_index_builders,\n        ip_addr_values,\n        ..\n    } = buffers;\n    send_to_serialize_column_mappable_to_u128(\n        column_operations_it,\n        cardinality,\n        num_docs,\n        value_index_builders,\n        ip_addr_values,\n        wrt,\n    )?;\n    Ok(())\n}\n\nfn send_to_serialize_column_mappable_to_u128<\n    T: Copy + Ord + std::fmt::Debug + Send + Sync + MonotonicallyMappableToU128 + PartialOrd,\n>(\n    op_iterator: impl Iterator<Item = ColumnOperation<T>>,\n    cardinality: Cardinality,\n    num_rows: RowId,\n    value_index_builders: &mut PreallocatedIndexBuilders,\n    values: &mut Vec<T>,\n    mut wrt: impl io::Write,\n) -> io::Result<()> {\n    values.clear();\n    // TODO: split index and values\n    let serializable_column_index = match cardinality {\n        Cardinality::Full => {\n            consume_operation_iterator(\n                op_iterator,\n                value_index_builders.borrow_required_index_builder(),\n                values,\n            );\n            SerializableColumnIndex::Full\n        }\n        Cardinality::Optional => {\n            let optional_index_builder = value_index_builders.borrow_optional_index_builder();\n            consume_operation_iterator(op_iterator, optional_index_builder, values);\n            let optional_index = optional_index_builder.finish(num_rows);\n            SerializableColumnIndex::Optional(SerializableOptionalIndex {\n                num_rows,\n                non_null_row_ids: Box::new(optional_index),\n            })\n        }\n        Cardinality::Multivalued => {\n            let multivalued_index_builder = value_index_builders.borrow_multivalued_index_builder();\n            consume_operation_iterator(op_iterator, multivalued_index_builder, values);\n            let serializable_multivalued_index = multivalued_index_builder.finish(num_rows);\n            SerializableColumnIndex::Multivalued(serializable_multivalued_index)\n        }\n    };\n    crate::column::serialize_column_mappable_to_u128(\n        serializable_column_index,\n        &&values[..],\n        &mut wrt,\n    )?;\n    Ok(())\n}\n\nfn send_to_serialize_column_mappable_to_u64(\n    op_iterator: impl Iterator<Item = ColumnOperation<u64>>,\n    cardinality: Cardinality,\n    num_rows: RowId,\n    sort_values_within_row: bool,\n    value_index_builders: &mut PreallocatedIndexBuilders,\n    values: &mut Vec<u64>,\n    mut wrt: impl io::Write,\n) -> io::Result<()> {\n    values.clear();\n    let serializable_column_index = match cardinality {\n        Cardinality::Full => {\n            consume_operation_iterator(\n                op_iterator,\n                value_index_builders.borrow_required_index_builder(),\n                values,\n            );\n            SerializableColumnIndex::Full\n        }\n        Cardinality::Optional => {\n            let optional_index_builder = value_index_builders.borrow_optional_index_builder();\n            consume_operation_iterator(op_iterator, optional_index_builder, values);\n            let optional_index = optional_index_builder.finish(num_rows);\n            SerializableColumnIndex::Optional(SerializableOptionalIndex {\n                non_null_row_ids: Box::new(optional_index),\n                num_rows,\n            })\n        }\n        Cardinality::Multivalued => {\n            let multivalued_index_builder = value_index_builders.borrow_multivalued_index_builder();\n            consume_operation_iterator(op_iterator, multivalued_index_builder, values);\n            let serializable_multivalued_index = multivalued_index_builder.finish(num_rows);\n            if sort_values_within_row {\n                sort_values_within_row_in_place(\n                    serializable_multivalued_index.start_offsets.boxed_iter(),\n                    values,\n                );\n            }\n            SerializableColumnIndex::Multivalued(serializable_multivalued_index)\n        }\n    };\n    crate::column::serialize_column_mappable_to_u64(\n        serializable_column_index,\n        &&values[..],\n        &mut wrt,\n    )?;\n    Ok(())\n}\n\nfn sort_values_within_row_in_place(\n    multivalued_index: impl Iterator<Item = RowId>,\n    values: &mut [u64],\n) {\n    let mut start_index: usize = 0;\n    for end_index in multivalued_index {\n        let end_index = end_index as usize;\n        values[start_index..end_index].sort_unstable();\n        start_index = end_index;\n    }\n}\n\nfn coerce_numerical_symbol<T>(\n    operation_iterator: impl Iterator<Item = ColumnOperation<NumericalValue>>,\n) -> impl Iterator<Item = ColumnOperation<u64>>\nwhere T: Coerce + MonotonicallyMappableToU64 {\n    operation_iterator.map(|symbol| match symbol {\n        ColumnOperation::NewDoc(doc) => ColumnOperation::NewDoc(doc),\n        ColumnOperation::Value(numerical_value) => {\n            ColumnOperation::Value(T::coerce(numerical_value).to_u64())\n        }\n    })\n}\n\nfn consume_operation_iterator<T: Ord, TIndexBuilder: IndexBuilder>(\n    operation_iterator: impl Iterator<Item = ColumnOperation<T>>,\n    index_builder: &mut TIndexBuilder,\n    values: &mut Vec<T>,\n) {\n    for symbol in operation_iterator {\n        match symbol {\n            ColumnOperation::NewDoc(doc) => {\n                index_builder.record_row(doc);\n            }\n            ColumnOperation::Value(value) => {\n                index_builder.record_value();\n                values.push(value);\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use stacker::MemoryArena;\n\n    use crate::columnar::writer::column_operation::ColumnOperation;\n    use crate::{Cardinality, NumericalValue};\n\n    #[test]\n    fn test_column_writer_required_simple() {\n        let mut arena = MemoryArena::default();\n        let mut column_writer = super::ColumnWriter::default();\n        column_writer.record(0u32, NumericalValue::from(14i64), &mut arena);\n        column_writer.record(1u32, NumericalValue::from(15i64), &mut arena);\n        column_writer.record(2u32, NumericalValue::from(-16i64), &mut arena);\n        assert_eq!(column_writer.get_cardinality(3), Cardinality::Full);\n        let mut buffer = Vec::new();\n        let symbols: Vec<ColumnOperation<NumericalValue>> = column_writer\n            .operation_iterator(&arena, &mut buffer)\n            .collect();\n        assert_eq!(symbols.len(), 6);\n        assert!(matches!(symbols[0], ColumnOperation::NewDoc(0u32)));\n        assert!(matches!(\n            symbols[1],\n            ColumnOperation::Value(NumericalValue::I64(14i64))\n        ));\n        assert!(matches!(symbols[2], ColumnOperation::NewDoc(1u32)));\n        assert!(matches!(\n            symbols[3],\n            ColumnOperation::Value(NumericalValue::I64(15i64))\n        ));\n        assert!(matches!(symbols[4], ColumnOperation::NewDoc(2u32)));\n        assert!(matches!(\n            symbols[5],\n            ColumnOperation::Value(NumericalValue::I64(-16i64))\n        ));\n    }\n\n    #[test]\n    fn test_column_writer_optional_cardinality_missing_first() {\n        let mut arena = MemoryArena::default();\n        let mut column_writer = super::ColumnWriter::default();\n        column_writer.record(1u32, NumericalValue::from(15i64), &mut arena);\n        column_writer.record(2u32, NumericalValue::from(-16i64), &mut arena);\n        assert_eq!(column_writer.get_cardinality(3), Cardinality::Optional);\n        let mut buffer = Vec::new();\n        let symbols: Vec<ColumnOperation<NumericalValue>> = column_writer\n            .operation_iterator(&arena, &mut buffer)\n            .collect();\n        assert_eq!(symbols.len(), 4);\n        assert!(matches!(symbols[0], ColumnOperation::NewDoc(1u32)));\n        assert!(matches!(\n            symbols[1],\n            ColumnOperation::Value(NumericalValue::I64(15i64))\n        ));\n        assert!(matches!(symbols[2], ColumnOperation::NewDoc(2u32)));\n        assert!(matches!(\n            symbols[3],\n            ColumnOperation::Value(NumericalValue::I64(-16i64))\n        ));\n    }\n\n    #[test]\n    fn test_column_writer_optional_cardinality_missing_last() {\n        let mut arena = MemoryArena::default();\n        let mut column_writer = super::ColumnWriter::default();\n        column_writer.record(0u32, NumericalValue::from(15i64), &mut arena);\n        assert_eq!(column_writer.get_cardinality(2), Cardinality::Optional);\n        let mut buffer = Vec::new();\n        let symbols: Vec<ColumnOperation<NumericalValue>> = column_writer\n            .operation_iterator(&arena, &mut buffer)\n            .collect();\n        assert_eq!(symbols.len(), 2);\n        assert!(matches!(symbols[0], ColumnOperation::NewDoc(0u32)));\n        assert!(matches!(\n            symbols[1],\n            ColumnOperation::Value(NumericalValue::I64(15i64))\n        ));\n    }\n\n    #[test]\n    fn test_column_writer_multivalued() {\n        let mut arena = MemoryArena::default();\n        let mut column_writer = super::ColumnWriter::default();\n        column_writer.record(0u32, NumericalValue::from(16i64), &mut arena);\n        column_writer.record(0u32, NumericalValue::from(17i64), &mut arena);\n        assert_eq!(column_writer.get_cardinality(1), Cardinality::Multivalued);\n        let mut buffer = Vec::new();\n        let symbols: Vec<ColumnOperation<NumericalValue>> = column_writer\n            .operation_iterator(&arena, &mut buffer)\n            .collect();\n        assert_eq!(symbols.len(), 3);\n        assert!(matches!(symbols[0], ColumnOperation::NewDoc(0u32)));\n        assert!(matches!(\n            symbols[1],\n            ColumnOperation::Value(NumericalValue::I64(16i64))\n        ));\n        assert!(matches!(\n            symbols[2],\n            ColumnOperation::Value(NumericalValue::I64(17i64))\n        ));\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/writer/serializer.rs",
    "content": "use std::io;\nuse std::io::Write;\n\nuse common::json_path_writer::JSON_END_OF_PATH;\nuse common::{BinarySerializable, CountingWriter};\nuse sstable::RangeSSTable;\nuse sstable::value::RangeValueWriter;\n\nuse crate::RowId;\nuse crate::columnar::ColumnType;\n\npub struct ColumnarSerializer<W: io::Write> {\n    wrt: CountingWriter<W>,\n    sstable_range: sstable::Writer<Vec<u8>, RangeValueWriter>,\n    prepare_key_buffer: Vec<u8>,\n}\n\n/// Returns a key consisting of the concatenation of the key and the column_type_and_cardinality\n/// code.\nfn prepare_key(key: &[u8], column_type: ColumnType, buffer: &mut Vec<u8>) {\n    buffer.clear();\n    buffer.extend_from_slice(key);\n    buffer.push(JSON_END_OF_PATH);\n    buffer.push(column_type.to_code());\n}\n\nimpl<W: io::Write> ColumnarSerializer<W> {\n    pub(crate) fn new(wrt: W) -> ColumnarSerializer<W> {\n        let sstable_range: sstable::Writer<Vec<u8>, RangeValueWriter> =\n            sstable::Dictionary::<RangeSSTable>::builder(Vec::with_capacity(100_000)).unwrap();\n        ColumnarSerializer {\n            wrt: CountingWriter::wrap(wrt),\n            sstable_range,\n            prepare_key_buffer: Vec::new(),\n        }\n    }\n\n    /// Creates a ColumnSerializer.\n    pub fn start_serialize_column<'a>(\n        &'a mut self,\n        column_name: &[u8],\n        column_type: ColumnType,\n    ) -> ColumnSerializer<'a, W> {\n        let start_offset = self.wrt.written_bytes();\n        prepare_key(column_name, column_type, &mut self.prepare_key_buffer);\n        ColumnSerializer {\n            columnar_serializer: self,\n            start_offset,\n        }\n    }\n\n    pub(crate) fn finalize(mut self, num_rows: RowId) -> io::Result<()> {\n        let sstable_bytes: Vec<u8> = self.sstable_range.finish()?;\n        let sstable_num_bytes: u64 = sstable_bytes.len() as u64;\n        self.wrt.write_all(&sstable_bytes)?;\n        self.wrt.write_all(&sstable_num_bytes.to_le_bytes()[..])?;\n        num_rows.serialize(&mut self.wrt)?;\n        self.wrt\n            .write_all(&super::super::format_version::footer())?;\n        self.wrt.flush()?;\n        Ok(())\n    }\n}\n\npub struct ColumnSerializer<'a, W: io::Write> {\n    columnar_serializer: &'a mut ColumnarSerializer<W>,\n    start_offset: u64,\n}\n\nimpl<W: io::Write> ColumnSerializer<'_, W> {\n    pub fn finalize(self) -> io::Result<()> {\n        let end_offset: u64 = self.columnar_serializer.wrt.written_bytes();\n        let byte_range = self.start_offset..end_offset;\n        self.columnar_serializer.sstable_range.insert(\n            &self.columnar_serializer.prepare_key_buffer[..],\n            &byte_range,\n        )?;\n        self.columnar_serializer.prepare_key_buffer.clear();\n        Ok(())\n    }\n}\n\nimpl<W: io::Write> io::Write for ColumnSerializer<'_, W> {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        self.columnar_serializer.wrt.write(buf)\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        self.columnar_serializer.wrt.flush()\n    }\n\n    fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {\n        self.columnar_serializer.wrt.write_all(buf)\n    }\n}\n"
  },
  {
    "path": "columnar/src/columnar/writer/value_index.rs",
    "content": "use crate::RowId;\nuse crate::column_index::{SerializableMultivalueIndex, SerializableOptionalIndex};\nuse crate::iterable::Iterable;\n\n/// The `IndexBuilder` interprets a sequence of\n/// calls of the form:\n/// (record_doc,record_value+)*\n/// and can then serialize the results into an index to associate docids with their value[s].\n///\n/// It has different implementation depending on whether the\n/// cardinality is required, optional, or multivalued.\npub(crate) trait IndexBuilder {\n    fn record_row(&mut self, doc: RowId);\n    #[inline]\n    fn record_value(&mut self) {}\n}\n\n/// The FullIndexBuilder does nothing.\n#[derive(Default)]\npub struct FullIndexBuilder;\n\nimpl IndexBuilder for FullIndexBuilder {\n    #[inline(always)]\n    fn record_row(&mut self, _doc: RowId) {}\n}\n\n#[derive(Default)]\npub struct OptionalIndexBuilder {\n    docs: Vec<RowId>,\n}\n\nimpl OptionalIndexBuilder {\n    pub fn finish(&mut self, num_rows: RowId) -> impl Iterable<RowId> + '_ {\n        debug_assert!(\n            self.docs\n                .last()\n                .copied()\n                .map(|last_doc| last_doc < num_rows)\n                .unwrap_or(true)\n        );\n        &self.docs[..]\n    }\n\n    fn reset(&mut self) {\n        self.docs.clear();\n    }\n}\n\nimpl IndexBuilder for OptionalIndexBuilder {\n    #[inline(always)]\n    fn record_row(&mut self, doc: RowId) {\n        debug_assert!(\n            self.docs\n                .last()\n                .copied()\n                .map(|prev_doc| doc > prev_doc)\n                .unwrap_or(true)\n        );\n        self.docs.push(doc);\n    }\n}\n\n#[derive(Default)]\npub struct MultivaluedIndexBuilder {\n    doc_with_values: Vec<RowId>,\n    start_offsets: Vec<u32>,\n    total_num_vals_seen: u32,\n    current_row: RowId,\n    current_row_has_value: bool,\n}\n\nimpl MultivaluedIndexBuilder {\n    pub fn finish(&mut self, num_docs: RowId) -> SerializableMultivalueIndex<'_> {\n        self.start_offsets.push(self.total_num_vals_seen);\n        let non_null_row_ids: Box<dyn Iterable<RowId>> = Box::new(&self.doc_with_values[..]);\n        SerializableMultivalueIndex {\n            doc_ids_with_values: SerializableOptionalIndex {\n                non_null_row_ids,\n                num_rows: num_docs,\n            },\n            start_offsets: Box::new(&self.start_offsets[..]),\n        }\n    }\n\n    fn reset(&mut self) {\n        self.doc_with_values.clear();\n        self.start_offsets.clear();\n        self.total_num_vals_seen = 0;\n        self.current_row = 0;\n        self.current_row_has_value = false;\n    }\n}\n\nimpl IndexBuilder for MultivaluedIndexBuilder {\n    fn record_row(&mut self, row_id: RowId) {\n        self.current_row = row_id;\n        self.current_row_has_value = false;\n    }\n\n    fn record_value(&mut self) {\n        if !self.current_row_has_value {\n            self.current_row_has_value = true;\n            self.doc_with_values.push(self.current_row);\n            self.start_offsets.push(self.total_num_vals_seen);\n        }\n        self.total_num_vals_seen += 1;\n    }\n}\n\n/// The `SpareIndexBuilders` is there to avoid allocating a\n/// new index builder for every single column.\n#[derive(Default)]\npub struct PreallocatedIndexBuilders {\n    required_index_builder: FullIndexBuilder,\n    optional_index_builder: OptionalIndexBuilder,\n    multivalued_index_builder: MultivaluedIndexBuilder,\n}\n\nimpl PreallocatedIndexBuilders {\n    pub fn borrow_required_index_builder(&mut self) -> &mut FullIndexBuilder {\n        &mut self.required_index_builder\n    }\n\n    pub fn borrow_optional_index_builder(&mut self) -> &mut OptionalIndexBuilder {\n        self.optional_index_builder.reset();\n        &mut self.optional_index_builder\n    }\n\n    pub fn borrow_multivalued_index_builder(&mut self) -> &mut MultivaluedIndexBuilder {\n        self.multivalued_index_builder.reset();\n        &mut self.multivalued_index_builder\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_optional_value_index_builder() {\n        let mut opt_value_index_builder = OptionalIndexBuilder::default();\n        opt_value_index_builder.record_row(0u32);\n        opt_value_index_builder.record_value();\n        assert_eq!(\n            &opt_value_index_builder\n                .finish(1u32)\n                .boxed_iter()\n                .collect::<Vec<u32>>(),\n            &[0]\n        );\n        opt_value_index_builder.reset();\n        opt_value_index_builder.record_row(1u32);\n        opt_value_index_builder.record_value();\n        assert_eq!(\n            &opt_value_index_builder\n                .finish(2u32)\n                .boxed_iter()\n                .collect::<Vec<u32>>(),\n            &[1]\n        );\n    }\n\n    #[test]\n    fn test_multivalued_value_index_builder_simple() {\n        let mut multivalued_value_index_builder = MultivaluedIndexBuilder::default();\n        {\n            multivalued_value_index_builder.record_row(0u32);\n            multivalued_value_index_builder.record_value();\n            multivalued_value_index_builder.record_value();\n            let serialized_multivalue_index = multivalued_value_index_builder.finish(1u32);\n            let start_offsets: Vec<u32> = serialized_multivalue_index\n                .start_offsets\n                .boxed_iter()\n                .collect();\n            assert_eq!(&start_offsets, &[0, 2]);\n        }\n        multivalued_value_index_builder.reset();\n        multivalued_value_index_builder.record_row(0u32);\n        multivalued_value_index_builder.record_value();\n        multivalued_value_index_builder.record_value();\n        let serialized_multivalue_index = multivalued_value_index_builder.finish(1u32);\n        let start_offsets: Vec<u32> = serialized_multivalue_index\n            .start_offsets\n            .boxed_iter()\n            .collect();\n        assert_eq!(&start_offsets, &[0, 2]);\n    }\n\n    #[test]\n    fn test_multivalued_value_index_builder() {\n        let mut multivalued_value_index_builder = MultivaluedIndexBuilder::default();\n        multivalued_value_index_builder.record_row(1u32);\n        multivalued_value_index_builder.record_value();\n        multivalued_value_index_builder.record_value();\n        multivalued_value_index_builder.record_row(2u32);\n        multivalued_value_index_builder.record_value();\n        let SerializableMultivalueIndex {\n            doc_ids_with_values,\n            start_offsets,\n        } = multivalued_value_index_builder.finish(4u32);\n        assert_eq!(doc_ids_with_values.num_rows, 4u32);\n        let doc_ids_with_values: Vec<u32> =\n            doc_ids_with_values.non_null_row_ids.boxed_iter().collect();\n        assert_eq!(&doc_ids_with_values, &[1u32, 2u32]);\n        let start_offsets: Vec<u32> = start_offsets.boxed_iter().collect();\n        assert_eq!(&start_offsets[..], &[0, 2, 3]);\n    }\n}\n"
  },
  {
    "path": "columnar/src/compat_tests.rs",
    "content": "use std::path::PathBuf;\n\nuse itertools::Itertools;\n\nuse crate::{\n    CURRENT_VERSION, Cardinality, Column, ColumnarReader, DynamicColumn, StackMergeOrder,\n    merge_columnar,\n};\n\nconst NUM_DOCS: u32 = u16::MAX as u32;\n\nfn generate_columnar(num_docs: u32, value_offset: u64) -> Vec<u8> {\n    use crate::ColumnarWriter;\n\n    let mut columnar_writer = ColumnarWriter::default();\n\n    for i in 0..num_docs {\n        if i % 100 == 0 {\n            columnar_writer.record_numerical(i, \"sparse\", value_offset + i as u64);\n        }\n        if i % 5 == 0 {\n            columnar_writer.record_numerical(i, \"dense\", value_offset + i as u64);\n        }\n        columnar_writer.record_numerical(i, \"full\", value_offset + i as u64);\n        columnar_writer.record_numerical(i, \"multi\", value_offset + i as u64);\n        columnar_writer.record_numerical(i, \"multi\", value_offset + i as u64);\n    }\n\n    let mut wrt: Vec<u8> = Vec::new();\n    columnar_writer.serialize(num_docs, &mut wrt).unwrap();\n\n    wrt\n}\n\n#[test]\n/// Writes a columnar for the CURRENT_VERSION to disk.\nfn create_format() {\n    let version = CURRENT_VERSION.to_string();\n    let file_path = path_for_version(&version);\n    if PathBuf::from(file_path.clone()).exists() {\n        return;\n    }\n    let columnar = generate_columnar(NUM_DOCS, 0);\n    std::fs::write(file_path, columnar).unwrap();\n}\n\nfn path_for_version(version: &str) -> String {\n    format!(\"./compat_tests_data/{}.columnar\", version)\n}\n\n#[test]\nfn test_format_v1() {\n    let path = path_for_version(\"v1\");\n    test_format(&path);\n}\n\n#[test]\nfn test_format_v2() {\n    let path = path_for_version(\"v2\");\n    test_format(&path);\n}\n\nfn test_format(path: &str) {\n    let file_content = std::fs::read(path).unwrap();\n    let reader = ColumnarReader::open(file_content).unwrap();\n\n    check_columns(&reader);\n\n    // Test merge\n    let reader2 = ColumnarReader::open(generate_columnar(NUM_DOCS, NUM_DOCS as u64)).unwrap();\n    let columnar_readers = vec![&reader, &reader2];\n    let merge_row_order = StackMergeOrder::stack(&columnar_readers[..]);\n    let mut out = Vec::new();\n    merge_columnar(&columnar_readers, &[], merge_row_order.into(), &mut out).unwrap();\n    let reader = ColumnarReader::open(out).unwrap();\n    check_columns(&reader);\n}\n\nfn check_columns(reader: &ColumnarReader) {\n    let column = open_column(reader, \"full\");\n    check_column(&column, |doc_id| vec![(doc_id, doc_id as u64).into()]);\n    assert_eq!(column.get_cardinality(), Cardinality::Full);\n\n    let column = open_column(reader, \"multi\");\n    check_column(&column, |doc_id| {\n        vec![\n            (doc_id * 2, doc_id as u64).into(),\n            (doc_id * 2 + 1, doc_id as u64).into(),\n        ]\n    });\n    assert_eq!(column.get_cardinality(), Cardinality::Multivalued);\n\n    let column = open_column(reader, \"sparse\");\n    check_column(&column, |doc_id| {\n        if doc_id % 100 == 0 {\n            vec![(doc_id / 100, doc_id as u64).into()]\n        } else {\n            vec![]\n        }\n    });\n    assert_eq!(column.get_cardinality(), Cardinality::Optional);\n\n    let column = open_column(reader, \"dense\");\n    check_column(&column, |doc_id| {\n        if doc_id % 5 == 0 {\n            vec![(doc_id / 5, doc_id as u64).into()]\n        } else {\n            vec![]\n        }\n    });\n    assert_eq!(column.get_cardinality(), Cardinality::Optional);\n}\n\nstruct RowIdAndValue {\n    row_id: u32,\n    value: u64,\n}\nimpl From<(u32, u64)> for RowIdAndValue {\n    fn from((row_id, value): (u32, u64)) -> Self {\n        Self { row_id, value }\n    }\n}\n\nfn check_column<F: Fn(u32) -> Vec<RowIdAndValue>>(column: &Column<u64>, expected: F) {\n    let num_docs = column.num_docs();\n    let test_doc = |doc: u32| {\n        if expected(doc).is_empty() {\n            assert_eq!(column.first(doc), None);\n        } else {\n            assert_eq!(column.first(doc), Some(expected(doc)[0].value));\n        }\n        let values = column.values_for_doc(doc).collect_vec();\n        assert_eq!(values, expected(doc).iter().map(|x| x.value).collect_vec());\n        let mut row_ids = Vec::new();\n        column.row_ids_for_docs(&[doc], &mut vec![], &mut row_ids);\n        assert_eq!(\n            row_ids,\n            expected(doc).iter().map(|x| x.row_id).collect_vec()\n        );\n        let values = column.values_for_doc(doc).collect_vec();\n        assert_eq!(values, expected(doc).iter().map(|x| x.value).collect_vec());\n\n        // Docid rowid conversion\n        let mut row_ids = Vec::new();\n        let safe_next_doc = |doc: u32| (doc + 1).min(num_docs - 1);\n        column\n            .index\n            .docids_to_rowids(&[doc, safe_next_doc(doc)], &mut vec![], &mut row_ids);\n        let expected_rowids = expected(doc)\n            .iter()\n            .map(|x| x.row_id)\n            .chain(expected(safe_next_doc(doc)).iter().map(|x| x.row_id))\n            .collect_vec();\n        assert_eq!(row_ids, expected_rowids);\n        let rowid_range = column\n            .index\n            .docid_range_to_rowids(doc..safe_next_doc(doc) + 1);\n        if expected_rowids.is_empty() {\n            assert!(rowid_range.is_empty());\n        } else {\n            assert_eq!(\n                rowid_range,\n                expected_rowids[0]..expected_rowids.last().unwrap() + 1\n            );\n        }\n    };\n    test_doc(0);\n    test_doc(num_docs - 1);\n    test_doc(num_docs - 2);\n    test_doc(65000);\n}\n\nfn open_column(reader: &ColumnarReader, name: &str) -> Column<u64> {\n    let column = reader.read_columns(name).unwrap()[0]\n        .open()\n        .unwrap()\n        .coerce_numerical(crate::NumericalType::U64)\n        .unwrap();\n    let DynamicColumn::U64(column) = column else {\n        panic!();\n    };\n    column\n}\n"
  },
  {
    "path": "columnar/src/dictionary.rs",
    "content": "use std::io;\n\nuse sstable::SSTable;\nuse stacker::{MemoryArena, SharedArenaHashMap};\n\npub(crate) struct TermIdMapping {\n    unordered_to_ord: Vec<OrderedId>,\n}\n\nimpl TermIdMapping {\n    pub fn to_ord(&self, unordered: UnorderedId) -> OrderedId {\n        self.unordered_to_ord[unordered.0 as usize]\n    }\n}\n\n/// When we add values, we cannot know their ordered id yet.\n/// For this reason, we temporarily assign them a `UnorderedId`\n/// that will be mapped to an `OrderedId` upon serialization.\n#[derive(Clone, Copy, Debug, Hash, PartialEq, Eq)]\npub struct UnorderedId(pub u32);\n\n#[derive(Clone, Copy, Hash, PartialEq, Eq, Debug)]\npub struct OrderedId(pub u32);\n\n/// `DictionaryBuilder` for dictionary encoding.\n///\n/// It stores the different terms encounterred and assigns them a temporary value\n/// we call unordered id.\n///\n/// Upon serialization, we will sort the ids and hence build a `UnorderedId -> Term ordinal`\n/// mapping.\n#[derive(Default)]\npub(crate) struct DictionaryBuilder {\n    dict: SharedArenaHashMap,\n}\n\nimpl DictionaryBuilder {\n    /// Get or allocate an unordered id.\n    /// (This ID is simply an auto-incremented id.)\n    pub fn get_or_allocate_id(&mut self, term: &[u8], arena: &mut MemoryArena) -> UnorderedId {\n        let next_id = self.dict.len() as u32;\n        let unordered_id = self\n            .dict\n            .mutate_or_create(term, arena, |unordered_id: Option<u32>| {\n                if let Some(unordered_id) = unordered_id {\n                    unordered_id\n                } else {\n                    next_id\n                }\n            });\n        UnorderedId(unordered_id)\n    }\n\n    /// Serialize the dictionary into an fst, and returns the\n    /// `UnorderedId -> TermOrdinal` map.\n    pub fn serialize<'a, W: io::Write + 'a>(\n        &self,\n        arena: &MemoryArena,\n        wrt: &mut W,\n    ) -> io::Result<TermIdMapping> {\n        let mut terms: Vec<(&[u8], UnorderedId)> = self\n            .dict\n            .iter(arena)\n            .map(|(k, v)| (k, arena.read(v)))\n            .collect();\n        terms.sort_unstable_by_key(|(key, _)| *key);\n        // TODO Remove the allocation.\n        let mut unordered_to_ord: Vec<OrderedId> = vec![OrderedId(0u32); terms.len()];\n        let mut sstable_builder = sstable::VoidSSTable::writer(wrt);\n        for (ord, (key, unordered_id)) in terms.into_iter().enumerate() {\n            let ordered_id = OrderedId(ord as u32);\n            sstable_builder.insert(key, &())?;\n            unordered_to_ord[unordered_id.0 as usize] = ordered_id;\n        }\n        sstable_builder.finish()?;\n        Ok(TermIdMapping { unordered_to_ord })\n    }\n\n    pub(crate) fn mem_usage(&self) -> usize {\n        self.dict.mem_usage()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_dictionary_builder() {\n        let mut arena = MemoryArena::default();\n        let mut dictionary_builder = DictionaryBuilder::default();\n        let hello_uid = dictionary_builder.get_or_allocate_id(b\"hello\", &mut arena);\n        let happy_uid = dictionary_builder.get_or_allocate_id(b\"happy\", &mut arena);\n        let tax_uid = dictionary_builder.get_or_allocate_id(b\"tax\", &mut arena);\n        let mut buffer = Vec::new();\n        let id_mapping = dictionary_builder.serialize(&arena, &mut buffer).unwrap();\n        assert_eq!(id_mapping.to_ord(hello_uid), OrderedId(1));\n        assert_eq!(id_mapping.to_ord(happy_uid), OrderedId(0));\n        assert_eq!(id_mapping.to_ord(tax_uid), OrderedId(2));\n    }\n}\n"
  },
  {
    "path": "columnar/src/dynamic_column.rs",
    "content": "use std::net::Ipv6Addr;\nuse std::sync::Arc;\nuse std::{fmt, io};\n\nuse common::file_slice::FileSlice;\nuse common::{ByteCount, DateTime, OwnedBytes};\nuse serde::{Deserialize, Serialize};\n\nuse crate::column::{BytesColumn, Column, StrColumn};\nuse crate::column_values::{StrictlyMonotonicFn, monotonic_map_column};\nuse crate::columnar::ColumnType;\nuse crate::{Cardinality, ColumnIndex, ColumnValues, NumericalType, Version};\n\n#[derive(Clone)]\npub enum DynamicColumn {\n    Bool(Column<bool>),\n    I64(Column<i64>),\n    U64(Column<u64>),\n    F64(Column<f64>),\n    IpAddr(Column<Ipv6Addr>),\n    DateTime(Column<DateTime>),\n    Bytes(BytesColumn),\n    Str(StrColumn),\n}\n\nimpl fmt::Debug for DynamicColumn {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"[{} {} |\", self.get_cardinality(), self.column_type())?;\n        match self {\n            DynamicColumn::Bool(col) => write!(f, \" {col:?}\")?,\n            DynamicColumn::I64(col) => write!(f, \" {col:?}\")?,\n            DynamicColumn::U64(col) => write!(f, \" {col:?}\")?,\n            DynamicColumn::F64(col) => write!(f, \"{col:?}\")?,\n            DynamicColumn::IpAddr(col) => write!(f, \"{col:?}\")?,\n            DynamicColumn::DateTime(col) => write!(f, \"{col:?}\")?,\n            DynamicColumn::Bytes(col) => write!(f, \"{col:?}\")?,\n            DynamicColumn::Str(col) => write!(f, \"{col:?}\")?,\n        }\n        write!(f, \"]\")\n    }\n}\n\nimpl DynamicColumn {\n    pub fn column_index(&self) -> &ColumnIndex {\n        match self {\n            DynamicColumn::Bool(c) => &c.index,\n            DynamicColumn::I64(c) => &c.index,\n            DynamicColumn::U64(c) => &c.index,\n            DynamicColumn::F64(c) => &c.index,\n            DynamicColumn::IpAddr(c) => &c.index,\n            DynamicColumn::DateTime(c) => &c.index,\n            DynamicColumn::Bytes(c) => &c.ords().index,\n            DynamicColumn::Str(c) => &c.ords().index,\n        }\n    }\n\n    pub fn get_cardinality(&self) -> Cardinality {\n        self.column_index().get_cardinality()\n    }\n\n    pub fn num_values(&self) -> u32 {\n        match self {\n            DynamicColumn::Bool(c) => c.values.num_vals(),\n            DynamicColumn::I64(c) => c.values.num_vals(),\n            DynamicColumn::U64(c) => c.values.num_vals(),\n            DynamicColumn::F64(c) => c.values.num_vals(),\n            DynamicColumn::IpAddr(c) => c.values.num_vals(),\n            DynamicColumn::DateTime(c) => c.values.num_vals(),\n            DynamicColumn::Bytes(c) => c.ords().values.num_vals(),\n            DynamicColumn::Str(c) => c.ords().values.num_vals(),\n        }\n    }\n\n    pub fn column_type(&self) -> ColumnType {\n        match self {\n            DynamicColumn::Bool(_) => ColumnType::Bool,\n            DynamicColumn::I64(_) => ColumnType::I64,\n            DynamicColumn::U64(_) => ColumnType::U64,\n            DynamicColumn::F64(_) => ColumnType::F64,\n            DynamicColumn::IpAddr(_) => ColumnType::IpAddr,\n            DynamicColumn::DateTime(_) => ColumnType::DateTime,\n            DynamicColumn::Bytes(_) => ColumnType::Bytes,\n            DynamicColumn::Str(_) => ColumnType::Str,\n        }\n    }\n\n    pub fn coerce_numerical(self, target_numerical_type: NumericalType) -> Option<Self> {\n        match target_numerical_type {\n            NumericalType::I64 => self.coerce_to_i64(),\n            NumericalType::U64 => self.coerce_to_u64(),\n            NumericalType::F64 => self.coerce_to_f64(),\n        }\n    }\n\n    pub fn is_numerical(&self) -> bool {\n        self.column_type().numerical_type().is_some()\n    }\n\n    pub fn is_f64(&self) -> bool {\n        self.column_type().numerical_type() == Some(NumericalType::F64)\n    }\n    pub fn is_i64(&self) -> bool {\n        self.column_type().numerical_type() == Some(NumericalType::I64)\n    }\n    pub fn is_u64(&self) -> bool {\n        self.column_type().numerical_type() == Some(NumericalType::U64)\n    }\n\n    fn coerce_to_f64(self) -> Option<DynamicColumn> {\n        match self {\n            DynamicColumn::I64(column) => Some(DynamicColumn::F64(Column {\n                index: column.index,\n                values: Arc::new(monotonic_map_column(column.values, MapI64ToF64)),\n            })),\n            DynamicColumn::U64(column) => Some(DynamicColumn::F64(Column {\n                index: column.index,\n                values: Arc::new(monotonic_map_column(column.values, MapU64ToF64)),\n            })),\n            DynamicColumn::F64(_) => Some(self),\n            _ => None,\n        }\n    }\n    fn coerce_to_i64(self) -> Option<DynamicColumn> {\n        match self {\n            DynamicColumn::U64(column) => {\n                if column.max_value() > i64::MAX as u64 {\n                    return None;\n                }\n                Some(DynamicColumn::I64(Column {\n                    index: column.index,\n                    values: Arc::new(monotonic_map_column(column.values, MapU64ToI64)),\n                }))\n            }\n            DynamicColumn::I64(_) => Some(self),\n            _ => None,\n        }\n    }\n    fn coerce_to_u64(self) -> Option<DynamicColumn> {\n        match self {\n            DynamicColumn::I64(column) => {\n                if column.min_value() < 0 {\n                    return None;\n                }\n                Some(DynamicColumn::U64(Column {\n                    index: column.index,\n                    values: Arc::new(monotonic_map_column(column.values, MapI64ToU64)),\n                }))\n            }\n            DynamicColumn::U64(_) => Some(self),\n            _ => None,\n        }\n    }\n}\n\nstruct MapI64ToF64;\nimpl StrictlyMonotonicFn<i64, f64> for MapI64ToF64 {\n    #[inline(always)]\n    fn mapping(&self, inp: i64) -> f64 {\n        inp as f64\n    }\n    #[inline(always)]\n    fn inverse(&self, out: f64) -> i64 {\n        out as i64\n    }\n}\n\nstruct MapU64ToF64;\nimpl StrictlyMonotonicFn<u64, f64> for MapU64ToF64 {\n    #[inline(always)]\n    fn mapping(&self, inp: u64) -> f64 {\n        inp as f64\n    }\n    #[inline(always)]\n    fn inverse(&self, out: f64) -> u64 {\n        out as u64\n    }\n}\n\nstruct MapU64ToI64;\nimpl StrictlyMonotonicFn<u64, i64> for MapU64ToI64 {\n    #[inline(always)]\n    fn mapping(&self, inp: u64) -> i64 {\n        inp as i64\n    }\n    #[inline(always)]\n    fn inverse(&self, out: i64) -> u64 {\n        out as u64\n    }\n}\n\nstruct MapI64ToU64;\nimpl StrictlyMonotonicFn<i64, u64> for MapI64ToU64 {\n    #[inline(always)]\n    fn mapping(&self, inp: i64) -> u64 {\n        inp as u64\n    }\n    #[inline(always)]\n    fn inverse(&self, out: u64) -> i64 {\n        out as i64\n    }\n}\n\nmacro_rules! static_dynamic_conversions {\n    ($typ:ty, $enum_name:ident) => {\n        impl From<DynamicColumn> for Option<$typ> {\n            fn from(dynamic_column: DynamicColumn) -> Option<$typ> {\n                if let DynamicColumn::$enum_name(col) = dynamic_column {\n                    Some(col)\n                } else {\n                    None\n                }\n            }\n        }\n\n        impl From<$typ> for DynamicColumn {\n            fn from(typed_column: $typ) -> Self {\n                DynamicColumn::$enum_name(typed_column)\n            }\n        }\n    };\n}\n\nstatic_dynamic_conversions!(Column<bool>, Bool);\nstatic_dynamic_conversions!(Column<u64>, U64);\nstatic_dynamic_conversions!(Column<i64>, I64);\nstatic_dynamic_conversions!(Column<f64>, F64);\nstatic_dynamic_conversions!(Column<DateTime>, DateTime);\nstatic_dynamic_conversions!(StrColumn, Str);\nstatic_dynamic_conversions!(BytesColumn, Bytes);\nstatic_dynamic_conversions!(Column<Ipv6Addr>, IpAddr);\n\n#[derive(Clone, Debug)]\npub struct DynamicColumnHandle {\n    pub(crate) file_slice: FileSlice,\n    pub(crate) column_type: ColumnType,\n    pub(crate) format_version: Version,\n}\n\nimpl DynamicColumnHandle {\n    // TODO rename load\n    pub fn open(&self) -> io::Result<DynamicColumn> {\n        let column_bytes: OwnedBytes = self.file_slice.read_bytes()?;\n        self.open_internal(column_bytes)\n    }\n\n    #[doc(hidden)]\n    pub fn file_slice(&self) -> &FileSlice {\n        &self.file_slice\n    }\n\n    /// Returns the `u64` fast field reader reader associated with `fields` of types\n    /// Str, u64, i64, f64, bool, ip, or datetime.\n    ///\n    /// Notice that for IpAddr, the fastfield reader will return the u64 representation of the\n    /// IpAddr.\n    /// In order to convert to u128 back cast to `CompactSpaceU64Accessor` and call\n    /// `compact_to_u128`.\n    ///\n    /// If not, the fastfield reader will returns the u64-value associated with the original\n    /// FastValue.\n    pub fn open_u64_lenient(&self) -> io::Result<Option<Column<u64>>> {\n        let column_bytes = self.file_slice.read_bytes()?;\n        match self.column_type {\n            ColumnType::Str | ColumnType::Bytes => {\n                let column: BytesColumn =\n                    crate::column::open_column_bytes(column_bytes, self.format_version)?;\n                Ok(Some(column.term_ord_column))\n            }\n            ColumnType::IpAddr => {\n                let column = crate::column::open_column_u128_as_compact_u64(\n                    column_bytes,\n                    self.format_version,\n                )?;\n                Ok(Some(column))\n            }\n            ColumnType::Bool\n            | ColumnType::I64\n            | ColumnType::U64\n            | ColumnType::F64\n            | ColumnType::DateTime => {\n                let column =\n                    crate::column::open_column_u64::<u64>(column_bytes, self.format_version)?;\n                Ok(Some(column))\n            }\n        }\n    }\n\n    fn open_internal(&self, column_bytes: OwnedBytes) -> io::Result<DynamicColumn> {\n        let dynamic_column: DynamicColumn = match self.column_type {\n            ColumnType::Bytes => {\n                crate::column::open_column_bytes(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::Str => {\n                crate::column::open_column_str(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::I64 => {\n                crate::column::open_column_u64::<i64>(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::U64 => {\n                crate::column::open_column_u64::<u64>(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::F64 => {\n                crate::column::open_column_u64::<f64>(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::Bool => {\n                crate::column::open_column_u64::<bool>(column_bytes, self.format_version)?.into()\n            }\n            ColumnType::IpAddr => {\n                crate::column::open_column_u128::<Ipv6Addr>(column_bytes, self.format_version)?\n                    .into()\n            }\n            ColumnType::DateTime => {\n                crate::column::open_column_u64::<DateTime>(column_bytes, self.format_version)?\n                    .into()\n            }\n        };\n        Ok(dynamic_column)\n    }\n\n    pub fn num_bytes(&self) -> ByteCount {\n        self.file_slice.num_bytes()\n    }\n\n    /// Legacy helper returning the column space usage.\n    pub fn column_and_dictionary_num_bytes(&self) -> io::Result<ColumnSpaceUsage> {\n        self.space_usage()\n    }\n\n    /// Return the space usage of the column, optionally broken down by dictionary and column\n    /// values.\n    ///\n    /// For dictionary encoded columns (strings and bytes), this splits the total footprint into\n    /// the dictionary and the remaining column data (including index and values).\n    /// For all other column types, the dictionary size is `None` and the column size\n    /// equals the total bytes.\n    pub fn space_usage(&self) -> io::Result<ColumnSpaceUsage> {\n        let total_num_bytes = self.num_bytes();\n        let dynamic_column = self.open()?;\n        let dictionary_num_bytes = match &dynamic_column {\n            DynamicColumn::Bytes(bytes_column) => bytes_column.dictionary().num_bytes(),\n            DynamicColumn::Str(str_column) => str_column.dictionary().num_bytes(),\n            _ => {\n                return Ok(ColumnSpaceUsage::new(self.num_bytes(), None));\n            }\n        };\n        assert!(dictionary_num_bytes <= total_num_bytes);\n        let column_num_bytes =\n            ByteCount::from(total_num_bytes.get_bytes() - dictionary_num_bytes.get_bytes());\n        Ok(ColumnSpaceUsage::new(\n            column_num_bytes,\n            Some(dictionary_num_bytes),\n        ))\n    }\n\n    pub fn column_type(&self) -> ColumnType {\n        self.column_type\n    }\n}\n\n/// Represents space usage of a column.\n///\n/// `column_num_bytes` tracks the column payload (index, values and footer).\n/// For dictionary encoded columns, `dictionary_num_bytes` captures the dictionary footprint.\n/// [`ColumnSpaceUsage::total_num_bytes`] returns the sum of both parts.\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct ColumnSpaceUsage {\n    column_num_bytes: ByteCount,\n    dictionary_num_bytes: Option<ByteCount>,\n}\n\nimpl ColumnSpaceUsage {\n    pub(crate) fn new(\n        column_num_bytes: ByteCount,\n        dictionary_num_bytes: Option<ByteCount>,\n    ) -> Self {\n        ColumnSpaceUsage {\n            column_num_bytes,\n            dictionary_num_bytes,\n        }\n    }\n\n    pub fn column_num_bytes(&self) -> ByteCount {\n        self.column_num_bytes\n    }\n\n    pub fn dictionary_num_bytes(&self) -> Option<ByteCount> {\n        self.dictionary_num_bytes\n    }\n\n    pub fn total_num_bytes(&self) -> ByteCount {\n        self.column_num_bytes + self.dictionary_num_bytes.unwrap_or_default()\n    }\n\n    /// Merge two space usage values by summing their components.\n    pub fn merge(&self, other: &ColumnSpaceUsage) -> ColumnSpaceUsage {\n        let dictionary_num_bytes = match (self.dictionary_num_bytes, other.dictionary_num_bytes) {\n            (Some(lhs), Some(rhs)) => Some(lhs + rhs),\n            (Some(val), None) | (None, Some(val)) => Some(val),\n            (None, None) => None,\n        };\n        ColumnSpaceUsage {\n            column_num_bytes: self.column_num_bytes + other.column_num_bytes,\n            dictionary_num_bytes,\n        }\n    }\n}\n"
  },
  {
    "path": "columnar/src/iterable.rs",
    "content": "use std::ops::Range;\nuse std::sync::Arc;\n\nuse crate::{ColumnValues, RowId};\n\npub trait Iterable<T = u64> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = T> + '_>;\n}\n\nimpl<T: Copy> Iterable<T> for &[T] {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = T> + '_> {\n        Box::new(self.iter().copied())\n    }\n}\n\nimpl<T: Copy> Iterable<T> for Range<T>\nwhere Range<T>: Iterator<Item = T>\n{\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = T> + '_> {\n        Box::new(self.clone())\n    }\n}\n\nimpl Iterable for Arc<dyn crate::ColumnValues<RowId>> {\n    fn boxed_iter(&self) -> Box<dyn Iterator<Item = u64> + '_> {\n        Box::new(self.iter().map(|row_id| row_id as u64))\n    }\n}\n"
  },
  {
    "path": "columnar/src/lib.rs",
    "content": "//! # Tantivy-Columnar\n//!\n//! `tantivy-columnar`provides a columnar storage for tantivy.\n//! The crate allows for efficient read operations on specific columns rather than entire records.\n//!\n//! ## Overview\n//!\n//! - **columnar**: Reading, writing, and merging multiple columns:\n//!   - **[ColumnarWriter]**: Makes it possible to create a new columnar.\n//!   - **[ColumnarReader]**: The ColumnarReader makes it possible to access a set of columns\n//!     associated to field names.\n//!   - **[merge_columnar]**: Contains the functionalities to merge multiple ColumnarReader or\n//!     segments into a single one.\n//!\n//! - **column**: A single column, which contains\n//!     - [column_index]: Resolves the rows for a document id. Manages the cardinality of the\n//!       column.\n//!     - [column_values]: Stores the values of a column in a dense format.\n\n#[cfg(test)]\n#[macro_use]\nextern crate more_asserts;\n\nuse std::fmt::Display;\nuse std::io;\n\nmod block_accessor;\nmod column;\npub mod column_index;\npub mod column_values;\nmod columnar;\nmod dictionary;\nmod dynamic_column;\nmod iterable;\npub(crate) mod utils;\nmod value;\n\npub use block_accessor::ColumnBlockAccessor;\npub use column::{BytesColumn, Column, StrColumn};\npub use column_index::ColumnIndex;\npub use column_values::{\n    ColumnValues, EmptyColumnValues, MonotonicallyMappableToU64, MonotonicallyMappableToU128,\n};\npub use columnar::{\n    CURRENT_VERSION, ColumnType, ColumnarReader, ColumnarWriter, HasAssociatedColumnType,\n    MergeRowOrder, ShuffleMergeOrder, StackMergeOrder, Version, merge_columnar,\n};\nuse sstable::VoidSSTable;\npub use value::{NumericalType, NumericalValue};\n\npub use self::dynamic_column::{ColumnSpaceUsage, DynamicColumn, DynamicColumnHandle};\n\npub type RowId = u32;\npub type DocId = u32;\n\n#[derive(Clone, Copy, Debug)]\npub struct RowAddr {\n    pub segment_ord: u32,\n    pub row_id: RowId,\n}\n\npub use sstable::{Dictionary, TermOrdHit};\npub type Streamer<'a> = sstable::Streamer<'a, VoidSSTable>;\n\npub use common::DateTime;\n\n#[derive(Copy, Clone, Debug)]\npub struct InvalidData;\n\nimpl From<InvalidData> for io::Error {\n    fn from(_: InvalidData) -> Self {\n        io::Error::new(io::ErrorKind::InvalidData, \"Invalid data\")\n    }\n}\n\n/// Enum describing the number of values that can exist per document\n/// (or per row if you will).\n///\n/// The cardinality must fit on 2 bits.\n#[derive(Clone, Copy, Hash, Default, Debug, PartialEq, Eq, PartialOrd, Ord)]\n#[repr(u8)]\npub enum Cardinality {\n    /// All documents contain exactly one value.\n    /// `Full` is the default for auto-detecting the Cardinality, since it is the most strict.\n    #[default]\n    Full = 0,\n    /// All documents contain at most one value.\n    Optional = 1,\n    /// All documents may contain any number of values.\n    Multivalued = 2,\n}\n\nimpl Display for Cardinality {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        let short_str = match self {\n            Cardinality::Full => \"full\",\n            Cardinality::Optional => \"opt\",\n            Cardinality::Multivalued => \"mult\",\n        };\n        write!(f, \"{short_str}\")\n    }\n}\n\nimpl Cardinality {\n    pub fn is_optional(&self) -> bool {\n        matches!(self, Cardinality::Optional)\n    }\n    pub fn is_multivalue(&self) -> bool {\n        matches!(self, Cardinality::Multivalued)\n    }\n    pub fn is_full(&self) -> bool {\n        matches!(self, Cardinality::Full)\n    }\n    pub(crate) fn to_code(self) -> u8 {\n        self as u8\n    }\n    pub(crate) fn try_from_code(code: u8) -> Result<Cardinality, InvalidData> {\n        match code {\n            0 => Ok(Cardinality::Full),\n            1 => Ok(Cardinality::Optional),\n            2 => Ok(Cardinality::Multivalued),\n            _ => Err(InvalidData),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests;\n\n#[cfg(test)]\nmod compat_tests;\n"
  },
  {
    "path": "columnar/src/tests.rs",
    "content": "use std::collections::HashMap;\nuse std::fmt::Debug;\nuse std::net::Ipv6Addr;\n\nuse common::DateTime;\nuse proptest::prelude::*;\nuse proptest::sample::subsequence;\n\nuse crate::column_values::MonotonicallyMappableToU128;\nuse crate::columnar::{ColumnType, ColumnTypeCategory};\nuse crate::dynamic_column::{DynamicColumn, DynamicColumnHandle};\nuse crate::value::{Coerce, NumericalValue};\nuse crate::{\n    BytesColumn, Cardinality, Column, ColumnarReader, ColumnarWriter, RowAddr, RowId,\n    ShuffleMergeOrder, StackMergeOrder,\n};\n\n#[test]\nfn test_dataframe_writer_str() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_str(1u32, \"my_string\", \"hello\");\n    dataframe_writer.record_str(3u32, \"my_string\", \"helloeee\");\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(5, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"my_string\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].num_bytes(), 73);\n}\n\n#[test]\nfn test_dataframe_writer_bytes() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_bytes(1u32, \"my_string\", b\"hello\");\n    dataframe_writer.record_bytes(3u32, \"my_string\", b\"helloeee\");\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(5, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"my_string\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].num_bytes(), 73);\n}\n\n#[test]\nfn test_dataframe_writer_bool() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_bool(1u32, \"bool.value\", false);\n    dataframe_writer.record_bool(3u32, \"bool.value\", true);\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(5, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"bool.value\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].num_bytes(), 22);\n    assert_eq!(cols[0].column_type(), ColumnType::Bool);\n    let dyn_bool_col = cols[0].open().unwrap();\n    let DynamicColumn::Bool(bool_col) = dyn_bool_col else {\n        panic!();\n    };\n    let vals: Vec<Option<bool>> = (0..5).map(|doc_id| bool_col.first(doc_id)).collect();\n    assert_eq!(&vals, &[None, Some(false), None, Some(true), None,]);\n}\n\n#[test]\nfn test_dataframe_writer_u64_multivalued() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_numerical(2u32, \"divisor\", 2u64);\n    dataframe_writer.record_numerical(3u32, \"divisor\", 3u64);\n    dataframe_writer.record_numerical(4u32, \"divisor\", 2u64);\n    dataframe_writer.record_numerical(5u32, \"divisor\", 5u64);\n    dataframe_writer.record_numerical(6u32, \"divisor\", 2u64);\n    dataframe_writer.record_numerical(6u32, \"divisor\", 3u64);\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(7, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"divisor\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].num_bytes(), 50);\n    let dyn_i64_col = cols[0].open().unwrap();\n    let DynamicColumn::I64(divisor_col) = dyn_i64_col else {\n        panic!();\n    };\n    assert_eq!(\n        divisor_col.get_cardinality(),\n        crate::Cardinality::Multivalued\n    );\n    assert_eq!(divisor_col.num_docs(), 7);\n}\n\n#[test]\nfn test_dataframe_writer_ip_addr() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_ip_addr(1, \"ip_addr\", Ipv6Addr::from_u128(1001));\n    dataframe_writer.record_ip_addr(3, \"ip_addr\", Ipv6Addr::from_u128(1050));\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(5, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"ip_addr\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].num_bytes(), 42);\n    assert_eq!(cols[0].column_type(), ColumnType::IpAddr);\n    let dyn_bool_col = cols[0].open().unwrap();\n    let DynamicColumn::IpAddr(ip_col) = dyn_bool_col else {\n        panic!();\n    };\n    let vals: Vec<Option<Ipv6Addr>> = (0..5).map(|doc_id| ip_col.first(doc_id)).collect();\n    assert_eq!(\n        &vals,\n        &[\n            None,\n            Some(Ipv6Addr::from_u128(1001)),\n            None,\n            Some(Ipv6Addr::from_u128(1050)),\n            None,\n        ]\n    );\n}\n\n#[test]\nfn test_dataframe_writer_numerical() {\n    let mut dataframe_writer = ColumnarWriter::default();\n    dataframe_writer.record_numerical(1u32, \"srical.value\", NumericalValue::U64(12u64));\n    dataframe_writer.record_numerical(2u32, \"srical.value\", NumericalValue::U64(13u64));\n    dataframe_writer.record_numerical(4u32, \"srical.value\", NumericalValue::U64(15u64));\n    let mut buffer: Vec<u8> = Vec::new();\n    dataframe_writer.serialize(6, &mut buffer).unwrap();\n    let columnar = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar.num_columns(), 1);\n    let cols: Vec<DynamicColumnHandle> = columnar.read_columns(\"srical.value\").unwrap();\n    assert_eq!(cols.len(), 1);\n    // Right now this 31 bytes are spent as follows\n    //\n    // - header 14 bytes\n    // - vals  8 //< due to padding? could have been 1byte?.\n    // - null footer 6 bytes\n    assert_eq!(cols[0].num_bytes(), 33);\n    let column = cols[0].open().unwrap();\n    let DynamicColumn::I64(column_i64) = column else {\n        panic!();\n    };\n    assert_eq!(column_i64.index.get_cardinality(), Cardinality::Optional);\n    assert_eq!(column_i64.first(0), None);\n    assert_eq!(column_i64.first(1), Some(12i64));\n    assert_eq!(column_i64.first(2), Some(13i64));\n    assert_eq!(column_i64.first(3), None);\n    assert_eq!(column_i64.first(4), Some(15i64));\n    assert_eq!(column_i64.first(5), None);\n    assert_eq!(column_i64.first(6), None); //< we can change the spec for that one.\n}\n\n#[test]\nfn test_dictionary_encoded_str() {\n    let mut buffer = Vec::new();\n    let mut columnar_writer = ColumnarWriter::default();\n    columnar_writer.record_str(1, \"my.column\", \"a\");\n    columnar_writer.record_str(3, \"my.column\", \"c\");\n    columnar_writer.record_str(3, \"my.column2\", \"different_column!\");\n    columnar_writer.record_str(4, \"my.column\", \"b\");\n    columnar_writer.serialize(5, &mut buffer).unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_columns(), 2);\n    let col_handles = columnar_reader.read_columns(\"my.column\").unwrap();\n    assert_eq!(col_handles.len(), 1);\n    let DynamicColumn::Str(str_col) = col_handles[0].open().unwrap() else {\n        panic!();\n    };\n    let index: Vec<Option<u64>> = (0..5).map(|doc_id| str_col.ords().first(doc_id)).collect();\n    assert_eq!(index, &[None, Some(0), None, Some(2), Some(1)]);\n    assert_eq!(str_col.num_rows(), 5);\n    let mut term_buffer = String::new();\n    let term_ords = str_col.ords();\n    assert_eq!(term_ords.first(0), None);\n    assert_eq!(term_ords.first(1), Some(0));\n    str_col.ord_to_str(0u64, &mut term_buffer).unwrap();\n    assert_eq!(term_buffer, \"a\");\n    assert_eq!(term_ords.first(2), None);\n    assert_eq!(term_ords.first(3), Some(2));\n    str_col.ord_to_str(2u64, &mut term_buffer).unwrap();\n    assert_eq!(term_buffer, \"c\");\n    assert_eq!(term_ords.first(4), Some(1));\n    str_col.ord_to_str(1u64, &mut term_buffer).unwrap();\n    assert_eq!(term_buffer, \"b\");\n}\n\n#[test]\nfn test_dictionary_encoded_bytes() {\n    let mut buffer = Vec::new();\n    let mut columnar_writer = ColumnarWriter::default();\n    columnar_writer.record_bytes(1, \"my.column\", b\"a\");\n    columnar_writer.record_bytes(3, \"my.column\", b\"c\");\n    columnar_writer.record_bytes(3, \"my.column2\", b\"different_column!\");\n    columnar_writer.record_bytes(4, \"my.column\", b\"b\");\n    columnar_writer.serialize(5, &mut buffer).unwrap();\n    let columnar_reader = ColumnarReader::open(buffer).unwrap();\n    assert_eq!(columnar_reader.num_columns(), 2);\n    let col_handles = columnar_reader.read_columns(\"my.column\").unwrap();\n    assert_eq!(col_handles.len(), 1);\n    let DynamicColumn::Bytes(bytes_col) = col_handles[0].open().unwrap() else {\n        panic!();\n    };\n    let index: Vec<Option<u64>> = (0..5)\n        .map(|doc_id| bytes_col.ords().first(doc_id))\n        .collect();\n    assert_eq!(index, &[None, Some(0), None, Some(2), Some(1)]);\n    assert_eq!(bytes_col.num_rows(), 5);\n    let mut term_buffer = Vec::new();\n    let term_ords = bytes_col.ords();\n    assert_eq!(term_ords.first(0), None);\n    assert_eq!(term_ords.first(1), Some(0));\n    bytes_col\n        .dictionary\n        .ord_to_term(0u64, &mut term_buffer)\n        .unwrap();\n    assert_eq!(term_buffer, b\"a\");\n    assert_eq!(term_ords.first(2), None);\n    assert_eq!(term_ords.first(3), Some(2));\n    bytes_col\n        .dictionary\n        .ord_to_term(2u64, &mut term_buffer)\n        .unwrap();\n    assert_eq!(term_buffer, b\"c\");\n    assert_eq!(term_ords.first(4), Some(1));\n    bytes_col\n        .dictionary\n        .ord_to_term(1u64, &mut term_buffer)\n        .unwrap();\n    assert_eq!(term_buffer, b\"b\");\n}\n\nfn num_strategy() -> impl Strategy<Value = NumericalValue> {\n    prop_oneof![\n        3 => Just(NumericalValue::U64(0u64)),\n        3 => Just(NumericalValue::U64(u64::MAX)),\n        3 => Just(NumericalValue::I64(0i64)),\n        3 => Just(NumericalValue::I64(i64::MIN)),\n        3 => Just(NumericalValue::I64(i64::MAX)),\n        3 => Just(NumericalValue::F64(1.2f64)),\n        1 => any::<f64>().prop_map(NumericalValue::from),\n        1 => any::<u64>().prop_map(NumericalValue::from),\n        1 => any::<i64>().prop_map(NumericalValue::from),\n    ]\n}\n\n#[derive(Debug, Clone, Copy)]\nenum ColumnValue {\n    Str(&'static str),\n    Bytes(&'static [u8]),\n    Numerical(NumericalValue),\n    IpAddr(Ipv6Addr),\n    Bool(bool),\n    DateTime(DateTime),\n}\n\nimpl<T: Into<NumericalValue>> From<T> for ColumnValue {\n    fn from(val: T) -> ColumnValue {\n        ColumnValue::Numerical(val.into())\n    }\n}\n\nimpl ColumnValue {\n    pub(crate) fn column_type_category(&self) -> ColumnTypeCategory {\n        match self {\n            ColumnValue::Str(_) => ColumnTypeCategory::Str,\n            ColumnValue::Bytes(_) => ColumnTypeCategory::Bytes,\n            ColumnValue::Numerical(_) => ColumnTypeCategory::Numerical,\n            ColumnValue::IpAddr(_) => ColumnTypeCategory::IpAddr,\n            ColumnValue::Bool(_) => ColumnTypeCategory::Bool,\n            ColumnValue::DateTime(_) => ColumnTypeCategory::DateTime,\n        }\n    }\n}\n\nfn column_name_strategy() -> impl Strategy<Value = &'static str> {\n    prop_oneof![Just(\"c1\"), Just(\"c2\")]\n}\n\nfn string_strategy() -> impl Strategy<Value = &'static str> {\n    prop_oneof![Just(\"a\"), Just(\"b\")]\n}\n\nfn bytes_strategy() -> impl Strategy<Value = &'static [u8]> {\n    prop_oneof![Just(&[0u8][..]), Just(&[1u8][..])]\n}\n\n// A random column value\nfn column_value_strategy() -> impl Strategy<Value = ColumnValue> {\n    prop_oneof![\n        10 => string_strategy().prop_map(ColumnValue::Str),\n        1 => bytes_strategy().prop_map(ColumnValue::Bytes),\n        40 => num_strategy().prop_map(ColumnValue::Numerical),\n        1 => (1u16..3u16).prop_map(|ip_addr_byte| ColumnValue::IpAddr(Ipv6Addr::new(\n            127,\n            0,\n            0,\n            0,\n            0,\n            0,\n            0,\n            ip_addr_byte\n        ))),\n        1 => any::<bool>().prop_map(ColumnValue::Bool),\n        1 => (679_723_993i64..1_679_723_995i64)\n            .prop_map(|val| { ColumnValue::DateTime(DateTime::from_timestamp_secs(val)) })\n    ]\n}\n\n// A document contains up to 4 values.\nfn doc_strategy() -> impl Strategy<Value = Vec<(&'static str, ColumnValue)>> {\n    proptest::collection::vec((column_name_strategy(), column_value_strategy()), 0..=4)\n}\n\nfn num_docs_strategy() -> impl Strategy<Value = usize> {\n    prop_oneof!(\n        // We focus heavily on the 0..2 case as we assume it is sufficient to cover all edge cases.\n        0usize..=3usize,\n        // We leave 50% of the effort exploring more defensively.\n        3usize..=12usize\n    )\n}\n\n// A columnar contains up to 2 docs.\nfn columnar_docs_strategy() -> impl Strategy<Value = Vec<Vec<(&'static str, ColumnValue)>>> {\n    num_docs_strategy()\n        .prop_flat_map(|num_docs| proptest::collection::vec(doc_strategy(), num_docs))\n}\n\nfn permutation_and_subset_strategy(n: usize) -> impl Strategy<Value = Vec<usize>> {\n    let vals: Vec<usize> = (0..n).collect();\n    subsequence(vals, 0..=n).prop_shuffle()\n}\n\nfn build_columnar_with_mapping(docs: &[Vec<(&'static str, ColumnValue)>]) -> ColumnarReader {\n    let num_docs = docs.len() as u32;\n    let mut buffer = Vec::new();\n    let mut columnar_writer = ColumnarWriter::default();\n    for (doc_id, vals) in docs.iter().enumerate() {\n        for (column_name, col_val) in vals {\n            match *col_val {\n                ColumnValue::Str(str_val) => {\n                    columnar_writer.record_str(doc_id as u32, column_name, str_val);\n                }\n                ColumnValue::Bytes(bytes) => {\n                    columnar_writer.record_bytes(doc_id as u32, column_name, bytes)\n                }\n                ColumnValue::Numerical(num) => {\n                    columnar_writer.record_numerical(doc_id as u32, column_name, num);\n                }\n                ColumnValue::IpAddr(ip_addr) => {\n                    columnar_writer.record_ip_addr(doc_id as u32, column_name, ip_addr);\n                }\n                ColumnValue::Bool(bool_val) => {\n                    columnar_writer.record_bool(doc_id as u32, column_name, bool_val);\n                }\n                ColumnValue::DateTime(date_time) => {\n                    columnar_writer.record_datetime(doc_id as u32, column_name, date_time);\n                }\n            }\n        }\n    }\n    columnar_writer.serialize(num_docs, &mut buffer).unwrap();\n\n    ColumnarReader::open(buffer).unwrap()\n}\n\nfn build_columnar(docs: &[Vec<(&'static str, ColumnValue)>]) -> ColumnarReader {\n    build_columnar_with_mapping(docs)\n}\n\nfn assert_columnar_eq_strict(left: &ColumnarReader, right: &ColumnarReader) {\n    assert_columnar_eq(left, right, false);\n}\n\nfn assert_columnar_eq(\n    left: &ColumnarReader,\n    right: &ColumnarReader,\n    lenient_on_numerical_value: bool,\n) {\n    assert_eq!(left.num_docs(), right.num_docs());\n    let left_columns = left.list_columns().unwrap();\n    let right_columns = right.list_columns().unwrap();\n    assert_eq!(left_columns.len(), right_columns.len());\n    for i in 0..left_columns.len() {\n        assert_eq!(left_columns[i].0, right_columns[i].0);\n        let left_column = left_columns[i].1.open().unwrap();\n        let right_column = right_columns[i].1.open().unwrap();\n        assert_dyn_column_eq(&left_column, &right_column, lenient_on_numerical_value);\n    }\n}\n\n#[track_caller]\nfn assert_column_eq<T: Copy + PartialOrd + Debug + Send + Sync + 'static>(\n    left: &Column<T>,\n    right: &Column<T>,\n) {\n    assert_eq!(left.get_cardinality(), right.get_cardinality());\n    assert_eq!(left.num_docs(), right.num_docs());\n    let num_docs = left.num_docs();\n    for doc in 0..num_docs {\n        assert_eq!(\n            left.index.value_row_ids(doc),\n            right.index.value_row_ids(doc)\n        );\n    }\n    assert_eq!(left.values.num_vals(), right.values.num_vals());\n    let num_vals = left.values.num_vals();\n    for i in 0..num_vals {\n        assert_eq!(left.values.get_val(i), right.values.get_val(i));\n    }\n}\n\nfn assert_bytes_column_eq(left: &BytesColumn, right: &BytesColumn) {\n    assert_eq!(\n        left.term_ord_column.get_cardinality(),\n        right.term_ord_column.get_cardinality()\n    );\n    assert_eq!(left.num_rows(), right.num_rows());\n    assert_column_eq(&left.term_ord_column, &right.term_ord_column);\n    assert_eq!(left.dictionary.num_terms(), right.dictionary.num_terms());\n    let num_terms = left.dictionary.num_terms();\n    let mut left_terms = left.dictionary.stream().unwrap();\n    let mut right_terms = right.dictionary.stream().unwrap();\n    for _ in 0..num_terms {\n        assert!(left_terms.advance());\n        assert!(right_terms.advance());\n        assert_eq!(left_terms.key(), right_terms.key());\n    }\n    assert!(!left_terms.advance());\n    assert!(!right_terms.advance());\n}\n\nfn assert_dyn_column_eq(\n    left_dyn_column: &DynamicColumn,\n    right_dyn_column: &DynamicColumn,\n    lenient_on_numerical_value: bool,\n) {\n    assert_eq!(\n        &left_dyn_column.get_cardinality(),\n        &right_dyn_column.get_cardinality()\n    );\n    match &(left_dyn_column, right_dyn_column) {\n        (DynamicColumn::Bool(left_col), DynamicColumn::Bool(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::I64(left_col), DynamicColumn::I64(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::U64(left_col), DynamicColumn::U64(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::F64(left_col), DynamicColumn::F64(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::DateTime(left_col), DynamicColumn::DateTime(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::IpAddr(left_col), DynamicColumn::IpAddr(right_col)) => {\n            assert_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::Bytes(left_col), DynamicColumn::Bytes(right_col)) => {\n            assert_bytes_column_eq(left_col, right_col);\n        }\n        (DynamicColumn::Str(left_col), DynamicColumn::Str(right_col)) => {\n            assert_bytes_column_eq(left_col, right_col);\n        }\n        (left, right) => {\n            if lenient_on_numerical_value {\n                assert_eq!(\n                    ColumnTypeCategory::from(left.column_type()),\n                    ColumnTypeCategory::from(right.column_type())\n                );\n            } else {\n                panic!(\n                    \"Column type are not the same: {:?} vs {:?}\",\n                    left.column_type(),\n                    right.column_type()\n                );\n            }\n        }\n    }\n}\n\ntrait AssertEqualToColumnValue {\n    fn assert_equal_to_column_value(&self, column_value: &ColumnValue);\n}\n\nimpl AssertEqualToColumnValue for bool {\n    fn assert_equal_to_column_value(&self, column_value: &ColumnValue) {\n        let ColumnValue::Bool(val) = column_value else {\n            panic!()\n        };\n        assert_eq!(self, val);\n    }\n}\n\nimpl AssertEqualToColumnValue for Ipv6Addr {\n    fn assert_equal_to_column_value(&self, column_value: &ColumnValue) {\n        let ColumnValue::IpAddr(val) = column_value else {\n            panic!()\n        };\n        assert_eq!(self, val);\n    }\n}\n\nimpl<T: Coerce + PartialEq + Debug + Into<NumericalValue>> AssertEqualToColumnValue for T {\n    fn assert_equal_to_column_value(&self, column_value: &ColumnValue) {\n        let ColumnValue::Numerical(num) = column_value else {\n            panic!()\n        };\n        assert_eq!(self, &T::coerce(*num));\n    }\n}\n\nimpl AssertEqualToColumnValue for DateTime {\n    fn assert_equal_to_column_value(&self, column_value: &ColumnValue) {\n        let ColumnValue::DateTime(dt) = column_value else {\n            panic!()\n        };\n        assert_eq!(self, dt);\n    }\n}\n\nfn assert_column_values<\n    T: AssertEqualToColumnValue + PartialEq + Copy + PartialOrd + Debug + Send + Sync + 'static,\n>(\n    col: &Column<T>,\n    expected: &HashMap<u32, Vec<&ColumnValue>>,\n) {\n    let mut num_non_empty_rows = 0;\n    for doc in 0..col.num_docs() {\n        let doc_vals: Vec<T> = col.values_for_doc(doc).collect();\n        if doc_vals.is_empty() {\n            continue;\n        }\n        num_non_empty_rows += 1;\n        let expected_vals = expected.get(&doc).unwrap();\n        assert_eq!(doc_vals.len(), expected_vals.len());\n        for (val, &expected) in doc_vals.iter().zip(expected_vals.iter()) {\n            val.assert_equal_to_column_value(expected)\n        }\n    }\n    assert_eq!(num_non_empty_rows, expected.len());\n}\n\nfn assert_bytes_column_values(\n    col: &BytesColumn,\n    expected: &HashMap<u32, Vec<&ColumnValue>>,\n    is_str: bool,\n) {\n    let mut num_non_empty_rows = 0;\n    let mut buffer = Vec::new();\n    for doc in 0..col.term_ord_column.num_docs() {\n        let doc_vals: Vec<u64> = col.term_ords(doc).collect();\n        if doc_vals.is_empty() {\n            continue;\n        }\n        let expected_vals = expected.get(&doc).unwrap();\n        assert_eq!(doc_vals.len(), expected_vals.len());\n        for (&expected_col_val, &ord) in expected_vals.iter().zip(&doc_vals) {\n            col.ord_to_bytes(ord, &mut buffer).unwrap();\n            match expected_col_val {\n                ColumnValue::Str(str_val) => {\n                    assert!(is_str);\n                    assert_eq!(str_val.as_bytes(), &buffer);\n                }\n                ColumnValue::Bytes(bytes_val) => {\n                    assert!(!is_str);\n                    assert_eq!(bytes_val, &buffer);\n                }\n                _ => {\n                    panic!();\n                }\n            }\n        }\n        num_non_empty_rows += 1;\n    }\n    assert_eq!(num_non_empty_rows, expected.len());\n}\n\n// This proptest attempts to create a tiny columnar based of up to 3 rows, and checks that the\n// resulting columnar matches the row data.\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(500))]\n    #[test]\n    fn test_single_columnar_builder_proptest(docs in columnar_docs_strategy()) {\n        let columnar = build_columnar(&docs[..]);\n        assert_eq!(columnar.num_docs() as usize, docs.len());\n        let mut expected_columns: HashMap<(&str, ColumnTypeCategory), HashMap<u32, Vec<&ColumnValue>> > = Default::default();\n        for (doc_id, doc_vals) in docs.iter().enumerate() {\n            for (col_name, col_val) in doc_vals {\n                expected_columns\n                    .entry((col_name, col_val.column_type_category()))\n                    .or_default()\n                    .entry(doc_id as u32)\n                    .or_default()\n                    .push(col_val);\n            }\n        }\n        let column_list = columnar.list_columns().unwrap();\n        assert_eq!(expected_columns.len(), column_list.len());\n        for (column_name, column) in column_list {\n            let dynamic_column = column.open().unwrap();\n            let col_category: ColumnTypeCategory = dynamic_column.column_type().into();\n            let expected_col_values: &HashMap<u32, Vec<&ColumnValue>> = expected_columns.get(&(column_name.as_str(), col_category)).unwrap();\n            match &dynamic_column {\n                DynamicColumn::Bool(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::I64(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::U64(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::F64(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::IpAddr(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::DateTime(col) =>\n                    assert_column_values(col, expected_col_values),\n                DynamicColumn::Bytes(col) =>\n                    assert_bytes_column_values(col, expected_col_values, false),\n                DynamicColumn::Str(col) =>\n                    assert_bytes_column_values(col, expected_col_values, true),\n            }\n        }\n    }\n}\n\n// This tests create 2 or 3 random small columnar and attempts to merge them.\n// It compares the resulting merged dataframe with what would have been obtained by building the\n// dataframe from the concatenated rows to begin with.\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(1000))]\n    #[test]\n    fn test_columnar_merge_proptest(columnar_docs in proptest::collection::vec(columnar_docs_strategy(), 2..=3)) {\n        let columnar_readers: Vec<ColumnarReader> = columnar_docs.iter()\n            .map(|docs| build_columnar(&docs[..]))\n            .collect::<Vec<_>>();\n        let columnar_readers_arr: Vec<&ColumnarReader> = columnar_readers.iter().collect();\n        let mut output: Vec<u8> = Vec::new();\n        let stack_merge_order = StackMergeOrder::stack(&columnar_readers_arr[..]).into();\n        crate::merge_columnar(&columnar_readers_arr[..], &[], stack_merge_order, &mut output).unwrap();\n        let merged_columnar = ColumnarReader::open(output).unwrap();\n        let concat_rows: Vec<Vec<(&'static str, ColumnValue)>> = columnar_docs.iter().flatten().cloned().collect();\n        let expected_merged_columnar = build_columnar(&concat_rows[..]);\n        assert_columnar_eq_strict(&merged_columnar, &expected_merged_columnar);\n    }\n}\n\n#[test]\nfn test_columnar_merging_empty_columnar() {\n    let columnar_docs: Vec<Vec<Vec<(&str, ColumnValue)>>> =\n        vec![vec![], vec![vec![(\"c1\", ColumnValue::Str(\"a\"))]]];\n    let columnar_readers: Vec<ColumnarReader> = columnar_docs\n        .iter()\n        .map(|docs| build_columnar(&docs[..]))\n        .collect::<Vec<_>>();\n    let columnar_readers_arr: Vec<&ColumnarReader> = columnar_readers.iter().collect();\n    let mut output: Vec<u8> = Vec::new();\n    let stack_merge_order = StackMergeOrder::stack(&columnar_readers_arr[..]);\n    crate::merge_columnar(\n        &columnar_readers_arr[..],\n        &[],\n        crate::MergeRowOrder::Stack(stack_merge_order),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    let concat_rows: Vec<Vec<(&'static str, ColumnValue)>> =\n        columnar_docs.iter().flatten().cloned().collect();\n    let expected_merged_columnar = build_columnar(&concat_rows[..]);\n    assert_columnar_eq_strict(&merged_columnar, &expected_merged_columnar);\n}\n\n#[test]\nfn test_columnar_merging_number_columns() {\n    let columnar_docs: Vec<Vec<Vec<(&str, ColumnValue)>>> = vec![\n        // columnar 1\n        vec![\n            // doc 1.1\n            vec![(\"c2\", ColumnValue::Numerical(0i64.into()))],\n        ],\n        // columnar2\n        vec![\n            // doc 2.1\n            vec![(\"c2\", ColumnValue::Numerical(0u64.into()))],\n            // doc 2.2\n            vec![(\"c2\", ColumnValue::Numerical(u64::MAX.into()))],\n        ],\n    ];\n    let columnar_readers: Vec<ColumnarReader> = columnar_docs\n        .iter()\n        .map(|docs| build_columnar(&docs[..]))\n        .collect::<Vec<_>>();\n    let columnar_readers_arr: Vec<&ColumnarReader> = columnar_readers.iter().collect();\n    let mut output: Vec<u8> = Vec::new();\n    let stack_merge_order = StackMergeOrder::stack(&columnar_readers_arr[..]);\n    crate::merge_columnar(\n        &columnar_readers_arr[..],\n        &[],\n        crate::MergeRowOrder::Stack(stack_merge_order),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    let concat_rows: Vec<Vec<(&'static str, ColumnValue)>> =\n        columnar_docs.iter().flatten().cloned().collect();\n    let expected_merged_columnar = build_columnar(&concat_rows[..]);\n    assert_columnar_eq_strict(&merged_columnar, &expected_merged_columnar);\n}\n\n// TODO add non trivial remap and merge\n// TODO test required_columns\n// TODO document edge case: required_columns incompatible with values.\n\n#[allow(clippy::type_complexity)]\nfn columnar_docs_and_remap()\n-> impl Strategy<Value = (Vec<Vec<Vec<(&'static str, ColumnValue)>>>, Vec<RowAddr>)> {\n    proptest::collection::vec(columnar_docs_strategy(), 2..=3).prop_flat_map(\n        |columnars_docs: Vec<Vec<Vec<(&str, ColumnValue)>>>| {\n            let row_addrs: Vec<RowAddr> = columnars_docs\n                .iter()\n                .enumerate()\n                .flat_map(|(segment_ord, columnar_docs)| {\n                    (0u32..columnar_docs.len() as u32).map(move |row_id| RowAddr {\n                        segment_ord: segment_ord as u32,\n                        row_id,\n                    })\n                })\n                .collect();\n            permutation_and_subset_strategy(row_addrs.len()).prop_map(move |shuffled_subset| {\n                let shuffled_row_addr_subset: Vec<RowAddr> =\n                    shuffled_subset.iter().map(|ord| row_addrs[*ord]).collect();\n                (columnars_docs.clone(), shuffled_row_addr_subset)\n            })\n        },\n    )\n}\n\nproptest! {\n    #![proptest_config(ProptestConfig::with_cases(1000))]\n    #[test]\n    fn test_columnar_merge_and_remap_proptest((columnar_docs, shuffle_merge_order) in\ncolumnar_docs_and_remap()) {\n        test_columnar_merge_and_remap(columnar_docs, shuffle_merge_order);\n    }\n}\n\nfn test_columnar_merge_and_remap(\n    columnar_docs: Vec<Vec<Vec<(&'static str, ColumnValue)>>>,\n    shuffle_merge_order: Vec<RowAddr>,\n) {\n    let shuffled_rows: Vec<Vec<(&'static str, ColumnValue)>> = shuffle_merge_order\n        .iter()\n        .map(|row_addr| {\n            columnar_docs[row_addr.segment_ord as usize][row_addr.row_id as usize].clone()\n        })\n        .collect();\n    let expected_merged_columnar = build_columnar(&shuffled_rows[..]);\n    let columnar_readers: Vec<ColumnarReader> = columnar_docs\n        .iter()\n        .map(|docs| build_columnar(&docs[..]))\n        .collect::<Vec<_>>();\n    let columnar_readers_ref: Vec<&ColumnarReader> = columnar_readers.iter().collect();\n    let mut output: Vec<u8> = Vec::new();\n    let segment_num_rows: Vec<RowId> = columnar_docs\n        .iter()\n        .map(|docs| docs.len() as RowId)\n        .collect();\n    let shuffle_merge_order = ShuffleMergeOrder::for_test(&segment_num_rows, shuffle_merge_order);\n    crate::merge_columnar(\n        &columnar_readers_ref[..],\n        &[],\n        shuffle_merge_order.into(),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    assert_columnar_eq(&merged_columnar, &expected_merged_columnar, true);\n}\n\n#[test]\nfn test_columnar_merge_and_remap_bug_1() {\n    let columnar_docs = vec![vec![\n        vec![\n            (\"c1\", ColumnValue::Numerical(NumericalValue::U64(0))),\n            (\"c1\", ColumnValue::Numerical(NumericalValue::U64(0))),\n        ],\n        vec![],\n    ]];\n    let shuffle_merge_order: Vec<RowAddr> = vec![\n        RowAddr {\n            segment_ord: 0,\n            row_id: 1,\n        },\n        RowAddr {\n            segment_ord: 0,\n            row_id: 0,\n        },\n    ];\n\n    test_columnar_merge_and_remap(columnar_docs, shuffle_merge_order);\n}\n\n#[test]\nfn test_columnar_merge_empty() {\n    let columnar_reader_1 = build_columnar(&[]);\n    let rows: &[Vec<_>] = &[vec![(\"c1\", ColumnValue::Str(\"a\"))]][..];\n    let columnar_reader_2 = build_columnar(rows);\n    let mut output: Vec<u8> = Vec::new();\n    let segment_num_rows: Vec<RowId> = vec![0, 0];\n    let shuffle_merge_order = ShuffleMergeOrder::for_test(&segment_num_rows, vec![]);\n    crate::merge_columnar(\n        &[&columnar_reader_1, &columnar_reader_2],\n        &[],\n        shuffle_merge_order.into(),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    assert_eq!(merged_columnar.num_docs(), 0);\n    assert_eq!(merged_columnar.num_columns(), 0);\n}\n\n#[test]\nfn test_columnar_merge_single_str_column() {\n    let columnar_reader_1 = build_columnar(&[]);\n    let rows: &[Vec<_>] = &[vec![(\"c1\", ColumnValue::Str(\"a\"))]][..];\n    let columnar_reader_2 = build_columnar(rows);\n    let mut output: Vec<u8> = Vec::new();\n    let segment_num_rows: Vec<RowId> = vec![0, 1];\n    let shuffle_merge_order = ShuffleMergeOrder::for_test(\n        &segment_num_rows,\n        vec![RowAddr {\n            segment_ord: 1u32,\n            row_id: 0u32,\n        }],\n    );\n    crate::merge_columnar(\n        &[&columnar_reader_1, &columnar_reader_2],\n        &[],\n        shuffle_merge_order.into(),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    assert_eq!(merged_columnar.num_docs(), 1);\n    assert_eq!(merged_columnar.num_columns(), 1);\n}\n\n#[test]\nfn test_delete_decrease_cardinality() {\n    let columnar_reader_1 = build_columnar(&[]);\n    let rows: &[Vec<_>] = &[\n        vec![\n            (\"c\", ColumnValue::from(0i64)),\n            (\"c\", ColumnValue::from(0i64)),\n        ],\n        vec![(\"c\", ColumnValue::from(0i64))],\n    ][..];\n    // c is multivalued here\n    let columnar_reader_2 = build_columnar(rows);\n    let mut output: Vec<u8> = Vec::new();\n    let shuffle_merge_order = ShuffleMergeOrder::for_test(\n        &[0, 2],\n        vec![RowAddr {\n            segment_ord: 1u32,\n            row_id: 1u32,\n        }],\n    );\n    crate::merge_columnar(\n        &[&columnar_reader_1, &columnar_reader_2],\n        &[],\n        shuffle_merge_order.into(),\n        &mut output,\n    )\n    .unwrap();\n    let merged_columnar = ColumnarReader::open(output).unwrap();\n    assert_eq!(merged_columnar.num_docs(), 1);\n    assert_eq!(merged_columnar.num_columns(), 1);\n    let cols = merged_columnar.read_columns(\"c\").unwrap();\n    assert_eq!(cols.len(), 1);\n    assert_eq!(cols[0].column_type(), ColumnType::I64);\n    assert_eq!(cols[0].open().unwrap().get_cardinality(), Cardinality::Full);\n}\n"
  },
  {
    "path": "columnar/src/utils.rs",
    "content": "const fn compute_mask(num_bits: u8) -> u8 {\n    if num_bits == 8 {\n        u8::MAX\n    } else {\n        (1u8 << num_bits) - 1\n    }\n}\n\n#[inline(always)]\n#[must_use]\npub(crate) fn select_bits<const START: u8, const END: u8>(code: u8) -> u8 {\n    assert!(START <= END);\n    assert!(END <= 8);\n    let num_bits: u8 = END - START;\n    let mask: u8 = compute_mask(num_bits);\n    (code >> START) & mask\n}\n\n#[inline(always)]\n#[must_use]\npub(crate) fn place_bits<const START: u8, const END: u8>(code: u8) -> u8 {\n    assert!(START <= END);\n    assert!(END <= 8);\n    let num_bits: u8 = END - START;\n    let mask: u8 = compute_mask(num_bits);\n    assert!(code <= mask);\n    code << START\n}\n\n/// Pop-front one bytes from a slice of bytes.\n#[inline(always)]\npub fn pop_first_byte(bytes: &mut &[u8]) -> Option<u8> {\n    if bytes.is_empty() {\n        return None;\n    }\n    let first_byte = bytes[0];\n    *bytes = &bytes[1..];\n    Some(first_byte)\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_select_bits() {\n        assert_eq!(255u8, select_bits::<0, 8>(255u8));\n        assert_eq!(0u8, select_bits::<0, 0>(255u8));\n        assert_eq!(8u8, select_bits::<0, 4>(8u8));\n        assert_eq!(4u8, select_bits::<1, 4>(8u8));\n        assert_eq!(0u8, select_bits::<1, 3>(8u8));\n    }\n\n    #[test]\n    fn test_place_bits() {\n        assert_eq!(255u8, place_bits::<0, 8>(255u8));\n        assert_eq!(4u8, place_bits::<2, 3>(1u8));\n        assert_eq!(0u8, place_bits::<2, 2>(0u8));\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_place_bits_overflows() {\n        let _ = place_bits::<1, 4>(8u8);\n    }\n\n    #[test]\n    fn test_pop_first_byte() {\n        let mut cursor: &[u8] = &b\"abcd\"[..];\n        assert_eq!(pop_first_byte(&mut cursor), Some(b'a'));\n        assert_eq!(pop_first_byte(&mut cursor), Some(b'b'));\n        assert_eq!(pop_first_byte(&mut cursor), Some(b'c'));\n        assert_eq!(pop_first_byte(&mut cursor), Some(b'd'));\n        assert_eq!(pop_first_byte(&mut cursor), None);\n    }\n}\n"
  },
  {
    "path": "columnar/src/value.rs",
    "content": "use std::str::FromStr;\n\nuse common::DateTime;\n\nuse crate::InvalidData;\n\n#[derive(Copy, Clone, PartialEq, Debug)]\npub enum NumericalValue {\n    I64(i64),\n    U64(u64),\n    F64(f64),\n}\n\nimpl FromStr for NumericalValue {\n    type Err = ();\n\n    fn from_str(s: &str) -> Result<Self, ()> {\n        if let Ok(val_i64) = s.parse::<i64>() {\n            return Ok(val_i64.into());\n        }\n        if let Ok(val_u64) = s.parse::<u64>() {\n            return Ok(val_u64.into());\n        }\n        if let Ok(val_f64) = s.parse::<f64>() {\n            return Ok(NumericalValue::from(val_f64).normalize());\n        }\n        Err(())\n    }\n}\n\nimpl NumericalValue {\n    pub fn numerical_type(&self) -> NumericalType {\n        match self {\n            NumericalValue::I64(_) => NumericalType::I64,\n            NumericalValue::U64(_) => NumericalType::U64,\n            NumericalValue::F64(_) => NumericalType::F64,\n        }\n    }\n\n    /// Tries to normalize the numerical value in the following priorities:\n    /// i64, i64, f64\n    pub fn normalize(self) -> Self {\n        match self {\n            NumericalValue::U64(val) => {\n                if val <= i64::MAX as u64 {\n                    NumericalValue::I64(val as i64)\n                } else {\n                    NumericalValue::U64(val)\n                }\n            }\n            NumericalValue::I64(val) => NumericalValue::I64(val),\n            NumericalValue::F64(val) => {\n                let fract = val.fract();\n                if fract == 0.0 && val >= i64::MIN as f64 && val <= i64::MAX as f64 {\n                    NumericalValue::I64(val as i64)\n                } else if fract == 0.0 && val >= u64::MIN as f64 && val <= u64::MAX as f64 {\n                    NumericalValue::U64(val as u64)\n                } else {\n                    NumericalValue::F64(val)\n                }\n            }\n        }\n    }\n}\n\nimpl From<u64> for NumericalValue {\n    fn from(val: u64) -> NumericalValue {\n        NumericalValue::U64(val)\n    }\n}\n\nimpl From<i64> for NumericalValue {\n    fn from(val: i64) -> Self {\n        NumericalValue::I64(val)\n    }\n}\n\nimpl From<f64> for NumericalValue {\n    fn from(val: f64) -> Self {\n        NumericalValue::F64(val)\n    }\n}\n\n#[derive(Clone, Copy, Debug, Default, Hash, Eq, PartialEq)]\n#[repr(u8)]\npub enum NumericalType {\n    #[default]\n    I64 = 0,\n    U64 = 1,\n    F64 = 2,\n}\n\nimpl NumericalType {\n    pub fn to_code(self) -> u8 {\n        self as u8\n    }\n\n    pub fn try_from_code(code: u8) -> Result<NumericalType, InvalidData> {\n        match code {\n            0 => Ok(NumericalType::I64),\n            1 => Ok(NumericalType::U64),\n            2 => Ok(NumericalType::F64),\n            _ => Err(InvalidData),\n        }\n    }\n}\n\n/// We voluntarily avoid using `Into` here to keep this\n/// implementation quirk as private as possible.\n///\n/// # Panics\n/// This coercion trait actually panics if it is used\n/// to convert a loose types to a stricter type.\n///\n/// The level is strictness is somewhat arbitrary.\n/// - i64\n/// - u64\n/// - f64.\npub(crate) trait Coerce {\n    fn coerce(numerical_value: NumericalValue) -> Self;\n}\n\nimpl Coerce for i64 {\n    fn coerce(value: NumericalValue) -> Self {\n        match value {\n            NumericalValue::I64(val) => val,\n            NumericalValue::U64(val) => val as i64,\n            NumericalValue::F64(_) => unreachable!(),\n        }\n    }\n}\n\nimpl Coerce for u64 {\n    fn coerce(value: NumericalValue) -> Self {\n        match value {\n            NumericalValue::I64(val) => val as u64,\n            NumericalValue::U64(val) => val,\n            NumericalValue::F64(_) => unreachable!(),\n        }\n    }\n}\n\nimpl Coerce for f64 {\n    fn coerce(value: NumericalValue) -> Self {\n        match value {\n            NumericalValue::I64(val) => val as f64,\n            NumericalValue::U64(val) => val as f64,\n            NumericalValue::F64(val) => val,\n        }\n    }\n}\n\nimpl Coerce for DateTime {\n    fn coerce(value: NumericalValue) -> Self {\n        let timestamp_micros = i64::coerce(value);\n        DateTime::from_timestamp_nanos(timestamp_micros)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::NumericalType;\n    use crate::NumericalValue;\n\n    #[test]\n    fn test_numerical_type_code() {\n        let mut num_numerical_type = 0;\n        for code in u8::MIN..=u8::MAX {\n            if let Ok(numerical_type) = NumericalType::try_from_code(code) {\n                assert_eq!(numerical_type.to_code(), code);\n                num_numerical_type += 1;\n            }\n        }\n        assert_eq!(num_numerical_type, 3);\n    }\n\n    #[test]\n    fn test_parse_numerical() {\n        assert_eq!(\n            \"123\".parse::<NumericalValue>().unwrap(),\n            NumericalValue::I64(123)\n        );\n        assert_eq!(\n            \"18446744073709551615\".parse::<NumericalValue>().unwrap(),\n            NumericalValue::U64(18446744073709551615u64)\n        );\n        assert_eq!(\n            \"1.0\".parse::<NumericalValue>().unwrap(),\n            NumericalValue::I64(1i64)\n        );\n        assert_eq!(\n            \"1.1\".parse::<NumericalValue>().unwrap(),\n            NumericalValue::F64(1.1f64)\n        );\n        assert_eq!(\n            \"-1.0\".parse::<NumericalValue>().unwrap(),\n            NumericalValue::I64(-1i64)\n        );\n    }\n\n    #[test]\n    fn test_normalize_numerical() {\n        assert_eq!(\n            NumericalValue::from(1u64).normalize(),\n            NumericalValue::I64(1i64),\n        );\n        let limit_val = i64::MAX as u64 + 1u64;\n        assert_eq!(\n            NumericalValue::from(limit_val).normalize(),\n            NumericalValue::U64(limit_val),\n        );\n        assert_eq!(\n            NumericalValue::from(-1i64).normalize(),\n            NumericalValue::I64(-1i64),\n        );\n        assert_eq!(\n            NumericalValue::from(-2.0f64).normalize(),\n            NumericalValue::I64(-2i64),\n        );\n        assert_eq!(\n            NumericalValue::from(-2.1f64).normalize(),\n            NumericalValue::F64(-2.1f64),\n        );\n        let large_float = 2.0f64.powf(70.0f64);\n        assert_eq!(\n            NumericalValue::from(large_float).normalize(),\n            NumericalValue::F64(large_float),\n        );\n    }\n}\n"
  },
  {
    "path": "common/Cargo.toml",
    "content": "[package]\nname = \"tantivy-common\"\nversion = \"0.10.0\"\nauthors = [\"Paul Masurel <paul@quickwit.io>\", \"Pascal Seitz <pascal@quickwit.io>\"]\nlicense = \"MIT\"\nedition = \"2024\"\ndescription = \"common traits and utility functions used by multiple tantivy subcrates\"\ndocumentation = \"https://docs.rs/tantivy_common/\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nbyteorder = \"1.4.3\"\nownedbytes = { version= \"0.9\", path=\"../ownedbytes\" }\nasync-trait = \"0.1\"\ntime = { version = \"0.3.47\", features = [\"serde-well-known\"] }\nserde = { version = \"1.0.136\", features = [\"derive\"] }\n\n[dev-dependencies]\nbinggan = \"0.14.0\"\nproptest = \"1.0.0\"\nrand = \"0.9\"\n"
  },
  {
    "path": "common/benches/bench.rs",
    "content": "use binggan::{BenchRunner, black_box};\nuse rand::rng;\nuse rand::seq::IteratorRandom;\nuse tantivy_common::{BitSet, TinySet, serialize_vint_u32};\n\nfn bench_vint() {\n    let mut runner = BenchRunner::new();\n\n    let vals: Vec<u32> = (0..20_000).collect();\n    runner.bench_function(\"bench_vint\", move |_| {\n        let mut out = 0u64;\n        for val in vals.iter().cloned() {\n            let mut buf = [0u8; 8];\n            serialize_vint_u32(val, &mut buf);\n            out += u64::from(buf[0]);\n        }\n        black_box(out);\n    });\n\n    let vals: Vec<u32> = (0..20_000).choose_multiple(&mut rng(), 100_000);\n    runner.bench_function(\"bench_vint_rand\", move |_| {\n        let mut out = 0u64;\n        for val in vals.iter().cloned() {\n            let mut buf = [0u8; 8];\n            serialize_vint_u32(val, &mut buf);\n            out += u64::from(buf[0]);\n        }\n        black_box(out);\n    });\n}\n\nfn bench_bitset() {\n    let mut runner = BenchRunner::new();\n\n    runner.bench_function(\"bench_tinyset_pop\", move |_| {\n        let mut tinyset = TinySet::singleton(black_box(31u32));\n        tinyset.pop_lowest();\n        tinyset.pop_lowest();\n        tinyset.pop_lowest();\n        tinyset.pop_lowest();\n        tinyset.pop_lowest();\n        tinyset.pop_lowest();\n        black_box(tinyset);\n    });\n\n    let tiny_set = TinySet::empty().insert(10u32).insert(14u32).insert(21u32);\n    runner.bench_function(\"bench_tinyset_sum\", move |_| {\n        assert_eq!(black_box(tiny_set).into_iter().sum::<u32>(), 45u32);\n    });\n\n    let v = [10u32, 14u32, 21u32];\n    runner.bench_function(\"bench_tinyarr_sum\", move |_| {\n        black_box(v.iter().cloned().sum::<u32>());\n    });\n\n    runner.bench_function(\"bench_bitset_initialize\", move |_| {\n        black_box(BitSet::with_max_value(1_000_000));\n    });\n}\n\nfn main() {\n    bench_vint();\n    bench_bitset();\n}\n"
  },
  {
    "path": "common/src/bitset.rs",
    "content": "use std::io::Write;\nuse std::{fmt, io};\n\nuse ownedbytes::OwnedBytes;\n\nuse crate::ByteCount;\n\n#[derive(Clone, Copy, Eq, PartialEq)]\npub struct TinySet(u64);\n\nimpl fmt::Debug for TinySet {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        self.into_iter().collect::<Vec<u32>>().fmt(f)\n    }\n}\n\npub struct TinySetIterator(TinySet);\nimpl Iterator for TinySetIterator {\n    type Item = u32;\n\n    #[inline]\n    fn next(&mut self) -> Option<Self::Item> {\n        self.0.pop_lowest()\n    }\n}\n\nimpl IntoIterator for TinySet {\n    type Item = u32;\n    type IntoIter = TinySetIterator;\n    fn into_iter(self) -> Self::IntoIter {\n        TinySetIterator(self)\n    }\n}\n\nimpl TinySet {\n    pub fn serialize<T: Write>(&self, writer: &mut T) -> io::Result<()> {\n        writer.write_all(self.0.to_le_bytes().as_ref())\n    }\n\n    pub fn into_bytes(self) -> [u8; 8] {\n        self.0.to_le_bytes()\n    }\n\n    #[inline]\n    pub fn deserialize(data: [u8; 8]) -> Self {\n        let val: u64 = u64::from_le_bytes(data);\n        TinySet(val)\n    }\n\n    /// Returns an empty `TinySet`.\n    #[inline]\n    pub fn empty() -> TinySet {\n        TinySet(0u64)\n    }\n\n    /// Returns a full `TinySet`.\n    #[inline]\n    pub fn full() -> TinySet {\n        TinySet::empty().complement()\n    }\n\n    pub fn clear(&mut self) {\n        self.0 = 0u64;\n    }\n\n    /// Returns the complement of the set in `[0, 64[`.\n    ///\n    /// Careful on making this function public, as it will break the padding handling in the last\n    /// bucket.\n    #[inline]\n    fn complement(self) -> TinySet {\n        TinySet(!self.0)\n    }\n\n    /// Returns true iff the `TinySet` contains the element `el`.\n    #[inline]\n    pub fn contains(self, el: u32) -> bool {\n        !self.intersect(TinySet::singleton(el)).is_empty()\n    }\n\n    /// Returns the number of elements in the TinySet.\n    #[inline]\n    pub fn len(self) -> u32 {\n        self.0.count_ones()\n    }\n\n    /// Returns the intersection of `self` and `other`\n    #[inline]\n    #[must_use]\n    pub fn intersect(self, other: TinySet) -> TinySet {\n        TinySet(self.0 & other.0)\n    }\n\n    /// Creates a new `TinySet` containing only one element\n    /// within `[0; 64[`\n    #[inline]\n    pub fn singleton(el: u32) -> TinySet {\n        TinySet(1u64 << u64::from(el))\n    }\n\n    /// Insert a new element within [0..64)\n    #[inline]\n    #[must_use]\n    pub fn insert(self, el: u32) -> TinySet {\n        self.union(TinySet::singleton(el))\n    }\n\n    /// Removes an element within [0..64)\n    #[inline]\n    #[must_use]\n    pub fn remove(self, el: u32) -> TinySet {\n        self.intersect(TinySet::singleton(el).complement())\n    }\n\n    /// Insert a new element within [0..64)\n    ///\n    /// returns true if the set changed\n    #[inline]\n    pub fn insert_mut(&mut self, el: u32) -> bool {\n        let old = *self;\n        *self = old.insert(el);\n        old != *self\n    }\n\n    /// Remove a element within [0..64)\n    ///\n    /// returns true if the set changed\n    #[inline]\n    pub fn remove_mut(&mut self, el: u32) -> bool {\n        let old = *self;\n        *self = old.remove(el);\n        old != *self\n    }\n\n    /// Returns the union of two tinysets\n    #[inline]\n    #[must_use]\n    pub fn union(self, other: TinySet) -> TinySet {\n        TinySet(self.0 | other.0)\n    }\n\n    /// Returns true iff the `TinySet` is empty.\n    #[inline]\n    pub fn is_empty(self) -> bool {\n        self.0 == 0u64\n    }\n\n    /// Returns the lowest element in the `TinySet`\n    /// and removes it.\n    #[inline]\n    pub fn pop_lowest(&mut self) -> Option<u32> {\n        if self.is_empty() {\n            None\n        } else {\n            let lowest = self.0.trailing_zeros();\n            self.0 ^= TinySet::singleton(lowest).0;\n            Some(lowest)\n        }\n    }\n\n    /// Returns a `TinySet` than contains all values up\n    /// to limit excluded.\n    ///\n    /// The limit is assumed to be strictly lower than 64.\n    pub fn range_lower(upper_bound: u32) -> TinySet {\n        TinySet((1u64 << u64::from(upper_bound % 64u32)) - 1u64)\n    }\n\n    /// Returns a `TinySet` that contains all values greater\n    /// or equal to the given limit, included. (and up to 63)\n    ///\n    /// The limit is assumed to be strictly lower than 64.\n    pub fn range_greater_or_equal(from_included: u32) -> TinySet {\n        TinySet::range_lower(from_included).complement()\n    }\n}\n\n#[derive(Clone)]\npub struct BitSet {\n    tinysets: Box<[TinySet]>,\n    len: u64,\n    max_value: u32,\n}\nimpl std::fmt::Debug for BitSet {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        f.debug_struct(\"BitSet\")\n            .field(\"len\", &self.len)\n            .field(\"max_value\", &self.max_value)\n            .finish()\n    }\n}\n\nfn num_buckets(max_val: u32) -> u32 {\n    max_val.div_ceil(64u32)\n}\n\nimpl BitSet {\n    /// serialize a `BitSet`.\n    pub fn serialize<T: Write>(&self, writer: &mut T) -> io::Result<()> {\n        writer.write_all(self.max_value.to_le_bytes().as_ref())?;\n        for tinyset in self.tinysets.iter().cloned() {\n            writer.write_all(&tinyset.into_bytes())?;\n        }\n        writer.flush()?;\n        Ok(())\n    }\n\n    /// Create a new `BitSet` that may contain elements\n    /// within `[0, max_val)`.\n    pub fn with_max_value(max_value: u32) -> BitSet {\n        let num_buckets = num_buckets(max_value);\n        let tinybitsets = vec![TinySet::empty(); num_buckets as usize].into_boxed_slice();\n        BitSet {\n            tinysets: tinybitsets,\n            len: 0,\n            max_value,\n        }\n    }\n\n    /// Create a new `BitSet` that may contain elements. Initially all values will be set.\n    /// within `[0, max_val)`.\n    pub fn with_max_value_and_full(max_value: u32) -> BitSet {\n        let num_buckets = num_buckets(max_value);\n        let mut tinybitsets = vec![TinySet::full(); num_buckets as usize].into_boxed_slice();\n\n        // Fix padding\n        let lower = max_value % 64u32;\n        if lower != 0 {\n            tinybitsets[tinybitsets.len() - 1] = TinySet::range_lower(lower);\n        }\n        BitSet {\n            tinysets: tinybitsets,\n            len: max_value as u64,\n            max_value,\n        }\n    }\n\n    /// Removes all elements from the `BitSet`.\n    pub fn clear(&mut self) {\n        for tinyset in self.tinysets.iter_mut() {\n            *tinyset = TinySet::empty();\n        }\n    }\n\n    /// Intersect with serialized bitset\n    pub fn intersect_update(&mut self, other: &ReadOnlyBitSet) {\n        self.intersect_update_with_iter(other.iter_tinysets());\n    }\n\n    /// Intersect with tinysets\n    fn intersect_update_with_iter(&mut self, other: impl Iterator<Item = TinySet>) {\n        self.len = 0;\n        for (left, right) in self.tinysets.iter_mut().zip(other) {\n            *left = left.intersect(right);\n            self.len += left.len() as u64;\n        }\n    }\n\n    /// Returns the number of elements in the `BitSet`.\n    #[inline]\n    pub fn len(&self) -> usize {\n        self.len as usize\n    }\n\n    /// Inserts an element in the `BitSet`\n    #[inline]\n    pub fn insert(&mut self, el: u32) {\n        // we do not check saturated els.\n        let higher = el / 64u32;\n        let lower = el % 64u32;\n        self.len += u64::from(self.tinysets[higher as usize].insert_mut(lower));\n    }\n\n    /// Inserts an element in the `BitSet`\n    #[inline]\n    pub fn remove(&mut self, el: u32) {\n        // we do not check saturated els.\n        let higher = el / 64u32;\n        let lower = el % 64u32;\n        self.len -= u64::from(self.tinysets[higher as usize].remove_mut(lower));\n    }\n\n    /// Returns true iff the elements is in the `BitSet`.\n    #[inline]\n    pub fn contains(&self, el: u32) -> bool {\n        self.tinyset(el / 64u32).contains(el % 64)\n    }\n\n    /// Returns the first non-empty `TinySet` associated with a bucket lower\n    /// or greater than bucket.\n    ///\n    /// Reminder: the tiny set with the bucket `bucket`, represents the\n    /// elements from `bucket * 64` to `(bucket+1) * 64`.\n    pub fn first_non_empty_bucket(&self, bucket: u32) -> Option<u32> {\n        self.tinysets[bucket as usize..]\n            .iter()\n            .cloned()\n            .position(|tinyset| !tinyset.is_empty())\n            .map(|delta_bucket| bucket + delta_bucket as u32)\n    }\n\n    #[inline]\n    pub fn max_value(&self) -> u32 {\n        self.max_value\n    }\n\n    /// Returns the tiny bitset representing the\n    /// the set restricted to the number range from\n    /// `bucket * 64` to `(bucket + 1) * 64`.\n    pub fn tinyset(&self, bucket: u32) -> TinySet {\n        self.tinysets[bucket as usize]\n    }\n}\n\n/// Serialized BitSet.\n#[derive(Clone)]\npub struct ReadOnlyBitSet {\n    data: OwnedBytes,\n    max_value: u32,\n}\n\npub fn intersect_bitsets(left: &ReadOnlyBitSet, other: &ReadOnlyBitSet) -> ReadOnlyBitSet {\n    assert_eq!(left.max_value(), other.max_value());\n    assert_eq!(left.data.len(), other.data.len());\n    let union_tinyset_it = left\n        .iter_tinysets()\n        .zip(other.iter_tinysets())\n        .map(|(left_tinyset, right_tinyset)| left_tinyset.intersect(right_tinyset));\n    let mut output_dataset: Vec<u8> = Vec::with_capacity(left.data.len());\n    for tinyset in union_tinyset_it {\n        output_dataset.extend_from_slice(&tinyset.into_bytes());\n    }\n    ReadOnlyBitSet {\n        data: OwnedBytes::new(output_dataset),\n        max_value: left.max_value(),\n    }\n}\n\nimpl ReadOnlyBitSet {\n    pub fn open(data: OwnedBytes) -> Self {\n        let (max_value_data, data) = data.split(4);\n        assert_eq!(data.len() % 8, 0);\n        let max_value: u32 = u32::from_le_bytes(max_value_data.as_ref().try_into().unwrap());\n        ReadOnlyBitSet { data, max_value }\n    }\n\n    /// Number of elements in the bitset.\n    #[inline]\n    pub fn len(&self) -> usize {\n        self.iter_tinysets()\n            .map(|tinyset| tinyset.len() as usize)\n            .sum()\n    }\n\n    /// Iterate the tinyset on the fly from serialized data.\n    #[inline]\n    fn iter_tinysets(&self) -> impl Iterator<Item = TinySet> + '_ {\n        self.data.chunks_exact(8).map(move |chunk| {\n            let tinyset: TinySet = TinySet::deserialize(chunk.try_into().unwrap());\n            tinyset\n        })\n    }\n\n    /// Iterate over the positions of the elements.\n    #[inline]\n    pub fn iter(&self) -> impl Iterator<Item = u32> + '_ {\n        self.iter_tinysets()\n            .enumerate()\n            .flat_map(move |(chunk_num, tinyset)| {\n                let chunk_base_val = chunk_num as u32 * 64;\n                tinyset\n                    .into_iter()\n                    .map(move |val| val + chunk_base_val)\n                    .take_while(move |doc| *doc < self.max_value)\n            })\n    }\n\n    /// Returns true iff the elements is in the `BitSet`.\n    #[inline]\n    pub fn contains(&self, el: u32) -> bool {\n        let byte_offset = el / 8u32;\n        let b: u8 = self.data[byte_offset as usize];\n        let shift = (el % 8) as u8;\n        b & (1u8 << shift) != 0\n    }\n\n    /// Maximum value the bitset may contain.\n    /// (Note this is not the maximum value contained in the set.)\n    ///\n    /// A bitset has an intrinsic capacity.\n    /// It only stores elements within [0..max_value).\n    #[inline]\n    pub fn max_value(&self) -> u32 {\n        self.max_value\n    }\n\n    /// Number of bytes used in the bitset representation.\n    pub fn num_bytes(&self) -> ByteCount {\n        self.data.len().into()\n    }\n}\n\nimpl<'a> From<&'a BitSet> for ReadOnlyBitSet {\n    fn from(bitset: &'a BitSet) -> ReadOnlyBitSet {\n        let mut buffer = Vec::with_capacity(bitset.tinysets.len() * 8 + 4);\n        bitset\n            .serialize(&mut buffer)\n            .expect(\"serializing into a buffer should never fail\");\n        ReadOnlyBitSet::open(OwnedBytes::new(buffer))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::HashSet;\n\n    use ownedbytes::OwnedBytes;\n    use rand::distr::Bernoulli;\n    use rand::rngs::StdRng;\n    use rand::{Rng, SeedableRng};\n\n    use super::{BitSet, ReadOnlyBitSet, TinySet};\n\n    #[test]\n    fn test_read_serialized_bitset_full_multi() {\n        for i in 0..1000 {\n            let bitset = BitSet::with_max_value_and_full(i);\n            let mut out = vec![];\n            bitset.serialize(&mut out).unwrap();\n\n            let bitset = ReadOnlyBitSet::open(OwnedBytes::new(out));\n            assert_eq!(bitset.len(), i as usize);\n        }\n    }\n\n    #[test]\n    fn test_read_serialized_bitset_full_block() {\n        let bitset = BitSet::with_max_value_and_full(64);\n        let mut out = vec![];\n        bitset.serialize(&mut out).unwrap();\n\n        let bitset = ReadOnlyBitSet::open(OwnedBytes::new(out));\n        assert_eq!(bitset.len(), 64);\n    }\n\n    #[test]\n    fn test_read_serialized_bitset_full() {\n        let mut bitset = BitSet::with_max_value_and_full(5);\n        bitset.remove(3);\n        let mut out = vec![];\n        bitset.serialize(&mut out).unwrap();\n\n        let bitset = ReadOnlyBitSet::open(OwnedBytes::new(out));\n        assert_eq!(bitset.len(), 4);\n    }\n\n    #[test]\n    fn test_bitset_intersect() {\n        let bitset_serialized = {\n            let mut bitset = BitSet::with_max_value_and_full(5);\n            bitset.remove(1);\n            bitset.remove(3);\n            let mut out = vec![];\n            bitset.serialize(&mut out).unwrap();\n\n            ReadOnlyBitSet::open(OwnedBytes::new(out))\n        };\n\n        let mut bitset = BitSet::with_max_value_and_full(5);\n        bitset.remove(1);\n        bitset.intersect_update(&bitset_serialized);\n\n        assert!(bitset.contains(0));\n        assert!(!bitset.contains(1));\n        assert!(bitset.contains(2));\n        assert!(!bitset.contains(3));\n        assert!(bitset.contains(4));\n\n        bitset.intersect_update_with_iter(vec![TinySet::singleton(0)].into_iter());\n\n        assert!(bitset.contains(0));\n        assert!(!bitset.contains(1));\n        assert!(!bitset.contains(2));\n        assert!(!bitset.contains(3));\n        assert!(!bitset.contains(4));\n        assert_eq!(bitset.len(), 1);\n\n        bitset.intersect_update_with_iter(vec![TinySet::singleton(1)].into_iter());\n        assert!(!bitset.contains(0));\n        assert!(!bitset.contains(1));\n        assert!(!bitset.contains(2));\n        assert!(!bitset.contains(3));\n        assert!(!bitset.contains(4));\n        assert_eq!(bitset.len(), 0);\n    }\n\n    #[test]\n    fn test_read_serialized_bitset_empty() {\n        let mut bitset = BitSet::with_max_value(5);\n        bitset.insert(3);\n        let mut out = vec![];\n        bitset.serialize(&mut out).unwrap();\n\n        let bitset = ReadOnlyBitSet::open(OwnedBytes::new(out));\n        assert_eq!(bitset.len(), 1);\n\n        {\n            let bitset = BitSet::with_max_value(5);\n            let mut out = vec![];\n            bitset.serialize(&mut out).unwrap();\n            let bitset = ReadOnlyBitSet::open(OwnedBytes::new(out));\n            assert_eq!(bitset.len(), 0);\n        }\n    }\n\n    #[test]\n    fn test_tiny_set_remove() {\n        {\n            let mut u = TinySet::empty().insert(63u32).insert(5).remove(63u32);\n            assert_eq!(u.pop_lowest(), Some(5u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let mut u = TinySet::empty()\n                .insert(63u32)\n                .insert(1)\n                .insert(5)\n                .remove(63u32);\n            assert_eq!(u.pop_lowest(), Some(1u32));\n            assert_eq!(u.pop_lowest(), Some(5u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let mut u = TinySet::empty().insert(1).remove(63u32);\n            assert_eq!(u.pop_lowest(), Some(1u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let mut u = TinySet::empty().insert(1).remove(1u32);\n            assert!(u.pop_lowest().is_none());\n        }\n    }\n    #[test]\n    fn test_tiny_set() {\n        assert!(TinySet::empty().is_empty());\n        {\n            let mut u = TinySet::empty().insert(1u32);\n            assert_eq!(u.pop_lowest(), Some(1u32));\n            assert!(u.pop_lowest().is_none())\n        }\n        {\n            let mut u = TinySet::empty().insert(1u32).insert(1u32);\n            assert_eq!(u.pop_lowest(), Some(1u32));\n            assert!(u.pop_lowest().is_none())\n        }\n        {\n            let mut u = TinySet::empty().insert(2u32);\n            assert_eq!(u.pop_lowest(), Some(2u32));\n            u.insert_mut(1u32);\n            assert_eq!(u.pop_lowest(), Some(1u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let mut u = TinySet::empty().insert(63u32);\n            assert_eq!(u.pop_lowest(), Some(63u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let mut u = TinySet::empty().insert(63u32).insert(5);\n            assert_eq!(u.pop_lowest(), Some(5u32));\n            assert_eq!(u.pop_lowest(), Some(63u32));\n            assert!(u.pop_lowest().is_none());\n        }\n        {\n            let original = TinySet::empty().insert(63u32).insert(5);\n            let after_serialize_deserialize = TinySet::deserialize(original.into_bytes());\n            assert_eq!(original, after_serialize_deserialize);\n        }\n    }\n\n    #[test]\n    fn test_bitset() {\n        let test_against_hashset = |els: &[u32], max_value: u32| {\n            let mut hashset: HashSet<u32> = HashSet::new();\n            let mut bitset = BitSet::with_max_value(max_value);\n            for &el in els {\n                assert!(el < max_value);\n                hashset.insert(el);\n                bitset.insert(el);\n            }\n            for el in 0..max_value {\n                assert_eq!(hashset.contains(&el), bitset.contains(el));\n            }\n            assert_eq!(bitset.max_value(), max_value);\n\n            // test deser\n            let mut data = vec![];\n            bitset.serialize(&mut data).unwrap();\n            let ro_bitset = ReadOnlyBitSet::open(OwnedBytes::new(data));\n            for el in 0..max_value {\n                assert_eq!(hashset.contains(&el), ro_bitset.contains(el));\n            }\n            assert_eq!(ro_bitset.max_value(), max_value);\n            assert_eq!(ro_bitset.len(), els.len());\n        };\n\n        test_against_hashset(&[], 0);\n        test_against_hashset(&[], 1);\n        test_against_hashset(&[0u32], 1);\n        test_against_hashset(&[0u32], 100);\n        test_against_hashset(&[1u32, 2u32], 4);\n        test_against_hashset(&[99u32], 100);\n        test_against_hashset(&[63u32], 64);\n        test_against_hashset(&[62u32, 63u32], 64);\n    }\n\n    #[test]\n    fn test_bitset_num_buckets() {\n        use super::num_buckets;\n        assert_eq!(num_buckets(0u32), 0);\n        assert_eq!(num_buckets(1u32), 1);\n        assert_eq!(num_buckets(64u32), 1);\n        assert_eq!(num_buckets(65u32), 2);\n        assert_eq!(num_buckets(128u32), 2);\n        assert_eq!(num_buckets(129u32), 3);\n    }\n\n    #[test]\n    fn test_tinyset_range() {\n        assert_eq!(\n            TinySet::range_lower(3).into_iter().collect::<Vec<u32>>(),\n            [0, 1, 2]\n        );\n        assert!(TinySet::range_lower(0).is_empty());\n        assert_eq!(\n            TinySet::range_lower(63).into_iter().collect::<Vec<u32>>(),\n            (0u32..63u32).collect::<Vec<_>>()\n        );\n        assert_eq!(\n            TinySet::range_lower(1).into_iter().collect::<Vec<u32>>(),\n            [0]\n        );\n        assert_eq!(\n            TinySet::range_lower(2).into_iter().collect::<Vec<u32>>(),\n            [0, 1]\n        );\n        assert_eq!(\n            TinySet::range_greater_or_equal(3)\n                .into_iter()\n                .collect::<Vec<u32>>(),\n            (3u32..64u32).collect::<Vec<_>>()\n        );\n    }\n\n    #[test]\n    fn test_bitset_len() {\n        let mut bitset = BitSet::with_max_value(1_000);\n        assert_eq!(bitset.len(), 0);\n        bitset.insert(3u32);\n        assert_eq!(bitset.len(), 1);\n        bitset.insert(103u32);\n        assert_eq!(bitset.len(), 2);\n        bitset.insert(3u32);\n        assert_eq!(bitset.len(), 2);\n        bitset.insert(103u32);\n        assert_eq!(bitset.len(), 2);\n        bitset.insert(104u32);\n        assert_eq!(bitset.len(), 3);\n        bitset.remove(105u32);\n        assert_eq!(bitset.len(), 3);\n        bitset.remove(104u32);\n        assert_eq!(bitset.len(), 2);\n        bitset.remove(3u32);\n        assert_eq!(bitset.len(), 1);\n        bitset.remove(103u32);\n        assert_eq!(bitset.len(), 0);\n    }\n\n    pub fn sample_with_seed(n: u32, ratio: f64, seed_val: u8) -> Vec<u32> {\n        StdRng::from_seed([seed_val; 32])\n            .sample_iter(&Bernoulli::new(ratio).unwrap())\n            .take(n as usize)\n            .enumerate()\n            .filter_map(|(val, keep)| if keep { Some(val as u32) } else { None })\n            .collect()\n    }\n\n    pub fn sample(n: u32, ratio: f64) -> Vec<u32> {\n        sample_with_seed(n, ratio, 4)\n    }\n\n    #[test]\n    fn test_bitset_clear() {\n        let mut bitset = BitSet::with_max_value(1_000);\n        let els = sample(1_000, 0.01f64);\n        for &el in &els {\n            bitset.insert(el);\n        }\n        assert!(els.iter().all(|el| bitset.contains(*el)));\n        bitset.clear();\n        for el in 0u32..1000u32 {\n            assert!(!bitset.contains(el));\n        }\n    }\n}\n"
  },
  {
    "path": "common/src/bounds.rs",
    "content": "use std::io;\nuse std::ops::Bound;\n\n#[derive(Clone, Debug)]\npub struct BoundsRange<T> {\n    pub lower_bound: Bound<T>,\n    pub upper_bound: Bound<T>,\n}\nimpl<T> BoundsRange<T> {\n    pub fn new(lower_bound: Bound<T>, upper_bound: Bound<T>) -> Self {\n        BoundsRange {\n            lower_bound,\n            upper_bound,\n        }\n    }\n    pub fn is_unbounded(&self) -> bool {\n        matches!(self.lower_bound, Bound::Unbounded) && matches!(self.upper_bound, Bound::Unbounded)\n    }\n    pub fn map_bound<TTo>(&self, transform: impl Fn(&T) -> TTo) -> BoundsRange<TTo> {\n        BoundsRange {\n            lower_bound: map_bound(&self.lower_bound, &transform),\n            upper_bound: map_bound(&self.upper_bound, &transform),\n        }\n    }\n\n    pub fn map_bound_res<TTo, Err>(\n        &self,\n        transform: impl Fn(&T) -> Result<TTo, Err>,\n    ) -> Result<BoundsRange<TTo>, Err> {\n        Ok(BoundsRange {\n            lower_bound: map_bound_res(&self.lower_bound, &transform)?,\n            upper_bound: map_bound_res(&self.upper_bound, &transform)?,\n        })\n    }\n\n    pub fn transform_inner<TTo>(\n        &self,\n        transform_lower: impl Fn(&T) -> TransformBound<TTo>,\n        transform_upper: impl Fn(&T) -> TransformBound<TTo>,\n    ) -> BoundsRange<TTo> {\n        BoundsRange {\n            lower_bound: transform_bound_inner(&self.lower_bound, &transform_lower),\n            upper_bound: transform_bound_inner(&self.upper_bound, &transform_upper),\n        }\n    }\n\n    /// Returns the first set inner value\n    pub fn get_inner(&self) -> Option<&T> {\n        inner_bound(&self.lower_bound).or(inner_bound(&self.upper_bound))\n    }\n}\n\npub enum TransformBound<T> {\n    /// Overwrite the bounds\n    NewBound(Bound<T>),\n    /// Use Existing bounds with new value\n    Existing(T),\n}\n\n/// Takes a bound and transforms the inner value into a new bound via a closure.\n/// The bound variant may change by the value returned value from the closure.\npub fn transform_bound_inner_res<TFrom, TTo>(\n    bound: &Bound<TFrom>,\n    transform: impl Fn(&TFrom) -> io::Result<TransformBound<TTo>>,\n) -> io::Result<Bound<TTo>> {\n    use self::Bound::*;\n    Ok(match bound {\n        Excluded(from_val) => match transform(from_val)? {\n            TransformBound::NewBound(new_val) => new_val,\n            TransformBound::Existing(new_val) => Excluded(new_val),\n        },\n        Included(from_val) => match transform(from_val)? {\n            TransformBound::NewBound(new_val) => new_val,\n            TransformBound::Existing(new_val) => Included(new_val),\n        },\n        Unbounded => Unbounded,\n    })\n}\n\n/// Takes a bound and transforms the inner value into a new bound via a closure.\n/// The bound variant may change by the value returned value from the closure.\npub fn transform_bound_inner<TFrom, TTo>(\n    bound: &Bound<TFrom>,\n    transform: impl Fn(&TFrom) -> TransformBound<TTo>,\n) -> Bound<TTo> {\n    use self::Bound::*;\n    match bound {\n        Excluded(from_val) => match transform(from_val) {\n            TransformBound::NewBound(new_val) => new_val,\n            TransformBound::Existing(new_val) => Excluded(new_val),\n        },\n        Included(from_val) => match transform(from_val) {\n            TransformBound::NewBound(new_val) => new_val,\n            TransformBound::Existing(new_val) => Included(new_val),\n        },\n        Unbounded => Unbounded,\n    }\n}\n\n/// Returns the inner value of a `Bound`\npub fn inner_bound<T>(val: &Bound<T>) -> Option<&T> {\n    match val {\n        Bound::Included(term) | Bound::Excluded(term) => Some(term),\n        Bound::Unbounded => None,\n    }\n}\n\npub fn map_bound<TFrom, TTo>(\n    bound: &Bound<TFrom>,\n    transform: impl Fn(&TFrom) -> TTo,\n) -> Bound<TTo> {\n    use self::Bound::*;\n    match bound {\n        Excluded(from_val) => Bound::Excluded(transform(from_val)),\n        Included(from_val) => Bound::Included(transform(from_val)),\n        Unbounded => Unbounded,\n    }\n}\n\npub fn map_bound_res<TFrom, TTo, Err>(\n    bound: &Bound<TFrom>,\n    transform: impl Fn(&TFrom) -> Result<TTo, Err>,\n) -> Result<Bound<TTo>, Err> {\n    use self::Bound::*;\n    Ok(match bound {\n        Excluded(from_val) => Excluded(transform(from_val)?),\n        Included(from_val) => Included(transform(from_val)?),\n        Unbounded => Unbounded,\n    })\n}\n"
  },
  {
    "path": "common/src/byte_count.rs",
    "content": "use std::iter::Sum;\nuse std::ops::{Add, AddAssign};\n\nuse serde::{Deserialize, Serialize};\n\n/// Indicates space usage in bytes\n#[derive(Copy, Clone, Default, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]\npub struct ByteCount(u64);\n\nimpl std::fmt::Debug for ByteCount {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.write_str(&self.human_readable())\n    }\n}\n\nimpl std::fmt::Display for ByteCount {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.write_str(&self.human_readable())\n    }\n}\n\nconst SUFFIX_AND_THRESHOLD: [(&str, u64); 5] = [\n    (\"KB\", 1_000),\n    (\"MB\", 1_000_000),\n    (\"GB\", 1_000_000_000),\n    (\"TB\", 1_000_000_000_000),\n    (\"PB\", 1_000_000_000_000_000),\n];\n\nimpl ByteCount {\n    #[inline]\n    pub fn get_bytes(&self) -> u64 {\n        self.0\n    }\n\n    pub fn human_readable(&self) -> String {\n        for (suffix, threshold) in SUFFIX_AND_THRESHOLD.iter().rev() {\n            if self.get_bytes() >= *threshold {\n                let unit_num = self.get_bytes() as f64 / *threshold as f64;\n                return format!(\"{unit_num:.2} {suffix}\");\n            }\n        }\n        format!(\"{:.2} B\", self.get_bytes())\n    }\n}\n\nimpl From<u64> for ByteCount {\n    fn from(value: u64) -> Self {\n        ByteCount(value)\n    }\n}\nimpl From<usize> for ByteCount {\n    fn from(value: usize) -> Self {\n        ByteCount(value as u64)\n    }\n}\n\nimpl Sum for ByteCount {\n    #[inline]\n    fn sum<I: Iterator<Item = Self>>(iter: I) -> Self {\n        iter.fold(ByteCount::default(), |acc, x| acc + x)\n    }\n}\n\nimpl PartialEq<u64> for ByteCount {\n    #[inline]\n    fn eq(&self, other: &u64) -> bool {\n        self.get_bytes() == *other\n    }\n}\n\nimpl PartialOrd<u64> for ByteCount {\n    #[inline]\n    fn partial_cmp(&self, other: &u64) -> Option<std::cmp::Ordering> {\n        self.get_bytes().partial_cmp(other)\n    }\n}\n\nimpl Add for ByteCount {\n    type Output = Self;\n\n    #[inline]\n    fn add(self, other: Self) -> Self {\n        Self(self.get_bytes() + other.get_bytes())\n    }\n}\n\nimpl AddAssign for ByteCount {\n    #[inline]\n    fn add_assign(&mut self, other: Self) {\n        *self = Self(self.get_bytes() + other.get_bytes());\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use crate::ByteCount;\n\n    #[test]\n    fn test_bytes() {\n        assert_eq!(ByteCount::from(0u64).human_readable(), \"0 B\");\n        assert_eq!(ByteCount::from(300u64).human_readable(), \"300 B\");\n        assert_eq!(ByteCount::from(1_000_000u64).human_readable(), \"1.00 MB\");\n        assert_eq!(ByteCount::from(1_500_000u64).human_readable(), \"1.50 MB\");\n        assert_eq!(\n            ByteCount::from(1_500_000_000u64).human_readable(),\n            \"1.50 GB\"\n        );\n        assert_eq!(\n            ByteCount::from(3_213_000_000_000u64).human_readable(),\n            \"3.21 TB\"\n        );\n    }\n}\n"
  },
  {
    "path": "common/src/datetime.rs",
    "content": "use std::fmt;\nuse std::io::{Read, Write};\n\nuse serde::{Deserialize, Serialize};\nuse time::format_description::well_known::Rfc3339;\nuse time::{OffsetDateTime, PrimitiveDateTime, UtcOffset};\n\nuse crate::BinarySerializable;\n\n/// Precision with which datetimes are truncated when stored in fast fields. This setting is only\n/// relevant for fast fields. In the docstore, datetimes are always saved with nanosecond precision.\n#[derive(\n    Clone, Copy, Debug, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, Default,\n)]\n#[serde(rename_all = \"lowercase\")]\npub enum DateTimePrecision {\n    /// Second precision.\n    #[default]\n    Seconds,\n    /// Millisecond precision.\n    Milliseconds,\n    /// Microsecond precision.\n    Microseconds,\n    /// Nanosecond precision.\n    Nanoseconds,\n}\n\n/// A date/time value with nanoseconds precision.\n///\n/// This timestamp does not carry any explicit time zone information.\n/// Users are responsible for applying the provided conversion\n/// functions consistently. Internally the time zone is assumed\n/// to be UTC, which is also used implicitly for JSON serialization.\n///\n/// All constructors and conversions are provided as explicit\n/// functions and not by implementing any `From`/`Into` traits\n/// to prevent unintended usage.\n#[derive(Clone, Default, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]\npub struct DateTime {\n    // Timestamp in nanoseconds.\n    pub(crate) timestamp_nanos: i64,\n}\n\nimpl DateTime {\n    /// Minimum possible `DateTime` value.\n    pub const MIN: DateTime = DateTime {\n        timestamp_nanos: i64::MIN,\n    };\n\n    /// Maximum possible `DateTime` value.\n    pub const MAX: DateTime = DateTime {\n        timestamp_nanos: i64::MAX,\n    };\n\n    /// Create new from UNIX timestamp in seconds\n    pub const fn from_timestamp_secs(seconds: i64) -> Self {\n        Self {\n            timestamp_nanos: seconds * 1_000_000_000,\n        }\n    }\n\n    /// Create new from UNIX timestamp in milliseconds\n    pub const fn from_timestamp_millis(milliseconds: i64) -> Self {\n        Self {\n            timestamp_nanos: milliseconds * 1_000_000,\n        }\n    }\n\n    /// Create new from UNIX timestamp in microseconds.\n    pub const fn from_timestamp_micros(microseconds: i64) -> Self {\n        Self {\n            timestamp_nanos: microseconds * 1_000,\n        }\n    }\n\n    /// Create new from UNIX timestamp in nanoseconds.\n    pub const fn from_timestamp_nanos(nanoseconds: i64) -> Self {\n        Self {\n            timestamp_nanos: nanoseconds,\n        }\n    }\n\n    /// Create new from `OffsetDateTime`\n    ///\n    /// The given date/time is converted to UTC and the actual\n    /// time zone is discarded.\n    pub fn from_utc(dt: OffsetDateTime) -> Self {\n        let timestamp_nanos = dt.unix_timestamp_nanos() as i64;\n        Self { timestamp_nanos }\n    }\n\n    /// Create new from `PrimitiveDateTime`\n    ///\n    /// Implicitly assumes that the given date/time is in UTC!\n    /// Otherwise the original value must only be reobtained with\n    /// [`Self::into_primitive()`].\n    pub fn from_primitive(dt: PrimitiveDateTime) -> Self {\n        Self::from_utc(dt.assume_utc())\n    }\n\n    /// Convert to UNIX timestamp in seconds.\n    pub const fn into_timestamp_secs(self) -> i64 {\n        self.timestamp_nanos / 1_000_000_000\n    }\n\n    /// Convert to UNIX timestamp in milliseconds.\n    pub const fn into_timestamp_millis(self) -> i64 {\n        self.timestamp_nanos / 1_000_000\n    }\n\n    /// Convert to UNIX timestamp in microseconds.\n    pub const fn into_timestamp_micros(self) -> i64 {\n        self.timestamp_nanos / 1_000\n    }\n\n    /// Convert to UNIX timestamp in nanoseconds.\n    pub const fn into_timestamp_nanos(self) -> i64 {\n        self.timestamp_nanos\n    }\n\n    /// Convert to UTC `OffsetDateTime`\n    pub fn into_utc(self) -> OffsetDateTime {\n        let utc_datetime = OffsetDateTime::from_unix_timestamp_nanos(self.timestamp_nanos as i128)\n            .expect(\"valid UNIX timestamp\");\n        debug_assert_eq!(UtcOffset::UTC, utc_datetime.offset());\n        utc_datetime\n    }\n\n    /// Convert to `OffsetDateTime` with the given time zone\n    pub fn into_offset(self, offset: UtcOffset) -> OffsetDateTime {\n        self.into_utc().to_offset(offset)\n    }\n\n    /// Convert to `PrimitiveDateTime` without any time zone\n    ///\n    /// The value should have been constructed with [`Self::from_primitive()`].\n    /// Otherwise the time zone is implicitly assumed to be UTC.\n    pub fn into_primitive(self) -> PrimitiveDateTime {\n        let utc_datetime = self.into_utc();\n        // Discard the UTC time zone offset\n        debug_assert_eq!(UtcOffset::UTC, utc_datetime.offset());\n        PrimitiveDateTime::new(utc_datetime.date(), utc_datetime.time())\n    }\n\n    /// Truncates the microseconds value to the corresponding precision.\n    pub fn truncate(self, precision: DateTimePrecision) -> Self {\n        let truncated_timestamp_micros = match precision {\n            DateTimePrecision::Seconds => (self.timestamp_nanos / 1_000_000_000) * 1_000_000_000,\n            DateTimePrecision::Milliseconds => (self.timestamp_nanos / 1_000_000) * 1_000_000,\n            DateTimePrecision::Microseconds => (self.timestamp_nanos / 1_000) * 1_000,\n            DateTimePrecision::Nanoseconds => self.timestamp_nanos,\n        };\n        Self {\n            timestamp_nanos: truncated_timestamp_micros,\n        }\n    }\n}\n\nimpl fmt::Debug for DateTime {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        let utc_rfc3339 = self.into_utc().format(&Rfc3339).map_err(|_| fmt::Error)?;\n        f.write_str(&utc_rfc3339)\n    }\n}\n\nimpl BinarySerializable for DateTime {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> std::io::Result<()> {\n        let timestamp_micros = self.into_timestamp_micros();\n        <i64 as BinarySerializable>::serialize(&timestamp_micros, writer)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> std::io::Result<Self> {\n        let timestamp_micros = <i64 as BinarySerializable>::deserialize(reader)?;\n        Ok(Self::from_timestamp_micros(timestamp_micros))\n    }\n}\n"
  },
  {
    "path": "common/src/file_slice.rs",
    "content": "use std::fs::File;\nuse std::ops::{Deref, Range, RangeBounds};\nuse std::path::Path;\nuse std::sync::Arc;\nuse std::{fmt, io};\n\nuse async_trait::async_trait;\nuse ownedbytes::{OwnedBytes, StableDeref};\n\nuse crate::{ByteCount, HasLen};\n\n/// Objects that represents files sections in tantivy.\n///\n/// By contract, whatever happens to the directory file, as long as a FileHandle\n/// is alive, the data associated with it cannot be altered or destroyed.\n///\n/// The underlying behavior is therefore specific to the `Directory` that\n/// created it. Despite its name, a [`FileSlice`] may or may not directly map to an actual file\n/// on the filesystem.\n\n#[async_trait]\npub trait FileHandle: 'static + Send + Sync + HasLen + fmt::Debug {\n    /// Reads a slice of bytes.\n    ///\n    /// This method may panic if the range requested is invalid.\n    fn read_bytes(&self, range: Range<usize>) -> io::Result<OwnedBytes>;\n\n    #[doc(hidden)]\n    async fn read_bytes_async(&self, _byte_range: Range<usize>) -> io::Result<OwnedBytes> {\n        Err(io::Error::new(\n            io::ErrorKind::Unsupported,\n            \"Async read is not supported.\",\n        ))\n    }\n}\n\n#[derive(Debug)]\n/// A File with it's length included.\npub struct WrapFile {\n    file: File,\n    len: usize,\n}\nimpl WrapFile {\n    /// Creates a new WrapFile and stores its length.\n    pub fn new(file: File) -> io::Result<Self> {\n        let len = file.metadata()?.len() as usize;\n        Ok(WrapFile { file, len })\n    }\n}\n\n#[async_trait]\nimpl FileHandle for WrapFile {\n    fn read_bytes(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        let file_len = self.len();\n\n        // Calculate the actual range to read, ensuring it stays within file boundaries\n        let start = range.start;\n        let end = range.end.min(file_len);\n\n        // Ensure the start is before the end of the range\n        if start >= end {\n            return Err(io::Error::new(io::ErrorKind::InvalidInput, \"Invalid range\"));\n        }\n\n        let mut buffer = vec![0; end - start];\n\n        #[cfg(unix)]\n        {\n            use std::os::unix::prelude::FileExt;\n            self.file.read_exact_at(&mut buffer, start as u64)?;\n        }\n\n        #[cfg(not(unix))]\n        {\n            use std::io::{Read, Seek};\n            let mut file = self.file.try_clone()?; // Clone the file to read from it separately\n            // Seek to the start position in the file\n            file.seek(io::SeekFrom::Start(start as u64))?;\n            // Read the data into the buffer\n            file.read_exact(&mut buffer)?;\n        }\n\n        Ok(OwnedBytes::new(buffer))\n    }\n    // todo implement async\n}\nimpl HasLen for WrapFile {\n    fn len(&self) -> usize {\n        self.len\n    }\n}\n\n#[async_trait]\nimpl FileHandle for &'static [u8] {\n    fn read_bytes(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        let bytes = &self[range];\n        Ok(OwnedBytes::new(bytes))\n    }\n\n    async fn read_bytes_async(&self, byte_range: Range<usize>) -> io::Result<OwnedBytes> {\n        Ok(self.read_bytes(byte_range)?)\n    }\n}\n\nimpl<B> From<B> for FileSlice\nwhere B: StableDeref + Deref<Target = [u8]> + 'static + Send + Sync\n{\n    fn from(bytes: B) -> FileSlice {\n        FileSlice::new(Arc::new(OwnedBytes::new(bytes)))\n    }\n}\n\n/// Logical slice of read only file in tantivy.\n///\n/// It can be cloned and sliced cheaply.\n#[derive(Clone)]\npub struct FileSlice {\n    data: Arc<dyn FileHandle>,\n    range: Range<usize>,\n}\n\nimpl fmt::Debug for FileSlice {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"FileSlice({:?}, {:?})\", &self.data, self.range)\n    }\n}\n\nimpl FileSlice {\n    pub fn stream_file_chunks(&self) -> impl Iterator<Item = io::Result<OwnedBytes>> + '_ {\n        let len = self.range.end;\n        let mut start = self.range.start;\n        std::iter::from_fn(move || {\n            /// Returns chunks of 1MB of data from the FileHandle.\n            const CHUNK_SIZE: usize = 1024 * 1024; // 1MB\n\n            if start < len {\n                let end = (start + CHUNK_SIZE).min(len);\n                let range = start..end;\n                let chunk = self.data.read_bytes(range);\n                start += CHUNK_SIZE;\n                match chunk {\n                    Ok(chunk) => Some(Ok(chunk)),\n                    Err(e) => Some(Err(e)),\n                }\n            } else {\n                None\n            }\n        })\n    }\n}\n\n/// Takes a range, a `RangeBounds` object, and returns\n/// a `Range` that corresponds to the relative application of the\n/// `RangeBounds` object to the original `Range`.\n///\n/// For instance, combine_ranges(`[2..11)`, `[5..7]`) returns `[7..10]`\n/// as it reads, what is the sub-range that starts at the 5 element of\n/// `[2..11)` and ends at the 9th element included.\n///\n/// This function panics, if the result would suggest something outside\n/// of the bounds of the original range.\nfn combine_ranges<R: RangeBounds<usize>>(orig_range: Range<usize>, rel_range: R) -> Range<usize> {\n    let start: usize = orig_range.start\n        + match rel_range.start_bound().cloned() {\n            std::ops::Bound::Included(rel_start) => rel_start,\n            std::ops::Bound::Excluded(rel_start) => rel_start + 1,\n            std::ops::Bound::Unbounded => 0,\n        };\n    assert!(start <= orig_range.end);\n    let end: usize = match rel_range.end_bound().cloned() {\n        std::ops::Bound::Included(rel_end) => orig_range.start + rel_end + 1,\n        std::ops::Bound::Excluded(rel_end) => orig_range.start + rel_end,\n        std::ops::Bound::Unbounded => orig_range.end,\n    };\n    assert!(end >= start);\n    assert!(end <= orig_range.end);\n    start..end\n}\n\nimpl FileSlice {\n    /// Creates a FileSlice from a path.\n    pub fn open(path: &Path) -> io::Result<FileSlice> {\n        let wrap_file = WrapFile::new(File::open(path)?)?;\n        Ok(FileSlice::new(Arc::new(wrap_file)))\n    }\n\n    /// Wraps a FileHandle.\n    pub fn new(file_handle: Arc<dyn FileHandle>) -> Self {\n        let num_bytes = file_handle.len();\n        FileSlice::new_with_num_bytes(file_handle, num_bytes)\n    }\n\n    /// Wraps a FileHandle.\n    #[doc(hidden)]\n    #[must_use]\n    pub fn new_with_num_bytes(file_handle: Arc<dyn FileHandle>, num_bytes: usize) -> Self {\n        FileSlice {\n            data: file_handle,\n            range: 0..num_bytes,\n        }\n    }\n\n    /// Creates a fileslice that is just a view over a slice of the data.\n    ///\n    /// # Panics\n    ///\n    /// Panics if `byte_range.end` exceeds the filesize.\n    #[must_use]\n    #[inline]\n    pub fn slice<R: RangeBounds<usize>>(&self, byte_range: R) -> FileSlice {\n        FileSlice {\n            data: self.data.clone(),\n            range: combine_ranges(self.range.clone(), byte_range),\n        }\n    }\n\n    /// Creates an empty FileSlice\n    pub fn empty() -> FileSlice {\n        const EMPTY_SLICE: &[u8] = &[];\n        FileSlice::from(EMPTY_SLICE)\n    }\n\n    /// Returns a `OwnedBytes` with all of the data in the `FileSlice`.\n    ///\n    /// The behavior is strongly dependent on the implementation of the underlying\n    /// `Directory` and the `FileSliceTrait` it creates.\n    /// In particular, it is  up to the `Directory` implementation\n    /// to handle caching if needed.\n    pub fn read_bytes(&self) -> io::Result<OwnedBytes> {\n        self.data.read_bytes(self.range.clone())\n    }\n\n    #[doc(hidden)]\n    pub async fn read_bytes_async(&self) -> io::Result<OwnedBytes> {\n        self.data.read_bytes_async(self.range.clone()).await\n    }\n\n    /// Reads a specific slice of data.\n    ///\n    /// This is equivalent to running `file_slice.slice(from, to).read_bytes()`.\n    pub fn read_bytes_slice(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        assert!(\n            range.end <= self.len(),\n            \"end of requested range exceeds the fileslice length ({} > {})\",\n            range.end,\n            self.len()\n        );\n        self.data\n            .read_bytes(self.range.start + range.start..self.range.start + range.end)\n    }\n\n    #[doc(hidden)]\n    pub async fn read_bytes_slice_async(&self, byte_range: Range<usize>) -> io::Result<OwnedBytes> {\n        assert!(\n            self.range.start + byte_range.end <= self.range.end,\n            \"`to` exceeds the fileslice length\"\n        );\n        self.data\n            .read_bytes_async(\n                self.range.start + byte_range.start..self.range.start + byte_range.end,\n            )\n            .await\n    }\n\n    /// Splits the FileSlice at the given offset and return two file slices.\n    /// `file_slice[..split_offset]` and `file_slice[split_offset..]`.\n    ///\n    /// This operation is cheap and must not copy any underlying data.\n    pub fn split(self, left_len: usize) -> (FileSlice, FileSlice) {\n        let left = self.slice_to(left_len);\n        let right = self.slice_from(left_len);\n        (left, right)\n    }\n\n    /// Splits the file slice at the given offset and return two file slices.\n    /// `file_slice[..split_offset]` and `file_slice[split_offset..]`.\n    pub fn split_from_end(self, right_len: usize) -> (FileSlice, FileSlice) {\n        let left_len = self.len() - right_len;\n        self.split(left_len)\n    }\n\n    /// Like `.slice(...)` but enforcing only the `from`\n    /// boundary.\n    ///\n    /// Equivalent to `.slice(from_offset, self.len())`\n    #[must_use]\n    pub fn slice_from(&self, from_offset: usize) -> FileSlice {\n        self.slice(from_offset..self.len())\n    }\n\n    /// Returns a slice from the end.\n    ///\n    /// Equivalent to `.slice(self.len() - from_offset, self.len())`\n    #[must_use]\n    pub fn slice_from_end(&self, from_offset: usize) -> FileSlice {\n        self.slice(self.len() - from_offset..self.len())\n    }\n\n    /// Like `.slice(...)` but enforcing only the `to`\n    /// boundary.\n    ///\n    /// Equivalent to `.slice(0, to_offset)`\n    #[must_use]\n    pub fn slice_to(&self, to_offset: usize) -> FileSlice {\n        self.slice(0..to_offset)\n    }\n\n    /// Returns the byte count of the FileSlice.\n    pub fn num_bytes(&self) -> ByteCount {\n        self.range.len().into()\n    }\n}\n\n#[async_trait]\nimpl FileHandle for FileSlice {\n    fn read_bytes(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        self.read_bytes_slice(range)\n    }\n\n    async fn read_bytes_async(&self, byte_range: Range<usize>) -> io::Result<OwnedBytes> {\n        self.read_bytes_slice_async(byte_range).await\n    }\n}\n\nimpl HasLen for FileSlice {\n    fn len(&self) -> usize {\n        self.range.len()\n    }\n}\n\n#[async_trait]\nimpl FileHandle for OwnedBytes {\n    fn read_bytes(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        Ok(self.slice(range))\n    }\n\n    async fn read_bytes_async(&self, range: Range<usize>) -> io::Result<OwnedBytes> {\n        self.read_bytes(range)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io;\n    use std::ops::Bound;\n    use std::sync::Arc;\n\n    use super::{FileHandle, FileSlice};\n    use crate::HasLen;\n    use crate::file_slice::combine_ranges;\n\n    #[test]\n    fn test_file_slice() -> io::Result<()> {\n        let file_slice = FileSlice::new(Arc::new(b\"abcdef\".as_ref()));\n        assert_eq!(file_slice.len(), 6);\n        assert_eq!(file_slice.slice_from(2).read_bytes()?.as_slice(), b\"cdef\");\n        assert_eq!(file_slice.slice_to(2).read_bytes()?.as_slice(), b\"ab\");\n        assert_eq!(\n            file_slice\n                .slice_from(1)\n                .slice_to(2)\n                .read_bytes()?\n                .as_slice(),\n            b\"bc\"\n        );\n        {\n            let (left, right) = file_slice.clone().split(0);\n            assert_eq!(left.read_bytes()?.as_slice(), b\"\");\n            assert_eq!(right.read_bytes()?.as_slice(), b\"abcdef\");\n        }\n        {\n            let (left, right) = file_slice.clone().split(2);\n            assert_eq!(left.read_bytes()?.as_slice(), b\"ab\");\n            assert_eq!(right.read_bytes()?.as_slice(), b\"cdef\");\n        }\n        {\n            let (left, right) = file_slice.clone().split_from_end(0);\n            assert_eq!(left.read_bytes()?.as_slice(), b\"abcdef\");\n            assert_eq!(right.read_bytes()?.as_slice(), b\"\");\n        }\n        {\n            let (left, right) = file_slice.split_from_end(2);\n            assert_eq!(left.read_bytes()?.as_slice(), b\"abcd\");\n            assert_eq!(right.read_bytes()?.as_slice(), b\"ef\");\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_file_slice_trait_slice_len() {\n        let blop: &'static [u8] = b\"abc\";\n        let owned_bytes: Box<dyn FileHandle> = Box::new(blop);\n        assert_eq!(owned_bytes.len(), 3);\n    }\n\n    #[test]\n    fn test_slice_simple_read() -> io::Result<()> {\n        let slice = FileSlice::new(Arc::new(&b\"abcdef\"[..]));\n        assert_eq!(slice.len(), 6);\n        assert_eq!(slice.read_bytes()?.as_ref(), b\"abcdef\");\n        assert_eq!(slice.slice(1..4).read_bytes()?.as_ref(), b\"bcd\");\n        Ok(())\n    }\n\n    #[test]\n    fn test_slice_read_slice() -> io::Result<()> {\n        let slice_deref = FileSlice::new(Arc::new(&b\"abcdef\"[..]));\n        assert_eq!(slice_deref.read_bytes_slice(1..4)?.as_ref(), b\"bcd\");\n        Ok(())\n    }\n\n    #[test]\n    #[should_panic(expected = \"end of requested range exceeds the fileslice length (10 > 6)\")]\n    fn test_slice_read_slice_invalid_range_exceeds() {\n        let slice_deref = FileSlice::new(Arc::new(&b\"abcdef\"[..]));\n        assert_eq!(\n            slice_deref.read_bytes_slice(0..10).unwrap().as_ref(),\n            b\"bcd\"\n        );\n    }\n\n    #[test]\n    fn test_combine_range() {\n        assert_eq!(combine_ranges(1..3, 0..1), 1..2);\n        assert_eq!(combine_ranges(1..3, 1..), 2..3);\n        assert_eq!(combine_ranges(1..4, ..2), 1..3);\n        assert_eq!(combine_ranges(3..10, 2..5), 5..8);\n        assert_eq!(combine_ranges(2..11, 5..=7), 7..10);\n        assert_eq!(\n            combine_ranges(2..11, (Bound::Excluded(5), Bound::Unbounded)),\n            8..11\n        );\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_combine_range_panics() {\n        let _ = combine_ranges(3..5, 1..4);\n    }\n}\n"
  },
  {
    "path": "common/src/group_by.rs",
    "content": "use std::cell::RefCell;\nuse std::iter::Peekable;\nuse std::rc::Rc;\n\npub trait GroupByIteratorExtended: Iterator {\n    /// Return an `Iterator` that groups iterator elements. Consecutive elements that map to the\n    /// same key are assigned to the same group.\n    ///\n    /// The returned Iterator item is `(K, impl Iterator)`, where Iterator are the items of the\n    /// group.\n    ///\n    /// ```\n    /// use tantivy_common::GroupByIteratorExtended;\n    ///\n    /// // group data into blocks of larger than zero or not.\n    /// let data: Vec<i32> = vec![1, 3, -2, -2, 1, 0, 1, 2];\n    /// // groups:               |---->|------>|--------->|\n    ///\n    /// let mut data_grouped = Vec::new();\n    /// // Note: group is an iterator\n    /// for (key, group) in data.into_iter().group_by(|val| *val >= 0) {\n    ///     data_grouped.push((key, group.collect()));\n    /// }\n    /// assert_eq!(data_grouped, vec![(true, vec![1, 3]), (false, vec![-2, -2]), (true, vec![1, 0, 1, 2])]);\n    /// ```\n    fn group_by<K, F>(self, key: F) -> GroupByIterator<Self, F, K>\n    where\n        Self: Sized,\n        F: FnMut(&Self::Item) -> K,\n        K: PartialEq + Clone,\n        Self::Item: Clone,\n    {\n        GroupByIterator::new(self, key)\n    }\n}\nimpl<I: Iterator> GroupByIteratorExtended for I {}\n\npub struct GroupByIterator<I, F, K: Clone>\nwhere\n    I: Iterator,\n    F: FnMut(&I::Item) -> K,\n{\n    // I really would like to avoid the Rc<RefCell>, but the Iterator is shared between\n    // `GroupByIterator` and `GroupIter`. In practice they are used consecutive and\n    // `GroupByIter` is finished before calling next on `GroupByIterator`. I'm not sure there\n    // is a solution with lifetimes for that, because we would need to enforce it in the usage\n    // somehow.\n    //\n    // One potential solution would be to replace the iterator approach with something similar.\n    inner: Rc<RefCell<GroupByShared<I, F, K>>>,\n}\n\nstruct GroupByShared<I, F, K: Clone>\nwhere\n    I: Iterator,\n    F: FnMut(&I::Item) -> K,\n{\n    iter: Peekable<I>,\n    group_by_fn: F,\n}\n\nimpl<I, F, K> GroupByIterator<I, F, K>\nwhere\n    I: Iterator,\n    F: FnMut(&I::Item) -> K,\n    K: Clone,\n{\n    fn new(inner: I, group_by_fn: F) -> Self {\n        let inner = GroupByShared {\n            iter: inner.peekable(),\n            group_by_fn,\n        };\n\n        Self {\n            inner: Rc::new(RefCell::new(inner)),\n        }\n    }\n}\n\nimpl<I, F, K> Iterator for GroupByIterator<I, F, K>\nwhere\n    I: Iterator,\n    I::Item: Clone,\n    F: FnMut(&I::Item) -> K,\n    K: Clone,\n{\n    type Item = (K, GroupIterator<I, F, K>);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let mut inner = self.inner.borrow_mut();\n        let value = inner.iter.peek()?.clone();\n        let key = (inner.group_by_fn)(&value);\n\n        let inner = self.inner.clone();\n\n        let group_iter = GroupIterator {\n            inner,\n            group_key: key.clone(),\n        };\n        Some((key, group_iter))\n    }\n}\n\npub struct GroupIterator<I, F, K: Clone>\nwhere\n    I: Iterator,\n    F: FnMut(&I::Item) -> K,\n{\n    inner: Rc<RefCell<GroupByShared<I, F, K>>>,\n    group_key: K,\n}\n\nimpl<I, F, K: PartialEq + Clone> Iterator for GroupIterator<I, F, K>\nwhere\n    I: Iterator,\n    I::Item: Clone,\n    F: FnMut(&I::Item) -> K,\n{\n    type Item = I::Item;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let mut inner = self.inner.borrow_mut();\n        // peek if next value is in group\n        let peek_val = inner.iter.peek()?.clone();\n        if (inner.group_by_fn)(&peek_val) == self.group_key {\n            inner.iter.next()\n        } else {\n            None\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    fn group_by_collect<I: Iterator<Item = u32>>(iter: I) -> Vec<(I::Item, Vec<I::Item>)> {\n        iter.group_by(|val| val / 10)\n            .map(|(el, iter)| (el, iter.collect::<Vec<_>>()))\n            .collect::<Vec<_>>()\n    }\n\n    #[test]\n    fn group_by_two_groups() {\n        let vals = vec![1u32, 4, 15];\n        let grouped_vals = group_by_collect(vals.into_iter());\n        assert_eq!(grouped_vals, vec![(0, vec![1, 4]), (1, vec![15])]);\n    }\n\n    #[test]\n    fn group_by_test_empty() {\n        let vals = vec![];\n        let grouped_vals = group_by_collect(vals.into_iter());\n        assert_eq!(grouped_vals, vec![]);\n    }\n\n    #[test]\n    fn group_by_three_groups() {\n        let vals = vec![1u32, 4, 15, 1];\n        let grouped_vals = group_by_collect(vals.into_iter());\n        assert_eq!(\n            grouped_vals,\n            vec![(0, vec![1, 4]), (1, vec![15]), (0, vec![1])]\n        );\n    }\n}\n"
  },
  {
    "path": "common/src/json_path_writer.rs",
    "content": "use crate::replace_in_place;\n\n/// Separates the different segments of a json path.\npub const JSON_PATH_SEGMENT_SEP: u8 = 1u8;\npub const JSON_PATH_SEGMENT_SEP_STR: &str =\n    unsafe { std::str::from_utf8_unchecked(&[JSON_PATH_SEGMENT_SEP]) };\n\n/// Separates the json path and the value in\n/// a JSON term binary representation.\npub const JSON_END_OF_PATH: u8 = 0u8;\npub const JSON_END_OF_PATH_STR: &str =\n    unsafe { std::str::from_utf8_unchecked(&[JSON_END_OF_PATH]) };\n\n/// Create a new JsonPathWriter, that creates flattened json paths for tantivy.\n#[derive(Clone, Debug, Default)]\npub struct JsonPathWriter {\n    path: String,\n    indices: Vec<usize>,\n    expand_dots: bool,\n}\n\nimpl JsonPathWriter {\n    pub fn with_expand_dots(expand_dots: bool) -> Self {\n        JsonPathWriter {\n            path: String::new(),\n            indices: Vec::new(),\n            expand_dots,\n        }\n    }\n\n    pub fn new() -> Self {\n        JsonPathWriter {\n            path: String::new(),\n            indices: Vec::new(),\n            expand_dots: false,\n        }\n    }\n\n    /// When expand_dots is enabled, json object like\n    /// `{\"k8s.node.id\": 5}` is processed as if it was\n    /// `{\"k8s\": {\"node\": {\"id\": 5}}}`.\n    /// This option has the merit of allowing users to\n    /// write queries  like `k8s.node.id:5`.\n    /// On the other, enabling that feature can lead to\n    /// ambiguity.\n    #[inline]\n    pub fn set_expand_dots(&mut self, expand_dots: bool) {\n        self.expand_dots = expand_dots;\n    }\n\n    /// Push a new segment to the path.\n    #[inline]\n    pub fn push(&mut self, segment: &str) {\n        let len_path = self.path.len();\n        self.indices.push(len_path);\n        if self.indices.len() > 1 {\n            self.path.push(JSON_PATH_SEGMENT_SEP as char);\n        }\n        self.path.push_str(segment);\n        if self.expand_dots {\n            // This might include the separation byte, which is ok because it is not a dot.\n            let appended_segment = &mut self.path[len_path..];\n            // The unsafe below is safe as long as b'.' and JSON_PATH_SEGMENT_SEP are\n            // valid single byte ut8 strings.\n            // By utf-8 design, they cannot be part of another codepoint.\n            unsafe {\n                replace_in_place(b'.', JSON_PATH_SEGMENT_SEP, appended_segment.as_bytes_mut())\n            };\n        }\n    }\n\n    /// Set the end of JSON path marker.\n    #[inline]\n    pub fn set_end(&mut self) {\n        self.path.push_str(JSON_END_OF_PATH_STR);\n    }\n\n    /// Remove the last segment. Does nothing if the path is empty.\n    #[inline]\n    pub fn pop(&mut self) {\n        if let Some(last_idx) = self.indices.pop() {\n            self.path.truncate(last_idx);\n        }\n    }\n\n    /// Clear the path.\n    #[inline]\n    pub fn clear(&mut self) {\n        self.path.clear();\n        self.indices.clear();\n    }\n\n    /// Get the current path.\n    #[inline]\n    pub fn as_str(&self) -> &str {\n        &self.path\n    }\n}\n\nimpl From<JsonPathWriter> for String {\n    #[inline]\n    fn from(value: JsonPathWriter) -> Self {\n        value.path\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn json_path_writer_test() {\n        let mut writer = JsonPathWriter::new();\n        writer.set_expand_dots(false);\n\n        writer.push(\"root\");\n        assert_eq!(writer.as_str(), \"root\");\n\n        writer.push(\"child\");\n        assert_eq!(writer.as_str(), \"root\\u{1}child\");\n\n        writer.pop();\n        assert_eq!(writer.as_str(), \"root\");\n\n        writer.push(\"k8s.node.id\");\n        assert_eq!(writer.as_str(), \"root\\u{1}k8s.node.id\");\n\n        writer.set_expand_dots(true);\n        writer.pop();\n        writer.push(\"k8s.node.id\");\n        assert_eq!(writer.as_str(), \"root\\u{1}k8s\\u{1}node\\u{1}id\");\n    }\n\n    #[test]\n    fn test_json_path_expand_dots_enabled_pop_segment() {\n        let mut json_writer = JsonPathWriter::with_expand_dots(true);\n        json_writer.push(\"hello\");\n        assert_eq!(json_writer.as_str(), \"hello\");\n        json_writer.push(\"color.hue\");\n        assert_eq!(json_writer.as_str(), \"hello\\x01color\\x01hue\");\n        json_writer.pop();\n        assert_eq!(json_writer.as_str(), \"hello\");\n    }\n}\n"
  },
  {
    "path": "common/src/lib.rs",
    "content": "#![allow(clippy::len_without_is_empty)]\n\nuse std::ops::Deref;\n\npub use byteorder::LittleEndian as Endianness;\n\nmod bitset;\npub mod bounds;\nmod byte_count;\nmod datetime;\npub mod file_slice;\nmod group_by;\npub mod json_path_writer;\nmod serialize;\nmod vint;\nmod writer;\npub use bitset::*;\npub use byte_count::ByteCount;\npub use datetime::{DateTime, DateTimePrecision};\npub use group_by::GroupByIteratorExtended;\npub use json_path_writer::JsonPathWriter;\npub use ownedbytes::{OwnedBytes, StableDeref};\npub use serialize::{BinarySerializable, DeserializeFrom, FixedSize};\npub use vint::{\n    VInt, VIntU128, read_u32_vint, read_u32_vint_no_advance, serialize_vint_u32, write_u32_vint,\n};\npub use writer::{AntiCallToken, CountingWriter, TerminatingWrite};\n\n/// Has length trait\npub trait HasLen {\n    /// Return length\n    fn len(&self) -> usize;\n\n    /// Returns true iff empty.\n    fn is_empty(&self) -> bool {\n        self.len() == 0\n    }\n}\n\nimpl<T: Deref<Target = [u8]>> HasLen for T {\n    fn len(&self) -> usize {\n        self.deref().len()\n    }\n}\n\nconst HIGHEST_BIT: u64 = 1 << 63;\n\n/// Maps a `i64` to `u64`\n///\n/// For simplicity, tantivy internally handles `i64` as `u64`.\n/// The mapping is defined by this function.\n///\n/// Maps `i64` to `u64` so that\n/// `-2^63 .. 2^63-1` is mapped\n///     to\n/// `0 .. 2^64-1`\n/// in that order.\n///\n/// This is more suited than simply casting (`val as u64`)\n/// because of bitpacking.\n///\n/// Imagine a list of `i64` ranging from -10 to 10.\n/// When casting negative values, the negative values are projected\n/// to values over 2^63, and all values end up requiring 64 bits.\n///\n/// # See also\n/// The reverse mapping is [`u64_to_i64()`].\n#[inline]\npub fn i64_to_u64(val: i64) -> u64 {\n    (val as u64) ^ HIGHEST_BIT\n}\n\n/// Reverse the mapping given by [`i64_to_u64()`].\n#[inline]\npub fn u64_to_i64(val: u64) -> i64 {\n    (val ^ HIGHEST_BIT) as i64\n}\n\n/// Maps a `f64` to `u64`\n///\n/// For simplicity, tantivy internally handles `f64` as `u64`.\n/// The mapping is defined by this function.\n///\n/// Maps `f64` to `u64` in a monotonic manner, so that bytes lexical order is preserved.\n///\n/// This is more suited than simply casting (`val as u64`)\n/// which would truncate the result\n///\n/// # Reference\n///\n/// Daniel Lemire's [blog post](https://lemire.me/blog/2020/12/14/converting-floating-point-numbers-to-integers-while-preserving-order/)\n/// explains the mapping in a clear manner.\n///\n/// # See also\n/// The reverse mapping is [`u64_to_f64()`].\n#[inline]\npub fn f64_to_u64(val: f64) -> u64 {\n    let bits = val.to_bits();\n    if val.is_sign_positive() {\n        bits ^ HIGHEST_BIT\n    } else {\n        !bits\n    }\n}\n\n/// Reverse the mapping given by [`f64_to_u64()`].\n#[inline]\npub fn u64_to_f64(val: u64) -> f64 {\n    f64::from_bits(if val & HIGHEST_BIT != 0 {\n        val ^ HIGHEST_BIT\n    } else {\n        !val\n    })\n}\n\n/// Replaces a given byte in the `bytes` slice of bytes.\n///\n/// This function assumes that the needle is rarely contained in the bytes string\n/// and offers a fast path if the needle is not present.\n#[inline]\npub fn replace_in_place(needle: u8, replacement: u8, bytes: &mut [u8]) {\n    if !bytes.contains(&needle) {\n        return;\n    }\n    for b in bytes {\n        if *b == needle {\n            *b = replacement;\n        }\n    }\n}\n\n#[cfg(test)]\npub(crate) mod test {\n\n    use proptest::prelude::*;\n\n    use super::{f64_to_u64, i64_to_u64, u64_to_f64, u64_to_i64};\n\n    fn test_i64_converter_helper(val: i64) {\n        assert_eq!(u64_to_i64(i64_to_u64(val)), val);\n    }\n\n    fn test_f64_converter_helper(val: f64) {\n        assert_eq!(u64_to_f64(f64_to_u64(val)), val);\n    }\n\n    proptest! {\n        #[test]\n        fn test_f64_converter_monotonicity_proptest((left, right) in (proptest::num::f64::NORMAL, proptest::num::f64::NORMAL)) {\n            let left_u64 = f64_to_u64(left);\n            let right_u64 = f64_to_u64(right);\n            assert_eq!(left_u64 < right_u64,  left < right);\n        }\n    }\n\n    #[test]\n    fn test_i64_converter() {\n        assert_eq!(i64_to_u64(i64::MIN), u64::MIN);\n        assert_eq!(i64_to_u64(i64::MAX), u64::MAX);\n        test_i64_converter_helper(0i64);\n        test_i64_converter_helper(i64::MIN);\n        test_i64_converter_helper(i64::MAX);\n        for i in -1000i64..1000i64 {\n            test_i64_converter_helper(i);\n        }\n    }\n\n    #[test]\n    fn test_f64_converter() {\n        test_f64_converter_helper(f64::INFINITY);\n        test_f64_converter_helper(f64::NEG_INFINITY);\n        test_f64_converter_helper(0.0);\n        test_f64_converter_helper(-0.0);\n        test_f64_converter_helper(1.0);\n        test_f64_converter_helper(-1.0);\n    }\n\n    #[test]\n    fn test_f64_order() {\n        assert!(\n            !(f64_to_u64(f64::NEG_INFINITY)..f64_to_u64(f64::INFINITY))\n                .contains(&f64_to_u64(f64::NAN))\n        ); // nan is not a number\n        assert!(f64_to_u64(1.5) > f64_to_u64(1.0)); // same exponent, different mantissa\n        assert!(f64_to_u64(2.0) > f64_to_u64(1.0)); // same mantissa, different exponent\n        assert!(f64_to_u64(2.0) > f64_to_u64(1.5)); // different exponent and mantissa\n        assert!(f64_to_u64(1.0) > f64_to_u64(-1.0)); // pos > neg\n        assert!(f64_to_u64(-1.5) < f64_to_u64(-1.0));\n        assert!(f64_to_u64(-2.0) < f64_to_u64(1.0));\n        assert!(f64_to_u64(-2.0) < f64_to_u64(-1.5));\n    }\n\n    #[test]\n    fn test_replace_in_place() {\n        let test_aux = |before_replacement: &[u8], expected: &[u8]| {\n            let mut bytes: Vec<u8> = before_replacement.to_vec();\n            super::replace_in_place(b'b', b'c', &mut bytes);\n            assert_eq!(&bytes[..], expected);\n        };\n        test_aux(b\"\", b\"\");\n        test_aux(b\"b\", b\"c\");\n        test_aux(b\"baaa\", b\"caaa\");\n        test_aux(b\"aaab\", b\"aaac\");\n        test_aux(b\"aaabaa\", b\"aaacaa\");\n        test_aux(b\"aaaaaa\", b\"aaaaaa\");\n        test_aux(b\"bbbb\", b\"cccc\");\n    }\n}\n"
  },
  {
    "path": "common/src/serialize.rs",
    "content": "use std::borrow::Cow;\nuse std::io::{Read, Write};\nuse std::{fmt, io};\n\nuse byteorder::{ReadBytesExt, WriteBytesExt};\n\nuse crate::{Endianness, VInt};\n\n#[derive(Default)]\nstruct Counter(u64);\n\nimpl io::Write for Counter {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        self.0 += buf.len() as u64;\n        Ok(buf.len())\n    }\n\n    fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {\n        self.0 += buf.len() as u64;\n        Ok(())\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        Ok(())\n    }\n}\n\n/// Trait for a simple binary serialization.\npub trait BinarySerializable: fmt::Debug + Sized {\n    /// Serialize\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()>;\n    /// Deserialize\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self>;\n\n    fn num_bytes(&self) -> u64 {\n        let mut counter = Counter::default();\n        self.serialize(&mut counter).unwrap();\n        counter.0\n    }\n}\n\npub trait DeserializeFrom<T: BinarySerializable> {\n    fn deserialize(&mut self) -> io::Result<T>;\n}\n\n/// Implement deserialize from &[u8] for all types which implement BinarySerializable.\n///\n/// TryFrom would actually be preferable, but not possible because of the orphan\n/// rules (not completely sure if this could be resolved)\nimpl<T: BinarySerializable> DeserializeFrom<T> for &[u8] {\n    fn deserialize(&mut self) -> io::Result<T> {\n        T::deserialize(self)\n    }\n}\n\n/// `FixedSize` marks a `BinarySerializable` as\n/// always serializing to the same size.\npub trait FixedSize: BinarySerializable {\n    const SIZE_IN_BYTES: usize;\n}\n\nimpl BinarySerializable for () {\n    fn serialize<W: Write + ?Sized>(&self, _: &mut W) -> io::Result<()> {\n        Ok(())\n    }\n    fn deserialize<R: Read>(_: &mut R) -> io::Result<Self> {\n        Ok(())\n    }\n}\n\nimpl FixedSize for () {\n    const SIZE_IN_BYTES: usize = 0;\n}\n\nimpl<T: BinarySerializable> BinarySerializable for Vec<T> {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        BinarySerializable::serialize(&VInt(self.len() as u64), writer)?;\n        for it in self {\n            it.serialize(writer)?;\n        }\n        Ok(())\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Vec<T>> {\n        let num_items = <VInt as BinarySerializable>::deserialize(reader)?.val();\n        let mut items: Vec<T> = Vec::with_capacity(num_items as usize);\n        for _ in 0..num_items {\n            let item = T::deserialize(reader)?;\n            items.push(item);\n        }\n        Ok(items)\n    }\n}\n\nimpl<Left: BinarySerializable, Right: BinarySerializable> BinarySerializable for (Left, Right) {\n    fn serialize<W: Write + ?Sized>(&self, write: &mut W) -> io::Result<()> {\n        self.0.serialize(write)?;\n        self.1.serialize(write)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        Ok((Left::deserialize(reader)?, Right::deserialize(reader)?))\n    }\n}\nimpl<Left: BinarySerializable + FixedSize, Right: BinarySerializable + FixedSize> FixedSize\n    for (Left, Right)\n{\n    const SIZE_IN_BYTES: usize = Left::SIZE_IN_BYTES + Right::SIZE_IN_BYTES;\n}\n\nimpl BinarySerializable for u32 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u32::<Endianness>(*self)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<u32> {\n        reader.read_u32::<Endianness>()\n    }\n}\n\nimpl FixedSize for u32 {\n    const SIZE_IN_BYTES: usize = 4;\n}\n\nimpl BinarySerializable for u16 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u16::<Endianness>(*self)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<u16> {\n        reader.read_u16::<Endianness>()\n    }\n}\n\nimpl FixedSize for u16 {\n    const SIZE_IN_BYTES: usize = 2;\n}\n\nimpl BinarySerializable for u64 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u64::<Endianness>(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        reader.read_u64::<Endianness>()\n    }\n}\n\nimpl FixedSize for u64 {\n    const SIZE_IN_BYTES: usize = 8;\n}\n\nimpl BinarySerializable for u128 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u128::<Endianness>(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        reader.read_u128::<Endianness>()\n    }\n}\n\nimpl FixedSize for u128 {\n    const SIZE_IN_BYTES: usize = 16;\n}\n\nimpl BinarySerializable for f32 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_f32::<Endianness>(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        reader.read_f32::<Endianness>()\n    }\n}\n\nimpl FixedSize for f32 {\n    const SIZE_IN_BYTES: usize = 4;\n}\n\nimpl BinarySerializable for i64 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_i64::<Endianness>(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        reader.read_i64::<Endianness>()\n    }\n}\n\nimpl FixedSize for i64 {\n    const SIZE_IN_BYTES: usize = 8;\n}\n\nimpl BinarySerializable for f64 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_f64::<Endianness>(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        reader.read_f64::<Endianness>()\n    }\n}\n\nimpl FixedSize for f64 {\n    const SIZE_IN_BYTES: usize = 8;\n}\n\nimpl BinarySerializable for u8 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u8(*self)\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<u8> {\n        reader.read_u8()\n    }\n}\n\nimpl FixedSize for u8 {\n    const SIZE_IN_BYTES: usize = 1;\n}\n\nimpl BinarySerializable for bool {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        writer.write_u8(u8::from(*self))\n    }\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<bool> {\n        let val = reader.read_u8()?;\n        match val {\n            0 => Ok(false),\n            1 => Ok(true),\n            _ => Err(io::Error::new(\n                io::ErrorKind::InvalidData,\n                \"invalid bool value on deserialization, data corrupted\",\n            )),\n        }\n    }\n}\n\nimpl FixedSize for bool {\n    const SIZE_IN_BYTES: usize = 1;\n}\n\nimpl BinarySerializable for String {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        let data: &[u8] = self.as_bytes();\n        BinarySerializable::serialize(&VInt(data.len() as u64), writer)?;\n        writer.write_all(data)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<String> {\n        let string_length = <VInt as BinarySerializable>::deserialize(reader)?.val() as usize;\n        let mut result = String::with_capacity(string_length);\n        reader\n            .take(string_length as u64)\n            .read_to_string(&mut result)?;\n        Ok(result)\n    }\n}\n\nimpl<'a> BinarySerializable for Cow<'a, str> {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        let data: &[u8] = self.as_bytes();\n        BinarySerializable::serialize(&VInt(data.len() as u64), writer)?;\n        writer.write_all(data)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Cow<'a, str>> {\n        let string_length = <VInt as BinarySerializable>::deserialize(reader)?.val() as usize;\n        let mut result = String::with_capacity(string_length);\n        reader\n            .take(string_length as u64)\n            .read_to_string(&mut result)?;\n        Ok(Cow::Owned(result))\n    }\n}\n\nimpl<'a> BinarySerializable for Cow<'a, [u8]> {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        BinarySerializable::serialize(&VInt(self.len() as u64), writer)?;\n        for it in self.iter() {\n            BinarySerializable::serialize(it, writer)?;\n        }\n        Ok(())\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Cow<'a, [u8]>> {\n        let num_items = <VInt as BinarySerializable>::deserialize(reader)?.val();\n        let mut items: Vec<u8> = Vec::with_capacity(num_items as usize);\n        for _ in 0..num_items {\n            let item = <u8 as BinarySerializable>::deserialize(reader)?;\n            items.push(item);\n        }\n        Ok(Cow::Owned(items))\n    }\n}\n\n#[cfg(test)]\npub mod test {\n\n    use super::*;\n    pub fn fixed_size_test<O: BinarySerializable + FixedSize + Default>() {\n        let mut buffer = Vec::new();\n        O::default().serialize(&mut buffer).unwrap();\n        assert_eq!(buffer.len(), O::SIZE_IN_BYTES);\n    }\n\n    fn serialize_test<T: BinarySerializable + Eq>(v: T) -> usize {\n        let mut buffer: Vec<u8> = Vec::new();\n        v.serialize(&mut buffer).unwrap();\n        let num_bytes = buffer.len();\n        let mut cursor = &buffer[..];\n        let deser = T::deserialize(&mut cursor).unwrap();\n        assert_eq!(deser, v);\n        num_bytes\n    }\n\n    #[test]\n    fn test_serialize_u8() {\n        fixed_size_test::<u8>();\n    }\n\n    #[test]\n    fn test_serialize_u32() {\n        fixed_size_test::<u32>();\n        assert_eq!(4, serialize_test(3u32));\n        assert_eq!(4, serialize_test(5u32));\n        assert_eq!(4, serialize_test(u32::MAX));\n    }\n\n    #[test]\n    fn test_serialize_i64() {\n        fixed_size_test::<i64>();\n    }\n\n    #[test]\n    fn test_serialize_f64() {\n        fixed_size_test::<f64>();\n    }\n\n    #[test]\n    fn test_serialize_u64() {\n        fixed_size_test::<u64>();\n    }\n\n    #[test]\n    fn test_serialize_bool() {\n        fixed_size_test::<bool>();\n    }\n\n    #[test]\n    fn test_serialize_string() {\n        assert_eq!(serialize_test(String::from(\"\")), 1);\n        assert_eq!(serialize_test(String::from(\"ぽよぽよ\")), 1 + 3 * 4);\n        assert_eq!(serialize_test(String::from(\"富士さん見える。\")), 1 + 3 * 8);\n    }\n\n    #[test]\n    fn test_serialize_vec() {\n        assert_eq!(serialize_test(Vec::<u8>::new()), 1);\n        assert_eq!(serialize_test(vec![1u32, 3u32]), 1 + 4 * 2);\n    }\n\n    #[test]\n    fn test_serialize_vint() {\n        for i in 0..10_000 {\n            serialize_test(VInt(i as u64));\n        }\n        assert_eq!(serialize_test(VInt(7u64)), 1);\n        assert_eq!(serialize_test(VInt(127u64)), 1);\n        assert_eq!(serialize_test(VInt(128u64)), 2);\n        assert_eq!(serialize_test(VInt(129u64)), 2);\n        assert_eq!(serialize_test(VInt(1234u64)), 2);\n        assert_eq!(serialize_test(VInt(16_383u64)), 2);\n        assert_eq!(serialize_test(VInt(16_384u64)), 3);\n        assert_eq!(serialize_test(VInt(u64::MAX)), 10);\n    }\n}\n"
  },
  {
    "path": "common/src/vint.rs",
    "content": "use std::io;\nuse std::io::{Read, Write};\n\nuse super::BinarySerializable;\n\n/// Variable int serializes a u128 number\npub fn serialize_vint_u128(mut val: u128, output: &mut Vec<u8>) {\n    loop {\n        let next_byte: u8 = (val % 128u128) as u8;\n        val /= 128u128;\n        if val == 0 {\n            output.push(next_byte | STOP_BIT);\n            return;\n        } else {\n            output.push(next_byte);\n        }\n    }\n}\n\n///   Wrapper over a `u128` that serializes as a variable int.\n#[derive(Clone, Copy, Debug, Eq, PartialEq)]\npub struct VIntU128(pub u128);\n\nimpl BinarySerializable for VIntU128 {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        let mut buffer = vec![];\n        serialize_vint_u128(self.0, &mut buffer);\n        writer.write_all(&buffer)\n    }\n\n    #[allow(clippy::unbuffered_bytes)]\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        #[allow(clippy::unbuffered_bytes)]\n        let mut bytes = reader.bytes();\n        let mut result = 0u128;\n        let mut shift = 0u64;\n        loop {\n            match bytes.next() {\n                Some(Ok(b)) => {\n                    result |= u128::from(b % 128u8) << shift;\n                    if b >= STOP_BIT {\n                        return Ok(VIntU128(result));\n                    }\n                    shift += 7;\n                }\n                _ => {\n                    return Err(io::Error::new(\n                        io::ErrorKind::InvalidData,\n                        \"Reach end of buffer while reading VInt\",\n                    ));\n                }\n            }\n        }\n    }\n}\n\n///   Wrapper over a `u64` that serializes as a variable int.\n#[derive(Clone, Copy, Debug, Eq, PartialEq)]\npub struct VInt(pub u64);\n\nconst STOP_BIT: u8 = 128;\n\n#[inline]\npub fn serialize_vint_u32(val: u32, buf: &mut [u8; 8]) -> &[u8] {\n    const START_2: u64 = 1 << 7;\n    const START_3: u64 = 1 << 14;\n    const START_4: u64 = 1 << 21;\n    const START_5: u64 = 1 << 28;\n\n    const MASK_1: u64 = 127;\n    const MASK_2: u64 = MASK_1 << 7;\n    const MASK_3: u64 = MASK_2 << 7;\n    const MASK_4: u64 = MASK_3 << 7;\n    const MASK_5: u64 = MASK_4 << 7;\n\n    let val = u64::from(val);\n    const STOP_BIT: u64 = 128u64;\n    let (res, num_bytes) = if val < START_2 {\n        (val | STOP_BIT, 1)\n    } else if val < START_3 {\n        (\n            (val & MASK_1) | ((val & MASK_2) << 1) | (STOP_BIT << (8)),\n            2,\n        )\n    } else if val < START_4 {\n        (\n            (val & MASK_1) | ((val & MASK_2) << 1) | ((val & MASK_3) << 2) | (STOP_BIT << (8 * 2)),\n            3,\n        )\n    } else if val < START_5 {\n        (\n            (val & MASK_1)\n                | ((val & MASK_2) << 1)\n                | ((val & MASK_3) << 2)\n                | ((val & MASK_4) << 3)\n                | (STOP_BIT << (8 * 3)),\n            4,\n        )\n    } else {\n        (\n            (val & MASK_1)\n                | ((val & MASK_2) << 1)\n                | ((val & MASK_3) << 2)\n                | ((val & MASK_4) << 3)\n                | ((val & MASK_5) << 4)\n                | (STOP_BIT << (8 * 4)),\n            5,\n        )\n    };\n    *buf = res.to_le_bytes();\n    &buf[0..num_bytes]\n}\n\n/// Returns the number of bytes covered by a\n/// serialized vint `u32`.\n///\n/// Expects a buffer data that starts\n/// by the serialized `vint`, scans at most 5 bytes ahead until\n/// it finds the vint final byte.\n///\n/// # May Panic\n/// If the payload does not start by a valid `vint`\nfn vint_len(data: &[u8]) -> usize {\n    for (i, &val) in data.iter().enumerate().take(5) {\n        if val >= STOP_BIT {\n            return i + 1;\n        }\n    }\n    panic!(\"Corrupted data. Invalid VInt 32\");\n}\n\n/// Reads a vint `u32` from a buffer, and\n/// consumes its payload data.\n///\n/// # Panics\n///\n/// If the buffer does not start by a valid\n/// vint payload\npub fn read_u32_vint(data: &mut &[u8]) -> u32 {\n    let (result, vlen) = read_u32_vint_no_advance(data);\n    *data = &data[vlen..];\n    result\n}\n\npub fn read_u32_vint_no_advance(data: &[u8]) -> (u32, usize) {\n    let vlen = vint_len(data);\n    let mut result = 0u32;\n    let mut shift = 0u64;\n    for &b in &data[..vlen] {\n        result |= u32::from(b & 127u8) << shift;\n        shift += 7;\n    }\n    (result, vlen)\n}\n/// Write a `u32` as a vint payload.\npub fn write_u32_vint<W: io::Write + ?Sized>(val: u32, writer: &mut W) -> io::Result<()> {\n    let mut buf = [0u8; 8];\n    let data = serialize_vint_u32(val, &mut buf);\n    writer.write_all(data)\n}\n\nimpl VInt {\n    pub fn val(&self) -> u64 {\n        self.0\n    }\n\n    pub fn deserialize_u64<R: Read>(reader: &mut R) -> io::Result<u64> {\n        VInt::deserialize(reader).map(|vint| vint.0)\n    }\n\n    pub fn serialize_into_vec(&self, output: &mut Vec<u8>) {\n        let mut buffer = [0u8; 10];\n        let num_bytes = self.serialize_into(&mut buffer);\n        output.extend(&buffer[0..num_bytes]);\n    }\n\n    pub fn serialize_into(&self, buffer: &mut [u8; 10]) -> usize {\n        let mut remaining = self.0;\n        for (i, b) in buffer.iter_mut().enumerate() {\n            let next_byte: u8 = (remaining % 128u64) as u8;\n            remaining /= 128u64;\n            if remaining == 0u64 {\n                *b = next_byte | STOP_BIT;\n                return i + 1;\n            } else {\n                *b = next_byte;\n            }\n        }\n        unreachable!();\n    }\n}\n\nimpl BinarySerializable for VInt {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        let mut buffer = [0u8; 10];\n        let num_bytes = self.serialize_into(&mut buffer);\n        writer.write_all(&buffer[0..num_bytes])\n    }\n\n    #[allow(clippy::unbuffered_bytes)]\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        #[allow(clippy::unbuffered_bytes)]\n        let mut bytes = reader.bytes();\n        let mut result = 0u64;\n        let mut shift = 0u64;\n        loop {\n            match bytes.next() {\n                Some(Ok(b)) => {\n                    result |= u64::from(b % 128u8) << shift;\n                    if b >= STOP_BIT {\n                        return Ok(VInt(result));\n                    }\n                    shift += 7;\n                }\n                _ => {\n                    return Err(io::Error::new(\n                        io::ErrorKind::InvalidData,\n                        \"Reach end of buffer while reading VInt\",\n                    ));\n                }\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::{BinarySerializable, VInt, serialize_vint_u32};\n\n    fn aux_test_vint(val: u64) {\n        let mut v = [14u8; 10];\n        let num_bytes = VInt(val).serialize_into(&mut v);\n        for el in &v[num_bytes..10] {\n            assert_eq!(el, &14u8);\n        }\n        assert!(num_bytes > 0);\n        if num_bytes < 10 {\n            assert!(1u64 << (7 * num_bytes) > val);\n        }\n        if num_bytes > 1 {\n            assert!(1u64 << (7 * (num_bytes - 1)) <= val);\n        }\n        let serdeser_val = VInt::deserialize(&mut &v[..]).unwrap();\n        assert_eq!(val, serdeser_val.0);\n    }\n\n    #[test]\n    fn test_vint() {\n        aux_test_vint(0);\n        aux_test_vint(1);\n        aux_test_vint(5);\n        aux_test_vint(u64::MAX);\n        for i in 1..9 {\n            let power_of_128 = 1u64 << (7 * i);\n            aux_test_vint(power_of_128 - 1u64);\n            aux_test_vint(power_of_128);\n            aux_test_vint(power_of_128 + 1u64);\n        }\n        aux_test_vint(10);\n    }\n\n    fn aux_test_serialize_vint_u32(val: u32) {\n        let mut buffer = [0u8; 10];\n        let mut buffer2 = [0u8; 8];\n        let len_vint = VInt(val as u64).serialize_into(&mut buffer);\n        let res2 = serialize_vint_u32(val, &mut buffer2);\n        assert_eq!(&buffer[..len_vint], res2, \"array wrong for {val}\");\n    }\n\n    #[test]\n    fn test_vint_u32() {\n        aux_test_serialize_vint_u32(0);\n        aux_test_serialize_vint_u32(1);\n        aux_test_serialize_vint_u32(5);\n        for i in 1..3 {\n            let power_of_128 = 1u32 << (7 * i);\n            aux_test_serialize_vint_u32(power_of_128 - 1u32);\n            aux_test_serialize_vint_u32(power_of_128);\n            aux_test_serialize_vint_u32(power_of_128 + 1u32);\n        }\n        aux_test_serialize_vint_u32(u32::MAX);\n    }\n}\n"
  },
  {
    "path": "common/src/writer.rs",
    "content": "use std::io::{self, BufWriter, Write};\n\npub struct CountingWriter<W> {\n    underlying: W,\n    written_bytes: u64,\n}\n\nimpl<W: Write> CountingWriter<W> {\n    pub fn wrap(underlying: W) -> CountingWriter<W> {\n        CountingWriter {\n            underlying,\n            written_bytes: 0,\n        }\n    }\n\n    #[inline]\n    pub fn written_bytes(&self) -> u64 {\n        self.written_bytes\n    }\n\n    /// Returns the underlying write object.\n    /// Note that this method does not trigger any flushing.\n    #[inline]\n    pub fn finish(self) -> W {\n        self.underlying\n    }\n}\n\nimpl<W: Write> Write for CountingWriter<W> {\n    #[inline]\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        let written_size = self.underlying.write(buf)?;\n        self.written_bytes += written_size as u64;\n        Ok(written_size)\n    }\n\n    #[inline]\n    fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {\n        self.underlying.write_all(buf)?;\n        self.written_bytes += buf.len() as u64;\n        Ok(())\n    }\n\n    #[inline]\n    fn flush(&mut self) -> io::Result<()> {\n        self.underlying.flush()\n    }\n}\n\nimpl<W: TerminatingWrite> TerminatingWrite for CountingWriter<W> {\n    #[inline]\n    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {\n        self.underlying.terminate_ref(token)\n    }\n}\n\n/// Struct used to prevent from calling\n/// [`terminate_ref`](TerminatingWrite::terminate_ref) directly\n///\n/// The point is that while the type is public, it cannot be built by anyone\n/// outside of this module.\npub struct AntiCallToken(());\n\n/// Trait used to indicate when no more write need to be done on a writer\n///\n/// Thread-safety is enforced at the call sites that require it.\npub trait TerminatingWrite: Write {\n    /// Indicate that the writer will no longer be used. Internally call terminate_ref.\n    fn terminate(mut self) -> io::Result<()>\n    where Self: Sized {\n        self.terminate_ref(AntiCallToken(()))\n    }\n\n    /// You should implement this function to define custom behavior.\n    /// This function should flush any buffer it may hold.\n    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()>;\n}\n\nimpl<W: TerminatingWrite + ?Sized> TerminatingWrite for Box<W> {\n    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {\n        self.as_mut().terminate_ref(token)\n    }\n}\n\nimpl<W: TerminatingWrite> TerminatingWrite for BufWriter<W> {\n    fn terminate_ref(&mut self, a: AntiCallToken) -> io::Result<()> {\n        self.flush()?;\n        self.get_mut().terminate_ref(a)\n    }\n}\n\nimpl TerminatingWrite for &mut Vec<u8> {\n    fn terminate_ref(&mut self, _a: AntiCallToken) -> io::Result<()> {\n        self.flush()\n    }\n}\n\n#[cfg(test)]\nmod test {\n\n    use std::io::Write;\n\n    use super::CountingWriter;\n\n    #[test]\n    fn test_counting_writer() {\n        let buffer: Vec<u8> = vec![];\n        let mut counting_writer = CountingWriter::wrap(buffer);\n        let bytes = (0u8..10u8).collect::<Vec<u8>>();\n        counting_writer.write_all(&bytes).unwrap();\n        let len = counting_writer.written_bytes();\n        let buffer_restituted: Vec<u8> = counting_writer.finish();\n        assert_eq!(len, 10u64);\n        assert_eq!(buffer_restituted.len(), 10);\n    }\n}\n"
  },
  {
    "path": "doc/.gitignore",
    "content": "book\n"
  },
  {
    "path": "doc/book.toml",
    "content": "[book]\nauthors = [\"Paul Masurel\"]\nmultilingual = false\nsrc = \"src\"\ntitle = \"Tantivy, the user guide\"\n"
  },
  {
    "path": "doc/src/SUMMARY.md",
    "content": "# Summary\n\n[Avant Propos](./avant-propos.md)\n\n- [Segments](./basis.md)\n- [Defining your schema](./schema.md)\n- [Facetting](./facetting.md)\n- [Index Sorting](./index_sorting.md)\n- [Innerworkings](./innerworkings.md)\n  - [Inverted index](./inverted_index.md)\n- [Best practise](./inverted_index.md)\n\n[Frequently Asked Questions](./faq.md)\n[Examples](./examples.md)\n"
  },
  {
    "path": "doc/src/avant-propos.md",
    "content": "# Foreword, what is the scope of tantivy?\n\n> Tantivy is a **search** engine **library** for Rust.\n\nIf you are familiar with Lucene, it's an excellent approximation to consider tantivy as Lucene for Rust. Tantivy is heavily inspired by Lucene's design and\nthey both have the same scope and targeted use cases.\n\nIf you are not familiar with Lucene, let's break down our little tagline.\n\n- **Search** here means full-text search : fundamentally, tantivy is here to help you\nidentify efficiently what are the documents matching a given query in your corpus.\nBut modern search UI are so much more : text processing, facetting, autocomplete, fuzzy search, good\nrelevancy, collapsing, highlighting, spatial search.\n\n  While some of these features are not available in tantivy yet, all of these are relevant\n  feature requests. Tantivy's objective is to offer a solid toolbox to create the best search\n  experience. But keep in mind this is just a toolbox.\n  Which bring us to the second keyword...\n\n- **Library** means that you will have to write code. Tantivy is not an *all-in-one* server solution like Elasticsearch for instance.\n\n  Sometimes a functionality will not be available in tantivy because it is too\n  specific to your use case. By design, tantivy should make it possible to extend\n  the available set of features using the existing rock-solid datastructures.\n\n  Most frequently this will mean writing your own `Collector`, your own `Scorer` or your own\n  `TokenFilter`... Some of your requirements may also be related to\n  something closer to architecture or operations. For instance, you may\n  want to build a large corpus on Hadoop, fine-tune the merge policy to keep your\n  index sharded in a time-wise fashion, or you may want to convert and existing\n  index from a different format.\n\n  Tantivy exposes a lot of low level API to do all of these things.\n  \n"
  },
  {
    "path": "doc/src/basis.md",
    "content": "# Anatomy of an index\n\n## Straight from disk\n\nTantivy accesses its data using an abstracting trait called `Directory`.\nIn theory, one can come and override the data access logic. In practise, the\ntrait somewhat assumes that your data can be mapped to memory, and tantivy\nseems deeply married to using `mmap` for its io [^1], and the only persisting\ndirectory shipped with tantivy is the `MmapDirectory`.\n\nWhile this design has some downsides, this greatly simplifies the source code of\ntantivy. Caching is also entirely delegated to the OS.\n\nTantivy works entirely (or almost) by directly reading the datastructures as they are laid on disk. As a result, the act of opening an indexing does not involve loading different datastructures from the disk into random access memory : starting a process, opening an index, and performing your first query can typically be done in a matter of milliseconds.\n\nThis is an interesting property for a command line search engine, or for some multi-tenant log search engine : spawning a new process for each new query can be a perfectly sensible solution in some use case.\n\nIn later chapters, we will discuss tantivy's inverted index data layout.\nOne key take away is that to achieve great performance, search indexes are extremely compact.\nOf course this is crucial to reduce IO, and ensure that as much of our index can sit in RAM.\n\nAlso, whenever possible its data is accessed sequentially. Of course, this is an amazing property when tantivy needs to access the data from your spinning hard disk, but this is also\ncritical for performance, if your data is read from and an `SSD` or even already in your pagecache.\n\n## Segments, and the log method\n\nThat kind of compact layout comes at one cost: it prevents our datastructures from being dynamic.\nIn fact, the `Directory` trait does not even allow you to modify part of a file.\n\nTo allow the addition / deletion of documents, and create the illusion that\nyour index is dynamic (i.e.: adding and deleting documents), tantivy uses a common database trick sometimes referred to as the *log method*.\n\nLet's forget about deletes for a moment.\n\nAs you add documents, these documents are processed and stored in a dedicated datastructure, in a `RAM` buffer. This datastructure is not ready for search, but it is useful to receive your data and rearrange it very rapidly.\n\nAs you add documents, this buffer will reach its capacity and tantivy will transparently stop adding document to it and start converting this datastructure to its final read-only format on disk. Once written, an brand empty buffer is available to resume adding documents.\n\nThe resulting chunk of index obtained after this serialization is called a `Segment`.\n\n> A segment is a self-contained atomic piece of index. It is identified with a UUID, and all of its files are identified using the naming scheme : `<UUID>.*`.\n\nWhich brings us to the nature of a tantivy `Index`.\n\n> A tantivy `Index` is a collection of `Segments`.\n\nPhysically, this really just means and index is a bunch of segment files in a given `Directory`,\nlinked together by a `meta.json` file. This transparency can become extremely handy\nto get tantivy to fit your use case:\n\n*Example 1* You could for instance use hadoop to build a very large search index in a timely manner, copy all of the resulting segment files in the same directory and edit the `meta.json` to get a functional index.[^2]\n\n*Example 2* You could also disable your merge policy and enforce daily segments. Removing data after one week can then be done very efficiently by just editing the `meta.json` and deleting the files associated with segment `D-7`.\n\n## Merging\n\nAs you index more and more data, your index will accumulate more and more segments.\nHaving a lot of small segments is not really optimal. There is a bit of redundancy in having\nall these term dictionary. Also when searching, we will need to do term lookups as many times as we have segments.  It can hurt search performance a bit.\n\nThat's where merging or compacting comes into place. Tantivy will continuously consider merge\nopportunities and start merging segments in the background.\n\n## Indexing throughput, number of indexing threads\n\n[^1]: This may eventually change.\n\n[^2]: Be careful however. By default these files will not be considered as *managed* by tantivy. This means they will never be garbage collected by tantivy, regardless of whether they become obsolete or not.\n"
  },
  {
    "path": "doc/src/best_practise.md.rs",
    "content": ""
  },
  {
    "path": "doc/src/examples.md",
    "content": "# Examples\n\n- [Basic search](/examples/basic_search.html)\n"
  },
  {
    "path": "doc/src/facetting.md",
    "content": "# Facetting\n\nwewew\n\n## weeewe\n"
  },
  {
    "path": "doc/src/faq.md",
    "content": ""
  },
  {
    "path": "doc/src/index_sorting.md",
    "content": "\n- [Index Sorting](#index-sorting)\n  - [Why Sorting](#why-sorting)\n    - [Compression](#compression)\n    - [Top-N Optimization](#top-n-optimization)\n    - [Pruning](#pruning)\n    - [Other](#other)\n  - [Usage](#usage)\n\n# Index Sorting has been removed!\nMore infos here:\n\nhttps://github.com/quickwit-oss/tantivy/issues/2352\n\n# Index Sorting\n\nTantivy allows you to sort the index according to a property.\n\n## Why Sorting\n\nPresorting an index has several advantages:\n\n### Compression\n\nWhen data is sorted it is easier to compress the data. E.g. the numbers sequence [5, 2, 3, 1, 4] would be sorted to [1, 2, 3, 4, 5].\nIf we apply delta encoding this list would be unsorted [5, -3, 1, -2, 3] vs. [1, 1, 1, 1, 1].\nCompression ratio is mainly affected on the fast field of the sorted property, every thing else is likely unaffected.\n\n### Top-N Optimization\n\nWhen data is presorted by a field and search queries request sorting by the same field, we can leverage the natural order of the documents.\nE.g. if the data is sorted by timestamp and want the top n newest docs containing a term, we can simply leveraging the order of the docids.\n\nNote: tantivy 0.16 does not do this optimization yet.\n\n### Pruning\n\nLet's say we want all documents and want to apply the filter `>= 2010-08-11`. When the data is sorted, we could make a lookup in the fast field to find the docid range and use this as the filter.\n\nNote: tantivy 0.16 does not do this optimization yet.\n\n### Other?\n\nIn principle there are many algorithms possible that exploit the monotonically increasing nature. (aggregations maybe?)\n\n## Usage\n\nThe index sorting can be configured setting [`sort_by_field`](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/core/index_meta.rs#L238) on `IndexSettings` and passing it to a `IndexBuilder`. As of tantivy 0.16 only fast fields are allowed to be used.\n\n```rust\nlet settings = IndexSettings {\n    sort_by_field: Some(IndexSortByField {\n        field: \"intval\".to_string(),\n        order: Order::Desc,\n    }),\n    ..Default::default()\n};\nlet mut index_builder = Index::builder().schema(schema);\nindex_builder = index_builder.settings(settings);\nlet index = index_builder.create_in_ram().unwrap();\n```\n\n## Implementation details\n\nSorting an index is applied in the serialization step. In general there are two serialization steps: [Finishing a single segment](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/indexer/segment_writer.rs#L338) and [merging multiple segments](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/indexer/merger.rs#L1073).\n\nIn both cases we generate a docid mapping reflecting the sort. This mapping is used when serializing the different components (doc store, fastfields, posting list, normfield, facets).\n"
  },
  {
    "path": "doc/src/innerworkings.md",
    "content": "# Innerworkings\n"
  },
  {
    "path": "doc/src/inverted_index.md",
    "content": "# Inverted index\n"
  },
  {
    "path": "doc/src/json.md",
    "content": "# Json\n\nAs of tantivy 0.17, tantivy supports a json object type.\nThis type can be used to allow for a schema-less search index.\n\nWhen indexing a json object, we \"flatten\" the JSON. This operation emits terms that represent a triplet `(json_path, value_type, value)`\n\nFor instance,  if user is a json field, the following document:\n\n```json\n{\n    \"user\": {\n        \"name\": \"Paul Masurel\",\n        \"address\": {\n            \"city\": \"Tokyo\",\n            \"country\": \"Japan\"\n        },\n        \"created_at\": \"2018-11-12T23:20:50.52Z\"\n    }\n}\n```\n\nemits the following tokens:\n\n- (\"name\", Text, \"Paul\")\n- (\"name\", Text, \"Masurel\")\n- (\"address.city\", Text, \"Tokyo\")\n- (\"address.country\", Text, \"Japan\")\n- (\"created_at\", Date, 15420648505)\n\n## Bytes-encoding and lexicographical sort\n\nLike any other terms, these triplets are encoded into a binary format as follows.\n\n- `json_path`: the json path is a sequence of \"segments\". In the example above, `address.city`\nis just a debug representation of the json path `[\"address\", \"city\"]`.\nIts representation is done by separating segments by a unicode char `\\x01`, and ending the path by `\\x00`.\n- `value type`: One byte represents the `Value` type.\n- `value`: The value representation is just the regular Value representation.\n\nThis representation is designed to align the natural sort of Terms with the lexicographical sort\nof their binary representation (tantivy's dictionary (whether fst or sstable) is sorted and does prefix encoding).\n\nIn the example above, the terms will be sorted as\n\n- (\"address.city\", Text, \"Tokyo\")\n- (\"address.country\", Text, \"Japan\")\n- (\"name\", Text, \"Masurel\")\n- (\"name\", Text, \"Paul\")\n- (\"created_at\", Date, 15420648505)\n\nAs seen in \"pitfalls\", we may end up having to search for a value for a same path in several different fields. Putting the field code after the path makes it maximizes compression opportunities but also increases the chances for the two terms to end up in the actual same term dictionary block.\n\n## Pitfalls, limitation and corner cases\n\nJson gives very little information about the type of the literals it stores.\nAll numeric types end up mapped as a \"Number\" and there are no types for dates.\n\nAt indexing, tantivy will try to interpret number and strings as different type with a\npriority order.\n\nNumbers will be interpreted as u64, i64 and f64 in that order.\nStrings will be interpreted as rfc3339 dates or simple strings.\n\nThe first working type is picked and is the only term that is emitted for indexing.\nNote this interpretation happens on a per-document basis, and there is no effort to try to sniff\na consistent field type at the scale of a segment.\n\nOn the query parser side on the other hand, we may end up emitting more than one type.\nFor instance, we do not even know if the type is a number or string based.\n\nSo the query\n\n```rust\nmy_path.my_segment:233\n```\n\nWill be interpreted as\n\n```rust\n(my_path.my_segment, String, 233) or (my_path.my_segment, u64, 233)\n```\n\nLikewise, we need to emit two tokens if the query contains an rfc3339 date.\nIndeed the date could have been actually a single token inside the text of a document at ingestion time. Generally speaking, we will always at least emit a string token in query parsing, and sometimes more.\n\nIf one more json field is defined, things get even more complicated.\n\n## Default json field\n\nIf the schema contains a text field called \"text\" and a json field that is set as a default field:\n`text:hello` could be reasonably interpreted as targeting the text field or as targeting the json field called `json_dynamic` with the json_path \"text\".\n\nIf there is such an ambiguity, we decide to only search in the \"text\" field: `text:hello`.\n\nIn other words, the parser will not search in default json fields if there is a schema hit.\nThis is a product decision.\n\nThe user can still target the JSON field by specifying its name explicitly:\n`json_dynamic.text:hello`.\n\n## Range queries are not supported\n\nJson field do not support range queries.\n\n## Arrays do not work like nested object\n\nIf json object contains an array, a search query might return more documents\nthan what might be expected.\n\nLet's take an example.\n\n```json\n{\n    \"cart_id\": 3234234 ,\n    \"cart\": [\n        {\"product_type\": \"sneakers\", \"attributes\": {\"color\": \"white\"} },\n        {\"product_type\": \"t-shirt\", \"attributes\": {\"color\": \"red\"}},\n    ]\n}\n```\n\nDespite the array structure, a document in tantivy is a bag of terms.\nThe query:\n\n```rust\ncart.product_type:sneakers AND cart.attributes.color:red\n```\n\nActually match the document above.\n"
  },
  {
    "path": "doc/src/schema.md",
    "content": "# Defining your schema\n"
  },
  {
    "path": "examples/aggregation.rs",
    "content": "// # Aggregation example\n//\n// This example shows how you can use built-in aggregations.\n// We will use nested aggregations with buckets and metrics:\n// - Range buckets and compute the average in each bucket.\n// - Term aggregation and compute the min price in each bucket\n// ---\n\nuse serde_json::{Deserializer, Value};\nuse tantivy::aggregation::agg_req::Aggregations;\nuse tantivy::aggregation::agg_result::AggregationResults;\nuse tantivy::aggregation::AggregationCollector;\nuse tantivy::query::AllQuery;\nuse tantivy::schema::{self, IndexRecordOption, Schema, TextFieldIndexing, FAST};\nuse tantivy::{Index, IndexWriter, TantivyDocument};\n\nfn main() -> tantivy::Result<()> {\n    // # Create Schema\n    //\n    // Lets create a schema for a footwear shop, with 4 fields: name, category, stock and price.\n    // category, stock and price will be fast fields as that's the requirement\n    // for aggregation queries.\n    //\n\n    let mut schema_builder = Schema::builder();\n    // In preparation of the `TermsAggregation`, the category field is configured with:\n    // - `set_fast`\n    // - `raw` tokenizer\n    //\n    // The tokenizer is set to \"raw\", because the fast field uses the same dictionary as the\n    // inverted index. (This behaviour will change in tantivy 0.20, where the fast field will\n    // always be raw tokenized independent from the regular tokenizing)\n    //\n    let text_fieldtype = schema::TextOptions::default()\n        .set_indexing_options(\n            TextFieldIndexing::default()\n                .set_index_option(IndexRecordOption::WithFreqs)\n                .set_tokenizer(\"raw\"),\n        )\n        .set_fast(None)\n        .set_stored();\n    schema_builder.add_text_field(\"category\", text_fieldtype);\n    schema_builder.add_f64_field(\"stock\", FAST);\n    schema_builder.add_f64_field(\"price\", FAST);\n\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    //\n    // Lets index a bunch of documents for this example.\n    let index = Index::create_in_ram(schema.clone());\n\n    let data = r#\"{\n        \"name\": \"Almond Toe Court Shoes, Patent Black\",\n        \"category\": \"Womens Footwear\",\n        \"price\": 99.00,\n        \"stock\": 5\n    }\n    {\n        \"name\": \"Suede Shoes, Blue\",\n        \"category\": \"Womens Footwear\",\n        \"price\": 42.00,\n        \"stock\": 4\n    }\n    {\n        \"name\": \"Leather Driver Saddle Loafers, Tan\",\n        \"category\": \"Mens Footwear\",\n        \"price\": 34.00,\n        \"stock\": 12\n    }\n    {\n        \"name\": \"Flip Flops, Red\",\n        \"category\": \"Mens Footwear\",\n        \"price\": 19.00,\n        \"stock\": 6\n    }\n    {\n        \"name\": \"Flip Flops, Blue\",\n        \"category\": \"Mens Footwear\",\n        \"price\": 19.00,\n        \"stock\": 0\n    }\n    {\n        \"name\": \"Gold Button Cardigan, Black\",\n        \"category\": \"Womens Casualwear\",\n        \"price\": 167.00,\n        \"stock\": 6\n    }\n    {\n        \"name\": \"Cotton Shorts, Medium Red\",\n        \"category\": \"Womens Casualwear\",\n        \"price\": 30.00,\n        \"stock\": 5\n    }\n    {\n        \"name\": \"Fine Stripe Short SleeveShirt, Grey\",\n        \"category\": \"Mens Casualwear\",\n        \"price\": 49.99,\n        \"stock\": 9\n    }\n    {\n        \"name\": \"Fine Stripe Short SleeveShirt, Green\",\n        \"category\": \"Mens Casualwear\",\n        \"price\": 49.99,\n        \"offer\": 39.99,\n        \"stock\": 9\n    }\n    {\n        \"name\": \"Sharkskin Waistcoat, Charcoal\",\n        \"category\": \"Mens Formalwear\",\n        \"price\": 75.00,\n        \"stock\": 2\n    }\n    {\n        \"name\": \"Lightweight Patch PocketBlazer, Deer\",\n        \"category\": \"Mens Formalwear\",\n        \"price\": 175.50,\n        \"stock\": 1\n    }\n    {\n        \"name\": \"Bird Print Dress, Black\",\n        \"category\": \"Womens Formalwear\",\n        \"price\": 270.00,\n        \"stock\": 10\n    }\n    {\n        \"name\": \"Mid Twist Cut-Out Dress, Pink\",\n        \"category\": \"Womens Formalwear\",\n        \"price\": 540.00,\n        \"stock\": 5\n    }\"#;\n\n    let stream = Deserializer::from_str(data).into_iter::<Value>();\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n    let mut num_indexed = 0;\n    for value in stream {\n        let doc = TantivyDocument::parse_json(&schema, &serde_json::to_string(&value.unwrap())?)?;\n        index_writer.add_document(doc)?;\n        num_indexed += 1;\n        if num_indexed > 4 {\n            // Writing the first segment\n            index_writer.commit()?;\n        }\n    }\n\n    // Writing the second segment\n    index_writer.commit()?;\n\n    // We have two segments now. The `AggregationCollector` will run the aggregation on each\n    // segment and then merge the results into an `IntermediateAggregationResult`.\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n    // ---\n    // # Aggregation Query\n    //\n    //\n    // We can construct the query by building the request structure or by deserializing from JSON.\n    // The JSON API is more stable and therefore recommended.\n    //\n    // ## Request 1\n\n    let agg_req_str = r#\"\n    {\n      \"group_by_stock\": {\n        \"aggs\": {\n          \"average_price\": { \"avg\": { \"field\": \"price\" } }\n        },\n        \"range\": {\n          \"field\": \"stock\",\n          \"ranges\": [\n            { \"key\": \"few\", \"to\": 1.0 },\n            { \"key\": \"some\", \"from\": 1.0, \"to\": 10.0 },\n            { \"key\": \"many\", \"from\": 10.0 }\n          ]\n        }\n      }\n    } \"#;\n\n    // In this Aggregation we want to get the average price for different groups, depending on how\n    // many items are in stock. We define custom ranges `few`, `some`, `many` via the\n    // range aggregation.\n    // For every bucket we want the average price, so we create a nested metric aggregation on the\n    // range bucket aggregation. Only buckets support nested aggregations.\n    // ### Request JSON API\n    //\n\n    let agg_req: Aggregations = serde_json::from_str(agg_req_str)?;\n    let collector = AggregationCollector::from_aggs(agg_req, Default::default());\n\n    // We use the `AllQuery` which will pass all documents to the AggregationCollector.\n    let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n    let res: Value = serde_json::to_value(agg_res)?;\n\n    // ### Aggregation Result\n    //\n    // The resulting structure deserializes in the same JSON format as elastic search.\n    //\n    let expected_res = r#\"\n    {\n        \"group_by_stock\":{\n            \"buckets\":[\n                {\"average_price\":{\"value\":19.0},\"doc_count\":1,\"key\":\"few\",\"to\":1.0},\n                {\"average_price\":{\"value\":124.748},\"doc_count\":10,\"from\":1.0,\"key\":\"some\",\"to\":10.0},\n                {\"average_price\":{\"value\":152.0},\"doc_count\":2,\"from\":10.0,\"key\":\"many\"}\n            ]\n        }\n    }\n    \"#;\n    let expected_json: Value = serde_json::from_str(expected_res)?;\n    assert_eq!(expected_json, res);\n\n    // ### Request 2\n    //\n    // Now we are interested in the minimum price per category, so we create a bucket per\n    // category via `TermsAggregation`. We are interested in the highest minimum prices, and set the\n    // order of the buckets `\"order\": { \"min_price\": \"desc\" }` to be sorted by the the metric of\n    // the sub aggregation. (awesome)\n    //\n    let agg_req_str = r#\"\n    {\n      \"min_price_per_category\": {\n        \"aggs\": {\n          \"min_price\": { \"min\": { \"field\": \"price\" } }\n        },\n        \"terms\": {\n          \"field\": \"category\",\n          \"min_doc_count\": 1,\n          \"order\": { \"min_price\": \"desc\" }\n        }\n      }\n    } \"#;\n\n    let agg_req: Aggregations = serde_json::from_str(agg_req_str)?;\n\n    let collector = AggregationCollector::from_aggs(agg_req, Default::default());\n\n    let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n    let res: Value = serde_json::to_value(agg_res)?;\n\n    // Minimum price per category, sorted by minimum price descending\n    //\n    // As you can see, the starting prices for `Formalwear` are higher than `Casualwear`.\n    //\n    let expected_res = r#\"\n    {\n      \"min_price_per_category\": {\n        \"buckets\": [\n          { \"doc_count\": 2, \"key\": \"Womens Formalwear\", \"min_price\": { \"value\": 270.0 } },\n          { \"doc_count\": 2, \"key\": \"Mens Formalwear\", \"min_price\": { \"value\": 75.0 } },\n          { \"doc_count\": 2, \"key\": \"Mens Casualwear\", \"min_price\": { \"value\": 49.99 } },\n          { \"doc_count\": 2, \"key\": \"Womens Footwear\", \"min_price\": { \"value\": 42.0 } },\n          { \"doc_count\": 2, \"key\": \"Womens Casualwear\", \"min_price\": { \"value\": 30.0 } },\n          { \"doc_count\": 3, \"key\": \"Mens Footwear\", \"min_price\": { \"value\": 19.0 } }\n        ],\n        \"sum_other_doc_count\": 0\n      }\n    }\n    \"#;\n    let expected_json: Value = serde_json::from_str(expected_res)?;\n\n    assert_eq!(expected_json, res);\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/basic_search.rs",
    "content": "// # Basic Example\n//\n// This example covers the basic functionalities of\n// tantivy.\n//\n// We will :\n// - define our schema\n// - create an index in a directory\n// - index a few documents into our index\n// - search for the best document matching a basic query\n// - retrieve the best document's original content.\n\n// ---\n// Importing tantivy...\nuse tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::{doc, Index, IndexWriter, ReloadPolicy};\nuse tempfile::TempDir;\n\nfn main() -> tantivy::Result<()> {\n    // Let's create a temporary directory for the\n    // sake of this example\n    let index_path = TempDir::new()?;\n\n    // # Defining the schema\n    //\n    // The Tantivy index requires a very strict schema.\n    // The schema declares which fields are in the index,\n    // and for each field, its type and \"the way it should\n    // be indexed\".\n\n    // First we need to define a schema ...\n    let mut schema_builder = Schema::builder();\n\n    // Our first field is title.\n    // We want full-text search for it, and we also want\n    // to be able to retrieve the document after the search.\n    //\n    // `TEXT | STORED` is some syntactic sugar to describe\n    // that.\n    //\n    // `TEXT` means the field should be tokenized and indexed,\n    // along with its term frequency and term positions.\n    //\n    // `STORED` means that the field will also be saved\n    // in a compressed, row-oriented key-value store.\n    // This store is useful for reconstructing the\n    // documents that were selected during the search phase.\n    schema_builder.add_text_field(\"title\", TEXT | STORED);\n\n    // Our second field is body.\n    // We want full-text search for it, but we do not\n    // need to be able to retrieve it\n    // for our application.\n    //\n    // We can make our index lighter by omitting the `STORED` flag.\n    schema_builder.add_text_field(\"body\", TEXT);\n\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    //\n    // Let's create a brand new index.\n    //\n    // This will actually just save a meta.json\n    // with our schema in the directory.\n    let index = Index::create_in_dir(&index_path, schema.clone())?;\n\n    // To insert a document we will need an index writer.\n    // There must be only one writer at a time.\n    // This single `IndexWriter` is already\n    // multithreaded.\n    //\n    // Here we give tantivy a budget of `50MB`.\n    // Using a bigger memory_arena for the indexer may increase\n    // throughput, but 50 MB is already plenty.\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // Let's index our documents!\n    // We first need a handle on the title and the body field.\n\n    // ### Adding documents\n    //\n    // We can create a document manually, by setting the fields\n    // one by one in a Document object.\n    let title = schema.get_field(\"title\").unwrap();\n    let body = schema.get_field(\"body\").unwrap();\n\n    let mut old_man_doc = TantivyDocument::default();\n    old_man_doc.add_text(title, \"The Old Man and the Sea\");\n    old_man_doc.add_text(\n        body,\n        \"He was an old man who fished alone in a skiff in the Gulf Stream and he had gone \\\n         eighty-four days now without taking a fish.\",\n    );\n\n    // ... and add it to the `IndexWriter`.\n    index_writer.add_document(old_man_doc)?;\n\n    // For convenience, tantivy also comes with a macro to\n    // reduce the boilerplate above.\n    index_writer.add_document(doc!(\n    title => \"Of Mice and Men\",\n    body => \"A few miles south of Soledad, the Salinas River drops in close to the hillside \\\n            bank and runs deep and green. The water is warm too, for it has slipped twinkling \\\n            over the yellow sands in the sunlight before reaching the narrow pool. On one \\\n            side of the river the golden foothill slopes curve up to the strong and rocky \\\n            Gabilan Mountains, but on the valley side the water is lined with trees—willows \\\n            fresh and green with every spring, carrying in their lower leaf junctures the \\\n            debris of the winter’s flooding; and sycamores with mottled, white, recumbent \\\n            limbs and branches that arch over the pool\"\n    ))?;\n\n    // Multivalued field just need to be repeated.\n    index_writer.add_document(doc!(\n    title => \"Frankenstein\",\n    title => \"The Modern Prometheus\",\n    body => \"You will rejoice to hear that no disaster has accompanied the commencement of an \\\n             enterprise which you have regarded with such evil forebodings.  I arrived here \\\n             yesterday, and my first task is to assure my dear sister of my welfare and \\\n             increasing confidence in the success of my undertaking.\"\n    ))?;\n\n    // This is an example, so we will only index 3 documents\n    // here. You can check out tantivy's tutorial to index\n    // the English wikipedia. Tantivy's indexing is rather fast.\n    // Indexing 5 million articles of the English wikipedia takes\n    // around 3 minutes on my computer!\n\n    // ### Committing\n    //\n    // At this point our documents are not searchable.\n    //\n    //\n    // We need to call `.commit()` explicitly to force the\n    // `index_writer` to finish processing the documents in the queue,\n    // flush the current index to the disk, and advertise\n    // the existence of new documents.\n    //\n    // This call is blocking.\n    index_writer.commit()?;\n\n    // If `.commit()` returns correctly, then all of the\n    // documents that have been added are guaranteed to be\n    // persistently indexed.\n    //\n    // In the scenario of a crash or a power failure,\n    // tantivy behaves as if it has rolled back to its last\n    // commit.\n\n    // # Searching\n    //\n    // ### Searcher\n    //\n    // A reader is required first in order to search an index.\n    // It acts as a `Searcher` pool that reloads itself,\n    // depending on a `ReloadPolicy`.\n    //\n    // For a search server you will typically create one reader for the entire lifetime of your\n    // program, and acquire a new searcher for every single request.\n    //\n    // In the code below, we rely on the 'ON_COMMIT' policy: the reader\n    // will reload the index automatically after each commit.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::OnCommitWithDelay)\n        .try_into()?;\n\n    // We now need to acquire a searcher.\n    //\n    // A searcher points to a snapshotted, immutable version of the index.\n    //\n    // Some search experience might require more than\n    // one query. Using the same searcher ensures that all of these queries will run on the\n    // same version of the index.\n    //\n    // Acquiring a `searcher` is very cheap.\n    //\n    // You should acquire a searcher every time you start processing a request and\n    // and release it right after your query is finished.\n    let searcher = reader.searcher();\n\n    // ### Query\n\n    // The query parser can interpret human queries.\n    // Here, if the user does not specify which\n    // field they want to search, tantivy will search\n    // in both title and body.\n    let query_parser = QueryParser::for_index(&index, vec![title, body]);\n\n    // `QueryParser` may fail if the query is not in the right\n    // format. For user facing applications, this can be a problem.\n    // A ticket has been opened regarding this problem.\n    let query = query_parser.parse_query(\"sea whale\")?;\n\n    // A query defines a set of documents, as\n    // well as the way they should be scored.\n    //\n    // A query created by the query parser is scored according\n    // to a metric called Tf-Idf, and will consider\n    // any document matching at least one of our terms.\n\n    // ### Collectors\n    //\n    // We are not interested in all of the documents but\n    // only in the top 10. Keeping track of our top 10 best documents\n    // is the role of the `TopDocs` collector.\n\n    // We can now perform our query.\n    let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n\n    // The actual documents still need to be\n    // retrieved from Tantivy's store.\n    //\n    // Since the body field was not configured as stored,\n    // the document returned will only contain\n    // a title.\n    for (_score, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"{}\", retrieved_doc.to_json(&schema));\n    }\n\n    // We can also get an explanation to understand\n    // how a found document got its score.\n    let query = query_parser.parse_query(\"title:sea^20 body:whale^70\")?;\n\n    let (_score, doc_address) = searcher\n        .search(&query, &TopDocs::with_limit(1).order_by_score())?\n        .into_iter()\n        .next()\n        .unwrap();\n\n    let explanation = query.explain(&searcher, doc_address)?;\n\n    println!(\"{}\", explanation.to_pretty_json());\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/custom_collector.rs",
    "content": "// # Custom collector example\n//\n// This example shows how you can implement your own\n// collector. As an example, we will compute a collector\n// that computes the standard deviation of a given fast field.\n//\n// Of course, you can have a look at the tantivy's built-in collectors\n// such as the `CountCollector` for more examples.\n\nuse columnar::Column;\n// ---\n// Importing tantivy...\nuse tantivy::collector::{Collector, SegmentCollector};\nuse tantivy::index::SegmentReader;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::{Schema, FAST, INDEXED, TEXT};\nuse tantivy::{doc, Index, IndexWriter, Score};\n\n#[derive(Default)]\nstruct Stats {\n    count: usize,\n    sum: f64,\n    squared_sum: f64,\n}\n\nimpl Stats {\n    pub fn count(&self) -> usize {\n        self.count\n    }\n\n    pub fn mean(&self) -> f64 {\n        self.sum / (self.count as f64)\n    }\n\n    fn square_mean(&self) -> f64 {\n        self.squared_sum / (self.count as f64)\n    }\n\n    pub fn standard_deviation(&self) -> f64 {\n        let mean = self.mean();\n        (self.square_mean() - mean * mean).sqrt()\n    }\n\n    fn non_zero_count(self) -> Option<Stats> {\n        if self.count == 0 {\n            None\n        } else {\n            Some(self)\n        }\n    }\n}\n\nstruct StatsCollector {\n    field: String,\n}\n\nimpl StatsCollector {\n    fn with_field(field: String) -> StatsCollector {\n        StatsCollector { field }\n    }\n}\n\nimpl Collector for StatsCollector {\n    // That's the type of our result.\n    // Our standard deviation will be a float.\n    type Fruit = Option<Stats>;\n\n    type Child = StatsSegmentCollector;\n\n    fn for_segment(\n        &self,\n        _segment_local_id: u32,\n        segment_reader: &SegmentReader,\n    ) -> tantivy::Result<StatsSegmentCollector> {\n        let fast_field_reader = segment_reader.fast_fields().u64(&self.field)?;\n        Ok(StatsSegmentCollector {\n            fast_field_reader,\n            stats: Stats::default(),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        // this collector does not care about score.\n        false\n    }\n\n    fn merge_fruits(&self, segment_stats: Vec<Option<Stats>>) -> tantivy::Result<Option<Stats>> {\n        let mut stats = Stats::default();\n        for segment_stats in segment_stats.into_iter().flatten() {\n            stats.count += segment_stats.count;\n            stats.sum += segment_stats.sum;\n            stats.squared_sum += segment_stats.squared_sum;\n        }\n        Ok(stats.non_zero_count())\n    }\n}\n\nstruct StatsSegmentCollector {\n    fast_field_reader: Column,\n    stats: Stats,\n}\n\nimpl SegmentCollector for StatsSegmentCollector {\n    type Fruit = Option<Stats>;\n\n    fn collect(&mut self, doc: u32, _score: Score) {\n        // Since we know the values are single value, we could call `first_or_default_col` on the\n        // column and fetch single values.\n        for value in self.fast_field_reader.values_for_doc(doc) {\n            let value = value as f64;\n            self.stats.count += 1;\n            self.stats.sum += value;\n            self.stats.squared_sum += value * value;\n        }\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        self.stats.non_zero_count()\n    }\n}\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    //\n    // The Tantivy index requires a very strict schema.\n    // The schema declares which fields are in the index,\n    // and for each field, its type and \"the way it should\n    // be indexed\".\n\n    // first we need to define a schema ...\n    let mut schema_builder = Schema::builder();\n\n    // We'll assume a fictional index containing\n    // products, and with a name, a description, and a price.\n    let product_name = schema_builder.add_text_field(\"name\", TEXT);\n    let product_description = schema_builder.add_text_field(\"description\", TEXT);\n    let price = schema_builder.add_u64_field(\"price\", INDEXED | FAST);\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    //\n    // Lets index a bunch of fake documents for the sake of\n    // this example.\n    let index = Index::create_in_ram(schema);\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n    index_writer.add_document(doc!(\n        product_name => \"Super Broom 2000\",\n        product_description => \"While it is ok for short distance travel, this broom \\\n        was designed quiditch. It will up your game.\",\n        price => 30_200u64\n    ))?;\n    index_writer.add_document(doc!(\n        product_name => \"Turbulobroom\",\n        product_description => \"You might have heard of this broom before : it is the sponsor of the Wales team.\\\n            You'll enjoy its sharp turns, and rapid acceleration\",\n        price => 29_240u64\n    ))?;\n    index_writer.add_document(doc!(\n        product_name => \"Broomio\",\n        product_description => \"Great value for the price. This broom is a market favorite\",\n        price => 21_240u64\n    ))?;\n    index_writer.add_document(doc!(\n        product_name => \"Whack a Mole\",\n        product_description => \"Prime quality bat.\",\n        price => 5_200u64\n    ))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n    let query_parser = QueryParser::for_index(&index, vec![product_name, product_description]);\n\n    // here we want to search for `broom` and use `StatsCollector` on the hits.\n    let query = query_parser.parse_query(\"broom\")?;\n    if let Some(stats) =\n        searcher.search(&query, &StatsCollector::with_field(\"price\".to_string()))?\n    {\n        println!(\"count: {}\", stats.count());\n        println!(\"mean: {}\", stats.mean());\n        println!(\"standard deviation: {}\", stats.standard_deviation());\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/custom_tokenizer.rs",
    "content": "// # Defining a tokenizer pipeline\n//\n// In this example, we'll see how to define a tokenizer\n// by creating a custom `NgramTokenizer`.\nuse tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::tokenizer::NgramTokenizer;\nuse tantivy::{doc, Index, IndexWriter};\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    //\n    // The Tantivy index requires a very strict schema.\n    // The schema declares which fields are in the index,\n    // and for each field, its type and \"the way it should\n    // be indexed\".\n\n    // first we need to define a schema ...\n    let mut schema_builder = Schema::builder();\n\n    // Our first field is title.\n    // In this example we want to use NGram searching\n    // we will set that to 3 characters, so any three\n    // char in the title should be findable.\n    let text_field_indexing = TextFieldIndexing::default()\n        .set_tokenizer(\"ngram3\")\n        .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n    let text_options = TextOptions::default()\n        .set_indexing_options(text_field_indexing)\n        .set_stored();\n    let title = schema_builder.add_text_field(\"title\", text_options);\n\n    // Our second field is body.\n    // We want full-text search for it, but we do not\n    // need to be able to be able to retrieve it\n    // for our application.\n    //\n    // We can make our index lighter by omitting the `STORED` flag.\n    let body = schema_builder.add_text_field(\"body\", TEXT);\n\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    //\n    // Let's create a brand new index.\n    // To simplify we will work entirely in RAM.\n    // This is not what you want in reality, but it is very useful\n    // for your unit tests... Or this example.\n    let index = Index::create_in_ram(schema.clone());\n\n    // here we are registering our custom tokenizer\n    // this will store tokens of 3 characters each\n    index\n        .tokenizers()\n        .register(\"ngram3\", NgramTokenizer::new(3, 3, false).unwrap());\n\n    // To insert document we need an index writer.\n    // There must be only one writer at a time.\n    // This single `IndexWriter` is already\n    // multithreaded.\n    //\n    // Here we use a buffer of 50MB per thread. Using a bigger\n    // memory arena for the indexer can increase its throughput.\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n    index_writer.add_document(doc!(\n    title => \"The Old Man and the Sea\",\n    body => \"He was an old man who fished alone in a skiff in the Gulf Stream and \\\n     he had gone eighty-four days now without taking a fish.\"\n    ))?;\n    index_writer.add_document(doc!(\n    title => \"Of Mice and Men\",\n       body => r#\"A few miles south of Soledad, the Salinas River drops in close to the hillside\n                bank and runs deep and green. The water is warm too, for it has slipped twinkling\n                over the yellow sands in the sunlight before reaching the narrow pool. On one\n                side of the river the golden foothill slopes curve up to the strong and rocky\n                Gabilan Mountains, but on the valley side the water is lined with trees—willows\n                fresh and green with every spring, carrying in their lower leaf junctures the\n                debris of the winter’s flooding; and sycamores with mottled, white, recumbent\n                limbs and branches that arch over the pool\"#\n    ))?;\n    index_writer.add_document(doc!(\n    title => \"Frankenstein\",\n        body => r#\"You will rejoice to hear that no disaster has accompanied the commencement of an\n                enterprise which you have regarded with such evil forebodings.  I arrived here\n                yesterday, and my first task is to assure my dear sister of my welfare and\n                increasing confidence in the success of my undertaking.\"#\n    ))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    // The query parser can interpret human queries.\n    // Here, if the user does not specify which\n    // field they want to search, tantivy will search\n    // in both title and body.\n    let query_parser = QueryParser::for_index(&index, vec![title, body]);\n\n    // here we want to get a hit on the 'ken' in Frankenstein\n    let query = query_parser.parse_query(\"ken\")?;\n\n    let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n\n    for (_, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"{}\", retrieved_doc.to_json(&schema));\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/date_time_field.rs",
    "content": "// # DateTime field example\n//\n// This example shows how the DateTime field can be used\n\nuse tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::{DateOptions, Document, Schema, Value, INDEXED, STORED, STRING};\nuse tantivy::{Index, IndexWriter, TantivyDocument};\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    let mut schema_builder = Schema::builder();\n    let opts = DateOptions::from(INDEXED)\n        .set_stored()\n        .set_fast()\n        .set_precision(tantivy::schema::DateTimePrecision::Seconds);\n    // Add `occurred_at` date field type\n    let occurred_at = schema_builder.add_date_field(\"occurred_at\", opts);\n    let event_type = schema_builder.add_text_field(\"event\", STRING | STORED);\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    let index = Index::create_in_ram(schema.clone());\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n    // The dates are passed as string in the RFC3339 format\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n        \"occurred_at\": \"2022-06-22T12:53:50.53Z\",\n        \"event\": \"pull-request\"\n    }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n        \"occurred_at\": \"2022-06-22T13:00:00.22Z\",\n        \"event\": \"comment\"\n    }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    // # Search\n    let query_parser = QueryParser::for_index(&index, vec![event_type]);\n    {\n        // Simple exact search on the date\n        let query = query_parser.parse_query(\"occurred_at:\\\"2022-06-22T12:53:50.53Z\\\"\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(5).order_by_score())?;\n        assert_eq!(count_docs.len(), 1);\n    }\n    {\n        // Range query on the date field\n        let query = query_parser\n            .parse_query(r#\"occurred_at:[2022-06-22T12:58:00Z TO 2022-06-23T00:00:00Z}\"#)?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(4).order_by_score())?;\n        assert_eq!(count_docs.len(), 1);\n        for (_score, doc_address) in count_docs {\n            let retrieved_doc = searcher.doc::<TantivyDocument>(doc_address)?;\n            assert!(retrieved_doc\n                .get_first(occurred_at)\n                .unwrap()\n                .as_value()\n                .as_datetime()\n                .is_some(),);\n            assert_eq!(\n                retrieved_doc.to_json(&schema),\n                r#\"{\"event\":[\"comment\"],\"occurred_at\":[\"2022-06-22T13:00:00.22Z\"]}\"#\n            );\n        }\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "examples/deleting_updating_documents.rs",
    "content": "// # Deleting and Updating (?) documents\n//\n// This example explains how to delete and update documents.\n// In fact there is actually no such thing as an update in tantivy.\n//\n// To update a document, you need to delete a document and then reinsert\n// its new version.\n//\n// ---\n// Importing tantivy...\nuse tantivy::collector::TopDocs;\nuse tantivy::query::TermQuery;\nuse tantivy::schema::*;\nuse tantivy::{doc, Index, IndexReader, IndexWriter};\n\n// A simple helper function to fetch a single document\n// given its id from our index.\n// It will be helpful to check our work.\nfn extract_doc_given_isbn(\n    reader: &IndexReader,\n    isbn_term: &Term,\n) -> tantivy::Result<Option<TantivyDocument>> {\n    let searcher = reader.searcher();\n\n    // This is the simplest query you can think of.\n    // It matches all of the documents containing a specific term.\n    //\n    // The second argument is here to tell we don't care about decoding positions,\n    // or term frequencies.\n    let term_query = TermQuery::new(isbn_term.clone(), IndexRecordOption::Basic);\n    let top_docs = searcher.search(&term_query, &TopDocs::with_limit(1).order_by_score())?;\n\n    if let Some((_score, doc_address)) = top_docs.first() {\n        let doc = searcher.doc(*doc_address)?;\n        Ok(Some(doc))\n    } else {\n        // no doc matching this ID.\n        Ok(None)\n    }\n}\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    //\n    // Check out the *basic_search* example if this makes\n    // small sense to you.\n    let mut schema_builder = Schema::builder();\n\n    // Tantivy does not really have a notion of primary id.\n    // This may change in the future.\n    //\n    // Still, we can create a `isbn` field and use it as an id. This\n    // field can be `u64` or a `text`, depending on your use case.\n    // It just needs to be indexed.\n    //\n    // If it is `text`, let's make sure to keep it `raw` and let's avoid\n    // running any text processing on it.\n    // This is done by associating this field to the tokenizer named `raw`.\n    // Rather than building our\n    // [`TextOptions`](//docs.rs/tantivy/~0/tantivy/schema/struct.TextOptions.html) manually, We\n    // use the `STRING` shortcut. `STRING` stands for indexed (without term frequency or positions)\n    // and untokenized.\n    //\n    // Because we also want to be able to see this `id` in our returned documents,\n    // we also mark the field as stored.\n    let isbn = schema_builder.add_text_field(\"isbn\", STRING | STORED);\n    let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema.clone());\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // Let's add a couple of documents, for the sake of the example.\n    let mut old_man_doc = TantivyDocument::default();\n    old_man_doc.add_text(title, \"The Old Man and the Sea\");\n    index_writer.add_document(doc!(\n        isbn => \"978-0099908401\",\n        title => \"The old Man and the see\"\n    ))?;\n    index_writer.add_document(doc!(\n        isbn => \"978-0140177398\",\n        title => \"Of Mice and Men\",\n    ))?;\n    index_writer.add_document(doc!(\n       title => \"Frankentein\", //< Oops there is a typo here.\n       isbn => \"978-9176370711\",\n    ))?;\n    index_writer.commit()?;\n    let reader = index.reader()?;\n\n    let frankenstein_isbn = Term::from_field_text(isbn, \"978-9176370711\");\n\n    // Oops our frankenstein doc seems misspelled\n    let frankenstein_doc_misspelled = extract_doc_given_isbn(&reader, &frankenstein_isbn)?.unwrap();\n    assert_eq!(\n        frankenstein_doc_misspelled.to_json(&schema),\n        r#\"{\"isbn\":[\"978-9176370711\"],\"title\":[\"Frankentein\"]}\"#,\n    );\n\n    // # Update = Delete + Insert\n    //\n    // Here we will want to update the typo in the `Frankenstein` book.\n    //\n    // Tantivy does not handle updates directly, we need to delete\n    // and reinsert the document.\n    //\n    // This can be complicated as it means you need to have access\n    // to the entire document. It is good practise to integrate tantivy\n    // with a key value store for this reason.\n    //\n    // To remove one of the document, we just call `delete_term`\n    // on its id.\n    //\n    // Note that `tantivy` does nothing to enforce the idea that\n    // there is only one document associated with this id.\n    //\n    // Also you might have noticed that we apply the delete before\n    // having committed. This does not matter really...\n    index_writer.delete_term(frankenstein_isbn.clone());\n\n    // We now need to reinsert our document without the typo.\n    index_writer.add_document(doc!(\n       title => \"Frankenstein\",\n       isbn => \"978-9176370711\",\n    ))?;\n\n    // You are guaranteed that your clients will only observe your index in\n    // the state it was in after a commit.\n    // In this example, your search engine will at no point be missing the *Frankenstein* document.\n    // Everything happened as if the document was updated.\n    index_writer.commit()?;\n    // We reload our searcher to make our change available to clients.\n    reader.reload()?;\n\n    // No more typo!\n    let frankenstein_new_doc = extract_doc_given_isbn(&reader, &frankenstein_isbn)?.unwrap();\n    assert_eq!(\n        frankenstein_new_doc.to_json(&schema),\n        r#\"{\"isbn\":[\"978-9176370711\"],\"title\":[\"Frankenstein\"]}\"#,\n    );\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/faceted_search.rs",
    "content": "// # Faceted Search\n//\n// This example covers the faceted search functionalities of\n// tantivy.\n//\n// We will :\n// - define a text field \"name\" in our schema\n// - define a facet field \"classification\" in our schema\n// - create an index in memory\n// - index few documents with respective facets in our index\n// - search and count the number of documents that the classifications start the facet \"/Felidae\"\n// - Search the facet \"/Felidae/Pantherinae\" and count the number of documents that the\n//   classifications include the facet.\n//\n// ---\n// Importing tantivy...\nuse tantivy::collector::FacetCollector;\nuse tantivy::query::{AllQuery, TermQuery};\nuse tantivy::schema::*;\nuse tantivy::{doc, Index, IndexWriter};\n\nfn main() -> tantivy::Result<()> {\n    // Let's create a temporary directory for the sake of this example\n    let mut schema_builder = Schema::builder();\n\n    let name = schema_builder.add_text_field(\"name\", TEXT | STORED);\n    // this is our faceted field: its scientific classification\n    let classification = schema_builder.add_facet_field(\"classification\", FacetOptions::default());\n\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n\n    let mut index_writer: IndexWriter = index.writer(30_000_000)?;\n\n    // For convenience, tantivy also comes with a macro to\n    // reduce the boilerplate above.\n    index_writer.add_document(doc!(\n        name => \"Cat\",\n        classification => Facet::from(\"/Felidae/Felinae/Felis\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Canada lynx\",\n        classification => Facet::from(\"/Felidae/Felinae/Lynx\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Cheetah\",\n        classification => Facet::from(\"/Felidae/Felinae/Acinonyx\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Tiger\",\n        classification => Facet::from(\"/Felidae/Pantherinae/Panthera\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Lion\",\n        classification => Facet::from(\"/Felidae/Pantherinae/Panthera\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Jaguar\",\n        classification => Facet::from(\"/Felidae/Pantherinae/Panthera\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Sunda clouded leopard\",\n        classification => Facet::from(\"/Felidae/Pantherinae/Neofelis\")\n    ))?;\n    index_writer.add_document(doc!(\n        name => \"Fossa\",\n        classification => Facet::from(\"/Eupleridae/Cryptoprocta\")\n    ))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n    {\n        let mut facet_collector = FacetCollector::for_field(\"classification\");\n        facet_collector.add_facet(\"/Felidae\");\n        let facet_counts = searcher.search(&AllQuery, &facet_collector)?;\n        // This lists all of the facet counts, right below \"/Felidae\".\n        let facets: Vec<(&Facet, u64)> = facet_counts.get(\"/Felidae\").collect();\n        assert_eq!(\n            facets,\n            vec![\n                (&Facet::from(\"/Felidae/Felinae\"), 3),\n                (&Facet::from(\"/Felidae/Pantherinae\"), 4),\n            ]\n        );\n    }\n\n    // Facets are also searchable.\n    //\n    // For instance a common UI pattern is to allow the user someone to click on a facet link\n    // (e.g: `Pantherinae`) to drill down and filter the current result set with this subfacet.\n    //\n    // The search would then look as follows.\n\n    // Check the reference doc for different ways to create a `Facet` object.\n    {\n        let facet = Facet::from(\"/Felidae/Pantherinae\");\n        let facet_term = Term::from_facet(classification, &facet);\n        let facet_term_query = TermQuery::new(facet_term, IndexRecordOption::Basic);\n        let mut facet_collector = FacetCollector::for_field(\"classification\");\n        facet_collector.add_facet(\"/Felidae/Pantherinae\");\n        let facet_counts = searcher.search(&facet_term_query, &facet_collector)?;\n        let facets: Vec<(&Facet, u64)> = facet_counts.get(\"/Felidae/Pantherinae\").collect();\n        assert_eq!(\n            facets,\n            vec![\n                (&Facet::from(\"/Felidae/Pantherinae/Neofelis\"), 1),\n                (&Facet::from(\"/Felidae/Pantherinae/Panthera\"), 3),\n            ]\n        );\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/faceted_search_with_tweaked_score.rs",
    "content": "// # Faceted Search With Tweak Score\n//\n// This example covers the faceted search functionalities of\n// tantivy.\n//\n// We will :\n// - define a text field \"name\" in our schema\n// - define a facet field \"classification\" in our schema\n\nuse std::collections::HashSet;\n\nuse tantivy::collector::TopDocs;\nuse tantivy::query::BooleanQuery;\nuse tantivy::schema::*;\nuse tantivy::{doc, DocId, Index, IndexWriter, Score, SegmentReader};\n\nfn main() -> tantivy::Result<()> {\n    let mut schema_builder = Schema::builder();\n\n    let title = schema_builder.add_text_field(\"title\", STORED);\n    let ingredient = schema_builder.add_facet_field(\"ingredient\", FacetOptions::default());\n\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n\n    let mut index_writer: IndexWriter = index.writer(30_000_000)?;\n\n    index_writer.add_document(doc!(\n        title => \"Fried egg\",\n        ingredient => Facet::from(\"/ingredient/egg\"),\n        ingredient => Facet::from(\"/ingredient/oil\"),\n    ))?;\n    index_writer.add_document(doc!(\n        title => \"Scrambled egg\",\n        ingredient => Facet::from(\"/ingredient/egg\"),\n        ingredient => Facet::from(\"/ingredient/butter\"),\n        ingredient => Facet::from(\"/ingredient/milk\"),\n        ingredient => Facet::from(\"/ingredient/salt\"),\n    ))?;\n    index_writer.add_document(doc!(\n        title => \"Egg rolls\",\n        ingredient => Facet::from(\"/ingredient/egg\"),\n        ingredient => Facet::from(\"/ingredient/garlic\"),\n        ingredient => Facet::from(\"/ingredient/salt\"),\n        ingredient => Facet::from(\"/ingredient/oil\"),\n        ingredient => Facet::from(\"/ingredient/tortilla-wrap\"),\n        ingredient => Facet::from(\"/ingredient/mushroom\"),\n    ))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n    {\n        let facets = [\n            Facet::from(\"/ingredient/egg\"),\n            Facet::from(\"/ingredient/oil\"),\n            Facet::from(\"/ingredient/garlic\"),\n            Facet::from(\"/ingredient/mushroom\"),\n        ];\n        let query = BooleanQuery::new_multiterms_query(\n            facets\n                .iter()\n                .map(|key| Term::from_facet(ingredient, key))\n                .collect(),\n        );\n        let top_docs_by_custom_score =\n            // Call TopDocs with a custom tweak score\n            TopDocs::with_limit(2).tweak_score(move |segment_reader: &SegmentReader| {\n                let ingredient_reader = segment_reader.facet_reader(\"ingredient\").unwrap();\n                let facet_dict = ingredient_reader.facet_dict();\n\n                let query_ords: HashSet<u64> = facets\n                    .iter()\n                    .filter_map(|key| facet_dict.term_ord(key.encoded_str()).unwrap())\n                    .collect();\n\n                move |doc: DocId, original_score: Score| {\n                    // Update the original score with a tweaked score\n                    let missing_ingredients = ingredient_reader\n                        .facet_ords(doc)\n                        .filter(|ord| !query_ords.contains(ord))\n                        .count();\n                    let tweak = 1.0 / 4_f32.powi(missing_ingredients as i32);\n\n                    original_score * tweak\n                }\n            });\n        let top_docs = searcher.search(&query, &top_docs_by_custom_score)?;\n\n        let titles: Vec<String> = top_docs\n            .iter()\n            .map(|(_, doc_id)| {\n                searcher\n                    .doc::<TantivyDocument>(*doc_id)\n                    .unwrap()\n                    .get_first(title)\n                    .and_then(|v| v.as_str().map(|el| el.to_string()))\n                    .unwrap()\n            })\n            .collect();\n        assert_eq!(titles, vec![\"Fried egg\", \"Egg rolls\"]);\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "examples/filter_aggregation.rs",
    "content": "// # Filter Aggregation Example\n//\n// This example demonstrates filter aggregations - creating buckets of documents\n// matching specific queries, with nested aggregations computed on each bucket.\n//\n// Filter aggregations are useful for computing metrics on different subsets of\n// your data in a single query, like \"average price overall + average price for\n// electronics + count of in-stock items\".\n\nuse serde_json::json;\nuse tantivy::aggregation::agg_req::Aggregations;\nuse tantivy::aggregation::AggregationCollector;\nuse tantivy::query::AllQuery;\nuse tantivy::schema::{Schema, FAST, INDEXED, TEXT};\nuse tantivy::{doc, Index};\n\nfn main() -> tantivy::Result<()> {\n    // Create a simple product schema\n    let mut schema_builder = Schema::builder();\n    schema_builder.add_text_field(\"category\", TEXT | FAST);\n    schema_builder.add_text_field(\"brand\", TEXT | FAST);\n    schema_builder.add_u64_field(\"price\", FAST);\n    schema_builder.add_f64_field(\"rating\", FAST);\n    schema_builder.add_bool_field(\"in_stock\", FAST | INDEXED);\n    let schema = schema_builder.build();\n\n    // Create index and add sample products\n    let index = Index::create_in_ram(schema.clone());\n    let mut writer = index.writer(50_000_000)?;\n\n    writer.add_document(doc!(\n        schema.get_field(\"category\")? => \"electronics\",\n        schema.get_field(\"brand\")? => \"apple\",\n        schema.get_field(\"price\")? => 999u64,\n        schema.get_field(\"rating\")? => 4.5f64,\n        schema.get_field(\"in_stock\")? => true\n    ))?;\n    writer.add_document(doc!(\n        schema.get_field(\"category\")? => \"electronics\",\n        schema.get_field(\"brand\")? => \"samsung\",\n        schema.get_field(\"price\")? => 799u64,\n        schema.get_field(\"rating\")? => 4.2f64,\n        schema.get_field(\"in_stock\")? => true\n    ))?;\n    writer.add_document(doc!(\n        schema.get_field(\"category\")? => \"clothing\",\n        schema.get_field(\"brand\")? => \"nike\",\n        schema.get_field(\"price\")? => 120u64,\n        schema.get_field(\"rating\")? => 4.1f64,\n        schema.get_field(\"in_stock\")? => false\n    ))?;\n    writer.add_document(doc!(\n        schema.get_field(\"category\")? => \"books\",\n        schema.get_field(\"brand\")? => \"penguin\",\n        schema.get_field(\"price\")? => 25u64,\n        schema.get_field(\"rating\")? => 4.8f64,\n        schema.get_field(\"in_stock\")? => true\n    ))?;\n\n    writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    // Example 1: Basic filter with metric aggregation\n    println!(\"=== Example 1: Electronics average price ===\");\n    let agg_req = json!({\n        \"electronics\": {\n            \"filter\": \"category:electronics\",\n            \"aggs\": {\n                \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n            }\n        }\n    });\n\n    let agg: Aggregations = serde_json::from_value(agg_req)?;\n    let collector = AggregationCollector::from_aggs(agg, Default::default());\n    let result = searcher.search(&AllQuery, &collector)?;\n\n    let expected = json!({\n        \"electronics\": {\n            \"doc_count\": 2,\n            \"avg_price\": { \"value\": 899.0 }\n        }\n    });\n    assert_eq!(serde_json::to_value(&result)?, expected);\n    println!(\"{}\\n\", serde_json::to_string_pretty(&result)?);\n\n    // Example 2: Multiple independent filters\n    println!(\"=== Example 2: Multiple filters in one query ===\");\n    let agg_req = json!({\n        \"electronics\": {\n            \"filter\": \"category:electronics\",\n            \"aggs\": { \"avg_price\": { \"avg\": { \"field\": \"price\" } } }\n        },\n        \"in_stock\": {\n            \"filter\": \"in_stock:true\",\n            \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n        },\n        \"high_rated\": {\n            \"filter\": \"rating:[4.5 TO *]\",\n            \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n        }\n    });\n\n    let agg: Aggregations = serde_json::from_value(agg_req)?;\n    let collector = AggregationCollector::from_aggs(agg, Default::default());\n    let result = searcher.search(&AllQuery, &collector)?;\n\n    let expected = json!({\n        \"electronics\": {\n            \"doc_count\": 2,\n            \"avg_price\": { \"value\": 899.0 }\n        },\n        \"in_stock\": {\n            \"doc_count\": 3,\n            \"count\": { \"value\": 3.0 }\n        },\n        \"high_rated\": {\n            \"doc_count\": 2,\n            \"count\": { \"value\": 2.0 }\n        }\n    });\n    assert_eq!(serde_json::to_value(&result)?, expected);\n    println!(\"{}\\n\", serde_json::to_string_pretty(&result)?);\n\n    // Example 3: Nested filters - progressive refinement\n    println!(\"=== Example 3: Nested filters ===\");\n    let agg_req = json!({\n        \"in_stock\": {\n            \"filter\": \"in_stock:true\",\n            \"aggs\": {\n                \"electronics\": {\n                    \"filter\": \"category:electronics\",\n                    \"aggs\": {\n                        \"expensive\": {\n                            \"filter\": \"price:[800 TO *]\",\n                            \"aggs\": {\n                                \"avg_rating\": { \"avg\": { \"field\": \"rating\" } }\n                            }\n                        }\n                    }\n                }\n            }\n        }\n    });\n\n    let agg: Aggregations = serde_json::from_value(agg_req)?;\n    let collector = AggregationCollector::from_aggs(agg, Default::default());\n    let result = searcher.search(&AllQuery, &collector)?;\n\n    let expected = json!({\n        \"in_stock\": {\n            \"doc_count\": 3,  // apple, samsung, penguin\n            \"electronics\": {\n                \"doc_count\": 2,  // apple, samsung\n                \"expensive\": {\n                    \"doc_count\": 1,  // only apple (999)\n                    \"avg_rating\": { \"value\": 4.5 }\n                }\n            }\n        }\n    });\n    assert_eq!(serde_json::to_value(&result)?, expected);\n    println!(\"{}\\n\", serde_json::to_string_pretty(&result)?);\n\n    // Example 4: Filter with sub-aggregation (terms)\n    println!(\"=== Example 4: Filter with terms sub-aggregation ===\");\n    let agg_req = json!({\n        \"electronics\": {\n            \"filter\": \"category:electronics\",\n            \"aggs\": {\n                \"by_brand\": {\n                    \"terms\": { \"field\": \"brand\" },\n                    \"aggs\": {\n                        \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n                    }\n                }\n            }\n        }\n    });\n\n    let agg: Aggregations = serde_json::from_value(agg_req)?;\n    let collector = AggregationCollector::from_aggs(agg, Default::default());\n    let result = searcher.search(&AllQuery, &collector)?;\n\n    let expected = json!({\n        \"electronics\": {\n            \"doc_count\": 2,\n            \"by_brand\": {\n                \"buckets\": [\n                    {\n                        \"key\": \"samsung\",\n                        \"doc_count\": 1,\n                        \"avg_price\": { \"value\": 799.0 }\n                    },\n                    {\n                        \"key\": \"apple\",\n                        \"doc_count\": 1,\n                        \"avg_price\": { \"value\": 999.0 }\n                    }\n                ],\n                \"sum_other_doc_count\": 0,\n                \"doc_count_error_upper_bound\": 0\n            }\n        }\n    });\n    assert_eq!(serde_json::to_value(&result)?, expected);\n    println!(\"{}\", serde_json::to_string_pretty(&result)?);\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/fuzzy_search.rs",
    "content": "// # Basic Example\n//\n// This example covers the basic functionalities of\n// tantivy.\n//\n// We will :\n// - define our schema\n// - create an index in a directory\n// - index a few documents into our index\n// - search for the best document matching a basic query\n// - retrieve the best document's original content.\n// ---\n// Importing tantivy...\nuse tantivy::collector::{Count, TopDocs};\nuse tantivy::query::FuzzyTermQuery;\nuse tantivy::schema::*;\nuse tantivy::{doc, Index, IndexWriter, ReloadPolicy};\nuse tempfile::TempDir;\n\nfn main() -> tantivy::Result<()> {\n    // Let's create a temporary directory for the\n    // sake of this example\n    let index_path = TempDir::new()?;\n\n    // # Defining the schema\n    //\n    // The Tantivy index requires a very strict schema.\n    // The schema declares which fields are in the index,\n    // and for each field, its type and \"the way it should\n    // be indexed\".\n\n    // First we need to define a schema ...\n    let mut schema_builder = Schema::builder();\n\n    // Our first field is title.\n    // We want full-text search for it, and we also want\n    // to be able to retrieve the document after the search.\n    //\n    // `TEXT | STORED` is some syntactic sugar to describe\n    // that.\n    //\n    // `TEXT` means the field should be tokenized and indexed,\n    // along with its term frequency and term positions.\n    //\n    // `STORED` means that the field will also be saved\n    // in a compressed, row-oriented key-value store.\n    // This store is useful for reconstructing the\n    // documents that were selected during the search phase.\n    let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    //\n    // Let's create a brand new index.\n    //\n    // This will actually just save a meta.json\n    // with our schema in the directory.\n    let index = Index::create_in_dir(&index_path, schema.clone())?;\n\n    // To insert a document we will need an index writer.\n    // There must be only one writer at a time.\n    // This single `IndexWriter` is already\n    // multithreaded.\n    //\n    // Here we give tantivy a budget of `50MB`.\n    // Using a bigger memory_arena for the indexer may increase\n    // throughput, but 50 MB is already plenty.\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // Let's index our documents!\n    // We first need a handle on the title and the body field.\n\n    // ### Adding documents\n    //\n    index_writer.add_document(doc!(\n        title => \"The Name of the Wind\",\n    ))?;\n    index_writer.add_document(doc!(\n        title => \"The Diary of Muadib\",\n    ))?;\n    index_writer.add_document(doc!(\n        title => \"A Dairy Cow\",\n    ))?;\n    index_writer.add_document(doc!(\n        title => \"The Diary of a Young Girl\",\n    ))?;\n\n    // ### Committing\n    //\n    // At this point our documents are not searchable.\n    //\n    //\n    // We need to call `.commit()` explicitly to force the\n    // `index_writer` to finish processing the documents in the queue,\n    // flush the current index to the disk, and advertise\n    // the existence of new documents.\n    //\n    // This call is blocking.\n    index_writer.commit()?;\n\n    // If `.commit()` returns correctly, then all of the\n    // documents that have been added are guaranteed to be\n    // persistently indexed.\n    //\n    // In the scenario of a crash or a power failure,\n    // tantivy behaves as if it has rolled back to its last\n    // commit.\n\n    // # Searching\n    //\n    // ### Searcher\n    //\n    // A reader is required first in order to search an index.\n    // It acts as a `Searcher` pool that reloads itself,\n    // depending on a `ReloadPolicy`.\n    //\n    // For a search server you will typically create one reader for the entire lifetime of your\n    // program, and acquire a new searcher for every single request.\n    //\n    // In the code below, we rely on the 'ON_COMMIT' policy: the reader\n    // will reload the index automatically after each commit.\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::OnCommitWithDelay)\n        .try_into()?;\n\n    // We now need to acquire a searcher.\n    //\n    // A searcher points to a snapshotted, immutable version of the index.\n    //\n    // Some search experience might require more than\n    // one query. Using the same searcher ensures that all of these queries will run on the\n    // same version of the index.\n    //\n    // Acquiring a `searcher` is very cheap.\n    //\n    // You should acquire a searcher every time you start processing a request and\n    // and release it right after your query is finished.\n    let searcher = reader.searcher();\n\n    // ### FuzzyTermQuery\n    {\n        let term = Term::from_field_text(title, \"Diary\");\n        let query = FuzzyTermQuery::new(term, 2, true);\n\n        let (top_docs, count) = searcher\n            .search(&query, &(TopDocs::with_limit(5).order_by_score(), Count))\n            .unwrap();\n        assert_eq!(count, 3);\n        assert_eq!(top_docs.len(), 3);\n        for (score, doc_address) in top_docs {\n            // Note that the score is not lower for the fuzzy hit.\n            // There's an issue open for that: https://github.com/quickwit-oss/tantivy/issues/563\n            let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n            println!(\"score {score:?} doc {}\", retrieved_doc.to_json(&schema));\n            // score 1.0 doc {\"title\":[\"The Diary of Muadib\"]}\n            //\n            // score 1.0 doc {\"title\":[\"The Diary of a Young Girl\"]}\n            //\n            // score 1.0 doc {\"title\":[\"A Dairy Cow\"]}\n        }\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/index_from_multiple_threads.rs",
    "content": "// # Indexing from different threads.\n//\n// It is fairly common to have to index from different threads.\n// Tantivy forbids to create more than one `IndexWriter` at a time.\n//\n// This `IndexWriter` itself has its own multithreaded layer, so managing your own\n// indexing threads will not help. However, it can still be useful for some applications.\n//\n// For instance, if preparing documents to send to tantivy before indexing is the bottleneck of\n// your application, it is reasonable to have multiple threads.\n//\n// Another very common reason to want to index from multiple threads, is implementing a webserver\n// with CRUD capabilities. The server framework will most likely handle request from\n// different threads.\n//\n// The recommended way to address both of these use case is to wrap your `IndexWriter` into a\n// `Arc<RwLock<IndexWriter>>`.\n//\n// While this is counterintuitive, adding and deleting documents do not require mutability\n// over the `IndexWriter`, so several threads will be able to do this operation concurrently.\n//\n// The example below does not represent an actual real-life use case (who would spawn thread to\n// index a single document?), but aims at demonstrating the mechanism that makes indexing\n// from several threads possible.\n\n// ---\n// Importing tantivy...\nuse std::sync::{Arc, RwLock};\nuse std::thread;\nuse std::time::Duration;\n\nuse tantivy::schema::{Schema, STORED, TEXT};\nuse tantivy::{doc, Index, IndexWriter, Opstamp, TantivyError};\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    let mut schema_builder = Schema::builder();\n    let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n    let body = schema_builder.add_text_field(\"body\", TEXT);\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema);\n    let index_writer: Arc<RwLock<IndexWriter>> = Arc::new(RwLock::new(index.writer(50_000_000)?));\n\n    // # First indexing thread.\n    let index_writer_clone_1 = index_writer.clone();\n    thread::spawn(move || {\n        // we index 100 times the document... for the sake of the example.\n        for i in 0..100 {\n            let opstamp = index_writer_clone_1\n                .read().unwrap() //< A read lock is sufficient here.\n                .add_document(\n                    doc!(\n                        title => \"Of Mice and Men\",\n                        body => \"A few miles south of Soledad, the Salinas River drops in close to the hillside \\\n                        bank and runs deep and green. The water is warm too, for it has slipped twinkling \\\n                        over the yellow sands in the sunlight before reaching the narrow pool. On one \\\n                        side of the river the golden foothill slopes curve up to the strong and rocky \\\n                        Gabilan Mountains, but on the valley side the water is lined with trees—willows \\\n                        fresh and green with every spring, carrying in their lower leaf junctures the \\\n                        debris of the winter’s flooding; and sycamores with mottled, white, recumbent \\\n                        limbs and branches that arch over the pool\"\n                    ))?;\n            println!(\"add doc {i} from thread 1 - opstamp {opstamp}\");\n            thread::sleep(Duration::from_millis(20));\n        }\n        Result::<(), TantivyError>::Ok(())\n    });\n\n    // # Second indexing thread.\n    let index_writer_clone_2 = index_writer.clone();\n    // For convenience, tantivy also comes with a macro to\n    // reduce the boilerplate above.\n    thread::spawn(move || {\n        // we index 100 times the document... for the sake of the example.\n        for i in 0..100 {\n            // A read lock is sufficient here.\n            let opstamp = {\n                let index_writer_rlock = index_writer_clone_2.read().unwrap();\n                index_writer_rlock.add_document(doc!(\n                    title => \"Manufacturing consent\",\n                    body => \"Some great book description...\"\n                ))?\n            };\n            println!(\"add doc {i} from thread 2 - opstamp {opstamp}\");\n            thread::sleep(Duration::from_millis(10));\n        }\n        Result::<(), TantivyError>::Ok(())\n    });\n\n    // # In the main thread, we commit 10 times, once every 500ms.\n    for _ in 0..10 {\n        let opstamp: Opstamp = {\n            // Committing or rollbacking on the other hand requires write lock. This will block\n            // other threads.\n            let mut index_writer_wlock = index_writer.write().unwrap();\n            index_writer_wlock.commit()?\n        };\n        println!(\"committed with opstamp {opstamp}\");\n        thread::sleep(Duration::from_millis(500));\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/index_with_json.rs",
    "content": "use tantivy::schema::*;\n\n// # Document from json\n//\n// For convenience, `Document` can be parsed directly from json.\nfn main() -> tantivy::Result<()> {\n    // Let's first define a schema and an index.\n    // Check out the basic example if this is confusing to you.\n    //\n    // first we need to define a schema ...\n    let mut schema_builder = Schema::builder();\n    schema_builder.add_text_field(\"title\", TEXT | STORED);\n    schema_builder.add_text_field(\"body\", TEXT);\n    schema_builder.add_u64_field(\"year\", INDEXED);\n    let schema = schema_builder.build();\n\n    // Let's assume we have a json-serialized document.\n    let mice_and_men_doc_json = r#\"{\n       \"title\": \"Of Mice and Men\",\n       \"year\": 1937\n    }\"#;\n\n    // We can parse our document\n    let _mice_and_men_doc = TantivyDocument::parse_json(&schema, mice_and_men_doc_json)?;\n\n    // Multi-valued field are allowed, they are\n    // expressed in JSON by an array.\n    // The following document has two titles.\n    let frankenstein_json = r#\"{\n       \"title\": [\"Frankenstein\", \"The Modern Prometheus\"],\n       \"year\": 1818\n    }\"#;\n    let _frankenstein_doc = TantivyDocument::parse_json(&schema, frankenstein_json)?;\n\n    // Note that the schema is saved in your index directory.\n    //\n    // As a result, Indexes are aware of their schema, and you can use this feature\n    // just by opening an existing `Index`, and calling `index.schema()..parse_document(json)`.\n    Ok(())\n}\n"
  },
  {
    "path": "examples/integer_range_search.rs",
    "content": "use std::ops::Bound;\n\n// # Searching a range on an indexed int field.\n//\n// Below is an example of creating an indexed integer field in your schema\n// You can use RangeQuery to get a Count of all occurrences in a given range.\nuse tantivy::collector::Count;\nuse tantivy::query::RangeQuery;\nuse tantivy::schema::{Schema, INDEXED};\nuse tantivy::{doc, Index, IndexWriter, Result, Term};\n\nfn main() -> Result<()> {\n    // For the sake of simplicity, this schema will only have 1 field\n    let mut schema_builder = Schema::builder();\n\n    // `INDEXED` is a short-hand to indicate that our field should be \"searchable\".\n    let year_field = schema_builder.add_u64_field(\"year\", INDEXED);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let reader = index.reader()?;\n    {\n        let mut index_writer: IndexWriter = index.writer_with_num_threads(1, 6_000_000)?;\n        for year in 1950u64..2019u64 {\n            index_writer.add_document(doc!(year_field => year))?;\n        }\n        index_writer.commit()?;\n        // The index will be a range of years\n    }\n    reader.reload()?;\n    let searcher = reader.searcher();\n    // The end is excluded i.e. here we are searching up to 1969\n    let docs_in_the_sixties = RangeQuery::new(\n        Bound::Included(Term::from_field_u64(year_field, 1960)),\n        Bound::Excluded(Term::from_field_u64(year_field, 1970)),\n    );\n    // Uses a Count collector to sum the total number of docs in the range\n    let num_60s_books = searcher.search(&docs_in_the_sixties, &Count)?;\n    assert_eq!(num_60s_books, 10);\n    Ok(())\n}\n"
  },
  {
    "path": "examples/ip_field.rs",
    "content": "// # IP Address example\n//\n// This example shows how the ip field can be used\n// with IpV6 and IpV4.\n\nuse tantivy::collector::{Count, TopDocs};\nuse tantivy::query::QueryParser;\nuse tantivy::schema::{Schema, FAST, INDEXED, STORED, STRING};\nuse tantivy::{Index, IndexWriter, TantivyDocument};\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    // We set the IP field as `INDEXED`, so it can be searched\n    // `FAST` will create a fast field. The fast field will be used to execute search queries.\n    // `FAST` is not a requirement for range queries, it can also be executed on the inverted index\n    // which is created by `INDEXED`.\n    let mut schema_builder = Schema::builder();\n    let event_type = schema_builder.add_text_field(\"event_type\", STRING | STORED);\n    let ip = schema_builder.add_ip_addr_field(\"ip\", STORED | INDEXED | FAST);\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    let index = Index::create_in_ram(schema.clone());\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // ### IPv4\n    // Adding documents that contain an IPv4 address. Notice that the IP addresses are passed as\n    // `String`. Since the field is of type ip, we parse the IP address from the string and store it\n    // internally as IPv6.\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n            \"ip\": \"192.168.0.33\",\n            \"event_type\": \"login\"\n        }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n            \"ip\": \"192.168.0.80\",\n            \"event_type\": \"checkout\"\n        }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    // ### IPv6\n    // Adding a document that contains an IPv6 address.\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n            \"ip\": \"2001:0db8:85a3:0000:0000:8a2e:0370:7334\",\n            \"event_type\": \"checkout\"\n        }\"#,\n    )?;\n\n    index_writer.add_document(doc)?;\n    // Commit will create a segment containing our documents.\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    // # Search\n    // Range queries on IPv4. Since we created a fast field, the fast field will be used to execute\n    // the search.\n    // ### Range Queries\n    let query_parser = QueryParser::for_index(&index, vec![event_type, ip]);\n    {\n        // Inclusive range queries\n        let query = query_parser.parse_query(\"ip:[192.168.0.80 TO 192.168.0.100]\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(5).order_by_score())?;\n        assert_eq!(count_docs.len(), 1);\n    }\n    {\n        // Exclusive range queries\n        let query = query_parser.parse_query(\"ip:{192.168.0.80 TO 192.168.1.100]\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(count_docs.len(), 0);\n    }\n    {\n        // Find docs with IP addresses smaller equal 192.168.1.100\n        let query = query_parser.parse_query(\"ip:[* TO 192.168.1.100]\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(count_docs.len(), 2);\n    }\n    {\n        // Find docs with IP addresses smaller than 192.168.1.100\n        let query = query_parser.parse_query(\"ip:[* TO 192.168.1.100}\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(count_docs.len(), 2);\n    }\n\n    // ### Exact Queries\n    // Exact search on IPv4.\n    {\n        let query = query_parser.parse_query(\"ip:192.168.0.80\")?;\n        let count_docs = searcher.search(&*query, &Count)?;\n        assert_eq!(count_docs, 1);\n    }\n    // Exact search on IPv6.\n    // IpV6 addresses need to be quoted because they contain `:`\n    {\n        let query = query_parser.parse_query(\"ip:\\\"2001:0db8:85a3:0000:0000:8a2e:0370:7334\\\"\")?;\n        let count_docs = searcher.search(&*query, &Count)?;\n        assert_eq!(count_docs, 1);\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/iterating_docs_and_positions.rs",
    "content": "// # Iterating docs and positions.\n//\n// At its core of tantivy, relies on a data structure\n// called an inverted index.\n//\n// This example shows how to manually iterate through\n// the list of documents containing a term, getting\n// its term frequency, and accessing its positions.\n\nuse tantivy::postings::Postings;\n// ---\n// Importing tantivy...\nuse tantivy::schema::*;\nuse tantivy::{doc, DocSet, Index, IndexWriter, TERMINATED};\n\nfn main() -> tantivy::Result<()> {\n    // We first create a schema for the sake of the\n    // example. Check the `basic_search` example for more information.\n    let mut schema_builder = Schema::builder();\n\n    // For this example, we need to make sure to index positions for our title\n    // field. `TEXT` precisely does this.\n    let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema);\n\n    let mut index_writer: IndexWriter = index.writer_with_num_threads(1, 50_000_000)?;\n    index_writer.add_document(doc!(title => \"The Old Man and the Sea\"))?;\n    index_writer.add_document(doc!(title => \"Of Mice and Men\"))?;\n    index_writer.add_document(doc!(title => \"The modern Prometheus\"))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n\n    let searcher = reader.searcher();\n\n    // A tantivy index is actually a collection of segments.\n    // Similarly, a searcher just wraps a list `segment_reader`.\n    //\n    // (Because we indexed a very small number of documents over one thread\n    // there is actually only one segment here, but let's iterate through the list\n    // anyway)\n    for segment_reader in searcher.segment_readers() {\n        // A segment contains different data structure.\n        // Inverted index stands for the combination of\n        // - the term dictionary\n        // - the inverted lists associated with each terms and their positions\n        let inverted_index = segment_reader.inverted_index(title)?;\n\n        // A `Term` is a text token associated with a field.\n        // Let's go through all docs containing the term `title:the` and access their position\n        let term_the = Term::from_field_text(title, \"the\");\n\n        // This segment posting object is like a cursor over the documents matching the term.\n        // The `IndexRecordOption` arguments tells tantivy we will be interested in both term\n        // frequencies and positions.\n        //\n        // If you don't need all this information, you may get better performance by decompressing\n        // less information.\n        if let Some(mut segment_postings) =\n            inverted_index.read_postings(&term_the, IndexRecordOption::WithFreqsAndPositions)?\n        {\n            // this buffer will be used to request for positions\n            let mut positions: Vec<u32> = Vec::with_capacity(100);\n            let mut doc_id = segment_postings.doc();\n            while doc_id != TERMINATED {\n                // This MAY contains deleted documents as well.\n                if segment_reader.is_deleted(doc_id) {\n                    doc_id = segment_postings.advance();\n                    continue;\n                }\n\n                // the number of time the term appears in the document.\n                let term_freq: u32 = segment_postings.term_freq();\n                // accessing positions is slightly expensive and lazy, do not request\n                // for them if you don't need them for some documents.\n                segment_postings.positions(&mut positions);\n\n                // By definition we should have `term_freq` positions.\n                assert_eq!(positions.len(), term_freq as usize);\n\n                // This prints:\n                // ```\n                // Doc 0: TermFreq 2: [0, 4]\n                // Doc 2: TermFreq 1: [0]\n                // ```\n                println!(\"Doc {doc_id}: TermFreq {term_freq}: {positions:?}\");\n                doc_id = segment_postings.advance();\n            }\n        }\n    }\n\n    // A `Term` is a text token associated with a field.\n    // Let's go through all docs containing the term `title:the` and access their position\n    let term_the = Term::from_field_text(title, \"the\");\n\n    // Some other powerful operations (especially `.skip_to`) may be useful to consume these\n    // posting lists rapidly.\n    // You can check for them in the [`DocSet`](https://docs.rs/tantivy/~0/tantivy/trait.DocSet.html) trait\n    // and the [`Postings`](https://docs.rs/tantivy/~0/tantivy/trait.Postings.html) trait\n\n    // Also, for some VERY specific high performance use case like an OLAP analysis of logs,\n    // you can get better performance by accessing directly the blocks of doc ids.\n    for segment_reader in searcher.segment_readers() {\n        // A segment contains different data structure.\n        // Inverted index stands for the combination of\n        // - the term dictionary\n        // - the inverted lists associated with each terms and their positions\n        let inverted_index = segment_reader.inverted_index(title)?;\n\n        // This segment posting object is like a cursor over the documents matching the term.\n        // The `IndexRecordOption` arguments tells tantivy we will be interested in both term\n        // frequencies and positions.\n        //\n        // If you don't need all this information, you may get better performance by decompressing\n        // less information.\n        if let Some(mut block_segment_postings) =\n            inverted_index.read_block_postings(&term_the, IndexRecordOption::Basic)?\n        {\n            loop {\n                let docs = block_segment_postings.docs();\n                if docs.is_empty() {\n                    break;\n                }\n                // Once again these docs MAY contains deleted documents as well.\n                let docs = block_segment_postings.docs();\n                // Prints `Docs [0, 2].`\n                println!(\"Docs {docs:?}\");\n                block_segment_postings.advance();\n            }\n        }\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/json_field.rs",
    "content": "// # Json field example\n//\n// This example shows how the json field can be used\n// to make tantivy partially schemaless by setting it as\n// default query parser field.\n\nuse tantivy::collector::{Count, TopDocs};\nuse tantivy::query::QueryParser;\nuse tantivy::schema::{Schema, FAST, STORED, STRING, TEXT};\nuse tantivy::{Index, IndexWriter, TantivyDocument};\n\nfn main() -> tantivy::Result<()> {\n    // # Defining the schema\n    let mut schema_builder = Schema::builder();\n    schema_builder.add_date_field(\"timestamp\", FAST | STORED);\n    let event_type = schema_builder.add_text_field(\"event_type\", STRING | STORED);\n    let attributes = schema_builder.add_json_field(\"attributes\", STORED | TEXT);\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    let index = Index::create_in_ram(schema.clone());\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n        \"timestamp\": \"2022-02-22T23:20:50.53Z\",\n        \"event_type\": \"click\",\n        \"attributes\": {\n            \"target\": \"submit-button\",\n            \"cart\": {\"product_id\": 103},\n            \"description\": \"the best vacuum cleaner ever\"\n        }\n    }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    let doc = TantivyDocument::parse_json(\n        &schema,\n        r#\"{\n        \"timestamp\": \"2022-02-22T23:20:51.53Z\",\n        \"event_type\": \"click\",\n        \"attributes\": {\n            \"target\": \"submit-button\",\n            \"cart\": {\"product_id\": 133},\n            \"description\": \"das keyboard\",\n            \"event_type\": \"holiday-sale\"\n        }\n    }\"#,\n    )?;\n    index_writer.add_document(doc)?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    // # Default fields: event_type and attributes\n    // By setting attributes as a default field it allows omitting attributes itself, e.g. \"target\",\n    // instead of \"attributes.target\"\n    let query_parser = QueryParser::for_index(&index, vec![event_type, attributes]);\n    {\n        let query = query_parser.parse_query(\"target:submit-button\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(count_docs.len(), 2);\n    }\n    {\n        let query = query_parser.parse_query(\"target:submit\")?;\n        let count_docs = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(count_docs.len(), 2);\n    }\n    {\n        let query = query_parser.parse_query(\"cart.product_id:103\")?;\n        let count_docs = searcher.search(&*query, &Count)?;\n        assert_eq!(count_docs, 1);\n    }\n    {\n        let query = query_parser.parse_query(\"click AND cart.product_id:133\")?;\n        let hits = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(hits.len(), 1);\n    }\n    {\n        // The sub-fields in the json field marked as default field still need to be explicitly\n        // addressed\n        let query = query_parser.parse_query(\"click AND 133\")?;\n        let hits = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(hits.len(), 0);\n    }\n    {\n        // Default json fields are ignored if they collide with the schema\n        let query = query_parser.parse_query(\"event_type:holiday-sale\")?;\n        let hits = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(hits.len(), 0);\n    }\n    // # Query via full attribute path\n    {\n        // This only searches in our schema's `event_type` field\n        let query = query_parser.parse_query(\"event_type:click\")?;\n        let hits = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(hits.len(), 2);\n    }\n    {\n        // Default json fields can still be accessed by full path\n        let query = query_parser.parse_query(\"attributes.event_type:holiday-sale\")?;\n        let hits = searcher.search(&*query, &TopDocs::with_limit(2).order_by_score())?;\n        assert_eq!(hits.len(), 1);\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "examples/phrase_prefix_search.rs",
    "content": "use tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::{doc, Index, IndexWriter, ReloadPolicy, Result};\nuse tempfile::TempDir;\n\nfn main() -> Result<()> {\n    let index_path = TempDir::new()?;\n\n    let mut schema_builder = Schema::builder();\n    schema_builder.add_text_field(\"title\", TEXT | STORED);\n    schema_builder.add_text_field(\"body\", TEXT);\n    let schema = schema_builder.build();\n\n    let title = schema.get_field(\"title\").unwrap();\n    let body = schema.get_field(\"body\").unwrap();\n\n    let index = Index::create_in_dir(&index_path, schema)?;\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    index_writer.add_document(doc!(\n    title => \"The Old Man and the Sea\",\n    body => \"He was an old man who fished alone in a skiff in the Gulf Stream and he had gone \\\n            eighty-four days now without taking a fish.\",\n    ))?;\n\n    index_writer.add_document(doc!(\n    title => \"Of Mice and Men\",\n    body => \"A few miles south of Soledad, the Salinas River drops in close to the hillside \\\n            bank and runs deep and green. The water is warm too, for it has slipped twinkling \\\n            over the yellow sands in the sunlight before reaching the narrow pool. On one \\\n            side of the river the golden foothill slopes curve up to the strong and rocky \\\n            Gabilan Mountains, but on the valley side the water is lined with trees—willows \\\n            fresh and green with every spring, carrying in their lower leaf junctures the \\\n            debris of the winter’s flooding; and sycamores with mottled, white, recumbent \\\n            limbs and branches that arch over the pool\"\n    ))?;\n\n    // Multivalued field just need to be repeated.\n    index_writer.add_document(doc!(\n    title => \"Frankenstein\",\n    title => \"The Modern Prometheus\",\n    body => \"You will rejoice to hear that no disaster has accompanied the commencement of an \\\n             enterprise which you have regarded with such evil forebodings.  I arrived here \\\n             yesterday, and my first task is to assure my dear sister of my welfare and \\\n             increasing confidence in the success of my undertaking.\"\n    ))?;\n\n    index_writer.commit()?;\n\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::OnCommitWithDelay)\n        .try_into()?;\n\n    let searcher = reader.searcher();\n\n    let query_parser = QueryParser::for_index(&index, vec![title, body]);\n    // This will match documents containing the phrase \"in the\"\n    // followed by some word starting with \"su\",\n    // i.e. it will match \"in the sunlight\" and \"in the success\",\n    // but not \"in the Gulf Stream\".\n    let query = query_parser.parse_query(\"\\\"in the su\\\"*\")?;\n\n    let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n    let mut titles = top_docs\n        .into_iter()\n        .map(|(_score, doc_address)| {\n            let doc = searcher.doc::<TantivyDocument>(doc_address)?;\n            let title = doc\n                .get_first(title)\n                .and_then(|v| v.as_str())\n                .unwrap()\n                .to_owned();\n            Ok(title)\n        })\n        .collect::<Result<Vec<_>>>()?;\n    titles.sort_unstable();\n    assert_eq!(titles, [\"Frankenstein\", \"Of Mice and Men\"]);\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/pre_tokenized_text.rs",
    "content": "// # Pre-tokenized text example\n//\n// This example shows how to use pre-tokenized text. Sometimes you might\n// want to index and search through text which is already split into\n// tokens by some external tool.\n//\n// In this example we will:\n// - use tantivy tokenizer to create tokens and load them directly into tantivy,\n// - import tokenized text straight from json,\n// - perform a search on documents with pre-tokenized text\n\nuse tantivy::collector::{Count, TopDocs};\nuse tantivy::query::TermQuery;\nuse tantivy::schema::*;\nuse tantivy::tokenizer::{PreTokenizedString, SimpleTokenizer, Token, TokenStream, Tokenizer};\nuse tantivy::{doc, Index, IndexWriter, ReloadPolicy};\nuse tempfile::TempDir;\n\nfn pre_tokenize_text(text: &str) -> Vec<Token> {\n    let mut tokenizer = SimpleTokenizer::default();\n    let mut token_stream = tokenizer.token_stream(text);\n    let mut tokens = vec![];\n    while token_stream.advance() {\n        tokens.push(token_stream.token().clone());\n    }\n    tokens\n}\n\nfn main() -> tantivy::Result<()> {\n    let index_path = TempDir::new()?;\n\n    let mut schema_builder = Schema::builder();\n\n    schema_builder.add_text_field(\"title\", TEXT | STORED);\n    schema_builder.add_text_field(\"body\", TEXT);\n\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_dir(&index_path, schema.clone())?;\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // We can create a document manually, by setting the fields\n    // one by one in a Document object.\n    let title = schema.get_field(\"title\").unwrap();\n    let body = schema.get_field(\"body\").unwrap();\n\n    let title_text = \"The Old Man and the Sea\";\n    let body_text = \"He was an old man who fished alone in a skiff in the Gulf Stream\";\n\n    // Content of our first document\n    // We create `PreTokenizedString` which contains original text and vector of tokens\n    let title_tok = PreTokenizedString {\n        text: String::from(title_text),\n        tokens: pre_tokenize_text(title_text),\n    };\n\n    println!(\n        \"Original text: \\\"{}\\\" and tokens: {:?}\",\n        title_tok.text, title_tok.tokens\n    );\n\n    let body_tok = PreTokenizedString {\n        text: String::from(body_text),\n        tokens: pre_tokenize_text(body_text),\n    };\n\n    // Now lets create a document and add our `PreTokenizedString`\n    let old_man_doc = doc!(title => title_tok, body => body_tok);\n\n    // ... now let's just add it to the IndexWriter\n    index_writer.add_document(old_man_doc)?;\n\n    // Pretokenized text can also be fed as JSON\n    let short_man_json = r#\"{\n        \"title\":[{\n            \"text\":\"The Old Man\",\n            \"tokens\":[\n                {\"offset_from\":0,\"offset_to\":3,\"position\":0,\"text\":\"The\",\"position_length\":1},\n                {\"offset_from\":4,\"offset_to\":7,\"position\":1,\"text\":\"Old\",\"position_length\":1},\n                {\"offset_from\":8,\"offset_to\":11,\"position\":2,\"text\":\"Man\",\"position_length\":1}\n            ]\n        }]\n    }\"#;\n\n    let short_man_doc = TantivyDocument::parse_json(&schema, short_man_json)?;\n\n    index_writer.add_document(short_man_doc)?;\n\n    // Let's commit changes\n    index_writer.commit()?;\n\n    // ... and now is the time to query our index\n\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::OnCommitWithDelay)\n        .try_into()?;\n\n    let searcher = reader.searcher();\n\n    // We want to get documents with token \"Man\", we will use TermQuery to do it\n    // Using PreTokenizedString means the tokens are stored as is avoiding stemming\n    // and lowercasing, which preserves full words in their original form\n    let query = TermQuery::new(\n        Term::from_field_text(title, \"Man\"),\n        IndexRecordOption::Basic,\n    );\n\n    let (top_docs, count) =\n        searcher.search(&query, &(TopDocs::with_limit(2).order_by_score(), Count))?;\n\n    assert_eq!(count, 2);\n\n    // Now let's print out the results.\n    // Note that the tokens are not stored along with the original text\n    // in the document store\n    for (_score, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"{}\", retrieved_doc.to_json(&schema));\n    }\n\n    // In contrary to the previous query, when we search for the \"man\" term we\n    // should get no results, as it's not one of the indexed tokens. SimpleTokenizer\n    // only splits text on whitespace / punctuation.\n\n    let query = TermQuery::new(\n        Term::from_field_text(title, \"man\"),\n        IndexRecordOption::Basic,\n    );\n\n    let (_top_docs, count) =\n        searcher.search(&query, &(TopDocs::with_limit(2).order_by_score(), Count))?;\n\n    assert_eq!(count, 0);\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/snippet.rs",
    "content": "// # Snippet example\n//\n// This example shows how to return a representative snippet of\n// your hit result.\n// Snippet are an extracted of a target document, and returned in HTML format.\n// The keyword searched by the user are highlighted with a `<b>` tag.\n\n// ---\n// Importing tantivy...\nuse tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::snippet::{Snippet, SnippetGenerator};\nuse tantivy::{doc, Index, IndexWriter};\nuse tempfile::TempDir;\n\nfn main() -> tantivy::Result<()> {\n    // Let's create a temporary directory for the\n    // sake of this example\n    let index_path = TempDir::new()?;\n\n    // # Defining the schema\n    let mut schema_builder = Schema::builder();\n    let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n    let body = schema_builder.add_text_field(\"body\", TEXT | STORED);\n    let schema = schema_builder.build();\n\n    // # Indexing documents\n    let index = Index::create_in_dir(&index_path, schema)?;\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    // we'll only need one doc for this example.\n    index_writer.add_document(doc!(\n    title => \"Of Mice and Men\",\n    body => \"A few miles south of Soledad, the Salinas River drops in close to the hillside \\\n            bank and runs deep and green. The water is warm too, for it has slipped twinkling \\\n            over the yellow sands in the sunlight before reaching the narrow pool. On one \\\n            side of the river the golden foothill slopes curve up to the strong and rocky \\\n            Gabilan Mountains, but on the valley side the water is lined with trees—willows \\\n            fresh and green with every spring, carrying in their lower leaf junctures the \\\n            debris of the winter’s flooding; and sycamores with mottled, white, recumbent \\\n            limbs and branches that arch over the pool\"\n    ))?;\n    // ...\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n    let query_parser = QueryParser::for_index(&index, vec![title, body]);\n    let query = query_parser.parse_query(\"sycamore spring\")?;\n\n    let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n\n    let snippet_generator = SnippetGenerator::create(&searcher, &*query, body)?;\n\n    for (score, doc_address) in top_docs {\n        let doc = searcher.doc::<TantivyDocument>(doc_address)?;\n        let snippet = snippet_generator.snippet_from_doc(&doc);\n        println!(\"Document score {score}:\");\n        println!(\"title: {}\", doc.get_first(title).unwrap().as_str().unwrap());\n        println!(\"snippet: {}\", snippet.to_html());\n        println!(\"custom highlighting: {}\", highlight(snippet));\n    }\n\n    Ok(())\n}\n\nfn highlight(snippet: Snippet) -> String {\n    let mut result = String::new();\n    let mut start_from = 0;\n\n    for fragment_range in snippet.highlighted() {\n        result.push_str(&snippet.fragment()[start_from..fragment_range.start]);\n        result.push_str(\" --> \");\n        result.push_str(&snippet.fragment()[fragment_range.clone()]);\n        result.push_str(\" <-- \");\n        start_from = fragment_range.end;\n    }\n\n    result.push_str(&snippet.fragment()[start_from..]);\n    result\n}\n"
  },
  {
    "path": "examples/stop_words.rs",
    "content": "// # Stop Words Example\n//\n// This example covers the basic usage of stop words\n// with tantivy\n//\n// We will :\n// - define our schema\n// - create an index in a directory\n// - add a few stop words\n// - index few documents in our index\n\n// ---\n// Importing tantivy...\nuse tantivy::collector::TopDocs;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::*;\nuse tantivy::tokenizer::*;\nuse tantivy::{doc, Index, IndexWriter};\n\nfn main() -> tantivy::Result<()> {\n    // this example assumes you understand the content in `basic_search`\n    let mut schema_builder = Schema::builder();\n\n    // This configures your custom options for how tantivy will\n    // store and process your content in the index; The key\n    // to note is that we are setting the tokenizer to `stoppy`\n    // which will be defined and registered below.\n    let text_field_indexing = TextFieldIndexing::default()\n        .set_tokenizer(\"stoppy\")\n        .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n    let text_options = TextOptions::default()\n        .set_indexing_options(text_field_indexing)\n        .set_stored();\n\n    // Our first field is title.\n    schema_builder.add_text_field(\"title\", text_options);\n\n    // Our second field is body.\n    let text_field_indexing = TextFieldIndexing::default()\n        .set_tokenizer(\"stoppy\")\n        .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n    let text_options = TextOptions::default()\n        .set_indexing_options(text_field_indexing)\n        .set_stored();\n    schema_builder.add_text_field(\"body\", text_options);\n\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema.clone());\n\n    // This tokenizer lowers all of the text (to help with stop word matching)\n    // then removes all instances of `the` and `and` from the corpus\n    let tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n        .filter(LowerCaser)\n        .filter(StopWordFilter::remove(vec![\n            \"the\".to_string(),\n            \"and\".to_string(),\n        ]))\n        .build();\n\n    index.tokenizers().register(\"stoppy\", tokenizer);\n\n    let mut index_writer: IndexWriter = index.writer(50_000_000)?;\n\n    let title = schema.get_field(\"title\").unwrap();\n    let body = schema.get_field(\"body\").unwrap();\n\n    index_writer.add_document(doc!(\n    title => \"The Old Man and the Sea\",\n    body => \"He was an old man who fished alone in a skiff in the Gulf Stream and \\\n     he had gone eighty-four days now without taking a fish.\"\n    ))?;\n\n    index_writer.add_document(doc!(\n    title => \"Of Mice and Men\",\n    body => \"A few miles south of Soledad, the Salinas River drops in close to the hillside \\\n            bank and runs deep and green. The water is warm too, for it has slipped twinkling \\\n            over the yellow sands in the sunlight before reaching the narrow pool. On one \\\n            side of the river the golden foothill slopes curve up to the strong and rocky \\\n            Gabilan Mountains, but on the valley side the water is lined with trees—willows \\\n            fresh and green with every spring, carrying in their lower leaf junctures the \\\n            debris of the winter’s flooding; and sycamores with mottled, white, recumbent \\\n            limbs and branches that arch over the pool\"\n    ))?;\n\n    index_writer.add_document(doc!(\n    title => \"Frankenstein\",\n    body => \"You will rejoice to hear that no disaster has accompanied the commencement of an \\\n             enterprise which you have regarded with such evil forebodings.  I arrived here \\\n             yesterday, and my first task is to assure my dear sister of my welfare and \\\n             increasing confidence in the success of my undertaking.\"\n    ))?;\n\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n\n    let searcher = reader.searcher();\n\n    let query_parser = QueryParser::for_index(&index, vec![title, body]);\n\n    // stop words are applied on the query as well.\n    // The following will be equivalent to `title:frankenstein`\n    let query = query_parser.parse_query(\"title:\\\"the Frankenstein\\\"\")?;\n    let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n\n    for (score, doc_address) in top_docs {\n        let retrieved_doc: TantivyDocument = searcher.doc(doc_address)?;\n        println!(\"\\n==\\nDocument score {score}:\");\n        println!(\"{}\", retrieved_doc.to_json(&schema));\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "examples/warmer.rs",
    "content": "use std::cmp::Reverse;\nuse std::collections::{HashMap, HashSet};\nuse std::sync::{Arc, RwLock, Weak};\n\nuse tantivy::collector::TopDocs;\nuse tantivy::index::SegmentId;\nuse tantivy::query::QueryParser;\nuse tantivy::schema::{Schema, FAST, TEXT};\nuse tantivy::{\n    doc, DocAddress, DocId, Index, IndexWriter, Opstamp, Searcher, SearcherGeneration,\n    SegmentReader, Warmer,\n};\n\n// This example shows how warmers can be used to\n// load values from an external sources and\n// tie their lifecycle to that of the index segments\n// using the Warmer API.\n//\n// In this example, we assume an e-commerce search engine.\n\ntype ProductId = u64;\n\ntype Price = u32;\n\npub trait PriceFetcher: Send + Sync + 'static {\n    fn fetch_prices(&self, product_ids: &[ProductId]) -> Vec<Price>;\n}\n\ntype SegmentKey = (SegmentId, Option<Opstamp>);\n\nstruct DynamicPriceColumn {\n    field: String,\n    price_cache: RwLock<HashMap<SegmentKey, Arc<Vec<Price>>>>,\n    price_fetcher: Box<dyn PriceFetcher>,\n}\n\nimpl DynamicPriceColumn {\n    pub fn with_product_id_field<T: PriceFetcher>(field: String, price_fetcher: T) -> Self {\n        DynamicPriceColumn {\n            field,\n            price_cache: Default::default(),\n            price_fetcher: Box::new(price_fetcher),\n        }\n    }\n\n    pub fn price_for_segment(&self, segment_reader: &SegmentReader) -> Option<Arc<Vec<Price>>> {\n        let segment_key = (segment_reader.segment_id(), segment_reader.delete_opstamp());\n        self.price_cache.read().unwrap().get(&segment_key).cloned()\n    }\n}\nimpl Warmer for DynamicPriceColumn {\n    fn warm(&self, searcher: &Searcher) -> tantivy::Result<()> {\n        for segment in searcher.segment_readers() {\n            let product_id_reader = segment\n                .fast_fields()\n                .u64(&self.field)?\n                .first_or_default_col(0);\n            let product_ids: Vec<ProductId> = segment\n                .doc_ids_alive()\n                .map(|doc| product_id_reader.get_val(doc))\n                .collect();\n\n            let mut prices = self.price_fetcher.fetch_prices(&product_ids).into_iter();\n\n            let prices: Vec<Price> = (0..segment.max_doc())\n                .map(|doc| {\n                    if !segment.is_deleted(doc) {\n                        prices.next().unwrap()\n                    } else {\n                        0\n                    }\n                })\n                .collect();\n\n            let key = (segment.segment_id(), segment.delete_opstamp());\n            self.price_cache\n                .write()\n                .unwrap()\n                .insert(key, Arc::new(prices));\n        }\n\n        Ok(())\n    }\n\n    fn garbage_collect(&self, live_generations: &[&SearcherGeneration]) {\n        let live_keys: HashSet<SegmentKey> = live_generations\n            .iter()\n            .flat_map(|gen| gen.segments())\n            .map(|(&segment_id, &opstamp)| (segment_id, opstamp))\n            .collect();\n\n        self.price_cache\n            .write()\n            .unwrap()\n            .retain(|key, _| live_keys.contains(key));\n    }\n}\n\n// For the sake of this example, the table is just an editable HashMap behind a RwLock.\n// This map represents a map (ProductId -> Price)\n//\n// In practise, it could be fetching things from an external service, like a SQL table.\n#[derive(Default, Clone)]\npub struct ExternalPriceTable {\n    prices: Arc<RwLock<HashMap<ProductId, Price>>>,\n}\n\nimpl ExternalPriceTable {\n    pub fn update_price(&self, product_id: ProductId, price: Price) {\n        self.prices.write().unwrap().insert(product_id, price);\n    }\n}\n\nimpl PriceFetcher for ExternalPriceTable {\n    fn fetch_prices(&self, product_ids: &[ProductId]) -> Vec<Price> {\n        let prices = self.prices.read().unwrap();\n\n        product_ids\n            .iter()\n            .map(|product_id| prices.get(product_id).cloned().unwrap_or(0))\n            .collect()\n    }\n}\n\nfn main() -> tantivy::Result<()> {\n    // Declaring our schema.\n    let mut schema_builder = Schema::builder();\n    // The product id is assumed to be a primary id for our external price source.\n    let product_id = schema_builder.add_u64_field(\"product_id\", FAST);\n    let text = schema_builder.add_text_field(\"text\", TEXT);\n    let schema: Schema = schema_builder.build();\n\n    let price_table = ExternalPriceTable::default();\n    let price_dynamic_column = Arc::new(DynamicPriceColumn::with_product_id_field(\n        \"product_id\".to_string(),\n        price_table.clone(),\n    ));\n    price_table.update_price(OLIVE_OIL, 12);\n    price_table.update_price(GLOVES, 13);\n    price_table.update_price(SNEAKERS, 80);\n\n    const OLIVE_OIL: ProductId = 323423;\n    const GLOVES: ProductId = 3966623;\n    const SNEAKERS: ProductId = 23222;\n\n    let index = Index::create_in_ram(schema);\n    let mut writer: IndexWriter = index.writer_with_num_threads(1, 15_000_000)?;\n    writer.add_document(doc!(product_id=>OLIVE_OIL, text=>\"cooking olive oil from greece\"))?;\n    writer.add_document(doc!(product_id=>GLOVES, text=>\"kitchen gloves, perfect for cooking\"))?;\n    writer.add_document(doc!(product_id=>SNEAKERS, text=>\"uber sweet sneakers\"))?;\n    writer.commit()?;\n\n    let warmers = vec![Arc::downgrade(&price_dynamic_column) as Weak<dyn Warmer>];\n    let reader = index.reader_builder().warmers(warmers).try_into()?;\n\n    let query_parser = QueryParser::for_index(&index, vec![text]);\n    let query = query_parser.parse_query(\"cooking\")?;\n\n    let searcher = reader.searcher();\n    let score_by_price = move |segment_reader: &SegmentReader| {\n        let price = price_dynamic_column\n            .price_for_segment(segment_reader)\n            .unwrap();\n        move |doc_id: DocId| Reverse(price[doc_id as usize])\n    };\n\n    let most_expensive_first = TopDocs::with_limit(10).order_by(score_by_price);\n\n    let hits = searcher.search(&query, &most_expensive_first)?;\n    assert_eq!(\n        &hits,\n        &[\n            (\n                Reverse(12u32),\n                DocAddress {\n                    segment_ord: 0,\n                    doc_id: 0u32\n                }\n            ),\n            (\n                Reverse(13u32),\n                DocAddress {\n                    segment_ord: 0,\n                    doc_id: 1u32\n                }\n            ),\n        ]\n    );\n\n    // Olive oil just got more expensive!\n    price_table.update_price(OLIVE_OIL, 15);\n\n    // The price update are directly reflected on `reload`.\n    //\n    // Be careful here though!...\n    // You may have spotted that we are still using the same `Searcher`.\n    //\n    // It is up to the `Warmer` implementer to decide how\n    // to control this behavior.\n\n    reader.reload()?;\n\n    let hits_with_new_prices = searcher.search(&query, &most_expensive_first)?;\n    assert_eq!(\n        &hits_with_new_prices,\n        &[\n            (\n                Reverse(13u32),\n                DocAddress {\n                    segment_ord: 0,\n                    doc_id: 1u32\n                }\n            ),\n            (\n                Reverse(15u32),\n                DocAddress {\n                    segment_ord: 0,\n                    doc_id: 0u32\n                }\n            ),\n        ]\n    );\n\n    Ok(())\n}\n"
  },
  {
    "path": "ownedbytes/Cargo.toml",
    "content": "[package]\nauthors = [\"Paul Masurel <paul@quickwit.io>\", \"Pascal Seitz <pascal@quickwit.io>\"]\nname = \"ownedbytes\"\nversion = \"0.9.0\"\nedition = \"2021\"\ndescription = \"Expose data as static slice\"\nlicense = \"MIT\"\ndocumentation = \"https://docs.rs/ownedbytes/\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nstable_deref_trait = \"1.2.0\"\n"
  },
  {
    "path": "ownedbytes/src/lib.rs",
    "content": "use std::ops::{Deref, Range};\nuse std::sync::Arc;\nuse std::{fmt, io};\n\npub use stable_deref_trait::StableDeref;\n\n/// An OwnedBytes simply wraps an object that owns a slice of data and exposes\n/// this data as a slice.\n///\n/// The backing object is required to be `StableDeref`.\n#[derive(Clone)]\npub struct OwnedBytes {\n    data: &'static [u8],\n    box_stable_deref: Arc<dyn Deref<Target = [u8]> + Sync + Send>,\n}\n\nimpl OwnedBytes {\n    /// Creates an empty `OwnedBytes`.\n    pub fn empty() -> OwnedBytes {\n        OwnedBytes::new(&[][..])\n    }\n\n    /// Creates an `OwnedBytes` instance given a `StableDeref` object.\n    pub fn new<T: StableDeref + Deref<Target = [u8]> + 'static + Send + Sync>(\n        data_holder: T,\n    ) -> OwnedBytes {\n        let box_stable_deref = Arc::new(data_holder);\n        let bytes: &[u8] = box_stable_deref.deref();\n        let data = unsafe { &*(bytes as *const [u8]) };\n        OwnedBytes {\n            data,\n            box_stable_deref,\n        }\n    }\n\n    /// creates a fileslice that is just a view over a slice of the data.\n    #[must_use]\n    #[inline]\n    pub fn slice(&self, range: Range<usize>) -> Self {\n        OwnedBytes {\n            data: &self.data[range],\n            box_stable_deref: self.box_stable_deref.clone(),\n        }\n    }\n\n    /// Returns the underlying slice of data.\n    /// `Deref` and `AsRef` are also available.\n    #[inline]\n    pub fn as_slice(&self) -> &[u8] {\n        self.data\n    }\n\n    /// Returns the len of the slice.\n    #[inline]\n    pub fn len(&self) -> usize {\n        self.data.len()\n    }\n\n    /// Returns true iff this `OwnedBytes` is empty.\n    #[inline]\n    pub fn is_empty(&self) -> bool {\n        self.data.is_empty()\n    }\n\n    /// Splits the OwnedBytes into two OwnedBytes `(left, right)`.\n    ///\n    /// Left will hold `split_len` bytes.\n    ///\n    /// This operation is cheap and does not require to copy any memory.\n    /// On the other hand, both `left` and `right` retain a handle over\n    /// the entire slice of memory. In other words, the memory will only\n    /// be released when both left and right are dropped.\n    #[inline]\n    #[must_use]\n    pub fn split(self, split_len: usize) -> (OwnedBytes, OwnedBytes) {\n        let (left_data, right_data) = self.data.split_at(split_len);\n        let right_box_stable_deref = self.box_stable_deref.clone();\n        let left = OwnedBytes {\n            data: left_data,\n            box_stable_deref: self.box_stable_deref,\n        };\n        let right = OwnedBytes {\n            data: right_data,\n            box_stable_deref: right_box_stable_deref,\n        };\n        (left, right)\n    }\n\n    /// Splits the OwnedBytes into two OwnedBytes `(left, right)`.\n    ///\n    /// Right will hold `split_len` bytes.\n    ///\n    /// This operation is cheap and does not require to copy any memory.\n    /// On the other hand, both `left` and `right` retain a handle over\n    /// the entire slice of memory. In other words, the memory will only\n    /// be released when both left and right are dropped.\n    #[inline]\n    #[must_use]\n    pub fn rsplit(self, split_len: usize) -> (OwnedBytes, OwnedBytes) {\n        let data_len = self.data.len();\n        self.split(data_len - split_len)\n    }\n\n    /// Splits the right part of the `OwnedBytes` at the given offset.\n    ///\n    /// `self` is truncated to `split_len`, left with the remaining bytes.\n    pub fn split_off(&mut self, split_len: usize) -> OwnedBytes {\n        let (left, right) = self.data.split_at(split_len);\n        let right_box_stable_deref = self.box_stable_deref.clone();\n        let right_piece = OwnedBytes {\n            data: right,\n            box_stable_deref: right_box_stable_deref,\n        };\n        self.data = left;\n        right_piece\n    }\n\n    /// Drops the left most `advance_len` bytes.\n    #[inline]\n    pub fn advance(&mut self, advance_len: usize) -> &[u8] {\n        let (data, rest) = self.data.split_at(advance_len);\n        self.data = rest;\n        data\n    }\n\n    /// Reads an `u8` from the `OwnedBytes` and advance by one byte.\n    #[inline]\n    pub fn read_u8(&mut self) -> u8 {\n        self.advance(1)[0]\n    }\n\n    #[inline]\n    fn read_n<const N: usize>(&mut self) -> [u8; N] {\n        self.advance(N).try_into().unwrap()\n    }\n\n    /// Reads an `u32` encoded as little-endian from the `OwnedBytes` and advance by 4 bytes.\n    #[inline]\n    pub fn read_u32(&mut self) -> u32 {\n        u32::from_le_bytes(self.read_n())\n    }\n\n    /// Reads an `u64` encoded as little-endian from the `OwnedBytes` and advance by 8 bytes.\n    #[inline]\n    pub fn read_u64(&mut self) -> u64 {\n        u64::from_le_bytes(self.read_n())\n    }\n}\n\nimpl fmt::Debug for OwnedBytes {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        // We truncate the bytes in order to make sure the debug string\n        // is not too long.\n        let bytes_truncated: &[u8] = if self.len() > 10 {\n            &self.as_slice()[..10]\n        } else {\n            self.as_slice()\n        };\n        write!(f, \"OwnedBytes({bytes_truncated:?}, len={})\", self.len())\n    }\n}\n\nimpl PartialEq for OwnedBytes {\n    fn eq(&self, other: &OwnedBytes) -> bool {\n        self.as_slice() == other.as_slice()\n    }\n}\n\nimpl Eq for OwnedBytes {}\n\nimpl PartialEq<[u8]> for OwnedBytes {\n    fn eq(&self, other: &[u8]) -> bool {\n        self.as_slice() == other\n    }\n}\n\nimpl PartialEq<str> for OwnedBytes {\n    fn eq(&self, other: &str) -> bool {\n        self.as_slice() == other.as_bytes()\n    }\n}\n\nimpl<'a, T: ?Sized> PartialEq<&'a T> for OwnedBytes\nwhere OwnedBytes: PartialEq<T>\n{\n    fn eq(&self, other: &&'a T) -> bool {\n        *self == **other\n    }\n}\n\nimpl Deref for OwnedBytes {\n    type Target = [u8];\n\n    #[inline]\n    fn deref(&self) -> &Self::Target {\n        self.as_slice()\n    }\n}\n\nimpl AsRef<[u8]> for OwnedBytes {\n    #[inline]\n    fn as_ref(&self) -> &[u8] {\n        self.as_slice()\n    }\n}\n\nimpl io::Read for OwnedBytes {\n    #[inline]\n    fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {\n        let data_len = self.data.len();\n        let buf_len = buf.len();\n        if data_len >= buf_len {\n            let data = self.advance(buf_len);\n            buf.copy_from_slice(data);\n            Ok(buf_len)\n        } else {\n            buf[..data_len].copy_from_slice(self.data);\n            self.data = &[];\n            Ok(data_len)\n        }\n    }\n    #[inline]\n    fn read_to_end(&mut self, buf: &mut Vec<u8>) -> io::Result<usize> {\n        buf.extend(self.data);\n        let read_len = self.data.len();\n        self.data = &[];\n        Ok(read_len)\n    }\n    #[inline]\n    fn read_exact(&mut self, buf: &mut [u8]) -> io::Result<()> {\n        let read_len = self.read(buf)?;\n        if read_len != buf.len() {\n            return Err(io::Error::new(\n                io::ErrorKind::UnexpectedEof,\n                \"failed to fill whole buffer\",\n            ));\n        }\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io::{self, Read};\n\n    use super::OwnedBytes;\n\n    #[test]\n    fn test_owned_bytes_debug() {\n        let short_bytes = OwnedBytes::new(b\"abcd\".as_ref());\n        assert_eq!(\n            format!(\"{short_bytes:?}\"),\n            \"OwnedBytes([97, 98, 99, 100], len=4)\"\n        );\n        let medium_bytes = OwnedBytes::new(b\"abcdefghi\".as_ref());\n        assert_eq!(\n            format!(\"{medium_bytes:?}\"),\n            \"OwnedBytes([97, 98, 99, 100, 101, 102, 103, 104, 105], len=9)\"\n        );\n        let long_bytes = OwnedBytes::new(b\"abcdefghijklmnopq\".as_ref());\n        assert_eq!(\n            format!(\"{long_bytes:?}\"),\n            \"OwnedBytes([97, 98, 99, 100, 101, 102, 103, 104, 105, 106], len=17)\"\n        );\n    }\n\n    #[test]\n    fn test_owned_bytes_read() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"abcdefghiklmnopqrstuvwxyz\".as_ref());\n        {\n            let mut buf = [0u8; 5];\n            bytes.read_exact(&mut buf[..]).unwrap();\n            assert_eq!(&buf, b\"abcde\");\n            assert_eq!(bytes.as_slice(), b\"fghiklmnopqrstuvwxyz\")\n        }\n        {\n            let mut buf = [0u8; 2];\n            bytes.read_exact(&mut buf[..]).unwrap();\n            assert_eq!(&buf, b\"fg\");\n            assert_eq!(bytes.as_slice(), b\"hiklmnopqrstuvwxyz\")\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_owned_bytes_read_right_at_the_end() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"abcde\".as_ref());\n        let mut buf = [0u8; 5];\n        assert_eq!(bytes.read(&mut buf[..]).unwrap(), 5);\n        assert_eq!(&buf, b\"abcde\");\n        assert_eq!(bytes.as_slice(), b\"\");\n        assert_eq!(bytes.read(&mut buf[..]).unwrap(), 0);\n        assert_eq!(&buf, b\"abcde\");\n        Ok(())\n    }\n    #[test]\n    fn test_owned_bytes_read_incomplete() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"abcde\".as_ref());\n        let mut buf = [0u8; 7];\n        assert_eq!(bytes.read(&mut buf[..]).unwrap(), 5);\n        assert_eq!(&buf[..5], b\"abcde\");\n        assert_eq!(bytes.read(&mut buf[..]).unwrap(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_owned_bytes_read_to_end() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"abcde\".as_ref());\n        let mut buf = Vec::new();\n        bytes.read_to_end(&mut buf)?;\n        assert_eq!(buf.as_slice(), b\"abcde\".as_ref());\n        Ok(())\n    }\n\n    #[test]\n    fn test_owned_bytes_read_u8() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"\\xFF\".as_ref());\n        assert_eq!(bytes.read_u8(), 255);\n        assert_eq!(bytes.len(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_owned_bytes_read_u64() -> io::Result<()> {\n        let mut bytes = OwnedBytes::new(b\"\\0\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\".as_ref());\n        assert_eq!(bytes.read_u64(), u64::MAX - 255);\n        assert_eq!(bytes.len(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_owned_bytes_split() {\n        let bytes = OwnedBytes::new(b\"abcdefghi\".as_ref());\n        let (left, right) = bytes.split(3);\n        assert_eq!(left.as_slice(), b\"abc\");\n        assert_eq!(right.as_slice(), b\"defghi\");\n    }\n\n    #[test]\n    fn test_owned_bytes_split_boundary() {\n        let bytes = OwnedBytes::new(b\"abcdefghi\".as_ref());\n        {\n            let (left, right) = bytes.clone().split(0);\n            assert_eq!(left.as_slice(), b\"\");\n            assert_eq!(right.as_slice(), b\"abcdefghi\");\n        }\n        {\n            let (left, right) = bytes.split(9);\n            assert_eq!(left.as_slice(), b\"abcdefghi\");\n            assert_eq!(right.as_slice(), b\"\");\n        }\n    }\n\n    #[test]\n    fn test_split_off() {\n        let mut data = OwnedBytes::new(b\"abcdef\".as_ref());\n        assert_eq!(data, \"abcdef\");\n        assert_eq!(data.split_off(2), \"cdef\");\n        assert_eq!(data, \"ab\");\n        assert_eq!(data.split_off(1), \"b\");\n        assert_eq!(data, \"a\");\n    }\n}\n"
  },
  {
    "path": "query-grammar/Cargo.toml",
    "content": "[package]\nname = \"tantivy-query-grammar\"\nversion = \"0.25.0\"\nauthors = [\"Paul Masurel <paul.masurel@gmail.com>\"]\nlicense = \"MIT\"\ncategories = [\"database-implementations\", \"data-structures\"]\ndescription = \"\"\"Search engine library\"\"\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\nreadme = \"README.md\"\nkeywords = [\"search\", \"information\", \"retrieval\"]\nedition = \"2024\"\n\n[dependencies]\nnom = \"7\"\nserde = { version = \"1.0.219\", features = [\"derive\"] }\nserde_json = \"1.0.140\"\nordered-float = \"5.0.0\"\nfnv = \"1.0.7\"\n"
  },
  {
    "path": "query-grammar/README.md",
    "content": "# Tantivy Query Grammar\n\nThis crate is used by tantivy to parse queries.\n"
  },
  {
    "path": "query-grammar/src/infallible.rs",
    "content": "//! nom combinators for infallible operations\n\nuse std::convert::Infallible;\n\nuse nom::{AsChar, IResult, InputLength, InputTakeAtPosition};\nuse serde::Serialize;\n\npub(crate) type ErrorList = Vec<LenientErrorInternal>;\npub(crate) type JResult<I, O> = IResult<I, (O, ErrorList), Infallible>;\n\n/// An error, with an end-of-string based offset\n#[derive(Debug)]\npub(crate) struct LenientErrorInternal {\n    pub pos: usize,\n    pub message: String,\n}\n\n/// A recoverable error and the position it happened at\n#[derive(Debug, PartialEq, Serialize)]\n#[serde(rename_all = \"snake_case\")]\npub struct LenientError {\n    pub pos: usize,\n    pub message: String,\n}\n\nimpl LenientError {\n    pub(crate) fn from_internal(internal: LenientErrorInternal, str_len: usize) -> LenientError {\n        LenientError {\n            pos: str_len - internal.pos,\n            message: internal.message,\n        }\n    }\n}\n\nfn unwrap_infallible<T>(res: Result<T, nom::Err<Infallible>>) -> T {\n    match res {\n        Ok(val) => val,\n        Err(_) => unreachable!(),\n    }\n}\n\n// when rfcs#1733 get stabilized, this can make things clearer\n// trait InfallibleParser<I, O> = nom::Parser<I, (O, ErrorList), std::convert::Infallible>;\n\n/// A variant of the classical `opt` parser, except it returns an infallible error type.\n///\n/// It's less generic than the original to ease type resolution in the rest of the code.\npub(crate) fn opt_i<I: Clone, O, F>(mut f: F) -> impl FnMut(I) -> JResult<I, Option<O>>\nwhere F: nom::Parser<I, O, nom::error::Error<I>> {\n    move |input: I| {\n        let i = input.clone();\n        match f.parse(input) {\n            Ok((i, o)) => Ok((i, (Some(o), Vec::new()))),\n            Err(_) => Ok((i, (None, Vec::new()))),\n        }\n    }\n}\n\npub(crate) fn opt_i_err<'a, I: Clone + InputLength, O, F>(\n    mut f: F,\n    message: impl ToString + 'a,\n) -> impl FnMut(I) -> JResult<I, Option<O>> + 'a\nwhere\n    F: nom::Parser<I, O, nom::error::Error<I>> + 'a,\n{\n    move |input: I| {\n        let i = input.clone();\n        match f.parse(input) {\n            Ok((i, o)) => Ok((i, (Some(o), Vec::new()))),\n            Err(_) => {\n                let errs = vec![LenientErrorInternal {\n                    pos: i.input_len(),\n                    message: message.to_string(),\n                }];\n                Ok((i, (None, errs)))\n            }\n        }\n    }\n}\n\npub(crate) fn space0_infallible<T>(input: T) -> JResult<T, T>\nwhere\n    T: InputTakeAtPosition + Clone,\n    <T as InputTakeAtPosition>::Item: AsChar + Clone,\n{\n    opt_i(nom::character::complete::multispace0)(input)\n        .map(|(left, (spaces, errors))| (left, (spaces.expect(\"multispace0 can't fail\"), errors)))\n}\n\npub(crate) fn space1_infallible<T>(input: T) -> JResult<T, Option<T>>\nwhere\n    T: InputTakeAtPosition + Clone + InputLength,\n    <T as InputTakeAtPosition>::Item: AsChar + Clone,\n{\n    opt_i(nom::character::complete::multispace1)(input).map(|(left, (spaces, mut errors))| {\n        if spaces.is_none() {\n            errors.push(LenientErrorInternal {\n                pos: left.input_len(),\n                message: \"missing space\".to_string(),\n            })\n        }\n        (left, (spaces, errors))\n    })\n}\n\npub(crate) fn fallible<I, O, E: nom::error::ParseError<I>, F>(\n    mut f: F,\n) -> impl FnMut(I) -> IResult<I, O, E>\nwhere F: nom::Parser<I, (O, ErrorList), Infallible> {\n    use nom::Err;\n    move |input: I| match f.parse(input) {\n        Ok((input, (output, _err))) => Ok((input, output)),\n        Err(Err::Incomplete(needed)) => Err(Err::Incomplete(needed)),\n        // old versions don't understand this is uninhabited and need the empty match to help,\n        // newer versions warn because this arm is unreachable (which it is indeed).\n        Err(Err::Error(val)) | Err(Err::Failure(val)) => match val {},\n    }\n}\n\npub(crate) fn terminated_infallible<I, O1, O2, F, G>(\n    mut first: F,\n    mut second: G,\n) -> impl FnMut(I) -> JResult<I, O1>\nwhere\n    F: nom::Parser<I, (O1, ErrorList), Infallible>,\n    G: nom::Parser<I, (O2, ErrorList), Infallible>,\n{\n    move |input: I| {\n        let (input, (o1, mut err)) = first.parse(input)?;\n        let (input, (_, mut err2)) = second.parse(input)?;\n        err.append(&mut err2);\n        Ok((input, (o1, err)))\n    }\n}\n\npub(crate) fn delimited_infallible<I, O1, O2, O3, F, G, H>(\n    mut first: F,\n    mut second: G,\n    mut third: H,\n) -> impl FnMut(I) -> JResult<I, O2>\nwhere\n    F: nom::Parser<I, (O1, ErrorList), Infallible>,\n    G: nom::Parser<I, (O2, ErrorList), Infallible>,\n    H: nom::Parser<I, (O3, ErrorList), Infallible>,\n{\n    move |input: I| {\n        let (input, (_, mut err)) = first.parse(input)?;\n        let (input, (o2, mut err2)) = second.parse(input)?;\n        err.append(&mut err2);\n        let (input, (_, mut err3)) = third.parse(input)?;\n        err.append(&mut err3);\n        Ok((input, (o2, err)))\n    }\n}\n\n// Parse nothing. Just a lazy way to not implement terminated/preceded and use delimited instead\npub(crate) fn nothing(i: &str) -> JResult<&str, ()> {\n    Ok((i, ((), Vec::new())))\n}\n\npub(crate) trait TupleInfallible<I, O> {\n    /// Parses the input and returns a tuple of results of each parser.\n    fn parse(&mut self, input: I) -> JResult<I, O>;\n}\n\nimpl<Input, Output, F: nom::Parser<Input, (Output, ErrorList), Infallible>>\n    TupleInfallible<Input, (Output,)> for (F,)\n{\n    fn parse(&mut self, input: Input) -> JResult<Input, (Output,)> {\n        self.0.parse(input).map(|(i, (o, e))| (i, ((o,), e)))\n    }\n}\n\n// these macros are heavily copied from nom, with some minor adaptations for our type\nmacro_rules! tuple_trait(\n  ($name1:ident $ty1:ident, $name2: ident $ty2:ident, $($name:ident $ty:ident),*) => (\n    tuple_trait!(__impl $name1 $ty1, $name2 $ty2; $($name $ty),*);\n  );\n  (__impl $($name:ident $ty: ident),+; $name1:ident $ty1:ident, $($name2:ident $ty2:ident),*) => (\n    tuple_trait_impl!($($name $ty),+);\n    tuple_trait!(__impl $($name $ty),+ , $name1 $ty1; $($name2 $ty2),*);\n  );\n  (__impl $($name:ident $ty: ident),+; $name1:ident $ty1:ident) => (\n    tuple_trait_impl!($($name $ty),+);\n    tuple_trait_impl!($($name $ty),+, $name1 $ty1);\n  );\n);\n\nmacro_rules! tuple_trait_impl(\n  ($($name:ident $ty: ident),+) => (\n    impl<\n      Input: Clone, $($ty),+ ,\n      $($name: nom::Parser<Input, ($ty, ErrorList), Infallible>),+\n    > TupleInfallible<Input, ( $($ty),+ )> for ( $($name),+ ) {\n\n      fn parse(&mut self, input: Input) -> JResult<Input, ( $($ty),+ )> {\n        let mut error_list = Vec::new();\n        tuple_trait_inner!(0, self, input, (), error_list, $($name)+)\n      }\n    }\n  );\n);\n\nmacro_rules! tuple_trait_inner(\n  ($it:tt, $self:expr_2021, $input:expr_2021, (), $error_list:expr_2021, $head:ident $($id:ident)+) => ({\n    let (i, (o, mut err)) = $self.$it.parse($input.clone())?;\n    $error_list.append(&mut err);\n\n    succ!($it, tuple_trait_inner!($self, i, ( o ), $error_list, $($id)+))\n  });\n  ($it:tt, $self:expr_2021, $input:expr_2021, ($($parsed:tt)*), $error_list:expr_2021, $head:ident $($id:ident)+) => ({\n    let (i, (o, mut err)) = $self.$it.parse($input.clone())?;\n    $error_list.append(&mut err);\n\n    succ!($it, tuple_trait_inner!($self, i, ($($parsed)* , o), $error_list, $($id)+))\n  });\n  ($it:tt, $self:expr_2021, $input:expr_2021, ($($parsed:tt)*), $error_list:expr_2021, $head:ident) => ({\n    let (i, (o, mut err)) = $self.$it.parse($input.clone())?;\n    $error_list.append(&mut err);\n\n    Ok((i, (($($parsed)* , o), $error_list)))\n  });\n);\n\nmacro_rules! succ (\n  (0, $submac:ident ! ($($rest:tt)*)) => ($submac!(1, $($rest)*));\n  (1, $submac:ident ! ($($rest:tt)*)) => ($submac!(2, $($rest)*));\n  (2, $submac:ident ! ($($rest:tt)*)) => ($submac!(3, $($rest)*));\n  (3, $submac:ident ! ($($rest:tt)*)) => ($submac!(4, $($rest)*));\n  (4, $submac:ident ! ($($rest:tt)*)) => ($submac!(5, $($rest)*));\n  (5, $submac:ident ! ($($rest:tt)*)) => ($submac!(6, $($rest)*));\n  (6, $submac:ident ! ($($rest:tt)*)) => ($submac!(7, $($rest)*));\n  (7, $submac:ident ! ($($rest:tt)*)) => ($submac!(8, $($rest)*));\n  (8, $submac:ident ! ($($rest:tt)*)) => ($submac!(9, $($rest)*));\n  (9, $submac:ident ! ($($rest:tt)*)) => ($submac!(10, $($rest)*));\n  (10, $submac:ident ! ($($rest:tt)*)) => ($submac!(11, $($rest)*));\n  (11, $submac:ident ! ($($rest:tt)*)) => ($submac!(12, $($rest)*));\n  (12, $submac:ident ! ($($rest:tt)*)) => ($submac!(13, $($rest)*));\n  (13, $submac:ident ! ($($rest:tt)*)) => ($submac!(14, $($rest)*));\n  (14, $submac:ident ! ($($rest:tt)*)) => ($submac!(15, $($rest)*));\n  (15, $submac:ident ! ($($rest:tt)*)) => ($submac!(16, $($rest)*));\n  (16, $submac:ident ! ($($rest:tt)*)) => ($submac!(17, $($rest)*));\n  (17, $submac:ident ! ($($rest:tt)*)) => ($submac!(18, $($rest)*));\n  (18, $submac:ident ! ($($rest:tt)*)) => ($submac!(19, $($rest)*));\n  (19, $submac:ident ! ($($rest:tt)*)) => ($submac!(20, $($rest)*));\n  (20, $submac:ident ! ($($rest:tt)*)) => ($submac!(21, $($rest)*));\n);\n\ntuple_trait!(FnA A, FnB B, FnC C, FnD D, FnE E, FnF F, FnG G, FnH H, FnI I, FnJ J, FnK K, FnL L,\n  FnM M, FnN N, FnO O, FnP P, FnQ Q, FnR R, FnS S, FnT T, FnU U);\n\n// Special case: implement `TupleInfallible` for `()`, the unit type.\n// This can come up in macros which accept a variable number of arguments.\n// Literally, `()` is an empty tuple, so it should simply parse nothing.\nimpl<I> TupleInfallible<I, ()> for () {\n    fn parse(&mut self, input: I) -> JResult<I, ()> {\n        Ok((input, ((), Vec::new())))\n    }\n}\n\npub(crate) fn tuple_infallible<I, O, List: TupleInfallible<I, O>>(\n    mut l: List,\n) -> impl FnMut(I) -> JResult<I, O> {\n    move |i: I| l.parse(i)\n}\n\npub(crate) fn separated_list_infallible<I, O, O2, F, G>(\n    mut sep: G,\n    mut f: F,\n) -> impl FnMut(I) -> JResult<I, Vec<O>>\nwhere\n    I: Clone + InputLength,\n    F: nom::Parser<I, (O, ErrorList), Infallible>,\n    G: nom::Parser<I, (O2, ErrorList), Infallible>,\n{\n    move |i: I| {\n        let mut res: Vec<O> = Vec::new();\n        let mut errors: ErrorList = Vec::new();\n\n        let (mut i, (o, mut err)) = unwrap_infallible(f.parse(i.clone()));\n        errors.append(&mut err);\n        res.push(o);\n\n        loop {\n            let (i_sep_parsed, (_, mut err_sep)) = unwrap_infallible(sep.parse(i.clone()));\n            let len_before = i_sep_parsed.input_len();\n\n            let (i_elem_parsed, (o, mut err_elem)) =\n                unwrap_infallible(f.parse(i_sep_parsed.clone()));\n\n            // infinite loop check: the parser must always consume\n            // if we consumed nothing here, don't produce an element.\n            if i_elem_parsed.input_len() == len_before {\n                return Ok((i, (res, errors)));\n            }\n            res.push(o);\n            errors.append(&mut err_sep);\n            errors.append(&mut err_elem);\n            i = i_elem_parsed;\n        }\n    }\n}\n\npub(crate) trait Alt<I, O> {\n    /// Tests each parser in the tuple and returns the result of the first one that succeeds\n    fn choice(&mut self, input: I) -> Option<JResult<I, O>>;\n}\n\nmacro_rules! alt_trait(\n  ($first_cond:ident $first:ident, $($id_cond:ident $id: ident),+) => (\n    alt_trait!(__impl $first_cond $first; $($id_cond $id),+);\n  );\n  (__impl $($current_cond:ident $current:ident),*; $head_cond:ident $head:ident, $($id_cond:ident $id:ident),+) => (\n    alt_trait_impl!($($current_cond $current),*);\n\n    alt_trait!(__impl $($current_cond $current,)* $head_cond $head; $($id_cond $id),+);\n  );\n  (__impl $($current_cond:ident $current:ident),*; $head_cond:ident $head:ident) => (\n    alt_trait_impl!($($current_cond $current),*);\n    alt_trait_impl!($($current_cond $current,)* $head_cond $head);\n  );\n);\n\nmacro_rules! alt_trait_impl(\n  ($($id_cond:ident $id:ident),+) => (\n    impl<\n      Input: Clone, Output,\n      $(\n          // () are to make things easier on me, but I'm not entirely sure whether we can do better\n          // with rule E0207\n          $id_cond: nom::Parser<Input, (), ()>,\n          $id: nom::Parser<Input, (Output, ErrorList), Infallible>\n      ),+\n    > Alt<Input, Output> for ( $(($id_cond, $id),)+ ) {\n\n      fn choice(&mut self, input: Input) -> Option<JResult<Input, Output>> {\n        match self.0.0.parse(input.clone()) {\n          Err(_) => alt_trait_inner!(1, self, input, $($id_cond $id),+),\n          Ok((input_left, _)) => Some(self.0.1.parse(input_left)),\n        }\n      }\n    }\n  );\n);\n\nmacro_rules! alt_trait_inner(\n  ($it:tt, $self:expr_2021, $input:expr_2021, $head_cond:ident $head:ident, $($id_cond:ident $id:ident),+) => (\n    match $self.$it.0.parse($input.clone()) {\n      Err(_) => succ!($it, alt_trait_inner!($self, $input, $($id_cond $id),+)),\n      Ok((input_left, _)) => Some($self.$it.1.parse(input_left)),\n    }\n  );\n  ($it:tt, $self:expr_2021, $input:expr_2021, $head_cond:ident $head:ident) => (\n    None\n  );\n);\n\nalt_trait!(A1 A, B1 B, C1 C, D1 D, E1 E, F1 F, G1 G, H1 H, I1 I, J1 J, K1 K,\n           L1 L, M1 M, N1 N, O1 O, P1 P, Q1 Q, R1 R, S1 S, T1 T, U1 U);\n\n/// An alt() like combinator. For each branch, it first tries a fallible parser, which commits to\n/// this branch, or tells to check next branch, and the execute the infallible parser which follow.\n///\n/// In case no branch match, the default (fallible) parser is executed.\npub(crate) fn alt_infallible<I: Clone, O, F, List: Alt<I, O>>(\n    mut l: List,\n    mut default: F,\n) -> impl FnMut(I) -> JResult<I, O>\nwhere\n    F: nom::Parser<I, (O, ErrorList), Infallible>,\n{\n    move |i: I| l.choice(i.clone()).unwrap_or_else(|| default.parse(i))\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_lenient_error_serialization() {\n        let error = LenientError {\n            pos: 42,\n            message: \"test error message\".to_string(),\n        };\n\n        assert_eq!(\n            serde_json::to_string(&error).unwrap(),\n            \"{\\\"pos\\\":42,\\\"message\\\":\\\"test error message\\\"}\"\n        );\n    }\n}\n"
  },
  {
    "path": "query-grammar/src/lib.rs",
    "content": "#![allow(clippy::derive_partial_eq_without_eq)]\n\nuse serde::Serialize;\n\nmod infallible;\nmod occur;\nmod query_grammar;\nmod user_input_ast;\n\npub use crate::infallible::LenientError;\npub use crate::occur::Occur;\nuse crate::query_grammar::{parse_to_ast, parse_to_ast_lenient};\npub use crate::user_input_ast::{\n    Delimiter, UserInputAst, UserInputBound, UserInputLeaf, UserInputLiteral,\n};\n\n#[derive(Debug, Serialize)]\n#[serde(rename_all = \"snake_case\")]\npub struct Error;\n\n/// Parse a query\npub fn parse_query(query: &str) -> Result<UserInputAst, Error> {\n    let (_remaining, user_input_ast) = parse_to_ast(query).map_err(|_| Error)?;\n    Ok(user_input_ast)\n}\n\n/// Parse a query, trying to recover from syntax errors, and giving hints toward fixing errors.\npub fn parse_query_lenient(query: &str) -> (UserInputAst, Vec<LenientError>) {\n    parse_to_ast_lenient(query)\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::{UserInputAst, parse_query, parse_query_lenient};\n\n    #[test]\n    fn test_deduplication() {\n        let ast: UserInputAst = parse_query(\"a a\").unwrap();\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"bool\",\"clauses\":[[null,{\"type\":\"literal\",\"field_name\":null,\"phrase\":\"a\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}]]}\"#\n        );\n    }\n\n    #[test]\n    fn test_parse_query_serialization() {\n        let ast = parse_query(\"title:hello OR title:x\").unwrap();\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"bool\",\"clauses\":[[\"should\",{\"type\":\"literal\",\"field_name\":\"title\",\"phrase\":\"hello\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}],[\"should\",{\"type\":\"literal\",\"field_name\":\"title\",\"phrase\":\"x\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}]]}\"#\n        );\n    }\n\n    #[test]\n    fn test_parse_query_wrong_query() {\n        assert!(parse_query(\"title:\").is_err());\n    }\n\n    #[test]\n    fn test_parse_query_lenient_wrong_query() {\n        let (_, errors) = parse_query_lenient(\"title:\");\n        assert!(errors.len() == 1);\n        let json = serde_json::to_string(&errors).unwrap();\n        assert_eq!(json, r#\"[{\"pos\":6,\"message\":\"expected word\"}]\"#);\n    }\n}\n"
  },
  {
    "path": "query-grammar/src/occur.rs",
    "content": "use std::fmt;\nuse std::fmt::Write;\n\nuse serde::Serialize;\n\n/// Defines whether a term in a query must be present,\n/// should be present or must not be present.\n#[derive(Debug, Clone, Hash, Copy, Eq, PartialEq, Serialize)]\n#[serde(rename_all = \"snake_case\")]\npub enum Occur {\n    /// For a given document to be considered for scoring,\n    /// at least one of the queries with the Should or the Must\n    /// Occur constraint must be within the document.\n    Should,\n    /// Document without the queries are excluded from the search.\n    Must,\n    /// Document that contain the query are excluded from the\n    /// search.\n    MustNot,\n}\n\nimpl Occur {\n    /// Returns the one-char prefix symbol for this `Occur`.\n    /// - `Should` => '?',\n    /// - `Must` => '+'\n    /// - `Not` => '-'\n    fn to_char(self) -> char {\n        match self {\n            Occur::Should => '?',\n            Occur::Must => '+',\n            Occur::MustNot => '-',\n        }\n    }\n\n    /// Compose two occur values.\n    pub fn compose(left: Occur, right: Occur) -> Occur {\n        match (left, right) {\n            (Occur::Should, _) => right,\n            (Occur::Must, Occur::MustNot) => Occur::MustNot,\n            (Occur::Must, _) => Occur::Must,\n            (Occur::MustNot, Occur::MustNot) => Occur::Must,\n            (Occur::MustNot, _) => Occur::MustNot,\n        }\n    }\n}\n\nimpl fmt::Display for Occur {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        f.write_char(self.to_char())\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use crate::Occur;\n\n    #[test]\n    fn test_occur_compose() {\n        assert_eq!(Occur::compose(Occur::Should, Occur::Should), Occur::Should);\n        assert_eq!(Occur::compose(Occur::Should, Occur::Must), Occur::Must);\n        assert_eq!(\n            Occur::compose(Occur::Should, Occur::MustNot),\n            Occur::MustNot\n        );\n        assert_eq!(Occur::compose(Occur::Must, Occur::Should), Occur::Must);\n        assert_eq!(Occur::compose(Occur::Must, Occur::Must), Occur::Must);\n        assert_eq!(Occur::compose(Occur::Must, Occur::MustNot), Occur::MustNot);\n        assert_eq!(\n            Occur::compose(Occur::MustNot, Occur::Should),\n            Occur::MustNot\n        );\n        assert_eq!(Occur::compose(Occur::MustNot, Occur::Must), Occur::MustNot);\n        assert_eq!(Occur::compose(Occur::MustNot, Occur::MustNot), Occur::Must);\n    }\n}\n"
  },
  {
    "path": "query-grammar/src/query_grammar.rs",
    "content": "use std::borrow::Cow;\nuse std::iter::once;\n\nuse fnv::FnvHashSet;\nuse nom::IResult;\nuse nom::branch::alt;\nuse nom::bytes::complete::tag;\nuse nom::character::complete::{\n    anychar, char, digit1, multispace0, multispace1, none_of, one_of, satisfy, u32,\n};\nuse nom::combinator::{eof, map, map_res, opt, peek, recognize, value, verify};\nuse nom::error::{Error, ErrorKind};\nuse nom::multi::{many0, many1, separated_list0};\nuse nom::sequence::{delimited, preceded, separated_pair, terminated, tuple};\n\nuse super::user_input_ast::{UserInputAst, UserInputBound, UserInputLeaf, UserInputLiteral};\nuse crate::Occur;\nuse crate::infallible::*;\nuse crate::user_input_ast::Delimiter;\n\n// Note: '-' char is only forbidden at the beginning of a field name, would be clearer to add it to\n// special characters.\nconst SPECIAL_CHARS: &[char] = &[\n    '+', '^', '`', ':', '{', '}', '\"', '\\'', '[', ']', '(', ')', '!', '\\\\', '*', ' ',\n];\n\n/// consume a field name followed by colon. Return the field name with escape sequence\n/// already interpreted\nfn field_name(inp: &str) -> IResult<&str, String> {\n    let simple_char = none_of(SPECIAL_CHARS);\n    let first_char = verify(none_of(SPECIAL_CHARS), |c| *c != '-');\n    let escape_sequence = || preceded(char('\\\\'), one_of(SPECIAL_CHARS));\n\n    map(\n        terminated(\n            tuple((\n                alt((first_char, escape_sequence())),\n                many0(alt((simple_char, escape_sequence(), char('\\\\')))),\n            )),\n            tuple((multispace0, char(':'), multispace0)),\n        ),\n        |(first_char, next)| once(first_char).chain(next).collect(),\n    )(inp)\n}\n\nconst ESCAPE_IN_WORD: &[char] = &['^', '`', ':', '{', '}', '\"', '\\'', '[', ']', '(', ')', '\\\\'];\n\nfn interpret_escape(source: &str) -> String {\n    let mut res = String::with_capacity(source.len());\n    let mut in_escape = false;\n    let require_escape = |c: char| c.is_whitespace() || ESCAPE_IN_WORD.contains(&c) || c == '-';\n\n    for c in source.chars() {\n        if in_escape {\n            if !require_escape(c) {\n                // we re-add the escape sequence\n                res.push('\\\\');\n            }\n            res.push(c);\n            in_escape = false;\n        } else if c == '\\\\' {\n            in_escape = true;\n        } else {\n            res.push(c);\n        }\n    }\n    res\n}\n\n/// Consume a word outside of any context.\n// TODO should support escape sequences\nfn word(inp: &str) -> IResult<&str, Cow<'_, str>> {\n    map_res(\n        recognize(tuple((\n            alt((\n                preceded(char('\\\\'), anychar),\n                satisfy(|c| !c.is_whitespace() && !ESCAPE_IN_WORD.contains(&c) && c != '-'),\n            )),\n            many0(alt((\n                preceded(char('\\\\'), anychar),\n                satisfy(|c: char| !c.is_whitespace() && !ESCAPE_IN_WORD.contains(&c)),\n            ))),\n        ))),\n        |s| match s {\n            \"OR\" | \"AND\" | \"NOT\" | \"IN\" => Err(Error::new(inp, ErrorKind::Tag)),\n            s if s.contains('\\\\') => Ok(Cow::Owned(interpret_escape(s))),\n            s => Ok(Cow::Borrowed(s)),\n        },\n    )(inp)\n}\n\nfn word_infallible(\n    delimiter: &str,\n    emit_error: bool,\n) -> impl Fn(&str) -> JResult<&str, Option<Cow<str>>> + '_ {\n    // emit error is set when receiving an unescaped `:` should emit an error\n\n    move |inp| {\n        map(\n            opt_i_err(\n                preceded(\n                    multispace0,\n                    recognize(many1(alt((\n                        preceded(char::<&str, _>('\\\\'), anychar),\n                        satisfy(|c| !c.is_whitespace() && !delimiter.contains(c)),\n                    )))),\n                ),\n                \"expected word\",\n            ),\n            |(opt_s, mut errors)| match opt_s {\n                Some(s) => {\n                    if emit_error\n                        && (s\n                            .as_bytes()\n                            .windows(2)\n                            .any(|window| window[0] != b'\\\\' && window[1] == b':')\n                            || s.starts_with(':'))\n                    {\n                        errors.push(LenientErrorInternal {\n                            pos: inp.len(),\n                            message: \"parsed possible invalid field as term\".to_string(),\n                        });\n                    }\n                    if s.contains('\\\\') {\n                        (Some(Cow::Owned(interpret_escape(s))), errors)\n                    } else {\n                        (Some(Cow::Borrowed(s)), errors)\n                    }\n                }\n                None => (None, errors),\n            },\n        )(inp)\n    }\n}\n\n/// Consume a word inside a Range context. More values are allowed as they are\n/// not ambiguous in this context.\nfn relaxed_word(inp: &str) -> IResult<&str, &str> {\n    recognize(tuple((\n        satisfy(|c| !c.is_whitespace() && !['`', '{', '}', '\"', '[', ']', '(', ')'].contains(&c)),\n        many0(satisfy(|c: char| {\n            !c.is_whitespace() && !['{', '}', '\"', '[', ']', '(', ')'].contains(&c)\n        })),\n    )))(inp)\n}\n\nfn negative_number(inp: &str) -> IResult<&str, &str> {\n    recognize(preceded(\n        char('-'),\n        tuple((digit1, opt(tuple((char('.'), digit1))))),\n    ))(inp)\n}\n\nfn simple_term(inp: &str) -> IResult<&str, (Delimiter, String)> {\n    let escaped_string = |delimiter| {\n        // we need this because none_of can't accept an owned array of char.\n        let not_delimiter = verify(anychar, move |parsed| *parsed != delimiter);\n        map(\n            delimited(\n                char(delimiter),\n                many0(alt((preceded(char('\\\\'), anychar), not_delimiter))),\n                char(delimiter),\n            ),\n            |res| res.into_iter().collect::<String>(),\n        )\n    };\n\n    let negative_number = map(negative_number, |number| {\n        (Delimiter::None, number.to_string())\n    });\n    let double_quotes = map(escaped_string('\"'), |phrase| {\n        (Delimiter::DoubleQuotes, phrase)\n    });\n    let simple_quotes = map(escaped_string('\\''), |phrase| {\n        (Delimiter::SingleQuotes, phrase)\n    });\n    let text_no_delimiter = map(word, |text| (Delimiter::None, text.to_string()));\n\n    alt((\n        negative_number,\n        simple_quotes,\n        double_quotes,\n        text_no_delimiter,\n    ))(inp)\n}\n\nfn simple_term_infallible(\n    delimiter: &str,\n) -> impl Fn(&str) -> JResult<&str, Option<(Delimiter, String)>> + '_ {\n    |inp| {\n        let escaped_string = |delimiter| {\n            // we need this because none_of can't accept an owned array of char.\n            let not_delimiter = verify(anychar, move |parsed| *parsed != delimiter);\n            map(\n                delimited_infallible(\n                    nothing,\n                    opt_i(many0(alt((preceded(char('\\\\'), anychar), not_delimiter)))),\n                    opt_i_err(char(delimiter), format!(\"missing delimiter \\\\{delimiter}\")),\n                ),\n                |(res, err)| {\n                    // many0 can't fail\n                    (res.unwrap().into_iter().collect::<String>(), err)\n                },\n            )\n        };\n\n        let double_quotes = map(escaped_string('\"'), |(phrase, errors)| {\n            (Some((Delimiter::DoubleQuotes, phrase)), errors)\n        });\n        let simple_quotes = map(escaped_string('\\''), |(phrase, errors)| {\n            (Some((Delimiter::SingleQuotes, phrase)), errors)\n        });\n\n        alt_infallible(\n            (\n                (value((), char('\"')), double_quotes),\n                (value((), char('\\'')), simple_quotes),\n            ),\n            // numbers are parsed with words in this case, as we allow string starting with a -\n            map(word_infallible(delimiter, true), |(text, errors)| {\n                (text.map(|text| (Delimiter::None, text.to_string())), errors)\n            }),\n        )(inp)\n    }\n}\n\nfn term_or_phrase(inp: &str) -> IResult<&str, UserInputLeaf> {\n    map(\n        tuple((simple_term, fallible(slop_or_prefix_val))),\n        |((delimiter, phrase), (slop, prefix))| {\n            UserInputLiteral {\n                field_name: None,\n                phrase,\n                delimiter,\n                slop,\n                prefix,\n            }\n            .into()\n        },\n    )(inp)\n}\n\nfn term_or_phrase_infallible(inp: &str) -> JResult<&str, Option<UserInputLeaf>> {\n    map(\n        // ~* for slop/prefix, ) inside group or ast tree, ^ if boost\n        tuple_infallible((simple_term_infallible(\")^\"), slop_or_prefix_val)),\n        |((delimiter_phrase, (slop, prefix)), errors)| {\n            let leaf = if let Some((delimiter, phrase)) = delimiter_phrase {\n                Some(\n                    UserInputLiteral {\n                        field_name: None,\n                        phrase,\n                        delimiter,\n                        slop,\n                        prefix,\n                    }\n                    .into(),\n                )\n            } else if slop != 0 {\n                Some(\n                    UserInputLiteral {\n                        field_name: None,\n                        phrase: \"\".to_string(),\n                        delimiter: Delimiter::None,\n                        slop,\n                        prefix,\n                    }\n                    .into(),\n                )\n            } else {\n                None\n            };\n            (leaf, errors)\n        },\n    )(inp)\n}\n\nfn term_group(inp: &str) -> IResult<&str, UserInputAst> {\n    map(\n        tuple((\n            terminated(field_name, multispace0),\n            delimited(tuple((char('('), multispace0)), ast, char(')')),\n        )),\n        |(field_name, mut ast)| {\n            ast.set_default_field(field_name);\n            ast\n        },\n    )(inp)\n}\n\n// this is a precondition for term_group_infallible. Without it, term_group_infallible can fail\n// with a panic. It does not consume its input.\nfn term_group_precond(inp: &str) -> IResult<&str, (), ()> {\n    value(\n        (),\n        peek(tuple((\n            field_name,\n            multispace0,\n            char('('), // when we are here, we know it can't be anything but a term group\n        ))),\n    )(inp)\n    .map_err(|e| e.map(|_| ()))\n}\n\nfn term_group_infallible(inp: &str) -> JResult<&str, UserInputAst> {\n    let (inp, (field_name, _, _, _)) =\n        tuple((field_name, multispace0, char('('), multispace0))(inp).expect(\"precondition failed\");\n\n    delimited_infallible(\n        nothing,\n        map(ast_infallible, |(mut ast, errors)| {\n            ast.set_default_field(field_name.to_string());\n            (ast, errors)\n        }),\n        opt_i_err(char(')'), \"expected ')'\"),\n    )(inp)\n}\n\nfn exists(inp: &str) -> IResult<&str, UserInputLeaf> {\n    value(\n        UserInputLeaf::Exists {\n            field: String::new(),\n        },\n        tuple((\n            multispace0,\n            char('*'),\n            peek(alt((\n                value(\n                    \"\",\n                    satisfy(|c: char| c.is_whitespace() || ESCAPE_IN_WORD.contains(&c)),\n                ),\n                eof,\n            ))),\n        )),\n    )(inp)\n}\n\nfn exists_precond(inp: &str) -> IResult<&str, (), ()> {\n    value(\n        (),\n        peek(tuple((\n            field_name,\n            multispace0,\n            char('*'),\n            peek(alt((\n                value(\n                    \"\",\n                    satisfy(|c: char| c.is_whitespace() || ESCAPE_IN_WORD.contains(&c)),\n                ),\n                eof,\n            ))), // we need to check this isn't a wildcard query\n        ))),\n    )(inp)\n    .map_err(|e| e.map(|_| ()))\n}\n\nfn exists_infallible(inp: &str) -> JResult<&str, UserInputAst> {\n    let (inp, (field_name, _, _)) =\n        tuple((field_name, multispace0, char('*')))(inp).expect(\"precondition failed\");\n\n    let exists = UserInputLeaf::Exists { field: field_name }.into();\n    Ok((inp, (exists, Vec::new())))\n}\n\nfn literal(inp: &str) -> IResult<&str, UserInputAst> {\n    // * alone is already parsed by our caller, so if `exists` succeed, we can be confident\n    // something (a field name) got parsed before\n    alt((\n        map(\n            tuple((\n                opt(field_name),\n                alt((range, set, exists, regex, term_or_phrase)),\n            )),\n            |(field_name, leaf): (Option<String>, UserInputLeaf)| leaf.set_field(field_name).into(),\n        ),\n        term_group,\n    ))(inp)\n}\n\nfn literal_no_group_infallible(inp: &str) -> JResult<&str, Option<UserInputAst>> {\n    map(\n        tuple_infallible((\n            opt_i(field_name),\n            space0_infallible,\n            alt_infallible(\n                (\n                    (\n                        value((), tuple((tag(\"IN\"), multispace0, char('[')))),\n                        map(set_infallible, |(set, errs)| (Some(set), errs)),\n                    ),\n                    (\n                        value((), peek(one_of(\"{[><\"))),\n                        map(range_infallible, |(range, errs)| (Some(range), errs)),\n                    ),\n                    (\n                        value((), peek(one_of(\"/\"))),\n                        map(regex_infallible, |(regex, errs)| (Some(regex), errs)),\n                    ),\n                ),\n                delimited_infallible(space0_infallible, term_or_phrase_infallible, nothing),\n            ),\n        )),\n        |((field_name, _, leaf), mut errors)| {\n            (\n                leaf.map(|leaf| {\n                    if matches!(&leaf, UserInputLeaf::Literal(literal)\n                            if literal.phrase == \"NOT\" && literal.delimiter == Delimiter::None)\n                        && field_name.is_none()\n                    {\n                        errors.push(LenientErrorInternal {\n                            pos: inp.len(),\n                            message: \"parsed keyword NOT as term. It should be quoted\".to_string(),\n                        });\n                    }\n                    leaf.set_field(field_name).into()\n                }),\n                errors,\n            )\n        },\n    )(inp)\n}\n\nfn literal_infallible(inp: &str) -> JResult<&str, Option<UserInputAst>> {\n    alt_infallible(\n        (\n            (\n                term_group_precond,\n                map(term_group_infallible, |(group, errs)| (Some(group), errs)),\n            ),\n            (\n                exists_precond,\n                map(exists_infallible, |(exists, errs)| (Some(exists), errs)),\n            ),\n        ),\n        literal_no_group_infallible,\n    )(inp)\n}\n\nfn slop_or_prefix_val(inp: &str) -> JResult<&str, (u32, bool)> {\n    map(\n        opt_i(alt((\n            value((0, true), char('*')),\n            map(preceded(char('~'), u32), |slop| (slop, false)),\n        ))),\n        |(slop_or_prefix_opt, err)| (slop_or_prefix_opt.unwrap_or_default(), err),\n    )(inp)\n}\n\n/// Function that parses a range out of a Stream\n/// Supports ranges like:\n/// [5 TO 10], {5 TO 10}, [* TO 10], [10 TO *], {10 TO *], >5, <=10\n/// [a TO *], [a TO c], [abc TO bcd}\nfn range(inp: &str) -> IResult<&str, UserInputLeaf> {\n    let range_term_val = || {\n        map(\n            alt((negative_number, relaxed_word, tag(\"*\"))),\n            ToString::to_string,\n        )\n    };\n\n    // check for unbounded range in the form of <5, <=10, >5, >=5\n    let elastic_unbounded_range = map(\n        tuple((\n            preceded(multispace0, alt((tag(\">=\"), tag(\"<=\"), tag(\"<\"), tag(\">\")))),\n            preceded(multispace0, range_term_val()),\n        )),\n        |(comparison_sign, bound)| match comparison_sign {\n            \">=\" => (UserInputBound::Inclusive(bound), UserInputBound::Unbounded),\n            \"<=\" => (UserInputBound::Unbounded, UserInputBound::Inclusive(bound)),\n            \"<\" => (UserInputBound::Unbounded, UserInputBound::Exclusive(bound)),\n            \">\" => (UserInputBound::Exclusive(bound), UserInputBound::Unbounded),\n            // unreachable case\n            _ => (UserInputBound::Unbounded, UserInputBound::Unbounded),\n        },\n    );\n\n    let lower_bound = map(\n        separated_pair(one_of(\"{[\"), multispace0, range_term_val()),\n        |(boundary_char, lower_bound)| {\n            if lower_bound == \"*\" {\n                UserInputBound::Unbounded\n            } else if boundary_char == '{' {\n                UserInputBound::Exclusive(lower_bound)\n            } else {\n                UserInputBound::Inclusive(lower_bound)\n            }\n        },\n    );\n\n    let upper_bound = map(\n        separated_pair(range_term_val(), multispace0, one_of(\"}]\")),\n        |(upper_bound, boundary_char)| {\n            if upper_bound == \"*\" {\n                UserInputBound::Unbounded\n            } else if boundary_char == '}' {\n                UserInputBound::Exclusive(upper_bound)\n            } else {\n                UserInputBound::Inclusive(upper_bound)\n            }\n        },\n    );\n\n    let lower_to_upper = separated_pair(\n        lower_bound,\n        tuple((multispace1, tag(\"TO\"), multispace1)),\n        upper_bound,\n    );\n\n    map(\n        alt((elastic_unbounded_range, lower_to_upper)),\n        |(lower, upper)| UserInputLeaf::Range {\n            field: None,\n            lower,\n            upper,\n        },\n    )(inp)\n}\n\nfn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {\n    let lower_to_upper = map(\n        tuple_infallible((\n            opt_i(anychar),\n            space0_infallible,\n            word_infallible(\"]}\", false),\n            space1_infallible,\n            opt_i_err(\n                terminated(tag(\"TO\"), alt((value((), multispace1), value((), eof)))),\n                \"missing keyword TO\",\n            ),\n            word_infallible(\"]}\", false),\n            opt_i_err(one_of(\"]}\"), \"missing range delimiter\"),\n        )),\n        |(\n            (lower_bound_kind, _multispace0, lower, _multispace1, to, upper, upper_bound_kind),\n            errs,\n        )| {\n            let lower_bound = match (lower_bound_kind, lower.as_deref()) {\n                (_, Some(\"*\")) => UserInputBound::Unbounded,\n                (_, None) => UserInputBound::Unbounded,\n                // if it is some, TO was actually the bound (i.e. [TO TO something])\n                (_, Some(\"TO\")) if to.is_none() => UserInputBound::Unbounded,\n                (Some('['), Some(bound)) => UserInputBound::Inclusive(bound.to_string()),\n                (Some('{'), Some(bound)) => UserInputBound::Exclusive(bound.to_string()),\n                _ => unreachable!(\"precondition failed, range did not start with [ or {{\"),\n            };\n            let upper_bound = match (upper_bound_kind, upper.as_deref()) {\n                (_, Some(\"*\")) => UserInputBound::Unbounded,\n                (_, None) => UserInputBound::Unbounded,\n                (Some(']'), Some(bound)) => UserInputBound::Inclusive(bound.to_string()),\n                (Some('}'), Some(bound)) => UserInputBound::Exclusive(bound.to_string()),\n                // the end is missing, assume this is an inclusive bound\n                (_, Some(bound)) => UserInputBound::Inclusive(bound.to_string()),\n            };\n            ((lower_bound, upper_bound), errs)\n        },\n    );\n\n    map(\n        alt_infallible(\n            (\n                (\n                    value((), tag(\">=\")),\n                    map(word_infallible(\")\", false), |(bound, err)| {\n                        (\n                            (\n                                bound\n                                    .map(|bound| UserInputBound::Inclusive(bound.to_string()))\n                                    .unwrap_or(UserInputBound::Unbounded),\n                                UserInputBound::Unbounded,\n                            ),\n                            err,\n                        )\n                    }),\n                ),\n                (\n                    value((), tag(\"<=\")),\n                    map(word_infallible(\")\", false), |(bound, err)| {\n                        (\n                            (\n                                UserInputBound::Unbounded,\n                                bound\n                                    .map(|bound| UserInputBound::Inclusive(bound.to_string()))\n                                    .unwrap_or(UserInputBound::Unbounded),\n                            ),\n                            err,\n                        )\n                    }),\n                ),\n                (\n                    value((), tag(\">\")),\n                    map(word_infallible(\")\", false), |(bound, err)| {\n                        (\n                            (\n                                bound\n                                    .map(|bound| UserInputBound::Exclusive(bound.to_string()))\n                                    .unwrap_or(UserInputBound::Unbounded),\n                                UserInputBound::Unbounded,\n                            ),\n                            err,\n                        )\n                    }),\n                ),\n                (\n                    value((), tag(\"<\")),\n                    map(word_infallible(\")\", false), |(bound, err)| {\n                        (\n                            (\n                                UserInputBound::Unbounded,\n                                bound\n                                    .map(|bound| UserInputBound::Exclusive(bound.to_string()))\n                                    .unwrap_or(UserInputBound::Unbounded),\n                            ),\n                            err,\n                        )\n                    }),\n                ),\n            ),\n            lower_to_upper,\n        ),\n        |((lower, upper), errors)| {\n            (\n                UserInputLeaf::Range {\n                    field: None,\n                    lower,\n                    upper,\n                },\n                errors,\n            )\n        },\n    )(inp)\n}\n\nfn set(inp: &str) -> IResult<&str, UserInputLeaf> {\n    map(\n        preceded(\n            tuple((multispace0, tag(\"IN\"), multispace1)),\n            delimited(\n                tuple((char('['), multispace0)),\n                separated_list0(multispace1, map(simple_term, |(_, term)| term)),\n                char(']'),\n            ),\n        ),\n        |elements| UserInputLeaf::Set {\n            field: None,\n            elements,\n        },\n    )(inp)\n}\n\nfn set_infallible(mut inp: &str) -> JResult<&str, UserInputLeaf> {\n    // `IN [` has already been parsed when we enter, we only need to parse simple terms until we\n    // find a `]`\n    let mut elements = Vec::new();\n    let mut errs = Vec::new();\n    let mut first_round = true;\n    loop {\n        let mut space_error = if first_round {\n            first_round = false;\n            Vec::new()\n        } else {\n            let (rest, (_, err)) = space1_infallible(inp)?;\n            inp = rest;\n            err\n        };\n        if inp.is_empty() {\n            // TODO push error about missing ]\n            //\n            errs.push(LenientErrorInternal {\n                pos: inp.len(),\n                message: \"missing ]\".to_string(),\n            });\n            let res = UserInputLeaf::Set {\n                field: None,\n                elements,\n            };\n            return Ok((inp, (res, errs)));\n        }\n        if let Some(inp) = inp.strip_prefix(']') {\n            let res = UserInputLeaf::Set {\n                field: None,\n                elements,\n            };\n            return Ok((inp, (res, errs)));\n        }\n        errs.append(&mut space_error);\n        // TODO\n        // here we do the assumption term_or_phrase_infallible always consume something if the\n        // first byte is not `)` or ' '. If it did not, we would end up looping.\n\n        let (rest, (delim_term, mut err)) = simple_term_infallible(\"]\")(inp)?;\n        errs.append(&mut err);\n        if let Some((_, term)) = delim_term {\n            elements.push(term);\n        }\n        inp = rest;\n    }\n}\n\nfn regex(inp: &str) -> IResult<&str, UserInputLeaf> {\n    map(\n        terminated(\n            delimited(\n                char('/'),\n                many1(alt((preceded(char('\\\\'), char('/')), none_of(\"/\")))),\n                char('/'),\n            ),\n            peek(alt((\n                value((), multispace1),\n                value((), char(')')),\n                value((), eof),\n            ))),\n        ),\n        |elements| UserInputLeaf::Regex {\n            field: None,\n            pattern: elements.into_iter().collect::<String>(),\n        },\n    )(inp)\n}\n\nfn regex_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {\n    match terminated_infallible(\n        delimited_infallible(\n            opt_i_err(char('/'), \"missing delimiter /\"),\n            opt_i(many1(alt((preceded(char('\\\\'), char('/')), none_of(\"/\"))))),\n            opt_i_err(char('/'), \"missing delimiter /\"),\n        ),\n        opt_i_err(\n            peek(alt((\n                value((), multispace1),\n                value((), char(')')),\n                value((), eof),\n            ))),\n            \"expected whitespace, closing parenthesis, or end of input\",\n        ),\n    )(inp)\n    {\n        Ok((rest, (elements_part, errors))) => {\n            let pattern = match elements_part {\n                Some(elements_part) => elements_part.into_iter().collect(),\n                None => String::new(),\n            };\n            let res = UserInputLeaf::Regex {\n                field: None,\n                pattern,\n            };\n            Ok((rest, (res, errors)))\n        }\n        Err(e) => {\n            let errs = vec![LenientErrorInternal {\n                pos: inp.len(),\n                message: e.to_string(),\n            }];\n            let res = UserInputLeaf::Regex {\n                field: None,\n                pattern: String::new(),\n            };\n            Ok((inp, (res, errs)))\n        }\n    }\n}\n\nfn negate(expr: UserInputAst) -> UserInputAst {\n    expr.unary(Occur::MustNot)\n}\n\nfn leaf(inp: &str) -> IResult<&str, UserInputAst> {\n    alt((\n        delimited(char('('), ast, char(')')),\n        map(\n            terminated(\n                char('*'),\n                peek(alt((\n                    value((), multispace1),\n                    value((), char(')')),\n                    value((), eof),\n                ))),\n            ),\n            |_| UserInputAst::from(UserInputLeaf::All),\n        ),\n        map(preceded(tuple((tag(\"NOT\"), multispace1)), leaf), negate),\n        literal,\n    ))(inp)\n}\n\nfn leaf_infallible(inp: &str) -> JResult<&str, Option<UserInputAst>> {\n    alt_infallible(\n        (\n            (\n                value((), char('(')),\n                map(\n                    delimited_infallible(\n                        nothing,\n                        ast_infallible,\n                        opt_i_err(char(')'), \"expected ')'\"),\n                    ),\n                    |(ast, errs)| (Some(ast), errs),\n                ),\n            ),\n            (\n                value(\n                    (),\n                    terminated(\n                        char('*'),\n                        peek(alt((\n                            value((), multispace1),\n                            value((), char(')')),\n                            value((), eof),\n                        ))),\n                    ),\n                ),\n                map(nothing, |_| {\n                    (Some(UserInputAst::from(UserInputLeaf::All)), Vec::new())\n                }),\n            ),\n            (\n                value((), tag(\"NOT \")),\n                delimited_infallible(\n                    space0_infallible,\n                    map(leaf_infallible, |(res, err)| (res.map(negate), err)),\n                    nothing,\n                ),\n            ),\n        ),\n        literal_infallible,\n    )(inp)\n}\n\nfn positive_float_number(inp: &str) -> IResult<&str, f64> {\n    map(\n        recognize(tuple((digit1, opt(tuple((char('.'), digit1)))))),\n        // TODO this is actually dangerous if the number is actually not representable as a f64\n        // (too big for instance)\n        |float_str: &str| float_str.parse::<f64>().unwrap(),\n    )(inp)\n}\n\nfn boost(inp: &str) -> JResult<&str, Option<f64>> {\n    opt_i(preceded(char('^'), positive_float_number))(inp)\n}\n\nfn boosted_leaf(inp: &str) -> IResult<&str, UserInputAst> {\n    map(\n        tuple((leaf, fallible(boost))),\n        |(leaf, boost_opt)| match boost_opt {\n            Some(boost) if (boost - 1.0).abs() > f64::EPSILON => {\n                UserInputAst::Boost(Box::new(leaf), boost.into())\n            }\n            _ => leaf,\n        },\n    )(inp)\n}\n\nfn boosted_leaf_infallible(inp: &str) -> JResult<&str, Option<UserInputAst>> {\n    map(\n        tuple_infallible((leaf_infallible, boost)),\n        |((leaf, boost_opt), error)| match boost_opt {\n            Some(boost) if (boost - 1.0).abs() > f64::EPSILON => (\n                leaf.map(|leaf| UserInputAst::Boost(Box::new(leaf), boost.into())),\n                error,\n            ),\n            _ => (leaf, error),\n        },\n    )(inp)\n}\n\nfn occur_symbol(inp: &str) -> JResult<&str, Option<Occur>> {\n    opt_i(alt((\n        value(Occur::MustNot, char('-')),\n        value(Occur::Must, char('+')),\n    )))(inp)\n}\n\nfn occur_leaf(inp: &str) -> IResult<&str, (Option<Occur>, UserInputAst)> {\n    tuple((fallible(occur_symbol), boosted_leaf))(inp)\n}\n\n#[expect(clippy::type_complexity)]\nfn operand_occur_leaf_infallible(\n    inp: &str,\n) -> JResult<&str, (Option<BinaryOperand>, Option<Occur>, Option<UserInputAst>)> {\n    // TODO maybe this should support multiple chained AND/OR, and \"fuse\" them?\n    tuple_infallible((\n        delimited_infallible(nothing, opt_i(binary_operand), space0_infallible),\n        occur_symbol,\n        boosted_leaf_infallible,\n    ))(inp)\n}\n\n#[derive(Clone, Copy, Debug, PartialEq, Eq)]\nenum BinaryOperand {\n    Or,\n    And,\n}\n\nfn binary_operand(inp: &str) -> IResult<&str, BinaryOperand> {\n    alt((\n        value(BinaryOperand::And, tag(\"AND \")),\n        value(BinaryOperand::Or, tag(\"OR \")),\n    ))(inp)\n}\n\nfn aggregate_binary_expressions(\n    left: (Option<Occur>, UserInputAst),\n    others: Vec<(Option<BinaryOperand>, Option<Occur>, UserInputAst)>,\n) -> Result<UserInputAst, LenientErrorInternal> {\n    let mut leafs = Vec::with_capacity(others.len() + 1);\n    leafs.push((None, left.0, Some(left.1)));\n    leafs.extend(\n        others\n            .into_iter()\n            .map(|(operand, occur, ast)| (operand, occur, Some(ast))),\n    );\n    // the parameters we pass should statically guarantee we can't get errors\n    // (no prefix BinaryOperand is provided)\n    let (res, mut errors) = aggregate_infallible_expressions(leafs);\n    if errors.is_empty() {\n        Ok(res)\n    } else {\n        Err(errors.swap_remove(0))\n    }\n}\n\nfn aggregate_infallible_expressions(\n    input_leafs: Vec<(Option<BinaryOperand>, Option<Occur>, Option<UserInputAst>)>,\n) -> (UserInputAst, ErrorList) {\n    let mut err = Vec::new();\n    let mut leafs: Vec<(_, _, UserInputAst)> = input_leafs\n        .into_iter()\n        .filter_map(|(operand, occur, ast)| ast.map(|ast| (operand, occur, ast)))\n        .collect();\n    if leafs.is_empty() {\n        return (UserInputAst::empty_query(), err);\n    }\n\n    let early_operand = leafs\n        .iter()\n        .take(1)\n        .all(|(operand, _, _)| operand.is_some());\n\n    if early_operand {\n        err.push(LenientErrorInternal {\n            pos: 0,\n            message: \"Found unexpected boolean operator before term\".to_string(),\n        });\n    }\n\n    let mut clauses: Vec<Vec<(Option<Occur>, UserInputAst)>> = vec![];\n    for ((prev_operator, occur, ast), (next_operator, _, _)) in\n        leafs.iter().zip(leafs.iter().skip(1))\n    {\n        match prev_operator {\n            Some(BinaryOperand::And) => {\n                if let Some(last) = clauses.last_mut() {\n                    last.push((occur.or(Some(Occur::Must)), ast.clone()));\n                } else {\n                    let last = vec![(occur.or(Some(Occur::Must)), ast.clone())];\n                    clauses.push(last);\n                }\n            }\n            Some(BinaryOperand::Or) => {\n                let default_op = match next_operator {\n                    Some(BinaryOperand::And) => Some(Occur::Must),\n                    _ => Some(Occur::Should),\n                };\n                if occur == &Some(Occur::MustNot) && default_op == Some(Occur::Should) {\n                    // if occur is MustNot *and* operation is OR, we synthesize a ShouldNot\n                    clauses.push(vec![(\n                        Some(Occur::Should),\n                        ast.clone().unary(Occur::MustNot),\n                    )])\n                } else {\n                    clauses.push(vec![(occur.or(default_op), ast.clone())]);\n                }\n            }\n            None => {\n                let default_op = match next_operator {\n                    Some(BinaryOperand::And) => Some(Occur::Must),\n                    Some(BinaryOperand::Or) => Some(Occur::Should),\n                    None => None,\n                };\n                if occur == &Some(Occur::MustNot) && default_op == Some(Occur::Should) {\n                    // if occur is MustNot *and* operation is OR, we synthesize a ShouldNot\n                    clauses.push(vec![(\n                        Some(Occur::Should),\n                        ast.clone().unary(Occur::MustNot),\n                    )])\n                } else {\n                    clauses.push(vec![(occur.or(default_op), ast.clone())])\n                }\n            }\n        }\n    }\n\n    // leaf isn't empty, so we can unwrap\n    let (last_operator, last_occur, last_ast) = leafs.pop().unwrap();\n    match last_operator {\n        Some(BinaryOperand::And) => {\n            if let Some(last) = clauses.last_mut() {\n                last.push((last_occur.or(Some(Occur::Must)), last_ast));\n            } else {\n                let last = vec![(last_occur.or(Some(Occur::Must)), last_ast)];\n                clauses.push(last);\n            }\n        }\n        Some(BinaryOperand::Or) => {\n            if last_occur == Some(Occur::MustNot) {\n                // if occur is MustNot *and* operation is OR, we synthesize a ShouldNot\n                clauses.push(vec![(Some(Occur::Should), last_ast.unary(Occur::MustNot))]);\n            } else {\n                clauses.push(vec![(last_occur.or(Some(Occur::Should)), last_ast)]);\n            }\n        }\n        None => clauses.push(vec![(last_occur, last_ast)]),\n    }\n\n    if clauses.len() == 1 {\n        let mut clause = clauses.pop().unwrap();\n        if clause.len() == 1 && clause[0].0 != Some(Occur::MustNot) {\n            (clause.pop().unwrap().1, err)\n        } else {\n            (UserInputAst::Clause(clause), err)\n        }\n    } else {\n        let mut final_clauses: Vec<(Option<Occur>, UserInputAst)> = Vec::new();\n        for mut sub_clauses in clauses {\n            if sub_clauses.len() == 1 {\n                final_clauses.push(sub_clauses.pop().unwrap());\n            } else {\n                final_clauses.push((Some(Occur::Should), UserInputAst::Clause(sub_clauses)));\n            }\n        }\n\n        (UserInputAst::Clause(final_clauses), err)\n    }\n}\n\nfn operand_leaf(inp: &str) -> IResult<&str, (Option<BinaryOperand>, Option<Occur>, UserInputAst)> {\n    map(\n        tuple((\n            terminated(opt(binary_operand), multispace0),\n            terminated(occur_leaf, multispace0),\n        )),\n        |(operand, (occur, ast))| (operand, occur, ast),\n    )(inp)\n}\n\nfn ast(inp: &str) -> IResult<&str, UserInputAst> {\n    let boolean_expr = map_res(\n        separated_pair(occur_leaf, multispace1, many1(operand_leaf)),\n        |(left, right)| aggregate_binary_expressions(left, right),\n    );\n    let single_leaf = map(occur_leaf, |(occur, ast)| {\n        if occur == Some(Occur::MustNot) {\n            ast.unary(Occur::MustNot)\n        } else {\n            ast\n        }\n    });\n    delimited(multispace0, alt((boolean_expr, single_leaf)), multispace0)(inp)\n}\n\nfn ast_infallible(inp: &str) -> JResult<&str, UserInputAst> {\n    // ast() parse either `term AND term OR term` or `+term term -term`\n    // both are locally ambiguous, and as we allow error, it's hard to permit backtracking.\n    // Instead, we allow a mix of both syntaxes, trying to make sense of what a user meant.\n    // For instance `term OR -term` is interpreted as `*term -term`, but `term AND -term`\n    // is interpreted as `+term -term`. We also allow `AND term` to make things easier for us,\n    // even if it's not very sensical.\n\n    let expression = map(\n        separated_list_infallible(space1_infallible, operand_occur_leaf_infallible),\n        |(leaf, mut err)| {\n            let (res, mut err2) = aggregate_infallible_expressions(leaf);\n            err.append(&mut err2);\n            (res, err)\n        },\n    );\n\n    delimited_infallible(space0_infallible, expression, space0_infallible)(inp)\n}\n\npub fn parse_to_ast(inp: &str) -> IResult<&str, UserInputAst> {\n    map(delimited(multispace0, opt(ast), eof), |opt_ast| {\n        rewrite_ast(opt_ast.unwrap_or_else(UserInputAst::empty_query))\n    })(inp)\n}\n\npub fn parse_to_ast_lenient(query_str: &str) -> (UserInputAst, Vec<LenientError>) {\n    if query_str.trim().is_empty() {\n        return (UserInputAst::Clause(Vec::new()), Vec::new());\n    }\n    let (left, (res, mut errors)) = ast_infallible(query_str).unwrap();\n    if !left.trim().is_empty() {\n        errors.push(LenientErrorInternal {\n            pos: left.len(),\n            message: \"unparsed end of query\".to_string(),\n        })\n    }\n\n    // convert end-based index to start-based index.\n    let errors = errors\n        .into_iter()\n        .map(|internal_error| LenientError::from_internal(internal_error, query_str.len()))\n        .collect();\n\n    (rewrite_ast(res), errors)\n}\n\nfn rewrite_ast(mut input: UserInputAst) -> UserInputAst {\n    if let UserInputAst::Clause(sub_clauses) = &mut input {\n        // call rewrite_ast recursively on children clauses if applicable\n        let mut new_clauses = Vec::with_capacity(sub_clauses.len());\n        for (occur, clause) in sub_clauses.drain(..) {\n            let rewritten_clause = rewrite_ast(clause);\n            new_clauses.push((occur, rewritten_clause));\n        }\n        *sub_clauses = new_clauses;\n\n        // remove duplicate child clauses\n        // e.g. (+a +b) OR (+c +d) OR (+a +b)  => (+a +b) OR (+c +d)\n        let mut seen = FnvHashSet::default();\n        sub_clauses.retain(|term| seen.insert(term.clone()));\n\n        // Removes unnecessary children clauses in AST\n        //\n        // Motivated by [issue #1433](https://github.com/quickwit-oss/tantivy/issues/1433)\n        for term in sub_clauses {\n            rewrite_ast_clause(term);\n        }\n    }\n    input\n}\n\nfn rewrite_ast_clause(input: &mut (Option<Occur>, UserInputAst)) {\n    match input {\n        (None, UserInputAst::Clause(clauses)) if clauses.len() == 1 => {\n            *input = clauses.pop().unwrap(); // safe because clauses.len() == 1\n        }\n        _ => {}\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    pub fn nearly_equals(a: f64, b: f64) -> bool {\n        (a - b).abs() < 0.0005 * (a + b).abs()\n    }\n\n    fn assert_nearly_equals(expected: f64, val: f64) {\n        assert!(\n            nearly_equals(val, expected),\n            \"Got {val}, expected {expected}.\"\n        );\n    }\n\n    // TODO test as part of occur_leaf\n    // #[test]\n    // fn test_occur_symbol() -> TestParseResult {\n    // assert_eq!(super::occur_symbol(\"-\")?, (\"\", Occur::MustNot));\n    // assert_eq!(super::occur_symbol(\"+\")?, (\"\", Occur::Must));\n    // Ok(())\n    // }\n\n    #[test]\n    fn test_positive_float_number() {\n        fn valid_parse(float_str: &str, expected_val: f64, expected_remaining: &str) {\n            let (remaining, val) = positive_float_number(float_str).unwrap();\n            assert_eq!(remaining, expected_remaining);\n            assert_nearly_equals(val, expected_val);\n        }\n        fn error_parse(float_str: &str) {\n            assert!(positive_float_number(float_str).is_err());\n        }\n        valid_parse(\"1.0\", 1.0, \"\");\n        valid_parse(\"1\", 1.0, \"\");\n        valid_parse(\"0.234234 aaa\", 0.234234f64, \" aaa\");\n        error_parse(\".3332\");\n        // TODO trinity-1686a: I disagree that it should fail, I think it should succeed,\n        // consuming only \"1\", and leave \".\" for the next thing (which will likely fail then)\n        // error_parse(\"1.\");\n        error_parse(\"-1.\");\n    }\n\n    #[test]\n    fn test_date_time() {\n        let (remaining, val) =\n            relaxed_word(\"2015-08-02T18:54:42+02:30\").expect(\"cannot parse date\");\n        assert_eq!(val, \"2015-08-02T18:54:42+02:30\");\n        assert_eq!(remaining, \"\");\n        // this isn't a valid date, but relaxed_word allows it.\n        // assert!(date_time().parse(\"2015-08-02T18:54:42+02\").is_err());\n\n        let (remaining, val) = relaxed_word(\"2021-04-13T19:46:26.266051969+00:00\")\n            .expect(\"cannot parse fractional date\");\n        assert_eq!(val, \"2021-04-13T19:46:26.266051969+00:00\");\n        assert_eq!(remaining, \"\");\n    }\n\n    #[track_caller]\n    fn test_parse_query_to_ast_helper(query: &str, expected: &str) {\n        let query_strict = parse_to_ast(query).unwrap().1;\n        let query_strict_str = format!(\"{query_strict:?}\");\n        assert_eq!(query_strict_str, expected, \"strict parser failed\");\n\n        let (query_lenient, errs) = parse_to_ast_lenient(query);\n        let query_lenient_str = format!(\"{query_lenient:?}\");\n        assert_eq!(query_lenient_str, expected, \"lenient parser failed\");\n        assert!(\n            errs.is_empty(),\n            \"lenient parser returned errors on valid query: {errs:?}\"\n        );\n    }\n\n    #[track_caller]\n    fn test_is_parse_err(query: &str, lenient_expected: &str) {\n        assert!(\n            parse_to_ast(query).is_err(),\n            \"strict parser succeeded where an error was expected.\"\n        );\n\n        let (query_lenient, errs) = parse_to_ast_lenient(query);\n        let query_lenient_str = format!(\"{query_lenient:?}\");\n        assert_eq!(query_lenient_str, lenient_expected, \"lenient parser failed\");\n        assert!(!errs.is_empty());\n    }\n\n    #[test]\n    fn test_parse_empty_to_ast() {\n        test_parse_query_to_ast_helper(\"\", \"<emptyclause>\");\n    }\n\n    #[test]\n    fn test_parse_query_to_ast_hyphen() {\n        test_parse_query_to_ast_helper(\"\\\"www-form-encoded\\\"\", \"\\\"www-form-encoded\\\"\");\n        test_parse_query_to_ast_helper(\"'www-form-encoded'\", \"'www-form-encoded'\");\n        test_parse_query_to_ast_helper(\"www-form-encoded\", \"www-form-encoded\");\n        test_parse_query_to_ast_helper(\"www-form-encoded\", \"www-form-encoded\");\n        test_parse_query_to_ast_helper(\"mr james bo?d\", \"(*mr *james *bo?d)\");\n        test_parse_query_to_ast_helper(\"mr james bo*\", \"(*mr *james *bo*)\");\n        test_parse_query_to_ast_helper(\"mr james b*d\", \"(*mr *james *b*d)\");\n    }\n\n    #[test]\n    fn test_parse_query_lenient_unfinished_quote() {\n        test_is_parse_err(\"\\\"www-form-encoded\", \"\\\"www-form-encoded\\\"\");\n        // TODO strict parser default to parsing a normal term, and parse \"'www-forme-encoded\" (note\n        // the initial \\')\n        // test_is_parse_err(\"'www-form-encoded\", \"'www-form-encoded'\");\n    }\n\n    #[test]\n    fn test_parse_query_to_ast_not_op() {\n        test_is_parse_err(\"NOT\", \"NOT\");\n        test_parse_query_to_ast_helper(\"NOTa\", \"NOTa\");\n        test_parse_query_to_ast_helper(\"NOT a\", \"(-a)\");\n    }\n\n    #[test]\n    fn test_boosting() {\n        test_is_parse_err(\"a^2^3\", \"(a)^2\");\n        test_is_parse_err(\"a^2^\", \"(a)^2\");\n        test_parse_query_to_ast_helper(\"a^3\", \"(a)^3\");\n        test_parse_query_to_ast_helper(\"a^3 b^2\", \"(*(a)^3 *(b)^2)\");\n        test_parse_query_to_ast_helper(\"a^1\", \"a\");\n    }\n\n    #[test]\n    fn test_parse_query_to_ast_binary_op() {\n        test_parse_query_to_ast_helper(\"a AND b\", \"(+a +b)\");\n        test_parse_query_to_ast_helper(\"a\\nAND b\", \"(+a +b)\");\n        test_parse_query_to_ast_helper(\"a OR b\", \"(?a ?b)\");\n        test_parse_query_to_ast_helper(\"a OR b AND c\", \"(?a ?(+b +c))\");\n        test_parse_query_to_ast_helper(\"a AND b         AND c\", \"(+a +b +c)\");\n        test_parse_query_to_ast_helper(\"a OR b aaa\", \"(?a ?b *aaa)\");\n        test_parse_query_to_ast_helper(\"a AND b aaa\", \"(?(+a +b) *aaa)\");\n        test_parse_query_to_ast_helper(\"aaa a OR b \", \"(*aaa ?a ?b)\");\n        test_parse_query_to_ast_helper(\"aaa ccc a OR b \", \"(*aaa *ccc ?a ?b)\");\n        test_parse_query_to_ast_helper(\"aaa a AND b \", \"(*aaa ?(+a +b))\");\n        test_parse_query_to_ast_helper(\"aaa ccc a AND b \", \"(*aaa *ccc ?(+a +b))\");\n    }\n\n    #[test]\n    fn test_parse_mixed_bool_occur() {\n        test_parse_query_to_ast_helper(\"+a OR +b\", \"(+a +b)\");\n\n        test_parse_query_to_ast_helper(\"a AND -b\", \"(+a -b)\");\n        test_parse_query_to_ast_helper(\"-a AND b\", \"(-a +b)\");\n        test_parse_query_to_ast_helper(\"a AND NOT b\", \"(+a +(-b))\");\n        test_parse_query_to_ast_helper(\"NOT a AND b\", \"(+(-a) +b)\");\n\n        test_parse_query_to_ast_helper(\"a AND NOT b AND c\", \"(+a +(-b) +c)\");\n        test_parse_query_to_ast_helper(\"a AND -b AND c\", \"(+a -b +c)\");\n\n        test_parse_query_to_ast_helper(\"a OR -b\", \"(?a ?(-b))\");\n        test_parse_query_to_ast_helper(\"-a OR b\", \"(?(-a) ?b)\");\n        test_parse_query_to_ast_helper(\"a OR NOT b\", \"(?a ?(-b))\");\n        test_parse_query_to_ast_helper(\"NOT a OR b\", \"(?(-a) ?b)\");\n\n        test_parse_query_to_ast_helper(\"a OR NOT b OR c\", \"(?a ?(-b) ?c)\");\n        test_parse_query_to_ast_helper(\"a OR -b OR c\", \"(?a ?(-b) ?c)\");\n\n        test_parse_query_to_ast_helper(\"a OR b +aaa\", \"(?a ?b +aaa)\");\n        test_parse_query_to_ast_helper(\"a AND b -aaa\", \"(?(+a +b) -aaa)\");\n        test_parse_query_to_ast_helper(\"+a OR +b aaa\", \"(+a +b *aaa)\");\n        test_parse_query_to_ast_helper(\"-a AND -b aaa\", \"(?(-a -b) *aaa)\");\n        test_parse_query_to_ast_helper(\"-aaa +ccc -a OR b \", \"(-aaa +ccc ?(-a) ?b)\");\n    }\n\n    #[test]\n    fn test_parse_elastic_query_ranges() {\n        test_parse_query_to_ast_helper(\"title: >a\", \"\\\"title\\\":{\\\"a\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\"title:>=a\", \"\\\"title\\\":[\\\"a\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\"title: <a\", \"\\\"title\\\":{\\\"*\\\" TO \\\"a\\\"}\");\n        test_parse_query_to_ast_helper(\"title:<=a\", \"\\\"title\\\":{\\\"*\\\" TO \\\"a\\\"]\");\n        test_parse_query_to_ast_helper(\"title:<=bsd\", \"\\\"title\\\":{\\\"*\\\" TO \\\"bsd\\\"]\");\n\n        test_parse_query_to_ast_helper(\"weight: >70\", \"\\\"weight\\\":{\\\"70\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\"weight:>=70\", \"\\\"weight\\\":[\\\"70\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\"weight: <70\", \"\\\"weight\\\":{\\\"*\\\" TO \\\"70\\\"}\");\n        test_parse_query_to_ast_helper(\"weight:<=70\", \"\\\"weight\\\":{\\\"*\\\" TO \\\"70\\\"]\");\n        test_parse_query_to_ast_helper(\"weight: >60.7\", \"\\\"weight\\\":{\\\"60.7\\\" TO \\\"*\\\"}\");\n\n        test_parse_query_to_ast_helper(\"weight: <= 70\", \"\\\"weight\\\":{\\\"*\\\" TO \\\"70\\\"]\");\n\n        test_parse_query_to_ast_helper(\"weight: <= 70.5\", \"\\\"weight\\\":{\\\"*\\\" TO \\\"70.5\\\"]\");\n\n        test_parse_query_to_ast_helper(\">a\", \"{\\\"a\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\">=a\", \"[\\\"a\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\"<a\", \"{\\\"*\\\" TO \\\"a\\\"}\");\n        test_parse_query_to_ast_helper(\"<=a\", \"{\\\"*\\\" TO \\\"a\\\"]\");\n        test_parse_query_to_ast_helper(\"<=bsd\", \"{\\\"*\\\" TO \\\"bsd\\\"]\");\n\n        test_parse_query_to_ast_helper(\"(<=42)\", \"{\\\"*\\\" TO \\\"42\\\"]\");\n        test_parse_query_to_ast_helper(\"(<=42 )\", \"{\\\"*\\\" TO \\\"42\\\"]\");\n        test_parse_query_to_ast_helper(\"(age:>5)\", \"\\\"age\\\":{\\\"5\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\n            \"(title:bar AND age:>12)\",\n            \"(+\\\"title\\\":bar +\\\"age\\\":{\\\"12\\\" TO \\\"*\\\"})\",\n        );\n    }\n\n    #[test]\n    fn test_occur_leaf() {\n        let (_, (occur, ast)) = super::occur_leaf(\"+abc\").unwrap();\n        assert_eq!(occur, Some(Occur::Must));\n        assert_eq!(format!(\"{ast:?}\"), \"abc\");\n    }\n\n    #[test]\n    fn test_field_name() {\n        assert_eq!(\n            super::field_name(\".my.field.name:a\"),\n            Ok((\"a\", \".my.field.name\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(r#\"にんじん:a\"#),\n            Ok((\"a\", \"にんじん\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(r#\"my\\field:a\"#),\n            Ok((\"a\", r#\"my\\field\"#.to_string()))\n        );\n        assert_eq!(\n            super::field_name(r#\"my\\\\field:a\"#),\n            Ok((\"a\", r#\"my\\field\"#.to_string()))\n        );\n        assert!(super::field_name(\"my field:a\").is_err());\n        assert_eq!(\n            super::field_name(\"\\\\(1\\\\+1\\\\):2\"),\n            Ok((\"2\", \"(1+1)\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(\"my_field_name:a\"),\n            Ok((\"a\", \"my_field_name\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(\"myfield.b:hello\").unwrap(),\n            (\"hello\", \"myfield.b\".to_string())\n        );\n        assert_eq!(\n            super::field_name(r#\"myfield\\.b:hello\"#).unwrap(),\n            (\"hello\", r#\"myfield\\.b\"#.to_string())\n        );\n        assert!(super::field_name(\"my_field_name\").is_err());\n        assert!(super::field_name(\":a\").is_err());\n        assert!(super::field_name(\"-my_field:a\").is_err());\n        assert_eq!(\n            super::field_name(\"_my_field:a\"),\n            Ok((\"a\", \"_my_field\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(\"~my~field:a\"),\n            Ok((\"a\", \"~my~field\".to_string()))\n        );\n        assert_eq!(\n            super::field_name(\".my.field.name : a\"),\n            Ok((\"a\", \".my.field.name\".to_string()))\n        );\n        for special_char in SPECIAL_CHARS.iter() {\n            let query = &format!(\"\\\\{special_char}my\\\\{special_char}field:a\");\n            assert_eq!(\n                super::field_name(query),\n                Ok((\"a\", format!(\"{special_char}my{special_char}field\")))\n            );\n        }\n    }\n\n    #[test]\n    fn test_range_parser() {\n        // testing the range() parser separately\n        let res = literal(\"title: <hello\")\n            .expect(\"Cannot parse flexible bound word\")\n            .1;\n        let expected = UserInputLeaf::Range {\n            field: Some(\"title\".to_string()),\n            lower: UserInputBound::Unbounded,\n            upper: UserInputBound::Exclusive(\"hello\".to_string()),\n        }\n        .into();\n        let res2 = literal(\"title:{* TO hello}\")\n            .expect(\"Cannot parse ununbounded to word\")\n            .1;\n        assert_eq!(res, expected);\n        assert_eq!(res2, expected);\n\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"weight\".to_string()),\n            lower: UserInputBound::Inclusive(\"71.2\".to_string()),\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res3 = literal(\"weight: >=71.2\")\n            .expect(\"Cannot parse flexible bound float\")\n            .1;\n        let res4 = literal(\"weight:[71.2 TO *}\")\n            .expect(\"Cannot parse float to unbounded\")\n            .1;\n        assert_eq!(res3, expected_weight);\n        assert_eq!(res4, expected_weight);\n\n        let expected_dates = UserInputLeaf::Range {\n            field: Some(\"date_field\".to_string()),\n            lower: UserInputBound::Exclusive(\"2015-08-02T18:54:42Z\".to_string()),\n            upper: UserInputBound::Inclusive(\"2021-08-02T18:54:42+02:30\".to_string()),\n        }\n        .into();\n        let res5 = literal(\"date_field:{2015-08-02T18:54:42Z TO 2021-08-02T18:54:42+02:30]\")\n            .expect(\"Cannot parse date range\")\n            .1;\n        assert_eq!(res5, expected_dates);\n\n        let expected_flexible_dates = UserInputLeaf::Range {\n            field: Some(\"date_field\".to_string()),\n            lower: UserInputBound::Unbounded,\n            upper: UserInputBound::Inclusive(\"2021-08-02T18:54:42.12345+02:30\".to_string()),\n        }\n        .into();\n\n        let res6 = literal(\"date_field: <=2021-08-02T18:54:42.12345+02:30\")\n            .expect(\"Cannot parse date range\")\n            .1;\n        assert_eq!(res6, expected_flexible_dates);\n        // IP Range Unbounded\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::1\".to_string()),\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res1 = literal(\"ip: >=::1\").expect(\"Cannot parse ip v6 format\").1;\n        let res2 = literal(\"ip:[::1 TO *}\")\n            .expect(\"Cannot parse ip v6 format\")\n            .1;\n        assert_eq!(res1, expected_weight);\n        assert_eq!(res2, expected_weight);\n\n        // IP Range Bounded\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::0.0.0.50\".to_string()),\n            upper: UserInputBound::Exclusive(\"::0.0.0.52\".to_string()),\n        }\n        .into();\n        let res1 = literal(\"ip:[::0.0.0.50 TO ::0.0.0.52}\")\n            .expect(\"Cannot parse ip v6 format\")\n            .1;\n        assert_eq!(res1, expected_weight);\n    }\n\n    #[test]\n    fn test_range_parser_lenient() {\n        let literal = |query| literal_infallible(query).unwrap().1.0.unwrap();\n\n        // same tests as non-lenient\n        let res = literal(\"title: <hello\");\n        let expected = UserInputLeaf::Range {\n            field: Some(\"title\".to_string()),\n            lower: UserInputBound::Unbounded,\n            upper: UserInputBound::Exclusive(\"hello\".to_string()),\n        }\n        .into();\n        let res2 = literal(\"title:{* TO hello}\");\n        assert_eq!(res, expected);\n        assert_eq!(res2, expected);\n\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"weight\".to_string()),\n            lower: UserInputBound::Inclusive(\"71.2\".to_string()),\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res3 = literal(\"weight: >=71.2\");\n        let res4 = literal(\"weight:[71.2 TO *}\");\n        assert_eq!(res3, expected_weight);\n        assert_eq!(res4, expected_weight);\n\n        let expected_dates = UserInputLeaf::Range {\n            field: Some(\"date_field\".to_string()),\n            lower: UserInputBound::Exclusive(\"2015-08-02T18:54:42Z\".to_string()),\n            upper: UserInputBound::Inclusive(\"2021-08-02T18:54:42+02:30\".to_string()),\n        }\n        .into();\n        let res5 = literal(\"date_field:{2015-08-02T18:54:42Z TO 2021-08-02T18:54:42+02:30]\");\n        assert_eq!(res5, expected_dates);\n\n        let expected_flexible_dates = UserInputLeaf::Range {\n            field: Some(\"date_field\".to_string()),\n            lower: UserInputBound::Unbounded,\n            upper: UserInputBound::Inclusive(\"2021-08-02T18:54:42.12345+02:30\".to_string()),\n        }\n        .into();\n\n        let res6 = literal(\"date_field: <=2021-08-02T18:54:42.12345+02:30\");\n        assert_eq!(res6, expected_flexible_dates);\n        // IP Range Unbounded\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::1\".to_string()),\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res1 = literal(\"ip: >=::1\");\n        let res2 = literal(\"ip:[::1 TO *}\");\n        assert_eq!(res1, expected_weight);\n        assert_eq!(res2, expected_weight);\n\n        // IP Range Bounded\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::0.0.0.50\".to_string()),\n            upper: UserInputBound::Exclusive(\"::0.0.0.52\".to_string()),\n        }\n        .into();\n        let res1 = literal(\"ip:[::0.0.0.50 TO ::0.0.0.52}\");\n        assert_eq!(res1, expected_weight);\n\n        // additional tests\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::0.0.0.50\".to_string()),\n            upper: UserInputBound::Inclusive(\"::0.0.0.52\".to_string()),\n        }\n        .into();\n        let res1 = literal(\"ip:[::0.0.0.50 TO ::0.0.0.52\");\n        let res2 = literal(\"ip:[::0.0.0.50 ::0.0.0.52\");\n        let res3 = literal(\"ip:[::0.0.0.50 ::0.0.0.52 AND ...\");\n        assert_eq!(res1, expected_weight);\n        assert_eq!(res2, expected_weight);\n        assert_eq!(res3, expected_weight);\n\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Inclusive(\"::0.0.0.50\".to_string()),\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res1 = literal(\"ip:[::0.0.0.50 TO \");\n        let res2 = literal(\"ip:[::0.0.0.50 TO\");\n        let res3 = literal(\"ip:[::0.0.0.50\");\n        assert_eq!(res1, expected_weight);\n        assert_eq!(res2, expected_weight);\n        assert_eq!(res3, expected_weight);\n\n        let expected_weight = UserInputLeaf::Range {\n            field: Some(\"ip\".to_string()),\n            lower: UserInputBound::Unbounded,\n            upper: UserInputBound::Unbounded,\n        }\n        .into();\n        let res1 = literal(\"ip:[ \");\n        let res2 = literal(\"ip:{ \");\n        let res3 = literal(\"ip:[\");\n        assert_eq!(res1, expected_weight);\n        assert_eq!(res2, expected_weight);\n        assert_eq!(res3, expected_weight);\n        // we don't test ip: as that is not a valid range request as per percondition\n    }\n\n    #[test]\n    fn test_parse_query_to_trimming_spaces() {\n        test_parse_query_to_ast_helper(\"   abc\", \"abc\");\n        test_parse_query_to_ast_helper(\"abc \", \"abc\");\n        test_parse_query_to_ast_helper(\"(  a OR abc)\", \"(?a ?abc)\");\n        test_parse_query_to_ast_helper(\"(a  OR abc)\", \"(?a ?abc)\");\n        test_parse_query_to_ast_helper(\"(a OR  abc)\", \"(?a ?abc)\");\n        test_parse_query_to_ast_helper(\"a OR abc \", \"(?a ?abc)\");\n        test_parse_query_to_ast_helper(\"(a OR abc )\", \"(?a ?abc)\");\n        test_parse_query_to_ast_helper(\"(a OR  abc) \", \"(?a ?abc)\");\n        test_is_parse_err(\"(a OR  abc \", \"(?a ?abc)\");\n    }\n\n    #[test]\n    fn test_parse_query_term_group() {\n        test_parse_query_to_ast_helper(r#\"field:(abc)\"#, r#\"\"field\":abc\"#);\n        test_parse_query_to_ast_helper(r#\"field:(+a -\"b c\")\"#, r#\"(+\"field\":a -\"field\":\"b c\")\"#);\n        test_parse_query_to_ast_helper(r#\"field:(a AND \"b c\")\"#, r#\"(+\"field\":a +\"field\":\"b c\")\"#);\n        test_parse_query_to_ast_helper(r#\"field:(a OR \"b c\")\"#, r#\"(?\"field\":a ?\"field\":\"b c\")\"#);\n        test_parse_query_to_ast_helper(\n            r#\"field:(a OR (b AND c))\"#,\n            r#\"(?\"field\":a ?(+\"field\":b +\"field\":c))\"#,\n        );\n        test_parse_query_to_ast_helper(\n            r#\"field:(a [b TO c])\"#,\n            r#\"(*\"field\":a *\"field\":[\"b\" TO \"c\"])\"#,\n        );\n\n        test_is_parse_err(r#\"field:(+a -\"b c\"\"#, r#\"(+\"field\":a -\"field\":\"b c\")\"#);\n    }\n\n    #[test]\n    fn field_re_specification() {\n        test_parse_query_to_ast_helper(r#\"field:(abc AND b:cde)\"#, r#\"(+\"field\":abc +\"b\":cde)\"#);\n    }\n\n    #[test]\n    fn test_parse_query_single_term() {\n        test_parse_query_to_ast_helper(\"abc\", \"abc\");\n    }\n\n    #[test]\n    fn test_parse_query_default_clause() {\n        test_parse_query_to_ast_helper(\"a b\", \"(*a *b)\");\n    }\n\n    #[test]\n    fn test_parse_query_must_default_clause() {\n        test_parse_query_to_ast_helper(\"+(a b)\", \"(*a *b)\");\n    }\n\n    #[test]\n    fn test_parse_query_must_single_term() {\n        test_parse_query_to_ast_helper(\"+d\", \"d\");\n    }\n\n    #[test]\n    fn test_single_term_with_field() {\n        test_parse_query_to_ast_helper(\"abc:toto\", \"\\\"abc\\\":toto\");\n    }\n\n    #[test]\n    fn test_phrase_with_field() {\n        test_parse_query_to_ast_helper(\"abc:\\\"happy tax payer\\\"\", \"\\\"abc\\\":\\\"happy tax payer\\\"\");\n        test_parse_query_to_ast_helper(\"abc:'happy tax payer'\", \"\\\"abc\\\":'happy tax payer'\");\n    }\n\n    #[test]\n    fn test_single_term_with_float() {\n        test_parse_query_to_ast_helper(\"abc:1.1\", \"\\\"abc\\\":1.1\");\n        test_parse_query_to_ast_helper(\"a.b.c:1.1\", \"\\\"a.b.c\\\":1.1\");\n        test_parse_query_to_ast_helper(\"a\\\\ b\\\\ c:1.1\", \"\\\"a b c\\\":1.1\");\n    }\n\n    #[test]\n    fn test_must_clause() {\n        test_parse_query_to_ast_helper(\"(+a +b)\", \"(+a +b)\");\n    }\n\n    #[test]\n    fn test_parse_test_query_plus_a_b_plus_d() {\n        test_parse_query_to_ast_helper(\"+(a b) +d\", \"(+(*a *b) +d)\");\n    }\n\n    #[test]\n    fn test_parse_test_query_set() {\n        test_parse_query_to_ast_helper(\"abc: IN [a b c]\", r#\"\"abc\": IN [\"a\" \"b\" \"c\"]\"#);\n        test_parse_query_to_ast_helper(\"abc: IN [1]\", r#\"\"abc\": IN [\"1\"]\"#);\n        test_parse_query_to_ast_helper(\"abc: IN []\", r#\"\"abc\": IN []\"#);\n        test_parse_query_to_ast_helper(\"IN [1 2]\", r#\"IN [\"1\" \"2\"]\"#);\n        test_is_parse_err(\"IN [1 2\", r#\"IN [\"1\" \"2\"]\"#);\n\n        // TODO maybe support these too?\n        // test_is_parse_err(\"IN (1 2\", r#\"IN [\"1\" \"2\"]\"#);\n        // test_is_parse_err(\"IN {1 2\", r#\"IN [\"1\" \"2\"]\"#);\n    }\n\n    #[test]\n    fn test_parse_test_query_other() {\n        test_parse_query_to_ast_helper(\"(+a +b) d\", \"(*(+a +b) *d)\");\n        test_parse_query_to_ast_helper(\"+abc:toto\", \"\\\"abc\\\":toto\");\n        test_parse_query_to_ast_helper(\"+a\\\\+b\\\\+c:toto\", \"\\\"a+b+c\\\":toto\");\n        test_parse_query_to_ast_helper(\"(+abc:toto -titi)\", \"(+\\\"abc\\\":toto -titi)\");\n        test_parse_query_to_ast_helper(\"-abc:toto\", \"(-\\\"abc\\\":toto)\");\n        // TODO not entirely sure about this one (it's seen as a NOT '-abc:toto')\n        test_is_parse_err(\"--abc:toto\", \"(--abc:toto)\");\n        test_parse_query_to_ast_helper(\"abc:a b\", \"(*\\\"abc\\\":a *b)\");\n        test_parse_query_to_ast_helper(\"abc:\\\"a b\\\"\", \"\\\"abc\\\":\\\"a b\\\"\");\n        test_parse_query_to_ast_helper(\"foo:[1 TO 5]\", \"\\\"foo\\\":[\\\"1\\\" TO \\\"5\\\"]\");\n\n        // Phrase prefixed with *\n        test_parse_query_to_ast_helper(\"foo:(*A)\", \"\\\"foo\\\":*A\");\n        test_parse_query_to_ast_helper(\"*A\", \"*A\");\n        test_parse_query_to_ast_helper(\"(*A)\", \"*A\");\n        test_parse_query_to_ast_helper(\"foo:(A OR B)\", \"(?\\\"foo\\\":A ?\\\"foo\\\":B)\");\n        test_parse_query_to_ast_helper(\"foo:(A* OR B*)\", \"(?\\\"foo\\\":A* ?\\\"foo\\\":B*)\");\n        test_parse_query_to_ast_helper(\"foo:(*A OR *B)\", \"(?\\\"foo\\\":*A ?\\\"foo\\\":*B)\");\n\n        // Regexes between parentheses\n        test_parse_query_to_ast_helper(\"foo:(/A.*/)\", \"\\\"foo\\\":/A.*/\");\n        test_parse_query_to_ast_helper(\"foo:(/A.*/ OR /B.*/)\", \"(?\\\"foo\\\":/A.*/ ?\\\"foo\\\":/B.*/)\");\n    }\n\n    #[test]\n    fn test_parse_query_all() {\n        test_parse_query_to_ast_helper(\"*\", \"*\");\n        test_parse_query_to_ast_helper(\"(*)\", \"*\");\n        test_parse_query_to_ast_helper(\"(* )\", \"*\");\n    }\n\n    #[test]\n    fn test_parse_query_with_range() {\n        test_parse_query_to_ast_helper(\"[1 TO 5]\", \"[\\\"1\\\" TO \\\"5\\\"]\");\n        test_parse_query_to_ast_helper(\"foo:{a TO z}\", \"\\\"foo\\\":{\\\"a\\\" TO \\\"z\\\"}\");\n        test_parse_query_to_ast_helper(\"foo:[1 TO toto}\", \"\\\"foo\\\":[\\\"1\\\" TO \\\"toto\\\"}\");\n        test_parse_query_to_ast_helper(\"foo:[* TO toto}\", \"\\\"foo\\\":{\\\"*\\\" TO \\\"toto\\\"}\");\n        test_parse_query_to_ast_helper(\"foo:[1 TO *}\", \"\\\"foo\\\":[\\\"1\\\" TO \\\"*\\\"}\");\n        test_parse_query_to_ast_helper(\n            \"1.2.foo.bar:[1.1 TO *}\",\n            \"\\\"1.2.foo.bar\\\":[\\\"1.1\\\" TO \\\"*\\\"}\",\n        );\n        test_is_parse_err(\"abc +    \", \"abc\");\n    }\n\n    #[test]\n    fn test_slop() {\n        test_is_parse_err(\"\\\"a b\\\"~\", \"(*\\\"a b\\\" *~)\");\n        test_is_parse_err(\"foo:\\\"a b\\\"~\", \"(*\\\"foo\\\":\\\"a b\\\" *~)\");\n        test_is_parse_err(\"\\\"a b\\\"~a\", \"(*\\\"a b\\\" *~a)\");\n        test_is_parse_err(\n            \"\\\"a b\\\"~100000000000000000\",\n            \"(*\\\"a b\\\" *~100000000000000000)\",\n        );\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"^2 ~4\", \"(*(\\\"a b\\\")^2 *~4)\");\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"~4^2\", \"(\\\"a b\\\"~4)^2\");\n        test_parse_query_to_ast_helper(\"\\\"~Document\\\"\", \"\\\"~Document\\\"\");\n        test_parse_query_to_ast_helper(\"~Document\", \"~Document\");\n        test_parse_query_to_ast_helper(\"a~2\", \"a~2\");\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"~0\", \"\\\"a b\\\"\");\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"~1\", \"\\\"a b\\\"~1\");\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"~3\", \"\\\"a b\\\"~3\");\n        test_parse_query_to_ast_helper(\"foo:\\\"a b\\\"~300\", \"\\\"foo\\\":\\\"a b\\\"~300\");\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"~300^2\", \"(\\\"a b\\\"~300)^2\");\n    }\n\n    #[test]\n    fn test_phrase_prefix() {\n        test_parse_query_to_ast_helper(\"\\\"a b\\\"*\", \"\\\"a b\\\"*\");\n        test_parse_query_to_ast_helper(\"\\\"a\\\"*\", \"\\\"a\\\"*\");\n        test_parse_query_to_ast_helper(\"\\\"\\\"*\", \"\\\"\\\"*\");\n        test_parse_query_to_ast_helper(\"foo:\\\"a b\\\"*\", \"\\\"foo\\\":\\\"a b\\\"*\");\n        test_parse_query_to_ast_helper(\"foo:\\\"a\\\"*\", \"\\\"foo\\\":\\\"a\\\"*\");\n        test_parse_query_to_ast_helper(\"foo:\\\"\\\"*\", \"\\\"foo\\\":\\\"\\\"*\");\n    }\n\n    #[test]\n    fn test_exist_query() {\n        test_parse_query_to_ast_helper(\"a:*\", \"$exists(\\\"a\\\")\");\n        test_parse_query_to_ast_helper(\"a: *\", \"$exists(\\\"a\\\")\");\n\n        test_parse_query_to_ast_helper(\n            \"(hello AND toto:*) OR happy\",\n            \"(?(+hello +$exists(\\\"toto\\\")) ?happy)\",\n        );\n        test_parse_query_to_ast_helper(\"(a:*)\", \"$exists(\\\"a\\\")\");\n\n        // these are term/wildcard query (not a phrase prefix)\n        test_parse_query_to_ast_helper(\"a:b*\", \"\\\"a\\\":b*\");\n        test_parse_query_to_ast_helper(\"a:*b\", \"\\\"a\\\":*b\");\n        test_parse_query_to_ast_helper(r#\"a:*def*\"#, \"\\\"a\\\":*def*\");\n    }\n\n    #[test]\n    fn test_not_queries_are_consistent() {\n        test_parse_query_to_ast_helper(\"tata -toto\", \"(*tata -toto)\");\n        test_parse_query_to_ast_helper(\"tata NOT toto\", \"(*tata -toto)\");\n    }\n\n    #[test]\n    fn test_escaping() {\n        test_parse_query_to_ast_helper(\n            r#\"myfield:\"hello\\\"happy\\'tax\"\"#,\n            r#\"\"myfield\":\"hello\"happy'tax\"\"#,\n        );\n        test_parse_query_to_ast_helper(\n            r#\"myfield:'hello\\\"happy\\'tax'\"#,\n            r#\"\"myfield\":'hello\"happy'tax'\"#,\n        );\n        // we don't process escape sequence for chars which don't require it\n        test_parse_query_to_ast_helper(r#\"abc\\*\"#, r#\"abc\\*\"#);\n    }\n\n    #[test]\n    fn test_queries_with_colons() {\n        test_parse_query_to_ast_helper(r#\"\"abc:def\"\"#, r#\"\"abc:def\"\"#);\n        test_parse_query_to_ast_helper(r#\"'abc:def'\"#, r#\"'abc:def'\"#);\n        test_parse_query_to_ast_helper(r#\"abc\\:def\"#, r#\"abc:def\"#);\n        test_parse_query_to_ast_helper(r#\"\"abc\\:def\"\"#, r#\"\"abc:def\"\"#);\n        test_parse_query_to_ast_helper(r#\"'abc\\:def'\"#, r#\"'abc:def'\"#);\n    }\n\n    #[test]\n    fn test_invalid_field() {\n        test_is_parse_err(r#\"!bc:def\"#, \"!bc:def\");\n    }\n\n    #[test]\n    fn test_regex_parser() {\n        let r = parse_to_ast(r#\"a:/joh?n(ath[oa]n)/\"#);\n        assert!(r.is_ok(), \"Failed to parse custom query: {r:?}\");\n        let (_, input) = r.unwrap();\n        match input {\n            UserInputAst::Leaf(leaf) => match leaf.as_ref() {\n                UserInputLeaf::Regex { field, pattern } => {\n                    assert_eq!(field, &Some(\"a\".to_string()));\n                    assert_eq!(pattern, \"joh?n(ath[oa]n)\");\n                }\n                _ => panic!(\"Expected a regex leaf, got {leaf:?}\"),\n            },\n            _ => panic!(\"Expected a leaf\"),\n        }\n        let r = parse_to_ast(r#\"a:/\\\\/cgi-bin\\\\/luci.*/\"#);\n        assert!(r.is_ok(), \"Failed to parse custom query: {r:?}\");\n        let (_, input) = r.unwrap();\n        match input {\n            UserInputAst::Leaf(leaf) => match leaf.as_ref() {\n                UserInputLeaf::Regex { field, pattern } => {\n                    assert_eq!(field, &Some(\"a\".to_string()));\n                    assert_eq!(pattern, \"\\\\/cgi-bin\\\\/luci.*\");\n                }\n                _ => panic!(\"Expected a regex leaf, got {leaf:?}\"),\n            },\n            _ => panic!(\"Expected a leaf\"),\n        }\n    }\n\n    #[test]\n    fn test_regex_parser_lenient() {\n        let literal = |query| literal_infallible(query).unwrap().1;\n\n        let (res, errs) = literal(r#\"a:/joh?n(ath[oa]n)/\"#);\n        let expected = UserInputLeaf::Regex {\n            field: Some(\"a\".to_string()),\n            pattern: \"joh?n(ath[oa]n)\".to_string(),\n        }\n        .into();\n        assert_eq!(res.unwrap(), expected);\n        assert!(errs.is_empty(), \"Expected no errors, got: {errs:?}\");\n\n        let (res, errs) = literal(\"title:/joh?n(ath[oa]n)\");\n        let expected = UserInputLeaf::Regex {\n            field: Some(\"title\".to_string()),\n            pattern: \"joh?n(ath[oa]n)\".to_string(),\n        }\n        .into();\n        assert_eq!(res.unwrap(), expected);\n        assert_eq!(errs.len(), 1, \"Expected 1 error, got: {errs:?}\");\n        assert_eq!(\n            errs[0].message, \"missing delimiter /\",\n            \"Unexpected error message\",\n        );\n    }\n\n    #[test]\n    fn test_space_before_value() {\n        test_parse_query_to_ast_helper(\"field : a\", r#\"\"field\":a\"#);\n        test_parse_query_to_ast_helper(\"field:    a\", r#\"\"field\":a\"#);\n        test_parse_query_to_ast_helper(\"field         :a\", r#\"\"field\":a\"#);\n        test_parse_query_to_ast_helper(\n            \"field : 'happy tax payer' AND other_field  : 1\",\n            r#\"(+\"field\":'happy tax payer' +\"other_field\":1)\"#,\n        );\n    }\n}\n"
  },
  {
    "path": "query-grammar/src/user_input_ast.rs",
    "content": "use std::fmt;\nuse std::fmt::{Debug, Formatter};\n\nuse serde::Serialize;\n\nuse crate::Occur;\n\n#[derive(PartialEq, Eq, Hash, Clone, Serialize)]\n#[serde(tag = \"type\")]\n#[serde(rename_all = \"snake_case\")]\npub enum UserInputLeaf {\n    Literal(UserInputLiteral),\n    All,\n    Range {\n        field: Option<String>,\n        lower: UserInputBound,\n        upper: UserInputBound,\n    },\n    Set {\n        field: Option<String>,\n        elements: Vec<String>,\n    },\n    Exists {\n        field: String,\n    },\n    Regex {\n        field: Option<String>,\n        pattern: String,\n    },\n}\n\nimpl UserInputLeaf {\n    pub(crate) fn set_field(self, field: Option<String>) -> Self {\n        match self {\n            UserInputLeaf::Literal(mut literal) => {\n                literal.field_name = field;\n                UserInputLeaf::Literal(literal)\n            }\n            UserInputLeaf::All => UserInputLeaf::All,\n            UserInputLeaf::Range {\n                field: _,\n                lower,\n                upper,\n            } => UserInputLeaf::Range {\n                field,\n                lower,\n                upper,\n            },\n            UserInputLeaf::Set { field: _, elements } => UserInputLeaf::Set { field, elements },\n            UserInputLeaf::Exists { field: _ } => UserInputLeaf::Exists {\n                field: field.expect(\"Exist query without a field isn't allowed\"),\n            },\n            UserInputLeaf::Regex { field: _, pattern } => UserInputLeaf::Regex { field, pattern },\n        }\n    }\n\n    pub(crate) fn set_default_field(&mut self, default_field: String) {\n        match self {\n            UserInputLeaf::Literal(literal) if literal.field_name.is_none() => {\n                literal.field_name = Some(default_field)\n            }\n            UserInputLeaf::All => {\n                *self = UserInputLeaf::Exists {\n                    field: default_field,\n                }\n            }\n            UserInputLeaf::Range { field, .. } if field.is_none() => *field = Some(default_field),\n            UserInputLeaf::Set { field, .. } if field.is_none() => *field = Some(default_field),\n            UserInputLeaf::Regex { field, .. } if field.is_none() => *field = Some(default_field),\n            _ => (), // field was already set, do nothing\n        }\n    }\n}\n\nimpl Debug for UserInputLeaf {\n    fn fmt(&self, formatter: &mut Formatter) -> Result<(), fmt::Error> {\n        match self {\n            UserInputLeaf::Literal(literal) => literal.fmt(formatter),\n            UserInputLeaf::Range {\n                field,\n                lower,\n                upper,\n            } => {\n                if let Some(field) = field {\n                    // TODO properly escape field (in case of \\\")\n                    write!(formatter, \"\\\"{field}\\\":\")?;\n                }\n                lower.display_lower(formatter)?;\n                write!(formatter, \" TO \")?;\n                upper.display_upper(formatter)?;\n                Ok(())\n            }\n            UserInputLeaf::Set { field, elements } => {\n                if let Some(field) = field {\n                    // TODO properly escape field (in case of \\\")\n                    write!(formatter, \"\\\"{field}\\\": \")?;\n                }\n                write!(formatter, \"IN [\")?;\n                for (i, text) in elements.iter().enumerate() {\n                    if i != 0 {\n                        write!(formatter, \" \")?;\n                    }\n                    // TODO properly escape element\n                    write!(formatter, \"\\\"{text}\\\"\")?;\n                }\n                write!(formatter, \"]\")\n            }\n            UserInputLeaf::All => write!(formatter, \"*\"),\n            UserInputLeaf::Exists { field } => {\n                write!(formatter, \"$exists(\\\"{field}\\\")\")\n            }\n            UserInputLeaf::Regex { field, pattern } => {\n                if let Some(field) = field {\n                    // TODO properly escape field (in case of \\\")\n                    write!(formatter, \"\\\"{field}\\\":\")?;\n                }\n                // TODO properly escape pattern (in case of \\\")\n                write!(formatter, \"/{pattern}/\")\n            }\n        }\n    }\n}\n\n#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug, Serialize)]\n#[serde(rename_all = \"snake_case\")]\npub enum Delimiter {\n    SingleQuotes,\n    DoubleQuotes,\n    None,\n}\n\n#[derive(PartialEq, Eq, Hash, Clone, Serialize)]\n#[serde(rename_all = \"snake_case\")]\npub struct UserInputLiteral {\n    pub field_name: Option<String>,\n    pub phrase: String,\n    pub delimiter: Delimiter,\n    pub slop: u32,\n    pub prefix: bool,\n}\n\nimpl fmt::Debug for UserInputLiteral {\n    fn fmt(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> {\n        if let Some(ref field) = self.field_name {\n            // TODO properly escape field (in case of \\\")\n            write!(formatter, \"\\\"{field}\\\":\")?;\n        }\n        match self.delimiter {\n            Delimiter::SingleQuotes => {\n                // TODO properly escape element (in case of \\')\n                write!(formatter, \"'{}'\", self.phrase)?;\n            }\n            Delimiter::DoubleQuotes => {\n                // TODO properly escape element (in case of \\\")\n                write!(formatter, \"\\\"{}\\\"\", self.phrase)?;\n            }\n            Delimiter::None => {\n                // TODO properly escape element\n                write!(formatter, \"{}\", self.phrase)?;\n            }\n        }\n        if self.slop > 0 {\n            write!(formatter, \"~{}\", self.slop)?;\n        } else if self.prefix {\n            write!(formatter, \"*\")?;\n        }\n        Ok(())\n    }\n}\n\n#[derive(PartialEq, Eq, Hash, Debug, Clone, Serialize)]\n#[serde(tag = \"type\", content = \"value\")]\n#[serde(rename_all = \"snake_case\")]\npub enum UserInputBound {\n    Inclusive(String),\n    Exclusive(String),\n    Unbounded,\n}\n\nimpl UserInputBound {\n    fn display_lower(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> {\n        match *self {\n            // TODO properly escape word if required\n            UserInputBound::Inclusive(ref word) => write!(formatter, \"[\\\"{word}\\\"\"),\n            UserInputBound::Exclusive(ref word) => write!(formatter, \"{{\\\"{word}\\\"\"),\n            UserInputBound::Unbounded => write!(formatter, \"{{\\\"*\\\"\"),\n        }\n    }\n\n    fn display_upper(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> {\n        match *self {\n            // TODO properly escape word if required\n            UserInputBound::Inclusive(ref word) => write!(formatter, \"\\\"{word}\\\"]\"),\n            UserInputBound::Exclusive(ref word) => write!(formatter, \"\\\"{word}\\\"}}\"),\n            UserInputBound::Unbounded => write!(formatter, \"\\\"*\\\"}}\"),\n        }\n    }\n\n    pub fn term_str(&self) -> &str {\n        match *self {\n            UserInputBound::Inclusive(ref contents) => contents,\n            UserInputBound::Exclusive(ref contents) => contents,\n            UserInputBound::Unbounded => \"*\",\n        }\n    }\n}\n\n#[derive(PartialEq, Eq, Hash, Clone, Serialize)]\n#[serde(into = \"UserInputAstSerde\")]\npub enum UserInputAst {\n    Clause(Vec<(Option<Occur>, UserInputAst)>),\n    Boost(Box<UserInputAst>, ordered_float::OrderedFloat<f64>),\n    Leaf(Box<UserInputLeaf>),\n}\n\n#[derive(Serialize)]\n#[serde(tag = \"type\", rename_all = \"snake_case\")]\nenum UserInputAstSerde {\n    Bool {\n        clauses: Vec<(Option<Occur>, UserInputAst)>,\n    },\n    Boost {\n        underlying: Box<UserInputAst>,\n        boost: f64,\n    },\n    #[serde(untagged)]\n    Leaf(Box<UserInputLeaf>),\n}\n\nimpl From<UserInputAst> for UserInputAstSerde {\n    fn from(ast: UserInputAst) -> Self {\n        match ast {\n            UserInputAst::Clause(clause) => UserInputAstSerde::Bool { clauses: clause },\n            UserInputAst::Boost(underlying, boost) => UserInputAstSerde::Boost {\n                underlying,\n                boost: boost.into_inner(),\n            },\n            UserInputAst::Leaf(leaf) => UserInputAstSerde::Leaf(leaf),\n        }\n    }\n}\n\nimpl UserInputAst {\n    #[must_use]\n    pub fn unary(self, occur: Occur) -> UserInputAst {\n        UserInputAst::Clause(vec![(Some(occur), self)])\n    }\n\n    fn compose(occur: Occur, asts: Vec<UserInputAst>) -> UserInputAst {\n        assert_ne!(occur, Occur::MustNot);\n        assert!(!asts.is_empty());\n        if asts.len() == 1 {\n            asts.into_iter().next().unwrap() //< safe\n        } else {\n            UserInputAst::Clause(\n                asts.into_iter()\n                    .map(|ast: UserInputAst| (Some(occur), ast))\n                    .collect::<Vec<_>>(),\n            )\n        }\n    }\n\n    pub fn empty_query() -> UserInputAst {\n        UserInputAst::Clause(Vec::default())\n    }\n\n    pub fn and(asts: Vec<UserInputAst>) -> UserInputAst {\n        UserInputAst::compose(Occur::Must, asts)\n    }\n\n    pub fn or(asts: Vec<UserInputAst>) -> UserInputAst {\n        UserInputAst::compose(Occur::Should, asts)\n    }\n\n    pub(crate) fn set_default_field(&mut self, field: String) {\n        match self {\n            UserInputAst::Clause(clauses) => clauses\n                .iter_mut()\n                .for_each(|(_, ast)| ast.set_default_field(field.clone())),\n            UserInputAst::Leaf(leaf) => leaf.set_default_field(field),\n            UserInputAst::Boost(ast, _) => ast.set_default_field(field),\n        }\n    }\n}\n\nimpl From<UserInputLiteral> for UserInputLeaf {\n    fn from(literal: UserInputLiteral) -> UserInputLeaf {\n        UserInputLeaf::Literal(literal)\n    }\n}\n\nimpl From<UserInputLeaf> for UserInputAst {\n    fn from(leaf: UserInputLeaf) -> UserInputAst {\n        UserInputAst::Leaf(Box::new(leaf))\n    }\n}\n\nfn print_occur_ast(\n    occur_opt: Option<Occur>,\n    ast: &UserInputAst,\n    formatter: &mut fmt::Formatter,\n) -> fmt::Result {\n    if let Some(occur) = occur_opt {\n        write!(formatter, \"{occur}{ast:?}\")?;\n    } else {\n        write!(formatter, \"*{ast:?}\")?;\n    }\n    Ok(())\n}\n\nimpl fmt::Debug for UserInputAst {\n    fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n        match *self {\n            UserInputAst::Clause(ref subqueries) => {\n                if subqueries.is_empty() {\n                    // TODO this will break ast reserialization, is writing \"( )\" enough?\n                    write!(formatter, \"<emptyclause>\")?;\n                } else {\n                    write!(formatter, \"(\")?;\n                    print_occur_ast(subqueries[0].0, &subqueries[0].1, formatter)?;\n                    for subquery in &subqueries[1..] {\n                        write!(formatter, \" \")?;\n                        print_occur_ast(subquery.0, &subquery.1, formatter)?;\n                    }\n                    write!(formatter, \")\")?;\n                }\n                Ok(())\n            }\n            UserInputAst::Leaf(ref subquery) => write!(formatter, \"{subquery:?}\"),\n            UserInputAst::Boost(ref leaf, boost) => write!(formatter, \"({leaf:?})^{boost}\"),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_all_leaf_serialization() {\n        let ast = UserInputAst::Leaf(Box::new(UserInputLeaf::All));\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(json, r#\"{\"type\":\"all\"}\"#);\n    }\n\n    #[test]\n    fn test_literal_leaf_serialization() {\n        let literal = UserInputLiteral {\n            field_name: Some(\"title\".to_string()),\n            phrase: \"hello\".to_string(),\n            delimiter: Delimiter::None,\n            slop: 0,\n            prefix: false,\n        };\n        let ast = UserInputAst::Leaf(Box::new(UserInputLeaf::Literal(literal)));\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"literal\",\"field_name\":\"title\",\"phrase\":\"hello\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}\"#\n        );\n    }\n\n    #[test]\n    fn test_range_leaf_serialization() {\n        let range = UserInputLeaf::Range {\n            field: Some(\"price\".to_string()),\n            lower: UserInputBound::Inclusive(\"10\".to_string()),\n            upper: UserInputBound::Exclusive(\"100\".to_string()),\n        };\n        let ast = UserInputAst::Leaf(Box::new(range));\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"range\",\"field\":\"price\",\"lower\":{\"type\":\"inclusive\",\"value\":\"10\"},\"upper\":{\"type\":\"exclusive\",\"value\":\"100\"}}\"#\n        );\n    }\n\n    #[test]\n    fn test_range_leaf_unbounded_serialization() {\n        let range = UserInputLeaf::Range {\n            field: Some(\"price\".to_string()),\n            lower: UserInputBound::Inclusive(\"10\".to_string()),\n            upper: UserInputBound::Unbounded,\n        };\n        let ast = UserInputAst::Leaf(Box::new(range));\n        let json = serde_json::to_string(&ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"range\",\"field\":\"price\",\"lower\":{\"type\":\"inclusive\",\"value\":\"10\"},\"upper\":{\"type\":\"unbounded\"}}\"#\n        );\n    }\n\n    #[test]\n    fn test_boost_serialization() {\n        let inner_ast = UserInputAst::Leaf(Box::new(UserInputLeaf::All));\n        let boost_ast = UserInputAst::Boost(Box::new(inner_ast), 2.5.into());\n        let json = serde_json::to_string(&boost_ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"boost\",\"underlying\":{\"type\":\"all\"},\"boost\":2.5}\"#\n        );\n    }\n\n    #[test]\n    fn test_boost_serialization2() {\n        let boost_ast = UserInputAst::Boost(\n            Box::new(UserInputAst::Clause(vec![\n                (\n                    Some(Occur::Must),\n                    UserInputAst::Leaf(Box::new(UserInputLeaf::All)),\n                ),\n                (\n                    Some(Occur::Should),\n                    UserInputAst::Leaf(Box::new(UserInputLeaf::Literal(UserInputLiteral {\n                        field_name: Some(\"title\".to_string()),\n                        phrase: \"hello\".to_string(),\n                        delimiter: Delimiter::None,\n                        slop: 0,\n                        prefix: false,\n                    }))),\n                ),\n            ])),\n            2.5.into(),\n        );\n        let json = serde_json::to_string(&boost_ast).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"boost\",\"underlying\":{\"type\":\"bool\",\"clauses\":[[\"must\",{\"type\":\"all\"}],[\"should\",{\"type\":\"literal\",\"field_name\":\"title\",\"phrase\":\"hello\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}]]},\"boost\":2.5}\"#\n        );\n    }\n\n    #[test]\n    fn test_clause_serialization() {\n        let clause = UserInputAst::Clause(vec![\n            (\n                Some(Occur::Must),\n                UserInputAst::Leaf(Box::new(UserInputLeaf::All)),\n            ),\n            (\n                Some(Occur::Should),\n                UserInputAst::Leaf(Box::new(UserInputLeaf::Literal(UserInputLiteral {\n                    field_name: Some(\"title\".to_string()),\n                    phrase: \"hello\".to_string(),\n                    delimiter: Delimiter::None,\n                    slop: 0,\n                    prefix: false,\n                }))),\n            ),\n        ]);\n        let json = serde_json::to_string(&clause).unwrap();\n        assert_eq!(\n            json,\n            r#\"{\"type\":\"bool\",\"clauses\":[[\"must\",{\"type\":\"all\"}],[\"should\",{\"type\":\"literal\",\"field_name\":\"title\",\"phrase\":\"hello\",\"delimiter\":\"none\",\"slop\":0,\"prefix\":false}]]}\"#\n        );\n    }\n}\n"
  },
  {
    "path": "rustfmt.toml",
    "content": "comment_width = 120\nformat_strings = true\ngroup_imports = \"StdExternalCrate\"\nimports_granularity = \"Module\"\nnormalize_comments = true\nwhere_single_line = true\nwrap_comments = true\n"
  },
  {
    "path": "src/aggregation/README.md",
    "content": "# Contributing\n\nWhen adding new bucket aggregation make sure to extend the \"test_aggregation_flushing\" test for at least 2 levels.\n\n\n\n# Code Organization\n\nTantivy's aggregations have been designed to mimic the \n[aggregations of elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html).\n\nThe code is organized in submodules:\n\n## bucket\nContains all bucket aggregations, like range aggregation. These bucket aggregations group documents into buckets and can contain sub-aggregations.\n\n## metric\nContains all metric aggregations, like average aggregation. Metric aggregations do not have sub aggregations.\n\n#### agg_req\nagg_req contains the users aggregation request. Deserialization from json is compatible with elasticsearch aggregation requests.\n\n#### agg_data\nagg_data contains the users aggregation request enriched with fast field accessors etc, which are\nused during collection.\n\n#### segment_agg_result\nsegment_agg_result contains the aggregation result tree, which is used for collection of a segment.\nagg_data is passed during collection.\n\n#### intermediate_agg_result\nintermediate_agg_result contains the aggregation tree for merging with other trees.\n\n#### agg_result\nagg_result contains the final aggregation tree.\n"
  },
  {
    "path": "src/aggregation/accessor_helpers.rs",
    "content": "//! This will enhance the request tree with access to the fastfield and metadata.\n\nuse std::io;\n\nuse columnar::{Column, ColumnType};\n\nuse crate::aggregation::{f64_to_fastfield_u64, Key};\nuse crate::index::SegmentReader;\n\n/// Get the missing value as internal u64 representation\n///\n/// For terms we use u64::MAX as sentinel value\n/// For numerical data we convert the value into the representation\n/// we would get from the fast field, when we open it as u64_lenient_for_type.\n///\n/// That way we can use it the same way as if it would come from the fastfield.\npub(crate) fn get_missing_val_as_u64_lenient(\n    column_type: ColumnType,\n    column_max_value: u64,\n    missing: &Key,\n    field_name: &str,\n) -> crate::Result<Option<u64>> {\n    let missing_val = match missing {\n        Key::Str(_) if column_type == ColumnType::Str => Some(column_max_value + 1),\n        // Allow fallback to number on text fields\n        Key::F64(_) if column_type == ColumnType::Str => Some(column_max_value + 1),\n        Key::U64(_) if column_type == ColumnType::Str => Some(column_max_value + 1),\n        Key::I64(_) if column_type == ColumnType::Str => Some(column_max_value + 1),\n        Key::F64(val) if column_type.numerical_type().is_some() => {\n            f64_to_fastfield_u64(*val, &column_type)\n        }\n        // NOTE: We may loose precision of the passed missing value by casting i64 and u64 to f64.\n        Key::I64(val) if column_type.numerical_type().is_some() => {\n            f64_to_fastfield_u64(*val as f64, &column_type)\n        }\n        Key::U64(val) if column_type.numerical_type().is_some() => {\n            f64_to_fastfield_u64(*val as f64, &column_type)\n        }\n        _ => {\n            return Err(crate::TantivyError::InvalidArgument(format!(\n                \"Missing value {missing:?} for field {field_name} is not supported for column \\\n                 type {column_type:?}\"\n            )));\n        }\n    };\n    Ok(missing_val)\n}\n\npub(crate) fn get_numeric_or_date_column_types() -> &'static [ColumnType] {\n    &[\n        ColumnType::F64,\n        ColumnType::U64,\n        ColumnType::I64,\n        ColumnType::DateTime,\n    ]\n}\n\n/// Get fast field reader or empty as default.\npub(crate) fn get_ff_reader(\n    reader: &SegmentReader,\n    field_name: &str,\n    allowed_column_types: Option<&[ColumnType]>,\n) -> crate::Result<(columnar::Column<u64>, ColumnType)> {\n    let ff_fields = reader.fast_fields();\n    let ff_field_with_type = ff_fields\n        .u64_lenient_for_type(allowed_column_types, field_name)?\n        .unwrap_or_else(|| {\n            (\n                Column::build_empty_column(reader.num_docs()),\n                ColumnType::U64,\n            )\n        });\n    Ok(ff_field_with_type)\n}\n\npub(crate) fn get_dynamic_columns(\n    reader: &SegmentReader,\n    field_name: &str,\n) -> crate::Result<Vec<columnar::DynamicColumn>> {\n    let ff_fields = reader.fast_fields().dynamic_column_handles(field_name)?;\n    let cols = ff_fields\n        .iter()\n        .map(|h| h.open())\n        .collect::<io::Result<_>>()?;\n    assert!(!ff_fields.is_empty(), \"field {field_name} not found\");\n    Ok(cols)\n}\n\n/// Get all fast field reader or empty as default.\n///\n/// Is guaranteed to return at least one column.\npub(crate) fn get_all_ff_reader_or_empty(\n    reader: &SegmentReader,\n    field_name: &str,\n    allowed_column_types: Option<&[ColumnType]>,\n    fallback_type: ColumnType,\n) -> crate::Result<Vec<(columnar::Column<u64>, ColumnType)>> {\n    let ff_fields = reader.fast_fields();\n    let mut ff_field_with_type =\n        ff_fields.u64_lenient_for_type_all(allowed_column_types, field_name)?;\n    if ff_field_with_type.is_empty() {\n        ff_field_with_type.push((Column::build_empty_column(reader.num_docs()), fallback_type));\n    }\n    Ok(ff_field_with_type)\n}\n"
  },
  {
    "path": "src/aggregation/agg_data.rs",
    "content": "use columnar::{Column, ColumnBlockAccessor, ColumnType, StrColumn};\nuse common::BitSet;\nuse rustc_hash::FxHashSet;\nuse serde::Serialize;\nuse tantivy_fst::Regex;\n\nuse crate::aggregation::accessor_helpers::{\n    get_all_ff_reader_or_empty, get_dynamic_columns, get_ff_reader, get_missing_val_as_u64_lenient,\n    get_numeric_or_date_column_types,\n};\nuse crate::aggregation::agg_req::{Aggregation, AggregationVariants, Aggregations};\nuse crate::aggregation::bucket::{\n    build_segment_filter_collector, build_segment_range_collector, CompositeAggReqData,\n    CompositeAggregation, CompositeSourceAccessors, FilterAggReqData, HistogramAggReqData,\n    HistogramBounds, IncludeExcludeParam, MissingTermAggReqData, RangeAggReqData,\n    SegmentHistogramCollector, TermMissingAgg, TermsAggReqData, TermsAggregation,\n    TermsAggregationInternal,\n};\nuse crate::aggregation::metric::{\n    build_segment_stats_collector, AverageAggregation, CardinalityAggReqData,\n    CardinalityAggregationReq, CountAggregation, ExtendedStatsAggregation, MaxAggregation,\n    MetricAggReqData, MinAggregation, SegmentCardinalityCollector, SegmentExtendedStatsCollector,\n    SegmentPercentilesCollector, StatsAggregation, StatsType, SumAggregation, TopHitsAggReqData,\n    TopHitsSegmentCollector,\n};\nuse crate::aggregation::segment_agg_result::{\n    GenericSegmentAggregationResultsCollector, SegmentAggregationCollector,\n};\nuse crate::aggregation::{f64_to_fastfield_u64, AggContextParams, Key};\nuse crate::{SegmentOrdinal, SegmentReader};\n\n#[derive(Default)]\n/// Datastructure holding all request data for executing aggregations on a segment.\n/// It is passed to the collectors during collection.\npub struct AggregationsSegmentCtx {\n    /// Request data for each aggregation type.\n    pub per_request: PerRequestAggSegCtx,\n    pub context: AggContextParams,\n    pub column_block_accessor: ColumnBlockAccessor<u64>,\n}\n\nimpl AggregationsSegmentCtx {\n    pub(crate) fn push_term_req_data(&mut self, data: TermsAggReqData) -> usize {\n        self.per_request.term_req_data.push(Some(Box::new(data)));\n        self.per_request.term_req_data.len() - 1\n    }\n    pub(crate) fn push_cardinality_req_data(&mut self, data: CardinalityAggReqData) -> usize {\n        self.per_request.cardinality_req_data.push(data);\n        self.per_request.cardinality_req_data.len() - 1\n    }\n    pub(crate) fn push_metric_req_data(&mut self, data: MetricAggReqData) -> usize {\n        self.per_request.stats_metric_req_data.push(data);\n        self.per_request.stats_metric_req_data.len() - 1\n    }\n    pub(crate) fn push_top_hits_req_data(&mut self, data: TopHitsAggReqData) -> usize {\n        self.per_request.top_hits_req_data.push(data);\n        self.per_request.top_hits_req_data.len() - 1\n    }\n    pub(crate) fn push_missing_term_req_data(&mut self, data: MissingTermAggReqData) -> usize {\n        self.per_request.missing_term_req_data.push(data);\n        self.per_request.missing_term_req_data.len() - 1\n    }\n    pub(crate) fn push_histogram_req_data(&mut self, data: HistogramAggReqData) -> usize {\n        self.per_request\n            .histogram_req_data\n            .push(Some(Box::new(data)));\n        self.per_request.histogram_req_data.len() - 1\n    }\n    pub(crate) fn push_range_req_data(&mut self, data: RangeAggReqData) -> usize {\n        self.per_request.range_req_data.push(Some(Box::new(data)));\n        self.per_request.range_req_data.len() - 1\n    }\n    pub(crate) fn push_filter_req_data(&mut self, data: FilterAggReqData) -> usize {\n        self.per_request.filter_req_data.push(Some(Box::new(data)));\n        self.per_request.filter_req_data.len() - 1\n    }\n    pub(crate) fn push_composite_req_data(&mut self, data: CompositeAggReqData) -> usize {\n        self.per_request\n            .composite_req_data\n            .push(Some(Box::new(data)));\n        self.per_request.composite_req_data.len() - 1\n    }\n\n    #[inline]\n    pub(crate) fn get_term_req_data(&self, idx: usize) -> &TermsAggReqData {\n        self.per_request.term_req_data[idx]\n            .as_deref()\n            .expect(\"term_req_data slot is empty (taken)\")\n    }\n    #[inline]\n    pub(crate) fn get_cardinality_req_data(&self, idx: usize) -> &CardinalityAggReqData {\n        &self.per_request.cardinality_req_data[idx]\n    }\n    #[inline]\n    pub(crate) fn get_metric_req_data(&self, idx: usize) -> &MetricAggReqData {\n        &self.per_request.stats_metric_req_data[idx]\n    }\n    #[inline]\n    pub(crate) fn get_top_hits_req_data(&self, idx: usize) -> &TopHitsAggReqData {\n        &self.per_request.top_hits_req_data[idx]\n    }\n    #[inline]\n    pub(crate) fn get_missing_term_req_data(&self, idx: usize) -> &MissingTermAggReqData {\n        &self.per_request.missing_term_req_data[idx]\n    }\n    #[inline]\n    pub(crate) fn get_histogram_req_data(&self, idx: usize) -> &HistogramAggReqData {\n        self.per_request.histogram_req_data[idx]\n            .as_deref()\n            .expect(\"histogram_req_data slot is empty (taken)\")\n    }\n    #[inline]\n    pub(crate) fn get_range_req_data(&self, idx: usize) -> &RangeAggReqData {\n        self.per_request.range_req_data[idx]\n            .as_deref()\n            .expect(\"range_req_data slot is empty (taken)\")\n    }\n    #[inline]\n    pub(crate) fn get_composite_req_data(&self, idx: usize) -> &CompositeAggReqData {\n        self.per_request.composite_req_data[idx]\n            .as_deref()\n            .expect(\"composite_req_data slot is empty (taken)\")\n    }\n\n    // ---------- mutable getters ----------\n\n    #[inline]\n    pub(crate) fn get_metric_req_data_mut(&mut self, idx: usize) -> &mut MetricAggReqData {\n        &mut self.per_request.stats_metric_req_data[idx]\n    }\n\n    #[inline]\n    pub(crate) fn get_cardinality_req_data_mut(\n        &mut self,\n        idx: usize,\n    ) -> &mut CardinalityAggReqData {\n        &mut self.per_request.cardinality_req_data[idx]\n    }\n\n    #[inline]\n    pub(crate) fn get_histogram_req_data_mut(&mut self, idx: usize) -> &mut HistogramAggReqData {\n        self.per_request.histogram_req_data[idx]\n            .as_deref_mut()\n            .expect(\"histogram_req_data slot is empty (taken)\")\n    }\n\n    // ---------- take / put (terms, histogram, range) ----------\n\n    /// Move out the boxed Histogram request at `idx`, leaving `None`.\n    #[inline]\n    pub(crate) fn take_histogram_req_data(&mut self, idx: usize) -> Box<HistogramAggReqData> {\n        self.per_request.histogram_req_data[idx]\n            .take()\n            .expect(\"histogram_req_data slot is empty (taken)\")\n    }\n\n    /// Put back a Histogram request into an empty slot at `idx`.\n    #[inline]\n    pub(crate) fn put_back_histogram_req_data(\n        &mut self,\n        idx: usize,\n        value: Box<HistogramAggReqData>,\n    ) {\n        debug_assert!(self.per_request.histogram_req_data[idx].is_none());\n        self.per_request.histogram_req_data[idx] = Some(value);\n    }\n\n    /// Move out the boxed Range request at `idx`, leaving `None`.\n    #[inline]\n    pub(crate) fn take_range_req_data(&mut self, idx: usize) -> Box<RangeAggReqData> {\n        self.per_request.range_req_data[idx]\n            .take()\n            .expect(\"range_req_data slot is empty (taken)\")\n    }\n\n    /// Put back a Range request into an empty slot at `idx`.\n    #[inline]\n    pub(crate) fn put_back_range_req_data(&mut self, idx: usize, value: Box<RangeAggReqData>) {\n        debug_assert!(self.per_request.range_req_data[idx].is_none());\n        self.per_request.range_req_data[idx] = Some(value);\n    }\n\n    /// Move out the boxed Filter request at `idx`, leaving `None`.\n    #[inline]\n    pub(crate) fn take_filter_req_data(&mut self, idx: usize) -> Box<FilterAggReqData> {\n        self.per_request.filter_req_data[idx]\n            .take()\n            .expect(\"filter_req_data slot is empty (taken)\")\n    }\n\n    /// Put back a Filter request into an empty slot at `idx`.\n    #[inline]\n    pub(crate) fn put_back_filter_req_data(&mut self, idx: usize, value: Box<FilterAggReqData>) {\n        debug_assert!(self.per_request.filter_req_data[idx].is_none());\n        self.per_request.filter_req_data[idx] = Some(value);\n    }\n\n    /// Move out the Composite request at `idx`.\n    #[inline]\n    pub(crate) fn take_composite_req_data(&mut self, idx: usize) -> Box<CompositeAggReqData> {\n        self.per_request.composite_req_data[idx]\n            .take()\n            .expect(\"composite_req_data slot is empty (taken)\")\n    }\n\n    /// Put back a Composite request into an empty slot at `idx`.\n    #[inline]\n    pub(crate) fn put_back_composite_req_data(\n        &mut self,\n        idx: usize,\n        value: Box<CompositeAggReqData>,\n    ) {\n        debug_assert!(self.per_request.composite_req_data[idx].is_none());\n        self.per_request.composite_req_data[idx] = Some(value);\n    }\n}\n\n/// Each type of aggregation has its own request data struct. This struct holds\n/// all request data to execute the aggregation request on a single segment.\n///\n/// The request tree is represented by `agg_tree`. Tree nodes contain the index\n/// of their context in corresponding request data vector (e.g. `term_req_data`\n/// for a node with [AggKind::Terms]).\n#[derive(Default)]\npub struct PerRequestAggSegCtx {\n    // Box for cheap take/put - Only necessary for bucket aggs that have sub-aggregations\n    /// TermsAggReqData contains the request data for a terms aggregation.\n    pub term_req_data: Vec<Option<Box<TermsAggReqData>>>,\n    /// HistogramAggReqData contains the request data for a histogram aggregation.\n    pub histogram_req_data: Vec<Option<Box<HistogramAggReqData>>>,\n    /// RangeAggReqData contains the request data for a range aggregation.\n    pub range_req_data: Vec<Option<Box<RangeAggReqData>>>,\n    /// FilterAggReqData contains the request data for a filter aggregation.\n    pub filter_req_data: Vec<Option<Box<FilterAggReqData>>>,\n    /// Shared by avg, min, max, sum, stats, extended_stats, count\n    pub stats_metric_req_data: Vec<MetricAggReqData>,\n    /// CardinalityAggReqData contains the request data for a cardinality aggregation.\n    pub cardinality_req_data: Vec<CardinalityAggReqData>,\n    /// TopHitsAggReqData contains the request data for a top_hits aggregation.\n    pub top_hits_req_data: Vec<TopHitsAggReqData>,\n    /// MissingTermAggReqData contains the request data for a missing term aggregation.\n    pub missing_term_req_data: Vec<MissingTermAggReqData>,\n    /// CompositeAggReqData contains the request data for a composite aggregation.\n    pub composite_req_data: Vec<Option<Box<CompositeAggReqData>>>,\n\n    /// Request tree used to build collectors.\n    pub agg_tree: Vec<AggRefNode>,\n}\n\nimpl PerRequestAggSegCtx {\n    /// Estimate the memory consumption of this struct in bytes.\n    fn get_memory_consumption(&self) -> usize {\n        self.term_req_data\n            .iter()\n            .map(|b| b.as_ref().unwrap().get_memory_consumption())\n            .sum::<usize>()\n            + self\n                .histogram_req_data\n                .iter()\n                .map(|b| b.as_ref().unwrap().get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .range_req_data\n                .iter()\n                .map(|b| b.as_ref().unwrap().get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .filter_req_data\n                .iter()\n                .map(|b| b.as_ref().unwrap().get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .stats_metric_req_data\n                .iter()\n                .map(|t| t.get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .cardinality_req_data\n                .iter()\n                .map(|t| t.get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .top_hits_req_data\n                .iter()\n                .map(|t| t.get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .missing_term_req_data\n                .iter()\n                .map(|t| t.get_memory_consumption())\n                .sum::<usize>()\n            + self\n                .composite_req_data\n                .iter()\n                .map(|b| b.as_ref().map(|d| d.get_memory_consumption()).unwrap_or(0))\n                .sum::<usize>()\n            + self.agg_tree.len() * std::mem::size_of::<AggRefNode>()\n    }\n\n    pub fn get_name(&self, node: &AggRefNode) -> &str {\n        let idx = node.idx_in_req_data;\n        let kind = node.kind;\n        match kind {\n            AggKind::Terms => self.term_req_data[idx]\n                .as_deref()\n                .expect(\"term_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n            AggKind::Cardinality => &self.cardinality_req_data[idx].name,\n            AggKind::StatsKind(_) => &self.stats_metric_req_data[idx].name,\n            AggKind::TopHits => &self.top_hits_req_data[idx].name,\n            AggKind::MissingTerm => &self.missing_term_req_data[idx].name,\n            AggKind::Histogram => self.histogram_req_data[idx]\n                .as_deref()\n                .expect(\"histogram_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n            AggKind::DateHistogram => self.histogram_req_data[idx]\n                .as_deref()\n                .expect(\"histogram_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n            AggKind::Range => self.range_req_data[idx]\n                .as_deref()\n                .expect(\"range_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n            AggKind::Filter => self.filter_req_data[idx]\n                .as_deref()\n                .expect(\"filter_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n            AggKind::Composite => self.composite_req_data[idx]\n                .as_deref()\n                .expect(\"composite_req_data slot is empty (taken)\")\n                .name\n                .as_str(),\n        }\n    }\n\n    /// Convert the aggregation tree into a serializable struct representation.\n    /// Each node contains: { name, kind, children }.\n    #[allow(dead_code)]\n    pub fn get_view_tree(&self) -> Vec<AggTreeViewNode> {\n        fn node_to_view(node: &AggRefNode, pr: &PerRequestAggSegCtx) -> AggTreeViewNode {\n            let mut children: Vec<AggTreeViewNode> =\n                node.children.iter().map(|c| node_to_view(c, pr)).collect();\n            children.sort_by_key(|v| serde_json::to_string(v).unwrap());\n            AggTreeViewNode {\n                name: pr.get_name(node).to_string(),\n                kind: node.kind.as_str().to_string(),\n                children,\n            }\n        }\n\n        let mut roots: Vec<AggTreeViewNode> = self\n            .agg_tree\n            .iter()\n            .map(|n| node_to_view(n, self))\n            .collect();\n        roots.sort_by_key(|v| serde_json::to_string(v).unwrap());\n        roots\n    }\n}\n\npub(crate) fn build_segment_agg_collectors_root(\n    req: &mut AggregationsSegmentCtx,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    build_segment_agg_collectors_generic(req, &req.per_request.agg_tree.clone())\n}\n\npub(crate) fn build_segment_agg_collectors(\n    req: &mut AggregationsSegmentCtx,\n    nodes: &[AggRefNode],\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    build_segment_agg_collectors_generic(req, nodes)\n}\n\nfn build_segment_agg_collectors_generic(\n    req: &mut AggregationsSegmentCtx,\n    nodes: &[AggRefNode],\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    let mut collectors = Vec::new();\n    for node in nodes.iter() {\n        collectors.push(build_segment_agg_collector(req, node)?);\n    }\n\n    req.context\n        .limits\n        .add_memory_consumed(req.per_request.get_memory_consumption() as u64)?;\n    // Single collector special case\n    if collectors.len() == 1 {\n        return Ok(collectors.pop().unwrap());\n    }\n    let agg = GenericSegmentAggregationResultsCollector { aggs: collectors };\n    Ok(Box::new(agg))\n}\n\npub(crate) fn build_segment_agg_collector(\n    req: &mut AggregationsSegmentCtx,\n    node: &AggRefNode,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    match node.kind {\n        AggKind::Terms => crate::aggregation::bucket::build_segment_term_collector(req, node),\n        AggKind::MissingTerm => {\n            let req_data = &mut req.per_request.missing_term_req_data[node.idx_in_req_data];\n            if req_data.accessors.is_empty() {\n                return Err(crate::TantivyError::InternalError(\n                    \"MissingTerm aggregation requires at least one field accessor.\".to_string(),\n                ));\n            }\n            Ok(Box::new(TermMissingAgg::new(req, node)?))\n        }\n        AggKind::Cardinality => {\n            let req_data = &mut req.get_cardinality_req_data_mut(node.idx_in_req_data);\n            Ok(Box::new(SegmentCardinalityCollector::from_req(\n                req_data.column_type,\n                node.idx_in_req_data,\n                req_data.accessor.clone(),\n                req_data.missing_value_for_accessor,\n            )))\n        }\n        AggKind::StatsKind(stats_type) => {\n            let req_data = &mut req.per_request.stats_metric_req_data[node.idx_in_req_data];\n            match stats_type {\n                StatsType::Sum\n                | StatsType::Average\n                | StatsType::Count\n                | StatsType::Max\n                | StatsType::Min\n                | StatsType::Stats => build_segment_stats_collector(req_data),\n                StatsType::ExtendedStats(sigma) => Ok(Box::new(\n                    SegmentExtendedStatsCollector::from_req(req_data, sigma),\n                )),\n                StatsType::Percentiles => {\n                    let req_data = req.get_metric_req_data_mut(node.idx_in_req_data);\n                    Ok(Box::new(\n                        SegmentPercentilesCollector::from_req_and_validate(\n                            req_data.field_type,\n                            req_data.missing_u64,\n                            req_data.accessor.clone(),\n                            node.idx_in_req_data,\n                        ),\n                    ))\n                }\n            }\n        }\n        AggKind::TopHits => {\n            let req_data = &mut req.per_request.top_hits_req_data[node.idx_in_req_data];\n            Ok(Box::new(TopHitsSegmentCollector::from_req(\n                &req_data.req,\n                node.idx_in_req_data,\n                req_data.segment_ordinal,\n            )))\n        }\n        AggKind::Histogram => Ok(Box::new(SegmentHistogramCollector::from_req_and_validate(\n            req, node,\n        )?)),\n        AggKind::DateHistogram => Ok(Box::new(SegmentHistogramCollector::from_req_and_validate(\n            req, node,\n        )?)),\n        AggKind::Range => Ok(build_segment_range_collector(req, node)?),\n        AggKind::Filter => build_segment_filter_collector(req, node),\n        AggKind::Composite => Ok(Box::new(\n            crate::aggregation::bucket::SegmentCompositeCollector::from_req_and_validate(\n                req, node,\n            )?,\n        )),\n    }\n}\n\n/// See [PerRequestAggSegCtx]\n#[derive(Debug, Clone)]\npub struct AggRefNode {\n    pub kind: AggKind,\n    pub idx_in_req_data: usize,\n    pub children: Vec<AggRefNode>,\n}\nimpl AggRefNode {\n    pub fn get_sub_agg(&self, name: &str, pr: &PerRequestAggSegCtx) -> Option<&AggRefNode> {\n        self.children\n            .iter()\n            .find(|&child| pr.get_name(child) == name)\n    }\n}\n\n#[derive(Copy, Clone, Debug)]\npub enum AggKind {\n    Terms,\n    Cardinality,\n    /// One of: Statistics, Average, Min, Max, Sum, Count, Stats, ExtendedStats\n    StatsKind(StatsType),\n    TopHits,\n    MissingTerm,\n    Histogram,\n    DateHistogram,\n    Range,\n    Filter,\n    Composite,\n}\n\nimpl AggKind {\n    #[cfg_attr(not(test), allow(dead_code))]\n    fn as_str(&self) -> &'static str {\n        match self {\n            AggKind::Terms => \"Terms\",\n            AggKind::Cardinality => \"Cardinality\",\n            AggKind::StatsKind(_) => \"Metric\",\n            AggKind::TopHits => \"TopHits\",\n            AggKind::MissingTerm => \"MissingTerm\",\n            AggKind::Histogram => \"Histogram\",\n            AggKind::DateHistogram => \"DateHistogram\",\n            AggKind::Range => \"Range\",\n            AggKind::Filter => \"Filter\",\n            AggKind::Composite => \"Composite\",\n        }\n    }\n}\n\n/// Build AggregationsData by walking the request tree.\npub(crate) fn build_aggregations_data_from_req(\n    aggs: &Aggregations,\n    reader: &SegmentReader,\n    segment_ordinal: SegmentOrdinal,\n    context: AggContextParams,\n) -> crate::Result<AggregationsSegmentCtx> {\n    let mut data = AggregationsSegmentCtx {\n        per_request: Default::default(),\n        context,\n        column_block_accessor: ColumnBlockAccessor::default(),\n    };\n\n    for (name, agg) in aggs.iter() {\n        let nodes = build_nodes(name, agg, reader, segment_ordinal, &mut data, true)?;\n        data.per_request.agg_tree.extend(nodes);\n    }\n    Ok(data)\n}\n\nfn build_nodes(\n    agg_name: &str,\n    req: &Aggregation,\n    reader: &SegmentReader,\n    segment_ordinal: SegmentOrdinal,\n    data: &mut AggregationsSegmentCtx,\n    is_top_level: bool,\n) -> crate::Result<Vec<AggRefNode>> {\n    use AggregationVariants::*;\n    match &req.agg {\n        Range(range_req) => {\n            let (accessor, field_type) = get_ff_reader(\n                reader,\n                &range_req.field,\n                Some(get_numeric_or_date_column_types()),\n            )?;\n            let idx_in_req_data = data.push_range_req_data(RangeAggReqData {\n                accessor,\n                field_type,\n                name: agg_name.to_string(),\n                req: range_req.clone(),\n                is_top_level,\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::Range,\n                idx_in_req_data,\n                children,\n            }])\n        }\n        Histogram(histo_req) => {\n            let (accessor, field_type) = get_ff_reader(\n                reader,\n                &histo_req.field,\n                Some(get_numeric_or_date_column_types()),\n            )?;\n            let idx_in_req_data = data.push_histogram_req_data(HistogramAggReqData {\n                accessor,\n                field_type,\n                name: agg_name.to_string(),\n                req: histo_req.clone(),\n                is_date_histogram: false,\n                bounds: HistogramBounds {\n                    min: f64::MIN,\n                    max: f64::MAX,\n                },\n                offset: 0.0,\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::Histogram,\n                idx_in_req_data,\n                children,\n            }])\n        }\n        DateHistogram(date_req) => {\n            let (accessor, field_type) =\n                get_ff_reader(reader, &date_req.field, Some(&[ColumnType::DateTime]))?;\n            // Convert to histogram request, normalize to ns precision\n            let mut histo_req = date_req.to_histogram_req()?;\n            histo_req.normalize_date_time();\n            let idx_in_req_data = data.push_histogram_req_data(HistogramAggReqData {\n                accessor,\n                field_type,\n                name: agg_name.to_string(),\n                req: histo_req,\n                is_date_histogram: true,\n                bounds: HistogramBounds {\n                    min: f64::MIN,\n                    max: f64::MAX,\n                },\n                offset: 0.0,\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::DateHistogram,\n                idx_in_req_data,\n                children,\n            }])\n        }\n        Terms(terms_req) => build_terms_or_cardinality_nodes(\n            agg_name,\n            &terms_req.field,\n            &terms_req.missing,\n            reader,\n            segment_ordinal,\n            data,\n            &req.sub_aggregation,\n            TermsOrCardinalityRequest::Terms(terms_req.clone()),\n            is_top_level,\n        ),\n        Cardinality(card_req) => build_terms_or_cardinality_nodes(\n            agg_name,\n            &card_req.field,\n            &card_req.missing,\n            reader,\n            segment_ordinal,\n            data,\n            &req.sub_aggregation,\n            TermsOrCardinalityRequest::Cardinality(card_req.clone()),\n            is_top_level,\n        ),\n        Average(AverageAggregation { field, missing, .. })\n        | Max(MaxAggregation { field, missing, .. })\n        | Min(MinAggregation { field, missing, .. })\n        | Stats(StatsAggregation { field, missing, .. })\n        | ExtendedStats(ExtendedStatsAggregation { field, missing, .. })\n        | Sum(SumAggregation { field, missing, .. })\n        | Count(CountAggregation { field, missing, .. }) => {\n            let allowed_column_types = if matches!(&req.agg, Count(_)) {\n                Some(\n                    &[\n                        ColumnType::I64,\n                        ColumnType::U64,\n                        ColumnType::F64,\n                        ColumnType::Str,\n                        ColumnType::DateTime,\n                        ColumnType::Bool,\n                        ColumnType::IpAddr,\n                    ][..],\n                )\n            } else {\n                Some(get_numeric_or_date_column_types())\n            };\n            let collecting_for = match &req.agg {\n                Average(_) => StatsType::Average,\n                Max(_) => StatsType::Max,\n                Min(_) => StatsType::Min,\n                Stats(_) => StatsType::Stats,\n                ExtendedStats(req) => StatsType::ExtendedStats(req.sigma),\n                Sum(_) => StatsType::Sum,\n                Count(_) => StatsType::Count,\n                _ => {\n                    return Err(crate::TantivyError::InvalidArgument(\n                        \"Internal error: unexpected aggregation type in metric aggregation \\\n                         handling.\"\n                            .to_string(),\n                    ))\n                }\n            };\n            let (accessor, field_type) = get_ff_reader(reader, field, allowed_column_types)?;\n            let idx_in_req_data = data.push_metric_req_data(MetricAggReqData {\n                accessor,\n                field_type,\n                name: agg_name.to_string(),\n                collecting_for,\n                missing: *missing,\n                missing_u64: (*missing).and_then(|m| f64_to_fastfield_u64(m, &field_type)),\n                is_number_or_date_type: matches!(\n                    field_type,\n                    ColumnType::I64 | ColumnType::U64 | ColumnType::F64 | ColumnType::DateTime\n                ),\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::StatsKind(collecting_for),\n                idx_in_req_data,\n                children,\n            }])\n        }\n        // Percentiles handled as Metric as well\n        AggregationVariants::Percentiles(percentiles_req) => {\n            percentiles_req.validate()?;\n            let (accessor, field_type) = get_ff_reader(\n                reader,\n                percentiles_req.field_name(),\n                Some(get_numeric_or_date_column_types()),\n            )?;\n            let idx_in_req_data = data.push_metric_req_data(MetricAggReqData {\n                accessor,\n                field_type,\n                name: agg_name.to_string(),\n                collecting_for: StatsType::Percentiles,\n                missing: percentiles_req.missing,\n                missing_u64: percentiles_req\n                    .missing\n                    .and_then(|m| f64_to_fastfield_u64(m, &field_type)),\n                is_number_or_date_type: matches!(\n                    field_type,\n                    ColumnType::I64 | ColumnType::U64 | ColumnType::F64 | ColumnType::DateTime\n                ),\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::StatsKind(StatsType::Percentiles),\n                idx_in_req_data,\n                children,\n            }])\n        }\n        AggregationVariants::TopHits(top_hits_req) => {\n            let mut top_hits = top_hits_req.clone();\n            top_hits.validate_and_resolve_field_names(reader.fast_fields().columnar())?;\n            let accessors: Vec<(Column<u64>, ColumnType)> = top_hits\n                .field_names()\n                .iter()\n                .map(|field| get_ff_reader(reader, field, Some(get_numeric_or_date_column_types())))\n                .collect::<crate::Result<_>>()?;\n\n            let value_accessors = top_hits\n                .value_field_names()\n                .iter()\n                .map(|field_name| {\n                    Ok((\n                        field_name.to_string(),\n                        get_dynamic_columns(reader, field_name)?,\n                    ))\n                })\n                .collect::<crate::Result<_>>()?;\n\n            let idx_in_req_data = data.push_top_hits_req_data(TopHitsAggReqData {\n                accessors,\n                value_accessors,\n                segment_ordinal,\n                name: agg_name.to_string(),\n                req: top_hits.clone(),\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::TopHits,\n                idx_in_req_data,\n                children,\n            }])\n        }\n        AggregationVariants::Composite(composite_req) => Ok(vec![build_composite_node(\n            agg_name,\n            reader,\n            segment_ordinal,\n            data,\n            &req.sub_aggregation,\n            composite_req,\n        )?]),\n        AggregationVariants::Filter(filter_req) => {\n            // Build the query and evaluator upfront\n            let schema = reader.schema();\n            let tokenizers = &data.context.tokenizers;\n            let query = filter_req.parse_query(schema, tokenizers)?;\n            let evaluator = crate::aggregation::bucket::DocumentQueryEvaluator::new(\n                query,\n                schema.clone(),\n                reader,\n            )?;\n\n            // Pre-allocate buffer for batch filtering\n            let max_doc = reader.max_doc();\n            let buffer_capacity = crate::docset::COLLECT_BLOCK_BUFFER_LEN.min(max_doc as usize);\n            let matching_docs_buffer = Vec::with_capacity(buffer_capacity);\n\n            let idx_in_req_data = data.push_filter_req_data(FilterAggReqData {\n                name: agg_name.to_string(),\n                req: filter_req.clone(),\n                segment_reader: reader.clone(),\n                evaluator,\n                matching_docs_buffer,\n                is_top_level,\n            });\n            let children = build_children(&req.sub_aggregation, reader, segment_ordinal, data)?;\n            Ok(vec![AggRefNode {\n                kind: AggKind::Filter,\n                idx_in_req_data,\n                children,\n            }])\n        }\n    }\n}\n\nfn build_composite_node(\n    agg_name: &str,\n    reader: &SegmentReader,\n    _segment_ordinal: SegmentOrdinal,\n    data: &mut AggregationsSegmentCtx,\n    sub_aggs: &Aggregations,\n    req: &CompositeAggregation,\n) -> crate::Result<AggRefNode> {\n    let mut composite_accessors = Vec::with_capacity(req.sources.len());\n    for source in &req.sources {\n        let source_after_key_opt = req.after.get(source.name()).map(|k| &k.0);\n        let source_accessor =\n            CompositeSourceAccessors::build_for_source(reader, source, source_after_key_opt)?;\n        composite_accessors.push(source_accessor);\n    }\n    let agg = CompositeAggReqData {\n        name: agg_name.to_string(),\n        req: req.clone(),\n        composite_accessors,\n    };\n    let idx = data.push_composite_req_data(agg);\n    let children = build_children(sub_aggs, reader, _segment_ordinal, data)?;\n    Ok(AggRefNode {\n        kind: AggKind::Composite,\n        idx_in_req_data: idx,\n        children,\n    })\n}\n\nfn build_children(\n    aggs: &Aggregations,\n    reader: &SegmentReader,\n    segment_ordinal: SegmentOrdinal,\n    data: &mut AggregationsSegmentCtx,\n) -> crate::Result<Vec<AggRefNode>> {\n    let mut children = Vec::new();\n    for (name, agg) in aggs.iter() {\n        children.extend(build_nodes(\n            name,\n            agg,\n            reader,\n            segment_ordinal,\n            data,\n            false,\n        )?);\n    }\n    Ok(children)\n}\n\nfn get_term_agg_accessors(\n    reader: &SegmentReader,\n    field_name: &str,\n    missing: &Option<Key>,\n) -> crate::Result<Vec<(Column<u64>, ColumnType)>> {\n    let allowed_column_types = [\n        ColumnType::I64,\n        ColumnType::U64,\n        ColumnType::F64,\n        ColumnType::Str,\n        ColumnType::DateTime,\n        ColumnType::Bool,\n        ColumnType::IpAddr,\n    ];\n\n    // In case the column is empty we want the shim column to match the missing type\n    let fallback_type = missing\n        .as_ref()\n        .map(|missing| match missing {\n            Key::Str(_) => ColumnType::Str,\n            Key::F64(_) => ColumnType::F64,\n            Key::I64(_) => ColumnType::I64,\n            Key::U64(_) => ColumnType::U64,\n        })\n        .unwrap_or(ColumnType::U64);\n\n    let column_and_types = get_all_ff_reader_or_empty(\n        reader,\n        field_name,\n        Some(&allowed_column_types),\n        fallback_type,\n    )?;\n\n    Ok(column_and_types)\n}\n\nenum TermsOrCardinalityRequest {\n    Terms(TermsAggregation),\n    Cardinality(CardinalityAggregationReq),\n}\nimpl TermsOrCardinalityRequest {\n    fn as_terms(&self) -> Option<&TermsAggregation> {\n        match self {\n            TermsOrCardinalityRequest::Terms(t) => Some(t),\n            _ => None,\n        }\n    }\n}\n\n#[allow(clippy::too_many_arguments)]\nfn build_terms_or_cardinality_nodes(\n    agg_name: &str,\n    field_name: &str,\n    missing: &Option<Key>,\n    reader: &SegmentReader,\n    segment_ordinal: SegmentOrdinal,\n    data: &mut AggregationsSegmentCtx,\n    sub_aggs: &Aggregations,\n    req: TermsOrCardinalityRequest,\n    is_top_level: bool,\n) -> crate::Result<Vec<AggRefNode>> {\n    let mut nodes = Vec::new();\n\n    let str_dict_column = reader.fast_fields().str(field_name)?;\n\n    let column_and_types = get_term_agg_accessors(reader, field_name, missing)?;\n\n    // Special handling when missing + multi column or incompatible type on text/date.\n    let missing_and_more_than_one_col = column_and_types.len() > 1 && missing.is_some();\n    let text_on_non_text_col = column_and_types.len() == 1\n        && column_and_types[0].1 != ColumnType::Str\n        && matches!(missing, Some(Key::Str(_)));\n\n    let use_special_missing_agg = missing_and_more_than_one_col || text_on_non_text_col;\n\n    // If special missing handling is required, build a MissingTerm node that carries all\n    // accessors (across any column types) for existence checks.\n    if use_special_missing_agg {\n        let fallback_type = missing\n            .as_ref()\n            .map(|missing| match missing {\n                Key::Str(_) => ColumnType::Str,\n                Key::F64(_) => ColumnType::F64,\n                Key::I64(_) => ColumnType::I64,\n                Key::U64(_) => ColumnType::U64,\n            })\n            .unwrap_or(ColumnType::U64);\n        let all_accessors = get_all_ff_reader_or_empty(reader, field_name, None, fallback_type)?\n            .into_iter()\n            .collect::<Vec<_>>();\n        // This case only happens when we have term aggregation, or we fail\n        let req = req.as_terms().cloned().ok_or_else(|| {\n            crate::TantivyError::InvalidArgument(\n                \"Cardinality aggregation with missing on non-text/number field is not supported.\"\n                    .to_string(),\n            )\n        })?;\n\n        let children = build_children(sub_aggs, reader, segment_ordinal, data)?;\n        let idx_in_req_data = data.push_missing_term_req_data(MissingTermAggReqData {\n            accessors: all_accessors,\n            name: agg_name.to_string(),\n            req,\n        });\n        nodes.push(AggRefNode {\n            kind: AggKind::MissingTerm,\n            idx_in_req_data,\n            children,\n        });\n    }\n\n    // Add one node per accessor\n    for (accessor, column_type) in column_and_types {\n        let missing_value_for_accessor = if use_special_missing_agg {\n            None\n        } else if let Some(m) = missing.as_ref() {\n            get_missing_val_as_u64_lenient(column_type, accessor.max_value(), m, field_name)?\n        } else {\n            None\n        };\n\n        let children = build_children(sub_aggs, reader, segment_ordinal, data)?;\n        let (idx, kind) = match req {\n            TermsOrCardinalityRequest::Terms(ref req) => {\n                let mut allowed_term_ids = None;\n                if req.include.is_some() || req.exclude.is_some() {\n                    if column_type != ColumnType::Str {\n                        // Skip non-string columns entirely when filtering is requested.\n                        // When excluding, the behavior could be to include non-string values\n                        continue;\n                    }\n                    let str_col = str_dict_column\n                        .as_ref()\n                        .expect(\"str_dict_column must exist for string column\");\n                    allowed_term_ids =\n                        build_allowed_term_ids_for_str(str_col, &req.include, &req.exclude)?;\n                };\n                let idx_in_req_data = data.push_term_req_data(TermsAggReqData {\n                    accessor,\n                    column_type,\n                    str_dict_column: str_dict_column.clone(),\n                    missing_value_for_accessor,\n                    name: agg_name.to_string(),\n                    req: TermsAggregationInternal::from_req(req),\n                    sug_aggregations: sub_aggs.clone(),\n                    allowed_term_ids,\n                    is_top_level,\n                });\n                (idx_in_req_data, AggKind::Terms)\n            }\n            TermsOrCardinalityRequest::Cardinality(ref req) => {\n                let idx_in_req_data = data.push_cardinality_req_data(CardinalityAggReqData {\n                    accessor,\n                    column_type,\n                    str_dict_column: str_dict_column.clone(),\n                    missing_value_for_accessor,\n                    name: agg_name.to_string(),\n                    req: req.clone(),\n                });\n                (idx_in_req_data, AggKind::Cardinality)\n            }\n        };\n        nodes.push(AggRefNode {\n            kind,\n            idx_in_req_data: idx,\n            children,\n        });\n    }\n\n    Ok(nodes)\n}\n\n/// Builds a single BitSet of allowed term ordinals for a string dictionary column according to\n/// include/exclude parameters.\nfn build_allowed_term_ids_for_str(\n    str_col: &StrColumn,\n    include: &Option<IncludeExcludeParam>,\n    exclude: &Option<IncludeExcludeParam>,\n) -> crate::Result<Option<BitSet>> {\n    let mut allowed: Option<BitSet> = None;\n    let num_terms = str_col.dictionary().num_terms() as u32;\n    if let Some(include) = include {\n        // add matches\n        allowed = Some(BitSet::with_max_value(num_terms));\n        let allowed = allowed.as_mut().unwrap();\n        for_each_matching_term_ord(str_col, include, |ord| allowed.insert(ord))?;\n    };\n\n    if let Some(exclude) = exclude {\n        if allowed.is_none() {\n            // Start with all terms allowed\n            allowed = Some(BitSet::with_max_value_and_full(num_terms));\n        }\n        let allowed = allowed.as_mut().unwrap();\n        for_each_matching_term_ord(str_col, exclude, |ord| allowed.remove(ord))?;\n    }\n\n    Ok(allowed)\n}\n\n/// Apply a callback to each matching term ordinal for the given include/exclude parameter.\nfn for_each_matching_term_ord(\n    str_col: &StrColumn,\n    param: &IncludeExcludeParam,\n    mut cb: impl FnMut(u32),\n) -> crate::Result<()> {\n    match param {\n        IncludeExcludeParam::Regex(pattern) => {\n            let re = Regex::new(pattern).map_err(|e| {\n                crate::TantivyError::InvalidArgument(format!(\"Invalid regex `{}`: {}\", pattern, e))\n            })?;\n            // TODO: we can handle patterns like `^prefix.*` more efficiently\n            let mut stream = str_col.dictionary().search(re).into_stream()?;\n            while stream.advance() {\n                cb(stream.term_ord() as u32);\n            }\n        }\n        IncludeExcludeParam::Values(values) => {\n            let set: FxHashSet<&str> = values.iter().map(|s| s.as_str()).collect();\n            let mut stream = str_col.dictionary().stream()?;\n            while stream.advance() {\n                if let Ok(key_str) = std::str::from_utf8(stream.key()) {\n                    if set.contains(key_str) {\n                        cb(stream.term_ord() as u32);\n                    }\n                }\n            }\n        }\n    }\n    Ok(())\n}\n\n/// Convert the aggregation tree to something serializable and easy to read.\n#[derive(Serialize, Debug, Clone, PartialEq, Eq)]\npub struct AggTreeViewNode {\n    pub name: String,\n    pub kind: String,\n    #[serde(skip_serializing_if = \"Vec::is_empty\", default)]\n    pub children: Vec<AggTreeViewNode>,\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::get_test_index_2_segments;\n\n    fn agg_from_json(val: serde_json::Value) -> crate::aggregation::agg_req::Aggregation {\n        serde_json::from_value(val).unwrap()\n    }\n\n    #[test]\n    fn test_tree_roots_and_expansion_terms_missing_on_numeric() -> crate::Result<()> {\n        let index = get_test_index_2_segments(true)?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let seg_reader = searcher.segment_reader(0u32);\n\n        // Build request with:\n        // 1) Terms on numeric field with missing as string => expands to MissingTerm + Terms\n        // 2) Avg metric\n        // 3) Terms on string with child histogram\n        let terms_score_missing = agg_from_json(json!({\n            \"terms\": {\"field\": \"score\", \"missing\": \"NA\"}\n        }));\n        let avg_score = agg_from_json(json!({\n            \"avg\": {\"field\": \"score\"}\n        }));\n        let terms_string_with_child = agg_from_json(json!({\n            \"terms\": {\"field\": \"string_id\"},\n            \"aggs\": {\n                \"histo\": {\"histogram\": {\"field\": \"score\", \"interval\": 10.0}}\n            }\n        }));\n\n        let aggs: Aggregations = vec![\n            (\"t_score_missing_str\".to_string(), terms_score_missing),\n            (\"avg_score\".to_string(), avg_score),\n            (\"terms_string\".to_string(), terms_string_with_child),\n        ]\n        .into_iter()\n        .collect();\n\n        let data = build_aggregations_data_from_req(&aggs, seg_reader, 0u32, Default::default())?;\n        let printed_nodes = data.per_request.get_view_tree();\n        let printed = serde_json::to_value(&printed_nodes).unwrap();\n\n        let expected = json!([\n            {\"name\": \"avg_score\", \"kind\": \"Metric\"},\n            {\"name\": \"t_score_missing_str\", \"kind\": \"MissingTerm\"},\n            {\"name\": \"t_score_missing_str\", \"kind\": \"Terms\"},\n            {\"name\": \"terms_string\", \"kind\": \"Terms\", \"children\": [\n                {\"name\": \"histo\", \"kind\": \"Histogram\"}\n            ]}\n        ]);\n        assert_eq!(\n            printed,\n            expected,\n            \"tree json:\\n{}\",\n            serde_json::to_string_pretty(&printed).unwrap()\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/agg_limits.rs",
    "content": "use std::collections::HashMap;\nuse std::sync::atomic::{AtomicU64, Ordering};\nuse std::sync::Arc;\n\nuse common::ByteCount;\n\nuse super::collector::DEFAULT_MEMORY_LIMIT;\nuse super::{AggregationError, DEFAULT_BUCKET_LIMIT};\n\n/// An estimate for memory consumption. Non recursive\npub trait MemoryConsumption {\n    fn memory_consumption(&self) -> usize;\n}\n\nimpl<K, V, S> MemoryConsumption for HashMap<K, V, S> {\n    fn memory_consumption(&self) -> usize {\n        let capacity = self.capacity();\n        (std::mem::size_of::<K>() + std::mem::size_of::<V>() + 1) * capacity\n    }\n}\n\n/// Aggregation memory limit after which the request fails. Defaults to DEFAULT_MEMORY_LIMIT\n/// (500MB). The limit is shared by all SegmentCollectors\n///\n/// The memory limit is also a guard, which tracks how much it allocated and releases it's memory\n/// on the shared counter. Cloning will create a new guard.\npub struct AggregationLimitsGuard {\n    /// The counter which is shared between the aggregations for one request.\n    memory_consumption: Arc<AtomicU64>,\n    /// The memory_limit in bytes\n    memory_limit: ByteCount,\n    /// The maximum number of buckets _returned_\n    /// This is not counting intermediate buckets.\n    bucket_limit: u32,\n    /// Allocated memory with this guard.\n    allocated_with_the_guard: u64,\n}\n\nimpl Clone for AggregationLimitsGuard {\n    fn clone(&self) -> Self {\n        Self {\n            memory_consumption: Arc::clone(&self.memory_consumption),\n            memory_limit: self.memory_limit,\n            bucket_limit: self.bucket_limit,\n            allocated_with_the_guard: 0,\n        }\n    }\n}\n\nimpl Drop for AggregationLimitsGuard {\n    /// Removes the memory consumed tracked by this _instance_ of AggregationLimits.\n    /// This is used to clear the segment specific memory consumption all at once.\n    fn drop(&mut self) {\n        self.memory_consumption\n            .fetch_sub(self.allocated_with_the_guard, Ordering::Relaxed);\n    }\n}\n\nimpl Default for AggregationLimitsGuard {\n    fn default() -> Self {\n        Self {\n            memory_consumption: Default::default(),\n            memory_limit: DEFAULT_MEMORY_LIMIT.into(),\n            bucket_limit: DEFAULT_BUCKET_LIMIT,\n            allocated_with_the_guard: 0,\n        }\n    }\n}\n\nimpl AggregationLimitsGuard {\n    /// *memory_limit*\n    /// memory_limit is defined in bytes.\n    /// Aggregation fails when the estimated memory consumption of the aggregation is higher than\n    /// memory_limit.\n    /// memory_limit will default to `DEFAULT_MEMORY_LIMIT` (500MB)\n    ///\n    /// *bucket_limit*\n    /// Limits the maximum number of buckets returned from an aggregation request.\n    /// bucket_limit will default to `DEFAULT_BUCKET_LIMIT` (65000)\n    ///\n    /// Note: The returned instance contains a Arc shared counter to track memory consumption.\n    pub fn new(memory_limit: Option<u64>, bucket_limit: Option<u32>) -> Self {\n        Self {\n            memory_consumption: Default::default(),\n            memory_limit: memory_limit.unwrap_or(DEFAULT_MEMORY_LIMIT).into(),\n            bucket_limit: bucket_limit.unwrap_or(DEFAULT_BUCKET_LIMIT),\n            allocated_with_the_guard: 0,\n        }\n    }\n\n    pub(crate) fn add_memory_consumed(&mut self, add_num_bytes: u64) -> crate::Result<()> {\n        let prev_value = self\n            .memory_consumption\n            .fetch_add(add_num_bytes, Ordering::Relaxed);\n        self.allocated_with_the_guard += add_num_bytes;\n        validate_memory_consumption(prev_value + add_num_bytes, self.memory_limit)?;\n        Ok(())\n    }\n\n    pub(crate) fn get_bucket_limit(&self) -> u32 {\n        self.bucket_limit\n    }\n}\n\nfn validate_memory_consumption(\n    memory_consumption: u64,\n    memory_limit: ByteCount,\n) -> Result<(), AggregationError> {\n    // Load the estimated memory consumed by the aggregations\n    let memory_consumed: ByteCount = memory_consumption.into();\n    if memory_consumed > memory_limit {\n        return Err(AggregationError::MemoryExceeded {\n            limit: memory_limit,\n            current: memory_consumed,\n        });\n    }\n    Ok(())\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::aggregation::tests::exec_request_with_query;\n\n    // https://github.com/quickwit-oss/quickwit/issues/3837\n    #[test]\n    fn test_agg_limits_with_empty_merge() {\n        use crate::aggregation::agg_req::Aggregations;\n        use crate::aggregation::bucket::tests::get_test_index_from_docs;\n\n        let docs = vec![\n            vec![r#\"{ \"date\": \"2015-01-02T00:00:00Z\", \"text\": \"bbb\", \"text2\": \"bbb\" }\"#],\n            vec![r#\"{ \"text\": \"aaa\", \"text2\": \"bbb\" }\"#],\n        ];\n        let index = get_test_index_from_docs(false, &docs).unwrap();\n\n        {\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"1\": {\n                        \"terms\": {\"field\": \"text2\", \"min_doc_count\": 0},\n                        \"aggs\": {\n                            \"2\":{\n                                \"date_histogram\": {\n                                    \"field\": \"date\",\n                                    \"fixed_interval\": \"1d\",\n                                    \"extended_bounds\": {\n                                        \"min\": \"2015-01-01T00:00:00Z\",\n                                        \"max\": \"2015-01-10T00:00:00Z\"\n                                    }\n                                }\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request_with_query(agg_req, &index, Some((\"text\", \"bbb\"))).unwrap();\n            let expected_res = json!({\n             \"1\": {\n                \"buckets\": [\n                  {\n                    \"2\": {\n                      \"buckets\": [\n                        { \"doc_count\": 0, \"key\": 1420070400000.0, \"key_as_string\": \"2015-01-01T00:00:00Z\" },\n                        { \"doc_count\": 1, \"key\": 1420156800000.0, \"key_as_string\": \"2015-01-02T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420243200000.0, \"key_as_string\": \"2015-01-03T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420329600000.0, \"key_as_string\": \"2015-01-04T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420416000000.0, \"key_as_string\": \"2015-01-05T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420502400000.0, \"key_as_string\": \"2015-01-06T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420588800000.0, \"key_as_string\": \"2015-01-07T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420675200000.0, \"key_as_string\": \"2015-01-08T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420761600000.0, \"key_as_string\": \"2015-01-09T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420848000000.0, \"key_as_string\": \"2015-01-10T00:00:00Z\" }\n                      ]\n                    },\n                    \"doc_count\": 1,\n                    \"key\": \"bbb\"\n                  }\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n              }\n            });\n            assert_eq!(res, expected_res);\n        }\n    }\n\n    // https://github.com/quickwit-oss/quickwit/issues/3837\n    #[test]\n    fn test_agg_limits_with_empty_data() {\n        use crate::aggregation::agg_req::Aggregations;\n        use crate::aggregation::bucket::tests::get_test_index_from_docs;\n\n        let docs = vec![vec![r#\"{ \"text\": \"aaa\", \"text2\": \"bbb\" }\"#]];\n        let index = get_test_index_from_docs(false, &docs).unwrap();\n\n        {\n            // Empty result since there is no doc with dates\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"1\": {\n                        \"terms\": {\"field\": \"text2\", \"min_doc_count\": 0},\n                        \"aggs\": {\n                            \"2\":{\n                                \"date_histogram\": {\n                                    \"field\": \"date\",\n                                    \"fixed_interval\": \"1d\",\n                                    \"extended_bounds\": {\n                                        \"min\": \"2015-01-01T00:00:00Z\",\n                                        \"max\": \"2015-01-10T00:00:00Z\"\n                                    }\n                                }\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request_with_query(agg_req, &index, Some((\"text\", \"bbb\"))).unwrap();\n            let expected_res = json!({\n             \"1\": {\n                \"buckets\": [\n                  {\n                    \"2\": {\n                      \"buckets\": [\n                        { \"doc_count\": 0, \"key\": 1420070400000.0, \"key_as_string\": \"2015-01-01T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420156800000.0, \"key_as_string\": \"2015-01-02T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420243200000.0, \"key_as_string\": \"2015-01-03T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420329600000.0, \"key_as_string\": \"2015-01-04T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420416000000.0, \"key_as_string\": \"2015-01-05T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420502400000.0, \"key_as_string\": \"2015-01-06T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420588800000.0, \"key_as_string\": \"2015-01-07T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420675200000.0, \"key_as_string\": \"2015-01-08T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420761600000.0, \"key_as_string\": \"2015-01-09T00:00:00Z\" },\n                        { \"doc_count\": 0, \"key\": 1420848000000.0, \"key_as_string\": \"2015-01-10T00:00:00Z\" }\n                      ]\n                    },\n                    \"doc_count\": 0,\n                    \"key\": \"bbb\"\n                  }\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n              }\n            });\n            assert_eq!(res, expected_res);\n        }\n    }\n}\n"
  },
  {
    "path": "src/aggregation/agg_req.rs",
    "content": "//! Contains the aggregation request tree. Used to build an\n//! [`AggregationCollector`](super::AggregationCollector).\n//!\n//! [`Aggregations`] is the top level entry point to create a request, which is a `HashMap<String,\n//! Aggregation>`.\n//!\n//! Requests are compatible with the json format of elasticsearch.\n//!\n//! # Example\n//!\n//! ```\n//! use tantivy::aggregation::agg_req::Aggregations;\n//!\n//! let elasticsearch_compatible_json_req = r#\"\n//! {\n//!   \"range\": {\n//!     \"range\": {\n//!       \"field\": \"score\",\n//!       \"ranges\": [\n//!         { \"from\": 3.0, \"to\": 7.0 },\n//!         { \"from\": 7.0, \"to\": 20.0 }\n//!       ]\n//!     }\n//!   }\n//! }\"#;\n//! let _agg_req: Aggregations = serde_json::from_str(elasticsearch_compatible_json_req).unwrap();\n//! ```\n\nuse std::collections::HashSet;\n\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse super::bucket::{\n    CompositeAggregation, DateHistogramAggregationReq, FilterAggregation, HistogramAggregation,\n    RangeAggregation, TermsAggregation,\n};\nuse super::metric::{\n    AverageAggregation, CardinalityAggregationReq, CountAggregation, ExtendedStatsAggregation,\n    MaxAggregation, MinAggregation, PercentilesAggregationReq, StatsAggregation, SumAggregation,\n    TopHitsAggregationReq,\n};\n\n/// The top-level aggregation request structure, which contains [`Aggregation`] and their user\n/// defined names. It is also used in buckets aggregations to define sub-aggregations.\n///\n/// The key is the user defined name of the aggregation.\npub type Aggregations = FxHashMap<String, Aggregation>;\n\n/// Aggregation request.\n///\n/// An aggregation is either a bucket or a metric.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(try_from = \"AggregationForDeserialization\")]\npub struct Aggregation {\n    /// The aggregation variant, which can be either a bucket or a metric.\n    #[serde(flatten)]\n    pub agg: AggregationVariants,\n    /// on the document set in the bucket.\n    #[serde(rename = \"aggs\")]\n    #[serde(skip_serializing_if = \"Aggregations::is_empty\")]\n    pub sub_aggregation: Aggregations,\n}\n\n/// In order to display proper error message, we cannot rely on flattening\n/// the json enum. Instead we introduce an intermediary struct to separate\n/// the aggregation from the subaggregation.\n#[derive(Deserialize)]\nstruct AggregationForDeserialization {\n    #[serde(flatten)]\n    pub aggs_remaining_json: serde_json::Value,\n    #[serde(rename = \"aggs\")]\n    #[serde(default)]\n    pub sub_aggregation: Aggregations,\n}\n\nimpl TryFrom<AggregationForDeserialization> for Aggregation {\n    type Error = serde_json::Error;\n\n    fn try_from(value: AggregationForDeserialization) -> serde_json::Result<Self> {\n        let AggregationForDeserialization {\n            aggs_remaining_json,\n            sub_aggregation,\n        } = value;\n        let agg: AggregationVariants = serde_json::from_value(aggs_remaining_json)?;\n        Ok(Aggregation {\n            agg,\n            sub_aggregation,\n        })\n    }\n}\n\nimpl Aggregation {\n    pub(crate) fn sub_aggregation(&self) -> &Aggregations {\n        &self.sub_aggregation\n    }\n\n    fn get_fast_field_names(&self, fast_field_names: &mut HashSet<String>) {\n        fast_field_names.extend(\n            self.agg\n                .get_fast_field_names()\n                .iter()\n                .map(|s| s.to_string()),\n        );\n        fast_field_names.extend(get_fast_field_names(&self.sub_aggregation));\n    }\n}\n\n/// Extract all fast field names used in the tree.\npub fn get_fast_field_names(aggs: &Aggregations) -> HashSet<String> {\n    let mut fast_field_names = Default::default();\n    for el in aggs.values() {\n        el.get_fast_field_names(&mut fast_field_names)\n    }\n    fast_field_names\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n/// All aggregation types.\npub enum AggregationVariants {\n    // Bucket aggregation types\n    /// Put data into buckets of user-defined ranges.\n    #[serde(rename = \"range\")]\n    Range(RangeAggregation),\n    /// Put data into a histogram.\n    #[serde(rename = \"histogram\")]\n    Histogram(HistogramAggregation),\n    /// Put data into a date histogram.\n    #[serde(rename = \"date_histogram\")]\n    DateHistogram(DateHistogramAggregationReq),\n    /// Put data into buckets of terms.\n    #[serde(rename = \"terms\")]\n    Terms(TermsAggregation),\n    /// Filter documents into a single bucket.\n    #[serde(rename = \"filter\")]\n    Filter(FilterAggregation),\n    /// Multi-dimensional, paginable bucket aggregation.\n    #[serde(rename = \"composite\")]\n    Composite(CompositeAggregation),\n\n    // Metric aggregation types\n    /// Computes the average of the extracted values.\n    #[serde(rename = \"avg\")]\n    Average(AverageAggregation),\n    /// Counts the number of extracted values.\n    #[serde(rename = \"value_count\")]\n    Count(CountAggregation),\n    /// Finds the maximum value.\n    #[serde(rename = \"max\")]\n    Max(MaxAggregation),\n    /// Finds the minimum value.\n    #[serde(rename = \"min\")]\n    Min(MinAggregation),\n    /// Computes a collection of statistics (`min`, `max`, `sum`, `count`, and `avg`) over the\n    /// extracted values.\n    #[serde(rename = \"stats\")]\n    Stats(StatsAggregation),\n    /// Computes a collection of estended statistics (`min`, `max`, `sum`, `count`, `avg`,\n    /// `sum_of_squares`, `variance`, `variance_sampling`, `std_deviation`,\n    /// `std_deviation_sampling`) over the  extracted values.\n    #[serde(rename = \"extended_stats\")]\n    ExtendedStats(ExtendedStatsAggregation),\n    /// Computes the sum of the extracted values.\n    #[serde(rename = \"sum\")]\n    Sum(SumAggregation),\n    /// Computes the sum of the extracted values.\n    #[serde(rename = \"percentiles\")]\n    Percentiles(PercentilesAggregationReq),\n    /// Finds the top k values matching some order\n    #[serde(rename = \"top_hits\")]\n    TopHits(TopHitsAggregationReq),\n    /// Computes an estimate of the number of unique values\n    #[serde(rename = \"cardinality\")]\n    Cardinality(CardinalityAggregationReq),\n}\n\nimpl AggregationVariants {\n    /// Returns the name of the fields used by the aggregation.\n    pub fn get_fast_field_names(&self) -> Vec<&str> {\n        match self {\n            AggregationVariants::Terms(terms) => vec![terms.field.as_str()],\n            AggregationVariants::Range(range) => vec![range.field.as_str()],\n            AggregationVariants::Histogram(histogram) => vec![histogram.field.as_str()],\n            AggregationVariants::DateHistogram(histogram) => vec![histogram.field.as_str()],\n            AggregationVariants::Filter(filter) => filter.get_fast_field_names(),\n            AggregationVariants::Composite(composite) => composite\n                .sources\n                .iter()\n                .map(|source| source.field())\n                .collect(),\n            AggregationVariants::Average(avg) => vec![avg.field_name()],\n            AggregationVariants::Count(count) => vec![count.field_name()],\n            AggregationVariants::Max(max) => vec![max.field_name()],\n            AggregationVariants::Min(min) => vec![min.field_name()],\n            AggregationVariants::Stats(stats) => vec![stats.field_name()],\n            AggregationVariants::ExtendedStats(extended_stats) => vec![extended_stats.field_name()],\n            AggregationVariants::Sum(sum) => vec![sum.field_name()],\n            AggregationVariants::Percentiles(per) => vec![per.field_name()],\n            AggregationVariants::TopHits(top_hits) => top_hits.field_names(),\n            AggregationVariants::Cardinality(per) => vec![per.field_name()],\n        }\n    }\n\n    pub(crate) fn as_range(&self) -> Option<&RangeAggregation> {\n        match &self {\n            AggregationVariants::Range(range) => Some(range),\n            _ => None,\n        }\n    }\n    pub(crate) fn as_histogram(&self) -> crate::Result<Option<HistogramAggregation>> {\n        match &self {\n            AggregationVariants::Histogram(histogram) => Ok(Some(histogram.clone())),\n            AggregationVariants::DateHistogram(histogram) => {\n                Ok(Some(histogram.to_histogram_req()?))\n            }\n            _ => Ok(None),\n        }\n    }\n    pub(crate) fn as_term(&self) -> Option<&TermsAggregation> {\n        match &self {\n            AggregationVariants::Terms(terms) => Some(terms),\n            _ => None,\n        }\n    }\n    pub(crate) fn as_composite(&self) -> Option<&CompositeAggregation> {\n        match &self {\n            AggregationVariants::Composite(composite) => Some(composite),\n            _ => None,\n        }\n    }\n    pub(crate) fn as_percentile(&self) -> Option<&PercentilesAggregationReq> {\n        match &self {\n            AggregationVariants::Percentiles(percentile_req) => Some(percentile_req),\n            _ => None,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n\n    #[test]\n    fn deser_json_test() {\n        let agg_req_json = r#\"{\n            \"price_avg\": { \"avg\": { \"field\": \"price\" } },\n            \"price_count\": { \"value_count\": { \"field\": \"price\" } },\n            \"price_max\": { \"max\": { \"field\": \"price\" } },\n            \"price_min\": { \"min\": { \"field\": \"price\" } },\n            \"price_stats\": { \"stats\": { \"field\": \"price\" } },\n            \"price_sum\": { \"sum\": { \"field\": \"price\" } }\n        }\"#;\n        let _agg_req: Aggregations = serde_json::from_str(agg_req_json).unwrap();\n    }\n\n    #[test]\n    fn deser_json_test_bucket() {\n        let agg_req_json = r#\"\n    {\n        \"termagg\": {\n            \"terms\": {\n                \"field\": \"json.mixed_type\",\n                \"order\": { \"min_price\": \"desc\" }\n            },\n            \"aggs\": {\n                \"min_price\": { \"min\": { \"field\": \"json.mixed_type\" } }\n            }\n        },\n        \"rangeagg\": {\n            \"range\": {\n                \"field\": \"json.mixed_type\",\n                \"ranges\": [\n                    { \"to\": 3.0 },\n                    { \"from\": 19.0, \"to\": 20.0 },\n                    { \"from\": 20.0 }\n                ]\n            },\n            \"aggs\": {\n                \"average_in_range\": { \"avg\": { \"field\": \"json.mixed_type\" } }\n            }\n        }\n    } \"#;\n\n        let _agg_req: Aggregations = serde_json::from_str(agg_req_json).unwrap();\n    }\n\n    #[test]\n    fn test_metric_aggregations_deser() {\n        let agg_req_json = r#\"{\n            \"price_avg\": { \"avg\": { \"field\": \"price\" } },\n            \"price_count\": { \"value_count\": { \"field\": \"price\" } },\n            \"price_max\": { \"max\": { \"field\": \"price\" } },\n            \"price_min\": { \"min\": { \"field\": \"price\" } },\n            \"price_stats\": { \"stats\": { \"field\": \"price\" } },\n            \"price_sum\": { \"sum\": { \"field\": \"price\" } }\n        }\"#;\n        let agg_req: Aggregations = serde_json::from_str(agg_req_json).unwrap();\n\n        assert!(\n            matches!(&agg_req.get(\"price_avg\").unwrap().agg, AggregationVariants::Average(avg) if avg.field == \"price\")\n        );\n        assert!(\n            matches!(&agg_req.get(\"price_count\").unwrap().agg, AggregationVariants::Count(count) if count.field == \"price\")\n        );\n        assert!(\n            matches!(&agg_req.get(\"price_max\").unwrap().agg, AggregationVariants::Max(max) if max.field == \"price\")\n        );\n        assert!(\n            matches!(&agg_req.get(\"price_min\").unwrap().agg, AggregationVariants::Min(min) if min.field == \"price\")\n        );\n        assert!(\n            matches!(&agg_req.get(\"price_stats\").unwrap().agg, AggregationVariants::Stats(stats) if stats.field == \"price\")\n        );\n        assert!(\n            matches!(&agg_req.get(\"price_sum\").unwrap().agg, AggregationVariants::Sum(sum) if sum.field == \"price\")\n        );\n    }\n\n    #[test]\n    fn serialize_to_json_test() {\n        let elasticsearch_compatible_json_req = r#\"{\n  \"range\": {\n    \"range\": {\n      \"field\": \"score\",\n      \"ranges\": [\n        {\n          \"to\": 3.0\n        },\n        {\n          \"from\": 3.0,\n          \"to\": 7.0\n        },\n        {\n          \"from\": 7.0,\n          \"to\": 20.0\n        },\n        {\n          \"from\": 20.0\n        }\n      ],\n      \"keyed\": true\n    }\n  }\n}\"#;\n\n        let agg_req1: Aggregations =\n            { serde_json::from_str(elasticsearch_compatible_json_req).unwrap() };\n\n        let agg_req2: String = serde_json::to_string_pretty(&agg_req1).unwrap();\n        assert_eq!(agg_req2, elasticsearch_compatible_json_req);\n    }\n\n    #[test]\n    fn test_get_fast_field_names() {\n        let range_agg: Aggregation = {\n            serde_json::from_value(json!({\n                \"range\": {\n                    \"field\": \"score\",\n                    \"ranges\": [\n                        { \"to\": 3.0 },\n                        { \"from\": 3.0, \"to\": 7.0 },\n                        { \"from\": 7.0, \"to\": 20.0 },\n                        { \"from\": 20.0 }\n                    ],\n                }\n\n            }))\n            .unwrap()\n        };\n\n        let agg_req1: Aggregations = {\n            serde_json::from_value(json!({\n                \"range1\": range_agg,\n                \"range2\":{\n                    \"range\": {\n                        \"field\": \"score2\",\n                        \"ranges\": [\n                            { \"to\": 3.0 },\n                            { \"from\": 3.0, \"to\": 7.0 },\n                            { \"from\": 7.0, \"to\": 20.0 },\n                            { \"from\": 20.0 }\n                        ],\n                    },\n                    \"aggs\": {\n                        \"metric\": {\n                            \"avg\": {\n                                \"field\": \"field123\"\n                            }\n                        }\n                    }\n                }\n            }))\n            .unwrap()\n        };\n\n        assert_eq!(\n            get_fast_field_names(&agg_req1),\n            vec![\n                \"score\".to_string(),\n                \"score2\".to_string(),\n                \"field123\".to_string()\n            ]\n            .into_iter()\n            .collect()\n        )\n    }\n}\n"
  },
  {
    "path": "src/aggregation/agg_result.rs",
    "content": "//! Contains the final aggregation tree.\n//!\n//! This tree can be converted via the `into()` method from `IntermediateAggregationResults`.\n//! This conversion computes the final result. For example: The intermediate result contains\n//! intermediate average results, which is the sum and the number of values. The actual average is\n//! calculated on the step from intermediate to final aggregation result tree.\n\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse super::bucket::GetDocCount;\nuse super::intermediate_agg_result::CompositeIntermediateKey;\nuse super::metric::{\n    ExtendedStats, PercentilesMetricResult, SingleMetricResult, Stats, TopHitsMetricResult,\n};\nuse super::{AggregationError, Key};\nuse crate::aggregation::bucket::AfterKey;\nuse crate::TantivyError;\n\n#[derive(Clone, Default, Debug, PartialEq, Serialize, Deserialize)]\n/// The final aggregation result.\npub struct AggregationResults(pub FxHashMap<String, AggregationResult>);\n\nimpl AggregationResults {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        self.0\n            .values()\n            .map(|agg| agg.get_bucket_count())\n            .sum::<u64>()\n    }\n\n    pub(crate) fn get_value_from_aggregation(\n        &self,\n        name: &str,\n        agg_property: &str,\n    ) -> crate::Result<Option<f64>> {\n        if let Some(agg) = self.0.get(name) {\n            agg.get_value_from_aggregation(name, agg_property)\n        } else {\n            // Validation is be done during request parsing, so we can't reach this state.\n            Err(TantivyError::InternalError(format!(\n                \"Can't find aggregation {name:?} in sub-aggregations\"\n            )))\n        }\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\n/// An aggregation is either a bucket or a metric.\npub enum AggregationResult {\n    /// Bucket result variant.\n    BucketResult(BucketResult),\n    /// Metric result variant.\n    MetricResult(MetricResult),\n}\n\nimpl AggregationResult {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        match self {\n            AggregationResult::BucketResult(bucket) => bucket.get_bucket_count(),\n            AggregationResult::MetricResult(_) => 0,\n        }\n    }\n\n    pub(crate) fn get_value_from_aggregation(\n        &self,\n        _name: &str,\n        agg_property: &str,\n    ) -> crate::Result<Option<f64>> {\n        match self {\n            AggregationResult::BucketResult(_bucket) => Err(TantivyError::InternalError(\n                \"Tried to retrieve value from bucket aggregation. This is not supported and \\\n                 should not happen during collection phase, but should be caught during validation\"\n                    .to_string(),\n            )),\n            AggregationResult::MetricResult(metric) => metric.get_value(agg_property),\n        }\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\n/// MetricResult\npub enum MetricResult {\n    /// Average metric result.\n    Average(SingleMetricResult),\n    /// Count metric result.\n    Count(SingleMetricResult),\n    /// Max metric result.\n    Max(SingleMetricResult),\n    /// Min metric result.\n    Min(SingleMetricResult),\n    /// Stats metric result.\n    Stats(Stats),\n    /// ExtendedStats metric result.\n    ExtendedStats(Box<ExtendedStats>),\n    /// Sum metric result.\n    Sum(SingleMetricResult),\n    /// Percentiles metric result.\n    Percentiles(PercentilesMetricResult),\n    /// Top hits metric result\n    TopHits(TopHitsMetricResult),\n    /// Cardinality metric result\n    Cardinality(SingleMetricResult),\n}\n\nimpl MetricResult {\n    fn get_value(&self, agg_property: &str) -> crate::Result<Option<f64>> {\n        match self {\n            MetricResult::Average(avg) => Ok(avg.value),\n            MetricResult::Count(count) => Ok(count.value),\n            MetricResult::Max(max) => Ok(max.value),\n            MetricResult::Min(min) => Ok(min.value),\n            MetricResult::Stats(stats) => stats.get_value(agg_property),\n            MetricResult::ExtendedStats(extended_stats) => extended_stats.get_value(agg_property),\n            MetricResult::Sum(sum) => Ok(sum.value),\n            MetricResult::Percentiles(_) => Err(TantivyError::AggregationError(\n                AggregationError::InvalidRequest(\"percentiles can't be used to order\".to_string()),\n            )),\n            MetricResult::TopHits(_) => Err(TantivyError::AggregationError(\n                AggregationError::InvalidRequest(\"top_hits can't be used to order\".to_string()),\n            )),\n            MetricResult::Cardinality(card) => Ok(card.value),\n        }\n    }\n}\n\n/// BucketEntry holds bucket aggregation result types.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\npub enum BucketResult {\n    /// This is the range entry for a bucket, which contains a key, count, from, to, and optionally\n    /// sub-aggregations.\n    Range {\n        /// The range buckets sorted by range.\n        buckets: BucketEntries<RangeBucketEntry>,\n    },\n    /// This is the histogram entry for a bucket, which contains a key, count, and optionally\n    /// sub-aggregations.\n    Histogram {\n        /// The buckets.\n        ///\n        /// If there are holes depends on the request, if min_doc_count is 0, then there are no\n        /// holes between the first and last bucket.\n        /// See [`HistogramAggregation`](super::bucket::HistogramAggregation)\n        buckets: BucketEntries<BucketEntry>,\n    },\n    /// This is the term result\n    Terms {\n        /// The buckets.\n        ///\n        /// See [`TermsAggregation`](super::bucket::TermsAggregation)\n        buckets: Vec<BucketEntry>,\n        /// The number of documents that didn’t make it into to TOP N due to shard_size or size\n        sum_other_doc_count: u64,\n        #[serde(skip_serializing_if = \"Option::is_none\")]\n        /// The upper bound error for the doc count of each term.\n        doc_count_error_upper_bound: Option<u64>,\n    },\n    /// This is the filter result - a single bucket with sub-aggregations\n    Filter(FilterBucketResult),\n    /// This is the composite result\n    Composite {\n        /// The buckets\n        buckets: Vec<CompositeBucketEntry>,\n        /// The key to start after when paginating\n        #[serde(skip_serializing_if = \"FxHashMap::is_empty\")]\n        after_key: FxHashMap<String, AfterKey>,\n    },\n}\n\nimpl BucketResult {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        match self {\n            BucketResult::Range { buckets } => {\n                buckets.iter().map(|bucket| bucket.get_bucket_count()).sum()\n            }\n            BucketResult::Histogram { buckets } => {\n                buckets.iter().map(|bucket| bucket.get_bucket_count()).sum()\n            }\n            BucketResult::Terms {\n                buckets,\n                sum_other_doc_count: _,\n                doc_count_error_upper_bound: _,\n            } => buckets.iter().map(|bucket| bucket.get_bucket_count()).sum(),\n            BucketResult::Filter(filter_result) => {\n                // Filter doesn't add to bucket count - it's not a user-facing bucket\n                // Only count sub-aggregation buckets\n                filter_result.sub_aggregations.get_bucket_count()\n            }\n            BucketResult::Composite { buckets, .. } => {\n                buckets.iter().map(|bucket| bucket.get_bucket_count()).sum()\n            }\n        }\n    }\n}\n\n/// This is the wrapper of buckets entries, which can be vector or hashmap\n/// depending on if it's keyed or not.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\npub enum BucketEntries<T> {\n    /// Vector format bucket entries\n    Vec(Vec<T>),\n    /// HashMap format bucket entries\n    HashMap(FxHashMap<String, T>),\n}\n\nimpl<T> BucketEntries<T> {\n    fn iter<'a>(&'a self) -> Box<dyn Iterator<Item = &'a T> + 'a> {\n        match self {\n            BucketEntries::Vec(vec) => Box::new(vec.iter()),\n            BucketEntries::HashMap(map) => Box::new(map.values()),\n        }\n    }\n}\n\n/// This is the default entry for a bucket, which contains a key, count, and optionally\n/// sub-aggregations.\n///\n/// # JSON Format\n/// ```json\n/// {\n///   ...\n///     \"my_histogram\": {\n///       \"buckets\": [\n///         {\n///           \"key\": \"2.0\",\n///           \"doc_count\": 5\n///         },\n///         {\n///           \"key\": \"4.0\",\n///           \"doc_count\": 2\n///         },\n///         {\n///           \"key\": \"6.0\",\n///           \"doc_count\": 3\n///         }\n///       ]\n///    }\n///    ...\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct BucketEntry {\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    /// The string representation of the bucket.\n    pub key_as_string: Option<String>,\n    /// The identifier of the bucket.\n    pub key: Key,\n    /// Number of documents in the bucket.\n    pub doc_count: u64,\n    #[serde(flatten)]\n    /// Sub-aggregations in this bucket.\n    pub sub_aggregation: AggregationResults,\n}\nimpl BucketEntry {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        1 + self.sub_aggregation.get_bucket_count()\n    }\n}\nimpl GetDocCount for &BucketEntry {\n    fn doc_count(&self) -> u64 {\n        self.doc_count\n    }\n}\nimpl GetDocCount for BucketEntry {\n    fn doc_count(&self) -> u64 {\n        self.doc_count\n    }\n}\n\n/// This is the range entry for a bucket, which contains a key, count, and optionally\n/// sub-aggregations.\n///\n/// # JSON Format\n/// ```json\n/// {\n///   ...\n///     \"my_ranges\": {\n///       \"buckets\": [\n///         {\n///           \"key\": \"*-10\",\n///           \"to\": 10,\n///           \"doc_count\": 5\n///         },\n///         {\n///           \"key\": \"10-20\",\n///           \"from\": 10,\n///           \"to\": 20,\n///           \"doc_count\": 2\n///         },\n///         {\n///           \"key\": \"20-*\",\n///           \"from\": 20,\n///           \"doc_count\": 3\n///         }\n///       ]\n///    }\n///    ...\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct RangeBucketEntry {\n    /// The identifier of the bucket.\n    pub key: Key,\n    /// Number of documents in the bucket.\n    pub doc_count: u64,\n    #[serde(flatten)]\n    /// Sub-aggregations in this bucket.\n    pub sub_aggregation: AggregationResults,\n    /// The from range of the bucket. Equals `f64::MIN` when `None`.\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub from: Option<f64>,\n    /// The to range of the bucket. Equals `f64::MAX` when `None`.\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub to: Option<f64>,\n    /// The optional string representation for the `from` range.\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub from_as_string: Option<String>,\n    /// The optional string representation for the `to` range.\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub to_as_string: Option<String>,\n}\nimpl RangeBucketEntry {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        1 + self.sub_aggregation.get_bucket_count()\n    }\n}\n\n/// This is the filter bucket result, which contains the document count and sub-aggregations.\n///\n/// # JSON Format\n/// ```json\n/// {\n///   \"electronics_only\": {\n///     \"doc_count\": 2,\n///     \"avg_price\": {\n///       \"value\": 150.0\n///     }\n///   }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct FilterBucketResult {\n    /// Number of documents in the filter bucket\n    pub doc_count: u64,\n    /// Sub-aggregation results\n    #[serde(flatten)]\n    pub sub_aggregations: AggregationResults,\n}\n\n/// Note the type information loss compared to `CompositeIntermediateKey`.\n/// Pagination is performed using `AfterKey`, which encodes type information.\n#[derive(Clone, Debug, Serialize, Deserialize)]\n#[serde(untagged)]\npub enum CompositeKey {\n    /// Boolean key\n    Bool(bool),\n    /// String key\n    Str(String),\n    /// `i64` key\n    I64(i64),\n    /// `u64` key\n    U64(u64),\n    /// `f64` key\n    F64(f64),\n    /// Null key\n    Null,\n}\nimpl Eq for CompositeKey {}\nimpl std::hash::Hash for CompositeKey {\n    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {\n        core::mem::discriminant(self).hash(state);\n        match self {\n            Self::Bool(val) => val.hash(state),\n            Self::Str(text) => text.hash(state),\n            Self::F64(val) => val.to_bits().hash(state),\n            Self::U64(val) => val.hash(state),\n            Self::I64(val) => val.hash(state),\n            Self::Null => {}\n        }\n    }\n}\nimpl PartialEq for CompositeKey {\n    fn eq(&self, other: &Self) -> bool {\n        match (self, other) {\n            (Self::Bool(l), Self::Bool(r)) => l == r,\n            (Self::Str(l), Self::Str(r)) => l == r,\n            (Self::F64(l), Self::F64(r)) => l.to_bits() == r.to_bits(),\n            (Self::I64(l), Self::I64(r)) => l == r,\n            (Self::U64(l), Self::U64(r)) => l == r,\n            (Self::Null, Self::Null) => true,\n            _ => false,\n        }\n    }\n}\nimpl From<CompositeIntermediateKey> for CompositeKey {\n    fn from(value: CompositeIntermediateKey) -> Self {\n        match value {\n            CompositeIntermediateKey::Str(s) => Self::Str(s),\n            CompositeIntermediateKey::IpAddr(s) => {\n                if let Some(ip) = s.to_ipv4_mapped() {\n                    Self::Str(ip.to_string())\n                } else {\n                    Self::Str(s.to_string())\n                }\n            }\n            CompositeIntermediateKey::F64(f) => Self::F64(f),\n            CompositeIntermediateKey::Bool(f) => Self::Bool(f),\n            CompositeIntermediateKey::U64(f) => Self::U64(f),\n            CompositeIntermediateKey::I64(f) => Self::I64(f),\n            CompositeIntermediateKey::DateTime(f) => Self::I64(f / 1_000_000), // ns to ms\n            CompositeIntermediateKey::Null => Self::Null,\n        }\n    }\n}\n\n/// Composite bucket entry with a multi-dimensional key.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct CompositeBucketEntry {\n    /// The identifier of the bucket.\n    pub key: FxHashMap<String, CompositeKey>,\n    /// Number of documents in the bucket.\n    pub doc_count: u64,\n    #[serde(flatten)]\n    /// Sub-aggregations in this bucket.\n    pub sub_aggregation: AggregationResults,\n}\n\nimpl CompositeBucketEntry {\n    pub(crate) fn get_bucket_count(&self) -> u64 {\n        1 + self.sub_aggregation.get_bucket_count()\n    }\n}\n"
  },
  {
    "path": "src/aggregation/agg_tests.rs",
    "content": "use serde_json::Value;\n\nuse crate::aggregation::agg_req::{Aggregation, Aggregations};\nuse crate::aggregation::agg_result::AggregationResults;\nuse crate::aggregation::collector::AggregationCollector;\nuse crate::aggregation::intermediate_agg_result::IntermediateAggregationResults;\nuse crate::aggregation::tests::{get_test_index_2_segments, get_test_index_from_values_and_terms};\nuse crate::aggregation::DistributedAggregationCollector;\nuse crate::docset::COLLECT_BLOCK_BUFFER_LEN;\nuse crate::query::{AllQuery, TermQuery};\nuse crate::schema::{IndexRecordOption, Schema, FAST};\nuse crate::{Index, IndexWriter, Term};\n\n// The following tests ensure that each bucket aggregation type correctly functions as a\n// sub-aggregation of another bucket aggregation in two scenarios:\n// 1) The parent has more buckets than the child sub-aggregation\n// 2) The child sub-aggregation has more buckets than the parent\n//\n// These scenarios exercise the bucket id mapping and sub-aggregation routing logic.\n\n#[test]\nfn test_terms_as_subagg_parent_more_vs_child_more() -> crate::Result<()> {\n    let index = get_test_index_2_segments(false)?;\n\n    // Case A: parent has more buckets than child\n    // Parent: range with 4 buckets\n    // Child: terms on text -> 2 buckets\n    let agg_parent_more: Aggregations = serde_json::from_value(json!({\n        \"parent_range\": {\n            \"range\": {\n                \"field\": \"score\",\n                \"ranges\": [\n                    {\"to\": 3.0},\n                    {\"from\": 3.0, \"to\": 7.0},\n                    {\"from\": 7.0, \"to\": 20.0},\n                    {\"from\": 20.0}\n                ]\n            },\n            \"aggs\": {\n                \"child_terms\": {\"terms\": {\"field\": \"text\", \"order\": {\"_key\": \"asc\"}}}\n            }\n        }\n    }))\n    .unwrap();\n\n    let res = crate::aggregation::tests::exec_request(agg_parent_more, &index)?;\n    // Exact expected structure and counts\n    assert_eq!(\n        res[\"parent_range\"][\"buckets\"],\n        json!([\n            {\n                \"key\": \"*-3\",\n                \"doc_count\": 1,\n                \"to\": 3.0,\n                \"child_terms\": {\n                    \"buckets\": [\n                        {\"doc_count\": 1, \"key\": \"cool\"}\n                    ],\n                    \"sum_other_doc_count\": 0\n                }\n            },\n            {\n                \"key\": \"3-7\",\n                \"doc_count\": 3,\n                \"from\": 3.0,\n                \"to\": 7.0,\n                \"child_terms\": {\n                    \"buckets\": [\n                        {\"doc_count\": 2, \"key\": \"cool\"},\n                        {\"doc_count\": 1, \"key\": \"nohit\"}\n                    ],\n                    \"sum_other_doc_count\": 0\n                }\n            },\n            {\n                \"key\": \"7-20\",\n                \"doc_count\": 3,\n                \"from\": 7.0,\n                \"to\": 20.0,\n                \"child_terms\": {\n                    \"buckets\": [\n                        {\"doc_count\": 3, \"key\": \"cool\"}\n                    ],\n                    \"sum_other_doc_count\": 0\n                }\n            },\n            {\n                \"key\": \"20-*\",\n                \"doc_count\": 2,\n                \"from\": 20.0,\n                \"child_terms\": {\n                    \"buckets\": [\n                        {\"doc_count\": 1, \"key\": \"cool\"},\n                        {\"doc_count\": 1, \"key\": \"nohit\"}\n                    ],\n                    \"sum_other_doc_count\": 0\n                }\n            }\n        ])\n    );\n\n    // Case B: child has more buckets than parent\n    // Parent: histogram on score with large interval -> 1 bucket\n    // Child: terms on text -> 2 buckets (cool/nohit)\n    let agg_child_more: Aggregations = serde_json::from_value(json!({\n        \"parent_hist\": {\n            \"histogram\": {\"field\": \"score\", \"interval\": 100.0},\n            \"aggs\": {\n                \"child_terms\": {\"terms\": {\"field\": \"text\", \"order\": {\"_key\": \"asc\"}}}\n            }\n        }\n    }))\n    .unwrap();\n\n    let res = crate::aggregation::tests::exec_request(agg_child_more, &index)?;\n    assert_eq!(\n        res[\"parent_hist\"],\n        json!({\n            \"buckets\": [\n                {\n                    \"key\": 0.0,\n                    \"doc_count\": 9,\n                    \"child_terms\": {\n                        \"buckets\": [\n                            {\"doc_count\": 7, \"key\": \"cool\"},\n                            {\"doc_count\": 2, \"key\": \"nohit\"}\n                        ],\n                        \"sum_other_doc_count\": 0\n                    }\n                }\n            ]\n        })\n    );\n\n    Ok(())\n}\n\n#[test]\nfn test_range_as_subagg_parent_more_vs_child_more() -> crate::Result<()> {\n    let index = get_test_index_2_segments(false)?;\n\n    // Case A: parent has more buckets than child\n    // Parent: range with 5 buckets\n    // Child: coarse range with 3 buckets\n    let agg_parent_more: Aggregations = serde_json::from_value(json!({\n        \"parent_range\": {\n            \"range\": {\n                \"field\": \"score\",\n                \"ranges\": [\n                    {\"to\": 3.0},\n                    {\"from\": 3.0, \"to\": 7.0},\n                    {\"from\": 7.0, \"to\": 11.0},\n                    {\"from\": 11.0, \"to\": 20.0},\n                    {\"from\": 20.0}\n                ]\n            },\n            \"aggs\": {\n                \"child_range\": {\n                    \"range\": {\n                        \"field\": \"score\",\n                        \"ranges\": [\n                            {\"to\": 3.0},\n                            {\"from\": 3.0, \"to\": 20.0}\n                        ]\n                    }\n                }\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_parent_more, &index)?;\n    assert_eq!(\n        res[\"parent_range\"][\"buckets\"],\n        json!([\n            {\"key\": \"*-3\", \"doc_count\": 1, \"to\": 3.0,\n                \"child_range\": {\"buckets\": [\n                    {\"key\": \"*-3\", \"doc_count\": 1, \"to\": 3.0},\n                    {\"key\": \"3-20\", \"doc_count\": 0, \"from\": 3.0, \"to\": 20.0},\n                    {\"key\": \"20-*\", \"doc_count\": 0, \"from\": 20.0}\n                ]}\n            },\n            {\"key\": \"3-7\", \"doc_count\": 3, \"from\": 3.0, \"to\": 7.0,\n                \"child_range\": {\"buckets\": [\n                    {\"key\": \"*-3\", \"doc_count\": 0, \"to\": 3.0},\n                    {\"key\": \"3-20\", \"doc_count\": 3, \"from\": 3.0, \"to\": 20.0},\n                    {\"key\": \"20-*\", \"doc_count\": 0, \"from\": 20.0}\n                ]}\n            },\n            {\"key\": \"7-11\", \"doc_count\": 1, \"from\": 7.0, \"to\": 11.0,\n                \"child_range\": {\"buckets\": [\n                    {\"key\": \"*-3\", \"doc_count\": 0, \"to\": 3.0},\n                    {\"key\": \"3-20\", \"doc_count\": 1, \"from\": 3.0, \"to\": 20.0},\n                    {\"key\": \"20-*\", \"doc_count\": 0, \"from\": 20.0}\n                ]}\n            },\n            {\"key\": \"11-20\", \"doc_count\": 2, \"from\": 11.0, \"to\": 20.0,\n                \"child_range\": {\"buckets\": [\n                    {\"key\": \"*-3\", \"doc_count\": 0, \"to\": 3.0},\n                    {\"key\": \"3-20\", \"doc_count\": 2, \"from\": 3.0, \"to\": 20.0},\n                    {\"key\": \"20-*\", \"doc_count\": 0, \"from\": 20.0}\n                ]}\n            },\n            {\"key\": \"20-*\", \"doc_count\": 2, \"from\": 20.0,\n                \"child_range\": {\"buckets\": [\n                    {\"key\": \"*-3\", \"doc_count\": 0, \"to\": 3.0},\n                    {\"key\": \"3-20\", \"doc_count\": 0, \"from\": 3.0, \"to\": 20.0},\n                    {\"key\": \"20-*\", \"doc_count\": 2, \"from\": 20.0}\n                ]}\n            }\n        ])\n    );\n\n    // Case B: child has more buckets than parent\n    // Parent: terms on text (2 buckets)\n    // Child: range with 4 buckets\n    let agg_child_more: Aggregations = serde_json::from_value(json!({\n        \"parent_terms\": {\n            \"terms\": {\"field\": \"text\"},\n            \"aggs\": {\n                \"child_range\": {\n                    \"range\": {\n                        \"field\": \"score\",\n                        \"ranges\": [\n                            {\"to\": 3.0},\n                            {\"from\": 3.0, \"to\": 7.0},\n                            {\"from\": 7.0, \"to\": 20.0}\n                        ]\n                    }\n                }\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_child_more, &index)?;\n\n    assert_eq!(\n        res[\"parent_terms\"],\n        json!({\n            \"buckets\": [\n                {\n                    \"key\": \"cool\",\n                    \"doc_count\": 7,\n                    \"child_range\": {\n                        \"buckets\": [\n                            {\"key\": \"*-3\", \"doc_count\": 1, \"to\": 3.0},\n                            {\"key\": \"3-7\", \"doc_count\": 2, \"from\": 3.0, \"to\": 7.0},\n                            {\"key\": \"7-20\", \"doc_count\": 3, \"from\": 7.0, \"to\": 20.0},\n                            {\"key\": \"20-*\", \"doc_count\": 1, \"from\": 20.0}\n                        ]\n                    }\n                },\n                {\n                    \"key\": \"nohit\",\n                    \"doc_count\": 2,\n                    \"child_range\": {\n                        \"buckets\": [\n                            {\"key\": \"*-3\", \"doc_count\": 0, \"to\": 3.0},\n                            {\"key\": \"3-7\", \"doc_count\": 1, \"from\": 3.0, \"to\": 7.0},\n                            {\"key\": \"7-20\", \"doc_count\": 0, \"from\": 7.0, \"to\": 20.0},\n                            {\"key\": \"20-*\", \"doc_count\": 1, \"from\": 20.0}\n                        ]\n                    }\n                }\n            ],\n            \"doc_count_error_upper_bound\": 0,\n            \"sum_other_doc_count\": 0\n        })\n    );\n\n    Ok(())\n}\n\n#[test]\nfn test_histogram_as_subagg_parent_more_vs_child_more() -> crate::Result<()> {\n    let index = get_test_index_2_segments(false)?;\n\n    // Case A: parent has more buckets than child\n    // Parent: range with several ranges\n    // Child: histogram with large interval (single bucket per parent)\n    let agg_parent_more: Aggregations = serde_json::from_value(json!({\n        \"parent_range\": {\n            \"range\": {\n                \"field\": \"score\",\n                \"ranges\": [\n                    {\"to\": 3.0},\n                    {\"from\": 3.0, \"to\": 7.0},\n                    {\"from\": 7.0, \"to\": 11.0},\n                    {\"from\": 11.0, \"to\": 20.0},\n                    {\"from\": 20.0}\n                ]\n            },\n            \"aggs\": {\n                \"child_hist\": {\"histogram\": {\"field\": \"score\", \"interval\": 100.0}}\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_parent_more, &index)?;\n    assert_eq!(\n        res[\"parent_range\"][\"buckets\"],\n        json!([\n            {\"key\": \"*-3\", \"doc_count\": 1, \"to\": 3.0,\n                \"child_hist\": {\"buckets\": [ {\"key\": 0.0, \"doc_count\": 1} ]}\n            },\n            {\"key\": \"3-7\", \"doc_count\": 3, \"from\": 3.0, \"to\": 7.0,\n                \"child_hist\": {\"buckets\": [ {\"key\": 0.0, \"doc_count\": 3} ]}\n            },\n            {\"key\": \"7-11\", \"doc_count\": 1, \"from\": 7.0, \"to\": 11.0,\n                \"child_hist\": {\"buckets\": [ {\"key\": 0.0, \"doc_count\": 1} ]}\n            },\n            {\"key\": \"11-20\", \"doc_count\": 2, \"from\": 11.0, \"to\": 20.0,\n                \"child_hist\": {\"buckets\": [ {\"key\": 0.0, \"doc_count\": 2} ]}\n            },\n            {\"key\": \"20-*\", \"doc_count\": 2, \"from\": 20.0,\n                \"child_hist\": {\"buckets\": [ {\"key\": 0.0, \"doc_count\": 2} ]}\n            }\n        ])\n    );\n\n    // Case B: child has more buckets than parent\n    // Parent: terms on text -> 2 buckets\n    // Child: histogram with small interval -> multiple buckets including empties\n    let agg_child_more: Aggregations = serde_json::from_value(json!({\n        \"parent_terms\": {\n            \"terms\": {\"field\": \"text\"},\n            \"aggs\": {\n                \"child_hist\": {\"histogram\": {\"field\": \"score\", \"interval\": 10.0}}\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_child_more, &index)?;\n    assert_eq!(\n        res[\"parent_terms\"],\n        json!({\n            \"buckets\": [\n                {\n                    \"key\": \"cool\",\n                    \"doc_count\": 7,\n                    \"child_hist\": {\n                        \"buckets\": [\n                            {\"key\": 0.0, \"doc_count\": 4},\n                            {\"key\": 10.0, \"doc_count\": 2},\n                            {\"key\": 20.0, \"doc_count\": 0},\n                            {\"key\": 30.0, \"doc_count\": 0},\n                            {\"key\": 40.0, \"doc_count\": 1}\n                        ]\n                    }\n                },\n                {\n                    \"key\": \"nohit\",\n                    \"doc_count\": 2,\n                    \"child_hist\": {\n                        \"buckets\": [\n                            {\"key\": 0.0, \"doc_count\": 1},\n                            {\"key\": 10.0, \"doc_count\": 0},\n                            {\"key\": 20.0, \"doc_count\": 0},\n                            {\"key\": 30.0, \"doc_count\": 0},\n                            {\"key\": 40.0, \"doc_count\": 1}\n                        ]\n                    }\n                }\n            ],\n            \"doc_count_error_upper_bound\": 0,\n            \"sum_other_doc_count\": 0\n        })\n    );\n\n    Ok(())\n}\n\n#[test]\nfn test_date_histogram_as_subagg_parent_more_vs_child_more() -> crate::Result<()> {\n    let index = get_test_index_2_segments(false)?;\n\n    // Case A: parent has more buckets than child\n    // Parent: range with several buckets\n    // Child: date_histogram with 30d -> single bucket per parent\n    let agg_parent_more: Aggregations = serde_json::from_value(json!({\n        \"parent_range\": {\n            \"range\": {\n                \"field\": \"score\",\n                \"ranges\": [\n                    {\"to\": 3.0},\n                    {\"from\": 3.0, \"to\": 7.0},\n                    {\"from\": 7.0, \"to\": 11.0},\n                    {\"from\": 11.0, \"to\": 20.0},\n                    {\"from\": 20.0}\n                ]\n            },\n            \"aggs\": {\n                \"child_date_hist\": {\"date_histogram\": {\"field\": \"date\", \"fixed_interval\": \"30d\"}}\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_parent_more, &index)?;\n    let buckets = res[\"parent_range\"][\"buckets\"].as_array().unwrap();\n    // Verify each parent bucket has exactly one child date bucket with matching doc_count\n    for bucket in buckets {\n        let parent_count = bucket[\"doc_count\"].as_u64().unwrap();\n        let child_buckets = bucket[\"child_date_hist\"][\"buckets\"].as_array().unwrap();\n        assert_eq!(child_buckets.len(), 1);\n        assert_eq!(child_buckets[0][\"doc_count\"], parent_count);\n    }\n\n    // Case B: child has more buckets than parent\n    // Parent: terms on text (2 buckets)\n    // Child: date_histogram with 1d -> multiple buckets\n    let agg_child_more: Aggregations = serde_json::from_value(json!({\n        \"parent_terms\": {\n            \"terms\": {\"field\": \"text\"},\n            \"aggs\": {\n                \"child_date_hist\": {\"date_histogram\": {\"field\": \"date\", \"fixed_interval\": \"1d\"}}\n            }\n        }\n    }))\n    .unwrap();\n    let res = crate::aggregation::tests::exec_request(agg_child_more, &index)?;\n    let buckets = res[\"parent_terms\"][\"buckets\"].as_array().unwrap();\n\n    // cool bucket\n    assert_eq!(buckets[0][\"key\"], \"cool\");\n    let cool_buckets = buckets[0][\"child_date_hist\"][\"buckets\"].as_array().unwrap();\n    assert_eq!(cool_buckets.len(), 3);\n    assert_eq!(cool_buckets[0][\"doc_count\"], 1); // day 0\n    assert_eq!(cool_buckets[1][\"doc_count\"], 4); // day 1\n    assert_eq!(cool_buckets[2][\"doc_count\"], 2); // day 2\n\n    // nohit bucket\n    assert_eq!(buckets[1][\"key\"], \"nohit\");\n    let nohit_buckets = buckets[1][\"child_date_hist\"][\"buckets\"].as_array().unwrap();\n    assert_eq!(nohit_buckets.len(), 2);\n    assert_eq!(nohit_buckets[0][\"doc_count\"], 1); // day 1\n    assert_eq!(nohit_buckets[1][\"doc_count\"], 1); // day 2\n\n    Ok(())\n}\n\nfn get_avg_req(field_name: &str) -> Aggregation {\n    serde_json::from_value(json!({\n        \"avg\": {\n            \"field\": field_name,\n        }\n    }))\n    .unwrap()\n}\n\nfn get_collector(agg_req: Aggregations) -> AggregationCollector {\n    AggregationCollector::from_aggs(agg_req, Default::default())\n}\n\n// *** EVERY BUCKET-TYPE SHOULD BE TESTED HERE ***\n// Note: The flushng part of these  tests are outdated, since the buffering change after converting\n// the collection into one collector per request instead of per bucket.\n//\n// However they are useful as they test a complex aggregation requests.\nfn test_aggregation_flushing(\n    merge_segments: bool,\n    use_distributed_collector: bool,\n) -> crate::Result<()> {\n    let mut values_and_terms = (0..80)\n        .map(|val| vec![(val as f64, \"terma\".to_string())])\n        .collect::<Vec<_>>();\n    values_and_terms.last_mut().unwrap()[0].1 = \"termb\".to_string();\n    let index = get_test_index_from_values_and_terms(merge_segments, &values_and_terms)?;\n\n    let reader = index.reader()?;\n\n    assert_eq!(COLLECT_BLOCK_BUFFER_LEN, 64);\n    // In the tree we cache documents of COLLECT_BLOCK_BUFFER_LEN before passing them down as one\n    // block.\n    //\n    // Build a request so that on the first level we have one full cache, which is then flushed.\n    // The same cache should have some residue docs at the end, which are flushed (Range 0-70)\n    // -> 70 docs\n    //\n    // The second level should also have some residue docs in the cache that are flushed at the\n    // end.\n    //\n    // A second bucket on the first level should have the cache unfilled\n\n    // let elasticsearch_compatible_json_req = r#\"\n    let elasticsearch_compatible_json = json!(\n    {\n    \"bucketsL1\": {\n        \"range\": {\n            \"field\": \"score\",\n            \"ranges\": [ { \"to\": 3.0f64 }, { \"from\": 3.0f64, \"to\": 70.0f64 }, { \"from\": 70.0f64 } ]\n        },\n        \"aggs\": {\n            \"bucketsL2\": {\n                \"range\": {\n                    \"field\": \"score\",\n                    \"ranges\": [ { \"to\": 30.0f64 }, { \"from\": 30.0f64, \"to\": 70.0f64 }, { \"from\": 70.0f64 } ]\n                }\n            }\n        }\n    },\n    \"top_hits_test\":{\n        \"terms\": {\n            \"field\": \"string_id\"\n        },\n        \"aggs\": {\n            \"bucketsL2\": {\n                \"top_hits\": {\n                    \"size\": 2,\n                    \"sort\": [\n                        { \"score\": \"asc\" }\n                    ],\n                    \"docvalue_fields\": [\"score\"]\n                }\n            }\n        }\n    },\n    \"histogram_test\":{\n        \"histogram\": {\n            \"field\": \"score\",\n            \"interval\":  70.0,\n            \"offset\": 3.0\n        },\n        \"aggs\": {\n            \"bucketsL2\": {\n                \"histogram\": {\n                    \"field\": \"score\",\n                    \"interval\":  70.0\n                }\n            }\n        }\n    },\n    \"term_agg_test\":{\n        \"terms\": {\n            \"field\": \"string_id\"\n        },\n        \"aggs\": {\n            \"bucketsL2\": {\n                \"histogram\": {\n                    \"field\": \"score\",\n                    \"interval\":  70.0\n                }\n            }\n        }\n    },\n    \"cardinality_string_id\":{\n        \"cardinality\": {\n            \"field\": \"string_id\"\n        }\n    },\n    \"cardinality_score\":{\n        \"cardinality\": {\n            \"field\": \"score\"\n        }\n    }\n    });\n\n    let agg_req: Aggregations =\n        serde_json::from_str(&serde_json::to_string(&elasticsearch_compatible_json).unwrap())\n            .unwrap();\n\n    let agg_res: AggregationResults = if use_distributed_collector {\n        let collector =\n            DistributedAggregationCollector::from_aggs(agg_req.clone(), Default::default());\n\n        let searcher = reader.searcher();\n        let intermediate_agg_result = searcher.search(&AllQuery, &collector).unwrap();\n\n        // Test postcard roundtrip serialization\n        let intermediate_agg_result_bytes = postcard::to_allocvec(&intermediate_agg_result).expect(\n            \"Postcard Serialization failed, flatten etc. is not supported in the intermediate \\\n             result\",\n        );\n        let intermediate_agg_result: IntermediateAggregationResults =\n            postcard::from_bytes(&intermediate_agg_result_bytes)\n                .expect(\"Post deserialization failed\");\n\n        intermediate_agg_result\n            .into_final_result(agg_req, Default::default())\n            .unwrap()\n    } else {\n        let collector = get_collector(agg_req);\n\n        let searcher = reader.searcher();\n        searcher.search(&AllQuery, &collector).unwrap()\n    };\n\n    let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n    assert_eq!(res[\"bucketsL1\"][\"buckets\"][0][\"doc_count\"], 3);\n    assert_eq!(\n        res[\"bucketsL1\"][\"buckets\"][0][\"bucketsL2\"][\"buckets\"][0][\"doc_count\"],\n        3\n    );\n    assert_eq!(res[\"bucketsL1\"][\"buckets\"][1][\"key\"], \"3-70\");\n    assert_eq!(res[\"bucketsL1\"][\"buckets\"][1][\"doc_count\"], 70 - 3);\n    assert_eq!(\n        res[\"bucketsL1\"][\"buckets\"][1][\"bucketsL2\"][\"buckets\"][0][\"doc_count\"],\n        27\n    );\n    assert_eq!(\n        res[\"bucketsL1\"][\"buckets\"][1][\"bucketsL2\"][\"buckets\"][1][\"doc_count\"],\n        40\n    );\n    assert_eq!(\n        res[\"bucketsL1\"][\"buckets\"][1][\"bucketsL2\"][\"buckets\"][2][\"doc_count\"],\n        0\n    );\n    assert_eq!(\n        res[\"bucketsL1\"][\"buckets\"][2][\"bucketsL2\"][\"buckets\"][2][\"doc_count\"],\n        80 - 70\n    );\n    assert_eq!(res[\"bucketsL1\"][\"buckets\"][2][\"doc_count\"], 80 - 70);\n\n    assert_eq!(\n        res[\"term_agg_test\"],\n        json!(\n        {\n            \"buckets\": [\n              {\n                \"bucketsL2\": {\n                  \"buckets\": [\n                    {\n                      \"doc_count\": 70,\n                      \"key\": 0.0\n                    },\n                    {\n                      \"doc_count\": 9,\n                      \"key\": 70.0\n                    }\n                  ]\n                },\n                \"doc_count\": 79,\n                \"key\": \"terma\"\n              },\n              {\n                \"bucketsL2\": {\n                  \"buckets\": [\n                    {\n                      \"doc_count\": 1,\n                      \"key\": 70.0\n                    }\n                  ]\n                },\n                \"doc_count\": 1,\n                \"key\": \"termb\"\n              }\n            ],\n            \"doc_count_error_upper_bound\": 0,\n            \"sum_other_doc_count\": 0\n          }\n        )\n    );\n\n    assert_eq!(res[\"cardinality_string_id\"][\"value\"], 2.0);\n    assert_eq!(res[\"cardinality_score\"][\"value\"], 80.0);\n\n    Ok(())\n}\n\n#[test]\nfn test_aggregation_flushing_variants() {\n    test_aggregation_flushing(false, false).unwrap();\n    test_aggregation_flushing(false, true).unwrap();\n    test_aggregation_flushing(true, false).unwrap();\n    test_aggregation_flushing(true, true).unwrap();\n}\n\n#[test]\nfn test_aggregation_level1_simple() -> crate::Result<()> {\n    let index = get_test_index_2_segments(true)?;\n\n    let reader = index.reader()?;\n    let text_field = reader.searcher().schema().get_field(\"text\").unwrap();\n\n    let term_query = TermQuery::new(\n        Term::from_field_text(text_field, \"cool\"),\n        IndexRecordOption::Basic,\n    );\n\n    let range_agg = |field_name: &str| -> Aggregation {\n        serde_json::from_value(json!({\n            \"range\": {\n                \"field\": field_name,\n                \"ranges\": [ { \"from\": 3.0f64, \"to\": 7.0f64 }, { \"from\": 7.0f64, \"to\": 20.0f64 } ]\n            }\n        }))\n        .unwrap()\n    };\n\n    let agg_req_1: Aggregations = vec![\n        (\"average\".to_string(), get_avg_req(\"score\")),\n        (\"range\".to_string(), range_agg(\"score\")),\n    ]\n    .into_iter()\n    .collect();\n\n    let collector = get_collector(agg_req_1);\n\n    let searcher = reader.searcher();\n    let agg_res: AggregationResults = searcher.search(&term_query, &collector).unwrap();\n\n    let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n    assert_eq!(res[\"average\"][\"value\"], 12.142857142857142);\n    assert_eq!(\n        res[\"range\"][\"buckets\"],\n        json!(\n        [\n        {\n          \"key\": \"*-3\",\n          \"doc_count\": 1,\n          \"to\": 3.0\n        },\n        {\n          \"key\": \"3-7\",\n          \"doc_count\": 2,\n          \"from\": 3.0,\n          \"to\": 7.0\n        },\n        {\n          \"key\": \"7-20\",\n          \"doc_count\": 3,\n          \"from\": 7.0,\n          \"to\": 20.0\n        },\n        {\n          \"key\": \"20-*\",\n          \"doc_count\": 1,\n          \"from\": 20.0\n        }\n        ])\n    );\n\n    Ok(())\n}\n\n#[test]\nfn test_aggregation_level1() -> crate::Result<()> {\n    let index = get_test_index_2_segments(true)?;\n\n    let reader = index.reader()?;\n    let text_field = reader.searcher().schema().get_field(\"text\").unwrap();\n\n    let term_query = TermQuery::new(\n        Term::from_field_text(text_field, \"cool\"),\n        IndexRecordOption::Basic,\n    );\n\n    let range_agg = |field_name: &str| -> Aggregation {\n        serde_json::from_value(json!({\n            \"range\": {\n                \"field\": field_name,\n                \"ranges\": [ { \"from\": 3.0f64, \"to\": 7.0f64 }, { \"from\": 7.0f64, \"to\": 20.0f64 } ]\n            }\n        }))\n        .unwrap()\n    };\n\n    let agg_req_1: Aggregations = vec![\n        (\"average_i64\".to_string(), get_avg_req(\"score_i64\")),\n        (\"average_f64\".to_string(), get_avg_req(\"score_f64\")),\n        (\"average\".to_string(), get_avg_req(\"score\")),\n        (\"range\".to_string(), range_agg(\"score\")),\n        (\"rangef64\".to_string(), range_agg(\"score_f64\")),\n        (\"rangei64\".to_string(), range_agg(\"score_i64\")),\n    ]\n    .into_iter()\n    .collect();\n\n    let collector = get_collector(agg_req_1);\n\n    let searcher = reader.searcher();\n    let agg_res: AggregationResults = searcher.search(&term_query, &collector).unwrap();\n\n    let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n    assert_eq!(res[\"average\"][\"value\"], 12.142857142857142);\n    assert_eq!(res[\"average_f64\"][\"value\"], 12.214285714285714);\n    assert_eq!(res[\"average_i64\"][\"value\"], 12.142857142857142);\n    assert_eq!(\n        res[\"range\"][\"buckets\"],\n        json!(\n        [\n        {\n          \"key\": \"*-3\",\n          \"doc_count\": 1,\n          \"to\": 3.0\n        },\n        {\n          \"key\": \"3-7\",\n          \"doc_count\": 2,\n          \"from\": 3.0,\n          \"to\": 7.0\n        },\n        {\n          \"key\": \"7-20\",\n          \"doc_count\": 3,\n          \"from\": 7.0,\n          \"to\": 20.0\n        },\n        {\n          \"key\": \"20-*\",\n          \"doc_count\": 1,\n          \"from\": 20.0\n        }\n        ])\n    );\n\n    Ok(())\n}\n\nfn test_aggregation_level2(\n    merge_segments: bool,\n    use_distributed_collector: bool,\n) -> crate::Result<()> {\n    let index = get_test_index_2_segments(merge_segments)?;\n\n    let reader = index.reader()?;\n    let text_field = reader.searcher().schema().get_field(\"text\").unwrap();\n\n    let term_query = TermQuery::new(\n        Term::from_field_text(text_field, \"cool\"),\n        IndexRecordOption::Basic,\n    );\n\n    let query_with_no_hits = TermQuery::new(\n        Term::from_field_text(text_field, \"thistermdoesnotexist\"),\n        IndexRecordOption::Basic,\n    );\n\n    let elasticsearch_compatible_json_req = r#\"\n{\n  \"rangef64\": {\n    \"range\": {\n      \"field\": \"score_f64\",\n      \"ranges\": [\n        { \"to\": 3.0 },\n        { \"from\": 3.0, \"to\": 7.0 },\n        { \"from\": 7.0, \"to\": 19.0 },\n        { \"from\": 19.0, \"to\": 20.0 },\n        { \"from\": 20.0 }\n      ]\n    },\n    \"aggs\": {\n      \"average_in_range\": { \"avg\": { \"field\": \"score\" } },\n      \"term_agg\": { \"terms\": { \"field\": \"text\" } }\n    }\n  },\n  \"rangei64\": {\n    \"range\": {\n      \"field\": \"score_i64\",\n      \"ranges\": [\n        { \"to\": 3.0 },\n        { \"from\": 3.0, \"to\": 7.0 },\n        { \"from\": 7.0, \"to\": 19.0 },\n        { \"from\": 19.0, \"to\": 20.0 },\n        { \"from\": 20.0 }\n      ]\n    },\n    \"aggs\": {\n      \"average_in_range\": { \"avg\": { \"field\": \"score\" } },\n      \"term_agg\": { \"terms\": { \"field\": \"text\" } }\n    }\n  },\n  \"average\": {\n    \"avg\": { \"field\": \"score\" }\n  },\n  \"range\": {\n    \"range\": {\n      \"field\": \"score\",\n      \"ranges\": [\n        { \"to\": 3.0 },\n        { \"from\": 3.0, \"to\": 7.0 },\n        { \"from\": 7.0, \"to\": 19.0 },\n        { \"from\": 19.0, \"to\": 20.0 },\n        { \"from\": 20.0 }\n      ]\n    },\n    \"aggs\": {\n      \"average_in_range\": { \"avg\": { \"field\": \"score\" } },\n      \"term_agg\": { \"terms\": { \"field\": \"text\" } }\n    }\n  }\n}\n\"#;\n    let agg_req: Aggregations = serde_json::from_str(elasticsearch_compatible_json_req).unwrap();\n\n    let agg_res: AggregationResults = if use_distributed_collector {\n        let collector =\n            DistributedAggregationCollector::from_aggs(agg_req.clone(), Default::default());\n\n        let searcher = reader.searcher();\n        let res = searcher.search(&term_query, &collector).unwrap();\n        res.into_final_result(agg_req.clone(), Default::default())\n            .unwrap()\n    } else {\n        let collector = get_collector(agg_req.clone());\n\n        let searcher = reader.searcher();\n        searcher.search(&term_query, &collector).unwrap()\n    };\n\n    let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n    assert_eq!(res[\"range\"][\"buckets\"][1][\"key\"], \"3-7\");\n    assert_eq!(res[\"range\"][\"buckets\"][1][\"doc_count\"], 2u64);\n    assert_eq!(res[\"rangef64\"][\"buckets\"][1][\"doc_count\"], 2u64);\n    assert_eq!(res[\"rangei64\"][\"buckets\"][1][\"doc_count\"], 2u64);\n\n    assert_eq!(res[\"average\"][\"value\"], 12.142857142857142f64);\n    assert_eq!(res[\"range\"][\"buckets\"][2][\"key\"], \"7-19\");\n    assert_eq!(res[\"range\"][\"buckets\"][2][\"doc_count\"], 3u64);\n    assert_eq!(res[\"rangef64\"][\"buckets\"][2][\"doc_count\"], 3u64);\n    assert_eq!(res[\"rangei64\"][\"buckets\"][2][\"doc_count\"], 3u64);\n    assert_eq!(res[\"rangei64\"][\"buckets\"][5], serde_json::Value::Null);\n\n    assert_eq!(res[\"range\"][\"buckets\"][4][\"key\"], \"20-*\");\n    assert_eq!(res[\"range\"][\"buckets\"][4][\"doc_count\"], 1u64);\n    assert_eq!(res[\"rangef64\"][\"buckets\"][4][\"doc_count\"], 1u64);\n    assert_eq!(res[\"rangei64\"][\"buckets\"][4][\"doc_count\"], 1u64);\n\n    assert_eq!(res[\"range\"][\"buckets\"][3][\"key\"], \"19-20\");\n    assert_eq!(res[\"range\"][\"buckets\"][3][\"doc_count\"], 0u64);\n    assert_eq!(res[\"rangef64\"][\"buckets\"][3][\"doc_count\"], 0u64);\n    assert_eq!(res[\"rangei64\"][\"buckets\"][3][\"doc_count\"], 0u64);\n\n    assert_eq!(\n        res[\"range\"][\"buckets\"][3][\"average_in_range\"][\"value\"],\n        serde_json::Value::Null\n    );\n\n    assert_eq!(\n        res[\"range\"][\"buckets\"][4][\"average_in_range\"][\"value\"],\n        44.0f64\n    );\n    assert_eq!(\n        res[\"rangef64\"][\"buckets\"][4][\"average_in_range\"][\"value\"],\n        44.0f64\n    );\n    assert_eq!(\n        res[\"rangei64\"][\"buckets\"][4][\"average_in_range\"][\"value\"],\n        44.0f64\n    );\n\n    assert_eq!(\n        res[\"range\"][\"7-19\"][\"average_in_range\"][\"value\"],\n        res[\"rangef64\"][\"7-19\"][\"average_in_range\"][\"value\"]\n    );\n    assert_eq!(\n        res[\"range\"][\"7-19\"][\"average_in_range\"][\"value\"],\n        res[\"rangei64\"][\"7-19\"][\"average_in_range\"][\"value\"]\n    );\n\n    // Test empty result set\n    let collector = get_collector(agg_req);\n    let searcher = reader.searcher();\n    searcher.search(&query_with_no_hits, &collector).unwrap();\n\n    Ok(())\n}\n\n#[test]\nfn test_aggregation_level2_multi_segments() -> crate::Result<()> {\n    test_aggregation_level2(false, false)\n}\n\n#[test]\nfn test_aggregation_level2_single_segment() -> crate::Result<()> {\n    test_aggregation_level2(true, false)\n}\n\n#[test]\nfn test_aggregation_level2_multi_segments_distributed_collector() -> crate::Result<()> {\n    test_aggregation_level2(false, true)\n}\n\n#[test]\nfn test_aggregation_level2_single_segment_distributed_collector() -> crate::Result<()> {\n    test_aggregation_level2(true, true)\n}\n\n#[test]\nfn test_aggregation_invalid_requests() -> crate::Result<()> {\n    let index = get_test_index_2_segments(false)?;\n\n    let reader = index.reader()?;\n\n    let avg_on_field = |field_name: &str| {\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"average\": {\n                \"avg\": {\n                    \"field\": field_name,\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = get_collector(agg_req_1);\n\n        let searcher = reader.searcher();\n\n        searcher.search(&AllQuery, &collector)\n    };\n\n    let agg_res = avg_on_field(\"dummy_text\").unwrap_err();\n    assert_eq!(\n        format!(\"{agg_res:?}\"),\n        r#\"InvalidArgument(\"Field \\\"dummy_text\\\" is not configured as fast field\")\"#\n    );\n\n    let agg_req_1: Result<Aggregations, serde_json::Error> = serde_json::from_value(json!({\n        \"average\": {\n            \"avg\": {\n                \"fieldd\": \"a\",\n            },\n        }\n    }));\n\n    assert_eq!(agg_req_1.is_err(), true);\n    assert_eq!(agg_req_1.unwrap_err().to_string(), \"missing field `field`\");\n\n    let agg_req_1: Result<Aggregations, serde_json::Error> = serde_json::from_value(json!({\n        \"average\": {\n            \"doesnotmatchanyagg\": {\n                \"field\": \"a\",\n            },\n        }\n    }));\n\n    assert_eq!(agg_req_1.is_err(), true);\n    // TODO: This should list valid values\n    assert!(agg_req_1\n        .unwrap_err()\n        .to_string()\n        .contains(\"unknown variant `doesnotmatchanyagg`, expected one of\"));\n\n    // TODO: This should return an error\n    // let agg_res = avg_on_field(\"not_exist_field\").unwrap_err();\n    // assert_eq!(\n    // format!(\"{:?}\", agg_res),\n    // r#\"InvalidArgument(\"No fast field found for field: not_exist_field\")\"#\n    //);\n\n    // TODO: This should return an error\n    // let agg_res = avg_on_field(\"ip_addr\").unwrap_err();\n    // assert_eq!(\n    // format!(\"{:?}\", agg_res),\n    // r#\"InvalidArgument(\"No fast field found for field: ip_addr\")\"#\n    //);\n\n    Ok(())\n}\n\n#[test]\nfn test_aggregation_on_json_object() {\n    let mut schema_builder = Schema::builder();\n    let json = schema_builder.add_json_field(\"json\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"red\"})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"red\"})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"blue\"})))\n        .unwrap();\n    index_writer.commit().unwrap();\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n\n    let agg: Aggregations = serde_json::from_value(json!({\n        \"jsonagg\": {\n            \"terms\": {\n                \"field\": \"json.color\",\n            }\n        }\n    }))\n    .unwrap();\n\n    let aggregation_collector = get_collector(agg);\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!({\n            \"jsonagg\": {\n                \"buckets\": [\n                    {\"doc_count\": 2, \"key\": \"red\"},\n                    {\"doc_count\": 1, \"key\": \"blue\"},\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n            }\n        })\n    );\n}\n\n#[test]\nfn test_aggregation_on_nested_json_object() {\n    let mut schema_builder = Schema::builder();\n    let json = schema_builder.add_json_field(\"json.blub\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color.dot\": \"red\", \"color\": {\"nested\":\"red\"} })))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color.dot\": \"blue\", \"color\": {\"nested\":\"blue\"} })))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color.dot\": \"blue\", \"color\": {\"nested\":\"blue\"} })))\n        .unwrap();\n    index_writer.commit().unwrap();\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n\n    let agg: Aggregations = serde_json::from_value(json!({\n        \"jsonagg1\": {\n            \"terms\": {\n                \"field\": \"json\\\\.blub.color\\\\.dot\",\n            }\n        },\n        \"jsonagg2\": {\n            \"terms\": {\n                \"field\": \"json\\\\.blub.color.nested\",\n            }\n        }\n\n    }))\n    .unwrap();\n\n    let aggregation_collector = get_collector(agg);\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!({\n            \"jsonagg1\": {\n                \"buckets\": [\n                    {\"doc_count\": 2, \"key\": \"blue\"},\n                    {\"doc_count\": 1, \"key\": \"red\"}\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n            },\n            \"jsonagg2\": {\n                \"buckets\": [\n                    {\"doc_count\": 2, \"key\": \"blue\"},\n                    {\"doc_count\": 1, \"key\": \"red\"}\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n            }\n\n        })\n    );\n}\n\n#[test]\nfn test_aggregation_on_json_object_empty_columns() {\n    let mut schema_builder = Schema::builder();\n    let json = schema_builder.add_json_field(\"json\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n    // => Empty column when accessing color\n    index_writer\n        .add_document(doc!(json => json!({\"price\": 10.0})))\n        .unwrap();\n    index_writer.commit().unwrap();\n    // => Empty column when accessing price\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"blue\"})))\n        .unwrap();\n    index_writer.commit().unwrap();\n\n    // => Non Empty columns\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"red\", \"price\": 10.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"red\", \"price\": 10.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"green\", \"price\": 20.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"green\", \"price\": 20.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"color\": \"green\", \"price\": 20.0})))\n        .unwrap();\n\n    index_writer.commit().unwrap();\n\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n\n    let agg: Aggregations = serde_json::from_value(json!({\n        \"jsonagg\": {\n            \"terms\": {\n                \"field\": \"json.color\",\n            }\n        }\n    }))\n    .unwrap();\n\n    let aggregation_collector = get_collector(agg);\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!({\n            \"jsonagg\": {\n                \"buckets\": [\n                    {\"doc_count\": 3, \"key\": \"green\"},\n                    {\"doc_count\": 2, \"key\": \"red\"},\n                    {\"doc_count\": 1, \"key\": \"blue\"}\n                ],\n                \"doc_count_error_upper_bound\": 0,\n                \"sum_other_doc_count\": 0\n            }\n        })\n    );\n\n    let agg_req_str = r#\"\n    {\n      \"jsonagg\": {\n        \"aggs\": {\n          \"min_price\": { \"min\": { \"field\": \"json.price\" } }\n        },\n        \"terms\": {\n          \"field\": \"json.color\",\n          \"order\": { \"min_price\": \"desc\" }\n        }\n      }\n    } \"#;\n    let agg: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n    let aggregation_collector = get_collector(agg);\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!(\n            {\n              \"jsonagg\": {\n                \"buckets\": [\n                  {\n                    \"key\": \"green\",\n                    \"doc_count\": 3,\n                    \"min_price\": {\n                      \"value\": 20.0\n                    }\n                  },\n                  {\n                    \"key\": \"red\",\n                    \"doc_count\": 2,\n                    \"min_price\": {\n                      \"value\": 10.0\n                    }\n                  },\n                  {\n                    \"key\": \"blue\",\n                    \"doc_count\": 1,\n                    \"min_price\": {\n                      \"value\": null\n                    }\n                  }\n                ],\n                \"sum_other_doc_count\": 0\n              }\n            }\n        )\n    );\n}\n\n#[test]\nfn test_aggregation_on_json_object_mixed_types() {\n    let mut schema_builder = Schema::builder();\n    let json = schema_builder.add_json_field(\"json\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n    // => Segment with all values numeric\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": 10.0, \"mixed_price\": 10.0})))\n        .unwrap();\n    index_writer.commit().unwrap();\n    // => Segment with all values text\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": \"blue\", \"mixed_price\": 5.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": \"blue\", \"mixed_price\": 5.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": \"blue\", \"mixed_price\": 5.0})))\n        .unwrap();\n    index_writer.commit().unwrap();\n    // => Segment with all boolean\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": true, \"mixed_price\": \"no_price\"})))\n        .unwrap();\n    index_writer.commit().unwrap();\n\n    // => Segment with mixed values\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": \"red\", \"mixed_price\": 1.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": \"red\", \"mixed_price\": 1.0})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": -20.5, \"mixed_price\": -20.5})))\n        .unwrap();\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_type\": true, \"mixed_price\": \"no_price\"})))\n        .unwrap();\n\n    index_writer.commit().unwrap();\n\n    // All bucket types\n    let agg_req_str = r#\"\n    {\n        \"termagg\": {\n            \"terms\": {\n                \"field\": \"json.mixed_type\",\n                \"order\": { \"min_price\": \"desc\" }\n            },\n            \"aggs\": {\n                \"min_price\": { \"min\": { \"field\": \"json.mixed_price\" } }\n            }\n        },\n        \"rangeagg\": {\n            \"range\": {\n                \"field\": \"json.mixed_type\",\n                \"ranges\": [\n                    { \"to\": 3.0 },\n                    { \"from\": 19.0, \"to\": 20.0 },\n                    { \"from\": 20.0 }\n                ]\n            },\n            \"aggs\": {\n                \"average_in_range\": { \"avg\": { \"field\": \"json.mixed_type\" } }\n            }\n        }\n    } \"#;\n    let agg: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n    let aggregation_collector = get_collector(agg);\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    use pretty_assertions::assert_eq;\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!({\n          \"rangeagg\": {\n            \"buckets\": [\n              { \"average_in_range\": { \"value\": -20.5 }, \"doc_count\": 1, \"key\": \"*-3\", \"to\": 3.0 },\n              { \"average_in_range\": { \"value\": 10.0 }, \"doc_count\": 1, \"from\": 3.0, \"key\": \"3-19\", \"to\": 19.0 },\n              { \"average_in_range\": { \"value\": null }, \"doc_count\": 0, \"from\": 19.0, \"key\": \"19-20\", \"to\": 20.0 },\n              { \"average_in_range\": { \"value\": null }, \"doc_count\": 0, \"from\": 20.0, \"key\": \"20-*\" }\n            ]\n          },\n          \"termagg\": {\n            \"buckets\": [\n              { \"doc_count\": 1, \"key\": 10, \"min_price\": { \"value\": 10.0 } },\n              { \"doc_count\": 3, \"key\": \"blue\", \"min_price\": { \"value\": 5.0 } },\n              { \"doc_count\": 2, \"key\": \"red\", \"min_price\": { \"value\": 1.0 } },\n              { \"doc_count\": 1, \"key\": -20.5, \"min_price\": { \"value\": -20.5 } },\n              { \"doc_count\": 2, \"key\": 1, \"key_as_string\": \"true\", \"min_price\": { \"value\": null } },\n            ],\n            \"sum_other_doc_count\": 0\n          }\n        }\n        )\n    );\n}\n\n#[test]\nfn test_aggregation_on_json_object_mixed_numerical_segments() {\n    let mut schema_builder = Schema::builder();\n    let json = schema_builder.add_json_field(\"json\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n    // => Segment with all values f64 numeric\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_price\": 10.5})))\n        .unwrap();\n    // Gets converted to f64!\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_price\": 10})))\n        .unwrap();\n    index_writer.commit().unwrap();\n    // => Segment with all values i64 numeric\n    index_writer\n        .add_document(doc!(json => json!({\"mixed_price\": 10})))\n        .unwrap();\n    index_writer.commit().unwrap();\n\n    index_writer.commit().unwrap();\n\n    // All bucket types\n    let agg_req_str = r#\"\n    {\n        \"termagg\": {\n            \"terms\": {\n                \"field\": \"json.mixed_price\"\n            }\n        }\n    } \"#;\n    let agg: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n    let aggregation_collector = get_collector(agg);\n    let reader = index.reader().unwrap();\n    let searcher = reader.searcher();\n\n    let aggregation_results = searcher.search(&AllQuery, &aggregation_collector).unwrap();\n    let aggregation_res_json = serde_json::to_value(aggregation_results).unwrap();\n    use pretty_assertions::assert_eq;\n    assert_eq!(\n        &aggregation_res_json,\n        &serde_json::json!({\n          \"termagg\": {\n            \"buckets\": [\n              { \"doc_count\": 2, \"key\": 10},\n              { \"doc_count\": 1, \"key\": 10.5},\n            ],\n            \"doc_count_error_upper_bound\": 0,\n            \"sum_other_doc_count\": 0\n          }\n        }\n        )\n    );\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/accessors.rs",
    "content": "use std::net::Ipv6Addr;\n\nuse columnar::column_values::{CompactHit, CompactSpaceU64Accessor};\nuse columnar::{Column, ColumnType, MonotonicallyMappableToU64, StrColumn, TermOrdHit};\n\nuse crate::aggregation::accessor_helpers::get_numeric_or_date_column_types;\nuse crate::aggregation::bucket::composite::numeric_types::num_proj;\nuse crate::aggregation::bucket::composite::numeric_types::num_proj::ProjectedNumber;\nuse crate::aggregation::bucket::composite::ToTypePaginationOrder;\nuse crate::aggregation::bucket::{\n    parse_into_milliseconds, CalendarInterval, CompositeAggregation, CompositeAggregationSource,\n    MissingOrder, Order,\n};\nuse crate::aggregation::intermediate_agg_result::CompositeIntermediateKey;\nuse crate::{SegmentReader, TantivyError};\n\n/// Contains all information required by the SegmentCompositeCollector to perform the\n/// composite aggregation on a segment.\npub struct CompositeAggReqData {\n    /// The name of the aggregation.\n    pub name: String,\n    /// The normalized term aggregation request.\n    pub req: CompositeAggregation,\n    /// Accessors for each source, each source can have multiple accessors (columns).\n    pub composite_accessors: Vec<CompositeSourceAccessors>,\n}\n\nimpl CompositeAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n            + self.composite_accessors.len() * std::mem::size_of::<CompositeSourceAccessors>()\n    }\n}\n\n/// Accessors for a single column in a composite source.\npub struct CompositeAccessor {\n    /// The fast field column\n    pub column: Column<u64>,\n    /// The column type\n    pub column_type: ColumnType,\n    /// Term dictionary if the column type is Str\n    ///\n    /// Only used by term sources\n    pub str_dict_column: Option<StrColumn>,\n    /// Parsed date interval for date histogram sources\n    pub date_histogram_interval: PrecomputedDateInterval,\n}\n\n/// Accessors to all the columns that belong to the field of a composite source.\npub struct CompositeSourceAccessors {\n    /// The accessors for this source\n    pub accessors: Vec<CompositeAccessor>,\n    /// The key after which to start collecting results. Applies to the first\n    /// column of the source.\n    pub after_key: PrecomputedAfterKey,\n\n    /// The column index the after_key applies to. The after_key only applies to\n    /// one column. Columns before should be skipped. Columns after should be\n    /// kept without comparison to the after_key.\n    pub after_key_accessor_idx: usize,\n\n    /// Whether to skip missing values because of the after_key. Skipping only\n    /// applies if the value for previous columns were exactly equal to the\n    /// corresponding after keys (is_on_after_key).\n    pub skip_missing: bool,\n\n    /// The after key was set to null to indicate that the last collected key\n    /// was a missing value.\n    pub is_after_key_explicit_missing: bool,\n}\n\nimpl CompositeSourceAccessors {\n    /// Creates a new set of accessors for the composite source.\n    ///\n    /// Precomputes some values to make collection faster.\n    pub fn build_for_source(\n        reader: &SegmentReader,\n        source: &CompositeAggregationSource,\n        // First option is None when no after key was set in the query, the\n        // second option is None when the after key was set but its value for\n        // this source was set to `null`\n        source_after_key_opt: Option<&CompositeIntermediateKey>,\n    ) -> crate::Result<Self> {\n        let is_after_key_explicit_missing = source_after_key_opt\n            .map(|after_key| matches!(after_key, CompositeIntermediateKey::Null))\n            .unwrap_or(false);\n        let mut skip_missing = false;\n        if let Some(CompositeIntermediateKey::Null) = source_after_key_opt {\n            if !source.missing_bucket() {\n                return Err(TantivyError::InvalidArgument(\n                    \"the 'after' key for a source cannot be null when 'missing_bucket' is false\"\n                        .to_string(),\n                ));\n            }\n        } else if source_after_key_opt.is_some() {\n            // if missing buckets come first and we have a non null after key, we skip missing\n            if MissingOrder::First == source.missing_order() {\n                skip_missing = true;\n            }\n            if MissingOrder::Default == source.missing_order() && Order::Asc == source.order() {\n                skip_missing = true;\n            }\n        };\n\n        match source {\n            CompositeAggregationSource::Terms(source) => {\n                let allowed_column_types = [\n                    ColumnType::I64,\n                    ColumnType::U64,\n                    ColumnType::F64,\n                    ColumnType::Str,\n                    ColumnType::DateTime,\n                    ColumnType::Bool,\n                    ColumnType::IpAddr,\n                    // ColumnType::Bytes Unsupported\n                ];\n                let mut columns_and_types = reader\n                    .fast_fields()\n                    .u64_lenient_for_type_all(Some(&allowed_column_types), &source.field)?;\n\n                // Sort columns by their pagination order and determine which to skip\n                columns_and_types.sort_by_key(|(_, col_type): &(Column, ColumnType)| {\n                    col_type.column_pagination_order()\n                });\n                if source.order == Order::Desc {\n                    columns_and_types.reverse();\n                }\n                let after_key_accessor_idx = find_first_column_to_collect(\n                    &columns_and_types,\n                    source_after_key_opt,\n                    source.missing_order,\n                    source.order,\n                )?;\n\n                let source_collectors: Vec<CompositeAccessor> = columns_and_types\n                    .into_iter()\n                    .map(|(column, column_type)| {\n                        Ok(CompositeAccessor {\n                            column,\n                            column_type,\n                            str_dict_column: reader.fast_fields().str(&source.field)?,\n                            date_histogram_interval: PrecomputedDateInterval::NotApplicable,\n                        })\n                    })\n                    .collect::<crate::Result<_>>()?;\n\n                let after_key = if let Some(first_col) =\n                    source_collectors.get(after_key_accessor_idx)\n                {\n                    match source_after_key_opt {\n                        Some(after_key) => PrecomputedAfterKey::precompute(\n                            first_col,\n                            after_key,\n                            &source.field,\n                            source.missing_order,\n                            source.order,\n                        )?,\n                        None => {\n                            precompute_missing_after_key(false, source.missing_order, source.order)\n                        }\n                    }\n                } else {\n                    // if no columns, we don't care about the after_key\n                    PrecomputedAfterKey::Next(0)\n                };\n\n                Ok(CompositeSourceAccessors {\n                    accessors: source_collectors,\n                    is_after_key_explicit_missing,\n                    skip_missing,\n                    after_key,\n                    after_key_accessor_idx,\n                })\n            }\n            CompositeAggregationSource::Histogram(source) => {\n                let column_and_types: Vec<(Column, ColumnType)> =\n                    reader.fast_fields().u64_lenient_for_type_all(\n                        Some(get_numeric_or_date_column_types()),\n                        &source.field,\n                    )?;\n                let source_collectors: Vec<CompositeAccessor> = column_and_types\n                    .into_iter()\n                    .map(|(column, column_type)| {\n                        Ok(CompositeAccessor {\n                            column,\n                            column_type,\n                            str_dict_column: None,\n                            date_histogram_interval: PrecomputedDateInterval::NotApplicable,\n                        })\n                    })\n                    .collect::<crate::Result<_>>()?;\n                let after_key = match source_after_key_opt {\n                    Some(CompositeIntermediateKey::F64(key)) => {\n                        let normalized_key = *key / source.interval;\n                        num_proj::f64_to_i64(normalized_key).into()\n                    }\n                    Some(CompositeIntermediateKey::Null) => {\n                        precompute_missing_after_key(true, source.missing_order, source.order)\n                    }\n                    None => precompute_missing_after_key(true, source.missing_order, source.order),\n                    _ => {\n                        return Err(crate::TantivyError::InvalidArgument(\n                            \"After key type invalid for interval composite source\".to_string(),\n                        ));\n                    }\n                };\n                Ok(CompositeSourceAccessors {\n                    accessors: source_collectors,\n                    is_after_key_explicit_missing,\n                    skip_missing,\n                    after_key,\n                    after_key_accessor_idx: 0,\n                })\n            }\n            CompositeAggregationSource::DateHistogram(source) => {\n                let column_and_types = reader\n                    .fast_fields()\n                    .u64_lenient_for_type_all(Some(&[ColumnType::DateTime]), &source.field)?;\n                let date_histogram_interval =\n                    PrecomputedDateInterval::from_date_histogram_source_intervals(\n                        &source.fixed_interval,\n                        source.calendar_interval,\n                    )?;\n                let source_collectors: Vec<CompositeAccessor> = column_and_types\n                    .into_iter()\n                    .map(|(column, column_type)| {\n                        Ok(CompositeAccessor {\n                            column,\n                            column_type,\n                            str_dict_column: None,\n                            date_histogram_interval,\n                        })\n                    })\n                    .collect::<crate::Result<_>>()?;\n                let after_key = match source_after_key_opt {\n                    Some(CompositeIntermediateKey::DateTime(key)) => {\n                        PrecomputedAfterKey::Exact(key.to_u64())\n                    }\n                    Some(CompositeIntermediateKey::Null) => {\n                        precompute_missing_after_key(true, source.missing_order, source.order)\n                    }\n                    None => precompute_missing_after_key(true, source.missing_order, source.order),\n                    _ => {\n                        return Err(crate::TantivyError::InvalidArgument(\n                            \"After key type invalid for interval composite source\".to_string(),\n                        ));\n                    }\n                };\n                Ok(CompositeSourceAccessors {\n                    accessors: source_collectors,\n                    is_after_key_explicit_missing,\n                    skip_missing,\n                    after_key,\n                    after_key_accessor_idx: 0,\n                })\n            }\n        }\n    }\n}\n\n/// Finds the index of the first column we should start collecting from to\n/// resume the pagination from the after_key.\nfn find_first_column_to_collect<T>(\n    sorted_columns: &[(T, ColumnType)],\n    after_key_opt: Option<&CompositeIntermediateKey>,\n    missing_order: MissingOrder,\n    order: Order,\n) -> crate::Result<usize> {\n    let after_key = match after_key_opt {\n        None => return Ok(0), // No pagination, start from beginning\n        Some(key) => key,\n    };\n    // Handle null after_key (we were on a missing value last time)\n    if matches!(after_key, CompositeIntermediateKey::Null) {\n        return match (missing_order, order) {\n            // Missing values come first, so all columns remain\n            (MissingOrder::First, _) | (MissingOrder::Default, Order::Asc) => Ok(0),\n            // Missing values come last, so all columns are done\n            (MissingOrder::Last, _) | (MissingOrder::Default, Order::Desc) => {\n                Ok(sorted_columns.len())\n            }\n        };\n    }\n    // Find the first column whose type order matches or follows the after_key's\n    // type in the pagination sequence\n    let after_key_column_order = after_key.column_pagination_order();\n    for (idx, (_, col_type)) in sorted_columns.iter().enumerate() {\n        let col_order = col_type.column_pagination_order();\n        let is_first_to_collect = match order {\n            Order::Asc => col_order >= after_key_column_order,\n            Order::Desc => col_order <= after_key_column_order,\n        };\n        if is_first_to_collect {\n            return Ok(idx);\n        }\n    }\n    // All columns are before the after_key, nothing left to collect\n    Ok(sorted_columns.len())\n}\n\nfn precompute_missing_after_key(\n    is_after_key_explicit_missing: bool,\n    missing_order: MissingOrder,\n    order: Order,\n) -> PrecomputedAfterKey {\n    let after_last = PrecomputedAfterKey::AfterLast;\n    let before_first = PrecomputedAfterKey::Next(0);\n    match (is_after_key_explicit_missing, missing_order, order) {\n        (true, MissingOrder::First, Order::Asc) => before_first,\n        (true, MissingOrder::First, Order::Desc) => after_last,\n        (true, MissingOrder::Last, Order::Asc) => after_last,\n        (true, MissingOrder::Last, Order::Desc) => before_first,\n        (true, MissingOrder::Default, Order::Asc) => before_first,\n        (true, MissingOrder::Default, Order::Desc) => after_last,\n        (false, _, Order::Asc) => before_first,\n        (false, _, Order::Desc) => after_last,\n    }\n}\n\n/// A parsed representation of the date interval for date histogram sources\n#[derive(Clone, Copy, Debug)]\npub enum PrecomputedDateInterval {\n    /// This is not a date histogram source\n    NotApplicable,\n    /// Source was configured with a fixed interval\n    FixedNanoseconds(i64),\n    /// Source was configured with a calendar interval\n    Calendar(CalendarInterval),\n}\n\nimpl PrecomputedDateInterval {\n    /// Validates the date histogram source interval fields and parses a date interval from them.\n    pub fn from_date_histogram_source_intervals(\n        fixed_interval: &Option<String>,\n        calendar_interval: Option<CalendarInterval>,\n    ) -> crate::Result<Self> {\n        match (fixed_interval, calendar_interval) {\n            (Some(_), Some(_)) | (None, None) => Err(TantivyError::InvalidArgument(\n                \"date histogram source must one and only one of fixed_interval or \\\n                 calendar_interval set\"\n                    .to_string(),\n            )),\n            (Some(fixed_interval), None) => {\n                let fixed_interval_ms = parse_into_milliseconds(fixed_interval)?;\n                Ok(PrecomputedDateInterval::FixedNanoseconds(\n                    fixed_interval_ms * 1_000_000,\n                ))\n            }\n            (None, Some(calendar_interval)) => {\n                Ok(PrecomputedDateInterval::Calendar(calendar_interval))\n            }\n        }\n    }\n}\n\n/// The after key projected to the u64 column space\n///\n/// Some column types (term, IP) might not have an exact representation of the\n/// specified after key\n#[derive(Debug)]\npub enum PrecomputedAfterKey {\n    /// The after key could be exactly represented in the column space.\n    Exact(u64),\n    /// The after key could not be exactly represented exactly represented, so\n    /// this is the next closest one.\n    Next(u64),\n    /// The after key could not be represented in the column space, it is\n    /// greater than all value\n    AfterLast,\n}\n\nimpl From<CompactHit> for PrecomputedAfterKey {\n    fn from(hit: CompactHit) -> Self {\n        match hit {\n            CompactHit::Exact(ord) => PrecomputedAfterKey::Exact(ord as u64),\n            CompactHit::Next(ord) => PrecomputedAfterKey::Next(ord as u64),\n            CompactHit::AfterLast => PrecomputedAfterKey::AfterLast,\n        }\n    }\n}\n\nimpl From<TermOrdHit> for PrecomputedAfterKey {\n    fn from(hit: TermOrdHit) -> Self {\n        match hit {\n            TermOrdHit::Exact(ord) => PrecomputedAfterKey::Exact(ord),\n            // TermOrdHit represents AfterLast as Next(u64::MAX), we keep it as is\n            TermOrdHit::Next(ord) => PrecomputedAfterKey::Next(ord),\n        }\n    }\n}\n\nimpl<T: MonotonicallyMappableToU64> From<ProjectedNumber<T>> for PrecomputedAfterKey {\n    fn from(num: ProjectedNumber<T>) -> Self {\n        match num {\n            ProjectedNumber::Exact(number) => PrecomputedAfterKey::Exact(number.to_u64()),\n            ProjectedNumber::Next(number) => PrecomputedAfterKey::Next(number.to_u64()),\n            ProjectedNumber::AfterLast => PrecomputedAfterKey::AfterLast,\n        }\n    }\n}\n\n// /!\\ These operators only makes sense if both values are in the same column space\nimpl PrecomputedAfterKey {\n    pub fn equals(&self, column_value: u64) -> bool {\n        match self {\n            PrecomputedAfterKey::Exact(v) => *v == column_value,\n            PrecomputedAfterKey::Next(_) => false,\n            PrecomputedAfterKey::AfterLast => false,\n        }\n    }\n\n    pub fn gt(&self, column_value: u64) -> bool {\n        match self {\n            PrecomputedAfterKey::Exact(v) => *v > column_value,\n            PrecomputedAfterKey::Next(v) => *v > column_value,\n            PrecomputedAfterKey::AfterLast => true,\n        }\n    }\n\n    pub fn lt(&self, column_value: u64) -> bool {\n        match self {\n            PrecomputedAfterKey::Exact(v) => *v < column_value,\n            // a value equal to the next is greater than the after key\n            PrecomputedAfterKey::Next(v) => *v <= column_value,\n            PrecomputedAfterKey::AfterLast => false,\n        }\n    }\n\n    fn precompute_ip_addr(column: &Column<u64>, key: &Ipv6Addr) -> crate::Result<Self> {\n        let compact_space_accessor = column\n            .values\n            .clone()\n            .downcast_arc::<CompactSpaceU64Accessor>()\n            .map_err(|_| {\n                TantivyError::AggregationError(crate::aggregation::AggregationError::InternalError(\n                    \"type mismatch: could not downcast to CompactSpaceU64Accessor\".to_string(),\n                ))\n            })?;\n        let ip_u128 = key.to_bits();\n        let ip_next_compact = compact_space_accessor.u128_to_next_compact(ip_u128);\n        Ok(ip_next_compact.into())\n    }\n\n    fn precompute_term_ord(\n        str_dict_column: &Option<StrColumn>,\n        key: &str,\n        field: &str,\n    ) -> crate::Result<Self> {\n        let dict = str_dict_column\n            .as_ref()\n            .expect(\"dictionary missing for str accessor\")\n            .dictionary();\n        let next_ord = dict.term_ord_or_next(key).map_err(|_| {\n            TantivyError::InvalidArgument(format!(\n                \"failed to lookup after_key '{}' for field '{}'\",\n                key, field\n            ))\n        })?;\n        Ok(next_ord.into())\n    }\n\n    /// Projects the after key into the column space of the given accessor.\n    ///\n    /// The computed after key will not take care of skipping entire columns\n    /// when the after key type is ordered after the accessor's type, that\n    /// should be performed earlier.\n    pub fn precompute(\n        composite_accessor: &CompositeAccessor,\n        source_after_key: &CompositeIntermediateKey,\n        field: &str,\n        missing_order: MissingOrder,\n        order: Order,\n    ) -> crate::Result<Self> {\n        use CompositeIntermediateKey as CIKey;\n        let precomputed_key = match (composite_accessor.column_type, source_after_key) {\n            (ColumnType::Bytes, _) => panic!(\"unsupported\"),\n            // null after key\n            (_, CIKey::Null) => precompute_missing_after_key(false, missing_order, order),\n            // numerical\n            (ColumnType::I64, CIKey::I64(k)) => PrecomputedAfterKey::Exact(k.to_u64()),\n            (ColumnType::I64, CIKey::U64(k)) => num_proj::u64_to_i64(*k).into(),\n            (ColumnType::I64, CIKey::F64(k)) => num_proj::f64_to_i64(*k).into(),\n            (ColumnType::U64, CIKey::I64(k)) => num_proj::i64_to_u64(*k).into(),\n            (ColumnType::U64, CIKey::U64(k)) => PrecomputedAfterKey::Exact(*k),\n            (ColumnType::U64, CIKey::F64(k)) => num_proj::f64_to_u64(*k).into(),\n            (ColumnType::F64, CIKey::I64(k)) => num_proj::i64_to_f64(*k).into(),\n            (ColumnType::F64, CIKey::U64(k)) => num_proj::u64_to_f64(*k).into(),\n            (ColumnType::F64, CIKey::F64(k)) => PrecomputedAfterKey::Exact(k.to_u64()),\n            // boolean\n            (ColumnType::Bool, CIKey::Bool(key)) => PrecomputedAfterKey::Exact(key.to_u64()),\n            // string\n            (ColumnType::Str, CIKey::Str(key)) => PrecomputedAfterKey::precompute_term_ord(\n                &composite_accessor.str_dict_column,\n                key,\n                field,\n            )?,\n            // date time\n            (ColumnType::DateTime, CIKey::DateTime(key)) => {\n                PrecomputedAfterKey::Exact(key.to_u64())\n            }\n            // ip address\n            (ColumnType::IpAddr, CIKey::IpAddr(key)) => {\n                PrecomputedAfterKey::precompute_ip_addr(&composite_accessor.column, key)?\n            }\n            // assume the column's type is ordered after the after_key's type\n            _ => PrecomputedAfterKey::keep_all(order),\n        };\n        Ok(precomputed_key)\n    }\n\n    fn keep_all(order: Order) -> Self {\n        match order {\n            Order::Asc => PrecomputedAfterKey::Next(0),\n            Order::Desc => PrecomputedAfterKey::Next(u64::MAX),\n        }\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/calendar_interval.rs",
    "content": "use time::convert::{Day, Nanosecond};\nuse time::{Time, UtcDateTime};\n\nconst NS_IN_DAY: i64 = Nanosecond::per_t::<i128>(Day) as i64;\n\n/// Computes the timestamp in nanoseconds corresponding to the beginning of the\n/// year (January 1st at midnight UTC).\npub(super) fn try_year_bucket(timestamp_ns: i64) -> crate::Result<i64> {\n    year_bucket_using_time_crate(timestamp_ns).map_err(|e| {\n        crate::TantivyError::InvalidArgument(format!(\n            \"Failed to compute year bucket for timestamp {}: {e}\",\n            timestamp_ns\n        ))\n    })\n}\n\n/// Computes the timestamp in nanoseconds corresponding to the beginning of the\n/// month (1st at midnight UTC).\npub(super) fn try_month_bucket(timestamp_ns: i64) -> crate::Result<i64> {\n    month_bucket_using_time_crate(timestamp_ns).map_err(|e| {\n        crate::TantivyError::InvalidArgument(format!(\n            \"Failed to compute month bucket for timestamp {}: {e}\",\n            timestamp_ns\n        ))\n    })\n}\n\n/// Computes the timestamp in nanoseconds corresponding to the beginning of the\n/// week (Monday at midnight UTC).\npub(super) fn week_bucket(timestamp_ns: i64) -> i64 {\n    // 1970-01-01 was a Thursday (weekday = 4)\n    let days_since_epoch = timestamp_ns.div_euclid(NS_IN_DAY);\n    // Find the weekday: 0=Monday, ..., 6=Sunday\n    let weekday = (days_since_epoch + 3).rem_euclid(7);\n    let monday_days_since_epoch = days_since_epoch - weekday;\n    monday_days_since_epoch * NS_IN_DAY\n}\n\nfn year_bucket_using_time_crate(timestamp_ns: i64) -> Result<i64, time::Error> {\n    let timestamp_ns = UtcDateTime::from_unix_timestamp_nanos(timestamp_ns as i128)?\n        .replace_ordinal(1)?\n        .replace_time(Time::MIDNIGHT)\n        .unix_timestamp_nanos();\n    Ok(timestamp_ns as i64)\n}\n\nfn month_bucket_using_time_crate(timestamp_ns: i64) -> Result<i64, time::Error> {\n    let timestamp_ns = UtcDateTime::from_unix_timestamp_nanos(timestamp_ns as i128)?\n        .replace_day(1)?\n        .replace_time(Time::MIDNIGHT)\n        .unix_timestamp_nanos();\n    Ok(timestamp_ns as i64)\n}\n\n#[cfg(test)]\nmod tests {\n    use std::i64;\n\n    use time::format_description::well_known::Iso8601;\n    use time::UtcDateTime;\n\n    use super::*;\n\n    fn ts_ns(iso: &str) -> i64 {\n        UtcDateTime::parse(iso, &Iso8601::DEFAULT)\n            .unwrap()\n            .unix_timestamp_nanos() as i64\n    }\n\n    #[test]\n    fn test_year_bucket() {\n        let ts = ts_ns(\"1970-01-01T00:00:00Z\");\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"1970-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-06-01T10:00:01.010Z\");\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"1970-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"2008-12-31T23:59:59.999999999Z\"); // leap year\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"2008-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"2008-01-01T00:00:00Z\"); // leap year\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"2008-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"2010-12-31T23:59:59.999999999Z\");\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"2010-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"1972-06-01T00:10:00Z\");\n        let res = try_year_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"1972-01-01T00:00:00Z\"));\n    }\n\n    #[test]\n    fn test_month_bucket() {\n        let ts = ts_ns(\"1970-01-15T00:00:00Z\");\n        let res = try_month_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"1970-01-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-02-01T00:00:00Z\");\n        let res = try_month_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"1970-02-01T00:00:00Z\"));\n\n        let ts = ts_ns(\"2000-01-31T23:59:59.999999999Z\");\n        let res = try_month_bucket(ts).unwrap();\n        assert_eq!(res, ts_ns(\"2000-01-01T00:00:00Z\"));\n    }\n\n    #[test]\n    fn test_week_bucket() {\n        let ts = ts_ns(\"1970-01-05T00:00:00Z\"); // Monday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"1970-01-05T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-01-05T23:59:59Z\"); // Monday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"1970-01-05T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-01-07T01:13:00Z\"); // Wednesday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"1970-01-05T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-01-11T23:59:59.999999999Z\"); // Sunday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"1970-01-05T00:00:00Z\"));\n\n        let ts = ts_ns(\"2025-10-16T10:41:59.010Z\"); // Thursday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"2025-10-13T00:00:00Z\"));\n\n        let ts = ts_ns(\"1970-01-01T00:00:00Z\"); // Thursday\n        let res = week_bucket(ts);\n        assert_eq!(res, ts_ns(\"1969-12-29T00:00:00Z\")); // Negative\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/collector.rs",
    "content": "use std::fmt::Debug;\nuse std::mem;\nuse std::net::Ipv6Addr;\n\nuse columnar::column_values::CompactSpaceU64Accessor;\nuse columnar::{\n    Column, ColumnType, Dictionary, MonotonicallyMappableToU128, MonotonicallyMappableToU64,\n    NumericalValue, StrColumn,\n};\nuse rustc_hash::FxHashMap;\nuse smallvec::SmallVec;\n\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::bucket::composite::accessors::{\n    CompositeAccessor, CompositeAggReqData, PrecomputedDateInterval,\n};\nuse crate::aggregation::bucket::composite::calendar_interval;\nuse crate::aggregation::bucket::composite::map::{DynArrayHeapMap, MAX_DYN_ARRAY_SIZE};\nuse crate::aggregation::bucket::{\n    CalendarInterval, CompositeAggregationSource, MissingOrder, Order,\n};\nuse crate::aggregation::cached_sub_aggs::{CachedSubAggs, HighCardSubAggCache};\nuse crate::aggregation::intermediate_agg_result::{\n    CompositeIntermediateKey, IntermediateAggregationResult, IntermediateAggregationResults,\n    IntermediateBucketResult, IntermediateCompositeBucketEntry, IntermediateCompositeBucketResult,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::BucketId;\nuse crate::TantivyError;\n\n#[derive(Clone, Debug)]\nstruct CompositeBucketCollector {\n    count: u32,\n    bucket_id: BucketId,\n}\n\n/// Compact sortable representation of a single source value within a composite key.\n///\n/// The struct encodes both the column identity and the fast field value in a way\n/// that preserves the desired sort order via the derived `Ord` implementation\n/// (fields are compared top-to-bottom: `sort_key` first, then `encoded_value`).\n///\n/// ## `sort_key` encoding\n/// - `0` — missing value, sorted first\n/// - `1..=254` — present value; the original accessor index is `sort_key - 1`\n/// - `u8::MAX` (255) — missing value, sorted last\n///\n/// ## `encoded_value` encoding\n/// - `0` when the field is missing\n/// - The raw u64 fast-field representation when order is ascending\n/// - Bitwise NOT of the raw u64 when order is descending\n#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Default, Hash)]\nstruct InternalValueRepr {\n    /// Column index biased by +1 (so 0 and u8::MAX are reserved for missing sentinels).\n    sort_key: u8,\n    /// Fast field value, possibly bit-flipped for descending order.\n    encoded_value: u64,\n}\n\nimpl InternalValueRepr {\n    #[inline]\n    fn new_term(raw: u64, accessor_idx: u8, order: Order) -> Self {\n        let encoded_value = match order {\n            Order::Asc => raw,\n            Order::Desc => !raw,\n        };\n        InternalValueRepr {\n            sort_key: accessor_idx + 1,\n            encoded_value,\n        }\n    }\n\n    /// For histogram sources the column index is irrelevant (always 1).\n    #[inline]\n    fn new_histogram(raw: u64, order: Order) -> Self {\n        let encoded_value = match order {\n            Order::Asc => raw,\n            Order::Desc => !raw,\n        };\n        InternalValueRepr {\n            sort_key: 1,\n            encoded_value,\n        }\n    }\n\n    #[inline]\n    fn new_missing(order: Order, missing_order: MissingOrder) -> Self {\n        let sort_key = match (missing_order, order) {\n            (MissingOrder::First, _) | (MissingOrder::Default, Order::Asc) => 0,\n            (MissingOrder::Last, _) | (MissingOrder::Default, Order::Desc) => u8::MAX,\n        };\n        InternalValueRepr {\n            sort_key,\n            encoded_value: 0,\n        }\n    }\n\n    /// Decode back to `(accessor_idx, raw_value)`.\n    /// Returns `None` when the value represents a missing field.\n    #[inline]\n    fn decode(self, order: Order) -> Option<(u8, u64)> {\n        if self.sort_key == 0 || self.sort_key == u8::MAX {\n            return None;\n        }\n        let raw = match order {\n            Order::Asc => self.encoded_value,\n            Order::Desc => !self.encoded_value,\n        };\n        Some((self.sort_key - 1, raw))\n    }\n}\n\n/// The collector puts values from the fast field into the correct buckets and\n/// does a conversion to the correct datatype.\n#[derive(Debug)]\npub struct SegmentCompositeCollector {\n    /// One DynArrayHeapMap per parent bucket.\n    parent_buckets: Vec<DynArrayHeapMap<InternalValueRepr, CompositeBucketCollector>>,\n    accessor_idx: usize,\n    sub_agg: Option<CachedSubAggs<HighCardSubAggCache>>,\n    bucket_id_provider: BucketIdProvider,\n    /// Number of sources, needed when creating new DynArrayHeapMaps.\n    num_sources: usize,\n}\n\nimpl SegmentAggregationCollector for SegmentCompositeCollector {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let name = agg_data\n            .get_composite_req_data(self.accessor_idx)\n            .name\n            .clone();\n\n        let buckets = self.add_intermediate_bucket_result(agg_data, parent_bucket_id)?;\n        results.push(\n            name,\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Composite { buckets }),\n        )?;\n\n        Ok(())\n    }\n\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let mem_pre = self.get_memory_consumption();\n        let composite_agg_data = agg_data.take_composite_req_data(self.accessor_idx);\n\n        for doc in docs {\n            let mut visitor = CompositeKeyVisitor {\n                doc_id: *doc,\n                composite_agg_data: &composite_agg_data,\n                buckets: &mut self.parent_buckets[parent_bucket_id as usize],\n                sub_agg: &mut self.sub_agg,\n                bucket_id_provider: &mut self.bucket_id_provider,\n                sub_level_values: SmallVec::new(),\n            };\n            visitor.visit(0, true)?;\n        }\n        agg_data.put_back_composite_req_data(self.accessor_idx, composite_agg_data);\n\n        if let Some(sub_agg) = &mut self.sub_agg {\n            sub_agg.check_flush_local(agg_data)?;\n        }\n\n        let mem_delta = self.get_memory_consumption() - mem_pre;\n        if mem_delta > 0 {\n            agg_data.context.limits.add_memory_consumed(mem_delta)?;\n        }\n\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(sub_agg) = &mut self.sub_agg {\n            sub_agg.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let required_len = max_bucket as usize + 1;\n        while self.parent_buckets.len() < required_len {\n            let map = DynArrayHeapMap::try_new(self.num_sources)?;\n            self.parent_buckets.push(map);\n        }\n        Ok(())\n    }\n}\n\nimpl SegmentCompositeCollector {\n    fn get_memory_consumption(&self) -> u64 {\n        self.parent_buckets\n            .iter()\n            .map(|m| m.memory_consumption())\n            .sum()\n    }\n\n    pub(crate) fn from_req_and_validate(\n        req_data: &mut AggregationsSegmentCtx,\n        node: &AggRefNode,\n    ) -> crate::Result<Self> {\n        validate_req(req_data, node.idx_in_req_data)?;\n\n        let has_sub_aggregations = !node.children.is_empty();\n        let sub_agg = if has_sub_aggregations {\n            let sub_agg_collector = build_segment_agg_collectors(req_data, &node.children)?;\n            Some(CachedSubAggs::new(sub_agg_collector))\n        } else {\n            None\n        };\n\n        let composite_req_data = req_data.get_composite_req_data(node.idx_in_req_data);\n        let num_sources = composite_req_data.req.sources.len();\n\n        Ok(SegmentCompositeCollector {\n            parent_buckets: vec![DynArrayHeapMap::try_new(num_sources)?],\n            accessor_idx: node.idx_in_req_data,\n            sub_agg,\n            bucket_id_provider: BucketIdProvider::default(),\n            num_sources,\n        })\n    }\n\n    #[inline]\n    fn add_intermediate_bucket_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<IntermediateCompositeBucketResult> {\n        let empty_map = DynArrayHeapMap::try_new(self.num_sources)?;\n        let heap_map = mem::replace(\n            &mut self.parent_buckets[parent_bucket_id as usize],\n            empty_map,\n        );\n\n        let mut dict: FxHashMap<Vec<CompositeIntermediateKey>, IntermediateCompositeBucketEntry> =\n            Default::default();\n        dict.reserve(heap_map.size());\n        let composite_data = agg_data.get_composite_req_data(self.accessor_idx);\n        for (key_internal_repr, agg) in heap_map.into_iter() {\n            let key = resolve_key(&key_internal_repr, composite_data)?;\n            let mut sub_aggregation_res = IntermediateAggregationResults::default();\n            if let Some(sub_agg) = &mut self.sub_agg {\n                sub_agg\n                    .get_sub_agg_collector()\n                    .add_intermediate_aggregation_result(\n                        agg_data,\n                        &mut sub_aggregation_res,\n                        agg.bucket_id,\n                    )?;\n            }\n\n            dict.insert(\n                key,\n                IntermediateCompositeBucketEntry {\n                    doc_count: agg.count,\n                    sub_aggregation: sub_aggregation_res,\n                },\n            );\n        }\n\n        Ok(IntermediateCompositeBucketResult {\n            entries: dict,\n            target_size: composite_data.req.size,\n            orders: composite_data\n                .req\n                .sources\n                .iter()\n                .map(|source| match source {\n                    CompositeAggregationSource::Terms(t) => (t.order, t.missing_order),\n                    CompositeAggregationSource::Histogram(h) => (h.order, h.missing_order),\n                    CompositeAggregationSource::DateHistogram(d) => (d.order, d.missing_order),\n                })\n                .collect(),\n        })\n    }\n}\n\nfn validate_req(req_data: &mut AggregationsSegmentCtx, accessor_idx: usize) -> crate::Result<()> {\n    let composite_data = req_data.get_composite_req_data(accessor_idx);\n    let req = &composite_data.req;\n    if req.sources.is_empty() {\n        return Err(TantivyError::InvalidArgument(\n            \"composite aggregation must have at least one source\".to_string(),\n        ));\n    }\n    if req.size == 0 {\n        return Err(TantivyError::InvalidArgument(\n            \"composite aggregation 'size' must be > 0\".to_string(),\n        ));\n    }\n\n    if composite_data.composite_accessors.len() > MAX_DYN_ARRAY_SIZE {\n        return Err(TantivyError::InvalidArgument(format!(\n            \"composite aggregation source supports maximum {MAX_DYN_ARRAY_SIZE} sources\",\n        )));\n    }\n\n    let column_types_for_sources = composite_data.composite_accessors.iter().map(|item| {\n        item.accessors\n            .iter()\n            .map(|a| a.column_type)\n            .collect::<Vec<_>>()\n    });\n\n    for column_types in column_types_for_sources {\n        if column_types.contains(&ColumnType::Bytes) {\n            return Err(TantivyError::InvalidArgument(\n                \"composite aggregation does not support 'bytes' field type\".to_string(),\n            ));\n        }\n    }\n    Ok(())\n}\n\nfn collect_bucket_with_limit(\n    doc_id: crate::DocId,\n    limit_num_buckets: usize,\n    buckets: &mut DynArrayHeapMap<InternalValueRepr, CompositeBucketCollector>,\n    key: &[InternalValueRepr],\n    sub_agg: &mut Option<CachedSubAggs<HighCardSubAggCache>>,\n    bucket_id_provider: &mut BucketIdProvider,\n) {\n    let mut record_in_bucket = |bucket: &mut CompositeBucketCollector| {\n        bucket.count += 1;\n        if let Some(sub_agg) = sub_agg {\n            sub_agg.push(bucket.bucket_id, doc_id);\n        }\n    };\n\n    // We still have room for buckets, just insert\n    if buckets.size() < limit_num_buckets {\n        let bucket = buckets.get_or_insert_with(key, || CompositeBucketCollector {\n            count: 0,\n            bucket_id: bucket_id_provider.next_bucket_id(),\n        });\n        record_in_bucket(bucket);\n        return;\n    }\n\n    // Map is full, but we can still update the bucket if it already exists\n    if let Some(bucket) = buckets.get_mut(key) {\n        record_in_bucket(bucket);\n        return;\n    }\n\n    // Check if the item qualifies to enter the top-k, and evict the highest if it does\n    if let Some(highest_key) = buckets.peek_highest() {\n        if key < highest_key {\n            buckets.evict_highest();\n            let bucket = buckets.get_or_insert_with(key, || CompositeBucketCollector {\n                count: 0,\n                bucket_id: bucket_id_provider.next_bucket_id(),\n            });\n            record_in_bucket(bucket);\n        }\n    }\n}\n\n/// Converts the composite key from its internal column space representation\n/// (segment specific) into its intermediate form.\nfn resolve_key(\n    internal_key: &[InternalValueRepr],\n    agg_data: &CompositeAggReqData,\n) -> crate::Result<Vec<CompositeIntermediateKey>> {\n    internal_key\n        .iter()\n        .enumerate()\n        .map(|(idx, val)| {\n            resolve_internal_value_repr(\n                *val,\n                &agg_data.req.sources[idx],\n                &agg_data.composite_accessors[idx].accessors,\n            )\n        })\n        .collect()\n}\n\nfn resolve_internal_value_repr(\n    internal_value_repr: InternalValueRepr,\n    source: &CompositeAggregationSource,\n    composite_accessors: &[CompositeAccessor],\n) -> crate::Result<CompositeIntermediateKey> {\n    let decoded_value_opt = match source {\n        CompositeAggregationSource::Terms(source) => internal_value_repr.decode(source.order),\n        CompositeAggregationSource::Histogram(source) => internal_value_repr.decode(source.order),\n        CompositeAggregationSource::DateHistogram(source) => {\n            internal_value_repr.decode(source.order)\n        }\n    };\n    let Some((decoded_accessor_idx, val)) = decoded_value_opt else {\n        return Ok(CompositeIntermediateKey::Null);\n    };\n    let key = match source {\n        CompositeAggregationSource::Terms(_) => {\n            let CompositeAccessor {\n                column_type,\n                str_dict_column,\n                column,\n                ..\n            } = &composite_accessors[decoded_accessor_idx as usize];\n            resolve_term(val, column_type, str_dict_column, column)?\n        }\n        CompositeAggregationSource::Histogram(source) => {\n            CompositeIntermediateKey::F64(i64::from_u64(val) as f64 * source.interval)\n        }\n        CompositeAggregationSource::DateHistogram(_) => {\n            CompositeIntermediateKey::DateTime(i64::from_u64(val))\n        }\n    };\n\n    Ok(key)\n}\n\nfn resolve_term(\n    val: u64,\n    column_type: &ColumnType,\n    str_dict_column: &Option<StrColumn>,\n    column: &Column,\n) -> crate::Result<CompositeIntermediateKey> {\n    let key = if *column_type == ColumnType::Str {\n        let fallback_dict = Dictionary::empty();\n        let term_dict = str_dict_column\n            .as_ref()\n            .map(|el| el.dictionary())\n            .unwrap_or_else(|| &fallback_dict);\n\n        let mut buffer = Vec::new();\n        term_dict.ord_to_term(val, &mut buffer)?;\n        CompositeIntermediateKey::Str(\n            String::from_utf8(buffer.to_vec()).expect(\"could not convert to String\"),\n        )\n    } else if *column_type == ColumnType::DateTime {\n        let val = i64::from_u64(val);\n        CompositeIntermediateKey::DateTime(val)\n    } else if *column_type == ColumnType::Bool {\n        let val = bool::from_u64(val);\n        CompositeIntermediateKey::Bool(val)\n    } else if *column_type == ColumnType::IpAddr {\n        let compact_space_accessor = column\n            .values\n            .clone()\n            .downcast_arc::<CompactSpaceU64Accessor>()\n            .map_err(|_| {\n                TantivyError::AggregationError(crate::aggregation::AggregationError::InternalError(\n                    \"Type mismatch: Could not downcast to CompactSpaceU64Accessor\".to_string(),\n                ))\n            })?;\n        let val: u128 = compact_space_accessor.compact_to_u128(val as u32);\n        let val = Ipv6Addr::from_u128(val);\n        CompositeIntermediateKey::IpAddr(val)\n    } else if *column_type == ColumnType::U64 {\n        CompositeIntermediateKey::U64(val)\n    } else if *column_type == ColumnType::I64 {\n        CompositeIntermediateKey::I64(i64::from_u64(val))\n    } else {\n        let val = f64::from_u64(val);\n        let val: NumericalValue = val.into();\n\n        match val.normalize() {\n            NumericalValue::U64(val) => CompositeIntermediateKey::U64(val),\n            NumericalValue::I64(val) => CompositeIntermediateKey::I64(val),\n            NumericalValue::F64(val) => CompositeIntermediateKey::F64(val),\n        }\n    };\n    Ok(key)\n}\n\n/// Browse through the cardinal product obtained by the different values of the doc composite key\n/// sources.\n///\n/// For each of those tuple-key, that are after the limit key, we call collect_bucket_with_limit.\nstruct CompositeKeyVisitor<'a> {\n    doc_id: crate::DocId,\n    composite_agg_data: &'a CompositeAggReqData,\n    buckets: &'a mut DynArrayHeapMap<InternalValueRepr, CompositeBucketCollector>,\n    sub_agg: &'a mut Option<CachedSubAggs<HighCardSubAggCache>>,\n    bucket_id_provider: &'a mut BucketIdProvider,\n    sub_level_values: SmallVec<[InternalValueRepr; MAX_DYN_ARRAY_SIZE]>,\n}\n\nimpl CompositeKeyVisitor<'_> {\n    /// Depth-first walk of the accessors to build the composite key combinations\n    /// and update the buckets.\n    ///\n    /// `source_idx` is the current source index in the recursion.\n    /// `is_on_after_key` tracks whether we still need to consider the after_key\n    /// for pruning at this level and below.\n    fn visit(&mut self, source_idx: usize, is_on_after_key: bool) -> crate::Result<()> {\n        if source_idx == self.composite_agg_data.req.sources.len() {\n            if !is_on_after_key {\n                collect_bucket_with_limit(\n                    self.doc_id,\n                    self.composite_agg_data.req.size as usize,\n                    self.buckets,\n                    &self.sub_level_values,\n                    self.sub_agg,\n                    self.bucket_id_provider,\n                );\n            }\n            return Ok(());\n        }\n\n        let current_level_accessors = &self.composite_agg_data.composite_accessors[source_idx];\n        let current_level_source = &self.composite_agg_data.req.sources[source_idx];\n        let mut missing = true;\n        for (accessor_idx, accessor) in current_level_accessors.accessors.iter().enumerate() {\n            let values = accessor.column.values_for_doc(self.doc_id);\n            for value in values {\n                missing = false;\n                match current_level_source {\n                    CompositeAggregationSource::Terms(_) => {\n                        let preceeds_after_key_type =\n                            accessor_idx < current_level_accessors.after_key_accessor_idx;\n                        if is_on_after_key && preceeds_after_key_type {\n                            break;\n                        }\n                        let matches_after_key_type =\n                            accessor_idx == current_level_accessors.after_key_accessor_idx;\n\n                        if matches_after_key_type && is_on_after_key {\n                            let should_skip = match current_level_source.order() {\n                                Order::Asc => current_level_accessors.after_key.gt(value),\n                                Order::Desc => current_level_accessors.after_key.lt(value),\n                            };\n                            if should_skip {\n                                continue;\n                            }\n                        }\n                        self.sub_level_values.push(InternalValueRepr::new_term(\n                            value,\n                            accessor_idx as u8,\n                            current_level_source.order(),\n                        ));\n                        let still_on_after_key = matches_after_key_type\n                            && current_level_accessors.after_key.equals(value);\n                        self.visit(source_idx + 1, is_on_after_key && still_on_after_key)?;\n                        self.sub_level_values.pop();\n                    }\n                    CompositeAggregationSource::Histogram(source) => {\n                        let float_value = match accessor.column_type {\n                            ColumnType::U64 => value as f64,\n                            ColumnType::I64 => i64::from_u64(value) as f64,\n                            ColumnType::DateTime => i64::from_u64(value) as f64 / 1_000_000.,\n                            ColumnType::F64 => f64::from_u64(value),\n                            _ => {\n                                panic!(\n                                    \"unexpected type {:?}. This should not happen\",\n                                    accessor.column_type\n                                )\n                            }\n                        };\n                        let bucket_index = (float_value / source.interval).floor() as i64;\n                        let bucket_value = i64::to_u64(bucket_index);\n                        if is_on_after_key {\n                            let should_skip = match current_level_source.order() {\n                                Order::Asc => current_level_accessors.after_key.gt(bucket_value),\n                                Order::Desc => current_level_accessors.after_key.lt(bucket_value),\n                            };\n                            if should_skip {\n                                continue;\n                            }\n                        }\n                        self.sub_level_values.push(InternalValueRepr::new_histogram(\n                            bucket_value,\n                            current_level_source.order(),\n                        ));\n                        let still_on_after_key =\n                            current_level_accessors.after_key.equals(bucket_value);\n                        self.visit(source_idx + 1, is_on_after_key && still_on_after_key)?;\n                        self.sub_level_values.pop();\n                    }\n                    CompositeAggregationSource::DateHistogram(_) => {\n                        let value_ns = match accessor.column_type {\n                            ColumnType::DateTime => i64::from_u64(value),\n                            _ => {\n                                panic!(\n                                    \"unexpected type {:?}. This should not happen\",\n                                    accessor.column_type\n                                )\n                            }\n                        };\n                        let bucket_index = match accessor.date_histogram_interval {\n                            PrecomputedDateInterval::FixedNanoseconds(fixed_interval_ns) => {\n                                (value_ns / fixed_interval_ns) * fixed_interval_ns\n                            }\n                            PrecomputedDateInterval::Calendar(CalendarInterval::Year) => {\n                                calendar_interval::try_year_bucket(value_ns)?\n                            }\n                            PrecomputedDateInterval::Calendar(CalendarInterval::Month) => {\n                                calendar_interval::try_month_bucket(value_ns)?\n                            }\n                            PrecomputedDateInterval::Calendar(CalendarInterval::Week) => {\n                                calendar_interval::week_bucket(value_ns)\n                            }\n                            PrecomputedDateInterval::NotApplicable => {\n                                panic!(\"interval not precomputed for date histogram source\")\n                            }\n                        };\n                        let bucket_value = i64::to_u64(bucket_index);\n                        if is_on_after_key {\n                            let should_skip = match current_level_source.order() {\n                                Order::Asc => current_level_accessors.after_key.gt(bucket_value),\n                                Order::Desc => current_level_accessors.after_key.lt(bucket_value),\n                            };\n                            if should_skip {\n                                continue;\n                            }\n                        }\n                        self.sub_level_values.push(InternalValueRepr::new_histogram(\n                            bucket_value,\n                            current_level_source.order(),\n                        ));\n                        let still_on_after_key =\n                            current_level_accessors.after_key.equals(bucket_value);\n                        self.visit(source_idx + 1, is_on_after_key && still_on_after_key)?;\n                        self.sub_level_values.pop();\n                    }\n                };\n            }\n        }\n        if missing && current_level_source.missing_bucket() {\n            if is_on_after_key && current_level_accessors.skip_missing {\n                return Ok(());\n            }\n            self.sub_level_values.push(InternalValueRepr::new_missing(\n                current_level_source.order(),\n                current_level_source.missing_order(),\n            ));\n            self.visit(\n                source_idx + 1,\n                is_on_after_key && current_level_accessors.is_after_key_explicit_missing,\n            )?;\n            self.sub_level_values.pop();\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/map.rs",
    "content": "use std::collections::BinaryHeap;\nuse std::fmt::Debug;\nuse std::hash::Hash;\n\nuse rustc_hash::FxHashMap;\nuse smallvec::SmallVec;\n\nuse crate::TantivyError;\n\n/// Map backed by a hash map for fast access and a binary heap to track the\n/// highest key. The key is an array of fixed size S.\n#[derive(Clone, Debug)]\nstruct ArrayHeapMap<K: Ord, V, const S: usize> {\n    pub(crate) buckets: FxHashMap<[K; S], V>,\n    pub(crate) heap: BinaryHeap<[K; S]>,\n}\n\nimpl<K: Ord, V, const S: usize> Default for ArrayHeapMap<K, V, S> {\n    fn default() -> Self {\n        ArrayHeapMap {\n            buckets: FxHashMap::default(),\n            heap: BinaryHeap::default(),\n        }\n    }\n}\n\nimpl<K: Eq + Hash + Clone + Ord, V, const S: usize> ArrayHeapMap<K, V, S> {\n    /// Panics if the length of `key` is not S.\n    fn get_or_insert_with<F: FnOnce() -> V>(&mut self, key: &[K], f: F) -> &mut V {\n        let key_array: &[K; S] = key.try_into().expect(\"Key length mismatch\");\n        self.buckets.entry(key_array.clone()).or_insert_with(|| {\n            self.heap.push(key_array.clone());\n            f()\n        })\n    }\n\n    /// Panics if the length of `key` is not S.\n    fn get_mut(&mut self, key: &[K]) -> Option<&mut V> {\n        let key_array: &[K; S] = key.try_into().expect(\"Key length mismatch\");\n        self.buckets.get_mut(key_array)\n    }\n\n    fn peek_highest(&self) -> Option<&[K]> {\n        self.heap.peek().map(|k_array| k_array.as_slice())\n    }\n\n    fn evict_highest(&mut self) {\n        if let Some(highest) = self.heap.pop() {\n            self.buckets.remove(&highest);\n        }\n    }\n\n    fn memory_consumption(&self) -> u64 {\n        let key_size = std::mem::size_of::<[K; S]>();\n        let map_size = (key_size + std::mem::size_of::<V>()) * self.buckets.capacity();\n        let heap_size = key_size * self.heap.capacity();\n        (map_size + heap_size) as u64\n    }\n}\n\nimpl<K: Copy + Ord + Clone + 'static, V: 'static, const S: usize> ArrayHeapMap<K, V, S> {\n    fn into_iter(self) -> Box<dyn Iterator<Item = (SmallVec<[K; MAX_DYN_ARRAY_SIZE]>, V)>> {\n        Box::new(\n            self.buckets\n                .into_iter()\n                .map(|(k, v)| (SmallVec::from_slice(&k), v)),\n        )\n    }\n}\n\npub(super) const MAX_DYN_ARRAY_SIZE: usize = 16;\nconst MAX_DYN_ARRAY_SIZE_PLUS_ONE: usize = MAX_DYN_ARRAY_SIZE + 1;\n\n/// A map optimized for memory footprint, fast access and efficient eviction of\n/// the highest key.\n///\n/// Keys are inlined arrays of size 1 to [MAX_DYN_ARRAY_SIZE] but for a given\n/// instance the key size is fixed. This allows to avoid heap allocations for the\n/// keys.\n#[derive(Clone, Debug)]\npub(super) struct DynArrayHeapMap<K: Ord, V>(DynArrayHeapMapInner<K, V>);\n\n/// Wrapper around ArrayHeapMap to dynamically dispatch on the array size.\n#[derive(Clone, Debug)]\nenum DynArrayHeapMapInner<K: Ord, V> {\n    Dim1(ArrayHeapMap<K, V, 1>),\n    Dim2(ArrayHeapMap<K, V, 2>),\n    Dim3(ArrayHeapMap<K, V, 3>),\n    Dim4(ArrayHeapMap<K, V, 4>),\n    Dim5(ArrayHeapMap<K, V, 5>),\n    Dim6(ArrayHeapMap<K, V, 6>),\n    Dim7(ArrayHeapMap<K, V, 7>),\n    Dim8(ArrayHeapMap<K, V, 8>),\n    Dim9(ArrayHeapMap<K, V, 9>),\n    Dim10(ArrayHeapMap<K, V, 10>),\n    Dim11(ArrayHeapMap<K, V, 11>),\n    Dim12(ArrayHeapMap<K, V, 12>),\n    Dim13(ArrayHeapMap<K, V, 13>),\n    Dim14(ArrayHeapMap<K, V, 14>),\n    Dim15(ArrayHeapMap<K, V, 15>),\n    Dim16(ArrayHeapMap<K, V, 16>),\n}\n\nimpl<K: Ord, V> DynArrayHeapMap<K, V> {\n    /// Creates a new heap map with dynamic array keys of size `key_dimension`.\n    pub(super) fn try_new(key_dimension: usize) -> crate::Result<Self> {\n        let inner = match key_dimension {\n            0 => {\n                return Err(TantivyError::InvalidArgument(\n                    \"DynArrayHeapMap dimension must be at least 1\".to_string(),\n                ))\n            }\n            1 => DynArrayHeapMapInner::Dim1(ArrayHeapMap::default()),\n            2 => DynArrayHeapMapInner::Dim2(ArrayHeapMap::default()),\n            3 => DynArrayHeapMapInner::Dim3(ArrayHeapMap::default()),\n            4 => DynArrayHeapMapInner::Dim4(ArrayHeapMap::default()),\n            5 => DynArrayHeapMapInner::Dim5(ArrayHeapMap::default()),\n            6 => DynArrayHeapMapInner::Dim6(ArrayHeapMap::default()),\n            7 => DynArrayHeapMapInner::Dim7(ArrayHeapMap::default()),\n            8 => DynArrayHeapMapInner::Dim8(ArrayHeapMap::default()),\n            9 => DynArrayHeapMapInner::Dim9(ArrayHeapMap::default()),\n            10 => DynArrayHeapMapInner::Dim10(ArrayHeapMap::default()),\n            11 => DynArrayHeapMapInner::Dim11(ArrayHeapMap::default()),\n            12 => DynArrayHeapMapInner::Dim12(ArrayHeapMap::default()),\n            13 => DynArrayHeapMapInner::Dim13(ArrayHeapMap::default()),\n            14 => DynArrayHeapMapInner::Dim14(ArrayHeapMap::default()),\n            15 => DynArrayHeapMapInner::Dim15(ArrayHeapMap::default()),\n            16 => DynArrayHeapMapInner::Dim16(ArrayHeapMap::default()),\n            MAX_DYN_ARRAY_SIZE_PLUS_ONE.. => {\n                return Err(TantivyError::InvalidArgument(format!(\n                    \"DynArrayHeapMap supports maximum {MAX_DYN_ARRAY_SIZE} dimensions, got \\\n                     {key_dimension}\",\n                )))\n            }\n        };\n        Ok(DynArrayHeapMap(inner))\n    }\n\n    /// Number of elements in the map. This is not the dimension of the keys.\n    pub(super) fn size(&self) -> usize {\n        match &self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim2(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim3(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim4(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim5(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim6(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim7(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim8(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim9(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim10(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim11(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim12(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim13(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim14(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim15(map) => map.buckets.len(),\n            DynArrayHeapMapInner::Dim16(map) => map.buckets.len(),\n        }\n    }\n}\n\nimpl<K: Ord + Hash + Clone, V> DynArrayHeapMap<K, V> {\n    /// Get a mutable reference to the value corresponding to `key` or inserts a new\n    /// value created by calling `f`.\n    ///\n    /// Panics if the length of `key` does not match the key dimension of the map.\n    pub(super) fn get_or_insert_with<F: FnOnce() -> V>(&mut self, key: &[K], f: F) -> &mut V {\n        match &mut self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim2(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim3(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim4(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim5(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim6(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim7(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim8(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim9(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim10(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim11(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim12(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim13(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim14(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim15(map) => map.get_or_insert_with(key, f),\n            DynArrayHeapMapInner::Dim16(map) => map.get_or_insert_with(key, f),\n        }\n    }\n\n    /// Returns a mutable reference to the value corresponding to `key`.\n    ///\n    /// Panics if the length of `key` does not match the key dimension of the map.\n    pub fn get_mut(&mut self, key: &[K]) -> Option<&mut V> {\n        match &mut self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim2(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim3(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim4(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim5(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim6(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim7(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim8(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim9(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim10(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim11(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim12(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim13(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim14(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim15(map) => map.get_mut(key),\n            DynArrayHeapMapInner::Dim16(map) => map.get_mut(key),\n        }\n    }\n\n    /// Returns a reference to the highest key in the map.\n    pub(super) fn peek_highest(&self) -> Option<&[K]> {\n        match &self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim2(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim3(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim4(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim5(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim6(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim7(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim8(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim9(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim10(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim11(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim12(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim13(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim14(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim15(map) => map.peek_highest(),\n            DynArrayHeapMapInner::Dim16(map) => map.peek_highest(),\n        }\n    }\n\n    /// Removes the entry with the highest key from the map.\n    pub(super) fn evict_highest(&mut self) {\n        match &mut self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim2(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim3(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim4(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim5(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim6(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim7(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim8(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim9(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim10(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim11(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim12(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim13(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim14(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim15(map) => map.evict_highest(),\n            DynArrayHeapMapInner::Dim16(map) => map.evict_highest(),\n        }\n    }\n\n    pub(crate) fn memory_consumption(&self) -> u64 {\n        match &self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim2(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim3(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim4(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim5(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim6(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim7(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim8(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim9(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim10(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim11(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim12(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim13(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim14(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim15(map) => map.memory_consumption(),\n            DynArrayHeapMapInner::Dim16(map) => map.memory_consumption(),\n        }\n    }\n}\n\nimpl<K: Ord + Clone + Copy + 'static, V: 'static> DynArrayHeapMap<K, V> {\n    /// Turns this map into an iterator over key-value pairs.\n    pub fn into_iter(self) -> impl Iterator<Item = (SmallVec<[K; MAX_DYN_ARRAY_SIZE]>, V)> {\n        match self.0 {\n            DynArrayHeapMapInner::Dim1(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim2(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim3(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim4(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim5(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim6(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim7(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim8(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim9(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim10(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim11(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim12(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim13(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim14(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim15(map) => map.into_iter(),\n            DynArrayHeapMapInner::Dim16(map) => map.into_iter(),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_dyn_array_heap_map() {\n        let mut map = DynArrayHeapMap::<u32, &str>::try_new(2).unwrap();\n        // insert\n        let key1 = [1u32, 2u32];\n        let key2 = [2u32, 1u32];\n        map.get_or_insert_with(&key1, || \"a\");\n        map.get_or_insert_with(&key2, || \"b\");\n        assert_eq!(map.size(), 2);\n\n        // evict highest\n        assert_eq!(map.peek_highest(), Some(&key2[..]));\n        map.evict_highest();\n        assert_eq!(map.size(), 1);\n        assert_eq!(map.peek_highest(), Some(&key1[..]));\n\n        // into_iter\n        let mut iter = map.into_iter();\n        let (k, v) = iter.next().unwrap();\n        assert_eq!(k.as_slice(), &key1);\n        assert_eq!(v, \"a\");\n        assert_eq!(iter.next(), None);\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/mod.rs",
    "content": "mod accessors;\nmod calendar_interval;\nmod collector;\nmod map;\nmod numeric_types;\n\nuse core::panic;\nuse std::cmp::Ordering;\nuse std::fmt::Debug;\nuse std::net::{AddrParseError, IpAddr};\nuse std::str::FromStr;\n\nuse columnar::ColumnType;\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse crate::aggregation::agg_result::CompositeKey;\npub use crate::aggregation::bucket::composite::accessors::{\n    CompositeAccessor, CompositeAggReqData, CompositeSourceAccessors, PrecomputedDateInterval,\n};\npub use crate::aggregation::bucket::composite::collector::SegmentCompositeCollector;\nuse crate::aggregation::bucket::composite::numeric_types::num_cmp::{\n    cmp_i64_f64, cmp_i64_u64, cmp_u64_f64,\n};\nuse crate::aggregation::bucket::Order;\nuse crate::aggregation::deserialize_f64;\nuse crate::aggregation::intermediate_agg_result::CompositeIntermediateKey;\nuse crate::schema::IntoIpv6Addr;\nuse crate::TantivyError;\n\n/// Position of missing keys in the ordering.\n#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, Default)]\n#[serde(rename_all = \"lowercase\")]\npub enum MissingOrder {\n    /// Missing keys appear first in ascending order, last in descending order.\n    #[default]\n    Default,\n    /// Missing keys should appear first.\n    First,\n    /// Missing keys should appear last.\n    Last,\n}\n\nfn agg_source_default_order() -> Order {\n    Order::Asc\n}\n\n/// Term source for a composite aggregation.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct TermCompositeAggregationSource {\n    /// The name used to refer to this source in the composite key.\n    #[serde(skip)]\n    pub name: String,\n    /// The field to aggregate on.\n    pub field: String,\n    /// The order for this source.\n    #[serde(default = \"agg_source_default_order\")]\n    pub order: Order,\n    /// Whether to create a `null` bucket for documents without value for this\n    /// field. By default documents without a value are ignored.\n    #[serde(default)]\n    pub missing_bucket: bool,\n    /// Whether missing keys should appear first or last.\n    #[serde(default)]\n    pub missing_order: MissingOrder,\n}\n\n/// Histogram source for a composite aggregation.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct HistogramCompositeAggregationSource {\n    /// The name used to refer to this source in the composite key.\n    #[serde(skip)]\n    pub name: String,\n    /// The field to aggregate on.\n    pub field: String,\n    /// The interval for the histogram. For datetime fields, this is expressed.\n    /// in milliseconds.\n    #[serde(deserialize_with = \"deserialize_f64\")]\n    pub interval: f64,\n    /// The order for this source.\n    #[serde(default = \"agg_source_default_order\")]\n    pub order: Order,\n    /// Whether to create a `null` bucket for documents without value for this\n    /// field. By default documents without a value are ignored.\n    #[serde(default)]\n    pub missing_bucket: bool,\n    /// Whether missing keys should appear first or last.\n    #[serde(default)]\n    pub missing_order: MissingOrder,\n}\n\n/// Calendar intervals supported for date histogram sources\n#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"lowercase\")]\npub enum CalendarInterval {\n    /// A year between Jan 1st and Dec 31st, taking into account leap years.\n    Year,\n    /// A month between the 1st and the last day of the month.\n    Month,\n    /// A week between Monday and Sunday.\n    Week,\n}\n\n/// Date histogram source for a composite aggregation.\n///\n/// Time zone not supported yet. Every interval is aligned on UTC.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct DateHistogramCompositeAggregationSource {\n    /// The name used to refer to this source in the composite key.\n    #[serde(skip)]\n    pub name: String,\n    /// The field to aggregate on.\n    pub field: String,\n    /// The fixed interval for the histogram. Either this or `calendar_interval`.\n    /// must be set.\n    #[serde(default, skip_serializing_if = \"Option::is_none\")]\n    pub fixed_interval: Option<String>,\n    /// The calendar adjusted interval for the histogram. Either this or\n    /// `fixed_interval` must be set.\n    #[serde(default, skip_serializing_if = \"Option::is_none\")]\n    pub calendar_interval: Option<CalendarInterval>,\n    /// The order for this source.\n    #[serde(default = \"agg_source_default_order\")]\n    pub order: Order,\n    /// Whether to create a `null` bucket for documents without value for this\n    /// field. By default documents without a value are ignored. Not supported\n    /// in Elasticsearch.\n    #[serde(default)]\n    pub missing_bucket: bool,\n    /// Whether missing keys should appear first or last.\n    #[serde(default)]\n    pub missing_order: MissingOrder,\n}\n\n/// Source for the composite aggregation. A composite aggregation can have\n/// multiple sources.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(rename_all = \"snake_case\")]\npub enum CompositeAggregationSource {\n    /// Terms source.\n    Terms(TermCompositeAggregationSource),\n    /// Histogram source.\n    Histogram(HistogramCompositeAggregationSource),\n    /// Date histogram source.\n    DateHistogram(DateHistogramCompositeAggregationSource),\n}\n\nimpl CompositeAggregationSource {\n    pub(crate) fn field(&self) -> &str {\n        match self {\n            CompositeAggregationSource::Terms(source) => &source.field,\n            CompositeAggregationSource::Histogram(source) => &source.field,\n            CompositeAggregationSource::DateHistogram(source) => &source.field,\n        }\n    }\n\n    pub(crate) fn name(&self) -> &str {\n        match self {\n            CompositeAggregationSource::Terms(source) => &source.name,\n            CompositeAggregationSource::Histogram(source) => &source.name,\n            CompositeAggregationSource::DateHistogram(source) => &source.name,\n        }\n    }\n\n    pub(crate) fn order(&self) -> Order {\n        match self {\n            CompositeAggregationSource::Terms(source) => source.order,\n            CompositeAggregationSource::Histogram(source) => source.order,\n            CompositeAggregationSource::DateHistogram(source) => source.order,\n        }\n    }\n\n    pub(crate) fn missing_order(&self) -> MissingOrder {\n        match self {\n            CompositeAggregationSource::Terms(source) => source.missing_order,\n            CompositeAggregationSource::Histogram(source) => source.missing_order,\n            CompositeAggregationSource::DateHistogram(source) => source.missing_order,\n        }\n    }\n\n    pub(crate) fn missing_bucket(&self) -> bool {\n        match self {\n            CompositeAggregationSource::Terms(source) => source.missing_bucket,\n            CompositeAggregationSource::Histogram(source) => source.missing_bucket,\n            CompositeAggregationSource::DateHistogram(source) => source.missing_bucket,\n        }\n    }\n}\n\n/// A paginable aggregation that performs on multiple dimensions (sources),\n/// potentially mixing term and range queries.\n///\n/// Pagination is made possible because the buckets are ordered by the composite\n/// key, so the next page can be fetched \"efficiently\" by filtering using range\n/// queries on the key dimensions.\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\n#[serde(\n    try_from = \"CompositeAggregationSerde\",\n    into = \"CompositeAggregationSerde\"\n)]\npub struct CompositeAggregation {\n    /// The fields and bucketting strategies.\n    pub sources: Vec<CompositeAggregationSource>,\n    /// Number of buckets to return (page size).\n    pub size: u32,\n    /// The key of the previous page's last bucket.\n    pub after: FxHashMap<String, AfterKey>,\n}\n\n#[derive(Serialize, Deserialize)]\nstruct CompositeAggregationSerde {\n    sources: Vec<FxHashMap<String, CompositeAggregationSource>>,\n    size: u32,\n    #[serde(default, skip_serializing_if = \"FxHashMap::is_empty\")]\n    after: FxHashMap<String, AfterKey>,\n}\n\nimpl TryFrom<CompositeAggregationSerde> for CompositeAggregation {\n    type Error = TantivyError;\n\n    fn try_from(value: CompositeAggregationSerde) -> Result<Self, Self::Error> {\n        let mut sources = Vec::with_capacity(value.sources.len());\n        for map in value.sources {\n            if map.len() != 1 {\n                return Err(TantivyError::InvalidArgument(\n                    \"each composite source must have exactly one named entry\".to_string(),\n                ));\n            }\n            let (name, mut source) = map.into_iter().next().unwrap();\n            match &mut source {\n                CompositeAggregationSource::Terms(source) => {\n                    source.name = name;\n                }\n                CompositeAggregationSource::Histogram(source) => {\n                    source.name = name;\n                }\n                CompositeAggregationSource::DateHistogram(source) => {\n                    source.name = name;\n                }\n            }\n            sources.push(source);\n        }\n        Ok(CompositeAggregation {\n            sources,\n            size: value.size,\n            after: value.after,\n        })\n    }\n}\n\nimpl From<CompositeAggregation> for CompositeAggregationSerde {\n    fn from(value: CompositeAggregation) -> Self {\n        let mut serde_sources = Vec::with_capacity(value.sources.len());\n        for source in value.sources {\n            let (name, stored_source) = match source {\n                CompositeAggregationSource::Terms(source) => {\n                    let name = source.name.clone();\n                    // name field is #[serde(skip)] so it won't be serialized inside the value\n                    (name, CompositeAggregationSource::Terms(source))\n                }\n                CompositeAggregationSource::Histogram(source) => {\n                    let name = source.name.clone();\n                    (name, CompositeAggregationSource::Histogram(source))\n                }\n                CompositeAggregationSource::DateHistogram(source) => {\n                    let name = source.name.clone();\n                    (name, CompositeAggregationSource::DateHistogram(source))\n                }\n            };\n            let mut map = FxHashMap::default();\n            map.insert(name, stored_source);\n            serde_sources.push(map);\n        }\n        CompositeAggregationSerde {\n            sources: serde_sources,\n            size: value.size,\n            after: value.after,\n        }\n    }\n}\n\n/// Key used to decide the order in which multi-type terms should be paginated.\n#[derive(Ord, PartialOrd, PartialEq, Eq)]\nenum ColumnPaginationOrder {\n    Bool = 1,\n    Str = 2,\n    Numeric = 3,\n    IpAddr = 4,\n    DateTime = 5,\n}\n\ntrait ToTypePaginationOrder {\n    /// Returns the pagination order key for the current type related variant.\n    ///\n    /// Panics if called on a variant representing null. Null values must be\n    /// handled separately.\n    fn column_pagination_order(&self) -> ColumnPaginationOrder;\n}\n\nimpl ToTypePaginationOrder for ColumnType {\n    fn column_pagination_order(&self) -> ColumnPaginationOrder {\n        match self {\n            ColumnType::Bool => ColumnPaginationOrder::Bool,\n            ColumnType::Str => ColumnPaginationOrder::Str,\n            ColumnType::F64 | ColumnType::I64 | ColumnType::U64 => ColumnPaginationOrder::Numeric,\n            ColumnType::IpAddr => ColumnPaginationOrder::IpAddr,\n            ColumnType::DateTime => ColumnPaginationOrder::DateTime,\n            ColumnType::Bytes => panic!(\"unsupported\"),\n        }\n    }\n}\n\nimpl ToTypePaginationOrder for CompositeIntermediateKey {\n    fn column_pagination_order(&self) -> ColumnPaginationOrder {\n        match self {\n            CompositeIntermediateKey::Bool(_) => ColumnPaginationOrder::Bool,\n            CompositeIntermediateKey::Str(_) => ColumnPaginationOrder::Str,\n            CompositeIntermediateKey::F64(_)\n            | CompositeIntermediateKey::I64(_)\n            | CompositeIntermediateKey::U64(_) => ColumnPaginationOrder::Numeric,\n            CompositeIntermediateKey::IpAddr(_) => ColumnPaginationOrder::IpAddr,\n            CompositeIntermediateKey::DateTime(_) => ColumnPaginationOrder::DateTime,\n            CompositeIntermediateKey::Null => panic!(\"null must be handled separately\"),\n        }\n    }\n}\n\nimpl ToTypePaginationOrder for CompositeKey {\n    fn column_pagination_order(&self) -> ColumnPaginationOrder {\n        match self {\n            CompositeKey::Bool(_) => ColumnPaginationOrder::Bool,\n            CompositeKey::Str(_) => ColumnPaginationOrder::Str,\n            CompositeKey::F64(_) | CompositeKey::I64(_) | CompositeKey::U64(_) => {\n                ColumnPaginationOrder::Numeric\n            }\n            CompositeKey::Null => panic!(\"null must be handled separately\"),\n        }\n    }\n}\n\n/// After key is a string that encodes the intermediate composite key as \"<type>:<value>\"\n/// A wrapper type for CompositeIntermediateKey that serializes/deserializes\n/// to/from the \"<type>:<value>\" format.\n#[derive(Clone, Debug, PartialEq)]\npub struct AfterKey(pub CompositeIntermediateKey);\n\nimpl Serialize for AfterKey {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: serde::Serializer {\n        let s = match &self.0 {\n            CompositeIntermediateKey::Bool(b) => format!(\"bool:{}\", b),\n            CompositeIntermediateKey::Str(s) => format!(\"str:{}\", s),\n            CompositeIntermediateKey::I64(i) => format!(\"i64:{}\", i),\n            CompositeIntermediateKey::U64(u) => format!(\"u64:{}\", u),\n            CompositeIntermediateKey::F64(f) => format!(\"f64:{}\", f),\n            CompositeIntermediateKey::IpAddr(ip) => format!(\"ip:{}\", ip),\n            CompositeIntermediateKey::DateTime(dt) => format!(\"dt:{}\", dt),\n            CompositeIntermediateKey::Null => \"null:\".to_string(),\n        };\n        serializer.serialize_str(&s)\n    }\n}\n\nimpl<'de> Deserialize<'de> for AfterKey {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: serde::Deserializer<'de> {\n        let s = String::deserialize(deserializer)?;\n        let parts: Vec<&str> = s.splitn(2, ':').collect();\n\n        if parts.len() != 2 {\n            return Err(serde::de::Error::custom(\"invalid after key format\"));\n        }\n\n        let key = match parts[0] {\n            \"bool\" => {\n                let b = parts[1].parse::<bool>().map_err(|e| {\n                    serde::de::Error::custom(format!(\"failed to parse bool: {}\", e))\n                })?;\n                CompositeIntermediateKey::Bool(b)\n            }\n            \"str\" => CompositeIntermediateKey::Str(parts[1].to_string()),\n            \"i64\" => {\n                let i = parts[1]\n                    .parse::<i64>()\n                    .map_err(|e| serde::de::Error::custom(format!(\"failed to parse i64: {}\", e)))?;\n                CompositeIntermediateKey::I64(i)\n            }\n            \"u64\" => {\n                let u = parts[1]\n                    .parse::<u64>()\n                    .map_err(|e| serde::de::Error::custom(format!(\"failed to parse u64: {}\", e)))?;\n                CompositeIntermediateKey::U64(u)\n            }\n            \"f64\" => {\n                let f = parts[1]\n                    .parse::<f64>()\n                    .map_err(|e| serde::de::Error::custom(format!(\"failed to parse f64: {}\", e)))?;\n                if f.is_nan() {\n                    return Err(serde::de::Error::custom(\n                        \"NaN is not supported in after key\",\n                    ));\n                }\n                CompositeIntermediateKey::F64(f)\n            }\n            \"ip\" => {\n                let ip = IpAddr::from_str(parts[1]).map_err(|e: AddrParseError| {\n                    serde::de::Error::custom(format!(\"failed to parse ip: {}\", e))\n                })?;\n                CompositeIntermediateKey::IpAddr(ip.into_ipv6_addr())\n            }\n            \"dt\" => {\n                let dt = parts[1].parse::<i64>().map_err(|e| {\n                    serde::de::Error::custom(format!(\"failed to parse datetime: {}\", e))\n                })?;\n                CompositeIntermediateKey::DateTime(dt)\n            }\n            \"null\" => CompositeIntermediateKey::Null,\n            _ => {\n                return Err(serde::de::Error::custom(\"invalid after key type\"));\n            }\n        };\n\n        Ok(AfterKey(key))\n    }\n}\n\nimpl From<CompositeIntermediateKey> for AfterKey {\n    fn from(key: CompositeIntermediateKey) -> Self {\n        AfterKey(key)\n    }\n}\n\nimpl From<AfterKey> for CompositeIntermediateKey {\n    fn from(value: AfterKey) -> Self {\n        value.0\n    }\n}\n\n/// Calculates the ordering between intermediate keys.\npub fn composite_intermediate_key_ordering(\n    left_opt: &CompositeIntermediateKey,\n    right_opt: &CompositeIntermediateKey,\n    order: Order,\n    missing_order: MissingOrder,\n) -> crate::Result<Ordering> {\n    use CompositeIntermediateKey as CIKey;\n    let mut forced_ordering = false;\n    let asc_ordering = match (left_opt, right_opt) {\n        // null comparisons\n        (CIKey::Null, CIKey::Null) => Ordering::Equal,\n        (CIKey::Null, _) => {\n            forced_ordering = missing_order != MissingOrder::Default;\n            match missing_order {\n                MissingOrder::First => Ordering::Less,\n                MissingOrder::Last => Ordering::Greater,\n                MissingOrder::Default => Ordering::Less,\n            }\n        }\n        (_, CIKey::Null) => {\n            forced_ordering = missing_order != MissingOrder::Default;\n            match missing_order {\n                MissingOrder::First => Ordering::Greater,\n                MissingOrder::Last => Ordering::Less,\n                MissingOrder::Default => Ordering::Greater,\n            }\n        }\n        // same type comparisons\n        (CIKey::Bool(left), CIKey::Bool(right)) => left.cmp(right),\n        (CIKey::I64(left), CIKey::I64(right)) => left.cmp(right),\n        (CIKey::Str(left), CIKey::Str(right)) => left.cmp(right),\n        (CIKey::IpAddr(left), CIKey::IpAddr(right)) => left.cmp(right),\n        (CIKey::DateTime(left), CIKey::DateTime(right)) => left.cmp(right),\n        (CIKey::U64(left), CIKey::U64(right)) => left.cmp(right),\n        (CIKey::F64(f), CIKey::F64(_)) | (CIKey::F64(_), CIKey::F64(f)) if f.is_nan() => {\n            return Err(TantivyError::InvalidArgument(\n                \"NaN comparison is not supported\".to_string(),\n            ))\n        }\n        (CIKey::F64(left), CIKey::F64(right)) => left.partial_cmp(right).unwrap_or(Ordering::Equal),\n        // numeric cross-type comparisons\n        (CIKey::F64(left), CIKey::I64(right)) => cmp_i64_f64(*right, *left)?.reverse(),\n        (CIKey::F64(left), CIKey::U64(right)) => cmp_u64_f64(*right, *left)?.reverse(),\n        (CIKey::I64(left), CIKey::F64(right)) => cmp_i64_f64(*left, *right)?,\n        (CIKey::I64(left), CIKey::U64(right)) => cmp_i64_u64(*left, *right),\n        (CIKey::U64(left), CIKey::I64(right)) => cmp_i64_u64(*right, *left).reverse(),\n        (CIKey::U64(left), CIKey::F64(right)) => cmp_u64_f64(*left, *right)?,\n        // other cross-type comparisons\n        (type_a, type_b) => type_a\n            .column_pagination_order()\n            .cmp(&type_b.column_pagination_order()),\n    };\n    if !forced_ordering && order == Order::Desc {\n        Ok(asc_ordering.reverse())\n    } else {\n        Ok(asc_ordering)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::{Ipv4Addr, Ipv6Addr};\n\n    use serde_json::json;\n    use time::format_description::well_known::Rfc3339;\n    use time::OffsetDateTime;\n\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::exec_request;\n    use crate::schema::{Schema, FAST, STRING};\n    use crate::Index;\n\n    fn datetime_from_iso_str(date_str: &str) -> common::DateTime {\n        let dt = OffsetDateTime::parse(date_str, &Rfc3339)\n            .expect(&format!(\"Failed to parse date: {}\", date_str));\n        let timestamp_secs = dt.unix_timestamp_nanos();\n        common::DateTime::from_timestamp_nanos(timestamp_secs as i64)\n    }\n\n    fn ms_timestamp_from_iso_str(date_str: &str) -> i64 {\n        let dt = OffsetDateTime::parse(date_str, &Rfc3339)\n            .expect(&format!(\"Failed to parse date: {}\", date_str));\n        (dt.unix_timestamp_nanos() / 1_000_000) as i64\n    }\n\n    /// Runs the query and compares the result buckets to the expected buckets,\n    /// then run the same query with a all possible `after` keys and different\n    /// page sizes.\n    fn exec_and_assert_all_paginations(\n        index: &Index,\n        composite_agg_sources: serde_json::Value,\n        expected_buckets: serde_json::Value,\n    ) {\n        let expected_buckets_vec = expected_buckets.as_array().unwrap();\n\n        for page_size in 1..=expected_buckets_vec.len() {\n            let page_count = (expected_buckets_vec.len() + page_size - 1) / page_size;\n            let mut after_key = None;\n            for page_idx in 0..page_count {\n                let mut agg_req_json = json!({\n                    \"my_composite\": {\n                        \"composite\": {\n                            \"sources\": composite_agg_sources,\n                            \"size\": page_size,\n                        }\n                    }\n                });\n                if page_idx > 0 {\n                    agg_req_json[\"my_composite\"][\"composite\"][\"after\"] = after_key.take().unwrap();\n                }\n                let agg_req: Aggregations = serde_json::from_value(agg_req_json).unwrap();\n                let res = exec_request(agg_req.clone(), &index).unwrap();\n                let expected_page_buckets = &expected_buckets_vec[page_idx * page_size\n                    ..std::cmp::min((page_idx + 1) * page_size, expected_buckets_vec.len())];\n                assert_eq!(\n                    &res[\"my_composite\"][\"buckets\"],\n                    &json!(expected_page_buckets),\n                    \"pagination failed at page {}, size {}, query: {:?}\",\n                    page_idx,\n                    page_size,\n                    agg_req,\n                );\n                if page_idx + 1 < page_count {\n                    assert!(\n                        res[\"my_composite\"].get(\"after_key\").is_some(),\n                        \"expected after_key on all but last page\"\n                    );\n                    after_key = Some(res[\"my_composite\"][\"after_key\"].clone());\n                } else if let Some(_) = res[\"my_composite\"].get(\"after_key\") {\n                    // currently we sometime have an after_key on the last page,\n                    // check that the next \"page\" is empty\n                    let agg_req_json = json!({\n                        \"my_composite\": {\n                            \"composite\": {\n                                \"sources\": composite_agg_sources,\n                                \"size\": page_size,\n                                \"after\": res[\"my_composite\"][\"after_key\"].clone(),\n                            }\n                        }\n                    });\n                    let agg_req: Aggregations = serde_json::from_value(agg_req_json).unwrap();\n                    let res = exec_request(agg_req.clone(), &index).unwrap();\n                    assert_eq!(\n                        res[\"my_composite\"][\"buckets\"],\n                        json!([]),\n                        \"expected no buckets when using after_key from last page, query: {:?}\",\n                        agg_req\n                    );\n                }\n            }\n        }\n    }\n\n    fn composite_aggregation_test(merge_segments: bool) -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.add_document(doc!(string_field => \"termb\"))?;\n            index_writer.add_document(doc!(string_field => \"termc\"))?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.add_document(doc!(string_field => \"termb\"))?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.commit()?;\n            if merge_segments {\n                index_writer.wait_merging_threads()?;\n            }\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"term1\": {\"terms\": {\"field\": \"string_id\"}}}\n                    ],\n                    \"size\": 10\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"term1\": \"terma\"}, \"doc_count\": 5},\n                {\"key\": {\"term1\": \"termb\"}, \"doc_count\": 2},\n                {\"key\": {\"term1\": \"termc\"}, \"doc_count\": 1}\n            ])\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_single_segment() -> crate::Result<()> {\n        composite_aggregation_test(true)\n    }\n\n    #[test]\n    fn composite_aggregation_term_multi_segment() -> crate::Result<()> {\n        composite_aggregation_test(false)\n    }\n\n    fn composite_aggregation_term_size_limit(merge_segments: bool) -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.add_document(doc!(string_field => \"termb\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(string_field => \"termc\"))?;\n            index_writer.add_document(doc!(string_field => \"termd\"))?;\n            index_writer.add_document(doc!(string_field => \"terme\"))?;\n            index_writer.commit()?;\n            if merge_segments {\n                index_writer.wait_merging_threads()?;\n            }\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\"}}}\n                    ],\n                    \"size\": 3\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n        // Should only return 3 buckets due to size limit\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"termc\"}, \"doc_count\": 1}\n            ])\n        );\n\n        // next page\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\"}}}\n                    ],\n                    \"size\": 3,\n                    \"after\":  &res[\"my_composite\"][\"after_key\"]\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"myterm\": \"termd\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"terme\"}, \"doc_count\": 1}\n            ])\n        );\n        assert!(res[\"my_composite\"].get(\"after_key\").is_none());\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_size_limit_single_segment() -> crate::Result<()> {\n        composite_aggregation_term_size_limit(true)\n    }\n\n    #[test]\n    fn composite_aggregation_term_size_limit_multi_segment() -> crate::Result<()> {\n        composite_aggregation_term_size_limit(false)\n    }\n\n    #[test]\n    fn composite_aggregation_term_ordering() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"zebra\"))?;\n            index_writer.add_document(doc!(string_field => \"apple\"))?;\n            index_writer.add_document(doc!(string_field => \"banana\"))?;\n            index_writer.add_document(doc!(string_field => \"cherry\"))?;\n            index_writer.add_document(doc!(string_field => \"dog\"))?;\n            index_writer.add_document(doc!(string_field => \"elephant\"))?;\n            index_writer.add_document(doc!(string_field => \"fox\"))?;\n            index_writer.add_document(doc!(string_field => \"grape\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test ascending order (default)\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"fruity_aggreg\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\", \"order\": \"asc\"}}}\n                    ],\n                    \"size\": 5\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"fruity_aggreg\"][\"buckets\"];\n        // Should return only 5 buckets due to size limit, in ascending order\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"myterm\": \"apple\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"banana\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"cherry\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"dog\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"elephant\"}, \"doc_count\": 1}\n            ])\n        );\n\n        // Test descending order\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"fruity_aggreg\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\", \"order\": \"desc\"}}}\n                    ],\n                    \"size\": 5\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"fruity_aggreg\"][\"buckets\"];\n        // Should return only 5 buckets due to size limit, in descending order\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"myterm\": \"zebra\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"grape\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"fox\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"elephant\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"dog\"}, \"doc_count\": 1}\n            ])\n        );\n\n        // next page in descending order\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"fruity_aggreg\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\", \"order\": \"desc\"}}}\n                    ],\n                    \"size\": 5,\n                    \"after\":  &res[\"fruity_aggreg\"][\"after_key\"]\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"fruity_aggreg\"][\"buckets\"];\n        // Should return only 5 buckets due to size limit, in descending order\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"myterm\": \"cherry\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"banana\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"apple\"}, \"doc_count\": 1}\n            ])\n        );\n        assert!(res[\"fruity_aggreg\"].get(\"after_key\").is_none());\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_missing_values() -> crate::Result<()> {\n        // Create index with some documents having missing values\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.add_document(doc!(string_field => \"termb\"))?;\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test without missing bucket (should ignore missing values)\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"myterm\": {\"terms\": {\"field\": \"string_id\", \"missing_bucket\": false}}}\n            ]),\n            json!([\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 2},\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        // Test with missing bucket enabled\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"myterm\": {\"terms\": {\"field\": \"string_id\", \"missing_bucket\": true}}}\n            ]),\n            // Should have 3 buckets including the missing bucket\n            // Missing bucket should come first in ascending order by default\n            json!([\n                {\"key\": {\"myterm\": null}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 2},\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_missing_order() -> crate::Result<()> {\n        // Create index with missing values\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"termb\"))?;\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(string_field => \"terma\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test missing_order: \"first\"\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\n                    \"myterm\": {\n                        \"terms\": {\n                            \"field\": \"string_id\",\n                            \"missing_bucket\": true,\n                            \"missing_order\": \"first\",\n                            \"order\": \"asc\"\n                        }\n                    }\n                }\n            ]),\n            json!([\n                {\"key\": {\"myterm\": null}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        // Test missing_order: \"last\"\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\n                    \"myterm\": {\n                        \"terms\": {\n                            \"field\": \"string_id\",\n                            \"missing_bucket\": true,\n                            \"missing_order\": \"last\",\n                            \"order\": \"asc\"\n                        }\n                    }\n                }\n            ]),\n            json!([\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": null}, \"doc_count\": 1}\n            ]),\n        );\n\n        // Test missing_order: \"default\" with desc order\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\n                    \"myterm\": {\n                        \"terms\": {\n                            \"field\": \"string_id\",\n                            \"missing_bucket\": true,\n                            \"missing_order\": \"default\",\n                            \"order\": \"desc\"\n                        }\n                    }\n                }\n            ]),\n            json!([\n                {\"key\": {\"myterm\": \"termb\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": \"terma\"}, \"doc_count\": 1},\n                {\"key\": {\"myterm\": null}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_multi_source() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let cat = schema_builder.add_text_field(\"category\", STRING | FAST);\n        let status = schema_builder.add_text_field(\"status\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(cat => \"electronics\", status => \"active\"))?;\n            index_writer.add_document(doc!(cat => \"electronics\", status => \"inactive\"))?;\n            index_writer.add_document(doc!(cat => \"electronics\", status => \"active\"))?;\n            index_writer.add_document(doc!(cat => \"books\", status => \"active\"))?;\n            index_writer.add_document(doc!(cat => \"books\", status => \"inactive\"))?;\n            index_writer.add_document(doc!(cat => \"clothing\", status => \"active\"))?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"category\": {\"terms\": {\"field\": \"category\"}}},\n                {\"status\": {\"terms\": {\"field\": \"status\"}}}\n            ]),\n            // Should have composite keys with both dimensions in sorted order\n            json!([\n                {\"key\": {\"category\": \"books\", \"status\": \"active\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"books\", \"status\": \"inactive\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"clothing\", \"status\": \"active\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"electronics\", \"status\": \"active\"}, \"doc_count\": 2},\n                {\"key\": {\"category\": \"electronics\", \"status\": \"inactive\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_multi_source_ordering() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let cat = schema_builder.add_text_field(\"category\", STRING | FAST);\n        let priority = schema_builder.add_text_field(\"priority\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(cat => \"zebra\", priority => \"high\"))?;\n            index_writer.add_document(doc!(cat => \"apple\", priority => \"low\"))?;\n            index_writer.add_document(doc!(cat => \"zebra\", priority => \"low\"))?;\n            index_writer.add_document(doc!(cat => \"apple\", priority => \"high\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test with different ordering on different sources\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"category\": {\"terms\": {\"field\": \"category\", \"order\": \"asc\"}}},\n                {\"priority\": {\"terms\": {\"field\": \"priority\", \"order\": \"desc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"category\": \"apple\", \"priority\": \"low\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"apple\", \"priority\": \"high\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"zebra\", \"priority\": \"low\"}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"zebra\", \"priority\": \"high\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_with_sub_aggregations() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let score_field = schema_builder.add_f64_field(\"score_f64\", FAST);\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(score_field => 5.0f64, string_field => \"terma\"))?;\n            index_writer.add_document(doc!(score_field => 2.0f64, string_field => \"termb\"))?;\n            index_writer.add_document(doc!(score_field => 3.0f64, string_field => \"terma\"))?;\n            index_writer.add_document(doc!(score_field => 7.0f64, string_field => \"termb\"))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\"}}}\n                    ],\n                    \"size\": 10\n                },\n                \"aggs\": {\n                    \"avg_score\": {\n                        \"avg\": {\n                            \"field\": \"score_f64\"\n                        }\n                    },\n                    \"max_score\": {\n                        \"max\": {\n                            \"field\": \"score_f64\"\n                        }\n                    }\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n\n        // Check that sub-aggregations are computed for each bucket with specific values\n        assert_eq!(\n            buckets,\n            &json!([\n                {\n                    \"key\": {\"myterm\": \"terma\"},\n                    \"doc_count\": 2,\n                    \"avg_score\": {\"value\": 4.0}, // (5+3)/2\n                    \"max_score\": {\"value\": 5.0}\n                },\n                {\n                    \"key\": {\"myterm\": \"termb\"},\n                    \"doc_count\": 2,\n                    \"avg_score\": {\"value\": 4.5}, // (2+7)/2\n                    \"max_score\": {\"value\": 7.0}\n                }\n            ])\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_validation_errors() -> crate::Result<()> {\n        // Create index with explicit document creation\n        let mut schema_builder = Schema::builder();\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(string_field => \"term\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test empty sources\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [],\n                    \"size\": 10\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index);\n        assert!(res.is_err());\n\n        // Test size = 0\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"myterm\": {\"terms\": {\"field\": \"string_id\"}}}\n                    ],\n                    \"size\": 0\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index);\n        assert!(res.is_err());\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_numeric_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let score_field = schema_builder.add_f64_field(\"score\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(score_field => 1.0f64))?;\n            index_writer.add_document(doc!(score_field => 2.0f64))?;\n            index_writer.add_document(doc!(score_field => 1.0f64))?;\n            index_writer.add_document(doc!(score_field => 3.33f64))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(score_field => 1.0f64))?;\n            index_writer.commit()?;\n        }\n\n        // Test composite aggregation on numeric field\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"score\": {\"terms\": {\"field\": \"score\"}}}\n            ]),\n            json!([\n                {\"key\": {\"score\": 1}, \"doc_count\": 3},\n                {\"key\": {\"score\": 2}, \"doc_count\": 1},\n                {\"key\": {\"score\": 3.33}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_date_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"timestamp\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            // Add documents with different dates\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2022-01-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T00:00:00Z\")))?; // duplicate\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2023-01-01T00:00:00Z\")))?;\n            index_writer.commit()?;\n        }\n\n        // Test composite aggregation on date field\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"timestamp\": {\"terms\": {\"field\": \"timestamp\"}}}\n            ]),\n            json!([\n                {\"key\": {\"timestamp\": 1609459200000i64}, \"doc_count\": 2},\n                {\"key\": {\"timestamp\": 1640995200000i64}, \"doc_count\": 1},\n                {\"key\": {\"timestamp\": 1672531200000i64}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_ip_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let ip_field = schema_builder.add_ip_addr_field(\"ip_addr\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let ipv4 = |ip: &str| ip.parse::<Ipv4Addr>().unwrap().to_ipv6_mapped();\n            let ipv6 = |ip: &str| ip.parse::<Ipv6Addr>().unwrap();\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(ip_field => ipv4(\"192.168.1.1\")))?;\n            index_writer.add_document(doc!(ip_field => ipv4(\"10.0.0.1\")))?;\n            index_writer.add_document(doc!(ip_field => ipv4(\"192.168.1.1\")))?; // duplicate\n            index_writer.add_document(doc!(ip_field => ipv4(\"172.16.0.1\")))?;\n            index_writer.add_document(doc!(ip_field => ipv6(\"2001:db8::1\")))?;\n            index_writer.add_document(doc!(ip_field => ipv6(\"::1\")))?; // localhost\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(ip_field => ipv6(\"2001:db8::1\")))?; // duplicate\n            index_writer.commit()?;\n        }\n\n        // Test composite aggregation on IP field\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"ip_addr\": {\"terms\": {\"field\": \"ip_addr\"}}}\n            ]),\n            json!([\n                {\"key\": {\"ip_addr\": \"::1\"}, \"doc_count\": 1},\n                {\"key\": {\"ip_addr\": \"10.0.0.1\"}, \"doc_count\": 1},\n                {\"key\": {\"ip_addr\": \"172.16.0.1\"}, \"doc_count\": 1},\n                {\"key\": {\"ip_addr\": \"192.168.1.1\"}, \"doc_count\": 2},\n                {\"key\": {\"ip_addr\": \"2001:db8::1\"}, \"doc_count\": 2}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_multiple_column_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let score_field = schema_builder.add_f64_field(\"score\", FAST);\n        let string_field = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(score_field => 1.0f64, string_field => \"apple\"))?;\n            index_writer.add_document(doc!(score_field => 2.0f64, string_field => \"banana\"))?;\n            index_writer.add_document(doc!(score_field => 1.0f64, string_field => \"apple\"))?;\n            index_writer.add_document(doc!(score_field => 2.0f64, string_field => \"banana\"))?;\n            index_writer.add_document(doc!(score_field => 3.0f64, string_field => \"cherry\"))?;\n            index_writer.add_document(doc!(score_field => 1.0f64, string_field => \"banana\"))?;\n            index_writer.commit()?;\n        }\n\n        // Test composite aggregation mixing numeric and text fields\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"category\": {\"terms\": {\"field\": \"string_id\", \"order\": \"asc\"}}},\n                {\"score\": {\"terms\": {\"field\": \"score\", \"order\": \"desc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"category\": \"apple\", \"score\": 1}, \"doc_count\": 2},\n                {\"key\": {\"category\": \"banana\", \"score\": 2}, \"doc_count\": 2},\n                {\"key\": {\"category\": \"banana\", \"score\": 1}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"cherry\", \"score\": 3}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_json_various_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json_data\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"cat\": \"elec\", \"price\": 999, \"avail\": true})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"cat\": \"books\", \"price\": 15, \"avail\": false})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"cat\": \"elec\", \"price\": 200, \"avail\": true})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"cat\": \"books\", \"price\": 25, \"avail\": true})),\n            )?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"cat\": {\"terms\": {\"field\": \"json_data.cat\"}}},\n                {\"avail\": {\"terms\": {\"field\": \"json_data.avail\"}}},\n                {\"price\": {\"terms\": {\"field\": \"json_data.price\", \"order\": \"desc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"cat\": \"books\", \"avail\": false, \"price\": 15}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"books\", \"avail\": true, \"price\": 25}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"avail\": true, \"price\": 999}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"avail\": true, \"price\": 200}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_json_missing_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json_data\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer\n                .add_document(doc!(json_field => json!({\"cat\": \"elec\", \"brand\": \"apple\"})))?;\n            index_writer\n                .add_document(doc!(json_field => json!({\"cat\": \"books\", \"brand\": \"gut\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"cat\": \"books\"})))?; // missing brand\n            index_writer.add_document(doc!(json_field => json!({\"brand\": \"samsung\"})))?; // missing category\n            index_writer\n                .add_document(doc!(json_field => json!({\"cat\": \"elec\", \"brand\": \"samsung\"})))?;\n            index_writer.commit()?;\n        }\n\n        // Test with missing bucket enabled\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"cat\": {\"terms\": {\"field\": \"json_data.cat\", \"missing_bucket\": true}}},\n                {\"brand\": {\"terms\": {\"field\": \"json_data.brand\", \"missing_bucket\": true, \"missing_order\": \"last\"}}}\n            ]),\n            json!([\n                {\"key\": {\"cat\": null, \"brand\": \"samsung\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"books\", \"brand\": \"gut\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"books\", \"brand\": null}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"brand\": \"apple\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"brand\": \"samsung\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        // Small twist on the missing order of the second source\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"cat\": {\"terms\": {\"field\": \"json_data.cat\", \"missing_bucket\": true}}},\n                {\"brand\": {\"terms\": {\"field\": \"json_data.brand\", \"missing_bucket\": true, \"missing_order\": \"first\"}}}\n            ]),\n            json!([\n                {\"key\": {\"cat\": null, \"brand\": \"samsung\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"books\", \"brand\": null}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"books\", \"brand\": \"gut\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"brand\": \"apple\"}, \"doc_count\": 1},\n                {\"key\": {\"cat\": \"elec\", \"brand\": \"samsung\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_json_nested_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json_data\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"prod\": {\"name\": \"laptop\", \"cpu\": \"intel\"}})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"prod\": {\"name\": \"phone\", \"cpu\": \"snap\"}})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"prod\": {\"name\": \"laptop\", \"cpu\": \"amd\"}})),\n            )?;\n            index_writer.add_document(\n                doc!(json_field => json!({\"prod\": {\"name\": \"tablet\", \"cpu\": \"intel\"}})),\n            )?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"name\": {\"terms\": {\"field\": \"json_data.prod.name\"}}},\n                {\"cpu\": {\"terms\": {\"field\": \"json_data.prod.cpu\"}}}\n            ]),\n            json!([\n                {\"key\": {\"name\": \"laptop\", \"cpu\": \"amd\"}, \"doc_count\": 1},\n                {\"key\": {\"name\": \"laptop\", \"cpu\": \"intel\"}, \"doc_count\": 1},\n                {\"key\": {\"name\": \"phone\", \"cpu\": \"snap\"}, \"doc_count\": 1},\n                {\"key\": {\"name\": \"tablet\", \"cpu\": \"intel\"}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_json_mixed_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json_data\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc1\"})))?;\n            // this segment's numeric is i64\n            index_writer.add_document(doc!(json_field => json!({\"id\": 100})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": true})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc2\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": 50})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": false})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc3\"})))?;\n            index_writer.commit()?;\n            // this segment's numeric is f64\n            index_writer.add_document(doc!(json_field => json!({\"id\": 33.3})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": 50})))?;\n            index_writer.commit()?;\n            // this segment contains dates\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc4\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"2023-01-01T00:00:00Z\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"2023-01-02T00:00:00Z\"})))?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"id\": {\"terms\": {\"field\": \"json_data.id\", \"order\": \"asc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"id\": false}, \"doc_count\": 1},\n                {\"key\": {\"id\": true}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc1\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc2\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc3\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc4\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": 33.3}, \"doc_count\": 1},\n                {\"key\": {\"id\": 50}, \"doc_count\": 2},\n                {\"key\": {\"id\": 100}, \"doc_count\": 1},\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-01T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-02T00:00:00Z\")}, \"doc_count\": 1},\n            ]),\n        );\n\n        // Test descending order\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"id\": {\"terms\": {\"field\": \"json_data.id\", \"order\": \"desc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-02T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-01T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"id\": 100}, \"doc_count\": 1},\n                {\"key\": {\"id\": 50}, \"doc_count\": 2},\n                {\"key\": {\"id\": 33.3}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc4\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc3\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc2\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": \"doc1\"}, \"doc_count\": 1},\n                {\"key\": {\"id\": true}, \"doc_count\": 1},\n                {\"key\": {\"id\": false}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_term_multi_value_fields() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", FAST | STRING);\n        let num_field = schema_builder.add_u64_field(\"num\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            // Document with multiple values for text and num fields\n            index_writer.add_document(doc!(\n                text_field => \"apple\",\n                text_field => \"banana\",\n                num_field => 10u64,\n                num_field => 20u64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"cherry\",\n                num_field => 30u64,\n            ))?;\n            // Multi valued document with duplicate values\n            index_writer.add_document(doc!(\n                text_field => \"elderberry\",\n                text_field => \"date\",\n                text_field => \"elderberry\",\n                num_field => 40u64,\n            ))?;\n\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"text_terms\": {\"terms\": {\"field\": \"text\"}}}\n            ]),\n            json!([\n                {\"key\": {\"text_terms\": \"apple\"}, \"doc_count\": 1},\n                {\"key\": {\"text_terms\": \"banana\"}, \"doc_count\": 1},\n                {\"key\": {\"text_terms\": \"cherry\"}, \"doc_count\": 1},\n                {\"key\": {\"text_terms\": \"date\"}, \"doc_count\": 1},\n                // this is not the doc count but the term occurrence count\n                // https://github.com/quickwit-oss/tantivy/issues/2721\n                {\"key\": {\"text_terms\": \"elderberry\"}, \"doc_count\": 2}\n            ]),\n        );\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"num_terms\": {\"terms\": {\"field\": \"num\"}}}\n            ]),\n            json!([\n                {\"key\": {\"num_terms\": 10}, \"doc_count\": 1},\n                {\"key\": {\"num_terms\": 20}, \"doc_count\": 1},\n                {\"key\": {\"num_terms\": 30}, \"doc_count\": 1},\n                {\"key\": {\"num_terms\": 40}, \"doc_count\": 1}\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_histogram_basic() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let num_field = schema_builder.add_f64_field(\"value\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(num_field => -0.5f64))?;\n            index_writer.add_document(doc!(num_field => 1.0f64))?;\n            index_writer.add_document(doc!(num_field => 2.0f64))?;\n            index_writer.add_document(doc!(num_field => 5.0f64))?;\n            index_writer.add_document(doc!(num_field => 7.0f64))?;\n            index_writer.add_document(doc!(num_field => 11.0f64))?;\n            index_writer.commit()?;\n        }\n\n        // Histogram with interval 5\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"val_hist\": {\"histogram\": {\"field\": \"value\", \"interval\": 5.0}}}\n            ]),\n            json!([\n                {\"key\": {\"val_hist\": -5.0}, \"doc_count\": 1},\n                {\"key\": {\"val_hist\": 0.0}, \"doc_count\": 2},\n                {\"key\": {\"val_hist\": 5.0}, \"doc_count\": 2},\n                {\"key\": {\"val_hist\": 10.0}, \"doc_count\": 1}\n            ]),\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_histogram_json_mixed_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json_data\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            // this segment's numeric is i64\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc1\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": 100})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": true})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc2\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": 50})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": false})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"doc3\"})))?;\n            index_writer.commit()?;\n            // this segment's numeric is f64 and it also contains a date column\n            index_writer.add_document(doc!(json_field => json!({\"id\": 33.3})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": 50})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": -0.01})))?;\n            index_writer.add_document(doc!(json_field => json!({\"id\": \"2023-01-01T00:00:00Z\"})))?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"id\": {\"histogram\": {\"field\": \"json_data.id\", \"interval\": 50, \"order\": \"asc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"id\": -50.0}, \"doc_count\": 1},\n                {\"key\": {\"id\": 0.0}, \"doc_count\": 1},\n                {\"key\": {\"id\": 50.0}, \"doc_count\": 2},\n                {\"key\": {\"id\": 100.0}, \"doc_count\": 1},\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-01T00:00:00Z\") as f64}, \"doc_count\": 1},\n            ]),\n        );\n\n        // Test descending order\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"id\": {\"histogram\": {\"field\": \"json_data.id\", \"interval\": 50, \"order\": \"desc\"}}}\n            ]),\n            json!([\n                {\"key\": {\"id\": ms_timestamp_from_iso_str(\"2023-01-01T00:00:00Z\") as f64},\"doc_count\": 1},\n                {\"key\": {\"id\": 100.0}, \"doc_count\": 1},\n                {\"key\": {\"id\": 50.0}, \"doc_count\": 2},\n                {\"key\": {\"id\": 0.0}, \"doc_count\": 1},\n                {\"key\": {\"id\": -50.0}, \"doc_count\": 1},\n            ]),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_date_histogram_calendar_interval() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"dt\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-02-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2022-01-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2023-01-01T00:00:00Z\")))?;\n            index_writer.commit()?;\n        }\n\n        // Date histogram with calendar_interval = \"year\"\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"dt_hist\": {\"date_histogram\": {\"field\": \"dt\", \"calendar_interval\": \"year\"}}}\n            ]),\n            json!([\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\")}, \"doc_count\": 2},\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2022-01-01T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2023-01-01T00:00:00Z\")}, \"doc_count\": 1}\n            ]),\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_date_histogram_fixed_interval() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"dt\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T00:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T05:30:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T06:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T12:00:00Z\")))?;\n            index_writer\n                .add_document(doc!(date_field => datetime_from_iso_str(\"2021-01-01T18:00:00Z\")))?;\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"dt_hist\": {\"date_histogram\": {\"field\": \"dt\", \"fixed_interval\": \"6h\"}}}\n            ]),\n            json!([\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\")}, \"doc_count\": 2},\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2021-01-01T06:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2021-01-01T12:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"dt_hist\": ms_timestamp_from_iso_str(\"2021-01-01T18:00:00Z\")}, \"doc_count\": 1}\n            ]),\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_mixed_term_and_date_histogram() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"timestamp\", FAST);\n        let category_field = schema_builder.add_text_field(\"category\", STRING | FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-01-01T05:00:00Z\"),\n                category_field => \"electronics\"\n            ))?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-01-15T10:30:00Z\"),\n                category_field => \"electronics\"\n            ))?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-01-05T12:00:00Z\"),\n                category_field => \"books\"\n            ))?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-02-10T08:45:00Z\"),\n                category_field => \"books\"\n            ))?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-02-05T14:20:00Z\"),\n                category_field => \"clothing\"\n            ))?;\n            index_writer.add_document(doc!(\n                date_field => datetime_from_iso_str(\"2021-02-20T09:15:00Z\"),\n                category_field => \"clothing\"\n            ))?;\n\n            index_writer.commit()?;\n        }\n\n        exec_and_assert_all_paginations(\n            &index,\n            json!([\n                {\"category\": {\"terms\": {\"field\": \"category\"}}},\n                {\"month\": {\"date_histogram\": {\"field\": \"timestamp\", \"calendar_interval\": \"month\"}}}\n            ]),\n            json!([\n                {\"key\": {\"category\": \"books\", \"month\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"books\", \"month\": ms_timestamp_from_iso_str(\"2021-02-01T00:00:00Z\")}, \"doc_count\": 1},\n                {\"key\": {\"category\": \"clothing\", \"month\": ms_timestamp_from_iso_str(\"2021-02-01T00:00:00Z\")}, \"doc_count\": 2},\n                {\"key\": {\"category\": \"electronics\", \"month\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\")}, \"doc_count\": 2}\n            ]),\n        );\n\n        // Test with different ordering for sources with a size limit\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"month\": {\"date_histogram\": {\"field\": \"timestamp\", \"calendar_interval\": \"month\"}}},\n                        {\"category\": {\"terms\": {\"field\": \"category\", \"order\": \"desc\"}}}\n                    ],\n                    \"size\": 3\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"month\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\"), \"category\": \"electronics\"}, \"doc_count\": 2},\n                {\"key\": {\"month\": ms_timestamp_from_iso_str(\"2021-01-01T00:00:00Z\"), \"category\": \"books\"}, \"doc_count\": 1},\n                {\"key\": {\"month\": ms_timestamp_from_iso_str(\"2021-02-01T00:00:00Z\"), \"category\": \"clothing\"}, \"doc_count\": 2},\n            ]),\n        );\n\n        // next page\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"month\": {\"date_histogram\": {\"field\": \"timestamp\", \"calendar_interval\": \"month\"}}},\n                        {\"category\": {\"terms\": {\"field\": \"category\", \"order\": \"desc\"}}}\n                    ],\n                    \"size\": 3,\n                    \"after\": res[\"my_composite\"][\"after_key\"]\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n        assert_eq!(\n            buckets,\n            &json!([\n                {\"key\": {\"month\": ms_timestamp_from_iso_str(\"2021-02-01T00:00:00Z\"), \"category\": \"books\"}, \"doc_count\": 1},\n            ]),\n        );\n        assert!(res[\"my_composite\"].get(\"after_key\").is_none());\n\n        Ok(())\n    }\n\n    #[test]\n    fn composite_aggregation_no_matching_columns() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_f64_field(\"dt\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.add_document(doc!(date_field => 1.0))?;\n            index_writer.add_document(doc!(date_field => 2.0))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"dt_hist\": {\"date_histogram\": {\"field\": \"dt\", \"fixed_interval\": \"6h\"}}}\n                    ],\n                    \"size\": 10\n                }\n            }\n        }))\n        .unwrap();\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n        assert_eq!(buckets, &json!([]));\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_composite\": {\n                \"composite\": {\n                    \"sources\": [\n                        {\"dt_hist\": {\"date_histogram\": {\"field\": \"dt\", \"fixed_interval\": \"6h\", \"missing_bucket\": true}}}\n                    ],\n                    \"size\": 10,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        let buckets = &res[\"my_composite\"][\"buckets\"];\n\n        assert_eq!(\n            buckets,\n            &json!([{\"key\": {\"dt_hist\": null}, \"doc_count\": 2}])\n        );\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/composite/numeric_types.rs",
    "content": "/// This module helps comparing numerical values of different types (i64, u64\n/// and f64).\npub(super) mod num_cmp {\n    use std::cmp::Ordering;\n\n    use crate::TantivyError;\n\n    pub fn cmp_i64_f64(left_i: i64, right_f: f64) -> crate::Result<Ordering> {\n        if right_f.is_nan() {\n            return Err(TantivyError::InvalidArgument(\n                \"NaN comparison is not supported\".to_string(),\n            ));\n        }\n\n        // If right_f is < i64::MIN then left_i > right_f (i64::MIN=-2^63 can be\n        // exactly represented as f64)\n        if right_f < i64::MIN as f64 {\n            return Ok(Ordering::Greater);\n        }\n        // If right_f is >= i64::MAX then left_i < right_f (i64::MAX=2^63-1 cannot\n        // be exactly represented as f64)\n        if right_f >= i64::MAX as f64 {\n            return Ok(Ordering::Less);\n        }\n\n        // Now right_f is in (i64::MIN, i64::MAX), so `right_f as i64` is\n        // well-defined (truncation toward 0)\n        let right_as_i = right_f as i64;\n\n        let result = match left_i.cmp(&right_as_i) {\n            Ordering::Less => Ordering::Less,\n            Ordering::Greater => Ordering::Greater,\n            Ordering::Equal => {\n                // they have the same integer part, compare the fraction\n                let rem = right_f - (right_as_i as f64);\n                if rem == 0.0 {\n                    Ordering::Equal\n                } else if right_f > 0.0 {\n                    Ordering::Less\n                } else {\n                    Ordering::Greater\n                }\n            }\n        };\n        Ok(result)\n    }\n\n    pub fn cmp_u64_f64(left_u: u64, right_f: f64) -> crate::Result<Ordering> {\n        if right_f.is_nan() {\n            return Err(TantivyError::InvalidArgument(\n                \"NaN comparison is not supported\".to_string(),\n            ));\n        }\n\n        // Negative floats are always less than any u64 >= 0\n        if right_f < 0.0 {\n            return Ok(Ordering::Greater);\n        }\n\n        // If right_f is >= u64::MAX then left_u < right_f (u64::MAX=2^64-1 cannot be exactly)\n        let max_as_f = u64::MAX as f64;\n        if right_f > max_as_f {\n            return Ok(Ordering::Less);\n        }\n\n        // Now right_f is in (0, u64::MAX), so `right_f as u64` is well-defined\n        // (truncation toward 0)\n        let right_as_u = right_f as u64;\n\n        let result = match left_u.cmp(&right_as_u) {\n            Ordering::Less => Ordering::Less,\n            Ordering::Greater => Ordering::Greater,\n            Ordering::Equal => {\n                // they have the same integer part, compare the fraction\n                let rem = right_f - (right_as_u as f64);\n                if rem == 0.0 {\n                    Ordering::Equal\n                } else {\n                    Ordering::Less\n                }\n            }\n        };\n        Ok(result)\n    }\n\n    pub fn cmp_i64_u64(left_i: i64, right_u: u64) -> Ordering {\n        if left_i < 0 {\n            Ordering::Less\n        } else {\n            let left_as_u = left_i as u64;\n            left_as_u.cmp(&right_u)\n        }\n    }\n}\n\n/// This module helps projecting numerical values to other numerical types.\n/// When the target value space cannot exactly represent the source value, the\n/// next representable value is returned (or AfterLast if the source value is\n/// larger than the largest representable value).\n///\n/// All functions in this module assume that f64 values are not NaN.\npub(super) mod num_proj {\n    #[derive(Debug, PartialEq)]\n    pub enum ProjectedNumber<T> {\n        Exact(T),\n        Next(T),\n        AfterLast,\n    }\n\n    pub fn i64_to_u64(value: i64) -> ProjectedNumber<u64> {\n        if value < 0 {\n            ProjectedNumber::Next(0)\n        } else {\n            ProjectedNumber::Exact(value as u64)\n        }\n    }\n\n    pub fn u64_to_i64(value: u64) -> ProjectedNumber<i64> {\n        if value > i64::MAX as u64 {\n            ProjectedNumber::AfterLast\n        } else {\n            ProjectedNumber::Exact(value as i64)\n        }\n    }\n\n    pub fn f64_to_u64(value: f64) -> ProjectedNumber<u64> {\n        if value < 0.0 {\n            ProjectedNumber::Next(0)\n        } else if value > u64::MAX as f64 {\n            ProjectedNumber::AfterLast\n        } else if value.fract() == 0.0 {\n            ProjectedNumber::Exact(value as u64)\n        } else {\n            // casting f64 to u64 truncates toward zero\n            ProjectedNumber::Next(value as u64 + 1)\n        }\n    }\n\n    pub fn f64_to_i64(value: f64) -> ProjectedNumber<i64> {\n        if value < (i64::MIN as f64) {\n            ProjectedNumber::Next(i64::MIN)\n        } else if value >= (i64::MAX as f64) {\n            ProjectedNumber::AfterLast\n        } else if value.fract() == 0.0 {\n            ProjectedNumber::Exact(value as i64)\n        } else if value > 0.0 {\n            // casting f64 to i64 truncates toward zero\n            ProjectedNumber::Next(value as i64 + 1)\n        } else {\n            ProjectedNumber::Next(value as i64)\n        }\n    }\n\n    pub fn i64_to_f64(value: i64) -> ProjectedNumber<f64> {\n        let value_f = value as f64;\n        let k_roundtrip = value_f as i64;\n        if k_roundtrip == value {\n            // between -2^53 and 2^53 all i64 are exactly represented as f64\n            ProjectedNumber::Exact(value_f)\n        } else {\n            // for very large/small i64 values, it is approximated to the closest f64\n            if k_roundtrip > value {\n                ProjectedNumber::Next(value_f)\n            } else {\n                ProjectedNumber::Next(value_f.next_up())\n            }\n        }\n    }\n\n    pub fn u64_to_f64(value: u64) -> ProjectedNumber<f64> {\n        let value_f = value as f64;\n        let k_roundtrip = value_f as u64;\n        if k_roundtrip == value {\n            // between 0 and 2^53 all u64 are exactly represented as f64\n            ProjectedNumber::Exact(value_f)\n        } else if k_roundtrip > value {\n            ProjectedNumber::Next(value_f)\n        } else {\n            ProjectedNumber::Next(value_f.next_up())\n        }\n    }\n}\n\n#[cfg(test)]\nmod num_cmp_tests {\n    use std::cmp::Ordering;\n\n    use super::num_cmp::*;\n\n    #[test]\n    fn test_cmp_u64_f64() {\n        // Basic comparisons\n        assert_eq!(cmp_u64_f64(5, 5.0).unwrap(), Ordering::Equal);\n        assert_eq!(cmp_u64_f64(5, 6.0).unwrap(), Ordering::Less);\n        assert_eq!(cmp_u64_f64(6, 5.0).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_u64_f64(0, 0.0).unwrap(), Ordering::Equal);\n        assert_eq!(cmp_u64_f64(0, 0.1).unwrap(), Ordering::Less);\n\n        // Negative float values should always be less than any u64\n        assert_eq!(cmp_u64_f64(0, -0.1).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_u64_f64(5, -5.0).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_u64_f64(u64::MAX, -1e20).unwrap(), Ordering::Greater);\n\n        // Tests with extreme values\n        assert_eq!(cmp_u64_f64(u64::MAX, 1e20).unwrap(), Ordering::Less);\n\n        // Precision edge cases: large u64 that loses precision when converted to f64\n        // => 2^54, exactly represented as f64\n        let large_f64 = 18_014_398_509_481_984.0;\n        let large_u64 = 18_014_398_509_481_984;\n        // prove that large_u64 is exactly represented as f64\n        assert_eq!(large_u64 as f64, large_f64);\n        assert_eq!(cmp_u64_f64(large_u64, large_f64).unwrap(), Ordering::Equal);\n        // => (2^54 + 1) cannot be exactly represented in f64\n        let large_u64_plus_1 = 18_014_398_509_481_985;\n        // prove that it is represented as f64 by large_f64\n        assert_eq!(large_u64_plus_1 as f64, large_f64);\n        assert_eq!(\n            cmp_u64_f64(large_u64_plus_1, large_f64).unwrap(),\n            Ordering::Greater\n        );\n        // => (2^54 - 1) cannot be exactly represented in f64\n        let large_u64_minus_1 = 18_014_398_509_481_983;\n        // prove that it is also represented as f64 by large_f64\n        assert_eq!(large_u64_minus_1 as f64, large_f64);\n        assert_eq!(\n            cmp_u64_f64(large_u64_minus_1, large_f64).unwrap(),\n            Ordering::Less\n        );\n\n        // NaN comparison results in an error\n        assert!(cmp_u64_f64(0, f64::NAN).is_err());\n    }\n\n    #[test]\n    fn test_cmp_i64_f64() {\n        // Basic comparisons\n        assert_eq!(cmp_i64_f64(5, 5.0).unwrap(), Ordering::Equal);\n        assert_eq!(cmp_i64_f64(5, 6.0).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(6, 5.0).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_i64_f64(-5, -5.0).unwrap(), Ordering::Equal);\n        assert_eq!(cmp_i64_f64(-5, -4.0).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(-4, -5.0).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_i64_f64(-5, 5.0).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(5, -5.0).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_i64_f64(0, -0.1).unwrap(), Ordering::Greater);\n        assert_eq!(cmp_i64_f64(0, 0.1).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(-1, -0.5).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(-1, 0.0).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(0, 0.0).unwrap(), Ordering::Equal);\n\n        // Tests with extreme values\n        assert_eq!(cmp_i64_f64(i64::MAX, 1e20).unwrap(), Ordering::Less);\n        assert_eq!(cmp_i64_f64(i64::MIN, -1e20).unwrap(), Ordering::Greater);\n\n        // Precision edge cases: large i64 that loses precision when converted to f64\n        // => 2^54, exactly represented as f64\n        let large_f64 = 18_014_398_509_481_984.0;\n        let large_i64 = 18_014_398_509_481_984;\n        // prove that large_i64 is exactly represented as f64\n        assert_eq!(large_i64 as f64, large_f64);\n        assert_eq!(cmp_i64_f64(large_i64, large_f64).unwrap(), Ordering::Equal);\n        // => (1_i64 << 54) + 1 cannot be exactly represented in f64\n        let large_i64_plus_1 = 18_014_398_509_481_985;\n        // prove that it is represented as f64 by large_f64\n        assert_eq!(large_i64_plus_1 as f64, large_f64);\n        assert_eq!(\n            cmp_i64_f64(large_i64_plus_1, large_f64).unwrap(),\n            Ordering::Greater\n        );\n        // => (1_i64 << 54) - 1 cannot be exactly represented in f64\n        let large_i64_minus_1 = 18_014_398_509_481_983;\n        // prove that it is also represented as f64 by large_f64\n        assert_eq!(large_i64_minus_1 as f64, large_f64);\n        assert_eq!(\n            cmp_i64_f64(large_i64_minus_1, large_f64).unwrap(),\n            Ordering::Less\n        );\n\n        // Same precision edge case but with negative values\n        // => -2^54, exactly represented as f64\n        let large_neg_f64 = -18_014_398_509_481_984.0;\n        let large_neg_i64 = -18_014_398_509_481_984;\n        // prove that large_neg_i64 is exactly represented as f64\n        assert_eq!(large_neg_i64 as f64, large_neg_f64);\n        assert_eq!(\n            cmp_i64_f64(large_neg_i64, large_neg_f64).unwrap(),\n            Ordering::Equal\n        );\n        // => (-2^54 + 1) cannot be exactly represented in f64\n        let large_neg_i64_plus_1 = -18_014_398_509_481_985;\n        // prove that it is represented as f64 by large_neg_f64\n        assert_eq!(large_neg_i64_plus_1 as f64, large_neg_f64);\n        assert_eq!(\n            cmp_i64_f64(large_neg_i64_plus_1, large_neg_f64).unwrap(),\n            Ordering::Less\n        );\n        // => (-2^54 - 1) cannot be exactly represented in f64\n        let large_neg_i64_minus_1 = -18_014_398_509_481_983;\n        // prove that it is also represented as f64 by large_neg_f64\n        assert_eq!(large_neg_i64_minus_1 as f64, large_neg_f64);\n        assert_eq!(\n            cmp_i64_f64(large_neg_i64_minus_1, large_neg_f64).unwrap(),\n            Ordering::Greater\n        );\n\n        // NaN comparison results in an error\n        assert!(cmp_i64_f64(0, f64::NAN).is_err());\n    }\n\n    #[test]\n    fn test_cmp_i64_u64() {\n        // Test with negative i64 values (should always be less than any u64)\n        assert_eq!(cmp_i64_u64(-1, 0), Ordering::Less);\n        assert_eq!(cmp_i64_u64(i64::MIN, 0), Ordering::Less);\n        assert_eq!(cmp_i64_u64(i64::MIN, u64::MAX), Ordering::Less);\n\n        // Test with positive i64 values\n        assert_eq!(cmp_i64_u64(0, 0), Ordering::Equal);\n        assert_eq!(cmp_i64_u64(1, 0), Ordering::Greater);\n        assert_eq!(cmp_i64_u64(1, 1), Ordering::Equal);\n        assert_eq!(cmp_i64_u64(0, 1), Ordering::Less);\n        assert_eq!(cmp_i64_u64(5, 10), Ordering::Less);\n        assert_eq!(cmp_i64_u64(10, 5), Ordering::Greater);\n\n        // Test with values near i64::MAX and u64 conversion\n        assert_eq!(cmp_i64_u64(i64::MAX, i64::MAX as u64), Ordering::Equal);\n        assert_eq!(cmp_i64_u64(i64::MAX, (i64::MAX as u64) + 1), Ordering::Less);\n        assert_eq!(cmp_i64_u64(i64::MAX, u64::MAX), Ordering::Less);\n    }\n}\n\n#[cfg(test)]\nmod num_proj_tests {\n    use super::num_proj::{self, ProjectedNumber};\n\n    #[test]\n    fn test_i64_to_u64() {\n        assert_eq!(num_proj::i64_to_u64(-1), ProjectedNumber::Next(0));\n        assert_eq!(num_proj::i64_to_u64(i64::MIN), ProjectedNumber::Next(0));\n        assert_eq!(num_proj::i64_to_u64(0), ProjectedNumber::Exact(0));\n        assert_eq!(num_proj::i64_to_u64(42), ProjectedNumber::Exact(42));\n        assert_eq!(\n            num_proj::i64_to_u64(i64::MAX),\n            ProjectedNumber::Exact(i64::MAX as u64)\n        );\n    }\n\n    #[test]\n    fn test_u64_to_i64() {\n        assert_eq!(num_proj::u64_to_i64(0), ProjectedNumber::Exact(0));\n        assert_eq!(num_proj::u64_to_i64(42), ProjectedNumber::Exact(42));\n        assert_eq!(\n            num_proj::u64_to_i64(i64::MAX as u64),\n            ProjectedNumber::Exact(i64::MAX)\n        );\n        assert_eq!(\n            num_proj::u64_to_i64((i64::MAX as u64) + 1),\n            ProjectedNumber::AfterLast\n        );\n        assert_eq!(num_proj::u64_to_i64(u64::MAX), ProjectedNumber::AfterLast);\n    }\n\n    #[test]\n    fn test_f64_to_u64() {\n        assert_eq!(num_proj::f64_to_u64(-1e25), ProjectedNumber::Next(0));\n        assert_eq!(num_proj::f64_to_u64(-0.1), ProjectedNumber::Next(0));\n        assert_eq!(num_proj::f64_to_u64(1e20), ProjectedNumber::AfterLast);\n        assert_eq!(\n            num_proj::f64_to_u64(f64::INFINITY),\n            ProjectedNumber::AfterLast\n        );\n        assert_eq!(num_proj::f64_to_u64(0.0), ProjectedNumber::Exact(0));\n        assert_eq!(num_proj::f64_to_u64(42.0), ProjectedNumber::Exact(42));\n        assert_eq!(num_proj::f64_to_u64(0.5), ProjectedNumber::Next(1));\n        assert_eq!(num_proj::f64_to_u64(42.1), ProjectedNumber::Next(43));\n    }\n\n    #[test]\n    fn test_f64_to_i64() {\n        assert_eq!(num_proj::f64_to_i64(-1e20), ProjectedNumber::Next(i64::MIN));\n        assert_eq!(\n            num_proj::f64_to_i64(f64::NEG_INFINITY),\n            ProjectedNumber::Next(i64::MIN)\n        );\n        assert_eq!(num_proj::f64_to_i64(1e20), ProjectedNumber::AfterLast);\n        assert_eq!(\n            num_proj::f64_to_i64(f64::INFINITY),\n            ProjectedNumber::AfterLast\n        );\n        assert_eq!(num_proj::f64_to_i64(0.0), ProjectedNumber::Exact(0));\n        assert_eq!(num_proj::f64_to_i64(42.0), ProjectedNumber::Exact(42));\n        assert_eq!(num_proj::f64_to_i64(-42.0), ProjectedNumber::Exact(-42));\n        assert_eq!(num_proj::f64_to_i64(0.5), ProjectedNumber::Next(1));\n        assert_eq!(num_proj::f64_to_i64(42.1), ProjectedNumber::Next(43));\n        assert_eq!(num_proj::f64_to_i64(-0.5), ProjectedNumber::Next(0));\n        assert_eq!(num_proj::f64_to_i64(-42.1), ProjectedNumber::Next(-42));\n    }\n\n    #[test]\n    fn test_i64_to_f64() {\n        assert_eq!(num_proj::i64_to_f64(0), ProjectedNumber::Exact(0.0));\n        assert_eq!(num_proj::i64_to_f64(42), ProjectedNumber::Exact(42.0));\n        assert_eq!(num_proj::i64_to_f64(-42), ProjectedNumber::Exact(-42.0));\n\n        let max_exact = 9_007_199_254_740_992; // 2^53\n        assert_eq!(\n            num_proj::i64_to_f64(max_exact),\n            ProjectedNumber::Exact(max_exact as f64)\n        );\n\n        // Test values that cannot be exactly represented as f64 (integers above 2^53)\n        let large_i64 = 9_007_199_254_740_993; // 2^53 + 1\n        let closest_f64 = 9_007_199_254_740_992.0;\n        assert_eq!(large_i64 as f64, closest_f64);\n        if let ProjectedNumber::Next(val) = num_proj::i64_to_f64(large_i64) {\n            // Verify that the returned float is different from the direct cast\n            assert!(val > closest_f64);\n            assert!(val - closest_f64 < 2. * f64::EPSILON * closest_f64);\n        } else {\n            panic!(\"Expected ProjectedNumber::Next for large_i64\");\n        }\n\n        // Test with very large negative value\n        let large_neg_i64 = -9_007_199_254_740_993; // -(2^53 + 1)\n        let closest_neg_f64 = -9_007_199_254_740_992.0;\n        assert_eq!(large_neg_i64 as f64, closest_neg_f64);\n        if let ProjectedNumber::Next(val) = num_proj::i64_to_f64(large_neg_i64) {\n            // Verify that the returned float is the closest representable f64\n            assert_eq!(val, closest_neg_f64);\n        } else {\n            panic!(\"Expected ProjectedNumber::Next for large_neg_i64\");\n        }\n    }\n\n    #[test]\n    fn test_u64_to_f64() {\n        assert_eq!(num_proj::u64_to_f64(0), ProjectedNumber::Exact(0.0));\n        assert_eq!(num_proj::u64_to_f64(42), ProjectedNumber::Exact(42.0));\n\n        // Test the largest u64 value that can be exactly represented as f64 (2^53)\n        let max_exact = 9_007_199_254_740_992; // 2^53\n        assert_eq!(\n            num_proj::u64_to_f64(max_exact),\n            ProjectedNumber::Exact(max_exact as f64)\n        );\n\n        // Test values that cannot be exactly represented as f64 (integers above 2^53)\n        let large_u64 = 9_007_199_254_740_993; // 2^53 + 1\n        let closest_f64 = 9_007_199_254_740_992.0;\n        assert_eq!(large_u64 as f64, closest_f64);\n        if let ProjectedNumber::Next(val) = num_proj::u64_to_f64(large_u64) {\n            // Verify that the returned float is different from the direct cast\n            assert!(val > closest_f64);\n            assert!(val - closest_f64 < 2. * f64::EPSILON * closest_f64);\n        } else {\n            panic!(\"Expected ProjectedNumber::Next for large_u64\");\n        }\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/filter.rs",
    "content": "use std::fmt::Debug;\n\nuse common::BitSet;\nuse serde::{Deserialize, Deserializer, Serialize, Serializer};\n\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::cached_sub_aggs::{\n    CachedSubAggs, HighCardSubAggCache, LowCardSubAggCache, SubAggCache,\n};\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateBucketResult,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::BucketId;\nuse crate::docset::DocSet;\nuse crate::query::{AllQuery, EnableScoring, Query, QueryParser};\nuse crate::schema::Schema;\nuse crate::tokenizer::TokenizerManager;\nuse crate::{DocId, SegmentReader, TantivyError};\n\n/// A trait for query builders that can build queries programmatically.\n///\n/// This trait enables programmatic query construction for filter aggregations with\n/// full serialization/deserialization support for distributed aggregation scenarios.\n///\n/// # Why This Exists\n///\n/// Filter aggregations need to support both:\n/// - Query strings (simple, always serializable)\n/// - Programmatic query construction (flexible, with serialization support)\n///\n/// This trait provides the programmatic query construction capability with full\n/// serialization support via the `typetag` crate.\n///\n/// # Implementation Requirements\n///\n/// Implementers must:\n/// 1. Derive `Debug`, `Clone`, `Serialize`, and `Deserialize`\n/// 2. Use `#[typetag::serde]` attribute on the impl block\n/// 3. Implement `build_query()` to construct the query from schema/tokenizers\n/// 4. Implement `box_clone()` to enable cloning (typically just `Box::new(self.clone())`)\n///\n/// # Example\n///\n/// ```rust\n/// use tantivy::aggregation::bucket::QueryBuilder;\n/// use tantivy::query::{Query, TermQuery};\n/// use tantivy::schema::{Schema, IndexRecordOption};\n/// use tantivy::tokenizer::TokenizerManager;\n/// use tantivy::Term;\n/// use serde::{Serialize, Deserialize};\n///\n/// #[derive(Debug, Clone, Serialize, Deserialize)]\n/// struct TermQueryBuilder {\n///     field_name: String,\n///     term_text: String,\n/// }\n///\n/// #[typetag::serde]\n/// impl QueryBuilder for TermQueryBuilder {\n///     fn build_query(\n///         &self,\n///         schema: &Schema,\n///         _tokenizers: &TokenizerManager,\n///     ) -> tantivy::Result<Box<dyn Query>> {\n///         let field = schema.get_field(&self.field_name)?;\n///         let term = Term::from_field_text(field, &self.term_text);\n///         Ok(Box::new(TermQuery::new(term, IndexRecordOption::Basic)))\n///     }\n///\n///     fn box_clone(&self) -> Box<dyn QueryBuilder> {\n///         Box::new(self.clone())\n///     }\n/// }\n///\n/// // Create an instance\n/// let builder = TermQueryBuilder {\n///     field_name: \"category\".to_string(),\n///     term_text: \"electronics\".to_string(),\n/// };\n/// ```\n#[typetag::serde(tag = \"type\")]\npub trait QueryBuilder: Debug + Send + Sync {\n    /// Build a query from the given schema and tokenizer manager.\n    ///\n    /// This method is called once when creating the FilterAggReqData for a segment.\n    ///\n    /// # Parameters\n    /// - `schema`: The index schema for field lookups\n    /// - `tokenizers`: The tokenizer manager for text analysis\n    ///\n    /// # Returns\n    /// A boxed Query object, or an error if construction fails\n    fn build_query(\n        &self,\n        schema: &Schema,\n        tokenizers: &TokenizerManager,\n    ) -> crate::Result<Box<dyn Query>>;\n\n    /// Clone this builder into a boxed trait object.\n    ///\n    /// Since builders are just data (no state), this simply clones the data.\n    /// The typical implementation is:\n    /// ```rust,ignore\n    /// fn box_clone(&self) -> Box<dyn QueryBuilder> {\n    ///     Box::new(self.clone())\n    /// }\n    /// ```\n    fn box_clone(&self) -> Box<dyn QueryBuilder>;\n}\n\n/// Filter aggregation creates a single bucket containing documents that match a query.\n///\n/// # Usage\n///\n/// ## Query String (Recommended)\n/// ```rust\n/// use tantivy::aggregation::bucket::FilterAggregation;\n///\n/// // Query strings are parsed using Tantivy's standard QueryParser\n/// let filter_agg = FilterAggregation::new(\"category:electronics AND price:[100 TO 500]\".to_string());\n/// ```\n///\n/// ## Custom Query Builder\n/// ```rust\n/// use tantivy::aggregation::bucket::{FilterAggregation, QueryBuilder};\n/// use tantivy::query::{Query, TermQuery};\n/// use tantivy::schema::{Schema, IndexRecordOption};\n/// use tantivy::tokenizer::TokenizerManager;\n/// use tantivy::Term;\n/// use serde::{Serialize, Deserialize};\n///\n/// #[derive(Debug, Clone, Serialize, Deserialize)]\n/// struct MyBuilder {\n///     field_name: String,\n///     term_text: String,\n/// }\n///\n/// #[typetag::serde]\n/// impl QueryBuilder for MyBuilder {\n///     fn build_query(\n///         &self,\n///         schema: &Schema,\n///         _tokenizers: &TokenizerManager,\n///     ) -> tantivy::Result<Box<dyn Query>> {\n///         let field = schema.get_field(&self.field_name)?;\n///         let term = Term::from_field_text(field, &self.term_text);\n///         Ok(Box::new(TermQuery::new(term, IndexRecordOption::Basic)))\n///     }\n///\n///     fn box_clone(&self) -> Box<dyn QueryBuilder> {\n///         Box::new(self.clone())\n///     }\n/// }\n///\n/// let builder = MyBuilder {\n///     field_name: \"category\".to_string(),\n///     term_text: \"electronics\".to_string(),\n/// };\n/// let filter_agg = FilterAggregation::new_with_builder(Box::new(builder));\n/// ```\n///\n/// # Result\n/// The filter aggregation returns a single bucket with:\n/// - `doc_count`: Number of documents matching the filter\n/// - Sub-aggregation results computed on the filtered document set\n#[derive(Debug, Clone)]\npub struct FilterAggregation {\n    /// The query for filtering - can be either a query string or a query builder\n    query: FilterQuery,\n}\n\n/// Represents different ways to specify a filter query\npub enum FilterQuery {\n    /// Query string that will be parsed using Tantivy's standard parsing facilities\n    ///\n    /// This is the recommended approach as it's serializable and doesn't carry runtime state.\n    QueryString(String),\n\n    /// Custom query builder for programmatic query building\n    ///\n    /// This variant stores a builder that builds the query once when creating FilterAggReqData.\n    ///\n    /// This is useful for:\n    /// - Custom query types not expressible as query strings\n    /// - Programmatic query construction based on schema\n    /// - Extension query types\n    ///\n    /// **Note**: The builder is serializable and can be deserialized.\n    CustomBuilder(Box<dyn QueryBuilder>),\n}\n\nimpl Debug for FilterQuery {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            FilterQuery::QueryString(s) => f.debug_tuple(\"QueryString\").field(s).finish(),\n            FilterQuery::CustomBuilder(_) => {\n                f.debug_struct(\"CustomBuilder\").finish_non_exhaustive()\n            }\n        }\n    }\n}\n\nimpl Clone for FilterQuery {\n    fn clone(&self) -> Self {\n        match self {\n            FilterQuery::QueryString(query_string) => {\n                FilterQuery::QueryString(query_string.clone())\n            }\n            FilterQuery::CustomBuilder(builder) => FilterQuery::CustomBuilder(builder.box_clone()),\n        }\n    }\n}\n\nimpl FilterAggregation {\n    /// Create a new filter aggregation with a query string\n    /// The query string will be parsed using the QueryParser::parse_query() method.\n    pub fn new(query_string: String) -> Self {\n        Self {\n            query: FilterQuery::QueryString(query_string),\n        }\n    }\n\n    /// Create a new filter aggregation with a query builder\n    ///\n    /// The builder will be called once when creating the FilterAggReqData for each segment.\n    ///\n    /// # Example\n    /// ```rust\n    /// use tantivy::aggregation::bucket::{FilterAggregation, QueryBuilder};\n    /// use tantivy::query::{Query, TermQuery};\n    /// use tantivy::schema::{Schema, IndexRecordOption};\n    /// use tantivy::tokenizer::TokenizerManager;\n    /// use tantivy::Term;\n    /// use serde::{Serialize, Deserialize};\n    ///\n    /// #[derive(Debug, Clone, Serialize, Deserialize)]\n    /// struct MyBuilder {\n    ///     field_name: String,\n    ///     term_text: String,\n    /// }\n    ///\n    /// #[typetag::serde]\n    /// impl QueryBuilder for MyBuilder {\n    ///     fn build_query(\n    ///         &self,\n    ///         schema: &Schema,\n    ///         _tokenizers: &TokenizerManager,\n    ///     ) -> tantivy::Result<Box<dyn Query>> {\n    ///         let field = schema.get_field(&self.field_name)?;\n    ///         let term = Term::from_field_text(field, &self.term_text);\n    ///         Ok(Box::new(TermQuery::new(term, IndexRecordOption::Basic)))\n    ///     }\n    ///\n    ///     fn box_clone(&self) -> Box<dyn QueryBuilder> {\n    ///         Box::new(self.clone())\n    ///     }\n    /// }\n    ///\n    /// let builder = MyBuilder {\n    ///     field_name: \"category\".to_string(),\n    ///     term_text: \"electronics\".to_string(),\n    /// };\n    /// let filter_agg = FilterAggregation::new_with_builder(Box::new(builder));\n    /// ```\n    pub fn new_with_builder(builder: Box<dyn QueryBuilder>) -> Self {\n        Self {\n            query: FilterQuery::CustomBuilder(builder),\n        }\n    }\n\n    /// Parse the query into a Tantivy Query object\n    ///\n    /// For query strings, this uses the QueryParser::parse_query() method.\n    /// For custom builders, builds the query using the builder.\n    pub(crate) fn parse_query(\n        &self,\n        schema: &Schema,\n        tokenizer_manager: &TokenizerManager,\n    ) -> crate::Result<Box<dyn Query>> {\n        match &self.query {\n            FilterQuery::QueryString(query_str) => {\n                let query_parser =\n                    QueryParser::new(schema.clone(), vec![], tokenizer_manager.clone());\n\n                query_parser\n                    .parse_query(query_str)\n                    .map_err(|e| TantivyError::InvalidArgument(e.to_string()))\n            }\n            FilterQuery::CustomBuilder(builder) => {\n                // Build the query using the builder\n                builder.build_query(schema, tokenizer_manager)\n            }\n        }\n    }\n\n    /// Parse the query with a custom QueryParser\n    ///\n    /// This method allows using a pre-configured QueryParser with custom settings\n    /// like field boosts, fuzzy matching, default fields, etc.\n    ///\n    /// For custom builders, this method is not supported and will return an error.\n    /// Custom builders need schema and tokenizers which are not accessible from QueryParser.\n    pub fn parse_query_with_parser(\n        &self,\n        query_parser: &QueryParser,\n    ) -> crate::Result<Box<dyn Query>> {\n        match &self.query {\n            FilterQuery::QueryString(query_str) => query_parser\n                .parse_query(query_str)\n                .map_err(|e| TantivyError::InvalidArgument(e.to_string())),\n            FilterQuery::CustomBuilder(_) => Err(TantivyError::InvalidArgument(\n                \"parse_query_with_parser is not supported for custom query builders. Use \\\n                 parse_query with explicit schema and tokenizers instead.\"\n                    .to_string(),\n            )),\n        }\n    }\n\n    /// Get the fast field names used by this aggregation (none for filter aggregation)\n    pub fn get_fast_field_names(&self) -> Vec<&str> {\n        // Filter aggregation cannot introspect query fast field dependencies.\n        //\n        // As of PR #2693, queries can fall back to fast fields when fields are not indexed\n        // (e.g., TermQuery falls back to RangeQuery on fast fields). However, the Query\n        // trait has no mechanism to report these dependencies.\n        //\n        // For prefetching optimization, callers must analyze the query themselves to\n        // determine fast field usage. This requires:\n        // 1. Parsing the query string to extract field references\n        // 2. Checking the schema to see if those fields are indexed or fast-only\n        // 3. Collecting fast field names for non-indexed fields\n        //\n        // This limitation exists because:\n        // - Query::weight() is called during execution, not during planning\n        // - The fallback decision is based on schema configuration\n        // - There's no Query trait method to declare potential fast field dependencies\n        vec![]\n    }\n}\n\n// Custom serialization implementation\nimpl Serialize for FilterAggregation {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: Serializer {\n        match &self.query {\n            FilterQuery::QueryString(query_string) => {\n                // Serialize query strings as plain strings\n                query_string.serialize(serializer)\n            }\n            FilterQuery::CustomBuilder(builder) => {\n                // Serialize custom builders using typetag (includes type information)\n                builder.serialize(serializer)\n            }\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for FilterAggregation {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: Deserializer<'de> {\n        // We need to peek at the value to determine if it's a string or an object\n        use serde::de::Error;\n        use serde_json::Value;\n\n        let value = Value::deserialize(deserializer)?;\n\n        let query = if let Some(query_string) = value.as_str() {\n            // It's a plain string - query string\n            FilterQuery::QueryString(query_string.to_string())\n        } else {\n            // It's an object - custom builder with typetag\n            let builder: Box<dyn QueryBuilder> = serde_json::from_value(value).map_err(|e| {\n                D::Error::custom(format!(\"Failed to deserialize QueryBuilder: {}\", e))\n            })?;\n            FilterQuery::CustomBuilder(builder)\n        };\n\n        Ok(FilterAggregation { query })\n    }\n}\n\n// PartialEq is required because AggregationVariants derives it\n// We implement it manually to handle custom builders which cannot be compared\nimpl PartialEq for FilterAggregation {\n    fn eq(&self, other: &Self) -> bool {\n        match (&self.query, &other.query) {\n            (FilterQuery::QueryString(a), FilterQuery::QueryString(b)) => a == b,\n            // Custom builders cannot be compared for equality\n            _ => false,\n        }\n    }\n}\n\n/// Request data for filter aggregation\n/// This struct holds the per-segment data needed to execute a filter aggregation\npub struct FilterAggReqData {\n    /// The name of the filter aggregation\n    pub name: String,\n    /// The filter aggregation\n    pub req: FilterAggregation,\n    /// The segment reader\n    pub segment_reader: SegmentReader,\n    /// Document evaluator for the filter query (precomputed BitSet)\n    /// This is built once when the request data is created\n    pub evaluator: DocumentQueryEvaluator,\n    /// Reusable buffer for matching documents to minimize allocations during collection\n    pub matching_docs_buffer: Vec<DocId>,\n    /// True if this filter aggregation is at the top level of the aggregation tree (not nested).\n    pub is_top_level: bool,\n}\n\nimpl FilterAggReqData {\n    pub(crate) fn get_memory_consumption(&self) -> usize {\n        // Estimate: name + segment reader reference + bitset + buffer capacity\n        self.name.len()\n        + std::mem::size_of::<SegmentReader>()\n        + self.evaluator.bitset.len() / 8 // BitSet memory (bits to bytes)\n        + self.matching_docs_buffer.capacity() * std::mem::size_of::<DocId>()\n        + std::mem::size_of::<bool>()\n    }\n}\n\n/// Document evaluator for filter queries using BitSet\npub struct DocumentQueryEvaluator {\n    /// BitSet containing all matching documents for this segment.\n    /// For AllQuery, this is a full BitSet (all bits set).\n    /// For other queries, only matching document bits are set.\n    pub(crate) bitset: BitSet,\n}\n\nimpl DocumentQueryEvaluator {\n    /// Create and initialize a document query evaluator for a segment\n    /// This executes the query upfront and collects results into a BitSet,\n    /// unless the query is AllQuery in which case we skip BitSet creation.\n    pub(crate) fn new(\n        query: Box<dyn Query>,\n        schema: Schema,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<Self> {\n        let max_doc = segment_reader.max_doc();\n\n        // Optimization: Detect AllQuery and create a full BitSet\n        if query.as_any().downcast_ref::<AllQuery>().is_some() {\n            return Ok(Self {\n                bitset: BitSet::with_max_value_and_full(max_doc),\n            });\n        }\n\n        // Get the weight for the query\n        let weight = query.weight(EnableScoring::disabled_from_schema(&schema))?;\n\n        // Get a scorer that iterates over matching documents\n        let mut scorer = weight.scorer(segment_reader, 1.0)?;\n\n        // Create a BitSet to hold all matching documents\n        let mut bitset = BitSet::with_max_value(max_doc);\n\n        // Collect all matching documents into the BitSet\n        // This is the upfront cost, but then lookups are O(1)\n        let mut doc = scorer.doc();\n        while doc != crate::TERMINATED {\n            bitset.insert(doc);\n            doc = scorer.advance();\n        }\n\n        Ok(Self { bitset })\n    }\n\n    /// Evaluate if a document matches the filter query\n    /// O(1) lookup in the precomputed BitSet\n    #[inline]\n    pub fn matches_document(&self, doc: DocId) -> bool {\n        self.bitset.contains(doc)\n    }\n\n    /// Filter a batch of documents\n    /// Returns matching documents from the input batch\n    #[inline]\n    pub fn filter_batch(&self, docs: &[DocId], output: &mut Vec<DocId>) {\n        for &doc in docs {\n            if self.bitset.contains(doc) {\n                output.push(doc);\n            }\n        }\n    }\n}\n\nimpl Debug for DocumentQueryEvaluator {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"DocumentQueryEvaluator\")\n            .field(\"num_matches\", &self.bitset.len())\n            .finish()\n    }\n}\n\n#[derive(Debug, Clone, PartialEq, Copy)]\nstruct DocCount {\n    doc_count: u64,\n    bucket_id: BucketId,\n}\n\n/// Segment collector for filter aggregation\npub struct SegmentFilterCollector<C: SubAggCache> {\n    /// Document counts per parent bucket\n    parent_buckets: Vec<DocCount>,\n    /// Sub-aggregation collectors\n    sub_aggregations: Option<CachedSubAggs<C>>,\n    bucket_id_provider: BucketIdProvider,\n    /// Accessor index for this filter aggregation (to access FilterAggReqData)\n    accessor_idx: usize,\n}\n\nimpl<C: SubAggCache> SegmentFilterCollector<C> {\n    /// Create a new filter segment collector following the new agg_data pattern\n    pub(crate) fn from_req_and_validate(\n        req: &mut AggregationsSegmentCtx,\n        node: &AggRefNode,\n    ) -> crate::Result<Self> {\n        // Build sub-aggregation collectors if any\n        let sub_agg_collector = if !node.children.is_empty() {\n            Some(build_segment_agg_collectors(req, &node.children)?)\n        } else {\n            None\n        };\n        let sub_agg_collector = sub_agg_collector.map(CachedSubAggs::new);\n\n        Ok(SegmentFilterCollector {\n            parent_buckets: Vec::new(),\n            sub_aggregations: sub_agg_collector,\n            accessor_idx: node.idx_in_req_data,\n            bucket_id_provider: BucketIdProvider::default(),\n        })\n    }\n}\n\npub(crate) fn build_segment_filter_collector(\n    req: &mut AggregationsSegmentCtx,\n    node: &AggRefNode,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    let is_top_level = req.per_request.filter_req_data[node.idx_in_req_data]\n        .as_ref()\n        .expect(\"filter_req_data slot is empty\")\n        .is_top_level;\n\n    if is_top_level {\n        Ok(Box::new(\n            SegmentFilterCollector::<LowCardSubAggCache>::from_req_and_validate(req, node)?,\n        ))\n    } else {\n        Ok(Box::new(\n            SegmentFilterCollector::<HighCardSubAggCache>::from_req_and_validate(req, node)?,\n        ))\n    }\n}\n\nimpl<C: SubAggCache> Debug for SegmentFilterCollector<C> {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"SegmentFilterCollector\")\n            .field(\"buckets\", &self.parent_buckets)\n            .field(\"has_sub_aggs\", &self.sub_aggregations.is_some())\n            .field(\"accessor_idx\", &self.accessor_idx)\n            .finish()\n    }\n}\n\nimpl<C: SubAggCache> SegmentAggregationCollector for SegmentFilterCollector<C> {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let mut sub_results = IntermediateAggregationResults::default();\n        let bucket_opt = self.parent_buckets.get(parent_bucket_id as usize);\n\n        if let Some(sub_aggs) = &mut self.sub_aggregations {\n            sub_aggs\n                .get_sub_agg_collector()\n                .add_intermediate_aggregation_result(\n                    agg_data,\n                    &mut sub_results,\n                    // Here we create a new bucket ID for sub-aggregations if the bucket doesn't\n                    // exist, so that sub-aggregations can still produce results (e.g., zero doc\n                    // count)\n                    bucket_opt\n                        .map(|bucket| bucket.bucket_id)\n                        .unwrap_or(self.bucket_id_provider.next_bucket_id()),\n                )?;\n        }\n\n        // Create the filter bucket result\n        let filter_bucket_result = IntermediateBucketResult::Filter {\n            doc_count: bucket_opt.map(|b| b.doc_count).unwrap_or(0),\n            sub_aggregations: sub_results,\n        };\n\n        // Get the name of this filter aggregation\n        let name = agg_data.per_request.filter_req_data[self.accessor_idx]\n            .as_ref()\n            .expect(\"filter_req_data slot is empty\")\n            .name\n            .clone();\n\n        results.push(\n            name,\n            IntermediateAggregationResult::Bucket(filter_bucket_result),\n        )?;\n\n        Ok(())\n    }\n\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        if docs.is_empty() {\n            return Ok(());\n        }\n\n        let mut bucket = self.parent_buckets[parent_bucket_id as usize];\n        // Take the request data to avoid borrow checker issues with sub-aggregations\n        let mut req = agg_data.take_filter_req_data(self.accessor_idx);\n\n        // Use batch filtering with O(1) BitSet lookups\n        req.matching_docs_buffer.clear();\n        req.evaluator\n            .filter_batch(docs, &mut req.matching_docs_buffer);\n\n        bucket.doc_count += req.matching_docs_buffer.len() as u64;\n\n        // Batch process sub-aggregations if we have matches\n        if !req.matching_docs_buffer.is_empty() {\n            if let Some(sub_aggs) = &mut self.sub_aggregations {\n                for &doc_id in &req.matching_docs_buffer {\n                    sub_aggs.push(bucket.bucket_id, doc_id);\n                }\n            }\n        }\n\n        // Put the request data back\n        agg_data.put_back_filter_req_data(self.accessor_idx, req);\n        if let Some(sub_aggs) = &mut self.sub_aggregations {\n            sub_aggs.check_flush_local(agg_data)?;\n        }\n        // put back bucket\n        self.parent_buckets[parent_bucket_id as usize] = bucket;\n\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(ref mut sub_aggs) = self.sub_aggregations {\n            sub_aggs.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.parent_buckets.len() <= max_bucket as usize {\n            let bucket_id = self.bucket_id_provider.next_bucket_id();\n            self.parent_buckets.push(DocCount {\n                doc_count: 0,\n                bucket_id,\n            });\n        }\n        Ok(())\n    }\n}\n\n/// Intermediate result for filter aggregation\n#[derive(Debug, Clone, PartialEq)]\npub struct IntermediateFilterBucketResult {\n    /// Document count in this bucket\n    pub doc_count: u64,\n    /// Sub-aggregation results\n    pub sub_aggregations: IntermediateAggregationResults,\n}\n\n#[cfg(test)]\nmod tests {\n    use serde_json::{json, Value};\n\n    use super::*;\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::{AggContextParams, AggregationCollector};\n    use crate::query::{AllQuery, TermQuery};\n    use crate::schema::{IndexRecordOption, Schema, Term, FAST, INDEXED, TEXT};\n    use crate::{doc, Index, IndexWriter};\n\n    // Test helper functions\n    fn aggregation_results_to_json(results: &AggregationResults) -> Value {\n        serde_json::to_value(results).expect(\"Failed to serialize aggregation results\")\n    }\n\n    fn json_values_match(actual: &Value, expected: &Value, tolerance: f64) -> bool {\n        match (actual, expected) {\n            (Value::Number(a), Value::Number(e)) => {\n                let a_f64 = a.as_f64().unwrap_or(0.0);\n                let e_f64 = e.as_f64().unwrap_or(0.0);\n                (a_f64 - e_f64).abs() < tolerance\n            }\n            (Value::Object(a_map), Value::Object(e_map)) => {\n                if a_map.len() != e_map.len() {\n                    return false;\n                }\n                for (key, expected_val) in e_map {\n                    match a_map.get(key) {\n                        Some(actual_val) => {\n                            if !json_values_match(actual_val, expected_val, tolerance) {\n                                return false;\n                            }\n                        }\n                        None => return false,\n                    }\n                }\n                true\n            }\n            (Value::Array(a_arr), Value::Array(e_arr)) => {\n                if a_arr.len() != e_arr.len() {\n                    return false;\n                }\n                for (actual_item, expected_item) in a_arr.iter().zip(e_arr.iter()) {\n                    if !json_values_match(actual_item, expected_item, tolerance) {\n                        return false;\n                    }\n                }\n                true\n            }\n            _ => actual == expected,\n        }\n    }\n\n    fn assert_aggregation_results_match(\n        actual_results: &AggregationResults,\n        expected_json: Value,\n        tolerance: f64,\n    ) {\n        let actual_json = aggregation_results_to_json(actual_results);\n\n        if !json_values_match(&actual_json, &expected_json, tolerance) {\n            panic!(\n                \"Aggregation results do not match expected JSON.\\nActual:\\n{}\\nExpected:\\n{}\",\n                serde_json::to_string_pretty(&actual_json).unwrap(),\n                serde_json::to_string_pretty(&expected_json).unwrap()\n            );\n        }\n    }\n\n    macro_rules! assert_agg_results {\n        ($actual:expr, $expected:expr) => {\n            assert_aggregation_results_match($actual, $expected, 0.1)\n        };\n        ($actual:expr, $expected:expr, $tolerance:expr) => {\n            assert_aggregation_results_match($actual, $expected, $tolerance)\n        };\n    }\n\n    fn create_standard_test_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let category = schema_builder.add_text_field(\"category\", TEXT | FAST);\n        let brand = schema_builder.add_text_field(\"brand\", TEXT | FAST);\n        let price = schema_builder.add_u64_field(\"price\", FAST | INDEXED);\n        let rating = schema_builder.add_f64_field(\"rating\", FAST);\n        let in_stock = schema_builder.add_bool_field(\"in_stock\", FAST | INDEXED);\n\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter = index.writer_for_tests()?;\n\n        writer.add_document(doc!(\n            category => \"electronics\", brand => \"apple\",\n            price => 999u64, rating => 4.5f64, in_stock => true\n        ))?;\n        writer.commit()?;\n        writer.add_document(doc!(\n            category => \"electronics\", brand => \"samsung\",\n            price => 799u64, rating => 4.2f64, in_stock => true\n        ))?;\n        writer.add_document(doc!(\n            category => \"clothing\", brand => \"nike\",\n            price => 120u64, rating => 4.1f64, in_stock => false\n        ))?;\n        writer.add_document(doc!(\n            category => \"books\", brand => \"penguin\",\n            price => 25u64, rating => 4.8f64, in_stock => true\n        ))?;\n\n        writer.commit()?;\n        Ok(index)\n    }\n\n    /// Helper to create aggregation collector with serialization roundtrip\n    /// This ensures all aggregations can be serialized and deserialized correctly\n    fn create_collector(\n        index: &Index,\n        aggregations: Aggregations,\n    ) -> crate::Result<AggregationCollector> {\n        // Serialize and deserialize the aggregations\n        let serialized = serde_json::to_string(&aggregations)?;\n        let deserialized: Aggregations = serde_json::from_str(&serialized)?;\n\n        // Create collector with deserialized aggregations\n        Ok(AggregationCollector::from_aggs(\n            deserialized,\n            AggContextParams::new(Default::default(), index.tokenizers().clone()),\n        ))\n    }\n\n    #[test]\n    fn test_basic_filter_with_metric_agg() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": {\n                    \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,\n                \"avg_price\": { \"value\": 899.0 }  // (999 + 799) / 2\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_filter_with_no_matches() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"furniture\": {\n                \"filter\": \"category:furniture\",\n                \"aggs\": {\n                    \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"furniture\": {\n                \"doc_count\": 0,\n                \"avg_price\": { \"value\": null }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_multiple_independent_filters() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": { \"avg_price\": { \"avg\": { \"field\": \"price\" } } }\n            },\n            \"in_stock\": {\n                \"filter\": \"in_stock:true\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            },\n            \"high_rated\": {\n                \"filter\": \"rating:[4.5 TO *]\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,\n                \"avg_price\": { \"value\": 899.0 }\n            },\n            \"in_stock\": {\n                \"doc_count\": 3,  // apple, samsung, penguin\n                \"count\": { \"value\": 3.0 }\n            },\n            \"high_rated\": {\n                \"doc_count\": 2,  // apple (4.5), penguin (4.8)\n                \"count\": { \"value\": 2.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    // ============================================================================\n    // Query Type Tests\n    // ============================================================================\n\n    #[test]\n    fn test_term_query_filter() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"apple_products\": {\n                \"filter\": \"brand:apple\",\n                \"aggs\": { \"max_price\": { \"max\": { \"field\": \"price\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"apple_products\": {\n                \"doc_count\": 1,\n                \"max_price\": { \"value\": 999.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_range_query_filter() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"mid_price\": {\n                \"filter\": \"price:[100 TO 900]\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"mid_price\": {\n                \"doc_count\": 2,  // samsung (799), nike (120)\n                \"count\": { \"value\": 2.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_boolean_query_filter() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 2);\n        let agg = json!({\n            \"premium_electronics\": {\n                \"filter\": \"category:electronics AND price:[800 TO *]\",\n                \"aggs\": { \"avg_rating\": { \"avg\": { \"field\": \"rating\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"premium_electronics\": {\n                \"doc_count\": 1,  // Only apple (999) is >= 800 in tantivy's range semantics\n                \"avg_rating\": { \"value\": 4.5 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_bool_field_filter() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"in_stock\": {\n                \"filter\": \"in_stock:true\",\n                \"aggs\": { \"avg_price\": { \"avg\": { \"field\": \"price\" } } }\n            },\n            \"out_of_stock\": {\n                \"filter\": \"in_stock:false\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"in_stock\": {\n                \"doc_count\": 3,  // apple, samsung, penguin\n                \"avg_price\": { \"value\": 607.67 }  // (999 + 799 + 25) / 3 ≈ 607.67\n            },\n            \"out_of_stock\": {\n                \"doc_count\": 1,  // nike\n                \"count\": { \"value\": 1.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected, 1.0);\n        Ok(())\n    }\n\n    // ============================================================================\n    // Nested Filter Tests\n    // ============================================================================\n\n    #[test]\n    fn test_two_level_nested_filters() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"all\": {\n                \"filter\": \"*\",\n                \"aggs\": {\n                    \"electronics\": {\n                        \"filter\": \"category:electronics\",\n                        \"aggs\": {\n                            \"expensive\": {\n                                \"filter\": \"price:[900 TO *]\",\n                                \"aggs\": {\n                                    \"count\": { \"value_count\": { \"field\": \"brand\" } }\n                                }\n                            }\n                        }\n                    }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"all\": {\n                \"doc_count\": 4,\n                \"electronics\": {\n                    \"doc_count\": 2,\n                    \"expensive\": {\n                        \"doc_count\": 1,  // Only apple (999) is >= 900\n                        \"count\": { \"value\": 1.0 }\n                    }\n                }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_deeply_nested_filters() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"level1\": {\n                \"filter\": \"*\",\n                \"aggs\": {\n                    \"level2\": {\n                        \"filter\": \"in_stock:true\",\n                        \"aggs\": {\n                            \"level3\": {\n                                \"filter\": \"rating:[4.0 TO *]\",\n                                \"aggs\": {\n                                    \"level4\": {\n                                        \"filter\": \"price:[500 TO *]\",\n                                        \"aggs\": {\n                                            \"final_count\": { \"value_count\": { \"field\": \"brand\" } }\n                                        }\n                                    }\n                                }\n                            }\n                        }\n                    }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"level1\": {\n                \"doc_count\": 4,\n                \"level2\": {\n                    \"doc_count\": 3,  // in_stock: apple, samsung, penguin\n                    \"level3\": {\n                        \"doc_count\": 3,  // all have rating >= 4.0\n                        \"level4\": {\n                            \"doc_count\": 2,  // apple (999), samsung (799)\n                            \"final_count\": { \"value\": 2.0 }\n                        }\n                    }\n                }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_multiple_nested_branches() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"root\": {\n                \"filter\": \"*\",\n                \"aggs\": {\n                    \"electronics_branch\": {\n                        \"filter\": \"category:electronics\",\n                        \"aggs\": {\n                            \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n                        }\n                    },\n                    \"in_stock_branch\": {\n                        \"filter\": \"in_stock:true\",\n                        \"aggs\": {\n                            \"count\": { \"value_count\": { \"field\": \"brand\" } }\n                        }\n                    }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"root\": {\n                \"doc_count\": 4,\n                \"electronics_branch\": {\n                    \"doc_count\": 2,\n                    \"avg_price\": { \"value\": 899.0 }\n                },\n                \"in_stock_branch\": {\n                    \"doc_count\": 3,\n                    \"count\": { \"value\": 3.0 }\n                }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_nested_filters_with_multiple_siblings_at_each_level() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Test complex nesting: multiple branches at each level\n        let agg = json!({\n            \"all\": {\n                \"filter\": \"*\",\n                \"aggs\": {\n                    // Level 2: Two independent filters\n                    \"expensive\": {\n                        \"filter\": \"price:[500 TO *]\",\n                        \"aggs\": {\n                            // Level 3: Multiple branches under \"expensive\"\n                            \"electronics\": {\n                                \"filter\": \"category:electronics\",\n                                \"aggs\": {\n                                    \"avg_rating\": { \"avg\": { \"field\": \"rating\" } }\n                                }\n                            },\n                            \"in_stock\": {\n                                \"filter\": \"in_stock:true\",\n                                \"aggs\": {\n                                    \"count\": { \"value_count\": { \"field\": \"brand\" } }\n                                }\n                            }\n                        }\n                    },\n                    \"affordable\": {\n                        \"filter\": \"price:[0 TO 200]\",\n                        \"aggs\": {\n                            // Level 3: Multiple branches under \"affordable\"\n                            \"books\": {\n                                \"filter\": \"category:books\",\n                                \"aggs\": {\n                                    \"max_rating\": { \"max\": { \"field\": \"rating\" } }\n                                }\n                            },\n                            \"clothing\": {\n                                \"filter\": \"category:clothing\",\n                                \"aggs\": {\n                                    \"min_price\": { \"min\": { \"field\": \"price\" } }\n                                }\n                            }\n                        }\n                    }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"all\": {\n                \"doc_count\": 4,\n                \"expensive\": {\n                    \"doc_count\": 2,  // apple (999), samsung (799)\n                    \"electronics\": {\n                        \"doc_count\": 2,  // both are electronics\n                        \"avg_rating\": { \"value\": 4.35 }  // (4.5 + 4.2) / 2\n                    },\n                    \"in_stock\": {\n                        \"doc_count\": 2,  // both are in stock\n                        \"count\": { \"value\": 2.0 }\n                    }\n                },\n                \"affordable\": {\n                    \"doc_count\": 2,  // nike (120), penguin (25)\n                    \"books\": {\n                        \"doc_count\": 1,  // penguin (25)\n                        \"max_rating\": { \"value\": 4.8 }\n                    },\n                    \"clothing\": {\n                        \"doc_count\": 1,  // nike (120)\n                        \"min_price\": { \"value\": 120.0 }\n                    }\n                }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    // ============================================================================\n    // Sub-Aggregation Combination Tests\n    // ============================================================================\n\n    #[test]\n    fn test_filter_with_terms_sub_agg() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": {\n                    \"brands\": {\n                        \"terms\": { \"field\": \"brand\" },\n                        \"aggs\": {\n                            \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n                        }\n                    }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        // Verify the structure exists and has expected doc_count\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,\n                \"brands\": {\n                    \"buckets\": [\n                        {\n                            \"key\": \"samsung\",\n                            \"doc_count\": 1,\n                            \"avg_price\": { \"value\": 799.0 }\n                        },\n                        {\n                            \"key\": \"apple\",\n                            \"doc_count\": 1,\n                            \"avg_price\": { \"value\": 999.0 }\n                        }\n                    ],\n                    \"sum_other_doc_count\": 0,\n                    \"doc_count_error_upper_bound\": 0\n                }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_filter_with_multiple_metric_aggs() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": {\n                    \"price_stats\": { \"stats\": { \"field\": \"price\" } },\n                    \"rating_avg\": { \"avg\": { \"field\": \"rating\" } },\n                    \"count\": { \"value_count\": { \"field\": \"brand\" } }\n                }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,\n                \"price_stats\": {\n                    \"count\": 2,\n                    \"min\": 799.0,\n                    \"max\": 999.0,\n                    \"sum\": 1798.0,\n                    \"avg\": 899.0\n                },\n                \"rating_avg\": { \"value\": 4.35 },\n                \"count\": { \"value\": 2.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    // ============================================================================\n    // Edge Cases and Error Handling\n    // ============================================================================\n\n    #[test]\n    fn test_filter_on_empty_index() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let _category = schema_builder.add_text_field(\"category\", TEXT | FAST);\n        let _price = schema_builder.add_u64_field(\"price\", FAST);\n\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter = index.writer(50_000_000)?;\n        writer.commit()?; // Commit empty index\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": { \"avg_price\": { \"avg\": { \"field\": \"price\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 0,\n                \"avg_price\": { \"value\": null }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    #[test]\n    fn test_malformed_query_string() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Empty query string\n        let agg = json!({\n            \"test\": {\n                \"filter\": \"\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let result = serde_json::from_value::<Aggregations>(agg)\n            .map_err(|e| crate::TantivyError::InvalidArgument(e.to_string()))\n            .and_then(|aggregations| {\n                let collector = create_collector(&index, aggregations)?;\n                searcher.search(&AllQuery, &collector)\n            });\n\n        // Empty string should either work (matching nothing) or error gracefully\n        assert!(result.is_ok() || result.is_err());\n        Ok(())\n    }\n\n    #[test]\n    fn test_filter_with_base_query() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let schema = index.schema();\n\n        // Use a base query to pre-filter to in_stock items only\n        let in_stock_field = schema.get_field(\"in_stock\").unwrap();\n        let base_query = TermQuery::new(\n            Term::from_field_bool(in_stock_field, true),\n            IndexRecordOption::Basic,\n        );\n\n        let agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&base_query, &collector)?;\n\n        let expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,  // Both in-stock electronics\n                \"count\": { \"value\": 2.0 }\n            }\n        });\n\n        assert_agg_results!(&result, expected);\n        Ok(())\n    }\n\n    // ============================================================================\n    // Custom Query Integration Tests\n    // ============================================================================\n\n    #[test]\n    fn test_custom_query_builder() -> crate::Result<()> {\n        // Define a query builder with full serde support\n        #[derive(Debug, Clone, Serialize, Deserialize)]\n        struct TestTermQueryBuilder {\n            field_name: String,\n            term_text: String,\n        }\n\n        #[typetag::serde(name = \"TestTermQueryBuilder\")]\n        impl QueryBuilder for TestTermQueryBuilder {\n            fn build_query(\n                &self,\n                schema: &Schema,\n                _tokenizers: &TokenizerManager,\n            ) -> crate::Result<Box<dyn Query>> {\n                let field = schema.get_field(&self.field_name)?;\n                let term = Term::from_field_text(field, &self.term_text);\n                Ok(Box::new(TermQuery::new(term, IndexRecordOption::Basic)))\n            }\n\n            fn box_clone(&self) -> Box<dyn QueryBuilder> {\n                Box::new(self.clone())\n            }\n        }\n\n        let index = create_standard_test_index()?;\n\n        // Create a filter aggregation with a custom query builder\n        let builder = TestTermQueryBuilder {\n            field_name: \"category\".to_string(),\n            term_text: \"electronics\".to_string(),\n        };\n        let filter_agg = FilterAggregation::new_with_builder(Box::new(builder));\n\n        // Test that the query can be parsed\n        let schema = index.schema();\n        let tokenizers = index.tokenizers();\n        let query = filter_agg.parse_query(&schema, tokenizers)?;\n\n        // Verify the query was built correctly (it should be a TermQuery)\n        assert!(format!(\"{:?}\", query).contains(\"TermQuery\"));\n\n        // Test that it can be cloned\n        let cloned = filter_agg.clone();\n        let query2 = cloned.parse_query(&schema, tokenizers)?;\n        assert!(format!(\"{:?}\", query2).contains(\"TermQuery\"));\n\n        // Verify that custom builders CAN be serialized with typetag\n        let serialized = serde_json::to_string(&filter_agg)?;\n        assert!(\n            serialized.contains(\"TestTermQueryBuilder\"),\n            \"Serialized JSON should contain the type tag\"\n        );\n        assert!(\n            serialized.contains(\"electronics\"),\n            \"Serialized JSON should contain the field data\"\n        );\n\n        // Verify that it can be deserialized\n        let deserialized: FilterAggregation = serde_json::from_str(&serialized)?;\n        let query3 = deserialized.parse_query(&schema, tokenizers)?;\n        assert!(format!(\"{:?}\", query3).contains(\"TermQuery\"));\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_query_string_serialization() -> crate::Result<()> {\n        // Query strings should serialize/deserialize correctly\n        let filter_agg = FilterAggregation::new(\"category:electronics\".to_string());\n\n        let serialized = serde_json::to_string(&filter_agg)?;\n        assert!(serialized.contains(\"electronics\"));\n\n        let deserialized: FilterAggregation = serde_json::from_str(&serialized)?;\n        // Verify it deserializes correctly by using it in an aggregation\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg = json!({\n                \"test\": {\n                    \"filter\": deserialized,\n                    \"aggs\": { \"count\": { \"value_count\": { \"field\": \"brand\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let result = searcher.search(&AllQuery, &collector)?;\n\n        // Should match 2 electronics\n        let result_json = serde_json::to_value(&result)?;\n        assert_eq!(result_json[\"test\"][\"doc_count\"], 2);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_query_builder_serialization_roundtrip() -> crate::Result<()> {\n        // Define a serializable query builder\n        #[derive(Debug, Clone, Serialize, Deserialize)]\n        struct RoundtripTermQueryBuilder {\n            field_name: String,\n            term_text: String,\n        }\n\n        #[typetag::serde(name = \"RoundtripTermQueryBuilder\")]\n        impl QueryBuilder for RoundtripTermQueryBuilder {\n            fn build_query(\n                &self,\n                schema: &Schema,\n                _tokenizers: &TokenizerManager,\n            ) -> crate::Result<Box<dyn Query>> {\n                let field = schema.get_field(&self.field_name)?;\n                let term = Term::from_field_text(field, &self.term_text);\n                Ok(Box::new(TermQuery::new(term, IndexRecordOption::Basic)))\n            }\n\n            fn box_clone(&self) -> Box<dyn QueryBuilder> {\n                Box::new(self.clone())\n            }\n        }\n\n        let index = create_standard_test_index()?;\n\n        // Create a filter aggregation with a custom query builder\n        let builder = RoundtripTermQueryBuilder {\n            field_name: \"category\".to_string(),\n            term_text: \"electronics\".to_string(),\n        };\n        let filter_agg = FilterAggregation::new_with_builder(Box::new(builder));\n\n        // Serialize the filter aggregation\n        let serialized = serde_json::to_string(&filter_agg)?;\n\n        // Verify the serialized JSON contains the builder data and type tag\n        assert!(\n            serialized.contains(\"RoundtripTermQueryBuilder\"),\n            \"Serialized JSON should contain type tag\"\n        );\n        assert!(\n            serialized.contains(\"category\"),\n            \"Serialized JSON should contain field_name\"\n        );\n        assert!(\n            serialized.contains(\"electronics\"),\n            \"Serialized JSON should contain term_text\"\n        );\n\n        // Deserialize back\n        let deserialized: FilterAggregation = serde_json::from_str(&serialized)?;\n\n        // Verify the aggregation produces correct results\n        let agg = json!({\n            \"filtered\": {\n                \"filter\": deserialized\n            }\n        });\n\n        let agg_req: Aggregations = serde_json::from_value(agg)?;\n        let searcher = index.reader()?.searcher();\n        let collector = create_collector(&index, agg_req)?;\n        let agg_res = searcher.search(&AllQuery, &collector)?;\n\n        let result_json = serde_json::to_value(&agg_res)?;\n        assert_eq!(result_json[\"filtered\"][\"doc_count\"], 2);\n\n        Ok(())\n    }\n\n    // ============================================================================\n    // Correctness Validation Tests\n    // ============================================================================\n\n    #[test]\n    fn test_filter_result_correctness_vs_separate_query() -> crate::Result<()> {\n        let index = create_standard_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let schema = index.schema();\n\n        // Method 1: Filter aggregation\n        let filter_agg = json!({\n            \"electronics\": {\n                \"filter\": \"category:electronics\",\n                \"aggs\": { \"avg_price\": { \"avg\": { \"field\": \"price\" } } }\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(filter_agg)?;\n        let collector = create_collector(&index, aggregations)?;\n        let filter_result = searcher.search(&AllQuery, &collector)?;\n\n        // Method 2: Separate query\n        let category_field = schema.get_field(\"category\").unwrap();\n        let term = Term::from_field_text(category_field, \"electronics\");\n        let term_query = TermQuery::new(term, IndexRecordOption::Basic);\n\n        let separate_agg = json!({\n            \"result\": { \"avg\": { \"field\": \"price\" } }\n        });\n\n        let separate_aggregations: Aggregations = serde_json::from_value(separate_agg)?;\n        let separate_collector =\n            AggregationCollector::from_aggs(separate_aggregations, Default::default());\n        let separate_result = searcher.search(&term_query, &separate_collector)?;\n\n        // Both methods should produce identical results\n        let filter_expected = json!({\n            \"electronics\": {\n                \"doc_count\": 2,\n                \"avg_price\": { \"value\": 899.0 }\n            }\n        });\n\n        let separate_expected = json!({\n            \"result\": {\n                \"value\": 899.0\n            }\n        });\n\n        // Verify filter aggregation result\n        assert_agg_results!(&filter_result, filter_expected);\n\n        // Verify separate query result matches\n        assert_agg_results!(&separate_result, separate_expected);\n\n        // This test demonstrates that filter aggregation produces the same results\n        // as running a separate query with the same condition\n        Ok(())\n    }\n\n    #[test]\n    fn test_custom_tokenizer_required() -> crate::Result<()> {\n        use crate::schema::{TextFieldIndexing, TextOptions};\n        use crate::tokenizer::{SimpleTokenizer, TextAnalyzer, TokenizerManager};\n\n        // Create a custom tokenizer that doesn't lowercase (just splits on whitespace)\n        let custom_tokenizer = TextAnalyzer::builder(SimpleTokenizer::default()).build();\n\n        // Register tokenizer\n        let tokenizers = TokenizerManager::default();\n        tokenizers.register(\"my_custom\", custom_tokenizer);\n\n        // Create a schema with a text field that uses our custom tokenizer\n        let mut schema_builder = Schema::builder();\n        let text_field_indexing = TextFieldIndexing::default()\n            .set_tokenizer(\"my_custom\")\n            .set_index_option(IndexRecordOption::Basic);\n        let text_options = TextOptions::default()\n            .set_indexing_options(text_field_indexing)\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n\n        // Build index with custom tokenizer\n        let index = crate::IndexBuilder::new()\n            .schema(schema.clone())\n            .tokenizers(tokenizers)\n            .create_in_ram()?;\n        let mut writer = index.writer(50_000_000)?;\n\n        // Add documents with UPPERCASE text\n        writer.add_document(doc!(text_field => \"HELLO\"))?;\n        writer.add_document(doc!(text_field => \"WORLD\"))?;\n        writer.add_document(doc!(text_field => \"hello\"))?; // lowercase version\n        writer.commit()?;\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Test: With correct tokenizer (from index) - should work\n        let agg = json!({\n            \"uppercase_hello\": {\n                \"filter\": \"text:HELLO\"\n            }\n        });\n\n        let aggregations: Aggregations = serde_json::from_value(agg)?;\n        let collector_with_tokenizer = create_collector(&index, aggregations.clone())?;\n        let result_with_tokenizer = searcher.search(&AllQuery, &collector_with_tokenizer)?;\n\n        // Should match only the UPPERCASE \"HELLO\" (1 document)\n        let result_json = serde_json::to_value(&result_with_tokenizer)?;\n        assert_eq!(\n            result_json[\"uppercase_hello\"][\"doc_count\"], 1,\n            \"With custom tokenizer from index, should match exactly 1 UPPERCASE document\"\n        );\n\n        // Test 2: With default tokenizer (wrong!) - should fail to parse the query\n        // because \"my_custom\" tokenizer is not in the default TokenizerManager\n        let collector_with_default = AggregationCollector::from_aggs(\n            aggregations,\n            AggContextParams::new(Default::default(), TokenizerManager::default()),\n        );\n        let result_with_default = searcher.search(&AllQuery, &collector_with_default);\n\n        // This should error because the tokenizer \"my_custom\" is not registered\n        assert!(\n            result_with_default.is_err(),\n            \"Without proper tokenizers, query parsing should fail\"\n        );\n        assert!(\n            result_with_default\n                .unwrap_err()\n                .to_string()\n                .contains(\"my_custom\"),\n            \"Error should mention the missing tokenizer\"\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/histogram/date_histogram.rs",
    "content": "use serde::{Deserialize, Serialize};\n\nuse super::{HistogramAggregation, HistogramBounds};\nuse crate::aggregation::*;\n\n/// DateHistogramAggregation is similar to `HistogramAggregation`, but it can only be used with date\n/// type.\n///\n/// Currently only **fixed time** intervals are supported. Calendar-aware time intervals are not\n/// supported.\n///\n/// Like the histogram, values are rounded down into the closest bucket.\n///\n/// For this calculation all fastfield values are converted to f64.\n///\n/// # Limitations/Compatibility\n/// Only fixed time intervals are supported.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"prices\": {\n///         \"date_histogram\": {\n///             \"field\": \"price\",\n///             \"fixed_interval\": \"30d\"\n///         }\n///     }\n/// }\n/// ```\n///\n/// Response\n/// See [`BucketEntry`](crate::aggregation::agg_result::BucketEntry)\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\npub struct DateHistogramAggregationReq {\n    #[doc(hidden)]\n    /// Only for validation\n    pub interval: Option<String>,\n    #[doc(hidden)]\n    /// Only for validation\n    pub calendar_interval: Option<String>,\n    /// The field to aggregate on.\n    pub field: String,\n    /// The format to format dates. Unsupported currently.\n    pub format: Option<String>,\n    /// The interval to chunk your data range. Each bucket spans a value range of\n    /// [0..fixed_interval). Accepted values\n    ///\n    /// Fixed intervals are configured with the `fixed_interval` parameter.\n    /// In contrast to calendar-aware intervals, fixed intervals are a fixed number of SI units and\n    /// never deviate, regardless of where they fall on the calendar. One second is always\n    /// composed of 1000ms. This allows fixed intervals to be specified in any multiple of the\n    /// supported units. However, it means fixed intervals cannot express other units such as\n    /// months, since the duration of a month is not a fixed quantity. Attempting to specify a\n    /// calendar interval like month or quarter will return an Error.\n    ///\n    /// The accepted units for fixed intervals are:\n    /// * `ms`: milliseconds\n    /// * `s`: seconds. Defined as 1000 milliseconds each.\n    /// * `m`: minutes. Defined as 60 seconds each (60_000 milliseconds).\n    /// * `h`: hours. Defined as 60 minutes each (3_600_000 milliseconds).\n    /// * `d`: days. Defined as 24 hours (86_400_000 milliseconds).\n    ///\n    /// Fractional time values are not supported, but you can address this by shifting to another\n    /// time unit (e.g., `1.5h` could instead be specified as `90m`).\n    ///\n    /// `Option` for validation, the parameter is not optional\n    pub fixed_interval: Option<String>,\n    /// Intervals implicitly defines an absolute grid of buckets `[interval * k, interval * (k +\n    /// 1))`.\n    ///\n    /// Offset makes it possible to shift this grid into\n    /// `[offset + interval * k, offset + interval * (k + 1))`. Offset has to be in the range [0,\n    /// interval).\n    ///\n    /// The `offset` parameter is has the same syntax as the `fixed_interval` parameter, but\n    /// also allows for negative values.\n    pub offset: Option<String>,\n    /// The minimum number of documents in a bucket to be returned. Defaults to 0.\n    pub min_doc_count: Option<u64>,\n    /// Limits the data range to `[min, max]` closed interval.\n    ///\n    /// This can be used to filter values if they are not in the data range.\n    ///\n    /// hard_bounds only limits the buckets, to force a range set both extended_bounds and\n    /// hard_bounds to the same range.\n    ///\n    /// Needs to be provided as timestamp in millisecond precision.\n    ///\n    /// ## Example\n    /// ```json\n    /// {\n    ///     \"sales_over_time\": {\n    ///        \"date_histogram\": {\n    ///            \"field\": \"dates\",\n    ///            \"interval\": \"1d\",\n    ///            \"hard_bounds\": {\n    ///                \"min\": 0,\n    ///                \"max\": 1420502400000\n    ///            }\n    ///        }\n    ///    }\n    /// }\n    /// ```\n    pub hard_bounds: Option<HistogramBounds>,\n    /// Can be set to extend your bounds. The range of the buckets is by default defined by the\n    /// data range of the values of the documents. As the name suggests, this can only be used to\n    /// extend the value range. If the bounds for min or max are not extending the range, the value\n    /// has no effect on the returned buckets.\n    ///\n    /// Cannot be set in conjunction with min_doc_count > 0, since the empty buckets from extended\n    /// bounds would not be returned.\n    pub extended_bounds: Option<HistogramBounds>,\n\n    /// Whether to return the buckets as a hash map\n    #[serde(default)]\n    pub keyed: bool,\n}\n\nimpl DateHistogramAggregationReq {\n    pub(crate) fn to_histogram_req(&self) -> crate::Result<HistogramAggregation> {\n        self.validate()?;\n        Ok(HistogramAggregation {\n            field: self.field.to_string(),\n            interval: parse_into_milliseconds(self.fixed_interval.as_ref().unwrap())? as f64,\n            offset: self\n                .offset\n                .as_ref()\n                .map(|offset| parse_offset_into_milliseconds(offset))\n                .transpose()?\n                .map(|el| el as f64),\n            min_doc_count: self.min_doc_count,\n            hard_bounds: self.hard_bounds,\n            extended_bounds: self.extended_bounds,\n            keyed: self.keyed,\n            is_normalized_to_ns: false,\n        })\n    }\n\n    fn validate(&self) -> crate::Result<()> {\n        if let Some(interval) = self.interval.as_ref() {\n            return Err(crate::TantivyError::InvalidArgument(format!(\n                \"`interval` parameter {interval:?} in date histogram is unsupported, only \\\n                 `fixed_interval` is supported\"\n            )));\n        }\n        if let Some(interval) = self.calendar_interval.as_ref() {\n            return Err(crate::TantivyError::InvalidArgument(format!(\n                \"`calendar_interval` parameter {interval:?} in date histogram is unsupported, \\\n                 only `fixed_interval` is supported\"\n            )));\n        }\n        if self.format.is_some() {\n            return Err(crate::TantivyError::InvalidArgument(\n                \"format parameter on date_histogram is unsupported\".to_string(),\n            ));\n        }\n\n        if self.fixed_interval.is_none() {\n            return Err(crate::TantivyError::InvalidArgument(\n                \"fixed_interval in date histogram is missing\".to_string(),\n            ));\n        }\n\n        parse_into_milliseconds(self.fixed_interval.as_ref().unwrap())?;\n\n        Ok(())\n    }\n}\n\n#[derive(Debug, Clone, PartialEq, Eq, Error)]\n/// Errors when parsing the fixed interval for `DateHistogramAggregationReq`.\npub enum DateHistogramParseError {\n    /// Unit not recognized in passed String\n    #[error(\"Unit not recognized in passed String {0:?}\")]\n    UnitNotRecognized(String),\n    /// Number not found in passed String\n    #[error(\"Number not found in passed String {0:?}\")]\n    NumberMissing(String),\n    /// Unit not found in passed String\n    #[error(\"Unit not found in passed String {0:?}\")]\n    UnitMissing(String),\n    /// Offset invalid\n    #[error(\"passed offset is invalid {0:?}\")]\n    InvalidOffset(String),\n    /// Value out of bounds\n    #[error(\"passed value is out of bounds: {0:?}\")]\n    OutOfBounds(String),\n}\n\nfn parse_offset_into_milliseconds(input: &str) -> Result<i64, AggregationError> {\n    let is_sign = |byte| &[byte] == b\"-\" || &[byte] == b\"+\";\n    if input.is_empty() {\n        return Err(DateHistogramParseError::InvalidOffset(input.to_string()).into());\n    }\n\n    let has_sign = is_sign(input.as_bytes()[0]);\n    if has_sign {\n        let (sign, input) = input.split_at(1);\n        let val = parse_into_milliseconds(input)?;\n        if sign == \"-\" {\n            Ok(-val)\n        } else {\n            Ok(val)\n        }\n    } else {\n        parse_into_milliseconds(input)\n    }\n}\n\npub(crate) fn parse_into_milliseconds(input: &str) -> Result<i64, AggregationError> {\n    let split_boundary = input\n        .as_bytes()\n        .iter()\n        .take_while(|byte| byte.is_ascii_digit())\n        .count();\n    let (number, unit) = input.split_at(split_boundary);\n    if number.is_empty() {\n        return Err(DateHistogramParseError::NumberMissing(input.to_string()).into());\n    }\n    if unit.is_empty() {\n        return Err(DateHistogramParseError::UnitMissing(input.to_string()).into());\n    }\n    let number: i64 = number\n        .parse()\n        // Technically this should never happen, but there was a bug\n        // here and being defensive does not hurt.\n        .map_err(|_err| DateHistogramParseError::NumberMissing(input.to_string()))?;\n\n    let unit_in_ms = match unit {\n        \"ms\" | \"milliseconds\" => 1,\n        \"s\" | \"seconds\" => 1000,\n        \"m\" | \"minutes\" => 60 * 1000,\n        \"h\" | \"hours\" => 60 * 60 * 1000,\n        \"d\" | \"days\" => 24 * 60 * 60 * 1000,\n        _ => return Err(DateHistogramParseError::UnitNotRecognized(unit.to_string()).into()),\n    };\n\n    let val = number * unit_in_ms;\n    // The field type is in nanoseconds precision, so validate the value to fit the range\n    val.checked_mul(1_000_000)\n        .ok_or_else(|| DateHistogramParseError::OutOfBounds(input.to_string()))?;\n\n    Ok(val)\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use pretty_assertions::assert_eq;\n\n    use super::*;\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::exec_request;\n    use crate::indexer::NoMergePolicy;\n    use crate::schema::{Schema, FAST, STRING};\n    use crate::{Index, IndexWriter, TantivyDocument};\n\n    #[test]\n    fn test_parse_into_millisecs() {\n        assert_eq!(parse_into_milliseconds(\"1m\").unwrap(), 60_000);\n        assert_eq!(parse_into_milliseconds(\"2m\").unwrap(), 120_000);\n        assert_eq!(parse_into_milliseconds(\"2minutes\").unwrap(), 120_000);\n        assert_eq!(\n            parse_into_milliseconds(\"2y\").unwrap_err(),\n            DateHistogramParseError::UnitNotRecognized(\"y\".to_string()).into()\n        );\n        assert_eq!(\n            parse_into_milliseconds(\"2000\").unwrap_err(),\n            DateHistogramParseError::UnitMissing(\"2000\".to_string()).into()\n        );\n        assert_eq!(\n            parse_into_milliseconds(\"ms\").unwrap_err(),\n            DateHistogramParseError::NumberMissing(\"ms\".to_string()).into()\n        );\n    }\n\n    #[test]\n    fn test_parse_offset_into_milliseconds() {\n        assert_eq!(parse_offset_into_milliseconds(\"1m\").unwrap(), 60_000);\n        assert_eq!(parse_offset_into_milliseconds(\"+1m\").unwrap(), 60_000);\n        assert_eq!(parse_offset_into_milliseconds(\"-1m\").unwrap(), -60_000);\n        assert_eq!(parse_offset_into_milliseconds(\"2m\").unwrap(), 120_000);\n        assert_eq!(parse_offset_into_milliseconds(\"+2m\").unwrap(), 120_000);\n        assert_eq!(parse_offset_into_milliseconds(\"-2m\").unwrap(), -120_000);\n        assert_eq!(parse_offset_into_milliseconds(\"-2ms\").unwrap(), -2);\n        assert_eq!(\n            parse_offset_into_milliseconds(\"2y\").unwrap_err(),\n            DateHistogramParseError::UnitNotRecognized(\"y\".to_string()).into()\n        );\n        assert_eq!(\n            parse_offset_into_milliseconds(\"2000\").unwrap_err(),\n            DateHistogramParseError::UnitMissing(\"2000\".to_string()).into()\n        );\n        assert_eq!(\n            parse_offset_into_milliseconds(\"ms\").unwrap_err(),\n            DateHistogramParseError::NumberMissing(\"ms\".to_string()).into()\n        );\n    }\n\n    #[test]\n    fn test_parse_into_milliseconds_do_not_accept_non_ascii() {\n        assert!(parse_into_milliseconds(\"１m\").is_err());\n    }\n\n    pub fn get_test_index_from_docs(\n        merge_segments: bool,\n        segment_and_docs: &[Vec<&str>],\n    ) -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_date_field(\"date\", FAST);\n        schema_builder.add_json_field(\"mixed\", FAST);\n        schema_builder.add_text_field(\"text\", FAST | STRING);\n        schema_builder.add_text_field(\"text2\", FAST | STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 30_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            for values in segment_and_docs {\n                for doc_str in values {\n                    let doc = TantivyDocument::parse_json(&schema, doc_str)?;\n                    index_writer.add_document(doc)?;\n                }\n                // writing the segment\n                index_writer.commit()?;\n            }\n        }\n        if merge_segments {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            if segment_ids.len() > 1 {\n                let mut index_writer: IndexWriter = index.writer_for_tests()?;\n                index_writer.merge(&segment_ids).wait()?;\n                index_writer.wait_merging_threads()?;\n            }\n        }\n\n        Ok(index)\n    }\n\n    #[test]\n    fn histogram_test_date_force_merge_segments() {\n        histogram_test_date_merge_segments(true)\n    }\n\n    #[test]\n    fn histogram_test_date() {\n        histogram_test_date_merge_segments(false)\n    }\n\n    fn histogram_test_date_merge_segments(merge_segments: bool) {\n        let docs = vec![\n            vec![r#\"{ \"date\": \"2015-01-01T12:10:30Z\", \"text\": \"aaa\" }\"#],\n            vec![r#\"{ \"date\": \"2015-01-01T11:11:30Z\", \"text\": \"bbb\" }\"#],\n            vec![r#\"{ \"date\": \"2015-01-01T11:11:30Z\", \"text\": \"bbb\" }\"#],\n            vec![r#\"{ \"date\": \"2015-01-02T00:00:00Z\", \"text\": \"bbb\" }\"#],\n            vec![r#\"{ \"date\": \"2015-01-06T00:00:00Z\", \"text\": \"ccc\" }\"#],\n            vec![r#\"{ \"date\": \"2015-01-06T00:00:00Z\", \"text\": \"ccc\" }\"#],\n        ];\n        let index = get_test_index_from_docs(merge_segments, &docs).unwrap();\n\n        {\n            // 30day + offset\n            let elasticsearch_compatible_json = json!(\n                {\n                \"sales_over_time\": {\n                    \"date_histogram\": {\n                    \"field\": \"date\",\n                    \"fixed_interval\": \"30d\",\n                    \"offset\": \"-4d\"\n                    }\n                }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!({\n                \"sales_over_time\" : {\n                    \"buckets\" : [\n                        {\n                            \"key_as_string\" : \"2015-01-01T00:00:00Z\",\n                            \"key\" : 1420070400000.0,\n                            \"doc_count\" : 6\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n\n        {\n            // 30day + offset + sub_agg\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"sales_over_time\": {\n                        \"date_histogram\": {\n                        \"field\": \"date\",\n                        \"fixed_interval\": \"30d\",\n                        \"offset\": \"-4d\"\n                        },\n                        \"aggs\": {\n                            \"texts\": {\n                                \"terms\": {\"field\": \"text\"}\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!({\n                \"sales_over_time\" : {\n                \"buckets\" : [\n                    {\n                        \"key_as_string\" : \"2015-01-01T00:00:00Z\",\n                        \"key\" : 1420070400000.0,\n                        \"doc_count\" : 6,\n                        \"texts\": {\n                            \"buckets\": [\n                                {\n                                \"doc_count\": 3,\n                                \"key\": \"bbb\"\n                                },\n                                {\n                                \"doc_count\": 2,\n                                \"key\": \"ccc\"\n                                },\n                                {\n                                \"doc_count\": 1,\n                                \"key\": \"aaa\"\n                                }\n                            ],\n                            \"doc_count_error_upper_bound\": 0,\n                            \"sum_other_doc_count\": 0\n                            }\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n        {\n            // 1day\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"sales_over_time\": {\n                        \"date_histogram\": {\n                            \"field\": \"date\",\n                            \"fixed_interval\": \"1d\"\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!( {\n                \"sales_over_time\": {\n                    \"buckets\": [\n                        {\n                            \"doc_count\": 3,\n                            \"key\": 1420070400000.0,\n                            \"key_as_string\": \"2015-01-01T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 1,\n                            \"key\": 1420156800000.0,\n                            \"key_as_string\": \"2015-01-02T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420243200000.0,\n                            \"key_as_string\": \"2015-01-03T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420329600000.0,\n                            \"key_as_string\": \"2015-01-04T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420416000000.0,\n                            \"key_as_string\": \"2015-01-05T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 2,\n                            \"key\": 1420502400000.0,\n                            \"key_as_string\": \"2015-01-06T00:00:00Z\"\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n\n        {\n            // 1day + extended_bounds\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"sales_over_time\": {\n                        \"date_histogram\": {\n                            \"field\": \"date\",\n                            \"fixed_interval\": \"1d\",\n                            \"extended_bounds\": {\n                                \"min\": 1419984000000.0,\n                                \"max\": 1420588800000.0\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!({\n                \"sales_over_time\" : {\n                    \"buckets\": [\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1419984000000.0,\n                            \"key_as_string\": \"2014-12-31T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 3,\n                            \"key\": 1420070400000.0,\n                            \"key_as_string\": \"2015-01-01T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 1,\n                            \"key\": 1420156800000.0,\n                            \"key_as_string\": \"2015-01-02T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420243200000.0,\n                            \"key_as_string\": \"2015-01-03T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420329600000.0,\n                            \"key_as_string\": \"2015-01-04T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420416000000.0,\n                            \"key_as_string\": \"2015-01-05T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 2,\n                            \"key\": 1420502400000.0,\n                            \"key_as_string\": \"2015-01-06T00:00:00Z\"\n                        },\n                        {\n                            \"doc_count\": 0,\n                            \"key\": 1420588800000.0,\n                            \"key_as_string\": \"2015-01-07T00:00:00Z\"\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n        {\n            // 1day + hard_bounds + extended_bounds\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"sales_over_time\": {\n                        \"date_histogram\": {\n                            \"field\": \"date\",\n                            \"fixed_interval\": \"1d\",\n                            \"hard_bounds\": {\n                                \"min\": 1420156800000.0,\n                                \"max\": 1420243200000.0\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!({\n                \"sales_over_time\" : {\n                    \"buckets\": [\n                        {\n                            \"doc_count\": 1,\n                            \"key\": 1420156800000.0,\n                            \"key_as_string\": \"2015-01-02T00:00:00Z\"\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n\n        {\n            // 1day + hard_bounds as Rfc3339\n            let elasticsearch_compatible_json = json!(\n                {\n                    \"sales_over_time\": {\n                        \"date_histogram\": {\n                            \"field\": \"date\",\n                            \"fixed_interval\": \"1d\",\n                            \"hard_bounds\": {\n                                \"min\": \"2015-01-02T00:00:00Z\",\n                                \"max\": \"2015-01-02T12:00:00Z\"\n                            }\n                        }\n                    }\n                }\n            );\n\n            let agg_req: Aggregations = serde_json::from_str(\n                &serde_json::to_string(&elasticsearch_compatible_json).unwrap(),\n            )\n            .unwrap();\n            let res = exec_request(agg_req, &index).unwrap();\n            let expected_res = json!({\n                \"sales_over_time\" : {\n                    \"buckets\": [\n                        {\n                            \"doc_count\": 1,\n                            \"key\": 1420156800000.0,\n                            \"key_as_string\": \"2015-01-02T00:00:00Z\"\n                        }\n                    ]\n                }\n            });\n            assert_eq!(res, expected_res);\n        }\n    }\n    #[test]\n    fn histogram_test_invalid_req() {\n        let docs = vec![];\n\n        let index = get_test_index_from_docs(false, &docs).unwrap();\n        let elasticsearch_compatible_json = json!(\n            {\n              \"sales_over_time\": {\n                \"date_histogram\": {\n                  \"field\": \"date\",\n                  \"interval\": \"30d\",\n                  \"offset\": \"-4d\"\n                }\n              }\n            }\n        );\n\n        let agg_req: Aggregations =\n            serde_json::from_str(&serde_json::to_string(&elasticsearch_compatible_json).unwrap())\n                .unwrap();\n        let err = exec_request(agg_req, &index).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            r#\"An invalid argument was passed: '`interval` parameter \"30d\" in date histogram is unsupported, only `fixed_interval` is supported'\"#\n        );\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/histogram/histogram.rs",
    "content": "use std::cmp::Ordering;\n\nuse columnar::{Column, ColumnType};\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\nuse tantivy_bitpacker::minmax;\n\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::agg_req::Aggregations;\nuse crate::aggregation::agg_result::BucketEntry;\nuse crate::aggregation::cached_sub_aggs::{CachedSubAggs, HighCardCachedSubAggs};\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateBucketResult,\n    IntermediateHistogramBucketEntry,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// Contains all information required by the SegmentHistogramCollector to perform the\n/// histogram or date_histogram aggregation on a segment.\npub struct HistogramAggReqData {\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n    /// The field type of the fast field.\n    pub field_type: ColumnType,\n    /// The name of the aggregation.\n    pub name: String,\n    /// The histogram aggregation request.\n    pub req: HistogramAggregation,\n    /// True if this is a date_histogram aggregation.\n    pub is_date_histogram: bool,\n    /// The bounds to limit the buckets to.\n    pub bounds: HistogramBounds,\n    /// The offset used to calculate the bucket position.\n    pub offset: f64,\n}\nimpl HistogramAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\n/// Histogram is a bucket aggregation, where buckets are created dynamically for given `interval`.\n/// Each document value is rounded down to its bucket.\n///\n/// E.g. if we have a price 18 and an interval of 5, the document will fall into the bucket with\n/// the key 15. The formula used for this is:\n/// `((val - offset) / interval).floor() * interval + offset`\n///\n/// For this calculation all fastfield values are converted to f64.\n///\n/// # Returned Buckets\n/// By default buckets are returned between the min and max value of the documents, including empty\n/// buckets.\n/// Setting min_doc_count to != 0 will filter empty buckets.\n///\n/// The value range of the buckets can bet extended via\n/// [extended_bounds](HistogramAggregation::extended_bounds) or limit the range via\n/// [hard_bounds](HistogramAggregation::hard_bounds).\n///\n/// # Result\n/// Result type is [`BucketResult`](crate::aggregation::agg_result::BucketResult) with\n/// [`BucketEntry`](crate::aggregation::agg_result::BucketEntry) on the\n/// `AggregationCollector`.\n///\n/// Result type is\n/// [`IntermediateBucketResult`](crate::aggregation::intermediate_agg_result::IntermediateBucketResult) with\n/// [`IntermediateHistogramBucketEntry`](crate::aggregation::intermediate_agg_result::IntermediateHistogramBucketEntry) on the\n/// `DistributedAggregationCollector`.\n///\n/// # Limitations/Compatibility\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"prices\": {\n///         \"histogram\": {\n///             \"field\": \"price\",\n///             \"interval\": 10\n///         }\n///     }\n/// }\n/// ```\n///\n/// Response\n/// See [`BucketEntry`](crate::aggregation::agg_result::BucketEntry)\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\npub struct HistogramAggregation {\n    /// The field to aggregate on.\n    pub field: String,\n    /// The interval to chunk your data range. Each bucket spans a value range of [0..interval).\n    /// Must be a positive value.\n    #[serde(deserialize_with = \"deserialize_f64\")]\n    pub interval: f64,\n    /// Intervals implicitly defines an absolute grid of buckets `[interval * k, interval * (k +\n    /// 1))`.\n    ///\n    /// Offset makes it possible to shift this grid into\n    /// `[offset + interval * k, offset + interval * (k + 1))`. Offset has to be in the range [0,\n    /// interval).\n    ///\n    /// As an example, if there are two documents with value 9 and 12 and interval 10.0, they would\n    /// fall into the buckets with the key 0 and 10.\n    /// With offset 5 and interval 10, they would both fall into the bucket with they key 5 and the\n    /// range [5..15)\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub offset: Option<f64>,\n    /// The minimum number of documents in a bucket to be returned. Defaults to 0.\n    pub min_doc_count: Option<u64>,\n    /// Limits the data range to `[min, max]` closed interval.\n    ///\n    /// This can be used to filter values if they are not in the data range.\n    ///\n    /// hard_bounds only limits the buckets, to force a range set both extended_bounds and\n    /// hard_bounds to the same range.\n    ///\n    /// ## Example\n    /// ```json\n    /// {\n    ///     \"prices\": {\n    ///        \"histogram\": {\n    ///            \"field\": \"price\",\n    ///            \"interval\": 10,\n    ///            \"hard_bounds\": {\n    ///                \"min\": 0,\n    ///                \"max\": 100\n    ///            }\n    ///        }\n    ///    }\n    /// }\n    /// ```\n    pub hard_bounds: Option<HistogramBounds>,\n    /// Can be set to extend your bounds. The range of the buckets is by default defined by the\n    /// data range of the values of the documents. As the name suggests, this can only be used to\n    /// extend the value range. If the bounds for min or max are not extending the range, the value\n    /// has no effect on the returned buckets.\n    ///\n    /// Cannot be set in conjunction with min_doc_count > 0, since the empty buckets from extended\n    /// bounds would not be returned.\n    pub extended_bounds: Option<HistogramBounds>,\n    /// Whether to return the buckets as a hash map\n    #[serde(default)]\n    pub keyed: bool,\n    /// Whether the values are normalized to ns for date time values. Defaults to false.\n    #[serde(default)]\n    pub is_normalized_to_ns: bool,\n}\n\nimpl HistogramAggregation {\n    pub(crate) fn normalize_date_time(&mut self) {\n        if !self.is_normalized_to_ns {\n            // values are provided in ms, but the fastfield is in nano seconds\n            self.interval *= 1_000_000.0;\n            self.offset = self.offset.map(|off| off * 1_000_000.0);\n            self.hard_bounds = self.hard_bounds.map(|bounds| HistogramBounds {\n                min: bounds.min * 1_000_000.0,\n                max: bounds.max * 1_000_000.0,\n            });\n            self.extended_bounds = self.extended_bounds.map(|bounds| HistogramBounds {\n                min: bounds.min * 1_000_000.0,\n                max: bounds.max * 1_000_000.0,\n            });\n            self.is_normalized_to_ns = true;\n        }\n    }\n\n    fn validate(&self) -> crate::Result<()> {\n        if self.interval <= 0.0f64 {\n            return Err(TantivyError::InvalidArgument(\n                \"interval must be a positive value\".to_string(),\n            ));\n        }\n\n        if self.min_doc_count.unwrap_or(0) > 0 && self.extended_bounds.is_some() {\n            return Err(TantivyError::InvalidArgument(\n                \"Cannot set min_doc_count and extended_bounds at the same time\".to_string(),\n            ));\n        }\n\n        if let (Some(hard_bounds), Some(extended_bounds)) = (self.hard_bounds, self.extended_bounds)\n        {\n            if extended_bounds.min < hard_bounds.min || extended_bounds.max > hard_bounds.max {\n                return Err(TantivyError::InvalidArgument(format!(\n                    \"extended_bounds have to be inside hard_bounds, extended_bounds: \\\n                     {extended_bounds}, hard_bounds {hard_bounds}\"\n                )));\n            }\n        }\n\n        Ok(())\n    }\n\n    /// Returns the minimum number of documents required for a bucket to be returned.\n    pub fn min_doc_count(&self) -> u64 {\n        self.min_doc_count.unwrap_or(0)\n    }\n}\n\n/// Used to set extended or hard bounds on the histogram.\n#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]\npub struct HistogramBounds {\n    /// The lower bounds.\n    #[serde(deserialize_with = \"deserialize_date_or_num\")]\n    pub min: f64,\n    /// The upper bounds.\n    #[serde(deserialize_with = \"deserialize_date_or_num\")]\n    pub max: f64,\n}\n\nfn deserialize_date_or_num<'de, D>(deserializer: D) -> Result<f64, D::Error>\nwhere D: serde::Deserializer<'de> {\n    let value: serde_json::Value = Deserialize::deserialize(deserializer)?;\n\n    // Check if the value is a string representing an Rfc3339 formatted date\n    if let serde_json::Value::String(date_str) = value {\n        // Parse the Rfc3339 formatted date string into a DateTime<Utc>\n        let date =\n            time::OffsetDateTime::parse(&date_str, &time::format_description::well_known::Rfc3339)\n                .map_err(|_| serde::de::Error::custom(\"Invalid Rfc3339 formatted date\"))?;\n\n        let milliseconds: i64 = (date.unix_timestamp_nanos() / 1_000_000)\n            .try_into()\n            .map_err(|_| serde::de::Error::custom(\"{date_str} out of allowed range\"))?;\n\n        // Return the milliseconds as f64\n        Ok(milliseconds as f64)\n    } else {\n        // The value is not a string, so assume it's a regular f64 number\n        value\n            .as_f64()\n            .ok_or_else(|| serde::de::Error::custom(\"Invalid number format\"))\n    }\n}\n\nimpl Display for HistogramBounds {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.write_fmt(format_args!(\"[{},{}]\", self.min, self.max))\n    }\n}\n\nimpl HistogramBounds {\n    fn contains(&self, val: f64) -> bool {\n        val >= self.min && val <= self.max\n    }\n}\n\n#[derive(Default, Clone, Debug, PartialEq)]\npub(crate) struct SegmentHistogramBucketEntry {\n    pub key: f64,\n    pub doc_count: u64,\n    pub bucket_id: BucketId,\n}\n\nimpl SegmentHistogramBucketEntry {\n    pub(crate) fn into_intermediate_bucket_entry(\n        self,\n        sub_aggregation: &mut Option<HighCardCachedSubAggs>,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<IntermediateHistogramBucketEntry> {\n        let mut sub_aggregation_res = IntermediateAggregationResults::default();\n        if let Some(sub_aggregation) = sub_aggregation {\n            sub_aggregation\n                .get_sub_agg_collector()\n                .add_intermediate_aggregation_result(\n                    agg_data,\n                    &mut sub_aggregation_res,\n                    self.bucket_id,\n                )?;\n        }\n        Ok(IntermediateHistogramBucketEntry {\n            key: self.key,\n            doc_count: self.doc_count,\n            sub_aggregation: sub_aggregation_res,\n        })\n    }\n}\n\n#[derive(Clone, Debug, Default)]\nstruct HistogramBuckets {\n    pub buckets: FxHashMap<i64, SegmentHistogramBucketEntry>,\n}\n\n/// The collector puts values from the fast field into the correct buckets and does a conversion to\n/// the correct datatype.\n#[derive(Debug)]\npub struct SegmentHistogramCollector {\n    /// The buckets containing the aggregation data.\n    /// One Histogram bucket per parent bucket id.\n    parent_buckets: Vec<HistogramBuckets>,\n    sub_agg: Option<HighCardCachedSubAggs>,\n    accessor_idx: usize,\n    bucket_id_provider: BucketIdProvider,\n}\n\nimpl SegmentAggregationCollector for SegmentHistogramCollector {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let name = agg_data\n            .get_histogram_req_data(self.accessor_idx)\n            .name\n            .clone();\n        // TODO: avoid prepare_max_bucket here and handle empty buckets.\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let histogram = std::mem::take(&mut self.parent_buckets[parent_bucket_id as usize]);\n        let bucket = self.add_intermediate_bucket_result(agg_data, histogram)?;\n        results.push(name, IntermediateAggregationResult::Bucket(bucket))?;\n\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let req = agg_data.take_histogram_req_data(self.accessor_idx);\n        let mem_pre = self.get_memory_consumption();\n        let buckets = &mut self.parent_buckets[parent_bucket_id as usize].buckets;\n\n        let bounds = req.bounds;\n        let interval = req.req.interval;\n        let offset = req.offset;\n        let get_bucket_pos = |val| get_bucket_pos_f64(val, interval, offset) as i64;\n\n        agg_data\n            .column_block_accessor\n            .fetch_block(docs, &req.accessor);\n        for (doc, val) in agg_data\n            .column_block_accessor\n            .iter_docid_vals(docs, &req.accessor)\n        {\n            let val = f64_from_fastfield_u64(val, req.field_type);\n            let bucket_pos = get_bucket_pos(val);\n            if bounds.contains(val) {\n                let bucket = buckets.entry(bucket_pos).or_insert_with(|| {\n                    let key = get_bucket_key_from_pos(bucket_pos as f64, interval, offset);\n                    SegmentHistogramBucketEntry {\n                        key,\n                        doc_count: 0,\n                        bucket_id: self.bucket_id_provider.next_bucket_id(),\n                    }\n                });\n                bucket.doc_count += 1;\n                if let Some(sub_agg) = &mut self.sub_agg {\n                    sub_agg.push(bucket.bucket_id, doc);\n                }\n            }\n        }\n        agg_data.put_back_histogram_req_data(self.accessor_idx, req);\n\n        let mem_delta = self.get_memory_consumption() - mem_pre;\n        if mem_delta > 0 {\n            agg_data\n                .context\n                .limits\n                .add_memory_consumed(mem_delta as u64)?;\n        }\n\n        if let Some(sub_agg) = &mut self.sub_agg {\n            sub_agg.check_flush_local(agg_data)?;\n        }\n\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(sub_aggregation) = &mut self.sub_agg {\n            sub_aggregation.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.parent_buckets.len() <= max_bucket as usize {\n            self.parent_buckets.push(HistogramBuckets {\n                buckets: FxHashMap::default(),\n            });\n        }\n        Ok(())\n    }\n}\n\nimpl SegmentHistogramCollector {\n    fn get_memory_consumption(&self) -> usize {\n        let self_mem = std::mem::size_of::<Self>();\n        let buckets_mem = self.parent_buckets.len() * std::mem::size_of::<HistogramBuckets>();\n        self_mem + buckets_mem\n    }\n    /// Converts the collector result into a intermediate bucket result.\n    fn add_intermediate_bucket_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        histogram: HistogramBuckets,\n    ) -> crate::Result<IntermediateBucketResult> {\n        let mut buckets = Vec::with_capacity(histogram.buckets.len());\n\n        for bucket in histogram.buckets.into_values() {\n            let bucket_res = bucket.into_intermediate_bucket_entry(&mut self.sub_agg, agg_data);\n\n            buckets.push(bucket_res?);\n        }\n        buckets.sort_unstable_by(|b1, b2| b1.key.total_cmp(&b2.key));\n\n        let is_date_agg = agg_data\n            .get_histogram_req_data(self.accessor_idx)\n            .field_type\n            == ColumnType::DateTime;\n        Ok(IntermediateBucketResult::Histogram {\n            buckets,\n            is_date_agg,\n        })\n    }\n\n    pub(crate) fn from_req_and_validate(\n        agg_data: &mut AggregationsSegmentCtx,\n        node: &AggRefNode,\n    ) -> crate::Result<Self> {\n        let sub_agg = if !node.children.is_empty() {\n            Some(build_segment_agg_collectors(agg_data, &node.children)?)\n        } else {\n            None\n        };\n        let req_data = agg_data.get_histogram_req_data_mut(node.idx_in_req_data);\n        req_data.req.validate()?;\n        if req_data.field_type == ColumnType::DateTime && !req_data.is_date_histogram {\n            req_data.req.normalize_date_time();\n        }\n        req_data.bounds = req_data.req.hard_bounds.unwrap_or(HistogramBounds {\n            min: f64::MIN,\n            max: f64::MAX,\n        });\n        req_data.offset = req_data.req.offset.unwrap_or(0.0);\n        let sub_agg = sub_agg.map(CachedSubAggs::new);\n\n        Ok(Self {\n            parent_buckets: Default::default(),\n            sub_agg,\n            accessor_idx: node.idx_in_req_data,\n            bucket_id_provider: BucketIdProvider::default(),\n        })\n    }\n}\n\n#[inline]\nfn get_bucket_pos_f64(val: f64, interval: f64, offset: f64) -> f64 {\n    ((val - offset) / interval).floor()\n}\n\n#[inline]\nfn get_bucket_key_from_pos(bucket_pos: f64, interval: f64, offset: f64) -> f64 {\n    bucket_pos * interval + offset\n}\n\n// Convert to BucketEntry and fill gaps\nfn intermediate_buckets_to_final_buckets_fill_gaps(\n    buckets: Vec<IntermediateHistogramBucketEntry>,\n    histogram_req: &HistogramAggregation,\n    sub_aggregation: &Aggregations,\n    limits: &mut AggregationLimitsGuard,\n) -> crate::Result<Vec<BucketEntry>> {\n    // Generate the full list of buckets without gaps.\n    //\n    // The bounds are the min max from the current buckets, optionally extended by\n    // extended_bounds from the request\n    let min_max = minmax(buckets.iter().map(|bucket| bucket.key));\n\n    // memory check upfront\n    let (_, first_bucket_num, last_bucket_num) =\n        generate_bucket_pos_with_opt_minmax(histogram_req, min_max);\n\n    // It's based on user input, so we need to account for overflows\n    let added_buckets = ((last_bucket_num.saturating_sub(first_bucket_num)).max(0) as u64)\n        .saturating_sub(buckets.len() as u64);\n    limits.add_memory_consumed(\n        added_buckets * std::mem::size_of::<IntermediateHistogramBucketEntry>() as u64,\n    )?;\n    // create buckets\n    let fill_gaps_buckets = generate_buckets_with_opt_minmax(histogram_req, min_max);\n\n    let empty_sub_aggregation = IntermediateAggregationResults::empty_from_req(sub_aggregation);\n\n    // Use merge_join_by to fill in gaps, since buckets are sorted\n\n    let final_buckets: Vec<BucketEntry> = buckets\n        .into_iter()\n        .merge_join_by(fill_gaps_buckets, |existing_bucket, fill_gaps_bucket| {\n            existing_bucket\n                .key\n                .partial_cmp(fill_gaps_bucket)\n                .unwrap_or(Ordering::Equal)\n        })\n        .map(|either| match either {\n            // Ignore the generated bucket\n            itertools::EitherOrBoth::Both(existing, _) => existing,\n            itertools::EitherOrBoth::Left(existing) => existing,\n            // Add missing bucket\n            itertools::EitherOrBoth::Right(missing_bucket) => IntermediateHistogramBucketEntry {\n                key: missing_bucket,\n                doc_count: 0,\n                sub_aggregation: empty_sub_aggregation.clone(),\n            },\n        })\n        .map(|intermediate_bucket| {\n            intermediate_bucket.into_final_bucket_entry(sub_aggregation, limits)\n        })\n        .collect::<crate::Result<Vec<_>>>()?;\n\n    Ok(final_buckets)\n}\n\n// Convert to BucketEntry\npub(crate) fn intermediate_histogram_buckets_to_final_buckets(\n    buckets: Vec<IntermediateHistogramBucketEntry>,\n    is_date_agg: bool,\n    histogram_req: &HistogramAggregation,\n    sub_aggregation: &Aggregations,\n    limits: &mut AggregationLimitsGuard,\n) -> crate::Result<Vec<BucketEntry>> {\n    // Normalization is column type dependent.\n    // The request used in the the call to final is not yet be normalized.\n    // Normalization is changing the precision from milliseconds to nanoseconds.\n    let mut histogram_req = histogram_req.clone();\n    if is_date_agg {\n        histogram_req.normalize_date_time();\n    }\n    let mut buckets = if histogram_req.min_doc_count() == 0 {\n        // With min_doc_count != 0, we may need to add buckets, so that there are no\n        // gaps, since intermediate result does not contain empty buckets (filtered to\n        // reduce serialization size).\n        intermediate_buckets_to_final_buckets_fill_gaps(\n            buckets,\n            &histogram_req,\n            sub_aggregation,\n            limits,\n        )?\n    } else {\n        buckets\n            .into_iter()\n            .filter(|histogram_bucket| histogram_bucket.doc_count >= histogram_req.min_doc_count())\n            .map(|histogram_bucket| {\n                histogram_bucket.into_final_bucket_entry(sub_aggregation, limits)\n            })\n            .collect::<crate::Result<Vec<_>>>()?\n    };\n\n    // If we have a date type on the histogram buckets, we add the `key_as_string` field as rfc339\n    // and normalize from nanoseconds to milliseconds\n    if is_date_agg {\n        for bucket in buckets.iter_mut() {\n            if let crate::aggregation::Key::F64(ref mut val) = bucket.key {\n                let key_as_string = format_date(*val as i64)?;\n                *val /= 1_000_000.0;\n                bucket.key_as_string = Some(key_as_string);\n            }\n        }\n    }\n\n    Ok(buckets)\n}\n\n/// Applies req extended_bounds/hard_bounds on the min_max value\n///\n/// May return `(f64::MAX, f64::MIN)`, if there is no range.\nfn get_req_min_max(req: &HistogramAggregation, min_max: Option<(f64, f64)>) -> (f64, f64) {\n    let (mut min, mut max) = min_max.unwrap_or((f64::MAX, f64::MIN));\n\n    if let Some(extended_bounds) = &req.extended_bounds {\n        min = min.min(extended_bounds.min);\n        max = max.max(extended_bounds.max);\n    }\n\n    if let Some(hard_bounds) = &req.hard_bounds {\n        min = min.max(hard_bounds.min);\n        max = max.min(hard_bounds.max);\n    }\n\n    (min, max)\n}\n\n/// Generates buckets with req.interval\n/// Range is computed for provided min_max and request extended_bounds/hard_bounds\n/// returns empty vec when there is no range to span\npub(crate) fn generate_bucket_pos_with_opt_minmax(\n    req: &HistogramAggregation,\n    min_max: Option<(f64, f64)>,\n) -> (f64, i64, i64) {\n    let (min, max) = get_req_min_max(req, min_max);\n\n    let offset = req.offset.unwrap_or(0.0);\n    let first_bucket_num = get_bucket_pos_f64(min, req.interval, offset) as i64;\n    let last_bucket_num = get_bucket_pos_f64(max, req.interval, offset) as i64;\n    (offset, first_bucket_num, last_bucket_num)\n}\n\n/// Generates buckets with req.interval\n/// Range is computed for provided min_max and request extended_bounds/hard_bounds\n/// returns empty vec when there is no range to span\npub(crate) fn generate_buckets_with_opt_minmax(\n    req: &HistogramAggregation,\n    min_max: Option<(f64, f64)>,\n) -> Vec<f64> {\n    let (offset, first_bucket_num, last_bucket_num) =\n        generate_bucket_pos_with_opt_minmax(req, min_max);\n    let mut buckets = Vec::with_capacity((first_bucket_num..=last_bucket_num).count());\n    for bucket_pos in first_bucket_num..=last_bucket_num {\n        let bucket_key = bucket_pos as f64 * req.interval + offset;\n        buckets.push(bucket_key);\n    }\n\n    buckets\n}\n\n#[cfg(test)]\nmod tests {\n\n    use pretty_assertions::assert_eq;\n    use serde_json::Value;\n\n    use super::*;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::tests::{\n        exec_request, exec_request_with_query, exec_request_with_query_and_memory_limit,\n        get_test_index_2_segments, get_test_index_from_values, get_test_index_with_num_docs,\n    };\n    use crate::query::AllQuery;\n\n    #[test]\n    fn histogram_test_crooked_values() -> crate::Result<()> {\n        let values = vec![-12.0, 12.31, 14.33, 16.23];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_interval\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 3.5,\n                    \"offset\": 0.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"key\"], -14.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][7][\"key\"], 10.5);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][7][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][8][\"key\"], 14.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][8][\"doc_count\"], 2);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][9], Value::Null);\n\n        // With offset\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_interval\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 3.5,\n                    \"offset\": 1.2,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"key\"], -12.8);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][1][\"key\"], -9.3);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][2][\"key\"], -5.8);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][3][\"key\"], -2.3);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][3][\"doc_count\"], 0);\n\n        assert_eq!(res[\"my_interval\"][\"buckets\"][7][\"key\"], 11.7);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][7][\"doc_count\"], 2);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][8][\"key\"], 15.2);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][8][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][9], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_test_min_value_positive_force_merge_segments() -> crate::Result<()> {\n        histogram_test_min_value_positive_merge_segments(true)\n    }\n\n    #[test]\n    fn histogram_test_min_value_positive() -> crate::Result<()> {\n        histogram_test_min_value_positive_merge_segments(false)\n    }\n    fn histogram_test_min_value_positive_merge_segments(merge_segments: bool) -> crate::Result<()> {\n        let values = vec![10.0, 12.0, 14.0, 16.23];\n\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_interval\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"key\"], 10.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][1][\"key\"], 11.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][2][\"key\"], 12.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][2][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][3][\"key\"], 13.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][3][\"doc_count\"], 0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][6][\"key\"], 16.0);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][6][\"doc_count\"], 1);\n        assert_eq!(res[\"my_interval\"][\"buckets\"][7], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_simple_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 0.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 1.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"histogram\"][\"buckets\"][99][\"key\"], 99.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][99][\"doc_count\"], 1);\n        assert_eq!(res[\"histogram\"][\"buckets\"][100], Value::Null);\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_memory_limit() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(true, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 0.1,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query_and_memory_limit(\n            agg_req,\n            &index,\n            None,\n            AggregationLimitsGuard::new(Some(5_000), None),\n        )\n        .unwrap_err();\n        assert!(res.to_string().starts_with(\n            \"Aborting aggregation because memory limit was exceeded. Limit: 5.00 KB, Current\"\n        ));\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_merge_test() -> crate::Result<()> {\n        // Merge buckets counts from different segments\n        let values = vec![10.0, 12.0, 14.0, 16.23, 10.0, 13.0, 10.0, 12.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 10.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 11.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 2);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"key\"], 13.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"doc_count\"], 1);\n\n        Ok(())\n    }\n    #[test]\n    fn histogram_min_doc_test_multi_segments() -> crate::Result<()> {\n        histogram_min_doc_test_with_opt(false)\n    }\n    #[test]\n    fn histogram_min_doc_test_single_segments() -> crate::Result<()> {\n        histogram_min_doc_test_with_opt(true)\n    }\n    fn histogram_min_doc_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let values = vec![10.0, 12.0, 14.0, 16.23, 10.0, 13.0, 10.0, 12.0];\n\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"min_doc_count\": 2,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 10.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_extended_bounds_test_multi_segment() -> crate::Result<()> {\n        histogram_extended_bounds_test_with_opt(false)\n    }\n    #[test]\n    fn histogram_extended_bounds_test_single_segment() -> crate::Result<()> {\n        histogram_extended_bounds_test_with_opt(true)\n    }\n    fn histogram_extended_bounds_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let values = vec![5.0];\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 2.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"doc_count\"], 0);\n\n        // 2 hits\n        let values = vec![5.0, 5.5];\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 3.0,\n                        \"max\": 6.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 4.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"key\"], 5.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 2);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"key\"], 6.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][4], Value::Null);\n\n        // 1 hit outside bounds\n        let values = vec![15.0];\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 3.0,\n                        \"max\": 6.0,\n                    },\n                    \"hard_bounds\": {\n                        \"min\": 3.0,\n                        \"max\": 6.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 4.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"key\"], 5.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"key\"], 6.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][3][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][4], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_hard_bounds_test_multi_segment() -> crate::Result<()> {\n        histogram_hard_bounds_test_with_opt(false)\n    }\n    #[test]\n    fn histogram_hard_bounds_test_single_segment() -> crate::Result<()> {\n        histogram_hard_bounds_test_with_opt(true)\n    }\n    fn histogram_hard_bounds_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let values = vec![10.0, 12.0, 14.0, 16.23, 10.0, 13.0, 10.0, 12.0];\n\n        let index = get_test_index_from_values(merge_segments, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"hard_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 10.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 11.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 2);\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][3], Value::Null);\n\n        // hard_bounds and extended_bounds will act like a force bounds\n        //\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                    \"hard_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 2.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"doc_count\"], 2);\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][11], Value::Null);\n\n        // invalid request\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 1.0,\n                        \"max\": 12.0,\n                    },\n                    \"hard_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index).unwrap_err();\n        assert_eq!(\n            res.to_string(),\n            \"An invalid argument was passed: 'extended_bounds have to be inside hard_bounds, \\\n             extended_bounds: [1,12], hard_bounds [2,12]'\"\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_empty_result_behaviour_test_single_segment() -> crate::Result<()> {\n        histogram_empty_result_behaviour_test_with_opt(true)\n    }\n\n    #[test]\n    fn histogram_empty_result_behaviour_test_multi_segment() -> crate::Result<()> {\n        histogram_empty_result_behaviour_test_with_opt(false)\n    }\n\n    fn histogram_empty_result_behaviour_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let index = get_test_index_2_segments(merge_segments)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req.clone(), &index, Some((\"text\", \"blubberasdf\")))?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"histogram\": {\n                    \"buckets\": []\n                }\n            })\n        );\n\n        // test index without segments\n        let values = vec![];\n\n        // Don't merge empty segments\n        let index = get_test_index_from_values(false, &values)?;\n\n        let res = exec_request_with_query(agg_req, &index, Some((\"text\", \"blubberasdf\")))?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"histogram\": {\n                    \"buckets\": []\n                }\n            })\n        );\n\n        // test index without segments\n        let values = vec![];\n\n        // Don't merge empty segments\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 2.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"doc_count\"], 0);\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 5.0,\n                    },\n                    \"hard_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 2.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10], Value::Null);\n\n        // hard_bounds will not extend the result\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"hard_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"histogram\": {\n                    \"buckets\": []\n                }\n            })\n        );\n\n        let sub_agg_req: Aggregations = serde_json::from_value(json!({\n            \"stats\": { \"stats\": { \"field\": \"score_f64\", } },\n            \"avg\": { \"avg\": { \"field\": \"score_f64\", } }\n\n        }))\n        .unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 1.0,\n                    \"extended_bounds\": {\n                        \"min\": 2.0,\n                        \"max\": 12.0,\n                    },\n                },\n                \"aggs\": sub_agg_req\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(\n            res[\"histogram\"][\"buckets\"][0],\n            json!({\n                \"avg\": {\n                    \"value\": Value::Null\n                },\n                \"doc_count\": 0,\n                \"key\": 2.0,\n                \"stats\": {\n                    \"sum\": 0.0,\n                    \"count\": 0,\n                    \"min\": Value::Null,\n                    \"max\": Value::Null,\n                    \"avg\": Value::Null,\n                }\n            })\n        );\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 2.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 3.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"doc_count\"], 0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"key\"], 12.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][10][\"doc_count\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_single_bucket_test_single_segment() -> crate::Result<()> {\n        histogram_single_bucket_test_with_opt(true)\n    }\n\n    #[test]\n    fn histogram_single_bucket_test_multi_segment() -> crate::Result<()> {\n        histogram_single_bucket_test_with_opt(false)\n    }\n\n    fn histogram_single_bucket_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let index = get_test_index_2_segments(merge_segments)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 100000.0,\n                },\n            }\n        }))\n        .unwrap();\n\n        let agg_res = exec_request(agg_req, &index)?;\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 0.0);\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 9);\n        assert_eq!(res[\"histogram\"][\"buckets\"][1], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_date_test_single_segment() -> crate::Result<()> {\n        histogram_date_test_with_opt(true)\n    }\n\n    #[test]\n    fn histogram_date_test_multi_segment() -> crate::Result<()> {\n        histogram_date_test_with_opt(false)\n    }\n\n    fn histogram_date_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let index = get_test_index_2_segments(merge_segments)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"date\",\n                    \"interval\": 86400000.0, // one day in milliseconds seconds\n                },\n            }\n        }))\n        .unwrap();\n\n        let agg_res = exec_request(agg_req, &index)?;\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"key\"], 1546300800000.0);\n        assert_eq!(\n            res[\"histogram\"][\"buckets\"][0][\"key_as_string\"],\n            \"2019-01-01T00:00:00Z\"\n        );\n        assert_eq!(res[\"histogram\"][\"buckets\"][0][\"doc_count\"], 1);\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"key\"], 1546387200000.0);\n        assert_eq!(\n            res[\"histogram\"][\"buckets\"][1][\"key_as_string\"],\n            \"2019-01-02T00:00:00Z\"\n        );\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][1][\"doc_count\"], 5);\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][2][\"key\"], 1546473600000.0);\n        assert_eq!(\n            res[\"histogram\"][\"buckets\"][2][\"key_as_string\"],\n            \"2019-01-03T00:00:00Z\"\n        );\n\n        assert_eq!(res[\"histogram\"][\"buckets\"][3], Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_invalid_request() -> crate::Result<()> {\n        let index = get_test_index_2_segments(true)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 0.0,\n                },\n            }\n        }))\n        .unwrap();\n\n        let agg_res = exec_request(agg_req, &index);\n\n        assert!(agg_res.is_err());\n\n        Ok(())\n    }\n\n    #[test]\n    fn histogram_keyed_buckets_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"histogram\": {\n                \"histogram\": {\n                    \"field\": \"score_f64\",\n                    \"interval\": 50.0,\n                    \"keyed\": true\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"histogram\": {\n                    \"buckets\": {\n                        \"0\": {\n                            \"key\": 0.0,\n                            \"doc_count\": 50\n                        },\n                        \"50\": {\n                            \"key\": 50.0,\n                            \"doc_count\": 50\n                        }\n                    }\n                }\n            })\n        );\n\n        Ok(())\n    }\n    #[test]\n    fn test_aggregation_histogram_empty_index() -> crate::Result<()> {\n        // test index without segments\n        let values = vec![];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"myhisto\": {\n                \"histogram\": {\n                    \"field\": \"score\",\n                    \"interval\": 10.0\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        // Make sure the result structure is correct\n        assert_eq!(res[\"myhisto\"][\"buckets\"].as_array().unwrap().len(), 0);\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/histogram/mod.rs",
    "content": "mod date_histogram;\nmod histogram;\npub use date_histogram::*;\npub use histogram::*;\n"
  },
  {
    "path": "src/aggregation/bucket/mod.rs",
    "content": "//! Module for all bucket aggregations.\n//!\n//! BucketAggregations create buckets of documents.\n//! Each bucket is associated with a rule which\n//! determines whether or not a document in the falls into it. In other words, the buckets\n//! effectively define document sets. Buckets are not necessarily disjunct, therefore a document can\n//! fall into multiple buckets. In addition to the buckets themselves, the bucket aggregations also\n//! compute and return the number of documents for each bucket. Bucket aggregations, as opposed to\n//! metric aggregations, can hold sub-aggregations. These sub-aggregations will be aggregated for\n//! the buckets created by their \"parent\" bucket aggregation. There are different bucket\n//! aggregators, each with a different \"bucketing\" strategy. Some define a single bucket, some\n//! define fixed number of multiple buckets, and others dynamically create the buckets during the\n//! aggregation process.\n//!\n//! Results of final buckets are [`BucketResult`](super::agg_result::BucketResult).\n//! Results of intermediate buckets are\n//! [`IntermediateBucketResult`](super::intermediate_agg_result::IntermediateBucketResult)\n//!\n//! ## Supported Bucket Aggregations\n//! - [Histogram](HistogramAggregation)\n//! - [DateHistogram](DateHistogramAggregationReq)\n//! - [Range](RangeAggregation)\n//! - [Terms](TermsAggregation)\n\nmod composite;\nmod filter;\nmod histogram;\nmod range;\nmod term_agg;\nmod term_missing_agg;\n\nuse std::collections::HashMap;\nuse std::fmt;\n\npub use composite::*;\npub use filter::*;\npub use histogram::*;\npub use range::*;\nuse serde::{de, Deserialize, Deserializer, Serialize, Serializer};\npub use term_agg::*;\npub use term_missing_agg::*;\n\n/// Order for buckets in a bucket aggregation.\n#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, Default)]\npub enum Order {\n    /// Asc order\n    #[serde(rename = \"asc\")]\n    Asc,\n    /// Desc order\n    #[serde(rename = \"desc\")]\n    #[default]\n    Desc,\n}\n\n#[derive(Clone, Debug, PartialEq)]\n/// Order property by which to apply the order\n#[derive(Default)]\npub enum OrderTarget {\n    /// The key of the bucket\n    Key,\n    /// The doc count of the bucket\n    #[default]\n    Count,\n    /// Order by value of the sub aggregation metric with identified by given `String`.\n    ///\n    /// Only single value metrics are supported currently\n    SubAggregation(String),\n}\n\nimpl From<&str> for OrderTarget {\n    fn from(val: &str) -> Self {\n        match val {\n            \"_key\" => OrderTarget::Key,\n            \"_count\" => OrderTarget::Count,\n            _ => OrderTarget::SubAggregation(val.to_string()),\n        }\n    }\n}\n\nimpl fmt::Display for OrderTarget {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        match self {\n            OrderTarget::Key => f.write_str(\"_key\"),\n            OrderTarget::Count => f.write_str(\"_count\"),\n            OrderTarget::SubAggregation(agg) => agg.fmt(f),\n        }\n    }\n}\n\n/// Set the order. target is either \"_count\", \"_key\", or the name of\n/// a metric sub_aggregation.\n///\n/// De/Serializes to elasticsearch compatible JSON.\n///\n/// Examples in JSON format:\n/// { \"_count\": \"asc\" }\n/// { \"_key\": \"asc\" }\n/// { \"average_price\": \"asc\" }\n#[derive(Clone, Default, Debug, PartialEq)]\npub struct CustomOrder {\n    /// The target property by which to sort by\n    pub target: OrderTarget,\n    /// The order asc or desc\n    pub order: Order,\n}\n\nimpl Serialize for CustomOrder {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: Serializer {\n        let map: HashMap<String, Order> =\n            std::iter::once((self.target.to_string(), self.order)).collect();\n        map.serialize(serializer)\n    }\n}\n\nimpl<'de> Deserialize<'de> for CustomOrder {\n    fn deserialize<D>(deserializer: D) -> Result<CustomOrder, D::Error>\n    where D: Deserializer<'de> {\n        let value = serde_json::Value::deserialize(deserializer)?;\n        let return_err = |message, val: serde_json::Value| {\n            de::Error::custom(format!(\n                \"{}, but got {}\",\n                message,\n                serde_json::to_string(&val).unwrap()\n            ))\n        };\n\n        match value {\n            serde_json::Value::Object(map) => {\n                if map.len() != 1 {\n                    return Err(return_err(\n                        \"expected exactly one key-value pair in the order map\",\n                        map.into(),\n                    ));\n                }\n\n                let (key, value) = map.into_iter().next().unwrap();\n                let order = serde_json::from_value(value).map_err(de::Error::custom)?;\n\n                Ok(CustomOrder {\n                    target: key.as_str().into(),\n                    order,\n                })\n            }\n            serde_json::Value::Array(arr) => {\n                if arr.is_empty() {\n                    return Err(return_err(\"unexpected empty array in order\", arr.into()));\n                }\n                if arr.len() != 1 {\n                    return Err(return_err(\n                        \"only one sort order supported currently\",\n                        arr.into(),\n                    ));\n                }\n                let entry = arr.into_iter().next().unwrap();\n                let map = entry\n                    .as_object()\n                    .ok_or_else(|| return_err(\"expected object as sort order\", entry.clone()))?;\n                let (key, value) = map.into_iter().next().ok_or_else(|| {\n                    return_err(\n                        \"expected exactly one key-value pair in the order map\",\n                        entry.clone(),\n                    )\n                })?;\n                let order = serde_json::from_value(value.clone()).map_err(de::Error::custom)?;\n\n                Ok(CustomOrder {\n                    target: key.as_str().into(),\n                    order,\n                })\n            }\n            _ => Err(return_err(\n                \"unexpected type, expected an object or array\",\n                value,\n            )),\n        }\n    }\n}\n\n#[test]\nfn custom_order_serde_test() {\n    let order = CustomOrder {\n        target: OrderTarget::Key,\n        order: Order::Desc,\n    };\n\n    let order_str = serde_json::to_string(&order).unwrap();\n    assert_eq!(order_str, \"{\\\"_key\\\":\\\"desc\\\"}\");\n    let order_deser = serde_json::from_str(&order_str).unwrap();\n\n    assert_eq!(order, order_deser);\n    let order_deser: CustomOrder = serde_json::from_str(\"[{\\\"_key\\\":\\\"desc\\\"}]\").unwrap();\n    assert_eq!(order, order_deser);\n\n    let order_deser: serde_json::Result<CustomOrder> = serde_json::from_str(\"{}\");\n    assert!(order_deser.is_err());\n\n    let order_deser: serde_json::Result<CustomOrder> = serde_json::from_str(\"[]\");\n    assert!(order_deser\n        .unwrap_err()\n        .to_string()\n        .contains(\"unexpected empty array in order\"));\n\n    let order_deser: serde_json::Result<CustomOrder> =\n        serde_json::from_str(r#\"[{\"_key\":\"desc\"},{\"_key\":\"desc\"}]\"#);\n    assert_eq!(\n        order_deser.unwrap_err().to_string(),\n        r#\"only one sort order supported currently, but got [{\"_key\":\"desc\"},{\"_key\":\"desc\"}]\"#\n    );\n}\n"
  },
  {
    "path": "src/aggregation/bucket/range.rs",
    "content": "use std::fmt::Debug;\nuse std::ops::Range;\n\nuse columnar::{Column, ColumnType};\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::agg_limits::AggregationLimitsGuard;\nuse crate::aggregation::cached_sub_aggs::{\n    CachedSubAggs, HighCardSubAggCache, LowCardCachedSubAggs, LowCardSubAggCache, SubAggCache,\n};\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateBucketResult,\n    IntermediateRangeBucketEntry, IntermediateRangeBucketResult,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// Contains all information required by the SegmentRangeCollector to perform the\n/// range aggregation on a segment.\npub struct RangeAggReqData {\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n    /// The type of the fast field.\n    pub field_type: ColumnType,\n    /// The range aggregation request.\n    pub req: RangeAggregation,\n    /// The name of the aggregation.\n    pub name: String,\n    /// Whether this is a top-level aggregation.\n    pub is_top_level: bool,\n}\n\nimpl RangeAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\n/// Provide user-defined buckets to aggregate on.\n///\n/// Two special buckets will automatically be created to cover the whole range of values.\n/// The provided buckets have to be continuous.\n/// During the aggregation, the values extracted from the fast_field `field` will be checked\n/// against each bucket range. Note that this aggregation includes the from value and excludes the\n/// to value for each range.\n///\n/// Result type is [`BucketResult`](crate::aggregation::agg_result::BucketResult) with\n/// [`RangeBucketEntry`](crate::aggregation::agg_result::RangeBucketEntry) on the\n/// `AggregationCollector`.\n///\n/// Result type is\n/// [`IntermediateBucketResult`](crate::aggregation::intermediate_agg_result::IntermediateBucketResult) with\n/// [`IntermediateRangeBucketEntry`](crate::aggregation::intermediate_agg_result::IntermediateRangeBucketEntry) on the\n/// `DistributedAggregationCollector`.\n///\n/// # Limitations/Compatibility\n/// Overlapping ranges are not yet supported.\n///\n/// # Request JSON Format\n/// ```json\n/// {\n///     \"my_ranges\": {\n///         \"field\": \"score\",\n///         \"ranges\": [\n///             { \"to\": 3.0 },\n///             { \"from\": 3.0, \"to\": 7.0 },\n///             { \"from\": 7.0, \"to\": 20.0 },\n///             { \"from\": 20.0 }\n///         ]\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\npub struct RangeAggregation {\n    /// The field to aggregate on.\n    pub field: String,\n    /// Note that this aggregation includes the from value and excludes the to value for each\n    /// range. Extra buckets will be created until the first to, and last from, if necessary.\n    pub ranges: Vec<RangeAggregationRange>,\n    /// Whether to return the buckets as a hash map\n    #[serde(default)]\n    pub keyed: bool,\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n/// The range for one range bucket.\npub struct RangeAggregationRange {\n    /// Custom key for the range bucket\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub key: Option<String>,\n    /// The from range value, which is inclusive in the range.\n    /// `None` equals to an open ended interval.\n    #[serde(\n        skip_serializing_if = \"Option::is_none\",\n        default,\n        deserialize_with = \"deserialize_option_f64\"\n    )]\n    pub from: Option<f64>,\n    /// The to range value, which is not inclusive in the range.\n    /// `None` equals to an open ended interval.\n    #[serde(\n        skip_serializing_if = \"Option::is_none\",\n        default,\n        deserialize_with = \"deserialize_option_f64\"\n    )]\n    pub to: Option<f64>,\n}\n\nimpl From<Range<f64>> for RangeAggregationRange {\n    fn from(range: Range<f64>) -> Self {\n        let from = if range.start == f64::MIN {\n            None\n        } else {\n            Some(range.start)\n        };\n        let to = if range.end == f64::MAX {\n            None\n        } else {\n            Some(range.end)\n        };\n        RangeAggregationRange {\n            key: None,\n            from,\n            to,\n        }\n    }\n}\n\n#[derive(Clone, Debug, PartialEq)]\n/// Internally used u64 range for one range bucket.\npub(crate) struct InternalRangeAggregationRange {\n    /// Custom key for the range bucket\n    key: Option<String>,\n    /// `u64` range value\n    range: Range<u64>,\n}\n\nimpl From<Range<u64>> for InternalRangeAggregationRange {\n    fn from(range: Range<u64>) -> Self {\n        InternalRangeAggregationRange { key: None, range }\n    }\n}\n\n#[derive(Clone, Debug)]\npub(crate) struct SegmentRangeAndBucketEntry {\n    range: Range<u64>,\n    bucket: SegmentRangeBucketEntry,\n}\n\n/// The collector puts values from the fast field into the correct buckets and does a conversion to\n/// the correct datatype.\npub struct SegmentRangeCollector<C: SubAggCache> {\n    /// The buckets containing the aggregation data.\n    /// One for each ParentBucketId\n    parent_buckets: Vec<Vec<SegmentRangeAndBucketEntry>>,\n    column_type: ColumnType,\n    pub(crate) accessor_idx: usize,\n    sub_agg: Option<CachedSubAggs<C>>,\n    /// Here things get a bit weird. We need to assign unique bucket ids across all\n    /// parent buckets. So we keep track of the next available bucket id here.\n    /// This allows a kind of flattening of the bucket ids across all parent buckets.\n    /// E.g. in nested aggregations:\n    /// Term Agg -> Range aggregation -> Stats aggregation\n    /// E.g. the Term Agg creates 3 buckets [\"INFO\", \"ERROR\", \"WARN\"], each of these has a Range\n    /// aggregation with 4 buckets. The Range aggregation will create buckets with ids:\n    /// - INFO: 0,1,2,3\n    /// - ERROR: 4,5,6,7\n    /// - WARN: 8,9,10,11\n    ///\n    /// This allows the Stats aggregation to have unique bucket ids to refer to.\n    bucket_id_provider: BucketIdProvider,\n    limits: AggregationLimitsGuard,\n}\n\nimpl<C: SubAggCache> Debug for SegmentRangeCollector<C> {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"SegmentRangeCollector\")\n            .field(\"parent_buckets_len\", &self.parent_buckets.len())\n            .field(\"column_type\", &self.column_type)\n            .field(\"accessor_idx\", &self.accessor_idx)\n            .field(\"has_sub_agg\", &self.sub_agg.is_some())\n            .finish()\n    }\n}\n\n/// TODO: Bad naming, there's also SegmentRangeAndBucketEntry\n#[derive(Clone)]\npub(crate) struct SegmentRangeBucketEntry {\n    pub key: Key,\n    pub doc_count: u64,\n    // pub sub_aggregation: Option<Box<dyn SegmentAggregationCollector>>,\n    pub bucket_id: BucketId,\n    /// The from range of the bucket. Equals `f64::MIN` when `None`.\n    pub from: Option<f64>,\n    /// The to range of the bucket. Equals `f64::MAX` when `None`. Open interval, `to` is not\n    /// inclusive.\n    pub to: Option<f64>,\n}\n\nimpl Debug for SegmentRangeBucketEntry {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"SegmentRangeBucketEntry\")\n            .field(\"key\", &self.key)\n            .field(\"doc_count\", &self.doc_count)\n            .field(\"from\", &self.from)\n            .field(\"to\", &self.to)\n            .finish()\n    }\n}\nimpl SegmentRangeBucketEntry {\n    pub(crate) fn into_intermediate_bucket_entry(\n        self,\n    ) -> crate::Result<IntermediateRangeBucketEntry> {\n        let sub_aggregation = IntermediateAggregationResults::default();\n\n        Ok(IntermediateRangeBucketEntry {\n            key: self.key.into(),\n            doc_count: self.doc_count,\n            sub_aggregation_res: sub_aggregation,\n            from: self.from,\n            to: self.to,\n        })\n    }\n}\n\nimpl<C: SubAggCache> SegmentAggregationCollector for SegmentRangeCollector<C> {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let field_type = self.column_type;\n        let name = agg_data\n            .get_range_req_data(self.accessor_idx)\n            .name\n            .to_string();\n\n        let buckets = std::mem::take(&mut self.parent_buckets[parent_bucket_id as usize]);\n\n        let buckets: FxHashMap<SerializedKey, IntermediateRangeBucketEntry> = buckets\n            .into_iter()\n            .map(|range_bucket| {\n                let bucket_id = range_bucket.bucket.bucket_id;\n                let mut agg = range_bucket.bucket.into_intermediate_bucket_entry()?;\n                if let Some(sub_aggregation) = &mut self.sub_agg {\n                    sub_aggregation\n                        .get_sub_agg_collector()\n                        .add_intermediate_aggregation_result(\n                            agg_data,\n                            &mut agg.sub_aggregation_res,\n                            bucket_id,\n                        )?;\n                }\n                Ok((range_to_string(&range_bucket.range, &field_type)?, agg))\n            })\n            .collect::<crate::Result<_>>()?;\n\n        let bucket = IntermediateBucketResult::Range(IntermediateRangeBucketResult {\n            buckets,\n            column_type: Some(self.column_type),\n        });\n\n        results.push(name, IntermediateAggregationResult::Bucket(bucket))?;\n\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let req = agg_data.take_range_req_data(self.accessor_idx);\n\n        agg_data\n            .column_block_accessor\n            .fetch_block(docs, &req.accessor);\n\n        let buckets = &mut self.parent_buckets[parent_bucket_id as usize];\n\n        for (doc, val) in agg_data\n            .column_block_accessor\n            .iter_docid_vals(docs, &req.accessor)\n        {\n            let bucket_pos = get_bucket_pos(val, buckets);\n            let bucket = &mut buckets[bucket_pos];\n            bucket.bucket.doc_count += 1;\n            if let Some(sub_agg) = self.sub_agg.as_mut() {\n                sub_agg.push(bucket.bucket.bucket_id, doc);\n            }\n        }\n\n        agg_data.put_back_range_req_data(self.accessor_idx, req);\n        if let Some(sub_agg) = self.sub_agg.as_mut() {\n            sub_agg.check_flush_local(agg_data)?;\n        }\n\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(sub_agg) = self.sub_agg.as_mut() {\n            sub_agg.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.parent_buckets.len() <= max_bucket as usize {\n            let new_buckets = self.create_new_buckets(agg_data)?;\n            self.parent_buckets.push(new_buckets);\n        }\n\n        Ok(())\n    }\n}\n/// Build a concrete `SegmentRangeCollector` with either a Vec- or HashMap-backed\n/// bucket storage, depending on the column type and aggregation level.\npub(crate) fn build_segment_range_collector(\n    agg_data: &mut AggregationsSegmentCtx,\n    node: &AggRefNode,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    let accessor_idx = node.idx_in_req_data;\n    let req_data = agg_data.get_range_req_data(node.idx_in_req_data);\n    let field_type = req_data.field_type;\n\n    // TODO: A better metric instead of is_top_level would be the number of buckets expected.\n    // E.g. If range agg is not top level, but the parent is a bucket agg with less than 10 buckets,\n    // we can are still in low cardinality territory.\n    let is_low_card = req_data.is_top_level && req_data.req.ranges.len() <= 64;\n\n    let sub_agg = if !node.children.is_empty() {\n        Some(build_segment_agg_collectors(agg_data, &node.children)?)\n    } else {\n        None\n    };\n\n    if is_low_card {\n        Ok(Box::new(SegmentRangeCollector::<LowCardSubAggCache> {\n            sub_agg: sub_agg.map(LowCardCachedSubAggs::new),\n            column_type: field_type,\n            accessor_idx,\n            parent_buckets: Vec::new(),\n            bucket_id_provider: BucketIdProvider::default(),\n            limits: agg_data.context.limits.clone(),\n        }))\n    } else {\n        Ok(Box::new(SegmentRangeCollector::<HighCardSubAggCache> {\n            sub_agg: sub_agg.map(CachedSubAggs::new),\n            column_type: field_type,\n            accessor_idx,\n            parent_buckets: Vec::new(),\n            bucket_id_provider: BucketIdProvider::default(),\n            limits: agg_data.context.limits.clone(),\n        }))\n    }\n}\n\nimpl<C: SubAggCache> SegmentRangeCollector<C> {\n    pub(crate) fn create_new_buckets(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<Vec<SegmentRangeAndBucketEntry>> {\n        let field_type = self.column_type;\n        let req_data = agg_data.get_range_req_data(self.accessor_idx);\n        // The range input on the request is f64.\n        // We need to convert to u64 ranges, because we read the values as u64.\n        // The mapping from the conversion is monotonic so ordering is preserved.\n        let buckets: Vec<_> = extend_validate_ranges(&req_data.req.ranges, &field_type)?\n            .iter()\n            .map(|range| {\n                let bucket_id = self.bucket_id_provider.next_bucket_id();\n                let key = range\n                    .key\n                    .clone()\n                    .map(|key| Ok(Key::Str(key)))\n                    .unwrap_or_else(|| range_to_key(&range.range, &field_type))?;\n                let to = if range.range.end == u64::MAX {\n                    None\n                } else {\n                    Some(f64_from_fastfield_u64(range.range.end, field_type))\n                };\n                let from = if range.range.start == u64::MIN {\n                    None\n                } else {\n                    Some(f64_from_fastfield_u64(range.range.start, field_type))\n                };\n                // let sub_aggregation = sub_agg_prototype.clone();\n\n                Ok(SegmentRangeAndBucketEntry {\n                    range: range.range.clone(),\n                    bucket: SegmentRangeBucketEntry {\n                        doc_count: 0,\n                        bucket_id,\n                        key,\n                        from,\n                        to,\n                    },\n                })\n            })\n            .collect::<crate::Result<_>>()?;\n\n        self.limits.add_memory_consumed(\n            buckets.len() as u64 * std::mem::size_of::<SegmentRangeAndBucketEntry>() as u64,\n        )?;\n        Ok(buckets)\n    }\n}\n#[inline]\nfn get_bucket_pos(val: u64, buckets: &[SegmentRangeAndBucketEntry]) -> usize {\n    let pos = buckets\n        .binary_search_by_key(&val, |probe| probe.range.start)\n        .unwrap_or_else(|pos| pos - 1);\n    debug_assert!(buckets[pos].range.contains(&val));\n    pos\n}\n\n/// Converts the user provided f64 range value to fast field value space.\n///\n/// Internally fast field values are always stored as u64.\n/// If the fast field has u64 `[1, 2, 5]`, these values are stored as is in the fast field.\n/// A fast field with f64 `[1.0, 2.0, 5.0]` is converted to u64 space, using a\n/// monotonic mapping function, so the order is preserved.\n///\n/// Consequently, a f64 user range 1.0..3.0 needs to be converted to fast field value space using\n/// the same monotonic mapping function, so that the provided ranges contain the u64 values in the\n/// fast field.\n/// The alternative would be that every value read would be converted to the f64 range, but that is\n/// more computational expensive when many documents are hit.\nfn to_u64_range(\n    range: &RangeAggregationRange,\n    field_type: &ColumnType,\n) -> crate::Result<InternalRangeAggregationRange> {\n    let start = if let Some(from) = range.from {\n        f64_to_fastfield_u64(from, field_type)\n            .ok_or_else(|| TantivyError::InvalidArgument(\"invalid field type\".to_string()))?\n    } else {\n        u64::MIN\n    };\n\n    let end = if let Some(to) = range.to {\n        f64_to_fastfield_u64(to, field_type)\n            .ok_or_else(|| TantivyError::InvalidArgument(\"invalid field type\".to_string()))?\n    } else {\n        u64::MAX\n    };\n\n    Ok(InternalRangeAggregationRange {\n        key: range.key.clone(),\n        range: start..end,\n    })\n}\n\n/// Extends the provided buckets to contain the whole value range, by inserting buckets at the\n/// beginning and end and filling gaps.\nfn extend_validate_ranges(\n    buckets: &[RangeAggregationRange],\n    field_type: &ColumnType,\n) -> crate::Result<Vec<InternalRangeAggregationRange>> {\n    let mut converted_buckets = buckets\n        .iter()\n        .map(|range| to_u64_range(range, field_type))\n        .collect::<crate::Result<Vec<_>>>()?;\n\n    converted_buckets.sort_by_key(|bucket| bucket.range.start);\n    if converted_buckets[0].range.start != u64::MIN {\n        converted_buckets.insert(0, (u64::MIN..converted_buckets[0].range.start).into());\n    }\n\n    if converted_buckets[converted_buckets.len() - 1].range.end != u64::MAX {\n        converted_buckets\n            .push((converted_buckets[converted_buckets.len() - 1].range.end..u64::MAX).into());\n    }\n\n    // fill up holes in the ranges\n    let find_hole = |converted_buckets: &[InternalRangeAggregationRange]| {\n        for (pos, ranges) in converted_buckets.windows(2).enumerate() {\n            if ranges[0].range.end > ranges[1].range.start {\n                return Err(TantivyError::InvalidArgument(format!(\n                    \"Overlapping ranges not supported range {:?}, range+1 {:?}\",\n                    ranges[0], ranges[1]\n                )));\n            }\n            if ranges[0].range.end != ranges[1].range.start {\n                return Ok(Some(pos));\n            }\n        }\n        Ok(None)\n    };\n\n    while let Some(hole_pos) = find_hole(&converted_buckets)? {\n        let new_range =\n            converted_buckets[hole_pos].range.end..converted_buckets[hole_pos + 1].range.start;\n        converted_buckets.insert(hole_pos + 1, new_range.into());\n    }\n\n    Ok(converted_buckets)\n}\n\npub(crate) fn range_to_string(\n    range: &Range<u64>,\n    field_type: &ColumnType,\n) -> crate::Result<String> {\n    // is_start is there for malformed requests, e.g. ig the user passes the range u64::MIN..0.0,\n    // it should be rendered as \"*-0\" and not \"*-*\"\n    let to_str = |val: u64, is_start: bool| {\n        if (is_start && val == u64::MIN) || (!is_start && val == u64::MAX) {\n            Ok(\"*\".to_string())\n        } else if *field_type == ColumnType::DateTime {\n            let val = i64::from_u64(val);\n            format_date(val)\n        } else {\n            Ok(f64_from_fastfield_u64(val, *field_type).to_string())\n        }\n    };\n\n    Ok(format!(\n        \"{}-{}\",\n        to_str(range.start, true)?,\n        to_str(range.end, false)?\n    ))\n}\n\npub(crate) fn range_to_key(range: &Range<u64>, field_type: &ColumnType) -> crate::Result<Key> {\n    Ok(Key::Str(range_to_string(range, field_type)?))\n}\n\n#[cfg(test)]\nmod tests {\n\n    use serde_json::Value;\n\n    use super::*;\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::{\n        exec_request, exec_request_with_query, get_test_index_2_segments,\n        get_test_index_with_num_docs,\n    };\n\n    pub fn get_collector_from_ranges(\n        ranges: Vec<RangeAggregationRange>,\n        field_type: ColumnType,\n    ) -> SegmentRangeCollector<HighCardSubAggCache> {\n        let req = RangeAggregation {\n            field: \"dummy\".to_string(),\n            ranges,\n            ..Default::default()\n        };\n        // Build buckets directly as in from_req_and_validate without AggregationsData\n        let buckets: Vec<_> = extend_validate_ranges(&req.ranges, &field_type)\n            .expect(\"unexpected error in extend_validate_ranges\")\n            .iter()\n            .map(|range| {\n                let key = range\n                    .key\n                    .clone()\n                    .map(|key| Ok(Key::Str(key)))\n                    .unwrap_or_else(|| range_to_key(&range.range, &field_type))\n                    .expect(\"unexpected error in range_to_key\");\n                let to = if range.range.end == u64::MAX {\n                    None\n                } else {\n                    Some(f64_from_fastfield_u64(range.range.end, field_type))\n                };\n                let from = if range.range.start == u64::MIN {\n                    None\n                } else {\n                    Some(f64_from_fastfield_u64(range.range.start, field_type))\n                };\n                SegmentRangeAndBucketEntry {\n                    range: range.range.clone(),\n                    bucket: SegmentRangeBucketEntry {\n                        doc_count: 0,\n                        key,\n                        from,\n                        to,\n                        bucket_id: 0,\n                    },\n                }\n            })\n            .collect();\n\n        SegmentRangeCollector {\n            parent_buckets: vec![buckets],\n            column_type: field_type,\n            accessor_idx: 0,\n            sub_agg: None,\n            bucket_id_provider: Default::default(),\n            limits: AggregationLimitsGuard::default(),\n        }\n    }\n\n    #[test]\n    fn range_fraction_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"range\": {\n                \"range\": {\n                    \"field\": \"fraction_f64\",\n                    \"ranges\": [\n                        {\"from\": 0.0, \"to\": 0.1},\n                        {\"from\": 0.1, \"to\": 0.2},\n                    ]\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(res[\"range\"][\"buckets\"][0][\"key\"], \"*-0\");\n        assert_eq!(res[\"range\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"range\"][\"buckets\"][1][\"key\"], \"0-0.1\");\n        assert_eq!(res[\"range\"][\"buckets\"][1][\"doc_count\"], 10);\n        assert_eq!(res[\"range\"][\"buckets\"][2][\"key\"], \"0.1-0.2\");\n        assert_eq!(res[\"range\"][\"buckets\"][2][\"doc_count\"], 10);\n        assert_eq!(res[\"range\"][\"buckets\"][3][\"key\"], \"0.2-*\");\n        assert_eq!(res[\"range\"][\"buckets\"][3][\"doc_count\"], 80);\n\n        Ok(())\n    }\n\n    #[test]\n    fn range_fraction_test_with_sub_agg() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let sub_agg_req: Aggregations = serde_json::from_value(json!({\n            \"avg\": { \"avg\": { \"field\": \"score_f64\", } }\n\n        }))\n        .unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"range\": {\n                \"range\": {\n                    \"field\": \"fraction_f64\",\n                    \"ranges\": [\n                        {\"from\": 0.0, \"to\": 0.1},\n                        {\"from\": 0.1, \"to\": 0.2},\n                    ]\n                },\n                \"aggs\": sub_agg_req\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(res[\"range\"][\"buckets\"][0][\"key\"], \"*-0\");\n        assert_eq!(res[\"range\"][\"buckets\"][0][\"doc_count\"], 0);\n        assert_eq!(res[\"range\"][\"buckets\"][1][\"key\"], \"0-0.1\");\n        assert_eq!(res[\"range\"][\"buckets\"][1][\"doc_count\"], 10);\n        assert_eq!(res[\"range\"][\"buckets\"][2][\"key\"], \"0.1-0.2\");\n        assert_eq!(res[\"range\"][\"buckets\"][2][\"doc_count\"], 10);\n        assert_eq!(res[\"range\"][\"buckets\"][3][\"key\"], \"0.2-*\");\n        assert_eq!(res[\"range\"][\"buckets\"][3][\"doc_count\"], 80);\n\n        Ok(())\n    }\n\n    #[test]\n    fn range_keyed_buckets_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"range\": {\n                \"range\": {\n                    \"field\": \"fraction_f64\",\n                    \"ranges\": [\n                        {\"from\": 0.0, \"to\": 0.1},\n                        {\"from\": 0.1, \"to\": 0.2},\n                    ],\n                    \"keyed\": true\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"range\": {\n                    \"buckets\": {\n                        \"*-0\": { \"key\": \"*-0\", \"doc_count\": 0, \"to\": 0.0},\n                        \"0-0.1\": {\"key\": \"0-0.1\", \"doc_count\": 10, \"from\": 0.0, \"to\": 0.1},\n                        \"0.1-0.2\": {\"key\": \"0.1-0.2\", \"doc_count\": 10, \"from\": 0.1, \"to\": 0.2},\n                        \"0.2-*\": {\"key\": \"0.2-*\", \"doc_count\": 80, \"from\": 0.2},\n                    }\n                }\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn range_custom_key_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"range\": {\n                \"range\": {\n                    \"field\": \"fraction_f64\",\n                    \"ranges\": [\n                        {\"key\": \"custom-key-0-to-0.1\", \"from\": 0.0, \"to\": 0.1},\n                        {\"from\": 0.1, \"to\": 0.2},\n                    ],\n                    \"keyed\": false\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"range\": {\n                    \"buckets\": [\n                        {\"key\": \"*-0\", \"doc_count\": 0, \"to\": 0.0},\n                        {\"key\": \"custom-key-0-to-0.1\", \"doc_count\": 10, \"from\": 0.0, \"to\": 0.1},\n                        {\"key\": \"0.1-0.2\", \"doc_count\": 10, \"from\": 0.1, \"to\": 0.2},\n                        {\"key\": \"0.2-*\", \"doc_count\": 80, \"from\": 0.2}\n                    ]\n                }\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn range_date_test_single_segment() -> crate::Result<()> {\n        range_date_test_with_opt(true)\n    }\n\n    #[test]\n    fn range_date_test_multi_segment() -> crate::Result<()> {\n        range_date_test_with_opt(false)\n    }\n\n    fn range_date_test_with_opt(merge_segments: bool) -> crate::Result<()> {\n        let index = get_test_index_2_segments(merge_segments)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"date_ranges\": {\n                \"range\": {\n                    \"field\": \"date\",\n                    \"ranges\": [\n                        {\"to\": 1546300800000000000i64},\n                        {\"from\": 1546300800000000000i64, \"to\": 1546387200000000000i64},\n                    ],\n                    \"keyed\": false\n                },\n            }\n        }))\n        .unwrap();\n\n        let agg_res = exec_request(agg_req, &index)?;\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][0][\"from_as_string\"],\n            Value::Null\n        );\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][0][\"key\"],\n            \"*-2019-01-01T00:00:00Z\"\n        );\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][1][\"from_as_string\"],\n            \"2019-01-01T00:00:00Z\"\n        );\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][1][\"to_as_string\"],\n            \"2019-01-02T00:00:00Z\"\n        );\n\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][2][\"from_as_string\"],\n            \"2019-01-02T00:00:00Z\"\n        );\n        assert_eq!(\n            res[\"date_ranges\"][\"buckets\"][2][\"to_as_string\"],\n            Value::Null\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn range_custom_key_keyed_buckets_test() -> crate::Result<()> {\n        let index = get_test_index_with_num_docs(false, 100)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"range\": {\n                \"range\": {\n                    \"field\": \"fraction_f64\",\n                    \"ranges\": [\n                        {\"key\": \"custom-key-0-to-0.1\", \"from\": 0.0, \"to\": 0.1},\n                    ],\n                    \"keyed\": true\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res,\n            json!({\n                \"range\": {\n                    \"buckets\": {\n                        \"*-0\": { \"key\": \"*-0\", \"doc_count\": 0, \"to\": 0.0},\n                        \"custom-key-0-to-0.1\": {\"key\": \"custom-key-0-to-0.1\", \"doc_count\": 10, \"from\": 0.0, \"to\": 0.1},\n                        \"0.1-*\": {\"key\": \"0.1-*\", \"doc_count\": 90, \"from\": 0.1},\n                    }\n                }\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn bucket_test_extend_range_hole() {\n        let buckets = vec![(10f64..20f64).into(), (30f64..40f64).into()];\n        let collector = get_collector_from_ranges(buckets, ColumnType::F64);\n\n        let buckets = collector.parent_buckets[0].clone();\n        assert_eq!(buckets[0].range.start, u64::MIN);\n        assert_eq!(buckets[0].range.end, 10f64.to_u64());\n        assert_eq!(buckets[1].range.start, 10f64.to_u64());\n        assert_eq!(buckets[1].range.end, 20f64.to_u64());\n        // Added bucket to fill hole\n        assert_eq!(buckets[2].range.start, 20f64.to_u64());\n        assert_eq!(buckets[2].range.end, 30f64.to_u64());\n        assert_eq!(buckets[3].range.start, 30f64.to_u64());\n        assert_eq!(buckets[3].range.end, 40f64.to_u64());\n    }\n\n    #[test]\n    fn bucket_test_range_conversion_special_case() {\n        // the monotonic conversion between f64 and u64, does not map f64::MIN.to_u64() ==\n        // u64::MIN, but the into trait converts f64::MIN/MAX to None\n        let buckets = vec![\n            (f64::MIN..10f64).into(),\n            (10f64..20f64).into(),\n            (20f64..f64::MAX).into(),\n        ];\n        let collector = get_collector_from_ranges(buckets, ColumnType::F64);\n\n        let buckets = collector.parent_buckets[0].clone();\n        assert_eq!(buckets[0].range.start, u64::MIN);\n        assert_eq!(buckets[0].range.end, 10f64.to_u64());\n        assert_eq!(buckets[1].range.start, 10f64.to_u64());\n        assert_eq!(buckets[1].range.end, 20f64.to_u64());\n        assert_eq!(buckets[2].range.start, 20f64.to_u64());\n        assert_eq!(buckets[2].range.end, u64::MAX);\n        assert_eq!(buckets.len(), 3);\n    }\n\n    #[test]\n    fn bucket_range_test_negative_vals() {\n        let buckets = vec![(-10f64..-1f64).into()];\n        let collector = get_collector_from_ranges(buckets, ColumnType::F64);\n\n        let buckets = collector.parent_buckets[0].clone();\n        assert_eq!(&buckets[0].bucket.key.to_string(), \"*--10\");\n        assert_eq!(&buckets[buckets.len() - 1].bucket.key.to_string(), \"-1-*\");\n    }\n    #[test]\n    fn bucket_range_test_positive_vals() {\n        let buckets = vec![(0f64..10f64).into()];\n        let collector = get_collector_from_ranges(buckets, ColumnType::F64);\n\n        let buckets = collector.parent_buckets[0].clone();\n        assert_eq!(&buckets[0].bucket.key.to_string(), \"*-0\");\n        assert_eq!(&buckets[buckets.len() - 1].bucket.key.to_string(), \"10-*\");\n    }\n\n    #[test]\n    fn range_binary_search_test_u64() {\n        let check_ranges = |ranges: Vec<RangeAggregationRange>| {\n            let collector = get_collector_from_ranges(ranges, ColumnType::U64);\n            let search = |val: u64| get_bucket_pos(val, &collector.parent_buckets[0]);\n\n            assert_eq!(search(u64::MIN), 0);\n            assert_eq!(search(9), 0);\n            assert_eq!(search(10), 1);\n            assert_eq!(search(11), 1);\n            assert_eq!(search(99), 1);\n            assert_eq!(search(100), 2);\n            assert_eq!(search(u64::MAX - 1), 2); // Since the end range is never included, the max\n                                                 // value\n        };\n\n        let ranges = vec![(10.0..100.0).into()];\n        check_ranges(ranges);\n\n        let ranges = vec![\n            RangeAggregationRange {\n                key: None,\n                to: Some(10.0),\n                from: None,\n            },\n            (10.0..100.0).into(),\n        ];\n        check_ranges(ranges);\n\n        let ranges = vec![\n            RangeAggregationRange {\n                key: None,\n                to: Some(10.0),\n                from: None,\n            },\n            (10.0..100.0).into(),\n            RangeAggregationRange {\n                key: None,\n                to: None,\n                from: Some(100.0),\n            },\n        ];\n        check_ranges(ranges);\n    }\n\n    #[test]\n    fn range_binary_search_test_f64() {\n        let ranges = vec![(10.0..100.0).into()];\n\n        let collector = get_collector_from_ranges(ranges, ColumnType::F64);\n        let search = |val: u64| get_bucket_pos(val, &collector.parent_buckets[0]);\n\n        assert_eq!(search(u64::MIN), 0);\n        assert_eq!(search(9f64.to_u64()), 0);\n        assert_eq!(search(10f64.to_u64()), 1);\n        assert_eq!(search(11f64.to_u64()), 1);\n        assert_eq!(search(99f64.to_u64()), 1);\n        assert_eq!(search(100f64.to_u64()), 2);\n        assert_eq!(search(u64::MAX - 1), 2); // Since the end range is never included,\n                                             // the max value\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/term_agg.rs",
    "content": "use std::fmt::Debug;\nuse std::io;\nuse std::net::Ipv6Addr;\n\nuse columnar::column_values::CompactSpaceU64Accessor;\nuse columnar::{\n    Column, ColumnType, Dictionary, MonotonicallyMappableToU128, MonotonicallyMappableToU64,\n    NumericalValue, StrColumn,\n};\nuse common::{BitSet, TinySet};\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse super::{CustomOrder, Order, OrderTarget};\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::agg_limits::MemoryConsumption;\nuse crate::aggregation::agg_req::Aggregations;\nuse crate::aggregation::cached_sub_aggs::{\n    CachedSubAggs, HighCardSubAggCache, LowCardCachedSubAggs, LowCardSubAggCache, SubAggCache,\n};\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateBucketResult,\n    IntermediateKey, IntermediateTermBucketEntry, IntermediateTermBucketResult,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::{format_date, BucketId, Key};\nuse crate::error::DataCorruption;\nuse crate::TantivyError;\n\n/// Contains all information required by the SegmentTermCollector to perform the\n/// terms aggregation on a segment.\n#[derive(Debug, Clone)]\npub struct TermsAggReqData {\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n    /// The type of the column.\n    pub column_type: ColumnType,\n    /// The string dictionary column if the field is of type text.\n    pub str_dict_column: Option<StrColumn>,\n    /// The missing value as u64 value.\n    pub missing_value_for_accessor: Option<u64>,\n    /// Used to build the correct nested result when we have an empty result.\n    pub sug_aggregations: Aggregations,\n    /// The name of the aggregation.\n    pub name: String,\n    /// The normalized term aggregation request.\n    pub req: TermsAggregationInternal,\n    /// Preloaded allowed term ords (string columns only). If set, only ords present are collected.\n    pub allowed_term_ids: Option<BitSet>,\n    /// True if this terms aggregation is at the top level of the aggregation tree (not nested).\n    pub is_top_level: bool,\n}\n\nimpl TermsAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n            + std::mem::size_of::<TermsAggregationInternal>()\n            + self\n                .allowed_term_ids\n                .as_ref()\n                .map(|bs| bs.len() / 8)\n                .unwrap_or(0)\n    }\n}\n\n/// Creates a bucket for every unique term and counts the number of occurrences.\n/// Note that doc_count in the response buckets equals term count here.\n///\n/// If the text is untokenized and single value, that means one term per document and therefore it\n/// is in fact doc count.\n///\n/// ## Prerequisite\n/// Term aggregations work only on [fast fields](`crate::fastfield`) of type `u64`, `f64`, `i64` and\n/// text.\n///\n/// ## Document count error\n/// To improve performance, results from one segment are cut off at `segment_size`. On a index with\n/// a single segment this is fine. When combining results of multiple segments, terms that\n/// don't make it in the top n of a shard increase the theoretical upper bound error by lowest\n/// term-count.\n///\n/// Even with a larger `segment_size` value, doc_count values for a terms aggregation may be\n/// approximate. As a result, any sub-aggregations on the terms aggregation may also be approximate.\n/// `sum_other_doc_count` is the number of documents that didn’t make it into the top size\n/// terms. If this is greater than 0, you can be sure that the terms agg had to throw away some\n/// buckets, either because they didn’t fit into size on the root node or they didn’t fit into\n/// `segment_size` on the segment node.\n///\n/// ## Per bucket document count error\n/// If you set the `show_term_doc_count_error` parameter to true, the terms aggregation will include\n/// doc_count_error_upper_bound, which is an upper bound to the error on the doc_count returned by\n/// each segment. It’s the sum of the size of the largest bucket on each segment that didn’t fit\n/// into segment_size.\n///\n/// Result type is [`BucketResult`](crate::aggregation::agg_result::BucketResult) with\n/// [`BucketEntry`](crate::aggregation::agg_result::BucketEntry) on the\n/// `AggregationCollector`.\n///\n/// Result type is\n/// [`IntermediateBucketResult`](crate::aggregation::intermediate_agg_result::IntermediateBucketResult) with\n/// [`IntermediateTermBucketEntry`](crate::aggregation::intermediate_agg_result::IntermediateTermBucketEntry) on the\n/// `DistributedAggregationCollector`.\n///\n/// # Limitations/Compatibility\n///\n/// Each segment returns up to [segment_size](TermsAggregation::segment_size) results. This\n/// differences to elasticsearch behaviour.\n///\n/// # Request JSON Format\n/// ```json\n/// {\n///     \"genres\": {\n///         \"terms\":{ \"field\": \"genre\" }\n///     }\n/// }\n/// ```\n///\n/// /// # Response JSON Format\n/// ```json\n/// {\n///     ...\n///     \"aggregations\": {\n///         \"genres\": {\n///             \"doc_count_error_upper_bound\": 0,\n///             \"sum_other_doc_count\": 0,\n///             \"buckets\": [\n///                 { \"key\": \"drumnbass\", \"doc_count\": 6 },\n///                 { \"key\": \"raggae\", \"doc_count\": 4 },\n///                 { \"key\": \"jazz\", \"doc_count\": 2 }\n///             ]\n///         }\n///     }\n/// }\n/// ```\n\n#[derive(Clone, Debug, PartialEq)]\npub enum IncludeExcludeParam {\n    /// A single string pattern is treated as regex.\n    Regex(String),\n    /// An array of strings is treated as exact values.\n    Values(Vec<String>),\n}\n\nimpl Serialize for IncludeExcludeParam {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: serde::Serializer {\n        match self {\n            IncludeExcludeParam::Regex(s) => serializer.serialize_str(s),\n            IncludeExcludeParam::Values(v) => v.serialize(serializer),\n        }\n    }\n}\n\n// Custom deserializer to accept either a single string (regex) or an array of strings (values).\nimpl<'de> Deserialize<'de> for IncludeExcludeParam {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: serde::Deserializer<'de> {\n        use serde::de::{self, SeqAccess, Visitor};\n        struct IncludeExcludeVisitor;\n\n        impl<'de> Visitor<'de> for IncludeExcludeVisitor {\n            type Value = IncludeExcludeParam;\n\n            fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {\n                formatter.write_str(\"a string (regex) or an array of strings\")\n            }\n\n            fn visit_str<E>(self, v: &str) -> Result<Self::Value, E>\n            where E: de::Error {\n                Ok(IncludeExcludeParam::Regex(v.to_string()))\n            }\n\n            fn visit_borrowed_str<E>(self, v: &'de str) -> Result<Self::Value, E>\n            where E: de::Error {\n                Ok(IncludeExcludeParam::Regex(v.to_string()))\n            }\n\n            fn visit_string<E>(self, v: String) -> Result<Self::Value, E>\n            where E: de::Error {\n                Ok(IncludeExcludeParam::Regex(v))\n            }\n\n            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n            where A: SeqAccess<'de> {\n                let mut values: Vec<String> = Vec::new();\n                while let Some(elem) = seq.next_element::<String>()? {\n                    values.push(elem);\n                }\n                Ok(IncludeExcludeParam::Values(values))\n            }\n        }\n\n        deserializer.deserialize_any(IncludeExcludeVisitor)\n    }\n}\n\n/// The terms aggregation allows you to group documents by unique values of a field.\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\npub struct TermsAggregation {\n    /// The field to aggregate on.\n    pub field: String,\n    /// By default, the top 10 terms with the most documents are returned.\n    /// Larger values for size are more expensive.\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub size: Option<u32>,\n\n    /// To get more accurate results, we fetch more than `size` from each segment.\n    ///\n    /// Increasing this value is will increase the cost for more accuracy.\n    ///\n    /// Defaults to 10 * size.\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    #[serde(alias = \"shard_size\")]\n    #[serde(alias = \"split_size\")]\n    pub segment_size: Option<u32>,\n\n    /// If you set the `show_term_doc_count_error` parameter to true, the terms aggregation will\n    /// include doc_count_error_upper_bound, which is an upper bound to the error on the\n    /// doc_count returned by each shard. It’s the sum of the size of the largest bucket on\n    /// each segment that didn’t fit into `shard_size`.\n    ///\n    /// Defaults to true when ordering by count desc.\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub show_term_doc_count_error: Option<bool>,\n\n    /// Filter all terms that are lower than `min_doc_count`. Defaults to 1.\n    ///\n    /// **Expensive**: When set to 0, this will return all terms in the field.\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub min_doc_count: Option<u64>,\n\n    /// Set the order. `String` is here a target, which is either \"_count\", \"_key\", or the name of\n    /// a metric sub_aggregation.\n    ///\n    /// Single value metrics like average can be addressed by its name.\n    /// Multi value metrics like stats are required to address their field by name e.g.\n    /// \"stats.avg\"\n    ///\n    /// Examples in JSON format:\n    /// { \"_count\": \"asc\" }\n    /// { \"_key\": \"asc\" }\n    /// { \"average_price\": \"asc\" }\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub order: Option<CustomOrder>,\n\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"missing\": \"NO_DATA\" }\n    ///\n    /// # Internal\n    ///\n    /// Internally, `missing` requires some specialized handling in some scenarios.\n    ///\n    /// Simple Case:\n    /// In the simplest case, we can just put the missing value in the termmap and use that. In\n    /// case of text we put a special u64::MAX and replace it at the end with the actual\n    /// missing value, when loading the text.\n    /// Special Case 1:\n    /// If we have multiple columns on one field, we need to have a union on the indices on both\n    /// columns, to find docids without a value. That requires a special missing aggregation.\n    /// Special Case 2: if the key is of type text and the column is numerical, we also need to use\n    /// the special missing aggregation, since there is no mechanism in the numerical column to\n    /// add text.\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub missing: Option<Key>,\n\n    /// Include terms by either regex (single string) or exact values (array).\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub include: Option<IncludeExcludeParam>,\n    /// Exclude terms by either regex (single string) or exact values (array).\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub exclude: Option<IncludeExcludeParam>,\n}\n\n/// Same as TermsAggregation, but with populated defaults.\n#[derive(Clone, Debug, PartialEq)]\npub struct TermsAggregationInternal {\n    /// The field to aggregate on.\n    pub field: String,\n    /// By default, the top 10 terms with the most documents are returned.\n    /// Larger values for size are more expensive.\n    ///\n    /// Defaults to 10.\n    pub size: u32,\n\n    /// If you set the `show_term_doc_count_error` parameter to true, the terms aggregation will\n    /// include doc_count_error_upper_bound, which is an upper bound to the error on the\n    /// doc_count returned by each shard. It’s the sum of the size of the largest bucket on\n    /// each segment that didn’t fit into `segment_size`.\n    pub show_term_doc_count_error: bool,\n\n    /// The get more accurate results, we fetch more than `size` from each segment.\n    ///\n    /// Increasing this value is will increase the cost for more accuracy.\n    pub segment_size: u32,\n\n    /// Filter all terms that are lower than `min_doc_count`. Defaults to 1.\n    ///\n    /// *Expensive*: When set to 0, this will return all terms in the field.\n    pub min_doc_count: u64,\n\n    /// Set the order. `String` is here a target, which is either \"_count\", \"_key\", or the name of\n    /// a metric sub_aggregation.\n    pub order: CustomOrder,\n\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    pub missing: Option<Key>,\n}\n\nimpl TermsAggregationInternal {\n    pub(crate) fn from_req(req: &TermsAggregation) -> Self {\n        let size = req.size.unwrap_or(10);\n\n        let mut segment_size = req.segment_size.unwrap_or(size * 10);\n\n        let order = req.order.clone().unwrap_or_default();\n        segment_size = segment_size.max(size);\n        TermsAggregationInternal {\n            field: req.field.to_string(),\n            size,\n            segment_size,\n            show_term_doc_count_error: req\n                .show_term_doc_count_error\n                .unwrap_or_else(|| order == CustomOrder::default()),\n            min_doc_count: req.min_doc_count.unwrap_or(1),\n            order,\n            missing: req.missing.clone(),\n        }\n    }\n}\n\n/// The treshold for maximum number of terms to use a Vec-backed bucket storage.\n/// TODO: Benchmark to validate the threshold\npub const MAX_NUM_TERMS_FOR_VEC: u64 = 100;\n\n/// Build a concrete `SegmentTermCollector` with either a Vec- or HashMap-backed\n/// bucket storage, depending on the column type and aggregation level.\npub(crate) fn build_segment_term_collector(\n    req_data: &mut AggregationsSegmentCtx,\n    node: &AggRefNode,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    let terms_req_data = req_data.get_term_req_data(node.idx_in_req_data).clone();\n    let column_type = terms_req_data.column_type;\n\n    if column_type == ColumnType::Bytes {\n        return Err(TantivyError::InvalidArgument(format!(\n            \"terms aggregation is not supported for column type {column_type:?}\"\n        )));\n    }\n\n    // Validate sub aggregation exists when ordering by sub-aggregation.\n    {\n        if let OrderTarget::SubAggregation(sub_agg_name) = &terms_req_data.req.order.target {\n            let (agg_name, _agg_property) = get_agg_name_and_property(sub_agg_name);\n\n            node.get_sub_agg(agg_name, &req_data.per_request)\n                .ok_or_else(|| {\n                    TantivyError::InvalidArgument(format!(\n                        \"could not find aggregation with name {agg_name} in metric \\\n                         sub_aggregations\"\n                    ))\n                })?;\n        }\n    }\n\n    // Build sub-aggregation blueprint if there are children.\n    let has_sub_aggregations = !node.children.is_empty();\n\n    // TODO: A better metric instead of is_top_level would be the number of buckets expected.\n    // E.g. If term agg is not top level, but the parent is a bucket agg with less than 10 buckets,\n    // we can still use Vec.\n    let is_top_level = terms_req_data.is_top_level;\n\n    // Let's see if we can use a vec to aggregate our data\n    // instead of a hashmap.\n    let col_max_value = terms_req_data.accessor.max_value();\n    let max_term_id: u64 =\n        col_max_value.max(terms_req_data.missing_value_for_accessor.unwrap_or(0u64));\n\n    let sub_agg_collector = if has_sub_aggregations {\n        Some(build_segment_agg_collectors(req_data, &node.children)?)\n    } else {\n        None\n    };\n\n    let mut bucket_id_provider = BucketIdProvider::default();\n    // Decide which bucket storage is best suited for this aggregation.\n    if is_top_level && max_term_id < MAX_NUM_TERMS_FOR_VEC && !has_sub_aggregations {\n        let term_buckets = VecTermBucketsNoAgg::new(max_term_id + 1, &mut bucket_id_provider);\n        let collector: SegmentTermCollector<_, HighCardSubAggCache> = SegmentTermCollector {\n            parent_buckets: vec![term_buckets],\n            sub_agg: None,\n            bucket_id_provider,\n            max_term_id,\n            terms_req_data,\n        };\n        Ok(Box::new(collector))\n    } else if is_top_level && max_term_id < MAX_NUM_TERMS_FOR_VEC {\n        let term_buckets = VecTermBuckets::new(max_term_id + 1, &mut bucket_id_provider);\n        let sub_agg = sub_agg_collector.map(LowCardCachedSubAggs::new);\n        let collector: SegmentTermCollector<_, LowCardSubAggCache> = SegmentTermCollector {\n            parent_buckets: vec![term_buckets],\n            sub_agg,\n            bucket_id_provider,\n            max_term_id,\n            terms_req_data,\n        };\n        Ok(Box::new(collector))\n    } else if max_term_id < 8_000_000 && is_top_level {\n        let term_buckets: PagedTermMap =\n            PagedTermMap::new(max_term_id + 1, &mut bucket_id_provider);\n        // Build sub-aggregation blueprint (flat pairs)\n        let sub_agg = sub_agg_collector.map(CachedSubAggs::new);\n        let collector: SegmentTermCollector<PagedTermMap, HighCardSubAggCache> =\n            SegmentTermCollector {\n                parent_buckets: vec![term_buckets],\n                sub_agg,\n                bucket_id_provider,\n                max_term_id,\n                terms_req_data,\n            };\n        Ok(Box::new(collector))\n    } else {\n        let term_buckets: HashMapTermBuckets = HashMapTermBuckets::default();\n        // Build sub-aggregation blueprint (flat pairs)\n        let sub_agg = sub_agg_collector.map(CachedSubAggs::new);\n        let collector: SegmentTermCollector<HashMapTermBuckets, HighCardSubAggCache> =\n            SegmentTermCollector {\n                parent_buckets: vec![term_buckets],\n                sub_agg,\n                bucket_id_provider,\n                max_term_id,\n                terms_req_data,\n            };\n        Ok(Box::new(collector))\n    }\n}\n\n#[derive(Debug, Clone, Copy, Default)]\nstruct Bucket {\n    pub count: u32,\n    pub bucket_id: BucketId,\n}\n\nimpl Bucket {\n    #[inline(always)]\n    fn new(bucket_id: BucketId) -> Self {\n        Self {\n            count: 0,\n            bucket_id,\n        }\n    }\n}\n\n/// Abstraction over the storage used for term buckets (counts only).\ntrait TermAggregationMap: Clone + Debug + 'static {\n    /// Create a new instance with a strict upper bound on term ids.\n    fn new(max_term_id: u64, bucket_id_provider: &mut BucketIdProvider) -> Self;\n\n    /// Estimate the memory consumption of this struct in bytes.\n    fn get_memory_consumption(&self) -> usize;\n\n    /// Increments the count and returns the bucket_id associated to a given term_id.\n    fn term_entry(&mut self, term_id: u64, bucket_id_provider: &mut BucketIdProvider) -> BucketId;\n\n    /// Returns the term aggregation as a vector of (term_id, bucket) pairs,\n    /// in any order.\n    fn into_vec(self) -> Vec<(u64, Bucket)>;\n}\n\n#[derive(Clone, Debug)]\nstruct HashMapTermBuckets {\n    bucket_map: FxHashMap<u64, Bucket>,\n}\n\nimpl Default for HashMapTermBuckets {\n    #[inline(always)]\n    fn default() -> Self {\n        Self {\n            bucket_map: FxHashMap::default(),\n        }\n    }\n}\n\nconst PAGE_SHIFT: usize = 10;\nconst PAGE_SIZE: usize = 1 << PAGE_SHIFT; // 1024\nconst PAGE_MASK: usize = PAGE_SIZE - 1;\nconst BITMASK_LEN: usize = PAGE_SIZE / 64;\n\n#[derive(Clone, Debug)]\nstruct Page {\n    /// Bitmask indicating which offsets are present.\n    /// It is chunked into TinySet words.\n    presence: [TinySet; BITMASK_LEN],\n    data: [Bucket; PAGE_SIZE],\n}\n\nimpl Page {\n    fn new() -> Self {\n        Self {\n            presence: [TinySet::empty(); BITMASK_LEN],\n            data: [Bucket::default(); PAGE_SIZE],\n        }\n    }\n\n    #[inline]\n    fn is_set(&self, offset: usize) -> bool {\n        let bucket_idx = offset / 64;\n        let bit_idx = offset % 64;\n        self.presence[bucket_idx].contains(bit_idx as u32)\n    }\n\n    #[inline]\n    fn set_present(&mut self, offset: usize) {\n        let bucket_idx = offset / 64;\n        let bit_idx = offset % 64;\n        self.presence[bucket_idx].insert_mut(bit_idx as u32);\n    }\n\n    // Flattened iteration logic\n    fn collect_items(&self, base_term_id: u64, result: &mut Vec<(u64, Bucket)>) {\n        for (bucket_pos, &tiny_set) in self.presence.iter().enumerate() {\n            let base_offset = bucket_pos * 64;\n\n            for bit in tiny_set.into_iter() {\n                let offset = base_offset + bit as usize;\n                result.push((base_term_id + offset as u64, self.data[offset]));\n            }\n        }\n    }\n}\n\n/// A paged term map implementation for moderate sized term id sets.\n/// Uses a fixed size vector of pages, each page containing a fixed size array of buckets.\n///\n/// Each page covers a range of term ids. Pages are allocated on demand.\n/// This implementation is more memory efficient than a full Vec for high cardinality term id sets,\n///\n/// It has a fixed cost of `num_pages * 8 bytes` for the page directory.\n/// For 1 million terms, this is 8 * 1024 = 8KB.\n///\n/// Note that for nested aggregations we create one TermAggregationMap per parent bucket.\n/// For example, with 100 parent buckets and 1 million terms, this is 800KB overhead for the page\n/// directories only. Therefore, this implementation is only enabled for top-level aggregations\n/// TODO: pass expected number of buckets from parent instead of strict is_top_level flag.\n#[derive(Clone, Debug, Default)]\nstruct PagedTermMap {\n    // Fixed size vector based on max_term_id\n    pages: Vec<Option<Box<Page>>>,\n    mem_usage: usize,\n}\n\nimpl PagedTermMap {}\n\nimpl TermAggregationMap for PagedTermMap {\n    #[inline]\n    fn get_memory_consumption(&self) -> usize {\n        self.mem_usage + std::mem::size_of::<Self>()\n    }\n\n    #[inline]\n    fn term_entry(&mut self, term_id: u64, bucket_id_provider: &mut BucketIdProvider) -> BucketId {\n        let term_id = term_id as usize;\n        let page_idx = term_id >> PAGE_SHIFT;\n        let offset = term_id & PAGE_MASK;\n\n        // This panics if term_id > max_term_id\n        let page = match &mut self.pages[page_idx] {\n            Some(p) => p,\n            None => {\n                let new_page = Box::new(Page::new());\n                self.mem_usage += std::mem::size_of::<Page>();\n                self.pages[page_idx] = Some(new_page);\n                self.pages[page_idx].as_mut().unwrap()\n            }\n        };\n\n        if page.is_set(offset) {\n            let bucket = &mut page.data[offset];\n            bucket.count += 1;\n            bucket.bucket_id\n        } else {\n            let new_id = bucket_id_provider.next_bucket_id();\n            page.data[offset] = Bucket {\n                count: 1,\n                bucket_id: new_id,\n            };\n            page.set_present(offset);\n            new_id\n        }\n    }\n\n    fn into_vec(self) -> Vec<(u64, Bucket)> {\n        // estimate 16 entries per non-empty page\n        let estimated_count = self.pages.iter().filter(|p| p.is_some()).count() * 16;\n        let mut result = Vec::with_capacity(estimated_count);\n\n        for (i, page_opt) in self.pages.into_iter().enumerate() {\n            if let Some(page) = page_opt {\n                let base_term_id = (i << PAGE_SHIFT) as u64;\n                page.collect_items(base_term_id, &mut result);\n            }\n        }\n        result\n    }\n\n    /// Initialize with a strict upper bound.\n    /// Panics if you try to insert a term_id > max_term_id.\n    fn new(max_term_id: u64, _bucket_id_provider: &mut BucketIdProvider) -> Self {\n        let max_page_idx = (max_term_id as usize) >> PAGE_SHIFT;\n        let num_pages = max_page_idx + 1;\n\n        // Pre-allocate the directory (pointers only, not the heavy pages)\n        // Memory cost: num_pages * 8 bytes\n        let pages = vec![None; num_pages];\n\n        let mem_usage = pages.capacity() * std::mem::size_of::<Option<Box<Page>>>();\n\n        Self { pages, mem_usage }\n    }\n}\n\nimpl TermAggregationMap for HashMapTermBuckets {\n    #[inline]\n    fn get_memory_consumption(&self) -> usize {\n        self.bucket_map.memory_consumption()\n    }\n\n    #[inline(always)]\n    fn term_entry(&mut self, term_id: u64, bucket_id_provider: &mut BucketIdProvider) -> BucketId {\n        let bucket = self\n            .bucket_map\n            .entry(term_id)\n            .or_insert_with(|| Bucket::new(bucket_id_provider.next_bucket_id()));\n        bucket.count += 1;\n        bucket.bucket_id\n    }\n\n    fn into_vec(self) -> Vec<(u64, Bucket)> {\n        self.bucket_map.into_iter().collect()\n    }\n\n    #[inline]\n    fn new(_max_term_id: u64, _bucket_id_provider: &mut BucketIdProvider) -> Self {\n        Self::default()\n    }\n}\n\n/// An optimized term map implementation for a compact set of term ordinals.\n#[derive(Clone, Debug)]\nstruct VecTermBucketsNoAgg {\n    buckets: Vec<u32>,\n}\n\nimpl TermAggregationMap for VecTermBucketsNoAgg {\n    /// Estimate the memory consumption of this struct in bytes.\n    fn get_memory_consumption(&self) -> usize {\n        // We do not include `std::mem::size_of::<Self>()`\n        // It is already measure by the parent aggregation.\n        //\n        self.buckets.capacity() * std::mem::size_of::<u32>()\n    }\n\n    /// Add an occurrence of the given term id.\n    #[inline(always)]\n    fn term_entry(&mut self, term_id: u64, _bucket_id_provider: &mut BucketIdProvider) -> BucketId {\n        let term_id_usize = term_id as usize;\n        debug_assert!(\n            term_id_usize < self.buckets.len(),\n            \"term_id {} out of bounds for VecTermBuckets (len={})\",\n            term_id,\n            self.buckets.len()\n        );\n        let count = unsafe { self.buckets.get_unchecked_mut(term_id_usize) };\n        *count += 1;\n        0 // unused\n    }\n\n    fn into_vec(self) -> Vec<(u64, Bucket)> {\n        self.buckets\n            .into_iter()\n            .enumerate()\n            .filter(|(_term_id, count)| *count > 0)\n            .map(|(term_id, count)| {\n                (\n                    term_id as u64,\n                    Bucket {\n                        count,\n                        bucket_id: 0, // unused, there are no sub-aggregations\n                    },\n                )\n            })\n            .collect()\n    }\n\n    fn new(num_terms: u64, _bucket_id_provider: &mut BucketIdProvider) -> Self {\n        Self {\n            buckets: std::iter::repeat_with(|| 0)\n                .take(num_terms as usize)\n                .collect(),\n        }\n    }\n}\n\n/// An optimized term map implementation for a compact set of term ordinals.\n#[derive(Clone, Debug)]\nstruct VecTermBuckets {\n    buckets: Vec<Bucket>,\n}\n\nimpl TermAggregationMap for VecTermBuckets {\n    /// Estimate the memory consumption of this struct in bytes.\n    fn get_memory_consumption(&self) -> usize {\n        // We do not include `std::mem::size_of::<Self>()`\n        // It is already measure by the parent aggregation.\n        //\n        // The root aggregation mem size is not measure but we do not care.\n        self.buckets.capacity() * std::mem::size_of::<Bucket>()\n    }\n\n    /// Add an occurrence of the given term id.\n    #[inline(always)]\n    fn term_entry(&mut self, term_id: u64, _bucket_id_provider: &mut BucketIdProvider) -> BucketId {\n        let term_id_usize = term_id as usize;\n        debug_assert!(\n            term_id_usize < self.buckets.len(),\n            \"term_id {} out of bounds for VecTermBuckets (len={})\",\n            term_id,\n            self.buckets.len()\n        );\n        let bucket = unsafe { self.buckets.get_unchecked_mut(term_id_usize) };\n        bucket.count += 1;\n        bucket.bucket_id\n    }\n\n    fn into_vec(self) -> Vec<(u64, Bucket)> {\n        self.buckets\n            .into_iter()\n            .enumerate()\n            .filter(|(_, bucket)| bucket.count > 0)\n            .map(|(term_id, bucket)| (term_id as u64, bucket))\n            .collect()\n    }\n\n    fn new(num_terms: u64, bucket_id_provider: &mut BucketIdProvider) -> Self {\n        VecTermBuckets {\n            buckets: std::iter::repeat_with(|| Bucket::new(bucket_id_provider.next_bucket_id()))\n                .take(num_terms as usize)\n                .collect(),\n        }\n    }\n}\n\n/// The collector puts values from the fast field into the correct buckets and does a conversion to\n/// the correct datatype.\n#[derive(Debug)]\nstruct SegmentTermCollector<TermMap: TermAggregationMap, C: SubAggCache> {\n    /// The buckets containing the aggregation data.\n    parent_buckets: Vec<TermMap>,\n    sub_agg: Option<CachedSubAggs<C>>,\n    bucket_id_provider: BucketIdProvider,\n    max_term_id: u64,\n    terms_req_data: TermsAggReqData,\n}\n\npub(crate) fn get_agg_name_and_property(name: &str) -> (&str, &str) {\n    let (agg_name, agg_property) = name.split_once('.').unwrap_or((name, \"\"));\n    (agg_name, agg_property)\n}\n\nimpl<TermMap: TermAggregationMap, C: SubAggCache> SegmentAggregationCollector\n    for SegmentTermCollector<TermMap, C>\n{\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        bucket: BucketId,\n    ) -> crate::Result<()> {\n        // TODO: avoid prepare_max_bucket here and handle empty buckets.\n        self.prepare_max_bucket(bucket, agg_data)?;\n        let bucket = std::mem::replace(\n            &mut self.parent_buckets[bucket as usize],\n            TermMap::new(0, &mut self.bucket_id_provider),\n        );\n        let term_req = &self.terms_req_data;\n        let name = term_req.name.clone();\n\n        let bucket =\n            Self::into_intermediate_bucket_result(term_req, &mut self.sub_agg, bucket, agg_data)?;\n        results.push(name, IntermediateAggregationResult::Bucket(bucket))?;\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let mem_pre = self.get_memory_consumption();\n\n        let req_data = &mut self.terms_req_data;\n\n        agg_data.column_block_accessor.fetch_block_with_missing(\n            docs,\n            &req_data.accessor,\n            req_data.missing_value_for_accessor,\n        );\n\n        if let Some(sub_agg) = &mut self.sub_agg {\n            let term_buckets = &mut self.parent_buckets[parent_bucket_id as usize];\n            let it = agg_data\n                .column_block_accessor\n                .iter_docid_vals(docs, &req_data.accessor);\n            if let Some(allowed_bs) = req_data.allowed_term_ids.as_ref() {\n                let it = it.filter(move |&(_doc, term_id)| allowed_bs.contains(term_id as u32));\n                Self::collect_terms_with_docs(\n                    it,\n                    term_buckets,\n                    &mut self.bucket_id_provider,\n                    sub_agg,\n                );\n            } else {\n                Self::collect_terms_with_docs(\n                    it,\n                    term_buckets,\n                    &mut self.bucket_id_provider,\n                    sub_agg,\n                );\n            }\n        } else {\n            let term_buckets = &mut self.parent_buckets[parent_bucket_id as usize];\n            let it = agg_data.column_block_accessor.iter_vals();\n            if let Some(allowed_bs) = req_data.allowed_term_ids.as_ref() {\n                let it = it.filter(move |&term_id| allowed_bs.contains(term_id as u32));\n                Self::collect_terms(it, term_buckets, &mut self.bucket_id_provider);\n            } else {\n                Self::collect_terms(it, term_buckets, &mut self.bucket_id_provider);\n            }\n        }\n\n        let mem_delta = self.get_memory_consumption() - mem_pre;\n        if mem_delta > 0 {\n            agg_data\n                .context\n                .limits\n                .add_memory_consumed(mem_delta as u64)?;\n        }\n        if let Some(sub_agg) = &mut self.sub_agg {\n            sub_agg.check_flush_local(agg_data)?;\n        }\n\n        Ok(())\n    }\n\n    #[inline]\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(sub_agg) = &mut self.sub_agg {\n            sub_agg.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.parent_buckets.len() <= max_bucket as usize {\n            let term_buckets: TermMap =\n                TermMap::new(self.max_term_id, &mut self.bucket_id_provider);\n            self.parent_buckets.push(term_buckets);\n        }\n        Ok(())\n    }\n}\n\n/// Missing value are represented as a sentinel value in the column.\n///\n/// This function extracts the missing value from the entries vector,\n/// computes the intermediate key, and returns it the key and the bucket\n/// in an Option.\nfn extract_missing_value<T>(\n    entries: &mut Vec<(u64, T)>,\n    term_req: &TermsAggReqData,\n) -> Option<(IntermediateKey, T)> {\n    let missing_sentinel = term_req.missing_value_for_accessor?;\n    let missing_value_entry_pos = entries\n        .iter()\n        .position(|(term_id, _)| *term_id == missing_sentinel)?;\n    let (_term_id, bucket) = entries.swap_remove(missing_value_entry_pos);\n    let missing_key = term_req.req.missing.as_ref()?;\n    let key = match missing_key {\n        Key::Str(missing) => IntermediateKey::Str(missing.clone()),\n        Key::F64(val) => IntermediateKey::F64(*val),\n        Key::U64(val) => IntermediateKey::U64(*val),\n        Key::I64(val) => IntermediateKey::I64(*val),\n    };\n    Some((key, bucket))\n}\n\nimpl<TermMap, C> SegmentTermCollector<TermMap, C>\nwhere\n    TermMap: TermAggregationMap,\n    C: SubAggCache,\n{\n    fn get_memory_consumption(&self) -> usize {\n        self.parent_buckets\n            .iter()\n            .map(|b| b.get_memory_consumption())\n            .sum()\n    }\n\n    #[inline]\n    pub(crate) fn into_intermediate_bucket_result(\n        term_req: &TermsAggReqData,\n        sub_agg: &mut Option<CachedSubAggs<C>>,\n        term_buckets: TermMap,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<IntermediateBucketResult> {\n        let mut entries: Vec<(u64, Bucket)> = term_buckets.into_vec();\n\n        let order_by_sub_aggregation =\n            matches!(term_req.req.order.target, OrderTarget::SubAggregation(_));\n\n        match &term_req.req.order.target {\n            OrderTarget::Key => {\n                // We rely on the fact, that term ordinals match the order of the strings\n                // TODO: We could have a special collector, that keeps only TOP n results at any\n                // time.\n                if term_req.req.order.order == Order::Desc {\n                    entries.sort_unstable_by_key(|bucket| std::cmp::Reverse(bucket.0));\n                } else {\n                    entries.sort_unstable_by_key(|bucket| bucket.0);\n                }\n            }\n            OrderTarget::SubAggregation(_name) => {\n                // don't sort and cut off since it's hard to make assumptions on the quality of the\n                // results when cutting off du to unknown nature of the sub_aggregation (possible\n                // to check).\n            }\n            OrderTarget::Count => {\n                if term_req.req.order.order == Order::Desc {\n                    entries.sort_unstable_by_key(|bucket| std::cmp::Reverse(bucket.1.count));\n                } else {\n                    entries.sort_unstable_by_key(|bucket| bucket.1.count);\n                }\n            }\n        }\n\n        let (term_doc_count_before_cutoff, sum_other_doc_count) = if order_by_sub_aggregation {\n            (0, 0)\n        } else {\n            cut_off_buckets(&mut entries, term_req.req.segment_size as usize)\n        };\n\n        let mut dict: FxHashMap<IntermediateKey, IntermediateTermBucketEntry> = Default::default();\n        dict.reserve(entries.len());\n\n        let into_intermediate_bucket_entry =\n            |bucket: Bucket,\n             sub_agg: &mut Option<CachedSubAggs<C>>|\n             -> crate::Result<IntermediateTermBucketEntry> {\n                if let Some(sub_agg) = sub_agg {\n                    let mut sub_aggregation_res = IntermediateAggregationResults::default();\n                    sub_agg\n                        .get_sub_agg_collector()\n                        .add_intermediate_aggregation_result(\n                            agg_data,\n                            &mut sub_aggregation_res,\n                            bucket.bucket_id,\n                        )?;\n                    Ok(IntermediateTermBucketEntry {\n                        doc_count: bucket.count,\n                        sub_aggregation: sub_aggregation_res,\n                    })\n                } else {\n                    Ok(IntermediateTermBucketEntry {\n                        doc_count: bucket.count,\n                        sub_aggregation: Default::default(),\n                    })\n                }\n            };\n\n        if term_req.column_type == ColumnType::Str {\n            let fallback_dict = Dictionary::empty();\n            let term_dict = term_req\n                .str_dict_column\n                .as_ref()\n                .map(|el| el.dictionary())\n                .unwrap_or_else(|| &fallback_dict);\n\n            if let Some((intermediate_key, bucket)) = extract_missing_value(&mut entries, term_req)\n            {\n                let intermediate_entry = into_intermediate_bucket_entry(bucket, sub_agg)?;\n                dict.insert(intermediate_key, intermediate_entry);\n            }\n\n            // Sort by term ord\n            entries.sort_unstable_by_key(|bucket| bucket.0);\n\n            let (term_ids, buckets): (Vec<u64>, Vec<Bucket>) = entries.into_iter().unzip();\n            let mut buckets_it = buckets.into_iter();\n\n            term_dict.sorted_ords_to_term_cb(term_ids.into_iter(), |term| {\n                let bucket = buckets_it.next().unwrap();\n                let intermediate_entry =\n                    into_intermediate_bucket_entry(bucket, sub_agg).map_err(io::Error::other)?;\n                dict.insert(\n                    IntermediateKey::Str(\n                        String::from_utf8(term.to_vec()).expect(\"could not convert to String\"),\n                    ),\n                    intermediate_entry,\n                );\n                Ok(())\n            })?;\n\n            if term_req.req.min_doc_count == 0 {\n                // TODO: Handle rev streaming for descending sorting by keys\n                let mut stream = term_dict.stream()?;\n                let empty_sub_aggregation =\n                    IntermediateAggregationResults::empty_from_req(&term_req.sug_aggregations);\n                while stream.advance() {\n                    if dict.len() >= term_req.req.segment_size as usize {\n                        break;\n                    }\n\n                    // Respect allowed filters if present\n                    if let Some(allowed_bs) = term_req.allowed_term_ids.as_ref() {\n                        if !allowed_bs.contains(stream.term_ord() as u32) {\n                            continue;\n                        }\n                    }\n\n                    let key = IntermediateKey::Str(\n                        std::str::from_utf8(stream.key())\n                            .map_err(|utf8_err| DataCorruption::comment_only(utf8_err.to_string()))?\n                            .to_string(),\n                    );\n\n                    dict.entry(key.clone())\n                        .or_insert_with(|| IntermediateTermBucketEntry {\n                            doc_count: 0,\n                            sub_aggregation: empty_sub_aggregation.clone(),\n                        });\n                }\n            }\n        } else if term_req.column_type == ColumnType::DateTime {\n            for (val, doc_count) in entries {\n                let intermediate_entry = into_intermediate_bucket_entry(doc_count, sub_agg)?;\n                let val = i64::from_u64(val);\n                let date = format_date(val)?;\n                dict.insert(IntermediateKey::Str(date), intermediate_entry);\n            }\n        } else if term_req.column_type == ColumnType::Bool {\n            for (val, doc_count) in entries {\n                let intermediate_entry = into_intermediate_bucket_entry(doc_count, sub_agg)?;\n                let val = bool::from_u64(val);\n                dict.insert(IntermediateKey::Bool(val), intermediate_entry);\n            }\n        } else if term_req.column_type == ColumnType::IpAddr {\n            let compact_space_accessor = term_req\n                .accessor\n                .values\n                .clone()\n                .downcast_arc::<CompactSpaceU64Accessor>()\n                .map_err(|_| {\n                    TantivyError::AggregationError(\n                        crate::aggregation::AggregationError::InternalError(\n                            \"Type mismatch: Could not downcast to CompactSpaceU64Accessor\"\n                                .to_string(),\n                        ),\n                    )\n                })?;\n\n            for (val, doc_count) in entries {\n                let intermediate_entry = into_intermediate_bucket_entry(doc_count, sub_agg)?;\n                let val: u128 = compact_space_accessor.compact_to_u128(val as u32);\n                let val = Ipv6Addr::from_u128(val);\n                dict.insert(IntermediateKey::IpAddr(val), intermediate_entry);\n            }\n        } else {\n            for (val, doc_count) in entries {\n                let intermediate_entry = into_intermediate_bucket_entry(doc_count, sub_agg)?;\n                if term_req.column_type == ColumnType::U64 {\n                    dict.insert(IntermediateKey::U64(val), intermediate_entry);\n                } else if term_req.column_type == ColumnType::I64 {\n                    dict.insert(IntermediateKey::I64(i64::from_u64(val)), intermediate_entry);\n                } else {\n                    let val = f64::from_u64(val);\n                    let val: NumericalValue = val.into();\n\n                    match val.normalize() {\n                        NumericalValue::U64(val) => {\n                            dict.insert(IntermediateKey::U64(val), intermediate_entry);\n                        }\n                        NumericalValue::I64(val) => {\n                            dict.insert(IntermediateKey::I64(val), intermediate_entry);\n                        }\n                        NumericalValue::F64(val) => {\n                            dict.insert(IntermediateKey::F64(val), intermediate_entry);\n                        }\n                    }\n                };\n            }\n        };\n\n        Ok(IntermediateBucketResult::Terms {\n            buckets: IntermediateTermBucketResult {\n                entries: dict,\n                sum_other_doc_count,\n                doc_count_error_upper_bound: term_doc_count_before_cutoff,\n            },\n        })\n    }\n}\n\nimpl<TermMap: TermAggregationMap, C: SubAggCache> SegmentTermCollector<TermMap, C> {\n    #[inline]\n    fn collect_terms_with_docs(\n        iter: impl Iterator<Item = (crate::DocId, u64)>,\n        term_buckets: &mut TermMap,\n        bucket_id_provider: &mut BucketIdProvider,\n        sub_agg: &mut CachedSubAggs<C>,\n    ) {\n        for (doc, term_id) in iter {\n            let bucket_id = term_buckets.term_entry(term_id, bucket_id_provider);\n            sub_agg.push(bucket_id, doc);\n        }\n    }\n\n    #[inline]\n    fn collect_terms(\n        iter: impl Iterator<Item = u64>,\n        term_buckets: &mut TermMap,\n        bucket_id_provider: &mut BucketIdProvider,\n    ) {\n        for term_id in iter {\n            term_buckets.term_entry(term_id, bucket_id_provider);\n        }\n    }\n}\n\npub(crate) trait GetDocCount {\n    fn doc_count(&self) -> u64;\n}\n\nimpl GetDocCount for (String, IntermediateTermBucketEntry) {\n    fn doc_count(&self) -> u64 {\n        self.1.doc_count as u64\n    }\n}\n\nimpl GetDocCount for (u64, Bucket) {\n    fn doc_count(&self) -> u64 {\n        self.1.count as u64\n    }\n}\n\npub(crate) fn cut_off_buckets<T: GetDocCount + Debug>(\n    entries: &mut Vec<T>,\n    num_elem: usize,\n) -> (u64, u64) {\n    let term_doc_count_before_cutoff = entries\n        .get(num_elem)\n        .map(|entry| entry.doc_count())\n        .unwrap_or(0);\n\n    let sum_other_doc_count = entries\n        .get(num_elem..)\n        .map(|cut_off_range| cut_off_range.iter().map(|entry| entry.doc_count()).sum())\n        .unwrap_or(0);\n\n    entries.truncate(num_elem);\n    (term_doc_count_before_cutoff, sum_other_doc_count)\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::IpAddr;\n    use std::str::FromStr;\n\n    use common::DateTime;\n    use time::{Date, Month};\n\n    use super::{PagedTermMap, TermAggregationMap, PAGE_SIZE};\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::intermediate_agg_result::IntermediateAggregationResults;\n    use crate::aggregation::segment_agg_result::BucketIdProvider;\n    use crate::aggregation::tests::{\n        exec_request, exec_request_with_query, exec_request_with_query_and_memory_limit,\n        get_test_index_from_terms, get_test_index_from_values_and_terms,\n    };\n    use crate::aggregation::{AggregationLimitsGuard, DistributedAggregationCollector};\n    use crate::indexer::NoMergePolicy;\n    use crate::query::AllQuery;\n    use crate::schema::{IntoIpv6Addr, Schema, FAST, STRING};\n    use crate::{Index, IndexWriter};\n\n    #[test]\n    fn paged_term_map_reuses_buckets_and_counts() {\n        let mut bucket_id_provider = BucketIdProvider::default();\n        let mut map = PagedTermMap::new((PAGE_SIZE * 2) as u64, &mut bucket_id_provider);\n\n        let bucket_first = map.term_entry(5, &mut bucket_id_provider);\n        let bucket_second_page = map.term_entry((PAGE_SIZE + 7) as u64, &mut bucket_id_provider);\n\n        // Reinsertions should increment counts and reuse bucket ids\n        assert_eq!(map.term_entry(5, &mut bucket_id_provider), bucket_first);\n        assert_eq!(\n            map.term_entry((PAGE_SIZE + 7) as u64, &mut bucket_id_provider),\n            bucket_second_page\n        );\n\n        // High offset exercises the TinySet presence word boundaries.\n        let bucket_high_bit = map.term_entry(63, &mut bucket_id_provider);\n\n        let mut entries = map.into_vec();\n        entries.sort_by_key(|(term_id, _)| *term_id);\n\n        let expected = vec![\n            (5u64, bucket_first, 2u32),\n            (63u64, bucket_high_bit, 1u32),\n            ((PAGE_SIZE + 7) as u64, bucket_second_page, 2u32),\n        ];\n\n        assert_eq!(entries.len(), expected.len());\n        for ((term_id, bucket), (expected_term, expected_bucket_id, expected_count)) in\n            entries.into_iter().zip(expected)\n        {\n            assert_eq!(term_id, expected_term);\n            assert_eq!(bucket.bucket_id, expected_bucket_id);\n            assert_eq!(bucket.count, expected_count);\n        }\n    }\n\n    #[test]\n    fn terms_aggregation_test_single_segment() -> crate::Result<()> {\n        terms_aggregation_test_merge_segment(true)\n    }\n    #[test]\n    fn terms_aggregation_test() -> crate::Result<()> {\n        terms_aggregation_test_merge_segment(false)\n    }\n    fn terms_aggregation_test_merge_segment(merge_segments: bool) -> crate::Result<()> {\n        let segment_and_terms = vec![\n            vec![\"terma\"],\n            vec![\"termb\"],\n            vec![\"termc\"],\n            vec![\"terma\"],\n            vec![\"terma\"],\n            vec![\"terma\"],\n            vec![\"termb\"],\n            vec![\"terma\"],\n        ];\n        let index = get_test_index_from_terms(merge_segments, &segment_and_terms)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 1);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"size\": 2,\n                    \"segment_size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 1);\n\n        // include filter: only terma and termc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"include\": [\"terma\", \"termc\"],\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // exclude filter: remove termc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"exclude\": [\"termc\"],\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // include regex (single string): only termb\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"include\": \"termb\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // include regex (term.*) with exclude regex (termc): expect terma and termb\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"include\": \"term.*\",\n                    \"exclude\": \"termc\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // test min_doc_count\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"size\": 2,\n                    \"min_doc_count\": 3,\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][1][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0); // TODO sum_other_doc_count with min_doc_count\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_test_order_count_single_segment() -> crate::Result<()> {\n        terms_aggregation_test_order_count_merge_segment(true)\n    }\n    #[test]\n    fn terms_aggregation_test_count_order() -> crate::Result<()> {\n        terms_aggregation_test_order_count_merge_segment(false)\n    }\n    fn terms_aggregation_test_order_count_merge_segment(merge_segments: bool) -> crate::Result<()> {\n        let segment_and_terms = vec![\n            vec![(5.0, \"terma\".to_string())],\n            vec![(2.0, \"termb\".to_string())],\n            vec![(2.0, \"terma\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(8.0, \"termb\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n        ];\n        let index = get_test_index_from_values_and_terms(merge_segments, &segment_and_terms)?;\n\n        let sub_agg: Aggregations = serde_json::from_value(json!({\n            \"avg_score\": {\n                \"avg\": {\n                    \"field\": \"score\",\n                }\n            },\n            \"stats_score\": {\n                \"stats\": {\n                    \"field\": \"score\",\n                }\n            }\n        }))\n        .unwrap();\n\n        // sub agg desc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_count\": \"asc\",\n                    },\n                },\n                \"aggs\": sub_agg,\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"avg_score\"][\"value\"], 5.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 6);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"avg_score\"][\"value\"], 4.5);\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // Agg on non string\n        //\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_scores1\": {\n                \"terms\": {\n                    \"field\": \"score\",\n                    \"order\": {\n                        \"_count\": \"asc\",\n                    },\n                },\n                \"aggs\": sub_agg,\n            },\n            \"my_scores2\": {\n                \"terms\": {\n                    \"field\": \"score_f64\",\n                    \"order\": {\n                        \"_count\": \"asc\",\n                    },\n                },\n                \"aggs\": sub_agg,\n            },\n            \"my_scores3\": {\n                \"terms\": {\n                    \"field\": \"score_i64\",\n                    \"order\": {\n                        \"_count\": \"asc\",\n                    },\n                },\n                \"aggs\": sub_agg,\n            }\n\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][0][\"key\"], 8.0);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][0][\"avg_score\"][\"value\"], 8.0);\n\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][1][\"key\"], 2.0);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][1][\"avg_score\"][\"value\"], 2.0);\n\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][2][\"key\"], 1.0);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][2][\"doc_count\"], 3);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][2][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][3][\"key\"], 5.0);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][3][\"doc_count\"], 5);\n        assert_eq!(res[\"my_scores1\"][\"buckets\"][3][\"avg_score\"][\"value\"], 5.0);\n\n        assert_eq!(res[\"my_scores1\"][\"sum_other_doc_count\"], 0);\n\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][0][\"key\"], 8.0);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][0][\"avg_score\"][\"value\"], 8.0);\n\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][1][\"key\"], 2.0);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][1][\"avg_score\"][\"value\"], 2.0);\n\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][2][\"key\"], 1.0);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][2][\"doc_count\"], 3);\n        assert_eq!(res[\"my_scores2\"][\"buckets\"][2][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_scores2\"][\"sum_other_doc_count\"], 0);\n\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][0][\"key\"], 8.0);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][0][\"avg_score\"][\"value\"], 8.0);\n\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][1][\"key\"], 2.0);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][1][\"avg_score\"][\"value\"], 2.0);\n\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][2][\"key\"], 1.0);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][2][\"doc_count\"], 3);\n        assert_eq!(res[\"my_scores3\"][\"buckets\"][2][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_scores3\"][\"sum_other_doc_count\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_simple_agg() {\n        let segment_and_terms = vec![vec![(5.0, \"terma\".to_string())]];\n        let index = get_test_index_from_values_and_terms(true, &segment_and_terms).unwrap();\n\n        let sub_agg: Aggregations = serde_json::from_value(json!({\n            \"avg_score\": {\n                \"avg\": {\n                    \"field\": \"score\",\n                }\n            }\n        }))\n        .unwrap();\n\n        // sub agg desc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_count\": \"asc\",\n                    },\n                        },\n                        \"aggs\": sub_agg,\n                    }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index).unwrap();\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"avg_score\"][\"value\"], 5.0);\n    }\n\n    #[test]\n    fn terms_aggregation_test_order_sub_agg_single_segment() -> crate::Result<()> {\n        terms_aggregation_test_order_sub_agg_merge_segment(true)\n    }\n    #[test]\n    fn terms_aggregation_test_sub_agg_order() -> crate::Result<()> {\n        terms_aggregation_test_order_sub_agg_merge_segment(false)\n    }\n    fn terms_aggregation_test_order_sub_agg_merge_segment(\n        merge_segments: bool,\n    ) -> crate::Result<()> {\n        let segment_and_terms = vec![\n            vec![(5.0, \"terma\".to_string())],\n            vec![(4.0, \"termb\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(8.0, \"termb\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n        ];\n        let index = get_test_index_from_values_and_terms(merge_segments, &segment_and_terms)?;\n\n        let sub_agg: Aggregations = serde_json::from_value(json!({\n            \"avg_score\": {\n                \"avg\": {\n                    \"field\": \"score\",\n                }\n            },\n            \"stats_score\": {\n                \"stats\": {\n                    \"field\": \"score\",\n                }\n            }\n        }))\n        .unwrap();\n\n        // sub agg desc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"avg_score\": \"desc\"\n                    }\n                },\n                \"aggs\": sub_agg,\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"avg_score\"][\"value\"], 6.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"avg_score\"][\"value\"], 5.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // sub agg asc\n        //\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"avg_score\": \"asc\"\n                    }\n                },\n                \"aggs\": sub_agg,\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"avg_score\"][\"value\"], 5.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"avg_score\"][\"value\"], 6.0);\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // sub agg multi value asc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"stats_score.avg\": \"asc\"\n                    }\n                },\n                \"aggs\": sub_agg,\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"avg_score\"][\"value\"], 1.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"avg_score\"][\"value\"], 5.0);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"avg_score\"][\"value\"], 6.0);\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // sub agg invalid request\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"doesnotexist\": \"asc\"\n                    }\n                },\n                \"aggs\": sub_agg,\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index);\n        assert!(res.is_err());\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_test_order_key_single_segment() -> crate::Result<()> {\n        terms_aggregation_test_order_key_merge_segment(true)\n    }\n    #[test]\n    fn terms_aggregation_test_key_order() -> crate::Result<()> {\n        terms_aggregation_test_order_key_merge_segment(false)\n    }\n    fn terms_aggregation_test_order_key_merge_segment(merge_segments: bool) -> crate::Result<()> {\n        let segment_and_terms = vec![\n            vec![(5.0, \"terma\".to_string())],\n            vec![(4.0, \"termb\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(1.0, \"termc\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n            vec![(8.0, \"termb\".to_string())],\n            vec![(5.0, \"terma\".to_string())],\n        ];\n        let index = get_test_index_from_values_and_terms(merge_segments, &segment_and_terms)?;\n\n        // key asc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"asc\"\n                    }\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // key desc and size cut_off\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"asc\"\n                    },\n                    \"size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"doc_count\"],\n            serde_json::Value::Null\n        );\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 3);\n\n        // key asc and segment_size cut_off\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"asc\"\n                    },\n                    \"size\": 2,\n                    \"segment_size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"doc_count\"],\n            serde_json::Value::Null\n        );\n\n        // key desc\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"desc\"\n                    },\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][2][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n\n        // key desc, size cut_off\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"desc\"\n                    },\n                    \"size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"doc_count\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 5);\n\n        // key desc, segment_size cut_off\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"order\": {\n                        \"_key\": \"desc\"\n                    },\n                    \"size\": 2,\n                    \"segment_size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"termc\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"doc_count\"],\n            serde_json::Value::Null\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_min_doc_count_special_case() -> crate::Result<()> {\n        let terms_per_segment = vec![\n            vec![\"terma\", \"terma\", \"termb\", \"termb\", \"termb\"],\n            vec![\"terma\", \"terma\", \"termb\"],\n        ];\n\n        let index = get_test_index_from_terms(false, &terms_per_segment)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"min_doc_count\": 0,\n                },\n            }\n        }))\n        .unwrap();\n\n        // searching for terma, but min_doc_count will return all terms\n        let res = exec_request_with_query(agg_req, &index, Some((\"string_id\", \"terma\")))?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_min_doc_count_special_case_with_sub_agg_empty_merge() -> crate::Result<()>\n    {\n        let mut schema_builder = Schema::builder();\n        let string_field_1 = schema_builder.add_text_field(\"string1\", STRING | FAST);\n        let string_field_2 = schema_builder.add_text_field(\"string2\", STRING | FAST);\n        let score_fieldtype = crate::schema::NumericOptions::default().set_fast();\n        let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            // writing the segment\n            index_writer.add_document(doc!(\n                string_field_1 => \"A\".to_string(),\n                string_field_2 => \"hit\".to_string(),\n                score_field => 1u64,\n            ))?;\n            index_writer.add_document(doc!(\n                string_field_1 => \"B\".to_string(),\n                string_field_2 => \"nohit\".to_string(), // this doc gets filtered in this segment,\n                                                       // but the term will still be loaded because\n                                                       // min_doc_count == 0\n                score_field => 2u64,\n            ))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(\n                string_field_1 => \"A\".to_string(),\n                string_field_2 => \"hit\".to_string(),\n                score_field => 2u64,\n            ))?;\n            index_writer.add_document(doc!(\n                string_field_1 => \"B\".to_string(),\n                string_field_2 => \"hit\".to_string(),\n                score_field => 4u64,\n            ))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string1\",\n                    \"min_doc_count\": 0,\n                },\n                \"aggs\":{\n                    \"elhistogram\": {\n                        \"histogram\": {\n                            \"field\": \"score\",\n                            \"interval\": 1\n                        }\n                    }\n                }\n            }\n        }))\n        .unwrap();\n\n        // searching for terma, but min_doc_count will return all terms\n        let res = exec_request_with_query(agg_req, &index, Some((\"string2\", \"hit\")))?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"A\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][0][\"elhistogram\"][\"buckets\"],\n            json!([{ \"doc_count\": 1, \"key\": 1.0 }, { \"doc_count\": 1, \"key\": 2.0 } ])\n        );\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"B\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][1][\"elhistogram\"][\"buckets\"],\n            json!([ { \"doc_count\": 1, \"key\": 4.0 } ])\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_error_count_test() -> crate::Result<()> {\n        let terms_per_segment = vec![\n            vec![\"terma\", \"terma\", \"termb\", \"termb\", \"termb\", \"termc\"], /* termc doesn't make it\n                                                                         * from this segment */\n            vec![\"terma\", \"terma\", \"termb\", \"termc\", \"termc\"], /* termb doesn't make it from\n                                                                * this segment */\n        ];\n\n        let index = get_test_index_from_terms(false, &terms_per_segment)?;\n        assert_eq!(index.searchable_segments().unwrap().len(), 2);\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"size\": 2,\n                    \"segment_size\": 2\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"terma\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"termb\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 3);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"doc_count\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 4);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 2);\n\n        // disable doc_count_error_upper_bound\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"size\": 2,\n                    \"segment_size\": 2,\n                    \"show_term_doc_count_error\": false\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 4);\n        assert_eq!(\n            res[\"my_texts\"][\"doc_count_error_upper_bound\"],\n            serde_json::Value::Null\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_term_bucket_limit() -> crate::Result<()> {\n        let terms: Vec<String> = (0..20_000).map(|el| el.to_string()).collect();\n        let terms_per_segment = vec![terms.iter().map(|el| el.as_str()).collect()];\n\n        let index = get_test_index_from_terms(true, &terms_per_segment)?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"min_doc_count\": 0,\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query_and_memory_limit(\n            agg_req,\n            &index,\n            None,\n            AggregationLimitsGuard::new(Some(50_000), None),\n        )\n        .unwrap_err();\n        assert!(res\n            .to_string()\n            .contains(\"Aborting aggregation because memory limit was exceeded. Limit: 50.00 KB\"));\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_different_tokenizer_on_ff_test() -> crate::Result<()> {\n        let terms = vec![\"Hello Hello\", \"Hallo Hallo\", \"Hallo Hallo\"];\n\n        let index = get_test_index_from_terms(true, &[terms])?;\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"text_id\",\n                    \"min_doc_count\": 0,\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None).unwrap();\n        println!(\"{}\", serde_json::to_string_pretty(&res).unwrap());\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"Hallo Hallo\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 2);\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"Hello Hello\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 1);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_json_format() -> crate::Result<()> {\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"term_agg_test\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"size\": 2,\n                    \"segment_size\": 2,\n                    \"order\": {\n                        \"_key\": \"desc\"\n                    }\n                },\n            }\n        }))\n        .unwrap();\n\n        let elasticsearch_compatible_json = json!(\n        {\n        \"term_agg_test\":{\n            \"terms\": {\n                \"field\": \"string_id\",\n                \"size\": 2u64,\n                \"segment_size\": 2u64,\n                \"order\": {\"_key\": \"desc\"}\n            }\n        }\n        });\n\n        let agg_req_deser: Aggregations =\n            serde_json::from_str(&serde_json::to_string(&elasticsearch_compatible_json).unwrap())\n                .unwrap();\n        assert_eq!(agg_req, agg_req_deser);\n\n        let elasticsearch_compatible_json = json!(\n        {\n        \"term_agg_test\":{\n            \"terms\": {\n                \"field\": \"string_id\",\n                \"split_size\": 2u64,\n            }\n        }\n        });\n\n        // test alias shard_size, split_size\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"term_agg_test\": {\n                \"terms\": {\n                    \"field\": \"string_id\",\n                    \"split_size\": 2,\n                },\n            }\n        }))\n        .unwrap();\n\n        let agg_req_deser: Aggregations =\n            serde_json::from_str(&serde_json::to_string(&elasticsearch_compatible_json).unwrap())\n                .unwrap();\n        assert_eq!(agg_req, agg_req_deser);\n\n        let elasticsearch_compatible_json = json!(\n        {\n        \"term_agg_test\":{\n            \"terms\": {\n                \"field\": \"string_id\",\n                \"shard_size\": 2u64,\n            }\n        }\n        });\n\n        let agg_req_deser: Aggregations =\n            serde_json::from_str(&serde_json::to_string(&elasticsearch_compatible_json).unwrap())\n                .unwrap();\n        assert_eq!(agg_req, agg_req_deser);\n\n        Ok(())\n    }\n    #[test]\n    fn terms_empty_json() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with empty json\n        index_writer.add_document(doc!()).unwrap();\n        index_writer.commit().unwrap();\n        // => Segment with json, but no field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"different_field\": \"blue\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        //// => Segment with field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"partially_empty\": \"blue\"})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"json.partially_empty\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"blue\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1], serde_json::Value::Null);\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_bytes() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let bytes_field = schema_builder.add_bytes_field(\"bytes\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                bytes_field => vec![1,2,3],\n            ))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"bytes\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // TODO: Returning an error would be better instead of an empty result, since this is not a\n        // JSON field\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][0][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_multi_value() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", FAST);\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n                text_field => \"Hello Hello\",\n                id_field => 1u64,\n                id_field => 1u64,\n            ))?;\n            // Missing\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n            ))?;\n            index_writer.commit()?;\n            // Empty segment special case\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n            // Full segment special case\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n                id_field => 1u64,\n            ))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": \"Empty\"\n                },\n            },\n            \"my_texts2\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": 1337\n                },\n            },\n            \"my_ids\": {\n                \"terms\": {\n                    \"field\": \"id\",\n                    \"missing\": 1337\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"Hello Hello\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"Empty\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"key\"],\n            serde_json::Value::Null\n        );\n        // text field with number as missing fallback\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"key\"], \"Hello Hello\");\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"doc_count\"], 5);\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][1][\"key\"], 1337.0);\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts2\"][\"buckets\"][2][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        // id field\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"key\"], 1.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"doc_count\"], 3);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n    #[test]\n    fn terms_aggregation_missing_simple_id() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                id_field => 1u64,\n            ))?;\n            // Missing\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_ids\": {\n                \"terms\": {\n                    \"field\": \"id\",\n                    \"missing\": 1337\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // id field\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"key\"], 1.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_u64_value() -> crate::Result<()> {\n        // Make sure that large u64 are not truncated\n        let mut schema_builder = Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                id_field => 9_223_372_036_854_775_807u64,\n            ))?;\n            index_writer.add_document(doc!(\n                id_field => 1_769_070_189_829_214_202u64,\n            ))?;\n            index_writer.add_document(doc!(\n                id_field => 1_769_070_189_829_214_202u64,\n            ))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_ids\": {\n                \"terms\": {\n                    \"field\": \"id\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // id field\n        assert_eq!(\n            res[\"my_ids\"][\"buckets\"][0][\"key\"],\n            1_769_070_189_829_214_202u64\n        );\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_ids\"][\"buckets\"][1][\"key\"],\n            9_223_372_036_854_775_807u64\n        );\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing1() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", FAST);\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n                id_field => 1u64,\n            ))?;\n            // Missing\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n            ))?;\n            index_writer.commit()?;\n            // Empty segment special case\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n            // Full segment special case\n            index_writer.add_document(doc!(\n                text_field => \"Hello Hello\",\n                id_field => 1u64,\n            ))?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": \"Empty\"\n                },\n            },\n            \"my_texts2\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": 1337\n                },\n            },\n            \"my_ids\": {\n                \"terms\": {\n                    \"field\": \"id\",\n                    \"missing\": 1337\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"Hello Hello\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"key\"], \"Empty\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][2][\"key\"],\n            serde_json::Value::Null\n        );\n        // text field with number as missing fallback\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"key\"], \"Hello Hello\");\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][1][\"key\"], 1337.0);\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"my_texts2\"][\"buckets\"][2][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        // id field\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"doc_count\"], 4);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"key\"], 1.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"doc_count\"], 2);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n    #[test]\n    fn terms_aggregation_missing_empty() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"text\", FAST);\n        schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            // Empty segment special case\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_texts\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": \"Empty\"\n                },\n            },\n            \"my_texts2\": {\n                \"terms\": {\n                    \"field\": \"text\",\n                    \"missing\": 1337\n                },\n            },\n            \"my_ids\": {\n                \"terms\": {\n                    \"field\": \"id\",\n                    \"missing\": 1337\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"key\"], \"Empty\");\n        assert_eq!(res[\"my_texts\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(\n            res[\"my_texts\"][\"buckets\"][1][\"key\"],\n            serde_json::Value::Null\n        );\n        // text field with number as missing fallback\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"my_texts2\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(\n            res[\"my_texts2\"][\"buckets\"][1][\"key\"],\n            serde_json::Value::Null\n        );\n        assert_eq!(res[\"my_texts\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"my_texts\"][\"doc_count_error_upper_bound\"], 0);\n\n        // id field\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][0][\"doc_count\"], 1);\n        assert_eq!(res[\"my_ids\"][\"buckets\"][1][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_date() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut writer = index.writer_with_num_threads(1, 15_000_000)?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1982, Month::September, 17)?.with_hms(0, 0, 0)?)))?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1982, Month::September, 17)?.with_hms(0, 0, 0)?)))?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1983, Month::September, 27)?.with_hms(0, 0, 0)?)))?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_date\": {\n                \"terms\": {\n                    \"field\": \"date_field\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // date_field field\n        assert_eq!(res[\"my_date\"][\"buckets\"][0][\"key\"], \"1982-09-17T00:00:00Z\");\n        assert_eq!(res[\"my_date\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_date\"][\"buckets\"][1][\"key\"], \"1983-09-27T00:00:00Z\");\n        assert_eq!(res[\"my_date\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_date\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n    #[test]\n    fn terms_aggregation_date_missing() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut writer = index.writer_with_num_threads(1, 15_000_000)?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1982, Month::September, 17)?.with_hms(0, 0, 0)?)))?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1982, Month::September, 17)?.with_hms(0, 0, 0)?)))?;\n            writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1983, Month::September, 27)?.with_hms(0, 0, 0)?)))?;\n            writer.add_document(doc!())?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_date\": {\n                \"terms\": {\n                    \"field\": \"date_field\",\n                    \"missing\": \"1982-09-17T00:00:00Z\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // date_field field\n        assert_eq!(res[\"my_date\"][\"buckets\"][0][\"key\"], \"1982-09-17T00:00:00Z\");\n        assert_eq!(res[\"my_date\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"my_date\"][\"buckets\"][1][\"key\"], \"1983-09-27T00:00:00Z\");\n        assert_eq!(res[\"my_date\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_date\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_bool() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_bool_field(\"bool_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut writer = index.writer_with_num_threads(1, 15_000_000)?;\n            writer.add_document(doc!(field=>true))?;\n            writer.add_document(doc!(field=>false))?;\n            writer.add_document(doc!(field=>true))?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_bool\": {\n                \"terms\": {\n                    \"field\": \"bool_field\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(res[\"my_bool\"][\"buckets\"][0][\"key\"], 1.0);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][0][\"key_as_string\"], \"true\");\n        assert_eq!(res[\"my_bool\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][1][\"key\"], 0.0);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][1][\"key_as_string\"], \"false\");\n        assert_eq!(res[\"my_bool\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_ip_addr() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_ip_addr_field(\"ip_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut writer = index.writer_with_num_threads(1, 15_000_000)?;\n            // IpV6 loopback\n            writer.add_document(doc!(field=>IpAddr::from_str(\"::1\").unwrap().into_ipv6_addr()))?;\n            writer.add_document(doc!(field=>IpAddr::from_str(\"::1\").unwrap().into_ipv6_addr()))?;\n            // IpV4\n            writer.add_document(\n                doc!(field=>IpAddr::from_str(\"127.0.0.1\").unwrap().into_ipv6_addr()),\n            )?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_bool\": {\n                \"terms\": {\n                    \"field\": \"ip_field\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n        // print as json\n        // println!(\"{}\", serde_json::to_string_pretty(&res).unwrap());\n\n        assert_eq!(res[\"my_bool\"][\"buckets\"][0][\"key\"], \"::1\");\n        assert_eq!(res[\"my_bool\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][1][\"key\"], \"127.0.0.1\");\n        assert_eq!(res[\"my_bool\"][\"buckets\"][1][\"doc_count\"], 1);\n        assert_eq!(res[\"my_bool\"][\"buckets\"][2][\"key\"], serde_json::Value::Null);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggs_hosts_and_tags_merge_on_mixed_order_request() -> crate::Result<()> {\n        // This test ensures that merging of aggregation results works correctly\n        // even if the order of the aggregation requests is different and\n        // running on different indexes with the same data.\n        let build_index = || -> crate::Result<Index> {\n            let mut schema_builder = Schema::builder();\n            let fielda = schema_builder.add_text_field(\"fielda\", FAST);\n            let fieldb = schema_builder.add_text_field(\"fieldb\", FAST);\n            let host = schema_builder.add_text_field(\"host\", FAST);\n            let tags = schema_builder.add_text_field(\"tags\", FAST);\n            let schema = schema_builder.build();\n\n            let index = Index::create_in_ram(schema.clone());\n            let mut writer = index.writer(50_000_000).unwrap();\n\n            // --- Ingest documents (batch #1) ---\n            writer.add_document(doc!(\n                host => \"192.168.0.10\",\n                tags => \"nice\",\n                fielda => \"a\",\n                fieldb => \"b\",\n            ))?;\n            writer.add_document(doc!(\n                host => \"192.168.0.1\",\n                tags => \"nice\",\n            ))?;\n            writer.add_document(doc!(\n                host => \"192.168.0.11\",\n                tags => \"nice\",\n            ))?;\n            writer.add_document(doc!(\n                host => \"192.168.0.10\",\n                tags => \"nice\",\n                tags => \"cool\",\n            ))?;\n            writer.add_document(doc!(\n                host => \"192.168.0.1\",\n                tags => \"nice\",\n                tags => \"cool\",\n            ))?;\n\n            writer.commit()?;\n\n            // --- Ingest documents (batch #2) ---\n            writer.add_document(doc!())?;\n            writer.add_document(doc!())?;\n            writer.add_document(doc!(\n                host => \"192.168.0.10\",\n            ))?;\n            writer.add_document(doc!(\n                host => \"192.168.0.10\",\n            ))?;\n            writer.add_document(doc!())?;\n\n            writer.commit()?;\n            Ok(index)\n        };\n        let index = build_index()?;\n        let index2 = build_index()?;\n\n        let search = |idx: &Index,\n                      agg_req: &Aggregations|\n         -> crate::Result<IntermediateAggregationResults> {\n            let collector =\n                DistributedAggregationCollector::from_aggs(agg_req.clone(), Default::default());\n            let reader = idx.reader()?;\n            let searcher = reader.searcher();\n            let agg_res = searcher.search(&AllQuery, &collector)?;\n            Ok(agg_res)\n        };\n\n        // --- Aggregations: terms on host and tags ---\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"hosts\": { \"terms\": { \"field\": \"host\" } },\n            \"tags\":  { \"terms\": { \"field\": \"tags\" } },\n            \"fielda\":  { \"terms\": { \"field\": \"fielda\" } },\n            \"fieldb\":  { \"terms\": { \"field\": \"fieldb\" } },\n        }))\n        .unwrap();\n\n        let mut agg_res = search(&index, &agg_req)?;\n\n        // --- Aggregations: terms on host and tags ---\n        let mut agg_req2: Aggregations =\n            Aggregations::with_capacity_and_hasher(20, Default::default());\n        agg_req2.insert(\n            \"tags\".to_string(),\n            serde_json::from_value(json!({ \"terms\": { \"field\": \"tags\" } }))?,\n        );\n        agg_req2.insert(\n            \"fielda\".to_string(),\n            serde_json::from_value(json!({ \"terms\": { \"field\": \"fielda\" } }))?,\n        );\n        agg_req2.insert(\n            \"hosts\".to_string(),\n            serde_json::from_value(json!({ \"terms\": { \"field\": \"host\" } }))?,\n        );\n        agg_req2.insert(\n            \"fieldb\".to_string(),\n            serde_json::from_value(json!({ \"terms\": { \"field\": \"fieldb\" } }))?,\n        );\n        // make sure the order of the aggregation request is different\n        // disabled to avoid flaky test with hashmap changes\n        // assert_ne!(agg_req.keys().next(), agg_req2.keys().next());\n\n        let agg_res2 = search(&index2, &agg_req2)?;\n\n        agg_res.merge_fruits(agg_res2).unwrap();\n        let agg_json =\n            serde_json::to_value(&agg_res.into_final_result(agg_req2, Default::default())?)?;\n\n        // hosts:\n        let hosts = &agg_json[\"hosts\"][\"buckets\"];\n        assert_eq!(hosts[0][\"key\"], \"192.168.0.10\");\n        assert_eq!(hosts[0][\"doc_count\"], 8);\n        assert_eq!(hosts[1][\"key\"], \"192.168.0.1\");\n        assert_eq!(hosts[1][\"doc_count\"], 4);\n        assert_eq!(hosts[2][\"key\"], \"192.168.0.11\");\n        assert_eq!(hosts[2][\"doc_count\"], 2);\n        // Implementation currently reports error bounds/other count; ensure zero.\n        assert_eq!(agg_json[\"hosts\"][\"doc_count_error_upper_bound\"], 0);\n        assert_eq!(agg_json[\"hosts\"][\"sum_other_doc_count\"], 0);\n\n        // tags:\n        let tags_buckets = &agg_json[\"tags\"][\"buckets\"];\n        assert_eq!(tags_buckets[0][\"key\"], \"nice\");\n        assert_eq!(tags_buckets[0][\"doc_count\"], 10);\n        assert_eq!(tags_buckets[1][\"key\"], \"cool\");\n        assert_eq!(tags_buckets[1][\"doc_count\"], 4);\n        assert_eq!(agg_json[\"tags\"][\"doc_count_error_upper_bound\"], 0);\n        assert_eq!(agg_json[\"tags\"][\"sum_other_doc_count\"], 0);\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/bucket/term_missing_agg.rs",
    "content": "use columnar::{Column, ColumnType};\nuse rustc_hash::FxHashMap;\n\nuse crate::aggregation::agg_data::{\n    build_segment_agg_collectors, AggRefNode, AggregationsSegmentCtx,\n};\nuse crate::aggregation::bucket::term_agg::TermsAggregation;\nuse crate::aggregation::cached_sub_aggs::{CachedSubAggs, HighCardCachedSubAggs};\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateBucketResult,\n    IntermediateKey, IntermediateTermBucketEntry, IntermediateTermBucketResult,\n};\nuse crate::aggregation::segment_agg_result::{BucketIdProvider, SegmentAggregationCollector};\nuse crate::aggregation::BucketId;\n\n/// Special aggregation to handle missing values for term aggregations.\n/// This missing aggregation will check multiple columns for existence.\n///\n/// This is needed when:\n/// - The field is multi-valued and we therefore have multiple columns\n/// - The field is not text and missing is provided as string (we cannot use the numeric missing\n///   value optimization)\n#[derive(Default)]\npub struct MissingTermAggReqData {\n    /// The accessors to check for existence of a value.\n    pub accessors: Vec<(Column<u64>, ColumnType)>,\n    /// The name of the aggregation.\n    pub name: String,\n    /// The original terms aggregation request.\n    pub req: TermsAggregation,\n}\n\nimpl MissingTermAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\n#[derive(Default, Debug, Clone)]\nstruct MissingCount {\n    missing_count: u32,\n    bucket_id: BucketId,\n}\n\n/// The specialized missing term aggregation.\n#[derive(Default, Debug)]\npub struct TermMissingAgg {\n    accessor_idx: usize,\n    sub_agg: Option<HighCardCachedSubAggs>,\n    /// Idx = parent bucket id, Value = missing count for that bucket\n    missing_count_per_bucket: Vec<MissingCount>,\n    bucket_id_provider: BucketIdProvider,\n}\nimpl TermMissingAgg {\n    pub(crate) fn new(\n        agg_data: &mut AggregationsSegmentCtx,\n        node: &AggRefNode,\n    ) -> crate::Result<Self> {\n        let has_sub_aggregations = !node.children.is_empty();\n        let accessor_idx = node.idx_in_req_data;\n        let sub_agg = if has_sub_aggregations {\n            let sub_aggregation = build_segment_agg_collectors(agg_data, &node.children)?;\n            Some(sub_aggregation)\n        } else {\n            None\n        };\n\n        let sub_agg = sub_agg.map(CachedSubAggs::new);\n        let bucket_id_provider = BucketIdProvider::default();\n\n        Ok(Self {\n            accessor_idx,\n            sub_agg,\n            missing_count_per_bucket: Vec::new(),\n            bucket_id_provider,\n        })\n    }\n}\n\nimpl SegmentAggregationCollector for TermMissingAgg {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let req_data = agg_data.get_missing_term_req_data(self.accessor_idx);\n        let term_agg = &req_data.req;\n        let missing = term_agg\n            .missing\n            .as_ref()\n            .expect(\"TermMissingAgg collector, but no missing found in agg req\")\n            .clone();\n        let mut entries: FxHashMap<IntermediateKey, IntermediateTermBucketEntry> =\n            Default::default();\n\n        let missing_count = &self.missing_count_per_bucket[parent_bucket_id as usize];\n        let mut missing_entry = IntermediateTermBucketEntry {\n            doc_count: missing_count.missing_count,\n            sub_aggregation: Default::default(),\n        };\n        if let Some(sub_agg) = &mut self.sub_agg {\n            let mut res = IntermediateAggregationResults::default();\n            sub_agg\n                .get_sub_agg_collector()\n                .add_intermediate_aggregation_result(agg_data, &mut res, missing_count.bucket_id)?;\n            missing_entry.sub_aggregation = res;\n        }\n        entries.insert(missing.into(), missing_entry);\n\n        let bucket = IntermediateBucketResult::Terms {\n            buckets: IntermediateTermBucketResult {\n                entries,\n                sum_other_doc_count: 0,\n                doc_count_error_upper_bound: 0,\n            },\n        };\n\n        results.push(\n            req_data.name.to_string(),\n            IntermediateAggregationResult::Bucket(bucket),\n        )?;\n\n        Ok(())\n    }\n\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let bucket = &mut self.missing_count_per_bucket[parent_bucket_id as usize];\n        let req_data = agg_data.get_missing_term_req_data(self.accessor_idx);\n\n        for doc in docs {\n            let doc = *doc;\n            let has_value = req_data\n                .accessors\n                .iter()\n                .any(|(acc, _)| acc.index.has_value(doc));\n            if !has_value {\n                bucket.missing_count += 1;\n\n                if let Some(sub_agg) = self.sub_agg.as_mut() {\n                    sub_agg.push(bucket.bucket_id, doc);\n                }\n            }\n        }\n\n        if let Some(sub_agg) = self.sub_agg.as_mut() {\n            sub_agg.check_flush_local(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.missing_count_per_bucket.len() <= max_bucket as usize {\n            let bucket_id = self.bucket_id_provider.next_bucket_id();\n            self.missing_count_per_bucket.push(MissingCount {\n                missing_count: 0,\n                bucket_id,\n            });\n        }\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if let Some(sub_agg) = self.sub_agg.as_mut() {\n            sub_agg.flush(agg_data)?;\n        }\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::exec_request_with_query;\n    use crate::schema::{Schema, FAST};\n    use crate::{Index, IndexWriter};\n\n    #[test]\n    fn terms_aggregation_missing_mixed_type_mult_seg_sub_agg() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let score = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with all values numeric\n        index_writer\n            .add_document(doc!(score => 1.0, json => json!({\"mixed_type\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!(score => 5.0))?;\n        // index_writer.commit().unwrap();\n        //// => Segment with all values text\n        index_writer\n            .add_document(doc!(score => 1.0, json => json!({\"mixed_type\": \"blue\"})))\n            .unwrap();\n        index_writer.add_document(doc!(score => 5.0))?;\n        // index_writer.commit().unwrap();\n\n        // => Segment with mixed values\n        index_writer.add_document(doc!(json => json!({\"mixed_type\": \"red\"})))?;\n        index_writer.add_document(doc!(json => json!({\"mixed_type\": -20.5})))?;\n        index_writer.add_document(doc!(json => json!({\"mixed_type\": true})))?;\n        index_writer.add_document(doc!(score => 5.0))?;\n\n        index_writer.commit().unwrap();\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n                \"aggs\": {\n                    \"sum_score\": {\n                        \"sum\": {\n                            \"field\": \"score\"\n                        }\n                    }\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(\n            res[\"replace_null\"][\"buckets\"][0][\"sum_score\"][\"value\"],\n            15.0\n        );\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_mixed_type_sub_agg_reg1() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let score = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with all values numeric\n        index_writer.add_document(doc!(score => 1.0, json => json!({\"mixed_type\": 10.0})))?;\n        index_writer.add_document(doc!(score => 5.0))?;\n        index_writer.add_document(doc!(score => 5.0))?;\n\n        index_writer.commit().unwrap();\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n                \"aggs\": {\n                    \"sum_score\": {\n                        \"sum\": {\n                            \"field\": \"score\"\n                        }\n                    }\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 2);\n        assert_eq!(\n            res[\"replace_null\"][\"buckets\"][0][\"sum_score\"][\"value\"],\n            10.0\n        );\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_mult_seg_empty() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let score = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n\n        index_writer.add_document(doc!(score => 5.0))?;\n        index_writer.commit().unwrap();\n        index_writer.add_document(doc!(score => 5.0))?;\n        index_writer.commit().unwrap();\n        index_writer.add_document(doc!(score => 5.0))?;\n\n        index_writer.commit().unwrap();\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n                \"aggs\": {\n                    \"sum_score\": {\n                        \"sum\": {\n                            \"field\": \"score\"\n                        }\n                    }\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(\n            res[\"replace_null\"][\"buckets\"][0][\"sum_score\"][\"value\"],\n            15.0\n        );\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_single_seg_empty() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let score = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n\n        index_writer.add_document(doc!(score => 5.0))?;\n        index_writer.add_document(doc!(score => 5.0))?;\n        index_writer.add_document(doc!(score => 5.0))?;\n\n        index_writer.commit().unwrap();\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n                \"aggs\": {\n                    \"sum_score\": {\n                        \"sum\": {\n                            \"field\": \"score\"\n                        }\n                    }\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(\n            res[\"replace_null\"][\"buckets\"][0][\"sum_score\"][\"value\"],\n            15.0\n        );\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_mixed_type_mult_seg() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with all values numeric\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n        //// => Segment with all values text\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": \"blue\"})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n\n        // => Segment with mixed values\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": \"red\"})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": -20.5})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": true})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n\n        index_writer.commit().unwrap();\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n            },\n            \"replace_num\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": 1337\n                },\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"replace_num\"][\"buckets\"][0][\"key\"], 1337.0);\n        assert_eq!(res[\"replace_num\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_str_on_numeric_field() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with all values numeric\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.add_document(doc!())?;\n\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": -20.5})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn terms_aggregation_missing_mixed_type_one_seg() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with all values numeric\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        //// => Segment with all values text\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": \"blue\"})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n\n        // => Segment with mixed values\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": \"red\"})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": -20.5})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"mixed_type\": true})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"replace_null\": {\n                \"terms\": {\n                    \"field\": \"json.mixed_type\",\n                    \"missing\": \"NULL\"\n                },\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        // text field\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"key\"], \"NULL\");\n        assert_eq!(res[\"replace_null\"][\"buckets\"][0][\"doc_count\"], 3);\n        assert_eq!(res[\"replace_null\"][\"sum_other_doc_count\"], 0);\n        assert_eq!(res[\"replace_null\"][\"doc_count_error_upper_bound\"], 0);\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/cached_sub_aggs.rs",
    "content": "use std::fmt::Debug;\n\nuse super::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::bucket::MAX_NUM_TERMS_FOR_VEC;\nuse crate::aggregation::BucketId;\nuse crate::DocId;\n\n/// A cache for sub-aggregations, storing doc ids per bucket id.\n/// Depending on the cardinality of the parent aggregation, we use different\n/// storage strategies.\n///\n/// ## Low Cardinality\n/// Cardinality here refers to the number of unique flattened buckets that can be created\n/// by the parent aggregation.\n/// Flattened buckets are the result of combining all buckets per collector\n/// into a single list of buckets, where each bucket is identified by its BucketId.\n///\n/// ## Usage\n/// Since this is caching for sub-aggregations, it is only used by bucket\n/// aggregations.\n///\n/// TODO: consider using a more advanced data structure for high cardinality\n/// aggregations.\n/// What this datastructure does in general is to group docs by bucket id.\n#[derive(Debug)]\npub(crate) struct CachedSubAggs<C: SubAggCache> {\n    cache: C,\n    sub_agg_collector: Box<dyn SegmentAggregationCollector>,\n    num_docs: usize,\n}\n\npub type LowCardCachedSubAggs = CachedSubAggs<LowCardSubAggCache>;\npub type HighCardCachedSubAggs = CachedSubAggs<HighCardSubAggCache>;\n\nconst FLUSH_THRESHOLD: usize = 2048;\n\n/// A trait for caching sub-aggregation doc ids per bucket id.\n/// Different implementations can be used depending on the cardinality\n/// of the parent aggregation.\npub trait SubAggCache: Debug {\n    fn new() -> Self;\n    fn push(&mut self, bucket_id: BucketId, doc_id: DocId);\n    fn flush_local(\n        &mut self,\n        sub_agg: &mut Box<dyn SegmentAggregationCollector>,\n        agg_data: &mut AggregationsSegmentCtx,\n        force: bool,\n    ) -> crate::Result<()>;\n}\n\nimpl<Backend: SubAggCache + Debug> CachedSubAggs<Backend> {\n    pub fn new(sub_agg: Box<dyn SegmentAggregationCollector>) -> Self {\n        Self {\n            cache: Backend::new(),\n            sub_agg_collector: sub_agg,\n            num_docs: 0,\n        }\n    }\n\n    pub fn get_sub_agg_collector(&mut self) -> &mut Box<dyn SegmentAggregationCollector> {\n        &mut self.sub_agg_collector\n    }\n\n    #[inline]\n    pub fn push(&mut self, bucket_id: BucketId, doc_id: DocId) {\n        self.cache.push(bucket_id, doc_id);\n        self.num_docs += 1;\n    }\n\n    /// Check if we need to flush based on the number of documents cached.\n    /// If so, flushes the cache to the provided aggregation collector.\n    pub fn check_flush_local(\n        &mut self,\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        if self.num_docs >= FLUSH_THRESHOLD {\n            self.cache\n                .flush_local(&mut self.sub_agg_collector, agg_data, false)?;\n            self.num_docs = 0;\n        }\n        Ok(())\n    }\n\n    /// Note: this _does_ flush the sub aggregations.\n    pub fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        if self.num_docs != 0 {\n            self.cache\n                .flush_local(&mut self.sub_agg_collector, agg_data, true)?;\n            self.num_docs = 0;\n        }\n        self.sub_agg_collector.flush(agg_data)?;\n        Ok(())\n    }\n}\n\n/// Number of partitions for high cardinality sub-aggregation cache.\nconst NUM_PARTITIONS: usize = 16;\n\n#[derive(Debug)]\npub(crate) struct HighCardSubAggCache {\n    /// This weird partitioning is used to do some cheap grouping on the bucket ids.\n    /// bucket ids are dense, e.g. when we don't detect the cardinality as low cardinality,\n    /// but there are just 16 bucket ids, each bucket id will go to its own partition.\n    ///\n    /// We want to keep this cheap, because high cardinality aggregations can have a lot of\n    /// buckets, and there may be nothing to group.\n    partitions: Box<[PartitionEntry; NUM_PARTITIONS]>,\n}\n\nimpl HighCardSubAggCache {\n    #[inline]\n    fn clear(&mut self) {\n        for partition in self.partitions.iter_mut() {\n            partition.clear();\n        }\n    }\n}\n\n#[derive(Debug, Clone, Default)]\nstruct PartitionEntry {\n    bucket_ids: Vec<BucketId>,\n    docs: Vec<DocId>,\n}\n\nimpl PartitionEntry {\n    #[inline]\n    fn clear(&mut self) {\n        self.bucket_ids.clear();\n        self.docs.clear();\n    }\n}\n\nimpl SubAggCache for HighCardSubAggCache {\n    fn new() -> Self {\n        Self {\n            partitions: Box::new(core::array::from_fn(|_| PartitionEntry::default())),\n        }\n    }\n\n    fn push(&mut self, bucket_id: BucketId, doc_id: DocId) {\n        let idx = bucket_id % NUM_PARTITIONS as u32;\n        let slot = &mut self.partitions[idx as usize];\n        slot.bucket_ids.push(bucket_id);\n        slot.docs.push(doc_id);\n    }\n\n    fn flush_local(\n        &mut self,\n        sub_agg: &mut Box<dyn SegmentAggregationCollector>,\n        agg_data: &mut AggregationsSegmentCtx,\n        _force: bool,\n    ) -> crate::Result<()> {\n        let mut max_bucket = 0u32;\n        for partition in self.partitions.iter() {\n            if let Some(&local_max) = partition.bucket_ids.iter().max() {\n                max_bucket = max_bucket.max(local_max);\n            }\n        }\n\n        sub_agg.prepare_max_bucket(max_bucket, agg_data)?;\n\n        for slot in self.partitions.iter() {\n            if !slot.bucket_ids.is_empty() {\n                // Reduce dynamic dispatch overhead by collecting a full partition in one call.\n                sub_agg.collect_multiple(&slot.bucket_ids, &slot.docs, agg_data)?;\n            }\n        }\n\n        self.clear();\n        Ok(())\n    }\n}\n\n#[derive(Debug)]\npub(crate) struct LowCardSubAggCache {\n    /// Cache doc ids per bucket for sub-aggregations.\n    ///\n    /// The outer Vec is indexed by BucketId.\n    per_bucket_docs: Vec<Vec<DocId>>,\n}\n\nimpl LowCardSubAggCache {\n    #[inline]\n    fn clear(&mut self) {\n        for v in &mut self.per_bucket_docs {\n            v.clear();\n        }\n    }\n}\n\nimpl SubAggCache for LowCardSubAggCache {\n    fn new() -> Self {\n        Self {\n            per_bucket_docs: Vec::new(),\n        }\n    }\n\n    fn push(&mut self, bucket_id: BucketId, doc_id: DocId) {\n        let idx = bucket_id as usize;\n        if self.per_bucket_docs.len() <= idx {\n            self.per_bucket_docs.resize_with(idx + 1, Vec::new);\n        }\n        self.per_bucket_docs[idx].push(doc_id);\n    }\n\n    fn flush_local(\n        &mut self,\n        sub_agg: &mut Box<dyn SegmentAggregationCollector>,\n        agg_data: &mut AggregationsSegmentCtx,\n        force: bool,\n    ) -> crate::Result<()> {\n        // Pre-aggregated: call collect per bucket.\n        let max_bucket = (self.per_bucket_docs.len() as BucketId).saturating_sub(1);\n        sub_agg.prepare_max_bucket(max_bucket, agg_data)?;\n        // The threshold above which we flush buckets individually.\n        // Note: We need to make sure that we don't lock ourselves into a situation where we hit\n        // the FLUSH_THRESHOLD, but never flush any buckets. (except the final flush)\n        let mut bucket_treshold = FLUSH_THRESHOLD / (self.per_bucket_docs.len().max(1) * 2);\n        const _: () = {\n            // MAX_NUM_TERMS_FOR_VEC threshold is used for term aggregations\n            // Note: There may be other flexible values, for other aggregations, but we can use the\n            // const value here as a upper bound. (better than nothing)\n            let bucket_treshold_limit = FLUSH_THRESHOLD / (MAX_NUM_TERMS_FOR_VEC as usize * 2);\n            assert!(\n                bucket_treshold_limit > 0,\n                \"Bucket threshold must be greater than 0\"\n            );\n        };\n        if force {\n            bucket_treshold = 0;\n        }\n        for (bucket_id, docs) in self\n            .per_bucket_docs\n            .iter()\n            .enumerate()\n            .filter(|(_, docs)| docs.len() > bucket_treshold)\n        {\n            sub_agg.collect(bucket_id as BucketId, docs, agg_data)?;\n        }\n\n        self.clear();\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/collector.rs",
    "content": "use super::agg_req::Aggregations;\nuse super::agg_result::AggregationResults;\nuse super::cached_sub_aggs::LowCardCachedSubAggs;\nuse super::intermediate_agg_result::IntermediateAggregationResults;\nuse super::AggContextParams;\n// group buffering strategy is chosen explicitly by callers; no need to hash-group on the fly.\nuse crate::aggregation::agg_data::{\n    build_aggregations_data_from_req, build_segment_agg_collectors_root, AggregationsSegmentCtx,\n};\nuse crate::collector::{Collector, SegmentCollector};\nuse crate::index::SegmentReader;\nuse crate::{DocId, SegmentOrdinal, TantivyError};\n\n/// The default max bucket count, before the aggregation fails.\npub const DEFAULT_BUCKET_LIMIT: u32 = 65000;\n\n/// The default memory limit in bytes before the aggregation fails. 500MB\npub const DEFAULT_MEMORY_LIMIT: u64 = 500_000_000;\n\n/// Collector for aggregations.\n///\n/// The collector collects all aggregations by the underlying aggregation request.\npub struct AggregationCollector {\n    agg: Aggregations,\n    context: AggContextParams,\n}\n\nimpl AggregationCollector {\n    /// Create collector from aggregation request.\n    ///\n    /// Aggregation fails when the limits in `AggregationLimits` is exceeded. (memory limit and\n    /// bucket limit)\n    pub fn from_aggs(agg: Aggregations, context: AggContextParams) -> Self {\n        Self { agg, context }\n    }\n}\n\n/// Collector for distributed aggregations.\n///\n/// The collector collects all aggregations by the underlying aggregation request.\n///\n/// # Purpose\n/// AggregationCollector returns `IntermediateAggregationResults` and not the final\n/// `AggregationResults`, so that results from different indices can be merged and then converted\n/// into the final `AggregationResults` via the `into_final_result()` method.\npub struct DistributedAggregationCollector {\n    agg: Aggregations,\n    context: AggContextParams,\n}\n\nimpl DistributedAggregationCollector {\n    /// Create collector from aggregation request.\n    ///\n    /// Aggregation fails when the limits in `AggregationLimits` is exceeded. (memory limit and\n    /// bucket limit)\n    pub fn from_aggs(agg: Aggregations, context: AggContextParams) -> Self {\n        Self { agg, context }\n    }\n}\n\nimpl Collector for DistributedAggregationCollector {\n    type Fruit = IntermediateAggregationResults;\n\n    type Child = AggregationSegmentCollector;\n\n    fn for_segment(\n        &self,\n        segment_local_id: crate::SegmentOrdinal,\n        reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        AggregationSegmentCollector::from_agg_req_and_reader(\n            &self.agg,\n            reader,\n            segment_local_id,\n            &self.context,\n        )\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit> {\n        merge_fruits(segment_fruits)\n    }\n}\n\nimpl Collector for AggregationCollector {\n    type Fruit = AggregationResults;\n\n    type Child = AggregationSegmentCollector;\n\n    fn for_segment(\n        &self,\n        segment_local_id: crate::SegmentOrdinal,\n        reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        AggregationSegmentCollector::from_agg_req_and_reader(\n            &self.agg,\n            reader,\n            segment_local_id,\n            &self.context,\n        )\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit> {\n        let res = merge_fruits(segment_fruits)?;\n        res.into_final_result(self.agg.clone(), self.context.limits.clone())\n    }\n}\n\nfn merge_fruits(\n    mut segment_fruits: Vec<crate::Result<IntermediateAggregationResults>>,\n) -> crate::Result<IntermediateAggregationResults> {\n    if let Some(fruit) = segment_fruits.pop() {\n        let mut fruit = fruit?;\n        for next_fruit in segment_fruits {\n            fruit.merge_fruits(next_fruit?)?;\n        }\n        Ok(fruit)\n    } else {\n        Ok(IntermediateAggregationResults::default())\n    }\n}\n\n/// `AggregationSegmentCollector` does the aggregation collection on a segment.\npub struct AggregationSegmentCollector {\n    aggs_with_accessor: AggregationsSegmentCtx,\n    agg_collector: LowCardCachedSubAggs,\n    error: Option<TantivyError>,\n}\n\nimpl AggregationSegmentCollector {\n    /// Creates an `AggregationSegmentCollector from` an [`Aggregations`] request and a segment\n    /// reader. Also includes validation, e.g. checking field types and existence.\n    pub fn from_agg_req_and_reader(\n        agg: &Aggregations,\n        reader: &SegmentReader,\n        segment_ordinal: SegmentOrdinal,\n        context: &AggContextParams,\n    ) -> crate::Result<Self> {\n        let mut agg_data =\n            build_aggregations_data_from_req(agg, reader, segment_ordinal, context.clone())?;\n        let mut result =\n            LowCardCachedSubAggs::new(build_segment_agg_collectors_root(&mut agg_data)?);\n        result\n            .get_sub_agg_collector()\n            .prepare_max_bucket(0, &agg_data)?; // prepare for bucket zero\n\n        Ok(AggregationSegmentCollector {\n            aggs_with_accessor: agg_data,\n            agg_collector: result,\n            error: None,\n        })\n    }\n}\n\nimpl SegmentCollector for AggregationSegmentCollector {\n    type Fruit = crate::Result<IntermediateAggregationResults>;\n\n    #[inline]\n    fn collect(&mut self, doc: DocId, _score: crate::Score) {\n        if self.error.is_some() {\n            return;\n        }\n        self.agg_collector.push(0, doc);\n        match self\n            .agg_collector\n            .check_flush_local(&mut self.aggs_with_accessor)\n        {\n            Ok(_) => {}\n            Err(e) => {\n                self.error = Some(e);\n            }\n        }\n    }\n    fn collect_block(&mut self, docs: &[DocId]) {\n        if self.error.is_some() {\n            return;\n        }\n\n        match self.agg_collector.get_sub_agg_collector().collect(\n            0,\n            docs,\n            &mut self.aggs_with_accessor,\n        ) {\n            Ok(_) => {}\n            Err(e) => {\n                self.error = Some(e);\n            }\n        }\n    }\n\n    fn harvest(mut self) -> Self::Fruit {\n        if let Some(err) = self.error {\n            return Err(err);\n        }\n        self.agg_collector.flush(&mut self.aggs_with_accessor)?;\n\n        let mut sub_aggregation_res = IntermediateAggregationResults::default();\n        self.agg_collector\n            .get_sub_agg_collector()\n            .add_intermediate_aggregation_result(\n                &self.aggs_with_accessor,\n                &mut sub_aggregation_res,\n                0,\n            )?;\n\n        Ok(sub_aggregation_res)\n    }\n}\n"
  },
  {
    "path": "src/aggregation/date.rs",
    "content": "use time::format_description::well_known::Rfc3339;\nuse time::OffsetDateTime;\n\nuse crate::TantivyError;\n\npub(crate) fn format_date(val: i64) -> crate::Result<String> {\n    let datetime = OffsetDateTime::from_unix_timestamp_nanos(val as i128).map_err(|err| {\n        TantivyError::InvalidArgument(format!(\n            \"Could not convert {val:?} to OffsetDateTime, err {err:?}\"\n        ))\n    })?;\n    let key_as_string = datetime\n        .format(&Rfc3339)\n        .map_err(|_err| TantivyError::InvalidArgument(\"Could not serialize date\".to_string()))?;\n    Ok(key_as_string)\n}\n"
  },
  {
    "path": "src/aggregation/error.rs",
    "content": "use common::ByteCount;\n\nuse super::bucket::DateHistogramParseError;\n\n/// Error that may occur when opening a directory\n#[derive(Debug, Clone, PartialEq, Eq, Error)]\npub enum AggregationError {\n    /// InternalError Aggregation Request\n    #[error(\"InternalError: {0:?}\")]\n    InternalError(String),\n    /// Invalid Aggregation Request\n    #[error(\"InvalidRequest: {0:?}\")]\n    InvalidRequest(String),\n    /// Date histogram parse error\n    #[error(\"Date histogram parse error: {0:?}\")]\n    DateHistogramParseError(#[from] DateHistogramParseError),\n    /// Memory limit exceeded\n    #[error(\n        \"Aborting aggregation because memory limit was exceeded. Limit: {limit:?}, Current: \\\n         {current:?}\"\n    )]\n    MemoryExceeded {\n        /// Memory consumption limit\n        limit: ByteCount,\n        /// Current memory consumption\n        current: ByteCount,\n    },\n    /// Bucket limit exceeded\n    #[error(\n        \"Aborting aggregation because bucket limit was exceeded. Limit: {limit:?}, Current: \\\n         {current:?}\"\n    )]\n    BucketLimitExceeded {\n        /// Bucket limit\n        limit: u32,\n        /// Current num buckets\n        current: u32,\n    },\n}\n"
  },
  {
    "path": "src/aggregation/intermediate_agg_result.rs",
    "content": "//! Contains the intermediate aggregation tree, that can be merged.\n//! Intermediate aggregation results can be used to merge results between segments or between\n//! indices.\n\nuse std::cmp::Ordering;\nuse std::collections::hash_map::Entry;\nuse std::hash::Hash;\nuse std::net::Ipv6Addr;\n\nuse columnar::ColumnType;\nuse itertools::Itertools;\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\n\nuse super::agg_req::{Aggregation, AggregationVariants, Aggregations};\nuse super::agg_result::{AggregationResult, BucketResult, MetricResult, RangeBucketEntry};\nuse super::bucket::{\n    composite_intermediate_key_ordering, cut_off_buckets, get_agg_name_and_property,\n    intermediate_histogram_buckets_to_final_buckets, CompositeAggregation, GetDocCount,\n    MissingOrder, Order, OrderTarget, RangeAggregation, TermsAggregation,\n};\nuse super::metric::{\n    IntermediateAverage, IntermediateCount, IntermediateExtendedStats, IntermediateMax,\n    IntermediateMin, IntermediateStats, IntermediateSum, PercentilesCollector, TopHitsTopNComputer,\n};\nuse super::segment_agg_result::AggregationLimitsGuard;\nuse super::{format_date, AggregationError, Key, SerializedKey};\nuse crate::aggregation::agg_result::{\n    AggregationResults, BucketEntries, BucketEntry, CompositeBucketEntry, FilterBucketResult,\n};\nuse crate::aggregation::bucket::TermsAggregationInternal;\nuse crate::aggregation::metric::CardinalityCollector;\nuse crate::TantivyError;\n\n/// Contains the intermediate aggregation result, which is optimized to be merged with other\n/// intermediate results.\n///\n/// Notice: This struct should not be de/serialized via JSON format.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateAggregationResults {\n    pub(crate) aggs_res: FxHashMap<String, IntermediateAggregationResult>,\n}\n\n#[derive(Clone, Debug, Serialize, Deserialize, PartialOrd, PartialEq)]\n/// The key to identify a bucket.\n/// This might seem redundant with `Key`, but the point is to have a different\n/// Serialize implementation.\npub enum IntermediateKey {\n    /// Ip Addr key\n    IpAddr(Ipv6Addr),\n    /// Bool key\n    Bool(bool),\n    /// String key\n    Str(String),\n    /// `f64` key\n    F64(f64),\n    /// `i64` key\n    I64(i64),\n    /// `u64` key\n    U64(u64),\n}\nimpl From<Key> for IntermediateKey {\n    fn from(value: Key) -> Self {\n        match value {\n            Key::Str(s) => Self::Str(s),\n            Key::F64(f) => Self::F64(f),\n            Key::U64(f) => Self::U64(f),\n            Key::I64(f) => Self::I64(f),\n        }\n    }\n}\nimpl From<IntermediateKey> for Key {\n    fn from(value: IntermediateKey) -> Self {\n        match value {\n            IntermediateKey::Str(s) => Self::Str(s),\n            IntermediateKey::IpAddr(s) => {\n                // Prefer to use the IPv4 representation if possible\n                if let Some(ip) = s.to_ipv4_mapped() {\n                    Self::Str(ip.to_string())\n                } else {\n                    Self::Str(s.to_string())\n                }\n            }\n            IntermediateKey::F64(f) => Self::F64(f),\n            IntermediateKey::Bool(f) => Self::U64(f as u64),\n            IntermediateKey::U64(f) => Self::U64(f),\n            IntermediateKey::I64(f) => Self::I64(f),\n        }\n    }\n}\n\nimpl Eq for IntermediateKey {}\n\nimpl std::fmt::Display for IntermediateKey {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            IntermediateKey::Str(val) => f.write_str(val),\n            IntermediateKey::F64(val) => f.write_str(&val.to_string()),\n            IntermediateKey::U64(val) => f.write_str(&val.to_string()),\n            IntermediateKey::I64(val) => f.write_str(&val.to_string()),\n            IntermediateKey::Bool(val) => f.write_str(&val.to_string()),\n            IntermediateKey::IpAddr(val) => f.write_str(&val.to_string()),\n        }\n    }\n}\n\nimpl std::hash::Hash for IntermediateKey {\n    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {\n        core::mem::discriminant(self).hash(state);\n        match self {\n            IntermediateKey::Str(text) => text.hash(state),\n            IntermediateKey::F64(val) => val.to_bits().hash(state),\n            IntermediateKey::U64(val) => val.hash(state),\n            IntermediateKey::I64(val) => val.hash(state),\n            IntermediateKey::Bool(val) => val.hash(state),\n            IntermediateKey::IpAddr(val) => val.hash(state),\n        }\n    }\n}\n\nimpl IntermediateAggregationResults {\n    /// Returns a reference to the intermediate aggregation result for the given key.\n    pub fn get(&self, key: &str) -> Option<&IntermediateAggregationResult> {\n        self.aggs_res.get(key)\n    }\n\n    /// Removes and returns the intermediate aggregation result for the given key.\n    pub fn remove(&mut self, key: &str) -> Option<IntermediateAggregationResult> {\n        self.aggs_res.remove(key)\n    }\n\n    /// Returns an iterator over the keys in the intermediate aggregation results.\n    pub fn keys(&self) -> impl Iterator<Item = &String> {\n        self.aggs_res.keys()\n    }\n\n    /// Add a result\n    pub fn push(&mut self, key: String, value: IntermediateAggregationResult) -> crate::Result<()> {\n        let entry = self.aggs_res.entry(key);\n        match entry {\n            Entry::Occupied(mut e) => {\n                // In case of term aggregation over different types, we need to merge the results.\n                e.get_mut().merge_fruits(value)?;\n            }\n            Entry::Vacant(e) => {\n                e.insert(value);\n            }\n        }\n        Ok(())\n    }\n\n    /// Convert intermediate result and its aggregation request to the final result.\n    pub fn into_final_result(\n        self,\n        req: Aggregations,\n        mut limits: AggregationLimitsGuard,\n    ) -> crate::Result<AggregationResults> {\n        let res = self.into_final_result_internal(&req, &mut limits)?;\n        let bucket_count = res.get_bucket_count() as u32;\n        if bucket_count > limits.get_bucket_limit() {\n            return Err(TantivyError::AggregationError(\n                AggregationError::BucketLimitExceeded {\n                    limit: limits.get_bucket_limit(),\n                    current: bucket_count,\n                },\n            ));\n        }\n        Ok(res)\n    }\n\n    /// Convert intermediate result and its aggregation request to the final result.\n    pub(crate) fn into_final_result_internal(\n        self,\n        req: &Aggregations,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<AggregationResults> {\n        let mut results: FxHashMap<String, AggregationResult> = FxHashMap::default();\n        for (key, agg_res) in self.aggs_res.into_iter() {\n            let req = req.get(key.as_str()).unwrap_or_else(|| {\n                panic!(\n                    \"Could not find key {:?} in request keys {:?}. This probably means that \\\n                     add_intermediate_aggregation_result passed the wrong agg object.\",\n                    key,\n                    req.keys().collect::<Vec<_>>()\n                )\n            });\n            results.insert(key, agg_res.into_final_result(req, limits)?);\n        }\n        // Handle empty results\n        if results.len() != req.len() {\n            for (key, req) in req.iter() {\n                if !results.contains_key(key) {\n                    let empty_res = empty_from_req(req);\n                    results.insert(key.to_string(), empty_res.into_final_result(req, limits)?);\n                }\n            }\n        }\n\n        Ok(AggregationResults(results))\n    }\n\n    pub(crate) fn empty_from_req(req: &Aggregations) -> Self {\n        let mut aggs_res: FxHashMap<String, IntermediateAggregationResult> = FxHashMap::default();\n        for (key, req) in req.iter() {\n            let empty_res = empty_from_req(req);\n            aggs_res.insert(key.to_string(), empty_res);\n        }\n\n        Self { aggs_res }\n    }\n\n    /// Merge another intermediate aggregation result into this result.\n    pub fn merge_fruits(&mut self, mut other: IntermediateAggregationResults) -> crate::Result<()> {\n        for (key, left) in self.aggs_res.iter_mut() {\n            if let Some(key) = other.aggs_res.remove(key) {\n                left.merge_fruits(key)?;\n            }\n        }\n        // Move remainder of other aggs_res into self.\n        // Note: Currently we don't expect this to happen, as we create empty intermediate results\n        // via [IntermediateAggregationResults::empty_from_req].\n        for (key, value) in other.aggs_res {\n            self.aggs_res.insert(key, value);\n        }\n        Ok(())\n    }\n}\n\npub(crate) fn empty_from_req(req: &Aggregation) -> IntermediateAggregationResult {\n    use AggregationVariants::*;\n    match req.agg {\n        Terms(_) => IntermediateAggregationResult::Bucket(IntermediateBucketResult::Terms {\n            buckets: Default::default(),\n        }),\n        Range(_) => IntermediateAggregationResult::Bucket(IntermediateBucketResult::Range(\n            Default::default(),\n        )),\n        Histogram(_) => {\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Histogram {\n                buckets: Vec::new(),\n                is_date_agg: false,\n            })\n        }\n        DateHistogram(_) => {\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Histogram {\n                buckets: Vec::new(),\n                is_date_agg: true,\n            })\n        }\n        Average(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Average(\n            IntermediateAverage::default(),\n        )),\n        Count(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Count(\n            IntermediateCount::default(),\n        )),\n        Max(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Max(\n            IntermediateMax::default(),\n        )),\n        Min(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Min(\n            IntermediateMin::default(),\n        )),\n        Stats(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Stats(\n            IntermediateStats::default(),\n        )),\n        ExtendedStats(_) => IntermediateAggregationResult::Metric(\n            IntermediateMetricResult::ExtendedStats(IntermediateExtendedStats::default()),\n        ),\n        Sum(_) => IntermediateAggregationResult::Metric(IntermediateMetricResult::Sum(\n            IntermediateSum::default(),\n        )),\n        Percentiles(_) => IntermediateAggregationResult::Metric(\n            IntermediateMetricResult::Percentiles(PercentilesCollector::default()),\n        ),\n        TopHits(ref req) => IntermediateAggregationResult::Metric(\n            IntermediateMetricResult::TopHits(TopHitsTopNComputer::new(req)),\n        ),\n        Cardinality(_) => IntermediateAggregationResult::Metric(\n            IntermediateMetricResult::Cardinality(CardinalityCollector::default()),\n        ),\n        Filter(_) => IntermediateAggregationResult::Bucket(IntermediateBucketResult::Filter {\n            doc_count: 0,\n            sub_aggregations: IntermediateAggregationResults::default(),\n        }),\n        Composite(_) => {\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Composite {\n                buckets: IntermediateCompositeBucketResult::default(),\n            })\n        }\n    }\n}\n\n/// An aggregation is either a bucket or a metric.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[allow(clippy::large_enum_variant)]\npub enum IntermediateAggregationResult {\n    /// Bucket variant\n    Bucket(IntermediateBucketResult),\n    /// Metric variant\n    Metric(IntermediateMetricResult),\n}\n\nimpl IntermediateAggregationResult {\n    pub(crate) fn into_final_result(\n        self,\n        req: &Aggregation,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<AggregationResult> {\n        let res = match self {\n            IntermediateAggregationResult::Bucket(bucket) => {\n                AggregationResult::BucketResult(bucket.into_final_bucket_result(req, limits)?)\n            }\n            IntermediateAggregationResult::Metric(metric) => {\n                AggregationResult::MetricResult(metric.into_final_metric_result(req))\n            }\n        };\n        Ok(res)\n    }\n    fn merge_fruits(&mut self, other: IntermediateAggregationResult) -> crate::Result<()> {\n        match (self, other) {\n            (\n                IntermediateAggregationResult::Bucket(b1),\n                IntermediateAggregationResult::Bucket(b2),\n            ) => b1.merge_fruits(b2),\n            (\n                IntermediateAggregationResult::Metric(m1),\n                IntermediateAggregationResult::Metric(m2),\n            ) => m1.merge_fruits(m2),\n            _ => panic!(\"aggregation result type mismatch (mixed metric and buckets)\"),\n        }\n    }\n}\n\n/// Holds the intermediate data for metric results\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub enum IntermediateMetricResult {\n    /// Intermediate average result.\n    Percentiles(PercentilesCollector),\n    /// Intermediate average result.\n    Average(IntermediateAverage),\n    /// Intermediate count result.\n    Count(IntermediateCount),\n    /// Intermediate max result.\n    Max(IntermediateMax),\n    /// Intermediate min result.\n    Min(IntermediateMin),\n    /// Intermediate stats result.\n    Stats(IntermediateStats),\n    /// Intermediate stats result.\n    ExtendedStats(IntermediateExtendedStats),\n    /// Intermediate sum result.\n    Sum(IntermediateSum),\n    /// Intermediate top_hits result\n    TopHits(TopHitsTopNComputer),\n    /// Intermediate cardinality result\n    Cardinality(CardinalityCollector),\n}\n\nimpl IntermediateMetricResult {\n    fn into_final_metric_result(self, req: &Aggregation) -> MetricResult {\n        match self {\n            IntermediateMetricResult::Average(intermediate_avg) => {\n                MetricResult::Average(intermediate_avg.finalize().into())\n            }\n            IntermediateMetricResult::Count(intermediate_count) => {\n                MetricResult::Count(intermediate_count.finalize().into())\n            }\n            IntermediateMetricResult::Max(intermediate_max) => {\n                MetricResult::Max(intermediate_max.finalize().into())\n            }\n            IntermediateMetricResult::Min(intermediate_min) => {\n                MetricResult::Min(intermediate_min.finalize().into())\n            }\n            IntermediateMetricResult::Stats(intermediate_stats) => {\n                MetricResult::Stats(intermediate_stats.finalize())\n            }\n            IntermediateMetricResult::ExtendedStats(intermediate_stats) => {\n                MetricResult::ExtendedStats(intermediate_stats.finalize())\n            }\n            IntermediateMetricResult::Sum(intermediate_sum) => {\n                MetricResult::Sum(intermediate_sum.finalize().into())\n            }\n            IntermediateMetricResult::Percentiles(percentiles) => MetricResult::Percentiles(\n                percentiles\n                    .into_final_result(req.agg.as_percentile().expect(\"unexpected metric type\")),\n            ),\n            IntermediateMetricResult::TopHits(top_hits) => {\n                MetricResult::TopHits(top_hits.into_final_result())\n            }\n            IntermediateMetricResult::Cardinality(cardinality) => {\n                MetricResult::Cardinality(cardinality.finalize().into())\n            }\n        }\n    }\n\n    // TODO: this is our top-of-the-chain fruit merge mech\n    fn merge_fruits(&mut self, other: IntermediateMetricResult) -> crate::Result<()> {\n        match (self, other) {\n            (\n                IntermediateMetricResult::Average(avg_left),\n                IntermediateMetricResult::Average(avg_right),\n            ) => {\n                avg_left.merge_fruits(avg_right);\n            }\n            (\n                IntermediateMetricResult::Count(count_left),\n                IntermediateMetricResult::Count(count_right),\n            ) => {\n                count_left.merge_fruits(count_right);\n            }\n            (IntermediateMetricResult::Max(max_left), IntermediateMetricResult::Max(max_right)) => {\n                max_left.merge_fruits(max_right);\n            }\n            (IntermediateMetricResult::Min(min_left), IntermediateMetricResult::Min(min_right)) => {\n                min_left.merge_fruits(min_right);\n            }\n            (\n                IntermediateMetricResult::Stats(stats_left),\n                IntermediateMetricResult::Stats(stats_right),\n            ) => {\n                stats_left.merge_fruits(stats_right);\n            }\n            (\n                IntermediateMetricResult::ExtendedStats(extended_stats_left),\n                IntermediateMetricResult::ExtendedStats(extended_stats_right),\n            ) => {\n                extended_stats_left.merge_fruits(extended_stats_right);\n            }\n            (IntermediateMetricResult::Sum(sum_left), IntermediateMetricResult::Sum(sum_right)) => {\n                sum_left.merge_fruits(sum_right);\n            }\n            (\n                IntermediateMetricResult::Percentiles(left),\n                IntermediateMetricResult::Percentiles(right),\n            ) => {\n                left.merge_fruits(right)?;\n            }\n            (IntermediateMetricResult::TopHits(left), IntermediateMetricResult::TopHits(right)) => {\n                left.merge_fruits(right)?;\n            }\n            (\n                IntermediateMetricResult::Cardinality(left),\n                IntermediateMetricResult::Cardinality(right),\n            ) => {\n                left.merge_fruits(right)?;\n            }\n            _ => {\n                panic!(\"incompatible fruit types in tree or missing merge_fruits handler\");\n            }\n        }\n\n        Ok(())\n    }\n}\n\n/// The intermediate bucket results. Internally they can be easily merged via the keys of the\n/// buckets.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub enum IntermediateBucketResult {\n    /// This is the range entry for a bucket, which contains a key, count, from, to, and optionally\n    /// sub_aggregations.\n    Range(IntermediateRangeBucketResult),\n    /// This is the histogram entry for a bucket, which contains a key, count, and optionally\n    /// sub_aggregations.\n    Histogram {\n        /// The column_type of the underlying `Column` is DateTime\n        is_date_agg: bool,\n        /// The histogram buckets\n        buckets: Vec<IntermediateHistogramBucketEntry>,\n    },\n    /// Term aggregation\n    Terms {\n        /// The term buckets\n        buckets: IntermediateTermBucketResult,\n    },\n    /// Filter aggregation - a single bucket with sub-aggregations\n    Filter {\n        /// Document count in the filter bucket\n        doc_count: u64,\n        /// Sub-aggregation results\n        sub_aggregations: IntermediateAggregationResults,\n    },\n    /// Composite aggregation\n    Composite {\n        /// The composite buckets\n        buckets: IntermediateCompositeBucketResult,\n    },\n}\n\nimpl IntermediateBucketResult {\n    pub(crate) fn into_final_bucket_result(\n        self,\n        req: &Aggregation,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<BucketResult> {\n        match self {\n            IntermediateBucketResult::Range(range_res) => {\n                let mut buckets: Vec<RangeBucketEntry> = range_res\n                    .buckets\n                    .into_values()\n                    .map(|bucket| {\n                        bucket.into_final_bucket_entry(\n                            req.sub_aggregation(),\n                            req.agg\n                                .as_range()\n                                .expect(\"unexpected aggregation, expected histogram aggregation\"),\n                            range_res.column_type,\n                            limits,\n                        )\n                    })\n                    .collect::<crate::Result<Vec<_>>>()?;\n\n                buckets.sort_by(|left, right| {\n                    left.from\n                        .unwrap_or(f64::MIN)\n                        .total_cmp(&right.from.unwrap_or(f64::MIN))\n                });\n\n                let is_keyed = req\n                    .agg\n                    .as_range()\n                    .expect(\"unexpected aggregation, expected range aggregation\")\n                    .keyed;\n                let buckets = if is_keyed {\n                    let mut bucket_map =\n                        FxHashMap::with_capacity_and_hasher(buckets.len(), Default::default());\n                    for bucket in buckets {\n                        bucket_map.insert(bucket.key.to_string(), bucket);\n                    }\n                    BucketEntries::HashMap(bucket_map)\n                } else {\n                    BucketEntries::Vec(buckets)\n                };\n                Ok(BucketResult::Range { buckets })\n            }\n            IntermediateBucketResult::Histogram {\n                is_date_agg,\n                buckets,\n            } => {\n                let histogram_req = &req\n                    .agg\n                    .as_histogram()?\n                    .expect(\"unexpected aggregation, expected histogram aggregation\");\n                let buckets = intermediate_histogram_buckets_to_final_buckets(\n                    buckets,\n                    is_date_agg,\n                    histogram_req,\n                    req.sub_aggregation(),\n                    limits,\n                )?;\n\n                let buckets = if histogram_req.keyed {\n                    let mut bucket_map =\n                        FxHashMap::with_capacity_and_hasher(buckets.len(), Default::default());\n                    for bucket in buckets {\n                        bucket_map.insert(bucket.key.to_string(), bucket);\n                    }\n                    BucketEntries::HashMap(bucket_map)\n                } else {\n                    BucketEntries::Vec(buckets)\n                };\n                Ok(BucketResult::Histogram { buckets })\n            }\n            IntermediateBucketResult::Terms { buckets: terms } => terms.into_final_result(\n                req.agg\n                    .as_term()\n                    .expect(\"unexpected aggregation, expected term aggregation\"),\n                req.sub_aggregation(),\n                limits,\n            ),\n            IntermediateBucketResult::Filter {\n                doc_count,\n                sub_aggregations,\n            } => {\n                // Convert sub-aggregation results to final format\n                let final_sub_aggregations = sub_aggregations\n                    .into_final_result(req.sub_aggregation().clone(), limits.clone())?;\n                Ok(BucketResult::Filter(FilterBucketResult {\n                    doc_count,\n                    sub_aggregations: final_sub_aggregations,\n                }))\n            }\n            IntermediateBucketResult::Composite { buckets } => {\n                let composite_req = req\n                    .agg\n                    .as_composite()\n                    .expect(\"unexpected aggregation, expected composite aggregation\");\n                buckets.into_final_result(composite_req, req.sub_aggregation(), limits)\n            }\n        }\n    }\n\n    fn merge_fruits(&mut self, other: IntermediateBucketResult) -> crate::Result<()> {\n        match (self, other) {\n            (\n                IntermediateBucketResult::Terms {\n                    buckets: term_res_left,\n                },\n                IntermediateBucketResult::Terms {\n                    buckets: term_res_right,\n                },\n            ) => {\n                merge_maps(&mut term_res_left.entries, term_res_right.entries)?;\n                term_res_left.sum_other_doc_count += term_res_right.sum_other_doc_count;\n                term_res_left.doc_count_error_upper_bound +=\n                    term_res_right.doc_count_error_upper_bound;\n            }\n\n            (\n                IntermediateBucketResult::Range(range_res_left),\n                IntermediateBucketResult::Range(range_res_right),\n            ) => {\n                merge_maps(&mut range_res_left.buckets, range_res_right.buckets)?;\n            }\n            (\n                IntermediateBucketResult::Histogram {\n                    buckets: buckets_left,\n                    is_date_agg: _,\n                },\n                IntermediateBucketResult::Histogram {\n                    buckets: buckets_right,\n                    is_date_agg: _,\n                },\n            ) => {\n                let buckets: Result<Vec<IntermediateHistogramBucketEntry>, TantivyError> =\n                    buckets_left\n                        .drain(..)\n                        .merge_join_by(buckets_right, |left, right| {\n                            left.key.partial_cmp(&right.key).unwrap_or(Ordering::Equal)\n                        })\n                        .map(|either| match either {\n                            itertools::EitherOrBoth::Both(mut left, right) => {\n                                left.merge_fruits(right)?;\n                                Ok(left)\n                            }\n                            itertools::EitherOrBoth::Left(left) => Ok(left),\n                            itertools::EitherOrBoth::Right(right) => Ok(right),\n                        })\n                        .collect::<Result<_, _>>();\n\n                *buckets_left = buckets?;\n            }\n            (\n                IntermediateBucketResult::Filter {\n                    doc_count: doc_count_left,\n                    sub_aggregations: sub_aggs_left,\n                },\n                IntermediateBucketResult::Filter {\n                    doc_count: doc_count_right,\n                    sub_aggregations: sub_aggs_right,\n                },\n            ) => {\n                *doc_count_left += doc_count_right;\n                sub_aggs_left.merge_fruits(sub_aggs_right)?;\n            }\n            (\n                IntermediateBucketResult::Composite {\n                    buckets: composite_left,\n                },\n                IntermediateBucketResult::Composite {\n                    buckets: composite_right,\n                },\n            ) => {\n                composite_left.merge_fruits(composite_right)?;\n            }\n            (IntermediateBucketResult::Range(_), _) => {\n                panic!(\"try merge on different types\")\n            }\n            (IntermediateBucketResult::Histogram { .. }, _) => {\n                panic!(\"try merge on different types\")\n            }\n            (IntermediateBucketResult::Terms { .. }, _) => {\n                panic!(\"try merge on different types\")\n            }\n            (IntermediateBucketResult::Filter { .. }, _) => {\n                panic!(\"try merge on different types\")\n            }\n            (IntermediateBucketResult::Composite { .. }, _) => {\n                panic!(\"try merge on different types\")\n            }\n        }\n        Ok(())\n    }\n}\n\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\n/// Range aggregation including error counts\npub struct IntermediateRangeBucketResult {\n    pub(crate) buckets: FxHashMap<SerializedKey, IntermediateRangeBucketEntry>,\n    pub(crate) column_type: Option<ColumnType>,\n}\n\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\n/// Term aggregation including error counts\npub struct IntermediateTermBucketResult {\n    pub(crate) entries: FxHashMap<IntermediateKey, IntermediateTermBucketEntry>,\n    pub(crate) sum_other_doc_count: u64,\n    pub(crate) doc_count_error_upper_bound: u64,\n}\n\nimpl IntermediateTermBucketResult {\n    /// Returns a reference to the map of bucket entries keyed by [`IntermediateKey`].\n    pub fn entries(&self) -> &FxHashMap<IntermediateKey, IntermediateTermBucketEntry> {\n        &self.entries\n    }\n\n    /// Returns the count of documents not included in the returned buckets.\n    pub fn sum_other_doc_count(&self) -> u64 {\n        self.sum_other_doc_count\n    }\n\n    /// Returns the upper bound of the error on document counts in the returned buckets.\n    pub fn doc_count_error_upper_bound(&self) -> u64 {\n        self.doc_count_error_upper_bound\n    }\n\n    pub(crate) fn into_final_result(\n        self,\n        req: &TermsAggregation,\n        sub_aggregation_req: &Aggregations,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<BucketResult> {\n        let req = TermsAggregationInternal::from_req(req);\n        let mut buckets: Vec<BucketEntry> = self\n            .entries\n            .into_iter()\n            .filter(|bucket| bucket.1.doc_count as u64 >= req.min_doc_count)\n            .map(|(key, entry)| {\n                let key_as_string = match key {\n                    IntermediateKey::Bool(key) => {\n                        let val = if key { \"true\" } else { \"false\" };\n                        Some(val.to_string())\n                    }\n                    _ => None,\n                };\n                Ok(BucketEntry {\n                    key_as_string,\n                    key: key.into(),\n                    doc_count: entry.doc_count as u64,\n                    sub_aggregation: entry\n                        .sub_aggregation\n                        .into_final_result_internal(sub_aggregation_req, limits)?,\n                })\n            })\n            .collect::<crate::Result<_>>()?;\n\n        let order = req.order.order;\n        match req.order.target {\n            OrderTarget::Key => {\n                buckets.sort_by(|left, right| {\n                    if req.order.order == Order::Asc {\n                        left.key.partial_cmp(&right.key)\n                    } else {\n                        right.key.partial_cmp(&left.key)\n                    }\n                    .expect(\"expected type string, which is always sortable\")\n                });\n            }\n            OrderTarget::Count => {\n                if req.order.order == Order::Desc {\n                    buckets.sort_unstable_by_key(|bucket| std::cmp::Reverse(bucket.doc_count()));\n                } else {\n                    buckets.sort_unstable_by_key(|bucket| bucket.doc_count());\n                }\n            }\n            OrderTarget::SubAggregation(name) => {\n                let (agg_name, agg_property) = get_agg_name_and_property(&name);\n                let mut buckets_with_val = buckets\n                    .into_iter()\n                    .map(|bucket| {\n                        let val = bucket\n                            .sub_aggregation\n                            .get_value_from_aggregation(agg_name, agg_property)?\n                            .unwrap_or(f64::MIN);\n                        Ok((bucket, val))\n                    })\n                    .collect::<crate::Result<Vec<_>>>()?;\n\n                buckets_with_val.sort_by(|(_, val1), (_, val2)| match &order {\n                    Order::Desc => val2.total_cmp(val1),\n                    Order::Asc => val1.total_cmp(val2),\n                });\n                buckets = buckets_with_val\n                    .into_iter()\n                    .map(|(bucket, _val)| bucket)\n                    .collect_vec();\n            }\n        }\n\n        // We ignore _term_doc_count_before_cutoff here, because it increases the upperbound error\n        // only for terms that didn't make it into the top N.\n        //\n        // This can be interesting, as a value of quality of the results, but not good to check the\n        // actual error count for the returned terms.\n        let (_term_doc_count_before_cutoff, sum_other_doc_count) =\n            cut_off_buckets(&mut buckets, req.size as usize);\n\n        let doc_count_error_upper_bound = if req.show_term_doc_count_error {\n            Some(self.doc_count_error_upper_bound)\n        } else {\n            None\n        };\n\n        Ok(BucketResult::Terms {\n            buckets,\n            sum_other_doc_count: self.sum_other_doc_count + sum_other_doc_count,\n            doc_count_error_upper_bound,\n        })\n    }\n}\n\ntrait MergeFruits {\n    fn merge_fruits(&mut self, other: Self) -> crate::Result<()>;\n}\n\nfn merge_maps<V: MergeFruits + Clone, T: Eq + PartialEq + Hash>(\n    entries_left: &mut FxHashMap<T, V>,\n    mut entries_right: FxHashMap<T, V>,\n) -> crate::Result<()> {\n    for (name, entry_left) in entries_left.iter_mut() {\n        if let Some(entry_right) = entries_right.remove(name) {\n            entry_left.merge_fruits(entry_right)?;\n        }\n    }\n\n    for (key, res) in entries_right.into_iter() {\n        entries_left.entry(key).or_insert(res);\n    }\n    Ok(())\n}\n\n/// This is the histogram entry for a bucket, which contains a key, count, and optionally\n/// sub_aggregations.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateHistogramBucketEntry {\n    /// The unique the bucket is identified.\n    pub key: f64,\n    /// The number of documents in the bucket.\n    pub doc_count: u64,\n    /// The sub_aggregation in this bucket.\n    pub sub_aggregation: IntermediateAggregationResults,\n}\n\nimpl IntermediateHistogramBucketEntry {\n    pub(crate) fn into_final_bucket_entry(\n        self,\n        req: &Aggregations,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<BucketEntry> {\n        Ok(BucketEntry {\n            key_as_string: None,\n            key: Key::F64(self.key),\n            doc_count: self.doc_count,\n            sub_aggregation: self\n                .sub_aggregation\n                .into_final_result_internal(req, limits)?,\n        })\n    }\n}\n\n/// This is the range entry for a bucket, which contains a key, count, and optionally\n/// sub_aggregations.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateRangeBucketEntry {\n    /// The unique key the bucket is identified with.\n    pub key: IntermediateKey,\n    /// The number of documents in the bucket.\n    pub doc_count: u64,\n    /// The sub_aggregation in this bucket.\n    pub sub_aggregation_res: IntermediateAggregationResults,\n    /// The from range of the bucket. Equals `f64::MIN` when `None`.\n    pub from: Option<f64>,\n    /// The to range of the bucket. Equals `f64::MAX` when `None`.\n    pub to: Option<f64>,\n}\n\nimpl IntermediateRangeBucketEntry {\n    pub(crate) fn into_final_bucket_entry(\n        self,\n        req: &Aggregations,\n        _range_req: &RangeAggregation,\n        column_type: Option<ColumnType>,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<RangeBucketEntry> {\n        let mut range_bucket_entry = RangeBucketEntry {\n            key: self.key.into(),\n            doc_count: self.doc_count,\n            sub_aggregation: self\n                .sub_aggregation_res\n                .into_final_result_internal(req, limits)?,\n            to: self.to,\n            from: self.from,\n            to_as_string: None,\n            from_as_string: None,\n        };\n\n        // If we have a date type on the histogram buckets, we add the `key_as_string` field as\n        // rfc3339\n        if column_type == Some(ColumnType::DateTime) {\n            if let Some(val) = range_bucket_entry.to {\n                let key_as_string = format_date(val as i64)?;\n                range_bucket_entry.to_as_string = Some(key_as_string);\n            }\n            if let Some(val) = range_bucket_entry.from {\n                let key_as_string = format_date(val as i64)?;\n                range_bucket_entry.from_as_string = Some(key_as_string);\n            }\n        }\n\n        Ok(range_bucket_entry)\n    }\n}\n\n/// This is the term entry for a bucket, which contains a count, and optionally\n/// sub_aggregations.\n#[derive(Clone, Default, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateTermBucketEntry {\n    /// The number of documents in the bucket.\n    pub doc_count: u32,\n    /// The sub_aggregation in this bucket.\n    pub sub_aggregation: IntermediateAggregationResults,\n}\n\nimpl MergeFruits for IntermediateTermBucketEntry {\n    fn merge_fruits(&mut self, other: IntermediateTermBucketEntry) -> crate::Result<()> {\n        self.doc_count += other.doc_count;\n        self.sub_aggregation.merge_fruits(other.sub_aggregation)?;\n        Ok(())\n    }\n}\n\nimpl MergeFruits for IntermediateRangeBucketEntry {\n    fn merge_fruits(&mut self, other: IntermediateRangeBucketEntry) -> crate::Result<()> {\n        self.doc_count += other.doc_count;\n        self.sub_aggregation_res\n            .merge_fruits(other.sub_aggregation_res)?;\n        Ok(())\n    }\n}\n\nimpl MergeFruits for IntermediateHistogramBucketEntry {\n    fn merge_fruits(&mut self, other: IntermediateHistogramBucketEntry) -> crate::Result<()> {\n        self.doc_count += other.doc_count;\n        self.sub_aggregation.merge_fruits(other.sub_aggregation)?;\n        Ok(())\n    }\n}\n\n/// Entry for the composite bucket.\npub type IntermediateCompositeBucketEntry = IntermediateTermBucketEntry;\n\n/// The fully typed key for composite aggregation\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub enum CompositeIntermediateKey {\n    /// Bool key\n    Bool(bool),\n    /// String key\n    Str(String),\n    /// Float key\n    F64(f64),\n    /// Signed integer key\n    I64(i64),\n    /// Unsigned integer key\n    U64(u64),\n    /// DateTime key, nanoseconds since epoch\n    DateTime(i64),\n    /// IP Address key\n    IpAddr(Ipv6Addr),\n    /// Missing value key\n    Null,\n}\n\nimpl Eq for CompositeIntermediateKey {}\n\nimpl std::hash::Hash for CompositeIntermediateKey {\n    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {\n        core::mem::discriminant(self).hash(state);\n        match self {\n            CompositeIntermediateKey::Bool(val) => val.hash(state),\n            CompositeIntermediateKey::Str(text) => text.hash(state),\n            CompositeIntermediateKey::F64(val) => val.to_bits().hash(state),\n            CompositeIntermediateKey::U64(val) => val.hash(state),\n            CompositeIntermediateKey::I64(val) => val.hash(state),\n            CompositeIntermediateKey::DateTime(val) => val.hash(state),\n            CompositeIntermediateKey::IpAddr(val) => val.hash(state),\n            CompositeIntermediateKey::Null => {}\n        }\n    }\n}\n\n/// Composite aggregation page.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateCompositeBucketResult {\n    pub(crate) entries: FxHashMap<Vec<CompositeIntermediateKey>, IntermediateCompositeBucketEntry>,\n    pub(crate) target_size: u32,\n    pub(crate) orders: Vec<(Order, MissingOrder)>,\n}\n\nimpl IntermediateCompositeBucketResult {\n    pub(crate) fn into_final_result(\n        self,\n        req: &CompositeAggregation,\n        sub_aggregation_req: &Aggregations,\n        limits: &mut AggregationLimitsGuard,\n    ) -> crate::Result<BucketResult> {\n        let trimmed_entry_vec =\n            trim_composite_buckets(self.entries, &self.orders, self.target_size)?;\n        let after_key = if trimmed_entry_vec.len() == req.size as usize {\n            trimmed_entry_vec\n                .last()\n                .map(|bucket| {\n                    let (intermediate_key, _entry) = bucket;\n                    intermediate_key\n                        .iter()\n                        .enumerate()\n                        .map(|(idx, intermediate_key)| {\n                            let source = &req.sources[idx];\n                            (source.name().to_string(), intermediate_key.clone().into())\n                        })\n                        .collect()\n                })\n                .unwrap()\n        } else {\n            FxHashMap::default()\n        };\n\n        let buckets = trimmed_entry_vec\n            .into_iter()\n            .map(|(intermediate_key, entry)| {\n                let key = intermediate_key\n                    .into_iter()\n                    .enumerate()\n                    .map(|(idx, intermediate_key)| {\n                        let source = &req.sources[idx];\n                        (source.name().to_string(), intermediate_key.into())\n                    })\n                    .collect();\n                Ok(CompositeBucketEntry {\n                    key,\n                    doc_count: entry.doc_count as u64,\n                    sub_aggregation: entry\n                        .sub_aggregation\n                        .into_final_result_internal(sub_aggregation_req, limits)?,\n                })\n            })\n            .collect::<crate::Result<Vec<_>>>()?;\n\n        Ok(BucketResult::Composite { after_key, buckets })\n    }\n\n    fn merge_fruits(&mut self, other: IntermediateCompositeBucketResult) -> crate::Result<()> {\n        merge_maps(&mut self.entries, other.entries)?;\n        if self.entries.len() as u32 > 2 * self.target_size {\n            self.trim()?;\n        }\n        Ok(())\n    }\n\n    /// Trim the composite buckets to the target size, according to the ordering.\n    pub(crate) fn trim(&mut self) -> crate::Result<()> {\n        if self.entries.len() as u32 <= self.target_size {\n            return Ok(());\n        }\n\n        let sorted_entries = trim_composite_buckets(\n            std::mem::take(&mut self.entries),\n            &self.orders,\n            self.target_size,\n        )?;\n\n        self.entries = sorted_entries.into_iter().collect();\n        Ok(())\n    }\n}\n\nfn trim_composite_buckets(\n    entries: FxHashMap<Vec<CompositeIntermediateKey>, IntermediateCompositeBucketEntry>,\n    orders: &[(Order, MissingOrder)],\n    target_size: u32,\n) -> crate::Result<\n    Vec<(\n        Vec<CompositeIntermediateKey>,\n        IntermediateCompositeBucketEntry,\n    )>,\n> {\n    let mut entries: Vec<_> = entries.into_iter().collect();\n    let mut sort_error: Option<TantivyError> = None;\n    entries.sort_by(|(left_key, _), (right_key, _)| {\n        if sort_error.is_some() {\n            return Ordering::Equal;\n        }\n\n        for idx in 0..orders.len() {\n            match composite_intermediate_key_ordering(\n                &left_key[idx],\n                &right_key[idx],\n                orders[idx].0,\n                orders[idx].1,\n            ) {\n                Ok(ordering) if ordering != Ordering::Equal => return ordering,\n                Ok(_) => continue,\n                Err(err) => {\n                    sort_error = Some(err);\n                    break;\n                }\n            }\n        }\n        Ordering::Equal\n    });\n\n    if let Some(err) = sort_error {\n        return Err(err);\n    }\n\n    entries.truncate(target_size as usize);\n    Ok(entries)\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashMap;\n\n    use pretty_assertions::assert_eq;\n\n    use super::*;\n\n    fn get_sub_test_tree(data: &[(String, u64)]) -> IntermediateAggregationResults {\n        let mut map = HashMap::new();\n        let mut buckets = FxHashMap::default();\n        for (key, doc_count) in data {\n            buckets.insert(\n                key.to_string(),\n                IntermediateRangeBucketEntry {\n                    key: IntermediateKey::Str(key.to_string()),\n                    doc_count: *doc_count,\n                    sub_aggregation_res: Default::default(),\n                    from: None,\n                    to: None,\n                },\n            );\n        }\n        map.insert(\n            \"my_agg_level2\".to_string(),\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Range(\n                IntermediateRangeBucketResult {\n                    buckets,\n                    column_type: None,\n                },\n            )),\n        );\n        IntermediateAggregationResults {\n            aggs_res: map.into_iter().collect(),\n        }\n    }\n\n    fn get_intermediate_tree_with_ranges(\n        data: &[(String, u64, String, u64)],\n    ) -> IntermediateAggregationResults {\n        let mut map = HashMap::new();\n        let mut buckets: FxHashMap<_, _> = Default::default();\n        for (key, doc_count, sub_aggregation_key, sub_aggregation_count) in data {\n            buckets.insert(\n                key.to_string(),\n                IntermediateRangeBucketEntry {\n                    key: IntermediateKey::Str(key.to_string()),\n                    doc_count: *doc_count,\n                    from: None,\n                    to: None,\n                    sub_aggregation_res: get_sub_test_tree(&[(\n                        sub_aggregation_key.to_string(),\n                        *sub_aggregation_count,\n                    )]),\n                },\n            );\n        }\n        map.insert(\n            \"my_agg_level1\".to_string(),\n            IntermediateAggregationResult::Bucket(IntermediateBucketResult::Range(\n                IntermediateRangeBucketResult {\n                    buckets,\n                    column_type: None,\n                },\n            )),\n        );\n        IntermediateAggregationResults {\n            aggs_res: map.into_iter().collect(),\n        }\n    }\n\n    #[test]\n    fn test_merge_fruits_tree_1() {\n        let mut tree_left = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 50, \"1900\".to_string(), 25),\n            (\"blue\".to_string(), 30, \"1900\".to_string(), 30),\n        ]);\n        let tree_right = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 60, \"1900\".to_string(), 30),\n            (\"blue\".to_string(), 25, \"1900\".to_string(), 50),\n        ]);\n\n        tree_left.merge_fruits(tree_right).unwrap();\n\n        let tree_expected = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 110, \"1900\".to_string(), 55),\n            (\"blue\".to_string(), 55, \"1900\".to_string(), 80),\n        ]);\n\n        assert_eq!(tree_left, tree_expected);\n    }\n\n    #[test]\n    fn test_merge_fruits_tree_2() {\n        let mut tree_left = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 50, \"1900\".to_string(), 25),\n            (\"blue\".to_string(), 30, \"1900\".to_string(), 30),\n        ]);\n        let tree_right = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 60, \"1900\".to_string(), 30),\n            (\"green\".to_string(), 25, \"1900\".to_string(), 50),\n        ]);\n\n        tree_left.merge_fruits(tree_right).unwrap();\n\n        let tree_expected = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 110, \"1900\".to_string(), 55),\n            (\"blue\".to_string(), 30, \"1900\".to_string(), 30),\n            (\"green\".to_string(), 25, \"1900\".to_string(), 50),\n        ]);\n\n        assert_eq!(tree_left, tree_expected);\n    }\n\n    #[test]\n    fn test_merge_fruits_tree_empty() {\n        let mut tree_left = get_intermediate_tree_with_ranges(&[\n            (\"red\".to_string(), 50, \"1900\".to_string(), 25),\n            (\"blue\".to_string(), 30, \"1900\".to_string(), 30),\n        ]);\n\n        let orig = tree_left.clone();\n\n        tree_left\n            .merge_fruits(IntermediateAggregationResults::default())\n            .unwrap();\n\n        assert_eq!(tree_left, orig);\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/average.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::*;\n\n/// A single-value metric aggregation that computes the average of numeric values that are\n/// extracted from the aggregated documents.\n/// See [super::SingleMetricResult] for return value.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"avg\": {\n///         \"field\": \"score\"\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct AverageAggregation {\n    /// The field name to compute the average on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl AverageAggregation {\n    /// Creates a new [`AverageAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Intermediate result of the average aggregation that can be combined with other intermediate\n/// results.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateAverage {\n    stats: IntermediateStats,\n}\n\nimpl IntermediateAverage {\n    /// Creates a new [`IntermediateAverage`] instance from a [`SegmentStatsCollector`].\n    pub(crate) fn from_stats(stats: IntermediateStats) -> Self {\n        Self { stats }\n    }\n\n    /// Returns a reference to the underlying [`IntermediateStats`].\n    pub fn stats(&self) -> &IntermediateStats {\n        &self.stats\n    }\n\n    /// Merges the other intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateAverage) {\n        self.stats.merge_fruits(other.stats);\n    }\n    /// Computes the final average value.\n    pub fn finalize(&self) -> Option<f64> {\n        self.stats.finalize().avg\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn deserialization_with_missing_test1() {\n        let json = r#\"{\n            \"field\": \"score\",\n            \"missing\": \"10.0\"\n        }\"#;\n        let avg: AverageAggregation = serde_json::from_str(json).unwrap();\n        assert_eq!(avg.field, \"score\");\n        assert_eq!(avg.missing, Some(10.0));\n        // no dot\n        let json = r#\"{\n            \"field\": \"score\",\n            \"missing\": \"10\"\n        }\"#;\n        let avg: AverageAggregation = serde_json::from_str(json).unwrap();\n        assert_eq!(avg.field, \"score\");\n        assert_eq!(avg.missing, Some(10.0));\n\n        // from value\n        let avg: AverageAggregation = serde_json::from_value(json!({\n            \"field\": \"score_f64\",\n            \"missing\": 10u64,\n        }))\n        .unwrap();\n        assert_eq!(avg.missing, Some(10.0));\n        // from value\n        let avg: AverageAggregation = serde_json::from_value(json!({\n            \"field\": \"score_f64\",\n            \"missing\": 10u32,\n        }))\n        .unwrap();\n        assert_eq!(avg.missing, Some(10.0));\n        let avg: AverageAggregation = serde_json::from_value(json!({\n            \"field\": \"score_f64\",\n            \"missing\": 10i8,\n        }))\n        .unwrap();\n        assert_eq!(avg.missing, Some(10.0));\n    }\n\n    #[test]\n    fn deserialization_with_missing_test_fail() {\n        let json = r#\"{\n            \"field\": \"score\",\n            \"missing\": \"a\"\n        }\"#;\n        let avg: Result<AverageAggregation, _> = serde_json::from_str(json);\n        assert!(avg.is_err());\n        assert!(avg\n            .unwrap_err()\n            .to_string()\n            .contains(\"Failed to parse f64 from string: \\\"a\\\"\"));\n\n        // Disallow NaN\n        let json = r#\"{\n            \"field\": \"score\",\n            \"missing\": \"NaN\"\n        }\"#;\n        let avg: Result<AverageAggregation, _> = serde_json::from_str(json);\n        assert!(avg.is_err());\n        assert!(avg.unwrap_err().to_string().contains(\"NaN\"));\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/cardinality.rs",
    "content": "use std::hash::Hash;\n\nuse columnar::column_values::CompactSpaceU64Accessor;\nuse columnar::{Column, ColumnType, Dictionary, StrColumn};\nuse common::f64_to_u64;\nuse datasketches::hll::{HllSketch, HllType, HllUnion};\nuse rustc_hash::FxHashSet;\nuse serde::{Deserialize, Deserializer, Serialize, Serializer};\n\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateMetricResult,\n};\nuse crate::aggregation::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// Log2 of the number of registers for the HLL sketch.\n/// 2^11 = 2048 registers, giving ~2.3% relative error and ~1KB per sketch (Hll4).\nconst LG_K: u8 = 11;\n\n/// # Cardinality\n///\n/// The cardinality aggregation allows for computing an estimate\n/// of the number of different values in a data set based on the\n/// Apache DataSketches HyperLogLog algorithm. This is particularly useful for\n/// understanding the uniqueness of values in a large dataset where counting\n/// each unique value individually would be computationally expensive.\n///\n/// For example, you might use a cardinality aggregation to estimate the number\n/// of unique visitors to a website by aggregating on a field that contains\n/// user IDs or session IDs.\n///\n/// To use the cardinality aggregation, you'll need to provide a field to\n/// aggregate on. The following example demonstrates a request for the cardinality\n/// of the \"user_id\" field:\n///\n/// ```JSON\n/// {\n///     \"cardinality\": {\n///         \"field\": \"user_id\"\n///     }\n/// }\n/// ```\n///\n/// This request will return an estimate of the number of unique values in the\n/// \"user_id\" field.\n///\n/// ## Missing Values\n///\n/// The `missing` parameter defines how documents that are missing a value should be treated.\n/// By default, documents without a value for the specified field are ignored. However, you can\n/// specify a default value for these documents using the `missing` parameter. This can be useful\n/// when you want to include documents with missing values in the aggregation.\n///\n/// For example, the following request treats documents with missing values in the \"user_id\"\n/// field as if they had a value of \"unknown\":\n///\n/// ```JSON\n/// {\n///     \"cardinality\": {\n///         \"field\": \"user_id\",\n///         \"missing\": \"unknown\"\n///     }\n/// }\n/// ```\n///\n/// # Estimation Accuracy\n///\n/// The cardinality aggregation provides an approximate count, which is usually\n/// accurate within a small error range. This trade-off allows for efficient\n/// computation even on very large datasets.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct CardinalityAggregationReq {\n    /// The field name to compute the percentiles on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(skip_serializing_if = \"Option::is_none\", default)]\n    pub missing: Option<Key>,\n}\n\n/// Contains all information required by the SegmentCardinalityCollector to perform the\n/// cardinality aggregation on a segment.\npub struct CardinalityAggReqData {\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n    /// The column_type of the field.\n    pub column_type: ColumnType,\n    /// The string dictionary column if the field is of type string.\n    pub str_dict_column: Option<StrColumn>,\n    /// The missing value normalized to the internal u64 representation of the field type.\n    pub missing_value_for_accessor: Option<u64>,\n    /// The name of the aggregation.\n    pub name: String,\n    /// The aggregation request.\n    pub req: CardinalityAggregationReq,\n}\n\nimpl CardinalityAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\nimpl CardinalityAggregationReq {\n    /// Creates a new [`CardinalityAggregationReq`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n#[derive(Clone, Debug)]\npub(crate) struct SegmentCardinalityCollector {\n    buckets: Vec<SegmentCardinalityCollectorBucket>,\n    accessor_idx: usize,\n    /// The column accessor to access the fast field values.\n    accessor: Column<u64>,\n    /// The column_type of the field.\n    column_type: ColumnType,\n    /// The missing value normalized to the internal u64 representation of the field type.\n    missing_value_for_accessor: Option<u64>,\n}\n\n#[derive(Clone, Debug, PartialEq, Default)]\npub(crate) struct SegmentCardinalityCollectorBucket {\n    cardinality: CardinalityCollector,\n    entries: FxHashSet<u64>,\n}\nimpl SegmentCardinalityCollectorBucket {\n    pub fn new(column_type: ColumnType) -> Self {\n        Self {\n            cardinality: CardinalityCollector::new(column_type as u8),\n            entries: FxHashSet::default(),\n        }\n    }\n    fn into_intermediate_metric_result(\n        mut self,\n        req_data: &CardinalityAggReqData,\n    ) -> crate::Result<IntermediateMetricResult> {\n        if req_data.column_type == ColumnType::Str {\n            let fallback_dict = Dictionary::empty();\n            let dict = req_data\n                .str_dict_column\n                .as_ref()\n                .map(|el| el.dictionary())\n                .unwrap_or_else(|| &fallback_dict);\n            let mut has_missing = false;\n\n            // TODO: replace FxHashSet with something that allows iterating in order\n            // (e.g. sparse bitvec)\n            let mut term_ids = Vec::new();\n            for term_ord in self.entries.into_iter() {\n                if term_ord == u64::MAX {\n                    has_missing = true;\n                } else {\n                    // we can reasonably exclude values above u32::MAX\n                    term_ids.push(term_ord as u32);\n                }\n            }\n\n            term_ids.sort_unstable();\n            dict.sorted_ords_to_term_cb(term_ids.iter().map(|term| *term as u64), |term| {\n                self.cardinality.insert(term);\n                Ok(())\n            })?;\n            if has_missing {\n                // Replace missing with the actual value provided\n                let missing_key =\n                    req_data.req.missing.as_ref().expect(\n                        \"Found sentinel value u64::MAX for term_ord but `missing` is not set\",\n                    );\n                match missing_key {\n                    Key::Str(missing) => {\n                        self.cardinality.insert(missing.as_str());\n                    }\n                    Key::F64(val) => {\n                        let val = f64_to_u64(*val);\n                        self.cardinality.insert(val);\n                    }\n                    Key::U64(val) => {\n                        self.cardinality.insert(*val);\n                    }\n                    Key::I64(val) => {\n                        self.cardinality.insert(*val);\n                    }\n                }\n            }\n        }\n\n        Ok(IntermediateMetricResult::Cardinality(self.cardinality))\n    }\n}\n\nimpl SegmentCardinalityCollector {\n    pub fn from_req(\n        column_type: ColumnType,\n        accessor_idx: usize,\n        accessor: Column<u64>,\n        missing_value_for_accessor: Option<u64>,\n    ) -> Self {\n        Self {\n            buckets: vec![SegmentCardinalityCollectorBucket::new(column_type); 1],\n            column_type,\n            accessor_idx,\n            accessor,\n            missing_value_for_accessor,\n        }\n    }\n\n    fn fetch_block_with_field(\n        &mut self,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) {\n        agg_data.column_block_accessor.fetch_block_with_missing(\n            docs,\n            &self.accessor,\n            self.missing_value_for_accessor,\n        );\n    }\n}\n\nimpl SegmentAggregationCollector for SegmentCardinalityCollector {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let req_data = &agg_data.get_cardinality_req_data(self.accessor_idx);\n        let name = req_data.name.to_string();\n        // take the bucket in buckets and replace it with a new empty one\n        let bucket = std::mem::take(&mut self.buckets[parent_bucket_id as usize]);\n\n        let intermediate_result = bucket.into_intermediate_metric_result(req_data)?;\n        results.push(\n            name,\n            IntermediateAggregationResult::Metric(intermediate_result),\n        )?;\n\n        Ok(())\n    }\n\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        self.fetch_block_with_field(docs, agg_data);\n        let bucket = &mut self.buckets[parent_bucket_id as usize];\n\n        let col_block_accessor = &agg_data.column_block_accessor;\n        if self.column_type == ColumnType::Str {\n            for term_ord in col_block_accessor.iter_vals() {\n                bucket.entries.insert(term_ord);\n            }\n        } else if self.column_type == ColumnType::IpAddr {\n            let compact_space_accessor = self\n                .accessor\n                .values\n                .clone()\n                .downcast_arc::<CompactSpaceU64Accessor>()\n                .map_err(|_| {\n                    TantivyError::AggregationError(\n                        crate::aggregation::AggregationError::InternalError(\n                            \"Type mismatch: Could not downcast to CompactSpaceU64Accessor\"\n                                .to_string(),\n                        ),\n                    )\n                })?;\n            for val in col_block_accessor.iter_vals() {\n                let val: u128 = compact_space_accessor.compact_to_u128(val as u32);\n                bucket.cardinality.insert(val);\n            }\n        } else {\n            for val in col_block_accessor.iter_vals() {\n                bucket.cardinality.insert(val);\n            }\n        }\n\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        if max_bucket as usize >= self.buckets.len() {\n            self.buckets.resize_with(max_bucket as usize + 1, || {\n                SegmentCardinalityCollectorBucket::new(self.column_type)\n            });\n        }\n        Ok(())\n    }\n}\n\n#[derive(Clone, Debug)]\n/// The cardinality collector used during segment collection and for merging results.\n/// Uses Apache DataSketches HLL (lg_k=11, Hll4) for compact binary serialization\n/// and cross-language compatibility (e.g. Java `datasketches` library).\npub struct CardinalityCollector {\n    sketch: HllSketch,\n    /// Salt derived from `ColumnType`, used to differentiate values of different column types\n    /// that map to the same u64 (e.g. bool `false` = 0 vs i64 `0`).\n    /// Not serialized — only needed during insertion, not after sketch registers are populated.\n    salt: u8,\n}\n\nimpl Default for CardinalityCollector {\n    fn default() -> Self {\n        Self::new(0)\n    }\n}\n\nimpl PartialEq for CardinalityCollector {\n    fn eq(&self, _other: &Self) -> bool {\n        false\n    }\n}\n\nimpl Serialize for CardinalityCollector {\n    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {\n        let bytes = self.sketch.serialize();\n        serializer.serialize_bytes(&bytes)\n    }\n}\n\nimpl<'de> Deserialize<'de> for CardinalityCollector {\n    fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {\n        let bytes: Vec<u8> = Deserialize::deserialize(deserializer)?;\n        let sketch = HllSketch::deserialize(&bytes).map_err(serde::de::Error::custom)?;\n        Ok(Self { sketch, salt: 0 })\n    }\n}\n\nimpl CardinalityCollector {\n    fn new(salt: u8) -> Self {\n        Self {\n            sketch: HllSketch::new(LG_K, HllType::Hll4),\n            salt,\n        }\n    }\n\n    /// Insert a value into the HLL sketch, salted by the column type.\n    /// The salt ensures that identical u64 values from different column types\n    /// (e.g. bool `false` vs i64 `0`) are counted as distinct.\n    pub(crate) fn insert<T: Hash>(&mut self, value: T) {\n        self.sketch.update((self.salt, value));\n    }\n\n    /// Compute the final cardinality estimate.\n    pub fn finalize(self) -> Option<f64> {\n        Some(self.sketch.estimate().trunc())\n    }\n\n    /// Serialize the HLL sketch to its compact binary representation.\n    /// The format is cross-language compatible with Apache DataSketches (Java, C++, Python).\n    pub fn to_sketch_bytes(&self) -> Vec<u8> {\n        self.sketch.serialize()\n    }\n\n    pub(crate) fn merge_fruits(&mut self, right: CardinalityCollector) -> crate::Result<()> {\n        let mut union = HllUnion::new(LG_K);\n        union.update(&self.sketch);\n        union.update(&right.sketch);\n        self.sketch = union.get_result(HllType::Hll4);\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::net::IpAddr;\n    use std::str::FromStr;\n\n    use columnar::MonotonicallyMappableToU64;\n\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::{exec_request, get_test_index_from_terms};\n    use crate::schema::{IntoIpv6Addr, Schema, FAST};\n    use crate::Index;\n\n    #[test]\n    fn cardinality_aggregation_test_empty_index() -> crate::Result<()> {\n        let values = vec![];\n        let index = get_test_index_from_terms(false, &values)?;\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"cardinality\": {\n                \"cardinality\": {\n                    \"field\": \"string_id\",\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"cardinality\"][\"value\"], 0.0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn cardinality_aggregation_test_single_segment() -> crate::Result<()> {\n        cardinality_aggregation_test_merge_segment(true)\n    }\n    #[test]\n    fn cardinality_aggregation_test() -> crate::Result<()> {\n        cardinality_aggregation_test_merge_segment(false)\n    }\n    fn cardinality_aggregation_test_merge_segment(merge_segments: bool) -> crate::Result<()> {\n        let segment_and_terms = vec![\n            vec![\"terma\"],\n            vec![\"termb\"],\n            vec![\"termc\"],\n            vec![\"terma\"],\n            vec![\"terma\"],\n            vec![\"terma\"],\n            vec![\"termb\"],\n            vec![\"terma\"],\n        ];\n        let index = get_test_index_from_terms(merge_segments, &segment_and_terms)?;\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"cardinality\": {\n                \"cardinality\": {\n                    \"field\": \"string_id\",\n                }\n            },\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"cardinality\"][\"value\"], 3.0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn cardinality_aggregation_u64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut writer = index.writer_for_tests()?;\n            writer.add_document(doc!(id_field => 1u64))?;\n            writer.add_document(doc!(id_field => 2u64))?;\n            writer.add_document(doc!(id_field => 3u64))?;\n            writer.add_document(doc!())?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"cardinality\": {\n                \"cardinality\": {\n                    \"field\": \"id\",\n                    \"missing\": 0u64\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"cardinality\"][\"value\"], 4.0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn cardinality_aggregation_ip_addr() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_ip_addr_field(\"ip_field\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut writer = index.writer_for_tests()?;\n            // IpV6 loopback\n            writer.add_document(doc!(field=>IpAddr::from_str(\"::1\").unwrap().into_ipv6_addr()))?;\n            writer.add_document(doc!(field=>IpAddr::from_str(\"::1\").unwrap().into_ipv6_addr()))?;\n            // IpV4\n            writer.add_document(\n                doc!(field=>IpAddr::from_str(\"127.0.0.1\").unwrap().into_ipv6_addr()),\n            )?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"cardinality\": {\n                \"cardinality\": {\n                    \"field\": \"ip_field\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"cardinality\"][\"value\"], 2.0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn cardinality_aggregation_json() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_json_field(\"json\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut writer = index.writer_for_tests()?;\n            writer.add_document(doc!(field => json!({\"value\": false})))?;\n            writer.add_document(doc!(field => json!({\"value\": true})))?;\n            writer.add_document(doc!(field => json!({\"value\": i64::from_u64(0u64)})))?;\n            writer.add_document(doc!(field => json!({\"value\": i64::from_u64(1u64)})))?;\n            writer.commit()?;\n        }\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"cardinality\": {\n                \"cardinality\": {\n                    \"field\": \"json.value\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request(agg_req, &index)?;\n        assert_eq!(res[\"cardinality\"][\"value\"], 4.0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn cardinality_collector_serde_roundtrip() {\n        use super::CardinalityCollector;\n\n        let mut collector = CardinalityCollector::default();\n        collector.insert(\"hello\");\n        collector.insert(\"world\");\n        collector.insert(\"hello\"); // duplicate\n\n        let serialized = serde_json::to_vec(&collector).unwrap();\n        let deserialized: CardinalityCollector = serde_json::from_slice(&serialized).unwrap();\n\n        let original_estimate = collector.finalize().unwrap();\n        let roundtrip_estimate = deserialized.finalize().unwrap();\n        assert_eq!(original_estimate, roundtrip_estimate);\n        assert_eq!(original_estimate, 2.0);\n    }\n\n    #[test]\n    fn cardinality_collector_merge() {\n        use super::CardinalityCollector;\n\n        let mut left = CardinalityCollector::default();\n        left.insert(\"a\");\n        left.insert(\"b\");\n\n        let mut right = CardinalityCollector::default();\n        right.insert(\"b\");\n        right.insert(\"c\");\n\n        left.merge_fruits(right).unwrap();\n        let estimate = left.finalize().unwrap();\n        assert_eq!(estimate, 3.0);\n    }\n\n    #[test]\n    fn cardinality_collector_serialize_deserialize_binary() {\n        use datasketches::hll::HllSketch;\n\n        use super::CardinalityCollector;\n\n        let mut collector = CardinalityCollector::default();\n        collector.insert(\"apple\");\n        collector.insert(\"banana\");\n        collector.insert(\"cherry\");\n\n        let bytes = collector.to_sketch_bytes();\n        let deserialized = HllSketch::deserialize(&bytes).unwrap();\n        assert!((deserialized.estimate() - 3.0).abs() < 0.01);\n    }\n\n    #[test]\n    fn cardinality_collector_salt_differentiates_types() {\n        use super::CardinalityCollector;\n\n        // Without salt, same u64 value from different column types would collide\n        let mut collector_bool = CardinalityCollector::new(5); // e.g. ColumnType::Bool\n        collector_bool.insert(0u64); // false\n        collector_bool.insert(1u64); // true\n\n        let mut collector_i64 = CardinalityCollector::new(2); // e.g. ColumnType::I64\n        collector_i64.insert(0u64);\n        collector_i64.insert(1u64);\n\n        // Merge them\n        collector_bool.merge_fruits(collector_i64).unwrap();\n        let estimate = collector_bool.finalize().unwrap();\n        // Should be 4 because salt makes (5, 0) != (2, 0) and (5, 1) != (2, 1)\n        assert_eq!(estimate, 4.0);\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/count.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::*;\n\n/// A single-value metric aggregation that counts the number of values that are\n/// extracted from the aggregated documents.\n/// See [super::SingleMetricResult] for return value.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"value_count\": {\n///         \"field\": \"score\"\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct CountAggregation {\n    /// The field name to compute the count on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl CountAggregation {\n    /// Creates a new [`CountAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Intermediate result of the count aggregation that can be combined with other intermediate\n/// results.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateCount {\n    stats: IntermediateStats,\n}\n\nimpl IntermediateCount {\n    /// Creates a new [`IntermediateCount`] instance from a [`SegmentStatsCollector`].\n    pub(crate) fn from_stats(stats: IntermediateStats) -> Self {\n        Self { stats }\n    }\n    /// Merges the other intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateCount) {\n        self.stats.merge_fruits(other.stats);\n    }\n    /// Computes the final count value.\n    pub fn finalize(&self) -> Option<f64> {\n        Some(self.stats.finalize().count as f64)\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/extended_stats.rs",
    "content": "use std::fmt::Debug;\nuse std::mem;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateMetricResult,\n};\nuse crate::aggregation::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// A multi-value metric aggregation that computes a collection of extended statistics\n/// on numeric values that are extracted\n/// from the aggregated documents.\n/// See [`ExtendedStats`] for returned statistics.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"extended_stats\": {\n///         \"field\": \"score\"\n///     }\n///  }\n/// ```\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct ExtendedStatsAggregation {\n    /// The field name to compute the stats on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default)]\n    pub missing: Option<f64>,\n    /// The sigma parameter defines how standard_deviation_bound_are_calculated.\n    /// This can be a useful way to visualize variance of your data.\n    /// The default value is 2. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"sigma\": \"3.0\" }\n    #[serde(default)]\n    pub sigma: Option<f64>,\n}\n\nimpl ExtendedStatsAggregation {\n    /// Creates a new [`ExtendedStatsAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        ExtendedStatsAggregation {\n            field: field_name,\n            missing: None,\n            sigma: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Extended stats contains a collection of statistics\n/// they extends stats adding variance, standard deviation\n/// and bound information\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct ExtendedStats {\n    /// The number of documents.\n    pub count: u64,\n    /// The sum of the fast field values.\n    pub sum: f64,\n    /// The min value of the fast field values.\n    pub min: Option<f64>,\n    /// The max value of the fast field values.\n    pub max: Option<f64>,\n    /// The average of the fast field values. `None` if count equals zero.\n    pub avg: Option<f64>,\n    /// The sum of squares of the fast field values. `None` if count equals zero.\n    pub sum_of_squares: Option<f64>,\n    /// The variance of the fast field values. `None` if count is is 0 or 1.\n    pub variance: Option<f64>,\n    /// The variance population of the fast field values, always equal to variance. `None` if count\n    /// is is 0 or 1.\n    pub variance_population: Option<f64>,\n    /// The variance sampling of the fast field values, always equal to variance. `None` if count\n    /// is is 0 or 1.\n    pub variance_sampling: Option<f64>,\n    /// The standard deviation of the fast field values. `None` if count is is 0 or 1.\n    pub std_deviation: Option<f64>,\n    /// The standard deviation of the fast field values, always equal to variance. `None` if count\n    /// is is 0 or 1.\n    pub std_deviation_population: Option<f64>,\n    /// The standard deviation sampling of the fast field values. `None`\n    /// if count is is 0 or 1.\n    pub std_deviation_sampling: Option<f64>,\n    /// The standard deviation bounds of the fast field values, always equal to variance. `None`\n    /// if count is is 0 or 1.\n    pub std_deviation_bounds: Option<StandardDeviationBounds>,\n}\n\n/// A sub struct for ExtendedStat containing deviation bounds\n/// the values depend on sigma and represent\n/// the bounds from the average with a distance of\n/// std_deviation*sigma\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct StandardDeviationBounds {\n    /// upper bound -> avg + std_dev*sigma\n    pub upper: f64,\n    /// lower bound -> avg - std_dev*sigma\n    pub lower: f64,\n    /// upper bound sampling -> avg + std_dev_sampling*sigma\n    pub upper_sampling: f64,\n    /// lower bound sampling -> avg - std_dev_sampling*sigma\n    pub lower_sampling: f64,\n    /// same as upper\n    pub upper_population: f64,\n    /// same as lower\n    pub lower_population: f64,\n}\n\nimpl ExtendedStats {\n    pub(crate) fn get_value(&self, agg_property: &str) -> crate::Result<Option<f64>> {\n        match agg_property {\n            \"count\" => Ok(Some(self.count as f64)),\n            \"sum\" => Ok(Some(self.sum)),\n            \"min\" => Ok(self.min),\n            \"max\" => Ok(self.max),\n            \"avg\" => Ok(self.avg),\n            \"variance\" => Ok(self.variance),\n            \"variance_sampling\" => Ok(self.variance_sampling),\n            \"variance_population\" => Ok(self.variance_population),\n            \"sum_of_squares\" => Ok(self.sum_of_squares),\n            \"std_deviation\" => Ok(self.std_deviation),\n            \"std_deviation_sampling\" => Ok(self.std_deviation_sampling),\n            \"std_deviation_population\" => Ok(self.std_deviation_population),\n            \"std_deviation_bounds.lower\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.lower)),\n            \"std_deviation_bounds.lower_population\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.lower_population)),\n            \"std_deviation_bounds.lower_sampling\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.lower_sampling)),\n            \"std_deviation_bounds.upper\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.upper)),\n            \"std_deviation_bounds.upper_population\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.upper_population)),\n            \"std_deviation_bounds.upper_sampling\" => Ok(self\n                .std_deviation_bounds\n                .as_ref()\n                .map(|bounds| bounds.upper_sampling)),\n            _ => Err(TantivyError::InvalidArgument(format!(\n                \"Unknown property {agg_property} on stats metric aggregation\"\n            ))),\n        }\n    }\n}\n\n/// Intermediate result of the extended stats aggregation that can be combined with other\n/// intermediate results.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateExtendedStats {\n    intermediate_stats: IntermediateStats,\n    /// The number of extracted values.\n    /// The sum of square values, it's referred as M2 in Welford's online algorithm\n    sum_of_squares: f64,\n    /// The sum of square values as computed by elastic search\n    sum_of_squares_elastic: f64,\n    /// The delta for sum of squares  as computed by elastic search needed for the Kahan algorithm\n    delta_sum_for_squares_elastic: f64,\n    /// The mean is an intermediate value need for calculating the variance\n    /// as per [Welford's online algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford's_online_algorithm)\n    mean: f64,\n    /// The value used for computing standard deviation bounds\n    sigma: f64,\n}\n\nimpl Default for IntermediateExtendedStats {\n    fn default() -> Self {\n        Self {\n            intermediate_stats: IntermediateStats::default(),\n            sum_of_squares: 0.0,\n            sum_of_squares_elastic: 0.0,\n            delta_sum_for_squares_elastic: 0.0,\n            mean: 0.0,\n            // The default value is the same of ElasticSearch\n            sigma: 2.0,\n        }\n    }\n}\n\nimpl IntermediateExtendedStats {\n    /// Creates a new IntermediateExtendedStats using an option\n    /// containing the sigma to be used for calculating bound values.\n    pub fn with_sigma(sigma: Option<f64>) -> Self {\n        Self {\n            intermediate_stats: IntermediateStats::default(),\n            sum_of_squares: 0.0,\n            sum_of_squares_elastic: 0.0,\n            delta_sum_for_squares_elastic: 0.0,\n            mean: 0.0,\n            // The default value is the same of ElasticSearch\n            sigma: sigma.unwrap_or(2.0),\n        }\n    }\n    /// Merges the other stats intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateExtendedStats) {\n        if other.intermediate_stats.count == 0 {\n            return;\n        }\n        if self.intermediate_stats.count == 0 {\n            let _ = mem::replace(self, other);\n            return;\n        }\n        let new_count = self.intermediate_stats.count + other.intermediate_stats.count;\n        let delta = other.mean - self.mean;\n        self.sum_of_squares += other.sum_of_squares\n            + delta\n                * delta\n                * self.intermediate_stats.count as f64\n                * other.intermediate_stats.count as f64\n                / new_count as f64;\n        self.mean = (self.intermediate_stats.sum + other.intermediate_stats.sum) / new_count as f64;\n        self.sum_of_squares_elastic += other.sum_of_squares_elastic;\n        self.delta_sum_for_squares_elastic += other.delta_sum_for_squares_elastic;\n        self.intermediate_stats\n            .merge_fruits(other.intermediate_stats);\n    }\n\n    /// Computes the final stats value.\n    pub fn finalize(&self) -> Box<ExtendedStats> {\n        let (min, max, avg, sum_of_squares) = if self.intermediate_stats.count == 0 {\n            (None, None, None, None)\n        } else {\n            (\n                Some(self.intermediate_stats.min),\n                Some(self.intermediate_stats.max),\n                Some(self.mean),\n                Some(self.sum_of_squares_elastic),\n            )\n        };\n        let (variance, variance_sampling) = if self.intermediate_stats.count <= 1 {\n            (None, None)\n        } else {\n            (\n                Some(self.sum_of_squares / self.intermediate_stats.count as f64),\n                Some(self.sum_of_squares / (self.intermediate_stats.count - 1) as f64),\n            )\n        };\n        let std_deviation = variance.map(|v| v.sqrt());\n        let std_deviation_sampling = variance_sampling.map(|v| v.sqrt());\n        let std_deviation_bounds =\n            if let (Some(std_deviation_val), Some(std_deviation_sampling_val)) =\n                (std_deviation, std_deviation_sampling)\n            {\n                let upper = self.mean + std_deviation_val * self.sigma;\n                let lower = self.mean - std_deviation_val * self.sigma;\n                let upper_sampling = self.mean + std_deviation_sampling_val * self.sigma;\n                let lower_sampling = self.mean - std_deviation_sampling_val * self.sigma;\n                Some(StandardDeviationBounds {\n                    upper,\n                    lower,\n                    upper_sampling,\n                    lower_sampling,\n                    upper_population: upper,\n                    lower_population: lower,\n                })\n            } else {\n                None\n            };\n        Box::new(ExtendedStats {\n            count: self.intermediate_stats.count,\n            sum: self.intermediate_stats.sum,\n            min,\n            max,\n            avg,\n            sum_of_squares,\n            variance,\n            variance_population: variance,\n            variance_sampling,\n            std_deviation,\n            std_deviation_population: std_deviation,\n            std_deviation_sampling,\n            std_deviation_bounds,\n        })\n    }\n\n    fn update_variance(&mut self, value: f64) {\n        let delta = value - self.mean;\n        // this is not what the Welford's online algorithm prescribes but\n        // using the pseudo code from wikipedia there was a small rounding\n        // error (in 15th decimal place) that caused a test\n        //(test_aggregation_level1 in agg_test.rs)\n        // failure\n        self.mean = self.intermediate_stats.sum / self.intermediate_stats.count as f64;\n        // self.mean += delta / self.count as f64;\n        let delta2 = value - self.mean;\n        self.sum_of_squares += delta * delta2;\n    }\n\n    #[inline]\n    fn collect(&mut self, value: f64) {\n        self.intermediate_stats.collect(value);\n        // kahan algorithm for sum_of_squares_elastic\n        let y = value * value - self.delta_sum_for_squares_elastic;\n        let t = self.sum_of_squares_elastic + y;\n        self.delta_sum_for_squares_elastic = (t - self.sum_of_squares_elastic) - y;\n        self.sum_of_squares_elastic = t;\n        self.update_variance(value);\n    }\n}\n\n#[derive(Clone, Debug)]\npub(crate) struct SegmentExtendedStatsCollector {\n    name: String,\n    missing: Option<u64>,\n    field_type: ColumnType,\n    accessor: columnar::Column<u64>,\n    buckets: Vec<IntermediateExtendedStats>,\n    sigma: Option<f64>,\n}\n\nimpl SegmentExtendedStatsCollector {\n    pub fn from_req(req: &MetricAggReqData, sigma: Option<f64>) -> Self {\n        let missing = req\n            .missing\n            .and_then(|val| f64_to_fastfield_u64(val, &req.field_type));\n        Self {\n            name: req.name.clone(),\n            field_type: req.field_type,\n            accessor: req.accessor.clone(),\n            missing,\n            buckets: vec![IntermediateExtendedStats::with_sigma(sigma); 16],\n            sigma,\n        }\n    }\n}\n\nimpl SegmentAggregationCollector for SegmentExtendedStatsCollector {\n    #[inline]\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let name = self.name.clone();\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let extended_stats = std::mem::take(&mut self.buckets[parent_bucket_id as usize]);\n        results.push(\n            name,\n            IntermediateAggregationResult::Metric(IntermediateMetricResult::ExtendedStats(\n                extended_stats,\n            )),\n        )?;\n\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let mut extended_stats = self.buckets[parent_bucket_id as usize].clone();\n\n        agg_data\n            .column_block_accessor\n            .fetch_block_with_missing(docs, &self.accessor, self.missing);\n        for val in agg_data.column_block_accessor.iter_vals() {\n            let val1 = f64_from_fastfield_u64(val, self.field_type);\n            extended_stats.collect(val1);\n        }\n\n        // store back\n        self.buckets[parent_bucket_id as usize] = extended_stats;\n\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        if self.buckets.len() <= max_bucket as usize {\n            self.buckets.resize_with(max_bucket as usize + 1, || {\n                IntermediateExtendedStats::with_sigma(self.sigma)\n            });\n        }\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::metric::IntermediateExtendedStats;\n    use crate::aggregation::tests::get_test_index_from_values;\n    use crate::aggregation::AggregationCollector;\n    use crate::assert_nearly_equals;\n    use crate::query::AllQuery;\n\n    const EPSILON_FOR_TEST: f64 = 0.000000000002;\n\n    #[test]\n    fn test_aggregation_extended_stats_no_variance() -> crate::Result<()> {\n        let values = vec![1.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"extended_stats\": {\n                    \"field\": \"score_f64\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"count\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"min\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"max\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"avg\")?\n                .unwrap(),\n            1.0\n        );\n\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_population\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_sampling\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_population\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_sampling\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_population\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_sampling\")?\n            .is_none());\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum_of_squares\")?\n                .unwrap(),\n            1.0\n        );\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"variance_population\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"variance\")?\n            .is_none());\n        assert!(agg_res\n            .get_value_from_aggregation(\"my_stats\", \"variance_sampling\")?\n            .is_none());\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_extended_stats() -> crate::Result<()> {\n        let values = vec![1.0, 3.0, 4.0, 5.0, 8.0, 10.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"extended_stats\": {\n                    \"field\": \"score_f64\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n        const EXPECTED_VARIANCE: f64 = 9.138888888888888;\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"count\")?\n                .unwrap(),\n            6.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"min\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"max\")?\n                .unwrap(),\n            10.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum\")?\n                .unwrap(),\n            31.0\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"avg\")?\n                .unwrap(),\n            5.166666666666667,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_sampling\")?\n                .unwrap(),\n            3.311595788538611,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower\")?\n                .unwrap(),\n            -0.8794523824056837,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_population\")?\n                .unwrap(),\n            -0.8794523824056837,\n            0.00000000000001\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_sampling\")?\n                .unwrap(),\n            -1.4565249104105549,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper\")?\n                .unwrap(),\n            11.212785715739017,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_population\")?\n                .unwrap(),\n            11.212785715739017,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_sampling\")?\n                .unwrap(),\n            11.78985824374389,\n            EPSILON_FOR_TEST\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum_of_squares\")?\n                .unwrap(),\n            215.0\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_sampling\")?\n                .unwrap(),\n            10.966666666666663,\n            EPSILON_FOR_TEST\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_extended_stats_with_sigma() -> crate::Result<()> {\n        let values = vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"extended_stats\": {\n                    \"field\": \"score_f64\",\n                    \"sigma\": 1.5\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        const EXPECTED_VARIANCE: f64 = 2.9166666666666665;\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"count\")?\n                .unwrap(),\n            6.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"min\")?\n                .unwrap(),\n            1.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"max\")?\n                .unwrap(),\n            6.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum\")?\n                .unwrap(),\n            21.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"avg\")?\n                .unwrap(),\n            3.5\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_sampling\")?\n                .unwrap(),\n            1.8708286933869709,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower\")?\n                .unwrap(),\n            0.9382623085101005,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_population\")?\n                .unwrap(),\n            0.9382623085101005,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_sampling\")?\n                .unwrap(),\n            0.6937569599195434,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper\")?\n                .unwrap(),\n            6.061737691489899,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_population\")?\n                .unwrap(),\n            6.061737691489899,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_sampling\")?\n                .unwrap(),\n            6.3062430400804566,\n            EPSILON_FOR_TEST\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum_of_squares\")?\n                .unwrap(),\n            91.0\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_sampling\")?\n                .unwrap(),\n            3.5,\n            EPSILON_FOR_TEST\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_extended_stats_with_variance_similar_to_mean() -> crate::Result<()> {\n        let values = vec![50.01, 50.02, 50.01, 50.03, 50.01, 50.02];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"extended_stats\": {\n                    \"field\": \"score_f64\",\n                    \"sigma\": 1.5\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n        const EXPECTED_VARIANCE: f64 = 5.5555555555608854e-5;\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"count\")?\n                .unwrap(),\n            6.0\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"min\")?\n                .unwrap(),\n            50.01\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"max\")?\n                .unwrap(),\n            50.03\n        );\n        assert_eq!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum\")?\n                .unwrap(),\n            300.1\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"avg\")?\n                .unwrap(),\n            50.01666666666667,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE.sqrt(),\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_sampling\")?\n                .unwrap(),\n            0.008164965809279263,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower\")?\n                .unwrap(),\n            50.00548632677917,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_population\")?\n                .unwrap(),\n            50.00548632677917,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.lower_sampling\")?\n                .unwrap(),\n            50.00441921795275,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper\")?\n                .unwrap(),\n            50.027847006554175,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_population\")?\n                .unwrap(),\n            50.027847006554175,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"std_deviation_bounds.upper_sampling\")?\n                .unwrap(),\n            50.028914115380594,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"sum_of_squares\")?\n                .unwrap(),\n            15010.002,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_population\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance\")?\n                .unwrap(),\n            EXPECTED_VARIANCE,\n            EPSILON_FOR_TEST\n        );\n        assert_nearly_equals!(\n            agg_res\n                .get_value_from_aggregation(\"my_stats\", \"variance_sampling\")?\n                .unwrap(),\n            6.666666666670718e-5,\n            EPSILON_FOR_TEST\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn extended_stat_zero_value() {\n        let intermediate_extend_stats = IntermediateExtendedStats::default();\n        let extended_stats = intermediate_extend_stats.finalize();\n        assert!(extended_stats.variance.is_none());\n        assert!(extended_stats.variance_population.is_none());\n        assert!(extended_stats.variance_sampling.is_none());\n        assert!(extended_stats.sum_of_squares.is_none());\n        assert!(extended_stats.std_deviation.is_none());\n        assert!(extended_stats.std_deviation_population.is_none());\n        assert!(extended_stats.std_deviation_sampling.is_none());\n        assert!(extended_stats.std_deviation_bounds.is_none());\n    }\n\n    #[test]\n    fn extended_stat_one_value() {\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.collect(1.0f64);\n        let extended_stats = intermediate_extend_stats.finalize();\n        assert!(extended_stats.variance.is_none());\n        assert!(extended_stats.variance_population.is_none());\n        assert!(extended_stats.variance_sampling.is_none());\n        assert!(extended_stats.std_deviation.is_none());\n        assert!(extended_stats.std_deviation_population.is_none());\n        assert!(extended_stats.std_deviation_sampling.is_none());\n        assert!(extended_stats.std_deviation_bounds.is_none());\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(1.0f64, sum_of_squares);\n    }\n\n    #[test]\n    fn extended_stat_multiple_values() {\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.collect(1.0f64);\n        intermediate_extend_stats.collect(3.0f64);\n        intermediate_extend_stats.collect(4.0f64);\n        intermediate_extend_stats.collect(5.0f64);\n        intermediate_extend_stats.collect(8.0f64);\n        intermediate_extend_stats.collect(10.0f64);\n        let extended_stats = intermediate_extend_stats.finalize();\n        let variance = extended_stats.variance.unwrap();\n        const EXPECTED_VARIANCE: f64 = 9.138888888888888;\n        assert_eq!(EXPECTED_VARIANCE, variance);\n        let variance_population = extended_stats.variance_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE, variance_population);\n        let variance_sampling = extended_stats.variance_sampling.unwrap();\n        assert_eq!(10.966666666666665f64, variance_sampling);\n        let std_deviation = extended_stats.std_deviation.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation);\n        let std_deviation_population = extended_stats.std_deviation_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation_population);\n        let std_deviation_sampling = extended_stats.std_deviation_sampling.unwrap();\n        assert_eq!(10.966666666666665f64.sqrt(), std_deviation_sampling);\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(215.0, sum_of_squares);\n        let avg = extended_stats.avg.unwrap();\n        assert_eq!(5.166666666666667, avg);\n    }\n\n    #[test]\n    fn merge_empty_with_one_value() {\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        let mut intermediate_extend_stats1 = IntermediateExtendedStats::default();\n        intermediate_extend_stats1.collect(1.0f64);\n        intermediate_extend_stats.merge_fruits(intermediate_extend_stats1);\n        let extended_stats = intermediate_extend_stats.finalize();\n        assert!(extended_stats.variance.is_none());\n        assert!(extended_stats.variance_population.is_none());\n        assert!(extended_stats.variance_sampling.is_none());\n        assert!(extended_stats.std_deviation.is_none());\n        assert!(extended_stats.std_deviation_population.is_none());\n        assert!(extended_stats.std_deviation_sampling.is_none());\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(1.0f64, sum_of_squares);\n    }\n\n    #[test]\n    fn merge_empty_with_multiple_values() {\n        let mut intermediate_extend_stats1 = IntermediateExtendedStats::default();\n        intermediate_extend_stats1.collect(1.0f64);\n        intermediate_extend_stats1.collect(2.0f64);\n        intermediate_extend_stats1.collect(3.0f64);\n        intermediate_extend_stats1.collect(4.0f64);\n        intermediate_extend_stats1.collect(5.0f64);\n\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.merge_fruits(intermediate_extend_stats1);\n        let extended_stats = intermediate_extend_stats.finalize();\n        const EXPECTED_VARIANCE: f64 = 2.0;\n        let variance = extended_stats.variance.unwrap();\n        assert_eq!(EXPECTED_VARIANCE, variance);\n        let variance_population = extended_stats.variance_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE, variance_population);\n        let variance_sampling = extended_stats.variance_sampling.unwrap();\n        assert_eq!(2.5f64, variance_sampling);\n        let std_deviation = extended_stats.std_deviation.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation);\n        let std_deviation_population = extended_stats.std_deviation_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation_population);\n        let std_deviation_sampling = extended_stats.std_deviation_sampling.unwrap();\n        assert_eq!(2.5f64.sqrt(), std_deviation_sampling);\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(55f64, sum_of_squares);\n    }\n\n    #[test]\n    fn merge_non_empty_extended_stats() {\n        let mut intermediate_extend_stats1 = IntermediateExtendedStats::default();\n        intermediate_extend_stats1.collect(3.0f64);\n        intermediate_extend_stats1.collect(4.0f64);\n        intermediate_extend_stats1.collect(5.0f64);\n\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.collect(1.0f64);\n        intermediate_extend_stats.collect(2.0f64);\n        intermediate_extend_stats.merge_fruits(intermediate_extend_stats1);\n        let extended_stats = intermediate_extend_stats.finalize();\n\n        let variance = extended_stats.variance.unwrap();\n        assert_eq!(2.0f64, variance);\n        let variance_population = extended_stats.variance_population.unwrap();\n        assert_eq!(2.0f64, variance_population);\n        let variance_sampling = extended_stats.variance_sampling.unwrap();\n        assert_eq!(2.5f64, variance_sampling);\n        let std_deviation = extended_stats.std_deviation.unwrap();\n        assert_eq!(2.0f64.sqrt(), std_deviation);\n        let std_deviation_population = extended_stats.std_deviation_population.unwrap();\n        assert_eq!(2.0f64.sqrt(), std_deviation_population);\n        let std_deviation_sampling = extended_stats.std_deviation_sampling.unwrap();\n        assert_eq!(2.5f64.sqrt(), std_deviation_sampling);\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(55f64, sum_of_squares);\n\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.collect(1.0f64);\n        intermediate_extend_stats.collect(3.0f64);\n        intermediate_extend_stats.collect(4.0f64);\n        let mut intermediate_extend_stats1 = IntermediateExtendedStats::default();\n        intermediate_extend_stats1.collect(5.0f64);\n        intermediate_extend_stats1.collect(8.0f64);\n        intermediate_extend_stats1.collect(10.0f64);\n        intermediate_extend_stats.merge_fruits(intermediate_extend_stats1);\n        let extended_stats = intermediate_extend_stats.finalize();\n        const EXPECTED_VARIANCE: f64 = 9.138888888888888;\n        let variance = extended_stats.variance.unwrap();\n        assert_eq!(EXPECTED_VARIANCE, variance);\n        let variance_population = extended_stats.variance_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE, variance_population);\n        let variance_sampling = extended_stats.variance_sampling.unwrap();\n        assert_eq!(10.966666666666665f64, variance_sampling);\n        let std_deviation = extended_stats.std_deviation.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation);\n        let std_deviation_population = extended_stats.std_deviation_population.unwrap();\n        assert_eq!(EXPECTED_VARIANCE.sqrt(), std_deviation_population);\n        let std_deviation_sampling = extended_stats.std_deviation_sampling.unwrap();\n        assert_eq!(10.966666666666665f64.sqrt(), std_deviation_sampling);\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(215f64, sum_of_squares);\n        let avg = extended_stats.avg.unwrap();\n        assert_eq!(5.166666666666667, avg);\n    }\n\n    #[test]\n    fn merge_and_then_collect_non_empty_extended_stats() {\n        let mut intermediate_extend_stats = IntermediateExtendedStats::default();\n        intermediate_extend_stats.collect(1.0f64);\n        intermediate_extend_stats.collect(3.0f64);\n\n        let mut intermediate_extend_stats1 = IntermediateExtendedStats::default();\n        intermediate_extend_stats1.collect(5.0f64);\n        intermediate_extend_stats1.collect(8.0f64);\n        intermediate_extend_stats1.collect(10.0f64);\n        intermediate_extend_stats.merge_fruits(intermediate_extend_stats1);\n        intermediate_extend_stats.collect(4.0f64);\n        let extended_stats = intermediate_extend_stats.finalize();\n        const EXPECTED_VARIANCE: f64 = 9.138888888888888;\n        let variance = extended_stats.variance.unwrap();\n        assert_nearly_equals!(EXPECTED_VARIANCE, variance, EPSILON_FOR_TEST);\n        let variance_population = extended_stats.variance_population.unwrap();\n        assert_nearly_equals!(EXPECTED_VARIANCE, variance_population, EPSILON_FOR_TEST);\n        let variance_sampling = extended_stats.variance_sampling.unwrap();\n        assert_nearly_equals!(10.966666666666665, variance_sampling, EPSILON_FOR_TEST);\n        let std_deviation = extended_stats.std_deviation.unwrap();\n        assert_nearly_equals!(EXPECTED_VARIANCE.sqrt(), std_deviation, EPSILON_FOR_TEST);\n        let std_deviation_population = extended_stats.std_deviation_population.unwrap();\n        assert_nearly_equals!(\n            EXPECTED_VARIANCE.sqrt(),\n            std_deviation_population,\n            EPSILON_FOR_TEST\n        );\n        let std_deviation_sampling = extended_stats.std_deviation_sampling.unwrap();\n        assert_nearly_equals!(\n            10.966666666666665_f64.sqrt(),\n            std_deviation_sampling,\n            EPSILON_FOR_TEST\n        );\n        let sum_of_squares = extended_stats.sum_of_squares.unwrap();\n        assert_eq!(215.0, sum_of_squares);\n        let avg = extended_stats.avg.unwrap();\n        assert_eq!(5.166666666666667, avg);\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/max.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::*;\n\n/// A single-value metric aggregation that computes the maximum of numeric values that are\n/// extracted from the aggregated documents.\n/// See [super::SingleMetricResult] for return value.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"max\": {\n///         \"field\": \"score\"\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct MaxAggregation {\n    /// The field name to compute the maximum on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl MaxAggregation {\n    /// Creates a new [`MaxAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Intermediate result of the maximum aggregation that can be combined with other intermediate\n/// results.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateMax {\n    stats: IntermediateStats,\n}\n\nimpl IntermediateMax {\n    /// Creates a new [`IntermediateMax`] instance from a [`SegmentStatsCollector`].\n    pub(crate) fn from_stats(stats: IntermediateStats) -> Self {\n        Self { stats }\n    }\n    /// Merges the other intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateMax) {\n        self.stats.merge_fruits(other.stats);\n    }\n    /// Computes the final maximum value.\n    pub fn finalize(&self) -> Option<f64> {\n        self.stats.finalize().max\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::tests::exec_request_with_query;\n    use crate::schema::{Schema, FAST};\n    use crate::{Index, IndexWriter};\n\n    #[test]\n    fn test_max_agg_with_missing() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with empty json\n        index_writer.add_document(doc!()).unwrap();\n        index_writer.commit().unwrap();\n        // => Segment with json, but no field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"different_field\": \"blue\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        //// => Segment with field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"partially_empty\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"max\": {\n                    \"field\": \"json.partially_empty\",\n                    \"missing\": 100.0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res[\"my_stats\"],\n            json!({\n                \"value\": 100.0,\n            })\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/min.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::*;\n\n/// A single-value metric aggregation that computes the minimum of numeric values that are\n/// extracted from the aggregated documents.\n/// See [super::SingleMetricResult] for return value.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"min\": {\n///         \"field\": \"score\"\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct MinAggregation {\n    /// The field name to compute the minimum on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl MinAggregation {\n    /// Creates a new [`MinAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Intermediate result of the minimum aggregation that can be combined with other intermediate\n/// results.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateMin {\n    stats: IntermediateStats,\n}\n\nimpl IntermediateMin {\n    /// Creates a new [`IntermediateMin`] instance from a [`SegmentStatsCollector`].\n    pub(crate) fn from_stats(stats: IntermediateStats) -> Self {\n        Self { stats }\n    }\n    /// Merges the other intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateMin) {\n        self.stats.merge_fruits(other.stats);\n    }\n    /// Computes the final minimum value.\n    pub fn finalize(&self) -> Option<f64> {\n        self.stats.finalize().min\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/mod.rs",
    "content": "//! Module for all metric aggregations.\n//!\n//! The aggregations in this family compute metrics based on values extracted\n//! from the documents that are being aggregated. Values are extracted from the fast field of\n//! the document.\n//! Some aggregations output a single numeric metric (e.g. Average) and are called\n//! single-value numeric metrics aggregation, others generate multiple metrics (e.g. Stats) and are\n//! called multi-value numeric metrics aggregation.\n//!\n//! ## Supported Metric Aggregations\n//! - [Average](AverageAggregation)\n//! - [Stats](StatsAggregation)\n//! - [Min](MinAggregation)\n//! - [Max](MaxAggregation)\n//! - [Sum](SumAggregation)\n//! - [Count](CountAggregation)\n//! - [Percentiles](PercentilesAggregationReq)\n\nmod average;\nmod cardinality;\nmod count;\nmod extended_stats;\nmod max;\nmod min;\nmod percentiles;\nmod stats;\nmod sum;\nmod top_hits;\n\nuse std::collections::HashMap;\n\npub use average::*;\npub use cardinality::*;\nuse columnar::{Column, ColumnType};\npub use count::*;\npub use extended_stats::*;\npub use max::*;\npub use min::*;\npub use percentiles::*;\nuse rustc_hash::FxHashMap;\nuse serde::{Deserialize, Serialize};\npub use stats::*;\npub use sum::*;\npub use top_hits::*;\n\nuse crate::schema::OwnedValue;\n\n/// Contains all information required by metric aggregations like avg, min, max, sum, stats,\n/// extended_stats, count, percentiles.\n#[repr(C)]\npub struct MetricAggReqData {\n    /// True if the field is of number or date type.\n    pub is_number_or_date_type: bool,\n    /// The type of the field.\n    pub field_type: ColumnType,\n    /// The missing value normalized to the internal u64 representation of the field type.\n    pub missing_u64: Option<u64>,\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n    /// Used when converting to intermediate result\n    pub collecting_for: StatsType,\n    /// The missing value\n    pub missing: Option<f64>,\n    /// The name of the aggregation.\n    pub name: String,\n}\n\nimpl MetricAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\n/// Single-metric aggregations use this common result structure.\n///\n/// Main reason to wrap it in value is to match elasticsearch output structure.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct SingleMetricResult {\n    /// The value of the single value metric.\n    pub value: Option<f64>,\n}\n\nimpl From<f64> for SingleMetricResult {\n    fn from(value: f64) -> Self {\n        Self { value: Some(value) }\n    }\n}\n\nimpl From<Option<f64>> for SingleMetricResult {\n    fn from(value: Option<f64>) -> Self {\n        Self { value }\n    }\n}\n\n/// This is the wrapper of percentile entries, which can be vector or hashmap\n/// depending on if it's keyed or not.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\npub enum PercentileValues {\n    /// Vector format percentile entries\n    Vec(Vec<PercentileValuesVecEntry>),\n    /// HashMap format percentile entries. Key is the serialized percentile\n    HashMap(FxHashMap<String, f64>),\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n/// The entry when requesting percentiles with keyed: false\npub struct PercentileValuesVecEntry {\n    /// Percentile\n    pub key: f64,\n\n    /// Value at the percentile\n    pub value: f64,\n}\n\n/// Single-metric aggregations use this common result structure.\n///\n/// Main reason to wrap it in value is to match elasticsearch output structure.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct PercentilesMetricResult {\n    /// The result of the percentile metric.\n    pub values: PercentileValues,\n}\n\n/// The top_hits metric results entry\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct TopHitsVecEntry {\n    /// The sort values of the document, depending on the sort criteria in the request.\n    pub sort: Vec<Option<u64>>,\n\n    /// Search results, for queries that include field retrieval requests\n    /// (`docvalue_fields`).\n    #[serde(rename = \"docvalue_fields\")]\n    #[serde(skip_serializing_if = \"HashMap::is_empty\")]\n    pub doc_value_fields: HashMap<String, OwnedValue>,\n}\n\n/// The top_hits metric aggregation results a list of top hits by sort criteria.\n///\n/// The main reason for wrapping it in `hits` is to match elasticsearch output structure.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct TopHitsMetricResult {\n    /// The result of the top_hits metric.\n    pub hits: Vec<TopHitsVecEntry>,\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::AggregationCollector;\n    use crate::query::AllQuery;\n    use crate::schema::{NumericOptions, Schema};\n    use crate::{Index, IndexWriter};\n\n    #[test]\n    fn test_metric_aggregations() {\n        let mut schema_builder = Schema::builder();\n        let field_options = NumericOptions::default().set_fast();\n        let field = schema_builder.add_f64_field(\"price\", field_options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n\n        for i in 0..3 {\n            index_writer\n                .add_document(doc!(\n                    field => i as f64,\n                ))\n                .unwrap();\n        }\n        index_writer.commit().unwrap();\n\n        for i in 3..6 {\n            index_writer\n                .add_document(doc!(\n                    field => i as f64,\n                ))\n                .unwrap();\n        }\n        index_writer.commit().unwrap();\n\n        let aggregations_json = r#\"{\n            \"price_avg\": { \"avg\": { \"field\": \"price\" } },\n            \"price_count\": { \"value_count\": { \"field\": \"price\" } },\n            \"price_max\": { \"max\": { \"field\": \"price\" } },\n            \"price_min\": { \"min\": { \"field\": \"price\" } },\n            \"price_stats\": { \"stats\": { \"field\": \"price\" } },\n            \"price_sum\": { \"sum\": { \"field\": \"price\" } }\n        }\"#;\n        let aggregations: Aggregations = serde_json::from_str(aggregations_json).unwrap();\n        let collector = AggregationCollector::from_aggs(aggregations, Default::default());\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let aggregations_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n        let aggregations_res_json = serde_json::to_value(aggregations_res).unwrap();\n\n        assert_eq!(aggregations_res_json[\"price_avg\"][\"value\"], 2.5);\n        assert_eq!(aggregations_res_json[\"price_count\"][\"value\"], 6.0);\n        assert_eq!(aggregations_res_json[\"price_max\"][\"value\"], 5.0);\n        assert_eq!(aggregations_res_json[\"price_min\"][\"value\"], 0.0);\n        assert_eq!(aggregations_res_json[\"price_sum\"][\"value\"], 15.0);\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/percentiles.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateMetricResult,\n};\nuse crate::aggregation::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// # Percentiles\n///\n/// The percentiles aggregation is a useful tool for understanding the distribution\n/// of a data set. It calculates the values below which a given percentage of the\n/// data falls. For instance, the 95th percentile indicates the value below which\n/// 95% of the data points can be found.\n///\n/// This aggregation can be particularly interesting for analyzing website or service response\n/// times. For example, if the 95th percentile website load time is significantly higher than the\n/// median, this indicates that a small percentage of users are experiencing much slower load times\n/// than the majority.\n///\n/// To use the percentiles aggregation, you'll need to provide a field to\n/// aggregate on. In the case of website load times, this would typically be a\n/// field containing the duration of time it takes for the site to load.\n///\n/// The following example demonstrates a request for the percentiles of the \"load_time\"\n/// field:\n///\n/// ```JSON\n/// {\n///     \"percentiles\": {\n///         \"field\": \"load_time\"\n///     }\n/// }\n/// ```\n///\n/// This request will return an object containing the default percentiles (1, 5,\n/// 25, 50 (median), 75, 95, and 99). You can also customize the percentiles you want to\n/// calculate by providing an array of values in the \"percents\" parameter:\n///\n/// ```JSON\n/// {\n///     \"percentiles\": {\n///         \"field\": \"load_time\",\n///         \"percents\": [10, 20, 30, 40, 50, 60, 70, 80, 90]\n///     }\n/// }\n/// ```\n///\n/// In this example, the aggregation will return the 10th, 20th, 30th, 40th, 50th,\n/// 60th, 70th, 80th, and 90th percentiles of the \"load_time\" field.\n///\n/// Analyzing the percentiles of website load times can help you understand the\n/// user experience and identify areas for optimization. For example, if the 95th\n/// percentile load time is significantly higher than the median, this indicates\n/// that a small percentage of users are experiencing much slower load times than\n/// the majority.\n///\n/// # Estimating Percentiles\n///\n/// While percentiles provide valuable insights into the distribution of data, it's\n/// important to understand that they are often estimates. This is because\n/// calculating exact percentiles for large data sets can be computationally\n/// expensive and time-consuming. As a result, many percentile aggregation\n/// algorithms use approximation techniques to provide faster results.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct PercentilesAggregationReq {\n    /// The field name to compute the percentiles on.\n    pub field: String,\n    /// The percentiles to compute.\n    /// Defaults to [1.0, 5.0, 25.0, 50.0, 75.0, 95.0, 99.0]\n    pub percents: Option<Vec<f64>>,\n    /// Whether to return the percentiles as a hash map\n    #[serde(default = \"default_as_true\")]\n    pub keyed: bool,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(\n        skip_serializing_if = \"Option::is_none\",\n        default,\n        deserialize_with = \"deserialize_option_f64\"\n    )]\n    pub missing: Option<f64>,\n}\nfn default_percentiles() -> &'static [f64] {\n    &[1.0, 5.0, 25.0, 50.0, 75.0, 95.0, 99.0]\n}\nfn default_as_true() -> bool {\n    true\n}\n\nimpl PercentilesAggregationReq {\n    /// Creates a new [`PercentilesAggregationReq`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        PercentilesAggregationReq {\n            field: field_name,\n            percents: None,\n            keyed: default_as_true(),\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n\n    /// Validates the request parameters.\n    pub fn validate(&self) -> crate::Result<()> {\n        if let Some(percents) = self.percents.as_ref() {\n            let all_in_range = percents\n                .iter()\n                .cloned()\n                .all(|percent| (0.0..=100.0).contains(&percent));\n            if !all_in_range {\n                return Err(TantivyError::AggregationError(\n                    AggregationError::InvalidRequest(\n                        \"All percentiles have to be between 0.0 and 100.0\".to_string(),\n                    ),\n                ));\n            }\n        }\n\n        Ok(())\n    }\n}\n\n#[derive(Clone, Debug)]\npub(crate) struct SegmentPercentilesCollector {\n    pub(crate) buckets: Vec<PercentilesCollector>,\n    pub(crate) accessor_idx: usize,\n    /// The type of the field.\n    pub field_type: ColumnType,\n    /// The missing value normalized to the internal u64 representation of the field type.\n    pub missing_u64: Option<u64>,\n    /// The column accessor to access the fast field values.\n    pub accessor: Column<u64>,\n}\n\n#[derive(Clone, Serialize, Deserialize)]\n/// The percentiles collector used during segment collection and for merging results.\npub struct PercentilesCollector {\n    sketch: sketches_ddsketch::DDSketch,\n}\nimpl Default for PercentilesCollector {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\nimpl Debug for PercentilesCollector {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        f.debug_struct(\"IntermediatePercentiles\")\n            .field(\"sketch_len\", &self.sketch.length())\n            .finish()\n    }\n}\nimpl PartialEq for PercentilesCollector {\n    fn eq(&self, _other: &Self) -> bool {\n        false\n    }\n}\n\nfn format_percentile(percentile: f64) -> String {\n    let mut out = percentile.to_string();\n    // Slightly silly way to format trailing decimals\n    if !out.contains('.') {\n        out.push_str(\".0\");\n    }\n    out\n}\n\nimpl PercentilesCollector {\n    /// Convert result into final result. This will query the quantils from the underlying quantil\n    /// collector.\n    pub fn into_final_result(self, req: &PercentilesAggregationReq) -> PercentilesMetricResult {\n        let percentiles: &[f64] = req\n            .percents\n            .as_ref()\n            .map(|el| el.as_ref())\n            .unwrap_or(default_percentiles());\n        let iter_quantile_and_values = percentiles.iter().cloned().map(|percentile| {\n            (\n                percentile,\n                self.sketch\n                    .quantile(percentile / 100.0)\n                    .expect(\n                        \"quantil out of range. This error should have been caught during \\\n                         validation phase\",\n                    )\n                    .unwrap_or(f64::NAN),\n            )\n        });\n\n        let values = if req.keyed {\n            PercentileValues::HashMap(\n                iter_quantile_and_values\n                    .map(|(val, quantil)| (format_percentile(val), quantil))\n                    .collect(),\n            )\n        } else {\n            PercentileValues::Vec(\n                iter_quantile_and_values\n                    .map(|(key, value)| PercentileValuesVecEntry { key, value })\n                    .collect(),\n            )\n        };\n        PercentilesMetricResult { values }\n    }\n\n    fn new() -> Self {\n        let ddsketch_config = sketches_ddsketch::Config::defaults();\n        let sketch = sketches_ddsketch::DDSketch::new(ddsketch_config);\n        Self { sketch }\n    }\n    fn collect(&mut self, val: f64) {\n        self.sketch.add(val);\n    }\n\n    /// Encode the underlying DDSketch to Java-compatible binary format\n    /// for cross-language serialization with Java consumers.\n    pub fn to_sketch_bytes(&self) -> Vec<u8> {\n        self.sketch.to_java_bytes()\n    }\n\n    pub(crate) fn merge_fruits(&mut self, right: PercentilesCollector) -> crate::Result<()> {\n        self.sketch.merge(&right.sketch).map_err(|err| {\n            TantivyError::AggregationError(AggregationError::InternalError(format!(\n                \"Error while merging percentiles {err:?}\"\n            )))\n        })?;\n\n        Ok(())\n    }\n}\n\nimpl SegmentPercentilesCollector {\n    pub fn from_req_and_validate(\n        field_type: ColumnType,\n        missing_u64: Option<u64>,\n        accessor: Column<u64>,\n        accessor_idx: usize,\n    ) -> Self {\n        Self {\n            buckets: Vec::with_capacity(64),\n            field_type,\n            missing_u64,\n            accessor,\n            accessor_idx,\n        }\n    }\n}\n\nimpl SegmentAggregationCollector for SegmentPercentilesCollector {\n    #[inline]\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let name = agg_data.get_metric_req_data(self.accessor_idx).name.clone();\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        // Swap collector with an empty one to avoid cloning\n        let percentiles_collector = std::mem::take(&mut self.buckets[parent_bucket_id as usize]);\n\n        let intermediate_metric_result =\n            IntermediateMetricResult::Percentiles(percentiles_collector);\n\n        results.push(\n            name,\n            IntermediateAggregationResult::Metric(intermediate_metric_result),\n        )?;\n\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let percentiles = &mut self.buckets[parent_bucket_id as usize];\n        agg_data.column_block_accessor.fetch_block_with_missing(\n            docs,\n            &self.accessor,\n            self.missing_u64,\n        );\n\n        for val in agg_data.column_block_accessor.iter_vals() {\n            let val1 = f64_from_fastfield_u64(val, self.field_type);\n            percentiles.collect(val1);\n        }\n\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        while self.buckets.len() <= max_bucket as usize {\n            self.buckets.push(PercentilesCollector::new());\n        }\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use itertools::Itertools;\n    use more_asserts::{assert_ge, assert_le};\n    use rand::rngs::StdRng;\n    use rand::SeedableRng;\n    use serde_json::Value;\n\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::tests::{\n        exec_request_with_query, get_test_index_from_values, get_test_index_from_values_and_terms,\n    };\n    use crate::aggregation::AggregationCollector;\n    use crate::query::AllQuery;\n    use crate::schema::{Schema, FAST};\n    use crate::{assert_nearly_equals, Index};\n\n    #[test]\n    fn test_aggregation_percentiles_empty_index() -> crate::Result<()> {\n        // test index without segments\n        let values = vec![];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"percentiles\": {\n                \"percentiles\": {\n                    \"field\": \"score\",\n                }\n            },\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        assert_eq!(\n            res[\"percentiles\"][\"values\"],\n            json!({\n                \"1.0\": Value::Null,\n                \"5.0\": Value::Null,\n                \"25.0\": Value::Null,\n                \"50.0\": Value::Null,\n                \"75.0\": Value::Null,\n                \"95.0\": Value::Null,\n                \"99.0\": Value::Null,\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_percentile_simple() -> crate::Result<()> {\n        let values = vec![10.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"percentiles\": {\n                \"percentiles\": {\n                    \"field\": \"score\",\n                }\n            },\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        let percents = vec![\"1.0\", \"5.0\", \"25.0\", \"50.0\", \"75.0\", \"95.0\", \"99.0\"];\n        let range = 9.9..10.1;\n        for percent in percents {\n            let val = res[\"percentiles\"][\"values\"][percent].as_f64().unwrap();\n            assert!(range.contains(&val));\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_percentile_parameters() -> crate::Result<()> {\n        let values = vec![10.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_str = r#\"\n        {\n          \"mypercentiles\": {\n            \"percentiles\": {\n              \"field\": \"score\",\n              \"percents\": [ 95, 99, 99.9 ]\n            }\n          }\n        } \"#;\n        let agg_req_1: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        let percents = vec![\"95.0\", \"99.0\", \"99.9\"];\n        let expected_range = 9.9..10.1;\n        for percent in percents {\n            let val = res[\"mypercentiles\"][\"values\"][percent].as_f64().unwrap();\n            assert!(expected_range.contains(&val));\n        }\n        // Keyed false\n        //\n        let agg_req_str = r#\"\n        {\n          \"mypercentiles\": {\n            \"percentiles\": {\n              \"field\": \"score\",\n              \"percents\": [ 95, 99, 99.9 ],\n              \"keyed\": false\n            }\n          }\n        } \"#;\n        let agg_req_1: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n\n        let vals = &res[\"mypercentiles\"][\"values\"];\n        assert_eq!(vals[0][\"key\"].as_f64().unwrap(), 95.0);\n        assert_eq!(vals[1][\"key\"].as_f64().unwrap(), 99.0);\n        assert_eq!(vals[2][\"key\"].as_f64().unwrap(), 99.9);\n        assert_eq!(vals[3][\"key\"], serde_json::Value::Null);\n        assert!(expected_range.contains(&vals[0][\"value\"].as_f64().unwrap()));\n        assert!(expected_range.contains(&vals[1][\"value\"].as_f64().unwrap()));\n        assert!(expected_range.contains(&vals[2][\"value\"].as_f64().unwrap()));\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_percentiles_single_seg() -> crate::Result<()> {\n        test_aggregation_percentiles(true)\n    }\n\n    #[test]\n    fn test_aggregation_percentiles_multi_seg() -> crate::Result<()> {\n        test_aggregation_percentiles(false)\n    }\n\n    fn test_aggregation_percentiles(merge_segments: bool) -> crate::Result<()> {\n        use rand_distr::Distribution;\n        let num_values_in_segment = [100, 30_000, 8000];\n        let lg_norm = rand_distr::LogNormal::new(2.996f64, 0.979f64).unwrap();\n        let mut rng = StdRng::from_seed([1u8; 32]);\n\n        let segment_data = |i| {\n            (0..num_values_in_segment[i])\n                .map(|_| lg_norm.sample(&mut rng))\n                .collect_vec()\n        };\n\n        let values = (0..=2).map(segment_data).collect_vec();\n\n        let mut all_values = values\n            .iter()\n            .flat_map(|el| el.iter().cloned())\n            .collect_vec();\n        all_values.sort_unstable_by(|a, b| a.total_cmp(b));\n\n        fn get_exact_quantil(q: f64, all_values: &[f64]) -> f64 {\n            let q = q / 100.0;\n            assert!((0f64..=1f64).contains(&q));\n\n            let index = (all_values.len() as f64 * q).ceil() as usize;\n            let index = index.min(all_values.len() - 1);\n            all_values[index]\n        }\n\n        let segment_and_values = values\n            .into_iter()\n            .map(|segment_data| {\n                segment_data\n                    .into_iter()\n                    .map(|val| (val, val.to_string()))\n                    .collect_vec()\n            })\n            .collect_vec();\n\n        let index =\n            get_test_index_from_values_and_terms(merge_segments, &segment_and_values).unwrap();\n\n        let reader = index.reader()?;\n\n        let agg_req_str = r#\"\n        {\n          \"mypercentiles\": {\n            \"percentiles\": {\n              \"field\": \"score_f64\",\n              \"percents\": [ 95, 99, 99.9 ]\n            }\n          }\n        } \"#;\n        let agg_req_1: Aggregations = serde_json::from_str(agg_req_str).unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        let vals = &res[\"mypercentiles\"][\"values\"];\n\n        let check_quantil = |exact_quantil: f64, val: f64| {\n            let lower = exact_quantil - exact_quantil * 0.02;\n            let upper = exact_quantil + exact_quantil * 0.02;\n            assert_le!(val, upper);\n            assert_ge!(val, lower);\n        };\n\n        let val = vals[\"95.0\"].as_f64().unwrap();\n        let exact_quantil = get_exact_quantil(95.0, &all_values);\n        check_quantil(exact_quantil, val);\n\n        let val = vals[\"99.0\"].as_f64().unwrap();\n        let exact_quantil = get_exact_quantil(99.0, &all_values);\n        check_quantil(exact_quantil, val);\n\n        let val = vals[\"99.9\"].as_f64().unwrap();\n        let exact_quantil = get_exact_quantil(99.9, &all_values);\n        check_quantil(exact_quantil, val);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_percentiles_missing_sub_agg() -> crate::Result<()> {\n        // This test verifies the `collect` method (in contrast to `collect_block`), which is\n        // called when the sub-aggregations are flushed.\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"texts\", FAST);\n        let score_field_f64 = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                score_field_f64 => 10.0f64,\n                text_field => \"a\"\n            ))?;\n            index_writer.add_document(doc!(\n                score_field_f64 => 10.0f64,\n                text_field => \"a\"\n            ))?;\n\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = {\n            serde_json::from_value(json!({\n                \"range_with_stats\": {\n                    \"terms\": {\n                        \"field\": \"texts\"\n                    },\n                    \"aggs\": {\n                        \"percentiles\": {\n                            \"percentiles\": {\n                                \"field\": \"score\",\n                                \"missing\": 5.0\n                            }\n                        }\n                    }\n                }\n            }))\n            .unwrap()\n        };\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n        assert_eq!(res[\"range_with_stats\"][\"buckets\"][0][\"doc_count\"], 3);\n\n        assert_nearly_equals!(\n            res[\"range_with_stats\"][\"buckets\"][0][\"percentiles\"][\"values\"][\"1.0\"]\n                .as_f64()\n                .unwrap(),\n            5.0028295751107414\n        );\n        assert_nearly_equals!(\n            res[\"range_with_stats\"][\"buckets\"][0][\"percentiles\"][\"values\"][\"99.0\"]\n                .as_f64()\n                .unwrap(),\n            10.07469668951144\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_percentiles_missing() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"texts\", FAST);\n        let score_field_f64 = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                score_field_f64 => 10.0f64,\n                text_field => \"a\"\n            ))?;\n            index_writer.add_document(doc!(\n                score_field_f64 => 10.0f64,\n                text_field => \"a\"\n            ))?;\n\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = {\n            serde_json::from_value(json!({\n                \"percentiles\": {\n                    \"percentiles\": {\n                        \"field\": \"score\",\n                        \"missing\": 5.0\n                    }\n                }\n            }))\n            .unwrap()\n        };\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_nearly_equals!(\n            res[\"percentiles\"][\"values\"][\"1.0\"].as_f64().unwrap(),\n            5.0028295751107414\n        );\n        assert_nearly_equals!(\n            res[\"percentiles\"][\"values\"][\"99.0\"].as_f64().unwrap(),\n            10.07469668951144\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/stats.rs",
    "content": "use std::fmt::Debug;\n\nuse columnar::{Column, ColumnType};\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateAggregationResults, IntermediateMetricResult,\n};\nuse crate::aggregation::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::*;\nuse crate::TantivyError;\n\n/// A multi-value metric aggregation that computes a collection of statistics on numeric values that\n/// are extracted from the aggregated documents.\n/// See [`Stats`] for returned statistics.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"stats\": {\n///         \"field\": \"score\"\n///     }\n///  }\n/// ```\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct StatsAggregation {\n    /// The field name to compute the stats on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl StatsAggregation {\n    /// Creates a new [`StatsAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        StatsAggregation {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Stats contains a collection of statistics.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct Stats {\n    /// The number of documents.\n    pub count: u64,\n    /// The sum of the fast field values.\n    pub sum: f64,\n    /// The min value of the fast field values.\n    pub min: Option<f64>,\n    /// The max value of the fast field values.\n    pub max: Option<f64>,\n    /// The average of the fast field values. `None` if count equals zero.\n    pub avg: Option<f64>,\n}\n\nimpl Stats {\n    pub(crate) fn get_value(&self, agg_property: &str) -> crate::Result<Option<f64>> {\n        match agg_property {\n            \"count\" => Ok(Some(self.count as f64)),\n            \"sum\" => Ok(Some(self.sum)),\n            \"min\" => Ok(self.min),\n            \"max\" => Ok(self.max),\n            \"avg\" => Ok(self.avg),\n            _ => Err(TantivyError::InvalidArgument(format!(\n                \"Unknown property {agg_property} on stats metric aggregation\"\n            ))),\n        }\n    }\n}\n\n/// Intermediate result of the stats aggregation that can be combined with other intermediate\n/// results.\n#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateStats {\n    /// The number of extracted values.\n    pub(crate) count: u64,\n    /// The sum of the extracted values.\n    pub(crate) sum: f64,\n    /// delta for sum needed for [Kahan algorithm for summation](https://en.wikipedia.org/wiki/Kahan_summation_algorithm)\n    pub(crate) delta: f64,\n    /// The min value.\n    pub(crate) min: f64,\n    /// The max value.\n    pub(crate) max: f64,\n}\n\nimpl Default for IntermediateStats {\n    fn default() -> Self {\n        Self {\n            count: 0,\n            sum: 0.0,\n            delta: 0.0,\n            min: f64::MAX,\n            max: f64::MIN,\n        }\n    }\n}\n\nimpl IntermediateStats {\n    /// Returns the number of values collected.\n    pub fn count(&self) -> u64 {\n        self.count\n    }\n\n    /// Returns the sum of all values collected.\n    pub fn sum(&self) -> f64 {\n        self.sum\n    }\n\n    /// Merges the other stats intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateStats) {\n        self.count += other.count;\n\n        // kahan algorithm for sum\n        let y = other.sum - (self.delta + other.delta);\n        let t = self.sum + y;\n        self.delta = (t - self.sum) - y;\n        self.sum = t;\n\n        self.min = self.min.min(other.min);\n        self.max = self.max.max(other.max);\n    }\n\n    /// Computes the final stats value.\n    pub fn finalize(&self) -> Stats {\n        let min = if self.count == 0 {\n            None\n        } else {\n            Some(self.min)\n        };\n        let max = if self.count == 0 {\n            None\n        } else {\n            Some(self.max)\n        };\n        let avg = if self.count == 0 {\n            None\n        } else {\n            Some(self.sum / (self.count as f64))\n        };\n        Stats {\n            count: self.count,\n            sum: self.sum,\n            min,\n            max,\n            avg,\n        }\n    }\n\n    #[inline]\n    pub(in crate::aggregation::metric) fn collect(&mut self, value: f64) {\n        self.count += 1;\n\n        // kahan algorithm for sum\n        let y = value - self.delta;\n        let t = self.sum + y;\n        self.delta = (t - self.sum) - y;\n        self.sum = t;\n\n        self.min = self.min.min(value);\n        self.max = self.max.max(value);\n    }\n}\n\n/// The type of stats aggregation to perform.\n/// Note that not all stats types are supported in the stats aggregation.\n#[derive(Clone, Copy, Debug)]\npub enum StatsType {\n    /// The average of the values.\n    Average,\n    /// The count of the values.\n    Count,\n    /// The maximum value.\n    Max,\n    /// The minimum value.\n    Min,\n    /// The stats (count, sum, min, max, avg) of the values.\n    Stats,\n    /// The extended stats (count, sum, min, max, avg, sum_of_squares, variance, std_deviation,\n    ExtendedStats(Option<f64>), // sigma\n    /// The sum of the values.\n    Sum,\n    /// The percentiles of the values.\n    Percentiles,\n}\n\nfn create_collector<const TYPE_ID: u8>(\n    req: &MetricAggReqData,\n) -> Box<dyn SegmentAggregationCollector> {\n    Box::new(SegmentStatsCollector::<TYPE_ID> {\n        name: req.name.clone(),\n        collecting_for: req.collecting_for,\n        is_number_or_date_type: req.is_number_or_date_type,\n        missing_u64: req.missing_u64,\n        accessor: req.accessor.clone(),\n        buckets: vec![IntermediateStats::default()],\n    })\n}\n\n/// Build a concrete `SegmentStatsCollector` depending on the column type.\npub(crate) fn build_segment_stats_collector(\n    req: &MetricAggReqData,\n) -> crate::Result<Box<dyn SegmentAggregationCollector>> {\n    match req.field_type {\n        ColumnType::I64 => Ok(create_collector::<{ ColumnType::I64 as u8 }>(req)),\n        ColumnType::U64 => Ok(create_collector::<{ ColumnType::U64 as u8 }>(req)),\n        ColumnType::F64 => Ok(create_collector::<{ ColumnType::F64 as u8 }>(req)),\n        ColumnType::Bool => Ok(create_collector::<{ ColumnType::Bool as u8 }>(req)),\n        ColumnType::DateTime => Ok(create_collector::<{ ColumnType::DateTime as u8 }>(req)),\n        ColumnType::Bytes => Ok(create_collector::<{ ColumnType::Bytes as u8 }>(req)),\n        ColumnType::Str => Ok(create_collector::<{ ColumnType::Str as u8 }>(req)),\n        ColumnType::IpAddr => Ok(create_collector::<{ ColumnType::IpAddr as u8 }>(req)),\n    }\n}\n\n#[repr(C)]\n#[derive(Clone, Debug)]\npub(crate) struct SegmentStatsCollector<const COLUMN_TYPE_ID: u8> {\n    pub(crate) missing_u64: Option<u64>,\n    pub(crate) accessor: Column<u64>,\n    pub(crate) is_number_or_date_type: bool,\n    pub(crate) buckets: Vec<IntermediateStats>,\n    pub(crate) name: String,\n    pub(crate) collecting_for: StatsType,\n}\n\nimpl<const COLUMN_TYPE_ID: u8> SegmentAggregationCollector\n    for SegmentStatsCollector<COLUMN_TYPE_ID>\n{\n    #[inline]\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let name = self.name.clone();\n\n        self.prepare_max_bucket(parent_bucket_id, agg_data)?;\n        let stats = self.buckets[parent_bucket_id as usize];\n        let intermediate_metric_result = match self.collecting_for {\n            StatsType::Average => {\n                IntermediateMetricResult::Average(IntermediateAverage::from_stats(stats))\n            }\n            StatsType::Count => {\n                IntermediateMetricResult::Count(IntermediateCount::from_stats(stats))\n            }\n            StatsType::Max => IntermediateMetricResult::Max(IntermediateMax::from_stats(stats)),\n            StatsType::Min => IntermediateMetricResult::Min(IntermediateMin::from_stats(stats)),\n            StatsType::Stats => IntermediateMetricResult::Stats(stats),\n            StatsType::Sum => IntermediateMetricResult::Sum(IntermediateSum::from_stats(stats)),\n            _ => {\n                return Err(TantivyError::InvalidArgument(format!(\n                    \"Unsupported stats type for stats aggregation: {:?}\",\n                    self.collecting_for\n                )))\n            }\n        };\n\n        results.push(\n            name,\n            IntermediateAggregationResult::Metric(intermediate_metric_result),\n        )?;\n\n        Ok(())\n    }\n\n    #[inline]\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        // TODO: remove once we fetch all values for all bucket ids in one go\n        if docs.len() == 1 && self.missing_u64.is_none() {\n            collect_stats::<COLUMN_TYPE_ID>(\n                &mut self.buckets[parent_bucket_id as usize],\n                self.accessor.values_for_doc(docs[0]),\n                self.is_number_or_date_type,\n            )?;\n\n            return Ok(());\n        }\n        agg_data.column_block_accessor.fetch_block_with_missing(\n            docs,\n            &self.accessor,\n            self.missing_u64,\n        );\n        collect_stats::<COLUMN_TYPE_ID>(\n            &mut self.buckets[parent_bucket_id as usize],\n            agg_data.column_block_accessor.iter_vals(),\n            self.is_number_or_date_type,\n        )?;\n\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let required_buckets = (max_bucket as usize) + 1;\n        if self.buckets.len() < required_buckets {\n            self.buckets\n                .resize_with(required_buckets, IntermediateStats::default);\n        }\n        Ok(())\n    }\n}\n\n#[inline]\nfn collect_stats<const COLUMN_TYPE_ID: u8>(\n    stats: &mut IntermediateStats,\n    vals: impl Iterator<Item = u64>,\n    is_number_or_date_type: bool,\n) -> crate::Result<()> {\n    if is_number_or_date_type {\n        for val in vals {\n            let val1 = convert_to_f64::<COLUMN_TYPE_ID>(val);\n            stats.collect(val1);\n        }\n    } else {\n        for _val in vals {\n            // we ignore the value and simply record that we got something\n            stats.collect(0.0);\n        }\n    }\n\n    Ok(())\n}\n\n#[cfg(test)]\nmod tests {\n    use serde_json::Value;\n\n    use crate::aggregation::agg_req::{Aggregation, Aggregations};\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::tests::{\n        exec_request_with_query, get_test_index_2_segments, get_test_index_from_values,\n    };\n    use crate::aggregation::AggregationCollector;\n    use crate::query::{AllQuery, TermQuery};\n    use crate::schema::{IndexRecordOption, Schema, FAST};\n    use crate::{Index, IndexWriter, Term};\n\n    #[test]\n    fn test_aggregation_stats_empty_index() -> crate::Result<()> {\n        // test index without segments\n        let values = vec![];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"stats\": {\n                \"stats\": {\n                    \"field\": \"score\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        assert_eq!(\n            res[\"stats\"],\n            json!({\n                \"avg\": Value::Null,\n                \"count\": 0,\n                \"max\": Value::Null,\n                \"min\": Value::Null,\n                \"sum\": 0.0\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_stats_simple() -> crate::Result<()> {\n        let values = vec![10.0];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"stats\": {\n                \"stats\": {\n                    \"field\": \"score\",\n                },\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        assert_eq!(\n            res[\"stats\"],\n            json!({\n                \"avg\": 10.0,\n                \"count\": 1,\n                \"max\": 10.0,\n                \"min\": 10.0,\n                \"sum\": 10.0\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_stats() -> crate::Result<()> {\n        let index = get_test_index_2_segments(false)?;\n\n        let reader = index.reader()?;\n        let text_field = reader.searcher().schema().get_field(\"text\").unwrap();\n\n        let term_query = TermQuery::new(\n            Term::from_field_text(text_field, \"cool\"),\n            IndexRecordOption::Basic,\n        );\n\n        let range_agg: Aggregation = {\n            serde_json::from_value(json!({\n                \"range\": {\n                    \"field\": \"score\",\n                    \"ranges\": [ { \"from\": 3.0f64, \"to\": 7.0f64 }, { \"from\": 7.0f64, \"to\": 19.0f64 }, { \"from\": 19.0f64, \"to\": 20.0f64 }  ]\n                },\n                \"aggs\": {\n                    \"stats\": {\n                        \"stats\": {\n                            \"field\": \"score\"\n                        }\n                    }\n                }\n            }))\n            .unwrap()\n        };\n\n        let agg_req_1: Aggregations = serde_json::from_value(json!({\n            \"stats_i64\": {\n                \"stats\": {\n                    \"field\": \"score_i64\",\n                },\n            },\n            \"stats_f64\": {\n                \"stats\": {\n                    \"field\": \"score_f64\",\n                },\n            },\n            \"stats\": {\n                \"stats\": {\n                    \"field\": \"score\",\n                },\n            },\n            \"count_str\": {\n                \"value_count\": {\n                    \"field\": \"text\",\n                },\n            },\n            \"range\": range_agg\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(agg_req_1, Default::default());\n\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&term_query, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        assert_eq!(\n            res[\"stats\"],\n            json!({\n                \"avg\": 12.142857142857142,\n                \"count\": 7,\n                \"max\": 44.0,\n                \"min\": 1.0,\n                \"sum\": 85.0\n            })\n        );\n\n        assert_eq!(\n            res[\"stats_i64\"],\n            json!({\n                \"avg\": 12.142857142857142,\n                \"count\": 7,\n                \"max\": 44.0,\n                \"min\": 1.0,\n                \"sum\": 85.0\n            })\n        );\n\n        assert_eq!(\n            res[\"stats_f64\"],\n            json!({\n                \"avg\":  12.214285714285714,\n                \"count\": 7,\n                \"max\": 44.5,\n                \"min\": 1.0,\n                \"sum\": 85.5\n            })\n        );\n\n        assert_eq!(\n            res[\"range\"][\"buckets\"][2][\"stats\"],\n            json!({\n                \"avg\": 10.666666666666666,\n                \"count\": 3,\n                \"max\": 14.0,\n                \"min\": 7.0,\n                \"sum\": 32.0\n            })\n        );\n\n        assert_eq!(\n            res[\"range\"][\"buckets\"][3][\"stats\"],\n            json!({\n                \"avg\": serde_json::Value::Null,\n                \"count\": 0,\n                \"max\": serde_json::Value::Null,\n                \"min\": serde_json::Value::Null,\n                \"sum\": 0.0,\n            })\n        );\n\n        assert_eq!(\n            res[\"count_str\"],\n            json!({\n                \"value\": 7.0,\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_stats_json() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with empty json\n        index_writer.add_document(doc!()).unwrap();\n        index_writer.commit().unwrap();\n        // => Segment with json, but no field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"different_field\": \"blue\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        //// => Segment with field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"partially_empty\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"stats\": {\n                    \"field\": \"json.partially_empty\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res[\"my_stats\"],\n            json!({\n                \"avg\":  10.0,\n                \"count\": 1,\n                \"max\": 10.0,\n                \"min\": 10.0,\n                \"sum\": 10.0\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_stats_json_missing() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        // => Segment with empty json\n        index_writer.add_document(doc!()).unwrap();\n        index_writer.commit().unwrap();\n        // => Segment with json, but no field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"different_field\": \"blue\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        //// => Segment with field partially_empty\n        index_writer\n            .add_document(doc!(json => json!({\"partially_empty\": 10.0})))\n            .unwrap();\n        index_writer.add_document(doc!())?;\n        index_writer.commit().unwrap();\n\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"stats\": {\n                    \"field\": \"json.partially_empty\",\n                    \"missing\": 0.0\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res[\"my_stats\"],\n            json!({\n                \"avg\":  2.5,\n                \"count\": 4,\n                \"max\": 10.0,\n                \"min\": 0.0,\n                \"sum\": 10.0\n            })\n        );\n\n        // From string\n        let agg_req: Aggregations = serde_json::from_value(json!({\n            \"my_stats\": {\n                \"stats\": {\n                    \"field\": \"json.partially_empty\",\n                    \"missing\": \"0.0\"\n                },\n            }\n        }))\n        .unwrap();\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res[\"my_stats\"],\n            json!({\n                \"avg\":  2.5,\n                \"count\": 4,\n                \"max\": 10.0,\n                \"min\": 0.0,\n                \"sum\": 10.0\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_stats_json_missing_sub_agg() -> crate::Result<()> {\n        // This test verifies the `collect` method (in contrast to `collect_block`), which is\n        // called when the sub-aggregations are flushed.\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"texts\", FAST);\n        let score_field_f64 = schema_builder.add_f64_field(\"score\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                score_field_f64 => 10.0f64,\n                text_field => \"a\"\n            ))?;\n\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n\n            index_writer.commit()?;\n        }\n\n        let agg_req: Aggregations = {\n            serde_json::from_value(json!({\n                \"range_with_stats\": {\n                    \"terms\": {\n                        \"field\": \"texts\"\n                    },\n                    \"aggs\": {\n                        \"my_stats\": {\n                            \"stats\": {\n                                \"field\": \"score\",\n                                \"missing\": 0.0\n                            }\n                        }\n                    }\n                }\n            }))\n            .unwrap()\n        };\n\n        let res = exec_request_with_query(agg_req, &index, None)?;\n\n        assert_eq!(\n            res[\"range_with_stats\"][\"buckets\"][0][\"my_stats\"][\"count\"],\n            2\n        );\n        assert_eq!(\n            res[\"range_with_stats\"][\"buckets\"][0][\"my_stats\"][\"min\"],\n            0.0\n        );\n        assert_eq!(\n            res[\"range_with_stats\"][\"buckets\"][0][\"my_stats\"][\"avg\"],\n            5.0\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/sum.rs",
    "content": "use std::fmt::Debug;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::*;\nuse crate::aggregation::*;\n\n/// A single-value metric aggregation that sums up numeric values that are\n/// extracted from the aggregated documents.\n/// See [super::SingleMetricResult] for return value.\n///\n/// # JSON Format\n/// ```json\n/// {\n///     \"sum\": {\n///         \"field\": \"score\"\n///     }\n/// }\n/// ```\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct SumAggregation {\n    /// The field name to compute the minimum on.\n    pub field: String,\n    /// The missing parameter defines how documents that are missing a value should be treated.\n    /// By default they will be ignored but it is also possible to treat them as if they had a\n    /// value. Examples in JSON format:\n    /// { \"field\": \"my_numbers\", \"missing\": \"10.0\" }\n    #[serde(default, deserialize_with = \"deserialize_option_f64\")]\n    pub missing: Option<f64>,\n}\n\nimpl SumAggregation {\n    /// Creates a new [`SumAggregation`] instance from a field name.\n    pub fn from_field_name(field_name: String) -> Self {\n        Self {\n            field: field_name,\n            missing: None,\n        }\n    }\n    /// Returns the field name the aggregation is computed on.\n    pub fn field_name(&self) -> &str {\n        &self.field\n    }\n}\n\n/// Intermediate result of the minimum aggregation that can be combined with other intermediate\n/// results.\n#[derive(Default, Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct IntermediateSum {\n    stats: IntermediateStats,\n}\n\nimpl IntermediateSum {\n    /// Creates a new [`IntermediateSum`] instance from a [`SegmentStatsCollector`].\n    pub(crate) fn from_stats(stats: IntermediateStats) -> Self {\n        Self { stats }\n    }\n    /// Merges the other intermediate result into self.\n    pub fn merge_fruits(&mut self, other: IntermediateSum) {\n        self.stats.merge_fruits(other.stats);\n    }\n    /// Computes the final minimum value.\n    pub fn finalize(&self) -> Option<f64> {\n        Some(self.stats.finalize().sum)\n    }\n}\n"
  },
  {
    "path": "src/aggregation/metric/top_hits.rs",
    "content": "use std::collections::HashMap;\nuse std::net::Ipv6Addr;\n\nuse columnar::{Column, ColumnType, ColumnarReader, DynamicColumn};\nuse common::json_path_writer::JSON_PATH_SEGMENT_SEP_STR;\nuse common::DateTime;\nuse regex::Regex;\nuse serde::ser::SerializeMap;\nuse serde::{Deserialize, Deserializer, Serialize, Serializer};\n\nuse super::{TopHitsMetricResult, TopHitsVecEntry};\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::bucket::Order;\nuse crate::aggregation::intermediate_agg_result::{\n    IntermediateAggregationResult, IntermediateMetricResult,\n};\nuse crate::aggregation::segment_agg_result::SegmentAggregationCollector;\nuse crate::aggregation::{AggregationError, BucketId};\nuse crate::collector::sort_key::ReverseComparator;\nuse crate::collector::TopNComputer;\nuse crate::schema::OwnedValue;\nuse crate::{DocAddress, DocId, SegmentOrdinal};\n\n/// Contains all information required by the TopHitsSegmentCollector to perform the\n/// top_hits aggregation on a segment.\n#[derive(Default)]\npub struct TopHitsAggReqData {\n    /// The accessors to access the fast field values.\n    pub accessors: Vec<(Column<u64>, ColumnType)>,\n    /// The accessors to access the fast field values for retrieving document fields.\n    pub value_accessors: HashMap<String, Vec<DynamicColumn>>,\n    /// The ordinal of the segment this request data is for.\n    pub segment_ordinal: SegmentOrdinal,\n    /// The name of the aggregation.\n    pub name: String,\n    /// The top_hits aggregation request.\n    pub req: TopHitsAggregationReq,\n}\n\nimpl TopHitsAggReqData {\n    /// Estimate the memory consumption of this struct in bytes.\n    pub fn get_memory_consumption(&self) -> usize {\n        std::mem::size_of::<Self>()\n    }\n}\n\n/// # Top Hits\n///\n/// The top hits aggregation is a useful tool to answer questions like:\n/// - \"What are the most recent posts by each author?\"\n/// - \"What are the most popular items in each category?\"\n///\n/// It does so by keeping track of the most relevant document being aggregated,\n/// in terms of a sort criterion that can consist of multiple fields and their\n/// sort-orders (ascending or descending).\n///\n/// `top_hits` should not be used as a top-level aggregation. It is intended to be\n/// used as a sub-aggregation, inside a `terms` aggregation or a `filters` aggregation,\n/// for example.\n///\n/// Note that this aggregator does not return the actual document addresses, but\n/// rather a list of the values of the fields that were requested to be retrieved.\n/// These values can be specified in the `docvalue_fields` parameter, which can include\n/// a list of fast fields to be retrieved. At the moment, only fast fields are supported\n/// but it is possible that we support the `fields` parameter to retrieve any stored\n/// field in the future.\n///\n/// The following example demonstrates a request for the top_hits aggregation:\n/// ```JSON\n/// {\n///     \"aggs\": {\n///         \"top_authors\": {\n///             \"terms\": {\n///                 \"field\": \"author\",\n///                 \"size\": 5\n///             }\n///         },\n///         \"aggs\": {\n///             \"top_hits\": {\n///                 \"size\": 2,\n///                 \"from\": 0\n///                 \"sort\": [\n///                     { \"date\": \"desc\" }\n///                 ]\n///                 \"docvalue_fields\": [\"date\", \"title\", \"iden\"]\n///             }\n///         }\n/// }\n/// ```\n///\n/// This request will return an object containing the top two documents, sorted\n/// by the `date` field in descending order. You can also sort by multiple fields, which\n/// helps to resolve ties. The aggregation object for each bucket will look like:\n/// ```JSON\n/// {\n///     \"hits\": [\n///         {\n///           \"score\": [<time_u64>],\n///           \"docvalue_fields\": {\n///             \"date\": \"<date_RFC3339>\",\n///             \"title\": \"<title>\",\n///             \"iden\": \"<iden>\"\n///           }\n///         },\n///         {\n///           \"score\": [<time_u64>]\n///           \"docvalue_fields\": {\n///             \"date\": \"<date_RFC3339>\",\n///             \"title\": \"<title>\",\n///             \"iden\": \"<iden>\"\n///           }\n///         }\n///     ]\n/// }\n/// ```\n#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, Default)]\npub struct TopHitsAggregationReq {\n    sort: Vec<KeyOrder>,\n    size: usize,\n    from: Option<usize>,\n\n    #[serde(rename = \"docvalue_fields\")]\n    #[serde(default)]\n    doc_value_fields: Vec<String>,\n\n    // Not supported\n    _source: Option<serde_json::Value>,\n    fields: Option<serde_json::Value>,\n    script_fields: Option<serde_json::Value>,\n    highlight: Option<serde_json::Value>,\n    explain: Option<serde_json::Value>,\n    version: Option<serde_json::Value>,\n}\n\n#[derive(Debug, Clone, PartialEq, Default)]\nstruct KeyOrder {\n    field: String,\n    order: Order,\n}\n\nimpl Serialize for KeyOrder {\n    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {\n        let KeyOrder { field, order } = self;\n        let mut map = serializer.serialize_map(Some(1))?;\n        map.serialize_entry(field, order)?;\n        map.end()\n    }\n}\n\nimpl<'de> Deserialize<'de> for KeyOrder {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: Deserializer<'de> {\n        let mut key_order = <HashMap<String, Order>>::deserialize(deserializer)?.into_iter();\n        let (field, order) = key_order.next().ok_or(serde::de::Error::custom(\n            \"Expected exactly one key-value pair in sort parameter of top_hits, found none\",\n        ))?;\n        if key_order.next().is_some() {\n            return Err(serde::de::Error::custom(format!(\n                \"Expected exactly one key-value pair in sort parameter of top_hits, found \\\n                 {key_order:?}\"\n            )));\n        }\n        Ok(Self { field, order })\n    }\n}\n\n// Transform a glob (`pattern*`, for example) into a regex::Regex (`^pattern.*$`)\nfn globbed_string_to_regex(glob: &str) -> Result<Regex, crate::TantivyError> {\n    // Replace `*` glob with `.*` regex\n    let sanitized = format!(\"^{}$\", regex::escape(glob).replace(r\"\\*\", \".*\"));\n    Regex::new(&sanitized.replace('*', \".*\")).map_err(|e| {\n        crate::TantivyError::SchemaError(format!(\"Invalid regex '{glob}' in docvalue_fields: {e}\"))\n    })\n}\n\nfn use_doc_value_fields_err(parameter: &str) -> crate::Result<()> {\n    Err(crate::TantivyError::AggregationError(\n        AggregationError::InvalidRequest(format!(\n            \"The `{parameter}` parameter is not supported, only `docvalue_fields` is supported in \\\n             `top_hits` aggregation\"\n        )),\n    ))\n}\nfn unsupported_err(parameter: &str) -> crate::Result<()> {\n    Err(crate::TantivyError::AggregationError(\n        AggregationError::InvalidRequest(format!(\n            \"The `{parameter}` parameter is not supported in the `top_hits` aggregation\"\n        )),\n    ))\n}\n\nimpl TopHitsAggregationReq {\n    /// Validate and resolve field retrieval parameters\n    pub fn validate_and_resolve_field_names(\n        &mut self,\n        reader: &ColumnarReader,\n    ) -> crate::Result<()> {\n        if self._source.is_some() {\n            use_doc_value_fields_err(\"_source\")?;\n        }\n        if self.fields.is_some() {\n            use_doc_value_fields_err(\"fields\")?;\n        }\n        if self.script_fields.is_some() {\n            use_doc_value_fields_err(\"script_fields\")?;\n        }\n        if self.explain.is_some() {\n            unsupported_err(\"explain\")?;\n        }\n        if self.highlight.is_some() {\n            unsupported_err(\"highlight\")?;\n        }\n        if self.version.is_some() {\n            unsupported_err(\"version\")?;\n        }\n\n        self.doc_value_fields = self\n            .doc_value_fields\n            .iter()\n            .map(|field| {\n                if !field.contains('*')\n                    && reader\n                        .iter_columns()?\n                        .any(|(name, _)| name.as_str() == field)\n                {\n                    return Ok(vec![field.to_owned()]);\n                }\n\n                let pattern = globbed_string_to_regex(field)?;\n                let fields = reader\n                    .iter_columns()?\n                    .map(|(name, _)| {\n                        // normalize path from internal fast field repr\n                        name.replace(JSON_PATH_SEGMENT_SEP_STR, \".\")\n                    })\n                    .filter(|name| pattern.is_match(name))\n                    .collect::<Vec<_>>();\n                assert!(\n                    !fields.is_empty(),\n                    \"No fields matched the glob '{field}' in docvalue_fields\"\n                );\n                Ok(fields)\n            })\n            .collect::<crate::Result<Vec<_>>>()?\n            .into_iter()\n            .flatten()\n            .collect();\n\n        Ok(())\n    }\n\n    /// Return fields accessed by the aggregator, in order.\n    pub fn field_names(&self) -> Vec<&str> {\n        self.sort\n            .iter()\n            .map(|KeyOrder { field, .. }| field.as_str())\n            .chain(self.doc_value_fields.iter().map(|s| s.as_str()))\n            .collect()\n    }\n\n    /// Return fields accessed by the aggregator's value retrieval.\n    pub fn value_field_names(&self) -> Vec<&str> {\n        self.doc_value_fields.iter().map(|s| s.as_str()).collect()\n    }\n\n    fn get_document_field_data(\n        &self,\n        accessors: &HashMap<String, Vec<DynamicColumn>>,\n        doc_id: DocId,\n    ) -> HashMap<String, FastFieldValue> {\n        let doc_value_fields = self\n            .doc_value_fields\n            .iter()\n            .map(|field| {\n                let accessors = accessors\n                    .get(field)\n                    .unwrap_or_else(|| panic!(\"field '{field}' not found in accessors\"));\n\n                let values: Vec<FastFieldValue> = accessors\n                    .iter()\n                    .flat_map(|accessor| match accessor {\n                        DynamicColumn::U64(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::U64)\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::I64(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::I64)\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::F64(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::F64)\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::Bytes(accessor) => accessor\n                            .term_ords(doc_id)\n                            .map(|term_ord| {\n                                let mut buffer = vec![];\n                                assert!(\n                                    accessor\n                                        .ord_to_bytes(term_ord, &mut buffer)\n                                        .expect(\"could not read term dictionary\"),\n                                    \"term corresponding to term_ord does not exist\"\n                                );\n                                FastFieldValue::Bytes(buffer)\n                            })\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::Str(accessor) => accessor\n                            .term_ords(doc_id)\n                            .map(|term_ord| {\n                                let mut buffer = vec![];\n                                assert!(\n                                    accessor\n                                        .ord_to_bytes(term_ord, &mut buffer)\n                                        .expect(\"could not read term dictionary\"),\n                                    \"term corresponding to term_ord does not exist\"\n                                );\n                                FastFieldValue::Str(String::from_utf8(buffer).unwrap())\n                            })\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::Bool(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::Bool)\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::IpAddr(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::IpAddr)\n                            .collect::<Vec<_>>(),\n                        DynamicColumn::DateTime(accessor) => accessor\n                            .values_for_doc(doc_id)\n                            .map(FastFieldValue::Date)\n                            .collect::<Vec<_>>(),\n                    })\n                    .collect();\n\n                (field.to_owned(), FastFieldValue::Array(values))\n            })\n            .collect();\n        doc_value_fields\n    }\n}\n\n/// A retrieved value from a fast field.\n#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]\npub enum FastFieldValue {\n    /// The str type is used for any text information.\n    Str(String),\n    /// Unsigned 64-bits Integer `u64`\n    U64(u64),\n    /// Signed 64-bits Integer `i64`\n    I64(i64),\n    /// 64-bits Float `f64`\n    F64(f64),\n    /// Bool value\n    Bool(bool),\n    /// Date/time with nanoseconds precision\n    Date(DateTime),\n    /// Arbitrarily sized byte array\n    Bytes(Vec<u8>),\n    /// IpV6 Address. Internally there is no IpV4, it needs to be converted to `Ipv6Addr`.\n    IpAddr(Ipv6Addr),\n    /// A list of values.\n    Array(Vec<Self>),\n}\n\nimpl From<FastFieldValue> for OwnedValue {\n    fn from(value: FastFieldValue) -> Self {\n        match value {\n            FastFieldValue::Str(s) => OwnedValue::Str(s),\n            FastFieldValue::U64(u) => OwnedValue::U64(u),\n            FastFieldValue::I64(i) => OwnedValue::I64(i),\n            FastFieldValue::F64(f) => OwnedValue::F64(f),\n            FastFieldValue::Bool(b) => OwnedValue::Bool(b),\n            FastFieldValue::Date(d) => OwnedValue::Date(d),\n            FastFieldValue::Bytes(b) => OwnedValue::Bytes(b),\n            FastFieldValue::IpAddr(ip) => OwnedValue::IpAddr(ip),\n            FastFieldValue::Array(a) => {\n                OwnedValue::Array(a.into_iter().map(OwnedValue::from).collect())\n            }\n        }\n    }\n}\n\n/// Holds a fast field value in its u64 representation, and the order in which it should be sorted.\n#[derive(Clone, Serialize, Deserialize, Debug)]\nstruct DocValueAndOrder {\n    /// A fast field value in its u64 representation.\n    value: Option<u64>,\n    /// Sort order for the value\n    order: Order,\n}\n\nimpl Ord for DocValueAndOrder {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        let invert = |cmp: std::cmp::Ordering| match self.order {\n            Order::Asc => cmp,\n            Order::Desc => cmp.reverse(),\n        };\n\n        match (self.value, other.value) {\n            (Some(self_value), Some(other_value)) => invert(self_value.cmp(&other_value)),\n            (Some(_), None) => std::cmp::Ordering::Greater,\n            (None, Some(_)) => std::cmp::Ordering::Less,\n            (None, None) => std::cmp::Ordering::Equal,\n        }\n    }\n}\n\nimpl PartialOrd for DocValueAndOrder {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl PartialEq for DocValueAndOrder {\n    fn eq(&self, other: &Self) -> bool {\n        self.value.cmp(&other.value) == std::cmp::Ordering::Equal\n    }\n}\n\nimpl Eq for DocValueAndOrder {}\n\n#[derive(Clone, Serialize, Deserialize, Debug)]\nstruct DocSortValuesAndFields {\n    sorts: Vec<DocValueAndOrder>,\n\n    #[serde(rename = \"docvalue_fields\")]\n    #[serde(skip_serializing_if = \"HashMap::is_empty\")]\n    doc_value_fields: HashMap<String, FastFieldValue>,\n}\n\nimpl Ord for DocSortValuesAndFields {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        for (self_feature, other_feature) in self.sorts.iter().zip(other.sorts.iter()) {\n            let cmp = self_feature.cmp(other_feature);\n            if cmp != std::cmp::Ordering::Equal {\n                return cmp;\n            }\n        }\n        std::cmp::Ordering::Equal\n    }\n}\n\nimpl PartialOrd for DocSortValuesAndFields {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl PartialEq for DocSortValuesAndFields {\n    fn eq(&self, other: &Self) -> bool {\n        self.cmp(other) == std::cmp::Ordering::Equal\n    }\n}\n\nimpl Eq for DocSortValuesAndFields {}\n\n/// The TopHitsCollector used for collecting over segments and merging results.\n#[derive(Clone, Serialize, Deserialize, Debug)]\npub struct TopHitsTopNComputer {\n    req: TopHitsAggregationReq,\n    top_n: TopNComputer<DocSortValuesAndFields, DocAddress, ReverseComparator>,\n}\n\nimpl std::cmp::PartialEq for TopHitsTopNComputer {\n    fn eq(&self, _other: &Self) -> bool {\n        false\n    }\n}\n\nimpl TopHitsTopNComputer {\n    /// Create a new TopHitsCollector\n    pub fn new(req: &TopHitsAggregationReq) -> Self {\n        Self {\n            top_n: TopNComputer::new_with_comparator(\n                req.size + req.from.unwrap_or(0),\n                ReverseComparator,\n            ),\n            req: req.clone(),\n        }\n    }\n\n    fn collect(&mut self, features: DocSortValuesAndFields, doc: DocAddress) {\n        self.top_n.push(features, doc);\n    }\n\n    pub(crate) fn merge_fruits(&mut self, other_fruit: Self) -> crate::Result<()> {\n        for doc in other_fruit.top_n.into_vec() {\n            self.collect(doc.sort_key, doc.doc);\n        }\n        Ok(())\n    }\n\n    /// Finalize by converting self into the final result form\n    pub fn into_final_result(self) -> TopHitsMetricResult {\n        let mut hits: Vec<TopHitsVecEntry> = self\n            .top_n\n            .into_sorted_vec()\n            .into_iter()\n            .map(|doc| TopHitsVecEntry {\n                sort: doc.sort_key.sorts.iter().map(|f| f.value).collect(),\n                doc_value_fields: doc\n                    .sort_key\n                    .doc_value_fields\n                    .into_iter()\n                    .map(|(k, v)| (k, v.into()))\n                    .collect(),\n            })\n            .collect();\n\n        // Remove the first `from` elements\n        // Truncating from end would be more efficient, but we need to truncate from the front\n        // because `into_sorted_vec` gives us a descending order because of the inverted\n        // `Ord` semantics of the heap elements.\n        hits.drain(..self.req.from.unwrap_or(0));\n        TopHitsMetricResult { hits }\n    }\n}\n\n#[derive(Clone, Debug)]\npub(crate) struct TopHitsSegmentCollector {\n    segment_ordinal: SegmentOrdinal,\n    accessor_idx: usize,\n    buckets: Vec<TopNComputer<Vec<DocValueAndOrder>, DocAddress, ReverseComparator>>,\n    num_hits: usize,\n}\n\nimpl TopHitsSegmentCollector {\n    pub fn from_req(\n        req: &TopHitsAggregationReq,\n        accessor_idx: usize,\n        segment_ordinal: SegmentOrdinal,\n    ) -> Self {\n        let num_hits = req.size + req.from.unwrap_or(0);\n        Self {\n            num_hits,\n            segment_ordinal,\n            accessor_idx,\n            buckets: vec![TopNComputer::new_with_comparator(num_hits, ReverseComparator); 1],\n        }\n    }\n    fn get_top_hits_computer(\n        &mut self,\n        parent_bucket_id: BucketId,\n        value_accessors: &HashMap<String, Vec<DynamicColumn>>,\n        req: &TopHitsAggregationReq,\n    ) -> TopHitsTopNComputer {\n        if parent_bucket_id as usize >= self.buckets.len() {\n            return TopHitsTopNComputer::new(req);\n        }\n        let top_n = std::mem::replace(\n            &mut self.buckets[parent_bucket_id as usize],\n            TopNComputer::new(0),\n        );\n        let mut top_hits_computer = TopHitsTopNComputer::new(req);\n        let top_results = top_n.into_vec();\n\n        for res in top_results {\n            let doc_value_fields = req.get_document_field_data(value_accessors, res.doc.doc_id);\n            top_hits_computer.collect(\n                DocSortValuesAndFields {\n                    sorts: res.sort_key,\n                    doc_value_fields,\n                },\n                res.doc,\n            );\n        }\n\n        top_hits_computer\n    }\n}\n\nimpl SegmentAggregationCollector for TopHitsSegmentCollector {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut crate::aggregation::intermediate_agg_result::IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        let req_data = agg_data.get_top_hits_req_data(self.accessor_idx);\n\n        let value_accessors = &req_data.value_accessors;\n\n        let intermediate_result = IntermediateMetricResult::TopHits(self.get_top_hits_computer(\n            parent_bucket_id,\n            value_accessors,\n            &req_data.req,\n        ));\n        results.push(\n            req_data.name.to_string(),\n            IntermediateAggregationResult::Metric(intermediate_result),\n        )\n    }\n\n    /// TODO: Consider a caching layer to reduce the call overhead\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        let top_n = &mut self.buckets[parent_bucket_id as usize];\n        let req_data = agg_data.get_top_hits_req_data(self.accessor_idx);\n        let req = &req_data.req;\n        let accessors = &req_data.accessors;\n        for &doc_id in docs {\n            // TODO: this is terrible, a new vec is allocated for every doc\n            // We can fetch blocks instead\n            // We don't need to store the order for every value\n            let sorts: Vec<DocValueAndOrder> = req\n                .sort\n                .iter()\n                .enumerate()\n                .map(|(idx, KeyOrder { order, .. })| {\n                    let order = *order;\n                    let value = accessors\n                        .get(idx)\n                        .expect(\"could not find field in accessors\")\n                        .0\n                        .values_for_doc(doc_id)\n                        .next();\n                    DocValueAndOrder { value, order }\n                })\n                .collect();\n\n            top_n.push(\n                sorts,\n                DocAddress {\n                    segment_ord: self.segment_ordinal,\n                    doc_id,\n                },\n            );\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        _agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        self.buckets.resize(\n            (max_bucket as usize) + 1,\n            TopNComputer::new_with_comparator(self.num_hits, ReverseComparator),\n        );\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use common::DateTime;\n    use pretty_assertions::assert_eq;\n    use serde_json::Value;\n    use time::macros::datetime;\n\n    use super::{DocSortValuesAndFields, DocValueAndOrder, Order};\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::bucket::tests::get_test_index_from_docs;\n    use crate::aggregation::tests::get_test_index_from_values;\n    use crate::aggregation::AggregationCollector;\n    use crate::collector::sort_key::ReverseComparator;\n    use crate::collector::ComparableDoc;\n    use crate::query::AllQuery;\n    use crate::schema::OwnedValue;\n\n    fn invert_order(cmp_feature: DocValueAndOrder) -> DocValueAndOrder {\n        let DocValueAndOrder { value, order } = cmp_feature;\n        let order = match order {\n            Order::Asc => Order::Desc,\n            Order::Desc => Order::Asc,\n        };\n        DocValueAndOrder { value, order }\n    }\n\n    fn collector_with_capacity(capacity: usize) -> super::TopHitsTopNComputer {\n        super::TopHitsTopNComputer {\n            top_n: super::TopNComputer::new_with_comparator(capacity, ReverseComparator),\n            req: Default::default(),\n        }\n    }\n\n    fn invert_order_features(mut cmp_features: DocSortValuesAndFields) -> DocSortValuesAndFields {\n        cmp_features.sorts = cmp_features\n            .sorts\n            .into_iter()\n            .map(invert_order)\n            .collect::<Vec<_>>();\n        cmp_features\n    }\n\n    #[test]\n    fn test_comparable_doc_feature() -> crate::Result<()> {\n        let small = DocValueAndOrder {\n            value: Some(1),\n            order: Order::Asc,\n        };\n        let big = DocValueAndOrder {\n            value: Some(2),\n            order: Order::Asc,\n        };\n        let none = DocValueAndOrder {\n            value: None,\n            order: Order::Asc,\n        };\n\n        assert!(small < big);\n        assert!(none < small);\n        assert!(none < big);\n\n        let small = invert_order(small);\n        let big = invert_order(big);\n        let none = invert_order(none);\n\n        assert!(small > big);\n        assert!(none < small);\n        assert!(none < big);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_comparable_doc_features() -> crate::Result<()> {\n        let features_1 = DocSortValuesAndFields {\n            sorts: vec![DocValueAndOrder {\n                value: Some(1),\n                order: Order::Asc,\n            }],\n            doc_value_fields: Default::default(),\n        };\n\n        let features_2 = DocSortValuesAndFields {\n            sorts: vec![DocValueAndOrder {\n                value: Some(2),\n                order: Order::Asc,\n            }],\n            doc_value_fields: Default::default(),\n        };\n\n        assert!(features_1 < features_2);\n\n        assert!(invert_order_features(features_1.clone()) > invert_order_features(features_2));\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_top_hits_empty_index() -> crate::Result<()> {\n        let values = vec![];\n\n        let index = get_test_index_from_values(false, &values)?;\n\n        let d: Aggregations = serde_json::from_value(json!({\n            \"top_hits_req\": {\n                \"top_hits\": {\n                    \"size\": 2,\n                    \"sort\": [\n                        { \"date\": \"desc\" }\n                    ],\n                    \"from\": 0,\n                }\n            }\n        }))\n        .unwrap();\n\n        let collector = AggregationCollector::from_aggs(d, Default::default());\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n\n        let res: Value = serde_json::from_str(\n            &serde_json::to_string(&agg_res).expect(\"JSON serialization failed\"),\n        )\n        .expect(\"JSON parsing failed\");\n\n        assert_eq!(\n            res,\n            json!({\n                \"top_hits_req\": {\n                    \"hits\": []\n                }\n            })\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_hits_collector_single_feature() -> crate::Result<()> {\n        let docs = vec![\n            ComparableDoc::<_, _> {\n                doc: crate::DocAddress {\n                    segment_ord: 0,\n                    doc_id: 0,\n                },\n                sort_key: DocSortValuesAndFields {\n                    sorts: vec![DocValueAndOrder {\n                        value: Some(1),\n                        order: Order::Asc,\n                    }],\n                    doc_value_fields: Default::default(),\n                },\n            },\n            ComparableDoc {\n                doc: crate::DocAddress {\n                    segment_ord: 0,\n                    doc_id: 2,\n                },\n                sort_key: DocSortValuesAndFields {\n                    sorts: vec![DocValueAndOrder {\n                        value: Some(3),\n                        order: Order::Asc,\n                    }],\n                    doc_value_fields: Default::default(),\n                },\n            },\n            ComparableDoc {\n                doc: crate::DocAddress {\n                    segment_ord: 0,\n                    doc_id: 1,\n                },\n                sort_key: DocSortValuesAndFields {\n                    sorts: vec![DocValueAndOrder {\n                        value: Some(5),\n                        order: Order::Asc,\n                    }],\n                    doc_value_fields: Default::default(),\n                },\n            },\n        ];\n\n        let mut collector = collector_with_capacity(3);\n        for doc in docs.clone() {\n            collector.collect(doc.sort_key, doc.doc);\n        }\n\n        let res = collector.into_final_result();\n\n        assert_eq!(\n            res,\n            super::TopHitsMetricResult {\n                hits: vec![\n                    super::TopHitsVecEntry {\n                        sort: vec![docs[0].sort_key.sorts[0].value],\n                        doc_value_fields: Default::default(),\n                    },\n                    super::TopHitsVecEntry {\n                        sort: vec![docs[1].sort_key.sorts[0].value],\n                        doc_value_fields: Default::default(),\n                    },\n                    super::TopHitsVecEntry {\n                        sort: vec![docs[2].sort_key.sorts[0].value],\n                        doc_value_fields: Default::default(),\n                    },\n                ]\n            }\n        );\n\n        Ok(())\n    }\n\n    fn test_aggregation_top_hits(merge_segments: bool) -> crate::Result<()> {\n        let docs = vec![\n            vec![\n                r#\"{ \"date\": \"2015-01-02T00:00:00Z\", \"text\": \"bbb\", \"text2\": \"bbb\", \"mixed\": { \"dyn_arr\": [1, \"2\"] } }\"#,\n                r#\"{ \"date\": \"2017-06-15T00:00:00Z\", \"text\": \"ccc\", \"text2\": \"ddd\", \"mixed\": { \"dyn_arr\": [3, \"4\"] } }\"#,\n            ],\n            vec![\n                r#\"{ \"text\": \"aaa\", \"text2\": \"bbb\", \"date\": \"2018-01-02T00:00:00Z\", \"mixed\": { \"dyn_arr\": [\"9\", 8] } }\"#,\n                r#\"{ \"text\": \"aaa\", \"text2\": \"bbb\", \"date\": \"2016-01-02T00:00:00Z\", \"mixed\": { \"dyn_arr\": [\"7\", 6] } }\"#,\n            ],\n        ];\n\n        let index = get_test_index_from_docs(merge_segments, &docs)?;\n\n        let d: Aggregations = serde_json::from_value(json!({\n            \"top_hits_req\": {\n                \"top_hits\": {\n                    \"size\": 2,\n                    \"sort\": [\n                        { \"date\": \"desc\" }\n                    ],\n                    \"from\": 1,\n                    \"docvalue_fields\": [\n                        \"date\",\n                        \"tex*\",\n                        \"mixed.*\",\n                    ],\n                }\n            }\n        }))?;\n\n        let collector = AggregationCollector::from_aggs(d, Default::default());\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let agg_res =\n            serde_json::to_value(searcher.search(&AllQuery, &collector).unwrap()).unwrap();\n\n        let date_2017 = datetime!(2017-06-15 00:00:00 UTC);\n        let date_2016 = datetime!(2016-01-02 00:00:00 UTC);\n\n        assert_eq!(\n            agg_res[\"top_hits_req\"],\n            json!({\n                \"hits\": [\n                    {\n                        \"sort\": [common::i64_to_u64(date_2017.unix_timestamp_nanos() as i64)],\n                        \"docvalue_fields\": {\n                            \"date\": [ OwnedValue::Date(DateTime::from_utc(date_2017)) ],\n                            \"text\": [ \"ccc\" ],\n                            \"text2\": [ \"ddd\" ],\n                            \"mixed.dyn_arr\": [ 3, \"4\" ],\n                        }\n                    },\n                    {\n                        \"sort\": [common::i64_to_u64(date_2016.unix_timestamp_nanos() as i64)],\n                        \"docvalue_fields\": {\n                            \"date\": [ OwnedValue::Date(DateTime::from_utc(date_2016)) ],\n                            \"text\": [ \"aaa\" ],\n                            \"text2\": [ \"bbb\" ],\n                            \"mixed.dyn_arr\": [ 6, \"7\" ],\n                        }\n                    }\n                ]\n            }),\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_aggregation_top_hits_single_segment() -> crate::Result<()> {\n        test_aggregation_top_hits(true)\n    }\n\n    #[test]\n    fn test_aggregation_top_hits_multi_segment() -> crate::Result<()> {\n        test_aggregation_top_hits(false)\n    }\n}\n"
  },
  {
    "path": "src/aggregation/mod.rs",
    "content": "//! # Aggregations\n//!\n//! An aggregation summarizes your data as statistics on buckets or metrics.\n//!\n//! Aggregations can provide answer to questions like:\n//! - What is the average price of all sold articles?\n//! - How many errors with status code 500 do we have per day?\n//! - What is the average listing price of cars grouped by color?\n//!\n//! There are two categories: [Metrics](metric) and [Buckets](bucket).\n//!\n//! ## Prerequisite\n//! Currently aggregations work only on [fast fields](`crate::fastfield`). Fast fields\n//! of type `u64`, `f64`, `i64`, `date` and fast fields on text fields.\n//!\n//! ## Usage\n//! To use aggregations, build an aggregation request by constructing\n//! [`Aggregations`](agg_req::Aggregations).\n//! Create an [`AggregationCollector`] from this request. `AggregationCollector` implements the\n//! [`Collector`](crate::collector::Collector) trait and can be passed as collector into\n//! [`Searcher::search()`](crate::Searcher::search).\n//!\n//!\n//! ## JSON Format\n//! Aggregations request and result structures de/serialize into elasticsearch compatible JSON.\n//!\n//! Notice: Intermediate aggregation results should not be de/serialized via JSON format.\n//! Postcard is a good choice.\n//!\n//! ```verbatim\n//! let agg_req: Aggregations = serde_json::from_str(json_request_string).unwrap();\n//! let collector = AggregationCollector::from_aggs(agg_req, None);\n//! let searcher = reader.searcher();\n//! let agg_res = searcher.search(&term_query, &collector).unwrap_err();\n//! let json_response_string: String = &serde_json::to_string(&agg_res)?;\n//! ```\n//!\n//! ## Supported Aggregations\n//! - [Bucket](bucket)\n//!     - [Histogram](bucket::HistogramAggregation)\n//!     - [DateHistogram](bucket::DateHistogramAggregationReq)\n//!     - [Range](bucket::RangeAggregation)\n//!     - [Terms](bucket::TermsAggregation)\n//! - [Metric](metric)\n//!     - [Average](metric::AverageAggregation)\n//!     - [Stats](metric::StatsAggregation)\n//!     - [ExtendedStats](metric::ExtendedStatsAggregation)\n//!     - [Min](metric::MinAggregation)\n//!     - [Max](metric::MaxAggregation)\n//!     - [Sum](metric::SumAggregation)\n//!     - [Count](metric::CountAggregation)\n//!     - [Percentiles](metric::PercentilesAggregationReq)\n//!     - [Cardinality](metric::CardinalityAggregationReq)\n//!     - [TopHits](metric::TopHitsAggregationReq)\n//!\n//! # Example\n//! Compute the average metric, by building [`agg_req::Aggregations`], which is built from an\n//! `(String, agg_req::Aggregation)` iterator.\n//!\n//! Requests are compatible with the elasticsearch JSON request format.\n//!\n//! ```\n//! use tantivy::aggregation::agg_req::Aggregations;\n//!\n//! let elasticsearch_compatible_json_req = r#\"\n//! {\n//!   \"average\": {\n//!     \"avg\": { \"field\": \"score\" }\n//!   },\n//!   \"range\": {\n//!     \"range\": {\n//!       \"field\": \"score\",\n//!       \"ranges\": [\n//!         { \"to\": 3.0 },\n//!         { \"from\": 3.0, \"to\": 7.0 },\n//!         { \"from\": 7.0, \"to\": 20.0 },\n//!         { \"from\": 20.0 }\n//!       ]\n//!     },\n//!     \"aggs\": {\n//!       \"average_in_range\": { \"avg\": { \"field\": \"score\" } }\n//!     }\n//!   }\n//! }\n//! \"#;\n//! let agg_req: Aggregations =\n//!     serde_json::from_str(elasticsearch_compatible_json_req).unwrap();\n//! ```\n//! # Code Organization\n//!\n//! Check the [README](https://github.com/quickwit-oss/tantivy/tree/main/src/aggregation#readme) on github to see how the code is organized.\n//!\n//! # Nested Aggregation\n//!\n//! Buckets can contain sub-aggregations. In this example we create buckets with the range\n//! aggregation and then calculate the average on each bucket.\n//! ```\n//! use tantivy::aggregation::agg_req::*;\n//! use serde_json::json;\n//!\n//! let agg_req_1: Aggregations = serde_json::from_value(json!({\n//!     \"rangef64\": {\n//!         \"range\": {\n//!             \"field\": \"score\",\n//!             \"ranges\": [\n//!                 { \"from\": 3, \"to\": 7000 },\n//!                 { \"from\": 7000, \"to\": 20000 },\n//!                 { \"from\": 50000, \"to\": 60000 }\n//!             ]\n//!         },\n//!         \"aggs\": {\n//!             \"average_in_range\": { \"avg\": { \"field\": \"score\" } }\n//!         }\n//!     },\n//! }))\n//! .unwrap();\n//! ```\n//!\n//! # Distributed Aggregation\n//! When the data is distributed on different [`Index`](crate::Index) instances, the\n//! [`DistributedAggregationCollector`] provides functionality to merge data between independent\n//! search calls by returning\n//! [`IntermediateAggregationResults`](intermediate_agg_result::IntermediateAggregationResults).\n//! `IntermediateAggregationResults` provides the\n//! [`merge_fruits`](intermediate_agg_result::IntermediateAggregationResults::merge_fruits) method\n//! to merge multiple results. The merged result can then be converted into\n//! [`AggregationResults`](agg_result::AggregationResults) via the\n//! [`into_final_result`](intermediate_agg_result::IntermediateAggregationResults::into_final_result) method.\n\nmod accessor_helpers;\nmod agg_data;\nmod agg_limits;\npub mod agg_req;\npub mod agg_result;\npub mod bucket;\npub(crate) mod cached_sub_aggs;\nmod collector;\nmod date;\nmod error;\npub mod intermediate_agg_result;\npub mod metric;\n\nmod segment_agg_result;\nuse std::fmt::Display;\n\n#[cfg(test)]\nmod agg_tests;\n\nuse core::fmt;\n\npub use agg_limits::AggregationLimitsGuard;\npub use collector::{\n    AggregationCollector, AggregationSegmentCollector, DistributedAggregationCollector,\n    DEFAULT_BUCKET_LIMIT,\n};\nuse columnar::{ColumnType, MonotonicallyMappableToU64};\npub(crate) use date::format_date;\npub use error::AggregationError;\nuse itertools::Itertools;\nuse serde::de::{self, Visitor};\nuse serde::{Deserialize, Deserializer, Serialize};\n\nuse crate::tokenizer::TokenizerManager;\n\n/// A bucket id is a dense identifier for a bucket within an aggregation.\n/// It is used to index into a Vec that hold per-bucket data.\n///\n/// For example, in a terms aggregation, each unique term will be assigned a incremental BucketId.\n/// This BucketId will be forwarded to sub-aggregations to identify the parent bucket.\n///\n/// This allows to have a single AggregationCollector instance per aggregation,\n/// that can handle multiple buckets efficiently.\n///\n/// The API to call sub-aggregations is therefore a &[(BucketId, &[DocId])].\n/// For that we'll need a buffer. One Vec per bucket aggregation is needed.\npub type BucketId = u32;\n\n/// Context parameters for aggregation execution\n///\n/// This struct holds shared resources needed during aggregation execution:\n/// - `limits`: Memory and bucket limits for the aggregation\n/// - `tokenizers`: TokenizerManager for parsing query strings in filter aggregations\n#[derive(Clone, Default)]\npub struct AggContextParams {\n    /// Aggregation limits (memory and bucket count)\n    pub limits: AggregationLimitsGuard,\n    /// Tokenizer manager for query string parsing\n    pub tokenizers: TokenizerManager,\n}\n\nimpl AggContextParams {\n    /// Create new aggregation context parameters\n    pub fn new(limits: AggregationLimitsGuard, tokenizers: TokenizerManager) -> Self {\n        Self { limits, tokenizers }\n    }\n}\n\nfn parse_str_into_f64<E: de::Error>(value: &str) -> Result<f64, E> {\n    let parsed = value\n        .parse::<f64>()\n        .map_err(|_err| de::Error::custom(format!(\"Failed to parse f64 from string: {value:?}\")))?;\n\n    // Check if the parsed value is NaN or infinity\n    if parsed.is_nan() || parsed.is_infinite() {\n        Err(de::Error::custom(format!(\n            \"Value is not a valid f64 (NaN or Infinity): {value:?}\"\n        )))\n    } else {\n        Ok(parsed)\n    }\n}\n\n/// deserialize Option<f64> from string or float\npub(crate) fn deserialize_option_f64<'de, D>(deserializer: D) -> Result<Option<f64>, D::Error>\nwhere D: Deserializer<'de> {\n    struct StringOrFloatVisitor;\n\n    impl Visitor<'_> for StringOrFloatVisitor {\n        type Value = Option<f64>;\n\n        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n            formatter.write_str(\"a string or a float\")\n        }\n\n        fn visit_str<E>(self, value: &str) -> Result<Self::Value, E>\n        where E: de::Error {\n            parse_str_into_f64(value).map(Some)\n        }\n\n        fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(Some(value))\n        }\n\n        fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(Some(value as f64))\n        }\n\n        fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(Some(value as f64))\n        }\n\n        fn visit_none<E>(self) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(None)\n        }\n\n        fn visit_unit<E>(self) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(None)\n        }\n    }\n\n    deserializer.deserialize_any(StringOrFloatVisitor)\n}\n\n/// deserialize f64 from string or float\npub(crate) fn deserialize_f64<'de, D>(deserializer: D) -> Result<f64, D::Error>\nwhere D: Deserializer<'de> {\n    struct StringOrFloatVisitor;\n\n    impl Visitor<'_> for StringOrFloatVisitor {\n        type Value = f64;\n\n        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {\n            formatter.write_str(\"a string or a float\")\n        }\n\n        fn visit_str<E>(self, value: &str) -> Result<Self::Value, E>\n        where E: de::Error {\n            parse_str_into_f64(value)\n        }\n\n        fn visit_f64<E>(self, value: f64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(value)\n        }\n\n        fn visit_i64<E>(self, value: i64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(value as f64)\n        }\n\n        fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>\n        where E: de::Error {\n            Ok(value as f64)\n        }\n    }\n\n    deserializer.deserialize_any(StringOrFloatVisitor)\n}\n\n/// The serialized key is used in a `HashMap`.\npub type SerializedKey = String;\n\n#[derive(Clone, Debug, Serialize, Deserialize, PartialOrd)]\n/// The key to identify a bucket.\n///\n/// The order is important, with serde untagged, that we try to deserialize into i64 first.\n#[serde(untagged)]\npub enum Key {\n    /// String key\n    Str(String),\n    /// `i64` key\n    I64(i64),\n    /// `u64` key\n    U64(u64),\n    /// `f64` key\n    F64(f64),\n}\nimpl Eq for Key {}\nimpl std::hash::Hash for Key {\n    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {\n        core::mem::discriminant(self).hash(state);\n        match self {\n            Key::Str(text) => text.hash(state),\n            Key::F64(val) => val.to_bits().hash(state),\n            Key::U64(val) => val.hash(state),\n            Key::I64(val) => val.hash(state),\n        }\n    }\n}\n\nimpl PartialEq for Key {\n    fn eq(&self, other: &Self) -> bool {\n        match (self, other) {\n            (Self::Str(l), Self::Str(r)) => l == r,\n            (Self::F64(l), Self::F64(r)) => l.to_bits() == r.to_bits(),\n            (Self::I64(l), Self::I64(r)) => l == r,\n            (Self::U64(l), Self::U64(r)) => l == r,\n            // we list all variant of left operand to make sure this gets updated when we add\n            // variants to the enum\n            (Self::Str(_) | Self::F64(_) | Self::I64(_) | Self::U64(_), _) => false,\n        }\n    }\n}\n\nimpl Display for Key {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            Key::Str(val) => f.write_str(val),\n            Key::F64(val) => f.write_str(&val.to_string()),\n            Key::U64(val) => f.write_str(&val.to_string()),\n            Key::I64(val) => f.write_str(&val.to_string()),\n        }\n    }\n}\n\npub(crate) fn convert_to_f64<const COLUMN_TYPE_ID: u8>(val: u64) -> f64 {\n    if COLUMN_TYPE_ID == ColumnType::U64 as u8 {\n        val as f64\n    } else if COLUMN_TYPE_ID == ColumnType::I64 as u8\n        || COLUMN_TYPE_ID == ColumnType::DateTime as u8\n    {\n        i64::from_u64(val) as f64\n    } else if COLUMN_TYPE_ID == ColumnType::F64 as u8 {\n        f64::from_u64(val)\n    } else if COLUMN_TYPE_ID == ColumnType::Bool as u8 {\n        val as f64\n    } else {\n        panic!(\n            \"ColumnType ID {} cannot be converted to f64 metric\",\n            COLUMN_TYPE_ID\n        )\n    }\n}\n\n/// Inverse of `to_fastfield_u64`. Used to convert to `f64` for metrics.\n///\n/// # Panics\n/// Only `u64`, `f64`, `date`, and `i64` are supported.\npub(crate) fn f64_from_fastfield_u64(val: u64, field_type: ColumnType) -> f64 {\n    match field_type {\n        ColumnType::U64 => convert_to_f64::<{ ColumnType::U64 as u8 }>(val),\n        ColumnType::I64 => convert_to_f64::<{ ColumnType::I64 as u8 }>(val),\n        ColumnType::F64 => convert_to_f64::<{ ColumnType::F64 as u8 }>(val),\n        ColumnType::Bool => convert_to_f64::<{ ColumnType::Bool as u8 }>(val),\n        ColumnType::DateTime => convert_to_f64::<{ ColumnType::DateTime as u8 }>(val),\n        _ => panic!(\"unexpected type {field_type:?}. This should not happen\"),\n    }\n}\n\n/// Converts the `f64` value to fast field value space, which is always u64.\n///\n/// If the fast field has `u64`, values are stored unchanged as `u64` in the fast field.\n///\n/// If the fast field has `f64` values are converted and stored to `u64` using a\n/// monotonic mapping.\n/// A `f64` value of e.g. `2.0` needs to be converted using the same monotonic\n/// conversion function, so that the value matches the `u64` value stored in the fast\n/// field.\npub(crate) fn f64_to_fastfield_u64(val: f64, field_type: &ColumnType) -> Option<u64> {\n    match field_type {\n        ColumnType::U64 => Some(val as u64),\n        ColumnType::I64 | ColumnType::DateTime => Some((val as i64).to_u64()),\n        ColumnType::F64 => Some(val.to_u64()),\n        ColumnType::Bool => Some(val as u64),\n        _ => None,\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::Ipv6Addr;\n\n    use columnar::DateTime;\n    use serde_json::Value;\n    use time::OffsetDateTime;\n\n    use super::agg_req::Aggregations;\n    use super::*;\n    use crate::indexer::NoMergePolicy;\n    use crate::query::{AllQuery, TermQuery};\n    use crate::schema::{IndexRecordOption, Schema, TextFieldIndexing, FAST, STRING};\n    use crate::{Index, IndexWriter, Term};\n\n    pub fn get_test_index_with_num_docs(\n        merge_segments: bool,\n        num_docs: usize,\n    ) -> crate::Result<Index> {\n        get_test_index_from_values(\n            merge_segments,\n            &(0..num_docs).map(|el| el as f64).collect::<Vec<f64>>(),\n        )\n    }\n\n    pub fn exec_request(agg_req: Aggregations, index: &Index) -> crate::Result<Value> {\n        exec_request_with_query(agg_req, index, None)\n    }\n    pub fn exec_request_with_query(\n        agg_req: Aggregations,\n        index: &Index,\n        query: Option<(&str, &str)>,\n    ) -> crate::Result<Value> {\n        exec_request_with_query_and_memory_limit(agg_req, index, query, Default::default())\n    }\n\n    pub fn exec_request_with_query_and_memory_limit(\n        agg_req: Aggregations,\n        index: &Index,\n        query: Option<(&str, &str)>,\n        limits: AggregationLimitsGuard,\n    ) -> crate::Result<Value> {\n        let collector = AggregationCollector::from_aggs(\n            agg_req,\n            AggContextParams::new(limits, index.tokenizers().clone()),\n        );\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let agg_res = if let Some((field, term)) = query {\n            let text_field = reader.searcher().schema().get_field(field).unwrap();\n\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, term),\n                IndexRecordOption::Basic,\n            );\n\n            searcher.search(&term_query, &collector)?\n        } else {\n            searcher.search(&AllQuery, &collector)?\n        };\n\n        // Test serialization/deserialization roundtrip\n        let res: Value = serde_json::from_str(&serde_json::to_string(&agg_res)?)?;\n        Ok(res)\n    }\n\n    pub fn get_test_index_from_values(\n        merge_segments: bool,\n        values: &[f64],\n    ) -> crate::Result<Index> {\n        // Every value gets its own segment\n        let mut segment_and_values = vec![];\n        for value in values {\n            segment_and_values.push(vec![(*value, value.to_string())]);\n        }\n        get_test_index_from_values_and_terms(merge_segments, &segment_and_values)\n    }\n\n    pub fn get_test_index_from_terms(\n        merge_segments: bool,\n        values: &[Vec<&str>],\n    ) -> crate::Result<Index> {\n        // Every value gets its own segment\n        let segment_and_values = values\n            .iter()\n            .map(|terms| {\n                terms\n                    .iter()\n                    .enumerate()\n                    .map(|(i, term)| (i as f64, term.to_string()))\n                    .collect()\n            })\n            .collect::<Vec<_>>();\n        get_test_index_from_values_and_terms(merge_segments, &segment_and_values)\n    }\n\n    pub fn get_test_index_from_values_and_terms(\n        merge_segments: bool,\n        segment_and_values: &[Vec<(f64, String)>],\n    ) -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text_fieldtype = crate::schema::TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_index_option(IndexRecordOption::Basic)\n                    .set_fieldnorms(false),\n            )\n            .set_fast(None)\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text\", text_fieldtype.clone());\n        let text_field_id = schema_builder.add_text_field(\"text_id\", text_fieldtype);\n        let string_field_id = schema_builder.add_text_field(\"string_id\", STRING | FAST);\n        let score_fieldtype = crate::schema::NumericOptions::default().set_fast();\n        let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype.clone());\n        let score_field_f64 = schema_builder.add_f64_field(\"score_f64\", score_fieldtype.clone());\n        let score_field_i64 = schema_builder.add_i64_field(\"score_i64\", score_fieldtype);\n        let fraction_field = schema_builder.add_f64_field(\n            \"fraction_f64\",\n            crate::schema::NumericOptions::default().set_fast(),\n        );\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            // let mut index_writer = index.writer_for_tests()?;\n            let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            for values in segment_and_values {\n                for (i, term) in values {\n                    let i = *i;\n                    // writing the segment\n                    index_writer.add_document(doc!(\n                        text_field => \"cool\",\n                        text_field_id => term.to_string(),\n                        string_field_id => term.to_string(),\n                        score_field => i as u64,\n                        score_field_f64 => i,\n                        score_field_i64 => i as i64,\n                        fraction_field => i/100.0,\n                    ))?;\n                }\n                index_writer.commit()?;\n            }\n        }\n        if merge_segments {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            if segment_ids.len() > 1 {\n                let mut index_writer: IndexWriter = index.writer_for_tests()?;\n                index_writer.merge(&segment_ids).wait()?;\n                index_writer.wait_merging_threads()?;\n            }\n        }\n\n        Ok(index)\n    }\n\n    pub fn get_test_index_2_segments(merge_segments: bool) -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text_fieldtype = crate::schema::TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default().set_index_option(IndexRecordOption::WithFreqs),\n            )\n            .set_fast(Some(\"raw\"))\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text\", text_fieldtype);\n        let date_field = schema_builder.add_date_field(\"date\", FAST);\n        schema_builder.add_text_field(\"dummy_text\", STRING);\n        let score_fieldtype = crate::schema::NumericOptions::default().set_fast();\n        let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype.clone());\n        let score_field_f64 = schema_builder.add_f64_field(\"score_f64\", score_fieldtype.clone());\n        let ip_addr_field = schema_builder.add_ip_addr_field(\"ip_addr\", FAST);\n\n        let multivalue = crate::schema::NumericOptions::default().set_fast();\n        let scores_field_i64 = schema_builder.add_i64_field(\"scores_i64\", multivalue);\n\n        let score_field_i64 = schema_builder.add_i64_field(\"score_i64\", score_fieldtype);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800).unwrap()),\n                score_field => 1u64,\n                ip_addr_field => Ipv6Addr::from(1u128),\n                score_field_f64 => 1f64,\n                score_field_i64 => 1i64,\n                scores_field_i64 => 1i64,\n                scores_field_i64 => 2i64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400).unwrap()),\n                score_field => 3u64,\n                score_field_f64 => 3f64,\n                score_field_i64 => 3i64,\n                scores_field_i64 => 5i64,\n                scores_field_i64 => 5i64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400).unwrap()),\n                score_field => 5u64,\n                score_field_f64 => 5f64,\n                score_field_i64 => 5i64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"nohit\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400).unwrap()),\n                score_field => 6u64,\n                score_field_f64 => 6f64,\n                score_field_i64 => 6i64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400).unwrap()),\n                score_field => 7u64,\n                score_field_f64 => 7f64,\n                score_field_i64 => 7i64,\n            ))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400).unwrap()),\n                score_field => 11u64,\n                score_field_f64 => 11f64,\n                score_field_i64 => 11i64,\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400 + 86400).unwrap()),\n                score_field => 14u64,\n                score_field_f64 => 14f64,\n                score_field_i64 => 14i64,\n            ))?;\n\n            index_writer.add_document(doc!(\n                text_field => \"cool\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400 + 86400).unwrap()),\n                score_field => 44u64,\n                score_field_f64 => 44.5f64,\n                score_field_i64 => 44i64,\n            ))?;\n\n            index_writer.commit()?;\n\n            // no hits segment\n            index_writer.add_document(doc!(\n                text_field => \"nohit\",\n                date_field => DateTime::from_utc(OffsetDateTime::from_unix_timestamp(1_546_300_800 + 86400 + 86400).unwrap()),\n                score_field => 44u64,\n                score_field_f64 => 44.5f64,\n                score_field_i64 => 44i64,\n            ))?;\n\n            index_writer.commit()?;\n        }\n        if merge_segments {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n\n        Ok(index)\n    }\n}\n"
  },
  {
    "path": "src/aggregation/segment_agg_result.rs",
    "content": "//! Contains aggregation trees which is used during collection in a segment.\n//! This tree contains datastructrues optimized for fast collection.\n//! The tree can be converted to an intermediate tree, which contains datastructrues optimized for\n//! merging.\n\nuse std::fmt::Debug;\n\npub(crate) use super::agg_limits::AggregationLimitsGuard;\nuse super::intermediate_agg_result::IntermediateAggregationResults;\nuse crate::aggregation::agg_data::AggregationsSegmentCtx;\nuse crate::aggregation::BucketId;\n\n/// Monotonically increasing provider of BucketIds.\n#[derive(Debug, Clone, Default)]\npub struct BucketIdProvider(u32);\nimpl BucketIdProvider {\n    /// Get the next BucketId.\n    pub fn next_bucket_id(&mut self) -> BucketId {\n        let bucket_id = self.0;\n        self.0 += 1;\n        bucket_id\n    }\n}\n\n/// A SegmentAggregationCollector is used to collect aggregation results.\npub trait SegmentAggregationCollector: Debug {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()>;\n\n    /// Note: The caller needs to call `prepare_max_bucket` before calling `collect`.\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()>;\n\n    /// Collect docs for multiple buckets in one call.\n    /// Minimizes dynamic dispatch overhead when collecting many buckets.\n    ///\n    /// Note: The caller needs to call `prepare_max_bucket` before calling `collect`.\n    fn collect_multiple(\n        &mut self,\n        bucket_ids: &[BucketId],\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        debug_assert_eq!(bucket_ids.len(), docs.len());\n        let mut start = 0;\n        while start < bucket_ids.len() {\n            let bucket_id = bucket_ids[start];\n            let mut end = start + 1;\n            while end < bucket_ids.len() && bucket_ids[end] == bucket_id {\n                end += 1;\n            }\n            self.collect(bucket_id, &docs[start..end], agg_data)?;\n            start = end;\n        }\n        Ok(())\n    }\n\n    /// Prepare the collector for collecting up to BucketId `max_bucket`.\n    /// This is useful so we can split allocation ahead of time of collecting.\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()>;\n\n    /// Finalize method. Some Aggregator collect blocks of docs before calling `collect_block`.\n    /// This method ensures those staged docs will be collected.\n    fn flush(&mut self, _agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        Ok(())\n    }\n}\n\n#[derive(Default)]\n/// The GenericSegmentAggregationResultsCollector is the generic version of the collector, which\n/// can handle arbitrary complexity of  sub-aggregations. Ideally we never have to pick this one\n/// and can provide specialized versions instead, that remove some of its overhead.\npub(crate) struct GenericSegmentAggregationResultsCollector {\n    pub(crate) aggs: Vec<Box<dyn SegmentAggregationCollector>>,\n}\n\nimpl Debug for GenericSegmentAggregationResultsCollector {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"SegmentAggregationResultsCollector\")\n            .field(\"aggs\", &self.aggs)\n            .finish()\n    }\n}\n\nimpl SegmentAggregationCollector for GenericSegmentAggregationResultsCollector {\n    fn add_intermediate_aggregation_result(\n        &mut self,\n        agg_data: &AggregationsSegmentCtx,\n        results: &mut IntermediateAggregationResults,\n        parent_bucket_id: BucketId,\n    ) -> crate::Result<()> {\n        for agg in &mut self.aggs {\n            agg.add_intermediate_aggregation_result(agg_data, results, parent_bucket_id)?;\n        }\n\n        Ok(())\n    }\n\n    fn collect(\n        &mut self,\n        parent_bucket_id: BucketId,\n        docs: &[crate::DocId],\n        agg_data: &mut AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        for collector in &mut self.aggs {\n            collector.collect(parent_bucket_id, docs, agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn flush(&mut self, agg_data: &mut AggregationsSegmentCtx) -> crate::Result<()> {\n        for collector in &mut self.aggs {\n            collector.flush(agg_data)?;\n        }\n        Ok(())\n    }\n\n    fn prepare_max_bucket(\n        &mut self,\n        max_bucket: BucketId,\n        agg_data: &AggregationsSegmentCtx,\n    ) -> crate::Result<()> {\n        for collector in &mut self.aggs {\n            collector.prepare_max_bucket(max_bucket, agg_data)?;\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/collector/count_collector.rs",
    "content": "use super::Collector;\nuse crate::collector::SegmentCollector;\nuse crate::{DocId, Score, SegmentOrdinal, SegmentReader};\n\n/// `CountCollector` collector only counts how many\n/// documents match the query.\n///\n/// ```rust\n/// use tantivy::collector::Count;\n/// use tantivy::query::QueryParser;\n/// use tantivy::schema::{Schema, TEXT};\n/// use tantivy::{doc, Index};\n///\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n///\n/// let mut index_writer = index.writer(15_000_000).unwrap();\n/// index_writer.add_document(doc!(title => \"The Name of the Wind\")).unwrap();\n/// index_writer.add_document(doc!(title => \"The Diary of Muadib\")).unwrap();\n/// index_writer.add_document(doc!(title => \"A Dairy Cow\")).unwrap();\n/// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\")).unwrap();\n/// assert!(index_writer.commit().is_ok());\n///\n/// let reader = index.reader().unwrap();\n/// let searcher = reader.searcher();\n///\n/// // Here comes the important part\n/// let query_parser = QueryParser::for_index(&index, vec![title]);\n/// let query = query_parser.parse_query(\"diary\").unwrap();\n/// let count = searcher.search(&query, &Count).unwrap();\n///\n/// assert_eq!(count, 2);\n/// ```\npub struct Count;\n\nimpl Collector for Count {\n    type Fruit = usize;\n\n    type Child = SegmentCountCollector;\n\n    fn for_segment(\n        &self,\n        _: SegmentOrdinal,\n        _: &SegmentReader,\n    ) -> crate::Result<SegmentCountCollector> {\n        Ok(SegmentCountCollector::default())\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(&self, segment_counts: Vec<usize>) -> crate::Result<usize> {\n        Ok(segment_counts.into_iter().sum())\n    }\n}\n\n#[derive(Default)]\npub struct SegmentCountCollector {\n    count: usize,\n}\n\nimpl SegmentCollector for SegmentCountCollector {\n    type Fruit = usize;\n\n    fn collect(&mut self, _: DocId, _: Score) {\n        self.count += 1;\n    }\n\n    fn harvest(self) -> usize {\n        self.count\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::{Count, SegmentCountCollector};\n    use crate::collector::{Collector, SegmentCollector};\n\n    #[test]\n    fn test_count_collect_does_not_requires_scoring() {\n        assert!(!Count.requires_scoring());\n    }\n\n    #[test]\n    fn test_segment_count_collector() {\n        {\n            let count_collector = SegmentCountCollector::default();\n            assert_eq!(count_collector.harvest(), 0);\n        }\n        {\n            let mut count_collector = SegmentCountCollector::default();\n            count_collector.collect(0u32, 1.0);\n            assert_eq!(count_collector.harvest(), 1);\n        }\n        {\n            let mut count_collector = SegmentCountCollector::default();\n            count_collector.collect(0u32, 1.0);\n            assert_eq!(count_collector.harvest(), 1);\n        }\n        {\n            let mut count_collector = SegmentCountCollector::default();\n            count_collector.collect(0u32, 1.0);\n            count_collector.collect(1u32, 1.0);\n            assert_eq!(count_collector.harvest(), 2);\n        }\n    }\n}\n"
  },
  {
    "path": "src/collector/docset_collector.rs",
    "content": "use std::collections::HashSet;\n\nuse super::{Collector, SegmentCollector};\nuse crate::{DocAddress, DocId, Score};\n\n/// Collectors that returns the set of DocAddress that matches the query.\n///\n/// This collector is mostly useful for tests.\npub struct DocSetCollector;\n\nimpl Collector for DocSetCollector {\n    type Fruit = HashSet<DocAddress>;\n    type Child = DocSetChildCollector;\n\n    fn for_segment(\n        &self,\n        segment_local_id: crate::SegmentOrdinal,\n        _segment: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        Ok(DocSetChildCollector {\n            segment_local_id,\n            docs: HashSet::new(),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<(u32, HashSet<DocId>)>,\n    ) -> crate::Result<Self::Fruit> {\n        let len: usize = segment_fruits.iter().map(|(_, docset)| docset.len()).sum();\n        let mut result = HashSet::with_capacity(len);\n        for (segment_local_id, docs) in segment_fruits {\n            for doc in docs {\n                result.insert(DocAddress::new(segment_local_id, doc));\n            }\n        }\n        Ok(result)\n    }\n}\n\npub struct DocSetChildCollector {\n    segment_local_id: u32,\n    docs: HashSet<DocId>,\n}\n\nimpl SegmentCollector for DocSetChildCollector {\n    type Fruit = (u32, HashSet<DocId>);\n\n    fn collect(&mut self, doc: crate::DocId, _score: Score) {\n        self.docs.insert(doc);\n    }\n\n    fn harvest(self) -> (u32, HashSet<DocId>) {\n        (self.segment_local_id, self.docs)\n    }\n}\n"
  },
  {
    "path": "src/collector/facet_collector.rs",
    "content": "use std::cmp::Ordering;\nuse std::collections::{btree_map, BTreeMap, BTreeSet, BinaryHeap};\nuse std::io;\nuse std::ops::Bound;\n\nuse crate::collector::{Collector, SegmentCollector};\nuse crate::fastfield::FacetReader;\nuse crate::schema::Facet;\nuse crate::{DocId, Score, SegmentOrdinal, SegmentReader};\n\nstruct Hit<'a> {\n    count: u64,\n    facet: &'a Facet,\n}\n\nimpl Eq for Hit<'_> {}\n\nimpl<'a> PartialEq<Hit<'a>> for Hit<'a> {\n    fn eq(&self, other: &Hit<'_>) -> bool {\n        self.count == other.count\n    }\n}\n\nimpl<'a> PartialOrd<Hit<'a>> for Hit<'a> {\n    fn partial_cmp(&self, other: &Hit<'_>) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for Hit<'_> {\n    fn cmp(&self, other: &Self) -> Ordering {\n        other\n            .count\n            .cmp(&self.count)\n            .then(self.facet.cmp(other.facet))\n    }\n}\n\nfn facet_depth(facet_bytes: &[u8]) -> usize {\n    if facet_bytes.is_empty() {\n        0\n    } else {\n        facet_bytes.iter().cloned().filter(|b| *b == 0u8).count() + 1\n    }\n}\n\n/// Collector for faceting\n///\n/// The collector collects all facets. You need to configure it\n/// beforehand with the facet you want to extract.\n///\n/// This is done by calling `.add_facet(...)` with the root of the\n/// facet you want to extract as argument.\n///\n/// Facet counts will only be computed for the facet that are direct children\n/// of such a root facet.\n///\n/// For instance, if your index represents books, your hierarchy of facets\n/// may contain `category`, `language`.\n///\n/// The category facet may include `subcategories`. For instance, a book\n/// could belong to `/category/fiction/fantasy`.\n///\n/// If you request the facet counts for `/category`, the result will be\n/// the breakdown of counts for the direct children of `/category`\n/// (e.g. `/category/fiction`, `/category/biography`, `/category/personal_development`).\n///\n/// Once collection is finished, you can harvest its results in the form\n/// of a [`FacetCounts`] object, and extract your facet counts from it.\n///\n/// This implementation assumes you are working with a number of facets that\n/// is many hundreds of times smaller than your number of documents.\n///\n///\n/// ```rust\n/// use tantivy::collector::FacetCollector;\n/// use tantivy::query::AllQuery;\n/// use tantivy::schema::{Facet, Schema, FacetOptions, TEXT};\n/// use tantivy::{doc, Index};\n///\n/// fn example() -> tantivy::Result<()> {\n///     let mut schema_builder = Schema::builder();\n///\n///     // Facet have their own specific type.\n///     // It is not a bad practise to put all of your\n///     // facet information in the same field.\n///     let facet = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n///     let title = schema_builder.add_text_field(\"title\", TEXT);\n///     let schema = schema_builder.build();\n///     let index = Index::create_in_ram(schema);\n///     {\n///         let mut index_writer = index.writer(15_000_000)?;\n///         // a document can be associated with any number of facets\n///         index_writer.add_document(doc!(\n///             title => \"The Name of the Wind\",\n///             facet => Facet::from(\"/lang/en\"),\n///             facet => Facet::from(\"/category/fiction/fantasy\")\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"Dune\",\n///             facet => Facet::from(\"/lang/en\"),\n///             facet => Facet::from(\"/category/fiction/sci-fi\")\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"La Vénus d'Ille\",\n///             facet => Facet::from(\"/lang/fr\"),\n///             facet => Facet::from(\"/category/fiction/fantasy\"),\n///             facet => Facet::from(\"/category/fiction/horror\")\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"The Diary of a Young Girl\",\n///             facet => Facet::from(\"/lang/en\"),\n///             facet => Facet::from(\"/category/biography\")\n///         ))?;\n///         index_writer.commit()?;\n///     }\n///     let reader = index.reader()?;\n///     let searcher = reader.searcher();\n///\n///     {\n///         let mut facet_collector = FacetCollector::for_field(\"facet\");\n///         facet_collector.add_facet(\"/lang\");\n///         facet_collector.add_facet(\"/category\");\n///         let facet_counts = searcher.search(&AllQuery, &facet_collector)?;\n///\n///         // This lists all of the facet counts\n///         let facets: Vec<(&Facet, u64)> = facet_counts\n///             .get(\"/category\")\n///             .collect();\n///         assert_eq!(facets, vec![\n///             (&Facet::from(\"/category/biography\"), 1),\n///             (&Facet::from(\"/category/fiction\"), 3)\n///         ]);\n///     }\n///\n///     {\n///         let mut facet_collector = FacetCollector::for_field(\"facet\");\n///         facet_collector.add_facet(\"/category/fiction\");\n///         let facet_counts = searcher.search(&AllQuery, &facet_collector)?;\n///\n///         // This lists all of the facet counts\n///         let facets: Vec<(&Facet, u64)> = facet_counts\n///             .get(\"/category/fiction\")\n///             .collect();\n///         assert_eq!(facets, vec![\n///             (&Facet::from(\"/category/fiction/fantasy\"), 2),\n///             (&Facet::from(\"/category/fiction/horror\"), 1),\n///             (&Facet::from(\"/category/fiction/sci-fi\"), 1)\n///         ]);\n///     }\n///\n///     {\n///         let mut facet_collector = FacetCollector::for_field(\"facet\");\n///         facet_collector.add_facet(\"/category/fiction\");\n///         let facet_counts = searcher.search(&AllQuery, &facet_collector)?;\n///\n///         // This lists all of the facet counts\n///         let facets: Vec<(&Facet, u64)> = facet_counts.top_k(\"/category/fiction\", 1);\n///         assert_eq!(facets, vec![\n///             (&Facet::from(\"/category/fiction/fantasy\"), 2)\n///         ]);\n///     }\n///\n///     {\n///         let mut facet_collector = FacetCollector::for_field(\"facet\");\n///         facet_collector.add_facet(\"/\");\n///         let facet_counts = searcher.search(&AllQuery, &facet_collector)?;\n///\n///         // This lists all of the facet counts\n///         let facets: Vec<(&Facet, u64)> = facet_counts\n///             .get(\"/\")\n///             .collect();\n///         assert_eq!(facets, vec![\n///             (&Facet::from(\"/category\"), 4),\n///             (&Facet::from(\"/lang\"), 4)\n///         ]);\n///     }\n///\n///     Ok(())\n/// }\n/// # assert!(example().is_ok());\n/// ```\npub struct FacetCollector {\n    field_name: String,\n    facets: BTreeSet<Facet>,\n}\n\npub struct FacetSegmentCollector {\n    reader: FacetReader,\n    // collapse facet_id -> count\n    counts: Vec<u64>,\n    // facet_ord -> compressed collapse facet_id\n    compressed_collapse_mapping: Vec<usize>,\n    // compressed collapse facet_id -> facet_ord\n    unique_facet_ords: Vec<(u64, usize)>,\n}\n\nimpl FacetCollector {\n    /// Create a facet collector to collect the facets\n    /// from a specific facet `Field`.\n    ///\n    /// This function does not check whether the field\n    /// is of the proper type.\n    pub fn for_field(field_name: impl ToString) -> FacetCollector {\n        FacetCollector {\n            field_name: field_name.to_string(),\n            facets: BTreeSet::default(),\n        }\n    }\n\n    /// Adds a facet that we want to record counts\n    ///\n    /// Adding facet `Facet::from(\"/country\")` for instance,\n    /// will record the counts of all of the direct children of the facet country\n    /// (e.g. `/country/FR`, `/country/UK`).\n    ///\n    /// Adding two facets within which one is the prefix of the other is forbidden.\n    /// If you need the correct number of unique documents for two such facets,\n    /// just add them in a separate `FacetCollector`.\n    pub fn add_facet<T>(&mut self, facet_from: T)\n    where Facet: From<T> {\n        let facet = Facet::from(facet_from);\n        for old_facet in &self.facets {\n            assert!(\n                !old_facet.is_prefix_of(&facet),\n                \"Tried to add a facet which is a descendant of an already added facet.\"\n            );\n            assert!(\n                !facet.is_prefix_of(old_facet),\n                \"Tried to add a facet which is an ancestor of an already added facet.\"\n            );\n        }\n        self.facets.insert(facet);\n    }\n}\n\nfn compress_mapping(mapping: &[(u64, usize)]) -> (Vec<usize>, Vec<(u64, usize)>) {\n    // facet_ord -> collapse facet_id\n    let mut compressed_collapse_mapping: Vec<usize> = Vec::with_capacity(mapping.len());\n    // collapse facet_id -> facet_ord\n    let mut unique_facet_ords: Vec<(u64, usize)> = Vec::new();\n    if mapping.is_empty() {\n        return (Vec::new(), Vec::new());\n    }\n    compressed_collapse_mapping.push(0);\n    unique_facet_ords.push(mapping[0]);\n    let mut last_facet_ord = mapping[0];\n    let mut last_facet_id = 0;\n    for &facet_ord in &mapping[1..] {\n        if facet_ord != last_facet_ord {\n            last_facet_id += 1;\n            last_facet_ord = facet_ord;\n            unique_facet_ords.push(facet_ord);\n        }\n        compressed_collapse_mapping.push(last_facet_id);\n    }\n    (compressed_collapse_mapping, unique_facet_ords)\n}\n\nimpl Collector for FacetCollector {\n    type Fruit = FacetCounts;\n\n    type Child = FacetSegmentCollector;\n\n    fn for_segment(\n        &self,\n        _: SegmentOrdinal,\n        reader: &SegmentReader,\n    ) -> crate::Result<FacetSegmentCollector> {\n        let facet_reader = reader.facet_reader(&self.field_name)?;\n        let facet_dict = facet_reader.facet_dict();\n        let collapse_mapping: Vec<(u64, usize)> =\n            compute_collapse_mapping(facet_dict, &self.facets)?;\n        let (compressed_collapse_mapping, unique_facet_ords) = compress_mapping(&collapse_mapping);\n        let counts = vec![0u64; unique_facet_ords.len()];\n        Ok(FacetSegmentCollector {\n            reader: facet_reader,\n            compressed_collapse_mapping,\n            counts,\n            unique_facet_ords,\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(&self, segments_facet_counts: Vec<FacetCounts>) -> crate::Result<FacetCounts> {\n        let mut facet_counts: BTreeMap<Facet, u64> = BTreeMap::new();\n        for segment_facet_counts in segments_facet_counts {\n            for (facet, count) in segment_facet_counts.facet_counts {\n                *(facet_counts.entry(facet).or_insert(0)) += count;\n            }\n        }\n        Ok(FacetCounts { facet_counts })\n    }\n}\n\nfn is_child_facet(parent_facet: &[u8], possible_child_facet: &[u8]) -> bool {\n    if !possible_child_facet.starts_with(parent_facet) {\n        return false;\n    }\n    if parent_facet.is_empty() {\n        return true;\n    }\n    possible_child_facet.get(parent_facet.len()).copied() == Some(0u8)\n}\n\nfn compute_collapse_mapping_one(\n    facet_terms: &mut columnar::Streamer,\n    facet_bytes: &[u8],\n    collapsed: &mut [(u64, usize)],\n) -> io::Result<bool> {\n    let mut facet_child: Vec<u8> = Vec::new();\n    let mut term_ord = 0;\n    let offset = facet_bytes.len() + 1;\n    let depth = facet_depth(facet_bytes);\n    loop {\n        match facet_terms.key().cmp(facet_bytes) {\n            Ordering::Less | Ordering::Equal => {}\n            Ordering::Greater => {\n                if !is_child_facet(facet_bytes, facet_terms.key()) {\n                    return Ok(true);\n                }\n                let suffix = &facet_terms.key()[offset..];\n                if facet_child.is_empty() || !is_child_facet(&facet_child, suffix) {\n                    facet_child.clear();\n                    term_ord = facet_terms.term_ord();\n                    let end = suffix\n                        .iter()\n                        .position(|b| *b == 0u8)\n                        .unwrap_or(suffix.len());\n                    facet_child.extend(&suffix[..end]);\n                }\n                collapsed[facet_terms.term_ord() as usize] = (term_ord, depth);\n            }\n        }\n        if !facet_terms.advance() {\n            return Ok(false);\n        }\n    }\n}\n\nfn compute_collapse_mapping(\n    facet_dict: &columnar::Dictionary,\n    facets: &BTreeSet<Facet>,\n) -> io::Result<Vec<(u64, usize)>> {\n    let mut collapsed = vec![(u64::MAX, 0); facet_dict.num_terms()];\n    if facets.is_empty() {\n        return Ok(collapsed);\n    }\n    let mut facet_terms: columnar::Streamer = facet_dict.range().into_stream()?;\n    if !facet_terms.advance() {\n        return Ok(collapsed);\n    }\n    let mut facet_bytes = Vec::new();\n    for facet in facets {\n        facet_bytes.clear();\n        facet_bytes.extend(facet.encoded_str().as_bytes());\n        if !compute_collapse_mapping_one(&mut facet_terms, &facet_bytes, &mut collapsed[..])? {\n            break;\n        }\n    }\n    Ok(collapsed)\n}\n\nimpl SegmentCollector for FacetSegmentCollector {\n    type Fruit = FacetCounts;\n\n    fn collect(&mut self, doc: DocId, _: Score) {\n        let mut previous_collapsed_ord: usize = usize::MAX;\n        for facet_ord in self.reader.facet_ords(doc) {\n            let collapsed_ord = self.compressed_collapse_mapping[facet_ord as usize];\n            self.counts[collapsed_ord] += u64::from(collapsed_ord != previous_collapsed_ord);\n            previous_collapsed_ord = collapsed_ord;\n        }\n    }\n\n    /// Returns the results of the collection.\n    ///\n    /// This method does not just return the counters,\n    /// it also translates the facet ordinals of the last segment.\n    fn harvest(self) -> FacetCounts {\n        let mut facet_counts = BTreeMap::new();\n        let facet_dict = self.reader.facet_dict();\n        for (collapsed_facet_ord, count) in self.counts.iter().cloned().enumerate() {\n            if count == 0 {\n                continue;\n            }\n            let mut facet = vec![];\n            let (facet_ord, facet_depth) = self.unique_facet_ords[collapsed_facet_ord];\n            // TODO handle errors.\n            if facet_dict.ord_to_term(facet_ord, &mut facet).is_ok() {\n                if let Some((end_collapsed_facet, _)) = facet\n                    .iter()\n                    .enumerate()\n                    .filter(|(_pos, &b)| b == 0u8)\n                    .nth(facet_depth)\n                {\n                    facet.truncate(end_collapsed_facet);\n                }\n                if let Ok(facet) = Facet::from_encoded(facet) {\n                    facet_counts.insert(facet, count);\n                }\n            }\n        }\n        FacetCounts { facet_counts }\n    }\n}\n\n/// Intermediary result of the `FacetCollector` that stores\n/// the facet counts for all the segments.\n#[derive(Default, Clone)]\npub struct FacetCounts {\n    facet_counts: BTreeMap<Facet, u64>,\n}\n\npub struct FacetChildIterator<'a> {\n    underlying: btree_map::Range<'a, Facet, u64>,\n}\n\nimpl<'a> Iterator for FacetChildIterator<'a> {\n    type Item = (&'a Facet, u64);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        self.underlying.next().map(|(facet, count)| (facet, *count))\n    }\n}\n\nimpl FacetCounts {\n    /// Returns an iterator over all of the facet count pairs inside this result.\n    /// See the documentation for [`FacetCollector`] for a usage example.\n    pub fn get<T>(&self, facet_from: T) -> FacetChildIterator<'_>\n    where Facet: From<T> {\n        let facet = Facet::from(facet_from);\n        let lower_bound = Bound::Excluded(facet.clone());\n        let upper_bound = if facet.is_root() {\n            Bound::Unbounded\n        } else {\n            let mut facet_after_bytes: String = facet.encoded_str().to_owned();\n            facet_after_bytes.push('\\u{1}');\n            let facet_after = Facet::from_encoded_string(facet_after_bytes);\n            Bound::Excluded(facet_after)\n        };\n        let underlying: btree_map::Range<'_, _, _> =\n            self.facet_counts.range((lower_bound, upper_bound));\n        FacetChildIterator { underlying }\n    }\n\n    /// Returns a vector of top `k` facets with their counts, sorted highest-to-lowest by counts.\n    /// See the documentation for [`FacetCollector`] for a usage example.\n    pub fn top_k<T>(&self, facet: T, k: usize) -> Vec<(&Facet, u64)>\n    where Facet: From<T> {\n        let mut heap = BinaryHeap::with_capacity(k);\n        let mut it = self.get(facet);\n\n        // push the first k elements to first bring the heap\n        // to capacity\n        for (facet, count) in (&mut it).take(k) {\n            heap.push(Hit { count, facet });\n        }\n\n        let mut lowest_count: u64 = heap.peek().map(|hit| hit.count).unwrap_or(u64::MIN); //< the `unwrap_or` case may be triggered but the value\n                                                                                          // is never used in that case.\n\n        for (facet, count) in it {\n            if count > lowest_count {\n                if let Some(mut head) = heap.peek_mut() {\n                    *head = Hit { count, facet };\n                }\n                // the heap gets reconstructed at this point\n                if let Some(head) = heap.peek() {\n                    lowest_count = head.count;\n                }\n            }\n        }\n        heap.into_sorted_vec()\n            .into_iter()\n            .map(|hit| (hit.facet, hit.count))\n            .collect::<Vec<_>>()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeSet;\n\n    use columnar::Dictionary;\n    use rand::distr::Uniform;\n    use rand::prelude::SliceRandom;\n    use rand::{rng, Rng};\n\n    use super::{FacetCollector, FacetCounts};\n    use crate::collector::facet_collector::compress_mapping;\n    use crate::collector::Count;\n    use crate::index::Index;\n    use crate::query::{AllQuery, QueryParser, TermQuery};\n    use crate::schema::{Facet, FacetOptions, IndexRecordOption, Schema, TantivyDocument};\n    use crate::{IndexWriter, Term};\n\n    fn test_collapse_mapping_aux(\n        facet_terms: &[&str],\n        facet_params: &[&str],\n        expected_collapsed_mapping: &[(u64, usize)],\n    ) {\n        let mut facets: Vec<Facet> = facet_terms.iter().map(Facet::from).collect();\n        facets.sort();\n        let facet_terms: Vec<&str> = facets.iter().map(|facet| facet.encoded_str()).collect();\n        let dictionary = Dictionary::build_for_tests(&facet_terms);\n        let facet_params: BTreeSet<Facet> = facet_params.iter().map(Facet::from).collect();\n        let collapse_mapping = super::compute_collapse_mapping(&dictionary, &facet_params).unwrap();\n        assert_eq!(&collapse_mapping[..], expected_collapsed_mapping);\n    }\n\n    #[test]\n    fn test_collapse_simple() {\n        test_collapse_mapping_aux(&[\"/facet/a\", \"/facet/b\"], &[\"/facet\"], &[(0, 1), (1, 1)]);\n        test_collapse_mapping_aux(\n            &[\"/facet/a\", \"/facet/a2\", \"/facet/b\"],\n            &[\"/facet\"],\n            &[(0, 1), (1, 1), (2, 1)],\n        );\n        test_collapse_mapping_aux(&[\"/facet/a\", \"/facet/a/2\"], &[\"/facet\"], &[(0, 1), (0, 1)]);\n        test_collapse_mapping_aux(\n            &[\"/facet/a\", \"/facet/a/2\", \"/facet/b\"],\n            &[\"/facet\"],\n            &[(0, 1), (0, 1), (2, 1)],\n        );\n    }\n\n    fn test_compress_mapping_aux(\n        collapsed_mapping: &[(u64, usize)],\n        expected_compressed_collapsed_mapping: &[usize],\n        expected_unique_facet_ords: &[(u64, usize)],\n    ) {\n        let (compressed_collapsed_mapping, unique_facet_ords) = compress_mapping(collapsed_mapping);\n        assert_eq!(\n            compressed_collapsed_mapping,\n            expected_compressed_collapsed_mapping\n        );\n        assert_eq!(unique_facet_ords, expected_unique_facet_ords);\n    }\n\n    #[test]\n    fn test_compress_mapping() {\n        test_compress_mapping_aux(&[], &[], &[]);\n        test_compress_mapping_aux(&[(1, 2)], &[0], &[(1, 2)]);\n        test_compress_mapping_aux(&[(1, 2), (1, 2)], &[0, 0], &[(1, 2)]);\n        test_compress_mapping_aux(\n            &[(1, 2), (5, 2), (5, 2), (6, 3), (8, 3)],\n            &[0, 1, 1, 2, 3],\n            &[(1, 2), (5, 2), (6, 3), (8, 3)],\n        );\n    }\n\n    #[test]\n    fn test_facet_collector_simple() {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from(\"/facet/a\")))\n            .unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from(\"/facet/b\")))\n            .unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from(\"/facet/b\")))\n            .unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from(\"/facet/c\")))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(\"/facet\");\n        let counts: FacetCounts = searcher.search(&AllQuery, &facet_collector).unwrap();\n        let facets: Vec<(&Facet, u64)> = counts.top_k(\"/facet\", 1);\n        assert_eq!(facets, vec![(&Facet::from(\"/facet/b\"), 2)]);\n    }\n\n    #[test]\n    fn test_facet_collector_drilldown() {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let num_facets: usize = 3 * 4 * 5;\n        let facets: Vec<Facet> = (0..num_facets)\n            .map(|mut n| {\n                let top = n % 3;\n                n /= 3;\n                let mid = n % 4;\n                n /= 4;\n                let leaf = n % 5;\n                Facet::from(&format!(\"/top{top}/mid{mid}/leaf{leaf}\"))\n            })\n            .collect();\n        for i in 0..num_facets * 10 {\n            let mut doc = TantivyDocument::new();\n            doc.add_facet(facet_field, facets[i % num_facets].clone());\n            index_writer.add_document(doc).unwrap();\n        }\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(Facet::from(\"/top1\"));\n        let counts = searcher.search(&AllQuery, &facet_collector).unwrap();\n\n        {\n            let facets: Vec<(String, u64)> = counts\n                .get(\"/top1\")\n                .map(|(facet, count)| (facet.to_string(), count))\n                .collect();\n            assert_eq!(\n                facets,\n                [\n                    (\"/top1/mid0\", 50),\n                    (\"/top1/mid1\", 50),\n                    (\"/top1/mid2\", 50),\n                    (\"/top1/mid3\", 50),\n                ]\n                .iter()\n                .map(|&(facet_str, count)| (String::from(facet_str), count))\n                .collect::<Vec<_>>()\n            );\n        }\n    }\n\n    #[test]\n    #[should_panic(\n        expected = \"Tried to add a facet which is a descendant of an already added facet.\"\n    )]\n    fn test_misused_facet_collector() {\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(Facet::from(\"/country\"));\n        facet_collector.add_facet(Facet::from(\"/country/europe\"));\n    }\n\n    #[test]\n    fn test_doc_unsorted_multifacet() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facets\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(\n            facet_field => Facet::from_text(&\"/subjects/A/a\").unwrap(),\n            facet_field => Facet::from_text(&\"/subjects/B/a\").unwrap(),\n            facet_field => Facet::from_text(&\"/subjects/A/b\").unwrap(),\n            facet_field => Facet::from_text(&\"/subjects/B/b\").unwrap(),\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 1);\n        let mut facet_collector = FacetCollector::for_field(\"facets\");\n        facet_collector.add_facet(\"/subjects\");\n        let counts = searcher.search(&AllQuery, &facet_collector)?;\n        let facets: Vec<(&Facet, u64)> = counts.get(\"/subjects\").collect();\n        assert_eq!(facets[0].1, 1);\n        Ok(())\n    }\n\n    #[test]\n    fn test_doc_search_by_facet() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(\n            facet_field => Facet::from_text(&\"/A/A\").unwrap(),\n        ))?;\n        index_writer.add_document(doc!(\n            facet_field => Facet::from_text(&\"/A/B\").unwrap(),\n        ))?;\n        index_writer.add_document(doc!(\n            facet_field => Facet::from_text(&\"/A/C/A\").unwrap(),\n        ))?;\n        index_writer.add_document(doc!(\n            facet_field => Facet::from_text(&\"/D/C/A\").unwrap(),\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 4);\n\n        let count_facet = |facet_str: &str| {\n            let term = Term::from_facet(facet_field, &Facet::from_text(facet_str).unwrap());\n            searcher\n                .search(&TermQuery::new(term, IndexRecordOption::Basic), &Count)\n                .unwrap()\n        };\n\n        assert_eq!(count_facet(\"/\"), 4);\n        assert_eq!(count_facet(\"/A\"), 3);\n        assert_eq!(count_facet(\"/A/B\"), 1);\n        assert_eq!(count_facet(\"/A/C\"), 1);\n        assert_eq!(count_facet(\"/A/C/A\"), 1);\n        assert_eq!(count_facet(\"/C/A\"), 0);\n\n        let query_parser = QueryParser::for_index(&index, vec![]);\n        {\n            let query = query_parser.parse_query(\"facet:/A/B\")?;\n            assert_eq!(1, searcher.search(&query, &Count).unwrap());\n        }\n        {\n            let query = query_parser.parse_query(\"facet:/A\")?;\n            assert_eq!(3, searcher.search(&query, &Count)?);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_non_used_facet_collector() {\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(Facet::from(\"/country\"));\n        facet_collector.add_facet(Facet::from(\"/countryeurope\"));\n    }\n\n    #[test]\n    fn test_facet_collector_topk() {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        let uniform = Uniform::new_inclusive(1, 100_000).unwrap();\n        let mut docs: Vec<TantivyDocument> =\n            vec![(\"a\", 10), (\"b\", 100), (\"c\", 7), (\"d\", 12), (\"e\", 21)]\n                .into_iter()\n                .flat_map(|(c, count)| {\n                    let facet = Facet::from(&format!(\"/facet/{c}\"));\n                    let doc = doc!(facet_field => facet);\n                    std::iter::repeat_n(doc, count)\n                })\n                .map(|mut doc| {\n                    doc.add_facet(facet_field, &format!(\"/facet/{}\", rng().sample(uniform)));\n                    doc\n                })\n                .collect();\n        docs[..].shuffle(&mut rng());\n\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        for doc in docs {\n            index_writer.add_document(doc).unwrap();\n        }\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(\"/facet\");\n        let counts: FacetCounts = searcher.search(&AllQuery, &facet_collector).unwrap();\n\n        {\n            let facets: Vec<(&Facet, u64)> = counts.top_k(\"/facet\", 3);\n            assert_eq!(\n                facets,\n                vec![\n                    (&Facet::from(\"/facet/b\"), 100),\n                    (&Facet::from(\"/facet/e\"), 21),\n                    (&Facet::from(\"/facet/d\"), 12),\n                ]\n            );\n        }\n    }\n\n    #[test]\n    fn test_facet_collector_topk_tie_break() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        let docs: Vec<TantivyDocument> = vec![(\"b\", 2), (\"a\", 2), (\"c\", 4)]\n            .into_iter()\n            .flat_map(|(c, count)| {\n                let facet = Facet::from(&format!(\"/facet/{c}\"));\n                let doc = doc!(facet_field => facet);\n                std::iter::repeat_n(doc, count)\n            })\n            .collect();\n\n        let mut index_writer = index.writer_for_tests()?;\n        for doc in docs {\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n        let mut facet_collector = FacetCollector::for_field(\"facet\");\n        facet_collector.add_facet(\"/facet\");\n        let counts: FacetCounts = searcher.search(&AllQuery, &facet_collector)?;\n\n        let facets: Vec<(&Facet, u64)> = counts.top_k(\"/facet\", 2);\n        assert_eq!(\n            facets,\n            vec![(&Facet::from(\"/facet/c\"), 4), (&Facet::from(\"/facet/a\"), 2)]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn is_child_facet() {\n        assert!(super::is_child_facet(&b\"foo\"[..], &b\"foo\\0bar\"[..]));\n        assert!(super::is_child_facet(&b\"\"[..], &b\"foo\\0bar\"[..]));\n        assert!(super::is_child_facet(&b\"\"[..], &b\"foo\"[..]));\n        assert!(!super::is_child_facet(&b\"foo\\0bar\"[..], &b\"foo\"[..]));\n        assert!(!super::is_child_facet(&b\"foo\"[..], &b\"foobar\\0baz\"[..]));\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use rand::rng;\n    use rand::seq::SliceRandom;\n    use test::Bencher;\n\n    use crate::collector::FacetCollector;\n    use crate::query::AllQuery;\n    use crate::schema::{Facet, Schema, INDEXED};\n    use crate::{Index, IndexWriter};\n\n    #[bench]\n    fn bench_facet_collector(b: &mut Bencher) {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        let mut docs = vec![];\n        for val in 0..50 {\n            let facet = Facet::from(&format!(\"/facet_{val}\"));\n            for _ in 0..val * val {\n                docs.push(doc!(facet_field=>facet.clone()));\n            }\n        }\n        // 40425 docs\n        docs[..].shuffle(&mut rng());\n\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        for doc in docs {\n            index_writer.add_document(doc).unwrap();\n        }\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        b.iter(|| {\n            let searcher = reader.searcher();\n            let facet_collector = FacetCollector::for_field(\"facet\");\n            searcher.search(&AllQuery, &facet_collector).unwrap();\n        });\n    }\n}\n"
  },
  {
    "path": "src/collector/filter_collector_wrapper.rs",
    "content": "// # Custom collector example\n//\n// This example shows how you can implement your own\n// collector. As an example, we will compute a collector\n// that computes the standard deviation of a given fast field.\n//\n// Of course, you can have a look at the tantivy's built-in collectors\n// such as the `CountCollector` for more examples.\nuse std::fmt::Debug;\nuse std::marker::PhantomData;\n\nuse columnar::{BytesColumn, Column, DynamicColumn, HasAssociatedColumnType};\n\nuse crate::collector::{Collector, SegmentCollector};\nuse crate::schema::Schema;\nuse crate::{DocId, Score, SegmentReader};\n\n/// The `FilterCollector` filters docs using a fast field value and a predicate.\n///\n/// Only the documents containing at least one value for which the predicate returns `true`\n/// will be passed on to the next collector.\n///\n/// In other words,\n/// - documents with no values are filtered out.\n/// - documents with several values are accepted if at least one value matches the predicate.\n///\n///\n/// ```rust\n/// use tantivy::collector::{TopDocs, FilterCollector};\n/// use tantivy::query::QueryParser;\n/// use tantivy::schema::{Schema, TEXT, FAST};\n/// use tantivy::{doc, DocAddress, Index};\n///\n/// # fn main() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let price = schema_builder.add_u64_field(\"price\", FAST);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n///\n/// let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n/// index_writer.add_document(doc!(title => \"The Name of the Wind\", price => 30_200u64))?;\n/// index_writer.add_document(doc!(title => \"The Diary of Muadib\", price => 29_240u64))?;\n/// index_writer.add_document(doc!(title => \"A Dairy Cow\", price => 21_240u64))?;\n/// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\", price => 20_120u64))?;\n/// index_writer.commit()?;\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n///\n/// let query_parser = QueryParser::for_index(&index, vec![title]);\n/// let query = query_parser.parse_query(\"diary\")?;\n/// let no_filter_collector = FilterCollector::new(\"price\".to_string(), |value: u64| value > 20_120u64, TopDocs::with_limit(2).order_by_score());\n/// let top_docs = searcher.search(&query, &no_filter_collector)?;\n///\n/// assert_eq!(top_docs.len(), 1);\n/// assert_eq!(top_docs[0].1, DocAddress::new(0, 1));\n///\n/// let filter_all_collector: FilterCollector<_, _, u64> = FilterCollector::new(\"price\".to_string(), |value| value < 5u64, TopDocs::with_limit(2).order_by_score());\n/// let filtered_top_docs = searcher.search(&query, &filter_all_collector)?;\n///\n/// assert_eq!(filtered_top_docs.len(), 0);\n/// # Ok(())\n/// # }\n/// ```\n///\n/// Note that this is limited to fast fields which implement the\n/// [`FastValue`][crate::fastfield::FastValue] trait, e.g. `u64` but not `&[u8]`.\n/// To filter based on a bytes fast field, use a [`BytesFilterCollector`] instead.\npub struct FilterCollector<TCollector, TPredicate, TPredicateValue>\nwhere TPredicate: 'static + Clone\n{\n    field: String,\n    collector: TCollector,\n    predicate: TPredicate,\n    t_predicate_value: PhantomData<TPredicateValue>,\n}\n\nimpl<TCollector, TPredicate, TPredicateValue>\n    FilterCollector<TCollector, TPredicate, TPredicateValue>\nwhere\n    TCollector: Collector + Send + Sync,\n    TPredicate: Fn(TPredicateValue) -> bool + Send + Sync + Clone,\n{\n    /// Create a new `FilterCollector`.\n    pub fn new(field: String, predicate: TPredicate, collector: TCollector) -> Self {\n        Self {\n            field,\n            predicate,\n            collector,\n            t_predicate_value: PhantomData,\n        }\n    }\n}\n\nimpl<TCollector, TPredicate, TPredicateValue> Collector\n    for FilterCollector<TCollector, TPredicate, TPredicateValue>\nwhere\n    TCollector: Collector + Send + Sync,\n    TPredicate: 'static + Fn(TPredicateValue) -> bool + Send + Sync + Clone,\n    TPredicateValue: HasAssociatedColumnType,\n    DynamicColumn: Into<Option<columnar::Column<TPredicateValue>>>,\n{\n    type Fruit = TCollector::Fruit;\n\n    type Child = FilterSegmentCollector<TCollector::Child, TPredicate, TPredicateValue>;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.collector.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let column_opt = segment_reader.fast_fields().column_opt(&self.field)?;\n\n        let segment_collector = self\n            .collector\n            .for_segment(segment_local_id, segment_reader)?;\n\n        Ok(FilterSegmentCollector {\n            column_opt,\n            segment_collector,\n            predicate: self.predicate.clone(),\n            t_predicate_value: PhantomData,\n            filtered_docs: Vec::with_capacity(crate::COLLECT_BLOCK_BUFFER_LEN),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.collector.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<TCollector::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<TCollector::Fruit> {\n        self.collector.merge_fruits(segment_fruits)\n    }\n}\n\npub struct FilterSegmentCollector<TSegmentCollector, TPredicate, TPredicateValue> {\n    column_opt: Option<Column<TPredicateValue>>,\n    segment_collector: TSegmentCollector,\n    predicate: TPredicate,\n    t_predicate_value: PhantomData<TPredicateValue>,\n    filtered_docs: Vec<DocId>,\n}\n\nimpl<TSegmentCollector, TPredicate, TPredicateValue>\n    FilterSegmentCollector<TSegmentCollector, TPredicate, TPredicateValue>\nwhere\n    TPredicateValue: PartialOrd + Copy + Debug + Send + Sync + 'static,\n    TPredicate: 'static + Fn(TPredicateValue) -> bool + Send + Sync,\n{\n    #[inline]\n    fn accept_document(&self, doc_id: DocId) -> bool {\n        if let Some(column) = &self.column_opt {\n            for val in column.values_for_doc(doc_id) {\n                if (self.predicate)(val) {\n                    return true;\n                }\n            }\n        }\n        false\n    }\n}\n\nimpl<TSegmentCollector, TPredicate, TPredicateValue> SegmentCollector\n    for FilterSegmentCollector<TSegmentCollector, TPredicate, TPredicateValue>\nwhere\n    TSegmentCollector: SegmentCollector,\n    TPredicateValue: HasAssociatedColumnType,\n    TPredicate: 'static + Fn(TPredicateValue) -> bool + Send + Sync, /* DynamicColumn: Into<Option<columnar::Column<TPredicateValue>>> */\n{\n    type Fruit = TSegmentCollector::Fruit;\n\n    fn collect(&mut self, doc: u32, score: Score) {\n        if self.accept_document(doc) {\n            self.segment_collector.collect(doc, score);\n        }\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.filtered_docs.clear();\n        for &doc in docs {\n            // TODO: `accept_document` could be further optimized to do batch lookups of column\n            // values for single-valued columns.\n            if self.accept_document(doc) {\n                self.filtered_docs.push(doc);\n            }\n        }\n        if !self.filtered_docs.is_empty() {\n            self.segment_collector.collect_block(&self.filtered_docs);\n        }\n    }\n\n    fn harvest(self) -> TSegmentCollector::Fruit {\n        self.segment_collector.harvest()\n    }\n}\n\n/// A variant of the [`FilterCollector`] specialized for bytes fast fields, i.e.\n///\n/// it transparently wraps an inner [`Collector`] but filters documents\n/// based on the result of applying the predicate to the bytes fast field.\n///\n/// A document is accepted if and only if the predicate returns `true` for at least one value.\n///\n/// In other words,\n/// - documents with no values are filtered out.\n/// - documents with several values are accepted if at least one value matches the predicate.\n///\n/// ```rust\n/// use tantivy::collector::{TopDocs, BytesFilterCollector};\n/// use tantivy::query::QueryParser;\n/// use tantivy::schema::{Schema, TEXT, FAST};\n/// use tantivy::{doc, DocAddress, Index};\n///\n/// # fn main() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let barcode = schema_builder.add_bytes_field(\"barcode\", FAST);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n///\n/// let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n/// index_writer.add_document(doc!(title => \"The Name of the Wind\", barcode => &b\"010101\"[..]))?;\n/// index_writer.add_document(doc!(title => \"The Diary of Muadib\", barcode => &b\"110011\"[..]))?;\n/// index_writer.add_document(doc!(title => \"A Dairy Cow\", barcode => &b\"110111\"[..]))?;\n/// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\", barcode => &b\"011101\"[..]))?;\n/// index_writer.add_document(doc!(title => \"Bridget Jones's Diary\"))?;\n/// index_writer.commit()?;\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n///\n/// let query_parser = QueryParser::for_index(&index, vec![title]);\n/// let query = query_parser.parse_query(\"diary\")?;\n/// let filter_collector = BytesFilterCollector::new(\"barcode\".to_string(), |bytes: &[u8]| bytes.starts_with(b\"01\"), TopDocs::with_limit(2).order_by_score());\n/// let top_docs = searcher.search(&query, &filter_collector)?;\n///\n/// assert_eq!(top_docs.len(), 1);\n/// assert_eq!(top_docs[0].1, DocAddress::new(0, 3));\n/// # Ok(())\n/// # }\n/// ```\npub struct BytesFilterCollector<TCollector, TPredicate>\nwhere TPredicate: 'static + Clone\n{\n    field: String,\n    collector: TCollector,\n    predicate: TPredicate,\n}\n\nimpl<TCollector, TPredicate> BytesFilterCollector<TCollector, TPredicate>\nwhere\n    TCollector: Collector + Send + Sync,\n    TPredicate: Fn(&[u8]) -> bool + Send + Sync + Clone,\n{\n    /// Create a new `BytesFilterCollector`.\n    pub fn new(field: String, predicate: TPredicate, collector: TCollector) -> Self {\n        Self {\n            field,\n            predicate,\n            collector,\n        }\n    }\n}\n\nimpl<TCollector, TPredicate> Collector for BytesFilterCollector<TCollector, TPredicate>\nwhere\n    TCollector: Collector + Send + Sync,\n    TPredicate: 'static + Fn(&[u8]) -> bool + Send + Sync + Clone,\n{\n    type Fruit = TCollector::Fruit;\n\n    type Child = BytesFilterSegmentCollector<TCollector::Child, TPredicate>;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.collector.check_schema(schema)\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let column_opt = segment_reader.fast_fields().bytes(&self.field)?;\n\n        let segment_collector = self\n            .collector\n            .for_segment(segment_local_id, segment_reader)?;\n\n        Ok(BytesFilterSegmentCollector {\n            column_opt,\n            segment_collector,\n            predicate: self.predicate.clone(),\n            buffer: Vec::new(),\n            filtered_docs: Vec::with_capacity(crate::COLLECT_BLOCK_BUFFER_LEN),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.collector.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<TCollector::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<TCollector::Fruit> {\n        self.collector.merge_fruits(segment_fruits)\n    }\n}\n\npub struct BytesFilterSegmentCollector<TSegmentCollector, TPredicate>\nwhere TPredicate: 'static\n{\n    column_opt: Option<BytesColumn>,\n    segment_collector: TSegmentCollector,\n    predicate: TPredicate,\n    buffer: Vec<u8>,\n    filtered_docs: Vec<DocId>,\n}\n\nimpl<TSegmentCollector, TPredicate> BytesFilterSegmentCollector<TSegmentCollector, TPredicate>\nwhere\n    TSegmentCollector: SegmentCollector,\n    TPredicate: 'static + Fn(&[u8]) -> bool + Send + Sync,\n{\n    #[inline]\n    fn accept_document(&mut self, doc_id: DocId) -> bool {\n        if let Some(column) = &self.column_opt {\n            for ord in column.term_ords(doc_id) {\n                self.buffer.clear();\n\n                let found = column.ord_to_bytes(ord, &mut self.buffer).unwrap_or(false);\n\n                if found && (self.predicate)(&self.buffer) {\n                    return true;\n                }\n            }\n        }\n        false\n    }\n}\n\nimpl<TSegmentCollector, TPredicate> SegmentCollector\n    for BytesFilterSegmentCollector<TSegmentCollector, TPredicate>\nwhere\n    TSegmentCollector: SegmentCollector,\n    TPredicate: 'static + Fn(&[u8]) -> bool + Send + Sync,\n{\n    type Fruit = TSegmentCollector::Fruit;\n\n    fn collect(&mut self, doc: u32, score: Score) {\n        if self.accept_document(doc) {\n            self.segment_collector.collect(doc, score);\n        }\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.filtered_docs.clear();\n        for &doc in docs {\n            // TODO: `accept_document` could be further optimized to do batch lookups of column\n            // values for single-valued columns.\n            if self.accept_document(doc) {\n                self.filtered_docs.push(doc);\n            }\n        }\n        if !self.filtered_docs.is_empty() {\n            self.segment_collector.collect_block(&self.filtered_docs);\n        }\n    }\n\n    fn harvest(self) -> TSegmentCollector::Fruit {\n        self.segment_collector.harvest()\n    }\n}\n"
  },
  {
    "path": "src/collector/histogram_collector.rs",
    "content": "use std::sync::Arc;\n\nuse columnar::ColumnValues;\nuse fastdivide::DividerU64;\n\nuse crate::collector::{Collector, SegmentCollector};\nuse crate::fastfield::{FastFieldNotAvailableError, FastValue};\nuse crate::schema::Type;\nuse crate::{DocId, Score};\n\n/// Histogram builds an histogram of the values of a fastfield for the\n/// collected DocSet.\n///\n/// At construction, it is given parameters that define a partition of an interval\n/// [min_val, max_val) into N buckets with the same width.\n/// The ith bucket is then defined by `[min_val + i * bucket_width, min_val + (i+1) * bucket_width)`\n///\n/// An histogram is then defined as a `Vec<u64>` of length `num_buckets`, that contains a count of\n/// documents for each value bucket.\n///\n/// See also [`HistogramCollector::new()`].\n///\n/// # Warning\n///\n/// f64 fields are not supported.\n#[derive(Clone)]\npub struct HistogramCollector {\n    min_value: u64,\n    num_buckets: usize,\n    divider: DividerU64,\n    field: String,\n}\n\nimpl HistogramCollector {\n    /// Builds a new HistogramCollector.\n    ///\n    /// The scale/range of the histogram is not dynamic. It is required to\n    /// define it by supplying following parameter:\n    ///  - `min_value`: the minimum value that can be recorded in the histogram.\n    ///  - `bucket_width`: the length of the interval that is associated with each buckets.\n    ///  - `num_buckets`: The overall number of buckets.\n    ///\n    /// Together, this parameters define a partition of `[min_value, min_value + num_buckets *\n    /// bucket_width)` into `num_buckets` intervals of width bucket that we call `bucket`.\n    ///\n    /// # Disclaimer\n    /// This function panics if the field given is of type f64.\n    pub fn new<TFastValue: FastValue>(\n        field: String,\n        min_value: TFastValue,\n        bucket_width: u64,\n        num_buckets: usize,\n    ) -> HistogramCollector {\n        let fast_type = TFastValue::to_type();\n        assert!(fast_type == Type::U64 || fast_type == Type::I64 || fast_type == Type::Date);\n        HistogramCollector {\n            min_value: min_value.to_u64(),\n            num_buckets,\n            field,\n            divider: DividerU64::divide_by(bucket_width),\n        }\n    }\n}\n\nstruct HistogramComputer {\n    counts: Vec<u64>,\n    min_value: u64,\n    divider: DividerU64,\n}\n\nimpl HistogramComputer {\n    #[inline]\n    pub(crate) fn add_value(&mut self, value: u64) {\n        if value < self.min_value {\n            return;\n        }\n        let delta = value - self.min_value;\n        let bucket_id: usize = self.divider.divide(delta) as usize;\n        if bucket_id < self.counts.len() {\n            self.counts[bucket_id] += 1;\n        }\n    }\n\n    fn harvest(self) -> Vec<u64> {\n        self.counts\n    }\n}\npub struct SegmentHistogramCollector {\n    histogram_computer: HistogramComputer,\n    column_u64: Arc<dyn ColumnValues<u64>>,\n}\n\nimpl SegmentCollector for SegmentHistogramCollector {\n    type Fruit = Vec<u64>;\n\n    fn collect(&mut self, doc: DocId, _score: Score) {\n        let value = self.column_u64.get_val(doc);\n        self.histogram_computer.add_value(value);\n    }\n\n    fn harvest(self) -> Self::Fruit {\n        self.histogram_computer.harvest()\n    }\n}\n\nimpl Collector for HistogramCollector {\n    type Fruit = Vec<u64>;\n    type Child = SegmentHistogramCollector;\n\n    fn for_segment(\n        &self,\n        _segment_local_id: crate::SegmentOrdinal,\n        segment: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let column_opt = segment.fast_fields().u64_lenient(&self.field)?;\n        let (column, _column_type) = column_opt.ok_or_else(|| FastFieldNotAvailableError {\n            field_name: self.field.clone(),\n        })?;\n        let column_u64 = column.first_or_default_col(0u64);\n        Ok(SegmentHistogramCollector {\n            histogram_computer: HistogramComputer {\n                counts: vec![0; self.num_buckets],\n                min_value: self.min_value,\n                divider: self.divider,\n            },\n            column_u64,\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(&self, child_histograms: Vec<Vec<u64>>) -> crate::Result<Vec<u64>> {\n        Ok(add_vecs(child_histograms, self.num_buckets))\n    }\n}\n\npub fn add_arrays_into(acc: &mut [u64], add: &[u64]) {\n    assert_eq!(acc.len(), add.len());\n    for (dest_bucket, bucket_count) in acc.iter_mut().zip(add) {\n        *dest_bucket += bucket_count;\n    }\n}\n\nfn add_vecs(mut vals_list: Vec<Vec<u64>>, len: usize) -> Vec<u64> {\n    let mut acc = vals_list.pop().unwrap_or_else(|| vec![0u64; len]);\n    assert_eq!(acc.len(), len);\n    for vals in vals_list {\n        add_arrays_into(&mut acc, &vals);\n    }\n    acc\n}\n\n#[cfg(test)]\nmod tests {\n    use fastdivide::DividerU64;\n    use query::AllQuery;\n\n    use super::{add_vecs, HistogramCollector, HistogramComputer};\n    use crate::schema::{Schema, FAST};\n    use crate::time::{Date, Month};\n    use crate::{query, DateTime, Index};\n\n    #[test]\n    fn test_add_histograms_simple() {\n        assert_eq!(\n            add_vecs(vec![vec![1, 0, 3], vec![11, 2, 3], vec![0, 0, 1]], 3),\n            vec![12, 2, 7]\n        )\n    }\n\n    #[test]\n    fn test_add_histograms_empty() {\n        assert_eq!(add_vecs(vec![], 3), vec![0, 0, 0])\n    }\n\n    #[test]\n    fn test_histogram_builder_simple() {\n        // [1..3)\n        // [3..5)\n        // ..\n        // [9..11)\n        let mut histogram_computer = HistogramComputer {\n            counts: vec![0; 5],\n            min_value: 1,\n            divider: DividerU64::divide_by(2),\n        };\n        histogram_computer.add_value(1);\n        histogram_computer.add_value(7);\n        assert_eq!(histogram_computer.harvest(), vec![1, 0, 0, 1, 0]);\n    }\n\n    #[test]\n    fn test_histogram_too_low_is_ignored() {\n        let mut histogram_computer = HistogramComputer {\n            counts: vec![0; 5],\n            min_value: 2,\n            divider: DividerU64::divide_by(2),\n        };\n        histogram_computer.add_value(0);\n        assert_eq!(histogram_computer.harvest(), vec![0, 0, 0, 0, 0]);\n    }\n\n    #[test]\n    fn test_histogram_too_high_is_ignored() {\n        let mut histogram_computer = HistogramComputer {\n            counts: vec![0u64; 5],\n            min_value: 0,\n            divider: DividerU64::divide_by(2),\n        };\n        histogram_computer.add_value(10);\n        assert_eq!(histogram_computer.harvest(), vec![0, 0, 0, 0, 0]);\n    }\n    #[test]\n    fn test_no_segments() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_u64_field(\"val_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let all_query = AllQuery;\n        let histogram_collector = HistogramCollector::new(\"val_field\".to_string(), 0u64, 2, 5);\n        let histogram = searcher.search(&all_query, &histogram_collector)?;\n        assert_eq!(histogram, vec![0; 5]);\n        Ok(())\n    }\n\n    #[test]\n    fn test_histogram_i64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let val_field = schema_builder.add_i64_field(\"val_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(val_field=>12i64))?;\n        writer.add_document(doc!(val_field=>-30i64))?;\n        writer.add_document(doc!(val_field=>-12i64))?;\n        writer.add_document(doc!(val_field=>-10i64))?;\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let all_query = AllQuery;\n        let histogram_collector =\n            HistogramCollector::new(\"val_field\".to_string(), -20i64, 10u64, 4);\n        let histogram = searcher.search(&all_query, &histogram_collector)?;\n        assert_eq!(histogram, vec![1, 1, 0, 1]);\n        Ok(())\n    }\n\n    #[test]\n    fn test_histogram_merge() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let val_field = schema_builder.add_i64_field(\"val_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(val_field=>12i64))?;\n        writer.commit()?;\n        writer.add_document(doc!(val_field=>-30i64))?;\n        writer.commit()?;\n        writer.add_document(doc!(val_field=>-12i64))?;\n        writer.commit()?;\n        writer.add_document(doc!(val_field=>-10i64))?;\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let all_query = AllQuery;\n        let histogram_collector =\n            HistogramCollector::new(\"val_field\".to_string(), -20i64, 10u64, 4);\n        let histogram = searcher.search(&all_query, &histogram_collector)?;\n        assert_eq!(histogram, vec![1, 1, 0, 1]);\n        Ok(())\n    }\n\n    #[test]\n    fn test_histogram_dates() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1982, Month::September, 17)?.with_hms(0, 0, 0)?)))?;\n        writer.add_document(\n            doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1986, Month::March, 9)?.with_hms(0, 0, 0)?)),\n        )?;\n        writer.add_document(doc!(date_field=>DateTime::from_primitive(Date::from_calendar_date(1983, Month::September, 27)?.with_hms(0, 0, 0)?)))?;\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let all_query = AllQuery;\n        let week_histogram_collector = HistogramCollector::new(\n            \"date_field\".to_string(),\n            DateTime::from_primitive(\n                Date::from_calendar_date(1980, Month::January, 1)?.with_hms(0, 0, 0)?,\n            ),\n            3_600_000_000_000 * 24 * 365, // it is just for a unit test... sorry leap years.\n            10,\n        );\n        let week_histogram = searcher.search(&all_query, &week_histogram_collector)?;\n        assert_eq!(week_histogram, vec![0, 0, 1, 1, 0, 0, 1, 0, 0, 0]);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/collector/mod.rs",
    "content": "//! # Collectors\n//!\n//! Collectors define the information you want to extract from the documents matching the queries.\n//! In tantivy jargon, we call this information your search \"fruit\".\n//!\n//! Your fruit could for instance be :\n//! - [the count of matching documents](crate::collector::Count)\n//! - [the top 10 documents, by relevancy or by a fast field](crate::collector::TopDocs)\n//! - [facet counts](FacetCollector)\n//!\n//! At some point in your code, you will trigger the actual search operation by calling\n//! [`Searcher::search()`](crate::Searcher::search).\n//! This call will look like this:\n//!\n//! ```verbatim\n//! let fruit = searcher.search(&query, &collector)?;\n//! ```\n//!\n//! Here the type of fruit is actually determined as an associated type of the collector\n//! (`Collector::Fruit`).\n//!\n//!\n//! # Combining several collectors\n//!\n//! A rich search experience often requires to run several collectors on your search query.\n//! For instance,\n//! - selecting the top-K products matching your query\n//! - counting the matching documents\n//! - computing several facets\n//! - computing statistics about the matching product prices\n//!\n//! A simple and efficient way to do that is to pass your collectors as one tuple.\n//! The resulting `Fruit` will then be a typed tuple with each collector's original fruits\n//! in their respective position.\n//!\n//! ```rust\n//! # use tantivy::schema::*;\n//! # use tantivy::*;\n//! # use tantivy::query::*;\n//! use tantivy::collector::{Count, TopDocs};\n//! #\n//! # fn main() -> tantivy::Result<()> {\n//! # let mut schema_builder = Schema::builder();\n//! #     let title = schema_builder.add_text_field(\"title\", TEXT);\n//! #     let schema = schema_builder.build();\n//! #     let index = Index::create_in_ram(schema);\n//! #     let mut index_writer = index.writer(15_000_000)?;\n//! #       index_writer.add_document(doc!(\n//! #       title => \"The Name of the Wind\",\n//! #      ))?;\n//! #     index_writer.add_document(doc!(\n//! #        title => \"The Diary of Muadib\",\n//! #     ))?;\n//! #     index_writer.commit()?;\n//! #     let reader = index.reader()?;\n//! #     let searcher = reader.searcher();\n//! #     let query_parser = QueryParser::for_index(&index, vec![title]);\n//! #     let query = query_parser.parse_query(\"diary\")?;\n//! let (doc_count, top_docs): (usize, Vec<(Score, DocAddress)>) =\n//! searcher.search(&query, &(Count, TopDocs::with_limit(2).order_by_score()))?;\n//! #     Ok(())\n//! # }\n//! ```\n//!\n//! The `Collector` trait is implemented for up to 4 collectors.\n//! If you have more than 4 collectors, you can either group them into\n//! tuples of tuples `(a,(b,(c,d)))`, or rely on [`MultiCollector`].\n//!\n//! # Combining several collectors dynamically\n//!\n//! Combining collectors into a tuple is a zero-cost abstraction: everything\n//! happens as if you had manually implemented a single collector\n//! combining all of our features.\n//!\n//! Unfortunately it requires you to know at compile time your collector types.\n//! If on the other hand, the collectors depend on some query parameter,\n//! you can rely on [`MultiCollector`]'s.\n//!\n//!\n//! # Implementing your own collectors.\n//!\n//! See the `custom_collector` example.\n\nuse downcast_rs::impl_downcast;\n\nuse crate::schema::Schema;\nuse crate::{DocId, Score, SegmentOrdinal, SegmentReader};\n\nmod count_collector;\npub use self::count_collector::Count;\n\n/// Sort keys\npub mod sort_key;\n\nmod histogram_collector;\npub use histogram_collector::HistogramCollector;\n\nmod multi_collector;\npub use self::multi_collector::{FruitHandle, MultiCollector, MultiFruit};\n\nmod top_collector;\npub use self::top_collector::ComparableDoc;\n\nmod top_score_collector;\npub use self::top_score_collector::{TopDocs, TopNComputer};\n\nmod sort_key_top_collector;\npub use self::sort_key::{SegmentSortKeyComputer, SortKeyComputer};\nmod facet_collector;\npub use self::facet_collector::{FacetCollector, FacetCounts};\nuse crate::query::Weight;\n\nmod docset_collector;\npub use self::docset_collector::DocSetCollector;\n\nmod filter_collector_wrapper;\npub use self::filter_collector_wrapper::{BytesFilterCollector, FilterCollector};\n\n/// `Fruit` is the type for the result of our collection.\n/// e.g. `usize` for the `Count` collector.\npub trait Fruit: Send + downcast_rs::Downcast {}\n\nimpl<T> Fruit for T where T: Send + downcast_rs::Downcast {}\n\n/// Collectors are in charge of collecting and retaining relevant\n/// information from the document found and scored by the query.\n///\n/// For instance,\n///\n/// - keeping track of the top 10 best documents\n/// - computing a breakdown over a fast field\n/// - computing the number of documents matching the query\n///\n/// Our search index is in fact a collection of segments, so\n/// a `Collector` trait is actually more of a factory to instance\n/// `SegmentCollector`s for each segments.\n///\n/// The collection logic itself is in the `SegmentCollector`.\n///\n/// Segments are not guaranteed to be visited in any specific order.\npub trait Collector: Sync + Send {\n    /// `Fruit` is the type for the result of our collection.\n    /// e.g. `usize` for the `Count` collector.\n    type Fruit: Fruit;\n\n    /// Type of the `SegmentCollector` associated with this collector.\n    type Child: SegmentCollector;\n\n    /// Returns an error if the schema is not compatible with the collector.\n    fn check_schema(&self, _schema: &Schema) -> crate::Result<()> {\n        Ok(())\n    }\n\n    /// `set_segment` is called before beginning to enumerate\n    /// on this segment.\n    fn for_segment(\n        &self,\n        segment_local_id: SegmentOrdinal,\n        segment: &SegmentReader,\n    ) -> crate::Result<Self::Child>;\n\n    /// Returns true iff the collector requires to compute scores for documents.\n    fn requires_scoring(&self) -> bool;\n\n    /// Combines the fruit associated with the collection of each segments\n    /// into one fruit.\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit>;\n\n    /// Created a segment collector and\n    fn collect_segment(\n        &self,\n        weight: &dyn Weight,\n        segment_ord: u32,\n        reader: &SegmentReader,\n    ) -> crate::Result<<Self::Child as SegmentCollector>::Fruit> {\n        let with_scoring = self.requires_scoring();\n        let mut segment_collector = self.for_segment(segment_ord, reader)?;\n        default_collect_segment_impl(&mut segment_collector, weight, reader, with_scoring)?;\n        Ok(segment_collector.harvest())\n    }\n}\n\npub(crate) fn default_collect_segment_impl<TSegmentCollector: SegmentCollector>(\n    segment_collector: &mut TSegmentCollector,\n    weight: &dyn Weight,\n    reader: &SegmentReader,\n    with_scoring: bool,\n) -> crate::Result<()> {\n    match (reader.alive_bitset(), with_scoring) {\n        (Some(alive_bitset), true) => {\n            weight.for_each(reader, &mut |doc, score| {\n                if alive_bitset.is_alive(doc) {\n                    segment_collector.collect(doc, score);\n                }\n            })?;\n        }\n        (Some(alive_bitset), false) => {\n            weight.for_each_no_score(reader, &mut |docs| {\n                for doc in docs.iter().cloned() {\n                    if alive_bitset.is_alive(doc) {\n                        segment_collector.collect(doc, 0.0);\n                    }\n                }\n            })?;\n        }\n        (None, true) => {\n            weight.for_each(reader, &mut |doc, score| {\n                segment_collector.collect(doc, score);\n            })?;\n        }\n        (None, false) => {\n            weight.for_each_no_score(reader, &mut |docs| {\n                segment_collector.collect_block(docs);\n            })?;\n        }\n    }\n    Ok(())\n}\n\nimpl<TSegmentCollector: SegmentCollector> SegmentCollector for Option<TSegmentCollector> {\n    type Fruit = Option<TSegmentCollector::Fruit>;\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        if let Some(segment_collector) = self {\n            segment_collector.collect(doc, score);\n        }\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        if let Some(segment_collector) = self {\n            segment_collector.collect_block(docs);\n        }\n    }\n\n    fn harvest(self) -> Self::Fruit {\n        self.map(|segment_collector| segment_collector.harvest())\n    }\n}\n\nimpl<TCollector: Collector> Collector for Option<TCollector> {\n    type Fruit = Option<TCollector::Fruit>;\n\n    type Child = Option<<TCollector as Collector>::Child>;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        if let Some(underlying_collector) = self {\n            underlying_collector.check_schema(schema)?;\n        }\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: SegmentOrdinal,\n        segment: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        Ok(if let Some(inner) = self {\n            let inner_segment_collector = inner.for_segment(segment_local_id, segment)?;\n            Some(inner_segment_collector)\n        } else {\n            None\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.as_ref()\n            .map(|inner| inner.requires_scoring())\n            .unwrap_or(false)\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit> {\n        if let Some(inner) = self.as_ref() {\n            let inner_segment_fruits: Vec<_> = segment_fruits\n                .into_iter()\n                .flat_map(|fruit_opt| fruit_opt.into_iter())\n                .collect();\n            let fruit = inner.merge_fruits(inner_segment_fruits)?;\n            Ok(Some(fruit))\n        } else {\n            Ok(None)\n        }\n    }\n}\n\n/// The `SegmentCollector` is the trait in charge of defining the\n/// collect operation at the scale of the segment.\n///\n/// `.collect(doc, score)` will be called for every documents\n/// matching the query.\npub trait SegmentCollector: 'static {\n    /// `Fruit` is the type for the result of our collection.\n    /// e.g. `usize` for the `Count` collector.\n    type Fruit: Fruit;\n\n    /// The query pushes the scored document to the collector via this method.\n    fn collect(&mut self, doc: DocId, score: Score);\n\n    /// The query pushes the scored document to the collector via this method.\n    /// This method is used when the collector does not require scoring.\n    ///\n    /// See [`COLLECT_BLOCK_BUFFER_LEN`](crate::COLLECT_BLOCK_BUFFER_LEN) for the\n    /// buffer size passed to the collector.\n    fn collect_block(&mut self, docs: &[DocId]) {\n        for doc in docs {\n            self.collect(*doc, 0.0);\n        }\n    }\n\n    /// Extract the fruit of the collection from the `SegmentCollector`.\n    fn harvest(self) -> Self::Fruit;\n}\n\n// -----------------------------------------------\n// Tuple implementations.\n\nimpl<Left, Right> Collector for (Left, Right)\nwhere\n    Left: Collector,\n    Right: Collector,\n{\n    type Fruit = (Left::Fruit, Right::Fruit);\n    type Child = (Left::Child, Right::Child);\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        segment: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let left = self.0.for_segment(segment_local_id, segment)?;\n        let right = self.1.for_segment(segment_local_id, segment)?;\n        Ok((left, right))\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring() || self.1.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        segment_fruits: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<(Left::Fruit, Right::Fruit)> {\n        let mut left_fruits = vec![];\n        let mut right_fruits = vec![];\n        for (left_fruit, right_fruit) in segment_fruits {\n            left_fruits.push(left_fruit);\n            right_fruits.push(right_fruit);\n        }\n        Ok((\n            self.0.merge_fruits(left_fruits)?,\n            self.1.merge_fruits(right_fruits)?,\n        ))\n    }\n}\n\nimpl<Left, Right> SegmentCollector for (Left, Right)\nwhere\n    Left: SegmentCollector,\n    Right: SegmentCollector,\n{\n    type Fruit = (Left::Fruit, Right::Fruit);\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        self.0.collect(doc, score);\n        self.1.collect(doc, score);\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.0.collect_block(docs);\n        self.1.collect_block(docs);\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        (self.0.harvest(), self.1.harvest())\n    }\n}\n\n// 3-Tuple\n\nimpl<One, Two, Three> Collector for (One, Two, Three)\nwhere\n    One: Collector,\n    Two: Collector,\n    Three: Collector,\n{\n    type Fruit = (One::Fruit, Two::Fruit, Three::Fruit);\n    type Child = (One::Child, Two::Child, Three::Child);\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        self.2.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        segment: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let one = self.0.for_segment(segment_local_id, segment)?;\n        let two = self.1.for_segment(segment_local_id, segment)?;\n        let three = self.2.for_segment(segment_local_id, segment)?;\n        Ok((one, two, three))\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring() || self.1.requires_scoring() || self.2.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        children: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit> {\n        let mut one_fruits = vec![];\n        let mut two_fruits = vec![];\n        let mut three_fruits = vec![];\n        for (one_fruit, two_fruit, three_fruit) in children {\n            one_fruits.push(one_fruit);\n            two_fruits.push(two_fruit);\n            three_fruits.push(three_fruit);\n        }\n        Ok((\n            self.0.merge_fruits(one_fruits)?,\n            self.1.merge_fruits(two_fruits)?,\n            self.2.merge_fruits(three_fruits)?,\n        ))\n    }\n}\n\nimpl<One, Two, Three> SegmentCollector for (One, Two, Three)\nwhere\n    One: SegmentCollector,\n    Two: SegmentCollector,\n    Three: SegmentCollector,\n{\n    type Fruit = (One::Fruit, Two::Fruit, Three::Fruit);\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        self.0.collect(doc, score);\n        self.1.collect(doc, score);\n        self.2.collect(doc, score);\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.0.collect_block(docs);\n        self.1.collect_block(docs);\n        self.2.collect_block(docs);\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        (self.0.harvest(), self.1.harvest(), self.2.harvest())\n    }\n}\n\n// 4-Tuple\n\nimpl<One, Two, Three, Four> Collector for (One, Two, Three, Four)\nwhere\n    One: Collector,\n    Two: Collector,\n    Three: Collector,\n    Four: Collector,\n{\n    type Fruit = (One::Fruit, Two::Fruit, Three::Fruit, Four::Fruit);\n    type Child = (One::Child, Two::Child, Three::Child, Four::Child);\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        self.2.check_schema(schema)?;\n        self.3.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        segment: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let one = self.0.for_segment(segment_local_id, segment)?;\n        let two = self.1.for_segment(segment_local_id, segment)?;\n        let three = self.2.for_segment(segment_local_id, segment)?;\n        let four = self.3.for_segment(segment_local_id, segment)?;\n        Ok((one, two, three, four))\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring()\n            || self.1.requires_scoring()\n            || self.2.requires_scoring()\n            || self.3.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        children: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Self::Fruit> {\n        let mut one_fruits = vec![];\n        let mut two_fruits = vec![];\n        let mut three_fruits = vec![];\n        let mut four_fruits = vec![];\n        for (one_fruit, two_fruit, three_fruit, four_fruit) in children {\n            one_fruits.push(one_fruit);\n            two_fruits.push(two_fruit);\n            three_fruits.push(three_fruit);\n            four_fruits.push(four_fruit);\n        }\n        Ok((\n            self.0.merge_fruits(one_fruits)?,\n            self.1.merge_fruits(two_fruits)?,\n            self.2.merge_fruits(three_fruits)?,\n            self.3.merge_fruits(four_fruits)?,\n        ))\n    }\n}\n\nimpl<One, Two, Three, Four> SegmentCollector for (One, Two, Three, Four)\nwhere\n    One: SegmentCollector,\n    Two: SegmentCollector,\n    Three: SegmentCollector,\n    Four: SegmentCollector,\n{\n    type Fruit = (One::Fruit, Two::Fruit, Three::Fruit, Four::Fruit);\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        self.0.collect(doc, score);\n        self.1.collect(doc, score);\n        self.2.collect(doc, score);\n        self.3.collect(doc, score);\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.0.collect_block(docs);\n        self.1.collect_block(docs);\n        self.2.collect_block(docs);\n        self.3.collect_block(docs);\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        (\n            self.0.harvest(),\n            self.1.harvest(),\n            self.2.harvest(),\n            self.3.harvest(),\n        )\n    }\n}\n\nimpl_downcast!(Fruit);\n\n#[cfg(test)]\npub(crate) mod tests;\n"
  },
  {
    "path": "src/collector/multi_collector.rs",
    "content": "use std::marker::PhantomData;\nuse std::ops::Deref;\n\nuse super::{Collector, SegmentCollector};\nuse crate::collector::Fruit;\nuse crate::schema::Schema;\nuse crate::{DocId, Score, SegmentOrdinal, SegmentReader, TantivyError};\n\n/// MultiFruit keeps Fruits from every nested Collector\npub struct MultiFruit {\n    sub_fruits: Vec<Option<Box<dyn Fruit>>>,\n}\n\npub struct CollectorWrapper<TCollector: Collector>(TCollector);\n\nimpl<TCollector: Collector> Collector for CollectorWrapper<TCollector> {\n    type Fruit = Box<dyn Fruit>;\n    type Child = Box<dyn BoxableSegmentCollector>;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: u32,\n        reader: &SegmentReader,\n    ) -> crate::Result<Box<dyn BoxableSegmentCollector>> {\n        let child = self.0.for_segment(segment_local_id, reader)?;\n        Ok(Box::new(SegmentCollectorWrapper(child)))\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring()\n    }\n\n    fn merge_fruits(\n        &self,\n        children: Vec<<Self::Child as SegmentCollector>::Fruit>,\n    ) -> crate::Result<Box<dyn Fruit>> {\n        let typed_fruit: Vec<<TCollector::Child as SegmentCollector>::Fruit> = children\n            .into_iter()\n            .map(|untyped_fruit| {\n                untyped_fruit\n                    .downcast::<<TCollector::Child as SegmentCollector>::Fruit>()\n                    .map(|boxed_but_typed| *boxed_but_typed)\n                    .map_err(|_| {\n                        TantivyError::InvalidArgument(\"Failed to cast child fruit.\".to_string())\n                    })\n            })\n            .collect::<crate::Result<_>>()?;\n        let merged_fruit = self.0.merge_fruits(typed_fruit)?;\n        Ok(Box::new(merged_fruit))\n    }\n}\n\nimpl SegmentCollector for Box<dyn BoxableSegmentCollector> {\n    type Fruit = Box<dyn Fruit>;\n\n    #[inline]\n    fn collect(&mut self, doc: u32, score: Score) {\n        self.as_mut().collect(doc, score);\n    }\n\n    #[inline]\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.as_mut().collect_block(docs);\n    }\n\n    fn harvest(self) -> Box<dyn Fruit> {\n        BoxableSegmentCollector::harvest_from_box(self)\n    }\n}\n\npub trait BoxableSegmentCollector {\n    fn collect(&mut self, doc: u32, score: Score);\n    fn collect_block(&mut self, docs: &[DocId]) {\n        for &doc in docs {\n            self.collect(doc, 0.0);\n        }\n    }\n    fn harvest_from_box(self: Box<Self>) -> Box<dyn Fruit>;\n}\n\npub struct SegmentCollectorWrapper<TSegmentCollector: SegmentCollector>(TSegmentCollector);\n\nimpl<TSegmentCollector: SegmentCollector> BoxableSegmentCollector\n    for SegmentCollectorWrapper<TSegmentCollector>\n{\n    #[inline]\n    fn collect(&mut self, doc: u32, score: Score) {\n        self.0.collect(doc, score);\n    }\n    #[inline]\n    fn collect_block(&mut self, docs: &[DocId]) {\n        self.0.collect_block(docs);\n    }\n\n    fn harvest_from_box(self: Box<Self>) -> Box<dyn Fruit> {\n        Box::new(self.0.harvest())\n    }\n}\n\n/// FruitHandle stores reference to the corresponding collector inside MultiCollector\npub struct FruitHandle<TFruit: Fruit> {\n    pos: usize,\n    _phantom: PhantomData<TFruit>,\n}\n\nimpl<TFruit: Fruit> FruitHandle<TFruit> {\n    /// Extract a typed fruit off a multifruit.\n    ///\n    /// This function involves downcasting and can panic if the multifruit was\n    /// created using faulty code.\n    pub fn extract(self, fruits: &mut MultiFruit) -> TFruit {\n        let boxed_fruit = fruits.sub_fruits[self.pos].take().expect(\"\");\n        *boxed_fruit\n            .downcast::<TFruit>()\n            .map_err(|_| ())\n            .expect(\"Failed to downcast collector fruit.\")\n    }\n}\n\n/// Multicollector makes it possible to collect on more than one collector.\n/// It should only be used for use cases where the Collector types is unknown\n/// at compile time.\n///\n/// If the type of the collectors is known, you can just group yours collectors\n/// in a tuple. See the\n/// [Combining several collectors section of the collector\n/// documentation](./index.html#combining-several-collectors).\n///\n/// ```rust\n/// use tantivy::collector::{Count, TopDocs, MultiCollector};\n/// use tantivy::query::QueryParser;\n/// use tantivy::schema::{Schema, TEXT};\n/// use tantivy::{doc, Index};\n///\n/// # fn main() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n/// let mut index_writer = index.writer(15_000_000)?;\n/// index_writer.add_document(doc!(title => \"The Name of the Wind\"))?;\n/// index_writer.add_document(doc!(title => \"The Diary of Muadib\"))?;\n/// index_writer.add_document(doc!(title => \"A Dairy Cow\"))?;\n/// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\"))?;\n/// index_writer.commit()?;\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n///\n/// let mut collectors = MultiCollector::new();\n/// let top_docs_handle = collectors.add_collector(TopDocs::with_limit(2).order_by_score());\n/// let count_handle = collectors.add_collector(Count);\n/// let query_parser = QueryParser::for_index(&index, vec![title]);\n/// let query = query_parser.parse_query(\"diary\").unwrap();\n/// let mut multi_fruit = searcher.search(&query, &collectors).unwrap();\n///\n/// let count = count_handle.extract(&mut multi_fruit);\n/// let top_docs = top_docs_handle.extract(&mut multi_fruit);\n///\n/// assert_eq!(count, 2);\n/// assert_eq!(top_docs.len(), 2);\n/// # Ok(())\n/// # }\n/// ```\n#[expect(clippy::type_complexity)]\n#[derive(Default)]\npub struct MultiCollector<'a> {\n    collector_wrappers: Vec<\n        Box<dyn Collector<Child = Box<dyn BoxableSegmentCollector>, Fruit = Box<dyn Fruit>> + 'a>,\n    >,\n}\n\nimpl<'a> MultiCollector<'a> {\n    /// Create a new `MultiCollector`\n    pub fn new() -> Self {\n        Default::default()\n    }\n\n    /// Add a new collector to our `MultiCollector`.\n    pub fn add_collector<'b: 'a, TCollector: Collector + 'b>(\n        &mut self,\n        collector: TCollector,\n    ) -> FruitHandle<TCollector::Fruit> {\n        let pos = self.collector_wrappers.len();\n        self.collector_wrappers\n            .push(Box::new(CollectorWrapper(collector)));\n        FruitHandle {\n            pos,\n            _phantom: PhantomData,\n        }\n    }\n}\n\nimpl Collector for MultiCollector<'_> {\n    type Fruit = MultiFruit;\n    type Child = MultiCollectorChild;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        for collector in &self.collector_wrappers {\n            collector.check_schema(schema)?;\n        }\n        Ok(())\n    }\n\n    fn for_segment(\n        &self,\n        segment_local_id: SegmentOrdinal,\n        segment: &SegmentReader,\n    ) -> crate::Result<MultiCollectorChild> {\n        let children = self\n            .collector_wrappers\n            .iter()\n            .map(|collector_wrapper| collector_wrapper.for_segment(segment_local_id, segment))\n            .collect::<crate::Result<Vec<_>>>()?;\n        Ok(MultiCollectorChild { children })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.collector_wrappers\n            .iter()\n            .map(Deref::deref)\n            .any(Collector::requires_scoring)\n    }\n\n    fn merge_fruits(&self, segments_multifruits: Vec<MultiFruit>) -> crate::Result<MultiFruit> {\n        let mut segment_fruits_list: Vec<Vec<Box<dyn Fruit>>> = (0..self.collector_wrappers.len())\n            .map(|_| Vec::with_capacity(segments_multifruits.len()))\n            .collect::<Vec<_>>();\n        for segment_multifruit in segments_multifruits {\n            for (idx, segment_fruit_opt) in segment_multifruit.sub_fruits.into_iter().enumerate() {\n                if let Some(segment_fruit) = segment_fruit_opt {\n                    segment_fruits_list[idx].push(segment_fruit);\n                }\n            }\n        }\n        let sub_fruits = self\n            .collector_wrappers\n            .iter()\n            .zip(segment_fruits_list)\n            .map(|(child_collector, segment_fruits)| {\n                Ok(Some(child_collector.merge_fruits(segment_fruits)?))\n            })\n            .collect::<crate::Result<_>>()?;\n        Ok(MultiFruit { sub_fruits })\n    }\n}\n\npub struct MultiCollectorChild {\n    children: Vec<Box<dyn BoxableSegmentCollector>>,\n}\n\nimpl SegmentCollector for MultiCollectorChild {\n    type Fruit = MultiFruit;\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        for child in &mut self.children {\n            child.collect(doc, score);\n        }\n    }\n\n    fn collect_block(&mut self, docs: &[DocId]) {\n        for child in &mut self.children {\n            child.collect_block(docs);\n        }\n    }\n\n    fn harvest(self) -> MultiFruit {\n        MultiFruit {\n            sub_fruits: self\n                .children\n                .into_iter()\n                .map(|child| Some(child.harvest()))\n                .collect(),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n    use crate::collector::{Count, TopDocs};\n    use crate::query::TermQuery;\n    use crate::schema::{IndexRecordOption, Schema, TEXT};\n    use crate::{Index, Term};\n\n    #[test]\n    fn test_multi_collector() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text=>\"abc\"))?;\n            index_writer.add_document(doc!(text=>\"abc abc abc\"))?;\n            index_writer.add_document(doc!(text=>\"abc abc\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(text=>\"\"))?;\n            index_writer.add_document(doc!(text=>\"abc abc abc abc\"))?;\n            index_writer.add_document(doc!(text=>\"abc\"))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let term = Term::from_field_text(text, \"abc\");\n        let query = TermQuery::new(term, IndexRecordOption::Basic);\n\n        let mut collectors = MultiCollector::new();\n        let topdocs_handler = collectors.add_collector(TopDocs::with_limit(2).order_by_score());\n        let count_handler = collectors.add_collector(Count);\n        let mut multifruits = searcher.search(&query, &collectors).unwrap();\n\n        assert_eq!(count_handler.extract(&mut multifruits), 5);\n        assert_eq!(topdocs_handler.extract(&mut multifruits).len(), 2);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/mod.rs",
    "content": "mod order;\nmod sort_by_bytes;\nmod sort_by_erased_type;\nmod sort_by_score;\nmod sort_by_static_fast_value;\nmod sort_by_string;\nmod sort_key_computer;\n\npub use order::*;\npub use sort_by_bytes::SortByBytes;\npub use sort_by_erased_type::SortByErasedType;\npub use sort_by_score::SortBySimilarityScore;\npub use sort_by_static_fast_value::SortByStaticFastValue;\npub use sort_by_string::SortByString;\npub use sort_key_computer::{SegmentSortKeyComputer, SortKeyComputer};\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    // By spec, regardless of whether ascending or descending order was requested, in presence of a\n    // tie, we sort by ascending doc id/doc address.\n    pub(crate) fn sort_hits<TSortKey: Ord, D: Ord>(\n        hits: &mut [ComparableDoc<TSortKey, D>],\n        order: Order,\n    ) {\n        if order.is_asc() {\n            hits.sort_by(|l, r| l.sort_key.cmp(&r.sort_key).then(l.doc.cmp(&r.doc)));\n        } else {\n            hits.sort_by(|l, r| {\n                l.sort_key\n                    .cmp(&r.sort_key)\n                    .reverse() // This is descending\n                    .then(l.doc.cmp(&r.doc))\n            });\n        }\n    }\n\n    use std::collections::HashMap;\n    use std::ops::Range;\n\n    use crate::collector::sort_key::{\n        SortByErasedType, SortBySimilarityScore, SortByStaticFastValue, SortByString,\n    };\n    use crate::collector::{ComparableDoc, DocSetCollector, TopDocs};\n    use crate::indexer::NoMergePolicy;\n    use crate::query::{AllQuery, QueryParser};\n    use crate::schema::{OwnedValue, Schema, FAST, TEXT};\n    use crate::{DocAddress, Document, Index, Order, Score, Searcher};\n\n    fn make_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let id = schema_builder.add_u64_field(\"id\", FAST);\n        let city = schema_builder.add_text_field(\"city\", TEXT | FAST);\n        let catchphrase = schema_builder.add_text_field(\"catchphrase\", TEXT);\n        let altitude = schema_builder.add_f64_field(\"altitude\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        fn create_segment(index: &Index, docs: Vec<impl Document>) -> crate::Result<()> {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            for doc in docs {\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n            Ok(())\n        }\n\n        create_segment(\n            &index,\n            vec![\n                doc!(\n                    id => 0_u64,\n                    city => \"austin\",\n                    catchphrase => \"Hills, Barbeque, Glow\",\n                    altitude => 149.0,\n                ),\n                doc!(\n                    id => 1_u64,\n                    city => \"greenville\",\n                    catchphrase => \"Grow, Glow, Glow\",\n                    altitude => 27.0,\n                ),\n            ],\n        )?;\n        create_segment(\n            &index,\n            vec![doc!(\n                id => 2_u64,\n                city => \"tokyo\",\n                catchphrase => \"Glow, Glow, Glow\",\n                altitude => 40.0,\n            )],\n        )?;\n        create_segment(\n            &index,\n            vec![doc!(\n                id => 3_u64,\n                catchphrase => \"No, No, No\",\n                altitude => 0.0,\n            )],\n        )?;\n        Ok(index)\n    }\n\n    // NOTE: You cannot determine the SegmentIds that will be generated for Segments\n    // ahead of time, so DocAddresses must be mapped back to a unique id for each Searcher.\n    fn id_mapping(searcher: &Searcher) -> HashMap<DocAddress, u64> {\n        searcher\n            .search(&AllQuery, &DocSetCollector)\n            .unwrap()\n            .into_iter()\n            .map(|doc_address| {\n                let column = searcher.segment_readers()[doc_address.segment_ord as usize]\n                    .fast_fields()\n                    .u64(\"id\")\n                    .unwrap();\n                (doc_address, column.first(doc_address.doc_id).unwrap())\n            })\n            .collect()\n    }\n\n    #[test]\n    fn test_order_by_string() -> crate::Result<()> {\n        let index = make_index()?;\n\n        #[track_caller]\n        fn assert_query(\n            index: &Index,\n            order: Order,\n            doc_range: Range<usize>,\n            expected: Vec<(Option<String>, u64)>,\n        ) -> crate::Result<()> {\n            let searcher = index.reader()?.searcher();\n            let ids = id_mapping(&searcher);\n\n            // Try as primitive.\n            let top_collector = TopDocs::for_doc_range(doc_range)\n                .order_by((SortByString::for_field(\"city\"), order));\n            let actual = searcher\n                .search(&AllQuery, &top_collector)?\n                .into_iter()\n                .map(|(sort_key_opt, doc)| (sort_key_opt, ids[&doc]))\n                .collect::<Vec<_>>();\n            assert_eq!(actual, expected);\n            Ok(())\n        }\n\n        assert_query(\n            &index,\n            Order::Asc,\n            0..4,\n            vec![\n                (Some(\"austin\".to_owned()), 0),\n                (Some(\"greenville\".to_owned()), 1),\n                (Some(\"tokyo\".to_owned()), 2),\n                (None, 3),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Asc,\n            0..3,\n            vec![\n                (Some(\"austin\".to_owned()), 0),\n                (Some(\"greenville\".to_owned()), 1),\n                (Some(\"tokyo\".to_owned()), 2),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Asc,\n            0..2,\n            vec![\n                (Some(\"austin\".to_owned()), 0),\n                (Some(\"greenville\".to_owned()), 1),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Asc,\n            0..1,\n            vec![(Some(\"austin\".to_string()), 0)],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Asc,\n            1..3,\n            vec![\n                (Some(\"greenville\".to_owned()), 1),\n                (Some(\"tokyo\".to_owned()), 2),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Desc,\n            0..4,\n            vec![\n                (Some(\"tokyo\".to_owned()), 2),\n                (Some(\"greenville\".to_owned()), 1),\n                (Some(\"austin\".to_owned()), 0),\n                (None, 3),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Desc,\n            1..3,\n            vec![\n                (Some(\"greenville\".to_owned()), 1),\n                (Some(\"austin\".to_owned()), 0),\n            ],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Desc,\n            0..1,\n            vec![(Some(\"tokyo\".to_owned()), 2)],\n        )?;\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_order_by_f64() -> crate::Result<()> {\n        let index = make_index()?;\n\n        fn assert_query(\n            index: &Index,\n            order: Order,\n            expected: Vec<(Option<f64>, u64)>,\n        ) -> crate::Result<()> {\n            let searcher = index.reader()?.searcher();\n            let ids = id_mapping(&searcher);\n\n            // Try as primitive.\n            let top_collector = TopDocs::with_limit(3)\n                .order_by((SortByStaticFastValue::<f64>::for_field(\"altitude\"), order));\n            let actual = searcher\n                .search(&AllQuery, &top_collector)?\n                .into_iter()\n                .map(|(altitude_opt, doc)| (altitude_opt, ids[&doc]))\n                .collect::<Vec<_>>();\n            assert_eq!(actual, expected);\n\n            Ok(())\n        }\n\n        assert_query(\n            &index,\n            Order::Asc,\n            vec![(Some(0.0), 3), (Some(27.0), 1), (Some(40.0), 2)],\n        )?;\n\n        assert_query(\n            &index,\n            Order::Desc,\n            vec![(Some(149.0), 0), (Some(40.0), 2), (Some(27.0), 1)],\n        )?;\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_order_by_score() -> crate::Result<()> {\n        let index = make_index()?;\n\n        fn query(index: &Index, order: Order) -> crate::Result<Vec<(Score, u64)>> {\n            let searcher = index.reader()?.searcher();\n            let ids = id_mapping(&searcher);\n\n            let top_collector = TopDocs::with_limit(4).order_by((SortBySimilarityScore, order));\n            let field = index.schema().get_field(\"catchphrase\").unwrap();\n            let query_parser = QueryParser::for_index(index, vec![field]);\n            let text_query = query_parser.parse_query(\"glow\")?;\n\n            Ok(searcher\n                .search(&text_query, &top_collector)?\n                .into_iter()\n                .map(|(score, doc)| (score, ids[&doc]))\n                .collect())\n        }\n\n        assert_eq!(\n            &query(&index, Order::Desc)?,\n            &[(0.5604893, 2), (0.4904281, 1), (0.35667497, 0),]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc)?,\n            &[(0.35667497, 0), (0.4904281, 1), (0.5604893, 2),]\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_order_by_score_then_string() -> crate::Result<()> {\n        let index = make_index()?;\n\n        type SortKey = (Score, Option<String>);\n\n        fn query(\n            index: &Index,\n            score_order: Order,\n            city_order: Order,\n        ) -> crate::Result<Vec<(SortKey, u64)>> {\n            let searcher = index.reader()?.searcher();\n            let ids = id_mapping(&searcher);\n\n            let top_collector = TopDocs::with_limit(4).order_by((\n                (SortBySimilarityScore, score_order),\n                (SortByString::for_field(\"city\"), city_order),\n            ));\n            let results: Vec<((Score, Option<String>), DocAddress)> =\n                searcher.search(&AllQuery, &top_collector)?;\n            Ok(results.into_iter().map(|(f, doc)| (f, ids[&doc])).collect())\n        }\n\n        assert_eq!(\n            &query(&index, Order::Asc, Order::Asc)?,\n            &[\n                ((1.0, Some(\"austin\".to_owned())), 0),\n                ((1.0, Some(\"greenville\".to_owned())), 1),\n                ((1.0, Some(\"tokyo\".to_owned())), 2),\n                ((1.0, None), 3),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc, Order::Desc)?,\n            &[\n                ((1.0, Some(\"tokyo\".to_owned())), 2),\n                ((1.0, Some(\"greenville\".to_owned())), 1),\n                ((1.0, Some(\"austin\".to_owned())), 0),\n                ((1.0, None), 3),\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_order_by_score_then_owned_value() -> crate::Result<()> {\n        let index = make_index()?;\n\n        type SortKey = (Score, OwnedValue);\n\n        fn query(\n            index: &Index,\n            score_order: Order,\n            city_order: Order,\n        ) -> crate::Result<Vec<(SortKey, u64)>> {\n            let searcher = index.reader()?.searcher();\n            let ids = id_mapping(&searcher);\n\n            let top_collector = TopDocs::with_limit(4).order_by::<(Score, OwnedValue)>((\n                (SortBySimilarityScore, score_order),\n                (SortByErasedType::for_field(\"city\"), city_order),\n            ));\n            let results: Vec<((Score, OwnedValue), DocAddress)> =\n                searcher.search(&AllQuery, &top_collector)?;\n            Ok(results.into_iter().map(|(f, doc)| (f, ids[&doc])).collect())\n        }\n\n        assert_eq!(\n            &query(&index, Order::Asc, Order::Asc)?,\n            &[\n                ((1.0, OwnedValue::Str(\"austin\".to_owned())), 0),\n                ((1.0, OwnedValue::Str(\"greenville\".to_owned())), 1),\n                ((1.0, OwnedValue::Str(\"tokyo\".to_owned())), 2),\n                ((1.0, OwnedValue::Null), 3),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc, Order::Desc)?,\n            &[\n                ((1.0, OwnedValue::Str(\"tokyo\".to_owned())), 2),\n                ((1.0, OwnedValue::Str(\"greenville\".to_owned())), 1),\n                ((1.0, OwnedValue::Str(\"austin\".to_owned())), 0),\n                ((1.0, OwnedValue::Null), 3),\n            ]\n        );\n        Ok(())\n    }\n\n    use proptest::prelude::*;\n\n    proptest! {\n    #[test]\n    fn test_order_by_string_prop(\n          order in prop_oneof!(Just(Order::Desc), Just(Order::Asc)),\n          limit in 1..64_usize,\n          offset in 0..64_usize,\n          segments_terms in\n            proptest::collection::vec(\n                proptest::collection::vec(0..32_u8, 1..32_usize),\n                0..8_usize,\n            )\n        ) {\n            let mut schema_builder = Schema::builder();\n            let city = schema_builder.add_text_field(\"city\", TEXT | FAST);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            let mut index_writer = index.writer_for_tests()?;\n\n            // A Vec<Vec<u8>>, where the outer Vec represents segments, and the inner Vec\n            // represents terms.\n            for segment_terms in segments_terms.into_iter() {\n                for term in segment_terms.into_iter() {\n                    let term = format!(\"{term:0>3}\");\n                    index_writer.add_document(doc!(\n                        city => term,\n                    ))?;\n                }\n                index_writer.commit()?;\n            }\n\n            let searcher = index.reader()?.searcher();\n            let top_n_results = searcher.search(&AllQuery, &TopDocs::with_limit(limit)\n                .and_offset(offset)\n                .order_by_string_fast_field(\"city\", order))?;\n            let all_results = searcher.search(&AllQuery, &DocSetCollector)?.into_iter().map(|doc_address| {\n                // Get the term for this address.\n                let column = searcher.segment_readers()[doc_address.segment_ord as usize].fast_fields().str(\"city\").unwrap().unwrap();\n                let value = column.term_ords(doc_address.doc_id).next().map(|term_ord| {\n                    let mut city = Vec::new();\n                    column.dictionary().ord_to_term(term_ord, &mut city).unwrap();\n                    String::try_from(city).unwrap()\n                });\n                (value, doc_address)\n            });\n\n            // Using the TopDocs collector should always be equivalent to sorting, skipping the\n            // offset, and then taking the limit.\n            let sorted_docs: Vec<_> = {\n                let mut comparable_docs: Vec<ComparableDoc<_, _>> =\n                    all_results.into_iter().map(|(sort_key, doc)| ComparableDoc { sort_key, doc}).collect();\n                sort_hits(&mut comparable_docs, order);\n                comparable_docs.into_iter().map(|cd| (cd.sort_key, cd.doc)).collect()\n            };\n            let expected_docs = sorted_docs.into_iter().skip(offset).take(limit).collect::<Vec<_>>();\n            prop_assert_eq!(\n                expected_docs,\n                top_n_results\n            );\n        }\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/order.rs",
    "content": "use std::cmp::Ordering;\n\nuse columnar::MonotonicallyMappableToU64;\nuse serde::{Deserialize, Serialize};\n\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::schema::{OwnedValue, Schema};\nuse crate::{DocId, Order, Score};\n\nfn compare_owned_value<const NULLS_FIRST: bool>(lhs: &OwnedValue, rhs: &OwnedValue) -> Ordering {\n    match (lhs, rhs) {\n        (OwnedValue::Null, OwnedValue::Null) => Ordering::Equal,\n        (OwnedValue::Null, _) => {\n            if NULLS_FIRST {\n                Ordering::Less\n            } else {\n                Ordering::Greater\n            }\n        }\n        (_, OwnedValue::Null) => {\n            if NULLS_FIRST {\n                Ordering::Greater\n            } else {\n                Ordering::Less\n            }\n        }\n        (OwnedValue::Str(a), OwnedValue::Str(b)) => a.cmp(b),\n        (OwnedValue::PreTokStr(a), OwnedValue::PreTokStr(b)) => a.cmp(b),\n        (OwnedValue::U64(a), OwnedValue::U64(b)) => a.cmp(b),\n        (OwnedValue::I64(a), OwnedValue::I64(b)) => a.cmp(b),\n        (OwnedValue::F64(a), OwnedValue::F64(b)) => a.to_u64().cmp(&b.to_u64()),\n        (OwnedValue::Bool(a), OwnedValue::Bool(b)) => a.cmp(b),\n        (OwnedValue::Date(a), OwnedValue::Date(b)) => a.cmp(b),\n        (OwnedValue::Facet(a), OwnedValue::Facet(b)) => a.cmp(b),\n        (OwnedValue::Bytes(a), OwnedValue::Bytes(b)) => a.cmp(b),\n        (OwnedValue::IpAddr(a), OwnedValue::IpAddr(b)) => a.cmp(b),\n        (OwnedValue::U64(a), OwnedValue::I64(b)) => {\n            if *b < 0 {\n                Ordering::Greater\n            } else {\n                a.cmp(&(*b as u64))\n            }\n        }\n        (OwnedValue::I64(a), OwnedValue::U64(b)) => {\n            if *a < 0 {\n                Ordering::Less\n            } else {\n                (*a as u64).cmp(b)\n            }\n        }\n        (OwnedValue::U64(a), OwnedValue::F64(b)) => (*a as f64).to_u64().cmp(&b.to_u64()),\n        (OwnedValue::F64(a), OwnedValue::U64(b)) => a.to_u64().cmp(&(*b as f64).to_u64()),\n        (OwnedValue::I64(a), OwnedValue::F64(b)) => (*a as f64).to_u64().cmp(&b.to_u64()),\n        (OwnedValue::F64(a), OwnedValue::I64(b)) => a.to_u64().cmp(&(*b as f64).to_u64()),\n        (a, b) => {\n            let ord = a.discriminant_value().cmp(&b.discriminant_value());\n            // If the discriminant is equal, it's because a new type was added, but hasn't been\n            // included in this `match` statement.\n            assert!(\n                ord != Ordering::Equal,\n                \"Unimplemented comparison for type of {a:?}, {b:?}\"\n            );\n            ord\n        }\n    }\n}\n\n/// Comparator trait defining the order in which documents should be ordered.\npub trait Comparator<T>: Send + Sync + std::fmt::Debug + Default {\n    /// Return the order between two values.\n    fn compare(&self, lhs: &T, rhs: &T) -> Ordering;\n}\n\n/// Compare values naturally (e.g. 1 < 2).\n///\n/// When used with `TopDocs`, which reverses the order, this results in a\n/// \"Descending\" sort (Greatest values first).\n///\n/// `None` (or Null for `OwnedValue`) values are considered to be smaller than any other value,\n/// and will therefore appear last in a descending sort (e.g. `[Some(20), Some(10), None]`).\n#[derive(Debug, Copy, Clone, Default, Serialize, Deserialize)]\npub struct NaturalComparator;\n\nimpl<T: PartialOrd> Comparator<T> for NaturalComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &T, rhs: &T) -> Ordering {\n        lhs.partial_cmp(rhs).unwrap_or(Ordering::Equal)\n    }\n}\n\n/// A (partial) implementation of comparison for OwnedValue.\n///\n/// Intended for use within columns of homogenous types, and so will panic for OwnedValues with\n/// mismatched types. The one exception is Null, for which we do define all comparisons.\nimpl Comparator<OwnedValue> for NaturalComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &OwnedValue, rhs: &OwnedValue) -> Ordering {\n        compare_owned_value::</* NULLS_FIRST= */ true>(lhs, rhs)\n    }\n}\n\n/// Compare values in reverse (e.g. 2 < 1).\n///\n/// When used with `TopDocs`, which reverses the order, this results in an\n/// \"Ascending\" sort (Smallest values first).\n///\n/// `None` is considered smaller than `Some` in the underlying comparator, but because the\n/// comparison is reversed, `None` is effectively treated as the lowest value in the resulting\n/// Ascending sort (e.g. `[None, Some(10), Some(20)]`).\n///\n/// The ReverseComparator does not necessarily imply that the sort order is reversed compared\n/// to the NaturalComparator. In presence of a tie on the sort key, documents will always be\n/// sorted by ascending `DocId`/`DocAddress` in TopN results, regardless of the sort key's order.\n#[derive(Debug, Copy, Clone, Default, Serialize, Deserialize)]\npub struct ReverseComparator;\n\nimpl<T> Comparator<T> for ReverseComparator\nwhere NaturalComparator: Comparator<T>\n{\n    #[inline(always)]\n    fn compare(&self, lhs: &T, rhs: &T) -> Ordering {\n        NaturalComparator.compare(rhs, lhs)\n    }\n}\n\n/// Compare values in reverse, but treating `None` as lower than `Some`.\n///\n/// When used with `TopDocs`, which reverses the order, this results in an\n/// \"Ascending\" sort (Smallest values first), but with `None` values appearing last\n/// (e.g. `[Some(10), Some(20), None]`).\n///\n/// This is usually what is wanted when sorting by a field in an ascending order.\n/// For instance, in an e-commerce website, if sorting by price ascending,\n/// the cheapest items would appear first, and items without a price would appear last.\n#[derive(Debug, Copy, Clone, Default)]\npub struct ReverseNoneIsLowerComparator;\n\nimpl<T> Comparator<Option<T>> for ReverseNoneIsLowerComparator\nwhere ReverseComparator: Comparator<T>\n{\n    #[inline(always)]\n    fn compare(&self, lhs_opt: &Option<T>, rhs_opt: &Option<T>) -> Ordering {\n        match (lhs_opt, rhs_opt) {\n            (None, None) => Ordering::Equal,\n            (None, Some(_)) => Ordering::Less,\n            (Some(_), None) => Ordering::Greater,\n            (Some(lhs), Some(rhs)) => ReverseComparator.compare(lhs, rhs),\n        }\n    }\n}\n\nimpl Comparator<u32> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &u32, rhs: &u32) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<u64> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &u64, rhs: &u64) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<f64> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &f64, rhs: &f64) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<f32> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &f32, rhs: &f32) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<i64> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &i64, rhs: &i64) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<String> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &String, rhs: &String) -> Ordering {\n        ReverseComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<OwnedValue> for ReverseNoneIsLowerComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &OwnedValue, rhs: &OwnedValue) -> Ordering {\n        compare_owned_value::</* NULLS_FIRST= */ false>(rhs, lhs)\n    }\n}\n\n/// Compare values naturally, but treating `None` as higher than `Some`.\n///\n/// When used with `TopDocs`, which reverses the order, this results in a\n/// \"Descending\" sort (Greatest values first), but with `None` values appearing first\n/// (e.g. `[None, Some(20), Some(10)]`).\n#[derive(Debug, Copy, Clone, Default, Serialize, Deserialize)]\npub struct NaturalNoneIsHigherComparator;\n\nimpl<T> Comparator<Option<T>> for NaturalNoneIsHigherComparator\nwhere NaturalComparator: Comparator<T>\n{\n    #[inline(always)]\n    fn compare(&self, lhs_opt: &Option<T>, rhs_opt: &Option<T>) -> Ordering {\n        match (lhs_opt, rhs_opt) {\n            (None, None) => Ordering::Equal,\n            (None, Some(_)) => Ordering::Greater,\n            (Some(_), None) => Ordering::Less,\n            (Some(lhs), Some(rhs)) => NaturalComparator.compare(lhs, rhs),\n        }\n    }\n}\n\nimpl Comparator<u32> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &u32, rhs: &u32) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<u64> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &u64, rhs: &u64) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<f64> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &f64, rhs: &f64) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<f32> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &f32, rhs: &f32) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<i64> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &i64, rhs: &i64) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<String> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &String, rhs: &String) -> Ordering {\n        NaturalComparator.compare(lhs, rhs)\n    }\n}\n\nimpl Comparator<OwnedValue> for NaturalNoneIsHigherComparator {\n    #[inline(always)]\n    fn compare(&self, lhs: &OwnedValue, rhs: &OwnedValue) -> Ordering {\n        compare_owned_value::</* NULLS_FIRST= */ false>(lhs, rhs)\n    }\n}\n\n/// An enum representing the different sort orders.\n#[derive(Debug, Clone, Copy, Eq, PartialEq, Default)]\npub enum ComparatorEnum {\n    /// Natural order (See [NaturalComparator])\n    #[default]\n    Natural,\n    /// Reverse order (See [ReverseComparator])\n    Reverse,\n    /// Reverse order by treating None as the lowest value. (See [ReverseNoneLowerComparator])\n    ReverseNoneLower,\n    /// Natural order but treating None as the highest value. (See [NaturalNoneIsHigherComparator])\n    NaturalNoneHigher,\n}\n\nimpl From<Order> for ComparatorEnum {\n    fn from(order: Order) -> Self {\n        match order {\n            Order::Asc => ComparatorEnum::ReverseNoneLower,\n            Order::Desc => ComparatorEnum::Natural,\n        }\n    }\n}\n\nimpl<T> Comparator<T> for ComparatorEnum\nwhere\n    ReverseNoneIsLowerComparator: Comparator<T>,\n    NaturalComparator: Comparator<T>,\n    ReverseComparator: Comparator<T>,\n    NaturalNoneIsHigherComparator: Comparator<T>,\n{\n    #[inline(always)]\n    fn compare(&self, lhs: &T, rhs: &T) -> Ordering {\n        match self {\n            ComparatorEnum::Natural => NaturalComparator.compare(lhs, rhs),\n            ComparatorEnum::Reverse => ReverseComparator.compare(lhs, rhs),\n            ComparatorEnum::ReverseNoneLower => ReverseNoneIsLowerComparator.compare(lhs, rhs),\n            ComparatorEnum::NaturalNoneHigher => NaturalNoneIsHigherComparator.compare(lhs, rhs),\n        }\n    }\n}\n\nimpl<Head, Tail, LeftComparator, RightComparator> Comparator<(Head, Tail)>\n    for (LeftComparator, RightComparator)\nwhere\n    LeftComparator: Comparator<Head>,\n    RightComparator: Comparator<Tail>,\n{\n    #[inline(always)]\n    fn compare(&self, lhs: &(Head, Tail), rhs: &(Head, Tail)) -> Ordering {\n        self.0\n            .compare(&lhs.0, &rhs.0)\n            .then_with(|| self.1.compare(&lhs.1, &rhs.1))\n    }\n}\n\nimpl<Type1, Type2, Type3, Comparator1, Comparator2, Comparator3> Comparator<(Type1, (Type2, Type3))>\n    for (Comparator1, Comparator2, Comparator3)\nwhere\n    Comparator1: Comparator<Type1>,\n    Comparator2: Comparator<Type2>,\n    Comparator3: Comparator<Type3>,\n{\n    #[inline(always)]\n    fn compare(&self, lhs: &(Type1, (Type2, Type3)), rhs: &(Type1, (Type2, Type3))) -> Ordering {\n        self.0\n            .compare(&lhs.0, &rhs.0)\n            .then_with(|| self.1.compare(&lhs.1 .0, &rhs.1 .0))\n            .then_with(|| self.2.compare(&lhs.1 .1, &rhs.1 .1))\n    }\n}\n\nimpl<Type1, Type2, Type3, Comparator1, Comparator2, Comparator3> Comparator<(Type1, Type2, Type3)>\n    for (Comparator1, Comparator2, Comparator3)\nwhere\n    Comparator1: Comparator<Type1>,\n    Comparator2: Comparator<Type2>,\n    Comparator3: Comparator<Type3>,\n{\n    #[inline(always)]\n    fn compare(&self, lhs: &(Type1, Type2, Type3), rhs: &(Type1, Type2, Type3)) -> Ordering {\n        self.0\n            .compare(&lhs.0, &rhs.0)\n            .then_with(|| self.1.compare(&lhs.1, &rhs.1))\n            .then_with(|| self.2.compare(&lhs.2, &rhs.2))\n    }\n}\n\nimpl<Type1, Type2, Type3, Type4, Comparator1, Comparator2, Comparator3, Comparator4>\n    Comparator<(Type1, (Type2, (Type3, Type4)))>\n    for (Comparator1, Comparator2, Comparator3, Comparator4)\nwhere\n    Comparator1: Comparator<Type1>,\n    Comparator2: Comparator<Type2>,\n    Comparator3: Comparator<Type3>,\n    Comparator4: Comparator<Type4>,\n{\n    #[inline(always)]\n    fn compare(\n        &self,\n        lhs: &(Type1, (Type2, (Type3, Type4))),\n        rhs: &(Type1, (Type2, (Type3, Type4))),\n    ) -> Ordering {\n        self.0\n            .compare(&lhs.0, &rhs.0)\n            .then_with(|| self.1.compare(&lhs.1 .0, &rhs.1 .0))\n            .then_with(|| self.2.compare(&lhs.1 .1 .0, &rhs.1 .1 .0))\n            .then_with(|| self.3.compare(&lhs.1 .1 .1, &rhs.1 .1 .1))\n    }\n}\n\nimpl<Type1, Type2, Type3, Type4, Comparator1, Comparator2, Comparator3, Comparator4>\n    Comparator<(Type1, Type2, Type3, Type4)>\n    for (Comparator1, Comparator2, Comparator3, Comparator4)\nwhere\n    Comparator1: Comparator<Type1>,\n    Comparator2: Comparator<Type2>,\n    Comparator3: Comparator<Type3>,\n    Comparator4: Comparator<Type4>,\n{\n    #[inline(always)]\n    fn compare(\n        &self,\n        lhs: &(Type1, Type2, Type3, Type4),\n        rhs: &(Type1, Type2, Type3, Type4),\n    ) -> Ordering {\n        self.0\n            .compare(&lhs.0, &rhs.0)\n            .then_with(|| self.1.compare(&lhs.1, &rhs.1))\n            .then_with(|| self.2.compare(&lhs.2, &rhs.2))\n            .then_with(|| self.3.compare(&lhs.3, &rhs.3))\n    }\n}\n\nimpl<TSortKeyComputer> SortKeyComputer for (TSortKeyComputer, ComparatorEnum)\nwhere\n    TSortKeyComputer: SortKeyComputer,\n    ComparatorEnum: Comparator<TSortKeyComputer::SortKey>,\n    ComparatorEnum: Comparator<\n        <<TSortKeyComputer as SortKeyComputer>::Child as SegmentSortKeyComputer>::SegmentSortKey,\n    >,\n{\n    type SortKey = TSortKeyComputer::SortKey;\n\n    type Child = SegmentSortKeyComputerWithComparator<TSortKeyComputer::Child, Self::Comparator>;\n\n    type Comparator = ComparatorEnum;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring()\n    }\n\n    fn comparator(&self) -> Self::Comparator {\n        self.1\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let child = self.0.segment_sort_key_computer(segment_reader)?;\n        Ok(SegmentSortKeyComputerWithComparator {\n            segment_sort_key_computer: child,\n            comparator: self.comparator(),\n        })\n    }\n}\n\nimpl<TSortKeyComputer> SortKeyComputer for (TSortKeyComputer, Order)\nwhere\n    TSortKeyComputer: SortKeyComputer,\n    ComparatorEnum: Comparator<TSortKeyComputer::SortKey>,\n    ComparatorEnum: Comparator<\n        <<TSortKeyComputer as SortKeyComputer>::Child as SegmentSortKeyComputer>::SegmentSortKey,\n    >,\n{\n    type SortKey = TSortKeyComputer::SortKey;\n\n    type Child = SegmentSortKeyComputerWithComparator<TSortKeyComputer::Child, Self::Comparator>;\n\n    type Comparator = ComparatorEnum;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring()\n    }\n\n    fn comparator(&self) -> Self::Comparator {\n        self.1.into()\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let child = self.0.segment_sort_key_computer(segment_reader)?;\n        Ok(SegmentSortKeyComputerWithComparator {\n            segment_sort_key_computer: child,\n            comparator: self.comparator(),\n        })\n    }\n}\n\n/// A segment sort key computer with a custom ordering.\npub struct SegmentSortKeyComputerWithComparator<TSegmentSortKeyComputer, TComparator> {\n    segment_sort_key_computer: TSegmentSortKeyComputer,\n    comparator: TComparator,\n}\n\nimpl<TSegmentSortKeyComputer, TSegmentSortKey, TComparator> SegmentSortKeyComputer\n    for SegmentSortKeyComputerWithComparator<TSegmentSortKeyComputer, TComparator>\nwhere\n    TSegmentSortKeyComputer: SegmentSortKeyComputer<SegmentSortKey = TSegmentSortKey>,\n    TSegmentSortKey: Clone + 'static + Sync + Send,\n    TComparator: Comparator<TSegmentSortKey> + 'static + Sync + Send,\n{\n    type SortKey = TSegmentSortKeyComputer::SortKey;\n    type SegmentSortKey = TSegmentSortKey;\n    type SegmentComparator = TComparator;\n\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Self::SegmentSortKey {\n        self.segment_sort_key_computer.segment_sort_key(doc, score)\n    }\n\n    #[inline(always)]\n    fn compare_segment_sort_key(\n        &self,\n        left: &Self::SegmentSortKey,\n        right: &Self::SegmentSortKey,\n    ) -> Ordering {\n        self.comparator.compare(left, right)\n    }\n\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        self.segment_sort_key_computer\n            .convert_segment_sort_key(sort_key)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::schema::OwnedValue;\n\n    #[test]\n    fn test_natural_none_is_higher() {\n        let comp = NaturalNoneIsHigherComparator;\n        let null = None;\n        let v1 = Some(1_u64);\n        let v2 = Some(2_u64);\n\n        // NaturalNoneIsGreaterComparator logic:\n        // 1. Delegates to NaturalComparator for non-nulls.\n        // NaturalComparator compare(2, 1) -> 2.cmp(1) -> Greater.\n        assert_eq!(comp.compare(&v2, &v1), Ordering::Greater);\n\n        // 2. Treats None (Null) as Greater than any value.\n        // compare(None, Some(2)) should be Greater.\n        assert_eq!(comp.compare(&null, &v2), Ordering::Greater);\n\n        // compare(Some(1), None) should be Less.\n        assert_eq!(comp.compare(&v1, &null), Ordering::Less);\n\n        // compare(None, None) should be Equal.\n        assert_eq!(comp.compare(&null, &null), Ordering::Equal);\n    }\n\n    #[test]\n    fn test_mixed_ownedvalue_compare() {\n        let u = OwnedValue::U64(10);\n        let i = OwnedValue::I64(10);\n        let f = OwnedValue::F64(10.0);\n\n        let nc = NaturalComparator;\n        assert_eq!(nc.compare(&u, &i), Ordering::Equal);\n        assert_eq!(nc.compare(&u, &f), Ordering::Equal);\n        assert_eq!(nc.compare(&i, &f), Ordering::Equal);\n\n        let u2 = OwnedValue::U64(11);\n        assert_eq!(nc.compare(&u2, &f), Ordering::Greater);\n\n        let s = OwnedValue::Str(\"a\".to_string());\n        // Str < U64\n        assert_eq!(nc.compare(&s, &u), Ordering::Less);\n        // Str < I64\n        assert_eq!(nc.compare(&s, &i), Ordering::Less);\n        // Str < F64\n        assert_eq!(nc.compare(&s, &f), Ordering::Less);\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_by_bytes.rs",
    "content": "use columnar::BytesColumn;\n\nuse crate::collector::sort_key::NaturalComparator;\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::termdict::TermOrdinal;\nuse crate::{DocId, Score};\n\n/// Sort by the first value of a bytes column.\n///\n/// If the field is multivalued, only the first value is considered.\n///\n/// Documents that do not have this value are still considered.\n/// Their sort key will simply be `None`.\n#[derive(Debug, Clone)]\npub struct SortByBytes {\n    column_name: String,\n}\n\nimpl SortByBytes {\n    /// Creates a new sort by bytes sort key computer.\n    pub fn for_field(column_name: impl ToString) -> Self {\n        SortByBytes {\n            column_name: column_name.to_string(),\n        }\n    }\n}\n\nimpl SortKeyComputer for SortByBytes {\n    type SortKey = Option<Vec<u8>>;\n    type Child = ByBytesColumnSegmentSortKeyComputer;\n    type Comparator = NaturalComparator;\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let bytes_column_opt = segment_reader.fast_fields().bytes(&self.column_name)?;\n        Ok(ByBytesColumnSegmentSortKeyComputer { bytes_column_opt })\n    }\n}\n\n/// Segment-level sort key computer for bytes columns.\npub struct ByBytesColumnSegmentSortKeyComputer {\n    bytes_column_opt: Option<BytesColumn>,\n}\n\nimpl SegmentSortKeyComputer for ByBytesColumnSegmentSortKeyComputer {\n    type SortKey = Option<Vec<u8>>;\n    type SegmentSortKey = Option<TermOrdinal>;\n    type SegmentComparator = NaturalComparator;\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, doc: DocId, _score: Score) -> Option<TermOrdinal> {\n        let bytes_column = self.bytes_column_opt.as_ref()?;\n        bytes_column.ords().first(doc)\n    }\n\n    fn convert_segment_sort_key(&self, term_ord_opt: Option<TermOrdinal>) -> Option<Vec<u8>> {\n        // TODO: Individual lookups to the dictionary like this are very likely to repeatedly\n        // decompress the same blocks. See https://github.com/quickwit-oss/tantivy/issues/2776\n        let term_ord = term_ord_opt?;\n        let bytes_column = self.bytes_column_opt.as_ref()?;\n        let mut bytes = Vec::new();\n        bytes_column\n            .dictionary()\n            .ord_to_term(term_ord, &mut bytes)\n            .ok()?;\n        Some(bytes)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::SortByBytes;\n    use crate::collector::TopDocs;\n    use crate::query::AllQuery;\n    use crate::schema::{BytesOptions, Schema, FAST, INDEXED};\n    use crate::{Index, IndexWriter, Order, TantivyDocument};\n\n    #[test]\n    fn test_sort_by_bytes_asc() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let bytes_field = schema_builder\n            .add_bytes_field(\"data\", BytesOptions::default().set_fast().set_indexed());\n        let id_field = schema_builder.add_u64_field(\"id\", FAST | INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // Insert documents with byte values in non-sorted order\n        let test_data: Vec<(u64, Vec<u8>)> = vec![\n            (1, vec![0x02, 0x00]),\n            (2, vec![0x00, 0x10]),\n            (3, vec![0x01, 0x00]),\n            (4, vec![0x00, 0x20]),\n        ];\n\n        for (id, bytes) in &test_data {\n            let mut doc = TantivyDocument::new();\n            doc.add_u64(id_field, *id);\n            doc.add_bytes(bytes_field, bytes);\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Sort ascending by bytes\n        let top_docs =\n            TopDocs::with_limit(10).order_by((SortByBytes::for_field(\"data\"), Order::Asc));\n        let results: Vec<(Option<Vec<u8>>, _)> = searcher.search(&AllQuery, &top_docs)?;\n\n        // Expected order: [0x00,0x10], [0x00,0x20], [0x01,0x00], [0x02,0x00]\n        let sorted_bytes: Vec<Option<Vec<u8>>> = results.into_iter().map(|(b, _)| b).collect();\n        assert_eq!(\n            sorted_bytes,\n            vec![\n                Some(vec![0x00, 0x10]),\n                Some(vec![0x00, 0x20]),\n                Some(vec![0x01, 0x00]),\n                Some(vec![0x02, 0x00]),\n            ]\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_sort_by_bytes_desc() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let bytes_field = schema_builder\n            .add_bytes_field(\"data\", BytesOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        let test_data: Vec<Vec<u8>> = vec![vec![0x00, 0x10], vec![0x02, 0x00], vec![0x01, 0x00]];\n\n        for bytes in &test_data {\n            let mut doc = TantivyDocument::new();\n            doc.add_bytes(bytes_field, bytes);\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Sort descending by bytes\n        let top_docs =\n            TopDocs::with_limit(10).order_by((SortByBytes::for_field(\"data\"), Order::Desc));\n        let results: Vec<(Option<Vec<u8>>, _)> = searcher.search(&AllQuery, &top_docs)?;\n\n        // Expected order (descending): [0x02,0x00], [0x01,0x00], [0x00,0x10]\n        let sorted_bytes: Vec<Option<Vec<u8>>> = results.into_iter().map(|(b, _)| b).collect();\n        assert_eq!(\n            sorted_bytes,\n            vec![\n                Some(vec![0x02, 0x00]),\n                Some(vec![0x01, 0x00]),\n                Some(vec![0x00, 0x10]),\n            ]\n        );\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_by_erased_type.rs",
    "content": "use columnar::{ColumnType, MonotonicallyMappableToU64};\n\nuse crate::collector::sort_key::{\n    NaturalComparator, SortByBytes, SortBySimilarityScore, SortByStaticFastValue, SortByString,\n};\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::fastfield::FastFieldNotAvailableError;\nuse crate::schema::OwnedValue;\nuse crate::{DateTime, DocId, Score};\n\n/// Sort by the boxed / OwnedValue representation of either a fast field, or of the score.\n///\n/// Using the OwnedValue representation allows for type erasure, and can be useful when sort orders\n/// are not known until runtime. But it comes with a performance cost: wherever possible, prefer to\n/// use a SortKeyComputer implementation with a known-type at compile time.\n#[derive(Debug, Clone)]\npub enum SortByErasedType {\n    /// Sort by a fast field\n    Field(String),\n    /// Sort by score\n    Score,\n}\n\nimpl SortByErasedType {\n    /// Creates a new sort key computer which will sort by the given fast field column, with type\n    /// erasure.\n    pub fn for_field(column_name: impl ToString) -> Self {\n        Self::Field(column_name.to_string())\n    }\n\n    /// Creates a new sort key computer which will sort by score, with type erasure.\n    pub fn for_score() -> Self {\n        Self::Score\n    }\n}\n\ntrait ErasedSegmentSortKeyComputer: Send + Sync {\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Option<u64>;\n    fn convert_segment_sort_key(&self, sort_key: Option<u64>) -> OwnedValue;\n}\n\nstruct ErasedSegmentSortKeyComputerWrapper<C, F> {\n    inner: C,\n    converter: F,\n}\n\nimpl<C, F> ErasedSegmentSortKeyComputer for ErasedSegmentSortKeyComputerWrapper<C, F>\nwhere\n    C: SegmentSortKeyComputer<SegmentSortKey = Option<u64>> + Send + Sync,\n    F: Fn(C::SortKey) -> OwnedValue + Send + Sync + 'static,\n{\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Option<u64> {\n        self.inner.segment_sort_key(doc, score)\n    }\n\n    fn convert_segment_sort_key(&self, sort_key: Option<u64>) -> OwnedValue {\n        let val = self.inner.convert_segment_sort_key(sort_key);\n        (self.converter)(val)\n    }\n}\n\nstruct ScoreSegmentSortKeyComputer {\n    segment_computer: SortBySimilarityScore,\n}\n\nimpl ErasedSegmentSortKeyComputer for ScoreSegmentSortKeyComputer {\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Option<u64> {\n        let score_value: f64 = self.segment_computer.segment_sort_key(doc, score).into();\n        Some(score_value.to_u64())\n    }\n\n    fn convert_segment_sort_key(&self, sort_key: Option<u64>) -> OwnedValue {\n        let score_value: u64 = sort_key.expect(\"This implementation always produces a score.\");\n        OwnedValue::F64(f64::from_u64(score_value))\n    }\n}\n\nimpl SortKeyComputer for SortByErasedType {\n    type SortKey = OwnedValue;\n    type Child = ErasedColumnSegmentSortKeyComputer;\n    type Comparator = NaturalComparator;\n\n    fn requires_scoring(&self) -> bool {\n        matches!(self, Self::Score)\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let inner: Box<dyn ErasedSegmentSortKeyComputer> = match self {\n            Self::Field(column_name) => {\n                let fast_fields = segment_reader.fast_fields();\n                // TODO: We currently double-open the column to avoid relying on the implementation\n                // details of `SortByString` or `SortByStaticFastValue`. Once\n                // https://github.com/quickwit-oss/tantivy/issues/2776 is resolved, we should\n                // consider directly constructing the appropriate `SegmentSortKeyComputer` type for\n                // the column that we open here.\n                let (_column, column_type) =\n                    fast_fields.u64_lenient(column_name)?.ok_or_else(|| {\n                        FastFieldNotAvailableError {\n                            field_name: column_name.to_owned(),\n                        }\n                    })?;\n\n                match column_type {\n                    ColumnType::Str => {\n                        let computer = SortByString::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<String>| {\n                                val.map(OwnedValue::Str).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::Bytes => {\n                        let computer = SortByBytes::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<Vec<u8>>| {\n                                val.map(OwnedValue::Bytes).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::U64 => {\n                        let computer = SortByStaticFastValue::<u64>::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<u64>| {\n                                val.map(OwnedValue::U64).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::I64 => {\n                        let computer = SortByStaticFastValue::<i64>::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<i64>| {\n                                val.map(OwnedValue::I64).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::F64 => {\n                        let computer = SortByStaticFastValue::<f64>::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<f64>| {\n                                val.map(OwnedValue::F64).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::Bool => {\n                        let computer = SortByStaticFastValue::<bool>::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<bool>| {\n                                val.map(OwnedValue::Bool).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    ColumnType::DateTime => {\n                        let computer = SortByStaticFastValue::<DateTime>::for_field(column_name);\n                        let inner = computer.segment_sort_key_computer(segment_reader)?;\n                        Box::new(ErasedSegmentSortKeyComputerWrapper {\n                            inner,\n                            converter: |val: Option<DateTime>| {\n                                val.map(OwnedValue::Date).unwrap_or(OwnedValue::Null)\n                            },\n                        })\n                    }\n                    column_type => {\n                        return Err(crate::TantivyError::SchemaError(format!(\n                            \"Field `{}` is of type {column_type:?}, which is not supported for \\\n                             sorting by owned value yet.\",\n                            column_name\n                        )))\n                    }\n                }\n            }\n            Self::Score => Box::new(ScoreSegmentSortKeyComputer {\n                segment_computer: SortBySimilarityScore,\n            }),\n        };\n        Ok(ErasedColumnSegmentSortKeyComputer { inner })\n    }\n}\n\npub struct ErasedColumnSegmentSortKeyComputer {\n    inner: Box<dyn ErasedSegmentSortKeyComputer>,\n}\n\nimpl SegmentSortKeyComputer for ErasedColumnSegmentSortKeyComputer {\n    type SortKey = OwnedValue;\n    type SegmentSortKey = Option<u64>;\n    type SegmentComparator = NaturalComparator;\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Option<u64> {\n        self.inner.segment_sort_key(doc, score)\n    }\n\n    fn convert_segment_sort_key(&self, segment_sort_key: Self::SegmentSortKey) -> OwnedValue {\n        self.inner.convert_segment_sort_key(segment_sort_key)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::collector::sort_key::{ComparatorEnum, SortByErasedType};\n    use crate::collector::TopDocs;\n    use crate::query::AllQuery;\n    use crate::schema::{OwnedValue, Schema, FAST, TEXT};\n    use crate::Index;\n\n    #[test]\n    fn test_sort_by_owned_u64() {\n        let mut schema_builder = Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(id_field => 10u64)).unwrap();\n        writer.add_document(doc!(id_field => 2u64)).unwrap();\n        writer.add_document(doc!()).unwrap();\n        writer.commit().unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let collector = TopDocs::with_limit(10)\n            .order_by((SortByErasedType::for_field(\"id\"), ComparatorEnum::Natural));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![OwnedValue::U64(10), OwnedValue::U64(2), OwnedValue::Null]\n        );\n\n        let collector = TopDocs::with_limit(10).order_by((\n            SortByErasedType::for_field(\"id\"),\n            ComparatorEnum::ReverseNoneLower,\n        ));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![OwnedValue::U64(2), OwnedValue::U64(10), OwnedValue::Null]\n        );\n    }\n\n    #[test]\n    fn test_sort_by_owned_string() {\n        let mut schema_builder = Schema::builder();\n        let city_field = schema_builder.add_text_field(\"city\", FAST | TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(city_field => \"tokyo\")).unwrap();\n        writer.add_document(doc!(city_field => \"austin\")).unwrap();\n        writer.add_document(doc!()).unwrap();\n        writer.commit().unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let collector = TopDocs::with_limit(10).order_by((\n            SortByErasedType::for_field(\"city\"),\n            ComparatorEnum::ReverseNoneLower,\n        ));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![\n                OwnedValue::Str(\"austin\".to_string()),\n                OwnedValue::Str(\"tokyo\".to_string()),\n                OwnedValue::Null\n            ]\n        );\n    }\n\n    #[test]\n    fn test_sort_by_owned_bytes() {\n        let mut schema_builder = Schema::builder();\n        let data_field = schema_builder.add_bytes_field(\"data\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer\n            .add_document(doc!(data_field => vec![0x03u8, 0x00]))\n            .unwrap();\n        writer\n            .add_document(doc!(data_field => vec![0x01u8, 0x00]))\n            .unwrap();\n        writer\n            .add_document(doc!(data_field => vec![0x02u8, 0x00]))\n            .unwrap();\n        writer.add_document(doc!()).unwrap();\n        writer.commit().unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        // Sort descending (Natural - highest first)\n        let collector = TopDocs::with_limit(10)\n            .order_by((SortByErasedType::for_field(\"data\"), ComparatorEnum::Natural));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![\n                OwnedValue::Bytes(vec![0x03, 0x00]),\n                OwnedValue::Bytes(vec![0x02, 0x00]),\n                OwnedValue::Bytes(vec![0x01, 0x00]),\n                OwnedValue::Null\n            ]\n        );\n\n        // Sort ascending (ReverseNoneLower - lowest first, nulls last)\n        let collector = TopDocs::with_limit(10).order_by((\n            SortByErasedType::for_field(\"data\"),\n            ComparatorEnum::ReverseNoneLower,\n        ));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![\n                OwnedValue::Bytes(vec![0x01, 0x00]),\n                OwnedValue::Bytes(vec![0x02, 0x00]),\n                OwnedValue::Bytes(vec![0x03, 0x00]),\n                OwnedValue::Null\n            ]\n        );\n    }\n\n    #[test]\n    fn test_sort_by_owned_reverse() {\n        let mut schema_builder = Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(id_field => 10u64)).unwrap();\n        writer.add_document(doc!(id_field => 2u64)).unwrap();\n        writer.add_document(doc!()).unwrap();\n        writer.commit().unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let collector = TopDocs::with_limit(10)\n            .order_by((SortByErasedType::for_field(\"id\"), ComparatorEnum::Reverse));\n        let top_docs = searcher.search(&AllQuery, &collector).unwrap();\n\n        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();\n\n        assert_eq!(\n            values,\n            vec![OwnedValue::Null, OwnedValue::U64(2), OwnedValue::U64(10)]\n        );\n    }\n\n    #[test]\n    fn test_sort_by_owned_score() {\n        let mut schema_builder = Schema::builder();\n        let body_field = schema_builder.add_text_field(\"body\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(body_field => \"a a\")).unwrap();\n        writer.add_document(doc!(body_field => \"a\")).unwrap();\n        writer.commit().unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let query_parser = crate::query::QueryParser::for_index(&index, vec![body_field]);\n        let query = query_parser.parse_query(\"a\").unwrap();\n\n        // Sort by score descending (Natural)\n        let collector = TopDocs::with_limit(10)\n            .order_by((SortByErasedType::for_score(), ComparatorEnum::Natural));\n        let top_docs = searcher.search(&query, &collector).unwrap();\n\n        let values: Vec<f64> = top_docs\n            .into_iter()\n            .map(|(key, _)| match key {\n                OwnedValue::F64(val) => val,\n                _ => panic!(\"Wrong type {key:?}\"),\n            })\n            .collect();\n\n        assert_eq!(values.len(), 2);\n        assert!(values[0] > values[1]);\n\n        // Sort by score ascending (ReverseNoneLower)\n        let collector = TopDocs::with_limit(10).order_by((\n            SortByErasedType::for_score(),\n            ComparatorEnum::ReverseNoneLower,\n        ));\n        let top_docs = searcher.search(&query, &collector).unwrap();\n\n        let values: Vec<f64> = top_docs\n            .into_iter()\n            .map(|(key, _)| match key {\n                OwnedValue::F64(val) => val,\n                _ => panic!(\"Wrong type {key:?}\"),\n            })\n            .collect();\n\n        assert_eq!(values.len(), 2);\n        assert!(values[0] < values[1]);\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_by_score.rs",
    "content": "use crate::collector::sort_key::NaturalComparator;\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer, TopNComputer};\nuse crate::{DocAddress, DocId, Score};\n\n/// Sort by similarity score.\n#[derive(Clone, Debug, Copy)]\npub struct SortBySimilarityScore;\n\nimpl SortKeyComputer for SortBySimilarityScore {\n    type SortKey = Score;\n\n    type Child = SortBySimilarityScore;\n\n    type Comparator = NaturalComparator;\n\n    fn requires_scoring(&self) -> bool {\n        true\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        _segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        Ok(SortBySimilarityScore)\n    }\n\n    // Sorting by score is special in that it allows for the Block-Wand optimization.\n    fn collect_segment_top_k(\n        &self,\n        k: usize,\n        weight: &dyn crate::query::Weight,\n        reader: &crate::SegmentReader,\n        segment_ord: u32,\n    ) -> crate::Result<Vec<(Self::SortKey, DocAddress)>> {\n        let mut top_n: TopNComputer<Score, DocId, Self::Comparator> =\n            TopNComputer::new_with_comparator(k, self.comparator());\n\n        if let Some(alive_bitset) = reader.alive_bitset() {\n            let mut threshold = Score::MIN;\n            top_n.threshold = Some(threshold);\n            weight.for_each_pruning(Score::MIN, reader, &mut |doc, score| {\n                if alive_bitset.is_deleted(doc) {\n                    return threshold;\n                }\n                top_n.push(score, doc);\n                threshold = top_n.threshold.unwrap_or(Score::MIN);\n                threshold\n            })?;\n        } else {\n            weight.for_each_pruning(Score::MIN, reader, &mut |doc, score| {\n                top_n.push(score, doc);\n                top_n.threshold.unwrap_or(Score::MIN)\n            })?;\n        }\n\n        Ok(top_n\n            .into_vec()\n            .into_iter()\n            .map(|cid| (cid.sort_key, DocAddress::new(segment_ord, cid.doc)))\n            .collect())\n    }\n}\n\nimpl SegmentSortKeyComputer for SortBySimilarityScore {\n    type SortKey = Score;\n    type SegmentSortKey = Score;\n    type SegmentComparator = NaturalComparator;\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, _doc: DocId, score: Score) -> Score {\n        score\n    }\n\n    fn convert_segment_sort_key(&self, score: Score) -> Score {\n        score\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_by_static_fast_value.rs",
    "content": "use std::marker::PhantomData;\n\nuse columnar::Column;\n\nuse crate::collector::sort_key::NaturalComparator;\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::fastfield::{FastFieldNotAvailableError, FastValue};\nuse crate::{DocId, Score, SegmentReader};\n\n/// Sorts by a fast value (u64, i64, f64, bool).\n///\n/// The field must appear explicitly in the schema, with the right type, and declared as\n/// a fast field..\n///\n/// If the field is multivalued, only the first value is considered.\n///\n/// Documents that do not have this value are still considered.\n/// Their sort key will simply be `None`.\n#[derive(Debug, Clone)]\npub struct SortByStaticFastValue<T: FastValue> {\n    field: String,\n    typ: PhantomData<T>,\n}\n\nimpl<T: FastValue> SortByStaticFastValue<T> {\n    /// Creates a new `SortByStaticFastValue` instance for the given field.\n    pub fn for_field(column_name: impl ToString) -> SortByStaticFastValue<T> {\n        Self {\n            field: column_name.to_string(),\n            typ: PhantomData,\n        }\n    }\n}\n\nimpl<T: FastValue> SortKeyComputer for SortByStaticFastValue<T> {\n    type Child = SortByFastValueSegmentSortKeyComputer<T>;\n    type SortKey = Option<T>;\n    type Comparator = NaturalComparator;\n\n    fn check_schema(&self, schema: &crate::schema::Schema) -> crate::Result<()> {\n        // At the segment sort key computer level, we rely on the u64 representation.\n        // The mapping is monotonic, so it is sufficient to compute our top-K docs.\n        let field = schema.get_field(&self.field)?;\n        let field_entry = schema.get_field_entry(field);\n        if !field_entry.is_fast() {\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"Field `{}` is not a fast field.\",\n                self.field,\n            )));\n        }\n        let schema_type = field_entry.field_type().value_type();\n        if schema_type != T::to_type() {\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"Field `{}` is of type {schema_type:?}, not of the type {:?}.\",\n                &self.field,\n                T::to_type()\n            )));\n        }\n        Ok(())\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let sort_column_opt = segment_reader.fast_fields().u64_lenient(&self.field)?;\n        let (sort_column, _sort_column_type) =\n            sort_column_opt.ok_or_else(|| FastFieldNotAvailableError {\n                field_name: self.field.clone(),\n            })?;\n        Ok(SortByFastValueSegmentSortKeyComputer {\n            sort_column,\n            typ: PhantomData,\n        })\n    }\n}\n\npub struct SortByFastValueSegmentSortKeyComputer<T> {\n    sort_column: Column<u64>,\n    typ: PhantomData<T>,\n}\n\nimpl<T: FastValue> SegmentSortKeyComputer for SortByFastValueSegmentSortKeyComputer<T> {\n    type SortKey = Option<T>;\n    type SegmentSortKey = Option<u64>;\n    type SegmentComparator = NaturalComparator;\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, doc: DocId, _score: Score) -> Self::SegmentSortKey {\n        self.sort_column.first(doc)\n    }\n\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        sort_key.map(T::from_u64)\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_by_string.rs",
    "content": "use columnar::StrColumn;\n\nuse crate::collector::sort_key::NaturalComparator;\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::termdict::TermOrdinal;\nuse crate::{DocId, Score};\n\n/// Sort by the first value of a string column.\n///\n/// The string can be dynamic (coming from a json field)\n/// or static (being specificaly defined in the configuration).\n///\n/// If the field is multivalued, only the first value is considered.\n///\n/// Documents that do not have this value are still considered.\n/// Their sort key will simply be `None`.\n#[derive(Debug, Clone)]\npub struct SortByString {\n    column_name: String,\n}\n\nimpl SortByString {\n    /// Creates a new sort by string sort key computer.\n    pub fn for_field(column_name: impl ToString) -> Self {\n        SortByString {\n            column_name: column_name.to_string(),\n        }\n    }\n}\n\nimpl SortKeyComputer for SortByString {\n    type SortKey = Option<String>;\n    type Child = ByStringColumnSegmentSortKeyComputer;\n    type Comparator = NaturalComparator;\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &crate::SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        let str_column_opt = segment_reader.fast_fields().str(&self.column_name)?;\n        Ok(ByStringColumnSegmentSortKeyComputer { str_column_opt })\n    }\n}\n\npub struct ByStringColumnSegmentSortKeyComputer {\n    str_column_opt: Option<StrColumn>,\n}\n\nimpl SegmentSortKeyComputer for ByStringColumnSegmentSortKeyComputer {\n    type SortKey = Option<String>;\n    type SegmentSortKey = Option<TermOrdinal>;\n    type SegmentComparator = NaturalComparator;\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, doc: DocId, _score: Score) -> Option<TermOrdinal> {\n        let str_column = self.str_column_opt.as_ref()?;\n        str_column.ords().first(doc)\n    }\n\n    fn convert_segment_sort_key(&self, term_ord_opt: Option<TermOrdinal>) -> Option<String> {\n        // TODO: Individual lookups to the dictionary like this are very likely to repeatedly\n        // decompress the same blocks. See https://github.com/quickwit-oss/tantivy/issues/2776\n        let term_ord = term_ord_opt?;\n        let str_column = self.str_column_opt.as_ref()?;\n        let mut bytes = Vec::new();\n        str_column\n            .dictionary()\n            .ord_to_term(term_ord, &mut bytes)\n            .ok()?;\n        String::try_from(bytes).ok()\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key/sort_key_computer.rs",
    "content": "use std::cmp::Ordering;\n\nuse crate::collector::sort_key::{Comparator, NaturalComparator};\nuse crate::collector::sort_key_top_collector::TopBySortKeySegmentCollector;\nuse crate::collector::{default_collect_segment_impl, SegmentCollector as _, TopNComputer};\nuse crate::schema::Schema;\nuse crate::{DocAddress, DocId, Result, Score, SegmentReader};\n\n/// A `SegmentSortKeyComputer` makes it possible to modify the default score\n/// for a given document belonging to a specific segment.\n///\n/// It is the segment local version of the [`SortKeyComputer`].\npub trait SegmentSortKeyComputer: 'static {\n    /// The final score being emitted.\n    type SortKey: 'static + Send + Sync + Clone;\n\n    /// Sort key used by at the segment level by the `SegmentSortKeyComputer`.\n    ///\n    /// It is typically small like a `u64`, and is meant to be converted\n    /// to the final score at the end of the collection of the segment.\n    type SegmentSortKey: 'static + Clone + Send + Sync + Clone;\n\n    /// Comparator type.\n    type SegmentComparator: Comparator<Self::SegmentSortKey> + 'static;\n\n    /// Returns the segment sort key comparator.\n    fn segment_comparator(&self) -> Self::SegmentComparator {\n        Self::SegmentComparator::default()\n    }\n\n    /// Computes the sort key for the given document and score.\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Self::SegmentSortKey;\n\n    /// Computes the sort key and pushes the document in a TopN Computer.\n    ///\n    /// When using a tuple as the sorting key, the sort key is evaluated in a lazy manner.\n    #[inline(always)]\n    fn compute_sort_key_and_collect<C: Comparator<Self::SegmentSortKey>>(\n        &mut self,\n        doc: DocId,\n        score: Score,\n        top_n_computer: &mut TopNComputer<Self::SegmentSortKey, DocId, C>,\n    ) {\n        let sort_key = self.segment_sort_key(doc, score);\n        top_n_computer.push(sort_key, doc);\n    }\n\n    /// A SegmentSortKeyComputer maps to a SegmentSortKey, but it can also decide on\n    /// its ordering.\n    ///\n    /// This method must be consistent with the `SortKey` ordering.\n    #[inline(always)]\n    fn compare_segment_sort_key(\n        &self,\n        left: &Self::SegmentSortKey,\n        right: &Self::SegmentSortKey,\n    ) -> Ordering {\n        self.segment_comparator().compare(left, right)\n    }\n\n    /// Implementing this method makes it possible to avoid computing\n    /// a sort_key entirely if we can assess that it won't pass a threshold\n    /// with a partial computation.\n    ///\n    /// This is currently used for lexicographic sorting.\n    fn accept_sort_key_lazy(\n        &mut self,\n        doc_id: DocId,\n        score: Score,\n        threshold: &Self::SegmentSortKey,\n    ) -> Option<(Ordering, Self::SegmentSortKey)> {\n        let sort_key = self.segment_sort_key(doc_id, score);\n        let cmp = self.compare_segment_sort_key(&sort_key, threshold);\n        if cmp == Ordering::Less {\n            None\n        } else {\n            Some((cmp, sort_key))\n        }\n    }\n\n    /// Convert a segment level sort key into the global sort key.\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey;\n}\n\n/// `SortKeyComputer` defines the sort key to be used by a TopK Collector.\n///\n/// The `SortKeyComputer` itself does not make much of the computation itself.\n/// Instead, it helps constructing `Self::Child` instances that will compute\n/// the sort key at a segment scale.\npub trait SortKeyComputer: Sync {\n    /// The sort key type.\n    type SortKey: 'static + Send + Sync + Clone + std::fmt::Debug;\n    /// Type of the associated [`SegmentSortKeyComputer`].\n    type Child: SegmentSortKeyComputer<SortKey = Self::SortKey>;\n    /// Comparator type.\n    type Comparator: Comparator<Self::SortKey>\n        + Comparator<<Self::Child as SegmentSortKeyComputer>::SegmentSortKey>\n        + 'static;\n\n    /// Checks whether the schema is compatible with the sort key computer.\n    fn check_schema(&self, _schema: &Schema) -> crate::Result<()> {\n        Ok(())\n    }\n\n    /// Returns the sort key comparator.\n    fn comparator(&self) -> Self::Comparator {\n        Self::Comparator::default()\n    }\n\n    /// Indicates whether the sort key actually uses the similarity score (by default BM25).\n    /// If set to false, the similary score might not be computed (as an optimization),\n    /// and the score fed in the segment sort key computer could take any value.\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    /// Sorting by score has a overriding implementation for BM25 scores, using Block-WAND.\n    fn collect_segment_top_k(\n        &self,\n        k: usize,\n        weight: &dyn crate::query::Weight,\n        reader: &crate::SegmentReader,\n        segment_ord: u32,\n    ) -> crate::Result<Vec<(Self::SortKey, DocAddress)>> {\n        let with_scoring = self.requires_scoring();\n        let segment_sort_key_computer = self.segment_sort_key_computer(reader)?;\n        let topn_computer = TopNComputer::new_with_comparator(k, self.comparator());\n        let mut segment_top_key_collector = TopBySortKeySegmentCollector {\n            topn_computer,\n            segment_ord,\n            segment_sort_key_computer,\n        };\n        default_collect_segment_impl(&mut segment_top_key_collector, weight, reader, with_scoring)?;\n        Ok(segment_top_key_collector.harvest())\n    }\n\n    /// Builds a child sort key computer for a specific segment.\n    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child>;\n}\n\nimpl<HeadSortKeyComputer, TailSortKeyComputer> SortKeyComputer\n    for (HeadSortKeyComputer, TailSortKeyComputer)\nwhere\n    HeadSortKeyComputer: SortKeyComputer,\n    TailSortKeyComputer: SortKeyComputer,\n{\n    type SortKey = (HeadSortKeyComputer::SortKey, TailSortKeyComputer::SortKey);\n    type Child = (HeadSortKeyComputer::Child, TailSortKeyComputer::Child);\n\n    type Comparator = (\n        HeadSortKeyComputer::Comparator,\n        TailSortKeyComputer::Comparator,\n    );\n\n    fn comparator(&self) -> Self::Comparator {\n        (self.0.comparator(), self.1.comparator())\n    }\n\n    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {\n        Ok((\n            self.0.segment_sort_key_computer(segment_reader)?,\n            self.1.segment_sort_key_computer(segment_reader)?,\n        ))\n    }\n\n    /// Checks whether the schema is compatible with the sort key computer.\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        Ok(())\n    }\n\n    /// Indicates whether the sort key actually uses the similarity score (by default BM25).\n    /// If set to false, the similary score might not be computed (as an optimization),\n    /// and the score fed in the segment sort key computer could take any value.\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring() || self.1.requires_scoring()\n    }\n}\n\nimpl<HeadSegmentSortKeyComputer, TailSegmentSortKeyComputer> SegmentSortKeyComputer\n    for (HeadSegmentSortKeyComputer, TailSegmentSortKeyComputer)\nwhere\n    HeadSegmentSortKeyComputer: SegmentSortKeyComputer,\n    TailSegmentSortKeyComputer: SegmentSortKeyComputer,\n{\n    type SortKey = (\n        HeadSegmentSortKeyComputer::SortKey,\n        TailSegmentSortKeyComputer::SortKey,\n    );\n    type SegmentSortKey = (\n        HeadSegmentSortKeyComputer::SegmentSortKey,\n        TailSegmentSortKeyComputer::SegmentSortKey,\n    );\n\n    type SegmentComparator = (\n        HeadSegmentSortKeyComputer::SegmentComparator,\n        TailSegmentSortKeyComputer::SegmentComparator,\n    );\n\n    /// A SegmentSortKeyComputer maps to a SegmentSortKey, but it can also decide on\n    /// its ordering.\n    ///\n    /// By default, it uses the natural ordering.\n    #[inline]\n    fn compare_segment_sort_key(\n        &self,\n        left: &Self::SegmentSortKey,\n        right: &Self::SegmentSortKey,\n    ) -> Ordering {\n        self.0\n            .compare_segment_sort_key(&left.0, &right.0)\n            .then_with(|| self.1.compare_segment_sort_key(&left.1, &right.1))\n    }\n\n    #[inline(always)]\n    fn compute_sort_key_and_collect<C: Comparator<Self::SegmentSortKey>>(\n        &mut self,\n        doc: DocId,\n        score: Score,\n        top_n_computer: &mut TopNComputer<Self::SegmentSortKey, DocId, C>,\n    ) {\n        let sort_key: Self::SegmentSortKey;\n        if let Some(threshold) = &top_n_computer.threshold {\n            if let Some((_cmp, lazy_sort_key)) = self.accept_sort_key_lazy(doc, score, threshold) {\n                sort_key = lazy_sort_key;\n            } else {\n                return;\n            }\n        } else {\n            sort_key = self.segment_sort_key(doc, score);\n        };\n        top_n_computer.append_doc(doc, sort_key);\n    }\n\n    #[inline(always)]\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Self::SegmentSortKey {\n        let head_sort_key = self.0.segment_sort_key(doc, score);\n        let tail_sort_key = self.1.segment_sort_key(doc, score);\n        (head_sort_key, tail_sort_key)\n    }\n\n    fn accept_sort_key_lazy(\n        &mut self,\n        doc_id: DocId,\n        score: Score,\n        threshold: &Self::SegmentSortKey,\n    ) -> Option<(Ordering, Self::SegmentSortKey)> {\n        let (head_threshold, tail_threshold) = threshold;\n        let (head_cmp, head_sort_key) =\n            self.0.accept_sort_key_lazy(doc_id, score, head_threshold)?;\n        if head_cmp == Ordering::Equal {\n            let (tail_cmp, tail_sort_key) =\n                self.1.accept_sort_key_lazy(doc_id, score, tail_threshold)?;\n            Some((tail_cmp, (head_sort_key, tail_sort_key)))\n        } else {\n            let tail_sort_key = self.1.segment_sort_key(doc_id, score);\n            Some((head_cmp, (head_sort_key, tail_sort_key)))\n        }\n    }\n\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        let (head_sort_key, tail_sort_key) = sort_key;\n        (\n            self.0.convert_segment_sort_key(head_sort_key),\n            self.1.convert_segment_sort_key(tail_sort_key),\n        )\n    }\n}\n\n/// This struct is used as an adapter to take a sort key computer and map its score to another\n/// new sort key.\npub struct MappedSegmentSortKeyComputer<T, PreviousSortKey, NewSortKey> {\n    sort_key_computer: T,\n    map: fn(PreviousSortKey) -> NewSortKey,\n}\n\nimpl<T, PreviousScore, NewScore> SegmentSortKeyComputer\n    for MappedSegmentSortKeyComputer<T, PreviousScore, NewScore>\nwhere\n    T: SegmentSortKeyComputer<SortKey = PreviousScore>,\n    PreviousScore: 'static + Clone + Send + Sync,\n    NewScore: 'static + Clone + Send + Sync,\n{\n    type SortKey = NewScore;\n    type SegmentSortKey = T::SegmentSortKey;\n    type SegmentComparator = T::SegmentComparator;\n\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> Self::SegmentSortKey {\n        self.sort_key_computer.segment_sort_key(doc, score)\n    }\n\n    fn accept_sort_key_lazy(\n        &mut self,\n        doc_id: DocId,\n        score: Score,\n        threshold: &Self::SegmentSortKey,\n    ) -> Option<(Ordering, Self::SegmentSortKey)> {\n        self.sort_key_computer\n            .accept_sort_key_lazy(doc_id, score, threshold)\n    }\n\n    #[inline(always)]\n    fn compute_sort_key_and_collect<C: Comparator<Self::SegmentSortKey>>(\n        &mut self,\n        doc: DocId,\n        score: Score,\n        top_n_computer: &mut TopNComputer<Self::SegmentSortKey, DocId, C>,\n    ) {\n        self.sort_key_computer\n            .compute_sort_key_and_collect(doc, score, top_n_computer);\n    }\n\n    fn convert_segment_sort_key(&self, segment_sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        (self.map)(\n            self.sort_key_computer\n                .convert_segment_sort_key(segment_sort_key),\n        )\n    }\n}\n\n// We then re-use our (head, tail) implement and our mapper by seeing mapping any tuple (a, b, c,\n// ...) as the chain (a, (b, (c, ...)))\n\nimpl<SortKeyComputer1, SortKeyComputer2, SortKeyComputer3> SortKeyComputer\n    for (SortKeyComputer1, SortKeyComputer2, SortKeyComputer3)\nwhere\n    SortKeyComputer1: SortKeyComputer,\n    SortKeyComputer2: SortKeyComputer,\n    SortKeyComputer3: SortKeyComputer,\n{\n    type SortKey = (\n        SortKeyComputer1::SortKey,\n        SortKeyComputer2::SortKey,\n        SortKeyComputer3::SortKey,\n    );\n    type Child = MappedSegmentSortKeyComputer<\n        <(SortKeyComputer1, (SortKeyComputer2, SortKeyComputer3)) as SortKeyComputer>::Child,\n        (\n            SortKeyComputer1::SortKey,\n            (SortKeyComputer2::SortKey, SortKeyComputer3::SortKey),\n        ),\n        Self::SortKey,\n    >;\n\n    type Comparator = (\n        SortKeyComputer1::Comparator,\n        SortKeyComputer2::Comparator,\n        SortKeyComputer3::Comparator,\n    );\n\n    fn comparator(&self) -> Self::Comparator {\n        (\n            self.0.comparator(),\n            self.1.comparator(),\n            self.2.comparator(),\n        )\n    }\n\n    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {\n        let sort_key_computer1 = self.0.segment_sort_key_computer(segment_reader)?;\n        let sort_key_computer2 = self.1.segment_sort_key_computer(segment_reader)?;\n        let sort_key_computer3 = self.2.segment_sort_key_computer(segment_reader)?;\n        let map = |(sort_key1, (sort_key2, sort_key3))| (sort_key1, sort_key2, sort_key3);\n        Ok(MappedSegmentSortKeyComputer {\n            sort_key_computer: (sort_key_computer1, (sort_key_computer2, sort_key_computer3)),\n            map,\n        })\n    }\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        self.2.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring() || self.1.requires_scoring() || self.2.requires_scoring()\n    }\n}\n\nimpl<SortKeyComputer1, SortKeyComputer2, SortKeyComputer3, SortKeyComputer4> SortKeyComputer\n    for (\n        SortKeyComputer1,\n        SortKeyComputer2,\n        SortKeyComputer3,\n        SortKeyComputer4,\n    )\nwhere\n    SortKeyComputer1: SortKeyComputer,\n    SortKeyComputer2: SortKeyComputer,\n    SortKeyComputer3: SortKeyComputer,\n    SortKeyComputer4: SortKeyComputer,\n{\n    type Child = MappedSegmentSortKeyComputer<\n        <(\n            SortKeyComputer1,\n            (SortKeyComputer2, (SortKeyComputer3, SortKeyComputer4)),\n        ) as SortKeyComputer>::Child,\n        (\n            SortKeyComputer1::SortKey,\n            (\n                SortKeyComputer2::SortKey,\n                (SortKeyComputer3::SortKey, SortKeyComputer4::SortKey),\n            ),\n        ),\n        Self::SortKey,\n    >;\n    type SortKey = (\n        SortKeyComputer1::SortKey,\n        SortKeyComputer2::SortKey,\n        SortKeyComputer3::SortKey,\n        SortKeyComputer4::SortKey,\n    );\n    type Comparator = (\n        SortKeyComputer1::Comparator,\n        SortKeyComputer2::Comparator,\n        SortKeyComputer3::Comparator,\n        SortKeyComputer4::Comparator,\n    );\n\n    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {\n        let sort_key_computer1 = self.0.segment_sort_key_computer(segment_reader)?;\n        let sort_key_computer2 = self.1.segment_sort_key_computer(segment_reader)?;\n        let sort_key_computer3 = self.2.segment_sort_key_computer(segment_reader)?;\n        let sort_key_computer4 = self.3.segment_sort_key_computer(segment_reader)?;\n        Ok(MappedSegmentSortKeyComputer {\n            sort_key_computer: (\n                sort_key_computer1,\n                (sort_key_computer2, (sort_key_computer3, sort_key_computer4)),\n            ),\n            map: |(sort_key1, (sort_key2, (sort_key3, sort_key4)))| {\n                (sort_key1, sort_key2, sort_key3, sort_key4)\n            },\n        })\n    }\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.0.check_schema(schema)?;\n        self.1.check_schema(schema)?;\n        self.2.check_schema(schema)?;\n        self.3.check_schema(schema)?;\n        Ok(())\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.0.requires_scoring()\n            || self.1.requires_scoring()\n            || self.2.requires_scoring()\n            || self.3.requires_scoring()\n    }\n}\n\nimpl<F, SegmentF, TSortKey> SortKeyComputer for F\nwhere\n    F: 'static + Send + Sync + Fn(&SegmentReader) -> SegmentF,\n    SegmentF: 'static + FnMut(DocId) -> TSortKey,\n    TSortKey: 'static + PartialOrd + Clone + Send + Sync + std::fmt::Debug,\n{\n    type SortKey = TSortKey;\n    type Child = SegmentF;\n    type Comparator = NaturalComparator;\n\n    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {\n        Ok((self)(segment_reader))\n    }\n}\n\nimpl<F, TSortKey> SegmentSortKeyComputer for F\nwhere\n    F: 'static + FnMut(DocId) -> TSortKey,\n    TSortKey: 'static + PartialOrd + Clone + Send + Sync,\n{\n    type SortKey = TSortKey;\n    type SegmentSortKey = TSortKey;\n    type SegmentComparator = NaturalComparator;\n\n    fn segment_sort_key(&mut self, doc: DocId, _score: Score) -> TSortKey {\n        (self)(doc)\n    }\n\n    /// Convert a segment level score into the global level score.\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        sort_key\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::cmp::Ordering;\n    use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};\n    use std::sync::Arc;\n\n    use crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\n    use crate::schema::Schema;\n    use crate::{DocId, Index, Order, SegmentReader};\n\n    fn build_test_index() -> Index {\n        let schema = Schema::builder().build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(crate::TantivyDocument::default())\n            .unwrap();\n        index_writer.commit().unwrap();\n        index\n    }\n\n    #[test]\n    fn test_lazy_score_computer() {\n        let score_computer_primary = |_segment_reader: &SegmentReader| |_doc: DocId| 200u32;\n        let call_count = Arc::new(AtomicUsize::new(0));\n        let call_count_clone = call_count.clone();\n        let score_computer_secondary = move |_segment_reader: &SegmentReader| {\n            let call_count_new_clone = call_count_clone.clone();\n            move |_doc: DocId| {\n                call_count_new_clone.fetch_add(1, AtomicOrdering::SeqCst);\n                \"b\"\n            }\n        };\n        let lazy_score_computer = (score_computer_primary, score_computer_secondary);\n        let index = build_test_index();\n        let searcher = index.reader().unwrap().searcher();\n        let mut segment_sort_key_computer = lazy_score_computer\n            .segment_sort_key_computer(searcher.segment_reader(0))\n            .unwrap();\n        let expected_sort_key = (200, \"b\");\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(100u32, \"a\"));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 1);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(100u32, \"c\"));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 2);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(200u32, \"a\"));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 3);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(200u32, \"c\"));\n            assert!(sort_key_opt.is_none());\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(300u32, \"a\"));\n            assert_eq!(sort_key_opt, None);\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(300u32, \"c\"));\n            assert_eq!(sort_key_opt, None);\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &expected_sort_key);\n            assert_eq!(sort_key_opt, Some((Ordering::Equal, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 5);\n        }\n    }\n\n    #[test]\n    fn test_lazy_score_computer_dynamic_ordering() {\n        let score_computer_primary = |_segment_reader: &SegmentReader| |_doc: DocId| 200u32;\n        let call_count = Arc::new(AtomicUsize::new(0));\n        let call_count_clone = call_count.clone();\n        let score_computer_secondary = move |_segment_reader: &SegmentReader| {\n            let call_count_new_clone = call_count_clone.clone();\n            move |_doc: DocId| {\n                call_count_new_clone.fetch_add(1, AtomicOrdering::SeqCst);\n                2u32\n            }\n        };\n        let lazy_score_computer = (\n            (score_computer_primary, Order::Desc),\n            (score_computer_secondary, Order::Asc),\n        );\n        let index = build_test_index();\n        let searcher = index.reader().unwrap().searcher();\n        let mut segment_sort_key_computer = lazy_score_computer\n            .segment_sort_key_computer(searcher.segment_reader(0))\n            .unwrap();\n        let expected_sort_key = (200, 2u32);\n\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(100u32, 1u32));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 1);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(100u32, 3u32));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 2);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(200u32, 1u32));\n            assert!(sort_key_opt.is_none());\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 3);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(200u32, 3u32));\n            assert_eq!(sort_key_opt, Some((Ordering::Greater, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(300u32, 1u32));\n            assert_eq!(sort_key_opt, None);\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &(300u32, 3u32));\n            assert_eq!(sort_key_opt, None);\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 4);\n        }\n        {\n            let sort_key_opt =\n                segment_sort_key_computer.accept_sort_key_lazy(0u32, 1f32, &expected_sort_key);\n            assert_eq!(sort_key_opt, Some((Ordering::Equal, expected_sort_key)));\n            assert_eq!(call_count.load(AtomicOrdering::SeqCst), 5);\n        }\n        assert_eq!(\n            segment_sort_key_computer.convert_segment_sort_key(expected_sort_key),\n            (200u32, 2u32)\n        );\n    }\n}\n"
  },
  {
    "path": "src/collector/sort_key_top_collector.rs",
    "content": "use std::ops::Range;\n\nuse crate::collector::sort_key::{Comparator, SegmentSortKeyComputer, SortKeyComputer};\nuse crate::collector::{Collector, SegmentCollector, TopNComputer};\nuse crate::query::Weight;\nuse crate::schema::Schema;\nuse crate::{DocAddress, DocId, Result, Score, SegmentReader};\n\npub(crate) struct TopBySortKeyCollector<TSortKeyComputer> {\n    sort_key_computer: TSortKeyComputer,\n    doc_range: Range<usize>,\n}\n\nimpl<TSortKeyComputer> TopBySortKeyCollector<TSortKeyComputer> {\n    pub fn new(sort_key_computer: TSortKeyComputer, doc_range: Range<usize>) -> Self {\n        TopBySortKeyCollector {\n            sort_key_computer,\n            doc_range,\n        }\n    }\n}\n\nimpl<TSortKeyComputer> Collector for TopBySortKeyCollector<TSortKeyComputer>\nwhere TSortKeyComputer: SortKeyComputer + Send + Sync + 'static\n{\n    type Fruit = Vec<(TSortKeyComputer::SortKey, DocAddress)>;\n\n    type Child =\n        TopBySortKeySegmentCollector<TSortKeyComputer::Child, TSortKeyComputer::Comparator>;\n\n    fn check_schema(&self, schema: &Schema) -> crate::Result<()> {\n        self.sort_key_computer.check_schema(schema)\n    }\n\n    fn for_segment(&self, segment_ord: u32, segment_reader: &SegmentReader) -> Result<Self::Child> {\n        let segment_sort_key_computer = self\n            .sort_key_computer\n            .segment_sort_key_computer(segment_reader)?;\n        let topn_computer = TopNComputer::new_with_comparator(\n            self.doc_range.end,\n            self.sort_key_computer.comparator(),\n        );\n        Ok(TopBySortKeySegmentCollector {\n            topn_computer,\n            segment_ord,\n            segment_sort_key_computer,\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.sort_key_computer.requires_scoring()\n    }\n\n    fn merge_fruits(&self, segment_fruits: Vec<Self::Fruit>) -> Result<Self::Fruit> {\n        Ok(merge_top_k(\n            segment_fruits.into_iter().flatten(),\n            self.doc_range.clone(),\n            self.sort_key_computer.comparator(),\n        ))\n    }\n\n    fn collect_segment(\n        &self,\n        weight: &dyn Weight,\n        segment_ord: u32,\n        reader: &SegmentReader,\n    ) -> crate::Result<Vec<(TSortKeyComputer::SortKey, DocAddress)>> {\n        let k = self.doc_range.end;\n        let docs = self\n            .sort_key_computer\n            .collect_segment_top_k(k, weight, reader, segment_ord)?;\n        Ok(docs)\n    }\n}\n\nfn merge_top_k<D: Ord, TSortKey: Clone + std::fmt::Debug, C: Comparator<TSortKey>>(\n    sort_key_docs: impl Iterator<Item = (TSortKey, D)>,\n    doc_range: Range<usize>,\n    comparator: C,\n) -> Vec<(TSortKey, D)> {\n    if doc_range.is_empty() {\n        return Vec::new();\n    }\n    let mut top_collector: TopNComputer<TSortKey, D, C> =\n        TopNComputer::new_with_comparator(doc_range.end, comparator);\n    for (sort_key, doc) in sort_key_docs {\n        top_collector.push(sort_key, doc);\n    }\n    top_collector\n        .into_sorted_vec()\n        .into_iter()\n        .skip(doc_range.start)\n        .map(|cdoc| (cdoc.sort_key, cdoc.doc))\n        .collect()\n}\n\npub struct TopBySortKeySegmentCollector<TSegmentSortKeyComputer, C>\nwhere\n    TSegmentSortKeyComputer: SegmentSortKeyComputer,\n    C: Comparator<TSegmentSortKeyComputer::SegmentSortKey>,\n{\n    pub(crate) topn_computer: TopNComputer<TSegmentSortKeyComputer::SegmentSortKey, DocId, C>,\n    pub(crate) segment_ord: u32,\n    pub(crate) segment_sort_key_computer: TSegmentSortKeyComputer,\n}\n\nimpl<TSegmentSortKeyComputer, C> SegmentCollector\n    for TopBySortKeySegmentCollector<TSegmentSortKeyComputer, C>\nwhere\n    TSegmentSortKeyComputer: 'static + SegmentSortKeyComputer,\n    C: Comparator<TSegmentSortKeyComputer::SegmentSortKey> + 'static,\n{\n    type Fruit = Vec<(TSegmentSortKeyComputer::SortKey, DocAddress)>;\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        self.segment_sort_key_computer.compute_sort_key_and_collect(\n            doc,\n            score,\n            &mut self.topn_computer,\n        );\n    }\n\n    fn harvest(self) -> Self::Fruit {\n        let segment_ord = self.segment_ord;\n        let segment_hits: Vec<(TSegmentSortKeyComputer::SortKey, DocAddress)> = self\n            .topn_computer\n            .into_vec()\n            .into_iter()\n            .map(|comparable_doc| {\n                let sort_key = self\n                    .segment_sort_key_computer\n                    .convert_segment_sort_key(comparable_doc.sort_key);\n                (\n                    sort_key,\n                    DocAddress {\n                        segment_ord,\n                        doc_id: comparable_doc.doc,\n                    },\n                )\n            })\n            .collect();\n        segment_hits\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::ops::Range;\n\n    use rand;\n    use rand::seq::SliceRandom as _;\n\n    use super::merge_top_k;\n    use crate::collector::sort_key::ComparatorEnum;\n    use crate::Order;\n\n    fn test_merge_top_k_aux(\n        order: Order,\n        doc_range: Range<usize>,\n        expected: &[(crate::Score, usize)],\n    ) {\n        let mut vals: Vec<(crate::Score, usize)> = (0..10).map(|val| (val as f32, val)).collect();\n        vals.shuffle(&mut rand::rng());\n        let vals_merged = merge_top_k(vals.into_iter(), doc_range, ComparatorEnum::from(order));\n        assert_eq!(&vals_merged, expected);\n    }\n\n    #[test]\n    fn test_merge_top_k() {\n        test_merge_top_k_aux(Order::Asc, 0..0, &[]);\n        test_merge_top_k_aux(Order::Asc, 3..3, &[]);\n        test_merge_top_k_aux(Order::Asc, 0..3, &[(0.0f32, 0), (1.0f32, 1), (2.0f32, 2)]);\n        test_merge_top_k_aux(\n            Order::Asc,\n            0..11,\n            &[\n                (0.0f32, 0),\n                (1.0f32, 1),\n                (2.0f32, 2),\n                (3.0f32, 3),\n                (4.0f32, 4),\n                (5.0f32, 5),\n                (6.0f32, 6),\n                (7.0f32, 7),\n                (8.0f32, 8),\n                (9.0f32, 9),\n            ],\n        );\n        test_merge_top_k_aux(Order::Asc, 1..3, &[(1.0f32, 1), (2.0f32, 2)]);\n        test_merge_top_k_aux(Order::Desc, 0..2, &[(9.0f32, 9), (8.0f32, 8)]);\n        test_merge_top_k_aux(Order::Desc, 2..4, &[(7.0f32, 7), (6.0f32, 6)]);\n    }\n}\n"
  },
  {
    "path": "src/collector/tests.rs",
    "content": "use columnar::{BytesColumn, Column};\n\nuse super::*;\nuse crate::query::{AllQuery, QueryParser};\nuse crate::schema::{Schema, FAST, TEXT};\nuse crate::time::format_description::well_known::Rfc3339;\nuse crate::time::OffsetDateTime;\nuse crate::{DateTime, DocAddress, Index, Searcher, TantivyDocument};\n\npub const TEST_COLLECTOR_WITH_SCORE: TestCollector = TestCollector {\n    compute_score: true,\n};\n\npub const TEST_COLLECTOR_WITHOUT_SCORE: TestCollector = TestCollector {\n    compute_score: true,\n};\n\n#[test]\npub fn test_filter_collector() -> crate::Result<()> {\n    let mut schema_builder = Schema::builder();\n    let title = schema_builder.add_text_field(\"title\", TEXT);\n    let price = schema_builder.add_u64_field(\"price\", FAST);\n    let date = schema_builder.add_date_field(\"date\", FAST);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n\n    let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n    index_writer.add_document(doc!(title => \"The Name of the Wind\", price => 30_200u64, date => DateTime::from_utc(OffsetDateTime::parse(\"1898-04-09T00:00:00+00:00\", &Rfc3339).unwrap())))?;\n    index_writer.add_document(doc!(title => \"The Diary of Muadib\", price => 29_240u64, date => DateTime::from_utc(OffsetDateTime::parse(\"2020-04-09T00:00:00+00:00\", &Rfc3339).unwrap())))?;\n    index_writer.add_document(doc!(title => \"The Diary of Anne Frank\", price => 18_240u64, date => DateTime::from_utc(OffsetDateTime::parse(\"2019-04-20T00:00:00+00:00\", &Rfc3339).unwrap())))?;\n    index_writer.add_document(doc!(title => \"A Dairy Cow\", price => 21_240u64, date => DateTime::from_utc(OffsetDateTime::parse(\"2019-04-09T00:00:00+00:00\", &Rfc3339).unwrap())))?;\n    index_writer.add_document(doc!(title => \"The Diary of a Young Girl\", price => 20_120u64, date => DateTime::from_utc(OffsetDateTime::parse(\"2018-04-09T00:00:00+00:00\", &Rfc3339).unwrap())))?;\n    index_writer.commit()?;\n\n    let reader = index.reader()?;\n    let searcher = reader.searcher();\n\n    let query_parser = QueryParser::for_index(&index, vec![title]);\n    let query = query_parser.parse_query(\"diary\")?;\n    let filter_some_collector = FilterCollector::new(\n        \"price\".to_string(),\n        &|value: u64| value > 20_120u64,\n        TopDocs::with_limit(2).order_by_score(),\n    );\n    let top_docs = searcher.search(&query, &filter_some_collector)?;\n\n    assert_eq!(top_docs.len(), 1);\n    assert_eq!(top_docs[0].1, DocAddress::new(0, 1));\n\n    let filter_all_collector: FilterCollector<_, _, u64> = FilterCollector::new(\n        \"price\".to_string(),\n        &|value| value < 5u64,\n        TopDocs::with_limit(2).order_by_score(),\n    );\n    let filtered_top_docs = searcher.search(&query, &filter_all_collector).unwrap();\n\n    assert_eq!(filtered_top_docs.len(), 0);\n\n    fn date_filter(value: DateTime) -> bool {\n        (value.into_utc() - OffsetDateTime::parse(\"2019-04-09T00:00:00+00:00\", &Rfc3339).unwrap())\n            .whole_weeks()\n            > 0\n    }\n\n    let filter_dates_collector = FilterCollector::new(\n        \"date\".to_string(),\n        &date_filter,\n        TopDocs::with_limit(5).order_by_score(),\n    );\n    let filtered_date_docs = searcher.search(&query, &filter_dates_collector)?;\n\n    assert_eq!(filtered_date_docs.len(), 2);\n    Ok(())\n}\n\n/// Stores all of the doc ids.\n/// This collector is only used for tests.\n/// It is unusable in practise, as it does\n/// not store the segment ordinals\npub struct TestCollector {\n    pub compute_score: bool,\n}\n\npub struct TestSegmentCollector {\n    segment_id: SegmentOrdinal,\n    fruit: TestFruit,\n}\n\n#[derive(Default)]\npub struct TestFruit {\n    docs: Vec<DocAddress>,\n    scores: Vec<Score>,\n}\n\nimpl TestFruit {\n    /// Return the list of matching documents exhaustively.\n    pub fn docs(&self) -> &[DocAddress] {\n        &self.docs[..]\n    }\n    pub fn scores(&self) -> &[Score] {\n        &self.scores[..]\n    }\n}\n\nimpl Collector for TestCollector {\n    type Fruit = TestFruit;\n    type Child = TestSegmentCollector;\n\n    fn for_segment(\n        &self,\n        segment_id: SegmentOrdinal,\n        _reader: &SegmentReader,\n    ) -> crate::Result<TestSegmentCollector> {\n        Ok(TestSegmentCollector {\n            segment_id,\n            fruit: TestFruit::default(),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        self.compute_score\n    }\n\n    fn merge_fruits(&self, mut children: Vec<TestFruit>) -> crate::Result<TestFruit> {\n        children.sort_by_key(|fruit| {\n            if fruit.docs().is_empty() {\n                0\n            } else {\n                fruit.docs()[0].segment_ord\n            }\n        });\n        let mut docs = vec![];\n        let mut scores = vec![];\n        for child in children {\n            docs.extend(child.docs());\n            scores.extend(child.scores);\n        }\n        Ok(TestFruit { docs, scores })\n    }\n}\n\nimpl SegmentCollector for TestSegmentCollector {\n    type Fruit = TestFruit;\n\n    fn collect(&mut self, doc: DocId, score: Score) {\n        self.fruit.docs.push(DocAddress::new(self.segment_id, doc));\n        self.fruit.scores.push(score);\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        self.fruit\n    }\n}\n\n/// Collects in order all of the fast fields for all of the\n/// doc in the `DocSet`\n///\n/// This collector is mainly useful for tests.\npub struct FastFieldTestCollector {\n    field: String,\n}\n\npub struct FastFieldSegmentCollector {\n    vals: Vec<u64>,\n    reader: Column,\n}\n\nimpl FastFieldTestCollector {\n    pub fn for_field(field: impl ToString) -> FastFieldTestCollector {\n        FastFieldTestCollector {\n            field: field.to_string(),\n        }\n    }\n}\n\nimpl Collector for FastFieldTestCollector {\n    type Fruit = Vec<u64>;\n    type Child = FastFieldSegmentCollector;\n\n    fn for_segment(\n        &self,\n        _: SegmentOrdinal,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<FastFieldSegmentCollector> {\n        let reader = segment_reader\n            .fast_fields()\n            .u64(&self.field)\n            .expect(\"Requested field is not a fast field.\");\n        Ok(FastFieldSegmentCollector {\n            vals: Vec::new(),\n            reader,\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(&self, children: Vec<Vec<u64>>) -> crate::Result<Vec<u64>> {\n        Ok(children.into_iter().flat_map(|v| v.into_iter()).collect())\n    }\n}\n\nimpl SegmentCollector for FastFieldSegmentCollector {\n    type Fruit = Vec<u64>;\n\n    fn collect(&mut self, doc: DocId, _score: Score) {\n        self.vals.extend(self.reader.values_for_doc(doc));\n    }\n\n    fn harvest(self) -> Vec<u64> {\n        self.vals\n    }\n}\n\n/// Collects in order all of the fast field bytes for all of the\n/// docs in the `DocSet`\n///\n/// This collector is mainly useful for tests.\n/// It is very slow.\npub struct BytesFastFieldTestCollector {\n    field: String,\n}\n\npub struct BytesFastFieldSegmentCollector {\n    vals: Vec<u8>,\n    column_opt: Option<BytesColumn>,\n    buffer: Vec<u8>,\n}\n\nimpl BytesFastFieldTestCollector {\n    pub fn for_field(field: impl ToString) -> BytesFastFieldTestCollector {\n        BytesFastFieldTestCollector {\n            field: field.to_string(),\n        }\n    }\n}\n\nimpl Collector for BytesFastFieldTestCollector {\n    type Fruit = Vec<u8>;\n    type Child = BytesFastFieldSegmentCollector;\n\n    fn for_segment(\n        &self,\n        _segment_local_id: u32,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<BytesFastFieldSegmentCollector> {\n        let column_opt = segment_reader.fast_fields().bytes(&self.field)?;\n        Ok(BytesFastFieldSegmentCollector {\n            vals: Vec::new(),\n            column_opt,\n            buffer: Vec::new(),\n        })\n    }\n\n    fn requires_scoring(&self) -> bool {\n        false\n    }\n\n    fn merge_fruits(&self, children: Vec<Vec<u8>>) -> crate::Result<Vec<u8>> {\n        Ok(children.into_iter().flat_map(|c| c.into_iter()).collect())\n    }\n}\n\nimpl SegmentCollector for BytesFastFieldSegmentCollector {\n    type Fruit = Vec<u8>;\n\n    fn collect(&mut self, doc: DocId, _score: Score) {\n        if let Some(column) = self.column_opt.as_ref() {\n            for term_ord in column.term_ords(doc) {\n                let (vals, buffer) = (&mut self.vals, &mut self.buffer);\n                if column.ord_to_bytes(term_ord, buffer).unwrap() {\n                    vals.extend(&buffer[..]);\n                }\n            }\n        }\n    }\n\n    fn harvest(self) -> <Self as SegmentCollector>::Fruit {\n        self.vals\n    }\n}\n\nfn make_test_searcher() -> crate::Result<Searcher> {\n    let schema = Schema::builder().build();\n    let index = Index::create_in_ram(schema);\n    let mut index_writer = index.writer_for_tests()?;\n    index_writer.add_document(TantivyDocument::default())?;\n    index_writer.add_document(TantivyDocument::default())?;\n    index_writer.commit()?;\n    Ok(index.reader()?.searcher())\n}\n\n#[test]\nfn test_option_collector_some() -> crate::Result<()> {\n    let searcher = make_test_searcher()?;\n    let counts = searcher.search(&AllQuery, &Some(Count))?;\n    assert_eq!(counts, Some(2));\n    Ok(())\n}\n\n#[test]\nfn test_option_collector_none() -> crate::Result<()> {\n    let searcher = make_test_searcher()?;\n    let none_collector: Option<Count> = None;\n    let counts = searcher.search(&AllQuery, &none_collector)?;\n    assert_eq!(counts, None);\n    Ok(())\n}\n"
  },
  {
    "path": "src/collector/top_collector.rs",
    "content": "use serde::{Deserialize, Serialize};\n\n/// Contains a feature (field, score, etc.) of a document along with the document address.\n///\n/// Used only by TopNComputer, which implements the actual comparison via a `Comparator`.\n#[derive(Clone, Default, Eq, PartialEq, Serialize, Deserialize)]\npub struct ComparableDoc<T, D> {\n    /// The feature of the document. In practice, this is\n    /// is a type which can be compared with a `Comparator<T>`.\n    pub sort_key: T,\n    /// The document address. In practice, this is either a `DocId` or `DocAddress`.\n    pub doc: D,\n}\n\nimpl<T: std::fmt::Debug, D: std::fmt::Debug> std::fmt::Debug for ComparableDoc<T, D> {\n    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {\n        f.debug_struct(\"ComparableDoc\")\n            .field(\"feature\", &self.sort_key)\n            .field(\"doc\", &self.doc)\n            .finish()\n    }\n}\n"
  },
  {
    "path": "src/collector/top_score_collector.rs",
    "content": "use std::cmp::Ordering;\nuse std::fmt;\nuse std::ops::Range;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::Collector;\nuse crate::collector::sort_key::{\n    Comparator, ComparatorEnum, NaturalComparator, ReverseComparator, SortBySimilarityScore,\n    SortByStaticFastValue, SortByString,\n};\nuse crate::collector::sort_key_top_collector::TopBySortKeyCollector;\nuse crate::collector::top_collector::ComparableDoc;\nuse crate::collector::{SegmentSortKeyComputer, SortKeyComputer};\nuse crate::fastfield::FastValue;\nuse crate::{DocAddress, DocId, Order, Score, SegmentReader};\n\n/// The `TopDocs` collector keeps track of the top `K` documents\n/// sorted by their score.\n///\n/// The implementation is based on a repeatedly truncating on the median after K * 2 documents\n/// with pattern defeating QuickSort.\n/// The theoretical complexity for collecting the top `K` out of `N` documents\n/// is `O(N + K)`.\n///\n/// This collector guarantees a stable sorting in case of a tie on the\n/// document score/sort key: The document address (`DocAddress`) is used as a tie breaker.\n/// In case of a tie on the sort key, documents are always sorted by ascending `DocAddress`.\n///\n/// ```rust\n/// use tantivy::collector::TopDocs;\n/// use tantivy::query::QueryParser;\n/// use tantivy::schema::{Schema, TEXT};\n/// use tantivy::{doc, DocAddress, Index};\n///\n/// # fn main() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n///\n/// let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n/// index_writer.add_document(doc!(title => \"The Name of the Wind\"))?;\n/// index_writer.add_document(doc!(title => \"The Diary of Muadib\"))?;\n/// index_writer.add_document(doc!(title => \"A Dairy Cow\"))?;\n/// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\"))?;\n/// index_writer.commit()?;\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n///\n/// let query_parser = QueryParser::for_index(&index, vec![title]);\n/// let query = query_parser.parse_query(\"diary\")?;\n/// let top_docs = searcher.search(&query, &TopDocs::with_limit(2).order_by_score())?;\n///\n/// assert_eq!(top_docs[0].1, DocAddress::new(0, 1));\n/// assert_eq!(top_docs[1].1, DocAddress::new(0, 3));\n/// # Ok(())\n/// # }\n/// ```\npub struct TopDocs {\n    limit: usize,\n    offset: usize,\n}\n\nimpl fmt::Debug for TopDocs {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"TopDocs(limit={}, offset={})\", self.limit, self.offset)\n    }\n}\n\nimpl TopDocs {\n    /// Builds a `TopDocs` capturing a given document range.\n    ///\n    /// The range start..end translates in a limit of `end - start`\n    /// and an offset of start.\n    pub fn for_doc_range(doc_range: Range<usize>) -> Self {\n        TopDocs {\n            limit: doc_range.end.saturating_sub(doc_range.start),\n            offset: doc_range.start,\n        }\n    }\n\n    /// Returns the doc range we are trying to capture.\n    pub fn doc_range(&self) -> Range<usize> {\n        self.offset..self.offset + self.limit\n    }\n\n    /// Creates a top score collector, with a number of documents equal to \"limit\".\n    ///\n    /// # Panics\n    /// The method panics if limit is 0\n    pub fn with_limit(limit: usize) -> TopDocs {\n        assert_ne!(limit, 0, \"Limit must be greater than 0\");\n        TopDocs { limit, offset: 0 }\n    }\n\n    /// Skip the first \"offset\" documents when collecting.\n    ///\n    /// This is equivalent to `OFFSET` in MySQL or PostgreSQL and `start` in\n    /// Lucene's TopDocsCollector.\n    ///\n    /// # Example\n    ///\n    /// ```rust\n    /// use tantivy::collector::TopDocs;\n    /// use tantivy::query::QueryParser;\n    /// use tantivy::schema::{Schema, TEXT};\n    /// use tantivy::{doc, DocAddress, Index};\n    ///\n    /// # fn main() -> tantivy::Result<()> {\n    /// let mut schema_builder = Schema::builder();\n    /// let title = schema_builder.add_text_field(\"title\", TEXT);\n    /// let schema = schema_builder.build();\n    /// let index = Index::create_in_ram(schema);\n    ///\n    /// let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n    /// index_writer.add_document(doc!(title => \"The Name of the Wind\"))?;\n    /// index_writer.add_document(doc!(title => \"The Diary of Muadib\"))?;\n    /// index_writer.add_document(doc!(title => \"A Dairy Cow\"))?;\n    /// index_writer.add_document(doc!(title => \"The Diary of a Young Girl\"))?;\n    /// index_writer.add_document(doc!(title => \"The Diary of Lena Mukhina\"))?;\n    /// index_writer.commit()?;\n    ///\n    /// let reader = index.reader()?;\n    /// let searcher = reader.searcher();\n    ///\n    /// let query_parser = QueryParser::for_index(&index, vec![title]);\n    /// let query = query_parser.parse_query(\"diary\")?;\n    /// let top_docs = searcher.search(&query, &TopDocs::with_limit(2).and_offset(1).order_by_score())?;\n    ///\n    /// assert_eq!(top_docs.len(), 2);\n    /// assert_eq!(top_docs[0].1, DocAddress::new(0, 4));\n    /// assert_eq!(top_docs[1].1, DocAddress::new(0, 3));\n    /// Ok(())\n    /// # }\n    /// ```\n    #[must_use]\n    pub fn and_offset(self, offset: usize) -> TopDocs {\n        TopDocs {\n            limit: self.limit,\n            offset,\n        }\n    }\n\n    /// Set top-K to rank documents by a given fast field.\n    ///\n    /// If the field is not a fast or does not exist, this method returns successfully (it is not\n    /// aware of any schema). An error will be returned at the moment of search.\n    ///\n    /// If the field is a FAST field but not a u64 field, search will return successfully but it\n    /// will return returns a monotonic u64-representation (ie. the order is still correct) of\n    /// the requested field type.\n    ///\n    /// # Example\n    ///\n    /// ```rust\n    /// # use tantivy::schema::{Schema, FAST, TEXT};\n    /// # use tantivy::{doc, Index, DocAddress, Order};\n    /// # use tantivy::query::{Query, QueryParser};\n    /// use tantivy::Searcher;\n    /// use tantivy::collector::TopDocs;\n    ///\n    /// # fn main() -> tantivy::Result<()> {\n    /// #   let mut schema_builder = Schema::builder();\n    /// #   let title = schema_builder.add_text_field(\"title\", TEXT);\n    /// #   let rating = schema_builder.add_u64_field(\"rating\", FAST);\n    /// #   let schema = schema_builder.build();\n    /// #\n    /// #   let index = Index::create_in_ram(schema);\n    /// #   let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n    /// #   index_writer.add_document(doc!(title => \"The Name of the Wind\", rating => 92u64))?;\n    /// #   index_writer.add_document(doc!(title => \"The Diary of Muadib\", rating => 97u64))?;\n    /// #   index_writer.add_document(doc!(title => \"A Dairy Cow\", rating => 63u64))?;\n    /// #   index_writer.add_document(doc!(title => \"The Diary of a Young Girl\", rating => 80u64))?;\n    /// #   index_writer.commit()?;\n    /// #   let reader = index.reader()?;\n    /// #   let query = QueryParser::for_index(&index, vec![title]).parse_query(\"diary\")?;\n    /// #   let top_docs = docs_sorted_by_rating(&reader.searcher(), &query)?;\n    /// #   assert_eq!(top_docs,\n    /// #            vec![(Some(97u64), DocAddress::new(0u32, 1)),\n    /// #                 (Some(80u64), DocAddress::new(0u32, 3))]);\n    /// #   Ok(())\n    /// # }\n    /// /// Searches the document matching the given query, and\n    /// /// collects the top 10 documents, order by the u64-`field`\n    /// /// given in argument.\n    /// fn docs_sorted_by_rating(searcher: &Searcher,\n    ///                          query: &dyn Query)\n    ///     -> tantivy::Result<Vec<(Option<u64>, DocAddress)>> {\n    ///\n    ///     // This is where we build our topdocs collector\n    ///     //\n    ///     // Note the `rating_field` needs to be a FAST field here.\n    ///     let top_books_by_rating = TopDocs\n    ///                 ::with_limit(10)\n    ///                  .order_by_fast_field(\"rating\", Order::Desc);\n    ///\n    ///     // ... and here are our documents. Note this is a simple vec.\n    ///     // The `u64` in the pair is the value of our fast field for\n    ///     // each documents.\n    ///     //\n    ///     // The vec is sorted decreasingly by `sort_by_field`, and has a\n    ///     // length of 10, or less if not enough documents matched the\n    ///     // query.\n    ///     let resulting_docs: Vec<(Option<u64>, DocAddress)> =\n    ///          searcher.search(query, &top_books_by_rating)?;\n    ///\n    ///     Ok(resulting_docs)\n    /// }\n    /// ```\n    ///\n    /// # See also\n    ///\n    /// To comfortably work with `u64`s, `i64`s, `f64`s, or `date`s, please refer to\n    /// the [.order_by_fast_field(...)](TopDocs::order_by_fast_field) method.\n    pub fn order_by_u64_field(\n        self,\n        field: impl ToString,\n        order: Order,\n    ) -> impl Collector<Fruit = Vec<(Option<u64>, DocAddress)>> {\n        self.order_by((SortByStaticFastValue::for_field(field), order))\n    }\n\n    /// Order docs by decreasing BM25 similarity score.\n    pub fn order_by_score(self) -> impl Collector<Fruit = Vec<(Score, DocAddress)>> {\n        TopBySortKeyCollector::new(SortBySimilarityScore, self.doc_range())\n    }\n\n    /// Set top-K to rank documents by a given fast field.\n    ///\n    /// If the field is not a fast field, or its field type does not match the generic type, this\n    /// method does not panic, but an explicit error will be returned at the moment of\n    /// collection.\n    ///\n    /// Note that this method is a generic. The requested fast field type will be often\n    /// inferred in your code by the rust compiler.\n    ///\n    /// Implementation-wise, for performance reason, tantivy will manipulate the u64 representation\n    /// of your fast field until the last moment.\n    ///\n    /// # Example\n    ///\n    /// ```rust\n    /// # use tantivy::schema::{Schema, FAST, TEXT};\n    /// # use tantivy::{doc, Index, DocAddress,Order};\n    /// # use tantivy::query::{Query, AllQuery};\n    /// use tantivy::Searcher;\n    /// use tantivy::collector::TopDocs;\n    ///\n    /// # fn main() -> tantivy::Result<()> {\n    /// #   let mut schema_builder = Schema::builder();\n    /// #   let title = schema_builder.add_text_field(\"company\", TEXT);\n    /// #   let revenue = schema_builder.add_i64_field(\"revenue\", FAST);\n    /// #   let schema = schema_builder.build();\n    /// #\n    /// #   let index = Index::create_in_ram(schema);\n    /// #   let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n    /// #   index_writer.add_document(doc!(title => \"MadCow Inc.\", revenue => 92_000_000i64))?;\n    /// #   index_writer.add_document(doc!(title => \"Zozo Cow KKK\", revenue => 119_000_000i64))?;\n    /// #   index_writer.add_document(doc!(title => \"Declining Cow\", revenue => -63_000_000i64))?;\n    /// #   assert!(index_writer.commit().is_ok());\n    /// #   let reader = index.reader()?;\n    /// #   let top_docs = docs_sorted_by_revenue(&reader.searcher(), &AllQuery, \"revenue\")?;\n    /// #   assert_eq!(top_docs,\n    /// #            vec![(Some(119_000_000i64), DocAddress::new(0, 1)),\n    /// #                 (Some(92_000_000i64), DocAddress::new(0, 0))]);\n    /// #   Ok(())\n    /// # }\n    /// /// Searches the document matching the given query, and\n    /// /// collects the top 10 documents, order by the u64-`field`\n    /// /// given in argument.\n    /// fn docs_sorted_by_revenue(searcher: &Searcher,\n    ///                          query: &dyn Query,\n    ///                          revenue_field: &str)\n    ///     -> tantivy::Result<Vec<(Option<i64>, DocAddress)>> {\n    ///\n    ///     // This is where we build our topdocs collector\n    ///     //\n    ///     // Note the generics parameter that needs to match the\n    ///     // type `sort_by_field`. revenue_field here is a FAST i64 field.\n    ///     let top_company_by_revenue = TopDocs\n    ///                 ::with_limit(2)\n    ///                  .order_by_fast_field(\"revenue\", Order::Desc);\n    ///\n    ///     // ... and here are our documents. Note this is a simple vec.\n    ///     // The `i64` in the pair is the value of our fast field for\n    ///     // each documents.\n    ///     //\n    ///     // The vec is sorted decreasingly by `sort_by_field`, and has a\n    ///     // length of 10, or less if not enough documents matched the\n    ///     // query.\n    ///     let resulting_docs: Vec<(Option<i64>, DocAddress)> =\n    ///          searcher.search(query, &top_company_by_revenue)?;\n    ///\n    ///     Ok(resulting_docs)\n    /// }\n    /// ```\n    pub fn order_by_fast_field<TFastValue>(\n        self,\n        fast_field: impl ToString,\n        order: Order,\n    ) -> impl Collector<Fruit = Vec<(Option<TFastValue>, DocAddress)>>\n    where\n        TFastValue: FastValue,\n        ComparatorEnum: Comparator<Option<TFastValue>>,\n    {\n        self.order_by((SortByStaticFastValue::for_field(fast_field), order))\n    }\n\n    /// Like `order_by_fast_field`, but for a `String` fast field.\n    pub fn order_by_string_fast_field(\n        self,\n        fast_field: impl ToString,\n        order: Order,\n    ) -> impl Collector<Fruit = Vec<(Option<String>, DocAddress)>> {\n        let by_string_sort_key_computer = SortByString::for_field(fast_field.to_string());\n        self.order_by((by_string_sort_key_computer, order))\n    }\n\n    /// Ranks the documents using a sort key.\n    pub fn order_by<TSortKey>(\n        self,\n        sort_key_computer: impl SortKeyComputer<SortKey = TSortKey> + Send + 'static,\n    ) -> impl Collector<Fruit = Vec<(TSortKey, DocAddress)>>\n    where\n        TSortKey: 'static + Clone + Send + Sync + std::fmt::Debug,\n    {\n        TopBySortKeyCollector::new(sort_key_computer, self.doc_range())\n    }\n\n    /// Helper function to tweak the similarity score of documents using a function.\n    /// (usually a closure).\n    ///\n    /// This method offers a convenient way to tweak or replace\n    /// the documents score. As suggested by the prototype you can\n    /// manually define your own [`SortKeyComputer`]\n    /// and pass it as an argument, but there is a much simpler way to\n    /// tweak your score: you can use a closure as in the following\n    /// example.\n    ///\n    /// # Example\n    ///\n    /// Typically, you will want to rely on one or more fast fields,\n    /// to alter the original relevance `Score`.\n    ///\n    /// For instance, in the following, we assume that we are implementing\n    /// an e-commerce website that has a fast field called `popularity`\n    /// that rates whether a product is typically often bought by users.\n    ///\n    /// In the following example will will tweak our ranking a bit by\n    /// boosting popular products a notch.\n    ///\n    /// In more serious application, this tweaking could involve running a\n    /// learning-to-rank model over various features\n    ///\n    /// ```rust\n    /// # use tantivy::schema::{Schema, FAST, TEXT};\n    /// # use tantivy::{doc, Index, DocAddress, DocId, Score};\n    /// # use tantivy::query::QueryParser;\n    /// use tantivy::SegmentReader;\n    /// use tantivy::collector::TopDocs;\n    /// use tantivy::schema::Field;\n    ///\n    /// fn create_schema() -> Schema {\n    ///    let mut schema_builder = Schema::builder();\n    ///    schema_builder.add_text_field(\"product_name\", TEXT);\n    ///    schema_builder.add_u64_field(\"popularity\", FAST);\n    ///    schema_builder.build()\n    /// }\n    ///\n    /// fn create_index() -> tantivy::Result<Index> {\n    ///   let schema = create_schema();\n    ///   let index = Index::create_in_ram(schema);\n    ///   let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n    ///   let product_name = index.schema().get_field(\"product_name\").unwrap();\n    ///   let popularity: Field = index.schema().get_field(\"popularity\").unwrap();\n    ///   index_writer.add_document(doc!(product_name => \"The Diary of Muadib\", popularity => 1u64))?;\n    ///   index_writer.add_document(doc!(product_name => \"A Dairy Cow\", popularity => 10u64))?;\n    ///   index_writer.add_document(doc!(product_name => \"The Diary of a Young Girl\", popularity => 15u64))?;\n    ///   index_writer.commit()?;\n    ///   Ok(index)\n    /// }\n    ///\n    /// let index = create_index().unwrap();\n    /// let product_name = index.schema().get_field(\"product_name\").unwrap();\n    /// let popularity: Field = index.schema().get_field(\"popularity\").unwrap();\n    ///\n    /// let user_query_str = \"diary\";\n    /// let query_parser = QueryParser::for_index(&index, vec![product_name]);\n    /// let query = query_parser.parse_query(user_query_str).unwrap();\n    ///\n    /// // This is where we build our collector with our custom score.\n    /// let top_docs_by_custom_score = TopDocs\n    ///         ::with_limit(10)\n    ///          .tweak_score(move |segment_reader: &SegmentReader| {\n    ///             // The argument is a function that returns our scoring\n    ///             // function.\n    ///             //\n    ///             // The point of this \"mother\" function is to gather all\n    ///             // of the segment level information we need for scoring.\n    ///             // Typically, fast_fields.\n    ///             //\n    ///             // In our case, we will get a reader for the popularity\n    ///             // fast field. For simplicity we read the first or default value in the fast\n    ///             // field.\n    ///             let popularity_reader =\n    ///                 segment_reader.fast_fields().u64(\"popularity\").unwrap().first_or_default_col(0);\n    ///\n    ///             // We can now define our actual scoring function\n    ///             move |doc: DocId, original_score: Score| {\n    ///                 let popularity: u64 = popularity_reader.get_val(doc);\n    ///                 // Well.. For the sake of the example we use a simple logarithm\n    ///                 // function.\n    ///                 let popularity_boost_score = ((2u64 + popularity) as Score).log2();\n    ///                 popularity_boost_score * original_score\n    ///             }\n    ///           });\n    /// let reader = index.reader().unwrap();\n    /// let searcher = reader.searcher();\n    /// // ... and here are our documents. Note this is a simple vec.\n    /// // The `Score` in the pair is our tweaked score.\n    /// let resulting_docs: Vec<(Score, DocAddress)> =\n    ///      searcher.search(&query, &top_docs_by_custom_score).unwrap();\n    /// ``\n    pub fn tweak_score<F, TSortKey>(\n        self,\n        sort_key_fn: F,\n    ) -> impl Collector<Fruit = Vec<(TSortKey, DocAddress)>>\n    where\n        F: 'static + Send + Sync,\n        TSortKey: 'static + PartialOrd + Clone + Send + Sync + std::fmt::Debug,\n        TweakScoreFn<F>: SortKeyComputer<SortKey = TSortKey>,\n    {\n        self.order_by(TweakScoreFn(sort_key_fn))\n    }\n}\n\n/// Helper struct to make it possible to define a sort key computer that does not use\n/// the similary score from a simple function.\npub struct TweakScoreFn<F>(F);\n\nimpl<F, TTweakScoreSortKeyFn, TSortKey> SortKeyComputer for TweakScoreFn<F>\nwhere\n    F: 'static + Send + Sync + Fn(&SegmentReader) -> TTweakScoreSortKeyFn,\n    TTweakScoreSortKeyFn: 'static + Fn(DocId, Score) -> TSortKey,\n    TweakScoreSegmentSortKeyComputer<TTweakScoreSortKeyFn>:\n        SegmentSortKeyComputer<SortKey = TSortKey, SegmentSortKey = TSortKey>,\n    TSortKey: 'static + PartialOrd + Clone + Send + Sync + std::fmt::Debug,\n{\n    type SortKey = TSortKey;\n    type Child = TweakScoreSegmentSortKeyComputer<TTweakScoreSortKeyFn>;\n    type Comparator = NaturalComparator;\n\n    fn requires_scoring(&self) -> bool {\n        true\n    }\n\n    fn segment_sort_key_computer(\n        &self,\n        segment_reader: &SegmentReader,\n    ) -> crate::Result<Self::Child> {\n        Ok({\n            TweakScoreSegmentSortKeyComputer {\n                sort_key_fn: (self.0)(segment_reader),\n            }\n        })\n    }\n}\n\npub struct TweakScoreSegmentSortKeyComputer<TTweakScoreSortKeyFn> {\n    sort_key_fn: TTweakScoreSortKeyFn,\n}\n\nimpl<TTweakScoreSortKeyFn, TSortKey> SegmentSortKeyComputer\n    for TweakScoreSegmentSortKeyComputer<TTweakScoreSortKeyFn>\nwhere\n    TTweakScoreSortKeyFn: 'static + Fn(DocId, Score) -> TSortKey,\n    TSortKey: 'static + PartialOrd + Clone + Send + Sync,\n{\n    type SortKey = TSortKey;\n    type SegmentSortKey = TSortKey;\n    type SegmentComparator = NaturalComparator;\n\n    fn segment_sort_key(&mut self, doc: DocId, score: Score) -> TSortKey {\n        (self.sort_key_fn)(doc, score)\n    }\n\n    /// Convert a segment level score into the global level score.\n    fn convert_segment_sort_key(&self, sort_key: Self::SegmentSortKey) -> Self::SortKey {\n        sort_key\n    }\n}\n\n/// Fast TopN Computation\n///\n/// Capacity of the vec is 2 * top_n.\n/// The buffer is truncated to the top_n elements when it reaches the capacity of the Vec.\n/// That means capacity has special meaning and should be carried over when cloning or serializing.\n///\n/// For TopN == 0, it will be relative expensive.\n///\n/// The TopNComputer will tiebreak by using ascending `D` (DocId or DocAddress):\n/// i.e., in case of a tie on the sort key, the `DocId|DocAddress` are always sorted in\n/// ascending order, regardless of the `Comparator` used for the `Score` type.\n///\n/// NOTE: Items must be `push`ed to the TopNComputer in ascending `DocId|DocAddress` order, as the\n/// threshold used to eliminate docs does not include the `DocId` or `DocAddress`: this provides\n/// the ascending `DocId|DocAddress` tie-breaking behavior without additional comparisons.\n#[derive(Serialize, Deserialize)]\n#[serde(from = \"TopNComputerDeser<Score, D, C>\")]\npub struct TopNComputer<Score, D, C> {\n    /// The buffer reverses sort order to get top-semantics instead of bottom-semantics\n    buffer: Vec<ComparableDoc<Score, D>>,\n    top_n: usize,\n    pub(crate) threshold: Option<Score>,\n    comparator: C,\n}\n\n// Intermediate struct for TopNComputer for deserialization, to keep vec capacity\n#[derive(Deserialize)]\nstruct TopNComputerDeser<Score, D, C> {\n    buffer: Vec<ComparableDoc<Score, D>>,\n    top_n: usize,\n    threshold: Option<Score>,\n    comparator: C,\n}\n\nimpl<Score, D, C> From<TopNComputerDeser<Score, D, C>> for TopNComputer<Score, D, C> {\n    fn from(mut value: TopNComputerDeser<Score, D, C>) -> Self {\n        let expected_cap = value.top_n.max(1) * 2;\n        let current_cap = value.buffer.capacity();\n        if current_cap < expected_cap {\n            value.buffer.reserve_exact(expected_cap - current_cap);\n        } else {\n            value.buffer.shrink_to(expected_cap);\n        }\n\n        TopNComputer {\n            buffer: value.buffer,\n            top_n: value.top_n,\n            threshold: value.threshold,\n            comparator: value.comparator,\n        }\n    }\n}\n\nimpl<Score: std::fmt::Debug, D, C> std::fmt::Debug for TopNComputer<Score, D, C>\nwhere C: Comparator<Score>\n{\n    fn fmt(&self, f: &mut fmt::Formatter) -> std::fmt::Result {\n        f.debug_struct(\"TopNComputer\")\n            .field(\"buffer_len\", &self.buffer.len())\n            .field(\"top_n\", &self.top_n)\n            .field(\"current_threshold\", &self.threshold)\n            .field(\"comparator\", &self.comparator)\n            .finish()\n    }\n}\n\n// Custom clone to keep capacity\nimpl<Score: Clone, D: Clone, C: Clone> Clone for TopNComputer<Score, D, C> {\n    fn clone(&self) -> Self {\n        let mut buffer_clone = Vec::with_capacity(self.buffer.capacity());\n        buffer_clone.extend(self.buffer.iter().cloned());\n        TopNComputer {\n            buffer: buffer_clone,\n            top_n: self.top_n,\n            threshold: self.threshold.clone(),\n            comparator: self.comparator.clone(),\n        }\n    }\n}\n\nimpl<TSortKey, D> TopNComputer<TSortKey, D, ReverseComparator>\nwhere\n    D: Ord,\n    TSortKey: Clone,\n    NaturalComparator: Comparator<TSortKey>,\n{\n    /// Create a new `TopNComputer`.\n    /// Internally it will allocate a buffer of size `2 * top_n`.\n    pub fn new(top_n: usize) -> Self {\n        TopNComputer::new_with_comparator(top_n, ReverseComparator)\n    }\n}\n\n#[inline(always)]\nfn compare_for_top_k<TSortKey, D: Ord, C: Comparator<TSortKey>>(\n    c: &C,\n    lhs: &ComparableDoc<TSortKey, D>,\n    rhs: &ComparableDoc<TSortKey, D>,\n) -> std::cmp::Ordering {\n    c.compare(&lhs.sort_key, &rhs.sort_key)\n        .reverse() // Reverse here because we want top K.\n        .then_with(|| lhs.doc.cmp(&rhs.doc)) // Regardless of asc/desc, in presence of a tie, we\n                                             // sort by doc id\n}\n\nimpl<TSortKey, D, C> TopNComputer<TSortKey, D, C>\nwhere\n    D: Ord,\n    TSortKey: Clone,\n    C: Comparator<TSortKey>,\n{\n    /// Create a new `TopNComputer`.\n    /// Internally it will allocate a buffer of size `2 * top_n`.\n    pub fn new_with_comparator(top_n: usize, comparator: C) -> Self {\n        let vec_cap = top_n.max(1) * 2;\n        TopNComputer {\n            buffer: Vec::with_capacity(vec_cap),\n            top_n,\n            threshold: None,\n            comparator,\n        }\n    }\n\n    /// Push a new document to the top n.\n    /// If the document is below the current threshold, it will be ignored.\n    ///\n    /// NOTE: `push` must be called in ascending `DocId`/`DocAddress` order.\n    #[inline]\n    pub fn push(&mut self, sort_key: TSortKey, doc: D) {\n        if let Some(last_median) = &self.threshold {\n            // See the struct docs for an explanation of why this comparison is strict.\n            if self.comparator.compare(&sort_key, last_median) != Ordering::Greater {\n                return;\n            }\n        }\n        self.append_doc(doc, sort_key);\n    }\n\n    // Append a document to the top n.\n    //\n    // At this point, we need to have established that the doc is above the threshold.\n    #[inline(always)]\n    pub(crate) fn append_doc(&mut self, doc: D, sort_key: TSortKey) {\n        if self.buffer.len() == self.buffer.capacity() {\n            let median = self.truncate_top_n();\n            self.threshold = Some(median);\n        }\n        // This cannot panic, because we truncate_median will at least remove one element, since\n        // the min capacity is 2.\n        let comparable_doc = ComparableDoc { doc, sort_key };\n        push_assuming_capacity(comparable_doc, &mut self.buffer);\n    }\n\n    #[inline(never)]\n    fn truncate_top_n(&mut self) -> TSortKey {\n        // Use select_nth_unstable to find the top nth score\n        let (_, median_el, _) = self.buffer.select_nth_unstable_by(self.top_n, |lhs, rhs| {\n            compare_for_top_k(&self.comparator, lhs, rhs)\n        });\n\n        let median_score = median_el.sort_key.clone();\n        // Remove all elements below the top_n\n        self.buffer.truncate(self.top_n);\n\n        median_score\n    }\n\n    /// Returns the top n elements in sorted order.\n    pub fn into_sorted_vec(mut self) -> Vec<ComparableDoc<TSortKey, D>> {\n        if self.buffer.len() > self.top_n {\n            self.truncate_top_n();\n        }\n        self.buffer\n            .sort_unstable_by(|lhs, rhs| compare_for_top_k(&self.comparator, lhs, rhs));\n        self.buffer\n    }\n\n    /// Returns the top n elements in stored order.\n    /// Useful if you do not need the elements in sorted order,\n    /// for example when merging the results of multiple segments.\n    pub fn into_vec(mut self) -> Vec<ComparableDoc<TSortKey, D>> {\n        if self.buffer.len() > self.top_n {\n            self.truncate_top_n();\n        }\n        self.buffer\n    }\n}\n\n// Push an element provided there is enough capacity to do so.\n//\n// Panics if there is not enough capacity to add an element.\n#[inline(always)]\nfn push_assuming_capacity<T>(el: T, buf: &mut Vec<T>) {\n    let prev_len = buf.len();\n    assert!(prev_len < buf.capacity());\n    // This is mimicking the current (non-stabilized) implementation in std.\n    // SAFETY: we just checked we have enough capacity.\n    unsafe {\n        let end = buf.as_mut_ptr().add(prev_len);\n        std::ptr::write(end, el);\n        buf.set_len(prev_len + 1);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use proptest::prelude::*;\n\n    use super::{TopDocs, TopNComputer};\n    use crate::collector::sort_key::{ComparatorEnum, NaturalComparator, ReverseComparator};\n    use crate::collector::top_collector::ComparableDoc;\n    use crate::collector::{Collector, DocSetCollector};\n    use crate::query::{AllQuery, Query, QueryParser};\n    use crate::schema::{Field, Schema, FAST, STORED, TEXT};\n    use crate::time::format_description::well_known::Rfc3339;\n    use crate::time::OffsetDateTime;\n    use crate::{\n        assert_nearly_equals, DateTime, DocAddress, DocId, Index, IndexWriter, Order, Score,\n        SegmentReader,\n    };\n\n    fn make_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        // writing the segment\n        let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n        index_writer.add_document(doc!(text_field=>\"Hello happy tax payer.\"))?;\n        index_writer.add_document(doc!(text_field=>\"Droopy says hello happy tax payer\"))?;\n        index_writer.add_document(doc!(text_field=>\"I like Droopy\"))?;\n        index_writer.commit()?;\n        Ok(index)\n    }\n\n    fn assert_results_equals(results: &[(Score, DocAddress)], expected: &[(Score, DocAddress)]) {\n        for (result, expected) in results.iter().zip(expected.iter()) {\n            assert_eq!(result.1, expected.1);\n            crate::assert_nearly_equals!(result.0, expected.0);\n        }\n    }\n\n    #[test]\n    fn test_empty_topn_computer() {\n        let mut computer: TopNComputer<u32, u32, NaturalComparator> =\n            TopNComputer::new_with_comparator(0, NaturalComparator);\n\n        computer.push(1u32, 1u32);\n        computer.push(1u32, 2u32);\n        computer.push(1u32, 3u32);\n        assert!(computer.into_vec().is_empty());\n    }\n\n    #[test]\n    fn test_topn_computer() {\n        let mut computer: TopNComputer<u32, u32, NaturalComparator> =\n            TopNComputer::new_with_comparator(2, NaturalComparator);\n\n        computer.push(1u32, 1u32);\n        computer.push(2u32, 2u32);\n        computer.push(3u32, 3u32);\n        computer.push(2u32, 4u32);\n        computer.push(1u32, 5u32);\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 3u32,\n                    doc: 3u32,\n                },\n                ComparableDoc {\n                    sort_key: 2u32,\n                    doc: 2u32,\n                }\n            ]\n        );\n    }\n\n    #[test]\n    fn test_topn_computer_duplicates() {\n        let mut computer: TopNComputer<u32, u32, NaturalComparator> =\n            TopNComputer::new_with_comparator(2, NaturalComparator);\n\n        computer.push(1u32, 1u32);\n        computer.push(1u32, 2u32);\n        computer.push(1u32, 3u32);\n        computer.push(1u32, 4u32);\n        computer.push(1u32, 5u32);\n\n        // In the presence of duplicates, DocIds are always ascending order.\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 1u32,\n                    doc: 1u32,\n                },\n                ComparableDoc {\n                    sort_key: 1u32,\n                    doc: 2u32,\n                }\n            ]\n        );\n    }\n\n    #[test]\n    fn test_topn_computer_no_panic() {\n        for top_n in 0..10 {\n            let mut computer: TopNComputer<u32, u32, NaturalComparator> =\n                TopNComputer::new_with_comparator(top_n, NaturalComparator);\n\n            for _ in 0..1 + top_n * 2 {\n                computer.push(1u32, 1u32);\n            }\n            let _vals = computer.into_sorted_vec();\n        }\n    }\n\n    proptest! {\n        #[test]\n        fn test_topn_computer_asc_prop(\n          limit in 0..10_usize,\n          mut docs in proptest::collection::vec((0..100_u64, 0..100_u64), 0..100_usize),\n        ) {\n            // NB: TopNComputer must receive inputs in ascending DocId order.\n            docs.sort_by_key(|(_, doc_id)| *doc_id);\n            let mut computer: TopNComputer<_, _, ReverseComparator> = TopNComputer::new_with_comparator(limit, ReverseComparator);\n            for (feature, doc) in &docs {\n                computer.push(*feature, *doc);\n            }\n            let mut comparable_docs: Vec<ComparableDoc<u64, u64>> =\n                docs.into_iter().map(|(sort_key, doc)| ComparableDoc { sort_key, doc }).collect();\n            crate::collector::sort_key::tests::sort_hits(&mut comparable_docs, Order::Asc);\n            comparable_docs.truncate(limit);\n            prop_assert_eq!(\n                computer.into_sorted_vec(),\n                comparable_docs,\n            );\n        }\n    }\n\n    #[test]\n    fn test_top_collector_not_at_capacity_without_offset() -> crate::Result<()> {\n        let index = make_index()?;\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\")?;\n        let score_docs: Vec<(Score, DocAddress)> = index\n            .reader()?\n            .searcher()\n            .search(&text_query, &TopDocs::with_limit(4).order_by_score())?;\n        assert_results_equals(\n            &score_docs,\n            &[\n                (0.81221175, DocAddress::new(0u32, 1)),\n                (0.5376842, DocAddress::new(0u32, 2)),\n                (0.48527452, DocAddress::new(0, 0)),\n            ],\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_collector_not_at_capacity_with_offset() {\n        let index = make_index().unwrap();\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\").unwrap();\n        let score_docs: Vec<(Score, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(\n                &text_query,\n                &TopDocs::with_limit(4).and_offset(2).order_by_score(),\n            )\n            .unwrap();\n        assert_results_equals(&score_docs[..], &[(0.48527452, DocAddress::new(0, 0))]);\n    }\n\n    #[test]\n    fn test_top_collector_at_capacity() {\n        let index = make_index().unwrap();\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\").unwrap();\n        let score_docs: Vec<(Score, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(&text_query, &TopDocs::with_limit(2).order_by_score())\n            .unwrap();\n        assert_results_equals(\n            &score_docs,\n            &[\n                (0.81221175, DocAddress::new(0u32, 1)),\n                (0.5376842, DocAddress::new(0u32, 2)),\n            ],\n        );\n    }\n\n    #[test]\n    fn test_top_collector_at_capacity_with_offset() {\n        let index = make_index().unwrap();\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\").unwrap();\n        let score_docs: Vec<(Score, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(\n                &text_query,\n                &TopDocs::with_limit(2).and_offset(1).order_by_score(),\n            )\n            .unwrap();\n        assert_results_equals(\n            &score_docs[..],\n            &[\n                (0.5376842, DocAddress::new(0u32, 2)),\n                (0.48527452, DocAddress::new(0, 0)),\n            ],\n        );\n    }\n\n    #[test]\n    fn test_top_collector_stable_sorting() {\n        let index = make_index().unwrap();\n\n        // using AllQuery to get a constant score\n        let searcher = index.reader().unwrap().searcher();\n\n        let page_0 = searcher\n            .search(&AllQuery, &TopDocs::with_limit(1).order_by_score())\n            .unwrap();\n\n        let page_1 = searcher\n            .search(&AllQuery, &TopDocs::with_limit(2).order_by_score())\n            .unwrap();\n\n        let page_2 = searcher\n            .search(&AllQuery, &TopDocs::with_limit(3).order_by_score())\n            .unwrap();\n\n        // precondition for the test to be meaningful: we did get documents\n        // with the same score\n        assert!(page_0.iter().all(|result| result.0 == page_1[0].0));\n        assert!(page_1.iter().all(|result| result.0 == page_1[0].0));\n        assert!(page_2.iter().all(|result| result.0 == page_2[0].0));\n\n        // sanity check since we're relying on make_index()\n        assert_eq!(page_0.len(), 1);\n        assert_eq!(page_1.len(), 2);\n        assert_eq!(page_2.len(), 3);\n\n        assert_eq!(page_1, &page_2[..page_1.len()]);\n        assert_eq!(page_0, &page_2[..page_0.len()]);\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(20))]\n        /// Build multiple segments with equal-scoring docs and verify stable ordering\n        /// across pages when increasing limit or offset.\n        #[test]\n        fn proptest_stable_ordering_across_segments_with_pagination(\n            docs_per_segment in proptest::collection::vec(1usize..50, 2..5)\n        ) {\n            use crate::indexer::NoMergePolicy;\n\n            // Build an index with multiple segments; all docs will have the same score using AllQuery.\n            let mut schema_builder = Schema::builder();\n            let text = schema_builder.add_text_field(\"text\", TEXT);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            let mut writer = index.writer_for_tests().unwrap();\n            writer.set_merge_policy(Box::new(NoMergePolicy));\n\n            for num_docs in &docs_per_segment {\n                for _ in 0..*num_docs {\n                    writer.add_document(doc!(text => \"x\")).unwrap();\n                }\n                writer.commit().unwrap();\n            }\n\n            let reader = index.reader().unwrap();\n            let searcher = reader.searcher();\n\n            let total_docs: usize = docs_per_segment.iter().sum();\n            // Full result set, first assert all scores are identical.\n            let full_with_scores: Vec<(Score, DocAddress)> = searcher\n                .search(&AllQuery, &TopDocs::with_limit(total_docs).order_by_score())\n                .unwrap();\n            // Sanity: at least one document was returned.\n            prop_assert!(!full_with_scores.is_empty());\n            let first_score = full_with_scores[0].0;\n            prop_assert!(full_with_scores.iter().all(|(score, _)| *score == first_score));\n\n            // Keep only the addresses for the remaining checks.\n            let full: Vec<DocAddress> = full_with_scores\n                .into_iter()\n                .map(|(_score, addr)| addr)\n                .collect();\n\n            // Sanity: we actually created multiple segments and have documents.\n            prop_assert!(docs_per_segment.len() >= 2);\n            prop_assert!(total_docs >= 2);\n\n            // 1) Increasing limit should preserve prefix ordering.\n            for k in 1..=total_docs {\n                let page: Vec<DocAddress> = searcher\n                    .search(&AllQuery, &TopDocs::with_limit(k).order_by_score())\n                    .unwrap()\n                    .into_iter()\n                    .map(|(_score, addr)| addr)\n                    .collect();\n                prop_assert_eq!(page, full[..k].to_vec());\n            }\n\n            // 2) Offset + limit pages should always match the corresponding slice.\n            //    For each offset, check three representative page sizes:\n            //    - first page (size 1)\n            //    - a middle page (roughly half of remaining)\n            //    - the last page (size = remaining)\n            for offset in 0..total_docs {\n                let remaining = total_docs - offset;\n\n                let assert_page_eq = |limit: usize| -> proptest::test_runner::TestCaseResult {\n                    let page: Vec<DocAddress> = searcher\n                        .search(&AllQuery, &TopDocs::with_limit(limit).and_offset(offset).order_by_score())\n                        .unwrap()\n                        .into_iter()\n                        .map(|(_score, addr)| addr)\n                        .collect();\n                    prop_assert_eq!(page, full[offset..offset + limit].to_vec());\n                    Ok(())\n                };\n\n                // Smallest page.\n                assert_page_eq(1)?;\n                // A middle-sized page (dedupes to 1 if remaining == 1).\n                assert_page_eq((remaining / 2).max(1))?;\n                // Largest page for this offset.\n                assert_page_eq(remaining)?;\n            }\n\n            // 3) Concatenating fixed-size pages by offset reproduces the full order.\n            for page_size in 1..=total_docs.min(5) {\n                let mut concat: Vec<DocAddress> = Vec::new();\n                let mut offset = 0;\n                while offset < total_docs {\n                    let size = page_size.min(total_docs - offset);\n                    let page: Vec<DocAddress> = searcher\n                        .search(&AllQuery, &TopDocs::with_limit(size).and_offset(offset).order_by_score())\n                        .unwrap()\n                        .into_iter()\n                        .map(|(_score, addr)| addr)\n                        .collect();\n                    concat.extend(page);\n                    offset += size;\n                }\n                // Avoid moving `full` across loop iterations.\n                prop_assert_eq!(concat, full.clone());\n            }\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(20))]\n        /// Build multiple segments with same-scoring term matches and verify stable ordering\n        /// across pages for a real scoring query (TermQuery with identical TF and fieldnorm).\n        #[test]\n        fn proptest_stable_ordering_across_segments_with_term_query_and_pagination(\n            docs_per_segment in proptest::collection::vec(1usize..50, 2..5)\n        ) {\n            use crate::indexer::NoMergePolicy;\n            use crate::schema::IndexRecordOption;\n            use crate::query::TermQuery;\n            use crate::Term;\n\n            // Build an index with multiple segments; each doc has exactly one token \"x\",\n            // ensuring equal BM25 scores across all matching docs (same TF=1 and fieldnorm=1).\n            let mut schema_builder = Schema::builder();\n            let text = schema_builder.add_text_field(\"text\", TEXT);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            let mut writer = index.writer_for_tests().unwrap();\n            writer.set_merge_policy(Box::new(NoMergePolicy));\n\n            for num_docs in &docs_per_segment {\n                for _ in 0..*num_docs {\n                    writer.add_document(doc!(text => \"x\")).unwrap();\n                }\n                writer.commit().unwrap();\n            }\n\n            let reader = index.reader().unwrap();\n            let searcher = reader.searcher();\n\n            let total_docs: usize = docs_per_segment.iter().sum();\n            let term = Term::from_field_text(text, \"x\");\n            let tq = TermQuery::new(term, IndexRecordOption::WithFreqs);\n\n            // Full result set, first assert all scores are identical across docs.\n            let full_with_scores: Vec<(Score, DocAddress)> = searcher\n                .search(&tq, &TopDocs::with_limit(total_docs).order_by_score())\n                .unwrap();\n            // Sanity: at least one document was returned.\n            prop_assert!(!full_with_scores.is_empty());\n            let first_score = full_with_scores[0].0;\n            prop_assert!(full_with_scores.iter().all(|(score, _)| *score == first_score));\n\n            // Keep only the addresses for the remaining checks.\n            let full: Vec<DocAddress> = full_with_scores\n                .into_iter()\n                .map(|(_score, addr)| addr)\n                .collect();\n\n            // Sanity: we actually created multiple segments and have documents.\n            prop_assert!(docs_per_segment.len() >= 2);\n            prop_assert!(total_docs >= 2);\n\n            // 1) Increasing limit should preserve prefix ordering.\n            for k in 1..=total_docs {\n                let page: Vec<DocAddress> = searcher\n                    .search(&tq, &TopDocs::with_limit(k).order_by_score())\n                    .unwrap()\n                    .into_iter()\n                    .map(|(_score, addr)| addr)\n                    .collect();\n                prop_assert_eq!(page, full[..k].to_vec());\n            }\n\n            // 2) Offset + limit pages should always match the corresponding slice.\n            //    Check three representative page sizes for each offset: 1, ~half, and remaining.\n            for offset in 0..total_docs {\n                let remaining = total_docs - offset;\n\n                let assert_page_eq = |limit: usize| -> proptest::test_runner::TestCaseResult {\n                    let page: Vec<DocAddress> = searcher\n                        .search(&tq, &TopDocs::with_limit(limit).and_offset(offset).order_by_score())\n                        .unwrap()\n                        .into_iter()\n                        .map(|(_score, addr)| addr)\n                        .collect();\n                    prop_assert_eq!(page, full[offset..offset + limit].to_vec());\n                    Ok(())\n                };\n\n                assert_page_eq(1)?;\n                assert_page_eq((remaining / 2).max(1))?;\n                assert_page_eq(remaining)?;\n            }\n\n            // 3) Concatenating fixed-size pages by offset reproduces the full order.\n            for page_size in 1..=total_docs.min(5) {\n                let mut concat: Vec<DocAddress> = Vec::new();\n                let mut offset = 0;\n                while offset < total_docs {\n                    let size = page_size.min(total_docs - offset);\n                    let page: Vec<DocAddress> = searcher\n                        .search(&tq, &TopDocs::with_limit(size).and_offset(offset).order_by_score())\n                        .unwrap()\n                        .into_iter()\n                        .map(|(_score, addr)| addr)\n                        .collect();\n                    concat.extend(page);\n                    offset += size;\n                }\n                prop_assert_eq!(concat, full.clone());\n            }\n        }\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_top_0() {\n        TopDocs::with_limit(0);\n    }\n\n    const TITLE: &str = \"title\";\n    const SIZE: &str = \"size\";\n\n    #[test]\n    fn test_top_field_collector_not_at_capacity() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(TITLE, TEXT);\n        let size = schema_builder.add_u64_field(SIZE, FAST);\n        let schema = schema_builder.build();\n        let (index, query) = index(\"beer\", title, schema, |index_writer| {\n            index_writer\n                .add_document(doc!(\n                    title => \"bottle of beer\",\n                    size => 12u64,\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    title => \"growler of beer\",\n                    size => 64u64,\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    title => \"pint of beer\",\n                    size => 16u64,\n                ))\n                .unwrap();\n        });\n        let searcher = index.reader()?.searcher();\n\n        let top_collector = TopDocs::with_limit(4).order_by_u64_field(SIZE, Order::Desc);\n        let top_docs: Vec<(Option<u64>, DocAddress)> = searcher.search(&query, &top_collector)?;\n        assert_eq!(\n            &top_docs[..],\n            &[\n                (Some(64), DocAddress::new(0, 1)),\n                (Some(16), DocAddress::new(0, 2)),\n                (Some(12), DocAddress::new(0, 0))\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_field_collector_datetime() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name = schema_builder.add_text_field(\"name\", TEXT);\n        let birthday = schema_builder.add_date_field(\"birthday\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        let pr_birthday = DateTime::from_utc(OffsetDateTime::parse(\n            \"1898-04-09T00:00:00+00:00\",\n            &Rfc3339,\n        )?);\n        index_writer.add_document(doc!(\n            name => \"Paul Robeson\",\n            birthday => pr_birthday,\n        ))?;\n        let mr_birthday = DateTime::from_utc(OffsetDateTime::parse(\n            \"1947-11-08T00:00:00+00:00\",\n            &Rfc3339,\n        )?);\n        index_writer.add_document(doc!(\n            name => \"Minnie Riperton\",\n            birthday => mr_birthday,\n        ))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let top_collector = TopDocs::with_limit(3).order_by_fast_field(\"birthday\", Order::Desc);\n        let top_docs: Vec<(Option<DateTime>, DocAddress)> =\n            searcher.search(&AllQuery, &top_collector)?;\n        assert_eq!(\n            &top_docs[..],\n            &[\n                (Some(mr_birthday), DocAddress::new(0, 1)),\n                (Some(pr_birthday), DocAddress::new(0, 0)),\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_field_collector_i64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let city = schema_builder.add_text_field(\"city\", TEXT);\n        let altitude = schema_builder.add_i64_field(\"altitude\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(\n                city => \"georgetown\",\n                altitude =>  -1i64,\n        ))?;\n        index_writer.add_document(doc!(\n            city => \"tokyo\",\n            altitude =>  40i64,\n        ))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let top_collector = TopDocs::with_limit(3).order_by_fast_field(\"altitude\", Order::Desc);\n        let top_docs: Vec<(Option<i64>, DocAddress)> =\n            searcher.search(&AllQuery, &top_collector)?;\n        assert_eq!(\n            &top_docs[..],\n            &[\n                (Some(40i64), DocAddress::new(0, 1)),\n                (Some(-1i64), DocAddress::new(0, 0)),\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_field_collector_f64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let city = schema_builder.add_text_field(\"city\", TEXT);\n        let altitude = schema_builder.add_f64_field(\"altitude\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(\n                city => \"georgetown\",\n                altitude =>  -1.0f64,\n        ))?;\n        index_writer.add_document(doc!(\n            city => \"tokyo\",\n            altitude =>  40f64,\n        ))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let top_collector = TopDocs::with_limit(3).order_by_fast_field(\"altitude\", Order::Desc);\n        let top_docs: Vec<(Option<f64>, DocAddress)> =\n            searcher.search(&AllQuery, &top_collector)?;\n        assert_eq!(\n            &top_docs[..],\n            &[\n                (Some(40f64), DocAddress::new(0, 1)),\n                (Some(-1.0f64), DocAddress::new(0, 0)),\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_top_field_collector_string() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let city = schema_builder.add_text_field(\"city\", TEXT | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(\n                city => \"austin\",\n        ))?;\n        index_writer.add_document(doc!(\n                city => \"greenville\",\n        ))?;\n        index_writer.add_document(doc!(\n            city => \"tokyo\",\n        ))?;\n        index_writer.commit()?;\n\n        fn query(\n            index: &Index,\n            order: Order,\n            limit: usize,\n            offset: usize,\n        ) -> crate::Result<Vec<(Option<String>, DocAddress)>> {\n            let searcher = index.reader()?.searcher();\n            let top_collector = TopDocs::with_limit(limit)\n                .and_offset(offset)\n                .order_by_string_fast_field(\"city\", order);\n            searcher.search(&AllQuery, &top_collector)\n        }\n\n        assert_eq!(\n            &query(&index, Order::Desc, 3, 0)?,\n            &[\n                (Some(\"tokyo\".to_owned()), DocAddress::new(0, 2)),\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n                (Some(\"austin\".to_owned()), DocAddress::new(0, 0)),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Desc, 2, 0)?,\n            &[\n                (Some(\"tokyo\".to_owned()), DocAddress::new(0, 2)),\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n            ]\n        );\n\n        assert_eq!(&query(&index, Order::Desc, 3, 3)?, &[]);\n\n        assert_eq!(\n            &query(&index, Order::Desc, 2, 1)?,\n            &[\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n                (Some(\"austin\".to_owned()), DocAddress::new(0, 0)),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc, 3, 0)?,\n            &[\n                (Some(\"austin\".to_owned()), DocAddress::new(0, 0)),\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n                (Some(\"tokyo\".to_owned()), DocAddress::new(0, 2)),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc, 2, 1)?,\n            &[\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n                (Some(\"tokyo\".to_owned()), DocAddress::new(0, 2)),\n            ]\n        );\n\n        assert_eq!(\n            &query(&index, Order::Asc, 2, 0)?,\n            &[\n                (Some(\"austin\".to_owned()), DocAddress::new(0, 0)),\n                (Some(\"greenville\".to_owned()), DocAddress::new(0, 1)),\n            ]\n        );\n\n        assert_eq!(&query(&index, Order::Asc, 3, 3)?, &[]);\n\n        Ok(())\n    }\n\n    proptest! {\n        #[test]\n        fn test_top_field_collect_string_prop(\n          order in prop_oneof!(Just(Order::Desc), Just(Order::Asc)),\n          limit in 1..256_usize,\n          offset in 0..256_usize,\n          segments_terms in\n            proptest::collection::vec(\n                proptest::collection::vec(0..32_u8, 1..32_usize),\n                0..8_usize,\n            )\n        ) {\n            let mut schema_builder = Schema::builder();\n            let city = schema_builder.add_text_field(\"city\", TEXT | FAST);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            let mut index_writer = index.writer_for_tests()?;\n\n            // A Vec<Vec<u8>>, where the outer Vec represents segments, and the inner Vec\n            // represents terms.\n            for segment_terms in segments_terms.into_iter() {\n                for term in segment_terms.into_iter() {\n                    let term = format!(\"{term:0>3}\");\n                    index_writer.add_document(doc!(\n                        city => term,\n                    ))?;\n                }\n                index_writer.commit()?;\n            }\n\n            let searcher = index.reader()?.searcher();\n            let top_n_results = searcher.search(&AllQuery, &TopDocs::with_limit(limit)\n                .and_offset(offset)\n                .order_by_string_fast_field(\"city\", order))?;\n            let all_results = searcher.search(&AllQuery, &DocSetCollector)?.into_iter().map(|doc_address| {\n                // Get the term for this address.\n                // NOTE: We can't determine the SegmentIds that will be generated for Segments\n                // ahead of time, so we can't pre-compute the expected `DocAddress`es.\n                let column = searcher.segment_readers()[doc_address.segment_ord as usize].fast_fields().str(\"city\").unwrap().unwrap();\n                let term_ord = column.term_ords(doc_address.doc_id).next().unwrap();\n                let mut city = Vec::new();\n                column.dictionary().ord_to_term(term_ord, &mut city).unwrap();\n                (Some(String::try_from(city).unwrap()), doc_address)\n            });\n\n            // Using the TopDocs collector should always be equivalent to sorting, skipping the\n            // offset, and then taking the limit.\n            let sorted_docs: Vec<_> = {\n                let mut comparable_docs: Vec<ComparableDoc<_, _>> =\n                    all_results.into_iter().map(|(sort_key, doc)| ComparableDoc { sort_key, doc}).collect();\n                crate::collector::sort_key::tests::sort_hits(&mut comparable_docs, order);\n                comparable_docs.into_iter().map(|cd| (cd.sort_key, cd.doc)).collect()\n            };\n            let expected_docs = sorted_docs.into_iter().skip(offset).take(limit).collect::<Vec<_>>();\n            prop_assert_eq!(\n                expected_docs,\n                top_n_results\n            );\n        }\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_field_does_not_exist() {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(TITLE, TEXT);\n        let size = schema_builder.add_u64_field(SIZE, FAST);\n        let schema = schema_builder.build();\n        let (index, _) = index(\"beer\", title, schema, |index_writer| {\n            index_writer\n                .add_document(doc!(\n                    title => \"bottle of beer\",\n                    size => 12u64,\n                ))\n                .unwrap();\n        });\n        let searcher = index.reader().unwrap().searcher();\n        let top_collector = TopDocs::with_limit(4).order_by_u64_field(\"missing_field\", Order::Desc);\n        let segment_reader = searcher.segment_reader(0u32);\n        top_collector\n            .for_segment(0, segment_reader)\n            .expect(\"should panic\");\n    }\n\n    #[test]\n    fn test_field_not_fast_field() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let size = schema_builder.add_u64_field(SIZE, STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(size=>1u64))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let segment = searcher.segment_reader(0);\n        let top_collector = TopDocs::with_limit(4).order_by_u64_field(SIZE, Order::Desc);\n        let err = top_collector.for_segment(0, segment).err().unwrap();\n        assert!(matches!(err, crate::TantivyError::InvalidArgument(_)));\n        Ok(())\n    }\n\n    #[test]\n    fn test_field_wrong_type() {\n        let mut schema_builder = Schema::builder();\n        let _size = schema_builder.add_u64_field(SIZE, STORED);\n        let schema = schema_builder.build();\n        let top_collector = TopDocs::with_limit(4).order_by_fast_field::<i64>(SIZE, Order::Desc);\n        let err = top_collector.check_schema(&schema).err().unwrap();\n        assert!(\n            matches!(err, crate::TantivyError::SchemaError(msg) if msg == \"Field `size` is not a fast field.\")\n        );\n    }\n\n    #[test]\n    fn test_sort_key_top_collector_with_offset() -> crate::Result<()> {\n        let index = make_index()?;\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\")?;\n        let collector = TopDocs::with_limit(2)\n            .and_offset(1)\n            .order_by(move |_segment_reader: &SegmentReader| move |doc: DocId| doc);\n        let score_docs: Vec<(u32, DocAddress)> =\n            index.reader()?.searcher().search(&text_query, &collector)?;\n        assert_eq!(\n            score_docs,\n            vec![(1, DocAddress::new(0, 1)), (0, DocAddress::new(0, 0)),]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_custom_score_top_collector_with_offset() {\n        let index = make_index().unwrap();\n        let field = index.schema().get_field(\"text\").unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![field]);\n        let text_query = query_parser.parse_query(\"droopy tax\").unwrap();\n        let collector = TopDocs::with_limit(2)\n            .and_offset(1)\n            .order_by(move |_segment_reader: &SegmentReader| move |doc: DocId| doc);\n        let score_docs: Vec<(u32, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(&text_query, &collector)\n            .unwrap();\n\n        assert_eq!(\n            score_docs,\n            vec![(1, DocAddress::new(0, 1)), (0, DocAddress::new(0, 0)),]\n        );\n    }\n\n    fn index(\n        query: &str,\n        query_field: Field,\n        schema: Schema,\n        mut doc_adder: impl FnMut(&mut IndexWriter),\n    ) -> (Index, Box<dyn Query>) {\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_with_num_threads(1, 15_000_000).unwrap();\n        doc_adder(&mut index_writer);\n        index_writer.commit().unwrap();\n        let query_parser = QueryParser::for_index(&index, vec![query_field]);\n        let query = query_parser.parse_query(query).unwrap();\n        (index, query)\n    }\n    #[test]\n    fn test_fast_field_ascending_order() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(TITLE, TEXT);\n        let size = schema_builder.add_u64_field(SIZE, FAST);\n        let schema = schema_builder.build();\n        let (index, query) = index(\"beer\", title, schema, |index_writer| {\n            index_writer\n                .add_document(doc!(\n                    title => \"bottle of beer\",\n                    size => 12u64,\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    title => \"growler of beer\",\n                    size => 64u64,\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    title => \"pint of beer\",\n                    size => 16u64,\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    title => \"empty beer\",\n                ))\n                .unwrap();\n        });\n        let searcher = index.reader()?.searcher();\n\n        let top_collector = TopDocs::with_limit(4).order_by_fast_field(SIZE, Order::Asc);\n        let top_docs: Vec<(Option<u64>, DocAddress)> = searcher.search(&query, &top_collector)?;\n        assert_eq!(\n            &top_docs[..],\n            &[\n                (Some(12), DocAddress::new(0, 0)),\n                (Some(16), DocAddress::new(0, 2)),\n                (Some(64), DocAddress::new(0, 1)),\n                (None, DocAddress::new(0, 3)),\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_topn_computer_desc() {\n        let mut computer: TopNComputer<u32, u32, _> =\n            TopNComputer::new_with_comparator(2, ComparatorEnum::from(Order::Desc));\n\n        computer.push(1u32, 1u32);\n        computer.push(2u32, 2u32);\n        computer.push(3u32, 3u32);\n        computer.push(2u32, 4u32);\n        computer.push(4u32, 5u32);\n        computer.push(1u32, 6u32);\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 4u32,\n                    doc: 5u32,\n                },\n                ComparableDoc {\n                    sort_key: 3u32,\n                    doc: 3u32,\n                }\n            ]\n        );\n    }\n\n    #[test]\n    fn test_topn_computer_asc() {\n        let mut computer: TopNComputer<u32, u32, _> =\n            TopNComputer::new_with_comparator(2, ComparatorEnum::from(Order::Asc));\n        computer.push(1u32, 1u32);\n        computer.push(2u32, 2u32);\n        computer.push(3u32, 3u32);\n        computer.push(2u32, 4u32);\n        computer.push(4u32, 5u32);\n        computer.push(1u32, 6u32);\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 1u32,\n                    doc: 1u32,\n                },\n                ComparableDoc {\n                    sort_key: 1u32,\n                    doc: 6u32,\n                }\n            ]\n        );\n    }\n\n    #[test]\n    fn test_topn_computer_option_asc_null_at_the_end() {\n        let mut computer: TopNComputer<Option<u32>, u32, _> =\n            TopNComputer::new_with_comparator(2, ComparatorEnum::ReverseNoneLower);\n        computer.push(Some(1u32), 1u32);\n        computer.push(Some(2u32), 2u32);\n        computer.push(None, 3u32);\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: Some(1u32),\n                    doc: 1u32,\n                },\n                ComparableDoc {\n                    sort_key: Some(2u32),\n                    doc: 2u32,\n                }\n            ]\n        );\n    }\n\n    #[test]\n    fn test_topn_computer_option_asc_null_at_the_begining() {\n        let mut computer: TopNComputer<Option<u32>, u32, _> =\n            TopNComputer::new_with_comparator(2, ComparatorEnum::Reverse);\n        computer.push(Some(1u32), 1u32);\n        computer.push(Some(2u32), 2u32);\n        computer.push(None, 3u32);\n        assert_eq!(\n            computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: None,\n                    doc: 3u32,\n                },\n                ComparableDoc {\n                    sort_key: Some(1u32),\n                    doc: 1u32,\n                },\n            ]\n        );\n    }\n\n    #[test]\n    fn test_push_assuming_capacity() {\n        let mut vec = Vec::with_capacity(2);\n        super::push_assuming_capacity(1, &mut vec);\n        assert_eq!(&vec, &[1]);\n        super::push_assuming_capacity(2, &mut vec);\n        assert_eq!(&vec, &[1, 2]);\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_push_assuming_capacity_panics_when_no_cap() {\n        let mut vec = Vec::with_capacity(1);\n        super::push_assuming_capacity(1, &mut vec);\n        assert_eq!(&vec, &[1]);\n        super::push_assuming_capacity(2, &mut vec);\n    }\n\n    #[test]\n    fn test_top_n_computer_not_at_capacity() {\n        let mut top_n_computer = TopNComputer::new_with_comparator(4, NaturalComparator);\n        top_n_computer.append_doc(1, 0.8);\n        top_n_computer.append_doc(3, 0.2);\n        top_n_computer.append_doc(5, 0.3);\n        assert_eq!(\n            &top_n_computer.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 0.8,\n                    doc: 1\n                },\n                ComparableDoc {\n                    sort_key: 0.3,\n                    doc: 5\n                },\n                ComparableDoc {\n                    sort_key: 0.2,\n                    doc: 3\n                },\n            ]\n        );\n    }\n\n    #[test]\n    fn test_top_n_computer_at_capacity() {\n        let mut top_collector = TopNComputer::new_with_comparator(4, NaturalComparator);\n        top_collector.append_doc(1, 0.8);\n        top_collector.append_doc(3, 0.2);\n        top_collector.append_doc(5, 0.3);\n        top_collector.append_doc(7, 0.9);\n        top_collector.append_doc(9, -0.2);\n        assert_eq!(\n            &top_collector.into_sorted_vec(),\n            &[\n                ComparableDoc {\n                    sort_key: 0.9,\n                    doc: 7\n                },\n                ComparableDoc {\n                    sort_key: 0.8,\n                    doc: 1\n                },\n                ComparableDoc {\n                    sort_key: 0.3,\n                    doc: 5\n                },\n                ComparableDoc {\n                    sort_key: 0.2,\n                    doc: 3\n                },\n            ]\n        );\n    }\n\n    #[test]\n    fn test_top_segment_collector_stable_ordering_for_equal_feature() {\n        // given that the documents are collected in ascending doc id order,\n        // when harvesting we have to guarantee stable sorting in case of a tie\n        // on the score\n        let doc_ids_collection = [4, 5, 6];\n        let score = 3.3f32;\n\n        let mut top_collector_limit_2 = TopNComputer::new_with_comparator(2, NaturalComparator);\n        for id in &doc_ids_collection {\n            top_collector_limit_2.append_doc(*id, score);\n        }\n\n        let mut top_collector_limit_3 = TopNComputer::new_with_comparator(3, NaturalComparator);\n        for id in &doc_ids_collection {\n            top_collector_limit_3.append_doc(*id, score);\n        }\n\n        let docs_limit_2 = top_collector_limit_2.into_sorted_vec();\n        let docs_limit_3 = top_collector_limit_3.into_sorted_vec();\n\n        assert_eq!(&docs_limit_2, &docs_limit_3[..2],);\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n    use test::Bencher;\n\n    use super::TopNComputer;\n    use crate::collector::sort_key::NaturalComparator;\n\n    #[bench]\n    fn bench_top_segment_collector_collect_at_capacity(b: &mut Bencher) {\n        let mut top_collector = TopNComputer::new_with_comparator(100, NaturalComparator);\n\n        for i in 0..100 {\n            top_collector.append_doc(i, 0.8);\n        }\n\n        b.iter(|| {\n            for i in 0..100 {\n                top_collector.append_doc(i, 0.8);\n            }\n        });\n    }\n}\n"
  },
  {
    "path": "src/compat_tests.rs",
    "content": "use std::path::PathBuf;\n\nuse schema::*;\n\nuse crate::*;\n\nfn create_index(path: &str) {\n    let mut schema_builder = Schema::builder();\n    let label = schema_builder.add_text_field(\"label\", TEXT | STORED);\n    let date = schema_builder.add_date_field(\"date\", INDEXED | STORED);\n    let schema = schema_builder.build();\n    std::fs::create_dir_all(path).unwrap();\n    let index = Index::create_in_dir(path, schema).unwrap();\n    let mut index_writer = index.writer_with_num_threads(1, 20_000_000).unwrap();\n    index_writer\n        .add_document(doc!(label => \"dateformat\", date => DateTime::from_timestamp_nanos(123456)))\n        .unwrap();\n    index_writer.commit().unwrap();\n}\n\n#[test]\n/// Writes an Index for the current INDEX_FORMAT_VERSION to disk.\nfn create_format() {\n    let version = INDEX_FORMAT_VERSION.to_string();\n    let file_path = path_for_version(&version);\n    if PathBuf::from(file_path.clone()).exists() {\n        return;\n    }\n    create_index(&file_path);\n}\n\nfn path_for_version(version: &str) -> String {\n    format!(\"./tests/compat_tests_data/index_v{version}/\")\n}\n\n/// feature flag quickwit uses a different dictionary type\n#[test]\n#[cfg(not(feature = \"quickwit\"))]\nfn test_format_6() {\n    let path = path_for_version(\"6\");\n\n    let index = Index::open_in_dir(path).expect(\"Failed to open index\");\n    // dates are truncated to Microseconds in v6\n    assert_date_time_precision(&index, DateTimePrecision::Microseconds);\n}\n\n/// feature flag quickwit uses a different dictionary type\n#[test]\n#[cfg(not(feature = \"quickwit\"))]\nfn test_format_7() {\n    let path = path_for_version(\"7\");\n\n    let index = Index::open_in_dir(path).expect(\"Failed to open index\");\n    // dates are not truncated in v7 in the docstore\n    assert_date_time_precision(&index, DateTimePrecision::Nanoseconds);\n}\n\n#[cfg(not(feature = \"quickwit\"))]\nfn assert_date_time_precision(index: &Index, doc_store_precision: DateTimePrecision) {\n    use collector::TopDocs;\n    let reader = index.reader().expect(\"Failed to create reader\");\n    let searcher = reader.searcher();\n\n    let schema = index.schema();\n    let label_field = schema.get_field(\"label\").expect(\"Field 'label' not found\");\n    let query_parser = query::QueryParser::for_index(index, vec![label_field]);\n\n    let query = query_parser\n        .parse_query(\"dateformat\")\n        .expect(\"Failed to parse query\");\n    let top_docs = searcher\n        .search(&query, &TopDocs::with_limit(1).order_by_score())\n        .expect(\"Search failed\");\n\n    assert_eq!(top_docs.len(), 1, \"Expected 1 search result\");\n\n    let doc_address = top_docs[0].1;\n    let retrieved_doc: TantivyDocument = searcher\n        .doc(doc_address)\n        .expect(\"Failed to retrieve document\");\n\n    let date_field = schema.get_field(\"date\").expect(\"Field 'date' not found\");\n    let date_value = retrieved_doc\n        .get_first(date_field)\n        .expect(\"Date field not found in document\")\n        .as_datetime()\n        .unwrap();\n\n    let expected = DateTime::from_timestamp_nanos(123456).truncate(doc_store_precision);\n    assert_eq!(date_value, expected,);\n}\n"
  },
  {
    "path": "src/core/executor.rs",
    "content": "use std::sync::Arc;\n\n#[cfg(feature = \"quickwit\")]\nuse futures_util::{future::Either, FutureExt};\n\nuse crate::TantivyError;\n\n/// Executor makes it possible to run tasks in single thread or\n/// in a thread pool.\n#[derive(Clone)]\npub enum Executor {\n    /// Single thread variant of an Executor\n    SingleThread,\n    /// Thread pool variant of an Executor\n    ThreadPool(Arc<rayon::ThreadPool>),\n}\n\n#[cfg(feature = \"quickwit\")]\nimpl From<Arc<rayon::ThreadPool>> for Executor {\n    fn from(thread_pool: Arc<rayon::ThreadPool>) -> Self {\n        Executor::ThreadPool(thread_pool)\n    }\n}\n\nimpl Executor {\n    /// Creates an Executor that performs all task in the caller thread.\n    pub fn single_thread() -> Executor {\n        Executor::SingleThread\n    }\n\n    /// Creates an Executor that dispatches the tasks in a thread pool.\n    pub fn multi_thread(num_threads: usize, prefix: &'static str) -> crate::Result<Executor> {\n        let pool = rayon::ThreadPoolBuilder::new()\n            .num_threads(num_threads)\n            .thread_name(move |num| format!(\"{prefix}{num}\"))\n            .build()?;\n        Ok(Executor::ThreadPool(Arc::new(pool)))\n    }\n\n    /// Perform a map in the thread pool.\n    ///\n    /// Regardless of the executor (`SingleThread` or `ThreadPool`), panics in the task\n    /// will propagate to the caller.\n    pub fn map<A, R, F>(&self, f: F, args: impl Iterator<Item = A>) -> crate::Result<Vec<R>>\n    where\n        A: Send,\n        R: Send,\n        F: Sized + Sync + Fn(A) -> crate::Result<R>,\n    {\n        match self {\n            Executor::SingleThread => {\n                // Avoid `collect`, since the stacktrace is blown up by it, which makes profiling\n                // harder.\n                let mut result = Vec::with_capacity(args.size_hint().0);\n                for arg in args {\n                    result.push(f(arg)?);\n                }\n                Ok(result)\n            }\n            Executor::ThreadPool(pool) => {\n                let args: Vec<A> = args.collect();\n                let num_fruits = args.len();\n                let fruit_receiver = {\n                    let (fruit_sender, fruit_receiver) = crossbeam_channel::unbounded();\n                    pool.scope(|scope| {\n                        for (idx, arg) in args.into_iter().enumerate() {\n                            // We name references for f and fruit_sender_ref because we do not\n                            // want these two to be moved into the closure.\n                            let f_ref = &f;\n                            let fruit_sender_ref = &fruit_sender;\n                            scope.spawn(move |_| {\n                                let fruit = f_ref(arg);\n                                if let Err(err) = fruit_sender_ref.send((idx, fruit)) {\n                                    error!(\n                                        \"Failed to send search task. It probably means all search \\\n                                         threads have panicked. {err:?}\"\n                                    );\n                                }\n                            });\n                        }\n                    });\n                    fruit_receiver\n                    // This ends the scope of fruit_sender.\n                    // This is important as it makes it possible for the fruit_receiver iteration to\n                    // terminate.\n                };\n                let mut result_placeholders: Vec<Option<R>> =\n                    std::iter::repeat_with(|| None).take(num_fruits).collect();\n                for (pos, fruit_res) in fruit_receiver {\n                    let fruit = fruit_res?;\n                    result_placeholders[pos] = Some(fruit);\n                }\n                let results: Vec<R> = result_placeholders.into_iter().flatten().collect();\n                if results.len() != num_fruits {\n                    return Err(TantivyError::InternalError(\n                        \"One of the mapped execution failed.\".to_string(),\n                    ));\n                }\n                Ok(results)\n            }\n        }\n    }\n\n    /// Spawn a task on the pool, returning a future completing on task success.\n    ///\n    /// If the task panics, returns `Err(())`.\n    #[cfg(feature = \"quickwit\")]\n    pub fn spawn_blocking<T: Send + 'static>(\n        &self,\n        cpu_intensive_task: impl FnOnce() -> T + Send + 'static,\n    ) -> impl std::future::Future<Output = Result<T, ()>> {\n        match self {\n            Executor::SingleThread => Either::Left(std::future::ready(Ok(cpu_intensive_task()))),\n            Executor::ThreadPool(pool) => {\n                let (sender, receiver) = oneshot::channel();\n                pool.spawn(|| {\n                    if sender.is_closed() {\n                        return;\n                    }\n                    let task_result = cpu_intensive_task();\n                    let _ = sender.send(task_result);\n                });\n\n                let res = receiver.map(|res| res.map_err(|_| ()));\n                Either::Right(res)\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::Executor;\n\n    #[test]\n    #[should_panic(expected = \"panic should propagate\")]\n    fn test_panic_propagates_single_thread() {\n        let _result: Vec<usize> = Executor::single_thread()\n            .map(\n                |_| {\n                    panic!(\"panic should propagate\");\n                },\n                vec![0].into_iter(),\n            )\n            .unwrap();\n    }\n\n    #[test]\n    #[should_panic] //< unfortunately the panic message is not propagated\n    fn test_panic_propagates_multi_thread() {\n        let _result: Vec<usize> = Executor::multi_thread(1, \"search-test\")\n            .unwrap()\n            .map(\n                |_| {\n                    panic!(\"panic should propagate\");\n                },\n                vec![0].into_iter(),\n            )\n            .unwrap();\n    }\n\n    #[test]\n    fn test_map_singlethread() {\n        let result: Vec<usize> = Executor::single_thread()\n            .map(|i| Ok(i * 2), 0..1_000)\n            .unwrap();\n        assert_eq!(result.len(), 1_000);\n        for i in 0..1_000 {\n            assert_eq!(result[i], i * 2);\n        }\n    }\n\n    #[test]\n    fn test_map_multithread() {\n        let result: Vec<usize> = Executor::multi_thread(3, \"search-test\")\n            .unwrap()\n            .map(|i| Ok(i * 2), 0..10)\n            .unwrap();\n        assert_eq!(result.len(), 10);\n        for i in 0..10 {\n            assert_eq!(result[i], i * 2);\n        }\n    }\n\n    #[cfg(feature = \"quickwit\")]\n    #[test]\n    fn test_cancel_cpu_intensive_tasks() {\n        use std::sync::atomic::{AtomicU64, Ordering};\n        use std::sync::Arc;\n\n        let counter: Arc<AtomicU64> = Default::default();\n\n        let other_counter: Arc<AtomicU64> = Default::default();\n\n        let mut futures = Vec::new();\n        let mut other_futures = Vec::new();\n\n        let (tx, rx) = crossbeam_channel::bounded::<()>(0);\n        let rx = Arc::new(rx);\n        let executor = Executor::multi_thread(3, \"search-test\").unwrap();\n        for _ in 0..1000 {\n            let counter_clone: Arc<AtomicU64> = counter.clone();\n            let other_counter_clone: Arc<AtomicU64> = other_counter.clone();\n\n            let rx_clone = rx.clone();\n            let rx_clone2 = rx.clone();\n            let fut = executor.spawn_blocking(move || {\n                counter_clone.fetch_add(1, Ordering::SeqCst);\n                let _ = rx_clone.recv();\n            });\n            futures.push(fut);\n            let other_fut = executor.spawn_blocking(move || {\n                other_counter_clone.fetch_add(1, Ordering::SeqCst);\n                let _ = rx_clone2.recv();\n            });\n            other_futures.push(other_fut);\n        }\n\n        // We execute 100 futures.\n        for _ in 0..100 {\n            tx.send(()).unwrap();\n        }\n\n        let counter_val = counter.load(Ordering::SeqCst);\n        let other_counter_val = other_counter.load(Ordering::SeqCst);\n        assert!(counter_val >= 30);\n        assert!(other_counter_val >= 30);\n\n        drop(other_futures);\n\n        // We execute 100 futures.\n        for _ in 0..100 {\n            tx.send(()).unwrap();\n        }\n\n        let counter_val2 = counter.load(Ordering::SeqCst);\n        assert!(counter_val2 >= counter_val + 100 - 6);\n\n        let other_counter_val2 = other_counter.load(Ordering::SeqCst);\n        assert!(other_counter_val2 <= other_counter_val + 6);\n    }\n}\n"
  },
  {
    "path": "src/core/json_utils.rs",
    "content": "use columnar::NumericalValue;\nuse common::json_path_writer::{JSON_END_OF_PATH, JSON_PATH_SEGMENT_SEP};\nuse common::{replace_in_place, JsonPathWriter};\nuse rustc_hash::FxHashMap;\n\nuse crate::indexer::indexing_term::IndexingTerm;\nuse crate::postings::{IndexingContext, IndexingPosition, PostingsWriter};\nuse crate::schema::document::{ReferenceValue, ReferenceValueLeaf, Value};\nuse crate::schema::{Type, DATE_TIME_PRECISION_INDEXED};\nuse crate::time::format_description::well_known::Rfc3339;\nuse crate::time::{OffsetDateTime, UtcOffset};\nuse crate::tokenizer::TextAnalyzer;\nuse crate::{DateTime, DocId, Term};\n\n/// This object is a map storing the last position for a given path for the current document\n/// being indexed.\n///\n/// It is key to solve the following problem:\n/// If we index a JsonObject emitting several terms with the same path\n/// we do not want to create false positive in phrase queries.\n///\n/// For instance:\n///\n/// ```json\n/// {\"bands\": [\n///     {\"band_name\": \"Elliot Smith\"},\n///     {\"band_name\": \"The Who\"},\n/// ]}\n/// ```\n///\n/// If we are careless and index each band names independently,\n/// `Elliot` and `The` will end up indexed at position 0, and `Smith` and `Who` will be indexed at\n/// position 1.\n/// As a result, with lemmatization, \"The Smiths\" will match our object.\n///\n/// Worse, if a same term appears in the second object, a non increasing value would be pushed\n/// to the position recorder probably provoking a panic.\n///\n/// This problem is solved for regular multivalued object by offsetting the position\n/// of values, with a position gap. Here we would like `The` and `Who` to get indexed at\n/// position 2 and 3 respectively.\n///\n/// With regular fields, we sort the fields beforehand, so that all terms with the same\n/// path are indexed consecutively.\n///\n/// In JSON object, we do not have this comfort, so we need to record these position offsets in\n/// a map.\n///\n/// Note that using a single position for the entire object would not hurt correctness.\n/// It would however hurt compression.\n///\n/// We can therefore afford working with a map that is not imperfect. It is fine if several\n/// path map to the same index position as long as the probability is relatively low.\n#[derive(Default)]\npub(crate) struct IndexingPositionsPerPath {\n    positions_per_path: FxHashMap<u32, IndexingPosition>,\n}\n\nimpl IndexingPositionsPerPath {\n    fn get_position_from_id(&mut self, id: u32) -> &mut IndexingPosition {\n        self.positions_per_path.entry(id).or_default()\n    }\n    pub fn clear(&mut self) {\n        self.positions_per_path.clear();\n    }\n}\n\n/// Convert JSON_PATH_SEGMENT_SEP to a dot.\npub fn json_path_sep_to_dot(path: &mut str) {\n    // This is safe since we are replacing a ASCII character by another ASCII character.\n    unsafe {\n        replace_in_place(JSON_PATH_SEGMENT_SEP, b'.', path.as_bytes_mut());\n    }\n}\n\n#[expect(clippy::too_many_arguments)]\nfn index_json_object<'a, V: Value<'a>>(\n    doc: DocId,\n    json_visitor: V::ObjectIter,\n    text_analyzer: &mut TextAnalyzer,\n    term_buffer: &mut IndexingTerm,\n    json_path_writer: &mut JsonPathWriter,\n    postings_writer: &mut dyn PostingsWriter,\n    ctx: &mut IndexingContext,\n    positions_per_path: &mut IndexingPositionsPerPath,\n) {\n    for (json_path_segment, json_value_visitor) in json_visitor {\n        if json_path_segment.as_bytes().contains(&JSON_END_OF_PATH) {\n            continue;\n        }\n        json_path_writer.push(json_path_segment);\n        index_json_value(\n            doc,\n            json_value_visitor,\n            text_analyzer,\n            term_buffer,\n            json_path_writer,\n            postings_writer,\n            ctx,\n            positions_per_path,\n        );\n        json_path_writer.pop();\n    }\n}\n\n#[expect(clippy::too_many_arguments)]\npub(crate) fn index_json_value<'a, V: Value<'a>>(\n    doc: DocId,\n    json_value: V,\n    text_analyzer: &mut TextAnalyzer,\n    term_buffer: &mut IndexingTerm,\n    json_path_writer: &mut JsonPathWriter,\n    postings_writer: &mut dyn PostingsWriter,\n    ctx: &mut IndexingContext,\n    positions_per_path: &mut IndexingPositionsPerPath,\n) {\n    let set_path_id = |term_buffer: &mut IndexingTerm, unordered_id: u32| {\n        term_buffer.truncate_value_bytes(0);\n        term_buffer.append_bytes(&unordered_id.to_be_bytes());\n    };\n    let set_type = |term_buffer: &mut IndexingTerm, typ: Type| {\n        term_buffer.append_bytes(&[typ.to_code()]);\n    };\n\n    match json_value.as_value() {\n        ReferenceValue::Leaf(leaf) => match leaf {\n            ReferenceValueLeaf::Null => {}\n            ReferenceValueLeaf::Str(val) => {\n                let mut token_stream = text_analyzer.token_stream(val);\n                let unordered_id = ctx\n                    .path_to_unordered_id\n                    .get_or_allocate_unordered_id(json_path_writer.as_str());\n\n                // TODO: make sure the chain position works out.\n                set_path_id(term_buffer, unordered_id);\n                set_type(term_buffer, Type::Str);\n                let indexing_position = positions_per_path.get_position_from_id(unordered_id);\n                postings_writer.index_text(\n                    doc,\n                    &mut *token_stream,\n                    term_buffer,\n                    ctx,\n                    indexing_position,\n                );\n            }\n            ReferenceValueLeaf::U64(val) => {\n                // try to parse to i64, since when querying we will apply the same logic and prefer\n                // i64 values\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                if let Ok(i64_val) = val.try_into() {\n                    term_buffer.append_type_and_fast_value::<i64>(i64_val);\n                } else {\n                    term_buffer.append_type_and_fast_value::<u64>(val);\n                }\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::I64(val) => {\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                term_buffer.append_type_and_fast_value(val);\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::F64(val) => {\n                if !val.is_finite() {\n                    return;\n                };\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                // Normalize here is important.\n                // In the inverted index, we coerce all numerical values to their canonical\n                // representation.\n                //\n                // (We do the same thing on the query side)\n                match NumericalValue::F64(val).normalize() {\n                    NumericalValue::I64(val_i64) => {\n                        term_buffer.append_type_and_fast_value::<i64>(val_i64);\n                    }\n                    NumericalValue::U64(val_u64) => {\n                        term_buffer.append_type_and_fast_value::<u64>(val_u64);\n                    }\n                    NumericalValue::F64(val_f64) => {\n                        term_buffer.append_type_and_fast_value::<f64>(val_f64);\n                    }\n                }\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::Bool(val) => {\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                term_buffer.append_type_and_fast_value(val);\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::Date(val) => {\n                set_path_id(\n                    term_buffer,\n                    ctx.path_to_unordered_id\n                        .get_or_allocate_unordered_id(json_path_writer.as_str()),\n                );\n                let val = val.truncate(DATE_TIME_PRECISION_INDEXED);\n                term_buffer.append_type_and_fast_value(val);\n                postings_writer.subscribe(doc, 0u32, term_buffer, ctx);\n            }\n            ReferenceValueLeaf::PreTokStr(_) => {\n                unimplemented!(\n                    \"Pre-tokenized string support in dynamic fields is not yet implemented\"\n                )\n            }\n            ReferenceValueLeaf::Bytes(_) => {\n                unimplemented!(\"Bytes support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::Facet(_) => {\n                unimplemented!(\"Facet support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::IpAddr(_) => {\n                unimplemented!(\"IP address support in dynamic fields is not yet implemented\")\n            }\n        },\n        ReferenceValue::Array(elements) => {\n            for val in elements {\n                index_json_value(\n                    doc,\n                    val,\n                    text_analyzer,\n                    term_buffer,\n                    json_path_writer,\n                    postings_writer,\n                    ctx,\n                    positions_per_path,\n                );\n            }\n        }\n        ReferenceValue::Object(object) => {\n            index_json_object::<V>(\n                doc,\n                object,\n                text_analyzer,\n                term_buffer,\n                json_path_writer,\n                postings_writer,\n                ctx,\n                positions_per_path,\n            );\n        }\n    }\n}\n\n/// Tries to infer a JSON type from a string and append it to the term.\n///\n/// The term must be json + JSON path.\npub fn convert_to_fast_value_and_append_to_json_term(\n    term: &Term,\n    text: &str,\n    truncate_date_for_search: bool,\n) -> Option<Term> {\n    assert_eq!(\n        term.value()\n            .as_json_value_bytes()\n            .expect(\"expecting a Term with a json type and json path\")\n            .as_serialized()\n            .len(),\n        0,\n        \"JSON value bytes should be empty\"\n    );\n    try_convert_to_datetime_and_append_to_json_term(term, text, truncate_date_for_search)\n        .or_else(|| try_convert_to_number_and_append_to_json_term(term, text))\n        .or_else(|| try_convert_to_bool_and_append_to_json_term_typed(term, text))\n}\n\nfn try_convert_to_datetime_and_append_to_json_term(\n    term: &Term,\n    text: &str,\n    truncate_date_for_search: bool,\n) -> Option<Term> {\n    let dt = OffsetDateTime::parse(text, &Rfc3339).ok()?;\n    let mut dt = DateTime::from_utc(dt.to_offset(UtcOffset::UTC));\n    if truncate_date_for_search {\n        dt = dt.truncate(DATE_TIME_PRECISION_INDEXED);\n    }\n    let mut term_clone = term.clone();\n    term_clone.append_type_and_fast_value(dt);\n    Some(term_clone)\n}\n\nfn try_convert_to_number_and_append_to_json_term(term: &Term, text: &str) -> Option<Term> {\n    let numerical_value: NumericalValue = str::parse::<NumericalValue>(text).ok()?;\n    let mut term_clone = term.clone();\n    // Parse is actually returning normalized values already today, but let's not\n    // not rely on that hidden contract.\n    match numerical_value.normalize() {\n        NumericalValue::I64(i64_value) => {\n            term_clone.append_type_and_fast_value::<i64>(i64_value);\n        }\n        NumericalValue::U64(u64_value) => {\n            term_clone.append_type_and_fast_value::<u64>(u64_value);\n        }\n        NumericalValue::F64(f64_value) => {\n            term_clone.append_type_and_fast_value::<f64>(f64_value);\n        }\n    }\n    Some(term_clone)\n}\n\nfn try_convert_to_bool_and_append_to_json_term_typed(term: &Term, text: &str) -> Option<Term> {\n    let val = str::parse::<bool>(text).ok()?;\n    let mut term_clone = term.clone();\n    term_clone.append_type_and_fast_value(val);\n    Some(term_clone)\n}\n\n/// Splits a json path supplied to the query parser in such a way that\n/// `.` can be escaped.\n///\n/// In other words,\n/// - `k8s.node` ends up as `[\"k8s\", \"node\"]`.\n/// - `k8s\\.node` ends up as `[\"k8s.node\"]`.\npub fn split_json_path(json_path: &str) -> Vec<String> {\n    let mut escaped_state: bool = false;\n    let mut json_path_segments = Vec::new();\n    let mut buffer = String::new();\n    for ch in json_path.chars() {\n        if escaped_state {\n            buffer.push(ch);\n            escaped_state = false;\n            continue;\n        }\n        match ch {\n            '\\\\' => {\n                escaped_state = true;\n            }\n            '.' => {\n                let new_segment = std::mem::take(&mut buffer);\n                json_path_segments.push(new_segment);\n            }\n            _ => {\n                buffer.push(ch);\n            }\n        }\n    }\n    json_path_segments.push(buffer);\n    json_path_segments\n}\n\n/// Takes a field name, a json path as supplied by a user, and whether we should expand dots, and\n/// return a column key, as expected by the columnar crate.\n///\n/// This function will detect unescaped dots in the path, and split over them.\n/// If expand_dots is enabled, then even escaped dots will be split over.\n///\n/// The resulting list of segment then gets stitched together, joined by \\1 separator,\n/// as defined in the columnar crate.\npub(crate) fn encode_column_name(\n    field_name: &str,\n    json_path: &str,\n    expand_dots_enabled: bool,\n) -> String {\n    let mut path = JsonPathWriter::default();\n    path.push(field_name);\n    path.set_expand_dots(expand_dots_enabled);\n    for segment in split_json_path(json_path) {\n        path.push(&segment);\n    }\n    path.into()\n}\n\n#[cfg(test)]\nmod tests {\n    use super::split_json_path;\n    use crate::schema::Field;\n    use crate::Term;\n\n    #[test]\n    fn test_json_writer() {\n        let field = Field::from_field_id(1);\n\n        let mut term = Term::from_field_json_path(field, \"attributes.color\", false);\n        term.append_type_and_str(\"red\");\n        assert_eq!(\n            format!(\"{term:?}\"),\n            \"Term(field=1, type=Json, path=attributes.color, type=Str, \\\"red\\\")\"\n        );\n\n        let mut term = Term::from_field_json_path(field, \"attributes.dimensions.width\", false);\n        term.append_type_and_fast_value(400i64);\n        assert_eq!(\n            format!(\"{term:?}\"),\n            \"Term(field=1, type=Json, path=attributes.dimensions.width, type=I64, 400)\"\n        );\n    }\n\n    #[test]\n    fn test_string_term() {\n        let field = Field::from_field_id(1);\n        let mut term = Term::from_field_json_path(field, \"color\", false);\n        term.append_type_and_str(\"red\");\n\n        assert_eq!(term.serialized_value_bytes(), b\"color\\x00sred\".to_vec())\n    }\n\n    #[test]\n    fn test_i64_term() {\n        let field = Field::from_field_id(1);\n        let mut term = Term::from_field_json_path(field, \"color\", false);\n        term.append_type_and_fast_value(-4i64);\n\n        assert_eq!(\n            term.serialized_value_bytes(),\n            b\"color\\x00i\\x7f\\xff\\xff\\xff\\xff\\xff\\xff\\xfc\".to_vec()\n        )\n    }\n\n    #[test]\n    fn test_u64_term() {\n        let field = Field::from_field_id(1);\n        let mut term = Term::from_field_json_path(field, \"color\", false);\n        term.append_type_and_fast_value(4u64);\n\n        assert_eq!(\n            term.serialized_value_bytes(),\n            b\"color\\x00u\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x04\".to_vec()\n        )\n    }\n\n    #[test]\n    fn test_f64_term() {\n        let field = Field::from_field_id(1);\n        let mut term = Term::from_field_json_path(field, \"color\", false);\n        term.append_type_and_fast_value(4.0f64);\n        assert_eq!(\n            term.serialized_value_bytes(),\n            b\"color\\x00f\\xc0\\x10\\x00\\x00\\x00\\x00\\x00\\x00\".to_vec()\n        )\n    }\n\n    #[test]\n    fn test_bool_term() {\n        let field = Field::from_field_id(1);\n        let mut term = Term::from_field_json_path(field, \"color\", false);\n        term.append_type_and_fast_value(true);\n        assert_eq!(\n            term.serialized_value_bytes(),\n            b\"color\\x00o\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\".to_vec()\n        )\n    }\n\n    #[test]\n    fn test_split_json_path_simple() {\n        let json_path = split_json_path(\"titi.toto\");\n        assert_eq!(&json_path, &[\"titi\", \"toto\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_single_segment() {\n        let json_path = split_json_path(\"toto\");\n        assert_eq!(&json_path, &[\"toto\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_trailing_dot() {\n        let json_path = split_json_path(\"toto.\");\n        assert_eq!(&json_path, &[\"toto\", \"\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_heading_dot() {\n        let json_path = split_json_path(\".toto\");\n        assert_eq!(&json_path, &[\"\", \"toto\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_escaped_dot() {\n        let json_path = split_json_path(r\"toto\\.titi\");\n        assert_eq!(&json_path, &[\"toto.titi\"]);\n        let json_path_2 = split_json_path(r\"k8s\\.container\\.name\");\n        assert_eq!(&json_path_2, &[\"k8s.container.name\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_escaped_backslash() {\n        let json_path = split_json_path(r\"toto\\\\titi\");\n        assert_eq!(&json_path, &[r\"toto\\titi\"]);\n    }\n\n    #[test]\n    fn test_split_json_path_escaped_normal_letter() {\n        let json_path = split_json_path(r\"toto\\titi\");\n        assert_eq!(&json_path, &[r#\"tototiti\"#]);\n    }\n}\n"
  },
  {
    "path": "src/core/mod.rs",
    "content": "mod executor;\n#[doc(hidden)]\npub mod json_utils;\npub mod searcher;\n\nuse std::path::Path;\n\nuse once_cell::sync::Lazy;\n\npub use self::executor::Executor;\npub use self::searcher::{Searcher, SearcherGeneration};\n\n/// The meta file contains all the information about the list of segments and the schema\n/// of the index.\npub static META_FILEPATH: Lazy<&'static Path> = Lazy::new(|| Path::new(\"meta.json\"));\n\n/// The managed file contains a list of files that were created by the tantivy\n/// and will therefore be garbage collected when they are deemed useless by tantivy.\n///\n/// Removing this file is safe, but will prevent the garbage collection of all of the file that\n/// are currently in the directory\npub static MANAGED_FILEPATH: Lazy<&'static Path> = Lazy::new(|| Path::new(\".managed.json\"));\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "src/core/searcher.rs",
    "content": "use std::collections::BTreeMap;\nuse std::sync::Arc;\nuse std::{fmt, io};\n\nuse crate::collector::Collector;\nuse crate::core::Executor;\nuse crate::index::{SegmentId, SegmentReader};\nuse crate::query::{Bm25StatisticsProvider, EnableScoring, Query};\nuse crate::schema::document::DocumentDeserialize;\nuse crate::schema::{Schema, Term};\nuse crate::space_usage::SearcherSpaceUsage;\nuse crate::store::{CacheStats, StoreReader};\nuse crate::{DocAddress, Index, Opstamp, TrackedObject};\n\n/// Identifies the searcher generation accessed by a [`Searcher`].\n///\n/// While this might seem redundant, a [`SearcherGeneration`] contains\n/// both a `generation_id` AND a list of `(SegmentId, DeleteOpstamp)`.\n///\n/// This is on purpose. This object is used by the [`Warmer`](crate::reader::Warmer) API.\n/// Having both information makes it possible to identify which\n/// artifact should be refreshed or garbage collected.\n///\n/// Depending on the use case, `Warmer`'s implementers can decide to\n/// produce artifacts per:\n/// - `generation_id` (e.g. some searcher level aggregates)\n/// - `(segment_id, delete_opstamp)` (e.g. segment level aggregates)\n/// - `segment_id` (e.g. for immutable document level information)\n/// - `(generation_id, segment_id)` (e.g. for consistent dynamic column)\n/// - ...\n#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]\npub struct SearcherGeneration {\n    segments: BTreeMap<SegmentId, Option<Opstamp>>,\n    generation_id: u64,\n}\n\nimpl SearcherGeneration {\n    pub(crate) fn from_segment_readers(\n        segment_readers: &[SegmentReader],\n        generation_id: u64,\n    ) -> Self {\n        let mut segment_id_to_del_opstamp = BTreeMap::new();\n        for segment_reader in segment_readers {\n            segment_id_to_del_opstamp\n                .insert(segment_reader.segment_id(), segment_reader.delete_opstamp());\n        }\n        Self {\n            segments: segment_id_to_del_opstamp,\n            generation_id,\n        }\n    }\n\n    /// Returns the searcher generation id.\n    pub fn generation_id(&self) -> u64 {\n        self.generation_id\n    }\n\n    /// Return a `(SegmentId -> DeleteOpstamp)` mapping.\n    pub fn segments(&self) -> &BTreeMap<SegmentId, Option<Opstamp>> {\n        &self.segments\n    }\n}\n\n/// Holds a list of `SegmentReader`s ready for search.\n///\n/// It guarantees that the `Segment` will not be removed before\n/// the destruction of the `Searcher`.\n#[derive(Clone)]\npub struct Searcher {\n    inner: Arc<SearcherInner>,\n}\n\nimpl Searcher {\n    /// Returns the `Index` associated with the `Searcher`\n    pub fn index(&self) -> &Index {\n        &self.inner.index\n    }\n\n    /// [`SearcherGeneration`] which identifies the version of the snapshot held by this `Searcher`.\n    pub fn generation(&self) -> &SearcherGeneration {\n        self.inner.generation.as_ref()\n    }\n\n    /// Fetches a document from tantivy's store given a [`DocAddress`].\n    ///\n    /// The searcher uses the segment ordinal to route the\n    /// request to the right `Segment`.\n    pub fn doc<D: DocumentDeserialize>(&self, doc_address: DocAddress) -> crate::Result<D> {\n        let store_reader = &self.inner.store_readers[doc_address.segment_ord as usize];\n        store_reader.get(doc_address.doc_id)\n    }\n\n    /// The cache stats for the underlying store reader.\n    ///\n    /// Aggregates the sum for each segment store reader.\n    pub fn doc_store_cache_stats(&self) -> CacheStats {\n        let cache_stats: CacheStats = self\n            .inner\n            .store_readers\n            .iter()\n            .map(|reader| reader.cache_stats())\n            .sum();\n        cache_stats\n    }\n\n    /// Fetches a document in an asynchronous manner.\n    #[cfg(feature = \"quickwit\")]\n    pub async fn doc_async<D: DocumentDeserialize>(\n        &self,\n        doc_address: DocAddress,\n    ) -> crate::Result<D> {\n        let executor = self.inner.index.search_executor();\n        let store_reader = &self.inner.store_readers[doc_address.segment_ord as usize];\n        store_reader.get_async(doc_address.doc_id, executor).await\n    }\n\n    /// Access the schema associated with the index of this searcher.\n    pub fn schema(&self) -> &Schema {\n        &self.inner.schema\n    }\n\n    /// Returns the overall number of documents in the index.\n    pub fn num_docs(&self) -> u64 {\n        self.inner\n            .segment_readers\n            .iter()\n            .map(|segment_reader| u64::from(segment_reader.num_docs()))\n            .sum::<u64>()\n    }\n\n    /// Return the overall number of documents containing\n    /// the given term.\n    pub fn doc_freq(&self, term: &Term) -> crate::Result<u64> {\n        let mut total_doc_freq = 0;\n        for segment_reader in &self.inner.segment_readers {\n            let inverted_index = segment_reader.inverted_index(term.field())?;\n            let doc_freq = inverted_index.doc_freq(term)?;\n            total_doc_freq += u64::from(doc_freq);\n        }\n        Ok(total_doc_freq)\n    }\n\n    /// Return the overall number of documents containing\n    /// the given term in an asynchronous manner.\n    #[cfg(feature = \"quickwit\")]\n    pub async fn doc_freq_async(&self, term: &Term) -> crate::Result<u64> {\n        let mut total_doc_freq = 0;\n        for segment_reader in &self.inner.segment_readers {\n            let inverted_index = segment_reader.inverted_index(term.field())?;\n            let doc_freq = inverted_index.doc_freq_async(term).await?;\n            total_doc_freq += u64::from(doc_freq);\n        }\n        Ok(total_doc_freq)\n    }\n\n    /// Return the list of segment readers\n    pub fn segment_readers(&self) -> &[SegmentReader] {\n        &self.inner.segment_readers\n    }\n\n    /// Returns the segment_reader associated with the given segment_ord\n    pub fn segment_reader(&self, segment_ord: u32) -> &SegmentReader {\n        &self.inner.segment_readers[segment_ord as usize]\n    }\n\n    /// Runs a query on the segment readers wrapped by the searcher.\n    ///\n    /// Search works as follows :\n    ///\n    ///  First the weight object associated with the query is created.\n    ///\n    ///  Then, the query loops over the segments and for each segment :\n    ///  - setup the collector and informs it that the segment being processed has changed.\n    ///  - creates a SegmentCollector for collecting documents associated with the segment\n    ///  - creates a `Scorer` object associated for this segment\n    ///  - iterate through the matched documents and push them to the segment collector.\n    ///\n    ///  Finally, the Collector merges each of the child collectors into itself for result usability\n    ///  by the caller.\n    pub fn search<C: Collector>(\n        &self,\n        query: &dyn Query,\n        collector: &C,\n    ) -> crate::Result<C::Fruit> {\n        self.search_with_statistics_provider(query, collector, self)\n    }\n\n    /// Same as [`search(...)`](Searcher::search) but allows specifying\n    /// a [Bm25StatisticsProvider].\n    ///\n    /// This can be used to adjust the statistics used in computing BM25\n    /// scores.\n    pub fn search_with_statistics_provider<C: Collector>(\n        &self,\n        query: &dyn Query,\n        collector: &C,\n        statistics_provider: &dyn Bm25StatisticsProvider,\n    ) -> crate::Result<C::Fruit> {\n        let enabled_scoring = if collector.requires_scoring() {\n            EnableScoring::enabled_from_statistics_provider(statistics_provider, self)\n        } else {\n            EnableScoring::disabled_from_searcher(self)\n        };\n        let executor = self.inner.index.search_executor();\n        self.search_with_executor(query, collector, executor, enabled_scoring)\n    }\n\n    /// Same as [`search(...)`](Searcher::search) but multithreaded.\n    ///\n    /// The current implementation is rather naive :\n    /// multithreading is by splitting search into as many task\n    /// as there are segments.\n    ///\n    /// It is powerless at making search faster if your index consists in\n    /// one large segment.\n    ///\n    /// Also, keep in mind multithreading a single query on several\n    /// threads will not improve your throughput. It can actually\n    /// hurt it. It will however, decrease the average response time.\n    pub fn search_with_executor<C: Collector>(\n        &self,\n        query: &dyn Query,\n        collector: &C,\n        executor: &Executor,\n        enabled_scoring: EnableScoring,\n    ) -> crate::Result<C::Fruit> {\n        let weight = query.weight(enabled_scoring)?;\n        collector.check_schema(self.schema())?;\n        let segment_readers = self.segment_readers();\n        let fruits = executor.map(\n            |(segment_ord, segment_reader)| {\n                collector.collect_segment(weight.as_ref(), segment_ord as u32, segment_reader)\n            },\n            segment_readers.iter().enumerate(),\n        )?;\n        collector.merge_fruits(fruits)\n    }\n\n    /// Summarize total space usage of this searcher.\n    pub fn space_usage(&self) -> io::Result<SearcherSpaceUsage> {\n        let mut space_usage = SearcherSpaceUsage::new();\n        for segment_reader in self.segment_readers() {\n            space_usage.add_segment(segment_reader.space_usage()?);\n        }\n        Ok(space_usage)\n    }\n}\n\nimpl From<Arc<SearcherInner>> for Searcher {\n    fn from(inner: Arc<SearcherInner>) -> Self {\n        Searcher { inner }\n    }\n}\n\n/// Holds a list of `SegmentReader`s ready for search.\n///\n/// It guarantees that the `Segment` will not be removed before\n/// the destruction of the `Searcher`.\npub(crate) struct SearcherInner {\n    schema: Schema,\n    index: Index,\n    segment_readers: Vec<SegmentReader>,\n    store_readers: Vec<StoreReader>,\n    generation: TrackedObject<SearcherGeneration>,\n}\n\nimpl SearcherInner {\n    /// Creates a new `Searcher`\n    pub(crate) fn new(\n        schema: Schema,\n        index: Index,\n        segment_readers: Vec<SegmentReader>,\n        generation: TrackedObject<SearcherGeneration>,\n        doc_store_cache_num_blocks: usize,\n    ) -> io::Result<SearcherInner> {\n        assert_eq!(\n            &segment_readers\n                .iter()\n                .map(|reader| (reader.segment_id(), reader.delete_opstamp()))\n                .collect::<BTreeMap<_, _>>(),\n            generation.segments(),\n            \"Set of segments referenced by this Searcher and its SearcherGeneration must match\"\n        );\n        let store_readers: Vec<StoreReader> = segment_readers\n            .iter()\n            .map(|segment_reader| segment_reader.get_store_reader(doc_store_cache_num_blocks))\n            .collect::<io::Result<Vec<_>>>()?;\n\n        Ok(SearcherInner {\n            schema,\n            index,\n            segment_readers,\n            store_readers,\n            generation,\n        })\n    }\n}\n\nimpl fmt::Debug for Searcher {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        let segment_ids = self\n            .segment_readers()\n            .iter()\n            .map(SegmentReader::segment_id)\n            .collect::<Vec<_>>();\n        write!(f, \"Searcher({segment_ids:?})\")\n    }\n}\n"
  },
  {
    "path": "src/core/tests.rs",
    "content": "use crate::collector::Count;\nuse crate::directory::{RamDirectory, WatchCallback};\nuse crate::index::SegmentId;\nuse crate::indexer::{LogMergePolicy, NoMergePolicy};\nuse crate::postings::Postings;\nuse crate::query::TermQuery;\nuse crate::schema::{Field, IndexRecordOption, Schema, INDEXED, STRING, TEXT};\nuse crate::tokenizer::TokenizerManager;\nuse crate::{\n    Directory, DocSet, Index, IndexBuilder, IndexReader, IndexSettings, IndexWriter, ReloadPolicy,\n    TantivyDocument, Term,\n};\n\n#[test]\nfn test_indexer_for_field() {\n    let mut schema_builder = Schema::builder();\n    let num_likes_field = schema_builder.add_u64_field(\"num_likes\", INDEXED);\n    let body_field = schema_builder.add_text_field(\"body\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    assert!(index.tokenizer_for_field(body_field).is_ok());\n    assert_eq!(\n        format!(\"{:?}\", index.tokenizer_for_field(num_likes_field).err()),\n        \"Some(SchemaError(\\\"\\\\\\\"num_likes\\\\\\\" is not a text field.\\\"))\"\n    );\n}\n\n#[test]\nfn test_set_tokenizer_manager() {\n    let mut schema_builder = Schema::builder();\n    schema_builder.add_u64_field(\"num_likes\", INDEXED);\n    schema_builder.add_text_field(\"body\", TEXT);\n    let schema = schema_builder.build();\n    let index = IndexBuilder::new()\n        // set empty tokenizer manager\n        .tokenizers(TokenizerManager::new())\n        .schema(schema)\n        .create_in_ram()\n        .unwrap();\n    assert!(index.tokenizers().get(\"raw\").is_none());\n}\n\n#[test]\nfn test_index_exists() {\n    let directory: Box<dyn Directory> = Box::new(RamDirectory::create());\n    assert!(!Index::exists(directory.as_ref()).unwrap());\n    assert!(Index::create(\n        directory.clone(),\n        throw_away_schema(),\n        IndexSettings::default()\n    )\n    .is_ok());\n    assert!(Index::exists(directory.as_ref()).unwrap());\n}\n\n#[test]\nfn open_or_create_should_create() {\n    let directory = RamDirectory::create();\n    assert!(!Index::exists(&directory).unwrap());\n    assert!(Index::open_or_create(directory.clone(), throw_away_schema()).is_ok());\n    assert!(Index::exists(&directory).unwrap());\n}\n\n#[test]\nfn open_or_create_should_open() {\n    let directory: Box<dyn Directory> = Box::new(RamDirectory::create());\n    assert!(Index::create(\n        directory.clone(),\n        throw_away_schema(),\n        IndexSettings::default()\n    )\n    .is_ok());\n    assert!(Index::exists(directory.as_ref()).unwrap());\n    assert!(Index::open_or_create(directory, throw_away_schema()).is_ok());\n}\n\n#[test]\nfn create_should_wipeoff_existing() {\n    let directory: Box<dyn Directory> = Box::new(RamDirectory::create());\n    assert!(Index::create(\n        directory.clone(),\n        throw_away_schema(),\n        IndexSettings::default()\n    )\n    .is_ok());\n    assert!(Index::exists(directory.as_ref()).unwrap());\n    assert!(Index::create(\n        directory,\n        Schema::builder().build(),\n        IndexSettings::default()\n    )\n    .is_ok());\n}\n\n#[test]\nfn open_or_create_exists_but_schema_does_not_match() {\n    let directory = RamDirectory::create();\n    assert!(Index::create(\n        directory.clone(),\n        throw_away_schema(),\n        IndexSettings::default()\n    )\n    .is_ok());\n    assert!(Index::exists(&directory).unwrap());\n    assert!(Index::open_or_create(directory.clone(), throw_away_schema()).is_ok());\n    let err = Index::open_or_create(directory, Schema::builder().build());\n    assert_eq!(\n        format!(\"{:?}\", err.unwrap_err()),\n        \"SchemaError(\\\"An index exists but the schema does not match.\\\")\"\n    );\n}\n\nfn throw_away_schema() -> Schema {\n    let mut schema_builder = Schema::builder();\n    let _ = schema_builder.add_u64_field(\"num_likes\", INDEXED);\n    schema_builder.build()\n}\n\n#[test]\nfn test_index_on_commit_reload_policy() -> crate::Result<()> {\n    let schema = throw_away_schema();\n    let field = schema.get_field(\"num_likes\").unwrap();\n    let index = Index::create_in_ram(schema);\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::OnCommitWithDelay)\n        .try_into()\n        .unwrap();\n    assert_eq!(reader.searcher().num_docs(), 0);\n    test_index_on_commit_reload_policy_aux(field, &index, &reader)\n}\n\n#[cfg(feature = \"mmap\")]\nmod mmap_specific {\n\n    use std::path::PathBuf;\n\n    use tempfile::TempDir;\n\n    use super::*;\n\n    #[test]\n    fn test_index_on_commit_reload_policy_mmap() -> crate::Result<()> {\n        let schema = throw_away_schema();\n        let field = schema.get_field(\"num_likes\").unwrap();\n        let tempdir = TempDir::new().unwrap();\n        let tempdir_path = PathBuf::from(tempdir.path());\n        let index = Index::create_in_dir(tempdir_path, schema).unwrap();\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::OnCommitWithDelay)\n            .try_into()\n            .unwrap();\n        assert_eq!(reader.searcher().num_docs(), 0);\n        test_index_on_commit_reload_policy_aux(field, &index, &reader)\n    }\n\n    #[test]\n    fn test_index_manual_policy_mmap() -> crate::Result<()> {\n        let schema = throw_away_schema();\n        let field = schema.get_field(\"num_likes\").unwrap();\n        let mut index = Index::create_from_tempdir(schema)?;\n        let mut writer: IndexWriter = index.writer_for_tests()?;\n        writer.commit()?;\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        assert_eq!(reader.searcher().num_docs(), 0);\n        writer.add_document(doc!(field=>1u64))?;\n        let (sender, receiver) = crossbeam_channel::unbounded();\n        let _handle = index.directory_mut().watch(WatchCallback::new(move || {\n            let _ = sender.send(());\n        }));\n        writer.commit()?;\n        assert!(receiver.recv().is_ok());\n        assert_eq!(reader.searcher().num_docs(), 0);\n        reader.reload()?;\n        assert_eq!(reader.searcher().num_docs(), 1);\n        Ok(())\n    }\n\n    #[test]\n    fn test_index_on_commit_reload_policy_different_directories() -> crate::Result<()> {\n        let schema = throw_away_schema();\n        let field = schema.get_field(\"num_likes\").unwrap();\n        let tempdir = TempDir::new().unwrap();\n        let tempdir_path = PathBuf::from(tempdir.path());\n        let write_index = Index::create_in_dir(&tempdir_path, schema).unwrap();\n        let read_index = Index::open_in_dir(&tempdir_path).unwrap();\n        let reader = read_index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::OnCommitWithDelay)\n            .try_into()\n            .unwrap();\n        assert_eq!(reader.searcher().num_docs(), 0);\n        test_index_on_commit_reload_policy_aux(field, &write_index, &reader)\n    }\n}\nfn test_index_on_commit_reload_policy_aux(\n    field: Field,\n    index: &Index,\n    reader: &IndexReader,\n) -> crate::Result<()> {\n    let mut reader_index = reader.index();\n    let (sender, receiver) = crossbeam_channel::unbounded();\n    let _watch_handle = reader_index\n        .directory_mut()\n        .watch(WatchCallback::new(move || {\n            let _ = sender.send(());\n        }));\n    let mut writer: IndexWriter = index.writer_for_tests()?;\n    assert_eq!(reader.searcher().num_docs(), 0);\n    writer.add_document(doc!(field=>1u64))?;\n    writer.commit().unwrap();\n    // We need a loop here because it is possible for notify to send more than\n    // one modify event. It was observed on CI on MacOS.\n    loop {\n        assert!(receiver.recv().is_ok());\n        if reader.searcher().num_docs() == 1 {\n            break;\n        }\n    }\n    writer.add_document(doc!(field=>2u64))?;\n    writer.commit().unwrap();\n    // ... Same as above\n    loop {\n        assert!(receiver.recv().is_ok());\n        if reader.searcher().num_docs() == 2 {\n            break;\n        }\n    }\n    Ok(())\n}\n\n// This test will not pass on windows, because windows\n// prevent deleting files that are MMapped.\n#[cfg(not(target_os = \"windows\"))]\n#[test]\nfn garbage_collect_works_as_intended() -> crate::Result<()> {\n    let directory = RamDirectory::create();\n    let schema = throw_away_schema();\n    let field = schema.get_field(\"num_likes\").unwrap();\n    let index = Index::create(directory.clone(), schema, IndexSettings::default())?;\n\n    let mut writer: IndexWriter = index.writer_with_num_threads(1, 32_000_000).unwrap();\n    for _seg in 0..8 {\n        for i in 0u64..1_000u64 {\n            writer.add_document(doc!(field => i))?;\n        }\n        writer.commit()?;\n    }\n\n    let mem_right_after_commit = directory.total_mem_usage();\n\n    let reader = index\n        .reader_builder()\n        .reload_policy(ReloadPolicy::Manual)\n        .try_into()?;\n    assert_eq!(reader.searcher().num_docs(), 8_000);\n    assert_eq!(reader.searcher().segment_readers().len(), 8);\n\n    writer.wait_merging_threads()?;\n\n    let mem_right_after_merge_finished = directory.total_mem_usage();\n\n    reader.reload().unwrap();\n    let searcher = reader.searcher();\n    assert_eq!(searcher.segment_readers().len(), 1);\n    assert_eq!(searcher.num_docs(), 8_000);\n    assert!(\n        mem_right_after_merge_finished < mem_right_after_commit,\n        \"(mem after merge){mem_right_after_merge_finished} is expected < (mem before \\\n         merge){mem_right_after_commit}\"\n    );\n    Ok(())\n}\n\n#[test]\nfn test_single_segment_index_writer() -> crate::Result<()> {\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let schema = schema_builder.build();\n    let directory = RamDirectory::default();\n    let mut single_segment_index_writer = Index::builder()\n        .schema(schema)\n        .single_segment_index_writer(directory, 15_000_000)?;\n    for _ in 0..10 {\n        let doc = doc!(text_field=>\"hello\");\n        single_segment_index_writer.add_document(doc)?;\n    }\n    let index = single_segment_index_writer.finalize()?;\n    let searcher = index.reader()?.searcher();\n    let term_query = TermQuery::new(\n        Term::from_field_text(text_field, \"hello\"),\n        IndexRecordOption::Basic,\n    );\n    let count = searcher.search(&term_query, &Count)?;\n    assert_eq!(count, 10);\n    Ok(())\n}\n\n#[test]\nfn test_merging_segment_update_docfreq() {\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let id_field = schema_builder.add_text_field(\"id\", STRING);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema);\n    let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n    writer.set_merge_policy(Box::new(NoMergePolicy));\n    for _ in 0..5 {\n        writer.add_document(doc!(text_field=>\"hello\")).unwrap();\n    }\n    writer\n        .add_document(doc!(text_field=>\"hello\", id_field=>\"TO_BE_DELETED\"))\n        .unwrap();\n    writer\n        .add_document(doc!(text_field=>\"hello\", id_field=>\"TO_BE_DELETED\"))\n        .unwrap();\n    writer.add_document(TantivyDocument::default()).unwrap();\n    writer.commit().unwrap();\n    for _ in 0..7 {\n        writer.add_document(doc!(text_field=>\"hello\")).unwrap();\n    }\n    writer.add_document(TantivyDocument::default()).unwrap();\n    writer.add_document(TantivyDocument::default()).unwrap();\n    writer.delete_term(Term::from_field_text(id_field, \"TO_BE_DELETED\"));\n    writer.commit().unwrap();\n\n    let segment_ids: Vec<SegmentId> = index\n        .list_all_segment_metas()\n        .into_iter()\n        .map(|reader| reader.id())\n        .collect();\n    writer.merge(&segment_ids[..]).wait().unwrap();\n    let index_reader = index.reader().unwrap();\n    let searcher = index_reader.searcher();\n    assert_eq!(searcher.segment_readers().len(), 1);\n    assert_eq!(searcher.num_docs(), 15);\n    let segment_reader = searcher.segment_reader(0);\n    assert_eq!(segment_reader.max_doc(), 15);\n    let inv_index = segment_reader.inverted_index(text_field).unwrap();\n    let term = Term::from_field_text(text_field, \"hello\");\n    let term_info = inv_index.get_term_info(&term).unwrap().unwrap();\n    assert_eq!(term_info.doc_freq, 12);\n}\n\n// motivated by https://github.com/quickwit-oss/quickwit/issues/4130\n#[test]\nfn test_positions_merge_bug_non_text_json_vint() {\n    let mut schema_builder = Schema::builder();\n    let field = schema_builder.add_json_field(\"dynamic\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n    let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n    let mut merge_policy = LogMergePolicy::default();\n    merge_policy.set_min_num_segments(2);\n    writer.set_merge_policy(Box::new(merge_policy));\n    // Here a string would work.\n    let doc_json = r#\"{\"tenant_id\":75}\"#;\n    let vals = serde_json::from_str(doc_json).unwrap();\n    let mut doc = TantivyDocument::default();\n    doc.add_object(field, vals);\n    writer.add_document(doc.clone()).unwrap();\n    writer.commit().unwrap();\n    writer.add_document(doc.clone()).unwrap();\n    writer.commit().unwrap();\n    writer.wait_merging_threads().unwrap();\n    let reader = index.reader().unwrap();\n    assert_eq!(reader.searcher().segment_readers().len(), 1);\n}\n\n// Same as above but with bitpacked blocks\n#[test]\nfn test_positions_merge_bug_non_text_json_bitpacked_block() {\n    let mut schema_builder = Schema::builder();\n    let field = schema_builder.add_json_field(\"dynamic\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n    let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n    let mut merge_policy = LogMergePolicy::default();\n    merge_policy.set_min_num_segments(2);\n    writer.set_merge_policy(Box::new(merge_policy));\n    // Here a string would work.\n    let doc_json = r#\"{\"tenant_id\":75}\"#;\n    let vals = serde_json::from_str(doc_json).unwrap();\n    let mut doc = TantivyDocument::default();\n    doc.add_object(field, vals);\n    for _ in 0..128 {\n        writer.add_document(doc.clone()).unwrap();\n    }\n    writer.commit().unwrap();\n    writer.add_document(doc.clone()).unwrap();\n    writer.commit().unwrap();\n    writer.wait_merging_threads().unwrap();\n    let reader = index.reader().unwrap();\n    assert_eq!(reader.searcher().segment_readers().len(), 1);\n}\n\n#[test]\nfn test_non_text_json_term_freq() {\n    let mut schema_builder = Schema::builder();\n    let field = schema_builder.add_json_field(\"dynamic\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n    let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n    // Here a string would work.\n    let doc_json = r#\"{\"tenant_id\":75}\"#;\n    let vals = serde_json::from_str(doc_json).unwrap();\n    let mut doc = TantivyDocument::default();\n    doc.add_object(field, vals);\n    writer.add_document(doc.clone()).unwrap();\n    writer.commit().unwrap();\n    let reader = index.reader().unwrap();\n    assert_eq!(reader.searcher().segment_readers().len(), 1);\n    let searcher = reader.searcher();\n    let segment_reader = searcher.segment_reader(0u32);\n    let inv_idx = segment_reader.inverted_index(field).unwrap();\n\n    let mut term = Term::from_field_json_path(field, \"tenant_id\", false);\n    term.append_type_and_fast_value(75i64);\n\n    let postings = inv_idx\n        .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n        .unwrap()\n        .unwrap();\n    assert_eq!(postings.doc(), 0);\n    assert_eq!(postings.term_freq(), 1u32);\n}\n\n#[test]\nfn test_non_text_json_term_freq_bitpacked() {\n    let mut schema_builder = Schema::builder();\n    let field = schema_builder.add_json_field(\"dynamic\", TEXT);\n    let schema = schema_builder.build();\n    let index = Index::create_in_ram(schema.clone());\n    let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n    // Here a string would work.\n    let doc_json = r#\"{\"tenant_id\":75}\"#;\n    let vals = serde_json::from_str(doc_json).unwrap();\n    let mut doc = TantivyDocument::default();\n    doc.add_object(field, vals);\n    let num_docs = 132;\n    for _ in 0..num_docs {\n        writer.add_document(doc.clone()).unwrap();\n    }\n    writer.commit().unwrap();\n    let reader = index.reader().unwrap();\n    assert_eq!(reader.searcher().segment_readers().len(), 1);\n    let searcher = reader.searcher();\n    let segment_reader = searcher.segment_reader(0u32);\n    let inv_idx = segment_reader.inverted_index(field).unwrap();\n\n    let mut term = Term::from_field_json_path(field, \"tenant_id\", false);\n    term.append_type_and_fast_value(75i64);\n\n    let mut postings = inv_idx\n        .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n        .unwrap()\n        .unwrap();\n    assert_eq!(postings.doc(), 0);\n    assert_eq!(postings.term_freq(), 1u32);\n    for i in 1..num_docs {\n        assert_eq!(postings.advance(), i);\n        assert_eq!(postings.term_freq(), 1u32);\n    }\n}\n"
  },
  {
    "path": "src/directory/composite_file.rs",
    "content": "use std::collections::HashMap;\nuse std::io::{self, Read, Write};\nuse std::ops::Range;\n\nuse common::{BinarySerializable, CountingWriter, HasLen, VInt};\n\nuse crate::directory::{FileSlice, TerminatingWrite, WritePtr};\nuse crate::schema::{Field, Schema};\nuse crate::space_usage::{FieldUsage, PerFieldSpaceUsage};\n\n#[derive(Eq, PartialEq, Hash, Copy, Ord, PartialOrd, Clone, Debug)]\npub struct FileAddr {\n    field: Field,\n    idx: usize,\n}\n\nimpl FileAddr {\n    fn new(field: Field, idx: usize) -> FileAddr {\n        FileAddr { field, idx }\n    }\n}\n\nimpl BinarySerializable for FileAddr {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.field.serialize(writer)?;\n        VInt(self.idx as u64).serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let field = Field::deserialize(reader)?;\n        let idx = VInt::deserialize(reader)?.0 as usize;\n        Ok(FileAddr { field, idx })\n    }\n}\n\n/// A `CompositeWrite` is used to write a `CompositeFile`.\npub struct CompositeWrite<W = WritePtr> {\n    write: CountingWriter<W>,\n    offsets: Vec<(FileAddr, u64)>,\n}\n\nimpl<W: TerminatingWrite + Write> CompositeWrite<W> {\n    /// Crate a new API writer that writes a composite file\n    /// in a given write.\n    pub fn wrap(w: W) -> CompositeWrite<W> {\n        CompositeWrite {\n            write: CountingWriter::wrap(w),\n            offsets: Vec::new(),\n        }\n    }\n\n    /// Start writing a new field.\n    pub fn for_field(&mut self, field: Field) -> &mut CountingWriter<W> {\n        self.for_field_with_idx(field, 0)\n    }\n\n    /// Start writing a new field.\n    pub fn for_field_with_idx(&mut self, field: Field, idx: usize) -> &mut CountingWriter<W> {\n        let offset = self.write.written_bytes();\n        let file_addr = FileAddr::new(field, idx);\n        assert!(!self.offsets.iter().any(|el| el.0 == file_addr));\n        self.offsets.push((file_addr, offset));\n        &mut self.write\n    }\n\n    /// Close the composite file\n    ///\n    /// An index of the different field offsets\n    /// will be written as a footer.\n    pub fn close(mut self) -> io::Result<()> {\n        let footer_offset = self.write.written_bytes();\n        VInt(self.offsets.len() as u64).serialize(&mut self.write)?;\n\n        let mut prev_offset = 0;\n        for (file_addr, offset) in self.offsets {\n            VInt(offset - prev_offset).serialize(&mut self.write)?;\n            file_addr.serialize(&mut self.write)?;\n            prev_offset = offset;\n        }\n\n        let footer_len = (self.write.written_bytes() - footer_offset) as u32;\n        footer_len.serialize(&mut self.write)?;\n        self.write.terminate()\n    }\n}\n\n/// A composite file is an abstraction to store a\n/// file partitioned by field.\n///\n/// The file needs to be written field by field.\n/// A footer describes the start and stop offsets\n/// for each field.\n#[derive(Clone)]\npub struct CompositeFile {\n    data: FileSlice,\n    offsets_index: HashMap<FileAddr, Range<usize>>,\n}\n\nimpl std::fmt::Debug for CompositeFile {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        f.debug_struct(\"CompositeFile\")\n            .field(\"offsets_index\", &self.offsets_index)\n            .finish()\n    }\n}\n\nimpl CompositeFile {\n    /// Opens a composite file stored in a given\n    /// `FileSlice`.\n    pub fn open(data: &FileSlice) -> io::Result<CompositeFile> {\n        let end = data.len();\n        let footer_len_data = data.slice_from(end - 4).read_bytes()?;\n        let footer_len = u32::deserialize(&mut footer_len_data.as_slice())? as usize;\n        let footer_start = end - 4 - footer_len;\n        let footer_data = data\n            .slice(footer_start..footer_start + footer_len)\n            .read_bytes()?;\n        let mut footer_buffer = footer_data.as_slice();\n        let num_fields = VInt::deserialize(&mut footer_buffer)?.0 as usize;\n\n        let mut file_addrs = vec![];\n        let mut offsets = vec![];\n        let mut field_index = HashMap::new();\n\n        let mut offset = 0;\n        for _ in 0..num_fields {\n            offset += VInt::deserialize(&mut footer_buffer)?.0 as usize;\n            let file_addr = FileAddr::deserialize(&mut footer_buffer)?;\n            offsets.push(offset);\n            file_addrs.push(file_addr);\n        }\n        offsets.push(footer_start);\n        for i in 0..num_fields {\n            let file_addr = file_addrs[i];\n            let start_offset = offsets[i];\n            let end_offset = offsets[i + 1];\n            field_index.insert(file_addr, start_offset..end_offset);\n        }\n\n        Ok(CompositeFile {\n            data: data.slice_to(footer_start),\n            offsets_index: field_index,\n        })\n    }\n\n    /// Returns a composite file that stores\n    /// no fields.\n    pub fn empty() -> CompositeFile {\n        CompositeFile {\n            offsets_index: HashMap::new(),\n            data: FileSlice::empty(),\n        }\n    }\n\n    /// Returns the `FileSlice` associated with\n    /// a given `Field` and stored in a `CompositeFile`.\n    pub fn open_read(&self, field: Field) -> Option<FileSlice> {\n        self.open_read_with_idx(field, 0)\n    }\n\n    /// Returns the `FileSlice` associated with\n    /// a given `Field` and stored in a `CompositeFile`.\n    pub fn open_read_with_idx(&self, field: Field, idx: usize) -> Option<FileSlice> {\n        self.offsets_index\n            .get(&FileAddr { field, idx })\n            .map(|byte_range| self.data.slice(byte_range.clone()))\n    }\n\n    /// Returns the space usage per field in this composite file.\n    pub fn space_usage(&self, schema: &Schema) -> PerFieldSpaceUsage {\n        let mut fields = Vec::new();\n        for (&field_addr, byte_range) in &self.offsets_index {\n            let field_name = schema.get_field_name(field_addr.field).to_string();\n            let mut field_usage = FieldUsage::empty(field_name);\n            field_usage.add_field_idx(field_addr.idx, byte_range.len().into());\n            fields.push(field_usage);\n        }\n        PerFieldSpaceUsage::new(fields)\n    }\n}\n\n#[cfg(test)]\nmod test {\n\n    use std::io::Write;\n    use std::path::Path;\n\n    use common::{BinarySerializable, VInt};\n\n    use super::{CompositeFile, CompositeWrite};\n    use crate::directory::{Directory, RamDirectory};\n    use crate::schema::Field;\n\n    #[test]\n    fn test_composite_file() -> crate::Result<()> {\n        let path = Path::new(\"test_path\");\n        let directory = RamDirectory::create();\n        {\n            let w = directory.open_write(path).unwrap();\n            let mut composite_write = CompositeWrite::wrap(w);\n            let mut write_0 = composite_write.for_field(Field::from_field_id(0u32));\n            VInt(32431123u64).serialize(&mut write_0)?;\n            write_0.flush()?;\n            let mut write_4 = composite_write.for_field(Field::from_field_id(4u32));\n            VInt(2).serialize(&mut write_4)?;\n            write_4.flush()?;\n            composite_write.close()?;\n        }\n        {\n            let r = directory.open_read(path)?;\n            let composite_file = CompositeFile::open(&r)?;\n            {\n                let file0 = composite_file\n                    .open_read(Field::from_field_id(0u32))\n                    .unwrap()\n                    .read_bytes()?;\n                let mut file0_buf = file0.as_slice();\n                let payload_0 = VInt::deserialize(&mut file0_buf)?.0;\n                assert_eq!(file0_buf.len(), 0);\n                assert_eq!(payload_0, 32431123u64);\n            }\n            {\n                let file4 = composite_file\n                    .open_read(Field::from_field_id(4u32))\n                    .unwrap()\n                    .read_bytes()?;\n                let mut file4_buf = file4.as_slice();\n                let payload_4 = VInt::deserialize(&mut file4_buf)?.0;\n                assert_eq!(file4_buf.len(), 0);\n                assert_eq!(payload_4, 2u64);\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_composite_file_bug() -> crate::Result<()> {\n        let path = Path::new(\"test_path\");\n        let directory = RamDirectory::create();\n        {\n            let w = directory.open_write(path).unwrap();\n            let mut composite_write = CompositeWrite::wrap(w);\n            let mut write = composite_write.for_field_with_idx(Field::from_field_id(1u32), 0);\n            VInt(32431123u64).serialize(&mut write)?;\n            write.flush()?;\n            let write = composite_write.for_field_with_idx(Field::from_field_id(1u32), 1);\n            write.flush()?;\n\n            let mut write = composite_write.for_field_with_idx(Field::from_field_id(0u32), 0);\n            VInt(1_000_000).serialize(&mut write)?;\n            write.flush()?;\n\n            composite_write.close()?;\n        }\n        {\n            let r = directory.open_read(path)?;\n            let composite_file = CompositeFile::open(&r)?;\n            {\n                let file = composite_file\n                    .open_read_with_idx(Field::from_field_id(1u32), 0)\n                    .unwrap()\n                    .read_bytes()?;\n                let mut file0_buf = file.as_slice();\n                let payload_0 = VInt::deserialize(&mut file0_buf)?.0;\n                assert_eq!(file0_buf.len(), 0);\n                assert_eq!(payload_0, 32431123u64);\n            }\n            {\n                let file = composite_file\n                    .open_read_with_idx(Field::from_field_id(1u32), 1)\n                    .unwrap()\n                    .read_bytes()?;\n                let file = file.as_slice();\n                assert_eq!(file.len(), 0);\n            }\n            {\n                let file = composite_file\n                    .open_read_with_idx(Field::from_field_id(0u32), 0)\n                    .unwrap()\n                    .read_bytes()?;\n                let file = file.as_slice();\n                assert_eq!(file.len(), 3);\n            }\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/directory/directory.rs",
    "content": "use std::io::Write;\nuse std::path::{Path, PathBuf};\nuse std::sync::Arc;\nuse std::time::Duration;\nuse std::{fmt, io, thread};\n\nuse crate::directory::directory_lock::Lock;\nuse crate::directory::error::{DeleteError, LockError, OpenReadError, OpenWriteError};\nuse crate::directory::{FileHandle, FileSlice, WatchCallback, WatchHandle, WritePtr};\n\n/// Retry the logic of acquiring locks is pretty simple.\n/// We just retry `n` times after a given `duratio`, both\n/// depending on the type of lock.\nstruct RetryPolicy {\n    num_retries: usize,\n    wait_in_ms: u64,\n}\n\nimpl RetryPolicy {\n    fn no_retry() -> RetryPolicy {\n        RetryPolicy {\n            num_retries: 0,\n            wait_in_ms: 0,\n        }\n    }\n\n    fn wait_and_retry(&mut self) -> bool {\n        if self.num_retries == 0 {\n            false\n        } else {\n            self.num_retries -= 1;\n            let wait_duration = Duration::from_millis(self.wait_in_ms);\n            thread::sleep(wait_duration);\n            true\n        }\n    }\n}\n\n/// The `DirectoryLock` is an object that represents a file lock.\n///\n/// It is associated with a lock file, that gets deleted on `Drop.`\n#[expect(dead_code)]\npub struct DirectoryLock(Box<dyn Send + Sync + 'static>);\n\nstruct DirectoryLockGuard {\n    directory: Box<dyn Directory>,\n    path: PathBuf,\n}\n\nimpl<T: Send + Sync + 'static> From<Box<T>> for DirectoryLock {\n    fn from(underlying: Box<T>) -> Self {\n        DirectoryLock(underlying)\n    }\n}\n\nimpl Drop for DirectoryLockGuard {\n    fn drop(&mut self) {\n        if let Err(e) = self.directory.delete(&self.path) {\n            error!(\"Failed to remove the lock file. {e:?}\");\n        }\n    }\n}\n\nenum TryAcquireLockError {\n    FileExists,\n    IoError(Arc<io::Error>),\n}\nimpl From<io::Error> for TryAcquireLockError {\n    fn from(io_error: io::Error) -> Self {\n        Self::IoError(Arc::new(io_error))\n    }\n}\n\nfn try_acquire_lock(\n    filepath: &Path,\n    directory: &dyn Directory,\n) -> Result<DirectoryLock, TryAcquireLockError> {\n    let mut write = directory.open_write(filepath).map_err(|e| match e {\n        OpenWriteError::FileAlreadyExists(_) => TryAcquireLockError::FileExists,\n        OpenWriteError::IoError { io_error, .. } => TryAcquireLockError::IoError(io_error),\n    })?;\n    write.flush().map_err(TryAcquireLockError::from)?;\n    Ok(DirectoryLock::from(Box::new(DirectoryLockGuard {\n        directory: directory.box_clone(),\n        path: filepath.to_owned(),\n    })))\n}\n\nfn retry_policy(is_blocking: bool) -> RetryPolicy {\n    if is_blocking {\n        RetryPolicy {\n            num_retries: 100,\n            wait_in_ms: 100,\n        }\n    } else {\n        RetryPolicy::no_retry()\n    }\n}\n\n/// Write-once read many (WORM) abstraction for where\n/// tantivy's data should be stored.\n///\n/// There are currently two implementations of `Directory`\n///\n/// - The [`MMapDirectory`][crate::directory::MmapDirectory], this should be your default choice.\n/// - The [`RamDirectory`][crate::directory::RamDirectory], which should be used mostly for tests.\npub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {\n    /// Opens a file and returns a boxed `FileHandle`.\n    ///\n    /// Users of `Directory` should typically call `Directory::open_read(...)`,\n    /// while `Directory` implementer should implement `get_file_handle()`.\n    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError>;\n\n    /// Once a virtual file is open, its data may not\n    /// change.\n    ///\n    /// Specifically, subsequent writes or flushes should\n    /// have no effect on the returned [`FileSlice`] object.\n    ///\n    /// You should only use this to read files create with [`Directory::open_write()`].\n    fn open_read(&self, path: &Path) -> Result<FileSlice, OpenReadError> {\n        let file_handle = self.get_file_handle(path)?;\n        Ok(FileSlice::new(file_handle))\n    }\n\n    /// Removes a file\n    ///\n    /// Removing a file will not affect an eventual\n    /// existing [`FileSlice`] pointing to it.\n    ///\n    /// Removing a nonexistent file, returns a\n    /// [`DeleteError::FileDoesNotExist`].\n    fn delete(&self, path: &Path) -> Result<(), DeleteError>;\n\n    /// Returns true if and only if the file exists\n    fn exists(&self, path: &Path) -> Result<bool, OpenReadError>;\n\n    /// Opens a writer for the *virtual file* associated with\n    /// a [`Path`].\n    ///\n    /// Right after this call, for the span of the execution of the program\n    /// the file should be created and any subsequent call to\n    /// [`Directory::open_read()`] for the same path should return\n    /// a [`FileSlice`].\n    ///\n    /// However, depending on the directory implementation,\n    /// it might be required to call [`Directory::sync_directory()`] to ensure\n    /// that the file is durably created.\n    /// (The semantics here are the same when dealing with\n    /// a POSIX filesystem.)\n    ///\n    /// Write operations may be aggressively buffered.\n    /// The client of this trait is responsible for calling flush\n    /// to ensure that subsequent `read` operations\n    /// will take into account preceding `write` operations.\n    ///\n    /// Flush operation should also be persistent.\n    ///\n    /// The user shall not rely on [`Drop`] triggering `flush`.\n    /// Note that [`RamDirectory`][crate::directory::RamDirectory] will\n    /// panic! if `flush` was not called.\n    ///\n    /// The file may not previously exist.\n    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError>;\n\n    /// Reads the full content file that has been written using\n    /// [`Directory::atomic_write()`].\n    ///\n    /// This should only be used for small files.\n    ///\n    /// You should only use this to read files create with [`Directory::atomic_write()`].\n    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError>;\n\n    /// Atomically replace the content of a file with data.\n    ///\n    /// This calls ensure that reads can never *observe*\n    /// a partially written file.\n    ///\n    /// The file may or may not previously exist.\n    fn atomic_write(&self, path: &Path, data: &[u8]) -> io::Result<()>;\n\n    /// Sync the directory.\n    ///\n    /// This call is required to ensure that newly created files are\n    /// effectively stored durably.\n    fn sync_directory(&self) -> io::Result<()>;\n\n    /// Acquire a lock in the directory given in the [`Lock`].\n    ///\n    /// The method is blocking or not depending on the [`Lock`] object.\n    fn acquire_lock(&self, lock: &Lock) -> Result<DirectoryLock, LockError> {\n        let box_directory = self.box_clone();\n        let mut retry_policy = retry_policy(lock.is_blocking);\n        loop {\n            match try_acquire_lock(&lock.filepath, &*box_directory) {\n                Ok(result) => {\n                    return Ok(result);\n                }\n                Err(TryAcquireLockError::FileExists) => {\n                    if !retry_policy.wait_and_retry() {\n                        return Err(LockError::LockBusy);\n                    }\n                }\n                Err(TryAcquireLockError::IoError(io_error)) => {\n                    return Err(LockError::IoError(io_error));\n                }\n            }\n        }\n    }\n\n    /// Registers a callback that will be called whenever a change on the `meta.json`\n    /// using the [`Directory::atomic_write()`] API is detected.\n    ///\n    /// The behavior when using `.watch()` on a file using [`Directory::open_write()`] is, on the\n    /// other hand, undefined.\n    ///\n    /// The file will be watched for the lifetime of the returned `WatchHandle`. The caller is\n    /// required to keep it.\n    /// It does not override previous callbacks. When the file is modified, all callback that are\n    /// registered (and whose [`WatchHandle`] is still alive) are triggered.\n    ///\n    /// Internally, tantivy only uses this API to detect new commits to implement the\n    /// `OnCommitWithDelay` `ReloadPolicy`. Not implementing watch in a `Directory` only prevents\n    /// the `OnCommitWithDelay` `ReloadPolicy` to work properly.\n    fn watch(&self, watch_callback: WatchCallback) -> crate::Result<WatchHandle>;\n}\n\n/// DirectoryClone\npub trait DirectoryClone {\n    /// Clones the directory and boxes the clone\n    fn box_clone(&self) -> Box<dyn Directory>;\n}\n\nimpl<T> DirectoryClone for T\nwhere T: 'static + Directory + Clone\n{\n    fn box_clone(&self) -> Box<dyn Directory> {\n        Box::new(self.clone())\n    }\n}\n\nimpl Clone for Box<dyn Directory> {\n    fn clone(&self) -> Self {\n        self.box_clone()\n    }\n}\n\nimpl<T: Directory + 'static> From<T> for Box<dyn Directory> {\n    fn from(t: T) -> Self {\n        Box::new(t)\n    }\n}\n"
  },
  {
    "path": "src/directory/directory_lock.rs",
    "content": "use std::path::PathBuf;\n\nuse once_cell::sync::Lazy;\n\n/// A directory lock.\n///\n/// A lock is associated with a specific path.\n///\n/// The lock will be passed to [`Directory::acquire_lock`](crate::Directory::acquire_lock).\n///\n/// Tantivy itself uses only two locks but client application\n/// can use the directory facility to define their own locks.\n/// - [`INDEX_WRITER_LOCK`]\n/// - [`META_LOCK`]\n///\n/// Check out these locks documentation for more information.\n#[derive(Debug)]\npub struct Lock {\n    /// The lock needs to be associated with its own file `path`.\n    /// Depending on the platform, the lock might rely on the creation\n    /// and deletion of this filepath.\n    pub filepath: PathBuf,\n    /// `is_blocking` describes whether acquiring the lock is meant\n    /// to be a blocking operation or a non-blocking.\n    ///\n    /// Acquiring a blocking lock blocks until the lock is\n    /// available.\n    ///\n    /// Acquiring a non-blocking lock returns rapidly, either successfully\n    /// or with an error signifying that someone is already holding\n    /// the lock.\n    pub is_blocking: bool,\n}\n\n/// Only one process should be able to write tantivy's index at a time.\n/// This lock file, when present, is in charge of preventing other processes to open an\n/// `IndexWriter`.\n///\n/// If the process is killed and this file remains, it is safe to remove it manually.\n///\n/// Failing to acquire this lock usually means a misuse of tantivy's API,\n/// (creating more than one instance of the `IndexWriter`), are a spurious\n/// lock file remaining after a crash. In the latter case, removing the file after\n/// checking no process running tantivy is running is safe.\npub static INDEX_WRITER_LOCK: Lazy<Lock> = Lazy::new(|| Lock {\n    filepath: PathBuf::from(\".tantivy-writer.lock\"),\n    is_blocking: false,\n});\n/// The meta lock file is here to protect the segment files being opened by\n/// `IndexReader::reload()` from being garbage collected.\n///\n/// It makes it possible for another process to safely consume\n/// our index in-writing. Ideally, we may have preferred `RWLock` semantics\n/// here, but it is difficult to achieve on Windows.\n///\n/// Opening segment readers is a very fast process.\npub static META_LOCK: Lazy<Lock> = Lazy::new(|| Lock {\n    filepath: PathBuf::from(\".tantivy-meta.lock\"),\n    is_blocking: true,\n});\n"
  },
  {
    "path": "src/directory/error.rs",
    "content": "use std::path::PathBuf;\nuse std::sync::Arc;\nuse std::{fmt, io};\n\nuse crate::Version;\n\n/// Error while trying to acquire a directory [lock](crate::directory::Lock).\n///\n/// This is returned from [`Directory::acquire_lock`](crate::Directory::acquire_lock).\n#[derive(Debug, Clone, Error)]\npub enum LockError {\n    /// Failed to acquired a lock as it is already held by another\n    /// client.\n    /// - In the context of a blocking lock, this means the lock was not released within some\n    ///   `timeout` period.\n    /// - In the context of a non-blocking lock, this means the lock was busy at the moment of the\n    ///   call.\n    #[error(\"Could not acquire lock as it is already held, possibly by a different process.\")]\n    LockBusy,\n    /// Trying to acquire a lock failed with an `IoError`\n    #[error(\"Failed to acquire the lock due to an io:Error.\")]\n    IoError(Arc<io::Error>),\n}\n\nimpl LockError {\n    /// Wraps an io error.\n    pub fn wrap_io_error(io_error: io::Error) -> Self {\n        Self::IoError(Arc::new(io_error))\n    }\n}\n\n/// Error that may occur when opening a directory\n#[derive(Debug, Clone, Error)]\npub enum OpenDirectoryError {\n    /// The underlying directory does not exist.\n    #[error(\"Directory does not exist: '{0}'.\")]\n    DoesNotExist(PathBuf),\n    /// The path exists but is not a directory.\n    #[error(\"Path exists but is not a directory: '{0}'.\")]\n    NotADirectory(PathBuf),\n    /// Failed to create a temp directory.\n    #[error(\"Failed to create a temporary directory: '{0}'.\")]\n    FailedToCreateTempDir(Arc<io::Error>),\n    /// IoError\n    #[error(\"IoError '{io_error:?}' while create directory in: '{directory_path:?}'.\")]\n    IoError {\n        /// underlying io Error.\n        io_error: Arc<io::Error>,\n        /// directory we tried to open.\n        directory_path: PathBuf,\n    },\n}\n\nimpl OpenDirectoryError {\n    /// Wraps an io error.\n    pub fn wrap_io_error(io_error: io::Error, directory_path: PathBuf) -> Self {\n        Self::IoError {\n            io_error: Arc::new(io_error),\n            directory_path,\n        }\n    }\n}\n\n/// Error that may occur when starting to write in a file\n#[derive(Debug, Clone, Error)]\npub enum OpenWriteError {\n    /// Our directory is WORM, writing an existing file is forbidden.\n    /// Checkout the `Directory` documentation.\n    #[error(\"File already exists: '{0}'\")]\n    FileAlreadyExists(PathBuf),\n    /// Any kind of IO error that happens when\n    /// writing in the underlying IO device.\n    #[error(\"IoError '{io_error:?}' while opening file for write: '{filepath}'.\")]\n    IoError {\n        /// The underlying `io::Error`.\n        io_error: Arc<io::Error>,\n        /// File path of the file that tantivy failed to open for write.\n        filepath: PathBuf,\n    },\n}\n\nimpl OpenWriteError {\n    /// Wraps an io error.\n    pub fn wrap_io_error(io_error: io::Error, filepath: PathBuf) -> Self {\n        Self::IoError {\n            io_error: Arc::new(io_error),\n            filepath,\n        }\n    }\n}\n/// Type of index incompatibility between the library and the index found on disk\n/// Used to catch and provide a hint to solve this incompatibility issue\n#[derive(Clone)]\npub enum Incompatibility {\n    /// This library cannot decompress the index found on disk\n    CompressionMismatch {\n        /// Compression algorithm used by the current version of tantivy\n        library_compression_format: String,\n        /// Compression algorithm that was used to serialise the index\n        index_compression_format: String,\n    },\n    /// The index format found on disk isn't supported by this version of the library\n    IndexMismatch {\n        /// Version used by the library\n        library_version: Version,\n        /// Version the index was built with\n        index_version: Version,\n    },\n}\n\nimpl fmt::Debug for Incompatibility {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        match self {\n            Incompatibility::CompressionMismatch {\n                library_compression_format,\n                index_compression_format,\n            } => {\n                let err = format!(\n                    \"Library was compiled with {library_compression_format:?} compression, index \\\n                     was compressed with {index_compression_format:?}\"\n                );\n                let advice = format!(\n                    \"Change the feature flag to {index_compression_format:?} and rebuild the \\\n                     library\"\n                );\n                write!(f, \"{err}. {advice}\")?;\n            }\n            Incompatibility::IndexMismatch {\n                library_version,\n                index_version,\n            } => {\n                let err = format!(\n                    \"Library version: {}, index version: {}\",\n                    library_version.index_format_version, index_version.index_format_version\n                );\n                // TODO make a more useful error message\n                // include the version range that supports this index_format_version\n                let advice = format!(\n                    \"Change tantivy to a version compatible with index format {} (e.g. {}.{}.x) \\\n                     and rebuild your project.\",\n                    index_version.index_format_version, index_version.major, index_version.minor\n                );\n                write!(f, \"{err}. {advice}\")?;\n            }\n        }\n\n        Ok(())\n    }\n}\n\n/// Error that may occur when accessing a file read\n#[derive(Debug, Clone, Error)]\npub enum OpenReadError {\n    /// The file does not exist.\n    #[error(\"Files does not exist: {0:?}\")]\n    FileDoesNotExist(PathBuf),\n    /// Any kind of io::Error.\n    #[error(\n        \"IoError: '{io_error:?}' happened while opening the following file for Read: {filepath}.\"\n    )]\n    IoError {\n        /// The underlying `io::Error`.\n        io_error: Arc<io::Error>,\n        /// File path of the file that tantivy failed to open for read.\n        filepath: PathBuf,\n    },\n    /// This library does not support the index version found in file footer.\n    #[error(\"Index version unsupported: {0:?}\")]\n    IncompatibleIndex(Incompatibility),\n}\n\nimpl OpenReadError {\n    /// Wraps an io error.\n    pub fn wrap_io_error(io_error: io::Error, filepath: PathBuf) -> Self {\n        Self::IoError {\n            io_error: Arc::new(io_error),\n            filepath,\n        }\n    }\n}\n/// Error that may occur when trying to delete a file\n#[derive(Debug, Clone, Error)]\npub enum DeleteError {\n    /// The file does not exist.\n    #[error(\"File does not exist: '{0}'.\")]\n    FileDoesNotExist(PathBuf),\n    /// Any kind of IO error that happens when\n    /// interacting with the underlying IO device.\n    #[error(\"The following IO error happened while deleting file '{filepath}': '{io_error:?}'.\")]\n    IoError {\n        /// The underlying `io::Error`.\n        io_error: Arc<io::Error>,\n        /// File path of the file that tantivy failed to delete.\n        filepath: PathBuf,\n    },\n}\n\nimpl From<Incompatibility> for OpenReadError {\n    fn from(incompatibility: Incompatibility) -> Self {\n        OpenReadError::IncompatibleIndex(incompatibility)\n    }\n}\n"
  },
  {
    "path": "src/directory/footer.rs",
    "content": "//! The footer is a small metadata structure that is appended at the end of every file.\n//!\n//! The footer is used to store a checksum of the file content.\n//! The footer also stores the version of the index format.\n//! This version is used to detect incompatibility between the index and the library version.\n\nuse std::io;\nuse std::io::Write;\n\nuse common::{BinarySerializable, CountingWriter, DeserializeFrom, FixedSize, HasLen};\nuse crc32fast::Hasher;\nuse serde::{Deserialize, Serialize};\n\nuse crate::directory::error::Incompatibility;\nuse crate::directory::{AntiCallToken, FileSlice, TerminatingWrite};\nuse crate::{Version, INDEX_FORMAT_OLDEST_SUPPORTED_VERSION, INDEX_FORMAT_VERSION};\n\nconst FOOTER_MAX_LEN: u32 = 50_000;\n\n/// The magic byte of the footer to identify corruption\n/// or an old version of the footer.\nconst FOOTER_MAGIC_NUMBER: u32 = 1337;\n\ntype CrcHashU32 = u32;\n\n/// A Footer is appended to every file\n#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]\npub struct Footer {\n    /// The version of the index format\n    pub version: Version,\n    /// The crc32 hash of the body\n    pub crc: CrcHashU32,\n}\n\nimpl Footer {\n    pub(crate) fn new(crc: CrcHashU32) -> Self {\n        let version = crate::VERSION.clone();\n        Footer { version, crc }\n    }\n\n    pub(crate) fn crc(&self) -> CrcHashU32 {\n        self.crc\n    }\n    pub(crate) fn append_footer<W: io::Write>(&self, mut write: &mut W) -> io::Result<()> {\n        let mut counting_write = CountingWriter::wrap(&mut write);\n        counting_write.write_all(serde_json::to_string(&self)?.as_ref())?;\n        let footer_payload_len = counting_write.written_bytes();\n        BinarySerializable::serialize(&(footer_payload_len as u32), write)?;\n        BinarySerializable::serialize(&FOOTER_MAGIC_NUMBER, write)?;\n        Ok(())\n    }\n\n    /// Extracts the tantivy Footer from the file and returns the footer and the rest of the file\n    pub fn extract_footer(file: FileSlice) -> io::Result<(Footer, FileSlice)> {\n        if file.len() < 4 {\n            return Err(io::Error::new(\n                io::ErrorKind::UnexpectedEof,\n                format!(\n                    \"File corrupted. The file is smaller than 4 bytes (len={}).\",\n                    file.len()\n                ),\n            ));\n        }\n\n        let footer_metadata_len = <(u32, u32)>::SIZE_IN_BYTES;\n        let (footer_len, footer_magic_byte): (u32, u32) = file\n            .slice_from_end(footer_metadata_len)\n            .read_bytes()?\n            .as_ref()\n            .deserialize()?;\n\n        if footer_magic_byte != FOOTER_MAGIC_NUMBER {\n            return Err(io::Error::new(\n                io::ErrorKind::InvalidData,\n                \"Footer magic byte mismatch. File corrupted or index was created using old an \\\n                 tantivy version which is not supported anymore. Please use tantivy 0.15 or above \\\n                 to recreate the index.\",\n            ));\n        }\n\n        if footer_len > FOOTER_MAX_LEN {\n            return Err(io::Error::new(\n                io::ErrorKind::InvalidData,\n                format!(\n                    \"Footer seems invalid as it suggests a footer len of {footer_len}. File is \\\n                     corrupted, or the index was created with a different & old version of \\\n                     tantivy.\"\n                ),\n            ));\n        }\n        let total_footer_size = footer_len as usize + footer_metadata_len;\n        if file.len() < total_footer_size {\n            return Err(io::Error::new(\n                io::ErrorKind::UnexpectedEof,\n                format!(\n                    \"File corrupted. The file is smaller than it's footer bytes \\\n                     (len={total_footer_size}).\"\n                ),\n            ));\n        }\n\n        let footer: Footer =\n            serde_json::from_slice(&file.read_bytes_slice(\n                file.len() - total_footer_size..file.len() - footer_metadata_len,\n            )?)?;\n\n        let body = file.slice_to(file.len() - total_footer_size);\n        Ok((footer, body))\n    }\n\n    /// Confirms that the index will be read correctly by this version of tantivy\n    /// Has to be called after `extract_footer` to make sure it's not accessing uninitialised memory\n    pub fn is_compatible(&self) -> Result<(), Incompatibility> {\n        const SUPPORTED_INDEX_FORMAT_VERSION_RANGE: std::ops::RangeInclusive<u32> =\n            INDEX_FORMAT_OLDEST_SUPPORTED_VERSION..=INDEX_FORMAT_VERSION;\n\n        let library_version = crate::version();\n        if !SUPPORTED_INDEX_FORMAT_VERSION_RANGE.contains(&self.version.index_format_version) {\n            return Err(Incompatibility::IndexMismatch {\n                library_version: library_version.clone(),\n                index_version: self.version.clone(),\n            });\n        }\n        Ok(())\n    }\n}\n\npub(crate) struct FooterProxy<W: TerminatingWrite> {\n    /// always Some except after terminate call\n    hasher: Option<Hasher>,\n    /// always Some except after terminate call\n    writer: Option<W>,\n}\n\nimpl<W: TerminatingWrite> FooterProxy<W> {\n    pub fn new(writer: W) -> Self {\n        FooterProxy {\n            hasher: Some(Hasher::new()),\n            writer: Some(writer),\n        }\n    }\n}\n\nimpl<W: TerminatingWrite> Write for FooterProxy<W> {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        let count = self.writer.as_mut().unwrap().write(buf)?;\n        self.hasher.as_mut().unwrap().update(&buf[..count]);\n        Ok(count)\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        self.writer.as_mut().unwrap().flush()\n    }\n}\n\nimpl<W: TerminatingWrite> TerminatingWrite for FooterProxy<W> {\n    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {\n        let crc32 = self.hasher.take().unwrap().finalize();\n        let footer = Footer::new(crc32);\n        let mut writer = self.writer.take().unwrap();\n        footer.append_footer(&mut writer)?;\n        writer.terminate()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::io;\n    use std::sync::Arc;\n\n    use common::BinarySerializable;\n\n    use crate::directory::footer::{Footer, FOOTER_MAGIC_NUMBER};\n    use crate::directory::{FileSlice, OwnedBytes};\n\n    #[test]\n    fn test_deserialize_footer() {\n        let mut buf: Vec<u8> = vec![];\n        let footer = Footer::new(123);\n        footer.append_footer(&mut buf).unwrap();\n        let owned_bytes = OwnedBytes::new(buf);\n        let fileslice = FileSlice::new(Arc::new(owned_bytes));\n        let (footer_deser, _body) = Footer::extract_footer(fileslice).unwrap();\n        assert_eq!(footer_deser.crc(), footer.crc());\n    }\n    #[test]\n    fn test_deserialize_footer_missing_magic_byte() {\n        let mut buf: Vec<u8> = vec![];\n        BinarySerializable::serialize(&0_u32, &mut buf).unwrap();\n        let wrong_magic_byte: u32 = 5555;\n        BinarySerializable::serialize(&wrong_magic_byte, &mut buf).unwrap();\n\n        let owned_bytes = OwnedBytes::new(buf);\n\n        let fileslice = FileSlice::new(Arc::new(owned_bytes));\n        let err = Footer::extract_footer(fileslice).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Footer magic byte mismatch. File corrupted or index was created using old an tantivy \\\n             version which is not supported anymore. Please use tantivy 0.15 or above to recreate \\\n             the index.\"\n        );\n    }\n    #[test]\n    fn test_deserialize_footer_wrong_filesize() {\n        let mut buf: Vec<u8> = vec![];\n        BinarySerializable::serialize(&100_u32, &mut buf).unwrap();\n        BinarySerializable::serialize(&FOOTER_MAGIC_NUMBER, &mut buf).unwrap();\n\n        let owned_bytes = OwnedBytes::new(buf);\n\n        let fileslice = FileSlice::new(Arc::new(owned_bytes));\n        let err = Footer::extract_footer(fileslice).unwrap_err();\n        assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof);\n        assert_eq!(\n            err.to_string(),\n            \"File corrupted. The file is smaller than it\\'s footer bytes (len=108).\"\n        );\n    }\n\n    #[test]\n    fn test_deserialize_too_large_footer() {\n        let mut buf: Vec<u8> = vec![];\n\n        let footer_length = super::FOOTER_MAX_LEN + 1;\n        BinarySerializable::serialize(&footer_length, &mut buf).unwrap();\n        BinarySerializable::serialize(&FOOTER_MAGIC_NUMBER, &mut buf).unwrap();\n\n        let owned_bytes = OwnedBytes::new(buf);\n\n        let fileslice = FileSlice::new(Arc::new(owned_bytes));\n        let err = Footer::extract_footer(fileslice).unwrap_err();\n        assert_eq!(err.kind(), io::ErrorKind::InvalidData);\n        assert_eq!(\n            err.to_string(),\n            \"Footer seems invalid as it suggests a footer len of 50001. File is corrupted, or the \\\n             index was created with a different & old version of tantivy.\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/directory/managed_directory.rs",
    "content": "use std::collections::HashSet;\nuse std::io::Write;\nuse std::path::{Path, PathBuf};\nuse std::sync::{Arc, RwLock, RwLockWriteGuard};\nuse std::{io, result};\n\nuse crc32fast::Hasher;\n\nuse crate::core::MANAGED_FILEPATH;\nuse crate::directory::error::{DeleteError, LockError, OpenReadError, OpenWriteError};\nuse crate::directory::footer::{Footer, FooterProxy};\nuse crate::directory::{\n    DirectoryLock, FileHandle, FileSlice, GarbageCollectionResult, Lock, WatchCallback,\n    WatchHandle, WritePtr, META_LOCK,\n};\nuse crate::error::DataCorruption;\nuse crate::Directory;\n\n/// Returns true if the file is \"managed\".\n/// Non-managed file are not subject to garbage collection.\n///\n/// Filenames that starts by a \".\" -typically locks-\n/// are not managed.\nfn is_managed(path: &Path) -> bool {\n    path.to_str()\n        .map(|p_str| !p_str.starts_with('.'))\n        .unwrap_or(true)\n}\n\n/// Wrapper of directories that keeps track of files created by Tantivy.\n///\n/// A managed directory is just a wrapper of a directory\n/// that keeps a (persisted) list of the files that\n/// have been created (and not deleted) by tantivy so far.\n///\n/// Thanks to this list, it implements a `garbage_collect` method\n/// that removes the files that were created by tantivy and are not\n/// useful anymore.\n#[derive(Debug)]\npub struct ManagedDirectory {\n    directory: Box<dyn Directory>,\n    meta_informations: Arc<RwLock<MetaInformation>>,\n}\n\n#[derive(Debug, Default)]\nstruct MetaInformation {\n    managed_paths: HashSet<PathBuf>,\n}\n\n/// Saves the file containing the list of existing files\n/// that were created by tantivy.\nfn save_managed_paths(\n    directory: &dyn Directory,\n    wlock: &RwLockWriteGuard<'_, MetaInformation>,\n) -> io::Result<()> {\n    let mut w = serde_json::to_vec(&wlock.managed_paths)?;\n    writeln!(&mut w)?;\n    directory.atomic_write(&MANAGED_FILEPATH, &w[..])?;\n    Ok(())\n}\n\nimpl ManagedDirectory {\n    /// Wraps a directory as managed directory.\n    pub fn wrap(directory: Box<dyn Directory>) -> crate::Result<ManagedDirectory> {\n        match directory.atomic_read(&MANAGED_FILEPATH) {\n            Ok(data) => {\n                let managed_files_json = String::from_utf8_lossy(&data);\n                let managed_files: HashSet<PathBuf> = serde_json::from_str(&managed_files_json)\n                    .map_err(|e| {\n                        DataCorruption::new(\n                            MANAGED_FILEPATH.to_path_buf(),\n                            format!(\"Managed file cannot be deserialized: {e:?}. \"),\n                        )\n                    })?;\n                Ok(ManagedDirectory {\n                    directory,\n                    meta_informations: Arc::new(RwLock::new(MetaInformation {\n                        managed_paths: managed_files,\n                    })),\n                })\n            }\n            Err(OpenReadError::FileDoesNotExist(_)) => Ok(ManagedDirectory {\n                directory,\n                meta_informations: Arc::default(),\n            }),\n            io_err @ Err(OpenReadError::IoError { .. }) => Err(io_err.err().unwrap().into()),\n            Err(OpenReadError::IncompatibleIndex(incompatibility)) => {\n                // For the moment, this should never happen  `meta.json`\n                // do not have any footer and cannot detect incompatibility.\n                Err(crate::TantivyError::IncompatibleIndex(incompatibility))\n            }\n        }\n    }\n\n    /// Garbage collect unused files.\n    ///\n    /// Removes the files that were created by `tantivy` and are not\n    /// used by any segment anymore.\n    ///\n    /// * `living_files` - List of files that are still used by the index.\n    ///\n    /// The use a callback ensures that the list of living_files is computed\n    /// while we hold the lock on meta.\n    ///\n    /// This method does not panick nor returns errors.\n    /// If a file cannot be deleted (for permission reasons for instance)\n    /// an error is simply logged, and the file remains in the list of managed\n    /// files.\n    pub fn garbage_collect<L: FnOnce() -> HashSet<PathBuf>>(\n        &mut self,\n        get_living_files: L,\n    ) -> crate::Result<GarbageCollectionResult> {\n        info!(\"Garbage collect\");\n        let mut files_to_delete = vec![];\n\n        // It is crucial to get the living files after acquiring the\n        // read lock of meta information. That way, we\n        // avoid the following scenario.\n        //\n        // 1) we get the list of living files.\n        // 2) someone creates a new file.\n        // 3) we start garbage collection and remove this file\n        // even though it is a living file.\n        //\n        // releasing the lock as .delete() will use it too.\n        {\n            let meta_informations_rlock = self\n                .meta_informations\n                .read()\n                .expect(\"Managed directory rlock poisoned in garbage collect.\");\n\n            // The point of this second \"file\" lock is to enforce the following scenario\n            // 1) process B tries to load a new set of searcher.\n            // The list of segments is loaded\n            // 2) writer change meta.json (for instance after a merge or a commit)\n            // 3) gc kicks in.\n            // 4) gc removes a file that was useful for process B, before process B opened it.\n            match self.acquire_lock(&META_LOCK) {\n                Ok(_meta_lock) => {\n                    let living_files = get_living_files();\n                    for managed_path in &meta_informations_rlock.managed_paths {\n                        if !living_files.contains(managed_path) {\n                            files_to_delete.push(managed_path.clone());\n                        }\n                    }\n                }\n                Err(err) => {\n                    error!(\"Failed to acquire lock for GC\");\n                    return Err(crate::TantivyError::from(err));\n                }\n            }\n        }\n\n        let mut failed_to_delete_files = vec![];\n        let mut deleted_files = vec![];\n\n        for file_to_delete in files_to_delete {\n            match self.delete(&file_to_delete) {\n                Ok(_) => {\n                    info!(\"Deleted {file_to_delete:?}\");\n                    deleted_files.push(file_to_delete);\n                }\n                Err(file_error) => {\n                    match file_error {\n                        DeleteError::FileDoesNotExist(_) => {\n                            deleted_files.push(file_to_delete.clone());\n                        }\n                        DeleteError::IoError { .. } => {\n                            failed_to_delete_files.push(file_to_delete.clone());\n                            if !cfg!(target_os = \"windows\") {\n                                // On windows, delete is expected to fail if the file\n                                // is mmapped.\n                                error!(\"Failed to delete {file_to_delete:?}\");\n                            }\n                        }\n                    }\n                }\n            }\n        }\n\n        if !deleted_files.is_empty() {\n            // update the list of managed files by removing\n            // the file that were removed.\n            let mut meta_informations_wlock = self\n                .meta_informations\n                .write()\n                .expect(\"Managed directory wlock poisoned (2).\");\n            let managed_paths_write = &mut meta_informations_wlock.managed_paths;\n            for delete_file in &deleted_files {\n                managed_paths_write.remove(delete_file);\n            }\n            self.directory.sync_directory()?;\n            save_managed_paths(self.directory.as_mut(), &meta_informations_wlock)?;\n        }\n\n        Ok(GarbageCollectionResult {\n            deleted_files,\n            failed_to_delete_files,\n        })\n    }\n\n    /// Registers a file as managed\n    ///\n    /// This method must be called before the file is\n    /// actually created to ensure that a failure between\n    /// registering the filepath and creating the file\n    /// will not lead to garbage files that will\n    /// never get removed.\n    ///\n    /// File starting by \".\" are reserved to locks.\n    /// They are not managed and cannot be subjected\n    /// to garbage collection.\n    fn register_file_as_managed(&self, filepath: &Path) -> io::Result<()> {\n        // Files starting by \".\" (e.g. lock files) are not managed.\n        if !is_managed(filepath) {\n            return Ok(());\n        }\n        let mut meta_wlock = self\n            .meta_informations\n            .write()\n            .expect(\"Managed file lock poisoned\");\n        let has_changed = meta_wlock.managed_paths.insert(filepath.to_owned());\n        if !has_changed {\n            return Ok(());\n        }\n        save_managed_paths(self.directory.as_ref(), &meta_wlock)?;\n        // This is not the first file we add.\n        // Therefore, we are sure that `.managed.json` has been already\n        // properly created and we do not need to sync its parent directory.\n        //\n        // (It might seem like a nicer solution to create the managed_json on the\n        // creation of the ManagedDirectory instance but it would actually\n        // prevent the use of read-only directories..)\n        let managed_file_definitely_already_exists = meta_wlock.managed_paths.len() > 1;\n        if managed_file_definitely_already_exists {\n            return Ok(());\n        }\n        self.directory.sync_directory()?;\n        Ok(())\n    }\n\n    /// Verify checksum of a managed file\n    pub fn validate_checksum(&self, path: &Path) -> result::Result<bool, OpenReadError> {\n        let reader = self.directory.open_read(path)?;\n        let (footer, data) = Footer::extract_footer(reader)\n            .map_err(|io_error| OpenReadError::wrap_io_error(io_error, path.to_path_buf()))?;\n        let bytes = data\n            .read_bytes()\n            .map_err(|io_error| OpenReadError::IoError {\n                io_error: Arc::new(io_error),\n                filepath: path.to_path_buf(),\n            })?;\n        let mut hasher = Hasher::new();\n        hasher.update(bytes.as_slice());\n        let crc = hasher.finalize();\n        Ok(footer.crc() == crc)\n    }\n\n    /// List all managed files\n    pub fn list_managed_files(&self) -> HashSet<PathBuf> {\n        let managed_paths = self\n            .meta_informations\n            .read()\n            .expect(\"Managed directory rlock poisoned in list damaged.\")\n            .managed_paths\n            .clone();\n        managed_paths\n    }\n}\n\nimpl Directory for ManagedDirectory {\n    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {\n        let file_slice = self.open_read(path)?;\n        Ok(Arc::new(file_slice))\n    }\n\n    fn open_read(&self, path: &Path) -> result::Result<FileSlice, OpenReadError> {\n        let file_slice = self.directory.open_read(path)?;\n        let (footer, reader) = Footer::extract_footer(file_slice)\n            .map_err(|io_error| OpenReadError::wrap_io_error(io_error, path.to_path_buf()))?;\n        footer.is_compatible()?;\n        Ok(reader)\n    }\n\n    fn open_write(&self, path: &Path) -> result::Result<WritePtr, OpenWriteError> {\n        self.register_file_as_managed(path)\n            .map_err(|io_error| OpenWriteError::wrap_io_error(io_error, path.to_path_buf()))?;\n        Ok(io::BufWriter::new(Box::new(FooterProxy::new(\n            self.directory\n                .open_write(path)?\n                .into_inner()\n                .map_err(|_| ())\n                .expect(\"buffer should be empty\"),\n        ))))\n    }\n\n    fn atomic_write(&self, path: &Path, data: &[u8]) -> io::Result<()> {\n        self.register_file_as_managed(path)?;\n        self.directory.atomic_write(path, data)\n    }\n\n    fn atomic_read(&self, path: &Path) -> result::Result<Vec<u8>, OpenReadError> {\n        self.directory.atomic_read(path)\n    }\n\n    fn delete(&self, path: &Path) -> result::Result<(), DeleteError> {\n        self.directory.delete(path)\n    }\n\n    fn exists(&self, path: &Path) -> Result<bool, OpenReadError> {\n        self.directory.exists(path)\n    }\n\n    fn acquire_lock(&self, lock: &Lock) -> result::Result<DirectoryLock, LockError> {\n        self.directory.acquire_lock(lock)\n    }\n\n    fn watch(&self, watch_callback: WatchCallback) -> crate::Result<WatchHandle> {\n        self.directory.watch(watch_callback)\n    }\n\n    fn sync_directory(&self) -> io::Result<()> {\n        self.directory.sync_directory()?;\n        Ok(())\n    }\n}\n\nimpl Clone for ManagedDirectory {\n    fn clone(&self) -> ManagedDirectory {\n        ManagedDirectory {\n            directory: self.directory.box_clone(),\n            meta_informations: Arc::clone(&self.meta_informations),\n        }\n    }\n}\n\n#[cfg(feature = \"mmap\")]\n#[cfg(test)]\nmod tests_mmap_specific {\n\n    use std::collections::HashSet;\n    use std::io::Write;\n    use std::path::{Path, PathBuf};\n\n    use tempfile::TempDir;\n\n    use crate::directory::{Directory, ManagedDirectory, MmapDirectory, TerminatingWrite};\n\n    #[test]\n    fn test_managed_directory() {\n        let tempdir = TempDir::new().unwrap();\n        let tempdir_path = PathBuf::from(tempdir.path());\n\n        let test_path1: &'static Path = Path::new(\"some_path_for_test\");\n        let test_path2: &'static Path = Path::new(\"some_path_for_test_2\");\n        {\n            let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();\n            let mut managed_directory = ManagedDirectory::wrap(Box::new(mmap_directory)).unwrap();\n            let write_file = managed_directory.open_write(test_path1).unwrap();\n            write_file.terminate().unwrap();\n            managed_directory\n                .atomic_write(test_path2, &[0u8, 1u8])\n                .unwrap();\n            assert!(managed_directory.exists(test_path1).unwrap());\n            assert!(managed_directory.exists(test_path2).unwrap());\n            let living_files: HashSet<PathBuf> = [test_path1.to_owned()].iter().cloned().collect();\n            assert!(managed_directory.garbage_collect(|| living_files).is_ok());\n            assert!(managed_directory.exists(test_path1).unwrap());\n            assert!(!managed_directory.exists(test_path2).unwrap());\n        }\n        {\n            let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();\n            let mut managed_directory = ManagedDirectory::wrap(Box::new(mmap_directory)).unwrap();\n            assert!(managed_directory.exists(test_path1).unwrap());\n            assert!(!managed_directory.exists(test_path2).unwrap());\n            let living_files: HashSet<PathBuf> = HashSet::new();\n            assert!(managed_directory.garbage_collect(|| living_files).is_ok());\n            assert!(!managed_directory.exists(test_path1).unwrap());\n            assert!(!managed_directory.exists(test_path2).unwrap());\n        }\n    }\n\n    #[test]\n    fn test_managed_directory_gc_while_mmapped() {\n        let test_path1: &'static Path = Path::new(\"some_path_for_test\");\n\n        let tempdir = TempDir::new().unwrap();\n        let tempdir_path = PathBuf::from(tempdir.path());\n        let living_files = HashSet::new();\n\n        let mmap_directory = MmapDirectory::open(tempdir_path).unwrap();\n        let mut managed_directory = ManagedDirectory::wrap(Box::new(mmap_directory)).unwrap();\n        let mut write = managed_directory.open_write(test_path1).unwrap();\n        write.write_all(&[0u8, 1u8]).unwrap();\n        write.terminate().unwrap();\n        assert!(managed_directory.exists(test_path1).unwrap());\n\n        let _mmap_read = managed_directory.open_read(test_path1).unwrap();\n        assert!(managed_directory\n            .garbage_collect(|| living_files.clone())\n            .is_ok());\n        if cfg!(target_os = \"windows\") {\n            // On Windows, gc should try and fail the file as it is mmapped.\n            assert!(managed_directory.exists(test_path1).unwrap());\n            // unmap should happen here.\n            drop(_mmap_read);\n            // The file should still be in the list of managed file and\n            // eventually be deleted once mmap is released.\n            assert!(managed_directory.garbage_collect(|| living_files).is_ok());\n        }\n        assert!(!managed_directory.exists(test_path1).unwrap());\n    }\n}\n"
  },
  {
    "path": "src/directory/mmap_directory/file_watcher.rs",
    "content": "use std::io::BufRead;\nuse std::path::Path;\nuse std::sync::atomic::{AtomicUsize, Ordering};\nuse std::sync::Arc;\nuse std::time::Duration;\nuse std::{fs, io, thread};\n\nuse crc32fast::Hasher;\n\nuse crate::directory::{WatchCallback, WatchCallbackList, WatchHandle};\n\nconst POLLING_INTERVAL: Duration = Duration::from_millis(if cfg!(test) { 1 } else { 500 });\n\n// Watches a file and executes registered callbacks when the file is modified.\npub struct FileWatcher {\n    path: Arc<Path>,\n    callbacks: Arc<WatchCallbackList>,\n    state: Arc<AtomicUsize>, // 0: new, 1: runnable, 2: terminated\n}\n\nimpl FileWatcher {\n    pub fn new(path: &Path) -> FileWatcher {\n        FileWatcher {\n            path: Arc::from(path),\n            callbacks: Default::default(),\n            state: Default::default(),\n        }\n    }\n\n    pub fn spawn(&self) {\n        if self\n            .state\n            .compare_exchange(0, 1, Ordering::SeqCst, Ordering::SeqCst)\n            .is_err()\n        {\n            return;\n        }\n\n        let path = self.path.clone();\n        let callbacks = self.callbacks.clone();\n        let state = self.state.clone();\n\n        thread::Builder::new()\n            .name(\"thread-tantivy-meta-file-watcher\".to_string())\n            .spawn(move || {\n                let mut current_checksum_opt = None;\n\n                while state.load(Ordering::SeqCst) == 1 {\n                    if let Ok(checksum) = FileWatcher::compute_checksum(&path) {\n                        let metafile_has_changed = current_checksum_opt\n                            .map(|current_checksum| current_checksum != checksum)\n                            .unwrap_or(true);\n                        if metafile_has_changed {\n                            info!(\"Meta file {path:?} was modified\");\n                            current_checksum_opt = Some(checksum);\n                            // We actually ignore callbacks failing here.\n                            // We just wait for the end of their execution.\n                            let _ = callbacks.broadcast().wait();\n                        }\n                    }\n\n                    thread::sleep(POLLING_INTERVAL);\n                }\n            })\n            .expect(\"Failed to spawn meta file watcher thread\");\n    }\n\n    pub fn watch(&self, callback: WatchCallback) -> WatchHandle {\n        let handle = self.callbacks.subscribe(callback);\n        self.spawn();\n        handle\n    }\n\n    fn compute_checksum(path: &Path) -> Result<u32, io::Error> {\n        let reader = match fs::File::open(path) {\n            Ok(f) => io::BufReader::new(f),\n            Err(e) => {\n                warn!(\"Failed to open meta file {path:?}: {e:?}\");\n                return Err(e);\n            }\n        };\n\n        let mut hasher = Hasher::new();\n\n        for line in reader.lines() {\n            hasher.update(line?.as_bytes())\n        }\n\n        Ok(hasher.finalize())\n    }\n}\n\nimpl Drop for FileWatcher {\n    fn drop(&mut self) {\n        self.state.store(2, Ordering::SeqCst);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::mem;\n\n    use super::*;\n    use crate::directory::mmap_directory::atomic_write;\n\n    #[test]\n    fn test_file_watcher_drop_watcher() -> crate::Result<()> {\n        let tmp_dir = tempfile::TempDir::new()?;\n        let tmp_file = tmp_dir.path().join(\"watched.txt\");\n\n        let counter: Arc<AtomicUsize> = Default::default();\n        let (tx, rx) = crossbeam_channel::unbounded();\n        let timeout = Duration::from_millis(100);\n\n        let watcher = FileWatcher::new(&tmp_file);\n\n        let state = watcher.state.clone();\n        assert_eq!(state.load(Ordering::SeqCst), 0);\n\n        let counter_clone = counter.clone();\n\n        let _handle = watcher.watch(WatchCallback::new(move || {\n            let val = counter_clone.fetch_add(1, Ordering::SeqCst);\n            tx.send(val + 1).unwrap();\n        }));\n\n        assert_eq!(counter.load(Ordering::SeqCst), 0);\n        assert_eq!(state.load(Ordering::SeqCst), 1);\n\n        atomic_write(&tmp_file, b\"foo\")?;\n        assert_eq!(rx.recv_timeout(timeout), Ok(1));\n\n        atomic_write(&tmp_file, b\"foo\")?;\n        assert!(rx.recv_timeout(timeout).is_err());\n\n        atomic_write(&tmp_file, b\"bar\")?;\n        assert_eq!(rx.recv_timeout(timeout), Ok(2));\n\n        mem::drop(watcher);\n\n        atomic_write(&tmp_file, b\"qux\")?;\n        thread::sleep(Duration::from_millis(10));\n        assert_eq!(counter.load(Ordering::SeqCst), 2);\n        assert_eq!(state.load(Ordering::SeqCst), 2);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_file_watcher_drop_handle() -> crate::Result<()> {\n        let tmp_dir = tempfile::TempDir::new()?;\n        let tmp_file = tmp_dir.path().join(\"watched.txt\");\n\n        let counter: Arc<AtomicUsize> = Default::default();\n        let (tx, rx) = crossbeam_channel::unbounded();\n        let timeout = Duration::from_millis(100);\n\n        let watcher = FileWatcher::new(&tmp_file);\n\n        let state = watcher.state.clone();\n        assert_eq!(state.load(Ordering::SeqCst), 0);\n\n        let counter_clone = counter.clone();\n\n        let handle = watcher.watch(WatchCallback::new(move || {\n            let val = counter_clone.fetch_add(1, Ordering::SeqCst);\n            tx.send(val + 1).unwrap();\n        }));\n\n        assert_eq!(counter.load(Ordering::SeqCst), 0);\n        assert_eq!(state.load(Ordering::SeqCst), 1);\n\n        atomic_write(&tmp_file, b\"foo\")?;\n        assert_eq!(rx.recv_timeout(timeout), Ok(1));\n\n        mem::drop(handle);\n\n        atomic_write(&tmp_file, b\"qux\")?;\n        assert_eq!(counter.load(Ordering::SeqCst), 1);\n        assert_eq!(state.load(Ordering::SeqCst), 1);\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/directory/mmap_directory/mod.rs",
    "content": "mod file_watcher;\n\nuse std::collections::HashMap;\nuse std::fmt;\nuse std::fs::{self, File, OpenOptions};\nuse std::io::{self, BufWriter, Read, Write};\nuse std::ops::Deref;\nuse std::path::{Path, PathBuf};\nuse std::sync::{Arc, RwLock, Weak};\n\nuse common::StableDeref;\nuse file_watcher::FileWatcher;\nuse fs4::fs_std::FileExt;\n#[cfg(all(feature = \"mmap\", unix))]\npub use memmap2::Advice;\nuse memmap2::Mmap;\nuse serde::{Deserialize, Serialize};\nuse tempfile::TempDir;\n\nuse crate::core::META_FILEPATH;\nuse crate::directory::error::{\n    DeleteError, LockError, OpenDirectoryError, OpenReadError, OpenWriteError,\n};\nuse crate::directory::{\n    AntiCallToken, Directory, DirectoryLock, FileHandle, Lock, OwnedBytes, TerminatingWrite,\n    WatchCallback, WatchHandle, WritePtr,\n};\n\npub type ArcBytes = Arc<dyn Deref<Target = [u8]> + Send + Sync + 'static>;\npub type WeakArcBytes = Weak<dyn Deref<Target = [u8]> + Send + Sync + 'static>;\n\n/// Create a default io error given a string.\npub(crate) fn make_io_err(msg: String) -> io::Error {\n    io::Error::other(msg)\n}\n\n/// Returns `None` iff the file exists, can be read, but is empty (and hence\n/// cannot be mmapped)\nfn open_mmap(full_path: &Path) -> Result<Option<Mmap>, OpenReadError> {\n    let file = File::open(full_path).map_err(|io_err| {\n        if io_err.kind() == io::ErrorKind::NotFound {\n            OpenReadError::FileDoesNotExist(full_path.to_path_buf())\n        } else {\n            OpenReadError::wrap_io_error(io_err, full_path.to_path_buf())\n        }\n    })?;\n\n    let meta_data = file\n        .metadata()\n        .map_err(|io_err| OpenReadError::wrap_io_error(io_err, full_path.to_owned()))?;\n    if meta_data.len() == 0 {\n        // if the file size is 0, it will not be possible\n        // to mmap the file, so we return None\n        // instead.\n        return Ok(None);\n    }\n    let mmap_opt: Option<memmap2::Mmap> = unsafe {\n        memmap2::Mmap::map(&file)\n            .map(Some)\n            .map_err(|io_err| OpenReadError::wrap_io_error(io_err, full_path.to_path_buf()))\n    }?;\n\n    Ok(mmap_opt)\n}\n\n#[derive(Default, Clone, Debug, Serialize, Deserialize)]\npub struct CacheCounters {\n    /// Number of time the cache prevents to call `mmap`\n    pub hit: usize,\n    /// Number of time tantivy had to call `mmap`\n    /// as no entry was in the cache.\n    pub miss: usize,\n}\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct CacheInfo {\n    pub counters: CacheCounters,\n    pub mmapped: Vec<PathBuf>,\n}\n\nstruct MmapCache {\n    counters: CacheCounters,\n    cache: HashMap<PathBuf, WeakArcBytes>,\n    #[cfg(unix)]\n    madvice_opt: Option<Advice>,\n}\n\nimpl MmapCache {\n    fn new() -> MmapCache {\n        MmapCache {\n            counters: CacheCounters::default(),\n            cache: HashMap::default(),\n            #[cfg(unix)]\n            madvice_opt: None,\n        }\n    }\n\n    #[cfg(unix)]\n    fn set_advice(&mut self, madvice: Advice) {\n        self.madvice_opt = Some(madvice);\n    }\n\n    fn get_info(&self) -> CacheInfo {\n        let paths: Vec<PathBuf> = self.cache.keys().cloned().collect();\n        CacheInfo {\n            counters: self.counters.clone(),\n            mmapped: paths,\n        }\n    }\n\n    fn remove_weak_ref(&mut self) {\n        let keys_to_remove: Vec<PathBuf> = self\n            .cache\n            .iter()\n            .filter(|(_, mmap_weakref)| mmap_weakref.upgrade().is_none())\n            .map(|(key, _)| key.clone())\n            .collect();\n        for key in keys_to_remove {\n            self.cache.remove(&key);\n        }\n    }\n\n    fn open_mmap_impl(&self, full_path: &Path) -> Result<Option<Mmap>, OpenReadError> {\n        let mmap_opt = open_mmap(full_path)?;\n        #[cfg(unix)]\n        if let (Some(mmap), Some(madvice)) = (mmap_opt.as_ref(), self.madvice_opt) {\n            // We ignore madvise errors.\n            let _ = mmap.advise(madvice);\n        }\n        Ok(mmap_opt)\n    }\n\n    // Returns None if the file exists but as a len of 0 (and hence is not mmappable).\n    fn get_mmap(&mut self, full_path: &Path) -> Result<Option<ArcBytes>, OpenReadError> {\n        if let Some(mmap_weak) = self.cache.get(full_path) {\n            if let Some(mmap_arc) = mmap_weak.upgrade() {\n                self.counters.hit += 1;\n                return Ok(Some(mmap_arc));\n            }\n        }\n        self.cache.remove(full_path);\n        self.counters.miss += 1;\n        let mmap_opt = self.open_mmap_impl(full_path)?;\n        Ok(mmap_opt.map(|mmap| {\n            let mmap_arc: ArcBytes = Arc::new(mmap);\n            let mmap_weak = Arc::downgrade(&mmap_arc);\n            self.cache.insert(full_path.to_owned(), mmap_weak);\n            mmap_arc\n        }))\n    }\n}\n\n/// Directory storing data in files, read via mmap.\n///\n/// The Mmap object are cached to limit the\n/// system calls.\n///\n/// In the `MmapDirectory`, locks are implemented using the `fs2` crate definition of locks.\n///\n/// On MacOS & linux, it relies on `flock` (aka `BSD Lock`). These locks solve most of the\n/// problems related to POSIX Locks, but may their contract may not be respected on `NFS`\n/// depending on the implementation.\n///\n/// On Windows the semantics are again different.\n#[derive(Clone)]\npub struct MmapDirectory {\n    inner: Arc<MmapDirectoryInner>,\n}\n\nstruct MmapDirectoryInner {\n    root_path: PathBuf,\n    mmap_cache: RwLock<MmapCache>,\n    _temp_directory: Option<TempDir>,\n    watcher: FileWatcher,\n}\n\nimpl MmapDirectoryInner {\n    fn new(root_path: PathBuf, temp_directory: Option<TempDir>) -> MmapDirectoryInner {\n        MmapDirectoryInner {\n            mmap_cache: RwLock::new(MmapCache::new()),\n            _temp_directory: temp_directory,\n            watcher: FileWatcher::new(&root_path.join(*META_FILEPATH)),\n            root_path,\n        }\n    }\n\n    fn watch(&self, callback: WatchCallback) -> WatchHandle {\n        self.watcher.watch(callback)\n    }\n}\n\nimpl fmt::Debug for MmapDirectory {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"MmapDirectory({:?})\", self.inner.root_path)\n    }\n}\n\nimpl MmapDirectory {\n    fn new(root_path: PathBuf, temp_directory: Option<TempDir>) -> MmapDirectory {\n        let inner = MmapDirectoryInner::new(root_path, temp_directory);\n        MmapDirectory {\n            inner: Arc::new(inner),\n        }\n    }\n\n    /// Creates a new MmapDirectory in a temporary directory.\n    ///\n    /// This is mostly useful to test the MmapDirectory itself.\n    /// For your unit tests, prefer the RamDirectory.\n    pub fn create_from_tempdir() -> Result<MmapDirectory, OpenDirectoryError> {\n        let tempdir = TempDir::new()\n            .map_err(|io_err| OpenDirectoryError::FailedToCreateTempDir(Arc::new(io_err)))?;\n        Ok(MmapDirectory::new(\n            tempdir.path().to_path_buf(),\n            Some(tempdir),\n        ))\n    }\n\n    /// Opens a MmapDirectory in a directory, with a given access pattern.\n    ///\n    /// This is only supported on unix platforms.\n    #[cfg(unix)]\n    pub fn open_with_madvice(\n        directory_path: impl AsRef<Path>,\n        madvice: Advice,\n    ) -> Result<MmapDirectory, OpenDirectoryError> {\n        let dir = Self::open_impl_to_avoid_monomorphization(directory_path.as_ref())?;\n        dir.inner.mmap_cache.write().unwrap().set_advice(madvice);\n        Ok(dir)\n    }\n\n    /// Opens a MmapDirectory in a directory.\n    ///\n    /// Returns an error if the `directory_path` does not\n    /// exist or if it is not a directory.\n    pub fn open(directory_path: impl AsRef<Path>) -> Result<MmapDirectory, OpenDirectoryError> {\n        Self::open_impl_to_avoid_monomorphization(directory_path.as_ref())\n    }\n\n    #[inline(never)]\n    fn open_impl_to_avoid_monomorphization(\n        directory_path: &Path,\n    ) -> Result<MmapDirectory, OpenDirectoryError> {\n        if !directory_path.exists() {\n            return Err(OpenDirectoryError::DoesNotExist(PathBuf::from(\n                directory_path,\n            )));\n        }\n        #[expect(clippy::bind_instead_of_map)]\n        let canonical_path: PathBuf = directory_path.canonicalize().or_else(|io_err| {\n            let directory_path = directory_path.to_owned();\n\n            #[cfg(windows)]\n            {\n                // `canonicalize` returns \"Incorrect function\" (error code 1)\n                // for virtual drives (network drives, ramdisk, etc.).\n                if io_err.raw_os_error() == Some(1) && directory_path.exists() {\n                    // Should call `std::path::absolute` when it is stabilised.\n                    return Ok(directory_path);\n                }\n            }\n\n            Err(OpenDirectoryError::wrap_io_error(io_err, directory_path))\n        })?;\n        if !canonical_path.is_dir() {\n            return Err(OpenDirectoryError::NotADirectory(PathBuf::from(\n                directory_path,\n            )));\n        }\n        Ok(MmapDirectory::new(canonical_path, None))\n    }\n\n    /// Joins a relative_path to the directory `root_path`\n    /// to create a proper complete `filepath`.\n    fn resolve_path(&self, relative_path: &Path) -> PathBuf {\n        self.inner.root_path.join(relative_path)\n    }\n\n    /// Returns some statistical information\n    /// about the Mmap cache.\n    ///\n    /// The `MmapDirectory` embeds a `MmapDirectory`\n    /// to avoid multiplying the `mmap` system calls.\n    pub fn get_cache_info(&self) -> CacheInfo {\n        self.inner\n            .mmap_cache\n            .write()\n            .expect(\"mmap cache lock is poisoned\")\n            .remove_weak_ref();\n        self.inner\n            .mmap_cache\n            .read()\n            .expect(\"Mmap cache lock is poisoned.\")\n            .get_info()\n    }\n}\n\n/// We rely on fs2 for file locking. On Windows & MacOS this\n/// uses BSD locks (`flock`). The lock is actually released when\n/// the `File` object is dropped and its associated file descriptor\n/// is closed.\nstruct ReleaseLockFile {\n    _file: File,\n    path: PathBuf,\n}\n\nimpl Drop for ReleaseLockFile {\n    fn drop(&mut self) {\n        debug!(\"Releasing lock {:?}\", self.path);\n    }\n}\n\n/// This Write wraps a File, but has the specificity of\n/// call `sync_all` on flush.\nstruct SafeFileWriter(File);\n\nimpl SafeFileWriter {\n    fn new(file: File) -> SafeFileWriter {\n        SafeFileWriter(file)\n    }\n}\n\nimpl Write for SafeFileWriter {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        self.0.write(buf)\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        Ok(())\n    }\n}\n\nimpl TerminatingWrite for SafeFileWriter {\n    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {\n        self.0.flush()?;\n        self.0.sync_data()?;\n        Ok(())\n    }\n}\n\n#[derive(Clone)]\nstruct MmapArc(Arc<dyn Deref<Target = [u8]> + Send + Sync>);\n\nimpl Deref for MmapArc {\n    type Target = [u8];\n\n    fn deref(&self) -> &[u8] {\n        self.0.deref()\n    }\n}\nunsafe impl StableDeref for MmapArc {}\n\n/// Writes a file in an atomic manner.\npub(crate) fn atomic_write(path: &Path, content: &[u8]) -> io::Result<()> {\n    // We create the temporary file in the same directory as the target file.\n    // Indeed the canonical temp directory and the target file might sit in different\n    // filesystem, in which case the atomic write may actually not work.\n    let parent_path = path.parent().ok_or_else(|| {\n        io::Error::new(\n            io::ErrorKind::InvalidInput,\n            \"Path {:?} does not have parent directory.\",\n        )\n    })?;\n    let mut tempfile = tempfile::Builder::new().tempfile_in(parent_path)?;\n    tempfile.write_all(content)?;\n    tempfile.flush()?;\n    tempfile.as_file_mut().sync_data()?;\n    tempfile.into_temp_path().persist(path)?;\n    Ok(())\n}\n\nimpl Directory for MmapDirectory {\n    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {\n        debug!(\"Open Read {path:?}\");\n        let full_path = self.resolve_path(path);\n\n        let mut mmap_cache = self.inner.mmap_cache.write().map_err(|_| {\n            let msg = format!(\"Failed to acquired write lock on mmap cache while reading {path:?}\");\n            let io_err = make_io_err(msg);\n            OpenReadError::wrap_io_error(io_err, path.to_path_buf())\n        })?;\n\n        let owned_bytes = mmap_cache\n            .get_mmap(&full_path)?\n            .map(|mmap_arc| {\n                let mmap_arc_obj = MmapArc(mmap_arc);\n                OwnedBytes::new(mmap_arc_obj)\n            })\n            .unwrap_or_else(OwnedBytes::empty);\n\n        Ok(Arc::new(owned_bytes))\n    }\n\n    /// Any entry associated with the path in the mmap will be\n    /// removed before the file is deleted.\n    fn delete(&self, path: &Path) -> Result<(), DeleteError> {\n        let full_path = self.resolve_path(path);\n        fs::remove_file(full_path).map_err(|e| {\n            if e.kind() == io::ErrorKind::NotFound {\n                DeleteError::FileDoesNotExist(path.to_owned())\n            } else {\n                DeleteError::IoError {\n                    io_error: Arc::new(e),\n                    filepath: path.to_path_buf(),\n                }\n            }\n        })?;\n        Ok(())\n    }\n\n    fn exists(&self, path: &Path) -> Result<bool, OpenReadError> {\n        let full_path = self.resolve_path(path);\n        full_path\n            .try_exists()\n            .map_err(|io_err| OpenReadError::wrap_io_error(io_err, path.to_path_buf()))\n    }\n\n    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError> {\n        debug!(\"Open Write {path:?}\");\n        let full_path = self.resolve_path(path);\n\n        let open_res = OpenOptions::new()\n            .write(true)\n            .create_new(true)\n            .open(full_path);\n\n        let mut file = open_res.map_err(|io_err| {\n            if io_err.kind() == io::ErrorKind::AlreadyExists {\n                OpenWriteError::FileAlreadyExists(path.to_path_buf())\n            } else {\n                OpenWriteError::wrap_io_error(io_err, path.to_path_buf())\n            }\n        })?;\n\n        // making sure the file is created.\n        file.flush()\n            .map_err(|io_error| OpenWriteError::wrap_io_error(io_error, path.to_path_buf()))?;\n\n        // Note we actually do not sync the parent directory here.\n        //\n        // A newly created file, may, in some case, be created and even flushed to disk.\n        // and then lost...\n        //\n        // The file will only be durably written after we terminate AND\n        // sync_directory() is called.\n\n        let writer = SafeFileWriter::new(file);\n        Ok(BufWriter::new(Box::new(writer)))\n    }\n\n    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError> {\n        let full_path = self.resolve_path(path);\n        let mut buffer = Vec::new();\n        match File::open(full_path) {\n            Ok(mut file) => {\n                file.read_to_end(&mut buffer).map_err(|io_error| {\n                    OpenReadError::wrap_io_error(io_error, path.to_path_buf())\n                })?;\n                Ok(buffer)\n            }\n            Err(io_error) => {\n                if io_error.kind() == io::ErrorKind::NotFound {\n                    Err(OpenReadError::FileDoesNotExist(path.to_owned()))\n                } else {\n                    Err(OpenReadError::wrap_io_error(io_error, path.to_path_buf()))\n                }\n            }\n        }\n    }\n\n    fn atomic_write(&self, path: &Path, content: &[u8]) -> io::Result<()> {\n        debug!(\"Atomic Write {path:?}\");\n        let full_path = self.resolve_path(path);\n        atomic_write(&full_path, content)?;\n        Ok(())\n    }\n\n    fn acquire_lock(&self, lock: &Lock) -> Result<DirectoryLock, LockError> {\n        let full_path = self.resolve_path(&lock.filepath);\n        // We make sure that the file exists.\n        let file: File = OpenOptions::new()\n            .write(true)\n            .create(true) //< if the file does not exist yet, create it.\n            .truncate(false)\n            .open(full_path)\n            .map_err(LockError::wrap_io_error)?;\n        if lock.is_blocking {\n            file.lock_exclusive().map_err(LockError::wrap_io_error)?;\n        } else if !file.try_lock_exclusive().map_err(|_| LockError::LockBusy)? {\n            return Err(LockError::LockBusy);\n        }\n        // dropping the file handle will release the lock.\n        Ok(DirectoryLock::from(Box::new(ReleaseLockFile {\n            path: lock.filepath.clone(),\n            _file: file,\n        })))\n    }\n\n    fn watch(&self, watch_callback: WatchCallback) -> crate::Result<WatchHandle> {\n        Ok(self.inner.watch(watch_callback))\n    }\n\n    #[cfg(windows)]\n    fn sync_directory(&self) -> Result<(), io::Error> {\n        // On Windows, it is not necessary to fsync the parent directory to\n        // ensure that the directory entry containing the file has also reached\n        // disk, and calling sync_data on a handle to directory is a no-op on\n        // local disks, but will return an error on virtual drives.\n        Ok(())\n    }\n\n    #[cfg(not(windows))]\n    fn sync_directory(&self) -> Result<(), io::Error> {\n        let mut open_opts = OpenOptions::new();\n\n        // Linux needs read to be set, otherwise returns EINVAL\n        // write must not be set, or it fails with EISDIR\n        open_opts.read(true);\n\n        let fd = open_opts.open(&self.inner.root_path)?;\n        fd.sync_data()?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    // There are more tests in directory/mod.rs\n    // The following tests are specific to the MmapDirectory\n\n    use std::time::Duration;\n\n    use common::HasLen;\n\n    use super::*;\n    use crate::indexer::LogMergePolicy;\n    use crate::schema::{Schema, SchemaBuilder, TEXT};\n    use crate::{Index, IndexSettings, IndexWriter, ReloadPolicy};\n\n    #[test]\n    fn test_open_non_existent_path() {\n        assert!(MmapDirectory::open(PathBuf::from(\"./nowhere\")).is_err());\n    }\n\n    #[test]\n    fn test_open_empty() {\n        // empty file is actually an edge case because those\n        // cannot be mmapped.\n        //\n        // In that case the directory returns a SharedVecSlice.\n        let mmap_directory = MmapDirectory::create_from_tempdir().unwrap();\n        let path = PathBuf::from(\"test\");\n        {\n            let mut w = mmap_directory.open_write(&path).unwrap();\n            w.flush().unwrap();\n        }\n        let readonlymap = mmap_directory.open_read(&path).unwrap();\n        assert_eq!(readonlymap.len(), 0);\n    }\n\n    #[test]\n    fn test_cache() {\n        let content = b\"abc\";\n\n        // here we test if the cache releases\n        // mmaps correctly.\n        let mmap_directory = MmapDirectory::create_from_tempdir().unwrap();\n        let num_paths = 10;\n        let paths: Vec<PathBuf> = (0..num_paths)\n            .map(|i| PathBuf::from(&*format!(\"file_{i}\")))\n            .collect();\n        {\n            for path in &paths {\n                let mut w = mmap_directory.open_write(path).unwrap();\n                w.write_all(content).unwrap();\n                w.flush().unwrap();\n            }\n        }\n\n        let mut keep = vec![];\n        for (i, path) in paths.iter().enumerate() {\n            keep.push(mmap_directory.open_read(path).unwrap());\n            assert_eq!(mmap_directory.get_cache_info().mmapped.len(), i + 1);\n        }\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 0);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 10);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 10);\n        for path in paths.iter() {\n            let _r = mmap_directory.open_read(path).unwrap();\n            assert_eq!(mmap_directory.get_cache_info().mmapped.len(), num_paths);\n        }\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 10);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 10);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 10);\n\n        for path in paths.iter() {\n            let _r = mmap_directory.open_read(path).unwrap();\n            assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 10);\n        }\n\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 20);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 10);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 10);\n        drop(keep);\n        for path in paths.iter() {\n            let _r = mmap_directory.open_read(path).unwrap();\n            assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 1);\n        }\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 20);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 20);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 0);\n\n        for path in &paths {\n            mmap_directory.delete(path).unwrap();\n        }\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 20);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 20);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 0);\n        for path in paths.iter() {\n            assert!(mmap_directory.open_read(path).is_err());\n        }\n        assert_eq!(mmap_directory.get_cache_info().counters.hit, 20);\n        assert_eq!(mmap_directory.get_cache_info().counters.miss, 30);\n        assert_eq!(mmap_directory.get_cache_info().mmapped.len(), 0);\n    }\n\n    fn assert_eventually<P: Fn() -> Option<String>>(predicate: P) {\n        for _ in 0..30 {\n            if predicate().is_none() {\n                break;\n            }\n            std::thread::sleep(Duration::from_millis(200));\n        }\n        if let Some(error_msg) = predicate() {\n            panic!(\"{}\", error_msg);\n        }\n    }\n\n    #[test]\n    fn test_mmap_released() {\n        let mmap_directory = MmapDirectory::create_from_tempdir().unwrap();\n        let mut schema_builder: SchemaBuilder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n\n        {\n            let index =\n                Index::create(mmap_directory.clone(), schema, IndexSettings::default()).unwrap();\n\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            let mut log_merge_policy = LogMergePolicy::default();\n            log_merge_policy.set_min_num_segments(3);\n            index_writer.set_merge_policy(Box::new(log_merge_policy));\n            for _num_commits in 0..10 {\n                for _ in 0..10 {\n                    index_writer.add_document(doc!(text_field=>\"abc\")).unwrap();\n                }\n                index_writer.commit().unwrap();\n            }\n\n            let reader = index\n                .reader_builder()\n                .reload_policy(ReloadPolicy::Manual)\n                .try_into()\n                .unwrap();\n\n            for _ in 0..4 {\n                index_writer.add_document(doc!(text_field=>\"abc\")).unwrap();\n                index_writer.commit().unwrap();\n                reader.reload().unwrap();\n            }\n            index_writer.wait_merging_threads().unwrap();\n\n            reader.reload().unwrap();\n            let num_segments = reader.searcher().segment_readers().len();\n            assert!(num_segments <= 4);\n            let num_components_except_deletes_and_tempstore =\n                crate::index::SegmentComponent::iterator().len() - 1;\n            let max_num_mmapped = num_components_except_deletes_and_tempstore * num_segments;\n            assert_eventually(|| {\n                let num_mmapped = mmap_directory.get_cache_info().mmapped.len();\n                if num_mmapped > max_num_mmapped {\n                    Some(format!(\n                        \"Expected at most {max_num_mmapped} mmapped files, got {num_mmapped}\"\n                    ))\n                } else {\n                    None\n                }\n            });\n        }\n        // This test failed on CI. The last Mmap is dropped from the merging thread so there might\n        // be a race condition indeed.\n        assert_eventually(|| {\n            let num_mmapped = mmap_directory.get_cache_info().mmapped.len();\n            if num_mmapped > 0 {\n                Some(format!(\"Expected no mmapped files, got {num_mmapped}\"))\n            } else {\n                None\n            }\n        });\n    }\n}\n"
  },
  {
    "path": "src/directory/mod.rs",
    "content": "//! WORM (Write Once Read Many) directory abstraction.\n\n#[cfg(feature = \"mmap\")]\nmod mmap_directory;\n\nmod directory;\nmod directory_lock;\npub mod footer;\nmod managed_directory;\nmod ram_directory;\nmod watch_event_router;\n\n/// Errors specific to the directory module.\npub mod error;\n\nmod composite_file;\n\nuse std::io::BufWriter;\nuse std::path::PathBuf;\n\npub use common::file_slice::{FileHandle, FileSlice};\npub use common::{AntiCallToken, OwnedBytes, TerminatingWrite};\n\npub use self::composite_file::{CompositeFile, CompositeWrite};\npub use self::directory::{Directory, DirectoryClone, DirectoryLock};\npub use self::directory_lock::{Lock, INDEX_WRITER_LOCK, META_LOCK};\npub use self::ram_directory::RamDirectory;\npub use self::watch_event_router::{WatchCallback, WatchCallbackList, WatchHandle};\n\n/// Outcome of the Garbage collection\npub struct GarbageCollectionResult {\n    /// List of files that were deleted in this cycle\n    pub deleted_files: Vec<PathBuf>,\n    /// List of files that were schedule to be deleted in this cycle,\n    /// but deletion did not work. This typically happens on windows,\n    /// as deleting a memory mapped file is forbidden.\n    ///\n    /// If a searcher is still held, a file cannot be deleted.\n    /// This is not considered a bug, the file will simply be deleted\n    /// in the next GC.\n    pub failed_to_delete_files: Vec<PathBuf>,\n}\n\n#[cfg(all(feature = \"mmap\", unix))]\npub use memmap2::Advice;\n\npub use self::managed_directory::ManagedDirectory;\n#[cfg(feature = \"mmap\")]\npub use self::mmap_directory::MmapDirectory;\n\n/// Write object for Directory.\n///\n/// `WritePtr` are required to implement both Write\n/// and Seek.\npub type WritePtr = BufWriter<Box<dyn TerminatingWrite + Send + Sync>>;\n\n#[cfg(test)]\nmod tests;\n"
  },
  {
    "path": "src/directory/ram_directory.rs",
    "content": "use std::collections::HashMap;\nuse std::io::{self, BufWriter, Cursor, Write};\nuse std::path::{Path, PathBuf};\nuse std::sync::{Arc, RwLock};\nuse std::{fmt, result};\n\nuse common::HasLen;\n\nuse super::FileHandle;\nuse crate::core::META_FILEPATH;\nuse crate::directory::error::{DeleteError, OpenReadError, OpenWriteError};\nuse crate::directory::{\n    AntiCallToken, Directory, FileSlice, TerminatingWrite, WatchCallback, WatchCallbackList,\n    WatchHandle, WritePtr,\n};\n\n/// Writer associated with the [`RamDirectory`].\n///\n/// The Writer just writes a buffer.\nstruct VecWriter {\n    path: PathBuf,\n    shared_directory: RamDirectory,\n    data: Cursor<Vec<u8>>,\n    is_flushed: bool,\n}\n\nimpl VecWriter {\n    fn new(path_buf: PathBuf, shared_directory: RamDirectory) -> VecWriter {\n        VecWriter {\n            path: path_buf,\n            data: Cursor::new(Vec::new()),\n            shared_directory,\n            is_flushed: true,\n        }\n    }\n}\n\nimpl Drop for VecWriter {\n    fn drop(&mut self) {\n        if !self.is_flushed {\n            warn!(\n                \"You forgot to flush {:?} before its writer got Drop. Do not rely on drop. This \\\n                 also occurs when the indexer crashed, so you may want to check the logs for the \\\n                 root cause.\",\n                self.path\n            )\n        }\n    }\n}\n\nimpl Write for VecWriter {\n    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n        self.is_flushed = false;\n        self.data.write_all(buf)?;\n        Ok(buf.len())\n    }\n\n    fn flush(&mut self) -> io::Result<()> {\n        self.is_flushed = true;\n        let mut fs = self.shared_directory.fs.write().unwrap();\n        fs.write(self.path.clone(), self.data.get_ref());\n        Ok(())\n    }\n}\n\nimpl TerminatingWrite for VecWriter {\n    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {\n        self.flush()\n    }\n}\n\n#[derive(Default)]\nstruct InnerDirectory {\n    fs: HashMap<PathBuf, FileSlice>,\n    watch_router: WatchCallbackList,\n}\n\nimpl InnerDirectory {\n    fn write(&mut self, path: PathBuf, data: &[u8]) -> bool {\n        let data = FileSlice::from(data.to_vec());\n        self.fs.insert(path, data).is_some()\n    }\n\n    fn open_read(&self, path: &Path) -> Result<FileSlice, OpenReadError> {\n        self.fs\n            .get(path)\n            .ok_or_else(|| OpenReadError::FileDoesNotExist(PathBuf::from(path)))\n            .cloned()\n    }\n\n    fn delete(&mut self, path: &Path) -> result::Result<(), DeleteError> {\n        match self.fs.remove(path) {\n            Some(_) => Ok(()),\n            None => Err(DeleteError::FileDoesNotExist(PathBuf::from(path))),\n        }\n    }\n\n    fn exists(&self, path: &Path) -> bool {\n        self.fs.contains_key(path)\n    }\n\n    fn watch(&mut self, watch_handle: WatchCallback) -> WatchHandle {\n        self.watch_router.subscribe(watch_handle)\n    }\n\n    fn total_mem_usage(&self) -> usize {\n        self.fs.values().map(|f| f.len()).sum()\n    }\n}\n\nimpl fmt::Debug for RamDirectory {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"RamDirectory\")\n    }\n}\n\n/// A Directory storing everything in anonymous memory.\n///\n/// It is mainly meant for unit testing.\n/// Writes are only made visible upon flushing.\n#[derive(Clone, Default)]\npub struct RamDirectory {\n    fs: Arc<RwLock<InnerDirectory>>,\n}\n\nimpl RamDirectory {\n    /// Constructor\n    pub fn create() -> RamDirectory {\n        Self::default()\n    }\n\n    /// Deep clones the directory.\n    ///\n    /// Ulterior writes on one of the copy\n    /// will not affect the other copy.\n    pub fn deep_clone(&self) -> RamDirectory {\n        let inner_clone = InnerDirectory {\n            fs: self.fs.read().unwrap().fs.clone(),\n            watch_router: Default::default(),\n        };\n        RamDirectory {\n            fs: Arc::new(RwLock::new(inner_clone)),\n        }\n    }\n\n    /// Returns the sum of the size of the different files\n    /// in the [`RamDirectory`].\n    pub fn total_mem_usage(&self) -> usize {\n        self.fs.read().unwrap().total_mem_usage()\n    }\n\n    /// Write a copy of all of the files saved in the [`RamDirectory`] in the target [`Directory`].\n    ///\n    /// Files are all written using the [`Directory::open_write()`] meaning, even if they were\n    /// written using the [`Directory::atomic_write()`] api.\n    ///\n    /// If an error is encountered, files may be persisted partially.\n    pub fn persist(&self, dest: &dyn Directory) -> crate::Result<()> {\n        let wlock = self.fs.write().unwrap();\n        for (path, file) in wlock.fs.iter() {\n            let mut dest_wrt = dest.open_write(path)?;\n            dest_wrt.write_all(file.read_bytes()?.as_slice())?;\n            dest_wrt.terminate()?;\n        }\n        Ok(())\n    }\n}\n\nimpl Directory for RamDirectory {\n    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {\n        let file_slice = self.open_read(path)?;\n        Ok(Arc::new(file_slice))\n    }\n\n    fn open_read(&self, path: &Path) -> result::Result<FileSlice, OpenReadError> {\n        self.fs.read().unwrap().open_read(path)\n    }\n\n    fn delete(&self, path: &Path) -> result::Result<(), DeleteError> {\n        crate::fail_point!(\"RamDirectory::delete\", |_| {\n            Err(DeleteError::IoError {\n                io_error: Arc::new(io::Error::from(io::ErrorKind::Other)),\n                filepath: path.to_path_buf(),\n            })\n        });\n        self.fs.write().unwrap().delete(path)\n    }\n\n    fn exists(&self, path: &Path) -> Result<bool, OpenReadError> {\n        Ok(self\n            .fs\n            .read()\n            .map_err(|e| OpenReadError::IoError {\n                io_error: Arc::new(io::Error::other(e.to_string())),\n                filepath: path.to_path_buf(),\n            })?\n            .exists(path))\n    }\n\n    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError> {\n        let mut fs = self.fs.write().unwrap();\n        let path_buf = PathBuf::from(path);\n        let vec_writer = VecWriter::new(path_buf.clone(), self.clone());\n        let exists = fs.write(path_buf.clone(), &[]);\n        // force the creation of the file to mimic the MMap directory.\n        if exists {\n            Err(OpenWriteError::FileAlreadyExists(path_buf))\n        } else {\n            Ok(BufWriter::new(Box::new(vec_writer)))\n        }\n    }\n\n    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError> {\n        let bytes =\n            self.open_read(path)?\n                .read_bytes()\n                .map_err(|io_error| OpenReadError::IoError {\n                    io_error: Arc::new(io_error),\n                    filepath: path.to_path_buf(),\n                })?;\n        Ok(bytes.as_slice().to_owned())\n    }\n\n    fn atomic_write(&self, path: &Path, data: &[u8]) -> io::Result<()> {\n        let path_buf = PathBuf::from(path);\n        self.fs.write().unwrap().write(path_buf, data);\n        if path == *META_FILEPATH {\n            drop(self.fs.write().unwrap().watch_router.broadcast());\n        }\n        Ok(())\n    }\n\n    fn watch(&self, watch_callback: WatchCallback) -> crate::Result<WatchHandle> {\n        Ok(self.fs.write().unwrap().watch(watch_callback))\n    }\n\n    fn sync_directory(&self) -> io::Result<()> {\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io::Write;\n    use std::path::Path;\n\n    use super::RamDirectory;\n    use crate::Directory;\n\n    #[test]\n    fn test_persist() {\n        let msg_atomic: &'static [u8] = b\"atomic is the way\";\n        let msg_seq: &'static [u8] = b\"sequential is the way\";\n        let path_atomic: &'static Path = Path::new(\"atomic\");\n        let path_seq: &'static Path = Path::new(\"seq\");\n        let directory = RamDirectory::create();\n        assert!(directory.atomic_write(path_atomic, msg_atomic).is_ok());\n        let mut wrt = directory.open_write(path_seq).unwrap();\n        assert!(wrt.write_all(msg_seq).is_ok());\n        assert!(wrt.flush().is_ok());\n        let directory_copy = RamDirectory::create();\n        assert!(directory.persist(&directory_copy).is_ok());\n        assert_eq!(directory_copy.atomic_read(path_atomic).unwrap(), msg_atomic);\n        assert_eq!(directory_copy.atomic_read(path_seq).unwrap(), msg_seq);\n    }\n\n    #[test]\n    fn test_ram_directory_deep_clone() {\n        let dir = RamDirectory::default();\n        let test = Path::new(\"test\");\n        let test2 = Path::new(\"test2\");\n        dir.atomic_write(test, b\"firstwrite\").unwrap();\n        let dir_clone = dir.deep_clone();\n        assert_eq!(\n            dir_clone.atomic_read(test).unwrap(),\n            dir.atomic_read(test).unwrap()\n        );\n        dir.atomic_write(test, b\"original\").unwrap();\n        dir_clone.atomic_write(test, b\"clone\").unwrap();\n        dir_clone.atomic_write(test2, b\"clone2\").unwrap();\n        assert_eq!(dir.atomic_read(test).unwrap(), b\"original\");\n        assert_eq!(&dir_clone.atomic_read(test).unwrap(), b\"clone\");\n        assert_eq!(&dir_clone.atomic_read(test2).unwrap(), b\"clone2\");\n    }\n}\n"
  },
  {
    "path": "src/directory/tests.rs",
    "content": "use std::io::Write;\nuse std::mem;\nuse std::path::Path;\nuse std::sync::atomic::Ordering::SeqCst;\nuse std::sync::atomic::{AtomicBool, AtomicUsize};\nuse std::sync::Arc;\nuse std::time::Duration;\n\nuse super::*;\n\n#[cfg(feature = \"mmap\")]\nmod mmap_directory_tests {\n    use crate::directory::MmapDirectory;\n\n    type DirectoryImpl = MmapDirectory;\n\n    fn make_directory() -> DirectoryImpl {\n        MmapDirectory::create_from_tempdir().unwrap()\n    }\n\n    #[test]\n    fn test_simple() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_simple(&directory)\n    }\n\n    #[test]\n    fn test_write_create_the_file() {\n        let directory = make_directory();\n        super::test_write_create_the_file(&directory);\n    }\n\n    #[test]\n    fn test_rewrite_forbidden() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_rewrite_forbidden(&directory)?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_directory_delete() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_directory_delete(&directory)?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_lock_non_blocking() {\n        let directory = make_directory();\n        super::test_lock_non_blocking(&directory);\n    }\n\n    #[test]\n    fn test_lock_blocking() {\n        let directory = make_directory();\n        super::test_lock_blocking(&directory);\n    }\n\n    #[test]\n    fn test_watch() {\n        let directory = make_directory();\n        super::test_watch(&directory);\n    }\n}\n\nmod ram_directory_tests {\n    use crate::directory::RamDirectory;\n\n    type DirectoryImpl = RamDirectory;\n\n    fn make_directory() -> DirectoryImpl {\n        RamDirectory::default()\n    }\n\n    #[test]\n    fn test_simple() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_simple(&directory)\n    }\n\n    #[test]\n    fn test_write_create_the_file() {\n        let directory = make_directory();\n        super::test_write_create_the_file(&directory);\n    }\n\n    #[test]\n    fn test_rewrite_forbidden() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_rewrite_forbidden(&directory)?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_directory_delete() -> crate::Result<()> {\n        let directory = make_directory();\n        super::test_directory_delete(&directory)?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_lock_non_blocking() {\n        let directory = make_directory();\n        super::test_lock_non_blocking(&directory);\n    }\n\n    #[test]\n    fn test_lock_blocking() {\n        let directory = make_directory();\n        super::test_lock_blocking(&directory);\n    }\n\n    #[test]\n    fn test_watch() {\n        let directory = make_directory();\n        super::test_watch(&directory);\n    }\n}\n\nfn test_simple(directory: &dyn Directory) -> crate::Result<()> {\n    let test_path: &'static Path = Path::new(\"some_path_for_test\");\n    let mut write_file = directory.open_write(test_path)?;\n    assert!(directory.exists(test_path).unwrap());\n    write_file.write_all(&[4])?;\n    write_file.write_all(&[3])?;\n    write_file.write_all(&[7, 3, 5])?;\n    write_file.flush()?;\n    let read_file = directory.open_read(test_path)?.read_bytes()?;\n    assert_eq!(read_file.as_slice(), &[4u8, 3u8, 7u8, 3u8, 5u8]);\n    mem::drop(read_file);\n    assert!(directory.delete(test_path).is_ok());\n    assert!(!directory.exists(test_path).unwrap());\n    Ok(())\n}\n\nfn test_rewrite_forbidden(directory: &dyn Directory) -> crate::Result<()> {\n    let test_path: &'static Path = Path::new(\"some_path_for_test\");\n    directory.open_write(test_path)?;\n    assert!(directory.exists(test_path).unwrap());\n    assert!(directory.open_write(test_path).is_err());\n    assert!(directory.delete(test_path).is_ok());\n    Ok(())\n}\n\nfn test_write_create_the_file(directory: &dyn Directory) {\n    let test_path: &'static Path = Path::new(\"some_path_for_test\");\n    {\n        assert!(directory.open_read(test_path).is_err());\n        let _w = directory.open_write(test_path).unwrap();\n        assert!(directory.exists(test_path).unwrap());\n        assert!(directory.open_read(test_path).is_ok());\n        assert!(directory.delete(test_path).is_ok());\n    }\n}\n\nfn test_directory_delete(directory: &dyn Directory) -> crate::Result<()> {\n    let test_path: &'static Path = Path::new(\"some_path_for_test\");\n    assert!(directory.open_read(test_path).is_err());\n    let mut write_file = directory.open_write(test_path)?;\n    write_file.write_all(&[1, 2, 3, 4])?;\n    write_file.flush()?;\n    {\n        let read_handle = directory.open_read(test_path)?.read_bytes()?;\n        assert_eq!(read_handle.as_slice(), &[1u8, 2u8, 3u8, 4u8]);\n        // Mapped files can't be deleted on Windows\n        if !cfg!(windows) {\n            assert!(directory.delete(test_path).is_ok());\n            assert_eq!(read_handle.as_slice(), &[1u8, 2u8, 3u8, 4u8]);\n        }\n        assert!(directory.delete(Path::new(\"SomeOtherPath\")).is_err());\n    }\n\n    if cfg!(windows) {\n        assert!(directory.delete(test_path).is_ok());\n    }\n\n    assert!(directory.open_read(test_path).is_err());\n    assert!(directory.delete(test_path).is_err());\n    Ok(())\n}\n\nfn test_watch(directory: &dyn Directory) {\n    let counter: Arc<AtomicUsize> = Default::default();\n    let (tx, rx) = crossbeam_channel::unbounded();\n    let timeout = Duration::from_millis(500);\n\n    let handle = directory\n        .watch(WatchCallback::new(move || {\n            let val = counter.fetch_add(1, SeqCst);\n            tx.send(val + 1).unwrap();\n        }))\n        .unwrap();\n\n    assert!(directory\n        .atomic_write(Path::new(\"meta.json\"), b\"foo\")\n        .is_ok());\n    assert_eq!(rx.recv_timeout(timeout), Ok(1));\n\n    assert!(directory\n        .atomic_write(Path::new(\"meta.json\"), b\"bar\")\n        .is_ok());\n    assert_eq!(rx.recv_timeout(timeout), Ok(2));\n\n    mem::drop(handle);\n\n    assert!(directory\n        .atomic_write(Path::new(\"meta.json\"), b\"qux\")\n        .is_ok());\n    assert!(rx.recv_timeout(timeout).is_err());\n}\n\nfn test_lock_non_blocking(directory: &dyn Directory) {\n    {\n        let lock_a_res = directory.acquire_lock(&Lock {\n            filepath: PathBuf::from(\"a.lock\"),\n            is_blocking: false,\n        });\n        assert!(lock_a_res.is_ok());\n        let lock_b_res = directory.acquire_lock(&Lock {\n            filepath: PathBuf::from(\"b.lock\"),\n            is_blocking: false,\n        });\n        assert!(lock_b_res.is_ok());\n        let lock_a_res2 = directory.acquire_lock(&Lock {\n            filepath: PathBuf::from(\"a.lock\"),\n            is_blocking: false,\n        });\n        assert!(lock_a_res2.is_err());\n    }\n    let lock_a_res = directory.acquire_lock(&Lock {\n        filepath: PathBuf::from(\"a.lock\"),\n        is_blocking: false,\n    });\n    assert!(lock_a_res.is_ok());\n}\n\nfn test_lock_blocking(directory: &dyn Directory) {\n    let lock_a_res = directory.acquire_lock(&Lock {\n        filepath: PathBuf::from(\"a.lock\"),\n        is_blocking: true,\n    });\n    assert!(lock_a_res.is_ok());\n    let in_thread = Arc::new(AtomicBool::default());\n    let in_thread_clone = in_thread.clone();\n    let (sender, receiver) = oneshot::channel();\n    std::thread::spawn(move || {\n        //< lock_a_res is sent to the thread.\n        in_thread_clone.store(true, SeqCst);\n        let _just_sync = receiver.recv();\n        // explicitly dropping lock_a_res. It would have been sufficient to just force it\n        // to be part of the move, but the intent seems clearer that way.\n        drop(lock_a_res);\n    });\n    {\n        // A non-blocking call should fail, as the thread is running and holding the lock.\n        let lock_a_res = directory.acquire_lock(&Lock {\n            filepath: PathBuf::from(\"a.lock\"),\n            is_blocking: false,\n        });\n        assert!(lock_a_res.is_err());\n    }\n    let directory_clone = directory.box_clone();\n    let (sender2, receiver2) = oneshot::channel();\n    let join_handle = std::thread::spawn(move || {\n        assert!(sender2.send(()).is_ok());\n        let lock_a_res = directory_clone.acquire_lock(&Lock {\n            filepath: PathBuf::from(\"a.lock\"),\n            is_blocking: true,\n        });\n        assert!(in_thread.load(SeqCst));\n        assert!(lock_a_res.is_ok());\n    });\n    assert!(receiver2.recv().is_ok());\n    assert!(sender.send(()).is_ok());\n    assert!(join_handle.join().is_ok());\n}\n"
  },
  {
    "path": "src/directory/watch_event_router.rs",
    "content": "use std::sync::{Arc, RwLock, Weak};\n\nuse crate::FutureResult;\n\n/// Cloneable wrapper for callbacks registered when watching files of a `Directory`.\n#[derive(Clone)]\npub struct WatchCallback(Arc<dyn Fn() + Sync + Send>);\n\nimpl WatchCallback {\n    /// Wraps a `Fn()` to create a WatchCallback.\n    pub fn new<F: Fn() + Sync + Send + 'static>(op: F) -> Self {\n        WatchCallback(Arc::new(op))\n    }\n\n    fn call(&self) {\n        self.0()\n    }\n}\n\n/// Helper struct to implement the watch method in `Directory` implementations.\n///\n/// It registers callbacks (See `.subscribe(...)`) and\n/// calls them upon calls to `.broadcast(...)`.\n#[derive(Default)]\npub struct WatchCallbackList {\n    router: RwLock<Vec<Weak<WatchCallback>>>,\n}\n\n/// Controls how long a directory should watch for a file change.\n///\n/// After all the clones of `WatchHandle` are dropped, the associated will not be called when a\n/// file change is detected.\n#[must_use = \"This `WatchHandle` controls the lifetime of the watch and should therefore be used.\"]\n#[derive(Clone)]\n#[expect(dead_code)]\npub struct WatchHandle(Arc<WatchCallback>);\n\nimpl WatchHandle {\n    /// Create a WatchHandle handle.\n    pub fn new(watch_callback: Arc<WatchCallback>) -> WatchHandle {\n        WatchHandle(watch_callback)\n    }\n\n    /// Returns an empty watch handle.\n    ///\n    /// This function is only useful when implementing a readonly directory.\n    pub fn empty() -> WatchHandle {\n        WatchHandle::new(Arc::new(WatchCallback::new(|| {})))\n    }\n}\n\nimpl WatchCallbackList {\n    /// Subscribes a new callback and returns a handle that controls the lifetime of the callback.\n    pub fn subscribe(&self, watch_callback: WatchCallback) -> WatchHandle {\n        let watch_callback_arc = Arc::new(watch_callback);\n        let watch_callback_weak = Arc::downgrade(&watch_callback_arc);\n        self.router.write().unwrap().push(watch_callback_weak);\n        WatchHandle::new(watch_callback_arc)\n    }\n\n    fn list_callback(&self) -> Vec<WatchCallback> {\n        let mut callbacks: Vec<WatchCallback> = vec![];\n        let mut router_wlock = self.router.write().unwrap();\n        let mut i = 0;\n        while i < router_wlock.len() {\n            if let Some(watch) = router_wlock[i].upgrade() {\n                callbacks.push(watch.as_ref().clone());\n                i += 1;\n            } else {\n                router_wlock.swap_remove(i);\n            }\n        }\n        callbacks\n    }\n\n    /// Triggers all callbacks\n    pub fn broadcast(&self) -> FutureResult<()> {\n        let callbacks = self.list_callback();\n        let (result, sender) = FutureResult::create(\"One of the callback panicked.\");\n        if callbacks.is_empty() {\n            let _ = sender.send(Ok(()));\n            return result;\n        }\n        let spawn_res = std::thread::Builder::new()\n            .name(\"watch-callbacks\".to_string())\n            .spawn(move || {\n                for callback in callbacks {\n                    callback.call();\n                }\n                let _ = sender.send(Ok(()));\n            });\n        if let Err(err) = spawn_res {\n            error!(\"Failed to spawn thread to call watch callbacks. Cause: {err:?}\");\n        }\n        result\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::mem;\n    use std::sync::atomic::{AtomicUsize, Ordering};\n    use std::sync::Arc;\n\n    use crate::directory::{WatchCallback, WatchCallbackList};\n\n    #[test]\n    fn test_watch_event_router_simple() {\n        let watch_event_router = WatchCallbackList::default();\n        let counter: Arc<AtomicUsize> = Default::default();\n        let counter_clone = counter.clone();\n        let inc_callback = WatchCallback::new(move || {\n            counter_clone.fetch_add(1, Ordering::SeqCst);\n        });\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(0, counter.load(Ordering::SeqCst));\n        let handle_a = watch_event_router.subscribe(inc_callback);\n        assert_eq!(0, counter.load(Ordering::SeqCst));\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(1, counter.load(Ordering::SeqCst));\n        watch_event_router.broadcast().wait().unwrap();\n        watch_event_router.broadcast().wait().unwrap();\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(4, counter.load(Ordering::SeqCst));\n        mem::drop(handle_a);\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(4, counter.load(Ordering::SeqCst));\n    }\n\n    #[test]\n    fn test_watch_event_router_multiple_callback_same_key() {\n        let watch_event_router = WatchCallbackList::default();\n        let counter: Arc<AtomicUsize> = Default::default();\n        let inc_callback = |inc: usize| {\n            let counter_clone = counter.clone();\n            WatchCallback::new(move || {\n                counter_clone.fetch_add(inc, Ordering::SeqCst);\n            })\n        };\n        let handle_a = watch_event_router.subscribe(inc_callback(1));\n        let handle_a2 = watch_event_router.subscribe(inc_callback(10));\n        assert_eq!(0, counter.load(Ordering::SeqCst));\n        watch_event_router.broadcast().wait().unwrap();\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(22, counter.load(Ordering::SeqCst));\n        mem::drop(handle_a);\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(32, counter.load(Ordering::SeqCst));\n        mem::drop(handle_a2);\n        watch_event_router.broadcast().wait().unwrap();\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(32, counter.load(Ordering::SeqCst));\n    }\n\n    #[test]\n    fn test_watch_event_router_multiple_callback_different_key() {\n        let watch_event_router = WatchCallbackList::default();\n        let counter: Arc<AtomicUsize> = Default::default();\n        let counter_clone = counter.clone();\n        let inc_callback = WatchCallback::new(move || {\n            counter_clone.fetch_add(1, Ordering::SeqCst);\n        });\n        let handle_a = watch_event_router.subscribe(inc_callback);\n        assert_eq!(0, counter.load(Ordering::SeqCst));\n        watch_event_router.broadcast().wait().unwrap();\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(2, counter.load(Ordering::SeqCst));\n        mem::drop(handle_a);\n        drop(watch_event_router.broadcast());\n        watch_event_router.broadcast().wait().unwrap();\n        assert_eq!(2, counter.load(Ordering::SeqCst));\n    }\n}\n"
  },
  {
    "path": "src/docset.rs",
    "content": "use std::borrow::{Borrow, BorrowMut};\n\nuse crate::fastfield::AliveBitSet;\nuse crate::DocId;\n\n/// Sentinel value returned when a [`DocSet`] has been entirely consumed.\n///\n/// This is not `u32::MAX` as one would have expected, due to the lack of SSE2 instructions\n/// to compare `[u32; 4]`.\npub const TERMINATED: DocId = i32::MAX as u32;\n\n/// The collect_block method on `SegmentCollector` uses a buffer of this size.\n/// Passed results to `collect_block` will not exceed this size and will be\n/// exactly this size as long as we can fill the buffer.\npub const COLLECT_BLOCK_BUFFER_LEN: usize = 64;\n\n/// Represents an iterable set of sorted doc ids.\npub trait DocSet: Send {\n    /// Goes to the next element.\n    ///\n    /// The DocId of the next element is returned.\n    /// In other words we should always have :\n    /// ```compile_fail\n    /// let doc = docset.advance();\n    /// assert_eq!(doc, docset.doc());\n    /// ```\n    ///\n    /// If we reached the end of the `DocSet`, [`TERMINATED`] should be returned.\n    ///\n    /// Calling `.advance()` on a terminated `DocSet` should be supported, and [`TERMINATED`] should\n    /// be returned.\n    fn advance(&mut self) -> DocId;\n\n    /// Advances the `DocSet` forward until reaching the target, or going to the\n    /// lowest [`DocId`] greater than the target.\n    ///\n    /// If the end of the `DocSet` is reached, [`TERMINATED`] is returned.\n    ///\n    /// Calling `.seek(target)` on a terminated `DocSet` is legal. Implementation\n    /// of `DocSet` should support it.\n    ///\n    /// Calling `seek(TERMINATED)` is also legal and is the normal way to consume a `DocSet`.\n    ///\n    /// `target` has to be larger or equal to `.doc()` when calling `seek`.\n    fn seek(&mut self, target: DocId) -> DocId {\n        let mut doc = self.doc();\n        debug_assert!(doc <= target);\n        while doc < target {\n            doc = self.advance();\n        }\n        doc\n    }\n\n    /// !!!Dragons ahead!!!\n    /// In spirit, this is an approximate and dangerous version of `seek`.\n    ///\n    /// It can leave the DocSet in an `invalid` state and might return a\n    /// lower bound of what the result of Seek would have been.\n    ///\n    ///\n    /// More accurately it returns either:\n    /// - Found if the target is in the docset. In that case, the DocSet is left in a valid state.\n    /// - SeekLowerBound(seek_lower_bound) if the target is not in the docset. In that case, The\n    ///   DocSet can be the left in a invalid state. The DocSet should then only receives call to\n    ///   `seek_danger(..)` until it returns `Found`, and get back to a valid state.\n    ///\n    /// `seek_lower_bound` can be any `DocId` (in the docset or not) as long as it is in\n    /// `(target .. seek_result] U {TERMINATED}` where `seek_result` is the first document in the\n    /// docset greater than to `target`.\n    ///\n    /// `seek_danger` may return `SeekLowerBound(TERMINATED)`.\n    ///\n    /// Calling `seek_danger` with TERMINATED as a target is allowed,\n    /// and should always return NewTarget(TERMINATED) or anything larger as TERMINATED is NOT in\n    /// the DocSet.\n    ///\n    /// DocSets that already have an efficient `seek` method don't need to implement\n    /// `seek_danger`.\n    ///\n    /// Consecutive calls to seek_danger are guaranteed to have strictly increasing `target`\n    /// values.\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        if target >= TERMINATED {\n            debug_assert!(target == TERMINATED);\n            // No need to advance.\n            return SeekDangerResult::SeekLowerBound(target);\n        }\n\n        // The default implementation does not include any\n        // `danger zone` behavior.\n        //\n        // It does not leave the scorer in an invalid state.\n        // For this reason, we can safely call `self.doc()`.\n        let mut doc = self.doc();\n        if doc < target {\n            doc = self.seek(target);\n        }\n        if doc == target {\n            SeekDangerResult::Found\n        } else {\n            SeekDangerResult::SeekLowerBound(doc)\n        }\n    }\n\n    /// Fills a given mutable buffer with the next doc ids from the\n    /// `DocSet`\n    ///\n    /// If that many `DocId`s are available, the method should\n    /// fill the entire buffer and return the length of the buffer.\n    ///\n    /// If we reach the end of the `DocSet` before filling\n    /// it entirely, then the buffer is filled up to this point, and\n    /// return value is the number of elements that were filled.\n    ///\n    /// # Warning\n    ///\n    /// This method is only here for specific high-performance\n    /// use case where batching. The normal way to\n    /// go through the `DocId`'s is to call `.advance()`.\n    fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {\n        if self.doc() == TERMINATED {\n            return 0;\n        }\n        for (i, buffer_val) in buffer.iter_mut().enumerate() {\n            *buffer_val = self.doc();\n            if self.advance() == TERMINATED {\n                return i + 1;\n            }\n        }\n        buffer.len()\n    }\n\n    /// Returns the current document\n    /// Right after creating a new `DocSet`, the docset points to the first document.\n    ///\n    /// If the `DocSet` is empty, `.doc()` should return [`TERMINATED`].\n    fn doc(&self) -> DocId;\n\n    /// Returns a best-effort hint of the\n    /// length of the docset.\n    fn size_hint(&self) -> u32;\n\n    /// Returns a best-effort hint of the cost to consume the entire docset.\n    ///\n    /// Consuming means calling advance until [`TERMINATED`] is returned.\n    /// The cost should be relative to the cost of driving a Term query,\n    /// which would be the number of documents in the DocSet.\n    ///\n    /// By default this returns `size_hint()`.\n    ///\n    /// DocSets may have vastly different cost depending on their type,\n    /// e.g. an intersection with 10 hits is much cheaper than\n    /// a phrase search with 10 hits, since it needs to load positions.\n    ///\n    /// ### Future Work\n    /// We may want to differentiate `DocSet` costs more more granular, e.g.\n    /// creation_cost, advance_cost, seek_cost on to get a good estimation\n    /// what query types to choose.\n    fn cost(&self) -> u64 {\n        self.size_hint() as u64\n    }\n\n    /// Returns the number documents matching.\n    /// Calling this method consumes the `DocSet`.\n    fn count(&mut self, alive_bitset: &AliveBitSet) -> u32 {\n        let mut count = 0u32;\n        let mut doc = self.doc();\n        while doc != TERMINATED {\n            if alive_bitset.is_alive(doc) {\n                count += 1u32;\n            }\n            doc = self.advance();\n        }\n        count\n    }\n\n    /// Returns the count of documents, deleted or not.\n    /// Calling this method consumes the `DocSet`.\n    ///\n    /// Of course, the result is an upper bound of the result\n    /// given by `count()`.\n    fn count_including_deleted(&mut self) -> u32 {\n        let mut count = 0u32;\n        let mut doc = self.doc();\n        while doc != TERMINATED {\n            count += 1u32;\n            doc = self.advance();\n        }\n        count\n    }\n}\n\n#[derive(Clone, Copy, Debug, PartialEq, Eq)]\npub enum SeekDangerResult {\n    /// The target was found in the DocSet.\n    Found,\n    /// The target was not found in the DocSet.\n    /// We return a range in which the value could be.\n    /// The given target can be any DocId, that is <= than the first document\n    /// in the docset after the target.\n    SeekLowerBound(DocId),\n}\n\nimpl DocSet for &mut dyn DocSet {\n    fn advance(&mut self) -> u32 {\n        (**self).advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        (**self).seek(target)\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        (**self).seek_danger(target)\n    }\n\n    fn doc(&self) -> u32 {\n        (**self).doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        (**self).size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        (**self).cost()\n    }\n\n    fn count(&mut self, alive_bitset: &AliveBitSet) -> u32 {\n        (**self).count(alive_bitset)\n    }\n\n    fn count_including_deleted(&mut self) -> u32 {\n        (**self).count_including_deleted()\n    }\n}\n\nimpl<TDocSet: DocSet + ?Sized> DocSet for Box<TDocSet> {\n    fn advance(&mut self) -> DocId {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.seek(target)\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.seek_danger(target)\n    }\n\n    fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.fill_buffer(buffer)\n    }\n\n    fn doc(&self) -> DocId {\n        let unboxed: &TDocSet = self.borrow();\n        unboxed.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        let unboxed: &TDocSet = self.borrow();\n        unboxed.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        let unboxed: &TDocSet = self.borrow();\n        unboxed.cost()\n    }\n\n    fn count(&mut self, alive_bitset: &AliveBitSet) -> u32 {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.count(alive_bitset)\n    }\n\n    fn count_including_deleted(&mut self) -> u32 {\n        let unboxed: &mut TDocSet = self.borrow_mut();\n        unboxed.count_including_deleted()\n    }\n}\n"
  },
  {
    "path": "src/error.rs",
    "content": "//! Definition of Tantivy's errors and results.\n\nuse std::path::PathBuf;\nuse std::sync::{Arc, PoisonError};\nuse std::{fmt, io};\n\nuse thiserror::Error;\n\nuse crate::aggregation::AggregationError;\nuse crate::directory::error::{\n    Incompatibility, LockError, OpenDirectoryError, OpenReadError, OpenWriteError,\n};\nuse crate::fastfield::FastFieldNotAvailableError;\nuse crate::schema::document::DeserializeError;\nuse crate::{query, schema};\n\n/// Represents a `DataCorruption` error.\n///\n/// When facing data corruption, tantivy actually panics or returns this error.\n#[derive(Clone)]\npub struct DataCorruption {\n    filepath: Option<PathBuf>,\n    comment: String,\n}\n\nimpl DataCorruption {\n    /// Creates a `DataCorruption` Error.\n    pub fn new(filepath: PathBuf, comment: String) -> DataCorruption {\n        DataCorruption {\n            filepath: Some(filepath),\n            comment,\n        }\n    }\n\n    /// Creates a `DataCorruption` Error, when the filepath is irrelevant.\n    pub fn comment_only<TStr: ToString>(comment: TStr) -> DataCorruption {\n        DataCorruption {\n            filepath: None,\n            comment: comment.to_string(),\n        }\n    }\n}\n\nimpl fmt::Debug for DataCorruption {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        write!(f, \"Data corruption\")?;\n        if let Some(ref filepath) = &self.filepath {\n            write!(f, \" (in file `{filepath:?}`)\")?;\n        }\n        write!(f, \": {}.\", self.comment)?;\n        Ok(())\n    }\n}\n\n/// The library's error enum\n#[derive(Debug, Clone, Error)]\npub enum TantivyError {\n    /// Error when handling aggregations.\n    #[error(transparent)]\n    AggregationError(#[from] AggregationError),\n    /// Failed to open the directory.\n    #[error(\"Failed to open the directory: '{0:?}'\")]\n    OpenDirectoryError(#[from] OpenDirectoryError),\n    /// Failed to open a file for read.\n    #[error(\"Failed to open file for read: '{0:?}'\")]\n    OpenReadError(#[from] OpenReadError),\n    /// Failed to open a file for write.\n    #[error(\"Failed to open file for write: '{0:?}'\")]\n    OpenWriteError(#[from] OpenWriteError),\n    /// Index already exists in this directory.\n    #[error(\"Index already exists\")]\n    IndexAlreadyExists,\n    /// Failed to acquire file lock.\n    #[error(\"Failed to acquire Lockfile: {0:?}. {1:?}\")]\n    LockFailure(LockError, Option<String>),\n    /// IO Error.\n    #[error(\"An IO error occurred: '{0}'\")]\n    IoError(Arc<io::Error>),\n    /// Data corruption.\n    #[error(\"Data corrupted: '{0:?}'\")]\n    DataCorruption(DataCorruption),\n    /// A thread holding the locked panicked and poisoned the lock.\n    #[error(\"A thread holding the locked panicked and poisoned the lock\")]\n    Poisoned,\n    /// The provided field name does not exist.\n    #[error(\"The field does not exist: '{0}'\")]\n    FieldNotFound(String),\n    /// Invalid argument was passed by the user.\n    #[error(\"An invalid argument was passed: '{0}'\")]\n    InvalidArgument(String),\n    /// An Error occurred in one of the threads.\n    #[error(\"An error occurred in a thread: '{0}'\")]\n    ErrorInThread(String),\n    /// An Error occurred related to opening or creating a index.\n    #[error(\"Missing required index builder argument when open/create index: '{0}'\")]\n    IndexBuilderMissingArgument(&'static str),\n    /// An Error occurred related to the schema.\n    #[error(\"Schema error: '{0}'\")]\n    SchemaError(String),\n    /// System error. (e.g.: We failed spawning a new thread).\n    #[error(\"System error.'{0}'\")]\n    SystemError(String),\n    /// Index incompatible with current version of Tantivy.\n    #[error(\"{0:?}\")]\n    IncompatibleIndex(Incompatibility),\n    /// An internal error occurred. This is are internal states that should not be reached.\n    /// e.g. a datastructure is incorrectly initialized.\n    #[error(\"Internal error: '{0}'\")]\n    InternalError(String),\n    #[error(\"Deserialize error: {0}\")]\n    /// An error occurred while attempting to deserialize a document.\n    DeserializeError(DeserializeError),\n}\n\nimpl From<io::Error> for TantivyError {\n    fn from(io_err: io::Error) -> TantivyError {\n        TantivyError::IoError(Arc::new(io_err))\n    }\n}\nimpl From<DataCorruption> for TantivyError {\n    fn from(data_corruption: DataCorruption) -> TantivyError {\n        TantivyError::DataCorruption(data_corruption)\n    }\n}\nimpl From<FastFieldNotAvailableError> for TantivyError {\n    fn from(fastfield_error: FastFieldNotAvailableError) -> TantivyError {\n        TantivyError::SchemaError(format!(\"{fastfield_error}\"))\n    }\n}\nimpl From<LockError> for TantivyError {\n    fn from(lock_error: LockError) -> TantivyError {\n        TantivyError::LockFailure(lock_error, None)\n    }\n}\n\nimpl From<query::QueryParserError> for TantivyError {\n    fn from(parsing_error: query::QueryParserError) -> TantivyError {\n        TantivyError::InvalidArgument(format!(\"Query is invalid. {parsing_error:?}\"))\n    }\n}\n\nimpl<Guard> From<PoisonError<Guard>> for TantivyError {\n    fn from(_: PoisonError<Guard>) -> TantivyError {\n        TantivyError::Poisoned\n    }\n}\n\nimpl From<time::error::Format> for TantivyError {\n    fn from(err: time::error::Format) -> TantivyError {\n        TantivyError::InvalidArgument(format!(\"Date formatting error: {err}\"))\n    }\n}\n\nimpl From<time::error::Parse> for TantivyError {\n    fn from(err: time::error::Parse) -> TantivyError {\n        TantivyError::InvalidArgument(format!(\"Date parsing error: {err}\"))\n    }\n}\n\nimpl From<time::error::ComponentRange> for TantivyError {\n    fn from(err: time::error::ComponentRange) -> TantivyError {\n        TantivyError::InvalidArgument(format!(\"Date range error: {err}\"))\n    }\n}\n\nimpl From<schema::DocParsingError> for TantivyError {\n    fn from(error: schema::DocParsingError) -> TantivyError {\n        TantivyError::InvalidArgument(format!(\"Failed to parse document {error:?}\"))\n    }\n}\n\nimpl From<serde_json::Error> for TantivyError {\n    fn from(error: serde_json::Error) -> TantivyError {\n        TantivyError::IoError(Arc::new(error.into()))\n    }\n}\n\nimpl From<rayon::ThreadPoolBuildError> for TantivyError {\n    fn from(error: rayon::ThreadPoolBuildError) -> TantivyError {\n        TantivyError::SystemError(error.to_string())\n    }\n}\n\nimpl From<DeserializeError> for TantivyError {\n    fn from(error: DeserializeError) -> TantivyError {\n        TantivyError::DeserializeError(error)\n    }\n}\n"
  },
  {
    "path": "src/fastfield/alive_bitset.rs",
    "content": "use std::io;\nuse std::io::Write;\n\nuse common::{intersect_bitsets, BitSet, ByteCount, OwnedBytes, ReadOnlyBitSet};\n\nuse crate::DocId;\n\n/// Write an alive `BitSet`\n///\n/// where `alive_bitset` is the set of alive `DocId`.\n/// Warning: this function does not call terminate. The caller is in charge of\n/// closing the writer properly.\npub fn write_alive_bitset<T: Write>(alive_bitset: &BitSet, writer: &mut T) -> io::Result<()> {\n    alive_bitset.serialize(writer)?;\n    Ok(())\n}\n\n/// Set of alive `DocId`s.\n#[derive(Clone)]\npub struct AliveBitSet {\n    num_alive_docs: usize,\n    bitset: ReadOnlyBitSet,\n}\n\n/// Intersects two AliveBitSets in a new one.\n/// The two bitsets need to have the same max_value.\npub fn intersect_alive_bitsets(left: AliveBitSet, right: AliveBitSet) -> AliveBitSet {\n    assert_eq!(left.bitset().max_value(), right.bitset().max_value());\n    let bitset = intersect_bitsets(left.bitset(), right.bitset());\n    let num_alive_docs = bitset.len();\n    AliveBitSet {\n        num_alive_docs,\n        bitset,\n    }\n}\n\nimpl AliveBitSet {\n    #[cfg(test)]\n    pub(crate) fn for_test_from_deleted_docs(deleted_docs: &[DocId], max_doc: u32) -> AliveBitSet {\n        assert!(deleted_docs.iter().all(|&doc| doc < max_doc));\n        let mut bitset = BitSet::with_max_value_and_full(max_doc);\n        for &doc in deleted_docs {\n            bitset.remove(doc);\n        }\n        let mut alive_bitset_buffer = Vec::new();\n        write_alive_bitset(&bitset, &mut alive_bitset_buffer).unwrap();\n        let alive_bitset_bytes = OwnedBytes::new(alive_bitset_buffer);\n        Self::open(alive_bitset_bytes)\n    }\n\n    /// Opens an alive bitset given its file.\n    pub fn open(bytes: OwnedBytes) -> AliveBitSet {\n        let bitset = ReadOnlyBitSet::open(bytes);\n        AliveBitSet::from(bitset)\n    }\n\n    /// Returns true if the document is still \"alive\". In other words, if it has not been deleted.\n    #[inline]\n    pub fn is_alive(&self, doc: DocId) -> bool {\n        self.bitset.contains(doc)\n    }\n\n    /// Returns true if the document has been marked as deleted.\n    #[inline]\n    pub fn is_deleted(&self, doc: DocId) -> bool {\n        !self.is_alive(doc)\n    }\n\n    /// Iterate over the alive doc_ids.\n    #[inline]\n    pub fn iter_alive(&self) -> impl Iterator<Item = DocId> + '_ {\n        self.bitset.iter()\n    }\n\n    /// Get underlying bitset.\n    #[inline]\n    pub fn bitset(&self) -> &ReadOnlyBitSet {\n        &self.bitset\n    }\n\n    /// The number of alive documents.\n    pub fn num_alive_docs(&self) -> usize {\n        self.num_alive_docs\n    }\n\n    /// Summarize total space usage of this bitset.\n    pub fn space_usage(&self) -> ByteCount {\n        self.bitset().num_bytes()\n    }\n}\n\nimpl From<ReadOnlyBitSet> for AliveBitSet {\n    fn from(bitset: ReadOnlyBitSet) -> AliveBitSet {\n        let num_alive_docs = bitset.len();\n        AliveBitSet {\n            num_alive_docs,\n            bitset,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::AliveBitSet;\n\n    #[test]\n    fn test_alive_bitset_empty() {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[], 10);\n        for doc in 0..10 {\n            assert_eq!(alive_bitset.is_deleted(doc), !alive_bitset.is_alive(doc));\n            assert!(!alive_bitset.is_deleted(doc));\n        }\n        assert_eq!(alive_bitset.num_alive_docs(), 10);\n    }\n\n    #[test]\n    fn test_alive_bitset() {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[1, 9], 10);\n        assert!(alive_bitset.is_alive(0));\n        assert!(alive_bitset.is_deleted(1));\n        assert!(alive_bitset.is_alive(2));\n        assert!(alive_bitset.is_alive(3));\n        assert!(alive_bitset.is_alive(4));\n        assert!(alive_bitset.is_alive(5));\n        assert!(alive_bitset.is_alive(6));\n        assert!(alive_bitset.is_alive(6));\n        assert!(alive_bitset.is_alive(7));\n        assert!(alive_bitset.is_alive(8));\n        assert!(alive_bitset.is_deleted(9));\n        for doc in 0..10 {\n            assert_eq!(alive_bitset.is_deleted(doc), !alive_bitset.is_alive(doc));\n        }\n        assert_eq!(alive_bitset.num_alive_docs(), 8);\n    }\n\n    #[test]\n    fn test_alive_bitset_iter_minimal() {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[7], 8);\n\n        let data: Vec<_> = alive_bitset.iter_alive().collect();\n        assert_eq!(data, vec![0, 1, 2, 3, 4, 5, 6]);\n    }\n\n    #[test]\n    fn test_alive_bitset_iter_small() {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[0, 2, 3, 6], 7);\n\n        let data: Vec<_> = alive_bitset.iter_alive().collect();\n        assert_eq!(data, vec![1, 4, 5]);\n    }\n    #[test]\n    fn test_alive_bitset_iter() {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[0, 1, 1000], 1001);\n\n        let data: Vec<_> = alive_bitset.iter_alive().collect();\n        assert_eq!(data, (2..=999).collect::<Vec<_>>());\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use rand::prelude::IteratorRandom;\n    use rand::rng;\n    use test::Bencher;\n\n    use super::AliveBitSet;\n\n    fn get_alive() -> Vec<u32> {\n        let mut data = (0..1_000_000_u32).collect::<Vec<u32>>();\n        for _ in 0..1_000_000 / 8 {\n            remove_rand(&mut data);\n        }\n        data\n    }\n\n    fn remove_rand(raw: &mut Vec<u32>) {\n        let i = (0..raw.len()).choose(&mut rng()).unwrap();\n        raw.remove(i);\n    }\n\n    #[bench]\n    fn bench_alive_bitset_iter_deser_on_fly(bench: &mut Bencher) {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[0, 1, 1000, 10000], 1_000_000);\n\n        bench.iter(|| alive_bitset.iter_alive().collect::<Vec<_>>());\n    }\n\n    #[bench]\n    fn bench_alive_bitset_access(bench: &mut Bencher) {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[0, 1, 1000, 10000], 1_000_000);\n\n        bench.iter(|| {\n            (0..1_000_000_u32)\n                .filter(|doc| alive_bitset.is_alive(*doc))\n                .collect::<Vec<_>>()\n        });\n    }\n\n    #[bench]\n    fn bench_alive_bitset_iter_deser_on_fly_1_8_alive(bench: &mut Bencher) {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&get_alive(), 1_000_000);\n\n        bench.iter(|| alive_bitset.iter_alive().collect::<Vec<_>>());\n    }\n\n    #[bench]\n    fn bench_alive_bitset_access_1_8_alive(bench: &mut Bencher) {\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&get_alive(), 1_000_000);\n\n        bench.iter(|| {\n            (0..1_000_000_u32)\n                .filter(|doc| alive_bitset.is_alive(*doc))\n                .collect::<Vec<_>>()\n        });\n    }\n}\n"
  },
  {
    "path": "src/fastfield/error.rs",
    "content": "use std::result;\n\nuse crate::schema::FieldEntry;\n\n/// `FastFieldNotAvailableError` is returned when the\n/// user requested for a fast field reader, and the field was not\n/// defined in the schema as a fast field.\n#[derive(Debug, Error)]\n#[error(\"Fast field not available: '{field_name:?}'\")]\npub struct FastFieldNotAvailableError {\n    pub(crate) field_name: String,\n}\n\nimpl FastFieldNotAvailableError {\n    /// Creates a `FastFieldNotAvailable` error.\n    /// `field_entry` is the configuration of the field\n    /// for which fast fields are not available.\n    pub fn new(field_entry: &FieldEntry) -> FastFieldNotAvailableError {\n        FastFieldNotAvailableError {\n            field_name: field_entry.name().to_string(),\n        }\n    }\n}\n\n/// Result when trying to access a fast field reader.\npub type Result<R> = result::Result<R, FastFieldNotAvailableError>;\n"
  },
  {
    "path": "src/fastfield/facet_reader.rs",
    "content": "use columnar::StrColumn;\n\nuse crate::schema::Facet;\nuse crate::termdict::TermOrdinal;\nuse crate::DocId;\n\n/// The facet reader makes it possible to access the list of\n/// facets associated with a given document in a specific\n/// segment.\n///\n/// Rather than manipulating `Facet` object directly, the API\n/// exposes those in the form of list of `Facet` ordinal.\n///\n/// A segment ordinal can then be translated into a facet via\n/// `.facet_from_ord(...)`.\n///\n/// Facet ordinals are defined as their position in the sorted\n/// list of facets. This ordinal is segment local and\n/// only makes sense for a given segment.\npub struct FacetReader {\n    facet_column: StrColumn,\n}\n\nimpl FacetReader {\n    /// Creates a new `FacetReader`.\n    ///\n    /// A facet reader just wraps :\n    /// - a `MultiValuedFastFieldReader` that makes it possible to access the list of facet ords for\n    ///   a given document.\n    /// - a `TermDictionary` that helps associating a facet to an ordinal and vice versa.\n    pub fn new(facet_column: StrColumn) -> FacetReader {\n        FacetReader { facet_column }\n    }\n\n    /// Returns the size of the sets of facets in the segment.\n    /// This does not take in account the documents that may be marked\n    /// as deleted.\n    ///\n    /// `Facet` ordinals range from `0` to `num_facets() - 1`.\n    pub fn num_facets(&self) -> usize {\n        self.facet_column.num_terms()\n    }\n\n    /// Given a term ordinal returns the term associated with it.\n    pub fn facet_from_ord(&self, facet_ord: TermOrdinal, output: &mut Facet) -> crate::Result<()> {\n        let found_term = self.facet_column.ord_to_str(facet_ord, &mut output.0)?;\n        assert!(found_term, \"Term ordinal {facet_ord} no found.\");\n        Ok(())\n    }\n\n    /// Return the list of facet ordinals associated with a document.\n    pub fn facet_ords(&self, doc: DocId) -> impl Iterator<Item = u64> + '_ {\n        self.facet_column.ords().values_for_doc(doc)\n    }\n\n    /// Accessor to the facet dictionary.\n    pub fn facet_dict(&self) -> &columnar::Dictionary {\n        self.facet_column.dictionary()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::schema::{Facet, FacetOptions, SchemaBuilder, Value, STORED};\n    use crate::{DocAddress, Index, IndexWriter, TantivyDocument};\n\n    #[test]\n    fn test_facet_only_indexed() {\n        let mut schema_builder = SchemaBuilder::default();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from_text(\"/a/b\").unwrap()))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let facet_reader = searcher.segment_reader(0u32).facet_reader(\"facet\").unwrap();\n        let mut facet_ords = Vec::new();\n        facet_ords.extend(facet_reader.facet_ords(0u32));\n        assert_eq!(&facet_ords, &[0u64]);\n        assert_eq!(facet_reader.num_facets(), 1);\n        let mut facet = Facet::default();\n        facet_reader.facet_from_ord(0, &mut facet).unwrap();\n        assert_eq!(facet.to_path_string(), \"/a/b\");\n        let doc = searcher\n            .doc::<TantivyDocument>(DocAddress::new(0u32, 0u32))\n            .unwrap();\n        let value = doc\n            .get_first(facet_field)\n            .and_then(|v| v.as_value().as_facet());\n        assert_eq!(value, None);\n    }\n\n    #[test]\n    fn test_facet_several_facets_sorted() {\n        let mut schema_builder = SchemaBuilder::default();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(facet_field=>Facet::from_text(\"/parent/child1\").unwrap()))\n            .unwrap();\n        index_writer\n            .add_document(doc!(\n                facet_field=>Facet::from_text(\"/parent/child2\").unwrap(),\n                facet_field=>Facet::from_text(\"/parent/child1/blop\").unwrap(),\n            ))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let facet_reader = searcher.segment_reader(0u32).facet_reader(\"facet\").unwrap();\n        let mut facet_ords = Vec::new();\n\n        facet_ords.extend(facet_reader.facet_ords(0u32));\n        assert_eq!(&facet_ords, &[0u64]);\n\n        facet_ords.clear();\n        facet_ords.extend(facet_reader.facet_ords(1u32));\n        assert_eq!(&facet_ords, &[1u64, 2u64]);\n\n        assert_eq!(facet_reader.num_facets(), 3);\n        let mut facet = Facet::default();\n        facet_reader.facet_from_ord(0, &mut facet).unwrap();\n        assert_eq!(facet.to_path_string(), \"/parent/child1\");\n        facet_reader.facet_from_ord(1, &mut facet).unwrap();\n        assert_eq!(facet.to_path_string(), \"/parent/child1/blop\");\n        facet_reader.facet_from_ord(2, &mut facet).unwrap();\n        assert_eq!(facet.to_path_string(), \"/parent/child2\");\n    }\n\n    #[test]\n    fn test_facet_stored_and_indexed() -> crate::Result<()> {\n        let mut schema_builder = SchemaBuilder::default();\n        let facet_field = schema_builder.add_facet_field(\"facet\", STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(facet_field=>Facet::from_text(\"/a/b\").unwrap()))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let facet_reader = searcher.segment_reader(0u32).facet_reader(\"facet\").unwrap();\n        let mut facet_ords = Vec::new();\n        facet_ords.extend(facet_reader.facet_ords(0u32));\n        assert_eq!(&facet_ords, &[0u64]);\n        let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0u32, 0u32))?;\n        let value: Option<Facet> = doc\n            .get_first(facet_field)\n            .and_then(|v| v.as_facet())\n            .map(|facet| Facet::from_encoded_string(facet.to_string()));\n        assert_eq!(value, Facet::from_text(\"/a/b\").ok());\n        Ok(())\n    }\n\n    #[test]\n    fn test_facet_not_populated_for_all_docs() -> crate::Result<()> {\n        let mut schema_builder = SchemaBuilder::default();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(facet_field=>Facet::from_text(\"/a/b\").unwrap()))?;\n        index_writer.add_document(TantivyDocument::default())?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let facet_reader = searcher.segment_reader(0u32).facet_reader(\"facet\").unwrap();\n        let mut facet_ords = Vec::new();\n        facet_ords.extend(facet_reader.facet_ords(0u32));\n        assert_eq!(&facet_ords, &[0u64]);\n        facet_ords.clear();\n        facet_ords.extend(facet_reader.facet_ords(1u32));\n        assert!(facet_ords.is_empty());\n        Ok(())\n    }\n\n    #[test]\n    fn test_facet_not_populated_for_any_docs() -> crate::Result<()> {\n        let mut schema_builder = SchemaBuilder::default();\n        schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(TantivyDocument::default())?;\n        index_writer.add_document(TantivyDocument::default())?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let facet_reader = searcher.segment_reader(0u32).facet_reader(\"facet\").unwrap();\n        assert!(facet_reader.facet_ords(0u32).next().is_none());\n        assert!(facet_reader.facet_ords(1u32).next().is_none());\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/fastfield/mod.rs",
    "content": "//! Column oriented field storage for tantivy.\n//!\n//! It is the equivalent of `Lucene`'s `DocValues`.\n//!\n//! A fast field is a column-oriented fashion storage for `tantivy`.\n//!\n//! It is designed for the fast random access of some document\n//! fields given a document id.\n//!\n//! Fast fields are useful when a field is required for all or most of\n//! the `DocSet`: for instance for scoring, grouping, aggregation, filtering, or faceting.\n//!\n//!\n//! Fields have to be declared as `FAST` in the schema.\n//! Currently supported fields are: u64, i64, f64, bytes, ip and text.\n//!\n//! Fast fields are stored in with [different codecs](columnar). The best codec is detected\n//! automatically, when serializing.\n//!\n//! Read access performance is comparable to that of an array lookup.\n\npub use columnar::Column;\nuse columnar::MonotonicallyMappableToU64;\n\npub use self::alive_bitset::{intersect_alive_bitsets, write_alive_bitset, AliveBitSet};\npub use self::error::{FastFieldNotAvailableError, Result};\npub use self::facet_reader::FacetReader;\npub use self::readers::FastFieldReaders;\npub use self::writer::FastFieldsWriter;\nuse crate::schema::Type;\nuse crate::DateTime;\n\nmod alive_bitset;\nmod error;\nmod facet_reader;\nmod readers;\nmod writer;\n\n/// Trait for types that are allowed for fast fields:\n/// (u64, i64 and f64, bool, DateTime).\npub trait FastValue: MonotonicallyMappableToU64 {\n    /// Returns the `schema::Type` for this FastValue.\n    fn to_type() -> Type;\n}\n\nimpl FastValue for u64 {\n    fn to_type() -> Type {\n        Type::U64\n    }\n}\n\nimpl FastValue for i64 {\n    fn to_type() -> Type {\n        Type::I64\n    }\n}\n\nimpl FastValue for f64 {\n    fn to_type() -> Type {\n        Type::F64\n    }\n}\n\nimpl FastValue for bool {\n    fn to_type() -> Type {\n        Type::Bool\n    }\n}\nimpl FastValue for DateTime {\n    fn to_type() -> Type {\n        Type::Date\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::net::Ipv6Addr;\n    use std::ops::{Range, RangeInclusive};\n    use std::path::Path;\n\n    use columnar::StrColumn;\n    use common::{ByteCount, DateTimePrecision, HasLen, TerminatingWrite};\n    use once_cell::sync::Lazy;\n    use rand::prelude::SliceRandom;\n    use rand::rngs::StdRng;\n    use rand::{Rng, SeedableRng};\n\n    use super::*;\n    use crate::directory::{Directory, RamDirectory, WritePtr};\n    use crate::index::SegmentId;\n    use crate::merge_policy::NoMergePolicy;\n    use crate::schema::{\n        DateOptions, Facet, FacetOptions, Field, JsonObjectOptions, Schema, SchemaBuilder,\n        TantivyDocument, TextOptions, FAST, INDEXED, STORED, STRING, TEXT,\n    };\n    use crate::time::OffsetDateTime;\n    use crate::tokenizer::{LowerCaser, RawTokenizer, TextAnalyzer, TokenizerManager};\n    use crate::{Index, IndexWriter, SegmentReader};\n\n    pub static SCHEMA: Lazy<Schema> = Lazy::new(|| {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_u64_field(\"field\", FAST);\n        schema_builder.build()\n    });\n    pub static FIELD: Lazy<Field> = Lazy::new(|| SCHEMA.get_field(\"field\").unwrap());\n\n    #[test]\n    pub fn test_convert_i64_u64() {\n        let datetime = DateTime::from_utc(OffsetDateTime::UNIX_EPOCH);\n        assert_eq!(i64::from_u64(datetime.to_u64()), 0i64);\n    }\n\n    #[test]\n    fn test_intfastfield_small() -> crate::Result<()> {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&SCHEMA).unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>13u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>14u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>2u64))\n                .unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n\n        assert_eq!(file.len(), 80);\n        let fast_field_readers = FastFieldReaders::open(file, SCHEMA.clone()).unwrap();\n        let column = fast_field_readers\n            .u64(\"field\")\n            .unwrap()\n            .first_or_default_col(0);\n        assert_eq!(column.get_val(0), 13u64);\n        assert_eq!(column.get_val(1), 14u64);\n        assert_eq!(column.get_val(2), 2u64);\n        Ok(())\n    }\n\n    #[test]\n    fn test_intfastfield_large() {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&SCHEMA).unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>4u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>14_082_001u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>3_052u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>9_002u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>15_001u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>777u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>1_002u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>1_501u64))\n                .unwrap();\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>215u64))\n                .unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 108);\n        let fast_field_readers = FastFieldReaders::open(file, SCHEMA.clone()).unwrap();\n        let col = fast_field_readers\n            .u64(\"field\")\n            .unwrap()\n            .first_or_default_col(0);\n        assert_eq!(col.get_val(0), 4u64);\n        assert_eq!(col.get_val(1), 14_082_001u64);\n        assert_eq!(col.get_val(2), 3_052u64);\n        assert_eq!(col.get_val(3), 9002u64);\n        assert_eq!(col.get_val(4), 15_001u64);\n        assert_eq!(col.get_val(5), 777u64);\n        assert_eq!(col.get_val(6), 1_002u64);\n        assert_eq!(col.get_val(7), 1_501u64);\n        assert_eq!(col.get_val(8), 215u64);\n    }\n\n    #[test]\n    fn test_intfastfield_null_amplitude() {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&SCHEMA).unwrap();\n            for _ in 0..10_000 {\n                fast_field_writers\n                    .add_document(&doc!(*FIELD=>100_000u64))\n                    .unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 81);\n        let fast_field_readers = FastFieldReaders::open(file, SCHEMA.clone()).unwrap();\n        let fast_field_reader = fast_field_readers\n            .u64(\"field\")\n            .unwrap()\n            .first_or_default_col(0);\n        for doc in 0..10_000 {\n            assert_eq!(fast_field_reader.get_val(doc), 100_000u64);\n        }\n    }\n\n    #[test]\n    fn test_intfastfield_large_numbers() {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&SCHEMA).unwrap();\n            // forcing the amplitude to be high\n            fast_field_writers\n                .add_document(&doc!(*FIELD=>0u64))\n                .unwrap();\n            for doc_id in 1u64..10_000u64 {\n                fast_field_writers\n                    .add_document(&doc!(*FIELD=>5_000_000_000_000_000_000u64 + doc_id))\n                    .unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 4476);\n        {\n            let fast_field_readers = FastFieldReaders::open(file, SCHEMA.clone()).unwrap();\n            let col = fast_field_readers\n                .u64(\"field\")\n                .unwrap()\n                .first_or_default_col(0);\n            for doc in 1..10_000 {\n                assert_eq!(col.get_val(doc), 5_000_000_000_000_000_000u64 + doc as u64);\n            }\n        }\n    }\n\n    #[test]\n    fn test_signed_intfastfield_normal() -> crate::Result<()> {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        let mut schema_builder = Schema::builder();\n\n        let i64_field = schema_builder.add_i64_field(\"field\", FAST);\n        let schema = schema_builder.build();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            for i in -100i64..10_000i64 {\n                let mut doc = TantivyDocument::default();\n                doc.add_i64(i64_field, i);\n                fast_field_writers.add_document(&doc).unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 252);\n\n        {\n            let fast_field_readers = FastFieldReaders::open(file, schema).unwrap();\n            let col = fast_field_readers\n                .i64(\"field\")\n                .unwrap()\n                .first_or_default_col(0);\n            assert_eq!(col.min_value(), -100i64);\n            assert_eq!(col.max_value(), 9_999i64);\n            for (doc, i) in (-100i64..10_000i64).enumerate() {\n                assert_eq!(col.get_val(doc as u32), i);\n            }\n            let mut buffer = vec![0i64; 100];\n            col.get_range(53, &mut buffer[..]);\n            for i in 0..100 {\n                assert_eq!(buffer[i], -100i64 + 53i64 + i as i64);\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_signed_intfastfield_default_val() {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_i64_field(\"field\", FAST);\n        let schema = schema_builder.build();\n\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            let doc = TantivyDocument::default();\n            fast_field_writers.add_document(&doc).unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n\n        let file = directory.open_read(path).unwrap();\n        let fast_field_readers = FastFieldReaders::open(file, schema).unwrap();\n        let col = fast_field_readers.i64(\"field\").unwrap();\n        assert_eq!(col.first(0), None);\n\n        let col = fast_field_readers\n            .i64(\"field\")\n            .unwrap()\n            .first_or_default_col(0);\n        assert_eq!(col.get_val(0), 0);\n        let col = fast_field_readers\n            .i64(\"field\")\n            .unwrap()\n            .first_or_default_col(-100);\n        assert_eq!(col.get_val(0), -100);\n    }\n\n    #[test]\n    fn test_date_fastfield_default() {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_date_field(\"date\", FAST);\n        let schema = schema_builder.build();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            let doc = TantivyDocument::default();\n            fast_field_writers.add_document(&doc).unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n\n        let file = directory.open_read(path).unwrap();\n        let fast_field_readers = FastFieldReaders::open(file, schema).unwrap();\n        let col = fast_field_readers\n            .date(\"date\")\n            .unwrap()\n            .first_or_default_col(DateTime::default());\n        assert_eq!(col.get_val(0), DateTime::default());\n    }\n\n    // Warning: this generates the same permutation at each call\n    pub fn generate_permutation() -> Vec<u64> {\n        let mut permutation: Vec<u64> = (0u64..100_000u64).collect();\n        permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n        permutation\n    }\n\n    // Warning: this generates the same permutation at each call\n    pub fn generate_permutation_gcd() -> Vec<u64> {\n        let mut permutation: Vec<u64> = (1u64..100_000u64).map(|el| el * 1000).collect();\n        permutation.shuffle(&mut StdRng::from_seed([1u8; 32]));\n        permutation\n    }\n\n    fn test_intfastfield_permutation_with_data(permutation: Vec<u64>) {\n        let path = Path::new(\"test\");\n        let n = permutation.len();\n        let directory = RamDirectory::create();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&SCHEMA).unwrap();\n            for &x in &permutation {\n                fast_field_writers.add_document(&doc!(*FIELD=>x)).unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        let fast_field_readers = FastFieldReaders::open(file, SCHEMA.clone()).unwrap();\n        let col = fast_field_readers\n            .u64(\"field\")\n            .unwrap()\n            .first_or_default_col(0);\n        for a in 0..n {\n            assert_eq!(col.get_val(a as u32), permutation[a]);\n        }\n    }\n\n    #[test]\n    fn test_intfastfield_permutation_gcd() {\n        let permutation = generate_permutation_gcd();\n        test_intfastfield_permutation_with_data(permutation);\n    }\n\n    #[test]\n    fn test_intfastfield_permutation() {\n        let permutation = generate_permutation();\n        test_intfastfield_permutation_with_data(permutation);\n    }\n\n    #[test]\n    fn test_merge_missing_date_fast_field() {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n        index_writer\n            .add_document(doc!(date_field => DateTime::from_utc(OffsetDateTime::now_utc())))\n            .unwrap();\n        index_writer.commit().unwrap();\n        index_writer.add_document(doc!()).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let segment_ids: Vec<SegmentId> = reader\n            .searcher()\n            .segment_readers()\n            .iter()\n            .map(SegmentReader::segment_id)\n            .collect();\n        assert_eq!(segment_ids.len(), 2);\n        index_writer.merge(&segment_ids[..]).wait().unwrap();\n        reader.reload().unwrap();\n        assert_eq!(reader.searcher().segment_readers().len(), 1);\n    }\n\n    fn get_vals_for_docs(column: &Column<u64>, docs: Range<u32>) -> Vec<u64> {\n        docs.into_iter()\n            .flat_map(|doc| column.values_for_doc(doc))\n            .collect()\n    }\n\n    #[test]\n    fn test_text_fastfield() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            // first segment\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer\n                .add_document(doc!(\n                text_field => \"BBBBB\", // term ord 1\n                text_field => \"AAAAA\", // term ord 0\n                ))\n                .unwrap();\n            index_writer.add_document(doc!()).unwrap();\n            index_writer\n                .add_document(doc!(\n                text_field => \"AAAAA\", // term_ord 0\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    text_field => \"AAAAA\",\n                    text_field => \"BBBBB\",\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                text_field => \"zumberthree\", // term_ord 2, after merge term_ord 3\n                ))\n                .unwrap();\n\n            index_writer.add_document(doc!()).unwrap();\n            index_writer.commit().unwrap();\n\n            let reader = index.reader().unwrap();\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            let segment_reader = searcher.segment_reader(0);\n            let fast_fields = segment_reader.fast_fields();\n            let str_column = fast_fields.str(\"text\").unwrap().unwrap();\n            assert!(str_column.ords().values_for_doc(0u32).eq([1, 0]),);\n            assert!(str_column.ords().values_for_doc(1u32).next().is_none());\n            assert!(str_column.ords().values_for_doc(2u32).eq([0]),);\n            assert!(str_column.ords().values_for_doc(3u32).eq([0, 1]),);\n            assert!(str_column.ords().values_for_doc(4u32).eq([2]),);\n\n            let mut str_term = String::default();\n            assert!(str_column.ord_to_str(0, &mut str_term).unwrap());\n            assert_eq!(\"AAAAA\", &str_term);\n\n            let inverted_index = segment_reader.inverted_index(text_field).unwrap();\n            assert_eq!(inverted_index.terms().num_terms(), 3);\n            let mut bytes = vec![];\n            assert!(inverted_index.terms().ord_to_term(0, &mut bytes).unwrap());\n            assert_eq!(bytes, \"aaaaa\".as_bytes());\n        }\n\n        {\n            // second segment\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n\n            index_writer\n                .add_document(doc!(\n                text_field => \"AAAAA\", // term_ord 0\n                ))\n                .unwrap();\n\n            index_writer\n                .add_document(doc!(\n                text_field => \"CCCCC AAAAA\", // term_ord 1, after merge 2\n                ))\n                .unwrap();\n\n            index_writer.add_document(doc!()).unwrap();\n            index_writer.commit().unwrap();\n\n            let reader = index.reader().unwrap();\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 2);\n            let segment_reader = searcher.segment_reader(1);\n            let fast_fields = segment_reader.fast_fields();\n            let text_fast_field = fast_fields.str(\"text\").unwrap().unwrap();\n\n            assert_eq!(&get_vals_for_docs(text_fast_field.ords(), 0..2), &[0, 1]);\n        }\n\n        // TODO uncomment once merging is available\n        // Merging the segments\n        {\n            let segment_ids = index.searchable_segment_ids().unwrap();\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer.merge(&segment_ids).wait().unwrap();\n            index_writer.wait_merging_threads().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let fast_fields = segment_reader.fast_fields();\n        let text_column = fast_fields.str(\"text\").unwrap().unwrap();\n\n        assert_eq!(\n            get_vals_for_docs(text_column.ords(), 0..8),\n            vec![1, 0, 0, 0, 1, 3 /* next segment */, 0, 2]\n        );\n    }\n\n    #[test]\n    fn test_string_fastfield_simple() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(text_field=>\"hello happy tax payer\", text_field=>\"aaa this string comes lexicographically before the other one.\")).unwrap();\n        writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let str_column = segment_reader.fast_fields().str(\"text\").unwrap().unwrap();\n        // The string values are not sorted here.\n        let term_ords: Vec<u64> = str_column.term_ords(0u32).collect();\n        assert_eq!(&term_ords, &[1, 0]);\n    }\n\n    #[test]\n    fn test_facet_fastfield_simple() {\n        let mut schema_builder = Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer\n            .add_document(doc!(facet_field=>Facet::from(\"/a/2\"), facet_field=>Facet::from(\"/a/1\")))\n            .unwrap();\n        writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let facet_reader = segment_reader.facet_reader(\"facet\").unwrap();\n        // facets, contrary to strings are sorted.\n        let mut facet_ords = Vec::new();\n        facet_ords.extend(facet_reader.facet_ords(0u32));\n        assert_eq!(&facet_ords, &[0, 1]);\n    }\n\n    #[test]\n    fn test_string_fastfield() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", STRING | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            // first segment\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            index_writer.add_document(doc!(\n                text_field => \"BBBBB\", // term_ord 1\n            ))?;\n            index_writer.add_document(doc!())?;\n            index_writer.add_document(doc!(\n                text_field => \"AAAAA\", // term_ord 0\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"AAAAA\", // term_ord 0\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"zumberthree\", // term_ord 2, after merge term_ord 3\n            ))?;\n\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n\n            let reader = index.reader()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            let segment_reader = searcher.segment_reader(0);\n            let fast_fields = segment_reader.fast_fields();\n            let text_col = fast_fields.str(\"text\").unwrap().unwrap();\n\n            assert_eq!(get_vals_for_docs(text_col.ords(), 0..6), vec![1, 0, 0, 2]);\n\n            let inverted_index = segment_reader.inverted_index(text_field)?;\n            assert_eq!(inverted_index.terms().num_terms(), 3);\n            let mut bytes = vec![];\n            assert!(inverted_index.terms().ord_to_term(0, &mut bytes)?);\n            assert_eq!(bytes, \"AAAAA\".as_bytes());\n        }\n\n        {\n            // second segment\n            let mut index_writer = index.writer_for_tests()?;\n\n            index_writer.add_document(doc!(\n                text_field => \"AAAAA\", // term_ord 0\n            ))?;\n\n            index_writer.add_document(doc!(\n                text_field => \"CCCCC\", // term_ord 1, after merge 2\n            ))?;\n\n            index_writer.add_document(doc!())?;\n            index_writer.commit()?;\n\n            let reader = index.reader()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 2);\n            let segment_reader = searcher.segment_reader(1);\n            let fast_fields = segment_reader.fast_fields();\n            let text_fast_field = fast_fields.str(\"text\").unwrap().unwrap();\n\n            assert_eq!(&get_vals_for_docs(text_fast_field.ords(), 0..2), &[0, 1]);\n        }\n        // Merging the segments\n        {\n            let segment_ids = index.searchable_segment_ids()?;\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let fast_fields = segment_reader.fast_fields();\n        let text_fast_field = fast_fields.str(\"text\").unwrap().unwrap();\n\n        assert_eq!(\n            get_vals_for_docs(text_fast_field.ords(), 0..9),\n            vec![1, 0, 0, 3 /* next segment */, 0, 2]\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_datefastfield() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\n            \"date\",\n            DateOptions::from(FAST).set_precision(DateTimePrecision::Nanoseconds),\n        );\n        let multi_date_field = schema_builder.add_date_field(\n            \"multi_date\",\n            DateOptions::default()\n                .set_precision(DateTimePrecision::Nanoseconds)\n                .set_fast(),\n        );\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n        index_writer.add_document(doc!(\n            date_field => DateTime::from_u64(1i64.to_u64()),\n            multi_date_field => DateTime::from_u64(2i64.to_u64()),\n            multi_date_field => DateTime::from_u64(3i64.to_u64())\n        ))?;\n        index_writer.add_document(doc!(\n            date_field => DateTime::from_u64(4i64.to_u64())\n        ))?;\n        index_writer.add_document(doc!(\n            multi_date_field => DateTime::from_u64(5i64.to_u64()),\n            multi_date_field => DateTime::from_u64(6i64.to_u64())\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let segment_reader = searcher.segment_reader(0);\n        let fast_fields = segment_reader.fast_fields();\n        let date_fast_field = fast_fields\n            .column_opt::<DateTime>(\"date\")\n            .unwrap()\n            .unwrap()\n            .first_or_default_col(Default::default());\n        let dates_fast_field = fast_fields\n            .column_opt::<DateTime>(\"multi_date\")\n            .unwrap()\n            .unwrap();\n\n        {\n            assert_eq!(date_fast_field.get_val(0).into_timestamp_nanos(), 1i64);\n            let dates: Vec<DateTime> = dates_fast_field.values_for_doc(0u32).collect();\n            assert_eq!(dates.len(), 2);\n            assert_eq!(dates[0].into_timestamp_nanos(), 2i64);\n            assert_eq!(dates[1].into_timestamp_nanos(), 3i64);\n        }\n        {\n            assert_eq!(date_fast_field.get_val(1).into_timestamp_nanos(), 4i64);\n            let dates: Vec<DateTime> = dates_fast_field.values_for_doc(1u32).collect();\n            assert!(dates.is_empty());\n        }\n        {\n            assert_eq!(date_fast_field.get_val(2).into_timestamp_nanos(), 0i64);\n            let dates: Vec<DateTime> = dates_fast_field.values_for_doc(2u32).collect();\n            assert_eq!(dates.len(), 2);\n            assert_eq!(dates[0].into_timestamp_nanos(), 5i64);\n            assert_eq!(dates[1].into_timestamp_nanos(), 6i64);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_fastfield_bool_small() {\n        let path = Path::new(\"test_bool\");\n        let directory: RamDirectory = RamDirectory::create();\n\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_bool_field(\"field_bool\", FAST);\n        let schema = schema_builder.build();\n        let field = schema.get_field(\"field_bool\").unwrap();\n\n        {\n            let mut write: WritePtr = directory.open_write(path).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            fast_field_writers.add_document(&doc!(field=>true)).unwrap();\n            fast_field_writers\n                .add_document(&doc!(field=>false))\n                .unwrap();\n            fast_field_writers.add_document(&doc!(field=>true)).unwrap();\n            fast_field_writers\n                .add_document(&doc!(field=>false))\n                .unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 84);\n        let fast_field_readers = FastFieldReaders::open(file, schema).unwrap();\n        let bool_col = fast_field_readers.bool(\"field_bool\").unwrap();\n        assert_eq!(bool_col.first(0), Some(true));\n        assert_eq!(bool_col.first(1), Some(false));\n        assert_eq!(bool_col.first(2), Some(true));\n        assert_eq!(bool_col.first(3), Some(false));\n    }\n\n    #[test]\n    pub fn test_fastfield_bool_large() {\n        let path = Path::new(\"test_bool\");\n        let directory: RamDirectory = RamDirectory::create();\n\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_bool_field(\"field_bool\", FAST);\n        let schema = schema_builder.build();\n        let field = schema.get_field(\"field_bool\").unwrap();\n\n        {\n            let mut write: WritePtr = directory.open_write(path).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            for _ in 0..50 {\n                fast_field_writers.add_document(&doc!(field=>true)).unwrap();\n                fast_field_writers\n                    .add_document(&doc!(field=>false))\n                    .unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 96);\n        let readers = FastFieldReaders::open(file, schema).unwrap();\n        let bool_col = readers.bool(\"field_bool\").unwrap();\n        for i in 0..25 {\n            assert_eq!(bool_col.first(i * 2), Some(true));\n            assert_eq!(bool_col.first(i * 2 + 1), Some(false));\n        }\n    }\n\n    #[test]\n    pub fn test_fastfield_bool_default_value() {\n        let path = Path::new(\"test_bool\");\n        let directory: RamDirectory = RamDirectory::create();\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_bool_field(\"field_bool\", FAST);\n        let schema = schema_builder.build();\n        {\n            let mut write: WritePtr = directory.open_write(path).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(&schema).unwrap();\n            let doc = TantivyDocument::default();\n            fast_field_writers.add_document(&doc).unwrap();\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        let file = directory.open_read(path).unwrap();\n        assert_eq!(file.len(), 86);\n        let fastfield_readers = FastFieldReaders::open(file, schema).unwrap();\n        let col = fastfield_readers.bool(\"field_bool\").unwrap();\n        assert_eq!(col.first(0), None);\n        let col = fastfield_readers\n            .bool(\"field_bool\")\n            .unwrap()\n            .first_or_default_col(false);\n        assert_eq!(col.get_val(0), false);\n        let col = fastfield_readers\n            .bool(\"field_bool\")\n            .unwrap()\n            .first_or_default_col(true);\n        assert_eq!(col.get_val(0), true);\n    }\n\n    fn get_index(docs: &[crate::TantivyDocument], schema: &Schema) -> crate::Result<RamDirectory> {\n        let directory: RamDirectory = RamDirectory::create();\n        {\n            let mut write: WritePtr = directory.open_write(Path::new(\"test\")).unwrap();\n            let mut fast_field_writers = FastFieldsWriter::from_schema(schema).unwrap();\n            for doc in docs {\n                fast_field_writers.add_document(doc).unwrap();\n            }\n            fast_field_writers.serialize(&mut write).unwrap();\n            write.terminate().unwrap();\n        }\n        Ok(directory)\n    }\n\n    #[test]\n    pub fn test_gcd_date() {\n        let size_prec_sec = test_gcd_date_with_codec(DateTimePrecision::Seconds);\n        assert!((1000 * 13 / 8..100 + 1000 * 13 / 8).contains(&size_prec_sec.get_bytes())); // 13 bits per val = ceil(log_2(number of seconds in 2hours);\n        let size_prec_micros = test_gcd_date_with_codec(DateTimePrecision::Microseconds);\n        assert!((1000 * 33 / 8..100 + 1000 * 33 / 8).contains(&size_prec_micros.get_bytes()));\n        // 33 bits per\n        // val = ceil(log_2(number\n        // of microsecsseconds\n        // in 2hours);\n    }\n\n    fn test_gcd_date_with_codec(precision: DateTimePrecision) -> ByteCount {\n        let mut rng = StdRng::seed_from_u64(2u64);\n        const T0: i64 = 1_662_345_825_012_529i64;\n        const ONE_HOUR_IN_MICROSECS: i64 = 3_600 * 1_000_000;\n        let times: Vec<DateTime> = std::iter::repeat_with(|| {\n            // +- One hour.\n            let t = T0 + rng.random_range(-ONE_HOUR_IN_MICROSECS..ONE_HOUR_IN_MICROSECS);\n            DateTime::from_timestamp_micros(t)\n        })\n        .take(1_000)\n        .collect();\n        let date_options = DateOptions::default().set_fast().set_precision(precision);\n        let mut schema_builder = SchemaBuilder::default();\n        let field = schema_builder.add_date_field(\"field\", date_options);\n        let schema = schema_builder.build();\n\n        let docs: Vec<TantivyDocument> = times.iter().map(|time| doc!(field=>*time)).collect();\n\n        let directory = get_index(&docs[..], &schema).unwrap();\n        let path = Path::new(\"test\");\n        let file = directory.open_read(path).unwrap();\n        let readers = FastFieldReaders::open(file, schema).unwrap();\n        let col = readers.date(\"field\").unwrap();\n\n        for (i, time) in times.iter().enumerate() {\n            let dt: DateTime = col.first(i as u32).unwrap();\n            assert_eq!(dt, time.truncate(precision));\n        }\n        readers.column_num_bytes(\"field\").unwrap()\n    }\n\n    #[test]\n    fn test_gcd_bug_regression_1757() {\n        let mut schema_builder = Schema::builder();\n        let num_field = schema_builder.add_u64_field(\"url_norm_hash\", FAST | INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut writer = index.writer_for_tests().unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 100u64,\n                })\n                .unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 200u64,\n                })\n                .unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 300u64,\n                })\n                .unwrap();\n\n            writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment = &searcher.segment_readers()[0];\n        let field = segment\n            .fast_fields()\n            .u64(\"url_norm_hash\")\n            .unwrap()\n            .first_or_default_col(0);\n\n        let numbers = [100, 200, 300];\n        let test_range = |range: RangeInclusive<u64>| {\n            let expected_count = numbers.iter().filter(|num| range.contains(*num)).count();\n            let mut vec = vec![];\n            field.get_row_ids_for_value_range(range, 0..u32::MAX, &mut vec);\n            assert_eq!(vec.len(), expected_count);\n        };\n        test_range(50..=50);\n        test_range(150..=150);\n        test_range(350..=350);\n        test_range(100..=250);\n        test_range(101..=200);\n        test_range(101..=199);\n        test_range(100..=300);\n        test_range(100..=299);\n    }\n\n    #[test]\n    fn test_ip_addr_columnar_simple() {\n        let mut schema_builder = Schema::builder();\n        let ip_field = schema_builder.add_u64_field(\"ip\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let ip_addr = Ipv6Addr::new(1, 2, 3, 4, 5, 1, 2, 3);\n        index_writer\n            .add_document(TantivyDocument::default())\n            .unwrap();\n        index_writer.add_document(doc!(ip_field=>ip_addr)).unwrap();\n        index_writer\n            .add_document(TantivyDocument::default())\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fastfields = searcher.segment_reader(0u32).fast_fields();\n        let column: Column<Ipv6Addr> = fastfields.column_opt(\"ip\").unwrap().unwrap();\n        assert_eq!(column.num_docs(), 3);\n        assert_eq!(column.first(0), None);\n        assert_eq!(column.first(1), Some(ip_addr));\n        assert_eq!(column.first(2), None);\n    }\n\n    #[test]\n    fn test_mapping_bug_docids_for_value_range() {\n        let mut schema_builder = Schema::builder();\n        let num_field = schema_builder.add_u64_field(\"url_norm_hash\", FAST | INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // Values without gcd, but with min_value\n            let mut writer = index.writer_for_tests().unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 1000u64,\n                })\n                .unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 1001u64,\n                })\n                .unwrap();\n            writer\n                .add_document(doc! {\n                    num_field => 1003u64,\n                })\n                .unwrap();\n            writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment = &searcher.segment_readers()[0];\n        let field = segment\n            .fast_fields()\n            .u64(\"url_norm_hash\")\n            .unwrap()\n            .first_or_default_col(0);\n\n        let numbers = [1000, 1001, 1003];\n        let test_range = |range: RangeInclusive<u64>| {\n            let expected_count = numbers.iter().filter(|num| range.contains(*num)).count();\n            let mut vec = vec![];\n            field.get_row_ids_for_value_range(range, 0..u32::MAX, &mut vec);\n            assert_eq!(vec.len(), expected_count);\n        };\n        let test_range_variant = |start, stop| {\n            let start_range = start..=stop;\n            test_range(start_range);\n            let start_range = start..=(stop - 1);\n            test_range(start_range);\n            let start_range = start..=(stop + 1);\n            test_range(start_range);\n            let start_range = (start - 1)..=stop;\n            test_range(start_range);\n            let start_range = (start - 1)..=(stop - 1);\n            test_range(start_range);\n            let start_range = (start - 1)..=(stop + 1);\n            test_range(start_range);\n            let start_range = (start + 1)..=stop;\n            test_range(start_range);\n            let start_range = (start + 1)..=(stop - 1);\n            test_range(start_range);\n            let start_range = (start + 1)..=(stop + 1);\n            test_range(start_range);\n        };\n        test_range_variant(50, 50);\n        test_range_variant(1000, 1000);\n        test_range_variant(1000, 1002);\n    }\n\n    #[test]\n    fn test_json_object_fast_field() {\n        let mut schema_builder = Schema::builder();\n        let without_fast_field = schema_builder.add_json_field(\"without\", STORED);\n        let with_fast_field = schema_builder.add_json_field(\"with\", STORED | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer\n            .add_document(doc!(without_fast_field=>json!({\"hello\": \"without\"})))\n            .unwrap();\n        writer\n            .add_document(doc!(with_fast_field=>json!({\"hello\": \"with\"})))\n            .unwrap();\n        writer\n            .add_document(doc!(with_fast_field=>json!({\"hello\": \"with2\"})))\n            .unwrap();\n        writer\n            .add_document(doc!(with_fast_field=>json!({\"hello\": \"with1\"})))\n            .unwrap();\n        writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n        let fast_fields = segment_reader.fast_fields();\n        let column_without_opt = fast_fields.str(\"without.hello\");\n        assert!(column_without_opt.is_err());\n        let column_with_opt: Option<StrColumn> = fast_fields.str(\"with.hello\").unwrap();\n        let column_with: StrColumn = column_with_opt.unwrap();\n        assert!(column_with.term_ords(0).next().is_none());\n        assert!(column_with.term_ords(1).eq([0]));\n        assert!(column_with.term_ords(2).eq([2]));\n        assert!(column_with.term_ords(3).eq([1]));\n    }\n\n    #[test]\n    fn test_fast_field_in_json_field_expand_dots_disabled() {\n        let mut schema_builder = Schema::builder();\n        let json_option = JsonObjectOptions::default().set_fast(None);\n        let json = schema_builder.add_json_field(\"json\", json_option);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"attr.age\": 32})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        assert!(fast_field_reader\n            .column_opt::<i64>(\"json.attr.age\")\n            .unwrap()\n            .is_none());\n        let column = fast_field_reader\n            .column_opt::<i64>(r\"json.attr\\.age\")\n            .unwrap()\n            .unwrap();\n        let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n        assert_eq!(&vals, &[32])\n    }\n\n    #[test]\n    fn test_fast_field_in_json_field_with_tokenizer() {\n        let mut schema_builder = Schema::builder();\n        let json_option = JsonObjectOptions::default().set_fast(Some(\"default\"));\n        let json = schema_builder.add_json_field(\"json\", json_option);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"age\": 32})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"age\": \"NEW\"})))\n            .unwrap();\n\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_fields = searcher.segment_reader(0u32).fast_fields();\n\n        let ff_str = fast_fields.str(\"json.age\").unwrap().unwrap();\n        let mut output = String::new();\n        ff_str.ord_to_str(0, &mut output).unwrap();\n        assert_eq!(output, \"new\");\n    }\n\n    #[test]\n    fn test_fast_field_in_json_field_expand_dots_enabled() {\n        let mut schema_builder = Schema::builder();\n        let json_option = JsonObjectOptions::default()\n            .set_fast(None)\n            .set_expand_dots_enabled();\n        let json = schema_builder.add_json_field(\"json\", json_option);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(json => json!({\"attr.age\": 32})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        for test_column_name in &[\"json.attr.age\", \"json.attr\\\\.age\"] {\n            let column = fast_field_reader\n                .column_opt::<i64>(test_column_name)\n                .unwrap()\n                .unwrap();\n            let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n            assert_eq!(&vals, &[32]);\n        }\n    }\n\n    #[test]\n    fn test_fast_field_dot_in_schema_field_name() {\n        let mut schema_builder = Schema::builder();\n        let field_with_dot = schema_builder.add_i64_field(\"field.with.dot\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(field_with_dot => 32i64))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        let column = fast_field_reader\n            .column_opt::<i64>(\"field.with.dot\")\n            .unwrap()\n            .unwrap();\n        let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n        assert_eq!(&vals, &[32]);\n    }\n\n    #[test]\n    fn test_shadowing_fast_field() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"jsonfield\", FAST);\n        let shadowing_json_field = schema_builder.add_json_field(\"jsonfield.attr\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(json_field=> json!({\"attr\": {\"age\": 32}}), shadowing_json_field=>json!({\"age\": 33})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        let column = fast_field_reader\n            .column_opt::<i64>(\"jsonfield.attr.age\")\n            .unwrap()\n            .unwrap();\n        let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n        assert_eq!(&vals, &[33]);\n    }\n\n    #[test]\n    fn test_fast_field_tokenizer() {\n        let mut schema_builder = Schema::builder();\n        let opt = TextOptions::default().set_fast(Some(\"custom_lowercase\"));\n        let text_field = schema_builder.add_text_field(\"text\", opt);\n        let schema = schema_builder.build();\n        let ff_tokenizer_manager = TokenizerManager::default();\n        ff_tokenizer_manager.register(\n            \"custom_lowercase\",\n            TextAnalyzer::builder(RawTokenizer::default())\n                .filter(LowerCaser)\n                .build(),\n        );\n\n        let mut index = Index::create_in_ram(schema);\n        index.set_fast_field_tokenizers(ff_tokenizer_manager);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"Test1 test2\"))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        let column = fast_field_reader.str(\"text\").unwrap().unwrap();\n        let mut out = String::new();\n        column.ord_to_str(0u64, &mut out).unwrap();\n        assert_eq!(&out, \"test1 test2\");\n    }\n\n    #[test]\n    fn test_text_fast_field_tokenizer() {\n        let mut schema_builder = Schema::builder();\n\n        let text_fieldtype = crate::schema::TextOptions::default()\n            .set_indexing_options(\n                crate::schema::TextFieldIndexing::default()\n                    .set_index_option(crate::schema::IndexRecordOption::WithFreqs)\n                    .set_tokenizer(\"raw\"),\n            )\n            .set_fast(Some(\"default\"))\n            .set_stored();\n\n        let log_field = schema_builder.add_text_field(\"log_level\", text_fieldtype);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(log_field => \"info\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(log_field => \"INFO\"))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n\n        let text_fast_field = fast_field_reader.str(\"log_level\").unwrap().unwrap();\n        let mut buffer = String::new();\n        assert!(text_fast_field.ord_to_str(0, &mut buffer).unwrap());\n        assert_eq!(buffer, \"info\");\n        assert!(!text_fast_field.ord_to_str(1, &mut buffer).unwrap());\n\n        assert!(text_fast_field.term_ords(0).eq([0].into_iter()));\n        assert!(text_fast_field.term_ords(1).eq([0].into_iter()));\n        assert!(text_fast_field.ords().values_for_doc(0u32).eq([0]));\n        assert!(text_fast_field.ords().values_for_doc(1u32).eq([0]));\n    }\n\n    #[test]\n    fn test_shadowing_fast_field_with_expand_dots() {\n        let mut schema_builder = Schema::builder();\n        let json_option = JsonObjectOptions::default()\n            .set_fast(None)\n            .set_expand_dots_enabled();\n        let json_field = schema_builder.add_json_field(\"jsonfield\", json_option.clone());\n        let shadowing_json_field = schema_builder.add_json_field(\"jsonfield.attr\", json_option);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(json_field=> json!({\"attr.age\": 32}), shadowing_json_field=>json!({\"age\": 33})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let fast_field_reader = searcher.segment_reader(0u32).fast_fields();\n        // Supported for now, maybe dropped in the future.\n        let column = fast_field_reader\n            .column_opt::<i64>(\"jsonfield.attr.age\")\n            .unwrap()\n            .unwrap();\n        let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n        assert_eq!(&vals, &[33]);\n        let column = fast_field_reader\n            .column_opt::<i64>(\"jsonfield\\\\.attr.age\")\n            .unwrap()\n            .unwrap();\n        let vals: Vec<i64> = column.values_for_doc(0u32).collect();\n        assert_eq!(&vals, &[33]);\n    }\n}\n"
  },
  {
    "path": "src/fastfield/readers.rs",
    "content": "use std::io;\nuse std::net::Ipv6Addr;\nuse std::sync::Arc;\n\nuse columnar::{\n    BytesColumn, Column, ColumnType, ColumnValues, ColumnarReader, DynamicColumn,\n    DynamicColumnHandle, HasAssociatedColumnType, StrColumn,\n};\nuse common::ByteCount;\n\nuse crate::core::json_utils::{encode_column_name, json_path_sep_to_dot};\nuse crate::directory::FileSlice;\nuse crate::schema::{Field, FieldEntry, FieldType, Schema};\nuse crate::space_usage::{FieldUsage, PerFieldSpaceUsage};\nuse crate::TantivyError;\n\n/// Provides access to all of the BitpackedFastFieldReader.\n///\n/// Internally, `FastFieldReaders` have preloaded fast field readers,\n/// and just wraps several `HashMap`.\n#[derive(Clone)]\npub struct FastFieldReaders {\n    columnar: Arc<ColumnarReader>,\n    schema: Schema,\n}\n\nimpl FastFieldReaders {\n    pub(crate) fn open(fast_field_file: FileSlice, schema: Schema) -> io::Result<FastFieldReaders> {\n        let columnar = Arc::new(ColumnarReader::open(fast_field_file)?);\n        Ok(FastFieldReaders { columnar, schema })\n    }\n\n    fn resolve_field(&self, column_name: &str) -> crate::Result<Option<String>> {\n        let default_field_opt: Option<Field> = if cfg!(feature = \"quickwit\") {\n            self.schema.get_field(\"_dynamic\").ok()\n        } else {\n            None\n        };\n        self.resolve_column_name_given_default_field(column_name, default_field_opt)\n    }\n\n    pub(crate) fn space_usage(&self) -> io::Result<PerFieldSpaceUsage> {\n        let mut per_field_usages: Vec<FieldUsage> = Default::default();\n        for (mut field_name, column_handle) in self.columnar.iter_columns()? {\n            json_path_sep_to_dot(&mut field_name);\n            let space_usage = column_handle.space_usage()?;\n            let mut field_usage = FieldUsage::empty(field_name);\n            field_usage.set_column_usage(space_usage);\n            per_field_usages.push(field_usage);\n        }\n        Ok(PerFieldSpaceUsage::new(per_field_usages))\n    }\n\n    pub(crate) fn columnar(&self) -> &ColumnarReader {\n        self.columnar.as_ref()\n    }\n\n    /// Transforms a user-supplied fast field name into a column name.\n    ///\n    /// A user-supplied fast field name is not necessarily a schema field name\n    /// because we handle fast fields.\n    ///\n    /// For instance, if the documents look like `{.., \"attributes\": {\"color\": \"red\"}}` and\n    /// `attributes` is a json fast field,  a user could want to run a term aggregation over\n    /// colors, by referring to the field as `attributes.color`.\n    ///\n    /// This function transforms `attributes.color` into a column key to be used in the `columnar`.\n    ///\n    /// The logic works as follows, first we identify which field is targeted by calling\n    /// `schema.find_field(..)`. This method will attempt to split the user splied fast field\n    /// name by non-escaped dots, and find the longest matching schema field name.\n    /// In our case, it would return the (attribute_field, \"color\").\n    ///\n    /// If no field is found, but a dynamic field is supplied, then we\n    /// will simply assume the user is targeting the dynamic field. (This feature is used in\n    /// Quickwit.)\n    ///\n    /// We then encode the `(field, path)` into the right `columnar_key`.\n    fn resolve_column_name_given_default_field<'a>(\n        &'a self,\n        field_name: &'a str,\n        default_field_opt: Option<Field>,\n    ) -> crate::Result<Option<String>> {\n        let Some((field, path)): Option<(Field, &str)> = self\n            .schema\n            .find_field_with_default(field_name, default_field_opt)\n        else {\n            return Ok(None);\n        };\n        let field_entry: &FieldEntry = self.schema.get_field_entry(field);\n        if !field_entry.is_fast() {\n            return Err(TantivyError::InvalidArgument(format!(\n                \"Field {field_name:?} is not configured as fast field\"\n            )));\n        }\n        Ok(match (field_entry.field_type(), path) {\n            (FieldType::JsonObject(json_options), path) if !path.is_empty() => {\n                Some(encode_column_name(\n                    field_entry.name(),\n                    path,\n                    json_options.is_expand_dots_enabled(),\n                ))\n            }\n            (_, \"\") => Some(field_entry.name().to_string()),\n            _ => None,\n        })\n    }\n\n    /// Returns a typed column associated to a given field name.\n    ///\n    /// If no column associated with that field_name exists,\n    /// or existing columns do not have the required type,\n    /// returns `None`.\n    pub fn column_opt<T>(&self, field_name: &str) -> crate::Result<Option<Column<T>>>\n    where\n        T: HasAssociatedColumnType,\n        DynamicColumn: Into<Option<Column<T>>>,\n    {\n        let Some(dynamic_column_handle) =\n            self.dynamic_column_handle(field_name, T::column_type())?\n        else {\n            return Ok(None);\n        };\n        let dynamic_column = dynamic_column_handle.open()?;\n        Ok(dynamic_column.into())\n    }\n\n    /// Returns the number of `bytes` associated with a column.\n    ///\n    /// Returns 0 if the column does not exist.\n    pub fn column_num_bytes(&self, field: &str) -> crate::Result<ByteCount> {\n        let Some(resolved_field_name) = self.resolve_field(field)? else {\n            return Ok(0u64.into());\n        };\n        Ok(self\n            .columnar\n            .read_columns(&resolved_field_name)?\n            .into_iter()\n            .map(|column_handle| column_handle.num_bytes())\n            .sum())\n    }\n\n    /// Returns a typed column value object.\n    ///\n    /// In that column value:\n    /// - Rows with no value are associated with the default value.\n    /// - Rows with several values are associated with the first value.\n    pub fn column_first_or_default<T>(&self, field: &str) -> crate::Result<Arc<dyn ColumnValues<T>>>\n    where\n        T: PartialOrd + Copy + HasAssociatedColumnType + Send + Sync + 'static,\n        DynamicColumn: Into<Option<Column<T>>>,\n    {\n        let col: Column<T> = self.column(field)?;\n        Ok(col.first_or_default_col(T::default_value()))\n    }\n\n    /// Returns a typed column associated to a given field name.\n    ///\n    /// Returns an error if no column associated with that field_name exists.\n    fn column<T>(&self, field: &str) -> crate::Result<Column<T>>\n    where\n        T: PartialOrd + Copy + HasAssociatedColumnType + Send + Sync + 'static,\n        DynamicColumn: Into<Option<Column<T>>>,\n    {\n        let col_opt: Option<Column<T>> = self.column_opt(field)?;\n        col_opt.ok_or_else(|| {\n            crate::TantivyError::SchemaError(format!(\n                \"Field `{field}` is missing or is not configured as a fast field.\"\n            ))\n        })\n    }\n\n    /// Returns the `u64` fast field reader reader associated with `field`.\n    ///\n    /// If `field` is not a u64 fast field, this method returns an Error.\n    pub fn u64(&self, field: &str) -> crate::Result<Column<u64>> {\n        self.column(field)\n    }\n\n    /// Returns the `date` fast field reader reader associated with `field`.\n    ///\n    /// If `field` is not a date fast field, this method returns an Error.\n    pub fn date(&self, field: &str) -> crate::Result<Column<common::DateTime>> {\n        self.column(field)\n    }\n\n    /// Returns the `ip` fast field reader reader associated to `field`.\n    ///\n    /// If `field` is not a u128 fast field, this method returns an Error.\n    pub fn ip_addr(&self, field: &str) -> crate::Result<Column<Ipv6Addr>> {\n        self.column(field)\n    }\n\n    /// Returns a `str` column.\n    pub fn str(&self, field_name: &str) -> crate::Result<Option<StrColumn>> {\n        let Some(dynamic_column_handle) =\n            self.dynamic_column_handle(field_name, ColumnType::Str)?\n        else {\n            return Ok(None);\n        };\n        let dynamic_column = dynamic_column_handle.open()?;\n        Ok(dynamic_column.into())\n    }\n\n    /// Returns a `bytes` column.\n    pub fn bytes(&self, field_name: &str) -> crate::Result<Option<BytesColumn>> {\n        let Some(dynamic_column_handle) =\n            self.dynamic_column_handle(field_name, ColumnType::Bytes)?\n        else {\n            return Ok(None);\n        };\n        let dynamic_column = dynamic_column_handle.open()?;\n        Ok(dynamic_column.into())\n    }\n\n    /// Returns a `dynamic_column_handle`.\n    pub fn dynamic_column_handle(\n        &self,\n        field_name: &str,\n        column_type: ColumnType,\n    ) -> crate::Result<Option<DynamicColumnHandle>> {\n        let Some(resolved_field_name) = self.resolve_field(field_name)? else {\n            return Ok(None);\n        };\n        let dynamic_column_handle_opt = self\n            .columnar\n            .read_columns(&resolved_field_name)?\n            .into_iter()\n            .find(|column| column.column_type() == column_type);\n        Ok(dynamic_column_handle_opt)\n    }\n\n    /// Returns all `dynamic_column_handle` that match the given field name.\n    pub fn dynamic_column_handles(\n        &self,\n        field_name: &str,\n    ) -> crate::Result<Vec<DynamicColumnHandle>> {\n        let Some(resolved_field_name) = self.resolve_field(field_name)? else {\n            return Ok(Vec::new());\n        };\n        let dynamic_column_handles = self\n            .columnar\n            .read_columns(&resolved_field_name)?\n            .into_iter()\n            .collect();\n        Ok(dynamic_column_handles)\n    }\n\n    /// Returns all `dynamic_column_handle` that are inner fields of the provided JSON path.\n    pub fn dynamic_subpath_column_handles(\n        &self,\n        root_path: &str,\n    ) -> crate::Result<Vec<DynamicColumnHandle>> {\n        let Some(resolved_field_name) = self.resolve_field(root_path)? else {\n            return Ok(Vec::new());\n        };\n        let dynamic_column_handles = self\n            .columnar\n            .read_subpath_columns(&resolved_field_name)?\n            .into_iter()\n            .collect();\n        Ok(dynamic_column_handles)\n    }\n\n    #[doc(hidden)]\n    pub async fn list_dynamic_column_handles(\n        &self,\n        field_name: &str,\n    ) -> crate::Result<Vec<DynamicColumnHandle>> {\n        let Some(resolved_field_name) = self.resolve_field(field_name)? else {\n            return Ok(Vec::new());\n        };\n        let columns = self\n            .columnar\n            .read_columns_async(&resolved_field_name)\n            .await?;\n        Ok(columns)\n    }\n\n    #[doc(hidden)]\n    pub async fn list_subpath_dynamic_column_handles(\n        &self,\n        root_path: &str,\n    ) -> crate::Result<Vec<DynamicColumnHandle>> {\n        let Some(resolved_field_name) = self.resolve_field(root_path)? else {\n            return Ok(Vec::new());\n        };\n        let columns = self\n            .columnar\n            .read_subpath_columns_async(&resolved_field_name)\n            .await?;\n        Ok(columns)\n    }\n\n    /// Returns the `u64` column used to represent any `u64`-mapped typed (String/Bytes term ids,\n    /// i64, u64, f64, DateTime).\n    ///\n    /// Returns Ok(None) for empty columns\n    #[doc(hidden)]\n    pub fn u64_lenient_for_type(\n        &self,\n        type_white_list_opt: Option<&[ColumnType]>,\n        field_name: &str,\n    ) -> crate::Result<Option<(Column<u64>, ColumnType)>> {\n        let Some(resolved_field_name) = self.resolve_field(field_name)? else {\n            return Ok(None);\n        };\n        for col in self.columnar.read_columns(&resolved_field_name)? {\n            if let Some(type_white_list) = type_white_list_opt {\n                if !type_white_list.contains(&col.column_type()) {\n                    continue;\n                }\n            }\n            if let Some(col_u64) = col.open_u64_lenient()? {\n                return Ok(Some((col_u64, col.column_type())));\n            }\n        }\n        Ok(None)\n    }\n\n    /// Returns the all `u64` column used to represent any `u64`-mapped typed (String/Bytes term\n    /// ids, i64, u64, f64, bool, DateTime).\n    ///\n    /// In case of JSON, there may be two columns. One for term and one for numerical types. (This\n    /// may change later to 3 types if JSON handles DateTime)\n    #[doc(hidden)]\n    pub fn u64_lenient_for_type_all(\n        &self,\n        type_white_list_opt: Option<&[ColumnType]>,\n        field_name: &str,\n    ) -> crate::Result<Vec<(Column<u64>, ColumnType)>> {\n        let mut columns_and_types = Vec::new();\n        let Some(resolved_field_name) = self.resolve_field(field_name)? else {\n            return Ok(columns_and_types);\n        };\n        for col in self.columnar.read_columns(&resolved_field_name)? {\n            if let Some(type_white_list) = type_white_list_opt {\n                if !type_white_list.contains(&col.column_type()) {\n                    continue;\n                }\n            }\n            if let Some(col_u64) = col.open_u64_lenient()? {\n                columns_and_types.push((col_u64, col.column_type()));\n            }\n        }\n        Ok(columns_and_types)\n    }\n\n    /// Returns the `u64` column used to represent any `u64`-mapped typed (i64, u64, f64, DateTime).\n    ///\n    /// Returns Ok(None) for empty columns\n    #[doc(hidden)]\n    pub fn u64_lenient(\n        &self,\n        field_name: &str,\n    ) -> crate::Result<Option<(Column<u64>, ColumnType)>> {\n        self.u64_lenient_for_type(None, field_name)\n    }\n\n    /// Returns the `i64` fast field reader reader associated with `field`.\n    ///\n    /// If `field` is not a i64 fast field, this method returns an Error.\n    pub fn i64(&self, field_name: &str) -> crate::Result<Column<i64>> {\n        self.column(field_name)\n    }\n\n    /// Returns the `f64` fast field reader reader associated with `field`.\n    ///\n    /// If `field` is not a f64 fast field, this method returns an Error.\n    pub fn f64(&self, field_name: &str) -> crate::Result<Column<f64>> {\n        self.column(field_name)\n    }\n\n    /// Returns the `bool` fast field reader reader associated with `field`.\n    ///\n    /// If `field` is not a bool fast field, this method returns an Error.\n    pub fn bool(&self, field_name: &str) -> crate::Result<Column<bool>> {\n        self.column(field_name)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use columnar::ColumnType;\n\n    use crate::schema::{JsonObjectOptions, Schema, FAST};\n    use crate::{Index, IndexWriter, TantivyDocument};\n\n    #[test]\n    fn test_fast_field_reader_resolve_with_dynamic_internal() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_i64_field(\"age\", FAST);\n        schema_builder.add_json_field(\"json_expand_dots_disabled\", FAST);\n        schema_builder.add_json_field(\n            \"json_expand_dots_enabled\",\n            JsonObjectOptions::default()\n                .set_fast(None)\n                .set_expand_dots_enabled(),\n        );\n        let dynamic_field = schema_builder.add_json_field(\"_dyna\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(TantivyDocument::default())\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let reader = searcher.segment_reader(0u32);\n        let fast_field_readers = reader.fast_fields();\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\"age\", None)\n                .unwrap(),\n            Some(\"age\".to_string())\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\"age\", Some(dynamic_field))\n                .unwrap(),\n            Some(\"age\".to_string())\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\n                    \"json_expand_dots_disabled.attr.color\",\n                    None\n                )\n                .unwrap(),\n            Some(\"json_expand_dots_disabled\\u{1}attr\\u{1}color\".to_string())\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\n                    \"json_expand_dots_disabled.attr\\\\.color\",\n                    Some(dynamic_field)\n                )\n                .unwrap(),\n            Some(\"json_expand_dots_disabled\\u{1}attr.color\".to_string())\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\n                    \"json_expand_dots_enabled.attr\\\\.color\",\n                    Some(dynamic_field)\n                )\n                .unwrap(),\n            Some(\"json_expand_dots_enabled\\u{1}attr\\u{1}color\".to_string())\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\"notinschema.attr.color\", None)\n                .unwrap(),\n            None\n        );\n        assert_eq!(\n            fast_field_readers\n                .resolve_column_name_given_default_field(\n                    \"notinschema.attr.color\",\n                    Some(dynamic_field)\n                )\n                .unwrap(),\n            Some(\"_dyna\\u{1}notinschema\\u{1}attr\\u{1}color\".to_string())\n        );\n    }\n\n    #[test]\n    fn test_fast_field_reader_dynamic_column_handles() {\n        let mut schema_builder = Schema::builder();\n        let id = schema_builder.add_u64_field(\"id\", FAST);\n        let json = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(id=> 1u64, json => json!({\"foo\": 42})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(id=> 2u64, json => json!({\"foo\": true})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(id=> 3u64, json => json!({\"foo\": \"bar\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let reader = searcher.segment_reader(0u32);\n        let fast_fields = reader.fast_fields();\n        let id_columns = fast_fields.dynamic_column_handles(\"id\").unwrap();\n        assert_eq!(id_columns.len(), 1);\n        assert_eq!(id_columns.first().unwrap().column_type(), ColumnType::U64);\n\n        let foo_columns = fast_fields.dynamic_column_handles(\"json.foo\").unwrap();\n        assert_eq!(foo_columns.len(), 3);\n        assert!(foo_columns\n            .iter()\n            .any(|column| column.column_type() == ColumnType::I64));\n        assert!(foo_columns\n            .iter()\n            .any(|column| column.column_type() == ColumnType::Bool));\n        assert!(foo_columns\n            .iter()\n            .any(|column| column.column_type() == ColumnType::Str));\n\n        let json_columns = fast_fields.dynamic_column_handles(\"json\").unwrap();\n        assert_eq!(json_columns.len(), 0);\n\n        let json_subcolumns = fast_fields.dynamic_subpath_column_handles(\"json\").unwrap();\n        assert_eq!(json_subcolumns.len(), 3);\n\n        let foo_subcolumns = fast_fields\n            .dynamic_subpath_column_handles(\"json.foo\")\n            .unwrap();\n        assert_eq!(foo_subcolumns.len(), 0);\n    }\n}\n"
  },
  {
    "path": "src/fastfield/writer.rs",
    "content": "use std::io;\n\nuse columnar::{ColumnarWriter, NumericalValue};\nuse common::{DateTimePrecision, JsonPathWriter};\nuse tokenizer_api::Token;\n\nuse crate::schema::document::{Document, ReferenceValue, ReferenceValueLeaf, Value};\nuse crate::schema::{value_type_to_column_type, Field, FieldType, Schema, Type};\nuse crate::tokenizer::{TextAnalyzer, TokenizerManager};\nuse crate::{DocId, TantivyError};\n\n/// Only index JSON down to a depth of 20.\n/// This is mostly to guard us from a stack overflow triggered by malicious input.\nconst JSON_DEPTH_LIMIT: usize = 20;\n\n/// The `FastFieldsWriter` groups all of the fast field writers.\npub struct FastFieldsWriter {\n    columnar_writer: ColumnarWriter,\n    fast_field_names: Vec<Option<String>>, //< TODO see if we can hash the field name hash too.\n    per_field_tokenizer: Vec<Option<TextAnalyzer>>,\n    date_precisions: Vec<DateTimePrecision>,\n    expand_dots: Vec<bool>,\n    num_docs: DocId,\n    // Buffer that we recycle to avoid allocation.\n    json_path_buffer: JsonPathWriter,\n}\n\nimpl FastFieldsWriter {\n    /// Create all `FastFieldWriter` required by the schema.\n    #[cfg(test)]\n    pub fn from_schema(schema: &Schema) -> crate::Result<FastFieldsWriter> {\n        FastFieldsWriter::from_schema_and_tokenizer_manager(schema, TokenizerManager::new())\n    }\n\n    /// Create all `FastFieldWriter` required by the schema.\n    pub fn from_schema_and_tokenizer_manager(\n        schema: &Schema,\n        tokenizer_manager: TokenizerManager,\n    ) -> crate::Result<FastFieldsWriter> {\n        let mut columnar_writer = ColumnarWriter::default();\n\n        let mut fast_field_names: Vec<Option<String>> = vec![None; schema.num_fields()];\n        let mut date_precisions: Vec<DateTimePrecision> =\n            std::iter::repeat_with(DateTimePrecision::default)\n                .take(schema.num_fields())\n                .collect();\n        let mut expand_dots = vec![false; schema.num_fields()];\n        let mut per_field_tokenizer: Vec<Option<TextAnalyzer>> = vec![None; schema.num_fields()];\n        // TODO see other types\n        for (field_id, field_entry) in schema.fields() {\n            if !field_entry.field_type().is_fast() {\n                continue;\n            }\n            fast_field_names[field_id.field_id() as usize] = Some(field_entry.name().to_string());\n            let value_type = field_entry.field_type().value_type();\n            if let FieldType::Date(date_options) = field_entry.field_type() {\n                date_precisions[field_id.field_id() as usize] = date_options.get_precision();\n            }\n            if let FieldType::JsonObject(json_object_options) = field_entry.field_type() {\n                if let Some(tokenizer_name) = json_object_options.get_fast_field_tokenizer_name() {\n                    let text_analyzer = tokenizer_manager.get(tokenizer_name).ok_or_else(|| {\n                        TantivyError::InvalidArgument(format!(\n                            \"Tokenizer {tokenizer_name:?} not found\"\n                        ))\n                    })?;\n                    per_field_tokenizer[field_id.field_id() as usize] = Some(text_analyzer);\n                }\n\n                expand_dots[field_id.field_id() as usize] =\n                    json_object_options.is_expand_dots_enabled();\n            }\n            if let FieldType::Str(text_options) = field_entry.field_type() {\n                if let Some(tokenizer_name) = text_options.get_fast_field_tokenizer_name() {\n                    let text_analyzer = tokenizer_manager.get(tokenizer_name).ok_or_else(|| {\n                        TantivyError::InvalidArgument(format!(\n                            \"Tokenizer {tokenizer_name:?} not found\"\n                        ))\n                    })?;\n                    per_field_tokenizer[field_id.field_id() as usize] = Some(text_analyzer);\n                }\n            }\n\n            let sort_values_within_row = value_type == Type::Facet;\n            if let Some(column_type) = value_type_to_column_type(value_type) {\n                columnar_writer.record_column_type(\n                    field_entry.name(),\n                    column_type,\n                    sort_values_within_row,\n                );\n            }\n        }\n        Ok(FastFieldsWriter {\n            columnar_writer,\n            fast_field_names,\n            per_field_tokenizer,\n            num_docs: 0u32,\n            date_precisions,\n            expand_dots,\n            json_path_buffer: JsonPathWriter::default(),\n        })\n    }\n\n    /// The memory used (inclusive childs)\n    pub fn mem_usage(&self) -> usize {\n        self.columnar_writer.mem_usage()\n    }\n\n    /// Indexes all of the fastfields of a new document.\n    pub fn add_document<D: Document>(&mut self, doc: &D) -> crate::Result<()> {\n        let doc_id = self.num_docs;\n        for (field, value) in doc.iter_fields_and_values() {\n            let value_access = value as D::Value<'_>;\n\n            self.add_doc_value(doc_id, field, value_access)?;\n        }\n        self.num_docs += 1;\n        Ok(())\n    }\n\n    fn add_doc_value<'a, V: Value<'a>>(\n        &mut self,\n        doc_id: DocId,\n        field: Field,\n        value: V,\n    ) -> crate::Result<()> {\n        let field_name = match &self.fast_field_names[field.field_id() as usize] {\n            None => return Ok(()),\n            Some(name) => name,\n        };\n\n        match value.as_value() {\n            ReferenceValue::Leaf(leaf) => match leaf {\n                ReferenceValueLeaf::Null => {}\n                ReferenceValueLeaf::Str(val) => {\n                    if let Some(tokenizer) =\n                        &mut self.per_field_tokenizer[field.field_id() as usize]\n                    {\n                        let mut token_stream = tokenizer.token_stream(val);\n                        token_stream.process(&mut |token: &Token| {\n                            self.columnar_writer\n                                .record_str(doc_id, field_name, &token.text);\n                        })\n                    } else {\n                        self.columnar_writer.record_str(doc_id, field_name, val);\n                    }\n                }\n                ReferenceValueLeaf::U64(val) => {\n                    self.columnar_writer.record_numerical(\n                        doc_id,\n                        field_name,\n                        NumericalValue::from(val),\n                    );\n                }\n                ReferenceValueLeaf::I64(val) => {\n                    self.columnar_writer.record_numerical(\n                        doc_id,\n                        field_name,\n                        NumericalValue::from(val),\n                    );\n                }\n                ReferenceValueLeaf::F64(val) => {\n                    self.columnar_writer.record_numerical(\n                        doc_id,\n                        field_name,\n                        NumericalValue::from(val),\n                    );\n                }\n                ReferenceValueLeaf::Date(val) => {\n                    let date_precision = self.date_precisions[field.field_id() as usize];\n                    let truncated_datetime = val.truncate(date_precision);\n                    self.columnar_writer\n                        .record_datetime(doc_id, field_name, truncated_datetime);\n                }\n                ReferenceValueLeaf::Facet(val) => {\n                    self.columnar_writer.record_str(doc_id, field_name, val);\n                }\n                ReferenceValueLeaf::Bytes(val) => {\n                    self.columnar_writer.record_bytes(doc_id, field_name, val);\n                }\n                ReferenceValueLeaf::IpAddr(val) => {\n                    self.columnar_writer.record_ip_addr(doc_id, field_name, val);\n                }\n                ReferenceValueLeaf::Bool(val) => {\n                    self.columnar_writer.record_bool(doc_id, field_name, val);\n                }\n                ReferenceValueLeaf::PreTokStr(val) => {\n                    for token in &val.tokens {\n                        self.columnar_writer\n                            .record_str(doc_id, field_name, &token.text);\n                    }\n                }\n            },\n            ReferenceValue::Array(val) => {\n                // TODO: Check this is the correct behaviour we want.\n                for value in val {\n                    self.add_doc_value(doc_id, field, value)?;\n                }\n            }\n            ReferenceValue::Object(val) => {\n                let expand_dots = self.expand_dots[field.field_id() as usize];\n                self.json_path_buffer.clear();\n                // First field should not be expanded.\n                self.json_path_buffer.set_expand_dots(false);\n                self.json_path_buffer.push(field_name);\n                self.json_path_buffer.set_expand_dots(expand_dots);\n\n                let text_analyzer = &mut self.per_field_tokenizer[field.field_id() as usize];\n\n                record_json_obj_to_columnar_writer::<V>(\n                    doc_id,\n                    val,\n                    JSON_DEPTH_LIMIT,\n                    &mut self.json_path_buffer,\n                    &mut self.columnar_writer,\n                    text_analyzer,\n                );\n            }\n        }\n\n        Ok(())\n    }\n\n    /// Serializes all of the `FastFieldWriter`s by pushing them in\n    /// order to the fast field serializer.\n    pub fn serialize(mut self, wrt: &mut dyn io::Write) -> io::Result<()> {\n        let num_docs = self.num_docs;\n        self.columnar_writer.serialize(num_docs, wrt)?;\n        Ok(())\n    }\n}\n\nfn record_json_obj_to_columnar_writer<'a, V: Value<'a>>(\n    doc: DocId,\n    json_visitor: V::ObjectIter,\n    remaining_depth_limit: usize,\n    json_path_buffer: &mut JsonPathWriter,\n    columnar_writer: &mut columnar::ColumnarWriter,\n    tokenizer: &mut Option<TextAnalyzer>,\n) {\n    for (key, child) in json_visitor {\n        json_path_buffer.push(key);\n        record_json_value_to_columnar_writer(\n            doc,\n            child,\n            remaining_depth_limit,\n            json_path_buffer,\n            columnar_writer,\n            tokenizer,\n        );\n        json_path_buffer.pop();\n    }\n}\n\nfn record_json_value_to_columnar_writer<'a, V: Value<'a>>(\n    doc: DocId,\n    json_val: V,\n    mut remaining_depth_limit: usize,\n    json_path_writer: &mut JsonPathWriter,\n    columnar_writer: &mut columnar::ColumnarWriter,\n    tokenizer: &mut Option<TextAnalyzer>,\n) {\n    if remaining_depth_limit == 0 {\n        return;\n    }\n    remaining_depth_limit -= 1;\n\n    match json_val.as_value() {\n        ReferenceValue::Leaf(leaf) => match leaf {\n            ReferenceValueLeaf::Null => {} // TODO: Handle null\n            ReferenceValueLeaf::Str(val) => {\n                if let Some(text_analyzer) = tokenizer.as_mut() {\n                    let mut token_stream = text_analyzer.token_stream(val);\n                    token_stream.process(&mut |token| {\n                        columnar_writer.record_str(doc, json_path_writer.as_str(), &token.text);\n                    })\n                } else {\n                    columnar_writer.record_str(doc, json_path_writer.as_str(), val);\n                }\n            }\n            ReferenceValueLeaf::U64(val) => {\n                columnar_writer.record_numerical(\n                    doc,\n                    json_path_writer.as_str(),\n                    NumericalValue::from(val),\n                );\n            }\n            ReferenceValueLeaf::I64(val) => {\n                columnar_writer.record_numerical(\n                    doc,\n                    json_path_writer.as_str(),\n                    NumericalValue::from(val),\n                );\n            }\n            ReferenceValueLeaf::F64(val) => {\n                columnar_writer.record_numerical(\n                    doc,\n                    json_path_writer.as_str(),\n                    NumericalValue::from(val),\n                );\n            }\n            ReferenceValueLeaf::Bool(val) => {\n                columnar_writer.record_bool(doc, json_path_writer.as_str(), val);\n            }\n            ReferenceValueLeaf::Date(val) => {\n                columnar_writer.record_datetime(doc, json_path_writer.as_str(), val);\n            }\n            ReferenceValueLeaf::Facet(_) => {\n                unimplemented!(\"Facet support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::Bytes(_) => {\n                // TODO: This can be re added once it is added to the JSON Utils section as well.\n                // columnar_writer.record_bytes(doc, json_path_writer.as_str(), val);\n                unimplemented!(\"Bytes support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::IpAddr(_) => {\n                unimplemented!(\"IP address support in dynamic fields is not yet implemented\")\n            }\n            ReferenceValueLeaf::PreTokStr(_) => {\n                unimplemented!(\n                    \"Pre-tokenized string support in dynamic fields is not yet implemented\"\n                )\n            }\n        },\n        ReferenceValue::Array(elements) => {\n            for el in elements {\n                record_json_value_to_columnar_writer(\n                    doc,\n                    el,\n                    remaining_depth_limit,\n                    json_path_writer,\n                    columnar_writer,\n                    tokenizer,\n                );\n            }\n        }\n        ReferenceValue::Object(object) => {\n            record_json_obj_to_columnar_writer::<V>(\n                doc,\n                object,\n                remaining_depth_limit,\n                json_path_writer,\n                columnar_writer,\n                tokenizer,\n            );\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use columnar::{Column, ColumnarReader, ColumnarWriter, StrColumn};\n    use common::JsonPathWriter;\n\n    use super::record_json_value_to_columnar_writer;\n    use crate::fastfield::writer::JSON_DEPTH_LIMIT;\n    use crate::DocId;\n\n    fn test_columnar_from_jsons_aux(\n        json_docs: &[serde_json::Value],\n        expand_dots: bool,\n    ) -> ColumnarReader {\n        let mut columnar_writer = ColumnarWriter::default();\n        let mut json_path = JsonPathWriter::default();\n        json_path.set_expand_dots(expand_dots);\n        for (doc, json_doc) in json_docs.iter().enumerate() {\n            record_json_value_to_columnar_writer(\n                doc as u32,\n                json_doc,\n                JSON_DEPTH_LIMIT,\n                &mut json_path,\n                &mut columnar_writer,\n                &mut None,\n            );\n        }\n        let mut buffer = Vec::new();\n        columnar_writer\n            .serialize(json_docs.len() as DocId, &mut buffer)\n            .unwrap();\n        ColumnarReader::open(buffer).unwrap()\n    }\n\n    #[test]\n    fn test_json_fastfield_record_simple() {\n        let json_doc = serde_json::json!({\n            \"float\": 1.02,\n            \"text\": \"hello happy tax payer\",\n            \"nested\": {\"child\": 3, \"child2\": 5},\n            \"arr\": [\"hello\", \"happy\", \"tax\", \"payer\"]\n        });\n        let columnar_reader = test_columnar_from_jsons_aux(&[json_doc], false);\n        let columns = columnar_reader.list_columns().unwrap();\n        {\n            assert_eq!(columns[0].0, \"arr\");\n            let column_arr_opt: Option<StrColumn> = columns[0].1.open().unwrap().into();\n            assert!(column_arr_opt\n                .unwrap()\n                .term_ords(0)\n                .eq([1, 0, 3, 2].into_iter()));\n        }\n        {\n            assert_eq!(columns[1].0, \"float\");\n            let column_float_opt: Option<Column<f64>> = columns[1].1.open().unwrap().into();\n            assert!(column_float_opt\n                .unwrap()\n                .values_for_doc(0)\n                .eq([1.02f64].into_iter()));\n        }\n        {\n            assert_eq!(columns[2].0, \"nested\\u{1}child\");\n            let column_nest_child_opt: Option<Column<i64>> = columns[2].1.open().unwrap().into();\n            assert!(column_nest_child_opt\n                .unwrap()\n                .values_for_doc(0)\n                .eq([3].into_iter()));\n        }\n        {\n            assert_eq!(columns[3].0, \"nested\\u{1}child2\");\n            let column_nest_child2_opt: Option<Column<i64>> = columns[3].1.open().unwrap().into();\n            assert!(column_nest_child2_opt\n                .unwrap()\n                .values_for_doc(0)\n                .eq([5].into_iter()));\n        }\n        {\n            assert_eq!(columns[4].0, \"text\");\n            let column_text_opt: Option<StrColumn> = columns[4].1.open().unwrap().into();\n            assert!(column_text_opt.unwrap().term_ords(0).eq([0].into_iter()));\n        }\n    }\n\n    #[test]\n    fn test_json_fastfield_deep_obj() {\n        let json_doc = serde_json::json!(\n            {\"a\": {\"a\": {\"a\": {\"a\": {\"a\":\n            {\"a\": {\"a\": {\"a\": {\"a\": {\"a\":\n            {\"a\": {\"a\": {\"a\": {\"a\": {\"a\":\n            {\"a\": {\"a\": {\"a\": {\"depth_accepted\": 19, \"a\": {  \"depth_truncated\": 20}\n        }}}}}}}}}}}}}}}}}}});\n        let columnar_reader = test_columnar_from_jsons_aux(&[json_doc], false);\n        let columns = columnar_reader.list_columns().unwrap();\n        assert_eq!(columns.len(), 1);\n        assert!(columns[0].0.ends_with(\"a\\u{1}a\\u{1}a\\u{1}depth_accepted\"));\n    }\n\n    #[test]\n    fn test_json_fastfield_deep_arr() {\n        let json_doc = json!(\n        {\"obj\":\n        [[[[[,\n        [[[[[,\n        [[[[[,\n        [[18, [19, //< within limits\n        [20]]]]]]]]]]]]]]]]]]]});\n        let columnar_reader = test_columnar_from_jsons_aux(&[json_doc], false);\n        let columns = columnar_reader.list_columns().unwrap();\n        assert_eq!(columns.len(), 1);\n        assert_eq!(columns[0].0, \"obj\");\n        let dynamic_column = columns[0].1.open().unwrap();\n        let col: Option<Column<i64>> = dynamic_column.into();\n        let vals: Vec<i64> = col.unwrap().values_for_doc(0).collect();\n        assert_eq!(&vals, &[18, 19])\n    }\n\n    #[test]\n    fn test_json_fast_field_do_not_expand_dots() {\n        let json_doc = json!({\"field.with.dots\": {\"child.with.dot\": \"hello\"}});\n        let columnar_reader = test_columnar_from_jsons_aux(&[json_doc], false);\n        let columns = columnar_reader.list_columns().unwrap();\n        assert_eq!(columns.len(), 1);\n        assert_eq!(columns[0].0, \"field.with.dots\\u{1}child.with.dot\");\n    }\n\n    #[test]\n    fn test_json_fast_field_expand_dots() {\n        let json_doc = json!({\"field.with.dots\": {\"child.with.dot\": \"hello\"}});\n        let columnar_reader = test_columnar_from_jsons_aux(&[json_doc], true);\n        let columns = columnar_reader.list_columns().unwrap();\n        assert_eq!(columns.len(), 1);\n        assert_eq!(\n            columns[0].0,\n            \"field\\u{1}with\\u{1}dots\\u{1}child\\u{1}with\\u{1}dot\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/fieldnorm/code.rs",
    "content": "#[inline]\npub fn id_to_fieldnorm(id: u8) -> u32 {\n    FIELD_NORMS_TABLE[id as usize]\n}\n\n#[inline]\npub fn fieldnorm_to_id(fieldnorm: u32) -> u8 {\n    FIELD_NORMS_TABLE\n        .binary_search(&fieldnorm)\n        .unwrap_or_else(|idx| idx - 1) as u8\n}\n\npub const FIELD_NORMS_TABLE: [u32; 256] = [\n    0,\n    1,\n    2,\n    3,\n    4,\n    5,\n    6,\n    7,\n    8,\n    9,\n    10,\n    11,\n    12,\n    13,\n    14,\n    15,\n    16,\n    17,\n    18,\n    19,\n    20,\n    21,\n    22,\n    23,\n    24,\n    25,\n    26,\n    27,\n    28,\n    29,\n    30,\n    31,\n    32,\n    33,\n    34,\n    35,\n    36,\n    37,\n    38,\n    39,\n    40,\n    42,\n    44,\n    46,\n    48,\n    50,\n    52,\n    54,\n    56,\n    60,\n    64,\n    68,\n    72,\n    76,\n    80,\n    84,\n    88,\n    96,\n    104,\n    112,\n    120,\n    128,\n    136,\n    144,\n    152,\n    168,\n    184,\n    200,\n    216,\n    232,\n    248,\n    264,\n    280,\n    312,\n    344,\n    376,\n    408,\n    440,\n    472,\n    504,\n    536,\n    600,\n    664,\n    728,\n    792,\n    856,\n    920,\n    984,\n    1_048,\n    1_176,\n    1_304,\n    1_432,\n    1_560,\n    1_688,\n    1_816,\n    1_944,\n    2_072,\n    2_328,\n    2_584,\n    2_840,\n    3_096,\n    3_352,\n    3_608,\n    3_864,\n    4_120,\n    4_632,\n    5_144,\n    5_656,\n    6_168,\n    6_680,\n    7_192,\n    7_704,\n    8_216,\n    9_240,\n    10_264,\n    11_288,\n    12_312,\n    13_336,\n    14_360,\n    15_384,\n    16_408,\n    18_456,\n    20_504,\n    22_552,\n    24_600,\n    26_648,\n    28_696,\n    30_744,\n    32_792,\n    36_888,\n    40_984,\n    45_080,\n    49_176,\n    53_272,\n    57_368,\n    61_464,\n    65_560,\n    73_752,\n    81_944,\n    90_136,\n    98_328,\n    106_520,\n    114_712,\n    122_904,\n    131_096,\n    147_480,\n    163_864,\n    180_248,\n    196_632,\n    213_016,\n    229_400,\n    245_784,\n    262_168,\n    294_936,\n    327_704,\n    360_472,\n    393_240,\n    426_008,\n    458_776,\n    491_544,\n    524_312,\n    589_848,\n    655_384,\n    720_920,\n    786_456,\n    851_992,\n    917_528,\n    983_064,\n    1_048_600,\n    1_179_672,\n    1_310_744,\n    1_441_816,\n    1_572_888,\n    1_703_960,\n    1_835_032,\n    1_966_104,\n    2_097_176,\n    2_359_320,\n    2_621_464,\n    2_883_608,\n    3_145_752,\n    3_407_896,\n    3_670_040,\n    3_932_184,\n    4_194_328,\n    4_718_616,\n    5_242_904,\n    5_767_192,\n    6_291_480,\n    6_815_768,\n    7_340_056,\n    7_864_344,\n    8_388_632,\n    9_437_208,\n    10_485_784,\n    11_534_360,\n    12_582_936,\n    13_631_512,\n    14_680_088,\n    15_728_664,\n    16_777_240,\n    18_874_392,\n    20_971_544,\n    23_068_696,\n    25_165_848,\n    27_263_000,\n    29_360_152,\n    31_457_304,\n    33_554_456,\n    37_748_760,\n    41_943_064,\n    46_137_368,\n    50_331_672,\n    54_525_976,\n    58_720_280,\n    62_914_584,\n    67_108_888,\n    75_497_496,\n    83_886_104,\n    92_274_712,\n    100_663_320,\n    109_051_928,\n    117_440_536,\n    125_829_144,\n    134_217_752,\n    150_994_968,\n    167_772_184,\n    184_549_400,\n    201_326_616,\n    218_103_832,\n    234_881_048,\n    251_658_264,\n    268_435_480,\n    301_989_912,\n    335_544_344,\n    369_098_776,\n    402_653_208,\n    436_207_640,\n    469_762_072,\n    503_316_504,\n    536_870_936,\n    603_979_800,\n    671_088_664,\n    738_197_528,\n    805_306_392,\n    872_415_256,\n    939_524_120,\n    1_006_632_984,\n    1_073_741_848,\n    1_207_959_576,\n    1_342_177_304,\n    1_476_395_032,\n    1_610_612_760,\n    1_744_830_488,\n    1_879_048_216,\n    2_013_265_944,\n];\n\n#[cfg(test)]\nmod tests {\n\n    use super::{fieldnorm_to_id, id_to_fieldnorm, FIELD_NORMS_TABLE};\n\n    #[test]\n    fn test_decode_code() {\n        assert_eq!(fieldnorm_to_id(0), 0);\n        assert_eq!(fieldnorm_to_id(1), 1);\n        for i in 0..41 {\n            assert_eq!(fieldnorm_to_id(i), i as u8);\n        }\n        assert_eq!(fieldnorm_to_id(41), 40);\n        assert_eq!(fieldnorm_to_id(42), 41);\n        for id in 43..256 {\n            let field_norm = FIELD_NORMS_TABLE[id];\n            assert_eq!(id_to_fieldnorm(id as u8), field_norm);\n            assert_eq!(fieldnorm_to_id(field_norm), id as u8);\n            assert_eq!(fieldnorm_to_id(field_norm - 1), id as u8 - 1);\n            assert_eq!(fieldnorm_to_id(field_norm + 1), id as u8);\n        }\n    }\n\n    #[test]\n    fn test_u32_max() {\n        assert_eq!(fieldnorm_to_id(u32::MAX), u8::MAX);\n    }\n\n    #[test]\n    fn test_fieldnorm_byte() {\n        // const expression are not really a thing\n        // yet... Therefore we do things the other way around.\n\n        // The array is defined as a const,\n        // and we check in the unit test that the const\n        // value is matching the logic.\n        const IDENTITY_PART: u8 = 24u8;\n        fn decode_field_norm_exp_part(b: u8) -> u32 {\n            let bits = (b & 0b00000111) as u32;\n            let shift = b >> 3;\n            if shift == 0 {\n                bits\n            } else {\n                (bits | 8u32) << ((shift - 1u8) as u32)\n            }\n        }\n        fn decode_fieldnorm_byte(b: u8) -> u32 {\n            if b < IDENTITY_PART {\n                b as u32\n            } else {\n                (IDENTITY_PART as u32) + decode_field_norm_exp_part(b - IDENTITY_PART)\n            }\n        }\n        for i in 0..256 {\n            assert_eq!(FIELD_NORMS_TABLE[i], decode_fieldnorm_byte(i as u8));\n        }\n    }\n}\n"
  },
  {
    "path": "src/fieldnorm/mod.rs",
    "content": "//! The fieldnorm represents the length associated with\n//! a given Field of a given document.\n//!\n//! This metric is important to compute the score of a\n//! document: a document having a query word in one of its short fields\n//! (e.g. title)  is likely to be more relevant than in one of its longer field\n//! (e.g. body).\n//!\n//! It encodes `fieldnorm` on one byte with some precision loss,\n//! using the exact same scheme as Lucene. Each value is placed on a log-scale\n//! that takes values from `0` to `255`.\n//!\n//! A value on this scale is identified by a `fieldnorm_id`.\n//! Apart from compression, this scale also makes it possible to\n//! precompute computationally expensive functions of the fieldnorm\n//! in a very short array.\n//!\n//! This trick is used by the Bm25 similarity.\nmod code;\nmod reader;\nmod serializer;\nmod writer;\n\nuse self::code::{fieldnorm_to_id, id_to_fieldnorm};\npub use self::reader::{FieldNormReader, FieldNormReaders};\npub use self::serializer::FieldNormsSerializer;\npub use self::writer::FieldNormsWriter;\n\n#[cfg(test)]\nmod tests {\n    use std::path::Path;\n\n    use once_cell::sync::Lazy;\n\n    use crate::directory::{CompositeFile, Directory, RamDirectory, WritePtr};\n    use crate::fieldnorm::{FieldNormReader, FieldNormsSerializer, FieldNormsWriter};\n    use crate::query::{EnableScoring, Query, TermQuery};\n    use crate::schema::{\n        Field, IndexRecordOption, Schema, TextFieldIndexing, TextOptions, STORED, TEXT,\n    };\n    use crate::{Index, Term, TERMINATED};\n\n    pub static SCHEMA: Lazy<Schema> = Lazy::new(|| {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"field\", STORED);\n        schema_builder.add_text_field(\"txt_field\", TEXT);\n        schema_builder.add_text_field(\n            \"str_field\",\n            TextOptions::default().set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_index_option(IndexRecordOption::Basic)\n                    .set_fieldnorms(false),\n            ),\n        );\n        schema_builder.build()\n    });\n\n    pub static FIELD: Lazy<Field> = Lazy::new(|| SCHEMA.get_field(\"field\").unwrap());\n    pub static TXT_FIELD: Lazy<Field> = Lazy::new(|| SCHEMA.get_field(\"txt_field\").unwrap());\n    pub static STR_FIELD: Lazy<Field> = Lazy::new(|| SCHEMA.get_field(\"str_field\").unwrap());\n\n    #[test]\n    #[should_panic(expected = \"Cannot register a given fieldnorm twice\")]\n    pub fn test_should_panic_when_recording_fieldnorm_twice_for_same_doc() {\n        let mut fieldnorm_writers = FieldNormsWriter::for_schema(&SCHEMA);\n        fieldnorm_writers.record(0u32, *TXT_FIELD, 5);\n        fieldnorm_writers.record(0u32, *TXT_FIELD, 3);\n    }\n\n    #[test]\n    pub fn test_fieldnorm() -> crate::Result<()> {\n        let path = Path::new(\"test\");\n        let directory: RamDirectory = RamDirectory::create();\n        {\n            let write: WritePtr = directory.open_write(Path::new(\"test\"))?;\n            let serializer = FieldNormsSerializer::from_write(write)?;\n            let mut fieldnorm_writers = FieldNormsWriter::for_schema(&SCHEMA);\n            fieldnorm_writers.record(2u32, *TXT_FIELD, 5);\n            fieldnorm_writers.record(3u32, *TXT_FIELD, 3);\n            fieldnorm_writers.serialize(serializer)?;\n        }\n        let file = directory.open_read(path)?;\n        {\n            let fields_composite = CompositeFile::open(&file)?;\n            assert!(fields_composite.open_read(*FIELD).is_none());\n            assert!(fields_composite.open_read(*STR_FIELD).is_none());\n            let data = fields_composite.open_read(*TXT_FIELD).unwrap();\n            let fieldnorm_reader = FieldNormReader::open(data)?;\n            assert_eq!(fieldnorm_reader.fieldnorm(0u32), 0u32);\n            assert_eq!(fieldnorm_reader.fieldnorm(1u32), 0u32);\n            assert_eq!(fieldnorm_reader.fieldnorm(2u32), 5u32);\n            assert_eq!(fieldnorm_reader.fieldnorm(3u32), 3u32);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_fieldnorm_disabled() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_options = TextOptions::default()\n            .set_indexing_options(TextFieldIndexing::default().set_fieldnorms(false));\n        let text = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(text=>\"hello\"))?;\n        writer.add_document(doc!(text=>\"hello hello hello\"))?;\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query = TermQuery::new(\n            Term::from_field_text(text, \"hello\"),\n            IndexRecordOption::WithFreqs,\n        );\n        let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n        let mut scorer = weight.scorer(searcher.segment_reader(0), 1.0f32)?;\n        assert_eq!(scorer.doc(), 0);\n        assert!((scorer.score() - 0.22920431).abs() < 0.001f32);\n        assert_eq!(scorer.advance(), 1);\n        assert_eq!(scorer.doc(), 1);\n        assert!((scorer.score() - 0.22920431).abs() < 0.001f32);\n        assert_eq!(scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_fieldnorm_enabled() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_options = TextOptions::default()\n            .set_indexing_options(TextFieldIndexing::default().set_fieldnorms(true));\n        let text = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(text=>\"hello\"))?;\n        writer.add_document(doc!(text=>\"hello hello hello\"))?;\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query = TermQuery::new(\n            Term::from_field_text(text, \"hello\"),\n            IndexRecordOption::WithFreqs,\n        );\n        let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n        let mut scorer = weight.scorer(searcher.segment_reader(0), 1.0f32)?;\n        assert_eq!(scorer.doc(), 0);\n        assert!((scorer.score() - 0.22920431).abs() < 0.001f32);\n        assert_eq!(scorer.advance(), 1);\n        assert_eq!(scorer.doc(), 1);\n        assert!((scorer.score() - 0.15136132).abs() < 0.001f32);\n        assert_eq!(scorer.advance(), TERMINATED);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/fieldnorm/reader.rs",
    "content": "use std::sync::Arc;\n\nuse super::{fieldnorm_to_id, id_to_fieldnorm};\nuse crate::directory::{CompositeFile, FileSlice, OwnedBytes};\nuse crate::schema::{Field, Schema};\nuse crate::space_usage::PerFieldSpaceUsage;\nuse crate::DocId;\n\n/// Reader for the fieldnorm (for each document, the number of tokens indexed in the\n/// field) of all indexed fields in the index.\n///\n/// Each fieldnorm is approximately compressed over one byte. We refer to this byte as\n/// `fieldnorm_id`.\n/// The mapping from `fieldnorm` to `fieldnorm_id` is given by monotonic.\n#[derive(Clone)]\npub struct FieldNormReaders {\n    data: Arc<CompositeFile>,\n}\n\nimpl FieldNormReaders {\n    /// Creates a field norm reader.\n    pub fn open(file: FileSlice) -> crate::Result<FieldNormReaders> {\n        let data = CompositeFile::open(&file)?;\n        Ok(FieldNormReaders {\n            data: Arc::new(data),\n        })\n    }\n\n    /// Returns the FieldNormReader for a specific field.\n    pub fn get_field(&self, field: Field) -> crate::Result<Option<FieldNormReader>> {\n        if let Some(file) = self.data.open_read(field) {\n            let fieldnorm_reader = FieldNormReader::open(file)?;\n            Ok(Some(fieldnorm_reader))\n        } else {\n            Ok(None)\n        }\n    }\n\n    /// Return a break down of the space usage per field.\n    pub fn space_usage(&self, schema: &Schema) -> PerFieldSpaceUsage {\n        self.data.space_usage(schema)\n    }\n\n    /// Returns a handle to inner file\n    pub fn get_inner_file(&self) -> Arc<CompositeFile> {\n        self.data.clone()\n    }\n}\n\n/// Reads the fieldnorm associated with a document.\n///\n/// The [fieldnorm](FieldNormReader::fieldnorm) represents the length associated with\n/// a given Field of a given document.\n#[derive(Clone)]\npub struct FieldNormReader(ReaderImplEnum);\n\nimpl From<ReaderImplEnum> for FieldNormReader {\n    fn from(reader_enum: ReaderImplEnum) -> FieldNormReader {\n        FieldNormReader(reader_enum)\n    }\n}\n\n#[derive(Clone)]\nenum ReaderImplEnum {\n    FromData(OwnedBytes),\n    Const {\n        num_docs: u32,\n        fieldnorm_id: u8,\n        fieldnorm: u32,\n    },\n}\n\nimpl FieldNormReader {\n    /// Creates a `FieldNormReader` with a constant fieldnorm.\n    ///\n    /// The fieldnorm will be subjected to compression as if it was coming\n    /// from an array-backed fieldnorm reader.\n    pub fn constant(num_docs: u32, fieldnorm: u32) -> FieldNormReader {\n        let fieldnorm_id = fieldnorm_to_id(fieldnorm);\n        let fieldnorm = id_to_fieldnorm(fieldnorm_id);\n        ReaderImplEnum::Const {\n            num_docs,\n            fieldnorm_id,\n            fieldnorm,\n        }\n        .into()\n    }\n\n    /// Opens a field norm reader given its file.\n    pub fn open(fieldnorm_file: FileSlice) -> crate::Result<Self> {\n        let data = fieldnorm_file.read_bytes()?;\n        Ok(FieldNormReader::new(data))\n    }\n\n    fn new(data: OwnedBytes) -> Self {\n        ReaderImplEnum::FromData(data).into()\n    }\n\n    /// Returns the number of documents in this segment.\n    pub fn num_docs(&self) -> u32 {\n        match &self.0 {\n            ReaderImplEnum::FromData(data) => data.len() as u32,\n            ReaderImplEnum::Const { num_docs, .. } => *num_docs,\n        }\n    }\n\n    /// Returns the `fieldnorm` associated with a doc id.\n    /// The fieldnorm is a value approximating the number\n    /// of tokens in a given field of the `doc_id`.\n    ///\n    /// It is imprecise, and equal or lower than\n    /// the actual number of tokens.\n    ///\n    /// The fieldnorm is effectively decoded from the\n    /// `fieldnorm_id` by doing a simple table lookup.\n    pub fn fieldnorm(&self, doc_id: DocId) -> u32 {\n        match &self.0 {\n            ReaderImplEnum::FromData(data) => {\n                let fieldnorm_id = data.as_slice()[doc_id as usize];\n                id_to_fieldnorm(fieldnorm_id)\n            }\n            ReaderImplEnum::Const { fieldnorm, .. } => *fieldnorm,\n        }\n    }\n\n    /// Returns the `fieldnorm_id` associated with a document.\n    #[inline]\n    pub fn fieldnorm_id(&self, doc_id: DocId) -> u8 {\n        match &self.0 {\n            ReaderImplEnum::FromData(data) => {\n                let fieldnorm_id = data.as_slice()[doc_id as usize];\n                fieldnorm_id\n            }\n            ReaderImplEnum::Const { fieldnorm_id, .. } => *fieldnorm_id,\n        }\n    }\n\n    /// Converts a `fieldnorm_id` into a fieldnorm.\n    #[inline]\n    pub fn id_to_fieldnorm(id: u8) -> u32 {\n        id_to_fieldnorm(id)\n    }\n\n    /// Converts a `fieldnorm` into a `fieldnorm_id`.\n    /// (This function is not injective).\n    #[inline]\n    pub fn fieldnorm_to_id(fieldnorm: u32) -> u8 {\n        fieldnorm_to_id(fieldnorm)\n    }\n\n    #[cfg(test)]\n    pub(crate) fn for_test(field_norms: &[u32]) -> FieldNormReader {\n        let field_norms_id = field_norms\n            .iter()\n            .cloned()\n            .map(FieldNormReader::fieldnorm_to_id)\n            .collect::<Vec<u8>>();\n        let field_norms_data = OwnedBytes::new(field_norms_id);\n        FieldNormReader::new(field_norms_data)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::fieldnorm::FieldNormReader;\n\n    #[test]\n    fn test_from_fieldnorms_array() {\n        let fieldnorms = &[1, 2, 3, 4, 1_000_000];\n        let fieldnorm_reader = FieldNormReader::for_test(fieldnorms);\n        assert_eq!(fieldnorm_reader.num_docs(), 5);\n        assert_eq!(fieldnorm_reader.fieldnorm(0), 1);\n        assert_eq!(fieldnorm_reader.fieldnorm(1), 2);\n        assert_eq!(fieldnorm_reader.fieldnorm(2), 3);\n        assert_eq!(fieldnorm_reader.fieldnorm(3), 4);\n        assert_eq!(fieldnorm_reader.fieldnorm(4), 983_064);\n    }\n\n    #[test]\n    fn test_const_fieldnorm_reader_small_fieldnorm_id() {\n        let fieldnorm_reader = FieldNormReader::constant(1_000_000u32, 10u32);\n        assert_eq!(fieldnorm_reader.num_docs(), 1_000_000u32);\n        assert_eq!(fieldnorm_reader.fieldnorm(0u32), 10u32);\n        assert_eq!(fieldnorm_reader.fieldnorm_id(0u32), 10u8);\n    }\n\n    #[test]\n    fn test_const_fieldnorm_reader_large_fieldnorm_id() {\n        let fieldnorm_reader = FieldNormReader::constant(1_000_000u32, 300u32);\n        assert_eq!(fieldnorm_reader.num_docs(), 1_000_000u32);\n        assert_eq!(fieldnorm_reader.fieldnorm(0u32), 280u32);\n        assert_eq!(fieldnorm_reader.fieldnorm_id(0u32), 72u8);\n    }\n}\n"
  },
  {
    "path": "src/fieldnorm/serializer.rs",
    "content": "use std::io;\nuse std::io::Write;\n\nuse crate::directory::{CompositeWrite, WritePtr};\nuse crate::schema::Field;\n\n/// The fieldnorms serializer is in charge of\n/// the serialization of field norms for all fields.\npub struct FieldNormsSerializer {\n    composite_write: CompositeWrite,\n}\n\nimpl FieldNormsSerializer {\n    /// Constructor\n    pub fn from_write(write: WritePtr) -> io::Result<FieldNormsSerializer> {\n        // just making room for the pointer to header.\n        let composite_write = CompositeWrite::wrap(write);\n        Ok(FieldNormsSerializer { composite_write })\n    }\n\n    /// Serialize the given field\n    pub fn serialize_field(&mut self, field: Field, fieldnorms_data: &[u8]) -> io::Result<()> {\n        let write = self.composite_write.for_field(field);\n        write.write_all(fieldnorms_data)?;\n        write.flush()?;\n        Ok(())\n    }\n\n    /// Clean up / flush / close\n    pub fn close(self) -> io::Result<()> {\n        self.composite_write.close()?;\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/fieldnorm/writer.rs",
    "content": "use std::cmp::Ordering;\nuse std::{io, iter};\n\nuse super::{fieldnorm_to_id, FieldNormsSerializer};\nuse crate::schema::{Field, Schema};\nuse crate::DocId;\n\n/// The `FieldNormsWriter` is in charge of tracking the fieldnorm byte\n/// of each document for each field with field norms.\n///\n/// `FieldNormsWriter` stores a `Vec<u8>` for each tracked field, using a\n/// byte per document per field.\npub struct FieldNormsWriter {\n    fieldnorms_buffers: Vec<Option<Vec<u8>>>,\n}\n\nimpl FieldNormsWriter {\n    /// Returns the fields that should have field norms computed\n    /// according to the given schema.\n    pub(crate) fn fields_with_fieldnorm(schema: &Schema) -> Vec<Field> {\n        schema\n            .fields()\n            .filter_map(|(field, field_entry)| {\n                if field_entry.is_indexed() && field_entry.has_fieldnorms() {\n                    Some(field)\n                } else {\n                    None\n                }\n            })\n            .collect::<Vec<_>>()\n    }\n\n    /// Initialize with state for tracking the field norm fields\n    /// specified in the schema.\n    pub fn for_schema(schema: &Schema) -> FieldNormsWriter {\n        let mut fieldnorms_buffers: Vec<Option<Vec<u8>>> = iter::repeat_with(|| None)\n            .take(schema.num_fields())\n            .collect();\n        for field in FieldNormsWriter::fields_with_fieldnorm(schema) {\n            fieldnorms_buffers[field.field_id() as usize] = Some(Vec::with_capacity(1_000));\n        }\n        FieldNormsWriter { fieldnorms_buffers }\n    }\n\n    /// The memory used inclusive childs\n    pub fn mem_usage(&self) -> usize {\n        self.fieldnorms_buffers\n            .iter()\n            .flatten()\n            .map(|buf| buf.capacity())\n            .sum()\n    }\n    /// Ensure that all documents in 0..max_doc have a byte associated with them\n    /// in each of the fieldnorm vectors.\n    ///\n    /// Will extend with 0-bytes for documents that have not been seen.\n    pub fn fill_up_to_max_doc(&mut self, max_doc: DocId) {\n        for fieldnorms_buffer_opt in self.fieldnorms_buffers.iter_mut() {\n            if let Some(fieldnorms_buffer) = fieldnorms_buffer_opt.as_mut() {\n                fieldnorms_buffer.resize(max_doc as usize, 0u8);\n            }\n        }\n    }\n\n    /// Set the fieldnorm byte for the given document for the given field.\n    ///\n    /// Will internally convert the u32 `fieldnorm` value to the appropriate byte\n    /// to approximate the field norm in less space.\n    ///\n    /// * doc       - the document id\n    /// * field     - the field being set\n    /// * fieldnorm - the number of terms present in document `doc` in field `field`\n    pub fn record(&mut self, doc: DocId, field: Field, fieldnorm: u32) {\n        if let Some(fieldnorm_buffer) = self\n            .fieldnorms_buffers\n            .get_mut(field.field_id() as usize)\n            .and_then(Option::as_mut)\n        {\n            match fieldnorm_buffer.len().cmp(&(doc as usize)) {\n                Ordering::Less => {\n                    // we fill intermediary `DocId` as  having a fieldnorm of 0.\n                    fieldnorm_buffer.resize(doc as usize, 0u8);\n                }\n                Ordering::Equal => {}\n                Ordering::Greater => {\n                    panic!(\"Cannot register a given fieldnorm twice\")\n                }\n            }\n            fieldnorm_buffer.push(fieldnorm_to_id(fieldnorm));\n        }\n    }\n\n    /// Serialize the seen fieldnorm values to the serializer for all fields.\n    pub fn serialize(&self, mut fieldnorms_serializer: FieldNormsSerializer) -> io::Result<()> {\n        for (field, fieldnorms_buffer) in self.fieldnorms_buffers.iter().enumerate().filter_map(\n            |(field_id, fieldnorms_buffer_opt)| {\n                fieldnorms_buffer_opt.as_ref().map(|fieldnorms_buffer| {\n                    (Field::from_field_id(field_id as u32), fieldnorms_buffer)\n                })\n            },\n        ) {\n            fieldnorms_serializer.serialize_field(field, fieldnorms_buffer)?;\n        }\n        fieldnorms_serializer.close()?;\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/functional_test.rs",
    "content": "use std::collections::HashSet;\n\nuse rand::{rng, Rng};\n\nuse crate::indexer::index_writer::MEMORY_BUDGET_NUM_BYTES_MIN;\nuse crate::schema::*;\nuse crate::{doc, schema, Index, IndexWriter, Searcher};\n\nfn check_index_content(searcher: &Searcher, vals: &[u64]) -> crate::Result<()> {\n    assert!(searcher.segment_readers().len() < 20);\n    assert_eq!(searcher.num_docs() as usize, vals.len());\n    for segment_reader in searcher.segment_readers() {\n        let store_reader = segment_reader.get_store_reader(1)?;\n        for doc_id in 0..segment_reader.max_doc() {\n            let _doc: TantivyDocument = store_reader.get(doc_id)?;\n        }\n    }\n    Ok(())\n}\n\n#[test]\n#[ignore]\nfn test_functional_store() -> crate::Result<()> {\n    let mut schema_builder = Schema::builder();\n\n    let id_field = schema_builder.add_u64_field(\"id\", INDEXED | STORED);\n    let schema = schema_builder.build();\n\n    let index = Index::create_in_ram(schema);\n    let reader = index.reader()?;\n\n    let mut rng = rng();\n\n    let mut index_writer: IndexWriter =\n        index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;\n\n    let mut doc_set: Vec<u64> = Vec::new();\n\n    let mut doc_id = 0u64;\n    for _iteration in 0..get_num_iterations() {\n        let num_docs: usize = rng.random_range(0..4);\n        if !doc_set.is_empty() {\n            let doc_to_remove_id = rng.random_range(0..doc_set.len());\n            let removed_doc_id = doc_set.swap_remove(doc_to_remove_id);\n            index_writer.delete_term(Term::from_field_u64(id_field, removed_doc_id));\n        }\n        for _ in 0..num_docs {\n            doc_set.push(doc_id);\n            index_writer.add_document(doc!(id_field=>doc_id))?;\n            doc_id += 1;\n        }\n        index_writer.commit()?;\n        reader.reload()?;\n        let searcher = reader.searcher();\n        check_index_content(&searcher, &doc_set)?;\n    }\n    Ok(())\n}\n\nfn get_num_iterations() -> usize {\n    std::env::var(\"NUM_FUNCTIONAL_TEST_ITERATIONS\")\n        .map(|str| str.parse().unwrap())\n        .unwrap_or(2000)\n}\n\nconst LOREM: &str = \"Doc Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod \\\n                     tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, \\\n                     quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo \\\n                     consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse \\\n                     cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat \\\n                     non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\";\nfn get_text() -> String {\n    use rand::seq::IndexedRandom;\n    let mut rng = rng();\n    let tokens: Vec<_> = LOREM.split(' ').collect();\n    let random_val = rng.random_range(0..20);\n\n    (0..random_val)\n        .map(|_| tokens.choose(&mut rng).unwrap())\n        .cloned()\n        .collect::<Vec<_>>()\n        .join(\" \")\n}\n\n#[test]\n#[ignore]\nfn test_functional_indexing_unsorted() -> crate::Result<()> {\n    let mut schema_builder = Schema::builder();\n\n    let id_field = schema_builder.add_u64_field(\"id\", INDEXED);\n    let multiples_field = schema_builder.add_u64_field(\"multiples\", INDEXED);\n    let text_field_options = TextOptions::default()\n        .set_indexing_options(\n            TextFieldIndexing::default()\n                .set_index_option(schema::IndexRecordOption::WithFreqsAndPositions),\n        )\n        .set_stored();\n    let text_field = schema_builder.add_text_field(\"text_field\", text_field_options);\n    let schema = schema_builder.build();\n\n    let index = Index::create_from_tempdir(schema)?;\n    let reader = index.reader()?;\n\n    let mut rng = rng();\n\n    let mut index_writer: IndexWriter =\n        index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;\n\n    let mut committed_docs: HashSet<u64> = HashSet::new();\n    let mut uncommitted_docs: HashSet<u64> = HashSet::new();\n\n    for _ in 0..get_num_iterations() {\n        let random_val = rng.random_range(0..20);\n        if random_val == 0 {\n            index_writer.commit()?;\n            committed_docs.extend(&uncommitted_docs);\n            uncommitted_docs.clear();\n            reader.reload()?;\n            let searcher = reader.searcher();\n            // check that everything is correct.\n            check_index_content(\n                &searcher,\n                &committed_docs.iter().cloned().collect::<Vec<u64>>(),\n            )?;\n        } else if committed_docs.remove(&random_val) || uncommitted_docs.remove(&random_val) {\n            let doc_id_term = Term::from_field_u64(id_field, random_val);\n            index_writer.delete_term(doc_id_term);\n        } else {\n            uncommitted_docs.insert(random_val);\n            let mut doc = TantivyDocument::new();\n            doc.add_u64(id_field, random_val);\n            for i in 1u64..10u64 {\n                doc.add_u64(multiples_field, random_val * i);\n            }\n            doc.add_text(text_field, get_text());\n            index_writer.add_document(doc)?;\n        }\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "src/future_result.rs",
    "content": "use std::future::Future;\nuse std::pin::Pin;\nuse std::task::Poll;\n\nuse crate::TantivyError;\n\n/// `FutureResult` is a handle that makes it possible to wait for the completion\n/// of an ongoing task.\n///\n/// Contrary to some `Future`, it does not need to be polled for the task to\n/// progress. Dropping the `FutureResult` does not cancel the task being executed\n/// either.\n///\n/// - In a sync context, you can call `FutureResult::wait()`. The function does not rely on\n///   `block_on`.\n/// - In an async context, you can call simply use `FutureResult` as a future.\npub struct FutureResult<T> {\n    inner: Inner<T>,\n}\n\nenum Inner<T> {\n    FailedBeforeStart(Option<TantivyError>),\n    InProgress {\n        receiver: oneshot::Receiver<crate::Result<T>>,\n        error_msg_if_failure: &'static str,\n    },\n}\n\nimpl<T> From<TantivyError> for FutureResult<T> {\n    fn from(err: TantivyError) -> Self {\n        FutureResult {\n            inner: Inner::FailedBeforeStart(Some(err)),\n        }\n    }\n}\n\nimpl<T> FutureResult<T> {\n    pub(crate) fn create(\n        error_msg_if_failure: &'static str,\n    ) -> (Self, oneshot::Sender<crate::Result<T>>) {\n        let (sender, receiver) = oneshot::channel();\n        let inner: Inner<T> = Inner::InProgress {\n            receiver,\n            error_msg_if_failure,\n        };\n        (FutureResult { inner }, sender)\n    }\n\n    /// Blocks until the scheduled result is available.\n    ///\n    /// In an async context, you should simply use `ScheduledResult` as a future.\n    pub fn wait(self) -> crate::Result<T> {\n        match self.inner {\n            Inner::FailedBeforeStart(err) => Err(err.unwrap()),\n            Inner::InProgress {\n                receiver,\n                error_msg_if_failure,\n            } => receiver.recv().unwrap_or_else(|_| {\n                Err(crate::TantivyError::SystemError(\n                    error_msg_if_failure.to_string(),\n                ))\n            }),\n        }\n    }\n}\n\nimpl<T> Future for FutureResult<T> {\n    type Output = crate::Result<T>;\n\n    fn poll(self: Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> Poll<Self::Output> {\n        unsafe {\n            match &mut Pin::get_unchecked_mut(self).inner {\n                Inner::FailedBeforeStart(err) => Poll::Ready(Err(err.take().unwrap())),\n                Inner::InProgress {\n                    receiver,\n                    error_msg_if_failure,\n                } => match Future::poll(Pin::new_unchecked(receiver), cx) {\n                    Poll::Ready(oneshot_res) => {\n                        let res = oneshot_res.unwrap_or_else(|_| {\n                            Err(crate::TantivyError::SystemError(\n                                error_msg_if_failure.to_string(),\n                            ))\n                        });\n                        Poll::Ready(res)\n                    }\n                    Poll::Pending => Poll::Pending,\n                },\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use futures::executor::block_on;\n\n    use super::FutureResult;\n    use crate::TantivyError;\n\n    #[test]\n    fn test_scheduled_result_failed_to_schedule() {\n        let scheduled_result: FutureResult<()> = FutureResult::from(TantivyError::Poisoned);\n        let res = block_on(scheduled_result);\n        assert!(matches!(res, Err(TantivyError::Poisoned)));\n    }\n\n    #[test]\n\n    fn test_scheduled_result_error() {\n        let (scheduled_result, tx): (FutureResult<()>, _) = FutureResult::create(\"failed\");\n        drop(tx);\n        let res = block_on(scheduled_result);\n        assert!(matches!(res, Err(TantivyError::SystemError(_))));\n    }\n\n    #[test]\n    fn test_scheduled_result_sent_success() {\n        let (scheduled_result, tx): (FutureResult<u64>, _) = FutureResult::create(\"failed\");\n        tx.send(Ok(2u64)).unwrap();\n        assert_eq!(block_on(scheduled_result).unwrap(), 2u64);\n    }\n\n    #[test]\n    fn test_scheduled_result_sent_error() {\n        let (scheduled_result, tx): (FutureResult<u64>, _) = FutureResult::create(\"failed\");\n        tx.send(Err(TantivyError::Poisoned)).unwrap();\n        let res = block_on(scheduled_result);\n        assert!(matches!(res, Err(TantivyError::Poisoned)));\n    }\n}\n"
  },
  {
    "path": "src/index/index.rs",
    "content": "use std::collections::HashSet;\nuse std::fmt;\n#[cfg(feature = \"mmap\")]\nuse std::path::Path;\nuse std::path::PathBuf;\nuse std::thread::available_parallelism;\n\nuse super::segment::Segment;\nuse super::segment_reader::merge_field_meta_data;\nuse super::{FieldMetadata, IndexSettings};\nuse crate::core::{Executor, META_FILEPATH};\nuse crate::directory::error::OpenReadError;\n#[cfg(feature = \"mmap\")]\nuse crate::directory::MmapDirectory;\nuse crate::directory::{Directory, ManagedDirectory, RamDirectory, INDEX_WRITER_LOCK};\nuse crate::error::{DataCorruption, TantivyError};\nuse crate::index::{IndexMeta, SegmentId, SegmentMeta, SegmentMetaInventory};\nuse crate::indexer::index_writer::{\n    IndexWriterOptions, MAX_NUM_THREAD, MEMORY_BUDGET_NUM_BYTES_MIN,\n};\nuse crate::indexer::segment_updater::save_metas;\nuse crate::indexer::{IndexWriter, SingleSegmentIndexWriter};\nuse crate::reader::{IndexReader, IndexReaderBuilder};\nuse crate::schema::document::Document;\nuse crate::schema::{Field, FieldType, Schema};\nuse crate::tokenizer::{TextAnalyzer, TokenizerManager};\nuse crate::SegmentReader;\n\nfn load_metas(\n    directory: &dyn Directory,\n    inventory: &SegmentMetaInventory,\n) -> crate::Result<IndexMeta> {\n    let meta_data = directory.atomic_read(&META_FILEPATH)?;\n    let meta_string = String::from_utf8(meta_data).map_err(|_utf8_err| {\n        error!(\"Meta data is not valid utf8.\");\n        DataCorruption::new(\n            META_FILEPATH.to_path_buf(),\n            \"Meta file does not contain valid utf8 file.\".to_string(),\n        )\n    })?;\n    IndexMeta::deserialize(&meta_string, inventory)\n        .map_err(|e| {\n            DataCorruption::new(\n                META_FILEPATH.to_path_buf(),\n                format!(\"Meta file cannot be deserialized. {e:?}. Content: {meta_string:?}\"),\n            )\n        })\n        .map_err(From::from)\n}\n\n/// Save the index meta file.\n/// This operation is atomic :\n/// Either\n/// - it fails, in which case an error is returned, and the `meta.json` remains untouched,\n/// - it succeeds, and `meta.json` is written and flushed.\n///\n/// This method is not part of tantivy's public API\nfn save_new_metas(\n    schema: Schema,\n    index_settings: IndexSettings,\n    directory: &dyn Directory,\n) -> crate::Result<()> {\n    save_metas(\n        &IndexMeta {\n            index_settings,\n            segments: Vec::new(),\n            schema,\n            opstamp: 0u64,\n            payload: None,\n        },\n        directory,\n    )?;\n    directory.sync_directory()?;\n    Ok(())\n}\n\n/// IndexBuilder can be used to create an index.\n///\n/// Use in conjunction with [`SchemaBuilder`][crate::schema::SchemaBuilder].\n/// Global index settings can be configured with [`IndexSettings`].\n///\n/// # Examples\n///\n/// ```\n/// use tantivy::schema::*;\n/// use tantivy::{Index, IndexSettings};\n///\n/// let mut schema_builder = Schema::builder();\n/// let id_field = schema_builder.add_text_field(\"id\", STRING);\n/// let title_field = schema_builder.add_text_field(\"title\", TEXT);\n/// let body_field = schema_builder.add_text_field(\"body\", TEXT);\n/// let number_field = schema_builder.add_u64_field(\n///     \"number\",\n///     NumericOptions::default().set_fast(),\n/// );\n///\n/// let schema = schema_builder.build();\n/// let settings = IndexSettings{\n///     docstore_blocksize: 100_000,\n///     ..Default::default()\n/// };\n/// let index = Index::builder().schema(schema).settings(settings).create_in_ram();\n/// ```\npub struct IndexBuilder {\n    schema: Option<Schema>,\n    index_settings: IndexSettings,\n    tokenizer_manager: TokenizerManager,\n    fast_field_tokenizer_manager: TokenizerManager,\n}\nimpl Default for IndexBuilder {\n    fn default() -> Self {\n        IndexBuilder::new()\n    }\n}\nimpl IndexBuilder {\n    /// Creates a new `IndexBuilder`\n    pub fn new() -> Self {\n        Self {\n            schema: None,\n            index_settings: IndexSettings::default(),\n            tokenizer_manager: TokenizerManager::default(),\n            fast_field_tokenizer_manager: TokenizerManager::default(),\n        }\n    }\n\n    /// Set the settings\n    #[must_use]\n    pub fn settings(mut self, settings: IndexSettings) -> Self {\n        self.index_settings = settings;\n        self\n    }\n\n    /// Set the schema\n    #[must_use]\n    pub fn schema(mut self, schema: Schema) -> Self {\n        self.schema = Some(schema);\n        self\n    }\n\n    /// Set the tokenizers.\n    pub fn tokenizers(mut self, tokenizers: TokenizerManager) -> Self {\n        self.tokenizer_manager = tokenizers;\n        self\n    }\n\n    /// Set the fast field tokenizers.\n    pub fn fast_field_tokenizers(mut self, tokenizers: TokenizerManager) -> Self {\n        self.fast_field_tokenizer_manager = tokenizers;\n        self\n    }\n\n    /// Creates a new index using the [`RamDirectory`].\n    ///\n    /// The index will be allocated in anonymous memory.\n    /// This is useful for indexing small set of documents\n    /// for instances like unit test or temporary in memory index.\n    pub fn create_in_ram(self) -> Result<Index, TantivyError> {\n        let ram_directory = RamDirectory::create();\n        self.create(ram_directory)\n    }\n\n    /// Creates a new index in a given filepath.\n    /// The index will use the [`MmapDirectory`].\n    ///\n    /// If a previous index was in this directory, it returns an\n    /// [`TantivyError::IndexAlreadyExists`] error.\n    #[cfg(feature = \"mmap\")]\n    pub fn create_in_dir<P: AsRef<Path>>(self, directory_path: P) -> crate::Result<Index> {\n        let mmap_directory: Box<dyn Directory> = Box::new(MmapDirectory::open(directory_path)?);\n        if Index::exists(&*mmap_directory)? {\n            return Err(TantivyError::IndexAlreadyExists);\n        }\n        self.create(mmap_directory)\n    }\n\n    /// Dragons ahead!!!\n    ///\n    /// The point of this API is to let users create a simple index with a single segment\n    /// and without starting any thread.\n    ///\n    /// Do not use this method if you are not sure what you are doing.\n    ///\n    /// It expects an originally empty directory, and will not run any GC operation.\n    #[doc(hidden)]\n    pub fn single_segment_index_writer<D: Document>(\n        self,\n        dir: impl Into<Box<dyn Directory>>,\n        mem_budget: usize,\n    ) -> crate::Result<SingleSegmentIndexWriter<D>> {\n        let index = self.create(dir)?;\n        let index_simple_writer = SingleSegmentIndexWriter::new(index, mem_budget)?;\n        Ok(index_simple_writer)\n    }\n\n    /// Creates a new index in a temp directory.\n    ///\n    /// The index will use the [`MmapDirectory`] in a newly created directory.\n    /// The temp directory will be destroyed automatically when the [`Index`] object\n    /// is destroyed.\n    ///\n    /// The temp directory is only used for testing the [`MmapDirectory`].\n    /// For other unit tests, prefer the [`RamDirectory`], see:\n    /// [`IndexBuilder::create_in_ram()`].\n    #[cfg(feature = \"mmap\")]\n    pub fn create_from_tempdir(self) -> crate::Result<Index> {\n        let mmap_directory: Box<dyn Directory> = Box::new(MmapDirectory::create_from_tempdir()?);\n        self.create(mmap_directory)\n    }\n\n    fn get_expect_schema(&self) -> crate::Result<Schema> {\n        self.schema\n            .as_ref()\n            .cloned()\n            .ok_or(TantivyError::IndexBuilderMissingArgument(\"schema\"))\n    }\n\n    /// Opens or creates a new index in the provided directory\n    pub fn open_or_create<T: Into<Box<dyn Directory>>>(self, dir: T) -> crate::Result<Index> {\n        let dir: Box<dyn Directory> = dir.into();\n        if !Index::exists(&*dir)? {\n            return self.create(dir);\n        }\n        let mut index = Index::open(dir)?;\n        index.set_tokenizers(self.tokenizer_manager.clone());\n        if index.schema() == self.get_expect_schema()? {\n            Ok(index)\n        } else {\n            Err(TantivyError::SchemaError(\n                \"An index exists but the schema does not match.\".to_string(),\n            ))\n        }\n    }\n\n    fn validate(&self) -> crate::Result<()> {\n        if let Some(_schema) = self.schema.as_ref() {\n            Ok(())\n        } else {\n            Err(TantivyError::InvalidArgument(\n                \"no schema passed\".to_string(),\n            ))\n        }\n    }\n\n    /// Creates a new index given an implementation of the trait `Directory`.\n    ///\n    /// If a directory previously existed, it will be erased.\n    fn create<T: Into<Box<dyn Directory>>>(self, dir: T) -> crate::Result<Index> {\n        self.validate()?;\n        let dir = dir.into();\n        let directory = ManagedDirectory::wrap(dir)?;\n        save_new_metas(\n            self.get_expect_schema()?,\n            self.index_settings.clone(),\n            &directory,\n        )?;\n        let mut metas = IndexMeta::with_schema(self.get_expect_schema()?);\n        metas.index_settings = self.index_settings;\n        let mut index = Index::open_from_metas(directory, &metas, SegmentMetaInventory::default());\n        index.set_tokenizers(self.tokenizer_manager);\n        index.set_fast_field_tokenizers(self.fast_field_tokenizer_manager);\n        Ok(index)\n    }\n}\n\n/// Search Index\n#[derive(Clone)]\npub struct Index {\n    directory: ManagedDirectory,\n    schema: Schema,\n    settings: IndexSettings,\n    executor: Executor,\n    tokenizers: TokenizerManager,\n    fast_field_tokenizers: TokenizerManager,\n    inventory: SegmentMetaInventory,\n}\n\nimpl Index {\n    /// Creates a new builder.\n    pub fn builder() -> IndexBuilder {\n        IndexBuilder::new()\n    }\n    /// Examines the directory to see if it contains an index.\n    ///\n    /// Effectively, it only checks for the presence of the `meta.json` file.\n    pub fn exists(dir: &dyn Directory) -> Result<bool, OpenReadError> {\n        dir.exists(&META_FILEPATH)\n    }\n\n    /// Accessor to the search executor.\n    ///\n    /// This pool is used by default when calling `searcher.search(...)`\n    /// to perform search on the individual segments.\n    ///\n    /// By default the executor is single thread, and simply runs in the calling thread.\n    pub fn search_executor(&self) -> &Executor {\n        &self.executor\n    }\n\n    /// Replace the default single thread search executor pool\n    /// by a thread pool with a given number of threads.\n    pub fn set_multithread_executor(&mut self, num_threads: usize) -> crate::Result<()> {\n        self.executor = Executor::multi_thread(num_threads, \"tantivy-search-\")?;\n        Ok(())\n    }\n\n    /// Custom thread pool by a outer thread pool.\n    pub fn set_executor(&mut self, executor: Executor) {\n        self.executor = executor;\n    }\n\n    /// Replace the default single thread search executor pool\n    /// by a thread pool with as many threads as there are CPUs on the system.\n    pub fn set_default_multithread_executor(&mut self) -> crate::Result<()> {\n        let default_num_threads = available_parallelism()?.get();\n        self.set_multithread_executor(default_num_threads)\n    }\n\n    /// Creates a new index using the [`RamDirectory`].\n    ///\n    /// The index will be allocated in anonymous memory.\n    /// This is useful for indexing small set of documents\n    /// for instances like unit test or temporary in memory index.\n    pub fn create_in_ram(schema: Schema) -> Index {\n        IndexBuilder::new().schema(schema).create_in_ram().unwrap()\n    }\n\n    /// Creates a new index in a given filepath.\n    /// The index will use the [`MmapDirectory`].\n    ///\n    /// If a previous index was in this directory, then it returns\n    /// a [`TantivyError::IndexAlreadyExists`] error.\n    #[cfg(feature = \"mmap\")]\n    pub fn create_in_dir<P: AsRef<Path>>(\n        directory_path: P,\n        schema: Schema,\n    ) -> crate::Result<Index> {\n        IndexBuilder::new()\n            .schema(schema)\n            .create_in_dir(directory_path)\n    }\n\n    /// Opens or creates a new index in the provided directory\n    pub fn open_or_create<T: Into<Box<dyn Directory>>>(\n        dir: T,\n        schema: Schema,\n    ) -> crate::Result<Index> {\n        let dir = dir.into();\n        IndexBuilder::new().schema(schema).open_or_create(dir)\n    }\n\n    /// Creates a new index in a temp directory.\n    ///\n    /// The index will use the [`MmapDirectory`] in a newly created directory.\n    /// The temp directory will be destroyed automatically when the [`Index`] object\n    /// is destroyed.\n    ///\n    /// The temp directory is only used for testing the [`MmapDirectory`].\n    /// For other unit tests, prefer the [`RamDirectory`],\n    /// see: [`IndexBuilder::create_in_ram()`].\n    #[cfg(feature = \"mmap\")]\n    pub fn create_from_tempdir(schema: Schema) -> crate::Result<Index> {\n        IndexBuilder::new().schema(schema).create_from_tempdir()\n    }\n\n    /// Creates a new index given an implementation of the trait `Directory`.\n    ///\n    /// If a directory previously existed, it will be erased.\n    pub fn create<T: Into<Box<dyn Directory>>>(\n        dir: T,\n        schema: Schema,\n        settings: IndexSettings,\n    ) -> crate::Result<Index> {\n        let dir: Box<dyn Directory> = dir.into();\n        let mut builder = IndexBuilder::new().schema(schema);\n        builder = builder.settings(settings);\n        builder.create(dir)\n    }\n\n    /// Creates a new index given a directory and an [`IndexMeta`].\n    fn open_from_metas(\n        directory: ManagedDirectory,\n        metas: &IndexMeta,\n        inventory: SegmentMetaInventory,\n    ) -> Index {\n        let schema = metas.schema.clone();\n        Index {\n            settings: metas.index_settings.clone(),\n            directory,\n            schema,\n            tokenizers: TokenizerManager::default(),\n            fast_field_tokenizers: TokenizerManager::default(),\n            executor: Executor::single_thread(),\n            inventory,\n        }\n    }\n\n    /// Setter for the tokenizer manager.\n    pub fn set_tokenizers(&mut self, tokenizers: TokenizerManager) {\n        self.tokenizers = tokenizers;\n    }\n\n    /// Accessor for the tokenizer manager.\n    pub fn tokenizers(&self) -> &TokenizerManager {\n        &self.tokenizers\n    }\n\n    /// Setter for the fast field tokenizer manager.\n    pub fn set_fast_field_tokenizers(&mut self, tokenizers: TokenizerManager) {\n        self.fast_field_tokenizers = tokenizers;\n    }\n\n    /// Accessor for the fast field tokenizer manager.\n    pub fn fast_field_tokenizer(&self) -> &TokenizerManager {\n        &self.fast_field_tokenizers\n    }\n\n    /// Get the tokenizer associated with a specific field.\n    pub fn tokenizer_for_field(&self, field: Field) -> crate::Result<TextAnalyzer> {\n        let field_entry = self.schema.get_field_entry(field);\n        let field_type = field_entry.field_type();\n        let tokenizer_manager: &TokenizerManager = self.tokenizers();\n        let indexing_options_opt = match field_type {\n            FieldType::JsonObject(options) => options.get_text_indexing_options(),\n            FieldType::Str(options) => options.get_indexing_options(),\n            _ => {\n                return Err(TantivyError::SchemaError(format!(\n                    \"{:?} is not a text field.\",\n                    field_entry.name()\n                )))\n            }\n        };\n        let indexing_options = indexing_options_opt.ok_or_else(|| {\n            TantivyError::InvalidArgument(format!(\n                \"No indexing options set for field {field_entry:?}\"\n            ))\n        })?;\n\n        tokenizer_manager\n            .get(indexing_options.tokenizer())\n            .ok_or_else(|| {\n                TantivyError::InvalidArgument(format!(\n                    \"No Tokenizer found for field {field_entry:?}\"\n                ))\n            })\n    }\n\n    /// Create a default [`IndexReader`] for the given index.\n    ///\n    /// See [`Index.reader_builder()`].\n    pub fn reader(&self) -> crate::Result<IndexReader> {\n        self.reader_builder().try_into()\n    }\n\n    /// Create a [`IndexReader`] for the given index.\n    ///\n    /// Most project should create at most one reader for a given index.\n    /// This method is typically called only once per `Index` instance.\n    pub fn reader_builder(&self) -> IndexReaderBuilder {\n        IndexReaderBuilder::new(self.clone())\n    }\n\n    /// Opens a new directory from an index path.\n    #[cfg(feature = \"mmap\")]\n    pub fn open_in_dir<P: AsRef<Path>>(directory_path: P) -> crate::Result<Index> {\n        let mmap_directory = MmapDirectory::open(directory_path)?;\n        Index::open(mmap_directory)\n    }\n\n    /// Returns the list of the segment metas tracked by the index.\n    ///\n    /// Such segments can of course be part of the index,\n    /// but also they could be segments being currently built or in the middle of a merge\n    /// operation.\n    pub(crate) fn list_all_segment_metas(&self) -> Vec<SegmentMeta> {\n        self.inventory.all()\n    }\n\n    /// Returns the list of fields that have been indexed in the Index.\n    /// The field list includes the field defined in the schema as well as the fields\n    /// that have been indexed as a part of a JSON field.\n    /// The returned field name is the full field name, including the name of the JSON field.\n    ///\n    /// The returned field names can be used in queries.\n    ///\n    /// Notice: If your data contains JSON fields this is **very expensive**, as it requires\n    /// browsing through the inverted index term dictionary and the columnar field dictionary.\n    ///\n    /// Disclaimer: Some fields may not be listed here. For instance, if the schema contains a json\n    /// field that is not indexed nor a fast field but is stored, it is possible for the field\n    /// to not be listed.\n    pub fn fields_metadata(&self) -> crate::Result<Vec<FieldMetadata>> {\n        let segments = self.searchable_segments()?;\n        let fields_metadata: Vec<Vec<FieldMetadata>> = segments\n            .into_iter()\n            .map(|segment| SegmentReader::open(&segment)?.fields_metadata())\n            .collect::<Result<_, _>>()?;\n        Ok(merge_field_meta_data(fields_metadata))\n    }\n\n    /// Creates a new segment_meta (Advanced user only).\n    ///\n    /// As long as the `SegmentMeta` lives, the files associated with the\n    /// `SegmentMeta` are guaranteed to not be garbage collected, regardless of\n    /// whether the segment is recorded as part of the index or not.\n    pub fn new_segment_meta(&self, segment_id: SegmentId, max_doc: u32) -> SegmentMeta {\n        self.inventory.new_segment_meta(segment_id, max_doc)\n    }\n\n    /// Open the index using the provided directory\n    pub fn open<T: Into<Box<dyn Directory>>>(directory: T) -> crate::Result<Index> {\n        let directory = directory.into();\n        let directory = ManagedDirectory::wrap(directory)?;\n        let inventory = SegmentMetaInventory::default();\n        let metas = load_metas(&directory, &inventory)?;\n        let index = Index::open_from_metas(directory, &metas, inventory);\n        Ok(index)\n    }\n\n    /// Reads the index meta file from the directory.\n    pub fn load_metas(&self) -> crate::Result<IndexMeta> {\n        load_metas(self.directory(), &self.inventory)\n    }\n\n    /// Open a new index writer with the given options. Attempts to acquire a lockfile.\n    ///\n    /// The lockfile should be deleted on drop, but it is possible\n    /// that due to a panic or other error, a stale lockfile will be\n    /// left in the index directory. If you are sure that no other\n    /// `IndexWriter` on the system is accessing the index directory,\n    /// it is safe to manually delete the lockfile.\n    ///\n    /// - `options` defines the writer configuration which includes things like buffer sizes,\n    ///   indexer threads, etc...\n    ///\n    /// # Errors\n    /// If the lockfile already exists, returns `TantivyError::LockFailure`.\n    /// If the memory arena per thread is too small or too big, returns\n    /// `TantivyError::InvalidArgument`\n    pub fn writer_with_options<D: Document>(\n        &self,\n        options: IndexWriterOptions,\n    ) -> crate::Result<IndexWriter<D>> {\n        let directory_lock = self\n            .directory\n            .acquire_lock(&INDEX_WRITER_LOCK)\n            .map_err(|err| {\n                TantivyError::LockFailure(\n                    err,\n                    Some(\n                        \"Failed to acquire index lock. If you are using a regular directory, this \\\n                         means there is already an `IndexWriter` working on this `Directory`, in \\\n                         this process or in a different process.\"\n                            .to_string(),\n                    ),\n                )\n            })?;\n\n        IndexWriter::new(self, options, directory_lock)\n    }\n\n    /// Open a new index writer. Attempts to acquire a lockfile.\n    ///\n    /// The lockfile should be deleted on drop, but it is possible\n    /// that due to a panic or other error, a stale lockfile will be\n    /// left in the index directory. If you are sure that no other\n    /// `IndexWriter` on the system is accessing the index directory,\n    /// it is safe to manually delete the lockfile.\n    ///\n    /// - `num_threads` defines the number of indexing workers that should work at the same time.\n    ///\n    /// - `overall_memory_budget_in_bytes` sets the amount of memory allocated for all indexing\n    ///   thread.\n    ///\n    /// Each thread will receive a budget of `overall_memory_budget_in_bytes / num_threads`.\n    ///\n    /// # Errors\n    /// If the lockfile already exists, returns `Error::DirectoryLockBusy` or an `Error::IoError`.\n    /// If the memory arena per thread is too small or too big, returns\n    /// `TantivyError::InvalidArgument`\n    pub fn writer_with_num_threads<D: Document>(\n        &self,\n        num_threads: usize,\n        overall_memory_budget_in_bytes: usize,\n    ) -> crate::Result<IndexWriter<D>> {\n        let memory_arena_in_bytes_per_thread = overall_memory_budget_in_bytes / num_threads;\n        let options = IndexWriterOptions::builder()\n            .num_worker_threads(num_threads)\n            .memory_budget_per_thread(memory_arena_in_bytes_per_thread)\n            .build();\n        self.writer_with_options(options)\n    }\n\n    /// Helper to create an index writer for tests.\n    ///\n    /// That index writer only simply has a single thread and a memory budget of 15 MB.\n    /// Using a single thread gives us a deterministic allocation of DocId.\n    #[cfg(test)]\n    pub fn writer_for_tests<D: Document>(&self) -> crate::Result<IndexWriter<D>> {\n        self.writer_with_num_threads(1, MEMORY_BUDGET_NUM_BYTES_MIN)\n    }\n\n    /// Creates a multithreaded writer\n    ///\n    /// Tantivy will automatically define the number of threads to use, but\n    /// no more than 8 threads.\n    /// `overall_memory_arena_in_bytes` is the total target memory usage that will be split\n    /// between a given number of threads.\n    ///\n    /// # Errors\n    /// If the lockfile already exists, returns `Error::FileAlreadyExists`.\n    /// If the memory arena per thread is too small or too big, returns\n    /// `TantivyError::InvalidArgument`\n    pub fn writer<D: Document>(\n        &self,\n        memory_budget_in_bytes: usize,\n    ) -> crate::Result<IndexWriter<D>> {\n        let mut num_threads = std::cmp::min(available_parallelism()?.get(), MAX_NUM_THREAD);\n        let memory_budget_num_bytes_per_thread = memory_budget_in_bytes / num_threads;\n        if memory_budget_num_bytes_per_thread < MEMORY_BUDGET_NUM_BYTES_MIN {\n            num_threads = (memory_budget_in_bytes / MEMORY_BUDGET_NUM_BYTES_MIN).max(1);\n        }\n        self.writer_with_num_threads(num_threads, memory_budget_in_bytes)\n    }\n\n    /// Accessor to the index settings\n    pub fn settings(&self) -> &IndexSettings {\n        &self.settings\n    }\n\n    /// Accessor to the index settings\n    pub fn settings_mut(&mut self) -> &mut IndexSettings {\n        &mut self.settings\n    }\n\n    /// Accessor to the index schema\n    ///\n    /// The schema is actually cloned.\n    pub fn schema(&self) -> Schema {\n        self.schema.clone()\n    }\n\n    /// Returns the list of segments that are searchable\n    pub fn searchable_segments(&self) -> crate::Result<Vec<Segment>> {\n        Ok(self\n            .searchable_segment_metas()?\n            .into_iter()\n            .map(|segment_meta| self.segment(segment_meta))\n            .collect())\n    }\n\n    #[doc(hidden)]\n    pub fn segment(&self, segment_meta: SegmentMeta) -> Segment {\n        Segment::for_index(self.clone(), segment_meta)\n    }\n\n    /// Creates a new segment.\n    pub fn new_segment(&self) -> Segment {\n        let segment_meta = self\n            .inventory\n            .new_segment_meta(SegmentId::generate_random(), 0);\n        self.segment(segment_meta)\n    }\n\n    /// Return a reference to the index directory.\n    pub fn directory(&self) -> &ManagedDirectory {\n        &self.directory\n    }\n\n    /// Return a mutable reference to the index directory.\n    pub fn directory_mut(&mut self) -> &mut ManagedDirectory {\n        &mut self.directory\n    }\n\n    /// Reads the meta.json and returns the list of\n    /// `SegmentMeta` from the last commit.\n    pub fn searchable_segment_metas(&self) -> crate::Result<Vec<SegmentMeta>> {\n        Ok(self.load_metas()?.segments)\n    }\n\n    /// Returns the list of segment ids that are searchable.\n    pub fn searchable_segment_ids(&self) -> crate::Result<Vec<SegmentId>> {\n        Ok(self\n            .searchable_segment_metas()?\n            .iter()\n            .map(SegmentMeta::id)\n            .collect())\n    }\n\n    /// Returns the set of corrupted files\n    pub fn validate_checksum(&self) -> crate::Result<HashSet<PathBuf>> {\n        let managed_files = self.directory.list_managed_files();\n        let active_segments_files: HashSet<PathBuf> = self\n            .searchable_segment_metas()?\n            .iter()\n            .flat_map(|segment_meta| segment_meta.list_files())\n            .collect();\n        let active_existing_files: HashSet<&PathBuf> =\n            active_segments_files.intersection(&managed_files).collect();\n\n        let mut damaged_files = HashSet::new();\n        for path in active_existing_files {\n            if !self.directory.validate_checksum(path)? {\n                damaged_files.insert((*path).clone());\n            }\n        }\n        Ok(damaged_files)\n    }\n}\n\nimpl fmt::Debug for Index {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"Index({:?})\", self.directory)\n    }\n}\n"
  },
  {
    "path": "src/index/index_meta.rs",
    "content": "use std::collections::HashSet;\nuse std::fmt;\nuse std::path::PathBuf;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::SegmentComponent;\nuse crate::index::SegmentId;\nuse crate::schema::Schema;\nuse crate::store::Compressor;\nuse crate::{Inventory, Opstamp, TrackedObject};\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct DeleteMeta {\n    num_deleted_docs: u32,\n    pub opstamp: Opstamp,\n}\n\n#[derive(Clone, Default)]\npub(crate) struct SegmentMetaInventory {\n    inventory: Inventory<InnerSegmentMeta>,\n}\n\nimpl SegmentMetaInventory {\n    /// Lists all living `SegmentMeta` object at the time of the call.\n    pub fn all(&self) -> Vec<SegmentMeta> {\n        self.inventory\n            .list()\n            .into_iter()\n            .map(SegmentMeta::from)\n            .collect::<Vec<_>>()\n    }\n\n    pub fn new_segment_meta(&self, segment_id: SegmentId, max_doc: u32) -> SegmentMeta {\n        let inner = InnerSegmentMeta {\n            segment_id,\n            max_doc,\n            deletes: None,\n        };\n        SegmentMeta::from(self.inventory.track(inner))\n    }\n}\n\n/// `SegmentMeta` contains simple meta information about a segment.\n///\n/// For instance the number of docs it contains,\n/// how many are deleted, etc.\n#[derive(Clone)]\npub struct SegmentMeta {\n    tracked: TrackedObject<InnerSegmentMeta>,\n}\n\nimpl fmt::Debug for SegmentMeta {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        self.tracked.fmt(f)\n    }\n}\n\nimpl serde::Serialize for SegmentMeta {\n    fn serialize<S>(\n        &self,\n        serializer: S,\n    ) -> Result<<S as serde::Serializer>::Ok, <S as serde::Serializer>::Error>\n    where\n        S: serde::Serializer,\n    {\n        self.tracked.serialize(serializer)\n    }\n}\n\nimpl From<TrackedObject<InnerSegmentMeta>> for SegmentMeta {\n    fn from(tracked: TrackedObject<InnerSegmentMeta>) -> SegmentMeta {\n        SegmentMeta { tracked }\n    }\n}\n\nimpl SegmentMeta {\n    // Creates a new `SegmentMeta` object.\n\n    /// Returns the segment id.\n    pub fn id(&self) -> SegmentId {\n        self.tracked.segment_id\n    }\n\n    /// Returns the number of deleted documents.\n    pub fn num_deleted_docs(&self) -> u32 {\n        self.tracked\n            .deletes\n            .as_ref()\n            .map(|delete_meta| delete_meta.num_deleted_docs)\n            .unwrap_or(0u32)\n    }\n\n    /// Returns the list of files that\n    /// are required for the segment meta.\n    /// Note: Some of the returned files may not exist depending on the state of the segment.\n    ///\n    /// This is useful as the way tantivy removes files\n    /// is by removing all files that have been created by tantivy\n    /// and are not used by any segment anymore.\n    pub fn list_files(&self) -> HashSet<PathBuf> {\n        SegmentComponent::iterator()\n            .map(|component| self.relative_path(*component))\n            .collect::<HashSet<PathBuf>>()\n    }\n\n    /// Returns the relative path of a component of our segment.\n    ///\n    /// It just joins the segment id with the extension\n    /// associated with a segment component.\n    pub fn relative_path(&self, component: SegmentComponent) -> PathBuf {\n        let mut path = self.id().uuid_string();\n        path.push_str(&match component {\n            SegmentComponent::Postings => \".idx\".to_string(),\n            SegmentComponent::Positions => \".pos\".to_string(),\n            SegmentComponent::Terms => \".term\".to_string(),\n            SegmentComponent::Store => \".store\".to_string(),\n            SegmentComponent::FastFields => \".fast\".to_string(),\n            SegmentComponent::FieldNorms => \".fieldnorm\".to_string(),\n            SegmentComponent::Delete => format!(\".{}.del\", self.delete_opstamp().unwrap_or(0)),\n        });\n        PathBuf::from(path)\n    }\n\n    /// Return the highest doc id + 1\n    ///\n    /// If there are no deletes, then num_docs = max_docs\n    /// and all the doc ids contains in this segment\n    /// are exactly (0..max_doc).\n    pub fn max_doc(&self) -> u32 {\n        self.tracked.max_doc\n    }\n\n    /// Return the number of documents in the segment.\n    pub fn num_docs(&self) -> u32 {\n        self.max_doc() - self.num_deleted_docs()\n    }\n\n    /// Returns the `Opstamp` of the last delete operation\n    /// taken in account in this segment.\n    pub fn delete_opstamp(&self) -> Option<Opstamp> {\n        self.tracked\n            .deletes\n            .as_ref()\n            .map(|delete_meta| delete_meta.opstamp)\n    }\n\n    /// Returns true iff the segment meta contains\n    /// delete information.\n    pub fn has_deletes(&self) -> bool {\n        self.num_deleted_docs() > 0\n    }\n\n    /// Updates the max_doc value from the `SegmentMeta`.\n    pub fn with_max_doc(self, max_doc: u32) -> SegmentMeta {\n        assert_eq!(self.tracked.max_doc, 0);\n        assert!(self.tracked.deletes.is_none());\n        let tracked = self.tracked.map(move |inner_meta| InnerSegmentMeta {\n            segment_id: inner_meta.segment_id,\n            max_doc,\n            deletes: None,\n        });\n        SegmentMeta { tracked }\n    }\n\n    #[doc(hidden)]\n    #[must_use]\n    pub fn with_delete_meta(self, num_deleted_docs: u32, opstamp: Opstamp) -> SegmentMeta {\n        assert!(\n            num_deleted_docs <= self.max_doc(),\n            \"There cannot be more deleted docs than there are docs.\"\n        );\n        let delete_meta = DeleteMeta {\n            num_deleted_docs,\n            opstamp,\n        };\n        let tracked = self.tracked.map(move |inner_meta| InnerSegmentMeta {\n            segment_id: inner_meta.segment_id,\n            max_doc: inner_meta.max_doc,\n            deletes: Some(delete_meta),\n        });\n        SegmentMeta { tracked }\n    }\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\nstruct InnerSegmentMeta {\n    segment_id: SegmentId,\n    max_doc: u32,\n    pub deletes: Option<DeleteMeta>,\n}\n\nimpl InnerSegmentMeta {\n    pub fn track(self, inventory: &SegmentMetaInventory) -> SegmentMeta {\n        SegmentMeta {\n            tracked: inventory.inventory.track(self),\n        }\n    }\n}\n\nfn return_true() -> bool {\n    true\n}\n\nfn is_true(val: &bool) -> bool {\n    *val\n}\n\n/// Search Index Settings.\n///\n/// Contains settings which are applied on the whole\n/// index, like presort documents.\n#[derive(Clone, Debug, Serialize, Deserialize, Eq, PartialEq)]\npub struct IndexSettings {\n    /// The `Compressor` used to compress the doc store.\n    #[serde(default)]\n    pub docstore_compression: Compressor,\n    /// If set to true, docstore compression will happen on a dedicated thread.\n    /// (defaults: true)\n    #[doc(hidden)]\n    #[serde(default = \"return_true\")]\n    #[serde(skip_serializing_if = \"is_true\")]\n    pub docstore_compress_dedicated_thread: bool,\n    #[serde(default = \"default_docstore_blocksize\")]\n    /// The size of each block that will be compressed and written to disk\n    pub docstore_blocksize: usize,\n}\n\n/// Must be a function to be compatible with serde defaults\nfn default_docstore_blocksize() -> usize {\n    16_384\n}\n\nimpl Default for IndexSettings {\n    fn default() -> Self {\n        Self {\n            docstore_compression: Compressor::default(),\n            docstore_blocksize: default_docstore_blocksize(),\n            docstore_compress_dedicated_thread: true,\n        }\n    }\n}\n\n/// The order to sort by\n#[derive(Clone, Copy, Debug, Serialize, Deserialize, Eq, PartialEq)]\npub enum Order {\n    /// Ascending Order\n    Asc,\n    /// Descending Order\n    Desc,\n}\n\nimpl Order {\n    /// return if the Order is ascending\n    pub fn is_asc(&self) -> bool {\n        self == &Order::Asc\n    }\n    /// return if the Order is descending\n    pub fn is_desc(&self) -> bool {\n        self == &Order::Desc\n    }\n}\n\n/// Meta information about the `Index`.\n///\n/// This object is serialized on disk in the `meta.json` file.\n/// It keeps information about\n/// * the searchable segments,\n/// * the index `docstamp`\n/// * the schema\n#[derive(Clone, Serialize)]\npub struct IndexMeta {\n    /// `IndexSettings` to configure index options.\n    #[serde(default)]\n    pub index_settings: IndexSettings,\n    /// List of `SegmentMeta` information associated with each finalized segment of the index.\n    pub segments: Vec<SegmentMeta>,\n    /// Index `Schema`\n    pub schema: Schema,\n    /// Opstamp associated with the last `commit` operation.\n    pub opstamp: Opstamp,\n    /// Payload associated with the last commit.\n    ///\n    /// Upon commit, clients can optionally add a small `String` payload to their commit\n    /// to help identify this commit.\n    /// This payload is entirely unused by tantivy.\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub payload: Option<String>,\n}\n\n#[derive(Deserialize, Debug)]\nstruct UntrackedIndexMeta {\n    pub segments: Vec<InnerSegmentMeta>,\n    #[serde(default)]\n    pub index_settings: IndexSettings,\n    pub schema: Schema,\n    pub opstamp: Opstamp,\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    pub payload: Option<String>,\n}\n\nimpl UntrackedIndexMeta {\n    pub fn track(self, inventory: &SegmentMetaInventory) -> IndexMeta {\n        IndexMeta {\n            index_settings: self.index_settings,\n            segments: self\n                .segments\n                .into_iter()\n                .map(|inner_seg_meta| inner_seg_meta.track(inventory))\n                .collect::<Vec<SegmentMeta>>(),\n            schema: self.schema,\n            opstamp: self.opstamp,\n            payload: self.payload,\n        }\n    }\n}\n\nimpl IndexMeta {\n    /// Create an `IndexMeta` object representing a brand new `Index`\n    /// with the given index.\n    ///\n    /// This new index does not contains any segments.\n    /// Opstamp will the value `0u64`.\n    pub fn with_schema(schema: Schema) -> IndexMeta {\n        IndexMeta {\n            index_settings: IndexSettings::default(),\n            segments: vec![],\n            schema,\n            opstamp: 0u64,\n            payload: None,\n        }\n    }\n\n    pub(crate) fn deserialize(\n        meta_json: &str,\n        inventory: &SegmentMetaInventory,\n    ) -> serde_json::Result<IndexMeta> {\n        let untracked_meta_json: UntrackedIndexMeta = serde_json::from_str(meta_json)?;\n        Ok(untracked_meta_json.track(inventory))\n    }\n}\n\nimpl fmt::Debug for IndexMeta {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(\n            f,\n            \"{}\",\n            serde_json::ser::to_string(self)\n                .expect(\"JSON serialization for IndexMeta should never fail.\")\n        )\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::IndexMeta;\n    use crate::index::index_meta::UntrackedIndexMeta;\n    use crate::schema::{Schema, TEXT};\n    use crate::store::Compressor;\n    #[cfg(feature = \"zstd-compression\")]\n    use crate::store::ZstdCompressor;\n    use crate::IndexSettings;\n\n    #[test]\n    fn test_serialize_metas() {\n        let schema = {\n            let mut schema_builder = Schema::builder();\n            schema_builder.add_text_field(\"text\", TEXT);\n            schema_builder.build()\n        };\n        let index_metas = IndexMeta {\n            index_settings: IndexSettings {\n                docstore_compression: Compressor::None,\n                ..Default::default()\n            },\n            segments: Vec::new(),\n            schema,\n            opstamp: 0u64,\n            payload: None,\n        };\n        let json = serde_json::ser::to_string(&index_metas).expect(\"serialization failed\");\n        assert_eq!(\n            json,\n            r#\"{\"index_settings\":{\"docstore_compression\":\"none\",\"docstore_blocksize\":16384},\"segments\":[],\"schema\":[{\"name\":\"text\",\"type\":\"text\",\"options\":{\"indexing\":{\"record\":\"position\",\"fieldnorms\":true,\"tokenizer\":\"default\"},\"stored\":false,\"fast\":false}}],\"opstamp\":0}\"#\n        );\n\n        let deser_meta: UntrackedIndexMeta = serde_json::from_str(&json).unwrap();\n        assert_eq!(index_metas.index_settings, deser_meta.index_settings);\n        assert_eq!(index_metas.schema, deser_meta.schema);\n        assert_eq!(index_metas.opstamp, deser_meta.opstamp);\n    }\n\n    #[test]\n    #[cfg(feature = \"zstd-compression\")]\n    fn test_serialize_metas_zstd_compressor() {\n        let schema = {\n            let mut schema_builder = Schema::builder();\n            schema_builder.add_text_field(\"text\", TEXT);\n            schema_builder.build()\n        };\n        let index_metas = IndexMeta {\n            index_settings: IndexSettings {\n                docstore_compression: crate::store::Compressor::Zstd(ZstdCompressor {\n                    compression_level: Some(4),\n                }),\n                docstore_blocksize: 1_000_000,\n                docstore_compress_dedicated_thread: true,\n            },\n            segments: Vec::new(),\n            schema,\n            opstamp: 0u64,\n            payload: None,\n        };\n        let json = serde_json::ser::to_string(&index_metas).expect(\"serialization failed\");\n        assert_eq!(\n            json,\n            r#\"{\"index_settings\":{\"docstore_compression\":\"zstd(compression_level=4)\",\"docstore_blocksize\":1000000},\"segments\":[],\"schema\":[{\"name\":\"text\",\"type\":\"text\",\"options\":{\"indexing\":{\"record\":\"position\",\"fieldnorms\":true,\"tokenizer\":\"default\"},\"stored\":false,\"fast\":false}}],\"opstamp\":0}\"#\n        );\n\n        let deser_meta: UntrackedIndexMeta = serde_json::from_str(&json).unwrap();\n        assert_eq!(index_metas.index_settings, deser_meta.index_settings);\n        assert_eq!(index_metas.schema, deser_meta.schema);\n        assert_eq!(index_metas.opstamp, deser_meta.opstamp);\n    }\n\n    #[test]\n    #[cfg(all(feature = \"lz4-compression\", feature = \"zstd-compression\"))]\n    fn test_serialize_metas_invalid_comp() {\n        let json = r#\"{\"index_settings\":{\"docstore_compression\":\"zsstd\",\"docstore_blocksize\":1000000},\"segments\":[],\"schema\":[{\"name\":\"text\",\"type\":\"text\",\"options\":{\"indexing\":{\"record\":\"position\",\"fieldnorms\":true,\"tokenizer\":\"default\"},\"stored\":false,\"fast\":false}}],\"opstamp\":0}\"#;\n\n        let err = serde_json::from_str::<UntrackedIndexMeta>(json).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"unknown variant `zsstd`, expected one of `none`, `lz4`, `zstd`, \\\n             `zstd(compression_level=5)` at line 1 column 49\"\n                .to_string()\n        );\n\n        let json = r#\"{\"index_settings\":{\"docstore_compression\":\"zstd(bla=10)\",\"docstore_blocksize\":1000000},\"segments\":[],\"schema\":[{\"name\":\"text\",\"type\":\"text\",\"options\":{\"indexing\":{\"record\":\"position\",\"fieldnorms\":true,\"tokenizer\":\"default\"},\"stored\":false,\"fast\":false}}],\"opstamp\":0}\"#;\n\n        let err = serde_json::from_str::<UntrackedIndexMeta>(json).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"unknown zstd option \\\"bla\\\" at line 1 column 56\".to_string()\n        );\n    }\n\n    #[test]\n    #[cfg(not(feature = \"zstd-compression\"))]\n    fn test_serialize_metas_unsupported_comp() {\n        let json = r#\"{\"index_settings\":{\"docstore_compression\":\"zstd\",\"docstore_blocksize\":1000000},\"segments\":[],\"schema\":[{\"name\":\"text\",\"type\":\"text\",\"options\":{\"indexing\":{\"record\":\"position\",\"fieldnorms\":true,\"tokenizer\":\"default\"},\"stored\":false,\"fast\":false}}],\"opstamp\":0}\"#;\n\n        let err = serde_json::from_str::<UntrackedIndexMeta>(json).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"unsupported variant `zstd`, please enable Tantivy's `zstd-compression` feature at \\\n             line 1 column 48\"\n                .to_string()\n        );\n    }\n\n    #[test]\n    #[cfg(feature = \"lz4-compression\")]\n    fn test_index_settings_default() {\n        use crate::store::Compressor;\n\n        let mut index_settings = IndexSettings::default();\n        assert_eq!(\n            index_settings,\n            IndexSettings {\n                docstore_compression: Compressor::default(),\n                docstore_compress_dedicated_thread: true,\n                docstore_blocksize: 16_384\n            }\n        );\n        {\n            let index_settings_json = serde_json::to_value(&index_settings).unwrap();\n            assert_eq!(\n                index_settings_json,\n                serde_json::json!({\n                    \"docstore_compression\": \"lz4\",\n                    \"docstore_blocksize\": 16384\n                })\n            );\n            let index_settings_deser: IndexSettings =\n                serde_json::from_value(index_settings_json).unwrap();\n            assert_eq!(index_settings_deser, index_settings);\n        }\n        {\n            index_settings.docstore_compress_dedicated_thread = false;\n            let index_settings_json = serde_json::to_value(&index_settings).unwrap();\n            assert_eq!(\n                index_settings_json,\n                serde_json::json!({\n                    \"docstore_compression\": \"lz4\",\n                    \"docstore_blocksize\": 16384,\n                    \"docstore_compress_dedicated_thread\": false,\n                })\n            );\n            let index_settings_deser: IndexSettings =\n                serde_json::from_value(index_settings_json).unwrap();\n            assert_eq!(index_settings_deser, index_settings);\n        }\n    }\n}\n"
  },
  {
    "path": "src/index/inverted_index_reader.rs",
    "content": "use std::io;\n\nuse common::json_path_writer::JSON_END_OF_PATH;\nuse common::{BinarySerializable, ByteCount};\n#[cfg(feature = \"quickwit\")]\nuse futures_util::{FutureExt, StreamExt, TryStreamExt};\n#[cfg(feature = \"quickwit\")]\nuse itertools::Itertools;\n#[cfg(feature = \"quickwit\")]\nuse tantivy_fst::automaton::{AlwaysMatch, Automaton};\n\nuse crate::directory::FileSlice;\nuse crate::positions::PositionReader;\nuse crate::postings::{BlockSegmentPostings, SegmentPostings, TermInfo};\nuse crate::schema::{IndexRecordOption, Term, Type};\nuse crate::termdict::TermDictionary;\n\n/// The inverted index reader is in charge of accessing\n/// the inverted index associated with a specific field.\n///\n/// # Note\n///\n/// It is safe to delete the segment associated with\n/// an `InvertedIndexReader`. As long as it is open,\n/// the [`FileSlice`] it is relying on should\n/// stay available.\n///\n/// `InvertedIndexReader` are created by calling\n/// [`SegmentReader::inverted_index()`](crate::SegmentReader::inverted_index).\npub struct InvertedIndexReader {\n    termdict: TermDictionary,\n    postings_file_slice: FileSlice,\n    positions_file_slice: FileSlice,\n    record_option: IndexRecordOption,\n    total_num_tokens: u64,\n}\n\n/// Object that records the amount of space used by a field in an inverted index.\npub(crate) struct InvertedIndexFieldSpace {\n    pub field_name: String,\n    pub field_type: Type,\n    pub postings_size: ByteCount,\n    pub positions_size: ByteCount,\n    pub num_terms: u64,\n}\n\n/// Returns None if the term is not a valid JSON path.\nfn extract_field_name_and_field_type_from_json_path(term: &[u8]) -> Option<(String, Type)> {\n    let index = term.iter().position(|&byte| byte == JSON_END_OF_PATH)?;\n    let field_type_code = term.get(index + 1).copied()?;\n    let field_type = Type::from_code(field_type_code)?;\n    // Let's flush the current field.\n    let field_name = String::from_utf8_lossy(&term[..index]).to_string();\n    Some((field_name, field_type))\n}\n\nimpl InvertedIndexFieldSpace {\n    fn record(&mut self, term_info: &TermInfo) {\n        self.postings_size += ByteCount::from(term_info.posting_num_bytes() as u64);\n        self.positions_size += ByteCount::from(term_info.positions_num_bytes() as u64);\n        self.num_terms += 1;\n    }\n}\n\nimpl InvertedIndexReader {\n    pub(crate) fn new(\n        termdict: TermDictionary,\n        postings_file_slice: FileSlice,\n        positions_file_slice: FileSlice,\n        record_option: IndexRecordOption,\n    ) -> io::Result<InvertedIndexReader> {\n        let (total_num_tokens_slice, postings_body) = postings_file_slice.split(8);\n        let total_num_tokens = u64::deserialize(&mut total_num_tokens_slice.read_bytes()?)?;\n        Ok(InvertedIndexReader {\n            termdict,\n            postings_file_slice: postings_body,\n            positions_file_slice,\n            record_option,\n            total_num_tokens,\n        })\n    }\n\n    /// Creates an empty `InvertedIndexReader` object, which\n    /// contains no terms at all.\n    pub fn empty(record_option: IndexRecordOption) -> InvertedIndexReader {\n        InvertedIndexReader {\n            termdict: TermDictionary::empty(),\n            postings_file_slice: FileSlice::empty(),\n            positions_file_slice: FileSlice::empty(),\n            record_option,\n            total_num_tokens: 0u64,\n        }\n    }\n\n    /// Returns the term info associated with the term.\n    pub fn get_term_info(&self, term: &Term) -> io::Result<Option<TermInfo>> {\n        self.termdict.get(term.serialized_value_bytes())\n    }\n\n    /// Return the term dictionary datastructure.\n    pub fn terms(&self) -> &TermDictionary {\n        &self.termdict\n    }\n\n    /// Return the fields and types encoded in the dictionary in lexicographic order.\n    /// Only valid on JSON fields.\n    ///\n    /// Notice: This requires a full scan and therefore **very expensive**.\n    /// TODO: Move to sstable to use the index.\n    pub(crate) fn list_encoded_json_fields(&self) -> io::Result<Vec<InvertedIndexFieldSpace>> {\n        let mut stream = self.termdict.stream()?;\n        let mut fields: Vec<InvertedIndexFieldSpace> = Vec::new();\n\n        let mut current_field_opt: Option<InvertedIndexFieldSpace> = None;\n        // Current field bytes, including the JSON_END_OF_PATH.\n        let mut current_field_bytes: Vec<u8> = Vec::new();\n\n        while let Some((term, term_info)) = stream.next() {\n            if let Some(current_field) = &mut current_field_opt {\n                if term.starts_with(&current_field_bytes) {\n                    // We are still in the same field.\n                    current_field.record(term_info);\n                    continue;\n                }\n            }\n\n            // This is a new field!\n            // Let's flush the current field.\n            fields.extend(current_field_opt.take());\n            current_field_bytes.clear();\n\n            // And create a new one.\n            let Some((field_name, field_type)) =\n                extract_field_name_and_field_type_from_json_path(term)\n            else {\n                error!(\n                    \"invalid term bytes encountered {term:?}. this only happens if the term \\\n                     dictionary is corrupted. please report\"\n                );\n                continue;\n            };\n            let mut field_space = InvertedIndexFieldSpace {\n                field_name,\n                field_type,\n                postings_size: ByteCount::default(),\n                positions_size: ByteCount::default(),\n                num_terms: 0u64,\n            };\n            field_space.record(term_info);\n\n            // We include the json type and the json end of path to make sure the prefix check\n            // is meaningful.\n            current_field_bytes.extend_from_slice(&term[..field_space.field_name.len() + 2]);\n            current_field_opt = Some(field_space);\n        }\n\n        // We need to flush the last field as well.\n        fields.extend(current_field_opt.take());\n\n        Ok(fields)\n    }\n\n    /// Resets the block segment to another position of the postings\n    /// file.\n    ///\n    /// This is useful for enumerating through a list of terms,\n    /// and consuming the associated posting lists while avoiding\n    /// reallocating a [`BlockSegmentPostings`].\n    ///\n    /// # Warning\n    ///\n    /// This does not reset the positions list.\n    pub fn reset_block_postings_from_terminfo(\n        &self,\n        term_info: &TermInfo,\n        block_postings: &mut BlockSegmentPostings,\n    ) -> io::Result<()> {\n        let postings_slice = self\n            .postings_file_slice\n            .slice(term_info.postings_range.clone());\n        let postings_bytes = postings_slice.read_bytes()?;\n        block_postings.reset(term_info.doc_freq, postings_bytes)?;\n        Ok(())\n    }\n\n    /// Returns a block postings given a `Term`.\n    /// This method is for an advanced usage only.\n    ///\n    /// Most users should prefer using [`Self::read_postings()`] instead.\n    pub fn read_block_postings(\n        &self,\n        term: &Term,\n        option: IndexRecordOption,\n    ) -> io::Result<Option<BlockSegmentPostings>> {\n        self.get_term_info(term)?\n            .map(move |term_info| self.read_block_postings_from_terminfo(&term_info, option))\n            .transpose()\n    }\n\n    /// Returns a block postings given a `term_info`.\n    /// This method is for an advanced usage only.\n    ///\n    /// Most users should prefer using [`Self::read_postings()`] instead.\n    pub fn read_block_postings_from_terminfo(\n        &self,\n        term_info: &TermInfo,\n        requested_option: IndexRecordOption,\n    ) -> io::Result<BlockSegmentPostings> {\n        let postings_data = self\n            .postings_file_slice\n            .slice(term_info.postings_range.clone());\n        BlockSegmentPostings::open(\n            term_info.doc_freq,\n            postings_data,\n            self.record_option,\n            requested_option,\n        )\n    }\n\n    /// Returns a posting object given a `term_info`.\n    /// This method is for an advanced usage only.\n    ///\n    /// Most users should prefer using [`Self::read_postings()`] instead.\n    pub fn read_postings_from_terminfo(\n        &self,\n        term_info: &TermInfo,\n        option: IndexRecordOption,\n    ) -> io::Result<SegmentPostings> {\n        let option = option.downgrade(self.record_option);\n\n        let block_postings = self.read_block_postings_from_terminfo(term_info, option)?;\n        let position_reader = {\n            if option.has_positions() {\n                let positions_data = self\n                    .positions_file_slice\n                    .read_bytes_slice(term_info.positions_range.clone())?;\n                let position_reader = PositionReader::open(positions_data)?;\n                Some(position_reader)\n            } else {\n                None\n            }\n        };\n        Ok(SegmentPostings::from_block_postings(\n            block_postings,\n            position_reader,\n        ))\n    }\n\n    /// Returns the total number of tokens recorded for all documents\n    /// (including deleted documents).\n    pub fn total_num_tokens(&self) -> u64 {\n        self.total_num_tokens\n    }\n\n    /// Returns the segment postings associated with the term, and with the given option,\n    /// or `None` if the term has never been encountered and indexed.\n    ///\n    /// If the field was not indexed with the indexing options that cover\n    /// the requested options, the returned [`SegmentPostings`] the method does not fail\n    /// and returns a `SegmentPostings` with as much information as possible.\n    ///\n    /// For instance, requesting [`IndexRecordOption::WithFreqs`] for a\n    /// [`TextOptions`](crate::schema::TextOptions) that does not index position\n    /// will return a [`SegmentPostings`] with `DocId`s and frequencies.\n    pub fn read_postings(\n        &self,\n        term: &Term,\n        option: IndexRecordOption,\n    ) -> io::Result<Option<SegmentPostings>> {\n        self.get_term_info(term)?\n            .map(move |term_info| self.read_postings_from_terminfo(&term_info, option))\n            .transpose()\n    }\n\n    /// Returns the number of documents containing the term.\n    pub fn doc_freq(&self, term: &Term) -> io::Result<u32> {\n        Ok(self\n            .get_term_info(term)?\n            .map(|term_info| term_info.doc_freq)\n            .unwrap_or(0u32))\n    }\n}\n\n#[cfg(feature = \"quickwit\")]\nimpl InvertedIndexReader {\n    pub(crate) async fn get_term_info_async(&self, term: &Term) -> io::Result<Option<TermInfo>> {\n        self.termdict.get_async(term.serialized_value_bytes()).await\n    }\n\n    async fn get_term_range_async<'a, A: Automaton + 'a>(\n        &'a self,\n        terms: impl std::ops::RangeBounds<Term>,\n        automaton: A,\n        limit: Option<u64>,\n        merge_holes_under_bytes: usize,\n    ) -> io::Result<impl Iterator<Item = TermInfo> + 'a>\n    where\n        A::State: Clone,\n    {\n        use std::ops::Bound;\n        let range_builder = self.termdict.search(automaton);\n        let range_builder = match terms.start_bound() {\n            Bound::Included(bound) => range_builder.ge(bound.serialized_value_bytes()),\n            Bound::Excluded(bound) => range_builder.gt(bound.serialized_value_bytes()),\n            Bound::Unbounded => range_builder,\n        };\n        let range_builder = match terms.end_bound() {\n            Bound::Included(bound) => range_builder.le(bound.serialized_value_bytes()),\n            Bound::Excluded(bound) => range_builder.lt(bound.serialized_value_bytes()),\n            Bound::Unbounded => range_builder,\n        };\n        let range_builder = if let Some(limit) = limit {\n            range_builder.limit(limit)\n        } else {\n            range_builder\n        };\n\n        let mut stream = range_builder\n            .into_stream_async_merging_holes(merge_holes_under_bytes)\n            .await?;\n\n        let iter = std::iter::from_fn(move || stream.next().map(|(_k, v)| v.clone()));\n\n        // limit on stream is only an optimization to load less data, the stream may still return\n        // more than limit elements.\n        let limit = limit.map(|limit| limit as usize).unwrap_or(usize::MAX);\n        let iter = iter.take(limit);\n\n        Ok(iter)\n    }\n\n    /// Warmup a block postings given a `Term`.\n    /// This method is for an advanced usage only.\n    ///\n    /// returns a boolean, whether the term was found in the dictionary\n    pub async fn warm_postings(&self, term: &Term, with_positions: bool) -> io::Result<bool> {\n        let term_info_opt: Option<TermInfo> = self.get_term_info_async(term).await?;\n        if let Some(term_info) = term_info_opt {\n            let postings = self\n                .postings_file_slice\n                .read_bytes_slice_async(term_info.postings_range.clone());\n            if with_positions {\n                let positions = self\n                    .positions_file_slice\n                    .read_bytes_slice_async(term_info.positions_range.clone());\n                futures_util::future::try_join(postings, positions).await?;\n            } else {\n                postings.await?;\n            }\n            Ok(true)\n        } else {\n            Ok(false)\n        }\n    }\n\n    /// Warmup a block postings given a range of `Term`s.\n    /// This method is for an advanced usage only.\n    ///\n    /// returns a boolean, whether a term matching the range was found in the dictionary\n    pub async fn warm_postings_range(\n        &self,\n        terms: impl std::ops::RangeBounds<Term>,\n        limit: Option<u64>,\n        with_positions: bool,\n    ) -> io::Result<bool> {\n        let mut term_info = self\n            .get_term_range_async(terms, AlwaysMatch, limit, 0)\n            .await?;\n\n        let Some(first_terminfo) = term_info.next() else {\n            // no key matches, nothing more to load\n            return Ok(false);\n        };\n\n        let last_terminfo = term_info.last().unwrap_or_else(|| first_terminfo.clone());\n\n        let postings_range = first_terminfo.postings_range.start..last_terminfo.postings_range.end;\n        let positions_range =\n            first_terminfo.positions_range.start..last_terminfo.positions_range.end;\n\n        let postings = self\n            .postings_file_slice\n            .read_bytes_slice_async(postings_range);\n        if with_positions {\n            let positions = self\n                .positions_file_slice\n                .read_bytes_slice_async(positions_range);\n            futures_util::future::try_join(postings, positions).await?;\n        } else {\n            postings.await?;\n        }\n        Ok(true)\n    }\n\n    /// Warmup a block postings given a range of `Term`s.\n    /// This method is for an advanced usage only.\n    ///\n    /// returns a boolean, whether a term matching the range was found in the dictionary\n    pub async fn warm_postings_automaton<\n        A: Automaton + Clone + Send + 'static,\n        E: FnOnce(Box<dyn FnOnce() -> io::Result<()> + Send>) -> F,\n        F: std::future::Future<Output = io::Result<()>>,\n    >(\n        &self,\n        automaton: A,\n        // with_positions: bool, at the moment we have no use for it, and supporting it would add\n        // complexity to the coalesce\n        executor: E,\n    ) -> io::Result<bool>\n    where\n        A::State: Clone,\n    {\n        // merge holes under 4MiB, that's how many bytes we can hope to receive during a TTFB from\n        // S3 (~80MiB/s, and 50ms latency)\n        const MERGE_HOLES_UNDER_BYTES: usize = (80 * 1024 * 1024 * 50) / 1000;\n        // we build a first iterator to download everything. Simply calling the function already\n        // download everything we need from the sstable, but doesn't start iterating over it.\n        let _term_info_iter = self\n            .get_term_range_async(.., automaton.clone(), None, MERGE_HOLES_UNDER_BYTES)\n            .await?;\n\n        let (sender, posting_ranges_to_load_stream) = futures_channel::mpsc::unbounded();\n        let termdict = self.termdict.clone();\n        let cpu_bound_task = move || {\n            // then we build a 2nd iterator, this one with no holes, so we don't go through blocks\n            // we can't match.\n            // This makes the assumption there is a caching layer below us, which gives sync read\n            // for free after the initial async access. This might not always be true, but is in\n            // Quickwit.\n            // We build things from this closure otherwise we get into lifetime issues that can only\n            // be solved with self referential strucs. Returning an io::Result from here is a bit\n            // more leaky abstraction-wise, but a lot better than the alternative\n            let mut stream = termdict.search(automaton).into_stream()?;\n\n            // we could do without an iterator, but this allows us access to coalesce which simplify\n            // things\n            let posting_ranges_iter =\n                std::iter::from_fn(move || stream.next().map(|(_k, v)| v.postings_range.clone()));\n\n            let merged_posting_ranges_iter = posting_ranges_iter.coalesce(|range1, range2| {\n                if range1.end + MERGE_HOLES_UNDER_BYTES >= range2.start {\n                    Ok(range1.start..range2.end)\n                } else {\n                    Err((range1, range2))\n                }\n            });\n\n            for posting_range in merged_posting_ranges_iter {\n                if let Err(_) = sender.unbounded_send(posting_range) {\n                    // this should happen only when search is cancelled\n                    return Err(io::Error::other(\"failed to send posting range back\"));\n                }\n            }\n            Ok(())\n        };\n        let task_handle = executor(Box::new(cpu_bound_task));\n\n        let posting_downloader = posting_ranges_to_load_stream\n            .map(|posting_slice| {\n                self.postings_file_slice\n                    .read_bytes_slice_async(posting_slice)\n                    .map(|result| result.map(|_slice| ()))\n            })\n            .buffer_unordered(5)\n            .try_collect::<Vec<()>>();\n\n        let (_, slices_downloaded) =\n            futures_util::future::try_join(task_handle, posting_downloader).await?;\n\n        Ok(!slices_downloaded.is_empty())\n    }\n\n    /// Warmup the block postings for all terms.\n    /// This method is for an advanced usage only.\n    ///\n    /// If you know which terms to pre-load, prefer using [`Self::warm_postings`] or\n    /// [`Self::warm_postings`] instead.\n    pub async fn warm_postings_full(&self, with_positions: bool) -> io::Result<()> {\n        self.postings_file_slice.read_bytes_async().await?;\n        if with_positions {\n            self.positions_file_slice.read_bytes_async().await?;\n        }\n        Ok(())\n    }\n\n    /// Returns the number of documents containing the term asynchronously.\n    pub async fn doc_freq_async(&self, term: &Term) -> io::Result<u32> {\n        Ok(self\n            .get_term_info_async(term)\n            .await?\n            .map(|term_info| term_info.doc_freq)\n            .unwrap_or(0u32))\n    }\n}\n"
  },
  {
    "path": "src/index/mod.rs",
    "content": "//! The `index` module in Tantivy contains core components to read and write indexes.\n//!\n//! It contains `Index` and `Segment`, where a `Index` consists of one or more `Segment`s.\n\nmod index;\nmod index_meta;\nmod inverted_index_reader;\nmod segment;\nmod segment_component;\nmod segment_id;\nmod segment_reader;\n\npub use self::index::{Index, IndexBuilder};\npub(crate) use self::index_meta::SegmentMetaInventory;\npub use self::index_meta::{IndexMeta, IndexSettings, Order, SegmentMeta};\npub use self::inverted_index_reader::InvertedIndexReader;\npub use self::segment::Segment;\npub use self::segment_component::SegmentComponent;\npub use self::segment_id::SegmentId;\npub use self::segment_reader::{FieldMetadata, SegmentReader};\n"
  },
  {
    "path": "src/index/segment.rs",
    "content": "use std::fmt;\nuse std::path::PathBuf;\n\nuse super::SegmentComponent;\nuse crate::directory::error::{OpenReadError, OpenWriteError};\nuse crate::directory::{Directory, FileSlice, WritePtr};\nuse crate::index::{Index, SegmentId, SegmentMeta};\nuse crate::schema::Schema;\nuse crate::Opstamp;\n\n/// A segment is a piece of the index.\n#[derive(Clone)]\npub struct Segment {\n    index: Index,\n    meta: SegmentMeta,\n}\n\nimpl fmt::Debug for Segment {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"Segment({:?})\", self.id().uuid_string())\n    }\n}\n\nimpl Segment {\n    /// Creates a new segment given an `Index` and a `SegmentId`\n    pub(crate) fn for_index(index: Index, meta: SegmentMeta) -> Segment {\n        Segment { index, meta }\n    }\n\n    /// Returns the index the segment belongs to.\n    pub fn index(&self) -> &Index {\n        &self.index\n    }\n\n    /// Returns our index's schema.\n    pub fn schema(&self) -> Schema {\n        self.index.schema()\n    }\n\n    /// Returns the segment meta-information\n    pub fn meta(&self) -> &SegmentMeta {\n        &self.meta\n    }\n\n    /// Updates the max_doc value from the `SegmentMeta`.\n    ///\n    /// This method is only used when updating `max_doc` from 0\n    /// as we finalize a fresh new segment.\n    pub fn with_max_doc(self, max_doc: u32) -> Segment {\n        Segment {\n            index: self.index,\n            meta: self.meta.with_max_doc(max_doc),\n        }\n    }\n\n    #[doc(hidden)]\n    #[must_use]\n    pub fn with_delete_meta(self, num_deleted_docs: u32, opstamp: Opstamp) -> Segment {\n        Segment {\n            index: self.index,\n            meta: self.meta.with_delete_meta(num_deleted_docs, opstamp),\n        }\n    }\n\n    /// Returns the segment's id.\n    pub fn id(&self) -> SegmentId {\n        self.meta.id()\n    }\n\n    /// Returns the relative path of a component of our segment.\n    ///\n    /// It just joins the segment id with the extension\n    /// associated with a segment component.\n    pub fn relative_path(&self, component: SegmentComponent) -> PathBuf {\n        self.meta.relative_path(component)\n    }\n\n    /// Open one of the component file for a *regular* read.\n    pub fn open_read(&self, component: SegmentComponent) -> Result<FileSlice, OpenReadError> {\n        let path = self.relative_path(component);\n        self.index.directory().open_read(&path)\n    }\n\n    /// Open one of the component file for *regular* write.\n    pub fn open_write(&mut self, component: SegmentComponent) -> Result<WritePtr, OpenWriteError> {\n        let path = self.relative_path(component);\n        let write = self.index.directory_mut().open_write(&path)?;\n        Ok(write)\n    }\n}\n"
  },
  {
    "path": "src/index/segment_component.rs",
    "content": "use std::slice;\n\n/// Enum describing each component of a tantivy segment.\n///\n/// Each component is stored in its own file,\n/// using the pattern `segment_uuid`.`component_extension`,\n/// except the delete component that takes an `segment_uuid`.`delete_opstamp`.`component_extension`\n#[derive(Copy, Clone, Eq, PartialEq)]\npub enum SegmentComponent {\n    /// Postings (or inverted list). Sorted lists of document ids, associated with terms\n    Postings,\n    /// Positions of terms in each document.\n    Positions,\n    /// Column-oriented random-access storage of fields.\n    FastFields,\n    /// Stores the sum  of the length (in terms) of each field for each document.\n    /// Field norms are stored as a special u64 fast field.\n    FieldNorms,\n    /// Dictionary associating `Term`s to `TermInfo`s which is\n    /// simply an address into the `postings` file and the `positions` file.\n    Terms,\n    /// Row-oriented, compressed storage of the documents.\n    /// Accessing a document from the store is relatively slow, as it\n    /// requires to decompress the entire block it belongs to.\n    Store,\n    /// Bitset describing which document of the segment is alive.\n    /// (It was representing deleted docs but changed to represent alive docs from v0.17)\n    Delete,\n}\n\nimpl SegmentComponent {\n    /// Iterates through the components.\n    pub fn iterator() -> slice::Iter<'static, SegmentComponent> {\n        static SEGMENT_COMPONENTS: [SegmentComponent; 7] = [\n            SegmentComponent::Postings,\n            SegmentComponent::Positions,\n            SegmentComponent::FastFields,\n            SegmentComponent::FieldNorms,\n            SegmentComponent::Terms,\n            SegmentComponent::Store,\n            SegmentComponent::Delete,\n        ];\n        SEGMENT_COMPONENTS.iter()\n    }\n}\n"
  },
  {
    "path": "src/index/segment_id.rs",
    "content": "use std::cmp::Ordering;\nuse std::error::Error;\nuse std::fmt;\nuse std::str::FromStr;\n#[cfg(test)]\nuse std::sync::atomic;\n\n#[cfg(test)]\nuse once_cell::sync::Lazy;\nuse serde::{Deserialize, Serialize};\nuse uuid::Uuid;\n\n/// Uuid identifying a segment.\n///\n/// Tantivy's segment are identified\n/// by a UUID which is used to prefix the filenames\n/// of all of the file associated with the segment.\n///\n/// In unit test, for reproducibility, the `SegmentId` are\n/// simply generated in an autoincrement fashion.\n#[derive(Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]\npub struct SegmentId(Uuid);\n\n#[cfg(test)]\nstatic AUTO_INC_COUNTER: Lazy<atomic::AtomicUsize> = Lazy::new(atomic::AtomicUsize::default);\n\n#[cfg(test)]\nconst ZERO_ARRAY: [u8; 8] = [0u8; 8];\n\n// During tests, we generate the segment id in a autoincrement manner\n// for consistency of segment id between run.\n//\n// The order of the test execution is not guaranteed, but the order\n// of segments within a single test is guaranteed.\n#[cfg(test)]\nfn create_uuid() -> Uuid {\n    let new_auto_inc_id = (*AUTO_INC_COUNTER).fetch_add(1, atomic::Ordering::SeqCst);\n    Uuid::from_fields(new_auto_inc_id as u32, 0, 0, &ZERO_ARRAY)\n}\n\n#[cfg(not(test))]\nfn create_uuid() -> Uuid {\n    Uuid::new_v4()\n}\n\nimpl SegmentId {\n    #[doc(hidden)]\n    pub fn generate_random() -> SegmentId {\n        SegmentId(create_uuid())\n    }\n\n    /// Returns a shorter identifier of the segment.\n    ///\n    /// We are using UUID4, so only 6 bits are fixed,\n    /// and the rest is random.\n    ///\n    /// Picking the first 8 chars is ok to identify\n    /// segments in a display message (e.g. a5c4dfcb).\n    pub fn short_uuid_string(&self) -> String {\n        self.0.as_simple().to_string()[..8].to_string()\n    }\n\n    /// Returns a segment uuid string.\n    ///\n    /// It consists in 32 lowercase hexadecimal chars\n    /// (e.g. a5c4dfcbdfe645089129e308e26d5523)\n    pub fn uuid_string(&self) -> String {\n        self.0.as_simple().to_string()\n    }\n\n    /// Build a `SegmentId` string from the full uuid string.\n    ///\n    /// E.g. \"a5c4dfcbdfe645089129e308e26d5523\"\n    pub fn from_uuid_string(uuid_string: &str) -> Result<SegmentId, SegmentIdParseError> {\n        FromStr::from_str(uuid_string)\n    }\n}\n\n/// Error type used when parsing a `SegmentId` from a string fails.\npub struct SegmentIdParseError(uuid::Error);\n\nimpl Error for SegmentIdParseError {}\n\nimpl fmt::Debug for SegmentIdParseError {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        self.0.fmt(f)\n    }\n}\n\nimpl fmt::Display for SegmentIdParseError {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        self.0.fmt(f)\n    }\n}\n\nimpl FromStr for SegmentId {\n    type Err = SegmentIdParseError;\n\n    fn from_str(uuid_string: &str) -> Result<Self, SegmentIdParseError> {\n        let uuid = Uuid::parse_str(uuid_string).map_err(SegmentIdParseError)?;\n        Ok(SegmentId(uuid))\n    }\n}\n\nimpl fmt::Debug for SegmentId {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"Seg({:?})\", self.short_uuid_string())\n    }\n}\n\nimpl fmt::Display for SegmentId {\n    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(f, \"Seg({:?})\", self.short_uuid_string())\n    }\n}\n\nimpl PartialOrd for SegmentId {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for SegmentId {\n    fn cmp(&self, other: &Self) -> Ordering {\n        self.0.as_bytes().cmp(other.0.as_bytes())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::SegmentId;\n\n    #[test]\n    fn test_to_uuid_string() {\n        let full_uuid = \"a5c4dfcbdfe645089129e308e26d5523\";\n        let segment_id = SegmentId::from_uuid_string(full_uuid).unwrap();\n        assert_eq!(segment_id.uuid_string(), full_uuid);\n        assert_eq!(segment_id.short_uuid_string(), \"a5c4dfcb\");\n        // one extra char\n        assert!(SegmentId::from_uuid_string(\"a5c4dfcbdfe645089129e308e26d5523b\").is_err());\n    }\n}\n"
  },
  {
    "path": "src/index/segment_reader.rs",
    "content": "use std::collections::HashMap;\nuse std::sync::{Arc, RwLock};\nuse std::{fmt, io};\n\nuse common::{ByteCount, HasLen};\nuse fnv::FnvHashMap;\nuse itertools::Itertools;\n\nuse crate::directory::{CompositeFile, FileSlice};\nuse crate::error::DataCorruption;\nuse crate::fastfield::{intersect_alive_bitsets, AliveBitSet, FacetReader, FastFieldReaders};\nuse crate::fieldnorm::{FieldNormReader, FieldNormReaders};\nuse crate::index::{InvertedIndexReader, Segment, SegmentComponent, SegmentId};\nuse crate::json_utils::json_path_sep_to_dot;\nuse crate::schema::{Field, IndexRecordOption, Schema, Type};\nuse crate::space_usage::SegmentSpaceUsage;\nuse crate::store::StoreReader;\nuse crate::termdict::TermDictionary;\nuse crate::{DocId, Opstamp};\n\n/// Entry point to access all of the datastructures of the `Segment`\n///\n/// - term dictionary\n/// - postings\n/// - store\n/// - fast field readers\n/// - field norm reader\n///\n/// The segment reader has a very low memory footprint,\n/// as close to all of the memory data is mmapped.\n#[derive(Clone)]\npub struct SegmentReader {\n    inv_idx_reader_cache: Arc<RwLock<HashMap<Field, Arc<InvertedIndexReader>>>>,\n\n    segment_id: SegmentId,\n    delete_opstamp: Option<Opstamp>,\n\n    max_doc: DocId,\n    num_docs: DocId,\n\n    termdict_composite: CompositeFile,\n    postings_composite: CompositeFile,\n    positions_composite: CompositeFile,\n    fast_fields_readers: FastFieldReaders,\n    fieldnorm_readers: FieldNormReaders,\n\n    store_file: FileSlice,\n    alive_bitset_opt: Option<AliveBitSet>,\n    schema: Schema,\n}\n\nimpl SegmentReader {\n    /// Returns the highest document id ever attributed in\n    /// this segment + 1.\n    pub fn max_doc(&self) -> DocId {\n        self.max_doc\n    }\n\n    /// Returns the number of alive documents.\n    /// Deleted documents are not counted.\n    pub fn num_docs(&self) -> DocId {\n        self.num_docs\n    }\n\n    /// Returns the schema of the index this segment belongs to.\n    pub fn schema(&self) -> &Schema {\n        &self.schema\n    }\n\n    /// Return the number of documents that have been\n    /// deleted in the segment.\n    pub fn num_deleted_docs(&self) -> DocId {\n        self.max_doc - self.num_docs\n    }\n\n    /// Returns true if some of the documents of the segment have been deleted.\n    pub fn has_deletes(&self) -> bool {\n        self.num_deleted_docs() > 0\n    }\n\n    /// Accessor to a segment's fast field reader given a field.\n    ///\n    /// Returns the u64 fast value reader if the field\n    /// is a u64 field indexed as \"fast\".\n    ///\n    /// Return a FastFieldNotAvailableError if the field is not\n    /// declared as a fast field in the schema.\n    ///\n    /// # Panics\n    /// May panic if the index is corrupted.\n    pub fn fast_fields(&self) -> &FastFieldReaders {\n        &self.fast_fields_readers\n    }\n\n    /// Accessor to the `FacetReader` associated with a given `Field`.\n    pub fn facet_reader(&self, field_name: &str) -> crate::Result<FacetReader> {\n        let schema = self.schema();\n        let field = schema.get_field(field_name)?;\n        let field_entry = schema.get_field_entry(field);\n        if field_entry.field_type().value_type() != Type::Facet {\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"`{field_name}` is not a facet field.`\"\n            )));\n        }\n        let Some(facet_column) = self.fast_fields().str(field_name)? else {\n            panic!(\"Facet Field `{field_name}` is missing. This should not happen\");\n        };\n        Ok(FacetReader::new(facet_column))\n    }\n\n    /// Accessor to the segment's `Field norms`'s reader.\n    ///\n    /// Field norms are the length (in tokens) of the fields.\n    /// It is used in the computation of the [TfIdf](https://fulmicoton.gitbooks.io/tantivy-doc/content/tfidf.html).\n    ///\n    /// They are simply stored as a fast field, serialized in\n    /// the `.fieldnorm` file of the segment.\n    pub fn get_fieldnorms_reader(&self, field: Field) -> crate::Result<FieldNormReader> {\n        self.fieldnorm_readers.get_field(field)?.ok_or_else(|| {\n            let field_name = self.schema.get_field_name(field);\n            let err_msg = format!(\n                \"Field norm not found for field {field_name:?}. Was the field set to record norm \\\n                 during indexing?\"\n            );\n            crate::TantivyError::SchemaError(err_msg)\n        })\n    }\n\n    #[doc(hidden)]\n    pub fn fieldnorms_readers(&self) -> &FieldNormReaders {\n        &self.fieldnorm_readers\n    }\n\n    /// Accessor to the segment's [`StoreReader`](crate::store::StoreReader).\n    ///\n    /// `cache_num_blocks` sets the number of decompressed blocks to be cached in an LRU.\n    /// The size of blocks is configurable, this should be reflexted in the\n    pub fn get_store_reader(&self, cache_num_blocks: usize) -> io::Result<StoreReader> {\n        StoreReader::open(self.store_file.clone(), cache_num_blocks)\n    }\n\n    /// Open a new segment for reading.\n    pub fn open(segment: &Segment) -> crate::Result<SegmentReader> {\n        Self::open_with_custom_alive_set(segment, None)\n    }\n\n    /// Open a new segment for reading.\n    pub fn open_with_custom_alive_set(\n        segment: &Segment,\n        custom_bitset: Option<AliveBitSet>,\n    ) -> crate::Result<SegmentReader> {\n        let termdict_file = segment.open_read(SegmentComponent::Terms)?;\n        let termdict_composite = CompositeFile::open(&termdict_file)?;\n\n        let store_file = segment.open_read(SegmentComponent::Store)?;\n\n        crate::fail_point!(\"SegmentReader::open#middle\");\n\n        let postings_file = segment.open_read(SegmentComponent::Postings)?;\n        let postings_composite = CompositeFile::open(&postings_file)?;\n\n        let positions_composite = {\n            if let Ok(positions_file) = segment.open_read(SegmentComponent::Positions) {\n                CompositeFile::open(&positions_file)?\n            } else {\n                CompositeFile::empty()\n            }\n        };\n\n        let schema = segment.schema();\n\n        let fast_fields_data = segment.open_read(SegmentComponent::FastFields)?;\n        let fast_fields_readers = FastFieldReaders::open(fast_fields_data, schema.clone())?;\n        let fieldnorm_data = segment.open_read(SegmentComponent::FieldNorms)?;\n        let fieldnorm_readers = FieldNormReaders::open(fieldnorm_data)?;\n\n        let original_bitset = if segment.meta().has_deletes() {\n            let alive_doc_file_slice = segment.open_read(SegmentComponent::Delete)?;\n            let alive_doc_data = alive_doc_file_slice.read_bytes()?;\n            Some(AliveBitSet::open(alive_doc_data))\n        } else {\n            None\n        };\n\n        let alive_bitset_opt = intersect_alive_bitset(original_bitset, custom_bitset);\n\n        let max_doc = segment.meta().max_doc();\n        let num_docs = alive_bitset_opt\n            .as_ref()\n            .map(|alive_bitset| alive_bitset.num_alive_docs() as u32)\n            .unwrap_or(max_doc);\n\n        Ok(SegmentReader {\n            inv_idx_reader_cache: Default::default(),\n            num_docs,\n            max_doc,\n            termdict_composite,\n            postings_composite,\n            fast_fields_readers,\n            fieldnorm_readers,\n            segment_id: segment.id(),\n            delete_opstamp: segment.meta().delete_opstamp(),\n            store_file,\n            alive_bitset_opt,\n            positions_composite,\n            schema,\n        })\n    }\n\n    /// Returns a field reader associated with the field given in argument.\n    /// If the field was not present in the index during indexing time,\n    /// the InvertedIndexReader is empty.\n    ///\n    /// The field reader is in charge of iterating through the\n    /// term dictionary associated with a specific field,\n    /// and opening the posting list associated with any term.\n    ///\n    /// If the field is not marked as index, a warning is logged and an empty `InvertedIndexReader`\n    /// is returned.\n    /// Similarly, if the field is marked as indexed but no term has been indexed for the given\n    /// index, an empty `InvertedIndexReader` is returned (but no warning is logged).\n    pub fn inverted_index(&self, field: Field) -> crate::Result<Arc<InvertedIndexReader>> {\n        if let Some(inv_idx_reader) = self\n            .inv_idx_reader_cache\n            .read()\n            .expect(\"Lock poisoned. This should never happen\")\n            .get(&field)\n        {\n            return Ok(Arc::clone(inv_idx_reader));\n        }\n        let field_entry = self.schema.get_field_entry(field);\n        let field_type = field_entry.field_type();\n        let record_option_opt = field_type.get_index_record_option();\n\n        if record_option_opt.is_none() {\n            warn!(\"Field {:?} does not seem indexed.\", field_entry.name());\n        }\n\n        let postings_file_opt = self.postings_composite.open_read(field);\n\n        if postings_file_opt.is_none() || record_option_opt.is_none() {\n            // no documents in the segment contained this field.\n            // As a result, no data is associated with the inverted index.\n            //\n            // Returns an empty inverted index.\n            let record_option = record_option_opt.unwrap_or(IndexRecordOption::Basic);\n            return Ok(Arc::new(InvertedIndexReader::empty(record_option)));\n        }\n\n        let record_option = record_option_opt.unwrap();\n        let postings_file = postings_file_opt.unwrap();\n\n        let termdict_file: FileSlice =\n            self.termdict_composite.open_read(field).ok_or_else(|| {\n                DataCorruption::comment_only(format!(\n                    \"Failed to open field {:?}'s term dictionary in the composite file. Has the \\\n                     schema been modified?\",\n                    field_entry.name()\n                ))\n            })?;\n\n        let positions_file = self.positions_composite.open_read(field).ok_or_else(|| {\n            let error_msg = format!(\n                \"Failed to open field {:?}'s positions in the composite file. Has the schema been \\\n                 modified?\",\n                field_entry.name()\n            );\n            DataCorruption::comment_only(error_msg)\n        })?;\n\n        let inv_idx_reader = Arc::new(InvertedIndexReader::new(\n            TermDictionary::open(termdict_file)?,\n            postings_file,\n            positions_file,\n            record_option,\n        )?);\n\n        // by releasing the lock in between, we may end up opening the inverting index\n        // twice, but this is fine.\n        self.inv_idx_reader_cache\n            .write()\n            .expect(\"Field reader cache lock poisoned. This should never happen.\")\n            .insert(field, Arc::clone(&inv_idx_reader));\n\n        Ok(inv_idx_reader)\n    }\n\n    /// Returns the list of fields that have been indexed in the segment.\n    /// The field list includes the field defined in the schema as well as the fields\n    /// that have been indexed as a part of a JSON field.\n    /// The returned field name is the full field name, including the name of the JSON field.\n    ///\n    /// The returned field names can be used in queries.\n    ///\n    /// Notice: If your data contains JSON fields this is **very expensive**, as it requires\n    /// browsing through the inverted index term dictionary and the columnar field dictionary.\n    ///\n    /// Disclaimer: Some fields may not be listed here. For instance, if the schema contains a json\n    /// field that is not indexed nor a fast field but is stored, it is possible for the field\n    /// to not be listed.\n    pub fn fields_metadata(&self) -> crate::Result<Vec<FieldMetadata>> {\n        let mut indexed_fields: Vec<FieldMetadata> = Vec::new();\n        let mut map_to_canonical = FnvHashMap::default();\n        for (field, field_entry) in self.schema().fields() {\n            let field_name = field_entry.name().to_string();\n            let is_indexed = field_entry.is_indexed();\n            if is_indexed {\n                let is_json = field_entry.field_type().value_type() == Type::Json;\n                if is_json {\n                    let term_dictionary_json_field_num_bytes: u64 = self\n                        .termdict_composite\n                        .open_read(field)\n                        .map(|file_slice| file_slice.len() as u64)\n                        .unwrap_or(0u64);\n                    let inv_index = self.inverted_index(field)?;\n                    let encoded_fields_in_index = inv_index.list_encoded_json_fields()?;\n                    let mut build_path = |field_name: &str, mut json_path: String| {\n                        // In this case we need to map the potential fast field to the field name\n                        // accepted by the query parser.\n                        let create_canonical =\n                            !field_entry.is_expand_dots_enabled() && json_path.contains('.');\n                        if create_canonical {\n                            // Without expand dots enabled dots need to be escaped.\n                            let escaped_json_path = json_path.replace('.', \"\\\\.\");\n                            let full_path = format!(\"{field_name}.{escaped_json_path}\");\n                            let full_path_unescaped = format!(\"{}.{}\", field_name, &json_path);\n                            map_to_canonical.insert(full_path_unescaped, full_path.to_string());\n                            full_path\n                        } else {\n                            // With expand dots enabled, we can use '.' instead of '\\u{1}'.\n                            json_path_sep_to_dot(&mut json_path);\n                            format!(\"{field_name}.{json_path}\")\n                        }\n                    };\n                    let total_num_terms = encoded_fields_in_index\n                        .iter()\n                        .map(|field_space| field_space.num_terms)\n                        .sum();\n                    indexed_fields.extend(encoded_fields_in_index.into_iter().map(|field_space| {\n                        let field_name = build_path(&field_name, field_space.field_name);\n                        // It is complex to attribute the exact amount of bytes required by specific\n                        // field in the json field. Instead, as a proxy, we\n                        // attribute the total amount of bytes for the entire json field,\n                        // proportionally to the number of terms in each\n                        // fields.\n                        let term_dictionary_size = (term_dictionary_json_field_num_bytes\n                            * field_space.num_terms)\n                            .checked_div(total_num_terms)\n                            .unwrap_or(0);\n                        FieldMetadata {\n                            postings_size: Some(field_space.postings_size),\n                            positions_size: Some(field_space.positions_size),\n                            term_dictionary_size: Some(ByteCount::from(term_dictionary_size)),\n                            fast_size: None,\n                            // The stored flag will be set at the end of this function!\n                            stored: field_entry.is_stored(),\n                            field_name,\n                            typ: field_space.field_type,\n                        }\n                    }));\n                } else {\n                    let postings_size: ByteCount = self\n                        .postings_composite\n                        .open_read(field)\n                        .map(|posting_fileslice| posting_fileslice.len())\n                        .unwrap_or(0)\n                        .into();\n                    let positions_size: ByteCount = self\n                        .positions_composite\n                        .open_read(field)\n                        .map(|positions_fileslice| positions_fileslice.len())\n                        .unwrap_or(0)\n                        .into();\n                    let term_dictionary_size: ByteCount = self\n                        .termdict_composite\n                        .open_read(field)\n                        .map(|term_dictionary_fileslice| term_dictionary_fileslice.len())\n                        .unwrap_or(0)\n                        .into();\n                    indexed_fields.push(FieldMetadata {\n                        field_name: field_name.to_string(),\n                        typ: field_entry.field_type().value_type(),\n                        // The stored flag will be set at the end of this function!\n                        stored: field_entry.is_stored(),\n                        fast_size: None,\n                        term_dictionary_size: Some(term_dictionary_size),\n                        postings_size: Some(postings_size),\n                        positions_size: Some(positions_size),\n                    });\n                }\n            }\n        }\n        let fast_fields: Vec<FieldMetadata> = self\n            .fast_fields()\n            .columnar()\n            .iter_columns()?\n            .map(|(mut field_name, handle)| {\n                json_path_sep_to_dot(&mut field_name);\n                // map to canonical path, to avoid similar but different entries.\n                // Eventually we should just accept '.' separated for all cases.\n                let field_name = map_to_canonical\n                    .get(&field_name)\n                    .unwrap_or(&field_name)\n                    .to_string();\n                let stored = is_field_stored(&field_name, &self.schema);\n                FieldMetadata {\n                    field_name,\n                    typ: Type::from(handle.column_type()),\n                    stored,\n                    fast_size: Some(handle.num_bytes()),\n                    term_dictionary_size: None,\n                    postings_size: None,\n                    positions_size: None,\n                }\n            })\n            .collect();\n        let merged_field_metadatas: Vec<FieldMetadata> =\n            merge_field_meta_data(vec![indexed_fields, fast_fields]);\n        Ok(merged_field_metadatas)\n    }\n\n    /// Returns the segment id\n    pub fn segment_id(&self) -> SegmentId {\n        self.segment_id\n    }\n\n    /// Returns the delete opstamp\n    pub fn delete_opstamp(&self) -> Option<Opstamp> {\n        self.delete_opstamp\n    }\n\n    /// Returns the bitset representing the alive `DocId`s.\n    pub fn alive_bitset(&self) -> Option<&AliveBitSet> {\n        self.alive_bitset_opt.as_ref()\n    }\n\n    /// Returns true if the `doc` is marked\n    /// as deleted.\n    pub fn is_deleted(&self, doc: DocId) -> bool {\n        self.alive_bitset()\n            .map(|alive_bitset| alive_bitset.is_deleted(doc))\n            .unwrap_or(false)\n    }\n\n    /// Returns an iterator that will iterate over the alive document ids\n    pub fn doc_ids_alive(&self) -> Box<dyn Iterator<Item = DocId> + Send + '_> {\n        if let Some(alive_bitset) = &self.alive_bitset_opt {\n            Box::new(alive_bitset.iter_alive())\n        } else {\n            Box::new(0u32..self.max_doc)\n        }\n    }\n\n    /// Summarize total space usage of this segment.\n    pub fn space_usage(&self) -> io::Result<SegmentSpaceUsage> {\n        Ok(SegmentSpaceUsage::new(\n            self.num_docs(),\n            self.termdict_composite.space_usage(self.schema()),\n            self.postings_composite.space_usage(self.schema()),\n            self.positions_composite.space_usage(self.schema()),\n            self.fast_fields_readers.space_usage()?,\n            self.fieldnorm_readers.space_usage(self.schema()),\n            self.get_store_reader(0)?.space_usage(),\n            self.alive_bitset_opt\n                .as_ref()\n                .map(AliveBitSet::space_usage)\n                .unwrap_or_default(),\n        ))\n    }\n}\n\n#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]\n/// FieldMetadata\npub struct FieldMetadata {\n    /// The field name\n    // Notice: Don't reorder the declaration of 1.field_name 2.typ, as it is used for ordering by\n    // field_name then typ.\n    pub field_name: String,\n    /// The field type\n    // Notice: Don't reorder the declaration of 1.field_name 2.typ, as it is used for ordering by\n    // field_name then typ.\n    pub typ: Type,\n    /// Is the field stored in the doc store\n    pub stored: bool,\n    /// Size occupied in the columnar storage (None if not fast)\n    pub fast_size: Option<ByteCount>,\n    /// term_dictionary\n    pub term_dictionary_size: Option<ByteCount>,\n    /// Size occupied in the index postings storage (None if not indexed)\n    pub postings_size: Option<ByteCount>,\n    /// Size occupied in the index postings storage (None if positions are not recorded)\n    pub positions_size: Option<ByteCount>,\n}\n\nfn merge_options(left: Option<ByteCount>, right: Option<ByteCount>) -> Option<ByteCount> {\n    match (left, right) {\n        (Some(l), Some(r)) => Some(l + r),\n        (None, right) => right,\n        (left, None) => left,\n    }\n}\n\nimpl FieldMetadata {\n    /// Returns true if and only if the field is indexed.\n    pub fn is_indexed(&self) -> bool {\n        self.postings_size.is_some()\n    }\n\n    /// Returns true if and only if the field is a fast field (i.e.: recorded in  columnar format).\n    pub fn is_fast(&self) -> bool {\n        self.fast_size.is_some()\n    }\n\n    /// Merges two field metadata.\n    pub fn merge(&mut self, rhs: Self) {\n        assert_eq!(self.field_name, rhs.field_name);\n        assert_eq!(self.typ, rhs.typ);\n        self.stored |= rhs.stored;\n        self.fast_size = merge_options(self.fast_size, rhs.fast_size);\n        self.term_dictionary_size =\n            merge_options(self.term_dictionary_size, rhs.term_dictionary_size);\n        self.postings_size = merge_options(self.postings_size, rhs.postings_size);\n        self.positions_size = merge_options(self.positions_size, rhs.positions_size);\n    }\n}\n\n// Maybe too slow for the high cardinality case\nfn is_field_stored(field_name: &str, schema: &Schema) -> bool {\n    schema\n        .find_field(field_name)\n        .map(|(field, _path)| schema.get_field_entry(field).is_stored())\n        .unwrap_or(false)\n}\n\n/// Helper to merge the field metadata from multiple segments.\npub fn merge_field_meta_data(mut field_metadatas: Vec<Vec<FieldMetadata>>) -> Vec<FieldMetadata> {\n    // READ BEFORE REMOVING THIS!\n    //\n    // Because we replace field sep by `.`, fields are not always sorted.\n    // Also, to enforce such an implicit contract, we would have to add\n    // assert here.\n    //\n    // Sorting is linear time on pre-sorted data, so we are simply better off sorting data here.\n    for field_metadatas in &mut field_metadatas {\n        field_metadatas.sort_unstable();\n    }\n    let mut merged_field_metadata = Vec::new();\n    for (_key, mut group) in &field_metadatas\n        .into_iter()\n        .kmerge()\n        // TODO: Remove allocation\n        .chunk_by(|el| (el.field_name.to_string(), el.typ))\n    {\n        let mut merged: FieldMetadata = group.next().unwrap();\n        for el in group {\n            merged.merge(el);\n        }\n        // Currently is_field_stored is maybe too slow for the high cardinality case\n        merged_field_metadata.push(merged);\n    }\n    merged_field_metadata\n}\n\nfn intersect_alive_bitset(\n    left_opt: Option<AliveBitSet>,\n    right_opt: Option<AliveBitSet>,\n) -> Option<AliveBitSet> {\n    match (left_opt, right_opt) {\n        (Some(left), Some(right)) => {\n            assert_eq!(left.bitset().max_value(), right.bitset().max_value());\n            Some(intersect_alive_bitsets(left, right))\n        }\n        (Some(left), None) => Some(left),\n        (None, Some(right)) => Some(right),\n        (None, None) => None,\n    }\n}\n\nimpl fmt::Debug for SegmentReader {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"SegmentReader({:?})\", self.segment_id)\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n    use crate::index::Index;\n    use crate::schema::{Term, STORED, TEXT};\n    use crate::IndexWriter;\n\n    #[track_caller]\n    fn assert_merge(fields_metadatas: &[Vec<FieldMetadata>], expected: &[FieldMetadata]) {\n        use itertools::Itertools;\n        let num_els = fields_metadatas.len();\n        for permutation in fields_metadatas.iter().cloned().permutations(num_els) {\n            let res = merge_field_meta_data(permutation);\n            assert_eq!(&res, &expected);\n        }\n    }\n\n    #[test]\n    fn test_merge_field_meta_data_same_field() {\n        let field_metadata1 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            stored: false,\n            term_dictionary_size: Some(ByteCount::from(100u64)),\n            postings_size: Some(ByteCount::from(1_000u64)),\n            positions_size: Some(ByteCount::from(2_000u64)),\n            fast_size: Some(ByteCount::from(1_000u64)),\n        };\n        let field_metadata2 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            stored: false,\n            term_dictionary_size: Some(ByteCount::from(80u64)),\n            postings_size: Some(ByteCount::from(1_500u64)),\n            positions_size: Some(ByteCount::from(2_500u64)),\n            fast_size: Some(ByteCount::from(3_000u64)),\n        };\n        let expected = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            stored: false,\n            term_dictionary_size: Some(ByteCount::from(180u64)),\n            postings_size: Some(ByteCount::from(2_500u64)),\n            positions_size: Some(ByteCount::from(4_500u64)),\n            fast_size: Some(ByteCount::from(4_000u64)),\n        };\n        assert_merge(\n            &[vec![field_metadata1.clone()], vec![field_metadata2]],\n            &[expected],\n        );\n    }\n\n    #[track_caller]\n    #[test]\n    fn test_merge_field_meta_data_different() {\n        let field_metadata1 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            stored: false,\n            fast_size: Some(1_000u64.into()),\n            term_dictionary_size: Some(100u64.into()),\n            postings_size: Some(2_000u64.into()),\n            positions_size: Some(4_000u64.into()),\n        };\n        let field_metadata2 = FieldMetadata {\n            field_name: \"b\".to_string(),\n            typ: crate::schema::Type::Str,\n            stored: false,\n            fast_size: Some(1_002u64.into()),\n            term_dictionary_size: None,\n            postings_size: None,\n            positions_size: None,\n        };\n        let field_metadata3 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            term_dictionary_size: Some(101u64.into()),\n            postings_size: Some(2_001u64.into()),\n            positions_size: Some(4_001u64.into()),\n            stored: false,\n            fast_size: None,\n        };\n        let expected = vec![\n            FieldMetadata {\n                field_name: \"a\".to_string(),\n                typ: crate::schema::Type::Str,\n                stored: false,\n                term_dictionary_size: Some(201u64.into()),\n                postings_size: Some(4_001u64.into()),\n                positions_size: Some(8_001u64.into()),\n                fast_size: Some(1_000u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"b\".to_string(),\n                typ: crate::schema::Type::Str,\n                stored: false,\n                term_dictionary_size: None,\n                postings_size: None,\n                positions_size: None,\n                fast_size: Some(1_002u64.into()),\n            },\n        ];\n        assert_merge(\n            &[\n                vec![field_metadata1.clone(), field_metadata2.clone()],\n                vec![field_metadata3],\n            ],\n            &expected,\n        );\n    }\n\n    #[test]\n    fn test_merge_field_meta_data_merge() {\n        let get_meta_data = |name: &str, typ: Type| FieldMetadata {\n            field_name: name.to_string(),\n            typ,\n            term_dictionary_size: None,\n            postings_size: None,\n            positions_size: None,\n            stored: false,\n            fast_size: Some(1u64.into()),\n        };\n        let metas = vec![get_meta_data(\"d\", Type::Str), get_meta_data(\"e\", Type::U64)];\n        assert_merge(\n            &[vec![get_meta_data(\"e\", Type::Str)], metas],\n            &[\n                get_meta_data(\"d\", Type::Str),\n                get_meta_data(\"e\", Type::Str),\n                get_meta_data(\"e\", Type::U64),\n            ],\n        );\n    }\n\n    #[test]\n    fn test_merge_field_meta_data_bitxor() {\n        let field_metadata1 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            term_dictionary_size: None,\n            postings_size: None,\n            positions_size: None,\n            stored: false,\n            fast_size: Some(10u64.into()),\n        };\n        let field_metadata2 = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            term_dictionary_size: Some(10u64.into()),\n            postings_size: Some(11u64.into()),\n            positions_size: Some(12u64.into()),\n            stored: false,\n            fast_size: None,\n        };\n        let field_metadata_expected = FieldMetadata {\n            field_name: \"a\".to_string(),\n            typ: crate::schema::Type::Str,\n            term_dictionary_size: Some(10u64.into()),\n            postings_size: Some(11u64.into()),\n            positions_size: Some(12u64.into()),\n            stored: false,\n            fast_size: Some(10u64.into()),\n        };\n        let mut res1 = field_metadata1.clone();\n        res1.merge(field_metadata2.clone());\n        let mut res2 = field_metadata2.clone();\n        res2.merge(field_metadata1);\n        assert_eq!(res1, field_metadata_expected);\n        assert_eq!(res2, field_metadata_expected);\n    }\n\n    #[test]\n    fn test_num_alive() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"name\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let name = schema.get_field(\"name\").unwrap();\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => \"tantivy\"))?;\n            index_writer.add_document(doc!(name => \"horse\"))?;\n            index_writer.add_document(doc!(name => \"jockey\"))?;\n            index_writer.add_document(doc!(name => \"cap\"))?;\n            // we should now have one segment with two docs\n            index_writer.delete_term(Term::from_field_text(name, \"horse\"));\n            index_writer.delete_term(Term::from_field_text(name, \"cap\"));\n\n            // ok, now we should have a deleted doc\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        assert_eq!(2, searcher.segment_reader(0).num_docs());\n        assert_eq!(4, searcher.segment_reader(0).max_doc());\n        Ok(())\n    }\n\n    #[test]\n    fn test_alive_docs_iterator() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"name\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let name = schema.get_field(\"name\").unwrap();\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => \"tantivy\"))?;\n            index_writer.add_document(doc!(name => \"horse\"))?;\n            index_writer.add_document(doc!(name => \"jockey\"))?;\n            index_writer.add_document(doc!(name => \"cap\"))?;\n            // we should now have one segment with two docs\n            index_writer.commit()?;\n        }\n\n        {\n            let mut index_writer2: IndexWriter = index.writer(50_000_000)?;\n            index_writer2.delete_term(Term::from_field_text(name, \"horse\"));\n            index_writer2.delete_term(Term::from_field_text(name, \"cap\"));\n\n            // ok, now we should have a deleted doc\n            index_writer2.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let docs: Vec<DocId> = searcher.segment_reader(0).doc_ids_alive().collect();\n        assert_eq!(vec![0u32, 2u32], docs);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/indexer/delete_queue.rs",
    "content": "use std::ops::DerefMut;\nuse std::sync::{Arc, RwLock, Weak};\n\nuse super::operation::DeleteOperation;\nuse crate::Opstamp;\n\n/// The DeleteQueue is similar in conceptually to a multiple\n/// consumer single producer broadcast channel.\n///\n/// All consumer will receive all messages.\n///\n/// Consumer of the delete queue are holding a `DeleteCursor`,\n/// which points to a specific place of the `DeleteQueue`.\n///\n/// New consumer can be created in two ways\n/// - calling `delete_queue.cursor()` returns a cursor, that will include all future delete\n///   operation (and some or none of the past operations... The client is in charge of checking the\n///   opstamps.).\n/// - cloning an existing cursor returns a new cursor, that is at the exact same position, and can\n///   now advance independently from the original cursor.\n#[derive(Default)]\nstruct InnerDeleteQueue {\n    writer: Vec<DeleteOperation>,\n    last_block: Weak<Block>,\n}\n\n/// The delete queue is a linked list storing delete operations.\n///\n/// Several consumers can hold a reference to it. Delete operations\n/// get dropped/gc'ed when no more consumers are holding a reference\n/// to them.\n#[derive(Clone, Default)]\npub struct DeleteQueue {\n    inner: Arc<RwLock<InnerDeleteQueue>>,\n}\n\nimpl DeleteQueue {\n    fn get_last_block(&self) -> Arc<Block> {\n        {\n            // try get the last block with simply acquiring the read lock.\n            let rlock = self.inner.read().unwrap();\n            if let Some(block) = rlock.last_block.upgrade() {\n                return block;\n            }\n        }\n        // It failed. Let's double check after acquiring the write, as someone could have called\n        // `get_last_block` right after we released the rlock.\n        let mut wlock = self.inner.write().unwrap();\n        if let Some(block) = wlock.last_block.upgrade() {\n            return block;\n        }\n        let block = Arc::new(Block {\n            operations: Arc::new([]),\n            next: NextBlock::from(self.clone()),\n        });\n        wlock.last_block = Arc::downgrade(&block);\n        block\n    }\n\n    /// Creates a new cursor that makes it possible to\n    /// consume future delete operations.\n    ///\n    /// Past delete operations are not accessible.\n    pub fn cursor(&self) -> DeleteCursor {\n        let last_block = self.get_last_block();\n        let operations_len = last_block.operations.len();\n        DeleteCursor {\n            block: last_block,\n            pos: operations_len,\n        }\n    }\n\n    /// Appends a new delete operations.\n    pub fn push(&self, delete_operation: DeleteOperation) {\n        self.inner\n            .write()\n            .expect(\"Failed to acquire write lock on delete queue writer\")\n            .writer\n            .push(delete_operation);\n    }\n\n    // DeleteQueue is a linked list of blocks of\n    // delete operations.\n    //\n    // Writing happens by simply appending to a vec.\n    // `.flush()` takes this pending delete operations vec\n    // creates a new read-only block from it,\n    // and appends it to the linked list.\n    //\n    // `.flush()` happens when, for instance,\n    // a consumer reaches the last read-only operations.\n    // It then ask the delete queue if there happen to\n    // be some unflushed operations.\n    //\n    fn flush(&self) -> Option<Arc<Block>> {\n        let mut self_wlock = self\n            .inner\n            .write()\n            .expect(\"Failed to acquire write lock on delete queue writer\");\n\n        if self_wlock.writer.is_empty() {\n            return None;\n        }\n\n        let delete_operations = std::mem::take(&mut self_wlock.writer);\n\n        let new_block = Arc::new(Block {\n            operations: Arc::from(delete_operations.into_boxed_slice()),\n            next: NextBlock::from(self.clone()),\n        });\n\n        self_wlock.last_block = Arc::downgrade(&new_block);\n        Some(new_block)\n    }\n}\n\nenum InnerNextBlock {\n    Writer(DeleteQueue),\n    Closed(Arc<Block>),\n}\n\nstruct NextBlock(RwLock<InnerNextBlock>);\n\nimpl From<DeleteQueue> for NextBlock {\n    fn from(delete_queue: DeleteQueue) -> NextBlock {\n        NextBlock(RwLock::new(InnerNextBlock::Writer(delete_queue)))\n    }\n}\n\nimpl NextBlock {\n    fn next_block(&self) -> Option<Arc<Block>> {\n        {\n            let next_read_lock = self\n                .0\n                .read()\n                .expect(\"Failed to acquire write lock in delete queue\");\n            if let InnerNextBlock::Closed(ref block) = *next_read_lock {\n                return Some(Arc::clone(block));\n            }\n        }\n        let next_block;\n        {\n            let mut next_write_lock = self\n                .0\n                .write()\n                .expect(\"Failed to acquire write lock in delete queue\");\n            match *next_write_lock {\n                InnerNextBlock::Closed(ref block) => {\n                    return Some(Arc::clone(block));\n                }\n                InnerNextBlock::Writer(ref writer) => match writer.flush() {\n                    Some(flushed_next_block) => {\n                        next_block = flushed_next_block;\n                    }\n                    None => {\n                        return None;\n                    }\n                },\n            }\n            *next_write_lock.deref_mut() = InnerNextBlock::Closed(Arc::clone(&next_block));\n            Some(next_block)\n        }\n    }\n}\n\nstruct Block {\n    operations: Arc<[DeleteOperation]>,\n    next: NextBlock,\n}\n\n/// As we process delete operations, keeps track of our position.\n#[derive(Clone)]\npub struct DeleteCursor {\n    block: Arc<Block>,\n    pos: usize,\n}\n\nimpl DeleteCursor {\n    /// Skips operations and position it so that\n    /// - either all of the delete operation currently in the queue are consume and the next get\n    ///   will return `None`.\n    /// - the next get will return the first operation with an `opstamp >= target_opstamp`.\n    pub fn skip_to(&mut self, target_opstamp: Opstamp) {\n        // TODO Can be optimize as we work with block.\n        while self.is_behind_opstamp(target_opstamp) {\n            self.advance();\n        }\n    }\n\n    fn is_behind_opstamp(&mut self, target_opstamp: Opstamp) -> bool {\n        self.get()\n            .map(|operation| operation.opstamp < target_opstamp)\n            .unwrap_or(false)\n    }\n\n    /// If the current block has been entirely\n    /// consumed, try to load the next one.\n    ///\n    /// Return `true`, if after this attempt,\n    /// the cursor is on a block that has not\n    /// been entirely consumed.\n    /// Return `false`, if we have reached the end of the queue.\n    fn load_block_if_required(&mut self) -> bool {\n        if self.pos >= self.block.operations.len() {\n            // we have consumed our operations entirely.\n            // let's ask our writer if he has more for us.\n            // self.go_next_block();\n            match self.block.next.next_block() {\n                Some(block) => {\n                    self.block = block;\n                    self.pos = 0;\n                    true\n                }\n                None => false,\n            }\n        } else {\n            true\n        }\n    }\n\n    /// Advance to the next delete operation.\n    /// Returns true if and only if there is such an operation.\n    pub fn advance(&mut self) -> bool {\n        if self.load_block_if_required() {\n            self.pos += 1;\n            true\n        } else {\n            false\n        }\n    }\n\n    /// Get the current delete operation.\n    /// Calling `.get` does not advance the cursor.\n    pub fn get(&mut self) -> Option<&DeleteOperation> {\n        if self.load_block_if_required() {\n            Some(&self.block.operations[self.pos])\n        } else {\n            None\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::{DeleteOperation, DeleteQueue};\n    use crate::index::SegmentReader;\n    use crate::query::{Explanation, Scorer, Weight};\n    use crate::{DocId, Score};\n\n    struct DummyWeight;\n    impl Weight for DummyWeight {\n        fn scorer(&self, _reader: &SegmentReader, _boost: Score) -> crate::Result<Box<dyn Scorer>> {\n            Err(crate::TantivyError::InternalError(\"dummy impl\".to_owned()))\n        }\n\n        fn explain(&self, _reader: &SegmentReader, _doc: DocId) -> crate::Result<Explanation> {\n            Err(crate::TantivyError::InternalError(\"dummy impl\".to_owned()))\n        }\n    }\n\n    #[test]\n    fn test_deletequeue() {\n        let delete_queue = DeleteQueue::default();\n\n        let make_op = |i: usize| DeleteOperation {\n            opstamp: i as u64,\n            target: Box::new(DummyWeight),\n        };\n\n        delete_queue.push(make_op(1));\n        delete_queue.push(make_op(2));\n\n        let snapshot = delete_queue.cursor();\n        {\n            let mut operations_it = snapshot.clone();\n            assert_eq!(operations_it.get().unwrap().opstamp, 1);\n            operations_it.advance();\n            assert_eq!(operations_it.get().unwrap().opstamp, 2);\n            operations_it.advance();\n            assert!(operations_it.get().is_none());\n            operations_it.advance();\n\n            let mut snapshot2 = delete_queue.cursor();\n            assert!(snapshot2.get().is_none());\n            delete_queue.push(make_op(3));\n            assert_eq!(snapshot2.get().unwrap().opstamp, 3);\n            assert_eq!(operations_it.get().unwrap().opstamp, 3);\n            assert_eq!(operations_it.get().unwrap().opstamp, 3);\n            operations_it.advance();\n            assert!(operations_it.get().is_none());\n            operations_it.advance();\n        }\n        {\n            let mut operations_it = snapshot;\n            assert_eq!(operations_it.get().unwrap().opstamp, 1);\n            operations_it.advance();\n            assert_eq!(operations_it.get().unwrap().opstamp, 2);\n            operations_it.advance();\n            assert_eq!(operations_it.get().unwrap().opstamp, 3);\n            operations_it.advance();\n            assert!(operations_it.get().is_none());\n        }\n    }\n}\n"
  },
  {
    "path": "src/indexer/doc_id_mapping.rs",
    "content": "//! This module is used when sorting the index by a property, e.g.\n//! to get mappings from old doc_id to new doc_id and vice versa, after sorting\n\nuse common::ReadOnlyBitSet;\n\nuse crate::DocAddress;\n\n#[derive(Copy, Clone, Eq, PartialEq)]\npub enum MappingType {\n    Stacked,\n    StackedWithDeletes,\n}\n\n/// Struct to provide mapping from new doc_id to old doc_id and segment.\n#[derive(Clone)]\npub(crate) struct SegmentDocIdMapping {\n    pub(crate) new_doc_id_to_old_doc_addr: Vec<DocAddress>,\n    pub(crate) alive_bitsets: Vec<Option<ReadOnlyBitSet>>,\n    mapping_type: MappingType,\n}\n\nimpl SegmentDocIdMapping {\n    pub(crate) fn new(\n        new_doc_id_to_old_doc_addr: Vec<DocAddress>,\n        mapping_type: MappingType,\n        alive_bitsets: Vec<Option<ReadOnlyBitSet>>,\n    ) -> Self {\n        Self {\n            new_doc_id_to_old_doc_addr,\n            mapping_type,\n            alive_bitsets,\n        }\n    }\n\n    pub fn mapping_type(&self) -> MappingType {\n        self.mapping_type\n    }\n\n    /// Returns an iterator over the old document addresses, ordered by the new document ids.\n    ///\n    /// In the returned `DocAddress`, the `segment_ord` is the ordinal of targeted segment\n    /// in the list of merged segments.\n    pub(crate) fn iter_old_doc_addrs(&self) -> impl Iterator<Item = DocAddress> + '_ {\n        self.new_doc_id_to_old_doc_addr.iter().copied()\n    }\n}\n"
  },
  {
    "path": "src/indexer/doc_opstamp_mapping.rs",
    "content": "use crate::{DocId, Opstamp};\n\n// Doc to opstamp is used to identify which\n// document should be deleted.\n//\n// Since the docset matching the query of a delete operation\n// is not computed right when the delete operation is received,\n// we need to find a way to evaluate, for each document,\n// whether the document was added before or after\n// the delete operation. This anteriority is used by comparing\n// the docstamp of the document.\n//\n// The doc to opstamp mapping stores precisely an array\n// indexed by doc id and storing the opstamp of the document.\n//\n// This mapping is NOT necessarily increasing, because\n// we might be sorting documents according to a fast field.\n#[derive(Clone)]\npub enum DocToOpstampMapping<'a> {\n    WithMap(&'a [Opstamp]),\n    None,\n}\n\nimpl DocToOpstampMapping<'_> {\n    /// Assess whether a document should be considered deleted given that it contains\n    /// a deleted term that was deleted at the opstamp: `delete_opstamp`.\n    ///\n    /// This function returns true if the `DocToOpstamp` mapping is none or if\n    /// the `doc_opstamp` is anterior to the delete opstamp.\n    pub fn is_deleted(&self, doc_id: DocId, delete_opstamp: Opstamp) -> bool {\n        match self {\n            Self::WithMap(doc_opstamps) => {\n                let doc_opstamp = doc_opstamps[doc_id as usize];\n                doc_opstamp < delete_opstamp\n            }\n            Self::None => true,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::DocToOpstampMapping;\n\n    #[test]\n    fn test_doc_to_opstamp_mapping_none() {\n        let doc_to_opstamp_mapping = DocToOpstampMapping::None;\n        assert!(doc_to_opstamp_mapping.is_deleted(1u32, 0u64));\n        assert!(doc_to_opstamp_mapping.is_deleted(1u32, 2u64));\n    }\n\n    #[test]\n    fn test_doc_to_opstamp_mapping_with_map() {\n        let doc_to_opstamp_mapping = DocToOpstampMapping::WithMap(&[5u64, 1u64, 0u64, 4u64, 3u64]);\n        assert_eq!(doc_to_opstamp_mapping.is_deleted(0u32, 2u64), false);\n        assert_eq!(doc_to_opstamp_mapping.is_deleted(1u32, 2u64), true);\n        assert_eq!(doc_to_opstamp_mapping.is_deleted(2u32, 2u64), true);\n        assert_eq!(doc_to_opstamp_mapping.is_deleted(3u32, 2u64), false);\n        assert_eq!(doc_to_opstamp_mapping.is_deleted(4u32, 2u64), false);\n    }\n}\n"
  },
  {
    "path": "src/indexer/flat_map_with_buffer.rs",
    "content": "pub struct FlatMapWithBuffer<T, F, Iter> {\n    buffer: Vec<T>,\n    fill_buffer: F,\n    underlying_it: Iter,\n}\n\nimpl<T, F, Iter, I> Iterator for FlatMapWithBuffer<T, F, Iter>\nwhere\n    Iter: Iterator<Item = I>,\n    F: Fn(I, &mut Vec<T>),\n{\n    type Item = T;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        while self.buffer.is_empty() {\n            let next_el = self.underlying_it.next()?;\n            (self.fill_buffer)(next_el, &mut self.buffer);\n            // We will pop elements, so we reverse the buffer first.\n            self.buffer.reverse();\n        }\n        self.buffer.pop()\n    }\n}\n\n#[allow(dead_code)]\npub trait FlatMapWithBufferIter: Iterator {\n    /// Function similar to `flat_map`, but allows reusing a shared `Vec`.\n    fn flat_map_with_buffer<F, T>(self, fill_buffer: F) -> FlatMapWithBuffer<T, F, Self>\n    where\n        F: Fn(Self::Item, &mut Vec<T>),\n        Self: Sized,\n    {\n        FlatMapWithBuffer {\n            buffer: Vec::with_capacity(10),\n            fill_buffer,\n            underlying_it: self,\n        }\n    }\n}\n\nimpl<T: ?Sized> FlatMapWithBufferIter for T where T: Iterator {}\n\n#[cfg(test)]\nmod tests {\n    use crate::indexer::flat_map_with_buffer::FlatMapWithBufferIter;\n\n    #[test]\n    fn test_flat_map_with_buffer_empty() {\n        let mut empty_iter = std::iter::empty::<usize>()\n            .flat_map_with_buffer(|_val: usize, _buffer: &mut Vec<usize>| {});\n        assert!(empty_iter.next().is_none());\n    }\n\n    #[test]\n    fn test_flat_map_with_buffer_simple() {\n        let vals: Vec<usize> = (1..5)\n            .flat_map_with_buffer(|val: usize, buffer: &mut Vec<usize>| buffer.extend(0..val))\n            .collect();\n        assert_eq!(&[0, 0, 1, 0, 1, 2, 0, 1, 2, 3], &vals[..]);\n    }\n\n    #[test]\n    fn test_flat_map_filling_no_elements_does_not_stop_iterator() {\n        let vals: Vec<usize> = [2, 0, 0, 3]\n            .into_iter()\n            .flat_map_with_buffer(|val: usize, buffer: &mut Vec<usize>| buffer.extend(0..val))\n            .collect();\n        assert_eq!(&[0, 1, 0, 1, 2], &vals[..]);\n    }\n}\n"
  },
  {
    "path": "src/indexer/index_writer.rs",
    "content": "use std::ops::Range;\nuse std::sync::Arc;\nuse std::thread;\nuse std::thread::JoinHandle;\n\nuse common::BitSet;\nuse smallvec::smallvec;\n\nuse super::operation::{AddOperation, UserOperation};\nuse super::segment_updater::SegmentUpdater;\nuse super::{AddBatch, AddBatchReceiver, AddBatchSender, PreparedCommit};\nuse crate::directory::{DirectoryLock, GarbageCollectionResult, TerminatingWrite};\nuse crate::error::TantivyError;\nuse crate::fastfield::write_alive_bitset;\nuse crate::index::{Index, Segment, SegmentComponent, SegmentId, SegmentMeta, SegmentReader};\nuse crate::indexer::delete_queue::{DeleteCursor, DeleteQueue};\nuse crate::indexer::doc_opstamp_mapping::DocToOpstampMapping;\nuse crate::indexer::index_writer_status::IndexWriterStatus;\nuse crate::indexer::operation::DeleteOperation;\nuse crate::indexer::stamper::Stamper;\nuse crate::indexer::{MergePolicy, SegmentEntry, SegmentWriter};\nuse crate::query::{EnableScoring, Query, TermQuery};\nuse crate::schema::document::Document;\nuse crate::schema::{IndexRecordOption, TantivyDocument, Term};\nuse crate::{FutureResult, Opstamp};\n\n// Size of the margin for the `memory_arena`. A segment is closed when the remaining memory\n// in the `memory_arena` goes below MARGIN_IN_BYTES.\npub const MARGIN_IN_BYTES: usize = 1_000_000;\n\n// We impose the memory per thread to be at least 15 MB, as the baseline consumption is 12MB.\npub const MEMORY_BUDGET_NUM_BYTES_MIN: usize = ((MARGIN_IN_BYTES as u32) * 15u32) as usize;\npub const MEMORY_BUDGET_NUM_BYTES_MAX: usize = u32::MAX as usize - MARGIN_IN_BYTES;\n\n// We impose the number of index writer threads to be at most this.\npub const MAX_NUM_THREAD: usize = 8;\n\n// Add document will block if the number of docs waiting in the queue to be indexed\n// reaches `PIPELINE_MAX_SIZE_IN_DOCS`\nconst PIPELINE_MAX_SIZE_IN_DOCS: usize = 10_000;\n\nfn error_in_index_worker_thread(context: &str) -> TantivyError {\n    TantivyError::ErrorInThread(format!(\n        \"{context}. A worker thread encountered an error (io::Error most likely) or panicked.\"\n    ))\n}\n\n#[derive(Clone, bon::Builder)]\n/// A builder for creating a new [IndexWriter] for an index.\npub struct IndexWriterOptions {\n    #[builder(default = MEMORY_BUDGET_NUM_BYTES_MIN)]\n    /// The memory budget per indexer thread.\n    ///\n    /// When an indexer thread has buffered this much data in memory\n    /// it will flush the segment to disk (although this is not searchable until commit is called.)\n    memory_budget_per_thread: usize,\n    #[builder(default = 1)]\n    /// The number of indexer worker threads to use.\n    num_worker_threads: usize,\n    #[builder(default = 4)]\n    /// Defines the number of merger threads to use.\n    num_merge_threads: usize,\n}\n\n/// `IndexWriter` is the user entry-point to add document to an index.\n///\n/// It manages a small number of indexing thread, as well as a shared\n/// indexing queue.\n/// Each indexing thread builds its own independent [`Segment`], via\n/// a `SegmentWriter` object.\npub struct IndexWriter<D: Document = TantivyDocument> {\n    // the lock is just used to bind the\n    // lifetime of the lock with that of the IndexWriter.\n    _directory_lock: Option<DirectoryLock>,\n\n    index: Index,\n\n    options: IndexWriterOptions,\n\n    workers_join_handle: Vec<JoinHandle<crate::Result<()>>>,\n\n    index_writer_status: IndexWriterStatus<D>,\n    operation_sender: AddBatchSender<D>,\n\n    segment_updater: SegmentUpdater,\n\n    worker_id: usize,\n\n    delete_queue: DeleteQueue,\n\n    stamper: Stamper,\n    committed_opstamp: Opstamp,\n}\n\nfn compute_deleted_bitset(\n    alive_bitset: &mut BitSet,\n    segment_reader: &SegmentReader,\n    delete_cursor: &mut DeleteCursor,\n    doc_opstamps: &DocToOpstampMapping,\n    target_opstamp: Opstamp,\n) -> crate::Result<bool> {\n    let mut might_have_changed = false;\n    while let Some(delete_op) = delete_cursor.get() {\n        if delete_op.opstamp > target_opstamp {\n            break;\n        }\n\n        // A delete operation should only affect\n        // document that were inserted before it.\n        delete_op\n            .target\n            .for_each_no_score(segment_reader, &mut |docs_matching_delete_query| {\n                for doc_matching_delete_query in docs_matching_delete_query.iter().cloned() {\n                    if doc_opstamps.is_deleted(doc_matching_delete_query, delete_op.opstamp) {\n                        alive_bitset.remove(doc_matching_delete_query);\n                        might_have_changed = true;\n                    }\n                }\n            })?;\n        delete_cursor.advance();\n    }\n    Ok(might_have_changed)\n}\n\n/// Advance delete for the given segment up to the target opstamp.\n///\n/// Note that there are no guarantee that the resulting `segment_entry` delete_opstamp\n/// is `==` target_opstamp.\n/// For instance, there was no delete operation between the state of the `segment_entry` and\n/// the `target_opstamp`, `segment_entry` is not updated.\npub fn advance_deletes(\n    mut segment: Segment,\n    segment_entry: &mut SegmentEntry,\n    target_opstamp: Opstamp,\n) -> crate::Result<()> {\n    if segment_entry.meta().delete_opstamp() == Some(target_opstamp) {\n        // We are already up-to-date here.\n        return Ok(());\n    }\n\n    if segment_entry.alive_bitset().is_none() && segment_entry.delete_cursor().get().is_none() {\n        // There has been no `DeleteOperation` between the segment status and `target_opstamp`.\n        return Ok(());\n    }\n\n    let segment_reader = SegmentReader::open(&segment)?;\n\n    let max_doc = segment_reader.max_doc();\n    let mut alive_bitset: BitSet = match segment_entry.alive_bitset() {\n        Some(previous_alive_bitset) => (*previous_alive_bitset).clone(),\n        None => BitSet::with_max_value_and_full(max_doc),\n    };\n\n    let num_deleted_docs_before = segment.meta().num_deleted_docs();\n\n    compute_deleted_bitset(\n        &mut alive_bitset,\n        &segment_reader,\n        segment_entry.delete_cursor(),\n        &DocToOpstampMapping::None,\n        target_opstamp,\n    )?;\n\n    if let Some(seg_alive_bitset) = segment_reader.alive_bitset() {\n        alive_bitset.intersect_update(seg_alive_bitset.bitset());\n    }\n\n    let num_alive_docs: u32 = alive_bitset.len() as u32;\n    let num_deleted_docs = max_doc - num_alive_docs;\n    if num_deleted_docs > num_deleted_docs_before {\n        // There are new deletes. We need to write a new delete file.\n        segment = segment.with_delete_meta(num_deleted_docs, target_opstamp);\n        let mut alive_doc_file = segment.open_write(SegmentComponent::Delete)?;\n        write_alive_bitset(&alive_bitset, &mut alive_doc_file)?;\n        alive_doc_file.terminate()?;\n    }\n\n    segment_entry.set_meta(segment.meta().clone());\n    Ok(())\n}\n\nfn index_documents<D: Document>(\n    memory_budget: usize,\n    segment: Segment,\n    grouped_document_iterator: &mut dyn Iterator<Item = AddBatch<D>>,\n    segment_updater: &SegmentUpdater,\n    mut delete_cursor: DeleteCursor,\n) -> crate::Result<()> {\n    let mut segment_writer = SegmentWriter::for_segment(memory_budget, segment.clone())?;\n    for document_group in grouped_document_iterator {\n        for doc in document_group {\n            segment_writer.add_document(doc)?;\n        }\n        let mem_usage = segment_writer.mem_usage();\n        if mem_usage >= memory_budget - MARGIN_IN_BYTES {\n            info!(\n                \"Buffer limit reached, flushing segment with maxdoc={}.\",\n                segment_writer.max_doc()\n            );\n            break;\n        }\n    }\n\n    if !segment_updater.is_alive() {\n        return Ok(());\n    }\n\n    let max_doc = segment_writer.max_doc();\n\n    // this is ensured by the call to peek before starting\n    // the worker thread.\n    assert!(max_doc > 0);\n\n    let doc_opstamps: Vec<Opstamp> = segment_writer.finalize()?;\n\n    let segment_with_max_doc = segment.with_max_doc(max_doc);\n\n    let alive_bitset_opt = apply_deletes(&segment_with_max_doc, &mut delete_cursor, &doc_opstamps)?;\n\n    let meta = segment_with_max_doc.meta().clone();\n\n    // update segment_updater inventory to remove tempstore\n    let segment_entry = SegmentEntry::new(meta, delete_cursor, alive_bitset_opt);\n    segment_updater.schedule_add_segment(segment_entry).wait()?;\n    Ok(())\n}\n\n/// `doc_opstamps` is required to be non-empty.\nfn apply_deletes(\n    segment: &Segment,\n    delete_cursor: &mut DeleteCursor,\n    doc_opstamps: &[Opstamp],\n) -> crate::Result<Option<BitSet>> {\n    if delete_cursor.get().is_none() {\n        // if there are no delete operation in the queue, no need\n        // to even open the segment.\n        return Ok(None);\n    }\n\n    let max_doc_opstamp: Opstamp = doc_opstamps\n        .iter()\n        .cloned()\n        .max()\n        .expect(\"Empty DocOpstamp is forbidden\");\n\n    let segment_reader = SegmentReader::open(segment)?;\n    let doc_to_opstamps = DocToOpstampMapping::WithMap(doc_opstamps);\n\n    let max_doc = segment.meta().max_doc();\n    let mut deleted_bitset = BitSet::with_max_value_and_full(max_doc);\n    let may_have_deletes = compute_deleted_bitset(\n        &mut deleted_bitset,\n        &segment_reader,\n        delete_cursor,\n        &doc_to_opstamps,\n        max_doc_opstamp,\n    )?;\n    Ok(if may_have_deletes {\n        Some(deleted_bitset)\n    } else {\n        None\n    })\n}\n\nimpl<D: Document> IndexWriter<D> {\n    /// Create a new index writer. Attempts to acquire a lockfile.\n    ///\n    /// The lockfile should be deleted on drop, but it is possible\n    /// that due to a panic or other error, a stale lockfile will be\n    /// left in the index directory. If you are sure that no other\n    /// `IndexWriter` on the system is accessing the index directory,\n    /// it is safe to manually delete the lockfile.\n    ///\n    /// `num_threads` specifies the number of indexing workers that\n    /// should work at the same time.\n    /// # Errors\n    /// If the lockfile already exists, returns `Error::FileAlreadyExists`.\n    /// If the memory arena per thread is too small or too big, returns\n    /// `TantivyError::InvalidArgument`\n    pub(crate) fn new(\n        index: &Index,\n        options: IndexWriterOptions,\n        directory_lock: DirectoryLock,\n    ) -> crate::Result<Self> {\n        if options.memory_budget_per_thread < MEMORY_BUDGET_NUM_BYTES_MIN {\n            let err_msg = format!(\n                \"The memory arena in bytes per thread needs to be at least \\\n                 {MEMORY_BUDGET_NUM_BYTES_MIN}.\"\n            );\n            return Err(TantivyError::InvalidArgument(err_msg));\n        }\n        if options.memory_budget_per_thread >= MEMORY_BUDGET_NUM_BYTES_MAX {\n            let err_msg = format!(\n                \"The memory arena in bytes per thread cannot exceed {MEMORY_BUDGET_NUM_BYTES_MAX}\"\n            );\n            return Err(TantivyError::InvalidArgument(err_msg));\n        }\n        if options.num_worker_threads == 0 {\n            let err_msg = \"At least one worker thread is required, got 0\".to_string();\n            return Err(TantivyError::InvalidArgument(err_msg));\n        }\n\n        let (document_sender, document_receiver) =\n            crossbeam_channel::bounded(PIPELINE_MAX_SIZE_IN_DOCS);\n\n        let delete_queue = DeleteQueue::default();\n\n        let current_opstamp = index.load_metas()?.opstamp;\n\n        let stamper = Stamper::new(current_opstamp);\n\n        let segment_updater = SegmentUpdater::create(\n            index.clone(),\n            stamper.clone(),\n            &delete_queue.cursor(),\n            options.num_merge_threads,\n        )?;\n\n        let mut index_writer = Self {\n            _directory_lock: Some(directory_lock),\n\n            options: options.clone(),\n            index: index.clone(),\n            index_writer_status: IndexWriterStatus::from(document_receiver),\n            operation_sender: document_sender,\n\n            segment_updater,\n\n            workers_join_handle: vec![],\n\n            delete_queue,\n\n            committed_opstamp: current_opstamp,\n            stamper,\n\n            worker_id: 0,\n        };\n        index_writer.start_workers()?;\n        Ok(index_writer)\n    }\n\n    fn drop_sender(&mut self) {\n        let (sender, _receiver) = crossbeam_channel::bounded(1);\n        self.operation_sender = sender;\n    }\n\n    /// Accessor to the index.\n    pub fn index(&self) -> &Index {\n        &self.index\n    }\n\n    /// If there are some merging threads, blocks until they all finish their work and\n    /// then drop the `IndexWriter`.\n    pub fn wait_merging_threads(mut self) -> crate::Result<()> {\n        // this will stop the indexing thread,\n        // dropping the last reference to the segment_updater.\n        self.drop_sender();\n\n        let former_workers_handles = std::mem::take(&mut self.workers_join_handle);\n        for join_handle in former_workers_handles {\n            join_handle\n                .join()\n                .map_err(|_| error_in_index_worker_thread(\"Worker thread panicked.\"))?\n                .map_err(|_| error_in_index_worker_thread(\"Worker thread failed.\"))?;\n        }\n\n        let result = self\n            .segment_updater\n            .wait_merging_thread()\n            .map_err(|_| error_in_index_worker_thread(\"Failed to join merging thread.\"));\n\n        if let Err(ref e) = result {\n            error!(\"Some merging thread failed {e:?}\");\n        }\n\n        result\n    }\n\n    #[doc(hidden)]\n    pub fn add_segment(&self, segment_meta: SegmentMeta) -> crate::Result<()> {\n        let delete_cursor = self.delete_queue.cursor();\n        let segment_entry = SegmentEntry::new(segment_meta, delete_cursor, None);\n        self.segment_updater\n            .schedule_add_segment(segment_entry)\n            .wait()\n    }\n\n    /// Creates a new segment.\n    ///\n    /// This method is useful only for users trying to do complex\n    /// operations, like converting an index format to another.\n    ///\n    /// It is safe to start writing file associated with the new `Segment`.\n    /// These will not be garbage collected as long as an instance object of\n    /// `SegmentMeta` object associated with the new `Segment` is \"alive\".\n    pub fn new_segment(&self) -> Segment {\n        self.index.new_segment()\n    }\n\n    fn operation_receiver(&self) -> crate::Result<AddBatchReceiver<D>> {\n        self.index_writer_status\n            .operation_receiver()\n            .ok_or_else(|| {\n                crate::TantivyError::ErrorInThread(\n                    \"The index writer was killed. It can happen if an indexing worker encountered \\\n                     an Io error for instance.\"\n                        .to_string(),\n                )\n            })\n    }\n\n    /// Spawns a new worker thread for indexing.\n    /// The thread consumes documents from the pipeline.\n    fn add_indexing_worker(&mut self) -> crate::Result<()> {\n        let document_receiver_clone = self.operation_receiver()?;\n        let index_writer_bomb = self.index_writer_status.create_bomb();\n\n        let segment_updater = self.segment_updater.clone();\n\n        let mut delete_cursor = self.delete_queue.cursor();\n\n        let mem_budget = self.options.memory_budget_per_thread;\n        let index = self.index.clone();\n        let join_handle: JoinHandle<crate::Result<()>> = thread::Builder::new()\n            .name(format!(\"thrd-tantivy-index{}\", self.worker_id))\n            .spawn(move || {\n                loop {\n                    let mut document_iterator = document_receiver_clone\n                        .clone()\n                        .into_iter()\n                        .filter(|batch| !batch.is_empty())\n                        .peekable();\n\n                    // The peeking here is to avoid creating a new segment's files\n                    // if no document are available.\n                    //\n                    // This is a valid guarantee as the peeked document now belongs to\n                    // our local iterator.\n                    if let Some(batch) = document_iterator.peek() {\n                        assert!(!batch.is_empty());\n                        delete_cursor.skip_to(batch[0].opstamp);\n                    } else {\n                        // No more documents.\n                        // It happens when there is a commit, or if the `IndexWriter`\n                        // was dropped.\n                        index_writer_bomb.defuse();\n                        return Ok(());\n                    }\n\n                    index_documents(\n                        mem_budget,\n                        index.new_segment(),\n                        &mut document_iterator,\n                        &segment_updater,\n                        delete_cursor.clone(),\n                    )?;\n                }\n            })?;\n        self.worker_id += 1;\n        self.workers_join_handle.push(join_handle);\n        Ok(())\n    }\n\n    /// Accessor to the merge policy.\n    pub fn get_merge_policy(&self) -> Arc<dyn MergePolicy> {\n        self.segment_updater.get_merge_policy()\n    }\n\n    /// Setter for the merge policy.\n    pub fn set_merge_policy(&self, merge_policy: Box<dyn MergePolicy>) {\n        self.segment_updater.set_merge_policy(merge_policy);\n    }\n\n    fn start_workers(&mut self) -> crate::Result<()> {\n        for _ in 0..self.options.num_worker_threads {\n            self.add_indexing_worker()?;\n        }\n        Ok(())\n    }\n\n    /// Detects and removes the files that are not used by the index anymore.\n    pub fn garbage_collect_files(&self) -> FutureResult<GarbageCollectionResult> {\n        self.segment_updater.schedule_garbage_collect()\n    }\n\n    /// Deletes all documents from the index\n    ///\n    /// Requires `commit`ing\n    /// Enables users to rebuild the index,\n    /// by clearing and resubmitting necessary documents\n    ///\n    /// ```rust\n    /// use tantivy::collector::TopDocs;\n    /// use tantivy::query::QueryParser;\n    /// use tantivy::schema::*;\n    /// use tantivy::{doc, Index};\n    ///\n    /// fn main() -> tantivy::Result<()> {\n    ///     let mut schema_builder = Schema::builder();\n    ///     let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n    ///     let schema = schema_builder.build();\n    ///\n    ///     let index = Index::create_in_ram(schema.clone());\n    ///\n    ///     let mut index_writer = index.writer_with_num_threads(1, 50_000_000)?;\n    ///     index_writer.add_document(doc!(title => \"The modern Prometheus\"))?;\n    ///     index_writer.commit()?;\n    ///\n    ///     let clear_res = index_writer.delete_all_documents().unwrap();\n    ///     // have to commit, otherwise deleted terms remain available\n    ///     index_writer.commit()?;\n    ///\n    ///     let searcher = index.reader()?.searcher();\n    ///     let query_parser = QueryParser::for_index(&index, vec![title]);\n    ///     let query_promo = query_parser.parse_query(\"Prometheus\")?;\n    ///     let top_docs_promo = searcher.search(&query_promo, &TopDocs::with_limit(1).order_by_score())?;\n    ///\n    ///     assert!(top_docs_promo.is_empty());\n    ///     Ok(())\n    /// }\n    /// ```\n    pub fn delete_all_documents(&self) -> crate::Result<Opstamp> {\n        // Delete segments\n        self.segment_updater.remove_all_segments();\n        // Return new stamp - reverted stamp\n        self.stamper.revert(self.committed_opstamp);\n        Ok(self.committed_opstamp)\n    }\n\n    /// Merges a given list of segments.\n    ///\n    /// If all segments are empty no new segment will be created.\n    ///\n    /// `segment_ids` is required to be non-empty.\n    pub fn merge(&mut self, segment_ids: &[SegmentId]) -> FutureResult<Option<SegmentMeta>> {\n        let merge_operation = self.segment_updater.make_merge_operation(segment_ids);\n        let segment_updater = self.segment_updater.clone();\n        segment_updater.start_merge(merge_operation)\n    }\n\n    /// Closes the current document channel send.\n    /// and replace all the channels by new ones.\n    ///\n    /// The current workers will keep on indexing\n    /// the pending document and stop\n    /// when no documents are remaining.\n    ///\n    /// Returns the former segment_ready channel.\n    fn recreate_document_channel(&mut self) {\n        let (document_sender, document_receiver) =\n            crossbeam_channel::bounded(PIPELINE_MAX_SIZE_IN_DOCS);\n        self.operation_sender = document_sender;\n        self.index_writer_status = IndexWriterStatus::from(document_receiver);\n    }\n\n    /// Rollback to the last commit\n    ///\n    /// This cancels all of the updates that\n    /// happened after the last commit.\n    /// After calling rollback, the index is in the same\n    /// state as it was after the last commit.\n    ///\n    /// The opstamp at the last commit is returned.\n    pub fn rollback(&mut self) -> crate::Result<Opstamp> {\n        info!(\"Rolling back to opstamp {}\", self.committed_opstamp);\n        // marks the segment updater as killed. From now on, all\n        // segment updates will be ignored.\n        self.segment_updater.kill();\n        let document_receiver_res = self.operation_receiver();\n\n        // take the directory lock to create a new index_writer.\n        let directory_lock = self\n            ._directory_lock\n            .take()\n            .expect(\"The IndexWriter does not have any lock. This is a bug, please report.\");\n\n        let new_index_writer = IndexWriter::new(&self.index, self.options.clone(), directory_lock)?;\n\n        // the current `self` is dropped right away because of this call.\n        //\n        // This will drop the document queue, and the thread\n        // should terminate.\n        *self = new_index_writer;\n\n        // Drains the document receiver pipeline :\n        // Workers don't need to index the pending documents.\n        //\n        // This will reach an end as the only document_sender\n        // was dropped with the index_writer.\n        if let Ok(document_receiver) = document_receiver_res {\n            for _ in document_receiver {}\n        }\n\n        Ok(self.committed_opstamp)\n    }\n\n    /// Prepares a commit.\n    ///\n    /// Calling `prepare_commit()` will cut the indexing\n    /// queue. All pending documents will be sent to the\n    /// indexing workers. They will then terminate, regardless\n    /// of the size of their current segment and flush their\n    /// work on disk.\n    ///\n    /// Once a commit is \"prepared\", you can either\n    /// call\n    /// * `.commit()`: to accept this commit\n    /// * `.abort()`: to cancel this commit.\n    ///\n    /// In the current implementation, [`PreparedCommit`] borrows\n    /// the [`IndexWriter`] mutably so we are guaranteed that no new\n    /// document can be added as long as it is committed or is\n    /// dropped.\n    ///\n    /// It is also possible to add a payload to the `commit`\n    /// using this API.\n    /// See [`PreparedCommit::set_payload()`].\n    pub fn prepare_commit(&mut self) -> crate::Result<PreparedCommit<'_, D>> {\n        // Here, because we join all of the worker threads,\n        // all of the segment update for this commit have been\n        // sent.\n        //\n        // No document belonging to the next commit have been\n        // pushed too, because add_document can only happen\n        // on this thread.\n        //\n        // This will move uncommitted segments to the state of\n        // committed segments.\n        info!(\"Preparing commit\");\n\n        // this will drop the current document channel\n        // and recreate a new one.\n        self.recreate_document_channel();\n\n        let former_workers_join_handle = std::mem::take(&mut self.workers_join_handle);\n\n        for worker_handle in former_workers_join_handle {\n            let indexing_worker_result = worker_handle\n                .join()\n                .map_err(|e| TantivyError::ErrorInThread(format!(\"{e:?}\")))?;\n            indexing_worker_result?;\n            self.add_indexing_worker()?;\n        }\n\n        let commit_opstamp = self.stamper.stamp();\n        let prepared_commit = PreparedCommit::new(self, commit_opstamp);\n        info!(\"Prepared commit {commit_opstamp}\");\n        Ok(prepared_commit)\n    }\n\n    /// Commits all of the pending changes\n    ///\n    /// A call to commit blocks.\n    /// After it returns, all of the document that\n    /// were added since the last commit are published\n    /// and persisted.\n    ///\n    /// In case of a crash or an hardware failure (as\n    /// long as the hard disk is spared), it will be possible\n    /// to resume indexing from this point.\n    ///\n    /// Commit returns the `opstamp` of the last document\n    /// that made it in the commit.\n    pub fn commit(&mut self) -> crate::Result<Opstamp> {\n        self.prepare_commit()?.commit()\n    }\n\n    pub(crate) fn segment_updater(&self) -> &SegmentUpdater {\n        &self.segment_updater\n    }\n\n    /// Delete all documents containing a given term.\n    ///\n    /// Delete operation only affects documents that\n    /// were added in previous commits, and documents\n    /// that were added previously in the same commit.\n    ///\n    /// Like adds, the deletion itself will be visible\n    /// only after calling `commit()`.\n    pub fn delete_term(&self, term: Term) -> Opstamp {\n        let query = TermQuery::new(term, IndexRecordOption::Basic);\n        // For backward compatibility, if Term is invalid for the index, do nothing but return an\n        // Opstamp\n        self.delete_query(Box::new(query))\n            .unwrap_or_else(|_| self.stamper.stamp())\n    }\n\n    /// Delete all documents matching a given query.\n    /// Returns an `Err` if the query can't be executed.\n    ///\n    /// Delete operation only affects documents that\n    /// were added in previous commits, and documents\n    /// that were added previously in the same commit.\n    ///\n    /// Like adds, the deletion itself will be visible\n    /// only after calling `commit()`.\n    #[doc(hidden)]\n    pub fn delete_query(&self, query: Box<dyn Query>) -> crate::Result<Opstamp> {\n        let weight = query.weight(EnableScoring::disabled_from_schema(&self.index.schema()))?;\n        let opstamp = self.stamper.stamp();\n        let delete_operation = DeleteOperation {\n            opstamp,\n            target: weight,\n        };\n        self.delete_queue.push(delete_operation);\n        Ok(opstamp)\n    }\n\n    /// Returns the opstamp of the last successful commit.\n    ///\n    /// This is, for instance, the opstamp the index will\n    /// rollback to if there is a failure like a power surge.\n    ///\n    /// This is also the opstamp of the commit that is currently\n    /// available for searchers.\n    pub fn commit_opstamp(&self) -> Opstamp {\n        self.committed_opstamp\n    }\n\n    /// Adds a document.\n    ///\n    /// If the indexing pipeline is full, this call may block.\n    ///\n    /// The opstamp is an increasing `u64` that can\n    /// be used by the client to align commits with its own\n    /// document queue.\n    pub fn add_document(&self, document: D) -> crate::Result<Opstamp> {\n        let opstamp = self.stamper.stamp();\n        self.send_add_documents_batch(smallvec![AddOperation { opstamp, document }])?;\n        Ok(opstamp)\n    }\n\n    /// Gets a range of stamps from the stamper and \"pops\" the last stamp\n    /// from the range returning a tuple of the last optstamp and the popped\n    /// range.\n    ///\n    /// The total number of stamps generated by this method is `count + 1`;\n    /// each operation gets a stamp from the `stamps` iterator and `last_opstamp`\n    /// is for the batch itself.\n    fn get_batch_opstamps(&self, count: Opstamp) -> (Opstamp, Range<Opstamp>) {\n        let Range { start, end } = self.stamper.stamps(count + 1u64);\n        let last_opstamp = end - 1;\n        (last_opstamp, start..last_opstamp)\n    }\n\n    /// Runs a group of document operations ensuring that the operations are\n    /// assigned contiguous u64 opstamps and that add operations of the same\n    /// group are flushed into the same segment.\n    ///\n    /// If the indexing pipeline is full, this call may block.\n    ///\n    /// Each operation of the given `user_operations` will receive an in-order,\n    /// contiguous u64 opstamp. The entire batch itself is also given an\n    /// opstamp that is 1 greater than the last given operation. This\n    /// `batch_opstamp` is the return value of `run`. An empty group of\n    /// `user_operations`, an empty `Vec<UserOperation>`, still receives\n    /// a valid opstamp even though no changes were _actually_ made to the index.\n    ///\n    /// Like adds and deletes (see `IndexWriter.add_document` and\n    /// `IndexWriter.delete_term`), the changes made by calling `run` will be\n    /// visible to readers only after calling `commit()`.\n    pub fn run<I>(&self, user_operations: I) -> crate::Result<Opstamp>\n    where\n        I: IntoIterator<Item = UserOperation<D>>,\n        I::IntoIter: ExactSizeIterator,\n    {\n        let user_operations_it = user_operations.into_iter();\n        let count = user_operations_it.len() as u64;\n        if count == 0 {\n            return Ok(self.stamper.stamp());\n        }\n        let (batch_opstamp, stamps) = self.get_batch_opstamps(count);\n\n        let mut adds = AddBatch::default();\n\n        for (user_op, opstamp) in user_operations_it.zip(stamps) {\n            match user_op {\n                UserOperation::Delete(term) => {\n                    let query = TermQuery::new(term, IndexRecordOption::Basic);\n                    let weight =\n                        query.weight(EnableScoring::disabled_from_schema(&self.index.schema()))?;\n                    let delete_operation = DeleteOperation {\n                        opstamp,\n                        target: weight,\n                    };\n                    self.delete_queue.push(delete_operation);\n                }\n                UserOperation::Add(document) => {\n                    let add_operation = AddOperation { opstamp, document };\n                    adds.push(add_operation);\n                }\n            }\n        }\n        self.send_add_documents_batch(adds)?;\n        Ok(batch_opstamp)\n    }\n\n    fn send_add_documents_batch(&self, add_ops: AddBatch<D>) -> crate::Result<()> {\n        if self.index_writer_status.is_alive() && self.operation_sender.send(add_ops).is_ok() {\n            Ok(())\n        } else {\n            Err(error_in_index_worker_thread(\"An index writer was killed.\"))\n        }\n    }\n}\n\nimpl<D: Document> Drop for IndexWriter<D> {\n    fn drop(&mut self) {\n        self.segment_updater.kill();\n        self.drop_sender();\n        for work in self.workers_join_handle.drain(..) {\n            let _ = work.join();\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::{HashMap, HashSet};\n    use std::net::Ipv6Addr;\n\n    use columnar::{Column, MonotonicallyMappableToU128};\n    use itertools::Itertools;\n    use proptest::prop_oneof;\n\n    use super::super::operation::UserOperation;\n    use crate::collector::{Count, TopDocs};\n    use crate::directory::error::LockError;\n    use crate::error::*;\n    use crate::indexer::index_writer::MEMORY_BUDGET_NUM_BYTES_MIN;\n    use crate::indexer::{IndexWriterOptions, NoMergePolicy};\n    use crate::query::{QueryParser, TermQuery};\n    use crate::schema::{\n        self, Facet, FacetOptions, IndexRecordOption, IpAddrOptions, JsonObjectOptions,\n        NumericOptions, Schema, TextFieldIndexing, TextOptions, Value, FAST, INDEXED, STORED,\n        STRING, TEXT,\n    };\n    use crate::store::DOCSTORE_CACHE_CAPACITY;\n    use crate::{\n        DateTime, DocAddress, Index, IndexSettings, IndexWriter, ReloadPolicy, TantivyDocument,\n        Term,\n    };\n\n    const LOREM: &str = \"Doc Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do \\\n                         eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad \\\n                         minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip \\\n                         ex ea commodo consequat. Duis aute irure dolor in reprehenderit in \\\n                         voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur \\\n                         sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt \\\n                         mollit anim id est laborum.\";\n\n    #[test]\n    fn test_operations_group() {\n        // an operations group with 2 items should cause 3 opstamps 0, 1, and 2.\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let index_writer = index.writer_for_tests().unwrap();\n        let operations = vec![\n            UserOperation::Add(doc!(text_field=>\"a\")),\n            UserOperation::Add(doc!(text_field=>\"b\")),\n        ];\n        let batch_opstamp1 = index_writer.run(operations).unwrap();\n        assert_eq!(batch_opstamp1, 2u64);\n    }\n\n    #[test]\n    fn test_no_need_to_rewrite_delete_file_if_no_new_deletes() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"hello1\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"hello2\"))\n            .unwrap();\n        assert!(index_writer.commit().is_ok());\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        assert_eq!(searcher.segment_reader(0u32).num_docs(), 2);\n\n        index_writer.delete_term(Term::from_field_text(text_field, \"hello1\"));\n        assert!(index_writer.commit().is_ok());\n\n        assert!(reader.reload().is_ok());\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        assert_eq!(searcher.segment_reader(0u32).num_docs(), 1);\n\n        let previous_delete_opstamp = index.load_metas().unwrap().segments[0].delete_opstamp();\n\n        // All docs containing hello1 have been already removed.\n        // We should not update the delete meta.\n        index_writer.delete_term(Term::from_field_text(text_field, \"hello1\"));\n        assert!(index_writer.commit().is_ok());\n\n        assert!(reader.reload().is_ok());\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        assert_eq!(searcher.segment_reader(0u32).num_docs(), 1);\n\n        let after_delete_opstamp = index.load_metas().unwrap().segments[0].delete_opstamp();\n        assert_eq!(after_delete_opstamp, previous_delete_opstamp);\n    }\n\n    #[test]\n    fn test_ordered_batched_operations() {\n        // * one delete for `doc!(field=>\"a\")`\n        // * one add for `doc!(field=>\"a\")`\n        // * one add for `doc!(field=>\"b\")`\n        // * one delete for `doc!(field=>\"b\")`\n        // after commit there is one doc with \"a\" and 0 doc with \"b\"\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()\n            .unwrap();\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let a_term = Term::from_field_text(text_field, \"a\");\n        let b_term = Term::from_field_text(text_field, \"b\");\n        let operations = vec![\n            UserOperation::Delete(a_term),\n            UserOperation::Add(doc!(text_field=>\"a\")),\n            UserOperation::Add(doc!(text_field=>\"b\")),\n            UserOperation::Delete(b_term),\n        ];\n\n        index_writer.run(operations).unwrap();\n        index_writer.commit().expect(\"failed to commit\");\n        reader.reload().expect(\"failed to load searchers\");\n\n        let a_term = Term::from_field_text(text_field, \"a\");\n        let b_term = Term::from_field_text(text_field, \"b\");\n\n        let a_query = TermQuery::new(a_term, IndexRecordOption::Basic);\n        let b_query = TermQuery::new(b_term, IndexRecordOption::Basic);\n\n        let searcher = reader.searcher();\n\n        let a_docs = searcher\n            .search(&a_query, &TopDocs::with_limit(1).order_by_score())\n            .expect(\"search for a failed\");\n\n        let b_docs = searcher\n            .search(&b_query, &TopDocs::with_limit(1).order_by_score())\n            .expect(\"search for b failed\");\n\n        assert_eq!(a_docs.len(), 1);\n        assert_eq!(b_docs.len(), 0);\n    }\n\n    #[test]\n    fn test_empty_operations_group() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let operations1 = vec![];\n        let batch_opstamp1 = index_writer.run(operations1).unwrap();\n        assert_eq!(batch_opstamp1, 0u64);\n        let operations2 = vec![];\n        let batch_opstamp2 = index_writer.run(operations2).unwrap();\n        assert_eq!(batch_opstamp2, 1u64);\n    }\n\n    #[test]\n    fn test_lockfile_stops_duplicates() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let _index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        match index.writer_for_tests::<TantivyDocument>() {\n            Err(TantivyError::LockFailure(LockError::LockBusy, _)) => {}\n            _ => panic!(\"Expected a `LockFailure` error\"),\n        }\n    }\n\n    #[test]\n    fn test_lockfile_already_exists_error_msg() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let _index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        match index.writer_for_tests::<TantivyDocument>() {\n            Err(err) => {\n                let err_msg = err.to_string();\n                assert!(err_msg.contains(\"already an `IndexWriter`\"));\n            }\n            _ => panic!(\"Expected LockfileAlreadyExists error\"),\n        }\n    }\n\n    #[test]\n    fn test_set_merge_policy() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        assert_eq!(\n            format!(\"{:?}\", index_writer.get_merge_policy()),\n            \"LogMergePolicy { min_num_segments: 8, max_docs_before_merge: 10000000, \\\n             min_layer_size: 10000, level_log_size: 0.75, del_docs_ratio_before_merge: 1.0 }\"\n        );\n        let merge_policy = Box::<NoMergePolicy>::default();\n        index_writer.set_merge_policy(merge_policy);\n        assert_eq!(\n            format!(\"{:?}\", index_writer.get_merge_policy()),\n            \"NoMergePolicy\"\n        );\n    }\n\n    #[test]\n    fn test_lockfile_released_on_drop() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let _index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            // the lock should be released when the\n            // index_writer leaves the scope.\n        }\n        let _index_writer_two: IndexWriter = index.writer_for_tests().unwrap();\n    }\n\n    #[test]\n    fn test_commit_and_rollback() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        let num_docs_containing = |s: &str| {\n            let searcher = reader.searcher();\n            let term = Term::from_field_text(text_field, s);\n            searcher.doc_freq(&term).unwrap()\n        };\n\n        {\n            // writing the segment\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n            index_writer.rollback()?;\n            assert_eq!(index_writer.commit_opstamp(), 0u64);\n            assert_eq!(num_docs_containing(\"a\"), 0);\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n            index_writer.add_document(doc!(text_field=>\"c\"))?;\n            index_writer.commit()?;\n            reader.reload()?;\n            assert_eq!(num_docs_containing(\"a\"), 0);\n            assert_eq!(num_docs_containing(\"b\"), 1);\n            assert_eq!(num_docs_containing(\"c\"), 1);\n        }\n        reader.reload()?;\n        reader.searcher();\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_on_empty_segments_single_segment() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        let num_docs_containing = |s: &str| {\n            let term_a = Term::from_field_text(text_field, s);\n            reader.searcher().doc_freq(&term_a).unwrap()\n        };\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        //  this should create 1 segment\n\n        let segments = index.searchable_segment_ids().unwrap();\n        assert_eq!(segments.len(), 1);\n\n        reader.reload().unwrap();\n        assert_eq!(num_docs_containing(\"a\"), 1);\n\n        index_writer.delete_term(Term::from_field_text(text_field, \"a\"));\n        index_writer.commit()?;\n\n        reader.reload().unwrap();\n        assert_eq!(num_docs_containing(\"a\"), 0);\n\n        index_writer.merge(&segments);\n        index_writer.wait_merging_threads().unwrap();\n\n        let segments = index.searchable_segment_ids().unwrap();\n        assert_eq!(segments.len(), 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_on_empty_segments() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        let num_docs_containing = |s: &str| {\n            let term_a = Term::from_field_text(text_field, s);\n            reader.searcher().doc_freq(&term_a).unwrap()\n        };\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        //  this should create 4 segments\n\n        let segments = index.searchable_segment_ids().unwrap();\n        assert_eq!(segments.len(), 4);\n\n        reader.reload().unwrap();\n        assert_eq!(num_docs_containing(\"a\"), 4);\n\n        index_writer.delete_term(Term::from_field_text(text_field, \"a\"));\n        index_writer.commit()?;\n\n        reader.reload().unwrap();\n        assert_eq!(num_docs_containing(\"a\"), 0);\n\n        index_writer.merge(&segments);\n        index_writer.wait_merging_threads().unwrap();\n\n        let segments = index.searchable_segment_ids().unwrap();\n        assert_eq!(segments.len(), 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_with_merges() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        let num_docs_containing = |s: &str| {\n            let term_a = Term::from_field_text(text_field, s);\n            reader.searcher().doc_freq(&term_a).unwrap()\n        };\n        // writing the segment\n        let mut index_writer = index.writer(MEMORY_BUDGET_NUM_BYTES_MIN).unwrap();\n        // create 8 segments with 100 tiny docs\n        for _doc in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n        }\n        index_writer.commit()?;\n        for _doc in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n        }\n        //  this should create 8 segments and trigger a merge.\n        index_writer.commit()?;\n        index_writer.wait_merging_threads()?;\n        reader.reload()?;\n        assert_eq!(num_docs_containing(\"a\"), 200);\n        assert!(index.searchable_segments()?.len() < 8);\n        Ok(())\n    }\n\n    #[test]\n    fn test_prepare_with_commit_message() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        let mut index_writer = index.writer_for_tests()?;\n        for _doc in 0..100 {\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n        }\n        {\n            let mut prepared_commit = index_writer.prepare_commit()?;\n            prepared_commit.set_payload(\"first commit\");\n            prepared_commit.commit()?;\n        }\n        {\n            let metas = index.load_metas()?;\n            assert_eq!(metas.payload.unwrap(), \"first commit\");\n        }\n        for _doc in 0..100 {\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n        }\n        index_writer.commit()?;\n        {\n            let metas = index.load_metas()?;\n            assert!(metas.payload.is_none());\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_prepare_but_rollback() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        {\n            // writing the segment\n            let mut index_writer =\n                index.writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)?;\n            // create 8 segments with 100 tiny docs\n            for _doc in 0..100 {\n                index_writer.add_document(doc!(text_field => \"a\"))?;\n            }\n            {\n                let mut prepared_commit = index_writer.prepare_commit()?;\n                prepared_commit.set_payload(\"first commit\");\n                prepared_commit.abort()?;\n            }\n            {\n                let metas = index.load_metas()?;\n                assert!(metas.payload.is_none());\n            }\n            for _doc in 0..100 {\n                index_writer.add_document(doc!(text_field => \"b\"))?;\n            }\n            index_writer.commit()?;\n        }\n        let num_docs_containing = |s: &str| {\n            let term_a = Term::from_field_text(text_field, s);\n            index\n                .reader_builder()\n                .reload_policy(ReloadPolicy::Manual)\n                .try_into()?\n                .searcher()\n                .doc_freq(&term_a)\n        };\n        assert_eq!(num_docs_containing(\"a\")?, 0);\n        assert_eq!(num_docs_containing(\"b\")?, 100);\n        Ok(())\n    }\n\n    #[test]\n    fn test_add_then_delete_all_documents() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()\n            .unwrap();\n        let num_docs_containing = |s: &str| {\n            reader.reload().unwrap();\n            let searcher = reader.searcher();\n            let term = Term::from_field_text(text_field, s);\n            searcher.doc_freq(&term).unwrap()\n        };\n        let mut index_writer = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n\n        let add_tstamp = index_writer.add_document(doc!(text_field => \"a\")).unwrap();\n        let commit_tstamp = index_writer.commit().unwrap();\n        assert!(commit_tstamp > add_tstamp);\n        index_writer.delete_all_documents().unwrap();\n        index_writer.commit().unwrap();\n\n        // Search for documents with the same term that we added\n        assert_eq!(num_docs_containing(\"a\"), 0);\n    }\n\n    #[test]\n    fn test_delete_all_documents_rollback_correct_stamp() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n\n        let add_tstamp = index_writer.add_document(doc!(text_field => \"a\")).unwrap();\n\n        // commit documents - they are now available\n        let first_commit = index_writer.commit();\n        assert!(first_commit.is_ok());\n        let first_commit_tstamp = first_commit.unwrap();\n        assert!(first_commit_tstamp > add_tstamp);\n\n        // delete_all_documents the index\n        let clear_tstamp = index_writer.delete_all_documents().unwrap();\n        assert_eq!(clear_tstamp, add_tstamp);\n\n        // commit the clear command - now documents aren't available\n        let second_commit = index_writer.commit();\n        assert!(second_commit.is_ok());\n        let second_commit_tstamp = second_commit.unwrap();\n\n        // add new documents again\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field => \"b\")).unwrap();\n        }\n\n        // rollback to last commit, when index was empty\n        let rollback = index_writer.rollback();\n        assert!(rollback.is_ok());\n        let rollback_tstamp = rollback.unwrap();\n        assert_eq!(rollback_tstamp, second_commit_tstamp);\n\n        // working with an empty index == no documents\n        let term_b = Term::from_field_text(text_field, \"b\");\n        assert_eq!(\n            index\n                .reader()\n                .unwrap()\n                .searcher()\n                .doc_freq(&term_b)\n                .unwrap(),\n            0\n        );\n    }\n\n    #[test]\n    fn test_delete_all_documents_then_add() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        // writing the segment\n        let mut index_writer = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n        let res = index_writer.delete_all_documents();\n        assert!(res.is_ok());\n\n        assert!(index_writer.commit().is_ok());\n        // add one simple doc\n        index_writer.add_document(doc!(text_field => \"a\")).unwrap();\n        assert!(index_writer.commit().is_ok());\n\n        let term_a = Term::from_field_text(text_field, \"a\");\n        // expect the document with that term to be in the index\n        assert_eq!(\n            index\n                .reader()\n                .unwrap()\n                .searcher()\n                .doc_freq(&term_a)\n                .unwrap(),\n            1\n        );\n    }\n\n    #[test]\n    fn test_delete_all_documents_and_rollback() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n\n        // add one simple doc\n        assert!(index_writer.add_document(doc!(text_field => \"a\")).is_ok());\n        let comm = index_writer.commit();\n        assert!(comm.is_ok());\n        let commit_tstamp = comm.unwrap();\n\n        // clear but don't commit!\n        let clear_tstamp = index_writer.delete_all_documents().unwrap();\n        // clear_tstamp should reset to before the last commit\n        assert!(clear_tstamp < commit_tstamp);\n\n        // rollback\n        let _rollback_tstamp = index_writer.rollback().unwrap();\n        // Find original docs in the index\n        let term_a = Term::from_field_text(text_field, \"a\");\n        // expect the document with that term to be in the index\n        assert_eq!(\n            index\n                .reader()\n                .unwrap()\n                .searcher()\n                .doc_freq(&term_a)\n                .unwrap(),\n            1\n        );\n    }\n\n    #[test]\n    fn test_delete_all_documents_empty_index() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n        let clear = index_writer.delete_all_documents();\n        let commit = index_writer.commit();\n        assert!(clear.is_ok());\n        assert!(commit.is_ok());\n    }\n\n    #[test]\n    fn test_delete_all_documents_index_twice() {\n        let schema_builder = schema::Schema::builder();\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index\n            .writer_with_num_threads(4, MEMORY_BUDGET_NUM_BYTES_MIN * 4)\n            .unwrap();\n        let clear = index_writer.delete_all_documents();\n        let commit = index_writer.commit();\n        assert!(clear.is_ok());\n        assert!(commit.is_ok());\n        let clear_again = index_writer.delete_all_documents();\n        let commit_again = index_writer.commit();\n        assert!(clear_again.is_ok());\n        assert!(commit_again.is_ok());\n    }\n\n    #[test]\n    fn test_delete_and_merge_removes_terms_fast_field_dict() {\n        let mut schema_builder = schema::Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", STRING | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::builder().schema(schema).create_in_ram().unwrap();\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"one\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"two\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(text_field => \"three\"))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let index_reader = index.reader().unwrap();\n        let searcher = index_reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let text_fast_field = segment_reader.fast_fields().str(\"text\").unwrap().unwrap();\n        let mut buffer = String::new();\n        assert!(text_fast_field.ord_to_str(0, &mut buffer).unwrap());\n        assert_eq!(buffer, \"one\");\n        assert!(text_fast_field.ord_to_str(1, &mut buffer).unwrap());\n        assert_eq!(buffer, \"three\");\n        assert!(text_fast_field.ord_to_str(2, &mut buffer).unwrap());\n        assert_eq!(buffer, \"two\");\n        assert!(!text_fast_field.ord_to_str(3, &mut buffer).unwrap());\n\n        assert_eq!(segment_reader.max_doc(), 3);\n        index_writer.delete_term(Term::from_field_text(text_field, \"three\"));\n        index_writer.commit().unwrap();\n        index_writer\n            .merge(&[segment_reader.segment_id()])\n            .wait()\n            .unwrap();\n        index_reader.reload().unwrap();\n        let searcher = index_reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        assert_eq!(segment_reader.max_doc(), 2);\n        let text_fast_field = segment_reader.fast_fields().str(\"text\").unwrap().unwrap();\n        let mut buffer = String::new();\n        assert!(text_fast_field.ord_to_str(0, &mut buffer).unwrap());\n        assert_eq!(buffer, \"one\");\n        assert!(text_fast_field.ord_to_str(1, &mut buffer).unwrap());\n        assert_eq!(buffer, \"two\");\n        assert!(!text_fast_field.ord_to_str(2, &mut buffer).unwrap());\n        assert!(text_fast_field.term_ords(0).eq([0].into_iter()));\n        assert!(text_fast_field.term_ords(1).eq([1].into_iter()));\n    }\n\n    #[derive(Debug, Clone)]\n    enum IndexingOp {\n        AddMultipleDoc {\n            id: u64,\n            num_docs: u64,\n            value: IndexValue,\n        },\n        AddDoc {\n            id: u64,\n            value: IndexValue,\n        },\n        DeleteDoc {\n            id: u64,\n        },\n        DeleteDocQuery {\n            id: u64,\n        },\n        Commit,\n        Merge,\n    }\n    impl IndexingOp {\n        fn add(id: u64) -> Self {\n            IndexingOp::AddDoc {\n                id,\n                value: IndexValue::F64(id as f64),\n            }\n        }\n    }\n\n    use serde::Serialize;\n    #[derive(Debug, Clone, Serialize)]\n    #[serde(untagged)]\n    enum IndexValue {\n        Str(String),\n        F64(f64),\n        U64(u64),\n        I64(i64),\n    }\n    impl Default for IndexValue {\n        fn default() -> Self {\n            IndexValue::F64(0.0)\n        }\n    }\n\n    fn value_strategy() -> impl Strategy<Value = IndexValue> {\n        prop_oneof![\n            any::<f64>().prop_map(IndexValue::F64),\n            any::<u64>().prop_map(IndexValue::U64),\n            any::<i64>().prop_map(IndexValue::I64),\n            any::<String>().prop_map(IndexValue::Str),\n        ]\n    }\n\n    fn balanced_operation_strategy() -> impl Strategy<Value = IndexingOp> {\n        prop_oneof![\n            (0u64..20u64).prop_map(|id| IndexingOp::DeleteDoc { id }),\n            (0u64..20u64).prop_map(|id| IndexingOp::DeleteDocQuery { id }),\n            (0u64..20u64, value_strategy())\n                .prop_map(move |(id, value)| IndexingOp::AddDoc { id, value }),\n            ((0u64..20u64), (1u64..100), value_strategy()).prop_map(\n                move |(id, num_docs, value)| {\n                    IndexingOp::AddMultipleDoc {\n                        id,\n                        num_docs,\n                        value,\n                    }\n                }\n            ),\n            (0u64..1u64).prop_map(|_| IndexingOp::Commit),\n            (0u64..1u64).prop_map(|_| IndexingOp::Merge),\n        ]\n    }\n\n    fn adding_operation_strategy() -> impl Strategy<Value = IndexingOp> {\n        prop_oneof![\n            5 => (0u64..100u64).prop_map(|id| IndexingOp::DeleteDoc { id }),\n            5 => (0u64..100u64).prop_map(|id| IndexingOp::DeleteDocQuery { id }),\n            50 => (0u64..100u64, value_strategy())\n                .prop_map(move |(id, value)| IndexingOp::AddDoc { id, value }),\n            50 => (0u64..100u64, (1u64..100), value_strategy()).prop_map(\n                move |(id, num_docs, value)| {\n                    IndexingOp::AddMultipleDoc {\n                        id,\n                        num_docs,\n                        value,\n                    }\n                }\n            ),\n            2 => (0u64..1u64).prop_map(|_| IndexingOp::Commit),\n            1 => (0u64..1u64).prop_map(|_| IndexingOp::Merge),\n        ]\n    }\n\n    fn expected_ids(ops: &[IndexingOp]) -> (HashMap<u64, u64>, HashSet<u64>) {\n        let mut existing_ids = HashMap::new();\n        let mut deleted_ids = HashSet::new();\n        for op in ops {\n            match op {\n                IndexingOp::AddDoc { id, value: _ } => {\n                    *existing_ids.entry(*id).or_insert(0) += 1;\n                    deleted_ids.remove(id);\n                }\n                IndexingOp::AddMultipleDoc {\n                    id,\n                    num_docs,\n                    value: _,\n                } => {\n                    *existing_ids.entry(*id).or_insert(0) += num_docs;\n                    deleted_ids.remove(id);\n                }\n                IndexingOp::DeleteDoc { id } => {\n                    existing_ids.remove(id);\n                    deleted_ids.insert(*id);\n                }\n                IndexingOp::DeleteDocQuery { id } => {\n                    existing_ids.remove(id);\n                    deleted_ids.insert(*id);\n                }\n                _ => {}\n            }\n        }\n        (existing_ids, deleted_ids)\n    }\n\n    fn get_id_list(ops: &[IndexingOp]) -> Vec<u64> {\n        let mut id_list = Vec::new();\n        for op in ops {\n            match op {\n                IndexingOp::AddDoc { id, value: _ } => {\n                    id_list.push(*id);\n                }\n                IndexingOp::AddMultipleDoc { id, .. } => {\n                    id_list.push(*id);\n                }\n                IndexingOp::DeleteDoc { id } => {\n                    id_list.retain(|el| el != id);\n                }\n                IndexingOp::DeleteDocQuery { id } => {\n                    id_list.retain(|el| el != id);\n                }\n                _ => {}\n            }\n        }\n        id_list\n    }\n\n    fn test_operation_strategy(ops: &[IndexingOp], force_end_merge: bool) -> crate::Result<Index> {\n        let mut schema_builder = schema::Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", FAST | TEXT | STORED);\n        let ip_field = schema_builder.add_ip_addr_field(\"ip\", FAST | INDEXED | STORED);\n        let ips_field = schema_builder\n            .add_ip_addr_field(\"ips\", IpAddrOptions::default().set_fast().set_indexed());\n        let i64_field = schema_builder.add_i64_field(\"i64\", INDEXED);\n        let id_field = schema_builder.add_u64_field(\"id\", FAST | INDEXED | STORED);\n        let id_opt_field = schema_builder.add_u64_field(\"id_opt\", FAST | INDEXED | STORED);\n        let f64_field = schema_builder.add_f64_field(\"f64\", INDEXED);\n        let date_field = schema_builder.add_date_field(\"date\", INDEXED);\n        let bytes_field = schema_builder.add_bytes_field(\"bytes\", FAST | INDEXED | STORED);\n        let bool_field = schema_builder.add_bool_field(\"bool\", FAST | INDEXED | STORED);\n        let text_field = schema_builder.add_text_field(\n            \"text_field\",\n            TextOptions::default()\n                .set_indexing_options(\n                    TextFieldIndexing::default()\n                        .set_index_option(schema::IndexRecordOption::WithFreqsAndPositions),\n                )\n                .set_stored(),\n        );\n\n        let large_text_field = schema_builder.add_text_field(\"large_text_field\", TEXT | STORED);\n        let multi_text_fields = schema_builder.add_text_field(\"multi_text_fields\", TEXT | STORED);\n\n        let multi_numbers = schema_builder.add_u64_field(\n            \"multi_numbers\",\n            NumericOptions::default().set_fast().set_stored(),\n        );\n        let multi_bools = schema_builder.add_bool_field(\n            \"multi_bools\",\n            NumericOptions::default().set_fast().set_stored(),\n        );\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n        let settings = {\n            IndexSettings {\n                ..Default::default()\n            }\n        };\n        let index = Index::builder()\n            .schema(schema)\n            .settings(settings)\n            .create_in_ram()?;\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n        let old_reader = index.reader()?;\n\n        // Every 3rd doc has only id field\n        let id_is_full_doc = |id| id % 3 != 0;\n\n        let multi_text_field_text1 = \"test1 test2 test3 test1 test2 test3\";\n        // rotate left\n        let multi_text_field_text2 = \"test2 test3 test1 test2 test3 test1\";\n        // rotate right\n        let multi_text_field_text3 = \"test3 test1 test2 test3 test1 test2\";\n\n        let ip_from_id = |id| Ipv6Addr::from_u128(id as u128);\n\n        let add_docs = |index_writer: &mut IndexWriter,\n                        id: u64,\n                        value: IndexValue,\n                        num: u64|\n         -> crate::Result<()> {\n            let facet = Facet::from(&(\"/cola/\".to_string() + &id.to_string()));\n            let ip = ip_from_id(id);\n            let doc = if !id_is_full_doc(id) {\n                // every 3rd doc has no ip field\n                doc!(\n                    id_field=>id,\n                )\n            } else {\n                let json = json!({\"date1\": format!(\"2022-{id}-01T00:00:01Z\"), \"date2\": format!(\"{id}-05-01T00:00:01Z\"), \"id\": id, \"ip\": ip.to_string(), \"val\": value});\n                doc!(id_field=>id,\n                        json_field=>json,\n                        bytes_field => id.to_le_bytes().as_slice(),\n                        id_opt_field => id,\n                        ip_field => ip,\n                        ips_field => ip,\n                        ips_field => ip,\n                        multi_numbers=> id,\n                        multi_numbers => id,\n                        bool_field => (id % 2u64) != 0,\n                        i64_field => id as i64,\n                        f64_field => id as f64,\n                        date_field => DateTime::from_timestamp_secs(id as i64),\n                        multi_bools => (id % 2u64) != 0,\n                        multi_bools => (id % 2u64) == 0,\n                        text_field => id.to_string(),\n                        facet_field => facet,\n                        large_text_field => LOREM,\n                        multi_text_fields => multi_text_field_text1,\n                        multi_text_fields => multi_text_field_text2,\n                        multi_text_fields => multi_text_field_text3,\n                )\n            };\n            for _ in 0..num {\n                index_writer.add_document(doc.clone())?;\n            }\n            Ok(())\n        };\n        for op in ops {\n            match op.clone() {\n                IndexingOp::AddMultipleDoc {\n                    id,\n                    num_docs,\n                    value,\n                } => {\n                    add_docs(&mut index_writer, id, value, num_docs)?;\n                }\n                IndexingOp::AddDoc { id, value } => {\n                    add_docs(&mut index_writer, id, value, 1)?;\n                }\n                IndexingOp::DeleteDoc { id } => {\n                    index_writer.delete_term(Term::from_field_u64(id_field, id));\n                }\n                IndexingOp::DeleteDocQuery { id } => {\n                    let term = Term::from_field_u64(id_field, id);\n                    let query = TermQuery::new(term, Default::default());\n                    index_writer.delete_query(Box::new(query))?;\n                }\n                IndexingOp::Commit => {\n                    index_writer.commit()?;\n                }\n                IndexingOp::Merge => {\n                    let mut segment_ids = index\n                        .searchable_segment_ids()\n                        .expect(\"Searchable segments failed.\");\n                    segment_ids.sort();\n                    if segment_ids.len() >= 2 {\n                        index_writer.merge(&segment_ids).wait().unwrap();\n                        assert!(index_writer.segment_updater().wait_merging_thread().is_ok());\n                    }\n                }\n            }\n        }\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n        let num_segments_before_merge = searcher.segment_readers().len();\n        if force_end_merge {\n            index_writer.wait_merging_threads()?;\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            if segment_ids.len() >= 2 {\n                index_writer.merge(&segment_ids).wait().unwrap();\n                assert!(index_writer.wait_merging_threads().is_ok());\n            }\n        }\n        let num_segments_after_merge = searcher.segment_readers().len();\n\n        old_reader.reload()?;\n        let old_searcher = old_reader.searcher();\n\n        let ids_old_searcher: HashSet<u64> = old_searcher\n            .segment_readers()\n            .iter()\n            .flat_map(|segment_reader| {\n                let ff_reader = segment_reader.fast_fields().u64(\"id\").unwrap();\n                segment_reader\n                    .doc_ids_alive()\n                    .flat_map(move |doc| ff_reader.values_for_doc(doc).collect_vec().into_iter())\n            })\n            .collect();\n\n        let ids: HashSet<u64> = searcher\n            .segment_readers()\n            .iter()\n            .flat_map(|segment_reader| {\n                let ff_reader = segment_reader.fast_fields().u64(\"id\").unwrap();\n                segment_reader\n                    .doc_ids_alive()\n                    .flat_map(move |doc| ff_reader.values_for_doc(doc).collect_vec().into_iter())\n            })\n            .collect();\n\n        let (expected_ids_and_num_occurrences, deleted_ids) = expected_ids(ops);\n\n        let id_list = get_id_list(ops);\n\n        // multivalue fast field content\n        let mut all_ips = Vec::new();\n        let mut num_ips = 0;\n        for segment_reader in searcher.segment_readers().iter() {\n            let ip_reader: Column<Ipv6Addr> = segment_reader\n                .fast_fields()\n                .column_opt(\"ips\")\n                .unwrap()\n                .unwrap();\n            for doc in segment_reader.doc_ids_alive() {\n                all_ips.extend(ip_reader.values_for_doc(doc));\n            }\n            num_ips += ip_reader.values.num_vals();\n        }\n\n        let num_docs_expected = expected_ids_and_num_occurrences\n            .values()\n            .map(|id_occurrences| *id_occurrences as usize)\n            .sum::<usize>();\n\n        let num_docs_with_values = expected_ids_and_num_occurrences\n            .iter()\n            .filter(|(id, _id_occurrences)| id_is_full_doc(**id))\n            .map(|(_, id_occurrences)| *id_occurrences as usize)\n            .sum::<usize>();\n\n        assert_eq!(searcher.num_docs() as usize, num_docs_expected);\n        assert_eq!(old_searcher.num_docs() as usize, num_docs_expected);\n        assert_eq!(\n            ids_old_searcher,\n            expected_ids_and_num_occurrences\n                .keys()\n                .cloned()\n                .collect::<HashSet<_>>()\n        );\n        assert_eq!(\n            ids,\n            expected_ids_and_num_occurrences\n                .keys()\n                .cloned()\n                .collect::<HashSet<_>>()\n        );\n\n        if force_end_merge && num_segments_before_merge > 1 && num_segments_after_merge == 1 {\n            let mut expected_multi_ips: Vec<_> = id_list\n                .iter()\n                .filter(|id| id_is_full_doc(**id))\n                .flat_map(|id| vec![ip_from_id(*id), ip_from_id(*id)])\n                .collect();\n            assert_eq!(num_ips, expected_multi_ips.len() as u32);\n\n            expected_multi_ips.sort();\n            all_ips.sort();\n            assert_eq!(expected_multi_ips, all_ips);\n\n            // Test fastfield num_docs\n            let num_docs: usize = searcher\n                .segment_readers()\n                .iter()\n                .map(|segment_reader| {\n                    let ff_reader = segment_reader\n                        .fast_fields()\n                        .column_opt::<Ipv6Addr>(\"ips\")\n                        .unwrap()\n                        .unwrap();\n                    ff_reader.num_docs() as usize\n                })\n                .sum();\n            assert_eq!(num_docs, num_docs_expected);\n        }\n\n        // Load all ips addr\n        let mut ips: HashSet<Ipv6Addr> = Default::default();\n        for reader in searcher.segment_readers() {\n            if let Some(ff_reader) = reader.fast_fields().column_opt::<Ipv6Addr>(\"ips\").unwrap() {\n                for doc in reader.doc_ids_alive() {\n                    ips.extend(ff_reader.values_for_doc(doc));\n                }\n            }\n        }\n\n        let expected_ips = expected_ids_and_num_occurrences\n            .keys()\n            .flat_map(|id| {\n                if !id_is_full_doc(*id) {\n                    None\n                } else {\n                    Some(Ipv6Addr::from_u128(*id as u128))\n                }\n            })\n            .collect::<HashSet<_>>();\n        assert_eq!(ips, expected_ips);\n\n        let expected_ips = expected_ids_and_num_occurrences\n            .keys()\n            .filter_map(|id| {\n                if !id_is_full_doc(*id) {\n                    None\n                } else {\n                    Some(Ipv6Addr::from_u128(*id as u128))\n                }\n            })\n            .collect::<HashSet<_>>();\n\n        let mut ips: HashSet<Ipv6Addr> = Default::default();\n        for reader in searcher.segment_readers() {\n            if let Some(ff_reader) = reader.fast_fields().column_opt::<Ipv6Addr>(\"ips\").unwrap() {\n                for doc in reader.doc_ids_alive() {\n                    ips.extend(ff_reader.values_for_doc(doc));\n                }\n            }\n        }\n        assert_eq!(ips, expected_ips);\n\n        // multivalue fast field tests\n        for segment_reader in searcher.segment_readers().iter() {\n            let id_reader = segment_reader.fast_fields().u64(\"id\").unwrap();\n            let ff_reader = segment_reader\n                .fast_fields()\n                .column_opt(\"multi_numbers\")\n                .unwrap()\n                .unwrap();\n            let bool_ff_reader = segment_reader\n                .fast_fields()\n                .column_opt::<bool>(\"multi_bools\")\n                .unwrap()\n                .unwrap();\n            for doc in segment_reader.doc_ids_alive() {\n                let id = id_reader.first(doc).unwrap();\n\n                let vals: Vec<u64> = ff_reader.values_for_doc(doc).collect();\n                if id_is_full_doc(id) {\n                    assert_eq!(vals.len(), 2);\n                    assert_eq!(vals[0], vals[1]);\n                    assert!(expected_ids_and_num_occurrences.contains_key(&vals[0]));\n                    assert_eq!(id_reader.first(doc), Some(vals[0]));\n                } else {\n                    assert_eq!(vals.len(), 0);\n                }\n\n                let bool_vals: Vec<bool> = bool_ff_reader.values_for_doc(doc).collect();\n                if id_is_full_doc(id) {\n                    assert_eq!(bool_vals.len(), 2);\n                    assert_ne!(bool_vals[0], bool_vals[1]);\n                } else {\n                    assert_eq!(bool_vals.len(), 0);\n                }\n            }\n        }\n\n        // doc store tests\n        for segment_reader in searcher.segment_readers().iter() {\n            let store_reader = segment_reader\n                .get_store_reader(DOCSTORE_CACHE_CAPACITY)\n                .unwrap();\n            // test store iterator\n            for doc in store_reader.iter::<TantivyDocument>(segment_reader.alive_bitset()) {\n                let id = doc\n                    .unwrap()\n                    .get_first(id_field)\n                    .unwrap()\n                    .as_value()\n                    .as_u64()\n                    .unwrap();\n                assert!(expected_ids_and_num_occurrences.contains_key(&id));\n            }\n            // test store random access\n            for doc_id in segment_reader.doc_ids_alive() {\n                let id = store_reader\n                    .get::<TantivyDocument>(doc_id)\n                    .unwrap()\n                    .get_first(id_field)\n                    .unwrap()\n                    .as_u64()\n                    .unwrap();\n                assert!(expected_ids_and_num_occurrences.contains_key(&id));\n                if id_is_full_doc(id) {\n                    let id2 = store_reader\n                        .get::<TantivyDocument>(doc_id)\n                        .unwrap()\n                        .get_first(multi_numbers)\n                        .unwrap()\n                        .as_u64()\n                        .unwrap();\n                    assert_eq!(id, id2);\n                    let bool = store_reader\n                        .get::<TantivyDocument>(doc_id)\n                        .unwrap()\n                        .get_first(bool_field)\n                        .unwrap()\n                        .as_bool()\n                        .unwrap();\n                    let doc = store_reader.get::<TantivyDocument>(doc_id).unwrap();\n                    let mut bool2 = doc.get_all(multi_bools);\n                    assert_eq!(bool, bool2.next().unwrap().as_bool().unwrap());\n                    assert_ne!(bool, bool2.next().unwrap().as_bool().unwrap());\n                    assert!(bool2.next().is_none())\n                }\n            }\n        }\n        // test search\n        let do_search = |term: &str, field| {\n            let query = QueryParser::for_index(&index, vec![field])\n                .parse_query(term)\n                .unwrap();\n            let top_docs: Vec<(f32, DocAddress)> = searcher\n                .search(&query, &TopDocs::with_limit(1000).order_by_score())\n                .unwrap();\n\n            top_docs.iter().map(|el| el.1).collect::<Vec<_>>()\n        };\n        let count_search = |term: &str, field| {\n            let query = QueryParser::for_index(&index, vec![field])\n                .parse_query(term)\n                .unwrap();\n            searcher.search(&query, &Count).unwrap()\n        };\n\n        let count_search2 = |term: Term| {\n            let query = TermQuery::new(term, IndexRecordOption::Basic);\n            searcher.search(&query, &Count).unwrap()\n        };\n\n        for (id, count) in &expected_ids_and_num_occurrences {\n            // skip expensive queries\n            let (existing_id, count) = (*id, *count);\n            let get_num_hits = |field| count_search(&existing_id.to_string(), field) as u64;\n            assert_eq!(get_num_hits(id_field), count);\n            if !id_is_full_doc(existing_id) {\n                continue;\n            }\n            assert_eq!(get_num_hits(text_field), count);\n            assert_eq!(get_num_hits(i64_field), count);\n            assert_eq!(get_num_hits(f64_field), count);\n\n            // Test multi text\n            if num_docs_with_values < 1000 {\n                assert_eq!(\n                    do_search(\"\\\"test1 test2\\\"\", multi_text_fields).len(),\n                    num_docs_with_values\n                );\n                assert_eq!(\n                    do_search(\"\\\"test2 test3\\\"\", multi_text_fields).len(),\n                    num_docs_with_values\n                );\n            }\n\n            // Test bytes\n            let term = Term::from_field_bytes(bytes_field, existing_id.to_le_bytes().as_slice());\n            assert_eq!(count_search2(term) as u64, count);\n\n            // Test date\n            let term = Term::from_field_date(\n                date_field,\n                DateTime::from_timestamp_secs(existing_id as i64),\n            );\n            assert_eq!(count_search2(term) as u64, count);\n        }\n        for deleted_id in deleted_ids {\n            let assert_field = |field| {\n                assert_eq!(count_search(&deleted_id.to_string(), field) as u64, 0);\n            };\n            assert_field(text_field);\n            assert_field(f64_field);\n            assert_field(i64_field);\n            assert_field(id_field);\n\n            // Test bytes\n            let term = Term::from_field_bytes(bytes_field, deleted_id.to_le_bytes().as_slice());\n            assert_eq!(count_search2(term), 0);\n\n            // Test date\n            let term =\n                Term::from_field_date(date_field, DateTime::from_timestamp_secs(deleted_id as i64));\n            assert_eq!(count_search2(term), 0);\n        }\n        // search ip address\n        //\n        for (existing_id, count) in &expected_ids_and_num_occurrences {\n            let (existing_id, count) = (*existing_id, *count);\n            if !id_is_full_doc(existing_id) {\n                continue;\n            }\n            let do_search_ip_field = |term: &str| count_search(term, ip_field) as u64;\n            let ip_addr = Ipv6Addr::from_u128(existing_id as u128);\n            // Test incoming ip as ipv6\n            assert_eq!(do_search_ip_field(&format!(\"\\\"{ip_addr}\\\"\")), count);\n\n            let term = Term::from_field_ip_addr(ip_field, ip_addr);\n            assert_eq!(count_search2(term) as u64, count);\n\n            // Test incoming ip as ipv4\n            if let Some(ip_addr) = ip_addr.to_ipv4_mapped() {\n                assert_eq!(do_search_ip_field(&format!(\"\\\"{ip_addr}\\\"\")), count);\n            }\n        }\n\n        // Range query\n        //\n        // Take half as sample\n        let mut sample: Vec<_> = expected_ids_and_num_occurrences.iter().collect();\n        sample.sort_by_key(|(k, _num_occurrences)| *k);\n        // sample.truncate(sample.len() / 2);\n        if !sample.is_empty() {\n            let (left_sample, right_sample) = sample.split_at(sample.len() / 2);\n\n            let calc_expected_count = |sample: &[(&u64, &u64)]| {\n                sample\n                    .iter()\n                    .filter(|(id, _)| id_is_full_doc(**id))\n                    .map(|(_id, num_occurrences)| **num_occurrences)\n                    .sum::<u64>()\n            };\n            fn gen_query_inclusive<T1: ToString, T2: ToString>(\n                field: &str,\n                from: T1,\n                to: T2,\n            ) -> String {\n                format!(\"{}:[{} TO {}]\", field, &from.to_string(), &to.to_string())\n            }\n\n            // Query first half\n            let expected_count = calc_expected_count(left_sample);\n            if !left_sample.is_empty() && expected_count < 1000 {\n                let start_range = *left_sample[0].0;\n                let end_range = *left_sample.last().unwrap().0;\n                let query = gen_query_inclusive(\"id_opt\", start_range, end_range);\n                assert_eq!(count_search(&query, id_opt_field) as u64, expected_count);\n\n                // Range query on ip field\n                let ip1 = ip_from_id(start_range);\n                let ip2 = ip_from_id(end_range);\n                let do_search_ip_field = |term: &str| count_search(term, ip_field) as u64;\n                let query = gen_query_inclusive(\"ip\", ip1, ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                let query = gen_query_inclusive(\"ip\", \"*\", ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                // Range query on multi value field\n                let query = gen_query_inclusive(\"ips\", ip1, ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                let query = gen_query_inclusive(\"ips\", \"*\", ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n            }\n            // Query second half\n            let expected_count = calc_expected_count(right_sample);\n            if !right_sample.is_empty() && expected_count < 1000 {\n                let start_range = *right_sample[0].0;\n                let end_range = *right_sample.last().unwrap().0;\n                // Range query on id opt field\n                let query =\n                    gen_query_inclusive(\"id_opt\", start_range.to_string(), end_range.to_string());\n                assert_eq!(count_search(&query, id_opt_field) as u64, expected_count);\n\n                // Range query on ip field\n                let ip1 = ip_from_id(start_range);\n                let ip2 = ip_from_id(end_range);\n                let do_search_ip_field = |term: &str| count_search(term, ip_field) as u64;\n                let query = gen_query_inclusive(\"ip\", ip1, ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                let query = gen_query_inclusive(\"ip\", ip1, \"*\");\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                // Range query on multi value field\n                let query = gen_query_inclusive(\"ips\", ip1, ip2);\n                assert_eq!(do_search_ip_field(&query), expected_count);\n                let query = gen_query_inclusive(\"ips\", ip1, \"*\");\n                assert_eq!(do_search_ip_field(&query), expected_count);\n            }\n        }\n\n        // ip range query on fast field\n        //\n        for (existing_id, count) in expected_ids_and_num_occurrences.iter().take(10) {\n            let (existing_id, count) = (*existing_id, *count);\n            if !id_is_full_doc(existing_id) {\n                continue;\n            }\n            let gen_query_inclusive = |field: &str, from: Ipv6Addr, to: Ipv6Addr| {\n                format!(\"{}:[{} TO {}]\", field, &from.to_string(), &to.to_string())\n            };\n            let ip = ip_from_id(existing_id);\n\n            let do_search_ip_field = |term: &str| count_search(term, ip_field) as u64;\n            // Range query on single value field\n            let query = gen_query_inclusive(\"ip\", ip, ip);\n            assert_eq!(do_search_ip_field(&query), count);\n\n            // Range query on multi value field\n            let query = gen_query_inclusive(\"ips\", ip, ip);\n            assert_eq!(do_search_ip_field(&query), count);\n        }\n\n        // test facets\n        for segment_reader in searcher.segment_readers().iter() {\n            let facet_reader = segment_reader.facet_reader(\"facet\").unwrap();\n            let ff_reader = segment_reader\n                .fast_fields()\n                .u64(\"id\")\n                .unwrap()\n                .first_or_default_col(9999);\n            for doc_id in segment_reader.doc_ids_alive() {\n                let id = ff_reader.get_val(doc_id);\n                if !id_is_full_doc(id) {\n                    continue;\n                }\n                let facet_ords: Vec<u64> = facet_reader.facet_ords(doc_id).collect();\n                assert_eq!(facet_ords.len(), 1);\n                let mut facet = Facet::default();\n                facet_reader\n                    .facet_from_ord(facet_ords[0], &mut facet)\n                    .unwrap();\n                let facet_expected = Facet::from(&(\"/cola/\".to_string() + &id.to_string()));\n\n                assert_eq!(facet, facet_expected);\n            }\n        }\n\n        Ok(index)\n    }\n\n    #[test]\n    fn test_fast_field_range() {\n        let ops: Vec<_> = (0..1000).map(IndexingOp::add).collect();\n        assert!(test_operation_strategy(&ops, true).is_ok());\n    }\n\n    #[test]\n    fn test_ip_range_query_multivalue_bug() {\n        assert!(test_operation_strategy(\n            &[\n                IndexingOp::add(2),\n                IndexingOp::Commit,\n                IndexingOp::add(1),\n                IndexingOp::add(1),\n                IndexingOp::Commit,\n                IndexingOp::Merge\n            ],\n            false\n        )\n        .is_ok());\n    }\n\n    #[test]\n    fn test_ff_num_ips_regression() {\n        assert!(test_operation_strategy(\n            &[\n                IndexingOp::add(13),\n                IndexingOp::add(1),\n                IndexingOp::Commit,\n                IndexingOp::DeleteDocQuery { id: 13 },\n                IndexingOp::add(1),\n                IndexingOp::Commit,\n            ],\n            true\n        )\n        .is_ok());\n    }\n\n    #[test]\n    fn test_minimal_sort_force_end_merge() {\n        assert!(\n            test_operation_strategy(&[IndexingOp::add(23), IndexingOp::add(13),], false).is_ok()\n        );\n    }\n\n    #[test]\n    fn test_minimal_no_force_end_merge() {\n        assert!(test_operation_strategy(\n            &[\n                IndexingOp::add(23),\n                IndexingOp::add(13),\n                IndexingOp::DeleteDoc { id: 13 }\n            ],\n            false\n        )\n        .is_ok());\n    }\n\n    use proptest::prelude::*;\n\n    proptest! {\n\n        #![proptest_config(ProptestConfig::with_cases(20))]\n        #[test]\n        fn test_delete_proptest_adding(ops in proptest::collection::vec(adding_operation_strategy(), 1..100)) {\n            assert!(test_operation_strategy(&ops[..],  false).is_ok());\n        }\n\n        #[test]\n        fn test_delete_proptest_with_merge_adding(ops in proptest::collection::vec(adding_operation_strategy(), 1..100)) {\n            assert!(test_operation_strategy(&ops[..],  true).is_ok());\n        }\n\n        #[test]\n        fn test_delete_proptest(ops in proptest::collection::vec(balanced_operation_strategy(), 1..10)) {\n            assert!(test_operation_strategy(&ops[..],  false).is_ok());\n        }\n\n        #[test]\n        fn test_delete_proptest_with_merge(ops in proptest::collection::vec(balanced_operation_strategy(), 1..100)) {\n            assert!(test_operation_strategy(&ops[..],  true).is_ok());\n        }\n    }\n\n    #[test]\n    fn test_delete_bug_reproduction_ip_addr() {\n        use IndexingOp::*;\n        let ops = &[\n            IndexingOp::add(1),\n            IndexingOp::add(2),\n            Commit,\n            IndexingOp::add(3),\n            DeleteDoc { id: 1 },\n            Commit,\n            Merge,\n            IndexingOp::add(4),\n            Commit,\n        ];\n        test_operation_strategy(&ops[..], true).unwrap();\n    }\n\n    #[test]\n    fn test_merge_regression_1() {\n        use IndexingOp::*;\n        let ops = &[\n            IndexingOp::add(15),\n            Commit,\n            IndexingOp::add(9),\n            Commit,\n            Merge,\n        ];\n        test_operation_strategy(&ops[..], true).unwrap();\n    }\n\n    #[test]\n    fn test_range_query_bug_1() {\n        use IndexingOp::*;\n        let ops = &[\n            IndexingOp::add(9),\n            IndexingOp::add(0),\n            IndexingOp::add(13),\n            Commit,\n        ];\n        test_operation_strategy(&ops[..], true).unwrap();\n    }\n\n    #[test]\n    fn test_range_query_bug_2() {\n        let ops = &[\n            IndexingOp::add(3),\n            IndexingOp::add(6),\n            IndexingOp::add(9),\n            IndexingOp::add(10),\n        ];\n        test_operation_strategy(&ops[..], false).unwrap();\n    }\n\n    #[test]\n    fn test_index_doc_missing_field() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let idfield = schema_builder.add_text_field(\"id\", STRING);\n        schema_builder.add_text_field(\"optfield\", STRING);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(idfield=>\"myid\"))?;\n        index_writer.commit()?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_bug_1617_3() {\n        assert!(test_operation_strategy(\n            &[\n                IndexingOp::DeleteDoc { id: 0 },\n                IndexingOp::add(6),\n                IndexingOp::DeleteDocQuery { id: 11 },\n                IndexingOp::Commit,\n                IndexingOp::Merge,\n                IndexingOp::Commit,\n                IndexingOp::Commit\n            ],\n            false\n        )\n        .is_ok());\n    }\n\n    #[test]\n    fn test_bug_1617_2() {\n        test_operation_strategy(\n            &[\n                IndexingOp::AddDoc {\n                    id: 13,\n                    value: Default::default(),\n                },\n                IndexingOp::DeleteDoc { id: 13 },\n                IndexingOp::Commit,\n                IndexingOp::add(30),\n                IndexingOp::Commit,\n                IndexingOp::Merge,\n            ],\n            true,\n        )\n        .unwrap();\n    }\n\n    #[test]\n    fn test_bug_1617() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let id_field = schema_builder.add_u64_field(\"id\", INDEXED);\n\n        let schema = schema_builder.build();\n        let index = Index::builder().schema(schema).create_in_ram()?;\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n        let existing_id = 16u64;\n        let deleted_id = 13u64;\n        index_writer.add_document(doc!(\n            id_field=>existing_id,\n        ))?;\n        index_writer.add_document(doc!(\n            id_field=>deleted_id,\n        ))?;\n        index_writer.delete_term(Term::from_field_u64(id_field, deleted_id));\n        index_writer.commit()?;\n\n        // Merge\n        {\n            assert!(index_writer.wait_merging_threads().is_ok());\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            index_writer.merge(&segment_ids).wait().unwrap();\n            assert!(index_writer.wait_merging_threads().is_ok());\n        }\n        let searcher = index.reader()?.searcher();\n\n        let query = TermQuery::new(\n            Term::from_field_u64(id_field, existing_id),\n            IndexRecordOption::Basic,\n        );\n        let top_docs: Vec<(f32, DocAddress)> = searcher\n            .search(&query, &TopDocs::with_limit(10).order_by_score())\n            .unwrap();\n\n        assert_eq!(top_docs.len(), 1); // Was failing\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_bug_1618() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let id_field = schema_builder.add_i64_field(\"id\", INDEXED);\n\n        let schema = schema_builder.build();\n        let index = Index::builder().schema(schema).create_in_ram()?;\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n        index_writer.add_document(doc!(\n            id_field=>10i64,\n        ))?;\n        index_writer.add_document(doc!(\n            id_field=>30i64,\n        ))?;\n        index_writer.commit()?;\n\n        // Merge\n        {\n            assert!(index_writer.wait_merging_threads().is_ok());\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            index_writer.merge(&segment_ids).wait().unwrap();\n            assert!(index_writer.wait_merging_threads().is_ok());\n        }\n        let searcher = index.reader()?.searcher();\n\n        let query = TermQuery::new(\n            Term::from_field_i64(id_field, 10i64),\n            IndexRecordOption::Basic,\n        );\n        let top_docs: Vec<(f32, DocAddress)> = searcher\n            .search(&query, &TopDocs::with_limit(10).order_by_score())\n            .unwrap();\n\n        assert_eq!(top_docs.len(), 1); // Fails\n\n        let query = TermQuery::new(\n            Term::from_field_i64(id_field, 30i64),\n            IndexRecordOption::Basic,\n        );\n        let top_docs: Vec<(f32, DocAddress)> = searcher\n            .search(&query, &TopDocs::with_limit(10).order_by_score())\n            .unwrap();\n\n        assert_eq!(top_docs.len(), 1); // Fails\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_bug_2442_reserved_character_fast_field() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", FAST | TEXT);\n\n        let schema = schema_builder.build();\n        let index = Index::builder().schema(schema).create_in_ram()?;\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n        index_writer\n            .add_document(doc!(\n                json_field=>json!({\"\\u{0000}B\":\"1\"})\n            ))\n            .unwrap();\n        index_writer\n            .add_document(doc!(\n                json_field=>json!({\" A\":\"1\"})\n            ))\n            .unwrap();\n        index_writer.commit()?;\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_bug_2442_reserved_character_columnar() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let options = JsonObjectOptions::from(FAST).set_expand_dots_enabled();\n        let field = schema_builder.add_json_field(\"json\", options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({\"\\u{0000}\": \"A\"})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({format!(\"\\u{0000}\\u{0000}\"): \"A\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        Ok(())\n    }\n\n    #[test]\n    fn test_writer_options_validation() {\n        let mut schema_builder = Schema::builder();\n        let _field = schema_builder.add_bool_field(\"example\", STORED);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        let opt_wo_threads = IndexWriterOptions::builder().num_worker_threads(0).build();\n        let result = index.writer_with_options::<TantivyDocument>(opt_wo_threads);\n        assert!(result.is_err(), \"Writer should reject 0 thread count\");\n        assert!(matches!(result, Err(TantivyError::InvalidArgument(_))));\n\n        let opt_with_low_memory = IndexWriterOptions::builder()\n            .memory_budget_per_thread(10 << 10)\n            .build();\n        let result = index.writer_with_options::<TantivyDocument>(opt_with_low_memory);\n        assert!(\n            result.is_err(),\n            \"Writer should reject options with too low memory size\"\n        );\n        assert!(matches!(result, Err(TantivyError::InvalidArgument(_))));\n\n        let opt_with_low_memory = IndexWriterOptions::builder()\n            .memory_budget_per_thread(5 << 30)\n            .build();\n        let result = index.writer_with_options::<TantivyDocument>(opt_with_low_memory);\n        assert!(\n            result.is_err(),\n            \"Writer should reject options with too high memory size\"\n        );\n        assert!(matches!(result, Err(TantivyError::InvalidArgument(_))));\n    }\n}\n"
  },
  {
    "path": "src/indexer/index_writer_status.rs",
    "content": "use std::sync::atomic::{AtomicBool, Ordering};\nuse std::sync::{Arc, RwLock};\n\nuse super::AddBatchReceiver;\nuse crate::schema::document::Document;\nuse crate::TantivyDocument;\n\n#[derive(Clone)]\npub(crate) struct IndexWriterStatus<D: Document = TantivyDocument> {\n    inner: Arc<Inner<D>>,\n}\n\nimpl<D: Document> IndexWriterStatus<D> {\n    /// Returns true iff the index writer is alive.\n    pub fn is_alive(&self) -> bool {\n        self.inner.as_ref().is_alive()\n    }\n\n    /// Returns a copy of the operation receiver.\n    /// If the index writer was killed, returns `None`.\n    pub fn operation_receiver(&self) -> Option<AddBatchReceiver<D>> {\n        let rlock = self\n            .inner\n            .receive_channel\n            .read()\n            .expect(\"This lock should never be poisoned\");\n        rlock.as_ref().cloned()\n    }\n\n    /// Create an index writer bomb.\n    /// If dropped, the index writer status will be killed.\n    pub(crate) fn create_bomb(&self) -> IndexWriterBomb<D> {\n        IndexWriterBomb {\n            inner: Some(self.inner.clone()),\n        }\n    }\n}\n\nstruct Inner<D: Document> {\n    is_alive: AtomicBool,\n    receive_channel: RwLock<Option<AddBatchReceiver<D>>>,\n}\n\nimpl<D: Document> Inner<D> {\n    fn is_alive(&self) -> bool {\n        self.is_alive.load(Ordering::Relaxed)\n    }\n\n    fn kill(&self) {\n        self.is_alive.store(false, Ordering::Relaxed);\n        self.receive_channel\n            .write()\n            .expect(\"This lock should never be poisoned\")\n            .take();\n    }\n}\n\nimpl<D: Document> From<AddBatchReceiver<D>> for IndexWriterStatus<D> {\n    fn from(receiver: AddBatchReceiver<D>) -> Self {\n        IndexWriterStatus {\n            inner: Arc::new(Inner {\n                is_alive: AtomicBool::new(true),\n                receive_channel: RwLock::new(Some(receiver)),\n            }),\n        }\n    }\n}\n\n/// If dropped, the index writer will be killed.\n/// To prevent this, clients can call `.defuse()`.\npub(crate) struct IndexWriterBomb<D: Document> {\n    inner: Option<Arc<Inner<D>>>,\n}\n\nimpl<D: Document> IndexWriterBomb<D> {\n    /// Defuses the bomb.\n    ///\n    /// This is the only way to drop the bomb without killing\n    /// the index writer.\n    pub fn defuse(mut self) {\n        self.inner = None;\n    }\n}\n\nimpl<D: Document> Drop for IndexWriterBomb<D> {\n    fn drop(&mut self) {\n        if let Some(inner) = self.inner.take() {\n            inner.kill();\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::mem;\n\n    use crossbeam_channel as channel;\n\n    use super::IndexWriterStatus;\n\n    #[test]\n    fn test_bomb_goes_boom() {\n        let (_tx, rx) = channel::bounded(10);\n        let index_writer_status: IndexWriterStatus = IndexWriterStatus::from(rx);\n        assert!(index_writer_status.operation_receiver().is_some());\n        let bomb = index_writer_status.create_bomb();\n        assert!(index_writer_status.operation_receiver().is_some());\n        mem::drop(bomb);\n        // boom!\n        assert!(index_writer_status.operation_receiver().is_none());\n    }\n\n    #[test]\n    fn test_bomb_defused() {\n        let (_tx, rx) = channel::bounded(10);\n        let index_writer_status: IndexWriterStatus = IndexWriterStatus::from(rx);\n        assert!(index_writer_status.operation_receiver().is_some());\n        let bomb = index_writer_status.create_bomb();\n        bomb.defuse();\n        assert!(index_writer_status.operation_receiver().is_some());\n    }\n}\n"
  },
  {
    "path": "src/indexer/indexing_term.rs",
    "content": "use std::net::Ipv6Addr;\n\nuse columnar::MonotonicallyMappableToU128;\n\nuse crate::fastfield::FastValue;\nuse crate::schema::Field;\n\n/// IndexingTerm is used to represent a term during indexing.\n/// It's a serialized representation over field and value.\n///\n/// It actually wraps a `Vec<u8>`. The first 4 bytes are the field.\n///\n/// We serialize the field, because we index everything in a single\n/// global term dictionary during indexing.\n#[derive(Clone)]\npub(crate) struct IndexingTerm<B = Vec<u8>>(B)\nwhere B: AsRef<[u8]>;\n\n/// The number of bytes used as metadata by `Term`.\nconst TERM_METADATA_LENGTH: usize = 4;\n\nimpl IndexingTerm {\n    /// Create a new Term with a buffer with a given capacity.\n    pub fn with_capacity(capacity: usize) -> IndexingTerm {\n        let mut data = Vec::with_capacity(TERM_METADATA_LENGTH + capacity);\n        data.resize(TERM_METADATA_LENGTH, 0u8);\n        IndexingTerm(data)\n    }\n\n    /// Panics when the term is not empty... ie: some value is set.\n    /// Use `clear_with_field_and_type` in that case.\n    ///\n    /// Sets field and the type.\n    pub(crate) fn set_field(&mut self, field: Field) {\n        assert!(self.is_empty());\n        self.0[0..4].clone_from_slice(field.field_id().to_be_bytes().as_ref());\n    }\n\n    /// Is empty if there are no value bytes.\n    pub fn is_empty(&self) -> bool {\n        self.0.len() == TERM_METADATA_LENGTH\n    }\n\n    /// Removes the value_bytes and set the field\n    pub(crate) fn clear_with_field(&mut self, field: Field) {\n        self.truncate_value_bytes(0);\n        self.set_field(field);\n    }\n\n    /// Sets a u64 value in the term.\n    ///\n    /// U64 are serialized using (8-byte) BigEndian\n    /// representation.\n    /// The use of BigEndian has the benefit of preserving\n    /// the natural order of the values.\n    pub fn set_u64(&mut self, val: u64) {\n        self.set_fast_value(val);\n    }\n\n    /// Sets a `i64` value in the term.\n    pub fn set_i64(&mut self, val: i64) {\n        self.set_fast_value(val);\n    }\n\n    /// Sets a `f64` value in the term.\n    pub fn set_f64(&mut self, val: f64) {\n        self.set_fast_value(val);\n    }\n\n    /// Sets a `bool` value in the term.\n    pub fn set_bool(&mut self, val: bool) {\n        self.set_fast_value(val);\n    }\n\n    fn set_fast_value<T: FastValue>(&mut self, val: T) {\n        self.set_bytes(val.to_u64().to_be_bytes().as_ref());\n    }\n\n    /// Append a type marker + fast value to a term.\n    /// This is used in JSON type to append a fast value after the path.\n    ///\n    /// It will not clear existing bytes.\n    pub fn append_type_and_fast_value<T: FastValue>(&mut self, val: T) {\n        self.0.push(T::to_type().to_code());\n        let value = val.to_u64();\n        self.0.extend(value.to_be_bytes().as_ref());\n    }\n\n    /// Sets a `Ipv6Addr` value in the term.\n    pub fn set_ip_addr(&mut self, val: Ipv6Addr) {\n        self.set_bytes(val.to_u128().to_be_bytes().as_ref());\n    }\n\n    /// Sets the value of a `Bytes` field.\n    pub fn set_bytes(&mut self, bytes: &[u8]) {\n        self.truncate_value_bytes(0);\n        self.0.extend(bytes);\n    }\n\n    /// Truncates the value bytes of the term. Value and field type stays the same.\n    pub fn truncate_value_bytes(&mut self, len: usize) {\n        self.0.truncate(len + TERM_METADATA_LENGTH);\n    }\n\n    /// The length of the bytes.\n    pub fn len_bytes(&self) -> usize {\n        self.0.len() - TERM_METADATA_LENGTH\n    }\n\n    /// Appends value bytes to the Term.\n    ///\n    /// This function returns the segment that has just been added.\n    #[inline]\n    pub fn append_bytes(&mut self, bytes: &[u8]) -> &mut [u8] {\n        let len_before = self.0.len();\n        self.0.extend_from_slice(bytes);\n        &mut self.0[len_before..]\n    }\n}\n\nimpl<B> IndexingTerm<B>\nwhere B: AsRef<[u8]>\n{\n    /// Wraps serialized term bytes.\n    ///\n    /// The input buffer is expected to be the concatenation of the big endian encoded field id\n    /// followed by the serialized value bytes (type tag + payload).\n    #[inline]\n    pub fn wrap(serialized_term: B) -> IndexingTerm<B> {\n        debug_assert!(serialized_term.as_ref().len() >= TERM_METADATA_LENGTH);\n        IndexingTerm(serialized_term)\n    }\n\n    /// Returns the field this term belongs to.\n    #[inline]\n    pub fn field(&self) -> Field {\n        let field_id_bytes: [u8; 4] = self.0.as_ref()[..4].try_into().unwrap();\n        Field::from_field_id(u32::from_be_bytes(field_id_bytes))\n    }\n\n    /// Returns the serialized representation of Term.\n    /// This includes field_id, value type and value.\n    ///\n    /// Do NOT rely on this byte representation in the index.\n    /// This value is likely to change in the future.\n    #[inline]\n    pub fn serialized_term(&self) -> &[u8] {\n        self.0.as_ref()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::IndexingTerm;\n    use crate::schema::*;\n\n    #[test]\n    pub fn test_term_str() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"text\", STRING);\n        let title_field = schema_builder.add_text_field(\"title\", STRING);\n        let mut term = IndexingTerm::with_capacity(0);\n        term.set_field(title_field);\n        term.set_bytes(b\"test\");\n        assert_eq!(term.field(), title_field);\n        assert_eq!(term.serialized_term(), b\"\\x00\\x00\\x00\\x01test\".to_vec())\n    }\n\n    /// Size (in bytes) of the buffer of a fast value (u64, i64, f64, or date) term.\n    /// <field> + <type byte> + <value len>\n    ///\n    /// - <field> is a big endian encoded u32 field id\n    /// - <value> is,  if this is not the json term, a binary representation specific to the type.\n    ///   If it is a JSON Term, then it is prepended with the path that leads to this leaf value.\n    const FAST_VALUE_TERM_LEN: usize = 4 + 8;\n\n    #[test]\n    pub fn test_term_u64() {\n        let mut schema_builder = Schema::builder();\n        let count_field = schema_builder.add_u64_field(\"count\", INDEXED);\n        let mut term = IndexingTerm::with_capacity(0);\n        term.set_field(count_field);\n        term.set_u64(983u64);\n        assert_eq!(term.field(), count_field);\n        assert_eq!(term.serialized_term().len(), FAST_VALUE_TERM_LEN);\n    }\n\n    #[test]\n    pub fn test_term_bool() {\n        let mut schema_builder = Schema::builder();\n        let bool_field = schema_builder.add_bool_field(\"bool\", INDEXED);\n        let term = {\n            let mut term = IndexingTerm::with_capacity(0);\n            term.set_field(bool_field);\n            term.set_bool(true);\n            term\n        };\n        assert_eq!(term.field(), bool_field);\n        assert_eq!(term.serialized_term().len(), FAST_VALUE_TERM_LEN);\n    }\n\n    #[test]\n    pub fn indexing_term_wrap_extracts_field() {\n        let field = Field::from_field_id(7u32);\n        let mut term = IndexingTerm::with_capacity(0);\n        term.set_field(field);\n        term.append_bytes(b\"abc\");\n\n        let wrapped = IndexingTerm::wrap(term.serialized_term());\n        assert_eq!(wrapped.field(), field);\n        assert_eq!(wrapped.serialized_term(), term.serialized_term());\n    }\n}\n"
  },
  {
    "path": "src/indexer/log_merge_policy.rs",
    "content": "use std::cmp;\n\nuse itertools::Itertools;\n\nuse super::merge_policy::{MergeCandidate, MergePolicy};\nuse crate::index::SegmentMeta;\n\nconst DEFAULT_LEVEL_LOG_SIZE: f64 = 0.75;\nconst DEFAULT_MIN_LAYER_SIZE: u32 = 10_000;\nconst DEFAULT_MIN_NUM_SEGMENTS_IN_MERGE: usize = 8;\nconst DEFAULT_MAX_DOCS_BEFORE_MERGE: usize = 10_000_000;\n// The default value of 1 means that deletes are not taken in account when\n// identifying merge candidates. This is not a very sensible default: it was\n// set like that for backward compatibility and might change in the near future.\nconst DEFAULT_DEL_DOCS_RATIO_BEFORE_MERGE: f32 = 1.0f32;\n\n/// `LogMergePolicy` tries to merge segments that have a similar number of\n/// documents.\n#[derive(Debug, Clone)]\npub struct LogMergePolicy {\n    min_num_segments: usize,\n    max_docs_before_merge: usize,\n    min_layer_size: u32,\n    level_log_size: f64,\n    del_docs_ratio_before_merge: f32,\n}\n\nimpl LogMergePolicy {\n    fn clip_min_size(&self, size: u32) -> u32 {\n        cmp::max(self.min_layer_size, size)\n    }\n\n    /// Set the minimum number of segments that may be merged together.\n    pub fn set_min_num_segments(&mut self, min_num_segments: usize) {\n        self.min_num_segments = min_num_segments;\n    }\n\n    /// Set the maximum number docs in a segment for it to be considered for\n    /// merging. A segment can still reach more than max_docs, by merging many\n    /// smaller ones.\n    pub fn set_max_docs_before_merge(&mut self, max_docs_merge_size: usize) {\n        self.max_docs_before_merge = max_docs_merge_size;\n    }\n\n    /// Set the minimum segment size under which all segment belong\n    /// to the same level.\n    pub fn set_min_layer_size(&mut self, min_layer_size: u32) {\n        self.min_layer_size = min_layer_size;\n    }\n\n    /// Set the ratio between two consecutive levels.\n    ///\n    /// Segments are grouped in levels according to their sizes.\n    /// These levels are defined as intervals of exponentially growing sizes.\n    /// level_log_size define the factor by which one should multiply the limit\n    /// to reach a level, in order to get the limit to reach the following\n    /// level.\n    pub fn set_level_log_size(&mut self, level_log_size: f64) {\n        self.level_log_size = level_log_size;\n    }\n\n    /// Set the ratio of deleted documents in a segment to tolerate.\n    ///\n    /// If it is exceeded by any segment at a log level, a merge\n    /// will be triggered for that level.\n    ///\n    /// If there is a single segment at a level, we effectively end up expunging\n    /// deleted documents from it.\n    ///\n    /// # Panics\n    ///\n    /// Panics if del_docs_ratio_before_merge is not within (0..1].\n    pub fn set_del_docs_ratio_before_merge(&mut self, del_docs_ratio_before_merge: f32) {\n        assert!(del_docs_ratio_before_merge <= 1.0f32);\n        assert!(del_docs_ratio_before_merge > 0f32);\n        self.del_docs_ratio_before_merge = del_docs_ratio_before_merge;\n    }\n\n    fn has_segment_above_deletes_threshold(&self, level: &[&SegmentMeta]) -> bool {\n        level\n            .iter()\n            .any(|segment| deletes_ratio(segment) > self.del_docs_ratio_before_merge)\n    }\n}\n\nfn deletes_ratio(segment: &SegmentMeta) -> f32 {\n    if segment.max_doc() == 0 {\n        return 0f32;\n    }\n    segment.num_deleted_docs() as f32 / segment.max_doc() as f32\n}\n\nimpl MergePolicy for LogMergePolicy {\n    fn compute_merge_candidates(&self, segments: &[SegmentMeta]) -> Vec<MergeCandidate> {\n        let size_sorted_segments = segments\n            .iter()\n            .filter(|seg| (seg.num_docs() as usize) <= self.max_docs_before_merge)\n            .sorted_by_key(|seg| std::cmp::Reverse(seg.max_doc()))\n            .collect::<Vec<&SegmentMeta>>();\n\n        if size_sorted_segments.is_empty() {\n            return vec![];\n        }\n\n        let mut current_max_log_size = f64::MAX;\n        let mut levels = vec![];\n        for (_, merge_group) in &size_sorted_segments.into_iter().chunk_by(|segment| {\n            let segment_log_size = f64::from(self.clip_min_size(segment.num_docs())).log2();\n            if segment_log_size < (current_max_log_size - self.level_log_size) {\n                // update current_max_log_size to create a new group\n                current_max_log_size = segment_log_size;\n            }\n            // return current_max_log_size to be grouped to the current group\n            current_max_log_size\n        }) {\n            levels.push(merge_group.collect::<Vec<&SegmentMeta>>());\n        }\n\n        levels\n            .iter()\n            .filter(|level| {\n                level.len() >= self.min_num_segments\n                    || self.has_segment_above_deletes_threshold(level)\n            })\n            .map(|segments| MergeCandidate(segments.iter().map(|&seg| seg.id()).collect()))\n            .collect()\n    }\n}\n\nimpl Default for LogMergePolicy {\n    fn default() -> LogMergePolicy {\n        LogMergePolicy {\n            min_num_segments: DEFAULT_MIN_NUM_SEGMENTS_IN_MERGE,\n            max_docs_before_merge: DEFAULT_MAX_DOCS_BEFORE_MERGE,\n            min_layer_size: DEFAULT_MIN_LAYER_SIZE,\n            level_log_size: DEFAULT_LEVEL_LOG_SIZE,\n            del_docs_ratio_before_merge: DEFAULT_DEL_DOCS_RATIO_BEFORE_MERGE,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use once_cell::sync::Lazy;\n\n    use super::*;\n    use crate::index::{SegmentId, SegmentMetaInventory};\n    use crate::schema;\n    use crate::schema::INDEXED;\n\n    static INVENTORY: Lazy<SegmentMetaInventory> = Lazy::new(SegmentMetaInventory::default);\n\n    use crate::Index;\n\n    #[test]\n    fn create_index_test_max_merge_issue_1035() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let int_field = schema_builder.add_u64_field(\"intval\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut log_merge_policy = LogMergePolicy::default();\n            log_merge_policy.set_min_num_segments(1);\n            log_merge_policy.set_max_docs_before_merge(1);\n            log_merge_policy.set_min_layer_size(0);\n\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.set_merge_policy(Box::new(log_merge_policy));\n\n            // after every commit the merge checker is started, it will merge only segments with 1\n            // element in it because of the max_docs_before_merge.\n            index_writer.add_document(doc!(int_field=>1_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>2_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>3_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>4_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>5_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>6_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>7_u64))?;\n            index_writer.commit()?;\n\n            index_writer.add_document(doc!(int_field=>8_u64))?;\n            index_writer.commit()?;\n        }\n\n        let _segment_ids = index\n            .searchable_segment_ids()\n            .expect(\"Searchable segments failed.\");\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_readers = searcher.segment_readers();\n        for segment in segment_readers {\n            if segment.num_docs() > 2 {\n                panic!(\"segment can't have more than two segments\");\n            } // don't know how to wait for the merge, then it could be a simple eq\n        }\n        Ok(())\n    }\n\n    fn test_merge_policy() -> LogMergePolicy {\n        let mut log_merge_policy = LogMergePolicy::default();\n        log_merge_policy.set_min_num_segments(3);\n        log_merge_policy.set_max_docs_before_merge(100_000);\n        log_merge_policy.set_min_layer_size(2);\n        log_merge_policy\n    }\n\n    #[test]\n    fn test_log_merge_policy_empty() {\n        let y = Vec::new();\n        let result_list = test_merge_policy().compute_merge_candidates(&y);\n        assert!(result_list.is_empty());\n    }\n\n    fn create_random_segment_meta(num_docs: u32) -> SegmentMeta {\n        INVENTORY.new_segment_meta(SegmentId::generate_random(), num_docs)\n    }\n\n    #[test]\n    fn test_log_merge_policy_pair() {\n        let test_input = vec![\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n        ];\n        let result_list = test_merge_policy().compute_merge_candidates(&test_input);\n        assert_eq!(result_list.len(), 1);\n    }\n\n    #[test]\n    fn test_log_merge_policy_levels() {\n        // multiple levels all get merged correctly\n        // 2 MergeCandidates expected:\n        // * one with the 6 * 10-docs segments\n        // * one with the 3 * 1000-docs segments\n        // no MergeCandidate expected for the 2 * 10_000-docs segments as min_merge_size=3\n        let test_input = vec![\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n            create_random_segment_meta(1_000),\n            create_random_segment_meta(1_000),\n            create_random_segment_meta(1_000),\n            create_random_segment_meta(10_000),\n            create_random_segment_meta(10_000),\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n            create_random_segment_meta(10),\n        ];\n        let result_list = test_merge_policy().compute_merge_candidates(&test_input);\n        assert_eq!(result_list.len(), 2);\n    }\n\n    #[test]\n    fn test_log_merge_policy_within_levels() {\n        // multiple levels all get merged correctly\n        let test_input = vec![\n            create_random_segment_meta(10),   // log2(10) = ~3.32 (> 3.58 - 0.75)\n            create_random_segment_meta(11),   // log2(11) = ~3.46\n            create_random_segment_meta(12),   // log2(12) = ~3.58\n            create_random_segment_meta(800),  // log2(800) = ~9.64 (> 9.97 - 0.75)\n            create_random_segment_meta(1000), // log2(1000) = ~9.97\n            create_random_segment_meta(1000),\n        ]; // log2(1000) = ~9.97\n        let result_list = test_merge_policy().compute_merge_candidates(&test_input);\n        assert_eq!(result_list.len(), 2);\n    }\n\n    #[test]\n    fn test_log_merge_policy_small_segments() {\n        // segments under min_layer_size are merged together\n        let test_input = vec![\n            create_random_segment_meta(1),\n            create_random_segment_meta(1),\n            create_random_segment_meta(1),\n            create_random_segment_meta(2),\n            create_random_segment_meta(2),\n            create_random_segment_meta(2),\n        ];\n        let result_list = test_merge_policy().compute_merge_candidates(&test_input);\n        assert_eq!(result_list.len(), 1);\n    }\n\n    #[test]\n    fn test_log_merge_policy_all_segments_too_large_to_merge() {\n        let eight_large_segments: Vec<SegmentMeta> =\n            std::iter::repeat_with(|| create_random_segment_meta(100_001))\n                .take(8)\n                .collect();\n        assert!(test_merge_policy()\n            .compute_merge_candidates(&eight_large_segments)\n            .is_empty());\n    }\n\n    #[test]\n    fn test_large_merge_segments() {\n        let test_input = vec![\n            create_random_segment_meta(1_000_000),\n            create_random_segment_meta(100_001),\n            create_random_segment_meta(100_000),\n            create_random_segment_meta(1_000_001),\n            create_random_segment_meta(100_000),\n            create_random_segment_meta(100_000),\n            create_random_segment_meta(1_500_000),\n        ];\n        let result_list = test_merge_policy().compute_merge_candidates(&test_input);\n        // Do not include large segments\n        assert_eq!(result_list.len(), 1);\n        assert_eq!(result_list[0].0.len(), 3);\n\n        // Making sure merge policy points to the correct index of the original input\n        assert_eq!(result_list[0].0[0], test_input[2].id());\n        assert_eq!(result_list[0].0[1], test_input[4].id());\n        assert_eq!(result_list[0].0[2], test_input[5].id());\n    }\n\n    #[test]\n    fn test_merge_single_segment_with_deletes_below_threshold() {\n        let mut test_merge_policy = test_merge_policy();\n        test_merge_policy.set_del_docs_ratio_before_merge(0.25f32);\n        let test_input = vec![create_random_segment_meta(40_000).with_delete_meta(10_000, 1)];\n        let merge_candidates = test_merge_policy.compute_merge_candidates(&test_input);\n        assert!(merge_candidates.is_empty());\n    }\n\n    #[test]\n    fn test_merge_single_segment_with_deletes_above_threshold() {\n        let mut test_merge_policy = test_merge_policy();\n        test_merge_policy.set_del_docs_ratio_before_merge(0.25f32);\n        let test_input = vec![create_random_segment_meta(40_000).with_delete_meta(10_001, 1)];\n        let merge_candidates = test_merge_policy.compute_merge_candidates(&test_input);\n        assert_eq!(merge_candidates.len(), 1);\n    }\n\n    #[test]\n    fn test_merge_segments_with_deletes_above_threshold_all_in_level() {\n        let mut test_merge_policy = test_merge_policy();\n        test_merge_policy.set_del_docs_ratio_before_merge(0.25f32);\n        let test_input = vec![\n            create_random_segment_meta(40_000).with_delete_meta(10_001, 1),\n            create_random_segment_meta(40_000),\n        ];\n        let merge_candidates = test_merge_policy.compute_merge_candidates(&test_input);\n        assert_eq!(merge_candidates.len(), 1);\n        assert_eq!(merge_candidates[0].0.len(), 2);\n    }\n\n    #[test]\n    fn test_merge_segments_with_deletes_above_threshold_different_level_not_involved() {\n        let mut test_merge_policy = test_merge_policy();\n        test_merge_policy.set_del_docs_ratio_before_merge(0.25f32);\n        let test_input = vec![\n            create_random_segment_meta(100),\n            create_random_segment_meta(40_000).with_delete_meta(10_001, 1),\n        ];\n        let merge_candidates = test_merge_policy.compute_merge_candidates(&test_input);\n        assert_eq!(merge_candidates.len(), 1);\n        assert_eq!(merge_candidates[0].0.len(), 1);\n        assert_eq!(merge_candidates[0].0[0], test_input[1].id());\n    }\n\n    #[test]\n    fn test_max_docs_before_merge_large_value() {\n        // Regression test: (max_docs_before_merge as u32) truncates values > u32::MAX.\n        // Casting num_docs() to usize instead avoids the truncation.\n        let mut policy = LogMergePolicy::default();\n        policy.set_min_num_segments(2);\n        policy.set_max_docs_before_merge(5_000_000_000usize);\n        let test_input = vec![\n            create_random_segment_meta(100_000),\n            create_random_segment_meta(100_000),\n        ];\n        let result = policy.compute_merge_candidates(&test_input);\n        // Both segments should be eligible (100_000 < 5_000_000_000)\n        assert_eq!(result.len(), 1);\n        assert_eq!(result[0].0.len(), 2);\n    }\n}\n"
  },
  {
    "path": "src/indexer/merge_index_test.rs",
    "content": "#[cfg(test)]\nmod tests {\n    use crate::collector::TopDocs;\n    use crate::fastfield::AliveBitSet;\n    use crate::index::Index;\n    use crate::postings::Postings;\n    use crate::query::QueryParser;\n    use crate::schema::{\n        self, BytesOptions, Facet, FacetOptions, IndexRecordOption, NumericOptions,\n        TextFieldIndexing, TextOptions,\n    };\n    use crate::{DocAddress, DocSet, IndexSettings, IndexWriter, Term};\n\n    fn create_test_index(index_settings: Option<IndexSettings>) -> crate::Result<Index> {\n        let mut schema_builder = schema::Schema::builder();\n        let int_options = NumericOptions::default()\n            .set_fast()\n            .set_stored()\n            .set_indexed();\n        let int_field = schema_builder.add_u64_field(\"intval\", int_options);\n\n        let bytes_options = BytesOptions::default().set_fast().set_indexed();\n        let bytes_field = schema_builder.add_bytes_field(\"bytes\", bytes_options);\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n\n        let multi_numbers =\n            schema_builder.add_u64_field(\"multi_numbers\", NumericOptions::default().set_fast());\n        let text_field_options = TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_index_option(schema::IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text_field\", text_field_options);\n        let schema = schema_builder.build();\n\n        let mut index_builder = Index::builder().schema(schema);\n        if let Some(settings) = index_settings {\n            index_builder = index_builder.settings(settings);\n        }\n        let index = index_builder.create_in_ram()?;\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n\n            // segment 1 - range 1-3\n            index_writer.add_document(doc!(int_field=>1_u64))?;\n            index_writer.add_document(\n                doc!(int_field=>3_u64, multi_numbers => 3_u64, multi_numbers => 4_u64, bytes_field => vec![1, 2, 3], text_field => \"some text\", facet_field=> Facet::from(\"/book/crime\")),\n            )?;\n            index_writer.add_document(\n                doc!(int_field=>1_u64, text_field=> \"deleteme\",  text_field => \"ok text more text\"),\n            )?;\n            index_writer.add_document(\n                doc!(int_field=>2_u64, multi_numbers => 2_u64, multi_numbers => 3_u64, text_field => \"ok text more text\"),\n            )?;\n\n            index_writer.commit()?;\n            index_writer.add_document(doc!(int_field=>20_u64, multi_numbers => 20_u64))?;\n\n            let in_val = 1u64;\n            index_writer.add_document(doc!(int_field=>in_val, text_field=> \"deleteme\" , text_field => \"ok text more text\", facet_field=> Facet::from(\"/book/crime\")))?;\n            index_writer.commit()?;\n            let int_vals = [10u64, 5];\n            index_writer.add_document( // position of this doc after delete in desc sorting = [2], in disjunct case [1]\n                doc!(int_field=>int_vals[0], multi_numbers => 10_u64, multi_numbers => 11_u64, text_field=> \"blubber\", facet_field=> Facet::from(\"/book/fantasy\")),\n            )?;\n            index_writer.add_document(doc!(int_field=>int_vals[1], text_field=> \"deleteme\"))?;\n            index_writer.add_document(\n                doc!(int_field=>1_000u64, multi_numbers => 1001_u64, multi_numbers => 1002_u64, bytes_field => vec![5, 5],text_field => \"the biggest num\")\n            )?;\n\n            index_writer.delete_term(Term::from_field_text(text_field, \"deleteme\"));\n            index_writer.commit()?;\n        }\n\n        // Merging the segments\n        {\n            let segment_ids = index.searchable_segment_ids()?;\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n        Ok(index)\n    }\n\n    #[test]\n    fn test_merge_index() {\n        let index = create_test_index(Some(IndexSettings {\n            ..Default::default()\n        }))\n        .unwrap();\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let segment_reader = searcher.segment_readers().last().unwrap();\n\n        let searcher = index.reader().unwrap().searcher();\n        {\n            let my_text_field = index.schema().get_field(\"text_field\").unwrap();\n\n            let do_search = |term: &str| {\n                let query = QueryParser::for_index(&index, vec![my_text_field])\n                    .parse_query(term)\n                    .unwrap();\n                let top_docs: Vec<(f32, DocAddress)> = searcher\n                    .search(&query, &TopDocs::with_limit(3).order_by_score())\n                    .unwrap();\n\n                top_docs.iter().map(|el| el.1.doc_id).collect::<Vec<_>>()\n            };\n\n            assert_eq!(do_search(\"some\"), vec![1]);\n            assert_eq!(do_search(\"blubber\"), vec![3]);\n            assert_eq!(do_search(\"biggest\"), vec![4]);\n        }\n\n        // postings file\n        {\n            let my_text_field = index.schema().get_field(\"text_field\").unwrap();\n            let term_a = Term::from_field_text(my_text_field, \"text\");\n            let inverted_index = segment_reader.inverted_index(my_text_field).unwrap();\n            let mut postings = inverted_index\n                .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)\n                .unwrap()\n                .unwrap();\n            assert_eq!(postings.doc_freq(), 2);\n            let fallback_bitset = AliveBitSet::for_test_from_deleted_docs(&[0], 100);\n            assert_eq!(\n                postings.doc_freq_given_deletes(\n                    segment_reader.alive_bitset().unwrap_or(&fallback_bitset)\n                ),\n                2\n            );\n\n            assert_eq!(postings.term_freq(), 1);\n            let mut output = vec![];\n            postings.positions(&mut output);\n            assert_eq!(output, vec![1]);\n            postings.advance();\n\n            assert_eq!(postings.term_freq(), 2);\n            postings.positions(&mut output);\n            assert_eq!(output, vec![1, 3]);\n        }\n    }\n}\n"
  },
  {
    "path": "src/indexer/merge_operation.rs",
    "content": "use std::collections::HashSet;\nuse std::ops::Deref;\n\nuse crate::index::SegmentId;\nuse crate::{Inventory, Opstamp, TrackedObject};\n\n#[derive(Default)]\npub(crate) struct MergeOperationInventory(Inventory<InnerMergeOperation>);\n\nimpl Deref for MergeOperationInventory {\n    type Target = Inventory<InnerMergeOperation>;\n\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n\nimpl MergeOperationInventory {\n    pub fn segment_in_merge(&self) -> HashSet<SegmentId> {\n        let mut segment_in_merge = HashSet::default();\n        for merge_op in self.list() {\n            for &segment_id in &merge_op.segment_ids {\n                segment_in_merge.insert(segment_id);\n            }\n        }\n        segment_in_merge\n    }\n}\n\n/// A `MergeOperation` has two roles.\n/// It carries all of the information required to describe a merge:\n/// - `target_opstamp` is the opstamp up to which we want to consume the delete queue and reflect\n///   their deletes.\n/// - `segment_ids` is the list of segment to be merged.\n///\n/// The second role is to ensure keep track of the fact that these\n/// segments are in merge and avoid starting a merge operation that\n/// may conflict with this one.\n///\n/// This works by tracking merge operations. When considering computing\n/// merge candidates, we simply list tracked merge operations and remove\n/// their segments from possible merge candidates.\npub struct MergeOperation {\n    inner: TrackedObject<InnerMergeOperation>,\n}\n\npub(crate) struct InnerMergeOperation {\n    target_opstamp: Opstamp,\n    segment_ids: Vec<SegmentId>,\n}\n\nimpl MergeOperation {\n    pub(crate) fn new(\n        inventory: &MergeOperationInventory,\n        target_opstamp: Opstamp,\n        segment_ids: Vec<SegmentId>,\n    ) -> MergeOperation {\n        let inner_merge_operation = InnerMergeOperation {\n            target_opstamp,\n            segment_ids,\n        };\n        MergeOperation {\n            inner: inventory.track(inner_merge_operation),\n        }\n    }\n\n    /// Returns the opstamp up to which we want to consume the delete queue and reflect their\n    /// deletes.\n    pub fn target_opstamp(&self) -> Opstamp {\n        self.inner.target_opstamp\n    }\n\n    /// Returns the list of segment to be merged.\n    pub fn segment_ids(&self) -> &[SegmentId] {\n        &self.inner.segment_ids[..]\n    }\n}\n"
  },
  {
    "path": "src/indexer/merge_policy.rs",
    "content": "use std::fmt::Debug;\nuse std::marker;\n\nuse crate::index::{SegmentId, SegmentMeta};\n\n/// Set of segment suggested for a merge.\n#[derive(Debug, Clone)]\npub struct MergeCandidate(pub Vec<SegmentId>);\n\n/// The `MergePolicy` defines which segments should be merged.\n///\n/// Every time the list of segments changes, the segment updater\n/// asks the merge policy if some segments should be merged.\npub trait MergePolicy: marker::Send + marker::Sync + Debug {\n    /// Given the list of segment metas, returns the list of merge candidates.\n    ///\n    /// This call happens on the segment updater thread, and will block\n    /// other segment updates, so all implementations should happen rapidly.\n    fn compute_merge_candidates(&self, segments: &[SegmentMeta]) -> Vec<MergeCandidate>;\n}\n\n/// Never merge segments.\n#[derive(Debug, Clone)]\npub struct NoMergePolicy;\n\nimpl Default for NoMergePolicy {\n    fn default() -> NoMergePolicy {\n        NoMergePolicy\n    }\n}\n\nimpl MergePolicy for NoMergePolicy {\n    fn compute_merge_candidates(&self, _segments: &[SegmentMeta]) -> Vec<MergeCandidate> {\n        Vec::new()\n    }\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use super::*;\n\n    /// `MergePolicy` useful for test purposes.\n    ///\n    /// Every time there is more than one segment,\n    /// it will suggest to merge them.\n    #[derive(Debug, Clone)]\n    pub struct MergeWheneverPossible;\n\n    impl MergePolicy for MergeWheneverPossible {\n        fn compute_merge_candidates(&self, segment_metas: &[SegmentMeta]) -> Vec<MergeCandidate> {\n            let segment_ids = segment_metas\n                .iter()\n                .map(|segment_meta| segment_meta.id())\n                .collect::<Vec<SegmentId>>();\n            if segment_ids.len() > 1 {\n                vec![MergeCandidate(segment_ids)]\n            } else {\n                vec![]\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "src/indexer/merger.rs",
    "content": "use std::sync::Arc;\n\nuse columnar::{\n    ColumnType, ColumnarReader, MergeRowOrder, RowAddr, ShuffleMergeOrder, StackMergeOrder,\n};\nuse common::ReadOnlyBitSet;\nuse itertools::Itertools;\nuse measure_time::debug_time;\n\nuse crate::directory::WritePtr;\nuse crate::docset::{DocSet, TERMINATED};\nuse crate::error::DataCorruption;\nuse crate::fastfield::AliveBitSet;\nuse crate::fieldnorm::{FieldNormReader, FieldNormReaders, FieldNormsSerializer, FieldNormsWriter};\nuse crate::index::{Segment, SegmentComponent, SegmentReader};\nuse crate::indexer::doc_id_mapping::{MappingType, SegmentDocIdMapping};\nuse crate::indexer::SegmentSerializer;\nuse crate::postings::{InvertedIndexSerializer, Postings, SegmentPostings};\nuse crate::schema::{value_type_to_column_type, Field, FieldType, Schema};\nuse crate::store::StoreWriter;\nuse crate::termdict::{TermMerger, TermOrdinal};\nuse crate::{DocAddress, DocId, InvertedIndexReader};\n\n/// Segment's max doc must be `< MAX_DOC_LIMIT`.\n///\n/// We do not allow segments with more than\npub const MAX_DOC_LIMIT: u32 = 1 << 31;\n\nfn estimate_total_num_tokens_in_single_segment(\n    reader: &SegmentReader,\n    field: Field,\n) -> crate::Result<u64> {\n    // There are no deletes. We can simply use the exact value saved into the posting list.\n    // Note that this value is not necessarily exact as it could have been the result of a merge\n    // between segments themselves containing deletes.\n    if !reader.has_deletes() {\n        return Ok(reader.inverted_index(field)?.total_num_tokens());\n    }\n\n    // When there are deletes, we use an approximation either\n    // by using the fieldnorm.\n    if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {\n        let mut count: [usize; 256] = [0; 256];\n        for doc in reader.doc_ids_alive() {\n            let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);\n            count[fieldnorm_id as usize] += 1;\n        }\n        let total_num_tokens = count\n            .iter()\n            .cloned()\n            .enumerate()\n            .map(|(fieldnorm_ord, count)| {\n                count as u64 * u64::from(FieldNormReader::id_to_fieldnorm(fieldnorm_ord as u8))\n            })\n            .sum::<u64>();\n        return Ok(total_num_tokens);\n    }\n\n    // There are no fieldnorms available.\n    // Here we just do a pro-rata with the overall number of tokens an the ratio of\n    // documents alive.\n    let segment_num_tokens = reader.inverted_index(field)?.total_num_tokens();\n    if reader.max_doc() == 0 {\n        // That supposedly never happens, but let's be a bit defensive here.\n        return Ok(0u64);\n    }\n    let ratio = reader.num_docs() as f64 / reader.max_doc() as f64;\n    Ok((segment_num_tokens as f64 * ratio) as u64)\n}\n\nfn estimate_total_num_tokens(readers: &[SegmentReader], field: Field) -> crate::Result<u64> {\n    let mut total_num_tokens: u64 = 0;\n    for reader in readers {\n        total_num_tokens += estimate_total_num_tokens_in_single_segment(reader, field)?;\n    }\n    Ok(total_num_tokens)\n}\n\npub struct IndexMerger {\n    schema: Schema,\n    pub(crate) readers: Vec<SegmentReader>,\n    max_doc: u32,\n}\n\nstruct DeltaComputer {\n    buffer: Vec<u32>,\n}\n\nimpl DeltaComputer {\n    fn new() -> DeltaComputer {\n        DeltaComputer {\n            buffer: vec![0u32; 512],\n        }\n    }\n\n    fn compute_delta(&mut self, positions: &[u32]) -> &[u32] {\n        if positions.len() > self.buffer.len() {\n            self.buffer.resize(positions.len(), 0u32);\n        }\n        let mut last_pos = 0u32;\n        for (cur_pos, dest) in positions.iter().cloned().zip(self.buffer.iter_mut()) {\n            *dest = cur_pos - last_pos;\n            last_pos = cur_pos;\n        }\n        &self.buffer[..positions.len()]\n    }\n}\n\nfn convert_to_merge_order(\n    columnars: &[&ColumnarReader],\n    doc_id_mapping: SegmentDocIdMapping,\n) -> MergeRowOrder {\n    match doc_id_mapping.mapping_type() {\n        MappingType::Stacked => MergeRowOrder::Stack(StackMergeOrder::stack(columnars)),\n        MappingType::StackedWithDeletes => {\n            // RUST/LLVM is amazing. The following conversion is actually a no-op:\n            // no allocation, no copy.\n            let new_row_id_to_old_row_id: Vec<RowAddr> = doc_id_mapping\n                .new_doc_id_to_old_doc_addr\n                .into_iter()\n                .map(|doc_addr| RowAddr {\n                    segment_ord: doc_addr.segment_ord,\n                    row_id: doc_addr.doc_id,\n                })\n                .collect();\n            MergeRowOrder::Shuffled(ShuffleMergeOrder {\n                new_row_id_to_old_row_id,\n                alive_bitsets: doc_id_mapping.alive_bitsets,\n            })\n        }\n    }\n}\n\nfn extract_fast_field_required_columns(schema: &Schema) -> Vec<(String, ColumnType)> {\n    schema\n        .fields()\n        .map(|(_, field_entry)| field_entry)\n        .filter(|field_entry| field_entry.is_fast())\n        .filter_map(|field_entry| {\n            let column_name = field_entry.name().to_string();\n            let column_type = value_type_to_column_type(field_entry.field_type().value_type())?;\n            Some((column_name, column_type))\n        })\n        .collect()\n}\n\nimpl IndexMerger {\n    pub fn open(schema: Schema, segments: &[Segment]) -> crate::Result<IndexMerger> {\n        let alive_bitset = segments.iter().map(|_| None).collect_vec();\n        Self::open_with_custom_alive_set(schema, segments, alive_bitset)\n    }\n\n    // Create merge with a custom delete set.\n    // For every Segment, a delete bitset can be provided, which\n    // will be merged with the existing bit set. Make sure the index\n    // corresponds to the segment index.\n    //\n    // If `None` is provided for custom alive set, the regular alive set will be used.\n    // If a alive_bitset is provided, the union between the provided and regular\n    // alive set will be used.\n    //\n    // This can be used to merge but also apply an additional filter.\n    // One use case is demux, which is basically taking a list of\n    // segments and partitions them e.g. by a value in a field.\n    pub fn open_with_custom_alive_set(\n        schema: Schema,\n        segments: &[Segment],\n        alive_bitset_opt: Vec<Option<AliveBitSet>>,\n    ) -> crate::Result<IndexMerger> {\n        let mut readers = vec![];\n        for (segment, new_alive_bitset_opt) in segments.iter().zip(alive_bitset_opt) {\n            if segment.meta().num_docs() > 0 {\n                let reader =\n                    SegmentReader::open_with_custom_alive_set(segment, new_alive_bitset_opt)?;\n                readers.push(reader);\n            }\n        }\n\n        let max_doc = readers.iter().map(|reader| reader.num_docs()).sum();\n        // sort segments by their natural sort setting\n        if max_doc >= MAX_DOC_LIMIT {\n            let err_msg = format!(\n                \"The segment resulting from this merge would have {max_doc} docs,which exceeds \\\n                 the limit {MAX_DOC_LIMIT}.\"\n            );\n            return Err(crate::TantivyError::InvalidArgument(err_msg));\n        }\n        Ok(IndexMerger {\n            schema,\n            readers,\n            max_doc,\n        })\n    }\n\n    fn write_fieldnorms(\n        &self,\n        mut fieldnorms_serializer: FieldNormsSerializer,\n        doc_id_mapping: &SegmentDocIdMapping,\n    ) -> crate::Result<()> {\n        let fields = FieldNormsWriter::fields_with_fieldnorm(&self.schema);\n        let mut fieldnorms_data = Vec::with_capacity(self.max_doc as usize);\n        for field in fields {\n            fieldnorms_data.clear();\n            let fieldnorms_readers: Vec<FieldNormReader> = self\n                .readers\n                .iter()\n                .map(|reader| reader.get_fieldnorms_reader(field))\n                .collect::<Result<_, _>>()?;\n            for old_doc_addr in doc_id_mapping.iter_old_doc_addrs() {\n                let fieldnorms_reader = &fieldnorms_readers[old_doc_addr.segment_ord as usize];\n                let fieldnorm_id = fieldnorms_reader.fieldnorm_id(old_doc_addr.doc_id);\n                fieldnorms_data.push(fieldnorm_id);\n            }\n            fieldnorms_serializer.serialize_field(field, &fieldnorms_data[..])?;\n        }\n        fieldnorms_serializer.close()?;\n        Ok(())\n    }\n\n    fn write_fast_fields(\n        &self,\n        fast_field_wrt: &mut WritePtr,\n        doc_id_mapping: SegmentDocIdMapping,\n    ) -> crate::Result<()> {\n        debug_time!(\"write-fast-fields\");\n        let required_columns = extract_fast_field_required_columns(&self.schema);\n        let columnars: Vec<&ColumnarReader> = self\n            .readers\n            .iter()\n            .map(|reader| reader.fast_fields().columnar())\n            .collect();\n        let merge_row_order = convert_to_merge_order(&columnars[..], doc_id_mapping);\n        columnar::merge_columnar(\n            &columnars[..],\n            &required_columns,\n            merge_row_order,\n            fast_field_wrt,\n        )?;\n        Ok(())\n    }\n\n    /// Creates a mapping if the segments are stacked. this is helpful to merge codelines between\n    /// index sorting and the others\n    pub(crate) fn get_doc_id_from_concatenated_data(&self) -> crate::Result<SegmentDocIdMapping> {\n        let total_num_new_docs = self\n            .readers\n            .iter()\n            .map(|reader| reader.num_docs() as usize)\n            .sum();\n\n        let mut mapping: Vec<DocAddress> = Vec::with_capacity(total_num_new_docs);\n\n        mapping.extend(\n            self.readers\n                .iter()\n                .enumerate()\n                .flat_map(|(segment_ord, reader)| {\n                    reader.doc_ids_alive().map(move |doc_id| DocAddress {\n                        segment_ord: segment_ord as u32,\n                        doc_id,\n                    })\n                }),\n        );\n\n        let has_deletes: bool = self.readers.iter().any(SegmentReader::has_deletes);\n        let mapping_type = if has_deletes {\n            MappingType::StackedWithDeletes\n        } else {\n            MappingType::Stacked\n        };\n        let alive_bitsets: Vec<Option<ReadOnlyBitSet>> = self\n            .readers\n            .iter()\n            .map(|reader| {\n                let alive_bitset = reader.alive_bitset()?;\n                Some(alive_bitset.bitset().clone())\n            })\n            .collect();\n        Ok(SegmentDocIdMapping::new(\n            mapping,\n            mapping_type,\n            alive_bitsets,\n        ))\n    }\n\n    fn write_postings_for_field(\n        &self,\n        indexed_field: Field,\n        _field_type: &FieldType,\n        serializer: &mut InvertedIndexSerializer,\n        fieldnorm_reader: Option<FieldNormReader>,\n        doc_id_mapping: &SegmentDocIdMapping,\n    ) -> crate::Result<()> {\n        debug_time!(\"write-postings-for-field\");\n        let mut positions_buffer: Vec<u32> = Vec::with_capacity(1_000);\n        let mut delta_computer = DeltaComputer::new();\n\n        let mut max_term_ords: Vec<TermOrdinal> = Vec::new();\n\n        let field_readers: Vec<Arc<InvertedIndexReader>> = self\n            .readers\n            .iter()\n            .map(|reader| reader.inverted_index(indexed_field))\n            .collect::<crate::Result<Vec<_>>>()?;\n\n        let mut field_term_streams = Vec::new();\n        for field_reader in &field_readers {\n            let terms = field_reader.terms();\n            field_term_streams.push(terms.stream()?);\n            max_term_ords.push(terms.num_terms() as u64);\n        }\n\n        let mut merged_terms = TermMerger::new(field_term_streams);\n\n        // map from segment doc ids to the resulting merged segment doc id.\n\n        let mut merged_doc_id_map: Vec<Vec<Option<DocId>>> = self\n            .readers\n            .iter()\n            .map(|reader| {\n                let mut segment_local_map = vec![];\n                segment_local_map.resize(reader.max_doc() as usize, None);\n                segment_local_map\n            })\n            .collect();\n        for (new_doc_id, old_doc_addr) in doc_id_mapping.iter_old_doc_addrs().enumerate() {\n            let segment_map = &mut merged_doc_id_map[old_doc_addr.segment_ord as usize];\n            segment_map[old_doc_addr.doc_id as usize] = Some(new_doc_id as DocId);\n        }\n\n        // Note that the total number of tokens is not exact.\n        // It is only used as a parameter in the BM25 formula.\n        let total_num_tokens: u64 = estimate_total_num_tokens(&self.readers, indexed_field)?;\n\n        // Create the total list of doc ids\n        // by stacking the doc ids from the different segment.\n        //\n        // In the new segments, the doc id from the different\n        // segment are stacked so that :\n        // - Segment 0's doc ids become doc id [0, seg.max_doc]\n        // - Segment 1's doc ids become  [seg0.max_doc, seg0.max_doc + seg.max_doc]\n        // - Segment 2's doc ids become  [seg0.max_doc + seg1.max_doc, seg0.max_doc + seg1.max_doc +\n        //   seg2.max_doc]\n        //\n        // This stacking applies only when the index is not sorted, in that case the\n        // doc_ids are kmerged by their sort property\n        let mut field_serializer =\n            serializer.new_field(indexed_field, total_num_tokens, fieldnorm_reader)?;\n\n        let field_entry = self.schema.get_field_entry(indexed_field);\n\n        // ... set segment postings option the new field.\n        let segment_postings_option = field_entry.field_type().get_index_record_option().expect(\n            \"Encountered a field that is not supposed to be\n                         indexed. Have you modified the schema?\",\n        );\n\n        let mut segment_postings_containing_the_term: Vec<(usize, SegmentPostings)> = vec![];\n\n        while merged_terms.advance() {\n            segment_postings_containing_the_term.clear();\n            let term_bytes: &[u8] = merged_terms.key();\n\n            let mut total_doc_freq = 0;\n\n            // Let's compute the list of non-empty posting lists\n            for (segment_ord, term_info) in merged_terms.current_segment_ords_and_term_infos() {\n                let segment_reader = &self.readers[segment_ord];\n                let inverted_index: &InvertedIndexReader = &field_readers[segment_ord];\n                let segment_postings = inverted_index\n                    .read_postings_from_terminfo(&term_info, segment_postings_option)?;\n                let alive_bitset_opt = segment_reader.alive_bitset();\n                let doc_freq = if let Some(alive_bitset) = alive_bitset_opt {\n                    segment_postings.doc_freq_given_deletes(alive_bitset)\n                } else {\n                    segment_postings.doc_freq()\n                };\n                if doc_freq > 0u32 {\n                    total_doc_freq += doc_freq;\n                    segment_postings_containing_the_term.push((segment_ord, segment_postings));\n                }\n            }\n\n            // At this point, `segment_postings` contains the posting list\n            // of all of the segments containing the given term (and that are non-empty)\n            //\n            // These segments are non-empty and advance has already been called.\n            if total_doc_freq == 0u32 {\n                // All docs that used to contain the term have been deleted. The `term` will be\n                // entirely removed.\n                continue;\n            }\n\n            // This should never happen as we early exited for total_doc_freq == 0.\n            assert!(!segment_postings_containing_the_term.is_empty());\n\n            let has_term_freq = {\n                let has_term_freq = !segment_postings_containing_the_term[0]\n                    .1\n                    .block_cursor\n                    .freqs()\n                    .is_empty();\n                for (_, postings) in &segment_postings_containing_the_term[1..] {\n                    // This may look at a strange way to test whether we have term freq or not.\n                    // With JSON object, the schema is not sufficient to know whether a term\n                    // has its term frequency encoded or not:\n                    // strings may have term frequencies, while number terms never have one.\n                    //\n                    // Ideally, we should have burnt one bit of two in the `TermInfo`.\n                    // However, we preferred not changing the codec too much and detect this\n                    // instead by\n                    // - looking at the size of the skip data for bitpacked blocks\n                    // - observing the absence of remaining data after reading the docs for vint\n                    // blocks.\n                    //\n                    // Overall the reliable way to know if we have actual frequencies loaded or not\n                    // is to check whether the actual decoded array is empty or not.\n                    if has_term_freq == postings.block_cursor.freqs().is_empty() {\n                        return Err(DataCorruption::comment_only(\n                            \"Term freqs are inconsistent across segments\",\n                        )\n                        .into());\n                    }\n                }\n                has_term_freq\n            };\n\n            field_serializer.new_term(term_bytes, total_doc_freq, has_term_freq)?;\n\n            // We can now serialize this postings, by pushing each document to the\n            // postings serializer.\n            for (segment_ord, mut segment_postings) in\n                segment_postings_containing_the_term.drain(..)\n            {\n                let old_to_new_doc_id = &merged_doc_id_map[segment_ord];\n\n                let mut doc = segment_postings.doc();\n                while doc != TERMINATED {\n                    // deleted doc are skipped as they do not have a `remapped_doc_id`.\n                    if let Some(remapped_doc_id) = old_to_new_doc_id[doc as usize] {\n                        // we make sure to only write the term if\n                        // there is at least one document.\n                        let term_freq = if has_term_freq {\n                            segment_postings.positions(&mut positions_buffer);\n                            segment_postings.term_freq()\n                        } else {\n                            // The positions_buffer may contain positions from the previous term\n                            // Existence of positions depend on the value type in JSON fields.\n                            // https://github.com/quickwit-oss/tantivy/issues/2283\n                            positions_buffer.clear();\n                            0u32\n                        };\n\n                        let delta_positions = delta_computer.compute_delta(&positions_buffer);\n                        field_serializer.write_doc(remapped_doc_id, term_freq, delta_positions);\n                    }\n\n                    doc = segment_postings.advance();\n                }\n            }\n            // closing the term.\n            field_serializer.close_term()?;\n        }\n        field_serializer.close()?;\n        Ok(())\n    }\n\n    fn write_postings(\n        &self,\n        serializer: &mut InvertedIndexSerializer,\n        fieldnorm_readers: FieldNormReaders,\n        doc_id_mapping: &SegmentDocIdMapping,\n    ) -> crate::Result<()> {\n        for (field, field_entry) in self.schema.fields() {\n            let fieldnorm_reader = fieldnorm_readers.get_field(field)?;\n            if field_entry.is_indexed() {\n                self.write_postings_for_field(\n                    field,\n                    field_entry.field_type(),\n                    serializer,\n                    fieldnorm_reader,\n                    doc_id_mapping,\n                )?;\n            }\n        }\n        Ok(())\n    }\n\n    fn write_storable_fields(&self, store_writer: &mut StoreWriter) -> crate::Result<()> {\n        debug_time!(\"write-storable-fields\");\n        debug!(\"write-storable-field\");\n\n        for reader in &self.readers {\n            let store_reader = reader.get_store_reader(1)?;\n            if reader.has_deletes()\n                    // If there is not enough data in the store, we avoid stacking in order to\n                    // avoid creating many small blocks in the doc store. Once we have 5 full blocks,\n                    // we start stacking. In the worst case 2/7 of the blocks would be very small.\n                    // [segment 1 - {1 doc}][segment 2 - {fullblock * 5}{1doc}]\n                    // => 5 * full blocks, 2 * 1 document blocks\n                    //\n                    // In a more realistic scenario the segments are of the same size, so 1/6 of\n                    // the doc stores would be on average half full, given total randomness (which\n                    // is not the case here, but not sure how it behaves exactly).\n                    //\n                    // https://github.com/quickwit-oss/tantivy/issues/1053\n                    //\n                    // take 7 in order to not walk over all checkpoints.\n                    || store_reader.block_checkpoints().take(7).count() < 6\n                    || store_reader.decompressor() != store_writer.compressor().into()\n            {\n                for doc_bytes_res in store_reader.iter_raw(reader.alive_bitset()) {\n                    let doc_bytes = doc_bytes_res?;\n                    store_writer.store_bytes(&doc_bytes)?;\n                }\n            } else {\n                store_writer.stack(store_reader)?;\n            }\n        }\n        Ok(())\n    }\n\n    /// Writes the merged segment by pushing information\n    /// to the `SegmentSerializer`.\n    ///\n    /// # Returns\n    /// The number of documents in the resulting segment.\n    pub fn write(&self, mut serializer: SegmentSerializer) -> crate::Result<u32> {\n        let doc_id_mapping = self.get_doc_id_from_concatenated_data()?;\n        debug!(\"write-fieldnorms\");\n        if let Some(fieldnorms_serializer) = serializer.extract_fieldnorms_serializer() {\n            self.write_fieldnorms(fieldnorms_serializer, &doc_id_mapping)?;\n        }\n        debug!(\"write-postings\");\n        let fieldnorm_data = serializer\n            .segment()\n            .open_read(SegmentComponent::FieldNorms)?;\n        let fieldnorm_readers = FieldNormReaders::open(fieldnorm_data)?;\n        self.write_postings(\n            serializer.get_postings_serializer(),\n            fieldnorm_readers,\n            &doc_id_mapping,\n        )?;\n\n        debug!(\"write-storagefields\");\n        self.write_storable_fields(serializer.get_store_writer())?;\n        debug!(\"write-fastfields\");\n        self.write_fast_fields(serializer.get_fast_field_write(), doc_id_mapping)?;\n\n        debug!(\"close-serializer\");\n        serializer.close()?;\n        Ok(self.max_doc)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use columnar::Column;\n    use proptest::prop_oneof;\n    use proptest::strategy::Strategy;\n    use schema::FAST;\n\n    use crate::collector::tests::{\n        BytesFastFieldTestCollector, FastFieldTestCollector, TEST_COLLECTOR_WITH_SCORE,\n    };\n    use crate::collector::{Count, FacetCollector};\n    use crate::index::{Index, SegmentId};\n    use crate::indexer::NoMergePolicy;\n    use crate::query::{AllQuery, BooleanQuery, EnableScoring, Scorer, TermQuery};\n    use crate::schema::{\n        Facet, FacetOptions, IndexRecordOption, NumericOptions, TantivyDocument, Term,\n        TextFieldIndexing, Value, INDEXED, TEXT,\n    };\n    use crate::time::OffsetDateTime;\n    use crate::{\n        assert_nearly_equals, schema, DateTime, DocAddress, DocId, DocSet, IndexSettings,\n        IndexWriter, Searcher,\n    };\n\n    #[test]\n    fn test_index_merger_no_deletes() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_fieldtype = schema::TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default().set_index_option(IndexRecordOption::WithFreqs),\n            )\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text\", text_fieldtype);\n        let date_field = schema_builder.add_date_field(\"date\", INDEXED);\n        let score_fieldtype = schema::NumericOptions::default().set_fast();\n        let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype);\n        let bytes_score_field = schema_builder.add_bytes_field(\"score_bytes\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index.reader()?;\n        let curr_time = OffsetDateTime::now_utc();\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                text_field => \"af b\",\n                score_field => 3u64,\n                date_field => DateTime::from_utc(curr_time),\n                bytes_score_field => 3u32.to_be_bytes().as_ref()\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"a b c\",\n                score_field => 5u64,\n                bytes_score_field => 5u32.to_be_bytes().as_ref()\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"a b c d\",\n                score_field => 7u64,\n                bytes_score_field => 7u32.to_be_bytes().as_ref()\n            ))?;\n            index_writer.commit()?;\n            // writing the segment\n            index_writer.add_document(doc!(\n                text_field => \"af b\",\n                date_field => DateTime::from_utc(curr_time),\n                score_field => 11u64,\n                bytes_score_field => 11u32.to_be_bytes().as_ref()\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"a b c g\",\n                score_field => 13u64,\n                bytes_score_field => 13u32.to_be_bytes().as_ref()\n            ))?;\n            index_writer.commit()?;\n        }\n        {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n        {\n            reader.reload()?;\n            let searcher = reader.searcher();\n            let get_doc_ids = |terms: Vec<Term>| {\n                let query = BooleanQuery::new_multiterms_query(terms);\n                searcher\n                    .search(&query, &TEST_COLLECTOR_WITH_SCORE)\n                    .map(|top_docs| top_docs.docs().to_vec())\n            };\n            {\n                assert_eq!(\n                    get_doc_ids(vec![Term::from_field_text(text_field, \"a\")])?,\n                    vec![\n                        DocAddress::new(0, 1),\n                        DocAddress::new(0, 2),\n                        DocAddress::new(0, 4)\n                    ]\n                );\n                assert_eq!(\n                    get_doc_ids(vec![Term::from_field_text(text_field, \"af\")])?,\n                    vec![DocAddress::new(0, 0), DocAddress::new(0, 3)]\n                );\n                assert_eq!(\n                    get_doc_ids(vec![Term::from_field_text(text_field, \"g\")])?,\n                    vec![DocAddress::new(0, 4)]\n                );\n                assert_eq!(\n                    get_doc_ids(vec![Term::from_field_text(text_field, \"b\")])?,\n                    vec![\n                        DocAddress::new(0, 0),\n                        DocAddress::new(0, 1),\n                        DocAddress::new(0, 2),\n                        DocAddress::new(0, 3),\n                        DocAddress::new(0, 4)\n                    ]\n                );\n                assert_eq!(\n                    get_doc_ids(vec![Term::from_field_date_for_search(\n                        date_field,\n                        DateTime::from_utc(curr_time)\n                    )])?,\n                    vec![DocAddress::new(0, 0), DocAddress::new(0, 3)]\n                );\n            }\n            {\n                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 0))?;\n                assert_eq!(\n                    doc.get_first(text_field).unwrap().as_value().as_str(),\n                    Some(\"af b\")\n                );\n            }\n            {\n                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 1))?;\n                assert_eq!(\n                    doc.get_first(text_field).unwrap().as_value().as_str(),\n                    Some(\"a b c\")\n                );\n            }\n            {\n                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 2))?;\n                assert_eq!(\n                    doc.get_first(text_field).unwrap().as_value().as_str(),\n                    Some(\"a b c d\")\n                );\n            }\n            {\n                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 3))?;\n                assert_eq!(doc.get_first(text_field).unwrap().as_str(), Some(\"af b\"));\n            }\n            {\n                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 4))?;\n                assert_eq!(doc.get_first(text_field).unwrap().as_str(), Some(\"a b c g\"));\n            }\n\n            {\n                let get_fast_vals = |terms: Vec<Term>| {\n                    let query = BooleanQuery::new_multiterms_query(terms);\n                    searcher.search(&query, &FastFieldTestCollector::for_field(\"score\"))\n                };\n                let get_fast_vals_bytes = |terms: Vec<Term>| {\n                    let query = BooleanQuery::new_multiterms_query(terms);\n                    searcher.search(\n                        &query,\n                        &BytesFastFieldTestCollector::for_field(\"score_bytes\"),\n                    )\n                };\n                assert_eq!(\n                    get_fast_vals(vec![Term::from_field_text(text_field, \"a\")])?,\n                    vec![5, 7, 13]\n                );\n                assert_eq!(\n                    get_fast_vals_bytes(vec![Term::from_field_text(text_field, \"a\")])?,\n                    vec![0, 0, 0, 5, 0, 0, 0, 7, 0, 0, 0, 13]\n                );\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_index_merger_with_deletes() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let text_fieldtype = schema::TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default().set_index_option(IndexRecordOption::WithFreqs),\n            )\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text\", text_fieldtype);\n        let score_fieldtype = schema::NumericOptions::default().set_fast();\n        let score_field = schema_builder.add_u64_field(\"score\", score_fieldtype);\n        let bytes_score_field = schema_builder.add_bytes_field(\"score_bytes\", FAST);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests()?;\n        let reader = index.reader().unwrap();\n        let search_term = |searcher: &Searcher, term: Term| {\n            let collector = FastFieldTestCollector::for_field(\"score\");\n            // let bytes_collector = BytesFastFieldTestCollector::for_field(bytes_score_field);\n            let term_query = TermQuery::new(term, IndexRecordOption::Basic);\n            // searcher\n            //     .search(&term_query, &(collector, bytes_collector))\n            //     .map(|(scores, bytes)| {\n            //         let mut score_bytes = &bytes[..];\n            //         for &score in &scores {\n            //             assert_eq!(score as u32, score_bytes.read_u32::<BigEndian>().unwrap());\n            //         }\n            //         scores\n            //     })\n            searcher.search(&term_query, &collector)\n        };\n\n        let empty_vec = Vec::<u64>::new();\n        {\n            // a first commit\n            index_writer.add_document(doc!(\n                text_field => \"a b d\",\n                score_field => 1u64,\n                bytes_score_field => vec![0u8, 0, 0, 1],\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"b c\",\n                score_field => 2u64,\n                bytes_score_field => vec![0u8, 0, 0, 2],\n            ))?;\n            index_writer.delete_term(Term::from_field_text(text_field, \"c\"));\n            index_writer.add_document(doc!(\n                text_field => \"c d\",\n                score_field => 3u64,\n                bytes_score_field => vec![0u8, 0, 0, 3],\n            ))?;\n            index_writer.commit()?;\n            reader.reload()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].max_doc(), 3);\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"a\"))?,\n                vec![1]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"b\"))?,\n                vec![1]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"c\"))?,\n                vec![3]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"d\"))?,\n                vec![1, 3]\n            );\n        }\n        {\n            // a second commit\n            index_writer.add_document(doc!(\n                text_field => \"a d e\",\n                score_field => 4_000u64,\n                bytes_score_field => vec![0u8, 0, 0, 4],\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"e f\",\n                score_field => 5_000u64,\n                bytes_score_field => vec![0u8, 0, 0, 5],\n            ))?;\n            index_writer.delete_term(Term::from_field_text(text_field, \"a\"));\n            index_writer.delete_term(Term::from_field_text(text_field, \"f\"));\n            index_writer.add_document(doc!(\n                text_field => \"f g\",\n                score_field => 6_000u64,\n                bytes_score_field => vec![0u8, 0, 23, 112],\n            ))?;\n            index_writer.add_document(doc!(\n                text_field => \"g h\",\n                score_field => 7_000u64,\n                bytes_score_field => vec![0u8, 0, 27, 88],\n            ))?;\n            index_writer.commit()?;\n            reader.reload()?;\n            let searcher = reader.searcher();\n\n            assert_eq!(searcher.segment_readers().len(), 2);\n            assert_eq!(searcher.num_docs(), 3);\n            assert_eq!(searcher.segment_readers()[0].num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].max_doc(), 4);\n            assert_eq!(searcher.segment_readers()[1].num_docs(), 1);\n            assert_eq!(searcher.segment_readers()[1].max_doc(), 3);\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"a\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"b\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"c\"))?,\n                vec![3]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"d\"))?,\n                vec![3]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"e\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"f\"))?,\n                vec![6_000]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"g\"))?,\n                vec![6_000, 7_000]\n            );\n\n            let score_field_reader = searcher\n                .segment_reader(0)\n                .fast_fields()\n                .u64(\"score\")\n                .unwrap();\n            assert_eq!(score_field_reader.min_value(), 4000);\n            assert_eq!(score_field_reader.max_value(), 7000);\n\n            let score_field_reader = searcher\n                .segment_reader(1)\n                .fast_fields()\n                .u64(\"score\")\n                .unwrap();\n            assert_eq!(score_field_reader.min_value(), 1);\n            assert_eq!(score_field_reader.max_value(), 3);\n        }\n        {\n            // merging the segments\n            let segment_ids = index.searchable_segment_ids()?;\n            index_writer.merge(&segment_ids).wait()?;\n            reader.reload()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            assert_eq!(searcher.num_docs(), 3);\n            assert_eq!(searcher.segment_readers()[0].num_docs(), 3);\n            assert_eq!(searcher.segment_readers()[0].max_doc(), 3);\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"a\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"b\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"c\"))?,\n                vec![3]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"d\"))?,\n                vec![3]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"e\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"f\"))?,\n                vec![6_000]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"g\"))?,\n                vec![6_000, 7_000]\n            );\n            let score_field_reader = searcher\n                .segment_reader(0)\n                .fast_fields()\n                .u64(\"score\")\n                .unwrap();\n            assert_eq!(score_field_reader.min_value(), 3);\n            assert_eq!(score_field_reader.max_value(), 7000);\n        }\n        {\n            // test a commit with only deletes\n            index_writer.delete_term(Term::from_field_text(text_field, \"c\"));\n            index_writer.commit()?;\n\n            reader.reload()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            assert_eq!(searcher.num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].max_doc(), 3);\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"a\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"b\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"c\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"d\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"e\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"f\"))?,\n                vec![6_000]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"g\"))?,\n                vec![6_000, 7_000]\n            );\n            let score_field_reader = searcher\n                .segment_reader(0)\n                .fast_fields()\n                .u64(\"score\")\n                .unwrap();\n            assert_eq!(score_field_reader.min_value(), 3);\n            assert_eq!(score_field_reader.max_value(), 7000);\n        }\n        {\n            // Test merging a single segment in order to remove deletes.\n            let segment_ids = index.searchable_segment_ids()?;\n            index_writer.merge(&segment_ids).wait()?;\n            reader.reload()?;\n\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            assert_eq!(searcher.num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].num_docs(), 2);\n            assert_eq!(searcher.segment_readers()[0].max_doc(), 2);\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"a\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"b\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"c\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"d\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"e\"))?,\n                empty_vec\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"f\"))?,\n                vec![6_000]\n            );\n            assert_eq!(\n                search_term(&searcher, Term::from_field_text(text_field, \"g\"))?,\n                vec![6_000, 7_000]\n            );\n            let score_field_reader = searcher\n                .segment_reader(0)\n                .fast_fields()\n                .u64(\"score\")\n                .unwrap();\n            assert_eq!(score_field_reader.min_value(), 6000);\n            assert_eq!(score_field_reader.max_value(), 7000);\n        }\n\n        {\n            // Test removing all docs\n            index_writer.delete_term(Term::from_field_text(text_field, \"g\"));\n            index_writer.commit()?;\n            let segment_ids = index.searchable_segment_ids()?;\n            reader.reload()?;\n\n            let searcher = reader.searcher();\n            assert!(segment_ids.is_empty());\n            assert!(searcher.segment_readers().is_empty());\n            assert_eq!(searcher.num_docs(), 0);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_facets_sort_none() {\n        test_merge_facets(None, true)\n    }\n\n    // force_segment_value_overlap forces the int value for sorting to have overlapping min and max\n    // ranges between segments so that merge algorithm can't apply certain optimizations\n    fn test_merge_facets(index_settings: Option<IndexSettings>, force_segment_value_overlap: bool) {\n        let mut schema_builder = schema::Schema::builder();\n        let facet_field = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let int_options = NumericOptions::default().set_fast().set_indexed();\n        let int_field = schema_builder.add_u64_field(\"intval\", int_options);\n        let mut index_builder = Index::builder().schema(schema_builder.build());\n        if let Some(settings) = index_settings {\n            index_builder = index_builder.settings(settings);\n        }\n        let index = index_builder.create_in_ram().unwrap();\n        // let index = Index::create_in_ram(schema_builder.build());\n        let reader = index.reader().unwrap();\n        let mut int_val = 0;\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            let index_doc =\n                |index_writer: &mut IndexWriter, doc_facets: &[&str], int_val: &mut u64| {\n                    let mut doc = TantivyDocument::default();\n                    for facet in doc_facets {\n                        doc.add_facet(facet_field, Facet::from(facet));\n                    }\n                    doc.add_u64(int_field, *int_val);\n                    *int_val += 1;\n                    index_writer.add_document(doc).unwrap();\n                };\n\n            index_doc(\n                &mut index_writer,\n                &[\"/top/a/firstdoc\", \"/top/b\"],\n                &mut int_val,\n            );\n            index_doc(\n                &mut index_writer,\n                &[\"/top/a/firstdoc\", \"/top/b\", \"/top/c\"],\n                &mut int_val,\n            );\n            index_doc(&mut index_writer, &[\"/top/a\", \"/top/b\"], &mut int_val);\n            index_doc(&mut index_writer, &[\"/top/a\"], &mut int_val);\n\n            index_doc(&mut index_writer, &[\"/top/b\", \"/top/d\"], &mut int_val);\n            if force_segment_value_overlap {\n                index_doc(&mut index_writer, &[\"/top/d\"], &mut 0);\n                index_doc(&mut index_writer, &[\"/top/e\"], &mut 10);\n                index_writer.commit().expect(\"committed\");\n                index_doc(&mut index_writer, &[\"/top/a\"], &mut 5); // 5 is between 0 - 10 so the\n                                                                   // segments don' have disjunct\n                                                                   // ranges\n            } else {\n                index_doc(&mut index_writer, &[\"/top/d\"], &mut int_val);\n                index_doc(&mut index_writer, &[\"/top/e\"], &mut int_val);\n                index_writer.commit().expect(\"committed\");\n                index_doc(&mut index_writer, &[\"/top/a\"], &mut int_val);\n            }\n            index_doc(&mut index_writer, &[\"/top/b\"], &mut int_val);\n            index_doc(&mut index_writer, &[\"/top/c\"], &mut int_val);\n            index_writer.commit().expect(\"committed\");\n\n            index_doc(&mut index_writer, &[\"/top/e\", \"/top/f\"], &mut int_val);\n            index_writer.commit().expect(\"committed\");\n        }\n\n        reader.reload().unwrap();\n        let test_searcher = |expected_num_docs: usize, expected: &[(&str, u64)]| {\n            let searcher = reader.searcher();\n            let mut facet_collector = FacetCollector::for_field(\"facet\");\n            facet_collector.add_facet(Facet::from(\"/top\"));\n            let (count, facet_counts) = searcher\n                .search(&AllQuery, &(Count, facet_collector))\n                .unwrap();\n            assert_eq!(count, expected_num_docs);\n            let facets: Vec<(String, u64)> = facet_counts\n                .get(\"/top\")\n                .map(|(facet, count)| (facet.to_string(), count))\n                .collect();\n            assert_eq!(\n                facets,\n                expected\n                    .iter()\n                    .map(|&(facet_str, count)| (String::from(facet_str), count))\n                    .collect::<Vec<_>>()\n            );\n        };\n        test_searcher(\n            11,\n            &[\n                (\"/top/a\", 5),\n                (\"/top/b\", 5),\n                (\"/top/c\", 2),\n                (\"/top/d\", 2),\n                (\"/top/e\", 2),\n                (\"/top/f\", 1),\n            ],\n        );\n        // Merging the segments\n        {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer\n                .merge(&segment_ids)\n                .wait()\n                .expect(\"Merging failed\");\n            index_writer.wait_merging_threads().unwrap();\n            reader.reload().unwrap();\n            test_searcher(\n                11,\n                &[\n                    (\"/top/a\", 5),\n                    (\"/top/b\", 5),\n                    (\"/top/c\", 2),\n                    (\"/top/d\", 2),\n                    (\"/top/e\", 2),\n                    (\"/top/f\", 1),\n                ],\n            );\n        }\n\n        // Deleting one term\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            let facet = Facet::from_path(vec![\"top\", \"a\", \"firstdoc\"]);\n            let facet_term = Term::from_facet(facet_field, &facet);\n            index_writer.delete_term(facet_term);\n            index_writer.commit().unwrap();\n            reader.reload().unwrap();\n            test_searcher(\n                9,\n                &[\n                    (\"/top/a\", 3),\n                    (\"/top/b\", 3),\n                    (\"/top/c\", 1),\n                    (\"/top/d\", 2),\n                    (\"/top/e\", 2),\n                    (\"/top/f\", 1),\n                ],\n            );\n        }\n    }\n\n    #[test]\n    fn test_bug_merge() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let int_field = schema_builder.add_u64_field(\"intvals\", INDEXED);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc!(int_field => 1u64))?;\n        index_writer.commit().expect(\"commit failed\");\n        index_writer.add_document(doc!(int_field => 1u64))?;\n        index_writer.commit().expect(\"commit failed\");\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 2);\n        index_writer.delete_term(Term::from_field_u64(int_field, 1));\n        let segment_ids = index\n            .searchable_segment_ids()\n            .expect(\"Searchable segments failed.\");\n        index_writer.merge(&segment_ids).wait()?;\n        reader.reload()?;\n        // commit has not been called yet. The document should still be\n        // there.\n        assert_eq!(reader.searcher().num_docs(), 2);\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_multivalued_int_fields_all_deleted() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let int_options = NumericOptions::default().set_fast().set_indexed();\n        let int_field = schema_builder.add_u64_field(\"intvals\", int_options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let reader = index.reader()?;\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            let mut doc = TantivyDocument::default();\n            doc.add_u64(int_field, 1);\n            index_writer.add_document(doc.clone())?;\n            index_writer.commit()?;\n            index_writer.add_document(doc)?;\n            index_writer.commit()?;\n            index_writer.delete_term(Term::from_field_u64(int_field, 1));\n            let segment_ids = index.searchable_segment_ids()?;\n            index_writer.merge(&segment_ids).wait()?;\n\n            // assert delete has not been committed\n            reader.reload()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.num_docs(), 2);\n\n            index_writer.commit()?;\n\n            index_writer.wait_merging_threads()?;\n        }\n\n        reader.reload()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 0);\n        Ok(())\n    }\n\n    #[derive(Debug, Clone, Copy, Eq, PartialEq)]\n    enum IndexingOp {\n        ZeroVal,\n        OneVal { val: u64 },\n        TwoVal { val: u64 },\n        Commit,\n    }\n\n    fn balanced_operation_strategy() -> impl Strategy<Value = IndexingOp> {\n        prop_oneof![\n            (0u64..1u64).prop_map(|_| IndexingOp::ZeroVal),\n            (0u64..1u64).prop_map(|val| IndexingOp::OneVal { val }),\n            (0u64..1u64).prop_map(|val| IndexingOp::TwoVal { val }),\n            (0u64..1u64).prop_map(|_| IndexingOp::Commit),\n        ]\n    }\n\n    use proptest::prelude::*;\n    proptest! {\n        #[test]\n        fn test_merge_columnar_int_proptest(ops in proptest::collection::vec(balanced_operation_strategy(), 1..20)) {\n            assert!(test_merge_int_fields(&ops[..]).is_ok());\n        }\n    }\n    fn test_merge_int_fields(ops: &[IndexingOp]) -> crate::Result<()> {\n        if ops.iter().all(|op| *op == IndexingOp::Commit) {\n            return Ok(());\n        }\n        let expected_doc_and_vals: Vec<(u32, Vec<u64>)> = ops\n            .iter()\n            .filter(|op| *op != &IndexingOp::Commit)\n            .map(|op| match op {\n                IndexingOp::ZeroVal => vec![],\n                IndexingOp::OneVal { val } => vec![*val],\n                IndexingOp::TwoVal { val } => vec![*val, *val],\n                IndexingOp::Commit => unreachable!(),\n            })\n            .enumerate()\n            .map(|(id, val)| (id as u32, val))\n            .collect();\n\n        let mut schema_builder = schema::Schema::builder();\n        let int_options = NumericOptions::default().set_fast().set_indexed();\n        let int_field = schema_builder.add_u64_field(\"intvals\", int_options);\n        let index = Index::create_in_ram(schema_builder.build());\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n            let index_doc = |index_writer: &mut IndexWriter, int_vals: &[u64]| {\n                let mut doc = TantivyDocument::default();\n                for &val in int_vals {\n                    doc.add_u64(int_field, val);\n                }\n                index_writer.add_document(doc).unwrap();\n            };\n\n            for op in ops {\n                match op {\n                    IndexingOp::ZeroVal => index_doc(&mut index_writer, &[]),\n                    IndexingOp::OneVal { val } => index_doc(&mut index_writer, &[*val]),\n                    IndexingOp::TwoVal { val } => index_doc(&mut index_writer, &[*val, *val]),\n                    IndexingOp::Commit => {\n                        index_writer.commit().expect(\"commit failed\");\n                    }\n                }\n            }\n            index_writer.commit().expect(\"commit failed\");\n        }\n        {\n            let mut segment_ids = index.searchable_segment_ids()?;\n            segment_ids.sort();\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n        let reader = index.reader()?;\n        reader.reload()?;\n\n        let mut vals: Vec<u64> = Vec::new();\n        let mut test_vals = move |col: &Column<u64>, doc: DocId, expected: &[u64]| {\n            vals.clear();\n            vals.extend(col.values_for_doc(doc));\n            assert_eq!(&vals[..], expected);\n        };\n\n        let mut test_col = move |col: &Column<u64>, column_expected: &[(u32, Vec<u64>)]| {\n            for (doc_id, vals) in column_expected.iter() {\n                test_vals(col, *doc_id, vals);\n            }\n        };\n\n        {\n            let searcher = reader.searcher();\n            let segment = searcher.segment_reader(0u32);\n            let col = segment\n                .fast_fields()\n                .column_opt::<u64>(\"intvals\")\n                .unwrap()\n                .unwrap();\n\n            test_col(&col, &expected_doc_and_vals);\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_multivalued_int_fields_simple() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n        let int_options = NumericOptions::default().set_fast().set_indexed();\n        let int_field = schema_builder.add_u64_field(\"intvals\", int_options);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        let mut vals: Vec<u64> = Vec::new();\n        let mut test_vals = move |col: &Column<u64>, doc: DocId, expected: &[u64]| {\n            vals.clear();\n            vals.extend(col.values_for_doc(doc));\n            assert_eq!(&vals[..], expected);\n        };\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            let index_doc = |index_writer: &mut IndexWriter, int_vals: &[u64]| {\n                let mut doc = TantivyDocument::default();\n                for &val in int_vals {\n                    doc.add_u64(int_field, val);\n                }\n                index_writer.add_document(doc).unwrap();\n            };\n            index_doc(&mut index_writer, &[1, 2]);\n            index_doc(&mut index_writer, &[1, 2, 3]);\n            index_doc(&mut index_writer, &[4, 5]);\n            index_doc(&mut index_writer, &[1, 2]);\n            index_doc(&mut index_writer, &[1, 5]);\n            index_doc(&mut index_writer, &[3]);\n            index_doc(&mut index_writer, &[17]);\n            assert!(index_writer.commit().is_ok());\n            index_doc(&mut index_writer, &[20]);\n            assert!(index_writer.commit().is_ok());\n            index_doc(&mut index_writer, &[28, 27]);\n            index_doc(&mut index_writer, &[1_000]);\n            assert!(index_writer.commit().is_ok());\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        {\n            let segment = searcher.segment_reader(0u32);\n            let column = segment\n                .fast_fields()\n                .column_opt::<u64>(\"intvals\")\n                .unwrap()\n                .unwrap();\n            test_vals(&column, 0, &[1, 2]);\n            test_vals(&column, 1, &[1, 2, 3]);\n            test_vals(&column, 2, &[4, 5]);\n            test_vals(&column, 3, &[1, 2]);\n            test_vals(&column, 4, &[1, 5]);\n            test_vals(&column, 5, &[3]);\n            test_vals(&column, 6, &[17]);\n        }\n\n        {\n            let segment = searcher.segment_reader(1u32);\n            let col = segment\n                .fast_fields()\n                .column_opt::<u64>(\"intvals\")\n                .unwrap()\n                .unwrap();\n            test_vals(&col, 0, &[28, 27]);\n            test_vals(&col, 1, &[1000]);\n        }\n\n        {\n            let segment = searcher.segment_reader(2u32);\n            let col = segment\n                .fast_fields()\n                .column_opt::<u64>(\"intvals\")\n                .unwrap()\n                .unwrap();\n            test_vals(&col, 0, &[20]);\n        }\n\n        // Merging the segments\n        {\n            let segment_ids = index.searchable_segment_ids()?;\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n        reader.reload()?;\n\n        {\n            let searcher = reader.searcher();\n            let segment = searcher.segment_reader(0u32);\n            let col = segment\n                .fast_fields()\n                .column_opt::<u64>(\"intvals\")\n                .unwrap()\n                .unwrap();\n            test_vals(&col, 0, &[1, 2]);\n            test_vals(&col, 1, &[1, 2, 3]);\n            test_vals(&col, 2, &[4, 5]);\n            test_vals(&col, 3, &[1, 2]);\n            test_vals(&col, 4, &[1, 5]);\n            test_vals(&col, 5, &[3]);\n            test_vals(&col, 6, &[17]);\n            test_vals(&col, 7, &[28, 27]);\n            test_vals(&col, 8, &[1000]);\n            test_vals(&col, 9, &[20]);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn merges_f64_fast_fields_correctly() -> crate::Result<()> {\n        let mut builder = schema::SchemaBuilder::new();\n\n        let fast_multi = NumericOptions::default().set_fast();\n\n        let field = builder.add_f64_field(\"f64\", schema::FAST);\n        let multi_field = builder.add_f64_field(\"f64s\", fast_multi);\n\n        let index = Index::create_in_ram(builder.build());\n\n        let mut writer = index.writer_for_tests()?;\n\n        // Make sure we'll attempt to merge every created segment\n        let mut policy = crate::indexer::LogMergePolicy::default();\n        policy.set_min_num_segments(2);\n        writer.set_merge_policy(Box::new(policy));\n\n        for i in 0..100 {\n            let mut doc = TantivyDocument::new();\n            doc.add_f64(field, 42.0);\n            doc.add_f64(multi_field, 0.24);\n            doc.add_f64(multi_field, 0.27);\n            writer.add_document(doc)?;\n            if i % 5 == 0 {\n                writer.commit()?;\n            }\n        }\n\n        writer.commit()?;\n        writer.wait_merging_threads()?;\n\n        // If a merging thread fails, we should end up with more\n        // than one segment here\n        assert_eq!(1, index.searchable_segments()?.len());\n        Ok(())\n    }\n\n    #[test]\n    fn test_merged_index_has_blockwand() -> crate::Result<()> {\n        let mut builder = schema::SchemaBuilder::new();\n        let text = builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(builder.build());\n        let mut writer = index.writer_for_tests()?;\n        let happy_term = Term::from_field_text(text, \"happy\");\n        let term_query = TermQuery::new(happy_term, IndexRecordOption::WithFreqs);\n        for _ in 0..62 {\n            writer.add_document(doc!(text=>\"hello happy tax payer\"))?;\n        }\n        writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let mut term_scorer = term_query\n            .specialized_weight(EnableScoring::enabled_from_searcher(&searcher))?\n            .term_scorer_for_test(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(term_scorer.doc(), 0);\n        assert_nearly_equals!(term_scorer.block_max_score(), 0.0079681855);\n        assert_nearly_equals!(term_scorer.score(), 0.0079681855);\n        for _ in 0..81 {\n            writer.add_document(doc!(text=>\"hello happy tax payer\"))?;\n        }\n        writer.commit()?;\n        reader.reload()?;\n        let searcher = reader.searcher();\n\n        assert_eq!(searcher.segment_readers().len(), 2);\n        for segment_reader in searcher.segment_readers() {\n            let mut term_scorer = term_query\n                .specialized_weight(EnableScoring::enabled_from_searcher(&searcher))?\n                .term_scorer_for_test(segment_reader, 1.0)?\n                .unwrap();\n            // the difference compared to before is intrinsic to the bm25 formula. no worries\n            // there.\n            for doc in segment_reader.doc_ids_alive() {\n                assert_eq!(term_scorer.doc(), doc);\n                assert_nearly_equals!(term_scorer.block_max_score(), 0.003478312);\n                assert_nearly_equals!(term_scorer.score(), 0.003478312);\n                term_scorer.advance();\n            }\n        }\n\n        let segment_ids: Vec<SegmentId> = searcher\n            .segment_readers()\n            .iter()\n            .map(|reader| reader.segment_id())\n            .collect();\n        writer.merge(&segment_ids[..]).wait()?;\n\n        reader.reload()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n\n        let segment_reader = searcher.segment_reader(0u32);\n        let mut term_scorer = term_query\n            .specialized_weight(EnableScoring::enabled_from_searcher(&searcher))?\n            .term_scorer_for_test(segment_reader, 1.0)?\n            .unwrap();\n        // the difference compared to before is intrinsic to the bm25 formula. no worries there.\n        for doc in segment_reader.doc_ids_alive() {\n            assert_eq!(term_scorer.doc(), doc);\n            assert_nearly_equals!(term_scorer.block_max_score(), 0.003478312);\n            assert_nearly_equals!(term_scorer.score(), 0.003478312);\n            term_scorer.advance();\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_max_doc() {\n        // this is the first time I write a unit test for a constant.\n        assert!(((super::MAX_DOC_LIMIT - 1) as i32) >= 0);\n        assert!((super::MAX_DOC_LIMIT as i32) < 0);\n    }\n}\n"
  },
  {
    "path": "src/indexer/mod.rs",
    "content": "//! Indexing and merging data.\n//!\n//! Contains code to create and merge segments.\n//! `IndexWriter` is the main entry point for that, which created from\n//! [`Index::writer`](crate::Index::writer).\n\n/// Delete queue implementation for broadcasting delete operations to consumers.\npub(crate) mod delete_queue;\npub(crate) mod path_to_unordered_id;\n\npub(crate) mod doc_id_mapping;\nmod doc_opstamp_mapping;\nmod flat_map_with_buffer;\npub(crate) mod index_writer;\npub(crate) mod index_writer_status;\npub(crate) mod indexing_term;\nmod log_merge_policy;\nmod merge_index_test;\nmod merge_operation;\npub(crate) mod merge_policy;\npub(crate) mod merger;\npub(crate) mod operation;\npub(crate) mod prepared_commit;\nmod segment_entry;\nmod segment_manager;\nmod segment_register;\npub(crate) mod segment_serializer;\npub(crate) mod segment_updater;\npub(crate) mod segment_writer;\npub(crate) mod single_segment_index_writer;\nmod stamper;\n\nuse crossbeam_channel as channel;\nuse smallvec::SmallVec;\n\npub use self::index_writer::{advance_deletes, IndexWriter, IndexWriterOptions};\npub use self::log_merge_policy::LogMergePolicy;\npub use self::merge_operation::MergeOperation;\npub use self::merge_policy::{MergeCandidate, MergePolicy, NoMergePolicy};\npub use self::operation::{AddOperation, DeleteOperation, UserOperation};\npub use self::prepared_commit::PreparedCommit;\npub use self::segment_entry::SegmentEntry;\npub(crate) use self::segment_serializer::SegmentSerializer;\npub use self::segment_updater::{merge_filtered_segments, merge_indices};\npub use self::segment_writer::SegmentWriter;\npub use self::single_segment_index_writer::SingleSegmentIndexWriter;\n\n/// Alias for the default merge policy, which is the `LogMergePolicy`.\npub type DefaultMergePolicy = LogMergePolicy;\n\n// Batch of documents.\n// Most of the time, users will send operation one-by-one, but it can be useful to\n// send them as a small block to ensure that\n// - all docs in the operation will happen on the same segment and continuous doc_ids.\n// - all operations in the group are committed at the same time, making the group\n// atomic.\ntype AddBatch<D> = SmallVec<[AddOperation<D>; 4]>;\ntype AddBatchSender<D> = channel::Sender<AddBatch<D>>;\ntype AddBatchReceiver<D> = channel::Receiver<AddBatch<D>>;\n\n#[cfg(feature = \"mmap\")]\n#[cfg(test)]\nmod tests_mmap {\n\n    use common::ByteCount;\n\n    use crate::aggregation::agg_req::Aggregations;\n    use crate::aggregation::agg_result::AggregationResults;\n    use crate::aggregation::AggregationCollector;\n    use crate::collector::{Count, TopDocs};\n    use crate::index::FieldMetadata;\n    use crate::query::{AllQuery, QueryParser};\n    use crate::schema::{JsonObjectOptions, Schema, Type, FAST, INDEXED, STORED, TEXT};\n    use crate::{Index, IndexWriter, Term};\n\n    #[test]\n    fn test_advance_delete_bug() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_from_tempdir(schema_builder.build())?;\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        // there must be one deleted document in the segment\n        index_writer.add_document(doc!(text_field=>\"b\"))?;\n        index_writer.delete_term(Term::from_field_text(text_field, \"b\"));\n        // we need enough data to trigger the bug (at least 32 documents)\n        for _ in 0..32 {\n            index_writer.add_document(doc!(text_field=>\"c\"))?;\n        }\n        index_writer.commit()?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_json_field_expand_dots_disabled_dot_escaped_required() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"prometheus\", \"val\": \"hello\"});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 1);\n        let parse_query = QueryParser::for_index(&index, Vec::new());\n        {\n            let query = parse_query\n                .parse_query(r\"json.k8s\\.container\\.name:prometheus\")\n                .unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 1);\n        }\n        {\n            let query = parse_query\n                .parse_query(r#\"json.k8s.container.name:prometheus\"#)\n                .unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 0);\n        }\n    }\n\n    #[test]\n    fn test_json_field_number() {\n        // this test was added specifically to reach some cases related to using json fields, with\n        // frequency enabled, to store integers, with enough documents containing a single integer\n        // that the posting list can be bitpacked.\n        let mut schema_builder = Schema::builder();\n\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        for _ in 0..256 {\n            let json = serde_json::json!({\"somekey\": 1u64, \"otherkey\": -2i64});\n            index_writer.add_document(doc!(json_field=>json)).unwrap();\n\n            let json = serde_json::json!({\"somekey\": \"1str\", \"otherkey\": \"2str\"});\n            index_writer.add_document(doc!(json_field=>json)).unwrap();\n        }\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 512);\n        let parse_query = QueryParser::for_index(&index, Vec::new());\n        {\n            let query = parse_query.parse_query(r\"json.somekey:1\").unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 256);\n        }\n    }\n    #[test]\n    fn test_json_field_null_byte_is_ignored() {\n        let mut schema_builder = Schema::builder();\n        let options = JsonObjectOptions::from(TEXT | FAST).set_expand_dots_enabled();\n        let field = schema_builder.add_json_field(\"json\", options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({\"key\": \"test1\", \"invalidkey\\u{0000}\": \"test2\"})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let inv_indexer = segment_reader.inverted_index(field).unwrap();\n        let term_dict = inv_indexer.terms();\n        assert_eq!(term_dict.num_terms(), 1);\n        let mut term_bytes = Vec::new();\n        term_dict.ord_to_term(0, &mut term_bytes).unwrap();\n        assert_eq!(term_bytes, b\"key\\0stest1\");\n    }\n\n    #[test]\n    fn test_json_field_1byte() {\n        // Test when field name contains a '1' byte, which has special meaning in tantivy.\n        // The 1 byte can be addressed as '1' byte or '.'.\n        let field_name_in = \"\\u{0001}\";\n        let field_name_out = \"\\u{0001}\";\n        test_json_field_name(field_name_in, field_name_out);\n\n        // Test when field name contains a '1' byte, which has special meaning in tantivy.\n        let field_name_in = \"\\u{0001}\";\n        let field_name_out = \".\";\n        test_json_field_name(field_name_in, field_name_out);\n    }\n\n    #[test]\n    fn test_json_field_dot() {\n        // Test when field name contains a '.'\n        let field_name_in = \".\";\n        let field_name_out = \".\";\n        test_json_field_name(field_name_in, field_name_out);\n    }\n    fn test_json_field_name(field_name_in: &str, field_name_out: &str) {\n        let mut schema_builder = Schema::builder();\n\n        let options = JsonObjectOptions::from(TEXT | FAST).set_expand_dots_enabled();\n        let field = schema_builder.add_json_field(\"json\", options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({format!(\"{field_name_in}\"): \"test1\", format!(\"num{field_name_in}\"): 10})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({format!(\"a{field_name_in}\"): \"test2\"})))\n            .unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({format!(\"a{field_name_in}a\"): \"test3\"})))\n            .unwrap();\n        index_writer\n            .add_document(\n                doc!(field=>json!({format!(\"a{field_name_in}a{field_name_in}\"): \"test4\"})),\n            )\n            .unwrap();\n        index_writer\n            .add_document(\n                doc!(field=>json!({format!(\"a{field_name_in}.ab{field_name_in}\"): \"test5\"})),\n            )\n            .unwrap();\n        index_writer\n            .add_document(\n                doc!(field=>json!({format!(\"a{field_name_in}\"): json!({format!(\"a{field_name_in}\"): \"test6\"}) })),\n            )\n            .unwrap();\n        index_writer\n            .add_document(doc!(field=>json!({format!(\"{field_name_in}a\" ): \"test7\"})))\n            .unwrap();\n\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let parse_query = QueryParser::for_index(&index, Vec::new());\n        let test_query = |query_str: &str| {\n            let query = parse_query.parse_query(query_str).unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 1, \"{query_str}\");\n        };\n        test_query(format!(\"json.{field_name_out}:test1\").as_str());\n        test_query(format!(\"json.a{field_name_out}:test2\").as_str());\n        test_query(format!(\"json.a{field_name_out}a:test3\").as_str());\n        test_query(format!(\"json.a{field_name_out}a{field_name_out}:test4\").as_str());\n        test_query(format!(\"json.a{field_name_out}.ab{field_name_out}:test5\").as_str());\n        test_query(format!(\"json.a{field_name_out}.a{field_name_out}:test6\").as_str());\n        test_query(format!(\"json.{field_name_out}a:test7\").as_str());\n\n        let test_agg = |field_name: &str, expected: &str| {\n            let agg_req_str = json!(\n            {\n              \"termagg\": {\n                \"terms\": {\n                  \"field\": field_name,\n                }\n              }\n            });\n\n            let agg_req: Aggregations = serde_json::from_value(agg_req_str).unwrap();\n            let collector = AggregationCollector::from_aggs(agg_req, Default::default());\n            let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n            let res = serde_json::to_value(agg_res).unwrap();\n            assert_eq!(res[\"termagg\"][\"buckets\"][0][\"doc_count\"], 1);\n            assert_eq!(res[\"termagg\"][\"buckets\"][0][\"key\"], expected);\n        };\n\n        test_agg(format!(\"json.{field_name_out}\").as_str(), \"test1\");\n        test_agg(format!(\"json.a{field_name_out}\").as_str(), \"test2\");\n        test_agg(format!(\"json.a{field_name_out}a\").as_str(), \"test3\");\n        test_agg(\n            format!(\"json.a{field_name_out}a{field_name_out}\").as_str(),\n            \"test4\",\n        );\n        test_agg(\n            format!(\"json.a{field_name_out}.ab{field_name_out}\").as_str(),\n            \"test5\",\n        );\n        test_agg(\n            format!(\"json.a{field_name_out}.a{field_name_out}\").as_str(),\n            \"test6\",\n        );\n        test_agg(format!(\"json.{field_name_out}a\").as_str(), \"test7\");\n\n        // `.` is stored as `\\u{0001}` internally in tantivy\n        let field_name_out_internal = if field_name_out == \".\" {\n            \"\\u{0001}\"\n        } else {\n            field_name_out\n        };\n\n        let mut fields: Vec<(String, Type)> = reader.searcher().segment_readers()[0]\n            .inverted_index(field)\n            .unwrap()\n            .list_encoded_json_fields()\n            .unwrap()\n            .into_iter()\n            .map(|field_space| (field_space.field_name, field_space.field_type))\n            .collect();\n        assert_eq!(fields.len(), 8);\n        fields.sort();\n        let mut expected_fields = vec![\n            (format!(\"a{field_name_out_internal}\"), Type::Str),\n            (format!(\"a{field_name_out_internal}a\"), Type::Str),\n            (\n                format!(\"a{field_name_out_internal}a{field_name_out_internal}\"),\n                Type::Str,\n            ),\n            (\n                format!(\"a{field_name_out_internal}\\u{1}ab{field_name_out_internal}\"),\n                Type::Str,\n            ),\n            (\n                format!(\"a{field_name_out_internal}\\u{1}a{field_name_out_internal}\"),\n                Type::Str,\n            ),\n            (format!(\"{field_name_out_internal}a\"), Type::Str),\n            (field_name_out_internal.to_string(), Type::Str),\n            (format!(\"num{field_name_out_internal}\"), Type::I64),\n        ];\n        expected_fields.sort();\n        assert_eq!(fields, expected_fields);\n        // Check columnar reader\n        let mut columns = reader.searcher().segment_readers()[0]\n            .fast_fields()\n            .columnar()\n            .list_columns()\n            .unwrap()\n            .into_iter()\n            .map(|(name, _)| name)\n            .collect::<Vec<_>>();\n        let mut expected_columns = vec![\n            format!(\"json\\u{1}{field_name_out_internal}\"),\n            format!(\"json\\u{1}{field_name_out_internal}a\"),\n            format!(\"json\\u{1}a{field_name_out_internal}\"),\n            format!(\"json\\u{1}a{field_name_out_internal}a\"),\n            format!(\"json\\u{1}a{field_name_out_internal}a{field_name_out_internal}\"),\n            format!(\"json\\u{1}a{field_name_out_internal}\\u{1}ab{field_name_out_internal}\"),\n            format!(\"json\\u{1}a{field_name_out_internal}\\u{1}a{field_name_out_internal}\"),\n            format!(\"json\\u{1}num{field_name_out_internal}\"),\n        ];\n        columns.sort();\n        expected_columns.sort();\n        assert_eq!(columns, expected_columns);\n    }\n\n    #[test]\n    fn test_json_field_expand_dots_enabled_dot_escape_not_required() {\n        let mut schema_builder = Schema::builder();\n        let json_options: JsonObjectOptions =\n            JsonObjectOptions::from(TEXT).set_expand_dots_enabled();\n        let json_field = schema_builder.add_json_field(\"json\", json_options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"prometheus\", \"val\": \"hello\"});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 1);\n        let parse_query = QueryParser::for_index(&index, Vec::new());\n        {\n            let query = parse_query\n                .parse_query(r#\"json.k8s.container.name:prometheus\"#)\n                .unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 1);\n        }\n        {\n            let query = parse_query\n                .parse_query(r\"json.k8s\\.container\\.name:prometheus\")\n                .unwrap();\n            let num_docs = searcher.search(&query, &Count).unwrap();\n            assert_eq!(num_docs, 1);\n        }\n    }\n\n    #[test]\n    fn test_json_field_list_fields() {\n        let mut schema_builder = Schema::builder();\n        let json_options: JsonObjectOptions = JsonObjectOptions::from(TEXT);\n        let json_field = schema_builder.add_json_field(\"json\", json_options);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"prometheus\", \"val\": \"hello\", \"sub\": {\"a\": 1, \"b\": 2}});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"prometheus\", \"val\": \"hello\", \"suber\": {\"a\": 1, \"b\": 2}});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"prometheus\", \"val\": \"hello\", \"suber\": {\"a\": \"mixed\", \"b\": 2}});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 3);\n\n        let reader = &searcher.segment_readers()[0];\n        let inverted_index = reader.inverted_index(json_field).unwrap();\n        assert_eq!(\n            inverted_index\n                .list_encoded_json_fields()\n                .unwrap()\n                .into_iter()\n                .map(|field_space| (field_space.field_name, field_space.field_type))\n                .collect::<Vec<_>>(),\n            [\n                (\"k8s.container.name\".to_string(), Type::Str),\n                (\"sub\\u{1}a\".to_string(), Type::I64),\n                (\"sub\\u{1}b\".to_string(), Type::I64),\n                (\"suber\\u{1}a\".to_string(), Type::I64),\n                (\"suber\\u{1}a\".to_string(), Type::Str),\n                (\"suber\\u{1}b\".to_string(), Type::I64),\n                (\"val\".to_string(), Type::Str),\n            ]\n        );\n    }\n\n    #[test]\n    fn test_json_fields_metadata_expanded_dots_one_segment() {\n        test_json_fields_metadata(true, true);\n    }\n\n    #[test]\n    fn test_json_fields_metadata_expanded_dots_multi_segment() {\n        test_json_fields_metadata(true, false);\n    }\n\n    #[test]\n    fn test_json_fields_metadata_no_expanded_dots_one_segment() {\n        test_json_fields_metadata(false, true);\n    }\n\n    #[test]\n    fn test_json_fields_metadata_no_expanded_dots_multi_segment() {\n        test_json_fields_metadata(false, false);\n    }\n\n    #[track_caller]\n    fn assert_size_eq(lhs: Option<ByteCount>, rhs: Option<ByteCount>) {\n        let ignore_actual_values = |size_opt: Option<ByteCount>| size_opt.map(|val| val > 0);\n        assert_eq!(ignore_actual_values(lhs), ignore_actual_values(rhs));\n    }\n\n    #[track_caller]\n    fn assert_field_metadata_eq_but_ignore_field_size(\n        expected: &FieldMetadata,\n        actual: &FieldMetadata,\n    ) {\n        assert_eq!(&expected.field_name, &actual.field_name);\n        assert_eq!(&expected.typ, &actual.typ);\n        assert_eq!(&expected.stored, &actual.stored);\n        assert_size_eq(expected.postings_size, actual.postings_size);\n        assert_size_eq(expected.positions_size, actual.positions_size);\n        assert_size_eq(expected.fast_size, actual.fast_size);\n    }\n\n    fn test_json_fields_metadata(expanded_dots: bool, one_segment: bool) {\n        use pretty_assertions::assert_eq;\n        let mut schema_builder = Schema::builder();\n        let json_options: JsonObjectOptions =\n            JsonObjectOptions::from(TEXT).set_fast(None).set_stored();\n        let json_options = if expanded_dots {\n            json_options.set_expand_dots_enabled()\n        } else {\n            json_options\n        };\n        schema_builder.add_json_field(\"json.confusing\", json_options.clone());\n        let json_field = schema_builder.add_json_field(\"json.shadow\", json_options.clone());\n        let json_field2 = schema_builder.add_json_field(\"json\", json_options.clone());\n        schema_builder.add_json_field(\"empty_json\", json_options);\n        let number_field = schema_builder.add_u64_field(\"numbers\", FAST);\n        schema_builder.add_u64_field(\"empty\", FAST | INDEXED | STORED);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        let json =\n            serde_json::json!({\"k8s.container.name\": \"a\", \"val\": \"a\", \"sub\": {\"a\": 1, \"b\": 1}});\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        let json =\n            serde_json::json!({\"k8s.container.name\": \"a\", \"val\": \"a\", \"suber\": {\"a\": 1, \"b\": 1}});\n        if !one_segment {\n            index_writer.commit().unwrap();\n        }\n        index_writer.add_document(doc!(json_field=>json)).unwrap();\n        let json = serde_json::json!({\"k8s.container.name\": \"a\", \"k8s.container.name\": \"a\", \"val\": \"a\", \"suber\": {\"a\": \"a\", \"b\": 1}});\n        index_writer\n            .add_document(doc!(number_field => 50u64, json_field=>json, json_field2=>json!({\"shadow\": {\"val\": \"a\"}})))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 3);\n\n        let fields_metadata = index.fields_metadata().unwrap();\n\n        let expected_fields = &[\n            FieldMetadata {\n                field_name: \"empty\".to_string(),\n                stored: true,\n                typ: Type::U64,\n                term_dictionary_size: Some(0u64.into()),\n                fast_size: Some(1u64.into()),\n                postings_size: Some(0u64.into()),\n                positions_size: Some(0u64.into()),\n            },\n            FieldMetadata {\n                field_name: if expanded_dots {\n                    \"json.shadow.k8s.container.name\".to_string()\n                } else {\n                    \"json.shadow.k8s\\\\.container\\\\.name\".to_string()\n                },\n                stored: true,\n                typ: Type::Str,\n                term_dictionary_size: Some(1u64.into()),\n                fast_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.sub.a\".to_string(),\n                typ: Type::I64,\n                stored: true,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.sub.b\".to_string(),\n                typ: Type::I64,\n                stored: true,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.suber.a\".to_string(),\n                stored: true,\n                typ: Type::I64,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.suber.a\".to_string(),\n                typ: Type::Str,\n                stored: true,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.suber.b\".to_string(),\n                typ: Type::I64,\n                stored: true,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"json.shadow.val\".to_string(),\n                typ: Type::Str,\n                stored: true,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: Some(1u64.into()),\n                postings_size: Some(1u64.into()),\n                positions_size: Some(1u64.into()),\n            },\n            FieldMetadata {\n                field_name: \"numbers\".to_string(),\n                stored: false,\n                typ: Type::U64,\n                fast_size: Some(1u64.into()),\n                term_dictionary_size: None,\n                postings_size: None,\n                positions_size: None,\n            },\n        ];\n        assert_eq!(fields_metadata.len(), expected_fields.len());\n        for (expected, value) in expected_fields.iter().zip(fields_metadata.iter()) {\n            assert_field_metadata_eq_but_ignore_field_size(expected, value);\n        }\n        let query_parser = QueryParser::for_index(&index, vec![]);\n        // Test if returned field name can be queried\n        for indexed_field in fields_metadata.iter().filter(|meta| meta.is_indexed()) {\n            let val = if indexed_field.typ == Type::Str {\n                \"a\"\n            } else {\n                \"1\"\n            };\n            let query_str = &format!(\"{}:{}\", indexed_field.field_name, val);\n            let query = query_parser.parse_query(query_str).unwrap();\n            let count_docs = searcher\n                .search(&*query, &TopDocs::with_limit(2).order_by_score())\n                .unwrap();\n            if indexed_field.field_name.contains(\"empty\") || indexed_field.typ == Type::Json {\n                assert_eq!(count_docs.len(), 0);\n            } else {\n                assert!(!count_docs.is_empty(), \"{}\", indexed_field.field_name);\n            }\n        }\n        // Test if returned field name can be used for aggregation\n        for fast_field in fields_metadata\n            .iter()\n            .filter(|field_metadata| field_metadata.is_fast())\n        {\n            let agg_req_str = json!(\n            {\n              \"termagg\": {\n                \"terms\": {\n                  \"field\": fast_field.field_name,\n                }\n              }\n            });\n\n            let agg_req: Aggregations = serde_json::from_value(agg_req_str).unwrap();\n            let collector = AggregationCollector::from_aggs(agg_req, Default::default());\n            let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n            let res = serde_json::to_value(agg_res).unwrap();\n            if !fast_field.field_name.contains(\"empty\") && fast_field.typ != Type::Json {\n                assert!(\n                    !res[\"termagg\"][\"buckets\"].as_array().unwrap().is_empty(),\n                    \"{}\",\n                    fast_field.field_name\n                );\n            }\n        }\n    }\n\n    #[test]\n    fn test_json_field_shadowing_field_name_bug() {\n        /// This test is only there to display a bug on addressing a field if it gets shadowed\n        /// The issues only occurs if the field name that shadows contains a dot.\n        ///\n        /// Happens independently of the `expand_dots` option. Since that option does not\n        /// affect the field name itself.\n        use pretty_assertions::assert_eq;\n        let mut schema_builder = Schema::builder();\n        let json_options: JsonObjectOptions =\n            JsonObjectOptions::from(TEXT).set_fast(None).set_stored();\n        // let json_options = json_options.set_expand_dots_enabled();\n        let json_field_shadow = schema_builder.add_json_field(\"json.shadow\", json_options.clone());\n        let json_field = schema_builder.add_json_field(\"json\", json_options.clone());\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(\n                doc!(json_field_shadow=>json!({\"val\": \"b\"}), json_field=>json!({\"shadow\": {\"val\": \"a\"}})),\n            )\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n\n        let searcher = reader.searcher();\n\n        let fields_and_vals = [\n            (\"json.shadow\\u{1}val\".to_string(), \"a\"), // Succeeds\n            //(\"json.shadow.val\".to_string(), \"a\"),   // Fails\n            (\"json.shadow.val\".to_string(), \"b\"),\n        ];\n\n        let query_parser = QueryParser::for_index(&index, vec![]);\n        // Test if field name can be queried\n        for (indexed_field, val) in fields_and_vals.iter() {\n            let query_str = &format!(\"{indexed_field}:{val}\");\n            let query = query_parser.parse_query(query_str).unwrap();\n            let count_docs = searcher\n                .search(&*query, &TopDocs::with_limit(2).order_by_score())\n                .unwrap();\n            assert!(!count_docs.is_empty(), \"{indexed_field}:{val}\");\n        }\n        // Test if field name can be used for aggregation\n        for (field_name, val) in fields_and_vals.iter() {\n            let agg_req_str = json!(\n            {\n              \"termagg\": {\n                \"terms\": {\n                  \"field\": field_name,\n                }\n              }\n            });\n\n            let agg_req: Aggregations = serde_json::from_value(agg_req_str).unwrap();\n            let collector = AggregationCollector::from_aggs(agg_req, Default::default());\n            let agg_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();\n            let res = serde_json::to_value(agg_res).unwrap();\n            assert_eq!(\n                res[\"termagg\"][\"buckets\"].as_array().unwrap()[0][\"key\"]\n                    .as_str()\n                    .unwrap(),\n                *val,\n                \"{}\",\n                field_name\n            );\n        }\n    }\n}\n"
  },
  {
    "path": "src/indexer/operation.rs",
    "content": "use crate::query::Weight;\nuse crate::schema::document::Document;\nuse crate::schema::{TantivyDocument, Term};\nuse crate::Opstamp;\n\n/// Timestamped Delete operation.\npub struct DeleteOperation {\n    /// Operation stamp.\n    /// It is used to check whether the delete operation\n    /// applies to an added document operation.\n    pub opstamp: Opstamp,\n    /// Weight is used to define the set of documents to be deleted.\n    pub target: Box<dyn Weight>,\n}\n\n/// Timestamped Add operation.\n#[derive(Eq, PartialEq, Debug)]\npub struct AddOperation<D: Document = TantivyDocument> {\n    /// Operation stamp.\n    pub opstamp: Opstamp,\n    /// Document to be added.\n    pub document: D,\n}\n\n/// UserOperation is an enum type that encapsulates other operation types.\n#[derive(Eq, PartialEq, Debug)]\npub enum UserOperation<D: Document = TantivyDocument> {\n    /// Add operation\n    Add(D),\n    /// Delete operation\n    Delete(Term),\n}\n"
  },
  {
    "path": "src/indexer/path_to_unordered_id.rs",
    "content": "use fnv::FnvHashMap;\n\n/// `Field` is represented by an unsigned 32-bit integer type.\n/// The schema holds the mapping between field names and `Field` objects.\n#[derive(Copy, Default, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash)]\npub struct OrderedPathId(u32);\n\nimpl OrderedPathId {\n    /// Create a new field object for the given PathId.\n    pub const fn from_ordered_id(field_id: u32) -> OrderedPathId {\n        OrderedPathId(field_id)\n    }\n\n    /// Returns a u32 identifying uniquely a path within a schema.\n    pub const fn path_id(self) -> u32 {\n        self.0\n    }\n}\nimpl From<u32> for OrderedPathId {\n    fn from(id: u32) -> Self {\n        Self(id)\n    }\n}\n\n#[derive(Default)]\npub(crate) struct PathToUnorderedId {\n    map: FnvHashMap<String, u32>,\n}\n\nimpl PathToUnorderedId {\n    #[inline]\n    pub(crate) fn get_or_allocate_unordered_id(&mut self, path: &str) -> u32 {\n        if let Some(id) = self.map.get(path) {\n            return *id;\n        }\n        self.insert_new_path(path)\n    }\n    #[cold]\n    fn insert_new_path(&mut self, path: &str) -> u32 {\n        let next_id = self.map.len() as u32;\n        let new_path = path.to_string();\n        self.map.insert(new_path, next_id);\n        next_id\n    }\n\n    /// Returns ids which reflect the lexical order of the paths.\n    ///\n    /// The returned vec can be indexed with the unordered id to get the ordered id.\n    pub(crate) fn unordered_id_to_ordered_id(&self) -> Vec<OrderedPathId> {\n        let mut sorted_ids: Vec<(&str, &u32)> =\n            self.map.iter().map(|(k, v)| (k.as_str(), v)).collect();\n        sorted_ids.sort_unstable_by_key(|(path, _)| *path);\n        let mut result = vec![OrderedPathId::default(); sorted_ids.len()];\n        for (ordered, unordered) in sorted_ids.iter().map(|(_k, v)| v).enumerate() {\n            result[**unordered as usize] = OrderedPathId::from_ordered_id(ordered as u32);\n        }\n        result\n    }\n\n    /// Returns the paths so they can be queried by the ordered id (which is the index).\n    pub(crate) fn ordered_id_to_path(&self) -> Vec<&str> {\n        let mut paths = self.map.keys().map(String::as_str).collect::<Vec<_>>();\n        paths.sort_unstable();\n        paths\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn path_to_unordered_test() {\n        let mut path_to_id = PathToUnorderedId::default();\n        let terms = vec![\"b\", \"a\", \"b\", \"c\"];\n        let ids = terms\n            .iter()\n            .map(|term| path_to_id.get_or_allocate_unordered_id(term))\n            .collect::<Vec<u32>>();\n        assert_eq!(ids, vec![0, 1, 0, 2]);\n        let ordered_ids = ids\n            .iter()\n            .map(|id| path_to_id.unordered_id_to_ordered_id()[*id as usize])\n            .collect::<Vec<OrderedPathId>>();\n        assert_eq!(ordered_ids, vec![1.into(), 0.into(), 1.into(), 2.into()]);\n        // Fetch terms\n        let terms_fetched = ordered_ids\n            .iter()\n            .map(|id| path_to_id.ordered_id_to_path()[id.path_id() as usize])\n            .collect::<Vec<&str>>();\n        assert_eq!(terms_fetched, terms);\n    }\n}\n"
  },
  {
    "path": "src/indexer/prepared_commit.rs",
    "content": "use super::IndexWriter;\nuse crate::schema::document::Document;\nuse crate::{FutureResult, Opstamp, TantivyDocument};\n\n/// A prepared commit\npub struct PreparedCommit<'a, D: Document = TantivyDocument> {\n    index_writer: &'a mut IndexWriter<D>,\n    payload: Option<String>,\n    opstamp: Opstamp,\n}\n\nimpl<'a, D: Document> PreparedCommit<'a, D> {\n    pub(crate) fn new(index_writer: &'a mut IndexWriter<D>, opstamp: Opstamp) -> Self {\n        Self {\n            index_writer,\n            payload: None,\n            opstamp,\n        }\n    }\n\n    /// Returns the opstamp associated with the prepared commit.\n    pub fn opstamp(&self) -> Opstamp {\n        self.opstamp\n    }\n\n    /// Adds an arbitrary payload to the commit.\n    pub fn set_payload(&mut self, payload: &str) {\n        self.payload = Some(payload.to_string())\n    }\n\n    /// Rollbacks any change.\n    pub fn abort(self) -> crate::Result<Opstamp> {\n        self.index_writer.rollback()\n    }\n\n    /// Proceeds to commit.\n    /// See `.commit_future()`.\n    pub fn commit(self) -> crate::Result<Opstamp> {\n        self.commit_future().wait()\n    }\n\n    /// Proceeds to commit.\n    ///\n    /// Unfortunately, contrary to what `PrepareCommit` may suggests,\n    /// this operation is not at all really light.\n    /// At this point deletes have not been flushed yet.\n    pub fn commit_future(self) -> FutureResult<Opstamp> {\n        info!(\"committing {}\", self.opstamp);\n        self.index_writer\n            .segment_updater()\n            .schedule_commit(self.opstamp, self.payload)\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_entry.rs",
    "content": "use std::fmt;\n\nuse common::BitSet;\n\nuse crate::index::{SegmentId, SegmentMeta};\nuse crate::indexer::delete_queue::DeleteCursor;\n\n/// A segment entry describes the state of\n/// a given segment, at a given instant.\n///\n/// In addition to segment `meta`,\n/// it contains a few transient states\n/// - `alive_bitset` is a bitset describing documents that were alive during the commit itself.\n/// - `delete_cursor` is the position in the delete queue. Deletes happening before the cursor are\n///   reflected either in the .del file or in the `alive_bitset`.\n#[derive(Clone)]\npub struct SegmentEntry {\n    meta: SegmentMeta,\n    alive_bitset: Option<BitSet>,\n    delete_cursor: DeleteCursor,\n}\n\nimpl SegmentEntry {\n    /// Create a new `SegmentEntry`\n    pub fn new(\n        segment_meta: SegmentMeta,\n        delete_cursor: DeleteCursor,\n        alive_bitset: Option<BitSet>,\n    ) -> SegmentEntry {\n        SegmentEntry {\n            meta: segment_meta,\n            alive_bitset,\n            delete_cursor,\n        }\n    }\n\n    /// Return a reference to the segment entry deleted bitset.\n    ///\n    /// `DocId` in this bitset are flagged as deleted.\n    pub fn alive_bitset(&self) -> Option<&BitSet> {\n        self.alive_bitset.as_ref()\n    }\n\n    /// Set the `SegmentMeta` for this segment.\n    pub fn set_meta(&mut self, segment_meta: SegmentMeta) {\n        self.meta = segment_meta;\n    }\n\n    /// Return a reference to the segment_entry's delete cursor\n    pub fn delete_cursor(&mut self) -> &mut DeleteCursor {\n        &mut self.delete_cursor\n    }\n\n    /// Returns the segment id.\n    pub fn segment_id(&self) -> SegmentId {\n        self.meta.id()\n    }\n\n    /// Accessor to the `SegmentMeta`\n    pub fn meta(&self) -> &SegmentMeta {\n        &self.meta\n    }\n}\n\nimpl fmt::Debug for SegmentEntry {\n    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {\n        write!(formatter, \"SegmentEntry({:?})\", self.meta)\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_manager.rs",
    "content": "use std::collections::hash_set::HashSet;\nuse std::fmt::{self, Debug, Formatter};\nuse std::sync::{RwLock, RwLockReadGuard, RwLockWriteGuard};\n\nuse super::segment_register::SegmentRegister;\nuse crate::error::TantivyError;\nuse crate::index::{SegmentId, SegmentMeta};\nuse crate::indexer::delete_queue::DeleteCursor;\nuse crate::indexer::SegmentEntry;\n\n#[derive(Default)]\nstruct SegmentRegisters {\n    uncommitted: SegmentRegister,\n    committed: SegmentRegister,\n}\n\n#[derive(PartialEq, Eq)]\npub(crate) enum SegmentsStatus {\n    Committed,\n    Uncommitted,\n}\n\nimpl SegmentRegisters {\n    /// Check if all the segments are committed or uncommitted.\n    ///\n    /// If some segment is missing or segments are in a different state (this should not happen\n    /// if tantivy is used correctly), returns `None`.\n    fn segments_status(&self, segment_ids: &[SegmentId]) -> Option<SegmentsStatus> {\n        if self.uncommitted.contains_all(segment_ids) {\n            Some(SegmentsStatus::Uncommitted)\n        } else if self.committed.contains_all(segment_ids) {\n            Some(SegmentsStatus::Committed)\n        } else {\n            warn!(\n                \"segment_ids: {:?}, committed_ids: {:?}, uncommitted_ids {:?}\",\n                segment_ids,\n                self.committed.segment_ids(),\n                self.uncommitted.segment_ids()\n            );\n            None\n        }\n    }\n}\n\n/// The segment manager stores the list of segments\n/// as well as their state.\n///\n/// It guarantees the atomicity of the\n/// changes (merges especially)\n#[derive(Default)]\npub struct SegmentManager {\n    registers: RwLock<SegmentRegisters>,\n}\n\nimpl Debug for SegmentManager {\n    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {\n        let lock = self.read();\n        write!(\n            f,\n            \"{{ uncommitted: {:?}, committed: {:?} }}\",\n            lock.uncommitted, lock.committed\n        )\n    }\n}\n\nimpl SegmentManager {\n    pub fn from_segments(\n        segment_metas: Vec<SegmentMeta>,\n        delete_cursor: &DeleteCursor,\n    ) -> SegmentManager {\n        SegmentManager {\n            registers: RwLock::new(SegmentRegisters {\n                uncommitted: SegmentRegister::default(),\n                committed: SegmentRegister::new(segment_metas, delete_cursor),\n            }),\n        }\n    }\n\n    pub fn get_mergeable_segments(\n        &self,\n        in_merge_segment_ids: &HashSet<SegmentId>,\n    ) -> (Vec<SegmentMeta>, Vec<SegmentMeta>) {\n        let registers_lock = self.read();\n        (\n            registers_lock\n                .committed\n                .get_mergeable_segments(in_merge_segment_ids),\n            registers_lock\n                .uncommitted\n                .get_mergeable_segments(in_merge_segment_ids),\n        )\n    }\n    /// Returns all of the segment entries (committed or uncommitted)\n    pub fn segment_entries(&self) -> Vec<SegmentEntry> {\n        let registers_lock = self.read();\n        let mut segment_entries = registers_lock.uncommitted.segment_entries();\n        segment_entries.extend(registers_lock.committed.segment_entries());\n        segment_entries\n    }\n\n    // Lock poisoning should never happen :\n    // The lock is acquired and released within this class,\n    // and the operations cannot panic.\n    fn read(&self) -> RwLockReadGuard<'_, SegmentRegisters> {\n        self.registers\n            .read()\n            .expect(\"Failed to acquire read lock on SegmentManager.\")\n    }\n\n    fn write(&self) -> RwLockWriteGuard<'_, SegmentRegisters> {\n        self.registers\n            .write()\n            .expect(\"Failed to acquire write lock on SegmentManager.\")\n    }\n\n    /// Deletes all empty segments\n    fn remove_empty_segments(&self) {\n        let mut registers_lock = self.write();\n        registers_lock\n            .committed\n            .segment_entries()\n            .iter()\n            .filter(|segment| segment.meta().num_docs() == 0)\n            .for_each(|segment| {\n                registers_lock\n                    .committed\n                    .remove_segment(&segment.segment_id())\n            });\n    }\n\n    pub(crate) fn remove_all_segments(&self) {\n        let mut registers_lock = self.write();\n        registers_lock.committed.clear();\n        registers_lock.uncommitted.clear();\n    }\n\n    pub fn commit(&self, segment_entries: Vec<SegmentEntry>) {\n        let mut registers_lock = self.write();\n        registers_lock.committed.clear();\n        registers_lock.uncommitted.clear();\n        for segment_entry in segment_entries {\n            registers_lock.committed.add_segment_entry(segment_entry);\n        }\n    }\n\n    /// Marks a list of segments as in merge.\n    ///\n    /// Returns an error if some segments are missing, or if\n    /// the `segment_ids` are not either all committed or all\n    /// uncommitted.\n    pub fn start_merge(&self, segment_ids: &[SegmentId]) -> crate::Result<Vec<SegmentEntry>> {\n        let registers_lock = self.read();\n        let mut segment_entries = vec![];\n        if registers_lock.uncommitted.contains_all(segment_ids) {\n            for segment_id in segment_ids {\n                let segment_entry = registers_lock.uncommitted.get(segment_id).expect(\n                    \"Segment id not found {}. Should never happen because of the contains all \\\n                     if-block.\",\n                );\n                segment_entries.push(segment_entry);\n            }\n        } else if registers_lock.committed.contains_all(segment_ids) {\n            for segment_id in segment_ids {\n                let segment_entry = registers_lock.committed.get(segment_id).expect(\n                    \"Segment id not found {}. Should never happen because of the contains all \\\n                     if-block.\",\n                );\n                segment_entries.push(segment_entry);\n            }\n        } else {\n            let error_msg = \"Merge operation sent for segments that are not all uncommitted or \\\n                             committed.\"\n                .to_string();\n            return Err(TantivyError::InvalidArgument(error_msg));\n        }\n\n        Ok(segment_entries)\n    }\n\n    pub fn add_segment(&self, segment_entry: SegmentEntry) {\n        let mut registers_lock = self.write();\n        registers_lock.uncommitted.add_segment_entry(segment_entry);\n    }\n    // Replace a list of segments for their equivalent merged segment.\n    //\n    // Returns true if these segments are committed, false if the merge segments are uncommitted.\n    pub(crate) fn end_merge(\n        &self,\n        before_merge_segment_ids: &[SegmentId],\n        after_merge_segment_entry: Option<SegmentEntry>,\n    ) -> crate::Result<SegmentsStatus> {\n        let mut registers_lock = self.write();\n        let segments_status = registers_lock\n            .segments_status(before_merge_segment_ids)\n            .ok_or_else(|| {\n                warn!(\"couldn't find segment in SegmentManager\");\n                crate::TantivyError::InvalidArgument(\n                    \"The segments that were merged could not be found in the SegmentManager. This \\\n                     is not necessarily a bug, and can happen after a rollback for instance.\"\n                        .to_string(),\n                )\n            })?;\n\n        let target_register: &mut SegmentRegister = match segments_status {\n            SegmentsStatus::Uncommitted => &mut registers_lock.uncommitted,\n            SegmentsStatus::Committed => &mut registers_lock.committed,\n        };\n        for segment_id in before_merge_segment_ids {\n            target_register.remove_segment(segment_id);\n        }\n        if let Some(entry) = after_merge_segment_entry {\n            target_register.add_segment_entry(entry);\n        }\n        Ok(segments_status)\n    }\n\n    pub fn committed_segment_metas(&self) -> Vec<SegmentMeta> {\n        self.remove_empty_segments();\n        let registers_lock = self.read();\n        registers_lock.committed.segment_metas()\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_register.rs",
    "content": "use std::collections::{HashMap, HashSet};\nuse std::fmt::{self, Debug, Display, Formatter};\n\nuse crate::index::{SegmentId, SegmentMeta};\nuse crate::indexer::delete_queue::DeleteCursor;\nuse crate::indexer::segment_entry::SegmentEntry;\n\n/// The segment register keeps track\n/// of the list of segment, their size as well\n/// as the state they are in.\n///\n/// It is consumed by indexes to get the list of\n/// segments that are currently searchable,\n/// and by the index merger to identify\n/// merge candidates.\n#[derive(Default)]\npub struct SegmentRegister {\n    segment_states: HashMap<SegmentId, SegmentEntry>,\n}\n\nimpl Debug for SegmentRegister {\n    fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), fmt::Error> {\n        write!(f, \"SegmentRegister(\")?;\n        for k in self.segment_states.keys() {\n            write!(f, \"{}, \", k.short_uuid_string())?;\n        }\n        write!(f, \")\")?;\n        Ok(())\n    }\n}\nimpl Display for SegmentRegister {\n    fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), fmt::Error> {\n        write!(f, \"SegmentRegister(\")?;\n        for k in self.segment_states.keys() {\n            write!(f, \"{}, \", k.short_uuid_string())?;\n        }\n        write!(f, \")\")?;\n        Ok(())\n    }\n}\n\nimpl SegmentRegister {\n    pub fn clear(&mut self) {\n        self.segment_states.clear();\n    }\n\n    pub fn get_mergeable_segments(\n        &self,\n        in_merge_segment_ids: &HashSet<SegmentId>,\n    ) -> Vec<SegmentMeta> {\n        self.segment_states\n            .values()\n            .filter(|segment_entry| !in_merge_segment_ids.contains(&segment_entry.segment_id()))\n            .map(|segment_entry| segment_entry.meta().clone())\n            .collect()\n    }\n\n    pub fn segment_ids(&self) -> Vec<SegmentId> {\n        self.segment_states.keys().cloned().collect()\n    }\n\n    pub fn segment_entries(&self) -> Vec<SegmentEntry> {\n        self.segment_states.values().cloned().collect()\n    }\n\n    pub fn segment_metas(&self) -> Vec<SegmentMeta> {\n        self.segment_states\n            .values()\n            .map(|segment_entry| segment_entry.meta().clone())\n            .collect()\n    }\n\n    pub fn contains_all(&self, segment_ids: &[SegmentId]) -> bool {\n        segment_ids\n            .iter()\n            .all(|segment_id| self.segment_states.contains_key(segment_id))\n    }\n\n    pub fn add_segment_entry(&mut self, segment_entry: SegmentEntry) {\n        let segment_id = segment_entry.segment_id();\n        self.segment_states.insert(segment_id, segment_entry);\n    }\n\n    pub fn remove_segment(&mut self, segment_id: &SegmentId) {\n        self.segment_states.remove(segment_id);\n    }\n\n    pub fn get(&self, segment_id: &SegmentId) -> Option<SegmentEntry> {\n        self.segment_states.get(segment_id).cloned()\n    }\n\n    pub fn new(segment_metas: Vec<SegmentMeta>, delete_cursor: &DeleteCursor) -> SegmentRegister {\n        let mut segment_states = HashMap::new();\n        for segment_meta in segment_metas {\n            let segment_id = segment_meta.id();\n            let segment_entry = SegmentEntry::new(segment_meta, delete_cursor.clone(), None);\n            segment_states.insert(segment_id, segment_entry);\n        }\n        SegmentRegister { segment_states }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::index::SegmentMetaInventory;\n    use crate::indexer::delete_queue::*;\n\n    fn segment_ids(segment_register: &SegmentRegister) -> Vec<SegmentId> {\n        segment_register\n            .segment_metas()\n            .into_iter()\n            .map(|segment_meta| segment_meta.id())\n            .collect()\n    }\n\n    #[test]\n    fn test_segment_register() {\n        let inventory = SegmentMetaInventory::default();\n        let delete_queue = DeleteQueue::default();\n\n        let mut segment_register = SegmentRegister::default();\n        let segment_id_a = SegmentId::generate_random();\n        let segment_id_b = SegmentId::generate_random();\n        let segment_id_merged = SegmentId::generate_random();\n\n        {\n            let segment_meta = inventory.new_segment_meta(segment_id_a, 0u32);\n            let segment_entry = SegmentEntry::new(segment_meta, delete_queue.cursor(), None);\n            segment_register.add_segment_entry(segment_entry);\n        }\n        assert_eq!(segment_ids(&segment_register), vec![segment_id_a]);\n        {\n            let segment_meta = inventory.new_segment_meta(segment_id_b, 0u32);\n            let segment_entry = SegmentEntry::new(segment_meta, delete_queue.cursor(), None);\n            segment_register.add_segment_entry(segment_entry);\n        }\n        segment_register.remove_segment(&segment_id_a);\n        segment_register.remove_segment(&segment_id_b);\n        {\n            let segment_meta_merged = inventory.new_segment_meta(segment_id_merged, 0u32);\n            let segment_entry = SegmentEntry::new(segment_meta_merged, delete_queue.cursor(), None);\n            segment_register.add_segment_entry(segment_entry);\n        }\n        assert_eq!(segment_ids(&segment_register), vec![segment_id_merged]);\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_serializer.rs",
    "content": "use common::TerminatingWrite;\n\nuse crate::directory::WritePtr;\nuse crate::fieldnorm::FieldNormsSerializer;\nuse crate::index::{Segment, SegmentComponent};\nuse crate::postings::InvertedIndexSerializer;\nuse crate::store::StoreWriter;\n\n/// Segment serializer is in charge of laying out on disk\n/// the data accumulated and sorted by the `SegmentWriter`.\npub struct SegmentSerializer {\n    segment: Segment,\n    pub(crate) store_writer: StoreWriter,\n    fast_field_write: WritePtr,\n    fieldnorms_serializer: Option<FieldNormsSerializer>,\n    postings_serializer: InvertedIndexSerializer,\n}\n\nimpl SegmentSerializer {\n    /// Creates a new `SegmentSerializer`.\n    pub fn for_segment(mut segment: Segment) -> crate::Result<SegmentSerializer> {\n        let settings = segment.index().settings().clone();\n        let store_writer = {\n            let store_write = segment.open_write(SegmentComponent::Store)?;\n            StoreWriter::new(\n                store_write,\n                settings.docstore_compression,\n                settings.docstore_blocksize,\n                settings.docstore_compress_dedicated_thread,\n            )?\n        };\n\n        let fast_field_write = segment.open_write(SegmentComponent::FastFields)?;\n\n        let fieldnorms_write = segment.open_write(SegmentComponent::FieldNorms)?;\n        let fieldnorms_serializer = FieldNormsSerializer::from_write(fieldnorms_write)?;\n\n        let postings_serializer = InvertedIndexSerializer::open(&mut segment)?;\n        Ok(SegmentSerializer {\n            segment,\n            store_writer,\n            fast_field_write,\n            fieldnorms_serializer: Some(fieldnorms_serializer),\n            postings_serializer,\n        })\n    }\n\n    /// The memory used (inclusive childs)\n    pub fn mem_usage(&self) -> usize {\n        self.store_writer.mem_usage()\n    }\n\n    pub fn segment(&self) -> &Segment {\n        &self.segment\n    }\n\n    /// Accessor to the `PostingsSerializer`.\n    pub fn get_postings_serializer(&mut self) -> &mut InvertedIndexSerializer {\n        &mut self.postings_serializer\n    }\n\n    /// Accessor to the `FastFieldSerializer`.\n    pub fn get_fast_field_write(&mut self) -> &mut WritePtr {\n        &mut self.fast_field_write\n    }\n\n    /// Extract the field norm serializer.\n    ///\n    /// Note the fieldnorms serializer can only be extracted once.\n    pub fn extract_fieldnorms_serializer(&mut self) -> Option<FieldNormsSerializer> {\n        self.fieldnorms_serializer.take()\n    }\n\n    /// Accessor to the `StoreWriter`.\n    pub fn get_store_writer(&mut self) -> &mut StoreWriter {\n        &mut self.store_writer\n    }\n\n    /// Finalize the segment serialization.\n    pub fn close(mut self) -> crate::Result<()> {\n        if let Some(fieldnorms_serializer) = self.extract_fieldnorms_serializer() {\n            fieldnorms_serializer.close()?;\n        }\n        self.fast_field_write.terminate()?;\n        self.postings_serializer.close()?;\n        self.store_writer.close()?;\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_updater.rs",
    "content": "use std::any::Any;\nuse std::borrow::BorrowMut;\nuse std::collections::HashSet;\nuse std::io::Write;\nuse std::ops::Deref;\nuse std::path::PathBuf;\nuse std::sync::atomic::{AtomicBool, Ordering};\nuse std::sync::{Arc, RwLock};\n\nuse rayon::{ThreadPool, ThreadPoolBuilder};\n\nuse super::segment_manager::SegmentManager;\nuse crate::core::META_FILEPATH;\nuse crate::directory::{Directory, DirectoryClone, GarbageCollectionResult};\nuse crate::fastfield::AliveBitSet;\nuse crate::index::{Index, IndexMeta, IndexSettings, Segment, SegmentId, SegmentMeta};\nuse crate::indexer::delete_queue::DeleteCursor;\nuse crate::indexer::index_writer::advance_deletes;\nuse crate::indexer::merge_operation::MergeOperationInventory;\nuse crate::indexer::merger::IndexMerger;\nuse crate::indexer::segment_manager::SegmentsStatus;\nuse crate::indexer::stamper::Stamper;\nuse crate::indexer::{\n    DefaultMergePolicy, MergeCandidate, MergeOperation, MergePolicy, SegmentEntry,\n    SegmentSerializer,\n};\nuse crate::{FutureResult, Opstamp, TantivyError};\n\nconst PANIC_CAUGHT: &str = \"Panic caught in merge thread\";\n\n/// Save the index meta file.\n/// This operation is atomic:\n/// Either\n/// - it fails, in which case an error is returned, and the `meta.json` remains untouched,\n/// - it success, and `meta.json` is written and flushed.\n///\n/// This method is not part of tantivy's public API\npub(crate) fn save_metas(metas: &IndexMeta, directory: &dyn Directory) -> crate::Result<()> {\n    info!(\"save metas\");\n    let mut buffer = serde_json::to_vec_pretty(metas)?;\n    // Just adding a new line at the end of the buffer.\n    writeln!(&mut buffer)?;\n    crate::fail_point!(\"save_metas\", |msg| Err(crate::TantivyError::from(\n        std::io::Error::new(\n            std::io::ErrorKind::Other,\n            msg.unwrap_or_else(|| \"Undefined\".to_string())\n        )\n    )));\n    directory.sync_directory()?;\n    directory.atomic_write(&META_FILEPATH, &buffer[..])?;\n    debug!(\"Saved metas {:?}\", serde_json::to_string_pretty(&metas));\n    Ok(())\n}\n\n// The segment update runner is in charge of processing all\n//  of the `SegmentUpdate`s.\n//\n// All this processing happens on a single thread\n// consuming a common queue.\n//\n// We voluntarily pass a merge_operation ref to guarantee that\n// the merge_operation is alive during the process\n#[derive(Clone)]\npub(crate) struct SegmentUpdater(Arc<InnerSegmentUpdater>);\n\nimpl Deref for SegmentUpdater {\n    type Target = InnerSegmentUpdater;\n\n    #[inline]\n    fn deref(&self) -> &Self::Target {\n        &self.0\n    }\n}\n\nfn garbage_collect_files(\n    segment_updater: SegmentUpdater,\n) -> crate::Result<GarbageCollectionResult> {\n    info!(\"Running garbage collection\");\n    let mut index = segment_updater.index.clone();\n    index\n        .directory_mut()\n        .garbage_collect(move || segment_updater.list_files())\n}\n\n/// Merges a list of segments the list of segment givens in the `segment_entries`.\n/// This function happens in the calling thread and is computationally expensive.\nfn merge(\n    index: &Index,\n    mut segment_entries: Vec<SegmentEntry>,\n    target_opstamp: Opstamp,\n) -> crate::Result<Option<SegmentEntry>> {\n    let num_docs = segment_entries\n        .iter()\n        .map(|segment| segment.meta().num_docs() as u64)\n        .sum::<u64>();\n    if num_docs == 0 {\n        return Ok(None);\n    }\n\n    // first we need to apply deletes to our segment.\n    let merged_segment = index.new_segment();\n\n    // First we apply all of the delete to the merged segment, up to the target opstamp.\n    for segment_entry in &mut segment_entries {\n        let segment = index.segment(segment_entry.meta().clone());\n        advance_deletes(segment, segment_entry, target_opstamp)?;\n    }\n\n    let delete_cursor = segment_entries[0].delete_cursor().clone();\n\n    let segments: Vec<Segment> = segment_entries\n        .iter()\n        .map(|segment_entry| index.segment(segment_entry.meta().clone()))\n        .collect();\n\n    // An IndexMerger is like a \"view\" of our merged segments.\n    let merger: IndexMerger = IndexMerger::open(index.schema(), &segments[..])?;\n\n    // ... we just serialize this index merger in our new segment to merge the segments.\n    let segment_serializer = SegmentSerializer::for_segment(merged_segment.clone())?;\n\n    let num_docs = merger.write(segment_serializer)?;\n\n    let merged_segment_id = merged_segment.id();\n\n    let segment_meta = index.new_segment_meta(merged_segment_id, num_docs);\n    Ok(Some(SegmentEntry::new(segment_meta, delete_cursor, None)))\n}\n\n/// Advanced: Merges a list of segments from different indices in a new index.\n///\n/// Returns `TantivyError` if the indices list is empty or their\n/// schemas don't match.\n///\n/// `output_directory`: is assumed to be empty.\n///\n/// # Warning\n/// This function does NOT check or take the `IndexWriter` is running. It is not\n/// meant to work if you have an `IndexWriter` running for the origin indices, or\n/// the destination `Index`.\n#[doc(hidden)]\npub fn merge_indices<T: Into<Box<dyn Directory>>>(\n    indices: &[Index],\n    output_directory: T,\n) -> crate::Result<Index> {\n    if indices.is_empty() {\n        // If there are no indices to merge, there is no need to do anything.\n        return Err(crate::TantivyError::InvalidArgument(\n            \"No indices given to merge\".to_string(),\n        ));\n    }\n\n    let target_settings = indices[0].settings().clone();\n\n    // let's check that all of the indices have the same index settings\n    if indices\n        .iter()\n        .skip(1)\n        .any(|index| index.settings() != &target_settings)\n    {\n        return Err(crate::TantivyError::InvalidArgument(\n            \"Attempt to merge indices with different index_settings\".to_string(),\n        ));\n    }\n\n    let mut segments: Vec<Segment> = Vec::new();\n    for index in indices {\n        segments.extend(index.searchable_segments()?);\n    }\n\n    let non_filter = segments.iter().map(|_| None).collect::<Vec<_>>();\n    merge_filtered_segments(&segments, target_settings, non_filter, output_directory)\n}\n\n/// Advanced: Merges a list of segments from different indices in a new index.\n/// Additional you can provide a delete bitset for each segment to ignore doc_ids.\n///\n/// Returns `TantivyError` if the indices list is empty or their\n/// schemas don't match.\n///\n/// `output_directory`: is assumed to be empty.\n///\n/// # Warning\n/// This function does NOT check or take the `IndexWriter` is running. It is not\n/// meant to work if you have an `IndexWriter` running for the origin indices, or\n/// the destination `Index`.\n#[doc(hidden)]\npub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(\n    segments: &[Segment],\n    target_settings: IndexSettings,\n    filter_doc_ids: Vec<Option<AliveBitSet>>,\n    output_directory: T,\n) -> crate::Result<Index> {\n    if segments.is_empty() {\n        // If there are no indices to merge, there is no need to do anything.\n        return Err(crate::TantivyError::InvalidArgument(\n            \"No segments given to merge\".to_string(),\n        ));\n    }\n\n    let target_schema = segments[0].schema();\n\n    // let's check that all of the indices have the same schema\n    if segments\n        .iter()\n        .skip(1)\n        .any(|index| index.schema() != target_schema)\n    {\n        return Err(crate::TantivyError::InvalidArgument(\n            \"Attempt to merge different schema indices\".to_string(),\n        ));\n    }\n\n    let mut merged_index = Index::create(\n        output_directory,\n        target_schema.clone(),\n        target_settings.clone(),\n    )?;\n    let merged_segment = merged_index.new_segment();\n    let merged_segment_id = merged_segment.id();\n    let merger: IndexMerger =\n        IndexMerger::open_with_custom_alive_set(merged_index.schema(), segments, filter_doc_ids)?;\n    let segment_serializer = SegmentSerializer::for_segment(merged_segment)?;\n    let num_docs = merger.write(segment_serializer)?;\n\n    let segment_meta = merged_index.new_segment_meta(merged_segment_id, num_docs);\n\n    let stats = format!(\n        \"Segments Merge: [{}]\",\n        segments\n            .iter()\n            .fold(String::new(), |sum, current| format!(\n                \"{sum}{} \",\n                current.meta().id().uuid_string()\n            ))\n            .trim_end()\n    );\n\n    let index_meta = IndexMeta {\n        index_settings: target_settings, // index_settings of all segments should be the same\n        segments: vec![segment_meta],\n        schema: target_schema,\n        opstamp: 0u64,\n        payload: Some(stats),\n    };\n\n    // save the meta.json\n    save_metas(&index_meta, merged_index.directory_mut())?;\n\n    Ok(merged_index)\n}\n\npub(crate) struct InnerSegmentUpdater {\n    // we keep a copy of the current active IndexMeta to\n    // avoid loading the file every time we need it in the\n    // `SegmentUpdater`.\n    //\n    // This should be up to date as all update happen through\n    // the unique active `SegmentUpdater`.\n    active_index_meta: RwLock<Arc<IndexMeta>>,\n    pool: ThreadPool,\n    merge_thread_pool: ThreadPool,\n\n    index: Index,\n    segment_manager: SegmentManager,\n    merge_policy: RwLock<Arc<dyn MergePolicy>>,\n    killed: AtomicBool,\n    stamper: Stamper,\n    merge_operations: MergeOperationInventory,\n}\n\nimpl SegmentUpdater {\n    pub fn create(\n        index: Index,\n        stamper: Stamper,\n        delete_cursor: &DeleteCursor,\n        num_merge_threads: usize,\n    ) -> crate::Result<SegmentUpdater> {\n        let segments = index.searchable_segment_metas()?;\n        let segment_manager = SegmentManager::from_segments(segments, delete_cursor);\n        let pool = ThreadPoolBuilder::new()\n            .thread_name(|_| \"segment_updater\".to_string())\n            .num_threads(1)\n            .build()\n            .map_err(|_| {\n                crate::TantivyError::SystemError(\n                    \"Failed to spawn segment updater thread\".to_string(),\n                )\n            })?;\n        let merge_thread_pool = ThreadPoolBuilder::new()\n            .thread_name(|i| format!(\"merge_thread_{i}\"))\n            .num_threads(num_merge_threads)\n            .panic_handler(move |panic| {\n                // We don't print the panic content itself,\n                // it is already printed during the unwinding\n                if let Some(message) = panic.downcast_ref::<&str>() {\n                    if *message != PANIC_CAUGHT {\n                        error!(\"uncaught merge panic\")\n                    }\n                }\n            })\n            .build()\n            .map_err(|_| {\n                crate::TantivyError::SystemError(\n                    \"Failed to spawn segment merging thread\".to_string(),\n                )\n            })?;\n        let index_meta = index.load_metas()?;\n        Ok(SegmentUpdater(Arc::new(InnerSegmentUpdater {\n            active_index_meta: RwLock::new(Arc::new(index_meta)),\n            pool,\n            merge_thread_pool,\n            index,\n            segment_manager,\n            merge_policy: RwLock::new(Arc::new(DefaultMergePolicy::default())),\n            killed: AtomicBool::new(false),\n            stamper,\n            merge_operations: Default::default(),\n        })))\n    }\n\n    pub fn get_merge_policy(&self) -> Arc<dyn MergePolicy> {\n        self.merge_policy.read().unwrap().clone()\n    }\n\n    pub fn set_merge_policy(&self, merge_policy: Box<dyn MergePolicy>) {\n        let arc_merge_policy = Arc::from(merge_policy);\n        *self.merge_policy.write().unwrap() = arc_merge_policy;\n    }\n\n    fn schedule_task<T: 'static + Send, F: FnOnce() -> crate::Result<T> + 'static + Send>(\n        &self,\n        task: F,\n    ) -> FutureResult<T> {\n        if !self.is_alive() {\n            return crate::TantivyError::SystemError(\"Segment updater killed\".to_string()).into();\n        }\n        let (scheduled_result, sender) = FutureResult::create(\n            \"A segment_updater future did not succeed. This should never happen.\",\n        );\n        self.pool.spawn(|| {\n            let task_result = task();\n            let _ = sender.send(task_result);\n        });\n        scheduled_result\n    }\n\n    pub fn schedule_add_segment(&self, segment_entry: SegmentEntry) -> FutureResult<()> {\n        let segment_updater = self.clone();\n        self.schedule_task(move || {\n            segment_updater.segment_manager.add_segment(segment_entry);\n            segment_updater.consider_merge_options();\n            Ok(())\n        })\n    }\n\n    /// Orders `SegmentManager` to remove all segments\n    pub(crate) fn remove_all_segments(&self) {\n        self.segment_manager.remove_all_segments();\n    }\n\n    pub fn kill(&mut self) {\n        self.killed.store(true, Ordering::Release);\n    }\n\n    pub fn is_alive(&self) -> bool {\n        !self.killed.load(Ordering::Acquire)\n    }\n\n    /// Apply deletes up to the target opstamp to all segments.\n    ///\n    /// The method returns copies of the segment entries,\n    /// updated with the delete information.\n    fn purge_deletes(&self, target_opstamp: Opstamp) -> crate::Result<Vec<SegmentEntry>> {\n        let mut segment_entries = self.segment_manager.segment_entries();\n        for segment_entry in &mut segment_entries {\n            let segment = self.index.segment(segment_entry.meta().clone());\n            advance_deletes(segment, segment_entry, target_opstamp)?;\n        }\n        Ok(segment_entries)\n    }\n\n    pub fn save_metas(\n        &self,\n        opstamp: Opstamp,\n        commit_message: Option<String>,\n    ) -> crate::Result<()> {\n        if self.is_alive() {\n            let index = &self.index;\n            let directory = index.directory();\n            let mut committed_segment_metas = self.segment_manager.committed_segment_metas();\n\n            // We sort segment_readers by number of documents.\n            // This is an heuristic to make multithreading more efficient.\n            //\n            // This is not done at the searcher level because I had a strange\n            // use case in which I was dealing with a large static index,\n            // dispatched over 5 SSD drives.\n            //\n            // A `UnionDirectory` makes it possible to read from these\n            // 5 different drives and creates a meta.json on the fly.\n            // In order to optimize the throughput, it creates a lasagna of segments\n            // from the different drives.\n            //\n            // Segment 1 from disk 1, Segment 1 from disk 2, etc.\n            committed_segment_metas\n                .sort_by_key(|segment_meta| std::cmp::Reverse(segment_meta.max_doc()));\n            let index_meta = IndexMeta {\n                index_settings: index.settings().clone(),\n                segments: committed_segment_metas,\n                schema: index.schema(),\n                opstamp,\n                payload: commit_message,\n            };\n            // TODO add context to the error.\n            save_metas(&index_meta, directory.box_clone().borrow_mut())?;\n            self.store_meta(&index_meta);\n        }\n        Ok(())\n    }\n\n    pub fn schedule_garbage_collect(&self) -> FutureResult<GarbageCollectionResult> {\n        let self_clone = self.clone();\n        self.schedule_task(move || garbage_collect_files(self_clone))\n    }\n\n    /// List the files that are useful to the index.\n    ///\n    /// This does not include lock files, or files that are obsolete\n    /// but have not yet been deleted by the garbage collector.\n    fn list_files(&self) -> HashSet<PathBuf> {\n        let mut files: HashSet<PathBuf> = self\n            .index\n            .list_all_segment_metas()\n            .into_iter()\n            .flat_map(|segment_meta| segment_meta.list_files())\n            .collect();\n        files.insert(META_FILEPATH.to_path_buf());\n        files\n    }\n\n    pub(crate) fn schedule_commit(\n        &self,\n        opstamp: Opstamp,\n        payload: Option<String>,\n    ) -> FutureResult<Opstamp> {\n        let segment_updater: SegmentUpdater = self.clone();\n        self.schedule_task(move || {\n            let segment_entries = segment_updater.purge_deletes(opstamp)?;\n            segment_updater.segment_manager.commit(segment_entries);\n            segment_updater.save_metas(opstamp, payload)?;\n            let _ = garbage_collect_files(segment_updater.clone());\n            segment_updater.consider_merge_options();\n            Ok(opstamp)\n        })\n    }\n\n    fn store_meta(&self, index_meta: &IndexMeta) {\n        *self.active_index_meta.write().unwrap() = Arc::new(index_meta.clone());\n    }\n\n    fn load_meta(&self) -> Arc<IndexMeta> {\n        self.active_index_meta.read().unwrap().clone()\n    }\n\n    pub(crate) fn make_merge_operation(&self, segment_ids: &[SegmentId]) -> MergeOperation {\n        let commit_opstamp = self.load_meta().opstamp;\n        MergeOperation::new(&self.merge_operations, commit_opstamp, segment_ids.to_vec())\n    }\n\n    // Starts a merge operation. This function will block until the merge operation is effectively\n    // started. Note that it does not wait for the merge to terminate.\n    // The calling thread should not be block for a long time, as this only involve waiting for the\n    // `SegmentUpdater` queue which in turns only contains lightweight operations.\n    //\n    // The merge itself happens on a different thread.\n    //\n    // When successful, this function returns a `Future` for a `Result<SegmentMeta>` that represents\n    // the actual outcome of the merge operation.\n    //\n    // It returns an error if for some reason the merge operation could not be started.\n    //\n    // At this point an error is not necessarily the sign of a malfunction.\n    // (e.g. A rollback could have happened, between the instant when the merge operation was\n    // suggested and the moment when it ended up being executed.)\n    //\n    // `segment_ids` is required to be non-empty.\n    pub fn start_merge(\n        &self,\n        merge_operation: MergeOperation,\n    ) -> FutureResult<Option<SegmentMeta>> {\n        assert!(\n            !merge_operation.segment_ids().is_empty(),\n            \"Segment_ids cannot be empty.\"\n        );\n\n        let segment_updater = self.clone();\n        let segment_entries: Vec<SegmentEntry> = match self\n            .segment_manager\n            .start_merge(merge_operation.segment_ids())\n        {\n            Ok(segment_entries) => segment_entries,\n            Err(err) => {\n                warn!(\n                    \"Starting the merge failed for the following reason. This is not fatal. {err}\"\n                );\n                return err.into();\n            }\n        };\n\n        info!(\"Starting merge  - {:?}\", merge_operation.segment_ids());\n\n        let (scheduled_result, merging_future_send) =\n            FutureResult::create(\"Merge operation failed.\");\n\n        self.merge_thread_pool.spawn(move || {\n            // The fact that `merge_operation` is moved here is important.\n            // Its lifetime is used to track how many merging thread are currently running,\n            // as well as which segment is currently in merge and therefore should not be\n            // candidate for another merge.\n            let merge_panic_res = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {\n                merge(\n                    &segment_updater.index,\n                    segment_entries,\n                    merge_operation.target_opstamp(),\n                )\n            }));\n            let merge_res = match merge_panic_res {\n                Ok(merge_res) => merge_res,\n                Err(panic_err) => {\n                    let panic_str = if let Some(msg) = panic_err.downcast_ref::<&str>() {\n                        *msg\n                    } else if let Some(msg) = panic_err.downcast_ref::<String>() {\n                        msg.as_str()\n                    } else {\n                        \"UNKNOWN\"\n                    };\n                    let _send_result = merging_future_send.send(Err(TantivyError::SystemError(\n                        format!(\"Merge thread panicked: {panic_str}\"),\n                    )));\n                    // Resume unwinding because we forced unwind safety with\n                    // `std::panic::AssertUnwindSafe` Use a specific message so\n                    // the panic_handler can double check that we properly caught the panic.\n                    let boxed_panic_message: Box<dyn Any + Send> = Box::new(PANIC_CAUGHT);\n                    std::panic::resume_unwind(boxed_panic_message);\n                }\n            };\n            match merge_res {\n                Ok(after_merge_segment_entry) => {\n                    let res = segment_updater.end_merge(merge_operation, after_merge_segment_entry);\n                    let _send_result = merging_future_send.send(res);\n                }\n                Err(merge_error) => {\n                    warn!(\n                        \"Merge of {:?} was cancelled: {:?}\",\n                        merge_operation.segment_ids().to_vec(),\n                        merge_error\n                    );\n                    if cfg!(test) {\n                        panic!(\"{merge_error:?}\");\n                    }\n                    let _send_result = merging_future_send.send(Err(merge_error));\n                }\n            }\n        });\n\n        scheduled_result\n    }\n\n    pub(crate) fn get_mergeable_segments(&self) -> (Vec<SegmentMeta>, Vec<SegmentMeta>) {\n        let merge_segment_ids: HashSet<SegmentId> = self.merge_operations.segment_in_merge();\n        self.segment_manager\n            .get_mergeable_segments(&merge_segment_ids)\n    }\n\n    fn consider_merge_options(&self) {\n        let (mut committed_segments, mut uncommitted_segments) = self.get_mergeable_segments();\n        if committed_segments.len() == 1 && committed_segments[0].num_deleted_docs() == 0 {\n            committed_segments.clear();\n        }\n        if uncommitted_segments.len() == 1 && uncommitted_segments[0].num_deleted_docs() == 0 {\n            uncommitted_segments.clear();\n        }\n\n        // Committed segments cannot be merged with uncommitted_segments.\n        // We therefore consider merges using these two sets of segments independently.\n        let merge_policy = self.get_merge_policy();\n\n        let current_opstamp = self.stamper.stamp();\n        let mut merge_candidates: Vec<MergeOperation> = merge_policy\n            .compute_merge_candidates(&uncommitted_segments)\n            .into_iter()\n            .map(|merge_candidate| {\n                MergeOperation::new(&self.merge_operations, current_opstamp, merge_candidate.0)\n            })\n            .collect();\n\n        let commit_opstamp = self.load_meta().opstamp;\n        let committed_merge_candidates = merge_policy\n            .compute_merge_candidates(&committed_segments)\n            .into_iter()\n            .map(|merge_candidate: MergeCandidate| {\n                MergeOperation::new(&self.merge_operations, commit_opstamp, merge_candidate.0)\n            });\n        merge_candidates.extend(committed_merge_candidates);\n\n        for merge_operation in merge_candidates {\n            // If a merge cannot be started this is not a fatal error.\n            // We do log a warning in `start_merge`.\n            drop(self.start_merge(merge_operation));\n        }\n    }\n\n    /// Queues a `end_merge` in the segment updater and blocks until it is successfully processed.\n    fn end_merge(\n        &self,\n        merge_operation: MergeOperation,\n        mut after_merge_segment_entry: Option<SegmentEntry>,\n    ) -> crate::Result<Option<SegmentMeta>> {\n        let segment_updater = self.clone();\n        let after_merge_segment_meta = after_merge_segment_entry\n            .as_ref()\n            .map(|after_merge_segment_entry| after_merge_segment_entry.meta().clone());\n        self.schedule_task(move || {\n            info!(\n                \"End merge {:?}\",\n                after_merge_segment_entry.as_ref().map(|entry| entry.meta())\n            );\n            {\n                if let Some(after_merge_segment_entry) = after_merge_segment_entry.as_mut() {\n                    // Deletes and commits could have happened as we were merging.\n                    // We need to make sure we are up to date with deletes before accepting the\n                    // segment.\n                    let mut delete_cursor = after_merge_segment_entry.delete_cursor().clone();\n                    if let Some(delete_operation) = delete_cursor.get() {\n                        let committed_opstamp = segment_updater.load_meta().opstamp;\n                        if delete_operation.opstamp < committed_opstamp {\n                            // We are not up to date! Let's create a new tombstone file for our\n                            // freshly create split.\n                            let index = &segment_updater.index;\n                            let segment = index.segment(after_merge_segment_entry.meta().clone());\n                            if let Err(advance_deletes_err) = advance_deletes(\n                                segment,\n                                after_merge_segment_entry,\n                                committed_opstamp,\n                            ) {\n                                error!(\n                                    \"Merge of {:?} was cancelled (advancing deletes failed): {:?}\",\n                                    merge_operation.segment_ids(),\n                                    advance_deletes_err\n                                );\n                                // `merge_operations` are tracked. As it is dropped, the\n                                // the segment_ids will be available again for merge.\n                                return Err(advance_deletes_err);\n                            }\n                        }\n                    }\n                }\n                let previous_metas = segment_updater.load_meta();\n                let segments_status = segment_updater\n                    .segment_manager\n                    .end_merge(merge_operation.segment_ids(), after_merge_segment_entry)?;\n\n                if segments_status == SegmentsStatus::Committed {\n                    segment_updater\n                        .save_metas(previous_metas.opstamp, previous_metas.payload.clone())?;\n                }\n\n                segment_updater.consider_merge_options();\n            } // we drop all possible handle to a now useless `SegmentMeta`.\n\n            let _ = garbage_collect_files(segment_updater);\n            Ok(())\n        })\n        .wait()?;\n        Ok(after_merge_segment_meta)\n    }\n\n    /// Wait for current merging threads.\n    ///\n    /// Upon termination of the current merging threads,\n    /// merge opportunity may appear.\n    ///\n    /// We keep waiting until the merge policy judges that\n    /// no opportunity is available.\n    ///\n    /// Note that it is not required to call this\n    /// method in your application.\n    /// Terminating your application without letting\n    /// merge terminate is perfectly safe.\n    ///\n    /// Obsolete files will eventually be cleaned up\n    /// by the directory garbage collector.\n    pub fn wait_merging_thread(&self) -> crate::Result<()> {\n        self.merge_operations.wait_until_empty();\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::merge_indices;\n    use crate::collector::TopDocs;\n    use crate::directory::RamDirectory;\n    use crate::fastfield::AliveBitSet;\n    use crate::index::{SegmentId, SegmentMetaInventory};\n    use crate::indexer::merge_policy::tests::MergeWheneverPossible;\n    use crate::indexer::merger::IndexMerger;\n    use crate::indexer::segment_updater::merge_filtered_segments;\n    use crate::query::QueryParser;\n    use crate::schema::*;\n    use crate::{Directory, DocAddress, Index, Segment};\n\n    #[test]\n    fn test_segment_sort_large_max_doc() {\n        // Regression test: -(max_doc as i32) overflows for max_doc >= 2^31.\n        // Using std::cmp::Reverse avoids this.\n        let inventory = SegmentMetaInventory::default();\n        let mut metas = [\n            inventory.new_segment_meta(SegmentId::generate_random(), 100),\n            inventory.new_segment_meta(SegmentId::generate_random(), (1u32 << 31) - 1),\n            inventory.new_segment_meta(SegmentId::generate_random(), 50_000),\n        ];\n        metas.sort_by_key(|m| std::cmp::Reverse(m.max_doc()));\n        assert_eq!(metas[0].max_doc(), (1u32 << 31) - 1);\n        assert_eq!(metas[1].max_doc(), 50_000);\n        assert_eq!(metas[2].max_doc(), 100);\n    }\n\n    #[test]\n    fn test_delete_during_merge() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        // writing the segment\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(MergeWheneverPossible));\n\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n        }\n        index_writer.commit()?;\n\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"c\"))?;\n            index_writer.add_document(doc!(text_field=>\"d\"))?;\n        }\n        index_writer.commit()?;\n\n        index_writer.add_document(doc!(text_field=>\"e\"))?;\n        index_writer.add_document(doc!(text_field=>\"f\"))?;\n        index_writer.commit()?;\n\n        let term = Term::from_field_text(text_field, \"a\");\n        index_writer.delete_term(term);\n        index_writer.commit()?;\n\n        let reader = index.reader()?;\n        assert_eq!(reader.searcher().num_docs(), 302);\n\n        index_writer.wait_merging_threads()?;\n\n        reader.reload()?;\n        assert_eq!(reader.searcher().segment_readers().len(), 1);\n        assert_eq!(reader.searcher().num_docs(), 302);\n        Ok(())\n    }\n\n    #[test]\n    fn delete_all_docs_min() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        // writing the segment\n        let mut index_writer = index.writer_for_tests()?;\n\n        for _ in 0..10 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n        }\n        index_writer.commit()?;\n\n        let seg_ids = index.searchable_segment_ids()?;\n        // docs exist, should have at least 1 segment\n        assert!(!seg_ids.is_empty());\n\n        let term = Term::from_field_text(text_field, \"a\");\n        index_writer.delete_term(term);\n        index_writer.commit()?;\n\n        let term = Term::from_field_text(text_field, \"b\");\n        index_writer.delete_term(term);\n        index_writer.commit()?;\n\n        index_writer.wait_merging_threads()?;\n\n        let reader = index.reader()?;\n        assert_eq!(reader.searcher().num_docs(), 0);\n\n        let seg_ids = index.searchable_segment_ids()?;\n        assert!(seg_ids.is_empty());\n\n        reader.reload()?;\n        assert_eq!(reader.searcher().num_docs(), 0);\n        // empty segments should be erased\n        assert!(index.searchable_segment_metas()?.is_empty());\n        assert!(reader.searcher().segment_readers().is_empty());\n\n        Ok(())\n    }\n\n    #[test]\n    fn delete_all_docs() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        // writing the segment\n        let mut index_writer = index.writer_for_tests()?;\n\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n        }\n        index_writer.commit()?;\n\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"c\"))?;\n            index_writer.add_document(doc!(text_field=>\"d\"))?;\n        }\n        index_writer.commit()?;\n\n        index_writer.add_document(doc!(text_field=>\"e\"))?;\n        index_writer.add_document(doc!(text_field=>\"f\"))?;\n        index_writer.commit()?;\n\n        let seg_ids = index.searchable_segment_ids()?;\n        // docs exist, should have at least 1 segment\n        assert!(!seg_ids.is_empty());\n\n        let term_vals = vec![\"a\", \"b\", \"c\", \"d\", \"e\", \"f\"];\n        for term_val in term_vals {\n            let term = Term::from_field_text(text_field, term_val);\n            index_writer.delete_term(term);\n            index_writer.commit()?;\n        }\n\n        index_writer.wait_merging_threads()?;\n\n        let reader = index.reader()?;\n        assert_eq!(reader.searcher().num_docs(), 0);\n\n        let seg_ids = index.searchable_segment_ids()?;\n        assert!(seg_ids.is_empty());\n\n        reader.reload()?;\n        assert_eq!(reader.searcher().num_docs(), 0);\n        // empty segments should be erased\n        assert!(index.searchable_segment_metas()?.is_empty());\n        assert!(reader.searcher().segment_readers().is_empty());\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_remove_all_segments() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n\n        // writing the segment\n        let mut index_writer = index.writer_for_tests()?;\n        for _ in 0..100 {\n            index_writer.add_document(doc!(text_field=>\"a\"))?;\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n        }\n        index_writer.commit()?;\n\n        index_writer.segment_updater().remove_all_segments();\n        let seg_vec = index_writer\n            .segment_updater()\n            .segment_manager\n            .segment_entries();\n        assert!(seg_vec.is_empty());\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_segments() -> crate::Result<()> {\n        let mut indices = vec![];\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n\n        for _ in 0..3 {\n            let index = Index::create_in_ram(schema.clone());\n\n            // writing two segments\n            let mut index_writer = index.writer_for_tests()?;\n            for _ in 0..100 {\n                index_writer.add_document(doc!(text_field=>\"fizz\"))?;\n                index_writer.add_document(doc!(text_field=>\"buzz\"))?;\n            }\n            index_writer.commit()?;\n\n            for _ in 0..1000 {\n                index_writer.add_document(doc!(text_field=>\"foo\"))?;\n                index_writer.add_document(doc!(text_field=>\"bar\"))?;\n            }\n            index_writer.commit()?;\n            indices.push(index);\n        }\n\n        assert_eq!(indices.len(), 3);\n        let output_directory: Box<dyn Directory> = Box::<RamDirectory>::default();\n        let index = merge_indices(&indices, output_directory)?;\n        assert_eq!(index.schema(), schema);\n\n        let segments = index.searchable_segments()?;\n        assert_eq!(segments.len(), 1);\n\n        let segment_metas = segments[0].meta();\n        assert_eq!(segment_metas.num_deleted_docs(), 0);\n        assert_eq!(segment_metas.num_docs(), 6600);\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_empty_indices_array() {\n        let merge_result = merge_indices(&[], RamDirectory::default());\n        assert!(merge_result.is_err());\n    }\n\n    #[test]\n    fn test_merge_mismatched_schema() -> crate::Result<()> {\n        let first_index = {\n            let mut schema_builder = Schema::builder();\n            let text_field = schema_builder.add_text_field(\"text\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"some text\"))?;\n            index_writer.commit()?;\n            index\n        };\n\n        let second_index = {\n            let mut schema_builder = Schema::builder();\n            let body_field = schema_builder.add_text_field(\"body\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(body_field=>\"some body\"))?;\n            index_writer.commit()?;\n            index\n        };\n\n        // mismatched schema index list\n        let result = merge_indices(&[first_index, second_index], RamDirectory::default());\n        assert!(result.is_err());\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_filtered_segments() -> crate::Result<()> {\n        let first_index = {\n            let mut schema_builder = Schema::builder();\n            let text_field = schema_builder.add_text_field(\"text\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"some text 1\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 2\"))?;\n            index_writer.commit()?;\n            index\n        };\n\n        let second_index = {\n            let mut schema_builder = Schema::builder();\n            let text_field = schema_builder.add_text_field(\"text\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"some text 3\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 4\"))?;\n            index_writer.delete_term(Term::from_field_text(text_field, \"4\"));\n\n            index_writer.commit()?;\n            index\n        };\n\n        let mut segments: Vec<Segment> = Vec::new();\n        segments.extend(first_index.searchable_segments()?);\n        segments.extend(second_index.searchable_segments()?);\n\n        let target_settings = first_index.settings().clone();\n\n        let filter_segment_1 = AliveBitSet::for_test_from_deleted_docs(&[1], 2);\n        let filter_segment_2 = AliveBitSet::for_test_from_deleted_docs(&[0], 2);\n\n        let filter_segments = vec![Some(filter_segment_1), Some(filter_segment_2)];\n\n        let merged_index = merge_filtered_segments(\n            &segments,\n            target_settings,\n            filter_segments,\n            RamDirectory::default(),\n        )?;\n\n        let segments = merged_index.searchable_segments()?;\n        assert_eq!(segments.len(), 1);\n\n        let segment_metas = segments[0].meta();\n        assert_eq!(segment_metas.num_deleted_docs(), 0);\n        assert_eq!(segment_metas.num_docs(), 1);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_single_filtered_segments() -> crate::Result<()> {\n        let first_index = {\n            let mut schema_builder = Schema::builder();\n            let text_field = schema_builder.add_text_field(\"text\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"test text\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 2\"))?;\n\n            index_writer.add_document(doc!(text_field=>\"some text 3\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 4\"))?;\n\n            index_writer.delete_term(Term::from_field_text(text_field, \"4\"));\n\n            index_writer.commit()?;\n            index\n        };\n\n        let mut segments: Vec<Segment> = Vec::new();\n        segments.extend(first_index.searchable_segments()?);\n\n        let target_settings = first_index.settings().clone();\n\n        let filter_segment = AliveBitSet::for_test_from_deleted_docs(&[0], 4);\n\n        let filter_segments = vec![Some(filter_segment)];\n\n        let index = merge_filtered_segments(\n            &segments,\n            target_settings,\n            filter_segments,\n            RamDirectory::default(),\n        )?;\n\n        let segments = index.searchable_segments()?;\n        assert_eq!(segments.len(), 1);\n\n        let segment_metas = segments[0].meta();\n        assert_eq!(segment_metas.num_deleted_docs(), 0);\n        assert_eq!(segment_metas.num_docs(), 2);\n\n        let searcher = index.reader()?.searcher();\n        {\n            let text_field = index.schema().get_field(\"text\").unwrap();\n\n            let do_search = |term: &str| {\n                let query = QueryParser::for_index(&index, vec![text_field])\n                    .parse_query(term)\n                    .unwrap();\n                let top_docs: Vec<(f32, DocAddress)> = searcher\n                    .search(&query, &TopDocs::with_limit(3).order_by_score())\n                    .unwrap();\n\n                top_docs.iter().map(|el| el.1.doc_id).collect::<Vec<_>>()\n            };\n\n            assert_eq!(do_search(\"test\"), vec![] as Vec<u32>);\n            assert_eq!(do_search(\"text\"), vec![0, 1]);\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_apply_doc_id_filter_in_merger() -> crate::Result<()> {\n        let first_index = {\n            let mut schema_builder = Schema::builder();\n            let text_field = schema_builder.add_text_field(\"text\", TEXT);\n            let index = Index::create_in_ram(schema_builder.build());\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"some text 1\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 2\"))?;\n\n            index_writer.add_document(doc!(text_field=>\"some text 3\"))?;\n            index_writer.add_document(doc!(text_field=>\"some text 4\"))?;\n\n            index_writer.delete_term(Term::from_field_text(text_field, \"4\"));\n\n            index_writer.commit()?;\n            index\n        };\n\n        let mut segments: Vec<Segment> = Vec::new();\n        segments.extend(first_index.searchable_segments()?);\n\n        let target_settings = first_index.settings().clone();\n        {\n            let filter_segment = AliveBitSet::for_test_from_deleted_docs(&[1], 4);\n            let filter_segments = vec![Some(filter_segment)];\n            let target_schema = segments[0].schema();\n            let merged_index = Index::create(\n                RamDirectory::default(),\n                target_schema,\n                target_settings.clone(),\n            )?;\n            let merger: IndexMerger = IndexMerger::open_with_custom_alive_set(\n                merged_index.schema(),\n                &segments[..],\n                filter_segments,\n            )?;\n\n            let doc_ids_alive: Vec<_> = merger.readers[0].doc_ids_alive().collect();\n            assert_eq!(doc_ids_alive, vec![0, 2]);\n        }\n\n        {\n            let filter_segments = vec![None];\n            let target_schema = segments[0].schema();\n            let merged_index =\n                Index::create(RamDirectory::default(), target_schema, target_settings)?;\n            let merger: IndexMerger = IndexMerger::open_with_custom_alive_set(\n                merged_index.schema(),\n                &segments[..],\n                filter_segments,\n            )?;\n\n            let doc_ids_alive: Vec<_> = merger.readers[0].doc_ids_alive().collect();\n            assert_eq!(doc_ids_alive, vec![0, 1, 2]);\n        }\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/indexer/segment_writer.rs",
    "content": "use columnar::MonotonicallyMappableToU64;\nuse common::JsonPathWriter;\nuse itertools::Itertools;\nuse tokenizer_api::BoxTokenStream;\n\nuse super::operation::AddOperation;\nuse crate::fastfield::FastFieldsWriter;\nuse crate::fieldnorm::{FieldNormReaders, FieldNormsWriter};\nuse crate::index::{Segment, SegmentComponent};\nuse crate::indexer::indexing_term::IndexingTerm;\nuse crate::indexer::segment_serializer::SegmentSerializer;\nuse crate::json_utils::{index_json_value, IndexingPositionsPerPath};\nuse crate::postings::{\n    compute_table_memory_size, serialize_postings, IndexingContext, IndexingPosition,\n    PerFieldPostingsWriter, PostingsWriter,\n};\nuse crate::schema::document::{Document, Value};\nuse crate::schema::{FieldEntry, FieldType, Schema, DATE_TIME_PRECISION_INDEXED};\nuse crate::tokenizer::{FacetTokenizer, PreTokenizedStream, TextAnalyzer, Tokenizer};\nuse crate::{DocId, Opstamp, TantivyError};\n\n/// Computes the initial size of the hash table.\n///\n/// Returns the recommended initial table size as a power of 2.\n///\n/// Note this is a very dumb way to compute log2, but it is easier to proofread that way.\nfn compute_initial_table_size(per_thread_memory_budget: usize) -> crate::Result<usize> {\n    let table_memory_upper_bound = per_thread_memory_budget / 3;\n    (10..20) // We cap it at 2^19 = 512K capacity.\n        // TODO: There are cases where this limit causes a\n        // reallocation in the hashmap. Check if this affects performance.\n        .map(|power| 1 << power)\n        .take_while(|capacity| compute_table_memory_size(*capacity) < table_memory_upper_bound)\n        .last()\n        .ok_or_else(|| {\n            crate::TantivyError::InvalidArgument(format!(\n                \"per thread memory budget (={per_thread_memory_budget}) is too small. Raise the \\\n                 memory budget or lower the number of threads.\"\n            ))\n        })\n}\n\n/// A `SegmentWriter` is in charge of creating segment index from a\n/// set of documents.\n///\n/// They creates the postings list in anonymous memory.\n/// The segment is laid on disk when the segment gets `finalized`.\npub struct SegmentWriter {\n    pub(crate) max_doc: DocId,\n    pub(crate) ctx: IndexingContext,\n    pub(crate) per_field_postings_writers: PerFieldPostingsWriter,\n    pub(crate) segment_serializer: SegmentSerializer,\n    pub(crate) fast_field_writers: FastFieldsWriter,\n    pub(crate) fieldnorms_writer: FieldNormsWriter,\n    pub(crate) json_path_writer: JsonPathWriter,\n    pub(crate) json_positions_per_path: IndexingPositionsPerPath,\n    pub(crate) doc_opstamps: Vec<Opstamp>,\n    per_field_text_analyzers: Vec<TextAnalyzer>,\n    term_buffer: IndexingTerm,\n    schema: Schema,\n}\n\nimpl SegmentWriter {\n    /// Creates a new `SegmentWriter`\n    ///\n    /// The arguments are defined as follows\n    ///\n    /// - memory_budget: most of the segment writer data (terms, and postings lists recorders) is\n    ///   stored in a memory arena. This makes it possible for the user to define the flushing\n    ///   behavior as a memory limit.\n    /// - segment: The segment being written\n    /// - schema\n    pub fn for_segment(memory_budget_in_bytes: usize, segment: Segment) -> crate::Result<Self> {\n        let schema = segment.schema();\n        let tokenizer_manager = segment.index().tokenizers().clone();\n        let tokenizer_manager_fast_field = segment.index().fast_field_tokenizer().clone();\n        let table_size = compute_initial_table_size(memory_budget_in_bytes)?;\n        let segment_serializer = SegmentSerializer::for_segment(segment)?;\n        let per_field_postings_writers = PerFieldPostingsWriter::for_schema(&schema);\n        let per_field_text_analyzers = schema\n            .fields()\n            .map(|(_, field_entry): (_, &FieldEntry)| {\n                let text_options = match field_entry.field_type() {\n                    FieldType::Str(ref text_options) => text_options.get_indexing_options(),\n                    FieldType::JsonObject(ref json_object_options) => {\n                        json_object_options.get_text_indexing_options()\n                    }\n                    _ => None,\n                };\n                let tokenizer_name = text_options\n                    .map(|text_index_option| text_index_option.tokenizer())\n                    .unwrap_or(\"default\");\n\n                tokenizer_manager.get(tokenizer_name).ok_or_else(|| {\n                    TantivyError::SchemaError(format!(\n                        \"Error getting tokenizer for field: {}\",\n                        field_entry.name()\n                    ))\n                })\n            })\n            .collect::<Result<Vec<_>, _>>()?;\n        Ok(Self {\n            max_doc: 0,\n            ctx: IndexingContext::new(table_size),\n            per_field_postings_writers,\n            fieldnorms_writer: FieldNormsWriter::for_schema(&schema),\n            json_path_writer: JsonPathWriter::default(),\n            json_positions_per_path: IndexingPositionsPerPath::default(),\n            segment_serializer,\n            fast_field_writers: FastFieldsWriter::from_schema_and_tokenizer_manager(\n                &schema,\n                tokenizer_manager_fast_field,\n            )?,\n            doc_opstamps: Vec::with_capacity(1_000),\n            per_field_text_analyzers,\n            term_buffer: IndexingTerm::with_capacity(16),\n            schema,\n        })\n    }\n\n    /// Lay on disk the current content of the `SegmentWriter`\n    ///\n    /// Finalize consumes the `SegmentWriter`, so that it cannot\n    /// be used afterwards.\n    pub fn finalize(mut self) -> crate::Result<Vec<u64>> {\n        self.fieldnorms_writer.fill_up_to_max_doc(self.max_doc);\n        remap_and_write(\n            self.schema,\n            &self.per_field_postings_writers,\n            self.ctx,\n            self.fast_field_writers,\n            &self.fieldnorms_writer,\n            self.segment_serializer,\n        )?;\n        Ok(self.doc_opstamps)\n    }\n\n    /// Returns an estimation of the current memory usage of the segment writer.\n    /// If the mem usage exceeds the `memory_budget`, the segment be serialized.\n    pub fn mem_usage(&self) -> usize {\n        self.ctx.mem_usage()\n            + self.fieldnorms_writer.mem_usage()\n            + self.fast_field_writers.mem_usage()\n            + self.segment_serializer.mem_usage()\n    }\n\n    fn index_document<D: Document>(&mut self, doc: &D) -> crate::Result<()> {\n        let doc_id = self.max_doc;\n\n        // TODO: Can this be optimised a bit?\n        let vals_grouped_by_field = doc\n            .iter_fields_and_values()\n            .sorted_by_key(|(field, _)| *field)\n            .chunk_by(|(field, _)| *field);\n\n        for (field, field_values) in &vals_grouped_by_field {\n            let values = field_values.map(|el| el.1);\n\n            let field_entry = self.schema.get_field_entry(field);\n            let make_schema_error = || {\n                TantivyError::SchemaError(format!(\n                    \"Expected a {:?} for field {:?}\",\n                    field_entry.field_type().value_type(),\n                    field_entry.name()\n                ))\n            };\n            if !field_entry.is_indexed() {\n                continue;\n            }\n\n            let (term_buffer, ctx) = (&mut self.term_buffer, &mut self.ctx);\n            let postings_writer: &mut dyn PostingsWriter =\n                self.per_field_postings_writers.get_for_field_mut(field);\n            term_buffer.clear_with_field(field);\n\n            match field_entry.field_type() {\n                FieldType::Facet(_) => {\n                    let mut facet_tokenizer = FacetTokenizer::default(); // this can be global\n                    for value in values {\n                        let value = value.as_value();\n\n                        let facet_str = value.as_facet().ok_or_else(make_schema_error)?;\n                        let mut facet_tokenizer = facet_tokenizer.token_stream(facet_str);\n                        let mut indexing_position = IndexingPosition::default();\n                        postings_writer.index_text(\n                            doc_id,\n                            &mut facet_tokenizer,\n                            term_buffer,\n                            ctx,\n                            &mut indexing_position,\n                        );\n                    }\n                }\n                FieldType::Str(_) => {\n                    let mut indexing_position = IndexingPosition::default();\n                    for value in values {\n                        let value = value.as_value();\n\n                        let mut token_stream = if let Some(text) = value.as_str() {\n                            let text_analyzer =\n                                &mut self.per_field_text_analyzers[field.field_id() as usize];\n                            text_analyzer.token_stream(text)\n                        } else if let Some(tok_str) = value.into_pre_tokenized_text() {\n                            BoxTokenStream::new(PreTokenizedStream::from(*tok_str.clone()))\n                        } else {\n                            continue;\n                        };\n\n                        assert!(term_buffer.is_empty());\n                        postings_writer.index_text(\n                            doc_id,\n                            &mut *token_stream,\n                            term_buffer,\n                            ctx,\n                            &mut indexing_position,\n                        );\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer\n                            .record(doc_id, field, indexing_position.num_tokens);\n                    }\n                }\n                FieldType::U64(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n\n                        num_vals += 1;\n                        let u64_val = value.as_u64().ok_or_else(make_schema_error)?;\n                        term_buffer.set_u64(u64_val);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::Date(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n\n                        num_vals += 1;\n                        let date_val = value.as_datetime().ok_or_else(make_schema_error)?;\n                        term_buffer\n                            .set_u64(date_val.truncate(DATE_TIME_PRECISION_INDEXED).to_u64());\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::I64(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n\n                        num_vals += 1;\n                        let i64_val = value.as_i64().ok_or_else(make_schema_error)?;\n                        term_buffer.set_i64(i64_val);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::F64(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n                        num_vals += 1;\n                        let f64_val = value.as_f64().ok_or_else(make_schema_error)?;\n                        term_buffer.set_f64(f64_val);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::Bool(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n                        num_vals += 1;\n                        let bool_val = value.as_bool().ok_or_else(make_schema_error)?;\n                        term_buffer.set_bool(bool_val);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::Bytes(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n                        num_vals += 1;\n                        let bytes = value.as_bytes().ok_or_else(make_schema_error)?;\n                        term_buffer.set_bytes(bytes);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n                FieldType::JsonObject(json_options) => {\n                    let text_analyzer =\n                        &mut self.per_field_text_analyzers[field.field_id() as usize];\n\n                    self.json_positions_per_path.clear();\n                    self.json_path_writer\n                        .set_expand_dots(json_options.is_expand_dots_enabled());\n                    for json_value in values {\n                        self.json_path_writer.clear();\n\n                        index_json_value(\n                            doc_id,\n                            json_value,\n                            text_analyzer,\n                            term_buffer,\n                            &mut self.json_path_writer,\n                            postings_writer,\n                            ctx,\n                            &mut self.json_positions_per_path,\n                        );\n                    }\n                }\n                FieldType::IpAddr(_) => {\n                    let mut num_vals = 0;\n                    for value in values {\n                        let value = value.as_value();\n\n                        num_vals += 1;\n                        let ip_addr = value.as_ip_addr().ok_or_else(make_schema_error)?;\n                        term_buffer.set_ip_addr(ip_addr);\n                        postings_writer.subscribe(doc_id, 0u32, term_buffer, ctx);\n                    }\n                    if field_entry.has_fieldnorms() {\n                        self.fieldnorms_writer.record(doc_id, field, num_vals);\n                    }\n                }\n            }\n        }\n        Ok(())\n    }\n\n    /// Indexes a new document\n    ///\n    /// As a user, you should rather use `IndexWriter`'s add_document.\n    pub fn add_document<D: Document>(\n        &mut self,\n        add_operation: AddOperation<D>,\n    ) -> crate::Result<()> {\n        let AddOperation { document, opstamp } = add_operation;\n        self.doc_opstamps.push(opstamp);\n        self.fast_field_writers.add_document(&document)?;\n        self.index_document(&document)?;\n        let doc_writer = self.segment_serializer.get_store_writer();\n        doc_writer.store(&document, &self.schema)?;\n        self.max_doc += 1;\n        Ok(())\n    }\n\n    /// Max doc is\n    /// - the number of documents in the segment assuming there is no deletes\n    /// - the maximum document id (including deleted documents) + 1\n    ///\n    /// Currently, **tantivy** does not handle deletes anyway,\n    /// so `max_doc == num_docs`\n    pub fn max_doc(&self) -> u32 {\n        self.max_doc\n    }\n\n    /// Number of documents in the index.\n    /// Deleted documents are not counted.\n    ///\n    /// Currently, **tantivy** does not handle deletes anyway,\n    /// so `max_doc == num_docs`\n    #[allow(dead_code)]\n    pub fn num_docs(&self) -> u32 {\n        self.max_doc\n    }\n}\n\n/// This method is used as a trick to workaround the borrow checker\n/// Writes a view of a segment by pushing information\n/// to the `SegmentSerializer`.\n///\n/// `doc_id_map` is used to map to the new doc_id order.\nfn remap_and_write(\n    schema: Schema,\n    per_field_postings_writers: &PerFieldPostingsWriter,\n    ctx: IndexingContext,\n    fast_field_writers: FastFieldsWriter,\n    fieldnorms_writer: &FieldNormsWriter,\n    mut serializer: SegmentSerializer,\n) -> crate::Result<()> {\n    debug!(\"remap-and-write\");\n    if let Some(fieldnorms_serializer) = serializer.extract_fieldnorms_serializer() {\n        fieldnorms_writer.serialize(fieldnorms_serializer)?;\n    }\n    let fieldnorm_data = serializer\n        .segment()\n        .open_read(SegmentComponent::FieldNorms)?;\n    let fieldnorm_readers = FieldNormReaders::open(fieldnorm_data)?;\n    serialize_postings(\n        ctx,\n        schema,\n        per_field_postings_writers,\n        fieldnorm_readers,\n        serializer.get_postings_serializer(),\n    )?;\n    debug!(\"fastfield-serialize\");\n    fast_field_writers.serialize(serializer.get_fast_field_write())?;\n\n    debug!(\"serializer-close\");\n    serializer.close()?;\n\n    Ok(())\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeMap;\n    use std::path::Path;\n\n    use columnar::ColumnType;\n\n    use crate::collector::{Count, TopDocs};\n    use crate::directory::RamDirectory;\n    use crate::fastfield::FastValue;\n    use crate::postings::{Postings, TermInfo};\n    use crate::query::{PhraseQuery, QueryParser};\n    use crate::schema::{\n        Document, IndexRecordOption, OwnedValue, Schema, TextFieldIndexing, TextOptions, Value,\n        DATE_TIME_PRECISION_INDEXED, FAST, STORED, STRING, TEXT,\n    };\n    use crate::store::{Compressor, StoreReader, StoreWriter};\n    use crate::time::format_description::well_known::Rfc3339;\n    use crate::time::OffsetDateTime;\n    use crate::tokenizer::{PreTokenizedString, Token};\n    use crate::{\n        DateTime, Directory, DocAddress, DocSet, Index, IndexWriter, SegmentReader,\n        TantivyDocument, Term, TERMINATED,\n    };\n\n    #[test]\n    #[cfg(not(feature = \"compare_hash_only\"))]\n    fn test_hashmap_size() {\n        use super::compute_initial_table_size;\n        assert_eq!(compute_initial_table_size(100_000).unwrap(), 1 << 12);\n        assert_eq!(compute_initial_table_size(1_000_000).unwrap(), 1 << 15);\n        assert_eq!(compute_initial_table_size(15_000_000).unwrap(), 1 << 19);\n        assert_eq!(compute_initial_table_size(1_000_000_000).unwrap(), 1 << 19);\n        assert_eq!(compute_initial_table_size(4_000_000_000).unwrap(), 1 << 19);\n    }\n\n    #[test]\n    fn test_prepare_for_store() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"title\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        let pre_tokenized_text = PreTokenizedString {\n            text: String::from(\"A\"),\n            tokens: vec![Token {\n                offset_from: 0,\n                offset_to: 1,\n                position: 0,\n                text: String::from(\"A\"),\n                position_length: 1,\n            }],\n        };\n\n        doc.add_pre_tokenized_text(text_field, pre_tokenized_text);\n        doc.add_text(text_field, \"title\");\n\n        let path = Path::new(\"store\");\n        let directory = RamDirectory::create();\n        let store_wrt = directory.open_write(path).unwrap();\n\n        let mut store_writer = StoreWriter::new(store_wrt, Compressor::None, 0, false).unwrap();\n        store_writer.store(&doc, &schema).unwrap();\n        store_writer.close().unwrap();\n\n        let reader = StoreReader::open(directory.open_read(path).unwrap(), 0).unwrap();\n        let doc = reader.get::<TantivyDocument>(0).unwrap();\n\n        assert_eq!(doc.field_values().count(), 2);\n        assert_eq!(\n            doc.get_all(text_field).next().unwrap().as_value().as_str(),\n            Some(\"A\")\n        );\n        assert_eq!(\n            doc.get_all(text_field).nth(1).unwrap().as_value().as_str(),\n            Some(\"title\")\n        );\n    }\n    #[test]\n    fn test_simple_json_indexing() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STORED | STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut writer = index.writer_for_tests().unwrap();\n        writer\n            .add_document(doc!(json_field=>json!({\"my_field\": \"b\"})))\n            .unwrap();\n        writer\n            .add_document(doc!(json_field=>json!({\"my_field\": \"a\"})))\n            .unwrap();\n        writer\n            .add_document(doc!(json_field=>json!({\"my_field\": \"b\"})))\n            .unwrap();\n        writer.commit().unwrap();\n\n        let query_parser = QueryParser::for_index(&index, vec![json_field]);\n        let text_query = query_parser.parse_query(\"my_field:a\").unwrap();\n        let score_docs: Vec<(_, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(&text_query, &TopDocs::with_limit(4).order_by_score())\n            .unwrap();\n        assert_eq!(score_docs.len(), 1);\n\n        let text_query = query_parser.parse_query(\"my_field:b\").unwrap();\n        let score_docs: Vec<(_, DocAddress)> = index\n            .reader()\n            .unwrap()\n            .searcher()\n            .search(&text_query, &TopDocs::with_limit(4).order_by_score())\n            .unwrap();\n        assert_eq!(score_docs.len(), 2);\n    }\n\n    #[test]\n    fn test_flat_json_indexing() {\n        // A JSON Object that contains mixed values on the first level\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STORED | STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut writer = index.writer_for_tests().unwrap();\n        // Text, i64, u64\n        writer.add_document(doc!(json_field=>\"b\")).unwrap();\n        writer\n            .add_document(doc!(json_field=>OwnedValue::I64(10i64)))\n            .unwrap();\n        writer\n            .add_document(doc!(json_field=>OwnedValue::U64(55u64)))\n            .unwrap();\n        writer\n            .add_document(doc!(json_field=>json!({\"my_field\": \"a\"})))\n            .unwrap();\n        writer.commit().unwrap();\n\n        let search_and_expect = |query| {\n            let query_parser = QueryParser::for_index(&index, vec![json_field]);\n            let text_query = query_parser.parse_query(query).unwrap();\n            let score_docs: Vec<(_, DocAddress)> = index\n                .reader()\n                .unwrap()\n                .searcher()\n                .search(&text_query, &TopDocs::with_limit(4).order_by_score())\n                .unwrap();\n            assert_eq!(score_docs.len(), 1);\n        };\n\n        search_and_expect(\"my_field:a\");\n        search_and_expect(\"b\");\n        search_and_expect(\"10\");\n        search_and_expect(\"55\");\n    }\n\n    #[test]\n    fn test_json_indexing() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STORED | TEXT);\n        let schema = schema_builder.build();\n        let json_val: serde_json::Value = serde_json::from_str(\n            r#\"{\n            \"toto\": \"titi\",\n            \"float\": -0.2,\n            \"bool\": true,\n            \"unsigned\": 1,\n            \"signed\": -2,\n            \"complexobject\": {\n                \"field.with.dot\": 1\n            },\n            \"date\": \"1985-04-12T23:20:50.52Z\",\n            \"my_arr\": [2, 3, {\"my_key\": \"two tokens\"}, 4]\n        }\"#,\n        )\n        .unwrap();\n        let doc = doc!(json_field=>json_val.clone());\n        let index = Index::create_in_ram(schema.clone());\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let doc = searcher\n            .doc::<TantivyDocument>(DocAddress {\n                segment_ord: 0u32,\n                doc_id: 0u32,\n            })\n            .unwrap();\n        let serdeser_json_val = serde_json::from_str::<serde_json::Value>(&doc.to_json(&schema))\n            .unwrap()\n            .get(\"json\")\n            .unwrap()[0]\n            .clone();\n        assert_eq!(json_val, serdeser_json_val);\n        let segment_reader = searcher.segment_reader(0u32);\n        let inv_idx = segment_reader.inverted_index(json_field).unwrap();\n        let term_dict = inv_idx.terms();\n\n        let mut term_stream = term_dict.stream().unwrap();\n\n        let term_from_path =\n            |path: &str| -> Term { Term::from_field_json_path(json_field, path, false) };\n\n        fn set_fast_val<T: FastValue>(val: T, mut term: Term) -> Term {\n            term.append_type_and_fast_value(val);\n            term\n        }\n        fn set_str(val: &str, mut term: Term) -> Term {\n            term.append_type_and_str(val);\n            term\n        }\n\n        let term = term_from_path(\"bool\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(true, term).serialized_value_bytes()\n        );\n\n        let term = term_from_path(\"complexobject.field\\\\.with\\\\.dot\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(1i64, term).serialized_value_bytes()\n        );\n\n        // Date\n        let term = term_from_path(\"date\");\n\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(\n                DateTime::from_utc(\n                    OffsetDateTime::parse(\"1985-04-12T23:20:50.52Z\", &Rfc3339).unwrap(),\n                )\n                .truncate(DATE_TIME_PRECISION_INDEXED),\n                term\n            )\n            .serialized_value_bytes()\n        );\n\n        // Float\n        let term = term_from_path(\"float\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(-0.2f64, term).serialized_value_bytes()\n        );\n\n        // Number In Array\n        let term = term_from_path(\"my_arr\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(2i64, term).serialized_value_bytes()\n        );\n\n        let term = term_from_path(\"my_arr\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(3i64, term).serialized_value_bytes()\n        );\n\n        let term = term_from_path(\"my_arr\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(4i64, term).serialized_value_bytes()\n        );\n\n        // El in Array\n        let term = term_from_path(\"my_arr.my_key\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_str(\"tokens\", term).serialized_value_bytes()\n        );\n        let term = term_from_path(\"my_arr.my_key\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_str(\"two\", term).serialized_value_bytes()\n        );\n\n        // Signed\n        let term = term_from_path(\"signed\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(-2i64, term).serialized_value_bytes()\n        );\n\n        let term = term_from_path(\"toto\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_str(\"titi\", term).serialized_value_bytes()\n        );\n        // Unsigned\n        let term = term_from_path(\"unsigned\");\n        assert!(term_stream.advance());\n        assert_eq!(\n            term_stream.key(),\n            set_fast_val(1i64, term).serialized_value_bytes()\n        );\n\n        assert!(!term_stream.advance());\n    }\n\n    #[test]\n    fn test_json_tokenized_with_position() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STORED | TEXT);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        let json_val: BTreeMap<String, crate::schema::OwnedValue> =\n            serde_json::from_str(r#\"{\"mykey\": \"repeated token token\"}\"#).unwrap();\n        doc.add_object(json_field, json_val);\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n        let inv_index = segment_reader.inverted_index(json_field).unwrap();\n        let mut term = Term::from_field_json_path(json_field, \"mykey\", false);\n        term.append_type_and_str(\"token\");\n        let term_info = inv_index.get_term_info(&term).unwrap().unwrap();\n        assert_eq!(\n            term_info,\n            TermInfo {\n                doc_freq: 1,\n                postings_range: 2..4,\n                positions_range: 2..5\n            }\n        );\n        let mut postings = inv_index\n            .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n            .unwrap()\n            .unwrap();\n        assert_eq!(postings.doc(), 0);\n        assert_eq!(postings.term_freq(), 2);\n        let mut positions = Vec::new();\n        postings.positions(&mut positions);\n        assert_eq!(&positions[..], &[1, 2]);\n        assert_eq!(postings.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_json_raw_no_position() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STRING);\n        let schema = schema_builder.build();\n        let json_val: serde_json::Value =\n            serde_json::from_str(r#\"{\"mykey\": \"two tokens\"}\"#).unwrap();\n        let doc = doc!(json_field=>json_val);\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n        let inv_index = segment_reader.inverted_index(json_field).unwrap();\n        let mut term = Term::from_field_json_path(json_field, \"mykey\", false);\n        term.append_type_and_str(\"two tokens\");\n        let term_info = inv_index.get_term_info(&term).unwrap().unwrap();\n        assert_eq!(\n            term_info,\n            TermInfo {\n                doc_freq: 1,\n                postings_range: 0..1,\n                positions_range: 0..0\n            }\n        );\n        let mut postings = inv_index\n            .read_postings(&term, IndexRecordOption::WithFreqs)\n            .unwrap()\n            .unwrap();\n        assert_eq!(postings.doc(), 0);\n        assert_eq!(postings.term_freq(), 1);\n        let mut positions = Vec::new();\n        postings.positions(&mut positions);\n        assert_eq!(postings.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_position_overlapping_path() {\n        // This test checks that we do not end up detecting phrase query due\n        // to several string literal in the same json object being overlapping.\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let schema = schema_builder.build();\n        let json_val: serde_json::Value = serde_json::from_str(\n            r#\"{\"mykey\": [{\"field\": \"hello happy tax payer\"}, {\"field\": \"nothello\"}]}\"#,\n        )\n        .unwrap();\n        let doc = doc!(json_field=>json_val);\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let term = Term::from_field_json_path(json_field, \"mykey.field\", false);\n\n        let mut hello_term = term.clone();\n        hello_term.append_type_and_str(\"hello\");\n\n        let mut nothello_term = term.clone();\n        nothello_term.append_type_and_str(\"nothello\");\n\n        let mut happy_term = term.clone();\n        happy_term.append_type_and_str(\"happy\");\n\n        let phrase_query = PhraseQuery::new(vec![hello_term, happy_term.clone()]);\n        assert_eq!(searcher.search(&phrase_query, &Count).unwrap(), 1);\n        let phrase_query = PhraseQuery::new(vec![nothello_term, happy_term]);\n        assert_eq!(searcher.search(&phrase_query, &Count).unwrap(), 0);\n    }\n\n    #[test]\n    fn test_json_fast() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let json_val: serde_json::Value = serde_json::from_str(\n            r#\"{\n            \"toto\": \"titi\",\n            \"float\": -0.2,\n            \"bool\": true,\n            \"unsigned\": 1,\n            \"signed\": -2,\n            \"complexobject\": {\n                \"field.with.dot\": 1\n            },\n            \"date\": \"1985-04-12T23:20:50.52Z\",\n            \"my_arr\": [2, 3, {\"my_key\": \"two tokens\"}, 4]\n        }\"#,\n        )\n        .unwrap();\n        let doc = doc!(json_field=>json_val.clone());\n        let index = Index::create_in_ram(schema.clone());\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n\n        fn assert_type(reader: &SegmentReader, field: &str, typ: ColumnType) {\n            let cols = reader.fast_fields().dynamic_column_handles(field).unwrap();\n            assert_eq!(cols.len(), 1, \"{field}\");\n            assert_eq!(cols[0].column_type(), typ, \"{field}\");\n        }\n        assert_type(segment_reader, \"json.toto\", ColumnType::Str);\n        assert_type(segment_reader, \"json.float\", ColumnType::F64);\n        assert_type(segment_reader, \"json.bool\", ColumnType::Bool);\n        assert_type(segment_reader, \"json.unsigned\", ColumnType::I64);\n        assert_type(segment_reader, \"json.signed\", ColumnType::I64);\n        assert_type(\n            segment_reader,\n            \"json.complexobject.field\\\\.with\\\\.dot\",\n            ColumnType::I64,\n        );\n        assert_type(segment_reader, \"json.date\", ColumnType::DateTime);\n        assert_type(segment_reader, \"json.my_arr\", ColumnType::I64);\n        assert_type(segment_reader, \"json.my_arr.my_key\", ColumnType::Str);\n\n        fn assert_empty(reader: &SegmentReader, field: &str) {\n            let cols = reader.fast_fields().dynamic_column_handles(field).unwrap();\n            assert_eq!(cols.len(), 0);\n        }\n        assert_empty(segment_reader, \"unknown\");\n        assert_empty(segment_reader, \"json\");\n        assert_empty(segment_reader, \"json.toto.titi\");\n\n        let sub_columns = segment_reader\n            .fast_fields()\n            .dynamic_subpath_column_handles(\"json\")\n            .unwrap();\n        assert_eq!(sub_columns.len(), 9);\n\n        let subsub_columns = segment_reader\n            .fast_fields()\n            .dynamic_subpath_column_handles(\"json.complexobject\")\n            .unwrap();\n        assert_eq!(subsub_columns.len(), 1);\n    }\n\n    #[test]\n    fn test_json_term_with_numeric_merge_panic_regression_bug_2283() {\n        // https://github.com/quickwit-oss/tantivy/issues/2283\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        let doc = json!({\"field\": \"a\"});\n        writer.add_document(doc!(json=>doc)).unwrap();\n        writer.commit().unwrap();\n        let doc = json!({\"field\": \"a\", \"id\": 1});\n        writer.add_document(doc!(json=>doc.clone())).unwrap();\n        writer.commit().unwrap();\n\n        // Force Merge\n        writer.wait_merging_threads().unwrap();\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        let segment_ids = index\n            .searchable_segment_ids()\n            .expect(\"Searchable segments failed.\");\n        index_writer.merge(&segment_ids).wait().unwrap();\n        assert!(index_writer.wait_merging_threads().is_ok());\n    }\n\n    #[test]\n    fn test_bug_regression_1629_position_when_array_with_a_field_value_that_does_not_contain_any_token(\n    ) {\n        // We experienced a bug where we would have a position underflow when computing position\n        // delta in an horrible corner case.\n        //\n        // See the commit with this unit test if you want the details.\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let doc = TantivyDocument::parse_json(&schema, r#\"{\"text\": [ \"bbb\", \"aaa\", \"\", \"aaa\"]}\"#)\n            .unwrap();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc).unwrap();\n        // On debug this did panic on the underflow\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let seg_reader = searcher.segment_reader(0);\n        let inv_index = seg_reader.inverted_index(text).unwrap();\n        let term = Term::from_field_text(text, \"aaa\");\n        let mut postings = inv_index\n            .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n            .unwrap()\n            .unwrap();\n        assert_eq!(postings.doc(), 0u32);\n        let mut positions = Vec::new();\n        postings.positions(&mut positions);\n        // On release this was [2, 1]. (< note the decreasing values)\n        assert_eq!(positions, &[2, 5]);\n    }\n\n    #[test]\n    fn test_multiple_field_value_and_long_tokens() {\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        // This is a bit of a contrived example.\n        let tokens = PreTokenizedString {\n            text: \"roller-coaster\".to_string(),\n            tokens: vec![Token {\n                offset_from: 0,\n                offset_to: 14,\n                position: 0,\n                text: \"rollercoaster\".to_string(),\n                position_length: 2,\n            }],\n        };\n        doc.add_pre_tokenized_text(text, tokens.clone());\n        doc.add_pre_tokenized_text(text, tokens);\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let seg_reader = searcher.segment_reader(0);\n        let inv_index = seg_reader.inverted_index(text).unwrap();\n        let term = Term::from_field_text(text, \"rollercoaster\");\n        let mut postings = inv_index\n            .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n            .unwrap()\n            .unwrap();\n        assert_eq!(postings.doc(), 0u32);\n        let mut positions = Vec::new();\n        postings.positions(&mut positions);\n        assert_eq!(positions, &[0, 3]); //< as opposed to 0, 2 if we had a position length of 1.\n    }\n\n    #[test]\n    fn test_last_token_not_ending_last() {\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        // This is a bit of a contrived example.\n        let tokens = PreTokenizedString {\n            text: \"contrived-example\".to_string(), //< I can't think of a use case where this corner case happens in real life.\n            tokens: vec![\n                Token {\n                    // Not the last token, yet ends after the last token.\n                    offset_from: 0,\n                    offset_to: 14,\n                    position: 0,\n                    text: \"long_token\".to_string(),\n                    position_length: 3,\n                },\n                Token {\n                    offset_from: 0,\n                    offset_to: 14,\n                    position: 1,\n                    text: \"short\".to_string(),\n                    position_length: 1,\n                },\n            ],\n        };\n        doc.add_pre_tokenized_text(text, tokens);\n        doc.add_text(text, \"hello\");\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let seg_reader = searcher.segment_reader(0);\n        let inv_index = seg_reader.inverted_index(text).unwrap();\n        let term = Term::from_field_text(text, \"hello\");\n        let mut postings = inv_index\n            .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)\n            .unwrap()\n            .unwrap();\n        assert_eq!(postings.doc(), 0u32);\n        let mut positions = Vec::new();\n        postings.positions(&mut positions);\n        assert_eq!(positions, &[4]); //< as opposed to 3 if we had a position length of 1.\n    }\n\n    #[test]\n    fn test_show_error_when_tokenizer_not_registered() {\n        let text_field_indexing = TextFieldIndexing::default()\n            .set_tokenizer(\"custom_en\")\n            .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n        let text_options = TextOptions::default()\n            .set_indexing_options(text_field_indexing)\n            .set_stored();\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"title\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let schema = index.schema();\n        let mut index_writer = index.writer(50_000_000).unwrap();\n        let title = schema.get_field(\"title\").unwrap();\n        let mut document = TantivyDocument::default();\n        document.add_text(title, \"The Old Man and the Sea\");\n        index_writer.add_document(document).unwrap();\n        let error = index_writer.commit().unwrap_err();\n        assert_eq!(\n            error.to_string(),\n            \"Schema error: 'Error getting tokenizer for field: title'\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/indexer/single_segment_index_writer.rs",
    "content": "use std::marker::PhantomData;\n\nuse crate::indexer::operation::AddOperation;\nuse crate::indexer::segment_updater::save_metas;\nuse crate::indexer::SegmentWriter;\nuse crate::schema::document::Document;\nuse crate::{Directory, Index, IndexMeta, Opstamp, Segment, TantivyDocument};\n\n#[doc(hidden)]\npub struct SingleSegmentIndexWriter<D: Document = TantivyDocument> {\n    segment_writer: SegmentWriter,\n    segment: Segment,\n    opstamp: Opstamp,\n    _phantom: PhantomData<D>,\n}\n\nimpl<D: Document> SingleSegmentIndexWriter<D> {\n    pub fn new(index: Index, mem_budget: usize) -> crate::Result<Self> {\n        let segment = index.new_segment();\n        let segment_writer = SegmentWriter::for_segment(mem_budget, segment.clone())?;\n        Ok(Self {\n            segment_writer,\n            segment,\n            opstamp: 0,\n            _phantom: PhantomData,\n        })\n    }\n\n    pub fn mem_usage(&self) -> usize {\n        self.segment_writer.mem_usage()\n    }\n\n    pub fn add_document(&mut self, document: D) -> crate::Result<()> {\n        let opstamp = self.opstamp;\n        self.opstamp += 1;\n        self.segment_writer\n            .add_document(AddOperation { opstamp, document })\n    }\n\n    pub fn finalize(self) -> crate::Result<Index> {\n        let max_doc = self.segment_writer.max_doc();\n        self.segment_writer.finalize()?;\n        let segment: Segment = self.segment.with_max_doc(max_doc);\n        let index = segment.index();\n        let index_meta = IndexMeta {\n            index_settings: index.settings().clone(),\n            segments: vec![segment.meta().clone()],\n            schema: index.schema(),\n            opstamp: 0,\n            payload: None,\n        };\n        save_metas(&index_meta, index.directory())?;\n        index.directory().sync_directory()?;\n        Ok(segment.index().clone())\n    }\n}\n"
  },
  {
    "path": "src/indexer/stamper.rs",
    "content": "use std::ops::Range;\nuse std::sync::atomic::{AtomicU64, Ordering};\nuse std::sync::Arc;\n\nuse crate::Opstamp;\n\n/// Stamper provides Opstamps, which is just an auto-increment id to label\n/// an operation.\n///\n/// Cloning does not \"fork\" the stamp generation. The stamper actually wraps an `Arc`.\n#[derive(Clone, Default)]\npub struct Stamper(Arc<AtomicU64>);\n\nimpl Stamper {\n    pub fn new(first_opstamp: Opstamp) -> Stamper {\n        Stamper(Arc::new(AtomicU64::new(first_opstamp)))\n    }\n\n    pub fn stamp(&self) -> Opstamp {\n        self.0.fetch_add(1u64, Ordering::SeqCst)\n    }\n\n    /// Given a desired count `n`, `stamps` returns an iterator that\n    /// will supply `n` number of u64 stamps.\n    pub fn stamps(&self, n: u64) -> Range<Opstamp> {\n        let start = self.0.fetch_add(n, Ordering::SeqCst);\n        Range {\n            start,\n            end: start + n,\n        }\n    }\n\n    /// Reverts the stamper to a given `Opstamp` value and returns it\n    pub fn revert(&self, to_opstamp: Opstamp) -> Opstamp {\n        self.0.store(to_opstamp, Ordering::SeqCst);\n        to_opstamp\n    }\n}\n\n#[cfg(test)]\nmod test {\n\n    use super::Stamper;\n\n    #[test]\n    fn test_stamper() {\n        let stamper = Stamper::new(7u64);\n        assert_eq!(stamper.stamp(), 7u64);\n        assert_eq!(stamper.stamp(), 8u64);\n\n        let stamper_clone = stamper.clone();\n        assert_eq!(stamper.stamp(), 9u64);\n\n        assert_eq!(stamper.stamp(), 10u64);\n        assert_eq!(stamper_clone.stamp(), 11u64);\n        assert_eq!(stamper.stamps(3u64), (12..15));\n        assert_eq!(stamper.stamp(), 15u64);\n    }\n\n    #[test]\n    fn test_stamper_revert() {\n        let stamper = Stamper::new(7u64);\n        assert_eq!(stamper.stamp(), 7u64);\n        assert_eq!(stamper.stamp(), 8u64);\n\n        let stamper_clone = stamper.clone();\n        assert_eq!(stamper_clone.stamp(), 9u64);\n\n        stamper.revert(6);\n        assert_eq!(stamper.stamp(), 6);\n        assert_eq!(stamper_clone.stamp(), 7);\n    }\n}\n"
  },
  {
    "path": "src/lib.rs",
    "content": "#![doc(html_logo_url = \"http://fulmicoton.com/tantivy-logo/tantivy-logo.png\")]\n#![cfg_attr(all(feature = \"unstable\", test), feature(test))]\n#![doc(test(attr(allow(unused_variables), deny(warnings))))]\n#![warn(missing_docs)]\n#![allow(\n    clippy::len_without_is_empty,\n    clippy::derive_partial_eq_without_eq,\n    clippy::module_inception,\n    clippy::needless_range_loop,\n    clippy::bool_assert_comparison\n)]\n\n//! # `tantivy`\n//!\n//! Tantivy is a search engine library.\n//! Think `Lucene`, but in Rust.\n//!\n//! ```rust\n//! # use std::path::Path;\n//! # use std::fs;\n//! # use tempfile::TempDir;\n//! # use tantivy::collector::TopDocs;\n//! # use tantivy::query::QueryParser;\n//! # use tantivy::schema::*;\n//! # use tantivy::{doc, DocAddress, Index, IndexWriter, Score};\n//! #\n//! # fn main() {\n//! #     // Let's create a temporary directory for the\n//! #     // sake of this example\n//! #     if let Ok(dir) = TempDir::new() {\n//! #         let index_path = dir.path().join(\"index\");\n//! #         // In case the directory already exists, we remove it\n//! #         let _ = fs::remove_dir_all(&index_path);\n//! #         fs::create_dir_all(&index_path).unwrap();\n//! #         run_example(&index_path).unwrap();\n//! #     }\n//! # }\n//! #\n//! # fn run_example(index_path: &Path) -> tantivy::Result<()> {\n//! // First we need to define a schema ...\n//!\n//! // `TEXT` means the field should be tokenized and indexed,\n//! // along with its term frequency and term positions.\n//! //\n//! // `STORED` means that the field will also be saved\n//! // in a compressed, row-oriented key-value store.\n//! // This store is useful to reconstruct the\n//! // documents that were selected during the search phase.\n//! let mut schema_builder = Schema::builder();\n//! let title = schema_builder.add_text_field(\"title\", TEXT | STORED);\n//! let body = schema_builder.add_text_field(\"body\", TEXT);\n//! let schema = schema_builder.build();\n//!\n//! // Indexing documents\n//!\n//! let index = Index::create_in_dir(index_path, schema.clone())?;\n//!\n//! // Here we use a buffer of 100MB that will be split\n//! // between indexing threads.\n//! let mut index_writer: IndexWriter = index.writer(100_000_000)?;\n//!\n//! // Let's index a document!\n//! index_writer.add_document(doc!(\n//!     title => \"The Old Man and the Sea\",\n//!     body => \"He was an old man who fished alone in a skiff in \\\n//!             the Gulf Stream and he had gone eighty-four days \\\n//!             now without taking a fish.\"\n//! ))?;\n//!\n//! // We need to call .commit() explicitly to force the\n//! // index_writer to finish processing the documents in the queue,\n//! // flush the current index to the disk, and advertise\n//! // the existence of new documents.\n//! index_writer.commit()?;\n//!\n//! // # Searching\n//!\n//! let reader = index.reader()?;\n//!\n//! let searcher = reader.searcher();\n//!\n//! let query_parser = QueryParser::for_index(&index, vec![title, body]);\n//!\n//! // QueryParser may fail if the query is not in the right\n//! // format. For user facing applications, this can be a problem.\n//! // A ticket has been opened regarding this problem.\n//! let query = query_parser.parse_query(\"sea whale\")?;\n//!\n//! // Perform search.\n//! // `topdocs` contains the 10 most relevant doc ids, sorted by decreasing scores...\n//! let top_docs: Vec<(Score, DocAddress)> =\n//!     searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n//!\n//! for (_score, doc_address) in top_docs {\n//!     // Retrieve the actual content of documents given its `doc_address`.\n//!     let retrieved_doc = searcher.doc::<TantivyDocument>(doc_address)?;\n//!     println!(\"{}\", retrieved_doc.to_json(&schema));\n//! }\n//!\n//! # Ok(())\n//! # }\n//! ```\n//!\n//!\n//!\n//! A good place for you to get started is to check out\n//! the example code (\n//! [literate programming](https://tantivy-search.github.io/examples/basic_search.html) /\n//! [source code](https://github.com/quickwit-oss/tantivy/blob/main/examples/basic_search.rs))\n//!\n//! # Tantivy Architecture Overview\n//!\n//! Tantivy is inspired by Lucene, the Architecture is very similar.\n//!\n//! ## Core Concepts\n//!\n//! - **[Index]**: A collection of segments. The top level entry point for tantivy users to search\n//!   and index data.\n//!\n//! - **[Segment]**: At the heart of Tantivy's indexing structure is the [Segment]. It contains\n//!   documents and indices and is the atomic unit of indexing and search.\n//!\n//! - **[Schema](schema)**: A schema is a set of fields in an index. Each field has a specific data\n//!   type and set of attributes.\n//!\n//! - **[IndexWriter]**: Responsible creating and merging segments. It executes the indexing\n//!   pipeline including tokenization, creating indices, and storing the index in the\n//!   [Directory](directory).\n//!\n//! - **Searching**: [Searcher] searches the segments with anything that implements\n//!   [Query](query::Query) and merges the results. The list of [supported\n//!   queries](query::Query#implementers). Custom Queries are supported by implementing the\n//!   [Query](query::Query) trait.\n//!\n//! - **[Directory](directory)**: Abstraction over the storage where the index data is stored.\n//!\n//! - **[Tokenizer](tokenizer)**: Breaks down text into individual tokens. Users can implement or\n//!   use provided tokenizers.\n//!\n//! ## Architecture Flow\n//!\n//! 1. **Document Addition**: Users create documents according to the defined schema. The documents\n//!    fields are tokenized, processed, and added to the current segment. See\n//!    [Document](schema::document) for the structure and usage.\n//!\n//! 2. **Segment Creation**: Once the memory limit threshold is reached or a commit is called, the\n//!    segment is written to the Directory. Documents are searchable after `commit`.\n//!\n//! 3. **Merging**: To optimize space and search speed, segments might be merged. This operation is\n//!    performed in the background. Customize the merge behaviour via\n//!    [IndexWriter::set_merge_policy].\n#[cfg_attr(test, macro_use)]\nextern crate serde_json;\n#[macro_use]\nextern crate log;\n\n#[macro_use]\nextern crate thiserror;\n\n#[cfg(all(test, feature = \"unstable\"))]\nextern crate test;\n\n#[cfg(feature = \"mmap\")]\n#[cfg(test)]\nmod functional_test;\n\n#[macro_use]\nmod macros;\nmod future_result;\n\n// Re-exports\npub use columnar;\npub use common::{ByteCount, DateTime};\npub use query_grammar;\npub use time;\n\npub use crate::error::TantivyError;\npub use crate::future_result::FutureResult;\n\n/// Tantivy result.\n///\n/// Within tantivy, please avoid importing `Result` using `use crate::Result`\n/// and instead, refer to this as `crate::Result<T>`.\npub type Result<T> = std::result::Result<T, TantivyError>;\n\nmod core;\npub mod indexer;\n\npub mod error;\npub mod tokenizer;\n\npub mod aggregation;\npub mod collector;\npub mod directory;\npub mod fastfield;\npub mod fieldnorm;\npub mod index;\npub mod positions;\npub mod postings;\n\n/// Module containing the different query implementations.\npub mod query;\npub mod schema;\npub mod space_usage;\npub mod store;\npub mod termdict;\n\nmod docset;\nmod reader;\n\n#[cfg(test)]\n#[cfg(feature = \"mmap\")]\nmod compat_tests;\n\npub use self::reader::{IndexReader, IndexReaderBuilder, ReloadPolicy, Warmer};\npub mod snippet;\n\nuse std::fmt;\n\npub use census::{Inventory, TrackedObject};\npub use common::{f64_to_u64, i64_to_u64, u64_to_f64, u64_to_i64, HasLen};\nuse once_cell::sync::Lazy;\nuse serde::{Deserialize, Serialize};\n\npub use self::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN, TERMINATED};\npub use crate::core::{json_utils, Executor, Searcher, SearcherGeneration};\npub use crate::directory::Directory;\npub use crate::index::{\n    Index, IndexBuilder, IndexMeta, IndexSettings, InvertedIndexReader, Order, Segment,\n    SegmentMeta, SegmentReader,\n};\npub use crate::indexer::{IndexWriter, SingleSegmentIndexWriter};\npub use crate::schema::{Document, TantivyDocument, Term};\n\n/// Index format version.\npub const INDEX_FORMAT_VERSION: u32 = 7;\n/// Oldest index format version this tantivy version can read.\npub const INDEX_FORMAT_OLDEST_SUPPORTED_VERSION: u32 = 4;\n\n/// Structure version for the index.\n#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]\npub struct Version {\n    major: u32,\n    minor: u32,\n    patch: u32,\n    index_format_version: u32,\n}\n\nimpl fmt::Debug for Version {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        fmt::Display::fmt(self, f)\n    }\n}\n\nstatic VERSION: Lazy<Version> = Lazy::new(|| Version {\n    major: env!(\"CARGO_PKG_VERSION_MAJOR\").parse().unwrap(),\n    minor: env!(\"CARGO_PKG_VERSION_MINOR\").parse().unwrap(),\n    patch: env!(\"CARGO_PKG_VERSION_PATCH\").parse().unwrap(),\n    index_format_version: INDEX_FORMAT_VERSION,\n});\n\nimpl fmt::Display for Version {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(\n            f,\n            \"tantivy v{}.{}.{}, index_format v{}\",\n            self.major, self.minor, self.patch, self.index_format_version\n        )\n    }\n}\n\nstatic VERSION_STRING: Lazy<String> = Lazy::new(|| VERSION.to_string());\n\n/// Expose the current version of tantivy as found in Cargo.toml during compilation.\n/// eg. \"0.11.0\" as well as the compression scheme used in the docstore.\npub fn version() -> &'static Version {\n    &VERSION\n}\n\n/// Exposes the complete version of tantivy as found in Cargo.toml during compilation as a string.\n/// eg. \"tantivy v0.11.0, index_format v1, store_compression: lz4\".\npub fn version_string() -> &'static str {\n    VERSION_STRING.as_str()\n}\n\n/// Defines tantivy's merging strategy\npub mod merge_policy {\n    pub use crate::indexer::{\n        DefaultMergePolicy, LogMergePolicy, MergeCandidate, MergePolicy, NoMergePolicy,\n    };\n}\n\n/// A `u32` identifying a document within a segment.\n/// Documents have their `DocId` assigned incrementally,\n/// as they are added in the segment.\n///\n/// At most, a segment can contain 2^31 documents.\npub type DocId = u32;\n\n/// A u64 assigned to every operation incrementally\n///\n/// All operations modifying the index receives an monotonic Opstamp.\n/// The resulting state of the index is consistent with the opstamp ordering.\n///\n/// For instance, a commit with opstamp `32_423` will reflect all Add and Delete operations\n/// with an opstamp `<= 32_423`. A delete operation with opstamp n will no affect a document added\n/// with opstamp `n+1`.\npub type Opstamp = u64;\n\n/// A Score that represents the relevance of the document to the query\n///\n/// This is modelled internally as a `f32`. The larger the number, the more relevant\n/// the document to the search query.\npub type Score = f32;\n\n/// A `SegmentOrdinal` identifies a segment, within a `Searcher` or `Merger`.\npub type SegmentOrdinal = u32;\n\nimpl DocAddress {\n    /// Creates a new DocAddress from the segment/docId pair.\n    pub fn new(segment_ord: SegmentOrdinal, doc_id: DocId) -> DocAddress {\n        DocAddress {\n            segment_ord,\n            doc_id,\n        }\n    }\n}\n\n/// `DocAddress` contains all the necessary information\n/// to identify a document given a `Searcher` object.\n///\n/// It consists of an id identifying its segment, and\n/// a segment-local `DocId`.\n///\n/// The id used for the segment is actually an ordinal\n/// in the list of `Segment`s held by a `Searcher`.\n#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]\npub struct DocAddress {\n    /// The segment ordinal id that identifies the segment\n    /// hosting the document in the `Searcher` it is called from.\n    pub segment_ord: SegmentOrdinal,\n    /// The segment-local `DocId`.\n    pub doc_id: DocId,\n}\n\n#[macro_export]\n/// Enable fail_point if feature is enabled.\nmacro_rules! fail_point {\n    ($name:expr) => {{\n        #[cfg(feature = \"failpoints\")]\n        {\n            fail::eval($name, |_| {\n                panic!(\"Return is not supported for the fail point \\\"{}\\\"\", $name);\n            });\n        }\n    }};\n    ($name:expr, $e:expr) => {{\n        #[cfg(feature = \"failpoints\")]\n        {\n            if let Some(res) = fail::eval($name, $e) {\n                return res;\n            }\n        }\n    }};\n    ($name:expr, $cond:expr, $e:expr) => {{\n        #[cfg(feature = \"failpoints\")]\n        {\n            if $cond {\n                fail::fail_point!($name, $e);\n            }\n        }\n    }};\n}\n\n/// Common test utilities.\n#[cfg(test)]\npub mod tests {\n    use std::collections::BTreeMap;\n\n    use common::{BinarySerializable, FixedSize};\n    use query_grammar::{UserInputAst, UserInputLeaf, UserInputLiteral};\n    use rand::distr::{Bernoulli, Uniform};\n    use rand::rngs::StdRng;\n    use rand::{Rng, SeedableRng};\n    use time::OffsetDateTime;\n\n    use crate::collector::tests::TEST_COLLECTOR_WITH_SCORE;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::index::SegmentReader;\n    use crate::merge_policy::NoMergePolicy;\n    use crate::postings::Postings;\n    use crate::query::{BooleanQuery, QueryParser};\n    use crate::schema::*;\n    use crate::{DateTime, DocAddress, Index, IndexWriter, ReloadPolicy};\n\n    /// Asserts that the serialized value is the value in the trait.\n    pub fn fixed_size_test<O: BinarySerializable + FixedSize + Default>() {\n        let mut buffer = Vec::new();\n        O::default().serialize(&mut buffer).unwrap();\n        assert_eq!(buffer.len(), O::SIZE_IN_BYTES);\n    }\n\n    /// Checks if left and right are close one to each other.\n    /// Panics if the two values are more than 0.5% apart.\n    #[macro_export]\n    macro_rules! assert_nearly_equals {\n        ($left:expr, $right:expr) => {{\n            assert_nearly_equals!($left, $right, 0.0005);\n        }};\n        ($left:expr, $right:expr, $epsilon:expr) => {{\n            match (&$left, &$right, &$epsilon) {\n                (left_val, right_val, epsilon_val) => {\n                    let diff = (left_val - right_val).abs();\n\n                    if diff > *epsilon_val {\n                        panic!(\n                            r#\"assertion failed: `abs(left-right)>epsilon`\n    left: `{:?}`,\n    right: `{:?}`,\n    epsilon: `{:?}`\"#,\n                            &*left_val, &*right_val, &*epsilon_val\n                        )\n                    }\n                }\n            }\n        }};\n    }\n\n    /// Generates random numbers\n    pub fn generate_nonunique_unsorted(max_value: u32, n_elems: usize) -> Vec<u32> {\n        let seed: [u8; 32] = [1; 32];\n        StdRng::from_seed(seed)\n            .sample_iter(&Uniform::new(0u32, max_value).unwrap())\n            .take(n_elems)\n            .collect::<Vec<u32>>()\n    }\n\n    /// Sample `n` elements with Bernoulli distribution.\n    pub fn sample_with_seed(n: u32, ratio: f64, seed_val: u8) -> Vec<u32> {\n        StdRng::from_seed([seed_val; 32])\n            .sample_iter(&Bernoulli::new(ratio).unwrap())\n            .take(n as usize)\n            .enumerate()\n            .filter_map(|(val, keep)| if keep { Some(val as u32) } else { None })\n            .collect()\n    }\n\n    /// Sample `n` elements with Bernoulli distribution.\n    pub fn sample(n: u32, ratio: f64) -> Vec<u32> {\n        sample_with_seed(n, ratio, 4)\n    }\n\n    #[test]\n    fn test_version_string() {\n        use regex::Regex;\n        let regex_ptn = Regex::new(\n            \"tantivy v[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.[0-9]{1,3}\\\\.{0,10}, index_format v[0-9]{1,5}\",\n        )\n        .unwrap();\n        let version = super::version().to_string();\n        assert!(regex_ptn.find(&version).is_some());\n    }\n\n    #[test]\n    #[cfg(feature = \"mmap\")]\n    fn test_indexing() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_from_tempdir(schema)?;\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        {\n            let doc = doc!(text_field=>\"af b\");\n            index_writer.add_document(doc)?;\n        }\n        {\n            let doc = doc!(text_field=>\"a b c\");\n            index_writer.add_document(doc)?;\n        }\n        {\n            let doc = doc!(text_field=>\"a b c d\");\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_docfreq1() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.add_document(doc!(text_field=>\"a a\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text_field=>\"c\"))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let term_a = Term::from_field_text(text_field, \"a\");\n        assert_eq!(searcher.doc_freq(&term_a)?, 3);\n        let term_b = Term::from_field_text(text_field, \"b\");\n        assert_eq!(searcher.doc_freq(&term_b)?, 1);\n        let term_c = Term::from_field_text(text_field, \"c\");\n        assert_eq!(searcher.doc_freq(&term_c)?, 2);\n        let term_d = Term::from_field_text(text_field, \"d\");\n        assert_eq!(searcher.doc_freq(&term_d)?, 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_fieldnorm_no_docs_with_field() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let title_field = schema_builder.add_text_field(\"title\", TEXT);\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n        index_writer.commit()?;\n        let index_reader = index.reader()?;\n        let searcher = index_reader.searcher();\n        let reader = searcher.segment_reader(0);\n        {\n            let fieldnorm_reader = reader.get_fieldnorms_reader(text_field)?;\n            assert_eq!(fieldnorm_reader.fieldnorm(0), 3);\n        }\n        {\n            let fieldnorm_reader = reader.get_fieldnorms_reader(title_field)?;\n            assert_eq!(fieldnorm_reader.fieldnorm_id(0), 0);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_fieldnorm() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let index = Index::create_in_ram(schema_builder.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n        index_writer.add_document(doc!())?;\n        index_writer.add_document(doc!(text_field=>\"a b\"))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let segment_reader: &SegmentReader = searcher.segment_reader(0);\n        let fieldnorms_reader = segment_reader.get_fieldnorms_reader(text_field)?;\n        assert_eq!(fieldnorms_reader.fieldnorm(0), 3);\n        assert_eq!(fieldnorms_reader.fieldnorm(1), 0);\n        assert_eq!(fieldnorms_reader.fieldnorm(2), 2);\n        Ok(())\n    }\n\n    fn advance_undeleted(docset: &mut dyn DocSet, reader: &SegmentReader) -> bool {\n        let mut doc = docset.advance();\n        while doc != TERMINATED {\n            if !reader.is_deleted(doc) {\n                return true;\n            }\n            doc = docset.advance();\n        }\n        false\n    }\n\n    #[test]\n    fn test_delete_postings1() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let term_abcd = Term::from_field_text(text_field, \"abcd\");\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let term_b = Term::from_field_text(text_field, \"b\");\n        let term_c = Term::from_field_text(text_field, \"c\");\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()\n            .unwrap();\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            // 0\n            index_writer.add_document(doc!(text_field=>\"a b\"))?;\n            // 1\n            index_writer.add_document(doc!(text_field=>\" a c\"))?;\n            // 2\n            index_writer.add_document(doc!(text_field=>\" b c\"))?;\n            // 3\n            index_writer.add_document(doc!(text_field=>\" b d\"))?;\n\n            index_writer.delete_term(Term::from_field_text(text_field, \"c\"));\n            index_writer.delete_term(Term::from_field_text(text_field, \"a\"));\n            // 4\n            index_writer.add_document(doc!(text_field=>\" b c\"))?;\n            // 5\n            index_writer.add_document(doc!(text_field=>\" a\"))?;\n            index_writer.commit()?;\n        }\n        {\n            reader.reload()?;\n            let searcher = reader.searcher();\n            let segment_reader = searcher.segment_reader(0);\n            let inverted_index = segment_reader.inverted_index(text_field)?;\n            assert!(inverted_index\n                .read_postings(&term_abcd, IndexRecordOption::WithFreqsAndPositions)?\n                .is_none());\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 5);\n                assert!(!advance_undeleted(&mut postings, segment_reader));\n            }\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_b, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 3);\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 4);\n                assert!(!advance_undeleted(&mut postings, segment_reader));\n            }\n        }\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            // 0\n            index_writer.add_document(doc!(text_field=>\"a b\"))?;\n            // 1\n            index_writer.delete_term(Term::from_field_text(text_field, \"c\"));\n            index_writer.rollback()?;\n        }\n        {\n            reader.reload()?;\n            let searcher = reader.searcher();\n            let seg_reader = searcher.segment_reader(0);\n            let inverted_index = seg_reader.inverted_index(term_abcd.field())?;\n\n            assert!(inverted_index\n                .read_postings(&term_abcd, IndexRecordOption::WithFreqsAndPositions)?\n                .is_none());\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, seg_reader));\n                assert_eq!(postings.doc(), 5);\n                assert!(!advance_undeleted(&mut postings, seg_reader));\n            }\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_b, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, seg_reader));\n                assert_eq!(postings.doc(), 3);\n                assert!(advance_undeleted(&mut postings, seg_reader));\n                assert_eq!(postings.doc(), 4);\n                assert!(!advance_undeleted(&mut postings, seg_reader));\n            }\n        }\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"a b\"))?;\n            index_writer.delete_term(Term::from_field_text(text_field, \"c\"));\n            index_writer.rollback()?;\n            index_writer.delete_term(Term::from_field_text(text_field, \"a\"));\n            index_writer.commit()?;\n        }\n        {\n            reader.reload()?;\n            let searcher = reader.searcher();\n            let segment_reader = searcher.segment_reader(0);\n            let inverted_index = segment_reader.inverted_index(term_abcd.field())?;\n            assert!(inverted_index\n                .read_postings(&term_abcd, IndexRecordOption::WithFreqsAndPositions)?\n                .is_none());\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(!advance_undeleted(&mut postings, segment_reader));\n            }\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_b, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 3);\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 4);\n                assert!(!advance_undeleted(&mut postings, segment_reader));\n            }\n            {\n                let mut postings = inverted_index\n                    .read_postings(&term_c, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert!(advance_undeleted(&mut postings, segment_reader));\n                assert_eq!(postings.doc(), 4);\n                assert!(!advance_undeleted(&mut postings, segment_reader));\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_indexed_u64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_u64_field(\"value\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(field=>1u64))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let term = Term::from_field_u64(field, 1u64);\n        let mut postings = searcher\n            .segment_reader(0)\n            .inverted_index(term.field())?\n            .read_postings(&term, IndexRecordOption::Basic)?\n            .unwrap();\n        assert_eq!(postings.doc(), 0);\n        assert_eq!(postings.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_indexed_i64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let value_field = schema_builder.add_i64_field(\"value\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        let negative_val = -1i64;\n        index_writer.add_document(doc!(value_field => negative_val))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let term = Term::from_field_i64(value_field, negative_val);\n        let mut postings = searcher\n            .segment_reader(0)\n            .inverted_index(term.field())?\n            .read_postings(&term, IndexRecordOption::Basic)?\n            .unwrap();\n        assert_eq!(postings.doc(), 0);\n        assert_eq!(postings.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_indexed_f64() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let value_field = schema_builder.add_f64_field(\"value\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        let val = std::f64::consts::PI;\n        index_writer.add_document(doc!(value_field => val))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let term = Term::from_field_f64(value_field, val);\n        let mut postings = searcher\n            .segment_reader(0)\n            .inverted_index(term.field())?\n            .read_postings(&term, IndexRecordOption::Basic)?\n            .unwrap();\n        assert_eq!(postings.doc(), 0);\n        assert_eq!(postings.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_indexedfield_not_in_documents() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let absent_field = schema_builder.add_text_field(\"absent_text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        assert!(index_writer.commit().is_ok());\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let inverted_index = segment_reader.inverted_index(absent_field)?;\n        assert_eq!(inverted_index.terms().num_terms(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_delete_postings2() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"63\"))?;\n        index_writer.add_document(doc!(text_field=>\"70\"))?;\n        index_writer.add_document(doc!(text_field=>\"34\"))?;\n        index_writer.add_document(doc!(text_field=>\"1\"))?;\n        index_writer.add_document(doc!(text_field=>\"38\"))?;\n        index_writer.add_document(doc!(text_field=>\"33\"))?;\n        index_writer.add_document(doc!(text_field=>\"40\"))?;\n        index_writer.add_document(doc!(text_field=>\"17\"))?;\n        index_writer.delete_term(Term::from_field_text(text_field, \"38\"));\n        index_writer.delete_term(Term::from_field_text(text_field, \"34\"));\n        index_writer.commit()?;\n        reader.reload()?;\n        assert_eq!(reader.searcher().num_docs(), 6);\n        Ok(())\n    }\n\n    #[test]\n    fn test_termfreq() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"af af af bc bc\"))?;\n            index_writer.commit()?;\n        }\n        {\n            let index_reader = index.reader()?;\n            let searcher = index_reader.searcher();\n            let reader = searcher.segment_reader(0);\n            let inverted_index = reader.inverted_index(text_field)?;\n            let term_abcd = Term::from_field_text(text_field, \"abcd\");\n            assert!(inverted_index\n                .read_postings(&term_abcd, IndexRecordOption::WithFreqsAndPositions)?\n                .is_none());\n            let term_af = Term::from_field_text(text_field, \"af\");\n            let mut postings = inverted_index\n                .read_postings(&term_af, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.doc(), 0);\n            assert_eq!(postings.term_freq(), 3);\n            assert_eq!(postings.advance(), TERMINATED);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_searcher_1() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let reader = index.reader()?;\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"af af af b\"))?;\n        index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n        index_writer.add_document(doc!(text_field=>\"a b c d\"))?;\n        index_writer.commit()?;\n\n        reader.reload()?;\n        let searcher = reader.searcher();\n        let get_doc_ids = |terms: Vec<Term>| {\n            let query = BooleanQuery::new_multiterms_query(terms);\n            searcher\n                .search(&query, &TEST_COLLECTOR_WITH_SCORE)\n                .map(|topdocs| topdocs.docs().to_vec())\n        };\n        assert_eq!(\n            get_doc_ids(vec![Term::from_field_text(text_field, \"a\")])?,\n            vec![DocAddress::new(0, 1), DocAddress::new(0, 2)]\n        );\n        assert_eq!(\n            get_doc_ids(vec![Term::from_field_text(text_field, \"af\")])?,\n            vec![DocAddress::new(0, 0)]\n        );\n        assert_eq!(\n            get_doc_ids(vec![Term::from_field_text(text_field, \"b\")])?,\n            vec![\n                DocAddress::new(0, 0),\n                DocAddress::new(0, 1),\n                DocAddress::new(0, 2)\n            ]\n        );\n        assert_eq!(\n            get_doc_ids(vec![Term::from_field_text(text_field, \"c\")])?,\n            vec![DocAddress::new(0, 1), DocAddress::new(0, 2)]\n        );\n        assert_eq!(\n            get_doc_ids(vec![Term::from_field_text(text_field, \"d\")])?,\n            vec![DocAddress::new(0, 2)]\n        );\n        assert_eq!(\n            get_doc_ids(vec![\n                Term::from_field_text(text_field, \"b\"),\n                Term::from_field_text(text_field, \"a\"),\n            ])?,\n            vec![\n                DocAddress::new(0, 0),\n                DocAddress::new(0, 1),\n                DocAddress::new(0, 2)\n            ]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_searcher_2() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .try_into()?;\n        assert_eq!(reader.searcher().num_docs(), 0u64);\n        // writing the segment\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"af b\"))?;\n        index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n        index_writer.add_document(doc!(text_field=>\"a b c d\"))?;\n        index_writer.commit()?;\n        reader.reload()?;\n        assert_eq!(reader.searcher().num_docs(), 3u64);\n        Ok(())\n    }\n\n    #[test]\n    fn test_searcher_on_json_field_with_type_inference() {\n        // When indexing and searching a json value, we infer its type.\n        // This tests aims to check the type infereence is consistent between indexing and search.\n        // Inference order is date, i64, u64, f64, bool.\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", STORED | TEXT);\n        let schema = schema_builder.build();\n        let json_val: serde_json::Value = serde_json::from_str(\n            r#\"{\n            \"signed\": 2,\n            \"float\": 2.0,\n            \"unsigned\": 10000000000000,\n            \"date\": \"1985-04-12T23:20:50.52Z\",\n            \"bool\": true\n        }\"#,\n        )\n        .unwrap();\n        let doc = doc!(json_field=>json_val);\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer.add_document(doc).unwrap();\n        writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let get_doc_ids = |user_input_literal: UserInputLiteral| {\n            let query_parser = crate::query::QueryParser::for_index(&index, Vec::new());\n            let query = query_parser\n                .build_query_from_user_input_ast(UserInputAst::from(UserInputLeaf::Literal(\n                    user_input_literal,\n                )))\n                .unwrap();\n            searcher\n                .search(&query, &TEST_COLLECTOR_WITH_SCORE)\n                .map(|topdocs| topdocs.docs().to_vec())\n                .unwrap()\n        };\n        {\n            let user_input_literal = UserInputLiteral {\n                field_name: Some(\"json.signed\".to_string()),\n                phrase: \"2\".to_string(),\n                delimiter: crate::query_grammar::Delimiter::None,\n                slop: 0,\n                prefix: false,\n            };\n            assert_eq!(get_doc_ids(user_input_literal), vec![DocAddress::new(0, 0)]);\n        }\n        {\n            let user_input_literal = UserInputLiteral {\n                field_name: Some(\"json.float\".to_string()),\n                phrase: \"2.0\".to_string(),\n                delimiter: crate::query_grammar::Delimiter::None,\n                slop: 0,\n                prefix: false,\n            };\n            assert_eq!(get_doc_ids(user_input_literal), vec![DocAddress::new(0, 0)]);\n        }\n        {\n            let user_input_literal = UserInputLiteral {\n                field_name: Some(\"json.date\".to_string()),\n                phrase: \"1985-04-12T23:20:50.52Z\".to_string(),\n                delimiter: crate::query_grammar::Delimiter::None,\n                slop: 0,\n                prefix: false,\n            };\n            assert_eq!(get_doc_ids(user_input_literal), vec![DocAddress::new(0, 0)]);\n        }\n        {\n            let user_input_literal = UserInputLiteral {\n                field_name: Some(\"json.unsigned\".to_string()),\n                phrase: \"10000000000000\".to_string(),\n                delimiter: crate::query_grammar::Delimiter::None,\n                slop: 0,\n                prefix: false,\n            };\n            assert_eq!(get_doc_ids(user_input_literal), vec![DocAddress::new(0, 0)]);\n        }\n        {\n            let user_input_literal = UserInputLiteral {\n                field_name: Some(\"json.bool\".to_string()),\n                phrase: \"true\".to_string(),\n                delimiter: crate::query_grammar::Delimiter::None,\n                slop: 0,\n                prefix: false,\n            };\n            assert_eq!(get_doc_ids(user_input_literal), vec![DocAddress::new(0, 0)]);\n        }\n    }\n\n    #[test]\n    fn test_doc_macro() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let other_text_field = schema_builder.add_text_field(\"text2\", TEXT);\n        let document = doc!(text_field => \"tantivy\",\n                            text_field => \"some other value\",\n                            other_text_field => \"short\");\n        assert_eq!(document.len(), 3);\n        let values: Vec<OwnedValue> = document.get_all(text_field).map(OwnedValue::from).collect();\n        assert_eq!(values.len(), 2);\n        assert_eq!(values[0].as_ref().as_str(), Some(\"tantivy\"));\n        assert_eq!(values[1].as_ref().as_str(), Some(\"some other value\"));\n        let values: Vec<OwnedValue> = document\n            .get_all(other_text_field)\n            .map(OwnedValue::from)\n            .collect();\n        assert_eq!(values.len(), 1);\n        assert_eq!(values[0].as_ref().as_str(), Some(\"short\"));\n    }\n\n    #[test]\n    fn test_wrong_fast_field_type() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let fast_field_unsigned = schema_builder.add_u64_field(\"unsigned\", FAST);\n        let fast_field_signed = schema_builder.add_i64_field(\"signed\", FAST);\n        let fast_field_float = schema_builder.add_f64_field(\"float\", FAST);\n        schema_builder.add_text_field(\"text\", TEXT);\n        schema_builder.add_u64_field(\"stored_int\", STORED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        {\n            let document =\n                doc!(fast_field_unsigned => 4u64, fast_field_signed=>4i64, fast_field_float=>4f64);\n            index_writer.add_document(document)?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let segment_reader: &SegmentReader = searcher.segment_reader(0);\n        {\n            let fast_field_reader_res = segment_reader.fast_fields().u64(\"text\");\n            assert!(fast_field_reader_res.is_err());\n        }\n        {\n            let fast_field_reader_opt = segment_reader.fast_fields().u64(\"stored_int\");\n            assert!(fast_field_reader_opt.is_err());\n        }\n        {\n            let fast_field_reader_opt = segment_reader.fast_fields().u64(\"signed\");\n            assert!(fast_field_reader_opt.is_err());\n        }\n        {\n            let fast_field_reader_opt = segment_reader.fast_fields().u64(\"float\");\n            assert!(fast_field_reader_opt.is_err());\n        }\n        {\n            let fast_field_reader_opt = segment_reader.fast_fields().u64(\"unsigned\");\n            assert!(fast_field_reader_opt.is_ok());\n            let fast_field_reader = fast_field_reader_opt.unwrap();\n            assert_eq!(fast_field_reader.first(0), Some(4u64))\n        }\n\n        {\n            let fast_field_reader_res = segment_reader.fast_fields().i64(\"signed\");\n            assert!(fast_field_reader_res.is_ok());\n            let fast_field_reader = fast_field_reader_res.unwrap();\n            assert_eq!(fast_field_reader.first(0), Some(4i64))\n        }\n\n        {\n            let fast_field_reader_res = segment_reader.fast_fields().f64(\"float\");\n            assert!(fast_field_reader_res.is_ok());\n            let fast_field_reader = fast_field_reader_res.unwrap();\n            assert_eq!(fast_field_reader.first(0), Some(4f64))\n        }\n        Ok(())\n    }\n\n    // motivated by #729\n    #[test]\n    fn test_update_via_delete_insert() -> crate::Result<()> {\n        use crate::collector::Count;\n        use crate::index::SegmentId;\n        use crate::indexer::NoMergePolicy;\n        use crate::query::AllQuery;\n\n        const DOC_COUNT: u64 = 2u64;\n\n        let mut schema_builder = SchemaBuilder::default();\n        let id = schema_builder.add_u64_field(\"id\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let index_reader = index.reader()?;\n\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n        for doc_id in 0u64..DOC_COUNT {\n            index_writer.add_document(doc!(id => doc_id))?;\n        }\n        index_writer.commit()?;\n\n        index_reader.reload()?;\n        let searcher = index_reader.searcher();\n\n        assert_eq!(\n            searcher.search(&AllQuery, &Count).unwrap(),\n            DOC_COUNT as usize\n        );\n\n        // update the 10 elements by deleting and re-adding\n        for doc_id in 0u64..DOC_COUNT {\n            index_writer.delete_term(Term::from_field_u64(id, doc_id));\n            index_writer.commit()?;\n            index_reader.reload()?;\n            index_writer.add_document(doc!(id =>  doc_id))?;\n            index_writer.commit()?;\n            index_reader.reload()?;\n            let searcher = index_reader.searcher();\n            // The number of document should be stable.\n            assert_eq!(\n                searcher.search(&AllQuery, &Count).unwrap(),\n                DOC_COUNT as usize\n            );\n        }\n\n        index_reader.reload()?;\n        let searcher = index_reader.searcher();\n        let segment_ids: Vec<SegmentId> = searcher\n            .segment_readers()\n            .iter()\n            .map(|reader| reader.segment_id())\n            .collect();\n        index_writer.merge(&segment_ids).wait()?;\n        index_reader.reload()?;\n        let searcher = index_reader.searcher();\n        assert_eq!(searcher.search(&AllQuery, &Count)?, DOC_COUNT as usize);\n        Ok(())\n    }\n\n    #[test]\n    fn test_validate_checksum() -> crate::Result<()> {\n        let mut builder = Schema::builder();\n        let body = builder.add_text_field(\"body\", TEXT | STORED);\n        let schema = builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter = index.writer_for_tests()?;\n        writer.set_merge_policy(Box::new(NoMergePolicy));\n        for _ in 0..5000 {\n            writer.add_document(doc!(body => \"foo\"))?;\n            writer.add_document(doc!(body => \"boo\"))?;\n        }\n        writer.commit()?;\n        assert!(index.validate_checksum()?.is_empty());\n\n        // delete few docs\n        writer.delete_term(Term::from_field_text(body, \"foo\"));\n        writer.commit()?;\n        let segment_ids = index.searchable_segment_ids()?;\n        writer.merge(&segment_ids).wait()?;\n        assert!(index.validate_checksum()?.is_empty());\n        Ok(())\n    }\n\n    #[test]\n    fn test_datetime() {\n        let now = OffsetDateTime::now_utc();\n\n        let dt = DateTime::from_utc(now).into_utc();\n        assert_eq!(dt.to_ordinal_date(), now.to_ordinal_date());\n        assert_eq!(dt.to_hms_micro(), now.to_hms_micro());\n        // We store nanosecond level precision.\n        assert_eq!(dt.nanosecond(), now.nanosecond());\n\n        let dt = DateTime::from_timestamp_secs(now.unix_timestamp()).into_utc();\n        assert_eq!(dt.to_ordinal_date(), now.to_ordinal_date());\n        assert_eq!(dt.to_hms(), now.to_hms());\n        // Constructed from a second precision.\n        assert_ne!(dt.to_hms_micro(), now.to_hms_micro());\n\n        let dt =\n            DateTime::from_timestamp_micros((now.unix_timestamp_nanos() / 1_000) as i64).into_utc();\n        assert_eq!(dt.to_ordinal_date(), now.to_ordinal_date());\n        assert_eq!(dt.to_hms_micro(), now.to_hms_micro());\n\n        let dt_from_ts_nanos =\n            OffsetDateTime::from_unix_timestamp_nanos(1492432621123456789).unwrap();\n        let offset_dt = DateTime::from_utc(dt_from_ts_nanos).into_utc();\n        assert_eq!(\n            dt_from_ts_nanos.to_ordinal_date(),\n            offset_dt.to_ordinal_date()\n        );\n        assert_eq!(dt_from_ts_nanos.to_hms_micro(), offset_dt.to_hms_micro());\n    }\n\n    #[test]\n    fn test_json_number_ambiguity() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"number\", crate::schema::TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests().unwrap();\n        {\n            let mut doc = TantivyDocument::new();\n            let mut obj = BTreeMap::default();\n            obj.insert(\"key\".to_string(), OwnedValue::I64(1i64));\n            doc.add_object(json_field, obj);\n            index_writer.add_document(doc).unwrap();\n        }\n        {\n            let mut doc = TantivyDocument::new();\n            let mut obj = BTreeMap::default();\n            obj.insert(\"key\".to_string(), OwnedValue::U64(1u64));\n            doc.add_object(json_field, obj);\n            index_writer.add_document(doc).unwrap();\n        }\n        {\n            let mut doc = TantivyDocument::new();\n            let mut obj = BTreeMap::default();\n            obj.insert(\"key\".to_string(), OwnedValue::F64(1.0f64));\n            doc.add_object(json_field, obj);\n            index_writer.add_document(doc).unwrap();\n        }\n        index_writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        assert_eq!(searcher.num_docs(), 3);\n        {\n            let parser = QueryParser::for_index(&index, vec![]);\n            let query = parser.parse_query(\"number.key:1\").unwrap();\n            let count = searcher.search(&query, &crate::collector::Count).unwrap();\n            assert_eq!(count, 3);\n        }\n        {\n            let parser = QueryParser::for_index(&index, vec![]);\n            let query = parser.parse_query(\"number.key:1.0\").unwrap();\n            let count = searcher.search(&query, &crate::collector::Count).unwrap();\n            assert_eq!(count, 3);\n        }\n    }\n}\n"
  },
  {
    "path": "src/macros.rs",
    "content": "/// `doc!` is a shortcut that helps building `Document`\n/// objects.\n///\n/// Assuming that `field1` and `field2` are `Field` instances.\n/// You can create a document with a value of `value1` for `field1`\n/// `value2` for `field2`, as follows :\n///\n/// ```c\n/// doc!(\n///     field1 => value1,\n///     field2 => value2,\n/// )\n/// ```\n///\n/// The value can be a `u64`, a `&str`, a `i64`, or a `String`.\n///\n/// # Warning\n///\n/// The document hence created, is not yet validated against a schema.\n/// Nothing prevents its user from creating an invalid document missing a\n/// field, or associating a `String` to a `u64` field for instance.\n///\n/// # Example\n///\n/// ```rust\n/// use tantivy::schema::{Schema, TEXT, FAST};\n/// use tantivy::doc;\n///\n/// //...\n///\n/// # fn main() {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let author = schema_builder.add_text_field(\"text\", TEXT);\n/// let likes = schema_builder.add_u64_field(\"num_u64\", FAST);\n/// let schema = schema_builder.build();\n/// let doc = doc!(\n///     title => \"Life Aquatic\",\n///     author => \"Wes Anderson\",\n///     likes => 4u64\n/// );\n/// # }\n/// ```\n#[macro_export]\nmacro_rules! doc(\n    () => {\n        {\n            ($crate::TantivyDocument::default())\n        }\n    }; // avoids a warning due to the useless `mut`.\n    ($($field:expr => $value:expr),*) => {\n        {\n            let mut document = $crate::TantivyDocument::default();\n            $(\n                document.add_field_value($field, &$value);\n            )*\n            document\n        }\n    };\n    // if there is a trailing comma retry with the trailing comma stripped.\n    ($($field:expr => $value:expr),+ ,) => {\n        doc!( $( $field => $value ), *)\n    };\n);\n\n#[cfg(test)]\nmod test {\n    use crate::schema::{Schema, FAST, TEXT};\n\n    #[test]\n    fn test_doc_basic() {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(\"title\", TEXT);\n        let author = schema_builder.add_text_field(\"text\", TEXT);\n        let likes = schema_builder.add_u64_field(\"num_u64\", FAST);\n        let _schema = schema_builder.build();\n        let _doc = doc!(\n        title => \"Life Aquatic\",\n        author => \"Wes Anderson\",\n        likes => 4u64\n        );\n    }\n\n    #[test]\n    fn test_doc_trailing_comma() {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(\"title\", TEXT);\n        let author = schema_builder.add_text_field(\"text\", TEXT);\n        let likes = schema_builder.add_u64_field(\"num_u64\", FAST);\n        let _schema = schema_builder.build();\n        let _doc = doc!(\n        title => \"Life Aquatic\",\n        author => \"Wes Anderson\",\n        likes => 4u64,\n        );\n    }\n}\n"
  },
  {
    "path": "src/positions/mod.rs",
    "content": "//! Tantivy can (if instructed to do so in the schema) store the term positions in a given field.\n//!\n//! This position is expressed as token ordinal. For instance,\n//! In \"The beauty and the beast\", the term \"the\" appears in position 0 and position 3.\n//! This information is useful to run phrase queries.\n//!\n//! The [position](crate::index::SegmentComponent::Positions) file contains all of the\n//! bitpacked positions delta, for all terms of a given field, one term after the other.\n//!\n//! Each term is encoded independently.\n//! Like for posting lists, tantivy relies on simd bitpacking to encode the positions delta in\n//! blocks of 128 deltas. Because we rarely have a multiple of 128, the final block encodes\n//! the remaining values with variable int encoding.\n//!\n//! In order to make reading possible, the term delta positions first encode the number of\n//! bitpacked blocks, then the bitwidth for each block, then the actual bitpacked blocks and finally\n//! the final variable int encoded block.\n//!\n//! Contrary to postings list, the reader does not have access on the number of positions that is\n//! encoded, and instead stops decoding the last block when its byte slice has been entirely read.\n//!\n//! More formally:\n//! * *Positions* := *NumBitPackedBlocks* *BitPackedPositionBlock*^(P/128)\n//!   *BitPackedPositionsDeltaBitWidth* *VIntPosDeltas*?\n//! * *NumBitPackedBlocks**: := *P* / 128 encoded as a variable byte integer.\n//! * *BitPackedPositionBlock* := bit width encoded block of 128 positions delta\n//! * *BitPackedPositionsDeltaBitWidth* := (*BitWidth*: u8)^*NumBitPackedBlocks*\n//! * *VIntPosDeltas* := *VIntPosDelta*^(*P* % 128).\n//!\n//! The skip widths encoded separately makes it easy and fast to rapidly skip over n positions.\nmod reader;\nmod serializer;\n\nuse bitpacking::{BitPacker, BitPacker4x};\n\npub use self::reader::PositionReader;\npub use self::serializer::PositionSerializer;\n\nconst COMPRESSION_BLOCK_SIZE: usize = BitPacker4x::BLOCK_LEN;\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use proptest::prelude::*;\n    use proptest::sample::select;\n\n    use super::PositionSerializer;\n    use crate::directory::OwnedBytes;\n    use crate::positions::reader::PositionReader;\n\n    fn create_positions_data(vals: &[u32]) -> crate::Result<OwnedBytes> {\n        let mut positions_buffer = vec![];\n        let mut serializer = PositionSerializer::new(&mut positions_buffer);\n        serializer.write_positions_delta(vals);\n        serializer.close_term()?;\n        serializer.close()?;\n        Ok(OwnedBytes::new(positions_buffer))\n    }\n\n    fn gen_delta_positions() -> BoxedStrategy<Vec<u32>> {\n        select(&[0, 1, 70, 127, 128, 129, 200, 255, 256, 257, 270][..])\n            .prop_flat_map(|num_delta_positions| {\n                proptest::collection::vec(\n                    select(&[1u32, 2u32, 4u32, 8u32, 16u32][..]),\n                    num_delta_positions,\n                )\n            })\n            .boxed()\n    }\n\n    proptest! {\n        #[test]\n        fn test_position_delta(delta_positions in gen_delta_positions()) {\n            let delta_positions_data = create_positions_data(&delta_positions).unwrap();\n            let mut position_reader = PositionReader::open(delta_positions_data).unwrap();\n            let mut minibuf = [0u32; 1];\n            for (offset, &delta_position) in delta_positions.iter().enumerate() {\n                position_reader.read(offset as u64, &mut minibuf[..]);\n                assert_eq!(delta_position, minibuf[0]);\n            }\n        }\n    }\n\n    #[test]\n    fn test_position_read() -> crate::Result<()> {\n        let position_deltas: Vec<u32> = (0..1000).collect();\n        let positions_data = create_positions_data(&position_deltas[..])?;\n        assert_eq!(positions_data.len(), 1224);\n        let mut position_reader = PositionReader::open(positions_data)?;\n        for &n in &[1, 10, 127, 128, 130, 312] {\n            let mut v = vec![0u32; n];\n            position_reader.read(0, &mut v[..]);\n            for i in 0..n {\n                assert_eq!(position_deltas[i], i as u32);\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_empty_position() -> crate::Result<()> {\n        let mut positions_buffer = vec![];\n        let mut serializer = PositionSerializer::new(&mut positions_buffer);\n        serializer.close_term()?;\n        serializer.close()?;\n        let position_delta = OwnedBytes::new(positions_buffer);\n        assert!(PositionReader::open(position_delta).is_ok());\n        Ok(())\n    }\n\n    #[test]\n    fn test_multiple_write_positions() -> crate::Result<()> {\n        let mut positions_buffer = vec![];\n        let mut serializer = PositionSerializer::new(&mut positions_buffer);\n        serializer.write_positions_delta(&[1u32, 12u32]);\n        serializer.write_positions_delta(&[4u32, 17u32]);\n        serializer.write_positions_delta(&[443u32]);\n        serializer.close_term()?;\n        serializer.close()?;\n        let position_delta = OwnedBytes::new(positions_buffer);\n        let mut output_delta_pos_buffer = [0u32; 5];\n        let mut position_reader = PositionReader::open(position_delta)?;\n        position_reader.read(0, &mut output_delta_pos_buffer[..]);\n        assert_eq!(\n            &output_delta_pos_buffer[..],\n            &[1u32, 12u32, 4u32, 17u32, 443u32]\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_position_read_with_offset() -> crate::Result<()> {\n        let position_deltas: Vec<u32> = (0..1000).collect();\n        let positions_data = create_positions_data(&position_deltas[..])?;\n        assert_eq!(positions_data.len(), 1224);\n        let mut position_reader = PositionReader::open(positions_data)?;\n        for &offset in &[1u64, 10u64, 127u64, 128u64, 130u64, 312u64] {\n            for &len in &[1, 10, 130, 500] {\n                let mut v = vec![0u32; len];\n                position_reader.read(offset, &mut v[..]);\n                for i in 0..len {\n                    assert_eq!(v[i], i as u32 + offset as u32);\n                }\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_position_read_after_skip() -> crate::Result<()> {\n        let position_deltas: Vec<u32> = (0..1_000).collect();\n        let positions_data = create_positions_data(&position_deltas[..])?;\n        assert_eq!(positions_data.len(), 1224);\n\n        let mut position_reader = PositionReader::open(positions_data)?;\n        let mut buf = [0u32; 7];\n        let mut c = 0;\n\n        let mut offset = 0;\n        for _ in 0..100 {\n            position_reader.read(offset, &mut buf);\n            position_reader.read(offset, &mut buf);\n            offset += 7;\n            for &el in &buf {\n                assert_eq!(c, el);\n                c += 1;\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_position_reread_anchor_different_than_block() -> crate::Result<()> {\n        let positions_delta: Vec<u32> = (0..2_000_000).collect();\n        let positions_data = create_positions_data(&positions_delta[..])?;\n        assert_eq!(positions_data.len(), 5003499);\n        let mut position_reader = PositionReader::open(positions_data)?;\n        let mut buf = [0u32; 256];\n        position_reader.read(128, &mut buf);\n        for i in 0..256 {\n            assert_eq!(buf[i], (128 + i) as u32);\n        }\n        position_reader.read(128, &mut buf);\n        for i in 0..256 {\n            assert_eq!(buf[i], (128 + i) as u32);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_position_requesting_passed_block() -> crate::Result<()> {\n        let positions_delta: Vec<u32> = (0..512).collect();\n        let positions_data = create_positions_data(&positions_delta[..])?;\n        assert_eq!(positions_data.len(), 533);\n        let mut buf = [0u32; 1];\n        let mut position_reader = PositionReader::open(positions_data)?;\n        position_reader.read(230, &mut buf);\n        assert_eq!(buf[0], 230);\n        position_reader.read(9, &mut buf);\n        assert_eq!(buf[0], 9);\n        Ok(())\n    }\n\n    #[test]\n    fn test_position() -> crate::Result<()> {\n        const CONST_VAL: u32 = 9u32;\n        let positions_delta: Vec<u32> = std::iter::repeat_n(CONST_VAL, 2_000_000).collect();\n        let positions_data = create_positions_data(&positions_delta[..])?;\n        assert_eq!(positions_data.len(), 1_015_627);\n        let mut position_reader = PositionReader::open(positions_data)?;\n        let mut buf = [0u32; 1];\n        position_reader.read(0, &mut buf);\n        assert_eq!(buf[0], CONST_VAL);\n        Ok(())\n    }\n\n    #[test]\n    fn test_position_advance() -> crate::Result<()> {\n        let positions_delta: Vec<u32> = (0..2_000_000).collect();\n        let positions_data = create_positions_data(&positions_delta[..])?;\n        assert_eq!(positions_data.len(), 5_003_499);\n        for &offset in &[\n            10,\n            128 * 1024,\n            128 * 1024 - 1,\n            128 * 1024 + 7,\n            128 * 10 * 1024 + 10,\n        ] {\n            let mut position_reader = PositionReader::open(positions_data.clone())?;\n            let mut buf = [0u32; 1];\n            position_reader.read(offset, &mut buf);\n            assert_eq!(buf[0], offset as u32);\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/positions/reader.rs",
    "content": "use std::io;\n\nuse common::{BinarySerializable, VInt};\n\nuse crate::directory::OwnedBytes;\nuse crate::positions::COMPRESSION_BLOCK_SIZE;\nuse crate::postings::compression::{BlockDecoder, VIntDecoder};\n\n/// When accessing the positions of a term, we get a positions_idx from the `Terminfo`.\n/// This means we need to skip to the `nth` position efficiently.\n///\n/// Blocks are compressed using bitpacking, so `skip_read` contains the number of bits\n/// (values can go from 0 to 32 bits) required to decompress every block.\n///\n/// A given block obviously takes `(128 x  num_bit_for_the_block / num_bits_in_a_byte)`,\n/// so skipping a block without decompressing it is just a matter of advancing that many\n/// bytes.\n\n#[derive(Clone)]\npub struct PositionReader {\n    bit_widths: OwnedBytes,\n    positions: OwnedBytes,\n\n    block_decoder: BlockDecoder,\n\n    // offset, expressed in positions, for the first position of the block currently loaded\n    // block_offset is a multiple of COMPRESSION_BLOCK_SIZE.\n    block_offset: u64,\n    // offset, expressed in positions, for the position of the first block encoded\n    // in the `self.positions` bytes, and if bitpacked, compressed using the bitwidth in\n    // `self.bit_widths`.\n    //\n    // As we advance, anchor increases simultaneously with bit_widths and positions get consumed.\n    anchor_offset: u64,\n\n    // These are just copies used for .reset().\n    original_bit_widths: OwnedBytes,\n    original_positions: OwnedBytes,\n}\n\nimpl PositionReader {\n    /// Open and reads the term positions encoded into the positions_data owned bytes.\n    pub fn open(mut positions_data: OwnedBytes) -> io::Result<PositionReader> {\n        let num_positions_bitpacked_blocks = VInt::deserialize(&mut positions_data)?.0 as usize;\n        let (bit_widths, positions) = positions_data.split(num_positions_bitpacked_blocks);\n        Ok(PositionReader {\n            bit_widths: bit_widths.clone(),\n            positions: positions.clone(),\n            block_decoder: BlockDecoder::default(),\n            block_offset: i64::MAX as u64,\n            anchor_offset: 0u64,\n            original_bit_widths: bit_widths,\n            original_positions: positions,\n        })\n    }\n\n    fn reset(&mut self) {\n        self.positions = self.original_positions.clone();\n        self.bit_widths = self.original_bit_widths.clone();\n        self.block_offset = i64::MAX as u64;\n        self.anchor_offset = 0u64;\n    }\n\n    /// Advance from num_blocks bitpacked blocks.\n    ///\n    /// Panics if there are not that many remaining blocks.\n    fn advance_num_blocks(&mut self, num_blocks: usize) {\n        let num_bits: usize = self.bit_widths.as_ref()[..num_blocks]\n            .iter()\n            .cloned()\n            .map(|num_bits| num_bits as usize)\n            .sum();\n        let num_bytes_to_skip = num_bits * COMPRESSION_BLOCK_SIZE / 8;\n        self.bit_widths.advance(num_blocks);\n        self.positions.advance(num_bytes_to_skip);\n        self.anchor_offset += (num_blocks * COMPRESSION_BLOCK_SIZE) as u64;\n    }\n\n    /// block_rel_id is counted relatively to the anchor.\n    /// block_rel_id = 0 means the anchor block.\n    /// block_rel_id = i means the ith block after the anchor block.\n    fn load_block(&mut self, block_rel_id: usize) {\n        let bit_widths = self.bit_widths.as_slice();\n        let byte_offset: usize = bit_widths[0..block_rel_id]\n            .iter()\n            .map(|&b| b as usize)\n            .sum::<usize>()\n            * COMPRESSION_BLOCK_SIZE\n            / 8;\n        let compressed_data = &self.positions.as_slice()[byte_offset..];\n        if bit_widths.len() > block_rel_id {\n            // that block is bitpacked.\n            let bit_width = bit_widths[block_rel_id];\n            self.block_decoder\n                .uncompress_block_unsorted(compressed_data, bit_width, false);\n        } else {\n            // that block is vint encoded.\n            self.block_decoder\n                .uncompress_vint_unsorted_until_end(compressed_data);\n        }\n        self.block_offset = self.anchor_offset + (block_rel_id * COMPRESSION_BLOCK_SIZE) as u64;\n    }\n\n    /// Fills a buffer with the positions `[offset..offset+output.len())` integers.\n    ///\n    /// This function is optimized to be called with increasing values of `offset`.\n    pub fn read(&mut self, mut offset: u64, mut output: &mut [u32]) {\n        if offset < self.anchor_offset {\n            self.reset();\n        }\n        let delta_to_block_offset = offset as i64 - self.block_offset as i64;\n        if !(0..128).contains(&delta_to_block_offset) {\n            // The first position is not within the first block.\n            // (Note that it could be before or after)\n            // We need to possibly skip a few blocks, and decompress the first relevant  block.\n            let delta_to_anchor_offset = offset - self.anchor_offset;\n            let num_blocks_to_skip =\n                (delta_to_anchor_offset / (COMPRESSION_BLOCK_SIZE as u64)) as usize;\n            self.advance_num_blocks(num_blocks_to_skip);\n            self.load_block(0);\n        } else {\n            // The request offset is within the loaded block.\n            // We still need to advance anchor_offset to our current block.\n            let num_blocks_to_skip =\n                ((self.block_offset - self.anchor_offset) / COMPRESSION_BLOCK_SIZE as u64) as usize;\n            self.advance_num_blocks(num_blocks_to_skip);\n        }\n\n        // At this point, the block containing offset is loaded, and anchor has\n        // been updated to point to it as well.\n        for i in 1.. {\n            // we copy the part from block i - 1 that is relevant.\n            let offset_in_block = (offset as usize) % COMPRESSION_BLOCK_SIZE;\n            let remaining_in_block = COMPRESSION_BLOCK_SIZE - offset_in_block;\n            if remaining_in_block >= output.len() {\n                output.copy_from_slice(\n                    &self.block_decoder.output_array()[offset_in_block..][..output.len()],\n                );\n                break;\n            }\n            output[..remaining_in_block]\n                .copy_from_slice(&self.block_decoder.output_array()[offset_in_block..]);\n            output = &mut output[remaining_in_block..];\n            // we load block #i if necessary.\n            offset += remaining_in_block as u64;\n            self.load_block(i);\n        }\n    }\n}\n"
  },
  {
    "path": "src/positions/serializer.rs",
    "content": "use std::io::{self, Write};\n\nuse common::{BinarySerializable, CountingWriter, VInt};\n\nuse crate::positions::COMPRESSION_BLOCK_SIZE;\nuse crate::postings::compression::{BlockEncoder, VIntEncoder};\n\n/// The PositionSerializer is in charge of serializing all of the positions\n/// of all of the terms of a given field.\n///\n/// It is valid to call write_position_delta more than once per term.\npub struct PositionSerializer<W: io::Write> {\n    block_encoder: BlockEncoder,\n    positions_wrt: CountingWriter<W>,\n    positions_buffer: Vec<u8>,\n    block: Vec<u32>,\n    bit_widths: Vec<u8>,\n}\n\nimpl<W: io::Write> PositionSerializer<W> {\n    /// Creates a new PositionSerializer writing into the given positions_wrt.\n    pub fn new(positions_wrt: W) -> PositionSerializer<W> {\n        PositionSerializer {\n            block_encoder: BlockEncoder::new(),\n            positions_wrt: CountingWriter::wrap(positions_wrt),\n            positions_buffer: Vec::with_capacity(128_000),\n            block: Vec::with_capacity(128),\n            bit_widths: Vec::new(),\n        }\n    }\n\n    /// Returns the number of bytes written in the positions write object\n    /// at this point.\n    /// When called before writing the positions of a term, this value is used as\n    /// start offset.\n    /// When called after writing the positions of a term, this value is used as\n    /// end offset.\n    pub fn written_bytes(&self) -> u64 {\n        self.positions_wrt.written_bytes()\n    }\n\n    fn remaining_block_len(&self) -> usize {\n        COMPRESSION_BLOCK_SIZE - self.block.len()\n    }\n\n    /// Writes all of the given positions delta.\n    pub fn write_positions_delta(&mut self, mut positions_delta: &[u32]) {\n        while !positions_delta.is_empty() {\n            let remaining_block_len = self.remaining_block_len();\n            let num_to_write = remaining_block_len.min(positions_delta.len());\n            self.block.extend(&positions_delta[..num_to_write]);\n            positions_delta = &positions_delta[num_to_write..];\n            if self.remaining_block_len() == 0 {\n                self.flush_block();\n            }\n        }\n    }\n\n    fn flush_block(&mut self) {\n        // encode the positions in the block\n        if self.block.is_empty() {\n            return;\n        }\n        if self.block.len() == COMPRESSION_BLOCK_SIZE {\n            let (bit_width, block_encoded): (u8, &[u8]) = self\n                .block_encoder\n                .compress_block_unsorted(&self.block[..], false);\n            self.bit_widths.push(bit_width);\n            self.positions_buffer.extend(block_encoded);\n        } else {\n            debug_assert!(self.block.len() < COMPRESSION_BLOCK_SIZE);\n            let block_vint_encoded = self.block_encoder.compress_vint_unsorted(&self.block[..]);\n            self.positions_buffer.extend_from_slice(block_vint_encoded);\n        }\n        self.block.clear();\n    }\n\n    /// Close the positions for the current term.\n    pub fn close_term(&mut self) -> io::Result<()> {\n        self.flush_block();\n        VInt(self.bit_widths.len() as u64).serialize(&mut self.positions_wrt)?;\n        self.positions_wrt.write_all(&self.bit_widths[..])?;\n        self.positions_wrt.write_all(&self.positions_buffer)?;\n        self.bit_widths.clear();\n        self.positions_buffer.clear();\n        Ok(())\n    }\n\n    /// Close the positions for this term and flushes the data.\n    pub fn close(mut self) -> io::Result<()> {\n        self.positions_wrt.flush()\n    }\n}\n"
  },
  {
    "path": "src/postings/block_search.rs",
    "content": "use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n\n/// Search the first index containing an element greater or equal to\n/// the target.\n///\n/// The results should be equivalent to\n/// ```compile_fail\n/// block[..]\n//       .iter()\n//       .take_while(|&&val| val < target)\n//       .count()\n/// ```\n/// \n/// the `start` argument is just used to hint that the response is\n/// greater than beyond `start`. The implementation may or may not use\n/// it for optimization.\n///\n/// # Assumption\n///\n/// - The block is sorted. Some elements may appear several times. This is the case at the\n///   end of the last block for instance.\n/// - The target is assumed smaller or equal to the last element of the block.\npub fn branchless_binary_search(arr: &[u32; COMPRESSION_BLOCK_SIZE], target: u32) -> usize {\n    let mut start = 0;\n    let mut len = arr.len();\n    for _ in 0..7 {\n        len /= 2;\n        let pivot = unsafe { *arr.get_unchecked(start + len - 1) };\n        if pivot < target {\n            start += len;\n        }\n    }\n    start\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n\n    use proptest::prelude::*;\n\n    use super::branchless_binary_search;\n    use crate::docset::TERMINATED;\n    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n\n    fn search_in_block_trivial_but_slow(block: &[u32], target: u32) -> usize {\n        block.iter().take_while(|&&val| val < target).count()\n    }\n\n    fn util_test_search_in_block(block: &[u32], target: u32) {\n        let cursor = search_in_block_trivial_but_slow(block, target);\n        assert!(cursor < COMPRESSION_BLOCK_SIZE);\n        assert!(block[cursor] >= target);\n        if cursor > 0 {\n            assert!(block[cursor - 1] < target);\n        }\n        assert_eq!(block.len(), COMPRESSION_BLOCK_SIZE);\n        let mut output_buffer = [TERMINATED; COMPRESSION_BLOCK_SIZE];\n        output_buffer[..block.len()].copy_from_slice(block);\n        assert_eq!(branchless_binary_search(&output_buffer, target), cursor);\n    }\n\n    fn util_test_search_in_block_all(block: &[u32]) {\n        let mut targets = HashSet::new();\n        targets.insert(0);\n        for &val in block {\n            if val > 0 {\n                targets.insert(val - 1);\n            }\n            targets.insert(val);\n        }\n        for target in targets {\n            util_test_search_in_block(block, target);\n        }\n    }\n\n    #[test]\n    fn test_search_in_branchless_binary_search() {\n        let v: Vec<u32> = (0..COMPRESSION_BLOCK_SIZE).map(|i| i as u32 * 2).collect();\n        util_test_search_in_block_all(&v[..]);\n    }\n\n    fn monotonous_block() -> impl Strategy<Value = Vec<u32>> {\n        prop::collection::vec(0u32..5u32, COMPRESSION_BLOCK_SIZE).prop_map(|mut deltas| {\n            let mut el = 0;\n            for i in 0..COMPRESSION_BLOCK_SIZE {\n                el += deltas[i];\n                deltas[i] = el;\n            }\n            deltas\n        })\n    }\n\n    proptest! {\n        #[test]\n        fn test_proptest_branchless_binary_search(block in monotonous_block()) {\n            util_test_search_in_block_all(&block[..]);\n        }\n    }\n}\n"
  },
  {
    "path": "src/postings/block_segment_postings.rs",
    "content": "use std::io;\n\nuse common::VInt;\n\nuse crate::directory::{FileSlice, OwnedBytes};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::postings::compression::{BlockDecoder, VIntDecoder, COMPRESSION_BLOCK_SIZE};\nuse crate::postings::{BlockInfo, FreqReadingOption, SkipReader};\nuse crate::query::Bm25Weight;\nuse crate::schema::IndexRecordOption;\nuse crate::{DocId, Score, TERMINATED};\n\nfn max_score<I: Iterator<Item = Score>>(mut it: I) -> Option<Score> {\n    it.next().map(|first| it.fold(first, Score::max))\n}\n\n/// `BlockSegmentPostings` is a cursor iterating over blocks\n/// of documents.\n///\n/// # Warning\n///\n/// While it is useful for some very specific high-performance\n/// use cases, you should prefer using `SegmentPostings` for most usage.\n#[derive(Clone)]\npub struct BlockSegmentPostings {\n    pub(crate) doc_decoder: BlockDecoder,\n    block_loaded: bool,\n    freq_decoder: BlockDecoder,\n    freq_reading_option: FreqReadingOption,\n    block_max_score_cache: Option<Score>,\n    doc_freq: u32,\n    data: OwnedBytes,\n    skip_reader: SkipReader,\n}\n\nfn decode_bitpacked_block(\n    doc_decoder: &mut BlockDecoder,\n    freq_decoder_opt: Option<&mut BlockDecoder>,\n    data: &[u8],\n    doc_offset: DocId,\n    doc_num_bits: u8,\n    tf_num_bits: u8,\n    strict_delta: bool,\n) {\n    let num_consumed_bytes =\n        doc_decoder.uncompress_block_sorted(data, doc_offset, doc_num_bits, strict_delta);\n    if let Some(freq_decoder) = freq_decoder_opt {\n        freq_decoder.uncompress_block_unsorted(\n            &data[num_consumed_bytes..],\n            tf_num_bits,\n            strict_delta,\n        );\n    }\n}\n\nfn decode_vint_block(\n    doc_decoder: &mut BlockDecoder,\n    freq_decoder_opt: Option<&mut BlockDecoder>,\n    data: &[u8],\n    doc_offset: DocId,\n    num_vint_docs: usize,\n) {\n    let num_consumed_bytes =\n        doc_decoder.uncompress_vint_sorted(data, doc_offset, num_vint_docs, TERMINATED);\n    if let Some(freq_decoder) = freq_decoder_opt {\n        // if it's a json term with freq, containing less than 256 docs, we can reach here thinking\n        // we have a freq, despite not really having one.\n        if data.len() > num_consumed_bytes {\n            freq_decoder.uncompress_vint_unsorted(\n                &data[num_consumed_bytes..],\n                num_vint_docs,\n                TERMINATED,\n            );\n        }\n    }\n}\n\nfn split_into_skips_and_postings(\n    doc_freq: u32,\n    mut bytes: OwnedBytes,\n) -> io::Result<(Option<OwnedBytes>, OwnedBytes)> {\n    if doc_freq < COMPRESSION_BLOCK_SIZE as u32 {\n        return Ok((None, bytes));\n    }\n    let skip_len = VInt::deserialize_u64(&mut bytes)? as usize;\n    let (skip_data, postings_data) = bytes.split(skip_len);\n    Ok((Some(skip_data), postings_data))\n}\n\nimpl BlockSegmentPostings {\n    /// Opens a `BlockSegmentPostings`.\n    /// `doc_freq` is the number of documents in the posting list.\n    /// `record_option` represents the amount of data available according to the schema.\n    /// `requested_option` is the amount of data requested by the user.\n    /// If for instance, we do not request for term frequencies, this function will not decompress\n    /// term frequency blocks.\n    pub(crate) fn open(\n        doc_freq: u32,\n        data: FileSlice,\n        mut record_option: IndexRecordOption,\n        requested_option: IndexRecordOption,\n    ) -> io::Result<BlockSegmentPostings> {\n        let bytes = data.read_bytes()?;\n        let (skip_data_opt, postings_data) = split_into_skips_and_postings(doc_freq, bytes)?;\n        let skip_reader = match skip_data_opt {\n            Some(skip_data) => {\n                let block_count = doc_freq as usize / COMPRESSION_BLOCK_SIZE;\n                // 8 is the minimum size of a block with frequency (can be more if pos are stored\n                // too)\n                if skip_data.len() < 8 * block_count {\n                    // the field might be encoded with frequency, but this term in particular isn't.\n                    // This can happen for JSON field with term frequencies:\n                    // - text terms are encoded with term freqs.\n                    // - numerical terms are encoded without term freqs.\n                    record_option = IndexRecordOption::Basic;\n                }\n                SkipReader::new(skip_data, doc_freq, record_option)\n            }\n            None => SkipReader::new(OwnedBytes::empty(), doc_freq, record_option),\n        };\n\n        let freq_reading_option = match (record_option, requested_option) {\n            (IndexRecordOption::Basic, _) => FreqReadingOption::NoFreq,\n            (_, IndexRecordOption::Basic) => FreqReadingOption::SkipFreq,\n            (_, _) => FreqReadingOption::ReadFreq,\n        };\n\n        let mut block_segment_postings = BlockSegmentPostings {\n            doc_decoder: BlockDecoder::with_val(TERMINATED),\n            block_loaded: false,\n            freq_decoder: BlockDecoder::with_val(1),\n            freq_reading_option,\n            block_max_score_cache: None,\n            doc_freq,\n            data: postings_data,\n            skip_reader,\n        };\n        block_segment_postings.load_block();\n        Ok(block_segment_postings)\n    }\n\n    /// Returns the block_max_score for the current block.\n    /// It does not require the block to be loaded. For instance, it is ok to call this method\n    /// after having called `.shallow_advance(..)`.\n    ///\n    /// See `TermScorer::block_max_score(..)` for more information.\n    pub fn block_max_score(\n        &mut self,\n        fieldnorm_reader: &FieldNormReader,\n        bm25_weight: &Bm25Weight,\n    ) -> Score {\n        if let Some(score) = self.block_max_score_cache {\n            return score;\n        }\n        if let Some(skip_reader_max_score) = self.skip_reader.block_max_score(bm25_weight) {\n            // if we are on a full block, the skip reader should have the block max information\n            // for us\n            self.block_max_score_cache = Some(skip_reader_max_score);\n            return skip_reader_max_score;\n        }\n        // this is the last block of the segment posting list.\n        // If it is actually loaded, we can compute block max manually.\n        if self.block_is_loaded() {\n            let docs = self.doc_decoder.output_array().iter().cloned();\n            let freqs = self.freq_decoder.output_array().iter().cloned();\n            let bm25_scores = docs.zip(freqs).map(|(doc, term_freq)| {\n                let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);\n                bm25_weight.score(fieldnorm_id, term_freq)\n            });\n            let block_max_score = max_score(bm25_scores).unwrap_or(0.0);\n            self.block_max_score_cache = Some(block_max_score);\n            return block_max_score;\n        }\n        // We do not have access to any good block max value. We return bm25_weight.max_score()\n        // as it is a valid upperbound.\n        //\n        // We do not cache it however, so that it gets computed when once block is loaded.\n        bm25_weight.max_score()\n    }\n\n    pub(crate) fn freq_reading_option(&self) -> FreqReadingOption {\n        self.freq_reading_option\n    }\n\n    // Resets the block segment postings on another position\n    // in the postings file.\n    //\n    // This is useful for enumerating through a list of terms,\n    // and consuming the associated posting lists while avoiding\n    // reallocating a `BlockSegmentPostings`.\n    //\n    // # Warning\n    //\n    // This does not reset the positions list.\n    pub(crate) fn reset(&mut self, doc_freq: u32, postings_data: OwnedBytes) -> io::Result<()> {\n        let (skip_data_opt, postings_data) =\n            split_into_skips_and_postings(doc_freq, postings_data)?;\n        self.data = postings_data;\n        self.block_max_score_cache = None;\n        self.block_loaded = false;\n        if let Some(skip_data) = skip_data_opt {\n            self.skip_reader.reset(skip_data, doc_freq);\n        } else {\n            self.skip_reader.reset(OwnedBytes::empty(), doc_freq);\n        }\n        self.doc_freq = doc_freq;\n        self.load_block();\n        Ok(())\n    }\n\n    /// Returns the overall number of documents in the block postings.\n    /// It does not take in account whether documents are deleted or not.\n    ///\n    /// This `doc_freq` is simply the sum of the length of all of the blocks\n    /// length, and it does not take in account deleted documents.\n    pub fn doc_freq(&self) -> u32 {\n        self.doc_freq\n    }\n\n    /// Returns the array of docs in the current block.\n    ///\n    /// Before the first call to `.advance()`, the block\n    /// returned by `.docs()` is empty.\n    #[inline]\n    pub fn docs(&self) -> &[DocId] {\n        debug_assert!(self.block_is_loaded());\n        self.doc_decoder.output_array()\n    }\n\n    /// Return the document at index `idx` of the block.\n    #[inline]\n    pub fn doc(&self, idx: usize) -> u32 {\n        self.doc_decoder.output(idx)\n    }\n\n    /// Return the array of `term freq` in the block.\n    #[inline]\n    pub fn freqs(&self) -> &[u32] {\n        debug_assert!(self.block_is_loaded());\n        self.freq_decoder.output_array()\n    }\n\n    /// Return the frequency at index `idx` of the block.\n    #[inline]\n    pub fn freq(&self, idx: usize) -> u32 {\n        debug_assert!(self.block_is_loaded());\n        self.freq_decoder.output(idx)\n    }\n\n    /// Returns the length of the current block.\n    ///\n    /// All blocks have a length of `NUM_DOCS_PER_BLOCK`,\n    /// except the last block that may have a length\n    /// of any number between 1 and `NUM_DOCS_PER_BLOCK - 1`\n    #[inline]\n    pub fn block_len(&self) -> usize {\n        debug_assert!(self.block_is_loaded());\n        self.doc_decoder.output_len\n    }\n\n    /// Position on a block that may contains `target_doc`.\n    ///\n    /// If all docs are smaller than target, the block loaded may be empty,\n    /// or be the last an incomplete VInt block.\n    pub fn seek(&mut self, target_doc: DocId) -> usize {\n        // Move to the block that might contain our document.\n        self.seek_block(target_doc);\n        self.load_block();\n\n        // At this point we are on the block that might contain our document.\n        let doc = self.doc_decoder.seek_within_block(target_doc);\n\n        // The last block is not full and padded with TERMINATED,\n        // so we are guaranteed to have at least one value (real or padding)\n        // that is >= target_doc.\n        debug_assert!(doc < COMPRESSION_BLOCK_SIZE);\n\n        // `doc` is now the first element >= `target_doc`.\n        // If all docs are smaller than target, the current block is incomplete and padded\n        // with TERMINATED. After the search, the cursor points to the first TERMINATED.\n        doc\n    }\n\n    pub(crate) fn position_offset(&self) -> u64 {\n        self.skip_reader.position_offset()\n    }\n\n    /// Dangerous API! This calls seeks the next block on the skip list,\n    /// but does not `.load_block()` afterwards.\n    ///\n    /// `.load_block()` needs to be called manually afterwards.\n    /// If all docs are smaller than target, the block loaded may be empty,\n    /// or be the last an incomplete VInt block.\n    pub(crate) fn seek_block(&mut self, target_doc: DocId) {\n        if self.skip_reader.seek(target_doc) {\n            self.block_max_score_cache = None;\n            self.block_loaded = false;\n        }\n    }\n\n    pub(crate) fn block_is_loaded(&self) -> bool {\n        self.block_loaded\n    }\n\n    pub(crate) fn load_block(&mut self) {\n        if self.block_is_loaded() {\n            return;\n        }\n        let offset = self.skip_reader.byte_offset();\n        match self.skip_reader.block_info() {\n            BlockInfo::BitPacked {\n                doc_num_bits,\n                strict_delta_encoded,\n                tf_num_bits,\n                ..\n            } => {\n                decode_bitpacked_block(\n                    &mut self.doc_decoder,\n                    if let FreqReadingOption::ReadFreq = self.freq_reading_option {\n                        Some(&mut self.freq_decoder)\n                    } else {\n                        None\n                    },\n                    &self.data.as_slice()[offset..],\n                    self.skip_reader.last_doc_in_previous_block,\n                    doc_num_bits,\n                    tf_num_bits,\n                    strict_delta_encoded,\n                );\n            }\n            BlockInfo::VInt { num_docs } => {\n                let data = {\n                    if num_docs == 0 {\n                        &[]\n                    } else {\n                        &self.data.as_slice()[offset..]\n                    }\n                };\n                decode_vint_block(\n                    &mut self.doc_decoder,\n                    if let FreqReadingOption::ReadFreq = self.freq_reading_option {\n                        Some(&mut self.freq_decoder)\n                    } else {\n                        None\n                    },\n                    data,\n                    self.skip_reader.last_doc_in_previous_block,\n                    num_docs as usize,\n                );\n            }\n        }\n        self.block_loaded = true;\n    }\n\n    /// Advance to the next block.\n    pub fn advance(&mut self) {\n        self.skip_reader.advance();\n        self.block_loaded = false;\n        self.block_max_score_cache = None;\n        self.load_block();\n    }\n\n    /// Returns an empty segment postings object\n    pub fn empty() -> BlockSegmentPostings {\n        BlockSegmentPostings {\n            doc_decoder: BlockDecoder::with_val(TERMINATED),\n            block_loaded: true,\n            freq_decoder: BlockDecoder::with_val(1),\n            freq_reading_option: FreqReadingOption::NoFreq,\n            block_max_score_cache: None,\n            doc_freq: 0,\n            data: OwnedBytes::empty(),\n            skip_reader: SkipReader::new(OwnedBytes::empty(), 0, IndexRecordOption::Basic),\n        }\n    }\n\n    pub(crate) fn skip_reader(&self) -> &SkipReader {\n        &self.skip_reader\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use common::HasLen;\n\n    use super::BlockSegmentPostings;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::index::Index;\n    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n    use crate::postings::postings::Postings;\n    use crate::postings::SegmentPostings;\n    use crate::schema::{IndexRecordOption, Schema, Term, INDEXED};\n    use crate::DocId;\n\n    #[test]\n    fn test_empty_segment_postings() {\n        let mut postings = SegmentPostings::empty();\n        assert_eq!(postings.doc(), TERMINATED);\n        assert_eq!(postings.advance(), TERMINATED);\n        assert_eq!(postings.advance(), TERMINATED);\n        assert_eq!(postings.doc_freq(), 0);\n        assert_eq!(postings.len(), 0);\n    }\n\n    #[test]\n    fn test_empty_postings_doc_returns_terminated() {\n        let mut postings = SegmentPostings::empty();\n        assert_eq!(postings.doc(), TERMINATED);\n        assert_eq!(postings.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_empty_postings_doc_term_freq_returns_0() {\n        let postings = SegmentPostings::empty();\n        assert_eq!(postings.term_freq(), 1);\n    }\n\n    #[test]\n    fn test_empty_block_segment_postings() {\n        let mut postings = BlockSegmentPostings::empty();\n        assert!(postings.docs().is_empty());\n        assert_eq!(postings.doc_freq(), 0);\n        postings.advance();\n        assert!(postings.docs().is_empty());\n        assert_eq!(postings.doc_freq(), 0);\n    }\n\n    #[test]\n    fn test_block_segment_postings() -> crate::Result<()> {\n        let mut block_segments = build_block_postings(&(0..100_000).collect::<Vec<u32>>())?;\n        let mut offset: u32 = 0u32;\n        // checking that the `doc_freq` is correct\n        assert_eq!(block_segments.doc_freq(), 100_000);\n        loop {\n            let block = block_segments.docs();\n            if block.is_empty() {\n                break;\n            }\n            for (i, doc) in block.iter().cloned().enumerate() {\n                assert_eq!(offset + (i as u32), doc);\n            }\n            offset += block.len() as u32;\n            block_segments.advance();\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_skip_right_at_new_block() -> crate::Result<()> {\n        let mut doc_ids = (0..128).collect::<Vec<u32>>();\n        // 128 is missing\n        doc_ids.push(129);\n        doc_ids.push(130);\n        {\n            let block_segments = build_block_postings(&doc_ids)?;\n            let mut docset = SegmentPostings::from_block_postings(block_segments, None);\n            assert_eq!(docset.seek(128), 129);\n            assert_eq!(docset.doc(), 129);\n            assert_eq!(docset.advance(), 130);\n            assert_eq!(docset.doc(), 130);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let block_segments = build_block_postings(&doc_ids).unwrap();\n            let mut docset = SegmentPostings::from_block_postings(block_segments, None);\n            assert_eq!(docset.seek(129), 129);\n            assert_eq!(docset.doc(), 129);\n            assert_eq!(docset.advance(), 130);\n            assert_eq!(docset.doc(), 130);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let block_segments = build_block_postings(&doc_ids)?;\n            let mut docset = SegmentPostings::from_block_postings(block_segments, None);\n            assert_eq!(docset.doc(), 0);\n            assert_eq!(docset.seek(131), TERMINATED);\n            assert_eq!(docset.doc(), TERMINATED);\n        }\n        Ok(())\n    }\n\n    fn build_block_postings(docs: &[DocId]) -> crate::Result<BlockSegmentPostings> {\n        let mut schema_builder = Schema::builder();\n        let int_field = schema_builder.add_u64_field(\"id\", INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        let mut last_doc = 0u32;\n        for &doc in docs {\n            for _ in last_doc..doc {\n                index_writer.add_document(doc!(int_field=>1u64))?;\n            }\n            index_writer.add_document(doc!(int_field=>0u64))?;\n            last_doc = doc + 1;\n        }\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let inverted_index = segment_reader.inverted_index(int_field).unwrap();\n        let term = Term::from_field_u64(int_field, 0u64);\n        let term_info = inverted_index.get_term_info(&term)?.unwrap();\n        let block_postings = inverted_index\n            .read_block_postings_from_terminfo(&term_info, IndexRecordOption::Basic)?;\n        Ok(block_postings)\n    }\n\n    #[test]\n    fn test_block_segment_postings_seek() -> crate::Result<()> {\n        let mut docs = vec![0];\n        for i in 0..1300 {\n            docs.push((i * i / 100) + i);\n        }\n        let mut block_postings = build_block_postings(&docs[..])?;\n        for i in &[0, 424, 10000] {\n            block_postings.seek(*i);\n            let docs = block_postings.docs();\n            assert!(docs[0] <= *i);\n            assert!(docs.last().cloned().unwrap_or(0u32) >= *i);\n        }\n        block_postings.seek(100_000);\n        assert_eq!(block_postings.doc(COMPRESSION_BLOCK_SIZE - 1), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_reset_block_segment_postings() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let int_field = schema_builder.add_u64_field(\"id\", INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        // create two postings list, one containing even number,\n        // the other containing odd numbers.\n        for i in 0..6 {\n            let doc = doc!(int_field=> (i % 2) as u64);\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let segment_reader = searcher.segment_reader(0);\n\n        let mut block_segments;\n        {\n            let term = Term::from_field_u64(int_field, 0u64);\n            let inverted_index = segment_reader.inverted_index(int_field)?;\n            let term_info = inverted_index.get_term_info(&term)?.unwrap();\n            block_segments = inverted_index\n                .read_block_postings_from_terminfo(&term_info, IndexRecordOption::Basic)?;\n        }\n        assert_eq!(block_segments.docs(), &[0, 2, 4]);\n        {\n            let term = Term::from_field_u64(int_field, 1u64);\n            let inverted_index = segment_reader.inverted_index(int_field)?;\n            let term_info = inverted_index.get_term_info(&term)?.unwrap();\n            inverted_index.reset_block_postings_from_terminfo(&term_info, &mut block_segments)?;\n        }\n        assert_eq!(block_segments.docs(), &[1, 3, 5]);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/postings/compression/mod.rs",
    "content": "use bitpacking::{BitPacker, BitPacker4x};\n\npub const COMPRESSION_BLOCK_SIZE: usize = BitPacker4x::BLOCK_LEN;\n// in vint encoding, each byte stores 7 bits of data, so we need at most 32 / 7 = 4.57 bytes to\n// store a u32 in the worst case, rounding up to 5 bytes total\nconst MAX_VINT_SIZE: usize = 5;\nconst COMPRESSED_BLOCK_MAX_SIZE: usize = COMPRESSION_BLOCK_SIZE * MAX_VINT_SIZE;\n\nmod vint;\n\n/// Returns the size in bytes of a compressed block, given `num_bits`.\n#[inline]\npub fn compressed_block_size(num_bits: u8) -> usize {\n    (num_bits as usize) * COMPRESSION_BLOCK_SIZE / 8\n}\n\npub struct BlockEncoder {\n    bitpacker: BitPacker4x,\n    pub output: [u8; COMPRESSED_BLOCK_MAX_SIZE],\n}\n\nimpl Default for BlockEncoder {\n    fn default() -> Self {\n        BlockEncoder::new()\n    }\n}\n\nimpl BlockEncoder {\n    pub fn new() -> BlockEncoder {\n        BlockEncoder {\n            bitpacker: BitPacker4x::new(),\n            output: [0u8; COMPRESSED_BLOCK_MAX_SIZE],\n        }\n    }\n\n    pub fn compress_block_sorted(&mut self, block: &[u32], offset: u32) -> (u8, &[u8]) {\n        // if offset is zero, convert it to None. This is correct as long as we do the same when\n        // decompressing. It's required in case the block starts with an actual zero.\n        let offset = if offset == 0u32 { None } else { Some(offset) };\n\n        let num_bits = self.bitpacker.num_bits_strictly_sorted(offset, block);\n        let written_size =\n            self.bitpacker\n                .compress_strictly_sorted(offset, block, &mut self.output[..], num_bits);\n        (num_bits, &self.output[..written_size])\n    }\n\n    /// Compress a single block of unsorted numbers.\n    ///\n    /// If `minus_one_encoded` is set, each value must be >= 1, and will be encoded in a sligly\n    /// more compact format. This is useful for some values where 0 isn't a correct value, such\n    /// as term frequency, but isn't correct for some usages like position lists, where 0 can\n    /// appear.\n    pub fn compress_block_unsorted(\n        &mut self,\n        block: &[u32],\n        minus_one_encoded: bool,\n    ) -> (u8, &[u8]) {\n        debug_assert!(!minus_one_encoded || !block.contains(&0));\n\n        let mut block_minus_one = [0; COMPRESSION_BLOCK_SIZE];\n        let block = if minus_one_encoded {\n            for (elem_min_one, elem) in block_minus_one.iter_mut().zip(block) {\n                *elem_min_one = elem - 1;\n            }\n            &block_minus_one\n        } else {\n            block\n        };\n\n        let num_bits = self.bitpacker.num_bits(block);\n        let written_size = self\n            .bitpacker\n            .compress(block, &mut self.output[..], num_bits);\n        (num_bits, &self.output[..written_size])\n    }\n}\n\n#[derive(Clone)]\npub struct BlockDecoder {\n    bitpacker: BitPacker4x,\n    output: [u32; COMPRESSION_BLOCK_SIZE],\n    pub output_len: usize,\n}\n\nimpl Default for BlockDecoder {\n    fn default() -> Self {\n        BlockDecoder::with_val(0u32)\n    }\n}\n\nimpl BlockDecoder {\n    pub fn with_val(val: u32) -> BlockDecoder {\n        BlockDecoder {\n            bitpacker: BitPacker4x::new(),\n            output: [val; COMPRESSION_BLOCK_SIZE],\n            output_len: 0,\n        }\n    }\n\n    /// Decompress block of sorted integers.\n    ///\n    /// `strict_delta` depends on what encoding was used. Older version of tantivy never use strict\n    /// deltas, newer versions always use them.\n    pub fn uncompress_block_sorted(\n        &mut self,\n        compressed_data: &[u8],\n        offset: u32,\n        num_bits: u8,\n        strict_delta: bool,\n    ) -> usize {\n        if strict_delta {\n            let offset = std::num::NonZeroU32::new(offset).map(std::num::NonZeroU32::get);\n\n            self.output_len = COMPRESSION_BLOCK_SIZE;\n            self.bitpacker.decompress_strictly_sorted(\n                offset,\n                compressed_data,\n                &mut self.output,\n                num_bits,\n            )\n        } else {\n            self.output_len = COMPRESSION_BLOCK_SIZE;\n            self.bitpacker\n                .decompress_sorted(offset, compressed_data, &mut self.output, num_bits)\n        }\n    }\n\n    /// Decompress block of unsorted integers.\n    ///\n    /// `minus_one_encoded` depends on what encoding was used. Older version of tantivy never use\n    /// that encoding. Newer version use it for some structures, but not all. See the corresponding\n    /// call to `BlockEncoder::compress_block_unsorted`.\n    pub fn uncompress_block_unsorted(\n        &mut self,\n        compressed_data: &[u8],\n        num_bits: u8,\n        minus_one_encoded: bool,\n    ) -> usize {\n        self.output_len = COMPRESSION_BLOCK_SIZE;\n        let res = self\n            .bitpacker\n            .decompress(compressed_data, &mut self.output, num_bits);\n        if minus_one_encoded {\n            for val in &mut self.output {\n                *val += 1;\n            }\n        }\n        res\n    }\n\n    #[inline]\n    pub fn output_array(&self) -> &[u32] {\n        &self.output[..self.output_len]\n    }\n\n    /// Return in-block index of first value >= `target`.\n    /// Uses the padded buffer to enable branchless search.\n    #[inline]\n    pub(crate) fn seek_within_block(&self, target: u32) -> usize {\n        crate::postings::branchless_binary_search(&self.output, target)\n    }\n\n    #[inline]\n    pub fn output(&self, idx: usize) -> u32 {\n        self.output[idx]\n    }\n}\n\npub trait VIntEncoder {\n    /// Compresses an array of `u32` integers,\n    /// using [delta-encoding](https://en.wikipedia.org/wiki/Delta_encoding)\n    /// and variable bytes encoding.\n    ///\n    /// The method takes an array of ints to compress, and returns\n    /// a `&[u8]` representing the compressed data.\n    ///\n    /// The method also takes an offset to give the value of the\n    /// hypothetical previous element in the delta-encoding.\n    fn compress_vint_sorted(&mut self, input: &[u32], offset: u32) -> &[u8];\n\n    /// Compresses an array of `u32` integers,\n    /// using variable bytes encoding.\n    ///\n    /// The method takes an array of ints to compress, and returns\n    /// a `&[u8]` representing the compressed data.\n    fn compress_vint_unsorted(&mut self, input: &[u32]) -> &[u8];\n}\n\npub trait VIntDecoder {\n    /// Uncompress an array of `u32` integers,\n    /// that were compressed using [delta-encoding](https://en.wikipedia.org/wiki/Delta_encoding)\n    /// and variable bytes encoding.\n    ///\n    /// The method takes a number of int to decompress, and returns\n    /// the amount of bytes that were read to decompress them.\n    ///\n    /// The method also takes an offset to give the value of the\n    /// hypothetical previous element in the delta-encoding.\n    ///\n    /// For instance, if delta encoded are `1, 3, 9`, and the\n    /// `offset` is 5, then the output will be:\n    /// `5 + 1 = 6, 6 + 3= 9, 9 + 9 = 18`\n    ///\n    /// The value given in `padding` will be used to fill the remaining `128 - num_els` values.\n    fn uncompress_vint_sorted(\n        &mut self,\n        compressed_data: &[u8],\n        offset: u32,\n        num_els: usize,\n        padding: u32,\n    ) -> usize;\n\n    /// Uncompress an array of `u32s`, compressed using variable\n    /// byte encoding.\n    ///\n    /// The method takes a number of int to decompress, and returns\n    /// the amount of bytes that were read to decompress them.\n    ///\n    /// The value given in `padding` will be used to fill the remaining `128 - num_els` values.\n    fn uncompress_vint_unsorted(\n        &mut self,\n        compressed_data: &[u8],\n        num_els: usize,\n        padding: u32,\n    ) -> usize;\n\n    fn uncompress_vint_unsorted_until_end(&mut self, compressed_data: &[u8]);\n}\n\nimpl VIntEncoder for BlockEncoder {\n    fn compress_vint_sorted(&mut self, input: &[u32], offset: u32) -> &[u8] {\n        vint::compress_sorted(input, &mut self.output, offset)\n    }\n\n    fn compress_vint_unsorted(&mut self, input: &[u32]) -> &[u8] {\n        vint::compress_unsorted(input, &mut self.output)\n    }\n}\n\nimpl VIntDecoder for BlockDecoder {\n    fn uncompress_vint_sorted(\n        &mut self,\n        compressed_data: &[u8],\n        offset: u32,\n        num_els: usize,\n        padding: u32,\n    ) -> usize {\n        self.output_len = num_els;\n        self.output.iter_mut().for_each(|el| *el = padding);\n        vint::uncompress_sorted(compressed_data, &mut self.output[..num_els], offset)\n    }\n\n    fn uncompress_vint_unsorted(\n        &mut self,\n        compressed_data: &[u8],\n        num_els: usize,\n        padding: u32,\n    ) -> usize {\n        self.output_len = num_els;\n        self.output.iter_mut().for_each(|el| *el = padding);\n        vint::uncompress_unsorted(compressed_data, &mut self.output[..num_els])\n    }\n\n    fn uncompress_vint_unsorted_until_end(&mut self, compressed_data: &[u8]) {\n        let num_els = vint::uncompress_unsorted_until_end(compressed_data, &mut self.output);\n        self.output_len = num_els;\n    }\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use super::*;\n    use crate::TERMINATED;\n\n    #[test]\n    fn test_encode_sorted_block() {\n        let vals: Vec<u32> = (0u32..128u32).map(|i| i * 7).collect();\n        let mut encoder = BlockEncoder::new();\n        let (num_bits, compressed_data) = encoder.compress_block_sorted(&vals, 0);\n        let mut decoder = BlockDecoder::default();\n        {\n            let consumed_num_bytes =\n                decoder.uncompress_block_sorted(compressed_data, 0, num_bits, true);\n            assert_eq!(consumed_num_bytes, compressed_data.len());\n        }\n        for i in 0..128 {\n            assert_eq!(vals[i], decoder.output(i));\n        }\n    }\n\n    #[test]\n    fn test_encode_sorted_block_with_offset() {\n        let vals: Vec<u32> = (0u32..128u32).map(|i| 11 + i * 7).collect();\n        let mut encoder = BlockEncoder::default();\n        let (num_bits, compressed_data) = encoder.compress_block_sorted(&vals, 10);\n        let mut decoder = BlockDecoder::default();\n        {\n            let consumed_num_bytes =\n                decoder.uncompress_block_sorted(compressed_data, 10, num_bits, true);\n            assert_eq!(consumed_num_bytes, compressed_data.len());\n        }\n        for i in 0..128 {\n            assert_eq!(vals[i], decoder.output(i));\n        }\n    }\n\n    #[test]\n    fn test_encode_sorted_block_with_junk() {\n        let mut compressed: Vec<u8> = Vec::new();\n        let n = 128;\n        let vals: Vec<u32> = (0..n).map(|i| 11u32 + (i as u32) * 7u32).collect();\n        let mut encoder = BlockEncoder::default();\n        let (num_bits, compressed_data) = encoder.compress_block_sorted(&vals, 10);\n        compressed.extend_from_slice(compressed_data);\n        compressed.push(173u8);\n        let mut decoder = BlockDecoder::default();\n        {\n            let consumed_num_bytes =\n                decoder.uncompress_block_sorted(&compressed, 10, num_bits, true);\n            assert_eq!(consumed_num_bytes, compressed.len() - 1);\n            assert_eq!(compressed[consumed_num_bytes], 173u8);\n        }\n        for i in 0..n {\n            assert_eq!(vals[i], decoder.output(i));\n        }\n    }\n\n    #[test]\n    fn test_encode_unsorted_block_with_junk() {\n        for minus_one_encode in [false, true] {\n            let mut compressed: Vec<u8> = Vec::new();\n            let n = 128;\n            let vals: Vec<u32> = (0..n).map(|i| 11u32 + (i as u32) * 7u32 % 12).collect();\n            let mut encoder = BlockEncoder::default();\n            let (num_bits, compressed_data) =\n                encoder.compress_block_unsorted(&vals, minus_one_encode);\n            compressed.extend_from_slice(compressed_data);\n            compressed.push(173u8);\n            let mut decoder = BlockDecoder::default();\n            {\n                let consumed_num_bytes =\n                    decoder.uncompress_block_unsorted(&compressed, num_bits, minus_one_encode);\n                assert_eq!(consumed_num_bytes + 1, compressed.len());\n                assert_eq!(compressed[consumed_num_bytes], 173u8);\n            }\n            for i in 0..n {\n                assert_eq!(vals[i], decoder.output(i));\n            }\n        }\n    }\n\n    #[test]\n    fn test_block_decoder_initialization() {\n        let block = BlockDecoder::with_val(TERMINATED);\n        assert_eq!(block.output(0), TERMINATED);\n    }\n    #[test]\n    fn test_encode_vint() {\n        const PADDING_VALUE: u32 = 234_234_345u32;\n        let expected_length = 154;\n        let mut encoder = BlockEncoder::new();\n        let input: Vec<u32> = (0u32..123u32).map(|i| 4 + i * 7 / 2).collect();\n        for offset in &[0u32, 1u32, 2u32] {\n            let encoded_data = encoder.compress_vint_sorted(&input, *offset);\n            assert!(encoded_data.len() <= expected_length);\n            let mut decoder = BlockDecoder::default();\n            let consumed_num_bytes =\n                decoder.uncompress_vint_sorted(encoded_data, *offset, input.len(), PADDING_VALUE);\n            assert_eq!(consumed_num_bytes, encoded_data.len());\n            assert_eq!(input, decoder.output_array());\n            for i in input.len()..COMPRESSION_BLOCK_SIZE {\n                assert_eq!(decoder.output(i), PADDING_VALUE);\n            }\n        }\n    }\n\n    #[test]\n    fn test_compress_vint_unsorted_does_not_overflow() {\n        let mut encoder = BlockEncoder::new();\n        let input: Vec<u32> = vec![u32::MAX; COMPRESSION_BLOCK_SIZE];\n        encoder.compress_vint_unsorted(&input);\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use rand::rngs::StdRng;\n    use rand::{Rng, SeedableRng};\n    use test::Bencher;\n\n    use super::*;\n    use crate::TERMINATED;\n\n    fn generate_array_with_seed(n: usize, ratio: f64, seed_val: u8) -> Vec<u32> {\n        let mut seed: [u8; 32] = [0; 32];\n        seed[31] = seed_val;\n        let mut rng = StdRng::from_seed(seed);\n        (0u32..)\n            .filter(|_| rng.random_bool(ratio))\n            .take(n)\n            .collect()\n    }\n\n    pub fn generate_array(n: usize, ratio: f64) -> Vec<u32> {\n        generate_array_with_seed(n, ratio, 4)\n    }\n\n    #[bench]\n    fn bench_compress(b: &mut Bencher) {\n        let mut encoder = BlockEncoder::new();\n        let data = generate_array(COMPRESSION_BLOCK_SIZE, 0.1);\n        b.iter(|| {\n            encoder.compress_block_sorted(&data, 0u32);\n        });\n    }\n\n    #[bench]\n    fn bench_uncompress(b: &mut Bencher) {\n        let mut encoder = BlockEncoder::new();\n        let data = generate_array(COMPRESSION_BLOCK_SIZE, 0.1);\n        let (num_bits, compressed) = encoder.compress_block_sorted(&data, 0u32);\n        let mut decoder = BlockDecoder::default();\n        b.iter(|| {\n            decoder.uncompress_block_sorted(compressed, 0u32, num_bits, true);\n        });\n    }\n\n    //#[test]\n    // fn test_all_docs_compression_numbits() {\n    // for expected_num_bits in 0u8.. {\n    // let mut data = [0u32; 128];\n    // if expected_num_bits > 0 {\n    // data[0] = (1u64 << (expected_num_bits as usize) - 1) as u32;\n    //}\n    // let mut encoder = BlockEncoder::new();\n    // let (num_bits, compressed) = encoder.compress_block_unsorted(&data);\n    // assert_eq!(compressed.len(), compressed_block_size(num_bits));\n    //}\n\n    const NUM_INTS_BENCH_VINT: usize = 10;\n\n    #[bench]\n    fn bench_compress_vint(b: &mut Bencher) {\n        let mut encoder = BlockEncoder::new();\n        let data = generate_array(NUM_INTS_BENCH_VINT, 0.001);\n        b.iter(|| {\n            encoder.compress_vint_sorted(&data, 0u32);\n        });\n    }\n\n    #[bench]\n    fn bench_uncompress_vint(b: &mut Bencher) {\n        let mut encoder = BlockEncoder::new();\n        let data = generate_array(NUM_INTS_BENCH_VINT, 0.001);\n        let compressed = encoder.compress_vint_sorted(&data, 0u32);\n        let mut decoder = BlockDecoder::default();\n        b.iter(|| {\n            decoder.uncompress_vint_sorted(compressed, 0u32, NUM_INTS_BENCH_VINT, TERMINATED);\n        });\n    }\n}\n"
  },
  {
    "path": "src/postings/compression/vint.rs",
    "content": "#[inline]\npub fn compress_sorted<'a>(input: &[u32], output: &'a mut [u8], mut offset: u32) -> &'a [u8] {\n    let mut byte_written = 0;\n    for &v in input {\n        let mut to_encode: u32 = v - offset;\n        offset = v;\n        loop {\n            let next_byte: u8 = (to_encode % 128u32) as u8;\n            to_encode /= 128u32;\n            if to_encode == 0u32 {\n                output[byte_written] = next_byte | 128u8;\n                byte_written += 1;\n                break;\n            } else {\n                output[byte_written] = next_byte;\n                byte_written += 1;\n            }\n        }\n    }\n    &output[..byte_written]\n}\n\n#[inline]\npub(crate) fn compress_unsorted<'a>(input: &[u32], output: &'a mut [u8]) -> &'a [u8] {\n    let mut byte_written = 0;\n    for &v in input {\n        let mut to_encode: u32 = v;\n        loop {\n            let next_byte: u8 = (to_encode % 128u32) as u8;\n            to_encode /= 128u32;\n            if to_encode == 0u32 {\n                output[byte_written] = next_byte | 128u8;\n                byte_written += 1;\n                break;\n            } else {\n                output[byte_written] = next_byte;\n                byte_written += 1;\n            }\n        }\n    }\n    &output[..byte_written]\n}\n\n#[inline]\npub fn uncompress_sorted(compressed_data: &[u8], output: &mut [u32], offset: u32) -> usize {\n    let mut read_byte = 0;\n    let mut result = offset;\n    for output_mut in output.iter_mut() {\n        let mut shift = 0u32;\n        loop {\n            let cur_byte = compressed_data[read_byte];\n            read_byte += 1;\n            result += u32::from(cur_byte % 128u8) << shift;\n            if cur_byte & 128u8 != 0u8 {\n                break;\n            }\n            shift += 7;\n        }\n        *output_mut = result;\n    }\n    read_byte\n}\n\n#[inline]\npub(crate) fn uncompress_unsorted(compressed_data: &[u8], output_arr: &mut [u32]) -> usize {\n    let mut num_read_bytes = 0;\n    for output_mut in output_arr.iter_mut() {\n        let mut result = 0u32;\n        let mut shift = 0u32;\n        loop {\n            let cur_byte = compressed_data[num_read_bytes];\n            num_read_bytes += 1;\n            result += u32::from(cur_byte % 128u8) << shift;\n            if cur_byte & 128u8 != 0u8 {\n                break;\n            }\n            shift += 7;\n        }\n        *output_mut = result;\n    }\n    num_read_bytes\n}\n\n#[inline]\npub(crate) fn uncompress_unsorted_until_end(\n    compressed_data: &[u8],\n    output_arr: &mut [u32],\n) -> usize {\n    let mut num_read_bytes = 0;\n    for (num_ints_written, output_mut) in output_arr.iter_mut().enumerate() {\n        if compressed_data.len() == num_read_bytes {\n            return num_ints_written;\n        }\n        let mut result = 0u32;\n        let mut shift = 0u32;\n        loop {\n            let cur_byte = compressed_data[num_read_bytes];\n            num_read_bytes += 1;\n            result += u32::from(cur_byte % 128u8) << shift;\n            if cur_byte & 128u8 != 0u8 {\n                break;\n            }\n            shift += 7;\n        }\n        *output_mut = result;\n    }\n    output_arr.len()\n}\n"
  },
  {
    "path": "src/postings/indexing_context.rs",
    "content": "use stacker::{ArenaHashMap, MemoryArena};\n\nuse crate::indexer::path_to_unordered_id::PathToUnorderedId;\n\n/// IndexingContext contains all of the transient memory arenas\n/// required for building the inverted index.\npub(crate) struct IndexingContext {\n    /// The term index is an adhoc hashmap,\n    /// itself backed by a dedicated memory arena.\n    pub term_index: ArenaHashMap,\n    /// Arena is a memory arena that stores posting lists / term frequencies / positions.\n    pub arena: MemoryArena,\n    pub path_to_unordered_id: PathToUnorderedId,\n}\n\nimpl IndexingContext {\n    /// Create a new IndexingContext given the size of the term hash map.\n    pub(crate) fn new(table_size: usize) -> IndexingContext {\n        let term_index = ArenaHashMap::with_capacity(table_size);\n        IndexingContext {\n            arena: MemoryArena::default(),\n            term_index,\n            path_to_unordered_id: PathToUnorderedId::default(),\n        }\n    }\n\n    /// Returns the memory usage for the inverted index memory arenas, in bytes.\n    pub(crate) fn mem_usage(&self) -> usize {\n        self.term_index.mem_usage() + self.arena.mem_usage()\n    }\n}\n"
  },
  {
    "path": "src/postings/json_postings_writer.rs",
    "content": "use std::io;\n\nuse common::json_path_writer::JSON_END_OF_PATH;\nuse stacker::Addr;\n\nuse crate::indexer::indexing_term::IndexingTerm;\nuse crate::indexer::path_to_unordered_id::OrderedPathId;\nuse crate::postings::postings_writer::SpecializedPostingsWriter;\nuse crate::postings::recorder::{BufferLender, DocIdRecorder, Recorder};\nuse crate::postings::{FieldSerializer, IndexingContext, IndexingPosition, PostingsWriter};\nuse crate::schema::{Field, Type};\nuse crate::tokenizer::TokenStream;\nuse crate::DocId;\n\n/// The `JsonPostingsWriter` is odd in that it relies on a hidden contract:\n///\n/// `subscribe` is called directly to index non-text tokens, while\n/// `index_text` is used to index text.\n#[derive(Default)]\npub(crate) struct JsonPostingsWriter<Rec: Recorder> {\n    str_posting_writer: SpecializedPostingsWriter<Rec>,\n    non_str_posting_writer: SpecializedPostingsWriter<DocIdRecorder>,\n}\n\nimpl<Rec: Recorder> From<JsonPostingsWriter<Rec>> for Box<dyn PostingsWriter> {\n    fn from(json_postings_writer: JsonPostingsWriter<Rec>) -> Box<dyn PostingsWriter> {\n        Box::new(json_postings_writer)\n    }\n}\n\nimpl<Rec: Recorder> PostingsWriter for JsonPostingsWriter<Rec> {\n    #[inline]\n    fn subscribe(\n        &mut self,\n        doc: crate::DocId,\n        pos: u32,\n        term: &IndexingTerm,\n        ctx: &mut IndexingContext,\n    ) {\n        self.non_str_posting_writer.subscribe(doc, pos, term, ctx);\n    }\n\n    fn index_text(\n        &mut self,\n        doc_id: DocId,\n        token_stream: &mut dyn TokenStream,\n        term_buffer: &mut IndexingTerm,\n        ctx: &mut IndexingContext,\n        indexing_position: &mut IndexingPosition,\n    ) {\n        self.str_posting_writer.index_text(\n            doc_id,\n            token_stream,\n            term_buffer,\n            ctx,\n            indexing_position,\n        );\n    }\n\n    /// The actual serialization format is handled by the `PostingsSerializer`.\n    fn serialize(\n        &self,\n        ordered_term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],\n        ordered_id_to_path: &[&str],\n        ctx: &IndexingContext,\n        serializer: &mut FieldSerializer,\n    ) -> io::Result<()> {\n        let mut term_buffer = JsonTermSerializer(Vec::with_capacity(48));\n        let mut buffer_lender = BufferLender::default();\n        let mut prev_term_id = u32::MAX;\n        let mut term_path_len = 0; // this will be set in the first iteration\n        for (_field, path_id, term, addr) in ordered_term_addrs {\n            if prev_term_id != path_id.path_id() {\n                term_buffer.clear();\n                term_buffer.append_json_path(ordered_id_to_path[path_id.path_id() as usize]);\n                term_path_len = term_buffer.len();\n                prev_term_id = path_id.path_id();\n            }\n            term_buffer.truncate(term_path_len);\n            term_buffer.append_bytes(term);\n\n            let typ = Type::from_code(term[0]).expect(\"Invalid type code in JSON term\");\n            if typ == Type::Str {\n                SpecializedPostingsWriter::<Rec>::serialize_one_term(\n                    term_buffer.as_bytes(),\n                    *addr,\n                    &mut buffer_lender,\n                    ctx,\n                    serializer,\n                )?;\n            } else {\n                SpecializedPostingsWriter::<DocIdRecorder>::serialize_one_term(\n                    term_buffer.as_bytes(),\n                    *addr,\n                    &mut buffer_lender,\n                    ctx,\n                    serializer,\n                )?;\n            }\n        }\n        Ok(())\n    }\n\n    fn total_num_tokens(&self) -> u64 {\n        self.str_posting_writer.total_num_tokens() + self.non_str_posting_writer.total_num_tokens()\n    }\n}\n\n/// Helper to build the JSON term bytes that land in the term dictionary.\n/// Format: `[json path utf8][JSON_END_OF_PATH][type tag][payload]`\nstruct JsonTermSerializer(Vec<u8>);\nimpl JsonTermSerializer {\n    /// Appends a JSON path to the Term.\n    /// The path is terminated by a special end-of-path 0 byte.\n    #[inline]\n    pub fn append_json_path(&mut self, path: &str) {\n        let bytes = path.as_bytes();\n        // Replace any occurrence of the end-of-path byte with Ascii '0' byte.\n        if bytes.contains(&JSON_END_OF_PATH) {\n            self.0.extend(\n                bytes\n                    .iter()\n                    .map(|&b| if b == JSON_END_OF_PATH { b'0' } else { b }),\n            );\n        } else {\n            self.0.extend_from_slice(bytes);\n        }\n        self.0.push(JSON_END_OF_PATH);\n    }\n\n    /// Appends value bytes to the Term.\n    ///\n    /// This function returns the segment that has just been added.\n    #[inline]\n    pub fn append_bytes(&mut self, bytes: &[u8]) -> &mut [u8] {\n        let len_before = self.0.len();\n        self.0.extend_from_slice(bytes);\n        &mut self.0[len_before..]\n    }\n\n    fn clear(&mut self) {\n        self.0.clear();\n    }\n    fn truncate(&mut self, len: usize) {\n        self.0.truncate(len);\n    }\n    fn len(&self) -> usize {\n        self.0.len()\n    }\n\n    fn as_bytes(&self) -> &[u8] {\n        &self.0\n    }\n}\n"
  },
  {
    "path": "src/postings/loaded_postings.rs",
    "content": "use crate::docset::{DocSet, TERMINATED};\nuse crate::postings::{Postings, SegmentPostings};\nuse crate::DocId;\n\n/// `LoadedPostings` is a `DocSet` and `Postings` implementation.\n/// It is used to represent the postings of a term in memory.\n/// It is suitable if there are few documents for a term.\n///\n/// It exists mainly to reduce memory usage.\n/// `SegmentPostings` uses 1840 bytes per instance due to its caches.\n/// If you need to keep many terms around with few docs, it's cheaper to load all the\n/// postings in memory.\n///\n/// This is relevant for `RegexPhraseQuery`, which may have a lot of\n/// terms.\n/// E.g. 100_000 terms would need 184MB due to SegmentPostings.\npub struct LoadedPostings {\n    doc_ids: Box<[DocId]>,\n    position_offsets: Box<[u32]>,\n    positions: Box<[u32]>,\n    cursor: usize,\n}\n\nimpl LoadedPostings {\n    /// Creates a new `LoadedPostings` from a `SegmentPostings`.\n    ///\n    /// It will also preload positions, if positions are available in the SegmentPostings.\n    pub fn load(segment_postings: &mut SegmentPostings) -> LoadedPostings {\n        let num_docs = segment_postings.doc_freq() as usize;\n        let mut doc_ids = Vec::with_capacity(num_docs);\n        let mut positions = Vec::with_capacity(num_docs);\n        let mut position_offsets = Vec::with_capacity(num_docs);\n        while segment_postings.doc() != TERMINATED {\n            position_offsets.push(positions.len() as u32);\n            doc_ids.push(segment_postings.doc());\n            segment_postings.append_positions_with_offset(0, &mut positions);\n            segment_postings.advance();\n        }\n        position_offsets.push(positions.len() as u32);\n        LoadedPostings {\n            doc_ids: doc_ids.into_boxed_slice(),\n            positions: positions.into_boxed_slice(),\n            position_offsets: position_offsets.into_boxed_slice(),\n            cursor: 0,\n        }\n    }\n}\n\n#[cfg(test)]\nimpl From<(Vec<DocId>, Vec<Vec<u32>>)> for LoadedPostings {\n    fn from(doc_ids_and_positions: (Vec<DocId>, Vec<Vec<u32>>)) -> LoadedPostings {\n        let mut position_offsets = Vec::new();\n        let mut all_positions = Vec::new();\n        let (doc_ids, docid_positions) = doc_ids_and_positions;\n        for positions in docid_positions {\n            position_offsets.push(all_positions.len() as u32);\n            all_positions.extend_from_slice(&positions);\n        }\n        position_offsets.push(all_positions.len() as u32);\n        LoadedPostings {\n            doc_ids: doc_ids.into_boxed_slice(),\n            positions: all_positions.into_boxed_slice(),\n            position_offsets: position_offsets.into_boxed_slice(),\n            cursor: 0,\n        }\n    }\n}\n\nimpl DocSet for LoadedPostings {\n    fn advance(&mut self) -> DocId {\n        self.cursor += 1;\n        if self.cursor >= self.doc_ids.len() {\n            self.cursor = self.doc_ids.len();\n            return TERMINATED;\n        }\n        self.doc()\n    }\n\n    fn doc(&self) -> DocId {\n        if self.cursor >= self.doc_ids.len() {\n            return TERMINATED;\n        }\n        self.doc_ids[self.cursor]\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.doc_ids.len() as u32\n    }\n}\nimpl Postings for LoadedPostings {\n    fn term_freq(&self) -> u32 {\n        let start = self.position_offsets[self.cursor] as usize;\n        let end = self.position_offsets[self.cursor + 1] as usize;\n        (end - start) as u32\n    }\n\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        let start = self.position_offsets[self.cursor] as usize;\n        let end = self.position_offsets[self.cursor + 1] as usize;\n        for pos in &self.positions[start..end] {\n            output.push(*pos + offset);\n        }\n    }\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use super::*;\n\n    #[test]\n    pub fn test_vec_postings() {\n        let doc_ids: Vec<DocId> = (0u32..1024u32).map(|e| e * 3).collect();\n        let mut postings = LoadedPostings::from((doc_ids, vec![]));\n        assert_eq!(postings.doc(), 0u32);\n        assert_eq!(postings.advance(), 3u32);\n        assert_eq!(postings.doc(), 3u32);\n        assert_eq!(postings.seek(14u32), 15u32);\n        assert_eq!(postings.doc(), 15u32);\n        assert_eq!(postings.seek(300u32), 300u32);\n        assert_eq!(postings.doc(), 300u32);\n        assert_eq!(postings.seek(6000u32), TERMINATED);\n    }\n\n    #[test]\n    pub fn test_vec_postings2() {\n        let doc_ids: Vec<DocId> = (0u32..1024u32).map(|e| e * 3).collect();\n        let mut positions = Vec::new();\n        positions.resize(1024, Vec::new());\n        positions[0] = vec![1u32, 2u32, 3u32];\n        positions[1] = vec![30u32];\n        positions[2] = vec![10u32];\n        positions[4] = vec![50u32];\n        let mut postings = LoadedPostings::from((doc_ids, positions));\n\n        let load = |postings: &mut LoadedPostings| {\n            let mut loaded_positions = Vec::new();\n            postings.positions(loaded_positions.as_mut());\n            loaded_positions\n        };\n        assert_eq!(postings.doc(), 0u32);\n        assert_eq!(load(&mut postings), vec![1u32, 2u32, 3u32]);\n\n        assert_eq!(postings.advance(), 3u32);\n        assert_eq!(postings.doc(), 3u32);\n\n        assert_eq!(load(&mut postings), vec![30u32]);\n\n        assert_eq!(postings.seek(14u32), 15u32);\n        assert_eq!(postings.doc(), 15u32);\n        assert_eq!(postings.seek(300u32), 300u32);\n        assert_eq!(postings.doc(), 300u32);\n        assert_eq!(postings.seek(6000u32), TERMINATED);\n    }\n}\n"
  },
  {
    "path": "src/postings/mod.rs",
    "content": "//! Postings module (also called inverted index)\n\nmod block_search;\n\npub(crate) use self::block_search::branchless_binary_search;\n\nmod block_segment_postings;\npub(crate) mod compression;\nmod indexing_context;\nmod json_postings_writer;\nmod loaded_postings;\nmod per_field_postings_writer;\nmod postings;\nmod postings_writer;\nmod recorder;\nmod segment_postings;\n/// Serializer module for the inverted index\npub mod serializer;\nmod skip;\nmod term_info;\n\npub(crate) use loaded_postings::LoadedPostings;\npub(crate) use stacker::compute_table_memory_size;\n\npub use self::block_segment_postings::BlockSegmentPostings;\npub(crate) use self::indexing_context::IndexingContext;\npub(crate) use self::per_field_postings_writer::PerFieldPostingsWriter;\npub use self::postings::Postings;\npub(crate) use self::postings_writer::{serialize_postings, IndexingPosition, PostingsWriter};\npub use self::segment_postings::SegmentPostings;\npub use self::serializer::{FieldSerializer, InvertedIndexSerializer};\npub(crate) use self::skip::{BlockInfo, SkipReader};\npub use self::term_info::TermInfo;\n\n#[expect(clippy::enum_variant_names)]\n#[derive(Debug, PartialEq, Clone, Copy, Eq)]\npub(crate) enum FreqReadingOption {\n    NoFreq,\n    SkipFreq,\n    ReadFreq,\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use std::mem;\n\n    use super::{InvertedIndexSerializer, Postings};\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::fieldnorm::FieldNormReader;\n    use crate::index::{Index, SegmentComponent, SegmentReader};\n    use crate::indexer::operation::AddOperation;\n    use crate::indexer::SegmentWriter;\n    use crate::query::Scorer;\n    use crate::schema::{\n        Field, IndexRecordOption, Schema, Term, TextFieldIndexing, TextOptions, INDEXED, TEXT,\n    };\n    use crate::tokenizer::{SimpleTokenizer, MAX_TOKEN_LEN};\n    use crate::{DocId, HasLen, IndexWriter, Score};\n\n    #[test]\n    pub fn test_position_write() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut segment = index.new_segment();\n        let mut posting_serializer = InvertedIndexSerializer::open(&mut segment)?;\n        let mut field_serializer = posting_serializer.new_field(text_field, 120 * 4, None)?;\n        field_serializer.new_term(\"abc\".as_bytes(), 12u32, true)?;\n        for doc_id in 0u32..120u32 {\n            let delta_positions = vec![1, 2, 3, 2];\n            field_serializer.write_doc(doc_id, 4, &delta_positions);\n        }\n        field_serializer.close_term()?;\n        mem::drop(field_serializer);\n        posting_serializer.close()?;\n        let read = segment.open_read(SegmentComponent::Positions)?;\n        assert_eq!(read.len(), 207);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_skip_positions() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(\"title\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(title => r#\"abc abc abc\"#))?;\n        index_writer.add_document(doc!(title => r#\"abc be be be be abc\"#))?;\n        for _ in 0..1_000 {\n            index_writer.add_document(doc!(title => r#\"abc abc abc\"#))?;\n        }\n        index_writer.add_document(doc!(title => r#\"abc be be be be abc\"#))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n        let inverted_index = searcher.segment_reader(0u32).inverted_index(title)?;\n        let term = Term::from_field_text(title, \"abc\");\n        let mut positions = Vec::new();\n        {\n            let mut postings = inverted_index\n                .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.doc(), 0);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 1, 2], &positions[..]);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 1, 2], &positions[..]);\n            assert_eq!(postings.advance(), 1);\n            assert_eq!(postings.doc(), 1);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 5], &positions[..]);\n        }\n        {\n            let mut postings = inverted_index\n                .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.doc(), 0);\n            assert_eq!(postings.advance(), 1);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 5], &positions[..]);\n        }\n        {\n            let mut postings = inverted_index\n                .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.seek(1), 1);\n            assert_eq!(postings.doc(), 1);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 5], &positions[..]);\n        }\n        {\n            let mut postings = inverted_index\n                .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.seek(1002), 1002);\n            assert_eq!(postings.doc(), 1002);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 5], &positions[..]);\n        }\n        {\n            let mut postings = inverted_index\n                .read_postings(&term, IndexRecordOption::WithFreqsAndPositions)?\n                .unwrap();\n            assert_eq!(postings.seek(100), 100);\n            assert_eq!(postings.seek(1002), 1002);\n            assert_eq!(postings.doc(), 1002);\n            postings.positions(&mut positions);\n            assert_eq!(&[0, 5], &positions[..]);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_index_max_length_token() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_options = TextOptions::default().set_indexing_options(\n            TextFieldIndexing::default()\n                .set_index_option(IndexRecordOption::WithFreqsAndPositions)\n                .set_tokenizer(\"simple_no_truncation\"),\n        );\n        let text_field = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        index\n            .tokenizers()\n            .register(\"simple_no_truncation\", SimpleTokenizer::default());\n        let reader = index.reader()?;\n        let mut index_writer = index.writer_for_tests()?;\n\n        let ok_token_text: String = \"A\".repeat(MAX_TOKEN_LEN);\n        index_writer.add_document(doc!(text_field=>ok_token_text.clone()))?;\n        index_writer.commit()?;\n        reader.reload()?;\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n        let inverted_index = segment_reader.inverted_index(text_field)?;\n        assert_eq!(inverted_index.terms().num_terms(), 1);\n        let mut bytes = vec![];\n        assert!(inverted_index.terms().ord_to_term(0, &mut bytes)?);\n        assert_eq!(&bytes[..], ok_token_text.as_bytes());\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_drop_token_that_are_too_long() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_options = TextOptions::default().set_indexing_options(\n            TextFieldIndexing::default()\n                .set_index_option(IndexRecordOption::WithFreqsAndPositions)\n                .set_tokenizer(\"simple_no_truncation\"),\n        );\n        let text_field = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        index\n            .tokenizers()\n            .register(\"simple_no_truncation\", SimpleTokenizer::default());\n        let reader = index.reader()?;\n        let mut index_writer = index.writer_for_tests()?;\n\n        let mut exceeding_token_text: String = \"A\".repeat(MAX_TOKEN_LEN + 1);\n        exceeding_token_text.push_str(\" hello\");\n        index_writer.add_document(doc!(text_field=>exceeding_token_text))?;\n        index_writer.commit()?;\n        reader.reload()?;\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0u32);\n        let inverted_index = segment_reader.inverted_index(text_field)?;\n        assert_eq!(inverted_index.terms().num_terms(), 1);\n        let mut bytes = vec![];\n        assert!(inverted_index.terms().ord_to_term(0, &mut bytes)?);\n        assert_eq!(&bytes, b\"hello\");\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_position_and_fieldnorm1() -> crate::Result<()> {\n        let mut positions = Vec::new();\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let segment = index.new_segment();\n\n        {\n            let mut segment_writer =\n                SegmentWriter::for_segment(15_000_000, segment.clone()).unwrap();\n            {\n                // checking that position works if the field has two values\n                let op = AddOperation {\n                    opstamp: 0u64,\n                    document: doc!(\n                       text_field => \"a b a c a d a a.\",\n                       text_field => \"d d d d a\"\n                    ),\n                };\n                segment_writer.add_document(op)?;\n            }\n            {\n                let op = AddOperation {\n                    opstamp: 1u64,\n                    document: doc!(text_field => \"b a\"),\n                };\n                segment_writer.add_document(op).unwrap();\n            }\n            for i in 2..1000 {\n                let mut text: String = \"e \".repeat(i);\n                text.push_str(\" a\");\n                let op = AddOperation {\n                    opstamp: 2u64,\n                    document: doc!(text_field => text),\n                };\n                segment_writer.add_document(op).unwrap();\n            }\n            segment_writer.finalize()?;\n        }\n        {\n            let segment_reader = SegmentReader::open(&segment)?;\n            {\n                let fieldnorm_reader = segment_reader.get_fieldnorms_reader(text_field)?;\n                assert_eq!(fieldnorm_reader.fieldnorm(0), 8 + 5);\n                assert_eq!(fieldnorm_reader.fieldnorm(1), 2);\n                for i in 2..1000 {\n                    assert_eq!(\n                        fieldnorm_reader.fieldnorm_id(i),\n                        FieldNormReader::fieldnorm_to_id(i + 1)\n                    );\n                }\n            }\n            {\n                let term_a = Term::from_field_text(text_field, \"abcdef\");\n                assert!(segment_reader\n                    .inverted_index(term_a.field())?\n                    .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n                    .is_none());\n            }\n            {\n                let term_a = Term::from_field_text(text_field, \"a\");\n                let mut postings_a = segment_reader\n                    .inverted_index(term_a.field())?\n                    .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert_eq!(postings_a.len(), 1000);\n                assert_eq!(postings_a.doc(), 0);\n                assert_eq!(postings_a.term_freq(), 6);\n                postings_a.positions(&mut positions);\n                assert_eq!(&positions[..], [0, 2, 4, 6, 7, 13]);\n                assert_eq!(postings_a.advance(), 1u32);\n                assert_eq!(postings_a.doc(), 1u32);\n                assert_eq!(postings_a.term_freq(), 1);\n                for i in 2u32..1000u32 {\n                    assert_eq!(postings_a.advance(), i);\n                    assert_eq!(postings_a.term_freq(), 1);\n                    postings_a.positions(&mut positions);\n                    assert_eq!(&positions[..], [i]);\n                    assert_eq!(postings_a.doc(), i);\n                }\n                assert_eq!(postings_a.advance(), TERMINATED);\n            }\n            {\n                let term_e = Term::from_field_text(text_field, \"e\");\n                let mut postings_e = segment_reader\n                    .inverted_index(term_e.field())?\n                    .read_postings(&term_e, IndexRecordOption::WithFreqsAndPositions)?\n                    .unwrap();\n                assert_eq!(postings_e.len(), 1000 - 2);\n                for i in 2u32..1000u32 {\n                    assert_eq!(postings_e.term_freq(), i);\n                    postings_e.positions(&mut positions);\n                    assert_eq!(positions.len(), i as usize);\n                    for j in 0..positions.len() {\n                        assert_eq!(positions[j], (j as u32));\n                    }\n                    assert_eq!(postings_e.doc(), i);\n                    postings_e.advance();\n                }\n                assert_eq!(postings_e.doc(), TERMINATED);\n            }\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_position_and_fieldnorm2() -> crate::Result<()> {\n        let mut positions: Vec<u32> = Vec::new();\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field => \"g b b d c g c\"))?;\n            index_writer.add_document(doc!(text_field => \"g a b b a d c g c\"))?;\n            index_writer.commit()?;\n        }\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let searcher = index.reader()?.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let mut postings = segment_reader\n            .inverted_index(text_field)?\n            .read_postings(&term_a, IndexRecordOption::WithFreqsAndPositions)?\n            .unwrap();\n        assert_eq!(postings.doc(), 1u32);\n        postings.positions(&mut positions);\n        assert_eq!(&positions[..], &[1u32, 4]);\n        Ok(())\n    }\n\n    #[test]\n    fn test_skip_next() -> crate::Result<()> {\n        let term_0 = Term::from_field_u64(Field::from_field_id(0), 0);\n        let term_1 = Term::from_field_u64(Field::from_field_id(0), 1);\n        let term_2 = Term::from_field_u64(Field::from_field_id(0), 2);\n\n        let num_docs = 300u32;\n\n        let index = {\n            let mut schema_builder = Schema::builder();\n            let value_field = schema_builder.add_u64_field(\"value\", INDEXED);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            {\n                let mut index_writer = index.writer_for_tests()?;\n                for i in 0u64..num_docs as u64 {\n                    let doc = doc!(value_field => 2u64, value_field => i % 2u64);\n                    index_writer.add_document(doc)?;\n                }\n                assert!(index_writer.commit().is_ok());\n            }\n            index\n        };\n        let searcher = index.reader()?.searcher();\n        let segment_reader = searcher.segment_reader(0);\n\n        // check that the basic usage works\n        for i in 0..num_docs - 1 {\n            for j in i + 1..num_docs {\n                let mut segment_postings = segment_reader\n                    .inverted_index(term_2.field())?\n                    .read_postings(&term_2, IndexRecordOption::Basic)?\n                    .unwrap();\n                assert_eq!(segment_postings.seek(i), i);\n                assert_eq!(segment_postings.doc(), i);\n\n                assert_eq!(segment_postings.seek(j), j);\n                assert_eq!(segment_postings.doc(), j);\n            }\n        }\n\n        {\n            let mut segment_postings = segment_reader\n                .inverted_index(term_2.field())?\n                .read_postings(&term_2, IndexRecordOption::Basic)?\n                .unwrap();\n\n            // check that `skip_next` advances the iterator\n            assert_eq!(segment_postings.doc(), 0);\n\n            assert_eq!(segment_postings.seek(1), 1);\n            assert_eq!(segment_postings.doc(), 1);\n\n            assert_eq!(segment_postings.seek(1), 1);\n            assert_eq!(segment_postings.doc(), 1);\n\n            // check that going beyond the end is handled\n            assert_eq!(segment_postings.seek(num_docs), TERMINATED);\n        }\n\n        // check that filtering works\n        {\n            let mut segment_postings = segment_reader\n                .inverted_index(term_0.field())?\n                .read_postings(&term_0, IndexRecordOption::Basic)?\n                .unwrap();\n\n            for i in 0..num_docs / 2 {\n                assert_eq!(segment_postings.seek(i * 2), i * 2);\n                assert_eq!(segment_postings.doc(), i * 2);\n            }\n\n            let mut segment_postings = segment_reader\n                .inverted_index(term_0.field())?\n                .read_postings(&term_0, IndexRecordOption::Basic)?\n                .unwrap();\n\n            for i in 0..num_docs / 2 - 1 {\n                assert!(segment_postings.seek(i * 2 + 1) > i * 2);\n                assert_eq!(segment_postings.doc(), (i + 1) * 2);\n            }\n        }\n\n        // delete some of the documents\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.delete_term(term_0);\n            assert!(index_writer.commit().is_ok());\n        }\n        let searcher = index.reader()?.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let segment_reader = searcher.segment_reader(0);\n\n        // make sure seeking still works\n        for i in 0..num_docs {\n            let mut segment_postings = segment_reader\n                .inverted_index(term_2.field())?\n                .read_postings(&term_2, IndexRecordOption::Basic)?\n                .unwrap();\n\n            assert_eq!(segment_postings.seek(i), i);\n            assert_eq!(segment_postings.doc(), i);\n            if i % 2 == 0 {\n                assert!(segment_reader.is_deleted(i));\n            }\n        }\n\n        // now try with a longer sequence\n        {\n            let mut segment_postings = segment_reader\n                .inverted_index(term_2.field())?\n                .read_postings(&term_2, IndexRecordOption::Basic)?\n                .unwrap();\n\n            let mut last = 2; // start from 5 to avoid seeking to 3 twice\n            let mut cur = 3;\n            loop {\n                let seek = segment_postings.seek(cur);\n                if seek == TERMINATED {\n                    break;\n                }\n                assert_eq!(seek, segment_postings.doc());\n                if seek == cur {\n                    assert_eq!(segment_postings.doc(), cur);\n                } else {\n                    assert_eq!(segment_postings.doc(), cur + 1);\n                }\n                let next = cur + last;\n                last = cur;\n                cur = next;\n            }\n            assert_eq!(cur, 377);\n        }\n\n        // delete everything else\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.delete_term(term_1);\n            assert!(index_writer.commit().is_ok());\n        }\n        let searcher = index.reader()?.searcher();\n\n        // finally, check that it's empty\n        {\n            let searchable_segment_ids = index.searchable_segment_ids()?;\n            assert!(searchable_segment_ids.is_empty());\n            assert_eq!(searcher.num_docs(), 0);\n        }\n        Ok(())\n    }\n\n    /// Wraps a given docset, and forward all call but the\n    /// `.skip_next(...)`. This is useful to test that a specialized\n    /// implementation of `.skip_next(...)` is consistent\n    /// with the default implementation.\n    pub(crate) struct UnoptimizedDocSet<TDocSet: DocSet>(TDocSet);\n\n    impl<TDocSet: DocSet> UnoptimizedDocSet<TDocSet> {\n        pub fn wrap(docset: TDocSet) -> UnoptimizedDocSet<TDocSet> {\n            UnoptimizedDocSet(docset)\n        }\n    }\n\n    impl<TDocSet: DocSet> DocSet for UnoptimizedDocSet<TDocSet> {\n        fn advance(&mut self) -> DocId {\n            self.0.advance()\n        }\n\n        fn doc(&self) -> DocId {\n            self.0.doc()\n        }\n\n        fn size_hint(&self) -> u32 {\n            self.0.size_hint()\n        }\n    }\n\n    impl<TScorer: Scorer> Scorer for UnoptimizedDocSet<TScorer> {\n        #[inline]\n        fn score(&mut self) -> Score {\n            self.0.score()\n        }\n    }\n\n    pub fn test_skip_against_unoptimized<F: Fn() -> Box<dyn DocSet>>(\n        postings_factory: F,\n        targets: Vec<u32>,\n    ) {\n        for target in targets {\n            let mut postings_opt = postings_factory();\n            if target < postings_opt.doc() {\n                continue;\n            }\n            let mut postings_unopt = UnoptimizedDocSet::wrap(postings_factory());\n            let skip_result_opt = postings_opt.seek(target);\n            let skip_result_unopt = postings_unopt.seek(target);\n            assert_eq!(\n                skip_result_unopt, skip_result_opt,\n                \"Failed while skipping to {target}\"\n            );\n            assert!(skip_result_opt >= target);\n            assert_eq!(skip_result_opt, postings_opt.doc());\n            if skip_result_opt == TERMINATED {\n                return;\n            }\n            while postings_opt.doc() != TERMINATED {\n                assert_eq!(postings_opt.doc(), postings_unopt.doc());\n                assert_eq!(postings_opt.advance(), postings_unopt.advance());\n            }\n        }\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n    use once_cell::sync::Lazy;\n    use rand::rngs::StdRng;\n    use rand::{Rng, SeedableRng};\n    use test::{self, Bencher};\n\n    use crate::docset::TERMINATED;\n    use crate::query::Intersection;\n    use crate::schema::{Field, IndexRecordOption, Schema, TantivyDocument, Term, STRING};\n    use crate::{tests, DocSet, Index, IndexWriter};\n\n    pub static TERM_A: Lazy<Term> = Lazy::new(|| {\n        let field = Field::from_field_id(0);\n        Term::from_field_text(field, \"a\")\n    });\n    pub static TERM_B: Lazy<Term> = Lazy::new(|| {\n        let field = Field::from_field_id(0);\n        Term::from_field_text(field, \"b\")\n    });\n    pub static TERM_C: Lazy<Term> = Lazy::new(|| {\n        let field = Field::from_field_id(0);\n        Term::from_field_text(field, \"c\")\n    });\n    pub static TERM_D: Lazy<Term> = Lazy::new(|| {\n        let field = Field::from_field_id(0);\n        Term::from_field_text(field, \"d\")\n    });\n\n    pub static INDEX: Lazy<Index> = Lazy::new(|| {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", STRING);\n        let schema = schema_builder.build();\n\n        let mut rng: StdRng = StdRng::from_seed([1u8; 32]);\n\n        let index = Index::create_in_ram(schema);\n        let posting_list_size = 1_000_000;\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            for _ in 0..posting_list_size {\n                let mut doc = TantivyDocument::default();\n                if rng.random_bool(1f64 / 15f64) {\n                    doc.add_text(text_field, \"a\");\n                }\n                if rng.random_bool(1f64 / 10f64) {\n                    doc.add_text(text_field, \"b\");\n                }\n                if rng.random_bool(1f64 / 5f64) {\n                    doc.add_text(text_field, \"c\");\n                }\n                doc.add_text(text_field, \"d\");\n                index_writer.add_document(doc).unwrap();\n            }\n            assert!(index_writer.commit().is_ok());\n        }\n        index\n    });\n\n    #[bench]\n    fn bench_segment_postings(b: &mut Bencher) {\n        let reader = INDEX.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n\n        b.iter(|| {\n            let mut segment_postings = segment_reader\n                .inverted_index(TERM_A.field())\n                .unwrap()\n                .read_postings(&TERM_A, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            while segment_postings.advance() != TERMINATED {}\n        });\n    }\n\n    #[bench]\n    fn bench_segment_intersection(b: &mut Bencher) {\n        let reader = INDEX.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        b.iter(|| {\n            let segment_postings_a = segment_reader\n                .inverted_index(TERM_A.field())\n                .unwrap()\n                .read_postings(&TERM_A, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            let segment_postings_b = segment_reader\n                .inverted_index(TERM_B.field())\n                .unwrap()\n                .read_postings(&TERM_B, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            let segment_postings_c = segment_reader\n                .inverted_index(TERM_C.field())\n                .unwrap()\n                .read_postings(&TERM_C, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            let segment_postings_d = segment_reader\n                .inverted_index(TERM_D.field())\n                .unwrap()\n                .read_postings(&TERM_D, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            let mut intersection = Intersection::new(\n                vec![\n                    segment_postings_a,\n                    segment_postings_b,\n                    segment_postings_c,\n                    segment_postings_d,\n                ],\n                reader.searcher().num_docs() as u32,\n            );\n            while intersection.advance() != TERMINATED {}\n        });\n    }\n\n    fn bench_skip_next(p: f64, b: &mut Bencher) {\n        let reader = INDEX.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        let docs = tests::sample(segment_reader.num_docs(), p);\n\n        let mut segment_postings = segment_reader\n            .inverted_index(TERM_A.field())\n            .unwrap()\n            .read_postings(&TERM_A, IndexRecordOption::Basic)\n            .unwrap()\n            .unwrap();\n\n        let mut existing_docs = Vec::new();\n        for doc in &docs {\n            if *doc >= segment_postings.doc() {\n                existing_docs.push(*doc);\n                if segment_postings.seek(*doc) == TERMINATED {\n                    break;\n                }\n            }\n        }\n\n        b.iter(|| {\n            let mut segment_postings = segment_reader\n                .inverted_index(TERM_A.field())\n                .unwrap()\n                .read_postings(&TERM_A, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            for doc in &existing_docs {\n                if segment_postings.seek(*doc) == TERMINATED {\n                    break;\n                }\n            }\n        });\n    }\n\n    #[bench]\n    fn bench_skip_next_p01(b: &mut Bencher) {\n        bench_skip_next(0.001, b);\n    }\n\n    #[bench]\n    fn bench_skip_next_p1(b: &mut Bencher) {\n        bench_skip_next(0.01, b);\n    }\n\n    #[bench]\n    fn bench_skip_next_p10(b: &mut Bencher) {\n        bench_skip_next(0.1, b);\n    }\n\n    #[bench]\n    fn bench_skip_next_p90(b: &mut Bencher) {\n        bench_skip_next(0.9, b);\n    }\n\n    #[bench]\n    fn bench_iterate_segment_postings(b: &mut Bencher) {\n        let reader = INDEX.reader().unwrap();\n        let searcher = reader.searcher();\n        let segment_reader = searcher.segment_reader(0);\n        b.iter(|| {\n            let n: u32 = test::black_box(17);\n            let mut segment_postings = segment_reader\n                .inverted_index(TERM_A.field())\n                .unwrap()\n                .read_postings(&TERM_A, IndexRecordOption::Basic)\n                .unwrap()\n                .unwrap();\n            let mut s = 0u32;\n            while segment_postings.doc() != TERMINATED {\n                s += (segment_postings.doc() & n) % 1024;\n                segment_postings.advance();\n            }\n            s\n        });\n    }\n}\n"
  },
  {
    "path": "src/postings/per_field_postings_writer.rs",
    "content": "use crate::postings::json_postings_writer::JsonPostingsWriter;\nuse crate::postings::postings_writer::SpecializedPostingsWriter;\nuse crate::postings::recorder::{DocIdRecorder, TermFrequencyRecorder, TfAndPositionRecorder};\nuse crate::postings::PostingsWriter;\nuse crate::schema::{Field, FieldEntry, FieldType, IndexRecordOption, Schema};\n\npub(crate) struct PerFieldPostingsWriter {\n    per_field_postings_writers: Vec<Box<dyn PostingsWriter>>,\n}\n\nimpl PerFieldPostingsWriter {\n    pub fn for_schema(schema: &Schema) -> Self {\n        let per_field_postings_writers = schema\n            .fields()\n            .map(|(_, field_entry)| posting_writer_from_field_entry(field_entry))\n            .collect();\n        PerFieldPostingsWriter {\n            per_field_postings_writers,\n        }\n    }\n\n    pub(crate) fn get_for_field(&self, field: Field) -> &dyn PostingsWriter {\n        self.per_field_postings_writers[field.field_id() as usize].as_ref()\n    }\n\n    pub(crate) fn get_for_field_mut(&mut self, field: Field) -> &mut dyn PostingsWriter {\n        self.per_field_postings_writers[field.field_id() as usize].as_mut()\n    }\n}\n\nfn posting_writer_from_field_entry(field_entry: &FieldEntry) -> Box<dyn PostingsWriter> {\n    match *field_entry.field_type() {\n        FieldType::Str(ref text_options) => text_options\n            .get_indexing_options()\n            .map(|indexing_options| match indexing_options.index_option() {\n                IndexRecordOption::Basic => {\n                    SpecializedPostingsWriter::<DocIdRecorder>::default().into()\n                }\n                IndexRecordOption::WithFreqs => {\n                    SpecializedPostingsWriter::<TermFrequencyRecorder>::default().into()\n                }\n                IndexRecordOption::WithFreqsAndPositions => {\n                    SpecializedPostingsWriter::<TfAndPositionRecorder>::default().into()\n                }\n            })\n            .unwrap_or_else(|| SpecializedPostingsWriter::<DocIdRecorder>::default().into()),\n        FieldType::U64(_)\n        | FieldType::I64(_)\n        | FieldType::F64(_)\n        | FieldType::Bool(_)\n        | FieldType::Date(_)\n        | FieldType::Bytes(_)\n        | FieldType::IpAddr(_)\n        | FieldType::Facet(_) => Box::<SpecializedPostingsWriter<DocIdRecorder>>::default(),\n        FieldType::JsonObject(ref json_object_options) => {\n            if let Some(text_indexing_option) = json_object_options.get_text_indexing_options() {\n                match text_indexing_option.index_option() {\n                    IndexRecordOption::Basic => {\n                        JsonPostingsWriter::<DocIdRecorder>::default().into()\n                    }\n                    IndexRecordOption::WithFreqs => {\n                        JsonPostingsWriter::<TermFrequencyRecorder>::default().into()\n                    }\n                    IndexRecordOption::WithFreqsAndPositions => {\n                        JsonPostingsWriter::<TfAndPositionRecorder>::default().into()\n                    }\n                }\n            } else {\n                JsonPostingsWriter::<DocIdRecorder>::default().into()\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "src/postings/postings.rs",
    "content": "use crate::docset::DocSet;\n\n/// Postings (also called inverted list)\n///\n/// For a given term, it is the list of doc ids of the doc\n/// containing the term. Optionally, for each document,\n/// it may also give access to the term frequency\n/// as well as the list of term positions.\n///\n/// Its main implementation is `SegmentPostings`,\n/// but other implementations mocking `SegmentPostings` exist,\n/// for merging segments or for testing.\npub trait Postings: DocSet + 'static {\n    /// The number of times the term appears in the document.\n    fn term_freq(&self) -> u32;\n\n    /// Returns the positions offsetted with a given value.\n    /// It is not necessary to clear the `output` before calling this method.\n    /// The output vector will be resized to the `term_freq`.\n    fn positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        output.clear();\n        self.append_positions_with_offset(offset, output);\n    }\n\n    /// Returns the positions offsetted with a given value.\n    /// Data will be appended to the output.\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>);\n\n    /// Returns the positions of the term in the given document.\n    /// The output vector will be resized to the `term_freq`.\n    fn positions(&mut self, output: &mut Vec<u32>) {\n        self.positions_with_offset(0u32, output);\n    }\n}\n\nimpl Postings for Box<dyn Postings> {\n    fn term_freq(&self) -> u32 {\n        (**self).term_freq()\n    }\n\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        (**self).append_positions_with_offset(offset, output);\n    }\n}\n"
  },
  {
    "path": "src/postings/postings_writer.rs",
    "content": "use std::io;\nuse std::marker::PhantomData;\nuse std::ops::Range;\n\nuse stacker::Addr;\n\nuse crate::fieldnorm::FieldNormReaders;\nuse crate::indexer::indexing_term::IndexingTerm;\nuse crate::indexer::path_to_unordered_id::OrderedPathId;\nuse crate::postings::recorder::{BufferLender, Recorder};\nuse crate::postings::{\n    FieldSerializer, IndexingContext, InvertedIndexSerializer, PerFieldPostingsWriter,\n};\nuse crate::schema::{Field, Schema, Type};\nuse crate::tokenizer::{Token, TokenStream, MAX_TOKEN_LEN};\nuse crate::DocId;\n\nconst POSITION_GAP: u32 = 1;\n\nfn make_field_partition(\n    term_offsets: &[(Field, OrderedPathId, &[u8], Addr)],\n) -> Vec<(Field, Range<usize>)> {\n    let term_offsets_it = term_offsets\n        .iter()\n        .map(|(field, _, _, _)| *field)\n        .enumerate();\n    let mut prev_field_opt = None;\n    let mut fields = vec![];\n    let mut offsets = vec![];\n    for (offset, field) in term_offsets_it {\n        if Some(field) != prev_field_opt {\n            prev_field_opt = Some(field);\n            fields.push(field);\n            offsets.push(offset);\n        }\n    }\n    offsets.push(term_offsets.len());\n    let mut field_offsets = vec![];\n    for i in 0..fields.len() {\n        field_offsets.push((fields[i], offsets[i]..offsets[i + 1]));\n    }\n    field_offsets\n}\n\n/// Serialize the inverted index.\n/// It pushes all term, one field at a time, towards the\n/// postings serializer.\npub(crate) fn serialize_postings(\n    ctx: IndexingContext,\n    schema: Schema,\n    per_field_postings_writers: &PerFieldPostingsWriter,\n    fieldnorm_readers: FieldNormReaders,\n    serializer: &mut InvertedIndexSerializer,\n) -> crate::Result<()> {\n    // Replace unordered ids by ordered ids to be able to sort\n    let unordered_id_to_ordered_id: Vec<OrderedPathId> =\n        ctx.path_to_unordered_id.unordered_id_to_ordered_id();\n\n    let mut term_offsets: Vec<(Field, OrderedPathId, &[u8], Addr)> =\n        Vec::with_capacity(ctx.term_index.len());\n    term_offsets.extend(ctx.term_index.iter().map(|(key, addr)| {\n        let field = IndexingTerm::wrap(key).field();\n        if schema.get_field_entry(field).field_type().value_type() == Type::Json {\n            let byte_range_path = 4..4 + 4;\n            let unordered_id = u32::from_be_bytes(key[byte_range_path.clone()].try_into().unwrap());\n            let path_id = unordered_id_to_ordered_id[unordered_id as usize];\n            (field, path_id, &key[byte_range_path.end..], addr)\n        } else {\n            (field, 0.into(), &key[4..], addr)\n        }\n    }));\n    // Sort by field, path, and term\n    term_offsets.sort_unstable_by(\n        |(field1, path_id1, bytes1, _), (field2, path_id2, bytes2, _)| {\n            (field1, path_id1, bytes1).cmp(&(field2, path_id2, bytes2))\n        },\n    );\n    let ordered_id_to_path = ctx.path_to_unordered_id.ordered_id_to_path();\n    let field_offsets = make_field_partition(&term_offsets);\n    for (field, byte_offsets) in field_offsets {\n        let postings_writer = per_field_postings_writers.get_for_field(field);\n        let fieldnorm_reader = fieldnorm_readers.get_field(field)?;\n        let mut field_serializer =\n            serializer.new_field(field, postings_writer.total_num_tokens(), fieldnorm_reader)?;\n        postings_writer.serialize(\n            &term_offsets[byte_offsets],\n            &ordered_id_to_path,\n            &ctx,\n            &mut field_serializer,\n        )?;\n        field_serializer.close()?;\n    }\n\n    Ok(())\n}\n\n#[derive(Default, Debug)]\npub(crate) struct IndexingPosition {\n    pub num_tokens: u32,\n    pub end_position: u32,\n}\n\n/// The `PostingsWriter` is in charge of receiving documenting\n/// and building a `Segment` in anonymous memory.\n///\n/// `PostingsWriter` writes in a `MemoryArena`.\npub(crate) trait PostingsWriter: Send + Sync {\n    /// Record that a document contains a term at a given position.\n    ///\n    /// * doc  - the document id\n    /// * pos  - the term position (expressed in tokens)\n    /// * term - the term\n    /// * ctx - Contains a term hashmap and a memory arena to store all necessary posting list\n    ///   information.\n    fn subscribe(&mut self, doc: DocId, pos: u32, term: &IndexingTerm, ctx: &mut IndexingContext);\n\n    /// Serializes the postings on disk.\n    /// The actual serialization format is handled by the `PostingsSerializer`.\n    fn serialize(\n        &self,\n        term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],\n        ordered_id_to_path: &[&str],\n        ctx: &IndexingContext,\n        serializer: &mut FieldSerializer,\n    ) -> io::Result<()>;\n\n    /// Tokenize a text and subscribe all of its token.\n    fn index_text(\n        &mut self,\n        doc_id: DocId,\n        token_stream: &mut dyn TokenStream,\n        term_buffer: &mut IndexingTerm,\n        ctx: &mut IndexingContext,\n        indexing_position: &mut IndexingPosition,\n    ) {\n        let end_of_path_idx = term_buffer.len_bytes();\n        let mut num_tokens = 0;\n        let mut end_position = indexing_position.end_position;\n        token_stream.process(&mut |token: &Token| {\n            // We skip all tokens with a len greater than u16.\n            if token.text.len() > MAX_TOKEN_LEN {\n                warn!(\n                    \"A token exceeding MAX_TOKEN_LEN ({}>{}) was dropped. Search for \\\n                     MAX_TOKEN_LEN in the documentation for more information.\",\n                    token.text.len(),\n                    MAX_TOKEN_LEN\n                );\n                return;\n            }\n            term_buffer.truncate_value_bytes(end_of_path_idx);\n            term_buffer.append_bytes(token.text.as_bytes());\n            let start_position = indexing_position.end_position + token.position as u32;\n            end_position = end_position.max(start_position + token.position_length as u32);\n            self.subscribe(doc_id, start_position, term_buffer, ctx);\n            num_tokens += 1;\n        });\n\n        indexing_position.end_position = end_position + POSITION_GAP;\n        indexing_position.num_tokens += num_tokens;\n        term_buffer.truncate_value_bytes(end_of_path_idx);\n    }\n\n    fn total_num_tokens(&self) -> u64;\n}\n\n/// The `SpecializedPostingsWriter` is just here to remove dynamic\n/// dispatch to the recorder information.\n#[derive(Default)]\npub(crate) struct SpecializedPostingsWriter<Rec: Recorder> {\n    total_num_tokens: u64,\n    _recorder_type: PhantomData<Rec>,\n}\n\nimpl<Rec: Recorder> From<SpecializedPostingsWriter<Rec>> for Box<dyn PostingsWriter> {\n    fn from(\n        specialized_postings_writer: SpecializedPostingsWriter<Rec>,\n    ) -> Box<dyn PostingsWriter> {\n        Box::new(specialized_postings_writer)\n    }\n}\n\nimpl<Rec: Recorder> SpecializedPostingsWriter<Rec> {\n    #[inline]\n    pub(crate) fn serialize_one_term(\n        term: &[u8],\n        addr: Addr,\n        buffer_lender: &mut BufferLender,\n        ctx: &IndexingContext,\n        serializer: &mut FieldSerializer,\n    ) -> io::Result<()> {\n        let recorder: Rec = ctx.term_index.read(addr);\n        let term_doc_freq = recorder.term_doc_freq().unwrap_or(0u32);\n        serializer.new_term(term, term_doc_freq, recorder.has_term_freq())?;\n        recorder.serialize(&ctx.arena, serializer, buffer_lender);\n        serializer.close_term()?;\n        Ok(())\n    }\n}\n\nimpl<Rec: Recorder> PostingsWriter for SpecializedPostingsWriter<Rec> {\n    #[inline]\n    fn subscribe(\n        &mut self,\n        doc: DocId,\n        position: u32,\n        term: &IndexingTerm,\n        ctx: &mut IndexingContext,\n    ) {\n        debug_assert!(term.serialized_term().len() >= 4);\n        self.total_num_tokens += 1;\n        let (term_index, arena) = (&mut ctx.term_index, &mut ctx.arena);\n        term_index.mutate_or_create(term.serialized_term(), |opt_recorder: Option<Rec>| {\n            if let Some(mut recorder) = opt_recorder {\n                let current_doc = recorder.current_doc();\n                if current_doc != doc {\n                    recorder.close_doc(arena);\n                    recorder.new_doc(doc, arena);\n                }\n                recorder.record_position(position, arena);\n                recorder\n            } else {\n                let mut recorder = Rec::default();\n                recorder.new_doc(doc, arena);\n                recorder.record_position(position, arena);\n                recorder\n            }\n        });\n    }\n\n    fn serialize(\n        &self,\n        term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],\n        _ordered_id_to_path: &[&str],\n        ctx: &IndexingContext,\n        serializer: &mut FieldSerializer,\n    ) -> io::Result<()> {\n        let mut buffer_lender = BufferLender::default();\n        for (_field, _path_id, term, addr) in term_addrs {\n            Self::serialize_one_term(term, *addr, &mut buffer_lender, ctx, serializer)?;\n        }\n        Ok(())\n    }\n\n    fn total_num_tokens(&self) -> u64 {\n        self.total_num_tokens\n    }\n}\n"
  },
  {
    "path": "src/postings/recorder.rs",
    "content": "use common::read_u32_vint;\nuse stacker::{ExpUnrolledLinkedList, MemoryArena};\n\nuse crate::postings::FieldSerializer;\nuse crate::DocId;\n\nconst POSITION_END: u32 = 0;\n\n#[derive(Default)]\npub(crate) struct BufferLender {\n    buffer_u8: Vec<u8>,\n    buffer_u32: Vec<u32>,\n}\n\nimpl BufferLender {\n    pub fn lend_u8(&mut self) -> &mut Vec<u8> {\n        self.buffer_u8.clear();\n        &mut self.buffer_u8\n    }\n    pub fn lend_all(&mut self) -> (&mut Vec<u8>, &mut Vec<u32>) {\n        self.buffer_u8.clear();\n        self.buffer_u32.clear();\n        (&mut self.buffer_u8, &mut self.buffer_u32)\n    }\n}\n\npub struct VInt32Reader<'a> {\n    data: &'a [u8],\n}\n\nimpl<'a> VInt32Reader<'a> {\n    fn new(data: &'a [u8]) -> VInt32Reader<'a> {\n        VInt32Reader { data }\n    }\n}\n\nimpl Iterator for VInt32Reader<'_> {\n    type Item = u32;\n\n    fn next(&mut self) -> Option<u32> {\n        if self.data.is_empty() {\n            None\n        } else {\n            Some(read_u32_vint(&mut self.data))\n        }\n    }\n}\n\n/// `Recorder` is in charge of recording relevant information about\n/// the presence of a term in a document.\n///\n/// Depending on the [`TextOptions`](crate::schema::TextOptions) associated\n/// with the field, the recorder may record:\n///   * the document frequency\n///   * the document id\n///   * the term frequency\n///   * the term positions\npub(crate) trait Recorder: Copy + Default + Send + Sync + 'static {\n    /// Returns the current document\n    fn current_doc(&self) -> u32;\n    /// Starts recording information about a new document\n    /// This method shall only be called if the term is within the document.\n    fn new_doc(&mut self, doc: DocId, arena: &mut MemoryArena);\n    /// Record the position of a term. For each document,\n    /// this method will be called `term_freq` times.\n    fn record_position(&mut self, position: u32, arena: &mut MemoryArena);\n    /// Close the document. It will help record the term frequency.\n    fn close_doc(&mut self, arena: &mut MemoryArena);\n    /// Pushes the postings information to the serializer.\n    fn serialize(\n        &self,\n        arena: &MemoryArena,\n        serializer: &mut FieldSerializer<'_>,\n        buffer_lender: &mut BufferLender,\n    );\n    /// Returns the number of document containing this term.\n    ///\n    /// Returns `None` if not available.\n    fn term_doc_freq(&self) -> Option<u32>;\n\n    #[inline]\n    fn has_term_freq(&self) -> bool {\n        true\n    }\n}\n\n/// Only records the doc ids\n#[derive(Clone, Copy, Default)]\npub struct DocIdRecorder {\n    stack: ExpUnrolledLinkedList,\n    current_doc: DocId,\n}\n\nimpl Recorder for DocIdRecorder {\n    #[inline]\n    fn current_doc(&self) -> DocId {\n        self.current_doc\n    }\n\n    #[inline]\n    fn new_doc(&mut self, doc: DocId, arena: &mut MemoryArena) {\n        let delta = doc - self.current_doc;\n        self.current_doc = doc;\n        self.stack.writer(arena).write_u32_vint(delta);\n    }\n\n    #[inline]\n    fn record_position(&mut self, _position: u32, _arena: &mut MemoryArena) {}\n\n    #[inline]\n    fn close_doc(&mut self, _arena: &mut MemoryArena) {}\n\n    fn serialize(\n        &self,\n        arena: &MemoryArena,\n        serializer: &mut FieldSerializer<'_>,\n        buffer_lender: &mut BufferLender,\n    ) {\n        let buffer = buffer_lender.lend_u8();\n        // TODO avoid reading twice.\n        self.stack.read_to_end(arena, buffer);\n        let iter = get_sum_reader(VInt32Reader::new(&buffer[..]));\n        for doc_id in iter {\n            serializer.write_doc(doc_id, 0u32, &[][..]);\n        }\n    }\n\n    fn term_doc_freq(&self) -> Option<u32> {\n        None\n    }\n\n    fn has_term_freq(&self) -> bool {\n        false\n    }\n}\n\n/// Takes an Iterator of delta encoded elements and returns an iterator\n/// that yields the sum of the elements.\nfn get_sum_reader(iter: impl Iterator<Item = u32>) -> impl Iterator<Item = u32> {\n    iter.scan(0, |state, delta| {\n        *state += delta;\n        Some(*state)\n    })\n}\n\n/// Recorder encoding document ids, and term frequencies\n#[derive(Clone, Copy, Default)]\npub struct TermFrequencyRecorder {\n    stack: ExpUnrolledLinkedList,\n    current_doc: DocId,\n    current_tf: u32,\n    term_doc_freq: u32,\n}\n\nimpl Recorder for TermFrequencyRecorder {\n    #[inline]\n    fn current_doc(&self) -> DocId {\n        self.current_doc\n    }\n\n    #[inline]\n    fn new_doc(&mut self, doc: DocId, arena: &mut MemoryArena) {\n        let delta = doc - self.current_doc;\n        self.term_doc_freq += 1;\n        self.current_doc = doc;\n        self.stack.writer(arena).write_u32_vint(delta);\n    }\n\n    #[inline]\n    fn record_position(&mut self, _position: u32, _arena: &mut MemoryArena) {\n        self.current_tf += 1;\n    }\n\n    #[inline]\n    fn close_doc(&mut self, arena: &mut MemoryArena) {\n        debug_assert!(self.current_tf > 0);\n        self.stack.writer(arena).write_u32_vint(self.current_tf);\n        self.current_tf = 0;\n    }\n\n    fn serialize(\n        &self,\n        arena: &MemoryArena,\n        serializer: &mut FieldSerializer<'_>,\n        buffer_lender: &mut BufferLender,\n    ) {\n        let buffer = buffer_lender.lend_u8();\n        self.stack.read_to_end(arena, buffer);\n        let mut u32_it = VInt32Reader::new(&buffer[..]);\n        let mut prev_doc = 0;\n        while let Some(delta_doc_id) = u32_it.next() {\n            let doc_id = prev_doc + delta_doc_id;\n            prev_doc = doc_id;\n            let term_freq = u32_it.next().unwrap_or(self.current_tf);\n            serializer.write_doc(doc_id, term_freq, &[][..]);\n        }\n    }\n\n    fn term_doc_freq(&self) -> Option<u32> {\n        Some(self.term_doc_freq)\n    }\n}\n\n/// Recorder encoding term frequencies as well as positions.\n#[derive(Clone, Copy, Default)]\npub struct TfAndPositionRecorder {\n    stack: ExpUnrolledLinkedList,\n    current_doc: DocId,\n    term_doc_freq: u32,\n}\n\nimpl Recorder for TfAndPositionRecorder {\n    #[inline]\n    fn current_doc(&self) -> DocId {\n        self.current_doc\n    }\n\n    #[inline]\n    fn new_doc(&mut self, doc: DocId, arena: &mut MemoryArena) {\n        let delta = doc - self.current_doc;\n        self.current_doc = doc;\n        self.term_doc_freq += 1u32;\n        self.stack.writer(arena).write_u32_vint(delta);\n    }\n\n    #[inline]\n    fn record_position(&mut self, position: u32, arena: &mut MemoryArena) {\n        self.stack\n            .writer(arena)\n            .write_u32_vint(position.wrapping_add(1u32));\n    }\n\n    #[inline]\n    fn close_doc(&mut self, arena: &mut MemoryArena) {\n        self.stack.writer(arena).write_u32_vint(POSITION_END);\n    }\n\n    fn serialize(\n        &self,\n        arena: &MemoryArena,\n        serializer: &mut FieldSerializer<'_>,\n        buffer_lender: &mut BufferLender,\n    ) {\n        let (buffer_u8, buffer_positions) = buffer_lender.lend_all();\n        self.stack.read_to_end(arena, buffer_u8);\n        let mut u32_it = VInt32Reader::new(&buffer_u8[..]);\n        let mut prev_doc = 0;\n        while let Some(delta_doc_id) = u32_it.next() {\n            let doc_id = prev_doc + delta_doc_id;\n            prev_doc = doc_id;\n            let mut prev_position_plus_one = 1u32;\n            buffer_positions.clear();\n            loop {\n                match u32_it.next() {\n                    Some(POSITION_END) | None => {\n                        break;\n                    }\n                    Some(position_plus_one) => {\n                        let delta_position = position_plus_one - prev_position_plus_one;\n                        buffer_positions.push(delta_position);\n                        prev_position_plus_one = position_plus_one;\n                    }\n                }\n            }\n            serializer.write_doc(doc_id, buffer_positions.len() as u32, buffer_positions);\n        }\n    }\n\n    fn term_doc_freq(&self) -> Option<u32> {\n        Some(self.term_doc_freq)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use common::write_u32_vint;\n\n    use super::{BufferLender, VInt32Reader};\n\n    #[test]\n    fn test_buffer_lender() {\n        let mut buffer_lender = BufferLender::default();\n        {\n            let buf = buffer_lender.lend_u8();\n            assert!(buf.is_empty());\n            buf.push(1u8);\n        }\n        {\n            let buf = buffer_lender.lend_u8();\n            assert!(buf.is_empty());\n            buf.push(1u8);\n        }\n        {\n            let (_, buf) = buffer_lender.lend_all();\n            assert!(buf.is_empty());\n            buf.push(1u32);\n        }\n        {\n            let (_, buf) = buffer_lender.lend_all();\n            assert!(buf.is_empty());\n            buf.push(1u32);\n        }\n    }\n\n    #[test]\n    fn test_vint_u32() {\n        let mut buffer = vec![];\n        let vals = [0, 1, 324_234_234, u32::MAX];\n        for &i in &vals {\n            assert!(write_u32_vint(i, &mut buffer).is_ok());\n        }\n        assert_eq!(buffer.len(), 1 + 1 + 5 + 5);\n        let res: Vec<u32> = VInt32Reader::new(&buffer[..]).collect();\n        assert_eq!(&res[..], &vals[..]);\n    }\n}\n"
  },
  {
    "path": "src/postings/segment_postings.rs",
    "content": "use common::HasLen;\n\nuse crate::docset::DocSet;\nuse crate::fastfield::AliveBitSet;\nuse crate::positions::PositionReader;\nuse crate::postings::compression::COMPRESSION_BLOCK_SIZE;\nuse crate::postings::{BlockSegmentPostings, Postings};\nuse crate::{DocId, TERMINATED};\n\n/// `SegmentPostings` represents the inverted list or postings associated with\n/// a term in a `Segment`.\n///\n/// As we iterate through the `SegmentPostings`, the frequencies are optionally decoded.\n/// Positions on the other hand, are optionally entirely decoded upfront.\n#[derive(Clone)]\npub struct SegmentPostings {\n    pub(crate) block_cursor: BlockSegmentPostings,\n    cur: usize,\n    position_reader: Option<PositionReader>,\n}\n\nimpl SegmentPostings {\n    /// Returns an empty segment postings object\n    pub fn empty() -> Self {\n        SegmentPostings {\n            block_cursor: BlockSegmentPostings::empty(),\n            cur: 0,\n            position_reader: None,\n        }\n    }\n\n    /// Compute the number of non-deleted documents.\n    ///\n    /// This method will clone and scan through the posting lists.\n    /// (this is a rather expensive operation).\n    pub fn doc_freq_given_deletes(&self, alive_bitset: &AliveBitSet) -> u32 {\n        let mut docset = self.clone();\n        let mut doc_freq = 0;\n        loop {\n            let doc = docset.doc();\n            if doc == TERMINATED {\n                return doc_freq;\n            }\n            if alive_bitset.is_alive(doc) {\n                doc_freq += 1u32;\n            }\n            docset.advance();\n        }\n    }\n\n    /// Returns the overall number of documents in the block postings.\n    /// It does not take in account whether documents are deleted or not.\n    pub fn doc_freq(&self) -> u32 {\n        self.block_cursor.doc_freq()\n    }\n\n    /// Creates a segment postings object with the given documents\n    /// and no frequency encoded.\n    ///\n    /// This method is mostly useful for unit tests.\n    ///\n    /// It serializes the doc ids using tantivy's codec\n    /// and returns a `SegmentPostings` object that embeds a\n    /// buffer with the serialized data.\n    #[cfg(test)]\n    pub fn create_from_docs(docs: &[u32]) -> SegmentPostings {\n        use crate::directory::FileSlice;\n        use crate::postings::serializer::PostingsSerializer;\n        use crate::schema::IndexRecordOption;\n        let mut buffer = Vec::new();\n        {\n            let mut postings_serializer =\n                PostingsSerializer::new(0.0, IndexRecordOption::Basic, None);\n            postings_serializer.new_term(docs.len() as u32, false);\n            for &doc in docs {\n                postings_serializer.write_doc(doc, 1u32);\n            }\n            postings_serializer\n                .close_term(docs.len() as u32, &mut buffer)\n                .expect(\"In memory Serialization should never fail.\");\n        }\n        let block_segment_postings = BlockSegmentPostings::open(\n            docs.len() as u32,\n            FileSlice::from(buffer),\n            IndexRecordOption::Basic,\n            IndexRecordOption::Basic,\n        )\n        .unwrap();\n        SegmentPostings::from_block_postings(block_segment_postings, None)\n    }\n\n    /// Helper functions to create `SegmentPostings` for tests.\n    #[cfg(test)]\n    pub fn create_from_docs_and_tfs(\n        doc_and_tfs: &[(u32, u32)],\n        fieldnorms: Option<&[u32]>,\n    ) -> SegmentPostings {\n        use crate::directory::FileSlice;\n        use crate::fieldnorm::FieldNormReader;\n        use crate::postings::serializer::PostingsSerializer;\n        use crate::schema::IndexRecordOption;\n        use crate::Score;\n        let mut buffer: Vec<u8> = Vec::new();\n        let fieldnorm_reader = fieldnorms.map(FieldNormReader::for_test);\n        let average_field_norm = fieldnorms\n            .map(|fieldnorms| {\n                if fieldnorms.is_empty() {\n                    return 0.0;\n                }\n                let total_num_tokens: u64 = fieldnorms\n                    .iter()\n                    .map(|&fieldnorm| fieldnorm as u64)\n                    .sum::<u64>();\n                total_num_tokens as Score / fieldnorms.len() as Score\n            })\n            .unwrap_or(0.0);\n        let mut postings_serializer = PostingsSerializer::new(\n            average_field_norm,\n            IndexRecordOption::WithFreqs,\n            fieldnorm_reader,\n        );\n        postings_serializer.new_term(doc_and_tfs.len() as u32, true);\n        for &(doc, tf) in doc_and_tfs {\n            postings_serializer.write_doc(doc, tf);\n        }\n        postings_serializer\n            .close_term(doc_and_tfs.len() as u32, &mut buffer)\n            .unwrap();\n        let block_segment_postings = BlockSegmentPostings::open(\n            doc_and_tfs.len() as u32,\n            FileSlice::from(buffer),\n            IndexRecordOption::WithFreqs,\n            IndexRecordOption::WithFreqs,\n        )\n        .unwrap();\n        SegmentPostings::from_block_postings(block_segment_postings, None)\n    }\n\n    /// Reads a Segment postings from an &[u8]\n    ///\n    /// * `len` - number of document in the posting lists.\n    /// * `data` - data array. The complete data is not necessarily used.\n    /// * `freq_handler` - the freq handler is in charge of decoding frequencies and/or positions\n    pub(crate) fn from_block_postings(\n        segment_block_postings: BlockSegmentPostings,\n        position_reader: Option<PositionReader>,\n    ) -> SegmentPostings {\n        SegmentPostings {\n            block_cursor: segment_block_postings,\n            cur: 0, // cursor within the block\n            position_reader,\n        }\n    }\n}\n\nimpl DocSet for SegmentPostings {\n    // goes to the next element.\n    // next needs to be called a first time to point to the correct element.\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        debug_assert!(self.block_cursor.block_is_loaded());\n        if self.cur == COMPRESSION_BLOCK_SIZE - 1 {\n            self.cur = 0;\n            self.block_cursor.advance();\n        } else {\n            self.cur += 1;\n        }\n        self.doc()\n    }\n\n    #[inline]\n    fn seek(&mut self, target: DocId) -> DocId {\n        debug_assert!(self.doc() <= target);\n        if self.doc() >= target {\n            return self.doc();\n        }\n\n        // As an optimization, if the block is already loaded, we can\n        // cheaply check the next doc.\n        self.cur = (self.cur + 1).min(COMPRESSION_BLOCK_SIZE - 1);\n        if self.doc() >= target {\n            return self.doc();\n        }\n\n        // Delegate block-local search to BlockSegmentPostings::seek, which returns\n        // the in-block index of the first doc >= target.\n        self.cur = self.block_cursor.seek(target);\n        let doc = self.doc();\n        debug_assert!(doc >= target);\n        doc\n    }\n\n    /// Return the current document's `DocId`.\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.block_cursor.doc(self.cur)\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.len() as u32\n    }\n}\n\nimpl HasLen for SegmentPostings {\n    fn len(&self) -> usize {\n        self.block_cursor.doc_freq() as usize\n    }\n}\n\nimpl Postings for SegmentPostings {\n    /// Returns the frequency associated with the current document.\n    /// If the schema is set up so that no frequency have been encoded,\n    /// this method should always return 1.\n    ///\n    /// # Panics\n    ///\n    /// Will panics if called without having called advance before.\n    fn term_freq(&self) -> u32 {\n        debug_assert!(\n            // Here we do not use the len of `freqs()`\n            // because it is actually ok to request for the freq of doc\n            // even if no frequency were encoded for the field.\n            //\n            // In that case we hit the block just as if the frequency had been\n            // decoded. The block is simply prefilled by the value 1.\n            self.cur < COMPRESSION_BLOCK_SIZE,\n            \"Have you forgotten to call `.advance()` at least once before calling `.term_freq()`.\"\n        );\n        self.block_cursor.freq(self.cur)\n    }\n\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        let term_freq = self.term_freq();\n        let prev_len = output.len();\n        if let Some(position_reader) = self.position_reader.as_mut() {\n            debug_assert!(\n                !self.block_cursor.freqs().is_empty(),\n                \"No positions available\"\n            );\n            let read_offset = self.block_cursor.position_offset()\n                + (self.block_cursor.freqs()[..self.cur]\n                    .iter()\n                    .cloned()\n                    .sum::<u32>() as u64);\n            // TODO: instead of zeroing the output, we could use MaybeUninit or similar.\n            output.resize(prev_len + term_freq as usize, 0u32);\n            position_reader.read(read_offset, &mut output[prev_len..]);\n            let mut cum = offset;\n            for output_mut in output[prev_len..].iter_mut() {\n                cum += *output_mut;\n                *output_mut = cum;\n            }\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use common::HasLen;\n\n    use super::SegmentPostings;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::fastfield::AliveBitSet;\n    use crate::postings::postings::Postings;\n\n    #[test]\n    fn test_empty_segment_postings() {\n        let mut postings = SegmentPostings::empty();\n        assert_eq!(postings.advance(), TERMINATED);\n        assert_eq!(postings.advance(), TERMINATED);\n        assert_eq!(postings.len(), 0);\n    }\n\n    #[test]\n    fn test_empty_postings_doc_returns_terminated() {\n        let mut postings = SegmentPostings::empty();\n        assert_eq!(postings.doc(), TERMINATED);\n        assert_eq!(postings.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_empty_postings_doc_term_freq_returns_0() {\n        let postings = SegmentPostings::empty();\n        assert_eq!(postings.term_freq(), 1);\n    }\n\n    #[test]\n    fn test_doc_freq() {\n        let docs = SegmentPostings::create_from_docs(&[0, 2, 10]);\n        assert_eq!(docs.doc_freq(), 3);\n        let alive_bitset = AliveBitSet::for_test_from_deleted_docs(&[2], 12);\n        assert_eq!(docs.doc_freq_given_deletes(&alive_bitset), 2);\n        let all_deleted =\n            AliveBitSet::for_test_from_deleted_docs(&[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], 12);\n        assert_eq!(docs.doc_freq_given_deletes(&all_deleted), 0);\n    }\n}\n"
  },
  {
    "path": "src/postings/serializer.rs",
    "content": "use std::cmp::Ordering;\nuse std::io::{self, Write};\n\nuse common::{BinarySerializable, CountingWriter, VInt};\n\nuse super::TermInfo;\nuse crate::directory::{CompositeWrite, WritePtr};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::index::Segment;\nuse crate::positions::PositionSerializer;\nuse crate::postings::compression::{BlockEncoder, VIntEncoder, COMPRESSION_BLOCK_SIZE};\nuse crate::postings::skip::SkipSerializer;\nuse crate::query::Bm25Weight;\nuse crate::schema::{Field, FieldEntry, IndexRecordOption, Schema};\nuse crate::termdict::TermDictionaryBuilder;\nuse crate::{DocId, Score};\n\n/// `InvertedIndexSerializer` is in charge of serializing\n/// postings on disk, in the\n/// * `.idx` (inverted index)\n/// * `.pos` (positions file)\n/// * `.term` (term dictionary)\n///\n/// `PostingsWriter` are in charge of pushing the data to the\n/// serializer.\n///\n/// The serializer expects to receive the following calls\n/// in this order :\n/// * `set_field(...)`\n/// * `new_term(...)`\n/// * `write_doc(...)`\n/// * `write_doc(...)`\n/// * `write_doc(...)`\n/// * ...\n/// * `close_term()`\n/// * `new_term(...)`\n/// * `write_doc(...)`\n/// * ...\n/// * `close_term()`\n/// * `set_field(...)`\n/// * ...\n/// * `close()`\n///\n/// Terms have to be pushed in a lexicographically-sorted order.\n/// Within a term, documents have to be pushed in increasing order.\n///\n/// A description of the serialization format is\n/// [available here](https://fulmicoton.gitbooks.io/tantivy-doc/content/inverted-index.html).\npub struct InvertedIndexSerializer {\n    terms_write: CompositeWrite<WritePtr>,\n    postings_write: CompositeWrite<WritePtr>,\n    positions_write: CompositeWrite<WritePtr>,\n    schema: Schema,\n}\n\nimpl InvertedIndexSerializer {\n    /// Open a new `InvertedIndexSerializer` for the given segment\n    pub fn open(segment: &mut Segment) -> crate::Result<InvertedIndexSerializer> {\n        use crate::index::SegmentComponent::{Positions, Postings, Terms};\n        let inv_index_serializer = InvertedIndexSerializer {\n            terms_write: CompositeWrite::wrap(segment.open_write(Terms)?),\n            postings_write: CompositeWrite::wrap(segment.open_write(Postings)?),\n            positions_write: CompositeWrite::wrap(segment.open_write(Positions)?),\n            schema: segment.schema(),\n        };\n        Ok(inv_index_serializer)\n    }\n\n    /// Must be called before starting pushing terms of\n    /// a given field.\n    ///\n    /// Loads the indexing options for the given field.\n    pub fn new_field(\n        &mut self,\n        field: Field,\n        total_num_tokens: u64,\n        fieldnorm_reader: Option<FieldNormReader>,\n    ) -> io::Result<FieldSerializer<'_>> {\n        let field_entry: &FieldEntry = self.schema.get_field_entry(field);\n        let term_dictionary_write = self.terms_write.for_field(field);\n        let postings_write = self.postings_write.for_field(field);\n        let positions_write = self.positions_write.for_field(field);\n        let index_record_option = field_entry\n            .field_type()\n            .index_record_option()\n            .unwrap_or(IndexRecordOption::Basic);\n        FieldSerializer::create(\n            index_record_option,\n            total_num_tokens,\n            term_dictionary_write,\n            postings_write,\n            positions_write,\n            fieldnorm_reader,\n        )\n    }\n\n    /// Closes the serializer.\n    pub fn close(self) -> io::Result<()> {\n        self.terms_write.close()?;\n        self.postings_write.close()?;\n        self.positions_write.close()?;\n        Ok(())\n    }\n}\n\n/// The field serializer is in charge of\n/// the serialization of a specific field.\npub struct FieldSerializer<'a, W: Write = WritePtr> {\n    term_dictionary_builder: TermDictionaryBuilder<&'a mut CountingWriter<W>>,\n    postings_serializer: PostingsSerializer,\n    positions_serializer_opt: Option<PositionSerializer<&'a mut CountingWriter<W>>>,\n    current_term_info: TermInfo,\n    term_open: bool,\n    postings_write: &'a mut CountingWriter<W>,\n    postings_start_offset: u64,\n}\n\nimpl<'a, W: Write> FieldSerializer<'a, W> {\n    /// Creates a new `FieldSerializer` for the given field type.\n    pub fn create(\n        index_record_option: IndexRecordOption,\n        total_num_tokens: u64,\n        term_dictionary_write: &'a mut CountingWriter<W>,\n        postings_write: &'a mut CountingWriter<W>,\n        positions_write: &'a mut CountingWriter<W>,\n        fieldnorm_reader: Option<FieldNormReader>,\n    ) -> io::Result<FieldSerializer<'a, W>> {\n        total_num_tokens.serialize(postings_write)?;\n        let term_dictionary_builder = TermDictionaryBuilder::create(term_dictionary_write)?;\n        let average_fieldnorm = fieldnorm_reader\n            .as_ref()\n            .map(|ff_reader| total_num_tokens as Score / ff_reader.num_docs() as Score)\n            .unwrap_or(0.0);\n        let postings_serializer =\n            PostingsSerializer::new(average_fieldnorm, index_record_option, fieldnorm_reader);\n        let positions_serializer_opt = if index_record_option.has_positions() {\n            Some(PositionSerializer::new(positions_write))\n        } else {\n            None\n        };\n\n        let postings_start_offset = postings_write.written_bytes();\n        Ok(FieldSerializer {\n            term_dictionary_builder,\n            postings_serializer,\n            positions_serializer_opt,\n            current_term_info: TermInfo::default(),\n            term_open: false,\n            postings_write,\n            postings_start_offset,\n        })\n    }\n\n    fn postings_offset(&self) -> usize {\n        (self.postings_write.written_bytes() - self.postings_start_offset) as usize\n    }\n\n    fn current_term_info(&self) -> TermInfo {\n        let positions_start =\n            if let Some(positions_serializer) = self.positions_serializer_opt.as_ref() {\n                positions_serializer.written_bytes()\n            } else {\n                0u64\n            } as usize;\n        let addr = self.postings_offset();\n        TermInfo {\n            doc_freq: 0,\n            postings_range: addr..addr,\n            positions_range: positions_start..positions_start,\n        }\n    }\n\n    /// Starts the postings for a new term.\n    /// * term - the term. It needs to come after the previous term according to the lexicographical\n    ///   order.\n    /// * term_doc_freq - return the number of document containing the term.\n    pub fn new_term(\n        &mut self,\n        term: &[u8],\n        term_doc_freq: u32,\n        record_term_freq: bool,\n    ) -> io::Result<()> {\n        assert!(\n            !self.term_open,\n            \"Called new_term, while the previous term was not closed.\"\n        );\n        self.term_open = true;\n        self.postings_serializer.clear();\n        self.current_term_info = self.current_term_info();\n        self.term_dictionary_builder.insert_key(term)?;\n        self.postings_serializer\n            .new_term(term_doc_freq, record_term_freq);\n        Ok(())\n    }\n\n    /// Starts the postings for a new term without recording term frequencies.\n    pub fn new_term_without_freq(&mut self, term: &[u8]) -> io::Result<()> {\n        self.new_term(term, 0, false)\n    }\n\n    /// Serialize the information that a document contains for the current term:\n    /// its term frequency, and the position deltas.\n    ///\n    /// At this point, the positions are already `delta-encoded`.\n    /// For instance, if the positions are `2, 3, 17`,\n    /// `position_deltas` is `2, 1, 14`\n    ///\n    /// Term frequencies and positions may be ignored by the serializer depending\n    /// on the configuration of the field in the `Schema`.\n    pub fn write_doc(&mut self, doc_id: DocId, term_freq: u32, position_deltas: &[u32]) {\n        self.current_term_info.doc_freq += 1;\n        self.postings_serializer.write_doc(doc_id, term_freq);\n        if let Some(ref mut positions_serializer) = self.positions_serializer_opt.as_mut() {\n            assert_eq!(term_freq as usize, position_deltas.len());\n            positions_serializer.write_positions_delta(position_deltas);\n        }\n    }\n\n    /// Finish the serialization for this term postings.\n    ///\n    /// If the current block is incomplete, it needs to be encoded\n    /// using `VInt` encoding.\n    pub fn close_term(&mut self) -> io::Result<()> {\n        crate::fail_point!(\"FieldSerializer::close_term\", |msg: Option<String>| {\n            Err(io::Error::new(io::ErrorKind::Other, format!(\"{msg:?}\")))\n        });\n\n        if !self.term_open {\n            return Ok(());\n        };\n\n        self.postings_serializer\n            .close_term(self.current_term_info.doc_freq, self.postings_write)?;\n        self.current_term_info.postings_range.end = self.postings_offset();\n        if let Some(positions_serializer) = self.positions_serializer_opt.as_mut() {\n            positions_serializer.close_term()?;\n            self.current_term_info.positions_range.end =\n                positions_serializer.written_bytes() as usize;\n        }\n        self.term_dictionary_builder\n            .insert_value(&self.current_term_info)?;\n        self.term_open = false;\n        Ok(())\n    }\n\n    /// Closes the current field.\n    pub fn close(mut self) -> io::Result<()> {\n        self.close_term()?;\n        if let Some(positions_serializer) = self.positions_serializer_opt {\n            positions_serializer.close()?;\n        }\n        self.postings_write.flush()?;\n        self.term_dictionary_builder.finish()?;\n        Ok(())\n    }\n}\n\nstruct Block {\n    doc_ids: [DocId; COMPRESSION_BLOCK_SIZE],\n    term_freqs: [u32; COMPRESSION_BLOCK_SIZE],\n    len: usize,\n}\n\nimpl Block {\n    fn new() -> Self {\n        Block {\n            doc_ids: [0u32; COMPRESSION_BLOCK_SIZE],\n            term_freqs: [0u32; COMPRESSION_BLOCK_SIZE],\n            len: 0,\n        }\n    }\n\n    fn doc_ids(&self) -> &[DocId] {\n        &self.doc_ids[..self.len]\n    }\n\n    fn term_freqs(&self) -> &[u32] {\n        &self.term_freqs[..self.len]\n    }\n\n    fn clear(&mut self) {\n        self.len = 0;\n    }\n\n    fn append_doc(&mut self, doc: DocId, term_freq: u32) {\n        let len = self.len;\n        self.doc_ids[len] = doc;\n        self.term_freqs[len] = term_freq;\n        self.len = len + 1;\n    }\n\n    fn is_full(&self) -> bool {\n        self.len == COMPRESSION_BLOCK_SIZE\n    }\n\n    fn is_empty(&self) -> bool {\n        self.len == 0\n    }\n\n    fn last_doc(&self) -> DocId {\n        assert_eq!(self.len, COMPRESSION_BLOCK_SIZE);\n        self.doc_ids[COMPRESSION_BLOCK_SIZE - 1]\n    }\n}\n\n/// Serializer for postings lists.\npub struct PostingsSerializer {\n    last_doc_id_encoded: u32,\n\n    block_encoder: BlockEncoder,\n    block: Box<Block>,\n\n    postings_write: Vec<u8>,\n    skip_write: SkipSerializer,\n\n    mode: IndexRecordOption,\n    fieldnorm_reader: Option<FieldNormReader>,\n\n    bm25_weight: Option<Bm25Weight>,\n    avg_fieldnorm: Score, /* Average number of term in the field for that segment.\n                           * this value is used to compute the block wand information. */\n    term_has_freq: bool,\n}\n\nimpl PostingsSerializer {\n    /// Creates a new `PostingsSerializer`.\n    /// * avg_fieldnorm - average field norm for the field being serialized.\n    /// * mode - indexing options for the field being serialized.\n    pub fn new(\n        avg_fieldnorm: Score,\n        mode: IndexRecordOption,\n        fieldnorm_reader: Option<FieldNormReader>,\n    ) -> PostingsSerializer {\n        PostingsSerializer {\n            block_encoder: BlockEncoder::new(),\n            block: Box::new(Block::new()),\n\n            postings_write: Vec::new(),\n            skip_write: SkipSerializer::new(),\n\n            last_doc_id_encoded: 0u32,\n            mode,\n\n            fieldnorm_reader,\n            bm25_weight: None,\n            avg_fieldnorm,\n            term_has_freq: false,\n        }\n    }\n\n    /// Starts the serialization for a new term.\n    /// * term_doc_freq - the number of documents containing the term.\n    pub fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool) {\n        self.bm25_weight = None;\n\n        self.term_has_freq = self.mode.has_freq() && record_term_freq;\n        if !self.term_has_freq {\n            return;\n        }\n\n        let num_docs_in_segment: u64 =\n            if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {\n                fieldnorm_reader.num_docs() as u64\n            } else {\n                return;\n            };\n\n        if num_docs_in_segment == 0 {\n            return;\n        }\n\n        self.bm25_weight = Some(Bm25Weight::for_one_term_without_explain(\n            term_doc_freq as u64,\n            num_docs_in_segment,\n            self.avg_fieldnorm,\n        ));\n    }\n\n    fn write_block(&mut self) {\n        {\n            // encode the doc ids\n            let (num_bits, block_encoded): (u8, &[u8]) = self\n                .block_encoder\n                .compress_block_sorted(self.block.doc_ids(), self.last_doc_id_encoded);\n            self.last_doc_id_encoded = self.block.last_doc();\n            self.skip_write\n                .write_doc(self.last_doc_id_encoded, num_bits);\n            // last el block 0, offset block 1,\n            self.postings_write.extend(block_encoded);\n        }\n        if self.term_has_freq {\n            // encode the term frequencies\n            let (num_bits, block_encoded): (u8, &[u8]) = self\n                .block_encoder\n                .compress_block_unsorted(self.block.term_freqs(), true);\n            self.postings_write.extend(block_encoded);\n            self.skip_write.write_term_freq(num_bits);\n            if self.mode.has_positions() {\n                // We serialize the sum of term freqs within the skip information\n                // in order to navigate through positions.\n                let sum_freq = self.block.term_freqs().iter().cloned().sum();\n                self.skip_write.write_total_term_freq(sum_freq);\n            }\n            let mut blockwand_params = (0u8, 0u32);\n            if let Some(bm25_weight) = self.bm25_weight.as_ref() {\n                if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {\n                    let docs = self.block.doc_ids().iter().cloned();\n                    let term_freqs = self.block.term_freqs().iter().cloned();\n                    let fieldnorms = docs.map(|doc| fieldnorm_reader.fieldnorm_id(doc));\n                    blockwand_params = fieldnorms\n                        .zip(term_freqs)\n                        .max_by(\n                            |(left_fieldnorm_id, left_term_freq),\n                             (right_fieldnorm_id, right_term_freq)| {\n                                let left_score =\n                                    bm25_weight.tf_factor(*left_fieldnorm_id, *left_term_freq);\n                                let right_score =\n                                    bm25_weight.tf_factor(*right_fieldnorm_id, *right_term_freq);\n                                left_score\n                                    .partial_cmp(&right_score)\n                                    .unwrap_or(Ordering::Equal)\n                            },\n                        )\n                        .unwrap();\n                }\n            }\n            let (fieldnorm_id, term_freq) = blockwand_params;\n            self.skip_write.write_blockwand_max(fieldnorm_id, term_freq);\n        }\n        self.block.clear();\n    }\n\n    /// Register that the given document contains the current term.\n    /// * doc_id - the document id.\n    /// * term_freq - the term frequency within the document.\n    pub fn write_doc(&mut self, doc_id: DocId, term_freq: u32) {\n        self.block.append_doc(doc_id, term_freq);\n        if self.block.is_full() {\n            self.write_block();\n        }\n    }\n\n    /// Finish the serialization for this term.\n    pub fn close_term(\n        &mut self,\n        doc_freq: u32,\n        output_write: &mut impl std::io::Write,\n    ) -> io::Result<()> {\n        if !self.block.is_empty() {\n            // we have doc ids waiting to be written\n            // this happens when the number of doc ids is\n            // not a perfect multiple of our block size.\n            //\n            // In that case, the remaining part is encoded\n            // using variable int encoding.\n            {\n                let block_encoded = self\n                    .block_encoder\n                    .compress_vint_sorted(self.block.doc_ids(), self.last_doc_id_encoded);\n                self.postings_write.write_all(block_encoded)?;\n            }\n            // ... Idem for term frequencies\n            if self.term_has_freq {\n                let block_encoded = self\n                    .block_encoder\n                    .compress_vint_unsorted(self.block.term_freqs());\n                self.postings_write.write_all(block_encoded)?;\n            }\n            self.block.clear();\n        }\n        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {\n            let skip_data = self.skip_write.data();\n            VInt(skip_data.len() as u64).serialize(output_write)?;\n            output_write.write_all(skip_data)?;\n        }\n        output_write.write_all(&self.postings_write[..])?;\n        self.skip_write.clear();\n        self.postings_write.clear();\n        self.bm25_weight = None;\n        Ok(())\n    }\n\n    fn clear(&mut self) {\n        self.block.clear();\n        self.last_doc_id_encoded = 0;\n    }\n}\n"
  },
  {
    "path": "src/postings/skip.rs",
    "content": "use crate::directory::OwnedBytes;\nuse crate::postings::compression::{compressed_block_size, COMPRESSION_BLOCK_SIZE};\nuse crate::query::Bm25Weight;\nuse crate::schema::IndexRecordOption;\nuse crate::{DocId, Score, TERMINATED};\n\n// doc num bits uses the following encoding:\n// given 0b a b cdefgh\n//         |1|2|3|  4  |\n// - 1: unused\n// - 2: is delta-1 encoded. 0 if not, 1, if yes\n// - 3: unused\n// - 4: a 5 bit number in 0..32, the actual bitwidth. Bitpacking could in theory say this is 32\n//   (requiring a 6th bit), but the biggest doc_id we can want to encode is TERMINATED-1, which can\n//   be represented on 31b without delta encoding.\nfn encode_bitwidth(bitwidth: u8, delta_1: bool) -> u8 {\n    assert!(\n        bitwidth < 32,\n        \"bitwidth needs to be less than 32, but got {}\",\n        bitwidth\n    );\n    bitwidth | ((delta_1 as u8) << 6)\n}\n\nfn decode_bitwidth(raw_bitwidth: u8) -> (u8, bool) {\n    let delta_1 = ((raw_bitwidth >> 6) & 1) != 0;\n    let bitwidth = raw_bitwidth & 0x1f;\n    (bitwidth, delta_1)\n}\n\n#[inline]\nfn encode_block_wand_max_tf(max_tf: u32) -> u8 {\n    max_tf.min(u8::MAX as u32) as u8\n}\n\n#[inline]\nfn decode_block_wand_max_tf(max_tf_code: u8) -> u32 {\n    if max_tf_code == u8::MAX {\n        u32::MAX\n    } else {\n        max_tf_code as u32\n    }\n}\n\n#[inline]\nfn read_u32(data: &[u8]) -> u32 {\n    u32::from_le_bytes(data[..4].try_into().unwrap())\n}\n\n#[inline]\nfn write_u32(val: u32, buf: &mut Vec<u8>) {\n    buf.extend_from_slice(&val.to_le_bytes());\n}\n\npub struct SkipSerializer {\n    buffer: Vec<u8>,\n}\n\nimpl SkipSerializer {\n    pub fn new() -> SkipSerializer {\n        SkipSerializer { buffer: Vec::new() }\n    }\n\n    pub fn write_doc(&mut self, last_doc: DocId, doc_num_bits: u8) {\n        write_u32(last_doc, &mut self.buffer);\n        self.buffer.push(encode_bitwidth(doc_num_bits, true));\n    }\n\n    pub fn write_term_freq(&mut self, tf_num_bits: u8) {\n        self.buffer.push(tf_num_bits);\n    }\n\n    pub fn write_total_term_freq(&mut self, tf_sum: u32) {\n        write_u32(tf_sum, &mut self.buffer);\n    }\n\n    pub fn write_blockwand_max(&mut self, fieldnorm_id: u8, term_freq: u32) {\n        let block_wand_tf = encode_block_wand_max_tf(term_freq);\n        self.buffer\n            .extend_from_slice(&[fieldnorm_id, block_wand_tf]);\n    }\n\n    pub fn data(&self) -> &[u8] {\n        &self.buffer[..]\n    }\n\n    pub fn clear(&mut self) {\n        self.buffer.clear();\n    }\n}\n\n#[derive(Clone)]\npub(crate) struct SkipReader {\n    last_doc_in_block: DocId,\n    pub(crate) last_doc_in_previous_block: DocId,\n    owned_read: OwnedBytes,\n    skip_info: IndexRecordOption,\n    byte_offset: usize,\n    remaining_docs: u32, // number of docs remaining, including the\n    // documents in the current block.\n    block_info: BlockInfo,\n\n    position_offset: u64,\n}\n\n#[derive(Clone, Eq, PartialEq, Copy, Debug)]\npub(crate) enum BlockInfo {\n    BitPacked {\n        doc_num_bits: u8,\n        strict_delta_encoded: bool,\n        tf_num_bits: u8,\n        tf_sum: u32,\n        block_wand_fieldnorm_id: u8,\n        block_wand_term_freq: u32,\n    },\n    VInt {\n        num_docs: u32,\n    },\n}\n\nimpl Default for BlockInfo {\n    fn default() -> Self {\n        BlockInfo::VInt { num_docs: 0u32 }\n    }\n}\n\nimpl SkipReader {\n    pub fn new(data: OwnedBytes, doc_freq: u32, skip_info: IndexRecordOption) -> SkipReader {\n        let mut skip_reader = SkipReader {\n            last_doc_in_block: if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {\n                0\n            } else {\n                TERMINATED\n            },\n            last_doc_in_previous_block: 0u32,\n            owned_read: data,\n            skip_info,\n            block_info: BlockInfo::VInt { num_docs: doc_freq },\n            byte_offset: 0,\n            remaining_docs: doc_freq,\n            position_offset: 0u64,\n        };\n        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {\n            skip_reader.read_block_info();\n        }\n        skip_reader\n    }\n\n    pub fn reset(&mut self, data: OwnedBytes, doc_freq: u32) {\n        self.last_doc_in_block = if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {\n            0\n        } else {\n            TERMINATED\n        };\n        self.last_doc_in_previous_block = 0u32;\n        self.owned_read = data;\n        self.block_info = BlockInfo::VInt { num_docs: doc_freq };\n        self.byte_offset = 0;\n        self.remaining_docs = doc_freq;\n        self.position_offset = 0u64;\n        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {\n            self.read_block_info();\n        }\n    }\n\n    // Returns the block max score for this block if available.\n    //\n    // The block max score is available for all full bitpacked block,\n    // but no available for the last VInt encoded incomplete block.\n    pub fn block_max_score(&self, bm25_weight: &Bm25Weight) -> Option<Score> {\n        match self.block_info {\n            BlockInfo::BitPacked {\n                block_wand_fieldnorm_id,\n                block_wand_term_freq,\n                ..\n            } => Some(bm25_weight.score(block_wand_fieldnorm_id, block_wand_term_freq)),\n            BlockInfo::VInt { .. } => None,\n        }\n    }\n\n    pub(crate) fn last_doc_in_block(&self) -> DocId {\n        self.last_doc_in_block\n    }\n\n    pub fn position_offset(&self) -> u64 {\n        self.position_offset\n    }\n\n    #[inline]\n    pub fn byte_offset(&self) -> usize {\n        self.byte_offset\n    }\n\n    fn read_block_info(&mut self) {\n        let bytes = self.owned_read.as_slice();\n        let advance_len: usize;\n        self.last_doc_in_block = read_u32(bytes);\n        let (doc_num_bits, strict_delta_encoded) = decode_bitwidth(bytes[4]);\n        match self.skip_info {\n            IndexRecordOption::Basic => {\n                advance_len = 5;\n                self.block_info = BlockInfo::BitPacked {\n                    doc_num_bits,\n                    strict_delta_encoded,\n                    tf_num_bits: 0,\n                    tf_sum: 0,\n                    block_wand_fieldnorm_id: 0,\n                    block_wand_term_freq: 0,\n                };\n            }\n            IndexRecordOption::WithFreqs => {\n                let tf_num_bits = bytes[5];\n                let block_wand_fieldnorm_id = bytes[6];\n                let block_wand_term_freq = decode_block_wand_max_tf(bytes[7]);\n                advance_len = 8;\n                self.block_info = BlockInfo::BitPacked {\n                    doc_num_bits,\n                    strict_delta_encoded,\n                    tf_num_bits,\n                    tf_sum: 0,\n                    block_wand_fieldnorm_id,\n                    block_wand_term_freq,\n                };\n            }\n            IndexRecordOption::WithFreqsAndPositions => {\n                let tf_num_bits = bytes[5];\n                let tf_sum = read_u32(&bytes[6..10]);\n                let block_wand_fieldnorm_id = bytes[10];\n                let block_wand_term_freq = decode_block_wand_max_tf(bytes[11]);\n                advance_len = 12;\n                self.block_info = BlockInfo::BitPacked {\n                    doc_num_bits,\n                    strict_delta_encoded,\n                    tf_num_bits,\n                    tf_sum,\n                    block_wand_fieldnorm_id,\n                    block_wand_term_freq,\n                };\n            }\n        }\n        self.owned_read.advance(advance_len);\n    }\n\n    pub fn block_info(&self) -> BlockInfo {\n        self.block_info\n    }\n\n    /// Advance the skip reader to the block that may contain the target.\n    ///\n    /// If the target is larger than all documents, the skip_reader\n    /// then advance to the last Variable In block.\n    pub fn seek(&mut self, target: DocId) -> bool {\n        if self.last_doc_in_block() >= target {\n            return false;\n        }\n        loop {\n            self.advance();\n            if self.last_doc_in_block() >= target {\n                return true;\n            }\n        }\n    }\n\n    pub fn advance(&mut self) {\n        match self.block_info {\n            BlockInfo::BitPacked {\n                doc_num_bits,\n                tf_num_bits,\n                tf_sum,\n                ..\n            } => {\n                self.remaining_docs -= COMPRESSION_BLOCK_SIZE as u32;\n                self.byte_offset += compressed_block_size(doc_num_bits + tf_num_bits);\n                self.position_offset += tf_sum as u64;\n            }\n            BlockInfo::VInt { num_docs } => {\n                debug_assert_eq!(num_docs, self.remaining_docs);\n                self.remaining_docs = 0;\n                self.byte_offset = usize::MAX;\n            }\n        }\n        self.last_doc_in_previous_block = self.last_doc_in_block;\n        if self.remaining_docs >= COMPRESSION_BLOCK_SIZE as u32 {\n            self.read_block_info();\n        } else {\n            self.last_doc_in_block = TERMINATED;\n            self.block_info = BlockInfo::VInt {\n                num_docs: self.remaining_docs,\n            };\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::{\n        decode_bitwidth, encode_bitwidth, BlockInfo, IndexRecordOption, SkipReader, SkipSerializer,\n    };\n    use crate::directory::OwnedBytes;\n    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n\n    #[test]\n    fn test_encode_block_wand_max_tf() {\n        for tf in 0..255 {\n            assert_eq!(super::encode_block_wand_max_tf(tf), tf as u8);\n        }\n        for &tf in &[255, 256, 1_000_000, u32::MAX] {\n            assert_eq!(super::encode_block_wand_max_tf(tf), 255);\n        }\n    }\n\n    #[test]\n    fn test_decode_block_wand_max_tf() {\n        for tf in 0..255 {\n            assert_eq!(super::decode_block_wand_max_tf(tf), tf as u32);\n        }\n        assert_eq!(super::decode_block_wand_max_tf(255), u32::MAX);\n    }\n\n    #[test]\n    fn test_skip_with_freq() {\n        let buf = {\n            let mut skip_serializer = SkipSerializer::new();\n            skip_serializer.write_doc(1u32, 2u8);\n            skip_serializer.write_term_freq(3u8);\n            skip_serializer.write_blockwand_max(13u8, 3u32);\n            skip_serializer.write_doc(5u32, 5u8);\n            skip_serializer.write_term_freq(2u8);\n            skip_serializer.write_blockwand_max(8u8, 2u32);\n            skip_serializer.data().to_owned()\n        };\n        let doc_freq = 3u32 + (COMPRESSION_BLOCK_SIZE * 2) as u32;\n        let mut skip_reader =\n            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::WithFreqs);\n        assert_eq!(skip_reader.last_doc_in_block(), 1u32);\n        assert_eq!(\n            skip_reader.block_info,\n            BlockInfo::BitPacked {\n                doc_num_bits: 2u8,\n                strict_delta_encoded: true,\n                tf_num_bits: 3u8,\n                tf_sum: 0,\n                block_wand_fieldnorm_id: 13,\n                block_wand_term_freq: 3\n            }\n        );\n        skip_reader.advance();\n        assert_eq!(skip_reader.last_doc_in_block(), 5u32);\n        assert_eq!(\n            skip_reader.block_info(),\n            BlockInfo::BitPacked {\n                doc_num_bits: 5u8,\n                strict_delta_encoded: true,\n                tf_num_bits: 2u8,\n                tf_sum: 0,\n                block_wand_fieldnorm_id: 8,\n                block_wand_term_freq: 2\n            }\n        );\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 3u32 });\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });\n    }\n\n    #[test]\n    fn test_skip_no_freq() {\n        let buf = {\n            let mut skip_serializer = SkipSerializer::new();\n            skip_serializer.write_doc(1u32, 2u8);\n            skip_serializer.write_doc(5u32, 5u8);\n            skip_serializer.data().to_owned()\n        };\n        let doc_freq = 3u32 + (COMPRESSION_BLOCK_SIZE * 2) as u32;\n        let mut skip_reader =\n            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::Basic);\n        assert_eq!(skip_reader.last_doc_in_block(), 1u32);\n        assert_eq!(\n            skip_reader.block_info(),\n            BlockInfo::BitPacked {\n                doc_num_bits: 2u8,\n                strict_delta_encoded: true,\n                tf_num_bits: 0,\n                tf_sum: 0u32,\n                block_wand_fieldnorm_id: 0,\n                block_wand_term_freq: 0\n            }\n        );\n        skip_reader.advance();\n        assert_eq!(skip_reader.last_doc_in_block(), 5u32);\n        assert_eq!(\n            skip_reader.block_info(),\n            BlockInfo::BitPacked {\n                doc_num_bits: 5u8,\n                strict_delta_encoded: true,\n                tf_num_bits: 0,\n                tf_sum: 0u32,\n                block_wand_fieldnorm_id: 0,\n                block_wand_term_freq: 0\n            }\n        );\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 3u32 });\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });\n    }\n\n    #[test]\n    fn test_skip_multiple_of_block_size() {\n        let buf = {\n            let mut skip_serializer = SkipSerializer::new();\n            skip_serializer.write_doc(1u32, 2u8);\n            skip_serializer.data().to_owned()\n        };\n        let doc_freq = COMPRESSION_BLOCK_SIZE as u32;\n        let mut skip_reader =\n            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::Basic);\n        assert_eq!(skip_reader.last_doc_in_block(), 1u32);\n        assert_eq!(\n            skip_reader.block_info(),\n            BlockInfo::BitPacked {\n                doc_num_bits: 2u8,\n                strict_delta_encoded: true,\n                tf_num_bits: 0,\n                tf_sum: 0u32,\n                block_wand_fieldnorm_id: 0,\n                block_wand_term_freq: 0\n            }\n        );\n        skip_reader.advance();\n        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });\n    }\n\n    #[test]\n    fn test_encode_decode_bitwidth() {\n        for bitwidth in 0..32 {\n            for delta_1 in [false, true] {\n                assert_eq!(\n                    (bitwidth, delta_1),\n                    decode_bitwidth(encode_bitwidth(bitwidth, delta_1))\n                );\n            }\n        }\n        assert_eq!(0b01000010, encode_bitwidth(0b10, true));\n        assert_eq!(0b00000010, encode_bitwidth(0b10, false));\n    }\n}\n"
  },
  {
    "path": "src/postings/term_info.rs",
    "content": "use std::io;\nuse std::ops::Range;\n\nuse common::{BinarySerializable, FixedSize};\n\n/// `TermInfo` wraps the metadata associated with a Term.\n/// It is segment-local.\n#[derive(Debug, Default, Eq, PartialEq, Clone)]\npub struct TermInfo {\n    /// Number of documents in the segment containing the term\n    pub doc_freq: u32,\n    /// Byte range of the posting list within the postings (`.idx`) file.\n    pub postings_range: Range<usize>,\n    /// Byte range of the positions of this terms in the positions (`.pos`) file.\n    pub positions_range: Range<usize>,\n}\n\nimpl TermInfo {\n    pub(crate) fn posting_num_bytes(&self) -> u32 {\n        let num_bytes = self.postings_range.len();\n        assert!(num_bytes <= u32::MAX as usize);\n        num_bytes as u32\n    }\n\n    pub(crate) fn positions_num_bytes(&self) -> u32 {\n        let num_bytes = self.positions_range.len();\n        assert!(num_bytes <= u32::MAX as usize);\n        num_bytes as u32\n    }\n}\n\nimpl FixedSize for TermInfo {\n    /// Size required for the binary serialization of a `TermInfo` object.\n    /// This is large, but in practise, `TermInfo` are encoded in blocks and\n    /// only the first `TermInfo` of a block is serialized uncompressed.\n    /// The subsequent `TermInfo` are delta encoded and bitpacked.\n    const SIZE_IN_BYTES: usize = 3 * u32::SIZE_IN_BYTES + 2 * u64::SIZE_IN_BYTES;\n}\n\nimpl BinarySerializable for TermInfo {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.doc_freq.serialize(writer)?;\n        (self.postings_range.start as u64).serialize(writer)?;\n        self.posting_num_bytes().serialize(writer)?;\n        (self.positions_range.start as u64).serialize(writer)?;\n        self.positions_num_bytes().serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let doc_freq = u32::deserialize(reader)?;\n        let postings_start_offset = u64::deserialize(reader)? as usize;\n        let postings_num_bytes = u32::deserialize(reader)? as usize;\n        let postings_end_offset = postings_start_offset + postings_num_bytes;\n        let positions_start_offset = u64::deserialize(reader)? as usize;\n        let positions_num_bytes = u32::deserialize(reader)? as usize;\n        let positions_end_offset = positions_start_offset + positions_num_bytes;\n        Ok(TermInfo {\n            doc_freq,\n            postings_range: postings_start_offset..postings_end_offset,\n            positions_range: positions_start_offset..positions_end_offset,\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::TermInfo;\n    use crate::tests::fixed_size_test;\n\n    // TODO add serialize/deserialize test for terminfo\n\n    #[test]\n    fn test_fixed_size() {\n        fixed_size_test::<TermInfo>();\n    }\n}\n"
  },
  {
    "path": "src/query/all_query.rs",
    "content": "use crate::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN, TERMINATED};\nuse crate::index::SegmentReader;\nuse crate::query::boost_query::BoostScorer;\nuse crate::query::explanation::does_not_match;\nuse crate::query::{EnableScoring, Explanation, Query, Scorer, Weight};\nuse crate::{DocId, Score};\n\n/// Query that matches all of the documents.\n///\n/// All of the documents get the score 1.0.\n#[derive(Clone, Debug)]\npub struct AllQuery;\n\nimpl Query for AllQuery {\n    fn weight(&self, _: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(AllWeight))\n    }\n}\n\n/// Weight associated with the `AllQuery` query.\npub struct AllWeight;\n\nimpl Weight for AllWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let all_scorer = AllScorer::new(reader.max_doc());\n        if boost != 1.0 {\n            Ok(Box::new(BoostScorer::new(all_scorer, boost)))\n        } else {\n            Ok(Box::new(all_scorer))\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        if doc >= reader.max_doc() {\n            return Err(does_not_match(doc));\n        }\n        Ok(Explanation::new(\"AllQuery\", 1.0))\n    }\n}\n\n/// Scorer associated with the `AllQuery` query.\npub struct AllScorer {\n    doc: DocId,\n    max_doc: DocId,\n}\n\nimpl AllScorer {\n    /// Creates a new AllScorer with `max_doc` docs.\n    pub fn new(max_doc: DocId) -> AllScorer {\n        AllScorer { doc: 0u32, max_doc }\n    }\n}\n\nimpl DocSet for AllScorer {\n    #[inline(always)]\n    fn advance(&mut self) -> DocId {\n        if self.doc + 1 >= self.max_doc {\n            self.doc = TERMINATED;\n            return TERMINATED;\n        }\n        self.doc += 1;\n        self.doc\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        debug_assert!(target >= self.doc);\n        self.doc = target;\n        if self.doc >= self.max_doc {\n            self.doc = TERMINATED;\n        }\n        self.doc\n    }\n\n    fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {\n        if self.doc() == TERMINATED {\n            return 0;\n        }\n        let is_safe_distance = self.doc() + (buffer.len() as u32) < self.max_doc;\n        if is_safe_distance {\n            let num_items = buffer.len();\n            for buffer_val in buffer {\n                *buffer_val = self.doc();\n                self.doc += 1;\n            }\n            num_items\n        } else {\n            for (i, buffer_val) in buffer.iter_mut().enumerate() {\n                *buffer_val = self.doc();\n                if self.advance() == TERMINATED {\n                    return i + 1;\n                }\n            }\n            buffer.len()\n        }\n    }\n\n    #[inline(always)]\n    fn doc(&self) -> DocId {\n        self.doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.max_doc\n    }\n}\n\nimpl Scorer for AllScorer {\n    #[inline]\n    fn score(&mut self) -> Score {\n        1.0\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::AllQuery;\n    use crate::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN, TERMINATED};\n    use crate::query::{AllScorer, EnableScoring, Query};\n    use crate::schema::{Schema, TEXT};\n    use crate::{Index, IndexWriter};\n\n    fn create_test_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(field=>\"aaa\"))?;\n        index_writer.add_document(doc!(field=>\"bbb\"))?;\n        index_writer.commit()?;\n        index_writer.add_document(doc!(field=>\"ccc\"))?;\n        index_writer.commit()?;\n        Ok(index)\n    }\n\n    #[test]\n    fn test_all_query() -> crate::Result<()> {\n        let index = create_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let weight = AllQuery.weight(EnableScoring::disabled_from_schema(&index.schema()))?;\n        {\n            let reader = searcher.segment_reader(0);\n            let mut scorer = weight.scorer(reader, 1.0)?;\n            assert_eq!(scorer.doc(), 0u32);\n            assert_eq!(scorer.advance(), 1u32);\n            assert_eq!(scorer.doc(), 1u32);\n            assert_eq!(scorer.advance(), TERMINATED);\n        }\n        {\n            let reader = searcher.segment_reader(1);\n            let mut scorer = weight.scorer(reader, 1.0)?;\n            assert_eq!(scorer.doc(), 0u32);\n            assert_eq!(scorer.advance(), TERMINATED);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_all_query_with_boost() -> crate::Result<()> {\n        let index = create_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let weight = AllQuery.weight(EnableScoring::disabled_from_schema(searcher.schema()))?;\n        let reader = searcher.segment_reader(0);\n        {\n            let mut scorer = weight.scorer(reader, 2.0)?;\n            assert_eq!(scorer.doc(), 0u32);\n            assert_eq!(scorer.score(), 2.0);\n        }\n        {\n            let mut scorer = weight.scorer(reader, 1.5)?;\n            assert_eq!(scorer.doc(), 0u32);\n            assert_eq!(scorer.score(), 1.5);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_fill_buffer() {\n        let mut postings = AllScorer {\n            doc: 0u32,\n            max_doc: COLLECT_BLOCK_BUFFER_LEN as u32 * 2 + 9,\n        };\n        let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n        assert_eq!(postings.fill_buffer(&mut buffer), COLLECT_BLOCK_BUFFER_LEN);\n        for i in 0u32..COLLECT_BLOCK_BUFFER_LEN as u32 {\n            assert_eq!(buffer[i as usize], i);\n        }\n        assert_eq!(postings.fill_buffer(&mut buffer), COLLECT_BLOCK_BUFFER_LEN);\n        for i in 0u32..COLLECT_BLOCK_BUFFER_LEN as u32 {\n            assert_eq!(buffer[i as usize], i + COLLECT_BLOCK_BUFFER_LEN as u32);\n        }\n        assert_eq!(postings.fill_buffer(&mut buffer), 9);\n    }\n}\n"
  },
  {
    "path": "src/query/automaton_weight.rs",
    "content": "use std::io;\nuse std::sync::Arc;\n\nuse common::BitSet;\nuse tantivy_fst::Automaton;\n\nuse super::phrase_prefix_query::prefix_end;\nuse crate::index::SegmentReader;\nuse crate::postings::TermInfo;\nuse crate::query::{BitSetDocSet, ConstScorer, Explanation, Scorer, Weight};\nuse crate::schema::{Field, IndexRecordOption};\nuse crate::termdict::{TermDictionary, TermStreamer};\nuse crate::{DocId, Score, TantivyError};\n\n/// A weight struct for Fuzzy Term and Regex Queries\npub struct AutomatonWeight<A> {\n    field: Field,\n    automaton: Arc<A>,\n    // For JSON fields, the term dictionary include terms from all paths.\n    // We apply additional filtering based on the given JSON path, when searching within the term\n    // dictionary. This prevents terms from unrelated paths from matching the search criteria.\n    json_path_bytes: Option<Box<[u8]>>,\n}\n\nimpl<A> AutomatonWeight<A>\nwhere\n    A: Automaton + Send + Sync + 'static,\n    A::State: Clone,\n{\n    /// Create a new AutomationWeight\n    pub fn new<IntoArcA: Into<Arc<A>>>(field: Field, automaton: IntoArcA) -> AutomatonWeight<A> {\n        AutomatonWeight {\n            field,\n            automaton: automaton.into(),\n            json_path_bytes: None,\n        }\n    }\n\n    /// Create a new AutomationWeight for a json path\n    pub fn new_for_json_path<IntoArcA: Into<Arc<A>>>(\n        field: Field,\n        automaton: IntoArcA,\n        json_path_bytes: &[u8],\n    ) -> AutomatonWeight<A> {\n        AutomatonWeight {\n            field,\n            automaton: automaton.into(),\n            json_path_bytes: Some(json_path_bytes.to_vec().into_boxed_slice()),\n        }\n    }\n\n    fn automaton_stream<'a>(\n        &'a self,\n        term_dict: &'a TermDictionary,\n    ) -> io::Result<TermStreamer<'a, &'a A>> {\n        let automaton: &A = &self.automaton;\n        let mut term_stream_builder = term_dict.search(automaton);\n\n        if let Some(json_path_bytes) = &self.json_path_bytes {\n            term_stream_builder = term_stream_builder.ge(json_path_bytes);\n            if let Some(end) = prefix_end(json_path_bytes) {\n                term_stream_builder = term_stream_builder.lt(&end);\n            }\n        }\n\n        term_stream_builder.into_stream()\n    }\n\n    /// Returns the term infos that match the automaton\n    pub fn get_match_term_infos(&self, reader: &SegmentReader) -> crate::Result<Vec<TermInfo>> {\n        let inverted_index = reader.inverted_index(self.field)?;\n        let term_dict = inverted_index.terms();\n        let mut term_stream = self.automaton_stream(term_dict)?;\n        let mut term_infos = Vec::new();\n        while term_stream.advance() {\n            term_infos.push(term_stream.value().clone());\n        }\n        Ok(term_infos)\n    }\n}\n\nimpl<A> Weight for AutomatonWeight<A>\nwhere\n    A: Automaton + Send + Sync + 'static,\n    A::State: Clone,\n{\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let max_doc = reader.max_doc();\n        let mut doc_bitset = BitSet::with_max_value(max_doc);\n        let inverted_index = reader.inverted_index(self.field)?;\n        let term_dict = inverted_index.terms();\n        let mut term_stream = self.automaton_stream(term_dict)?;\n        while term_stream.advance() {\n            let term_info = term_stream.value();\n            let mut block_segment_postings = inverted_index\n                .read_block_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;\n            loop {\n                let docs = block_segment_postings.docs();\n                if docs.is_empty() {\n                    break;\n                }\n                for &doc in docs {\n                    doc_bitset.insert(doc);\n                }\n                block_segment_postings.advance();\n            }\n        }\n        let doc_bitset = BitSetDocSet::from(doc_bitset);\n        let const_scorer = ConstScorer::new(doc_bitset, boost);\n        Ok(Box::new(const_scorer))\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) == doc {\n            Ok(Explanation::new(\"AutomatonScorer\", 1.0))\n        } else {\n            Err(TantivyError::InvalidArgument(\n                \"Document does not exist\".to_string(),\n            ))\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use tantivy_fst::Automaton;\n\n    use super::AutomatonWeight;\n    use crate::docset::TERMINATED;\n    use crate::query::Weight;\n    use crate::schema::{Schema, STRING};\n    use crate::{Index, IndexWriter};\n\n    fn create_index() -> crate::Result<Index> {\n        let mut schema = Schema::builder();\n        let title = schema.add_text_field(\"title\", STRING);\n        let index = Index::create_in_ram(schema.build());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(title=>\"abc\"))?;\n        index_writer.add_document(doc!(title=>\"bcd\"))?;\n        index_writer.add_document(doc!(title=>\"abcd\"))?;\n        index_writer.commit()?;\n        Ok(index)\n    }\n\n    #[derive(Clone, Copy)]\n    enum State {\n        Start,\n        NotMatching,\n        AfterA,\n    }\n\n    struct PrefixedByA;\n\n    impl Automaton for PrefixedByA {\n        type State = State;\n\n        fn start(&self) -> Self::State {\n            State::Start\n        }\n\n        fn is_match(&self, state: &Self::State) -> bool {\n            matches!(*state, State::AfterA)\n        }\n\n        fn accept(&self, state: &Self::State, byte: u8) -> Self::State {\n            match *state {\n                State::Start => {\n                    if byte == b'a' {\n                        State::AfterA\n                    } else {\n                        State::NotMatching\n                    }\n                }\n                State::AfterA => State::AfterA,\n                State::NotMatching => State::NotMatching,\n            }\n        }\n    }\n\n    #[test]\n    fn test_automaton_weight() -> crate::Result<()> {\n        let index = create_index()?;\n        let field = index.schema().get_field(\"title\").unwrap();\n        let automaton_weight = AutomatonWeight::new(field, PrefixedByA);\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let mut scorer = automaton_weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n        assert_eq!(scorer.doc(), 0u32);\n        assert_eq!(scorer.score(), 1.0);\n        assert_eq!(scorer.advance(), 2u32);\n        assert_eq!(scorer.doc(), 2u32);\n        assert_eq!(scorer.score(), 1.0);\n        assert_eq!(scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_automaton_weight_boost() -> crate::Result<()> {\n        let index = create_index()?;\n        let field = index.schema().get_field(\"title\").unwrap();\n        let automaton_weight = AutomatonWeight::new(field, PrefixedByA);\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let mut scorer = automaton_weight.scorer(searcher.segment_reader(0u32), 1.32)?;\n        assert_eq!(scorer.doc(), 0u32);\n        assert_eq!(scorer.score(), 1.32);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/bitset/mod.rs",
    "content": "use common::{BitSet, TinySet};\n\nuse crate::docset::{DocSet, TERMINATED};\nuse crate::DocId;\n\n/// A `BitSetDocSet` makes it possible to iterate through a bitset as if it was a `DocSet`.\n///\n/// # Implementation detail\n///\n/// Skipping is relatively fast here as we can directly point to the\n/// right tiny bitset bucket.\n///\n/// TODO: Consider implementing a `BitTreeSet` in order to advance faster\n/// when the bitset is sparse\npub struct BitSetDocSet {\n    docs: BitSet,\n    cursor_bucket: u32, //< index associated with the current tiny bitset\n    cursor_tinybitset: TinySet,\n    doc: u32,\n}\n\nimpl BitSetDocSet {\n    fn go_to_bucket(&mut self, bucket_addr: u32) {\n        self.cursor_bucket = bucket_addr;\n        self.cursor_tinybitset = self.docs.tinyset(bucket_addr);\n    }\n}\n\nimpl From<BitSet> for BitSetDocSet {\n    fn from(docs: BitSet) -> BitSetDocSet {\n        let first_tiny_bitset = if docs.max_value() == 0 {\n            TinySet::empty()\n        } else {\n            docs.tinyset(0)\n        };\n        let mut docset = BitSetDocSet {\n            docs,\n            cursor_bucket: 0,\n            cursor_tinybitset: first_tiny_bitset,\n            doc: 0u32,\n        };\n        docset.advance();\n        docset\n    }\n}\n\nimpl DocSet for BitSetDocSet {\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        if let Some(lower) = self.cursor_tinybitset.pop_lowest() {\n            self.doc = (self.cursor_bucket * 64u32) | lower;\n            return self.doc;\n        }\n        if let Some(cursor_bucket) = self.docs.first_non_empty_bucket(self.cursor_bucket + 1) {\n            self.go_to_bucket(cursor_bucket);\n            let lower = self.cursor_tinybitset.pop_lowest().unwrap();\n            self.doc = (cursor_bucket * 64u32) | lower;\n            self.doc\n        } else {\n            self.doc = TERMINATED;\n            TERMINATED\n        }\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        if target >= self.docs.max_value() {\n            self.doc = TERMINATED;\n            return TERMINATED;\n        }\n        let target_bucket = target / 64u32;\n        if target_bucket > self.cursor_bucket {\n            self.go_to_bucket(target_bucket);\n            let greater_filter: TinySet = TinySet::range_greater_or_equal(target);\n            self.cursor_tinybitset = self.cursor_tinybitset.intersect(greater_filter);\n            self.advance()\n        } else {\n            let mut doc = self.doc();\n            while doc < target {\n                doc = self.advance();\n            }\n            doc\n        }\n    }\n\n    /// Returns the current document\n    fn doc(&self) -> DocId {\n        self.doc\n    }\n\n    /// Returns the number of values set in the underlying bitset.\n    fn size_hint(&self) -> u32 {\n        self.docs.len() as u32\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeSet;\n\n    use common::BitSet;\n\n    use super::BitSetDocSet;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::tests::generate_nonunique_unsorted;\n    use crate::DocId;\n\n    fn create_docbitset(docs: &[DocId], max_doc: DocId) -> BitSetDocSet {\n        let mut docset = BitSet::with_max_value(max_doc);\n        for &doc in docs {\n            docset.insert(doc);\n        }\n        BitSetDocSet::from(docset)\n    }\n\n    #[test]\n    fn test_bitset_large() {\n        let arr = generate_nonunique_unsorted(100_000, 5_000);\n        let mut btreeset: BTreeSet<u32> = BTreeSet::new();\n        let mut bitset = BitSet::with_max_value(100_000);\n        for el in arr {\n            btreeset.insert(el);\n            bitset.insert(el);\n        }\n        for i in 0..100_000 {\n            assert_eq!(btreeset.contains(&i), bitset.contains(i));\n        }\n        assert_eq!(btreeset.len(), bitset.len());\n        let mut bitset_docset = BitSetDocSet::from(bitset);\n        let mut remaining = true;\n        for el in btreeset.into_iter() {\n            assert!(remaining);\n            assert_eq!(bitset_docset.doc(), el);\n            remaining = bitset_docset.advance() != TERMINATED;\n        }\n        assert!(!remaining);\n    }\n\n    #[test]\n    fn test_empty() {\n        let bitset = BitSet::with_max_value(1000);\n        let mut empty = BitSetDocSet::from(bitset);\n        assert_eq!(empty.advance(), TERMINATED)\n    }\n\n    #[test]\n    fn test_seek_terminated() {\n        let bitset = BitSet::with_max_value(1000);\n        let mut empty = BitSetDocSet::from(bitset);\n        assert_eq!(empty.seek(TERMINATED), TERMINATED)\n    }\n\n    fn test_go_through_sequential(docs: &[DocId]) {\n        let mut docset = create_docbitset(docs, 1_000u32);\n        for &doc in docs {\n            assert_eq!(doc, docset.doc());\n            docset.advance();\n        }\n        assert_eq!(docset.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_docbitset_sequential() {\n        test_go_through_sequential(&[1, 2, 3]);\n        test_go_through_sequential(&[1, 2, 3, 4, 5, 63, 64, 65]);\n        test_go_through_sequential(&[63, 64, 65]);\n        test_go_through_sequential(&[1, 2, 3, 4, 95, 96, 97, 98, 99]);\n    }\n\n    #[test]\n    fn test_docbitset_skip() {\n        {\n            let mut docset = create_docbitset(&[1, 5, 6, 7, 5112], 10_000);\n            assert_eq!(docset.seek(7), 7);\n            assert_eq!(docset.doc(), 7);\n            assert_eq!(docset.advance(), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[1, 5, 6, 7, 5112], 10_000);\n            assert_eq!(docset.seek(3), 5);\n            assert_eq!(docset.doc(), 5);\n            assert_eq!(docset.advance(), 6);\n        }\n        {\n            let mut docset = create_docbitset(&[5112], 10_000);\n            assert_eq!(docset.seek(5112), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[5112], 10_000);\n            assert_eq!(docset.seek(5113), TERMINATED);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[5112], 10_000);\n            assert_eq!(docset.seek(5111), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[1, 5, 6, 7, 5112, 5500, 6666], 10_000);\n            assert_eq!(docset.seek(5112), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), 5500);\n            assert_eq!(docset.doc(), 5500);\n            assert_eq!(docset.advance(), 6666);\n            assert_eq!(docset.doc(), 6666);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[1, 5, 6, 7, 5112, 5500, 6666], 10_000);\n            assert_eq!(docset.seek(5111), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), 5500);\n            assert_eq!(docset.doc(), 5500);\n            assert_eq!(docset.advance(), 6666);\n            assert_eq!(docset.doc(), 6666);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n        {\n            let mut docset = create_docbitset(&[1, 5, 6, 7, 5112, 5513, 6666], 10_000);\n            assert_eq!(docset.seek(5111), 5112);\n            assert_eq!(docset.doc(), 5112);\n            assert_eq!(docset.advance(), 5513);\n            assert_eq!(docset.doc(), 5513);\n            assert_eq!(docset.advance(), 6666);\n            assert_eq!(docset.doc(), 6666);\n            assert_eq!(docset.advance(), TERMINATED);\n        }\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use super::{BitSet, BitSetDocSet};\n    use crate::docset::TERMINATED;\n    use crate::{test, tests, DocSet};\n\n    #[bench]\n    fn bench_bitset_1pct_insert(b: &mut test::Bencher) {\n        let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);\n        b.iter(|| {\n            let mut bitset = BitSet::with_max_value(1_000_000);\n            for el in els.iter().cloned() {\n                bitset.insert(el);\n            }\n        });\n    }\n\n    #[bench]\n    fn bench_bitset_1pct_clone(b: &mut test::Bencher) {\n        let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);\n        let mut bitset = BitSet::with_max_value(1_000_000);\n        for el in els {\n            bitset.insert(el);\n        }\n        b.iter(|| bitset.clone());\n    }\n\n    #[bench]\n    fn bench_bitset_1pct_clone_iterate(b: &mut test::Bencher) {\n        let els = tests::sample(1_000_000u32, 0.01);\n        let mut bitset = BitSet::with_max_value(1_000_000);\n        for el in els {\n            bitset.insert(el);\n        }\n        b.iter(|| {\n            let mut docset = BitSetDocSet::from(bitset.clone());\n            while docset.advance() != TERMINATED {}\n        });\n    }\n}\n"
  },
  {
    "path": "src/query/bm25.rs",
    "content": "use std::sync::Arc;\n\nuse crate::fieldnorm::FieldNormReader;\nuse crate::query::Explanation;\nuse crate::schema::Field;\nuse crate::{Score, Searcher, Term};\n\nconst K1: Score = 1.2;\nconst B: Score = 0.75;\n\n/// An interface to compute the statistics needed in BM25 scoring.\n///\n/// The standard implementation is a [Searcher] but you can also\n/// create your own to adjust the statistics.\npub trait Bm25StatisticsProvider {\n    /// The total number of tokens in a given field across all documents in\n    /// the index.\n    fn total_num_tokens(&self, field: Field) -> crate::Result<u64>;\n\n    /// The total number of documents in the index.\n    fn total_num_docs(&self) -> crate::Result<u64>;\n\n    /// The number of documents containing the given term.\n    fn doc_freq(&self, term: &Term) -> crate::Result<u64>;\n}\n\nimpl Bm25StatisticsProvider for Searcher {\n    fn total_num_tokens(&self, field: Field) -> crate::Result<u64> {\n        let mut total_num_tokens = 0u64;\n\n        for segment_reader in self.segment_readers() {\n            let inverted_index = segment_reader.inverted_index(field)?;\n            total_num_tokens += inverted_index.total_num_tokens();\n        }\n        Ok(total_num_tokens)\n    }\n\n    fn total_num_docs(&self) -> crate::Result<u64> {\n        let mut total_num_docs = 0u64;\n\n        for segment_reader in self.segment_readers() {\n            total_num_docs += u64::from(segment_reader.max_doc());\n        }\n        Ok(total_num_docs)\n    }\n\n    fn doc_freq(&self, term: &Term) -> crate::Result<u64> {\n        self.doc_freq(term)\n    }\n}\n\npub(crate) fn idf(doc_freq: u64, doc_count: u64) -> Score {\n    assert!(doc_count >= doc_freq, \"{doc_count} >= {doc_freq}\");\n    let x = ((doc_count - doc_freq) as Score + 0.5) / (doc_freq as Score + 0.5);\n    (1.0 + x).ln()\n}\n\nfn cached_tf_component(fieldnorm: u32, average_fieldnorm: Score) -> Score {\n    K1 * (1.0 - B + B * fieldnorm as Score / average_fieldnorm)\n}\n\nfn compute_tf_cache(average_fieldnorm: Score) -> Arc<[Score; 256]> {\n    let mut cache: [Score; 256] = [0.0; 256];\n    for (fieldnorm_id, cache_mut) in cache.iter_mut().enumerate() {\n        let fieldnorm = FieldNormReader::id_to_fieldnorm(fieldnorm_id as u8);\n        *cache_mut = cached_tf_component(fieldnorm, average_fieldnorm);\n    }\n    Arc::new(cache)\n}\n\n/// A struct used for computing BM25 scores.\n#[derive(Clone)]\npub struct Bm25Weight {\n    idf_explain: Option<Explanation>,\n    weight: Score,\n    cache: Arc<[Score; 256]>,\n    average_fieldnorm: Score,\n}\n\nimpl Bm25Weight {\n    /// Increase the weight by a multiplicative factor.\n    pub fn boost_by(&self, boost: Score) -> Bm25Weight {\n        if boost == 1.0f32 {\n            return self.clone();\n        }\n        Bm25Weight {\n            idf_explain: self.idf_explain.clone(),\n            weight: self.weight * boost,\n            cache: self.cache.clone(),\n            average_fieldnorm: self.average_fieldnorm,\n        }\n    }\n\n    /// Construct a [Bm25Weight] for a phrase of terms.\n    pub fn for_terms(\n        statistics: &dyn Bm25StatisticsProvider,\n        terms: &[Term],\n    ) -> crate::Result<Bm25Weight> {\n        assert!(!terms.is_empty(), \"Bm25 requires at least one term\");\n        let field = terms[0].field();\n        for term in &terms[1..] {\n            assert_eq!(\n                term.field(),\n                field,\n                \"All terms must belong to the same field.\"\n            );\n        }\n\n        let total_num_tokens = statistics.total_num_tokens(field)?;\n        let total_num_docs = statistics.total_num_docs()?;\n        let average_fieldnorm = total_num_tokens as Score / total_num_docs as Score;\n\n        if terms.len() == 1 {\n            let term_doc_freq = statistics.doc_freq(&terms[0])?;\n            Ok(Bm25Weight::for_one_term(\n                term_doc_freq,\n                total_num_docs,\n                average_fieldnorm,\n            ))\n        } else {\n            let mut idf_sum: Score = 0.0;\n            for term in terms {\n                let term_doc_freq = statistics.doc_freq(term)?;\n                idf_sum += idf(term_doc_freq, total_num_docs);\n            }\n            let idf_explain = Explanation::new(\"idf\", idf_sum);\n            Ok(Bm25Weight::new(idf_explain, average_fieldnorm))\n        }\n    }\n\n    /// Construct a [Bm25Weight] for a single term.\n    pub fn for_one_term(\n        term_doc_freq: u64,\n        total_num_docs: u64,\n        avg_fieldnorm: Score,\n    ) -> Bm25Weight {\n        let idf = idf(term_doc_freq, total_num_docs);\n        let mut idf_explain =\n            Explanation::new(\"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5))\", idf);\n        idf_explain.add_const(\n            \"n, number of docs containing this term\",\n            term_doc_freq as Score,\n        );\n        idf_explain.add_const(\"N, total number of docs\", total_num_docs as Score);\n        Bm25Weight::new(idf_explain, avg_fieldnorm)\n    }\n    /// Construct a [Bm25Weight] for a single term.\n    /// This method does not carry the [Explanation] for the idf.\n    pub fn for_one_term_without_explain(\n        term_doc_freq: u64,\n        total_num_docs: u64,\n        avg_fieldnorm: Score,\n    ) -> Bm25Weight {\n        let idf = idf(term_doc_freq, total_num_docs);\n        Bm25Weight::new_without_explain(idf, avg_fieldnorm)\n    }\n\n    pub(crate) fn new(idf_explain: Explanation, average_fieldnorm: Score) -> Bm25Weight {\n        let weight = idf_explain.value() * (1.0 + K1);\n        Bm25Weight {\n            idf_explain: Some(idf_explain),\n            weight,\n            cache: compute_tf_cache(average_fieldnorm),\n            average_fieldnorm,\n        }\n    }\n    pub(crate) fn new_without_explain(idf: f32, average_fieldnorm: Score) -> Bm25Weight {\n        let weight = idf * (1.0 + K1);\n        Bm25Weight {\n            idf_explain: None,\n            weight,\n            cache: compute_tf_cache(average_fieldnorm),\n            average_fieldnorm,\n        }\n    }\n\n    /// Compute the BM25 score of a single document.\n    #[inline]\n    pub fn score(&self, fieldnorm_id: u8, term_freq: u32) -> Score {\n        self.weight * self.tf_factor(fieldnorm_id, term_freq)\n    }\n\n    /// Compute the maximum possible BM25 score given this weight.\n    pub fn max_score(&self) -> Score {\n        self.score(255u8, 2_013_265_944)\n    }\n\n    #[inline]\n    pub(crate) fn tf_factor(&self, fieldnorm_id: u8, term_freq: u32) -> Score {\n        let term_freq = term_freq as Score;\n        let norm = self.cache[fieldnorm_id as usize];\n        term_freq / (term_freq + norm)\n    }\n\n    /// Produce an [Explanation] of a BM25 score.\n    pub fn explain(&self, fieldnorm_id: u8, term_freq: u32) -> Explanation {\n        // The explain format is directly copied from Lucene's.\n        // (So, Kudos to Lucene)\n        let score = self.score(fieldnorm_id, term_freq);\n\n        let norm = self.cache[fieldnorm_id as usize];\n        let term_freq = term_freq as Score;\n        let right_factor = term_freq / (term_freq + norm);\n\n        let mut tf_explanation = Explanation::new(\n            \"freq / (freq + k1 * (1 - b + b * dl / avgdl))\",\n            right_factor,\n        );\n\n        tf_explanation.add_const(\"freq, occurrences of term within document\", term_freq);\n        tf_explanation.add_const(\"k1, term saturation parameter\", K1);\n        tf_explanation.add_const(\"b, length normalization parameter\", B);\n        tf_explanation.add_const(\n            \"dl, length of field\",\n            FieldNormReader::id_to_fieldnorm(fieldnorm_id) as Score,\n        );\n        tf_explanation.add_const(\"avgdl, average length of field\", self.average_fieldnorm);\n\n        let mut explanation = Explanation::new(\"TermQuery, product of...\", score);\n        explanation.add_detail(Explanation::new(\"(K1+1)\", K1 + 1.0));\n        if let Some(idf_explain) = &self.idf_explain {\n            explanation.add_detail(idf_explain.clone());\n        }\n        explanation.add_detail(tf_explanation);\n        explanation\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::idf;\n    use crate::{assert_nearly_equals, Score};\n\n    #[test]\n    fn test_idf() {\n        let score: Score = 2.0;\n        assert_nearly_equals!(idf(1, 2), score.ln());\n    }\n}\n"
  },
  {
    "path": "src/query/boolean_query/block_wand.rs",
    "content": "use std::ops::{Deref, DerefMut};\n\nuse crate::query::term_query::TermScorer;\nuse crate::query::Scorer;\nuse crate::{DocId, DocSet, Score, TERMINATED};\n\n/// Takes a term_scorers sorted by their current doc() and a threshold and returns\n/// Returns (pivot_len, pivot_ord) defined as follows:\n/// - `pivot_doc` lowest document that has a chance of exceeding (>) the threshold score.\n/// - `before_pivot_len` number of term_scorers such that term_scorer.doc() < pivot.\n/// - `pivot_len` number of term_scorers such that term_scorer.doc() <= pivot.\n///\n/// We always have `before_pivot_len` < `pivot_len`.\n///\n/// `None` is returned if we establish that no document can exceed the threshold.\nfn find_pivot_doc(\n    term_scorers: &[TermScorerWithMaxScore],\n    threshold: Score,\n) -> Option<(usize, usize, DocId)> {\n    let mut max_score = 0.0;\n    let mut before_pivot_len = 0;\n    let mut pivot_doc = TERMINATED;\n    while before_pivot_len < term_scorers.len() {\n        let term_scorer = &term_scorers[before_pivot_len];\n        max_score += term_scorer.max_score;\n        if max_score > threshold {\n            pivot_doc = term_scorer.doc();\n            break;\n        }\n        before_pivot_len += 1;\n    }\n    if pivot_doc == TERMINATED {\n        return None;\n    }\n    // Right now i is an ordinal, we want a len.\n    let mut pivot_len = before_pivot_len + 1;\n    // Some other term_scorer may be positioned on the same document.\n    pivot_len += term_scorers[pivot_len..]\n        .iter()\n        .take_while(|term_scorer| term_scorer.doc() == pivot_doc)\n        .count();\n    Some((before_pivot_len, pivot_len, pivot_doc))\n}\n\n/// Advance the scorer with best score among the scorers[..pivot_len] to\n/// the next doc candidate defined by the min of `last_doc_in_block + 1` for\n/// scorer in scorers[..pivot_len] and `scorer.doc()` for scorer in scorers[pivot_len..].\n/// Note: before and after calling this method, scorers need to be sorted by their `.doc()`.\nfn block_max_was_too_low_advance_one_scorer(\n    scorers: &mut [TermScorerWithMaxScore],\n    pivot_len: usize,\n) {\n    debug_assert!(is_sorted(scorers.iter().map(|scorer| scorer.doc())));\n    let mut scorer_to_seek = pivot_len - 1;\n    let mut global_max_score = scorers[scorer_to_seek].max_score;\n    let mut doc_to_seek_after = scorers[scorer_to_seek].last_doc_in_block();\n    for scorer_ord in (0..pivot_len - 1).rev() {\n        let scorer = &scorers[scorer_ord];\n        if scorer.last_doc_in_block() <= doc_to_seek_after {\n            doc_to_seek_after = scorer.last_doc_in_block();\n        }\n        if scorers[scorer_ord].max_score > global_max_score {\n            global_max_score = scorers[scorer_ord].max_score;\n            scorer_to_seek = scorer_ord;\n        }\n    }\n    // Add +1 to go to the next block unless we are already at the end.\n    if doc_to_seek_after != TERMINATED {\n        doc_to_seek_after += 1;\n    }\n    for scorer in &scorers[pivot_len..] {\n        if scorer.doc() <= doc_to_seek_after {\n            doc_to_seek_after = scorer.doc();\n        }\n    }\n    scorers[scorer_to_seek].seek(doc_to_seek_after);\n\n    restore_ordering(scorers, scorer_to_seek);\n    debug_assert!(is_sorted(scorers.iter().map(|scorer| scorer.doc())));\n}\n\n// Given a list of term_scorers and a `ord` and assuming that `term_scorers[ord]` is sorted\n// except term_scorers[ord] that might be in advance compared to its ranks,\n// bubble up term_scorers[ord] in order to restore the ordering.\nfn restore_ordering(term_scorers: &mut [TermScorerWithMaxScore], ord: usize) {\n    let doc = term_scorers[ord].doc();\n    for i in ord + 1..term_scorers.len() {\n        if term_scorers[i].doc() >= doc {\n            break;\n        }\n        term_scorers.swap(i, i - 1);\n    }\n    debug_assert!(is_sorted(term_scorers.iter().map(|scorer| scorer.doc())));\n}\n\n// Attempts to advance all term_scorers between `&term_scorers[0..before_len]` to the pivot.\n// If this works, return true.\n// If this fails (ie: one of the term_scorer does not contain `pivot_doc` and seek goes past the\n// pivot), reorder the term_scorers to ensure the list is still sorted and returns `false`.\n// If a term_scorer reach TERMINATED in the process return false remove the term_scorer and return.\nfn align_scorers(\n    term_scorers: &mut Vec<TermScorerWithMaxScore>,\n    pivot_doc: DocId,\n    before_pivot_len: usize,\n) -> bool {\n    debug_assert_ne!(pivot_doc, TERMINATED);\n    for i in (0..before_pivot_len).rev() {\n        let new_doc = term_scorers[i].seek(pivot_doc);\n        if new_doc != pivot_doc {\n            if new_doc == TERMINATED {\n                term_scorers.swap_remove(i);\n            }\n            // We went past the pivot.\n            // We just go through the outer loop mechanic (Note that pivot is\n            // still a possible candidate).\n            //\n            // Termination is still guaranteed since we can only consider the same\n            // pivot at most term_scorers.len() - 1 times.\n            restore_ordering(term_scorers, i);\n            return false;\n        }\n    }\n    true\n}\n\n// Assumes terms_scorers[..pivot_len] are positioned on the same doc (pivot_doc).\n// Advance term_scorers[..pivot_len] and out of these removes the terminated scores.\n// Restores the ordering of term_scorers.\nfn advance_all_scorers_on_pivot(term_scorers: &mut Vec<TermScorerWithMaxScore>, pivot_len: usize) {\n    for term_scorer in &mut term_scorers[..pivot_len] {\n        term_scorer.advance();\n    }\n    // TODO use drain_filter when available.\n    let mut i = 0;\n    while i != term_scorers.len() {\n        if term_scorers[i].doc() == TERMINATED {\n            term_scorers.swap_remove(i);\n        } else {\n            i += 1;\n        }\n    }\n    term_scorers.sort_by_key(|scorer| scorer.doc());\n}\n\n/// Implements the WAND (Weak AND) algorithm for dynamic pruning\n/// described in the paper \"Faster Top-k Document Retrieval Using Block-Max Indexes\".\n/// Link: <http://engineering.nyu.edu/~suel/papers/bmw.pdf>\npub fn block_wand(\n    mut scorers: Vec<TermScorer>,\n    mut threshold: Score,\n    callback: &mut dyn FnMut(u32, Score) -> Score,\n) {\n    let mut scorers: Vec<TermScorerWithMaxScore> = scorers\n        .iter_mut()\n        .map(TermScorerWithMaxScore::from)\n        .collect();\n    scorers.sort_by_key(|scorer| scorer.doc());\n    // At this point we need to ensure that the scorers are sorted!\n    debug_assert!(is_sorted(scorers.iter().map(|scorer| scorer.doc())));\n    while let Some((before_pivot_len, pivot_len, pivot_doc)) =\n        find_pivot_doc(&scorers[..], threshold)\n    {\n        debug_assert!(is_sorted(scorers.iter().map(|scorer| scorer.doc())));\n        debug_assert_ne!(pivot_doc, TERMINATED);\n        debug_assert!(before_pivot_len < pivot_len);\n\n        let block_max_score_upperbound: Score = scorers[..pivot_len]\n            .iter_mut()\n            .map(|scorer| {\n                scorer.seek_block(pivot_doc);\n                scorer.block_max_score()\n            })\n            .sum();\n\n        // Beware after shallow advance, skip readers can be in advance compared to\n        // the segment posting lists.\n        //\n        // `block_segment_postings.load_block()` need to be called separately.\n        if block_max_score_upperbound <= threshold {\n            // Block max condition was not reached\n            // We could get away by simply advancing the scorers to DocId + 1 but it would\n            // be inefficient. The optimization requires proper explanation and was\n            // isolated in a different function.\n            block_max_was_too_low_advance_one_scorer(&mut scorers, pivot_len);\n            continue;\n        }\n\n        // Block max condition is observed.\n        //\n        // Let's try and advance all scorers before the pivot to the pivot.\n        if !align_scorers(&mut scorers, pivot_doc, before_pivot_len) {\n            // At least of the scorer does not contain the pivot.\n            //\n            // Let's stop scoring this pivot and go through the pivot selection again.\n            // Note that the current pivot is not necessarily a bad candidate and it\n            // may be picked again.\n            continue;\n        }\n\n        // At this point, all scorers are positioned on the doc.\n        let score = scorers[..pivot_len]\n            .iter_mut()\n            .map(|scorer| scorer.score())\n            .sum();\n\n        if score > threshold {\n            threshold = callback(pivot_doc, score);\n        }\n        // let's advance all of the scorers that are currently positioned on the pivot.\n        advance_all_scorers_on_pivot(&mut scorers, pivot_len);\n    }\n}\n\n/// Specialized version of [`block_wand`] for a single scorer.\n/// In this case, the algorithm is simple, readable and faster (~ x3)\n/// than the generic algorithm.\n/// The algorithm behaves as follows:\n/// - While we don't hit the end of the docset:\n///   - While the block max score is under the `threshold`, go to the next block.\n///   - On a block, advance until the end and execute `callback` when the doc score is greater or\n///     equal to the `threshold`.\npub fn block_wand_single_scorer(\n    mut scorer: TermScorer,\n    mut threshold: Score,\n    callback: &mut dyn FnMut(u32, Score) -> Score,\n) {\n    let mut doc = scorer.doc();\n    loop {\n        // We position the scorer on a block that can reach\n        // the threshold.\n        while scorer.block_max_score() < threshold {\n            let last_doc_in_block = scorer.last_doc_in_block();\n            if last_doc_in_block == TERMINATED {\n                return;\n            }\n            doc = last_doc_in_block + 1;\n            scorer.seek_block(doc);\n        }\n        // Seek will effectively load that block.\n        doc = scorer.seek(doc);\n        if doc == TERMINATED {\n            break;\n        }\n        loop {\n            let score = scorer.score();\n            if score > threshold {\n                threshold = callback(doc, score);\n            }\n            debug_assert!(doc <= scorer.last_doc_in_block());\n            if doc == scorer.last_doc_in_block() {\n                break;\n            }\n            doc = scorer.advance();\n            if doc == TERMINATED {\n                return;\n            }\n        }\n        doc += 1;\n        scorer.seek_block(doc);\n    }\n}\n\nstruct TermScorerWithMaxScore<'a> {\n    scorer: &'a mut TermScorer,\n    max_score: Score,\n}\n\nimpl<'a> From<&'a mut TermScorer> for TermScorerWithMaxScore<'a> {\n    fn from(scorer: &'a mut TermScorer) -> Self {\n        let max_score = scorer.max_score();\n        TermScorerWithMaxScore { scorer, max_score }\n    }\n}\n\nimpl Deref for TermScorerWithMaxScore<'_> {\n    type Target = TermScorer;\n\n    fn deref(&self) -> &Self::Target {\n        self.scorer\n    }\n}\n\nimpl DerefMut for TermScorerWithMaxScore<'_> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        self.scorer\n    }\n}\n\nfn is_sorted<I: Iterator<Item = DocId>>(mut it: I) -> bool {\n    if let Some(first) = it.next() {\n        let mut prev = first;\n        for doc in it {\n            if doc < prev {\n                return false;\n            }\n            prev = doc;\n        }\n    }\n    true\n}\n#[cfg(test)]\nmod tests {\n    use std::cmp::Ordering;\n    use std::collections::BinaryHeap;\n\n    use proptest::prelude::*;\n\n    use crate::query::score_combiner::SumCombiner;\n    use crate::query::term_query::TermScorer;\n    use crate::query::{Bm25Weight, BufferedUnionScorer, Scorer};\n    use crate::{DocId, DocSet, Score, TERMINATED};\n\n    struct Float(Score);\n\n    impl Eq for Float {}\n\n    impl PartialEq for Float {\n        fn eq(&self, other: &Self) -> bool {\n            self.cmp(other) == Ordering::Equal\n        }\n    }\n\n    impl PartialOrd for Float {\n        fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n            Some(self.cmp(other))\n        }\n    }\n\n    impl Ord for Float {\n        fn cmp(&self, other: &Self) -> Ordering {\n            other.0.partial_cmp(&self.0).unwrap_or(Ordering::Equal)\n        }\n    }\n\n    fn nearly_equals(left: Score, right: Score) -> bool {\n        (left - right).abs() < 0.0001 * (left + right).abs()\n    }\n\n    fn compute_checkpoints_for_each_pruning(\n        mut term_scorers: Vec<TermScorer>,\n        n: usize,\n    ) -> Vec<(DocId, Score)> {\n        let mut heap: BinaryHeap<Float> = BinaryHeap::with_capacity(n);\n        let mut checkpoints: Vec<(DocId, Score)> = Vec::new();\n        let mut limit: Score = 0.0;\n\n        let callback = &mut |doc, score| {\n            heap.push(Float(score));\n            if heap.len() > n {\n                heap.pop().unwrap();\n            }\n            if heap.len() == n {\n                limit = heap.peek().unwrap().0;\n            }\n            if !nearly_equals(score, limit) {\n                checkpoints.push((doc, score));\n            }\n            limit\n        };\n\n        if term_scorers.len() == 1 {\n            let scorer = term_scorers.pop().unwrap();\n            super::block_wand_single_scorer(scorer, Score::MIN, callback);\n        } else {\n            super::block_wand(term_scorers, Score::MIN, callback);\n        }\n        checkpoints\n    }\n\n    fn compute_checkpoints_manual(\n        term_scorers: Vec<TermScorer>,\n        n: usize,\n        max_doc: u32,\n    ) -> Vec<(DocId, Score)> {\n        let mut heap: BinaryHeap<Float> = BinaryHeap::with_capacity(n);\n        let mut checkpoints: Vec<(DocId, Score)> = Vec::new();\n        let mut scorer = BufferedUnionScorer::build(term_scorers, SumCombiner::default, max_doc);\n\n        let mut limit = Score::MIN;\n        loop {\n            if scorer.doc() == TERMINATED {\n                break;\n            }\n            let doc = scorer.doc();\n            let score = scorer.score();\n            if score > limit {\n                heap.push(Float(score));\n                if heap.len() > n {\n                    heap.pop().unwrap();\n                }\n                if heap.len() == n {\n                    limit = heap.peek().unwrap().0;\n                }\n                if !nearly_equals(score, limit) {\n                    checkpoints.push((doc, score));\n                }\n            }\n            scorer.advance();\n        }\n        checkpoints\n    }\n\n    const MAX_TERM_FREQ: u32 = 100u32;\n\n    fn posting_list(max_doc: u32) -> BoxedStrategy<Vec<(DocId, u32)>> {\n        (1..max_doc + 1)\n            .prop_flat_map(move |doc_freq| {\n                (\n                    proptest::bits::bitset::sampled(doc_freq as usize, 0..max_doc as usize),\n                    proptest::collection::vec(1u32..MAX_TERM_FREQ, doc_freq as usize),\n                )\n            })\n            .prop_map(|(docset, term_freqs)| {\n                docset\n                    .iter()\n                    .map(|doc| doc as u32)\n                    .zip(term_freqs.iter().cloned())\n                    .collect::<Vec<_>>()\n            })\n            .boxed()\n    }\n\n    #[expect(clippy::type_complexity)]\n    fn gen_term_scorers(num_scorers: usize) -> BoxedStrategy<(Vec<Vec<(DocId, u32)>>, Vec<u32>)> {\n        (1u32..100u32)\n            .prop_flat_map(move |max_doc: u32| {\n                (\n                    proptest::collection::vec(posting_list(max_doc), num_scorers),\n                    proptest::collection::vec(2u32..10u32 * MAX_TERM_FREQ, max_doc as usize),\n                )\n            })\n            .boxed()\n    }\n\n    fn test_block_wand_aux(posting_lists: &[Vec<(DocId, u32)>], fieldnorms: &[u32]) {\n        // We virtually repeat all docs 64 times in order to emulate blocks of 2 documents\n        // and surface blogs more easily.\n        const REPEAT: usize = 64;\n        let fieldnorms_expanded = fieldnorms\n            .iter()\n            .cloned()\n            .flat_map(|fieldnorm| std::iter::repeat_n(fieldnorm, REPEAT))\n            .collect::<Vec<u32>>();\n\n        let postings_lists_expanded: Vec<Vec<(DocId, u32)>> = posting_lists\n            .iter()\n            .map(|posting_list| {\n                posting_list\n                    .iter()\n                    .cloned()\n                    .flat_map(|(doc, term_freq)| {\n                        (0_u32..REPEAT as u32).map(move |offset| {\n                            (\n                                doc * (REPEAT as u32) + offset,\n                                if offset == 0 { term_freq } else { 1 },\n                            )\n                        })\n                    })\n                    .collect::<Vec<(DocId, u32)>>()\n            })\n            .collect::<Vec<_>>();\n\n        let total_fieldnorms: u64 = fieldnorms_expanded\n            .iter()\n            .cloned()\n            .map(|fieldnorm| fieldnorm as u64)\n            .sum();\n        let average_fieldnorm = (total_fieldnorms as Score) / (fieldnorms_expanded.len() as Score);\n        let max_doc = fieldnorms_expanded.len();\n\n        let term_scorers: Vec<TermScorer> = postings_lists_expanded\n            .iter()\n            .map(|postings| {\n                let bm25_weight = Bm25Weight::for_one_term(\n                    postings.len() as u64,\n                    max_doc as u64,\n                    average_fieldnorm,\n                );\n                TermScorer::create_for_test(postings, &fieldnorms_expanded[..], bm25_weight)\n            })\n            .collect();\n        for top_k in 1..4 {\n            let checkpoints_for_each_pruning =\n                compute_checkpoints_for_each_pruning(term_scorers.clone(), top_k);\n            let checkpoints_manual =\n                compute_checkpoints_manual(term_scorers.clone(), top_k, max_doc as u32);\n            assert_eq!(checkpoints_for_each_pruning.len(), checkpoints_manual.len());\n            for (&(left_doc, left_score), &(right_doc, right_score)) in checkpoints_for_each_pruning\n                .iter()\n                .zip(checkpoints_manual.iter())\n            {\n                assert_eq!(left_doc, right_doc);\n                assert!(nearly_equals(left_score, right_score));\n            }\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(500))]\n        #[test]\n        fn test_block_wand_two_term_scorers((posting_lists, fieldnorms) in gen_term_scorers(2)) {\n            test_block_wand_aux(&posting_lists[..], &fieldnorms[..]);\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(500))]\n        #[test]\n        fn test_block_wand_single_term_scorer((posting_lists, fieldnorms) in gen_term_scorers(1)) {\n            test_block_wand_aux(&posting_lists[..], &fieldnorms[..]);\n        }\n    }\n\n    #[test]\n    fn test_fn_reproduce_proptest() {\n        let postings_lists = &[\n            vec![\n                (0, 1),\n                (1, 1),\n                (2, 1),\n                (3, 1),\n                (4, 1),\n                (6, 1),\n                (7, 7),\n                (8, 1),\n                (10, 1),\n                (12, 1),\n                (13, 1),\n                (14, 1),\n                (15, 1),\n                (16, 1),\n                (19, 1),\n                (20, 1),\n                (21, 1),\n                (22, 1),\n                (24, 1),\n                (25, 1),\n                (26, 1),\n                (28, 1),\n                (30, 1),\n                (31, 1),\n                (33, 1),\n                (34, 1),\n                (35, 1),\n                (36, 95),\n                (37, 1),\n                (39, 1),\n                (41, 1),\n                (44, 1),\n                (46, 1),\n            ],\n            vec![\n                (0, 5),\n                (2, 1),\n                (4, 1),\n                (5, 84),\n                (6, 47),\n                (7, 26),\n                (8, 50),\n                (9, 34),\n                (11, 73),\n                (12, 11),\n                (13, 51),\n                (14, 45),\n                (15, 18),\n                (18, 60),\n                (19, 80),\n                (20, 63),\n                (23, 79),\n                (24, 69),\n                (26, 35),\n                (28, 82),\n                (29, 19),\n                (30, 2),\n                (31, 7),\n                (33, 40),\n                (34, 1),\n                (35, 33),\n                (36, 27),\n                (37, 24),\n                (38, 65),\n                (39, 32),\n                (40, 85),\n                (41, 1),\n                (42, 69),\n                (43, 11),\n                (45, 45),\n                (47, 97),\n            ],\n            vec![\n                (2, 1),\n                (4, 1),\n                (7, 94),\n                (8, 1),\n                (9, 1),\n                (10, 1),\n                (12, 1),\n                (15, 1),\n                (22, 1),\n                (23, 1),\n                (26, 1),\n                (27, 1),\n                (32, 1),\n                (33, 1),\n                (34, 1),\n                (36, 96),\n                (39, 1),\n                (41, 1),\n            ],\n        ];\n        let fieldnorms = &[\n            685, 239, 780, 564, 664, 827, 5, 56, 930, 887, 263, 665, 167, 127, 120, 919, 292, 92,\n            489, 734, 814, 724, 700, 304, 128, 779, 311, 877, 774, 15, 866, 368, 894, 371, 982,\n            502, 507, 669, 680, 76, 594, 626, 578, 331, 170, 639, 665, 186,\n        ][..];\n        test_block_wand_aux(postings_lists, fieldnorms);\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(500))]\n        #[ignore]\n        #[test]\n        #[ignore]\n        fn test_block_wand_three_term_scorers((posting_lists, fieldnorms) in gen_term_scorers(3)) {\n            test_block_wand_aux(&posting_lists[..], &fieldnorms[..]);\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/boolean_query/boolean_query.rs",
    "content": "use super::boolean_weight::BooleanWeight;\nuse crate::query::{EnableScoring, Occur, Query, SumCombiner, TermQuery, Weight};\nuse crate::schema::{IndexRecordOption, Term};\n\n/// The boolean query returns a set of documents\n/// that matches the Boolean combination of constituent subqueries.\n///\n/// The documents matched by the boolean query are those which\n/// - match all of the sub queries associated with the `Must` occurrence\n/// - match none of the sub queries associated with the `MustNot` occurrence.\n/// - match at least one of the sub queries associated with the `Must` or `Should` occurrence.\n///\n/// You can combine other query types and their `Occur`ances into one `BooleanQuery`\n///\n/// ```rust\n/// use tantivy::collector::Count;\n/// use tantivy::doc;\n/// use tantivy::query::{BooleanQuery, Occur, PhraseQuery, Query, TermQuery};\n/// use tantivy::schema::{IndexRecordOption, Schema, TEXT};\n/// use tantivy::Term;\n/// use tantivy::Index;\n/// use tantivy::IndexWriter;\n///\n/// fn main() -> tantivy::Result<()> {\n///    let mut schema_builder = Schema::builder();\n///    let title = schema_builder.add_text_field(\"title\", TEXT);\n///    let body = schema_builder.add_text_field(\"body\", TEXT);\n///    let schema = schema_builder.build();\n///    let index = Index::create_in_ram(schema);\n///    {\n///        let mut index_writer: IndexWriter = index.writer(15_000_000)?;\n///        index_writer.add_document(doc!(\n///            title => \"The Name of the Wind\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"The Diary of Muadib\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"A Dairy Cow\",\n///            body => \"hidden\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"A Dairy Cow\",\n///            body => \"found\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"The Diary of a Young Girl\",\n///        ))?;\n///        index_writer.commit()?;\n///    }\n///\n///    let reader = index.reader()?;\n///    let searcher = reader.searcher();\n///\n///    // Make TermQuery's for \"girl\" and \"diary\" in the title\n///    let girl_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(title, \"girl\"),\n///        IndexRecordOption::Basic,\n///    ));\n///    let diary_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(title, \"diary\"),\n///        IndexRecordOption::Basic,\n///    ));\n///    let cow_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(title, \"cow\"),\n///        IndexRecordOption::Basic\n///    ));\n///    // A TermQuery with \"found\" in the body\n///    let body_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(body, \"found\"),\n///        IndexRecordOption::Basic,\n///    ));\n///    // TermQuery \"diary\" must and \"girl\" must not be present\n///    let queries_with_occurs1 = vec![\n///        (Occur::Must, diary_term_query.box_clone()),\n///        (Occur::MustNot, girl_term_query.box_clone()),\n///    ];\n///    // Make a BooleanQuery equivalent to\n///    // title:+diary title:-girl\n///    let diary_must_and_girl_mustnot = BooleanQuery::new(queries_with_occurs1);\n///    let count1 = searcher.search(&diary_must_and_girl_mustnot, &Count)?;\n///    assert_eq!(count1, 1);\n///\n///    // \"title:diary OR title:cow\"\n///    let title_diary_or_cow = BooleanQuery::new(vec![\n///        (Occur::Should, diary_term_query.box_clone()),\n///        (Occur::Should, cow_term_query.box_clone()),\n///    ]);\n///    let count2 = searcher.search(&title_diary_or_cow, &Count)?;\n///    assert_eq!(count2, 4);\n///\n///    // Make a `PhraseQuery` from a vector of `Term`s\n///    let phrase_query: Box<dyn Query> = Box::new(PhraseQuery::new(vec![\n///        Term::from_field_text(title, \"dairy\"),\n///        Term::from_field_text(title, \"cow\"),\n///    ]));\n///    // You can combine subqueries of different types into 1 BooleanQuery:\n///    // `TermQuery` and `PhraseQuery`\n///    // \"title:diary OR \"dairy cow\"\n///    let term_of_phrase_query = BooleanQuery::new(vec![\n///        (Occur::Should, diary_term_query.box_clone()),\n///        (Occur::Should, phrase_query.box_clone()),\n///    ]);\n///    let count3 = searcher.search(&term_of_phrase_query, &Count)?;\n///    assert_eq!(count3, 4);\n///\n///    // You can nest one BooleanQuery inside another\n///    // body:found AND (\"title:diary OR \"dairy cow\")\n///    let nested_query = BooleanQuery::new(vec![\n///        (Occur::Must, body_term_query),\n///        (Occur::Must, Box::new(term_of_phrase_query))\n///    ]);\n///    let count4 = searcher.search(&nested_query, &Count)?;\n///    assert_eq!(count4, 1);\n///\n///    // You may call `with_minimum_required_clauses` to\n///    // specify the number of should clauses the returned documents must match.\n///    let minimum_required_query = BooleanQuery::with_minimum_required_clauses(vec![\n///         (Occur::Should, cow_term_query.box_clone()),\n///         (Occur::Should, girl_term_query.box_clone()),\n///         (Occur::Should, diary_term_query.box_clone()),\n///    ], 2);\n///    // Return documents contains \"Diary Cow\", \"Diary Girl\" or \"Cow Girl\"\n///    // Notice: \"Diary\" isn't \"Dairy\". ;-)\n///    let count5 = searcher.search(&minimum_required_query, &Count)?;\n///    assert_eq!(count5, 1);\n///    Ok(())\n/// }\n/// ```\n#[derive(Debug)]\npub struct BooleanQuery {\n    subqueries: Vec<(Occur, Box<dyn Query>)>,\n    minimum_number_should_match: usize,\n}\n\nimpl Clone for BooleanQuery {\n    fn clone(&self) -> Self {\n        let subqueries = self\n            .subqueries\n            .iter()\n            .map(|(occur, subquery)| (*occur, subquery.box_clone()))\n            .collect::<Vec<_>>();\n        Self {\n            subqueries,\n            minimum_number_should_match: self.minimum_number_should_match,\n        }\n    }\n}\n\nimpl From<Vec<(Occur, Box<dyn Query>)>> for BooleanQuery {\n    fn from(subqueries: Vec<(Occur, Box<dyn Query>)>) -> BooleanQuery {\n        BooleanQuery::new(subqueries)\n    }\n}\n\nimpl Query for BooleanQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let sub_weights = self\n            .subqueries\n            .iter()\n            .map(|(occur, subquery)| Ok((*occur, subquery.weight(enable_scoring)?)))\n            .collect::<crate::Result<_>>()?;\n        Ok(Box::new(BooleanWeight::with_minimum_number_should_match(\n            sub_weights,\n            self.minimum_number_should_match,\n            enable_scoring.is_scoring_enabled(),\n            Box::new(SumCombiner::default),\n        )))\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        for (_occur, subquery) in &self.subqueries {\n            subquery.query_terms(visitor);\n        }\n    }\n}\n\nimpl BooleanQuery {\n    /// Creates a new boolean query.\n    pub fn new(subqueries: Vec<(Occur, Box<dyn Query>)>) -> BooleanQuery {\n        // If the bool query includes at least one should clause\n        // and no Must or MustNot clauses, the default value is 1. Otherwise, the default value is\n        // 0. Keep compatible with Elasticsearch.\n        let mut minimum_required = 0;\n        for (occur, _) in &subqueries {\n            match occur {\n                Occur::Should => minimum_required = 1,\n                Occur::Must | Occur::MustNot => {\n                    minimum_required = 0;\n                    break;\n                }\n            }\n        }\n        Self::with_minimum_required_clauses(subqueries, minimum_required)\n    }\n\n    /// Create a new boolean query with minimum number of required should clauses specified.\n    pub fn with_minimum_required_clauses(\n        subqueries: Vec<(Occur, Box<dyn Query>)>,\n        minimum_number_should_match: usize,\n    ) -> BooleanQuery {\n        BooleanQuery {\n            subqueries,\n            minimum_number_should_match,\n        }\n    }\n\n    /// Getter for `minimum_number_should_match`\n    pub fn get_minimum_number_should_match(&self) -> usize {\n        self.minimum_number_should_match\n    }\n\n    /// Setter for `minimum_number_should_match`\n    pub fn set_minimum_number_should_match(&mut self, minimum_number_should_match: usize) {\n        self.minimum_number_should_match = minimum_number_should_match;\n    }\n\n    /// Returns the intersection of the queries.\n    pub fn intersection(queries: Vec<Box<dyn Query>>) -> BooleanQuery {\n        let subqueries = queries.into_iter().map(|s| (Occur::Must, s)).collect();\n        BooleanQuery::new(subqueries)\n    }\n\n    /// Returns the union of the queries.\n    pub fn union(queries: Vec<Box<dyn Query>>) -> BooleanQuery {\n        let subqueries = queries.into_iter().map(|s| (Occur::Should, s)).collect();\n        BooleanQuery::new(subqueries)\n    }\n\n    /// Returns the union of the queries with minimum required clause.\n    pub fn union_with_minimum_required_clauses(\n        queries: Vec<Box<dyn Query>>,\n        minimum_required_clauses: usize,\n    ) -> BooleanQuery {\n        let subqueries = queries\n            .into_iter()\n            .map(|sub_query| (Occur::Should, sub_query))\n            .collect();\n        BooleanQuery::with_minimum_required_clauses(subqueries, minimum_required_clauses)\n    }\n\n    /// Helper method to create a boolean query matching a given list of terms.\n    /// The resulting query is a disjunction of the terms.\n    pub fn new_multiterms_query(terms: Vec<Term>) -> BooleanQuery {\n        let occur_term_queries: Vec<(Occur, Box<dyn Query>)> = terms\n            .into_iter()\n            .map(|term| {\n                let term_query: Box<dyn Query> =\n                    Box::new(TermQuery::new(term, IndexRecordOption::WithFreqs));\n                (Occur::Should, term_query)\n            })\n            .collect();\n        BooleanQuery::new(occur_term_queries)\n    }\n\n    /// Deconstructed view of the clauses making up this query.\n    pub fn clauses(&self) -> &[(Occur, Box<dyn Query>)] {\n        &self.subqueries[..]\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n\n    use super::BooleanQuery;\n    use crate::collector::{Count, DocSetCollector};\n    use crate::query::{Query, QueryClone, QueryParser, TermQuery};\n    use crate::schema::{Field, IndexRecordOption, Schema, TEXT};\n    use crate::{DocAddress, DocId, Index, Term};\n\n    fn create_test_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests()?;\n        writer.add_document(doc!(text=>\"b c\"))?;\n        writer.add_document(doc!(text=>\"a c\"))?;\n        writer.add_document(doc!(text=>\"a b\"))?;\n        writer.add_document(doc!(text=>\"a d\"))?;\n        writer.commit()?;\n        Ok(index)\n    }\n\n    #[test]\n    fn test_minimum_required() -> crate::Result<()> {\n        fn create_test_index_with<T: IntoIterator<Item = &'static str>>(\n            docs: T,\n        ) -> crate::Result<Index> {\n            let mut schema_builder = Schema::builder();\n            let text = schema_builder.add_text_field(\"text\", TEXT);\n            let schema = schema_builder.build();\n            let index = Index::create_in_ram(schema);\n            let mut writer = index.writer_for_tests()?;\n            for doc in docs {\n                writer.add_document(doc!(text => doc))?;\n            }\n            writer.commit()?;\n            Ok(index)\n        }\n        fn create_boolean_query_with_mr<T: IntoIterator<Item = &'static str>>(\n            queries: T,\n            field: Field,\n            mr: usize,\n        ) -> BooleanQuery {\n            let terms = queries\n                .into_iter()\n                .map(|t| Term::from_field_text(field, t))\n                .map(|t| TermQuery::new(t, IndexRecordOption::Basic))\n                .map(|q| -> Box<dyn Query> { Box::new(q) })\n                .collect();\n            BooleanQuery::union_with_minimum_required_clauses(terms, mr)\n        }\n        fn check_doc_id<T: IntoIterator<Item = DocId>>(\n            expected: T,\n            actually: HashSet<DocAddress>,\n            seg: u32,\n        ) {\n            assert_eq!(\n                actually,\n                expected\n                    .into_iter()\n                    .map(|id| DocAddress::new(seg, id))\n                    .collect()\n            );\n        }\n        let index = create_test_index_with([\"a b c\", \"a c e\", \"d f g\", \"z z z\", \"c i b\"])?;\n        let searcher = index.reader()?.searcher();\n        let text = index.schema().get_field(\"text\").unwrap();\n        // Documents contains 'a c' 'a z' 'a i' 'c z' 'c i' or 'z i' shall be return.\n        let q1 = create_boolean_query_with_mr([\"a\", \"c\", \"z\", \"i\"], text, 2);\n        let docs = searcher.search(&q1, &DocSetCollector)?;\n        check_doc_id([0, 1, 4], docs, 0);\n        // Documents contains 'a b c', 'a b e', 'a c e' or 'b c e' shall be return.\n        let q2 = create_boolean_query_with_mr([\"a\", \"b\", \"c\", \"e\"], text, 3);\n        let docs = searcher.search(&q2, &DocSetCollector)?;\n        check_doc_id([0, 1], docs, 0);\n        // Nothing queried since minimum_required is too large.\n        let q3 = create_boolean_query_with_mr([\"a\", \"b\"], text, 3);\n        let docs = searcher.search(&q3, &DocSetCollector)?;\n        assert!(docs.is_empty());\n        // When mr is set to zero or one, there are no difference with `Boolean::Union`.\n        let q4 = create_boolean_query_with_mr([\"a\", \"z\"], text, 1);\n        let docs = searcher.search(&q4, &DocSetCollector)?;\n        check_doc_id([0, 1, 3], docs, 0);\n        let q5 = create_boolean_query_with_mr([\"a\", \"b\"], text, 0);\n        let docs = searcher.search(&q5, &DocSetCollector)?;\n        check_doc_id([0, 1, 4], docs, 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_union() -> crate::Result<()> {\n        let index = create_test_index()?;\n        let searcher = index.reader()?.searcher();\n        let text = index.schema().get_field(\"text\").unwrap();\n        let term_a = TermQuery::new(Term::from_field_text(text, \"a\"), IndexRecordOption::Basic);\n        let term_d = TermQuery::new(Term::from_field_text(text, \"d\"), IndexRecordOption::Basic);\n        let union_ad = BooleanQuery::union(vec![term_a.box_clone(), term_d.box_clone()]);\n        let docs = searcher.search(&union_ad, &DocSetCollector)?;\n        assert_eq!(\n            docs,\n            vec![\n                DocAddress::new(0u32, 1u32),\n                DocAddress::new(0u32, 2u32),\n                DocAddress::new(0u32, 3u32)\n            ]\n            .into_iter()\n            .collect()\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_intersection() -> crate::Result<()> {\n        let index = create_test_index()?;\n        let searcher = index.reader()?.searcher();\n        let text = index.schema().get_field(\"text\").unwrap();\n        let term_a = TermQuery::new(Term::from_field_text(text, \"a\"), IndexRecordOption::Basic);\n        let term_b = TermQuery::new(Term::from_field_text(text, \"b\"), IndexRecordOption::Basic);\n        let term_c = TermQuery::new(Term::from_field_text(text, \"c\"), IndexRecordOption::Basic);\n        let intersection_ab =\n            BooleanQuery::intersection(vec![term_a.box_clone(), term_b.box_clone()]);\n        let intersection_ac =\n            BooleanQuery::intersection(vec![term_a.box_clone(), term_c.box_clone()]);\n        let intersection_bc =\n            BooleanQuery::intersection(vec![term_b.box_clone(), term_c.box_clone()]);\n        {\n            let docs = searcher.search(&intersection_ab, &DocSetCollector)?;\n            assert_eq!(\n                docs,\n                vec![DocAddress::new(0u32, 2u32)].into_iter().collect()\n            );\n        }\n        {\n            let docs = searcher.search(&intersection_ac, &DocSetCollector)?;\n            assert_eq!(\n                docs,\n                vec![DocAddress::new(0u32, 1u32)].into_iter().collect()\n            );\n        }\n        {\n            let docs = searcher.search(&intersection_bc, &DocSetCollector)?;\n            assert_eq!(\n                docs,\n                vec![DocAddress::new(0u32, 0u32)].into_iter().collect()\n            );\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_json_array_pitfall_bag_of_terms() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(json_field=>json!({\n                \"cart\": [\n                    {\"product_type\": \"sneakers\", \"attributes\": {\"color\": \"white\"}},\n                    {\"product_type\": \"t-shirt\", \"attributes\": {\"color\": \"red\"}},\n                    {\"product_type\": \"cd\", \"attributes\": {\"genre\": \"blues\"}},\n                ]\n            })))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let doc_matches = |query: &str| {\n            let query_parser = QueryParser::for_index(&index, vec![json_field]);\n            let query = query_parser.parse_query(query).unwrap();\n            searcher.search(&query, &Count).unwrap() == 1\n        };\n        // As expected\n        assert!(doc_matches(\n            r#\"cart.product_type:sneakers AND cart.attributes.color:white\"#\n        ));\n        // Unexpected match, due to the fact that array do not act as nested docs.\n        assert!(doc_matches(\n            r#\"cart.product_type:sneakers AND cart.attributes.color:red\"#\n        ));\n        // However, bviously this works...\n        assert!(!doc_matches(\n            r#\"cart.product_type:sneakers AND cart.attributes.color:blues\"#\n        ));\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/boolean_query/boolean_weight.rs",
    "content": "use std::collections::HashMap;\n\nuse crate::docset::COLLECT_BLOCK_BUFFER_LEN;\nuse crate::index::SegmentReader;\nuse crate::postings::FreqReadingOption;\nuse crate::query::disjunction::Disjunction;\nuse crate::query::explanation::does_not_match;\nuse crate::query::score_combiner::{DoNothingCombiner, ScoreCombiner};\nuse crate::query::term_query::TermScorer;\nuse crate::query::weight::{for_each_docset_buffered, for_each_pruning_scorer, for_each_scorer};\nuse crate::query::{\n    intersect_scorers, AllScorer, BufferedUnionScorer, EmptyScorer, Exclude, Explanation, Occur,\n    RequiredOptionalScorer, Scorer, Weight,\n};\nuse crate::{DocId, Score};\n\nenum SpecializedScorer {\n    TermUnion(Vec<TermScorer>),\n    Other(Box<dyn Scorer>),\n}\n\nfn scorer_disjunction<TScoreCombiner>(\n    scorers: Vec<Box<dyn Scorer>>,\n    score_combiner: TScoreCombiner,\n    minimum_match_required: usize,\n) -> Box<dyn Scorer>\nwhere\n    TScoreCombiner: ScoreCombiner,\n{\n    debug_assert!(!scorers.is_empty());\n    debug_assert!(minimum_match_required > 1);\n    if scorers.len() == 1 {\n        return scorers.into_iter().next().unwrap(); // Safe unwrap.\n    }\n    Box::new(Disjunction::new(\n        scorers,\n        score_combiner,\n        minimum_match_required,\n    ))\n}\n\n/// num_docs is the number of documents in the segment.\nfn scorer_union<TScoreCombiner>(\n    scorers: Vec<Box<dyn Scorer>>,\n    score_combiner_fn: impl Fn() -> TScoreCombiner,\n    num_docs: u32,\n) -> SpecializedScorer\nwhere\n    TScoreCombiner: ScoreCombiner,\n{\n    assert!(!scorers.is_empty());\n    if scorers.len() == 1 {\n        return SpecializedScorer::Other(scorers.into_iter().next().unwrap()); //< we checked the size beforehand\n    }\n\n    {\n        let is_all_term_queries = scorers.iter().all(|scorer| scorer.is::<TermScorer>());\n        if is_all_term_queries {\n            let scorers: Vec<TermScorer> = scorers\n                .into_iter()\n                .map(|scorer| *(scorer.downcast::<TermScorer>().map_err(|_| ()).unwrap()))\n                .collect();\n            if scorers\n                .iter()\n                .all(|scorer| scorer.freq_reading_option() == FreqReadingOption::ReadFreq)\n            {\n                // Block wand is only available if we read frequencies.\n                return SpecializedScorer::TermUnion(scorers);\n            } else {\n                return SpecializedScorer::Other(Box::new(BufferedUnionScorer::build(\n                    scorers,\n                    score_combiner_fn,\n                    num_docs,\n                )));\n            }\n        }\n    }\n    SpecializedScorer::Other(Box::new(BufferedUnionScorer::build(\n        scorers,\n        score_combiner_fn,\n        num_docs,\n    )))\n}\n\nfn into_box_scorer<TScoreCombiner: ScoreCombiner>(\n    scorer: SpecializedScorer,\n    score_combiner_fn: impl Fn() -> TScoreCombiner,\n    num_docs: u32,\n) -> Box<dyn Scorer> {\n    match scorer {\n        SpecializedScorer::TermUnion(term_scorers) => {\n            let union_scorer =\n                BufferedUnionScorer::build(term_scorers, score_combiner_fn, num_docs);\n            Box::new(union_scorer)\n        }\n        SpecializedScorer::Other(scorer) => scorer,\n    }\n}\n\n/// Returns the effective MUST scorer, accounting for removed AllScorers.\n///\n/// When AllScorer instances are removed from must_scorers as an optimization,\n/// we must restore the \"match all\" semantics if the list becomes empty.\nfn effective_must_scorer(\n    must_scorers: Vec<Box<dyn Scorer>>,\n    removed_all_scorer_count: usize,\n    max_doc: DocId,\n    num_docs: u32,\n) -> Option<Box<dyn Scorer>> {\n    if must_scorers.is_empty() {\n        if removed_all_scorer_count > 0 {\n            // Had AllScorer(s) only - all docs match\n            Some(Box::new(AllScorer::new(max_doc)))\n        } else {\n            // No MUST constraint at all\n            None\n        }\n    } else {\n        Some(intersect_scorers(must_scorers, num_docs))\n    }\n}\n\n/// Returns a SHOULD scorer with AllScorer union if any were removed.\n///\n/// For union semantics (OR): if any SHOULD clause was an AllScorer, the result\n/// should include all documents. We restore this by unioning with AllScorer.\n///\n/// When `scoring_enabled` is false, we can just return AllScorer alone since\n/// we don't need score contributions from the should_scorer.\nfn effective_should_scorer_for_union<TScoreCombiner: ScoreCombiner>(\n    should_scorer: SpecializedScorer,\n    removed_all_scorer_count: usize,\n    max_doc: DocId,\n    num_docs: u32,\n    score_combiner_fn: impl Fn() -> TScoreCombiner,\n    scoring_enabled: bool,\n) -> SpecializedScorer {\n    if removed_all_scorer_count > 0 {\n        if scoring_enabled {\n            // Need to union to get score contributions from both\n            let all_scorers: Vec<Box<dyn Scorer>> = vec![\n                into_box_scorer(should_scorer, &score_combiner_fn, num_docs),\n                Box::new(AllScorer::new(max_doc)),\n            ];\n            SpecializedScorer::Other(Box::new(BufferedUnionScorer::build(\n                all_scorers,\n                score_combiner_fn,\n                num_docs,\n            )))\n        } else {\n            // Scoring disabled - AllScorer alone is sufficient\n            SpecializedScorer::Other(Box::new(AllScorer::new(max_doc)))\n        }\n    } else {\n        should_scorer\n    }\n}\n\nenum ShouldScorersCombinationMethod {\n    // Should scorers are irrelevant.\n    Ignored,\n    // Only contributes to final score.\n    Optional(SpecializedScorer),\n    // Regardless of score, the should scorers may impact whether a document is matching or not.\n    Required(SpecializedScorer),\n}\n\n/// Weight associated to the `BoolQuery`.\npub struct BooleanWeight<TScoreCombiner: ScoreCombiner> {\n    weights: Vec<(Occur, Box<dyn Weight>)>,\n    minimum_number_should_match: usize,\n    scoring_enabled: bool,\n    score_combiner_fn: Box<dyn Fn() -> TScoreCombiner + Sync + Send>,\n}\n\nimpl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {\n    /// Creates a new boolean weight.\n    pub fn new(\n        weights: Vec<(Occur, Box<dyn Weight>)>,\n        scoring_enabled: bool,\n        score_combiner_fn: Box<dyn Fn() -> TScoreCombiner + Sync + Send + 'static>,\n    ) -> BooleanWeight<TScoreCombiner> {\n        BooleanWeight {\n            weights,\n            scoring_enabled,\n            score_combiner_fn,\n            minimum_number_should_match: 1,\n        }\n    }\n\n    /// Create a new boolean weight with minimum number of required should clauses specified.\n    pub fn with_minimum_number_should_match(\n        weights: Vec<(Occur, Box<dyn Weight>)>,\n        minimum_number_should_match: usize,\n        scoring_enabled: bool,\n        score_combiner_fn: Box<dyn Fn() -> TScoreCombiner + Sync + Send + 'static>,\n    ) -> BooleanWeight<TScoreCombiner> {\n        BooleanWeight {\n            weights,\n            minimum_number_should_match,\n            scoring_enabled,\n            score_combiner_fn,\n        }\n    }\n\n    fn per_occur_scorers(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<HashMap<Occur, Vec<Box<dyn Scorer>>>> {\n        let mut per_occur_scorers: HashMap<Occur, Vec<Box<dyn Scorer>>> = HashMap::new();\n        for (occur, subweight) in &self.weights {\n            let sub_scorer: Box<dyn Scorer> = subweight.scorer(reader, boost)?;\n            per_occur_scorers\n                .entry(*occur)\n                .or_default()\n                .push(sub_scorer);\n        }\n        Ok(per_occur_scorers)\n    }\n\n    fn complex_scorer<TComplexScoreCombiner: ScoreCombiner>(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n        score_combiner_fn: impl Fn() -> TComplexScoreCombiner,\n    ) -> crate::Result<SpecializedScorer> {\n        let num_docs = reader.num_docs();\n        let mut per_occur_scorers = self.per_occur_scorers(reader, boost)?;\n\n        // Indicate how should clauses are combined with must clauses.\n        let mut must_scorers: Vec<Box<dyn Scorer>> =\n            per_occur_scorers.remove(&Occur::Must).unwrap_or_default();\n        let must_special_scorer_counts = remove_and_count_all_and_empty_scorers(&mut must_scorers);\n\n        if must_special_scorer_counts.num_empty_scorers > 0 {\n            return Ok(SpecializedScorer::Other(Box::new(EmptyScorer)));\n        }\n\n        let mut should_scorers = per_occur_scorers.remove(&Occur::Should).unwrap_or_default();\n        let should_special_scorer_counts =\n            remove_and_count_all_and_empty_scorers(&mut should_scorers);\n\n        let mut exclude_scorers: Vec<Box<dyn Scorer>> = per_occur_scorers\n            .remove(&Occur::MustNot)\n            .unwrap_or_default();\n        let exclude_special_scorer_counts =\n            remove_and_count_all_and_empty_scorers(&mut exclude_scorers);\n\n        if exclude_special_scorer_counts.num_all_scorers > 0 {\n            // We exclude all documents at one point.\n            return Ok(SpecializedScorer::Other(Box::new(EmptyScorer)));\n        }\n\n        let effective_minimum_number_should_match = self\n            .minimum_number_should_match\n            .saturating_sub(should_special_scorer_counts.num_all_scorers);\n\n        let should_scorers: ShouldScorersCombinationMethod = {\n            let num_of_should_scorers = should_scorers.len();\n            if effective_minimum_number_should_match > num_of_should_scorers {\n                // We don't have enough scorers to satisfy the minimum number of should matches.\n                // The request will match no documents.\n                return Ok(SpecializedScorer::Other(Box::new(EmptyScorer)));\n            }\n            match effective_minimum_number_should_match {\n                0 if num_of_should_scorers == 0 => ShouldScorersCombinationMethod::Ignored,\n                0 => ShouldScorersCombinationMethod::Optional(scorer_union(\n                    should_scorers,\n                    &score_combiner_fn,\n                    num_docs,\n                )),\n                1 => ShouldScorersCombinationMethod::Required(scorer_union(\n                    should_scorers,\n                    &score_combiner_fn,\n                    num_docs,\n                )),\n                n if num_of_should_scorers == n => {\n                    // When num_of_should_scorers equals the number of should clauses,\n                    // they are no different from must clauses.\n                    must_scorers.append(&mut should_scorers);\n                    ShouldScorersCombinationMethod::Ignored\n                }\n                _ => ShouldScorersCombinationMethod::Required(SpecializedScorer::Other(\n                    scorer_disjunction(\n                        should_scorers,\n                        score_combiner_fn(),\n                        effective_minimum_number_should_match,\n                    ),\n                )),\n            }\n        };\n\n        let include_scorer = match (should_scorers, must_scorers) {\n            (ShouldScorersCombinationMethod::Ignored, must_scorers) => {\n                // No SHOULD clauses (or they were absorbed into MUST).\n                // Result depends entirely on MUST + any removed AllScorers.\n                let combined_all_scorer_count = must_special_scorer_counts.num_all_scorers\n                    + should_special_scorer_counts.num_all_scorers;\n                let boxed_scorer: Box<dyn Scorer> = effective_must_scorer(\n                    must_scorers,\n                    combined_all_scorer_count,\n                    reader.max_doc(),\n                    num_docs,\n                )\n                .unwrap_or_else(|| Box::new(EmptyScorer));\n                SpecializedScorer::Other(boxed_scorer)\n            }\n            (ShouldScorersCombinationMethod::Optional(should_scorer), must_scorers) => {\n                // Optional SHOULD: contributes to scoring but not required for matching.\n                match effective_must_scorer(\n                    must_scorers,\n                    must_special_scorer_counts.num_all_scorers,\n                    reader.max_doc(),\n                    num_docs,\n                ) {\n                    None => {\n                        // No MUST constraint: promote SHOULD to required.\n                        // Must preserve any removed AllScorers from SHOULD via union.\n                        effective_should_scorer_for_union(\n                            should_scorer,\n                            should_special_scorer_counts.num_all_scorers,\n                            reader.max_doc(),\n                            num_docs,\n                            &score_combiner_fn,\n                            self.scoring_enabled,\n                        )\n                    }\n                    Some(must_scorer) => {\n                        // Has MUST constraint: SHOULD only affects scoring.\n                        if self.scoring_enabled {\n                            SpecializedScorer::Other(Box::new(RequiredOptionalScorer::<\n                                _,\n                                _,\n                                TScoreCombiner,\n                            >::new(\n                                must_scorer,\n                                into_box_scorer(should_scorer, &score_combiner_fn, num_docs),\n                            )))\n                        } else {\n                            SpecializedScorer::Other(must_scorer)\n                        }\n                    }\n                }\n            }\n            (ShouldScorersCombinationMethod::Required(should_scorer), must_scorers) => {\n                // Required SHOULD: at least `minimum_number_should_match` must match.\n                // Semantics: (MUST constraint) AND (SHOULD constraint)\n                match effective_must_scorer(\n                    must_scorers,\n                    must_special_scorer_counts.num_all_scorers,\n                    reader.max_doc(),\n                    num_docs,\n                ) {\n                    None => {\n                        // No MUST constraint: SHOULD alone determines matching.\n                        should_scorer\n                    }\n                    Some(must_scorer) => {\n                        // Has MUST constraint: intersect MUST with SHOULD.\n                        let should_boxed =\n                            into_box_scorer(should_scorer, &score_combiner_fn, num_docs);\n                        SpecializedScorer::Other(intersect_scorers(\n                            vec![must_scorer, should_boxed],\n                            num_docs,\n                        ))\n                    }\n                }\n            }\n        };\n        if exclude_scorers.is_empty() {\n            return Ok(include_scorer);\n        }\n\n        let include_scorer_boxed = into_box_scorer(include_scorer, &score_combiner_fn, num_docs);\n        let scorer: Box<dyn Scorer> = if exclude_scorers.len() == 1 {\n            let exclude_scorer = exclude_scorers.pop().unwrap();\n            match exclude_scorer.downcast::<TermScorer>() {\n                // Cast to TermScorer succeeded\n                Ok(exclude_scorer) => Box::new(Exclude::new(include_scorer_boxed, *exclude_scorer)),\n                // We get back the original Box<dyn Scorer>\n                Err(exclude_scorer) => Box::new(Exclude::new(include_scorer_boxed, exclude_scorer)),\n            }\n        } else {\n            Box::new(Exclude::new(include_scorer_boxed, exclude_scorers))\n        };\n        Ok(SpecializedScorer::Other(scorer))\n    }\n}\n\n#[derive(Default, Copy, Clone, Debug)]\nstruct AllAndEmptyScorerCounts {\n    num_all_scorers: usize,\n    num_empty_scorers: usize,\n}\n\nfn remove_and_count_all_and_empty_scorers(\n    scorers: &mut Vec<Box<dyn Scorer>>,\n) -> AllAndEmptyScorerCounts {\n    let mut counts = AllAndEmptyScorerCounts::default();\n    scorers.retain(|scorer| {\n        if scorer.is::<AllScorer>() {\n            counts.num_all_scorers += 1;\n            false\n        } else if scorer.is::<EmptyScorer>() {\n            counts.num_empty_scorers += 1;\n            false\n        } else {\n            true\n        }\n    });\n    counts\n}\n\nimpl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombiner> {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let num_docs = reader.num_docs();\n        if self.weights.is_empty() {\n            Ok(Box::new(EmptyScorer))\n        } else if self.weights.len() == 1 {\n            let &(occur, ref weight) = &self.weights[0];\n            if occur == Occur::MustNot {\n                Ok(Box::new(EmptyScorer))\n            } else {\n                weight.scorer(reader, boost)\n            }\n        } else if self.scoring_enabled {\n            self.complex_scorer(reader, boost, &self.score_combiner_fn)\n                .map(|specialized_scorer| {\n                    into_box_scorer(specialized_scorer, &self.score_combiner_fn, num_docs)\n                })\n        } else {\n            self.complex_scorer(reader, boost, DoNothingCombiner::default)\n                .map(|specialized_scorer| {\n                    into_box_scorer(specialized_scorer, DoNothingCombiner::default, num_docs)\n                })\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        if !self.scoring_enabled {\n            return Ok(Explanation::new(\"BooleanQuery with no scoring\", 1.0));\n        }\n\n        let mut explanation = Explanation::new(\"BooleanClause. sum of ...\", scorer.score());\n        for (occur, subweight) in &self.weights {\n            if is_include_occur(*occur) {\n                if let Ok(child_explanation) = subweight.explain(reader, doc) {\n                    explanation.add_detail(child_explanation);\n                }\n            }\n        }\n        Ok(explanation)\n    }\n\n    fn for_each(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score),\n    ) -> crate::Result<()> {\n        let scorer = self.complex_scorer(reader, 1.0, &self.score_combiner_fn)?;\n        match scorer {\n            SpecializedScorer::TermUnion(term_scorers) => {\n                let mut union_scorer = BufferedUnionScorer::build(\n                    term_scorers,\n                    &self.score_combiner_fn,\n                    reader.num_docs(),\n                );\n                for_each_scorer(&mut union_scorer, callback);\n            }\n            SpecializedScorer::Other(mut scorer) => {\n                for_each_scorer(scorer.as_mut(), callback);\n            }\n        }\n        Ok(())\n    }\n\n    fn for_each_no_score(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(&[DocId]),\n    ) -> crate::Result<()> {\n        let scorer = self.complex_scorer(reader, 1.0, || DoNothingCombiner)?;\n        let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n\n        match scorer {\n            SpecializedScorer::TermUnion(term_scorers) => {\n                let mut union_scorer = BufferedUnionScorer::build(\n                    term_scorers,\n                    &self.score_combiner_fn,\n                    reader.num_docs(),\n                );\n                for_each_docset_buffered(&mut union_scorer, &mut buffer, callback);\n            }\n            SpecializedScorer::Other(mut scorer) => {\n                for_each_docset_buffered(scorer.as_mut(), &mut buffer, callback);\n            }\n        }\n        Ok(())\n    }\n\n    /// Calls `callback` with all of the `(doc, score)` for which score\n    /// is exceeding a given threshold.\n    ///\n    /// This method is useful for the TopDocs collector.\n    /// For all docsets, the blanket implementation has the benefit\n    /// of prefiltering (doc, score) pairs, avoiding the\n    /// virtual dispatch cost.\n    ///\n    /// More importantly, it makes it possible for scorers to implement\n    /// important optimization (e.g. BlockWAND for union).\n    fn for_each_pruning(\n        &self,\n        threshold: Score,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score) -> Score,\n    ) -> crate::Result<()> {\n        let scorer = self.complex_scorer(reader, 1.0, &self.score_combiner_fn)?;\n        match scorer {\n            SpecializedScorer::TermUnion(term_scorers) => {\n                super::block_wand(term_scorers, threshold, callback);\n            }\n            SpecializedScorer::Other(mut scorer) => {\n                for_each_pruning_scorer(scorer.as_mut(), threshold, callback);\n            }\n        }\n        Ok(())\n    }\n}\n\nfn is_include_occur(occur: Occur) -> bool {\n    match occur {\n        Occur::Must | Occur::Should => true,\n        Occur::MustNot => false,\n    }\n}\n"
  },
  {
    "path": "src/query/boolean_query/mod.rs",
    "content": "mod block_wand;\nmod boolean_query;\nmod boolean_weight;\n\npub(crate) use self::block_wand::{block_wand, block_wand_single_scorer};\npub use self::boolean_query::BooleanQuery;\npub use self::boolean_weight::BooleanWeight;\n\n#[cfg(test)]\nmod tests {\n\n    use std::ops::Bound;\n\n    use super::*;\n    use crate::collector::tests::TEST_COLLECTOR_WITH_SCORE;\n    use crate::collector::{Count, TopDocs};\n    use crate::query::term_query::TermScorer;\n    use crate::query::{\n        AllScorer, EmptyScorer, EnableScoring, Intersection, Occur, Query, QueryParser, RangeQuery,\n        RequiredOptionalScorer, Scorer, SumCombiner, TermQuery,\n    };\n    use crate::schema::*;\n    use crate::{assert_nearly_equals, DocAddress, DocId, Index, IndexWriter, Score};\n\n    fn aux_test_helper() -> crate::Result<(Index, Field)> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field => \"a b c\"))?;\n            index_writer.add_document(doc!(text_field => \"a c\"))?;\n            index_writer.add_document(doc!(text_field => \"b c\"))?;\n            index_writer.add_document(doc!(text_field => \"a b c d\"))?;\n            index_writer.add_document(doc!(text_field => \"d\"))?;\n            index_writer.commit()?;\n        }\n        Ok((index, text_field))\n    }\n\n    #[test]\n    pub fn test_boolean_non_all_term_disjunction() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let query = query_parser.parse_query(\"(+a +b) d\")?;\n        let searcher = index.reader()?.searcher();\n        assert_eq!(query.count(&searcher)?, 3);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_single_must_clause() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let query = query_parser.parse_query(\"+a\")?;\n        let searcher = index.reader()?.searcher();\n        let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n        let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n        assert!(scorer.is::<TermScorer>());\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_termonly_intersection() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let searcher = index.reader()?.searcher();\n        {\n            let query = query_parser.parse_query(\"+a +b +c\")?;\n            let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n            assert!(scorer.is::<Intersection<TermScorer>>());\n        }\n        {\n            let query = query_parser.parse_query(\"+a +(b c)\")?;\n            let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n            assert!(scorer.is::<Intersection<Box<dyn Scorer>>>());\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_reqopt() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let searcher = index.reader()?.searcher();\n        {\n            let query = query_parser.parse_query(\"+a b\")?;\n            let weight = query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n            assert!(scorer\n                .is::<RequiredOptionalScorer<Box<dyn Scorer>, Box<dyn Scorer>, SumCombiner>>());\n        }\n        {\n            let query = query_parser.parse_query(\"+a b\")?;\n            let weight = query.weight(EnableScoring::disabled_from_schema(searcher.schema()))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n            assert!(scorer.is::<TermScorer>());\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_query() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n\n        let make_term_query = |text: &str| {\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, text),\n                IndexRecordOption::Basic,\n            );\n            let query: Box<dyn Query> = Box::new(term_query);\n            query\n        };\n\n        let reader = index.reader()?;\n\n        let matching_docs = |boolean_query: &dyn Query| {\n            reader\n                .searcher()\n                .search(boolean_query, &TEST_COLLECTOR_WITH_SCORE)\n                .unwrap()\n                .docs()\n                .iter()\n                .cloned()\n                .map(|doc| doc.doc_id)\n                .collect::<Vec<DocId>>()\n        };\n        {\n            let boolean_query = BooleanQuery::new(vec![(Occur::Must, make_term_query(\"a\"))]);\n            assert_eq!(matching_docs(&boolean_query), vec![0, 1, 3]);\n        }\n        {\n            let boolean_query = BooleanQuery::new(vec![(Occur::Should, make_term_query(\"a\"))]);\n            assert_eq!(matching_docs(&boolean_query), vec![0, 1, 3]);\n        }\n        {\n            let boolean_query = BooleanQuery::new(vec![\n                (Occur::Should, make_term_query(\"a\")),\n                (Occur::Should, make_term_query(\"b\")),\n            ]);\n            assert_eq!(matching_docs(&boolean_query), vec![0, 1, 2, 3]);\n        }\n        {\n            let boolean_query = BooleanQuery::new(vec![\n                (Occur::Must, make_term_query(\"a\")),\n                (Occur::Should, make_term_query(\"b\")),\n            ]);\n            assert_eq!(matching_docs(&boolean_query), vec![0, 1, 3]);\n        }\n        {\n            let boolean_query = BooleanQuery::new(vec![\n                (Occur::Must, make_term_query(\"a\")),\n                (Occur::Should, make_term_query(\"b\")),\n                (Occur::MustNot, make_term_query(\"d\")),\n            ]);\n            assert_eq!(matching_docs(&boolean_query), vec![0, 1]);\n        }\n        {\n            let boolean_query = BooleanQuery::new(vec![(Occur::MustNot, make_term_query(\"d\"))]);\n            assert_eq!(matching_docs(&boolean_query), Vec::<u32>::new());\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_query_two_excluded() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n\n        let make_term_query = |text: &str| {\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, text),\n                IndexRecordOption::Basic,\n            );\n            let query: Box<dyn Query> = Box::new(term_query);\n            query\n        };\n\n        let reader = index.reader()?;\n\n        let matching_topdocs = |query: &dyn Query| {\n            reader\n                .searcher()\n                .search(query, &TopDocs::with_limit(3).order_by_score())\n                .unwrap()\n        };\n\n        let score_doc_4: Score; // score of doc 4 should not be influenced by exclusion\n        {\n            let boolean_query_no_excluded =\n                BooleanQuery::new(vec![(Occur::Must, make_term_query(\"d\"))]);\n            let topdocs_no_excluded = matching_topdocs(&boolean_query_no_excluded);\n            assert_eq!(topdocs_no_excluded.len(), 2);\n            let (top_score, top_doc) = topdocs_no_excluded[0];\n            assert_eq!(top_doc, DocAddress::new(0, 4));\n            assert_eq!(topdocs_no_excluded[1].1, DocAddress::new(0, 3)); // ignore score of doc 3.\n            score_doc_4 = top_score;\n        }\n\n        {\n            let boolean_query_two_excluded = BooleanQuery::new(vec![\n                (Occur::Must, make_term_query(\"d\")),\n                (Occur::MustNot, make_term_query(\"a\")),\n                (Occur::MustNot, make_term_query(\"b\")),\n            ]);\n            let topdocs_excluded = matching_topdocs(&boolean_query_two_excluded);\n            assert_eq!(topdocs_excluded.len(), 1);\n            let (top_score, top_doc) = topdocs_excluded[0];\n            assert_eq!(top_doc, DocAddress::new(0, 4));\n            assert_eq!(top_score, score_doc_4);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_query_with_weight() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field => \"a b c\"))?;\n            index_writer.add_document(doc!(text_field => \"a c\"))?;\n            index_writer.add_document(doc!(text_field => \"b c\"))?;\n            index_writer.commit()?;\n        }\n        let term_a: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"a\"),\n            IndexRecordOption::WithFreqs,\n        ));\n        let term_b: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"b\"),\n            IndexRecordOption::WithFreqs,\n        ));\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let boolean_query =\n            BooleanQuery::new(vec![(Occur::Should, term_a), (Occur::Should, term_b)]);\n        let boolean_weight = boolean_query\n            .weight(EnableScoring::enabled_from_searcher(&searcher))\n            .unwrap();\n        {\n            let mut boolean_scorer = boolean_weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n            assert_eq!(boolean_scorer.doc(), 0u32);\n            assert_nearly_equals!(boolean_scorer.score(), 0.84163445);\n        }\n        {\n            let mut boolean_scorer = boolean_weight.scorer(searcher.segment_reader(0u32), 2.0)?;\n            assert_eq!(boolean_scorer.doc(), 0u32);\n            assert_nearly_equals!(boolean_scorer.score(), 1.6832689);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_intersection_score() -> crate::Result<()> {\n        let (index, text_field) = aux_test_helper()?;\n\n        let make_term_query = |text: &str| {\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, text),\n                IndexRecordOption::Basic,\n            );\n            let query: Box<dyn Query> = Box::new(term_query);\n            query\n        };\n        let reader = index.reader()?;\n        let score_docs = |boolean_query: &dyn Query| {\n            let fruit = reader\n                .searcher()\n                .search(boolean_query, &TEST_COLLECTOR_WITH_SCORE)\n                .unwrap();\n            fruit.scores().to_vec()\n        };\n\n        {\n            let boolean_query = BooleanQuery::new(vec![\n                (Occur::Must, make_term_query(\"a\")),\n                (Occur::Must, make_term_query(\"b\")),\n            ]);\n            let scores = score_docs(&boolean_query);\n            assert_nearly_equals!(scores[0], 0.977973);\n            assert_nearly_equals!(scores[1], 0.84699446);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_explain() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text = schema_builder.add_text_field(\"text\", STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text=>\"a\"))?;\n        index_writer.add_document(doc!(text=>\"b\"))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let term_a: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text, \"a\"),\n            IndexRecordOption::Basic,\n        ));\n        let term_b: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text, \"b\"),\n            IndexRecordOption::Basic,\n        ));\n        let query = BooleanQuery::from(vec![(Occur::Should, term_a), (Occur::Should, term_b)]);\n        let explanation = query.explain(&searcher, DocAddress::new(0, 0u32))?;\n        assert_nearly_equals!(explanation.value(), std::f32::consts::LN_2);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_boolean_weight_optimization() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"hello\"))?;\n        index_writer.add_document(doc!(text_field=>\"hello happy\"))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let term_match_all: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"hello\"),\n            IndexRecordOption::Basic,\n        ));\n        let term_match_some: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"happy\"),\n            IndexRecordOption::Basic,\n        ));\n        let term_match_none: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"tax\"),\n            IndexRecordOption::Basic,\n        ));\n        {\n            let query = BooleanQuery::from(vec![\n                (Occur::Must, term_match_all.box_clone()),\n                (Occur::Must, term_match_some.box_clone()),\n            ]);\n            let weight = query.weight(EnableScoring::disabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0f32)?;\n            assert!(scorer.is::<TermScorer>());\n        }\n        {\n            let query = BooleanQuery::from(vec![\n                (Occur::Must, term_match_all.box_clone()),\n                (Occur::Must, term_match_some.box_clone()),\n                (Occur::Must, term_match_none.box_clone()),\n            ]);\n            let weight = query.weight(EnableScoring::disabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0f32)?;\n            assert!(scorer.is::<EmptyScorer>());\n        }\n        {\n            let query = BooleanQuery::from(vec![\n                (Occur::Should, term_match_all.box_clone()),\n                (Occur::Should, term_match_none.box_clone()),\n            ]);\n            let weight = query.weight(EnableScoring::disabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0f32)?;\n            assert!(scorer.is::<AllScorer>());\n        }\n        {\n            let query = BooleanQuery::from(vec![\n                (Occur::Should, term_match_some.box_clone()),\n                (Occur::Should, term_match_none.box_clone()),\n            ]);\n            let weight = query.weight(EnableScoring::disabled_from_searcher(&searcher))?;\n            let scorer = weight.scorer(searcher.segment_reader(0u32), 1.0f32)?;\n            assert!(scorer.is::<TermScorer>());\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_min_should_match_with_all_query() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let num_field =\n            schema_builder.add_i64_field(\"num\", NumericOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        index_writer.add_document(doc!(text_field => \"apple\", num_field => 10i64))?;\n        index_writer.add_document(doc!(text_field => \"banana\", num_field => 20i64))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n\n        let effective_all_match_query: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(num_field, 0)),\n            Bound::Unbounded,\n        ));\n        let term_query: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"apple\"),\n            IndexRecordOption::Basic,\n        ));\n\n        // in some previous version, we would remove the 2 all_match, but then say we need *4*\n        // matches out of the 3 term queries, which matches nothing.\n        let mut bool_query = BooleanQuery::new(vec![\n            (Occur::Should, effective_all_match_query.box_clone()),\n            (Occur::Should, effective_all_match_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        bool_query.set_minimum_number_should_match(4);\n        let count = searcher.search(&bool_query, &Count)?;\n        assert_eq!(count, 1);\n\n        Ok(())\n    }\n\n    // =========================================================================\n    // AllScorer Preservation Regression Tests\n    // =========================================================================\n    //\n    // These tests verify the fix for a bug where AllScorer instances (produced by\n    // queries matching all documents, such as range queries covering all values)\n    // were incorrectly removed from Boolean query processing, causing documents\n    // to be unexpectedly excluded from results.\n    //\n    // The bug manifested in several scenarios:\n    // 1. SHOULD + SHOULD where one clause is AllScorer\n    // 2. MUST (AllScorer) + SHOULD\n    // 3. Range queries in Boolean clauses when all documents match the range\n\n    /// Regression test: SHOULD clause with AllScorer combined with other SHOULD clauses.\n    ///\n    /// When a SHOULD clause produces an AllScorer (e.g., from a range query matching\n    /// all documents), the Boolean query should still match all documents.\n    ///\n    /// Bug before fix: AllScorer was removed during optimization, leaving only the\n    /// other SHOULD clauses, which incorrectly excluded documents.\n    #[test]\n    pub fn test_should_with_all_scorer_regression() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let num_field =\n            schema_builder.add_i64_field(\"num\", NumericOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // All docs have num > 0, so range query will return AllScorer\n        index_writer.add_document(doc!(text_field => \"hello\", num_field => 10i64))?;\n        index_writer.add_document(doc!(text_field => \"world\", num_field => 20i64))?;\n        index_writer.add_document(doc!(text_field => \"hello world\", num_field => 30i64))?;\n        index_writer.add_document(doc!(text_field => \"foo\", num_field => 40i64))?;\n        index_writer.add_document(doc!(text_field => \"bar\", num_field => 50i64))?;\n        index_writer.add_document(doc!(text_field => \"baz\", num_field => 60i64))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n\n        // Range query matching all docs (returns AllScorer)\n        let all_match_query: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(num_field, 0)),\n            Bound::Unbounded,\n        ));\n        let term_query: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"hello\"),\n            IndexRecordOption::Basic,\n        ));\n\n        // Verify range matches all 6 docs\n        assert_eq!(searcher.search(all_match_query.as_ref(), &Count)?, 6);\n\n        // RangeQuery(all) OR TermQuery should match all 6 docs\n        let bool_query = BooleanQuery::new(vec![\n            (Occur::Should, all_match_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        let count = searcher.search(&bool_query, &Count)?;\n        assert_eq!(count, 6, \"SHOULD with AllScorer should match all docs\");\n\n        // Order should not matter\n        let bool_query_reversed = BooleanQuery::new(vec![\n            (Occur::Should, term_query.box_clone()),\n            (Occur::Should, all_match_query.box_clone()),\n        ]);\n        let count_reversed = searcher.search(&bool_query_reversed, &Count)?;\n        assert_eq!(\n            count_reversed, 6,\n            \"Order of SHOULD clauses should not matter\"\n        );\n\n        Ok(())\n    }\n\n    /// Regression test: MUST clause with AllScorer combined with SHOULD clause.\n    ///\n    /// When MUST contains an AllScorer, all documents satisfy the MUST constraint.\n    /// The SHOULD clause should only affect scoring, not filtering.\n    ///\n    /// Bug before fix: AllScorer was removed, leaving an empty must_scorers vector.\n    /// intersect_scorers([]) incorrectly returned EmptyScorer, matching 0 documents.\n    #[test]\n    pub fn test_must_all_with_should_regression() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let num_field =\n            schema_builder.add_i64_field(\"num\", NumericOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // All docs have num > 0, so range query will return AllScorer\n        index_writer.add_document(doc!(text_field => \"apple\", num_field => 10i64))?;\n        index_writer.add_document(doc!(text_field => \"banana\", num_field => 20i64))?;\n        index_writer.add_document(doc!(text_field => \"cherry\", num_field => 30i64))?;\n        index_writer.add_document(doc!(text_field => \"date\", num_field => 40i64))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n\n        // Range query matching all docs (returns AllScorer)\n        let all_match_query: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(num_field, 0)),\n            Bound::Unbounded,\n        ));\n        let term_query: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"apple\"),\n            IndexRecordOption::Basic,\n        ));\n\n        // Verify range matches all 4 docs\n        assert_eq!(searcher.search(all_match_query.as_ref(), &Count)?, 4);\n\n        // MUST(range matching all) AND SHOULD(term) should match all 4 docs\n        let bool_query = BooleanQuery::new(vec![\n            (Occur::Must, all_match_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        let count = searcher.search(&bool_query, &Count)?;\n        assert_eq!(count, 4, \"MUST AllScorer + SHOULD should match all docs\");\n\n        Ok(())\n    }\n\n    /// Regression test: Range queries in Boolean clauses when all documents match.\n    ///\n    /// Range queries can return AllScorer as an optimization when all indexed values\n    /// fall within the range. This test ensures such queries work correctly in\n    /// Boolean combinations.\n    ///\n    /// This is the most common real-world manifestation of the bug, occurring in\n    /// queries like: (age > 50 OR name = 'Alice') AND status = 'active'\n    /// when all documents have age > 50.\n    #[test]\n    pub fn test_range_query_all_match_in_boolean() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name_field = schema_builder.add_text_field(\"name\", TEXT);\n        let age_field =\n            schema_builder.add_i64_field(\"age\", NumericOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // All documents have age > 50, so range query will return AllScorer\n        index_writer.add_document(doc!(name_field => \"alice\", age_field => 55_i64))?;\n        index_writer.add_document(doc!(name_field => \"bob\", age_field => 60_i64))?;\n        index_writer.add_document(doc!(name_field => \"charlie\", age_field => 70_i64))?;\n        index_writer.add_document(doc!(name_field => \"diana\", age_field => 80_i64))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n\n        let range_query: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(age_field, 50)),\n            Bound::Unbounded,\n        ));\n        let term_query: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(name_field, \"alice\"),\n            IndexRecordOption::Basic,\n        ));\n\n        // Verify preconditions\n        assert_eq!(searcher.search(range_query.as_ref(), &Count)?, 4);\n        assert_eq!(searcher.search(term_query.as_ref(), &Count)?, 1);\n\n        // SHOULD(range) OR SHOULD(term): range matches all, so result is 4\n        let should_query = BooleanQuery::new(vec![\n            (Occur::Should, range_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        assert_eq!(\n            searcher.search(&should_query, &Count)?,\n            4,\n            \"SHOULD range OR term should match all\"\n        );\n\n        // MUST(range) AND SHOULD(term): range matches all, term is optional\n        let must_should_query = BooleanQuery::new(vec![\n            (Occur::Must, range_query.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        assert_eq!(\n            searcher.search(&must_should_query, &Count)?,\n            4,\n            \"MUST range + SHOULD term should match all\"\n        );\n\n        Ok(())\n    }\n\n    /// Test multiple AllScorer instances in different clause types.\n    ///\n    /// Verifies correct behavior when AllScorers appear in multiple positions.\n    #[test]\n    pub fn test_multiple_all_scorers() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let num_field =\n            schema_builder.add_i64_field(\"num\", NumericOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // All docs have num > 0, so range queries will return AllScorer\n        index_writer.add_document(doc!(text_field => \"doc1\", num_field => 10i64))?;\n        index_writer.add_document(doc!(text_field => \"doc2\", num_field => 20i64))?;\n        index_writer.add_document(doc!(text_field => \"doc3\", num_field => 30i64))?;\n        index_writer.commit()?;\n\n        let searcher = index.reader()?.searcher();\n\n        // Two different range queries that both match all docs (return AllScorer)\n        let all_query1: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(num_field, 0)),\n            Bound::Unbounded,\n        ));\n        let all_query2: Box<dyn Query> = Box::new(RangeQuery::new(\n            Bound::Excluded(Term::from_field_i64(num_field, 5)),\n            Bound::Unbounded,\n        ));\n        let term_query: Box<dyn Query> = Box::new(TermQuery::new(\n            Term::from_field_text(text_field, \"doc1\"),\n            IndexRecordOption::Basic,\n        ));\n\n        // Multiple AllScorers in SHOULD\n        let multi_all_should = BooleanQuery::new(vec![\n            (Occur::Should, all_query1.box_clone()),\n            (Occur::Should, all_query2.box_clone()),\n            (Occur::Should, term_query.box_clone()),\n        ]);\n        assert_eq!(\n            searcher.search(&multi_all_should, &Count)?,\n            3,\n            \"Multiple AllScorers in SHOULD\"\n        );\n\n        // AllScorer in both MUST and SHOULD\n        let all_must_and_should = BooleanQuery::new(vec![\n            (Occur::Must, all_query1.box_clone()),\n            (Occur::Should, all_query2.box_clone()),\n        ]);\n        assert_eq!(\n            searcher.search(&all_must_and_should, &Count)?,\n            3,\n            \"AllScorer in both MUST and SHOULD\"\n        );\n\n        Ok(())\n    }\n}\n\n/// A proptest which generates arbitrary permutations of a simple boolean AST, and then matches\n/// the result against an index which contains all permutations of documents with N fields.\n#[cfg(test)]\nmod proptest_boolean_query {\n    use std::collections::{BTreeMap, HashSet};\n    use std::ops::{Bound, Range};\n\n    use proptest::collection::vec;\n    use proptest::prelude::*;\n\n    use crate::collector::DocSetCollector;\n    use crate::query::{AllQuery, BooleanQuery, Occur, Query, RangeQuery, TermQuery};\n    use crate::schema::{Field, NumericOptions, OwnedValue, Schema, TEXT};\n    use crate::{DocId, Index, Term};\n\n    #[derive(Debug, Clone)]\n    enum BooleanQueryAST {\n        /// Matches all documents via AllQuery (wraps AllScorer in BoostScorer)\n        All,\n        /// Matches all documents via RangeQuery (returns bare AllScorer)\n        /// This is the actual trigger for the AllScorer preservation bug\n        RangeAll,\n        /// Matches documents where the field has value \"true\"\n        Leaf {\n            field_idx: usize,\n        },\n        Union(Vec<BooleanQueryAST>),\n        Intersection(Vec<BooleanQueryAST>),\n    }\n\n    impl BooleanQueryAST {\n        fn matches(&self, doc_id: DocId) -> bool {\n            match self {\n                BooleanQueryAST::All => true,\n                BooleanQueryAST::RangeAll => true,\n                BooleanQueryAST::Leaf { field_idx } => Self::matches_field(doc_id, *field_idx),\n                BooleanQueryAST::Union(children) => {\n                    children.iter().any(|child| child.matches(doc_id))\n                }\n                BooleanQueryAST::Intersection(children) => {\n                    children.iter().all(|child| child.matches(doc_id))\n                }\n            }\n        }\n\n        fn matches_field(doc_id: DocId, field_idx: usize) -> bool {\n            ((doc_id as usize) >> field_idx) & 1 == 1\n        }\n\n        fn to_query(&self, fields: &[Field], range_field: Field) -> Box<dyn Query> {\n            match self {\n                BooleanQueryAST::All => Box::new(AllQuery),\n                BooleanQueryAST::RangeAll => {\n                    // Range query that matches all docs (all have value >= 0)\n                    // This returns bare AllScorer, triggering the bug we fixed\n                    Box::new(RangeQuery::new(\n                        Bound::Included(Term::from_field_i64(range_field, 0)),\n                        Bound::Unbounded,\n                    ))\n                }\n                BooleanQueryAST::Leaf { field_idx } => Box::new(TermQuery::new(\n                    Term::from_field_text(fields[*field_idx], \"true\"),\n                    crate::schema::IndexRecordOption::Basic,\n                )),\n                BooleanQueryAST::Union(children) => {\n                    let sub_queries = children\n                        .iter()\n                        .map(|child| (Occur::Should, child.to_query(fields, range_field)))\n                        .collect();\n                    Box::new(BooleanQuery::new(sub_queries))\n                }\n                BooleanQueryAST::Intersection(children) => {\n                    let sub_queries = children\n                        .iter()\n                        .map(|child| (Occur::Must, child.to_query(fields, range_field)))\n                        .collect();\n                    Box::new(BooleanQuery::new(sub_queries))\n                }\n            }\n        }\n    }\n\n    fn doc_ids(num_docs: usize, num_fields: usize) -> Range<DocId> {\n        let permutations = 1 << num_fields;\n        let copies = (num_docs as f32 / permutations as f32).ceil() as u32;\n        0..(permutations * copies)\n    }\n\n    fn create_index_with_boolean_permutations(\n        num_docs: usize,\n        num_fields: usize,\n    ) -> (Index, Vec<Field>, Field) {\n        let mut schema_builder = Schema::builder();\n        let fields: Vec<Field> = (0..num_fields)\n            .map(|i| schema_builder.add_text_field(&format!(\"field_{}\", i), TEXT))\n            .collect();\n        // Add a numeric field for RangeQuery tests - all docs have value = doc_id\n        let range_field = schema_builder.add_i64_field(\n            \"range_field\",\n            NumericOptions::default().set_fast().set_indexed(),\n        );\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n\n        for doc_id in doc_ids(num_docs, num_fields) {\n            let mut doc: BTreeMap<_, OwnedValue> = BTreeMap::default();\n            for (field_idx, &field) in fields.iter().enumerate() {\n                if (doc_id >> field_idx) & 1 == 1 {\n                    doc.insert(field, \"true\".into());\n                }\n            }\n            // All docs have non-negative values, so RangeQuery(>=0) matches all\n            doc.insert(range_field, (doc_id as i64).into());\n            writer.add_document(doc).unwrap();\n        }\n        writer.commit().unwrap();\n        (index, fields, range_field)\n    }\n\n    fn arb_boolean_query_ast(num_fields: usize) -> impl Strategy<Value = BooleanQueryAST> {\n        // Leaf strategies: term queries, AllQuery, and RangeQuery matching all docs\n        let leaf = prop_oneof![\n            (0..num_fields).prop_map(|field_idx| BooleanQueryAST::Leaf { field_idx }),\n            Just(BooleanQueryAST::All),\n            Just(BooleanQueryAST::RangeAll),\n        ];\n        leaf.prop_recursive(\n            8,   // 8 levels of recursion\n            256, // 256 nodes max\n            10,  // 10 items per collection\n            |inner| {\n                prop_oneof![\n                    vec(inner.clone(), 1..10).prop_map(BooleanQueryAST::Union),\n                    vec(inner, 1..10).prop_map(BooleanQueryAST::Intersection),\n                ]\n            },\n        )\n    }\n\n    #[test]\n    fn proptest_boolean_query() {\n        // In the presence of optimizations around buffering, it can take large numbers of\n        // documents to uncover some issues.\n        let num_fields = 8;\n        let num_docs = 1 << num_fields;\n        let (index, fields, range_field) =\n            create_index_with_boolean_permutations(num_docs, num_fields);\n        let searcher = index.reader().unwrap().searcher();\n        proptest!(|(ast in arb_boolean_query_ast(num_fields))| {\n            let query = ast.to_query(&fields, range_field);\n\n            let mut matching_docs = HashSet::new();\n            for doc_id in doc_ids(num_docs, num_fields) {\n                if ast.matches(doc_id as DocId) {\n                    matching_docs.insert(doc_id as DocId);\n                }\n            }\n\n            let doc_addresses = searcher.search(&*query, &DocSetCollector).unwrap();\n            let result_docs: HashSet<DocId> =\n                doc_addresses.into_iter().map(|doc_address| doc_address.doc_id).collect();\n            prop_assert_eq!(result_docs, matching_docs);\n        });\n    }\n}\n"
  },
  {
    "path": "src/query/boost_query.rs",
    "content": "use std::fmt;\n\nuse crate::docset::{SeekDangerResult, COLLECT_BLOCK_BUFFER_LEN};\nuse crate::fastfield::AliveBitSet;\nuse crate::query::{EnableScoring, Explanation, Query, Scorer, Weight};\nuse crate::{DocId, DocSet, Score, SegmentReader, Term};\n\n/// `BoostQuery` is a wrapper over a query used to boost its score.\n///\n/// The document set matched by the `BoostQuery` is strictly the same as the underlying query.\n/// The score of each document, is the score of the underlying query multiplied by the `boost`\n/// factor.\npub struct BoostQuery {\n    query: Box<dyn Query>,\n    boost: Score,\n}\n\nimpl BoostQuery {\n    /// Builds a boost query.\n    pub fn new(query: Box<dyn Query>, boost: Score) -> BoostQuery {\n        BoostQuery { query, boost }\n    }\n}\n\nimpl Clone for BoostQuery {\n    fn clone(&self) -> Self {\n        BoostQuery {\n            query: self.query.box_clone(),\n            boost: self.boost,\n        }\n    }\n}\n\nimpl fmt::Debug for BoostQuery {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"Boost(query={:?}, boost={})\", self.query, self.boost)\n    }\n}\n\nimpl Query for BoostQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let weight_without_boost = self.query.weight(enable_scoring)?;\n        let boosted_weight = if enable_scoring.is_scoring_enabled() {\n            Box::new(BoostWeight::new(weight_without_boost, self.boost))\n        } else {\n            weight_without_boost\n        };\n        Ok(boosted_weight)\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        self.query.query_terms(visitor)\n    }\n}\n\n/// Weight associated to the BoostQuery.\npub struct BoostWeight {\n    weight: Box<dyn Weight>,\n    boost: Score,\n}\n\nimpl BoostWeight {\n    /// Creates a new BoostWeight.\n    pub fn new(weight: Box<dyn Weight>, boost: Score) -> Self {\n        BoostWeight { weight, boost }\n    }\n}\n\nimpl Weight for BoostWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        self.weight.scorer(reader, boost * self.boost)\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: u32) -> crate::Result<Explanation> {\n        let underlying_explanation = self.weight.explain(reader, doc)?;\n        let score = underlying_explanation.value() * self.boost;\n        let mut explanation =\n            Explanation::new_with_string(format!(\"Boost x{} of ...\", self.boost), score);\n        explanation.add_detail(underlying_explanation);\n        Ok(explanation)\n    }\n\n    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {\n        self.weight.count(reader)\n    }\n}\n\npub(crate) struct BoostScorer<S: Scorer> {\n    underlying: S,\n    boost: Score,\n}\n\nimpl<S: Scorer> BoostScorer<S> {\n    pub fn new(underlying: S, boost: Score) -> BoostScorer<S> {\n        BoostScorer { underlying, boost }\n    }\n}\n\nimpl<S: Scorer> DocSet for BoostScorer<S> {\n    fn advance(&mut self) -> DocId {\n        self.underlying.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.underlying.seek(target)\n    }\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        self.underlying.seek_danger(target)\n    }\n\n    fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {\n        self.underlying.fill_buffer(buffer)\n    }\n\n    fn doc(&self) -> u32 {\n        self.underlying.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.underlying.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        self.underlying.cost()\n    }\n\n    fn count(&mut self, alive_bitset: &AliveBitSet) -> u32 {\n        self.underlying.count(alive_bitset)\n    }\n\n    fn count_including_deleted(&mut self) -> u32 {\n        self.underlying.count_including_deleted()\n    }\n}\n\nimpl<S: Scorer> Scorer for BoostScorer<S> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.underlying.score() * self.boost\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::BoostQuery;\n    use crate::query::{AllQuery, Query};\n    use crate::schema::Schema;\n    use crate::{DocAddress, Index, IndexWriter, TantivyDocument};\n\n    #[test]\n    fn test_boost_query_explain() -> crate::Result<()> {\n        let schema = Schema::builder().build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(TantivyDocument::new())?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query = BoostQuery::new(Box::new(AllQuery), 0.2);\n        let explanation = query.explain(&searcher, DocAddress::new(0, 0u32)).unwrap();\n        assert_eq!(\n            explanation.to_pretty_json(),\n            \"{\\n  \\\"value\\\": 0.2,\\n  \\\"description\\\": \\\"Boost x0.2 of ...\\\",\\n  \\\"details\\\": [\\n    {\\n      \\\"value\\\": 1.0,\\n      \\\"description\\\": \\\"AllQuery\\\"\\n    }\\n  ]\\n}\"\n        );\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/const_score_query.rs",
    "content": "use std::fmt;\n\nuse crate::docset::COLLECT_BLOCK_BUFFER_LEN;\nuse crate::query::{EnableScoring, Explanation, Query, Scorer, Weight};\nuse crate::{DocId, DocSet, Score, SegmentReader, TantivyError, Term};\n\n/// `ConstScoreQuery` is a wrapper over a query to provide a constant score.\n/// It can avoid unnecessary score computation on the wrapped query.\n///\n/// The document set matched by the `ConstScoreQuery` is strictly the same as the underlying query.\n/// The configured score is used for each document.\npub struct ConstScoreQuery {\n    query: Box<dyn Query>,\n    score: Score,\n}\n\nimpl ConstScoreQuery {\n    /// Builds a const score query.\n    pub fn new(query: Box<dyn Query>, score: Score) -> ConstScoreQuery {\n        ConstScoreQuery { query, score }\n    }\n}\n\nimpl Clone for ConstScoreQuery {\n    fn clone(&self) -> Self {\n        ConstScoreQuery {\n            query: self.query.box_clone(),\n            score: self.score,\n        }\n    }\n}\n\nimpl fmt::Debug for ConstScoreQuery {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"Const(score={}, query={:?})\", self.score, self.query)\n    }\n}\n\nimpl Query for ConstScoreQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let inner_weight = self.query.weight(enable_scoring)?;\n        Ok(if enable_scoring.is_scoring_enabled() {\n            Box::new(ConstWeight::new(inner_weight, self.score))\n        } else {\n            inner_weight\n        })\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        self.query.query_terms(visitor);\n    }\n}\n\nstruct ConstWeight {\n    weight: Box<dyn Weight>,\n    score: Score,\n}\n\nimpl ConstWeight {\n    pub fn new(weight: Box<dyn Weight>, score: Score) -> Self {\n        ConstWeight { weight, score }\n    }\n}\n\nimpl Weight for ConstWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let inner_scorer = self.weight.scorer(reader, boost)?;\n        Ok(Box::new(ConstScorer::new(inner_scorer, boost * self.score)))\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: u32) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) != doc {\n            return Err(TantivyError::InvalidArgument(format!(\n                \"Document #({doc}) does not match\"\n            )));\n        }\n        let mut explanation = Explanation::new(\"Const\", self.score);\n        let underlying_explanation = self.weight.explain(reader, doc)?;\n        explanation.add_detail(underlying_explanation);\n        Ok(explanation)\n    }\n\n    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {\n        self.weight.count(reader)\n    }\n}\n\n/// Wraps a `DocSet` and simply returns a constant `Scorer`.\n/// The `ConstScorer` is useful if you have a `DocSet` where\n/// you needed a scorer.\n///\n/// The `ConstScorer`'s constant score can be set\n/// by calling `.set_score(...)`.\npub struct ConstScorer<TDocSet: DocSet> {\n    docset: TDocSet,\n    score: Score,\n}\n\nimpl<TDocSet: DocSet> ConstScorer<TDocSet> {\n    /// Creates a new `ConstScorer`.\n    pub fn new(docset: TDocSet, score: Score) -> ConstScorer<TDocSet> {\n        ConstScorer { docset, score }\n    }\n}\n\nimpl<TDocSet: DocSet> From<TDocSet> for ConstScorer<TDocSet> {\n    fn from(docset: TDocSet) -> Self {\n        ConstScorer::new(docset, 1.0)\n    }\n}\n\nimpl<TDocSet: DocSet> DocSet for ConstScorer<TDocSet> {\n    fn advance(&mut self) -> DocId {\n        self.docset.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.docset.seek(target)\n    }\n\n    fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {\n        self.docset.fill_buffer(buffer)\n    }\n\n    fn doc(&self) -> DocId {\n        self.docset.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.docset.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        self.docset.cost()\n    }\n}\n\nimpl<TDocSet: DocSet + 'static> Scorer for ConstScorer<TDocSet> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.score\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::ConstScoreQuery;\n    use crate::query::{AllQuery, Query};\n    use crate::schema::Schema;\n    use crate::{DocAddress, Index, IndexWriter, TantivyDocument};\n\n    #[test]\n    fn test_const_score_query_explain() -> crate::Result<()> {\n        let schema = Schema::builder().build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(TantivyDocument::new())?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query = ConstScoreQuery::new(Box::new(AllQuery), 0.42);\n        let explanation = query.explain(&searcher, DocAddress::new(0, 0u32)).unwrap();\n        assert_eq!(\n            explanation.to_pretty_json(),\n            r#\"{\n  \"value\": 0.42,\n  \"description\": \"Const\",\n  \"details\": [\n    {\n      \"value\": 1.0,\n      \"description\": \"AllQuery\"\n    }\n  ]\n}\"#\n        );\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/disjunction.rs",
    "content": "use std::cmp::Ordering;\nuse std::collections::BinaryHeap;\n\nuse crate::docset::SeekDangerResult;\nuse crate::query::score_combiner::DoNothingCombiner;\nuse crate::query::{ScoreCombiner, Scorer};\nuse crate::{DocId, DocSet, Score, TERMINATED};\n\n/// `Disjunction` is responsible for merging `DocSet` from multiple\n/// source. Specifically, It takes the union of two or more `DocSet`s\n/// then filtering out elements that appear fewer times than a\n/// specified threshold.\npub struct Disjunction<TScorer, TScoreCombiner = DoNothingCombiner> {\n    chains: BinaryHeap<ScorerWrapper<TScorer>>,\n    minimum_matches_required: usize,\n    score_combiner: TScoreCombiner,\n\n    current_doc: DocId,\n    current_score: Score,\n}\n\n/// A wrapper around a `Scorer` that caches the current `doc_id` and implements the `DocSet` trait.\n/// Also, the `Ord` trait and it's family are implemented reversely. So that we can combine\n/// `std::BinaryHeap<ScorerWrapper<T>>` to gain a min-heap with current doc id as key.\nstruct ScorerWrapper<T> {\n    scorer: T,\n    current_doc: DocId,\n}\n\nimpl<T: Scorer> ScorerWrapper<T> {\n    fn new(scorer: T) -> Self {\n        let current_doc = scorer.doc();\n        Self {\n            scorer,\n            current_doc,\n        }\n    }\n}\n\nimpl<T: Scorer> PartialEq for ScorerWrapper<T> {\n    fn eq(&self, other: &Self) -> bool {\n        self.doc() == other.doc()\n    }\n}\n\nimpl<T: Scorer> Eq for ScorerWrapper<T> {}\n\nimpl<T: Scorer> PartialOrd for ScorerWrapper<T> {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl<T: Scorer> Ord for ScorerWrapper<T> {\n    fn cmp(&self, other: &Self) -> Ordering {\n        self.doc().cmp(&other.doc()).reverse()\n    }\n}\n\nimpl<T: Scorer> DocSet for ScorerWrapper<T> {\n    fn advance(&mut self) -> DocId {\n        let doc_id = self.scorer.advance();\n        self.current_doc = doc_id;\n        doc_id\n    }\n    fn seek(&mut self, target: DocId) -> DocId {\n        let doc_id = self.scorer.seek(target);\n        self.current_doc = doc_id;\n        doc_id\n    }\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        let result = self.scorer.seek_danger(target);\n        if result == SeekDangerResult::Found {\n            self.current_doc = target;\n        }\n        result\n    }\n\n    fn doc(&self) -> DocId {\n        self.current_doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.scorer.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        self.scorer.cost()\n    }\n}\n\nimpl<TScorer: Scorer, TScoreCombiner: ScoreCombiner> Disjunction<TScorer, TScoreCombiner> {\n    pub fn new<T: IntoIterator<Item = TScorer>>(\n        docsets: T,\n        score_combiner: TScoreCombiner,\n        minimum_matches_required: usize,\n    ) -> Self {\n        debug_assert!(\n            minimum_matches_required > 1,\n            \"union scorer works better if just one matches required\"\n        );\n        let chains = docsets\n            .into_iter()\n            .map(|doc| ScorerWrapper::new(doc))\n            .collect();\n        let mut disjunction = Self {\n            chains,\n            score_combiner,\n            current_doc: TERMINATED,\n            minimum_matches_required,\n            current_score: 0.0,\n        };\n        if minimum_matches_required > disjunction.chains.len() {\n            return disjunction;\n        }\n        disjunction.advance();\n        disjunction\n    }\n}\n\nimpl<TScorer: Scorer, TScoreCombiner: ScoreCombiner> DocSet\n    for Disjunction<TScorer, TScoreCombiner>\n{\n    fn advance(&mut self) -> DocId {\n        let mut current_num_matches = 0;\n        while let Some(mut candidate) = self.chains.pop() {\n            let next = candidate.doc();\n            if next != TERMINATED {\n                // Peek next doc.\n                if self.current_doc != next {\n                    if current_num_matches >= self.minimum_matches_required {\n                        self.chains.push(candidate);\n                        self.current_score = self.score_combiner.score();\n                        return self.current_doc;\n                    }\n                    // Reset current_num_matches and scores.\n                    current_num_matches = 0;\n                    self.current_doc = next;\n                    self.score_combiner.clear();\n                }\n                current_num_matches += 1;\n                self.score_combiner.update(&mut candidate.scorer);\n                candidate.advance();\n                self.chains.push(candidate);\n            }\n        }\n        if current_num_matches < self.minimum_matches_required {\n            self.current_doc = TERMINATED;\n        }\n        self.current_score = self.score_combiner.score();\n        self.current_doc\n    }\n\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.current_doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.chains\n            .iter()\n            .map(|docset| docset.size_hint())\n            .max()\n            .unwrap_or(0u32)\n    }\n\n    fn cost(&self) -> u64 {\n        self.chains\n            .iter()\n            .map(|docset| docset.cost())\n            .max()\n            .unwrap_or(0u64)\n    }\n}\n\nimpl<TScorer: Scorer, TScoreCombiner: ScoreCombiner> Scorer\n    for Disjunction<TScorer, TScoreCombiner>\n{\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.current_score\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeMap;\n\n    use super::Disjunction;\n    use crate::query::score_combiner::DoNothingCombiner;\n    use crate::query::{ConstScorer, Scorer, SumCombiner, VecDocSet};\n    use crate::{DocId, DocSet, Score, TERMINATED};\n\n    fn conjunct<T: Ord + Copy>(arrays: &[Vec<T>], pass_line: usize) -> Vec<T> {\n        let mut counts = BTreeMap::new();\n        for array in arrays {\n            for &element in array {\n                *counts.entry(element).or_insert(0) += 1;\n            }\n        }\n        counts\n            .iter()\n            .filter_map(|(&element, &count)| {\n                if count >= pass_line {\n                    Some(element)\n                } else {\n                    None\n                }\n            })\n            .collect()\n    }\n\n    fn aux_test_conjunction(vals: Vec<Vec<u32>>, min_match: usize) {\n        let mut union_expected = VecDocSet::from(conjunct(&vals, min_match));\n        let make_scorer = || {\n            Disjunction::new(\n                vals.iter()\n                    .cloned()\n                    .map(VecDocSet::from)\n                    .map(|d| ConstScorer::new(d, 1.0)),\n                DoNothingCombiner,\n                min_match,\n            )\n        };\n        let mut scorer: Disjunction<_, DoNothingCombiner> = make_scorer();\n        let mut count = 0;\n        while scorer.doc() != TERMINATED {\n            assert_eq!(union_expected.doc(), scorer.doc());\n            assert_eq!(union_expected.advance(), scorer.advance());\n            count += 1;\n        }\n        assert_eq!(union_expected.advance(), TERMINATED);\n        assert_eq!(count, make_scorer().count_including_deleted());\n    }\n\n    #[should_panic]\n    #[test]\n    fn test_arg_check1() {\n        aux_test_conjunction(vec![], 0);\n    }\n\n    #[should_panic]\n    #[test]\n    fn test_arg_check2() {\n        aux_test_conjunction(vec![], 1);\n    }\n\n    #[test]\n    fn test_corner_case() {\n        aux_test_conjunction(vec![], 2);\n        aux_test_conjunction(vec![vec![]; 1000], 2);\n        aux_test_conjunction(vec![vec![]; 100], usize::MAX);\n        aux_test_conjunction(vec![vec![0xC0FFEE]; 10000], usize::MAX);\n        aux_test_conjunction((1..10000u32).map(|i| vec![i]).collect::<Vec<_>>(), 2);\n    }\n\n    #[test]\n    fn test_conjunction() {\n        aux_test_conjunction(\n            vec![\n                vec![1, 3333, 100000000u32],\n                vec![1, 2, 100000000u32],\n                vec![1, 2, 100000000u32],\n            ],\n            2,\n        );\n        aux_test_conjunction(\n            vec![vec![8], vec![3, 4, 0xC0FFEEu32], vec![1, 2, 100000000u32]],\n            2,\n        );\n        aux_test_conjunction(\n            vec![\n                vec![1, 3333, 100000000u32],\n                vec![1, 2, 100000000u32],\n                vec![1, 2, 100000000u32],\n            ],\n            3,\n        )\n    }\n\n    // This dummy scorer does nothing but yield doc id increasingly.\n    // with constant score 1.0\n    #[derive(Clone)]\n    struct DummyScorer {\n        cursor: usize,\n        foo: Vec<(DocId, f32)>,\n    }\n\n    impl DummyScorer {\n        fn new(doc_score: Vec<(DocId, f32)>) -> Self {\n            Self {\n                cursor: 0,\n                foo: doc_score,\n            }\n        }\n    }\n\n    impl DocSet for DummyScorer {\n        fn advance(&mut self) -> DocId {\n            self.cursor += 1;\n            self.doc()\n        }\n\n        fn doc(&self) -> DocId {\n            self.foo.get(self.cursor).map(|x| x.0).unwrap_or(TERMINATED)\n        }\n\n        fn size_hint(&self) -> u32 {\n            self.foo.len() as u32\n        }\n    }\n\n    impl Scorer for DummyScorer {\n        #[inline]\n        fn score(&mut self) -> Score {\n            self.foo.get(self.cursor).map(|x| x.1).unwrap_or(0.0)\n        }\n    }\n\n    #[test]\n    fn test_score_calculate() {\n        let mut scorer = Disjunction::new(\n            vec![\n                DummyScorer::new(vec![(1, 1f32), (2, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (3, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (4, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (2, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (2, 1f32)]),\n            ],\n            SumCombiner::default(),\n            3,\n        );\n        assert_eq!(scorer.score(), 5.0);\n        assert_eq!(scorer.advance(), 2);\n        assert_eq!(scorer.score(), 3.0);\n    }\n\n    #[test]\n    fn test_score_calculate_corner_case() {\n        let mut scorer = Disjunction::new(\n            vec![\n                DummyScorer::new(vec![(1, 1f32), (2, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (3, 1f32)]),\n                DummyScorer::new(vec![(1, 1f32), (3, 1f32)]),\n            ],\n            SumCombiner::default(),\n            2,\n        );\n        assert_eq!(scorer.doc(), 1);\n        assert_eq!(scorer.score(), 3.0);\n        assert_eq!(scorer.advance(), 3);\n        assert_eq!(scorer.score(), 2.0);\n    }\n}\n"
  },
  {
    "path": "src/query/disjunction_max_query.rs",
    "content": "use crate::query::{BooleanWeight, DisjunctionMaxCombiner, EnableScoring, Occur, Query, Weight};\nuse crate::{Score, Term};\n\n/// The disjunction max query returns documents matching one or more wrapped queries,\n/// called query clauses or clauses.\n///\n/// If a returned document matches multiple query clauses,\n/// the `DisjunctionMaxQuery` assigns the document the highest relevance score from any matching\n/// clause, plus a tie breaking increment for any additional matching subqueries.\n///\n/// ```rust\n/// use tantivy::collector::TopDocs;\n/// use tantivy::doc;\n/// use tantivy::query::{DisjunctionMaxQuery, Query, QueryClone, TermQuery};\n/// use tantivy::schema::{IndexRecordOption, Schema, TEXT};\n/// use tantivy::Term;\n/// use tantivy::Index;\n/// use tantivy::IndexWriter;\n///\n/// fn main() -> tantivy::Result<()> {\n///    let mut schema_builder = Schema::builder();\n///    let title = schema_builder.add_text_field(\"title\", TEXT);\n///    let body = schema_builder.add_text_field(\"body\", TEXT);\n///    let schema = schema_builder.build();\n///    let index = Index::create_in_ram(schema);\n///    {\n///        let mut index_writer: IndexWriter = index.writer(15_000_000)?;\n///        index_writer.add_document(doc!(\n///            title => \"The Name of Girl\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"The Diary of Muadib\",\n///        ))?;\n///        index_writer.add_document(doc!(\n///            title => \"The Diary of Girl\",\n///        ))?;\n///        index_writer.commit()?;\n///    }\n///\n///    let reader = index.reader()?;\n///    let searcher = reader.searcher();\n///\n///    // Make TermQuery's for \"girl\" and \"diary\" in the title\n///    let girl_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(title, \"girl\"),\n///        IndexRecordOption::Basic,\n///    ));\n///    let diary_term_query: Box<dyn Query> = Box::new(TermQuery::new(\n///        Term::from_field_text(title, \"diary\"),\n///        IndexRecordOption::Basic,\n///    ));\n///\n///    // TermQuery \"diary\" and \"girl\" should be present and only one should be accounted in score\n///    let queries1 = vec![diary_term_query.box_clone(), girl_term_query.box_clone()];\n///    let diary_and_girl = DisjunctionMaxQuery::new(queries1);\n///    let documents = searcher.search(&diary_and_girl, &TopDocs::with_limit(3).order_by_score())?;\n///    assert_eq!(documents[0].0, documents[1].0);\n///    assert_eq!(documents[1].0, documents[2].0);\n///\n///    // TermQuery \"diary\" and \"girl\" should be present\n///    // and one should be accounted with multiplier 0.7\n///    let queries2 = vec![diary_term_query.box_clone(), girl_term_query.box_clone()];\n///    let tie_breaker = 0.7;\n///    let diary_and_girl_with_tie_breaker = DisjunctionMaxQuery::with_tie_breaker(queries2, tie_breaker);\n///    let documents = searcher.search(&diary_and_girl_with_tie_breaker, &TopDocs::with_limit(3).order_by_score())?;\n///    assert_eq!(documents[1].0, documents[2].0);\n///    // For this test all terms brings the same score. So we can do easy math and assume that\n///    // `DisjunctionMaxQuery` with tie breakers score should be equal\n///    // to term1 score + `tie_breaker` * term2 score or (1.0 + tie_breaker) * term score\n///    assert!(f32::abs(documents[0].0 - documents[1].0 * (1.0 + tie_breaker)) < 0.001);\n///    Ok(())\n/// }\n/// ```\n#[derive(Debug)]\npub struct DisjunctionMaxQuery {\n    disjuncts: Vec<Box<dyn Query>>,\n    tie_breaker: Score,\n}\n\nimpl Clone for DisjunctionMaxQuery {\n    fn clone(&self) -> Self {\n        DisjunctionMaxQuery::with_tie_breaker(\n            self.disjuncts\n                .iter()\n                .map(|disjunct| disjunct.box_clone())\n                .collect::<Vec<_>>(),\n            self.tie_breaker,\n        )\n    }\n}\n\nimpl Query for DisjunctionMaxQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let disjuncts = self\n            .disjuncts\n            .iter()\n            .map(|disjunct| Ok((Occur::Should, disjunct.weight(enable_scoring)?)))\n            .collect::<crate::Result<_>>()?;\n        let tie_breaker = self.tie_breaker;\n        Ok(Box::new(BooleanWeight::new(\n            disjuncts,\n            enable_scoring.is_scoring_enabled(),\n            Box::new(move || DisjunctionMaxCombiner::with_tie_breaker(tie_breaker)),\n        )))\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        for disjunct in &self.disjuncts {\n            disjunct.query_terms(visitor);\n        }\n    }\n}\n\nimpl DisjunctionMaxQuery {\n    /// Creates a new `DisjunctionMaxQuery` with tie breaker.\n    pub fn with_tie_breaker(\n        disjuncts: Vec<Box<dyn Query>>,\n        tie_breaker: Score,\n    ) -> DisjunctionMaxQuery {\n        DisjunctionMaxQuery {\n            disjuncts,\n            tie_breaker,\n        }\n    }\n\n    /// Creates a new `DisjunctionMaxQuery` with no tie breaker.\n    pub fn new(disjuncts: Vec<Box<dyn Query>>) -> DisjunctionMaxQuery {\n        DisjunctionMaxQuery::with_tie_breaker(disjuncts, 0.0)\n    }\n}\n"
  },
  {
    "path": "src/query/empty_query.rs",
    "content": "use super::Scorer;\nuse crate::docset::TERMINATED;\nuse crate::index::SegmentReader;\nuse crate::query::explanation::does_not_match;\nuse crate::query::{EnableScoring, Explanation, Query, Weight};\nuse crate::{DocId, DocSet, Score, Searcher};\n\n/// `EmptyQuery` is a dummy `Query` in which no document matches.\n///\n/// It is useful for tests and handling edge cases.\n#[derive(Clone, Debug)]\npub struct EmptyQuery;\n\nimpl Query for EmptyQuery {\n    fn weight(&self, _enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(EmptyWeight))\n    }\n\n    fn count(&self, _searcher: &Searcher) -> crate::Result<usize> {\n        Ok(0)\n    }\n}\n\n/// `EmptyWeight` is a dummy `Weight` in which no document matches.\n///\n/// It is useful for tests and handling edge cases.\npub struct EmptyWeight;\nimpl Weight for EmptyWeight {\n    fn scorer(&self, _reader: &SegmentReader, _boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        Ok(Box::new(EmptyScorer))\n    }\n\n    fn explain(&self, _reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        Err(does_not_match(doc))\n    }\n}\n\n/// `EmptyScorer` is a dummy `Scorer` in which no document matches.\n///\n/// It is useful for tests and handling edge cases.\npub struct EmptyScorer;\n\nimpl DocSet for EmptyScorer {\n    fn advance(&mut self) -> DocId {\n        TERMINATED\n    }\n\n    fn doc(&self) -> DocId {\n        TERMINATED\n    }\n\n    fn size_hint(&self) -> u32 {\n        0\n    }\n}\n\nimpl Scorer for EmptyScorer {\n    #[inline]\n    fn score(&mut self) -> Score {\n        0.0\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::docset::TERMINATED;\n    use crate::query::EmptyScorer;\n    use crate::DocSet;\n\n    #[test]\n    fn test_empty_scorer() {\n        let mut empty_scorer = EmptyScorer;\n        assert_eq!(empty_scorer.doc(), TERMINATED);\n        assert_eq!(empty_scorer.advance(), TERMINATED);\n        assert_eq!(empty_scorer.doc(), TERMINATED);\n    }\n}\n"
  },
  {
    "path": "src/query/exclude.rs",
    "content": "use crate::docset::{DocSet, SeekDangerResult, TERMINATED};\nuse crate::query::Scorer;\nuse crate::{DocId, Score};\n\n/// An exclusion set is a set of documents\n/// that should be excluded from a given DocSet.\n///\n/// It can be a single DocSet, or a Vec of DocSets.\npub trait ExclusionSet: Send {\n    /// Returns `true` if the given `doc` is in the exclusion set.\n    fn contains(&mut self, doc: DocId) -> bool;\n}\n\nimpl<TDocSet: DocSet> ExclusionSet for TDocSet {\n    #[inline]\n    fn contains(&mut self, doc: DocId) -> bool {\n        self.seek_danger(doc) == SeekDangerResult::Found\n    }\n}\n\nimpl<TDocSet: DocSet> ExclusionSet for Vec<TDocSet> {\n    #[inline]\n    fn contains(&mut self, doc: DocId) -> bool {\n        for docset in self.iter_mut() {\n            if docset.seek_danger(doc) == SeekDangerResult::Found {\n                return true;\n            }\n        }\n        false\n    }\n}\n\n/// Filters a given `DocSet` by removing the docs from an exclusion set.\n///\n/// The excluding docsets have no impact on scoring.\npub struct Exclude<TDocSet, TExclusionSet> {\n    underlying_docset: TDocSet,\n    exclusion_set: TExclusionSet,\n}\n\nimpl<TDocSet, TExclusionSet> Exclude<TDocSet, TExclusionSet>\nwhere\n    TDocSet: DocSet,\n    TExclusionSet: ExclusionSet,\n{\n    /// Creates a new `ExcludeScorer`\n    pub fn new(\n        mut underlying_docset: TDocSet,\n        mut exclusion_set: TExclusionSet,\n    ) -> Exclude<TDocSet, TExclusionSet> {\n        while underlying_docset.doc() != TERMINATED {\n            let target = underlying_docset.doc();\n            if !exclusion_set.contains(target) {\n                break;\n            }\n            underlying_docset.advance();\n        }\n        Exclude {\n            underlying_docset,\n            exclusion_set,\n        }\n    }\n}\n\nimpl<TDocSet, TExclusionSet> DocSet for Exclude<TDocSet, TExclusionSet>\nwhere\n    TDocSet: DocSet,\n    TExclusionSet: ExclusionSet,\n{\n    fn advance(&mut self) -> DocId {\n        loop {\n            let candidate = self.underlying_docset.advance();\n            if candidate == TERMINATED {\n                return TERMINATED;\n            }\n            if !self.exclusion_set.contains(candidate) {\n                return candidate;\n            }\n        }\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        let candidate = self.underlying_docset.seek(target);\n        if candidate == TERMINATED {\n            return TERMINATED;\n        }\n        if !self.exclusion_set.contains(candidate) {\n            return candidate;\n        }\n        self.advance()\n    }\n\n    fn doc(&self) -> DocId {\n        self.underlying_docset.doc()\n    }\n\n    /// `.size_hint()` directly returns the size\n    /// of the underlying docset without taking in account\n    /// the fact that docs might be deleted.\n    fn size_hint(&self) -> u32 {\n        self.underlying_docset.size_hint()\n    }\n}\n\nimpl<TScorer, TExclusionSet> Scorer for Exclude<TScorer, TExclusionSet>\nwhere\n    TScorer: Scorer,\n    TExclusionSet: ExclusionSet + 'static,\n{\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.underlying_docset.score()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n    use crate::postings::tests::test_skip_against_unoptimized;\n    use crate::query::VecDocSet;\n    use crate::tests::sample_with_seed;\n\n    #[test]\n    fn test_exclude() {\n        let mut exclude_scorer = Exclude::new(\n            VecDocSet::from(vec![1, 2, 5, 8, 10, 15, 24]),\n            VecDocSet::from(vec![1, 2, 3, 10, 16, 24]),\n        );\n        let mut els = vec![];\n        while exclude_scorer.doc() != TERMINATED {\n            els.push(exclude_scorer.doc());\n            exclude_scorer.advance();\n        }\n        assert_eq!(els, vec![5, 8, 15]);\n    }\n\n    #[test]\n    fn test_exclude_skip() {\n        test_skip_against_unoptimized(\n            || {\n                Box::new(Exclude::new(\n                    VecDocSet::from(vec![1, 2, 5, 8, 10, 15, 24]),\n                    VecDocSet::from(vec![1, 2, 3, 10, 16, 24]),\n                ))\n            },\n            vec![5, 8, 10, 15, 24],\n        );\n    }\n\n    #[test]\n    fn test_exclude_skip_random() {\n        let sample_include = sample_with_seed(10_000, 0.1, 1);\n        let sample_exclude = sample_with_seed(10_000, 0.05, 2);\n        let sample_skip = sample_with_seed(10_000, 0.005, 3);\n        test_skip_against_unoptimized(\n            || {\n                Box::new(Exclude::new(\n                    VecDocSet::from(sample_include.clone()),\n                    VecDocSet::from(sample_exclude.clone()),\n                ))\n            },\n            sample_skip,\n        );\n    }\n}\n"
  },
  {
    "path": "src/query/exist_query.rs",
    "content": "use core::fmt::Debug;\n\nuse columnar::{ColumnIndex, DynamicColumn};\nuse common::BitSet;\n\nuse super::{ConstScorer, EmptyScorer};\nuse crate::docset::{DocSet, TERMINATED};\nuse crate::index::SegmentReader;\nuse crate::query::all_query::AllScorer;\nuse crate::query::boost_query::BoostScorer;\nuse crate::query::explanation::does_not_match;\nuse crate::query::{BitSetDocSet, EnableScoring, Explanation, Query, Scorer, Weight};\nuse crate::schema::Type;\nuse crate::{DocId, Score, TantivyError};\n\n/// Query that matches all documents with a non-null value in the specified\n/// field.\n///\n/// When querying inside a JSON field, \"exists\" queries can be executed strictly\n/// on the field name or check all the subpaths. In that second case a document\n/// will be matched if a non-null value exists in any subpath. For example,\n/// assuming the following document where `myfield` is a JSON fast field:\n/// ```json\n/// {\n///   \"myfield\": {\n///     \"mysubfield\": \"hello\"\n///   }\n/// }\n/// ```\n/// With `json_subpaths` enabled queries on either `myfield` or\n/// `myfield.mysubfield` will match the document. If it is set to false, only\n/// `myfield.mysubfield` will match it.\n///\n/// All of the matched documents get the score 1.0.\n#[derive(Clone, Debug)]\npub struct ExistsQuery {\n    field_name: String,\n    json_subpaths: bool,\n}\n\nimpl ExistsQuery {\n    /// Creates a new `ExistQuery` from the given field.\n    ///\n    /// This query matches all documents with at least one non-null value in the specified field.\n    /// This constructor never fails, but executing the search with this query will return an\n    /// error if the specified field doesn't exists or is not a fast field.\n    #[deprecated]\n    pub fn new_exists_query(field: String) -> ExistsQuery {\n        ExistsQuery {\n            field_name: field,\n            json_subpaths: false,\n        }\n    }\n\n    /// Creates a new `ExistQuery` from the given field.\n    ///\n    /// This query matches all documents with at least one non-null value in the\n    /// specified field. If `json_subpaths` is set to true, documents with\n    /// non-null values in any JSON subpath will also be matched.\n    ///\n    /// This constructor never fails, but executing the search with this query will\n    /// return an error if the specified field doesn't exists or is not a fast\n    /// field.\n    pub fn new(field: String, json_subpaths: bool) -> Self {\n        Self {\n            field_name: field,\n            json_subpaths,\n        }\n    }\n}\n\nimpl Query for ExistsQuery {\n    fn weight(&self, enable_scoring: EnableScoring) -> crate::Result<Box<dyn Weight>> {\n        let schema = enable_scoring.schema();\n        let Some((field, _path)) = schema.find_field(&self.field_name) else {\n            return Err(TantivyError::FieldNotFound(self.field_name.clone()));\n        };\n        let field_type = schema.get_field_entry(field).field_type();\n        if !field_type.is_fast() {\n            return Err(TantivyError::SchemaError(format!(\n                \"Field {} is not a fast field.\",\n                self.field_name\n            )));\n        }\n        Ok(Box::new(ExistsWeight {\n            field_name: self.field_name.clone(),\n            field_type: field_type.value_type(),\n            json_subpaths: self.json_subpaths,\n        }))\n    }\n}\n\n/// Weight associated with the `ExistsQuery` query.\npub struct ExistsWeight {\n    field_name: String,\n    field_type: Type,\n    json_subpaths: bool,\n}\n\nimpl Weight for ExistsWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let fast_field_reader = reader.fast_fields();\n        let mut column_handles = fast_field_reader.dynamic_column_handles(&self.field_name)?;\n        if self.field_type == Type::Json && self.json_subpaths {\n            let mut sub_columns =\n                fast_field_reader.dynamic_subpath_column_handles(&self.field_name)?;\n            column_handles.append(&mut sub_columns);\n        }\n        let dynamic_columns: crate::Result<Vec<DynamicColumn>> = column_handles\n            .into_iter()\n            .map(|handle| handle.open().map_err(|io_error| io_error.into()))\n            .collect();\n        let mut non_empty_columns = Vec::new();\n        for column in dynamic_columns? {\n            if !matches!(column.column_index(), ColumnIndex::Empty { .. }) {\n                non_empty_columns.push(column)\n            }\n        }\n        if non_empty_columns.is_empty() {\n            return Ok(Box::new(EmptyScorer));\n        }\n\n        // If any column is full, all docs match.\n        let max_doc = reader.max_doc();\n        if non_empty_columns\n            .iter()\n            .any(|col| matches!(col.column_index(), ColumnIndex::Full))\n        {\n            let all_scorer = AllScorer::new(max_doc);\n            if boost != 1.0f32 {\n                return Ok(Box::new(BoostScorer::new(all_scorer, boost)));\n            } else {\n                return Ok(Box::new(all_scorer));\n            }\n        }\n\n        // If we have a single dynamic column, use ExistsDocSet\n        // NOTE: A lower number may be better for very sparse columns\n        if non_empty_columns.len() < 4 {\n            let docset = ExistsDocSet::new(non_empty_columns, reader.max_doc());\n            return Ok(Box::new(ConstScorer::new(docset, boost)));\n        }\n\n        // If we have many dynamic columns, precompute a bitset of matching docs\n        let mut doc_bitset = BitSet::with_max_value(max_doc);\n        for column in &non_empty_columns {\n            match column.column_index() {\n                ColumnIndex::Empty { .. } => {}\n                ColumnIndex::Full => {\n                    // Handled by AllScorer return above.\n                }\n                ColumnIndex::Optional(optional_index) => {\n                    for doc in optional_index.iter_non_null_docs() {\n                        doc_bitset.insert(doc);\n                    }\n                }\n                ColumnIndex::Multivalued(multi_idx) => {\n                    for doc in multi_idx.iter_non_null_docs() {\n                        doc_bitset.insert(doc);\n                    }\n                }\n            }\n        }\n        let docset = BitSetDocSet::from(doc_bitset);\n        Ok(Box::new(ConstScorer::new(docset, boost)))\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        Ok(Explanation::new(\"ExistsQuery\", 1.0))\n    }\n}\n\npub(crate) struct ExistsDocSet {\n    columns: Vec<DynamicColumn>,\n    doc: DocId,\n    max_doc: DocId,\n}\n\nimpl ExistsDocSet {\n    pub(crate) fn new(columns: Vec<DynamicColumn>, max_doc: DocId) -> Self {\n        let mut set = Self {\n            columns,\n            doc: 0u32,\n            max_doc,\n        };\n        set.find_next();\n        set\n    }\n\n    fn find_next(&mut self) -> DocId {\n        while self.doc < self.max_doc {\n            if self\n                .columns\n                .iter()\n                .any(|col| col.column_index().has_value(self.doc))\n            {\n                return self.doc;\n            }\n            self.doc += 1;\n        }\n        self.doc = TERMINATED;\n        TERMINATED\n    }\n}\n\nimpl DocSet for ExistsDocSet {\n    fn advance(&mut self) -> DocId {\n        self.seek(self.doc + 1)\n    }\n\n    fn size_hint(&self) -> u32 {\n        0\n    }\n\n    fn doc(&self) -> DocId {\n        self.doc\n    }\n\n    #[inline(always)]\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.doc = target;\n        self.find_next()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::Ipv6Addr;\n    use std::ops::Bound;\n\n    use common::DateTime;\n    use time::OffsetDateTime;\n\n    use crate::collector::Count;\n    use crate::query::exist_query::ExistsQuery;\n    use crate::query::{BooleanQuery, RangeQuery};\n    use crate::schema::{Facet, FacetOptions, Schema, FAST, INDEXED, STRING, TEXT};\n    use crate::{Index, Searcher, Term};\n\n    #[test]\n    fn test_exists_query_simple() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let all_field = schema_builder.add_u64_field(\"all\", INDEXED | FAST);\n        let even_field = schema_builder.add_u64_field(\"even\", INDEXED | FAST);\n        let odd_field = schema_builder.add_text_field(\"odd\", STRING | FAST);\n        let multi_field = schema_builder.add_text_field(\"multi\", FAST);\n        let _never_field = schema_builder.add_u64_field(\"never\", INDEXED | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            for i in 0u64..100u64 {\n                if i % 2 == 0 {\n                    if i % 10 == 0 {\n                        index_writer.add_document(doc!(all_field => i, even_field => i, multi_field => i.to_string(), multi_field => (i + 1).to_string()))?;\n                    } else {\n                        index_writer.add_document(doc!(all_field => i, even_field => i))?;\n                    }\n                } else {\n                    index_writer.add_document(doc!(all_field => i, odd_field => i.to_string()))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        assert_eq!(count_existing_fields(&searcher, \"all\", false)?, 100);\n        assert_eq!(count_existing_fields(&searcher, \"odd\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"even\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"multi\", false)?, 10);\n        assert_eq!(count_existing_fields(&searcher, \"multi\", true)?, 10);\n        assert_eq!(count_existing_fields(&searcher, \"never\", false)?, 0);\n\n        // exercise seek\n        let query = BooleanQuery::intersection(vec![\n            Box::new(RangeQuery::new(\n                Bound::Included(Term::from_field_u64(all_field, 50)),\n                Bound::Unbounded,\n            )),\n            Box::new(ExistsQuery::new(\"even\".to_string(), false)),\n        ]);\n        assert_eq!(searcher.search(&query, &Count)?, 25);\n\n        let query = BooleanQuery::intersection(vec![\n            Box::new(RangeQuery::new(\n                Bound::Included(Term::from_field_u64(all_field, 0)),\n                Bound::Included(Term::from_field_u64(all_field, 50)),\n            )),\n            Box::new(ExistsQuery::new(\"odd\".to_string(), false)),\n        ]);\n        assert_eq!(searcher.search(&query, &Count)?, 25);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_exists_query_json() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", TEXT | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            for i in 0u64..100u64 {\n                if i % 2 == 0 {\n                    index_writer.add_document(doc!(json => json!({\"all\": i, \"even\": true})))?;\n                } else {\n                    index_writer\n                        .add_document(doc!(json => json!({\"all\": i.to_string(), \"odd\": true})))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        assert_eq!(count_existing_fields(&searcher, \"json.all\", false)?, 100);\n        assert_eq!(count_existing_fields(&searcher, \"json.even\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"json.even\", true)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"json.odd\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"json\", false)?, 0);\n        assert_eq!(count_existing_fields(&searcher, \"json\", true)?, 100);\n\n        // Handling of non-existing fields:\n        assert_eq!(count_existing_fields(&searcher, \"json.absent\", false)?, 0);\n        assert_eq!(count_existing_fields(&searcher, \"json.absent\", true)?, 0);\n        assert_does_not_exist(&searcher, \"does_not_exists.absent\", true);\n        assert_does_not_exist(&searcher, \"does_not_exists.absent\", false);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_exists_query_json_union_no_single_full_subpath() -> crate::Result<()> {\n        // Build docs where no single subpath exists for all docs, but the union does.\n        let mut schema_builder = Schema::builder();\n        let json = schema_builder.add_json_field(\"json\", TEXT | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            for i in 0u64..100u64 {\n                if i % 2 == 0 {\n                    // only subpath `a`\n                    index_writer.add_document(doc!(json => json!({\"a\": i})))?;\n                } else {\n                    // only subpath `b`\n                    index_writer.add_document(doc!(json => json!({\"b\": i})))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // No single subpath is full\n        assert_eq!(count_existing_fields(&searcher, \"json.a\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"json.b\", false)?, 50);\n\n        // Root exists with subpaths disabled is zero\n        assert_eq!(count_existing_fields(&searcher, \"json\", false)?, 0);\n\n        // Root exists with subpaths enabled should match all docs via union\n        assert_eq!(count_existing_fields(&searcher, \"json\", true)?, 100);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_exists_query_misc_supported_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let bool = schema_builder.add_bool_field(\"bool\", FAST);\n        let bytes = schema_builder.add_bytes_field(\"bytes\", FAST);\n        let date = schema_builder.add_date_field(\"date\", FAST);\n        let f64 = schema_builder.add_f64_field(\"f64\", FAST);\n        let ip_addr = schema_builder.add_ip_addr_field(\"ip_addr\", FAST);\n        let facet = schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            let now = OffsetDateTime::now_utc().unix_timestamp();\n            for i in 0u8..100u8 {\n                if i % 2 == 0 {\n                    let date_val = DateTime::from_utc(OffsetDateTime::from_unix_timestamp(\n                        now + i as i64 * 100,\n                    )?);\n                    index_writer.add_document(\n                        doc!(bool => i % 3 == 0, bytes => vec![i, i + 1,  i + 2], date => date_val),\n                    )?;\n                } else {\n                    let ip_addr_v6 = Ipv6Addr::new(0, 0, 0, 0, 0, 0xffff, 0xc00a, i.into());\n                    index_writer\n                        .add_document(doc!(f64 => i as f64 * 0.5, ip_addr => ip_addr_v6, facet => Facet::from(\"/facet/foo\"), facet => Facet::from(\"/facet/bar\")))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        assert_eq!(count_existing_fields(&searcher, \"bool\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"bool\", true)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"bytes\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"date\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"f64\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"ip_addr\", false)?, 50);\n        assert_eq!(count_existing_fields(&searcher, \"facet\", false)?, 50);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_exists_query_unsupported_types() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let not_fast = schema_builder.add_text_field(\"not_fast\", TEXT);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(\n                not_fast => \"slow\",\n            ))?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        assert_eq!(\n            searcher\n                .search(&ExistsQuery::new(\"not_fast\".to_string(), false), &Count)\n                .unwrap_err()\n                .to_string(),\n            \"Schema error: 'Field not_fast is not a fast field.'\"\n        );\n\n        assert_does_not_exist(&searcher, \"does_not_exists\", false);\n\n        Ok(())\n    }\n\n    fn count_existing_fields(\n        searcher: &Searcher,\n        field: &str,\n        json_subpaths: bool,\n    ) -> crate::Result<usize> {\n        let query = ExistsQuery::new(field.to_string(), json_subpaths);\n        searcher.search(&query, &Count)\n    }\n\n    fn assert_does_not_exist(searcher: &Searcher, field: &str, json_subpaths: bool) {\n        assert_eq!(\n            searcher\n                .search(&ExistsQuery::new(field.to_string(), json_subpaths), &Count)\n                .unwrap_err()\n                .to_string(),\n            format!(\"The field does not exist: '{field}'\")\n        );\n    }\n}\n"
  },
  {
    "path": "src/query/explanation.rs",
    "content": "use std::borrow::Cow;\nuse std::fmt;\n\nuse serde::Serialize;\n\nuse crate::{DocId, Score, TantivyError};\n\npub(crate) fn does_not_match(doc: DocId) -> TantivyError {\n    TantivyError::InvalidArgument(format!(\"Document #({doc}) does not match\"))\n}\n\n/// Object describing the score of a given document.\n/// It is organized in trees.\n///\n/// `.to_pretty_json()` can be useful to print out a human readable\n/// representation of this tree when debugging a given score.\n#[derive(Clone, Serialize)]\npub struct Explanation {\n    value: Score,\n    description: Cow<'static, str>,\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    details: Option<Vec<Explanation>>,\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    context: Option<Vec<String>>,\n}\nimpl fmt::Debug for Explanation {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"Explanation({})\", self.to_pretty_json())\n    }\n}\n\nimpl Explanation {\n    /// Creates a new explanation object.\n    pub fn new_with_string(description: String, value: Score) -> Explanation {\n        Explanation {\n            value,\n            description: Cow::Owned(description),\n            details: None,\n            context: None,\n        }\n    }\n    /// Creates a new explanation object.\n    pub fn new(description: &'static str, value: Score) -> Explanation {\n        Explanation {\n            value,\n            description: Cow::Borrowed(description),\n            details: None,\n            context: None,\n        }\n    }\n\n    /// Returns the value associated with the current node.\n    pub fn value(&self) -> Score {\n        self.value\n    }\n\n    /// Add some detail, explaining some part of the current node formula.\n    ///\n    /// Details are treated as child of the current node.\n    pub fn add_detail(&mut self, child_explanation: Explanation) {\n        self.details\n            .get_or_insert_with(Vec::new)\n            .push(child_explanation);\n    }\n\n    /// Adds some extra context to the explanation.\n    pub fn add_context(&mut self, context: String) {\n        self.context.get_or_insert_with(Vec::new).push(context);\n    }\n\n    /// Shortcut for `self.details.push(Explanation::new(name, value));`\n    pub fn add_const(&mut self, name: &'static str, value: Score) {\n        self.details\n            .get_or_insert_with(Vec::new)\n            .push(Explanation::new(name, value));\n    }\n\n    /// Returns an indented json representation of the explanation tree for debug usage.\n    pub fn to_pretty_json(&self) -> String {\n        serde_json::to_string_pretty(self).unwrap()\n    }\n}\n"
  },
  {
    "path": "src/query/fuzzy_query.rs",
    "content": "use levenshtein_automata::{Distance, LevenshteinAutomatonBuilder, DFA};\nuse once_cell::sync::OnceCell;\nuse tantivy_fst::Automaton;\n\nuse crate::query::{AutomatonWeight, EnableScoring, Query, Weight};\nuse crate::schema::{Term, Type};\nuse crate::TantivyError::InvalidArgument;\n\npub(crate) struct DfaWrapper(pub DFA);\n\nimpl Automaton for DfaWrapper {\n    type State = u32;\n\n    fn start(&self) -> Self::State {\n        self.0.initial_state()\n    }\n\n    fn is_match(&self, state: &Self::State) -> bool {\n        match self.0.distance(*state) {\n            Distance::Exact(_) => true,\n            Distance::AtLeast(_) => false,\n        }\n    }\n\n    fn can_match(&self, state: &u32) -> bool {\n        *state != levenshtein_automata::SINK_STATE\n    }\n\n    fn accept(&self, state: &Self::State, byte: u8) -> Self::State {\n        self.0.transition(*state, byte)\n    }\n}\n\n/// A Fuzzy Query matches all of the documents\n/// containing a specific term that is within\n/// Levenshtein distance\n/// ```rust\n/// use tantivy::collector::{Count, TopDocs};\n/// use tantivy::query::FuzzyTermQuery;\n/// use tantivy::schema::{Schema, TEXT};\n/// use tantivy::{doc, Index, IndexWriter, Term};\n///\n/// fn example() -> tantivy::Result<()> {\n///     let mut schema_builder = Schema::builder();\n///     let title = schema_builder.add_text_field(\"title\", TEXT);\n///     let schema = schema_builder.build();\n///     let index = Index::create_in_ram(schema);\n///     {\n///         let mut index_writer: IndexWriter = index.writer(15_000_000)?;\n///         index_writer.add_document(doc!(\n///             title => \"The Name of the Wind\",\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"The Diary of Muadib\",\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"A Dairy Cow\",\n///         ))?;\n///         index_writer.add_document(doc!(\n///             title => \"The Diary of a Young Girl\",\n///         ))?;\n///         index_writer.commit()?;\n///     }\n///     let reader = index.reader()?;\n///     let searcher = reader.searcher();\n///\n///     {\n///         let term = Term::from_field_text(title, \"Diary\");\n///         let query = FuzzyTermQuery::new(term, 1, true);\n///         let (top_docs, count) = searcher.search(&query, &(TopDocs::with_limit(2).order_by_score(), Count)).unwrap();\n///         assert_eq!(count, 2);\n///         assert_eq!(top_docs.len(), 2);\n///     }\n///\n///     Ok(())\n/// }\n/// # assert!(example().is_ok());\n/// ```\n#[derive(Debug, Clone)]\npub struct FuzzyTermQuery {\n    /// What term are we searching\n    term: Term,\n    /// How many changes are we going to allow\n    distance: u8,\n    /// Should a transposition cost 1 or 2?\n    transposition_cost_one: bool,\n    /// is a starts with query\n    prefix: bool,\n}\n\nimpl FuzzyTermQuery {\n    /// Creates a new Fuzzy Query\n    pub fn new(term: Term, distance: u8, transposition_cost_one: bool) -> FuzzyTermQuery {\n        FuzzyTermQuery {\n            term,\n            distance,\n            transposition_cost_one,\n            prefix: false,\n        }\n    }\n\n    /// Creates a new Fuzzy Query of the Term prefix\n    pub fn new_prefix(term: Term, distance: u8, transposition_cost_one: bool) -> FuzzyTermQuery {\n        FuzzyTermQuery {\n            term,\n            distance,\n            transposition_cost_one,\n            prefix: true,\n        }\n    }\n\n    fn specialized_weight(&self) -> crate::Result<AutomatonWeight<DfaWrapper>> {\n        static AUTOMATON_BUILDER: [[OnceCell<LevenshteinAutomatonBuilder>; 2]; 3] = [\n            [OnceCell::new(), OnceCell::new()],\n            [OnceCell::new(), OnceCell::new()],\n            [OnceCell::new(), OnceCell::new()],\n        ];\n\n        let automaton_builder = AUTOMATON_BUILDER\n            .get(self.distance as usize)\n            .ok_or_else(|| {\n                InvalidArgument(format!(\n                    \"Levenshtein distance of {} is not allowed. Choose a value less than {}\",\n                    self.distance,\n                    AUTOMATON_BUILDER.len()\n                ))\n            })?\n            .get(self.transposition_cost_one as usize)\n            .unwrap()\n            .get_or_init(|| {\n                LevenshteinAutomatonBuilder::new(self.distance, self.transposition_cost_one)\n            });\n\n        let term_value = self.term.value();\n\n        let term_text = if term_value.typ() == Type::Json {\n            if let Some(json_path_type) = term_value.json_path_type() {\n                if json_path_type != Type::Str {\n                    return Err(InvalidArgument(format!(\n                        \"The fuzzy term query requires a string path type for a json term. Found \\\n                         {json_path_type:?}\"\n                    )));\n                }\n            }\n\n            std::str::from_utf8(self.term.serialized_value_bytes()).map_err(|_| {\n                InvalidArgument(\n                    \"Failed to convert json term value bytes to utf8 string.\".to_string(),\n                )\n            })?\n        } else {\n            term_value.as_str().ok_or_else(|| {\n                InvalidArgument(\"The fuzzy term query requires a string term.\".to_string())\n            })?\n        };\n        let automaton = if self.prefix {\n            automaton_builder.build_prefix_dfa(term_text)\n        } else {\n            automaton_builder.build_dfa(term_text)\n        };\n\n        if let Some((json_path_bytes, _)) = term_value.as_json() {\n            Ok(AutomatonWeight::new_for_json_path(\n                self.term.field(),\n                DfaWrapper(automaton),\n                json_path_bytes,\n            ))\n        } else {\n            Ok(AutomatonWeight::new(\n                self.term.field(),\n                DfaWrapper(automaton),\n            ))\n        }\n    }\n}\n\nimpl Query for FuzzyTermQuery {\n    fn weight(&self, _enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(self.specialized_weight()?))\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use super::FuzzyTermQuery;\n    use crate::collector::{Count, TopDocs};\n    use crate::indexer::NoMergePolicy;\n    use crate::query::QueryParser;\n    use crate::schema::{Schema, STORED, TEXT};\n    use crate::{assert_nearly_equals, Index, IndexWriter, TantivyDocument, Term};\n\n    #[test]\n    pub fn test_fuzzy_json_path() -> crate::Result<()> {\n        // # Defining the schema\n        let mut schema_builder = Schema::builder();\n        let attributes = schema_builder.add_json_field(\"attributes\", TEXT | STORED);\n        let schema = schema_builder.build();\n\n        // # Indexing documents\n        let index = Index::create_in_ram(schema.clone());\n\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n        let doc = TantivyDocument::parse_json(\n            &schema,\n            r#\"{\n            \"attributes\": {\n                \"a\": \"japan\"\n            }\n        }\"#,\n        )?;\n        index_writer.add_document(doc)?;\n        let doc = TantivyDocument::parse_json(\n            &schema,\n            r#\"{\n            \"attributes\": {\n                \"aa\": \"japan\"\n            }\n        }\"#,\n        )?;\n        index_writer.add_document(doc)?;\n        index_writer.commit()?;\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // # Fuzzy search\n        let query_parser = QueryParser::for_index(&index, vec![attributes]);\n\n        let get_json_path_term = |query: &str| -> crate::Result<Term> {\n            let query = query_parser.parse_query(query)?;\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, _| {\n                terms.push(term.clone());\n            });\n\n            Ok(terms[0].clone())\n        };\n\n        // shall not match the first document due to json path mismatch\n        {\n            let term = get_json_path_term(\"attributes.aa:japan\")?;\n            let fuzzy_query = FuzzyTermQuery::new(term, 2, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected only 1 document\");\n            assert_eq!(top_docs[0].1.doc_id, 1, \"Expected the second document\");\n        }\n\n        // shall match the first document because Levenshtein distance is 1 (substitute 'o' with\n        // 'a')\n        {\n            let term = get_json_path_term(\"attributes.a:japon\")?;\n\n            let fuzzy_query = FuzzyTermQuery::new(term, 1, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected only 1 document\");\n            assert_eq!(top_docs[0].1.doc_id, 0, \"Expected the first document\");\n        }\n\n        // shall not match because non-prefix Levenshtein distance is more than 1 (add 'a' and 'n')\n        {\n            let term = get_json_path_term(\"attributes.a:jap\")?;\n\n            let fuzzy_query = FuzzyTermQuery::new(term, 1, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 0, \"Expected no document\");\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_fuzzy_term() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let country_field = schema_builder.add_text_field(\"country\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(\n                country_field => \"japan\",\n            ))?;\n            index_writer.add_document(doc!(\n                country_field => \"korea\",\n            ))?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // passes because Levenshtein distance is 1 (substitute 'o' with 'a')\n        {\n            let term = Term::from_field_text(country_field, \"japon\");\n            let fuzzy_query = FuzzyTermQuery::new(term, 1, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected only 1 document\");\n            let (score, _) = top_docs[0];\n            assert_nearly_equals!(1.0, score);\n        }\n\n        // fails because non-prefix Levenshtein distance is more than 1 (add 'a' and 'n')\n        {\n            let term = Term::from_field_text(country_field, \"jap\");\n\n            let fuzzy_query = FuzzyTermQuery::new(term, 1, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 0, \"Expected no document\");\n        }\n\n        // passes because prefix Levenshtein distance is 0\n        {\n            let term = Term::from_field_text(country_field, \"jap\");\n            let fuzzy_query = FuzzyTermQuery::new_prefix(term, 1, true);\n            let top_docs =\n                searcher.search(&fuzzy_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected only 1 document\");\n            let (score, _) = top_docs[0];\n            assert_nearly_equals!(1.0, score);\n        }\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_fuzzy_term_transposition_cost_one() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let country_field = schema_builder.add_text_field(\"country\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(country_field => \"japan\"))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let term_jaapn = Term::from_field_text(country_field, \"jaapn\");\n        {\n            let fuzzy_query_transposition = FuzzyTermQuery::new(term_jaapn.clone(), 1, true);\n            let count = searcher.search(&fuzzy_query_transposition, &Count)?;\n            assert_eq!(count, 1);\n        }\n        {\n            let fuzzy_query_transposition = FuzzyTermQuery::new(term_jaapn, 1, false);\n            let count = searcher.search(&fuzzy_query_transposition, &Count)?;\n            assert_eq!(count, 0);\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/intersection.rs",
    "content": "use super::size_hint::estimate_intersection;\nuse crate::docset::{DocSet, SeekDangerResult, TERMINATED};\nuse crate::query::term_query::TermScorer;\nuse crate::query::{EmptyScorer, Scorer};\nuse crate::{DocId, Score};\n\n/// Returns the intersection scorer.\n///\n/// The score associated with the documents is the sum of the\n/// score of the `Scorer`s given in argument.\n///\n/// For better performance, the function uses a\n/// specialized implementation if the two\n/// shortest scorers are `TermScorer`s.\n///\n/// num_docs_segment is the number of documents in the segment. It is used for estimating the\n/// `size_hint` of the intersection.\npub fn intersect_scorers(\n    mut scorers: Vec<Box<dyn Scorer>>,\n    num_docs_segment: u32,\n) -> Box<dyn Scorer> {\n    if scorers.is_empty() {\n        return Box::new(EmptyScorer);\n    }\n    if scorers.len() == 1 {\n        return scorers.pop().unwrap();\n    }\n    // Order by estimated cost to drive each scorer.\n    scorers.sort_by_key(|scorer| scorer.cost());\n    let doc = go_to_first_doc(&mut scorers[..]);\n    if doc == TERMINATED {\n        return Box::new(EmptyScorer);\n    }\n    // We know that we have at least 2 elements.\n    let left = scorers.remove(0);\n    let right = scorers.remove(0);\n    let all_term_scorers = [&left, &right]\n        .iter()\n        .all(|&scorer| scorer.is::<TermScorer>());\n    if all_term_scorers {\n        return Box::new(Intersection {\n            left: *(left.downcast::<TermScorer>().map_err(|_| ()).unwrap()),\n            right: *(right.downcast::<TermScorer>().map_err(|_| ()).unwrap()),\n            others: scorers,\n            num_docs: num_docs_segment,\n        });\n    }\n    Box::new(Intersection {\n        left,\n        right,\n        others: scorers,\n        num_docs: num_docs_segment,\n    })\n}\n\n/// Creates a `DocSet` that iterate through the intersection of two or more `DocSet`s.\npub struct Intersection<TDocSet: DocSet, TOtherDocSet: DocSet = Box<dyn Scorer>> {\n    left: TDocSet,\n    right: TDocSet,\n    others: Vec<TOtherDocSet>,\n    num_docs: u32,\n}\n\nfn go_to_first_doc<TDocSet: DocSet>(docsets: &mut [TDocSet]) -> DocId {\n    assert!(!docsets.is_empty());\n    let mut candidate = docsets.iter().map(TDocSet::doc).max().unwrap();\n    'outer: loop {\n        for docset in docsets.iter_mut() {\n            let seek_doc = docset.seek(candidate);\n            if seek_doc > candidate {\n                candidate = docset.doc();\n                continue 'outer;\n            }\n        }\n        return candidate;\n    }\n}\n\nimpl<TDocSet: DocSet> Intersection<TDocSet, TDocSet> {\n    /// num_docs is the number of documents in the segment.\n    pub(crate) fn new(mut docsets: Vec<TDocSet>, num_docs: u32) -> Intersection<TDocSet, TDocSet> {\n        let num_docsets = docsets.len();\n        assert!(num_docsets >= 2);\n        docsets.sort_by_key(|docset| docset.cost());\n        go_to_first_doc(&mut docsets);\n        let left = docsets.remove(0);\n        debug_assert!({\n            let doc = left.doc();\n            if doc == TERMINATED {\n                true\n            } else {\n                docsets.iter().all(|docset| docset.doc() == doc)\n            }\n        });\n        let right = docsets.remove(0);\n        Intersection {\n            left,\n            right,\n            others: docsets,\n            num_docs,\n        }\n    }\n}\n\nimpl<TDocSet: DocSet> Intersection<TDocSet, TDocSet> {\n    pub(crate) fn docset_mut_specialized(&mut self, ord: usize) -> &mut TDocSet {\n        match ord {\n            0 => &mut self.left,\n            1 => &mut self.right,\n            n => &mut self.others[n - 2],\n        }\n    }\n}\n\nimpl<TDocSet: DocSet, TOtherDocSet: DocSet> DocSet for Intersection<TDocSet, TOtherDocSet> {\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        let (left, right) = (&mut self.left, &mut self.right);\n\n        // Invariant:\n        // - candidate is always <= to the next document in the intersection.\n        // - candidate strictly increases at every occurence of the loop.\n        let mut candidate = left.doc() + 1;\n\n        // Termination: candidate strictly increases.\n        'outer: while candidate < TERMINATED {\n            // As we enter the loop, we should always have candidate < next_doc.\n\n            candidate = left.seek(candidate);\n\n            // Left is positionned on `candidate`.\n            debug_assert_eq!(left.doc(), candidate);\n\n            if let SeekDangerResult::SeekLowerBound(seek_lower_bound) = right.seek_danger(candidate)\n            {\n                debug_assert!(\n                    seek_lower_bound == TERMINATED || seek_lower_bound > candidate,\n                    \"seek_lower_bound {seek_lower_bound} must be greater than candidate \\\n                     {candidate}\"\n                );\n                candidate = seek_lower_bound;\n                continue;\n            }\n\n            // Left and right are positionned on `candidate`.\n            debug_assert_eq!(right.doc(), candidate);\n\n            for other in &mut self.others {\n                if let SeekDangerResult::SeekLowerBound(seek_lower_bound) =\n                    other.seek_danger(candidate)\n                {\n                    // One of the scorer does not match, let's restart at the top of the loop.\n                    debug_assert!(\n                        seek_lower_bound == TERMINATED || seek_lower_bound > candidate,\n                        \"seek_lower_bound {seek_lower_bound} must be greater than candidate \\\n                         {candidate}\"\n                    );\n                    candidate = seek_lower_bound;\n                    continue 'outer;\n                }\n            }\n\n            // At this point all scorers are in a valid state, aligned on the next document in the\n            // intersection.\n            debug_assert!(self.others.iter().all(|docset| docset.doc() == candidate));\n            return candidate;\n        }\n\n        // We make sure our docset is in a valid state.\n        // In particular, we want .doc() to return TERMINATED.\n        left.seek(TERMINATED);\n\n        TERMINATED\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.left.seek(target);\n        let mut docsets: Vec<&mut dyn DocSet> = vec![&mut self.left, &mut self.right];\n        for docset in &mut self.others {\n            docsets.push(docset);\n        }\n        let doc = go_to_first_doc(&mut docsets[..]);\n        debug_assert!(docsets.iter().all(|docset| docset.doc() == doc));\n        debug_assert!(doc >= target);\n        doc\n    }\n\n    /// Seeks to the target if necessary and checks if the target is an exact match.\n    ///\n    /// Some implementations may choose to advance past the target if beneficial for performance.\n    /// The return value is `true` if the target is in the docset, and `false` otherwise.\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        if let SeekDangerResult::SeekLowerBound(new_target) = self.left.seek_danger(target) {\n            return SeekDangerResult::SeekLowerBound(new_target);\n        }\n        if let SeekDangerResult::SeekLowerBound(new_target) = self.right.seek_danger(target) {\n            return SeekDangerResult::SeekLowerBound(new_target);\n        }\n        for docset in &mut self.others {\n            if let SeekDangerResult::SeekLowerBound(new_target) = docset.seek_danger(target) {\n                return SeekDangerResult::SeekLowerBound(new_target);\n            }\n        }\n        SeekDangerResult::Found\n    }\n\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.left.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        estimate_intersection(\n            [self.left.size_hint(), self.right.size_hint()]\n                .into_iter()\n                .chain(self.others.iter().map(DocSet::size_hint)),\n            self.num_docs,\n        )\n    }\n\n    fn cost(&self) -> u64 {\n        // What's the best way to compute the cost of an intersection?\n        // For now we take the cost of the docset driver, which is the first docset.\n        // If there are docsets that are bad at skipping, they should also influence the cost.\n        self.left.cost()\n    }\n}\n\nimpl<TScorer, TOtherScorer> Scorer for Intersection<TScorer, TOtherScorer>\nwhere\n    TScorer: Scorer,\n    TOtherScorer: Scorer,\n{\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.left.score()\n            + self.right.score()\n            + self.others.iter_mut().map(Scorer::score).sum::<Score>()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use proptest::prelude::*;\n\n    use super::Intersection;\n    use crate::collector::Count;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::postings::tests::test_skip_against_unoptimized;\n    use crate::query::{QueryParser, VecDocSet};\n    use crate::schema::{Schema, TEXT};\n    use crate::Index;\n\n    #[test]\n    fn test_intersection() {\n        {\n            let left = VecDocSet::from(vec![1, 3, 9]);\n            let right = VecDocSet::from(vec![3, 4, 9, 18]);\n            let mut intersection = Intersection::new(vec![left, right], 10);\n            assert_eq!(intersection.doc(), 3);\n            assert_eq!(intersection.advance(), 9);\n            assert_eq!(intersection.doc(), 9);\n            assert_eq!(intersection.advance(), TERMINATED);\n        }\n        {\n            let a = VecDocSet::from(vec![1, 3, 9]);\n            let b = VecDocSet::from(vec![3, 4, 9, 18]);\n            let c = VecDocSet::from(vec![1, 5, 9, 111]);\n            let mut intersection = Intersection::new(vec![a, b, c], 10);\n            assert_eq!(intersection.doc(), 9);\n            assert_eq!(intersection.advance(), TERMINATED);\n        }\n    }\n\n    #[test]\n    fn test_intersection_zero() {\n        let left = VecDocSet::from(vec![0]);\n        let right = VecDocSet::from(vec![0]);\n        let mut intersection = Intersection::new(vec![left, right], 10);\n        assert_eq!(intersection.doc(), 0);\n        assert_eq!(intersection.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_intersection_skip() {\n        let left = VecDocSet::from(vec![0, 1, 2, 4]);\n        let right = VecDocSet::from(vec![2, 5]);\n        let mut intersection = Intersection::new(vec![left, right], 10);\n        assert_eq!(intersection.seek(2), 2);\n        assert_eq!(intersection.doc(), 2);\n    }\n\n    #[test]\n    fn test_intersection_skip_against_unoptimized() {\n        test_skip_against_unoptimized(\n            || {\n                let left = VecDocSet::from(vec![4]);\n                let right = VecDocSet::from(vec![2, 5]);\n                Box::new(Intersection::new(vec![left, right], 10))\n            },\n            vec![0, 2, 4, 5, 6],\n        );\n        test_skip_against_unoptimized(\n            || {\n                let mut left = VecDocSet::from(vec![1, 4, 5, 6]);\n                let mut right = VecDocSet::from(vec![2, 5, 10]);\n                left.advance();\n                right.advance();\n                Box::new(Intersection::new(vec![left, right], 10))\n            },\n            vec![0, 1, 2, 3, 4, 5, 6, 7, 10, 11],\n        );\n        test_skip_against_unoptimized(\n            || {\n                Box::new(Intersection::new(\n                    vec![\n                        VecDocSet::from(vec![1, 4, 5, 6]),\n                        VecDocSet::from(vec![1, 2, 5, 6]),\n                        VecDocSet::from(vec![1, 4, 5, 6]),\n                        VecDocSet::from(vec![1, 5, 6]),\n                        VecDocSet::from(vec![2, 4, 5, 7, 8]),\n                    ],\n                    10,\n                ))\n            },\n            vec![0, 1, 2, 3, 4, 5, 6, 7, 10, 11],\n        );\n    }\n\n    #[test]\n    fn test_intersection_empty() {\n        let a = VecDocSet::from(vec![1, 3]);\n        let b = VecDocSet::from(vec![1, 4]);\n        let c = VecDocSet::from(vec![3, 9]);\n        let intersection = Intersection::new(vec![a, b, c], 10);\n        assert_eq!(intersection.doc(), TERMINATED);\n    }\n\n    #[test]\n    fn test_intersection_abc() {\n        let a = VecDocSet::from(vec![2, 3, 6]);\n        let b = VecDocSet::from(vec![1, 3, 5]);\n        let c = VecDocSet::from(vec![1, 3, 5]);\n        let mut intersection = Intersection::new(vec![c, b, a], 10);\n        let mut docs = Vec::new();\n        use crate::DocSet;\n        while intersection.doc() != TERMINATED {\n            docs.push(intersection.doc());\n            intersection.advance();\n        }\n        assert_eq!(&docs, &[3]);\n    }\n\n    #[test]\n    fn test_intersection_termination() {\n        use crate::query::score_combiner::DoNothingCombiner;\n        use crate::query::{BufferedUnionScorer, ConstScorer, VecDocSet};\n\n        let a1 = ConstScorer::new(VecDocSet::from(vec![0u32, 10000]), 1.0);\n        let a2 = ConstScorer::new(VecDocSet::from(vec![0u32, 10000]), 1.0);\n\n        let mut b_scorers = vec![];\n        for _ in 0..2 {\n            // Union matches 0 and 10000.\n            b_scorers.push(ConstScorer::new(VecDocSet::from(vec![0, 10000]), 1.0));\n        }\n        // That's the union of two scores matching 0, and 10_000.\n        let union = BufferedUnionScorer::build(b_scorers, DoNothingCombiner::default, 30000);\n\n        // Mismatching scorer: matches 0 and 20000. We then append more docs at the end to ensure it\n        // is last.\n        let mut m_docs = vec![0, 20000];\n        for i in 30000..30100 {\n            m_docs.push(i);\n        }\n        let m = ConstScorer::new(VecDocSet::from(m_docs), 1.0);\n\n        // Costs: A1=2, A2=2, Union=4, M=102.\n        // Sorted: A1, A2, Union, M.\n        // Left=A1, Right=A2, Others=[Union, M].\n        let mut intersection = crate::query::intersect_scorers(\n            vec![Box::new(a1), Box::new(a2), Box::new(union), Box::new(m)],\n            40000,\n        );\n\n        while intersection.doc() != TERMINATED {\n            intersection.advance();\n        }\n    }\n\n    // Strategy to generate sorted and deduplicated vectors of u32 document IDs\n    fn sorted_deduped_vec(max_val: u32, max_size: usize) -> impl Strategy<Value = Vec<u32>> {\n        prop::collection::vec(0..max_val, 0..max_size).prop_map(|mut vec| {\n            vec.sort();\n            vec.dedup();\n            vec\n        })\n    }\n\n    proptest! {\n        #[test]\n        fn prop_test_intersection_consistency(\n            a in sorted_deduped_vec(100, 10),\n            b in sorted_deduped_vec(100, 10),\n            num_docs in 100u32..500u32\n        ) {\n            let left = VecDocSet::from(a.clone());\n            let right = VecDocSet::from(b.clone());\n            let mut intersection = Intersection::new(vec![left, right], num_docs);\n\n            let expected: Vec<u32> = a.iter()\n                .cloned()\n                .filter(|doc| b.contains(doc))\n                .collect();\n\n            for expected_doc in expected {\n                assert_eq!(intersection.doc(), expected_doc);\n                intersection.advance();\n            }\n            assert_eq!(intersection.doc(), TERMINATED);\n        }\n    }\n\n    #[test]\n    fn test_bug_2811_intersection_candidate_should_increase() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let mut writer = index.writer_for_tests().unwrap();\n        writer\n            .add_document(doc!(text_field=>\"hello happy tax\"))\n            .unwrap();\n        writer.add_document(doc!(text_field=>\"hello\")).unwrap();\n        writer.add_document(doc!(text_field=>\"hello\")).unwrap();\n        writer.add_document(doc!(text_field=>\"happy tax\")).unwrap();\n\n        writer.commit().unwrap();\n        let query_parser = QueryParser::for_index(&index, Vec::new());\n        let query = query_parser\n            .parse_query(r#\"+text:hello +text:\"happy tax\"\"#)\n            .unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let c = searcher.search(&*query, &Count).unwrap();\n        assert_eq!(c, 1);\n    }\n}\n"
  },
  {
    "path": "src/query/mod.rs",
    "content": "mod all_query;\nmod automaton_weight;\nmod bitset;\nmod bm25;\nmod boolean_query;\nmod boost_query;\nmod const_score_query;\nmod disjunction;\nmod disjunction_max_query;\nmod empty_query;\nmod exclude;\nmod exist_query;\nmod explanation;\nmod fuzzy_query;\nmod intersection;\nmod more_like_this;\nmod phrase_prefix_query;\nmod phrase_query;\nmod query;\nmod query_parser;\nmod range_query;\nmod regex_query;\nmod reqopt_scorer;\nmod scorer;\nmod set_query;\nmod size_hint;\nmod term_query;\nmod union;\nmod weight;\n\n#[cfg(test)]\nmod vec_docset;\n\npub(crate) mod score_combiner;\npub use query_grammar::Occur;\n\npub use self::all_query::{AllQuery, AllScorer, AllWeight};\npub use self::automaton_weight::AutomatonWeight;\npub use self::bitset::BitSetDocSet;\npub use self::bm25::{Bm25StatisticsProvider, Bm25Weight};\npub use self::boolean_query::{BooleanQuery, BooleanWeight};\npub use self::boost_query::{BoostQuery, BoostWeight};\npub use self::const_score_query::{ConstScoreQuery, ConstScorer};\npub use self::disjunction_max_query::DisjunctionMaxQuery;\npub use self::empty_query::{EmptyQuery, EmptyScorer, EmptyWeight};\npub use self::exclude::{Exclude, ExclusionSet};\npub use self::exist_query::ExistsQuery;\npub use self::explanation::Explanation;\n#[cfg(test)]\npub(crate) use self::fuzzy_query::DfaWrapper;\npub use self::fuzzy_query::FuzzyTermQuery;\npub use self::intersection::{intersect_scorers, Intersection};\npub use self::more_like_this::{MoreLikeThisQuery, MoreLikeThisQueryBuilder};\npub use self::phrase_prefix_query::PhrasePrefixQuery;\npub use self::phrase_query::regex_phrase_query::{wildcard_query_to_regex_str, RegexPhraseQuery};\npub use self::phrase_query::PhraseQuery;\npub use self::query::{EnableScoring, Query, QueryClone};\npub use self::query_parser::{QueryParser, QueryParserError};\npub use self::range_query::*;\npub use self::regex_query::RegexQuery;\npub use self::reqopt_scorer::RequiredOptionalScorer;\npub use self::score_combiner::{DisjunctionMaxCombiner, ScoreCombiner, SumCombiner};\npub use self::scorer::Scorer;\npub use self::set_query::TermSetQuery;\npub use self::term_query::TermQuery;\npub use self::union::BufferedUnionScorer;\n#[cfg(test)]\npub use self::vec_docset::VecDocSet;\npub use self::weight::Weight;\n\n#[cfg(test)]\nmod tests {\n    use crate::collector::TopDocs;\n    use crate::query::phrase_query::tests::create_index;\n    use crate::query::QueryParser;\n    use crate::schema::{Schema, TEXT};\n    use crate::{DocAddress, Index, Term};\n\n    #[test]\n    pub fn test_mixed_intersection_and_union() -> crate::Result<()> {\n        let index = create_index(&[\"a b\", \"a c\", \"a b c\", \"b\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n\n        let do_search = |term: &str| {\n            let query = QueryParser::for_index(&index, vec![text_field])\n                .parse_query(term)\n                .unwrap();\n            let top_docs: Vec<(f32, DocAddress)> = searcher\n                .search(&query, &TopDocs::with_limit(10).order_by_score())\n                .unwrap();\n\n            top_docs.iter().map(|el| el.1.doc_id).collect::<Vec<_>>()\n        };\n\n        assert_eq!(do_search(\"a AND b\"), vec![0, 2]);\n        assert_eq!(do_search(\"(a OR b) AND C\"), vec![2, 1]);\n        // The intersection code has special code for more than 2 intersections\n        // left, right + others\n        // The will place the union in the \"others\" insersection to that seek_into_the_danger_zone\n        // is called\n        assert_eq!(\n            do_search(\"(a OR b) AND (c OR a) AND (b OR c)\"),\n            vec![2, 1, 0]\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_mixed_intersection_and_union_with_skip() -> crate::Result<()> {\n        // Test 4096 skip in BufferedUnionScorer\n        let mut data: Vec<&str> = Vec::new();\n        data.push(\"a b\");\n        let zz_data = vec![\"z z\"; 5000];\n        data.extend_from_slice(&zz_data);\n        data.extend_from_slice(&[\"a c\"]);\n        data.extend_from_slice(&zz_data);\n        data.extend_from_slice(&[\"a b c\", \"b\"]);\n        let index = create_index(&data)?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n\n        let do_search = |term: &str| {\n            let query = QueryParser::for_index(&index, vec![text_field])\n                .parse_query(term)\n                .unwrap();\n            let top_docs: Vec<(f32, DocAddress)> = searcher\n                .search(&query, &TopDocs::with_limit(10).order_by_score())\n                .unwrap();\n\n            top_docs.iter().map(|el| el.1.doc_id).collect::<Vec<_>>()\n        };\n\n        assert_eq!(do_search(\"a AND b\"), vec![0, 10002]);\n        assert_eq!(do_search(\"(a OR b) AND C\"), vec![10002, 5001]);\n        // The intersection code has special code for more than 2 intersections\n        // left, right + others\n        // The will place the union in the \"others\" insersection to that seek_into_the_danger_zone\n        // is called\n        assert_eq!(\n            do_search(\"(a OR b) AND (c OR a) AND (b OR c)\"),\n            vec![10002, 5001, 0]\n        );\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_query_terms() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let term_b = Term::from_field_text(text_field, \"b\");\n        {\n            let query = query_parser.parse_query(\"a\").unwrap();\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, pos| terms.push((term, pos)));\n            assert_eq!(vec![(&term_a, false)], terms);\n        }\n        {\n            let query = query_parser.parse_query(\"a b\").unwrap();\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, pos| terms.push((term, pos)));\n            assert_eq!(vec![(&term_a, false), (&term_b, false)], terms);\n        }\n        {\n            let query = query_parser.parse_query(\"\\\"a b\\\"\").unwrap();\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, pos| terms.push((term, pos)));\n            assert_eq!(vec![(&term_a, true), (&term_b, true)], terms);\n        }\n        {\n            let query = query_parser.parse_query(\"a a a a a\").unwrap();\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, pos| terms.push((term, pos)));\n            assert_eq!(vec![(&term_a, false); 1], terms);\n        }\n        {\n            let query = query_parser.parse_query(\"a -b\").unwrap();\n            let mut terms = Vec::new();\n            query.query_terms(&mut |term, pos| terms.push((term, pos)));\n            assert_eq!(vec![(&term_a, false), (&term_b, false)], terms);\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/more_like_this/mod.rs",
    "content": "mod more_like_this;\n\n/// Module containing the different query implementations.\nmod query;\n\npub use self::more_like_this::MoreLikeThis;\npub use self::query::{MoreLikeThisQuery, MoreLikeThisQueryBuilder};\n"
  },
  {
    "path": "src/query/more_like_this/more_like_this.rs",
    "content": "use std::cmp::Reverse;\nuse std::collections::{BinaryHeap, HashMap};\n\nuse tokenizer_api::Token;\n\nuse crate::query::bm25::idf;\nuse crate::query::{BooleanQuery, BoostQuery, Occur, Query, TermQuery};\nuse crate::schema::document::{Document, Value};\nuse crate::schema::{Field, FieldType, IndexRecordOption, Term};\nuse crate::tokenizer::{FacetTokenizer, PreTokenizedStream, TokenStream, Tokenizer};\nuse crate::{DocAddress, Result, Searcher, TantivyDocument, TantivyError};\n\n#[derive(Debug, PartialEq)]\nstruct ScoreTerm {\n    pub term: Term,\n    pub score: f32,\n}\n\nimpl ScoreTerm {\n    fn new(term: Term, score: f32) -> Self {\n        Self { term, score }\n    }\n}\n\nimpl Eq for ScoreTerm {}\n\nimpl PartialOrd for ScoreTerm {\n    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl Ord for ScoreTerm {\n    fn cmp(&self, other: &Self) -> std::cmp::Ordering {\n        self.score\n            .partial_cmp(&other.score)\n            .unwrap_or(std::cmp::Ordering::Equal)\n    }\n}\n\n/// A struct used as helper to build [`MoreLikeThisQuery`](crate::query::MoreLikeThisQuery)\n/// This more-like-this implementation is inspired by the Apache Lucene\n/// and closely follows the same implementation with adaptation to Tantivy vocabulary and API.\n///\n/// [MoreLikeThis](https://github.com/apache/lucene/blob/main/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java#L147)\n/// [MoreLikeThisQuery](https://github.com/apache/lucene/blob/main/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThisQuery.java#L36)\n#[derive(Debug, Clone)]\npub struct MoreLikeThis {\n    /// Ignore words which do not occur in at least this many docs.\n    pub min_doc_frequency: Option<u64>,\n    /// Ignore words which occur in more than this many docs.\n    pub max_doc_frequency: Option<u64>,\n    /// Ignore words less frequent than this.\n    pub min_term_frequency: Option<usize>,\n    /// Don't return a query longer than this.\n    pub max_query_terms: Option<usize>,\n    /// Ignore words if less than this length.\n    pub min_word_length: Option<usize>,\n    /// Ignore words if greater than this length.\n    pub max_word_length: Option<usize>,\n    /// Boost factor to use when boosting the terms\n    pub boost_factor: Option<f32>,\n    /// Current set of stop words.\n    pub stop_words: Vec<String>,\n}\n\nimpl Default for MoreLikeThis {\n    fn default() -> Self {\n        Self {\n            min_doc_frequency: Some(5),\n            max_doc_frequency: None,\n            min_term_frequency: Some(2),\n            max_query_terms: Some(25),\n            min_word_length: None,\n            max_word_length: None,\n            boost_factor: Some(1.0),\n            stop_words: vec![],\n        }\n    }\n}\n\nimpl MoreLikeThis {\n    /// Creates a [`BooleanQuery`] using a document address to collect\n    /// the top stored field values.\n    pub fn query_with_document(\n        &self,\n        searcher: &Searcher,\n        doc_address: DocAddress,\n    ) -> Result<BooleanQuery> {\n        let score_terms = self.retrieve_terms_from_doc_address(searcher, doc_address)?;\n        let query = self.create_query(score_terms);\n        Ok(query)\n    }\n\n    /// Creates a [`BooleanQuery`] using a set of field values.\n    pub fn query_with_document_fields<'a, V: Value<'a>>(\n        &self,\n        searcher: &Searcher,\n        doc_fields: &[(Field, Vec<V>)],\n    ) -> Result<BooleanQuery> {\n        let score_terms = self.retrieve_terms_from_doc_fields(searcher, doc_fields)?;\n        let query = self.create_query(score_terms);\n        Ok(query)\n    }\n\n    /// Creates a [`BooleanQuery`] from an ascendingly sorted list of ScoreTerm\n    /// This will map the list of ScoreTerm to a list of [`TermQuery`]  and compose a\n    /// BooleanQuery using that list as sub queries.\n    fn create_query(&self, mut score_terms: Vec<ScoreTerm>) -> BooleanQuery {\n        score_terms.sort_by(|left_ts, right_ts| right_ts.cmp(left_ts));\n        let best_score = score_terms.first().map_or(1f32, |x| x.score);\n        let mut queries = Vec::new();\n\n        for ScoreTerm { term, score } in score_terms {\n            let mut query: Box<dyn Query> =\n                Box::new(TermQuery::new(term, IndexRecordOption::Basic));\n            if let Some(factor) = self.boost_factor {\n                query = Box::new(BoostQuery::new(query, score * factor / best_score));\n            }\n            queries.push((Occur::Should, query));\n        }\n        BooleanQuery::from(queries)\n    }\n\n    /// Finds terms for a more-like-this query.\n    /// doc_address is the address of document from which to find terms.\n    fn retrieve_terms_from_doc_address(\n        &self,\n        searcher: &Searcher,\n        doc_address: DocAddress,\n    ) -> Result<Vec<ScoreTerm>> {\n        let doc = searcher.doc::<TantivyDocument>(doc_address)?;\n\n        let field_to_values = doc.get_sorted_field_values();\n        self.retrieve_terms_from_doc_fields(searcher, &field_to_values)\n    }\n\n    /// Finds terms for a more-like-this query.\n    /// field_to_field_values is a mapping from field to possible values of that field.\n    fn retrieve_terms_from_doc_fields<'a, V: Value<'a>>(\n        &self,\n        searcher: &Searcher,\n        field_to_values: &[(Field, Vec<V>)],\n    ) -> Result<Vec<ScoreTerm>> {\n        if field_to_values.is_empty() {\n            return Err(TantivyError::InvalidArgument(\n                \"Cannot create more like this query on empty field values. The document may not \\\n                 have stored fields\"\n                    .to_string(),\n            ));\n        }\n        let mut field_to_term_freq_map = HashMap::new();\n        for (field, values) in field_to_values {\n            self.add_term_frequencies(searcher, *field, values, &mut field_to_term_freq_map)?;\n        }\n        self.create_score_term(searcher, field_to_term_freq_map)\n    }\n\n    /// Computes the frequency of values for a field while updating the term frequencies\n    /// Note: A FieldValue can be made up of multiple terms.\n    /// We are interested in extracting terms within FieldValue\n    fn add_term_frequencies<'a, V: Value<'a>>(\n        &self,\n        searcher: &Searcher,\n        field: Field,\n        values: &[V],\n        term_frequencies: &mut HashMap<Term, usize>,\n    ) -> Result<()> {\n        let schema = searcher.schema();\n        let tokenizer_manager = searcher.index().tokenizers();\n\n        let field_entry = schema.get_field_entry(field);\n        if !field_entry.is_indexed() {\n            return Ok(());\n        }\n\n        // extract the raw value, possibly tokenizing & filtering to update the term frequency map\n        match field_entry.field_type() {\n            FieldType::Facet(_) => {\n                let facets: Vec<&str> = values\n                    .iter()\n                    .map(|value| {\n                        value.as_facet().ok_or_else(|| {\n                            TantivyError::InvalidArgument(\"invalid field value\".to_string())\n                        })\n                    })\n                    .collect::<Result<Vec<_>>>()?;\n                for fake_str in facets {\n                    FacetTokenizer::default()\n                        .token_stream(fake_str)\n                        .process(&mut |token| {\n                            if self.is_noise_word(token.text.clone()) {\n                                let term = Term::from_field_text(field, &token.text);\n                                *term_frequencies.entry(term).or_insert(0) += 1;\n                            }\n                        });\n                }\n            }\n            FieldType::Str(text_options) => {\n                let mut tokenizer_opt = text_options\n                    .get_indexing_options()\n                    .map(|options| options.tokenizer())\n                    .and_then(|tokenizer_name| tokenizer_manager.get(tokenizer_name));\n\n                let sink = &mut |token: &Token| {\n                    if !self.is_noise_word(token.text.clone()) {\n                        let term = Term::from_field_text(field, &token.text);\n                        *term_frequencies.entry(term).or_insert(0) += 1;\n                    }\n                };\n\n                // TODO: Validate these changed align with the HEAD branch.\n                for value in values {\n                    if let Some(text) = value.as_str() {\n                        let tokenizer = match &mut tokenizer_opt {\n                            None => continue,\n                            Some(tokenizer) => tokenizer,\n                        };\n\n                        let mut token_stream = tokenizer.token_stream(text);\n                        token_stream.process(sink);\n                    } else if let Some(tok_str) = value.as_pre_tokenized_text() {\n                        let mut token_stream = PreTokenizedStream::from(*tok_str.clone());\n                        token_stream.process(sink);\n                    }\n                }\n            }\n            FieldType::U64(_) => {\n                for value in values {\n                    let val = value.as_u64().ok_or_else(|| {\n                        TantivyError::InvalidArgument(\"invalid value\".to_string())\n                    })?;\n                    if !self.is_noise_word(val.to_string()) {\n                        let term = Term::from_field_u64(field, val);\n                        *term_frequencies.entry(term).or_insert(0) += 1;\n                    }\n                }\n            }\n            FieldType::Date(_) => {\n                for value in values {\n                    let timestamp = value.as_datetime().ok_or_else(|| {\n                        TantivyError::InvalidArgument(\"invalid value\".to_string())\n                    })?;\n                    let term = Term::from_field_date_for_search(field, timestamp);\n                    *term_frequencies.entry(term).or_insert(0) += 1;\n                }\n            }\n            FieldType::I64(_) => {\n                for value in values {\n                    let val = value.as_i64().ok_or_else(|| {\n                        TantivyError::InvalidArgument(\"invalid value\".to_string())\n                    })?;\n                    if !self.is_noise_word(val.to_string()) {\n                        let term = Term::from_field_i64(field, val);\n                        *term_frequencies.entry(term).or_insert(0) += 1;\n                    }\n                }\n            }\n            FieldType::F64(_) => {\n                for value in values {\n                    let val = value.as_f64().ok_or_else(|| {\n                        TantivyError::InvalidArgument(\"invalid value\".to_string())\n                    })?;\n                    if !self.is_noise_word(val.to_string()) {\n                        let term = Term::from_field_f64(field, val);\n                        *term_frequencies.entry(term).or_insert(0) += 1;\n                    }\n                }\n            }\n            _ => {}\n        }\n        Ok(())\n    }\n\n    /// Determines if the term is likely to be of interest based on \"more-like-this\" settings\n    fn is_noise_word(&self, word: String) -> bool {\n        let word_length = word.len();\n        if word_length == 0 {\n            return true;\n        }\n        if self\n            .min_word_length\n            .map(|min| word_length < min)\n            .unwrap_or(false)\n        {\n            return true;\n        }\n        if self\n            .max_word_length\n            .map(|max| word_length > max)\n            .unwrap_or(false)\n        {\n            return true;\n        }\n        self.stop_words.contains(&word)\n    }\n\n    /// Computes the score for each term while ignoring not useful terms\n    fn create_score_term(\n        &self,\n        searcher: &Searcher,\n        per_field_term_frequencies: HashMap<Term, usize>,\n    ) -> Result<Vec<ScoreTerm>> {\n        let mut score_terms: BinaryHeap<Reverse<ScoreTerm>> = BinaryHeap::new();\n        let num_docs = searcher\n            .segment_readers()\n            .iter()\n            .map(|segment_reader| segment_reader.num_docs() as u64)\n            .sum::<u64>();\n\n        for (term, term_frequency) in per_field_term_frequencies.iter() {\n            // ignore terms with less than min_term_frequency\n            if self\n                .min_term_frequency\n                .map(|min_term_frequency| *term_frequency < min_term_frequency)\n                .unwrap_or(false)\n            {\n                continue;\n            }\n\n            let doc_freq = searcher.doc_freq(term)?;\n\n            // ignore terms with less than min_doc_frequency\n            if self\n                .min_doc_frequency\n                .map(|min_doc_frequency| doc_freq < min_doc_frequency)\n                .unwrap_or(false)\n            {\n                continue;\n            }\n\n            // ignore terms with more than max_doc_frequency\n            if self\n                .max_doc_frequency\n                .map(|max_doc_frequency| doc_freq > max_doc_frequency)\n                .unwrap_or(false)\n            {\n                continue;\n            }\n\n            // ignore terms with zero frequency\n            if doc_freq == 0 {\n                continue;\n            }\n\n            // compute similarity & score\n            let idf = idf(doc_freq, num_docs);\n            let score = (*term_frequency as f32) * idf;\n            if let Some(limit) = self.max_query_terms {\n                if score_terms.len() > limit {\n                    // update the least significant term\n                    let least_significant_term_score = score_terms.peek().unwrap().0.score;\n                    if least_significant_term_score < score {\n                        score_terms.peek_mut().unwrap().0 = ScoreTerm::new(term.clone(), score);\n                    }\n                } else {\n                    score_terms.push(Reverse(ScoreTerm::new(term.clone(), score)));\n                }\n            } else {\n                score_terms.push(Reverse(ScoreTerm::new(term.clone(), score)));\n            }\n        }\n\n        let score_terms_vec: Vec<ScoreTerm> = score_terms\n            .into_iter()\n            .map(|reverse_score_term| reverse_score_term.0)\n            .collect();\n\n        Ok(score_terms_vec)\n    }\n}\n"
  },
  {
    "path": "src/query/more_like_this/query.rs",
    "content": "use std::fmt::Debug;\n\nuse super::MoreLikeThis;\nuse crate::query::{EnableScoring, Query, Weight};\nuse crate::schema::{Field, OwnedValue};\nuse crate::DocAddress;\n\n/// A query that matches all of the documents similar to a document\n/// or a set of field values provided.\n///\n/// # Examples\n///\n/// ```\n/// use tantivy::DocAddress;\n/// use tantivy::query::MoreLikeThisQuery;\n///\n/// let query = MoreLikeThisQuery::builder()\n///     .with_min_doc_frequency(1)\n///     .with_max_doc_frequency(10)\n///     .with_min_term_frequency(1)\n///     .with_min_word_length(2)\n///     .with_max_word_length(5)\n///     .with_boost_factor(1.0)\n///     .with_stop_words(vec![\"for\".to_string()])\n///     .with_document(DocAddress::new(2, 1));\n/// ```\n#[derive(Debug, Clone)]\npub struct MoreLikeThisQuery {\n    mlt: MoreLikeThis,\n    target: TargetDocument,\n}\n\n#[derive(Debug, Clone, PartialEq)]\nenum TargetDocument {\n    DocumentAddress(DocAddress),\n    DocumentFields(Vec<(Field, Vec<OwnedValue>)>),\n}\n\nimpl MoreLikeThisQuery {\n    /// Creates a new builder.\n    pub fn builder() -> MoreLikeThisQueryBuilder {\n        MoreLikeThisQueryBuilder::default()\n    }\n}\n\nimpl Query for MoreLikeThisQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let searcher = match enable_scoring {\n            EnableScoring::Enabled { searcher, .. } => searcher,\n            EnableScoring::Disabled { .. } => {\n                let err = \"MoreLikeThisQuery requires to enable scoring.\".to_string();\n                return Err(crate::TantivyError::InvalidArgument(err));\n            }\n        };\n        match &self.target {\n            TargetDocument::DocumentAddress(doc_address) => self\n                .mlt\n                .query_with_document(searcher, *doc_address)?\n                .weight(enable_scoring),\n            TargetDocument::DocumentFields(doc_fields) => {\n                let values = doc_fields\n                    .iter()\n                    .map(|(field, values)| (*field, values.iter().collect::<Vec<&OwnedValue>>()))\n                    .collect::<Vec<_>>();\n\n                self.mlt\n                    .query_with_document_fields(searcher, &values)?\n                    .weight(enable_scoring)\n            }\n        }\n    }\n}\n\n/// The builder for more-like-this query\n#[derive(Debug, Clone, Default)]\npub struct MoreLikeThisQueryBuilder {\n    mlt: MoreLikeThis,\n}\n\nimpl MoreLikeThisQueryBuilder {\n    /// Sets the minimum document frequency.\n    ///\n    /// The resulting query will ignore words which do not occur\n    /// in at least this many docs.\n    #[must_use]\n    pub fn with_min_doc_frequency(mut self, value: u64) -> Self {\n        self.mlt.min_doc_frequency = Some(value);\n        self\n    }\n\n    /// Sets the maximum document frequency.\n    ///\n    /// The resulting query will ignore words which occur\n    /// in more than this many docs.\n    #[must_use]\n    pub fn with_max_doc_frequency(mut self, value: u64) -> Self {\n        self.mlt.max_doc_frequency = Some(value);\n        self\n    }\n\n    /// Sets the minimum term frequency.\n    ///\n    /// The resulting query will ignore words less\n    /// frequent that this number.\n    #[must_use]\n    pub fn with_min_term_frequency(mut self, value: usize) -> Self {\n        self.mlt.min_term_frequency = Some(value);\n        self\n    }\n\n    /// Sets the maximum query terms.\n    ///\n    /// The resulting query will not return a query with more clause than this.\n    #[must_use]\n    pub fn with_max_query_terms(mut self, value: usize) -> Self {\n        self.mlt.max_query_terms = Some(value);\n        self\n    }\n\n    /// Sets the minimum word length.\n    ///\n    /// The resulting query will ignore words shorter than this length.\n    #[must_use]\n    pub fn with_min_word_length(mut self, value: usize) -> Self {\n        self.mlt.min_word_length = Some(value);\n        self\n    }\n\n    /// Sets the maximum word length.\n    ///\n    /// The resulting query will ignore words longer than this length.\n    #[must_use]\n    pub fn with_max_word_length(mut self, value: usize) -> Self {\n        self.mlt.max_word_length = Some(value);\n        self\n    }\n\n    /// Sets the boost factor\n    ///\n    /// The boost factor used by the resulting query for boosting terms.\n    #[must_use]\n    pub fn with_boost_factor(mut self, value: f32) -> Self {\n        self.mlt.boost_factor = Some(value);\n        self\n    }\n\n    /// Sets the set of stop words\n    ///\n    /// The resulting query will ignore these set of words.\n    #[must_use]\n    pub fn with_stop_words(mut self, value: Vec<String>) -> Self {\n        self.mlt.stop_words = value;\n        self\n    }\n\n    /// Sets the document address\n    /// Returns the constructed [`MoreLikeThisQuery`]\n    ///\n    /// This document will be used to collect field values, extract frequent terms\n    /// needed for composing the query.\n    ///\n    /// Note that field values will only be collected from stored fields in the index.\n    /// You can construct your own field values from any source.\n    pub fn with_document(self, doc_address: DocAddress) -> MoreLikeThisQuery {\n        MoreLikeThisQuery {\n            mlt: self.mlt,\n            target: TargetDocument::DocumentAddress(doc_address),\n        }\n    }\n\n    /// Sets the document fields\n    /// Returns the constructed [`MoreLikeThisQuery`]\n    ///\n    /// This represents the list field values possibly collected from multiple documents\n    /// that will be used to compose the resulting query.\n    /// This interface is meant to be used when you want to provide your own set of fields\n    /// not necessarily from a specific document.\n    pub fn with_document_fields(\n        self,\n        doc_fields: Vec<(Field, Vec<OwnedValue>)>,\n    ) -> MoreLikeThisQuery {\n        MoreLikeThisQuery {\n            mlt: self.mlt,\n            target: TargetDocument::DocumentFields(doc_fields),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::{MoreLikeThisQuery, TargetDocument};\n    use crate::collector::TopDocs;\n    use crate::schema::{Schema, STORED, TEXT};\n    use crate::{DocAddress, Index, IndexWriter};\n\n    fn create_test_index() -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(\"title\", TEXT);\n        let body = schema_builder.add_text_field(\"body\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc!(title => \"aaa\", body => \"the old man and the sea\"))?;\n        index_writer.add_document(doc!(title => \"bbb\", body => \"an old man sailing on the sea\"))?;\n        index_writer.add_document(doc!(title => \"ccc\", body=> \"send this message to alice\"))?;\n        index_writer.add_document(doc!(title => \"ddd\", body=> \"a lady was riding and old bike\"))?;\n        index_writer.add_document(doc!(title => \"eee\", body=> \"Yes, my lady.\"))?;\n        index_writer.commit()?;\n        Ok(index)\n    }\n\n    #[test]\n    fn test_more_like_this_query_builder() {\n        // default settings\n        let query = MoreLikeThisQuery::builder().with_document_fields(vec![]);\n\n        assert_eq!(query.mlt.min_doc_frequency, Some(5));\n        assert_eq!(query.mlt.max_doc_frequency, None);\n        assert_eq!(query.mlt.min_term_frequency, Some(2));\n        assert_eq!(query.mlt.max_query_terms, Some(25));\n        assert_eq!(query.mlt.min_word_length, None);\n        assert_eq!(query.mlt.max_word_length, None);\n        assert_eq!(query.mlt.boost_factor, Some(1.0));\n        assert_eq!(query.mlt.stop_words, Vec::<String>::new());\n        assert_eq!(query.target, TargetDocument::DocumentFields(vec![]));\n\n        // custom settings\n        let query = MoreLikeThisQuery::builder()\n            .with_min_doc_frequency(2)\n            .with_max_doc_frequency(5)\n            .with_min_term_frequency(2)\n            .with_min_word_length(2)\n            .with_max_word_length(4)\n            .with_boost_factor(0.5)\n            .with_stop_words(vec![\"all\".to_string(), \"for\".to_string()])\n            .with_document(DocAddress::new(1, 2));\n\n        assert_eq!(query.mlt.min_doc_frequency, Some(2));\n        assert_eq!(query.mlt.max_doc_frequency, Some(5));\n        assert_eq!(query.mlt.min_term_frequency, Some(2));\n        assert_eq!(query.mlt.min_word_length, Some(2));\n        assert_eq!(query.mlt.max_word_length, Some(4));\n        assert_eq!(query.mlt.boost_factor, Some(0.5));\n        assert_eq!(\n            query.mlt.stop_words,\n            vec![\"all\".to_string(), \"for\".to_string()]\n        );\n        assert_eq!(\n            query.target,\n            TargetDocument::DocumentAddress(DocAddress::new(1, 2))\n        );\n    }\n\n    #[test]\n    fn test_more_like_this_query() -> crate::Result<()> {\n        let index = create_test_index()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // search base 1st doc with words [sea, and] skipping [old]\n        let query = MoreLikeThisQuery::builder()\n            .with_min_doc_frequency(1)\n            .with_max_doc_frequency(10)\n            .with_min_term_frequency(1)\n            .with_min_word_length(2)\n            .with_max_word_length(5)\n            .with_boost_factor(1.0)\n            .with_stop_words(vec![\"old\".to_string()])\n            .with_document(DocAddress::new(0, 0));\n        let top_docs = searcher.search(&query, &TopDocs::with_limit(5).order_by_score())?;\n        let mut doc_ids: Vec<_> = top_docs.iter().map(|item| item.1.doc_id).collect();\n        doc_ids.sort_unstable();\n\n        assert_eq!(doc_ids.len(), 3);\n        assert_eq!(doc_ids, vec![0, 1, 3]);\n\n        // search base 5th doc with words [lady]\n        let query = MoreLikeThisQuery::builder()\n            .with_min_doc_frequency(1)\n            .with_max_doc_frequency(10)\n            .with_min_term_frequency(1)\n            .with_min_word_length(2)\n            .with_max_word_length(5)\n            .with_boost_factor(1.0)\n            .with_document(DocAddress::new(0, 4));\n        let top_docs = searcher.search(&query, &TopDocs::with_limit(5).order_by_score())?;\n        let mut doc_ids: Vec<_> = top_docs.iter().map(|item| item.1.doc_id).collect();\n        doc_ids.sort_unstable();\n\n        assert_eq!(doc_ids.len(), 2);\n        assert_eq!(doc_ids, vec![3, 4]);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_prefix_query/mod.rs",
    "content": "mod phrase_prefix_query;\nmod phrase_prefix_scorer;\nmod phrase_prefix_weight;\n\npub use phrase_prefix_query::PhrasePrefixQuery;\npub use phrase_prefix_scorer::PhrasePrefixScorer;\npub use phrase_prefix_weight::PhrasePrefixWeight;\n\npub(crate) fn prefix_end(prefix_start: &[u8]) -> Option<Vec<u8>> {\n    let mut res = prefix_start.to_owned();\n    while !res.is_empty() {\n        let end = res.len() - 1;\n        if res[end] == u8::MAX {\n            res.pop();\n        } else {\n            res[end] += 1;\n            return Some(res);\n        }\n    }\n    None\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_prefix_end() {\n        assert_eq!(prefix_end(b\"aaa\"), Some(b\"aab\".to_vec()));\n        assert_eq!(prefix_end(b\"aa\\xff\"), Some(b\"ab\".to_vec()));\n        assert_eq!(prefix_end(b\"a\\xff\\xff\"), Some(b\"b\".to_vec()));\n        assert_eq!(prefix_end(b\"\\xff\\xff\\xff\"), None);\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_prefix_query/phrase_prefix_query.rs",
    "content": "use std::ops::Bound;\n\nuse super::{prefix_end, PhrasePrefixWeight};\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::{EnableScoring, InvertedIndexRangeWeight, Query, Weight};\nuse crate::schema::{Field, IndexRecordOption, Term};\n\nconst DEFAULT_MAX_EXPANSIONS: u32 = 50;\n\n/// `PhrasePrefixQuery` matches a specific sequence of words followed by term of which only a\n/// prefix is known.\n///\n/// For instance the phrase prefix query for `\"part t\"` will match\n/// the sentence\n///\n/// **Alan just got a part time job.**\n///\n/// On the other hand it will not match the sentence.\n///\n/// **This is my favorite part of the job.**\n///\n/// Using a `PhrasePrefixQuery` on a field requires positions\n/// to be indexed for this field.\n#[derive(Clone, Debug)]\npub struct PhrasePrefixQuery {\n    field: Field,\n    phrase_terms: Vec<(usize, Term)>,\n    prefix: (usize, Term),\n    max_expansions: u32,\n}\n\nimpl PhrasePrefixQuery {\n    /// Creates a new `PhrasePrefixQuery` given a list of terms.\n    ///\n    /// There must be at least two terms, and all terms\n    /// must belong to the same field.\n    /// Offset for each term will be same as index in the Vector\n    /// The last Term is a prefix and not a full value\n    pub fn new(terms: Vec<Term>) -> PhrasePrefixQuery {\n        let terms_with_offset = terms.into_iter().enumerate().collect();\n        PhrasePrefixQuery::new_with_offset(terms_with_offset)\n    }\n\n    /// Creates a new `PhrasePrefixQuery` given a list of terms and their offsets.\n    ///\n    /// Can be used to provide custom offset for each term.\n    pub fn new_with_offset(mut terms: Vec<(usize, Term)>) -> PhrasePrefixQuery {\n        assert!(\n            !terms.is_empty(),\n            \"A phrase prefix query is required to have at least one term.\"\n        );\n        terms.sort_by_key(|&(offset, _)| offset);\n        let field = terms[0].1.field();\n        assert!(\n            terms[1..].iter().all(|term| term.1.field() == field),\n            \"All terms from a phrase query must belong to the same field\"\n        );\n        PhrasePrefixQuery {\n            field,\n            prefix: terms.pop().unwrap(),\n            phrase_terms: terms,\n            max_expansions: DEFAULT_MAX_EXPANSIONS,\n        }\n    }\n\n    /// Maximum number of terms to which the last provided term will expand.\n    pub fn set_max_expansions(&mut self, value: u32) {\n        self.max_expansions = value;\n    }\n\n    /// The [`Field`] this `PhrasePrefixQuery` is targeting.\n    pub fn field(&self) -> Field {\n        self.field\n    }\n\n    /// `Term`s in the phrase without the associated offsets.\n    pub fn phrase_terms(&self) -> Vec<Term> {\n        // TODO should we include the last term too?\n        self.phrase_terms\n            .iter()\n            .map(|(_, term)| term.clone())\n            .collect::<Vec<Term>>()\n    }\n\n    /// Returns the [`PhrasePrefixWeight`] for the given phrase query given a specific `searcher`.\n    ///\n    /// This function is the same as [`Query::weight()`] except it returns\n    /// a specialized type [`PhraseQueryWeight`] instead of a Boxed trait.\n    /// If the query was only one term long, this returns `None` whereas [`Query::weight`]\n    /// returns a boxed [`RangeWeight`]\n    pub(crate) fn phrase_prefix_query_weight(\n        &self,\n        enable_scoring: EnableScoring<'_>,\n    ) -> crate::Result<Option<PhrasePrefixWeight>> {\n        if self.phrase_terms.is_empty() {\n            return Ok(None);\n        }\n        let schema = enable_scoring.schema();\n        let field_entry = schema.get_field_entry(self.field);\n        let has_positions = field_entry\n            .field_type()\n            .get_index_record_option()\n            .map(IndexRecordOption::has_positions)\n            .unwrap_or(false);\n        if !has_positions {\n            let field_name = field_entry.name();\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"Applied phrase query on field {field_name:?}, which does not have positions \\\n                 indexed\"\n            )));\n        }\n        let terms = self.phrase_terms();\n        let bm25_weight_opt = match enable_scoring {\n            EnableScoring::Enabled { searcher, .. } => {\n                Some(Bm25Weight::for_terms(searcher, &terms)?)\n            }\n            EnableScoring::Disabled { .. } => None,\n        };\n        let weight = PhrasePrefixWeight::new(\n            self.phrase_terms.clone(),\n            self.prefix.clone(),\n            bm25_weight_opt,\n            self.max_expansions,\n        );\n        Ok(Some(weight))\n    }\n}\n\nimpl Query for PhrasePrefixQuery {\n    /// Create the weight associated with a query.\n    ///\n    /// See [`Weight`].\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        if let Some(phrase_weight) = self.phrase_prefix_query_weight(enable_scoring)? {\n            Ok(Box::new(phrase_weight))\n        } else {\n            // There are no prefix. Let's just match the suffix.\n            let end_term =\n                if let Some(end_value) = prefix_end(self.prefix.1.serialized_value_bytes()) {\n                    let mut end_term = Term::with_capacity(end_value.len());\n                    end_term.set_field_and_type(self.field, self.prefix.1.typ());\n                    end_term.append_bytes(&end_value);\n                    Bound::Excluded(end_term)\n                } else {\n                    Bound::Unbounded\n                };\n\n            let lower_bound = Bound::Included(self.prefix.1.clone());\n            let upper_bound = end_term;\n\n            Ok(Box::new(InvertedIndexRangeWeight::new(\n                self.field,\n                &lower_bound,\n                &upper_bound,\n                Some(self.max_expansions as u64),\n            )))\n        }\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        for (_, term) in &self.phrase_terms {\n            visitor(term, true);\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_prefix_query/phrase_prefix_scorer.rs",
    "content": "use crate::docset::{DocSet, SeekDangerResult, TERMINATED};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::postings::Postings;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::phrase_query::{intersection_count, PhraseScorer};\nuse crate::query::Scorer;\nuse crate::{DocId, Score};\n\n// MultiPrefix is the larger variant, and also the one we expect most often. PhraseScorer is > 1kB\n// though, it would be interesting to slim it down if possible.\n#[expect(clippy::large_enum_variant)]\nenum PhraseKind<TPostings: Postings> {\n    SinglePrefix {\n        position_offset: u32,\n        postings: TPostings,\n        positions: Vec<u32>,\n    },\n    MultiPrefix(PhraseScorer<TPostings>),\n}\n\nimpl<TPostings: Postings> PhraseKind<TPostings> {\n    fn get_intersection(&mut self) -> &[u32] {\n        match self {\n            PhraseKind::SinglePrefix {\n                position_offset,\n                postings,\n                positions,\n            } => {\n                if positions.is_empty() {\n                    postings.positions_with_offset(*position_offset, positions);\n                }\n                positions\n            }\n            PhraseKind::MultiPrefix(postings) => postings.get_intersection(),\n        }\n    }\n}\n\nimpl<TPostings: Postings> DocSet for PhraseKind<TPostings> {\n    fn advance(&mut self) -> DocId {\n        match self {\n            PhraseKind::SinglePrefix {\n                postings,\n                positions,\n                ..\n            } => {\n                positions.clear();\n                postings.advance()\n            }\n            PhraseKind::MultiPrefix(postings) => postings.advance(),\n        }\n    }\n\n    fn doc(&self) -> DocId {\n        match self {\n            PhraseKind::SinglePrefix { postings, .. } => postings.doc(),\n            PhraseKind::MultiPrefix(postings) => postings.doc(),\n        }\n    }\n\n    fn size_hint(&self) -> u32 {\n        match self {\n            PhraseKind::SinglePrefix { postings, .. } => postings.size_hint(),\n            PhraseKind::MultiPrefix(postings) => postings.size_hint(),\n        }\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        match self {\n            PhraseKind::SinglePrefix {\n                postings,\n                positions,\n                ..\n            } => {\n                positions.clear();\n                postings.seek(target)\n            }\n            PhraseKind::MultiPrefix(postings) => postings.seek(target),\n        }\n    }\n}\n\nimpl<TPostings: Postings> Scorer for PhraseKind<TPostings> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        match self {\n            PhraseKind::SinglePrefix { positions, .. } => {\n                if positions.is_empty() {\n                    0.0\n                } else {\n                    1.0\n                }\n            }\n            PhraseKind::MultiPrefix(postings) => postings.score(),\n        }\n    }\n}\n\npub struct PhrasePrefixScorer<TPostings: Postings> {\n    phrase_scorer: PhraseKind<TPostings>,\n    suffixes: Vec<TPostings>,\n    suffix_offset: u32,\n    phrase_count: u32,\n    suffix_position_buffer: Vec<u32>,\n}\n\nimpl<TPostings: Postings> PhrasePrefixScorer<TPostings> {\n    // If similarity_weight is None, then scoring is disabled.\n    pub fn new(\n        mut term_postings: Vec<(usize, TPostings)>,\n        similarity_weight_opt: Option<Bm25Weight>,\n        fieldnorm_reader: FieldNormReader,\n        suffixes: Vec<TPostings>,\n        suffix_pos: usize,\n    ) -> PhrasePrefixScorer<TPostings> {\n        // correct indices so we can merge with our suffix term the PhraseScorer doesn't know about\n        let max_offset = term_postings\n            .iter()\n            .map(|(pos, _)| *pos)\n            .chain(std::iter::once(suffix_pos))\n            .max()\n            .unwrap();\n\n        let phrase_scorer = if term_postings.len() > 1 {\n            PhraseKind::MultiPrefix(PhraseScorer::new_with_offset(\n                term_postings,\n                similarity_weight_opt,\n                fieldnorm_reader,\n                0,\n                1,\n            ))\n        } else {\n            let (pos, postings) = term_postings\n                .pop()\n                .expect(\"PhrasePrefixScorer must have at least two terms\");\n            let offset = suffix_pos - pos;\n            PhraseKind::SinglePrefix {\n                position_offset: offset as u32,\n                postings,\n                positions: Vec::with_capacity(100),\n            }\n        };\n        let mut phrase_prefix_scorer = PhrasePrefixScorer {\n            phrase_scorer,\n            suffixes,\n            suffix_offset: (max_offset - suffix_pos) as u32,\n            phrase_count: 0,\n            suffix_position_buffer: Vec::with_capacity(100),\n        };\n        if phrase_prefix_scorer.doc() != TERMINATED && !phrase_prefix_scorer.matches_prefix() {\n            phrase_prefix_scorer.advance();\n        }\n        phrase_prefix_scorer\n    }\n\n    pub fn phrase_count(&self) -> u32 {\n        self.phrase_count\n    }\n\n    fn matches_prefix(&mut self) -> bool {\n        let mut count = 0;\n        let current_doc = self.doc();\n        let pos_matching = self.phrase_scorer.get_intersection();\n        for suffix in &mut self.suffixes {\n            if suffix.doc() > current_doc {\n                continue;\n            }\n            let doc = suffix.seek(current_doc);\n            if doc == current_doc {\n                suffix.positions_with_offset(self.suffix_offset, &mut self.suffix_position_buffer);\n                count += intersection_count(pos_matching, &self.suffix_position_buffer);\n            }\n        }\n        self.phrase_count = count as u32;\n        count != 0\n    }\n}\n\nimpl<TPostings: Postings> DocSet for PhrasePrefixScorer<TPostings> {\n    fn advance(&mut self) -> DocId {\n        loop {\n            let doc = self.phrase_scorer.advance();\n            if doc == TERMINATED || self.matches_prefix() {\n                return doc;\n            }\n        }\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        let doc = self.phrase_scorer.seek(target);\n        if doc == TERMINATED || self.matches_prefix() {\n            return doc;\n        }\n        self.advance()\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        let seek_res = self.phrase_scorer.seek_danger(target);\n        if seek_res != SeekDangerResult::Found {\n            return seek_res;\n        }\n        // The intersection matched. Now let's see if we match the prefix.\n        if self.matches_prefix() {\n            SeekDangerResult::Found\n        } else {\n            SeekDangerResult::SeekLowerBound(target + 1)\n        }\n    }\n\n    fn doc(&self) -> DocId {\n        self.phrase_scorer.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.phrase_scorer.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        self.phrase_scorer.cost()\n    }\n}\n\nimpl<TPostings: Postings> Scorer for PhrasePrefixScorer<TPostings> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        // TODO modify score??\n        self.phrase_scorer.score()\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_prefix_query/phrase_prefix_weight.rs",
    "content": "use super::{prefix_end, PhrasePrefixScorer};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::index::SegmentReader;\nuse crate::postings::SegmentPostings;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::explanation::does_not_match;\nuse crate::query::{EmptyScorer, Explanation, Scorer, Weight};\nuse crate::schema::{IndexRecordOption, Term};\nuse crate::{DocId, DocSet, Score};\n\npub struct PhrasePrefixWeight {\n    phrase_terms: Vec<(usize, Term)>,\n    prefix: (usize, Term),\n    similarity_weight_opt: Option<Bm25Weight>,\n    max_expansions: u32,\n}\n\nimpl PhrasePrefixWeight {\n    /// Creates a new phrase weight.\n    /// If `similarity_weight_opt` is None, then scoring is disabled\n    pub fn new(\n        phrase_terms: Vec<(usize, Term)>,\n        prefix: (usize, Term),\n        similarity_weight_opt: Option<Bm25Weight>,\n        max_expansions: u32,\n    ) -> PhrasePrefixWeight {\n        PhrasePrefixWeight {\n            phrase_terms,\n            prefix,\n            similarity_weight_opt,\n            max_expansions,\n        }\n    }\n\n    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {\n        let field = self.phrase_terms[0].1.field();\n        if self.similarity_weight_opt.is_some() {\n            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {\n                return Ok(fieldnorm_reader);\n            }\n        }\n        Ok(FieldNormReader::constant(reader.max_doc(), 1))\n    }\n\n    pub(crate) fn phrase_scorer(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<Option<PhrasePrefixScorer<SegmentPostings>>> {\n        let similarity_weight_opt = self\n            .similarity_weight_opt\n            .as_ref()\n            .map(|similarity_weight| similarity_weight.boost_by(boost));\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let mut term_postings_list = Vec::new();\n        for &(offset, ref term) in &self.phrase_terms {\n            if let Some(postings) = reader\n                .inverted_index(term.field())?\n                .read_postings(term, IndexRecordOption::WithFreqsAndPositions)?\n            {\n                term_postings_list.push((offset, postings));\n            } else {\n                return Ok(None);\n            }\n        }\n\n        let inv_index = reader.inverted_index(self.prefix.1.field())?;\n        let mut stream = inv_index\n            .terms()\n            .range()\n            .ge(self.prefix.1.serialized_value_bytes());\n        if let Some(end) = prefix_end(self.prefix.1.serialized_value_bytes()) {\n            stream = stream.lt(&end);\n        }\n\n        #[cfg(feature = \"quickwit\")]\n        {\n            // We don't have this on the fst, hence  we end up needing a feature flag.\n            //\n            // This is not a problem however as we enforce the limit below too.\n            // The point of `stream.limit` is to limit the number of term dictionary\n            // blocks being downloaded.\n            stream = stream.limit(self.max_expansions as u64);\n        }\n\n        let mut stream = stream.into_stream()?;\n\n        let mut suffixes = Vec::with_capacity(self.max_expansions as usize);\n        let mut new_term = self.prefix.1.clone();\n        while stream.advance() && (suffixes.len() as u32) < self.max_expansions {\n            new_term.clear_with_type(new_term.typ());\n            new_term.append_bytes(stream.key());\n            if reader.has_deletes() {\n                if let Some(postings) =\n                    inv_index.read_postings(&new_term, IndexRecordOption::WithFreqsAndPositions)?\n                {\n                    suffixes.push(postings);\n                }\n            } else if let Some(postings) =\n                inv_index.read_postings(&new_term, IndexRecordOption::WithFreqsAndPositions)?\n            {\n                suffixes.push(postings);\n            }\n        }\n\n        Ok(Some(PhrasePrefixScorer::new(\n            term_postings_list,\n            similarity_weight_opt,\n            fieldnorm_reader,\n            suffixes,\n            self.prefix.0,\n        )))\n    }\n}\n\nimpl Weight for PhrasePrefixWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        if let Some(scorer) = self.phrase_scorer(reader, boost)? {\n            Ok(Box::new(scorer))\n        } else {\n            Ok(Box::new(EmptyScorer))\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let scorer_opt = self.phrase_scorer(reader, 1.0)?;\n        if scorer_opt.is_none() {\n            return Err(does_not_match(doc));\n        }\n        let mut scorer = scorer_opt.unwrap();\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);\n        let phrase_count = scorer.phrase_count();\n        let mut explanation = Explanation::new(\"Phrase Prefix Scorer\", scorer.score());\n        if let Some(similarity_weight) = self.similarity_weight_opt.as_ref() {\n            explanation.add_detail(similarity_weight.explain(fieldnorm_id, phrase_count));\n        }\n        Ok(explanation)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::docset::TERMINATED;\n    use crate::index::Index;\n    use crate::query::{EnableScoring, PhrasePrefixQuery, Query};\n    use crate::schema::{Schema, TEXT};\n    use crate::{DocSet, IndexWriter, Term};\n\n    pub fn create_index(texts: &[&'static str]) -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            for &text in texts {\n                let doc = doc!(text_field=>text);\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n        }\n        Ok(index)\n    }\n\n    #[test]\n    pub fn test_phrase_count_long() -> crate::Result<()> {\n        let index = create_index(&[\n            \"aa bb dd cc\",\n            \"aa aa bb c dd aa bb cc aa bb dc\",\n            \" aa bb cd\",\n        ])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhrasePrefixQuery::new(vec![\n            Term::from_field_text(text_field, \"aa\"),\n            Term::from_field_text(text_field, \"bb\"),\n            Term::from_field_text(text_field, \"c\"),\n        ]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query\n            .phrase_prefix_query_weight(enable_scoring)\n            .unwrap()\n            .unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_count_mid() -> crate::Result<()> {\n        let index = create_index(&[\"aa dd cc\", \"aa aa bb c dd aa bb cc aa dc\", \" aa bb cd\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhrasePrefixQuery::new(vec![\n            Term::from_field_text(text_field, \"aa\"),\n            Term::from_field_text(text_field, \"b\"),\n        ]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query\n            .phrase_prefix_query_weight(enable_scoring)\n            .unwrap()\n            .unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_count_short() -> crate::Result<()> {\n        let index = create_index(&[\"aa dd\", \"aa aa bb c dd aa bb cc aa dc\", \" aa bb cd\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhrasePrefixQuery::new(vec![Term::from_field_text(text_field, \"c\")]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        assert!(phrase_query\n            .phrase_prefix_query_weight(enable_scoring)\n            .unwrap()\n            .is_none());\n        let weight = phrase_query.weight(enable_scoring).unwrap();\n        let mut phrase_scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_no_match() -> crate::Result<()> {\n        let index = create_index(&[\"aa dd\", \"aa aa bb c dd aa bb cc aa dc\", \" aa bb cd\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhrasePrefixQuery::new(vec![\n            Term::from_field_text(text_field, \"aa\"),\n            Term::from_field_text(text_field, \"cc\"),\n            Term::from_field_text(text_field, \"d\"),\n        ]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let weight = phrase_query.weight(enable_scoring).unwrap();\n        let mut phrase_scorer = weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/mod.rs",
    "content": "mod phrase_query;\nmod phrase_scorer;\nmod phrase_weight;\npub mod regex_phrase_query;\nmod regex_phrase_weight;\n\npub use self::phrase_query::PhraseQuery;\npub(crate) use self::phrase_scorer::intersection_count;\npub use self::phrase_scorer::PhraseScorer;\npub use self::phrase_weight::PhraseWeight;\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use serde_json::json;\n\n    use super::*;\n    use crate::collector::tests::{TEST_COLLECTOR_WITHOUT_SCORE, TEST_COLLECTOR_WITH_SCORE};\n    use crate::index::Index;\n    use crate::query::{EnableScoring, QueryParser, Weight};\n    use crate::schema::{Schema, Term, TEXT};\n    use crate::{assert_nearly_equals, DocAddress, DocId, IndexWriter, TERMINATED};\n\n    pub fn create_index<S: AsRef<str>>(texts: &[S]) -> crate::Result<Index> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            for text in texts {\n                let doc = doc!(text_field=>text.as_ref());\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n        }\n        Ok(index)\n    }\n\n    #[test]\n    pub fn test_phrase_query() -> crate::Result<()> {\n        let index = create_index(&[\n            \"b b b d c g c\",\n            \"a b b d c g c\",\n            \"a b a b c\",\n            \"c a b a d ga a\",\n            \"a b c\",\n        ])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let test_query = |texts: Vec<&str>| {\n            let terms: Vec<Term> = texts\n                .iter()\n                .map(|text| Term::from_field_text(text_field, text))\n                .collect();\n            let phrase_query = PhraseQuery::new(terms);\n            let test_fruits = searcher\n                .search(&phrase_query, &TEST_COLLECTOR_WITH_SCORE)\n                .unwrap();\n            test_fruits\n                .docs()\n                .iter()\n                .map(|docaddr| docaddr.doc_id)\n                .collect::<Vec<_>>()\n        };\n        assert_eq!(test_query(vec![\"a\", \"b\"]), vec![1, 2, 3, 4]);\n        assert_eq!(test_query(vec![\"a\", \"b\", \"c\"]), vec![2, 4]);\n        assert_eq!(test_query(vec![\"b\", \"b\"]), vec![0, 1]);\n        assert!(test_query(vec![\"g\", \"ewrwer\"]).is_empty());\n        assert!(test_query(vec![\"g\", \"a\"]).is_empty());\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_query_simple() -> crate::Result<()> {\n        let index = create_index(&[\"a b b d c g c\", \"a b a b c\"])?;\n        let text_field = index.schema().get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let terms: Vec<Term> = [\"a\", \"b\", \"c\"]\n            .iter()\n            .map(|text| Term::from_field_text(text_field, text))\n            .collect();\n        let phrase_query = PhraseQuery::new(terms);\n        let phrase_weight =\n            phrase_query.phrase_weight(EnableScoring::disabled_from_schema(searcher.schema()))?;\n        let mut phrase_scorer = phrase_weight.scorer(searcher.segment_reader(0), 1.0)?;\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_query_no_score() -> crate::Result<()> {\n        let index = create_index(&[\n            \"b b b d c g c\",\n            \"a b b d c g c\",\n            \"a b a b c\",\n            \"c a b a d ga a\",\n            \"a b c\",\n        ])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let test_query = |texts: Vec<&str>| {\n            let terms: Vec<Term> = texts\n                .iter()\n                .map(|text| Term::from_field_text(text_field, text))\n                .collect();\n            let phrase_query = PhraseQuery::new(terms);\n            let test_fruits = searcher\n                .search(&phrase_query, &TEST_COLLECTOR_WITHOUT_SCORE)\n                .unwrap();\n            test_fruits\n                .docs()\n                .iter()\n                .map(|docaddr| docaddr.doc_id)\n                .collect::<Vec<_>>()\n        };\n        assert_eq!(test_query(vec![\"a\", \"b\", \"c\"]), vec![2, 4]);\n        assert_eq!(test_query(vec![\"a\", \"b\"]), vec![1, 2, 3, 4]);\n        assert_eq!(test_query(vec![\"b\", \"b\"]), vec![0, 1]);\n        assert!(test_query(vec![\"g\", \"ewrwer\"]).is_empty());\n        assert!(test_query(vec![\"g\", \"a\"]).is_empty());\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_query_no_positions() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        use crate::schema::{IndexRecordOption, TextFieldIndexing, TextOptions};\n        let no_positions = TextOptions::default().set_indexing_options(\n            TextFieldIndexing::default().set_index_option(IndexRecordOption::WithFreqs),\n        );\n\n        let text_field = schema_builder.add_text_field(\"text\", no_positions);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"a b c\"))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhraseQuery::new(vec![\n            Term::from_field_text(text_field, \"a\"),\n            Term::from_field_text(text_field, \"b\"),\n        ]);\n\n        let search_error = searcher\n            .search(&phrase_query, &TEST_COLLECTOR_WITH_SCORE)\n            .err();\n        assert!(matches!(\n            search_error,\n            Some(crate::TantivyError::SchemaError(msg))\n            if msg == \"Applied phrase query on field \\\"text\\\", which does not have positions \\\n            indexed\"\n        ));\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_score() -> crate::Result<()> {\n        let index = create_index(&[\"a b c\", \"a b c a b\"])?;\n        let scores = test_query(0, &index, vec![\"a\", \"b\"]);\n        assert_nearly_equals!(scores[0], 0.40618482);\n        assert_nearly_equals!(scores[1], 0.46844664);\n        Ok(())\n    }\n\n    #[ignore]\n    #[test]\n    pub fn test_phrase_score_with_slop() -> crate::Result<()> {\n        let index = create_index(&[\"a c b\", \"a b c a b\"])?;\n        let scores = test_query(1, &index, vec![\"a\", \"b\"]);\n        assert_nearly_equals!(scores[0], 0.40618482);\n        assert_nearly_equals!(scores[1], 0.46844664);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_score_with_slop_bug() -> crate::Result<()> {\n        let index = create_index(&[\"asdf asdf Captain Subject Wendy\", \"Captain\"])?;\n        let scores = test_query(1, &index, vec![\"captain\", \"wendy\"]);\n        assert_eq!(scores.len(), 1);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_score_with_slop_bug_2() -> crate::Result<()> {\n        // fails\n        let index = create_index(&[\"a x b x c\", \"a a c\"])?;\n        let scores = test_query(2, &index, vec![\"a\", \"b\", \"c\"]);\n        assert_eq!(scores.len(), 1);\n\n        let index = create_index(&[\"a x b x c\", \"b c c\"])?;\n        let scores = test_query(2, &index, vec![\"a\", \"b\", \"c\"]);\n        assert_eq!(scores.len(), 1);\n\n        Ok(())\n    }\n\n    fn test_query(slop: u32, index: &Index, texts: Vec<&str>) -> Vec<f32> {\n        let text_field = index.schema().get_field(\"text\").unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let terms: Vec<Term> = texts\n            .iter()\n            .map(|text| Term::from_field_text(text_field, text))\n            .collect();\n        let mut phrase_query = PhraseQuery::new(terms);\n        phrase_query.set_slop(slop);\n        searcher\n            .search(&phrase_query, &TEST_COLLECTOR_WITH_SCORE)\n            .expect(\"search should succeed\")\n            .scores()\n            .to_vec()\n    }\n\n    #[test]\n    pub fn test_phrase_score_with_slop_repeating() -> crate::Result<()> {\n        let index = create_index(&[\"wendy subject subject captain\", \"Captain\"])?;\n        let scores = test_query(1, &index, vec![\"wendy\", \"subject\", \"captain\"]);\n        assert_eq!(scores.len(), 1);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_score_with_slop_size() -> crate::Result<()> {\n        let index = create_index(&[\"a b e c\", \"a e e e c\", \"a e e e e c\"])?;\n        let scores = test_query(3, &index, vec![\"a\", \"c\"]);\n        assert_eq!(scores.len(), 2);\n        assert_nearly_equals!(scores[0], 0.29086056);\n        assert_nearly_equals!(scores[1], 0.26706287);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_slop() -> crate::Result<()> {\n        let index = create_index(&[\"a x b c\"])?;\n        let scores = test_query(1, &index, vec![\"a\", \"b\", \"c\"]);\n        assert_eq!(scores.len(), 1);\n\n        let index = create_index(&[\"a x b x c\"])?;\n        let scores = test_query(1, &index, vec![\"a\", \"b\", \"c\"]);\n        assert_eq!(scores.len(), 0);\n\n        let index = create_index(&[\"a b\"])?;\n        let scores = test_query(1, &index, vec![\"b\", \"a\"]);\n        assert_eq!(scores.len(), 0);\n\n        let index = create_index(&[\"a b\"])?;\n        let scores = test_query(2, &index, vec![\"b\", \"a\"]);\n        assert_eq!(scores.len(), 1);\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_score_with_slop_ordering() -> crate::Result<()> {\n        let index = create_index(&[\n            \"a e b e c\",\n            \"a e e e e e b e e e e c\",\n            \"a c b\", // also matches\n            \"a c e b e\",\n            \"a e c b\",\n            \"a e b c\",\n        ])?;\n        let scores = test_query(3, &index, vec![\"a\", \"b\", \"c\"]);\n        // The first and last matches.\n        assert_nearly_equals!(scores[0], 0.23091172);\n        assert_nearly_equals!(scores[1], 0.27310878);\n        assert_nearly_equals!(scores[3], 0.25024384);\n        Ok(())\n    }\n\n    #[test] // motivated by #234\n    pub fn test_phrase_query_docfreq_order() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"b\"))?;\n            index_writer.add_document(doc!(text_field=>\"a b\"))?;\n            index_writer.add_document(doc!(text_field=>\"b a\"))?;\n            index_writer.commit()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        let test_query = |texts: Vec<&str>| {\n            let terms: Vec<Term> = texts\n                .iter()\n                .map(|text| Term::from_field_text(text_field, text))\n                .collect();\n            let phrase_query = PhraseQuery::new(terms);\n            searcher\n                .search(&phrase_query, &TEST_COLLECTOR_WITH_SCORE)\n                .expect(\"search should succeed\")\n                .docs()\n                .to_vec()\n        };\n        assert_eq!(test_query(vec![\"a\", \"b\"]), vec![DocAddress::new(0, 1)]);\n        assert_eq!(test_query(vec![\"b\", \"a\"]), vec![DocAddress::new(0, 2)]);\n        Ok(())\n    }\n\n    #[test] // motivated by #234\n    pub fn test_phrase_query_non_trivial_offsets() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=>\"a b c d e f g h\"))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader().unwrap().searcher();\n        let test_query = |texts: Vec<(usize, &str)>| {\n            let terms: Vec<(usize, Term)> = texts\n                .iter()\n                .map(|(offset, text)| (*offset, Term::from_field_text(text_field, text)))\n                .collect();\n            let phrase_query = PhraseQuery::new_with_offset(terms);\n            searcher\n                .search(&phrase_query, &TEST_COLLECTOR_WITH_SCORE)\n                .expect(\"search should succeed\")\n                .docs()\n                .iter()\n                .map(|doc_address| doc_address.doc_id)\n                .collect::<Vec<DocId>>()\n        };\n        assert_eq!(test_query(vec![(0, \"a\"), (1, \"b\")]), vec![0]);\n        assert_eq!(test_query(vec![(1, \"b\"), (0, \"a\")]), vec![0]);\n        assert!(test_query(vec![(0, \"a\"), (2, \"b\")]).is_empty());\n        assert_eq!(test_query(vec![(0, \"a\"), (2, \"c\")]), vec![0]);\n        assert_eq!(test_query(vec![(0, \"a\"), (2, \"c\"), (3, \"d\")]), vec![0]);\n        assert_eq!(test_query(vec![(0, \"a\"), (2, \"c\"), (4, \"e\")]), vec![0]);\n        assert_eq!(test_query(vec![(4, \"e\"), (0, \"a\"), (2, \"c\")]), vec![0]);\n        assert!(test_query(vec![(0, \"a\"), (2, \"d\")]).is_empty());\n        assert_eq!(test_query(vec![(1, \"a\"), (3, \"c\")]), vec![0]);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_query_on_json() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(json_field=>json!({\n                \"text\": \"elliot smith the happy who\"\n            })))?;\n            index_writer.add_document(doc!(json_field=>json!({\n                \"text\": \"the who elliot smith\"\n            })))?;\n            index_writer.add_document(doc!(json_field=>json!({\n                \"arr\": [{\"text\":\"the who\"}, {\"text\":\"elliot smith\"}]\n            })))?;\n            index_writer.add_document(doc!(json_field=>json!({\n                \"text2\": \"the smith\"\n            })))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let matching_docs = |query: &str| {\n            let query_parser = QueryParser::for_index(&index, vec![json_field]);\n            let phrase_query = query_parser.parse_query(query).unwrap();\n            let phrase_weight = phrase_query\n                .weight(EnableScoring::disabled_from_schema(searcher.schema()))\n                .unwrap();\n            let mut phrase_scorer = phrase_weight\n                .scorer(searcher.segment_reader(0), 1.0f32)\n                .unwrap();\n            let mut docs = Vec::new();\n            loop {\n                let doc = phrase_scorer.doc();\n                if doc == TERMINATED {\n                    break;\n                }\n                docs.push(doc);\n                phrase_scorer.advance();\n            }\n            docs\n        };\n        assert!(matching_docs(r#\"text:\"the smith\"\"#).is_empty());\n        assert_eq!(&matching_docs(r#\"text:the\"#), &[0u32, 1u32]);\n        assert_eq!(&matching_docs(r#\"text:\"the\"\"#), &[0u32, 1u32]);\n        assert_eq!(&matching_docs(r#\"text:\"smith\"\"#), &[0u32, 1u32]);\n        assert_eq!(&matching_docs(r#\"text:\"elliot smith\"\"#), &[0u32, 1u32]);\n        assert_eq!(&matching_docs(r#\"text2:\"the smith\"\"#), &[3u32]);\n        assert!(&matching_docs(r#\"arr.text:\"the smith\"\"#).is_empty());\n        assert_eq!(&matching_docs(r#\"arr.text:\"elliot smith\"\"#), &[2]);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/phrase_query.rs",
    "content": "use super::PhraseWeight;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::{EnableScoring, Query, Weight};\nuse crate::schema::{Field, IndexRecordOption, Term};\n\n/// `PhraseQuery` matches a specific sequence of words.\n///\n/// For instance, the phrase query for `\"part time\"` will match\n/// the sentence:\n///\n/// **Alan just got a part time job.**\n///\n/// On the other hand it will not match the sentence.\n///\n/// **This is my favorite part of the job.**\n///\n/// [Slop](PhraseQuery::set_slop) allows leniency in term proximity\n/// for some performance trade-off.\n///\n/// Using a `PhraseQuery` on a field requires positions\n/// to be indexed for this field.\n#[derive(Clone, Debug)]\npub struct PhraseQuery {\n    field: Field,\n    phrase_terms: Vec<(usize, Term)>,\n    slop: u32,\n}\n\nimpl PhraseQuery {\n    /// Creates a new `PhraseQuery` given a list of terms.\n    ///\n    /// There must be at least two terms, and all terms\n    /// must belong to the same field.\n    /// Offset for each term will be same as index in the Vector\n    pub fn new(terms: Vec<Term>) -> PhraseQuery {\n        let terms_with_offset = terms.into_iter().enumerate().collect();\n        PhraseQuery::new_with_offset(terms_with_offset)\n    }\n\n    /// Creates a new `PhraseQuery` given a list of terms and their offsets.\n    ///\n    /// Can be used to provide custom offset for each term.\n    pub fn new_with_offset(terms: Vec<(usize, Term)>) -> PhraseQuery {\n        PhraseQuery::new_with_offset_and_slop(terms, 0)\n    }\n\n    /// Creates a new `PhraseQuery` given a list of terms, their offsets and a slop\n    pub fn new_with_offset_and_slop(mut terms: Vec<(usize, Term)>, slop: u32) -> PhraseQuery {\n        assert!(\n            terms.len() > 1,\n            \"A phrase query is required to have strictly more than one term.\"\n        );\n        terms.sort_by_key(|&(offset, _)| offset);\n        let field = terms[0].1.field();\n        assert!(\n            terms[1..].iter().all(|term| term.1.field() == field),\n            \"All terms from a phrase query must belong to the same field\"\n        );\n        PhraseQuery {\n            field,\n            phrase_terms: terms,\n            slop,\n        }\n    }\n\n    /// Slop allowed for the phrase.\n    ///\n    /// The query will match if its terms are separated by `slop` terms at most.\n    /// The slop can be considered a budget between all terms.\n    /// E.g. \"A B C\" with slop 1 allows \"A X B C\", \"A B X C\", but not \"A X B X C\".\n    ///\n    /// Transposition costs 2, e.g. \"A B\" with slop 1 will not match \"B A\" but it would with slop 2\n    /// Transposition is not a special case, in the example above A is moved 1 position and B is\n    /// moved 1 position, so the slop is 2.\n    ///\n    /// As a result slop works in both directions, so the order of the terms may changed as long as\n    /// they respect the slop.\n    ///\n    /// By default the slop is 0 meaning query terms need to be adjacent.\n    pub fn set_slop(&mut self, value: u32) {\n        self.slop = value;\n    }\n\n    /// The [`Field`] this `PhraseQuery` is targeting.\n    pub fn field(&self) -> Field {\n        self.field\n    }\n\n    /// `Term`s in the phrase without the associated offsets.\n    pub fn phrase_terms(&self) -> Vec<Term> {\n        self.phrase_terms\n            .iter()\n            .map(|(_, term)| term.clone())\n            .collect::<Vec<Term>>()\n    }\n\n    /// Returns the [`PhraseWeight`] for the given phrase query given a specific `searcher`.\n    ///\n    /// This function is the same as [`Query::weight()`] except it returns\n    /// a specialized type [`PhraseWeight`] instead of a Boxed trait.\n    pub(crate) fn phrase_weight(\n        &self,\n        enable_scoring: EnableScoring<'_>,\n    ) -> crate::Result<PhraseWeight> {\n        let schema = enable_scoring.schema();\n        let field_entry = schema.get_field_entry(self.field);\n        let has_positions = field_entry\n            .field_type()\n            .get_index_record_option()\n            .map(IndexRecordOption::has_positions)\n            .unwrap_or(false);\n        if !has_positions {\n            let field_name = field_entry.name();\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"Applied phrase query on field {field_name:?}, which does not have positions \\\n                 indexed\"\n            )));\n        }\n        let terms = self.phrase_terms();\n        let bm25_weight_opt = match enable_scoring {\n            EnableScoring::Enabled {\n                statistics_provider,\n                ..\n            } => Some(Bm25Weight::for_terms(statistics_provider, &terms)?),\n            EnableScoring::Disabled { .. } => None,\n        };\n        let mut weight = PhraseWeight::new(self.phrase_terms.clone(), bm25_weight_opt);\n        if self.slop > 0 {\n            weight.slop(self.slop);\n        }\n        Ok(weight)\n    }\n}\n\nimpl Query for PhraseQuery {\n    /// Create the weight associated with a query.\n    ///\n    /// See [`Weight`].\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let phrase_weight = self.phrase_weight(enable_scoring)?;\n        Ok(Box::new(phrase_weight))\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        for (_, term) in &self.phrase_terms {\n            visitor(term, true);\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/phrase_scorer.rs",
    "content": "use std::cmp::Ordering;\n\nuse crate::docset::{DocSet, SeekDangerResult, TERMINATED};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::postings::Postings;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::{Intersection, Scorer};\nuse crate::{DocId, Score};\n\nstruct PostingsWithOffset<TPostings> {\n    offset: u32,\n    postings: TPostings,\n}\n\nimpl<TPostings: Postings> PostingsWithOffset<TPostings> {\n    pub fn new(segment_postings: TPostings, offset: u32) -> PostingsWithOffset<TPostings> {\n        PostingsWithOffset {\n            offset,\n            postings: segment_postings,\n        }\n    }\n\n    pub fn positions(&mut self, output: &mut Vec<u32>) {\n        self.postings.positions_with_offset(self.offset, output)\n    }\n}\n\nimpl<TPostings: Postings> DocSet for PostingsWithOffset<TPostings> {\n    fn advance(&mut self) -> DocId {\n        self.postings.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.postings.seek(target)\n    }\n\n    fn doc(&self) -> DocId {\n        self.postings.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.postings.size_hint()\n    }\n}\n\npub struct PhraseScorer<TPostings: Postings> {\n    intersection_docset: Intersection<PostingsWithOffset<TPostings>, PostingsWithOffset<TPostings>>,\n    num_terms: usize,\n    left_positions: Vec<u32>,\n    right_positions: Vec<u32>,\n    phrase_count: u32,\n    fieldnorm_reader: FieldNormReader,\n    similarity_weight_opt: Option<Bm25Weight>,\n    slop: u32,\n    left_slops: Vec<u8>,\n    positions_buffer: Vec<u32>,\n    slops_buffer: Vec<u8>,\n}\n\n/// Returns true if and only if the two sorted arrays contain a common element\nfn intersection_exists(left: &[u32], right: &[u32]) -> bool {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    while left_index < left.len() && right_index < right.len() {\n        let left_val = left[left_index];\n        let right_val = right[right_index];\n        match left_val.cmp(&right_val) {\n            Ordering::Less => {\n                left_index += 1;\n            }\n            Ordering::Equal => {\n                return true;\n            }\n            Ordering::Greater => {\n                right_index += 1;\n            }\n        }\n    }\n    false\n}\n\npub(crate) fn intersection_count(left: &[u32], right: &[u32]) -> usize {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    let mut count = 0;\n    while left_index < left.len() && right_index < right.len() {\n        let left_val = left[left_index];\n        let right_val = right[right_index];\n        match left_val.cmp(&right_val) {\n            Ordering::Less => {\n                left_index += 1;\n            }\n            Ordering::Equal => {\n                count += 1;\n                left_index += 1;\n                right_index += 1;\n            }\n            Ordering::Greater => {\n                right_index += 1;\n            }\n        }\n    }\n    count\n}\n\n/// Intersect twos sorted arrays `left` and `right` and outputs the\n/// resulting array in left.\n///\n/// Returns the length of the intersection\n#[inline]\nfn intersection(left: &mut Vec<u32>, right: &[u32]) {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    let mut count = 0;\n    let left_len = left.len();\n    let right_len = right.len();\n    while left_index < left_len && right_index < right_len {\n        let left_val = left[left_index];\n        let right_val = right[right_index];\n        match left_val.cmp(&right_val) {\n            Ordering::Less => {\n                left_index += 1;\n            }\n            Ordering::Equal => {\n                left[count] = left_val;\n                count += 1;\n                left_index += 1;\n                right_index += 1;\n            }\n            Ordering::Greater => {\n                right_index += 1;\n            }\n        }\n    }\n    left.truncate(count);\n}\n\n/// Intersect twos sorted arrays `left` and `right` and outputs the\n/// resulting array in left_positions if update_left is true.\n///\n/// Condition for match is that the distance between left and right is less than or equal to `slop`.\n///\n/// Returns the length of the intersection\n#[inline]\nfn intersection_count_with_slop(\n    left_positions: &mut Vec<u32>,\n    right_positions: &[u32],\n    slop: u32,\n    update_left: bool,\n) -> usize {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    let mut count = 0;\n    let left_len = left_positions.len();\n    let right_len = right_positions.len();\n    while left_index < left_len && right_index < right_len {\n        let left_val = left_positions[left_index];\n        let right_val = right_positions[right_index];\n\n        let distance = left_val.abs_diff(right_val);\n        if distance <= slop {\n            while left_index + 1 < left_len {\n                // there could be a better match\n                let next_left_val = left_positions[left_index + 1];\n                if next_left_val > right_val {\n                    // the next value is outside the range, so current one is the best.\n                    break;\n                }\n                // the next value is better.\n                left_index += 1;\n            }\n\n            // store the match in left.\n            if update_left {\n                left_positions[count] = right_val;\n            }\n            count += 1;\n            left_index += 1;\n            right_index += 1;\n        } else if left_val < right_val {\n            left_index += 1;\n        } else {\n            right_index += 1;\n        }\n    }\n    if update_left {\n        left_positions.truncate(count);\n    }\n\n    count\n}\n\nfn intersection_exists_with_slop(\n    left_positions: &[u32],\n    right_positions: &[u32],\n    slop: u32,\n) -> bool {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    let left_len = left_positions.len();\n    let right_len = right_positions.len();\n    while left_index < left_len && right_index < right_len {\n        let left_val = left_positions[left_index];\n        let right_val = right_positions[right_index];\n        let distance = left_val.abs_diff(right_val);\n        if distance <= slop {\n            return true;\n        } else if left_val < right_val {\n            left_index += 1;\n        } else {\n            right_index += 1;\n        }\n    }\n    false\n}\n\n/// Intersection variant for multi term searches that keeps track of slop so far.\n///\n/// In contrast to the regular algorithm this solves some issues:\n/// - Keep track of the slop so far. Slop is a budget that is spent on the distance between terms.\n/// - When encountering a match between two positions, which position is the best match is unclear\n///   and depends on intersections afterwards, therefore this algorithm keeps left and right as\n///   matches, but only counts one.\n///\n/// This algorithm may return an incorrect count in some cases (e.g. left, right expansion and is\n/// then matches both on the following term.)\n/// I think to fix this we would need to iterate all positions simultaneously,\n/// but not sure if that's worth it. (It may be considerable slower - untested)\n///\n/// left_slops is allowed to be empty, which equals to a slop of 0 so far.\n#[inline]\nfn intersection_count_with_carrying_slop(\n    left_positions: &mut Vec<u32>,\n    left_slops: &mut Vec<u8>,\n    right_positions: &[u32],\n    max_slop: u32,\n    update_left: bool,\n    positions_buffer: &mut Vec<u32>,\n    slops_buffer: &mut Vec<u8>,\n) -> u32 {\n    let mut left_index = 0;\n    let mut right_index = 0;\n    let mut count = 0;\n\n    if left_positions.is_empty() || right_positions.is_empty() {\n        if update_left {\n            left_positions.clear();\n            left_slops.clear();\n        }\n        return 0;\n    }\n\n    let add_val = |val: (u8, u32), new_left: &mut Vec<u32>, new_slops: &mut Vec<u8>| {\n        if update_left {\n            let pos_exists = new_left.last().map(|v| *v == val.1).unwrap_or(false);\n            if pos_exists {\n                let last_slop = new_slops.last_mut().unwrap();\n                *last_slop = (*last_slop).min(val.0);\n            } else {\n                new_left.push(val.1);\n                new_slops.push(val.0);\n            }\n        }\n    };\n    loop {\n        let left_val = left_positions[left_index];\n        let slop_so_far = left_slops.get(left_index).cloned().unwrap_or(0);\n        let right_val = right_positions[right_index];\n\n        let distance = slop_so_far as u32 + left_val.abs_diff(right_val);\n        if distance <= max_slop {\n            let (smaller_val, larger_val, mut smaller_val_idx, smaller_val_positions) =\n                if left_val < right_val {\n                    (left_val, right_val, left_index, left_positions.as_slice())\n                } else {\n                    (right_val, left_val, right_index, right_positions)\n                };\n\n            let mut new_slop = distance;\n            add_val(\n                (new_slop as u8, smaller_val),\n                positions_buffer,\n                slops_buffer,\n            );\n            while smaller_val_idx + 1 < smaller_val_positions.len() {\n                // there could be a better match\n                let next_val = smaller_val_positions[smaller_val_idx + 1];\n                if next_val > larger_val {\n                    // the next value is outside the range, so current one is the best.\n                    break;\n                }\n                let distance = next_val.abs_diff(larger_val);\n\n                // the next value is better.\n                smaller_val_idx += 1;\n                // better slop\n                new_slop = slop_so_far as u32 + distance;\n                add_val((new_slop as u8, next_val), positions_buffer, slops_buffer);\n            }\n\n            add_val((new_slop as u8, larger_val), positions_buffer, slops_buffer);\n            count += 1;\n            left_index += 1;\n            right_index += 1;\n        } else if left_val < right_val {\n            left_index += 1;\n        } else {\n            right_index += 1;\n        }\n\n        if left_index >= left_positions.len() || right_index >= right_positions.len() {\n            // finish rest\n            if left_index >= left_positions.len() {\n                let left_val = *left_positions.last().unwrap();\n                let slop_so_far: u8 = *left_slops.last().unwrap_or(&0);\n                for right_val in &right_positions[right_index..] {\n                    let new_slop = left_val.abs_diff(*right_val) + slop_so_far as u32;\n                    if new_slop <= max_slop {\n                        add_val((new_slop as u8, *right_val), positions_buffer, slops_buffer);\n                    }\n                }\n            } else {\n                let right_val = *right_positions.last().unwrap();\n                for left_idx in left_index..left_positions.len() {\n                    let left_val = left_positions[left_idx];\n                    let slop_so_far = *left_slops.get(left_idx).unwrap_or(&0);\n                    let new_slop = left_val.abs_diff(right_val) + slop_so_far as u32;\n                    if new_slop <= max_slop {\n                        add_val((new_slop as u8, left_val), positions_buffer, slops_buffer);\n                    }\n                }\n            };\n\n            break;\n        }\n    }\n    if update_left {\n        std::mem::swap(left_positions, positions_buffer);\n        std::mem::swap(left_slops, slops_buffer);\n        positions_buffer.clear();\n        slops_buffer.clear();\n    }\n\n    count\n}\n\nimpl<TPostings: Postings> PhraseScorer<TPostings> {\n    // If similarity_weight is None, then scoring is disabled.\n    pub fn new(\n        term_postings: Vec<(usize, TPostings)>,\n        similarity_weight_opt: Option<Bm25Weight>,\n        fieldnorm_reader: FieldNormReader,\n        slop: u32,\n    ) -> PhraseScorer<TPostings> {\n        Self::new_with_offset(\n            term_postings,\n            similarity_weight_opt,\n            fieldnorm_reader,\n            slop,\n            0,\n        )\n    }\n\n    pub(crate) fn new_with_offset(\n        term_postings_with_offset: Vec<(usize, TPostings)>,\n        similarity_weight_opt: Option<Bm25Weight>,\n        fieldnorm_reader: FieldNormReader,\n        slop: u32,\n        offset: usize,\n    ) -> PhraseScorer<TPostings> {\n        let num_docs = fieldnorm_reader.num_docs();\n        let max_offset = term_postings_with_offset\n            .iter()\n            .map(|&(offset, _)| offset)\n            .max()\n            .unwrap_or(0)\n            + offset;\n        let num_docsets = term_postings_with_offset.len();\n        let postings_with_offsets = term_postings_with_offset\n            .into_iter()\n            .map(|(offset, postings)| {\n                PostingsWithOffset::new(postings, (max_offset - offset) as u32)\n            })\n            .collect::<Vec<_>>();\n        let intersection_docset = Intersection::new(postings_with_offsets, num_docs);\n        let mut scorer = PhraseScorer {\n            intersection_docset,\n            num_terms: num_docsets,\n            left_positions: Vec::with_capacity(100),\n            right_positions: Vec::with_capacity(100),\n            phrase_count: 0u32,\n            similarity_weight_opt,\n            fieldnorm_reader,\n            slop,\n            left_slops: Vec::with_capacity(100),\n            slops_buffer: Vec::with_capacity(100),\n            positions_buffer: Vec::with_capacity(100),\n        };\n        if scorer.doc() != TERMINATED && !scorer.phrase_match() {\n            scorer.advance();\n        }\n        scorer\n    }\n\n    pub fn phrase_count(&self) -> u32 {\n        self.phrase_count\n    }\n\n    pub(crate) fn get_intersection(&mut self) -> &[u32] {\n        intersection(&mut self.left_positions, &self.right_positions);\n        &self.left_positions\n    }\n\n    fn phrase_match(&mut self) -> bool {\n        if self.similarity_weight_opt.is_some() {\n            let count = self.compute_phrase_count();\n            self.phrase_count = count;\n            count > 0u32\n        } else {\n            self.phrase_exists()\n        }\n    }\n\n    fn phrase_exists(&mut self) -> bool {\n        self.compute_phrase_match();\n        if self.has_slop() {\n            intersection_exists_with_slop(\n                &self.left_positions,\n                &self.right_positions[..],\n                self.slop,\n            )\n        } else {\n            intersection_exists(&self.left_positions, &self.right_positions[..])\n        }\n    }\n\n    fn compute_phrase_count(&mut self) -> u32 {\n        self.compute_phrase_match();\n        if self.has_slop() {\n            if self.num_terms > 2 {\n                intersection_count_with_carrying_slop(\n                    &mut self.left_positions,\n                    &mut self.left_slops,\n                    &self.right_positions[..],\n                    self.slop,\n                    false,\n                    &mut self.positions_buffer,\n                    &mut self.slops_buffer,\n                )\n            } else {\n                intersection_count_with_slop(\n                    &mut self.left_positions,\n                    &self.right_positions[..],\n                    self.slop,\n                    false,\n                ) as u32\n            }\n        } else {\n            intersection_count(&self.left_positions, &self.right_positions[..]) as u32\n        }\n    }\n\n    fn compute_phrase_match(&mut self) {\n        {\n            self.intersection_docset\n                .docset_mut_specialized(0)\n                .positions(&mut self.left_positions);\n            if self.has_slop() {\n                self.left_slops.clear();\n            }\n        }\n        for i in 1..self.num_terms - 1 {\n            {\n                self.intersection_docset\n                    .docset_mut_specialized(i)\n                    .positions(&mut self.right_positions);\n            }\n            if self.has_slop() {\n                if self.num_terms > 2 {\n                    intersection_count_with_carrying_slop(\n                        &mut self.left_positions,\n                        &mut self.left_slops,\n                        &self.right_positions[..],\n                        self.slop,\n                        true,\n                        &mut self.positions_buffer,\n                        &mut self.slops_buffer,\n                    );\n                } else {\n                    intersection_count_with_slop(\n                        &mut self.left_positions,\n                        &self.right_positions[..],\n                        self.slop,\n                        true,\n                    );\n                }\n            } else {\n                intersection(&mut self.left_positions, &self.right_positions);\n            };\n            if self.left_positions.is_empty() {\n                return;\n            }\n        }\n        self.intersection_docset\n            .docset_mut_specialized(self.num_terms - 1)\n            .positions(&mut self.right_positions);\n    }\n\n    fn has_slop(&self) -> bool {\n        self.slop > 0\n    }\n}\n\nimpl<TPostings: Postings> DocSet for PhraseScorer<TPostings> {\n    fn advance(&mut self) -> DocId {\n        loop {\n            let doc = self.intersection_docset.advance();\n            if doc == TERMINATED || self.phrase_match() {\n                return doc;\n            }\n        }\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        debug_assert!(target >= self.doc());\n        let doc = self.intersection_docset.seek(target);\n        if doc == TERMINATED || self.phrase_match() {\n            return doc;\n        }\n        self.advance()\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        debug_assert!(\n            target >= self.doc(),\n            \"target ({}) should be greater than or equal to doc ({})\",\n            target,\n            self.doc()\n        );\n        let seek_res = self.intersection_docset.seek_danger(target);\n        if seek_res != SeekDangerResult::Found {\n            return seek_res;\n        }\n        // The intersection matched. Now let's see if we match the phrase.\n        if self.phrase_match() {\n            SeekDangerResult::Found\n        } else {\n            SeekDangerResult::SeekLowerBound(target + 1)\n        }\n    }\n\n    fn doc(&self) -> DocId {\n        self.intersection_docset.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        // We adjust the intersection estimate, since actual phrase hits are much lower than where\n        // the all appear.\n        // The estimate should depend on average field length, e.g. if the field is really short\n        // a phrase hit is more likely\n        self.intersection_docset.size_hint() / (10 * self.num_terms as u32)\n    }\n\n    /// Returns a best-effort hint of the\n    /// cost to drive the docset.\n    fn cost(&self) -> u64 {\n        // While determing a potential hit is cheap for phrases, evaluating an actual hit is\n        // expensive since it requires to load positions for a doc and check if they are next to\n        // each other.\n        // So the cost estimation would be the number of times we need to check if a doc is a hit *\n        // 10 * self.num_terms.\n        self.intersection_docset.size_hint() as u64 * 10 * self.num_terms as u64\n    }\n}\n\nimpl<TPostings: Postings> Scorer for PhraseScorer<TPostings> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        let doc = self.doc();\n        let fieldnorm_id = self.fieldnorm_reader.fieldnorm_id(doc);\n        if let Some(similarity_weight) = self.similarity_weight_opt.as_ref() {\n            similarity_weight.score(fieldnorm_id, self.phrase_count)\n        } else {\n            1.0f32\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    fn test_intersection_sym(left: &[u32], right: &[u32], expected: &[u32]) {\n        test_intersection_aux(left, right, expected, 0);\n        test_intersection_aux(right, left, expected, 0);\n    }\n\n    fn test_intersection_aux(left: &[u32], right: &[u32], expected: &[u32], slop: u32) {\n        let mut left_vec = Vec::from(left);\n        if slop == 0 {\n            assert_eq!(intersection_count(&left_vec, right), expected.len());\n            intersection(&mut left_vec, right);\n            assert_eq!(&left_vec, expected);\n        } else {\n            let mut right_vec = Vec::from(right);\n            let right_mut = &mut right_vec[..];\n            intersection_count_with_slop(&mut left_vec, right_mut, slop, true);\n            assert_eq!(&left_vec, expected);\n        }\n    }\n\n    #[test]\n    fn test_intersection() {\n        test_intersection_sym(&[1], &[1], &[1]);\n        test_intersection_sym(&[1], &[2], &[]);\n        test_intersection_sym(&[], &[2], &[]);\n        test_intersection_sym(&[5, 7], &[1, 5, 10, 12], &[5]);\n        test_intersection_sym(&[1, 5, 6, 9, 10, 12], &[6, 8, 9, 12], &[6, 9, 12]);\n    }\n    #[test]\n    fn test_slop() {\n        // The slop is not symmetric. It does not allow for the phrase to be out of order.\n        test_intersection_aux(&[1], &[2], &[2], 1);\n        test_intersection_aux(&[1], &[3], &[], 1);\n        test_intersection_aux(&[1], &[3], &[3], 2);\n        test_intersection_aux(&[], &[2], &[], 100000);\n        test_intersection_aux(&[5, 7, 11], &[1, 5, 10, 12], &[5, 10], 1);\n        test_intersection_aux(&[1, 5, 6, 9, 10, 12], &[6, 8, 9, 12], &[6, 8, 9, 12], 1);\n        test_intersection_aux(&[1, 5, 6, 9, 10, 12], &[6, 8, 9, 12], &[6, 8, 9, 12], 10);\n        test_intersection_aux(&[1, 3, 5], &[2, 4, 6], &[2, 4, 6], 1);\n        test_intersection_aux(&[1, 3, 5], &[2, 4, 6], &[], 0);\n    }\n\n    fn test_merge(left: &[u32], right: &[u32], expected_left: &[u32], slop: u32) {\n        let mut left_vec = Vec::from(left);\n        let mut right_vec = Vec::from(right);\n        let right_mut = &mut right_vec[..];\n        intersection_count_with_slop(&mut left_vec, right_mut, slop, true);\n        assert_eq!(&left_vec, expected_left);\n    }\n\n    #[test]\n    fn test_merge_slop() {\n        test_merge(&[1, 2], &[1], &[1], 1);\n        test_merge(&[3], &[4], &[4], 2);\n        test_merge(&[3], &[4], &[4], 2);\n        test_merge(&[1, 5, 6, 9, 10, 12], &[6, 8, 9, 12], &[6, 8, 9, 12], 10);\n    }\n\n    fn test_carry_slop_intersection_aux(\n        right: &[&[u32]],\n        expected: &[(u8, u32)],\n        slop: u32,\n        expected_count: u32,\n    ) {\n        let mut left_vec = right[0].to_vec();\n        let mut slops = vec![0; left_vec.len()];\n        let mut count = 0;\n        for right in &right[1..] {\n            count = intersection_count_with_carrying_slop(\n                &mut left_vec,\n                &mut slops,\n                right,\n                slop,\n                true,\n                &mut Vec::new(),\n                &mut Vec::new(),\n            );\n        }\n        let out: Vec<(u8, u32)> = slops\n            .iter()\n            .cloned()\n            .zip(left_vec.iter().cloned())\n            .collect();\n        assert_eq!(&out, expected);\n        assert_eq!(count, expected_count);\n    }\n\n    #[test]\n    fn test_carry_slop_intersection() {\n        test_carry_slop_intersection_aux(&[&[1], &[]], &[], 1, 0);\n        test_carry_slop_intersection_aux(&[&[1], &[2]], &[(1, 1), (1, 2)], 1, 1);\n        test_carry_slop_intersection_aux(&[&[1], &[3]], &[], 1, 0);\n        test_carry_slop_intersection_aux(&[&[1], &[2]], &[(1, 1), (1, 2)], 1, 1);\n\n        // The order may still matter\n        test_carry_slop_intersection_aux(&[&[1], &[2], &[2]], &[(1, 2)], 1, 1);\n        test_carry_slop_intersection_aux(&[&[2], &[1], &[2]], &[(1, 2)], 1, 1);\n        test_carry_slop_intersection_aux(&[&[2], &[2], &[1]], &[(1, 1), (1, 2)], 1, 1);\n\n        test_carry_slop_intersection_aux(&[&[2], &[2], &[1], &[2]], &[(1, 2)], 1, 1);\n        test_carry_slop_intersection_aux(&[&[1], &[2], &[2], &[2]], &[(1, 2)], 1, 1);\n\n        test_carry_slop_intersection_aux(&[&[1], &[2], &[1]], &[(1, 1)], 1, 1);\n\n        test_carry_slop_intersection_aux(&[&[11], &[10, 12]], &[(1, 10), (1, 11), (1, 12)], 1, 1);\n        test_carry_slop_intersection_aux(&[&[10, 12], &[11]], &[(1, 10), (1, 11), (1, 12)], 1, 1);\n\n        test_carry_slop_intersection_aux(\n            &[&[5, 7, 11], &[1, 5, 10, 12]],\n            &[(0, 5), (1, 10), (1, 11), (1, 12)],\n            1,\n            2,\n        );\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use test::Bencher;\n\n    use super::*;\n\n    #[bench]\n    fn bench_intersection_short_slop_carrying(b: &mut Bencher) {\n        let mut left = Vec::new();\n        let mut left_slops = Vec::new();\n        let mut buffer = Vec::new();\n        let mut slop_buffer = Vec::new();\n        b.iter(|| {\n            left.clear();\n            left.extend_from_slice(&[1, 5, 10, 12]);\n            left_slops.extend_from_slice(&[0, 0, 0, 0]);\n            let right = [5, 7];\n            intersection(&mut left, &right);\n\n            intersection_count_with_carrying_slop(\n                &mut left,\n                &mut left_slops,\n                &right,\n                2,\n                true,\n                &mut buffer,\n                &mut slop_buffer,\n            )\n        });\n    }\n\n    #[bench]\n    fn bench_intersection_short(b: &mut Bencher) {\n        let mut left = Vec::new();\n        b.iter(|| {\n            left.clear();\n            left.extend_from_slice(&[1, 5, 10, 12]);\n            let right = [5, 7];\n            intersection(&mut left, &right);\n        });\n    }\n\n    #[bench]\n    fn bench_intersection_medium_slop_carrying(b: &mut Bencher) {\n        let mut left = Vec::new();\n        let mut left_slops: Vec<u8> = Vec::new();\n        let mut buffer = Vec::new();\n        let mut slop_buffer = Vec::new();\n        let left_data: Vec<u32> = (0..100).collect();\n        let left_slop_data: Vec<u8> = (0..100).map(|_| 0).collect();\n\n        b.iter(|| {\n            left.clear();\n            left.extend_from_slice(&left_data);\n            left_slops.clear();\n            left_slops.extend_from_slice(&left_slop_data);\n            let right = [5, 7, 55, 200];\n\n            intersection_count_with_carrying_slop(\n                &mut left,\n                &mut left_slops,\n                &right,\n                2,\n                true,\n                &mut buffer,\n                &mut slop_buffer,\n            )\n        });\n    }\n\n    #[bench]\n    fn bench_intersection_medium_slop(b: &mut Bencher) {\n        let mut left = Vec::new();\n        let left_data: Vec<u32> = (0..100).collect();\n\n        b.iter(|| {\n            left.clear();\n            left.extend_from_slice(&left_data);\n            let right = [5, 7, 55, 200];\n            intersection_count_with_slop(&mut left, &right[..], 2, true) as u32\n        });\n    }\n\n    #[bench]\n    fn bench_intersection_medium(b: &mut Bencher) {\n        let mut left = Vec::new();\n        let left_data: Vec<u32> = (0..100).collect();\n        b.iter(|| {\n            left.clear();\n            left.extend_from_slice(&left_data);\n            let right = [5, 7, 55, 200];\n            intersection(&mut left, &right);\n        });\n    }\n\n    #[bench]\n    fn bench_intersection_count_short(b: &mut Bencher) {\n        b.iter(|| {\n            let left = [1, 5, 10, 12];\n            let right = [5, 7];\n            intersection_count(&left, &right);\n        });\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/phrase_weight.rs",
    "content": "use super::PhraseScorer;\nuse crate::fieldnorm::FieldNormReader;\nuse crate::index::SegmentReader;\nuse crate::postings::SegmentPostings;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::explanation::does_not_match;\nuse crate::query::{EmptyScorer, Explanation, Scorer, Weight};\nuse crate::schema::{IndexRecordOption, Term};\nuse crate::{DocId, DocSet, Score};\n\npub struct PhraseWeight {\n    phrase_terms: Vec<(usize, Term)>,\n    similarity_weight_opt: Option<Bm25Weight>,\n    slop: u32,\n}\n\nimpl PhraseWeight {\n    /// Creates a new phrase weight.\n    /// If `similarity_weight_opt` is None, then scoring is disabled\n    pub fn new(\n        phrase_terms: Vec<(usize, Term)>,\n        similarity_weight_opt: Option<Bm25Weight>,\n    ) -> PhraseWeight {\n        let slop = 0;\n        PhraseWeight {\n            phrase_terms,\n            similarity_weight_opt,\n            slop,\n        }\n    }\n\n    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {\n        let field = self.phrase_terms[0].1.field();\n        if self.similarity_weight_opt.is_some() {\n            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {\n                return Ok(fieldnorm_reader);\n            }\n        }\n        Ok(FieldNormReader::constant(reader.max_doc(), 1))\n    }\n\n    pub(crate) fn phrase_scorer(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<Option<PhraseScorer<SegmentPostings>>> {\n        let similarity_weight_opt = self\n            .similarity_weight_opt\n            .as_ref()\n            .map(|similarity_weight| similarity_weight.boost_by(boost));\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let mut term_postings_list = Vec::new();\n        for &(offset, ref term) in &self.phrase_terms {\n            if let Some(postings) = reader\n                .inverted_index(term.field())?\n                .read_postings(term, IndexRecordOption::WithFreqsAndPositions)?\n            {\n                term_postings_list.push((offset, postings));\n            } else {\n                return Ok(None);\n            }\n        }\n        Ok(Some(PhraseScorer::new(\n            term_postings_list,\n            similarity_weight_opt,\n            fieldnorm_reader,\n            self.slop,\n        )))\n    }\n\n    pub fn slop(&mut self, slop: u32) {\n        self.slop = slop;\n    }\n}\n\nimpl Weight for PhraseWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        if let Some(scorer) = self.phrase_scorer(reader, boost)? {\n            Ok(Box::new(scorer))\n        } else {\n            Ok(Box::new(EmptyScorer))\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let scorer_opt = self.phrase_scorer(reader, 1.0)?;\n        if scorer_opt.is_none() {\n            return Err(does_not_match(doc));\n        }\n        let mut scorer = scorer_opt.unwrap();\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);\n        let phrase_count = scorer.phrase_count();\n        let mut explanation = Explanation::new(\"Phrase Scorer\", scorer.score());\n        if let Some(similarity_weight) = self.similarity_weight_opt.as_ref() {\n            explanation.add_detail(similarity_weight.explain(fieldnorm_id, phrase_count));\n        }\n        Ok(explanation)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::super::tests::create_index;\n    use crate::docset::TERMINATED;\n    use crate::query::{EnableScoring, PhraseQuery};\n    use crate::{DocSet, Term};\n\n    #[test]\n    pub fn test_phrase_count() -> crate::Result<()> {\n        let index = create_index(&[\"a c\", \"a a b d a b c\", \" a b\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = PhraseQuery::new(vec![\n            Term::from_field_text(text_field, \"a\"),\n            Term::from_field_text(text_field, \"b\"),\n        ]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/regex_phrase_query.rs",
    "content": "use super::regex_phrase_weight::RegexPhraseWeight;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::{EnableScoring, Query, Weight};\nuse crate::schema::{Field, IndexRecordOption, Term, Type};\n\n/// `RegexPhraseQuery` matches a specific sequence of regex queries.\n///\n/// For instance, the phrase query for `\"pa.* time\"` will match\n/// the sentence:\n///\n/// **Alan just got a part time job.**\n///\n/// On the other hand it will not match the sentence.\n///\n/// **This is my favorite part of the job.**\n///\n/// [Slop](RegexPhraseQuery::set_slop) allows leniency in term proximity\n/// for some performance trade-off.\n///\n/// Using a `RegexPhraseQuery` on a field requires positions\n/// to be indexed for this field.\n#[derive(Clone, Debug)]\npub struct RegexPhraseQuery {\n    field: Field,\n    phrase_terms: Vec<(usize, String)>,\n    slop: u32,\n    max_expansions: u32,\n}\n\n/// Transform a wildcard query to a regex string.\n///\n/// `AB*CD` for example is converted to `AB.*CD`\n///\n/// All other chars are regex escaped.\npub fn wildcard_query_to_regex_str(term: &str) -> String {\n    regex::escape(term).replace(r\"\\*\", \".*\")\n}\n\nimpl RegexPhraseQuery {\n    /// Creates a new `RegexPhraseQuery` given a list of terms.\n    ///\n    /// There must be at least two terms, and all terms\n    /// must belong to the same field.\n    ///\n    /// Offset for each term will be same as index in the Vector\n    pub fn new(field: Field, terms: Vec<String>) -> RegexPhraseQuery {\n        let terms_with_offset = terms.into_iter().enumerate().collect();\n        RegexPhraseQuery::new_with_offset(field, terms_with_offset)\n    }\n\n    /// Creates a new `RegexPhraseQuery` given a list of terms and their offsets.\n    ///\n    /// Can be used to provide custom offset for each term.\n    pub fn new_with_offset(field: Field, terms: Vec<(usize, String)>) -> RegexPhraseQuery {\n        RegexPhraseQuery::new_with_offset_and_slop(field, terms, 0)\n    }\n\n    /// Creates a new `RegexPhraseQuery` given a list of terms, their offsets and a slop\n    pub fn new_with_offset_and_slop(\n        field: Field,\n        mut terms: Vec<(usize, String)>,\n        slop: u32,\n    ) -> RegexPhraseQuery {\n        assert!(\n            terms.len() > 1,\n            \"A phrase query is required to have strictly more than one term.\"\n        );\n        terms.sort_by_key(|&(offset, _)| offset);\n        RegexPhraseQuery {\n            field,\n            phrase_terms: terms,\n            slop,\n            max_expansions: 1 << 14,\n        }\n    }\n\n    /// Slop allowed for the phrase.\n    ///\n    /// The query will match if its terms are separated by `slop` terms at most.\n    /// The slop can be considered a budget between all terms.\n    /// E.g. \"A B C\" with slop 1 allows \"A X B C\", \"A B X C\", but not \"A X B X C\".\n    ///\n    /// Transposition costs 2, e.g. \"A B\" with slop 1 will not match \"B A\" but it would with slop 2\n    /// Transposition is not a special case, in the example above A is moved 1 position and B is\n    /// moved 1 position, so the slop is 2.\n    ///\n    /// As a result slop works in both directions, so the order of the terms may changed as long as\n    /// they respect the slop.\n    ///\n    /// By default the slop is 0 meaning query terms need to be adjacent.\n    pub fn set_slop(&mut self, value: u32) {\n        self.slop = value;\n    }\n\n    /// Sets the max expansions a regex term can match. The limit will be over all terms.\n    /// After the limit is hit an error will be returned.\n    pub fn set_max_expansions(&mut self, value: u32) {\n        self.max_expansions = value;\n    }\n\n    /// The [`Field`] this `RegexPhraseQuery` is targeting.\n    pub fn field(&self) -> Field {\n        self.field\n    }\n\n    /// `Term`s in the phrase without the associated offsets.\n    pub fn phrase_terms(&self) -> Vec<Term> {\n        self.phrase_terms\n            .iter()\n            .map(|(_, term)| Term::from_field_text(self.field, term))\n            .collect::<Vec<Term>>()\n    }\n\n    /// Returns the [`RegexPhraseWeight`] for the given phrase query given a specific `searcher`.\n    ///\n    /// This function is the same as [`Query::weight()`] except it returns\n    /// a specialized type [`RegexPhraseWeight`] instead of a Boxed trait.\n    pub(crate) fn regex_phrase_weight(\n        &self,\n        enable_scoring: EnableScoring<'_>,\n    ) -> crate::Result<RegexPhraseWeight> {\n        let schema = enable_scoring.schema();\n        let field_type = schema.get_field_entry(self.field).field_type().value_type();\n        if field_type != Type::Str {\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"RegexPhraseQuery can only be used with a field of type text currently, but got \\\n                 {field_type:?}\"\n            )));\n        }\n\n        let field_entry = schema.get_field_entry(self.field);\n        let has_positions = field_entry\n            .field_type()\n            .get_index_record_option()\n            .map(IndexRecordOption::has_positions)\n            .unwrap_or(false);\n        if !has_positions {\n            let field_name = field_entry.name();\n            return Err(crate::TantivyError::SchemaError(format!(\n                \"Applied phrase query on field {field_name:?}, which does not have positions \\\n                 indexed\"\n            )));\n        }\n        let terms = self.phrase_terms();\n        let bm25_weight_opt = match enable_scoring {\n            EnableScoring::Enabled {\n                statistics_provider,\n                ..\n            } => Some(Bm25Weight::for_terms(statistics_provider, &terms)?),\n            EnableScoring::Disabled { .. } => None,\n        };\n        let weight = RegexPhraseWeight::new(\n            self.field,\n            self.phrase_terms.clone(),\n            bm25_weight_opt,\n            self.max_expansions,\n            self.slop,\n        );\n        Ok(weight)\n    }\n}\n\nimpl Query for RegexPhraseQuery {\n    /// Create the weight associated with a query.\n    ///\n    /// See [`Weight`].\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let phrase_weight = self.regex_phrase_weight(enable_scoring)?;\n        Ok(Box::new(phrase_weight))\n    }\n}\n"
  },
  {
    "path": "src/query/phrase_query/regex_phrase_weight.rs",
    "content": "use std::sync::Arc;\n\nuse common::BitSet;\nuse tantivy_fst::Regex;\n\nuse super::PhraseScorer;\nuse crate::fieldnorm::FieldNormReader;\nuse crate::index::SegmentReader;\nuse crate::postings::{LoadedPostings, Postings, SegmentPostings, TermInfo};\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::explanation::does_not_match;\nuse crate::query::union::{BitSetPostingUnion, SimpleUnion};\nuse crate::query::{AutomatonWeight, BitSetDocSet, EmptyScorer, Explanation, Scorer, Weight};\nuse crate::schema::{Field, IndexRecordOption};\nuse crate::{DocId, DocSet, InvertedIndexReader, Score};\n\ntype UnionType = SimpleUnion<Box<dyn Postings + 'static>>;\n\n/// The `RegexPhraseWeight` is the weight associated to a regex phrase query.\n/// See RegexPhraseWeight::get_union_from_term_infos for some design decisions.\npub struct RegexPhraseWeight {\n    field: Field,\n    phrase_terms: Vec<(usize, String)>,\n    similarity_weight_opt: Option<Bm25Weight>,\n    slop: u32,\n    max_expansions: u32,\n}\n\nimpl RegexPhraseWeight {\n    /// Creates a new phrase weight.\n    /// If `similarity_weight_opt` is None, then scoring is disabled\n    pub fn new(\n        field: Field,\n        phrase_terms: Vec<(usize, String)>,\n        similarity_weight_opt: Option<Bm25Weight>,\n        max_expansions: u32,\n        slop: u32,\n    ) -> RegexPhraseWeight {\n        RegexPhraseWeight {\n            field,\n            phrase_terms,\n            similarity_weight_opt,\n            slop,\n            max_expansions,\n        }\n    }\n\n    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {\n        if self.similarity_weight_opt.is_some() {\n            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(self.field)? {\n                return Ok(fieldnorm_reader);\n            }\n        }\n        Ok(FieldNormReader::constant(reader.max_doc(), 1))\n    }\n\n    pub(crate) fn phrase_scorer(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<Option<PhraseScorer<UnionType>>> {\n        let similarity_weight_opt = self\n            .similarity_weight_opt\n            .as_ref()\n            .map(|similarity_weight| similarity_weight.boost_by(boost));\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let mut posting_lists = Vec::new();\n        let inverted_index = reader.inverted_index(self.field)?;\n        let mut num_terms = 0;\n        for &(offset, ref term) in &self.phrase_terms {\n            let regex = Regex::new(term)\n                .map_err(|e| crate::TantivyError::InvalidArgument(format!(\"Invalid regex: {e}\")))?;\n\n            let automaton: AutomatonWeight<Regex> =\n                AutomatonWeight::new(self.field, Arc::new(regex));\n            let term_infos = automaton.get_match_term_infos(reader)?;\n            // If term_infos is empty, the phrase can not match any documents.\n            if term_infos.is_empty() {\n                return Ok(None);\n            }\n            num_terms += term_infos.len();\n            if num_terms > self.max_expansions as usize {\n                return Err(crate::TantivyError::InvalidArgument(format!(\n                    \"Phrase query exceeded max expansions {num_terms}\"\n                )));\n            }\n            let union = Self::get_union_from_term_infos(&term_infos, reader, &inverted_index)?;\n\n            posting_lists.push((offset, union));\n        }\n\n        Ok(Some(PhraseScorer::new(\n            posting_lists,\n            similarity_weight_opt,\n            fieldnorm_reader,\n            self.slop,\n        )))\n    }\n\n    /// Add all docs of the term to the docset\n    fn add_to_bitset(\n        inverted_index: &InvertedIndexReader,\n        term_info: &TermInfo,\n        doc_bitset: &mut BitSet,\n    ) -> crate::Result<()> {\n        let mut block_segment_postings = inverted_index\n            .read_block_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;\n        loop {\n            let docs = block_segment_postings.docs();\n            if docs.is_empty() {\n                break;\n            }\n            for &doc in docs {\n                doc_bitset.insert(doc);\n            }\n            block_segment_postings.advance();\n        }\n        Ok(())\n    }\n\n    /// This function generates a union of document sets from multiple term information\n    /// (`TermInfo`).\n    ///\n    /// It uses bucketing based on term frequency to optimize query performance and memory usage.\n    /// The terms are divided into buckets based on their document frequency (the number of\n    /// documents they appear in).\n    ///\n    /// ### Bucketing Strategy:\n    /// Once a bucket contains more than 512 terms, it is moved to the end of the list and replaced\n    /// with a new empty bucket.\n    ///\n    /// - **Sparse Term Buckets**: Terms with document frequency `< 100`.\n    ///\n    ///   Each sparse bucket contains:\n    ///   - A `BitSet` to efficiently track which document IDs are present in the bucket, which is\n    ///     used to drive the `DocSet`.\n    ///   - A `Vec<LoadedPostings>` to store the postings for each term in that bucket.\n    ///\n    /// - **Other Term Buckets**:\n    ///   - **Bucket 0**: Terms appearing in less than `0.1%` of documents.\n    ///   - **Bucket 1**: Terms appearing in `0.1%` to `1%` of documents.\n    ///   - **Bucket 2**: Terms appearing in `1%` to `10%` of documents.\n    ///   - **Bucket 3**: Terms appearing in more than `10%` of documents.\n    ///\n    ///   Each bucket contains:\n    ///   - A `BitSet` to efficiently track which document IDs are present in the bucket.\n    ///   - A `Vec<SegmentPostings>` to store the postings for each term in that bucket.\n    ///\n    /// ### Design Choices:\n    /// The main cost for a _unbucketed_ regex phrase query with a medium/high amount of terms is\n    /// the `append_positions_with_offset` from `Postings`.\n    /// We don't know which docsets hit, so we need to scan all of them to check if they contain the\n    /// docid.\n    /// The bucketing strategy groups less common DocSets together, so we can rule out the\n    /// whole docset group in many cases.\n    ///\n    /// E.g. consider the phrase \"th* world\"\n    /// It contains the term \"the\", which may occur in almost all documents.\n    /// It may also contain 10_000s very rare terms like \"theologian\".\n    ///\n    /// For very low-frequency terms (sparse terms), we use `LoadedPostings` and aggregate\n    /// their document IDs into a `BitSet`, which is more memory-efficient than using\n    /// `SegmentPostings`. E.g. 100_000 terms with SegmentPostings would consume 184MB.\n    /// `SegmentPostings` uses memory equivalent to 460 docids. The 100 docs limit should be\n    /// fine as long as a term doesn't have too many positions per doc.\n    ///\n    /// ### Future Optimization:\n    /// A larger performance improvement would be an additional partitioning of the space\n    /// vertically of u16::MAX blocks, where we mark which docset ord has values in each block.\n    /// E.g. partitioning in a index with 5 million documents this would reduce the number of\n    /// docsets to scan to around 1/20 in the sparse term bucket where the terms only have a few\n    /// docs. For higher cardinality buckets this is irrelevant as they are in most blocks.\n    ///\n    /// Use Roaring Bitmaps for sparse terms. The full bitvec is main memory consumer currently.\n    pub(crate) fn get_union_from_term_infos(\n        term_infos: &[TermInfo],\n        reader: &SegmentReader,\n        inverted_index: &InvertedIndexReader,\n    ) -> crate::Result<UnionType> {\n        let max_doc = reader.max_doc();\n\n        // Buckets for sparse terms\n        let mut sparse_buckets: Vec<(BitSet, Vec<LoadedPostings>)> =\n            vec![(BitSet::with_max_value(max_doc), Vec::new())];\n\n        // Buckets for other terms based on document frequency percentages:\n        // - Bucket 0: Terms appearing in less than 0.1% of documents\n        // - Bucket 1: Terms appearing in 0.1% to 1% of documents\n        // - Bucket 2: Terms appearing in 1% to 10% of documents\n        // - Bucket 3: Terms appearing in more than 10% of documents\n        let mut buckets: Vec<(BitSet, Vec<SegmentPostings>)> = (0..4)\n            .map(|_| (BitSet::with_max_value(max_doc), Vec::new()))\n            .collect();\n\n        const SPARSE_TERM_DOC_THRESHOLD: u32 = 100;\n\n        for term_info in term_infos {\n            let mut term_posting = inverted_index\n                .read_postings_from_terminfo(term_info, IndexRecordOption::WithFreqsAndPositions)?;\n            let num_docs = term_posting.doc_freq();\n\n            if num_docs < SPARSE_TERM_DOC_THRESHOLD {\n                let current_bucket = &mut sparse_buckets[0];\n                Self::add_to_bitset(inverted_index, term_info, &mut current_bucket.0)?;\n                let docset = LoadedPostings::load(&mut term_posting);\n                current_bucket.1.push(docset);\n\n                // Move the bucket to the end if the term limit is reached\n                if current_bucket.1.len() == 512 {\n                    sparse_buckets.push((BitSet::with_max_value(max_doc), Vec::new()));\n                    let end_index = sparse_buckets.len() - 1;\n                    sparse_buckets.swap(0, end_index);\n                }\n            } else {\n                // Calculate the percentage of documents the term appears in\n                let doc_freq_percentage = (num_docs as f32) / (max_doc as f32) * 100.0;\n\n                // Determine the appropriate bucket based on percentage thresholds\n                let bucket_index = if doc_freq_percentage < 0.1 {\n                    0\n                } else if doc_freq_percentage < 1.0 {\n                    1\n                } else if doc_freq_percentage < 10.0 {\n                    2\n                } else {\n                    3\n                };\n                let bucket = &mut buckets[bucket_index];\n\n                // Add term postings to the appropriate bucket\n                Self::add_to_bitset(inverted_index, term_info, &mut bucket.0)?;\n                bucket.1.push(term_posting);\n\n                // Move the bucket to the end if the term limit is reached\n                if bucket.1.len() == 512 {\n                    buckets.push((BitSet::with_max_value(max_doc), Vec::new()));\n                    let end_index = buckets.len() - 1;\n                    buckets.swap(bucket_index, end_index);\n                }\n            }\n        }\n\n        // Build unions for sparse term buckets\n        let sparse_term_docsets: Vec<_> = sparse_buckets\n            .into_iter()\n            .filter(|(_, postings)| !postings.is_empty())\n            .map(|(bitset, postings)| {\n                BitSetPostingUnion::build(postings, BitSetDocSet::from(bitset))\n            })\n            .collect();\n        let sparse_term_unions = SimpleUnion::build(sparse_term_docsets);\n\n        // Build unions for other term buckets\n        let bitset_unions_per_bucket: Vec<_> = buckets\n            .into_iter()\n            .filter(|(_, postings)| !postings.is_empty())\n            .map(|(bitset, postings)| {\n                BitSetPostingUnion::build(postings, BitSetDocSet::from(bitset))\n            })\n            .collect();\n        let other_union = SimpleUnion::build(bitset_unions_per_bucket);\n\n        let union: SimpleUnion<Box<dyn Postings + 'static>> =\n            SimpleUnion::build(vec![Box::new(sparse_term_unions), Box::new(other_union)]);\n\n        // Return a union of sparse term unions and other term unions\n        Ok(union)\n    }\n}\n\nimpl Weight for RegexPhraseWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        if let Some(scorer) = self.phrase_scorer(reader, boost)? {\n            Ok(Box::new(scorer))\n        } else {\n            Ok(Box::new(EmptyScorer))\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let scorer_opt = self.phrase_scorer(reader, 1.0)?;\n        if scorer_opt.is_none() {\n            return Err(does_not_match(doc));\n        }\n        let mut scorer = scorer_opt.unwrap();\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);\n        let phrase_count = scorer.phrase_count();\n        let mut explanation = Explanation::new(\"Phrase Scorer\", scorer.score());\n        if let Some(similarity_weight) = self.similarity_weight_opt.as_ref() {\n            explanation.add_detail(similarity_weight.explain(fieldnorm_id, phrase_count));\n        }\n        Ok(explanation)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use proptest::prelude::*;\n    use rand::seq::SliceRandom;\n\n    use super::super::tests::create_index;\n    use crate::docset::TERMINATED;\n    use crate::query::{wildcard_query_to_regex_str, EnableScoring, RegexPhraseQuery};\n    use crate::DocSet;\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(50))]\n        #[test]\n        fn test_phrase_regex_with_random_strings(mut random_strings in proptest::collection::vec(\"[c-z ]{0,10}\", 1..100), num_occurrences in 1..150_usize) {\n            let mut rng = rand::rng();\n\n            // Insert \"aaa ccc\" the specified number of times into the list\n            for _ in 0..num_occurrences {\n                random_strings.push(\"aaa ccc\".to_string());\n            }\n            // Shuffle the list, which now contains random strings and the inserted \"aaa ccc\"\n            random_strings.shuffle(&mut rng);\n\n            // Compute the positions of \"aaa ccc\" after the shuffle\n            let aaa_ccc_positions: Vec<usize> = random_strings\n                .iter()\n                .enumerate()\n                .filter_map(|(idx, s)| if s == \"aaa ccc\" { Some(idx) } else { None })\n                .collect();\n\n            // Create the index with random strings and the fixed string \"aaa ccc\"\n            let index = create_index(&random_strings.iter().map(AsRef::as_ref).collect::<Vec<&str>>())?;\n            let schema = index.schema();\n            let text_field = schema.get_field(\"text\").unwrap();\n            let searcher = index.reader()?.searcher();\n\n            let phrase_query = RegexPhraseQuery::new(text_field, vec![wildcard_query_to_regex_str(\"a*\"), wildcard_query_to_regex_str(\"c*\")]);\n\n            let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n            let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n            let mut phrase_scorer = phrase_weight\n                .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n                .unwrap();\n\n            // Check if the scorer returns the correct document positions for \"aaa ccc\"\n            for expected_doc in aaa_ccc_positions {\n                prop_assert_eq!(phrase_scorer.doc(), expected_doc as u32);\n                prop_assert_eq!(phrase_scorer.phrase_count(), 1);\n                phrase_scorer.advance();\n            }\n            prop_assert_eq!(phrase_scorer.advance(), TERMINATED);\n        }\n    }\n\n    #[test]\n    pub fn test_phrase_count() -> crate::Result<()> {\n        let index = create_index(&[\"a c\", \"a a b d a b c\", \" a b\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = RegexPhraseQuery::new(text_field, vec![\"a\".into(), \"b\".into()]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_wildcard() -> crate::Result<()> {\n        let index = create_index(&[\"a c\", \"a aa b d ad b c\", \" ac b\", \"bac b\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = RegexPhraseQuery::new(text_field, vec![\"a.*\".into(), \"b\".into()]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_regex() -> crate::Result<()> {\n        let index = create_index(&[\"ba b\", \"a aa b d ad b c\", \"bac b\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = RegexPhraseQuery::new(text_field, vec![\"b?a.*\".into(), \"b\".into()]);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 0);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), 1);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), 2);\n        assert_eq!(phrase_scorer.doc(), 2);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_regex_with_slop() -> crate::Result<()> {\n        let index = create_index(&[\"aaa bbb ccc ___ abc ddd bbb ccc\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let mut phrase_query = RegexPhraseQuery::new(text_field, vec![\"a.*\".into(), \"c.*\".into()]);\n        phrase_query.set_slop(1);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 0);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n\n        phrase_query.set_slop(2);\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 0);\n        assert_eq!(phrase_scorer.phrase_count(), 2);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_phrase_regex_double_wildcard() -> crate::Result<()> {\n        let index = create_index(&[\"baaab bccccb\"])?;\n        let schema = index.schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        let searcher = index.reader()?.searcher();\n        let phrase_query = RegexPhraseQuery::new(\n            text_field,\n            vec![\n                wildcard_query_to_regex_str(\"*a*\"),\n                wildcard_query_to_regex_str(\"*c*\"),\n            ],\n        );\n        let enable_scoring = EnableScoring::enabled_from_searcher(&searcher);\n        let phrase_weight = phrase_query.regex_phrase_weight(enable_scoring).unwrap();\n        let mut phrase_scorer = phrase_weight\n            .phrase_scorer(searcher.segment_reader(0u32), 1.0)?\n            .unwrap();\n        assert_eq!(phrase_scorer.doc(), 0);\n        assert_eq!(phrase_scorer.phrase_count(), 1);\n        assert_eq!(phrase_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/query.rs",
    "content": "use std::fmt;\n\nuse downcast_rs::impl_downcast;\n\nuse super::bm25::Bm25StatisticsProvider;\nuse super::Weight;\nuse crate::core::searcher::Searcher;\nuse crate::query::Explanation;\nuse crate::schema::Schema;\nuse crate::{DocAddress, Term};\n\n/// Argument used in `Query::weight(..)`\n#[derive(Copy, Clone)]\npub enum EnableScoring<'a> {\n    /// Pass this to enable scoring.\n    Enabled {\n        /// The searcher to use during scoring.\n        searcher: &'a Searcher,\n\n        /// A [Bm25StatisticsProvider] used to compute BM25 scores.\n        ///\n        /// Normally this should be the [Searcher], but you can specify a custom\n        /// one to adjust the statistics.\n        statistics_provider: &'a dyn Bm25StatisticsProvider,\n    },\n    /// Pass this to disable scoring.\n    /// This can improve performance.\n    Disabled {\n        /// Schema is required.\n        schema: &'a Schema,\n        /// Searcher should be provided if available.\n        searcher_opt: Option<&'a Searcher>,\n    },\n}\n\nimpl<'a> EnableScoring<'a> {\n    /// Create using [Searcher] with scoring enabled.\n    pub fn enabled_from_searcher(searcher: &'a Searcher) -> EnableScoring<'a> {\n        EnableScoring::Enabled {\n            searcher,\n            statistics_provider: searcher,\n        }\n    }\n\n    /// Create using a custom [Bm25StatisticsProvider] with scoring enabled.\n    pub fn enabled_from_statistics_provider(\n        statistics_provider: &'a dyn Bm25StatisticsProvider,\n        searcher: &'a Searcher,\n    ) -> EnableScoring<'a> {\n        EnableScoring::Enabled {\n            statistics_provider,\n            searcher,\n        }\n    }\n\n    /// Create using [Searcher] with scoring disabled.\n    pub fn disabled_from_searcher(searcher: &'a Searcher) -> EnableScoring<'a> {\n        EnableScoring::Disabled {\n            schema: searcher.schema(),\n            searcher_opt: Some(searcher),\n        }\n    }\n\n    /// Create using [Schema] with scoring disabled.\n    pub fn disabled_from_schema(schema: &'a Schema) -> EnableScoring<'a> {\n        Self::Disabled {\n            schema,\n            searcher_opt: None,\n        }\n    }\n\n    /// Returns the searcher if available.\n    pub fn searcher(&self) -> Option<&Searcher> {\n        match self {\n            EnableScoring::Enabled { searcher, .. } => Some(*searcher),\n            EnableScoring::Disabled { searcher_opt, .. } => searcher_opt.to_owned(),\n        }\n    }\n\n    /// Returns the schema.\n    pub fn schema(&self) -> &Schema {\n        match self {\n            EnableScoring::Enabled { searcher, .. } => searcher.schema(),\n            EnableScoring::Disabled { schema, .. } => schema,\n        }\n    }\n\n    /// Returns true if the scoring is enabled.\n    pub fn is_scoring_enabled(&self) -> bool {\n        matches!(self, EnableScoring::Enabled { .. })\n    }\n}\n\n/// The `Query` trait defines a set of documents and a scoring method\n/// for those documents.\n///\n/// The `Query` trait is in charge of defining :\n///\n/// - a set of documents\n/// - a way to score these documents\n///\n/// When performing a [search](Searcher::search), these documents will then\n/// be pushed to a [`Collector`](crate::collector::Collector),\n/// which will in turn be in charge of deciding what to do with them.\n///\n/// Concretely, this scored docset is represented by the\n/// [`Scorer`] trait.\n///\n/// Because our index is actually split into segments, the\n/// query does not actually directly creates [`DocSet`](crate::DocSet) object.\n/// Instead, the query creates a [`Weight`] object for a given searcher.\n///\n/// The weight object, in turn, makes it possible to create\n/// a scorer for a specific [`SegmentReader`].\n///\n/// So to sum it up :\n/// - a `Query` is a recipe to define a set of documents as well the way to score them.\n/// - a [`Weight`] is this recipe tied to a specific [`Searcher`]. It may for instance hold\n///   statistics about the different term of the query. It is created by the query.\n/// - a [`Scorer`] is a cursor over the set of matching documents, for a specific [`SegmentReader`].\n///   It is created by the [`Weight`].\n///\n/// When implementing a new type of `Query`, it is normal to implement a\n/// dedicated `Query`, [`Weight`] and [`Scorer`].\n///\n/// [`Scorer`]: crate::query::Scorer\n/// [`SegmentReader`]: crate::SegmentReader\npub trait Query: QueryClone + Send + Sync + downcast_rs::Downcast + fmt::Debug {\n    /// Create the weight associated with a query.\n    ///\n    /// If scoring is not required, setting `scoring_enabled` to `false`\n    /// can increase performances.\n    ///\n    /// See [`Weight`].\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>>;\n\n    /// Returns an `Explanation` for the score of the document.\n    fn explain(&self, searcher: &Searcher, doc_address: DocAddress) -> crate::Result<Explanation> {\n        let weight = self.weight(EnableScoring::enabled_from_searcher(searcher))?;\n        let reader = searcher.segment_reader(doc_address.segment_ord);\n        weight.explain(reader, doc_address.doc_id)\n    }\n\n    /// Returns the number of documents matching the query.\n    fn count(&self, searcher: &Searcher) -> crate::Result<usize> {\n        let weight = self.weight(EnableScoring::disabled_from_searcher(searcher))?;\n        let mut result = 0;\n        for reader in searcher.segment_readers() {\n            result += weight.count(reader)? as usize;\n        }\n        Ok(result)\n    }\n\n    /// Extract all of the terms associated with the query and pass them to the\n    /// given closure.\n    ///\n    /// Each term is associated with a boolean indicating whether\n    /// positions are required or not.\n    ///\n    /// Note that there can be multiple instances of any given term\n    /// in a query and deduplication must be handled by the visitor.\n    fn query_terms<'a>(&'a self, _visitor: &mut dyn FnMut(&'a Term, bool)) {}\n}\n\n/// Implements `box_clone`.\npub trait QueryClone {\n    /// Returns a boxed clone of `self`.\n    fn box_clone(&self) -> Box<dyn Query>;\n}\n\nimpl<T> QueryClone for T\nwhere T: 'static + Query + Clone\n{\n    fn box_clone(&self) -> Box<dyn Query> {\n        Box::new(self.clone())\n    }\n}\n\nimpl Query for Box<dyn Query> {\n    fn weight(&self, enabled_scoring: EnableScoring) -> crate::Result<Box<dyn Weight>> {\n        self.as_ref().weight(enabled_scoring)\n    }\n\n    fn count(&self, searcher: &Searcher) -> crate::Result<usize> {\n        self.as_ref().count(searcher)\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        self.as_ref().query_terms(visitor);\n    }\n}\n\nimpl QueryClone for Box<dyn Query> {\n    fn box_clone(&self) -> Box<dyn Query> {\n        self.as_ref().box_clone()\n    }\n}\n\nimpl_downcast!(Query);\n"
  },
  {
    "path": "src/query/query_parser/logical_ast.rs",
    "content": "use std::fmt;\nuse std::ops::Bound;\nuse std::sync::Arc;\n\nuse tantivy_fst::Regex;\n\nuse crate::query::Occur;\nuse crate::schema::{Field, Term};\nuse crate::Score;\n\n#[derive(Clone)]\npub enum LogicalLiteral {\n    Term(Term),\n    Phrase {\n        terms: Vec<(usize, Term)>,\n        slop: u32,\n        prefix: bool,\n    },\n    Range {\n        lower: Bound<Term>,\n        upper: Bound<Term>,\n    },\n    Set {\n        elements: Vec<Term>,\n    },\n    All,\n    Regex {\n        pattern: Arc<Regex>,\n        field: Field,\n    },\n}\n\npub enum LogicalAst {\n    Clause(Vec<(Occur, LogicalAst)>),\n    Leaf(Box<LogicalLiteral>),\n    Boost(Box<LogicalAst>, Score),\n}\n\nimpl LogicalAst {\n    pub fn boost(self, boost: Score) -> LogicalAst {\n        if (boost - 1.0).abs() < Score::EPSILON {\n            self\n        } else {\n            LogicalAst::Boost(Box::new(self), boost)\n        }\n    }\n\n    // TODO: Move to rewrite_ast in query_grammar\n    pub fn simplify(self) -> LogicalAst {\n        match self {\n            LogicalAst::Clause(clauses) => {\n                let mut new_clauses: Vec<(Occur, LogicalAst)> = Vec::new();\n\n                for (occur, sub_ast) in clauses {\n                    let simplified_sub_ast = sub_ast.simplify();\n\n                    // If clauses below have the same `Occur`, we can pull them up\n                    match simplified_sub_ast {\n                        LogicalAst::Clause(sub_clauses)\n                            if (occur == Occur::Should || occur == Occur::Must)\n                                && sub_clauses.iter().all(|(o, _)| *o == occur) =>\n                        {\n                            for sub_clause in sub_clauses {\n                                new_clauses.push(sub_clause);\n                            }\n                        }\n                        _ => new_clauses.push((occur, simplified_sub_ast)),\n                    }\n                }\n\n                LogicalAst::Clause(new_clauses)\n            }\n            LogicalAst::Leaf(_) | LogicalAst::Boost(_, _) => self,\n        }\n    }\n}\n\nfn occur_letter(occur: Occur) -> &'static str {\n    match occur {\n        Occur::Must => \"+\",\n        Occur::MustNot => \"-\",\n        Occur::Should => \"\",\n    }\n}\n\nimpl fmt::Debug for LogicalAst {\n    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        match *self {\n            LogicalAst::Clause(ref clause) => {\n                if clause.is_empty() {\n                    write!(formatter, \"<emptyclause>\")?;\n                } else {\n                    let (occur, subquery) = &clause[0];\n                    write!(formatter, \"({}{subquery:?}\", occur_letter(*occur))?;\n                    for (occur, subquery) in &clause[1..] {\n                        write!(formatter, \" {}{subquery:?}\", occur_letter(*occur))?;\n                    }\n                    formatter.write_str(\")\")?;\n                }\n                Ok(())\n            }\n            LogicalAst::Boost(ref ast, boost) => write!(formatter, \"{ast:?}^{boost}\"),\n            LogicalAst::Leaf(ref literal) => write!(formatter, \"{literal:?}\"),\n        }\n    }\n}\n\nimpl From<LogicalLiteral> for LogicalAst {\n    fn from(literal: LogicalLiteral) -> LogicalAst {\n        LogicalAst::Leaf(Box::new(literal))\n    }\n}\n\nimpl fmt::Debug for LogicalLiteral {\n    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> Result<(), fmt::Error> {\n        match *self {\n            LogicalLiteral::Term(ref term) => write!(formatter, \"{term:?}\"),\n            LogicalLiteral::Phrase {\n                ref terms,\n                slop,\n                prefix,\n            } => {\n                write!(formatter, \"\\\"{terms:?}\\\"\")?;\n                if slop > 0 {\n                    write!(formatter, \"~{slop:?}\")\n                } else if prefix {\n                    write!(formatter, \"*\")\n                } else {\n                    Ok(())\n                }\n            }\n            LogicalLiteral::Range {\n                ref lower,\n                ref upper,\n                ..\n            } => write!(formatter, \"({lower:?} TO {upper:?})\"),\n            LogicalLiteral::Set { ref elements, .. } => {\n                const MAX_DISPLAYED: usize = 10;\n\n                write!(formatter, \"IN [\")?;\n                for (i, element) in elements.iter().enumerate() {\n                    if i == 0 {\n                        write!(formatter, \"{element:?}\")?;\n                    } else if i == MAX_DISPLAYED - 1 {\n                        write!(\n                            formatter,\n                            \", {element:?}, ... ({} more)\",\n                            elements.len() - i - 1\n                        )?;\n                        break;\n                    } else {\n                        write!(formatter, \", {element:?}\")?;\n                    }\n                }\n                write!(formatter, \"]\")\n            }\n            LogicalLiteral::All => write!(formatter, \"*\"),\n            LogicalLiteral::Regex {\n                ref pattern,\n                ref field,\n            } => write!(formatter, \"Regex({field:?}, {pattern:?})\"),\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/query_parser/mod.rs",
    "content": "mod query_parser;\n\npub mod logical_ast;\npub use self::query_parser::{QueryParser, QueryParserError};\n"
  },
  {
    "path": "src/query/query_parser/query_parser.rs",
    "content": "use std::net::{AddrParseError, IpAddr};\nuse std::num::{ParseFloatError, ParseIntError};\nuse std::ops::Bound;\nuse std::str::{FromStr, ParseBoolError};\nuse std::sync::Arc;\n\nuse base64::engine::general_purpose::STANDARD as BASE64;\nuse base64::Engine;\nuse itertools::Itertools;\nuse query_grammar::{UserInputAst, UserInputBound, UserInputLeaf, UserInputLiteral};\nuse rustc_hash::FxHashMap;\nuse tantivy_fst::Regex;\n\nuse super::logical_ast::*;\nuse crate::index::Index;\nuse crate::json_utils::convert_to_fast_value_and_append_to_json_term;\nuse crate::query::range_query::{is_type_valid_for_fastfield_range_query, RangeQuery};\nuse crate::query::{\n    AllQuery, BooleanQuery, BoostQuery, EmptyQuery, FuzzyTermQuery, Occur, PhrasePrefixQuery,\n    PhraseQuery, Query, RegexQuery, TermQuery, TermSetQuery,\n};\nuse crate::schema::{\n    Facet, FacetParseError, Field, FieldType, IndexRecordOption, IntoIpv6Addr, JsonObjectOptions,\n    Schema, Term, TextFieldIndexing, Type,\n};\nuse crate::time::format_description::well_known::Rfc3339;\nuse crate::time::OffsetDateTime;\nuse crate::tokenizer::{TextAnalyzer, TokenizerManager};\nuse crate::{DateTime, Score};\n\n/// Possible error that may happen when parsing a query.\n#[derive(Debug, PartialEq, Eq, Error)]\npub enum QueryParserError {\n    /// Error in the query syntax\n    #[error(\"Syntax Error: {0}\")]\n    SyntaxError(String),\n    /// This query is unsupported.\n    #[error(\"Unsupported query: {0}\")]\n    UnsupportedQuery(String),\n    /// The query references a field that is not in the schema\n    #[error(\"Field does not exist: '{0}'\")]\n    FieldDoesNotExist(String),\n    /// The query contains a term for a `u64` or `i64`-field, but the value\n    /// is neither.\n    #[error(\"Expected a valid integer: '{0:?}'\")]\n    ExpectedInt(#[from] ParseIntError),\n    /// The query contains a term for a bytes field, but the value is not valid\n    /// base64.\n    #[error(\"Expected base64: '{0:?}'\")]\n    ExpectedBase64(#[from] base64::DecodeError),\n    /// The query contains a term for a `f64`-field, but the value\n    /// is not a f64.\n    #[error(\"Invalid query: Only excluding terms given\")]\n    ExpectedFloat(#[from] ParseFloatError),\n    /// The query contains a term for a bool field, but the value\n    /// is not a bool.\n    #[error(\"Expected a bool value: '{0:?}'\")]\n    ExpectedBool(#[from] ParseBoolError),\n    /// It is forbidden queries that are only \"excluding\". (e.g. -title:pop)\n    #[error(\"Invalid query: Only excluding terms given\")]\n    AllButQueryForbidden,\n    /// If no default field is declared, running a query without any\n    /// field specified is forbidden.\n    #[error(\"No default field declared and no field specified in query\")]\n    NoDefaultFieldDeclared,\n    /// The field searched for is not declared\n    /// as indexed in the schema.\n    #[error(\"The field '{0}' is not declared as indexed\")]\n    FieldNotIndexed(String),\n    /// A phrase query was requested for a field that does not\n    /// have any positions indexed.\n    #[error(\"The field '{0}' does not have positions indexed\")]\n    FieldDoesNotHavePositionsIndexed(String),\n    /// A phrase-prefix query requires at least two terms\n    #[error(\n        \"The phrase '{phrase:?}' does not produce at least two terms using the tokenizer \\\n         '{tokenizer:?}'\"\n    )]\n    PhrasePrefixRequiresAtLeastTwoTerms {\n        /// The phrase which triggered the issue\n        phrase: String,\n        /// The tokenizer configured for the field\n        tokenizer: String,\n    },\n    /// The tokenizer for the given field is unknown\n    /// The two argument strings are the name of the field, the name of the tokenizer\n    #[error(\"The tokenizer '{tokenizer:?}' for the field '{field:?}' is unknown\")]\n    UnknownTokenizer {\n        /// The name of the tokenizer\n        tokenizer: String,\n        /// The field name\n        field: String,\n    },\n    /// The query contains a range query with a phrase as one of the bounds.\n    /// Only terms can be used as bounds.\n    #[error(\"A range query cannot have a phrase as one of the bounds\")]\n    RangeMustNotHavePhrase,\n    /// The format for the date field is not RFC 3339 compliant.\n    #[error(\"The date field has an invalid format\")]\n    DateFormatError(#[from] time::error::Parse),\n    /// The format for the facet field is invalid.\n    #[error(\"The facet field is malformed: {0}\")]\n    FacetFormatError(#[from] FacetParseError),\n    /// The format for the ip field is invalid.\n    #[error(\"The ip field is malformed: {0}\")]\n    IpFormatError(#[from] AddrParseError),\n}\n\n/// Recursively remove empty clause from the AST\n///\n/// Returns `None` if and only if the `logical_ast` ended up being empty.\nfn trim_ast(logical_ast: LogicalAst) -> Option<LogicalAst> {\n    match logical_ast {\n        LogicalAst::Clause(children) => {\n            let trimmed_children = children\n                .into_iter()\n                .flat_map(|(occur, child)| {\n                    trim_ast(child).map(|trimmed_child| (occur, trimmed_child))\n                })\n                .collect::<Vec<_>>();\n            if trimmed_children.is_empty() {\n                None\n            } else {\n                Some(LogicalAst::Clause(trimmed_children))\n            }\n        }\n        _ => Some(logical_ast),\n    }\n}\n\n/// Tantivy's Query parser\n///\n/// The language covered by the current parser is extremely simple.\n///\n/// * simple terms: \"e.g.: `Barack Obama` will be seen as a sequence of two tokens Barack and Obama.\n///   By default, the query parser will interpret this as a disjunction (see\n///   `.set_conjunction_by_default()`) and will match all documents that contains either \"Barack\" or\n///   \"Obama\" or both. Since we did not target a specific field, the query parser will look into the\n///   so-called default fields (as set up in the constructor).\n///\n///   Assuming that the default fields are `body` and `title`, and the query parser is set with\n///   conjunction as a default, our query will be interpreted as.\n///   `(body:Barack OR title:Barack) AND (title:Obama OR body:Obama)`.\n///   By default, all tokenized and indexed fields are default fields.\n///\n///   It is possible to explicitly target a field by prefixing the text by the `fieldname:`.\n///   Note this only applies to the term directly following.\n///   For instance, assuming the query parser is configured to use conjunction by default,\n///   `body:Barack Obama` is not interpreted as `body:Barack AND body:Obama` but as\n///   `body:Barack OR (body:Barack OR text:Obama)` .\n///\n/// * boolean operators `AND`, `OR`. `AND` takes precedence over `OR`, so that `a AND b OR c` is\n///   interpreted as `(a AND b) OR c`.\n///\n/// * In addition to the boolean operators, the `-`, `+` can help define. These operators are\n///   sufficient to express all queries using boolean operators. For instance `x AND y OR z` can be\n///   written (`(+x +y) z`). In addition, these operators can help define \"required optional\"\n///   queries. `(+x y)` matches the same document set as simply `x`, but `y` will help refining the\n///   score.\n///\n/// * negative terms: By prepending a term by a `-`, a term can be excluded from the search. This is\n///   useful for disambiguating a query. e.g. `apple -fruit`\n///\n/// * must terms: By prepending a term by a `+`, a term can be made required for the search.\n///\n/// * phrase terms: Quoted terms become phrase searches on fields that have positions indexed. e.g.,\n///   `title:\"Barack Obama\"` will only find documents that have \"barack\" immediately followed by\n///   \"obama\". Single quotes can also be used. If the text to be searched contains quotation mark,\n///   it is possible to escape them with a `\\`.\n///\n/// * range terms: Range searches can be done by specifying the start and end bound. These can be\n///   inclusive or exclusive. e.g., `title:[a TO c}` will find all documents whose title contains a\n///   word lexicographically between `a` and `c` (inclusive lower bound, exclusive upper bound).\n///   Inclusive bounds are `[]`, exclusive are `{}`.\n///\n/// * set terms: Using the `IN` operator, a field can be matched against a set of literals, e.g.\n///   `title: IN [a b cd]` will match documents where `title` is either `a`, `b` or `cd`, but do so\n///   more efficiently than the alternative query `title:a OR title:b OR title:c` does.\n///\n/// * date values: The query parser supports rfc3339 formatted dates. For example\n///   `\"2002-10-02T15:00:00.05Z\"` or `some_date_field:[2002-10-02T15:00:00Z TO\n///   2002-10-02T18:00:00Z}`\n///\n/// * all docs query: A plain `*` will match all documents in the index.\n///\n/// Parts of the queries can be boosted by appending `^boostfactor`.\n/// For instance, `\"SRE\"^2.0 OR devops^0.4` will boost documents containing `SRE` instead of\n/// devops. Negative boosts are not allowed.\n///\n/// It is also possible to define a boost for a some specific field, at the query parser level.\n/// (See [`set_field_boost(...)`](QueryParser::set_field_boost)). Typically you may want to boost a\n/// title field.\n///\n/// Additionally, specific fields can be marked to use fuzzy term queries for each literal\n/// via the [`QueryParser::set_field_fuzzy`] method.\n///\n/// Phrase terms support the `~` slop operator which allows to set the phrase's matching\n/// distance in words. `\"big wolf\"~1` will return documents containing the phrase `\"big bad wolf\"`.\n///\n/// Phrase terms also support the `*` prefix operator which switches the phrase's matching\n/// to consider all documents which contain the last term as a prefix, e.g. `\"big bad wo\"*` will\n/// match `\"big bad wolf\"`.\n#[derive(Clone)]\npub struct QueryParser {\n    schema: Schema,\n    default_fields: Vec<Field>,\n    conjunction_by_default: bool,\n    tokenizer_manager: TokenizerManager,\n    boost: FxHashMap<Field, Score>,\n    fuzzy: FxHashMap<Field, Fuzzy>,\n    regexes_allowed: bool,\n}\n\n#[derive(Clone)]\nstruct Fuzzy {\n    prefix: bool,\n    distance: u8,\n    transpose_cost_one: bool,\n}\n\nfn all_negative(ast: &LogicalAst) -> bool {\n    match ast {\n        LogicalAst::Leaf(_) => false,\n        LogicalAst::Boost(ref child_ast, _) => all_negative(child_ast),\n        LogicalAst::Clause(children) => children\n            .iter()\n            .all(|(ref occur, child)| (*occur == Occur::MustNot) || all_negative(child)),\n    }\n}\n\n// Make an all-negative ast into a normal ast. Must not be used on an already okay ast.\nfn make_non_negative(ast: &mut LogicalAst) {\n    match ast {\n        LogicalAst::Leaf(_) => (),\n        LogicalAst::Boost(ref mut child_ast, _) => make_non_negative(child_ast),\n        LogicalAst::Clause(children) => children.push((Occur::Should, LogicalLiteral::All.into())),\n    }\n}\n\n/// Similar to the try/? macro, but returns a tuple of (None, Vec<Error>) instead of Err(Error)\nmacro_rules! try_tuple {\n    ($expr:expr) => {{\n        match $expr {\n            Ok(val) => val,\n            Err(e) => return (None, vec![e.into()]),\n        }\n    }};\n}\n\nimpl QueryParser {\n    /// Creates a `QueryParser`, given\n    /// * schema - index Schema\n    /// * default_fields - fields used to search if no field is specifically defined in the query.\n    pub fn new(\n        schema: Schema,\n        default_fields: Vec<Field>,\n        tokenizer_manager: TokenizerManager,\n    ) -> QueryParser {\n        QueryParser {\n            schema,\n            default_fields,\n            tokenizer_manager,\n            conjunction_by_default: false,\n            boost: Default::default(),\n            fuzzy: Default::default(),\n            regexes_allowed: false,\n        }\n    }\n\n    // Splits a full_path as written in a query, into a field name and a\n    // json path.\n    pub(crate) fn split_full_path<'a>(&self, full_path: &'a str) -> Option<(Field, &'a str)> {\n        self.schema.find_field(full_path)\n    }\n\n    /// Creates a `QueryParser`, given\n    ///  * an index\n    ///  * a set of default fields used to search if no field is specifically defined in the query.\n    pub fn for_index(index: &Index, default_fields: Vec<Field>) -> QueryParser {\n        QueryParser::new(index.schema(), default_fields, index.tokenizers().clone())\n    }\n\n    /// Set the default way to compose queries to a conjunction.\n    ///\n    /// By default, the query `happy tax payer` is equivalent to the query\n    /// `happy OR tax OR payer`. After calling `.set_conjunction_by_default()`\n    /// `happy tax payer` will be interpreted by the parser as `happy AND tax AND payer`.\n    pub fn set_conjunction_by_default(&mut self) {\n        self.conjunction_by_default = true;\n    }\n\n    /// Sets a boost for a specific field.\n    ///\n    /// The parse query will automatically boost this field.\n    ///\n    /// If the query defines a query boost through the query language (e.g: `country:France^3.0`),\n    /// the two boosts (the one defined in the query, and the one defined in the `QueryParser`)\n    /// are multiplied together.\n    pub fn set_field_boost(&mut self, field: Field, boost: Score) {\n        self.boost.insert(field, boost);\n    }\n\n    /// Sets the given [field][`Field`] to use [fuzzy term queries][`FuzzyTermQuery`]\n    ///\n    /// If set, the parse will produce queries using fuzzy term queries\n    /// with the given parameters for each literal matched against the given field.\n    ///\n    /// See the [`FuzzyTermQuery::new`] and [`FuzzyTermQuery::new_prefix`] methods\n    /// for the meaning of the individual parameters.\n    pub fn set_field_fuzzy(\n        &mut self,\n        field: Field,\n        prefix: bool,\n        distance: u8,\n        transpose_cost_one: bool,\n    ) {\n        self.fuzzy.insert(\n            field,\n            Fuzzy {\n                prefix,\n                distance,\n                transpose_cost_one,\n            },\n        );\n    }\n\n    /// Allow regexes in queries\n    pub fn allow_regexes(&mut self) {\n        self.regexes_allowed = true;\n    }\n\n    /// Parse a query\n    ///\n    /// Note that `parse_query` returns an error if the input\n    /// is not a valid query.\n    pub fn parse_query(&self, query: &str) -> Result<Box<dyn Query>, QueryParserError> {\n        let logical_ast = self.parse_query_to_logical_ast(query)?;\n        Ok(convert_to_query(&self.fuzzy, logical_ast))\n    }\n\n    /// Parse a query leniently\n    ///\n    /// This variant parses invalid query on a best effort basis. If some part of the query can't\n    /// reasonably be executed (range query without field, searching on a non existing field,\n    /// searching without precising field when no default field is provided...), they may get\n    /// turned into a \"match-nothing\" subquery.\n    ///\n    /// In case it encountered such issues, they are reported as a Vec of errors.\n    pub fn parse_query_lenient(&self, query: &str) -> (Box<dyn Query>, Vec<QueryParserError>) {\n        let (logical_ast, errors) = self.parse_query_to_logical_ast_lenient(query);\n        (convert_to_query(&self.fuzzy, logical_ast), errors)\n    }\n\n    /// Build a query from an already parsed user input AST\n    ///\n    /// This can be useful if the user input AST parsed using [`query_grammar`]\n    /// needs to be inspected before the query is re-interpreted w.r.t.\n    /// index specifics like field names and tokenizers.\n    pub fn build_query_from_user_input_ast(\n        &self,\n        user_input_ast: UserInputAst,\n    ) -> Result<Box<dyn Query>, QueryParserError> {\n        let (logical_ast, mut err) = self.compute_logical_ast_lenient(user_input_ast);\n        if !err.is_empty() {\n            return Err(err.swap_remove(0));\n        }\n        Ok(convert_to_query(&self.fuzzy, logical_ast))\n    }\n\n    /// Build leniently a query from an already parsed user input AST.\n    ///\n    /// See also [`QueryParser::build_query_from_user_input_ast`]\n    pub fn build_query_from_user_input_ast_lenient(\n        &self,\n        user_input_ast: UserInputAst,\n    ) -> (Box<dyn Query>, Vec<QueryParserError>) {\n        let (logical_ast, errors) = self.compute_logical_ast_lenient(user_input_ast);\n        (convert_to_query(&self.fuzzy, logical_ast), errors)\n    }\n\n    /// Parse the user query into an AST.\n    fn parse_query_to_logical_ast(&self, query: &str) -> Result<LogicalAst, QueryParserError> {\n        let user_input_ast = query_grammar::parse_query(query)\n            .map_err(|_| QueryParserError::SyntaxError(query.to_string()))?;\n        let (ast, mut err) = self.compute_logical_ast_lenient(user_input_ast);\n        if !err.is_empty() {\n            return Err(err.swap_remove(0));\n        }\n        Ok(ast.simplify())\n    }\n\n    /// Parse the user query into an AST.\n    fn parse_query_to_logical_ast_lenient(\n        &self,\n        query: &str,\n    ) -> (LogicalAst, Vec<QueryParserError>) {\n        let (user_input_ast, errors) = query_grammar::parse_query_lenient(query);\n        let mut errors: Vec<_> = errors\n            .into_iter()\n            .map(|error| {\n                QueryParserError::SyntaxError(format!(\n                    \"{} at position {}\",\n                    error.message, error.pos\n                ))\n            })\n            .collect();\n        let (ast, mut ast_errors) = self.compute_logical_ast_lenient(user_input_ast);\n        errors.append(&mut ast_errors);\n        (ast, errors)\n    }\n\n    fn compute_logical_ast_lenient(\n        &self,\n        user_input_ast: UserInputAst,\n    ) -> (LogicalAst, Vec<QueryParserError>) {\n        let (mut ast, mut err) = self.compute_logical_ast_with_occur_lenient(user_input_ast);\n        if let LogicalAst::Clause(children) = &ast {\n            if children.is_empty() {\n                return (ast, err);\n            }\n        }\n        if all_negative(&ast) {\n            err.push(QueryParserError::AllButQueryForbidden);\n            make_non_negative(&mut ast);\n        }\n        (ast, err)\n    }\n\n    fn compute_boundary_term(\n        &self,\n        field: Field,\n        json_path: &str,\n        phrase: &str,\n    ) -> Result<Term, QueryParserError> {\n        let field_entry = self.schema.get_field_entry(field);\n        let field_type = field_entry.field_type();\n        let field_supports_ff_range_queries = field_type.is_fast()\n            && is_type_valid_for_fastfield_range_query(field_type.value_type());\n\n        if !field_type.is_indexed() && !field_supports_ff_range_queries {\n            return Err(QueryParserError::FieldNotIndexed(\n                field_entry.name().to_string(),\n            ));\n        }\n        if !json_path.is_empty() && field_type.value_type() != Type::Json {\n            return Err(QueryParserError::UnsupportedQuery(format!(\n                \"Json path is not supported for field {:?}\",\n                field_entry.name()\n            )));\n        }\n        match *field_type {\n            FieldType::U64(_) => {\n                let val: u64 = u64::from_str(phrase)?;\n                Ok(Term::from_field_u64(field, val))\n            }\n            FieldType::I64(_) => {\n                let val: i64 = i64::from_str(phrase)?;\n                Ok(Term::from_field_i64(field, val))\n            }\n            FieldType::F64(_) => {\n                let val: f64 = f64::from_str(phrase)?;\n                Ok(Term::from_field_f64(field, val))\n            }\n            FieldType::Bool(_) => {\n                let val: bool = bool::from_str(phrase)?;\n                Ok(Term::from_field_bool(field, val))\n            }\n            FieldType::Date(_) => {\n                let dt = OffsetDateTime::parse(phrase, &Rfc3339)?;\n                Ok(Term::from_field_date(field, DateTime::from_utc(dt)))\n            }\n            FieldType::Str(ref str_options) => {\n                let option = str_options.get_indexing_options().ok_or_else(|| {\n                    // This should have been seen earlier really.\n                    QueryParserError::FieldNotIndexed(field_entry.name().to_string())\n                })?;\n                let mut text_analyzer =\n                    self.tokenizer_manager\n                        .get(option.tokenizer())\n                        .ok_or_else(|| QueryParserError::UnknownTokenizer {\n                            field: field_entry.name().to_string(),\n                            tokenizer: option.tokenizer().to_string(),\n                        })?;\n                let mut terms: Vec<Term> = Vec::new();\n                let mut token_stream = text_analyzer.token_stream(phrase);\n                token_stream.process(&mut |token| {\n                    let term = Term::from_field_text(field, &token.text);\n                    terms.push(term);\n                });\n                if terms.len() != 1 {\n                    return Err(QueryParserError::UnsupportedQuery(format!(\n                        \"Range query boundary cannot have multiple tokens: {phrase:?} [{terms:?}].\"\n                    )));\n                }\n                Ok(terms.into_iter().next().unwrap())\n            }\n            FieldType::JsonObject(ref json_options) => {\n                let mut term = Term::from_field_json_path(\n                    field,\n                    json_path,\n                    json_options.is_expand_dots_enabled(),\n                );\n                if let Some(term) =\n                    // Try to convert the phrase to a fast value\n                    convert_to_fast_value_and_append_to_json_term(&term, phrase, false)\n                {\n                    Ok(term)\n                } else {\n                    term.append_type_and_str(phrase);\n                    Ok(term)\n                }\n            }\n            FieldType::Facet(_) => match Facet::from_text(phrase) {\n                Ok(facet) => Ok(Term::from_facet(field, &facet)),\n                Err(e) => Err(QueryParserError::from(e)),\n            },\n            FieldType::Bytes(_) => {\n                let bytes = BASE64\n                    .decode(phrase)\n                    .map_err(QueryParserError::ExpectedBase64)?;\n                Ok(Term::from_field_bytes(field, &bytes))\n            }\n            FieldType::IpAddr(_) => {\n                let ip_v6 = IpAddr::from_str(phrase)?.into_ipv6_addr();\n                Ok(Term::from_field_ip_addr(field, ip_v6))\n            }\n        }\n    }\n\n    fn compute_logical_ast_for_leaf(\n        &self,\n        field: Field,\n        json_path: &str,\n        phrase: &str,\n        slop: u32,\n        prefix: bool,\n    ) -> Result<Vec<LogicalLiteral>, QueryParserError> {\n        let field_entry = self.schema.get_field_entry(field);\n        let field_type = field_entry.field_type();\n        let field_name = field_entry.name();\n        if !field_type.is_indexed() {\n            return Err(QueryParserError::FieldNotIndexed(field_name.to_string()));\n        }\n        if field_type.value_type() != Type::Json && !json_path.is_empty() {\n            let field_name = self.schema.get_field_name(field);\n            return Err(QueryParserError::FieldDoesNotExist(format!(\n                \"{field_name}.{json_path}\"\n            )));\n        }\n        match *field_type {\n            FieldType::U64(_) => {\n                let val: u64 = u64::from_str(phrase)?;\n                let i64_term = Term::from_field_u64(field, val);\n                Ok(vec![LogicalLiteral::Term(i64_term)])\n            }\n            FieldType::I64(_) => {\n                let val: i64 = i64::from_str(phrase)?;\n                let i64_term = Term::from_field_i64(field, val);\n                Ok(vec![LogicalLiteral::Term(i64_term)])\n            }\n            FieldType::F64(_) => {\n                let val: f64 = f64::from_str(phrase)?;\n                let f64_term = Term::from_field_f64(field, val);\n                Ok(vec![LogicalLiteral::Term(f64_term)])\n            }\n            FieldType::Bool(_) => {\n                let val: bool = bool::from_str(phrase)?;\n                let bool_term = Term::from_field_bool(field, val);\n                Ok(vec![LogicalLiteral::Term(bool_term)])\n            }\n            FieldType::Date(_) => {\n                let dt = OffsetDateTime::parse(phrase, &Rfc3339)?;\n                let dt_term = Term::from_field_date_for_search(field, DateTime::from_utc(dt));\n                Ok(vec![LogicalLiteral::Term(dt_term)])\n            }\n            FieldType::Str(ref str_options) => {\n                let indexing_options = str_options.get_indexing_options().ok_or_else(|| {\n                    // This should have been seen earlier really.\n                    QueryParserError::FieldNotIndexed(field_name.to_string())\n                })?;\n                let mut text_analyzer = self\n                    .tokenizer_manager\n                    .get(indexing_options.tokenizer())\n                    .ok_or_else(|| QueryParserError::UnknownTokenizer {\n                        field: field_name.to_string(),\n                        tokenizer: indexing_options.tokenizer().to_string(),\n                    })?;\n                Ok(generate_literals_for_str(\n                    field_name,\n                    field,\n                    phrase,\n                    slop,\n                    prefix,\n                    indexing_options,\n                    &mut text_analyzer,\n                )?\n                .into_iter()\n                .collect())\n            }\n            FieldType::JsonObject(ref json_options) => generate_literals_for_json_object(\n                field_name,\n                field,\n                json_path,\n                phrase,\n                &self.tokenizer_manager,\n                json_options,\n            ),\n            FieldType::Facet(_) => match Facet::from_text(phrase) {\n                Ok(facet) => {\n                    let facet_term = Term::from_facet(field, &facet);\n                    Ok(vec![LogicalLiteral::Term(facet_term)])\n                }\n                Err(e) => Err(QueryParserError::from(e)),\n            },\n            FieldType::Bytes(_) => {\n                let bytes = BASE64\n                    .decode(phrase)\n                    .map_err(QueryParserError::ExpectedBase64)?;\n                let bytes_term = Term::from_field_bytes(field, &bytes);\n                Ok(vec![LogicalLiteral::Term(bytes_term)])\n            }\n            FieldType::IpAddr(_) => {\n                let ip_v6 = IpAddr::from_str(phrase)?.into_ipv6_addr();\n                let term = Term::from_field_ip_addr(field, ip_v6);\n                Ok(vec![LogicalLiteral::Term(term)])\n            }\n        }\n    }\n\n    fn default_occur(&self) -> Occur {\n        if self.conjunction_by_default {\n            Occur::Must\n        } else {\n            Occur::Should\n        }\n    }\n\n    fn resolve_bound(\n        &self,\n        field: Field,\n        json_path: &str,\n        bound: &UserInputBound,\n    ) -> Result<Bound<Term>, QueryParserError> {\n        if bound.term_str() == \"*\" {\n            return Ok(Bound::Unbounded);\n        }\n        let term = self.compute_boundary_term(field, json_path, bound.term_str())?;\n        match *bound {\n            UserInputBound::Inclusive(_) => Ok(Bound::Included(term)),\n            UserInputBound::Exclusive(_) => Ok(Bound::Excluded(term)),\n            UserInputBound::Unbounded => Ok(Bound::Unbounded),\n        }\n    }\n\n    fn compute_logical_ast_with_occur_lenient(\n        &self,\n        user_input_ast: UserInputAst,\n    ) -> (LogicalAst, Vec<QueryParserError>) {\n        match user_input_ast {\n            UserInputAst::Clause(sub_queries) => {\n                let default_occur = self.default_occur();\n                let mut logical_sub_queries: Vec<(Occur, LogicalAst)> = Vec::new();\n                let mut errors = Vec::new();\n                for (occur_opt, sub_ast) in sub_queries {\n                    let (sub_ast, mut sub_errors) =\n                        self.compute_logical_ast_with_occur_lenient(sub_ast);\n                    let occur = occur_opt.unwrap_or(default_occur);\n                    logical_sub_queries.push((occur, sub_ast));\n                    errors.append(&mut sub_errors);\n                }\n                (LogicalAst::Clause(logical_sub_queries), errors)\n            }\n            UserInputAst::Boost(ast, boost) => {\n                let (ast, errors) = self.compute_logical_ast_with_occur_lenient(*ast);\n                (ast.boost(boost.into_inner() as Score), errors)\n            }\n            UserInputAst::Leaf(leaf) => {\n                let (ast, errors) = self.compute_logical_ast_from_leaf_lenient(*leaf);\n                // if the error is not recoverable, replace it with an empty clause. We will end up\n                // trimming those later\n                (\n                    ast.unwrap_or_else(|| LogicalAst::Clause(Vec::new())),\n                    errors,\n                )\n            }\n        }\n    }\n\n    fn field_boost(&self, field: Field) -> Score {\n        self.boost.get(&field).cloned().unwrap_or(1.0)\n    }\n\n    fn default_indexed_json_fields(&self) -> impl Iterator<Item = Field> + '_ {\n        let schema = self.schema.clone();\n        self.default_fields.iter().cloned().filter(move |field| {\n            let field_type = schema.get_field_entry(*field).field_type();\n            field_type.value_type() == Type::Json && field_type.is_indexed()\n        })\n    }\n\n    /// Given a literal, returns the list of terms that should be searched.\n    ///\n    /// The terms are identified by a triplet:\n    /// - tantivy field\n    /// - field_path: tantivy has JSON fields. It is possible to target a member of a JSON object by\n    ///   naturally extending the json field name with a \".\" separated field_path\n    /// - field_phrase: the phrase that is being searched.\n    ///\n    /// The literal identifies the targeted field by a so-called *full field path*,\n    /// specified before the \":\". (e.g. identity.username:fulmicoton).\n    ///\n    /// The way we split the full field path into (field_name, field_path) can be ambiguous,\n    /// because field_names can contain \".\" themselves.\n    // For instance if a field is named `one.two` and another one is named `one`,\n    /// should `one.two:three` target `one.two` with field path `` or or `one` with\n    /// the field path `two`.\n    ///\n    /// In this case tantivy, just picks the solution with the longest field name.\n    ///\n    /// Quirk: As a hack for quickwit, we do not split over a dot that appear escaped '\\.'.\n    fn compute_path_triplets_for_literal<'a>(\n        &self,\n        literal: &'a UserInputLiteral,\n    ) -> Result<Vec<(Field, &'a str, &'a str)>, QueryParserError> {\n        let full_path = if let Some(full_path) = &literal.field_name {\n            full_path\n        } else {\n            // The user did not specify any path...\n            // We simply target default fields.\n            if self.default_fields.is_empty() {\n                return Err(QueryParserError::NoDefaultFieldDeclared);\n            }\n            return Ok(self\n                .default_fields\n                .iter()\n                .map(|default_field| (*default_field, \"\", literal.phrase.as_str()))\n                .collect::<Vec<(Field, &str, &str)>>());\n        };\n        if let Some((field, path)) = self.split_full_path(full_path) {\n            return Ok(vec![(field, path, literal.phrase.as_str())]);\n        }\n        // We need to add terms associated with json default fields.\n        let triplets: Vec<(Field, &str, &str)> = self\n            .default_indexed_json_fields()\n            .map(|json_field| (json_field, full_path.as_str(), literal.phrase.as_str()))\n            .collect();\n        if triplets.is_empty() {\n            return Err(QueryParserError::FieldDoesNotExist(full_path.to_string()));\n        }\n        Ok(triplets)\n    }\n\n    fn compute_logical_ast_from_leaf_lenient(\n        &self,\n        leaf: UserInputLeaf,\n    ) -> (Option<LogicalAst>, Vec<QueryParserError>) {\n        match leaf {\n            UserInputLeaf::Literal(literal) => {\n                let term_phrases: Vec<(Field, &str, &str)> =\n                    try_tuple!(self.compute_path_triplets_for_literal(&literal));\n                let mut asts: Vec<LogicalAst> = Vec::new();\n                let mut errors: Vec<QueryParserError> = Vec::new();\n                for (field, json_path, phrase) in term_phrases {\n                    let unboosted_asts = match self.compute_logical_ast_for_leaf(\n                        field,\n                        json_path,\n                        phrase,\n                        literal.slop,\n                        literal.prefix,\n                    ) {\n                        Ok(asts) => asts,\n                        Err(e) => {\n                            errors.push(e);\n                            continue;\n                        }\n                    };\n                    for ast in unboosted_asts {\n                        // Apply some field specific boost defined at the query parser level.\n                        let boost = self.field_boost(field);\n                        asts.push(LogicalAst::Leaf(Box::new(ast)).boost(boost));\n                    }\n                }\n                if !asts.is_empty() {\n                    // if some fields failed but other succeeded, we consider this a success, it\n                    // probably means the default_fields contains\n                    // text and non-text fields, and the non-text ones failed\n                    errors.clear();\n                }\n                let result_ast: LogicalAst = if asts.len() == 1 {\n                    asts.into_iter().next().unwrap()\n                } else {\n                    LogicalAst::Clause(asts.into_iter().map(|ast| (Occur::Should, ast)).collect())\n                };\n                (Some(result_ast), errors)\n            }\n            UserInputLeaf::All => (\n                Some(LogicalAst::Leaf(Box::new(LogicalLiteral::All))),\n                Vec::new(),\n            ),\n            UserInputLeaf::Range {\n                field: full_field_opt,\n                lower,\n                upper,\n            } => {\n                let Some(full_path) = full_field_opt else {\n                    return (\n                        None,\n                        vec![QueryParserError::UnsupportedQuery(\n                            \"Range query need to target a specific field.\".to_string(),\n                        )],\n                    );\n                };\n                let (field, json_path) = try_tuple!(self\n                    .split_full_path(&full_path)\n                    .ok_or_else(|| QueryParserError::FieldDoesNotExist(full_path.clone())));\n                let mut errors = Vec::new();\n                let lower = match self.resolve_bound(field, json_path, &lower) {\n                    Ok(bound) => bound,\n                    Err(error) => {\n                        errors.push(error);\n                        Bound::Unbounded\n                    }\n                };\n                let upper = match self.resolve_bound(field, json_path, &upper) {\n                    Ok(bound) => bound,\n                    Err(error) => {\n                        errors.push(error);\n                        Bound::Unbounded\n                    }\n                };\n                if lower == Bound::Unbounded && upper == Bound::Unbounded {\n                    // this range is useless, either because a user requested [* TO *], or because\n                    // we failed to parse something. Either way, there is no point emitting it\n                    return (None, errors);\n                }\n                let logical_ast =\n                    LogicalAst::Leaf(Box::new(LogicalLiteral::Range { lower, upper }));\n                (Some(logical_ast), errors)\n            }\n            UserInputLeaf::Set {\n                field: full_field_opt,\n                elements,\n            } => {\n                let full_path = try_tuple!(full_field_opt.ok_or_else(|| {\n                    QueryParserError::UnsupportedQuery(\n                        \"Range query need to target a specific field.\".to_string(),\n                    )\n                }));\n                let (field, json_path) = try_tuple!(self\n                    .split_full_path(&full_path)\n                    .ok_or_else(|| QueryParserError::FieldDoesNotExist(full_path.clone())));\n                let (elements, errors) = elements\n                    .into_iter()\n                    .map(|element| self.compute_boundary_term(field, json_path, &element))\n                    .partition_result();\n                let logical_ast = LogicalAst::Leaf(Box::new(LogicalLiteral::Set { elements }));\n                (Some(logical_ast), errors)\n            }\n            UserInputLeaf::Exists { .. } => (\n                None,\n                vec![QueryParserError::UnsupportedQuery(\n                    \"Range query need to target a specific field.\".to_string(),\n                )],\n            ),\n            UserInputLeaf::Regex { field, pattern } => {\n                if !self.regexes_allowed {\n                    return (\n                        None,\n                        vec![QueryParserError::UnsupportedQuery(\n                            \"Regex queries are not allowed.\".to_string(),\n                        )],\n                    );\n                }\n                let full_path = try_tuple!(field.ok_or_else(|| {\n                    QueryParserError::UnsupportedQuery(\n                        \"Regex query need to target a specific field.\".to_string(),\n                    )\n                }));\n                let (field, json_path) = try_tuple!(self\n                    .split_full_path(&full_path)\n                    .ok_or_else(|| QueryParserError::FieldDoesNotExist(full_path.clone())));\n                if !json_path.is_empty() {\n                    return (\n                        None,\n                        vec![QueryParserError::UnsupportedQuery(\n                            \"Regex query does not support json paths.\".to_string(),\n                        )],\n                    );\n                }\n                if !matches!(\n                    self.schema.get_field_entry(field).field_type(),\n                    FieldType::Str(_)\n                ) {\n                    return (\n                        None,\n                        vec![QueryParserError::UnsupportedQuery(\n                            \"Regex query only supported on text fields\".to_string(),\n                        )],\n                    );\n                }\n                let pattern = try_tuple!(Regex::new(&pattern).map_err(|e| {\n                    QueryParserError::UnsupportedQuery(format!(\"Invalid regex: {e}\"))\n                }));\n                let logical_ast = LogicalAst::Leaf(Box::new(LogicalLiteral::Regex {\n                    pattern: Arc::new(pattern),\n                    field,\n                }));\n                (Some(logical_ast), Vec::new())\n            }\n        }\n    }\n}\n\nfn convert_literal_to_query(\n    fuzzy: &FxHashMap<Field, Fuzzy>,\n    logical_literal: LogicalLiteral,\n) -> Box<dyn Query> {\n    match logical_literal {\n        LogicalLiteral::Term(term) => {\n            if let Some(fuzzy) = fuzzy.get(&term.field()) {\n                if fuzzy.prefix {\n                    Box::new(FuzzyTermQuery::new_prefix(\n                        term,\n                        fuzzy.distance,\n                        fuzzy.transpose_cost_one,\n                    ))\n                } else {\n                    Box::new(FuzzyTermQuery::new(\n                        term,\n                        fuzzy.distance,\n                        fuzzy.transpose_cost_one,\n                    ))\n                }\n            } else {\n                Box::new(TermQuery::new(term, IndexRecordOption::WithFreqs))\n            }\n        }\n        LogicalLiteral::Phrase {\n            terms,\n            slop,\n            prefix,\n        } => {\n            if prefix {\n                Box::new(PhrasePrefixQuery::new_with_offset(terms))\n            } else {\n                Box::new(PhraseQuery::new_with_offset_and_slop(terms, slop))\n            }\n        }\n        LogicalLiteral::Range { lower, upper } => Box::new(RangeQuery::new(lower, upper)),\n        LogicalLiteral::Set { elements, .. } => Box::new(TermSetQuery::new(elements)),\n        LogicalLiteral::All => Box::new(AllQuery),\n        LogicalLiteral::Regex { pattern, field } => {\n            Box::new(RegexQuery::from_regex(pattern, field))\n        }\n    }\n}\n\nfn generate_literals_for_str(\n    field_name: &str,\n    field: Field,\n    phrase: &str,\n    slop: u32,\n    prefix: bool,\n    indexing_options: &TextFieldIndexing,\n    text_analyzer: &mut TextAnalyzer,\n) -> Result<Option<LogicalLiteral>, QueryParserError> {\n    let mut terms: Vec<(usize, Term)> = Vec::new();\n    let mut token_stream = text_analyzer.token_stream(phrase);\n    token_stream.process(&mut |token| {\n        let term = Term::from_field_text(field, &token.text);\n        terms.push((token.position, term));\n    });\n    if terms.len() <= 1 {\n        if prefix {\n            return Err(QueryParserError::PhrasePrefixRequiresAtLeastTwoTerms {\n                phrase: phrase.to_owned(),\n                tokenizer: indexing_options.tokenizer().to_owned(),\n            });\n        }\n        let term_literal_opt = terms\n            .into_iter()\n            .next()\n            .map(|(_, term)| LogicalLiteral::Term(term));\n        return Ok(term_literal_opt);\n    }\n    if !indexing_options.index_option().has_positions() {\n        return Err(QueryParserError::FieldDoesNotHavePositionsIndexed(\n            field_name.to_string(),\n        ));\n    }\n    Ok(Some(LogicalLiteral::Phrase {\n        terms,\n        slop,\n        prefix,\n    }))\n}\n\nfn generate_literals_for_json_object(\n    field_name: &str,\n    field: Field,\n    json_path: &str,\n    phrase: &str,\n    tokenizer_manager: &TokenizerManager,\n    json_options: &JsonObjectOptions,\n) -> Result<Vec<LogicalLiteral>, QueryParserError> {\n    let text_options = json_options.get_text_indexing_options().ok_or_else(|| {\n        // This should have been seen earlier really.\n        QueryParserError::FieldNotIndexed(field_name.to_string())\n    })?;\n    let mut text_analyzer = tokenizer_manager\n        .get(text_options.tokenizer())\n        .ok_or_else(|| QueryParserError::UnknownTokenizer {\n            field: field_name.to_string(),\n            tokenizer: text_options.tokenizer().to_string(),\n        })?;\n    let index_record_option = text_options.index_option();\n    let mut logical_literals = Vec::new();\n\n    let get_term_with_path =\n        || Term::from_field_json_path(field, json_path, json_options.is_expand_dots_enabled());\n\n    // Try to convert the phrase to a fast value\n    if let Some(term) =\n        convert_to_fast_value_and_append_to_json_term(&get_term_with_path(), phrase, true)\n    {\n        logical_literals.push(LogicalLiteral::Term(term));\n    }\n\n    // Try to tokenize the phrase and create Terms.\n    let mut positions_and_terms = Vec::<(usize, Term)>::new();\n    let mut token_stream = text_analyzer.token_stream(phrase);\n    token_stream.process(&mut |token| {\n        let mut term = get_term_with_path();\n        term.append_type_and_str(&token.text);\n        positions_and_terms.push((token.position, term.clone()));\n    });\n\n    if positions_and_terms.len() <= 1 {\n        for (_, term) in positions_and_terms {\n            logical_literals.push(LogicalLiteral::Term(term));\n        }\n        return Ok(logical_literals);\n    }\n    if !index_record_option.has_positions() {\n        return Err(QueryParserError::FieldDoesNotHavePositionsIndexed(\n            field_name.to_string(),\n        ));\n    }\n    logical_literals.push(LogicalLiteral::Phrase {\n        terms: positions_and_terms,\n        slop: 0,\n        prefix: false,\n    });\n    Ok(logical_literals)\n}\n\nfn convert_to_query(fuzzy: &FxHashMap<Field, Fuzzy>, logical_ast: LogicalAst) -> Box<dyn Query> {\n    match trim_ast(logical_ast) {\n        Some(LogicalAst::Clause(trimmed_clause)) => {\n            let occur_subqueries = trimmed_clause\n                .into_iter()\n                .map(|(occur, subquery)| (occur, convert_to_query(fuzzy, subquery)))\n                .collect::<Vec<_>>();\n            assert!(\n                !occur_subqueries.is_empty(),\n                \"Should not be empty after trimming\"\n            );\n            Box::new(BooleanQuery::new(occur_subqueries))\n        }\n        Some(LogicalAst::Leaf(trimmed_logical_literal)) => {\n            convert_literal_to_query(fuzzy, *trimmed_logical_literal)\n        }\n        Some(LogicalAst::Boost(ast, boost)) => {\n            let query = convert_to_query(fuzzy, *ast);\n            let boosted_query = BoostQuery::new(query, boost);\n            Box::new(boosted_query)\n        }\n        None => Box::new(EmptyQuery),\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use matches::assert_matches;\n\n    use super::super::logical_ast::*;\n    use super::{QueryParser, QueryParserError};\n    use crate::query::Query;\n    use crate::schema::{\n        FacetOptions, Field, IndexRecordOption, Schema, Term, TextFieldIndexing, TextOptions, FAST,\n        INDEXED, STORED, STRING, TEXT,\n    };\n    use crate::tokenizer::{\n        LowerCaser, SimpleTokenizer, StopWordFilter, TextAnalyzer, TokenizerManager,\n    };\n    use crate::Index;\n\n    fn make_schema() -> Schema {\n        let mut schema_builder = Schema::builder();\n        let text_field_indexing = TextFieldIndexing::default()\n            .set_tokenizer(\"en_with_stop_words\")\n            .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n        let text_options = TextOptions::default()\n            .set_indexing_options(text_field_indexing)\n            .set_stored();\n        schema_builder.add_text_field(\"title\", TEXT);\n        schema_builder.add_text_field(\"text\", TEXT);\n        schema_builder.add_i64_field(\"signed\", INDEXED);\n        schema_builder.add_u64_field(\"unsigned\", INDEXED);\n        schema_builder.add_text_field(\"notindexed_text\", STORED);\n        schema_builder.add_text_field(\"notindexed_u64\", STORED);\n        schema_builder.add_text_field(\"notindexed_i64\", STORED);\n        schema_builder.add_text_field(\"nottokenized\", STRING);\n        schema_builder.add_text_field(\"with_stop_words\", text_options);\n        schema_builder.add_date_field(\"date\", INDEXED);\n        schema_builder.add_f64_field(\"float\", INDEXED);\n        schema_builder.add_facet_field(\"facet\", FacetOptions::default());\n        schema_builder.add_bytes_field(\"bytes\", INDEXED);\n        schema_builder.add_bytes_field(\"bytes_not_indexed\", STORED);\n        schema_builder.add_json_field(\"json\", TEXT);\n        schema_builder.add_json_field(\"json_not_indexed\", STORED);\n        schema_builder.add_bool_field(\"bool\", INDEXED);\n        schema_builder.add_bool_field(\"notindexed_bool\", STORED);\n        schema_builder.add_u64_field(\"u64_ff\", FAST);\n        schema_builder.build()\n    }\n\n    fn make_query_parser_with_default_fields(default_fields: &[&'static str]) -> QueryParser {\n        let schema = make_schema();\n        let default_fields: Vec<Field> = default_fields\n            .iter()\n            .flat_map(|field_name| schema.get_field(field_name))\n            .collect();\n        let tokenizer_manager = TokenizerManager::default();\n        tokenizer_manager.register(\n            \"en_with_stop_words\",\n            TextAnalyzer::builder(SimpleTokenizer::default())\n                .filter(LowerCaser)\n                .filter(StopWordFilter::remove(vec![\"the\".to_string()]))\n                .build(),\n        );\n        QueryParser::new(schema, default_fields, tokenizer_manager)\n    }\n\n    fn make_query_parser() -> QueryParser {\n        make_query_parser_with_default_fields(&[\"title\", \"text\"])\n    }\n\n    fn parse_query_to_logical_ast_with_default_fields(\n        query: &str,\n        default_conjunction: bool,\n        default_fields: &[&'static str],\n        allow_regexes: bool,\n    ) -> Result<LogicalAst, QueryParserError> {\n        let mut query_parser = make_query_parser_with_default_fields(default_fields);\n        if default_conjunction {\n            query_parser.set_conjunction_by_default();\n        }\n        if allow_regexes {\n            query_parser.allow_regexes();\n        }\n        query_parser.parse_query_to_logical_ast(query)\n    }\n\n    fn parse_query_to_logical_ast(\n        query: &str,\n        default_conjunction: bool,\n    ) -> Result<LogicalAst, QueryParserError> {\n        parse_query_to_logical_ast_with_default_fields(\n            query,\n            default_conjunction,\n            &[\"title\", \"text\"],\n            true,\n        )\n    }\n\n    #[track_caller]\n    fn test_parse_query_to_logical_ast_helper_with_default_fields(\n        query: &str,\n        expected: &str,\n        default_conjunction: bool,\n        default_fields: &[&'static str],\n    ) {\n        let query = parse_query_to_logical_ast_with_default_fields(\n            query,\n            default_conjunction,\n            default_fields,\n            true,\n        )\n        .unwrap();\n        let query_str = format!(\"{query:?}\");\n        assert_eq!(query_str, expected);\n    }\n\n    #[track_caller]\n    fn test_parse_query_to_logical_ast_helper(\n        query: &str,\n        expected: &str,\n        default_conjunction: bool,\n    ) {\n        test_parse_query_to_logical_ast_helper_with_default_fields(\n            query,\n            expected,\n            default_conjunction,\n            &[\"title\", \"text\"],\n        )\n    }\n\n    #[test]\n    pub fn test_parse_query_facet() {\n        let query_parser = make_query_parser();\n        let query = query_parser.parse_query(\"facet:/root/branch/leaf\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            r#\"TermQuery(Term(field=11, type=Facet, Facet(/root/branch/leaf)))\"#\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_with_boost() {\n        let mut query_parser = make_query_parser();\n        let schema = make_schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        query_parser.set_field_boost(text_field, 2.0);\n        let query = query_parser.parse_query(\"text:hello\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            r#\"Boost(query=TermQuery(Term(field=1, type=Str, \"hello\")), boost=2)\"#\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_range_with_boost() {\n        let query = make_query_parser().parse_query(\"title:[A TO B]\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            \"RangeQuery { bounds: BoundsRange { lower_bound: Included(Term(field=0, type=Str, \\\n             \\\"a\\\")), upper_bound: Included(Term(field=0, type=Str, \\\"b\\\")) } }\"\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_with_default_boost_and_custom_boost() {\n        let mut query_parser = make_query_parser();\n        let schema = make_schema();\n        let text_field = schema.get_field(\"text\").unwrap();\n        query_parser.set_field_boost(text_field, 2.0);\n        let query = query_parser.parse_query(\"text:hello^2\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            r#\"Boost(query=Boost(query=TermQuery(Term(field=1, type=Str, \"hello\")), boost=2), boost=2)\"#\n        );\n    }\n\n    #[test]\n    pub fn test_parse_nonindexed_field_yields_error() {\n        let query_parser = make_query_parser();\n\n        let is_not_indexed_err = |query: &str| {\n            let result: Result<Box<dyn Query>, QueryParserError> = query_parser.parse_query(query);\n            if let Err(QueryParserError::FieldNotIndexed(field_name)) = result {\n                Some(field_name)\n            } else {\n                None\n            }\n        };\n\n        assert_eq!(\n            is_not_indexed_err(\"notindexed_text:titi\"),\n            Some(String::from(\"notindexed_text\"))\n        );\n        assert_eq!(\n            is_not_indexed_err(\"notindexed_u64:23424\"),\n            Some(String::from(\"notindexed_u64\"))\n        );\n        assert_eq!(\n            is_not_indexed_err(\"notindexed_i64:-234324\"),\n            Some(String::from(\"notindexed_i64\"))\n        );\n        assert_eq!(\n            is_not_indexed_err(\"notindexed_bool:true\"),\n            Some(String::from(\"notindexed_bool\"))\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_untokenized() {\n        test_parse_query_to_logical_ast_helper(\n            \"nottokenized:\\\"wordone wordtwo\\\"\",\n            r#\"Term(field=7, type=Str, \"wordone wordtwo\")\"#,\n            false,\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_empty() {\n        test_parse_query_to_logical_ast_helper(\"\", \"<emptyclause>\", false);\n        test_parse_query_to_logical_ast_helper(\" \", \"<emptyclause>\", false);\n        let query_parser = make_query_parser();\n        let query_result = query_parser.parse_query(\"\");\n        let query = query_result.unwrap();\n        assert_eq!(format!(\"{query:?}\"), \"EmptyQuery\");\n    }\n\n    #[test]\n    pub fn test_parse_query_ints() {\n        let query_parser = make_query_parser();\n        assert!(query_parser.parse_query(\"signed:2324\").is_ok());\n        assert!(query_parser.parse_query(\"signed:\\\"22\\\"\").is_ok());\n        assert!(query_parser.parse_query(\"signed:\\\"-2234\\\"\").is_ok());\n        assert!(query_parser\n            .parse_query(\"signed:\\\"-9999999999999\\\"\")\n            .is_ok());\n        assert!(query_parser.parse_query(\"signed:\\\"a\\\"\").is_err());\n        assert!(query_parser.parse_query(\"signed:\\\"2a\\\"\").is_err());\n        assert!(query_parser\n            .parse_query(\"signed:\\\"18446744073709551615\\\"\")\n            .is_err());\n        assert!(query_parser.parse_query(\"unsigned:\\\"2\\\"\").is_ok());\n        assert!(query_parser.parse_query(\"unsigned:\\\"-2\\\"\").is_err());\n        assert!(query_parser\n            .parse_query(\"unsigned:\\\"18446744073709551615\\\"\")\n            .is_ok());\n        assert!(query_parser.parse_query(\"float:\\\"3.1\\\"\").is_ok());\n        assert!(query_parser.parse_query(\"float:\\\"-2.4\\\"\").is_ok());\n        assert!(query_parser.parse_query(\"float:\\\"2.1.2\\\"\").is_err());\n        assert!(query_parser.parse_query(\"float:\\\"2.1a\\\"\").is_err());\n        assert!(query_parser\n            .parse_query(\"float:\\\"18446744073709551615.0\\\"\")\n            .is_ok());\n        test_parse_query_to_logical_ast_helper(\n            \"unsigned:2324\",\n            \"Term(field=3, type=U64, 2324)\",\n            false,\n        );\n\n        test_parse_query_to_logical_ast_helper(\n            \"signed:-2324\",\n            &format!(\n                \"{:?}\",\n                Term::from_field_i64(Field::from_field_id(2u32), -2324)\n            ),\n            false,\n        );\n\n        test_parse_query_to_logical_ast_helper(\n            \"float:2.5\",\n            &format!(\n                \"{:?}\",\n                Term::from_field_f64(Field::from_field_id(10u32), 2.5)\n            ),\n            false,\n        );\n    }\n\n    #[test]\n    fn test_parse_bytes() {\n        test_parse_query_to_logical_ast_helper(\n            \"bytes:YnVidQ==\",\n            \"Term(field=12, type=Bytes, [98, 117, 98, 117])\",\n            false,\n        );\n    }\n\n    #[test]\n    fn test_parse_bool() {\n        test_parse_query_to_logical_ast_helper(\n            \"bool:true\",\n            &format!(\n                \"{:?}\",\n                Term::from_field_bool(Field::from_field_id(16u32), true),\n            ),\n            false,\n        );\n    }\n\n    #[test]\n    fn test_parse_bytes_not_indexed() {\n        let error = parse_query_to_logical_ast(\"bytes_not_indexed:aaa\", false).unwrap_err();\n        assert!(matches!(error, QueryParserError::FieldNotIndexed(_)));\n    }\n\n    #[test]\n    fn test_json_field() {\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:hello\",\n            \"Term(field=14, type=Json, path=titi, type=Str, \\\"hello\\\")\",\n            false,\n        );\n    }\n\n    fn extract_query_term_json_path(query: &str) -> String {\n        let LogicalAst::Leaf(literal) = parse_query_to_logical_ast(query, false).unwrap() else {\n            panic!();\n        };\n        let LogicalLiteral::Term(term) = *literal else {\n            panic!();\n        };\n        std::str::from_utf8(term.serialized_value_bytes())\n            .unwrap()\n            .to_string()\n    }\n\n    #[test]\n    fn test_json_field_query_with_escaped_dot() {\n        assert_eq!(\n            extract_query_term_json_path(r#\"json.k8s.node.name:hello\"#),\n            \"k8s\\u{1}node\\u{1}name\\0shello\"\n        );\n        assert_eq!(\n            extract_query_term_json_path(r\"json.k8s\\.node\\.name:hello\"),\n            \"k8s.node.name\\0shello\"\n        );\n    }\n\n    #[test]\n    fn test_json_field_possibly_a_number() {\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:5\",\n            r#\"(Term(field=14, type=Json, path=titi, type=I64, 5) Term(field=14, type=Json, path=titi, type=Str, \"5\"))\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:-5\",\n            r#\"(Term(field=14, type=Json, path=titi, type=I64, -5) Term(field=14, type=Json, path=titi, type=Str, \"5\"))\"#, //< Yes this is a bit weird after going through the tokenizer we lose the \"-\".\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:10000000000000000000\",\n            r#\"(Term(field=14, type=Json, path=titi, type=U64, 10000000000000000000) Term(field=14, type=Json, path=titi, type=Str, \"10000000000000000000\"))\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:-5.2\",\n            r#\"(Term(field=14, type=Json, path=titi, type=F64, -5.2) \"[(0, Term(field=14, type=Json, path=titi, type=Str, \"5\")), (1, Term(field=14, type=Json, path=titi, type=Str, \"2\"))]\")\"#,\n            true,\n        );\n    }\n\n    #[test]\n    fn test_json_field_possibly_a_date() {\n        test_parse_query_to_logical_ast_helper(\n            r#\"json.date:\"2019-10-12T07:20:50.52Z\"\"#,\n            r#\"(Term(field=14, type=Json, path=date, type=Date, 2019-10-12T07:20:50Z) \"[(0, Term(field=14, type=Json, path=date, type=Str, \"2019\")), (1, Term(field=14, type=Json, path=date, type=Str, \"10\")), (2, Term(field=14, type=Json, path=date, type=Str, \"12t07\")), (3, Term(field=14, type=Json, path=date, type=Str, \"20\")), (4, Term(field=14, type=Json, path=date, type=Str, \"50\")), (5, Term(field=14, type=Json, path=date, type=Str, \"52z\"))]\")\"#,\n            true,\n        );\n    }\n\n    #[test]\n    fn test_json_field_possibly_a_bool() {\n        test_parse_query_to_logical_ast_helper(\n            \"json.titi:true\",\n            r#\"(Term(field=14, type=Json, path=titi, type=Bool, true) Term(field=14, type=Json, path=titi, type=Str, \"true\"))\"#,\n            true,\n        );\n    }\n\n    #[test]\n    fn test_json_field_not_indexed() {\n        let error = parse_query_to_logical_ast(\"json_not_indexed.titi:hello\", false).unwrap_err();\n        assert!(matches!(error, QueryParserError::FieldNotIndexed(_)));\n    }\n\n    fn test_query_to_logical_ast_with_default_json(\n        query: &str,\n        expected: &str,\n        default_conjunction: bool,\n    ) {\n        let mut query_parser = make_query_parser_with_default_fields(&[\"json\"]);\n        if default_conjunction {\n            query_parser.set_conjunction_by_default();\n        }\n        let ast = query_parser.parse_query_to_logical_ast(query).unwrap();\n        let ast_str = format!(\"{ast:?}\");\n        assert_eq!(ast_str, expected);\n    }\n\n    #[test]\n    fn test_json_default() {\n        test_query_to_logical_ast_with_default_json(\n            \"titi:4\",\n            \"(Term(field=14, type=Json, path=titi, type=I64, 4) Term(field=14, type=Json, \\\n             path=titi, type=Str, \\\"4\\\"))\",\n            false,\n        );\n    }\n\n    #[test]\n    fn test_json_default_with_different_field() {\n        for conjunction in [false, true] {\n            test_query_to_logical_ast_with_default_json(\n                \"text:4\",\n                r#\"Term(field=1, type=Str, \"4\")\"#,\n                conjunction,\n            );\n        }\n    }\n\n    #[test]\n    fn test_json_default_with_same_field() {\n        for conjunction in [false, true] {\n            test_query_to_logical_ast_with_default_json(\n                \"json:4\",\n                r#\"(Term(field=14, type=Json, path=, type=I64, 4) Term(field=14, type=Json, path=, type=Str, \"4\"))\"#,\n                conjunction,\n            );\n        }\n    }\n\n    #[test]\n    fn test_parse_bytes_phrase() {\n        test_parse_query_to_logical_ast_helper(\n            \"bytes:\\\"YnVidQ==\\\"\",\n            \"Term(field=12, type=Bytes, [98, 117, 98, 117])\",\n            false,\n        );\n    }\n\n    #[test]\n    fn test_parse_bytes_invalid_base64() {\n        let base64_err: QueryParserError =\n            parse_query_to_logical_ast(\"bytes:aa\", false).unwrap_err();\n        assert!(matches!(base64_err, QueryParserError::ExpectedBase64(_)));\n    }\n\n    #[test]\n    fn test_parse_query_to_ast_ab_c() {\n        test_parse_query_to_logical_ast_helper(\n            \"(+title:a +title:b) title:c\",\n            r#\"((+Term(field=0, type=Str, \"a\") +Term(field=0, type=Str, \"b\")) Term(field=0, type=Str, \"c\"))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"(+title:a +title:b) title:c\",\n            r#\"(+Term(field=0, type=Str, \"a\") +Term(field=0, type=Str, \"b\") +Term(field=0, type=Str, \"c\"))\"#,\n            true,\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_to_ast_single_term() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:toto\",\n            r#\"Term(field=0, type=Str, \"toto\")\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"+title:toto\",\n            r#\"Term(field=0, type=Str, \"toto\")\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"+title:toto -titi\",\n            r#\"(+Term(field=0, type=Str, \"toto\") -(Term(field=0, type=Str, \"titi\") Term(field=1, type=Str, \"titi\")))\"#,\n            false,\n        );\n    }\n\n    #[test]\n    fn test_single_negative_term() {\n        assert_matches!(\n            parse_query_to_logical_ast(\"-title:toto\", false),\n            Err(QueryParserError::AllButQueryForbidden)\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_to_ast_two_terms() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:a b\",\n            r#\"(Term(field=0, type=Str, \"a\") Term(field=0, type=Str, \"b\") Term(field=1, type=Str, \"b\"))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            r#\"title:\"a b\"\"#,\n            r#\"\"[(0, Term(field=0, type=Str, \"a\")), (1, Term(field=0, type=Str, \"b\"))]\"\"#,\n            false,\n        );\n    }\n    #[test]\n    pub fn test_parse_query_all_query() {\n        let logical_ast = parse_query_to_logical_ast(\"*\", false).unwrap();\n        assert_eq!(format!(\"{logical_ast:?}\"), \"*\");\n    }\n\n    #[test]\n    pub fn test_parse_query_range_require_a_target_field() {\n        let query_parser_error = parse_query_to_logical_ast(\"[A TO B]\", false).err().unwrap();\n        assert_eq!(\n            query_parser_error.to_string(),\n            \"Unsupported query: Range query need to target a specific field.\"\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_to_ast_ranges() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:[a TO b]\",\n            r#\"(Included(Term(field=0, type=Str, \"a\")) TO Included(Term(field=0, type=Str, \"b\")))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:{titi TO toto}\",\n            r#\"(Excluded(Term(field=0, type=Str, \"titi\")) TO Excluded(Term(field=0, type=Str, \"toto\")))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:{* TO toto}\",\n            r#\"(Unbounded TO Excluded(Term(field=0, type=Str, \"toto\")))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:{titi TO *}\",\n            r#\"(Excluded(Term(field=0, type=Str, \"titi\")) TO Unbounded)\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"signed:{-5 TO 3}\",\n            r#\"(Excluded(Term(field=2, type=I64, -5)) TO Excluded(Term(field=2, type=I64, 3)))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"float:{-1.5 TO 1.5}\",\n            r#\"(Excluded(Term(field=10, type=F64, -1.5)) TO Excluded(Term(field=10, type=F64, 1.5)))\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"u64_ff:[7 TO 77]\",\n            r#\"(Included(Term(field=18, type=U64, 7)) TO Included(Term(field=18, type=U64, 77)))\"#,\n            false,\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_field_does_not_exist() {\n        let query_parser = make_query_parser();\n        assert_eq!(\n            query_parser\n                .parse_query(\"boujou:\\\"18446744073709551615\\\"\")\n                .unwrap_err(),\n            QueryParserError::FieldDoesNotExist(\"boujou\".to_string())\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_field_not_indexed() {\n        let query_parser = make_query_parser();\n        assert_matches!(\n            query_parser.parse_query(\"notindexed_text:\\\"18446744073709551615\\\"\"),\n            Err(QueryParserError::FieldNotIndexed(_))\n        );\n    }\n\n    #[test]\n    pub fn test_unknown_tokenizer() {\n        let mut schema_builder = Schema::builder();\n        let text_field_indexing = TextFieldIndexing::default()\n            .set_tokenizer(\"nonexistingtokenizer\")\n            .set_index_option(IndexRecordOption::Basic);\n        let text_options = TextOptions::default().set_indexing_options(text_field_indexing);\n        let title = schema_builder.add_text_field(\"title\", text_options);\n        let schema = schema_builder.build();\n        let default_fields = vec![title];\n        let tokenizer_manager = TokenizerManager::default();\n        let query_parser = QueryParser::new(schema, default_fields, tokenizer_manager);\n\n        assert_matches!(\n            query_parser.parse_query(\"title:\\\"happy tax payer\\\"\"),\n            Err(QueryParserError::UnknownTokenizer { .. })\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_no_positions() {\n        let mut schema_builder = Schema::builder();\n        let text_field_indexing = TextFieldIndexing::default()\n            .set_tokenizer(\"customtokenizer\")\n            .set_index_option(IndexRecordOption::Basic);\n        let text_options = TextOptions::default().set_indexing_options(text_field_indexing);\n        let title = schema_builder.add_text_field(\"title\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        index\n            .tokenizers()\n            .register(\"customtokenizer\", SimpleTokenizer::default());\n        let query_parser = QueryParser::for_index(&index, vec![title]);\n        assert_eq!(\n            query_parser.parse_query(\"title:\\\"happy tax\\\"\").unwrap_err(),\n            QueryParserError::FieldDoesNotHavePositionsIndexed(\"title\".to_string())\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_expected_int() {\n        let query_parser = make_query_parser();\n        assert_matches!(\n            query_parser.parse_query(\"unsigned:18a\"),\n            Err(QueryParserError::ExpectedInt(_))\n        );\n        assert!(query_parser.parse_query(\"unsigned:\\\"18\\\"\").is_ok());\n        assert_matches!(\n            query_parser.parse_query(\"signed:18b\"),\n            Err(QueryParserError::ExpectedInt(_))\n        );\n        assert!(query_parser.parse_query(\"float:\\\"1.8\\\"\").is_ok());\n        assert_matches!(\n            query_parser.parse_query(\"float:1.8a\"),\n            Err(QueryParserError::ExpectedFloat(_))\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_expected_bool() {\n        let query_parser = make_query_parser();\n        assert_matches!(\n            query_parser.parse_query(\"bool:brie\"),\n            Err(QueryParserError::ExpectedBool(_))\n        );\n        assert!(query_parser.parse_query(\"bool:\\\"true\\\"\").is_ok());\n        assert!(query_parser.parse_query(\"bool:\\\"false\\\"\").is_ok());\n    }\n\n    #[test]\n    pub fn test_query_parser_expected_date() {\n        let query_parser = make_query_parser();\n        assert_matches!(\n            query_parser.parse_query(\"date:18a\"),\n            Err(QueryParserError::DateFormatError(_))\n        );\n        test_parse_query_to_logical_ast_helper(\n            r#\"date:\"2010-11-21T09:55:06.000000000+02:00\"\"#,\n            r#\"Term(field=9, type=Date, 2010-11-21T07:55:06Z)\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            r#\"date:\"1985-04-12T23:20:50.52Z\"\"#,\n            r#\"Term(field=9, type=Date, 1985-04-12T23:20:50Z)\"#,\n            true,\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_expected_facet() {\n        let query_parser = make_query_parser();\n        match query_parser.parse_query(\"facet:INVALID\") {\n            Ok(_) => panic!(\"should never succeed\"),\n            Err(e) => assert_eq!(\n                \"The facet field is malformed: Failed to parse the facet string: 'INVALID'\",\n                format!(\"{e}\")\n            ),\n        }\n        assert!(query_parser.parse_query(\"facet:\\\"/foo/bar\\\"\").is_ok());\n    }\n\n    #[test]\n    pub fn test_query_parser_not_empty_but_no_tokens() {\n        let query_parser = make_query_parser();\n        assert!(query_parser.parse_query(\" !, \").is_ok());\n        assert!(query_parser.parse_query(\"with_stop_words:the\").is_ok());\n    }\n\n    #[test]\n    pub fn test_parse_query_single_negative_term_through_error() {\n        assert_matches!(\n            parse_query_to_logical_ast(\"-title:toto\", true),\n            Err(QueryParserError::AllButQueryForbidden)\n        );\n        assert_matches!(\n            parse_query_to_logical_ast(\"-title:toto\", false),\n            Err(QueryParserError::AllButQueryForbidden)\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_to_ast_conjunction() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:toto\",\n            r#\"Term(field=0, type=Str, \"toto\")\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"+title:toto\",\n            r#\"Term(field=0, type=Str, \"toto\")\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"+title:toto -titi\",\n            r#\"(+Term(field=0, type=Str, \"toto\") -(Term(field=0, type=Str, \"titi\") Term(field=1, type=Str, \"titi\")))\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:a b\",\n            r#\"(+Term(field=0, type=Str, \"a\") +(Term(field=0, type=Str, \"b\") Term(field=1, type=Str, \"b\")))\"#,\n            true,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:\\\"a b\\\"\",\n            r#\"\"[(0, Term(field=0, type=Str, \"a\")), (1, Term(field=0, type=Str, \"b\"))]\"\"#,\n            true,\n        );\n    }\n\n    #[test]\n    pub fn test_parse_query_negative() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:b -title:a\",\n            r#\"(+Term(field=0, type=Str, \"b\") -Term(field=0, type=Str, \"a\"))\"#,\n            true,\n        );\n\n        test_parse_query_to_logical_ast_helper(\n            \"title:b -(-title:a -title:c)\",\n            r#\"(+Term(field=0, type=Str, \"b\") -(-Term(field=0, type=Str, \"a\") -Term(field=0, type=Str, \"c\")))\"#,\n            true,\n        );\n    }\n\n    #[test]\n    pub fn test_query_parser_hyphen() {\n        test_parse_query_to_logical_ast_helper(\n            \"title:www-form-encoded\",\n            r#\"\"[(0, Term(field=0, type=Str, \"www\")), (1, Term(field=0, type=Str, \"form\")), (2, Term(field=0, type=Str, \"encoded\"))]\"\"#,\n            false,\n        );\n    }\n\n    #[test]\n    fn test_and_default_regardless_of_default_conjunctive() {\n        for &default_conjunction in &[false, true] {\n            test_parse_query_to_logical_ast_helper(\n                \"title:a AND title:b\",\n                r#\"(+Term(field=0, type=Str, \"a\") +Term(field=0, type=Str, \"b\"))\"#,\n                default_conjunction,\n            );\n        }\n    }\n\n    #[test]\n    fn test_or_default_conjunctive() {\n        for &default_conjunction in &[false, true] {\n            test_parse_query_to_logical_ast_helper(\n                \"title:a OR title:b\",\n                r#\"(Term(field=0, type=Str, \"a\") Term(field=0, type=Str, \"b\"))\"#,\n                default_conjunction,\n            );\n        }\n    }\n\n    #[test]\n    fn test_space_before_value() {\n        test_parse_query_to_logical_ast_helper(\n            \"title: a\",\n            r#\"Term(field=0, type=Str, \"a\")\"#,\n            false,\n        );\n    }\n\n    #[test]\n    fn test_escaped_field() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(r\"a\\.b\", STRING);\n        let schema = schema_builder.build();\n        let query_parser = QueryParser::new(schema, Vec::new(), TokenizerManager::default());\n        let query = query_parser.parse_query(r\"a\\.b:hello\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            \"TermQuery(Term(field=0, type=Str, \\\"hello\\\"))\"\n        );\n    }\n\n    #[test]\n    fn test_split_full_path() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"second\", STRING);\n        schema_builder.add_text_field(\"first\", STRING);\n        schema_builder.add_text_field(\"first.toto\", STRING);\n        schema_builder.add_text_field(\"first.toto.titi\", STRING);\n        schema_builder.add_text_field(\"third.a.b.c\", STRING);\n        let schema = schema_builder.build();\n        let query_parser =\n            QueryParser::new(schema.clone(), Vec::new(), TokenizerManager::default());\n        assert_eq!(\n            query_parser.split_full_path(\"first.toto\"),\n            Some((schema.get_field(\"first.toto\").unwrap(), \"\"))\n        );\n        assert_eq!(\n            query_parser.split_full_path(\"first.toto.bubu\"),\n            Some((schema.get_field(\"first.toto\").unwrap(), \"bubu\"))\n        );\n        assert_eq!(\n            query_parser.split_full_path(\"first.toto.titi\"),\n            Some((schema.get_field(\"first.toto.titi\").unwrap(), \"\"))\n        );\n        assert_eq!(\n            query_parser.split_full_path(\"first.titi\"),\n            Some((schema.get_field(\"first\").unwrap(), \"titi\"))\n        );\n        assert_eq!(query_parser.split_full_path(\"third\"), None);\n        assert_eq!(query_parser.split_full_path(\"hello.toto\"), None);\n        assert_eq!(query_parser.split_full_path(\"\"), None);\n        assert_eq!(query_parser.split_full_path(\"firsty\"), None);\n    }\n\n    #[test]\n    pub fn test_phrase_slop() {\n        test_parse_query_to_logical_ast_helper(\n            \"\\\"a b\\\"~0\",\n            r#\"(\"[(0, Term(field=0, type=Str, \"a\")), (1, Term(field=0, type=Str, \"b\"))]\" \"[(0, Term(field=1, type=Str, \"a\")), (1, Term(field=1, type=Str, \"b\"))]\")\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"\\\"a b\\\"~2\",\n            r#\"(\"[(0, Term(field=0, type=Str, \"a\")), (1, Term(field=0, type=Str, \"b\"))]\"~2 \"[(0, Term(field=1, type=Str, \"a\")), (1, Term(field=1, type=Str, \"b\"))]\"~2)\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"title:\\\"a b~4\\\"~2\",\n            r#\"\"[(0, Term(field=0, type=Str, \"a\")), (1, Term(field=0, type=Str, \"b\")), (2, Term(field=0, type=Str, \"4\"))]\"~2\"#,\n            false,\n        );\n    }\n\n    #[test]\n    pub fn test_phrase_prefix() {\n        test_parse_query_to_logical_ast_helper(\n            \"\\\"big bad wo\\\"*\",\n            r#\"(\"[(0, Term(field=0, type=Str, \"big\")), (1, Term(field=0, type=Str, \"bad\")), (2, Term(field=0, type=Str, \"wo\"))]\"* \"[(0, Term(field=1, type=Str, \"big\")), (1, Term(field=1, type=Str, \"bad\")), (2, Term(field=1, type=Str, \"wo\"))]\"*)\"#,\n            false,\n        );\n\n        let query_parser = make_query_parser();\n        let query = query_parser.parse_query(\"\\\"big bad wo\\\"*\").unwrap();\n        assert_eq!(\n            format!(\"{query:?}\"),\n            \"BooleanQuery { subqueries: [(Should, PhrasePrefixQuery { field: Field(0), \\\n             phrase_terms: [(0, Term(field=0, type=Str, \\\"big\\\")), (1, Term(field=0, type=Str, \\\n             \\\"bad\\\"))], prefix: (2, Term(field=0, type=Str, \\\"wo\\\")), max_expansions: 50 }), \\\n             (Should, PhrasePrefixQuery { field: Field(1), phrase_terms: [(0, Term(field=1, \\\n             type=Str, \\\"big\\\")), (1, Term(field=1, type=Str, \\\"bad\\\"))], prefix: (2, \\\n             Term(field=1, type=Str, \\\"wo\\\")), max_expansions: 50 })], \\\n             minimum_number_should_match: 1 }\"\n        );\n    }\n\n    #[test]\n    pub fn test_phrase_prefix_too_short() {\n        let err = parse_query_to_logical_ast(\"\\\"wo\\\"*\", true).unwrap_err();\n        assert_eq!(\n            err,\n            QueryParserError::PhrasePrefixRequiresAtLeastTwoTerms {\n                phrase: \"wo\".to_owned(),\n                tokenizer: \"default\".to_owned()\n            }\n        );\n\n        let err = parse_query_to_logical_ast(\"\\\"\\\"*\", true).unwrap_err();\n        assert_eq!(\n            err,\n            QueryParserError::PhrasePrefixRequiresAtLeastTwoTerms {\n                phrase: \"\".to_owned(),\n                tokenizer: \"default\".to_owned()\n            }\n        );\n    }\n\n    #[test]\n    pub fn test_term_set_query() {\n        test_parse_query_to_logical_ast_helper(\n            \"title: IN [a b cd]\",\n            r#\"IN [Term(field=0, type=Str, \"a\"), Term(field=0, type=Str, \"b\"), Term(field=0, type=Str, \"cd\")]\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"bytes: IN [AA== ABA= ABCD]\",\n            r#\"IN [Term(field=12, type=Bytes, [0]), Term(field=12, type=Bytes, [0, 16]), Term(field=12, type=Bytes, [0, 16, 131])]\"#,\n            false,\n        );\n        test_parse_query_to_logical_ast_helper(\n            \"signed: IN [1 2 -3]\",\n            r#\"IN [Term(field=2, type=I64, 1), Term(field=2, type=I64, 2), Term(field=2, type=I64, -3)]\"#,\n            false,\n        );\n\n        test_parse_query_to_logical_ast_helper(\n            \"float: IN [1.1 2.2 -3.3]\",\n            r#\"IN [Term(field=10, type=F64, 1.1), Term(field=10, type=F64, 2.2), Term(field=10, type=F64, -3.3)]\"#,\n            false,\n        );\n    }\n\n    #[test]\n    pub fn test_set_field_fuzzy() {\n        {\n            let mut query_parser = make_query_parser();\n            query_parser.set_field_fuzzy(\n                query_parser.schema.get_field(\"title\").unwrap(),\n                false,\n                1,\n                true,\n            );\n            let query = query_parser.parse_query(\"abc\").unwrap();\n            assert_eq!(\n                format!(\"{query:?}\"),\n                \"BooleanQuery { subqueries: [(Should, FuzzyTermQuery { term: Term(field=0, \\\n                 type=Str, \\\"abc\\\"), distance: 1, transposition_cost_one: true, prefix: false }), \\\n                 (Should, TermQuery(Term(field=1, type=Str, \\\"abc\\\")))], \\\n                 minimum_number_should_match: 1 }\"\n            );\n        }\n\n        {\n            let mut query_parser = make_query_parser();\n            query_parser.set_field_fuzzy(\n                query_parser.schema.get_field(\"text\").unwrap(),\n                true,\n                2,\n                false,\n            );\n            let query = query_parser.parse_query(\"abc\").unwrap();\n            assert_eq!(\n                format!(\"{query:?}\"),\n                \"BooleanQuery { subqueries: [(Should, TermQuery(Term(field=0, type=Str, \\\n                 \\\"abc\\\"))), (Should, FuzzyTermQuery { term: Term(field=1, type=Str, \\\"abc\\\"), \\\n                 distance: 2, transposition_cost_one: false, prefix: true })], \\\n                 minimum_number_should_match: 1 }\"\n            );\n        }\n    }\n\n    #[test]\n    pub fn test_set_default_field_integer() {\n        test_parse_query_to_logical_ast_helper_with_default_fields(\n            \"2324\",\n            \"(Term(field=0, type=Str, \\\"2324\\\") Term(field=2, type=I64, 2324))\",\n            false,\n            &[\"title\", \"signed\"],\n        );\n\n        test_parse_query_to_logical_ast_helper_with_default_fields(\n            \"abc\",\n            \"Term(field=0, type=Str, \\\"abc\\\")\",\n            false,\n            &[\"title\", \"signed\"],\n        );\n\n        let query_parser = make_query_parser_with_default_fields(&[\"signed\"]);\n        assert_matches!(\n            query_parser.parse_query(\"abc\"),\n            Err(QueryParserError::ExpectedInt(_))\n        );\n    }\n\n    #[test]\n    pub fn test_deduplication() {\n        let query = \"be be\";\n        test_parse_query_to_logical_ast_helper(\n            query,\n            \"(Term(field=0, type=Str, \\\"be\\\") Term(field=1, type=Str, \\\"be\\\"))\",\n            false,\n        );\n    }\n\n    #[test]\n    pub fn test_regex() {\n        let expected_regex = tantivy_fst::Regex::new(r\".*b\").unwrap();\n        test_parse_query_to_logical_ast_helper(\n            \"title:/.*b/\",\n            format!(\"Regex(Field(0), {:#?})\", expected_regex).as_str(),\n            false,\n        );\n        let expected_regex2 = tantivy_fst::Regex::new(r\".*a\").unwrap();\n        test_parse_query_to_logical_ast_helper(\n            \"title:(/.*b/ OR /.*a/)\",\n            format!(\n                \"(Regex(Field(0), {:#?}) Regex(Field(0), {:#?}))\",\n                expected_regex, expected_regex2\n            )\n            .as_str(),\n            false,\n        );\n\n        // Invalid field\n        let err = parse_query_to_logical_ast(\"float:/.*b/\", false).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Unsupported query: Regex query only supported on text fields\"\n        );\n\n        // No field specified\n        let err = parse_query_to_logical_ast(\"/.*b/\", false).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Unsupported query: Regex query need to target a specific field.\"\n        );\n\n        // Regex on a json path\n        let err = parse_query_to_logical_ast(\"title.subpath:/.*b/\", false).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Unsupported query: Regex query does not support json paths.\"\n        );\n\n        // Invalid regex\n        let err = parse_query_to_logical_ast(\"title:/[A-Z*b/\", false).unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Unsupported query: Invalid regex: regex parse error:\\n    [A-Z*b\\n    ^\\nerror: \\\n             unclosed character class\"\n        );\n\n        // Regexes not allowed\n        let err = parse_query_to_logical_ast_with_default_fields(\n            \"title:/.*b/\",\n            false,\n            &[\"title\", \"text\"],\n            false,\n        )\n        .unwrap_err();\n        assert_eq!(\n            err.to_string(),\n            \"Unsupported query: Regex queries are not allowed.\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/query/range_query/fast_field_range_doc_set.rs",
    "content": "use core::fmt::Debug;\nuse std::ops::RangeInclusive;\n\nuse columnar::Column;\n\nuse crate::{DocId, DocSet, TERMINATED};\n\n/// Helper to have a cursor over a vec of docids\n#[derive(Debug)]\nstruct VecCursor {\n    docs: Vec<u32>,\n    current_pos: usize,\n}\nimpl VecCursor {\n    fn new() -> Self {\n        Self {\n            docs: Vec::with_capacity(32),\n            current_pos: 0,\n        }\n    }\n    fn next(&mut self) -> Option<u32> {\n        self.current_pos += 1;\n        self.current()\n    }\n    #[inline]\n    fn current(&self) -> Option<u32> {\n        self.docs.get(self.current_pos).copied()\n    }\n    fn get_cleared_data(&mut self) -> &mut Vec<u32> {\n        self.docs.clear();\n        self.current_pos = 0;\n        &mut self.docs\n    }\n    fn last_doc(&self) -> Option<u32> {\n        self.docs.last().cloned()\n    }\n    fn is_empty(&self) -> bool {\n        self.current().is_none()\n    }\n}\n\npub(crate) struct RangeDocSet<T> {\n    /// The range filter on the values.\n    value_range: RangeInclusive<T>,\n    column: Column<T>,\n    /// The next docid start range to fetch (inclusive).\n    next_fetch_start: u32,\n    /// Number of docs range checked in a batch.\n    ///\n    /// There are two patterns.\n    /// - We do a full scan. => We can load large chunks. We don't know in advance if seek call\n    ///   will come, so we start with small chunks\n    /// - We load docs, interspersed with seek calls. When there are big jumps in the seek, we\n    ///   should load small chunks. When the seeks are small, we can employ the same strategy as on\n    ///   a full scan.\n    fetch_horizon: u32,\n    /// Current batch of loaded docs.\n    loaded_docs: VecCursor,\n    last_seek_pos_opt: Option<u32>,\n}\n\nconst DEFAULT_FETCH_HORIZON: u32 = 128;\nimpl<T: Send + Sync + PartialOrd + Copy + Debug + 'static> RangeDocSet<T> {\n    pub(crate) fn new(value_range: RangeInclusive<T>, column: Column<T>) -> Self {\n        if *value_range.start() > column.max_value() || *value_range.end() < column.min_value() {\n            return Self {\n                value_range,\n                column,\n                loaded_docs: VecCursor::new(),\n                next_fetch_start: TERMINATED,\n                fetch_horizon: DEFAULT_FETCH_HORIZON,\n                last_seek_pos_opt: None,\n            };\n        }\n\n        let mut range_docset = Self {\n            value_range,\n            column,\n            loaded_docs: VecCursor::new(),\n            next_fetch_start: 0,\n            fetch_horizon: DEFAULT_FETCH_HORIZON,\n            last_seek_pos_opt: None,\n        };\n        range_docset.reset_fetch_range();\n        range_docset.fetch_block();\n        range_docset\n    }\n\n    fn reset_fetch_range(&mut self) {\n        self.fetch_horizon = DEFAULT_FETCH_HORIZON;\n    }\n\n    /// Returns true if more data could be fetched\n    fn fetch_block(&mut self) {\n        if self.next_fetch_start >= self.column.num_docs() {\n            return;\n        }\n        const MAX_HORIZON: u32 = 100_000;\n        while self.loaded_docs.is_empty() {\n            let finished_to_end = self.fetch_horizon(self.fetch_horizon);\n            if finished_to_end {\n                break;\n            }\n            // Fetch more data, increase horizon. Horizon only gets reset when doing a seek.\n            self.fetch_horizon = (self.fetch_horizon * 2).min(MAX_HORIZON);\n        }\n    }\n\n    /// check if the distance between the seek calls is large\n    fn is_last_seek_distance_large(&self, new_seek: DocId) -> bool {\n        if let Some(last_seek_pos) = self.last_seek_pos_opt {\n            (new_seek - last_seek_pos) >= 128\n        } else {\n            true\n        }\n    }\n\n    /// Fetches a block for docid range [next_fetch_start .. next_fetch_start + HORIZON]\n    fn fetch_horizon(&mut self, horizon: u32) -> bool {\n        let mut finished_to_end = false;\n\n        let num_docs = self.column.num_docs();\n        let mut fetch_end = self.next_fetch_start + horizon;\n        if fetch_end >= num_docs {\n            fetch_end = num_docs;\n            finished_to_end = true;\n        }\n\n        let last_doc = self.loaded_docs.last_doc();\n        let doc_buffer: &mut Vec<DocId> = self.loaded_docs.get_cleared_data();\n        self.column.get_docids_for_value_range(\n            self.value_range.clone(),\n            self.next_fetch_start..fetch_end,\n            doc_buffer,\n        );\n        if let Some(last_doc) = last_doc {\n            while self.loaded_docs.current() == Some(last_doc) {\n                self.loaded_docs.next();\n            }\n        }\n        self.next_fetch_start = fetch_end;\n\n        finished_to_end\n    }\n}\n\nimpl<T: Send + Sync + PartialOrd + Copy + Debug + 'static> DocSet for RangeDocSet<T> {\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        if let Some(docid) = self.loaded_docs.next() {\n            return docid;\n        }\n        self.fetch_block();\n        self.loaded_docs.current().unwrap_or(TERMINATED)\n    }\n\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.loaded_docs.current().unwrap_or(TERMINATED)\n    }\n\n    /// Advances the `DocSet` forward until reaching the target, or going to the\n    /// lowest [`DocId`] greater than the target.\n    ///\n    /// If the end of the `DocSet` is reached, [`TERMINATED`] is returned.\n    ///\n    /// Calling `.seek(target)` on a terminated `DocSet` is legal. Implementation\n    /// of `DocSet` should support it.\n    ///\n    /// Calling `seek(TERMINATED)` is also legal and is the normal way to consume a `DocSet`.\n    fn seek(&mut self, target: DocId) -> DocId {\n        if self.is_last_seek_distance_large(target) {\n            self.reset_fetch_range();\n        }\n        if target > self.next_fetch_start {\n            self.next_fetch_start = target;\n        }\n        let mut doc = self.doc();\n        debug_assert!(doc <= target);\n        while doc < target {\n            doc = self.advance();\n        }\n        self.last_seek_pos_opt = Some(target);\n        doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        // TODO: Implement a better size hint\n        self.column.num_docs() / 10\n    }\n\n    /// Returns a best-effort hint of the\n    /// cost to drive the docset.\n    fn cost(&self) -> u64 {\n        // Advancing the docset is pretty expensive since it scans the whole column, there is no\n        // index currently (will change with an kd-tree)\n        // Since we use SIMD to scan the fast field range query we lower the cost a little bit,\n        // assuming that we hit 10% of the docs like in size_hint.\n        //\n        // If we would return a cost higher than num_docs, we would never choose ff range query as\n        // the driver in a DocSet, when intersecting a term query with a fast field. But\n        // it's the faster choice when the term query has a lot of docids and the range\n        // query has not.\n        //\n        // Ideally this would take the fast field codec into account\n        (self.column.num_docs() as f64 * 0.8) as u64\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::ops::Bound;\n\n    use crate::collector::Count;\n    use crate::directory::RamDirectory;\n    use crate::query::RangeQuery;\n    use crate::{schema, IndexBuilder, TantivyDocument, Term};\n\n    #[test]\n    fn range_query_fast_optional_field_minimum() {\n        let mut schema_builder = schema::SchemaBuilder::new();\n        let id_field = schema_builder.add_text_field(\"id\", schema::STRING);\n        let score_field = schema_builder.add_u64_field(\"score\", schema::FAST | schema::INDEXED);\n\n        let dir = RamDirectory::default();\n        let index = IndexBuilder::new()\n            .schema(schema_builder.build())\n            .open_or_create(dir)\n            .unwrap();\n\n        {\n            let mut writer = index.writer(15_000_000).unwrap();\n\n            let count = 1000;\n            for i in 0..count {\n                let mut doc = TantivyDocument::new();\n                doc.add_text(id_field, format!(\"doc{i}\"));\n\n                let nb_scores = i % 2; // 0 or 1 scores\n                for _ in 0..nb_scores {\n                    doc.add_u64(score_field, 80);\n                }\n\n                writer.add_document(doc).unwrap();\n            }\n            writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let query = RangeQuery::new(\n            Bound::Included(Term::from_field_u64(score_field, 70)),\n            Bound::Unbounded,\n        );\n\n        let count = searcher.search(&query, &Count).unwrap();\n        assert_eq!(count, 500);\n    }\n\n    #[test]\n    fn range_query_no_overlap_optimization() {\n        let mut schema_builder = schema::SchemaBuilder::new();\n        let id_field = schema_builder.add_text_field(\"id\", schema::STRING);\n        let value_field = schema_builder.add_u64_field(\"value\", schema::FAST | schema::INDEXED);\n\n        let dir = RamDirectory::default();\n        let index = IndexBuilder::new()\n            .schema(schema_builder.build())\n            .open_or_create(dir)\n            .unwrap();\n\n        {\n            let mut writer = index.writer(15_000_000).unwrap();\n\n            // Add documents with values in the range [10, 20]\n            for i in 0..100 {\n                let mut doc = TantivyDocument::new();\n                doc.add_text(id_field, format!(\"doc{i}\"));\n                doc.add_u64(value_field, 10 + (i % 11) as u64); // values in range 10-20\n\n                writer.add_document(doc).unwrap();\n            }\n            writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        // Test a range query [100, 200] that has no overlap with data range [10, 20]\n        let query = RangeQuery::new(\n            Bound::Included(Term::from_field_u64(value_field, 100)),\n            Bound::Included(Term::from_field_u64(value_field, 200)),\n        );\n\n        let count = searcher.search(&query, &Count).unwrap();\n        assert_eq!(count, 0); // should return 0 results since there's no overlap\n\n        // Test another non-overlapping range: [0, 5] while data range is [10, 20]\n        let query2 = RangeQuery::new(\n            Bound::Included(Term::from_field_u64(value_field, 0)),\n            Bound::Included(Term::from_field_u64(value_field, 5)),\n        );\n\n        let count2 = searcher.search(&query2, &Count).unwrap();\n        assert_eq!(count2, 0); // should return 0 results since there's no overlap\n    }\n}\n"
  },
  {
    "path": "src/query/range_query/mod.rs",
    "content": "use crate::schema::Type;\n\nmod fast_field_range_doc_set;\nmod range_query;\nmod range_query_fastfield;\n\npub use common::bounds::BoundsRange;\n\npub use self::range_query::*;\npub use self::range_query_fastfield::*;\n\n// TODO is this correct?\npub(crate) fn is_type_valid_for_fastfield_range_query(typ: Type) -> bool {\n    match typ {\n        Type::Str\n        | Type::U64\n        | Type::I64\n        | Type::F64\n        | Type::Bool\n        | Type::Date\n        | Type::Json\n        | Type::IpAddr\n        | Type::Bytes => true,\n        Type::Facet => false,\n    }\n}\n"
  },
  {
    "path": "src/query/range_query/range_query.rs",
    "content": "use std::io;\nuse std::ops::Bound;\n\nuse common::bounds::{map_bound, BoundsRange};\nuse common::BitSet;\n\nuse super::range_query_fastfield::FastFieldRangeWeight;\nuse crate::index::SegmentReader;\nuse crate::query::explanation::does_not_match;\nuse crate::query::range_query::is_type_valid_for_fastfield_range_query;\nuse crate::query::{BitSetDocSet, ConstScorer, EnableScoring, Explanation, Query, Scorer, Weight};\nuse crate::schema::{Field, IndexRecordOption, Term, Type};\nuse crate::termdict::{TermDictionary, TermStreamer};\nuse crate::{DocId, Score};\n\n/// `RangeQuery` matches all documents that have at least one term within a defined range.\n///\n/// Matched document will all get a constant `Score` of one.\n///\n/// # Implementation\n///\n/// ## Default\n/// The default implementation collects all documents _upfront_ into a `BitSet`.\n/// This is done by iterating over the terms within the range and loading all docs for each\n/// `TermInfo` from the inverted index (posting list) and put them into a `BitSet`.\n/// Depending on the number of terms matched, this is a potentially expensive operation.\n///\n/// ## IP fast field\n/// For IP fast fields a custom variant is used, by scanning the fast field. Unlike the default\n/// variant we can walk in a lazy fashion over it, since the fastfield is implicit orderered by\n/// DocId.\n///\n///\n/// # Example\n///\n/// ```rust\n/// use tantivy::collector::Count;\n/// use tantivy::query::RangeQuery;\n/// use tantivy::Term;\n/// use tantivy::schema::{Schema, INDEXED};\n/// use tantivy::{doc, Index, IndexWriter};\n/// use std::ops::Bound;\n/// # fn test() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let year_field = schema_builder.add_u64_field(\"year\", INDEXED);\n/// let schema = schema_builder.build();\n///\n/// let index = Index::create_in_ram(schema);\n/// let mut index_writer: IndexWriter = index.writer_with_num_threads(1, 20_000_000)?;\n/// for year in 1950u64..2017u64 {\n///     let num_docs_within_year = 10 + (year - 1950) * (year - 1950);\n///     for _ in 0..num_docs_within_year {\n///       index_writer.add_document(doc!(year_field => year))?;\n///     }\n/// }\n/// index_writer.commit()?;\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n/// let docs_in_the_sixties = RangeQuery::new(\n///     Bound::Included(Term::from_field_u64(year_field, 1960)),\n///     Bound::Excluded(Term::from_field_u64(year_field, 1970)),\n/// );\n/// let num_60s_books = searcher.search(&docs_in_the_sixties, &Count)?;\n/// assert_eq!(num_60s_books, 2285);\n/// Ok(())\n/// # }\n/// # assert!(test().is_ok());\n/// ```\n#[derive(Clone, Debug)]\npub struct RangeQuery {\n    bounds: BoundsRange<Term>,\n}\n\nimpl RangeQuery {\n    /// Creates a new `RangeQuery` from bounded start and end terms.\n    ///\n    /// If the value type is not correct, something may go terribly wrong when\n    /// the `Weight` object is created.\n    pub fn new(lower_bound: Bound<Term>, upper_bound: Bound<Term>) -> RangeQuery {\n        RangeQuery {\n            bounds: BoundsRange::new(lower_bound, upper_bound),\n        }\n    }\n\n    /// Field to search over\n    pub fn field(&self) -> Field {\n        self.get_term().field()\n    }\n\n    /// The value type of the field\n    pub fn value_type(&self) -> Type {\n        self.get_term().typ()\n    }\n\n    pub(crate) fn get_term(&self) -> &Term {\n        self.bounds\n            .get_inner()\n            .expect(\"At least one bound must be set\")\n    }\n}\n\nimpl Query for RangeQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let schema = enable_scoring.schema();\n        let field_type = schema.get_field_entry(self.field()).field_type();\n\n        if field_type.is_fast() && is_type_valid_for_fastfield_range_query(self.value_type()) {\n            Ok(Box::new(FastFieldRangeWeight::new(self.bounds.clone())))\n        } else {\n            if field_type.is_json() {\n                return Err(crate::TantivyError::InvalidArgument(\n                    \"RangeQuery on JSON is only supported for fast fields currently\".to_string(),\n                ));\n            }\n            Ok(Box::new(InvertedIndexRangeWeight::new(\n                self.field(),\n                &self.bounds.lower_bound,\n                &self.bounds.upper_bound,\n                None,\n            )))\n        }\n    }\n}\n\n#[derive(Clone, Debug)]\n/// `InvertedIndexRangeQuery` is the same as [RangeQuery] but only uses the inverted index\npub struct InvertedIndexRangeQuery {\n    bounds: BoundsRange<Term>,\n    limit: Option<u64>,\n}\nimpl InvertedIndexRangeQuery {\n    /// Create new `InvertedIndexRangeQuery`\n    pub fn new(lower_bound: Bound<Term>, upper_bound: Bound<Term>) -> InvertedIndexRangeQuery {\n        InvertedIndexRangeQuery {\n            bounds: BoundsRange::new(lower_bound, upper_bound),\n            limit: None,\n        }\n    }\n    /// Limit the number of term the `RangeQuery` will go through.\n    ///\n    /// This does not limit the number of matching document, only the number of\n    /// different terms that get matched.\n    pub fn limit(&mut self, limit: u64) {\n        self.limit = Some(limit);\n    }\n}\n\nimpl Query for InvertedIndexRangeQuery {\n    fn weight(&self, _enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        let field = self\n            .bounds\n            .get_inner()\n            .expect(\"At least one bound must be set\")\n            .field();\n\n        Ok(Box::new(InvertedIndexRangeWeight::new(\n            field,\n            &self.bounds.lower_bound,\n            &self.bounds.upper_bound,\n            self.limit,\n        )))\n    }\n}\n\n/// Range weight on the inverted index\npub struct InvertedIndexRangeWeight {\n    field: Field,\n    lower_bound: Bound<Vec<u8>>,\n    upper_bound: Bound<Vec<u8>>,\n    limit: Option<u64>,\n}\n\nimpl InvertedIndexRangeWeight {\n    /// Creates a new RangeWeight\n    ///\n    /// Note: The limit is only enabled with the quickwit feature flag.\n    pub fn new(\n        field: Field,\n        lower_bound: &Bound<Term>,\n        upper_bound: &Bound<Term>,\n        limit: Option<u64>,\n    ) -> Self {\n        let verify_and_unwrap_term = |val: &Term| val.serialized_value_bytes().to_owned();\n        Self {\n            field,\n            lower_bound: map_bound(lower_bound, verify_and_unwrap_term),\n            upper_bound: map_bound(upper_bound, verify_and_unwrap_term),\n            limit,\n        }\n    }\n\n    fn term_range<'a>(&self, term_dict: &'a TermDictionary) -> io::Result<TermStreamer<'a>> {\n        use std::ops::Bound::*;\n        let mut term_stream_builder = term_dict.range();\n        term_stream_builder = match self.lower_bound {\n            Included(ref term_val) => term_stream_builder.ge(term_val),\n            Excluded(ref term_val) => term_stream_builder.gt(term_val),\n            Unbounded => term_stream_builder,\n        };\n        term_stream_builder = match self.upper_bound {\n            Included(ref term_val) => term_stream_builder.le(term_val),\n            Excluded(ref term_val) => term_stream_builder.lt(term_val),\n            Unbounded => term_stream_builder,\n        };\n        #[cfg(feature = \"quickwit\")]\n        if let Some(limit) = self.limit {\n            term_stream_builder = term_stream_builder.limit(limit);\n        }\n        term_stream_builder.into_stream()\n    }\n}\n\nimpl Weight for InvertedIndexRangeWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        let max_doc = reader.max_doc();\n        let mut doc_bitset = BitSet::with_max_value(max_doc);\n\n        let inverted_index = reader.inverted_index(self.field)?;\n        let term_dict = inverted_index.terms();\n        let mut term_range = self.term_range(term_dict)?;\n        let mut processed_count = 0;\n        while term_range.advance() {\n            if let Some(limit) = self.limit {\n                if limit <= processed_count {\n                    break;\n                }\n            }\n            processed_count += 1;\n            let term_info = term_range.value();\n            let mut block_segment_postings = inverted_index\n                .read_block_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;\n            loop {\n                let docs = block_segment_postings.docs();\n                if docs.is_empty() {\n                    break;\n                }\n                for &doc in block_segment_postings.docs() {\n                    doc_bitset.insert(doc);\n                }\n                block_segment_postings.advance();\n            }\n        }\n        let doc_bitset = BitSetDocSet::from(doc_bitset);\n        Ok(Box::new(ConstScorer::new(doc_bitset, boost)))\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) != doc {\n            return Err(does_not_match(doc));\n        }\n        Ok(Explanation::new(\"RangeQuery\", 1.0))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::net::IpAddr;\n    use std::ops::Bound;\n    use std::str::FromStr;\n\n    use rand::seq::SliceRandom;\n\n    use super::RangeQuery;\n    use crate::collector::{Count, TopDocs};\n    use crate::indexer::NoMergePolicy;\n    use crate::query::range_query::fast_field_range_doc_set::RangeDocSet;\n    use crate::query::range_query::range_query::InvertedIndexRangeQuery;\n    use crate::query::{AllScorer, ConstScorer, EmptyScorer, EnableScoring, Query, QueryParser};\n    use crate::schema::{\n        Field, IntoIpv6Addr, Schema, TantivyDocument, FAST, INDEXED, STORED, TEXT,\n    };\n    use crate::{Index, IndexWriter, Term};\n\n    #[test]\n    fn test_range_query_simple() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let year_field = schema_builder.add_u64_field(\"year\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            for year in 1950u64..2017u64 {\n                let num_docs_within_year = 10 + (year - 1950) * (year - 1950);\n                for _ in 0..num_docs_within_year {\n                    index_writer.add_document(doc!(year_field => year))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let docs_in_the_sixties = InvertedIndexRangeQuery::new(\n            Bound::Included(Term::from_field_u64(year_field, 1960)),\n            Bound::Excluded(Term::from_field_u64(year_field, 1970)),\n        );\n\n        // ... or `1960..=1969` if inclusive range is enabled.\n        let count = searcher.search(&docs_in_the_sixties, &Count)?;\n        assert_eq!(count, 2285);\n        Ok(())\n    }\n\n    #[test]\n    fn test_range_query_with_limit() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let year_field = schema_builder.add_u64_field(\"year\", INDEXED);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            for year in 1950u64..2017u64 {\n                if year == 1963 {\n                    continue;\n                }\n                let num_docs_within_year = 10 + (year - 1950) * (year - 1950);\n                for _ in 0..num_docs_within_year {\n                    index_writer.add_document(doc!(year_field => year))?;\n                }\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        let mut docs_in_the_sixties = InvertedIndexRangeQuery::new(\n            Bound::Included(Term::from_field_u64(year_field, 1960)),\n            Bound::Excluded(Term::from_field_u64(year_field, 1970)),\n        );\n        docs_in_the_sixties.limit(5);\n\n        // due to the limit and no docs in 1963, it's really only 1960..=1965\n        let count = searcher.search(&docs_in_the_sixties, &Count)?;\n        assert_eq!(count, 836);\n        Ok(())\n    }\n\n    #[test]\n    fn test_range_query() -> crate::Result<()> {\n        let int_field: Field;\n        let schema = {\n            let mut schema_builder = Schema::builder();\n            int_field = schema_builder.add_i64_field(\"intfield\", INDEXED);\n            schema_builder.build()\n        };\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 60_000_000)?;\n            index_writer.set_merge_policy(Box::new(NoMergePolicy));\n\n            for i in 1..100 {\n                let mut doc = TantivyDocument::new();\n                for j in 1..100 {\n                    if i % j == 0 {\n                        doc.add_i64(int_field, j as i64);\n                    }\n                }\n                index_writer.add_document(doc)?;\n                if i == 10 {\n                    index_writer.commit()?;\n                }\n            }\n\n            index_writer.commit()?;\n        }\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 2);\n        let count_multiples =\n            |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_i64(int_field, 10)),\n                Bound::Excluded(Term::from_field_i64(int_field, 11)),\n            )),\n            9\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_i64(int_field, 10)),\n                Bound::Included(Term::from_field_i64(int_field, 11)),\n            )),\n            18\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Excluded(Term::from_field_i64(int_field, 9)),\n                Bound::Included(Term::from_field_i64(int_field, 10)),\n            )),\n            9\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_i64(int_field, 9)),\n                Bound::Unbounded\n            )),\n            91\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_range_float() -> crate::Result<()> {\n        let float_field: Field;\n        let schema = {\n            let mut schema_builder = Schema::builder();\n            float_field = schema_builder.add_f64_field(\"floatfield\", INDEXED);\n            schema_builder.build()\n        };\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 60_000_000).unwrap();\n            let mut docs = Vec::new();\n            for i in 1..100 {\n                let mut doc = TantivyDocument::new();\n                for j in 1..100 {\n                    if i % j == 0 {\n                        doc.add_f64(float_field, j as f64);\n                    }\n                }\n                docs.push(doc);\n            }\n\n            docs.shuffle(&mut rand::rng());\n            let mut docs_it = docs.into_iter();\n            for doc in (&mut docs_it).take(50) {\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n            for doc in docs_it {\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 2);\n        let count_multiples =\n            |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_f64(float_field, 10.0)),\n                Bound::Excluded(Term::from_field_f64(float_field, 11.0)),\n            )),\n            9\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_f64(float_field, 10.0)),\n                Bound::Included(Term::from_field_f64(float_field, 11.0)),\n            )),\n            18\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Excluded(Term::from_field_f64(float_field, 9.0)),\n                Bound::Included(Term::from_field_f64(float_field, 10.0)),\n            )),\n            9\n        );\n        assert_eq!(\n            count_multiples(RangeQuery::new(\n                Bound::Included(Term::from_field_f64(float_field, 9.0)),\n                Bound::Unbounded\n            )),\n            91\n        );\n        Ok(())\n    }\n\n    #[test]\n    fn test_bug_reproduce_range_query() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"title\", TEXT);\n        schema_builder.add_i64_field(\"year\", INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer = index.writer_for_tests()?;\n        let title = schema.get_field(\"title\").unwrap();\n        let year = schema.get_field(\"year\").unwrap();\n        index_writer.add_document(doc!(\n          title => \"hemoglobin blood\",\n          year => 1990_i64\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![title]);\n        let query = query_parser.parse_query(\"hemoglobin AND year:[1970 TO 1990]\")?;\n        let top_docs = searcher.search(&query, &TopDocs::with_limit(10).order_by_score())?;\n        assert_eq!(top_docs.len(), 1);\n        Ok(())\n    }\n\n    #[test]\n    fn search_ip_range_test_posting_list() {\n        search_ip_range_test_opt(false);\n    }\n\n    #[test]\n    fn search_ip_range_test() {\n        search_ip_range_test_opt(true);\n    }\n\n    fn search_ip_range_test_opt(with_fast_field: bool) {\n        let mut schema_builder = Schema::builder();\n        let ip_field = if with_fast_field {\n            schema_builder.add_ip_addr_field(\"ip\", INDEXED | STORED | FAST)\n        } else {\n            schema_builder.add_ip_addr_field(\"ip\", INDEXED | STORED)\n        };\n        let text_field = schema_builder.add_text_field(\"text\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let ip_addr_1 = IpAddr::from_str(\"127.0.0.10\").unwrap().into_ipv6_addr();\n        let ip_addr_2 = IpAddr::from_str(\"127.0.0.20\").unwrap().into_ipv6_addr();\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            for _ in 0..1_000 {\n                index_writer\n                    .add_document(doc!(\n                        ip_field => ip_addr_1,\n                        text_field => \"BLUBBER\"\n                    ))\n                    .unwrap();\n            }\n            for _ in 0..1_000 {\n                index_writer\n                    .add_document(doc!(\n                        ip_field => ip_addr_2,\n                        text_field => \"BLOBBER\"\n                    ))\n                    .unwrap();\n            }\n            index_writer.commit().unwrap();\n        }\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n\n        let get_num_hits = |query| {\n            let (_top_docs, count) = searcher\n                .search(&query, &(TopDocs::with_limit(10).order_by_score(), Count))\n                .unwrap();\n            count\n        };\n        let query_from_text = |text: &str| {\n            QueryParser::for_index(&index, vec![])\n                .parse_query(text)\n                .unwrap()\n        };\n\n        // Inclusive range\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:[127.0.0.1 TO 127.0.0.20]\")),\n            2000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:[127.0.0.10 TO 127.0.0.20]\")),\n            2000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:[127.0.0.11 TO 127.0.0.20]\")),\n            1000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:[127.0.0.11 TO 127.0.0.19]\")),\n            0\n        );\n\n        assert_eq!(get_num_hits(query_from_text(\"ip:[127.0.0.11 TO *]\")), 1000);\n        assert_eq!(get_num_hits(query_from_text(\"ip:[127.0.0.21 TO *]\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:[* TO 127.0.0.9]\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:[* TO 127.0.0.10]\")), 1000);\n\n        // Exclusive range\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.1 TO 127.0.0.20}\")),\n            1000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.1 TO 127.0.0.21}\")),\n            2000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.10 TO 127.0.0.20}\")),\n            0\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.11 TO 127.0.0.20}\")),\n            0\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.11 TO 127.0.0.19}\")),\n            0\n        );\n\n        assert_eq!(get_num_hits(query_from_text(\"ip:{127.0.0.11 TO *}\")), 1000);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{127.0.0.10 TO *}\")), 1000);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{127.0.0.21 TO *}\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{127.0.0.20 TO *}\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{127.0.0.19 TO *}\")), 1000);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{* TO 127.0.0.9}\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{* TO 127.0.0.10}\")), 0);\n        assert_eq!(get_num_hits(query_from_text(\"ip:{* TO 127.0.0.11}\")), 1000);\n\n        // Inclusive/Exclusive range\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:[127.0.0.1 TO 127.0.0.20}\")),\n            1000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\"ip:{127.0.0.1 TO 127.0.0.20]\")),\n            2000\n        );\n\n        // Intersection\n        assert_eq!(\n            get_num_hits(query_from_text(\n                \"text:BLUBBER AND ip:[127.0.0.10 TO 127.0.0.10]\"\n            )),\n            1000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\n                \"text:BLOBBER AND ip:[127.0.0.10 TO 127.0.0.10]\"\n            )),\n            0\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\n                \"text:BLOBBER AND ip:[127.0.0.20 TO 127.0.0.20]\"\n            )),\n            1000\n        );\n\n        assert_eq!(\n            get_num_hits(query_from_text(\n                \"text:BLUBBER AND ip:[127.0.0.20 TO 127.0.0.20]\"\n            )),\n            0\n        );\n    }\n\n    #[test]\n    fn test_range_query_simplified() {\n        // This test checks that if the targeted column values are entirely\n        // within the range, and the column is full, we end up with a AllScorer.\n        let mut schema_builder = Schema::builder();\n        let u64_field = schema_builder.add_u64_field(\"u64_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer.add_document(doc!(u64_field=> 2u64)).unwrap();\n        index_writer.add_document(doc!(u64_field=> 4u64)).unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let make_term = |value: u64| Term::from_field_u64(u64_field, value);\n        let make_scorer = move |lower_bound: Bound<u64>, upper_bound: Bound<u64>| {\n            let lower_bound_term = lower_bound.map(make_term);\n            let upper_bound_term = upper_bound.map(make_term);\n            let range_query = RangeQuery::new(lower_bound_term, upper_bound_term);\n            let range_weight = range_query\n                .weight(EnableScoring::disabled_from_schema(&schema))\n                .unwrap();\n            let range_scorer = range_weight\n                .scorer(&searcher.segment_readers()[0], 1.0f32)\n                .unwrap();\n            range_scorer\n        };\n        let range_scorer = make_scorer(Bound::Included(1), Bound::Included(4));\n        assert!(range_scorer.is::<AllScorer>());\n        let range_scorer = make_scorer(Bound::Included(0), Bound::Included(2));\n        assert!(range_scorer.is::<ConstScorer<RangeDocSet<u64>>>());\n        let range_scorer = make_scorer(Bound::Included(3), Bound::Included(10));\n        assert!(range_scorer.is::<ConstScorer<RangeDocSet<u64>>>());\n        let range_scorer = make_scorer(Bound::Included(10), Bound::Included(12));\n        assert!(range_scorer.is::<ConstScorer<RangeDocSet<u64>>>());\n        let range_scorer = make_scorer(Bound::Included(0), Bound::Included(1));\n        assert!(range_scorer.is::<EmptyScorer>());\n        let range_scorer = make_scorer(Bound::Included(0), Bound::Excluded(2));\n        assert!(range_scorer.is::<EmptyScorer>());\n    }\n}\n"
  },
  {
    "path": "src/query/range_query/range_query_fastfield.rs",
    "content": "//! Fastfields support efficient scanning for range queries.\n//! We use this variant only if the fastfield exists, otherwise the default in `range_query` is\n//! used, which uses the term dictionary + postings.\n\nuse std::net::Ipv6Addr;\nuse std::ops::{Bound, RangeInclusive};\n\nuse columnar::{\n    BytesColumn, Cardinality, Column, ColumnType, MonotonicallyMappableToU128,\n    MonotonicallyMappableToU64, NumericalType, StrColumn,\n};\nuse common::bounds::{BoundsRange, TransformBound};\n\nuse super::fast_field_range_doc_set::RangeDocSet;\nuse crate::query::{\n    AllScorer, ConstScorer, EmptyScorer, EnableScoring, Explanation, Query, Scorer, Weight,\n};\nuse crate::schema::{Type, ValueBytes};\nuse crate::{DocId, DocSet, Score, SegmentReader, TantivyError, Term};\n\n#[derive(Clone, Debug)]\n/// `FastFieldRangeQuery` is the same as [RangeQuery] but only uses the fast field\npub struct FastFieldRangeQuery {\n    bounds: BoundsRange<Term>,\n}\nimpl FastFieldRangeQuery {\n    /// Create new `FastFieldRangeQuery`\n    pub fn new(lower_bound: Bound<Term>, upper_bound: Bound<Term>) -> FastFieldRangeQuery {\n        Self {\n            bounds: BoundsRange::new(lower_bound, upper_bound),\n        }\n    }\n}\n\nimpl Query for FastFieldRangeQuery {\n    fn weight(&self, _enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(FastFieldRangeWeight::new(self.bounds.clone())))\n    }\n}\n\n/// `FastFieldRangeWeight` uses the fast field to execute range queries.\n#[derive(Clone, Debug)]\npub struct FastFieldRangeWeight {\n    bounds: BoundsRange<Term>,\n}\n\nimpl FastFieldRangeWeight {\n    /// Create a new FastFieldRangeWeight\n    pub fn new(bounds: BoundsRange<Term>) -> Self {\n        Self { bounds }\n    }\n}\n\nimpl Weight for FastFieldRangeWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        // Check if both bounds are Bound::Unbounded\n        if self.bounds.is_unbounded() {\n            return Ok(Box::new(AllScorer::new(reader.max_doc())));\n        }\n\n        let term = self\n            .bounds\n            .get_inner()\n            .expect(\"At least one bound must be set\");\n        let schema = reader.schema();\n        let field_type = schema.get_field_entry(term.field()).field_type();\n        assert_eq!(\n            term.typ(),\n            field_type.value_type(),\n            \"Field is of type {:?}, but got term of type {:?}\",\n            field_type,\n            term.typ()\n        );\n        let field_name = term.get_full_path(reader.schema());\n\n        let get_value_bytes = |term: &Term| term.value().value_bytes_payload();\n\n        let term_value = term.value();\n        if field_type.is_json() {\n            let bounds = self\n                .bounds\n                .map_bound(|term| term.value().as_json_value_bytes().unwrap().to_owned());\n            // Unlike with other field types JSON may have multiple columns of different types\n            // under the same name\n            //\n            // In the JSON case the provided type in term may not exactly match the column type,\n            // especially with the numeric type interpolation\n            let json_value_bytes = term_value\n                .as_json_value_bytes()\n                .expect(\"expected json type in term\");\n            let typ = json_value_bytes.typ();\n\n            match typ {\n                Type::Str => {\n                    let Some(str_dict_column): Option<StrColumn> =\n                        reader.fast_fields().str(&field_name)?\n                    else {\n                        return Ok(Box::new(EmptyScorer));\n                    };\n                    let dict = str_dict_column.dictionary();\n\n                    let bounds = self.bounds.map_bound(get_value_bytes);\n                    // Get term ids for terms\n                    let (lower_bound, upper_bound) =\n                        dict.term_bounds_to_ord(bounds.lower_bound, bounds.upper_bound)?;\n                    let fast_field_reader = reader.fast_fields();\n                    let Some((column, _col_type)) = fast_field_reader\n                        .u64_lenient_for_type(Some(&[ColumnType::Str]), &field_name)?\n                    else {\n                        return Ok(Box::new(EmptyScorer));\n                    };\n                    search_on_u64_ff(column, boost, BoundsRange::new(lower_bound, upper_bound))\n                }\n                Type::U64 | Type::I64 | Type::F64 => {\n                    search_on_json_numerical_field(reader, &field_name, typ, bounds, boost)\n                }\n                Type::Date => {\n                    let fast_field_reader = reader.fast_fields();\n                    let Some((column, _col_type)) = fast_field_reader\n                        .u64_lenient_for_type(Some(&[ColumnType::DateTime]), &field_name)?\n                    else {\n                        return Ok(Box::new(EmptyScorer));\n                    };\n                    let bounds = bounds.map_bound(|term| term.as_date().unwrap().to_u64());\n                    search_on_u64_ff(\n                        column,\n                        boost,\n                        BoundsRange::new(bounds.lower_bound, bounds.upper_bound),\n                    )\n                }\n                Type::Bool | Type::Facet | Type::Bytes | Type::Json | Type::IpAddr => {\n                    Err(crate::TantivyError::InvalidArgument(format!(\n                        \"unsupported value bytes type in json term value_bytes {:?}\",\n                        term_value.typ()\n                    )))\n                }\n            }\n        } else if field_type.is_ip_addr() {\n            let parse_ip_from_bytes = |term: &Term| {\n                term.value().as_ip_addr().ok_or_else(|| {\n                    crate::TantivyError::InvalidArgument(\"Expected ip address\".to_string())\n                })\n            };\n            let bounds: BoundsRange<Ipv6Addr> = self.bounds.map_bound_res(parse_ip_from_bytes)?;\n\n            let Some(ip_addr_column): Option<Column<Ipv6Addr>> =\n                reader.fast_fields().column_opt(&field_name)?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            let value_range = bound_range_inclusive_ip(\n                &bounds.lower_bound,\n                &bounds.upper_bound,\n                ip_addr_column.min_value(),\n                ip_addr_column.max_value(),\n            );\n            let docset = RangeDocSet::new(value_range, ip_addr_column);\n            Ok(Box::new(ConstScorer::new(docset, boost)))\n        } else if field_type.is_str() {\n            let Some(str_dict_column): Option<StrColumn> = reader.fast_fields().str(&field_name)?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            let dict = str_dict_column.dictionary();\n\n            let bounds = self.bounds.map_bound(get_value_bytes);\n            // Get term ids for terms\n            let (lower_bound, upper_bound) =\n                dict.term_bounds_to_ord(bounds.lower_bound, bounds.upper_bound)?;\n            let fast_field_reader = reader.fast_fields();\n            let Some((column, _col_type)) =\n                fast_field_reader.u64_lenient_for_type(None, &field_name)?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            search_on_u64_ff(column, boost, BoundsRange::new(lower_bound, upper_bound))\n        } else if field_type.is_bytes() {\n            let Some(bytes_column): Option<BytesColumn> =\n                reader.fast_fields().bytes(&field_name)?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            let dict = bytes_column.dictionary();\n\n            let bounds = self.bounds.map_bound(get_value_bytes);\n            // Get term ids for terms\n            let (lower_bound, upper_bound) =\n                dict.term_bounds_to_ord(bounds.lower_bound, bounds.upper_bound)?;\n            let fast_field_reader = reader.fast_fields();\n            let Some((column, _col_type)) =\n                fast_field_reader.u64_lenient_for_type(None, &field_name)?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            search_on_u64_ff(column, boost, BoundsRange::new(lower_bound, upper_bound))\n        } else {\n            assert!(\n                maps_to_u64_fastfield(field_type.value_type()),\n                \"{field_type:?}\"\n            );\n\n            let bounds = self.bounds.map_bound_res(|term| {\n                let value = term.value();\n                let val = if let Some(val) = value.as_u64() {\n                    val\n                } else if let Some(val) = value.as_i64() {\n                    val.to_u64()\n                } else if let Some(val) = value.as_f64() {\n                    val.to_u64()\n                } else if let Some(val) = value.as_date() {\n                    val.to_u64()\n                } else {\n                    return Err(TantivyError::InvalidArgument(format!(\n                        \"Expected term with u64, i64, f64 or date, but got {term:?}\"\n                    )));\n                };\n                Ok(val)\n            })?;\n\n            let fast_field_reader = reader.fast_fields();\n            let Some((column, _col_type)) = fast_field_reader.u64_lenient_for_type(\n                Some(&[\n                    ColumnType::U64,\n                    ColumnType::I64,\n                    ColumnType::F64,\n                    ColumnType::DateTime,\n                ]),\n                &field_name,\n            )?\n            else {\n                return Ok(Box::new(EmptyScorer));\n            };\n            search_on_u64_ff(\n                column,\n                boost,\n                BoundsRange::new(bounds.lower_bound, bounds.upper_bound),\n            )\n        }\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if scorer.seek(doc) != doc {\n            return Err(TantivyError::InvalidArgument(format!(\n                \"Document #({doc}) does not match\"\n            )));\n        }\n        let explanation = Explanation::new(\"Const\", scorer.score());\n\n        Ok(explanation)\n    }\n}\n\n/// On numerical fields the column type may not match the user provided one.\n///\n/// Convert into fast field value space and search.\nfn search_on_json_numerical_field(\n    reader: &SegmentReader,\n    field_name: &str,\n    typ: Type,\n    bounds: BoundsRange<ValueBytes<Vec<u8>>>,\n    boost: Score,\n) -> crate::Result<Box<dyn Scorer>> {\n    // Since we don't know which type was interpolated for the internal column we\n    // have to check for all numeric types (only one exists)\n    let allowed_column_types: Option<&[ColumnType]> =\n        Some(&[ColumnType::F64, ColumnType::I64, ColumnType::U64]);\n    let fast_field_reader = reader.fast_fields();\n    let Some((column, col_type)) =\n        fast_field_reader.u64_lenient_for_type(allowed_column_types, field_name)?\n    else {\n        return Ok(Box::new(EmptyScorer));\n    };\n    let actual_column_type: NumericalType = col_type\n        .numerical_type()\n        .unwrap_or_else(|| panic!(\"internal error: couldn't cast to numerical_type: {col_type:?}\"));\n\n    let bounds = match typ.numerical_type().unwrap() {\n        NumericalType::I64 => {\n            let bounds = bounds.map_bound(|term| term.as_i64().unwrap());\n            match actual_column_type {\n                NumericalType::I64 => bounds.map_bound(|&term| term.to_u64()),\n                NumericalType::U64 => {\n                    bounds.transform_inner(\n                        |&val| {\n                            if val < 0 {\n                                return TransformBound::NewBound(Bound::Unbounded);\n                            }\n                            TransformBound::Existing(val as u64)\n                        },\n                        |&val| {\n                            if val < 0 {\n                                // no hits case\n                                return TransformBound::NewBound(Bound::Excluded(0));\n                            }\n                            TransformBound::Existing(val as u64)\n                        },\n                    )\n                }\n                NumericalType::F64 => bounds.map_bound(|&term| (term as f64).to_u64()),\n            }\n        }\n        NumericalType::U64 => {\n            let bounds = bounds.map_bound(|term| term.as_u64().unwrap());\n            match actual_column_type {\n                NumericalType::U64 => bounds.map_bound(|&term| term.to_u64()),\n                NumericalType::I64 => {\n                    bounds.transform_inner(\n                        |&val| {\n                            if val > i64::MAX as u64 {\n                                // Actual no hits case\n                                return TransformBound::NewBound(Bound::Excluded(i64::MAX as u64));\n                            }\n                            TransformBound::Existing((val as i64).to_u64())\n                        },\n                        |&val| {\n                            if val > i64::MAX as u64 {\n                                return TransformBound::NewBound(Bound::Unbounded);\n                            }\n                            TransformBound::Existing((val as i64).to_u64())\n                        },\n                    )\n                }\n                NumericalType::F64 => bounds.map_bound(|&term| (term as f64).to_u64()),\n            }\n        }\n        NumericalType::F64 => {\n            let bounds = bounds.map_bound(|term| term.as_f64().unwrap());\n            match actual_column_type {\n                NumericalType::U64 => transform_from_f64_bounds::<u64>(&bounds),\n                NumericalType::I64 => transform_from_f64_bounds::<i64>(&bounds),\n                NumericalType::F64 => bounds.map_bound(|&term| term.to_u64()),\n            }\n        }\n    };\n    search_on_u64_ff(\n        column,\n        boost,\n        BoundsRange::new(bounds.lower_bound, bounds.upper_bound),\n    )\n}\n\ntrait IntType {\n    fn min() -> Self;\n    fn max() -> Self;\n    fn to_f64(self) -> f64;\n    fn from_f64(val: f64) -> Self;\n}\nimpl IntType for i64 {\n    fn min() -> Self {\n        Self::MIN\n    }\n    fn max() -> Self {\n        Self::MAX\n    }\n    fn to_f64(self) -> f64 {\n        self as f64\n    }\n    fn from_f64(val: f64) -> Self {\n        val as Self\n    }\n}\nimpl IntType for u64 {\n    fn min() -> Self {\n        Self::MIN\n    }\n    fn max() -> Self {\n        Self::MAX\n    }\n    fn to_f64(self) -> f64 {\n        self as f64\n    }\n    fn from_f64(val: f64) -> Self {\n        val as Self\n    }\n}\n\nfn transform_from_f64_bounds<T: IntType + MonotonicallyMappableToU64>(\n    bounds: &BoundsRange<f64>,\n) -> BoundsRange<u64> {\n    bounds.transform_inner(\n        |&lower_bound| {\n            if lower_bound < T::min().to_f64() {\n                return TransformBound::NewBound(Bound::Unbounded);\n            }\n            if lower_bound > T::max().to_f64() {\n                // no hits case\n                return TransformBound::NewBound(Bound::Excluded(u64::MAX));\n            }\n\n            if lower_bound.fract() == 0.0 {\n                TransformBound::Existing(T::from_f64(lower_bound).to_u64())\n            } else {\n                TransformBound::NewBound(Bound::Included(T::from_f64(lower_bound.trunc()).to_u64()))\n            }\n        },\n        |&upper_bound| {\n            if upper_bound < T::min().to_f64() {\n                return TransformBound::NewBound(Bound::Unbounded);\n            }\n            if upper_bound > T::max().to_f64() {\n                // no hits case\n                return TransformBound::NewBound(Bound::Included(u64::MAX));\n            }\n            if upper_bound.fract() == 0.0 {\n                TransformBound::Existing(T::from_f64(upper_bound).to_u64())\n            } else {\n                TransformBound::NewBound(Bound::Included(T::from_f64(upper_bound.trunc()).to_u64()))\n            }\n        },\n    )\n}\n\nfn search_on_u64_ff(\n    column: Column<u64>,\n    boost: Score,\n    bounds: BoundsRange<u64>,\n) -> crate::Result<Box<dyn Scorer>> {\n    let col_min_value = column.min_value();\n    let col_max_value = column.max_value();\n    #[expect(clippy::reversed_empty_ranges)]\n    let value_range = bound_to_value_range(\n        &bounds.lower_bound,\n        &bounds.upper_bound,\n        column.min_value(),\n        column.max_value(),\n    )\n    .unwrap_or(1..=0); // empty range\n    if value_range.is_empty() {\n        return Ok(Box::new(EmptyScorer));\n    }\n    if col_min_value >= *value_range.start() && col_max_value <= *value_range.end() {\n        // all values in the column are within the range.\n        if column.index.get_cardinality() == Cardinality::Full {\n            if boost != 1.0f32 {\n                return Ok(Box::new(ConstScorer::new(\n                    AllScorer::new(column.num_docs()),\n                    boost,\n                )));\n            } else {\n                return Ok(Box::new(AllScorer::new(column.num_docs())));\n            }\n        } else {\n            // TODO Make it a field presence request for that specific column\n        }\n    }\n\n    let docset = RangeDocSet::new(value_range, column);\n    Ok(Box::new(ConstScorer::new(docset, boost)))\n}\n\n/// Returns true if the type maps to a u64 fast field\npub(crate) fn maps_to_u64_fastfield(typ: Type) -> bool {\n    match typ {\n        Type::U64 | Type::I64 | Type::F64 | Type::Bool | Type::Date => true,\n        Type::IpAddr => false,\n        Type::Str | Type::Facet | Type::Bytes | Type::Json => false,\n    }\n}\n\nfn bound_range_inclusive_ip(\n    lower_bound: &Bound<Ipv6Addr>,\n    upper_bound: &Bound<Ipv6Addr>,\n    min_value: Ipv6Addr,\n    max_value: Ipv6Addr,\n) -> RangeInclusive<Ipv6Addr> {\n    let start_value = match lower_bound {\n        Bound::Included(ip_addr) => *ip_addr,\n        Bound::Excluded(ip_addr) => Ipv6Addr::from(ip_addr.to_u128() + 1),\n        Bound::Unbounded => min_value,\n    };\n\n    let end_value = match upper_bound {\n        Bound::Included(ip_addr) => *ip_addr,\n        Bound::Excluded(ip_addr) => Ipv6Addr::from(ip_addr.to_u128() - 1),\n        Bound::Unbounded => max_value,\n    };\n    start_value..=end_value\n}\n\n// Returns None, if the range cannot be converted to a inclusive range (which equals to a empty\n// range).\nfn bound_to_value_range<T: MonotonicallyMappableToU64>(\n    lower_bound: &Bound<T>,\n    upper_bound: &Bound<T>,\n    min_value: T,\n    max_value: T,\n) -> Option<RangeInclusive<T>> {\n    let mut start_value = match lower_bound {\n        Bound::Included(val) => *val,\n        Bound::Excluded(val) => T::from_u64(val.to_u64().checked_add(1)?),\n        Bound::Unbounded => min_value,\n    };\n    if start_value.partial_cmp(&min_value) == Some(std::cmp::Ordering::Less) {\n        start_value = min_value;\n    }\n    let end_value = match upper_bound {\n        Bound::Included(val) => *val,\n        Bound::Excluded(val) => T::from_u64(val.to_u64().checked_sub(1)?),\n        Bound::Unbounded => max_value,\n    };\n    Some(start_value..=end_value)\n}\n\n#[cfg(test)]\nmod tests {\n    use std::ops::{Bound, RangeInclusive};\n\n    use common::bounds::BoundsRange;\n    use common::DateTime;\n    use proptest::prelude::*;\n    use rand::rngs::StdRng;\n    use rand::seq::IndexedRandom;\n    use rand::SeedableRng;\n    use time::format_description::well_known::Rfc3339;\n    use time::OffsetDateTime;\n\n    use crate::collector::{Count, TopDocs};\n    use crate::fastfield::FastValue;\n    use crate::query::range_query::range_query_fastfield::FastFieldRangeWeight;\n    use crate::query::{QueryParser, RangeQuery, Weight};\n    use crate::schema::{\n        DateOptions, Field, NumericOptions, Schema, SchemaBuilder, FAST, INDEXED, STORED, STRING,\n        TEXT,\n    };\n    use crate::{Index, IndexWriter, TantivyDocument, Term, TERMINATED};\n\n    #[test]\n    fn test_text_field_ff_range_query() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"title\", TEXT | FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer = index.writer_for_tests()?;\n        let title = schema.get_field(\"title\").unwrap();\n        index_writer.add_document(doc!(\n          title => \"bbb\"\n        ))?;\n        index_writer.add_document(doc!(\n          title => \"ddd\"\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![title]);\n\n        let test_query = |query, num_hits| {\n            let query = query_parser.parse_query(query).unwrap();\n            let top_docs = searcher\n                .search(&query, &TopDocs::with_limit(10).order_by_score())\n                .unwrap();\n            assert_eq!(top_docs.len(), num_hits);\n        };\n\n        test_query(\"title:[aaa TO ccc]\", 1);\n        test_query(\"title:[aaa TO bbb]\", 1);\n        test_query(\"title:[bbb TO bbb]\", 1);\n        test_query(\"title:[bbb TO ddd]\", 2);\n        test_query(\"title:[bbb TO eee]\", 2);\n        test_query(\"title:[bb TO eee]\", 2);\n        test_query(\"title:[ccc TO ccc]\", 0);\n        test_query(\"title:[ccc TO ddd]\", 1);\n        test_query(\"title:[ccc TO eee]\", 1);\n\n        test_query(\"title:[aaa TO *}\", 2);\n        test_query(\"title:[bbb TO *]\", 2);\n        test_query(\"title:[bb TO *]\", 2);\n        test_query(\"title:[ccc TO *]\", 1);\n        test_query(\"title:[ddd TO *]\", 1);\n        test_query(\"title:[dddd TO *]\", 0);\n\n        test_query(\"title:{aaa TO *}\", 2);\n        test_query(\"title:{bbb TO *]\", 1);\n        test_query(\"title:{bb TO *]\", 2);\n        test_query(\"title:{ccc TO *]\", 1);\n        test_query(\"title:{ddd TO *]\", 0);\n        test_query(\"title:{dddd TO *]\", 0);\n\n        test_query(\"title:[* TO bb]\", 0);\n        test_query(\"title:[* TO bbb]\", 1);\n        test_query(\"title:[* TO ccc]\", 1);\n        test_query(\"title:[* TO ddd]\", 2);\n        test_query(\"title:[* TO ddd}\", 1);\n        test_query(\"title:[* TO eee]\", 2);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_date_range_query() {\n        let mut schema_builder = Schema::builder();\n        let options = DateOptions::default()\n            .set_precision(common::DateTimePrecision::Microseconds)\n            .set_fast();\n        let date_field = schema_builder.add_date_field(\"date\", options);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema.clone());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n            // This is added a string and creates a string column!\n            index_writer\n                .add_document(doc!(date_field => DateTime::from_utc(\n                    OffsetDateTime::parse(\"2022-12-01T00:00:01Z\", &Rfc3339).unwrap(),\n                )))\n                .unwrap();\n            index_writer\n                .add_document(doc!(date_field => DateTime::from_utc(\n                    OffsetDateTime::parse(\"2023-12-01T00:00:01Z\", &Rfc3339).unwrap(),\n                )))\n                .unwrap();\n            index_writer\n                .add_document(doc!(date_field => DateTime::from_utc(\n                    OffsetDateTime::parse(\"2015-02-01T00:00:00.001Z\", &Rfc3339).unwrap(),\n                )))\n                .unwrap();\n            index_writer.commit().unwrap();\n        }\n\n        // Date field\n        let dt1 =\n            DateTime::from_utc(OffsetDateTime::parse(\"2022-12-01T00:00:01Z\", &Rfc3339).unwrap());\n        let dt2 =\n            DateTime::from_utc(OffsetDateTime::parse(\"2023-12-01T00:00:01Z\", &Rfc3339).unwrap());\n        let dt3 = DateTime::from_utc(\n            OffsetDateTime::parse(\"2015-02-01T00:00:00.001Z\", &Rfc3339).unwrap(),\n        );\n        let dt4 = DateTime::from_utc(\n            OffsetDateTime::parse(\"2015-02-01T00:00:00.002Z\", &Rfc3339).unwrap(),\n        );\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![date_field]);\n        let test_query = |query, num_hits| {\n            let query = query_parser.parse_query(query).unwrap();\n            let top_docs = searcher\n                .search(&query, &TopDocs::with_limit(10).order_by_score())\n                .unwrap();\n            assert_eq!(top_docs.len(), num_hits);\n        };\n\n        test_query(\n            \"date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.001Z]\",\n            1,\n        );\n        test_query(\n            \"date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z}\",\n            1,\n        );\n        test_query(\n            \"date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z]\",\n            1,\n        );\n        test_query(\n            \"date:{2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z]\",\n            0,\n        );\n\n        let count = |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(Term::from_field_date(date_field, dt3)),\n                Bound::Excluded(Term::from_field_date(date_field, dt4)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(Term::from_field_date(date_field, dt3)),\n                Bound::Included(Term::from_field_date(date_field, dt4)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(Term::from_field_date(date_field, dt1)),\n                Bound::Included(Term::from_field_date(date_field, dt2)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(Term::from_field_date(date_field, dt1)),\n                Bound::Excluded(Term::from_field_date(date_field, dt2)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Excluded(Term::from_field_date(date_field, dt1)),\n                Bound::Excluded(Term::from_field_date(date_field, dt2)),\n            )),\n            0\n        );\n    }\n\n    fn get_json_term<T: FastValue>(field: Field, path: &str, value: T) -> Term {\n        let mut term = Term::from_field_json_path(field, path, true);\n        term.append_type_and_fast_value(value);\n        term\n    }\n\n    #[test]\n    fn mixed_numerical_test() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_i64_field(\"id_i64\", STORED | FAST);\n        schema_builder.add_u64_field(\"id_u64\", STORED | FAST);\n        schema_builder.add_f64_field(\"id_f64\", STORED | FAST);\n        let schema = schema_builder.build();\n\n        fn get_json_term<T: FastValue>(schema: &Schema, path: &str, value: T) -> Term {\n            let field = schema.get_field(path).unwrap();\n            Term::from_fast_value(field, &value)\n            // term.append_type_and_fast_value(value);\n            // term\n        }\n        let index = Index::create_in_ram(schema.clone());\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n\n            let doc = json!({\n                \"id_u64\": 0,\n                \"id_i64\": 50,\n            });\n            let doc = TantivyDocument::parse_json(&schema, &serde_json::to_string(&doc).unwrap())\n                .unwrap();\n            index_writer.add_document(doc).unwrap();\n            let doc = json!({\n                \"id_u64\": 10,\n                \"id_i64\": 1000,\n            });\n            let doc = TantivyDocument::parse_json(&schema, &serde_json::to_string(&doc).unwrap())\n                .unwrap();\n            index_writer.add_document(doc).unwrap();\n\n            index_writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let count = |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n\n        // u64 on u64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(&schema, \"id_u64\", 10u64)),\n                Bound::Included(get_json_term(&schema, \"id_u64\", 10u64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(&schema, \"id_u64\", 9u64)),\n                Bound::Excluded(get_json_term(&schema, \"id_u64\", 10u64)),\n            )),\n            0\n        );\n\n        // i64 on i64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(&schema, \"id_i64\", 50i64)),\n                Bound::Included(get_json_term(&schema, \"id_i64\", 1000i64)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(&schema, \"id_i64\", 50i64)),\n                Bound::Excluded(get_json_term(&schema, \"id_i64\", 1000i64)),\n            )),\n            1\n        );\n    }\n\n    #[test]\n    fn json_range_mixed_val() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT | STORED | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n            let doc = json!({\n                \"mixed_val\": 10000,\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            let doc = json!({\n                \"mixed_val\": 20000,\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            let doc = json!({\n                \"mixed_val\": \"1000a\",\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            let doc = json!({\n                \"mixed_val\": \"2000a\",\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            index_writer.commit().unwrap();\n        }\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let count = |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"mixed_val\", 10000u64)),\n                Bound::Included(get_json_term(json_field, \"mixed_val\", 20000u64)),\n            )),\n            2\n        );\n        fn get_json_term_str(field: Field, path: &str, value: &str) -> Term {\n            let mut term = Term::from_field_json_path(field, path, true);\n            term.append_type_and_str(value);\n            term\n        }\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term_str(json_field, \"mixed_val\", \"1000a\")),\n                Bound::Included(get_json_term_str(json_field, \"mixed_val\", \"2000b\")),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term_str(json_field, \"mixed_val\", \"1000\")),\n                Bound::Included(get_json_term_str(json_field, \"mixed_val\", \"2000a\")),\n            )),\n            2\n        );\n    }\n\n    #[test]\n    fn json_range_test() {\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT | STORED | FAST);\n        let schema = schema_builder.build();\n\n        let index = Index::create_in_ram(schema);\n        let u64_val = u64::MAX - 1;\n        {\n            let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n            let doc = json!({\n                \"id_u64\": 0,\n                \"id_f64\": 10.5,\n                \"id_i64\": -100,\n                \"date\": \"2022-12-01T00:00:01Z\"\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            let doc = json!({\n                \"id_u64\": u64_val,\n                \"id_f64\": 1000.5,\n                \"id_i64\": 1000,\n                \"date\": \"2023-12-01T00:00:01Z\"\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n            let doc = json!({\n                \"date\": \"2015-02-01T00:00:00.001Z\"\n            });\n            index_writer.add_document(doc!(json_field => doc)).unwrap();\n\n            index_writer.commit().unwrap();\n        }\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let count = |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();\n\n        // u64 on u64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_u64\", u64_val)),\n                Bound::Included(get_json_term(json_field, \"id_u64\", u64_val)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_u64\", u64_val)),\n                Bound::Excluded(get_json_term(json_field, \"id_u64\", u64_val)),\n            )),\n            0\n        );\n        // f64 on u64 field\n        assert_eq!(\n            count(RangeQuery::new(\n                // We need to subtract since there is some inaccuracy\n                Bound::Included(get_json_term(\n                    json_field,\n                    \"id_u64\",\n                    (u64_val - 10000) as f64\n                )),\n                Bound::Included(get_json_term(json_field, \"id_u64\", (u64_val) as f64)),\n            )),\n            1\n        );\n        // i64 on u64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_u64\", 0_i64)),\n                Bound::Included(get_json_term(json_field, \"id_u64\", 0_i64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_u64\", 1_i64)),\n                Bound::Included(get_json_term(json_field, \"id_u64\", 1_i64)),\n            )),\n            0\n        );\n        // u64 on f64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_f64\", 10_u64)),\n                Bound::Included(get_json_term(json_field, \"id_f64\", 11_u64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_f64\", 10_u64)),\n                Bound::Included(get_json_term(json_field, \"id_f64\", 2000_u64)),\n            )),\n            2\n        );\n        // i64 on f64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_f64\", 10_i64)),\n                Bound::Included(get_json_term(json_field, \"id_f64\", 2000_i64)),\n            )),\n            2\n        );\n\n        // i64 on i64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000i64)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 1000i64)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", 1000i64)),\n                Bound::Excluded(get_json_term(json_field, \"id_i64\", 1001i64)),\n            )),\n            1\n        );\n\n        // u64 on i64\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", 0_u64)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 1000u64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", 0_u64)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 999u64)),\n            )),\n            0\n        );\n        // f64 on i64 field\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000.0)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 1000.0)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000.0f64)),\n                Bound::Excluded(get_json_term(json_field, \"id_i64\", 1000.0f64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000.0f64)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 1000.0f64)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000.0f64)),\n                Bound::Excluded(get_json_term(json_field, \"id_i64\", 1000.01f64)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"id_i64\", -1000.0f64)),\n                Bound::Included(get_json_term(json_field, \"id_i64\", 999.99f64)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Excluded(get_json_term(json_field, \"id_i64\", 999.9)),\n                Bound::Excluded(get_json_term(json_field, \"id_i64\", 1000.1)),\n            )),\n            1\n        );\n\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![json_field]);\n        let test_query = |query, num_hits| {\n            let query = query_parser.parse_query(query).unwrap();\n            let top_docs = searcher\n                .search(&query, &TopDocs::with_limit(10).order_by_score())\n                .unwrap();\n            assert_eq!(top_docs.len(), num_hits);\n        };\n\n        test_query(\n            \"json.date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.001Z]\",\n            1,\n        );\n        test_query(\n            \"json.date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z}\",\n            1,\n        );\n        test_query(\n            \"json.date:[2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z]\",\n            1,\n        );\n        test_query(\n            \"json.date:{2015-02-01T00:00:00.001Z TO 2015-02-01T00:00:00.002Z]\",\n            0,\n        );\n\n        // Date field\n        let dt1 =\n            DateTime::from_utc(OffsetDateTime::parse(\"2022-12-01T00:00:01Z\", &Rfc3339).unwrap());\n        let dt2 =\n            DateTime::from_utc(OffsetDateTime::parse(\"2023-12-01T00:00:01Z\", &Rfc3339).unwrap());\n\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"date\", dt1)),\n                Bound::Included(get_json_term(json_field, \"date\", dt2)),\n            )),\n            2\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Included(get_json_term(json_field, \"date\", dt1)),\n                Bound::Excluded(get_json_term(json_field, \"date\", dt2)),\n            )),\n            1\n        );\n        assert_eq!(\n            count(RangeQuery::new(\n                Bound::Excluded(get_json_term(json_field, \"date\", dt1)),\n                Bound::Excluded(get_json_term(json_field, \"date\", dt2)),\n            )),\n            0\n        );\n        // Date precision test. We don't want to truncate the precision\n        let dt3 = DateTime::from_utc(\n            OffsetDateTime::parse(\"2015-02-01T00:00:00.001Z\", &Rfc3339).unwrap(),\n        );\n        let dt4 = DateTime::from_utc(\n            OffsetDateTime::parse(\"2015-02-01T00:00:00.002Z\", &Rfc3339).unwrap(),\n        );\n        let query = RangeQuery::new(\n            Bound::Included(get_json_term(json_field, \"date\", dt3)),\n            Bound::Excluded(get_json_term(json_field, \"date\", dt4)),\n        );\n        assert_eq!(count(query), 1);\n    }\n\n    #[derive(Clone, Debug)]\n    pub struct Doc {\n        pub id_name: String,\n        pub id: u64,\n    }\n\n    fn operation_strategy() -> impl Strategy<Value = Doc> {\n        prop_oneof![\n            (0u64..10_000u64).prop_map(doc_from_id_1),\n            (1u64..10_000u64).prop_map(doc_from_id_2),\n        ]\n    }\n\n    fn doc_from_id_1(id: u64) -> Doc {\n        let id = id * 1000;\n        Doc {\n            id_name: format!(\"id_name{id:010}\"),\n            id,\n        }\n    }\n    fn doc_from_id_2(id: u64) -> Doc {\n        let id = id * 1000;\n        Doc {\n            id_name: format!(\"id_name{:010}\", id - 1),\n            id,\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(10))]\n        #[test]\n        fn test_range_for_docs_prop(ops in proptest::collection::vec(operation_strategy(), 1..1000)) {\n            assert!(test_id_range_for_docs(ops).is_ok());\n        }\n    }\n\n    #[test]\n    fn range_regression1_test() {\n        let ops = vec![doc_from_id_1(0)];\n        assert!(test_id_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn range_regression1_test_json() {\n        let ops = vec![doc_from_id_1(0)];\n        assert!(test_id_range_for_docs_json(ops).is_ok());\n    }\n\n    #[test]\n    fn test_range_regression2() {\n        let ops = vec![\n            doc_from_id_1(52),\n            doc_from_id_1(63),\n            doc_from_id_1(12),\n            doc_from_id_2(91),\n            doc_from_id_2(33),\n        ];\n        assert!(test_id_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn test_range_regression3() {\n        let ops = vec![doc_from_id_1(9), doc_from_id_1(0), doc_from_id_1(13)];\n        assert!(test_id_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn test_range_regression_simplified() {\n        let mut schema_builder = SchemaBuilder::new();\n        let field = schema_builder.add_u64_field(\"test_field\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n        writer.add_document(doc!(field=>52_000u64)).unwrap();\n        writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let range_query = FastFieldRangeWeight::new(BoundsRange::new(\n            Bound::Included(Term::from_field_u64(field, 50_000)),\n            Bound::Included(Term::from_field_u64(field, 50_002)),\n        ));\n        let scorer = range_query\n            .scorer(searcher.segment_reader(0), 1.0f32)\n            .unwrap();\n        assert_eq!(scorer.doc(), TERMINATED);\n    }\n\n    #[test]\n    fn range_regression3_test() {\n        let ops = vec![doc_from_id_1(1), doc_from_id_1(2), doc_from_id_1(3)];\n        assert!(test_id_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn range_regression4_test() {\n        let ops = vec![doc_from_id_2(100)];\n        assert!(test_id_range_for_docs(ops).is_ok());\n    }\n\n    pub fn create_index_from_docs(docs: &[Doc], json_field: bool) -> Index {\n        let mut schema_builder = Schema::builder();\n        if json_field {\n            let json_field = schema_builder.add_json_field(\"json\", TEXT | STORED | FAST);\n            let schema = schema_builder.build();\n\n            let index = Index::create_in_ram(schema);\n\n            {\n                let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n                for doc in docs.iter() {\n                    let doc = json!({\n                        \"ids_i64\": doc.id as i64,\n                        \"ids_i64\": doc.id as i64,\n                        \"ids_f64\": doc.id as f64,\n                        \"ids_f64\": doc.id as f64,\n                        \"ids\": doc.id,\n                        \"ids\": doc.id,\n                        \"id\": doc.id,\n                        \"id_f64\": doc.id as f64,\n                        \"id_i64\": doc.id as i64,\n                        \"id_name\": doc.id_name.to_string(),\n                        \"id_name_fast\": doc.id_name.to_string(),\n                    });\n                    index_writer.add_document(doc!(json_field => doc)).unwrap();\n                }\n\n                index_writer.commit().unwrap();\n            }\n            index\n        } else {\n            let id_u64_field = schema_builder.add_u64_field(\"id\", INDEXED | STORED | FAST);\n            let ids_u64_field = schema_builder\n                .add_u64_field(\"ids\", NumericOptions::default().set_fast().set_indexed());\n\n            let id_f64_field = schema_builder.add_f64_field(\"id_f64\", INDEXED | STORED | FAST);\n            let ids_f64_field = schema_builder.add_f64_field(\n                \"ids_f64\",\n                NumericOptions::default().set_fast().set_indexed(),\n            );\n\n            let id_i64_field = schema_builder.add_i64_field(\"id_i64\", INDEXED | STORED | FAST);\n            let ids_i64_field = schema_builder.add_i64_field(\n                \"ids_i64\",\n                NumericOptions::default().set_fast().set_indexed(),\n            );\n\n            let text_field = schema_builder.add_text_field(\"id_name\", STRING | STORED);\n            let text_field2 = schema_builder.add_text_field(\"id_name_fast\", STRING | STORED | FAST);\n            let schema = schema_builder.build();\n\n            let index = Index::create_in_ram(schema);\n\n            {\n                let mut index_writer = index.writer_with_num_threads(1, 50_000_000).unwrap();\n                for doc in docs.iter() {\n                    index_writer\n                        .add_document(doc!(\n                            ids_i64_field => doc.id as i64,\n                            ids_i64_field => doc.id as i64,\n                            ids_f64_field => doc.id as f64,\n                            ids_f64_field => doc.id as f64,\n                            ids_u64_field => doc.id,\n                            ids_u64_field => doc.id,\n                            id_u64_field => doc.id,\n                            id_f64_field => doc.id as f64,\n                            id_i64_field => doc.id as i64,\n                            text_field => doc.id_name.to_string(),\n                            text_field2 => doc.id_name.to_string(),\n                        ))\n                        .unwrap();\n                }\n\n                index_writer.commit().unwrap();\n            }\n            index\n        }\n    }\n\n    fn test_id_range_for_docs(docs: Vec<Doc>) -> crate::Result<()> {\n        test_id_range_for_docs_with_opt(docs, false)\n    }\n    fn test_id_range_for_docs_json(docs: Vec<Doc>) -> crate::Result<()> {\n        test_id_range_for_docs_with_opt(docs, true)\n    }\n\n    fn test_id_range_for_docs_with_opt(docs: Vec<Doc>, json: bool) -> crate::Result<()> {\n        let index = create_index_from_docs(&docs, json);\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let mut rng: StdRng = StdRng::from_seed([1u8; 32]);\n\n        let get_num_hits = |query| searcher.search(&query, &Count).unwrap();\n        let query_from_text = |text: &str| {\n            QueryParser::for_index(&index, vec![])\n                .parse_query(text)\n                .unwrap()\n        };\n\n        let field_path = |field: &str| {\n            if json {\n                format!(\"json.{field}\")\n            } else {\n                field.to_string()\n            }\n        };\n\n        let gen_query_inclusive = |field: &str, range: RangeInclusive<u64>| {\n            format!(\n                \"{}:[{} TO {}]\",\n                field_path(field),\n                range.start(),\n                range.end()\n            )\n        };\n        let gen_query_exclusive = |field: &str, range: RangeInclusive<u64>| {\n            format!(\n                \"{}:{{{} TO {}}}\",\n                field_path(field),\n                range.start(),\n                range.end()\n            )\n        };\n\n        let test_sample = |sample_docs: Vec<Doc>| {\n            let mut ids: Vec<u64> = sample_docs.iter().map(|doc| doc.id).collect();\n            ids.sort();\n            let expected_num_hits = docs\n                .iter()\n                .filter(|doc| (ids[0]..=ids[1]).contains(&doc.id))\n                .count();\n\n            let query = gen_query_inclusive(\"id\", ids[0]..=ids[1]);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            let query = gen_query_inclusive(\"ids\", ids[0]..=ids[1]);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            // Text query\n            {\n                let test_text_query = |field_name: &str| {\n                    let mut id_names: Vec<&str> =\n                        sample_docs.iter().map(|doc| doc.id_name.as_str()).collect();\n                    id_names.sort();\n                    let expected_num_hits = docs\n                        .iter()\n                        .filter(|doc| (id_names[0]..=id_names[1]).contains(&doc.id_name.as_str()))\n                        .count();\n                    let query = format!(\n                        \"{}:[{} TO {}]\",\n                        field_path(field_name),\n                        id_names[0],\n                        id_names[1]\n                    );\n                    assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n                };\n\n                test_text_query(\"id_name\");\n                test_text_query(\"id_name_fast\");\n            }\n\n            // Exclusive range\n            let expected_num_hits = docs\n                .iter()\n                .filter(|doc| {\n                    (ids[0].saturating_add(1)..=ids[1].saturating_sub(1)).contains(&doc.id)\n                })\n                .count();\n\n            let query = gen_query_exclusive(\"id\", ids[0]..=ids[1]);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            let query = gen_query_exclusive(\"ids\", ids[0]..=ids[1]);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            // Intersection search\n            let id_filter = sample_docs[0].id_name.to_string();\n            let expected_num_hits = docs\n                .iter()\n                .filter(|doc| (ids[0]..=ids[1]).contains(&doc.id) && doc.id_name == id_filter)\n                .count();\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"id\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"id_f64\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"id_i64\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            // Intersection search on multivalue id field\n            let id_filter = sample_docs[0].id_name.to_string();\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"ids\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"ids_f64\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n            let query = format!(\n                \"{} AND {}:{}\",\n                gen_query_inclusive(\"ids_i64\", ids[0]..=ids[1]),\n                field_path(\"id_name\"),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n        };\n\n        test_sample(vec![docs[0].clone(), docs[0].clone()]);\n\n        let samples: Vec<_> = docs.choose_multiple(&mut rng, 3).collect();\n\n        if samples.len() > 1 {\n            test_sample(vec![samples[0].clone(), samples[1].clone()]);\n            test_sample(vec![samples[1].clone(), samples[1].clone()]);\n        }\n        if samples.len() > 2 {\n            test_sample(vec![samples[1].clone(), samples[2].clone()]);\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_bytes_field_ff_range_query() -> crate::Result<()> {\n        use crate::schema::BytesOptions;\n\n        let mut schema_builder = Schema::builder();\n        let bytes_field = schema_builder\n            .add_bytes_field(\"data\", BytesOptions::default().set_fast().set_indexed());\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n\n        // Insert documents with lexicographically sortable byte values\n        // Using simple byte sequences that have clear ordering\n        let values: Vec<Vec<u8>> = vec![\n            vec![0x00, 0x10],\n            vec![0x00, 0x20],\n            vec![0x00, 0x30],\n            vec![0x01, 0x00],\n            vec![0x01, 0x10],\n            vec![0x02, 0x00],\n        ];\n\n        for value in &values {\n            let mut doc = TantivyDocument::new();\n            doc.add_bytes(bytes_field, value);\n            index_writer.add_document(doc)?;\n        }\n        index_writer.commit()?;\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // Test: Range query [0x00, 0x20] to [0x01, 0x00] (inclusive)\n        // Should match: [0x00, 0x20], [0x00, 0x30], [0x01, 0x00]\n        let lower = Term::from_field_bytes(bytes_field, &[0x00, 0x20]);\n        let upper = Term::from_field_bytes(bytes_field, &[0x01, 0x00]);\n        let range_query = RangeQuery::new(Bound::Included(lower), Bound::Included(upper));\n        let count = searcher.search(&range_query, &Count)?;\n        assert_eq!(\n            count, 3,\n            \"Expected 3 documents in range [0x00,0x20] to [0x01,0x00]\"\n        );\n\n        // Test: Range query > [0x01, 0x00] (exclusive lower bound)\n        // Should match: [0x01, 0x10], [0x02, 0x00]\n        let lower = Term::from_field_bytes(bytes_field, &[0x01, 0x00]);\n        let range_query = RangeQuery::new(Bound::Excluded(lower), Bound::Unbounded);\n        let count = searcher.search(&range_query, &Count)?;\n        assert_eq!(count, 2, \"Expected 2 documents > [0x01,0x00]\");\n\n        // Test: Range query < [0x00, 0x30] (exclusive upper bound)\n        // Should match: [0x00, 0x10], [0x00, 0x20]\n        let upper = Term::from_field_bytes(bytes_field, &[0x00, 0x30]);\n        let range_query = RangeQuery::new(Bound::Unbounded, Bound::Excluded(upper));\n        let count = searcher.search(&range_query, &Count)?;\n        assert_eq!(count, 2, \"Expected 2 documents < [0x00,0x30]\");\n\n        Ok(())\n    }\n}\n\n#[cfg(test)]\npub(crate) mod ip_range_tests {\n    use proptest::prelude::ProptestConfig;\n    use proptest::strategy::Strategy;\n    use proptest::{prop_oneof, proptest};\n\n    use super::*;\n    use crate::collector::Count;\n    use crate::query::QueryParser;\n    use crate::schema::{Schema, FAST, INDEXED, STORED, STRING};\n    use crate::{Index, IndexWriter};\n\n    #[derive(Clone, Debug)]\n    pub struct Doc {\n        pub id: String,\n        pub ip: Ipv6Addr,\n    }\n\n    fn operation_strategy() -> impl Strategy<Value = Doc> {\n        prop_oneof![\n            (0u64..10_000u64).prop_map(doc_from_id_1),\n            (1u64..10_000u64).prop_map(doc_from_id_2),\n        ]\n    }\n\n    pub fn doc_from_id_1(id: u64) -> Doc {\n        let id = id * 1000;\n        Doc {\n            // ip != id\n            id: id.to_string(),\n            ip: Ipv6Addr::from_u128(id as u128),\n        }\n    }\n    fn doc_from_id_2(id: u64) -> Doc {\n        let id = id * 1000;\n        Doc {\n            // ip != id\n            id: (id - 1).to_string(),\n            ip: Ipv6Addr::from_u128(id as u128),\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(10))]\n        #[test]\n        fn test_ip_range_for_docs_prop(ops in proptest::collection::vec(operation_strategy(), 1..1000)) {\n            assert!(test_ip_range_for_docs(&ops).is_ok());\n        }\n    }\n\n    #[test]\n    fn test_ip_range_regression1() {\n        let ops = &[doc_from_id_1(0)];\n        assert!(test_ip_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn test_ip_range_regression2() {\n        let ops = &[\n            doc_from_id_1(52),\n            doc_from_id_1(63),\n            doc_from_id_1(12),\n            doc_from_id_2(91),\n            doc_from_id_2(33),\n        ];\n        assert!(test_ip_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn test_ip_range_regression3() {\n        let ops = &[doc_from_id_1(1), doc_from_id_1(2), doc_from_id_1(3)];\n        assert!(test_ip_range_for_docs(ops).is_ok());\n    }\n\n    #[test]\n    fn test_ip_range_regression3_simple() {\n        let mut schema_builder = Schema::builder();\n        let ips_field = schema_builder.add_ip_addr_field(\"ips\", FAST | INDEXED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter = index.writer_for_tests().unwrap();\n        let ip_addrs: Vec<Ipv6Addr> = [1000, 2000, 3000]\n            .into_iter()\n            .map(Ipv6Addr::from_u128)\n            .collect();\n        for &ip_addr in &ip_addrs {\n            writer\n                .add_document(doc!(ips_field=>ip_addr, ips_field=>ip_addr))\n                .unwrap();\n        }\n        writer.commit().unwrap();\n        let searcher = index.reader().unwrap().searcher();\n        let range_weight = FastFieldRangeWeight::new(BoundsRange::new(\n            Bound::Included(Term::from_field_ip_addr(ips_field, ip_addrs[1])),\n            Bound::Included(Term::from_field_ip_addr(ips_field, ip_addrs[2])),\n        ));\n\n        let count =\n            crate::query::weight::Weight::count(&range_weight, searcher.segment_reader(0)).unwrap();\n        assert_eq!(count, 2);\n    }\n\n    pub fn create_index_from_ip_docs(docs: &[Doc]) -> Index {\n        let mut schema_builder = Schema::builder();\n        let ip_field = schema_builder.add_ip_addr_field(\"ip\", STORED | FAST);\n        let ips_field = schema_builder.add_ip_addr_field(\"ips\", FAST | INDEXED);\n        let text_field = schema_builder.add_text_field(\"id\", STRING | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_with_num_threads(2, 60_000_000).unwrap();\n            for doc in docs.iter() {\n                index_writer\n                    .add_document(doc!(\n                        ips_field => doc.ip,\n                        ips_field => doc.ip,\n                        ip_field => doc.ip,\n                        text_field => doc.id.to_string(),\n                    ))\n                    .unwrap();\n            }\n\n            index_writer.commit().unwrap();\n        }\n        index\n    }\n\n    fn test_ip_range_for_docs(docs: &[Doc]) -> crate::Result<()> {\n        let index = create_index_from_ip_docs(docs);\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let get_num_hits = |query| searcher.search(&query, &Count).unwrap();\n        let query_from_text = |text: &str| {\n            QueryParser::for_index(&index, vec![])\n                .parse_query(text)\n                .unwrap()\n        };\n\n        let gen_query_inclusive = |field: &str, ip_range: &RangeInclusive<Ipv6Addr>| {\n            format!(\"{field}:[{} TO {}]\", ip_range.start(), ip_range.end())\n        };\n\n        let test_sample = |sample_docs: &[Doc]| {\n            let mut ips: Vec<Ipv6Addr> = sample_docs.iter().map(|doc| doc.ip).collect();\n            ips.sort();\n            let ip_range = ips[0]..=ips[1];\n            let expected_num_hits = docs\n                .iter()\n                .filter(|doc| (ips[0]..=ips[1]).contains(&doc.ip))\n                .count();\n\n            let query = gen_query_inclusive(\"ip\", &ip_range);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            let query = gen_query_inclusive(\"ips\", &ip_range);\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            // Intersection search\n            let id_filter = sample_docs[0].id.to_string();\n            let expected_num_hits = docs\n                .iter()\n                .filter(|doc| ip_range.contains(&doc.ip) && doc.id == id_filter)\n                .count();\n            let query = format!(\n                \"{} AND id:{}\",\n                gen_query_inclusive(\"ip\", &ip_range),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n\n            // Intersection search on multivalue ip field\n            let id_filter = sample_docs[0].id.to_string();\n            let query = format!(\n                \"{} AND id:{}\",\n                gen_query_inclusive(\"ips\", &ip_range),\n                &id_filter\n            );\n            assert_eq!(get_num_hits(query_from_text(&query)), expected_num_hits);\n        };\n\n        test_sample(&[docs[0].clone(), docs[0].clone()]);\n        if docs.len() > 1 {\n            test_sample(&[docs[0].clone(), docs[1].clone()]);\n            test_sample(&[docs[1].clone(), docs[1].clone()]);\n        }\n        if docs.len() > 2 {\n            test_sample(&[docs[1].clone(), docs[2].clone()]);\n        }\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/regex_query.rs",
    "content": "use std::clone::Clone;\nuse std::sync::Arc;\n\nuse tantivy_fst::Regex;\n\nuse crate::error::TantivyError;\nuse crate::query::{AutomatonWeight, EnableScoring, Query, Weight};\nuse crate::schema::Field;\n\n/// A Regex Query matches all of the documents\n/// containing a specific term that matches\n/// a regex pattern.\n///\n/// Wildcard queries (e.g. ho*se) can be achieved\n/// by converting them to their regex counterparts.\n///\n/// ```rust\n/// use tantivy::collector::Count;\n/// use tantivy::query::RegexQuery;\n/// use tantivy::schema::{Schema, TEXT};\n/// use tantivy::{doc, Index, IndexWriter, Term};\n///\n/// # fn test() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n/// {\n///     let mut index_writer: IndexWriter = index.writer(15_000_000)?;\n///     index_writer.add_document(doc!(\n///         title => \"The Name of the Wind\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"The Diary of Muadib\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"A Dairy Cow\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"The Diary of a Young Girl\",\n///     ))?;\n///     index_writer.commit()?;\n/// }\n///\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n///\n/// let term = Term::from_field_text(title, \"Diary\");\n/// let query = RegexQuery::from_pattern(\"d[ai]{2}ry\", title)?;\n/// let count = searcher.search(&query, &Count)?;\n/// assert_eq!(count, 3);\n/// Ok(())\n/// # }\n/// # assert!(test().is_ok());\n/// ```\n#[derive(Debug, Clone)]\npub struct RegexQuery {\n    regex: Arc<Regex>,\n    field: Field,\n}\n\nimpl RegexQuery {\n    /// Creates a new RegexQuery from a given pattern\n    pub fn from_pattern(regex_pattern: &str, field: Field) -> crate::Result<Self> {\n        let regex = Regex::new(regex_pattern)\n            .map_err(|err| TantivyError::InvalidArgument(format!(\"RegexQueryError: {err}\")))?;\n        Ok(RegexQuery::from_regex(regex, field))\n    }\n\n    /// Creates a new RegexQuery from a fully built Regex\n    pub fn from_regex<T: Into<Arc<Regex>>>(regex: T, field: Field) -> Self {\n        RegexQuery {\n            regex: regex.into(),\n            field,\n        }\n    }\n\n    fn specialized_weight(&self) -> AutomatonWeight<Regex> {\n        AutomatonWeight::new(self.field, self.regex.clone())\n    }\n}\n\nimpl Query for RegexQuery {\n    fn weight(&self, _enabled_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(self.specialized_weight()))\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use std::sync::Arc;\n\n    use tantivy_fst::Regex;\n\n    use super::RegexQuery;\n    use crate::collector::TopDocs;\n    use crate::schema::{Field, Schema, TEXT};\n    use crate::{assert_nearly_equals, Index, IndexReader, IndexWriter};\n\n    fn build_test_index() -> crate::Result<(IndexReader, Field)> {\n        let mut schema_builder = Schema::builder();\n        let country_field = schema_builder.add_text_field(\"country\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer.add_document(doc!(\n                country_field => \"japan\",\n            ))?;\n            index_writer.add_document(doc!(\n                country_field => \"korea\",\n            ))?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n\n        Ok((reader, country_field))\n    }\n\n    fn verify_regex_query(\n        query_matching_one: RegexQuery,\n        query_matching_zero: RegexQuery,\n        reader: IndexReader,\n    ) {\n        let searcher = reader.searcher();\n        {\n            let scored_docs = searcher\n                .search(\n                    &query_matching_one,\n                    &TopDocs::with_limit(2).order_by_score(),\n                )\n                .unwrap();\n            assert_eq!(scored_docs.len(), 1, \"Expected only 1 document\");\n            let (score, _) = scored_docs[0];\n            assert_nearly_equals!(1.0, score);\n        }\n        let top_docs = searcher\n            .search(\n                &query_matching_zero,\n                &TopDocs::with_limit(2).order_by_score(),\n            )\n            .unwrap();\n        assert!(top_docs.is_empty(), \"Expected ZERO document\");\n    }\n\n    #[test]\n    pub fn test_regex_query() -> crate::Result<()> {\n        let (reader, field) = build_test_index()?;\n\n        let matching_one = RegexQuery::from_pattern(\"jap[ao]n\", field)?;\n        let matching_zero = RegexQuery::from_pattern(\"jap[A-Z]n\", field)?;\n        verify_regex_query(matching_one, matching_zero, reader);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_construct_from_regex() -> crate::Result<()> {\n        let (reader, field) = build_test_index()?;\n\n        let matching_one = RegexQuery::from_regex(Regex::new(\"jap[ao]n\").unwrap(), field);\n        let matching_zero = RegexQuery::from_regex(Regex::new(\"jap[A-Z]n\").unwrap(), field);\n\n        verify_regex_query(matching_one, matching_zero, reader);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_construct_from_reused_regex() -> crate::Result<()> {\n        let r1 = Arc::new(Regex::new(\"jap[ao]n\").unwrap());\n        let r2 = Arc::new(Regex::new(\"jap[A-Z]n\").unwrap());\n\n        let (reader, field) = build_test_index()?;\n\n        let matching_one = RegexQuery::from_regex(r1.clone(), field);\n        let matching_zero = RegexQuery::from_regex(r2.clone(), field);\n\n        verify_regex_query(matching_one, matching_zero, reader.clone());\n\n        let matching_one = RegexQuery::from_regex(r1, field);\n        let matching_zero = RegexQuery::from_regex(r2, field);\n\n        verify_regex_query(matching_one, matching_zero, reader);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_pattern_error() {\n        let (_reader, field) = build_test_index().unwrap();\n\n        match RegexQuery::from_pattern(r\"(foo\", field) {\n            Err(crate::TantivyError::InvalidArgument(msg)) => {\n                assert!(msg.contains(\"error: unclosed group\"))\n            }\n            res => panic!(\"unexpected result: {res:?}\"),\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/reqopt_scorer.rs",
    "content": "use std::marker::PhantomData;\n\nuse crate::docset::{DocSet, SeekDangerResult};\nuse crate::query::score_combiner::ScoreCombiner;\nuse crate::query::Scorer;\nuse crate::{DocId, Score};\n\n/// Given a required scorer and an optional scorer\n/// matches all document from the required scorer\n/// and complements the score using the optional scorer.\n///\n/// This is useful for queries like `+somethingrequired somethingoptional`.\n///\n/// Note that `somethingoptional` has no impact on the `DocSet`.\npub struct RequiredOptionalScorer<TReqScorer, TOptScorer, TScoreCombiner: ScoreCombiner> {\n    req_scorer: TReqScorer,\n    opt_scorer: TOptScorer,\n    score_cache: Option<Score>,\n    _phantom: PhantomData<TScoreCombiner>,\n}\n\nimpl<TReqScorer, TOptScorer, TScoreCombiner>\n    RequiredOptionalScorer<TReqScorer, TOptScorer, TScoreCombiner>\nwhere\n    TOptScorer: DocSet,\n    TScoreCombiner: ScoreCombiner,\n{\n    /// Creates a new `RequiredOptionalScorer`.\n    pub fn new(\n        req_scorer: TReqScorer,\n        opt_scorer: TOptScorer,\n    ) -> RequiredOptionalScorer<TReqScorer, TOptScorer, TScoreCombiner> {\n        RequiredOptionalScorer {\n            req_scorer,\n            opt_scorer,\n            score_cache: None,\n            _phantom: PhantomData,\n        }\n    }\n}\n\nimpl<TReqScorer, TOptScorer, TScoreCombiner> DocSet\n    for RequiredOptionalScorer<TReqScorer, TOptScorer, TScoreCombiner>\nwhere\n    TReqScorer: DocSet,\n    TOptScorer: DocSet,\n    TScoreCombiner: ScoreCombiner,\n{\n    fn advance(&mut self) -> DocId {\n        self.score_cache = None;\n        self.req_scorer.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.score_cache = None;\n        self.req_scorer.seek(target)\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        self.score_cache = None;\n        self.req_scorer.seek_danger(target)\n    }\n\n    fn doc(&self) -> DocId {\n        self.req_scorer.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.req_scorer.size_hint()\n    }\n\n    fn cost(&self) -> u64 {\n        self.req_scorer.cost()\n    }\n}\n\nimpl<TReqScorer, TOptScorer, TScoreCombiner> Scorer\n    for RequiredOptionalScorer<TReqScorer, TOptScorer, TScoreCombiner>\nwhere\n    TReqScorer: Scorer,\n    TOptScorer: Scorer,\n    TScoreCombiner: ScoreCombiner,\n{\n    #[inline]\n    fn score(&mut self) -> Score {\n        if let Some(score) = self.score_cache {\n            return score;\n        }\n        let doc = self.doc();\n        let mut score_combiner = TScoreCombiner::default();\n        score_combiner.update(&mut self.req_scorer);\n        if self.opt_scorer.doc() <= doc && self.opt_scorer.seek(doc) == doc {\n            score_combiner.update(&mut self.opt_scorer);\n        }\n        let score = score_combiner.score();\n        self.score_cache = Some(score);\n        score\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::RequiredOptionalScorer;\n    use crate::docset::{DocSet, TERMINATED};\n    use crate::postings::tests::test_skip_against_unoptimized;\n    use crate::query::score_combiner::{DoNothingCombiner, SumCombiner};\n    use crate::query::{ConstScorer, Scorer, VecDocSet};\n    use crate::tests::sample_with_seed;\n\n    #[test]\n    fn test_reqopt_scorer_empty() {\n        let req = vec![1, 3, 7];\n        let mut reqoptscorer: RequiredOptionalScorer<_, _, SumCombiner> =\n            RequiredOptionalScorer::new(\n                ConstScorer::from(VecDocSet::from(req.clone())),\n                ConstScorer::from(VecDocSet::from(vec![])),\n            );\n        let mut docs = vec![];\n        while reqoptscorer.doc() != TERMINATED {\n            docs.push(reqoptscorer.doc());\n            reqoptscorer.advance();\n        }\n        assert_eq!(docs, req);\n    }\n\n    #[test]\n    fn test_reqopt_scorer() {\n        let mut reqoptscorer: RequiredOptionalScorer<_, _, SumCombiner> =\n            RequiredOptionalScorer::new(\n                ConstScorer::new(VecDocSet::from(vec![1, 3, 7, 8, 9, 10, 13, 15]), 1.0),\n                ConstScorer::new(VecDocSet::from(vec![1, 2, 7, 11, 12, 15]), 1.0),\n            );\n        {\n            assert_eq!(reqoptscorer.doc(), 1);\n            assert_eq!(reqoptscorer.score(), 2.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 3);\n            assert_eq!(reqoptscorer.doc(), 3);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 7);\n            assert_eq!(reqoptscorer.doc(), 7);\n            assert_eq!(reqoptscorer.score(), 2.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 8);\n            assert_eq!(reqoptscorer.doc(), 8);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 9);\n            assert_eq!(reqoptscorer.doc(), 9);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 10);\n            assert_eq!(reqoptscorer.doc(), 10);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 13);\n            assert_eq!(reqoptscorer.doc(), 13);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n        {\n            assert_eq!(reqoptscorer.advance(), 15);\n            assert_eq!(reqoptscorer.doc(), 15);\n            assert_eq!(reqoptscorer.score(), 2.0);\n        }\n        assert_eq!(reqoptscorer.advance(), TERMINATED);\n    }\n\n    #[test]\n    fn test_reqopt_scorer_skip() {\n        let req_docs = sample_with_seed(10_000, 0.02, 1);\n        let opt_docs = sample_with_seed(10_000, 0.02, 2);\n        let skip_docs = sample_with_seed(10_000, 0.001, 3);\n        test_skip_against_unoptimized(\n            || {\n                Box::new(RequiredOptionalScorer::<_, _, DoNothingCombiner>::new(\n                    ConstScorer::from(VecDocSet::from(req_docs.clone())),\n                    ConstScorer::from(VecDocSet::from(opt_docs.clone())),\n                ))\n            },\n            skip_docs,\n        );\n    }\n\n    #[test]\n    fn test_reqopt_scorer_seek() {\n        let mut reqoptscorer: RequiredOptionalScorer<_, _, SumCombiner> =\n            RequiredOptionalScorer::new(\n                ConstScorer::new(VecDocSet::from(vec![1, 3, 7, 8, 9, 10, 13, 15]), 1.0),\n                ConstScorer::new(VecDocSet::from(vec![2, 7, 11, 12, 15]), 1.0),\n            );\n        {\n            assert_eq!(reqoptscorer.score(), 1.0);\n            assert_eq!(reqoptscorer.seek(7), 7);\n            assert_eq!(reqoptscorer.score(), 2.0);\n        }\n        {\n            assert_eq!(reqoptscorer.score(), 2.0);\n            assert_eq!(reqoptscorer.seek(12), 13);\n            assert_eq!(reqoptscorer.score(), 1.0);\n        }\n    }\n}\n"
  },
  {
    "path": "src/query/score_combiner.rs",
    "content": "use crate::query::Scorer;\nuse crate::Score;\n\n/// The `ScoreCombiner` trait defines how to compute\n/// an overall score given a list of scores.\npub trait ScoreCombiner: Default + Clone + Send + Copy + 'static {\n    /// Aggregates the score combiner with the given scorer.\n    ///\n    /// The `ScoreCombiner` may decide to call `.scorer.score()`\n    /// or not.\n    fn update<TScorer: Scorer>(&mut self, scorer: &mut TScorer);\n\n    /// Clears the score combiner state back to its initial state.\n    fn clear(&mut self);\n\n    /// Returns the aggregate score.\n    fn score(&self) -> Score;\n}\n\n/// Just ignores scores. The `DoNothingCombiner` does not\n/// even call the scorers `.score()` function.\n///\n/// It is useful to optimize the case when scoring is disabled.\n#[derive(Default, Clone, Copy)] //< these should not be too much work :)\npub struct DoNothingCombiner;\n\nimpl ScoreCombiner for DoNothingCombiner {\n    fn update<TScorer: Scorer>(&mut self, _scorer: &mut TScorer) {}\n\n    fn clear(&mut self) {}\n\n    #[inline]\n    fn score(&self) -> Score {\n        1.0\n    }\n}\n\n/// Sums the score of different scorers.\n#[derive(Default, Clone, Copy)]\npub struct SumCombiner {\n    score: Score,\n}\n\nimpl ScoreCombiner for SumCombiner {\n    fn update<TScorer: Scorer>(&mut self, scorer: &mut TScorer) {\n        self.score += scorer.score();\n    }\n\n    fn clear(&mut self) {\n        self.score = 0.0;\n    }\n\n    #[inline]\n    fn score(&self) -> Score {\n        self.score\n    }\n}\n\n/// Take max score of different scorers\n/// and optionally sum it with other matches multiplied by `tie_breaker`\n#[derive(Default, Clone, Copy)]\npub struct DisjunctionMaxCombiner {\n    max: Score,\n    sum: Score,\n    tie_breaker: Score,\n}\n\nimpl DisjunctionMaxCombiner {\n    /// Creates `DisjunctionMaxCombiner` with tie breaker\n    pub fn with_tie_breaker(tie_breaker: Score) -> DisjunctionMaxCombiner {\n        DisjunctionMaxCombiner {\n            max: 0.0,\n            sum: 0.0,\n            tie_breaker,\n        }\n    }\n}\n\nimpl ScoreCombiner for DisjunctionMaxCombiner {\n    fn update<TScorer: Scorer>(&mut self, scorer: &mut TScorer) {\n        let score = scorer.score();\n        self.max = Score::max(score, self.max);\n        self.sum += score;\n    }\n\n    fn clear(&mut self) {\n        self.max = 0.0;\n        self.sum = 0.0;\n    }\n\n    #[inline]\n    fn score(&self) -> Score {\n        self.max + (self.sum - self.max) * self.tie_breaker\n    }\n}\n"
  },
  {
    "path": "src/query/scorer.rs",
    "content": "use std::ops::DerefMut;\n\nuse downcast_rs::impl_downcast;\n\nuse crate::docset::DocSet;\nuse crate::Score;\n\n/// Scored set of documents matching a query within a specific segment.\n///\n/// See [`Query`](crate::query::Query).\npub trait Scorer: downcast_rs::Downcast + DocSet + 'static {\n    /// Returns the score.\n    ///\n    /// This method will perform a bit of computation and is not cached.\n    fn score(&mut self) -> Score;\n}\n\nimpl_downcast!(Scorer);\n\nimpl Scorer for Box<dyn Scorer> {\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.deref_mut().score()\n    }\n}\n"
  },
  {
    "path": "src/query/set_query.rs",
    "content": "use std::collections::HashMap;\n\nuse tantivy_fst::raw::CompiledAddr;\nuse tantivy_fst::{Automaton, Map};\n\nuse crate::query::score_combiner::DoNothingCombiner;\nuse crate::query::{AutomatonWeight, BooleanWeight, EnableScoring, Occur, Query, Weight};\nuse crate::schema::{Field, Schema};\nuse crate::Term;\n\n/// A Term Set Query matches all of the documents containing any of the Term provided\n#[derive(Debug, Clone)]\npub struct TermSetQuery {\n    terms_map: HashMap<Field, Vec<Term>>,\n}\n\nimpl TermSetQuery {\n    /// Create a Term Set Query\n    pub fn new<T: IntoIterator<Item = Term>>(terms: T) -> Self {\n        let mut terms_map: HashMap<_, Vec<_>> = HashMap::new();\n        for term in terms {\n            terms_map.entry(term.field()).or_default().push(term);\n        }\n\n        for terms in terms_map.values_mut() {\n            terms.sort_unstable();\n            terms.dedup();\n        }\n\n        TermSetQuery { terms_map }\n    }\n\n    fn specialized_weight(\n        &self,\n        schema: &Schema,\n    ) -> crate::Result<BooleanWeight<DoNothingCombiner>> {\n        let mut sub_queries: Vec<(_, Box<dyn Weight>)> = Vec::with_capacity(self.terms_map.len());\n\n        for (&field, sorted_terms) in self.terms_map.iter() {\n            let field_entry = schema.get_field_entry(field);\n            let field_type = field_entry.field_type();\n            if !field_type.is_indexed() {\n                let error_msg = format!(\"Field {:?} is not indexed.\", field_entry.name());\n                return Err(crate::TantivyError::SchemaError(error_msg));\n            }\n\n            // In practice this won't fail because:\n            // - we are writing to memory, so no IoError\n            // - Terms are ordered\n            let map = Map::from_iter(\n                sorted_terms\n                    .iter()\n                    .map(|key| (key.serialized_value_bytes(), 0)),\n            )\n            .map_err(std::io::Error::other)?;\n\n            sub_queries.push((\n                Occur::Should,\n                Box::new(AutomatonWeight::new(field, SetDfaWrapper(map))),\n            ));\n        }\n\n        Ok(BooleanWeight::new(\n            sub_queries,\n            false,\n            Box::new(|| DoNothingCombiner),\n        ))\n    }\n}\n\nimpl Query for TermSetQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        Ok(Box::new(self.specialized_weight(enable_scoring.schema())?))\n    }\n\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        for terms in self.terms_map.values() {\n            for term in terms {\n                visitor(term, false);\n            }\n        }\n    }\n}\n\nstruct SetDfaWrapper(Map<Vec<u8>>);\n\nimpl Automaton for SetDfaWrapper {\n    type State = Option<CompiledAddr>;\n\n    fn start(&self) -> Option<CompiledAddr> {\n        Some(self.0.as_ref().root().addr())\n    }\n\n    fn is_match(&self, state_opt: &Option<CompiledAddr>) -> bool {\n        if let Some(state) = state_opt {\n            self.0.as_ref().node(*state).is_final()\n        } else {\n            false\n        }\n    }\n\n    fn accept(&self, state_opt: &Option<CompiledAddr>, byte: u8) -> Option<CompiledAddr> {\n        let state = state_opt.as_ref()?;\n        let node = self.0.as_ref().node(*state);\n        let transition = node.find_input(byte)?;\n        Some(node.transition_addr(transition))\n    }\n\n    fn can_match(&self, state: &Self::State) -> bool {\n        state.is_some()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::collector::TopDocs;\n    use crate::query::{QueryParser, TermSetQuery};\n    use crate::schema::{Schema, TEXT};\n    use crate::{assert_nearly_equals, Index, IndexWriter, Term};\n\n    #[test]\n    pub fn test_term_set_query() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field1 = schema_builder.add_text_field(\"field1\", TEXT);\n        let field2 = schema_builder.add_text_field(\"field2\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(\n                field1 => \"doc1\",\n                field2 => \"val1\",\n            ))?;\n            index_writer.add_document(doc!(\n                field1 => \"doc2\",\n                field2 => \"val2\",\n            ))?;\n            index_writer.add_document(doc!(\n                field1 => \"doc3\",\n                field2 => \"val3\",\n            ))?;\n            index_writer.add_document(doc!(\n                field1 => \"val3\",\n                field2 => \"doc3\",\n            ))?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        {\n            // single element\n            let terms = vec![Term::from_field_text(field1, \"doc1\")];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected 1 document\");\n            let (score, _) = top_docs[0];\n            assert_nearly_equals!(1.0, score);\n        }\n\n        {\n            // single element, absent\n            let terms = vec![Term::from_field_text(field1, \"doc4\")];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(1).order_by_score())?;\n            assert!(top_docs.is_empty(), \"Expected 0 document\");\n        }\n\n        {\n            // multiple elements\n            let terms = vec![\n                Term::from_field_text(field1, \"doc1\"),\n                Term::from_field_text(field1, \"doc2\"),\n            ];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 2, \"Expected 2 documents\");\n            for (score, _) in top_docs {\n                assert_nearly_equals!(1.0, score);\n            }\n        }\n\n        {\n            // multiple elements, mixed fields\n            let terms = vec![\n                Term::from_field_text(field1, \"doc1\"),\n                Term::from_field_text(field1, \"doc1\"),\n                Term::from_field_text(field2, \"val2\"),\n            ];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(3).order_by_score())?;\n\n            assert_eq!(top_docs.len(), 2, \"Expected 2 document\");\n            for (score, _) in top_docs {\n                assert_nearly_equals!(1.0, score);\n            }\n        }\n\n        {\n            // no field crosstalk\n            let terms = vec![Term::from_field_text(field1, \"doc3\")];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(3).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected 1 document\");\n\n            let terms = vec![Term::from_field_text(field2, \"doc3\")];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(3).order_by_score())?;\n            assert_eq!(top_docs.len(), 1, \"Expected 1 document\");\n\n            let terms = vec![\n                Term::from_field_text(field1, \"doc3\"),\n                Term::from_field_text(field2, \"doc3\"),\n            ];\n\n            let term_set_query = TermSetQuery::new(terms);\n            let top_docs =\n                searcher.search(&term_set_query, &TopDocs::with_limit(3).order_by_score())?;\n            assert_eq!(top_docs.len(), 2, \"Expected 2 document\");\n        }\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_set_query_parser() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"field\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        let field = schema.get_field(\"field\").unwrap();\n        index_writer.add_document(doc!(\n          field => \"val1\",\n        ))?;\n        index_writer.add_document(doc!(\n          field => \"val2\",\n        ))?;\n        index_writer.add_document(doc!(\n          field => \"val3\",\n        ))?;\n        index_writer.commit()?;\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![]);\n        let query = query_parser.parse_query(\"field: IN [val1 val2]\")?;\n        let top_docs = searcher.search(&query, &TopDocs::with_limit(3).order_by_score())?;\n        assert_eq!(top_docs.len(), 2);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/size_hint.rs",
    "content": "/// Computes the estimated number of documents in the intersection of multiple docsets\n/// given their sizes.\n///\n/// # Arguments\n/// * `docset_sizes` - An iterator over the sizes of the docsets (number of documents in each set).\n/// * `max_docs` - The maximum number of docs that can hit, usually number of documents in the\n///   segment.\n///\n/// # Returns\n/// The estimated number of documents in the intersection.\npub fn estimate_intersection<I>(mut docset_sizes: I, max_docs: u32) -> u32\nwhere I: Iterator<Item = u32> {\n    if max_docs == 0u32 {\n        return 0u32;\n    }\n    // Terms tend to be not really randomly distributed.\n    // This factor is used to adjust the estimate.\n    let mut co_loc_factor: f64 = 1.3;\n\n    let mut intersection_estimate = match docset_sizes.next() {\n        Some(first_size) => first_size as f64,\n        None => return 0, // No docsets provided, so return 0.\n    };\n\n    let mut smallest_docset_size = intersection_estimate;\n    // Assuming random distribution of terms, the probability of a document being in the\n    // intersection\n    for size in docset_sizes {\n        // Diminish the co-location factor for each additional set, or we will overestimate.\n        co_loc_factor = (co_loc_factor - 0.1).max(1.0);\n        intersection_estimate *= (size as f64 / max_docs as f64) * co_loc_factor;\n        smallest_docset_size = smallest_docset_size.min(size as f64);\n    }\n\n    intersection_estimate.round().min(smallest_docset_size) as u32\n}\n\n/// Computes the estimated number of documents in the union of multiple docsets\n/// given their sizes.\n///\n/// # Arguments\n/// * `docset_sizes` - An iterator over the sizes of the docsets (number of documents in each set).\n/// * `max_docs` - The maximum number of docs that can hit, usually number of documents in the\n///   segment.\n///\n/// # Returns\n/// The estimated number of documents in the union.\npub fn estimate_union<I>(docset_sizes: I, max_docs: u32) -> u32\nwhere I: Iterator<Item = u32> {\n    // Terms tend to be not really randomly distributed.\n    // This factor is used to adjust the estimate.\n    // Unlike intersection, the co-location reduces the estimate.\n    let co_loc_factor = 0.8;\n\n    // The approach for union is to compute the probability of a document not being in any of the\n    // sets\n    let mut not_in_any_set_prob = 1.0;\n\n    // Assuming random distribution of terms, the probability of a document being in the\n    // union is the complement of the probability of it not being in any of the sets.\n    for size in docset_sizes {\n        let prob_in_set = (size as f64 / max_docs as f64) * co_loc_factor;\n        not_in_any_set_prob *= 1.0 - prob_in_set;\n    }\n\n    let union_estimate = (max_docs as f64 * (1.0 - not_in_any_set_prob)).round();\n\n    union_estimate.min(max_docs as f64) as u32\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_estimate_intersection_small1() {\n        let docset_sizes = &[500, 1000];\n        let n = 10_000;\n        let result = estimate_intersection(docset_sizes.iter().copied(), n);\n        assert_eq!(result, 60);\n    }\n\n    #[test]\n    fn test_estimate_intersection_small2() {\n        let docset_sizes = &[500, 1000, 1500];\n        let n = 10_000;\n        let result = estimate_intersection(docset_sizes.iter().copied(), n);\n        assert_eq!(result, 10);\n    }\n\n    #[test]\n    fn test_estimate_intersection_large_values() {\n        let docset_sizes = &[100_000, 50_000, 30_000];\n        let n = 1_000_000;\n        let result = estimate_intersection(docset_sizes.iter().copied(), n);\n        assert_eq!(result, 198);\n    }\n\n    #[test]\n    fn test_estimate_union_small() {\n        let docset_sizes = &[500, 1000, 1500];\n        let n = 10000;\n        let result = estimate_union(docset_sizes.iter().copied(), n);\n        assert_eq!(result, 2228);\n    }\n\n    #[test]\n    fn test_estimate_union_large_values() {\n        let docset_sizes = &[100000, 50000, 30000];\n        let n = 1000000;\n        let result = estimate_union(docset_sizes.iter().copied(), n);\n        assert_eq!(result, 137997);\n    }\n\n    #[test]\n    fn test_estimate_intersection_large() {\n        let docset_sizes: Vec<_> = (0..10).map(|_| 4_000_000).collect();\n        let n = 5_000_000;\n        let result = estimate_intersection(docset_sizes.iter().copied(), n);\n        // Check that it doesn't overflow and returns a reasonable result\n        assert_eq!(result, 708_670);\n    }\n\n    #[test]\n    fn test_estimate_intersection_overflow_safety() {\n        let docset_sizes: Vec<_> = (0..100).map(|_| 4_000_000).collect();\n        let n = 5_000_000;\n        let result = estimate_intersection(docset_sizes.iter().copied(), n);\n        // Check that it doesn't overflow and returns a reasonable result\n        assert_eq!(result, 0);\n    }\n\n    #[test]\n    fn test_estimate_union_overflow_safety() {\n        let docset_sizes: Vec<_> = (0..100).map(|_| 1_000_000).collect();\n        let n = 20_000_000;\n        let result = estimate_union(docset_sizes.iter().copied(), n);\n        // Check that it doesn't overflow and returns a reasonable result\n        assert_eq!(result, 19_662_594);\n    }\n}\n"
  },
  {
    "path": "src/query/term_query/mod.rs",
    "content": "mod term_query;\nmod term_scorer;\nmod term_weight;\n\npub use self::term_query::TermQuery;\npub use self::term_scorer::TermScorer;\n#[cfg(test)]\nmod tests {\n\n    use crate::collector::TopDocs;\n    use crate::docset::DocSet;\n    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n    use crate::query::term_query::TermScorer;\n    use crate::query::{\n        AllScorer, EmptyScorer, EnableScoring, Query, QueryParser, Scorer, TermQuery,\n    };\n    use crate::schema::{Field, IndexRecordOption, Schema, FAST, STRING, TEXT};\n    use crate::{assert_nearly_equals, DocAddress, Index, IndexWriter, Term, TERMINATED};\n\n    #[test]\n    pub fn test_term_query_no_freq() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            let doc = doc!(text_field => \"a\");\n            index_writer.add_document(doc)?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let term_query = TermQuery::new(\n            Term::from_field_text(text_field, \"a\"),\n            IndexRecordOption::Basic,\n        );\n        let term_weight = term_query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n        let segment_reader = searcher.segment_reader(0);\n        let mut term_scorer = term_weight.scorer(segment_reader, 1.0)?;\n        assert_eq!(term_scorer.doc(), 0);\n        assert_nearly_equals!(term_scorer.score(), 0.28768212);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_term_query_multiple_of_block_len() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", STRING);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            for _ in 0..COMPRESSION_BLOCK_SIZE {\n                let doc = doc!(text_field => \"a\");\n                index_writer.add_document(doc)?;\n            }\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let term_query = TermQuery::new(\n            Term::from_field_text(text_field, \"a\"),\n            IndexRecordOption::Basic,\n        );\n        let term_weight = term_query.weight(EnableScoring::enabled_from_searcher(&searcher))?;\n        let segment_reader = searcher.segment_reader(0);\n        let mut term_scorer = term_weight.scorer(segment_reader, 1.0)?;\n        for i in 0u32..COMPRESSION_BLOCK_SIZE as u32 {\n            assert_eq!(term_scorer.doc(), i);\n            if i == COMPRESSION_BLOCK_SIZE as u32 - 1u32 {\n                assert_eq!(term_scorer.advance(), TERMINATED);\n            } else {\n                assert_eq!(term_scorer.advance(), i + 1);\n            }\n        }\n        assert_eq!(term_scorer.doc(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    pub fn test_term_weight() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let left_field = schema_builder.add_text_field(\"left\", TEXT);\n        let right_field = schema_builder.add_text_field(\"right\", TEXT);\n        let large_field = schema_builder.add_text_field(\"large\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(\n                left_field => \"left1 left2 left2 left2f2 left2f2 left3 abcde abcde abcde abcde abcde abcde abcde abcde abcde abcewde abcde abcde\",\n                right_field => \"right1 right2\",\n                large_field => \"large0 large1 large2 large3 large4 large5 large6 large7 large8 large9 large10 large11 large12 large13 large14 large15 large16 large17 large18 large19 large20 large21 large22 large23 large24 large25 large26 large27 large28 large29 large30 large31 large32 large33 large34 large35 large36 large37 large38 large39 large40 large41 large42 large43 large44 large45 large46 large47 large48 large49 large50 large51 large52 large53 large54 large55 large56 large57 large58 large59 large60 large61 large62 large63 large64 large65 large66 large67 large68 large69 large70 large71 large72 large73 large74 large75 large76 large77 large78 large79 large80 large81 large82 large83 large84 large85 large86 large87 large88 large89 large90 large91 large92 large93 large94 large95 large96 large97 large98 large99 large100 large101 large102 large103 large104 large105 large106 large107 large108 large109 large110 large111 large112 large113 large114 large115 large116 large117 large118 large119 large120 large121 large122 large123 large124 large125 large126 large127 large128 large129 large130 large131 large132 large133 large134 large135 large136 large137 large138 large139 large140 large141 large142 large143 large144 large145 large146 large147 large148 large149 large150 large151 large152 large153 large154 large155 large156 large157 large158 large159 large160 large161 large162 large163 large164 large165 large166 large167 large168 large169 large170 large171 large172 large173 large174 large175 large176 large177 large178 large179 large180 large181 large182 large183 large184 large185 large186 large187 large188 large189 large190 large191 large192 large193 large194 large195 large196 large197 large198 large199 large200 large201 large202 large203 large204 large205 large206 large207 large208 large209 large210 large211 large212 large213 large214 large215 large216 large217 large218 large219 large220 large221 large222 large223 large224 large225 large226 large227 large228 large229 large230 large231 large232 large233 large234 large235 large236 large237 large238 large239 large240 large241 large242 large243 large244 large245 large246 large247 large248 large249 large250 large251 large252 large253 large254 large255 large256 large257 large258 large259 large260 large261 large262 large263 large264 large265 large266 large267 large268 large269 large270 large271 large272 large273 large274 large275 large276 large277 large278 large279 large280 large281 large282 large283 large284 large285 large286\"\n            ))?;\n            index_writer.add_document(doc!(left_field => \"left4 left1\"))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        {\n            let term = Term::from_field_text(left_field, \"left2\");\n            let term_query = TermQuery::new(term, IndexRecordOption::WithFreqs);\n            let topdocs = searcher.search(&term_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(topdocs.len(), 1);\n            let (score, _) = topdocs[0];\n            assert_nearly_equals!(0.77802235, score);\n        }\n        {\n            let term = Term::from_field_text(left_field, \"left1\");\n            let term_query = TermQuery::new(term, IndexRecordOption::WithFreqs);\n            let top_docs =\n                searcher.search(&term_query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 2);\n            let (score1, _) = top_docs[0];\n            assert_nearly_equals!(0.27101856, score1);\n            let (score2, _) = top_docs[1];\n            assert_nearly_equals!(0.13736556, score2);\n        }\n        {\n            let query_parser = QueryParser::for_index(&index, Vec::new());\n            let query = query_parser.parse_query(\"left:left2 left:left1\")?;\n            let top_docs = searcher.search(&query, &TopDocs::with_limit(2).order_by_score())?;\n            assert_eq!(top_docs.len(), 2);\n            let (score1, _) = top_docs[0];\n            assert_nearly_equals!(0.9153879, score1);\n            let (score2, _) = top_docs[1];\n            assert_nearly_equals!(0.27101856, score2);\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_count_when_there_are_deletes() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a b\"))?;\n        index_writer.add_document(doc!(text_field=>\"a c\"))?;\n        index_writer.delete_term(Term::from_field_text(text_field, \"b\"));\n        index_writer.commit()?;\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let term_query = TermQuery::new(term_a, IndexRecordOption::Basic);\n        let reader = index.reader()?;\n        assert_eq!(term_query.count(&reader.searcher())?, 1);\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_simple_seek() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.commit()?;\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let term_query = TermQuery::new(term_a, IndexRecordOption::Basic);\n        let searcher = index.reader()?.searcher();\n        let term_weight =\n            term_query.weight(EnableScoring::disabled_from_schema(searcher.schema()))?;\n        let mut term_scorer = term_weight.scorer(searcher.segment_reader(0u32), 1.0)?;\n        assert_eq!(term_scorer.doc(), 0u32);\n        term_scorer.seek(1u32);\n        assert_eq!(term_scorer.doc(), 1u32);\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_debug() {\n        let term_query = TermQuery::new(\n            Term::from_field_text(Field::from_field_id(1), \"hello\"),\n            IndexRecordOption::WithFreqs,\n        );\n        assert_eq!(\n            format!(\"{term_query:?}\"),\n            r#\"TermQuery(Term(field=1, type=Str, \"hello\"))\"#\n        );\n    }\n\n    #[test]\n    fn test_term_query_explain() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text_field=>\"b\"))?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.add_document(doc!(text_field=>\"a\"))?;\n        index_writer.add_document(doc!(text_field=>\"b\"))?;\n        index_writer.commit()?;\n        let term_a = Term::from_field_text(text_field, \"a\");\n        let term_query = TermQuery::new(term_a, IndexRecordOption::Basic);\n        let searcher = index.reader()?.searcher();\n        {\n            let explanation = term_query.explain(&searcher, DocAddress::new(0u32, 1u32))?;\n            assert_nearly_equals!(explanation.value(), std::f32::consts::LN_2);\n        }\n        {\n            let explanation_err = term_query.explain(&searcher, DocAddress::new(0u32, 0u32));\n            assert!(matches!(\n                explanation_err,\n                Err(crate::TantivyError::InvalidArgument(_msg))\n            ));\n        }\n        {\n            let explanation_err = term_query.explain(&searcher, DocAddress::new(0u32, 3u32));\n            assert!(matches!(\n                explanation_err,\n                Err(crate::TantivyError::InvalidArgument(_msg))\n            ));\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_fallback_to_fastfield() -> crate::Result<()> {\n        use crate::collector::Count;\n        use crate::schema::FAST;\n\n        // Create a FAST-only numeric field (not indexed)\n        let mut schema_builder = Schema::builder();\n        let num_field = schema_builder.add_u64_field(\"num\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(num_field => 10u64))?;\n            index_writer.add_document(doc!(num_field => 20u64))?;\n            index_writer.add_document(doc!(num_field => 10u64))?;\n            index_writer.commit()?;\n        }\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n\n        // TermQuery should fall back to a fastfield range query and match correctly.\n        let tq_10 = TermQuery::new(\n            Term::from_field_u64(num_field, 10u64),\n            IndexRecordOption::Basic,\n        );\n        let tq_20 = TermQuery::new(\n            Term::from_field_u64(num_field, 20u64),\n            IndexRecordOption::Basic,\n        );\n        let tq_30 = TermQuery::new(\n            Term::from_field_u64(num_field, 30u64),\n            IndexRecordOption::Basic,\n        );\n\n        let count_10 = searcher.search(&tq_10, &Count)?;\n        let count_20 = searcher.search(&tq_20, &Count)?;\n        let count_30 = searcher.search(&tq_30, &Count)?;\n\n        assert_eq!(count_10, 2);\n        assert_eq!(count_20, 1);\n        assert_eq!(count_30, 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_fallback_text_fast_only() -> crate::Result<()> {\n        use crate::collector::Count;\n\n        // FAST-only text field (not indexed)\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field => \"hello\"))?;\n            index_writer.add_document(doc!(text_field => \"world\"))?;\n            index_writer.add_document(doc!(text_field => \"hello\"))?;\n            index_writer.commit()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        let tq_hello = TermQuery::new(\n            Term::from_field_text(text_field, \"hello\"),\n            IndexRecordOption::Basic,\n        );\n        let tq_world = TermQuery::new(\n            Term::from_field_text(text_field, \"world\"),\n            IndexRecordOption::Basic,\n        );\n        let tq_missing = TermQuery::new(\n            Term::from_field_text(text_field, \"nope\"),\n            IndexRecordOption::Basic,\n        );\n\n        assert_eq!(searcher.search(&tq_hello, &Count)?, 2);\n        assert_eq!(searcher.search(&tq_world, &Count)?, 1);\n        assert_eq!(searcher.search(&tq_missing, &Count)?, 0);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_fallback_json_fast_only() -> crate::Result<()> {\n        use crate::collector::Count;\n        use crate::fastfield::FastValue;\n        use crate::schema::FAST;\n\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(json_field => json!({\"a\": 10, \"b\": \"x\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"a\": 20, \"b\": \"y\"})))?;\n            index_writer.add_document(doc!(json_field => json!({\"a\": 10, \"b\": \"z\"})))?;\n            index_writer.commit()?;\n        }\n\n        fn json_term_fast<T: FastValue>(field: Field, path: &str, v: T) -> Term {\n            let mut term = Term::from_field_json_path(field, path, true);\n            term.append_type_and_fast_value(v);\n            term\n        }\n        fn json_term_str(field: Field, path: &str, v: &str) -> Term {\n            let mut term = Term::from_field_json_path(field, path, true);\n            term.append_type_and_str(v);\n            term\n        }\n\n        let searcher = index.reader()?.searcher();\n        // numeric path match\n        let tq_a10 = TermQuery::new(\n            json_term_fast(json_field, \"a\", 10u64),\n            IndexRecordOption::Basic,\n        );\n        let tq_a20 = TermQuery::new(\n            json_term_fast(json_field, \"a\", 20u64),\n            IndexRecordOption::Basic,\n        );\n        let tq_a30 = TermQuery::new(\n            json_term_fast(json_field, \"a\", 30u64),\n            IndexRecordOption::Basic,\n        );\n        assert_eq!(searcher.search(&tq_a10, &Count)?, 2);\n        assert_eq!(searcher.search(&tq_a20, &Count)?, 1);\n        assert_eq!(searcher.search(&tq_a30, &Count)?, 0);\n\n        // string path match\n        let tq_bx = TermQuery::new(\n            json_term_str(json_field, \"b\", \"x\"),\n            IndexRecordOption::Basic,\n        );\n        let tq_by = TermQuery::new(\n            json_term_str(json_field, \"b\", \"y\"),\n            IndexRecordOption::Basic,\n        );\n        let tq_bm = TermQuery::new(\n            json_term_str(json_field, \"b\", \"missing\"),\n            IndexRecordOption::Basic,\n        );\n        assert_eq!(searcher.search(&tq_bx, &Count)?, 1);\n        assert_eq!(searcher.search(&tq_by, &Count)?, 1);\n        assert_eq!(searcher.search(&tq_bm, &Count)?, 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_fallback_ip_fast_only() -> crate::Result<()> {\n        use std::net::IpAddr;\n        use std::str::FromStr;\n\n        use crate::collector::Count;\n        use crate::schema::{IntoIpv6Addr, FAST};\n\n        let mut schema_builder = Schema::builder();\n        let ip_field = schema_builder.add_ip_addr_field(\"ip\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        let ip1 = IpAddr::from_str(\"127.0.0.1\").unwrap().into_ipv6_addr();\n        let ip2 = IpAddr::from_str(\"127.0.0.2\").unwrap().into_ipv6_addr();\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(ip_field => ip1))?;\n            index_writer.add_document(doc!(ip_field => ip2))?;\n            index_writer.add_document(doc!(ip_field => ip1))?;\n            index_writer.commit()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        let tq_ip1 = TermQuery::new(\n            Term::from_field_ip_addr(ip_field, ip1),\n            IndexRecordOption::Basic,\n        );\n        let tq_ip2 = TermQuery::new(\n            Term::from_field_ip_addr(ip_field, ip2),\n            IndexRecordOption::Basic,\n        );\n        let ip3 = IpAddr::from_str(\"127.0.0.3\").unwrap().into_ipv6_addr();\n        let tq_ip3 = TermQuery::new(\n            Term::from_field_ip_addr(ip_field, ip3),\n            IndexRecordOption::Basic,\n        );\n\n        assert_eq!(searcher.search(&tq_ip1, &Count)?, 2);\n        assert_eq!(searcher.search(&tq_ip2, &Count)?, 1);\n        assert_eq!(searcher.search(&tq_ip3, &Count)?, 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_query_fallback_fastfield_with_scores_errors() -> crate::Result<()> {\n        use crate::collector::TopDocs;\n\n        // FAST-only numeric field (not indexed) should error when scoring is required\n        let mut schema_builder = Schema::builder();\n        let num_field = schema_builder.add_u64_field(\"num\", FAST);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(num_field => 10u64))?;\n            index_writer.add_document(doc!(num_field => 20u64))?;\n            index_writer.commit()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        let tq = TermQuery::new(\n            Term::from_field_u64(num_field, 10u64),\n            IndexRecordOption::Basic,\n        );\n\n        // Using TopDocs requires scoring; since the field is not indexed,\n        // TermQuery cannot score and should return a SchemaError.\n        let res = searcher.search(&tq, &TopDocs::with_limit(1).order_by_score());\n        assert!(matches!(res, Err(crate::TantivyError::SchemaError(_))));\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_weight_all_query_optimization() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", crate::schema::TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(text_field=>\"hello\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(text_field=>\"hello happy\"))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let get_scorer_for_term = |term: &str| {\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, term),\n                IndexRecordOption::Basic,\n            );\n            let term_weight = term_query\n                .weight(EnableScoring::disabled_from_schema(&schema))\n                .unwrap();\n            term_weight\n                .scorer(searcher.segment_reader(0u32), 1.0f32)\n                .unwrap()\n        };\n        // Should be an allscorer\n        let match_all_scorer = get_scorer_for_term(\"hello\");\n        // Should be a term scorer\n        let match_some_scorer = get_scorer_for_term(\"happy\");\n        // Should be an empty scorer\n        let empty_scorer = get_scorer_for_term(\"tax\");\n        assert!(match_all_scorer.is::<AllScorer>());\n        assert!(match_some_scorer.is::<TermScorer>());\n        assert!(empty_scorer.is::<EmptyScorer>());\n    }\n\n    #[test]\n    fn test_term_weight_all_query_optimization_disable_when_scoring_enabled() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", crate::schema::TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema.clone());\n        let mut index_writer = index.writer_for_tests().unwrap();\n        index_writer\n            .add_document(doc!(text_field=>\"hello\"))\n            .unwrap();\n        index_writer\n            .add_document(doc!(text_field=>\"hello happy\"))\n            .unwrap();\n        index_writer.commit().unwrap();\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let get_scorer_for_term = |term: &str| {\n            let term_query = TermQuery::new(\n                Term::from_field_text(text_field, term),\n                IndexRecordOption::Basic,\n            );\n            let term_weight = term_query\n                .weight(EnableScoring::enabled_from_searcher(&searcher))\n                .unwrap();\n            term_weight\n                .scorer(searcher.segment_reader(0u32), 1.0f32)\n                .unwrap()\n        };\n        // Should be an allscorer\n        let match_all_scorer = get_scorer_for_term(\"hello\");\n        // Should be a term scorer\n        let one_scorer = get_scorer_for_term(\"happy\");\n        // Should be an empty scorer\n        let empty_scorer = get_scorer_for_term(\"tax\");\n        assert!(match_all_scorer.is::<TermScorer>());\n        assert!(one_scorer.is::<TermScorer>());\n        assert!(empty_scorer.is::<EmptyScorer>());\n    }\n}\n"
  },
  {
    "path": "src/query/term_query/term_query.rs",
    "content": "use std::fmt;\nuse std::ops::Bound;\n\nuse super::term_weight::TermWeight;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::range_query::is_type_valid_for_fastfield_range_query;\nuse crate::query::{EnableScoring, Explanation, Query, RangeQuery, Weight};\nuse crate::schema::IndexRecordOption;\nuse crate::Term;\n\n/// A Term query matches all of the documents\n/// containing a specific term.\n///\n/// The score associated is defined as\n/// `idf` *  sqrt(`term_freq` / `field norm`)\n/// in which :\n/// * `idf`        - inverse document frequency.\n/// * `term_freq`  - number of occurrences of the term in the field\n/// * `field norm` - number of tokens in the field.\n///\n/// ```rust\n/// use tantivy::collector::{Count, TopDocs};\n/// use tantivy::query::TermQuery;\n/// use tantivy::schema::{Schema, TEXT, IndexRecordOption};\n/// use tantivy::{doc, Index, IndexWriter, Term};\n/// # fn test() -> tantivy::Result<()> {\n/// let mut schema_builder = Schema::builder();\n/// let title = schema_builder.add_text_field(\"title\", TEXT);\n/// let schema = schema_builder.build();\n/// let index = Index::create_in_ram(schema);\n/// {\n///     let mut index_writer: IndexWriter = index.writer(15_000_000)?;\n///     index_writer.add_document(doc!(\n///         title => \"The Name of the Wind\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"The Diary of Muadib\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"A Dairy Cow\",\n///     ))?;\n///     index_writer.add_document(doc!(\n///         title => \"The Diary of a Young Girl\",\n///     ))?;\n///     index_writer.commit()?;\n/// }\n/// let reader = index.reader()?;\n/// let searcher = reader.searcher();\n/// let query = TermQuery::new(\n///     Term::from_field_text(title, \"diary\"),\n///     IndexRecordOption::Basic,\n/// );\n/// let (top_docs, count) = searcher.search(&query, &(TopDocs::with_limit(2).order_by_score(), Count))?;\n/// assert_eq!(count, 2);\n/// Ok(())\n/// # }\n/// # assert!(test().is_ok());\n/// ```\n#[derive(Clone)]\npub struct TermQuery {\n    term: Term,\n    index_record_option: IndexRecordOption,\n}\n\nimpl fmt::Debug for TermQuery {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"TermQuery({:?})\", self.term)\n    }\n}\n\nimpl TermQuery {\n    /// Creates a new term query.\n    pub fn new(term: Term, segment_postings_options: IndexRecordOption) -> TermQuery {\n        TermQuery {\n            term,\n            index_record_option: segment_postings_options,\n        }\n    }\n\n    /// The `Term` this query is built out of.\n    pub fn term(&self) -> &Term {\n        &self.term\n    }\n\n    /// Returns a weight object.\n    ///\n    /// While `.weight(...)` returns a boxed trait object,\n    /// this method return a specific implementation.\n    /// This is useful for optimization purpose.\n    pub fn specialized_weight(\n        &self,\n        enable_scoring: EnableScoring<'_>,\n    ) -> crate::Result<TermWeight> {\n        let schema = enable_scoring.schema();\n        let field_entry = schema.get_field_entry(self.term.field());\n        if !field_entry.is_indexed() {\n            let error_msg = format!(\"Field {:?} is not indexed.\", field_entry.name());\n            return Err(crate::TantivyError::SchemaError(error_msg));\n        }\n        let bm25_weight = match enable_scoring {\n            EnableScoring::Enabled {\n                statistics_provider,\n                ..\n            } => Bm25Weight::for_terms(statistics_provider, std::slice::from_ref(&self.term))?,\n            EnableScoring::Disabled { .. } => {\n                Bm25Weight::new(Explanation::new(\"<no score>\", 1.0f32), 1.0f32)\n            }\n        };\n        let scoring_enabled = enable_scoring.is_scoring_enabled();\n        let index_record_option = if scoring_enabled {\n            self.index_record_option\n        } else {\n            IndexRecordOption::Basic\n        };\n\n        Ok(TermWeight::new(\n            self.term.clone(),\n            index_record_option,\n            bm25_weight,\n            scoring_enabled,\n        ))\n    }\n}\n\nimpl Query for TermQuery {\n    fn weight(&self, enable_scoring: EnableScoring<'_>) -> crate::Result<Box<dyn Weight>> {\n        // If the field is not indexed but is a suitable fast field, fall back to a range query\n        // on the fast field matching exactly this term.\n        //\n        // Note: This is considerable slower since it requires to scan the entire fast field.\n        // TODO: The range query would gain from having a single-value optimization\n        let schema = enable_scoring.schema();\n        let field_entry = schema.get_field_entry(self.term.field());\n        if !field_entry.is_indexed()\n            && field_entry.is_fast()\n            && is_type_valid_for_fastfield_range_query(self.term.typ())\n            && !enable_scoring.is_scoring_enabled()\n        {\n            let range_query = RangeQuery::new(\n                Bound::Included(self.term.clone()),\n                Bound::Included(self.term.clone()),\n            );\n            return range_query.weight(enable_scoring);\n        }\n        Ok(Box::new(self.specialized_weight(enable_scoring)?))\n    }\n    fn query_terms<'a>(&'a self, visitor: &mut dyn FnMut(&'a Term, bool)) {\n        visitor(&self.term, false);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::net::{IpAddr, Ipv6Addr};\n    use std::str::FromStr;\n\n    use columnar::MonotonicallyMappableToU128;\n\n    use crate::collector::{Count, TopDocs};\n    use crate::query::{Query, QueryParser, TermQuery};\n    use crate::schema::{IndexRecordOption, IntoIpv6Addr, Schema, INDEXED, STORED};\n    use crate::{Index, IndexWriter, Term};\n\n    #[test]\n    fn search_ip_test() {\n        let mut schema_builder = Schema::builder();\n        let ip_field = schema_builder.add_ip_addr_field(\"ip\", INDEXED | STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let ip_addr_1 = IpAddr::from_str(\"127.0.0.1\").unwrap().into_ipv6_addr();\n        let ip_addr_2 = Ipv6Addr::from_u128(10);\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer\n                .add_document(doc!(\n                    ip_field => ip_addr_1\n                ))\n                .unwrap();\n            index_writer\n                .add_document(doc!(\n                    ip_field => ip_addr_2\n                ))\n                .unwrap();\n\n            index_writer.commit().unwrap();\n        }\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n\n        let assert_single_hit = |query| {\n            let (_top_docs, count) = searcher\n                .search(&query, &(TopDocs::with_limit(2).order_by_score(), Count))\n                .unwrap();\n            assert_eq!(count, 1);\n        };\n        let query_from_text = |text: String| {\n            QueryParser::for_index(&index, vec![ip_field])\n                .parse_query(&text)\n                .unwrap()\n        };\n\n        let query_from_ip = |ip_addr| -> Box<dyn Query> {\n            Box::new(TermQuery::new(\n                Term::from_field_ip_addr(ip_field, ip_addr),\n                IndexRecordOption::Basic,\n            ))\n        };\n\n        assert_single_hit(query_from_ip(ip_addr_1));\n        assert_single_hit(query_from_ip(ip_addr_2));\n        assert_single_hit(query_from_text(\"127.0.0.1\".to_string()));\n        assert_single_hit(query_from_text(\"\\\"127.0.0.1\\\"\".to_string()));\n        assert_single_hit(query_from_text(format!(\"\\\"{ip_addr_1}\\\"\")));\n        assert_single_hit(query_from_text(format!(\"\\\"{ip_addr_2}\\\"\")));\n    }\n}\n"
  },
  {
    "path": "src/query/term_query/term_scorer.rs",
    "content": "use crate::docset::DocSet;\nuse crate::fieldnorm::FieldNormReader;\nuse crate::postings::{FreqReadingOption, Postings, SegmentPostings};\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::{Explanation, Scorer};\nuse crate::{DocId, Score};\n\n#[derive(Clone)]\npub struct TermScorer {\n    postings: SegmentPostings,\n    fieldnorm_reader: FieldNormReader,\n    similarity_weight: Bm25Weight,\n}\n\nimpl TermScorer {\n    pub fn new(\n        postings: SegmentPostings,\n        fieldnorm_reader: FieldNormReader,\n        similarity_weight: Bm25Weight,\n    ) -> TermScorer {\n        TermScorer {\n            postings,\n            fieldnorm_reader,\n            similarity_weight,\n        }\n    }\n\n    pub(crate) fn seek_block(&mut self, target_doc: DocId) {\n        self.postings.block_cursor.seek_block(target_doc);\n    }\n\n    #[cfg(test)]\n    pub fn create_for_test(\n        doc_and_tfs: &[(DocId, u32)],\n        fieldnorms: &[u32],\n        similarity_weight: Bm25Weight,\n    ) -> TermScorer {\n        assert!(!doc_and_tfs.is_empty());\n        assert!(\n            doc_and_tfs\n                .iter()\n                .map(|(doc, _tf)| *doc)\n                .max()\n                .unwrap_or(0u32)\n                < fieldnorms.len() as u32\n        );\n        let segment_postings =\n            SegmentPostings::create_from_docs_and_tfs(doc_and_tfs, Some(fieldnorms));\n        let fieldnorm_reader = FieldNormReader::for_test(fieldnorms);\n        TermScorer::new(segment_postings, fieldnorm_reader, similarity_weight)\n    }\n\n    /// See `FreqReadingOption`.\n    pub(crate) fn freq_reading_option(&self) -> FreqReadingOption {\n        self.postings.block_cursor.freq_reading_option()\n    }\n\n    /// Returns the maximum score for the current block.\n    ///\n    /// In some rare case, the result may not be exact. In this case a lower value is returned,\n    /// (and may lead us to return a lesser document).\n    ///\n    /// At index time, we store the (fieldnorm_id, term frequency) pair that maximizes the\n    /// score assuming the average fieldnorm computed on this segment.\n    ///\n    /// Though extremely rare, it is theoretically possible that the actual average fieldnorm\n    /// is different enough from the current segment average fieldnorm that the maximum over a\n    /// specific is achieved on a different document.\n    ///\n    /// (The result is on the other hand guaranteed to be correct if there is only one segment).\n    pub fn block_max_score(&mut self) -> Score {\n        self.postings\n            .block_cursor\n            .block_max_score(&self.fieldnorm_reader, &self.similarity_weight)\n    }\n\n    pub fn term_freq(&self) -> u32 {\n        self.postings.term_freq()\n    }\n\n    pub fn fieldnorm_id(&self) -> u8 {\n        self.fieldnorm_reader.fieldnorm_id(self.doc())\n    }\n\n    pub fn explain(&self) -> Explanation {\n        let fieldnorm_id = self.fieldnorm_id();\n        let term_freq = self.term_freq();\n        self.similarity_weight.explain(fieldnorm_id, term_freq)\n    }\n\n    pub fn max_score(&self) -> Score {\n        self.similarity_weight.max_score()\n    }\n\n    pub fn last_doc_in_block(&self) -> DocId {\n        self.postings.block_cursor.skip_reader().last_doc_in_block()\n    }\n}\n\nimpl DocSet for TermScorer {\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        self.postings.advance()\n    }\n\n    #[inline]\n    fn seek(&mut self, target: DocId) -> DocId {\n        debug_assert!(target >= self.doc());\n        self.postings.seek(target)\n    }\n\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.postings.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.postings.size_hint()\n    }\n}\n\nimpl Scorer for TermScorer {\n    #[inline]\n    fn score(&mut self) -> Score {\n        let fieldnorm_id = self.fieldnorm_id();\n        let term_freq = self.term_freq();\n        self.similarity_weight.score(fieldnorm_id, term_freq)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use proptest::prelude::*;\n\n    use crate::index::SegmentId;\n    use crate::indexer::index_writer::MEMORY_BUDGET_NUM_BYTES_MIN;\n    use crate::merge_policy::NoMergePolicy;\n    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;\n    use crate::query::term_query::TermScorer;\n    use crate::query::{Bm25Weight, EnableScoring, Scorer, TermQuery};\n    use crate::schema::{IndexRecordOption, Schema, TEXT};\n    use crate::{\n        assert_nearly_equals, DocId, DocSet, Index, IndexWriter, Score, Searcher, Term, TERMINATED,\n    };\n\n    #[test]\n    fn test_term_scorer_max_score() -> crate::Result<()> {\n        let bm25_weight = Bm25Weight::for_one_term(3, 6, 10.0);\n        let mut term_scorer = TermScorer::create_for_test(\n            &[(2, 3), (3, 12), (7, 8)],\n            &[0, 0, 10, 12, 0, 0, 0, 100],\n            bm25_weight,\n        );\n        let max_scorer = term_scorer.max_score();\n        crate::assert_nearly_equals!(max_scorer, 1.3990127);\n        assert_eq!(term_scorer.doc(), 2);\n        assert_eq!(term_scorer.term_freq(), 3);\n        assert_nearly_equals!(term_scorer.block_max_score(), 1.3676447);\n        assert_nearly_equals!(term_scorer.score(), 1.0892314);\n        assert_eq!(term_scorer.advance(), 3);\n        assert_eq!(term_scorer.doc(), 3);\n        assert_eq!(term_scorer.term_freq(), 12);\n        assert_nearly_equals!(term_scorer.score(), 1.3676447);\n        assert_eq!(term_scorer.advance(), 7);\n        assert_eq!(term_scorer.doc(), 7);\n        assert_eq!(term_scorer.term_freq(), 8);\n        assert_nearly_equals!(term_scorer.score(), 0.72015285);\n        assert_eq!(term_scorer.advance(), TERMINATED);\n        Ok(())\n    }\n\n    #[test]\n    fn test_term_scorer_shallow_advance() -> crate::Result<()> {\n        let bm25_weight = Bm25Weight::for_one_term(300, 1024, 10.0);\n        let mut doc_and_tfs = vec![];\n        for i in 0u32..300u32 {\n            let doc = i * 10;\n            doc_and_tfs.push((doc, 1u32 + doc % 3u32));\n        }\n        let fieldnorms: Vec<u32> = std::iter::repeat_n(10u32, 3_000).collect();\n        let mut term_scorer = TermScorer::create_for_test(&doc_and_tfs, &fieldnorms, bm25_weight);\n        assert_eq!(term_scorer.doc(), 0u32);\n        term_scorer.seek_block(1289);\n        assert_eq!(term_scorer.doc(), 0u32);\n        term_scorer.seek(1289);\n        assert_eq!(term_scorer.doc(), 1290);\n        Ok(())\n    }\n\n    proptest! {\n        #[test]\n        fn test_term_scorer_block_max_score(term_freqs_fieldnorms in proptest::collection::vec((1u32..10u32, 0u32..100u32), 80..300)) {\n        let term_doc_freq = term_freqs_fieldnorms.len();\n         let doc_tfs: Vec<(u32, u32)> = term_freqs_fieldnorms.iter()\n                   .cloned()\n                  .enumerate()\n                  .map(|(doc, (tf, _))| (doc as u32, tf))\n                  .collect();\n\n         let mut fieldnorms: Vec<u32> = vec![];\n         for item in term_freqs_fieldnorms.iter().take(term_doc_freq) {\n             let (tf, num_extra_terms) = item;\n             fieldnorms.push(tf + num_extra_terms);\n         }\n         let average_fieldnorm = fieldnorms\n             .iter()\n             .cloned()\n             .sum::<u32>() as Score / term_doc_freq as Score;\n             // Average fieldnorm is over the entire index,\n             // not necessarily the docs that are in the posting list.\n             // For this reason we multiply by 1.1 to make a realistic value.\n         let bm25_weight = Bm25Weight::for_one_term(term_doc_freq as u64,\n            term_doc_freq as u64 * 10u64,\n            average_fieldnorm);\n\n         let mut term_scorer =\n              TermScorer::create_for_test(&doc_tfs[..], &fieldnorms[..], bm25_weight);\n\n         let docs: Vec<DocId> = (0..term_doc_freq).map(|doc| doc as DocId).collect();\n         for block in docs.chunks(COMPRESSION_BLOCK_SIZE) {\n             let block_max_score: Score = term_scorer.block_max_score();\n             let mut block_max_score_computed: Score = 0.0;\n             for &doc in block {\n                assert_eq!(term_scorer.doc(), doc);\n                block_max_score_computed = block_max_score_computed.max(term_scorer.score());\n                term_scorer.advance();\n             }\n             assert_nearly_equals!(block_max_score_computed, block_max_score);\n         }\n        }\n    }\n\n    #[test]\n    fn test_block_wand() {\n        let mut doc_tfs: Vec<(u32, u32)> = vec![];\n        for doc in 0u32..128u32 {\n            doc_tfs.push((doc, 1u32));\n        }\n        for doc in 128u32..256u32 {\n            doc_tfs.push((doc, if doc == 200 { 2u32 } else { 1u32 }));\n        }\n        doc_tfs.push((256, 1u32));\n        doc_tfs.push((257, 3u32));\n        doc_tfs.push((258, 1u32));\n\n        let fieldnorms: Vec<u32> = std::iter::repeat_n(20u32, 300).collect();\n        let bm25_weight = Bm25Weight::for_one_term(10, 129, 20.0);\n        let mut docs = TermScorer::create_for_test(&doc_tfs[..], &fieldnorms[..], bm25_weight);\n        assert_nearly_equals!(docs.block_max_score(), 2.5161593);\n        docs.seek_block(135);\n        assert_nearly_equals!(docs.block_max_score(), 3.4597192);\n        docs.seek_block(256);\n        // the block is not loaded yet.\n        assert_nearly_equals!(docs.block_max_score(), 5.2971773);\n        assert_eq!(256, docs.seek(256));\n        assert_nearly_equals!(docs.block_max_score(), 3.9539647);\n    }\n\n    fn test_block_wand_aux(term_query: &TermQuery, searcher: &Searcher) -> crate::Result<()> {\n        let term_weight =\n            term_query.specialized_weight(EnableScoring::enabled_from_searcher(searcher))?;\n        for reader in searcher.segment_readers() {\n            let mut block_max_scores = vec![];\n            let mut block_max_scores_b = vec![];\n            let mut docs = vec![];\n            {\n                let mut term_scorer = term_weight.term_scorer_for_test(reader, 1.0)?.unwrap();\n                while term_scorer.doc() != TERMINATED {\n                    let mut score = term_scorer.score();\n                    docs.push(term_scorer.doc());\n                    for _ in 0..128 {\n                        score = score.max(term_scorer.score());\n                        if term_scorer.advance() == TERMINATED {\n                            break;\n                        }\n                    }\n                    block_max_scores.push(score);\n                }\n            }\n            {\n                let mut term_scorer = term_weight.term_scorer_for_test(reader, 1.0)?.unwrap();\n                for d in docs {\n                    term_scorer.seek_block(d);\n                    block_max_scores_b.push(term_scorer.block_max_score());\n                }\n            }\n            for (l, r) in block_max_scores\n                .iter()\n                .cloned()\n                .zip(block_max_scores_b.iter().cloned())\n            {\n                assert_nearly_equals!(l, r);\n            }\n        }\n        Ok(())\n    }\n\n    #[ignore]\n    #[test]\n    fn test_block_wand_long_test() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut writer: IndexWriter =\n            index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;\n        use rand::Rng;\n        let mut rng = rand::rng();\n        writer.set_merge_policy(Box::new(NoMergePolicy));\n        for _ in 0..3_000 {\n            let term_freq = rng.random_range(1..10000);\n            let words: Vec<&str> = std::iter::repeat_n(\"bbbb\", term_freq).collect();\n            let text = words.join(\" \");\n            writer.add_document(doc!(text_field=>text))?;\n        }\n        writer.commit()?;\n        let term_query = TermQuery::new(\n            Term::from_field_text(text_field, \"bbbb\"),\n            IndexRecordOption::WithFreqs,\n        );\n        let segment_ids: Vec<SegmentId>;\n        let reader = index.reader()?;\n        {\n            let searcher = reader.searcher();\n            segment_ids = searcher\n                .segment_readers()\n                .iter()\n                .map(|segment| segment.segment_id())\n                .collect();\n            test_block_wand_aux(&term_query, &searcher)?;\n        }\n        writer.merge(&segment_ids[..]).wait().unwrap();\n        {\n            reader.reload()?;\n            let searcher = reader.searcher();\n            assert_eq!(searcher.segment_readers().len(), 1);\n            test_block_wand_aux(&term_query, &searcher)?;\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/query/term_query/term_weight.rs",
    "content": "use super::term_scorer::TermScorer;\nuse crate::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN};\nuse crate::fieldnorm::FieldNormReader;\nuse crate::index::SegmentReader;\nuse crate::postings::SegmentPostings;\nuse crate::query::bm25::Bm25Weight;\nuse crate::query::explanation::does_not_match;\nuse crate::query::weight::{for_each_docset_buffered, for_each_scorer};\nuse crate::query::{AllScorer, AllWeight, EmptyScorer, Explanation, Scorer, Weight};\nuse crate::schema::IndexRecordOption;\nuse crate::{DocId, Score, TantivyError, Term};\n\npub struct TermWeight {\n    term: Term,\n    index_record_option: IndexRecordOption,\n    similarity_weight: Bm25Weight,\n    scoring_enabled: bool,\n}\n\nenum TermOrEmptyOrAllScorer {\n    TermScorer(Box<TermScorer>),\n    Empty,\n    AllMatch(AllScorer),\n}\n\nimpl TermOrEmptyOrAllScorer {\n    pub fn into_boxed_scorer(self) -> Box<dyn Scorer> {\n        match self {\n            TermOrEmptyOrAllScorer::TermScorer(scorer) => scorer,\n            TermOrEmptyOrAllScorer::Empty => Box::new(EmptyScorer),\n            TermOrEmptyOrAllScorer::AllMatch(scorer) => Box::new(scorer),\n        }\n    }\n}\n\nimpl Weight for TermWeight {\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {\n        Ok(self.specialized_scorer(reader, boost)?.into_boxed_scorer())\n    }\n\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {\n        match self.specialized_scorer(reader, 1.0)? {\n            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {\n                if term_scorer.doc() > doc || term_scorer.seek(doc) != doc {\n                    return Err(does_not_match(doc));\n                }\n                let mut explanation = term_scorer.explain();\n                explanation.add_context(format!(\"Term={:?}\", self.term,));\n                Ok(explanation)\n            }\n            TermOrEmptyOrAllScorer::Empty => Err(does_not_match(doc)),\n            TermOrEmptyOrAllScorer::AllMatch(_) => AllWeight.explain(reader, doc),\n        }\n    }\n\n    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {\n        if let Some(alive_bitset) = reader.alive_bitset() {\n            Ok(self.scorer(reader, 1.0)?.count(alive_bitset))\n        } else {\n            let field = self.term.field();\n            let inv_index = reader.inverted_index(field)?;\n            let term_info = inv_index.get_term_info(&self.term)?;\n            Ok(term_info.map(|term_info| term_info.doc_freq).unwrap_or(0))\n        }\n    }\n\n    /// Iterates through all of the document matched by the DocSet\n    /// `DocSet` and push the scored documents to the collector.\n    fn for_each(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score),\n    ) -> crate::Result<()> {\n        match self.specialized_scorer(reader, 1.0)? {\n            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {\n                for_each_scorer(&mut *term_scorer, callback);\n            }\n            TermOrEmptyOrAllScorer::Empty => {}\n            TermOrEmptyOrAllScorer::AllMatch(mut all_scorer) => {\n                for_each_scorer(&mut all_scorer, callback);\n            }\n        }\n        Ok(())\n    }\n\n    /// Iterates through all of the document matched by the DocSet\n    /// `DocSet` and push the scored documents to the collector.\n    fn for_each_no_score(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(&[DocId]),\n    ) -> crate::Result<()> {\n        match self.specialized_scorer(reader, 1.0)? {\n            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {\n                let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n                for_each_docset_buffered(&mut term_scorer, &mut buffer, callback);\n            }\n            TermOrEmptyOrAllScorer::Empty => {}\n            TermOrEmptyOrAllScorer::AllMatch(mut all_scorer) => {\n                let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n                for_each_docset_buffered(&mut all_scorer, &mut buffer, callback);\n            }\n        };\n\n        Ok(())\n    }\n\n    /// Calls `callback` with all of the `(doc, score)` for which score\n    /// is exceeding a given threshold.\n    ///\n    /// This method is useful for the TopDocs collector.\n    /// For all docsets, the blanket implementation has the benefit\n    /// of prefiltering (doc, score) pairs, avoiding the\n    /// virtual dispatch cost.\n    ///\n    /// More importantly, it makes it possible for scorers to implement\n    /// important optimization (e.g. BlockWAND for union).\n    fn for_each_pruning(\n        &self,\n        threshold: Score,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score) -> Score,\n    ) -> crate::Result<()> {\n        let specialized_scorer = self.specialized_scorer(reader, 1.0)?;\n        match specialized_scorer {\n            TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {\n                crate::query::boolean_query::block_wand_single_scorer(\n                    *term_scorer,\n                    threshold,\n                    callback,\n                );\n            }\n            TermOrEmptyOrAllScorer::Empty => {}\n            TermOrEmptyOrAllScorer::AllMatch(_) => {\n                return Err(TantivyError::InvalidArgument(\n                    \"for each pruning should only be called if scoring is enabled\".to_string(),\n                ));\n            }\n        }\n        Ok(())\n    }\n}\n\nimpl TermWeight {\n    pub fn new(\n        term: Term,\n        index_record_option: IndexRecordOption,\n        similarity_weight: Bm25Weight,\n        scoring_enabled: bool,\n    ) -> TermWeight {\n        TermWeight {\n            term,\n            index_record_option,\n            similarity_weight,\n            scoring_enabled,\n        }\n    }\n\n    pub fn term(&self) -> &Term {\n        &self.term\n    }\n\n    /// We need a method to access the actual `TermScorer` implementation\n    /// for `white box` test, checking in particular that the block max\n    /// is correct.\n    #[cfg(test)]\n    pub(crate) fn term_scorer_for_test(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<Option<TermScorer>> {\n        let scorer = self.specialized_scorer(reader, boost)?;\n        Ok(match scorer {\n            TermOrEmptyOrAllScorer::TermScorer(scorer) => Some(*scorer),\n            _ => None,\n        })\n    }\n\n    fn specialized_scorer(\n        &self,\n        reader: &SegmentReader,\n        boost: Score,\n    ) -> crate::Result<TermOrEmptyOrAllScorer> {\n        let field = self.term.field();\n        let inverted_index = reader.inverted_index(field)?;\n        let Some(term_info) = inverted_index.get_term_info(&self.term)? else {\n            // The term was not found.\n            return Ok(TermOrEmptyOrAllScorer::Empty);\n        };\n\n        // If we don't care about scores, and our posting lists matches all doc, we can return the\n        // AllMatch scorer.\n        if !self.scoring_enabled && term_info.doc_freq == reader.max_doc() {\n            return Ok(TermOrEmptyOrAllScorer::AllMatch(AllScorer::new(\n                reader.max_doc(),\n            )));\n        }\n\n        let segment_postings: SegmentPostings =\n            inverted_index.read_postings_from_terminfo(&term_info, self.index_record_option)?;\n\n        let fieldnorm_reader = self.fieldnorm_reader(reader)?;\n        let similarity_weight = self.similarity_weight.boost_by(boost);\n        Ok(TermOrEmptyOrAllScorer::TermScorer(Box::new(\n            TermScorer::new(segment_postings, fieldnorm_reader, similarity_weight),\n        )))\n    }\n\n    fn fieldnorm_reader(&self, segment_reader: &SegmentReader) -> crate::Result<FieldNormReader> {\n        if self.scoring_enabled {\n            if let Some(field_norm_reader) = segment_reader\n                .fieldnorms_readers()\n                .get_field(self.term.field())?\n            {\n                return Ok(field_norm_reader);\n            }\n        }\n        Ok(FieldNormReader::constant(segment_reader.max_doc(), 1))\n    }\n}\n"
  },
  {
    "path": "src/query/union/bitset_union.rs",
    "content": "use std::cell::RefCell;\n\nuse crate::docset::DocSet;\nuse crate::postings::Postings;\nuse crate::query::BitSetDocSet;\nuse crate::DocId;\n\n/// Creates a `Posting` that uses the bitset for hits and the docsets for PostingLists.\n///\n/// It is used for the regex phrase query, where we need the union of a large amount of\n/// terms, but need to keep the docsets for the postings.\npub struct BitSetPostingUnion<TDocSet> {\n    /// The docsets are required to load positions\n    ///\n    /// RefCell because we mutate in term_freq\n    docsets: RefCell<Vec<TDocSet>>,\n    /// The already unionized BitSet of the docsets\n    bitset: BitSetDocSet,\n}\n\nimpl<TDocSet: DocSet> BitSetPostingUnion<TDocSet> {\n    pub(crate) fn build(\n        docsets: Vec<TDocSet>,\n        bitset: BitSetDocSet,\n    ) -> BitSetPostingUnion<TDocSet> {\n        BitSetPostingUnion {\n            docsets: RefCell::new(docsets),\n            bitset,\n        }\n    }\n}\n\nimpl<TDocSet: Postings> Postings for BitSetPostingUnion<TDocSet> {\n    fn term_freq(&self) -> u32 {\n        let curr_doc = self.bitset.doc();\n        let mut term_freq = 0;\n        let mut docsets = self.docsets.borrow_mut();\n        for docset in docsets.iter_mut() {\n            if docset.doc() < curr_doc {\n                docset.seek(curr_doc);\n            }\n            if docset.doc() == curr_doc {\n                term_freq += docset.term_freq();\n            }\n        }\n        term_freq\n    }\n\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        let curr_doc = self.bitset.doc();\n        let mut docsets = self.docsets.borrow_mut();\n        for docset in docsets.iter_mut() {\n            if docset.doc() < curr_doc {\n                docset.seek(curr_doc);\n            }\n            if docset.doc() == curr_doc {\n                docset.append_positions_with_offset(offset, output);\n            }\n        }\n        debug_assert!(\n            !output.is_empty(),\n            \"this method should only be called if positions are available\"\n        );\n        output.sort_unstable();\n        output.dedup();\n    }\n}\n\nimpl<TDocSet: DocSet> DocSet for BitSetPostingUnion<TDocSet> {\n    fn advance(&mut self) -> DocId {\n        self.bitset.advance()\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.bitset.seek(target)\n    }\n\n    fn doc(&self) -> DocId {\n        self.bitset.doc()\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.bitset.size_hint()\n    }\n\n    fn count_including_deleted(&mut self) -> u32 {\n        self.bitset.count_including_deleted()\n    }\n}\n"
  },
  {
    "path": "src/query/union/buffered_union.rs",
    "content": "use common::TinySet;\n\nuse crate::docset::{DocSet, SeekDangerResult, TERMINATED};\nuse crate::query::score_combiner::{DoNothingCombiner, ScoreCombiner};\nuse crate::query::size_hint::estimate_union;\nuse crate::query::Scorer;\nuse crate::{DocId, Score};\n\n// The buffered union looks ahead within a fixed-size sliding window\n// of upcoming document IDs (the \"horizon\").\nconst HORIZON_NUM_TINYBITSETS: usize = HORIZON as usize / 64;\nconst HORIZON: u32 = 64u32 * 64u32;\n\n// `drain_filter` is not stable yet.\n// This function is similar except that it does is not unstable, and\n// it does not keep the original vector ordering.\n//\n// Elements are dropped and not yielded.\nfn unordered_drain_filter<T, P>(v: &mut Vec<T>, mut predicate: P)\nwhere P: FnMut(&mut T) -> bool {\n    let mut i = 0;\n    while i < v.len() {\n        if predicate(&mut v[i]) {\n            v.swap_remove(i);\n        } else {\n            i += 1;\n        }\n    }\n}\n\n/// Creates a `DocSet` that iterate through the union of two or more `DocSet`s.\npub struct BufferedUnionScorer<TScorer, TScoreCombiner = DoNothingCombiner> {\n    /// Active scorers (already filtered of `TERMINATED`).\n    docsets: Vec<TScorer>,\n    /// Sliding window presence map for upcoming docs.\n    ///\n    /// There are `HORIZON_NUM_TINYBITSETS` buckets, each covering\n    /// a span of 64 doc IDs. Bucket `i` represents the range\n    /// `[window_start_doc + i*64, window_start_doc + (i+1)*64)`.\n    bitsets: Box<[TinySet; HORIZON_NUM_TINYBITSETS]>,\n    // Index of the current TinySet bucket within the sliding window.\n    bucket_idx: usize,\n    /// Per-doc score combiners for the current window.\n    ///\n    /// these accumulators merge contributions from all scorers that\n    /// hit the same doc within the buffered window.\n    scores: Box<[TScoreCombiner; HORIZON as usize]>,\n    /// Start doc ID (inclusive) of the current sliding window.\n    window_start_doc: DocId,\n    /// Current doc ID of the union.\n    doc: DocId,\n    /// Combined score for current `doc` as produced by `TScoreCombiner`.\n    score: Score,\n    /// Number of documents in the segment.\n    num_docs: u32,\n}\n\nfn refill<TScorer: Scorer, TScoreCombiner: ScoreCombiner>(\n    scorers: &mut Vec<TScorer>,\n    bitsets: &mut [TinySet; HORIZON_NUM_TINYBITSETS],\n    score_combiner: &mut [TScoreCombiner; HORIZON as usize],\n    min_doc: DocId,\n) {\n    unordered_drain_filter(scorers, |scorer| {\n        let horizon = min_doc + HORIZON;\n        loop {\n            let doc = scorer.doc();\n            if doc >= horizon {\n                return false;\n            }\n            // add this document\n            let delta = doc - min_doc;\n            bitsets[(delta / 64) as usize].insert_mut(delta % 64u32);\n            score_combiner[delta as usize].update(scorer);\n            if scorer.advance() == TERMINATED {\n                // remove the docset, it has been entirely consumed.\n                return true;\n            }\n        }\n    });\n}\n\nimpl<TScorer: Scorer, TScoreCombiner: ScoreCombiner> BufferedUnionScorer<TScorer, TScoreCombiner> {\n    /// num_docs is the number of documents in the segment.\n    pub(crate) fn build(\n        docsets: Vec<TScorer>,\n        score_combiner_fn: impl FnOnce() -> TScoreCombiner,\n        num_docs: u32,\n    ) -> BufferedUnionScorer<TScorer, TScoreCombiner> {\n        let non_empty_docsets: Vec<TScorer> = docsets\n            .into_iter()\n            .filter(|docset| docset.doc() != TERMINATED)\n            .collect();\n        let mut union = BufferedUnionScorer {\n            docsets: non_empty_docsets,\n            bitsets: Box::new([TinySet::empty(); HORIZON_NUM_TINYBITSETS]),\n            scores: Box::new([score_combiner_fn(); HORIZON as usize]),\n            bucket_idx: HORIZON_NUM_TINYBITSETS,\n            window_start_doc: 0,\n            doc: 0,\n            score: 0.0,\n            num_docs,\n        };\n        if union.refill() {\n            union.advance();\n        } else {\n            union.doc = TERMINATED;\n        }\n        union\n    }\n\n    fn refill(&mut self) -> bool {\n        if let Some(min_doc) = self.docsets.iter().map(DocSet::doc).min() {\n            // Reset the sliding window to start at the smallest doc\n            // across all scorers and prebuffer within the horizon.\n            self.window_start_doc = min_doc;\n            self.bucket_idx = 0;\n            self.doc = min_doc;\n            refill(\n                &mut self.docsets,\n                &mut self.bitsets,\n                &mut self.scores,\n                min_doc,\n            );\n            true\n        } else {\n            false\n        }\n    }\n\n    #[inline]\n    fn advance_buffered(&mut self) -> bool {\n        while self.bucket_idx < HORIZON_NUM_TINYBITSETS {\n            if let Some(val) = self.bitsets[self.bucket_idx].pop_lowest() {\n                let delta = val + (self.bucket_idx as u32) * 64;\n                self.doc = self.window_start_doc + delta;\n                let score_combiner = &mut self.scores[delta as usize];\n                self.score = score_combiner.score();\n                score_combiner.clear();\n                return true;\n            } else {\n                self.bucket_idx += 1;\n            }\n        }\n        false\n    }\n\n    fn is_in_horizon(&self, target: DocId) -> bool {\n        // wrapping_sub, because target may be < window_start_doc\n        let gap = target.wrapping_sub(self.window_start_doc);\n        gap < HORIZON\n    }\n}\n\nimpl<TScorer, TScoreCombiner> DocSet for BufferedUnionScorer<TScorer, TScoreCombiner>\nwhere\n    TScorer: Scorer,\n    TScoreCombiner: ScoreCombiner,\n{\n    #[inline]\n    fn advance(&mut self) -> DocId {\n        if self.advance_buffered() {\n            return self.doc;\n        }\n        if !self.refill() {\n            self.doc = TERMINATED;\n            return TERMINATED;\n        }\n        if !self.advance_buffered() {\n            return TERMINATED;\n        }\n        self.doc\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        if self.doc >= target {\n            return self.doc;\n        }\n        let gap = target - self.window_start_doc;\n        if gap < HORIZON {\n            // Our value is within the buffered horizon.\n\n            // Skipping to corresponding bucket.\n            let new_bucket_idx = gap as usize / 64;\n            for obsolete_tinyset in &mut self.bitsets[self.bucket_idx..new_bucket_idx] {\n                obsolete_tinyset.clear();\n            }\n            for score_combiner in &mut self.scores[self.bucket_idx * 64..new_bucket_idx * 64] {\n                score_combiner.clear();\n            }\n            self.bucket_idx = new_bucket_idx;\n\n            // Advancing until we reach the end of the bucket\n            // or we reach a doc greater or equal to the target.\n            let mut doc = self.doc();\n            while doc < target {\n                doc = self.advance();\n            }\n            doc\n        } else {\n            // clear the buffered info.\n            for obsolete_tinyset in self.bitsets.iter_mut() {\n                *obsolete_tinyset = TinySet::empty();\n            }\n            for score_combiner in self.scores.iter_mut() {\n                score_combiner.clear();\n            }\n\n            // The target is outside of the buffered horizon.\n            // advance all docsets to a doc >= to the target.\n            unordered_drain_filter(&mut self.docsets, |docset| {\n                if docset.doc() < target {\n                    docset.seek(target);\n                }\n                docset.doc() == TERMINATED\n            });\n\n            // at this point all of the docsets\n            // are positioned on a doc >= to the target.\n            if !self.refill() {\n                self.doc = TERMINATED;\n                return TERMINATED;\n            }\n            self.advance()\n        }\n    }\n\n    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {\n        if target >= TERMINATED {\n            return SeekDangerResult::SeekLowerBound(TERMINATED);\n        }\n        if self.is_in_horizon(target) {\n            // Our value is within the buffered horizon and the docset may already have been\n            // processed and removed, so we need to use seek, which uses the regular advance.\n            let seek_doc = self.seek(target);\n            if seek_doc == target {\n                return SeekDangerResult::Found;\n            } else {\n                return SeekDangerResult::SeekLowerBound(seek_doc);\n            };\n        }\n\n        // The docsets are not in the buffered range, so we can use seek_into_the_danger_zone\n        // of the underlying docsets\n        let mut is_hit = false;\n        let mut min_new_target = TERMINATED;\n\n        for docset in self.docsets.iter_mut() {\n            match docset.seek_danger(target) {\n                SeekDangerResult::Found => {\n                    is_hit = true;\n                    break;\n                }\n                SeekDangerResult::SeekLowerBound(new_target) => {\n                    min_new_target = min_new_target.min(new_target);\n                }\n            }\n        }\n\n        // The API requires the DocSet to be in a valid state when `seek_into_the_danger_zone`\n        // returns Found.\n        if is_hit {\n            // The doc is found. Let's make sure we position the union on the target\n            // to bring it back to a valid state.\n            self.seek(target);\n            SeekDangerResult::Found\n        } else {\n            SeekDangerResult::SeekLowerBound(min_new_target)\n        }\n    }\n\n    #[inline]\n    fn doc(&self) -> DocId {\n        self.doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        estimate_union(self.docsets.iter().map(DocSet::size_hint), self.num_docs)\n    }\n\n    fn cost(&self) -> u64 {\n        self.docsets.iter().map(|docset| docset.cost()).sum()\n    }\n\n    // TODO Also implement `count` with deletes efficiently.\n    fn count_including_deleted(&mut self) -> u32 {\n        if self.doc == TERMINATED {\n            return 0;\n        }\n        let mut count = self.bitsets[self.bucket_idx..HORIZON_NUM_TINYBITSETS]\n            .iter()\n            .map(|bitset| bitset.len())\n            .sum::<u32>()\n            + 1;\n        for bitset in self.bitsets.iter_mut() {\n            bitset.clear();\n        }\n        while self.refill() {\n            count += self.bitsets.iter().map(|bitset| bitset.len()).sum::<u32>();\n            for bitset in self.bitsets.iter_mut() {\n                bitset.clear();\n            }\n        }\n        self.bucket_idx = HORIZON_NUM_TINYBITSETS;\n        count\n    }\n}\n\nimpl<TScorer, TScoreCombiner> Scorer for BufferedUnionScorer<TScorer, TScoreCombiner>\nwhere\n    TScoreCombiner: ScoreCombiner,\n    TScorer: Scorer,\n{\n    #[inline]\n    fn score(&mut self) -> Score {\n        self.score\n    }\n}\n"
  },
  {
    "path": "src/query/union/mod.rs",
    "content": "mod bitset_union;\nmod buffered_union;\nmod simple_union;\n\npub use bitset_union::BitSetPostingUnion;\npub use buffered_union::BufferedUnionScorer;\npub use simple_union::SimpleUnion;\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::BTreeSet;\n\n    use common::BitSet;\n\n    use super::{SimpleUnion, *};\n    use crate::docset::{DocSet, SeekDangerResult, TERMINATED};\n    use crate::postings::tests::test_skip_against_unoptimized;\n    use crate::query::score_combiner::DoNothingCombiner;\n    use crate::query::union::bitset_union::BitSetPostingUnion;\n    use crate::query::{BitSetDocSet, ConstScorer, VecDocSet};\n    use crate::{tests, DocId};\n\n    fn vec_doc_set_from_docs_list(\n        docs_list: &[Vec<DocId>],\n    ) -> impl Iterator<Item = VecDocSet> + '_ {\n        docs_list.iter().cloned().map(VecDocSet::from)\n    }\n    fn union_from_docs_list(docs_list: &[Vec<DocId>]) -> Box<dyn DocSet> {\n        let max_doc = docs_list\n            .iter()\n            .flat_map(|docs| docs.iter().copied())\n            .max()\n            .unwrap_or(0);\n        Box::new(BufferedUnionScorer::build(\n            vec_doc_set_from_docs_list(docs_list)\n                .map(|docset| ConstScorer::new(docset, 1.0))\n                .collect::<Vec<ConstScorer<VecDocSet>>>(),\n            DoNothingCombiner::default,\n            max_doc,\n        ))\n    }\n\n    fn posting_list_union_from_docs_list(docs_list: &[Vec<DocId>]) -> Box<dyn DocSet> {\n        Box::new(BitSetPostingUnion::build(\n            vec_doc_set_from_docs_list(docs_list).collect::<Vec<VecDocSet>>(),\n            bitset_from_docs_list(docs_list),\n        ))\n    }\n    fn simple_union_from_docs_list(docs_list: &[Vec<DocId>]) -> Box<dyn DocSet> {\n        Box::new(SimpleUnion::build(\n            vec_doc_set_from_docs_list(docs_list).collect::<Vec<VecDocSet>>(),\n        ))\n    }\n    fn bitset_from_docs_list(docs_list: &[Vec<DocId>]) -> BitSetDocSet {\n        let max_doc = docs_list\n            .iter()\n            .flat_map(|docs| docs.iter().copied())\n            .max()\n            .unwrap_or(0);\n        let mut doc_bitset = BitSet::with_max_value(max_doc + 1);\n        for docs in docs_list {\n            for &doc in docs {\n                doc_bitset.insert(doc);\n            }\n        }\n        BitSetDocSet::from(doc_bitset)\n    }\n    fn aux_test_union(docs_list: &[Vec<DocId>]) {\n        for constructor in [\n            posting_list_union_from_docs_list,\n            simple_union_from_docs_list,\n            union_from_docs_list,\n        ] {\n            aux_test_union_with_constructor(constructor, docs_list);\n        }\n    }\n    fn aux_test_union_with_constructor<F>(constructor: F, docs_list: &[Vec<DocId>])\n    where F: Fn(&[Vec<DocId>]) -> Box<dyn DocSet> {\n        let mut val_set: BTreeSet<u32> = BTreeSet::new();\n        for vs in docs_list {\n            for &v in vs {\n                val_set.insert(v);\n            }\n        }\n        let union_vals: Vec<u32> = val_set.into_iter().collect();\n        let mut union_expected = VecDocSet::from(union_vals);\n        let make_union = || constructor(docs_list);\n        let mut union = make_union();\n        let mut count = 0;\n        while union.doc() != TERMINATED {\n            assert_eq!(union_expected.doc(), union.doc());\n            assert_eq!(union_expected.advance(), union.advance());\n            count += 1;\n        }\n        assert_eq!(union_expected.advance(), TERMINATED);\n        assert_eq!(count, make_union().count_including_deleted());\n    }\n\n    use proptest::prelude::*;\n\n    proptest! {\n        #[test]\n        fn test_union_is_same(vecs in prop::collection::vec(\n            prop::collection::vec(0u32..100, 1..10)\n                .prop_map(|mut inner| {\n                    inner.sort_unstable();\n                    inner.dedup();\n                    inner\n                }),\n            1..10\n        ),\n        seek_docids in prop::collection::vec(0u32..100, 0..10).prop_map(|mut inner| {\n            inner.sort_unstable();\n            inner\n        })) {\n            test_docid_with_skip(&vecs, &seek_docids);\n        }\n    }\n\n    fn test_docid_with_skip(vecs: &[Vec<DocId>], skip_targets: &[DocId]) {\n        let mut union1 = posting_list_union_from_docs_list(vecs);\n        let mut union2 = simple_union_from_docs_list(vecs);\n        let mut union3 = union_from_docs_list(vecs);\n\n        // Check initial sequential advance\n        while union1.doc() != TERMINATED {\n            assert_eq!(union1.doc(), union2.doc());\n            assert_eq!(union1.doc(), union3.doc());\n            assert_eq!(union1.advance(), union2.advance());\n            assert_eq!(union1.doc(), union3.advance());\n        }\n\n        // Reset and test seek functionality\n        let mut union1 = posting_list_union_from_docs_list(vecs);\n        let mut union2 = simple_union_from_docs_list(vecs);\n        let mut union3 = union_from_docs_list(vecs);\n\n        for &seek_docid in skip_targets {\n            union1.seek(seek_docid);\n            union2.seek(seek_docid);\n            union3.seek(seek_docid);\n\n            // Verify that all unions have the same document after seeking\n            assert_eq!(union3.doc(), union1.doc());\n            assert_eq!(union3.doc(), union2.doc());\n        }\n    }\n\n    #[test]\n    fn test_union() {\n        aux_test_union(&[\n            vec![1, 3333, 100000000u32],\n            vec![1, 2, 100000000u32],\n            vec![1, 2, 100000000u32],\n            vec![],\n        ]);\n        aux_test_union(&[\n            vec![1, 3333, 100000000u32],\n            vec![1, 2, 100000000u32],\n            vec![1, 2, 100000000u32],\n            vec![],\n        ]);\n        aux_test_union(&[\n            tests::sample_with_seed(100_000, 0.01, 1),\n            tests::sample_with_seed(100_000, 0.05, 2),\n            tests::sample_with_seed(100_000, 0.001, 3),\n        ]);\n    }\n\n    fn test_aux_union_skip(docs_list: &[Vec<DocId>], skip_targets: Vec<DocId>) {\n        for constructor in [\n            posting_list_union_from_docs_list,\n            simple_union_from_docs_list,\n            union_from_docs_list,\n        ] {\n            test_aux_union_skip_with_constructor(constructor, docs_list, skip_targets.clone());\n        }\n    }\n    fn test_aux_union_skip_with_constructor<F>(\n        constructor: F,\n        docs_list: &[Vec<DocId>],\n        skip_targets: Vec<DocId>,\n    ) where\n        F: Fn(&[Vec<DocId>]) -> Box<dyn DocSet>,\n    {\n        let mut btree_set = BTreeSet::new();\n        for docs in docs_list {\n            btree_set.extend(docs.iter().cloned());\n        }\n        let docset_factory = || {\n            let res: Box<dyn DocSet> = constructor(docs_list);\n            res\n        };\n        let mut docset = constructor(docs_list);\n        for el in btree_set {\n            assert_eq!(el, docset.doc());\n            docset.advance();\n        }\n        assert_eq!(docset.doc(), TERMINATED);\n        test_skip_against_unoptimized(docset_factory, skip_targets);\n    }\n\n    #[test]\n    fn test_union_skip_corner_case() {\n        test_aux_union_skip(&[vec![165132, 167382], vec![25029, 25091]], vec![25029]);\n    }\n\n    #[test]\n    fn test_union_skip_corner_case2() {\n        test_aux_union_skip(\n            &[vec![1u32, 1u32 + 100], vec![2u32, 1000u32, 10_000u32]],\n            vec![0u32, 1u32, 2u32, 3u32, 1u32 + 100, 2u32 + 100],\n        );\n    }\n\n    #[test]\n    fn test_union_skip_corner_case3() {\n        let mut docset = posting_list_union_from_docs_list(&[vec![0u32, 5u32], vec![1u32, 4u32]]);\n        assert_eq!(docset.doc(), 0u32);\n        assert_eq!(docset.seek(0u32), 0u32);\n        assert_eq!(docset.seek(0u32), 0u32);\n        assert_eq!(docset.doc(), 0u32)\n    }\n\n    #[test]\n    fn test_union_skip_random() {\n        test_aux_union_skip(\n            &[\n                vec![1, 2, 3, 7],\n                vec![1, 3, 9, 10000],\n                vec![1, 3, 8, 9, 100],\n            ],\n            vec![1, 2, 3, 5, 6, 7, 8, 100],\n        );\n        test_aux_union_skip(\n            &[\n                tests::sample_with_seed(100_000, 0.001, 1),\n                tests::sample_with_seed(100_000, 0.002, 2),\n                tests::sample_with_seed(100_000, 0.005, 3),\n            ],\n            tests::sample_with_seed(100_000, 0.01, 4),\n        );\n    }\n\n    #[test]\n    fn test_union_skip_specific() {\n        test_aux_union_skip(\n            &[\n                vec![1, 2, 3, 7],\n                vec![1, 3, 9, 10000],\n                vec![1, 3, 8, 9, 100],\n            ],\n            vec![1, 2, 3, 7, 8, 9, 99, 100, 101, 500, 20000],\n        );\n    }\n\n    #[test]\n    fn test_buffered_union_seek_into_danger_zone_terminated() {\n        let scorer1 = ConstScorer::new(VecDocSet::from(vec![1, 2]), 1.0);\n        let scorer2 = ConstScorer::new(VecDocSet::from(vec![2, 3]), 1.0);\n\n        let mut union_scorer =\n            BufferedUnionScorer::build(vec![scorer1, scorer2], DoNothingCombiner::default, 100);\n\n        // Advance to end\n        while union_scorer.doc() != TERMINATED {\n            union_scorer.advance();\n        }\n\n        assert_eq!(union_scorer.doc(), TERMINATED);\n\n        assert_eq!(\n            union_scorer.seek_danger(TERMINATED),\n            SeekDangerResult::SeekLowerBound(TERMINATED)\n        );\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use test::Bencher;\n\n    use crate::query::score_combiner::DoNothingCombiner;\n    use crate::query::{BufferedUnionScorer, ConstScorer, VecDocSet};\n    use crate::{tests, DocId, DocSet, TERMINATED};\n\n    #[bench]\n    fn bench_union_3_high(bench: &mut Bencher) {\n        let union_docset: Vec<Vec<DocId>> = vec![\n            tests::sample_with_seed(100_000, 0.1, 0),\n            tests::sample_with_seed(100_000, 0.2, 1),\n        ];\n        bench.iter(|| {\n            let mut v = BufferedUnionScorer::build(\n                union_docset\n                    .iter()\n                    .map(|doc_ids| VecDocSet::from(doc_ids.clone()))\n                    .map(|docset| ConstScorer::new(docset, 1.0))\n                    .collect::<Vec<_>>(),\n                DoNothingCombiner::default,\n                100_000,\n            );\n            while v.doc() != TERMINATED {\n                v.advance();\n            }\n        });\n    }\n    #[bench]\n    fn bench_union_3_low(bench: &mut Bencher) {\n        let union_docset: Vec<Vec<DocId>> = vec![\n            tests::sample_with_seed(100_000, 0.01, 0),\n            tests::sample_with_seed(100_000, 0.05, 1),\n            tests::sample_with_seed(100_000, 0.001, 2),\n        ];\n        bench.iter(|| {\n            let mut v = BufferedUnionScorer::build(\n                union_docset\n                    .iter()\n                    .map(|doc_ids| VecDocSet::from(doc_ids.clone()))\n                    .map(|docset| ConstScorer::new(docset, 1.0))\n                    .collect::<Vec<_>>(),\n                DoNothingCombiner::default,\n                100_000,\n            );\n            while v.doc() != TERMINATED {\n                v.advance();\n            }\n        });\n    }\n}\n"
  },
  {
    "path": "src/query/union/simple_union.rs",
    "content": "use crate::docset::{DocSet, TERMINATED};\nuse crate::postings::Postings;\nuse crate::DocId;\n\n/// A `SimpleUnion` is a `DocSet` that is the union of multiple `DocSet`.\n/// Unlike `BufferedUnion`, it doesn't do any horizon precomputation.\n///\n/// For that reason SimpleUnion is a good choice for queries that skip a lot.\npub struct SimpleUnion<TDocSet> {\n    docsets: Vec<TDocSet>,\n    doc: DocId,\n}\n\nimpl<TDocSet: DocSet> SimpleUnion<TDocSet> {\n    pub(crate) fn build(mut docsets: Vec<TDocSet>) -> SimpleUnion<TDocSet> {\n        docsets.retain(|docset| docset.doc() != TERMINATED);\n        let mut docset = SimpleUnion { docsets, doc: 0 };\n\n        docset.initialize_first_doc_id();\n\n        docset\n    }\n\n    fn initialize_first_doc_id(&mut self) {\n        let mut next_doc = TERMINATED;\n\n        for docset in &self.docsets {\n            next_doc = next_doc.min(docset.doc());\n        }\n        self.doc = next_doc;\n    }\n\n    fn advance_to_next(&mut self) -> DocId {\n        let mut next_doc = TERMINATED;\n\n        for docset in &mut self.docsets {\n            if docset.doc() <= self.doc {\n                docset.advance();\n            }\n            next_doc = next_doc.min(docset.doc());\n        }\n        self.doc = next_doc;\n        self.doc\n    }\n}\n\nimpl<TDocSet: Postings> Postings for SimpleUnion<TDocSet> {\n    fn term_freq(&self) -> u32 {\n        let mut term_freq = 0;\n        for docset in &self.docsets {\n            let doc = docset.doc();\n            if doc == self.doc {\n                term_freq += docset.term_freq();\n            }\n        }\n        term_freq\n    }\n\n    fn append_positions_with_offset(&mut self, offset: u32, output: &mut Vec<u32>) {\n        for docset in &mut self.docsets {\n            let doc = docset.doc();\n            if doc == self.doc {\n                docset.append_positions_with_offset(offset, output);\n            }\n        }\n        output.sort_unstable();\n        output.dedup();\n    }\n}\n\nimpl<TDocSet: DocSet> DocSet for SimpleUnion<TDocSet> {\n    fn advance(&mut self) -> DocId {\n        self.advance_to_next();\n        self.doc\n    }\n\n    fn seek(&mut self, target: DocId) -> DocId {\n        self.doc = TERMINATED;\n        for docset in &mut self.docsets {\n            if docset.doc() < target {\n                docset.seek(target);\n            }\n            if docset.doc() < self.doc {\n                self.doc = docset.doc();\n            }\n        }\n        self.doc\n    }\n\n    fn doc(&self) -> DocId {\n        self.doc\n    }\n\n    fn size_hint(&self) -> u32 {\n        // TODO: use estimate_union\n        self.docsets\n            .iter()\n            .map(|docset| docset.size_hint())\n            .max()\n            .unwrap_or(0u32)\n    }\n\n    fn cost(&self) -> u64 {\n        self.docsets.iter().map(|docset| docset.cost()).sum()\n    }\n\n    fn count_including_deleted(&mut self) -> u32 {\n        if self.doc == TERMINATED {\n            return 0u32;\n        }\n        let mut count = 1u32;\n        while self.advance_to_next() != TERMINATED {\n            count += 1;\n        }\n        count\n    }\n}\n"
  },
  {
    "path": "src/query/vec_docset.rs",
    "content": "#![allow(dead_code)]\n\nuse common::HasLen;\n\nuse crate::docset::{DocSet, TERMINATED};\nuse crate::DocId;\n\n/// Simulate a `Postings` objects from a `VecPostings`.\n/// `VecPostings` only exist for testing purposes.\n///\n/// Term frequencies always return 1.\n/// No positions are returned.\npub struct VecDocSet {\n    doc_ids: Vec<DocId>,\n    cursor: usize,\n}\n\nimpl From<Vec<DocId>> for VecDocSet {\n    fn from(doc_ids: Vec<DocId>) -> VecDocSet {\n        // We do not use `slice::is_sorted`, as we want to check for doc ids to be strictly\n        // sorted.\n        assert!(doc_ids.windows(2).all(|w| w[0] < w[1]));\n        VecDocSet { doc_ids, cursor: 0 }\n    }\n}\n\nimpl DocSet for VecDocSet {\n    fn advance(&mut self) -> DocId {\n        self.cursor += 1;\n        if self.cursor >= self.doc_ids.len() {\n            self.cursor = self.doc_ids.len();\n            return TERMINATED;\n        }\n        self.doc()\n    }\n\n    fn doc(&self) -> DocId {\n        if self.cursor == self.doc_ids.len() {\n            return TERMINATED;\n        }\n        self.doc_ids[self.cursor]\n    }\n\n    fn size_hint(&self) -> u32 {\n        self.len() as u32\n    }\n}\n\nimpl HasLen for VecDocSet {\n    fn len(&self) -> usize {\n        self.doc_ids.len()\n    }\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use super::*;\n    use crate::docset::COLLECT_BLOCK_BUFFER_LEN;\n\n    #[test]\n    pub fn test_vec_postings() {\n        let doc_ids: Vec<DocId> = (0u32..1024u32).map(|e| e * 3).collect();\n        let mut postings = VecDocSet::from(doc_ids);\n        assert_eq!(postings.doc(), 0u32);\n        assert_eq!(postings.advance(), 3u32);\n        assert_eq!(postings.doc(), 3u32);\n        assert_eq!(postings.seek(14u32), 15u32);\n        assert_eq!(postings.doc(), 15u32);\n        assert_eq!(postings.seek(300u32), 300u32);\n        assert_eq!(postings.doc(), 300u32);\n        assert_eq!(postings.seek(6000u32), TERMINATED);\n    }\n\n    #[test]\n    pub fn test_fill_buffer() {\n        let doc_ids: Vec<DocId> = (1u32..=(COLLECT_BLOCK_BUFFER_LEN as u32 * 2 + 9)).collect();\n        let mut postings = VecDocSet::from(doc_ids);\n        let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n        assert_eq!(postings.fill_buffer(&mut buffer), COLLECT_BLOCK_BUFFER_LEN);\n        for i in 0u32..COLLECT_BLOCK_BUFFER_LEN as u32 {\n            assert_eq!(buffer[i as usize], i + 1);\n        }\n        assert_eq!(postings.fill_buffer(&mut buffer), COLLECT_BLOCK_BUFFER_LEN);\n        for i in 0u32..COLLECT_BLOCK_BUFFER_LEN as u32 {\n            assert_eq!(buffer[i as usize], i + 1 + COLLECT_BLOCK_BUFFER_LEN as u32);\n        }\n        assert_eq!(postings.fill_buffer(&mut buffer), 9);\n    }\n}\n"
  },
  {
    "path": "src/query/weight.rs",
    "content": "use super::Scorer;\nuse crate::docset::COLLECT_BLOCK_BUFFER_LEN;\nuse crate::index::SegmentReader;\nuse crate::query::Explanation;\nuse crate::{DocId, DocSet, Score, TERMINATED};\n\n/// Iterates through all of the documents and scores matched by the DocSet\n/// `DocSet`.\npub(crate) fn for_each_scorer<TScorer: Scorer + ?Sized>(\n    scorer: &mut TScorer,\n    callback: &mut dyn FnMut(DocId, Score),\n) {\n    let mut doc = scorer.doc();\n    while doc != TERMINATED {\n        callback(doc, scorer.score());\n        doc = scorer.advance();\n    }\n}\n\n/// Iterates through all of the documents matched by the DocSet\n/// `DocSet`.\n#[inline]\npub(crate) fn for_each_docset_buffered<T: DocSet + ?Sized>(\n    docset: &mut T,\n    buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN],\n    mut callback: impl FnMut(&[DocId]),\n) {\n    loop {\n        let num_items = docset.fill_buffer(buffer);\n        callback(&buffer[..num_items]);\n        if num_items != buffer.len() {\n            break;\n        }\n    }\n}\n\n/// Calls `callback` with all of the `(doc, score)` for which score\n/// is exceeding a given threshold.\n///\n/// This method is useful for the [`TopDocs`](crate::collector::TopDocs) collector.\n/// For all docsets, the blanket implementation has the benefit\n/// of prefiltering (doc, score) pairs, avoiding the\n/// virtual dispatch cost.\n///\n/// More importantly, it makes it possible for scorers to implement\n/// important optimization (e.g. BlockWAND for union).\npub(crate) fn for_each_pruning_scorer<TScorer: Scorer + ?Sized>(\n    scorer: &mut TScorer,\n    mut threshold: Score,\n    callback: &mut dyn FnMut(DocId, Score) -> Score,\n) {\n    let mut doc = scorer.doc();\n    while doc != TERMINATED {\n        let score = scorer.score();\n        if score > threshold {\n            threshold = callback(doc, score);\n        }\n        doc = scorer.advance();\n    }\n}\n\n/// A Weight is the specialization of a `Query`\n/// for a given set of segments.\n///\n/// See [`Query`](crate::query::Query).\npub trait Weight: Send + Sync + 'static {\n    /// Returns the scorer for the given segment.\n    ///\n    /// `boost` is a multiplier to apply to the score.\n    ///\n    /// See [`Query`](crate::query::Query).\n    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>>;\n\n    /// Returns an [`Explanation`] for the given document.\n    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation>;\n\n    /// Returns the number documents within the given [`SegmentReader`].\n    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        if let Some(alive_bitset) = reader.alive_bitset() {\n            Ok(scorer.count(alive_bitset))\n        } else {\n            Ok(scorer.count_including_deleted())\n        }\n    }\n\n    /// Iterates through all of the document matched by the DocSet\n    /// `DocSet` and push the scored documents to the collector.\n    fn for_each(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score),\n    ) -> crate::Result<()> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        for_each_scorer(scorer.as_mut(), callback);\n        Ok(())\n    }\n\n    /// Iterates through all of the document matched by the DocSet\n    /// `DocSet` and push the scored documents to the collector.\n    fn for_each_no_score(\n        &self,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(&[DocId]),\n    ) -> crate::Result<()> {\n        let mut docset = self.scorer(reader, 1.0)?;\n\n        let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];\n        for_each_docset_buffered(&mut docset, &mut buffer, callback);\n        Ok(())\n    }\n\n    /// Calls `callback` with all of the `(doc, score)` for which score\n    /// is exceeding a given threshold.\n    ///\n    /// This method is useful for the [`TopDocs`](crate::collector::TopDocs) collector.\n    /// For all docsets, the blanket implementation has the benefit\n    /// of prefiltering (doc, score) pairs, avoiding the\n    /// virtual dispatch cost.\n    ///\n    /// More importantly, it makes it possible for scorers to implement\n    /// important optimization (e.g. BlockWAND for union).\n    fn for_each_pruning(\n        &self,\n        threshold: Score,\n        reader: &SegmentReader,\n        callback: &mut dyn FnMut(DocId, Score) -> Score,\n    ) -> crate::Result<()> {\n        let mut scorer = self.scorer(reader, 1.0)?;\n        for_each_pruning_scorer(scorer.as_mut(), threshold, callback);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/reader/mod.rs",
    "content": "mod warming;\n\nuse std::sync::atomic::AtomicU64;\nuse std::sync::{atomic, Arc, Weak};\n\nuse arc_swap::ArcSwap;\npub use warming::Warmer;\n\nuse self::warming::WarmingState;\nuse crate::core::searcher::{SearcherGeneration, SearcherInner};\nuse crate::directory::{Directory, WatchCallback, WatchHandle, META_LOCK};\nuse crate::store::DOCSTORE_CACHE_CAPACITY;\nuse crate::{Index, Inventory, Searcher, SegmentReader, TrackedObject};\n\n/// Defines when a new version of the index should be reloaded.\n///\n/// Regardless of whether you search and index in the same process, tantivy does not necessarily\n/// reflects the change that are committed to your index. `ReloadPolicy` precisely helps you define\n/// when you want your index to be reloaded.\n#[derive(Clone, Copy)]\npub enum ReloadPolicy {\n    /// The index is entirely reloaded manually.\n    /// All updates of the index should be manual.\n    ///\n    /// No change is reflected automatically. You are required to call [`IndexReader::reload()`]\n    /// manually.\n    Manual,\n    /// The index is reloaded within milliseconds after a new commit is available.\n    /// This is made possible by watching changes in the `meta.json` file.\n    OnCommitWithDelay, // TODO add NEAR_REAL_TIME(target_ms)\n}\n\n/// [`IndexReader`] builder\n///\n/// It makes it possible to configure:\n/// - [`ReloadPolicy`] defining when new index versions are detected\n/// - [`Warmer`] implementations\n/// - number of warming threads, for parallelizing warming work\n/// - The cache size of the underlying doc store readers.\n#[derive(Clone)]\npub struct IndexReaderBuilder {\n    reload_policy: ReloadPolicy,\n    index: Index,\n    warmers: Vec<Weak<dyn Warmer>>,\n    num_warming_threads: usize,\n    doc_store_cache_num_blocks: usize,\n}\n\nimpl IndexReaderBuilder {\n    #[must_use]\n    pub(crate) fn new(index: Index) -> IndexReaderBuilder {\n        IndexReaderBuilder {\n            reload_policy: ReloadPolicy::OnCommitWithDelay,\n            index,\n            warmers: Vec::new(),\n            num_warming_threads: 1,\n            doc_store_cache_num_blocks: DOCSTORE_CACHE_CAPACITY,\n        }\n    }\n\n    /// Builds the reader.\n    ///\n    /// Building the reader is a non-trivial operation that requires\n    /// to open different segment readers. It may take hundreds of milliseconds\n    /// of time and it may return an error.\n    pub fn try_into(self) -> crate::Result<IndexReader> {\n        let searcher_generation_inventory = Inventory::default();\n        let warming_state = WarmingState::new(\n            self.num_warming_threads,\n            self.warmers,\n            searcher_generation_inventory.clone(),\n        )?;\n        let inner_reader = InnerIndexReader::new(\n            self.doc_store_cache_num_blocks,\n            self.index,\n            warming_state,\n            searcher_generation_inventory,\n        )?;\n        let inner_reader_arc = Arc::new(inner_reader);\n        let watch_handle_opt: Option<WatchHandle> = match self.reload_policy {\n            ReloadPolicy::Manual => {\n                // No need to set anything...\n                None\n            }\n            ReloadPolicy::OnCommitWithDelay => {\n                let inner_reader_arc_clone = inner_reader_arc.clone();\n                let callback = move || {\n                    if let Err(err) = inner_reader_arc_clone.reload() {\n                        error!(\"Error while loading searcher after commit was detected. {err:?}\");\n                    }\n                };\n                let watch_handle = inner_reader_arc\n                    .index\n                    .directory()\n                    .watch(WatchCallback::new(callback))?;\n                Some(watch_handle)\n            }\n        };\n        Ok(IndexReader {\n            inner: inner_reader_arc,\n            _watch_handle_opt: watch_handle_opt,\n        })\n    }\n\n    /// Sets the reload_policy.\n    ///\n    /// See [`ReloadPolicy`] for more details.\n    #[must_use]\n    pub fn reload_policy(mut self, reload_policy: ReloadPolicy) -> IndexReaderBuilder {\n        self.reload_policy = reload_policy;\n        self\n    }\n\n    /// Sets the cache size of the doc store readers.\n    ///\n    /// The doc store readers cache by default DOCSTORE_CACHE_CAPACITY(100) decompressed blocks.\n    #[must_use]\n    pub fn doc_store_cache_num_blocks(\n        mut self,\n        doc_store_cache_num_blocks: usize,\n    ) -> IndexReaderBuilder {\n        self.doc_store_cache_num_blocks = doc_store_cache_num_blocks;\n        self\n    }\n\n    /// Set the [`Warmer`]s that are invoked when reloading searchable segments.\n    #[must_use]\n    pub fn warmers(mut self, warmers: Vec<Weak<dyn Warmer>>) -> IndexReaderBuilder {\n        self.warmers = warmers;\n        self\n    }\n\n    /// Sets the number of warming threads.\n    ///\n    /// This allows parallelizing warming work when there are multiple [`Warmer`] registered with\n    /// the [`IndexReader`].\n    #[must_use]\n    pub fn num_warming_threads(mut self, num_warming_threads: usize) -> IndexReaderBuilder {\n        self.num_warming_threads = num_warming_threads;\n        self\n    }\n}\n\nimpl TryInto<IndexReader> for IndexReaderBuilder {\n    type Error = crate::TantivyError;\n\n    fn try_into(self) -> crate::Result<IndexReader> {\n        IndexReaderBuilder::try_into(self)\n    }\n}\n\nstruct InnerIndexReader {\n    doc_store_cache_num_blocks: usize,\n    index: Index,\n    warming_state: WarmingState,\n    searcher: arc_swap::ArcSwap<SearcherInner>,\n    searcher_generation_counter: Arc<AtomicU64>,\n    searcher_generation_inventory: Inventory<SearcherGeneration>,\n}\n\nimpl InnerIndexReader {\n    fn new(\n        doc_store_cache_num_blocks: usize,\n        index: Index,\n        warming_state: WarmingState,\n        // The searcher_generation_inventory is not used as source, but as target to track the\n        // loaded segments.\n        searcher_generation_inventory: Inventory<SearcherGeneration>,\n    ) -> crate::Result<Self> {\n        let searcher_generation_counter: Arc<AtomicU64> = Default::default();\n\n        let searcher = Self::create_searcher(\n            &index,\n            doc_store_cache_num_blocks,\n            &warming_state,\n            &searcher_generation_counter,\n            &searcher_generation_inventory,\n        )?;\n        Ok(InnerIndexReader {\n            doc_store_cache_num_blocks,\n            index,\n            warming_state,\n            searcher: ArcSwap::from(searcher),\n            searcher_generation_counter,\n            searcher_generation_inventory,\n        })\n    }\n    /// Opens the freshest segments [`SegmentReader`].\n    ///\n    /// This function acquires a lock to prevent GC from removing files\n    /// as we are opening our index.\n    fn open_segment_readers(index: &Index) -> crate::Result<Vec<SegmentReader>> {\n        // Prevents segment files from getting deleted while we are in the process of opening them\n        let _meta_lock = index.directory().acquire_lock(&META_LOCK)?;\n        let searchable_segments = index.searchable_segments()?;\n        let segment_readers = searchable_segments\n            .iter()\n            .map(SegmentReader::open)\n            .collect::<crate::Result<_>>()?;\n        Ok(segment_readers)\n    }\n\n    fn track_segment_readers_in_inventory(\n        segment_readers: &[SegmentReader],\n        searcher_generation_counter: &Arc<AtomicU64>,\n        searcher_generation_inventory: &Inventory<SearcherGeneration>,\n    ) -> TrackedObject<SearcherGeneration> {\n        let generation_id = searcher_generation_counter.fetch_add(1, atomic::Ordering::AcqRel);\n        let searcher_generation =\n            SearcherGeneration::from_segment_readers(segment_readers, generation_id);\n        searcher_generation_inventory.track(searcher_generation)\n    }\n\n    fn create_searcher(\n        index: &Index,\n        doc_store_cache_num_blocks: usize,\n        warming_state: &WarmingState,\n        searcher_generation_counter: &Arc<AtomicU64>,\n        searcher_generation_inventory: &Inventory<SearcherGeneration>,\n    ) -> crate::Result<Arc<SearcherInner>> {\n        let segment_readers = Self::open_segment_readers(index)?;\n        let searcher_generation = Self::track_segment_readers_in_inventory(\n            &segment_readers,\n            searcher_generation_counter,\n            searcher_generation_inventory,\n        );\n\n        let schema = index.schema();\n        let searcher = Arc::new(SearcherInner::new(\n            schema,\n            index.clone(),\n            segment_readers,\n            searcher_generation,\n            doc_store_cache_num_blocks,\n        )?);\n\n        warming_state.warm_new_searcher_generation(&searcher.clone().into())?;\n        Ok(searcher)\n    }\n\n    fn reload(&self) -> crate::Result<()> {\n        let searcher = Self::create_searcher(\n            &self.index,\n            self.doc_store_cache_num_blocks,\n            &self.warming_state,\n            &self.searcher_generation_counter,\n            &self.searcher_generation_inventory,\n        )?;\n\n        self.searcher.store(searcher);\n\n        Ok(())\n    }\n\n    fn searcher(&self) -> Searcher {\n        self.searcher.load().clone().into()\n    }\n}\n\n/// `IndexReader` is your entry point to read and search the index.\n///\n/// It controls when a new version of the index should be loaded and lends\n/// you instances of `Searcher` for the last loaded version.\n///\n/// `IndexReader` just wraps an `Arc`.\n#[derive(Clone)]\npub struct IndexReader {\n    inner: Arc<InnerIndexReader>,\n    _watch_handle_opt: Option<WatchHandle>,\n}\n\nimpl IndexReader {\n    #[cfg(test)]\n    pub(crate) fn index(&self) -> Index {\n        self.inner.index.clone()\n    }\n\n    /// Update searchers so that they reflect the state of the last\n    /// `.commit()`.\n    ///\n    /// If you set up the [`ReloadPolicy::OnCommitWithDelay`] (which is the default)\n    /// every commit should be rapidly reflected on your `IndexReader` and you should\n    /// not need to call `reload()` at all.\n    ///\n    /// This automatic reload can take 10s of milliseconds to kick in however, and in unit tests\n    /// it can be nice to deterministically force the reload of searchers.\n    pub fn reload(&self) -> crate::Result<()> {\n        self.inner.reload()\n    }\n\n    /// Returns a searcher\n    ///\n    /// This method should be called every single time a search\n    /// query is performed.\n    ///\n    /// The same searcher must be used for a given query, as it ensures\n    /// the use of a consistent segment set.\n    pub fn searcher(&self) -> Searcher {\n        self.inner.searcher()\n    }\n}\n"
  },
  {
    "path": "src/reader/warming.rs",
    "content": "use std::collections::HashSet;\nuse std::ops::Deref;\nuse std::sync::{Arc, Mutex, Weak};\nuse std::thread::JoinHandle;\nuse std::time::Duration;\n\nuse crate::{Executor, Inventory, Searcher, SearcherGeneration, TantivyError};\n\npub const GC_INTERVAL: Duration = Duration::from_secs(1);\n\n/// `Warmer` can be used to maintain segment-level state e.g. caches.\n///\n/// They must be registered with the [`IndexReaderBuilder`](super::IndexReaderBuilder).\npub trait Warmer: Sync + Send {\n    /// Perform any warming work using the provided [`Searcher`].\n    fn warm(&self, searcher: &Searcher) -> crate::Result<()>;\n\n    /// Discards internal state for any [`SearcherGeneration`] not provided.\n    fn garbage_collect(&self, live_generations: &[&SearcherGeneration]);\n}\n\n/// Warming-related state with interior mutability.\n#[derive(Clone)]\npub(crate) struct WarmingState(Arc<Mutex<WarmingStateInner>>);\n\nimpl WarmingState {\n    pub fn new(\n        num_warming_threads: usize,\n        warmers: Vec<Weak<dyn Warmer>>,\n        searcher_generation_inventory: Inventory<SearcherGeneration>,\n    ) -> crate::Result<Self> {\n        Ok(Self(Arc::new(Mutex::new(WarmingStateInner {\n            num_warming_threads,\n            warmers,\n            gc_thread: None,\n            warmed_generation_ids: Default::default(),\n            searcher_generation_inventory,\n        }))))\n    }\n\n    /// Start tracking a new generation of [`Searcher`], and [`Warmer::warm`] it if there are active\n    /// warmers.\n    ///\n    /// A background GC thread for [`Warmer::garbage_collect`] calls is uniquely created if there\n    /// are active warmers.\n    pub fn warm_new_searcher_generation(&self, searcher: &Searcher) -> crate::Result<()> {\n        self.0\n            .lock()\n            .unwrap()\n            .warm_new_searcher_generation(searcher, &self.0)\n    }\n\n    #[cfg(test)]\n    fn gc_maybe(&self) -> bool {\n        self.0.lock().unwrap().gc_maybe()\n    }\n}\n\nstruct WarmingStateInner {\n    num_warming_threads: usize,\n    warmers: Vec<Weak<dyn Warmer>>,\n    gc_thread: Option<JoinHandle<()>>,\n    // Contains all generations that have been warmed up.\n    // This list is used to avoid triggers the individual Warmer GCs\n    // if no warmed generation needs to be collected.\n    warmed_generation_ids: HashSet<u64>,\n    searcher_generation_inventory: Inventory<SearcherGeneration>,\n}\n\nimpl WarmingStateInner {\n    /// Start tracking provided searcher as an exemplar of a new generation.\n    /// If there are active warmers, warm them with the provided searcher, and kick background GC\n    /// thread if it has not yet been kicked. Otherwise, prune state for dropped searcher\n    /// generations inline.\n    fn warm_new_searcher_generation(\n        &mut self,\n        searcher: &Searcher,\n        this: &Arc<Mutex<Self>>,\n    ) -> crate::Result<()> {\n        let warmers = self.pruned_warmers();\n        // Avoid threads (warming as well as background GC) if there are no warmers\n        if warmers.is_empty() {\n            return Ok(());\n        }\n        self.start_gc_thread_maybe(this)?;\n        self.warmed_generation_ids\n            .insert(searcher.generation().generation_id());\n        warming_executor(self.num_warming_threads.min(warmers.len()))?\n            .map(|warmer| warmer.warm(searcher), warmers.into_iter())?;\n        Ok(())\n    }\n\n    /// Attempt to upgrade the weak `Warmer` references, pruning those which cannot be upgraded.\n    /// Return the strong references.\n    fn pruned_warmers(&mut self) -> Vec<Arc<dyn Warmer>> {\n        let strong_warmers = self\n            .warmers\n            .iter()\n            .flat_map(|weak_warmer| weak_warmer.upgrade())\n            .collect::<Vec<_>>();\n        self.warmers = strong_warmers.iter().map(Arc::downgrade).collect();\n        strong_warmers\n    }\n\n    /// [`Warmer::garbage_collect`] active warmers if some searcher generation is observed to have\n    /// been dropped.\n    fn gc_maybe(&mut self) -> bool {\n        let live_generations = self.searcher_generation_inventory.list();\n        let live_generation_ids: HashSet<u64> = live_generations\n            .iter()\n            .map(|searcher_generation| searcher_generation.generation_id())\n            .collect();\n        let gc_not_required = self\n            .warmed_generation_ids\n            .iter()\n            .all(|warmed_up_generation| live_generation_ids.contains(warmed_up_generation));\n        if gc_not_required {\n            return false;\n        }\n        let live_generation_refs = live_generations\n            .iter()\n            .map(Deref::deref)\n            .collect::<Vec<_>>();\n        for warmer in self.pruned_warmers() {\n            warmer.garbage_collect(&live_generation_refs);\n        }\n        self.warmed_generation_ids = live_generation_ids;\n        true\n    }\n\n    /// Start GC thread if one has not already been started.\n    fn start_gc_thread_maybe(&mut self, this: &Arc<Mutex<Self>>) -> crate::Result<bool> {\n        if self.gc_thread.is_some() {\n            return Ok(false);\n        }\n        let weak_inner = Arc::downgrade(this);\n        let handle = std::thread::Builder::new()\n            .name(\"tantivy-warm-gc\".to_owned())\n            .spawn(|| Self::gc_loop(weak_inner))\n            .map_err(|_| {\n                TantivyError::SystemError(\"Failed to spawn warmer GC thread\".to_owned())\n            })?;\n        self.gc_thread = Some(handle);\n        Ok(true)\n    }\n\n    /// Every [`GC_INTERVAL`] attempt to GC, with panics caught and logged using\n    /// [`std::panic::catch_unwind`].\n    fn gc_loop(inner: Weak<Mutex<WarmingStateInner>>) {\n        for _ in crossbeam_channel::tick(GC_INTERVAL) {\n            if let Some(inner) = inner.upgrade() {\n                // rely on deterministic gc in tests\n                #[cfg(not(test))]\n                if let Err(err) = std::panic::catch_unwind(|| inner.lock().unwrap().gc_maybe()) {\n                    error!(\"Panic in Warmer GC {err:?}\");\n                }\n                // avoid unused var warning in tests\n                #[cfg(test)]\n                drop(inner);\n            }\n        }\n    }\n}\n\nfn warming_executor(num_threads: usize) -> crate::Result<Executor> {\n    if num_threads <= 1 {\n        Ok(Executor::single_thread())\n    } else {\n        Executor::multi_thread(num_threads, \"tantivy-warm-\")\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::HashSet;\n    use std::sync::atomic::{self, AtomicUsize};\n    use std::sync::{Arc, RwLock, Weak};\n\n    use super::Warmer;\n    use crate::core::searcher::SearcherGeneration;\n    use crate::directory::RamDirectory;\n    use crate::index::SegmentId;\n    use crate::indexer::index_writer::MEMORY_BUDGET_NUM_BYTES_MIN;\n    use crate::schema::{Schema, INDEXED};\n    use crate::{Index, IndexSettings, ReloadPolicy, Searcher};\n\n    #[derive(Default)]\n    struct TestWarmer {\n        active_segment_ids: RwLock<HashSet<SegmentId>>,\n        warm_calls: AtomicUsize,\n        gc_calls: AtomicUsize,\n    }\n\n    impl TestWarmer {\n        fn live_segment_ids(&self) -> HashSet<SegmentId> {\n            self.active_segment_ids.read().unwrap().clone()\n        }\n\n        fn warm_calls(&self) -> usize {\n            self.warm_calls.load(atomic::Ordering::Acquire)\n        }\n\n        fn gc_calls(&self) -> usize {\n            self.gc_calls.load(atomic::Ordering::Acquire)\n        }\n\n        fn verify(\n            &self,\n            expected_warm_calls: usize,\n            expected_gc_calls: usize,\n            expected_segment_ids: HashSet<SegmentId>,\n        ) {\n            assert_eq!(self.warm_calls(), expected_warm_calls);\n            assert_eq!(self.gc_calls(), expected_gc_calls);\n            assert_eq!(self.live_segment_ids(), expected_segment_ids);\n        }\n    }\n\n    impl Warmer for TestWarmer {\n        fn warm(&self, searcher: &crate::Searcher) -> crate::Result<()> {\n            self.warm_calls.fetch_add(1, atomic::Ordering::SeqCst);\n            for reader in searcher.segment_readers() {\n                self.active_segment_ids\n                    .write()\n                    .unwrap()\n                    .insert(reader.segment_id());\n            }\n            Ok(())\n        }\n\n        fn garbage_collect(&self, live_generations: &[&SearcherGeneration]) {\n            self.gc_calls\n                .fetch_add(1, std::sync::atomic::Ordering::SeqCst);\n            let active_segment_ids = live_generations\n                .iter()\n                .flat_map(|searcher_generation| searcher_generation.segments().keys().copied())\n                .collect();\n            *self.active_segment_ids.write().unwrap() = active_segment_ids;\n        }\n    }\n\n    fn segment_ids(searcher: &Searcher) -> HashSet<SegmentId> {\n        searcher\n            .segment_readers()\n            .iter()\n            .map(|reader| reader.segment_id())\n            .collect()\n    }\n\n    fn test_warming(num_warming_threads: usize) -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let field = schema_builder.add_u64_field(\"pk\", INDEXED);\n        let schema = schema_builder.build();\n\n        let directory = RamDirectory::create();\n        let index = Index::create(directory, schema, IndexSettings::default())?;\n\n        let num_writer_threads = 4;\n        let mut writer = index\n            .writer_with_num_threads(\n                num_writer_threads,\n                MEMORY_BUDGET_NUM_BYTES_MIN * num_writer_threads,\n            )\n            .unwrap();\n\n        for i in 0u64..1000u64 {\n            writer.add_document(doc!(field => i))?;\n        }\n        writer.commit()?;\n\n        let warmer1 = Arc::new(TestWarmer::default());\n        let warmer2 = Arc::new(TestWarmer::default());\n        warmer1.verify(0, 0, HashSet::new());\n        warmer2.verify(0, 0, HashSet::new());\n\n        let num_searchers = 4;\n        let reader = index\n            .reader_builder()\n            .reload_policy(ReloadPolicy::Manual)\n            .num_warming_threads(num_warming_threads)\n            .warmers(vec![\n                Arc::downgrade(&warmer1) as Weak<dyn Warmer>,\n                Arc::downgrade(&warmer2) as Weak<dyn Warmer>,\n            ])\n            .try_into()?;\n\n        let warming_state = &reader.inner.warming_state;\n\n        let searcher = reader.searcher();\n        assert!(\n            !warming_state.gc_maybe(),\n            \"no GC after first searcher generation\"\n        );\n        warmer1.verify(1, 0, segment_ids(&searcher));\n        warmer2.verify(1, 0, segment_ids(&searcher));\n        assert_eq!(searcher.num_docs(), 1000);\n\n        for i in 1000u64..2000u64 {\n            writer.add_document(doc!(field => i))?;\n        }\n        writer.commit()?;\n        writer.wait_merging_threads()?;\n\n        drop(warmer1);\n\n        let old_searcher = searcher;\n\n        reader.reload()?;\n\n        assert!(!warming_state.gc_maybe(), \"old searcher still around\");\n\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 2000);\n\n        warmer2.verify(\n            2,\n            0,\n            segment_ids(&old_searcher)\n                .union(&segment_ids(&searcher))\n                .copied()\n                .collect(),\n        );\n\n        drop(old_searcher);\n        for _ in 0..num_searchers {\n            // make sure the old searcher is dropped by the pool too\n            let _ = reader.searcher();\n        }\n        assert!(warming_state.gc_maybe(), \"old searcher dropped\");\n\n        warmer2.verify(2, 1, segment_ids(&searcher));\n\n        Ok(())\n    }\n\n    #[test]\n    fn warming_single_thread() -> crate::Result<()> {\n        test_warming(1)\n    }\n\n    #[test]\n    fn warming_four_threads() -> crate::Result<()> {\n        test_warming(4)\n    }\n}\n"
  },
  {
    "path": "src/schema/bytes_options.rs",
    "content": "use std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::flags::{FastFlag, IndexedFlag, SchemaFlagList, StoredFlag};\n/// Define how a bytes field should be handled by tantivy.\n#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]\n#[serde(from = \"BytesOptionsDeser\")]\npub struct BytesOptions {\n    indexed: bool,\n    fieldnorms: bool,\n    fast: bool,\n    stored: bool,\n}\n\n/// For backward compatibility we add an intermediary to interpret the\n/// lack of fieldnorms attribute as \"true\" if and only if indexed.\n///\n/// (Downstream, for the moment, this attribute is not used if not indexed...)\n/// Note that: newly serialized NumericOptions will include the new attribute.\n#[derive(Deserialize)]\nstruct BytesOptionsDeser {\n    indexed: bool,\n    #[serde(default)]\n    fieldnorms: Option<bool>,\n    fast: bool,\n    stored: bool,\n}\n\nimpl From<BytesOptionsDeser> for BytesOptions {\n    fn from(deser: BytesOptionsDeser) -> Self {\n        BytesOptions {\n            indexed: deser.indexed,\n            fieldnorms: deser.fieldnorms.unwrap_or(deser.indexed),\n            fast: deser.fast,\n            stored: deser.stored,\n        }\n    }\n}\n\nimpl BytesOptions {\n    /// Returns true if the value is indexed.\n    #[inline]\n    pub fn is_indexed(&self) -> bool {\n        self.indexed\n    }\n\n    /// Returns true if and only if the value is normed.\n    #[inline]\n    pub fn fieldnorms(&self) -> bool {\n        self.fieldnorms\n    }\n\n    /// Returns true if the value is a fast field.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        self.fast\n    }\n\n    /// Returns true if the value is stored.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Set the field as indexed.\n    ///\n    /// Setting an integer as indexed will generate\n    /// a posting list for each value taken by the integer.\n    #[must_use]\n    pub fn set_indexed(mut self) -> BytesOptions {\n        self.indexed = true;\n        self\n    }\n\n    /// Set the field as normed.\n    ///\n    /// Setting an integer as normed will generate\n    /// the fieldnorm data for it.\n    #[must_use]\n    pub fn set_fieldnorms(mut self) -> BytesOptions {\n        self.fieldnorms = true;\n        self\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    #[must_use]\n    pub fn set_fast(mut self) -> BytesOptions {\n        self.fast = true;\n        self\n    }\n\n    /// Set the field as stored.\n    ///\n    /// Only the fields that are set as *stored* are\n    /// persisted into the Tantivy's store.\n    #[must_use]\n    pub fn set_stored(mut self) -> BytesOptions {\n        self.stored = true;\n        self\n    }\n}\n\nimpl<T: Into<BytesOptions>> BitOr<T> for BytesOptions {\n    type Output = BytesOptions;\n\n    fn bitor(self, other: T) -> BytesOptions {\n        let other = other.into();\n        BytesOptions {\n            indexed: self.indexed | other.indexed,\n            fieldnorms: self.fieldnorms | other.fieldnorms,\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n        }\n    }\n}\n\nimpl From<()> for BytesOptions {\n    fn from(_: ()) -> Self {\n        Self::default()\n    }\n}\n\nimpl From<FastFlag> for BytesOptions {\n    fn from(_: FastFlag) -> Self {\n        BytesOptions {\n            indexed: false,\n            fieldnorms: false,\n            stored: false,\n            fast: true,\n        }\n    }\n}\n\nimpl From<StoredFlag> for BytesOptions {\n    fn from(_: StoredFlag) -> Self {\n        BytesOptions {\n            indexed: false,\n            fieldnorms: false,\n            stored: true,\n            fast: false,\n        }\n    }\n}\n\nimpl From<IndexedFlag> for BytesOptions {\n    fn from(_: IndexedFlag) -> Self {\n        BytesOptions {\n            indexed: true,\n            fieldnorms: true,\n            stored: false,\n            fast: false,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for BytesOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::schema::{BytesOptions, FAST, INDEXED, STORED};\n\n    #[test]\n    fn test_bytes_option_fast_flag() {\n        assert_eq!(BytesOptions::default().set_fast(), FAST.into());\n        assert_eq!(\n            BytesOptions::default().set_indexed().set_fieldnorms(),\n            INDEXED.into()\n        );\n        assert_eq!(BytesOptions::default().set_stored(), STORED.into());\n    }\n    #[test]\n    fn test_bytes_option_fast_flag_composition() {\n        assert_eq!(\n            BytesOptions::default().set_fast().set_stored(),\n            (FAST | STORED).into()\n        );\n        assert_eq!(\n            BytesOptions::default()\n                .set_indexed()\n                .set_fieldnorms()\n                .set_fast(),\n            (INDEXED | FAST).into()\n        );\n        assert_eq!(\n            BytesOptions::default()\n                .set_stored()\n                .set_fieldnorms()\n                .set_indexed(),\n            (STORED | INDEXED).into()\n        );\n    }\n\n    #[test]\n    fn test_bytes_option_fast_() {\n        assert!(!BytesOptions::default().is_stored());\n        assert!(!BytesOptions::default().is_fast());\n        assert!(!BytesOptions::default().is_indexed());\n        assert!(!BytesOptions::default().fieldnorms());\n        assert!(BytesOptions::default().set_stored().is_stored());\n        assert!(BytesOptions::default().set_fast().is_fast());\n        assert!(BytesOptions::default().set_indexed().is_indexed());\n        assert!(BytesOptions::default().set_fieldnorms().fieldnorms());\n    }\n\n    #[test]\n    fn test_bytes_options_deser_if_fieldnorm_missing_indexed_true() {\n        let json = r#\"{\n            \"indexed\": true,\n            \"fast\": false,\n            \"stored\": false\n        }\"#;\n        let bytes_options: BytesOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &bytes_options,\n            &BytesOptions {\n                indexed: true,\n                fieldnorms: true,\n                fast: false,\n                stored: false\n            }\n        );\n    }\n\n    #[test]\n    fn test_bytes_options_deser_if_fieldnorm_missing_indexed_false() {\n        let json = r#\"{\n            \"indexed\": false,\n            \"stored\": false,\n            \"fast\": false\n        }\"#;\n        let bytes_options: BytesOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &bytes_options,\n            &BytesOptions {\n                indexed: false,\n                fieldnorms: false,\n                fast: false,\n                stored: false\n            }\n        );\n    }\n\n    #[test]\n    fn test_bytes_options_deser_if_fieldnorm_false_indexed_true() {\n        let json = r#\"{\n            \"indexed\": true,\n            \"fieldnorms\": false,\n            \"fast\": false,\n            \"stored\": false\n        }\"#;\n        let bytes_options: BytesOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &bytes_options,\n            &BytesOptions {\n                indexed: true,\n                fieldnorms: false,\n                fast: false,\n                stored: false\n            }\n        );\n    }\n\n    #[test]\n    fn test_bytes_options_deser_if_fieldnorm_true_indexed_false() {\n        // this one is kind of useless, at least at the moment\n        let json = r#\"{\n            \"indexed\": false,\n            \"fieldnorms\": true,\n            \"fast\": false,\n            \"stored\": false\n        }\"#;\n        let bytes_options: BytesOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &bytes_options,\n            &BytesOptions {\n                indexed: false,\n                fieldnorms: true,\n                fast: false,\n                stored: false\n            }\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/date_time_options.rs",
    "content": "use std::ops::BitOr;\n\npub use common::DateTimePrecision;\nuse serde::{Deserialize, Serialize};\n\nuse crate::schema::flags::{FastFlag, IndexedFlag, SchemaFlagList, StoredFlag};\n\n/// The precision of the indexed date/time values in the inverted index.\npub const DATE_TIME_PRECISION_INDEXED: DateTimePrecision = DateTimePrecision::Seconds;\n\n/// Defines how DateTime field should be handled by tantivy.\n#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)]\npub struct DateOptions {\n    indexed: bool,\n    // This boolean has no effect if the field is not marked as indexed true.\n    fieldnorms: bool,\n    #[serde(default)]\n    fast: bool,\n    stored: bool,\n    // Internal storage precision, used to optimize storage\n    // compression on fast fields.\n    #[serde(default)]\n    precision: DateTimePrecision,\n}\n\nimpl DateOptions {\n    /// Returns true iff the value is stored.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Returns true iff the value is indexed and therefore searchable.\n    #[inline]\n    pub fn is_indexed(&self) -> bool {\n        self.indexed\n    }\n\n    /// Returns true iff the field has fieldnorm.\n    #[inline]\n    pub fn fieldnorms(&self) -> bool {\n        self.fieldnorms && self.indexed\n    }\n\n    /// Returns true iff the value is a fast field.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        self.fast\n    }\n\n    /// Set the field as stored.\n    ///\n    /// Only the fields that are set as *stored* are\n    /// persisted into the Tantivy's store.\n    #[must_use]\n    pub fn set_stored(mut self) -> DateOptions {\n        self.stored = true;\n        self\n    }\n\n    /// Set the field as indexed.\n    ///\n    /// Setting an integer as indexed will generate\n    /// a posting list for each value taken by the integer.\n    ///\n    /// This is required for the field to be searchable.\n    #[must_use]\n    pub fn set_indexed(mut self) -> DateOptions {\n        self.indexed = true;\n        self\n    }\n\n    /// Set the field with fieldnorm.\n    ///\n    /// Setting an integer as fieldnorm will generate\n    /// the fieldnorm data for it.\n    #[must_use]\n    pub fn set_fieldnorm(mut self) -> DateOptions {\n        self.fieldnorms = true;\n        self\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    #[must_use]\n    pub fn set_fast(mut self) -> DateOptions {\n        self.fast = true;\n        self\n    }\n\n    /// Sets the precision for this DateTime field on the fast field.\n    /// Indexed precision is always [`DATE_TIME_PRECISION_INDEXED`].\n    ///\n    /// Internal storage precision, used to optimize storage\n    /// compression on fast fields.\n    pub fn set_precision(mut self, precision: DateTimePrecision) -> DateOptions {\n        self.precision = precision;\n        self\n    }\n\n    /// Returns the storage precision for this DateTime field.\n    ///\n    /// Internal storage precision, used to optimize storage\n    /// compression on fast fields.\n    pub fn get_precision(&self) -> DateTimePrecision {\n        self.precision\n    }\n}\n\nimpl From<()> for DateOptions {\n    fn from(_: ()) -> DateOptions {\n        DateOptions::default()\n    }\n}\n\nimpl From<FastFlag> for DateOptions {\n    fn from(_: FastFlag) -> Self {\n        DateOptions {\n            fast: true,\n            ..Default::default()\n        }\n    }\n}\n\nimpl From<StoredFlag> for DateOptions {\n    fn from(_: StoredFlag) -> Self {\n        DateOptions {\n            stored: true,\n            ..Default::default()\n        }\n    }\n}\n\nimpl From<IndexedFlag> for DateOptions {\n    fn from(_: IndexedFlag) -> Self {\n        DateOptions {\n            indexed: true,\n            fieldnorms: true,\n            ..Default::default()\n        }\n    }\n}\n\nimpl<T: Into<DateOptions>> BitOr<T> for DateOptions {\n    type Output = DateOptions;\n\n    fn bitor(self, other: T) -> DateOptions {\n        let other = other.into();\n        DateOptions {\n            indexed: self.indexed | other.indexed,\n            fieldnorms: self.fieldnorms | other.fieldnorms,\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n            precision: self.precision,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for DateOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_date_options_consistent_with_default() {\n        let date_time_options: DateOptions = serde_json::from_str(\n            r#\"{\n            \"indexed\": false,\n            \"fieldnorms\": false,\n            \"stored\": false\n        }\"#,\n        )\n        .unwrap();\n        assert_eq!(date_time_options, DateOptions::default());\n    }\n\n    #[test]\n    fn test_serialize_date_option() {\n        let date_options = serde_json::from_str::<DateOptions>(\n            r#\"\n            {\n                \"indexed\": true,\n                \"fieldnorms\": false,\n                \"stored\": false,\n                \"precision\": \"milliseconds\"\n            }\"#,\n        )\n        .unwrap();\n\n        let date_options_json = serde_json::to_value(date_options).unwrap();\n        assert_eq!(\n            date_options_json,\n            serde_json::json!({\n                \"precision\": \"milliseconds\",\n                \"indexed\": true,\n                \"fast\": false,\n                \"fieldnorms\": false,\n                \"stored\": false\n            })\n        );\n    }\n\n    #[test]\n    fn test_deserialize_date_options_with_wrong_options() {\n        assert!(serde_json::from_str::<DateOptions>(\n            r#\"{\n            \"indexed\": true,\n            \"fieldnorms\": false,\n            \"stored\": \"wrong_value\"\n        }\"#\n        )\n        .unwrap_err()\n        .to_string()\n        .contains(\"expected a boolean\"));\n\n        assert!(serde_json::from_str::<DateOptions>(\n            r#\"{\n            \"indexed\": true,\n            \"fieldnorms\": false,\n            \"stored\": false,\n            \"precision\": \"hours\"\n        }\"#\n        )\n        .unwrap_err()\n        .to_string()\n        .contains(\"unknown variant `hours`\"));\n    }\n}\n"
  },
  {
    "path": "src/schema/document/de.rs",
    "content": "//! Document binary deserialization API\n//!\n//! The deserialization API is strongly inspired by serde's API but with\n//! some tweaks, mostly around some of the types being concrete (errors)\n//! and some more specific types being visited (Ips, datetime, etc...)\n//!\n//! The motivation behind this API is to provide a easy to implement and\n//! efficient way of deserializing a potentially arbitrarily nested object.\n\nuse std::collections::{BTreeMap, HashMap};\nuse std::fmt::Display;\nuse std::io;\nuse std::io::Read;\nuse std::marker::PhantomData;\nuse std::net::Ipv6Addr;\nuse std::sync::Arc;\n\nuse columnar::MonotonicallyMappableToU128;\nuse common::{u64_to_f64, BinarySerializable, DateTime, VInt};\n\nuse super::se::BinaryObjectSerializer;\nuse super::{OwnedValue, Value};\nuse crate::schema::document::type_codes;\nuse crate::schema::{Facet, Field};\nuse crate::store::DocStoreVersion;\nuse crate::tokenizer::PreTokenizedString;\n\n#[derive(Debug, thiserror::Error, Clone)]\n/// An error which occurs while attempting to deserialize a given value\n/// by using the provided value visitor.\npub enum DeserializeError {\n    #[error(\"Unsupported Type: {0:?} cannot be deserialized from the given visitor\")]\n    /// The value cannot be deserialized from the given type.\n    UnsupportedType(ValueType),\n    #[error(\"Type Mismatch: Expected {expected:?} but found {actual:?}\")]\n    /// The value cannot be deserialized from the given type.\n    TypeMismatch {\n        /// The expected value type.\n        expected: ValueType,\n        /// The actual value type read.\n        actual: ValueType,\n    },\n    #[error(\"The value could not be read: {0}\")]\n    /// The value was unable to be read due to the error.\n    CorruptedValue(Arc<io::Error>),\n    #[error(\"{0}\")]\n    /// A custom error message.\n    Custom(String),\n    #[error(\"Version {0}, Max version supported: {1}\")]\n    /// Unsupported version error.\n    UnsupportedVersion(u32, u32),\n}\n\nimpl DeserializeError {\n    /// Creates a new custom deserialize error.\n    pub fn custom(msg: impl Display) -> Self {\n        Self::Custom(msg.to_string())\n    }\n}\n\nimpl From<io::Error> for DeserializeError {\n    fn from(error: io::Error) -> Self {\n        Self::CorruptedValue(Arc::new(error))\n    }\n}\n\n/// The core trait for deserializing a document.\n///\n/// TODO: Improve docs\npub trait DocumentDeserialize: Sized {\n    /// Attempts to deserialize Self from a given document deserializer.\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: DocumentDeserializer<'de>;\n}\n\n/// A deserializer that can walk through each entry in the document.\npub trait DocumentDeserializer<'de> {\n    /// A indicator as to how many values are in the document.\n    ///\n    /// This can be used to pre-allocate entries but should not\n    /// be depended on as a fixed size.\n    fn size_hint(&self) -> usize;\n\n    /// Attempts to deserialize the next field in the document.\n    fn next_field<V: ValueDeserialize>(&mut self) -> Result<Option<(Field, V)>, DeserializeError>;\n}\n\n/// The core trait for deserializing values.\n///\n/// TODO: Improve docs\npub trait ValueDeserialize: Sized {\n    /// Attempts to deserialize Self from a given value deserializer.\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de>;\n}\n\n/// A value deserializer.\npub trait ValueDeserializer<'de> {\n    /// Attempts to deserialize a null value from the deserializer.\n    fn deserialize_null(self) -> Result<(), DeserializeError>;\n\n    /// Attempts to deserialize a string value from the deserializer.\n    fn deserialize_string(self) -> Result<String, DeserializeError>;\n\n    /// Attempts to deserialize a u64 value from the deserializer.\n    fn deserialize_u64(self) -> Result<u64, DeserializeError>;\n\n    /// Attempts to deserialize an i64 value from the deserializer.\n    fn deserialize_i64(self) -> Result<i64, DeserializeError>;\n\n    /// Attempts to deserialize a f64 value from the deserializer.\n    fn deserialize_f64(self) -> Result<f64, DeserializeError>;\n\n    /// Attempts to deserialize a datetime value from the deserializer.\n    fn deserialize_datetime(self) -> Result<DateTime, DeserializeError>;\n\n    /// Attempts to deserialize a facet value from the deserializer.\n    fn deserialize_facet(self) -> Result<Facet, DeserializeError>;\n\n    /// Attempts to deserialize a bytes value from the deserializer.\n    fn deserialize_bytes(self) -> Result<Vec<u8>, DeserializeError>;\n\n    /// Attempts to deserialize an IP address value from the deserializer.\n    fn deserialize_ip_address(self) -> Result<Ipv6Addr, DeserializeError>;\n\n    /// Attempts to deserialize a bool value from the deserializer.\n    fn deserialize_bool(self) -> Result<bool, DeserializeError>;\n\n    /// Attempts to deserialize a pre-tokenized string value from the deserializer.\n    fn deserialize_pre_tokenized_string(self) -> Result<PreTokenizedString, DeserializeError>;\n\n    /// Attempts to deserialize the value using a given visitor.\n    fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, DeserializeError>\n    where V: ValueVisitor;\n}\n\n#[derive(Debug, Copy, Clone, Eq, PartialEq)]\n/// The type of the value attempting to be deserialized.\npub enum ValueType {\n    /// A null value.\n    Null,\n    /// A string value.\n    String,\n    /// A u64 value.\n    U64,\n    /// A i64 value.\n    I64,\n    /// A f64 value.\n    F64,\n    /// A datetime value.\n    DateTime,\n    /// A facet value.\n    Facet,\n    /// A bytes value.\n    Bytes,\n    /// A IP address value.\n    IpAddr,\n    /// A boolean value.\n    Bool,\n    /// A pre-tokenized string value.\n    PreTokStr,\n    /// An array of value.\n    Array,\n    /// A dynamic object value.\n    Object,\n    /// A JSON object value. Deprecated.\n    #[deprecated(note = \"We keep this for backwards compatibility, use Object instead\")]\n    JSONObject,\n}\n\n/// A value visitor for deserializing a document value.\n///\n/// This is strongly inspired by serde but has a few extra types.\n///\n/// TODO: Improve docs\npub trait ValueVisitor {\n    /// The value produced by the visitor.\n    type Value;\n\n    #[inline]\n    /// Called when the deserializer visits a string value.\n    fn visit_null(&self) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::Null))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a string value.\n    fn visit_string(&self, _val: String) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::String))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a u64 value.\n    fn visit_u64(&self, _val: u64) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::U64))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a i64 value.\n    fn visit_i64(&self, _val: i64) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::I64))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a f64 value.\n    fn visit_f64(&self, _val: f64) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::F64))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a bool value.\n    fn visit_bool(&self, _val: bool) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::Bool))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a datetime value.\n    fn visit_datetime(&self, _val: DateTime) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::DateTime))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits an IP address value.\n    fn visit_ip_address(&self, _val: Ipv6Addr) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::IpAddr))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a facet value.\n    fn visit_facet(&self, _val: Facet) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::Facet))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a bytes value.\n    fn visit_bytes(&self, _val: Vec<u8>) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::Bytes))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a pre-tokenized string value.\n    fn visit_pre_tokenized_string(\n        &self,\n        _val: PreTokenizedString,\n    ) -> Result<Self::Value, DeserializeError> {\n        Err(DeserializeError::UnsupportedType(ValueType::PreTokStr))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits an array.\n    fn visit_array<'de, A>(&self, _access: A) -> Result<Self::Value, DeserializeError>\n    where A: ArrayAccess<'de> {\n        Err(DeserializeError::UnsupportedType(ValueType::Array))\n    }\n\n    #[inline]\n    /// Called when the deserializer visits a object value.\n    fn visit_object<'de, A>(&self, _access: A) -> Result<Self::Value, DeserializeError>\n    where A: ObjectAccess<'de> {\n        Err(DeserializeError::UnsupportedType(ValueType::Object))\n    }\n}\n\n/// Access to a sequence of values which can be deserialized.\npub trait ArrayAccess<'de> {\n    /// A indicator as to how many values are in the object.\n    ///\n    /// This can be used to pre-allocate entries but should not\n    /// be depended on as a fixed size.\n    fn size_hint(&self) -> usize;\n\n    /// Attempts to deserialize the next element in the sequence.\n    fn next_element<V: ValueDeserialize>(&mut self) -> Result<Option<V>, DeserializeError>;\n}\n\n/// TODO: Improve docs\npub trait ObjectAccess<'de> {\n    /// A indicator as to how many values are in the object.\n    ///\n    /// This can be used to pre-allocate entries but should not\n    /// be depended on as a fixed size.\n    fn size_hint(&self) -> usize;\n\n    /// Attempts to deserialize the next key-value pair in the object.\n    fn next_entry<V: ValueDeserialize>(&mut self) -> Result<Option<(String, V)>, DeserializeError>;\n}\n\n/// The document deserializer used to read the tantivy documents serialized with\n/// `BinarySerializable`.\n///\n/// This acts very similarly to serde's deserialize types and can incrementally\n/// deserialize each field of the document from the provided reader (`R`).\n///\n/// TODO: Switch to slice instead?\npub struct BinaryDocumentDeserializer<'de, R> {\n    length: usize,\n    position: usize,\n    doc_store_version: DocStoreVersion,\n    reader: &'de mut R,\n}\n\nimpl<'de, R> BinaryDocumentDeserializer<'de, R>\nwhere R: Read\n{\n    /// Attempts to create a new document deserializer from a given reader.\n    pub(crate) fn from_reader(\n        reader: &'de mut R,\n        doc_store_version: DocStoreVersion,\n    ) -> Result<Self, DeserializeError> {\n        let length = VInt::deserialize(reader)?;\n\n        Ok(Self {\n            length: length.val() as usize,\n            position: 0,\n            doc_store_version,\n            reader,\n        })\n    }\n\n    /// Returns true if the deserializer has deserialized all the entries\n    /// within the document.\n    fn is_complete(&self) -> bool {\n        self.position >= self.length\n    }\n}\n\nimpl<'de, R> DocumentDeserializer<'de> for BinaryDocumentDeserializer<'de, R>\nwhere R: Read\n{\n    #[inline]\n    fn size_hint(&self) -> usize {\n        self.length\n    }\n\n    fn next_field<V: ValueDeserialize>(&mut self) -> Result<Option<(Field, V)>, DeserializeError> {\n        if self.is_complete() {\n            return Ok(None);\n        }\n\n        let field = Field::deserialize(self.reader).map_err(DeserializeError::from)?;\n        let deserializer =\n            BinaryValueDeserializer::from_reader(self.reader, self.doc_store_version)?;\n        let value = V::deserialize(deserializer)?;\n\n        self.position += 1;\n\n        Ok(Some((field, value)))\n    }\n}\n\n/// A single value deserializer that deserializes a value serialized with `BinarySerializable`.\n/// TODO: Improve docs\npub struct BinaryValueDeserializer<'de, R> {\n    value_type: ValueType,\n    reader: &'de mut R,\n    doc_store_version: DocStoreVersion,\n}\n\nimpl<'de, R> BinaryValueDeserializer<'de, R>\nwhere R: Read\n{\n    /// Attempts to create a new value deserializer from a given reader.\n    fn from_reader(\n        reader: &'de mut R,\n        doc_store_version: DocStoreVersion,\n    ) -> Result<Self, DeserializeError> {\n        let type_code = <u8 as BinarySerializable>::deserialize(reader)?;\n\n        let value_type = match type_code {\n            type_codes::TEXT_CODE => ValueType::String,\n            type_codes::U64_CODE => ValueType::U64,\n            type_codes::I64_CODE => ValueType::I64,\n            type_codes::F64_CODE => ValueType::F64,\n            type_codes::BOOL_CODE => ValueType::Bool,\n            type_codes::DATE_CODE => ValueType::DateTime,\n            type_codes::HIERARCHICAL_FACET_CODE => ValueType::Facet,\n            type_codes::BYTES_CODE => ValueType::Bytes,\n            type_codes::EXT_CODE => {\n                let ext_type_code = <u8 as BinarySerializable>::deserialize(reader)?;\n\n                match ext_type_code {\n                    type_codes::TOK_STR_EXT_CODE => ValueType::PreTokStr,\n                    _ => {\n                        return Err(DeserializeError::from(io::Error::new(\n                            io::ErrorKind::InvalidData,\n                            format!(\n                                \"No extended field type is associated with code {ext_type_code:?}\"\n                            ),\n                        )))\n                    }\n                }\n            }\n            type_codes::IP_CODE => ValueType::IpAddr,\n            type_codes::NULL_CODE => ValueType::Null,\n            type_codes::ARRAY_CODE => ValueType::Array,\n            type_codes::OBJECT_CODE => ValueType::Object,\n            #[expect(deprecated)]\n            type_codes::JSON_OBJ_CODE => ValueType::JSONObject,\n            _ => {\n                return Err(DeserializeError::from(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\"No field type is associated with code {type_code:?}\"),\n                )))\n            }\n        };\n\n        Ok(Self {\n            value_type,\n            reader,\n            doc_store_version,\n        })\n    }\n\n    fn validate_type(&self, expected_type: ValueType) -> Result<(), DeserializeError> {\n        if self.value_type == expected_type {\n            Ok(())\n        } else {\n            Err(DeserializeError::TypeMismatch {\n                expected: expected_type,\n                actual: self.value_type,\n            })\n        }\n    }\n}\n\nimpl<'de, R> ValueDeserializer<'de> for BinaryValueDeserializer<'de, R>\nwhere R: Read\n{\n    fn deserialize_null(self) -> Result<(), DeserializeError> {\n        self.validate_type(ValueType::Null)?;\n        Ok(())\n    }\n\n    fn deserialize_string(self) -> Result<String, DeserializeError> {\n        self.validate_type(ValueType::String)?;\n        <String as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_u64(self) -> Result<u64, DeserializeError> {\n        self.validate_type(ValueType::U64)?;\n        <u64 as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_i64(self) -> Result<i64, DeserializeError> {\n        self.validate_type(ValueType::I64)?;\n        <i64 as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_f64(self) -> Result<f64, DeserializeError> {\n        self.validate_type(ValueType::F64)?;\n        <u64 as BinarySerializable>::deserialize(self.reader)\n            .map(u64_to_f64)\n            .map_err(DeserializeError::from)\n    }\n\n    fn deserialize_datetime(self) -> Result<DateTime, DeserializeError> {\n        self.validate_type(ValueType::DateTime)?;\n        match self.doc_store_version {\n            DocStoreVersion::V1 => {\n                let timestamp_micros = <i64 as BinarySerializable>::deserialize(self.reader)?;\n                Ok(DateTime::from_timestamp_micros(timestamp_micros))\n            }\n            DocStoreVersion::V2 => {\n                let timestamp_nanos = <i64 as BinarySerializable>::deserialize(self.reader)?;\n                Ok(DateTime::from_timestamp_nanos(timestamp_nanos))\n            }\n        }\n    }\n\n    fn deserialize_facet(self) -> Result<Facet, DeserializeError> {\n        self.validate_type(ValueType::Facet)?;\n        <Facet as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_bytes(self) -> Result<Vec<u8>, DeserializeError> {\n        self.validate_type(ValueType::Bytes)?;\n        <Vec<u8> as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_ip_address(self) -> Result<Ipv6Addr, DeserializeError> {\n        self.validate_type(ValueType::IpAddr)?;\n        <u128 as BinarySerializable>::deserialize(self.reader)\n            .map(Ipv6Addr::from_u128)\n            .map_err(DeserializeError::from)\n    }\n\n    fn deserialize_bool(self) -> Result<bool, DeserializeError> {\n        self.validate_type(ValueType::Bool)?;\n        <bool as BinarySerializable>::deserialize(self.reader).map_err(DeserializeError::from)\n    }\n\n    fn deserialize_pre_tokenized_string(self) -> Result<PreTokenizedString, DeserializeError> {\n        self.validate_type(ValueType::PreTokStr)?;\n        <PreTokenizedString as BinarySerializable>::deserialize(self.reader)\n            .map_err(DeserializeError::from)\n    }\n\n    fn deserialize_any<V>(self, visitor: V) -> Result<V::Value, DeserializeError>\n    where V: ValueVisitor {\n        match self.value_type {\n            ValueType::Null => visitor.visit_null(),\n            ValueType::String => {\n                let val = self.deserialize_string()?;\n                visitor.visit_string(val)\n            }\n            ValueType::U64 => {\n                let val = self.deserialize_u64()?;\n                visitor.visit_u64(val)\n            }\n            ValueType::I64 => {\n                let val = self.deserialize_i64()?;\n                visitor.visit_i64(val)\n            }\n            ValueType::F64 => {\n                let val = self.deserialize_f64()?;\n                visitor.visit_f64(val)\n            }\n            ValueType::DateTime => {\n                let val = self.deserialize_datetime()?;\n                visitor.visit_datetime(val)\n            }\n            ValueType::Facet => {\n                let val = self.deserialize_facet()?;\n                visitor.visit_facet(val)\n            }\n            ValueType::Bytes => {\n                let val = self.deserialize_bytes()?;\n                visitor.visit_bytes(val)\n            }\n            ValueType::IpAddr => {\n                let val = self.deserialize_ip_address()?;\n                visitor.visit_ip_address(val)\n            }\n            ValueType::Bool => {\n                let val = self.deserialize_bool()?;\n                visitor.visit_bool(val)\n            }\n            ValueType::PreTokStr => {\n                let val = self.deserialize_pre_tokenized_string()?;\n                visitor.visit_pre_tokenized_string(val)\n            }\n            ValueType::Array => {\n                let access =\n                    BinaryArrayDeserializer::from_reader(self.reader, self.doc_store_version)?;\n                visitor.visit_array(access)\n            }\n            ValueType::Object => {\n                let access =\n                    BinaryObjectDeserializer::from_reader(self.reader, self.doc_store_version)?;\n                visitor.visit_object(access)\n            }\n            #[allow(deprecated)]\n            ValueType::JSONObject => {\n                // This is a compatibility layer\n                // The implementation is slow, but is temporary anyways\n                let mut de = serde_json::Deserializer::from_reader(self.reader);\n                let json_map = <serde_json::Map::<String, serde_json::Value> as serde::Deserialize>::deserialize(&mut de).map_err(|err| DeserializeError::Custom(err.to_string()))?;\n                let mut out = Vec::new();\n                let mut serializer = BinaryObjectSerializer::begin(json_map.len(), &mut out)?;\n                for (key, val) in json_map {\n                    let val: OwnedValue = val.into();\n                    serializer.serialize_entry(&key, (&val).as_value())?;\n                }\n                serializer.end()?;\n\n                let out_rc = std::rc::Rc::new(out);\n                let mut slice: &[u8] = &out_rc;\n                let access =\n                    BinaryObjectDeserializer::from_reader(&mut slice, self.doc_store_version)?;\n\n                visitor.visit_object(access)\n            }\n        }\n    }\n}\n\n/// A deserializer for an array of values serialized with `BinarySerializable`.\n/// TODO: Improve docs\npub struct BinaryArrayDeserializer<'de, R> {\n    length: usize,\n    position: usize,\n    reader: &'de mut R,\n    doc_store_version: DocStoreVersion,\n}\n\nimpl<'de, R> BinaryArrayDeserializer<'de, R>\nwhere R: Read\n{\n    /// Attempts to create a new array deserializer from a given reader.\n    fn from_reader(\n        reader: &'de mut R,\n        doc_store_version: DocStoreVersion,\n    ) -> Result<Self, DeserializeError> {\n        let length = <VInt as BinarySerializable>::deserialize(reader)?;\n\n        Ok(Self {\n            length: length.val() as usize,\n            position: 0,\n            reader,\n            doc_store_version,\n        })\n    }\n\n    /// Returns true if the deserializer has deserialized all the elements\n    /// within the array.\n    fn is_complete(&self) -> bool {\n        self.position >= self.length\n    }\n}\n\nimpl<'de, R> ArrayAccess<'de> for BinaryArrayDeserializer<'de, R>\nwhere R: Read\n{\n    #[inline]\n    fn size_hint(&self) -> usize {\n        self.length\n    }\n\n    fn next_element<V: ValueDeserialize>(&mut self) -> Result<Option<V>, DeserializeError> {\n        if self.is_complete() {\n            return Ok(None);\n        }\n\n        let deserializer =\n            BinaryValueDeserializer::from_reader(self.reader, self.doc_store_version)?;\n        let value = V::deserialize(deserializer)?;\n\n        // Advance the position cursor.\n        self.position += 1;\n\n        Ok(Some(value))\n    }\n}\n\n/// A deserializer for a object consisting of key-value pairs.\npub struct BinaryObjectDeserializer<'de, R> {\n    /// The inner deserializer.\n    ///\n    /// Internally an object is just represented by an array\n    /// in the format of `[key, value, key, value, key, value]`.\n    inner: BinaryArrayDeserializer<'de, R>,\n}\n\nimpl<'de, R> BinaryObjectDeserializer<'de, R>\nwhere R: Read\n{\n    /// Attempts to create a new object deserializer from a given reader.\n    fn from_reader(\n        reader: &'de mut R,\n        doc_store_version: DocStoreVersion,\n    ) -> Result<Self, DeserializeError> {\n        let inner = BinaryArrayDeserializer::from_reader(reader, doc_store_version)?;\n        Ok(Self { inner })\n    }\n}\n\nimpl<'de, R> ObjectAccess<'de> for BinaryObjectDeserializer<'de, R>\nwhere R: Read\n{\n    #[inline]\n    /// A indicator as to how many values are in the object.\n    ///\n    /// This can be used to pre-allocate entries but should not\n    /// be depended on as a fixed size.\n    fn size_hint(&self) -> usize {\n        // We divide by 2 here as we know our elements are going to be\n        // in the format of `[key, value, key, value, key, value]`.\n        self.inner.size_hint() / 2\n    }\n\n    /// Attempts to deserialize the next key-value pair in the object.\n    fn next_entry<V: ValueDeserialize>(&mut self) -> Result<Option<(String, V)>, DeserializeError> {\n        if self.inner.is_complete() {\n            return Ok(None);\n        }\n\n        let key = self.inner.next_element::<String>()?.expect(\n            \"Deserializer should not be empty as it is not marked as complete, this is a bug\",\n        );\n        let value = self.inner.next_element::<V>()?.expect(\n            \"Deserializer should not be empty as it is not marked as complete, this is a bug\",\n        );\n\n        Ok(Some((key, value)))\n    }\n}\n\n// Core type implementations\n\nimpl ValueDeserialize for String {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_string()\n    }\n}\n\nimpl ValueDeserialize for u64 {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_u64()\n    }\n}\n\nimpl ValueDeserialize for i64 {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_i64()\n    }\n}\n\nimpl ValueDeserialize for f64 {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_f64()\n    }\n}\n\nimpl ValueDeserialize for DateTime {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_datetime()\n    }\n}\n\nimpl ValueDeserialize for Ipv6Addr {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_ip_address()\n    }\n}\n\nimpl ValueDeserialize for Facet {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_facet()\n    }\n}\n\nimpl ValueDeserialize for Vec<u8> {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_bytes()\n    }\n}\n\nimpl ValueDeserialize for PreTokenizedString {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_pre_tokenized_string()\n    }\n}\n\n// Collections kind of suck, but can't think of a nicer way of doing this generically\n// without quite literally cloning serde entirely...\n\nstruct VecVisitor<T: ValueDeserialize>(PhantomData<T>);\nimpl<T: ValueDeserialize> ValueVisitor for VecVisitor<T> {\n    type Value = Vec<T>;\n\n    fn visit_array<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n    where A: ArrayAccess<'de> {\n        let mut entries = Vec::with_capacity(access.size_hint());\n        while let Some(value) = access.next_element()? {\n            entries.push(value);\n        }\n        Ok(entries)\n    }\n}\nimpl<T: ValueDeserialize> ValueDeserialize for Vec<T> {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_any(VecVisitor(PhantomData))\n    }\n}\n\nstruct BTreeMapVisitor<T: ValueDeserialize>(PhantomData<T>);\nimpl<T: ValueDeserialize> ValueVisitor for BTreeMapVisitor<T> {\n    type Value = BTreeMap<String, T>;\n\n    fn visit_object<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n    where A: ObjectAccess<'de> {\n        let mut entries = BTreeMap::new();\n        while let Some((key, value)) = access.next_entry()? {\n            entries.insert(key, value);\n        }\n        Ok(entries)\n    }\n}\nimpl<T: ValueDeserialize> ValueDeserialize for BTreeMap<String, T> {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_any(BTreeMapVisitor(PhantomData))\n    }\n}\n\nstruct HashMapVisitor<T: ValueDeserialize>(PhantomData<T>);\nimpl<T: ValueDeserialize> ValueVisitor for HashMapVisitor<T> {\n    type Value = HashMap<String, T>;\n\n    fn visit_object<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n    where A: ObjectAccess<'de> {\n        let mut entries = HashMap::with_capacity(access.size_hint());\n        while let Some((key, value)) = access.next_entry()? {\n            entries.insert(key, value);\n        }\n        Ok(entries)\n    }\n}\nimpl<T: ValueDeserialize> ValueDeserialize for HashMap<String, T> {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_any(HashMapVisitor(PhantomData))\n    }\n}\n\nstruct KeyValuesVecVisitor<T: ValueDeserialize>(PhantomData<T>);\nimpl<T: ValueDeserialize> ValueVisitor for KeyValuesVecVisitor<T> {\n    type Value = Vec<(String, T)>;\n\n    fn visit_object<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n    where A: ObjectAccess<'de> {\n        let mut entries = Vec::with_capacity(access.size_hint());\n        while let Some(entry) = access.next_entry()? {\n            entries.push(entry);\n        }\n        Ok(entries)\n    }\n}\nimpl<T: ValueDeserialize> ValueDeserialize for Vec<(String, T)> {\n    #[inline]\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        deserializer.deserialize_any(KeyValuesVecVisitor(PhantomData))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io::Cursor;\n\n    use serde_json::Number;\n    use tokenizer_api::Token;\n\n    use super::*;\n    use crate::schema::document::existing_type_impls::JsonObjectIter;\n    use crate::schema::document::se::BinaryValueSerializer;\n    use crate::schema::document::{ReferenceValue, ReferenceValueLeaf};\n    use crate::store::DOC_STORE_VERSION;\n\n    fn serialize_value<'a>(value: ReferenceValue<'a, &'a serde_json::Value>) -> Vec<u8> {\n        let mut writer = Vec::new();\n\n        let mut serializer = BinaryValueSerializer::new(&mut writer);\n        serializer.serialize_value(value).expect(\"Serialize value\");\n\n        writer\n    }\n\n    fn serialize_owned_value<'a>(value: ReferenceValue<'a, &'a OwnedValue>) -> Vec<u8> {\n        let mut writer = Vec::new();\n\n        let mut serializer = BinaryValueSerializer::new(&mut writer);\n        serializer.serialize_value(value).expect(\"Serialize value\");\n\n        writer\n    }\n\n    fn deserialize_value(buffer: Vec<u8>) -> crate::schema::OwnedValue {\n        let mut cursor = Cursor::new(buffer);\n        let deserializer =\n            BinaryValueDeserializer::from_reader(&mut cursor, DOC_STORE_VERSION).unwrap();\n        crate::schema::OwnedValue::deserialize(deserializer).expect(\"Deserialize value\")\n    }\n\n    #[test]\n    fn test_simple_value_serialize() {\n        let result = serialize_value(ReferenceValueLeaf::Null.into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::Null);\n\n        let result = serialize_value(ReferenceValueLeaf::Str(\"hello, world\").into());\n        let value = deserialize_value(result);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Str(String::from(\"hello, world\"))\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::U64(123).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::U64(123));\n\n        let result = serialize_value(ReferenceValueLeaf::I64(-123).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::I64(-123));\n\n        let result = serialize_value(ReferenceValueLeaf::F64(123.3845).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::F64(123.3845));\n\n        let result = serialize_value(ReferenceValueLeaf::Bool(false).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::Bool(false));\n\n        let result =\n            serialize_value(ReferenceValueLeaf::Date(DateTime::from_timestamp_micros(100)).into());\n        let value = deserialize_value(result);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Date(DateTime::from_timestamp_micros(100))\n        );\n\n        let facet = Facet::from_text(\"/hello/world\").unwrap();\n        let result = serialize_value(ReferenceValueLeaf::Facet(facet.encoded_str()).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::Facet(facet));\n\n        let pre_tok_str = PreTokenizedString {\n            text: \"hello, world\".to_string(),\n            tokens: vec![Token::default(), Token::default()],\n        };\n        let result =\n            serialize_value(ReferenceValueLeaf::PreTokStr(pre_tok_str.clone().into()).into());\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::PreTokStr(pre_tok_str));\n    }\n\n    #[test]\n    fn test_array_serialize() {\n        let elements = [serde_json::Value::Null, serde_json::Value::Null];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let value = deserialize_value(result);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Array(vec![\n                crate::schema::OwnedValue::Null,\n                crate::schema::OwnedValue::Null,\n            ]),\n        );\n\n        let elements = [\n            serde_json::Value::String(\"Hello, world\".into()),\n            serde_json::Value::String(\"Some demo\".into()),\n        ];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let value = deserialize_value(result);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Array(vec![\n                crate::schema::OwnedValue::Str(String::from(\"Hello, world\")),\n                crate::schema::OwnedValue::Str(String::from(\"Some demo\")),\n            ]),\n        );\n\n        let elements = [];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let value = deserialize_value(result);\n        assert_eq!(value, crate::schema::OwnedValue::Array(vec![]));\n\n        let elements = [\n            serde_json::Value::Null,\n            serde_json::Value::String(\"Hello, world\".into()),\n            serde_json::Value::Number(12345.into()),\n        ];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let value = deserialize_value(result);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Array(vec![\n                crate::schema::OwnedValue::Null,\n                crate::schema::OwnedValue::Str(String::from(\"Hello, world\")),\n                crate::schema::OwnedValue::I64(12345),\n            ]),\n        );\n    }\n\n    #[test]\n    fn test_object_serialize() {\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-first-key\".into(),\n            serde_json::Value::String(\"Hello\".into()),\n        );\n        object.insert(\"my-second-key\".into(), serde_json::Value::Null);\n        object.insert(\n            \"my-third-key\".into(),\n            serde_json::Value::Number(Number::from_f64(123.0).unwrap()),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let value = deserialize_value(result);\n\n        let mut expected_object = BTreeMap::new();\n        expected_object.insert(\n            \"my-first-key\".to_string(),\n            crate::schema::OwnedValue::Str(String::from(\"Hello\")),\n        );\n        expected_object.insert(\"my-second-key\".to_string(), crate::schema::OwnedValue::Null);\n        expected_object.insert(\n            \"my-third-key\".to_string(),\n            crate::schema::OwnedValue::F64(123.0),\n        );\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Object(expected_object.into_iter().collect())\n        );\n\n        let object = serde_json::Map::new();\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let value = deserialize_value(result);\n        let expected_object = BTreeMap::new();\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Object(expected_object.into_iter().collect())\n        );\n\n        let mut object = serde_json::Map::new();\n        object.insert(\"my-first-key\".into(), serde_json::Value::Null);\n        object.insert(\"my-second-key\".into(), serde_json::Value::Null);\n        object.insert(\"my-third-key\".into(), serde_json::Value::Null);\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let value = deserialize_value(result);\n        let mut expected_object = BTreeMap::new();\n        expected_object.insert(\"my-first-key\".to_string(), crate::schema::OwnedValue::Null);\n        expected_object.insert(\"my-second-key\".to_string(), crate::schema::OwnedValue::Null);\n        expected_object.insert(\"my-third-key\".to_string(), crate::schema::OwnedValue::Null);\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Object(expected_object.into_iter().collect())\n        );\n    }\n\n    #[test]\n    fn test_json_compat() {\n        let data = [\n            8, 123, 34, 107, 101, 121, 97, 58, 34, 58, 34, 98, 108, 117, 98, 34, 44, 34, 118, 97,\n            108, 115, 34, 58, 123, 34, 104, 101, 121, 34, 58, 34, 104, 111, 34, 125, 125,\n        ]\n        .to_vec();\n        let expected = json!({\n            \"keya:\": \"blub\",\n            \"vals\": {\n                \"hey\": \"ho\"\n            }\n        });\n        let expected_val: OwnedValue = expected.clone().into();\n\n        let value = deserialize_value(data);\n        assert_eq!(value, expected_val);\n    }\n\n    #[test]\n    fn test_nested_date_precision() {\n        let object = OwnedValue::Object(vec![(\n            \"my-date\".into(),\n            OwnedValue::Date(DateTime::from_timestamp_nanos(323456)),\n        )]);\n        let result = serialize_owned_value((&object).as_value());\n        let value = deserialize_value(result);\n        assert_eq!(value, object);\n    }\n\n    #[test]\n    fn test_nested_serialize() {\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-array\".into(),\n            serde_json::Value::Array(vec![\n                serde_json::Value::Null,\n                serde_json::Value::String(String::from(\"bobby of the sea\")),\n            ]),\n        );\n        object.insert(\n            \"my-object\".into(),\n            serde_json::Value::Object(\n                vec![\n                    (\n                        \"inner-1\".to_string(),\n                        serde_json::Value::Number((-123i64).into()),\n                    ),\n                    (\n                        \"inner-2\".to_string(),\n                        serde_json::Value::String(String::from(\"bobby of the sea 2\")),\n                    ),\n                ]\n                .into_iter()\n                .collect(),\n            ),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let value = deserialize_value(result);\n\n        let mut expected_object = BTreeMap::new();\n        expected_object.insert(\n            \"my-array\".to_string(),\n            crate::schema::OwnedValue::Array(vec![\n                crate::schema::OwnedValue::Null,\n                crate::schema::OwnedValue::Str(String::from(\"bobby of the sea\")),\n            ]),\n        );\n        expected_object.insert(\n            \"my-object\".to_string(),\n            crate::schema::OwnedValue::Object(\n                vec![\n                    (\n                        \"inner-1\".to_string(),\n                        crate::schema::OwnedValue::I64(-123i64),\n                    ),\n                    (\n                        \"inner-2\".to_string(),\n                        crate::schema::OwnedValue::Str(String::from(\"bobby of the sea 2\")),\n                    ),\n                ]\n                .into_iter()\n                .collect(),\n            ),\n        );\n        assert_eq!(\n            value,\n            crate::schema::OwnedValue::Object(expected_object.into_iter().collect())\n        );\n\n        // Some more extreme nesting that might behave weirdly\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-array\".into(),\n            serde_json::Value::Array(vec![serde_json::Value::Array(vec![\n                serde_json::Value::Array(vec![]),\n                serde_json::Value::Array(vec![serde_json::Value::Null]),\n            ])]),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let value = deserialize_value(result);\n\n        let mut expected_object = BTreeMap::new();\n        expected_object.insert(\n            \"my-array\".to_string(),\n            OwnedValue::Array(vec![OwnedValue::Array(vec![\n                OwnedValue::Array(vec![]),\n                OwnedValue::Array(vec![OwnedValue::Null]),\n            ])]),\n        );\n        assert_eq!(\n            value,\n            OwnedValue::Object(expected_object.into_iter().collect())\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/document/default_document.rs",
    "content": "use std::collections::{BTreeMap, HashMap, HashSet};\nuse std::io::{self, Read, Write};\nuse std::net::Ipv6Addr;\n\nuse columnar::MonotonicallyMappableToU128;\nuse common::{read_u32_vint_no_advance, serialize_vint_u32, BinarySerializable, DateTime, VInt};\nuse serde_json::Map;\npub use CompactDoc as TantivyDocument;\n\nuse super::{ReferenceValue, ReferenceValueLeaf, Value};\nuse crate::schema::document::{\n    DeserializeError, Document, DocumentDeserialize, DocumentDeserializer,\n};\nuse crate::schema::field_type::ValueParsingError;\nuse crate::schema::{Facet, Field, NamedFieldDocument, OwnedValue, Schema};\nuse crate::tokenizer::PreTokenizedString;\n\n#[repr(C, packed)]\n#[derive(Debug, Clone)]\n/// A field value pair in the compact tantivy document\nstruct FieldValueAddr {\n    pub field: u16,\n    pub value_addr: ValueAddr,\n}\n\n#[derive(Debug, Clone)]\n/// The default document in tantivy. It encodes data in a compact form.\npub struct CompactDoc {\n    /// `node_data` is a vec of bytes, where each value is serialized into bytes and stored. It\n    /// includes all the data of the document and also metadata like where the nodes are located\n    /// in an object or array.\n    pub node_data: Vec<u8>,\n    /// The root (Field, Value) pairs\n    field_values: Vec<FieldValueAddr>,\n}\n\nimpl Default for CompactDoc {\n    fn default() -> Self {\n        Self::new()\n    }\n}\n\nimpl CompactDoc {\n    /// Creates a new, empty document object\n    /// The reserved capacity is for the total serialized data\n    pub fn with_capacity(bytes: usize) -> CompactDoc {\n        CompactDoc {\n            node_data: Vec::with_capacity(bytes),\n            field_values: Vec::with_capacity(4),\n        }\n    }\n\n    /// Creates a new, empty document object\n    pub fn new() -> CompactDoc {\n        CompactDoc::with_capacity(1024)\n    }\n\n    /// Skrinks the capacity of the document to fit the data\n    pub fn shrink_to_fit(&mut self) {\n        self.node_data.shrink_to_fit();\n        self.field_values.shrink_to_fit();\n    }\n\n    /// Returns the length of the document.\n    pub fn len(&self) -> usize {\n        self.field_values.len()\n    }\n\n    /// Adding a facet to the document.\n    pub fn add_facet<F>(&mut self, field: Field, path: F)\n    where Facet: From<F> {\n        let facet = Facet::from(path);\n        self.add_leaf_field_value(field, ReferenceValueLeaf::Facet(facet.encoded_str()));\n    }\n\n    /// Add a text field.\n    pub fn add_text<S: AsRef<str>>(&mut self, field: Field, text: S) {\n        self.add_leaf_field_value(field, ReferenceValueLeaf::Str(text.as_ref()));\n    }\n\n    /// Add a pre-tokenized text field.\n    pub fn add_pre_tokenized_text(&mut self, field: Field, pre_tokenized_text: PreTokenizedString) {\n        self.add_leaf_field_value(field, pre_tokenized_text);\n    }\n\n    /// Add a u64 field\n    pub fn add_u64(&mut self, field: Field, value: u64) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a IP address field. Internally only Ipv6Addr is used.\n    pub fn add_ip_addr(&mut self, field: Field, value: Ipv6Addr) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a i64 field\n    pub fn add_i64(&mut self, field: Field, value: i64) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a f64 field\n    pub fn add_f64(&mut self, field: Field, value: f64) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a bool field\n    pub fn add_bool(&mut self, field: Field, value: bool) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a date field with unspecified time zone offset\n    pub fn add_date(&mut self, field: Field, value: DateTime) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a bytes field\n    pub fn add_bytes(&mut self, field: Field, value: &[u8]) {\n        self.add_leaf_field_value(field, value);\n    }\n\n    /// Add a dynamic object field\n    pub fn add_object(&mut self, field: Field, object: BTreeMap<String, OwnedValue>) {\n        self.add_field_value(field, &OwnedValue::from(object));\n    }\n\n    /// Add a (field, value) to the document.\n    ///\n    /// `OwnedValue` implements Value, which should be easiest to use, but is not the most\n    /// performant.\n    pub fn add_field_value<'a, V: Value<'a>>(&mut self, field: Field, value: V) {\n        let field_value = FieldValueAddr {\n            field: field\n                .field_id()\n                .try_into()\n                .expect(\"support only up to u16::MAX field ids\"),\n            value_addr: self.add_value(value),\n        };\n        self.field_values.push(field_value);\n    }\n\n    /// Add a (field, leaf value) to the document.\n    /// Leaf values don't have nested values.\n    pub fn add_leaf_field_value<'a, T: Into<ReferenceValueLeaf<'a>>>(\n        &mut self,\n        field: Field,\n        typed_val: T,\n    ) {\n        let value = typed_val.into();\n        let field_value = FieldValueAddr {\n            field: field\n                .field_id()\n                .try_into()\n                .expect(\"support only up to u16::MAX field ids\"),\n            value_addr: self.add_value_leaf(value),\n        };\n        self.field_values.push(field_value);\n    }\n\n    /// field_values accessor\n    pub fn field_values(&self) -> impl Iterator<Item = (Field, CompactDocValue<'_>)> {\n        self.field_values.iter().map(|field_val| {\n            let field = Field::from_field_id(field_val.field as u32);\n            let val = self.get_compact_doc_value(field_val.value_addr);\n            (field, val)\n        })\n    }\n\n    /// Returns all of the `ReferenceValue`s associated the given field\n    pub fn get_all(&self, field: Field) -> impl Iterator<Item = CompactDocValue<'_>> + '_ {\n        self.field_values\n            .iter()\n            .filter(move |field_value| Field::from_field_id(field_value.field as u32) == field)\n            .map(|val| self.get_compact_doc_value(val.value_addr))\n    }\n\n    /// Returns the first `ReferenceValue` associated the given field\n    pub fn get_first(&self, field: Field) -> Option<CompactDocValue<'_>> {\n        self.get_all(field).next()\n    }\n\n    /// Create document from a named doc.\n    pub fn convert_named_doc(\n        schema: &Schema,\n        named_doc: NamedFieldDocument,\n    ) -> Result<Self, DocParsingError> {\n        let mut document = Self::new();\n        for (field_name, values) in named_doc.0 {\n            if let Ok(field) = schema.get_field(&field_name) {\n                for value in values {\n                    document.add_field_value(field, &value);\n                }\n            }\n        }\n        Ok(document)\n    }\n\n    /// Build a document object from a json-object.\n    pub fn parse_json(schema: &Schema, doc_json: &str) -> Result<Self, DocParsingError> {\n        let json_obj: Map<String, serde_json::Value> =\n            serde_json::from_str(doc_json).map_err(|_| DocParsingError::invalid_json(doc_json))?;\n        Self::from_json_object(schema, json_obj)\n    }\n\n    /// Build a document object from a json-object.\n    pub fn from_json_object(\n        schema: &Schema,\n        json_obj: Map<String, serde_json::Value>,\n    ) -> Result<Self, DocParsingError> {\n        let mut doc = Self::default();\n        for (field_name, json_value) in json_obj {\n            if let Ok(field) = schema.get_field(&field_name) {\n                let field_entry = schema.get_field_entry(field);\n                let field_type = field_entry.field_type();\n                match json_value {\n                    serde_json::Value::Array(json_items) => {\n                        for json_item in json_items {\n                            let value = field_type\n                                .value_from_json(json_item)\n                                .map_err(|e| DocParsingError::ValueError(field_name.clone(), e))?;\n                            doc.add_field_value(field, &value);\n                        }\n                    }\n                    _ => {\n                        let value = field_type\n                            .value_from_json(json_value)\n                            .map_err(|e| DocParsingError::ValueError(field_name.clone(), e))?;\n                        doc.add_field_value(field, &value);\n                    }\n                }\n            }\n        }\n        Ok(doc)\n    }\n\n    fn add_value_leaf(&mut self, leaf: ReferenceValueLeaf) -> ValueAddr {\n        let type_id = ValueType::from(&leaf);\n        // Write into `node_data` and return u32 position as its address\n        // Null and bool are inlined into the address\n        let val_addr = match leaf {\n            ReferenceValueLeaf::Null => 0,\n            ReferenceValueLeaf::Str(bytes) => {\n                write_bytes_into(&mut self.node_data, bytes.as_bytes())\n            }\n            ReferenceValueLeaf::Facet(bytes) => {\n                write_bytes_into(&mut self.node_data, bytes.as_bytes())\n            }\n            ReferenceValueLeaf::Bytes(bytes) => write_bytes_into(&mut self.node_data, bytes),\n            ReferenceValueLeaf::U64(num) => write_into(&mut self.node_data, num),\n            ReferenceValueLeaf::I64(num) => write_into(&mut self.node_data, num),\n            ReferenceValueLeaf::F64(num) => write_into(&mut self.node_data, num),\n            ReferenceValueLeaf::Bool(b) => b as u32,\n            ReferenceValueLeaf::Date(date) => {\n                write_into(&mut self.node_data, date.into_timestamp_nanos())\n            }\n            ReferenceValueLeaf::IpAddr(num) => write_into(&mut self.node_data, num.to_u128()),\n            ReferenceValueLeaf::PreTokStr(pre_tok) => write_into(&mut self.node_data, *pre_tok),\n        };\n        ValueAddr { type_id, val_addr }\n    }\n    /// Adds a value and returns in address into the\n    fn add_value<'a, V: Value<'a>>(&mut self, value: V) -> ValueAddr {\n        let value = value.as_value();\n        let type_id = ValueType::from(&value);\n        match value {\n            ReferenceValue::Leaf(leaf) => self.add_value_leaf(leaf),\n            ReferenceValue::Array(elements) => {\n                // addresses of the elements in node_data\n                // Reusing a vec would be nicer, but it's not easy because of the recursion\n                // A global vec would work if every writer get it's discriminator\n                let mut addresses = Vec::new();\n                for elem in elements {\n                    let value_addr = self.add_value(elem);\n                    write_into(&mut addresses, value_addr);\n                }\n                ValueAddr {\n                    type_id,\n                    val_addr: write_bytes_into(&mut self.node_data, &addresses),\n                }\n            }\n            ReferenceValue::Object(entries) => {\n                // addresses of the elements in node_data\n                let mut addresses = Vec::new();\n                for (key, value) in entries {\n                    let key_addr = self.add_value_leaf(ReferenceValueLeaf::Str(key));\n                    let value_addr = self.add_value(value);\n                    write_into(&mut addresses, key_addr);\n                    write_into(&mut addresses, value_addr);\n                }\n                ValueAddr {\n                    type_id,\n                    val_addr: write_bytes_into(&mut self.node_data, &addresses),\n                }\n            }\n        }\n    }\n\n    /// Get CompactDocValue for address\n    fn get_compact_doc_value(&self, value_addr: ValueAddr) -> CompactDocValue<'_> {\n        CompactDocValue {\n            container: self,\n            value_addr,\n        }\n    }\n\n    /// get &[u8] reference from node_data\n    fn extract_bytes(&self, addr: Addr) -> &[u8] {\n        binary_deserialize_bytes(self.get_slice(addr))\n    }\n\n    /// get &str reference from node_data\n    fn extract_str(&self, addr: Addr) -> &str {\n        let data = self.extract_bytes(addr);\n        // Utf-8 checks would have a noticeable performance overhead here\n        unsafe { std::str::from_utf8_unchecked(data) }\n    }\n\n    /// deserialized owned value from node_data\n    fn read_from<T: BinarySerializable>(&self, addr: Addr) -> io::Result<T> {\n        let data_slice = &self.node_data[addr as usize..];\n        let mut cursor = std::io::Cursor::new(data_slice);\n        T::deserialize(&mut cursor)\n    }\n\n    /// get slice from address. The returned slice is open ended\n    fn get_slice(&self, addr: Addr) -> &[u8] {\n        &self.node_data[addr as usize..]\n    }\n}\n\n/// BinarySerializable alternative to read references\nfn binary_deserialize_bytes(data: &[u8]) -> &[u8] {\n    let (len, bytes_read) = read_u32_vint_no_advance(data);\n    &data[bytes_read..bytes_read + len as usize]\n}\n\n/// Write bytes and return the position of the written data.\n///\n/// BinarySerializable alternative to write references\nfn write_bytes_into(vec: &mut Vec<u8>, data: &[u8]) -> u32 {\n    let pos = vec.len() as u32;\n    let mut buf = [0u8; 8];\n    let len_vint_bytes = serialize_vint_u32(data.len() as u32, &mut buf);\n    vec.extend_from_slice(len_vint_bytes);\n    vec.extend_from_slice(data);\n    pos\n}\n\n/// Serialize and return the position\nfn write_into<T: BinarySerializable>(vec: &mut Vec<u8>, value: T) -> u32 {\n    let pos = vec.len() as u32;\n    value.serialize(vec).unwrap();\n    pos\n}\n\nimpl PartialEq for CompactDoc {\n    fn eq(&self, other: &Self) -> bool {\n        // super slow, but only here for tests\n        let convert_to_comparable_map = |doc: &CompactDoc| {\n            let mut field_value_set: HashMap<Field, HashSet<String>> = Default::default();\n            for field_value in doc.field_values.iter() {\n                let value: OwnedValue = doc.get_compact_doc_value(field_value.value_addr).into();\n                let value = serde_json::to_string(&value).unwrap();\n                field_value_set\n                    .entry(Field::from_field_id(field_value.field as u32))\n                    .or_default()\n                    .insert(value);\n            }\n            field_value_set\n        };\n        let self_field_values: HashMap<Field, HashSet<String>> = convert_to_comparable_map(self);\n        let other_field_values: HashMap<Field, HashSet<String>> = convert_to_comparable_map(other);\n        self_field_values.eq(&other_field_values)\n    }\n}\n\nimpl Eq for CompactDoc {}\n\nimpl DocumentDeserialize for CompactDoc {\n    fn deserialize<'de, D>(mut deserializer: D) -> Result<Self, DeserializeError>\n    where D: DocumentDeserializer<'de> {\n        let mut doc = CompactDoc::default();\n        // TODO: Deserializing into OwnedValue is wasteful. The deserializer should be able to work\n        // on slices and referenced data.\n        while let Some((field, value)) = deserializer.next_field::<OwnedValue>()? {\n            doc.add_field_value(field, &value);\n        }\n        Ok(doc)\n    }\n}\n\n/// A value of Compact Doc needs a reference to the container to extract its payload\n#[derive(Debug, Clone, Copy)]\npub struct CompactDocValue<'a> {\n    container: &'a CompactDoc,\n    value_addr: ValueAddr,\n}\nimpl PartialEq for CompactDocValue<'_> {\n    fn eq(&self, other: &Self) -> bool {\n        let value1: OwnedValue = (*self).into();\n        let value2: OwnedValue = (*other).into();\n        value1 == value2\n    }\n}\nimpl From<CompactDocValue<'_>> for OwnedValue {\n    fn from(value: CompactDocValue) -> Self {\n        value.as_value().into()\n    }\n}\nimpl<'a> Value<'a> for CompactDocValue<'a> {\n    type ArrayIter = CompactDocArrayIter<'a>;\n\n    type ObjectIter = CompactDocObjectIter<'a>;\n\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        self.get_ref_value().unwrap()\n    }\n}\nimpl<'a> CompactDocValue<'a> {\n    fn get_ref_value(&self) -> io::Result<ReferenceValue<'a, CompactDocValue<'a>>> {\n        let addr = self.value_addr.val_addr;\n        match self.value_addr.type_id {\n            ValueType::Null => Ok(ReferenceValueLeaf::Null.into()),\n            ValueType::Str => {\n                let str_ref = self.container.extract_str(addr);\n                Ok(ReferenceValueLeaf::Str(str_ref).into())\n            }\n            ValueType::Facet => {\n                let str_ref = self.container.extract_str(addr);\n                Ok(ReferenceValueLeaf::Facet(str_ref).into())\n            }\n            ValueType::Bytes => {\n                let data = self.container.extract_bytes(addr);\n                Ok(ReferenceValueLeaf::Bytes(data).into())\n            }\n            ValueType::U64 => self\n                .container\n                .read_from::<u64>(addr)\n                .map(ReferenceValueLeaf::U64)\n                .map(Into::into),\n            ValueType::I64 => self\n                .container\n                .read_from::<i64>(addr)\n                .map(ReferenceValueLeaf::I64)\n                .map(Into::into),\n            ValueType::F64 => self\n                .container\n                .read_from::<f64>(addr)\n                .map(ReferenceValueLeaf::F64)\n                .map(Into::into),\n            ValueType::Bool => Ok(ReferenceValueLeaf::Bool(addr != 0).into()),\n            ValueType::Date => self\n                .container\n                .read_from::<i64>(addr)\n                .map(|ts| ReferenceValueLeaf::Date(DateTime::from_timestamp_nanos(ts)))\n                .map(Into::into),\n            ValueType::IpAddr => self\n                .container\n                .read_from::<u128>(addr)\n                .map(|num| ReferenceValueLeaf::IpAddr(Ipv6Addr::from_u128(num)))\n                .map(Into::into),\n            ValueType::PreTokStr => self\n                .container\n                .read_from::<PreTokenizedString>(addr)\n                .map(Into::into)\n                .map(ReferenceValueLeaf::PreTokStr)\n                .map(Into::into),\n            ValueType::Object => Ok(ReferenceValue::Object(CompactDocObjectIter::new(\n                self.container,\n                addr,\n            )?)),\n            ValueType::Array => Ok(ReferenceValue::Array(CompactDocArrayIter::new(\n                self.container,\n                addr,\n            )?)),\n        }\n    }\n}\n\n/// The address in the vec\ntype Addr = u32;\n\n#[derive(Clone, Copy, Default)]\n#[repr(C, packed)]\n/// The value type and the address to its payload in the container.\nstruct ValueAddr {\n    type_id: ValueType,\n    /// This is the address to the value in the vec, except for bool and null, which are inlined\n    val_addr: Addr,\n}\nimpl BinarySerializable for ValueAddr {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.type_id.serialize(writer)?;\n        VInt(self.val_addr as u64).serialize(writer)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let type_id = ValueType::deserialize(reader)?;\n        let val_addr = VInt::deserialize(reader)?.0 as u32;\n        Ok(ValueAddr { type_id, val_addr })\n    }\n}\nimpl std::fmt::Debug for ValueAddr {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        let val_addr = self.val_addr;\n        f.write_fmt(format_args!(\"{:?} at {:?}\", self.type_id, val_addr))\n    }\n}\n\n/// A enum representing a value for tantivy to index.\n///\n/// ** Any changes need to be reflected in `BinarySerializable` for `ValueType` **\n///\n/// We can't use [schema::Type] or [columnar::ColumnType] here, because they are missing\n/// some items like Array and PreTokStr.\n#[derive(Default, Clone, Copy, Debug, PartialEq)]\n#[repr(u8)]\npub enum ValueType {\n    /// A null value.\n    #[default]\n    Null = 0,\n    /// The str type is used for any text information.\n    Str = 1,\n    /// Unsigned 64-bits Integer `u64`\n    U64 = 2,\n    /// Signed 64-bits Integer `i64`\n    I64 = 3,\n    /// 64-bits Float `f64`\n    F64 = 4,\n    /// Date/time with nanoseconds precision\n    Date = 5,\n    /// Facet\n    Facet = 6,\n    /// Arbitrarily sized byte array\n    Bytes = 7,\n    /// IpV6 Address. Internally there is no IpV4, it needs to be converted to `Ipv6Addr`.\n    IpAddr = 8,\n    /// Bool value\n    Bool = 9,\n    /// Pre-tokenized str type,\n    PreTokStr = 10,\n    /// Object\n    Object = 11,\n    /// Pre-tokenized str type,\n    Array = 12,\n}\n\nimpl BinarySerializable for ValueType {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        (*self as u8).serialize(writer)?;\n        Ok(())\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let num = u8::deserialize(reader)?;\n        let type_id = if (0..=12).contains(&num) {\n            unsafe { std::mem::transmute::<u8, ValueType>(num) }\n        } else {\n            return Err(io::Error::new(\n                io::ErrorKind::InvalidData,\n                format!(\"Invalid value type id: {num}\"),\n            ));\n        };\n        Ok(type_id)\n    }\n}\n\nimpl<'a, V: Value<'a>> From<&ReferenceValue<'a, V>> for ValueType {\n    fn from(value: &ReferenceValue<'a, V>) -> Self {\n        match value {\n            ReferenceValue::Leaf(leaf) => leaf.into(),\n            ReferenceValue::Array(_) => ValueType::Array,\n            ReferenceValue::Object(_) => ValueType::Object,\n        }\n    }\n}\nimpl<'a> From<&ReferenceValueLeaf<'a>> for ValueType {\n    fn from(value: &ReferenceValueLeaf<'a>) -> Self {\n        match value {\n            ReferenceValueLeaf::Null => ValueType::Null,\n            ReferenceValueLeaf::Str(_) => ValueType::Str,\n            ReferenceValueLeaf::U64(_) => ValueType::U64,\n            ReferenceValueLeaf::I64(_) => ValueType::I64,\n            ReferenceValueLeaf::F64(_) => ValueType::F64,\n            ReferenceValueLeaf::Bool(_) => ValueType::Bool,\n            ReferenceValueLeaf::Date(_) => ValueType::Date,\n            ReferenceValueLeaf::IpAddr(_) => ValueType::IpAddr,\n            ReferenceValueLeaf::PreTokStr(_) => ValueType::PreTokStr,\n            ReferenceValueLeaf::Facet(_) => ValueType::Facet,\n            ReferenceValueLeaf::Bytes(_) => ValueType::Bytes,\n        }\n    }\n}\n\n#[derive(Debug, Clone)]\n/// The Iterator for the object values in the compact document\npub struct CompactDocObjectIter<'a> {\n    container: &'a CompactDoc,\n    node_addresses_slice: &'a [u8],\n}\n\nimpl<'a> CompactDocObjectIter<'a> {\n    fn new(container: &'a CompactDoc, addr: Addr) -> io::Result<Self> {\n        // Objects are `&[ValueAddr]` serialized into bytes\n        let node_addresses_slice = container.extract_bytes(addr);\n        Ok(Self {\n            container,\n            node_addresses_slice,\n        })\n    }\n}\n\nimpl<'a> Iterator for CompactDocObjectIter<'a> {\n    type Item = (&'a str, CompactDocValue<'a>);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        if self.node_addresses_slice.is_empty() {\n            return None;\n        }\n        let key_addr = ValueAddr::deserialize(&mut self.node_addresses_slice).ok()?;\n        let key = self.container.extract_str(key_addr.val_addr);\n        let value = ValueAddr::deserialize(&mut self.node_addresses_slice).ok()?;\n        let value = CompactDocValue {\n            container: self.container,\n            value_addr: value,\n        };\n        Some((key, value))\n    }\n}\n\n#[derive(Debug, Clone)]\n/// The Iterator for the array values in the compact document\npub struct CompactDocArrayIter<'a> {\n    container: &'a CompactDoc,\n    node_addresses_slice: &'a [u8],\n}\n\nimpl<'a> CompactDocArrayIter<'a> {\n    fn new(container: &'a CompactDoc, addr: Addr) -> io::Result<Self> {\n        // Arrays are &[ValueAddr] serialized into bytes\n        let node_addresses_slice = container.extract_bytes(addr);\n        Ok(Self {\n            container,\n            node_addresses_slice,\n        })\n    }\n}\n\nimpl<'a> Iterator for CompactDocArrayIter<'a> {\n    type Item = CompactDocValue<'a>;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        if self.node_addresses_slice.is_empty() {\n            return None;\n        }\n        let value = ValueAddr::deserialize(&mut self.node_addresses_slice).ok()?;\n        let value = CompactDocValue {\n            container: self.container,\n            value_addr: value,\n        };\n        Some(value)\n    }\n}\n\nimpl Document for CompactDoc {\n    type Value<'a> = CompactDocValue<'a>;\n    type FieldsValuesIter<'a> = FieldValueIterRef<'a>;\n\n    fn iter_fields_and_values(&self) -> Self::FieldsValuesIter<'_> {\n        FieldValueIterRef {\n            slice: self.field_values.iter(),\n            container: self,\n        }\n    }\n}\n\n/// A helper wrapper for creating an iterator over the field values\npub struct FieldValueIterRef<'a> {\n    slice: std::slice::Iter<'a, FieldValueAddr>,\n    container: &'a CompactDoc,\n}\n\nimpl<'a> Iterator for FieldValueIterRef<'a> {\n    type Item = (Field, CompactDocValue<'a>);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        self.slice.next().map(|field_value| {\n            (\n                Field::from_field_id(field_value.field as u32),\n                CompactDocValue::<'a> {\n                    container: self.container,\n                    value_addr: field_value.value_addr,\n                },\n            )\n        })\n    }\n}\n\n/// Error that may happen when deserializing\n/// a document from JSON.\n#[derive(Debug, Error, PartialEq)]\npub enum DocParsingError {\n    /// The payload given is not valid JSON.\n    #[error(\"The provided string is not valid JSON\")]\n    InvalidJson(String),\n    /// One of the value node could not be parsed.\n    #[error(\"The field '{0:?}' could not be parsed: {1:?}\")]\n    ValueError(String, ValueParsingError),\n}\n\nimpl DocParsingError {\n    /// Builds a NotJson DocParsingError\n    fn invalid_json(invalid_json: &str) -> Self {\n        let sample = invalid_json.chars().take(20).collect();\n        DocParsingError::InvalidJson(sample)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::schema::*;\n\n    #[test]\n    fn test_doc() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"title\", TEXT);\n        let mut doc = TantivyDocument::default();\n        doc.add_text(text_field, \"My title\");\n        assert_eq!(doc.field_values().count(), 1);\n\n        let schema = schema_builder.build();\n        let _val = doc.get_first(text_field).unwrap();\n        let _json = doc.to_named_doc(&schema);\n    }\n\n    #[test]\n    fn test_json_value() {\n        let json_str = r#\"{\n            \"toto\": \"titi\",\n            \"float\": -0.2,\n            \"bool\": true,\n            \"unsigned\": 1,\n            \"signed\": -2,\n            \"complexobject\": {\n                \"field.with.dot\": 1\n            },\n            \"date\": \"1985-04-12T23:20:50.52Z\",\n            \"my_arr\": [2, 3, {\"my_key\": \"two tokens\"}, 4, {\"nested_array\": [2, 5, 6, [7, 8, {\"a\": [{\"d\": {\"e\":[99]}}, 9000]}, 9, 10], [5, 5]]}]\n        }\"#;\n        let json_val: std::collections::BTreeMap<String, OwnedValue> =\n            serde_json::from_str(json_str).unwrap();\n\n        let mut schema_builder = Schema::builder();\n        let json_field = schema_builder.add_json_field(\"json\", TEXT);\n        let mut doc = TantivyDocument::default();\n        doc.add_object(json_field, json_val);\n\n        let schema = schema_builder.build();\n        let json = doc.to_json(&schema);\n        let actual_json: serde_json::Value = serde_json::from_str(&json).unwrap();\n        let expected_json: serde_json::Value = serde_json::from_str(json_str).unwrap();\n        assert_eq!(actual_json[\"json\"][0], expected_json);\n    }\n\n    // TODO: Should this be re-added with the serialize method\n    //       technically this is no longer useful since the doc types\n    //       do not implement BinarySerializable due to orphan rules.\n    // #[test]\n    // fn test_doc_serialization_issue() {\n    //     let mut doc = Document::default();\n    //     doc.add_json_object(\n    //         Field::from_field_id(0),\n    //         serde_json::json!({\"key\": 2u64})\n    //             .as_object()\n    //             .unwrap()\n    //             .clone(),\n    //     );\n    //     doc.add_text(Field::from_field_id(1), \"hello\");\n    //     assert_eq!(doc.field_values().len(), 2);\n    //     let mut payload: Vec<u8> = Vec::new();\n    //     doc_binary_wrappers::serialize(&doc, &mut payload).unwrap();\n    //     assert_eq!(payload.len(), 26);\n    //     doc_binary_wrappers::deserialize::<Document, _>(&mut &payload[..]).unwrap();\n    // }\n}\n"
  },
  {
    "path": "src/schema/document/existing_type_impls.rs",
    "content": "//! Implementations of some of the core traits on various types to improve the ergonomics\n//! of the API when providing custom documents.\n//!\n//! This allows users a bit more freedom and ergonomics if they want a simple API\n//! and don't care about some of the more specialised types or only want to customise\n//! part of the document structure.\nuse std::collections::{btree_map, hash_map, BTreeMap, HashMap};\nuse std::iter::Empty;\nuse std::net::Ipv6Addr;\n\nuse common::DateTime;\nuse serde_json::Number;\nuse time::format_description::well_known::Rfc3339;\nuse time::OffsetDateTime;\n\nuse super::facet::Facet;\nuse super::ReferenceValueLeaf;\nuse crate::schema::document::{\n    ArrayAccess, DeserializeError, Document, DocumentDeserialize, DocumentDeserializer,\n    ObjectAccess, ReferenceValue, Value, ValueDeserialize, ValueDeserializer, ValueVisitor,\n};\nuse crate::schema::Field;\nuse crate::tokenizer::PreTokenizedString;\n\n// Serde compatibility support.\npub fn can_be_rfc3339_date_time(text: &str) -> bool {\n    if let Some(&first_byte) = text.as_bytes().first() {\n        if first_byte.is_ascii_digit() {\n            return true;\n        }\n    }\n\n    false\n}\n\nimpl<'a> Value<'a> for &'a serde_json::Value {\n    type ArrayIter = std::slice::Iter<'a, serde_json::Value>;\n    type ObjectIter = JsonObjectIter<'a>;\n\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        match self {\n            serde_json::Value::Null => ReferenceValueLeaf::Null.into(),\n            serde_json::Value::Bool(value) => ReferenceValueLeaf::Bool(*value).into(),\n            serde_json::Value::Number(number) => {\n                if let Some(val) = number.as_i64() {\n                    ReferenceValueLeaf::I64(val).into()\n                } else if let Some(val) = number.as_u64() {\n                    ReferenceValueLeaf::U64(val).into()\n                } else if let Some(val) = number.as_f64() {\n                    ReferenceValueLeaf::F64(val).into()\n                } else {\n                    panic!(\"Unsupported serde_json number {number}\");\n                }\n            }\n            serde_json::Value::String(text) => {\n                if can_be_rfc3339_date_time(text) {\n                    match OffsetDateTime::parse(text, &Rfc3339) {\n                        Ok(dt) => {\n                            let dt_utc = dt.to_offset(time::UtcOffset::UTC);\n                            ReferenceValueLeaf::Date(DateTime::from_utc(dt_utc)).into()\n                        }\n                        Err(_) => ReferenceValueLeaf::Str(text).into(),\n                    }\n                } else {\n                    ReferenceValueLeaf::Str(text).into()\n                }\n            }\n            serde_json::Value::Array(elements) => ReferenceValue::Array(elements.iter()),\n            serde_json::Value::Object(object) => {\n                ReferenceValue::Object(JsonObjectIter(object.iter()))\n            }\n        }\n    }\n}\n\nimpl<'a> Value<'a> for &'a String {\n    type ArrayIter = Empty<&'a String>;\n    type ObjectIter = Empty<(&'a str, &'a String)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Str(self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a Facet {\n    type ArrayIter = Empty<&'a Facet>;\n    type ObjectIter = Empty<(&'a str, &'a Facet)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Facet(self.encoded_str()))\n    }\n}\n\nimpl<'a> Value<'a> for &'a u64 {\n    type ArrayIter = Empty<&'a u64>;\n    type ObjectIter = Empty<(&'a str, &'a u64)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::U64(**self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a i64 {\n    type ArrayIter = Empty<&'a i64>;\n    type ObjectIter = Empty<(&'a str, &'a i64)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::I64(**self))\n    }\n}\nimpl<'a> Value<'a> for &'a f64 {\n    type ArrayIter = Empty<&'a f64>;\n    type ObjectIter = Empty<(&'a str, &'a f64)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::F64(**self))\n    }\n}\nimpl<'a> Value<'a> for &'a bool {\n    type ArrayIter = Empty<&'a bool>;\n    type ObjectIter = Empty<(&'a str, &'a bool)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Bool(**self))\n    }\n}\nimpl<'a> Value<'a> for &'a str {\n    type ArrayIter = Empty<&'a str>;\n    type ObjectIter = Empty<(&'a str, &'a str)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Str(self))\n    }\n}\nimpl<'a> Value<'a> for &'a &'a str {\n    type ArrayIter = Empty<&'a &'a str>;\n    type ObjectIter = Empty<(&'a str, &'a &'a str)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Str(self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a [u8] {\n    type ArrayIter = Empty<&'a [u8]>;\n    type ObjectIter = Empty<(&'a str, &'a [u8])>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Bytes(self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a &'a [u8] {\n    type ArrayIter = Empty<&'a &'a [u8]>;\n    type ObjectIter = Empty<(&'a str, &'a &'a [u8])>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Bytes(self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a Vec<u8> {\n    type ArrayIter = Empty<&'a Vec<u8>>;\n    type ObjectIter = Empty<(&'a str, &'a Vec<u8>)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Bytes(self))\n    }\n}\n\nimpl<'a> Value<'a> for &'a DateTime {\n    type ArrayIter = Empty<&'a DateTime>;\n    type ObjectIter = Empty<(&'a str, &'a DateTime)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::Date(**self))\n    }\n}\nimpl<'a> Value<'a> for &'a Ipv6Addr {\n    type ArrayIter = Empty<&'a Ipv6Addr>;\n    type ObjectIter = Empty<(&'a str, &'a Ipv6Addr)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::IpAddr(**self))\n    }\n}\nimpl<'a> Value<'a> for &'a PreTokenizedString {\n    type ArrayIter = Empty<&'a PreTokenizedString>;\n    type ObjectIter = Empty<(&'a str, &'a PreTokenizedString)>;\n    #[inline]\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        ReferenceValue::Leaf(ReferenceValueLeaf::PreTokStr(Box::new((*self).clone())))\n    }\n}\n\nimpl ValueDeserialize for serde_json::Value {\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        struct SerdeValueVisitor;\n\n        impl ValueVisitor for SerdeValueVisitor {\n            type Value = serde_json::Value;\n\n            fn visit_null(&self) -> Result<Self::Value, DeserializeError> {\n                Ok(serde_json::Value::Null)\n            }\n\n            fn visit_string(&self, val: String) -> Result<Self::Value, DeserializeError> {\n                Ok(serde_json::Value::String(val))\n            }\n\n            fn visit_u64(&self, val: u64) -> Result<Self::Value, DeserializeError> {\n                Ok(serde_json::Value::Number(val.into()))\n            }\n\n            fn visit_i64(&self, val: i64) -> Result<Self::Value, DeserializeError> {\n                Ok(serde_json::Value::Number(val.into()))\n            }\n\n            fn visit_f64(&self, val: f64) -> Result<Self::Value, DeserializeError> {\n                let num = Number::from_f64(val).ok_or_else(|| {\n                    DeserializeError::custom(format!(\n                        \"serde_json::Value cannot deserialize float {val}\"\n                    ))\n                })?;\n                Ok(serde_json::Value::Number(num))\n            }\n\n            fn visit_bool(&self, val: bool) -> Result<Self::Value, DeserializeError> {\n                Ok(serde_json::Value::Bool(val))\n            }\n\n            fn visit_array<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n            where A: ArrayAccess<'de> {\n                let mut elements = Vec::with_capacity(access.size_hint());\n\n                while let Some(value) = access.next_element()? {\n                    elements.push(value);\n                }\n\n                Ok(serde_json::Value::Array(elements))\n            }\n\n            fn visit_object<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n            where A: ObjectAccess<'de> {\n                let mut object = serde_json::Map::with_capacity(access.size_hint());\n\n                while let Some((key, value)) = access.next_entry()? {\n                    object.insert(key, value);\n                }\n\n                Ok(serde_json::Value::Object(object))\n            }\n        }\n\n        deserializer.deserialize_any(SerdeValueVisitor)\n    }\n}\n\n/// A wrapper struct for an iterator producing [Value]s.\npub struct JsonObjectIter<'a>(pub(crate) serde_json::map::Iter<'a>);\n\nimpl<'a> Iterator for JsonObjectIter<'a> {\n    type Item = (&'a str, &'a serde_json::Value);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let (key, value) = self.0.next()?;\n        Some((key, value))\n    }\n}\n\n// Custom document types\n\n// BTreeMap based documents\nimpl Document for BTreeMap<Field, crate::schema::OwnedValue> {\n    type Value<'a> = &'a crate::schema::OwnedValue;\n    type FieldsValuesIter<'a> = FieldCopyingIterator<\n        'a,\n        btree_map::Iter<'a, Field, crate::schema::OwnedValue>,\n        crate::schema::OwnedValue,\n    >;\n\n    fn iter_fields_and_values(&self) -> Self::FieldsValuesIter<'_> {\n        FieldCopyingIterator(self.iter())\n    }\n}\nimpl DocumentDeserialize for BTreeMap<Field, crate::schema::OwnedValue> {\n    fn deserialize<'de, D>(mut deserializer: D) -> Result<Self, DeserializeError>\n    where D: DocumentDeserializer<'de> {\n        let mut document = BTreeMap::new();\n\n        while let Some((field, value)) = deserializer.next_field()? {\n            document.insert(field, value);\n        }\n\n        Ok(document)\n    }\n}\n\n// HashMap based documents\nimpl Document for HashMap<Field, crate::schema::OwnedValue> {\n    type Value<'a> = &'a crate::schema::OwnedValue;\n    type FieldsValuesIter<'a> = FieldCopyingIterator<\n        'a,\n        hash_map::Iter<'a, Field, crate::schema::OwnedValue>,\n        crate::schema::OwnedValue,\n    >;\n\n    fn iter_fields_and_values(&self) -> Self::FieldsValuesIter<'_> {\n        FieldCopyingIterator(self.iter())\n    }\n}\nimpl DocumentDeserialize for HashMap<Field, crate::schema::OwnedValue> {\n    fn deserialize<'de, D>(mut deserializer: D) -> Result<Self, DeserializeError>\n    where D: DocumentDeserializer<'de> {\n        let mut document = HashMap::with_capacity(deserializer.size_hint());\n\n        while let Some((field, value)) = deserializer.next_field()? {\n            document.insert(field, value);\n        }\n\n        Ok(document)\n    }\n}\n\npub struct FieldCopyingIterator<'a, I, V>(I)\nwhere\n    V: 'a,\n    I: Iterator<Item = (&'a Field, &'a V)>;\n\nimpl<'a, I, V> Iterator for FieldCopyingIterator<'a, I, V>\nwhere\n    V: 'a,\n    I: Iterator<Item = (&'a Field, &'a V)>,\n{\n    type Item = (Field, &'a V);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let (field, value) = self.0.next()?;\n        Some((*field, value))\n    }\n}\n"
  },
  {
    "path": "src/schema/document/mod.rs",
    "content": "//! Document definition for Tantivy to index and store.\n//!\n//! A document and its values are defined by a couple core traits:\n//! - [Document] which describes your top-level document and it's fields.\n//! - [Value] which provides tantivy with a way to access the document's values in a common way\n//!   without performing any additional allocations.\n//! - [DocumentDeserialize] which implements the necessary code to deserialize the document from the\n//!   doc store. If you are fine with fetching [TantivyDocument] from the doc store, you can skip\n//!   implementing this trait for your type.\n//!\n//! Tantivy provides a few out-of-box implementations of these core traits to provide\n//! some simple usage if you don't want to implement these traits on a custom type yourself.\n//!\n//! # Out-of-box document implementations\n//! - [TantivyDocument] the old document type used by Tantivy before the trait based approach was\n//!   implemented. This type is still valid and provides all of the original behaviour you might\n//!   expect.\n//! - `BTreeMap<Field, OwnedValue>` a mapping of field_ids to their relevant schema value using a\n//!   BTreeMap.\n//! - `HashMap<Field, OwnedValue>` a mapping of field_ids to their relevant schema value using a\n//!   HashMap.\n//!\n//! # Implementing your custom documents\n//! Often in larger projects or higher performance applications you want to avoid the extra overhead\n//! of converting your own types to the [TantivyDocument] type, this can often save you a\n//! significant amount of time when indexing by avoiding the additional allocations.\n//!\n//! ### Important Note\n//! The implementer of the `Document` trait must be `'static` and safe to send across\n//! thread boundaries.\n//!\n//! ## Reusing existing types\n//! The API design of the document traits allow you to reuse as much of as little of the\n//! existing trait implementations as you like, this can save quite a bit of boilerplate\n//! as shown by the following example.\n//!\n//! ## A basic custom document\n//! ```\n//! use std::collections::{btree_map, BTreeMap};\n//! use tantivy::schema::{Document, Field};\n//! use tantivy::schema::document::{DeserializeError, DocumentDeserialize, DocumentDeserializer};\n//!\n//! /// Our custom document to let us use a map of `serde_json::Values`.\n//! #[allow(dead_code)]\n//! pub struct MyCustomDocument {\n//!     // Tantivy provides trait implementations for common `serde_json` types.\n//!     fields: BTreeMap<Field, serde_json::Value>\n//! }\n//!\n//! impl Document for MyCustomDocument {\n//!     // The value type produced by the `iter_fields_and_values` iterator.\n//!     // tantivy already implements the Value trait for serde_json::Value.\n//!     type Value<'a> = &'a serde_json::Value;\n//!     // The iterator which is produced by `iter_fields_and_values`.\n//!     // Often this is a simple new-type wrapper unless you like super long generics.\n//!     type FieldsValuesIter<'a> = MyCustomIter<'a>;\n//!\n//!     /// Produces an iterator over the document fields and values.\n//!     /// This method will be called multiple times, it's important\n//!     /// to not do anything too heavy in this step, any heavy operations\n//!     /// should be done before and effectively cached.\n//!     fn iter_fields_and_values(&self) -> Self::FieldsValuesIter<'_> {\n//!         MyCustomIter(self.fields.iter())\n//!     }\n//! }\n//!\n//! // Our document must also provide a way to get the original doc\n//! // back when it's deserialized from the doc store.\n//! // The API for this is very similar to serde but a little bit\n//! // more specialised, giving you access to types like IP addresses, datetime, etc...\n//! impl DocumentDeserialize for MyCustomDocument {\n//!     fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n//!     where D: DocumentDeserializer<'de>\n//!     {\n//!         // We're not going to implement the necessary logic for this example\n//!         // see the `Deserialization` section of implementing a custom document\n//!         // for more information on how this works.\n//!         unimplemented!()\n//!     }\n//! }\n//!\n//! /// Our custom iterator just helps us to avoid some messy generics.\n//! #[allow(dead_code)]\n//! pub struct MyCustomIter<'a>(btree_map::Iter<'a, Field, serde_json::Value>);\n//! impl<'a> Iterator for MyCustomIter<'a> {\n//!     // Here we can see our field-value pairs being produced by the iterator.\n//!     // The value returned alongside the field is the same type as `Document::Value<'_>`.\n//!     type Item = (Field, &'a serde_json::Value);\n//!\n//!     fn next(&mut self) -> Option<Self::Item> {\n//!         let (field, value) = self.0.next()?;\n//!         Some((*field, value))\n//!     }\n//! }\n//! ```\n//!\n//! You may have noticed in this example that we haven't needed to implement any custom value types,\n//! instead we've just used a [serde_json::Value] type which tantivy provides an existing\n//! implementation for.\n//!\n//! ## Implementing custom values\n//! In order to allow documents to return custom types, they must implement\n//! the [Value] trait which provides a way for Tantivy to get a `ReferenceValue` that it can then\n//! index and store.\n//! Internally, Tantivy only works with `ReferenceValue` which is an enum that tries to borrow\n//! as much data as it can\n//!\n//! Values can just as easily be customised as documents by implementing the `Value` trait.\n//!\n//! The implementer of this type should not own the data it's returning, instead it should just\n//! hold references of the data held by the parent [Document] which can then be passed\n//! on to the [ReferenceValue].\n//!\n//! This is why [Value] is implemented for `&'a serde_json::Value` and\n//! [&'a tantivy::schema::document::OwnedValue](OwnedValue) but not for their owned counterparts, as\n//! we cannot satisfy the lifetime bounds necessary when indexing the documents.\n//!\n//! ### A note about returning values\n//! The custom value type does not have to be the type stored by the document, instead the\n//! implementer of a `Value` can just be used as a way to convert between the owned type\n//! kept in the parent document, and the value passed into Tantivy.\n//!\n//! ```\n//! use tantivy::schema::document::ReferenceValue;\n//! use tantivy::schema::document::ReferenceValueLeaf;\n//! use tantivy::schema::{Value};\n//!\n//! #[derive(Debug)]\n//! /// Our custom value type which has 3 types, a string, float and bool.\n//! #[allow(dead_code)]\n//! pub enum MyCustomValue<'a> {\n//!     // Our string data is owned by the parent document, instead we just\n//!     // hold onto a reference of this data.\n//!     String(&'a str),\n//!     Float(f64),\n//!     Bool(bool),\n//! }\n//!\n//! impl<'a> Value<'a> for MyCustomValue<'a> {\n//!     // We don't need to worry about these types here as we're not\n//!     // working with nested types, but if we wanted to we would\n//!     // define our two iterator types, a sequence of ReferenceValues\n//!     // for the array iterator and a sequence of key-value pairs for objects.\n//!     type ArrayIter = std::iter::Empty<Self>;\n//!     type ObjectIter = std::iter::Empty<(&'a str, Self)>;\n//!\n//!     // The ReferenceValue which Tantivy can use.\n//!     fn as_value(&self) -> ReferenceValue<'a, Self> {\n//!         // We can support any type that Tantivy itself supports.\n//!         match self {\n//!             MyCustomValue::String(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Str(*val)),\n//!             MyCustomValue::Float(val) => ReferenceValue::Leaf(ReferenceValueLeaf::F64(*val)),\n//!             MyCustomValue::Bool(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Bool(*val)),\n//!         }\n//!     }\n//!\n//! }\n//! ```\n//!\n//! TODO: Complete this section...\n\nmod de;\nmod default_document;\nmod existing_type_impls;\nmod owned_value;\nmod se;\nmod value;\n\nuse std::collections::BTreeMap;\nuse std::mem;\n\npub(crate) use self::de::BinaryDocumentDeserializer;\npub use self::de::{\n    ArrayAccess, DeserializeError, DocumentDeserialize, DocumentDeserializer, ObjectAccess,\n    ValueDeserialize, ValueDeserializer, ValueType, ValueVisitor,\n};\npub use self::default_document::{\n    CompactDocArrayIter, CompactDocObjectIter, CompactDocValue, DocParsingError, TantivyDocument,\n};\npub use self::owned_value::OwnedValue;\npub(crate) use self::se::BinaryDocumentSerializer;\npub use self::value::{ReferenceValue, ReferenceValueLeaf, Value};\nuse super::*;\n\n/// The core trait representing a document within the index.\npub trait Document: Send + Sync + 'static {\n    /// The value of the field.\n    type Value<'a>: Value<'a> + Clone\n    where Self: 'a;\n\n    /// The iterator over all of the fields and values within the doc.\n    type FieldsValuesIter<'a>: Iterator<Item = (Field, Self::Value<'a>)>\n    where Self: 'a;\n\n    /// Get an iterator iterating over all fields and values in a document.\n    fn iter_fields_and_values(&self) -> Self::FieldsValuesIter<'_>;\n\n    /// Sort and groups the field_values by field.\n    ///\n    /// The result of this method is not cached and is\n    /// computed on the fly when this method is called.\n    fn get_sorted_field_values(&self) -> Vec<(Field, Vec<Self::Value<'_>>)> {\n        let mut field_values: Vec<(Field, Self::Value<'_>)> =\n            self.iter_fields_and_values().collect();\n        field_values.sort_by_key(|(field, _)| *field);\n\n        let mut field_values_it = field_values.into_iter();\n\n        let first_field_value = if let Some(first_field_value) = field_values_it.next() {\n            first_field_value\n        } else {\n            return Vec::new();\n        };\n\n        let mut grouped_field_values = vec![];\n        let mut current_field = first_field_value.0;\n        let mut current_group = vec![first_field_value.1];\n\n        for (field, value) in field_values_it {\n            if field == current_field {\n                current_group.push(value);\n            } else {\n                grouped_field_values\n                    .push((current_field, mem::replace(&mut current_group, vec![value])));\n                current_field = field;\n            }\n        }\n\n        grouped_field_values.push((current_field, current_group));\n        grouped_field_values\n    }\n\n    /// Create a named document from the doc.\n    fn to_named_doc(&self, schema: &Schema) -> NamedFieldDocument {\n        let mut field_map = BTreeMap::new();\n        for (field, field_values) in self.get_sorted_field_values() {\n            let field_name = schema.get_field_name(field);\n            let values: Vec<OwnedValue> = field_values\n                .into_iter()\n                .map(|val| OwnedValue::from(val.as_value()))\n                .collect();\n            field_map.insert(field_name.to_string(), values);\n        }\n        NamedFieldDocument(field_map)\n    }\n\n    /// Encode the doc in JSON.\n    ///\n    /// Encoding a document cannot fail.\n    fn to_json(&self, schema: &Schema) -> String {\n        serde_json::to_string(&self.to_named_doc(schema))\n            .expect(\"doc encoding failed. This is a bug\")\n    }\n}\n\npub(crate) mod type_codes {\n    pub const TEXT_CODE: u8 = 0;\n    pub const U64_CODE: u8 = 1;\n    pub const I64_CODE: u8 = 2;\n    pub const HIERARCHICAL_FACET_CODE: u8 = 3;\n    pub const BYTES_CODE: u8 = 4;\n    pub const DATE_CODE: u8 = 5;\n    pub const F64_CODE: u8 = 6;\n    pub const EXT_CODE: u8 = 7;\n\n    #[deprecated]\n    pub const JSON_OBJ_CODE: u8 = 8; // Replaced by the `OBJECT_CODE`.\n    pub const BOOL_CODE: u8 = 9;\n    pub const IP_CODE: u8 = 10;\n    pub const NULL_CODE: u8 = 11;\n    pub const ARRAY_CODE: u8 = 12;\n    pub const OBJECT_CODE: u8 = 13;\n\n    // Extended type codes\n    pub const TOK_STR_EXT_CODE: u8 = 0;\n}\n"
  },
  {
    "path": "src/schema/document/owned_value.rs",
    "content": "use std::collections::BTreeMap;\nuse std::fmt;\nuse std::net::Ipv6Addr;\n\nuse base64::engine::general_purpose::STANDARD as BASE64;\nuse base64::Engine;\nuse serde::de::{MapAccess, SeqAccess};\nuse time::format_description::well_known::Rfc3339;\nuse time::OffsetDateTime;\n\nuse super::existing_type_impls::can_be_rfc3339_date_time;\nuse super::ReferenceValueLeaf;\nuse crate::schema::document::{\n    ArrayAccess, DeserializeError, ObjectAccess, ReferenceValue, Value, ValueDeserialize,\n    ValueDeserializer, ValueVisitor,\n};\nuse crate::schema::Facet;\nuse crate::tokenizer::PreTokenizedString;\nuse crate::DateTime;\n\n/// This is a owned variant of `Value`, that can be passed around without lifetimes.\n/// Represents the value of a any field.\n/// It is an enum over all over all of the possible field type.\n#[derive(Debug, Clone, PartialEq)]\npub enum OwnedValue {\n    /// A null value.\n    Null,\n    /// The str type is used for any text information.\n    Str(String),\n    /// Pre-tokenized str type,\n    PreTokStr(PreTokenizedString),\n    /// Unsigned 64-bits Integer `u64`\n    U64(u64),\n    /// Signed 64-bits Integer `i64`\n    I64(i64),\n    /// 64-bits Float `f64`\n    F64(f64),\n    /// Bool value\n    Bool(bool),\n    /// Date/time with nanoseconds precision\n    Date(DateTime),\n    /// Facet\n    Facet(Facet),\n    /// Arbitrarily sized byte array\n    Bytes(Vec<u8>),\n    /// A set of values.\n    Array(Vec<Self>),\n    /// Dynamic object value.\n    Object(Vec<(String, Self)>),\n    /// IpV6 Address. Internally there is no IpV4, it needs to be converted to `Ipv6Addr`.\n    IpAddr(Ipv6Addr),\n}\n\nimpl AsRef<OwnedValue> for OwnedValue {\n    #[inline]\n    fn as_ref(&self) -> &OwnedValue {\n        self\n    }\n}\n\nimpl OwnedValue {\n    /// Returns a u8 discriminant value for the `OwnedValue` variant.\n    ///\n    /// This can be used to sort `OwnedValue` instances by their type.\n    pub fn discriminant_value(&self) -> u8 {\n        match self {\n            OwnedValue::Null => 0,\n            OwnedValue::Str(_) => 1,\n            OwnedValue::PreTokStr(_) => 2,\n            // It is key to make sure U64, I64, F64 are grouped together in there, otherwise we\n            // might be breaking transivity.\n            OwnedValue::U64(_) => 3,\n            OwnedValue::I64(_) => 4,\n            OwnedValue::F64(_) => 5,\n            OwnedValue::Bool(_) => 6,\n            OwnedValue::Date(_) => 7,\n            OwnedValue::Facet(_) => 8,\n            OwnedValue::Bytes(_) => 9,\n            OwnedValue::Array(_) => 10,\n            OwnedValue::Object(_) => 11,\n            OwnedValue::IpAddr(_) => 12,\n        }\n    }\n}\n\nimpl<'a> Value<'a> for &'a OwnedValue {\n    type ArrayIter = std::slice::Iter<'a, OwnedValue>;\n    type ObjectIter = ObjectMapIter<'a>;\n\n    fn as_value(&self) -> ReferenceValue<'a, Self> {\n        match self {\n            OwnedValue::Null => ReferenceValueLeaf::Null.into(),\n            OwnedValue::Str(val) => ReferenceValueLeaf::Str(val).into(),\n            OwnedValue::PreTokStr(val) => ReferenceValueLeaf::PreTokStr(val.clone().into()).into(),\n            OwnedValue::U64(val) => ReferenceValueLeaf::U64(*val).into(),\n            OwnedValue::I64(val) => ReferenceValueLeaf::I64(*val).into(),\n            OwnedValue::F64(val) => ReferenceValueLeaf::F64(*val).into(),\n            OwnedValue::Bool(val) => ReferenceValueLeaf::Bool(*val).into(),\n            OwnedValue::Date(val) => ReferenceValueLeaf::Date(*val).into(),\n            OwnedValue::Facet(val) => ReferenceValueLeaf::Facet(val.encoded_str()).into(),\n            OwnedValue::Bytes(val) => ReferenceValueLeaf::Bytes(val).into(),\n            OwnedValue::IpAddr(val) => ReferenceValueLeaf::IpAddr(*val).into(),\n            OwnedValue::Array(array) => ReferenceValue::Array(array.iter()),\n            OwnedValue::Object(object) => ReferenceValue::Object(ObjectMapIter(object.iter())),\n        }\n    }\n}\n\nimpl ValueDeserialize for OwnedValue {\n    fn deserialize<'de, D>(deserializer: D) -> Result<Self, DeserializeError>\n    where D: ValueDeserializer<'de> {\n        struct Visitor;\n\n        impl ValueVisitor for Visitor {\n            type Value = OwnedValue;\n\n            fn visit_null(&self) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Null)\n            }\n\n            fn visit_string(&self, val: String) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Str(val))\n            }\n\n            fn visit_u64(&self, val: u64) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::U64(val))\n            }\n\n            fn visit_i64(&self, val: i64) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::I64(val))\n            }\n\n            fn visit_f64(&self, val: f64) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::F64(val))\n            }\n\n            fn visit_bool(&self, val: bool) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Bool(val))\n            }\n\n            fn visit_datetime(&self, val: DateTime) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Date(val))\n            }\n\n            fn visit_ip_address(&self, val: Ipv6Addr) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::IpAddr(val))\n            }\n\n            fn visit_facet(&self, val: Facet) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Facet(val))\n            }\n\n            fn visit_bytes(&self, val: Vec<u8>) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::Bytes(val))\n            }\n\n            fn visit_pre_tokenized_string(\n                &self,\n                val: PreTokenizedString,\n            ) -> Result<Self::Value, DeserializeError> {\n                Ok(OwnedValue::PreTokStr(val))\n            }\n\n            fn visit_array<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n            where A: ArrayAccess<'de> {\n                let mut elements = Vec::with_capacity(access.size_hint());\n\n                while let Some(value) = access.next_element()? {\n                    elements.push(value);\n                }\n\n                Ok(OwnedValue::Array(elements))\n            }\n\n            fn visit_object<'de, A>(&self, mut access: A) -> Result<Self::Value, DeserializeError>\n            where A: ObjectAccess<'de> {\n                let mut elements = Vec::with_capacity(access.size_hint());\n\n                while let Some((key, value)) = access.next_entry()? {\n                    elements.push((key, value));\n                }\n\n                Ok(OwnedValue::Object(elements))\n            }\n        }\n\n        deserializer.deserialize_any(Visitor)\n    }\n}\n\nimpl Eq for OwnedValue {}\n\nimpl serde::Serialize for OwnedValue {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: serde::Serializer {\n        use serde::ser::SerializeMap;\n        match *self {\n            OwnedValue::Null => serializer.serialize_unit(),\n            OwnedValue::Str(ref v) => serializer.serialize_str(v),\n            OwnedValue::PreTokStr(ref v) => v.serialize(serializer),\n            OwnedValue::U64(u) => serializer.serialize_u64(u),\n            OwnedValue::I64(u) => serializer.serialize_i64(u),\n            OwnedValue::F64(u) => serializer.serialize_f64(u),\n            OwnedValue::Bool(b) => serializer.serialize_bool(b),\n            OwnedValue::Date(ref date) => {\n                time::serde::rfc3339::serialize(&date.into_utc(), serializer)\n            }\n            OwnedValue::Facet(ref facet) => facet.serialize(serializer),\n            OwnedValue::Bytes(ref bytes) => serializer.serialize_str(&BASE64.encode(bytes)),\n            OwnedValue::Object(ref obj) => {\n                let mut map = serializer.serialize_map(Some(obj.len()))?;\n                for (k, v) in obj {\n                    map.serialize_entry(k, v)?;\n                }\n                map.end()\n            }\n            OwnedValue::IpAddr(ref ip_v6) => {\n                // Ensure IpV4 addresses get serialized as IpV4, but excluding IpV6 loopback.\n                if let Some(ip_v4) = ip_v6.to_ipv4_mapped() {\n                    ip_v4.serialize(serializer)\n                } else {\n                    ip_v6.serialize(serializer)\n                }\n            }\n            OwnedValue::Array(ref array) => array.serialize(serializer),\n        }\n    }\n}\n\nimpl<'de> serde::Deserialize<'de> for OwnedValue {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: serde::Deserializer<'de> {\n        struct ValueVisitor;\n\n        impl<'de> serde::de::Visitor<'de> for ValueVisitor {\n            type Value = OwnedValue;\n\n            fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {\n                formatter.write_str(\"a string or u32\")\n            }\n\n            fn visit_bool<E>(self, v: bool) -> Result<Self::Value, E> {\n                Ok(OwnedValue::Bool(v))\n            }\n\n            fn visit_i64<E>(self, v: i64) -> Result<Self::Value, E> {\n                Ok(OwnedValue::I64(v))\n            }\n\n            fn visit_u64<E>(self, v: u64) -> Result<Self::Value, E> {\n                Ok(OwnedValue::U64(v))\n            }\n\n            fn visit_f64<E>(self, v: f64) -> Result<Self::Value, E> {\n                Ok(OwnedValue::F64(v))\n            }\n\n            fn visit_str<E>(self, v: &str) -> Result<Self::Value, E> {\n                Ok(OwnedValue::Str(v.to_owned()))\n            }\n\n            fn visit_string<E>(self, v: String) -> Result<Self::Value, E> {\n                Ok(OwnedValue::Str(v))\n            }\n\n            fn visit_unit<E>(self) -> Result<Self::Value, E>\n            where E: serde::de::Error {\n                Ok(OwnedValue::Null)\n            }\n\n            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n            where A: SeqAccess<'de> {\n                let mut elements = Vec::with_capacity(seq.size_hint().unwrap_or_default());\n\n                while let Some(value) = seq.next_element()? {\n                    elements.push(value);\n                }\n\n                Ok(OwnedValue::Array(elements))\n            }\n\n            fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>\n            where A: MapAccess<'de> {\n                let mut object = map.size_hint().map(Vec::with_capacity).unwrap_or_default();\n                while let Some((key, value)) = map.next_entry()? {\n                    object.push((key, value));\n                }\n                Ok(OwnedValue::Object(object))\n            }\n        }\n\n        deserializer.deserialize_any(ValueVisitor)\n    }\n}\n\nimpl<'a, V: Value<'a>> From<ReferenceValue<'a, V>> for OwnedValue {\n    fn from(val: ReferenceValue<'a, V>) -> OwnedValue {\n        match val {\n            ReferenceValue::Leaf(leaf) => match leaf {\n                ReferenceValueLeaf::Null => OwnedValue::Null,\n                ReferenceValueLeaf::Str(val) => OwnedValue::Str(val.to_string()),\n                ReferenceValueLeaf::U64(val) => OwnedValue::U64(val),\n                ReferenceValueLeaf::I64(val) => OwnedValue::I64(val),\n                ReferenceValueLeaf::F64(val) => OwnedValue::F64(val),\n                ReferenceValueLeaf::Date(val) => OwnedValue::Date(val),\n                ReferenceValueLeaf::Facet(val) => {\n                    OwnedValue::Facet(Facet::from_encoded_string(val.to_string()))\n                }\n                ReferenceValueLeaf::Bytes(val) => OwnedValue::Bytes(val.to_vec()),\n                ReferenceValueLeaf::IpAddr(val) => OwnedValue::IpAddr(val),\n                ReferenceValueLeaf::Bool(val) => OwnedValue::Bool(val),\n                ReferenceValueLeaf::PreTokStr(val) => OwnedValue::PreTokStr(*val.clone()),\n            },\n            ReferenceValue::Array(val) => {\n                OwnedValue::Array(val.map(|v| v.as_value().into()).collect())\n            }\n            ReferenceValue::Object(val) => OwnedValue::Object(\n                val.map(|(k, v)| (k.to_string(), v.as_value().into()))\n                    .collect(),\n            ),\n        }\n    }\n}\n\nimpl From<String> for OwnedValue {\n    fn from(s: String) -> OwnedValue {\n        OwnedValue::Str(s)\n    }\n}\n\nimpl From<Ipv6Addr> for OwnedValue {\n    fn from(v: Ipv6Addr) -> OwnedValue {\n        OwnedValue::IpAddr(v)\n    }\n}\n\nimpl From<u64> for OwnedValue {\n    fn from(v: u64) -> OwnedValue {\n        OwnedValue::U64(v)\n    }\n}\n\nimpl From<i64> for OwnedValue {\n    fn from(v: i64) -> OwnedValue {\n        OwnedValue::I64(v)\n    }\n}\n\nimpl From<f64> for OwnedValue {\n    fn from(v: f64) -> OwnedValue {\n        OwnedValue::F64(v)\n    }\n}\n\nimpl From<bool> for OwnedValue {\n    fn from(b: bool) -> Self {\n        OwnedValue::Bool(b)\n    }\n}\n\nimpl From<DateTime> for OwnedValue {\n    fn from(dt: DateTime) -> OwnedValue {\n        OwnedValue::Date(dt)\n    }\n}\n\nimpl<'a> From<&'a str> for OwnedValue {\n    fn from(s: &'a str) -> OwnedValue {\n        OwnedValue::Str(s.to_string())\n    }\n}\n\nimpl<'a> From<&'a [u8]> for OwnedValue {\n    fn from(bytes: &'a [u8]) -> OwnedValue {\n        OwnedValue::Bytes(bytes.to_vec())\n    }\n}\n\nimpl From<Facet> for OwnedValue {\n    fn from(facet: Facet) -> OwnedValue {\n        OwnedValue::Facet(facet)\n    }\n}\n\nimpl From<Vec<u8>> for OwnedValue {\n    fn from(bytes: Vec<u8>) -> OwnedValue {\n        OwnedValue::Bytes(bytes)\n    }\n}\n\nimpl From<PreTokenizedString> for OwnedValue {\n    fn from(pretokenized_string: PreTokenizedString) -> OwnedValue {\n        OwnedValue::PreTokStr(pretokenized_string)\n    }\n}\n\nimpl From<BTreeMap<String, OwnedValue>> for OwnedValue {\n    fn from(object: BTreeMap<String, OwnedValue>) -> OwnedValue {\n        let key_values = object.into_iter().collect();\n        OwnedValue::Object(key_values)\n    }\n}\n\nimpl From<serde_json::Value> for OwnedValue {\n    fn from(value: serde_json::Value) -> Self {\n        match value {\n            serde_json::Value::Null => Self::Null,\n            serde_json::Value::Bool(val) => Self::Bool(val),\n            serde_json::Value::Number(number) => {\n                if let Some(val) = number.as_i64() {\n                    Self::I64(val)\n                } else if let Some(val) = number.as_u64() {\n                    Self::U64(val)\n                } else if let Some(val) = number.as_f64() {\n                    Self::F64(val)\n                } else {\n                    panic!(\"Unsupported serde_json number {number}\");\n                }\n            }\n            serde_json::Value::String(text) => {\n                if can_be_rfc3339_date_time(&text) {\n                    match OffsetDateTime::parse(&text, &Rfc3339) {\n                        Ok(dt) => {\n                            let dt_utc = dt.to_offset(time::UtcOffset::UTC);\n                            Self::Date(DateTime::from_utc(dt_utc))\n                        }\n                        Err(_) => Self::Str(text),\n                    }\n                } else {\n                    Self::Str(text)\n                }\n            }\n            serde_json::Value::Array(elements) => {\n                let converted_elements = elements.into_iter().map(Self::from).collect();\n                Self::Array(converted_elements)\n            }\n            serde_json::Value::Object(object) => Self::from(object),\n        }\n    }\n}\n\nimpl From<serde_json::Map<String, serde_json::Value>> for OwnedValue {\n    fn from(map: serde_json::Map<String, serde_json::Value>) -> Self {\n        let object: Vec<(String, OwnedValue)> = map\n            .into_iter()\n            .map(|(key, value)| (key, OwnedValue::from(value)))\n            .collect();\n        OwnedValue::Object(object)\n    }\n}\n\n/// A wrapper type for iterating over a serde_json object producing reference values.\npub struct ObjectMapIter<'a>(std::slice::Iter<'a, (String, OwnedValue)>);\n\nimpl<'a> Iterator for ObjectMapIter<'a> {\n    type Item = (&'a str, &'a OwnedValue);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        let (key, value) = self.0.next()?;\n        Some((key.as_str(), value))\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::schema::{BytesOptions, Schema};\n    use crate::{Document, TantivyDocument};\n\n    #[test]\n    fn test_parse_bytes_doc() {\n        let mut schema_builder = Schema::builder();\n        let bytes_options = BytesOptions::default();\n        let bytes_field = schema_builder.add_bytes_field(\"my_bytes\", bytes_options);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        doc.add_bytes(bytes_field, \"this is a test\".as_bytes());\n        let json_string = doc.to_json(&schema);\n        assert_eq!(json_string, r#\"{\"my_bytes\":[\"dGhpcyBpcyBhIHRlc3Q=\"]}\"#);\n    }\n\n    #[test]\n    fn test_parse_empty_bytes_doc() {\n        let mut schema_builder = Schema::builder();\n        let bytes_options = BytesOptions::default();\n        let bytes_field = schema_builder.add_bytes_field(\"my_bytes\", bytes_options);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        doc.add_bytes(bytes_field, \"\".as_bytes());\n        let json_string = doc.to_json(&schema);\n\n        assert_eq!(json_string, r#\"{\"my_bytes\":[\"\"]}\"#);\n    }\n\n    #[test]\n    fn test_parse_many_bytes_doc() {\n        let mut schema_builder = Schema::builder();\n        let bytes_options = BytesOptions::default();\n        let bytes_field = schema_builder.add_bytes_field(\"my_bytes\", bytes_options);\n        let schema = schema_builder.build();\n        let mut doc = TantivyDocument::default();\n        doc.add_bytes(\n            bytes_field,\n            \"A bigger test I guess\\nspanning on multiple lines\\nhoping this will work\".as_bytes(),\n        );\n        let json_string = doc.to_json(&schema);\n        assert_eq!(\n            json_string,\n            r#\"{\"my_bytes\":[\"QSBiaWdnZXIgdGVzdCBJIGd1ZXNzCnNwYW5uaW5nIG9uIG11bHRpcGxlIGxpbmVzCmhvcGluZyB0aGlzIHdpbGwgd29yaw==\"]}\"#\n        );\n    }\n\n    #[test]\n    fn test_serialize_date() {\n        let value = OwnedValue::from(DateTime::from_utc(\n            OffsetDateTime::parse(\"1996-12-20T00:39:57+00:00\", &Rfc3339).unwrap(),\n        ));\n        let serialized_value_json = serde_json::to_string_pretty(&value).unwrap();\n        assert_eq!(serialized_value_json, r#\"\"1996-12-20T00:39:57Z\"\"#);\n        let value = OwnedValue::from(DateTime::from_utc(\n            OffsetDateTime::parse(\"1996-12-20T00:39:57-01:00\", &Rfc3339).unwrap(),\n        ));\n        let serialized_value_json = serde_json::to_string_pretty(&value).unwrap();\n        // The time zone information gets lost by conversion into `Value::Date` and\n        // implicitly becomes UTC.\n        assert_eq!(serialized_value_json, r#\"\"1996-12-20T01:39:57Z\"\"#);\n    }\n}\n"
  },
  {
    "path": "src/schema/document/se.rs",
    "content": "use std::borrow::Cow;\nuse std::io;\nuse std::io::Write;\n\nuse columnar::MonotonicallyMappableToU128;\nuse common::{f64_to_u64, BinarySerializable, VInt};\n\nuse super::{OwnedValue, ReferenceValueLeaf};\nuse crate::schema::document::{type_codes, Document, ReferenceValue, Value};\nuse crate::schema::Schema;\n\n/// A serializer writing documents which implement [`Document`] to a provided writer.\npub struct BinaryDocumentSerializer<'se, W> {\n    writer: &'se mut W,\n    schema: &'se Schema,\n}\n\nimpl<'se, W> BinaryDocumentSerializer<'se, W>\nwhere W: Write\n{\n    /// Creates a new serializer with a provided writer.\n    pub(crate) fn new(writer: &'se mut W, schema: &'se Schema) -> Self {\n        Self { writer, schema }\n    }\n\n    /// Attempts to serialize a given document and write the output\n    /// to the writer.\n    #[inline]\n    pub(crate) fn serialize_doc<D>(&mut self, doc: &D) -> io::Result<()>\n    where D: Document {\n        let stored_field_values = || {\n            doc.iter_fields_and_values()\n                .filter(|(field, _)| self.schema.get_field_entry(*field).is_stored())\n        };\n        let num_field_values = stored_field_values().count();\n        let mut actual_length = 0;\n\n        VInt(num_field_values as u64).serialize(self.writer)?;\n        for (field, value_access) in stored_field_values() {\n            field.serialize(self.writer)?;\n\n            let mut serializer = BinaryValueSerializer::new(self.writer);\n            match value_access.as_value() {\n                ReferenceValue::Leaf(ReferenceValueLeaf::PreTokStr(pre_tokenized_text)) => {\n                    serializer.serialize_value(ReferenceValue::Leaf::<&'_ OwnedValue>(\n                        ReferenceValueLeaf::Str(&pre_tokenized_text.text),\n                    ))?;\n                }\n                _ => {\n                    serializer.serialize_value(value_access.as_value())?;\n                }\n            }\n\n            actual_length += 1;\n        }\n\n        if num_field_values != actual_length {\n            return Err(io::Error::other(format!(\n                \"Unexpected number of entries written to serializer, expected {num_field_values} \\\n                 entries, got {actual_length} entries\",\n            )));\n        }\n\n        Ok(())\n    }\n}\n\n/// A serializer for a single value.\npub struct BinaryValueSerializer<'se, W> {\n    writer: &'se mut W,\n}\n\nimpl<'se, W> BinaryValueSerializer<'se, W>\nwhere W: Write\n{\n    /// Creates a new serializer with a provided writer.\n    pub(crate) fn new(writer: &'se mut W) -> Self {\n        Self { writer }\n    }\n\n    fn serialize_with_type_code<T: BinarySerializable>(\n        &mut self,\n        code: u8,\n        val: &T,\n    ) -> io::Result<()> {\n        self.write_type_code(code)?;\n        BinarySerializable::serialize(val, self.writer)\n    }\n\n    /// Attempts to serialize a given value and write the output\n    /// to the writer.\n    pub(crate) fn serialize_value<'a, V>(\n        &mut self,\n        value: ReferenceValue<'a, V>,\n    ) -> io::Result<()>\n    where\n        V: Value<'a>,\n    {\n        match value {\n            ReferenceValue::Leaf(leaf) => match leaf {\n                ReferenceValueLeaf::Null => self.write_type_code(type_codes::NULL_CODE),\n                ReferenceValueLeaf::Str(val) => {\n                    self.serialize_with_type_code(type_codes::TEXT_CODE, &Cow::Borrowed(val))\n                }\n                ReferenceValueLeaf::U64(val) => {\n                    self.serialize_with_type_code(type_codes::U64_CODE, &val)\n                }\n                ReferenceValueLeaf::I64(val) => {\n                    self.serialize_with_type_code(type_codes::I64_CODE, &val)\n                }\n                ReferenceValueLeaf::F64(val) => {\n                    self.serialize_with_type_code(type_codes::F64_CODE, &f64_to_u64(val))\n                }\n                ReferenceValueLeaf::Date(val) => {\n                    self.write_type_code(type_codes::DATE_CODE)?;\n                    let timestamp_nanos: i64 = val.into_timestamp_nanos();\n                    BinarySerializable::serialize(&timestamp_nanos, self.writer)\n                }\n                ReferenceValueLeaf::Facet(val) => self.serialize_with_type_code(\n                    type_codes::HIERARCHICAL_FACET_CODE,\n                    &Cow::Borrowed(val),\n                ),\n                ReferenceValueLeaf::Bytes(val) => {\n                    self.serialize_with_type_code(type_codes::BYTES_CODE, &Cow::Borrowed(val))\n                }\n                ReferenceValueLeaf::IpAddr(val) => {\n                    self.serialize_with_type_code(type_codes::IP_CODE, &val.to_u128())\n                }\n                ReferenceValueLeaf::Bool(val) => {\n                    self.serialize_with_type_code(type_codes::BOOL_CODE, &val)\n                }\n                ReferenceValueLeaf::PreTokStr(val) => {\n                    self.write_type_code(type_codes::EXT_CODE)?;\n                    self.serialize_with_type_code(type_codes::TOK_STR_EXT_CODE, &*val)\n                }\n            },\n            ReferenceValue::Array(elements) => {\n                self.write_type_code(type_codes::ARRAY_CODE)?;\n\n                // Somewhat unfortunate that we do this here however, writing the\n                // length at the end of the complicates things quite considerably.\n                let elements: Vec<V> = elements.collect();\n\n                let mut serializer = BinaryArraySerializer::begin(elements.len(), self.writer)?;\n\n                for value in elements {\n                    serializer.serialize_value(value.as_value())?;\n                }\n\n                serializer.end()\n            }\n            ReferenceValue::Object(object) => {\n                self.write_type_code(type_codes::OBJECT_CODE)?;\n\n                // Somewhat unfortunate that we do this here however, writing the\n                // length at the end of the complicates things quite considerably.\n                let entries: Vec<(&str, V)> = object.collect();\n\n                let mut serializer = BinaryObjectSerializer::begin(entries.len(), self.writer)?;\n\n                for (key, value) in entries {\n                    serializer.serialize_entry(key, value.as_value())?;\n                }\n\n                serializer.end()\n            }\n        }\n    }\n\n    fn write_type_code(&mut self, code: u8) -> io::Result<()> {\n        code.serialize(self.writer)\n    }\n}\n\n/// A serializer for writing a sequence of values to a writer.\npub struct BinaryArraySerializer<'se, W> {\n    writer: &'se mut W,\n    expected_length: usize,\n    actual_length: usize,\n}\n\nimpl<'se, W> BinaryArraySerializer<'se, W>\nwhere W: Write\n{\n    /// Creates a new array serializer and writes the length of the array to the writer.\n    pub(crate) fn begin(length: usize, writer: &'se mut W) -> io::Result<Self> {\n        VInt(length as u64).serialize(writer)?;\n\n        Ok(Self {\n            writer,\n            expected_length: length,\n            actual_length: 0,\n        })\n    }\n\n    /// Attempts to serialize a given value and write the output\n    /// to the writer.\n    pub(crate) fn serialize_value<'a, V>(\n        &mut self,\n        value: ReferenceValue<'a, V>,\n    ) -> io::Result<()>\n    where\n        V: Value<'a>,\n    {\n        let mut serializer = BinaryValueSerializer::new(self.writer);\n        serializer.serialize_value(value)?;\n\n        self.actual_length += 1;\n        Ok(())\n    }\n\n    /// Finishes writing the array to the writer and validates it.\n    pub(crate) fn end(self) -> io::Result<()> {\n        if self.expected_length != self.actual_length {\n            return Err(io::Error::other(format!(\n                \"Unexpected number of entries written to serializer, expected {} entries, got {} \\\n                 entries\",\n                self.expected_length, self.actual_length,\n            )));\n        }\n        Ok(())\n    }\n}\n\n/// A serializer for writing a set of key-value pairs to a writer.\npub struct BinaryObjectSerializer<'se, W> {\n    inner: BinaryArraySerializer<'se, W>,\n    expected_length: usize,\n    actual_length: usize,\n}\n\nimpl<'se, W> BinaryObjectSerializer<'se, W>\nwhere W: Write\n{\n    /// Creates a new object serializer and writes the length of the object to the writer.\n    pub(crate) fn begin(length: usize, writer: &'se mut W) -> io::Result<Self> {\n        // We mul by 2 here to count the keys and values separately:\n        // [(\"a\", 1), (\"b\", 2)] is actually stored as [\"a\", 1, \"b\", 2]\n        let inner = BinaryArraySerializer::begin(length * 2, writer)?;\n\n        Ok(Self {\n            inner,\n            expected_length: length,\n            actual_length: 0,\n        })\n    }\n\n    /// Attempts to serialize a given value and write the output\n    /// to the writer.\n    pub(crate) fn serialize_entry<'a, V>(\n        &mut self,\n        key: &'a str,\n        value: ReferenceValue<'a, V>,\n    ) -> io::Result<()>\n    where\n        V: Value<'a>,\n    {\n        // Keys and values are stored inline with one another.\n        // Technically this isn't the *most* optimal way of storing the objects\n        // as we could avoid writing the extra byte per key. But the gain is\n        // largely not worth it for the extra complexity it brings.\n        self.inner\n            .serialize_value(ReferenceValue::<'a, V>::Leaf(ReferenceValueLeaf::Str(key)))?;\n        self.inner.serialize_value(value)?;\n\n        self.actual_length += 1;\n        Ok(())\n    }\n\n    /// Finishes writing the array to the writer and validates it.\n    pub(crate) fn end(self) -> io::Result<()> {\n        if self.expected_length != self.actual_length {\n            return Err(io::Error::other(format!(\n                \"Unexpected number of entries written to serializer, expected {} entries, got {} \\\n                 entries\",\n                self.expected_length, self.actual_length,\n            )));\n        }\n\n        // This should never fail if the above statement is valid.\n        self.inner.end()?;\n\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeMap;\n\n    use serde_json::Number;\n    use tokenizer_api::Token;\n\n    use super::*;\n    use crate::schema::document::existing_type_impls::JsonObjectIter;\n    use crate::schema::{Facet, Field, FAST, STORED, TEXT};\n    use crate::tokenizer::PreTokenizedString;\n\n    fn serialize_value<'a>(value: ReferenceValue<'a, &'a serde_json::Value>) -> Vec<u8> {\n        let mut writer = Vec::new();\n\n        let mut serializer = BinaryValueSerializer::new(&mut writer);\n        serializer.serialize_value(value).expect(\"Serialize value\");\n\n        writer\n    }\n\n    /// A macro for defining the expected binary representation\n    /// of the serialized values in a somewhat human readable way.\n    macro_rules! binary_repr {\n        ($( $type_code:expr $(, $ext_code:expr)? => $value:expr $(,)?)*) => {{\n            let mut writer = Vec::new();\n\n            $(\n                $type_code.serialize(&mut writer).unwrap();\n\n                $(\n                    $ext_code.serialize(&mut writer).unwrap();\n                )?\n\n                BinarySerializable::serialize(\n                    &$value,\n                    &mut writer,\n                ).unwrap();\n            )*\n\n            writer\n        }};\n        (collection $code:expr, length $len:expr, $( $type_code:expr $(, $ext_code:expr)? => $value:expr $(,)?)*) => {{\n            let mut writer = Vec::new();\n\n            $code.serialize(&mut writer).unwrap();\n            VInt($len as u64).serialize(&mut writer).unwrap();\n\n            $(\n                $type_code.serialize(&mut writer).unwrap();\n\n                $(\n                    $ext_code.serialize(&mut writer).unwrap();\n                )?\n\n                BinarySerializable::serialize(\n                    &$value,\n                    &mut writer,\n                ).unwrap();\n            )*\n\n            writer\n        }};\n    }\n\n    #[test]\n    fn test_simple_value_serialize() {\n        let result = serialize_value(ReferenceValueLeaf::Null.into());\n        let expected = binary_repr!(\n            type_codes::NULL_CODE => (),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::Str(\"hello, world\").into());\n        let expected = binary_repr!(\n            type_codes::TEXT_CODE => String::from(\"hello, world\"),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::U64(123).into());\n        let expected = binary_repr!(\n            type_codes::U64_CODE => 123u64,\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::I64(-123).into());\n        let expected = binary_repr!(\n            type_codes::I64_CODE => -123i64,\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::F64(123.3845f64).into());\n        let expected = binary_repr!(\n            type_codes::F64_CODE => f64_to_u64(123.3845f64),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let result = serialize_value(ReferenceValueLeaf::Bool(false).into());\n        let expected = binary_repr!(\n            type_codes::BOOL_CODE => false,\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let facet = Facet::from_text(\"/hello/world\").unwrap();\n        let result = serialize_value(ReferenceValueLeaf::Facet(facet.encoded_str()).into());\n        let expected = binary_repr!(\n            type_codes::HIERARCHICAL_FACET_CODE => Facet::from_text(\"/hello/world\").unwrap(),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let pre_tok_str = PreTokenizedString {\n            text: \"hello, world\".to_string(),\n            tokens: vec![Token::default(), Token::default()],\n        };\n        let result =\n            serialize_value(ReferenceValueLeaf::PreTokStr(pre_tok_str.clone().into()).into());\n        let expected = binary_repr!(\n            type_codes::EXT_CODE, type_codes::TOK_STR_EXT_CODE => pre_tok_str,\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n    }\n\n    #[test]\n    fn test_array_serialize() {\n        let elements = [serde_json::Value::Null, serde_json::Value::Null];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let expected = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length elements.len(),\n            type_codes::NULL_CODE => (),\n            type_codes::NULL_CODE => (),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let elements = [\n            serde_json::Value::String(\"Hello, world\".into()),\n            serde_json::Value::String(\"Some demo\".into()),\n        ];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let expected = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length elements.len(),\n            type_codes::TEXT_CODE => String::from(\"Hello, world\"),\n            type_codes::TEXT_CODE => String::from(\"Some demo\"),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let elements = [];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let expected = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length elements.len(),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let elements = [\n            serde_json::Value::Null,\n            serde_json::Value::String(\"Hello, world\".into()),\n            serde_json::Value::Number(12345.into()),\n        ];\n        let result = serialize_value(ReferenceValue::Array(elements.iter()));\n        let expected = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length elements.len(),\n            type_codes::NULL_CODE => (),\n            type_codes::TEXT_CODE => String::from(\"Hello, world\"),\n            type_codes::I64_CODE => 12345i64,\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n    }\n\n    #[test]\n    fn test_object_serialize() {\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-first-key\".into(),\n            serde_json::Value::String(\"Hello\".into()),\n        );\n        object.insert(\"my-second-key\".into(), serde_json::Value::Null);\n        object.insert(\n            \"my-third-key\".into(),\n            serde_json::Value::Number(Number::from_f64(123.0).unwrap()),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let expected = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length object.len() * 2,  // To account for keys counting towards the length\n            type_codes::TEXT_CODE => String::from(\"my-first-key\"),\n            type_codes::TEXT_CODE => String::from(\"Hello\"),\n            type_codes::TEXT_CODE => String::from(\"my-second-key\"),\n            type_codes::NULL_CODE => (),\n            type_codes::TEXT_CODE => String::from(\"my-third-key\"),\n            type_codes::F64_CODE => f64_to_u64(123.0),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let object = serde_json::Map::new();\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let expected = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length object.len(),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        let mut object = serde_json::Map::new();\n        object.insert(\"my-first-key\".into(), serde_json::Value::Null);\n        object.insert(\"my-second-key\".into(), serde_json::Value::Null);\n        object.insert(\"my-third-key\".into(), serde_json::Value::Null);\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n        let expected = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length object.len() * 2, // To account for keys counting towards the length\n            type_codes::TEXT_CODE => String::from(\"my-first-key\"),\n            type_codes::NULL_CODE => (),\n            type_codes::TEXT_CODE => String::from(\"my-second-key\"),\n            type_codes::NULL_CODE => (),\n            type_codes::TEXT_CODE => String::from(\"my-third-key\"),\n            type_codes::NULL_CODE => (),\n        );\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n    }\n\n    #[test]\n    fn test_nested_serialize() {\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-array\".into(),\n            serde_json::Value::Array(vec![\n                serde_json::Value::Null,\n                serde_json::Value::String(String::from(\"bobby of the sea\")),\n            ]),\n        );\n        object.insert(\n            \"my-object\".into(),\n            serde_json::Value::Object(\n                vec![\n                    (\n                        \"inner-1\".to_string(),\n                        serde_json::Value::Number((-123i64).into()),\n                    ),\n                    (\n                        \"inner-2\".to_string(),\n                        serde_json::Value::String(String::from(\"bobby of the sea 2\")),\n                    ),\n                ]\n                .into_iter()\n                .collect(),\n            ),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n\n        let mut expected = Vec::new();\n        let header = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length object.len() * 2,\n        );\n        expected.extend_from_slice(&header);\n        expected\n            .extend_from_slice(&binary_repr!(type_codes::TEXT_CODE => String::from(\"my-array\")));\n        let nested_array = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length 2,\n            type_codes::NULL_CODE => (),\n            type_codes::TEXT_CODE => String::from(\"bobby of the sea\"),\n        );\n        expected.extend_from_slice(&nested_array);\n        expected\n            .extend_from_slice(&binary_repr!(type_codes::TEXT_CODE => String::from(\"my-object\")));\n        let nested_object = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length 4,   // 2 keys, 2 values\n            type_codes::TEXT_CODE => String::from(\"inner-1\"),\n            type_codes::I64_CODE => -123i64,\n            type_codes::TEXT_CODE => String::from(\"inner-2\"),\n            type_codes::TEXT_CODE => String::from(\"bobby of the sea 2\"),\n        );\n        expected.extend_from_slice(&nested_object);\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n\n        // Some more extreme nesting that might behave weirdly\n        let mut object = serde_json::Map::new();\n        object.insert(\n            \"my-array\".into(),\n            serde_json::Value::Array(vec![serde_json::Value::Array(vec![\n                serde_json::Value::Array(vec![]),\n                serde_json::Value::Array(vec![serde_json::Value::Null]),\n            ])]),\n        );\n        let result = serialize_value(ReferenceValue::Object(JsonObjectIter(object.iter())));\n\n        let mut expected = Vec::new();\n        let header = binary_repr!(\n            collection type_codes::OBJECT_CODE,\n            length object.len() * 2,\n        );\n        expected.extend_from_slice(&header);\n        expected\n            .extend_from_slice(&binary_repr!(type_codes::TEXT_CODE => String::from(\"my-array\")));\n        let nested_array = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length 1,\n        );\n        expected.extend_from_slice(&nested_array);\n        let nested_array = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length 2,\n        );\n        expected.extend_from_slice(&nested_array);\n        let nested_array = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length 0,\n        );\n        expected.extend_from_slice(&nested_array);\n        let nested_array = binary_repr!(\n            collection type_codes::ARRAY_CODE,\n            length 1,\n            type_codes::NULL_CODE => (),\n        );\n        expected.extend_from_slice(&nested_array);\n        assert_eq!(\n            result, expected,\n            \"Expected serialized value to match the binary representation\"\n        );\n    }\n\n    #[inline]\n    fn serialize_doc<D: Document>(doc: &D, schema: &Schema) -> Vec<u8> {\n        let mut writer = Vec::new();\n\n        let mut serializer = BinaryDocumentSerializer::new(&mut writer, schema);\n        serializer.serialize_doc(doc).expect(\"Serialize value\");\n\n        writer\n    }\n\n    /// A helper macro for generating the expected binary representation of the document.\n    macro_rules! expected_doc_data {\n        (length $len:expr) => {{\n            let mut writer = Vec::new();\n            VInt($len as u64).serialize(&mut writer).unwrap();\n            writer\n        }};\n        (length $len:expr, $( $field_id:expr => $value:expr $(,)?)*) => {{\n            let mut writer = Vec::new();\n\n            VInt($len as u64).serialize(&mut writer).unwrap();\n            $(\n                $field_id.serialize(&mut writer).unwrap();\n                $value.serialize(&mut writer).unwrap();\n            )*\n\n            writer\n        }};\n    }\n\n    #[test]\n    fn test_document_serialize() {\n        let mut builder = Schema::builder();\n        let name = builder.add_text_field(\"name\", TEXT | STORED);\n        let age = builder.add_u64_field(\"age\", FAST | STORED);\n        let schema = builder.build();\n\n        let mut document = BTreeMap::new();\n        document.insert(name, crate::schema::OwnedValue::Str(\"ChillFish8\".into()));\n        document.insert(age, crate::schema::OwnedValue::U64(20));\n\n        let result = serialize_doc(&document, &schema);\n        let mut expected = expected_doc_data!(length document.len());\n        name.serialize(&mut expected).unwrap();\n        expected\n            .extend_from_slice(&binary_repr!(type_codes::TEXT_CODE => String::from(\"ChillFish8\")));\n        age.serialize(&mut expected).unwrap();\n        expected.extend_from_slice(&binary_repr!(type_codes::U64_CODE => 20u64));\n        assert_eq!(\n            result, expected,\n            \"Expected serialized document to match the binary representation\"\n        );\n\n        let mut builder = Schema::builder();\n        let name = builder.add_text_field(\"name\", TEXT | STORED);\n        // This should be skipped when serializing.\n        let age = builder.add_u64_field(\"age\", FAST);\n        let schema = builder.build();\n\n        let mut document = BTreeMap::new();\n        document.insert(name, crate::schema::OwnedValue::Str(\"ChillFish8\".into()));\n        document.insert(age, crate::schema::OwnedValue::U64(20));\n\n        let result = serialize_doc(&document, &schema);\n        let mut expected = expected_doc_data!(length 1);\n        name.serialize(&mut expected).unwrap();\n        expected\n            .extend_from_slice(&binary_repr!(type_codes::TEXT_CODE => String::from(\"ChillFish8\")));\n        assert_eq!(\n            result, expected,\n            \"Expected serialized document to match the binary representation\"\n        );\n\n        let builder = Schema::builder();\n        let schema = builder.build();\n        let document = BTreeMap::<Field, crate::schema::OwnedValue>::new();\n        let result = serialize_doc(&document, &schema);\n        let expected = expected_doc_data!(length document.len());\n        assert_eq!(\n            result, expected,\n            \"Expected serialized document to match the binary representation\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/document/value.rs",
    "content": "use std::fmt::Debug;\nuse std::net::Ipv6Addr;\n\nuse common::DateTime;\n\nuse crate::tokenizer::PreTokenizedString;\n\n/// A single field value.\npub trait Value<'a>: Send + Sync + Debug {\n    /// The child value type returned by this doc value.\n    /// The iterator for walking through the elements within the array.\n    type ArrayIter: Iterator<Item = Self>;\n    /// The visitor walking through the key-value pairs within\n    /// the object.\n    type ObjectIter: Iterator<Item = (&'a str, Self)>;\n\n    /// Returns the field value represented by an enum which borrows it's data.\n    fn as_value(&self) -> ReferenceValue<'a, Self>;\n\n    #[inline]\n    /// If the Value is a leaf, returns the associated leaf. Returns None otherwise.\n    fn as_leaf(&self) -> Option<ReferenceValueLeaf<'a>> {\n        if let ReferenceValue::Leaf(val) = self.as_value() {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a String, returns the associated str. Returns None otherwise.\n    fn as_str(&self) -> Option<&'a str> {\n        self.as_leaf().and_then(|leaf| leaf.as_str())\n    }\n\n    #[inline]\n    /// If the Value is a u64, returns the associated u64. Returns None otherwise.\n    fn as_u64(&self) -> Option<u64> {\n        self.as_leaf().and_then(|leaf| leaf.as_u64())\n    }\n\n    #[inline]\n    /// If the Value is a i64, returns the associated i64. Returns None otherwise.\n    fn as_i64(&self) -> Option<i64> {\n        self.as_leaf().and_then(|leaf| leaf.as_i64())\n    }\n\n    #[inline]\n    /// If the Value is a f64, returns the associated f64. Returns None otherwise.\n    fn as_f64(&self) -> Option<f64> {\n        self.as_leaf().and_then(|leaf| leaf.as_f64())\n    }\n\n    #[inline]\n    /// If the Value is a datetime, returns the associated datetime. Returns None otherwise.\n    fn as_datetime(&self) -> Option<DateTime> {\n        self.as_leaf().and_then(|leaf| leaf.as_datetime())\n    }\n\n    #[inline]\n    /// If the Value is a IP address, returns the associated IP. Returns None otherwise.\n    fn as_ip_addr(&self) -> Option<Ipv6Addr> {\n        self.as_leaf().and_then(|leaf| leaf.as_ip_addr())\n    }\n\n    #[inline]\n    /// If the Value is a bool, returns the associated bool. Returns None otherwise.\n    fn as_bool(&self) -> Option<bool> {\n        self.as_leaf().and_then(|leaf| leaf.as_bool())\n    }\n\n    #[inline]\n    /// If the Value is a pre-tokenized string, returns the associated string. Returns None\n    /// otherwise.\n    fn as_pre_tokenized_text(&self) -> Option<Box<PreTokenizedString>> {\n        self.as_leaf()\n            .and_then(|leaf| leaf.into_pre_tokenized_text())\n    }\n\n    #[inline]\n    /// If the Value is a bytes value, returns the associated set of bytes. Returns None otherwise.\n    fn as_bytes(&self) -> Option<&'a [u8]> {\n        self.as_leaf().and_then(|leaf| leaf.as_bytes())\n    }\n\n    #[inline]\n    /// If the Value is a facet, returns the associated facet. Returns None otherwise.\n    fn as_facet(&self) -> Option<&'a str> {\n        self.as_leaf().and_then(|leaf| leaf.as_facet())\n    }\n\n    #[inline]\n    /// Returns the iterator over the array if the Value is an array.\n    fn as_array(&self) -> Option<Self::ArrayIter> {\n        if let ReferenceValue::Array(val) = self.as_value() {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// Returns the iterator over the object if the Value is an object.\n    fn as_object(&self) -> Option<Self::ObjectIter> {\n        if let ReferenceValue::Object(val) = self.as_value() {\n            Some(val)\n        } else {\n            None\n        }\n    }\n}\n\n/// A enum representing a leaf value for tantivy to index.\n#[derive(Clone, Debug, PartialEq)]\npub enum ReferenceValueLeaf<'a> {\n    /// A null value.\n    Null,\n    /// The str type is used for any text information.\n    Str(&'a str),\n    /// Unsigned 64-bits Integer `u64`\n    U64(u64),\n    /// Signed 64-bits Integer `i64`\n    I64(i64),\n    /// 64-bits Float `f64`\n    F64(f64),\n    /// Date/time with nanoseconds precision\n    Date(DateTime),\n    /// Facet string needs to match the format of\n    /// [Facet::encoded_str](crate::schema::Facet::encoded_str).\n    Facet(&'a str),\n    /// Arbitrarily sized byte array\n    Bytes(&'a [u8]),\n    /// IpV6 Address. Internally there is no IpV4, it needs to be converted to `Ipv6Addr`.\n    IpAddr(Ipv6Addr),\n    /// Bool value\n    Bool(bool),\n    /// Pre-tokenized str type,\n    PreTokStr(Box<PreTokenizedString>),\n}\n\nimpl From<u64> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: u64) -> Self {\n        ReferenceValueLeaf::U64(value)\n    }\n}\n\nimpl From<i64> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: i64) -> Self {\n        ReferenceValueLeaf::I64(value)\n    }\n}\n\nimpl From<f64> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: f64) -> Self {\n        ReferenceValueLeaf::F64(value)\n    }\n}\n\nimpl From<bool> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: bool) -> Self {\n        ReferenceValueLeaf::Bool(value)\n    }\n}\n\nimpl<'a> From<&'a str> for ReferenceValueLeaf<'a> {\n    #[inline]\n    fn from(value: &'a str) -> Self {\n        ReferenceValueLeaf::Str(value)\n    }\n}\n\nimpl<'a> From<&'a [u8]> for ReferenceValueLeaf<'a> {\n    #[inline]\n    fn from(value: &'a [u8]) -> Self {\n        ReferenceValueLeaf::Bytes(value)\n    }\n}\n\nimpl From<DateTime> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: DateTime) -> Self {\n        ReferenceValueLeaf::Date(value)\n    }\n}\n\nimpl From<Ipv6Addr> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(value: Ipv6Addr) -> Self {\n        ReferenceValueLeaf::IpAddr(value)\n    }\n}\n\nimpl From<PreTokenizedString> for ReferenceValueLeaf<'_> {\n    #[inline]\n    fn from(val: PreTokenizedString) -> Self {\n        ReferenceValueLeaf::PreTokStr(Box::new(val))\n    }\n}\n\nimpl<'a, T: Value<'a> + ?Sized> From<ReferenceValueLeaf<'a>> for ReferenceValue<'a, T> {\n    #[inline]\n    fn from(value: ReferenceValueLeaf<'a>) -> Self {\n        match value {\n            ReferenceValueLeaf::Null => ReferenceValue::Leaf(ReferenceValueLeaf::Null),\n            ReferenceValueLeaf::Str(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Str(val)),\n            ReferenceValueLeaf::U64(val) => ReferenceValue::Leaf(ReferenceValueLeaf::U64(val)),\n            ReferenceValueLeaf::I64(val) => ReferenceValue::Leaf(ReferenceValueLeaf::I64(val)),\n            ReferenceValueLeaf::F64(val) => ReferenceValue::Leaf(ReferenceValueLeaf::F64(val)),\n            ReferenceValueLeaf::Date(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Date(val)),\n            ReferenceValueLeaf::Facet(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Facet(val)),\n            ReferenceValueLeaf::Bytes(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Bytes(val)),\n            ReferenceValueLeaf::IpAddr(val) => {\n                ReferenceValue::Leaf(ReferenceValueLeaf::IpAddr(val))\n            }\n            ReferenceValueLeaf::Bool(val) => ReferenceValue::Leaf(ReferenceValueLeaf::Bool(val)),\n            ReferenceValueLeaf::PreTokStr(val) => {\n                ReferenceValue::Leaf(ReferenceValueLeaf::PreTokStr(val))\n            }\n        }\n    }\n}\n\nimpl<'a> ReferenceValueLeaf<'a> {\n    #[inline]\n    /// Returns if the value is `null` or not.\n    pub fn is_null(&self) -> bool {\n        matches!(self, Self::Null)\n    }\n\n    #[inline]\n    /// If the Value is a String, returns the associated str. Returns None otherwise.\n    pub fn as_str(&self) -> Option<&'a str> {\n        if let Self::Str(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a u64, returns the associated u64. Returns None otherwise.\n    pub fn as_u64(&self) -> Option<u64> {\n        if let Self::U64(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a i64, returns the associated i64. Returns None otherwise.\n    pub fn as_i64(&self) -> Option<i64> {\n        if let Self::I64(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a f64, returns the associated f64. Returns None otherwise.\n    pub fn as_f64(&self) -> Option<f64> {\n        if let Self::F64(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a datetime, returns the associated datetime. Returns None otherwise.\n    pub fn as_datetime(&self) -> Option<DateTime> {\n        if let Self::Date(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a IP address, returns the associated IP. Returns None otherwise.\n    pub fn as_ip_addr(&self) -> Option<Ipv6Addr> {\n        if let Self::IpAddr(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a bool, returns the associated bool. Returns None otherwise.\n    pub fn as_bool(&self) -> Option<bool> {\n        if let Self::Bool(val) = self {\n            Some(*val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a pre-tokenized string, consumes it and returns the string.\n    /// Returns None otherwise.\n    pub fn into_pre_tokenized_text(self) -> Option<Box<PreTokenizedString>> {\n        if let Self::PreTokStr(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a bytes value, returns the associated set of bytes. Returns None otherwise.\n    pub fn as_bytes(&self) -> Option<&'a [u8]> {\n        if let Self::Bytes(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a facet, returns the associated facet. Returns None otherwise.\n    pub fn as_facet(&self) -> Option<&'a str> {\n        if let Self::Facet(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n}\n\n/// A enum representing a value for tantivy to index.\n#[derive(Clone, Debug, PartialEq)]\npub enum ReferenceValue<'a, V>\nwhere V: Value<'a> + ?Sized\n{\n    /// A null value.\n    Leaf(ReferenceValueLeaf<'a>),\n    /// A an array containing multiple values.\n    Array(V::ArrayIter),\n    /// A nested / dynamic object.\n    Object(V::ObjectIter),\n}\n\nimpl<'a, V> ReferenceValue<'a, V>\nwhere V: Value<'a>\n{\n    #[inline]\n    /// Returns if the value is `null` or not.\n    pub fn is_null(&self) -> bool {\n        matches!(self, Self::Leaf(ReferenceValueLeaf::Null))\n    }\n\n    #[inline]\n    /// If the Value is a leaf, returns the associated leaf. Returns None otherwise.\n    pub fn as_leaf(&self) -> Option<&ReferenceValueLeaf<'a>> {\n        if let Self::Leaf(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a leaf, consume it and return the leaf. Returns None otherwise.\n    pub fn into_leaf(self) -> Option<ReferenceValueLeaf<'a>> {\n        if let Self::Leaf(val) = self {\n            Some(val)\n        } else {\n            None\n        }\n    }\n\n    #[inline]\n    /// If the Value is a String, returns the associated str. Returns None otherwise.\n    pub fn as_str(&self) -> Option<&'a str> {\n        self.as_leaf().and_then(|leaf| leaf.as_str())\n    }\n\n    #[inline]\n    /// If the Value is a u64, returns the associated u64. Returns None otherwise.\n    pub fn as_u64(&self) -> Option<u64> {\n        self.as_leaf().and_then(|leaf| leaf.as_u64())\n    }\n\n    #[inline]\n    /// If the Value is a i64, returns the associated i64. Returns None otherwise.\n    pub fn as_i64(&self) -> Option<i64> {\n        self.as_leaf().and_then(|leaf| leaf.as_i64())\n    }\n\n    #[inline]\n    /// If the Value is a f64, returns the associated f64. Returns None otherwise.\n    pub fn as_f64(&self) -> Option<f64> {\n        self.as_leaf().and_then(|leaf| leaf.as_f64())\n    }\n\n    #[inline]\n    /// If the Value is a datetime, returns the associated datetime. Returns None otherwise.\n    pub fn as_datetime(&self) -> Option<DateTime> {\n        self.as_leaf().and_then(|leaf| leaf.as_datetime())\n    }\n\n    #[inline]\n    /// If the Value is a IP address, returns the associated IP. Returns None otherwise.\n    pub fn as_ip_addr(&self) -> Option<Ipv6Addr> {\n        self.as_leaf().and_then(|leaf| leaf.as_ip_addr())\n    }\n\n    #[inline]\n    /// If the Value is a bool, returns the associated bool. Returns None otherwise.\n    pub fn as_bool(&self) -> Option<bool> {\n        self.as_leaf().and_then(|leaf| leaf.as_bool())\n    }\n\n    #[inline]\n    /// If the Value is a pre-tokenized string, consumes it and returns the string.\n    /// Returns None otherwise.\n    pub fn into_pre_tokenized_text(self) -> Option<Box<PreTokenizedString>> {\n        self.into_leaf()\n            .and_then(|leaf| leaf.into_pre_tokenized_text())\n    }\n\n    #[inline]\n    /// If the Value is a bytes value, returns the associated set of bytes. Returns None otherwise.\n    pub fn as_bytes(&self) -> Option<&'a [u8]> {\n        self.as_leaf().and_then(|leaf| leaf.as_bytes())\n    }\n\n    #[inline]\n    /// If the Value is a facet, returns the associated facet. Returns None otherwise.\n    pub fn as_facet(&self) -> Option<&'a str> {\n        self.as_leaf().and_then(|leaf| leaf.as_facet())\n    }\n\n    #[inline]\n    /// Returns true if the Value is an array.\n    pub fn is_array(&self) -> bool {\n        matches!(self, Self::Array(_))\n    }\n\n    #[inline]\n    /// Returns true if the Value is an object.\n    pub fn is_object(&self) -> bool {\n        matches!(self, Self::Object(_))\n    }\n}\n"
  },
  {
    "path": "src/schema/facet.rs",
    "content": "use std::borrow::{Borrow, Cow};\nuse std::fmt::{self, Debug, Display, Formatter};\nuse std::io::{self, Read, Write};\nuse std::str;\nuse std::string::FromUtf8Error;\n\nuse common::*;\nuse once_cell::sync::Lazy;\nuse regex::Regex;\nuse serde::de::Error as _;\nuse serde::{Deserialize, Deserializer, Serialize, Serializer};\n\nconst SLASH_BYTE: u8 = b'/';\nconst ESCAPE_BYTE: u8 = b'\\\\';\n\n/// BYTE used as a level separation in the binary\n/// representation of facets.\npub const FACET_SEP_BYTE: u8 = 0u8;\n\n/// `char` used as a level separation in the binary\n/// representation of facets. (It is the null codepoint.)\npub const FACET_SEP_CHAR: char = '\\u{0}';\n\n/// An error enum for facet parser.\n#[derive(Debug, PartialEq, Eq, Error)]\npub enum FacetParseError {\n    /// The facet text representation is unparsable.\n    #[error(\"Failed to parse the facet string: '{0}'\")]\n    FacetParseError(String),\n}\n\n/// A Facet represent a point in a given hierarchy.\n///\n/// They are typically represented similarly to a filepath.\n/// For instance, an e-commerce website could\n/// have a `Facet` for `/electronics/tv_and_video/led_tv`.\n///\n/// A document can be associated with any number of facets.\n/// The hierarchy implicitly imply that a document\n/// belonging to a facet also belongs to the ancestor of\n/// its facet. In the example above, `/electronics/tv_and_video/`\n/// and `/electronics`.\n#[derive(Clone, Default, Eq, Hash, PartialEq, Ord, PartialOrd)]\npub struct Facet(pub(crate) String);\n\nimpl Facet {\n    /// Returns a new instance of the \"root facet\"\n    /// Equivalent to `/`.\n    pub fn root() -> Facet {\n        Facet(\"\".to_string())\n    }\n\n    /// Returns true if the facet is the root facet `/`.\n    pub fn is_root(&self) -> bool {\n        self.encoded_str().is_empty()\n    }\n\n    /// Returns a binary representation of the facet.\n    ///\n    /// In this representation, `0u8` is used as a separator\n    /// and the string parts of the facet are unescaped.\n    /// (The first `/` is not encoded at all).\n    ///\n    /// This representation has the benefit of making it possible to\n    /// express \"being a child of a given facet\" as a range over\n    /// the term ordinals.\n    pub fn encoded_str(&self) -> &str {\n        &self.0\n    }\n\n    pub(crate) fn from_encoded_string(facet_string: String) -> Facet {\n        Facet(facet_string)\n    }\n\n    /// Creates a `Facet` from its binary representation.\n    pub fn from_encoded(encoded_bytes: Vec<u8>) -> Result<Facet, FromUtf8Error> {\n        // facet bytes validation. `0u8` is used a separator but that is still legal utf-8\n        String::from_utf8(encoded_bytes).map(Facet)\n    }\n\n    /// Parse a text representation of a facet.\n    ///\n    /// If one of the segments of this path\n    /// contains a `/`, it should be escaped\n    /// using an anti-slash `\\`.\n    pub fn from_text<T>(path: &T) -> Result<Facet, FacetParseError>\n    where T: ?Sized + AsRef<str> {\n        #[derive(Copy, Clone)]\n        enum State {\n            Escaped,\n            Idle,\n        }\n        let path_ref = path.as_ref();\n        if path_ref.is_empty() {\n            return Err(FacetParseError::FacetParseError(path_ref.to_string()));\n        }\n        if !path_ref.starts_with('/') {\n            return Err(FacetParseError::FacetParseError(path_ref.to_string()));\n        }\n        let mut facet_encoded = String::new();\n        let mut state = State::Idle;\n        let path_bytes = path_ref.as_bytes();\n        let mut last_offset = 1;\n        for i in 1..path_bytes.len() {\n            let c = path_bytes[i];\n            match (state, c) {\n                (State::Idle, ESCAPE_BYTE) => {\n                    facet_encoded.push_str(&path_ref[last_offset..i]);\n                    last_offset = i + 1;\n                    state = State::Escaped\n                }\n                (State::Idle, SLASH_BYTE) => {\n                    facet_encoded.push_str(&path_ref[last_offset..i]);\n                    facet_encoded.push(FACET_SEP_CHAR);\n                    last_offset = i + 1;\n                }\n                (State::Escaped, _escaped_char) => {\n                    state = State::Idle;\n                }\n                (State::Idle, _any_char) => {}\n            }\n        }\n        facet_encoded.push_str(&path_ref[last_offset..]);\n        Ok(Facet(facet_encoded))\n    }\n\n    /// Returns a `Facet` from an iterator over the different\n    /// steps of the facet path.\n    ///\n    /// The steps are expected to be unescaped.\n    pub fn from_path<Path>(path: Path) -> Facet\n    where\n        Path: IntoIterator,\n        Path::Item: AsRef<str>,\n    {\n        let mut facet_string: String = String::with_capacity(100);\n        let mut step_it = path.into_iter();\n        if let Some(step) = step_it.next() {\n            facet_string.push_str(step.as_ref());\n        }\n        for step in step_it {\n            facet_string.push(FACET_SEP_CHAR);\n            facet_string.push_str(step.as_ref());\n        }\n        Facet(facet_string)\n    }\n\n    /// Returns `true` if other is a `strict` subfacet of `self`.\n    ///\n    /// Disclaimer: By strict we mean that the relation is not reflexive.\n    /// `/happy` is not a prefix of `/happy`.\n    pub fn is_prefix_of(&self, other: &Facet) -> bool {\n        let self_str = self.encoded_str();\n        let other_str = other.encoded_str();\n\n        // Fast path, but also required to ensure that / is not a prefix of /.\n        if other_str.len() <= self_str.len() {\n            return false;\n        }\n\n        // Root is a prefix of every other path.\n        // This is not just an optimisation. It is necessary for correctness.\n        if self.is_root() {\n            return true;\n        }\n\n        other_str.starts_with(self_str) && other_str.as_bytes()[self_str.len()] == FACET_SEP_BYTE\n    }\n\n    /// Extract path from the `Facet`.\n    pub fn to_path(&self) -> Vec<&str> {\n        self.encoded_str().split(FACET_SEP_CHAR).collect()\n    }\n\n    /// This function is the inverse of Facet::from(&str).\n    pub fn to_path_string(&self) -> String {\n        format!(\"{self}\")\n    }\n}\n\nimpl Borrow<str> for Facet {\n    fn borrow(&self) -> &str {\n        self.encoded_str()\n    }\n}\n\nimpl<'a, T: ?Sized + AsRef<str>> From<&'a T> for Facet {\n    fn from(path_asref: &'a T) -> Facet {\n        Facet::from_text(path_asref).unwrap()\n    }\n}\n\nimpl BinarySerializable for Facet {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        <String as BinarySerializable>::serialize(&self.0, writer)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        Ok(Facet(<String as BinarySerializable>::deserialize(reader)?))\n    }\n}\n\nimpl Display for Facet {\n    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {\n        for step in self.0.split(FACET_SEP_CHAR) {\n            write!(f, \"/\")?;\n            write!(f, \"{}\", escape_slashes(step))?;\n        }\n        Ok(())\n    }\n}\n\nfn escape_slashes(s: &str) -> Cow<'_, str> {\n    static SLASH_PTN: Lazy<Regex> = Lazy::new(|| Regex::new(r\"[\\\\/]\").unwrap());\n    SLASH_PTN.replace_all(s, \"\\\\/\")\n}\n\nimpl Serialize for Facet {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: Serializer {\n        serializer.serialize_str(&self.to_string())\n    }\n}\n\nimpl<'de> Deserialize<'de> for Facet {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: Deserializer<'de> {\n        <Cow<'de, str> as Deserialize<'de>>::deserialize(deserializer).and_then(|path| {\n            Facet::from_text(&*path).map_err(|err| D::Error::custom(err.to_string()))\n        })\n    }\n}\n\nimpl Debug for Facet {\n    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {\n        write!(f, \"Facet({self})\")?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::{Facet, FacetParseError};\n\n    #[test]\n    fn test_root() {\n        assert_eq!(Facet::root(), Facet::from(\"/\"));\n        assert_eq!(format!(\"{}\", Facet::root()), \"/\");\n        assert!(Facet::root().is_root());\n        assert_eq!(Facet::root().encoded_str(), \"\");\n    }\n\n    #[test]\n    fn test_from_path() {\n        assert_eq!(\n            Facet::from_path(vec![\"top\", \"a\", \"firstdoc\"]),\n            Facet::from(\"/top/a/firstdoc\")\n        );\n    }\n\n    #[test]\n    fn test_facet_display() {\n        {\n            let v = [\"first\", \"second\", \"third\"];\n            let facet = Facet::from_path(v.iter());\n            assert_eq!(format!(\"{facet}\"), \"/first/second/third\");\n        }\n        {\n            let v = [\"first\", \"sec/ond\", \"third\"];\n            let facet = Facet::from_path(v.iter());\n            assert_eq!(format!(\"{facet}\"), \"/first/sec\\\\/ond/third\");\n        }\n    }\n\n    #[test]\n    fn test_facet_debug() {\n        let v = [\"first\", \"second\", \"third\"];\n        let facet = Facet::from_path(v.iter());\n        assert_eq!(format!(\"{facet:?}\"), \"Facet(/first/second/third)\");\n    }\n\n    #[test]\n    fn test_to_path() {\n        let v = [\"first\", \"second\", \"third\\\\/not_fourth\"];\n        let facet = Facet::from_path(v.iter());\n        assert_eq!(facet.to_path(), v);\n    }\n\n    #[test]\n    fn test_to_path_string() {\n        let v = [\"first\", \"second\", \"third/not_fourth\"];\n        let facet = Facet::from_path(v.iter());\n        assert_eq!(\n            facet.to_path_string(),\n            String::from(\"/first/second/third\\\\/not_fourth\")\n        );\n    }\n\n    #[test]\n    fn test_to_path_string_empty() {\n        let v: Vec<&str> = vec![];\n        let facet = Facet::from_path(v.iter());\n        assert_eq!(facet.to_path_string(), \"/\");\n    }\n\n    #[test]\n    fn test_from_text() {\n        assert_eq!(\n            Err(FacetParseError::FacetParseError(\"INVALID\".to_string())),\n            Facet::from_text(\"INVALID\")\n        );\n    }\n\n    #[test]\n    fn only_proper_prefixes() {\n        assert!(Facet::from(\"/foo\").is_prefix_of(&Facet::from(\"/foo/bar\")));\n\n        assert!(!Facet::from(\"/foo/bar\").is_prefix_of(&Facet::from(\"/foo/bar\")));\n    }\n\n    #[test]\n    fn root_is_a_prefix() {\n        assert!(Facet::from(\"/\").is_prefix_of(&Facet::from(\"/foobar\")));\n        assert!(!Facet::from(\"/\").is_prefix_of(&Facet::from(\"/\")));\n    }\n\n    #[test]\n    fn deserialize_from_borrowed_string() {\n        let facet = serde_json::from_str::<Facet>(r#\"\"/foo/bar\"\"#).unwrap();\n        assert_eq!(facet, Facet::from_path([\"foo\", \"bar\"]));\n    }\n\n    #[test]\n    fn deserialize_from_owned_string() {\n        let facet = serde_json::from_str::<Facet>(r#\"\"/foo/\\u263A\"\"#).unwrap();\n        assert_eq!(facet, Facet::from_path([\"foo\", \"☺\"]));\n    }\n\n    #[test]\n    fn deserialize_from_invalid_string() {\n        let error = serde_json::from_str::<Facet>(r#\"\"foo/bar\"\"#).unwrap_err();\n        assert_eq!(\n            error.to_string(),\n            \"Failed to parse the facet string: 'foo/bar'\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/facet_options.rs",
    "content": "use std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse crate::schema::flags::{IndexedFlag, SchemaFlagList, StoredFlag};\n\n/// Define how a facet field should be handled by tantivy.\n///\n/// Note that a Facet is always indexed and stored as a fastfield.\n#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)]\npub struct FacetOptions {\n    stored: bool,\n}\n\nimpl FacetOptions {\n    /// Returns true if the value is stored.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Set the field as stored.\n    ///\n    /// Only the fields that are set as *stored* are\n    /// persisted into the Tantivy's store.\n    #[must_use]\n    pub fn set_stored(mut self) -> FacetOptions {\n        self.stored = true;\n        self\n    }\n}\n\nimpl From<()> for FacetOptions {\n    fn from(_: ()) -> FacetOptions {\n        FacetOptions::default()\n    }\n}\n\nimpl From<StoredFlag> for FacetOptions {\n    fn from(_: StoredFlag) -> Self {\n        FacetOptions { stored: true }\n    }\n}\n\nimpl<T: Into<FacetOptions>> BitOr<T> for FacetOptions {\n    type Output = FacetOptions;\n\n    fn bitor(self, other: T) -> FacetOptions {\n        let other = other.into();\n        FacetOptions {\n            stored: self.stored | other.stored,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for FacetOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\nimpl From<IndexedFlag> for FacetOptions {\n    fn from(_: IndexedFlag) -> Self {\n        FacetOptions { stored: false }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::schema::{FacetOptions, INDEXED};\n\n    #[test]\n    fn test_from_index_flag() {\n        let facet_option = FacetOptions::from(INDEXED);\n        assert_eq!(facet_option, FacetOptions::default());\n    }\n}\n"
  },
  {
    "path": "src/schema/field.rs",
    "content": "use std::io;\nuse std::io::{Read, Write};\n\nuse common::BinarySerializable;\n\n/// `Field` is represented by an unsigned 32-bit integer type.\n/// The schema holds the mapping between field names and `Field` objects.\n#[derive(\n    Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash, serde::Serialize, serde::Deserialize,\n)]\npub struct Field(u32);\n\nimpl Field {\n    /// Create a new field object for the given FieldId.\n    pub const fn from_field_id(field_id: u32) -> Field {\n        Field(field_id)\n    }\n\n    /// Returns a u32 identifying uniquely a field within a schema.\n    pub const fn field_id(self) -> u32 {\n        self.0\n    }\n}\n\nimpl BinarySerializable for Field {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        self.0.serialize(writer)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Field> {\n        u32::deserialize(reader).map(Field)\n    }\n}\n"
  },
  {
    "path": "src/schema/field_entry.rs",
    "content": "use serde::{Deserialize, Serialize};\n\nuse super::ip_options::IpAddrOptions;\nuse crate::schema::bytes_options::BytesOptions;\nuse crate::schema::{\n    is_valid_field_name, DateOptions, FacetOptions, FieldType, JsonObjectOptions, NumericOptions,\n    TextOptions,\n};\n\n/// A `FieldEntry` represents a field and its configuration.\n/// `Schema` are a collection of `FieldEntry`\n///\n/// It consists of\n/// - a field name\n/// - a field type, itself wrapping up options describing how the field should be indexed.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\npub struct FieldEntry {\n    name: String,\n    #[serde(flatten)]\n    field_type: FieldType,\n}\n\nimpl FieldEntry {\n    /// Creates a new field entry given a name and a field type\n    pub fn new(field_name: String, field_type: FieldType) -> FieldEntry {\n        assert!(is_valid_field_name(&field_name));\n        FieldEntry {\n            name: field_name,\n            field_type,\n        }\n    }\n\n    /// Creates a new text field entry.\n    pub fn new_text(field_name: String, text_options: TextOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::Str(text_options))\n    }\n\n    /// Creates a new u64 field entry.\n    pub fn new_u64(field_name: String, int_options: NumericOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::U64(int_options))\n    }\n\n    /// Creates a new i64 field entry.\n    pub fn new_i64(field_name: String, int_options: NumericOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::I64(int_options))\n    }\n\n    /// Creates a new f64 field entry.\n    pub fn new_f64(field_name: String, f64_options: NumericOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::F64(f64_options))\n    }\n\n    /// Creates a new bool field entry.\n    pub fn new_bool(field_name: String, bool_options: NumericOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::Bool(bool_options))\n    }\n\n    /// Creates a new date field entry.\n    pub fn new_date(field_name: String, date_options: DateOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::Date(date_options))\n    }\n\n    /// Creates a new ip address field entry.\n    pub fn new_ip_addr(field_name: String, ip_options: IpAddrOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::IpAddr(ip_options))\n    }\n\n    /// Creates a field entry for a facet.\n    pub fn new_facet(field_name: String, facet_options: FacetOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::Facet(facet_options))\n    }\n\n    /// Creates a field entry for a bytes field\n    pub fn new_bytes(field_name: String, bytes_options: BytesOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::Bytes(bytes_options))\n    }\n\n    /// Creates a field entry for a json field\n    pub fn new_json(field_name: String, json_object_options: JsonObjectOptions) -> FieldEntry {\n        Self::new(field_name, FieldType::JsonObject(json_object_options))\n    }\n\n    /// Returns the name of the field\n    pub fn name(&self) -> &str {\n        &self.name\n    }\n\n    /// Returns the field type\n    pub fn field_type(&self) -> &FieldType {\n        &self.field_type\n    }\n\n    /// Returns true if the field is indexed.\n    ///\n    /// An indexed field is searchable.\n    pub fn is_indexed(&self) -> bool {\n        self.field_type.is_indexed()\n    }\n\n    /// Returns true if the field is normed\n    pub fn has_fieldnorms(&self) -> bool {\n        self.field_type.has_fieldnorms()\n    }\n\n    /// Returns true if the field is a fast field\n    pub fn is_fast(&self) -> bool {\n        self.field_type.is_fast()\n    }\n\n    /// Returns true if the field has the expand dots option set (for json fields)\n    pub fn is_expand_dots_enabled(&self) -> bool {\n        match self.field_type {\n            FieldType::JsonObject(ref options) => options.is_expand_dots_enabled(),\n            _ => false,\n        }\n    }\n\n    /// Returns true if the field is stored\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        match self.field_type {\n            FieldType::U64(ref options)\n            | FieldType::I64(ref options)\n            | FieldType::F64(ref options)\n            | FieldType::Bool(ref options) => options.is_stored(),\n            FieldType::Date(ref options) => options.is_stored(),\n            FieldType::Str(ref options) => options.is_stored(),\n            FieldType::Facet(ref options) => options.is_stored(),\n            FieldType::Bytes(ref options) => options.is_stored(),\n            FieldType::JsonObject(ref options) => options.is_stored(),\n            FieldType::IpAddr(ref options) => options.is_stored(),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n    use crate::schema::{Schema, TextFieldIndexing, TEXT};\n    use crate::Index;\n\n    #[test]\n    #[should_panic]\n    fn test_invalid_field_name_should_panic() {\n        FieldEntry::new_text(\"-hello\".to_string(), TEXT);\n    }\n\n    #[test]\n    fn test_json_serialization() {\n        let field_value = FieldEntry::new_text(String::from(\"title\"), TEXT);\n\n        let expected = r#\"{\n  \"name\": \"title\",\n  \"type\": \"text\",\n  \"options\": {\n    \"indexing\": {\n      \"record\": \"position\",\n      \"fieldnorms\": true,\n      \"tokenizer\": \"default\"\n    },\n    \"stored\": false,\n    \"fast\": false\n  }\n}\"#;\n        let field_value_json = serde_json::to_string_pretty(&field_value).unwrap();\n\n        assert_eq!(expected, &field_value_json);\n\n        let field_value: FieldEntry = serde_json::from_str(expected).unwrap();\n\n        assert_eq!(\"title\", field_value.name);\n\n        match field_value.field_type {\n            FieldType::Str(_) => {}\n            _ => panic!(\"expected FieldType::Str\"),\n        }\n    }\n\n    #[test]\n    fn test_json_deserialization() {\n        let json_str = r#\"{\n  \"name\": \"title\",\n  \"options\": {\n    \"indexing\": {\n      \"record\": \"position\",\n      \"fieldnorms\": true,\n      \"tokenizer\": \"default\"\n    },\n    \"stored\": false\n  },\n  \"type\": \"text\"\n}\"#;\n        let field_entry: FieldEntry = serde_json::from_str(json_str).unwrap();\n        match field_entry.field_type {\n            FieldType::Str(_) => {}\n            _ => panic!(\"expected FieldType::Str\"),\n        }\n    }\n\n    #[test]\n    fn test_missing_fieldnorms() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let no_field_norm = TextOptions::default()\n            .set_indexing_options(TextFieldIndexing::default().set_fieldnorms(false));\n        let text = schema_builder.add_text_field(\"text\", no_field_norm);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer = index.writer_for_tests()?;\n        index_writer.add_document(doc!(text=>\"abc\"))?;\n        index_writer.commit()?;\n        let searcher = index.reader()?.searcher();\n        let err = searcher.segment_reader(0u32).get_fieldnorms_reader(text);\n        assert!(matches!(err, Err(crate::TantivyError::SchemaError(_))));\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/schema/field_type.rs",
    "content": "use std::net::IpAddr;\nuse std::str::FromStr;\n\nuse base64::engine::general_purpose::STANDARD as BASE64;\nuse base64::Engine;\nuse columnar::{ColumnType, NumericalType};\nuse serde::{Deserialize, Serialize};\nuse serde_json::Value as JsonValue;\nuse thiserror::Error;\n\nuse super::ip_options::IpAddrOptions;\nuse super::IntoIpv6Addr;\nuse crate::schema::bytes_options::BytesOptions;\nuse crate::schema::facet_options::FacetOptions;\nuse crate::schema::{\n    DateOptions, Facet, IndexRecordOption, JsonObjectOptions, NumericOptions, OwnedValue,\n    TextFieldIndexing, TextOptions,\n};\nuse crate::time::format_description::well_known::Rfc3339;\nuse crate::time::OffsetDateTime;\nuse crate::tokenizer::PreTokenizedString;\nuse crate::DateTime;\n\n/// Possible error that may occur while parsing a field value\n/// At this point the JSON is known to be valid.\n#[derive(Debug, PartialEq, Error)]\npub enum ValueParsingError {\n    #[error(\"Overflow error. Expected {expected}, got {json}\")]\n    OverflowError {\n        expected: &'static str,\n        json: serde_json::Value,\n    },\n    #[error(\"Type error. Expected {expected}, got {json}\")]\n    TypeError {\n        expected: &'static str,\n        json: serde_json::Value,\n    },\n    #[error(\"Parse  error on {json}: {error}\")]\n    ParseError {\n        error: String,\n        json: serde_json::Value,\n    },\n    #[error(\"Invalid base64: {base64}\")]\n    InvalidBase64 { base64: String },\n}\n\n/// Type of the value that a field can take.\n///\n/// Contrary to FieldType, this does\n/// not include the way the field must be indexed.\n#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]\n#[repr(u8)]\npub enum Type {\n    /// `&str`\n    Str = b's',\n    /// `u64`\n    U64 = b'u',\n    /// `i64`\n    I64 = b'i',\n    /// `f64`\n    F64 = b'f',\n    /// `bool`\n    Bool = b'o',\n    /// `date(i64) timestamp`\n    Date = b'd',\n    /// `tantivy::schema::Facet`. Passed as a string in JSON.\n    Facet = b'h',\n    /// `Vec<u8>`\n    Bytes = b'b',\n    /// Leaf in a Json object.\n    Json = b'j',\n    /// IpAddr\n    IpAddr = b'p',\n}\n\nimpl From<ColumnType> for Type {\n    fn from(value: ColumnType) -> Self {\n        match value {\n            ColumnType::Str => Type::Str,\n            ColumnType::U64 => Type::U64,\n            ColumnType::I64 => Type::I64,\n            ColumnType::F64 => Type::F64,\n            ColumnType::Bool => Type::Bool,\n            ColumnType::DateTime => Type::Date,\n            ColumnType::Bytes => Type::Bytes,\n            ColumnType::IpAddr => Type::IpAddr,\n        }\n    }\n}\n\nconst ALL_TYPES: [Type; 10] = [\n    Type::Str,\n    Type::U64,\n    Type::I64,\n    Type::F64,\n    Type::Bool,\n    Type::Date,\n    Type::Facet,\n    Type::Bytes,\n    Type::Json,\n    Type::IpAddr,\n];\n\nimpl Type {\n    /// Returns the numerical type if applicable\n    /// It does not do any mapping, e.g. Date is None although it's also stored as I64 in the\n    /// column store\n    pub fn numerical_type(&self) -> Option<NumericalType> {\n        match self {\n            Type::I64 => Some(NumericalType::I64),\n            Type::U64 => Some(NumericalType::U64),\n            Type::F64 => Some(NumericalType::F64),\n            _ => None,\n        }\n    }\n\n    /// Returns an iterator over the different values\n    /// the Type enum can tape.\n    pub fn iter_values() -> impl Iterator<Item = Type> {\n        ALL_TYPES.iter().cloned()\n    }\n\n    /// Returns a 1 byte code used to identify the type.\n    #[inline]\n    pub fn to_code(&self) -> u8 {\n        *self as u8\n    }\n\n    /// Returns a human readable name for the Type.\n    pub fn name(&self) -> &'static str {\n        match self {\n            Type::Str => \"Str\",\n            Type::U64 => \"U64\",\n            Type::I64 => \"I64\",\n            Type::F64 => \"F64\",\n            Type::Bool => \"Bool\",\n            Type::Date => \"Date\",\n            Type::Facet => \"Facet\",\n            Type::Bytes => \"Bytes\",\n            Type::Json => \"Json\",\n            Type::IpAddr => \"IpAddr\",\n        }\n    }\n\n    /// Interprets a 1byte code as a type.\n    /// Returns `None` if the code is invalid.\n    #[inline]\n    pub fn from_code(code: u8) -> Option<Self> {\n        match code {\n            b's' => Some(Type::Str),\n            b'u' => Some(Type::U64),\n            b'i' => Some(Type::I64),\n            b'f' => Some(Type::F64),\n            b'o' => Some(Type::Bool),\n            b'd' => Some(Type::Date),\n            b'h' => Some(Type::Facet),\n            b'b' => Some(Type::Bytes),\n            b'j' => Some(Type::Json),\n            b'p' => Some(Type::IpAddr),\n            _ => None,\n        }\n    }\n}\n\n/// A `FieldType` describes the type (text, u64) of a field as well as\n/// how it should be handled by tantivy.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(tag = \"type\", content = \"options\")]\n#[serde(rename_all = \"snake_case\")]\npub enum FieldType {\n    /// String field type configuration\n    #[serde(rename = \"text\")]\n    Str(TextOptions),\n    /// Unsigned 64-bits integers field type configuration\n    U64(NumericOptions),\n    /// Signed 64-bits integers 64 field type configuration\n    I64(NumericOptions),\n    /// 64-bits float 64 field type configuration\n    F64(NumericOptions),\n    /// Bool field type configuration\n    Bool(NumericOptions),\n    /// Signed 64-bits Date 64 field type configuration,\n    Date(DateOptions),\n    /// Hierarchical Facet\n    Facet(FacetOptions),\n    /// Bytes (one per document)\n    Bytes(BytesOptions),\n    /// Json object\n    JsonObject(JsonObjectOptions),\n    /// IpAddr field\n    IpAddr(IpAddrOptions),\n}\n\nimpl FieldType {\n    /// Returns the value type associated for this field.\n    pub fn value_type(&self) -> Type {\n        match *self {\n            FieldType::Str(_) => Type::Str,\n            FieldType::U64(_) => Type::U64,\n            FieldType::I64(_) => Type::I64,\n            FieldType::F64(_) => Type::F64,\n            FieldType::Bool(_) => Type::Bool,\n            FieldType::Date(_) => Type::Date,\n            FieldType::Facet(_) => Type::Facet,\n            FieldType::Bytes(_) => Type::Bytes,\n            FieldType::JsonObject(_) => Type::Json,\n            FieldType::IpAddr(_) => Type::IpAddr,\n        }\n    }\n\n    /// returns true if this is an json field\n    pub fn is_json(&self) -> bool {\n        matches!(self, FieldType::JsonObject(_))\n    }\n\n    /// returns true if this is an ip address field\n    pub fn is_ip_addr(&self) -> bool {\n        matches!(self, FieldType::IpAddr(_))\n    }\n\n    /// returns true if this is an str field\n    pub fn is_str(&self) -> bool {\n        matches!(self, FieldType::Str(_))\n    }\n\n    /// returns true if this is a bytes field\n    pub fn is_bytes(&self) -> bool {\n        matches!(self, FieldType::Bytes(_))\n    }\n\n    /// returns true if this is an date field\n    pub fn is_date(&self) -> bool {\n        matches!(self, FieldType::Date(_))\n    }\n\n    /// returns true if the field is indexed.\n    pub fn is_indexed(&self) -> bool {\n        match *self {\n            FieldType::Str(ref text_options) => text_options.get_indexing_options().is_some(),\n            FieldType::U64(ref int_options)\n            | FieldType::I64(ref int_options)\n            | FieldType::F64(ref int_options)\n            | FieldType::Bool(ref int_options) => int_options.is_indexed(),\n            FieldType::Date(ref date_options) => date_options.is_indexed(),\n            FieldType::Facet(ref _facet_options) => true,\n            FieldType::Bytes(ref bytes_options) => bytes_options.is_indexed(),\n            FieldType::JsonObject(ref json_object_options) => json_object_options.is_indexed(),\n            FieldType::IpAddr(ref ip_addr_options) => ip_addr_options.is_indexed(),\n        }\n    }\n\n    /// Returns the index record option for the field.\n    ///\n    /// If the field is not indexed, returns `None`.\n    pub fn index_record_option(&self) -> Option<IndexRecordOption> {\n        match self {\n            FieldType::Str(text_options) => text_options\n                .get_indexing_options()\n                .map(|text_indexing| text_indexing.index_option()),\n            FieldType::JsonObject(json_object_options) => json_object_options\n                .get_text_indexing_options()\n                .map(|text_indexing| text_indexing.index_option()),\n            field_type => {\n                if field_type.is_indexed() {\n                    Some(IndexRecordOption::Basic)\n                } else {\n                    None\n                }\n            }\n        }\n    }\n\n    /// returns true if the field is fast.\n    pub fn is_fast(&self) -> bool {\n        match *self {\n            FieldType::Bytes(ref bytes_options) => bytes_options.is_fast(),\n            FieldType::Str(ref text_options) => text_options.is_fast(),\n            FieldType::U64(ref int_options)\n            | FieldType::I64(ref int_options)\n            | FieldType::F64(ref int_options)\n            | FieldType::Bool(ref int_options) => int_options.is_fast(),\n            FieldType::Date(ref date_options) => date_options.is_fast(),\n            FieldType::IpAddr(ref ip_addr_options) => ip_addr_options.is_fast(),\n            FieldType::Facet(_) => true,\n            FieldType::JsonObject(ref json_object_options) => json_object_options.is_fast(),\n        }\n    }\n\n    /// returns true if the field is normed (see [fieldnorms](crate::fieldnorm)).\n    pub fn has_fieldnorms(&self) -> bool {\n        match *self {\n            FieldType::Str(ref text_options) => text_options\n                .get_indexing_options()\n                .map(|options| options.fieldnorms())\n                .unwrap_or(false),\n            FieldType::U64(ref int_options)\n            | FieldType::I64(ref int_options)\n            | FieldType::F64(ref int_options)\n            | FieldType::Bool(ref int_options) => int_options.fieldnorms(),\n            FieldType::Date(ref date_options) => date_options.fieldnorms(),\n            FieldType::Facet(_) => false,\n            FieldType::Bytes(ref bytes_options) => bytes_options.fieldnorms(),\n            FieldType::JsonObject(ref _json_object_options) => false,\n            FieldType::IpAddr(ref ip_addr_options) => ip_addr_options.fieldnorms(),\n        }\n    }\n\n    /// Given a field configuration, return the maximal possible\n    /// `IndexRecordOption` available.\n    ///\n    /// For the Json object, this does not necessarily mean it is the index record\n    /// option level is available for all terms.\n    /// (Non string terms have the Basic indexing option at most.)\n    ///\n    /// If the field is not indexed, then returns `None`.\n    pub fn get_index_record_option(&self) -> Option<IndexRecordOption> {\n        match *self {\n            FieldType::Str(ref text_options) => text_options\n                .get_indexing_options()\n                .map(TextFieldIndexing::index_option),\n            FieldType::U64(ref int_options)\n            | FieldType::I64(ref int_options)\n            | FieldType::F64(ref int_options)\n            | FieldType::Bool(ref int_options) => {\n                if int_options.is_indexed() {\n                    Some(IndexRecordOption::Basic)\n                } else {\n                    None\n                }\n            }\n            FieldType::Date(ref date_options) => {\n                if date_options.is_indexed() {\n                    Some(IndexRecordOption::Basic)\n                } else {\n                    None\n                }\n            }\n            FieldType::Facet(ref _facet_options) => Some(IndexRecordOption::Basic),\n            FieldType::Bytes(ref bytes_options) => {\n                if bytes_options.is_indexed() {\n                    Some(IndexRecordOption::Basic)\n                } else {\n                    None\n                }\n            }\n            FieldType::JsonObject(ref json_obj_options) => json_obj_options\n                .get_text_indexing_options()\n                .map(TextFieldIndexing::index_option),\n            FieldType::IpAddr(ref ip_addr_options) => {\n                if ip_addr_options.is_indexed() {\n                    Some(IndexRecordOption::Basic)\n                } else {\n                    None\n                }\n            }\n        }\n    }\n\n    /// Parses a field value from json, given the target FieldType.\n    ///\n    /// Tantivy will try to cast values only with the coerce option.\n    /// For instance, If the json value is the integer `3` and the\n    /// target field is a `Str`, this method will return an Error if `coerce`\n    /// is not enabled.\n    pub fn value_from_json(&self, json: JsonValue) -> Result<OwnedValue, ValueParsingError> {\n        match json {\n            JsonValue::String(field_text) => {\n                match self {\n                    FieldType::Date(_) => {\n                        let dt_with_fixed_tz = OffsetDateTime::parse(&field_text, &Rfc3339)\n                            .map_err(|_err| ValueParsingError::TypeError {\n                                expected: \"rfc3339 format\",\n                                json: JsonValue::String(field_text),\n                            })?;\n                        Ok(DateTime::from_utc(dt_with_fixed_tz).into())\n                    }\n                    FieldType::Str(_) => Ok(OwnedValue::Str(field_text)),\n                    FieldType::U64(opt) => {\n                        if opt.should_coerce() {\n                            Ok(OwnedValue::U64(field_text.parse().map_err(|_| {\n                                ValueParsingError::TypeError {\n                                    expected: \"a u64 or a u64 as string\",\n                                    json: JsonValue::String(field_text),\n                                }\n                            })?))\n                        } else {\n                            Err(ValueParsingError::TypeError {\n                                expected: \"a u64\",\n                                json: JsonValue::String(field_text),\n                            })\n                        }\n                    }\n                    FieldType::I64(opt) => {\n                        if opt.should_coerce() {\n                            Ok(OwnedValue::I64(field_text.parse().map_err(|_| {\n                                ValueParsingError::TypeError {\n                                    expected: \"a i64 or a i64 as string\",\n                                    json: JsonValue::String(field_text),\n                                }\n                            })?))\n                        } else {\n                            Err(ValueParsingError::TypeError {\n                                expected: \"a i64\",\n                                json: JsonValue::String(field_text),\n                            })\n                        }\n                    }\n                    FieldType::F64(opt) => {\n                        if opt.should_coerce() {\n                            Ok(OwnedValue::F64(field_text.parse().map_err(|_| {\n                                ValueParsingError::TypeError {\n                                    expected: \"a f64 or a f64 as string\",\n                                    json: JsonValue::String(field_text),\n                                }\n                            })?))\n                        } else {\n                            Err(ValueParsingError::TypeError {\n                                expected: \"a f64\",\n                                json: JsonValue::String(field_text),\n                            })\n                        }\n                    }\n                    FieldType::Bool(opt) => {\n                        if opt.should_coerce() {\n                            Ok(OwnedValue::Bool(field_text.parse().map_err(|_| {\n                                ValueParsingError::TypeError {\n                                    expected: \"a i64 or a bool as string\",\n                                    json: JsonValue::String(field_text),\n                                }\n                            })?))\n                        } else {\n                            Err(ValueParsingError::TypeError {\n                                expected: \"a boolean\",\n                                json: JsonValue::String(field_text),\n                            })\n                        }\n                    }\n                    FieldType::Facet(_) => Ok(OwnedValue::Facet(Facet::from(&field_text))),\n                    FieldType::Bytes(_) => BASE64\n                        .decode(&field_text)\n                        .map(OwnedValue::Bytes)\n                        .map_err(|_| ValueParsingError::InvalidBase64 { base64: field_text }),\n                    FieldType::JsonObject(_) => Err(ValueParsingError::TypeError {\n                        expected: \"a json object\",\n                        json: JsonValue::String(field_text),\n                    }),\n                    FieldType::IpAddr(_) => {\n                        let ip_addr: IpAddr = IpAddr::from_str(&field_text).map_err(|err| {\n                            ValueParsingError::ParseError {\n                                error: err.to_string(),\n                                json: JsonValue::String(field_text),\n                            }\n                        })?;\n\n                        Ok(OwnedValue::IpAddr(ip_addr.into_ipv6_addr()))\n                    }\n                }\n            }\n            JsonValue::Number(field_val_num) => match self {\n                FieldType::I64(_) | FieldType::Date(_) => {\n                    if let Some(field_val_i64) = field_val_num.as_i64() {\n                        Ok(OwnedValue::I64(field_val_i64))\n                    } else {\n                        Err(ValueParsingError::OverflowError {\n                            expected: \"an i64 int\",\n                            json: JsonValue::Number(field_val_num),\n                        })\n                    }\n                }\n                FieldType::U64(_) => {\n                    if let Some(field_val_u64) = field_val_num.as_u64() {\n                        Ok(OwnedValue::U64(field_val_u64))\n                    } else {\n                        Err(ValueParsingError::OverflowError {\n                            expected: \"u64\",\n                            json: JsonValue::Number(field_val_num),\n                        })\n                    }\n                }\n                FieldType::F64(_) => {\n                    if let Some(field_val_f64) = field_val_num.as_f64() {\n                        Ok(OwnedValue::F64(field_val_f64))\n                    } else {\n                        Err(ValueParsingError::OverflowError {\n                            expected: \"a f64\",\n                            json: JsonValue::Number(field_val_num),\n                        })\n                    }\n                }\n                FieldType::Bool(_) => Err(ValueParsingError::TypeError {\n                    expected: \"a boolean\",\n                    json: JsonValue::Number(field_val_num),\n                }),\n                FieldType::Str(opt) => {\n                    if opt.should_coerce() {\n                        Ok(OwnedValue::Str(field_val_num.to_string()))\n                    } else {\n                        Err(ValueParsingError::TypeError {\n                            expected: \"a string\",\n                            json: JsonValue::Number(field_val_num),\n                        })\n                    }\n                }\n                FieldType::Facet(_) | FieldType::Bytes(_) => Err(ValueParsingError::TypeError {\n                    expected: \"a string\",\n                    json: JsonValue::Number(field_val_num),\n                }),\n                FieldType::JsonObject(_) => Err(ValueParsingError::TypeError {\n                    expected: \"a json object\",\n                    json: JsonValue::Number(field_val_num),\n                }),\n                FieldType::IpAddr(_) => Err(ValueParsingError::TypeError {\n                    expected: \"a string with an ip addr\",\n                    json: JsonValue::Number(field_val_num),\n                }),\n            },\n            JsonValue::Object(json_map) => match self {\n                FieldType::Str(_) => {\n                    if let Ok(tok_str_val) = serde_json::from_value::<PreTokenizedString>(\n                        serde_json::Value::Object(json_map.clone()),\n                    ) {\n                        Ok(OwnedValue::PreTokStr(tok_str_val))\n                    } else {\n                        Err(ValueParsingError::TypeError {\n                            expected: \"a string or an pretokenized string\",\n                            json: JsonValue::Object(json_map),\n                        })\n                    }\n                }\n                FieldType::JsonObject(_) => Ok(OwnedValue::from(json_map)),\n                _ => Err(ValueParsingError::TypeError {\n                    expected: self.value_type().name(),\n                    json: JsonValue::Object(json_map),\n                }),\n            },\n            JsonValue::Bool(json_bool_val) => match self {\n                FieldType::Bool(_) => Ok(OwnedValue::Bool(json_bool_val)),\n                FieldType::Str(opt) => {\n                    if opt.should_coerce() {\n                        Ok(OwnedValue::Str(json_bool_val.to_string()))\n                    } else {\n                        Err(ValueParsingError::TypeError {\n                            expected: \"a string\",\n                            json: JsonValue::Bool(json_bool_val),\n                        })\n                    }\n                }\n                _ => Err(ValueParsingError::TypeError {\n                    expected: self.value_type().name(),\n                    json: JsonValue::Bool(json_bool_val),\n                }),\n            },\n            // Could also just filter them\n            JsonValue::Null => match self {\n                FieldType::Str(opt) => {\n                    if opt.should_coerce() {\n                        Ok(OwnedValue::Str(\"null\".to_string()))\n                    } else {\n                        Err(ValueParsingError::TypeError {\n                            expected: \"a string\",\n                            json: JsonValue::Null,\n                        })\n                    }\n                }\n                _ => Err(ValueParsingError::TypeError {\n                    expected: self.value_type().name(),\n                    json: JsonValue::Null,\n                }),\n            },\n            _ => Err(ValueParsingError::TypeError {\n                expected: self.value_type().name(),\n                json: json.clone(),\n            }),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use serde_json::json;\n\n    use super::FieldType;\n    use crate::schema::field_type::ValueParsingError;\n    use crate::schema::{\n        Document, NumericOptions, OwnedValue, Schema, TextOptions, Type, COERCE, INDEXED,\n    };\n    use crate::time::{Date, Month, PrimitiveDateTime, Time};\n    use crate::tokenizer::{PreTokenizedString, Token};\n    use crate::{DateTime, TantivyDocument};\n\n    #[test]\n    fn test_to_string_coercion() {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"id\", COERCE);\n        let schema = schema_builder.build();\n        let doc = TantivyDocument::parse_json(&schema, r#\"{\"id\": 100}\"#).unwrap();\n        assert_eq!(\n            OwnedValue::Str(\"100\".to_string()),\n            doc.get_first(text_field).unwrap().into()\n        );\n\n        let doc = TantivyDocument::parse_json(&schema, r#\"{\"id\": true}\"#).unwrap();\n        assert_eq!(\n            OwnedValue::Str(\"true\".to_string()),\n            doc.get_first(text_field).unwrap().into()\n        );\n\n        // Not sure if this null coercion is the best approach\n        let doc = TantivyDocument::parse_json(&schema, r#\"{\"id\": null}\"#).unwrap();\n        assert_eq!(\n            OwnedValue::Str(\"null\".to_string()),\n            doc.get_first(text_field).unwrap().into()\n        );\n    }\n\n    #[test]\n    fn test_to_number_coercion() {\n        let mut schema_builder = Schema::builder();\n        let i64_field = schema_builder.add_i64_field(\"i64\", COERCE);\n        let u64_field = schema_builder.add_u64_field(\"u64\", COERCE);\n        let f64_field = schema_builder.add_f64_field(\"f64\", COERCE);\n        let schema = schema_builder.build();\n        let doc_json = r#\"{\"i64\": \"100\", \"u64\": \"100\", \"f64\": \"100\"}\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n        assert_eq!(\n            OwnedValue::I64(100),\n            doc.get_first(i64_field).unwrap().into()\n        );\n        assert_eq!(\n            OwnedValue::U64(100),\n            doc.get_first(u64_field).unwrap().into()\n        );\n        assert_eq!(\n            OwnedValue::F64(100.0),\n            doc.get_first(f64_field).unwrap().into()\n        );\n    }\n\n    #[test]\n    fn test_to_bool_coercion() {\n        let mut schema_builder = Schema::builder();\n        let bool_field = schema_builder.add_bool_field(\"bool\", COERCE);\n        let schema = schema_builder.build();\n        let doc_json = r#\"{\"bool\": \"true\"}\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n        assert_eq!(\n            OwnedValue::Bool(true),\n            doc.get_first(bool_field).unwrap().into()\n        );\n\n        let doc_json = r#\"{\"bool\": \"false\"}\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n        assert_eq!(\n            OwnedValue::Bool(false),\n            doc.get_first(bool_field).unwrap().into()\n        );\n    }\n\n    #[test]\n    fn test_to_number_no_coercion() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_i64_field(\"i64\", NumericOptions::default());\n        schema_builder.add_u64_field(\"u64\", NumericOptions::default());\n        schema_builder.add_f64_field(\"f64\", NumericOptions::default());\n        let schema = schema_builder.build();\n        assert!(TantivyDocument::parse_json(&schema, r#\"{\"u64\": \"100\"}\"#)\n            .unwrap_err()\n            .to_string()\n            .contains(\"a u64\"));\n\n        assert!(TantivyDocument::parse_json(&schema, r#\"{\"i64\": \"100\"}\"#)\n            .unwrap_err()\n            .to_string()\n            .contains(\"a i64\"));\n\n        assert!(TantivyDocument::parse_json(&schema, r#\"{\"f64\": \"100\"}\"#)\n            .unwrap_err()\n            .to_string()\n            .contains(\"a f64\"));\n    }\n\n    #[test]\n    fn test_deserialize_json_date() {\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date\", INDEXED);\n        let schema = schema_builder.build();\n        let doc_json = r#\"{\"date\": \"2019-10-12T07:20:50.52+02:00\"}\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n        let date = OwnedValue::from(doc.get_first(date_field).unwrap());\n        // Time zone is converted to UTC\n        assert_eq!(\"Date(2019-10-12T05:20:50.52Z)\", format!(\"{date:?}\"));\n    }\n\n    #[test]\n    fn test_serialize_json_date() {\n        let mut doc = TantivyDocument::new();\n        let mut schema_builder = Schema::builder();\n        let date_field = schema_builder.add_date_field(\"date\", INDEXED);\n        let schema = schema_builder.build();\n        let naive_date = Date::from_calendar_date(1982, Month::September, 17).unwrap();\n        let naive_time = Time::from_hms(13, 20, 0).unwrap();\n        let date_time = PrimitiveDateTime::new(naive_date, naive_time);\n        doc.add_date(date_field, DateTime::from_primitive(date_time));\n        let doc_json = doc.to_json(&schema);\n        assert_eq!(doc_json, r#\"{\"date\":[\"1982-09-17T13:20:00Z\"]}\"#);\n    }\n\n    #[test]\n    fn test_bytes_value_from_json() {\n        let result = FieldType::Bytes(Default::default())\n            .value_from_json(json!(\"dGhpcyBpcyBhIHRlc3Q=\"))\n            .unwrap();\n        assert_eq!(\n            result,\n            OwnedValue::Bytes(\"this is a test\".as_bytes().to_vec())\n        );\n\n        let result = FieldType::Bytes(Default::default()).value_from_json(json!(521));\n        match result {\n            Err(ValueParsingError::TypeError { .. }) => {}\n            _ => panic!(\"Expected parse failure for wrong type\"),\n        }\n\n        let result = FieldType::Bytes(Default::default()).value_from_json(json!(\"-\"));\n        match result {\n            Err(ValueParsingError::InvalidBase64 { .. }) => {}\n            _ => panic!(\"Expected parse failure for invalid base64\"),\n        }\n    }\n\n    #[test]\n    fn test_pre_tok_str_value_from_json() {\n        let pre_tokenized_string_json = r#\"{\n  \"text\": \"The Old Man\",\n  \"tokens\": [\n    {\n      \"offset_from\": 0,\n      \"offset_to\": 3,\n      \"position\": 0,\n      \"text\": \"The\",\n      \"position_length\": 1\n    },\n    {\n      \"offset_from\": 4,\n      \"offset_to\": 7,\n      \"position\": 1,\n      \"text\": \"Old\",\n      \"position_length\": 1\n    },\n    {\n      \"offset_from\": 8,\n      \"offset_to\": 11,\n      \"position\": 2,\n      \"text\": \"Man\",\n      \"position_length\": 1\n    }\n  ]\n}\"#;\n\n        let expected_value = OwnedValue::PreTokStr(PreTokenizedString {\n            text: String::from(\"The Old Man\"),\n            tokens: vec![\n                Token {\n                    offset_from: 0,\n                    offset_to: 3,\n                    position: 0,\n                    text: String::from(\"The\"),\n                    position_length: 1,\n                },\n                Token {\n                    offset_from: 4,\n                    offset_to: 7,\n                    position: 1,\n                    text: String::from(\"Old\"),\n                    position_length: 1,\n                },\n                Token {\n                    offset_from: 8,\n                    offset_to: 11,\n                    position: 2,\n                    text: String::from(\"Man\"),\n                    position_length: 1,\n                },\n            ],\n        });\n\n        let deserialized_value = FieldType::Str(TextOptions::default())\n            .value_from_json(serde_json::from_str(pre_tokenized_string_json).unwrap())\n            .unwrap();\n\n        assert_eq!(deserialized_value, expected_value);\n\n        let serialized_value_json = serde_json::to_string_pretty(&expected_value).unwrap();\n\n        assert_eq!(serialized_value_json, pre_tokenized_string_json);\n    }\n\n    #[test]\n    fn test_type_codes() {\n        for type_val in Type::iter_values() {\n            let code = type_val.to_code();\n            assert_eq!(Type::from_code(code), Some(type_val));\n        }\n        assert_eq!(Type::from_code(b'z'), None);\n    }\n}\n"
  },
  {
    "path": "src/schema/flags.rs",
    "content": "use std::ops::BitOr;\n\nuse crate::schema::{DateOptions, NumericOptions, TextOptions};\n\n#[derive(Clone)]\npub struct StoredFlag;\n/// Flag to mark the field as stored.\n/// This flag can apply to any kind of field.\n///\n/// A stored fields of a document can be retrieved given its `DocId`.\n/// Stored field are stored together and compressed.\n/// Reading the stored fields of a document is relatively slow.\n/// (~ 100 microsecs)\n///\n/// It should not be used during scoring or collection.\npub const STORED: SchemaFlagList<StoredFlag, ()> = SchemaFlagList {\n    head: StoredFlag,\n    tail: (),\n};\n\n#[derive(Clone)]\npub struct IndexedFlag;\n/// Flag to mark the field as indexed. An indexed field is searchable and has a fieldnorm.\n///\n/// The `INDEXED` flag can only be used when building `NumericOptions` (`u64`, `i64`, `f64` and\n/// `bool` fields) Of course, text fields can also be indexed... But this is expressed by using\n/// either the `STRING` (untokenized) or `TEXT` (tokenized with the english tokenizer) flags.\npub const INDEXED: SchemaFlagList<IndexedFlag, ()> = SchemaFlagList {\n    head: IndexedFlag,\n    tail: (),\n};\n\n#[derive(Clone)]\npub struct CoerceFlag;\n/// Flag to mark the field as coerced.\n///\n/// `COERCE` will try to convert values into its value type if they don't match.\n///\n/// See [fast fields](`crate::fastfield`).\npub const COERCE: SchemaFlagList<CoerceFlag, ()> = SchemaFlagList {\n    head: CoerceFlag,\n    tail: (),\n};\n\n#[derive(Clone)]\npub struct FastFlag;\n/// Flag to mark the field as a fast field (similar to Lucene's DocValues)\n///\n/// Fast fields can be random-accessed rapidly. Fields useful for scoring, filtering\n/// or collection should be mark as fast fields.\n///\n/// See [fast fields](`crate::fastfield`).\npub const FAST: SchemaFlagList<FastFlag, ()> = SchemaFlagList {\n    head: FastFlag,\n    tail: (),\n};\n\nimpl<Head, OldHead, OldTail> BitOr<SchemaFlagList<Head, ()>> for SchemaFlagList<OldHead, OldTail>\nwhere\n    Head: Clone,\n    OldHead: Clone,\n    OldTail: Clone,\n{\n    type Output = SchemaFlagList<Head, SchemaFlagList<OldHead, OldTail>>;\n\n    fn bitor(self, head: SchemaFlagList<Head, ()>) -> Self::Output {\n        SchemaFlagList {\n            head: head.head,\n            tail: self,\n        }\n    }\n}\n\nimpl<T: Clone + Into<NumericOptions>> BitOr<NumericOptions> for SchemaFlagList<T, ()> {\n    type Output = NumericOptions;\n\n    fn bitor(self, rhs: NumericOptions) -> Self::Output {\n        self.head.into() | rhs\n    }\n}\n\nimpl<T: Clone + Into<DateOptions>> BitOr<DateOptions> for SchemaFlagList<T, ()> {\n    type Output = DateOptions;\n\n    fn bitor(self, rhs: DateOptions) -> Self::Output {\n        self.head.into() | rhs\n    }\n}\n\nimpl<T: Clone + Into<TextOptions>> BitOr<TextOptions> for SchemaFlagList<T, ()> {\n    type Output = TextOptions;\n\n    fn bitor(self, rhs: TextOptions) -> Self::Output {\n        self.head.into() | rhs\n    }\n}\n\n#[derive(Clone)]\npub struct SchemaFlagList<Head: Clone, Tail: Clone> {\n    pub head: Head,\n    pub tail: Tail,\n}\n"
  },
  {
    "path": "src/schema/index_record_option.rs",
    "content": "use serde::{Deserialize, Serialize};\n\n/// `IndexRecordOption` describes an amount information associated\n/// with a given indexed field.\n///\n/// It is both used to:\n///\n///  * describe in the schema the amount of information that should be retained during indexing (See\n///    [`TextFieldIndexing::set_index_option()`](crate::schema::TextFieldIndexing::set_index_option))\n///  * request that a given amount of information to be decoded as one goes through a posting list.\n///    (See [`InvertedIndexReader::read_postings()`](crate::InvertedIndexReader::read_postings))\n#[derive(\n    Clone, Copy, Debug, PartialEq, PartialOrd, Ord, Eq, Hash, Serialize, Deserialize, Default,\n)]\npub enum IndexRecordOption {\n    /// records only the `DocId`s\n    #[serde(rename = \"basic\")]\n    #[default]\n    Basic,\n    /// records the document ids as well as the term frequency.\n    /// The term frequency can help giving better scoring of the documents.\n    #[serde(rename = \"freq\")]\n    WithFreqs,\n    /// records the document id, the term frequency and the positions of\n    /// the occurrences in the document.\n    /// Positions are required to run a [`PhraseQuery`](crate::query::PhraseQuery).\n    #[serde(rename = \"position\")]\n    WithFreqsAndPositions,\n}\n\nimpl IndexRecordOption {\n    /// Returns true if this option includes encoding\n    /// term frequencies.\n    pub fn has_freq(self) -> bool {\n        match self {\n            IndexRecordOption::Basic => false,\n            IndexRecordOption::WithFreqs | IndexRecordOption::WithFreqsAndPositions => true,\n        }\n    }\n\n    /// Returns true if this option include encoding\n    ///  term positions.\n    pub fn has_positions(self) -> bool {\n        match self {\n            IndexRecordOption::Basic | IndexRecordOption::WithFreqs => false,\n            IndexRecordOption::WithFreqsAndPositions => true,\n        }\n    }\n\n    /// Downgrades to the next level if provided `IndexRecordOption` is unavailable.\n    pub fn downgrade(&self, other: IndexRecordOption) -> IndexRecordOption {\n        use IndexRecordOption::*;\n\n        match (other, self) {\n            (WithFreqsAndPositions, WithFreqsAndPositions) => WithFreqsAndPositions,\n            (WithFreqs, WithFreqs) => WithFreqs,\n            (WithFreqsAndPositions, WithFreqs) => WithFreqs,\n            (WithFreqs, WithFreqsAndPositions) => WithFreqs,\n            _ => Basic,\n        }\n    }\n}\n"
  },
  {
    "path": "src/schema/ip_options.rs",
    "content": "use std::net::{IpAddr, Ipv6Addr};\nuse std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::flags::{FastFlag, IndexedFlag, SchemaFlagList, StoredFlag};\n\n/// Trait to convert into an Ipv6Addr.\npub trait IntoIpv6Addr {\n    /// Consumes the object and returns an Ipv6Addr.\n    fn into_ipv6_addr(self) -> Ipv6Addr;\n}\n\nimpl IntoIpv6Addr for IpAddr {\n    fn into_ipv6_addr(self) -> Ipv6Addr {\n        match self {\n            IpAddr::V4(addr) => addr.to_ipv6_mapped(),\n            IpAddr::V6(addr) => addr,\n        }\n    }\n}\n\n/// Define how an ip field should be handled by tantivy.\n#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)]\npub struct IpAddrOptions {\n    fast: bool,\n    stored: bool,\n    indexed: bool,\n    fieldnorms: bool,\n}\n\nimpl IpAddrOptions {\n    /// Returns true iff the value is a fast field.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        self.fast\n    }\n\n    /// Returns `true` if the ip address should be stored in the doc store.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Returns true iff the value is indexed and therefore searchable.\n    #[inline]\n    pub fn is_indexed(&self) -> bool {\n        self.indexed\n    }\n\n    /// Returns true if and only if the value is normed.\n    #[inline]\n    pub fn fieldnorms(&self) -> bool {\n        self.fieldnorms\n    }\n\n    /// Set the field as normed.\n    ///\n    /// Setting an integer as normed will generate\n    /// the fieldnorm data for it.\n    #[must_use]\n    pub fn set_fieldnorms(mut self) -> Self {\n        self.fieldnorms = true;\n        self\n    }\n\n    /// Sets the field as stored\n    #[must_use]\n    pub fn set_stored(mut self) -> Self {\n        self.stored = true;\n        self\n    }\n\n    /// Set the field as indexed.\n    ///\n    /// Setting an ip address as indexed will generate\n    /// a posting list for each value taken by the ip address.\n    /// Ips are normalized to IpV6.\n    ///\n    /// This is required for the field to be searchable.\n    #[must_use]\n    pub fn set_indexed(mut self) -> Self {\n        self.indexed = true;\n        self\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    #[must_use]\n    pub fn set_fast(mut self) -> Self {\n        self.fast = true;\n        self\n    }\n}\n\nimpl From<()> for IpAddrOptions {\n    fn from(_: ()) -> IpAddrOptions {\n        IpAddrOptions::default()\n    }\n}\n\nimpl From<FastFlag> for IpAddrOptions {\n    fn from(_: FastFlag) -> Self {\n        IpAddrOptions {\n            fieldnorms: false,\n            indexed: false,\n            stored: false,\n            fast: true,\n        }\n    }\n}\n\nimpl From<StoredFlag> for IpAddrOptions {\n    fn from(_: StoredFlag) -> Self {\n        IpAddrOptions {\n            fieldnorms: false,\n            indexed: false,\n            stored: true,\n            fast: false,\n        }\n    }\n}\n\nimpl From<IndexedFlag> for IpAddrOptions {\n    fn from(_: IndexedFlag) -> Self {\n        IpAddrOptions {\n            fieldnorms: true,\n            indexed: true,\n            stored: false,\n            fast: false,\n        }\n    }\n}\n\nimpl<T: Into<IpAddrOptions>> BitOr<T> for IpAddrOptions {\n    type Output = IpAddrOptions;\n\n    fn bitor(self, other: T) -> IpAddrOptions {\n        let other = other.into();\n        IpAddrOptions {\n            fieldnorms: self.fieldnorms | other.fieldnorms,\n            indexed: self.indexed | other.indexed,\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for IpAddrOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n"
  },
  {
    "path": "src/schema/json_object_options.rs",
    "content": "use std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::text_options::{FastFieldTextOptions, TokenizerName};\nuse crate::schema::flags::{FastFlag, SchemaFlagList, StoredFlag};\nuse crate::schema::{TextFieldIndexing, TextOptions};\n\n/// The `JsonObjectOptions` make it possible to\n/// configure how a json object field should be indexed and stored.\n#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]\npub struct JsonObjectOptions {\n    stored: bool,\n    // If set to some, int, date, f64 and text will be indexed.\n    // Text will use the TextFieldIndexing setting for indexing.\n    indexing: Option<TextFieldIndexing>,\n    // Store all field as fast fields with an optional tokenizer for text.\n    fast: FastFieldTextOptions,\n    /// tantivy will generate paths to the different nodes of the json object\n    /// both in:\n    /// - the inverted index (for the terms)\n    /// - fast fields (for the column names).\n    ///\n    /// These json path are encoded by concatenating the list of object keys that\n    /// are visited from the root to the leaf.\n    ///\n    /// By default, if an object key contains a `.`, we keep it as a `.` it as is.\n    /// On the search side, users will then have to escape this `.` in the query parser\n    /// or when referring to a column name.\n    ///\n    /// For instance:\n    /// `{\"root\": {\"child.with.dot\": \"hello\"}}`\n    ///\n    /// Can be searched using the following query\n    /// `root.child\\.with\\.dot:hello`\n    ///\n    /// If `expand_dots_enabled` is set to true, we will treat this `.` in object keys\n    /// as json separators. In other words, if set to true, our object will be\n    /// processed as if it was\n    /// `{\"root\": {\"child\": {\"with\": {\"dot\": \"hello\"}}}}`\n    /// and it can be search using the following query:\n    /// `root.child.with.dot:hello`\n    #[serde(default)]\n    expand_dots_enabled: bool,\n}\n\nimpl JsonObjectOptions {\n    /// Returns `true` if the json object should be stored.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Returns `true` iff the json object should be indexed.\n    #[inline]\n    pub fn is_indexed(&self) -> bool {\n        self.indexing.is_some()\n    }\n\n    /// Returns true if and only if the json object fields are\n    /// to be treated as fast fields.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        matches!(self.fast, FastFieldTextOptions::IsEnabled(true))\n            || matches!(\n                &self.fast,\n                FastFieldTextOptions::EnabledWithTokenizer { with_tokenizer: _ }\n            )\n    }\n\n    /// Returns true if and only if the value is a fast field.\n    #[inline]\n    pub fn get_fast_field_tokenizer_name(&self) -> Option<&str> {\n        match &self.fast {\n            FastFieldTextOptions::IsEnabled(true) | FastFieldTextOptions::IsEnabled(false) => None,\n            FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: tokenizer,\n            } => Some(tokenizer.name()),\n        }\n    }\n\n    /// Returns `true` iff dots in json keys should be expanded.\n    ///\n    /// When expand_dots is enabled, json object like\n    /// `{\"k8s.node.id\": 5}` is processed as if it was\n    /// `{\"k8s\": {\"node\": {\"id\": 5}}}`.\n    /// This option has the merit of allowing users to\n    /// write queries  like `k8s.node.id:5`.\n    /// On the other, enabling that feature can lead to\n    /// ambiguity.\n    ///\n    /// If disabled, the \".\" needs to be escaped:\n    /// `k8s\\.node\\.id:5`.\n    #[inline]\n    pub fn is_expand_dots_enabled(&self) -> bool {\n        self.expand_dots_enabled\n    }\n\n    /// Sets `expands_dots` to true.\n    /// See `is_expand_dots_enabled` for more information.\n    pub fn set_expand_dots_enabled(mut self) -> Self {\n        self.expand_dots_enabled = true;\n        self\n    }\n\n    /// Returns the text indexing options.\n    ///\n    /// If set to `Some` then both int and str values will be indexed.\n    /// The inner `TextFieldIndexing` will however, only apply to the str values\n    /// in the json object.\n    #[inline]\n    pub fn get_text_indexing_options(&self) -> Option<&TextFieldIndexing> {\n        self.indexing.as_ref()\n    }\n\n    /// Sets the field as stored\n    #[must_use]\n    pub fn set_stored(mut self) -> Self {\n        self.stored = true;\n        self\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    /// Access time are similar to a random lookup in an array.\n    /// Text fast fields will have the term ids stored in the fast field.\n    ///\n    /// The effective cardinality depends on the tokenizer. Without a tokenizer, the text will be\n    /// stored as is, which equals to the \"raw\" tokenizer. The tokenizer can be used to apply\n    /// normalization like lower case.\n    /// The passed tokenizer_name must be available on the fast field tokenizer manager.\n    /// `Index::fast_field_tokenizer`.\n    ///\n    /// The original text can be retrieved via\n    /// [`TermDictionary::ord_to_term()`](crate::termdict::TermDictionary::ord_to_term)\n    /// from the dictionary.\n    #[must_use]\n    pub fn set_fast(mut self, tokenizer_name: Option<&str>) -> Self {\n        if let Some(tokenizer) = tokenizer_name {\n            let tokenizer = TokenizerName::from_name(tokenizer);\n            self.fast = FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: tokenizer,\n            }\n        } else {\n            self.fast = FastFieldTextOptions::IsEnabled(true);\n        }\n        self\n    }\n\n    /// Sets the field as indexed, with the specific indexing options.\n    #[must_use]\n    pub fn set_indexing_options(mut self, indexing: TextFieldIndexing) -> Self {\n        self.indexing = Some(indexing);\n        self\n    }\n}\n\nimpl From<StoredFlag> for JsonObjectOptions {\n    fn from(_stored_flag: StoredFlag) -> Self {\n        JsonObjectOptions {\n            stored: true,\n            indexing: None,\n            fast: FastFieldTextOptions::default(),\n            expand_dots_enabled: false,\n        }\n    }\n}\n\nimpl From<FastFlag> for JsonObjectOptions {\n    fn from(_fast_flag: FastFlag) -> Self {\n        JsonObjectOptions {\n            stored: false,\n            indexing: None,\n            fast: FastFieldTextOptions::IsEnabled(true),\n            expand_dots_enabled: false,\n        }\n    }\n}\n\nimpl From<()> for JsonObjectOptions {\n    fn from(_: ()) -> Self {\n        Self::default()\n    }\n}\n\nimpl<T: Into<JsonObjectOptions>> BitOr<T> for JsonObjectOptions {\n    type Output = JsonObjectOptions;\n\n    fn bitor(self, other: T) -> Self {\n        let other: JsonObjectOptions = other.into();\n        JsonObjectOptions {\n            indexing: self.indexing.or(other.indexing),\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n            expand_dots_enabled: self.expand_dots_enabled | other.expand_dots_enabled,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for JsonObjectOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\nimpl From<TextOptions> for JsonObjectOptions {\n    fn from(text_options: TextOptions) -> Self {\n        JsonObjectOptions {\n            stored: text_options.is_stored(),\n            indexing: text_options.get_indexing_options().cloned(),\n            fast: text_options.fast,\n            expand_dots_enabled: false,\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::schema::{FAST, STORED, TEXT};\n\n    #[test]\n    fn test_json_options() {\n        {\n            let json_options: JsonObjectOptions = (STORED | TEXT).into();\n            assert!(json_options.is_stored());\n            assert!(json_options.is_indexed());\n            assert!(!json_options.is_fast());\n        }\n        {\n            let json_options: JsonObjectOptions = TEXT.into();\n            assert!(!json_options.is_stored());\n            assert!(json_options.is_indexed());\n            assert!(!json_options.is_fast());\n        }\n        {\n            let json_options: JsonObjectOptions = STORED.into();\n            assert!(json_options.is_stored());\n            assert!(!json_options.is_indexed());\n            assert!(!json_options.is_fast());\n        }\n        {\n            let json_options: JsonObjectOptions = FAST.into();\n            assert!(!json_options.is_stored());\n            assert!(!json_options.is_indexed());\n            assert!(json_options.is_fast());\n        }\n        {\n            let json_options: JsonObjectOptions = (FAST | STORED).into();\n            assert!(json_options.is_stored());\n            assert!(!json_options.is_indexed());\n            assert!(json_options.is_fast());\n        }\n    }\n}\n"
  },
  {
    "path": "src/schema/mod.rs",
    "content": "//! Schema definition for tantivy's indices.\n//!\n//! # Setting your schema in Tantivy\n//!\n//! Tantivy has a very strict schema.\n//! The schema defines information about the fields your index contains, that is, for each field:\n//!\n//! - the field name (may contain any character, can't start with a `-` and can't be empty. Some\n//!   characters may require escaping when using the query parser).\n//! - the type of the field (currently `text`, `u64`, `i64`, `f64`, `bool`, `date`, `IpAddr`,\n//!   facets, bytes and json are supported)\n//! - how the field should be indexed / stored.\n//!\n//! This very last point is critical as it will enable / disable some of the functionality\n//! for your index.\n//!\n//! Tantivy's schema is stored within the `meta.json` file at the root of your\n//! directory.\n//!\n//!\n//!\n//! # Building a schema \"programmatically\"\n//!\n//!\n//! ## Setting a text field\n//!\n//! ### Example\n//!\n//! ```\n//! use tantivy::schema::*;\n//! let mut schema_builder = Schema::builder();\n//! let title_options = TextOptions::default()\n//!     .set_stored()\n//!     .set_indexing_options(TextFieldIndexing::default()\n//!     .set_tokenizer(\"default\")\n//!     .set_index_option(IndexRecordOption::WithFreqsAndPositions));\n//! schema_builder.add_text_field(\"title\", title_options);\n//! let schema = schema_builder.build();\n//! ```\n//!\n//! We can split the problem of generating a search result page into two phases:\n//!\n//! - identifying the list of 10 or so documents to be displayed (Conceptually `query -> doc_ids[]`)\n//! - for each of these documents, retrieving the information required to generate the search\n//!   results page. (`doc_ids[] -> Document[]`)\n//!\n//! In the first phase, the ability to search for documents by the given field is determined by the\n//! [`IndexRecordOption`] of our [`TextOptions`].\n//!\n//! The effect of each possible setting is described more in detail in [`TextOptions`].\n//!\n//! On the other hand setting the field as stored or not determines whether the field should be\n//! returned when [`Searcher::doc()`](crate::Searcher::doc) is called.\n//!\n//!\n//! ## Setting a u64, a i64 or a f64 field\n//!\n//! ### Example\n//!\n//! ```\n//! use tantivy::schema::*;\n//! let mut schema_builder = Schema::builder();\n//! let num_stars_options = NumericOptions::default()\n//!     .set_stored()\n//!     .set_indexed();\n//! schema_builder.add_u64_field(\"num_stars\", num_stars_options);\n//! let schema = schema_builder.build();\n//! ```\n//!\n//! Just like for Text fields (see above),\n//! setting the field as stored defines whether the field will be\n//! returned when [`Searcher::doc()`](crate::Searcher::doc) is called,\n//! and setting the field as indexed means that we will be able perform queries such as\n//! `num_stars:10`. Note that unlike text fields, numeric fields can only be indexed in one way for\n//! the moment.\n//!\n//! ### Shortcuts\n//!\n//!\n//! For convenience, it is possible to define your field indexing options by combining different\n//! flags using the  `|` operator.\n//!\n//! For instance, a schema containing the two fields defined in the example above could be\n//! rewritten:\n//!\n//! ```\n//! use tantivy::schema::*;\n//! let mut schema_builder = Schema::builder();\n//! schema_builder.add_u64_field(\"num_stars\", INDEXED | STORED);\n//! schema_builder.add_text_field(\"title\", TEXT | STORED);\n//! let schema = schema_builder.build();\n//! ```\n//!\n//! ### Fast fields\n//! This functionality is somewhat similar to Lucene's `DocValues`.\n//!\n//! Fields that are indexed as [`FAST`] will be stored in a special data structure that will\n//! make it possible to access the value given the doc id rapidly. This is useful if the value\n//! of the field is required during scoring or collection for instance.\n//!\n//! Some queries may leverage Fast fields when run on a field that is not indexed. This can be\n//! handy if that kind of request is infrequent, however note that searching on a Fast field is\n//! generally much slower than searching in an index.\n//!\n//! ```\n//! use tantivy::schema::*;\n//! let mut schema_builder = Schema::builder();\n//! schema_builder.add_u64_field(\"population\", STORED | FAST);\n//! schema_builder.add_text_field(\"zip_code\", STRING | FAST);\n//! let schema = schema_builder.build();\n//! ```\n\npub mod document;\nmod facet;\nmod facet_options;\nmod schema;\npub(crate) mod term;\n\nmod field_entry;\nmod field_type;\n\nmod bytes_options;\nmod date_time_options;\nmod field;\nmod flags;\nmod index_record_option;\nmod ip_options;\nmod json_object_options;\nmod named_field_document;\nmod numeric_options;\nmod text_options;\n\nuse columnar::ColumnType;\n\npub use self::bytes_options::BytesOptions;\npub use self::date_time_options::{DateOptions, DateTimePrecision, DATE_TIME_PRECISION_INDEXED};\npub use self::document::{DocParsingError, Document, OwnedValue, TantivyDocument, Value};\npub(crate) use self::facet::FACET_SEP_BYTE;\npub use self::facet::{Facet, FacetParseError};\npub use self::facet_options::FacetOptions;\npub use self::field::Field;\npub use self::field_entry::FieldEntry;\npub use self::field_type::{FieldType, Type};\npub use self::flags::{COERCE, FAST, INDEXED, STORED};\npub use self::index_record_option::IndexRecordOption;\npub use self::ip_options::{IntoIpv6Addr, IpAddrOptions};\npub use self::json_object_options::JsonObjectOptions;\npub use self::named_field_document::NamedFieldDocument;\npub use self::numeric_options::NumericOptions;\npub use self::schema::{Schema, SchemaBuilder};\npub use self::term::{Term, ValueBytes};\npub use self::text_options::{TextFieldIndexing, TextOptions, STRING, TEXT};\n\n/// Validator for a potential `field_name`.\n/// Returns true if the name can be use for a field name.\n///\n/// A field name can be any character, must have at least one character\n/// and must not start with a `-`.\npub fn is_valid_field_name(field_name: &str) -> bool {\n    !field_name.is_empty() && !field_name.starts_with('-')\n}\n\npub(crate) fn value_type_to_column_type(typ: Type) -> Option<ColumnType> {\n    match typ {\n        Type::Str => Some(ColumnType::Str),\n        Type::U64 => Some(ColumnType::U64),\n        Type::I64 => Some(ColumnType::I64),\n        Type::F64 => Some(ColumnType::F64),\n        Type::Bool => Some(ColumnType::Bool),\n        Type::Date => Some(ColumnType::DateTime),\n        Type::Facet => Some(ColumnType::Str),\n        Type::Bytes => Some(ColumnType::Bytes),\n        Type::IpAddr => Some(ColumnType::IpAddr),\n        Type::Json => None,\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::is_valid_field_name;\n\n    #[test]\n    fn test_is_valid_name() {\n        assert!(is_valid_field_name(\"シャボン玉\"));\n        assert!(!is_valid_field_name(\"-fieldname\"));\n        assert!(!is_valid_field_name(\"\"));\n    }\n}\n"
  },
  {
    "path": "src/schema/named_field_document.rs",
    "content": "use std::collections::BTreeMap;\n\nuse serde::{Deserialize, Serialize};\n\nuse crate::schema::OwnedValue;\n\n/// Internal representation of a document used for JSON\n/// serialization.\n///\n/// A `NamedFieldDocument` is a simple representation of a document\n/// as a `BTreeMap<String, Vec<Value>>`.\n#[derive(Debug, Deserialize, Serialize)]\npub struct NamedFieldDocument(pub BTreeMap<String, Vec<OwnedValue>>);\n"
  },
  {
    "path": "src/schema/numeric_options.rs",
    "content": "use std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::flags::CoerceFlag;\nuse crate::schema::flags::{FastFlag, IndexedFlag, SchemaFlagList, StoredFlag};\n\n/// Define how an `u64`, `i64`, or `f64` field should be handled by tantivy.\n#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Default)]\n#[serde(from = \"NumericOptionsDeser\")]\npub struct NumericOptions {\n    indexed: bool,\n    // This boolean has no effect if the field is not marked as indexed too.\n    fieldnorms: bool, // This attribute only has an effect if indexed is true.\n    fast: bool,\n    stored: bool,\n    #[serde(skip_serializing_if = \"is_false\")]\n    coerce: bool,\n}\n\nfn is_false(val: &bool) -> bool {\n    !val\n}\n\n/// For backward compatibility we add an intermediary to interpret the\n/// lack of fieldnorms attribute as \"true\" if and only if indexed.\n///\n/// (Downstream, for the moment, this attribute is not used anyway if not indexed...)\n/// Note that: newly serialized `NumericOptions` will include the new attribute.\n#[derive(Deserialize)]\nstruct NumericOptionsDeser {\n    indexed: bool,\n    #[serde(default)]\n    fieldnorms: Option<bool>, // This attribute only has an effect if indexed is true.\n    #[serde(default)]\n    fast: bool,\n    stored: bool,\n    #[serde(default)]\n    coerce: bool,\n}\n\nimpl From<NumericOptionsDeser> for NumericOptions {\n    fn from(deser: NumericOptionsDeser) -> Self {\n        NumericOptions {\n            indexed: deser.indexed,\n            fieldnorms: deser.fieldnorms.unwrap_or(deser.indexed),\n            fast: deser.fast,\n            stored: deser.stored,\n            coerce: deser.coerce,\n        }\n    }\n}\n\nimpl NumericOptions {\n    /// Returns true iff the value is stored in the doc store.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Returns true iff the value is indexed and therefore searchable.\n    #[inline]\n    pub fn is_indexed(&self) -> bool {\n        self.indexed\n    }\n\n    /// Returns true iff the field has fieldnorm.\n    #[inline]\n    pub fn fieldnorms(&self) -> bool {\n        self.fieldnorms && self.indexed\n    }\n\n    /// Returns true iff the value is a fast field.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        self.fast\n    }\n\n    /// Returns true if values should be coerced to numbers.\n    #[inline]\n    pub fn should_coerce(&self) -> bool {\n        self.coerce\n    }\n\n    /// Try to coerce values if they are not a number. Defaults to false.\n    #[must_use]\n    pub fn set_coerce(mut self) -> Self {\n        self.coerce = true;\n        self\n    }\n\n    /// Set the field as stored.\n    ///\n    /// Only the fields that are set as *stored* are\n    /// persisted into the Tantivy's store.\n    #[must_use]\n    pub fn set_stored(mut self) -> NumericOptions {\n        self.stored = true;\n        self\n    }\n\n    /// Set the field as indexed.\n    ///\n    /// Setting an integer as indexed will generate\n    /// a posting list for each value taken by the integer.\n    ///\n    /// This is required for the field to be searchable.\n    #[must_use]\n    pub fn set_indexed(mut self) -> NumericOptions {\n        self.indexed = true;\n        self\n    }\n\n    /// Set the field with fieldnorm.\n    ///\n    /// Setting an integer as fieldnorm will generate\n    /// the fieldnorm data for it.\n    #[must_use]\n    pub fn set_fieldnorm(mut self) -> NumericOptions {\n        self.fieldnorms = true;\n        self\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    #[must_use]\n    pub fn set_fast(mut self) -> NumericOptions {\n        self.fast = true;\n        self\n    }\n}\n\nimpl From<()> for NumericOptions {\n    fn from(_: ()) -> NumericOptions {\n        NumericOptions::default()\n    }\n}\n\nimpl From<CoerceFlag> for NumericOptions {\n    fn from(_: CoerceFlag) -> NumericOptions {\n        NumericOptions {\n            indexed: false,\n            fieldnorms: false,\n            stored: false,\n            fast: false,\n            coerce: true,\n        }\n    }\n}\n\nimpl From<FastFlag> for NumericOptions {\n    fn from(_: FastFlag) -> Self {\n        NumericOptions {\n            indexed: false,\n            fieldnorms: false,\n            stored: false,\n            fast: true,\n            coerce: false,\n        }\n    }\n}\n\nimpl From<StoredFlag> for NumericOptions {\n    fn from(_: StoredFlag) -> Self {\n        NumericOptions {\n            indexed: false,\n            fieldnorms: false,\n            stored: true,\n            fast: false,\n            coerce: false,\n        }\n    }\n}\n\nimpl From<IndexedFlag> for NumericOptions {\n    fn from(_: IndexedFlag) -> Self {\n        NumericOptions {\n            indexed: true,\n            fieldnorms: true,\n            stored: false,\n            fast: false,\n            coerce: false,\n        }\n    }\n}\n\nimpl<T: Into<NumericOptions>> BitOr<T> for NumericOptions {\n    type Output = NumericOptions;\n\n    fn bitor(self, other: T) -> NumericOptions {\n        let other = other.into();\n        NumericOptions {\n            indexed: self.indexed | other.indexed,\n            fieldnorms: self.fieldnorms | other.fieldnorms,\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n            coerce: self.coerce | other.coerce,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for NumericOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_int_options_deser_if_fieldnorm_missing_indexed_true() {\n        let json = r#\"{\n            \"indexed\": true,\n            \"stored\": false\n        }\"#;\n        let int_options: NumericOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &int_options,\n            &NumericOptions {\n                indexed: true,\n                fieldnorms: true,\n                fast: false,\n                stored: false,\n                coerce: false,\n            }\n        );\n    }\n\n    #[test]\n    fn test_int_options_deser_if_fieldnorm_missing_indexed_false() {\n        let json = r#\"{\n            \"indexed\": false,\n            \"stored\": false\n        }\"#;\n        let int_options: NumericOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &int_options,\n            &NumericOptions {\n                indexed: false,\n                fieldnorms: false,\n                fast: false,\n                stored: false,\n                coerce: false,\n            }\n        );\n    }\n\n    #[test]\n    fn test_int_options_deser_if_fieldnorm_false_indexed_true() {\n        let json = r#\"{\n            \"indexed\": true,\n            \"fieldnorms\": false,\n            \"stored\": false\n        }\"#;\n        let int_options: NumericOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &int_options,\n            &NumericOptions {\n                indexed: true,\n                fieldnorms: false,\n                fast: false,\n                stored: false,\n                coerce: false,\n            }\n        );\n    }\n\n    #[test]\n    fn test_int_options_deser_if_fieldnorm_true_indexed_false() {\n        // this one is kind of useless, at least at the moment\n        let json = r#\"{\n            \"indexed\": false,\n            \"fieldnorms\": true,\n            \"stored\": false\n        }\"#;\n        let int_options: NumericOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &int_options,\n            &NumericOptions {\n                indexed: false,\n                fieldnorms: true,\n                fast: false,\n                stored: false,\n                coerce: false,\n            }\n        );\n    }\n\n    #[test]\n    fn test_int_options_deser_if_coerce_true() {\n        // this one is kind of useless, at least at the moment\n        let json = r#\"{\n            \"indexed\": false,\n            \"fieldnorms\": true,\n            \"stored\": false,\n            \"coerce\": true\n        }\"#;\n        let int_options: NumericOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            &int_options,\n            &NumericOptions {\n                indexed: false,\n                fieldnorms: true,\n                fast: false,\n                stored: false,\n                coerce: true,\n            }\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/schema.rs",
    "content": "use std::collections::HashMap;\nuse std::fmt;\nuse std::sync::Arc;\n\nuse serde::de::{SeqAccess, Visitor};\nuse serde::ser::SerializeSeq;\nuse serde::{Deserialize, Deserializer, Serialize, Serializer};\n\nuse super::*;\nuse crate::json_utils::split_json_path;\nuse crate::TantivyError;\n\n/// Tantivy has a very strict schema.\n/// You need to specify in advance whether a field is indexed or not,\n/// stored or not, and RAM-based or not.\n///\n/// This is done by creating a schema object, and\n/// setting up the fields one by one.\n/// It is for the moment impossible to remove fields.\n///\n/// # Examples\n///\n/// ```\n/// use tantivy::schema::*;\n///\n/// let mut schema_builder = Schema::builder();\n/// let id_field = schema_builder.add_text_field(\"id\", STRING);\n/// let title_field = schema_builder.add_text_field(\"title\", TEXT);\n/// let body_field = schema_builder.add_text_field(\"body\", TEXT);\n/// let schema = schema_builder.build();\n/// ```\n#[derive(Debug, Default)]\npub struct SchemaBuilder {\n    fields: Vec<FieldEntry>,\n    fields_map: HashMap<String, Field>,\n}\n\nimpl SchemaBuilder {\n    /// Create a new `SchemaBuilder`\n    pub fn new() -> SchemaBuilder {\n        SchemaBuilder::default()\n    }\n\n    /// Adds a new u64 field.\n    /// Returns the associated field handle\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_u64_field<T: Into<NumericOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_u64(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a new i64 field.\n    /// Returns the associated field handle\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_i64_field<T: Into<NumericOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_i64(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a new f64 field.\n    /// Returns the associated field handle\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_f64_field<T: Into<NumericOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_f64(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a new bool field.\n    /// Returns the associated field handle\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_bool_field<T: Into<NumericOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_bool(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a new date field.\n    /// Returns the associated field handle\n    /// Internally, Tantivy simply stores dates as i64 UTC timestamps,\n    /// while the user supplies DateTime values for convenience.\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_date_field<T: Into<DateOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_date(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a ip field.\n    /// Returns the associated field handle.\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_ip_addr_field<T: Into<IpAddrOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_ip_addr(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a new text field.\n    /// Returns the associated field handle\n    ///\n    /// # Panics\n    ///\n    /// Panics when field already exists.\n    pub fn add_text_field<T: Into<TextOptions>>(\n        &mut self,\n        field_name_str: &str,\n        field_options: T,\n    ) -> Field {\n        let field_name = String::from(field_name_str);\n        let field_entry = FieldEntry::new_text(field_name, field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a facet field to the schema.\n    pub fn add_facet_field(\n        &mut self,\n        field_name: &str,\n        facet_options: impl Into<FacetOptions>,\n    ) -> Field {\n        let field_entry = FieldEntry::new_facet(field_name.to_string(), facet_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a fast bytes field to the schema.\n    ///\n    /// Bytes field are not searchable and are only used\n    /// as fast field, to associate any kind of payload\n    /// to a document.\n    ///\n    /// For instance, learning-to-rank often requires to access\n    /// some document features at scoring time.\n    /// These can be serializing and stored as a bytes field to\n    /// get access rapidly when scoring each document.\n    pub fn add_bytes_field<T: Into<BytesOptions>>(\n        &mut self,\n        field_name: &str,\n        field_options: T,\n    ) -> Field {\n        let field_entry = FieldEntry::new_bytes(field_name.to_string(), field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a json object field to the schema.\n    pub fn add_json_field<T: Into<JsonObjectOptions>>(\n        &mut self,\n        field_name: &str,\n        field_options: T,\n    ) -> Field {\n        let field_entry = FieldEntry::new_json(field_name.to_string(), field_options.into());\n        self.add_field(field_entry)\n    }\n\n    /// Adds a field entry to the schema in build.\n    pub fn add_field(&mut self, field_entry: FieldEntry) -> Field {\n        let field = Field::from_field_id(self.fields.len() as u32);\n        let field_name = field_entry.name().to_string();\n        if let Some(_previous_value) = self.fields_map.insert(field_name, field) {\n            panic!(\"Field already exists in schema {}\", field_entry.name());\n        };\n        self.fields.push(field_entry);\n        field\n    }\n\n    /// Finalize the creation of a `Schema`\n    /// This will consume your `SchemaBuilder`\n    pub fn build(self) -> Schema {\n        Schema(Arc::new(InnerSchema {\n            fields: self.fields,\n            fields_map: self.fields_map,\n        }))\n    }\n}\n#[derive(Debug)]\nstruct InnerSchema {\n    fields: Vec<FieldEntry>,\n    fields_map: HashMap<String, Field>, // transient\n}\n\nimpl PartialEq for InnerSchema {\n    fn eq(&self, other: &InnerSchema) -> bool {\n        self.fields == other.fields\n    }\n}\n\nimpl Eq for InnerSchema {}\n\n/// Tantivy has a very strict schema.\n/// You need to specify in advance, whether a field is indexed or not,\n/// stored or not, and RAM-based or not.\n///\n/// This is done by creating a schema object, and\n/// setting up the fields one by one.\n/// It is for the moment impossible to remove fields.\n///\n/// # Examples\n///\n/// ```\n/// use tantivy::schema::*;\n///\n/// let mut schema_builder = Schema::builder();\n/// let id_field = schema_builder.add_text_field(\"id\", STRING);\n/// let title_field = schema_builder.add_text_field(\"title\", TEXT);\n/// let body_field = schema_builder.add_text_field(\"body\", TEXT);\n/// let schema = schema_builder.build();\n/// ```\n#[derive(Clone, Eq, PartialEq, Debug)]\npub struct Schema(Arc<InnerSchema>);\n\n// Returns the position (in byte offsets) of the unescaped '.' in the `field_path`.\n//\n// This function operates directly on bytes (as opposed to codepoint), relying\n// on a encoding property of utf-8 for its correctness.\nfn locate_splitting_dots(field_path: &str) -> Vec<usize> {\n    let mut splitting_dots_pos = Vec::new();\n    let mut escape_state = false;\n    for (pos, b) in field_path.bytes().enumerate() {\n        if escape_state {\n            escape_state = false;\n            continue;\n        }\n        match b {\n            b'\\\\' => {\n                escape_state = true;\n            }\n            b'.' => {\n                splitting_dots_pos.push(pos);\n            }\n            _ => {}\n        }\n    }\n    splitting_dots_pos\n}\n\nimpl Schema {\n    /// Return the `FieldEntry` associated with a `Field`.\n    #[inline]\n    pub fn get_field_entry(&self, field: Field) -> &FieldEntry {\n        &self.0.fields[field.field_id() as usize]\n    }\n\n    /// Return the field name for a given `Field`.\n    pub fn get_field_name(&self, field: Field) -> &str {\n        self.get_field_entry(field).name()\n    }\n\n    /// Returns the number of fields in the schema.\n    pub fn num_fields(&self) -> usize {\n        self.0.fields.len()\n    }\n\n    /// Return the list of all the `Field`s.\n    pub fn fields(&self) -> impl Iterator<Item = (Field, &FieldEntry)> {\n        self.0\n            .fields\n            .iter()\n            .enumerate()\n            .map(|(field_id, field_entry)| (Field::from_field_id(field_id as u32), field_entry))\n    }\n\n    /// Creates a new builder.\n    pub fn builder() -> SchemaBuilder {\n        SchemaBuilder::default()\n    }\n\n    /// Returns the field option associated with a given name.\n    pub fn get_field(&self, field_name: &str) -> crate::Result<Field> {\n        self.0\n            .fields_map\n            .get(field_name)\n            .cloned()\n            .ok_or_else(|| TantivyError::FieldNotFound(field_name.to_string()))\n    }\n\n    /// Searches for a full_path in the schema, returning the field name and a JSON path.\n    ///\n    /// This function works by checking if the field exists for the exact given full_path.\n    /// If it's not, it splits the full_path at non-escaped '.' chars and tries to match the\n    /// prefix with the field names, favoring the longest field names.\n    ///\n    /// This does not check if field is a JSON field. It is possible for this functions to\n    /// return a non-empty JSON path with a non-JSON field.\n    pub fn find_field<'a>(&self, full_path: &'a str) -> Option<(Field, &'a str)> {\n        if let Some(field) = self.0.fields_map.get(full_path) {\n            return Some((*field, \"\"));\n        }\n\n        let mut splitting_period_pos: Vec<usize> = locate_splitting_dots(full_path);\n        while let Some(pos) = splitting_period_pos.pop() {\n            let (prefix, suffix) = full_path.split_at(pos);\n\n            if let Some(field) = self.0.fields_map.get(prefix) {\n                return Some((*field, &suffix[1..]));\n            }\n            // JSON path may contain a dot, for now we try both variants to find the field.\n            let prefix = split_json_path(prefix).join(\".\");\n            if let Some(field) = self.0.fields_map.get(&prefix) {\n                return Some((*field, &suffix[1..]));\n            }\n        }\n        None\n    }\n\n    /// Transforms a user-supplied fast field name into a column name.\n    ///\n    /// This is similar to `.find_field` except it includes some fallback logic to\n    /// a default json field. This functionality is used in Quickwit.\n    ///\n    /// If the remaining path is empty and seems to target JSON field, we return None.\n    /// If the remaining path is non-empty and seems to target a non-JSON field, we return None.\n    #[doc(hidden)]\n    pub fn find_field_with_default<'a>(\n        &self,\n        full_path: &'a str,\n\n        default_field_opt: Option<Field>,\n    ) -> Option<(Field, &'a str)> {\n        let (field, json_path) = self\n            .find_field(full_path)\n            .or(default_field_opt.map(|field| (field, full_path)))?;\n        let field_entry = self.get_field_entry(field);\n        let is_json = field_entry.field_type().value_type() == Type::Json;\n        if !is_json && !json_path.is_empty() {\n            return None;\n        }\n        Some((field, json_path))\n    }\n}\n\nimpl Serialize for Schema {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: Serializer {\n        let mut seq = serializer.serialize_seq(Some(self.0.fields.len()))?;\n        for e in &self.0.fields {\n            seq.serialize_element(e)?;\n        }\n        seq.end()\n    }\n}\n\nimpl<'de> Deserialize<'de> for Schema {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: Deserializer<'de> {\n        struct SchemaVisitor;\n\n        impl<'de> Visitor<'de> for SchemaVisitor {\n            type Value = Schema;\n\n            fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {\n                formatter.write_str(\"struct Schema\")\n            }\n\n            fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>\n            where A: SeqAccess<'de> {\n                let mut schema = SchemaBuilder {\n                    fields: Vec::with_capacity(seq.size_hint().unwrap_or(0)),\n                    fields_map: HashMap::with_capacity(seq.size_hint().unwrap_or(0)),\n                };\n\n                while let Some(value) = seq.next_element()? {\n                    schema.add_field(value);\n                }\n\n                Ok(schema.build())\n            }\n        }\n\n        deserializer.deserialize_seq(SchemaVisitor)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::BTreeMap;\n\n    use matches::{assert_matches, matches};\n    use pretty_assertions::assert_eq;\n\n    use crate::schema::field_type::ValueParsingError;\n    use crate::schema::schema::DocParsingError::InvalidJson;\n    use crate::schema::*;\n\n    #[test]\n    fn test_locate_splitting_dots() {\n        assert_eq!(&super::locate_splitting_dots(\"a.b.c\"), &[1, 3]);\n        assert_eq!(&super::locate_splitting_dots(r\"a\\.b.c\"), &[4]);\n        assert_eq!(&super::locate_splitting_dots(r\"a\\..b.c\"), &[3, 5]);\n    }\n\n    #[test]\n    pub fn is_indexed_test() {\n        let mut schema_builder = Schema::builder();\n        let field_str = schema_builder.add_text_field(\"field_str\", STRING);\n        let schema = schema_builder.build();\n        assert!(schema.get_field_entry(field_str).is_indexed());\n    }\n\n    #[test]\n    pub fn test_schema_serialization() {\n        let mut schema_builder = Schema::builder();\n        let count_options = NumericOptions::default().set_stored().set_fast();\n        let popularity_options = NumericOptions::default().set_stored().set_fast();\n        let score_options = NumericOptions::default()\n            .set_indexed()\n            .set_fieldnorm()\n            .set_fast();\n        let is_read_options = NumericOptions::default().set_stored().set_fast();\n        schema_builder.add_text_field(\"title\", TEXT);\n        schema_builder.add_text_field(\n            \"author\",\n            TextOptions::default().set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_tokenizer(\"raw\")\n                    .set_fieldnorms(false),\n            ),\n        );\n        schema_builder.add_u64_field(\"count\", count_options);\n        schema_builder.add_i64_field(\"popularity\", popularity_options);\n        schema_builder.add_f64_field(\"score\", score_options);\n        schema_builder.add_bool_field(\"is_read\", is_read_options);\n        let schema = schema_builder.build();\n        let schema_json = serde_json::to_string_pretty(&schema).unwrap();\n        let expected = r#\"[\n  {\n    \"name\": \"title\",\n    \"type\": \"text\",\n    \"options\": {\n      \"indexing\": {\n        \"record\": \"position\",\n        \"fieldnorms\": true,\n        \"tokenizer\": \"default\"\n      },\n      \"stored\": false,\n      \"fast\": false\n    }\n  },\n  {\n    \"name\": \"author\",\n    \"type\": \"text\",\n    \"options\": {\n      \"indexing\": {\n        \"record\": \"basic\",\n        \"fieldnorms\": false,\n        \"tokenizer\": \"raw\"\n      },\n      \"stored\": false,\n      \"fast\": false\n    }\n  },\n  {\n    \"name\": \"count\",\n    \"type\": \"u64\",\n    \"options\": {\n      \"indexed\": false,\n      \"fieldnorms\": false,\n      \"fast\": true,\n      \"stored\": true\n    }\n  },\n  {\n    \"name\": \"popularity\",\n    \"type\": \"i64\",\n    \"options\": {\n      \"indexed\": false,\n      \"fieldnorms\": false,\n      \"fast\": true,\n      \"stored\": true\n    }\n  },\n  {\n    \"name\": \"score\",\n    \"type\": \"f64\",\n    \"options\": {\n      \"indexed\": true,\n      \"fieldnorms\": true,\n      \"fast\": true,\n      \"stored\": false\n    }\n  },\n  {\n    \"name\": \"is_read\",\n    \"type\": \"bool\",\n    \"options\": {\n      \"indexed\": false,\n      \"fieldnorms\": false,\n      \"fast\": true,\n      \"stored\": true\n    }\n  }\n]\"#;\n        assert_eq!(schema_json, expected);\n\n        let schema: Schema = serde_json::from_str(expected).unwrap();\n\n        let mut fields = schema.fields();\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"title\", field_entry.name());\n            assert_eq!(0, field.field_id());\n        }\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"author\", field_entry.name());\n            assert_eq!(1, field.field_id());\n        }\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"count\", field_entry.name());\n            assert_eq!(2, field.field_id());\n        }\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"popularity\", field_entry.name());\n            assert_eq!(3, field.field_id());\n        }\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"score\", field_entry.name());\n            assert_eq!(4, field.field_id());\n        }\n        {\n            let (field, field_entry) = fields.next().unwrap();\n            assert_eq!(\"is_read\", field_entry.name());\n            assert_eq!(5, field.field_id());\n        }\n        assert!(fields.next().is_none());\n    }\n\n    #[test]\n    pub fn test_document_to_json() {\n        let mut schema_builder = Schema::builder();\n        let count_options = NumericOptions::default().set_stored().set_fast();\n        let is_read_options = NumericOptions::default().set_stored().set_fast();\n        schema_builder.add_text_field(\"title\", TEXT);\n        schema_builder.add_text_field(\"author\", STRING);\n        schema_builder.add_u64_field(\"count\", count_options);\n        schema_builder.add_ip_addr_field(\"ip\", FAST | STORED);\n        schema_builder.add_bool_field(\"is_read\", is_read_options);\n        let schema = schema_builder.build();\n        let doc_json = r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 4,\n                \"ip\": \"127.0.0.1\",\n                \"is_read\": true\n        }\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n\n        let doc_serdeser = TantivyDocument::parse_json(&schema, &doc.to_json(&schema)).unwrap();\n        assert_eq!(doc, doc_serdeser);\n    }\n\n    #[test]\n    pub fn test_document_to_ipv4_json() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_ip_addr_field(\"ip\", FAST | STORED);\n        let schema = schema_builder.build();\n\n        // IpV4 loopback\n        let doc_json = r#\"{\n                \"ip\": \"127.0.0.1\"\n        }\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n        let value: serde_json::Value = serde_json::from_str(&doc.to_json(&schema)).unwrap();\n        assert_eq!(value[\"ip\"][0], \"127.0.0.1\");\n\n        // Special case IpV6 loopback. We don't want to map that to IPv4\n        let doc_json = r#\"{\n                \"ip\": \"::1\"\n        }\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n\n        let value: serde_json::Value = serde_json::from_str(&doc.to_json(&schema)).unwrap();\n        assert_eq!(value[\"ip\"][0], \"::1\");\n\n        // testing ip address of every router in the world\n        let doc_json = r#\"{\n                \"ip\": \"192.168.0.1\"\n        }\"#;\n        let doc = TantivyDocument::parse_json(&schema, doc_json).unwrap();\n\n        let value: serde_json::Value = serde_json::from_str(&doc.to_json(&schema)).unwrap();\n        assert_eq!(value[\"ip\"][0], \"192.168.0.1\");\n    }\n\n    #[test]\n    pub fn test_document_from_nameddoc() {\n        let mut schema_builder = Schema::builder();\n        let title = schema_builder.add_text_field(\"title\", TEXT);\n        let val = schema_builder.add_i64_field(\"val\", INDEXED);\n        let schema = schema_builder.build();\n        let mut named_doc_map = BTreeMap::default();\n        named_doc_map.insert(\n            \"title\".to_string(),\n            vec![OwnedValue::from(\"title1\"), OwnedValue::from(\"title2\")],\n        );\n        named_doc_map.insert(\n            \"val\".to_string(),\n            vec![OwnedValue::from(14u64), OwnedValue::from(-1i64)],\n        );\n        let doc =\n            TantivyDocument::convert_named_doc(&schema, NamedFieldDocument(named_doc_map)).unwrap();\n        assert_eq!(\n            doc.get_all(title).map(OwnedValue::from).collect::<Vec<_>>(),\n            vec![\n                OwnedValue::from(\"title1\".to_string()),\n                OwnedValue::from(\"title2\".to_string())\n            ]\n        );\n        assert_eq!(\n            doc.get_all(val).map(OwnedValue::from).collect::<Vec<_>>(),\n            vec![OwnedValue::from(14u64), OwnedValue::from(-1i64)]\n        );\n    }\n\n    #[test]\n    pub fn test_document_missing_field_no_error() {\n        let schema = Schema::builder().build();\n        let mut named_doc_map = BTreeMap::default();\n        named_doc_map.insert(\n            \"title\".to_string(),\n            vec![OwnedValue::from(\"title1\"), OwnedValue::from(\"title2\")],\n        );\n        TantivyDocument::convert_named_doc(&schema, NamedFieldDocument(named_doc_map)).unwrap();\n    }\n\n    #[test]\n    pub fn test_parse_document() {\n        let mut schema_builder = Schema::builder();\n        let count_options = NumericOptions::default().set_stored().set_fast();\n        let popularity_options = NumericOptions::default().set_stored().set_fast();\n        let score_options = NumericOptions::default().set_indexed().set_fast();\n        let title_field = schema_builder.add_text_field(\"title\", TEXT);\n        let author_field = schema_builder.add_text_field(\"author\", STRING);\n        let count_field = schema_builder.add_u64_field(\"count\", count_options);\n        let popularity_field = schema_builder.add_i64_field(\"popularity\", popularity_options);\n        let score_field = schema_builder.add_f64_field(\"score\", score_options);\n        let schema = schema_builder.build();\n        {\n            let doc = TantivyDocument::parse_json(&schema, \"{}\").unwrap();\n            assert!(doc.field_values().next().is_none());\n        }\n        {\n            let doc = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 4,\n                \"popularity\": 10,\n                \"score\": 80.5\n            }\"#,\n            )\n            .unwrap();\n            assert_eq!(\n                doc.get_first(title_field).unwrap().as_str(),\n                Some(\"my title\")\n            );\n            assert_eq!(\n                doc.get_first(author_field).unwrap().as_str(),\n                Some(\"fulmicoton\")\n            );\n            assert_eq!(doc.get_first(count_field).unwrap().as_u64(), Some(4));\n            assert_eq!(doc.get_first(popularity_field).unwrap().as_i64(), Some(10));\n            assert_eq!(doc.get_first(score_field).unwrap().as_f64(), Some(80.5f64));\n        }\n        {\n            let res = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"thisfieldisnotdefinedintheschema\": \"my title\",\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 4,\n                \"popularity\": 10,\n                \"score\": 80.5,\n                \"jambon\": \"bayonne\"\n            }\"#,\n            );\n            assert!(res.is_ok());\n        }\n        {\n            let json_err = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": \"5\",\n                \"popularity\": \"10\",\n                \"score\": \"80.5\",\n                \"jambon\": \"bayonne\"\n            }\"#,\n            );\n            assert_matches!(\n                json_err,\n                Err(DocParsingError::ValueError(\n                    _,\n                    ValueParsingError::TypeError { .. }\n                ))\n            );\n        }\n        {\n            let json_err = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": -5,\n                \"popularity\": 10,\n                \"score\": 80.5\n            }\"#,\n            );\n            assert_matches!(\n                json_err,\n                Err(DocParsingError::ValueError(\n                    _,\n                    ValueParsingError::OverflowError { .. }\n                ))\n            );\n        }\n        {\n            let json_err = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 9223372036854775808,\n                \"popularity\": 10,\n                \"score\": 80.5\n            }\"#,\n            );\n            assert!(!matches!(\n                json_err,\n                Err(DocParsingError::ValueError(\n                    _,\n                    ValueParsingError::OverflowError { .. }\n                ))\n            ));\n        }\n        {\n            let json_err = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 50,\n                \"popularity\": 9223372036854775808,\n                \"score\": 80.5\n            }\"#,\n            );\n            assert_matches!(\n                json_err,\n                Err(DocParsingError::ValueError(\n                    _,\n                    ValueParsingError::OverflowError { .. }\n                ))\n            );\n        }\n        {\n            // Short JSON, under the 20 char take.\n            let json_err = TantivyDocument::parse_json(&schema, r#\"{\"count\": 50,}\"#);\n            assert_matches!(json_err, Err(InvalidJson(_)));\n        }\n        {\n            let json_err = TantivyDocument::parse_json(\n                &schema,\n                r#\"{\n                \"title\": \"my title\",\n                \"author\": \"fulmicoton\",\n                \"count\": 50,\n            }\"#,\n            );\n            assert_matches!(json_err, Err(InvalidJson(_)));\n        }\n    }\n\n    #[test]\n    pub fn test_schema_add_field() {\n        let mut schema_builder = SchemaBuilder::default();\n        let id_options = TextOptions::default().set_stored().set_indexing_options(\n            TextFieldIndexing::default()\n                .set_tokenizer(\"raw\")\n                .set_index_option(IndexRecordOption::Basic),\n        );\n        let timestamp_options = DateOptions::default()\n            .set_stored()\n            .set_indexed()\n            .set_fieldnorm()\n            .set_fast();\n        schema_builder.add_text_field(\"_id\", id_options);\n        schema_builder.add_date_field(\"_timestamp\", timestamp_options);\n\n        let schema_content = r#\"[\n  {\n    \"name\": \"text\",\n    \"type\": \"text\",\n    \"options\": {\n      \"indexing\": {\n        \"record\": \"position\",\n        \"fieldnorms\": true,\n        \"tokenizer\": \"default\"\n      },\n      \"stored\": false,\n      \"fast\": false\n    }\n  },\n  {\n    \"name\": \"popularity\",\n    \"type\": \"i64\",\n    \"options\": {\n      \"indexed\": false,\n      \"fieldnorms\": false,\n      \"fast\": true,\n      \"stored\": true\n    }\n  }\n]\"#;\n        let tmp_schema: Schema =\n            serde_json::from_str(schema_content).expect(\"error while reading json\");\n        for (_field, field_entry) in tmp_schema.fields() {\n            schema_builder.add_field(field_entry.clone());\n        }\n\n        let schema = schema_builder.build();\n        let schema_json = serde_json::to_string_pretty(&schema).unwrap();\n        let expected = r#\"[\n  {\n    \"name\": \"_id\",\n    \"type\": \"text\",\n    \"options\": {\n      \"indexing\": {\n        \"record\": \"basic\",\n        \"fieldnorms\": true,\n        \"tokenizer\": \"raw\"\n      },\n      \"stored\": true,\n      \"fast\": false\n    }\n  },\n  {\n    \"name\": \"_timestamp\",\n    \"type\": \"date\",\n    \"options\": {\n      \"indexed\": true,\n      \"fieldnorms\": true,\n      \"fast\": true,\n      \"stored\": true,\n      \"precision\": \"seconds\"\n    }\n  },\n  {\n    \"name\": \"text\",\n    \"type\": \"text\",\n    \"options\": {\n      \"indexing\": {\n        \"record\": \"position\",\n        \"fieldnorms\": true,\n        \"tokenizer\": \"default\"\n      },\n      \"stored\": false,\n      \"fast\": false\n    }\n  },\n  {\n    \"name\": \"popularity\",\n    \"type\": \"i64\",\n    \"options\": {\n      \"indexed\": false,\n      \"fieldnorms\": false,\n      \"fast\": true,\n      \"stored\": true\n    }\n  }\n]\"#;\n        assert_eq!(schema_json, expected);\n    }\n\n    #[test]\n    fn test_find_field() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_json_field(\"foo\", STRING);\n\n        schema_builder.add_text_field(\"bar\", STRING);\n        schema_builder.add_text_field(\"foo.bar\", STRING);\n        schema_builder.add_text_field(\"foo.bar.baz\", STRING);\n        schema_builder.add_text_field(\"bar.a.b.c\", STRING);\n        let schema = schema_builder.build();\n\n        assert_eq!(\n            schema.find_field(\"foo.bar\"),\n            Some((schema.get_field(\"foo.bar\").unwrap(), \"\"))\n        );\n        assert_eq!(\n            schema.find_field(\"foo.bar.bar\"),\n            Some((schema.get_field(\"foo.bar\").unwrap(), \"bar\"))\n        );\n        assert_eq!(\n            schema.find_field(\"foo.bar.baz\"),\n            Some((schema.get_field(\"foo.bar.baz\").unwrap(), \"\"))\n        );\n        assert_eq!(\n            schema.find_field(\"foo.toto\"),\n            Some((schema.get_field(\"foo\").unwrap(), \"toto\"))\n        );\n        assert_eq!(\n            schema.find_field(\"foo.bar\"),\n            Some((schema.get_field(\"foo.bar\").unwrap(), \"\"))\n        );\n        assert_eq!(\n            schema.find_field(\"bar.toto.titi\"),\n            Some((schema.get_field(\"bar\").unwrap(), \"toto.titi\"))\n        );\n\n        assert_eq!(schema.find_field(\"hello\"), None);\n        assert_eq!(schema.find_field(\"\"), None);\n        assert_eq!(schema.find_field(\"thiswouldbeareallylongfieldname\"), None);\n        assert_eq!(schema.find_field(\"baz.bar.foo\"), None);\n    }\n\n    #[test]\n    fn test_find_field_with_default() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_json_field(\"_default\", JsonObjectOptions::default());\n        let default = Field::from_field_id(0);\n        schema_builder.add_json_field(\"foo\", STRING);\n        let foo = Field::from_field_id(1);\n        schema_builder.add_text_field(\"foo.bar\", STRING);\n        let foo_bar = Field::from_field_id(2);\n        schema_builder.add_text_field(\"bar\", STRING);\n        let bar = Field::from_field_id(3);\n        schema_builder.add_json_field(\"baz\", JsonObjectOptions::default());\n        let baz = Field::from_field_id(4);\n        let schema = schema_builder.build();\n\n        assert_eq!(schema.find_field_with_default(\"foo\", None), Some((foo, \"\")));\n        assert_eq!(\n            schema.find_field_with_default(\"foo.bar\", None),\n            Some((foo_bar, \"\"))\n        );\n        assert_eq!(schema.find_field_with_default(\"bar\", None), Some((bar, \"\")));\n        assert_eq!(schema.find_field_with_default(\"bar.baz\", None), None);\n        assert_eq!(schema.find_field_with_default(\"baz\", None), Some((baz, \"\")));\n        assert_eq!(\n            schema.find_field_with_default(\"baz.foobar\", None),\n            Some((baz, \"foobar\"))\n        );\n        assert_eq!(schema.find_field_with_default(\"foobar\", None), None);\n\n        assert_eq!(\n            schema.find_field_with_default(\"foo\", Some(default)),\n            Some((foo, \"\"))\n        );\n        assert_eq!(\n            schema.find_field_with_default(\"foo.bar\", Some(default)),\n            Some((foo_bar, \"\"))\n        );\n        assert_eq!(\n            schema.find_field_with_default(\"bar\", Some(default)),\n            Some((bar, \"\"))\n        );\n        // still None, we are under an existing field\n        assert_eq!(\n            schema.find_field_with_default(\"bar.baz\", Some(default)),\n            None\n        );\n        assert_eq!(\n            schema.find_field_with_default(\"baz\", Some(default)),\n            Some((baz, \"\"))\n        );\n        assert_eq!(\n            schema.find_field_with_default(\"baz.foobar\", Some(default)),\n            Some((baz, \"foobar\"))\n        );\n        assert_eq!(\n            schema.find_field_with_default(\"foobar\", Some(default)),\n            Some((default, \"foobar\"))\n        );\n    }\n}\n"
  },
  {
    "path": "src/schema/term.rs",
    "content": "use std::hash::Hash;\nuse std::net::Ipv6Addr;\nuse std::{fmt, str};\n\nuse columnar::MonotonicallyMappableToU128;\nuse common::json_path_writer::{JSON_END_OF_PATH, JSON_PATH_SEGMENT_SEP_STR};\nuse common::JsonPathWriter;\nuse serde::{Deserialize, Serialize};\n\nuse super::date_time_options::DATE_TIME_PRECISION_INDEXED;\nuse super::{Field, Schema};\nuse crate::fastfield::FastValue;\nuse crate::json_utils::split_json_path;\nuse crate::schema::{Facet, Type};\nuse crate::DateTime;\n\n/// Term represents the value that the token can take.\n/// It's a serialized representation over different types.\n///\n/// A term is composed of Field and the serialized value bytes.\n/// The serialized value bytes themselves start with a one byte type tag followed by the payload.\n#[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]\npub struct Term {\n    field: Field,\n    serialized_value_bytes: Vec<u8>,\n}\n\n/// The number of bytes used as metadata when serializing a term.\nconst TERM_TYPE_TAG_LEN: usize = 1;\n\nimpl Term {\n    /// Takes a serialized term and wraps it as a Term.\n    /// First 4 bytes are the field id\n    #[deprecated(\n        note = \"we want to avoid working on the serialized representation directly, replace with \\\n                typed API calls (add more if needed) or use serde to serialize/deserialize\"\n    )]\n    pub fn wrap(serialized: &[u8]) -> Term {\n        let field_id_bytes: [u8; 4] = serialized[0..4].try_into().unwrap();\n        let field_id = u32::from_be_bytes(field_id_bytes);\n        Term {\n            field: Field::from_field_id(field_id),\n            serialized_value_bytes: serialized[4..].to_vec(),\n        }\n    }\n\n    /// Returns the serialized representation of the term.\n    /// First 4 bytes are the field id\n    #[deprecated(\n        note = \"we want to avoid working on the serialized representation directly, replace with \\\n                typed API calls (add more if needed) or use serde to serialize/deserialize\"\n    )]\n    pub fn serialized_term(&self) -> Vec<u8> {\n        let mut serialized = Vec::with_capacity(4 + self.serialized_value_bytes.len());\n        serialized.extend(self.field.field_id().to_be_bytes().as_ref());\n        serialized.extend_from_slice(&self.serialized_value_bytes);\n        serialized\n    }\n\n    /// Create a new Term with a buffer with a given capacity.\n    pub fn with_capacity(capacity: usize) -> Term {\n        let mut data = Vec::with_capacity(TERM_TYPE_TAG_LEN + capacity);\n        data.resize(TERM_TYPE_TAG_LEN, 0u8);\n        Term {\n            field: Field::from_field_id(0u32),\n            serialized_value_bytes: data,\n        }\n    }\n\n    /// Creates a term from a json path.\n    ///\n    /// The json path can address a nested value in a JSON object.\n    /// e.g. `{\"k8s\": {\"node\": {\"id\": 5}}}` can be addressed via `k8s.node.id`.\n    ///\n    /// In case there are dots in the field name, and the `expand_dots_enabled` parameter is not\n    /// set they need to be escaped with a backslash.\n    /// e.g. `{\"k8s.node\": {\"id\": 5}}` can be addressed via `k8s\\.node.id`.\n    pub fn from_field_json_path(field: Field, json_path: &str, expand_dots_enabled: bool) -> Term {\n        let paths = split_json_path(json_path);\n        let mut json_path = JsonPathWriter::with_expand_dots(expand_dots_enabled);\n        for path in paths {\n            json_path.push(&path);\n        }\n        json_path.set_end();\n        let mut term = Term::with_type_and_field(Type::Json, field);\n\n        term.append_bytes(json_path.as_str().as_bytes());\n\n        term\n    }\n\n    /// Gets the full path of the field name + optional json path.\n    pub fn get_full_path(&self, schema: &Schema) -> String {\n        let field = self.field();\n        let mut field = schema.get_field_name(field).to_string();\n        if let Some(json_path) = self.get_json_path() {\n            field.push('.');\n            field.push_str(&json_path);\n        };\n        field\n    }\n\n    /// Gets the json path if the type is JSON\n    pub fn get_json_path(&self) -> Option<String> {\n        let value = self.value();\n        if let Some((json_path, _)) = value.as_json() {\n            Some(unsafe {\n                std::str::from_utf8_unchecked(&json_path[..json_path.len() - 1]).to_string()\n            })\n        } else {\n            None\n        }\n    }\n\n    pub(crate) fn with_type_and_field(typ: Type, field: Field) -> Term {\n        let mut term = Self::with_capacity(8);\n        term.set_field_and_type(field, typ);\n        term\n    }\n\n    fn with_bytes_and_field_and_payload(typ: Type, field: Field, bytes: &[u8]) -> Term {\n        let mut term = Self::with_capacity(bytes.len());\n        term.set_field_and_type(field, typ);\n        term.serialized_value_bytes.extend_from_slice(bytes);\n        term\n    }\n\n    pub(crate) fn from_fast_value<T: FastValue>(field: Field, val: &T) -> Term {\n        let mut term = Self::with_type_and_field(T::to_type(), field);\n        term.set_bytes(val.to_u64().to_be_bytes().as_ref());\n        term\n    }\n\n    /// Panics when the term is not empty... ie: some value is set.\n    /// Use `clear_with_field_and_type` in that case.\n    ///\n    /// Sets field and the type.\n    pub(crate) fn set_field_and_type(&mut self, field: Field, typ: Type) {\n        assert!(self.is_empty());\n        self.field = field;\n        self.serialized_value_bytes[0] = typ.to_code();\n    }\n\n    /// Is empty if there are no value bytes.\n    pub fn is_empty(&self) -> bool {\n        self.serialized_value_bytes.len() == TERM_TYPE_TAG_LEN\n    }\n\n    /// Builds a term given a field, and a `Ipv6Addr`-value\n    pub fn from_field_ip_addr(field: Field, ip_addr: Ipv6Addr) -> Term {\n        let mut term = Self::with_type_and_field(Type::IpAddr, field);\n        term.set_bytes(ip_addr.to_u128().to_be_bytes().as_ref());\n        term\n    }\n\n    /// Builds a term given a field, and a `u64`-value\n    pub fn from_field_u64(field: Field, val: u64) -> Term {\n        Term::from_fast_value(field, &val)\n    }\n\n    /// Builds a term given a field, and a `i64`-value\n    pub fn from_field_i64(field: Field, val: i64) -> Term {\n        Term::from_fast_value(field, &val)\n    }\n\n    /// Builds a term given a field, and a `f64`-value\n    pub fn from_field_f64(field: Field, val: f64) -> Term {\n        Term::from_fast_value(field, &val)\n    }\n\n    /// Builds a term given a field, and a `bool`-value\n    pub fn from_field_bool(field: Field, val: bool) -> Term {\n        Term::from_fast_value(field, &val)\n    }\n\n    /// Builds a term given a field, and a `DateTime` value.\n    ///\n    /// The contained value may not match the value, due do the truncation used\n    /// for indexed data [super::DATE_TIME_PRECISION_INDEXED].\n    /// To create a term used for search use `from_field_date_for_search`.\n    pub fn from_field_date(field: Field, val: DateTime) -> Term {\n        Term::from_fast_value(field, &val)\n    }\n\n    /// Builds a term given a field, and a `DateTime` value to be used in searching the inverted\n    /// index.\n    /// It truncates the `DateTime` to the precision used in the index\n    /// ([super::DATE_TIME_PRECISION_INDEXED]).\n    pub fn from_field_date_for_search(field: Field, val: DateTime) -> Term {\n        Term::from_fast_value(field, &val.truncate(DATE_TIME_PRECISION_INDEXED))\n    }\n\n    /// Creates a `Term` given a facet.\n    pub fn from_facet(field: Field, facet: &Facet) -> Term {\n        let facet_encoded_str = facet.encoded_str();\n        Term::with_bytes_and_field_and_payload(Type::Facet, field, facet_encoded_str.as_bytes())\n    }\n\n    /// Builds a term given a field, and a string value\n    pub fn from_field_text(field: Field, text: &str) -> Term {\n        Term::with_bytes_and_field_and_payload(Type::Str, field, text.as_bytes())\n    }\n\n    /// Builds a term bytes.\n    pub fn from_field_bytes(field: Field, bytes: &[u8]) -> Term {\n        Term::with_bytes_and_field_and_payload(Type::Bytes, field, bytes)\n    }\n\n    /// Removes the value_bytes and set the type code.\n    pub fn clear_with_type(&mut self, typ: Type) {\n        self.truncate_value_bytes(0);\n        self.serialized_value_bytes[0] = typ.to_code();\n    }\n\n    /// Append a type marker + fast value to a term.\n    /// This is used in JSON type to append a fast value after the path.\n    ///\n    /// It will not clear existing bytes.\n    pub fn append_type_and_fast_value<T: FastValue>(&mut self, val: T) {\n        self.serialized_value_bytes.push(T::to_type().to_code());\n        let value = val.to_u64();\n        self.serialized_value_bytes\n            .extend(value.to_be_bytes().as_ref());\n    }\n\n    /// Append a string type marker + string to a term.\n    /// This is used in JSON type to append a str after the path.\n    ///\n    /// It will not clear existing bytes.\n    pub fn append_type_and_str(&mut self, val: &str) {\n        self.serialized_value_bytes.push(Type::Str.to_code());\n        self.serialized_value_bytes.extend(val.as_bytes().as_ref());\n    }\n\n    /// Sets the value of a `Bytes` field.\n    pub fn set_bytes(&mut self, bytes: &[u8]) {\n        self.truncate_value_bytes(0);\n        self.serialized_value_bytes.extend(bytes);\n    }\n\n    /// Truncates the value bytes of the term. Value and field type stays the same.\n    pub fn truncate_value_bytes(&mut self, len: usize) {\n        self.serialized_value_bytes\n            .truncate(len + TERM_TYPE_TAG_LEN);\n    }\n\n    /// The length of the bytes.\n    pub fn len_bytes(&self) -> usize {\n        self.serialized_value_bytes.len() - TERM_TYPE_TAG_LEN\n    }\n\n    /// Appends value bytes to the Term.\n    ///\n    /// This function returns the segment that has just been added.\n    #[inline]\n    pub fn append_bytes(&mut self, bytes: &[u8]) -> &mut [u8] {\n        let len_before = self.serialized_value_bytes.len();\n        self.serialized_value_bytes.extend_from_slice(bytes);\n        &mut self.serialized_value_bytes[len_before..]\n    }\n\n    /// Return the type of the term.\n    pub fn typ(&self) -> Type {\n        self.value().typ()\n    }\n\n    /// Returns the field.\n    pub fn field(&self) -> Field {\n        self.field\n    }\n\n    /// Returns the serialized representation of the value.\n    /// (this does neither include the field id nor the value type.)\n    ///\n    /// If the term is a string, its value is utf-8 encoded.\n    /// If the term is a u64, its value is encoded according\n    /// to `byteorder::BigEndian`.\n    pub fn serialized_value_bytes(&self) -> &[u8] {\n        &self.serialized_value_bytes[TERM_TYPE_TAG_LEN..]\n    }\n\n    /// Returns the value of the term.\n    /// address or JSON path + value. (this does not include the field.)\n    pub fn value(&self) -> ValueBytes<&[u8]> {\n        ValueBytes::wrap(self.serialized_value_bytes.as_ref())\n    }\n}\n\n/// ValueBytes represents a serialized value.\n///\n/// The value can be of any type of [`Type`] (e.g. string, u64, f64, bool, date, JSON).\n/// The serialized representation matches the lexicographical order of the type.\n///\n/// The `ValueBytes` format is as follow:\n/// `[type code: u8][serialized value]`\n///\n/// For JSON `ValueBytes` equals to:\n/// `[type code=JSON][JSON path][JSON_END_OF_PATH][ValueBytes]`\n///\n/// The nested ValueBytes in JSON is never of type JSON. (there's no recursion)\n#[derive(Clone)]\npub struct ValueBytes<B>(B)\nwhere B: AsRef<[u8]>;\n\nimpl<B> ValueBytes<B>\nwhere B: AsRef<[u8]>\n{\n    /// Wraps a object holding bytes\n    pub fn wrap(data: B) -> ValueBytes<B> {\n        ValueBytes(data)\n    }\n\n    /// Wraps a object holding Vec<u8>\n    pub fn to_owned(&self) -> ValueBytes<Vec<u8>> {\n        ValueBytes(self.0.as_ref().to_vec())\n    }\n\n    fn typ_code(&self) -> u8 {\n        self.0.as_ref()[0]\n    }\n\n    /// Return the type of the term.\n    pub fn typ(&self) -> Type {\n        Type::from_code(self.typ_code()).expect(\"The term has an invalid type code\")\n    }\n\n    /// Returns the `u64` value stored in a term.\n    ///\n    /// Returns `None` if the term is not of the u64 type, or if the term byte representation\n    /// is invalid.\n    pub fn as_u64(&self) -> Option<u64> {\n        self.get_fast_type::<u64>()\n    }\n\n    fn get_fast_type<T: FastValue>(&self) -> Option<T> {\n        if self.typ() != T::to_type() {\n            return None;\n        }\n        let value_bytes = self.raw_value_bytes_payload();\n        let value_u64 = u64::from_be_bytes(value_bytes.try_into().ok()?);\n        Some(T::from_u64(value_u64))\n    }\n\n    /// Returns the `i64` value stored in a term.\n    ///\n    /// Returns `None` if the term is not of the i64 type, or if the term byte representation\n    /// is invalid.\n    pub fn as_i64(&self) -> Option<i64> {\n        self.get_fast_type::<i64>()\n    }\n\n    /// Returns the `f64` value stored in a term.\n    ///\n    /// Returns `None` if the term is not of the f64 type, or if the term byte representation\n    /// is invalid.\n    pub fn as_f64(&self) -> Option<f64> {\n        self.get_fast_type::<f64>()\n    }\n\n    /// Returns the `bool` value stored in a term.\n    ///\n    /// Returns `None` if the term is not of the bool type, or if the term byte representation\n    /// is invalid.\n    pub fn as_bool(&self) -> Option<bool> {\n        self.get_fast_type::<bool>()\n    }\n\n    /// Returns the `Date` value stored in a term.\n    ///\n    /// Returns `None` if the term is not of the Date type, or if the term byte representation\n    /// is invalid.\n    pub fn as_date(&self) -> Option<DateTime> {\n        self.get_fast_type::<DateTime>()\n    }\n\n    /// Returns the text associated with the term.\n    ///\n    /// Returns `None` if the field is not of string type\n    /// or if the bytes are not valid utf-8.\n    pub fn as_str(&self) -> Option<&str> {\n        if self.typ() != Type::Str {\n            return None;\n        }\n        str::from_utf8(self.raw_value_bytes_payload()).ok()\n    }\n\n    /// Returns the facet associated with the term.\n    ///\n    /// Returns `None` if the field is not of facet type\n    /// or if the bytes are not valid utf-8.\n    pub fn as_facet(&self) -> Option<Facet> {\n        if self.typ() != Type::Facet {\n            return None;\n        }\n        let facet_encode_str = str::from_utf8(self.raw_value_bytes_payload()).ok()?;\n        Some(Facet::from_encoded_string(facet_encode_str.to_string()))\n    }\n\n    /// Returns the bytes associated with the term.\n    ///\n    /// Returns `None` if the field is not of bytes type.\n    pub fn as_bytes(&self) -> Option<&[u8]> {\n        if self.typ() != Type::Bytes {\n            return None;\n        }\n        Some(self.raw_value_bytes_payload())\n    }\n\n    /// Returns a `Ipv6Addr` value from the term.\n    pub fn as_ip_addr(&self) -> Option<Ipv6Addr> {\n        if self.typ() != Type::IpAddr {\n            return None;\n        }\n        let ip_u128 = u128::from_be_bytes(self.raw_value_bytes_payload().try_into().ok()?);\n        Some(Ipv6Addr::from_u128(ip_u128))\n    }\n\n    /// Returns the json path type.\n    ///\n    /// Returns `None` if the value is not JSON.\n    pub fn json_path_type(&self) -> Option<Type> {\n        let json_value_bytes = self.as_json_value_bytes()?;\n\n        Some(json_value_bytes.typ())\n    }\n\n    /// Returns the json path bytes (including the JSON_END_OF_PATH byte),\n    /// and the encoded ValueBytes after the json path.\n    ///\n    /// Returns `None` if the value is not JSON.\n    pub(crate) fn as_json(&self) -> Option<(&[u8], ValueBytes<&[u8]>)> {\n        if self.typ() != Type::Json {\n            return None;\n        }\n        let bytes = self.raw_value_bytes_payload();\n\n        let pos = bytes.iter().cloned().position(|b| b == JSON_END_OF_PATH)?;\n        // split at pos + 1, so that json_path_bytes includes the JSON_END_OF_PATH byte.\n        let (json_path_bytes, term) = bytes.split_at(pos + 1);\n        Some((json_path_bytes, ValueBytes::wrap(term)))\n    }\n\n    /// Returns the encoded ValueBytes after the json path.\n    ///\n    /// Returns `None` if the value is not JSON.\n    pub(crate) fn as_json_value_bytes(&self) -> Option<ValueBytes<&[u8]>> {\n        if self.typ() != Type::Json {\n            return None;\n        }\n        let bytes = self.raw_value_bytes_payload();\n        let pos = bytes.iter().cloned().position(|b| b == JSON_END_OF_PATH)?;\n        Some(ValueBytes::wrap(&bytes[pos + 1..]))\n    }\n\n    /// Returns the raw value of ValueBytes payload, without the type tag.\n    pub(crate) fn raw_value_bytes_payload(&self) -> &[u8] {\n        &self.0.as_ref()[1..]\n    }\n\n    /// Returns the serialized value of ValueBytes payload, without the type tag.\n    pub(crate) fn value_bytes_payload(&self) -> Vec<u8> {\n        if let Some(value_bytes) = self.as_json_value_bytes() {\n            value_bytes.raw_value_bytes_payload().to_vec()\n        } else {\n            self.raw_value_bytes_payload().to_vec()\n        }\n    }\n\n    /// Returns the serialized representation of the value bytes including the type tag.\n    pub fn as_serialized(&self) -> &[u8] {\n        self.0.as_ref()\n    }\n\n    fn debug_value_bytes(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        let typ = self.typ();\n        write!(f, \"type={typ:?}, \")?;\n        match typ {\n            Type::Str => {\n                let s = self.as_str();\n                write_opt(f, s)?;\n            }\n            Type::U64 => {\n                write_opt(f, self.as_u64())?;\n            }\n            Type::I64 => {\n                write_opt(f, self.as_i64())?;\n            }\n            Type::F64 => {\n                write_opt(f, self.as_f64())?;\n            }\n            Type::Bool => {\n                write_opt(f, self.as_bool())?;\n            }\n            // TODO pretty print these types too.\n            Type::Date => {\n                write_opt(f, self.as_date())?;\n            }\n            Type::Facet => {\n                write_opt(f, self.as_facet())?;\n            }\n            Type::Bytes => {\n                write_opt(f, self.as_bytes())?;\n            }\n            Type::Json => {\n                if let Some((path_bytes, sub_value_bytes)) = self.as_json() {\n                    // Remove the JSON_END_OF_PATH byte & convert to utf8.\n                    let path = str::from_utf8(&path_bytes[..path_bytes.len() - 1])\n                        .map_err(|_| std::fmt::Error)?;\n                    let path_pretty = path.replace(JSON_PATH_SEGMENT_SEP_STR, \".\");\n                    write!(f, \"path={path_pretty}, \")?;\n                    sub_value_bytes.debug_value_bytes(f)?;\n                }\n            }\n            Type::IpAddr => {\n                write_opt(f, self.as_ip_addr())?;\n            }\n        }\n        Ok(())\n    }\n}\n\nfn write_opt<T: std::fmt::Debug>(f: &mut fmt::Formatter, val_opt: Option<T>) -> fmt::Result {\n    if let Some(val) = val_opt {\n        write!(f, \"{val:?}\")?;\n    }\n    Ok(())\n}\n\nimpl fmt::Debug for Term {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        let field_id = self.field.field_id();\n        write!(f, \"Term(field={field_id}, \")?;\n        let value_bytes = ValueBytes::wrap(&self.serialized_value_bytes);\n        value_bytes.debug_value_bytes(f)?;\n        write!(f, \")\",)?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use crate::schema::*;\n\n    #[test]\n    pub fn test_term_str() {\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"text\", STRING);\n        let title_field = schema_builder.add_text_field(\"title\", STRING);\n        let term = Term::from_field_text(title_field, \"test\");\n        assert_eq!(term.field(), title_field);\n        assert_eq!(term.typ(), Type::Str);\n        assert_eq!(term.value().as_str(), Some(\"test\"))\n    }\n\n    #[test]\n    pub fn test_term_u64() {\n        let mut schema_builder = Schema::builder();\n        let count_field = schema_builder.add_u64_field(\"count\", INDEXED);\n        let term = Term::from_field_u64(count_field, 983u64);\n        assert_eq!(term.field(), count_field);\n        assert_eq!(term.typ(), Type::U64);\n        assert_eq!(term.serialized_value_bytes().len(), 8);\n        assert_eq!(term.value().as_u64(), Some(983u64))\n    }\n\n    #[test]\n    pub fn test_term_bool() {\n        let mut schema_builder = Schema::builder();\n        let bool_field = schema_builder.add_bool_field(\"bool\", INDEXED);\n        let term = Term::from_field_bool(bool_field, true);\n        assert_eq!(term.field(), bool_field);\n        assert_eq!(term.typ(), Type::Bool);\n        assert_eq!(term.serialized_value_bytes().len(), 8);\n        assert_eq!(term.value().as_bool(), Some(true))\n    }\n}\n"
  },
  {
    "path": "src/schema/text_options.rs",
    "content": "use std::borrow::Cow;\nuse std::ops::BitOr;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::flags::{CoerceFlag, FastFlag};\nuse crate::schema::flags::{SchemaFlagList, StoredFlag};\nuse crate::schema::IndexRecordOption;\n\n/// Define how a text field should be handled by tantivy.\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Default)]\npub struct TextOptions {\n    #[serde(default)]\n    #[serde(skip_serializing_if = \"Option::is_none\")]\n    indexing: Option<TextFieldIndexing>,\n    #[serde(default)]\n    stored: bool,\n    #[serde(default)]\n    pub(crate) fast: FastFieldTextOptions,\n    #[serde(default)]\n    #[serde(skip_serializing_if = \"is_false\")]\n    /// coerce values into string if they are not of type string\n    coerce: bool,\n}\n\n#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]\n#[serde(untagged)]\n/// Enum to control how the fast field setting of a text field.\npub(crate) enum FastFieldTextOptions {\n    /// Flag to enable/disable\n    IsEnabled(bool),\n    /// Enable with tokenizer. The tokenizer must be available on the fast field tokenizer manager.\n    /// `Index::fast_field_tokenizer`.\n    EnabledWithTokenizer { with_tokenizer: TokenizerName },\n}\n\nimpl Default for FastFieldTextOptions {\n    fn default() -> Self {\n        FastFieldTextOptions::IsEnabled(false)\n    }\n}\n\nimpl BitOr<FastFieldTextOptions> for FastFieldTextOptions {\n    type Output = FastFieldTextOptions;\n\n    fn bitor(self, other: FastFieldTextOptions) -> FastFieldTextOptions {\n        match (self, other) {\n            (\n                FastFieldTextOptions::EnabledWithTokenizer {\n                    with_tokenizer: tokenizer,\n                },\n                _,\n            )\n            | (\n                _,\n                FastFieldTextOptions::EnabledWithTokenizer {\n                    with_tokenizer: tokenizer,\n                },\n            ) => FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: tokenizer,\n            },\n            (FastFieldTextOptions::IsEnabled(true), _)\n            | (_, FastFieldTextOptions::IsEnabled(true)) => FastFieldTextOptions::IsEnabled(true),\n            (_, FastFieldTextOptions::IsEnabled(false)) => FastFieldTextOptions::IsEnabled(false),\n        }\n    }\n}\n\nfn is_false(val: &bool) -> bool {\n    !val\n}\n\nimpl TextOptions {\n    /// Returns the indexing options.\n    #[inline]\n    pub fn get_indexing_options(&self) -> Option<&TextFieldIndexing> {\n        self.indexing.as_ref()\n    }\n\n    /// Returns true if the text is to be stored.\n    #[inline]\n    pub fn is_stored(&self) -> bool {\n        self.stored\n    }\n\n    /// Returns true if and only if the value is a fast field.\n    #[inline]\n    pub fn is_fast(&self) -> bool {\n        matches!(self.fast, FastFieldTextOptions::IsEnabled(true))\n            || matches!(\n                &self.fast,\n                FastFieldTextOptions::EnabledWithTokenizer { with_tokenizer: _ }\n            )\n    }\n\n    /// Returns true if and only if the value is a fast field.\n    #[inline]\n    pub fn get_fast_field_tokenizer_name(&self) -> Option<&str> {\n        match &self.fast {\n            FastFieldTextOptions::IsEnabled(true) | FastFieldTextOptions::IsEnabled(false) => None,\n            FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: tokenizer,\n            } => Some(tokenizer.name()),\n        }\n    }\n\n    /// Returns true if values should be coerced to strings (numbers, null).\n    #[inline]\n    pub fn should_coerce(&self) -> bool {\n        self.coerce\n    }\n\n    /// Set the field as a fast field.\n    ///\n    /// Fast fields are designed for random access.\n    /// Access time are similar to a random lookup in an array.\n    /// Text fast fields will have the term ids stored in the fast field.\n    ///\n    /// The effective cardinality depends on the tokenizer. Without a tokenizer, the text will be\n    /// stored as is, which equals to the \"raw\" tokenizer. The tokenizer can be used to apply\n    /// normalization like lower case.\n    /// The passed tokenizer_name must be available on the fast field tokenizer manager.\n    /// `Index::fast_field_tokenizer`.\n    ///\n    /// The original text can be retrieved via\n    /// [`TermDictionary::ord_to_term()`](crate::termdict::TermDictionary::ord_to_term)\n    /// from the dictionary.\n    #[must_use]\n    pub fn set_fast(mut self, tokenizer_name: Option<&str>) -> TextOptions {\n        if let Some(tokenizer) = tokenizer_name {\n            let tokenizer = TokenizerName::from_name(tokenizer);\n            self.fast = FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: tokenizer,\n            }\n        } else {\n            self.fast = FastFieldTextOptions::IsEnabled(true);\n        }\n        self\n    }\n\n    /// Coerce values if they are not of type string. Defaults to false.\n    #[must_use]\n    pub fn set_coerce(mut self) -> TextOptions {\n        self.coerce = true;\n        self\n    }\n\n    /// Sets the field as stored.\n    #[must_use]\n    pub fn set_stored(mut self) -> TextOptions {\n        self.stored = true;\n        self\n    }\n\n    /// Sets the field as indexed, with the specific indexing options.\n    #[must_use]\n    pub fn set_indexing_options(mut self, indexing: TextFieldIndexing) -> TextOptions {\n        self.indexing = Some(indexing);\n        self\n    }\n}\n\n#[derive(Clone, PartialEq, Debug, Eq, Serialize, Deserialize)]\npub(crate) struct TokenizerName(Cow<'static, str>);\n\nconst DEFAULT_TOKENIZER_NAME: &str = \"default\";\n\nconst NO_TOKENIZER_NAME: &str = \"raw\";\n\nimpl Default for TokenizerName {\n    fn default() -> Self {\n        TokenizerName::from_static(DEFAULT_TOKENIZER_NAME)\n    }\n}\n\nimpl TokenizerName {\n    pub const fn from_static(name: &'static str) -> Self {\n        TokenizerName(Cow::Borrowed(name))\n    }\n    pub(crate) fn from_name(name: &str) -> Self {\n        TokenizerName(Cow::Owned(name.to_string()))\n    }\n    pub(crate) fn name(&self) -> &str {\n        &self.0\n    }\n}\n\n/// Configuration defining indexing for a text field.\n///\n/// It defines\n/// - The amount of information that should be stored about the presence of a term in a document.\n///   Essentially, should we store the term frequency and/or the positions (See\n///   [`IndexRecordOption`]).\n/// - The name of the `Tokenizer` that should be used to process the field.\n/// - Flag indicating, if fieldnorms should be stored (See [fieldnorm](crate::fieldnorm)). Defaults\n///   to `true`.\n#[derive(Clone, PartialEq, Debug, Eq, Serialize, Deserialize)]\npub struct TextFieldIndexing {\n    #[serde(default)]\n    record: IndexRecordOption,\n    #[serde(default = \"default_fieldnorms\")]\n    fieldnorms: bool,\n    #[serde(default)]\n    tokenizer: TokenizerName,\n}\n\npub(crate) fn default_fieldnorms() -> bool {\n    true\n}\n\nimpl Default for TextFieldIndexing {\n    fn default() -> TextFieldIndexing {\n        TextFieldIndexing {\n            tokenizer: TokenizerName::default(),\n            record: IndexRecordOption::default(),\n            fieldnorms: default_fieldnorms(),\n        }\n    }\n}\n\nimpl TextFieldIndexing {\n    /// Sets the tokenizer to be used for a given field.\n    #[must_use]\n    pub fn set_tokenizer(mut self, tokenizer_name: &str) -> TextFieldIndexing {\n        self.tokenizer = TokenizerName::from_name(tokenizer_name);\n        self\n    }\n\n    /// Returns the tokenizer that will be used for this field.\n    pub fn tokenizer(&self) -> &str {\n        self.tokenizer.name()\n    }\n\n    /// Sets fieldnorms\n    #[must_use]\n    pub fn set_fieldnorms(mut self, fieldnorms: bool) -> TextFieldIndexing {\n        self.fieldnorms = fieldnorms;\n        self\n    }\n\n    /// Returns true if and only if [fieldnorms](crate::fieldnorm) are stored.\n    pub fn fieldnorms(&self) -> bool {\n        self.fieldnorms\n    }\n\n    /// Sets which information should be indexed with the tokens.\n    ///\n    /// See [`IndexRecordOption`] for more detail.\n    #[must_use]\n    pub fn set_index_option(mut self, index_option: IndexRecordOption) -> TextFieldIndexing {\n        self.record = index_option;\n        self\n    }\n\n    /// Returns the indexing options associated with this field.\n    ///\n    /// See [`IndexRecordOption`] for more detail.\n    pub fn index_option(&self) -> IndexRecordOption {\n        self.record\n    }\n}\n\n/// The field will be untokenized and indexed.\npub const STRING: TextOptions = TextOptions {\n    indexing: Some(TextFieldIndexing {\n        tokenizer: TokenizerName::from_static(NO_TOKENIZER_NAME),\n        fieldnorms: true,\n        record: IndexRecordOption::Basic,\n    }),\n    stored: false,\n    fast: FastFieldTextOptions::IsEnabled(false),\n    coerce: false,\n};\n\n/// The field will be tokenized and indexed.\npub const TEXT: TextOptions = TextOptions {\n    indexing: Some(TextFieldIndexing {\n        tokenizer: TokenizerName::from_static(DEFAULT_TOKENIZER_NAME),\n        fieldnorms: true,\n        record: IndexRecordOption::WithFreqsAndPositions,\n    }),\n    stored: false,\n    coerce: false,\n    fast: FastFieldTextOptions::IsEnabled(false),\n};\n\nimpl<T: Into<TextOptions>> BitOr<T> for TextOptions {\n    type Output = TextOptions;\n\n    fn bitor(self, other: T) -> TextOptions {\n        let other = other.into();\n        TextOptions {\n            indexing: self.indexing.or(other.indexing),\n            stored: self.stored | other.stored,\n            fast: self.fast | other.fast,\n            coerce: self.coerce | other.coerce,\n        }\n    }\n}\n\nimpl From<()> for TextOptions {\n    fn from(_: ()) -> TextOptions {\n        TextOptions::default()\n    }\n}\n\nimpl From<StoredFlag> for TextOptions {\n    fn from(_: StoredFlag) -> TextOptions {\n        TextOptions {\n            indexing: None,\n            stored: true,\n            fast: FastFieldTextOptions::default(),\n            coerce: false,\n        }\n    }\n}\n\nimpl From<CoerceFlag> for TextOptions {\n    fn from(_: CoerceFlag) -> TextOptions {\n        TextOptions {\n            indexing: None,\n            stored: false,\n            fast: FastFieldTextOptions::default(),\n            coerce: true,\n        }\n    }\n}\n\nimpl From<FastFlag> for TextOptions {\n    fn from(_: FastFlag) -> TextOptions {\n        TextOptions {\n            indexing: None,\n            stored: false,\n            fast: FastFieldTextOptions::IsEnabled(true),\n            coerce: false,\n        }\n    }\n}\n\nimpl<Head, Tail> From<SchemaFlagList<Head, Tail>> for TextOptions\nwhere\n    Head: Clone,\n    Tail: Clone,\n    Self: BitOr<Output = Self> + From<Head> + From<Tail>,\n{\n    fn from(head_tail: SchemaFlagList<Head, Tail>) -> Self {\n        Self::from(head_tail.head) | Self::from(head_tail.tail)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::schema::text_options::{FastFieldTextOptions, TokenizerName};\n    use crate::schema::*;\n\n    #[test]\n    fn test_field_options() {\n        let field_options = STORED | TEXT;\n        assert!(field_options.is_stored());\n        assert!(field_options.get_indexing_options().is_some());\n        let mut schema_builder = Schema::builder();\n        schema_builder.add_text_field(\"body\", TEXT);\n        let schema = schema_builder.build();\n        let field = schema.get_field(\"body\").unwrap();\n        let field_entry = schema.get_field_entry(field);\n        assert!(matches!(field_entry.field_type(),\n                FieldType::Str(text_options)\n                if text_options.get_indexing_options().unwrap().tokenizer() == \"default\"));\n    }\n\n    #[test]\n    fn test_cmp_index_record_option() {\n        assert!(IndexRecordOption::WithFreqsAndPositions > IndexRecordOption::WithFreqs);\n        assert!(IndexRecordOption::WithFreqs > IndexRecordOption::Basic);\n    }\n\n    #[test]\n    fn serde_default_test() {\n        let json = r#\"\n        {\n            \"indexing\": {\n                \"record\": \"basic\",\n                \"fieldnorms\": true,\n                \"tokenizer\": \"default\"\n            },\n            \"stored\": false\n        }\n        \"#;\n        let options: TextOptions = serde_json::from_str(json).unwrap();\n        let options2: TextOptions = serde_json::from_str(\"{\\\"indexing\\\": {}}\").unwrap();\n        assert_eq!(options, options2);\n        assert_eq!(options.indexing.unwrap().record, IndexRecordOption::Basic);\n        let options3: TextOptions = serde_json::from_str(\"{}\").unwrap();\n        assert_eq!(options3.indexing, None);\n    }\n\n    #[test]\n    fn serde_fast_field_tokenizer() {\n        let json = r#\" {\n            \"fast\": { \"with_tokenizer\": \"default\" }\n        } \"#;\n        let options: TextOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(\n            options.fast,\n            FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: TokenizerName::from_static(\"default\")\n            }\n        );\n        let options: TextOptions =\n            serde_json::from_str(&serde_json::to_string(&options).unwrap()).unwrap();\n        assert_eq!(\n            options.fast,\n            FastFieldTextOptions::EnabledWithTokenizer {\n                with_tokenizer: TokenizerName::from_static(\"default\")\n            }\n        );\n\n        let json = r#\" {\n            \"fast\": true\n        } \"#;\n        let options: TextOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(options.fast, FastFieldTextOptions::IsEnabled(true));\n        let options: TextOptions =\n            serde_json::from_str(&serde_json::to_string(&options).unwrap()).unwrap();\n        assert_eq!(options.fast, FastFieldTextOptions::IsEnabled(true));\n\n        let json = r#\" {\n            \"fast\": false\n        } \"#;\n        let options: TextOptions = serde_json::from_str(json).unwrap();\n        assert_eq!(options.fast, FastFieldTextOptions::IsEnabled(false));\n        let options: TextOptions =\n            serde_json::from_str(&serde_json::to_string(&options).unwrap()).unwrap();\n        assert_eq!(options.fast, FastFieldTextOptions::IsEnabled(false));\n    }\n}\n"
  },
  {
    "path": "src/snippet/mod.rs",
    "content": "//! [`SnippetGenerator`]\n//! Generates a text snippet for a given document, and some highlighted parts inside it.\n//!\n//! Imagine you doing a text search in a document\n//! and want to show a preview of where in the document the search terms occur,\n//! along with some surrounding text to give context, and the search terms highlighted.\n//!\n//! [`SnippetGenerator`] serves this purpose.\n//! It scans a document and constructs a snippet, which consists of sections where the search terms\n//! have been found, stitched together with \"...\" in between sections if necessary.\n//!\n//! ## Example\n//!\n//! ```rust\n//! # use tantivy::query::QueryParser;\n//! # use tantivy::schema::{Schema, TEXT};\n//! # use tantivy::{doc, Index};\n//! use tantivy::snippet::SnippetGenerator;\n//!\n//! # fn main() -> tantivy::Result<()> {\n//! #    let mut schema_builder = Schema::builder();\n//! #    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n//! #    let schema = schema_builder.build();\n//! #    let index = Index::create_in_ram(schema);\n//! #    let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n//! #    let doc = doc!(text_field => r#\"Comme je descendais des Fleuves impassibles,\n//! #   Je ne me sentis plus guidé par les haleurs :\n//! #  Des Peaux-Rouges criards les avaient pris pour cibles,\n//! #  Les ayant cloués nus aux poteaux de couleurs.\n//! #\n//! #  J'étais insoucieux de tous les équipages,\n//! #  Porteur de blés flamands ou de cotons anglais.\n//! #  Quand avec mes haleurs ont fini ces tapages,\n//! #  Les Fleuves m'ont laissé descendre où je voulais.\n//! #  \"#);\n//! #    index_writer.add_document(doc.clone())?;\n//! #    index_writer.commit()?;\n//! #    let query_parser = QueryParser::for_index(&index, vec![text_field]);\n//! // ...\n//! let query = query_parser.parse_query(\"haleurs flamands\").unwrap();\n//! # let reader = index.reader()?;\n//! # let searcher = reader.searcher();\n//! let mut snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n//! snippet_generator.set_max_num_chars(100);\n//! let snippet = snippet_generator.snippet_from_doc(&doc);\n//! let snippet_html: String = snippet.to_html();\n//! assert_eq!(snippet_html, \"Comme je descendais des Fleuves impassibles,\\n  Je ne me sentis plus guidé par les <b>haleurs</b> :\\n Des\");\n//! #    Ok(())\n//! # }\n//! ```\n//!\n//! You can also specify the maximum number of characters for the snippets generated with the\n//! `set_max_num_chars` method. By default, this limit is set to 150.\n//!\n//! SnippetGenerator needs to be created from the `Searcher` and the query, and the field on which\n//! the `SnippetGenerator` should generate the snippets.\n\nuse std::cmp::Ordering;\nuse std::collections::{BTreeMap, BTreeSet};\nuse std::ops::Range;\n\nuse htmlescape::encode_minimal;\n\nuse crate::query::Query;\nuse crate::schema::document::{Document, Value};\nuse crate::schema::Field;\nuse crate::tokenizer::{TextAnalyzer, Token};\nuse crate::{Score, Searcher, Term};\n\nconst DEFAULT_MAX_NUM_CHARS: usize = 150;\n\nconst DEFAULT_SNIPPET_PREFIX: &str = \"<b>\";\nconst DEFAULT_SNIPPET_POSTFIX: &str = \"</b>\";\n\n#[derive(Debug)]\npub(crate) struct FragmentCandidate {\n    score: Score,\n    start_offset: usize,\n    stop_offset: usize,\n    highlighted: Vec<Range<usize>>,\n}\n\nimpl FragmentCandidate {\n    /// Create a basic `FragmentCandidate`\n    ///\n    /// `score`, `num_chars` are set to 0\n    /// and `highlighted` is set to empty vec\n    /// stop_offset is set to start_offset, which is taken as a param.\n    fn new(start_offset: usize) -> FragmentCandidate {\n        FragmentCandidate {\n            score: 0.0,\n            start_offset,\n            stop_offset: start_offset,\n            highlighted: vec![],\n        }\n    }\n\n    /// Updates `score` and `highlighted` fields of the objects.\n    ///\n    /// taking the token and terms, the token is added to the fragment.\n    /// if the token is one of the terms, the score\n    /// and highlighted fields are updated in the fragment.\n    fn try_add_token(&mut self, token: &Token, terms: &BTreeMap<String, Score>) {\n        self.stop_offset = token.offset_to;\n\n        if let Some(&score) = terms.get(&token.text.to_lowercase()) {\n            self.score += score;\n            self.highlighted.push(token.offset_from..token.offset_to);\n        }\n    }\n}\n\n/// `Snippet`\n/// Contains a fragment of a document, and some highlighted parts inside it.\n#[derive(Debug)]\npub struct Snippet {\n    fragment: String,\n    highlighted: Vec<Range<usize>>,\n    snippet_prefix: String,\n    snippet_postfix: String,\n}\n\nimpl Snippet {\n    /// Create a new `Snippet`.\n    fn new(fragment: &str, highlighted: Vec<Range<usize>>) -> Self {\n        Self {\n            fragment: fragment.to_string(),\n            highlighted,\n            snippet_prefix: DEFAULT_SNIPPET_PREFIX.to_string(),\n            snippet_postfix: DEFAULT_SNIPPET_POSTFIX.to_string(),\n        }\n    }\n\n    /// Create a new, empty, `Snippet`.\n    pub fn empty() -> Snippet {\n        Snippet {\n            fragment: String::new(),\n            highlighted: Vec::new(),\n            snippet_prefix: String::new(),\n            snippet_postfix: String::new(),\n        }\n    }\n\n    /// Returns `true` if the snippet is empty.\n    pub fn is_empty(&self) -> bool {\n        self.highlighted.len() == 0\n    }\n\n    /// Returns a highlighted html from the `Snippet`.\n    pub fn to_html(&self) -> String {\n        let mut html = String::new();\n        let mut start_from: usize = 0;\n\n        for item in collapse_overlapped_ranges(&self.highlighted) {\n            html.push_str(&encode_minimal(&self.fragment[start_from..item.start]));\n            html.push_str(&self.snippet_prefix);\n            html.push_str(&encode_minimal(&self.fragment[item.clone()]));\n            html.push_str(&self.snippet_postfix);\n            start_from = item.end;\n        }\n        html.push_str(&encode_minimal(\n            &self.fragment[start_from..self.fragment.len()],\n        ));\n        html\n    }\n\n    /// Returns the fragment of text used in the  snippet.\n    pub fn fragment(&self) -> &str {\n        &self.fragment\n    }\n\n    /// Returns a list of highlighted positions from the `Snippet`.\n    pub fn highlighted(&self) -> &[Range<usize>] {\n        &self.highlighted\n    }\n\n    /// Sets highlighted prefix and postfix.\n    pub fn set_snippet_prefix_postfix(&mut self, prefix: &str, postfix: &str) {\n        self.snippet_prefix = prefix.to_string();\n        self.snippet_postfix = postfix.to_string()\n    }\n}\n\n/// Returns a non-empty list of \"good\" fragments.\n///\n/// If no target term is within the text, then the function\n/// should return an empty Vec.\n///\n/// If a target term is within the text, then the returned\n/// list is required to be non-empty.\n///\n/// The returned list is non-empty and contain less\n/// than 12 possibly overlapping fragments.\n///\n/// All fragments should contain at least one target term\n/// and have at most `max_num_chars` characters (not bytes).\n///\n/// It is ok to emit non-overlapping fragments, for instance,\n/// one short and one long containing the same keyword, in order\n/// to leave optimization opportunity to the fragment selector\n/// upstream.\n///\n/// Fragments must be valid in the sense that `&text[fragment.start..fragment.stop]`\\\n/// has to be a valid string.\nfn search_fragments(\n    tokenizer: &mut TextAnalyzer,\n    text: &str,\n    terms: &BTreeMap<String, Score>,\n    max_num_chars: usize,\n) -> Vec<FragmentCandidate> {\n    let mut token_stream = tokenizer.token_stream(text);\n    let mut fragment = FragmentCandidate::new(0);\n    let mut fragments: Vec<FragmentCandidate> = vec![];\n    while let Some(next) = token_stream.next() {\n        if (next.offset_to - fragment.start_offset) > max_num_chars {\n            if fragment.score > 0.0 {\n                fragments.push(fragment)\n            };\n            fragment = FragmentCandidate::new(next.offset_from);\n        }\n        fragment.try_add_token(next, terms);\n    }\n    if fragment.score > 0.0 {\n        fragments.push(fragment)\n    }\n\n    fragments\n}\n\n/// Returns a Snippet\n///\n/// Takes a vector of `FragmentCandidate`s and the text.\n/// Figures out the best fragment from it and creates a snippet.\nfn select_best_fragment_combination(fragments: &[FragmentCandidate], text: &str) -> Snippet {\n    let best_fragment_opt = fragments.iter().max_by(|left, right| {\n        let cmp_score = left\n            .score\n            .partial_cmp(&right.score)\n            .unwrap_or(Ordering::Equal);\n        if cmp_score == Ordering::Equal {\n            (right.start_offset, right.stop_offset).cmp(&(left.start_offset, left.stop_offset))\n        } else {\n            cmp_score\n        }\n    });\n    if let Some(fragment) = best_fragment_opt {\n        let fragment_text = &text[fragment.start_offset..fragment.stop_offset];\n        let highlighted = fragment\n            .highlighted\n            .iter()\n            .map(|item| item.start - fragment.start_offset..item.end - fragment.start_offset)\n            .collect();\n        Snippet::new(fragment_text, highlighted)\n    } else {\n        // When there are no fragments to chose from,\n        // for now create an empty snippet.\n        Snippet::empty()\n    }\n}\n\n/// Sorts and removes duplicate ranges from the input.\n///\n/// This function first sorts the ranges by their start position,\n/// then by their end position, and finally removes any duplicate ranges.\n///\n/// ## Examples\n/// - [0..3, 3..6, 0..3, 3..6] -> [0..3, 3..6]\n/// - [2..4, 1..3, 2..4, 0..2] -> [0..2, 1..3, 2..4]\nfn sort_and_deduplicate_ranges(ranges: &[Range<usize>]) -> Vec<Range<usize>> {\n    let mut sorted_ranges = ranges.to_vec();\n    sorted_ranges.sort_by_key(|range| (range.start, range.end));\n    sorted_ranges.dedup();\n    sorted_ranges\n}\n\n/// Merges overlapping or adjacent ranges into non-overlapping ranges.\n///\n/// This function assumes that the input ranges are already sorted\n/// and deduplicated. Use `sort_and_deduplicate_ranges` before calling\n/// this function if the input might contain unsorted or duplicate ranges.\n///\n/// ## Examples\n/// - [0..1, 2..3] -> [0..1, 2..3]  # no overlap\n/// - [0..1, 1..2] -> [0..2]  # adjacent, merged\n/// - [0..2, 1..3] -> [0..3]  # overlapping, merged\n/// - [0..3, 1..2] -> [0..3]  # second range is completely within the first\nfn merge_overlapping_ranges(ranges: &[Range<usize>]) -> Vec<Range<usize>> {\n    debug_assert!(is_sorted(ranges.iter().map(|range| range.start)));\n    let mut result = Vec::<Range<usize>>::new();\n    for range in ranges {\n        if let Some(last) = result.last_mut() {\n            if last.end > range.start {\n                // Only merge when there is a true overlap.\n                last.end = std::cmp::max(last.end, range.end);\n            } else {\n                // Do not overlap or only adjacent, add new scope.\n                result.push(range.clone());\n            }\n        } else {\n            // The first range\n            result.push(range.clone());\n        }\n    }\n    result\n}\n\n/// Collapses ranges into non-overlapped ranges.\n///\n/// This function first sorts and deduplicates the input ranges,\n/// then merges any overlapping or adjacent ranges.\n///\n/// ## Examples\n/// - [0..1, 2..3] -> [0..1, 2..3]  # no overlap\n/// - [0..1, 1..2] -> [0..2]  # adjacent, merged\n/// - [0..2, 1..3] -> [0..3]  # overlapping, merged\n/// - [0..3, 1..2] -> [0..3]  # second range is completely within the first\n/// - [0..3, 3..6, 0..3, 3..6] -> [0..6]  # duplicates removed, then merged\npub fn collapse_overlapped_ranges(ranges: &[Range<usize>]) -> Vec<Range<usize>> {\n    let prepared = sort_and_deduplicate_ranges(ranges);\n    merge_overlapping_ranges(&prepared)\n}\n\nfn is_sorted(mut it: impl Iterator<Item = usize>) -> bool {\n    if let Some(first) = it.next() {\n        let mut prev = first;\n        for item in it {\n            if item < prev {\n                return false;\n            }\n            prev = item;\n        }\n    }\n    true\n}\n\n/// `SnippetGenerator`\n///\n/// # Example\n///\n/// ```rust\n/// # use tantivy::query::QueryParser;\n/// # use tantivy::schema::{Schema, TEXT};\n/// # use tantivy::{doc, Index};\n/// use tantivy::snippet::SnippetGenerator;\n///\n/// # fn main() -> tantivy::Result<()> {\n/// #    let mut schema_builder = Schema::builder();\n/// #    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n/// #    let schema = schema_builder.build();\n/// #    let index = Index::create_in_ram(schema);\n/// #    let mut index_writer = index.writer_with_num_threads(1, 20_000_000)?;\n/// #    let doc = doc!(text_field => r#\"Comme je descendais des Fleuves impassibles,\n/// #   Je ne me sentis plus guidé par les haleurs :\n/// #  Des Peaux-Rouges criards les avaient pris pour cibles,\n/// #  Les ayant cloués nus aux poteaux de couleurs.\n/// #\n/// #  J'étais insoucieux de tous les équipages,\n/// #  Porteur de blés flamands ou de cotons anglais.\n/// #  Quand avec mes haleurs ont fini ces tapages,\n/// #  Les Fleuves m'ont laissé descendre où je voulais.\n/// #  \"#);\n/// #    index_writer.add_document(doc.clone())?;\n/// #    index_writer.commit()?;\n/// #    let query_parser = QueryParser::for_index(&index, vec![text_field]);\n/// // ...\n/// let query = query_parser.parse_query(\"haleurs flamands\").unwrap();\n/// # let reader = index.reader()?;\n/// # let searcher = reader.searcher();\n/// let mut snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n/// snippet_generator.set_max_num_chars(100);\n/// let snippet = snippet_generator.snippet_from_doc(&doc);\n/// let snippet_html: String = snippet.to_html();\n/// assert_eq!(snippet_html, \"Comme je descendais des Fleuves impassibles,\\n  Je ne me sentis plus guidé par les <b>haleurs</b> :\\n Des\");\n/// #    Ok(())\n/// # }\n/// ```\npub struct SnippetGenerator {\n    terms_text: BTreeMap<String, Score>,\n    tokenizer: TextAnalyzer,\n    field: Field,\n    max_num_chars: usize,\n}\n\nimpl SnippetGenerator {\n    /// Creates a new snippet generator\n    pub fn new(\n        terms_text: BTreeMap<String, Score>,\n        tokenizer: TextAnalyzer,\n        field: Field,\n        max_num_chars: usize,\n    ) -> Self {\n        SnippetGenerator {\n            terms_text,\n            tokenizer,\n            field,\n            max_num_chars,\n        }\n    }\n    /// Creates a new snippet generator\n    pub fn create(\n        searcher: &Searcher,\n        query: &dyn Query,\n        field: Field,\n    ) -> crate::Result<SnippetGenerator> {\n        let mut terms: BTreeSet<&Term> = BTreeSet::new();\n        query.query_terms(&mut |term, _| {\n            if term.field() == field {\n                terms.insert(term);\n            }\n        });\n        let mut terms_text: BTreeMap<String, Score> = Default::default();\n        for term in terms {\n            let term_value = term.value();\n            let term_str = if let Some(term_str) = term_value.as_str() {\n                term_str\n            } else {\n                continue;\n            };\n            let doc_freq = searcher.doc_freq(term)?;\n            if doc_freq > 0 {\n                let score = 1.0 / (1.0 + doc_freq as Score);\n                terms_text.insert(term_str.to_string(), score);\n            }\n        }\n        let tokenizer = searcher.index().tokenizer_for_field(field)?;\n        Ok(SnippetGenerator {\n            terms_text,\n            tokenizer,\n            field,\n            max_num_chars: DEFAULT_MAX_NUM_CHARS,\n        })\n    }\n\n    /// Sets a maximum number of chars. Default is 150.\n    pub fn set_max_num_chars(&mut self, max_num_chars: usize) {\n        self.max_num_chars = max_num_chars;\n    }\n\n    #[cfg(test)]\n    pub(crate) fn terms_text(&self) -> &BTreeMap<String, Score> {\n        &self.terms_text\n    }\n\n    /// Generates a snippet for the given `Document`.\n    ///\n    /// This method extract the text associated with the `SnippetGenerator`'s field\n    /// and computes a snippet.\n    pub fn snippet_from_doc<D: Document>(&self, doc: &D) -> Snippet {\n        let mut text = String::new();\n        for (field, value) in doc.iter_fields_and_values() {\n            let value = value as D::Value<'_>;\n            if field != self.field {\n                continue;\n            }\n\n            if let Some(val) = value.as_str() {\n                text.push(' ');\n                text.push_str(val);\n            }\n        }\n\n        self.snippet(text.trim())\n    }\n\n    /// Generates a snippet for the given text.\n    pub fn snippet(&self, text: &str) -> Snippet {\n        let fragment_candidates = search_fragments(\n            &mut self.tokenizer.clone(),\n            text,\n            &self.terms_text,\n            self.max_num_chars,\n        );\n        select_best_fragment_combination(&fragment_candidates[..], text)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::collections::BTreeMap;\n    use std::ops::Range;\n\n    use maplit::btreemap;\n\n    use super::{collapse_overlapped_ranges, search_fragments, select_best_fragment_combination};\n    use crate::query::QueryParser;\n    use crate::schema::{Schema, TEXT};\n    use crate::snippet::SnippetGenerator;\n    use crate::tokenizer::{NgramTokenizer, SimpleTokenizer};\n    use crate::Index;\n\n    const TEST_TEXT: &str = r#\"Rust is a systems programming language sponsored by\nMozilla which describes it as a \"safe, concurrent, practical language\", supporting functional and\nimperative-procedural paradigms. Rust is syntactically similar to C++[according to whom?],\nbut its designers intend it to provide better memory safety while still maintaining\nperformance.\n\nRust is free and open-source software, released under an MIT License, or Apache License\n2.0. Its designers have refined the language through the experiences of writing the Servo\nweb browser layout engine[14] and the Rust compiler. A large proportion of current commits\nto the project are from community members.[15]\n\nRust won first place for \"most loved programming language\" in the Stack Overflow Developer\nSurvey in 2016, 2017, and 2018.\"#;\n\n    #[test]\n    fn test_snippet() {\n        let terms = btreemap! {\n            String::from(\"rust\") => 1.0,\n            String::from(\"language\") => 0.9\n        };\n        let fragments = search_fragments(\n            &mut From::from(SimpleTokenizer::default()),\n            TEST_TEXT,\n            &terms,\n            100,\n        );\n        assert_eq!(fragments.len(), 7);\n        {\n            let first = &fragments[0];\n            assert_eq!(first.score, 1.9);\n            assert_eq!(first.stop_offset, 89);\n        }\n        let snippet = select_best_fragment_combination(&fragments[..], TEST_TEXT);\n        assert_eq!(\n            snippet.fragment,\n            \"Rust is a systems programming language sponsored by\\nMozilla which describes it as a \\\n             \\\"safe\"\n        );\n        assert_eq!(\n            snippet.to_html(),\n            \"<b>Rust</b> is a systems programming <b>language</b> sponsored by\\nMozilla which \\\n             describes it as a &quot;safe\"\n        )\n    }\n\n    #[test]\n    fn test_snippet_scored_fragment() {\n        {\n            let terms = btreemap! {\n                String::from(\"rust\") =>1.0,\n                String::from(\"language\") => 0.9\n            };\n            let fragments = search_fragments(\n                &mut From::from(SimpleTokenizer::default()),\n                TEST_TEXT,\n                &terms,\n                20,\n            );\n            {\n                let first = &fragments[0];\n                assert_eq!(first.score, 1.0);\n                assert_eq!(first.stop_offset, 17);\n            }\n            let snippet = select_best_fragment_combination(&fragments[..], TEST_TEXT);\n            assert_eq!(snippet.to_html(), \"<b>Rust</b> is a systems\")\n        }\n        {\n            let terms = btreemap! {\n                String::from(\"rust\") =>0.9,\n                String::from(\"language\") => 1.0\n            };\n            let fragments = search_fragments(\n                &mut From::from(SimpleTokenizer::default()),\n                TEST_TEXT,\n                &terms,\n                20,\n            );\n            // assert_eq!(fragments.len(), 7);\n            {\n                let first = &fragments[0];\n                assert_eq!(first.score, 0.9);\n                assert_eq!(first.stop_offset, 17);\n            }\n            let snippet = select_best_fragment_combination(&fragments[..], TEST_TEXT);\n            assert_eq!(snippet.to_html(), \"programming <b>language</b>\")\n        }\n    }\n\n    #[test]\n    fn test_snippet_in_second_fragment() {\n        let text = \"a b c d e f g\";\n\n        let mut terms = BTreeMap::new();\n        terms.insert(String::from(\"c\"), 1.0);\n\n        let fragments =\n            search_fragments(&mut From::from(SimpleTokenizer::default()), text, &terms, 3);\n\n        assert_eq!(fragments.len(), 1);\n        {\n            let first = &fragments[0];\n            assert_eq!(first.score, 1.0);\n            assert_eq!(first.start_offset, 4);\n            assert_eq!(first.stop_offset, 7);\n        }\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"c d\");\n        assert_eq!(snippet.to_html(), \"<b>c</b> d\");\n    }\n\n    #[test]\n    fn test_snippet_with_term_at_the_end_of_fragment() {\n        let text = \"a b c d e f f g\";\n\n        let mut terms = BTreeMap::new();\n        terms.insert(String::from(\"f\"), 1.0);\n\n        let fragments =\n            search_fragments(&mut From::from(SimpleTokenizer::default()), text, &terms, 3);\n\n        assert_eq!(fragments.len(), 2);\n        {\n            let first = &fragments[0];\n            assert_eq!(first.score, 1.0);\n            assert_eq!(first.stop_offset, 11);\n            assert_eq!(first.start_offset, 8);\n        }\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"e f\");\n        assert_eq!(snippet.to_html(), \"e <b>f</b>\");\n    }\n\n    #[test]\n    fn test_snippet_with_second_fragment_has_the_highest_score() {\n        let text = \"a b c d e f g\";\n\n        let mut terms = BTreeMap::new();\n        terms.insert(String::from(\"f\"), 1.0);\n        terms.insert(String::from(\"a\"), 0.9);\n\n        let fragments =\n            search_fragments(&mut From::from(SimpleTokenizer::default()), text, &terms, 7);\n\n        assert_eq!(fragments.len(), 2);\n        {\n            let first = &fragments[0];\n            assert_eq!(first.score, 0.9);\n            assert_eq!(first.stop_offset, 7);\n            assert_eq!(first.start_offset, 0);\n        }\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"e f g\");\n        assert_eq!(snippet.to_html(), \"e <b>f</b> g\");\n    }\n\n    #[test]\n    fn test_snippet_with_term_not_in_text() {\n        let text = \"a b c d\";\n\n        let mut terms = BTreeMap::new();\n        terms.insert(String::from(\"z\"), 1.0);\n\n        let fragments =\n            search_fragments(&mut From::from(SimpleTokenizer::default()), text, &terms, 3);\n\n        assert_eq!(fragments.len(), 0);\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"\");\n        assert_eq!(snippet.to_html(), \"\");\n        assert!(snippet.is_empty());\n    }\n\n    #[test]\n    fn test_snippet_with_no_terms() {\n        let text = \"a b c d\";\n\n        let terms = BTreeMap::new();\n        let fragments =\n            search_fragments(&mut From::from(SimpleTokenizer::default()), text, &terms, 3);\n        assert_eq!(fragments.len(), 0);\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"\");\n        assert_eq!(snippet.to_html(), \"\");\n        assert!(snippet.is_empty());\n    }\n\n    #[test]\n    fn test_snippet_generator_term_score() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let text_field = schema_builder.add_text_field(\"text\", TEXT);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n            index_writer.add_document(doc!(text_field => \"a\"))?;\n            index_writer.add_document(doc!(text_field => \"a b\"))?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader()?.searcher();\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        {\n            let query = query_parser.parse_query(\"e\")?;\n            let snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n            assert!(snippet_generator.terms_text().is_empty());\n        }\n        {\n            let query = query_parser.parse_query(\"a\")?;\n            let snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n            assert_eq!(\n                &btreemap!(\"a\".to_string() => 0.25),\n                snippet_generator.terms_text()\n            );\n        }\n        {\n            let query = query_parser.parse_query(\"a b\")?;\n            let snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n            assert_eq!(\n                &btreemap!(\"a\".to_string() => 0.25, \"b\".to_string() => 0.5),\n                snippet_generator.terms_text()\n            );\n        }\n        {\n            let query = query_parser.parse_query(\"a b c\")?;\n            let snippet_generator = SnippetGenerator::create(&searcher, &*query, text_field)?;\n            assert_eq!(\n                &btreemap!(\"a\".to_string() => 0.25, \"b\".to_string() => 0.5),\n                snippet_generator.terms_text()\n            );\n        }\n        Ok(())\n    }\n\n    #[cfg(feature = \"stemmer\")]\n    #[test]\n    fn test_snippet_generator() -> crate::Result<()> {\n        use crate::schema::{IndexRecordOption, TextFieldIndexing, TextOptions};\n        let mut schema_builder = Schema::builder();\n        let text_options = TextOptions::default().set_indexing_options(\n            TextFieldIndexing::default()\n                .set_tokenizer(\"en_stem\")\n                .set_index_option(IndexRecordOption::Basic),\n        );\n        let text_field = schema_builder.add_text_field(\"text\", text_options);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        {\n            // writing the segment\n            let mut index_writer = index.writer_for_tests()?;\n            let doc = doc!(text_field => TEST_TEXT);\n            index_writer.add_document(doc)?;\n            index_writer.commit()?;\n        }\n        let searcher = index.reader().unwrap().searcher();\n        let query_parser = QueryParser::for_index(&index, vec![text_field]);\n        let query = query_parser.parse_query(\"rust design\").unwrap();\n        let mut snippet_generator =\n            SnippetGenerator::create(&searcher, &*query, text_field).unwrap();\n        {\n            let snippet = snippet_generator.snippet(TEST_TEXT);\n            assert_eq!(\n                snippet.to_html(),\n                \"imperative-procedural paradigms. <b>Rust</b> is syntactically similar to \\\n                 C++[according to whom?],\\nbut its <b>designers</b> intend it to provide better \\\n                 memory safety\"\n            );\n        }\n        {\n            snippet_generator.set_max_num_chars(90);\n            let snippet = snippet_generator.snippet(TEST_TEXT);\n            assert_eq!(\n                snippet.to_html(),\n                \"<b>Rust</b> is syntactically similar to C++[according to whom?],\\nbut its \\\n                 <b>designers</b> intend it to\"\n            );\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_snippet_with_overlapped_highlighted_ranges() {\n        let text = \"abc\";\n\n        let mut terms = BTreeMap::new();\n        terms.insert(String::from(\"ab\"), 0.9);\n        terms.insert(String::from(\"bc\"), 1.0);\n\n        let fragments = search_fragments(\n            &mut From::from(NgramTokenizer::all_ngrams(2, 2).unwrap()),\n            text,\n            &terms,\n            3,\n        );\n\n        assert_eq!(fragments.len(), 1);\n        {\n            let first = &fragments[0];\n            assert_eq!(first.score, 1.9);\n            assert_eq!(first.start_offset, 0);\n            assert_eq!(first.stop_offset, 3);\n        }\n\n        let snippet = select_best_fragment_combination(&fragments[..], text);\n        assert_eq!(snippet.fragment, \"abc\");\n        assert_eq!(snippet.to_html(), \"<b>abc</b>\");\n    }\n\n    #[test]\n    fn test_snippet_generator_custom_highlighted_elements() {\n        let terms = btreemap! { String::from(\"rust\") => 1.0, String::from(\"language\") => 0.9 };\n        let fragments = search_fragments(\n            &mut From::from(SimpleTokenizer::default()),\n            TEST_TEXT,\n            &terms,\n            100,\n        );\n        let mut snippet = select_best_fragment_combination(&fragments[..], TEST_TEXT);\n        assert_eq!(\n            snippet.to_html(),\n            \"<b>Rust</b> is a systems programming <b>language</b> sponsored by\\nMozilla which \\\n             describes it as a &quot;safe\"\n        );\n        snippet.set_snippet_prefix_postfix(\"<q class=\\\"super\\\">\", \"</q>\");\n        assert_eq!(\n            snippet.to_html(),\n            \"<q class=\\\"super\\\">Rust</q> is a systems programming <q class=\\\"super\\\">language</q> \\\n             sponsored by\\nMozilla which describes it as a &quot;safe\"\n        );\n    }\n\n    #[test]\n    fn test_collapse_overlapped_ranges() {\n        #![allow(clippy::single_range_in_vec_init)]\n        assert_eq!(&collapse_overlapped_ranges(&[0..1, 2..3]), &[0..1, 2..3]);\n        assert_eq!(&collapse_overlapped_ranges(&[0..1, 1..2]), &[0..1, 1..2]);\n        assert_eq!(&collapse_overlapped_ranges(&[0..2, 1..2]), &[0..2]);\n        assert_eq!(&collapse_overlapped_ranges(&[0..2, 1..3]), &[0..3]);\n        assert_eq!(&collapse_overlapped_ranges(&[0..3, 1..2]), &[0..3]);\n    }\n\n    #[test]\n    fn test_no_overlap() {\n        let ranges = vec![0..1, 2..3, 4..5];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..1, 2..3, 4..5]);\n    }\n\n    #[test]\n    fn test_adjacent_ranges() {\n        let ranges = vec![0..1, 1..2, 2..3];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..1, 1..2, 2..3]);\n    }\n\n    #[test]\n    fn test_overlapping_ranges() {\n        let ranges = vec![0..2, 1..3, 2..4];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..4]);\n    }\n\n    #[test]\n    fn test_contained_ranges() {\n        let ranges = vec![0..5, 1..2, 3..4];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..5]);\n    }\n\n    #[test]\n    fn test_duplicate_ranges() {\n        let ranges = vec![0..2, 2..4, 0..2, 2..4];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..2, 2..4]);\n    }\n\n    #[test]\n    fn test_unsorted_ranges() {\n        let ranges = vec![2..4, 0..2, 1..3];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..4]);\n    }\n\n    #[test]\n    fn test_complex_scenario() {\n        let ranges = vec![0..2, 5..7, 1..3, 8..9, 2..4, 3..6, 8..10];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..7, 8..10]);\n    }\n\n    #[test]\n    fn test_empty_input() {\n        let ranges: Vec<Range<usize>> = vec![];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, ranges);\n    }\n\n    #[test]\n    fn test_single_range() {\n        #![allow(clippy::single_range_in_vec_init)]\n        let ranges = vec![0..5];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..5]);\n    }\n\n    #[test]\n    fn test_zero_length_ranges() {\n        let ranges = vec![0..0, 1..1, 2..2, 3..3];\n        let result = collapse_overlapped_ranges(&ranges);\n        assert_eq!(result, vec![0..0, 1..1, 2..2, 3..3]);\n    }\n}\n"
  },
  {
    "path": "src/space_usage/mod.rs",
    "content": "//! Representations for the space usage of various parts of a Tantivy index.\n//!\n//! This can be used programmatically, and will also be exposed in a human readable fashion in\n//! tantivy-cli.\n//!\n//! One important caveat for all of this functionality is that none of it currently takes\n//! storage-level details into consideration. For example, if your file system block size is 4096\n//! bytes, we can under-count actual resultant space usage by up to 4095 bytes per file.\n\nuse std::collections::btree_map::Entry;\nuse std::collections::BTreeMap;\n\nuse columnar::ColumnSpaceUsage;\nuse common::ByteCount;\nuse serde::{Deserialize, Serialize};\n\nuse crate::index::SegmentComponent;\n\n/// Enum containing any of the possible space usage results for segment components.\npub enum ComponentSpaceUsage {\n    /// Data is stored per field in a uniform way\n    PerField(PerFieldSpaceUsage),\n    /// Data is stored in separate pieces in the store\n    Store(StoreSpaceUsage),\n    /// Some sort of raw byte count\n    Basic(ByteCount),\n}\n\n/// Represents combined space usage of an entire searcher and its component segments.\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct SearcherSpaceUsage {\n    segments: Vec<SegmentSpaceUsage>,\n    total: ByteCount,\n}\n\nimpl SearcherSpaceUsage {\n    pub(crate) fn new() -> SearcherSpaceUsage {\n        SearcherSpaceUsage {\n            segments: Vec::new(),\n            total: Default::default(),\n        }\n    }\n\n    /// Add a segment, to `self`.\n    /// Performs no deduplication or other intelligence.\n    pub(crate) fn add_segment(&mut self, segment: SegmentSpaceUsage) {\n        self.total += segment.total();\n        self.segments.push(segment);\n    }\n\n    /// Per segment space usage\n    pub fn segments(&self) -> &[SegmentSpaceUsage] {\n        &self.segments[..]\n    }\n\n    /// Returns total byte usage of this searcher, including all large subcomponents.\n    /// Does not account for smaller things like `meta.json`.\n    pub fn total(&self) -> ByteCount {\n        self.total\n    }\n}\n\n/// Represents combined space usage for all of the large components comprising a segment.\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct SegmentSpaceUsage {\n    num_docs: u32,\n\n    termdict: PerFieldSpaceUsage,\n    postings: PerFieldSpaceUsage,\n    positions: PerFieldSpaceUsage,\n    fast_fields: PerFieldSpaceUsage,\n    fieldnorms: PerFieldSpaceUsage,\n\n    store: StoreSpaceUsage,\n\n    deletes: ByteCount,\n\n    total: ByteCount,\n}\n\nimpl SegmentSpaceUsage {\n    #[expect(clippy::too_many_arguments)]\n    pub(crate) fn new(\n        num_docs: u32,\n        termdict: PerFieldSpaceUsage,\n        postings: PerFieldSpaceUsage,\n        positions: PerFieldSpaceUsage,\n        fast_fields: PerFieldSpaceUsage,\n        fieldnorms: PerFieldSpaceUsage,\n        store: StoreSpaceUsage,\n        deletes: ByteCount,\n    ) -> SegmentSpaceUsage {\n        let total = termdict.total()\n            + postings.total()\n            + positions.total()\n            + fast_fields.total()\n            + fieldnorms.total()\n            + store.total()\n            + deletes;\n        SegmentSpaceUsage {\n            num_docs,\n            termdict,\n            postings,\n            positions,\n            fast_fields,\n            fieldnorms,\n            store,\n            deletes,\n            total,\n        }\n    }\n\n    /// Space usage for the given component\n    ///\n    /// Clones the underlying data.\n    /// Use the components directly if this is somehow in performance critical code.\n    pub fn component(&self, component: SegmentComponent) -> ComponentSpaceUsage {\n        use self::ComponentSpaceUsage::*;\n        use crate::index::SegmentComponent::*;\n        match component {\n            Postings => PerField(self.postings().clone()),\n            Positions => PerField(self.positions().clone()),\n            FastFields => PerField(self.fast_fields().clone()),\n            FieldNorms => PerField(self.fieldnorms().clone()),\n            Terms => PerField(self.termdict().clone()),\n            SegmentComponent::Store => ComponentSpaceUsage::Store(self.store().clone()),\n            Delete => Basic(self.deletes()),\n        }\n    }\n\n    /// Num docs in segment\n    pub fn num_docs(&self) -> u32 {\n        self.num_docs\n    }\n\n    /// Space usage for term dictionary\n    pub fn termdict(&self) -> &PerFieldSpaceUsage {\n        &self.termdict\n    }\n\n    /// Space usage for postings list\n    pub fn postings(&self) -> &PerFieldSpaceUsage {\n        &self.postings\n    }\n\n    /// Space usage for positions\n    pub fn positions(&self) -> &PerFieldSpaceUsage {\n        &self.positions\n    }\n\n    /// Space usage for fast fields\n    pub fn fast_fields(&self) -> &PerFieldSpaceUsage {\n        &self.fast_fields\n    }\n\n    /// Space usage for field norms\n    pub fn fieldnorms(&self) -> &PerFieldSpaceUsage {\n        &self.fieldnorms\n    }\n\n    /// Space usage for stored documents\n    pub fn store(&self) -> &StoreSpaceUsage {\n        &self.store\n    }\n\n    /// Space usage for document deletions\n    pub fn deletes(&self) -> ByteCount {\n        self.deletes\n    }\n\n    /// Total space usage in bytes for this segment.\n    pub fn total(&self) -> ByteCount {\n        self.total\n    }\n}\n\n/// Represents space usage for the Store for this segment.\n///\n/// This is composed of two parts.\n/// `data` represents the compressed data itself.\n/// `offsets` represents a lookup to find the start of a block\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct StoreSpaceUsage {\n    data: ByteCount,\n    offsets: ByteCount,\n}\n\nimpl StoreSpaceUsage {\n    pub(crate) fn new(data: ByteCount, offsets: ByteCount) -> StoreSpaceUsage {\n        StoreSpaceUsage { data, offsets }\n    }\n\n    /// Space usage for the data part of the store\n    pub fn data_usage(&self) -> ByteCount {\n        self.data\n    }\n\n    /// Space usage for the offsets part of the store (doc ID -> offset)\n    pub fn offsets_usage(&self) -> ByteCount {\n        self.offsets\n    }\n\n    /// Total space usage in bytes for this Store\n    pub fn total(&self) -> ByteCount {\n        self.data + self.offsets\n    }\n}\n\n/// Represents space usage for all of the (field, index) pairs that appear in a `CompositeFile`.\n///\n/// A field can appear with a single index (typically 0) or with multiple indexes.\n/// Multiple indexes are used to handle variable length things, where\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct PerFieldSpaceUsage {\n    fields: BTreeMap<String, FieldUsage>,\n    total: ByteCount,\n}\n\nimpl PerFieldSpaceUsage {\n    pub(crate) fn new(fields: Vec<FieldUsage>) -> PerFieldSpaceUsage {\n        let mut total = ByteCount::default();\n        let mut field_usage_map: BTreeMap<String, FieldUsage> = BTreeMap::new();\n        for field_usage in fields {\n            total += field_usage.total();\n            let field_name = field_usage.field_name().to_string();\n            match field_usage_map.entry(field_name) {\n                Entry::Vacant(entry) => {\n                    entry.insert(field_usage);\n                }\n                Entry::Occupied(mut entry) => {\n                    entry.get_mut().merge(field_usage);\n                }\n            }\n        }\n        PerFieldSpaceUsage {\n            fields: field_usage_map,\n            total,\n        }\n    }\n\n    /// Per field space usage\n    pub fn fields(&self) -> impl Iterator<Item = &FieldUsage> {\n        self.fields.values()\n    }\n\n    /// Bytes used by the represented file\n    pub fn total(&self) -> ByteCount {\n        self.total\n    }\n}\n\n/// Represents space usage of a given field, breaking it down into the (field, index) pairs that\n/// comprise it.\n///\n/// See documentation for [`PerFieldSpaceUsage`] for slightly more information.\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct FieldUsage {\n    field_name: String,\n    num_bytes: ByteCount,\n    /// A field can be composed of more than one piece.\n    /// These pieces are indexed by arbitrary numbers starting at zero.\n    /// `self.num_bytes` includes all of `self.sub_num_bytes`.\n    sub_num_bytes: Vec<Option<ByteCount>>,\n    /// Space usage of the column for fast fields, if relevant.\n    column_space_usage: Option<ColumnSpaceUsage>,\n}\n\nimpl FieldUsage {\n    pub(crate) fn empty(field_name: impl Into<String>) -> FieldUsage {\n        FieldUsage {\n            field_name: field_name.into(),\n            num_bytes: Default::default(),\n            sub_num_bytes: Vec::new(),\n            column_space_usage: None,\n        }\n    }\n\n    pub(crate) fn add_field_idx(&mut self, idx: usize, size: ByteCount) {\n        if self.sub_num_bytes.len() < idx + 1 {\n            self.sub_num_bytes.resize(idx + 1, None);\n        }\n        assert!(self.sub_num_bytes[idx].is_none());\n        self.sub_num_bytes[idx] = Some(size);\n        self.num_bytes += size\n    }\n\n    pub(crate) fn set_column_usage(&mut self, column_space_usage: ColumnSpaceUsage) {\n        self.num_bytes += column_space_usage.total_num_bytes();\n        self.column_space_usage = Some(column_space_usage);\n    }\n\n    /// Field\n    pub fn field_name(&self) -> &str {\n        &self.field_name\n    }\n\n    /// Space usage for each index\n    pub fn sub_num_bytes(&self) -> &[Option<ByteCount>] {\n        &self.sub_num_bytes[..]\n    }\n\n    /// Returns the number of bytes used by the column payload, if the field is columnar.\n    pub fn column_num_bytes(&self) -> Option<ByteCount> {\n        self.column_space_usage\n            .as_ref()\n            .map(ColumnSpaceUsage::column_num_bytes)\n    }\n\n    /// Returns the number of bytes used by the dictionary for dictionary-encoded columns.\n    pub fn dictionary_num_bytes(&self) -> Option<ByteCount> {\n        self.column_space_usage\n            .as_ref()\n            .and_then(ColumnSpaceUsage::dictionary_num_bytes)\n    }\n\n    /// Returns the space usage of the column, if any.\n    pub fn column_space_usage(&self) -> Option<&ColumnSpaceUsage> {\n        self.column_space_usage.as_ref()\n    }\n\n    /// Total bytes used for this field in this context\n    pub fn total(&self) -> ByteCount {\n        self.num_bytes\n    }\n\n    fn merge(&mut self, other: FieldUsage) {\n        assert_eq!(self.field_name, other.field_name);\n        self.num_bytes += other.num_bytes;\n        if other.sub_num_bytes.len() > self.sub_num_bytes.len() {\n            self.sub_num_bytes.resize(other.sub_num_bytes.len(), None);\n        }\n        for (idx, num_bytes_opt) in other.sub_num_bytes.into_iter().enumerate() {\n            if let Some(num_bytes) = num_bytes_opt {\n                match self.sub_num_bytes[idx] {\n                    Some(existing) => self.sub_num_bytes[idx] = Some(existing + num_bytes),\n                    None => self.sub_num_bytes[idx] = Some(num_bytes),\n                }\n            }\n        }\n        self.column_space_usage =\n            merge_column_space_usage(self.column_space_usage.take(), other.column_space_usage);\n    }\n}\n\nfn merge_column_space_usage(\n    left: Option<ColumnSpaceUsage>,\n    right: Option<ColumnSpaceUsage>,\n) -> Option<ColumnSpaceUsage> {\n    match (left, right) {\n        (Some(lhs), Some(rhs)) => Some(lhs.merge(&rhs)),\n        (Some(space), None) | (None, Some(space)) => Some(space),\n        (None, None) => None,\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use crate::index::Index;\n    use crate::schema::{Schema, FAST, INDEXED, STORED, TEXT};\n    use crate::space_usage::PerFieldSpaceUsage;\n    use crate::{IndexWriter, Term};\n\n    #[test]\n    fn test_empty() {\n        let schema = Schema::builder().build();\n        let index = Index::create_in_ram(schema);\n        let reader = index.reader().unwrap();\n        let searcher = reader.searcher();\n        let searcher_space_usage = searcher.space_usage().unwrap();\n        assert_eq!(searcher_space_usage.total(), 0u64);\n    }\n\n    fn expect_single_field(\n        field_space: &PerFieldSpaceUsage,\n        field: &str,\n        min_size: u64,\n        max_size: u64,\n    ) {\n        assert!(field_space.total() >= min_size);\n        assert!(field_space.total() <= max_size);\n        assert_eq!(\n            vec![(field.to_string(), field_space.total())],\n            field_space\n                .fields()\n                .map(|usage| (usage.field_name().to_string(), usage.total()))\n                .collect::<Vec<_>>()\n        );\n    }\n\n    #[test]\n    fn test_fast_indexed() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name = schema_builder.add_u64_field(\"name\", FAST | INDEXED);\n        let schema = schema_builder.build();\n        let field_name = schema.get_field_name(name).to_string();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => 1u64))?;\n            index_writer.add_document(doc!(name => 2u64))?;\n            index_writer.add_document(doc!(name => 10u64))?;\n            index_writer.add_document(doc!(name => 20u64))?;\n            index_writer.commit()?;\n        }\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let searcher_space_usage = searcher.space_usage()?;\n        assert!(searcher_space_usage.total() > 0);\n        assert_eq!(1, searcher_space_usage.segments().len());\n\n        let segment = &searcher_space_usage.segments()[0];\n        assert!(segment.total() > 0);\n\n        assert_eq!(4, segment.num_docs());\n\n        expect_single_field(segment.termdict(), &field_name, 1, 512);\n        expect_single_field(segment.postings(), &field_name, 1, 512);\n        assert_eq!(segment.positions().total(), 0);\n        expect_single_field(segment.fast_fields(), &field_name, 1, 512);\n        expect_single_field(segment.fieldnorms(), &field_name, 1, 512);\n        // TODO: understand why the following fails\n        //        assert_eq!(0, segment.store().total());\n        assert_eq!(segment.deletes(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_text() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name = schema_builder.add_text_field(\"name\", TEXT);\n        let schema = schema_builder.build();\n        let field_name = schema.get_field_name(name).to_string();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => \"hi\"))?;\n            index_writer.add_document(doc!(name => \"this is a test\"))?;\n            index_writer.add_document(\n                doc!(name => \"some more documents with some word overlap with the other test\"),\n            )?;\n            index_writer.add_document(doc!(name => \"hello hi goodbye\"))?;\n            index_writer.commit()?;\n        }\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let searcher_space_usage = searcher.space_usage()?;\n        assert!(searcher_space_usage.total() > 0);\n        assert_eq!(1, searcher_space_usage.segments().len());\n\n        let segment = &searcher_space_usage.segments()[0];\n        assert!(segment.total() > 0);\n\n        assert_eq!(4, segment.num_docs());\n\n        expect_single_field(segment.termdict(), &field_name, 1, 512);\n        expect_single_field(segment.postings(), &field_name, 1, 512);\n        expect_single_field(segment.positions(), &field_name, 1, 512);\n        assert_eq!(segment.fast_fields().total(), 0);\n        expect_single_field(segment.fieldnorms(), &field_name, 1, 512);\n        // TODO: understand why the following fails\n        //        assert_eq!(0, segment.store().total());\n        assert_eq!(segment.deletes(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_store() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name = schema_builder.add_text_field(\"name\", STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => \"hi\"))?;\n            index_writer.add_document(doc!(name => \"this is a test\"))?;\n            index_writer.add_document(\n                doc!(name => \"some more documents with some word overlap with the other test\"),\n            )?;\n            index_writer.add_document(doc!(name => \"hello hi goodbye\"))?;\n            index_writer.commit()?;\n        }\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let searcher_space_usage = searcher.space_usage()?;\n        assert!(searcher_space_usage.total() > 0);\n        assert_eq!(1, searcher_space_usage.segments().len());\n\n        let segment = &searcher_space_usage.segments()[0];\n        assert!(segment.total() > 0);\n\n        assert_eq!(4, segment.num_docs());\n\n        assert_eq!(segment.termdict().total(), 0);\n        assert!(segment.termdict().fields().next().is_none());\n        assert_eq!(segment.postings().total(), 0);\n        assert!(segment.postings().fields().next().is_none());\n        assert_eq!(segment.positions().total(), 0);\n        assert!(segment.positions().fields().next().is_none());\n        assert_eq!(segment.fast_fields().total(), 0);\n        assert!(segment.fast_fields().fields().next().is_none());\n        assert_eq!(segment.fieldnorms().total(), 0);\n        assert!(segment.fieldnorms().fields().next().is_none());\n        assert!(segment.store().total() > 0);\n        assert!(segment.store().total() < 512);\n        assert_eq!(segment.deletes(), 0);\n        Ok(())\n    }\n\n    #[test]\n    fn test_deletes() -> crate::Result<()> {\n        let mut schema_builder = Schema::builder();\n        let name = schema_builder.add_u64_field(\"name\", INDEXED);\n        let schema = schema_builder.build();\n        let field_name = schema.get_field_name(name).to_string();\n        let index = Index::create_in_ram(schema);\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.add_document(doc!(name => 1u64))?;\n            index_writer.add_document(doc!(name => 2u64))?;\n            index_writer.add_document(doc!(name => 3u64))?;\n            index_writer.add_document(doc!(name => 4u64))?;\n            index_writer.commit()?;\n        }\n\n        {\n            let mut index_writer2: IndexWriter = index.writer(50_000_000)?;\n            index_writer2.delete_term(Term::from_field_u64(name, 2u64));\n            index_writer2.delete_term(Term::from_field_u64(name, 3u64));\n            // ok, now we should have a deleted doc\n            index_writer2.commit()?;\n        }\n\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        let searcher_space_usage = searcher.space_usage()?;\n        assert!(searcher_space_usage.total() > 0);\n        assert_eq!(1, searcher_space_usage.segments().len());\n\n        let segment_space_usage = &searcher_space_usage.segments()[0];\n        assert!(segment_space_usage.total() > 0);\n\n        assert_eq!(2, segment_space_usage.num_docs());\n\n        expect_single_field(segment_space_usage.termdict(), &field_name, 1, 512);\n        expect_single_field(segment_space_usage.postings(), &field_name, 1, 512);\n        assert_eq!(segment_space_usage.positions().total(), 0u64);\n        assert_eq!(segment_space_usage.fast_fields().total(), 0u64);\n        expect_single_field(segment_space_usage.fieldnorms(), &field_name, 1, 512);\n        assert!(segment_space_usage.deletes() > 0);\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/store/compression_lz4_block.rs",
    "content": "use std::io::{self};\nuse std::mem;\n\nuse lz4_flex::{compress_into, decompress_into};\n\n#[inline]\n#[expect(clippy::uninit_vec)]\npub fn compress(uncompressed: &[u8], compressed: &mut Vec<u8>) -> io::Result<()> {\n    compressed.clear();\n    let maximum_output_size =\n        mem::size_of::<u32>() + lz4_flex::block::get_maximum_output_size(uncompressed.len());\n    compressed.reserve(maximum_output_size);\n    unsafe {\n        compressed.set_len(maximum_output_size);\n    }\n    let bytes_written = compress_into(uncompressed, &mut compressed[4..])\n        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err.to_string()))?;\n    let num_bytes = uncompressed.len() as u32;\n    compressed[0..4].copy_from_slice(&num_bytes.to_le_bytes());\n    unsafe {\n        compressed.set_len(bytes_written + mem::size_of::<u32>());\n    }\n    Ok(())\n}\n\n#[inline]\n#[expect(clippy::uninit_vec)]\npub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {\n    decompressed.clear();\n    let uncompressed_size_bytes: &[u8; 4] = compressed\n        .get(..4)\n        .ok_or(io::ErrorKind::InvalidData)?\n        .try_into()\n        .unwrap();\n    let uncompressed_size = u32::from_le_bytes(*uncompressed_size_bytes) as usize;\n    decompressed.reserve(uncompressed_size);\n    unsafe {\n        decompressed.set_len(uncompressed_size);\n    }\n    let bytes_written = decompress_into(&compressed[4..], decompressed)\n        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err.to_string()))?;\n    if bytes_written != uncompressed_size {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"doc store block not completely decompressed, data corruption\".to_string(),\n        ));\n    }\n    Ok(())\n}\n"
  },
  {
    "path": "src/store/compression_zstd_block.rs",
    "content": "use std::io;\n\nuse zstd::bulk::{compress_to_buffer, decompress_to_buffer};\nuse zstd::DEFAULT_COMPRESSION_LEVEL;\n\n#[inline]\npub fn compress(\n    uncompressed: &[u8],\n    compressed: &mut Vec<u8>,\n    compression_level: Option<i32>,\n) -> io::Result<()> {\n    let count_size = std::mem::size_of::<u32>();\n    let max_size = zstd::zstd_safe::compress_bound(uncompressed.len()) + count_size;\n\n    compressed.clear();\n    compressed.resize(max_size, 0);\n\n    let compressed_size = compress_to_buffer(\n        uncompressed,\n        &mut compressed[count_size..],\n        compression_level.unwrap_or(DEFAULT_COMPRESSION_LEVEL),\n    )?;\n\n    compressed[0..count_size].copy_from_slice(&(uncompressed.len() as u32).to_le_bytes());\n    compressed.resize(compressed_size + count_size, 0);\n\n    Ok(())\n}\n\n#[inline]\npub fn decompress(compressed: &[u8], decompressed: &mut Vec<u8>) -> io::Result<()> {\n    let count_size = std::mem::size_of::<u32>();\n    let uncompressed_size = u32::from_le_bytes(\n        compressed\n            .get(..count_size)\n            .ok_or(io::ErrorKind::InvalidData)?\n            .try_into()\n            .unwrap(),\n    ) as usize;\n\n    decompressed.clear();\n    decompressed.resize(uncompressed_size, 0);\n\n    let decompressed_size = decompress_to_buffer(&compressed[count_size..], decompressed)?;\n\n    if decompressed_size != uncompressed_size {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"doc store block not completely decompressed, data corruption\".to_string(),\n        ));\n    }\n\n    Ok(())\n}\n"
  },
  {
    "path": "src/store/compressors.rs",
    "content": "use std::io;\n\nuse serde::{Deserialize, Deserializer, Serialize};\n\n/// Compressor can be used on `IndexSettings` to choose\n/// the compressor used to compress the doc store.\n///\n/// The default is Lz4Block, but also depends on the enabled feature flags.\n#[derive(Clone, Debug, Copy, PartialEq, Eq)]\npub enum Compressor {\n    /// No compression\n    None,\n    /// Use the lz4 compressor (block format)\n    #[cfg(feature = \"lz4-compression\")]\n    Lz4,\n    /// Use the zstd compressor\n    #[cfg(feature = \"zstd-compression\")]\n    Zstd(ZstdCompressor),\n}\n\nimpl Serialize for Compressor {\n    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>\n    where S: serde::Serializer {\n        match *self {\n            Compressor::None => serializer.serialize_str(\"none\"),\n            #[cfg(feature = \"lz4-compression\")]\n            Compressor::Lz4 => serializer.serialize_str(\"lz4\"),\n            #[cfg(feature = \"zstd-compression\")]\n            Compressor::Zstd(zstd) => serializer.serialize_str(&zstd.ser_to_string()),\n        }\n    }\n}\n\nimpl<'de> Deserialize<'de> for Compressor {\n    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>\n    where D: Deserializer<'de> {\n        let buf = String::deserialize(deserializer)?;\n        let compressor = match buf.as_str() {\n            \"none\" => Compressor::None,\n            #[cfg(feature = \"lz4-compression\")]\n            \"lz4\" => Compressor::Lz4,\n            #[cfg(not(feature = \"lz4-compression\"))]\n            \"lz4\" => {\n                return Err(serde::de::Error::custom(\n                    \"unsupported variant `lz4`, please enable Tantivy's `lz4-compression` feature\",\n                ))\n            }\n            #[cfg(feature = \"zstd-compression\")]\n            _ if buf.starts_with(\"zstd\") => Compressor::Zstd(\n                ZstdCompressor::deser_from_str(&buf).map_err(serde::de::Error::custom)?,\n            ),\n            #[cfg(not(feature = \"zstd-compression\"))]\n            _ if buf.starts_with(\"zstd\") => {\n                return Err(serde::de::Error::custom(\n                    \"unsupported variant `zstd`, please enable Tantivy's `zstd-compression` \\\n                     feature\",\n                ))\n            }\n            _ => {\n                return Err(serde::de::Error::unknown_variant(\n                    &buf,\n                    &[\n                        \"none\",\n                        #[cfg(feature = \"lz4-compression\")]\n                        \"lz4\",\n                        #[cfg(feature = \"zstd-compression\")]\n                        \"zstd\",\n                        #[cfg(feature = \"zstd-compression\")]\n                        \"zstd(compression_level=5)\",\n                    ],\n                ));\n            }\n        };\n\n        Ok(compressor)\n    }\n}\n\n#[derive(Clone, Default, Debug, Copy, PartialEq, Eq, Serialize, Deserialize)]\n/// The Zstd compressor, with optional compression level.\npub struct ZstdCompressor {\n    /// The compression level, if unset defaults to zstd::DEFAULT_COMPRESSION_LEVEL = 3\n    pub compression_level: Option<i32>,\n}\n\n#[cfg(feature = \"zstd-compression\")]\nimpl ZstdCompressor {\n    fn deser_from_str(val: &str) -> Result<ZstdCompressor, String> {\n        if !val.starts_with(\"zstd\") {\n            return Err(format!(\"needs to start with zstd, but got {val}\"));\n        }\n        if val == \"zstd\" {\n            return Ok(ZstdCompressor::default());\n        }\n        let options = &val[\"zstd\".len() + 1..val.len() - 1];\n\n        let mut compressor = ZstdCompressor::default();\n        for option in options.split(',') {\n            let (opt_name, value) = options\n                .split_once('=')\n                .ok_or_else(|| format!(\"no '=' found in option {option:?}\"))?;\n\n            match opt_name {\n                \"compression_level\" => {\n                    let value = value.parse::<i32>().map_err(|err| {\n                        format!(\"Could not parse value {value} of option {opt_name}, e: {err}\")\n                    })?;\n                    if value >= 15 {\n                        warn!(\n                            \"High zstd compression level detected: {:?}. High compression levels \\\n                             (>=15) are slow and will limit indexing speed.\",\n                            value\n                        )\n                    }\n                    compressor.compression_level = Some(value);\n                }\n                _ => {\n                    return Err(format!(\"unknown zstd option {opt_name:?}\"));\n                }\n            }\n        }\n        Ok(compressor)\n    }\n    fn ser_to_string(&self) -> String {\n        if let Some(compression_level) = self.compression_level {\n            format!(\"zstd(compression_level={compression_level})\")\n        } else {\n            \"zstd\".to_string()\n        }\n    }\n}\n\nimpl Default for Compressor {\n    #[allow(unreachable_code)]\n    fn default() -> Self {\n        #[cfg(feature = \"lz4-compression\")]\n        return Compressor::Lz4;\n\n        #[cfg(feature = \"zstd-compression\")]\n        return Compressor::Zstd(ZstdCompressor::default());\n\n        Compressor::None\n    }\n}\n\nimpl Compressor {\n    #[inline]\n    pub(crate) fn compress_into(\n        &self,\n        uncompressed: &[u8],\n        compressed: &mut Vec<u8>,\n    ) -> io::Result<()> {\n        match self {\n            Self::None => {\n                compressed.clear();\n                compressed.extend_from_slice(uncompressed);\n                Ok(())\n            }\n            #[cfg(feature = \"lz4-compression\")]\n            Self::Lz4 => super::compression_lz4_block::compress(uncompressed, compressed),\n            #[cfg(feature = \"zstd-compression\")]\n            Self::Zstd(_zstd_compressor) => super::compression_zstd_block::compress(\n                uncompressed,\n                compressed,\n                _zstd_compressor.compression_level,\n            ),\n        }\n    }\n}\n\n#[cfg(all(feature = \"zstd-compression\", test))]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn zstd_serde_roundtrip() {\n        let compressor = ZstdCompressor {\n            compression_level: Some(15),\n        };\n\n        assert_eq!(\n            ZstdCompressor::deser_from_str(&compressor.ser_to_string()).unwrap(),\n            compressor\n        );\n\n        assert_eq!(\n            ZstdCompressor::deser_from_str(&ZstdCompressor::default().ser_to_string()).unwrap(),\n            ZstdCompressor::default()\n        );\n    }\n\n    #[test]\n    fn deser_zstd_test() {\n        assert_eq!(\n            ZstdCompressor::deser_from_str(\"zstd\").unwrap(),\n            ZstdCompressor::default()\n        );\n\n        assert!(ZstdCompressor::deser_from_str(\"zzstd\").is_err());\n        assert!(ZstdCompressor::deser_from_str(\"zzstd()\").is_err());\n        assert_eq!(\n            ZstdCompressor::deser_from_str(\"zstd(compression_level=15)\").unwrap(),\n            ZstdCompressor {\n                compression_level: Some(15)\n            }\n        );\n        assert_eq!(\n            ZstdCompressor::deser_from_str(\"zstd(compresion_level=15)\").unwrap_err(),\n            \"unknown zstd option \\\"compresion_level\\\"\"\n        );\n        assert_eq!(\n            ZstdCompressor::deser_from_str(\"zstd(compression_level->2)\").unwrap_err(),\n            \"no '=' found in option \\\"compression_level->2\\\"\"\n        );\n        assert_eq!(\n            ZstdCompressor::deser_from_str(\"zstd(compression_level=over9000)\").unwrap_err(),\n            \"Could not parse value over9000 of option compression_level, e: invalid digit found \\\n             in string\"\n        );\n    }\n}\n"
  },
  {
    "path": "src/store/decompressors.rs",
    "content": "use std::io;\n\nuse serde::{Deserialize, Serialize};\n\nuse super::Compressor;\n\n/// Decompressor is deserialized from the doc store footer, when opening an index.\n#[derive(Clone, Debug, Copy, PartialEq, Eq, Serialize, Deserialize)]\npub enum Decompressor {\n    /// No compression\n    None,\n    /// Use the lz4 decompressor (block format)\n    #[cfg(feature = \"lz4-compression\")]\n    Lz4,\n    /// Use the zstd decompressor\n    #[cfg(feature = \"zstd-compression\")]\n    Zstd,\n}\n\nimpl From<Compressor> for Decompressor {\n    fn from(compressor: Compressor) -> Self {\n        match compressor {\n            Compressor::None => Decompressor::None,\n            #[cfg(feature = \"lz4-compression\")]\n            Compressor::Lz4 => Decompressor::Lz4,\n            #[cfg(feature = \"zstd-compression\")]\n            Compressor::Zstd(_) => Decompressor::Zstd,\n        }\n    }\n}\n\nimpl Decompressor {\n    pub(crate) fn from_id(id: u8) -> Decompressor {\n        match id {\n            0 => Decompressor::None,\n            #[cfg(feature = \"lz4-compression\")]\n            1 => Decompressor::Lz4,\n            #[cfg(feature = \"zstd-compression\")]\n            4 => Decompressor::Zstd,\n            _ => panic!(\"unknown compressor id {id:?}\"),\n        }\n    }\n\n    pub(crate) fn get_id(&self) -> u8 {\n        match self {\n            Self::None => 0,\n            #[cfg(feature = \"lz4-compression\")]\n            Self::Lz4 => 1,\n            #[cfg(feature = \"zstd-compression\")]\n            Self::Zstd => 4,\n        }\n    }\n\n    pub(crate) fn decompress(&self, compressed_block: &[u8]) -> io::Result<Vec<u8>> {\n        let mut decompressed_block = vec![];\n        self.decompress_into(compressed_block, &mut decompressed_block)?;\n        Ok(decompressed_block)\n    }\n\n    #[inline]\n    pub(crate) fn decompress_into(\n        &self,\n        compressed: &[u8],\n        decompressed: &mut Vec<u8>,\n    ) -> io::Result<()> {\n        match self {\n            Self::None => {\n                decompressed.clear();\n                decompressed.extend_from_slice(compressed);\n                Ok(())\n            }\n            #[cfg(feature = \"lz4-compression\")]\n            Self::Lz4 => super::compression_lz4_block::decompress(compressed, decompressed),\n            #[cfg(feature = \"zstd-compression\")]\n            Self::Zstd => super::compression_zstd_block::decompress(compressed, decompressed),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn compressor_decompressor_id_test() {\n        assert_eq!(Decompressor::from(Compressor::None), Decompressor::None);\n        #[cfg(feature = \"lz4-compression\")]\n        assert_eq!(Decompressor::from(Compressor::Lz4), Decompressor::Lz4);\n        #[cfg(feature = \"zstd-compression\")]\n        assert_eq!(\n            Decompressor::from(Compressor::Zstd(Default::default())),\n            Decompressor::Zstd\n        );\n    }\n}\n"
  },
  {
    "path": "src/store/footer.rs",
    "content": "use std::io;\n\nuse common::{BinarySerializable, FixedSize, HasLen};\n\nuse super::{Decompressor, DocStoreVersion, DOC_STORE_VERSION};\nuse crate::directory::FileSlice;\n\n#[derive(Debug, Clone, PartialEq)]\npub struct DocStoreFooter {\n    pub offset: u64,\n    pub doc_store_version: DocStoreVersion,\n    pub decompressor: Decompressor,\n}\n\n/// Serialises the footer to a byte-array\n/// - offset : 8 bytes\n/// - compressor id: 1 byte\n/// - reserved for future use: 15 bytes\nimpl BinarySerializable for DocStoreFooter {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        BinarySerializable::serialize(&DOC_STORE_VERSION, writer)?;\n        BinarySerializable::serialize(&self.offset, writer)?;\n        BinarySerializable::serialize(&self.decompressor.get_id(), writer)?;\n        writer.write_all(&[0; 15])?;\n        Ok(())\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        let doc_store_version = DocStoreVersion::deserialize(reader)?;\n        if doc_store_version > DOC_STORE_VERSION {\n            panic!(\n                \"actual doc store version: {doc_store_version}, max_supported: {DOC_STORE_VERSION}\"\n            );\n        }\n        let offset = u64::deserialize(reader)?;\n        let compressor_id = u8::deserialize(reader)?;\n        let mut skip_buf = [0; 15];\n        reader.read_exact(&mut skip_buf)?;\n        Ok(DocStoreFooter {\n            offset,\n            doc_store_version,\n            decompressor: Decompressor::from_id(compressor_id),\n        })\n    }\n}\n\nimpl FixedSize for DocStoreFooter {\n    const SIZE_IN_BYTES: usize = 28;\n}\n\nimpl DocStoreFooter {\n    pub fn new(\n        offset: u64,\n        decompressor: Decompressor,\n        doc_store_version: DocStoreVersion,\n    ) -> Self {\n        DocStoreFooter {\n            offset,\n            doc_store_version,\n            decompressor,\n        }\n    }\n\n    pub fn extract_footer(file: FileSlice) -> io::Result<(DocStoreFooter, FileSlice)> {\n        if file.len() < DocStoreFooter::SIZE_IN_BYTES {\n            return Err(io::Error::new(\n                io::ErrorKind::UnexpectedEof,\n                format!(\n                    \"File corrupted. The file is smaller than Footer::SIZE_IN_BYTES (len={}).\",\n                    file.len()\n                ),\n            ));\n        }\n        let (body, footer_slice) = file.split_from_end(DocStoreFooter::SIZE_IN_BYTES);\n        let mut footer_bytes = footer_slice.read_bytes()?;\n        let footer = DocStoreFooter::deserialize(&mut footer_bytes)?;\n        Ok((footer, body))\n    }\n}\n\n#[test]\nfn doc_store_footer_test() {\n    // This test is just to safe guard changes on the footer.\n    // When the doc store footer is updated, make sure to update also the serialize/deserialize\n    // methods\n    assert_eq!(core::mem::size_of::<DocStoreFooter>(), 16);\n}\n"
  },
  {
    "path": "src/store/index/block.rs",
    "content": "use std::io;\nuse std::ops::Range;\n\nuse common::{read_u32_vint, VInt};\n\nuse crate::store::index::{Checkpoint, CHECKPOINT_PERIOD};\nuse crate::DocId;\n\n/// Represents a block of checkpoints.\n///\n/// The DocStore index checkpoints are organized into block\n/// for code-readability and compression purpose.\n///\n/// A block can be of any size.\npub struct CheckpointBlock {\n    pub checkpoints: Vec<Checkpoint>,\n}\n\nimpl Default for CheckpointBlock {\n    fn default() -> CheckpointBlock {\n        CheckpointBlock {\n            checkpoints: Vec::with_capacity(2 * CHECKPOINT_PERIOD),\n        }\n    }\n}\n\nimpl CheckpointBlock {\n    /// If non-empty returns [start_doc, end_doc)\n    /// for the overall block.\n    pub fn doc_interval(&self) -> Option<Range<DocId>> {\n        let start_doc_opt = self\n            .checkpoints\n            .first()\n            .cloned()\n            .map(|checkpoint| checkpoint.doc_range.start);\n        let end_doc_opt = self\n            .checkpoints\n            .last()\n            .cloned()\n            .map(|checkpoint| checkpoint.doc_range.end);\n        match (start_doc_opt, end_doc_opt) {\n            (Some(start_doc), Some(end_doc)) => Some(start_doc..end_doc),\n            _ => None,\n        }\n    }\n\n    /// Adding another checkpoint in the block.\n    pub fn push(&mut self, checkpoint: Checkpoint) {\n        if let Some(prev_checkpoint) = self.checkpoints.last() {\n            assert!(checkpoint.follows(prev_checkpoint));\n        }\n        self.checkpoints.push(checkpoint);\n    }\n\n    /// Returns the number of checkpoints in the block.\n    pub fn len(&self) -> usize {\n        self.checkpoints.len()\n    }\n\n    pub fn get(&self, idx: usize) -> Checkpoint {\n        self.checkpoints[idx].clone()\n    }\n\n    pub fn clear(&mut self) {\n        self.checkpoints.clear();\n    }\n\n    pub fn serialize(&mut self, buffer: &mut Vec<u8>) {\n        VInt(self.checkpoints.len() as u64).serialize_into_vec(buffer);\n        if self.checkpoints.is_empty() {\n            return;\n        }\n        VInt(self.checkpoints[0].doc_range.start as u64).serialize_into_vec(buffer);\n        VInt(self.checkpoints[0].byte_range.start as u64).serialize_into_vec(buffer);\n        for checkpoint in &self.checkpoints {\n            let delta_doc = checkpoint.doc_range.end - checkpoint.doc_range.start;\n            VInt(delta_doc as u64).serialize_into_vec(buffer);\n            VInt((checkpoint.byte_range.end - checkpoint.byte_range.start) as u64)\n                .serialize_into_vec(buffer);\n        }\n    }\n\n    pub fn deserialize(&mut self, data: &mut &[u8]) -> io::Result<()> {\n        if data.is_empty() {\n            return Err(io::Error::new(io::ErrorKind::UnexpectedEof, \"\"));\n        }\n        self.checkpoints.clear();\n        let len = read_u32_vint(data);\n        if len == 0 {\n            return Ok(());\n        }\n        let mut doc = read_u32_vint(data);\n        let mut start_offset = VInt::deserialize_u64(data)? as usize;\n        for _ in 0..len {\n            let num_docs = read_u32_vint(data);\n            let block_num_bytes = read_u32_vint(data) as usize;\n            self.checkpoints.push(Checkpoint {\n                doc_range: doc..doc + num_docs,\n                byte_range: start_offset..start_offset + block_num_bytes,\n            });\n            doc += num_docs;\n            start_offset += block_num_bytes;\n        }\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io;\n\n    use crate::store::index::block::CheckpointBlock;\n    use crate::store::index::Checkpoint;\n    use crate::DocId;\n\n    fn test_aux_ser_deser(checkpoints: &[Checkpoint]) -> io::Result<()> {\n        let mut block = CheckpointBlock::default();\n        for checkpoint in checkpoints {\n            block.push(checkpoint.clone());\n        }\n        let mut buffer = Vec::new();\n        block.serialize(&mut buffer);\n        let mut block_deser = CheckpointBlock::default();\n        let checkpoint = Checkpoint {\n            doc_range: 0..1,\n            byte_range: 2..3,\n        };\n        block_deser.push(checkpoint); // < check that value is erased before deser\n        let mut data = &buffer[..];\n        block_deser.deserialize(&mut data)?;\n        assert!(data.is_empty());\n        assert_eq!(checkpoints, &block_deser.checkpoints[..]);\n        Ok(())\n    }\n\n    #[test]\n    fn test_block_serialize_empty() -> io::Result<()> {\n        test_aux_ser_deser(&[])\n    }\n\n    #[test]\n    fn test_block_serialize_simple() -> io::Result<()> {\n        let checkpoints = vec![Checkpoint {\n            doc_range: 10..12,\n            byte_range: 100..120,\n        }];\n        test_aux_ser_deser(&checkpoints)\n    }\n\n    #[test]\n    fn test_block_serialize_large_byte_range() -> io::Result<()> {\n        let checkpoints = vec![Checkpoint {\n            doc_range: 10..12,\n            byte_range: 8_000_000_000..9_000_000_000,\n        }];\n        test_aux_ser_deser(&checkpoints)\n    }\n\n    #[test]\n    fn test_block_serialize() -> io::Result<()> {\n        let offsets: Vec<usize> = (0..11).map(|i| i * i * i).collect();\n        let mut checkpoints = vec![];\n        let mut start_doc = 0;\n        for i in 0..10 {\n            let end_doc = (i * i) as DocId;\n            checkpoints.push(Checkpoint {\n                doc_range: start_doc..end_doc,\n                byte_range: offsets[i]..offsets[i + 1],\n            });\n            start_doc = end_doc;\n        }\n        test_aux_ser_deser(&checkpoints)\n    }\n}\n"
  },
  {
    "path": "src/store/index/mod.rs",
    "content": "const CHECKPOINT_PERIOD: usize = 8;\n\nuse std::fmt;\nuse std::ops::Range;\nmod block;\nmod skip_index;\nmod skip_index_builder;\n\npub use self::skip_index::SkipIndex;\npub use self::skip_index_builder::SkipIndexBuilder;\nuse crate::DocId;\n\n/// A checkpoint contains meta-information about\n/// a block. Either a block of documents, or another block\n/// of checkpoints.\n///\n/// All of the intervals here defined are semi-open.\n/// The checkpoint describes that the block within the `byte_range`\n/// and spans over the `doc_range`.\n#[derive(Clone, Eq, PartialEq, Default)]\npub struct Checkpoint {\n    pub doc_range: Range<DocId>,\n    pub byte_range: Range<usize>,\n}\n\nimpl Checkpoint {\n    pub(crate) fn follows(&self, other: &Checkpoint) -> bool {\n        (self.doc_range.start == other.doc_range.end)\n            && (self.byte_range.start == other.byte_range.end)\n    }\n}\n\nimpl fmt::Debug for Checkpoint {\n    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {\n        write!(f, \"(doc={:?}, bytes={:?})\", self.doc_range, self.byte_range)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::io;\n\n    use proptest::prelude::*;\n\n    use super::{SkipIndex, SkipIndexBuilder};\n    use crate::directory::OwnedBytes;\n    use crate::indexer::NoMergePolicy;\n    use crate::schema::{SchemaBuilder, STORED, TEXT};\n    use crate::store::index::Checkpoint;\n    use crate::{DocAddress, DocId, Index, IndexWriter, TantivyDocument, Term};\n\n    #[test]\n    fn test_skip_index_empty() -> io::Result<()> {\n        let mut output: Vec<u8> = Vec::new();\n        let skip_index_builder: SkipIndexBuilder = SkipIndexBuilder::new();\n        skip_index_builder.serialize_into(&mut output)?;\n        let skip_index: SkipIndex = SkipIndex::open(OwnedBytes::new(output));\n        let mut skip_cursor = skip_index.checkpoints();\n        assert!(skip_cursor.next().is_none());\n        Ok(())\n    }\n\n    #[test]\n    fn test_skip_index_single_el() -> io::Result<()> {\n        let mut output: Vec<u8> = Vec::new();\n        let mut skip_index_builder: SkipIndexBuilder = SkipIndexBuilder::new();\n        let checkpoint = Checkpoint {\n            doc_range: 0..2,\n            byte_range: 0..3,\n        };\n        skip_index_builder.insert(checkpoint.clone());\n        skip_index_builder.serialize_into(&mut output)?;\n        let skip_index: SkipIndex = SkipIndex::open(OwnedBytes::new(output));\n        let mut skip_cursor = skip_index.checkpoints();\n        assert_eq!(skip_cursor.next(), Some(checkpoint));\n        assert_eq!(skip_cursor.next(), None);\n        Ok(())\n    }\n\n    #[test]\n    fn test_skip_index() -> io::Result<()> {\n        let mut output: Vec<u8> = Vec::new();\n        let checkpoints = vec![\n            Checkpoint {\n                doc_range: 0..3,\n                byte_range: 0..9,\n            },\n            Checkpoint {\n                doc_range: 3..4,\n                byte_range: 9..25,\n            },\n            Checkpoint {\n                doc_range: 4..6,\n                byte_range: 25..49,\n            },\n            Checkpoint {\n                doc_range: 6..8,\n                byte_range: 49..81,\n            },\n            Checkpoint {\n                doc_range: 8..10,\n                byte_range: 81..100,\n            },\n        ];\n\n        let mut skip_index_builder: SkipIndexBuilder = SkipIndexBuilder::new();\n        for checkpoint in &checkpoints {\n            skip_index_builder.insert(checkpoint.clone());\n        }\n        skip_index_builder.serialize_into(&mut output)?;\n\n        let skip_index: SkipIndex = SkipIndex::open(OwnedBytes::new(output));\n        assert_eq!(\n            &skip_index.checkpoints().collect::<Vec<_>>()[..],\n            &checkpoints[..]\n        );\n        Ok(())\n    }\n\n    fn offset_test(doc: DocId) -> usize {\n        (doc as usize) * (doc as usize)\n    }\n\n    #[test]\n    fn test_merge_store_with_stacking_reproducing_issue969() -> crate::Result<()> {\n        let mut schema_builder = SchemaBuilder::default();\n        let text = schema_builder.add_text_field(\"text\", STORED | TEXT);\n        let body = schema_builder.add_text_field(\"body\", STORED);\n        let schema = schema_builder.build();\n        let index = Index::create_in_ram(schema);\n        let mut index_writer: IndexWriter = index.writer_for_tests()?;\n        index_writer.set_merge_policy(Box::new(NoMergePolicy));\n        let long_text: String = \"abcdefghijklmnopqrstuvwxyz\".repeat(1_000);\n        for _ in 0..20 {\n            index_writer.add_document(doc!(body=>long_text.clone()))?;\n        }\n        index_writer.commit()?;\n        index_writer.add_document(doc!(text=>\"testb\"))?;\n        for _ in 0..10 {\n            index_writer.add_document(doc!(text=>\"testd\", body=>long_text.clone()))?;\n        }\n        index_writer.commit()?;\n        index_writer.delete_term(Term::from_field_text(text, \"testb\"));\n        index_writer.commit()?;\n        let segment_ids = index.searchable_segment_ids()?;\n        index_writer.merge(&segment_ids).wait().unwrap();\n        let reader = index.reader()?;\n        let searcher = reader.searcher();\n        assert_eq!(searcher.num_docs(), 30);\n        for i in 0..searcher.num_docs() as u32 {\n            let _doc = searcher.doc::<TantivyDocument>(DocAddress::new(0u32, i))?;\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_skip_index_long() -> io::Result<()> {\n        let mut output: Vec<u8> = Vec::new();\n        let checkpoints: Vec<Checkpoint> = (0..1000)\n            .map(|i| Checkpoint {\n                doc_range: i..(i + 1),\n                byte_range: offset_test(i)..offset_test(i + 1),\n            })\n            .collect();\n        let mut skip_index_builder = SkipIndexBuilder::new();\n        for checkpoint in &checkpoints {\n            skip_index_builder.insert(checkpoint.clone());\n        }\n        skip_index_builder.serialize_into(&mut output)?;\n        assert_eq!(output.len(), 4035);\n        let resulting_checkpoints: Vec<Checkpoint> = SkipIndex::open(OwnedBytes::new(output))\n            .checkpoints()\n            .collect();\n        assert_eq!(&resulting_checkpoints, &checkpoints);\n        Ok(())\n    }\n\n    fn integrate_delta(vals: Vec<usize>) -> Vec<usize> {\n        let mut output = Vec::with_capacity(vals.len() + 1);\n        output.push(0);\n        let mut prev = 0;\n        for val in vals {\n            let new_val = val + prev;\n            prev = new_val;\n            output.push(new_val);\n        }\n        output\n    }\n\n    // Generates a sequence of n valid checkpoints, with n < max_len.\n    fn monotonic_checkpoints(max_len: usize) -> BoxedStrategy<Vec<Checkpoint>> {\n        (0..max_len)\n            .prop_flat_map(move |len: usize| {\n                (\n                    proptest::collection::vec(1usize..20, len).prop_map(integrate_delta),\n                    proptest::collection::vec(1usize..26, len).prop_map(integrate_delta),\n                )\n                    .prop_map(|(docs, offsets)| {\n                        (0..docs.len() - 1)\n                            .map(move |i| Checkpoint {\n                                doc_range: docs[i] as DocId..docs[i + 1] as DocId,\n                                byte_range: offsets[i]..offsets[i + 1],\n                            })\n                            .collect::<Vec<Checkpoint>>()\n                    })\n            })\n            .boxed()\n    }\n\n    fn seek_manual<I: Iterator<Item = Checkpoint>>(\n        checkpoints: I,\n        target: DocId,\n    ) -> Option<Checkpoint> {\n        checkpoints\n            .into_iter()\n            .find(|checkpoint| checkpoint.doc_range.end > target)\n    }\n\n    fn test_skip_index_aux(skip_index: SkipIndex, checkpoints: &[Checkpoint]) {\n        if let Some(last_checkpoint) = checkpoints.last() {\n            for doc in 0u32..last_checkpoint.doc_range.end {\n                let expected = seek_manual(skip_index.checkpoints(), doc);\n                assert_eq!(expected, skip_index.seek(doc), \"Doc {doc}\");\n            }\n            assert!(skip_index.seek(last_checkpoint.doc_range.end).is_none());\n        }\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(20))]\n        #[test]\n        fn test_proptest_skip(checkpoints in monotonic_checkpoints(100)) {\n             let mut skip_index_builder = SkipIndexBuilder::new();\n             for checkpoint in checkpoints.iter().cloned() {\n                 skip_index_builder.insert(checkpoint);\n             }\n             let mut buffer = Vec::new();\n             skip_index_builder.serialize_into(&mut buffer).unwrap();\n             let skip_index = SkipIndex::open(OwnedBytes::new(buffer));\n             let iter_checkpoints: Vec<Checkpoint> = skip_index.checkpoints().collect();\n             assert_eq!(&checkpoints[..], &iter_checkpoints[..]);\n             test_skip_index_aux(skip_index, &checkpoints[..]);\n         }\n    }\n}\n"
  },
  {
    "path": "src/store/index/skip_index.rs",
    "content": "use common::{BinarySerializable, VInt};\n\nuse crate::directory::OwnedBytes;\nuse crate::store::index::block::CheckpointBlock;\nuse crate::store::index::Checkpoint;\nuse crate::DocId;\n\npub struct LayerCursor<'a> {\n    remaining: &'a [u8],\n    block: CheckpointBlock,\n    cursor: usize,\n}\n\nimpl Iterator for LayerCursor<'_> {\n    type Item = Checkpoint;\n\n    fn next(&mut self) -> Option<Checkpoint> {\n        if self.cursor == self.block.len() {\n            if self.remaining.is_empty() {\n                return None;\n            }\n            let (block_mut, remaining_mut) = (&mut self.block, &mut self.remaining);\n            block_mut.deserialize(remaining_mut).ok()?;\n            self.cursor = 0;\n        }\n        let res = Some(self.block.get(self.cursor));\n        self.cursor += 1;\n        res\n    }\n}\n\nstruct Layer {\n    data: OwnedBytes,\n}\n\nimpl Layer {\n    fn cursor(&self) -> impl Iterator<Item = Checkpoint> + '_ {\n        self.cursor_at_offset(0)\n    }\n\n    fn cursor_at_offset(&self, start_offset: usize) -> impl Iterator<Item = Checkpoint> + '_ {\n        let data = &self.data.as_slice();\n        LayerCursor {\n            remaining: &data[start_offset..],\n            block: CheckpointBlock::default(),\n            cursor: 0,\n        }\n    }\n\n    fn seek_start_at_offset(&self, target: DocId, offset: usize) -> Option<Checkpoint> {\n        self.cursor_at_offset(offset)\n            .find(|checkpoint| checkpoint.doc_range.end > target)\n    }\n}\n\npub struct SkipIndex {\n    layers: Vec<Layer>,\n}\n\nimpl SkipIndex {\n    pub fn open(mut data: OwnedBytes) -> SkipIndex {\n        let offsets: Vec<u64> = Vec::<VInt>::deserialize(&mut data)\n            .unwrap()\n            .into_iter()\n            .map(|el| el.0)\n            .collect();\n        let mut start_offset = 0;\n        let mut layers = Vec::new();\n        for end_offset in offsets {\n            let layer = Layer {\n                data: data.slice(start_offset as usize..end_offset as usize),\n            };\n            layers.push(layer);\n            start_offset = end_offset;\n        }\n        SkipIndex { layers }\n    }\n\n    pub(crate) fn checkpoints(&self) -> impl Iterator<Item = Checkpoint> + '_ {\n        self.layers\n            .last()\n            .into_iter()\n            .flat_map(|layer| layer.cursor())\n    }\n\n    pub fn seek(&self, target: DocId) -> Option<Checkpoint> {\n        let first_layer_len = self\n            .layers\n            .first()\n            .map(|layer| layer.data.len())\n            .unwrap_or(0);\n        let mut cur_checkpoint = Checkpoint {\n            doc_range: 0u32..1u32,\n            byte_range: 0..first_layer_len,\n        };\n        for layer in &self.layers {\n            if let Some(checkpoint) =\n                layer.seek_start_at_offset(target, cur_checkpoint.byte_range.start)\n            {\n                cur_checkpoint = checkpoint;\n            } else {\n                return None;\n            }\n        }\n        Some(cur_checkpoint)\n    }\n}\n"
  },
  {
    "path": "src/store/index/skip_index_builder.rs",
    "content": "use std::io;\nuse std::io::Write;\n\nuse common::{BinarySerializable, VInt};\n\nuse crate::store::index::block::CheckpointBlock;\nuse crate::store::index::{Checkpoint, CHECKPOINT_PERIOD};\n\n// Each skip contains iterator over pairs (last doc in block, offset to start of block).\n\nstruct LayerBuilder {\n    buffer: Vec<u8>,\n    pub block: CheckpointBlock,\n}\n\nimpl LayerBuilder {\n    fn finish(self) -> Vec<u8> {\n        self.buffer\n    }\n\n    fn new() -> LayerBuilder {\n        LayerBuilder {\n            buffer: Vec::new(),\n            block: CheckpointBlock::default(),\n        }\n    }\n\n    /// Serializes the block, and return a checkpoint representing\n    /// the entire block.\n    ///\n    /// If the block was empty to begin with, simply return `None`.\n    fn flush_block(&mut self) -> Option<Checkpoint> {\n        if let Some(doc_range) = self.block.doc_interval() {\n            let start_offset = self.buffer.len();\n            self.block.serialize(&mut self.buffer);\n            let end_offset = self.buffer.len();\n            self.block.clear();\n            Some(Checkpoint {\n                doc_range,\n                byte_range: start_offset..end_offset,\n            })\n        } else {\n            None\n        }\n    }\n\n    fn push(&mut self, checkpoint: Checkpoint) {\n        self.block.push(checkpoint);\n    }\n\n    fn insert(&mut self, checkpoint: Checkpoint) -> Option<Checkpoint> {\n        self.push(checkpoint);\n        let emit_skip_info = self.block.len() >= CHECKPOINT_PERIOD;\n        if emit_skip_info {\n            self.flush_block()\n        } else {\n            None\n        }\n    }\n}\n\npub struct SkipIndexBuilder {\n    layers: Vec<LayerBuilder>,\n}\n\nimpl SkipIndexBuilder {\n    pub fn new() -> SkipIndexBuilder {\n        SkipIndexBuilder { layers: Vec::new() }\n    }\n\n    fn get_layer(&mut self, layer_id: usize) -> &mut LayerBuilder {\n        if layer_id == self.layers.len() {\n            let layer_builder = LayerBuilder::new();\n            self.layers.push(layer_builder);\n        }\n        &mut self.layers[layer_id]\n    }\n\n    pub fn insert(&mut self, checkpoint: Checkpoint) {\n        let mut skip_pointer = Some(checkpoint);\n        for layer_id in 0.. {\n            if let Some(checkpoint) = skip_pointer {\n                skip_pointer = self.get_layer(layer_id).insert(checkpoint);\n            } else {\n                break;\n            }\n        }\n    }\n\n    pub fn serialize_into<W: Write>(mut self, output: &mut W) -> io::Result<()> {\n        let mut last_pointer = None;\n        for skip_layer in self.layers.iter_mut() {\n            if let Some(checkpoint) = last_pointer {\n                skip_layer.push(checkpoint);\n            }\n            last_pointer = skip_layer.flush_block();\n        }\n        let layer_buffers: Vec<Vec<u8>> = self\n            .layers\n            .into_iter()\n            .rev()\n            .map(|layer| layer.finish())\n            .collect();\n\n        let mut layer_offset = 0;\n        let mut layer_sizes = Vec::new();\n        for layer_buffer in &layer_buffers {\n            layer_offset += layer_buffer.len() as u64;\n            layer_sizes.push(VInt(layer_offset));\n        }\n        layer_sizes.serialize(output)?;\n        for layer_buffer in layer_buffers {\n            output.write_all(&layer_buffer[..])?;\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/store/mod.rs",
    "content": "//! Compressed/slow/row-oriented storage for documents.\n//!\n//! A field needs to be marked as stored in the schema in\n//! order to be handled in the `Store`.\n//!\n//! Internally, documents (or rather their stored fields) are serialized to a buffer.\n//! When the buffer exceeds `block_size` (defaults to 16K), the buffer is compressed\n//! using LZ4 or Zstd and the resulting block is written to disk.\n//!\n//! One can then request for a specific `DocId`.\n//! A skip list helps navigating to the right block,\n//! decompresses it entirely and returns the document within it.\n//!\n//! If the last document requested was in the same block,\n//! the reader is smart enough to avoid decompressing\n//! the block a second time, but their is no real\n//! uncompressed block* cache.\n//!\n//! A typical use case for the store is, once\n//! the search result page has been computed, returning\n//! the actual content of the 10 best document.\n//!\n//! # Usage\n//!\n//! Most users should not access the `StoreReader` directly\n//! and should rely on either\n//!\n//! - at the segment level, the [`SegmentReader`'s `doc`\n//!   method](../struct.SegmentReader.html#method.doc)\n//! - at the index level, the [`Searcher::doc()`](crate::Searcher::doc) method\n\nmod compressors;\nmod decompressors;\nmod footer;\nmod index;\nmod reader;\nmod writer;\n\npub use self::compressors::{Compressor, ZstdCompressor};\npub use self::decompressors::Decompressor;\npub use self::reader::{CacheStats, StoreReader};\npub(crate) use self::reader::{DocStoreVersion, DOCSTORE_CACHE_CAPACITY};\npub use self::writer::StoreWriter;\nmod store_compressor;\n\n/// Doc store version in footer to handle format changes.\npub(crate) const DOC_STORE_VERSION: DocStoreVersion = DocStoreVersion::V2;\n\n#[cfg(feature = \"lz4-compression\")]\nmod compression_lz4_block;\n\n#[cfg(feature = \"zstd-compression\")]\nmod compression_zstd_block;\n\n#[cfg(test)]\npub(crate) mod tests {\n\n    use std::path::Path;\n\n    use super::*;\n    use crate::directory::{Directory, RamDirectory, WritePtr};\n    use crate::fastfield::AliveBitSet;\n    use crate::schema::{\n        self, Schema, TantivyDocument, TextFieldIndexing, TextOptions, Value, STORED, TEXT,\n    };\n    use crate::{Index, IndexWriter, Term};\n\n    const LOREM: &str = \"Doc Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do \\\n                         eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad \\\n                         minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip \\\n                         ex ea commodo consequat. Duis aute irure dolor in reprehenderit in \\\n                         voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur \\\n                         sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt \\\n                         mollit anim id est laborum.\";\n\n    const BLOCK_SIZE: usize = 16_384;\n\n    pub fn write_lorem_ipsum_store(\n        writer: WritePtr,\n        num_docs: usize,\n        compressor: Compressor,\n        blocksize: usize,\n        separate_thread: bool,\n    ) -> Schema {\n        let mut schema_builder = Schema::builder();\n        let field_body = schema_builder.add_text_field(\"body\", TextOptions::default().set_stored());\n        let field_title =\n            schema_builder.add_text_field(\"title\", TextOptions::default().set_stored());\n        let schema = schema_builder.build();\n        {\n            let mut store_writer =\n                StoreWriter::new(writer, compressor, blocksize, separate_thread).unwrap();\n            for i in 0..num_docs {\n                let mut doc = TantivyDocument::default();\n                doc.add_text(field_body, LOREM);\n                doc.add_text(field_title, format!(\"Doc {i}\"));\n                store_writer.store(&doc, &schema).unwrap();\n            }\n            store_writer.close().unwrap();\n        }\n        schema\n    }\n\n    const NUM_DOCS: usize = 1_000;\n\n    #[test]\n    fn test_doc_store_iter_with_delete_bug_1077() -> crate::Result<()> {\n        // this will cover deletion of the first element in a checkpoint\n        let deleted_doc_ids = (200..300).collect::<Vec<_>>();\n        let alive_bitset =\n            AliveBitSet::for_test_from_deleted_docs(&deleted_doc_ids, NUM_DOCS as u32);\n\n        let path = Path::new(\"store\");\n        let directory = RamDirectory::create();\n        let store_wrt = directory.open_write(path)?;\n        let schema =\n            write_lorem_ipsum_store(store_wrt, NUM_DOCS, Compressor::default(), BLOCK_SIZE, true);\n        let field_title = schema.get_field(\"title\").unwrap();\n        let store_file = directory.open_read(path)?;\n        let store = StoreReader::open(store_file, 10)?;\n        for i in 0..NUM_DOCS as u32 {\n            assert_eq!(\n                store\n                    .get::<TantivyDocument>(i)?\n                    .get_first(field_title)\n                    .unwrap()\n                    .as_value()\n                    .as_str()\n                    .unwrap(),\n                format!(\"Doc {i}\")\n            );\n        }\n\n        for doc in store.iter::<TantivyDocument>(Some(&alive_bitset)) {\n            let doc = doc?;\n            let title_content = doc\n                .get_first(field_title)\n                .unwrap()\n                .as_value()\n                .as_str()\n                .unwrap()\n                .to_string();\n            if !title_content.starts_with(\"Doc \") {\n                panic!(\"unexpected title_content {title_content}\");\n            }\n\n            let id = title_content\n                .strip_prefix(\"Doc \")\n                .unwrap()\n                .parse::<u32>()\n                .unwrap();\n            if alive_bitset.is_deleted(id) {\n                panic!(\"unexpected deleted document {id}\");\n            }\n        }\n\n        Ok(())\n    }\n\n    fn test_store(\n        compressor: Compressor,\n        blocksize: usize,\n        separate_thread: bool,\n    ) -> crate::Result<()> {\n        let path = Path::new(\"store\");\n        let directory = RamDirectory::create();\n        let store_wrt = directory.open_write(path)?;\n        let schema =\n            write_lorem_ipsum_store(store_wrt, NUM_DOCS, compressor, blocksize, separate_thread);\n        let field_title = schema.get_field(\"title\").unwrap();\n        let store_file = directory.open_read(path)?;\n        let store = StoreReader::open(store_file, 10)?;\n        for i in 0..NUM_DOCS as u32 {\n            assert_eq!(\n                *store\n                    .get::<TantivyDocument>(i)?\n                    .get_first(field_title)\n                    .unwrap()\n                    .as_str()\n                    .unwrap(),\n                format!(\"Doc {i}\")\n            );\n        }\n        for (i, doc) in store.iter::<TantivyDocument>(None).enumerate() {\n            assert_eq!(\n                *doc?.get_first(field_title).unwrap().as_str().unwrap(),\n                format!(\"Doc {i}\")\n            );\n        }\n        Ok(())\n    }\n\n    #[test]\n    fn test_store_no_compression_same_thread() -> crate::Result<()> {\n        test_store(Compressor::None, BLOCK_SIZE, false)\n    }\n\n    #[test]\n    fn test_store_no_compression() -> crate::Result<()> {\n        test_store(Compressor::None, BLOCK_SIZE, true)\n    }\n\n    #[cfg(feature = \"lz4-compression\")]\n    #[test]\n    fn test_store_lz4_block() -> crate::Result<()> {\n        test_store(Compressor::Lz4, BLOCK_SIZE, true)\n    }\n\n    #[cfg(feature = \"zstd-compression\")]\n    #[test]\n    fn test_store_zstd() -> crate::Result<()> {\n        test_store(\n            Compressor::Zstd(ZstdCompressor::default()),\n            BLOCK_SIZE,\n            true,\n        )\n    }\n\n    #[test]\n    fn test_store_with_delete() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n\n        let text_field_options = TextOptions::default()\n            .set_indexing_options(\n                TextFieldIndexing::default()\n                    .set_index_option(schema::IndexRecordOption::WithFreqsAndPositions),\n            )\n            .set_stored();\n        let text_field = schema_builder.add_text_field(\"text_field\", text_field_options);\n        let schema = schema_builder.build();\n        let index_builder = Index::builder().schema(schema);\n\n        let index = index_builder.create_in_ram()?;\n\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            index_writer.add_document(doc!(text_field=> \"deleteme\"))?;\n            index_writer.add_document(doc!(text_field=> \"deletemenot\"))?;\n            index_writer.add_document(doc!(text_field=> \"deleteme\"))?;\n            index_writer.add_document(doc!(text_field=> \"deletemenot\"))?;\n            index_writer.add_document(doc!(text_field=> \"deleteme\"))?;\n\n            index_writer.delete_term(Term::from_field_text(text_field, \"deleteme\"));\n            index_writer.commit()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        let reader = searcher.segment_reader(0);\n        let store = reader.get_store_reader(10)?;\n        for doc in store.iter::<TantivyDocument>(reader.alive_bitset()) {\n            assert_eq!(\n                *doc?.get_first(text_field).unwrap().as_str().unwrap(),\n                \"deletemenot\".to_string()\n            );\n        }\n        Ok(())\n    }\n\n    #[cfg(feature = \"lz4-compression\")]\n    #[cfg(feature = \"zstd-compression\")]\n    #[test]\n    fn test_merge_with_changed_compressor() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n\n        let text_field = schema_builder.add_text_field(\"text_field\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index_builder = Index::builder().schema(schema);\n\n        let mut index = index_builder.create_in_ram().unwrap();\n        index.settings_mut().docstore_compression = Compressor::Lz4;\n        {\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            // put enough data create enough blocks in the doc store to be considered for stacking\n            for _ in 0..200 {\n                index_writer.add_document(doc!(text_field=> LOREM))?;\n            }\n            assert!(index_writer.commit().is_ok());\n            for _ in 0..200 {\n                index_writer.add_document(doc!(text_field=> LOREM))?;\n            }\n            assert!(index_writer.commit().is_ok());\n        }\n        assert_eq!(\n            index.reader().unwrap().searcher().segment_readers()[0]\n                .get_store_reader(10)\n                .unwrap()\n                .decompressor(),\n            Decompressor::Lz4\n        );\n        // Change compressor, this disables stacking on merging\n        let index_settings = index.settings_mut();\n        index_settings.docstore_compression = Compressor::Zstd(Default::default());\n        // Merging the segments\n        {\n            let segment_ids = index\n                .searchable_segment_ids()\n                .expect(\"Searchable segments failed.\");\n            let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();\n            assert!(index_writer.merge(&segment_ids).wait().is_ok());\n            assert!(index_writer.wait_merging_threads().is_ok());\n        }\n\n        let searcher = index.reader().unwrap().searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let reader = searcher.segment_readers().iter().last().unwrap();\n        let store = reader.get_store_reader(10).unwrap();\n\n        for doc in store\n            .iter::<TantivyDocument>(reader.alive_bitset())\n            .take(50)\n        {\n            assert_eq!(\n                *doc?.get_first(text_field).and_then(|v| v.as_str()).unwrap(),\n                LOREM.to_string()\n            );\n        }\n        assert_eq!(store.decompressor(), Decompressor::Zstd);\n\n        Ok(())\n    }\n\n    #[test]\n    fn test_merge_of_small_segments() -> crate::Result<()> {\n        let mut schema_builder = schema::Schema::builder();\n\n        let text_field = schema_builder.add_text_field(\"text_field\", TEXT | STORED);\n        let schema = schema_builder.build();\n        let index_builder = Index::builder().schema(schema);\n\n        let index = index_builder.create_in_ram().unwrap();\n\n        {\n            let mut index_writer = index.writer_for_tests()?;\n            index_writer.add_document(doc!(text_field=> \"1\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(text_field=> \"2\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(text_field=> \"3\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(text_field=> \"4\"))?;\n            index_writer.commit()?;\n            index_writer.add_document(doc!(text_field=> \"5\"))?;\n            index_writer.commit()?;\n        }\n        // Merging the segments\n        {\n            let segment_ids = index.searchable_segment_ids()?;\n            let mut index_writer: IndexWriter = index.writer_for_tests()?;\n            index_writer.merge(&segment_ids).wait()?;\n            index_writer.wait_merging_threads()?;\n        }\n\n        let searcher = index.reader()?.searcher();\n        assert_eq!(searcher.segment_readers().len(), 1);\n        let reader = searcher.segment_readers().iter().last().unwrap();\n        let store = reader.get_store_reader(10)?;\n        assert_eq!(store.block_checkpoints().count(), 1);\n        Ok(())\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n\n    use std::path::Path;\n\n    use test::Bencher;\n\n    use super::tests::write_lorem_ipsum_store;\n    use crate::directory::{Directory, RamDirectory};\n    use crate::store::{Compressor, StoreReader};\n    use crate::TantivyDocument;\n\n    #[bench]\n    #[cfg(feature = \"mmap\")]\n    fn bench_store_encode(b: &mut Bencher) {\n        let directory = RamDirectory::create();\n        let path = Path::new(\"store\");\n        b.iter(|| {\n            write_lorem_ipsum_store(\n                directory.open_write(path).unwrap(),\n                1_000,\n                Compressor::default(),\n                16_384,\n                true,\n            );\n            directory.delete(path).unwrap();\n        });\n    }\n\n    #[bench]\n    fn bench_store_decode(b: &mut Bencher) {\n        let directory = RamDirectory::create();\n        let path = Path::new(\"store\");\n        write_lorem_ipsum_store(\n            directory.open_write(path).unwrap(),\n            1_000,\n            Compressor::default(),\n            16_384,\n            true,\n        );\n        let store_file = directory.open_read(path).unwrap();\n        let store = StoreReader::open(store_file, 10).unwrap();\n        b.iter(|| store.iter::<TantivyDocument>(None).collect::<Vec<_>>());\n    }\n}\n"
  },
  {
    "path": "src/store/reader.rs",
    "content": "use std::fmt::Display;\nuse std::io;\nuse std::iter::Sum;\nuse std::num::NonZeroUsize;\nuse std::ops::{AddAssign, Range};\nuse std::sync::atomic::{AtomicUsize, Ordering};\nuse std::sync::{Arc, Mutex};\n\nuse common::{BinarySerializable, OwnedBytes};\nuse lru::LruCache;\n\nuse super::footer::DocStoreFooter;\nuse super::index::SkipIndex;\nuse super::Decompressor;\nuse crate::directory::FileSlice;\nuse crate::error::DataCorruption;\nuse crate::fastfield::AliveBitSet;\nuse crate::schema::document::{BinaryDocumentDeserializer, DocumentDeserialize};\nuse crate::space_usage::StoreSpaceUsage;\nuse crate::store::index::Checkpoint;\nuse crate::DocId;\n#[cfg(feature = \"quickwit\")]\nuse crate::Executor;\n\npub(crate) const DOCSTORE_CACHE_CAPACITY: usize = 100;\n\ntype Block = OwnedBytes;\n\n/// The format version of the document store.\n#[derive(Clone, Copy, Debug, PartialEq, PartialOrd)]\npub(crate) enum DocStoreVersion {\n    V1 = 1,\n    V2 = 2,\n}\nimpl Display for DocStoreVersion {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        match self {\n            DocStoreVersion::V1 => write!(f, \"V1\"),\n            DocStoreVersion::V2 => write!(f, \"V2\"),\n        }\n    }\n}\nimpl BinarySerializable for DocStoreVersion {\n    fn serialize<W: io::Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        (*self as u32).serialize(writer)\n    }\n\n    fn deserialize<R: io::Read>(reader: &mut R) -> io::Result<Self> {\n        Ok(match u32::deserialize(reader)? {\n            1 => DocStoreVersion::V1,\n            2 => DocStoreVersion::V2,\n            v => {\n                return Err(io::Error::new(\n                    io::ErrorKind::InvalidData,\n                    format!(\"Invalid doc store version {v}\"),\n                ))\n            }\n        })\n    }\n}\n\n/// Reads document off tantivy's [`Store`](./index.html)\npub struct StoreReader {\n    decompressor: Decompressor,\n    doc_store_version: DocStoreVersion,\n    data: FileSlice,\n    skip_index: Arc<SkipIndex>,\n    space_usage: StoreSpaceUsage,\n    cache: BlockCache,\n}\n\n/// The cache for decompressed blocks.\nstruct BlockCache {\n    cache: Option<Mutex<LruCache<usize, Block>>>,\n    cache_hits: AtomicUsize,\n    cache_misses: AtomicUsize,\n}\n\nimpl BlockCache {\n    fn get_from_cache(&self, pos: usize) -> Option<Block> {\n        if let Some(block) = self\n            .cache\n            .as_ref()\n            .and_then(|cache| cache.lock().unwrap().get(&pos).cloned())\n        {\n            self.cache_hits.fetch_add(1, Ordering::SeqCst);\n            return Some(block);\n        }\n        self.cache_misses.fetch_add(1, Ordering::SeqCst);\n        None\n    }\n\n    fn put_into_cache(&self, pos: usize, data: Block) {\n        if let Some(cache) = self.cache.as_ref() {\n            cache.lock().unwrap().put(pos, data);\n        }\n    }\n\n    fn stats(&self) -> CacheStats {\n        CacheStats {\n            cache_hits: self.cache_hits.load(Ordering::Relaxed),\n            cache_misses: self.cache_misses.load(Ordering::Relaxed),\n            num_entries: self.len(),\n        }\n    }\n\n    fn len(&self) -> usize {\n        self.cache\n            .as_ref()\n            .map_or(0, |cache| cache.lock().unwrap().len())\n    }\n\n    #[cfg(test)]\n    fn peek_lru(&self) -> Option<usize> {\n        self.cache\n            .as_ref()\n            .and_then(|cache| cache.lock().unwrap().peek_lru().map(|(&k, _)| k))\n    }\n}\n\n#[derive(Debug, Default)]\n/// CacheStats for the `StoreReader`.\npub struct CacheStats {\n    /// The number of entries in the cache\n    pub num_entries: usize,\n    /// The number of cache hits.\n    pub cache_hits: usize,\n    /// The number of cache misses.\n    pub cache_misses: usize,\n}\n\nimpl AddAssign for CacheStats {\n    fn add_assign(&mut self, other: Self) {\n        *self = Self {\n            num_entries: self.num_entries + other.num_entries,\n            cache_hits: self.cache_hits + other.cache_hits,\n            cache_misses: self.cache_misses + other.cache_misses,\n        };\n    }\n}\n\nimpl Sum for CacheStats {\n    fn sum<I: Iterator<Item = Self>>(mut iter: I) -> Self {\n        let mut first = iter.next().unwrap_or_default();\n        for el in iter {\n            first += el;\n        }\n        first\n    }\n}\n\nimpl StoreReader {\n    /// Opens a store reader\n    ///\n    /// `cache_num_blocks` sets the number of decompressed blocks to be cached in an LRU.\n    /// The size of blocks is configurable, this should be reflexted in the\n    pub fn open(store_file: FileSlice, cache_num_blocks: usize) -> io::Result<StoreReader> {\n        let (footer, data_and_offset) = DocStoreFooter::extract_footer(store_file)?;\n\n        let (data_file, offset_index_file) = data_and_offset.split(footer.offset as usize);\n        let index_data = offset_index_file.read_bytes()?;\n        let space_usage =\n            StoreSpaceUsage::new(data_file.num_bytes(), offset_index_file.num_bytes());\n        let skip_index = SkipIndex::open(index_data);\n        Ok(StoreReader {\n            decompressor: footer.decompressor,\n            doc_store_version: footer.doc_store_version,\n            data: data_file,\n            cache: BlockCache {\n                cache: NonZeroUsize::new(cache_num_blocks)\n                    .map(|cache_num_blocks| Mutex::new(LruCache::new(cache_num_blocks))),\n                cache_hits: Default::default(),\n                cache_misses: Default::default(),\n            },\n            skip_index: Arc::new(skip_index),\n            space_usage,\n        })\n    }\n\n    pub(crate) fn block_checkpoints(&self) -> impl Iterator<Item = Checkpoint> + '_ {\n        self.skip_index.checkpoints()\n    }\n\n    pub(crate) fn decompressor(&self) -> Decompressor {\n        self.decompressor\n    }\n\n    /// Returns the cache hit and miss statistics of the store reader.\n    pub(crate) fn cache_stats(&self) -> CacheStats {\n        self.cache.stats()\n    }\n\n    /// Get checkpoint for `DocId`. The checkpoint can be used to load a block containing the\n    /// document.\n    ///\n    /// Advanced API. In most cases use [`get`](Self::get).\n    fn block_checkpoint(&self, doc_id: DocId) -> crate::Result<Checkpoint> {\n        self.skip_index.seek(doc_id).ok_or_else(|| {\n            crate::TantivyError::InvalidArgument(format!(\"Failed to lookup Doc #{doc_id}.\"))\n        })\n    }\n\n    pub(crate) fn block_data(&self) -> io::Result<OwnedBytes> {\n        self.data.read_bytes()\n    }\n\n    fn get_compressed_block(&self, checkpoint: &Checkpoint) -> io::Result<OwnedBytes> {\n        self.data.slice(checkpoint.byte_range.clone()).read_bytes()\n    }\n\n    /// Loads and decompresses a block.\n    ///\n    /// Advanced API. In most cases use [`get`](Self::get).\n    fn read_block(&self, checkpoint: &Checkpoint) -> io::Result<Block> {\n        let cache_key = checkpoint.byte_range.start;\n        if let Some(block) = self.cache.get_from_cache(cache_key) {\n            return Ok(block);\n        }\n\n        let compressed_block = self.get_compressed_block(checkpoint)?;\n        let decompressed_block =\n            OwnedBytes::new(self.decompressor.decompress(compressed_block.as_ref())?);\n\n        self.cache\n            .put_into_cache(cache_key, decompressed_block.clone());\n\n        Ok(decompressed_block)\n    }\n\n    /// Reads a given document.\n    ///\n    /// Calling `.get(doc)` is relatively costly as it requires\n    /// decompressing a compressed block. The store utilizes a LRU cache,\n    /// so accessing docs from the same compressed block should be faster.\n    /// For that reason a store reader should be kept and reused.\n    ///\n    /// It should not be called to score documents\n    /// for instance.\n    pub fn get<D: DocumentDeserialize>(&self, doc_id: DocId) -> crate::Result<D> {\n        let mut doc_bytes = self.get_document_bytes(doc_id)?;\n\n        let deserializer =\n            BinaryDocumentDeserializer::from_reader(&mut doc_bytes, self.doc_store_version)\n                .map_err(crate::TantivyError::from)?;\n        D::deserialize(deserializer).map_err(crate::TantivyError::from)\n    }\n\n    /// Returns raw bytes of a given document.\n    ///\n    /// Calling `.get(doc)` is relatively costly as it requires\n    /// decompressing a compressed block. The store utilizes a LRU cache,\n    /// so accessing docs from the same compressed block should be faster.\n    /// For that reason a store reader should be kept and reused.\n    pub fn get_document_bytes(&self, doc_id: DocId) -> crate::Result<OwnedBytes> {\n        let checkpoint = self.block_checkpoint(doc_id)?;\n        let block = self.read_block(&checkpoint)?;\n        Self::get_document_bytes_from_block(block, doc_id, &checkpoint)\n    }\n\n    /// Advanced API.\n    ///\n    /// In most cases use [`get_document_bytes`](Self::get_document_bytes).\n    fn get_document_bytes_from_block(\n        block: OwnedBytes,\n        doc_id: DocId,\n        checkpoint: &Checkpoint,\n    ) -> crate::Result<OwnedBytes> {\n        let doc_pos = doc_id - checkpoint.doc_range.start;\n\n        let range = block_read_index(&block, doc_pos)?;\n        Ok(block.slice(range))\n    }\n\n    /// Iterator over all Documents in their order as they are stored in the doc store.\n    /// Use this, if you want to extract all Documents from the doc store.\n    /// The `alive_bitset` has to be forwarded from the `SegmentReader` or the results may be wrong.\n    pub fn iter<'a: 'b, 'b, D: DocumentDeserialize>(\n        &'b self,\n        alive_bitset: Option<&'a AliveBitSet>,\n    ) -> impl Iterator<Item = crate::Result<D>> + 'b {\n        self.iter_raw(alive_bitset).map(|doc_bytes_res| {\n            let mut doc_bytes = doc_bytes_res?;\n\n            let deserializer =\n                BinaryDocumentDeserializer::from_reader(&mut doc_bytes, self.doc_store_version)\n                    .map_err(crate::TantivyError::from)?;\n            D::deserialize(deserializer).map_err(crate::TantivyError::from)\n        })\n    }\n\n    /// Iterator over all raw Documents in their order as they are stored in the doc store.\n    /// Use this, if you want to extract all Documents from the doc store.\n    /// The `alive_bitset` has to be forwarded from the `SegmentReader` or the results may be wrong.\n    pub(crate) fn iter_raw<'a: 'b, 'b>(\n        &'b self,\n        alive_bitset: Option<&'a AliveBitSet>,\n    ) -> impl Iterator<Item = crate::Result<OwnedBytes>> + 'b {\n        let last_doc_id = self\n            .block_checkpoints()\n            .last()\n            .map(|checkpoint| checkpoint.doc_range.end)\n            .unwrap_or(0);\n        let mut checkpoint_block_iter = self.block_checkpoints();\n        let mut curr_checkpoint = checkpoint_block_iter.next();\n        let mut curr_block = curr_checkpoint\n            .as_ref()\n            .map(|checkpoint| self.read_block(checkpoint).map_err(|e| e.kind())); // map error in order to enable cloning\n        let mut doc_pos = 0;\n        (0..last_doc_id)\n            .filter_map(move |doc_id| {\n                // filter_map is only used to resolve lifetime issues between the two closures on\n                // the outer variables\n\n                // check move to next checkpoint\n                if doc_id >= curr_checkpoint.as_ref().unwrap().doc_range.end {\n                    curr_checkpoint = checkpoint_block_iter.next();\n                    curr_block = curr_checkpoint\n                        .as_ref()\n                        .map(|checkpoint| self.read_block(checkpoint).map_err(|e| e.kind()));\n                    doc_pos = 0;\n                }\n\n                let alive = alive_bitset\n                    .map(|bitset| bitset.is_alive(doc_id))\n                    .unwrap_or(true);\n                let res = if alive {\n                    Some((curr_block.clone(), doc_pos))\n                } else {\n                    None\n                };\n                doc_pos += 1;\n                res\n            })\n            .map(move |(block, doc_pos)| {\n                let block = block\n                    .ok_or_else(|| {\n                        DataCorruption::comment_only(\n                            \"the current checkpoint in the doc store iterator is none, this \\\n                             should never happen\",\n                        )\n                    })?\n                    .map_err(|error_kind| {\n                        std::io::Error::new(error_kind, \"error when reading block in doc store\")\n                    })?;\n\n                let range = block_read_index(&block, doc_pos)?;\n                Ok(block.slice(range))\n            })\n    }\n\n    /// Summarize total space usage of this store reader.\n    pub fn space_usage(&self) -> StoreSpaceUsage {\n        self.space_usage.clone()\n    }\n}\n\nfn block_read_index(block: &[u8], doc_pos: u32) -> crate::Result<Range<usize>> {\n    let doc_pos = doc_pos as usize;\n    let size_of_u32 = std::mem::size_of::<u32>();\n\n    let index_len_pos = block.len() - size_of_u32;\n    let index_len = u32::deserialize(&mut &block[index_len_pos..])? as usize;\n\n    if doc_pos > index_len {\n        return Err(crate::TantivyError::InternalError(\n            \"Attempted to read doc from wrong block\".to_owned(),\n        ));\n    }\n\n    let index_start = block.len() - (index_len + 1) * size_of_u32;\n    let index = &block[index_start..index_start + index_len * size_of_u32];\n\n    let start_offset = u32::deserialize(&mut &index[doc_pos * size_of_u32..])? as usize;\n    let end_offset = u32::deserialize(&mut &index[(doc_pos + 1) * size_of_u32..])\n        .unwrap_or(index_start as u32) as usize;\n    Ok(start_offset..end_offset)\n}\n\n#[cfg(feature = \"quickwit\")]\nimpl StoreReader {\n    /// Advanced API.\n    ///\n    /// In most cases use [`get_async`](Self::get_async)\n    ///\n    /// Loads and decompresses a block asynchronously.\n    async fn read_block_async(\n        &self,\n        checkpoint: &Checkpoint,\n        executor: &Executor,\n    ) -> io::Result<Block> {\n        let cache_key = checkpoint.byte_range.start;\n        if let Some(block) = self.cache.get_from_cache(checkpoint.byte_range.start) {\n            return Ok(block);\n        }\n\n        let compressed_block = self\n            .data\n            .slice(checkpoint.byte_range.clone())\n            .read_bytes_async()\n            .await?;\n\n        let decompressor = self.decompressor;\n        let maybe_decompressed_block = executor\n            .spawn_blocking(move || decompressor.decompress(compressed_block.as_ref()))\n            .await\n            .expect(\"decompression panicked\");\n        let decompressed_block = OwnedBytes::new(maybe_decompressed_block?);\n\n        self.cache\n            .put_into_cache(cache_key, decompressed_block.clone());\n\n        Ok(decompressed_block)\n    }\n\n    /// Reads raw bytes of a given document asynchronously.\n    pub async fn get_document_bytes_async(\n        &self,\n        doc_id: DocId,\n        executor: &Executor,\n    ) -> crate::Result<OwnedBytes> {\n        let checkpoint = self.block_checkpoint(doc_id)?;\n        let block = self.read_block_async(&checkpoint, executor).await?;\n        Self::get_document_bytes_from_block(block, doc_id, &checkpoint)\n    }\n\n    /// Fetches a document asynchronously. Async version of [`get`](Self::get).\n    pub async fn get_async<D: DocumentDeserialize>(\n        &self,\n        doc_id: DocId,\n        executor: &Executor,\n    ) -> crate::Result<D> {\n        let mut doc_bytes = self.get_document_bytes_async(doc_id, executor).await?;\n\n        let deserializer =\n            BinaryDocumentDeserializer::from_reader(&mut doc_bytes, self.doc_store_version)\n                .map_err(crate::TantivyError::from)?;\n        D::deserialize(deserializer).map_err(crate::TantivyError::from)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::path::Path;\n\n    use super::*;\n    use crate::directory::RamDirectory;\n    use crate::schema::{Field, TantivyDocument, Value};\n    use crate::store::tests::write_lorem_ipsum_store;\n    use crate::store::Compressor;\n    use crate::Directory;\n\n    const BLOCK_SIZE: usize = 16_384;\n\n    fn get_text_field<'a>(doc: &'a TantivyDocument, field: &'a Field) -> Option<&'a str> {\n        doc.get_first(*field).and_then(|f| f.as_value().as_str())\n    }\n\n    #[test]\n    fn test_doc_store_version_ord() {\n        assert!(DocStoreVersion::V1 < DocStoreVersion::V2);\n    }\n\n    #[test]\n    fn test_store_lru_cache() -> crate::Result<()> {\n        let directory = RamDirectory::create();\n        let path = Path::new(\"store\");\n        let writer = directory.open_write(path)?;\n        let schema = write_lorem_ipsum_store(writer, 500, Compressor::None, BLOCK_SIZE, true);\n        let title = schema.get_field(\"title\").unwrap();\n        let store_file = directory.open_read(path)?;\n        let store = StoreReader::open(store_file, DOCSTORE_CACHE_CAPACITY)?;\n\n        assert_eq!(store.cache.len(), 0);\n        assert_eq!(store.cache_stats().cache_hits, 0);\n        assert_eq!(store.cache_stats().cache_misses, 0);\n\n        let doc = store.get(0)?;\n        assert_eq!(get_text_field(&doc, &title), Some(\"Doc 0\"));\n\n        assert_eq!(store.cache.len(), 1);\n        assert_eq!(store.cache_stats().cache_hits, 0);\n        assert_eq!(store.cache_stats().cache_misses, 1);\n\n        assert_eq!(store.cache.peek_lru(), Some(0));\n\n        let doc = store.get(499)?;\n        assert_eq!(get_text_field(&doc, &title), Some(\"Doc 499\"));\n\n        assert_eq!(store.cache.len(), 2);\n        assert_eq!(store.cache_stats().cache_hits, 0);\n        assert_eq!(store.cache_stats().cache_misses, 2);\n\n        assert_eq!(store.cache.peek_lru(), Some(0));\n\n        let doc = store.get(0)?;\n        assert_eq!(get_text_field(&doc, &title), Some(\"Doc 0\"));\n\n        assert_eq!(store.cache.len(), 2);\n        assert_eq!(store.cache_stats().cache_hits, 1);\n        assert_eq!(store.cache_stats().cache_misses, 2);\n\n        assert_eq!(store.cache.peek_lru(), Some(232206));\n\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/store/store_compressor.rs",
    "content": "use std::io::Write;\nuse std::sync::mpsc::{sync_channel, Receiver, SyncSender};\nuse std::thread::JoinHandle;\nuse std::{io, thread};\n\nuse common::{BinarySerializable, CountingWriter, TerminatingWrite};\n\nuse super::DOC_STORE_VERSION;\nuse crate::directory::WritePtr;\nuse crate::store::footer::DocStoreFooter;\nuse crate::store::index::{Checkpoint, SkipIndexBuilder};\nuse crate::store::{Compressor, Decompressor, StoreReader};\nuse crate::DocId;\n\npub struct BlockCompressor(BlockCompressorVariants);\n\n// The struct wrapping an enum is just here to keep the\n// impls private.\nenum BlockCompressorVariants {\n    SameThread(BlockCompressorImpl),\n    DedicatedThread(DedicatedThreadBlockCompressorImpl),\n}\n\nimpl BlockCompressor {\n    pub fn new(compressor: Compressor, wrt: WritePtr, dedicated_thread: bool) -> io::Result<Self> {\n        let block_compressor_impl = BlockCompressorImpl::new(compressor, wrt);\n        if dedicated_thread {\n            let dedicated_thread_compressor =\n                DedicatedThreadBlockCompressorImpl::new(block_compressor_impl)?;\n            Ok(BlockCompressor(BlockCompressorVariants::DedicatedThread(\n                dedicated_thread_compressor,\n            )))\n        } else {\n            Ok(BlockCompressor(BlockCompressorVariants::SameThread(\n                block_compressor_impl,\n            )))\n        }\n    }\n\n    pub fn compress_block_and_write(\n        &mut self,\n        bytes: &[u8],\n        num_docs_in_block: u32,\n    ) -> io::Result<()> {\n        match &mut self.0 {\n            BlockCompressorVariants::SameThread(block_compressor) => {\n                block_compressor.compress_block_and_write(bytes, num_docs_in_block)?;\n            }\n            BlockCompressorVariants::DedicatedThread(different_thread_block_compressor) => {\n                different_thread_block_compressor\n                    .compress_block_and_write(bytes, num_docs_in_block)?;\n            }\n        }\n        Ok(())\n    }\n\n    pub fn stack_reader(&mut self, store_reader: StoreReader) -> io::Result<()> {\n        match &mut self.0 {\n            BlockCompressorVariants::SameThread(block_compressor) => {\n                block_compressor.stack(store_reader)?;\n            }\n            BlockCompressorVariants::DedicatedThread(different_thread_block_compressor) => {\n                different_thread_block_compressor.stack_reader(store_reader)?;\n            }\n        }\n        Ok(())\n    }\n\n    pub fn close(self) -> io::Result<()> {\n        let imp = self.0;\n        match imp {\n            BlockCompressorVariants::SameThread(block_compressor) => block_compressor.close(),\n            BlockCompressorVariants::DedicatedThread(different_thread_block_compressor) => {\n                different_thread_block_compressor.close()\n            }\n        }\n    }\n}\n\nstruct BlockCompressorImpl {\n    compressor: Compressor,\n    first_doc_in_block: DocId,\n    offset_index_writer: SkipIndexBuilder,\n    intermediary_buffer: Vec<u8>,\n    writer: CountingWriter<WritePtr>,\n}\n\nimpl BlockCompressorImpl {\n    fn new(compressor: Compressor, writer: WritePtr) -> Self {\n        Self {\n            compressor,\n            first_doc_in_block: 0,\n            offset_index_writer: SkipIndexBuilder::new(),\n            intermediary_buffer: Vec::new(),\n            writer: CountingWriter::wrap(writer),\n        }\n    }\n\n    fn compress_block_and_write(&mut self, data: &[u8], num_docs_in_block: u32) -> io::Result<()> {\n        assert!(num_docs_in_block > 0);\n        self.intermediary_buffer.clear();\n        self.compressor\n            .compress_into(data, &mut self.intermediary_buffer)?;\n\n        let start_offset = self.writer.written_bytes() as usize;\n        self.writer.write_all(&self.intermediary_buffer)?;\n        let end_offset = self.writer.written_bytes() as usize;\n\n        self.register_checkpoint(Checkpoint {\n            doc_range: self.first_doc_in_block..self.first_doc_in_block + num_docs_in_block,\n            byte_range: start_offset..end_offset,\n        });\n        Ok(())\n    }\n\n    fn register_checkpoint(&mut self, checkpoint: Checkpoint) {\n        self.offset_index_writer.insert(checkpoint.clone());\n        self.first_doc_in_block = checkpoint.doc_range.end;\n    }\n\n    /// Stacks a store reader on top of the documents written so far.\n    /// This method is an optimization compared to iterating over the documents\n    /// in the store and adding them one by one, as the store's data will\n    /// not be decompressed and then recompressed.\n    fn stack(&mut self, store_reader: StoreReader) -> io::Result<()> {\n        let doc_shift = self.first_doc_in_block;\n        let start_shift = self.writer.written_bytes() as usize;\n\n        // just bulk write all of the block of the given reader.\n        self.writer\n            .write_all(store_reader.block_data()?.as_slice())?;\n\n        // concatenate the index of the `store_reader`, after translating\n        // its start doc id and its start file offset.\n        for mut checkpoint in store_reader.block_checkpoints() {\n            checkpoint.doc_range.start += doc_shift;\n            checkpoint.doc_range.end += doc_shift;\n            checkpoint.byte_range.start += start_shift;\n            checkpoint.byte_range.end += start_shift;\n            self.register_checkpoint(checkpoint);\n        }\n        Ok(())\n    }\n\n    fn close(mut self) -> io::Result<()> {\n        let header_offset: u64 = self.writer.written_bytes();\n        let docstore_footer = DocStoreFooter::new(\n            header_offset,\n            Decompressor::from(self.compressor),\n            DOC_STORE_VERSION,\n        );\n        self.offset_index_writer.serialize_into(&mut self.writer)?;\n        docstore_footer.serialize(&mut self.writer)?;\n        self.writer.terminate()\n    }\n}\n\n// ---------------------------------\nenum BlockCompressorMessage {\n    CompressBlockAndWrite {\n        block_data: Vec<u8>,\n        num_docs_in_block: u32,\n    },\n    Stack(StoreReader),\n}\n\nstruct DedicatedThreadBlockCompressorImpl {\n    join_handle: Option<JoinHandle<io::Result<()>>>,\n    tx: SyncSender<BlockCompressorMessage>,\n}\n\nimpl DedicatedThreadBlockCompressorImpl {\n    fn new(mut block_compressor: BlockCompressorImpl) -> io::Result<Self> {\n        let (tx, rx): (\n            SyncSender<BlockCompressorMessage>,\n            Receiver<BlockCompressorMessage>,\n        ) = sync_channel(3);\n        let join_handle = thread::Builder::new()\n            .name(\"docstore-compressor-thread\".to_string())\n            .spawn(move || {\n                while let Ok(packet) = rx.recv() {\n                    match packet {\n                        BlockCompressorMessage::CompressBlockAndWrite {\n                            block_data,\n                            num_docs_in_block,\n                        } => {\n                            block_compressor\n                                .compress_block_and_write(&block_data[..], num_docs_in_block)?;\n                        }\n                        BlockCompressorMessage::Stack(store_reader) => {\n                            block_compressor.stack(store_reader)?;\n                        }\n                    }\n                }\n                block_compressor.close()?;\n                Ok(())\n            })?;\n        Ok(DedicatedThreadBlockCompressorImpl {\n            join_handle: Some(join_handle),\n            tx,\n        })\n    }\n\n    fn compress_block_and_write(&mut self, bytes: &[u8], num_docs_in_block: u32) -> io::Result<()> {\n        self.send(BlockCompressorMessage::CompressBlockAndWrite {\n            block_data: bytes.to_vec(),\n            num_docs_in_block,\n        })\n    }\n\n    fn stack_reader(&mut self, store_reader: StoreReader) -> io::Result<()> {\n        self.send(BlockCompressorMessage::Stack(store_reader))\n    }\n\n    fn send(&mut self, msg: BlockCompressorMessage) -> io::Result<()> {\n        if self.tx.send(msg).is_err() {\n            harvest_thread_result(self.join_handle.take())?;\n            return Err(io::Error::other(\"Unidentified error.\"));\n        }\n        Ok(())\n    }\n\n    fn close(self) -> io::Result<()> {\n        drop(self.tx);\n        harvest_thread_result(self.join_handle)\n    }\n}\n\n/// Wait for the thread result to terminate and returns its result.\n///\n/// If the thread panicked, or if the result has already been harvested,\n/// returns an explicit error.\nfn harvest_thread_result(join_handle_opt: Option<JoinHandle<io::Result<()>>>) -> io::Result<()> {\n    let join_handle = join_handle_opt.ok_or_else(|| io::Error::other(\"Thread already joined.\"))?;\n    join_handle\n        .join()\n        .map_err(|_err| io::Error::other(\"Compressing thread panicked.\"))?\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io;\n    use std::path::Path;\n\n    use crate::directory::RamDirectory;\n    use crate::store::store_compressor::BlockCompressor;\n    use crate::store::Compressor;\n    use crate::Directory;\n\n    fn populate_block_compressor(mut block_compressor: BlockCompressor) -> io::Result<()> {\n        block_compressor.compress_block_and_write(b\"hello\", 1)?;\n        block_compressor.compress_block_and_write(b\"happy\", 1)?;\n        block_compressor.close()?;\n        Ok(())\n    }\n\n    #[test]\n    fn test_block_store_compressor_impls_yield_the_same_result() {\n        let ram_directory = RamDirectory::default();\n        let path1 = Path::new(\"path1\");\n        let path2 = Path::new(\"path2\");\n        let wrt1 = ram_directory.open_write(path1).unwrap();\n        let wrt2 = ram_directory.open_write(path2).unwrap();\n        let block_compressor1 = BlockCompressor::new(Compressor::None, wrt1, true).unwrap();\n        let block_compressor2 = BlockCompressor::new(Compressor::None, wrt2, false).unwrap();\n        populate_block_compressor(block_compressor1).unwrap();\n        populate_block_compressor(block_compressor2).unwrap();\n        let data1 = ram_directory.open_read(path1).unwrap();\n        let data2 = ram_directory.open_read(path2).unwrap();\n        assert_eq!(data1.read_bytes().unwrap(), data2.read_bytes().unwrap());\n    }\n}\n"
  },
  {
    "path": "src/store/writer.rs",
    "content": "use std::io;\n\nuse common::BinarySerializable;\n\nuse super::compressors::Compressor;\nuse super::StoreReader;\nuse crate::directory::WritePtr;\nuse crate::schema::document::{BinaryDocumentSerializer, Document};\nuse crate::schema::Schema;\nuse crate::store::store_compressor::BlockCompressor;\nuse crate::DocId;\n\n/// Write tantivy's [`Store`](./index.html)\n///\n/// Contrary to the other components of `tantivy`,\n/// the store is written to disc as document as being added,\n/// as opposed to when the segment is getting finalized.\n///\n/// The skip list index on the other hand, is built in memory.\npub struct StoreWriter {\n    compressor: Compressor,\n    block_size: usize,\n    num_docs_in_current_block: DocId,\n    current_block: Vec<u8>,\n    doc_pos: Vec<u32>,\n    block_compressor: BlockCompressor,\n}\n\nimpl StoreWriter {\n    /// Create a store writer.\n    ///\n    /// The store writer will writes blocks on disc as\n    /// document are added.\n    pub fn new(\n        writer: WritePtr,\n        compressor: Compressor,\n        block_size: usize,\n        dedicated_thread: bool,\n    ) -> io::Result<StoreWriter> {\n        let block_compressor = BlockCompressor::new(compressor, writer, dedicated_thread)?;\n        Ok(StoreWriter {\n            compressor,\n            block_size,\n            num_docs_in_current_block: 0,\n            doc_pos: Vec::new(),\n            current_block: Vec::new(),\n            block_compressor,\n        })\n    }\n\n    pub(crate) fn compressor(&self) -> Compressor {\n        self.compressor\n    }\n\n    /// The memory used (inclusive childs)\n    pub fn mem_usage(&self) -> usize {\n        self.current_block.capacity() + self.doc_pos.capacity() * std::mem::size_of::<u32>()\n    }\n\n    /// Checks if the current block is full, and if so, compresses and flushes it.\n    fn check_flush_block(&mut self) -> io::Result<()> {\n        // this does not count the VInt storing the index length itself, but it is negligible in\n        // front of everything else.\n        let index_len = self.doc_pos.len() * std::mem::size_of::<usize>();\n        if self.current_block.len() + index_len > self.block_size {\n            self.send_current_block_to_compressor()?;\n        }\n        Ok(())\n    }\n\n    /// Flushes current uncompressed block and sends to compressor.\n    fn send_current_block_to_compressor(&mut self) -> io::Result<()> {\n        // We don't do anything if the current block is empty to begin with.\n        if self.current_block.is_empty() {\n            return Ok(());\n        }\n\n        let size_of_u32 = std::mem::size_of::<u32>();\n        self.current_block\n            .reserve((self.doc_pos.len() + 1) * size_of_u32);\n\n        for pos in self.doc_pos.iter() {\n            pos.serialize(&mut self.current_block)?;\n        }\n        (self.doc_pos.len() as u32).serialize(&mut self.current_block)?;\n\n        self.block_compressor\n            .compress_block_and_write(&self.current_block, self.num_docs_in_current_block)?;\n        self.doc_pos.clear();\n        self.current_block.clear();\n        self.num_docs_in_current_block = 0;\n        Ok(())\n    }\n\n    /// Store a new document.\n    ///\n    /// The document id is implicitly the current number\n    /// of documents.\n    pub fn store<D: Document>(&mut self, document: &D, schema: &Schema) -> io::Result<()> {\n        self.doc_pos.push(self.current_block.len() as u32);\n\n        let mut serializer = BinaryDocumentSerializer::new(&mut self.current_block, schema);\n        serializer.serialize_doc(document)?;\n\n        self.num_docs_in_current_block += 1;\n        self.check_flush_block()?;\n        Ok(())\n    }\n\n    /// Store bytes of a serialized document.\n    ///\n    /// The document id is implicitly the current number\n    /// of documents.\n    pub fn store_bytes(&mut self, serialized_document: &[u8]) -> io::Result<()> {\n        self.doc_pos.push(self.current_block.len() as u32);\n        self.current_block.extend_from_slice(serialized_document);\n        self.num_docs_in_current_block += 1;\n        self.check_flush_block()?;\n        Ok(())\n    }\n\n    /// Stacks a store reader on top of the documents written so far.\n    /// This method is an optimization compared to iterating over the documents\n    /// in the store and adding them one by one, as the store's data will\n    /// not be decompressed and then recompressed.\n    pub fn stack(&mut self, store_reader: StoreReader) -> io::Result<()> {\n        // We flush the current block first before stacking\n        self.send_current_block_to_compressor()?;\n        self.block_compressor.stack_reader(store_reader)?;\n        Ok(())\n    }\n\n    /// Finalized the store writer.\n    ///\n    /// Compress the last unfinished block if any,\n    /// and serializes the skip list index on disc.\n    pub fn close(mut self) -> io::Result<()> {\n        self.send_current_block_to_compressor()?;\n        self.block_compressor.close()?;\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/termdict/fst_termdict/merger.rs",
    "content": "use tantivy_fst::map::{OpBuilder, Union};\nuse tantivy_fst::raw::IndexedValue;\nuse tantivy_fst::Streamer;\n\nuse super::termdict::TermDictionary;\nuse crate::postings::TermInfo;\nuse crate::termdict::{TermOrdinal, TermStreamer};\n\n/// Given a list of sorted term streams,\n/// returns an iterator over sorted unique terms.\n///\n/// The item yielded is actually a pair with\n/// - the term\n/// - a slice with the ordinal of the segments containing the term.\npub struct TermMerger<'a> {\n    dictionaries: Vec<&'a TermDictionary>,\n    union: Union<'a>,\n    current_key: Vec<u8>,\n    current_segment_and_term_ordinals: Vec<IndexedValue>,\n}\n\nimpl<'a> TermMerger<'a> {\n    /// Stream of merged term dictionary\n    pub fn new(streams: Vec<TermStreamer<'a>>) -> TermMerger<'a> {\n        let mut op_builder = OpBuilder::new();\n        let mut dictionaries = vec![];\n        for streamer in streams {\n            op_builder.push(streamer.stream);\n            dictionaries.push(streamer.fst_map);\n        }\n        TermMerger {\n            dictionaries,\n            union: op_builder.union(),\n            current_key: vec![],\n            current_segment_and_term_ordinals: vec![],\n        }\n    }\n\n    /// Iterator over `(segment ordinal, TermOrdinal)` pairs sorted by segment ordinal\n    ///\n    /// This method may be called\n    /// if [`Self::advance`] has been called before\n    /// and `true` was returned.\n    pub fn matching_segments<'b: 'a>(&'b self) -> impl 'b + Iterator<Item = (usize, TermOrdinal)> {\n        self.current_segment_and_term_ordinals\n            .iter()\n            .map(|iv| (iv.index, iv.value))\n    }\n\n    /// Advance the term iterator to the next term.\n    /// Returns `true` if there is indeed another term\n    /// `false` if there is none.\n    pub fn advance(&mut self) -> bool {\n        let (key, values) = if let Some((key, values)) = self.union.next() {\n            (key, values)\n        } else {\n            return false;\n        };\n        self.current_key.clear();\n        self.current_key.extend_from_slice(key);\n        self.current_segment_and_term_ordinals.clear();\n        self.current_segment_and_term_ordinals\n            .extend_from_slice(values);\n        self.current_segment_and_term_ordinals\n            .sort_by_key(|iv| iv.index);\n        true\n    }\n\n    /// Returns the current term.\n    ///\n    /// This method may be called if [`Self::advance`] has been called before\n    /// and `true` was returned.\n    pub fn key(&self) -> &[u8] {\n        &self.current_key\n    }\n\n    /// Iterator over `(segment ordinal, TermInfo)` pairs sorted by the ordinal.\n    ///\n    /// This method may be called if [`Self::advance`] has been called before\n    /// and `true` was returned.\n    pub fn current_segment_ords_and_term_infos<'b: 'a>(\n        &'b self,\n    ) -> impl 'b + Iterator<Item = (usize, TermInfo)> {\n        self.current_segment_and_term_ordinals\n            .iter()\n            .map(move |iv| {\n                (\n                    iv.index,\n                    self.dictionaries[iv.index].term_info_from_ord(iv.value),\n                )\n            })\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n    use rand::distributions::Alphanumeric;\n    use rand::{rng, Rng};\n    use test::{self, Bencher};\n\n    use super::TermMerger;\n    use crate::directory::FileSlice;\n    use crate::postings::TermInfo;\n    use crate::termdict::{TermDictionary, TermDictionaryBuilder};\n\n    fn make_term_info(term_ord: u64) -> TermInfo {\n        let offset = |term_ord: u64| (term_ord * 100 + term_ord * term_ord) as usize;\n        TermInfo {\n            doc_freq: term_ord as u32,\n            postings_range: offset(term_ord)..offset(term_ord + 1),\n            positions_range: offset(term_ord)..offset(term_ord + 1),\n        }\n    }\n\n    /// Create a dictionary of random strings.\n    fn rand_dict(num_terms: usize) -> std::io::Result<TermDictionary> {\n        let buffer: Vec<u8> = {\n            let mut terms = vec![];\n            for _i in 0..num_terms {\n                let rand_string: String = rng()\n                    .sample_iter(&Alphanumeric)\n                    .take(rng().random_range(30..42))\n                    .map(char::from)\n                    .collect();\n                terms.push(rand_string);\n            }\n            terms.sort();\n\n            let mut term_dictionary_builder = TermDictionaryBuilder::create(Vec::new())?;\n            for i in 0..num_terms {\n                term_dictionary_builder.insert(terms[i].as_bytes(), &make_term_info(i as u64))?;\n            }\n            term_dictionary_builder.finish()?\n        };\n        let file = FileSlice::from(buffer);\n        TermDictionary::open(file)\n    }\n\n    #[bench]\n    fn bench_termmerger(b: &mut Bencher) -> crate::Result<()> {\n        let dict1 = rand_dict(100_000)?;\n        let dict2 = rand_dict(100_000)?;\n        b.iter(|| -> crate::Result<u32> {\n            let stream1 = dict1.stream()?;\n            let stream2 = dict2.stream()?;\n            let mut merger = TermMerger::new(vec![stream1, stream2]);\n            let mut count = 0;\n            while merger.advance() {\n                count += 1;\n            }\n            Ok(count)\n        });\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/termdict/fst_termdict/mod.rs",
    "content": "//! The term dictionary main role is to associate the sorted [`Term`](crate::Term)s with\n//! a [`TermInfo`](crate::postings::TermInfo) struct that contains some meta-information\n//! about the term.\n//!\n//! Internally, the term dictionary relies on the `fst` crate to store\n//! a sorted mapping that associates each term to its rank in the lexicographical order.\n//! For instance, in a dictionary containing the sorted terms \"abba\", \"bjork\", \"blur\" and \"donovan\",\n//! the `TermOrdinal` are respectively `0`, `1`, `2`, and `3`.\n//!\n//! For `u64`-terms, tantivy explicitly uses a `BigEndian` representation to ensure that the\n//! lexicographical order matches the natural order of integers.\n//!\n//! `i64`-terms are transformed to `u64` using a continuous mapping `val ⟶ val - i64::MIN`\n//! and then treated as a `u64`.\n//!\n//! `f64`-terms are transformed to `u64` using a mapping that preserve order, and are then treated\n//! as `u64`.\n//!\n//! A second datastructure makes it possible to access a\n//! [`TermInfo`](crate::postings::TermInfo).\nmod merger;\nmod streamer;\nmod term_info_store;\nmod termdict;\n\npub use self::merger::TermMerger;\npub use self::streamer::{TermStreamer, TermStreamerBuilder};\npub use self::termdict::{TermDictionary, TermDictionaryBuilder};\n"
  },
  {
    "path": "src/termdict/fst_termdict/streamer.rs",
    "content": "use std::io;\n\nuse tantivy_fst::automaton::AlwaysMatch;\nuse tantivy_fst::map::{Stream, StreamBuilder};\nuse tantivy_fst::{Automaton, IntoStreamer, Streamer};\n\nuse super::TermDictionary;\nuse crate::postings::TermInfo;\nuse crate::termdict::TermOrdinal;\n\n/// `TermStreamerBuilder` is a helper object used to define\n/// a range of terms that should be streamed.\npub struct TermStreamerBuilder<'a, A = AlwaysMatch>\nwhere A: Automaton\n{\n    fst_map: &'a TermDictionary,\n    stream_builder: StreamBuilder<'a, A>,\n}\n\nimpl<'a, A> TermStreamerBuilder<'a, A>\nwhere A: Automaton\n{\n    pub(crate) fn new(fst_map: &'a TermDictionary, stream_builder: StreamBuilder<'a, A>) -> Self {\n        TermStreamerBuilder {\n            fst_map,\n            stream_builder,\n        }\n    }\n\n    /// Limit the range to terms greater or equal to the bound\n    pub fn ge<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.stream_builder = self.stream_builder.ge(bound);\n        self\n    }\n\n    /// Limit the range to terms strictly greater than the bound\n    pub fn gt<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.stream_builder = self.stream_builder.gt(bound);\n        self\n    }\n\n    /// Limit the range to terms lesser or equal to the bound\n    pub fn le<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.stream_builder = self.stream_builder.le(bound);\n        self\n    }\n\n    /// Limit the range to terms lesser or equal to the bound\n    pub fn lt<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.stream_builder = self.stream_builder.lt(bound);\n        self\n    }\n\n    /// Iterate over the range backwards.\n    pub fn backward(mut self) -> Self {\n        self.stream_builder = self.stream_builder.backward();\n        self\n    }\n\n    /// Creates the stream corresponding to the range\n    /// of terms defined using the `TermStreamerBuilder`.\n    pub fn into_stream(self) -> io::Result<TermStreamer<'a, A>> {\n        Ok(TermStreamer {\n            fst_map: self.fst_map,\n            stream: self.stream_builder.into_stream(),\n            term_ord: 0u64,\n            current_key: Vec::with_capacity(100),\n            current_value: TermInfo::default(),\n        })\n    }\n}\n\n/// `TermStreamer` acts as a cursor over a range of terms of a segment.\n/// Terms are guaranteed to be sorted.\npub struct TermStreamer<'a, A = AlwaysMatch>\nwhere A: Automaton\n{\n    pub(crate) fst_map: &'a TermDictionary,\n    pub(crate) stream: Stream<'a, A>,\n    term_ord: TermOrdinal,\n    current_key: Vec<u8>,\n    current_value: TermInfo,\n}\n\nimpl<A> TermStreamer<'_, A>\nwhere A: Automaton\n{\n    /// Advance position the stream on the next item.\n    /// Before the first call to `.advance()`, the stream\n    /// is an uninitialized state.\n    pub fn advance(&mut self) -> bool {\n        if let Some((term, term_ord)) = self.stream.next() {\n            self.current_key.clear();\n            self.current_key.extend_from_slice(term);\n            self.term_ord = term_ord;\n            self.current_value = self.fst_map.term_info_from_ord(term_ord);\n            true\n        } else {\n            false\n        }\n    }\n\n    /// Returns the `TermOrdinal` of the given term.\n    ///\n    /// May panic if the called as `.advance()` as never\n    /// been called before.\n    pub fn term_ord(&self) -> TermOrdinal {\n        self.term_ord\n    }\n\n    /// Accesses the current key.\n    ///\n    /// `.key()` should return the key that was returned\n    /// by the `.next()` method.\n    ///\n    /// If the end of the stream as been reached, and `.next()`\n    /// has been called and returned `None`, `.key()` remains\n    /// the value of the last key encountered.\n    ///\n    /// Before any call to `.next()`, `.key()` returns an empty array.\n    pub fn key(&self) -> &[u8] {\n        &self.current_key\n    }\n\n    /// Accesses the current value.\n    ///\n    /// Calling `.value()` after the end of the stream will return the\n    /// last `.value()` encountered.\n    ///\n    /// # Panics\n    ///\n    /// Calling `.value()` before the first call to `.advance()` returns\n    /// `V::default()`.\n    pub fn value(&self) -> &TermInfo {\n        &self.current_value\n    }\n\n    /// Return the next `(key, value)` pair.\n    #[expect(clippy::should_implement_trait)]\n    pub fn next(&mut self) -> Option<(&[u8], &TermInfo)> {\n        if self.advance() {\n            Some((self.key(), self.value()))\n        } else {\n            None\n        }\n    }\n}\n"
  },
  {
    "path": "src/termdict/fst_termdict/term_info_store.rs",
    "content": "use std::cmp;\nuse std::io::{self, Read, Write};\n\nuse byteorder::{ByteOrder, LittleEndian};\nuse common::{BinarySerializable, FixedSize};\nuse tantivy_bitpacker::{compute_num_bits, BitPacker};\n\nuse crate::directory::{FileSlice, OwnedBytes};\nuse crate::postings::TermInfo;\nuse crate::termdict::TermOrdinal;\n\nconst BLOCK_LEN: usize = 256;\n\n#[derive(Debug, Eq, PartialEq, Default)]\nstruct TermInfoBlockMeta {\n    offset: u64,\n    ref_term_info: TermInfo,\n    doc_freq_nbits: u8,\n    postings_offset_nbits: u8,\n    positions_offset_nbits: u8,\n}\n\nimpl BinarySerializable for TermInfoBlockMeta {\n    fn serialize<W: Write + ?Sized>(&self, write: &mut W) -> io::Result<()> {\n        self.offset.serialize(write)?;\n        self.ref_term_info.serialize(write)?;\n        write.write_all(&[\n            self.doc_freq_nbits,\n            self.postings_offset_nbits,\n            self.positions_offset_nbits,\n        ])?;\n        Ok(())\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let offset = u64::deserialize(reader)?;\n        let ref_term_info = TermInfo::deserialize(reader)?;\n        let mut buffer = [0u8; 3];\n        reader.read_exact(&mut buffer)?;\n        Ok(TermInfoBlockMeta {\n            offset,\n            ref_term_info,\n            doc_freq_nbits: buffer[0],\n            postings_offset_nbits: buffer[1],\n            positions_offset_nbits: buffer[2],\n        })\n    }\n}\n\nimpl FixedSize for TermInfoBlockMeta {\n    const SIZE_IN_BYTES: usize = u64::SIZE_IN_BYTES + TermInfo::SIZE_IN_BYTES + 3;\n}\n\nimpl TermInfoBlockMeta {\n    fn num_bits(&self) -> u8 {\n        self.doc_freq_nbits + self.postings_offset_nbits + self.positions_offset_nbits\n    }\n\n    // Here inner_offset is the offset within the block, WITHOUT the first term_info.\n    // In other word, term_info #1,#2,#3 gets inner_offset 0,1,2... While term_info #0\n    // is encoded without bitpacking.\n    fn deserialize_term_info(&self, data: &[u8], inner_offset: usize) -> TermInfo {\n        assert!(inner_offset < BLOCK_LEN - 1);\n        let num_bits = self.num_bits() as usize;\n\n        let posting_start_addr = num_bits * inner_offset;\n        // the posting_start is the posting_start of the next term info.\n        let posting_end_addr = posting_start_addr + num_bits;\n        let positions_start_addr = posting_start_addr + self.postings_offset_nbits as usize;\n        // the position_end is the positions_start of the next term info.\n        let positions_end_addr = positions_start_addr + num_bits;\n\n        let doc_freq_addr = positions_start_addr + self.positions_offset_nbits as usize;\n\n        let postings_start_offset = self.ref_term_info.postings_range.start\n            + extract_bits(data, posting_start_addr, self.postings_offset_nbits) as usize;\n        let postings_end_offset = self.ref_term_info.postings_range.start\n            + extract_bits(data, posting_end_addr, self.postings_offset_nbits) as usize;\n\n        let positions_start_offset = self.ref_term_info.positions_range.start\n            + extract_bits(data, positions_start_addr, self.positions_offset_nbits) as usize;\n        let positions_end_offset = self.ref_term_info.positions_range.start\n            + extract_bits(data, positions_end_addr, self.positions_offset_nbits) as usize;\n\n        let doc_freq = extract_bits(data, doc_freq_addr, self.doc_freq_nbits) as u32;\n\n        TermInfo {\n            doc_freq,\n            postings_range: postings_start_offset..postings_end_offset,\n            positions_range: positions_start_offset..positions_end_offset,\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct TermInfoStore {\n    num_terms: usize,\n    block_meta_bytes: OwnedBytes,\n    term_info_bytes: OwnedBytes,\n}\n\nfn extract_bits(data: &[u8], addr_bits: usize, num_bits: u8) -> u64 {\n    assert!(num_bits <= 56);\n    let addr_byte = addr_bits / 8;\n    let bit_shift = (addr_bits % 8) as u64;\n    let val_unshifted_unmasked: u64 = if data.len() >= addr_byte + 8 {\n        LittleEndian::read_u64(&data[addr_byte..][..8])\n    } else {\n        // the buffer is not large enough.\n        // Let's copy the few remaining bytes to a 8 byte buffer\n        // padded with 0s.\n        let mut buf = [0u8; 8];\n        let data_to_copy = &data[addr_byte..];\n        let nbytes = data_to_copy.len();\n        buf[..nbytes].copy_from_slice(data_to_copy);\n        LittleEndian::read_u64(&buf)\n    };\n    let val_shifted_unmasked = val_unshifted_unmasked >> bit_shift;\n    let mask = (1u64 << u64::from(num_bits)) - 1;\n    val_shifted_unmasked & mask\n}\n\nimpl TermInfoStore {\n    pub fn open(term_info_store_file: FileSlice) -> io::Result<TermInfoStore> {\n        let (len_slice, main_slice) = term_info_store_file.split(16);\n        let mut bytes = len_slice.read_bytes()?;\n        let len = u64::deserialize(&mut bytes)? as usize;\n        let num_terms = u64::deserialize(&mut bytes)? as usize;\n        let (block_meta_file, term_info_file) = main_slice.split(len);\n        let term_info_bytes = term_info_file.read_bytes()?;\n        Ok(TermInfoStore {\n            num_terms,\n            block_meta_bytes: block_meta_file.read_bytes()?,\n            term_info_bytes,\n        })\n    }\n\n    pub fn get(&self, term_ord: TermOrdinal) -> TermInfo {\n        let block_id = (term_ord as usize) / BLOCK_LEN;\n        let buffer = self.block_meta_bytes.as_slice();\n        let mut block_data: &[u8] = &buffer[block_id * TermInfoBlockMeta::SIZE_IN_BYTES..];\n        let term_info_block_data = TermInfoBlockMeta::deserialize(&mut block_data)\n            .expect(\"Failed to deserialize terminfoblockmeta\");\n        let inner_offset = (term_ord as usize) % BLOCK_LEN;\n        if inner_offset == 0 {\n            return term_info_block_data.ref_term_info;\n        }\n        let term_info_data = self.term_info_bytes.as_slice();\n        term_info_block_data.deserialize_term_info(\n            &term_info_data[term_info_block_data.offset as usize..],\n            inner_offset - 1,\n        )\n    }\n\n    pub fn num_terms(&self) -> usize {\n        self.num_terms\n    }\n}\n\npub struct TermInfoStoreWriter {\n    buffer_block_metas: Vec<u8>,\n    buffer_term_infos: Vec<u8>,\n    term_infos: Vec<TermInfo>,\n    num_terms: u64,\n}\n\nfn bitpack_serialize<W: Write>(\n    write: &mut W,\n    bit_packer: &mut BitPacker,\n    term_info_block_meta: &TermInfoBlockMeta,\n    term_info: &TermInfo,\n) -> io::Result<()> {\n    bit_packer.write(\n        term_info.postings_range.start as u64,\n        term_info_block_meta.postings_offset_nbits,\n        write,\n    )?;\n    bit_packer.write(\n        term_info.positions_range.start as u64,\n        term_info_block_meta.positions_offset_nbits,\n        write,\n    )?;\n    bit_packer.write(\n        u64::from(term_info.doc_freq),\n        term_info_block_meta.doc_freq_nbits,\n        write,\n    )?;\n    Ok(())\n}\n\nimpl TermInfoStoreWriter {\n    pub fn new() -> TermInfoStoreWriter {\n        TermInfoStoreWriter {\n            buffer_block_metas: Vec::new(),\n            buffer_term_infos: Vec::new(),\n            term_infos: Vec::with_capacity(BLOCK_LEN),\n            num_terms: 0u64,\n        }\n    }\n\n    fn flush_block(&mut self) -> io::Result<()> {\n        let mut bit_packer = BitPacker::new();\n        let ref_term_info = self.term_infos[0].clone();\n\n        let last_term_info = if let Some(last_term_info) = self.term_infos.last().cloned() {\n            last_term_info\n        } else {\n            return Ok(());\n        };\n        let postings_end_offset =\n            last_term_info.postings_range.end - ref_term_info.postings_range.start;\n        let positions_end_offset =\n            last_term_info.positions_range.end - ref_term_info.positions_range.start;\n        for term_info in &mut self.term_infos[1..] {\n            term_info.postings_range.start -= ref_term_info.postings_range.start;\n            term_info.positions_range.start -= ref_term_info.positions_range.start;\n        }\n\n        let mut max_doc_freq: u32 = 0u32;\n\n        for term_info in &self.term_infos[1..] {\n            max_doc_freq = cmp::max(max_doc_freq, term_info.doc_freq);\n        }\n\n        let max_doc_freq_nbits: u8 = compute_num_bits(u64::from(max_doc_freq));\n        let max_postings_offset_nbits = compute_num_bits(postings_end_offset as u64);\n        let max_positions_offset_nbits = compute_num_bits(positions_end_offset as u64);\n\n        let term_info_block_meta = TermInfoBlockMeta {\n            offset: self.buffer_term_infos.len() as u64,\n            ref_term_info,\n            doc_freq_nbits: max_doc_freq_nbits,\n            postings_offset_nbits: max_postings_offset_nbits,\n            positions_offset_nbits: max_positions_offset_nbits,\n        };\n\n        term_info_block_meta.serialize(&mut self.buffer_block_metas)?;\n        for term_info in &self.term_infos[1..] {\n            bitpack_serialize(\n                &mut self.buffer_term_infos,\n                &mut bit_packer,\n                &term_info_block_meta,\n                term_info,\n            )?;\n        }\n\n        // We still need to serialize the end offset for postings & positions.\n        bit_packer.write(\n            postings_end_offset as u64,\n            term_info_block_meta.postings_offset_nbits,\n            &mut self.buffer_term_infos,\n        )?;\n        bit_packer.write(\n            positions_end_offset as u64,\n            term_info_block_meta.positions_offset_nbits,\n            &mut self.buffer_term_infos,\n        )?;\n\n        // Block need end up at the end of a byte.\n        bit_packer.flush(&mut self.buffer_term_infos)?;\n        self.term_infos.clear();\n\n        Ok(())\n    }\n\n    pub fn write_term_info(&mut self, term_info: &TermInfo) -> io::Result<()> {\n        self.num_terms += 1u64;\n        self.term_infos.push(term_info.clone());\n        if self.term_infos.len() >= BLOCK_LEN {\n            self.flush_block()?;\n        }\n        Ok(())\n    }\n\n    pub fn serialize<W: io::Write + ?Sized>(&mut self, write: &mut W) -> io::Result<()> {\n        if !self.term_infos.is_empty() {\n            self.flush_block()?;\n        }\n        let len = self.buffer_block_metas.len() as u64;\n        len.serialize(write)?;\n        self.num_terms.serialize(write)?;\n        write.write_all(&self.buffer_block_metas)?;\n        write.write_all(&self.buffer_term_infos)?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use common::BinarySerializable;\n    use tantivy_bitpacker::{compute_num_bits, BitPacker};\n\n    use super::{extract_bits, TermInfoBlockMeta, TermInfoStore, TermInfoStoreWriter};\n    use crate::directory::FileSlice;\n    use crate::postings::TermInfo;\n\n    #[test]\n    fn test_term_info_block() {\n        crate::tests::fixed_size_test::<TermInfoBlockMeta>();\n    }\n\n    #[test]\n    fn test_bitpacked() {\n        let mut buffer = Vec::new();\n        let mut bitpack = BitPacker::new();\n        bitpack.write(321u64, 9, &mut buffer).unwrap();\n        assert_eq!(compute_num_bits(321u64), 9);\n        bitpack.write(2u64, 2, &mut buffer).unwrap();\n        assert_eq!(compute_num_bits(2u64), 2);\n        bitpack.write(51, 6, &mut buffer).unwrap();\n        assert_eq!(compute_num_bits(51), 6);\n        bitpack.close(&mut buffer).unwrap();\n        assert_eq!(buffer.len(), 3);\n        assert_eq!(extract_bits(&buffer[..], 0, 9), 321u64);\n        assert_eq!(extract_bits(&buffer[..], 9, 2), 2u64);\n        assert_eq!(extract_bits(&buffer[..], 11, 6), 51u64);\n    }\n\n    #[test]\n    fn test_term_info_block_meta_serialization() {\n        let term_info_block_meta = TermInfoBlockMeta {\n            offset: 2009u64,\n            ref_term_info: TermInfo {\n                doc_freq: 512,\n                postings_range: 51..57,\n                positions_range: 110..134,\n            },\n            doc_freq_nbits: 10,\n            postings_offset_nbits: 5,\n            positions_offset_nbits: 8,\n        };\n        let mut buffer: Vec<u8> = Vec::new();\n        term_info_block_meta.serialize(&mut buffer).unwrap();\n        let mut cursor: &[u8] = &buffer[..];\n        let term_info_block_meta_serde = TermInfoBlockMeta::deserialize(&mut cursor).unwrap();\n        assert_eq!(term_info_block_meta_serde, term_info_block_meta);\n    }\n\n    #[test]\n    fn test_pack() -> crate::Result<()> {\n        let mut store_writer = TermInfoStoreWriter::new();\n        let mut term_infos = vec![];\n        let offset = |i| i * 13 + i * i;\n        for i in 0usize..1000usize {\n            let term_info = TermInfo {\n                doc_freq: i as u32,\n                postings_range: offset(i)..offset(i + 1),\n                positions_range: offset(i) * 3..offset(i + 1) * 3,\n            };\n            store_writer.write_term_info(&term_info)?;\n            term_infos.push(term_info);\n        }\n        let mut buffer = Vec::new();\n        store_writer.serialize(&mut buffer)?;\n        let term_info_store = TermInfoStore::open(FileSlice::from(buffer))?;\n        for i in 0..1000 {\n            assert_eq!(\n                term_info_store.get(i as u64),\n                term_infos[i],\n                \"term info {i}\"\n            );\n        }\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "src/termdict/fst_termdict/termdict.rs",
    "content": "use std::io::{self, Write};\nuse std::sync::Arc;\n\nuse common::{BinarySerializable, CountingWriter};\nuse once_cell::sync::Lazy;\nuse tantivy_fst::raw::Fst;\nuse tantivy_fst::Automaton;\n\nuse super::term_info_store::{TermInfoStore, TermInfoStoreWriter};\nuse super::{TermStreamer, TermStreamerBuilder};\nuse crate::directory::{FileSlice, OwnedBytes};\nuse crate::postings::TermInfo;\nuse crate::termdict::TermOrdinal;\n\nfn convert_fst_error(e: tantivy_fst::Error) -> io::Error {\n    io::Error::other(e)\n}\n\nconst FST_VERSION: u32 = 1;\n\n/// Builder for the new term dictionary.\n///\n/// Inserting must be done in the order of the `keys`.\npub struct TermDictionaryBuilder<W> {\n    fst_builder: tantivy_fst::MapBuilder<W>,\n    term_info_store_writer: TermInfoStoreWriter,\n    term_ord: u64,\n}\n\nimpl<W> TermDictionaryBuilder<W>\nwhere W: Write\n{\n    /// Creates a new `TermDictionaryBuilder`\n    pub fn create(w: W) -> io::Result<Self> {\n        let fst_builder = tantivy_fst::MapBuilder::new(w).map_err(convert_fst_error)?;\n        Ok(TermDictionaryBuilder {\n            fst_builder,\n            term_info_store_writer: TermInfoStoreWriter::new(),\n            term_ord: 0,\n        })\n    }\n\n    /// Inserts a `(key, value)` pair in the term dictionary.\n    ///\n    /// *Keys have to be inserted in order.*\n    pub fn insert<K: AsRef<[u8]>>(&mut self, key_ref: K, value: &TermInfo) -> io::Result<()> {\n        let key = key_ref.as_ref();\n        self.insert_key(key)?;\n        self.insert_value(value)?;\n        Ok(())\n    }\n\n    /// # Warning\n    /// Horribly dangerous internal API\n    ///\n    /// If used, it must be used by systematically alternating calls\n    /// to insert_key and insert_value.\n    ///\n    /// Prefer using `.insert(key, value)`\n    pub fn insert_key(&mut self, key: &[u8]) -> io::Result<()> {\n        self.fst_builder\n            .insert(key, self.term_ord)\n            .map_err(convert_fst_error)?;\n        self.term_ord += 1;\n        Ok(())\n    }\n\n    /// # Warning\n    ///\n    /// Horribly dangerous internal API. See `.insert_key(...)`.\n    pub fn insert_value(&mut self, term_info: &TermInfo) -> io::Result<()> {\n        self.term_info_store_writer.write_term_info(term_info)?;\n        Ok(())\n    }\n\n    /// Finalize writing the builder, and returns the underlying\n    /// `Write` object.\n    pub fn finish(mut self) -> io::Result<W> {\n        let mut file = self.fst_builder.into_inner().map_err(convert_fst_error)?;\n        {\n            let mut counting_writer = CountingWriter::wrap(&mut file);\n            self.term_info_store_writer\n                .serialize(&mut counting_writer)?;\n            let footer_size = counting_writer.written_bytes();\n            footer_size.serialize(&mut counting_writer)?;\n            FST_VERSION.serialize(&mut counting_writer)?;\n        }\n        Ok(file)\n    }\n}\n\nfn open_fst_index(fst_file: FileSlice) -> io::Result<tantivy_fst::Map<OwnedBytes>> {\n    let bytes = fst_file.read_bytes()?;\n    let fst = Fst::new(bytes).map_err(|err| {\n        io::Error::new(\n            io::ErrorKind::InvalidData,\n            format!(\"Fst data is corrupted: {err:?}\"),\n        )\n    })?;\n    Ok(tantivy_fst::Map::from(fst))\n}\n\nstatic EMPTY_TERM_DICT_FILE: Lazy<FileSlice> = Lazy::new(|| {\n    let term_dictionary_data: Vec<u8> = TermDictionaryBuilder::create(Vec::<u8>::new())\n        .expect(\"Creating a TermDictionaryBuilder in a Vec<u8> should never fail\")\n        .finish()\n        .expect(\"Writing in a Vec<u8> should never fail\");\n    FileSlice::from(term_dictionary_data)\n});\n\n/// The term dictionary contains all of the terms in\n/// `tantivy index` in a sorted manner.\n///\n/// The `Fst` crate is used to associate terms to their\n/// respective `TermOrdinal`. The `TermInfoStore` then makes it\n/// possible to fetch the associated `TermInfo`.\n#[derive(Clone)]\npub struct TermDictionary {\n    fst_index: Arc<tantivy_fst::Map<OwnedBytes>>,\n    term_info_store: TermInfoStore,\n}\n\nimpl TermDictionary {\n    /// Opens a `TermDictionary`.\n    pub fn open(file: FileSlice) -> io::Result<Self> {\n        let (main_slice, footer_len_slice) = file.split_from_end(12);\n        let mut footer_len_bytes = footer_len_slice.read_bytes()?;\n        let footer_size = u64::deserialize(&mut footer_len_bytes)?;\n        let version = u32::deserialize(&mut footer_len_bytes)?;\n        if version != FST_VERSION {\n            return Err(io::Error::other(format!(\n                \"Unsupported fst version, expected {version}, found {FST_VERSION}\",\n            )));\n        }\n\n        let (fst_file_slice, values_file_slice) = main_slice.split_from_end(footer_size as usize);\n        let fst_index = open_fst_index(fst_file_slice)?;\n        let term_info_store = TermInfoStore::open(values_file_slice)?;\n        Ok(TermDictionary {\n            fst_index: Arc::new(fst_index),\n            term_info_store,\n        })\n    }\n\n    /// Creates an empty term dictionary which contains no terms.\n    pub fn empty() -> Self {\n        TermDictionary::open(EMPTY_TERM_DICT_FILE.clone()).unwrap()\n    }\n\n    /// Returns the number of terms in the dictionary.\n    /// Term ordinals range from 0 to `num_terms() - 1`.\n    pub fn num_terms(&self) -> usize {\n        self.term_info_store.num_terms()\n    }\n\n    /// Returns the ordinal associated with a given term.\n    pub fn term_ord<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermOrdinal>> {\n        Ok(self.fst_index.get(key))\n    }\n\n    /// Stores the term associated with a given term ordinal in\n    /// a `bytes` buffer.\n    ///\n    /// Term ordinals are defined as the position of the term in\n    /// the sorted list of terms.\n    ///\n    /// Returns true if and only if the term has been found.\n    ///\n    /// Regardless of whether the term is found or not,\n    /// the buffer may be modified.\n    pub fn ord_to_term(&self, mut ord: TermOrdinal, bytes: &mut Vec<u8>) -> io::Result<bool> {\n        bytes.clear();\n        let fst = self.fst_index.as_fst();\n        let mut node = fst.root();\n        while ord != 0 || !node.is_final() {\n            if let Some(transition) = node\n                .transitions()\n                .take_while(|transition| transition.out.value() <= ord)\n                .last()\n            {\n                ord -= transition.out.value();\n                bytes.push(transition.inp);\n                let new_node_addr = transition.addr;\n                node = fst.node(new_node_addr);\n            } else {\n                return Ok(false);\n            }\n        }\n        Ok(true)\n    }\n\n    /// Returns the number of terms in the dictionary.\n    pub fn term_info_from_ord(&self, term_ord: TermOrdinal) -> TermInfo {\n        self.term_info_store.get(term_ord)\n    }\n\n    /// Lookups the value corresponding to the key.\n    pub fn get<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermInfo>> {\n        Ok(self\n            .term_ord(key)?\n            .map(|term_ord| self.term_info_from_ord(term_ord)))\n    }\n\n    /// Returns a range builder, to stream all of the terms\n    /// within an interval.\n    pub fn range(&self) -> TermStreamerBuilder<'_> {\n        TermStreamerBuilder::new(self, self.fst_index.range())\n    }\n\n    /// A stream of all the sorted terms.\n    pub fn stream(&self) -> io::Result<TermStreamer<'_>> {\n        self.range().into_stream()\n    }\n\n    /// Returns a search builder, to stream all of the terms\n    /// within the Automaton\n    pub fn search<'a, A: Automaton + 'a>(&'a self, automaton: A) -> TermStreamerBuilder<'a, A> {\n        let stream_builder = self.fst_index.search(automaton);\n        TermStreamerBuilder::<A>::new(self, stream_builder)\n    }\n}\n"
  },
  {
    "path": "src/termdict/mod.rs",
    "content": "//! The term dictionary main role is to associate the sorted [`Term`s](crate::Term) to\n//! a [`TermInfo`] struct that contains some meta-information\n//! about the term.\n//!\n//! Internally, the term dictionary relies on the `fst` crate to store\n//! a sorted mapping that associate each term to its rank in the lexicographical order.\n//! For instance, in a dictionary containing the sorted terms \"abba\", \"bjork\", \"blur\" and \"donovan\",\n//! the [`TermOrdinal`] are respectively `0`, `1`, `2`, and `3`.\n//!\n//! For `u64`-terms, tantivy explicitly uses a `BigEndian` representation to ensure that the\n//! lexicographical order matches the natural order of integers.\n//!\n//! `i64`-terms are transformed to `u64` using a continuous mapping `val ⟶ val - i64::MIN`\n//! and then treated as a `u64`.\n//!\n//! `f64`-terms are transformed to `u64` using a mapping that preserve order, and are then treated\n//! as `u64`.\n//!\n//! A second datastructure makes it possible to access a [`TermInfo`].\n\n#[cfg(not(feature = \"quickwit\"))]\nmod fst_termdict;\n#[cfg(not(feature = \"quickwit\"))]\nuse fst_termdict as termdict;\n\n#[cfg(feature = \"quickwit\")]\nmod sstable_termdict;\n#[cfg(feature = \"quickwit\")]\nuse sstable_termdict as termdict;\n\n#[cfg(test)]\nmod tests;\n\n/// Position of the term in the sorted list of terms.\npub type TermOrdinal = u64;\n\nuse std::io;\n\nuse common::file_slice::FileSlice;\nuse common::BinarySerializable;\nuse tantivy_fst::Automaton;\n\nuse self::termdict::{\n    TermDictionary as InnerTermDict, TermDictionaryBuilder as InnerTermDictBuilder,\n    TermStreamerBuilder,\n};\npub use self::termdict::{TermMerger, TermStreamer};\nuse crate::postings::TermInfo;\n\n#[derive(Debug, Eq, PartialEq)]\n#[repr(u32)]\nenum DictionaryType {\n    Fst = 1,\n    SSTable = 2,\n}\n\nimpl TryFrom<u32> for DictionaryType {\n    type Error = &'static str;\n\n    fn try_from(value: u32) -> Result<Self, Self::Error> {\n        match value {\n            1 => Ok(DictionaryType::Fst),\n            2 => Ok(DictionaryType::SSTable),\n            _ => Err(\"Invalid value for DictionaryType\"),\n        }\n    }\n}\n\n#[cfg(not(feature = \"quickwit\"))]\nconst CURRENT_TYPE: DictionaryType = DictionaryType::Fst;\n\n#[cfg(feature = \"quickwit\")]\nconst CURRENT_TYPE: DictionaryType = DictionaryType::SSTable;\n\n// TODO in the future this should become an enum of supported dictionaries\n/// A TermDictionary wrapping either an FST based dictionary or a SSTable based one.\n#[derive(Clone)]\npub struct TermDictionary(InnerTermDict);\n\nimpl TermDictionary {\n    /// Opens a `TermDictionary`.\n    pub fn open(file: FileSlice) -> io::Result<Self> {\n        let (main_slice, dict_type) = file.split_from_end(4);\n        let mut dict_type = dict_type.read_bytes()?;\n        let dict_type = u32::deserialize(&mut dict_type)?;\n        let dict_type = DictionaryType::try_from(dict_type).map_err(|_| {\n            io::Error::other(format!(\"Unsupported dictionary type, found {dict_type}\"))\n        })?;\n\n        if dict_type != CURRENT_TYPE {\n            return Err(io::Error::other(format!(\n                \"Unsupported dictionary type, compiled tantivy with {CURRENT_TYPE:?}, but got \\\n                 {dict_type:?}\",\n            )));\n        }\n\n        InnerTermDict::open(main_slice).map(TermDictionary)\n    }\n\n    /// Creates an empty term dictionary which contains no terms.\n    pub fn empty() -> Self {\n        TermDictionary(InnerTermDict::empty())\n    }\n\n    /// Returns the number of terms in the dictionary.\n    /// Term ordinals range from 0 to `num_terms() - 1`.\n    pub fn num_terms(&self) -> usize {\n        self.0.num_terms()\n    }\n\n    /// Returns the ordinal associated with a given term.\n    pub fn term_ord<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermOrdinal>> {\n        self.0.term_ord(key)\n    }\n\n    /// Stores the term associated with a given term ordinal in\n    /// a `bytes` buffer.\n    ///\n    /// Term ordinals are defined as the position of the term in\n    /// the sorted list of terms.\n    ///\n    /// Returns true if and only if the term has been found.\n    ///\n    /// Regardless of whether the term is found or not,\n    /// the buffer may be modified.\n    pub fn ord_to_term(&self, ord: TermOrdinal, bytes: &mut Vec<u8>) -> io::Result<bool> {\n        self.0.ord_to_term(ord, bytes)\n    }\n\n    // this isn't used, and has different prototype in Fst and SSTable\n    // Returns the number of terms in the dictionary.\n    // pub fn term_info_from_ord(&self, term_ord: TermOrdinal) -> TermInfo {\n    // self.0.term_info_from_ord(term_ord)\n    // }\n\n    /// Lookups the value corresponding to the key.\n    pub fn get<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermInfo>> {\n        self.0.get(key)\n    }\n\n    /// Returns a range builder, to stream all of the terms\n    /// within an interval.\n    pub fn range(&self) -> TermStreamerBuilder<'_> {\n        self.0.range()\n    }\n\n    /// A stream of all the sorted terms.\n    pub fn stream(&self) -> io::Result<TermStreamer<'_>> {\n        self.0.stream()\n    }\n\n    /// Returns a search builder, to stream all of the terms\n    /// within the Automaton\n    pub fn search<'a, A: Automaton + 'a>(&'a self, automaton: A) -> TermStreamerBuilder<'a, A>\n    where A::State: Clone {\n        self.0.search(automaton)\n    }\n\n    #[cfg(feature = \"quickwit\")]\n    /// Lookups the value corresponding to the key.\n    pub async fn get_async<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermInfo>> {\n        self.0.get_async(key).await\n    }\n\n    #[cfg(feature = \"quickwit\")]\n    #[doc(hidden)]\n    pub async fn warm_up_dictionary(&self) -> io::Result<()> {\n        self.0.warm_up_dictionary().await\n    }\n\n    #[cfg(feature = \"quickwit\")]\n    /// Returns a file slice covering a set of sstable blocks\n    /// that includes the key range passed in arguments.\n    pub fn file_slice_for_range(\n        &self,\n        key_range: impl std::ops::RangeBounds<[u8]>,\n        limit: Option<u64>,\n    ) -> FileSlice {\n        self.0.file_slice_for_range(key_range, limit)\n    }\n}\n\n/// A TermDictionaryBuilder wrapping either an FST or a SSTable dictionary builder.\npub struct TermDictionaryBuilder<W: io::Write>(InnerTermDictBuilder<W>);\n\nimpl<W: io::Write> TermDictionaryBuilder<W> {\n    /// Creates a new `TermDictionaryBuilder`\n    pub fn create(w: W) -> io::Result<Self> {\n        InnerTermDictBuilder::create(w).map(TermDictionaryBuilder)\n    }\n\n    /// Inserts a `(key, value)` pair in the term dictionary.\n    ///\n    /// *Keys have to be inserted in order.*\n    pub fn insert<K: AsRef<[u8]>>(&mut self, key_ref: K, value: &TermInfo) -> io::Result<()> {\n        self.0.insert(key_ref, value)\n    }\n\n    /// # Warning\n    /// Horribly dangerous internal API\n    ///\n    /// If used, it must be used by systematically alternating calls\n    /// to insert_key and insert_value.\n    ///\n    /// Prefer using `.insert(key, value)`\n    pub fn insert_key(&mut self, key: &[u8]) -> io::Result<()> {\n        self.0.insert_key(key)\n    }\n\n    /// # Warning\n    ///\n    /// Horribly dangerous internal API. See `.insert_key(...)`.\n    pub fn insert_value(&mut self, term_info: &TermInfo) -> io::Result<()> {\n        self.0.insert_value(term_info)\n    }\n\n    /// Finalize writing the builder, and returns the underlying\n    /// `Write` object.\n    pub fn finish(self) -> io::Result<W> {\n        let mut writer = self.0.finish()?;\n        (CURRENT_TYPE as u32).serialize(&mut writer)?;\n        Ok(writer)\n    }\n}\n"
  },
  {
    "path": "src/termdict/sstable_termdict/merger.rs",
    "content": "use std::cmp::Ordering;\nuse std::collections::BinaryHeap;\n\nuse crate::postings::TermInfo;\nuse crate::termdict::TermStreamer;\n\npub struct HeapItem<'a> {\n    pub streamer: TermStreamer<'a>,\n    pub segment_ord: usize,\n}\n\nimpl<'a> PartialEq for HeapItem<'a> {\n    fn eq(&self, other: &Self) -> bool {\n        self.segment_ord == other.segment_ord\n    }\n}\n\nimpl<'a> Eq for HeapItem<'a> {}\n\nimpl<'a> PartialOrd for HeapItem<'a> {\n    fn partial_cmp(&self, other: &HeapItem<'a>) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl<'a> Ord for HeapItem<'a> {\n    fn cmp(&self, other: &HeapItem<'a>) -> Ordering {\n        (&other.streamer.key(), &other.segment_ord).cmp(&(&self.streamer.key(), &self.segment_ord))\n    }\n}\n\n/// Given a list of sorted term streams,\n/// returns an iterator over sorted unique terms.\n///\n/// The item yield is actually a pair with\n/// - the term\n/// - a slice with the ordinal of the segments containing the terms.\npub struct TermMerger<'a> {\n    heap: BinaryHeap<HeapItem<'a>>,\n    current_streamers: Vec<HeapItem<'a>>,\n}\n\nimpl<'a> TermMerger<'a> {\n    /// Stream of merged term dictionary\n    pub fn new(streams: Vec<TermStreamer<'a>>) -> TermMerger<'a> {\n        TermMerger {\n            heap: BinaryHeap::new(),\n            current_streamers: streams\n                .into_iter()\n                .enumerate()\n                .map(|(ord, streamer)| HeapItem {\n                    streamer,\n                    segment_ord: ord,\n                })\n                .collect(),\n        }\n    }\n\n    fn advance_segments(&mut self) {\n        let streamers = &mut self.current_streamers;\n        let heap = &mut self.heap;\n        for mut heap_item in streamers.drain(..) {\n            if heap_item.streamer.advance() {\n                heap.push(heap_item);\n            }\n        }\n    }\n\n    /// Advance the term iterator to the next term.\n    /// Returns true if there is indeed another term\n    /// False if there is none.\n    pub fn advance(&mut self) -> bool {\n        self.advance_segments();\n        let Some(head) = self.heap.pop() else {\n            return false;\n        };\n        self.current_streamers.push(head);\n        while let Some(next_streamer) = self.heap.peek() {\n            if self.current_streamers[0].streamer.key() != next_streamer.streamer.key() {\n                break;\n            }\n            let next_heap_it = self.heap.pop().unwrap(); // safe : we peeked beforehand\n            self.current_streamers.push(next_heap_it);\n        }\n        true\n    }\n\n    /// Returns the current term.\n    ///\n    /// This method may be called\n    /// if and only if advance() has been called before\n    /// and \"true\" was returned.\n    pub fn key(&self) -> &[u8] {\n        self.current_streamers[0].streamer.key()\n    }\n\n    /// Returns the sorted list of segment ordinals\n    /// that include the current term.\n    ///\n    /// This method may be called\n    /// if and only if advance() has been called before\n    /// and \"true\" was returned.\n    pub fn current_segment_ords_and_term_infos<'b: 'a>(\n        &'b self,\n    ) -> impl 'b + Iterator<Item = (usize, TermInfo)> {\n        self.current_streamers\n            .iter()\n            .map(|heap_item| (heap_item.segment_ord, heap_item.streamer.value().clone()))\n    }\n}\n"
  },
  {
    "path": "src/termdict/sstable_termdict/mod.rs",
    "content": "use std::io;\n\nmod merger;\n\nuse std::iter::ExactSizeIterator;\n\nuse common::VInt;\nuse sstable::value::{ValueReader, ValueWriter};\nuse sstable::SSTable;\nuse tantivy_fst::automaton::AlwaysMatch;\n\npub use self::merger::TermMerger;\nuse crate::postings::TermInfo;\n\n/// The term dictionary contains all of the terms in\n/// `tantivy index` in a sorted manner.\n///\n/// The `Fst` crate is used to associate terms to their\n/// respective `TermOrdinal`. The `TermInfoStore` then makes it\n/// possible to fetch the associated `TermInfo`.\npub type TermDictionary = sstable::Dictionary<TermSSTable>;\n\n/// Builder for the new term dictionary.\npub type TermDictionaryBuilder<W> = sstable::Writer<W, TermInfoValueWriter>;\n\n/// `TermStreamer` acts as a cursor over a range of terms of a segment.\n/// Terms are guaranteed to be sorted.\npub type TermStreamer<'a, A = AlwaysMatch> = sstable::Streamer<'a, TermSSTable, A>;\n\n/// SSTable used to store TermInfo objects.\n#[derive(Clone)]\npub struct TermSSTable;\n\npub type TermStreamerBuilder<'a, A = AlwaysMatch> = sstable::StreamerBuilder<'a, TermSSTable, A>;\n\nimpl SSTable for TermSSTable {\n    type Value = TermInfo;\n    type ValueReader = TermInfoValueReader;\n    type ValueWriter = TermInfoValueWriter;\n}\n\n#[derive(Default)]\npub struct TermInfoValueReader {\n    term_infos: Vec<TermInfo>,\n}\n\nimpl ValueReader for TermInfoValueReader {\n    type Value = TermInfo;\n\n    #[inline(always)]\n    fn value(&self, idx: usize) -> &TermInfo {\n        &self.term_infos[idx]\n    }\n\n    fn load(&mut self, mut data: &[u8]) -> io::Result<usize> {\n        let len_before = data.len();\n        self.term_infos.clear();\n        let num_els = VInt::deserialize_u64(&mut data)?;\n        let mut postings_start = VInt::deserialize_u64(&mut data)? as usize;\n        let mut positions_start = VInt::deserialize_u64(&mut data)? as usize;\n        for _ in 0..num_els {\n            let doc_freq = VInt::deserialize_u64(&mut data)? as u32;\n            let postings_num_bytes = VInt::deserialize_u64(&mut data)?;\n            let positions_num_bytes = VInt::deserialize_u64(&mut data)?;\n            let postings_end = postings_start + postings_num_bytes as usize;\n            let positions_end = positions_start + positions_num_bytes as usize;\n            let term_info = TermInfo {\n                doc_freq,\n                postings_range: postings_start..postings_end,\n                positions_range: positions_start..positions_end,\n            };\n            self.term_infos.push(term_info);\n            postings_start = postings_end;\n            positions_start = positions_end;\n        }\n        let consumed_len = len_before - data.len();\n        Ok(consumed_len)\n    }\n}\n\n#[derive(Default)]\npub struct TermInfoValueWriter {\n    term_infos: Vec<TermInfo>,\n}\n\nimpl ValueWriter for TermInfoValueWriter {\n    type Value = TermInfo;\n\n    fn write(&mut self, term_info: &TermInfo) {\n        self.term_infos.push(term_info.clone());\n    }\n\n    fn serialize_block(&self, buffer: &mut Vec<u8>) {\n        VInt(self.term_infos.len() as u64).serialize_into_vec(buffer);\n        if self.term_infos.is_empty() {\n            return;\n        }\n        VInt(self.term_infos[0].postings_range.start as u64).serialize_into_vec(buffer);\n        VInt(self.term_infos[0].positions_range.start as u64).serialize_into_vec(buffer);\n        for term_info in &self.term_infos {\n            VInt(term_info.doc_freq as u64).serialize_into_vec(buffer);\n            VInt(term_info.postings_range.len() as u64).serialize_into_vec(buffer);\n            VInt(term_info.positions_range.len() as u64).serialize_into_vec(buffer);\n        }\n    }\n\n    fn clear(&mut self) {\n        self.term_infos.clear();\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use sstable::value::{ValueReader, ValueWriter};\n\n    use crate::postings::TermInfo;\n    use crate::termdict::sstable_termdict::TermInfoValueReader;\n\n    #[test]\n    fn test_block_terminfos() {\n        let mut term_info_writer = super::TermInfoValueWriter::default();\n        term_info_writer.write(&TermInfo {\n            doc_freq: 120u32,\n            postings_range: 17..45,\n            positions_range: 10..122,\n        });\n        term_info_writer.write(&TermInfo {\n            doc_freq: 10u32,\n            postings_range: 45..450,\n            positions_range: 122..1100,\n        });\n        term_info_writer.write(&TermInfo {\n            doc_freq: 17u32,\n            postings_range: 450..462,\n            positions_range: 1100..1302,\n        });\n        let mut buffer = Vec::new();\n        term_info_writer.serialize_block(&mut buffer);\n        let mut term_info_reader = TermInfoValueReader::default();\n        let num_bytes: usize = term_info_reader.load(&buffer[..]).unwrap();\n        assert_eq!(\n            term_info_reader.value(0),\n            &TermInfo {\n                doc_freq: 120u32,\n                postings_range: 17..45,\n                positions_range: 10..122\n            }\n        );\n        assert_eq!(buffer.len(), num_bytes);\n    }\n}\n"
  },
  {
    "path": "src/termdict/tests.rs",
    "content": "use std::path::PathBuf;\nuse std::{io, str};\n\nuse super::{TermDictionary, TermDictionaryBuilder, TermStreamer};\nuse crate::directory::{Directory, FileSlice, RamDirectory, TerminatingWrite};\nuse crate::postings::TermInfo;\n\nconst BLOCK_SIZE: usize = 1_500;\n\nfn make_term_info(term_ord: u64) -> TermInfo {\n    let offset = |term_ord: u64| (term_ord * 100 + term_ord * term_ord) as usize;\n    TermInfo {\n        doc_freq: term_ord as u32,\n        postings_range: offset(term_ord)..offset(term_ord + 1),\n        positions_range: offset(term_ord) * 2..offset(term_ord + 1) * 2,\n    }\n}\n\n#[test]\nfn test_empty_term_dictionary() {\n    let empty = TermDictionary::empty();\n    assert!(empty.stream().unwrap().next().is_none());\n}\n\n#[test]\nfn test_term_ordinals() -> crate::Result<()> {\n    const COUNTRIES: [&str; 7] = [\n        \"San Marino\",\n        \"Serbia\",\n        \"Slovakia\",\n        \"Slovenia\",\n        \"Spain\",\n        \"Sweden\",\n        \"Switzerland\",\n    ];\n    let directory = RamDirectory::create();\n    let path = PathBuf::from(\"TermDictionary\");\n    {\n        let write = directory.open_write(&path)?;\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(write)?;\n        for term in COUNTRIES.iter() {\n            term_dictionary_builder.insert(term.as_bytes(), &make_term_info(0u64))?;\n        }\n        term_dictionary_builder.finish()?.terminate()?;\n    }\n    let term_file = directory.open_read(&path)?;\n    let term_dict: TermDictionary = TermDictionary::open(term_file)?;\n    for (term_ord, term) in COUNTRIES.iter().enumerate() {\n        assert_eq!(term_dict.term_ord(term)?, Some(term_ord as u64));\n        let mut bytes = vec![];\n        assert!(term_dict.ord_to_term(term_ord as u64, &mut bytes)?);\n        assert_eq!(bytes, term.as_bytes());\n    }\n    Ok(())\n}\n\n#[test]\nfn test_term_dictionary_simple() -> crate::Result<()> {\n    let directory = RamDirectory::create();\n    let path = PathBuf::from(\"TermDictionary\");\n    {\n        let write = directory.open_write(&path)?;\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(write)?;\n        term_dictionary_builder.insert(\"abc\".as_bytes(), &make_term_info(34u64))?;\n        term_dictionary_builder.insert(\"abcd\".as_bytes(), &make_term_info(346u64))?;\n        term_dictionary_builder.finish()?.terminate()?;\n    }\n    let file = directory.open_read(&path)?;\n    let term_dict: TermDictionary = TermDictionary::open(file)?;\n    assert_eq!(term_dict.get(\"abc\")?.unwrap().doc_freq, 34u32);\n    assert_eq!(term_dict.get(\"abcd\")?.unwrap().doc_freq, 346u32);\n    let mut stream = term_dict.stream()?;\n    {\n        {\n            let (k, v) = stream.next().unwrap();\n            assert_eq!(k, \"abc\".as_bytes());\n            assert_eq!(v.doc_freq, 34u32);\n        }\n        assert_eq!(stream.key(), \"abc\".as_bytes());\n        assert_eq!(stream.value().doc_freq, 34u32);\n    }\n    {\n        {\n            let (k, v) = stream.next().unwrap();\n            assert_eq!(k, \"abcd\".as_bytes());\n            assert_eq!(v.doc_freq, 346u32);\n        }\n        assert_eq!(stream.key(), \"abcd\".as_bytes());\n        assert_eq!(stream.value().doc_freq, 346u32);\n    }\n    assert!(!stream.advance());\n    Ok(())\n}\n\n#[test]\nfn test_term_dictionary_stream() -> crate::Result<()> {\n    let ids: Vec<_> = (0u32..10_000u32)\n        .map(|i| (format!(\"doc{i:0>6}\"), i))\n        .collect();\n    let buffer: Vec<u8> = {\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(vec![]).unwrap();\n        for (id, i) in &ids {\n            term_dictionary_builder\n                .insert(id.as_bytes(), &make_term_info(*i as u64))\n                .unwrap();\n        }\n        term_dictionary_builder.finish()?\n    };\n    let term_file = FileSlice::from(buffer);\n    let term_dictionary: TermDictionary = TermDictionary::open(term_file)?;\n    {\n        let mut streamer = term_dictionary.stream()?;\n        let mut i = 0;\n        while let Some((streamer_k, streamer_v)) = streamer.next() {\n            let (key, v) = &ids[i];\n            assert_eq!(streamer_k, key.as_bytes());\n            assert_eq!(streamer_v, &make_term_info(*v as u64));\n            i += 1;\n        }\n    }\n\n    let (key, val) = &ids[2047];\n    assert_eq!(\n        term_dictionary.get(key.as_bytes())?,\n        Some(make_term_info(*val as u64))\n    );\n    Ok(())\n}\n\n#[test]\nfn test_stream_high_range_prefix_suffix() -> crate::Result<()> {\n    let buffer: Vec<u8> = {\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(vec![]).unwrap();\n        // term requires more than 16bits\n        term_dictionary_builder.insert(\"abcdefghijklmnopqrstuvwxy\", &make_term_info(1))?;\n        term_dictionary_builder.insert(\"abcdefghijklmnopqrstuvwxyz\", &make_term_info(2))?;\n        term_dictionary_builder.insert(\"abr\", &make_term_info(3))?;\n        term_dictionary_builder.finish()?\n    };\n    let term_dict_file = FileSlice::from(buffer);\n    let term_dictionary: TermDictionary = TermDictionary::open(term_dict_file)?;\n    let mut kv_stream = term_dictionary.stream()?;\n    assert!(kv_stream.advance());\n    assert_eq!(kv_stream.key(), \"abcdefghijklmnopqrstuvwxy\".as_bytes());\n    assert_eq!(kv_stream.value(), &make_term_info(1));\n    assert!(kv_stream.advance());\n    assert_eq!(kv_stream.key(), \"abcdefghijklmnopqrstuvwxyz\".as_bytes());\n    assert_eq!(kv_stream.value(), &make_term_info(2));\n    assert!(kv_stream.advance());\n    assert_eq!(kv_stream.key(), \"abr\".as_bytes());\n    assert_eq!(kv_stream.value(), &make_term_info(3));\n    assert!(!kv_stream.advance());\n    Ok(())\n}\n\n#[test]\nfn test_stream_range() -> crate::Result<()> {\n    let ids: Vec<_> = (0u32..10_000u32)\n        .map(|i| (format!(\"doc{i:0>6}\"), i))\n        .collect();\n    let buffer: Vec<u8> = {\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(vec![]).unwrap();\n        for (id, i) in &ids {\n            term_dictionary_builder\n                .insert(id.as_bytes(), &make_term_info(*i as u64))\n                .unwrap();\n        }\n        term_dictionary_builder.finish()?\n    };\n\n    let file = FileSlice::from(buffer);\n\n    let term_dictionary: TermDictionary = TermDictionary::open(file)?;\n    {\n        for i in (0..20).chain(6000..8_000) {\n            let (target_key, _) = &ids[i];\n            let mut streamer = term_dictionary\n                .range()\n                .ge(target_key.as_bytes())\n                .into_stream()?;\n            for j in 0..3 {\n                let (streamer_k, streamer_v) = streamer.next().unwrap();\n                let (key, v) = &ids[i + j];\n                assert_eq!(str::from_utf8(streamer_k).unwrap(), key);\n                assert_eq!(streamer_v.doc_freq, *v);\n                assert_eq!(streamer_v, &make_term_info(*v as u64));\n            }\n        }\n    }\n\n    {\n        for i in (0..20).chain(BLOCK_SIZE - 10..BLOCK_SIZE + 10) {\n            let (target_key, _) = &ids[i];\n            let mut streamer = term_dictionary\n                .range()\n                .gt(target_key.as_bytes())\n                .into_stream()?;\n            for j in 0..3 {\n                let (streamer_k, streamer_v) = streamer.next().unwrap();\n                let (key, v) = &ids[i + j + 1];\n                assert_eq!(streamer_k, key.as_bytes());\n                assert_eq!(streamer_v.doc_freq, *v);\n            }\n        }\n    }\n\n    {\n        for i in (0..20).chain(BLOCK_SIZE - 10..BLOCK_SIZE + 10) {\n            for j in 0..3 {\n                let (fst_key, _) = &ids[i];\n                let (last_key, _) = &ids[i + j];\n                let mut streamer = term_dictionary\n                    .range()\n                    .ge(fst_key.as_bytes())\n                    .lt(last_key.as_bytes())\n                    .into_stream()?;\n                for _ in 0..j {\n                    assert!(streamer.next().is_some());\n                }\n                assert!(streamer.next().is_none());\n            }\n        }\n    }\n    Ok(())\n}\n\n#[test]\nfn test_empty_string() -> crate::Result<()> {\n    let buffer: Vec<u8> = {\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(vec![]).unwrap();\n        term_dictionary_builder\n            .insert([], &make_term_info(1_u64))\n            .unwrap();\n        term_dictionary_builder\n            .insert([1u8], &make_term_info(2_u64))\n            .unwrap();\n        term_dictionary_builder.finish()?\n    };\n    let file = FileSlice::from(buffer);\n    let term_dictionary: TermDictionary = TermDictionary::open(file)?;\n    let mut stream = term_dictionary.stream()?;\n    assert!(stream.advance());\n    assert!(stream.key().is_empty());\n    assert!(stream.advance());\n    assert_eq!(stream.key(), &[1u8]);\n    assert!(!stream.advance());\n    Ok(())\n}\n\nfn stream_range_test_dict() -> io::Result<TermDictionary> {\n    let buffer: Vec<u8> = {\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(Vec::new())?;\n        for i in 0u8..10u8 {\n            let number_arr = [i; 1];\n            term_dictionary_builder.insert(number_arr, &make_term_info(i as u64))?;\n        }\n        term_dictionary_builder.finish()?\n    };\n    let file = FileSlice::from(buffer);\n    TermDictionary::open(file)\n}\n\n#[test]\nfn test_stream_range_boundaries_forward() -> crate::Result<()> {\n    let term_dictionary = stream_range_test_dict()?;\n    let value_list = |mut streamer: TermStreamer<'_>| {\n        let mut res: Vec<u32> = vec![];\n        while let Some((_, v)) = streamer.next() {\n            res.push(v.doc_freq);\n        }\n        res\n    };\n    {\n        let range = term_dictionary.range().ge([2u8]).into_stream()?;\n        assert_eq!(\n            value_list(range),\n            vec![2u32, 3u32, 4u32, 5u32, 6u32, 7u32, 8u32, 9u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().gt([2u8]).into_stream()?;\n        assert_eq!(\n            value_list(range),\n            vec![3u32, 4u32, 5u32, 6u32, 7u32, 8u32, 9u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().lt([6u8]).into_stream()?;\n        assert_eq!(value_list(range), vec![0u32, 1u32, 2u32, 3u32, 4u32, 5u32]);\n    }\n    {\n        let range = term_dictionary.range().le([6u8]).into_stream()?;\n        assert_eq!(\n            value_list(range),\n            vec![0u32, 1u32, 2u32, 3u32, 4u32, 5u32, 6u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().ge([0u8]).lt([5u8]).into_stream()?;\n        assert_eq!(value_list(range), vec![0u32, 1u32, 2u32, 3u32, 4u32]);\n    }\n    Ok(())\n}\n\n#[cfg(not(feature = \"quickwit\"))]\n#[test]\nfn test_stream_range_boundaries_backward() -> crate::Result<()> {\n    let term_dictionary = stream_range_test_dict()?;\n    let value_list_backward = |mut streamer: TermStreamer<'_>| {\n        let mut res: Vec<u32> = vec![];\n        while let Some((_, v)) = streamer.next() {\n            res.push(v.doc_freq);\n        }\n        res.reverse();\n        res\n    };\n    {\n        let range = term_dictionary.range().backward().into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![0u32, 1u32, 2u32, 3u32, 4u32, 5u32, 6u32, 7u32, 8u32, 9u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().ge([2u8]).backward().into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![2u32, 3u32, 4u32, 5u32, 6u32, 7u32, 8u32, 9u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().gt([2u8]).backward().into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![3u32, 4u32, 5u32, 6u32, 7u32, 8u32, 9u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().lt([6u8]).backward().into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![0u32, 1u32, 2u32, 3u32, 4u32, 5u32]\n        );\n    }\n    {\n        let range = term_dictionary.range().le([6u8]).backward().into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![0u32, 1u32, 2u32, 3u32, 4u32, 5u32, 6u32]\n        );\n    }\n    {\n        let range = term_dictionary\n            .range()\n            .ge([0u8])\n            .lt([5u8])\n            .backward()\n            .into_stream()?;\n        assert_eq!(\n            value_list_backward(range),\n            vec![0u32, 1u32, 2u32, 3u32, 4u32]\n        );\n    }\n    Ok(())\n}\n\n#[test]\nfn test_ord_to_term() -> crate::Result<()> {\n    let termdict = stream_range_test_dict()?;\n    let mut bytes = vec![];\n    for b in 0u8..10u8 {\n        termdict.ord_to_term(b as u64, &mut bytes)?;\n        assert_eq!(&bytes, &[b]);\n    }\n    Ok(())\n}\n\n#[test]\nfn test_stream_term_ord() -> crate::Result<()> {\n    let termdict = stream_range_test_dict()?;\n    let mut stream = termdict.stream()?;\n    for b in 0u8..10u8 {\n        assert!(stream.advance());\n        assert_eq!(stream.term_ord(), b as u64);\n        assert_eq!(stream.key(), &[b]);\n    }\n    assert!(!stream.advance());\n    Ok(())\n}\n\n#[test]\nfn test_automaton_search() -> crate::Result<()> {\n    use levenshtein_automata::LevenshteinAutomatonBuilder;\n\n    use crate::query::DfaWrapper;\n\n    const COUNTRIES: [&str; 7] = [\n        \"San Marino\",\n        \"Serbia\",\n        \"Slovakia\",\n        \"Slovenia\",\n        \"Spain\",\n        \"Sweden\",\n        \"Switzerland\",\n    ];\n\n    let directory = RamDirectory::create();\n    let path = PathBuf::from(\"TermDictionary\");\n    {\n        let write = directory.open_write(&path)?;\n        let mut term_dictionary_builder = TermDictionaryBuilder::create(write)?;\n        for term in COUNTRIES.iter() {\n            term_dictionary_builder.insert(term.as_bytes(), &make_term_info(0u64))?;\n        }\n        term_dictionary_builder.finish()?.terminate()?;\n    }\n    let file = directory.open_read(&path)?;\n    let term_dict: TermDictionary = TermDictionary::open(file)?;\n\n    // We can now build an entire dfa.\n    let lev_automaton_builder = LevenshteinAutomatonBuilder::new(2, true);\n    let automaton = DfaWrapper(lev_automaton_builder.build_dfa(\"Spaen\"));\n\n    let mut range = term_dict.search(automaton).into_stream()?;\n\n    // get the first finding\n    assert!(range.advance());\n    assert_eq!(\"Spain\".as_bytes(), range.key());\n    assert!(!range.advance());\n    Ok(())\n}\n"
  },
  {
    "path": "src/tokenizer/alphanum_only.rs",
    "content": "//! # Example\n//! ```rust\n//! use tantivy::tokenizer::*;\n//!\n//! let mut tokenizer = TextAnalyzer::builder(RawTokenizer::default())\n//!   .filter(AlphaNumOnlyFilter)\n//!   .build();\n//!\n//! let mut stream = tokenizer.token_stream(\"hello there\");\n//! // is none because the raw filter emits one token that\n//! // contains a space\n//! assert!(stream.next().is_none());\n//!\n//! let mut tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n//!   .filter(AlphaNumOnlyFilter)\n//!   .build();\n//!\n//! let mut stream = tokenizer.token_stream(\"hello there 💣\");\n//! assert!(stream.next().is_some());\n//! assert!(stream.next().is_some());\n//! // the \"emoji\" is dropped because its not an alphanum\n//! assert!(stream.next().is_none());\n//! ```\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// `TokenFilter` that removes all tokens that contain non\n/// ascii alphanumeric characters.\n#[derive(Clone)]\npub struct AlphaNumOnlyFilter;\n\npub struct AlphaNumOnlyFilterStream<T> {\n    tail: T,\n}\n\nimpl<T> AlphaNumOnlyFilterStream<T> {\n    fn predicate(&self, token: &Token) -> bool {\n        token.text.chars().all(|c| c.is_ascii_alphanumeric())\n    }\n}\n\nimpl TokenFilter for AlphaNumOnlyFilter {\n    type Tokenizer<T: Tokenizer> = AlphaNumOnlyFilterWrapper<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> AlphaNumOnlyFilterWrapper<T> {\n        AlphaNumOnlyFilterWrapper(tokenizer)\n    }\n}\n\n#[derive(Clone)]\npub struct AlphaNumOnlyFilterWrapper<T>(T);\n\nimpl<T: Tokenizer> Tokenizer for AlphaNumOnlyFilterWrapper<T> {\n    type TokenStream<'a> = AlphaNumOnlyFilterStream<T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        AlphaNumOnlyFilterStream {\n            tail: self.0.token_stream(text),\n        }\n    }\n}\n\nimpl<T: TokenStream> TokenStream for AlphaNumOnlyFilterStream<T> {\n    fn advance(&mut self) -> bool {\n        while self.tail.advance() {\n            if self.predicate(self.tail.token()) {\n                return true;\n            }\n        }\n\n        false\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{AlphaNumOnlyFilter, SimpleTokenizer, TextAnalyzer, Token};\n\n    #[test]\n    fn test_alphanum_only() {\n        let tokens = token_stream_helper(\"I am a cat. 我輩は猫である。(1906)\");\n        assert_eq!(tokens.len(), 5);\n        assert_token(&tokens[0], 0, \"I\", 0, 1);\n        assert_token(&tokens[1], 1, \"am\", 2, 4);\n        assert_token(&tokens[2], 2, \"a\", 5, 6);\n        assert_token(&tokens[3], 3, \"cat\", 7, 10);\n        assert_token(&tokens[4], 5, \"1906\", 37, 41);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut a = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(AlphaNumOnlyFilter)\n            .build();\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/ascii_folding_filter.rs",
    "content": "use std::mem;\n\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// This class converts alphabetic, numeric, and symbolic Unicode characters\n/// which are not in the first 127 ASCII characters (the \"Basic Latin\" Unicode\n/// block) into their ASCII equivalents, if one exists.\n#[derive(Clone)]\npub struct AsciiFoldingFilter;\n\nimpl TokenFilter for AsciiFoldingFilter {\n    type Tokenizer<T: Tokenizer> = AsciiFoldingFilterWrapper<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> AsciiFoldingFilterWrapper<T> {\n        AsciiFoldingFilterWrapper {\n            tokenizer,\n            buffer: String::new(),\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct AsciiFoldingFilterWrapper<T> {\n    tokenizer: T,\n    buffer: String,\n}\n\nimpl<T: Tokenizer> Tokenizer for AsciiFoldingFilterWrapper<T> {\n    type TokenStream<'a> = AsciiFoldingFilterTokenStream<'a, T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        self.buffer.clear();\n        AsciiFoldingFilterTokenStream {\n            buffer: &mut self.buffer,\n            tail: self.tokenizer.token_stream(text),\n        }\n    }\n}\n\npub struct AsciiFoldingFilterTokenStream<'a, T> {\n    buffer: &'a mut String,\n    tail: T,\n}\n\nimpl<T: TokenStream> TokenStream for AsciiFoldingFilterTokenStream<'_, T> {\n    fn advance(&mut self) -> bool {\n        if !self.tail.advance() {\n            return false;\n        }\n        if !self.token_mut().text.is_ascii() {\n            // ignore its already ascii\n            to_ascii(&self.tail.token().text, self.buffer);\n            mem::swap(&mut self.tail.token_mut().text, self.buffer);\n        }\n        true\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n// Returns a string that represents the ascii folded version of\n// the character. If the `char` does not require ascii folding\n// (e.g. simple ASCII chars like `A`) or if the `char`\n// does not have a sensible ascii equivalent (e.g.: Kanjis like 馬,\n// this function returns `None`.\nfn fold_non_ascii_char(c: char) -> Option<&'static str> {\n    match c {\n        '\\u{00C0}' | // À  [LATIN CAPITAL LETTER A WITH GRAVE]\n        '\\u{00C1}' | // Á  [LATIN CAPITAL LETTER A WITH ACUTE]\n        '\\u{00C2}' | //   [LATIN CAPITAL LETTER A WITH CIRCUMFLEX]\n        '\\u{00C3}' | // à  [LATIN CAPITAL LETTER A WITH TILDE]\n        '\\u{00C4}' | // Ä  [LATIN CAPITAL LETTER A WITH DIAERESIS]\n        '\\u{00C5}' | // Å  [LATIN CAPITAL LETTER A WITH RING ABOVE]\n        '\\u{0100}' | // Ā  [LATIN CAPITAL LETTER A WITH MACRON]\n        '\\u{0102}' | // Ă  [LATIN CAPITAL LETTER A WITH BREVE]\n        '\\u{0104}' | // Ą  [LATIN CAPITAL LETTER A WITH OGONEK]\n        '\\u{018F}' | // Ə  http://en.wikipedia.org/wiki/Schwa  [LATIN CAPITAL LETTER SCHWA]\n        '\\u{01CD}' | // Ǎ  [LATIN CAPITAL LETTER A WITH CARON]\n        '\\u{01DE}' | // Ǟ  [LATIN CAPITAL LETTER A WITH DIAERESIS AND MACRON]\n        '\\u{01E0}' | // Ǡ  [LATIN CAPITAL LETTER A WITH DOT ABOVE AND MACRON]\n        '\\u{01FA}' | // Ǻ  [LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE]\n        '\\u{0200}' | // Ȁ  [LATIN CAPITAL LETTER A WITH DOUBLE GRAVE]\n        '\\u{0202}' | // Ȃ  [LATIN CAPITAL LETTER A WITH INVERTED BREVE]\n        '\\u{0226}' | // Ȧ  [LATIN CAPITAL LETTER A WITH DOT ABOVE]\n        '\\u{023A}' | // Ⱥ  [LATIN CAPITAL LETTER A WITH STROKE]\n        '\\u{1D00}' | // ᴀ  [LATIN LETTER SMALL CAPITAL A]\n        '\\u{1E00}' | // Ḁ  [LATIN CAPITAL LETTER A WITH RING BELOW]\n        '\\u{1EA0}' | // Ạ  [LATIN CAPITAL LETTER A WITH DOT BELOW]\n        '\\u{1EA2}' | // Ả  [LATIN CAPITAL LETTER A WITH HOOK ABOVE]\n        '\\u{1EA4}' | // Ấ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1EA6}' | // Ầ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1EA8}' | // Ẩ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1EAA}' | // Ẫ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND TILDE]\n        '\\u{1EAC}' | // Ậ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{1EAE}' | // Ắ  [LATIN CAPITAL LETTER A WITH BREVE AND ACUTE]\n        '\\u{1EB0}' | // Ằ  [LATIN CAPITAL LETTER A WITH BREVE AND GRAVE]\n        '\\u{1EB2}' | // Ẳ  [LATIN CAPITAL LETTER A WITH BREVE AND HOOK ABOVE]\n        '\\u{1EB4}' | // Ẵ  [LATIN CAPITAL LETTER A WITH BREVE AND TILDE]\n        '\\u{1EB6}' | // Ặ  [LATIN CAPITAL LETTER A WITH BREVE AND DOT BELOW]\n        '\\u{24B6}' | // Ⓐ  [CIRCLED LATIN CAPITAL LETTER A]\n        '\\u{FF21}'  // Ａ  [FULLWIDTH LATIN CAPITAL LETTER A]\n        => Some(\"A\"),\n        '\\u{00E0}' | // à  [LATIN SMALL LETTER A WITH GRAVE]\n        '\\u{00E1}' | // á  [LATIN SMALL LETTER A WITH ACUTE]\n        '\\u{00E2}' | // â  [LATIN SMALL LETTER A WITH CIRCUMFLEX]\n        '\\u{00E3}' | // ã  [LATIN SMALL LETTER A WITH TILDE]\n        '\\u{00E4}' | // ä  [LATIN SMALL LETTER A WITH DIAERESIS]\n        '\\u{00E5}' | // å  [LATIN SMALL LETTER A WITH RING ABOVE]\n        '\\u{0101}' | // ā  [LATIN SMALL LETTER A WITH MACRON]\n        '\\u{0103}' | // ă  [LATIN SMALL LETTER A WITH BREVE]\n        '\\u{0105}' | // ą  [LATIN SMALL LETTER A WITH OGONEK]\n        '\\u{01CE}' | // ǎ  [LATIN SMALL LETTER A WITH CARON]\n        '\\u{01DF}' | // ǟ  [LATIN SMALL LETTER A WITH DIAERESIS AND MACRON]\n        '\\u{01E1}' | // ǡ  [LATIN SMALL LETTER A WITH DOT ABOVE AND MACRON]\n        '\\u{01FB}' | // ǻ  [LATIN SMALL LETTER A WITH RING ABOVE AND ACUTE]\n        '\\u{0201}' | // ȁ  [LATIN SMALL LETTER A WITH DOUBLE GRAVE]\n        '\\u{0203}' | // ȃ  [LATIN SMALL LETTER A WITH INVERTED BREVE]\n        '\\u{0227}' | // ȧ  [LATIN SMALL LETTER A WITH DOT ABOVE]\n        '\\u{0250}' | // ɐ  [LATIN SMALL LETTER TURNED A]\n        '\\u{0259}' | // ə  [LATIN SMALL LETTER SCHWA]\n        '\\u{025A}' | // ɚ  [LATIN SMALL LETTER SCHWA WITH HOOK]\n        '\\u{1D8F}' | // ᶏ  [LATIN SMALL LETTER A WITH RETROFLEX HOOK]\n        '\\u{1D95}' | // ᶕ  [LATIN SMALL LETTER SCHWA WITH RETROFLEX HOOK]\n        '\\u{1E01}' | // ạ  [LATIN SMALL LETTER A WITH RING BELOW]\n        '\\u{1E9A}' | // ả  [LATIN SMALL LETTER A WITH RIGHT HALF RING]\n        '\\u{1EA1}' | // ạ  [LATIN SMALL LETTER A WITH DOT BELOW]\n        '\\u{1EA3}' | // ả  [LATIN SMALL LETTER A WITH HOOK ABOVE]\n        '\\u{1EA5}' | // ấ  [LATIN SMALL LETTER A WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1EA7}' | // ầ  [LATIN SMALL LETTER A WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1EA9}' | // ẩ  [LATIN SMALL LETTER A WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1EAB}' | // ẫ  [LATIN SMALL LETTER A WITH CIRCUMFLEX AND TILDE]\n        '\\u{1EAD}' | // ậ  [LATIN SMALL LETTER A WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{1EAF}' | // ắ  [LATIN SMALL LETTER A WITH BREVE AND ACUTE]\n        '\\u{1EB1}' | // ằ  [LATIN SMALL LETTER A WITH BREVE AND GRAVE]\n        '\\u{1EB3}' | // ẳ  [LATIN SMALL LETTER A WITH BREVE AND HOOK ABOVE]\n        '\\u{1EB5}' | // ẵ  [LATIN SMALL LETTER A WITH BREVE AND TILDE]\n        '\\u{1EB7}' | // ặ  [LATIN SMALL LETTER A WITH BREVE AND DOT BELOW]\n        '\\u{2090}' | // ₐ  [LATIN SUBSCRIPT SMALL LETTER A]\n        '\\u{2094}' | // ₔ  [LATIN SUBSCRIPT SMALL LETTER SCHWA]\n        '\\u{24D0}' | // ⓐ  [CIRCLED LATIN SMALL LETTER A]\n        '\\u{2C65}' | // ⱥ  [LATIN SMALL LETTER A WITH STROKE]\n        '\\u{2C6F}' | // Ɐ  [LATIN CAPITAL LETTER TURNED A]\n        '\\u{FF41}'  // ａ  [FULLWIDTH LATIN SMALL LETTER A]\n        => Some(\"a\"),\n        '\\u{A732}'  // Ꜳ  [LATIN CAPITAL LETTER AA]\n        => Some(\"AA\"),\n        '\\u{00C6}' | // Æ  [LATIN CAPITAL LETTER AE]\n        '\\u{01E2}' | // Ǣ  [LATIN CAPITAL LETTER AE WITH MACRON]\n        '\\u{01FC}' | // Ǽ  [LATIN CAPITAL LETTER AE WITH ACUTE]\n        '\\u{1D01}' // ᴁ  [LATIN LETTER SMALL CAPITAL AE]\n        => Some(\"AE\"),\n        '\\u{A734}' // Ꜵ  [LATIN CAPITAL LETTER AO]\n        => Some(\"AO\"),\n        '\\u{A736}'  // Ꜷ  [LATIN CAPITAL LETTER AU]\n        => Some(\"AU\"),\n        '\\u{A738}' | // Ꜹ  [LATIN CAPITAL LETTER AV]\n        '\\u{A73A}'  // Ꜻ  [LATIN CAPITAL LETTER AV WITH HORIZONTAL BAR]\n        => Some(\"AV\"),\n        '\\u{A73C}'  // Ꜽ  [LATIN CAPITAL LETTER AY]\n        => Some(\"AY\"),\n        '\\u{249C}'  // ⒜  [PARENTHESIZED LATIN SMALL LETTER A]\n        => Some(\"(a)\"),\n        '\\u{A733}' // ꜳ  [LATIN SMALL LETTER AA]\n        => Some(\"aa\"),\n        '\\u{00E6}' | // æ  [LATIN SMALL LETTER AE]\n        '\\u{01E3}' | // ǣ  [LATIN SMALL LETTER AE WITH MACRON]\n        '\\u{01FD}' | // ǽ  [LATIN SMALL LETTER AE WITH ACUTE]\n        '\\u{1D02}' // ᴂ  [LATIN SMALL LETTER TURNED AE]\n        => Some(\"ae\"),\n        '\\u{A735}' // ꜵ  [LATIN SMALL LETTER AO]\n        => Some(\"ao\"),\n        '\\u{A737}' // ꜷ  [LATIN SMALL LETTER AU]\n        => Some(\"au\"),\n        '\\u{A739}' | // ꜹ  [LATIN SMALL LETTER AV]\n        '\\u{A73B}' // ꜻ  [LATIN SMALL LETTER AV WITH HORIZONTAL BAR]\n        => Some(\"av\"),\n        '\\u{A73D}' // ꜽ  [LATIN SMALL LETTER AY]\n        => Some(\"ay\"),\n        '\\u{0181}' | // Ɓ  [LATIN CAPITAL LETTER B WITH HOOK]\n        '\\u{0182}' | // Ƃ  [LATIN CAPITAL LETTER B WITH TOPBAR]\n        '\\u{0243}' | // Ƀ  [LATIN CAPITAL LETTER B WITH STROKE]\n        '\\u{0299}' | // ʙ  [LATIN LETTER SMALL CAPITAL B]\n        '\\u{1D03}' | // ᴃ  [LATIN LETTER SMALL CAPITAL BARRED B]\n        '\\u{1E02}' | // Ḃ  [LATIN CAPITAL LETTER B WITH DOT ABOVE]\n        '\\u{1E04}' | // Ḅ  [LATIN CAPITAL LETTER B WITH DOT BELOW]\n        '\\u{1E06}' | // Ḇ  [LATIN CAPITAL LETTER B WITH LINE BELOW]\n        '\\u{24B7}' | // Ⓑ  [CIRCLED LATIN CAPITAL LETTER B]\n        '\\u{FF22}' // Ｂ  [FULLWIDTH LATIN CAPITAL LETTER B]\n        => Some(\"B\"),\n        '\\u{0180}' | // ƀ  [LATIN SMALL LETTER B WITH STROKE]\n        '\\u{0183}' | // ƃ  [LATIN SMALL LETTER B WITH TOPBAR]\n        '\\u{0253}' | // ɓ  [LATIN SMALL LETTER B WITH HOOK]\n        '\\u{1D6C}' | // ᵬ  [LATIN SMALL LETTER B WITH MIDDLE TILDE]\n        '\\u{1D80}' | // ᶀ  [LATIN SMALL LETTER B WITH PALATAL HOOK]\n        '\\u{1E03}' | // ḃ  [LATIN SMALL LETTER B WITH DOT ABOVE]\n        '\\u{1E05}' | // ḅ  [LATIN SMALL LETTER B WITH DOT BELOW]\n        '\\u{1E07}' | // ḇ  [LATIN SMALL LETTER B WITH LINE BELOW]\n        '\\u{24D1}' | // ⓑ  [CIRCLED LATIN SMALL LETTER B]\n        '\\u{FF42}' // ｂ  [FULLWIDTH LATIN SMALL LETTER B]\n        => Some(\"b\"),\n        '\\u{249D}' // ⒝  [PARENTHESIZED LATIN SMALL LETTER B]\n        => Some(\"(b)\"),\n        '\\u{00C7}' | // Ç  [LATIN CAPITAL LETTER C WITH CEDILLA]\n        '\\u{0106}' | // Ć  [LATIN CAPITAL LETTER C WITH ACUTE]\n        '\\u{0108}' | // Ĉ  [LATIN CAPITAL LETTER C WITH CIRCUMFLEX]\n        '\\u{010A}' | // Ċ  [LATIN CAPITAL LETTER C WITH DOT ABOVE]\n        '\\u{010C}' | // Č  [LATIN CAPITAL LETTER C WITH CARON]\n        '\\u{0187}' | // Ƈ  [LATIN CAPITAL LETTER C WITH HOOK]\n        '\\u{023B}' | // Ȼ  [LATIN CAPITAL LETTER C WITH STROKE]\n        '\\u{0297}' | // ʗ  [LATIN LETTER STRETCHED C]\n        '\\u{1D04}' | // ᴄ  [LATIN LETTER SMALL CAPITAL C]\n        '\\u{1E08}' | // Ḉ  [LATIN CAPITAL LETTER C WITH CEDILLA AND ACUTE]\n        '\\u{24B8}' | // Ⓒ  [CIRCLED LATIN CAPITAL LETTER C]\n        '\\u{FF23}' // Ｃ  [FULLWIDTH LATIN CAPITAL LETTER C]\n        => Some(\"C\"),\n        '\\u{00E7}' | // ç  [LATIN SMALL LETTER C WITH CEDILLA]\n        '\\u{0107}' | // ć  [LATIN SMALL LETTER C WITH ACUTE]\n        '\\u{0109}' | // ĉ  [LATIN SMALL LETTER C WITH CIRCUMFLEX]\n        '\\u{010B}' | // ċ  [LATIN SMALL LETTER C WITH DOT ABOVE]\n        '\\u{010D}' | // č  [LATIN SMALL LETTER C WITH CARON]\n        '\\u{0188}' | // ƈ  [LATIN SMALL LETTER C WITH HOOK]\n        '\\u{023C}' | // ȼ  [LATIN SMALL LETTER C WITH STROKE]\n        '\\u{0255}' | // ɕ  [LATIN SMALL LETTER C WITH CURL]\n        '\\u{1E09}' | // ḉ  [LATIN SMALL LETTER C WITH CEDILLA AND ACUTE]\n        '\\u{2184}' | // ↄ  [LATIN SMALL LETTER REVERSED C]\n        '\\u{24D2}' | // ⓒ  [CIRCLED LATIN SMALL LETTER C]\n        '\\u{A73E}' | // Ꜿ  [LATIN CAPITAL LETTER REVERSED C WITH DOT]\n        '\\u{A73F}' | // ꜿ  [LATIN SMALL LETTER REVERSED C WITH DOT]\n        '\\u{FF43}' // ｃ  [FULLWIDTH LATIN SMALL LETTER C]\n        => Some(\"c\"),\n        '\\u{249E}' // ⒞  [PARENTHESIZED LATIN SMALL LETTER C]\n        => Some(\"(c)\"),\n        '\\u{00D0}' | // Ð  [LATIN CAPITAL LETTER ETH]\n        '\\u{010E}' | // Ď  [LATIN CAPITAL LETTER D WITH CARON]\n        '\\u{0110}' | // Đ  [LATIN CAPITAL LETTER D WITH STROKE]\n        '\\u{0189}' | // Ɖ  [LATIN CAPITAL LETTER AFRICAN D]\n        '\\u{018A}' | // Ɗ  [LATIN CAPITAL LETTER D WITH HOOK]\n        '\\u{018B}' | // Ƌ  [LATIN CAPITAL LETTER D WITH TOPBAR]\n        '\\u{1D05}' | // ᴅ  [LATIN LETTER SMALL CAPITAL D]\n        '\\u{1D06}' | // ᴆ  [LATIN LETTER SMALL CAPITAL ETH]\n        '\\u{1E0A}' | // Ḋ  [LATIN CAPITAL LETTER D WITH DOT ABOVE]\n        '\\u{1E0C}' | // Ḍ  [LATIN CAPITAL LETTER D WITH DOT BELOW]\n        '\\u{1E0E}' | // Ḏ  [LATIN CAPITAL LETTER D WITH LINE BELOW]\n        '\\u{1E10}' | // Ḑ  [LATIN CAPITAL LETTER D WITH CEDILLA]\n        '\\u{1E12}' | // Ḓ  [LATIN CAPITAL LETTER D WITH CIRCUMFLEX BELOW]\n        '\\u{24B9}' | // Ⓓ  [CIRCLED LATIN CAPITAL LETTER D]\n        '\\u{A779}' | // Ꝺ  [LATIN CAPITAL LETTER INSULAR D]\n        '\\u{FF24}' // Ｄ  [FULLWIDTH LATIN CAPITAL LETTER D]\n        => Some(\"D\"),\n        '\\u{00F0}' | // ð  [LATIN SMALL LETTER ETH]\n        '\\u{010F}' | // ď  [LATIN SMALL LETTER D WITH CARON]\n        '\\u{0111}' | // đ  [LATIN SMALL LETTER D WITH STROKE]\n        '\\u{018C}' | // ƌ  [LATIN SMALL LETTER D WITH TOPBAR]\n        '\\u{0221}' | // ȡ  [LATIN SMALL LETTER D WITH CURL]\n        '\\u{0256}' | // ɖ  [LATIN SMALL LETTER D WITH TAIL]\n        '\\u{0257}' | // ɗ  [LATIN SMALL LETTER D WITH HOOK]\n        '\\u{1D6D}' | // ᵭ  [LATIN SMALL LETTER D WITH MIDDLE TILDE]\n        '\\u{1D81}' | // ᶁ  [LATIN SMALL LETTER D WITH PALATAL HOOK]\n        '\\u{1D91}' | // ᶑ  [LATIN SMALL LETTER D WITH HOOK AND TAIL]\n        '\\u{1E0B}' | // ḋ  [LATIN SMALL LETTER D WITH DOT ABOVE]\n        '\\u{1E0D}' | // ḍ  [LATIN SMALL LETTER D WITH DOT BELOW]\n        '\\u{1E0F}' | // ḏ  [LATIN SMALL LETTER D WITH LINE BELOW]\n        '\\u{1E11}' | // ḑ  [LATIN SMALL LETTER D WITH CEDILLA]\n        '\\u{1E13}' | // ḓ  [LATIN SMALL LETTER D WITH CIRCUMFLEX BELOW]\n        '\\u{24D3}' | // ⓓ  [CIRCLED LATIN SMALL LETTER D]\n        '\\u{A77A}' | // ꝺ  [LATIN SMALL LETTER INSULAR D]\n        '\\u{FF44}' // ｄ  [FULLWIDTH LATIN SMALL LETTER D]\n        => Some(\"d\"),\n        '\\u{01C4}' | // DŽ  [LATIN CAPITAL LETTER DZ WITH CARON]\n        '\\u{01F1}' // DZ  [LATIN CAPITAL LETTER DZ]\n        => Some(\"DZ\"),\n        '\\u{01C5}' | // Dž  [LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON]\n        '\\u{01F2}' // Dz  [LATIN CAPITAL LETTER D WITH SMALL LETTER Z]\n        => Some(\"Dz\"),\n        '\\u{249F}' // ⒟  [PARENTHESIZED LATIN SMALL LETTER D]\n        => Some(\"(d)\"),\n        '\\u{0238}' // ȸ  [LATIN SMALL LETTER DB DIGRAPH]\n        => Some(\"db\"),\n        '\\u{01C6}' | // dž  [LATIN SMALL LETTER DZ WITH CARON]\n        '\\u{01F3}' | // dz  [LATIN SMALL LETTER DZ]\n        '\\u{02A3}' | // ʣ  [LATIN SMALL LETTER DZ DIGRAPH]\n        '\\u{02A5}' // ʥ  [LATIN SMALL LETTER DZ DIGRAPH WITH CURL]\n        => Some(\"dz\"),\n        '\\u{00C8}' | // È  [LATIN CAPITAL LETTER E WITH GRAVE]\n        '\\u{00C9}' | // É  [LATIN CAPITAL LETTER E WITH ACUTE]\n        '\\u{00CA}' | // Ê  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX]\n        '\\u{00CB}' | // Ë  [LATIN CAPITAL LETTER E WITH DIAERESIS]\n        '\\u{0112}' | // Ē  [LATIN CAPITAL LETTER E WITH MACRON]\n        '\\u{0114}' | // Ĕ  [LATIN CAPITAL LETTER E WITH BREVE]\n        '\\u{0116}' | // Ė  [LATIN CAPITAL LETTER E WITH DOT ABOVE]\n        '\\u{0118}' | // Ę  [LATIN CAPITAL LETTER E WITH OGONEK]\n        '\\u{011A}' | // Ě  [LATIN CAPITAL LETTER E WITH CARON]\n        '\\u{018E}' | // Ǝ  [LATIN CAPITAL LETTER REVERSED E]\n        '\\u{0190}' | // Ɛ  [LATIN CAPITAL LETTER OPEN E]\n        '\\u{0204}' | // Ȅ  [LATIN CAPITAL LETTER E WITH DOUBLE GRAVE]\n        '\\u{0206}' | // Ȇ  [LATIN CAPITAL LETTER E WITH INVERTED BREVE]\n        '\\u{0228}' | // Ȩ  [LATIN CAPITAL LETTER E WITH CEDILLA]\n        '\\u{0246}' | // Ɇ  [LATIN CAPITAL LETTER E WITH STROKE]\n        '\\u{1D07}' | // ᴇ  [LATIN LETTER SMALL CAPITAL E]\n        '\\u{1E14}' | // Ḕ  [LATIN CAPITAL LETTER E WITH MACRON AND GRAVE]\n        '\\u{1E16}' | // Ḗ  [LATIN CAPITAL LETTER E WITH MACRON AND ACUTE]\n        '\\u{1E18}' | // Ḙ  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX BELOW]\n        '\\u{1E1A}' | // Ḛ  [LATIN CAPITAL LETTER E WITH TILDE BELOW]\n        '\\u{1E1C}' | // Ḝ  [LATIN CAPITAL LETTER E WITH CEDILLA AND BREVE]\n        '\\u{1EB8}' | // Ẹ  [LATIN CAPITAL LETTER E WITH DOT BELOW]\n        '\\u{1EBA}' | // Ẻ  [LATIN CAPITAL LETTER E WITH HOOK ABOVE]\n        '\\u{1EBC}' | // Ẽ  [LATIN CAPITAL LETTER E WITH TILDE]\n        '\\u{1EBE}' | // Ế  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1EC0}' | // Ề  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1EC2}' | // Ể  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1EC4}' | // Ễ  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND TILDE]\n        '\\u{1EC6}' | // Ệ  [LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{24BA}' | // Ⓔ  [CIRCLED LATIN CAPITAL LETTER E]\n        '\\u{2C7B}' | // ⱻ  [LATIN LETTER SMALL CAPITAL TURNED E]\n        '\\u{FF25}' // Ｅ  [FULLWIDTH LATIN CAPITAL LETTER E]\n        => Some(\"E\"),\n        '\\u{00E8}' | // è  [LATIN SMALL LETTER E WITH GRAVE]\n        '\\u{00E9}' | // é  [LATIN SMALL LETTER E WITH ACUTE]\n        '\\u{00EA}' | // ê  [LATIN SMALL LETTER E WITH CIRCUMFLEX]\n        '\\u{00EB}' | // ë  [LATIN SMALL LETTER E WITH DIAERESIS]\n        '\\u{0113}' | // ē  [LATIN SMALL LETTER E WITH MACRON]\n        '\\u{0115}' | // ĕ  [LATIN SMALL LETTER E WITH BREVE]\n        '\\u{0117}' | // ė  [LATIN SMALL LETTER E WITH DOT ABOVE]\n        '\\u{0119}' | // ę  [LATIN SMALL LETTER E WITH OGONEK]\n        '\\u{011B}' | // ě  [LATIN SMALL LETTER E WITH CARON]\n        '\\u{01DD}' | // ǝ  [LATIN SMALL LETTER TURNED E]\n        '\\u{0205}' | // ȅ  [LATIN SMALL LETTER E WITH DOUBLE GRAVE]\n        '\\u{0207}' | // ȇ  [LATIN SMALL LETTER E WITH INVERTED BREVE]\n        '\\u{0229}' | // ȩ  [LATIN SMALL LETTER E WITH CEDILLA]\n        '\\u{0247}' | // ɇ  [LATIN SMALL LETTER E WITH STROKE]\n        '\\u{0258}' | // ɘ  [LATIN SMALL LETTER REVERSED E]\n        '\\u{025B}' | // ɛ  [LATIN SMALL LETTER OPEN E]\n        '\\u{025C}' | // ɜ  [LATIN SMALL LETTER REVERSED OPEN E]\n        '\\u{025D}' | // ɝ  [LATIN SMALL LETTER REVERSED OPEN E WITH HOOK]\n        '\\u{025E}' | // ɞ  [LATIN SMALL LETTER CLOSED REVERSED OPEN E]\n        '\\u{029A}' | // ʚ  [LATIN SMALL LETTER CLOSED OPEN E]\n        '\\u{1D08}' | // ᴈ  [LATIN SMALL LETTER TURNED OPEN E]\n        '\\u{1D92}' | // ᶒ  [LATIN SMALL LETTER E WITH RETROFLEX HOOK]\n        '\\u{1D93}' | // ᶓ  [LATIN SMALL LETTER OPEN E WITH RETROFLEX HOOK]\n        '\\u{1D94}' | // ᶔ  [LATIN SMALL LETTER REVERSED OPEN E WITH RETROFLEX HOOK]\n        '\\u{1E15}' | // ḕ  [LATIN SMALL LETTER E WITH MACRON AND GRAVE]\n        '\\u{1E17}' | // ḗ  [LATIN SMALL LETTER E WITH MACRON AND ACUTE]\n        '\\u{1E19}' | // ḙ  [LATIN SMALL LETTER E WITH CIRCUMFLEX BELOW]\n        '\\u{1E1B}' | // ḛ  [LATIN SMALL LETTER E WITH TILDE BELOW]\n        '\\u{1E1D}' | // ḝ  [LATIN SMALL LETTER E WITH CEDILLA AND BREVE]\n        '\\u{1EB9}' | // ẹ  [LATIN SMALL LETTER E WITH DOT BELOW]\n        '\\u{1EBB}' | // ẻ  [LATIN SMALL LETTER E WITH HOOK ABOVE]\n        '\\u{1EBD}' | // ẽ  [LATIN SMALL LETTER E WITH TILDE]\n        '\\u{1EBF}' | // ế  [LATIN SMALL LETTER E WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1EC1}' | // ề  [LATIN SMALL LETTER E WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1EC3}' | // ể  [LATIN SMALL LETTER E WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1EC5}' | // ễ  [LATIN SMALL LETTER E WITH CIRCUMFLEX AND TILDE]\n        '\\u{1EC7}' | // ệ  [LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{2091}' | // ₑ  [LATIN SUBSCRIPT SMALL LETTER E]\n        '\\u{24D4}' | // ⓔ  [CIRCLED LATIN SMALL LETTER E]\n        '\\u{2C78}' | // ⱸ  [LATIN SMALL LETTER E WITH NOTCH]\n        '\\u{FF45}' // ｅ  [FULLWIDTH LATIN SMALL LETTER E]\n        => Some(\"e\"),\n        '\\u{24A0}' // ⒠  [PARENTHESIZED LATIN SMALL LETTER E]\n        => Some(\"(e)\"),\n        '\\u{0191}' | // Ƒ  [LATIN CAPITAL LETTER F WITH HOOK]\n        '\\u{1E1E}' | // Ḟ  [LATIN CAPITAL LETTER F WITH DOT ABOVE]\n        '\\u{24BB}' | // Ⓕ  [CIRCLED LATIN CAPITAL LETTER F]\n        '\\u{A730}' | // ꜰ  [LATIN LETTER SMALL CAPITAL F]\n        '\\u{A77B}' | // Ꝼ  [LATIN CAPITAL LETTER INSULAR F]\n        '\\u{A7FB}' | // ꟻ  [LATIN EPIGRAPHIC LETTER REVERSED F]\n        '\\u{FF26}' // Ｆ  [FULLWIDTH LATIN CAPITAL LETTER F]\n        => Some(\"F\"),\n        '\\u{0192}' | // ƒ  [LATIN SMALL LETTER F WITH HOOK]\n        '\\u{1D6E}' | // ᵮ  [LATIN SMALL LETTER F WITH MIDDLE TILDE]\n        '\\u{1D82}' | // ᶂ  [LATIN SMALL LETTER F WITH PALATAL HOOK]\n        '\\u{1E1F}' | // ḟ  [LATIN SMALL LETTER F WITH DOT ABOVE]\n        '\\u{1E9B}' | // ẛ  [LATIN SMALL LETTER LONG S WITH DOT ABOVE]\n        '\\u{24D5}' | // ⓕ  [CIRCLED LATIN SMALL LETTER F]\n        '\\u{A77C}' | // ꝼ  [LATIN SMALL LETTER INSULAR F]\n        '\\u{FF46}' // ｆ  [FULLWIDTH LATIN SMALL LETTER F]\n        => Some(\"f\"),\n        '\\u{24A1}' // ⒡  [PARENTHESIZED LATIN SMALL LETTER F]\n        => Some(\"(f)\"),\n        '\\u{FB00}' // ff  [LATIN SMALL LIGATURE FF]\n        => Some(\"ff\"),\n        '\\u{FB03}' // ffi  [LATIN SMALL LIGATURE FFI]\n        => Some(\"ffi\"),\n        '\\u{FB04}' // ffl  [LATIN SMALL LIGATURE FFL]\n        => Some(\"ffl\"),\n        '\\u{FB01}' // fi  [LATIN SMALL LIGATURE FI]\n        => Some(\"fi\"),\n        '\\u{FB02}' // fl  [LATIN SMALL LIGATURE FL]\n        => Some(\"fl\"),\n        '\\u{011C}' | // Ĝ  [LATIN CAPITAL LETTER G WITH CIRCUMFLEX]\n        '\\u{011E}' | // Ğ  [LATIN CAPITAL LETTER G WITH BREVE]\n        '\\u{0120}' | // Ġ  [LATIN CAPITAL LETTER G WITH DOT ABOVE]\n        '\\u{0122}' | // Ģ  [LATIN CAPITAL LETTER G WITH CEDILLA]\n        '\\u{0193}' | // Ɠ  [LATIN CAPITAL LETTER G WITH HOOK]\n        '\\u{01E4}' | // Ǥ  [LATIN CAPITAL LETTER G WITH STROKE]\n        '\\u{01E5}' | // ǥ  [LATIN SMALL LETTER G WITH STROKE]\n        '\\u{01E6}' | // Ǧ  [LATIN CAPITAL LETTER G WITH CARON]\n        '\\u{01E7}' | // ǧ  [LATIN SMALL LETTER G WITH CARON]\n        '\\u{01F4}' | // Ǵ  [LATIN CAPITAL LETTER G WITH ACUTE]\n        '\\u{0262}' | // ɢ  [LATIN LETTER SMALL CAPITAL G]\n        '\\u{029B}' | // ʛ  [LATIN LETTER SMALL CAPITAL G WITH HOOK]\n        '\\u{1E20}' | // Ḡ  [LATIN CAPITAL LETTER G WITH MACRON]\n        '\\u{24BC}' | // Ⓖ  [CIRCLED LATIN CAPITAL LETTER G]\n        '\\u{A77D}' | // Ᵹ  [LATIN CAPITAL LETTER INSULAR G]\n        '\\u{A77E}' | // Ꝿ  [LATIN CAPITAL LETTER TURNED INSULAR G]\n        '\\u{FF27}' // Ｇ  [FULLWIDTH LATIN CAPITAL LETTER G]\n        => Some(\"G\"),\n        '\\u{011D}' | // ĝ  [LATIN SMALL LETTER G WITH CIRCUMFLEX]\n        '\\u{011F}' | // ğ  [LATIN SMALL LETTER G WITH BREVE]\n        '\\u{0121}' | // ġ  [LATIN SMALL LETTER G WITH DOT ABOVE]\n        '\\u{0123}' | // ģ  [LATIN SMALL LETTER G WITH CEDILLA]\n        '\\u{01F5}' | // ǵ  [LATIN SMALL LETTER G WITH ACUTE]\n        '\\u{0260}' | // ɠ  [LATIN SMALL LETTER G WITH HOOK]\n        '\\u{0261}' | // ɡ  [LATIN SMALL LETTER SCRIPT G]\n        '\\u{1D77}' | // ᵷ  [LATIN SMALL LETTER TURNED G]\n        '\\u{1D79}' | // ᵹ  [LATIN SMALL LETTER INSULAR G]\n        '\\u{1D83}' | // ᶃ  [LATIN SMALL LETTER G WITH PALATAL HOOK]\n        '\\u{1E21}' | // ḡ  [LATIN SMALL LETTER G WITH MACRON]\n        '\\u{24D6}' | // ⓖ  [CIRCLED LATIN SMALL LETTER G]\n        '\\u{A77F}' | // ꝿ  [LATIN SMALL LETTER TURNED INSULAR G]\n        '\\u{FF47}' // ｇ  [FULLWIDTH LATIN SMALL LETTER G]\n        => Some(\"g\"),\n        '\\u{24A2}' // ⒢  [PARENTHESIZED LATIN SMALL LETTER G]\n        => Some(\"(g)\"),\n        '\\u{0124}' | // Ĥ  [LATIN CAPITAL LETTER H WITH CIRCUMFLEX]\n        '\\u{0126}' | // Ħ  [LATIN CAPITAL LETTER H WITH STROKE]\n        '\\u{021E}' | // Ȟ  [LATIN CAPITAL LETTER H WITH CARON]\n        '\\u{029C}' | // ʜ  [LATIN LETTER SMALL CAPITAL H]\n        '\\u{1E22}' | // Ḣ  [LATIN CAPITAL LETTER H WITH DOT ABOVE]\n        '\\u{1E24}' | // Ḥ  [LATIN CAPITAL LETTER H WITH DOT BELOW]\n        '\\u{1E26}' | // Ḧ  [LATIN CAPITAL LETTER H WITH DIAERESIS]\n        '\\u{1E28}' | // Ḩ  [LATIN CAPITAL LETTER H WITH CEDILLA]\n        '\\u{1E2A}' | // Ḫ  [LATIN CAPITAL LETTER H WITH BREVE BELOW]\n        '\\u{24BD}' | // Ⓗ  [CIRCLED LATIN CAPITAL LETTER H]\n        '\\u{2C67}' | // Ⱨ  [LATIN CAPITAL LETTER H WITH DESCENDER]\n        '\\u{2C75}' | // Ⱶ  [LATIN CAPITAL LETTER HALF H]\n        '\\u{FF28}' // Ｈ  [FULLWIDTH LATIN CAPITAL LETTER H]\n        => Some(\"H\"),\n        '\\u{0125}' | // ĥ  [LATIN SMALL LETTER H WITH CIRCUMFLEX]\n        '\\u{0127}' | // ħ  [LATIN SMALL LETTER H WITH STROKE]\n        '\\u{021F}' | // ȟ  [LATIN SMALL LETTER H WITH CARON]\n        '\\u{0265}' | // ɥ  [LATIN SMALL LETTER TURNED H]\n        '\\u{0266}' | // ɦ  [LATIN SMALL LETTER H WITH HOOK]\n        '\\u{02AE}' | // ʮ  [LATIN SMALL LETTER TURNED H WITH FISHHOOK]\n        '\\u{02AF}' | // ʯ  [LATIN SMALL LETTER TURNED H WITH FISHHOOK AND TAIL]\n        '\\u{1E23}' | // ḣ  [LATIN SMALL LETTER H WITH DOT ABOVE]\n        '\\u{1E25}' | // ḥ  [LATIN SMALL LETTER H WITH DOT BELOW]\n        '\\u{1E27}' | // ḧ  [LATIN SMALL LETTER H WITH DIAERESIS]\n        '\\u{1E29}' | // ḩ  [LATIN SMALL LETTER H WITH CEDILLA]\n        '\\u{1E2B}' | // ḫ  [LATIN SMALL LETTER H WITH BREVE BELOW]\n        '\\u{1E96}' | // ẖ  [LATIN SMALL LETTER H WITH LINE BELOW]\n        '\\u{24D7}' | // ⓗ  [CIRCLED LATIN SMALL LETTER H]\n        '\\u{2C68}' | // ⱨ  [LATIN SMALL LETTER H WITH DESCENDER]\n        '\\u{2C76}' | // ⱶ  [LATIN SMALL LETTER HALF H]\n        '\\u{FF48}' // ｈ  [FULLWIDTH LATIN SMALL LETTER H]\n        => Some(\"h\"),\n        '\\u{01F6}' // Ƕ  http://en.wikipedia.org/wiki/Hwair  [LATIN CAPITAL LETTER HWAIR]\n        => Some(\"HV\"),\n        '\\u{24A3}' // ⒣  [PARENTHESIZED LATIN SMALL LETTER H]\n        => Some(\"(h)\"),\n        '\\u{0195}' // ƕ  [LATIN SMALL LETTER HV]\n        => Some(\"hv\"),\n        '\\u{00CC}' | // Ì  [LATIN CAPITAL LETTER I WITH GRAVE]\n        '\\u{00CD}' | // Í  [LATIN CAPITAL LETTER I WITH ACUTE]\n        '\\u{00CE}' | // Î  [LATIN CAPITAL LETTER I WITH CIRCUMFLEX]\n        '\\u{00CF}' | // Ï  [LATIN CAPITAL LETTER I WITH DIAERESIS]\n        '\\u{0128}' | // Ĩ  [LATIN CAPITAL LETTER I WITH TILDE]\n        '\\u{012A}' | // Ī  [LATIN CAPITAL LETTER I WITH MACRON]\n        '\\u{012C}' | // Ĭ  [LATIN CAPITAL LETTER I WITH BREVE]\n        '\\u{012E}' | // Į  [LATIN CAPITAL LETTER I WITH OGONEK]\n        '\\u{0130}' | // İ  [LATIN CAPITAL LETTER I WITH DOT ABOVE]\n        '\\u{0196}' | // Ɩ  [LATIN CAPITAL LETTER IOTA]\n        '\\u{0197}' | // Ɨ  [LATIN CAPITAL LETTER I WITH STROKE]\n        '\\u{01CF}' | // Ǐ  [LATIN CAPITAL LETTER I WITH CARON]\n        '\\u{0208}' | // Ȉ  [LATIN CAPITAL LETTER I WITH DOUBLE GRAVE]\n        '\\u{020A}' | // Ȋ  [LATIN CAPITAL LETTER I WITH INVERTED BREVE]\n        '\\u{026A}' | // ɪ  [LATIN LETTER SMALL CAPITAL I]\n        '\\u{1D7B}' | // ᵻ  [LATIN SMALL CAPITAL LETTER I WITH STROKE]\n        '\\u{1E2C}' | // Ḭ  [LATIN CAPITAL LETTER I WITH TILDE BELOW]\n        '\\u{1E2E}' | // Ḯ  [LATIN CAPITAL LETTER I WITH DIAERESIS AND ACUTE]\n        '\\u{1EC8}' | // Ỉ  [LATIN CAPITAL LETTER I WITH HOOK ABOVE]\n        '\\u{1ECA}' | // Ị  [LATIN CAPITAL LETTER I WITH DOT BELOW]\n        '\\u{24BE}' | // Ⓘ  [CIRCLED LATIN CAPITAL LETTER I]\n        '\\u{A7FE}' | // ꟾ  [LATIN EPIGRAPHIC LETTER I LONGA]\n        '\\u{FF29}' // Ｉ  [FULLWIDTH LATIN CAPITAL LETTER I]\n        => Some(\"I\"),\n        '\\u{00EC}' | // ì  [LATIN SMALL LETTER I WITH GRAVE]\n        '\\u{00ED}' | // í  [LATIN SMALL LETTER I WITH ACUTE]\n        '\\u{00EE}' | // î  [LATIN SMALL LETTER I WITH CIRCUMFLEX]\n        '\\u{00EF}' | // ï  [LATIN SMALL LETTER I WITH DIAERESIS]\n        '\\u{0129}' | // ĩ  [LATIN SMALL LETTER I WITH TILDE]\n        '\\u{012B}' | // ī  [LATIN SMALL LETTER I WITH MACRON]\n        '\\u{012D}' | // ĭ  [LATIN SMALL LETTER I WITH BREVE]\n        '\\u{012F}' | // į  [LATIN SMALL LETTER I WITH OGONEK]\n        '\\u{0131}' | // ı  [LATIN SMALL LETTER DOTLESS I]\n        '\\u{01D0}' | // ǐ  [LATIN SMALL LETTER I WITH CARON]\n        '\\u{0209}' | // ȉ  [LATIN SMALL LETTER I WITH DOUBLE GRAVE]\n        '\\u{020B}' | // ȋ  [LATIN SMALL LETTER I WITH INVERTED BREVE]\n        '\\u{0268}' | // ɨ  [LATIN SMALL LETTER I WITH STROKE]\n        '\\u{1D09}' | // ᴉ  [LATIN SMALL LETTER TURNED I]\n        '\\u{1D62}' | // ᵢ  [LATIN SUBSCRIPT SMALL LETTER I]\n        '\\u{1D7C}' | // ᵼ  [LATIN SMALL LETTER IOTA WITH STROKE]\n        '\\u{1D96}' | // ᶖ  [LATIN SMALL LETTER I WITH RETROFLEX HOOK]\n        '\\u{1E2D}' | // ḭ  [LATIN SMALL LETTER I WITH TILDE BELOW]\n        '\\u{1E2F}' | // ḯ  [LATIN SMALL LETTER I WITH DIAERESIS AND ACUTE]\n        '\\u{1EC9}' | // ỉ  [LATIN SMALL LETTER I WITH HOOK ABOVE]\n        '\\u{1ECB}' | // ị  [LATIN SMALL LETTER I WITH DOT BELOW]\n        '\\u{2071}' | // ⁱ  [SUPERSCRIPT LATIN SMALL LETTER I]\n        '\\u{24D8}' | // ⓘ  [CIRCLED LATIN SMALL LETTER I]\n        '\\u{FF49}' // ｉ  [FULLWIDTH LATIN SMALL LETTER I]\n        => Some(\"i\"),\n        '\\u{0132}' // IJ  [LATIN CAPITAL LIGATURE IJ]\n        => Some(\"IJ\"),\n        '\\u{24A4}' // ⒤  [PARENTHESIZED LATIN SMALL LETTER I]\n        => Some(\"(i)\"),\n        '\\u{0133}' // ij  [LATIN SMALL LIGATURE IJ]\n        => Some(\"ij\"),\n        '\\u{0134}' | // Ĵ  [LATIN CAPITAL LETTER J WITH CIRCUMFLEX]\n        '\\u{0248}' | // Ɉ  [LATIN CAPITAL LETTER J WITH STROKE]\n        '\\u{1D0A}' | // ᴊ  [LATIN LETTER SMALL CAPITAL J]\n        '\\u{24BF}' | // Ⓙ  [CIRCLED LATIN CAPITAL LETTER J]\n        '\\u{FF2A}' // Ｊ  [FULLWIDTH LATIN CAPITAL LETTER J]\n        => Some(\"J\"),\n        '\\u{0135}' | // ĵ  [LATIN SMALL LETTER J WITH CIRCUMFLEX]\n        '\\u{01F0}' | // ǰ  [LATIN SMALL LETTER J WITH CARON]\n        '\\u{0237}' | // ȷ  [LATIN SMALL LETTER DOTLESS J]\n        '\\u{0249}' | // ɉ  [LATIN SMALL LETTER J WITH STROKE]\n        '\\u{025F}' | // ɟ  [LATIN SMALL LETTER DOTLESS J WITH STROKE]\n        '\\u{0284}' | // ʄ  [LATIN SMALL LETTER DOTLESS J WITH STROKE AND HOOK]\n        '\\u{029D}' | // ʝ  [LATIN SMALL LETTER J WITH CROSSED-TAIL]\n        '\\u{24D9}' | // ⓙ  [CIRCLED LATIN SMALL LETTER J]\n        '\\u{2C7C}' | // ⱼ  [LATIN SUBSCRIPT SMALL LETTER J]\n        '\\u{FF4A}' // ｊ  [FULLWIDTH LATIN SMALL LETTER J]\n        => Some(\"j\"),\n        '\\u{24A5}' // ⒥  [PARENTHESIZED LATIN SMALL LETTER J]\n        => Some(\"(j)\"),\n        '\\u{0136}' | // Ķ  [LATIN CAPITAL LETTER K WITH CEDILLA]\n        '\\u{0198}' | // Ƙ  [LATIN CAPITAL LETTER K WITH HOOK]\n        '\\u{01E8}' | // Ǩ  [LATIN CAPITAL LETTER K WITH CARON]\n        '\\u{1D0B}' | // ᴋ  [LATIN LETTER SMALL CAPITAL K]\n        '\\u{1E30}' | // Ḱ  [LATIN CAPITAL LETTER K WITH ACUTE]\n        '\\u{1E32}' | // Ḳ  [LATIN CAPITAL LETTER K WITH DOT BELOW]\n        '\\u{1E34}' | // Ḵ  [LATIN CAPITAL LETTER K WITH LINE BELOW]\n        '\\u{24C0}' | // Ⓚ  [CIRCLED LATIN CAPITAL LETTER K]\n        '\\u{2C69}' | // Ⱪ  [LATIN CAPITAL LETTER K WITH DESCENDER]\n        '\\u{A740}' | // Ꝁ  [LATIN CAPITAL LETTER K WITH STROKE]\n        '\\u{A742}' | // Ꝃ  [LATIN CAPITAL LETTER K WITH DIAGONAL STROKE]\n        '\\u{A744}' | // Ꝅ  [LATIN CAPITAL LETTER K WITH STROKE AND DIAGONAL STROKE]\n        '\\u{FF2B}' // Ｋ  [FULLWIDTH LATIN CAPITAL LETTER K]\n        => Some(\"K\"),\n        '\\u{0137}' | // ķ  [LATIN SMALL LETTER K WITH CEDILLA]\n        '\\u{0199}' | // ƙ  [LATIN SMALL LETTER K WITH HOOK]\n        '\\u{01E9}' | // ǩ  [LATIN SMALL LETTER K WITH CARON]\n        '\\u{029E}' | // ʞ  [LATIN SMALL LETTER TURNED K]\n        '\\u{1D84}' | // ᶄ  [LATIN SMALL LETTER K WITH PALATAL HOOK]\n        '\\u{1E31}' | // ḱ  [LATIN SMALL LETTER K WITH ACUTE]\n        '\\u{1E33}' | // ḳ  [LATIN SMALL LETTER K WITH DOT BELOW]\n        '\\u{1E35}' | // ḵ  [LATIN SMALL LETTER K WITH LINE BELOW]\n        '\\u{24DA}' | // ⓚ  [CIRCLED LATIN SMALL LETTER K]\n        '\\u{2C6A}' | // ⱪ  [LATIN SMALL LETTER K WITH DESCENDER]\n        '\\u{A741}' | // ꝁ  [LATIN SMALL LETTER K WITH STROKE]\n        '\\u{A743}' | // ꝃ  [LATIN SMALL LETTER K WITH DIAGONAL STROKE]\n        '\\u{A745}' | // ꝅ  [LATIN SMALL LETTER K WITH STROKE AND DIAGONAL STROKE]\n        '\\u{FF4B}' // ｋ  [FULLWIDTH LATIN SMALL LETTER K]\n        => Some(\"k\"),\n        '\\u{24A6}' // ⒦  [PARENTHESIZED LATIN SMALL LETTER K]\n        => Some(\"(k)\"),\n        '\\u{0139}' | // Ĺ  [LATIN CAPITAL LETTER L WITH ACUTE]\n        '\\u{013B}' | // Ļ  [LATIN CAPITAL LETTER L WITH CEDILLA]\n        '\\u{013D}' | // Ľ  [LATIN CAPITAL LETTER L WITH CARON]\n        '\\u{013F}' | // Ŀ  [LATIN CAPITAL LETTER L WITH MIDDLE DOT]\n        '\\u{0141}' | // Ł  [LATIN CAPITAL LETTER L WITH STROKE]\n        '\\u{023D}' | // Ƚ  [LATIN CAPITAL LETTER L WITH BAR]\n        '\\u{029F}' | // ʟ  [LATIN LETTER SMALL CAPITAL L]\n        '\\u{1D0C}' | // ᴌ  [LATIN LETTER SMALL CAPITAL L WITH STROKE]\n        '\\u{1E36}' | // Ḷ  [LATIN CAPITAL LETTER L WITH DOT BELOW]\n        '\\u{1E38}' | // Ḹ  [LATIN CAPITAL LETTER L WITH DOT BELOW AND MACRON]\n        '\\u{1E3A}' | // Ḻ  [LATIN CAPITAL LETTER L WITH LINE BELOW]\n        '\\u{1E3C}' | // Ḽ  [LATIN CAPITAL LETTER L WITH CIRCUMFLEX BELOW]\n        '\\u{24C1}' | // Ⓛ  [CIRCLED LATIN CAPITAL LETTER L]\n        '\\u{2C60}' | // Ⱡ  [LATIN CAPITAL LETTER L WITH DOUBLE BAR]\n        '\\u{2C62}' | // Ɫ  [LATIN CAPITAL LETTER L WITH MIDDLE TILDE]\n        '\\u{A746}' | // Ꝇ  [LATIN CAPITAL LETTER BROKEN L]\n        '\\u{A748}' | // Ꝉ  [LATIN CAPITAL LETTER L WITH HIGH STROKE]\n        '\\u{A780}' | // Ꞁ  [LATIN CAPITAL LETTER TURNED L]\n        '\\u{FF2C}' // Ｌ  [FULLWIDTH LATIN CAPITAL LETTER L]\n        => Some(\"L\"),\n        '\\u{013A}' | // ĺ  [LATIN SMALL LETTER L WITH ACUTE]\n        '\\u{013C}' | // ļ  [LATIN SMALL LETTER L WITH CEDILLA]\n        '\\u{013E}' | // ľ  [LATIN SMALL LETTER L WITH CARON]\n        '\\u{0140}' | // ŀ  [LATIN SMALL LETTER L WITH MIDDLE DOT]\n        '\\u{0142}' | // ł  [LATIN SMALL LETTER L WITH STROKE]\n        '\\u{019A}' | // ƚ  [LATIN SMALL LETTER L WITH BAR]\n        '\\u{0234}' | // ȴ  [LATIN SMALL LETTER L WITH CURL]\n        '\\u{026B}' | // ɫ  [LATIN SMALL LETTER L WITH MIDDLE TILDE]\n        '\\u{026C}' | // ɬ  [LATIN SMALL LETTER L WITH BELT]\n        '\\u{026D}' | // ɭ  [LATIN SMALL LETTER L WITH RETROFLEX HOOK]\n        '\\u{1D85}' | // ᶅ  [LATIN SMALL LETTER L WITH PALATAL HOOK]\n        '\\u{1E37}' | // ḷ  [LATIN SMALL LETTER L WITH DOT BELOW]\n        '\\u{1E39}' | // ḹ  [LATIN SMALL LETTER L WITH DOT BELOW AND MACRON]\n        '\\u{1E3B}' | // ḻ  [LATIN SMALL LETTER L WITH LINE BELOW]\n        '\\u{1E3D}' | // ḽ  [LATIN SMALL LETTER L WITH CIRCUMFLEX BELOW]\n        '\\u{24DB}' | // ⓛ  [CIRCLED LATIN SMALL LETTER L]\n        '\\u{2C61}' | // ⱡ  [LATIN SMALL LETTER L WITH DOUBLE BAR]\n        '\\u{A747}' | // ꝇ  [LATIN SMALL LETTER BROKEN L]\n        '\\u{A749}' | // ꝉ  [LATIN SMALL LETTER L WITH HIGH STROKE]\n        '\\u{A781}' | // ꞁ  [LATIN SMALL LETTER TURNED L]\n        '\\u{FF4C}' // ｌ  [FULLWIDTH LATIN SMALL LETTER L]\n        => Some(\"l\"),\n        '\\u{01C7}' // LJ  [LATIN CAPITAL LETTER LJ]\n        => Some(\"LJ\"),\n        '\\u{1EFA}' // Ỻ  [LATIN CAPITAL LETTER MIDDLE-WELSH LL]\n        => Some(\"LL\"),\n        '\\u{01C8}' // Lj  [LATIN CAPITAL LETTER L WITH SMALL LETTER J]\n        => Some(\"Lj\"),\n        '\\u{24A7}' // ⒧  [PARENTHESIZED LATIN SMALL LETTER L]\n        => Some(\"(l)\"),\n        '\\u{01C9}' // lj  [LATIN SMALL LETTER LJ]\n        => Some(\"lj\"),\n        '\\u{1EFB}' // ỻ  [LATIN SMALL LETTER MIDDLE-WELSH LL]\n        => Some(\"ll\"),\n        '\\u{02AA}' // ʪ  [LATIN SMALL LETTER LS DIGRAPH]\n        => Some(\"ls\"),\n        '\\u{02AB}' // ʫ  [LATIN SMALL LETTER LZ DIGRAPH]\n        => Some(\"lz\"),\n        '\\u{019C}' | // Ɯ  [LATIN CAPITAL LETTER TURNED M]\n        '\\u{1D0D}' | // ᴍ  [LATIN LETTER SMALL CAPITAL M]\n        '\\u{1E3E}' | // Ḿ  [LATIN CAPITAL LETTER M WITH ACUTE]\n        '\\u{1E40}' | // Ṁ  [LATIN CAPITAL LETTER M WITH DOT ABOVE]\n        '\\u{1E42}' | // Ṃ  [LATIN CAPITAL LETTER M WITH DOT BELOW]\n        '\\u{24C2}' | // Ⓜ  [CIRCLED LATIN CAPITAL LETTER M]\n        '\\u{2C6E}' | // Ɱ  [LATIN CAPITAL LETTER M WITH HOOK]\n        '\\u{A7FD}' | // ꟽ  [LATIN EPIGRAPHIC LETTER INVERTED M]\n        '\\u{A7FF}' | // ꟿ  [LATIN EPIGRAPHIC LETTER ARCHAIC M]\n        '\\u{FF2D}' // Ｍ  [FULLWIDTH LATIN CAPITAL LETTER M]\n        => Some(\"M\"),\n        '\\u{026F}' | // ɯ  [LATIN SMALL LETTER TURNED M]\n        '\\u{0270}' | // ɰ  [LATIN SMALL LETTER TURNED M WITH LONG LEG]\n        '\\u{0271}' | // ɱ  [LATIN SMALL LETTER M WITH HOOK]\n        '\\u{1D6F}' | // ᵯ  [LATIN SMALL LETTER M WITH MIDDLE TILDE]\n        '\\u{1D86}' | // ᶆ  [LATIN SMALL LETTER M WITH PALATAL HOOK]\n        '\\u{1E3F}' | // ḿ  [LATIN SMALL LETTER M WITH ACUTE]\n        '\\u{1E41}' | // ṁ  [LATIN SMALL LETTER M WITH DOT ABOVE]\n        '\\u{1E43}' | // ṃ  [LATIN SMALL LETTER M WITH DOT BELOW]\n        '\\u{24DC}' | // ⓜ  [CIRCLED LATIN SMALL LETTER M]\n        '\\u{FF4D}' // ｍ  [FULLWIDTH LATIN SMALL LETTER M]\n        => Some(\"m\"),\n        '\\u{24A8}' // ⒨  [PARENTHESIZED LATIN SMALL LETTER M]\n        => Some(\"(m)\"),\n        '\\u{00D1}' | // Ñ  [LATIN CAPITAL LETTER N WITH TILDE]\n        '\\u{0143}' | // Ń  [LATIN CAPITAL LETTER N WITH ACUTE]\n        '\\u{0145}' | // Ņ  [LATIN CAPITAL LETTER N WITH CEDILLA]\n        '\\u{0147}' | // Ň  [LATIN CAPITAL LETTER N WITH CARON]\n        '\\u{014A}' | // Ŋ  http://en.wikipedia.org/wiki/Eng_(letter)  [LATIN CAPITAL LETTER ENG]\n        '\\u{019D}' | // Ɲ  [LATIN CAPITAL LETTER N WITH LEFT HOOK]\n        '\\u{01F8}' | // Ǹ  [LATIN CAPITAL LETTER N WITH GRAVE]\n        '\\u{0220}' | // Ƞ  [LATIN CAPITAL LETTER N WITH LONG RIGHT LEG]\n        '\\u{0274}' | // ɴ  [LATIN LETTER SMALL CAPITAL N]\n        '\\u{1D0E}' | // ᴎ  [LATIN LETTER SMALL CAPITAL REVERSED N]\n        '\\u{1E44}' | // Ṅ  [LATIN CAPITAL LETTER N WITH DOT ABOVE]\n        '\\u{1E46}' | // Ṇ  [LATIN CAPITAL LETTER N WITH DOT BELOW]\n        '\\u{1E48}' | // Ṉ  [LATIN CAPITAL LETTER N WITH LINE BELOW]\n        '\\u{1E4A}' | // Ṋ  [LATIN CAPITAL LETTER N WITH CIRCUMFLEX BELOW]\n        '\\u{24C3}' | // Ⓝ  [CIRCLED LATIN CAPITAL LETTER N]\n        '\\u{FF2E}' // Ｎ  [FULLWIDTH LATIN CAPITAL LETTER N]\n        => Some(\"N\"),\n        '\\u{00F1}' | // ñ  [LATIN SMALL LETTER N WITH TILDE]\n        '\\u{0144}' | // ń  [LATIN SMALL LETTER N WITH ACUTE]\n        '\\u{0146}' | // ņ  [LATIN SMALL LETTER N WITH CEDILLA]\n        '\\u{0148}' | // ň  [LATIN SMALL LETTER N WITH CARON]\n        '\\u{0149}' | // ʼn  [LATIN SMALL LETTER N PRECEDED BY APOSTROPHE]\n        '\\u{014B}' | // ŋ  http://en.wikipedia.org/wiki/Eng_(letter)  [LATIN SMALL LETTER ENG]\n        '\\u{019E}' | // ƞ  [LATIN SMALL LETTER N WITH LONG RIGHT LEG]\n        '\\u{01F9}' | // ǹ  [LATIN SMALL LETTER N WITH GRAVE]\n        '\\u{0235}' | // ȵ  [LATIN SMALL LETTER N WITH CURL]\n        '\\u{0272}' | // ɲ  [LATIN SMALL LETTER N WITH LEFT HOOK]\n        '\\u{0273}' | // ɳ  [LATIN SMALL LETTER N WITH RETROFLEX HOOK]\n        '\\u{1D70}' | // ᵰ  [LATIN SMALL LETTER N WITH MIDDLE TILDE]\n        '\\u{1D87}' | // ᶇ  [LATIN SMALL LETTER N WITH PALATAL HOOK]\n        '\\u{1E45}' | // ṅ  [LATIN SMALL LETTER N WITH DOT ABOVE]\n        '\\u{1E47}' | // ṇ  [LATIN SMALL LETTER N WITH DOT BELOW]\n        '\\u{1E49}' | // ṉ  [LATIN SMALL LETTER N WITH LINE BELOW]\n        '\\u{1E4B}' | // ṋ  [LATIN SMALL LETTER N WITH CIRCUMFLEX BELOW]\n        '\\u{207F}' | // ⁿ  [SUPERSCRIPT LATIN SMALL LETTER N]\n        '\\u{24DD}' | // ⓝ  [CIRCLED LATIN SMALL LETTER N]\n        '\\u{FF4E}' // ｎ  [FULLWIDTH LATIN SMALL LETTER N]\n        => Some(\"n\"),\n        '\\u{01CA}' // NJ  [LATIN CAPITAL LETTER NJ]\n        => Some(\"NJ\"),\n        '\\u{01CB}' // Nj  [LATIN CAPITAL LETTER N WITH SMALL LETTER J]\n        => Some(\"Nj\"),\n        '\\u{24A9}' // ⒩  [PARENTHESIZED LATIN SMALL LETTER N]\n        => Some(\"(n)\"),\n        '\\u{01CC}' // nj  [LATIN SMALL LETTER NJ]\n        => Some(\"nj\"),\n        '\\u{00D2}' | // Ò  [LATIN CAPITAL LETTER O WITH GRAVE]\n        '\\u{00D3}' | // Ó  [LATIN CAPITAL LETTER O WITH ACUTE]\n        '\\u{00D4}' | // Ô  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX]\n        '\\u{00D5}' | // Õ  [LATIN CAPITAL LETTER O WITH TILDE]\n        '\\u{00D6}' | // Ö  [LATIN CAPITAL LETTER O WITH DIAERESIS]\n        '\\u{00D8}' | // Ø  [LATIN CAPITAL LETTER O WITH STROKE]\n        '\\u{014C}' | // Ō  [LATIN CAPITAL LETTER O WITH MACRON]\n        '\\u{014E}' | // Ŏ  [LATIN CAPITAL LETTER O WITH BREVE]\n        '\\u{0150}' | // Ő  [LATIN CAPITAL LETTER O WITH DOUBLE ACUTE]\n        '\\u{0186}' | // Ɔ  [LATIN CAPITAL LETTER OPEN O]\n        '\\u{019F}' | // Ɵ  [LATIN CAPITAL LETTER O WITH MIDDLE TILDE]\n        '\\u{01A0}' | // Ơ  [LATIN CAPITAL LETTER O WITH HORN]\n        '\\u{01D1}' | // Ǒ  [LATIN CAPITAL LETTER O WITH CARON]\n        '\\u{01EA}' | // Ǫ  [LATIN CAPITAL LETTER O WITH OGONEK]\n        '\\u{01EC}' | // Ǭ  [LATIN CAPITAL LETTER O WITH OGONEK AND MACRON]\n        '\\u{01FE}' | // Ǿ  [LATIN CAPITAL LETTER O WITH STROKE AND ACUTE]\n        '\\u{020C}' | // Ȍ  [LATIN CAPITAL LETTER O WITH DOUBLE GRAVE]\n        '\\u{020E}' | // Ȏ  [LATIN CAPITAL LETTER O WITH INVERTED BREVE]\n        '\\u{022A}' | // Ȫ  [LATIN CAPITAL LETTER O WITH DIAERESIS AND MACRON]\n        '\\u{022C}' | // Ȭ  [LATIN CAPITAL LETTER O WITH TILDE AND MACRON]\n        '\\u{022E}' | // Ȯ  [LATIN CAPITAL LETTER O WITH DOT ABOVE]\n        '\\u{0230}' | // Ȱ  [LATIN CAPITAL LETTER O WITH DOT ABOVE AND MACRON]\n        '\\u{1D0F}' | // ᴏ  [LATIN LETTER SMALL CAPITAL O]\n        '\\u{1D10}' | // ᴐ  [LATIN LETTER SMALL CAPITAL OPEN O]\n        '\\u{1E4C}' | // Ṍ  [LATIN CAPITAL LETTER O WITH TILDE AND ACUTE]\n        '\\u{1E4E}' | // Ṏ  [LATIN CAPITAL LETTER O WITH TILDE AND DIAERESIS]\n        '\\u{1E50}' | // Ṑ  [LATIN CAPITAL LETTER O WITH MACRON AND GRAVE]\n        '\\u{1E52}' | // Ṓ  [LATIN CAPITAL LETTER O WITH MACRON AND ACUTE]\n        '\\u{1ECC}' | // Ọ  [LATIN CAPITAL LETTER O WITH DOT BELOW]\n        '\\u{1ECE}' | // Ỏ  [LATIN CAPITAL LETTER O WITH HOOK ABOVE]\n        '\\u{1ED0}' | // Ố  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1ED2}' | // Ồ  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1ED4}' | // Ổ  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1ED6}' | // Ỗ  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND TILDE]\n        '\\u{1ED8}' | // Ộ  [LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{1EDA}' | // Ớ  [LATIN CAPITAL LETTER O WITH HORN AND ACUTE]\n        '\\u{1EDC}' | // Ờ  [LATIN CAPITAL LETTER O WITH HORN AND GRAVE]\n        '\\u{1EDE}' | // Ở  [LATIN CAPITAL LETTER O WITH HORN AND HOOK ABOVE]\n        '\\u{1EE0}' | // Ỡ  [LATIN CAPITAL LETTER O WITH HORN AND TILDE]\n        '\\u{1EE2}' | // Ợ  [LATIN CAPITAL LETTER O WITH HORN AND DOT BELOW]\n        '\\u{24C4}' | // Ⓞ  [CIRCLED LATIN CAPITAL LETTER O]\n        '\\u{A74A}' | // Ꝋ  [LATIN CAPITAL LETTER O WITH LONG STROKE OVERLAY]\n        '\\u{A74C}' | // Ꝍ  [LATIN CAPITAL LETTER O WITH LOOP]\n        '\\u{FF2F}' // Ｏ  [FULLWIDTH LATIN CAPITAL LETTER O]\n        => Some(\"O\"),\n        '\\u{00F2}' | // ò  [LATIN SMALL LETTER O WITH GRAVE]\n        '\\u{00F3}' | // ó  [LATIN SMALL LETTER O WITH ACUTE]\n        '\\u{00F4}' | // ô  [LATIN SMALL LETTER O WITH CIRCUMFLEX]\n        '\\u{00F5}' | // õ  [LATIN SMALL LETTER O WITH TILDE]\n        '\\u{00F6}' | // ö  [LATIN SMALL LETTER O WITH DIAERESIS]\n        '\\u{00F8}' | // ø  [LATIN SMALL LETTER O WITH STROKE]\n        '\\u{014D}' | // ō  [LATIN SMALL LETTER O WITH MACRON]\n        '\\u{014F}' | // ŏ  [LATIN SMALL LETTER O WITH BREVE]\n        '\\u{0151}' | // ő  [LATIN SMALL LETTER O WITH DOUBLE ACUTE]\n        '\\u{01A1}' | // ơ  [LATIN SMALL LETTER O WITH HORN]\n        '\\u{01D2}' | // ǒ  [LATIN SMALL LETTER O WITH CARON]\n        '\\u{01EB}' | // ǫ  [LATIN SMALL LETTER O WITH OGONEK]\n        '\\u{01ED}' | // ǭ  [LATIN SMALL LETTER O WITH OGONEK AND MACRON]\n        '\\u{01FF}' | // ǿ  [LATIN SMALL LETTER O WITH STROKE AND ACUTE]\n        '\\u{020D}' | // ȍ  [LATIN SMALL LETTER O WITH DOUBLE GRAVE]\n        '\\u{020F}' | // ȏ  [LATIN SMALL LETTER O WITH INVERTED BREVE]\n        '\\u{022B}' | // ȫ  [LATIN SMALL LETTER O WITH DIAERESIS AND MACRON]\n        '\\u{022D}' | // ȭ  [LATIN SMALL LETTER O WITH TILDE AND MACRON]\n        '\\u{022F}' | // ȯ  [LATIN SMALL LETTER O WITH DOT ABOVE]\n        '\\u{0231}' | // ȱ  [LATIN SMALL LETTER O WITH DOT ABOVE AND MACRON]\n        '\\u{0254}' | // ɔ  [LATIN SMALL LETTER OPEN O]\n        '\\u{0275}' | // ɵ  [LATIN SMALL LETTER BARRED O]\n        '\\u{1D16}' | // ᴖ  [LATIN SMALL LETTER TOP HALF O]\n        '\\u{1D17}' | // ᴗ  [LATIN SMALL LETTER BOTTOM HALF O]\n        '\\u{1D97}' | // ᶗ  [LATIN SMALL LETTER OPEN O WITH RETROFLEX HOOK]\n        '\\u{1E4D}' | // ṍ  [LATIN SMALL LETTER O WITH TILDE AND ACUTE]\n        '\\u{1E4F}' | // ṏ  [LATIN SMALL LETTER O WITH TILDE AND DIAERESIS]\n        '\\u{1E51}' | // ṑ  [LATIN SMALL LETTER O WITH MACRON AND GRAVE]\n        '\\u{1E53}' | // ṓ  [LATIN SMALL LETTER O WITH MACRON AND ACUTE]\n        '\\u{1ECD}' | // ọ  [LATIN SMALL LETTER O WITH DOT BELOW]\n        '\\u{1ECF}' | // ỏ  [LATIN SMALL LETTER O WITH HOOK ABOVE]\n        '\\u{1ED1}' | // ố  [LATIN SMALL LETTER O WITH CIRCUMFLEX AND ACUTE]\n        '\\u{1ED3}' | // ồ  [LATIN SMALL LETTER O WITH CIRCUMFLEX AND GRAVE]\n        '\\u{1ED5}' | // ổ  [LATIN SMALL LETTER O WITH CIRCUMFLEX AND HOOK ABOVE]\n        '\\u{1ED7}' | // ỗ  [LATIN SMALL LETTER O WITH CIRCUMFLEX AND TILDE]\n        '\\u{1ED9}' | // ộ  [LATIN SMALL LETTER O WITH CIRCUMFLEX AND DOT BELOW]\n        '\\u{1EDB}' | // ớ  [LATIN SMALL LETTER O WITH HORN AND ACUTE]\n        '\\u{1EDD}' | // ờ  [LATIN SMALL LETTER O WITH HORN AND GRAVE]\n        '\\u{1EDF}' | // ở  [LATIN SMALL LETTER O WITH HORN AND HOOK ABOVE]\n        '\\u{1EE1}' | // ỡ  [LATIN SMALL LETTER O WITH HORN AND TILDE]\n        '\\u{1EE3}' | // ợ  [LATIN SMALL LETTER O WITH HORN AND DOT BELOW]\n        '\\u{2092}' | // ₒ  [LATIN SUBSCRIPT SMALL LETTER O]\n        '\\u{24DE}' | // ⓞ  [CIRCLED LATIN SMALL LETTER O]\n        '\\u{2C7A}' | // ⱺ  [LATIN SMALL LETTER O WITH LOW RING INSIDE]\n        '\\u{A74B}' | // ꝋ  [LATIN SMALL LETTER O WITH LONG STROKE OVERLAY]\n        '\\u{A74D}' | // ꝍ  [LATIN SMALL LETTER O WITH LOOP]\n        '\\u{FF4F}' // ｏ  [FULLWIDTH LATIN SMALL LETTER O]\n        => Some(\"o\"),\n        '\\u{0152}' | // Œ  [LATIN CAPITAL LIGATURE OE]\n        '\\u{0276}' // ɶ  [LATIN LETTER SMALL CAPITAL OE]\n        => Some(\"OE\"),\n        '\\u{A74E}' // Ꝏ  [LATIN CAPITAL LETTER OO]\n        => Some(\"OO\"),\n        '\\u{0222}' | // Ȣ  http://en.wikipedia.org/wiki/OU  [LATIN CAPITAL LETTER OU]\n        '\\u{1D15}' // ᴕ  [LATIN LETTER SMALL CAPITAL OU]\n        => Some(\"OU\"),\n        '\\u{24AA}' // ⒪  [PARENTHESIZED LATIN SMALL LETTER O]\n        => Some(\"(o)\"),\n        '\\u{0153}' | // œ  [LATIN SMALL LIGATURE OE]\n        '\\u{1D14}' // ᴔ  [LATIN SMALL LETTER TURNED OE]\n        => Some(\"oe\"),\n        '\\u{A74F}' // ꝏ  [LATIN SMALL LETTER OO]\n        => Some(\"oo\"),\n        '\\u{0223}' // ȣ  http://en.wikipedia.org/wiki/OU  [LATIN SMALL LETTER OU]\n        => Some(\"ou\"),\n        '\\u{01A4}' | // Ƥ  [LATIN CAPITAL LETTER P WITH HOOK]\n        '\\u{1D18}' | // ᴘ  [LATIN LETTER SMALL CAPITAL P]\n        '\\u{1E54}' | // Ṕ  [LATIN CAPITAL LETTER P WITH ACUTE]\n        '\\u{1E56}' | // Ṗ  [LATIN CAPITAL LETTER P WITH DOT ABOVE]\n        '\\u{24C5}' | // Ⓟ  [CIRCLED LATIN CAPITAL LETTER P]\n        '\\u{2C63}' | // Ᵽ  [LATIN CAPITAL LETTER P WITH STROKE]\n        '\\u{A750}' | // Ꝑ  [LATIN CAPITAL LETTER P WITH STROKE THROUGH DESCENDER]\n        '\\u{A752}' | // Ꝓ  [LATIN CAPITAL LETTER P WITH FLOURISH]\n        '\\u{A754}' | // Ꝕ  [LATIN CAPITAL LETTER P WITH SQUIRREL TAIL]\n        '\\u{FF30}' // Ｐ  [FULLWIDTH LATIN CAPITAL LETTER P]\n        => Some(\"P\"),\n        '\\u{01A5}' | // ƥ  [LATIN SMALL LETTER P WITH HOOK]\n        '\\u{1D71}' | // ᵱ  [LATIN SMALL LETTER P WITH MIDDLE TILDE]\n        '\\u{1D7D}' | // ᵽ  [LATIN SMALL LETTER P WITH STROKE]\n        '\\u{1D88}' | // ᶈ  [LATIN SMALL LETTER P WITH PALATAL HOOK]\n        '\\u{1E55}' | // ṕ  [LATIN SMALL LETTER P WITH ACUTE]\n        '\\u{1E57}' | // ṗ  [LATIN SMALL LETTER P WITH DOT ABOVE]\n        '\\u{24DF}' | // ⓟ  [CIRCLED LATIN SMALL LETTER P]\n        '\\u{A751}' | // ꝑ  [LATIN SMALL LETTER P WITH STROKE THROUGH DESCENDER]\n        '\\u{A753}' | // ꝓ  [LATIN SMALL LETTER P WITH FLOURISH]\n        '\\u{A755}' | // ꝕ  [LATIN SMALL LETTER P WITH SQUIRREL TAIL]\n        '\\u{A7FC}' | // ꟼ  [LATIN EPIGRAPHIC LETTER REVERSED P]\n        '\\u{FF50}' // ｐ  [FULLWIDTH LATIN SMALL LETTER P]\n        => Some(\"p\"),\n        '\\u{24AB}' // ⒫  [PARENTHESIZED LATIN SMALL LETTER P]\n        => Some(\"(p)\"),\n        '\\u{024A}' | // Ɋ  [LATIN CAPITAL LETTER SMALL Q WITH HOOK TAIL]\n        '\\u{24C6}' | // Ⓠ  [CIRCLED LATIN CAPITAL LETTER Q]\n        '\\u{A756}' | // Ꝗ  [LATIN CAPITAL LETTER Q WITH STROKE THROUGH DESCENDER]\n        '\\u{A758}' | // Ꝙ  [LATIN CAPITAL LETTER Q WITH DIAGONAL STROKE]\n        '\\u{FF31}' // Ｑ  [FULLWIDTH LATIN CAPITAL LETTER Q]\n        => Some(\"Q\"),\n        '\\u{0138}' | // ĸ  http://en.wikipedia.org/wiki/Kra_(letter)  [LATIN SMALL LETTER KRA]\n        '\\u{024B}' | // ɋ  [LATIN SMALL LETTER Q WITH HOOK TAIL]\n        '\\u{02A0}' | // ʠ  [LATIN SMALL LETTER Q WITH HOOK]\n        '\\u{24E0}' | // ⓠ  [CIRCLED LATIN SMALL LETTER Q]\n        '\\u{A757}' | // ꝗ  [LATIN SMALL LETTER Q WITH STROKE THROUGH DESCENDER]\n        '\\u{A759}' | // ꝙ  [LATIN SMALL LETTER Q WITH DIAGONAL STROKE]\n        '\\u{FF51}' // ｑ  [FULLWIDTH LATIN SMALL LETTER Q]\n        => Some(\"q\"),\n        '\\u{24AC}' // ⒬  [PARENTHESIZED LATIN SMALL LETTER Q]\n        => Some(\"(q)\"),\n        '\\u{0239}' // ȹ  [LATIN SMALL LETTER QP DIGRAPH]\n        => Some(\"qp\"),\n        '\\u{0154}' | // Ŕ  [LATIN CAPITAL LETTER R WITH ACUTE]\n        '\\u{0156}' | // Ŗ  [LATIN CAPITAL LETTER R WITH CEDILLA]\n        '\\u{0158}' | // Ř  [LATIN CAPITAL LETTER R WITH CARON]\n        '\\u{0210}' | // Ȓ  [LATIN CAPITAL LETTER R WITH DOUBLE GRAVE]\n        '\\u{0212}' | // Ȓ  [LATIN CAPITAL LETTER R WITH INVERTED BREVE]\n        '\\u{024C}' | // Ɍ  [LATIN CAPITAL LETTER R WITH STROKE]\n        '\\u{0280}' | // ʀ  [LATIN LETTER SMALL CAPITAL R]\n        '\\u{0281}' | // ʁ  [LATIN LETTER SMALL CAPITAL INVERTED R]\n        '\\u{1D19}' | // ᴙ  [LATIN LETTER SMALL CAPITAL REVERSED R]\n        '\\u{1D1A}' | // ᴚ  [LATIN LETTER SMALL CAPITAL TURNED R]\n        '\\u{1E58}' | // Ṙ  [LATIN CAPITAL LETTER R WITH DOT ABOVE]\n        '\\u{1E5A}' | // Ṛ  [LATIN CAPITAL LETTER R WITH DOT BELOW]\n        '\\u{1E5C}' | // Ṝ  [LATIN CAPITAL LETTER R WITH DOT BELOW AND MACRON]\n        '\\u{1E5E}' | // Ṟ  [LATIN CAPITAL LETTER R WITH LINE BELOW]\n        '\\u{24C7}' | // Ⓡ  [CIRCLED LATIN CAPITAL LETTER R]\n        '\\u{2C64}' | // Ɽ  [LATIN CAPITAL LETTER R WITH TAIL]\n        '\\u{A75A}' | // Ꝛ  [LATIN CAPITAL LETTER R ROTUNDA]\n        '\\u{A782}' | // Ꞃ  [LATIN CAPITAL LETTER INSULAR R]\n        '\\u{FF32}' // Ｒ  [FULLWIDTH LATIN CAPITAL LETTER R]\n        => Some(\"R\"),\n        '\\u{0155}' | // ŕ  [LATIN SMALL LETTER R WITH ACUTE]\n        '\\u{0157}' | // ŗ  [LATIN SMALL LETTER R WITH CEDILLA]\n        '\\u{0159}' | // ř  [LATIN SMALL LETTER R WITH CARON]\n        '\\u{0211}' | // ȑ  [LATIN SMALL LETTER R WITH DOUBLE GRAVE]\n        '\\u{0213}' | // ȓ  [LATIN SMALL LETTER R WITH INVERTED BREVE]\n        '\\u{024D}' | // ɍ  [LATIN SMALL LETTER R WITH STROKE]\n        '\\u{027C}' | // ɼ  [LATIN SMALL LETTER R WITH LONG LEG]\n        '\\u{027D}' | // ɽ  [LATIN SMALL LETTER R WITH TAIL]\n        '\\u{027E}' | // ɾ  [LATIN SMALL LETTER R WITH FISHHOOK]\n        '\\u{027F}' | // ɿ  [LATIN SMALL LETTER REVERSED R WITH FISHHOOK]\n        '\\u{1D63}' | // ᵣ  [LATIN SUBSCRIPT SMALL LETTER R]\n        '\\u{1D72}' | // ᵲ  [LATIN SMALL LETTER R WITH MIDDLE TILDE]\n        '\\u{1D73}' | // ᵳ  [LATIN SMALL LETTER R WITH FISHHOOK AND MIDDLE TILDE]\n        '\\u{1D89}' | // ᶉ  [LATIN SMALL LETTER R WITH PALATAL HOOK]\n        '\\u{1E59}' | // ṙ  [LATIN SMALL LETTER R WITH DOT ABOVE]\n        '\\u{1E5B}' | // ṛ  [LATIN SMALL LETTER R WITH DOT BELOW]\n        '\\u{1E5D}' | // ṝ  [LATIN SMALL LETTER R WITH DOT BELOW AND MACRON]\n        '\\u{1E5F}' | // ṟ  [LATIN SMALL LETTER R WITH LINE BELOW]\n        '\\u{24E1}' | // ⓡ  [CIRCLED LATIN SMALL LETTER R]\n        '\\u{A75B}' | // ꝛ  [LATIN SMALL LETTER R ROTUNDA]\n        '\\u{A783}' | // ꞃ  [LATIN SMALL LETTER INSULAR R]\n        '\\u{FF52}' // ｒ  [FULLWIDTH LATIN SMALL LETTER R]\n        => Some(\"r\"),\n        '\\u{24AD}' // ⒭  [PARENTHESIZED LATIN SMALL LETTER R]\n        => Some(\"(r)\"),\n        '\\u{015A}' | // Ś  [LATIN CAPITAL LETTER S WITH ACUTE]\n        '\\u{015C}' | // Ŝ  [LATIN CAPITAL LETTER S WITH CIRCUMFLEX]\n        '\\u{015E}' | // Ş  [LATIN CAPITAL LETTER S WITH CEDILLA]\n        '\\u{0160}' | // Š  [LATIN CAPITAL LETTER S WITH CARON]\n        '\\u{0218}' | // Ș  [LATIN CAPITAL LETTER S WITH COMMA BELOW]\n        '\\u{1E60}' | // Ṡ  [LATIN CAPITAL LETTER S WITH DOT ABOVE]\n        '\\u{1E62}' | // Ṣ  [LATIN CAPITAL LETTER S WITH DOT BELOW]\n        '\\u{1E64}' | // Ṥ  [LATIN CAPITAL LETTER S WITH ACUTE AND DOT ABOVE]\n        '\\u{1E66}' | // Ṧ  [LATIN CAPITAL LETTER S WITH CARON AND DOT ABOVE]\n        '\\u{1E68}' | // Ṩ  [LATIN CAPITAL LETTER S WITH DOT BELOW AND DOT ABOVE]\n        '\\u{24C8}' | // Ⓢ  [CIRCLED LATIN CAPITAL LETTER S]\n        '\\u{A731}' | // ꜱ  [LATIN LETTER SMALL CAPITAL S]\n        '\\u{A785}' | // ꞅ  [LATIN SMALL LETTER INSULAR S]\n        '\\u{FF33}' // Ｓ  [FULLWIDTH LATIN CAPITAL LETTER S]\n        => Some(\"S\"),\n        '\\u{015B}' | // ś  [LATIN SMALL LETTER S WITH ACUTE]\n        '\\u{015D}' | // ŝ  [LATIN SMALL LETTER S WITH CIRCUMFLEX]\n        '\\u{015F}' | // ş  [LATIN SMALL LETTER S WITH CEDILLA]\n        '\\u{0161}' | // š  [LATIN SMALL LETTER S WITH CARON]\n        '\\u{017F}' | // ſ  http://en.wikipedia.org/wiki/Long_S  [LATIN SMALL LETTER LONG S]\n        '\\u{0219}' | // ș  [LATIN SMALL LETTER S WITH COMMA BELOW]\n        '\\u{023F}' | // ȿ  [LATIN SMALL LETTER S WITH SWASH TAIL]\n        '\\u{0282}' | // ʂ  [LATIN SMALL LETTER S WITH HOOK]\n        '\\u{1D74}' | // ᵴ  [LATIN SMALL LETTER S WITH MIDDLE TILDE]\n        '\\u{1D8A}' | // ᶊ  [LATIN SMALL LETTER S WITH PALATAL HOOK]\n        '\\u{1E61}' | // ṡ  [LATIN SMALL LETTER S WITH DOT ABOVE]\n        '\\u{1E63}' | // ṣ  [LATIN SMALL LETTER S WITH DOT BELOW]\n        '\\u{1E65}' | // ṥ  [LATIN SMALL LETTER S WITH ACUTE AND DOT ABOVE]\n        '\\u{1E67}' | // ṧ  [LATIN SMALL LETTER S WITH CARON AND DOT ABOVE]\n        '\\u{1E69}' | // ṩ  [LATIN SMALL LETTER S WITH DOT BELOW AND DOT ABOVE]\n        '\\u{1E9C}' | // ẜ  [LATIN SMALL LETTER LONG S WITH DIAGONAL STROKE]\n        '\\u{1E9D}' | // ẝ  [LATIN SMALL LETTER LONG S WITH HIGH STROKE]\n        '\\u{24E2}' | // ⓢ  [CIRCLED LATIN SMALL LETTER S]\n        '\\u{A784}' | // Ꞅ  [LATIN CAPITAL LETTER INSULAR S]\n        '\\u{FF53}' // ｓ  [FULLWIDTH LATIN SMALL LETTER S]\n        => Some(\"s\"),\n        '\\u{1E9E}' // ẞ  [LATIN CAPITAL LETTER SHARP S]\n        => Some(\"SS\"),\n        '\\u{24AE}' // ⒮  [PARENTHESIZED LATIN SMALL LETTER S]\n        => Some(\"(s)\"),\n        '\\u{00DF}' // ß  [LATIN SMALL LETTER SHARP S]\n        => Some(\"ss\"),\n        '\\u{FB06}' // st  [LATIN SMALL LIGATURE ST]\n        => Some(\"st\"),\n        '\\u{0162}' | // Ţ  [LATIN CAPITAL LETTER T WITH CEDILLA]\n        '\\u{0164}' | // Ť  [LATIN CAPITAL LETTER T WITH CARON]\n        '\\u{0166}' | // Ŧ  [LATIN CAPITAL LETTER T WITH STROKE]\n        '\\u{01AC}' | // Ƭ  [LATIN CAPITAL LETTER T WITH HOOK]\n        '\\u{01AE}' | // Ʈ  [LATIN CAPITAL LETTER T WITH RETROFLEX HOOK]\n        '\\u{021A}' | // Ț  [LATIN CAPITAL LETTER T WITH COMMA BELOW]\n        '\\u{023E}' | // Ⱦ  [LATIN CAPITAL LETTER T WITH DIAGONAL STROKE]\n        '\\u{1D1B}' | // ᴛ  [LATIN LETTER SMALL CAPITAL T]\n        '\\u{1E6A}' | // Ṫ  [LATIN CAPITAL LETTER T WITH DOT ABOVE]\n        '\\u{1E6C}' | // Ṭ  [LATIN CAPITAL LETTER T WITH DOT BELOW]\n        '\\u{1E6E}' | // Ṯ  [LATIN CAPITAL LETTER T WITH LINE BELOW]\n        '\\u{1E70}' | // Ṱ  [LATIN CAPITAL LETTER T WITH CIRCUMFLEX BELOW]\n        '\\u{24C9}' | // Ⓣ  [CIRCLED LATIN CAPITAL LETTER T]\n        '\\u{A786}' | // Ꞇ  [LATIN CAPITAL LETTER INSULAR T]\n        '\\u{FF34}' // Ｔ  [FULLWIDTH LATIN CAPITAL LETTER T]\n        => Some(\"T\"),\n        '\\u{0163}' | // ţ  [LATIN SMALL LETTER T WITH CEDILLA]\n        '\\u{0165}' | // ť  [LATIN SMALL LETTER T WITH CARON]\n        '\\u{0167}' | // ŧ  [LATIN SMALL LETTER T WITH STROKE]\n        '\\u{01AB}' | // ƫ  [LATIN SMALL LETTER T WITH PALATAL HOOK]\n        '\\u{01AD}' | // ƭ  [LATIN SMALL LETTER T WITH HOOK]\n        '\\u{021B}' | // ț  [LATIN SMALL LETTER T WITH COMMA BELOW]\n        '\\u{0236}' | // ȶ  [LATIN SMALL LETTER T WITH CURL]\n        '\\u{0287}' | // ʇ  [LATIN SMALL LETTER TURNED T]\n        '\\u{0288}' | // ʈ  [LATIN SMALL LETTER T WITH RETROFLEX HOOK]\n        '\\u{1D75}' | // ᵵ  [LATIN SMALL LETTER T WITH MIDDLE TILDE]\n        '\\u{1E6B}' | // ṫ  [LATIN SMALL LETTER T WITH DOT ABOVE]\n        '\\u{1E6D}' | // ṭ  [LATIN SMALL LETTER T WITH DOT BELOW]\n        '\\u{1E6F}' | // ṯ  [LATIN SMALL LETTER T WITH LINE BELOW]\n        '\\u{1E71}' | // ṱ  [LATIN SMALL LETTER T WITH CIRCUMFLEX BELOW]\n        '\\u{1E97}' | // ẗ  [LATIN SMALL LETTER T WITH DIAERESIS]\n        '\\u{24E3}' | // ⓣ  [CIRCLED LATIN SMALL LETTER T]\n        '\\u{2C66}' | // ⱦ  [LATIN SMALL LETTER T WITH DIAGONAL STROKE]\n        '\\u{FF54}' // ｔ  [FULLWIDTH LATIN SMALL LETTER T]\n        => Some(\"t\"),\n        '\\u{00DE}' | // Þ  [LATIN CAPITAL LETTER THORN]\n        '\\u{A766}' // Ꝧ  [LATIN CAPITAL LETTER THORN WITH STROKE THROUGH DESCENDER]\n        => Some(\"TH\"),\n        '\\u{A728}' // Ꜩ  [LATIN CAPITAL LETTER TZ]\n        => Some(\"TZ\"),\n        '\\u{24AF}' // ⒯  [PARENTHESIZED LATIN SMALL LETTER T]\n        => Some(\"(t)\"),\n        '\\u{02A8}' // ʨ  [LATIN SMALL LETTER TC DIGRAPH WITH CURL]\n        => Some(\"tc\"),\n        '\\u{00FE}' | // þ  [LATIN SMALL LETTER THORN]\n        '\\u{1D7A}' | // ᵺ  [LATIN SMALL LETTER TH WITH STRIKETHROUGH]\n        '\\u{A767}' // ꝧ  [LATIN SMALL LETTER THORN WITH STROKE THROUGH DESCENDER]\n        => Some(\"th\"),\n        '\\u{02A6}' // ʦ  [LATIN SMALL LETTER TS DIGRAPH]\n        => Some(\"ts\"),\n        '\\u{A729}' // ꜩ  [LATIN SMALL LETTER TZ]\n        => Some(\"tz\"),\n        '\\u{00D9}' | // Ù  [LATIN CAPITAL LETTER U WITH GRAVE]\n        '\\u{00DA}' | // Ú  [LATIN CAPITAL LETTER U WITH ACUTE]\n        '\\u{00DB}' | // Û  [LATIN CAPITAL LETTER U WITH CIRCUMFLEX]\n        '\\u{00DC}' | // Ü  [LATIN CAPITAL LETTER U WITH DIAERESIS]\n        '\\u{0168}' | // Ũ  [LATIN CAPITAL LETTER U WITH TILDE]\n        '\\u{016A}' | // Ū  [LATIN CAPITAL LETTER U WITH MACRON]\n        '\\u{016C}' | // Ŭ  [LATIN CAPITAL LETTER U WITH BREVE]\n        '\\u{016E}' | // Ů  [LATIN CAPITAL LETTER U WITH RING ABOVE]\n        '\\u{0170}' | // Ű  [LATIN CAPITAL LETTER U WITH DOUBLE ACUTE]\n        '\\u{0172}' | // Ų  [LATIN CAPITAL LETTER U WITH OGONEK]\n        '\\u{01AF}' | // Ư  [LATIN CAPITAL LETTER U WITH HORN]\n        '\\u{01D3}' | // Ǔ  [LATIN CAPITAL LETTER U WITH CARON]\n        '\\u{01D5}' | // Ǖ  [LATIN CAPITAL LETTER U WITH DIAERESIS AND MACRON]\n        '\\u{01D7}' | // Ǘ  [LATIN CAPITAL LETTER U WITH DIAERESIS AND ACUTE]\n        '\\u{01D9}' | // Ǚ  [LATIN CAPITAL LETTER U WITH DIAERESIS AND CARON]\n        '\\u{01DB}' | // Ǜ  [LATIN CAPITAL LETTER U WITH DIAERESIS AND GRAVE]\n        '\\u{0214}' | // Ȕ  [LATIN CAPITAL LETTER U WITH DOUBLE GRAVE]\n        '\\u{0216}' | // Ȗ  [LATIN CAPITAL LETTER U WITH INVERTED BREVE]\n        '\\u{0244}' | // Ʉ  [LATIN CAPITAL LETTER U BAR]\n        '\\u{1D1C}' | // ᴜ  [LATIN LETTER SMALL CAPITAL U]\n        '\\u{1D7E}' | // ᵾ  [LATIN SMALL CAPITAL LETTER U WITH STROKE]\n        '\\u{1E72}' | // Ṳ  [LATIN CAPITAL LETTER U WITH DIAERESIS BELOW]\n        '\\u{1E74}' | // Ṵ  [LATIN CAPITAL LETTER U WITH TILDE BELOW]\n        '\\u{1E76}' | // Ṷ  [LATIN CAPITAL LETTER U WITH CIRCUMFLEX BELOW]\n        '\\u{1E78}' | // Ṹ  [LATIN CAPITAL LETTER U WITH TILDE AND ACUTE]\n        '\\u{1E7A}' | // Ṻ  [LATIN CAPITAL LETTER U WITH MACRON AND DIAERESIS]\n        '\\u{1EE4}' | // Ụ  [LATIN CAPITAL LETTER U WITH DOT BELOW]\n        '\\u{1EE6}' | // Ủ  [LATIN CAPITAL LETTER U WITH HOOK ABOVE]\n        '\\u{1EE8}' | // Ứ  [LATIN CAPITAL LETTER U WITH HORN AND ACUTE]\n        '\\u{1EEA}' | // Ừ  [LATIN CAPITAL LETTER U WITH HORN AND GRAVE]\n        '\\u{1EEC}' | // Ử  [LATIN CAPITAL LETTER U WITH HORN AND HOOK ABOVE]\n        '\\u{1EEE}' | // Ữ  [LATIN CAPITAL LETTER U WITH HORN AND TILDE]\n        '\\u{1EF0}' | // Ự  [LATIN CAPITAL LETTER U WITH HORN AND DOT BELOW]\n        '\\u{24CA}' | // Ⓤ  [CIRCLED LATIN CAPITAL LETTER U]\n        '\\u{FF35}' // Ｕ  [FULLWIDTH LATIN CAPITAL LETTER U]\n        => Some(\"U\"),\n        '\\u{00F9}' | // ù  [LATIN SMALL LETTER U WITH GRAVE]\n        '\\u{00FA}' | // ú  [LATIN SMALL LETTER U WITH ACUTE]\n        '\\u{00FB}' | // û  [LATIN SMALL LETTER U WITH CIRCUMFLEX]\n        '\\u{00FC}' | // ü  [LATIN SMALL LETTER U WITH DIAERESIS]\n        '\\u{0169}' | // ũ  [LATIN SMALL LETTER U WITH TILDE]\n        '\\u{016B}' | // ū  [LATIN SMALL LETTER U WITH MACRON]\n        '\\u{016D}' | // ŭ  [LATIN SMALL LETTER U WITH BREVE]\n        '\\u{016F}' | // ů  [LATIN SMALL LETTER U WITH RING ABOVE]\n        '\\u{0171}' | // ű  [LATIN SMALL LETTER U WITH DOUBLE ACUTE]\n        '\\u{0173}' | // ų  [LATIN SMALL LETTER U WITH OGONEK]\n        '\\u{01B0}' | // ư  [LATIN SMALL LETTER U WITH HORN]\n        '\\u{01D4}' | // ǔ  [LATIN SMALL LETTER U WITH CARON]\n        '\\u{01D6}' | // ǖ  [LATIN SMALL LETTER U WITH DIAERESIS AND MACRON]\n        '\\u{01D8}' | // ǘ  [LATIN SMALL LETTER U WITH DIAERESIS AND ACUTE]\n        '\\u{01DA}' | // ǚ  [LATIN SMALL LETTER U WITH DIAERESIS AND CARON]\n        '\\u{01DC}' | // ǜ  [LATIN SMALL LETTER U WITH DIAERESIS AND GRAVE]\n        '\\u{0215}' | // ȕ  [LATIN SMALL LETTER U WITH DOUBLE GRAVE]\n        '\\u{0217}' | // ȗ  [LATIN SMALL LETTER U WITH INVERTED BREVE]\n        '\\u{0289}' | // ʉ  [LATIN SMALL LETTER U BAR]\n        '\\u{1D64}' | // ᵤ  [LATIN SUBSCRIPT SMALL LETTER U]\n        '\\u{1D99}' | // ᶙ  [LATIN SMALL LETTER U WITH RETROFLEX HOOK]\n        '\\u{1E73}' | // ṳ  [LATIN SMALL LETTER U WITH DIAERESIS BELOW]\n        '\\u{1E75}' | // ṵ  [LATIN SMALL LETTER U WITH TILDE BELOW]\n        '\\u{1E77}' | // ṷ  [LATIN SMALL LETTER U WITH CIRCUMFLEX BELOW]\n        '\\u{1E79}' | // ṹ  [LATIN SMALL LETTER U WITH TILDE AND ACUTE]\n        '\\u{1E7B}' | // ṻ  [LATIN SMALL LETTER U WITH MACRON AND DIAERESIS]\n        '\\u{1EE5}' | // ụ  [LATIN SMALL LETTER U WITH DOT BELOW]\n        '\\u{1EE7}' | // ủ  [LATIN SMALL LETTER U WITH HOOK ABOVE]\n        '\\u{1EE9}' | // ứ  [LATIN SMALL LETTER U WITH HORN AND ACUTE]\n        '\\u{1EEB}' | // ừ  [LATIN SMALL LETTER U WITH HORN AND GRAVE]\n        '\\u{1EED}' | // ử  [LATIN SMALL LETTER U WITH HORN AND HOOK ABOVE]\n        '\\u{1EEF}' | // ữ  [LATIN SMALL LETTER U WITH HORN AND TILDE]\n        '\\u{1EF1}' | // ự  [LATIN SMALL LETTER U WITH HORN AND DOT BELOW]\n        '\\u{24E4}' | // ⓤ  [CIRCLED LATIN SMALL LETTER U]\n        '\\u{FF55}' // ｕ  [FULLWIDTH LATIN SMALL LETTER U]\n        => Some(\"u\"),\n        '\\u{24B0}' // ⒰  [PARENTHESIZED LATIN SMALL LETTER U]\n        => Some(\"(u)\"),\n        '\\u{1D6B}' // ᵫ  [LATIN SMALL LETTER UE]\n        => Some(\"ue\"),\n        '\\u{01B2}' | // Ʋ  [LATIN CAPITAL LETTER V WITH HOOK]\n        '\\u{0245}' | // Ʌ  [LATIN CAPITAL LETTER TURNED V]\n        '\\u{1D20}' | // ᴠ  [LATIN LETTER SMALL CAPITAL V]\n        '\\u{1E7C}' | // Ṽ  [LATIN CAPITAL LETTER V WITH TILDE]\n        '\\u{1E7E}' | // Ṿ  [LATIN CAPITAL LETTER V WITH DOT BELOW]\n        '\\u{1EFC}' | // Ỽ  [LATIN CAPITAL LETTER MIDDLE-WELSH V]\n        '\\u{24CB}' | // Ⓥ  [CIRCLED LATIN CAPITAL LETTER V]\n        '\\u{A75E}' | // Ꝟ  [LATIN CAPITAL LETTER V WITH DIAGONAL STROKE]\n        '\\u{A768}' | // Ꝩ  [LATIN CAPITAL LETTER VEND]\n        '\\u{FF36}' // Ｖ  [FULLWIDTH LATIN CAPITAL LETTER V]\n        => Some(\"V\"),\n        '\\u{028B}' | // ʋ  [LATIN SMALL LETTER V WITH HOOK]\n        '\\u{028C}' | // ʌ  [LATIN SMALL LETTER TURNED V]\n        '\\u{1D65}' | // ᵥ  [LATIN SUBSCRIPT SMALL LETTER V]\n        '\\u{1D8C}' | // ᶌ  [LATIN SMALL LETTER V WITH PALATAL HOOK]\n        '\\u{1E7D}' | // ṽ  [LATIN SMALL LETTER V WITH TILDE]\n        '\\u{1E7F}' | // ṿ  [LATIN SMALL LETTER V WITH DOT BELOW]\n        '\\u{24E5}' | // ⓥ  [CIRCLED LATIN SMALL LETTER V]\n        '\\u{2C71}' | // ⱱ  [LATIN SMALL LETTER V WITH RIGHT HOOK]\n        '\\u{2C74}' | // ⱴ  [LATIN SMALL LETTER V WITH CURL]\n        '\\u{A75F}' | // ꝟ  [LATIN SMALL LETTER V WITH DIAGONAL STROKE]\n        '\\u{FF56}' // ｖ  [FULLWIDTH LATIN SMALL LETTER V]\n        => Some(\"v\"),\n        '\\u{A760}' // Ꝡ  [LATIN CAPITAL LETTER VY]\n        => Some(\"VY\"),\n        '\\u{24B1}' // ⒱  [PARENTHESIZED LATIN SMALL LETTER V]\n        => Some(\"(v)\"),\n        '\\u{A761}' // ꝡ  [LATIN SMALL LETTER VY]\n        => Some(\"vy\"),\n        '\\u{0174}' | // Ŵ  [LATIN CAPITAL LETTER W WITH CIRCUMFLEX]\n        '\\u{01F7}' | // Ƿ  http://en.wikipedia.org/wiki/Wynn  [LATIN CAPITAL LETTER WYNN]\n        '\\u{1D21}' | // ᴡ  [LATIN LETTER SMALL CAPITAL W]\n        '\\u{1E80}' | // Ẁ  [LATIN CAPITAL LETTER W WITH GRAVE]\n        '\\u{1E82}' | // Ẃ  [LATIN CAPITAL LETTER W WITH ACUTE]\n        '\\u{1E84}' | // Ẅ  [LATIN CAPITAL LETTER W WITH DIAERESIS]\n        '\\u{1E86}' | // Ẇ  [LATIN CAPITAL LETTER W WITH DOT ABOVE]\n        '\\u{1E88}' | // Ẉ  [LATIN CAPITAL LETTER W WITH DOT BELOW]\n        '\\u{24CC}' | // Ⓦ  [CIRCLED LATIN CAPITAL LETTER W]\n        '\\u{2C72}' | // Ⱳ  [LATIN CAPITAL LETTER W WITH HOOK]\n        '\\u{FF37}' // Ｗ  [FULLWIDTH LATIN CAPITAL LETTER W]\n        => Some(\"W\"),\n        '\\u{0175}' | // ŵ  [LATIN SMALL LETTER W WITH CIRCUMFLEX]\n        '\\u{01BF}' | // ƿ  http://en.wikipedia.org/wiki/Wynn  [LATIN LETTER WYNN]\n        '\\u{028D}' | // ʍ  [LATIN SMALL LETTER TURNED W]\n        '\\u{1E81}' | // ẁ  [LATIN SMALL LETTER W WITH GRAVE]\n        '\\u{1E83}' | // ẃ  [LATIN SMALL LETTER W WITH ACUTE]\n        '\\u{1E85}' | // ẅ  [LATIN SMALL LETTER W WITH DIAERESIS]\n        '\\u{1E87}' | // ẇ  [LATIN SMALL LETTER W WITH DOT ABOVE]\n        '\\u{1E89}' | // ẉ  [LATIN SMALL LETTER W WITH DOT BELOW]\n        '\\u{1E98}' | // ẘ  [LATIN SMALL LETTER W WITH RING ABOVE]\n        '\\u{24E6}' | // ⓦ  [CIRCLED LATIN SMALL LETTER W]\n        '\\u{2C73}' | // ⱳ  [LATIN SMALL LETTER W WITH HOOK]\n        '\\u{FF57}' // ｗ  [FULLWIDTH LATIN SMALL LETTER W]\n        => Some(\"w\"),\n        '\\u{24B2}' // ⒲  [PARENTHESIZED LATIN SMALL LETTER W]\n        => Some(\"(w)\"),\n        '\\u{1E8A}' | // Ẋ  [LATIN CAPITAL LETTER X WITH DOT ABOVE]\n        '\\u{1E8C}' | // Ẍ  [LATIN CAPITAL LETTER X WITH DIAERESIS]\n        '\\u{24CD}' | // Ⓧ  [CIRCLED LATIN CAPITAL LETTER X]\n        '\\u{FF38}' // Ｘ  [FULLWIDTH LATIN CAPITAL LETTER X]\n        => Some(\"X\"),\n        '\\u{1D8D}' | // ᶍ  [LATIN SMALL LETTER X WITH PALATAL HOOK]\n        '\\u{1E8B}' | // ẋ  [LATIN SMALL LETTER X WITH DOT ABOVE]\n        '\\u{1E8D}' | // ẍ  [LATIN SMALL LETTER X WITH DIAERESIS]\n        '\\u{2093}' | // ₓ  [LATIN SUBSCRIPT SMALL LETTER X]\n        '\\u{24E7}' | // ⓧ  [CIRCLED LATIN SMALL LETTER X]\n        '\\u{FF58}' // ｘ  [FULLWIDTH LATIN SMALL LETTER X]\n        => Some(\"x\"),\n        '\\u{24B3}' // ⒳  [PARENTHESIZED LATIN SMALL LETTER X]\n        => Some(\"(x)\"),\n        '\\u{00DD}' | // Ý  [LATIN CAPITAL LETTER Y WITH ACUTE]\n        '\\u{0176}' | // Ŷ  [LATIN CAPITAL LETTER Y WITH CIRCUMFLEX]\n        '\\u{0178}' | // Ÿ  [LATIN CAPITAL LETTER Y WITH DIAERESIS]\n        '\\u{01B3}' | // Ƴ  [LATIN CAPITAL LETTER Y WITH HOOK]\n        '\\u{0232}' | // Ȳ  [LATIN CAPITAL LETTER Y WITH MACRON]\n        '\\u{024E}' | // Ɏ  [LATIN CAPITAL LETTER Y WITH STROKE]\n        '\\u{028F}' | // ʏ  [LATIN LETTER SMALL CAPITAL Y]\n        '\\u{1E8E}' | // Ẏ  [LATIN CAPITAL LETTER Y WITH DOT ABOVE]\n        '\\u{1EF2}' | // Ỳ  [LATIN CAPITAL LETTER Y WITH GRAVE]\n        '\\u{1EF4}' | // Ỵ  [LATIN CAPITAL LETTER Y WITH DOT BELOW]\n        '\\u{1EF6}' | // Ỷ  [LATIN CAPITAL LETTER Y WITH HOOK ABOVE]\n        '\\u{1EF8}' | // Ỹ  [LATIN CAPITAL LETTER Y WITH TILDE]\n        '\\u{1EFE}' | // Ỿ  [LATIN CAPITAL LETTER Y WITH LOOP]\n        '\\u{24CE}' | // Ⓨ  [CIRCLED LATIN CAPITAL LETTER Y]\n        '\\u{FF39}' // Ｙ  [FULLWIDTH LATIN CAPITAL LETTER Y]\n        => Some(\"Y\"),\n        '\\u{00FD}' | // ý  [LATIN SMALL LETTER Y WITH ACUTE]\n        '\\u{00FF}' | // ÿ  [LATIN SMALL LETTER Y WITH DIAERESIS]\n        '\\u{0177}' | // ŷ  [LATIN SMALL LETTER Y WITH CIRCUMFLEX]\n        '\\u{01B4}' | // ƴ  [LATIN SMALL LETTER Y WITH HOOK]\n        '\\u{0233}' | // ȳ  [LATIN SMALL LETTER Y WITH MACRON]\n        '\\u{024F}' | // ɏ  [LATIN SMALL LETTER Y WITH STROKE]\n        '\\u{028E}' | // ʎ  [LATIN SMALL LETTER TURNED Y]\n        '\\u{1E8F}' | // ẏ  [LATIN SMALL LETTER Y WITH DOT ABOVE]\n        '\\u{1E99}' | // ẙ  [LATIN SMALL LETTER Y WITH RING ABOVE]\n        '\\u{1EF3}' | // ỳ  [LATIN SMALL LETTER Y WITH GRAVE]\n        '\\u{1EF5}' | // ỵ  [LATIN SMALL LETTER Y WITH DOT BELOW]\n        '\\u{1EF7}' | // ỷ  [LATIN SMALL LETTER Y WITH HOOK ABOVE]\n        '\\u{1EF9}' | // ỹ  [LATIN SMALL LETTER Y WITH TILDE]\n        '\\u{1EFF}' | // ỿ  [LATIN SMALL LETTER Y WITH LOOP]\n        '\\u{24E8}' | // ⓨ  [CIRCLED LATIN SMALL LETTER Y]\n        '\\u{FF59}' // ｙ  [FULLWIDTH LATIN SMALL LETTER Y]\n        => Some(\"y\"),\n        '\\u{24B4}' // ⒴  [PARENTHESIZED LATIN SMALL LETTER Y]\n        => Some(\"(y)\"),\n        '\\u{0179}' | // Ź  [LATIN CAPITAL LETTER Z WITH ACUTE]\n        '\\u{017B}' | // Ż  [LATIN CAPITAL LETTER Z WITH DOT ABOVE]\n        '\\u{017D}' | // Ž  [LATIN CAPITAL LETTER Z WITH CARON]\n        '\\u{01B5}' | // Ƶ  [LATIN CAPITAL LETTER Z WITH STROKE]\n        '\\u{021C}' | // Ȝ  http://en.wikipedia.org/wiki/Yogh  [LATIN CAPITAL LETTER YOGH]\n        '\\u{0224}' | // Ȥ  [LATIN CAPITAL LETTER Z WITH HOOK]\n        '\\u{1D22}' | // ᴢ  [LATIN LETTER SMALL CAPITAL Z]\n        '\\u{1E90}' | // Ẑ  [LATIN CAPITAL LETTER Z WITH CIRCUMFLEX]\n        '\\u{1E92}' | // Ẓ  [LATIN CAPITAL LETTER Z WITH DOT BELOW]\n        '\\u{1E94}' | // Ẕ  [LATIN CAPITAL LETTER Z WITH LINE BELOW]\n        '\\u{24CF}' | // Ⓩ  [CIRCLED LATIN CAPITAL LETTER Z]\n        '\\u{2C6B}' | // Ⱬ  [LATIN CAPITAL LETTER Z WITH DESCENDER]\n        '\\u{A762}' | // Ꝣ  [LATIN CAPITAL LETTER VISIGOTHIC Z]\n        '\\u{FF3A}' // Ｚ  [FULLWIDTH LATIN CAPITAL LETTER Z]\n        => Some(\"Z\"),\n        '\\u{017A}' | // ź  [LATIN SMALL LETTER Z WITH ACUTE]\n        '\\u{017C}' | // ż  [LATIN SMALL LETTER Z WITH DOT ABOVE]\n        '\\u{017E}' | // ž  [LATIN SMALL LETTER Z WITH CARON]\n        '\\u{01B6}' | // ƶ  [LATIN SMALL LETTER Z WITH STROKE]\n        '\\u{021D}' | // ȝ  http://en.wikipedia.org/wiki/Yogh  [LATIN SMALL LETTER YOGH]\n        '\\u{0225}' | // ȥ  [LATIN SMALL LETTER Z WITH HOOK]\n        '\\u{0240}' | // ɀ  [LATIN SMALL LETTER Z WITH SWASH TAIL]\n        '\\u{0290}' | // ʐ  [LATIN SMALL LETTER Z WITH RETROFLEX HOOK]\n        '\\u{0291}' | // ʑ  [LATIN SMALL LETTER Z WITH CURL]\n        '\\u{1D76}' | // ᵶ  [LATIN SMALL LETTER Z WITH MIDDLE TILDE]\n        '\\u{1D8E}' | // ᶎ  [LATIN SMALL LETTER Z WITH PALATAL HOOK]\n        '\\u{1E91}' | // ẑ  [LATIN SMALL LETTER Z WITH CIRCUMFLEX]\n        '\\u{1E93}' | // ẓ  [LATIN SMALL LETTER Z WITH DOT BELOW]\n        '\\u{1E95}' | // ẕ  [LATIN SMALL LETTER Z WITH LINE BELOW]\n        '\\u{24E9}' | // ⓩ  [CIRCLED LATIN SMALL LETTER Z]\n        '\\u{2C6C}' | // ⱬ  [LATIN SMALL LETTER Z WITH DESCENDER]\n        '\\u{A763}' | // ꝣ  [LATIN SMALL LETTER VISIGOTHIC Z]\n        '\\u{FF5A}' // ｚ  [FULLWIDTH LATIN SMALL LETTER Z]\n        => Some(\"z\"),\n        '\\u{24B5}' // ⒵  [PARENTHESIZED LATIN SMALL LETTER Z]\n        => Some(\"(z)\"),\n        '\\u{2070}' | // ⁰  [SUPERSCRIPT ZERO]\n        '\\u{2080}' | // ₀  [SUBSCRIPT ZERO]\n        '\\u{24EA}' | // ⓪  [CIRCLED DIGIT ZERO]\n        '\\u{24FF}' | // ⓿  [NEGATIVE CIRCLED DIGIT ZERO]\n        '\\u{FF10}' // ０  [FULLWIDTH DIGIT ZERO]\n        => Some(\"0\"),\n        '\\u{00B9}' | // ¹  [SUPERSCRIPT ONE]\n        '\\u{2081}' | // ₁  [SUBSCRIPT ONE]\n        '\\u{2460}' | // ①  [CIRCLED DIGIT ONE]\n        '\\u{24F5}' | // ⓵  [DOUBLE CIRCLED DIGIT ONE]\n        '\\u{2776}' | // ❶  [DINGBAT NEGATIVE CIRCLED DIGIT ONE]\n        '\\u{2780}' | // ➀  [DINGBAT CIRCLED SANS-SERIF DIGIT ONE]\n        '\\u{278A}' | // ➊  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT ONE]\n        '\\u{FF11}' // １  [FULLWIDTH DIGIT ONE]\n        => Some(\"1\"),\n        '\\u{2488}' // ⒈  [DIGIT ONE FULL STOP]\n        => Some(\"1.\"),\n        '\\u{2474}' // ⑴  [PARENTHESIZED DIGIT ONE]\n        => Some(\"(1)\"),\n        '\\u{00B2}' | // ²  [SUPERSCRIPT TWO]\n        '\\u{2082}' | // ₂  [SUBSCRIPT TWO]\n        '\\u{2461}' | // ②  [CIRCLED DIGIT TWO]\n        '\\u{24F6}' | // ⓶  [DOUBLE CIRCLED DIGIT TWO]\n        '\\u{2777}' | // ❷  [DINGBAT NEGATIVE CIRCLED DIGIT TWO]\n        '\\u{2781}' | // ➁  [DINGBAT CIRCLED SANS-SERIF DIGIT TWO]\n        '\\u{278B}' | // ➋  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT TWO]\n        '\\u{FF12}' // ２  [FULLWIDTH DIGIT TWO]\n        => Some(\"2\"),\n        '\\u{2489}' // ⒉  [DIGIT TWO FULL STOP]\n        => Some(\"2.\"),\n        '\\u{2475}' // ⑵  [PARENTHESIZED DIGIT TWO]\n        => Some(\"(2)\"),\n        '\\u{00B3}' | // ³  [SUPERSCRIPT THREE]\n        '\\u{2083}' | // ₃  [SUBSCRIPT THREE]\n        '\\u{2462}' | // ③  [CIRCLED DIGIT THREE]\n        '\\u{24F7}' | // ⓷  [DOUBLE CIRCLED DIGIT THREE]\n        '\\u{2778}' | // ❸  [DINGBAT NEGATIVE CIRCLED DIGIT THREE]\n        '\\u{2782}' | // ➂  [DINGBAT CIRCLED SANS-SERIF DIGIT THREE]\n        '\\u{278C}' | // ➌  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT THREE]\n        '\\u{FF13}' // ３  [FULLWIDTH DIGIT THREE]\n        => Some(\"3\"),\n        '\\u{248A}' // ⒊  [DIGIT THREE FULL STOP]\n        => Some(\"3.\"),\n        '\\u{2476}' // ⑶  [PARENTHESIZED DIGIT THREE]\n        => Some(\"(3)\"),\n        '\\u{2074}' | // ⁴  [SUPERSCRIPT FOUR]\n        '\\u{2084}' | // ₄  [SUBSCRIPT FOUR]\n        '\\u{2463}' | // ④  [CIRCLED DIGIT FOUR]\n        '\\u{24F8}' | // ⓸  [DOUBLE CIRCLED DIGIT FOUR]\n        '\\u{2779}' | // ❹  [DINGBAT NEGATIVE CIRCLED DIGIT FOUR]\n        '\\u{2783}' | // ➃  [DINGBAT CIRCLED SANS-SERIF DIGIT FOUR]\n        '\\u{278D}' | // ➍  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT FOUR]\n        '\\u{FF14}' // ４  [FULLWIDTH DIGIT FOUR]\n        => Some(\"4\"),\n        '\\u{248B}' // ⒋  [DIGIT FOUR FULL STOP]\n        => Some(\"4.\"),\n        '\\u{2477}' // ⑷  [PARENTHESIZED DIGIT FOUR]\n        => Some(\"(4)\"),\n        '\\u{2075}' | // ⁵  [SUPERSCRIPT FIVE]\n        '\\u{2085}' | // ₅  [SUBSCRIPT FIVE]\n        '\\u{2464}' | // ⑤  [CIRCLED DIGIT FIVE]\n        '\\u{24F9}' | // ⓹  [DOUBLE CIRCLED DIGIT FIVE]\n        '\\u{277A}' | // ❺  [DINGBAT NEGATIVE CIRCLED DIGIT FIVE]\n        '\\u{2784}' | // ➄  [DINGBAT CIRCLED SANS-SERIF DIGIT FIVE]\n        '\\u{278E}' | // ➎  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT FIVE]\n        '\\u{FF15}' // ５  [FULLWIDTH DIGIT FIVE]\n        => Some(\"5\"),\n        '\\u{248C}' // ⒌  [DIGIT FIVE FULL STOP]\n        => Some(\"5.\"),\n        '\\u{2478}' // ⑸  [PARENTHESIZED DIGIT FIVE]\n        => Some(\"(5)\"),\n        '\\u{2076}' | // ⁶  [SUPERSCRIPT SIX]\n        '\\u{2086}' | // ₆  [SUBSCRIPT SIX]\n        '\\u{2465}' | // ⑥  [CIRCLED DIGIT SIX]\n        '\\u{24FA}' | // ⓺  [DOUBLE CIRCLED DIGIT SIX]\n        '\\u{277B}' | // ❻  [DINGBAT NEGATIVE CIRCLED DIGIT SIX]\n        '\\u{2785}' | // ➅  [DINGBAT CIRCLED SANS-SERIF DIGIT SIX]\n        '\\u{278F}' | // ➏  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT SIX]\n        '\\u{FF16}' // ６  [FULLWIDTH DIGIT SIX]\n        => Some(\"6\"),\n        '\\u{248D}' // ⒍  [DIGIT SIX FULL STOP]\n        => Some(\"6.\"),\n        '\\u{2479}' // ⑹  [PARENTHESIZED DIGIT SIX]\n        => Some(\"(6)\"),\n        '\\u{2077}' | // ⁷  [SUPERSCRIPT SEVEN]\n        '\\u{2087}' | // ₇  [SUBSCRIPT SEVEN]\n        '\\u{2466}' | // ⑦  [CIRCLED DIGIT SEVEN]\n        '\\u{24FB}' | // ⓻  [DOUBLE CIRCLED DIGIT SEVEN]\n        '\\u{277C}' | // ❼  [DINGBAT NEGATIVE CIRCLED DIGIT SEVEN]\n        '\\u{2786}' | // ➆  [DINGBAT CIRCLED SANS-SERIF DIGIT SEVEN]\n        '\\u{2790}' | // ➐  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT SEVEN]\n        '\\u{FF17}' // ７  [FULLWIDTH DIGIT SEVEN]\n        => Some(\"7\"),\n        '\\u{248E}' // ⒎  [DIGIT SEVEN FULL STOP]\n        => Some(\"7.\"),\n        '\\u{247A}' // ⑺  [PARENTHESIZED DIGIT SEVEN]\n        => Some(\"(7)\"),\n        '\\u{2078}' | // ⁸  [SUPERSCRIPT EIGHT]\n        '\\u{2088}' | // ₈  [SUBSCRIPT EIGHT]\n        '\\u{2467}' | // ⑧  [CIRCLED DIGIT EIGHT]\n        '\\u{24FC}' | // ⓼  [DOUBLE CIRCLED DIGIT EIGHT]\n        '\\u{277D}' | // ❽  [DINGBAT NEGATIVE CIRCLED DIGIT EIGHT]\n        '\\u{2787}' | // ➇  [DINGBAT CIRCLED SANS-SERIF DIGIT EIGHT]\n        '\\u{2791}' | // ➑  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT EIGHT]\n        '\\u{FF18}' // ８  [FULLWIDTH DIGIT EIGHT]\n        => Some(\"8\"),\n        '\\u{248F}' // ⒏  [DIGIT EIGHT FULL STOP]\n        => Some(\"8.\"),\n        '\\u{247B}' // ⑻  [PARENTHESIZED DIGIT EIGHT]\n        => Some(\"(8)\"),\n        '\\u{2079}' | // ⁹  [SUPERSCRIPT NINE]\n        '\\u{2089}' | // ₉  [SUBSCRIPT NINE]\n        '\\u{2468}' | // ⑨  [CIRCLED DIGIT NINE]\n        '\\u{24FD}' | // ⓽  [DOUBLE CIRCLED DIGIT NINE]\n        '\\u{277E}' | // ❾  [DINGBAT NEGATIVE CIRCLED DIGIT NINE]\n        '\\u{2788}' | // ➈  [DINGBAT CIRCLED SANS-SERIF DIGIT NINE]\n        '\\u{2792}' | // ➒  [DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT NINE]\n        '\\u{FF19}' // ９  [FULLWIDTH DIGIT NINE]\n        => Some(\"9\"),\n        '\\u{2490}' // ⒐  [DIGIT NINE FULL STOP]\n        => Some(\"9.\"),\n        '\\u{247C}' // ⑼  [PARENTHESIZED DIGIT NINE]\n        => Some(\"(9)\"),\n        '\\u{2469}' | // ⑩  [CIRCLED NUMBER TEN]\n        '\\u{24FE}' | // ⓾  [DOUBLE CIRCLED NUMBER TEN]\n        '\\u{277F}' | // ❿  [DINGBAT NEGATIVE CIRCLED NUMBER TEN]\n        '\\u{2789}' | // ➉  [DINGBAT CIRCLED SANS-SERIF NUMBER TEN]\n        '\\u{2793}' // ➓  [DINGBAT NEGATIVE CIRCLED SANS-SERIF NUMBER TEN]\n        => Some(\"10\"),\n        '\\u{2491}' // ⒑  [NUMBER TEN FULL STOP]\n        => Some(\"10.\"),\n        '\\u{247D}' // ⑽  [PARENTHESIZED NUMBER TEN]\n        => Some(\"(10)\"),\n        '\\u{246A}' | // ⑪  [CIRCLED NUMBER ELEVEN]\n        '\\u{24EB}' // ⓫  [NEGATIVE CIRCLED NUMBER ELEVEN]\n        => Some(\"11\"),\n        '\\u{2492}' // ⒒  [NUMBER ELEVEN FULL STOP]\n        => Some(\"11.\"),\n        '\\u{247E}' // ⑾  [PARENTHESIZED NUMBER ELEVEN]\n        => Some(\"(11)\"),\n        '\\u{246B}' | // ⑫  [CIRCLED NUMBER TWELVE]\n        '\\u{24EC}' // ⓬  [NEGATIVE CIRCLED NUMBER TWELVE]\n        => Some(\"12\"),\n        '\\u{2493}' // ⒓  [NUMBER TWELVE FULL STOP]\n        => Some(\"12.\"),\n        '\\u{247F}' // ⑿  [PARENTHESIZED NUMBER TWELVE]\n        => Some(\"(12)\"),\n        '\\u{246C}' | // ⑬  [CIRCLED NUMBER THIRTEEN]\n        '\\u{24ED}' // ⓭  [NEGATIVE CIRCLED NUMBER THIRTEEN]\n        => Some(\"13\"),\n        '\\u{2494}' // ⒔  [NUMBER THIRTEEN FULL STOP]\n        => Some(\"13.\"),\n        '\\u{2480}' // ⒀  [PARENTHESIZED NUMBER THIRTEEN]\n        => Some(\"(13)\"),\n        '\\u{246D}' | // ⑭  [CIRCLED NUMBER FOURTEEN]\n        '\\u{24EE}' // ⓮  [NEGATIVE CIRCLED NUMBER FOURTEEN]\n        => Some(\"14\"),\n        '\\u{2495}' // ⒕  [NUMBER FOURTEEN FULL STOP]\n        => Some(\"14.\"),\n        '\\u{2481}' // ⒁  [PARENTHESIZED NUMBER FOURTEEN]\n        => Some(\"(14)\"),\n        '\\u{246E}' | // ⑮  [CIRCLED NUMBER FIFTEEN]\n        '\\u{24EF}' // ⓯  [NEGATIVE CIRCLED NUMBER FIFTEEN]\n        => Some(\"15\"),\n        '\\u{2496}' // ⒖  [NUMBER FIFTEEN FULL STOP]\n        => Some(\"15.\"),\n        '\\u{2482}' // ⒂  [PARENTHESIZED NUMBER FIFTEEN]\n        => Some(\"(15)\"),\n        '\\u{246F}' | // ⑯  [CIRCLED NUMBER SIXTEEN]\n        '\\u{24F0}' // ⓰  [NEGATIVE CIRCLED NUMBER SIXTEEN]\n        => Some(\"16\"),\n        '\\u{2497}' // ⒗  [NUMBER SIXTEEN FULL STOP]\n        => Some(\"16.\"),\n        '\\u{2483}' // ⒃  [PARENTHESIZED NUMBER SIXTEEN]\n        => Some(\"(16)\"),\n        '\\u{2470}' | // ⑰  [CIRCLED NUMBER SEVENTEEN]\n        '\\u{24F1}' // ⓱  [NEGATIVE CIRCLED NUMBER SEVENTEEN]\n        => Some(\"17\"),\n        '\\u{2498}' // ⒘  [NUMBER SEVENTEEN FULL STOP]\n        => Some(\"17.\"),\n        '\\u{2484}' // ⒄  [PARENTHESIZED NUMBER SEVENTEEN]\n        => Some(\"(17)\"),\n        '\\u{2471}' | // ⑱  [CIRCLED NUMBER EIGHTEEN]\n        '\\u{24F2}' // ⓲  [NEGATIVE CIRCLED NUMBER EIGHTEEN]\n        => Some(\"18\"),\n        '\\u{2499}' // ⒙  [NUMBER EIGHTEEN FULL STOP]\n        => Some(\"18.\"),\n        '\\u{2485}' // ⒅  [PARENTHESIZED NUMBER EIGHTEEN]\n        => Some(\"(18)\"),\n        '\\u{2472}' | // ⑲  [CIRCLED NUMBER NINETEEN]\n        '\\u{24F3}' // ⓳  [NEGATIVE CIRCLED NUMBER NINETEEN]\n        => Some(\"19\"),\n        '\\u{249A}' // ⒚  [NUMBER NINETEEN FULL STOP]\n        => Some(\"19.\"),\n        '\\u{2486}' // ⒆  [PARENTHESIZED NUMBER NINETEEN]\n        => Some(\"(19)\"),\n        '\\u{2473}' | // ⑳  [CIRCLED NUMBER TWENTY]\n        '\\u{24F4}' // ⓴  [NEGATIVE CIRCLED NUMBER TWENTY]\n        => Some(\"20\"),\n        '\\u{249B}' // ⒛  [NUMBER TWENTY FULL STOP]\n        => Some(\"20.\"),\n        '\\u{2487}' // ⒇  [PARENTHESIZED NUMBER TWENTY]\n        => Some(\"(20)\"),\n        '\\u{00AB}' | // «  [LEFT-POINTING DOUBLE ANGLE QUOTATION MARK]\n        '\\u{00BB}' | // »  [RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK]\n        '\\u{201C}' | // “  [LEFT DOUBLE QUOTATION MARK]\n        '\\u{201D}' | // ”  [RIGHT DOUBLE QUOTATION MARK]\n        '\\u{201E}' | // „  [DOUBLE LOW-9 QUOTATION MARK]\n        '\\u{2033}' | // ″  [DOUBLE PRIME]\n        '\\u{2036}' | // ‶  [REVERSED DOUBLE PRIME]\n        '\\u{275D}' | // ❝  [HEAVY DOUBLE TURNED COMMA QUOTATION MARK ORNAMENT]\n        '\\u{275E}' | // ❞  [HEAVY DOUBLE COMMA QUOTATION MARK ORNAMENT]\n        '\\u{276E}' | // ❮  [HEAVY LEFT-POINTING ANGLE QUOTATION MARK ORNAMENT]\n        '\\u{276F}' | // ❯  [HEAVY RIGHT-POINTING ANGLE QUOTATION MARK ORNAMENT]\n        '\\u{FF02}' // ＂  [FULLWIDTH QUOTATION MARK]\n        => Some(\"\\\"\"),\n        '\\u{2018}' | // ‘  [LEFT SINGLE QUOTATION MARK]\n        '\\u{2019}' | // ’  [RIGHT SINGLE QUOTATION MARK]\n        '\\u{201A}' | // ‚  [SINGLE LOW-9 QUOTATION MARK]\n        '\\u{201B}' | // ‛  [SINGLE HIGH-REVERSED-9 QUOTATION MARK]\n        '\\u{2032}' | // ′  [PRIME]\n        '\\u{2035}' | // ‵  [REVERSED PRIME]\n        '\\u{2039}' | // ‹  [SINGLE LEFT-POINTING ANGLE QUOTATION MARK]\n        '\\u{203A}' | // ›  [SINGLE RIGHT-POINTING ANGLE QUOTATION MARK]\n        '\\u{275B}' | // ❛  [HEAVY SINGLE TURNED COMMA QUOTATION MARK ORNAMENT]\n        '\\u{275C}' | // ❜  [HEAVY SINGLE COMMA QUOTATION MARK ORNAMENT]\n        '\\u{FF07}' // ＇  [FULLWIDTH APOSTROPHE]\n        => Some(\"\\'\"),\n        '\\u{2010}' | // ‐  [HYPHEN]\n        '\\u{2011}' | // ‑  [NON-BREAKING HYPHEN]\n        '\\u{2012}' | // ‒  [FIGURE DASH]\n        '\\u{2013}' | // –  [EN DASH]\n        '\\u{2014}' | // —  [EM DASH]\n        '\\u{207B}' | // ⁻  [SUPERSCRIPT MINUS]\n        '\\u{208B}' | // ₋  [SUBSCRIPT MINUS]\n        '\\u{FF0D}' // －  [FULLWIDTH HYPHEN-MINUS]\n        => Some(\"-\"),\n        '\\u{2045}' | // ⁅  [LEFT SQUARE BRACKET WITH QUILL]\n        '\\u{2772}' | // ❲  [LIGHT LEFT TORTOISE SHELL BRACKET ORNAMENT]\n        '\\u{FF3B}' // ［  [FULLWIDTH LEFT SQUARE BRACKET]\n        => Some(\"[\"),\n        '\\u{2046}' | // ⁆  [RIGHT SQUARE BRACKET WITH QUILL]\n        '\\u{2773}' | // ❳  [LIGHT RIGHT TORTOISE SHELL BRACKET ORNAMENT]\n        '\\u{FF3D}' // ］  [FULLWIDTH RIGHT SQUARE BRACKET]\n        => Some(\"]\"),\n        '\\u{207D}' | // ⁽  [SUPERSCRIPT LEFT PARENTHESIS]\n        '\\u{208D}' | // ₍  [SUBSCRIPT LEFT PARENTHESIS]\n        '\\u{2768}' | // ❨  [MEDIUM LEFT PARENTHESIS ORNAMENT]\n        '\\u{276A}' | // ❪  [MEDIUM FLATTENED LEFT PARENTHESIS ORNAMENT]\n        '\\u{FF08}' // （  [FULLWIDTH LEFT PARENTHESIS]\n        => Some(\"(\"),\n        '\\u{2E28}' // ⸨  [LEFT DOUBLE PARENTHESIS]\n        => Some(\"((\"),\n        '\\u{207E}' | // ⁾  [SUPERSCRIPT RIGHT PARENTHESIS]\n        '\\u{208E}' | // ₎  [SUBSCRIPT RIGHT PARENTHESIS]\n        '\\u{2769}' | // ❩  [MEDIUM RIGHT PARENTHESIS ORNAMENT]\n        '\\u{276B}' | // ❫  [MEDIUM FLATTENED RIGHT PARENTHESIS ORNAMENT]\n        '\\u{FF09}' // ）  [FULLWIDTH RIGHT PARENTHESIS]\n        => Some(\")\"),\n        '\\u{2E29}' // ⸩  [RIGHT DOUBLE PARENTHESIS]\n        => Some(\"))\"),\n        '\\u{276C}' | // ❬  [MEDIUM LEFT-POINTING ANGLE BRACKET ORNAMENT]\n        '\\u{2770}' | // ❰  [HEAVY LEFT-POINTING ANGLE BRACKET ORNAMENT]\n        '\\u{FF1C}' // ＜  [FULLWIDTH LESS-THAN SIGN]\n        => Some(\"<\"),\n        '\\u{276D}' | // ❭  [MEDIUM RIGHT-POINTING ANGLE BRACKET ORNAMENT]\n        '\\u{2771}' | // ❱  [HEAVY RIGHT-POINTING ANGLE BRACKET ORNAMENT]\n        '\\u{FF1E}' // ＞  [FULLWIDTH GREATER-THAN SIGN]\n        => Some(\">\"),\n        '\\u{2774}' | // ❴  [MEDIUM LEFT CURLY BRACKET ORNAMENT]\n        '\\u{FF5B}' // ｛  [FULLWIDTH LEFT CURLY BRACKET]\n        => Some(\"{\"),\n        '\\u{2775}' | // ❵  [MEDIUM RIGHT CURLY BRACKET ORNAMENT]\n        '\\u{FF5D}' // ｝  [FULLWIDTH RIGHT CURLY BRACKET]\n        => Some(\"}\"),\n        '\\u{207A}' | // ⁺  [SUPERSCRIPT PLUS SIGN]\n        '\\u{208A}' | // ₊  [SUBSCRIPT PLUS SIGN]\n        '\\u{FF0B}' // ＋  [FULLWIDTH PLUS SIGN]\n        => Some(\"+\"),\n        '\\u{207C}' | // ⁼  [SUPERSCRIPT EQUALS SIGN]\n        '\\u{208C}' | // ₌  [SUBSCRIPT EQUALS SIGN]\n        '\\u{FF1D}' // ＝  [FULLWIDTH EQUALS SIGN]\n        => Some(\"=\"),\n        '\\u{FF01}' // ！  [FULLWIDTH EXCLAMATION MARK]\n        => Some(\"!\"),\n        '\\u{203C}' // ‼  [DOUBLE EXCLAMATION MARK]\n        => Some(\"!!\"),\n        '\\u{2049}' // ⁉  [EXCLAMATION QUESTION MARK]\n        => Some(\"!?\"),\n        '\\u{FF03}' // ＃  [FULLWIDTH NUMBER SIGN]\n        => Some(\"#\"),\n        '\\u{FF04}' // ＄  [FULLWIDTH DOLLAR SIGN]\n        => Some(\"$\"),\n        '\\u{2052}' | // ⁒  [COMMERCIAL MINUS SIGN]\n        '\\u{FF05}' // ％  [FULLWIDTH PERCENT SIGN]\n        => Some(\"%\"),\n        '\\u{FF06}' // ＆  [FULLWIDTH AMPERSAND]\n        => Some(\"&\"),\n        '\\u{204E}' | // ⁎  [LOW ASTERISK]\n        '\\u{FF0A}' // ＊  [FULLWIDTH ASTERISK]\n        => Some(\"*\"),\n        '\\u{FF0C}' // ，  [FULLWIDTH COMMA]\n        => Some(\",\"),\n        '\\u{FF0E}' // ．  [FULLWIDTH FULL STOP]\n        => Some(\".\"),\n        '\\u{2044}' | // ⁄  [FRACTION SLASH]\n        '\\u{FF0F}' // ／  [FULLWIDTH SOLIDUS]\n        => Some(\"/\"),\n        '\\u{FF1A}' // ：  [FULLWIDTH COLON]\n        => Some(\":\"),\n        '\\u{204F}' | // ⁏  [REVERSED SEMICOLON]\n        '\\u{FF1B}' // ；  [FULLWIDTH SEMICOLON]\n        => Some(\";\"),\n        '\\u{FF1F}' // ？  [FULLWIDTH QUESTION MARK]\n        => Some(\"?\"),\n        '\\u{2047}' // ⁇  [DOUBLE QUESTION MARK]\n        => Some(\"??\"),\n        '\\u{2048}' // ⁈  [QUESTION EXCLAMATION MARK]\n        => Some(\"?!\"),\n        '\\u{FF20}' // ＠  [FULLWIDTH COMMERCIAL AT]\n        => Some(\"@\"),\n        '\\u{FF3C}' // ＼  [FULLWIDTH REVERSE SOLIDUS]\n        => Some(\"\\\\\"),\n        '\\u{2038}' | // ‸  [CARET]\n        '\\u{FF3E}' // ＾  [FULLWIDTH CIRCUMFLEX ACCENT]\n        => Some(\"^\"),\n        '\\u{FF3F}' // ＿  [FULLWIDTH LOW LINE]\n        => Some(\"_\"),\n        '\\u{2053}' | // ⁓  [SWUNG DASH]\n        '\\u{FF5E}' // ～  [FULLWIDTH TILDE]\n        => Some(\"~\"),\n        _ => None\n    }\n}\n\n// https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java#L187\nfn to_ascii(text: &str, output: &mut String) {\n    output.clear();\n\n    for c in text.chars() {\n        if let Some(folded) = fold_non_ascii_char(c) {\n            output.push_str(folded);\n        } else {\n            output.push(c);\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::to_ascii;\n    use crate::tokenizer::{AsciiFoldingFilter, RawTokenizer, SimpleTokenizer, TextAnalyzer};\n\n    #[test]\n    fn test_ascii_folding() {\n        assert_eq!(&folding_helper(\"Ràmon\"), &[\"Ramon\"]);\n        assert_eq!(&folding_helper(\"accentué\"), &[\"accentue\"]);\n        assert_eq!(&folding_helper(\"âäàéè\"), &[\"aaaee\"]);\n    }\n\n    #[test]\n    fn test_no_change() {\n        assert_eq!(&folding_helper(\"Usagi\"), &[\"Usagi\"]);\n    }\n\n    fn folding_helper(text: &str) -> Vec<String> {\n        let mut tokens = Vec::new();\n        TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(AsciiFoldingFilter)\n            .build()\n            .token_stream(text)\n            .process(&mut |token| {\n                tokens.push(token.text.clone());\n            });\n        tokens\n    }\n\n    fn folding_using_raw_tokenizer_helper(text: &str) -> String {\n        let mut tokenizer = TextAnalyzer::builder(RawTokenizer::default())\n            .filter(AsciiFoldingFilter)\n            .build();\n        let mut token_stream = tokenizer.token_stream(text);\n        token_stream.advance();\n        token_stream.token().text.clone()\n    }\n\n    #[test]\n    fn test_latin1_characters() {\n        let latin1_string = \"Des mot clés À LA CHAÎNE À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï IJ Ð Ñ\n                   Ò Ó Ô Õ Ö Ø Œ Þ Ù Ú Û Ü Ý Ÿ à á â ã ä å æ ç è é ê ë ì í î ï ij\n                   ð ñ ò ó ô õ ö ø œ ß þ ù ú û ü ý ÿ fi fl\";\n        let mut vec: Vec<&str> = vec![\"Des\", \"mot\", \"cles\", \"A\", \"LA\", \"CHAINE\"];\n        vec.extend(std::iter::repeat_n(\"A\", 6));\n        vec.extend(std::iter::repeat_n(\"AE\", 1));\n        vec.extend(std::iter::repeat_n(\"C\", 1));\n        vec.extend(std::iter::repeat_n(\"E\", 4));\n        vec.extend(std::iter::repeat_n(\"I\", 4));\n        vec.extend(std::iter::repeat_n(\"IJ\", 1));\n        vec.extend(std::iter::repeat_n(\"D\", 1));\n        vec.extend(std::iter::repeat_n(\"N\", 1));\n        vec.extend(std::iter::repeat_n(\"O\", 6));\n        vec.extend(std::iter::repeat_n(\"OE\", 1));\n        vec.extend(std::iter::repeat_n(\"TH\", 1));\n        vec.extend(std::iter::repeat_n(\"U\", 4));\n        vec.extend(std::iter::repeat_n(\"Y\", 2));\n        vec.extend(std::iter::repeat_n(\"a\", 6));\n        vec.extend(std::iter::repeat_n(\"ae\", 1));\n        vec.extend(std::iter::repeat_n(\"c\", 1));\n        vec.extend(std::iter::repeat_n(\"e\", 4));\n        vec.extend(std::iter::repeat_n(\"i\", 4));\n        vec.extend(std::iter::repeat_n(\"ij\", 1));\n        vec.extend(std::iter::repeat_n(\"d\", 1));\n        vec.extend(std::iter::repeat_n(\"n\", 1));\n        vec.extend(std::iter::repeat_n(\"o\", 6));\n        vec.extend(std::iter::repeat_n(\"oe\", 1));\n        vec.extend(std::iter::repeat_n(\"ss\", 1));\n        vec.extend(std::iter::repeat_n(\"th\", 1));\n        vec.extend(std::iter::repeat_n(\"u\", 4));\n        vec.extend(std::iter::repeat_n(\"y\", 2));\n        vec.extend(std::iter::repeat_n(\"fi\", 1));\n        vec.extend(std::iter::repeat_n(\"fl\", 1));\n        assert_eq!(folding_helper(latin1_string), vec);\n    }\n\n    #[test]\n    fn test_unmodified_letters() {\n        assert_eq!(\n            folding_using_raw_tokenizer_helper(\"§ ¦ ¤ END\"),\n            \"§ ¦ ¤ END\".to_string()\n        );\n    }\n\n    #[test]\n    fn test_to_ascii() {\n        let input = \"Rámon\".to_string();\n        let mut buffer = String::new();\n        to_ascii(&input, &mut buffer);\n        assert_eq!(\"Ramon\", buffer);\n    }\n\n    #[test]\n    fn test_all_foldings() {\n        // those folding is a copy of\n        // https://github.com/apache/lucene-solr/blob/28d187acd1e391723eb6e1b5445f22abf5580a80/lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestASCIIFoldingFilter.java\n        // useful regex to adapt to a Rust structure:\n        // 1. Preg and replace folded:\n        //    - **REGEX** |,\"(.){3,5}\", // Folded result|\n        //    - **REPLACEMENT** ], \"$1\".to_string(), ), ( vec![\n        // 2. Preg and replace characters:\n        //    - **REGEX** |[\\+]{0,1} \"(.{1,3})\"  // U\\+|\n        //    - **REPLACEMENT** \"$1\",  // U+\n        let foldings: Vec<(&[&str], &str)> = vec![\n            (\n                &[\n                    \"À\",  // U+00C0: LATIN CAPITAL LETTER A WITH GRAVE\n                    \"Á\",  // U+00C1: LATIN CAPITAL LETTER A WITH ACUTE\n                    \"Â\",  // U+00C2: LATIN CAPITAL LETTER A WITH CIRCUMFLEX\n                    \"Ã\",  // U+00C3: LATIN CAPITAL LETTER A WITH TILDE\n                    \"Ä\",  // U+00C4: LATIN CAPITAL LETTER A WITH DIAERESIS\n                    \"Å\",  // U+00C5: LATIN CAPITAL LETTER A WITH RING ABOVE\n                    \"Ā\",  // U+0100: LATIN CAPITAL LETTER A WITH MACRON\n                    \"Ă\",  // U+0102: LATIN CAPITAL LETTER A WITH BREVE\n                    \"Ą\",  // U+0104: LATIN CAPITAL LETTER A WITH OGONEK\n                    \"Ə\",  // U+018F: LATIN CAPITAL LETTER SCHWA\n                    \"Ǎ\",  // U+01CD: LATIN CAPITAL LETTER A WITH CARON\n                    \"Ǟ\",  // U+01DE: LATIN CAPITAL LETTER A WITH DIAERESIS AND MACRON\n                    \"Ǡ\",  // U+01E0: LATIN CAPITAL LETTER A WITH DOT ABOVE AND MACRON\n                    \"Ǻ\",  // U+01FA: LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE\n                    \"Ȁ\",  // U+0200: LATIN CAPITAL LETTER A WITH DOUBLE GRAVE\n                    \"Ȃ\",  // U+0202: LATIN CAPITAL LETTER A WITH INVERTED BREVE\n                    \"Ȧ\",  // U+0226: LATIN CAPITAL LETTER A WITH DOT ABOVE\n                    \"Ⱥ\",  // U+023A: LATIN CAPITAL LETTER A WITH STROKE\n                    \"ᴀ\",  // U+1D00: LATIN LETTER SMALL CAPITAL A\n                    \"Ḁ\",  // U+1E00: LATIN CAPITAL LETTER A WITH RING BELOW\n                    \"Ạ\",  // U+1EA0: LATIN CAPITAL LETTER A WITH DOT BELOW\n                    \"Ả\",  // U+1EA2: LATIN CAPITAL LETTER A WITH HOOK ABOVE\n                    \"Ấ\",  // U+1EA4: LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND ACUTE\n                    \"Ầ\",  // U+1EA6: LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND GRAVE\n                    \"Ẩ\",  // U+1EA8: LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"Ẫ\",  // U+1EAA: LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND TILDE\n                    \"Ậ\",  // U+1EAC: LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND DOT BELOW\n                    \"Ắ\",  // U+1EAE: LATIN CAPITAL LETTER A WITH BREVE AND ACUTE\n                    \"Ằ\",  // U+1EB0: LATIN CAPITAL LETTER A WITH BREVE AND GRAVE\n                    \"Ẳ\",  // U+1EB2: LATIN CAPITAL LETTER A WITH BREVE AND HOOK ABOVE\n                    \"Ẵ\",  // U+1EB4: LATIN CAPITAL LETTER A WITH BREVE AND TILDE\n                    \"Ặ\",  // U+1EB6: LATIN CAPITAL LETTER A WITH BREVE AND DOT BELOW\n                    \"Ⓐ\",  // U+24B6: CIRCLED LATIN CAPITAL LETTER A\n                    \"Ａ\", // U+FF21: FULLWIDTH LATIN CAPITAL LETTER A\n                ],\n                \"A\",\n            ),\n            (\n                &[\n                    \"à\",  // U+00E0: LATIN SMALL LETTER A WITH GRAVE\n                    \"á\",  // U+00E1: LATIN SMALL LETTER A WITH ACUTE\n                    \"â\",  // U+00E2: LATIN SMALL LETTER A WITH CIRCUMFLEX\n                    \"ã\",  // U+00E3: LATIN SMALL LETTER A WITH TILDE\n                    \"ä\",  // U+00E4: LATIN SMALL LETTER A WITH DIAERESIS\n                    \"å\",  // U+00E5: LATIN SMALL LETTER A WITH RING ABOVE\n                    \"ā\",  // U+0101: LATIN SMALL LETTER A WITH MACRON\n                    \"ă\",  // U+0103: LATIN SMALL LETTER A WITH BREVE\n                    \"ą\",  // U+0105: LATIN SMALL LETTER A WITH OGONEK\n                    \"ǎ\",  // U+01CE: LATIN SMALL LETTER A WITH CARON\n                    \"ǟ\",  // U+01DF: LATIN SMALL LETTER A WITH DIAERESIS AND MACRON\n                    \"ǡ\",  // U+01E1: LATIN SMALL LETTER A WITH DOT ABOVE AND MACRON\n                    \"ǻ\",  // U+01FB: LATIN SMALL LETTER A WITH RING ABOVE AND ACUTE\n                    \"ȁ\",  // U+0201: LATIN SMALL LETTER A WITH DOUBLE GRAVE\n                    \"ȃ\",  // U+0203: LATIN SMALL LETTER A WITH INVERTED BREVE\n                    \"ȧ\",  // U+0227: LATIN SMALL LETTER A WITH DOT ABOVE\n                    \"ɐ\",  // U+0250: LATIN SMALL LETTER TURNED A\n                    \"ə\",  // U+0259: LATIN SMALL LETTER SCHWA\n                    \"ɚ\",  // U+025A: LATIN SMALL LETTER SCHWA WITH HOOK\n                    \"ᶏ\",  // U+1D8F: LATIN SMALL LETTER A WITH RETROFLEX HOOK\n                    \"ḁ\",  // U+1E01: LATIN SMALL LETTER A WITH RING BELOW\n                    \"ᶕ\",  // U+1D95: LATIN SMALL LETTER SCHWA WITH RETROFLEX HOOK\n                    \"ẚ\",  // U+1E9A: LATIN SMALL LETTER A WITH RIGHT HALF RING\n                    \"ạ\",  // U+1EA1: LATIN SMALL LETTER A WITH DOT BELOW\n                    \"ả\",  // U+1EA3: LATIN SMALL LETTER A WITH HOOK ABOVE\n                    \"ấ\",  // U+1EA5: LATIN SMALL LETTER A WITH CIRCUMFLEX AND ACUTE\n                    \"ầ\",  // U+1EA7: LATIN SMALL LETTER A WITH CIRCUMFLEX AND GRAVE\n                    \"ẩ\",  // U+1EA9: LATIN SMALL LETTER A WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"ẫ\",  // U+1EAB: LATIN SMALL LETTER A WITH CIRCUMFLEX AND TILDE\n                    \"ậ\",  // U+1EAD: LATIN SMALL LETTER A WITH CIRCUMFLEX AND DOT BELOW\n                    \"ắ\",  // U+1EAF: LATIN SMALL LETTER A WITH BREVE AND ACUTE\n                    \"ằ\",  // U+1EB1: LATIN SMALL LETTER A WITH BREVE AND GRAVE\n                    \"ẳ\",  // U+1EB3: LATIN SMALL LETTER A WITH BREVE AND HOOK ABOVE\n                    \"ẵ\",  // U+1EB5: LATIN SMALL LETTER A WITH BREVE AND TILDE\n                    \"ặ\",  // U+1EB7: LATIN SMALL LETTER A WITH BREVE AND DOT BELOW\n                    \"ₐ\",  // U+2090: LATIN SUBSCRIPT SMALL LETTER A\n                    \"ₔ\",  // U+2094: LATIN SUBSCRIPT SMALL LETTER SCHWA\n                    \"ⓐ\",  // U+24D0: CIRCLED LATIN SMALL LETTER A\n                    \"ⱥ\",  // U+2C65: LATIN SMALL LETTER A WITH STROKE\n                    \"Ɐ\",  // U+2C6F: LATIN CAPITAL LETTER TURNED A\n                    \"ａ\", // U+FF41: FULLWIDTH LATIN SMALL LETTER A\n                ],\n                \"a\",\n            ),\n            (\n                &[\n                    \"Ꜳ\", // U+A732: LATIN CAPITAL LETTER AA\n                ],\n                \"AA\",\n            ),\n            (\n                &[\n                    \"Æ\", // U+00C6: LATIN CAPITAL LETTER AE\n                    \"Ǣ\", // U+01E2: LATIN CAPITAL LETTER AE WITH MACRON\n                    \"Ǽ\", // U+01FC: LATIN CAPITAL LETTER AE WITH ACUTE\n                    \"ᴁ\", // U+1D01: LATIN LETTER SMALL CAPITAL AE\n                ],\n                \"AE\",\n            ),\n            (\n                &[\n                    \"Ꜵ\", // U+A734: LATIN CAPITAL LETTER AO\n                ],\n                \"AO\",\n            ),\n            (\n                &[\n                    \"Ꜷ\", // U+A736: LATIN CAPITAL LETTER AU\n                ],\n                \"AU\",\n            ),\n            (\n                &[\n                    \"Ꜹ\", // U+A738: LATIN CAPITAL LETTER AV\n                    \"Ꜻ\", // U+A73A: LATIN CAPITAL LETTER AV WITH HORIZONTAL BAR\n                ],\n                \"AV\",\n            ),\n            (\n                &[\n                    \"Ꜽ\", // U+A73C: LATIN CAPITAL LETTER AY\n                ],\n                \"AY\",\n            ),\n            (\n                &[\n                    \"⒜\", // U+249C: PARENTHESIZED LATIN SMALL LETTER A\n                ],\n                \"(a)\",\n            ),\n            (\n                &[\n                    \"ꜳ\", // U+A733: LATIN SMALL LETTER AA\n                ],\n                \"aa\",\n            ),\n            (\n                &[\n                    \"æ\", // U+00E6: LATIN SMALL LETTER AE\n                    \"ǣ\", // U+01E3: LATIN SMALL LETTER AE WITH MACRON\n                    \"ǽ\", // U+01FD: LATIN SMALL LETTER AE WITH ACUTE\n                    \"ᴂ\", // U+1D02: LATIN SMALL LETTER TURNED AE\n                ],\n                \"ae\",\n            ),\n            (\n                &[\n                    \"ꜵ\", // U+A735: LATIN SMALL LETTER AO\n                ],\n                \"ao\",\n            ),\n            (\n                &[\n                    \"ꜷ\", // U+A737: LATIN SMALL LETTER AU\n                ],\n                \"au\",\n            ),\n            (\n                &[\n                    \"ꜹ\", // U+A739: LATIN SMALL LETTER AV\n                    \"ꜻ\", // U+A73B: LATIN SMALL LETTER AV WITH HORIZONTAL BAR\n                ],\n                \"av\",\n            ),\n            (\n                &[\n                    \"ꜽ\", // U+A73D: LATIN SMALL LETTER AY\n                ],\n                \"ay\",\n            ),\n            (\n                &[\n                    \"Ɓ\",  // U+0181: LATIN CAPITAL LETTER B WITH HOOK\n                    \"Ƃ\",  // U+0182: LATIN CAPITAL LETTER B WITH TOPBAR\n                    \"Ƀ\",  // U+0243: LATIN CAPITAL LETTER B WITH STROKE\n                    \"ʙ\",  // U+0299: LATIN LETTER SMALL CAPITAL B\n                    \"ᴃ\",  // U+1D03: LATIN LETTER SMALL CAPITAL BARRED B\n                    \"Ḃ\",  // U+1E02: LATIN CAPITAL LETTER B WITH DOT ABOVE\n                    \"Ḅ\",  // U+1E04: LATIN CAPITAL LETTER B WITH DOT BELOW\n                    \"Ḇ\",  // U+1E06: LATIN CAPITAL LETTER B WITH LINE BELOW\n                    \"Ⓑ\",  // U+24B7: CIRCLED LATIN CAPITAL LETTER B\n                    \"Ｂ\", // U+FF22: FULLWIDTH LATIN CAPITAL LETTER B\n                ],\n                \"B\",\n            ),\n            (\n                &[\n                    \"ƀ\",  // U+0180: LATIN SMALL LETTER B WITH STROKE\n                    \"ƃ\",  // U+0183: LATIN SMALL LETTER B WITH TOPBAR\n                    \"ɓ\",  // U+0253: LATIN SMALL LETTER B WITH HOOK\n                    \"ᵬ\",  // U+1D6C: LATIN SMALL LETTER B WITH MIDDLE TILDE\n                    \"ᶀ\",  // U+1D80: LATIN SMALL LETTER B WITH PALATAL HOOK\n                    \"ḃ\",  // U+1E03: LATIN SMALL LETTER B WITH DOT ABOVE\n                    \"ḅ\",  // U+1E05: LATIN SMALL LETTER B WITH DOT BELOW\n                    \"ḇ\",  // U+1E07: LATIN SMALL LETTER B WITH LINE BELOW\n                    \"ⓑ\",  // U+24D1: CIRCLED LATIN SMALL LETTER B\n                    \"ｂ\", // U+FF42: FULLWIDTH LATIN SMALL LETTER B\n                ],\n                \"b\",\n            ),\n            (\n                &[\n                    \"⒝\", // U+249D: PARENTHESIZED LATIN SMALL LETTER B\n                ],\n                \"(b)\",\n            ),\n            (\n                &[\n                    \"Ç\",  // U+00C7: LATIN CAPITAL LETTER C WITH CEDILLA\n                    \"Ć\",  // U+0106: LATIN CAPITAL LETTER C WITH ACUTE\n                    \"Ĉ\",  // U+0108: LATIN CAPITAL LETTER C WITH CIRCUMFLEX\n                    \"Ċ\",  // U+010A: LATIN CAPITAL LETTER C WITH DOT ABOVE\n                    \"Č\",  // U+010C: LATIN CAPITAL LETTER C WITH CARON\n                    \"Ƈ\",  // U+0187: LATIN CAPITAL LETTER C WITH HOOK\n                    \"Ȼ\",  // U+023B: LATIN CAPITAL LETTER C WITH STROKE\n                    \"ʗ\",  // U+0297: LATIN LETTER STRETCHED C\n                    \"ᴄ\",  // U+1D04: LATIN LETTER SMALL CAPITAL C\n                    \"Ḉ\",  // U+1E08: LATIN CAPITAL LETTER C WITH CEDILLA AND ACUTE\n                    \"Ⓒ\",  // U+24B8: CIRCLED LATIN CAPITAL LETTER C\n                    \"Ｃ\", // U+FF23: FULLWIDTH LATIN CAPITAL LETTER C\n                ],\n                \"C\",\n            ),\n            (\n                &[\n                    \"ç\",  // U+00E7: LATIN SMALL LETTER C WITH CEDILLA\n                    \"ć\",  // U+0107: LATIN SMALL LETTER C WITH ACUTE\n                    \"ĉ\",  // U+0109: LATIN SMALL LETTER C WITH CIRCUMFLEX\n                    \"ċ\",  // U+010B: LATIN SMALL LETTER C WITH DOT ABOVE\n                    \"č\",  // U+010D: LATIN SMALL LETTER C WITH CARON\n                    \"ƈ\",  // U+0188: LATIN SMALL LETTER C WITH HOOK\n                    \"ȼ\",  // U+023C: LATIN SMALL LETTER C WITH STROKE\n                    \"ɕ\",  // U+0255: LATIN SMALL LETTER C WITH CURL\n                    \"ḉ\",  // U+1E09: LATIN SMALL LETTER C WITH CEDILLA AND ACUTE\n                    \"ↄ\",  // U+2184: LATIN SMALL LETTER REVERSED C\n                    \"ⓒ\",  // U+24D2: CIRCLED LATIN SMALL LETTER C\n                    \"Ꜿ\",  // U+A73E: LATIN CAPITAL LETTER REVERSED C WITH DOT\n                    \"ꜿ\",  // U+A73F: LATIN SMALL LETTER REVERSED C WITH DOT\n                    \"ｃ\", // U+FF43: FULLWIDTH LATIN SMALL LETTER C\n                ],\n                \"c\",\n            ),\n            (\n                &[\n                    \"⒞\", // U+249E: PARENTHESIZED LATIN SMALL LETTER C\n                ],\n                \"(c)\",\n            ),\n            (\n                &[\n                    \"Ð\",  // U+00D0: LATIN CAPITAL LETTER ETH\n                    \"Ď\",  // U+010E: LATIN CAPITAL LETTER D WITH CARON\n                    \"Đ\",  // U+0110: LATIN CAPITAL LETTER D WITH STROKE\n                    \"Ɖ\",  // U+0189: LATIN CAPITAL LETTER AFRICAN D\n                    \"Ɗ\",  // U+018A: LATIN CAPITAL LETTER D WITH HOOK\n                    \"Ƌ\",  // U+018B: LATIN CAPITAL LETTER D WITH TOPBAR\n                    \"ᴅ\",  // U+1D05: LATIN LETTER SMALL CAPITAL D\n                    \"ᴆ\",  // U+1D06: LATIN LETTER SMALL CAPITAL ETH\n                    \"Ḋ\",  // U+1E0A: LATIN CAPITAL LETTER D WITH DOT ABOVE\n                    \"Ḍ\",  // U+1E0C: LATIN CAPITAL LETTER D WITH DOT BELOW\n                    \"Ḏ\",  // U+1E0E: LATIN CAPITAL LETTER D WITH LINE BELOW\n                    \"Ḑ\",  // U+1E10: LATIN CAPITAL LETTER D WITH CEDILLA\n                    \"Ḓ\",  // U+1E12: LATIN CAPITAL LETTER D WITH CIRCUMFLEX BELOW\n                    \"Ⓓ\",  // U+24B9: CIRCLED LATIN CAPITAL LETTER D\n                    \"Ꝺ\",  // U+A779: LATIN CAPITAL LETTER INSULAR D\n                    \"Ｄ\", // U+FF24: FULLWIDTH LATIN CAPITAL LETTER D\n                ],\n                \"D\",\n            ),\n            (\n                &[\n                    \"ð\",  // U+00F0: LATIN SMALL LETTER ETH\n                    \"ď\",  // U+010F: LATIN SMALL LETTER D WITH CARON\n                    \"đ\",  // U+0111: LATIN SMALL LETTER D WITH STROKE\n                    \"ƌ\",  // U+018C: LATIN SMALL LETTER D WITH TOPBAR\n                    \"ȡ\",  // U+0221: LATIN SMALL LETTER D WITH CURL\n                    \"ɖ\",  // U+0256: LATIN SMALL LETTER D WITH TAIL\n                    \"ɗ\",  // U+0257: LATIN SMALL LETTER D WITH HOOK\n                    \"ᵭ\",  // U+1D6D: LATIN SMALL LETTER D WITH MIDDLE TILDE\n                    \"ᶁ\",  // U+1D81: LATIN SMALL LETTER D WITH PALATAL HOOK\n                    \"ᶑ\",  // U+1D91: LATIN SMALL LETTER D WITH HOOK AND TAIL\n                    \"ḋ\",  // U+1E0B: LATIN SMALL LETTER D WITH DOT ABOVE\n                    \"ḍ\",  // U+1E0D: LATIN SMALL LETTER D WITH DOT BELOW\n                    \"ḏ\",  // U+1E0F: LATIN SMALL LETTER D WITH LINE BELOW\n                    \"ḑ\",  // U+1E11: LATIN SMALL LETTER D WITH CEDILLA\n                    \"ḓ\",  // U+1E13: LATIN SMALL LETTER D WITH CIRCUMFLEX BELOW\n                    \"ⓓ\",  // U+24D3: CIRCLED LATIN SMALL LETTER D\n                    \"ꝺ\",  // U+A77A: LATIN SMALL LETTER INSULAR D\n                    \"ｄ\", // U+FF44: FULLWIDTH LATIN SMALL LETTER D\n                ],\n                \"d\",\n            ),\n            (\n                &[\n                    \"DŽ\", // U+01C4: LATIN CAPITAL LETTER DZ WITH CARON\n                    \"DZ\", // U+01F1: LATIN CAPITAL LETTER DZ\n                ],\n                \"DZ\",\n            ),\n            (\n                &[\n                    \"Dž\", // U+01C5: LATIN CAPITAL LETTER D WITH SMALL LETTER Z WITH CARON\n                    \"Dz\", // U+01F2: LATIN CAPITAL LETTER D WITH SMALL LETTER Z\n                ],\n                \"Dz\",\n            ),\n            (\n                &[\n                    \"⒟\", // U+249F: PARENTHESIZED LATIN SMALL LETTER D\n                ],\n                \"(d)\",\n            ),\n            (\n                &[\n                    \"ȸ\", // U+0238: LATIN SMALL LETTER DB DIGRAPH\n                ],\n                \"db\",\n            ),\n            (\n                &[\n                    \"dž\", // U+01C6: LATIN SMALL LETTER DZ WITH CARON\n                    \"dz\", // U+01F3: LATIN SMALL LETTER DZ\n                    \"ʣ\", // U+02A3: LATIN SMALL LETTER DZ DIGRAPH\n                    \"ʥ\", // U+02A5: LATIN SMALL LETTER DZ DIGRAPH WITH CURL\n                ],\n                \"dz\",\n            ),\n            (\n                &[\n                    \"È\",  // U+00C8: LATIN CAPITAL LETTER E WITH GRAVE\n                    \"É\",  // U+00C9: LATIN CAPITAL LETTER E WITH ACUTE\n                    \"Ê\",  // U+00CA: LATIN CAPITAL LETTER E WITH CIRCUMFLEX\n                    \"Ë\",  // U+00CB: LATIN CAPITAL LETTER E WITH DIAERESIS\n                    \"Ē\",  // U+0112: LATIN CAPITAL LETTER E WITH MACRON\n                    \"Ĕ\",  // U+0114: LATIN CAPITAL LETTER E WITH BREVE\n                    \"Ė\",  // U+0116: LATIN CAPITAL LETTER E WITH DOT ABOVE\n                    \"Ę\",  // U+0118: LATIN CAPITAL LETTER E WITH OGONEK\n                    \"Ě\",  // U+011A: LATIN CAPITAL LETTER E WITH CARON\n                    \"Ǝ\",  // U+018E: LATIN CAPITAL LETTER REVERSED E\n                    \"Ɛ\",  // U+0190: LATIN CAPITAL LETTER OPEN E\n                    \"Ȅ\",  // U+0204: LATIN CAPITAL LETTER E WITH DOUBLE GRAVE\n                    \"Ȇ\",  // U+0206: LATIN CAPITAL LETTER E WITH INVERTED BREVE\n                    \"Ȩ\",  // U+0228: LATIN CAPITAL LETTER E WITH CEDILLA\n                    \"Ɇ\",  // U+0246: LATIN CAPITAL LETTER E WITH STROKE\n                    \"ᴇ\",  // U+1D07: LATIN LETTER SMALL CAPITAL E\n                    \"Ḕ\",  // U+1E14: LATIN CAPITAL LETTER E WITH MACRON AND GRAVE\n                    \"Ḗ\",  // U+1E16: LATIN CAPITAL LETTER E WITH MACRON AND ACUTE\n                    \"Ḙ\",  // U+1E18: LATIN CAPITAL LETTER E WITH CIRCUMFLEX BELOW\n                    \"Ḛ\",  // U+1E1A: LATIN CAPITAL LETTER E WITH TILDE BELOW\n                    \"Ḝ\",  // U+1E1C: LATIN CAPITAL LETTER E WITH CEDILLA AND BREVE\n                    \"Ẹ\",  // U+1EB8: LATIN CAPITAL LETTER E WITH DOT BELOW\n                    \"Ẻ\",  // U+1EBA: LATIN CAPITAL LETTER E WITH HOOK ABOVE\n                    \"Ẽ\",  // U+1EBC: LATIN CAPITAL LETTER E WITH TILDE\n                    \"Ế\",  // U+1EBE: LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND ACUTE\n                    \"Ề\",  // U+1EC0: LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND GRAVE\n                    \"Ể\",  // U+1EC2: LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"Ễ\",  // U+1EC4: LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND TILDE\n                    \"Ệ\",  // U+1EC6: LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND DOT BELOW\n                    \"Ⓔ\",  // U+24BA: CIRCLED LATIN CAPITAL LETTER E\n                    \"ⱻ\",  // U+2C7B: LATIN LETTER SMALL CAPITAL TURNED E\n                    \"Ｅ\", // U+FF25: FULLWIDTH LATIN CAPITAL LETTER E\n                ],\n                \"E\",\n            ),\n            (\n                &[\n                    \"è\",  // U+00E8: LATIN SMALL LETTER E WITH GRAVE\n                    \"é\",  // U+00E9: LATIN SMALL LETTER E WITH ACUTE\n                    \"ê\",  // U+00EA: LATIN SMALL LETTER E WITH CIRCUMFLEX\n                    \"ë\",  // U+00EB: LATIN SMALL LETTER E WITH DIAERESIS\n                    \"ē\",  // U+0113: LATIN SMALL LETTER E WITH MACRON\n                    \"ĕ\",  // U+0115: LATIN SMALL LETTER E WITH BREVE\n                    \"ė\",  // U+0117: LATIN SMALL LETTER E WITH DOT ABOVE\n                    \"ę\",  // U+0119: LATIN SMALL LETTER E WITH OGONEK\n                    \"ě\",  // U+011B: LATIN SMALL LETTER E WITH CARON\n                    \"ǝ\",  // U+01DD: LATIN SMALL LETTER TURNED E\n                    \"ȅ\",  // U+0205: LATIN SMALL LETTER E WITH DOUBLE GRAVE\n                    \"ȇ\",  // U+0207: LATIN SMALL LETTER E WITH INVERTED BREVE\n                    \"ȩ\",  // U+0229: LATIN SMALL LETTER E WITH CEDILLA\n                    \"ɇ\",  // U+0247: LATIN SMALL LETTER E WITH STROKE\n                    \"ɘ\",  // U+0258: LATIN SMALL LETTER REVERSED E\n                    \"ɛ\",  // U+025B: LATIN SMALL LETTER OPEN E\n                    \"ɜ\",  // U+025C: LATIN SMALL LETTER REVERSED OPEN E\n                    \"ɝ\",  // U+025D: LATIN SMALL LETTER REVERSED OPEN E WITH HOOK\n                    \"ɞ\",  // U+025E: LATIN SMALL LETTER CLOSED REVERSED OPEN E\n                    \"ʚ\",  // U+029A: LATIN SMALL LETTER CLOSED OPEN E\n                    \"ᴈ\",  // U+1D08: LATIN SMALL LETTER TURNED OPEN E\n                    \"ᶒ\",  // U+1D92: LATIN SMALL LETTER E WITH RETROFLEX HOOK\n                    \"ᶓ\",  // U+1D93: LATIN SMALL LETTER OPEN E WITH RETROFLEX HOOK\n                    \"ᶔ\",  // U+1D94: LATIN SMALL LETTER REVERSED OPEN E WITH RETROFLEX HOOK\n                    \"ḕ\",  // U+1E15: LATIN SMALL LETTER E WITH MACRON AND GRAVE\n                    \"ḗ\",  // U+1E17: LATIN SMALL LETTER E WITH MACRON AND ACUTE\n                    \"ḙ\",  // U+1E19: LATIN SMALL LETTER E WITH CIRCUMFLEX BELOW\n                    \"ḛ\",  // U+1E1B: LATIN SMALL LETTER E WITH TILDE BELOW\n                    \"ḝ\",  // U+1E1D: LATIN SMALL LETTER E WITH CEDILLA AND BREVE\n                    \"ẹ\",  // U+1EB9: LATIN SMALL LETTER E WITH DOT BELOW\n                    \"ẻ\",  // U+1EBB: LATIN SMALL LETTER E WITH HOOK ABOVE\n                    \"ẽ\",  // U+1EBD: LATIN SMALL LETTER E WITH TILDE\n                    \"ế\",  // U+1EBF: LATIN SMALL LETTER E WITH CIRCUMFLEX AND ACUTE\n                    \"ề\",  // U+1EC1: LATIN SMALL LETTER E WITH CIRCUMFLEX AND GRAVE\n                    \"ể\",  // U+1EC3: LATIN SMALL LETTER E WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"ễ\",  // U+1EC5: LATIN SMALL LETTER E WITH CIRCUMFLEX AND TILDE\n                    \"ệ\",  // U+1EC7: LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW\n                    \"ₑ\",  // U+2091: LATIN SUBSCRIPT SMALL LETTER E\n                    \"ⓔ\",  // U+24D4: CIRCLED LATIN SMALL LETTER E\n                    \"ⱸ\",  // U+2C78: LATIN SMALL LETTER E WITH NOTCH\n                    \"ｅ\", // U+FF45: FULLWIDTH LATIN SMALL LETTER E\n                ],\n                \"e\",\n            ),\n            (\n                &[\n                    \"⒠\", // U+24A0: PARENTHESIZED LATIN SMALL LETTER E\n                ],\n                \"(e)\",\n            ),\n            (\n                &[\n                    \"Ƒ\",  // U+0191: LATIN CAPITAL LETTER F WITH HOOK\n                    \"Ḟ\",  // U+1E1E: LATIN CAPITAL LETTER F WITH DOT ABOVE\n                    \"Ⓕ\",  // U+24BB: CIRCLED LATIN CAPITAL LETTER F\n                    \"ꜰ\",  // U+A730: LATIN LETTER SMALL CAPITAL F\n                    \"Ꝼ\",  // U+A77B: LATIN CAPITAL LETTER INSULAR F\n                    \"ꟻ\",  // U+A7FB: LATIN EPIGRAPHIC LETTER REVERSED F\n                    \"Ｆ\", // U+FF26: FULLWIDTH LATIN CAPITAL LETTER F\n                ],\n                \"F\",\n            ),\n            (\n                &[\n                    \"ƒ\",  // U+0192: LATIN SMALL LETTER F WITH HOOK\n                    \"ᵮ\",  // U+1D6E: LATIN SMALL LETTER F WITH MIDDLE TILDE\n                    \"ᶂ\",  // U+1D82: LATIN SMALL LETTER F WITH PALATAL HOOK\n                    \"ḟ\",  // U+1E1F: LATIN SMALL LETTER F WITH DOT ABOVE\n                    \"ẛ\",  // U+1E9B: LATIN SMALL LETTER LONG S WITH DOT ABOVE\n                    \"ⓕ\",  // U+24D5: CIRCLED LATIN SMALL LETTER F\n                    \"ꝼ\",  // U+A77C: LATIN SMALL LETTER INSULAR F\n                    \"ｆ\", // U+FF46: FULLWIDTH LATIN SMALL LETTER F\n                ],\n                \"f\",\n            ),\n            (\n                &[\n                    \"⒡\", // U+24A1: PARENTHESIZED LATIN SMALL LETTER F\n                ],\n                \"(f)\",\n            ),\n            (\n                &[\n                    \"ff\", // U+FB00: LATIN SMALL LIGATURE FF\n                ],\n                \"ff\",\n            ),\n            (\n                &[\n                    \"ffi\", // U+FB03: LATIN SMALL LIGATURE FFI\n                ],\n                \"ffi\",\n            ),\n            (\n                &[\n                    \"ffl\", // U+FB04: LATIN SMALL LIGATURE FFL\n                ],\n                \"ffl\",\n            ),\n            (\n                &[\n                    \"fi\", // U+FB01: LATIN SMALL LIGATURE FI\n                ],\n                \"fi\",\n            ),\n            (\n                &[\n                    \"fl\", // U+FB02: LATIN SMALL LIGATURE FL\n                ],\n                \"fl\",\n            ),\n            (\n                &[\n                    \"Ĝ\",  // U+011C: LATIN CAPITAL LETTER G WITH CIRCUMFLEX\n                    \"Ğ\",  // U+011E: LATIN CAPITAL LETTER G WITH BREVE\n                    \"Ġ\",  // U+0120: LATIN CAPITAL LETTER G WITH DOT ABOVE\n                    \"Ģ\",  // U+0122: LATIN CAPITAL LETTER G WITH CEDILLA\n                    \"Ɠ\",  // U+0193: LATIN CAPITAL LETTER G WITH HOOK\n                    \"Ǥ\",  // U+01E4: LATIN CAPITAL LETTER G WITH STROKE\n                    \"ǥ\",  // U+01E5: LATIN SMALL LETTER G WITH STROKE\n                    \"Ǧ\",  // U+01E6: LATIN CAPITAL LETTER G WITH CARON\n                    \"ǧ\",  // U+01E7: LATIN SMALL LETTER G WITH CARON\n                    \"Ǵ\",  // U+01F4: LATIN CAPITAL LETTER G WITH ACUTE\n                    \"ɢ\",  // U+0262: LATIN LETTER SMALL CAPITAL G\n                    \"ʛ\",  // U+029B: LATIN LETTER SMALL CAPITAL G WITH HOOK\n                    \"Ḡ\",  // U+1E20: LATIN CAPITAL LETTER G WITH MACRON\n                    \"Ⓖ\",  // U+24BC: CIRCLED LATIN CAPITAL LETTER G\n                    \"Ᵹ\",  // U+A77D: LATIN CAPITAL LETTER INSULAR G\n                    \"Ꝿ\",  // U+A77E: LATIN CAPITAL LETTER TURNED INSULAR G\n                    \"Ｇ\", // U+FF27: FULLWIDTH LATIN CAPITAL LETTER G\n                ],\n                \"G\",\n            ),\n            (\n                &[\n                    \"ĝ\",  // U+011D: LATIN SMALL LETTER G WITH CIRCUMFLEX\n                    \"ğ\",  // U+011F: LATIN SMALL LETTER G WITH BREVE\n                    \"ġ\",  // U+0121: LATIN SMALL LETTER G WITH DOT ABOVE\n                    \"ģ\",  // U+0123: LATIN SMALL LETTER G WITH CEDILLA\n                    \"ǵ\",  // U+01F5: LATIN SMALL LETTER G WITH ACUTE\n                    \"ɠ\",  // U+0260: LATIN SMALL LETTER G WITH HOOK\n                    \"ɡ\",  // U+0261: LATIN SMALL LETTER SCRIPT G\n                    \"ᵷ\",  // U+1D77: LATIN SMALL LETTER TURNED G\n                    \"ᵹ\",  // U+1D79: LATIN SMALL LETTER INSULAR G\n                    \"ᶃ\",  // U+1D83: LATIN SMALL LETTER G WITH PALATAL HOOK\n                    \"ḡ\",  // U+1E21: LATIN SMALL LETTER G WITH MACRON\n                    \"ⓖ\",  // U+24D6: CIRCLED LATIN SMALL LETTER G\n                    \"ꝿ\",  // U+A77F: LATIN SMALL LETTER TURNED INSULAR G\n                    \"ｇ\", // U+FF47: FULLWIDTH LATIN SMALL LETTER G\n                ],\n                \"g\",\n            ),\n            (\n                &[\n                    \"⒢\", // U+24A2: PARENTHESIZED LATIN SMALL LETTER G\n                ],\n                \"(g)\",\n            ),\n            (\n                &[\n                    \"Ĥ\",  // U+0124: LATIN CAPITAL LETTER H WITH CIRCUMFLEX\n                    \"Ħ\",  // U+0126: LATIN CAPITAL LETTER H WITH STROKE\n                    \"Ȟ\",  // U+021E: LATIN CAPITAL LETTER H WITH CARON\n                    \"ʜ\",  // U+029C: LATIN LETTER SMALL CAPITAL H\n                    \"Ḣ\",  // U+1E22: LATIN CAPITAL LETTER H WITH DOT ABOVE\n                    \"Ḥ\",  // U+1E24: LATIN CAPITAL LETTER H WITH DOT BELOW\n                    \"Ḧ\",  // U+1E26: LATIN CAPITAL LETTER H WITH DIAERESIS\n                    \"Ḩ\",  // U+1E28: LATIN CAPITAL LETTER H WITH CEDILLA\n                    \"Ḫ\",  // U+1E2A: LATIN CAPITAL LETTER H WITH BREVE BELOW\n                    \"Ⓗ\",  // U+24BD: CIRCLED LATIN CAPITAL LETTER H\n                    \"Ⱨ\",  // U+2C67: LATIN CAPITAL LETTER H WITH DESCENDER\n                    \"Ⱶ\",  // U+2C75: LATIN CAPITAL LETTER HALF H\n                    \"Ｈ\", // U+FF28: FULLWIDTH LATIN CAPITAL LETTER H\n                ],\n                \"H\",\n            ),\n            (\n                &[\n                    \"ĥ\",  // U+0125: LATIN SMALL LETTER H WITH CIRCUMFLEX\n                    \"ħ\",  // U+0127: LATIN SMALL LETTER H WITH STROKE\n                    \"ȟ\",  // U+021F: LATIN SMALL LETTER H WITH CARON\n                    \"ɥ\",  // U+0265: LATIN SMALL LETTER TURNED H\n                    \"ɦ\",  // U+0266: LATIN SMALL LETTER H WITH HOOK\n                    \"ʮ\",  // U+02AE: LATIN SMALL LETTER TURNED H WITH FISHHOOK\n                    \"ʯ\",  // U+02AF: LATIN SMALL LETTER TURNED H WITH FISHHOOK AND TAIL\n                    \"ḣ\",  // U+1E23: LATIN SMALL LETTER H WITH DOT ABOVE\n                    \"ḥ\",  // U+1E25: LATIN SMALL LETTER H WITH DOT BELOW\n                    \"ḧ\",  // U+1E27: LATIN SMALL LETTER H WITH DIAERESIS\n                    \"ḩ\",  // U+1E29: LATIN SMALL LETTER H WITH CEDILLA\n                    \"ḫ\",  // U+1E2B: LATIN SMALL LETTER H WITH BREVE BELOW\n                    \"ẖ\",  // U+1E96: LATIN SMALL LETTER H WITH LINE BELOW\n                    \"ⓗ\",  // U+24D7: CIRCLED LATIN SMALL LETTER H\n                    \"ⱨ\",  // U+2C68: LATIN SMALL LETTER H WITH DESCENDER\n                    \"ⱶ\",  // U+2C76: LATIN SMALL LETTER HALF H\n                    \"ｈ\", // U+FF48: FULLWIDTH LATIN SMALL LETTER H\n                ],\n                \"h\",\n            ),\n            (\n                &[\n                    \"Ƕ\", // U+01F6: LATIN CAPITAL LETTER HWAIR\n                ],\n                \"HV\",\n            ),\n            (\n                &[\n                    \"⒣\", // U+24A3: PARENTHESIZED LATIN SMALL LETTER H\n                ],\n                \"(h)\",\n            ),\n            (\n                &[\n                    \"ƕ\", // U+0195: LATIN SMALL LETTER HV\n                ],\n                \"hv\",\n            ),\n            (\n                &[\n                    \"Ì\",  // U+00CC: LATIN CAPITAL LETTER I WITH GRAVE\n                    \"Í\",  // U+00CD: LATIN CAPITAL LETTER I WITH ACUTE\n                    \"Î\",  // U+00CE: LATIN CAPITAL LETTER I WITH CIRCUMFLEX\n                    \"Ï\",  // U+00CF: LATIN CAPITAL LETTER I WITH DIAERESIS\n                    \"Ĩ\",  // U+0128: LATIN CAPITAL LETTER I WITH TILDE\n                    \"Ī\",  // U+012A: LATIN CAPITAL LETTER I WITH MACRON\n                    \"Ĭ\",  // U+012C: LATIN CAPITAL LETTER I WITH BREVE\n                    \"Į\",  // U+012E: LATIN CAPITAL LETTER I WITH OGONEK\n                    \"İ\",  // U+0130: LATIN CAPITAL LETTER I WITH DOT ABOVE\n                    \"Ɩ\",  // U+0196: LATIN CAPITAL LETTER IOTA\n                    \"Ɨ\",  // U+0197: LATIN CAPITAL LETTER I WITH STROKE\n                    \"Ǐ\",  // U+01CF: LATIN CAPITAL LETTER I WITH CARON\n                    \"Ȉ\",  // U+0208: LATIN CAPITAL LETTER I WITH DOUBLE GRAVE\n                    \"Ȋ\",  // U+020A: LATIN CAPITAL LETTER I WITH INVERTED BREVE\n                    \"ɪ\",  // U+026A: LATIN LETTER SMALL CAPITAL I\n                    \"ᵻ\",  // U+1D7B: LATIN SMALL CAPITAL LETTER I WITH STROKE\n                    \"Ḭ\",  // U+1E2C: LATIN CAPITAL LETTER I WITH TILDE BELOW\n                    \"Ḯ\",  // U+1E2E: LATIN CAPITAL LETTER I WITH DIAERESIS AND ACUTE\n                    \"Ỉ\",  // U+1EC8: LATIN CAPITAL LETTER I WITH HOOK ABOVE\n                    \"Ị\",  // U+1ECA: LATIN CAPITAL LETTER I WITH DOT BELOW\n                    \"Ⓘ\",  // U+24BE: CIRCLED LATIN CAPITAL LETTER I\n                    \"ꟾ\",  // U+A7FE: LATIN EPIGRAPHIC LETTER I LONGA\n                    \"Ｉ\", // U+FF29: FULLWIDTH LATIN CAPITAL LETTER I\n                ],\n                \"I\",\n            ),\n            (\n                &[\n                    \"ì\",  // U+00EC: LATIN SMALL LETTER I WITH GRAVE\n                    \"í\",  // U+00ED: LATIN SMALL LETTER I WITH ACUTE\n                    \"î\",  // U+00EE: LATIN SMALL LETTER I WITH CIRCUMFLEX\n                    \"ï\",  // U+00EF: LATIN SMALL LETTER I WITH DIAERESIS\n                    \"ĩ\",  // U+0129: LATIN SMALL LETTER I WITH TILDE\n                    \"ī\",  // U+012B: LATIN SMALL LETTER I WITH MACRON\n                    \"ĭ\",  // U+012D: LATIN SMALL LETTER I WITH BREVE\n                    \"į\",  // U+012F: LATIN SMALL LETTER I WITH OGONEK\n                    \"ı\",  // U+0131: LATIN SMALL LETTER DOTLESS I\n                    \"ǐ\",  // U+01D0: LATIN SMALL LETTER I WITH CARON\n                    \"ȉ\",  // U+0209: LATIN SMALL LETTER I WITH DOUBLE GRAVE\n                    \"ȋ\",  // U+020B: LATIN SMALL LETTER I WITH INVERTED BREVE\n                    \"ɨ\",  // U+0268: LATIN SMALL LETTER I WITH STROKE\n                    \"ᴉ\",  // U+1D09: LATIN SMALL LETTER TURNED I\n                    \"ᵢ\",  // U+1D62: LATIN SUBSCRIPT SMALL LETTER I\n                    \"ᵼ\",  // U+1D7C: LATIN SMALL LETTER IOTA WITH STROKE\n                    \"ᶖ\",  // U+1D96: LATIN SMALL LETTER I WITH RETROFLEX HOOK\n                    \"ḭ\",  // U+1E2D: LATIN SMALL LETTER I WITH TILDE BELOW\n                    \"ḯ\",  // U+1E2F: LATIN SMALL LETTER I WITH DIAERESIS AND ACUTE\n                    \"ỉ\",  // U+1EC9: LATIN SMALL LETTER I WITH HOOK ABOVE\n                    \"ị\",  // U+1ECB: LATIN SMALL LETTER I WITH DOT BELOW\n                    \"ⁱ\",  // U+2071: SUPERSCRIPT LATIN SMALL LETTER I\n                    \"ⓘ\",  // U+24D8: CIRCLED LATIN SMALL LETTER I\n                    \"ｉ\", // U+FF49: FULLWIDTH LATIN SMALL LETTER I\n                ],\n                \"i\",\n            ),\n            (\n                &[\n                    \"IJ\", // U+0132: LATIN CAPITAL LIGATURE IJ\n                ],\n                \"IJ\",\n            ),\n            (\n                &[\n                    \"⒤\", // U+24A4: PARENTHESIZED LATIN SMALL LETTER I\n                ],\n                \"(i)\",\n            ),\n            (\n                &[\n                    \"ij\", // U+0133: LATIN SMALL LIGATURE IJ\n                ],\n                \"ij\",\n            ),\n            (\n                &[\n                    \"Ĵ\",  // U+0134: LATIN CAPITAL LETTER J WITH CIRCUMFLEX\n                    \"Ɉ\",  // U+0248: LATIN CAPITAL LETTER J WITH STROKE\n                    \"ᴊ\",  // U+1D0A: LATIN LETTER SMALL CAPITAL J\n                    \"Ⓙ\",  // U+24BF: CIRCLED LATIN CAPITAL LETTER J\n                    \"Ｊ\", // U+FF2A: FULLWIDTH LATIN CAPITAL LETTER J\n                ],\n                \"J\",\n            ),\n            (\n                &[\n                    \"ĵ\",  // U+0135: LATIN SMALL LETTER J WITH CIRCUMFLEX\n                    \"ǰ\",  // U+01F0: LATIN SMALL LETTER J WITH CARON\n                    \"ȷ\",  // U+0237: LATIN SMALL LETTER DOTLESS J\n                    \"ɉ\",  // U+0249: LATIN SMALL LETTER J WITH STROKE\n                    \"ɟ\",  // U+025F: LATIN SMALL LETTER DOTLESS J WITH STROKE\n                    \"ʄ\",  // U+0284: LATIN SMALL LETTER DOTLESS J WITH STROKE AND HOOK\n                    \"ʝ\",  // U+029D: LATIN SMALL LETTER J WITH CROSSED-TAIL\n                    \"ⓙ\",  // U+24D9: CIRCLED LATIN SMALL LETTER J\n                    \"ⱼ\",  // U+2C7C: LATIN SUBSCRIPT SMALL LETTER J\n                    \"ｊ\", // U+FF4A: FULLWIDTH LATIN SMALL LETTER J\n                ],\n                \"j\",\n            ),\n            (\n                &[\n                    \"⒥\", // U+24A5: PARENTHESIZED LATIN SMALL LETTER J\n                ],\n                \"(j)\",\n            ),\n            (\n                &[\n                    \"Ķ\",  // U+0136: LATIN CAPITAL LETTER K WITH CEDILLA\n                    \"Ƙ\",  // U+0198: LATIN CAPITAL LETTER K WITH HOOK\n                    \"Ǩ\",  // U+01E8: LATIN CAPITAL LETTER K WITH CARON\n                    \"ᴋ\",  // U+1D0B: LATIN LETTER SMALL CAPITAL K\n                    \"Ḱ\",  // U+1E30: LATIN CAPITAL LETTER K WITH ACUTE\n                    \"Ḳ\",  // U+1E32: LATIN CAPITAL LETTER K WITH DOT BELOW\n                    \"Ḵ\",  // U+1E34: LATIN CAPITAL LETTER K WITH LINE BELOW\n                    \"Ⓚ\",  // U+24C0: CIRCLED LATIN CAPITAL LETTER K\n                    \"Ⱪ\",  // U+2C69: LATIN CAPITAL LETTER K WITH DESCENDER\n                    \"Ꝁ\",  // U+A740: LATIN CAPITAL LETTER K WITH STROKE\n                    \"Ꝃ\",  // U+A742: LATIN CAPITAL LETTER K WITH DIAGONAL STROKE\n                    \"Ꝅ\",  // U+A744: LATIN CAPITAL LETTER K WITH STROKE AND DIAGONAL STROKE\n                    \"Ｋ\", // U+FF2B: FULLWIDTH LATIN CAPITAL LETTER K\n                ],\n                \"K\",\n            ),\n            (\n                &[\n                    \"ķ\",  // U+0137: LATIN SMALL LETTER K WITH CEDILLA\n                    \"ƙ\",  // U+0199: LATIN SMALL LETTER K WITH HOOK\n                    \"ǩ\",  // U+01E9: LATIN SMALL LETTER K WITH CARON\n                    \"ʞ\",  // U+029E: LATIN SMALL LETTER TURNED K\n                    \"ᶄ\",  // U+1D84: LATIN SMALL LETTER K WITH PALATAL HOOK\n                    \"ḱ\",  // U+1E31: LATIN SMALL LETTER K WITH ACUTE\n                    \"ḳ\",  // U+1E33: LATIN SMALL LETTER K WITH DOT BELOW\n                    \"ḵ\",  // U+1E35: LATIN SMALL LETTER K WITH LINE BELOW\n                    \"ⓚ\",  // U+24DA: CIRCLED LATIN SMALL LETTER K\n                    \"ⱪ\",  // U+2C6A: LATIN SMALL LETTER K WITH DESCENDER\n                    \"ꝁ\",  // U+A741: LATIN SMALL LETTER K WITH STROKE\n                    \"ꝃ\",  // U+A743: LATIN SMALL LETTER K WITH DIAGONAL STROKE\n                    \"ꝅ\",  // U+A745: LATIN SMALL LETTER K WITH STROKE AND DIAGONAL STROKE\n                    \"ｋ\", // U+FF4B: FULLWIDTH LATIN SMALL LETTER K\n                ],\n                \"k\",\n            ),\n            (\n                &[\n                    \"⒦\", // U+24A6: PARENTHESIZED LATIN SMALL LETTER K\n                ],\n                \"(k)\",\n            ),\n            (\n                &[\n                    \"Ĺ\",  // U+0139: LATIN CAPITAL LETTER L WITH ACUTE\n                    \"Ļ\",  // U+013B: LATIN CAPITAL LETTER L WITH CEDILLA\n                    \"Ľ\",  // U+013D: LATIN CAPITAL LETTER L WITH CARON\n                    \"Ŀ\",  // U+013F: LATIN CAPITAL LETTER L WITH MIDDLE DOT\n                    \"Ł\",  // U+0141: LATIN CAPITAL LETTER L WITH STROKE\n                    \"Ƚ\",  // U+023D: LATIN CAPITAL LETTER L WITH BAR\n                    \"ʟ\",  // U+029F: LATIN LETTER SMALL CAPITAL L\n                    \"ᴌ\",  // U+1D0C: LATIN LETTER SMALL CAPITAL L WITH STROKE\n                    \"Ḷ\",  // U+1E36: LATIN CAPITAL LETTER L WITH DOT BELOW\n                    \"Ḹ\",  // U+1E38: LATIN CAPITAL LETTER L WITH DOT BELOW AND MACRON\n                    \"Ḻ\",  // U+1E3A: LATIN CAPITAL LETTER L WITH LINE BELOW\n                    \"Ḽ\",  // U+1E3C: LATIN CAPITAL LETTER L WITH CIRCUMFLEX BELOW\n                    \"Ⓛ\",  // U+24C1: CIRCLED LATIN CAPITAL LETTER L\n                    \"Ⱡ\",  // U+2C60: LATIN CAPITAL LETTER L WITH DOUBLE BAR\n                    \"Ɫ\",  // U+2C62: LATIN CAPITAL LETTER L WITH MIDDLE TILDE\n                    \"Ꝇ\",  // U+A746: LATIN CAPITAL LETTER BROKEN L\n                    \"Ꝉ\",  // U+A748: LATIN CAPITAL LETTER L WITH HIGH STROKE\n                    \"Ꞁ\",  // U+A780: LATIN CAPITAL LETTER TURNED L\n                    \"Ｌ\", // U+FF2C: FULLWIDTH LATIN CAPITAL LETTER L\n                ],\n                \"L\",\n            ),\n            (\n                &[\n                    \"ĺ\",  // U+013A: LATIN SMALL LETTER L WITH ACUTE\n                    \"ļ\",  // U+013C: LATIN SMALL LETTER L WITH CEDILLA\n                    \"ľ\",  // U+013E: LATIN SMALL LETTER L WITH CARON\n                    \"ŀ\",  // U+0140: LATIN SMALL LETTER L WITH MIDDLE DOT\n                    \"ł\",  // U+0142: LATIN SMALL LETTER L WITH STROKE\n                    \"ƚ\",  // U+019A: LATIN SMALL LETTER L WITH BAR\n                    \"ȴ\",  // U+0234: LATIN SMALL LETTER L WITH CURL\n                    \"ɫ\",  // U+026B: LATIN SMALL LETTER L WITH MIDDLE TILDE\n                    \"ɬ\",  // U+026C: LATIN SMALL LETTER L WITH BELT\n                    \"ɭ\",  // U+026D: LATIN SMALL LETTER L WITH RETROFLEX HOOK\n                    \"ᶅ\",  // U+1D85: LATIN SMALL LETTER L WITH PALATAL HOOK\n                    \"ḷ\",  // U+1E37: LATIN SMALL LETTER L WITH DOT BELOW\n                    \"ḹ\",  // U+1E39: LATIN SMALL LETTER L WITH DOT BELOW AND MACRON\n                    \"ḻ\",  // U+1E3B: LATIN SMALL LETTER L WITH LINE BELOW\n                    \"ḽ\",  // U+1E3D: LATIN SMALL LETTER L WITH CIRCUMFLEX BELOW\n                    \"ⓛ\",  // U+24DB: CIRCLED LATIN SMALL LETTER L\n                    \"ⱡ\",  // U+2C61: LATIN SMALL LETTER L WITH DOUBLE BAR\n                    \"ꝇ\",  // U+A747: LATIN SMALL LETTER BROKEN L\n                    \"ꝉ\",  // U+A749: LATIN SMALL LETTER L WITH HIGH STROKE\n                    \"ꞁ\",  // U+A781: LATIN SMALL LETTER TURNED L\n                    \"ｌ\", // U+FF4C: FULLWIDTH LATIN SMALL LETTER L\n                ],\n                \"l\",\n            ),\n            (\n                &[\n                    \"LJ\", // U+01C7: LATIN CAPITAL LETTER LJ\n                ],\n                \"LJ\",\n            ),\n            (\n                &[\n                    \"Ỻ\", // U+1EFA: LATIN CAPITAL LETTER MIDDLE-WELSH LL\n                ],\n                \"LL\",\n            ),\n            (\n                &[\n                    \"Lj\", // U+01C8: LATIN CAPITAL LETTER L WITH SMALL LETTER J\n                ],\n                \"Lj\",\n            ),\n            (\n                &[\n                    \"⒧\", // U+24A7: PARENTHESIZED LATIN SMALL LETTER L\n                ],\n                \"(l)\",\n            ),\n            (\n                &[\n                    \"lj\", // U+01C9: LATIN SMALL LETTER LJ\n                ],\n                \"lj\",\n            ),\n            (\n                &[\n                    \"ỻ\", // U+1EFB: LATIN SMALL LETTER MIDDLE-WELSH LL\n                ],\n                \"ll\",\n            ),\n            (\n                &[\n                    \"ʪ\", // U+02AA: LATIN SMALL LETTER LS DIGRAPH\n                ],\n                \"ls\",\n            ),\n            (\n                &[\n                    \"ʫ\", // U+02AB: LATIN SMALL LETTER LZ DIGRAPH\n                ],\n                \"lz\",\n            ),\n            (\n                &[\n                    \"Ɯ\",  // U+019C: LATIN CAPITAL LETTER TURNED M\n                    \"ᴍ\",  // U+1D0D: LATIN LETTER SMALL CAPITAL M\n                    \"Ḿ\",  // U+1E3E: LATIN CAPITAL LETTER M WITH ACUTE\n                    \"Ṁ\",  // U+1E40: LATIN CAPITAL LETTER M WITH DOT ABOVE\n                    \"Ṃ\",  // U+1E42: LATIN CAPITAL LETTER M WITH DOT BELOW\n                    \"Ⓜ\",  // U+24C2: CIRCLED LATIN CAPITAL LETTER M\n                    \"Ɱ\",  // U+2C6E: LATIN CAPITAL LETTER M WITH HOOK\n                    \"ꟽ\",  // U+A7FD: LATIN EPIGRAPHIC LETTER INVERTED M\n                    \"ꟿ\",  // U+A7FF: LATIN EPIGRAPHIC LETTER ARCHAIC M\n                    \"Ｍ\", // U+FF2D: FULLWIDTH LATIN CAPITAL LETTER M\n                ],\n                \"M\",\n            ),\n            (\n                &[\n                    \"ɯ\",  // U+026F: LATIN SMALL LETTER TURNED M\n                    \"ɰ\",  // U+0270: LATIN SMALL LETTER TURNED M WITH LONG LEG\n                    \"ɱ\",  // U+0271: LATIN SMALL LETTER M WITH HOOK\n                    \"ᵯ\",  // U+1D6F: LATIN SMALL LETTER M WITH MIDDLE TILDE\n                    \"ᶆ\",  // U+1D86: LATIN SMALL LETTER M WITH PALATAL HOOK\n                    \"ḿ\",  // U+1E3F: LATIN SMALL LETTER M WITH ACUTE\n                    \"ṁ\",  // U+1E41: LATIN SMALL LETTER M WITH DOT ABOVE\n                    \"ṃ\",  // U+1E43: LATIN SMALL LETTER M WITH DOT BELOW\n                    \"ⓜ\",  // U+24DC: CIRCLED LATIN SMALL LETTER M\n                    \"ｍ\", // U+FF4D: FULLWIDTH LATIN SMALL LETTER M\n                ],\n                \"m\",\n            ),\n            (\n                &[\n                    \"⒨\", // U+24A8: PARENTHESIZED LATIN SMALL LETTER M\n                ],\n                \"(m)\",\n            ),\n            (\n                &[\n                    \"Ñ\",  // U+00D1: LATIN CAPITAL LETTER N WITH TILDE\n                    \"Ń\",  // U+0143: LATIN CAPITAL LETTER N WITH ACUTE\n                    \"Ņ\",  // U+0145: LATIN CAPITAL LETTER N WITH CEDILLA\n                    \"Ň\",  // U+0147: LATIN CAPITAL LETTER N WITH CARON\n                    \"Ŋ\",  // U+014A: LATIN CAPITAL LETTER ENG\n                    \"Ɲ\",  // U+019D: LATIN CAPITAL LETTER N WITH LEFT HOOK\n                    \"Ǹ\",  // U+01F8: LATIN CAPITAL LETTER N WITH GRAVE\n                    \"Ƞ\",  // U+0220: LATIN CAPITAL LETTER N WITH LONG RIGHT LEG\n                    \"ɴ\",  // U+0274: LATIN LETTER SMALL CAPITAL N\n                    \"ᴎ\",  // U+1D0E: LATIN LETTER SMALL CAPITAL REVERSED N\n                    \"Ṅ\",  // U+1E44: LATIN CAPITAL LETTER N WITH DOT ABOVE\n                    \"Ṇ\",  // U+1E46: LATIN CAPITAL LETTER N WITH DOT BELOW\n                    \"Ṉ\",  // U+1E48: LATIN CAPITAL LETTER N WITH LINE BELOW\n                    \"Ṋ\",  // U+1E4A: LATIN CAPITAL LETTER N WITH CIRCUMFLEX BELOW\n                    \"Ⓝ\",  // U+24C3: CIRCLED LATIN CAPITAL LETTER N\n                    \"Ｎ\", // U+FF2E: FULLWIDTH LATIN CAPITAL LETTER N\n                ],\n                \"N\",\n            ),\n            (\n                &[\n                    \"ñ\",  // U+00F1: LATIN SMALL LETTER N WITH TILDE\n                    \"ń\",  // U+0144: LATIN SMALL LETTER N WITH ACUTE\n                    \"ņ\",  // U+0146: LATIN SMALL LETTER N WITH CEDILLA\n                    \"ň\",  // U+0148: LATIN SMALL LETTER N WITH CARON\n                    \"ʼn\",  // U+0149: LATIN SMALL LETTER N PRECEDED BY APOSTROPHE\n                    \"ŋ\",  // U+014B: LATIN SMALL LETTER ENG\n                    \"ƞ\",  // U+019E: LATIN SMALL LETTER N WITH LONG RIGHT LEG\n                    \"ǹ\",  // U+01F9: LATIN SMALL LETTER N WITH GRAVE\n                    \"ȵ\",  // U+0235: LATIN SMALL LETTER N WITH CURL\n                    \"ɲ\",  // U+0272: LATIN SMALL LETTER N WITH LEFT HOOK\n                    \"ɳ\",  // U+0273: LATIN SMALL LETTER N WITH RETROFLEX HOOK\n                    \"ᵰ\",  // U+1D70: LATIN SMALL LETTER N WITH MIDDLE TILDE\n                    \"ᶇ\",  // U+1D87: LATIN SMALL LETTER N WITH PALATAL HOOK\n                    \"ṅ\",  // U+1E45: LATIN SMALL LETTER N WITH DOT ABOVE\n                    \"ṇ\",  // U+1E47: LATIN SMALL LETTER N WITH DOT BELOW\n                    \"ṉ\",  // U+1E49: LATIN SMALL LETTER N WITH LINE BELOW\n                    \"ṋ\",  // U+1E4B: LATIN SMALL LETTER N WITH CIRCUMFLEX BELOW\n                    \"ⁿ\",  // U+207F: SUPERSCRIPT LATIN SMALL LETTER N\n                    \"ⓝ\",  // U+24DD: CIRCLED LATIN SMALL LETTER N\n                    \"ｎ\", // U+FF4E: FULLWIDTH LATIN SMALL LETTER N\n                ],\n                \"n\",\n            ),\n            (\n                &[\n                    \"NJ\", // U+01CA: LATIN CAPITAL LETTER NJ\n                ],\n                \"NJ\",\n            ),\n            (\n                &[\n                    \"Nj\", // U+01CB: LATIN CAPITAL LETTER N WITH SMALL LETTER J\n                ],\n                \"Nj\",\n            ),\n            (\n                &[\n                    \"⒩\", // U+24A9: PARENTHESIZED LATIN SMALL LETTER N\n                ],\n                \"(n)\",\n            ),\n            (\n                &[\n                    \"nj\", // U+01CC: LATIN SMALL LETTER NJ\n                ],\n                \"nj\",\n            ),\n            (\n                &[\n                    \"Ò\",  // U+00D2: LATIN CAPITAL LETTER O WITH GRAVE\n                    \"Ó\",  // U+00D3: LATIN CAPITAL LETTER O WITH ACUTE\n                    \"Ô\",  // U+00D4: LATIN CAPITAL LETTER O WITH CIRCUMFLEX\n                    \"Õ\",  // U+00D5: LATIN CAPITAL LETTER O WITH TILDE\n                    \"Ö\",  // U+00D6: LATIN CAPITAL LETTER O WITH DIAERESIS\n                    \"Ø\",  // U+00D8: LATIN CAPITAL LETTER O WITH STROKE\n                    \"Ō\",  // U+014C: LATIN CAPITAL LETTER O WITH MACRON\n                    \"Ŏ\",  // U+014E: LATIN CAPITAL LETTER O WITH BREVE\n                    \"Ő\",  // U+0150: LATIN CAPITAL LETTER O WITH DOUBLE ACUTE\n                    \"Ɔ\",  // U+0186: LATIN CAPITAL LETTER OPEN O\n                    \"Ɵ\",  // U+019F: LATIN CAPITAL LETTER O WITH MIDDLE TILDE\n                    \"Ơ\",  // U+01A0: LATIN CAPITAL LETTER O WITH HORN\n                    \"Ǒ\",  // U+01D1: LATIN CAPITAL LETTER O WITH CARON\n                    \"Ǫ\",  // U+01EA: LATIN CAPITAL LETTER O WITH OGONEK\n                    \"Ǭ\",  // U+01EC: LATIN CAPITAL LETTER O WITH OGONEK AND MACRON\n                    \"Ǿ\",  // U+01FE: LATIN CAPITAL LETTER O WITH STROKE AND ACUTE\n                    \"Ȍ\",  // U+020C: LATIN CAPITAL LETTER O WITH DOUBLE GRAVE\n                    \"Ȏ\",  // U+020E: LATIN CAPITAL LETTER O WITH INVERTED BREVE\n                    \"Ȫ\",  // U+022A: LATIN CAPITAL LETTER O WITH DIAERESIS AND MACRON\n                    \"Ȭ\",  // U+022C: LATIN CAPITAL LETTER O WITH TILDE AND MACRON\n                    \"Ȯ\",  // U+022E: LATIN CAPITAL LETTER O WITH DOT ABOVE\n                    \"Ȱ\",  // U+0230: LATIN CAPITAL LETTER O WITH DOT ABOVE AND MACRON\n                    \"ᴏ\",  // U+1D0F: LATIN LETTER SMALL CAPITAL O\n                    \"ᴐ\",  // U+1D10: LATIN LETTER SMALL CAPITAL OPEN O\n                    \"Ṍ\",  // U+1E4C: LATIN CAPITAL LETTER O WITH TILDE AND ACUTE\n                    \"Ṏ\",  // U+1E4E: LATIN CAPITAL LETTER O WITH TILDE AND DIAERESIS\n                    \"Ṑ\",  // U+1E50: LATIN CAPITAL LETTER O WITH MACRON AND GRAVE\n                    \"Ṓ\",  // U+1E52: LATIN CAPITAL LETTER O WITH MACRON AND ACUTE\n                    \"Ọ\",  // U+1ECC: LATIN CAPITAL LETTER O WITH DOT BELOW\n                    \"Ỏ\",  // U+1ECE: LATIN CAPITAL LETTER O WITH HOOK ABOVE\n                    \"Ố\",  // U+1ED0: LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND ACUTE\n                    \"Ồ\",  // U+1ED2: LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND GRAVE\n                    \"Ổ\",  // U+1ED4: LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"Ỗ\",  // U+1ED6: LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND TILDE\n                    \"Ộ\",  // U+1ED8: LATIN CAPITAL LETTER O WITH CIRCUMFLEX AND DOT BELOW\n                    \"Ớ\",  // U+1EDA: LATIN CAPITAL LETTER O WITH HORN AND ACUTE\n                    \"Ờ\",  // U+1EDC: LATIN CAPITAL LETTER O WITH HORN AND GRAVE\n                    \"Ở\",  // U+1EDE: LATIN CAPITAL LETTER O WITH HORN AND HOOK ABOVE\n                    \"Ỡ\",  // U+1EE0: LATIN CAPITAL LETTER O WITH HORN AND TILDE\n                    \"Ợ\",  // U+1EE2: LATIN CAPITAL LETTER O WITH HORN AND DOT BELOW\n                    \"Ⓞ\",  // U+24C4: CIRCLED LATIN CAPITAL LETTER O\n                    \"Ꝋ\",  // U+A74A: LATIN CAPITAL LETTER O WITH LONG STROKE OVERLAY\n                    \"Ꝍ\",  // U+A74C: LATIN CAPITAL LETTER O WITH LOOP\n                    \"Ｏ\", // U+FF2F: FULLWIDTH LATIN CAPITAL LETTER O\n                ],\n                \"O\",\n            ),\n            (\n                &[\n                    \"ò\",  // U+00F2: LATIN SMALL LETTER O WITH GRAVE\n                    \"ó\",  // U+00F3: LATIN SMALL LETTER O WITH ACUTE\n                    \"ô\",  // U+00F4: LATIN SMALL LETTER O WITH CIRCUMFLEX\n                    \"õ\",  // U+00F5: LATIN SMALL LETTER O WITH TILDE\n                    \"ö\",  // U+00F6: LATIN SMALL LETTER O WITH DIAERESIS\n                    \"ø\",  // U+00F8: LATIN SMALL LETTER O WITH STROKE\n                    \"ō\",  // U+014D: LATIN SMALL LETTER O WITH MACRON\n                    \"ŏ\",  // U+014F: LATIN SMALL LETTER O WITH BREVE\n                    \"ő\",  // U+0151: LATIN SMALL LETTER O WITH DOUBLE ACUTE\n                    \"ơ\",  // U+01A1: LATIN SMALL LETTER O WITH HORN\n                    \"ǒ\",  // U+01D2: LATIN SMALL LETTER O WITH CARON\n                    \"ǫ\",  // U+01EB: LATIN SMALL LETTER O WITH OGONEK\n                    \"ǭ\",  // U+01ED: LATIN SMALL LETTER O WITH OGONEK AND MACRON\n                    \"ǿ\",  // U+01FF: LATIN SMALL LETTER O WITH STROKE AND ACUTE\n                    \"ȍ\",  // U+020D: LATIN SMALL LETTER O WITH DOUBLE GRAVE\n                    \"ȏ\",  // U+020F: LATIN SMALL LETTER O WITH INVERTED BREVE\n                    \"ȫ\",  // U+022B: LATIN SMALL LETTER O WITH DIAERESIS AND MACRON\n                    \"ȭ\",  // U+022D: LATIN SMALL LETTER O WITH TILDE AND MACRON\n                    \"ȯ\",  // U+022F: LATIN SMALL LETTER O WITH DOT ABOVE\n                    \"ȱ\",  // U+0231: LATIN SMALL LETTER O WITH DOT ABOVE AND MACRON\n                    \"ɔ\",  // U+0254: LATIN SMALL LETTER OPEN O\n                    \"ɵ\",  // U+0275: LATIN SMALL LETTER BARRED O\n                    \"ᴖ\",  // U+1D16: LATIN SMALL LETTER TOP HALF O\n                    \"ᴗ\",  // U+1D17: LATIN SMALL LETTER BOTTOM HALF O\n                    \"ᶗ\",  // U+1D97: LATIN SMALL LETTER OPEN O WITH RETROFLEX HOOK\n                    \"ṍ\",  // U+1E4D: LATIN SMALL LETTER O WITH TILDE AND ACUTE\n                    \"ṏ\",  // U+1E4F: LATIN SMALL LETTER O WITH TILDE AND DIAERESIS\n                    \"ṑ\",  // U+1E51: LATIN SMALL LETTER O WITH MACRON AND GRAVE\n                    \"ṓ\",  // U+1E53: LATIN SMALL LETTER O WITH MACRON AND ACUTE\n                    \"ọ\",  // U+1ECD: LATIN SMALL LETTER O WITH DOT BELOW\n                    \"ỏ\",  // U+1ECF: LATIN SMALL LETTER O WITH HOOK ABOVE\n                    \"ố\",  // U+1ED1: LATIN SMALL LETTER O WITH CIRCUMFLEX AND ACUTE\n                    \"ồ\",  // U+1ED3: LATIN SMALL LETTER O WITH CIRCUMFLEX AND GRAVE\n                    \"ổ\",  // U+1ED5: LATIN SMALL LETTER O WITH CIRCUMFLEX AND HOOK ABOVE\n                    \"ỗ\",  // U+1ED7: LATIN SMALL LETTER O WITH CIRCUMFLEX AND TILDE\n                    \"ộ\",  // U+1ED9: LATIN SMALL LETTER O WITH CIRCUMFLEX AND DOT BELOW\n                    \"ớ\",  // U+1EDB: LATIN SMALL LETTER O WITH HORN AND ACUTE\n                    \"ờ\",  // U+1EDD: LATIN SMALL LETTER O WITH HORN AND GRAVE\n                    \"ở\",  // U+1EDF: LATIN SMALL LETTER O WITH HORN AND HOOK ABOVE\n                    \"ỡ\",  // U+1EE1: LATIN SMALL LETTER O WITH HORN AND TILDE\n                    \"ợ\",  // U+1EE3: LATIN SMALL LETTER O WITH HORN AND DOT BELOW\n                    \"ₒ\",  // U+2092: LATIN SUBSCRIPT SMALL LETTER O\n                    \"ⓞ\",  // U+24DE: CIRCLED LATIN SMALL LETTER O\n                    \"ⱺ\",  // U+2C7A: LATIN SMALL LETTER O WITH LOW RING INSIDE\n                    \"ꝋ\",  // U+A74B: LATIN SMALL LETTER O WITH LONG STROKE OVERLAY\n                    \"ꝍ\",  // U+A74D: LATIN SMALL LETTER O WITH LOOP\n                    \"ｏ\", // U+FF4F: FULLWIDTH LATIN SMALL LETTER O\n                ],\n                \"o\",\n            ),\n            (\n                &[\n                    \"Œ\", // U+0152: LATIN CAPITAL LIGATURE OE\n                    \"ɶ\", // U+0276: LATIN LETTER SMALL CAPITAL OE\n                ],\n                \"OE\",\n            ),\n            (\n                &[\n                    \"Ꝏ\", // U+A74E: LATIN CAPITAL LETTER OO\n                ],\n                \"OO\",\n            ),\n            (\n                &[\n                    \"Ȣ\", // U+0222: LATIN CAPITAL LETTER OU\n                    \"ᴕ\", // U+1D15: LATIN LETTER SMALL CAPITAL OU\n                ],\n                \"OU\",\n            ),\n            (\n                &[\n                    \"⒪\", // U+24AA: PARENTHESIZED LATIN SMALL LETTER O\n                ],\n                \"(o)\",\n            ),\n            (\n                &[\n                    \"œ\", // U+0153: LATIN SMALL LIGATURE OE\n                    \"ᴔ\", // U+1D14: LATIN SMALL LETTER TURNED OE\n                ],\n                \"oe\",\n            ),\n            (\n                &[\n                    \"ꝏ\", // U+A74F: LATIN SMALL LETTER OO\n                ],\n                \"oo\",\n            ),\n            (\n                &[\n                    \"ȣ\", // U+0223: LATIN SMALL LETTER OU\n                ],\n                \"ou\",\n            ),\n            (\n                &[\n                    \"Ƥ\",  // U+01A4: LATIN CAPITAL LETTER P WITH HOOK\n                    \"ᴘ\",  // U+1D18: LATIN LETTER SMALL CAPITAL P\n                    \"Ṕ\",  // U+1E54: LATIN CAPITAL LETTER P WITH ACUTE\n                    \"Ṗ\",  // U+1E56: LATIN CAPITAL LETTER P WITH DOT ABOVE\n                    \"Ⓟ\",  // U+24C5: CIRCLED LATIN CAPITAL LETTER P\n                    \"Ᵽ\",  // U+2C63: LATIN CAPITAL LETTER P WITH STROKE\n                    \"Ꝑ\",  // U+A750: LATIN CAPITAL LETTER P WITH STROKE THROUGH DESCENDER\n                    \"Ꝓ\",  // U+A752: LATIN CAPITAL LETTER P WITH FLOURISH\n                    \"Ꝕ\",  // U+A754: LATIN CAPITAL LETTER P WITH SQUIRREL TAIL\n                    \"Ｐ\", // U+FF30: FULLWIDTH LATIN CAPITAL LETTER P\n                ],\n                \"P\",\n            ),\n            (\n                &[\n                    \"ƥ\",  // U+01A5: LATIN SMALL LETTER P WITH HOOK\n                    \"ᵱ\",  // U+1D71: LATIN SMALL LETTER P WITH MIDDLE TILDE\n                    \"ᵽ\",  // U+1D7D: LATIN SMALL LETTER P WITH STROKE\n                    \"ᶈ\",  // U+1D88: LATIN SMALL LETTER P WITH PALATAL HOOK\n                    \"ṕ\",  // U+1E55: LATIN SMALL LETTER P WITH ACUTE\n                    \"ṗ\",  // U+1E57: LATIN SMALL LETTER P WITH DOT ABOVE\n                    \"ⓟ\",  // U+24DF: CIRCLED LATIN SMALL LETTER P\n                    \"ꝑ\",  // U+A751: LATIN SMALL LETTER P WITH STROKE THROUGH DESCENDER\n                    \"ꝓ\",  // U+A753: LATIN SMALL LETTER P WITH FLOURISH\n                    \"ꝕ\",  // U+A755: LATIN SMALL LETTER P WITH SQUIRREL TAIL\n                    \"ꟼ\",  // U+A7FC: LATIN EPIGRAPHIC LETTER REVERSED P\n                    \"ｐ\", // U+FF50: FULLWIDTH LATIN SMALL LETTER P\n                ],\n                \"p\",\n            ),\n            (\n                &[\n                    \"⒫\", // U+24AB: PARENTHESIZED LATIN SMALL LETTER P\n                ],\n                \"(p)\",\n            ),\n            (\n                &[\n                    \"Ɋ\",  // U+024A: LATIN CAPITAL LETTER SMALL Q WITH HOOK TAIL\n                    \"Ⓠ\",  // U+24C6: CIRCLED LATIN CAPITAL LETTER Q\n                    \"Ꝗ\",  // U+A756: LATIN CAPITAL LETTER Q WITH STROKE THROUGH DESCENDER\n                    \"Ꝙ\",  // U+A758: LATIN CAPITAL LETTER Q WITH DIAGONAL STROKE\n                    \"Ｑ\", // U+FF31: FULLWIDTH LATIN CAPITAL LETTER Q\n                ],\n                \"Q\",\n            ),\n            (\n                &[\n                    \"ĸ\",  // U+0138: LATIN SMALL LETTER KRA\n                    \"ɋ\",  // U+024B: LATIN SMALL LETTER Q WITH HOOK TAIL\n                    \"ʠ\",  // U+02A0: LATIN SMALL LETTER Q WITH HOOK\n                    \"ⓠ\",  // U+24E0: CIRCLED LATIN SMALL LETTER Q\n                    \"ꝗ\",  // U+A757: LATIN SMALL LETTER Q WITH STROKE THROUGH DESCENDER\n                    \"ꝙ\",  // U+A759: LATIN SMALL LETTER Q WITH DIAGONAL STROKE\n                    \"ｑ\", // U+FF51: FULLWIDTH LATIN SMALL LETTER Q\n                ],\n                \"q\",\n            ),\n            (\n                &[\n                    \"⒬\", // U+24AC: PARENTHESIZED LATIN SMALL LETTER Q\n                ],\n                \"(q)\",\n            ),\n            (\n                &[\n                    \"ȹ\", // U+0239: LATIN SMALL LETTER QP DIGRAPH\n                ],\n                \"qp\",\n            ),\n            (\n                &[\n                    \"Ŕ\",  // U+0154: LATIN CAPITAL LETTER R WITH ACUTE\n                    \"Ŗ\",  // U+0156: LATIN CAPITAL LETTER R WITH CEDILLA\n                    \"Ř\",  // U+0158: LATIN CAPITAL LETTER R WITH CARON\n                    \"Ȑ\",  // U+0210: LATIN CAPITAL LETTER R WITH DOUBLE GRAVE\n                    \"Ȓ\",  // U+0212: LATIN CAPITAL LETTER R WITH INVERTED BREVE\n                    \"Ɍ\",  // U+024C: LATIN CAPITAL LETTER R WITH STROKE\n                    \"ʀ\",  // U+0280: LATIN LETTER SMALL CAPITAL R\n                    \"ʁ\",  // U+0281: LATIN LETTER SMALL CAPITAL INVERTED R\n                    \"ᴙ\",  // U+1D19: LATIN LETTER SMALL CAPITAL REVERSED R\n                    \"ᴚ\",  // U+1D1A: LATIN LETTER SMALL CAPITAL TURNED R\n                    \"Ṙ\",  // U+1E58: LATIN CAPITAL LETTER R WITH DOT ABOVE\n                    \"Ṛ\",  // U+1E5A: LATIN CAPITAL LETTER R WITH DOT BELOW\n                    \"Ṝ\",  // U+1E5C: LATIN CAPITAL LETTER R WITH DOT BELOW AND MACRON\n                    \"Ṟ\",  // U+1E5E: LATIN CAPITAL LETTER R WITH LINE BELOW\n                    \"Ⓡ\",  // U+24C7: CIRCLED LATIN CAPITAL LETTER R\n                    \"Ɽ\",  // U+2C64: LATIN CAPITAL LETTER R WITH TAIL\n                    \"Ꝛ\",  // U+A75A: LATIN CAPITAL LETTER R ROTUNDA\n                    \"Ꞃ\",  // U+A782: LATIN CAPITAL LETTER INSULAR R\n                    \"Ｒ\", // U+FF32: FULLWIDTH LATIN CAPITAL LETTER R\n                ],\n                \"R\",\n            ),\n            (\n                &[\n                    \"ŕ\",  // U+0155: LATIN SMALL LETTER R WITH ACUTE\n                    \"ŗ\",  // U+0157: LATIN SMALL LETTER R WITH CEDILLA\n                    \"ř\",  // U+0159: LATIN SMALL LETTER R WITH CARON\n                    \"ȑ\",  // U+0211: LATIN SMALL LETTER R WITH DOUBLE GRAVE\n                    \"ȓ\",  // U+0213: LATIN SMALL LETTER R WITH INVERTED BREVE\n                    \"ɍ\",  // U+024D: LATIN SMALL LETTER R WITH STROKE\n                    \"ɼ\",  // U+027C: LATIN SMALL LETTER R WITH LONG LEG\n                    \"ɽ\",  // U+027D: LATIN SMALL LETTER R WITH TAIL\n                    \"ɾ\",  // U+027E: LATIN SMALL LETTER R WITH FISHHOOK\n                    \"ɿ\",  // U+027F: LATIN SMALL LETTER REVERSED R WITH FISHHOOK\n                    \"ᵣ\",  // U+1D63: LATIN SUBSCRIPT SMALL LETTER R\n                    \"ᵲ\",  // U+1D72: LATIN SMALL LETTER R WITH MIDDLE TILDE\n                    \"ᵳ\",  // U+1D73: LATIN SMALL LETTER R WITH FISHHOOK AND MIDDLE TILDE\n                    \"ᶉ\",  // U+1D89: LATIN SMALL LETTER R WITH PALATAL HOOK\n                    \"ṙ\",  // U+1E59: LATIN SMALL LETTER R WITH DOT ABOVE\n                    \"ṛ\",  // U+1E5B: LATIN SMALL LETTER R WITH DOT BELOW\n                    \"ṝ\",  // U+1E5D: LATIN SMALL LETTER R WITH DOT BELOW AND MACRON\n                    \"ṟ\",  // U+1E5F: LATIN SMALL LETTER R WITH LINE BELOW\n                    \"ⓡ\",  // U+24E1: CIRCLED LATIN SMALL LETTER R\n                    \"ꝛ\",  // U+A75B: LATIN SMALL LETTER R ROTUNDA\n                    \"ꞃ\",  // U+A783: LATIN SMALL LETTER INSULAR R\n                    \"ｒ\", // U+FF52: FULLWIDTH LATIN SMALL LETTER R\n                ],\n                \"r\",\n            ),\n            (\n                &[\n                    \"⒭\", // U+24AD: PARENTHESIZED LATIN SMALL LETTER R\n                ],\n                \"(r)\",\n            ),\n            (\n                &[\n                    \"Ś\",  // U+015A: LATIN CAPITAL LETTER S WITH ACUTE\n                    \"Ŝ\",  // U+015C: LATIN CAPITAL LETTER S WITH CIRCUMFLEX\n                    \"Ş\",  // U+015E: LATIN CAPITAL LETTER S WITH CEDILLA\n                    \"Š\",  // U+0160: LATIN CAPITAL LETTER S WITH CARON\n                    \"Ș\",  // U+0218: LATIN CAPITAL LETTER S WITH COMMA BELOW\n                    \"Ṡ\",  // U+1E60: LATIN CAPITAL LETTER S WITH DOT ABOVE\n                    \"Ṣ\",  // U+1E62: LATIN CAPITAL LETTER S WITH DOT BELOW\n                    \"Ṥ\",  // U+1E64: LATIN CAPITAL LETTER S WITH ACUTE AND DOT ABOVE\n                    \"Ṧ\",  // U+1E66: LATIN CAPITAL LETTER S WITH CARON AND DOT ABOVE\n                    \"Ṩ\",  // U+1E68: LATIN CAPITAL LETTER S WITH DOT BELOW AND DOT ABOVE\n                    \"Ⓢ\",  // U+24C8: CIRCLED LATIN CAPITAL LETTER S\n                    \"ꜱ\",  // U+A731: LATIN LETTER SMALL CAPITAL S\n                    \"ꞅ\",  // U+A785: LATIN SMALL LETTER INSULAR S\n                    \"Ｓ\", // U+FF33: FULLWIDTH LATIN CAPITAL LETTER S\n                ],\n                \"S\",\n            ),\n            (\n                &[\n                    \"ś\",  // U+015B: LATIN SMALL LETTER S WITH ACUTE\n                    \"ŝ\",  // U+015D: LATIN SMALL LETTER S WITH CIRCUMFLEX\n                    \"ş\",  // U+015F: LATIN SMALL LETTER S WITH CEDILLA\n                    \"š\",  // U+0161: LATIN SMALL LETTER S WITH CARON\n                    \"ſ\",  // U+017F: LATIN SMALL LETTER LONG S\n                    \"ș\",  // U+0219: LATIN SMALL LETTER S WITH COMMA BELOW\n                    \"ȿ\",  // U+023F: LATIN SMALL LETTER S WITH SWASH TAIL\n                    \"ʂ\",  // U+0282: LATIN SMALL LETTER S WITH HOOK\n                    \"ᵴ\",  // U+1D74: LATIN SMALL LETTER S WITH MIDDLE TILDE\n                    \"ᶊ\",  // U+1D8A: LATIN SMALL LETTER S WITH PALATAL HOOK\n                    \"ṡ\",  // U+1E61: LATIN SMALL LETTER S WITH DOT ABOVE\n                    \"ṣ\",  // U+1E63: LATIN SMALL LETTER S WITH DOT BELOW\n                    \"ṥ\",  // U+1E65: LATIN SMALL LETTER S WITH ACUTE AND DOT ABOVE\n                    \"ṧ\",  // U+1E67: LATIN SMALL LETTER S WITH CARON AND DOT ABOVE\n                    \"ṩ\",  // U+1E69: LATIN SMALL LETTER S WITH DOT BELOW AND DOT ABOVE\n                    \"ẜ\",  // U+1E9C: LATIN SMALL LETTER LONG S WITH DIAGONAL STROKE\n                    \"ẝ\",  // U+1E9D: LATIN SMALL LETTER LONG S WITH HIGH STROKE\n                    \"ⓢ\",  // U+24E2: CIRCLED LATIN SMALL LETTER S\n                    \"Ꞅ\",  // U+A784: LATIN CAPITAL LETTER INSULAR S\n                    \"ｓ\", // U+FF53: FULLWIDTH LATIN SMALL LETTER S\n                ],\n                \"s\",\n            ),\n            (\n                &[\n                    \"ẞ\", // U+1E9E: LATIN CAPITAL LETTER SHARP S\n                ],\n                \"SS\",\n            ),\n            (\n                &[\n                    \"⒮\", // U+24AE: PARENTHESIZED LATIN SMALL LETTER S\n                ],\n                \"(s)\",\n            ),\n            (\n                &[\n                    \"ß\", // U+00DF: LATIN SMALL LETTER SHARP S\n                ],\n                \"ss\",\n            ),\n            (\n                &[\n                    \"st\", // U+FB06: LATIN SMALL LIGATURE ST\n                ],\n                \"st\",\n            ),\n            (\n                &[\n                    \"Ţ\",  // U+0162: LATIN CAPITAL LETTER T WITH CEDILLA\n                    \"Ť\",  // U+0164: LATIN CAPITAL LETTER T WITH CARON\n                    \"Ŧ\",  // U+0166: LATIN CAPITAL LETTER T WITH STROKE\n                    \"Ƭ\",  // U+01AC: LATIN CAPITAL LETTER T WITH HOOK\n                    \"Ʈ\",  // U+01AE: LATIN CAPITAL LETTER T WITH RETROFLEX HOOK\n                    \"Ț\",  // U+021A: LATIN CAPITAL LETTER T WITH COMMA BELOW\n                    \"Ⱦ\",  // U+023E: LATIN CAPITAL LETTER T WITH DIAGONAL STROKE\n                    \"ᴛ\",  // U+1D1B: LATIN LETTER SMALL CAPITAL T\n                    \"Ṫ\",  // U+1E6A: LATIN CAPITAL LETTER T WITH DOT ABOVE\n                    \"Ṭ\",  // U+1E6C: LATIN CAPITAL LETTER T WITH DOT BELOW\n                    \"Ṯ\",  // U+1E6E: LATIN CAPITAL LETTER T WITH LINE BELOW\n                    \"Ṱ\",  // U+1E70: LATIN CAPITAL LETTER T WITH CIRCUMFLEX BELOW\n                    \"Ⓣ\",  // U+24C9: CIRCLED LATIN CAPITAL LETTER T\n                    \"Ꞇ\",  // U+A786: LATIN CAPITAL LETTER INSULAR T\n                    \"Ｔ\", // U+FF34: FULLWIDTH LATIN CAPITAL LETTER T\n                ],\n                \"T\",\n            ),\n            (\n                &[\n                    \"ţ\",  // U+0163: LATIN SMALL LETTER T WITH CEDILLA\n                    \"ť\",  // U+0165: LATIN SMALL LETTER T WITH CARON\n                    \"ŧ\",  // U+0167: LATIN SMALL LETTER T WITH STROKE\n                    \"ƫ\",  // U+01AB: LATIN SMALL LETTER T WITH PALATAL HOOK\n                    \"ƭ\",  // U+01AD: LATIN SMALL LETTER T WITH HOOK\n                    \"ț\",  // U+021B: LATIN SMALL LETTER T WITH COMMA BELOW\n                    \"ȶ\",  // U+0236: LATIN SMALL LETTER T WITH CURL\n                    \"ʇ\",  // U+0287: LATIN SMALL LETTER TURNED T\n                    \"ʈ\",  // U+0288: LATIN SMALL LETTER T WITH RETROFLEX HOOK\n                    \"ᵵ\",  // U+1D75: LATIN SMALL LETTER T WITH MIDDLE TILDE\n                    \"ṫ\",  // U+1E6B: LATIN SMALL LETTER T WITH DOT ABOVE\n                    \"ṭ\",  // U+1E6D: LATIN SMALL LETTER T WITH DOT BELOW\n                    \"ṯ\",  // U+1E6F: LATIN SMALL LETTER T WITH LINE BELOW\n                    \"ṱ\",  // U+1E71: LATIN SMALL LETTER T WITH CIRCUMFLEX BELOW\n                    \"ẗ\",  // U+1E97: LATIN SMALL LETTER T WITH DIAERESIS\n                    \"ⓣ\",  // U+24E3: CIRCLED LATIN SMALL LETTER T\n                    \"ⱦ\",  // U+2C66: LATIN SMALL LETTER T WITH DIAGONAL STROKE\n                    \"ｔ\", // U+FF54: FULLWIDTH LATIN SMALL LETTER T\n                ],\n                \"t\",\n            ),\n            (\n                &[\n                    \"Þ\", // U+00DE: LATIN CAPITAL LETTER THORN\n                    \"Ꝧ\", // U+A766: LATIN CAPITAL LETTER THORN WITH STROKE THROUGH DESCENDER\n                ],\n                \"TH\",\n            ),\n            (\n                &[\n                    \"Ꜩ\", // U+A728: LATIN CAPITAL LETTER TZ\n                ],\n                \"TZ\",\n            ),\n            (\n                &[\n                    \"⒯\", // U+24AF: PARENTHESIZED LATIN SMALL LETTER T\n                ],\n                \"(t)\",\n            ),\n            (\n                &[\n                    \"ʨ\", // U+02A8: LATIN SMALL LETTER TC DIGRAPH WITH CURL\n                ],\n                \"tc\",\n            ),\n            (\n                &[\n                    \"þ\", // U+00FE: LATIN SMALL LETTER THORN\n                    \"ᵺ\", // U+1D7A: LATIN SMALL LETTER TH WITH STRIKETHROUGH\n                    \"ꝧ\", // U+A767: LATIN SMALL LETTER THORN WITH STROKE THROUGH DESCENDER\n                ],\n                \"th\",\n            ),\n            (\n                &[\n                    \"ʦ\", // U+02A6: LATIN SMALL LETTER TS DIGRAPH\n                ],\n                \"ts\",\n            ),\n            (\n                &[\n                    \"ꜩ\", // U+A729: LATIN SMALL LETTER TZ\n                ],\n                \"tz\",\n            ),\n            (\n                &[\n                    \"Ù\",  // U+00D9: LATIN CAPITAL LETTER U WITH GRAVE\n                    \"Ú\",  // U+00DA: LATIN CAPITAL LETTER U WITH ACUTE\n                    \"Û\",  // U+00DB: LATIN CAPITAL LETTER U WITH CIRCUMFLEX\n                    \"Ü\",  // U+00DC: LATIN CAPITAL LETTER U WITH DIAERESIS\n                    \"Ũ\",  // U+0168: LATIN CAPITAL LETTER U WITH TILDE\n                    \"Ū\",  // U+016A: LATIN CAPITAL LETTER U WITH MACRON\n                    \"Ŭ\",  // U+016C: LATIN CAPITAL LETTER U WITH BREVE\n                    \"Ů\",  // U+016E: LATIN CAPITAL LETTER U WITH RING ABOVE\n                    \"Ű\",  // U+0170: LATIN CAPITAL LETTER U WITH DOUBLE ACUTE\n                    \"Ų\",  // U+0172: LATIN CAPITAL LETTER U WITH OGONEK\n                    \"Ư\",  // U+01AF: LATIN CAPITAL LETTER U WITH HORN\n                    \"Ǔ\",  // U+01D3: LATIN CAPITAL LETTER U WITH CARON\n                    \"Ǖ\",  // U+01D5: LATIN CAPITAL LETTER U WITH DIAERESIS AND MACRON\n                    \"Ǘ\",  // U+01D7: LATIN CAPITAL LETTER U WITH DIAERESIS AND ACUTE\n                    \"Ǚ\",  // U+01D9: LATIN CAPITAL LETTER U WITH DIAERESIS AND CARON\n                    \"Ǜ\",  // U+01DB: LATIN CAPITAL LETTER U WITH DIAERESIS AND GRAVE\n                    \"Ȕ\",  // U+0214: LATIN CAPITAL LETTER U WITH DOUBLE GRAVE\n                    \"Ȗ\",  // U+0216: LATIN CAPITAL LETTER U WITH INVERTED BREVE\n                    \"Ʉ\",  // U+0244: LATIN CAPITAL LETTER U BAR\n                    \"ᴜ\",  // U+1D1C: LATIN LETTER SMALL CAPITAL U\n                    \"ᵾ\",  // U+1D7E: LATIN SMALL CAPITAL LETTER U WITH STROKE\n                    \"Ṳ\",  // U+1E72: LATIN CAPITAL LETTER U WITH DIAERESIS BELOW\n                    \"Ṵ\",  // U+1E74: LATIN CAPITAL LETTER U WITH TILDE BELOW\n                    \"Ṷ\",  // U+1E76: LATIN CAPITAL LETTER U WITH CIRCUMFLEX BELOW\n                    \"Ṹ\",  // U+1E78: LATIN CAPITAL LETTER U WITH TILDE AND ACUTE\n                    \"Ṻ\",  // U+1E7A: LATIN CAPITAL LETTER U WITH MACRON AND DIAERESIS\n                    \"Ụ\",  // U+1EE4: LATIN CAPITAL LETTER U WITH DOT BELOW\n                    \"Ủ\",  // U+1EE6: LATIN CAPITAL LETTER U WITH HOOK ABOVE\n                    \"Ứ\",  // U+1EE8: LATIN CAPITAL LETTER U WITH HORN AND ACUTE\n                    \"Ừ\",  // U+1EEA: LATIN CAPITAL LETTER U WITH HORN AND GRAVE\n                    \"Ử\",  // U+1EEC: LATIN CAPITAL LETTER U WITH HORN AND HOOK ABOVE\n                    \"Ữ\",  // U+1EEE: LATIN CAPITAL LETTER U WITH HORN AND TILDE\n                    \"Ự\",  // U+1EF0: LATIN CAPITAL LETTER U WITH HORN AND DOT BELOW\n                    \"Ⓤ\",  // U+24CA: CIRCLED LATIN CAPITAL LETTER U\n                    \"Ｕ\", // U+FF35: FULLWIDTH LATIN CAPITAL LETTER U\n                ],\n                \"U\",\n            ),\n            (\n                &[\n                    \"ù\",  // U+00F9: LATIN SMALL LETTER U WITH GRAVE\n                    \"ú\",  // U+00FA: LATIN SMALL LETTER U WITH ACUTE\n                    \"û\",  // U+00FB: LATIN SMALL LETTER U WITH CIRCUMFLEX\n                    \"ü\",  // U+00FC: LATIN SMALL LETTER U WITH DIAERESIS\n                    \"ũ\",  // U+0169: LATIN SMALL LETTER U WITH TILDE\n                    \"ū\",  // U+016B: LATIN SMALL LETTER U WITH MACRON\n                    \"ŭ\",  // U+016D: LATIN SMALL LETTER U WITH BREVE\n                    \"ů\",  // U+016F: LATIN SMALL LETTER U WITH RING ABOVE\n                    \"ű\",  // U+0171: LATIN SMALL LETTER U WITH DOUBLE ACUTE\n                    \"ų\",  // U+0173: LATIN SMALL LETTER U WITH OGONEK\n                    \"ư\",  // U+01B0: LATIN SMALL LETTER U WITH HORN\n                    \"ǔ\",  // U+01D4: LATIN SMALL LETTER U WITH CARON\n                    \"ǖ\",  // U+01D6: LATIN SMALL LETTER U WITH DIAERESIS AND MACRON\n                    \"ǘ\",  // U+01D8: LATIN SMALL LETTER U WITH DIAERESIS AND ACUTE\n                    \"ǚ\",  // U+01DA: LATIN SMALL LETTER U WITH DIAERESIS AND CARON\n                    \"ǜ\",  // U+01DC: LATIN SMALL LETTER U WITH DIAERESIS AND GRAVE\n                    \"ȕ\",  // U+0215: LATIN SMALL LETTER U WITH DOUBLE GRAVE\n                    \"ȗ\",  // U+0217: LATIN SMALL LETTER U WITH INVERTED BREVE\n                    \"ʉ\",  // U+0289: LATIN SMALL LETTER U BAR\n                    \"ᵤ\",  // U+1D64: LATIN SUBSCRIPT SMALL LETTER U\n                    \"ᶙ\",  // U+1D99: LATIN SMALL LETTER U WITH RETROFLEX HOOK\n                    \"ṳ\",  // U+1E73: LATIN SMALL LETTER U WITH DIAERESIS BELOW\n                    \"ṵ\",  // U+1E75: LATIN SMALL LETTER U WITH TILDE BELOW\n                    \"ṷ\",  // U+1E77: LATIN SMALL LETTER U WITH CIRCUMFLEX BELOW\n                    \"ṹ\",  // U+1E79: LATIN SMALL LETTER U WITH TILDE AND ACUTE\n                    \"ṻ\",  // U+1E7B: LATIN SMALL LETTER U WITH MACRON AND DIAERESIS\n                    \"ụ\",  // U+1EE5: LATIN SMALL LETTER U WITH DOT BELOW\n                    \"ủ\",  // U+1EE7: LATIN SMALL LETTER U WITH HOOK ABOVE\n                    \"ứ\",  // U+1EE9: LATIN SMALL LETTER U WITH HORN AND ACUTE\n                    \"ừ\",  // U+1EEB: LATIN SMALL LETTER U WITH HORN AND GRAVE\n                    \"ử\",  // U+1EED: LATIN SMALL LETTER U WITH HORN AND HOOK ABOVE\n                    \"ữ\",  // U+1EEF: LATIN SMALL LETTER U WITH HORN AND TILDE\n                    \"ự\",  // U+1EF1: LATIN SMALL LETTER U WITH HORN AND DOT BELOW\n                    \"ⓤ\",  // U+24E4: CIRCLED LATIN SMALL LETTER U\n                    \"ｕ\", // U+FF55: FULLWIDTH LATIN SMALL LETTER U\n                ],\n                \"u\",\n            ),\n            (\n                &[\n                    \"⒰\", // U+24B0: PARENTHESIZED LATIN SMALL LETTER U\n                ],\n                \"(u)\",\n            ),\n            (\n                &[\n                    \"ᵫ\", // U+1D6B: LATIN SMALL LETTER UE\n                ],\n                \"ue\",\n            ),\n            (\n                &[\n                    \"Ʋ\",  // U+01B2: LATIN CAPITAL LETTER V WITH HOOK\n                    \"Ʌ\",  // U+0245: LATIN CAPITAL LETTER TURNED V\n                    \"ᴠ\",  // U+1D20: LATIN LETTER SMALL CAPITAL V\n                    \"Ṽ\",  // U+1E7C: LATIN CAPITAL LETTER V WITH TILDE\n                    \"Ṿ\",  // U+1E7E: LATIN CAPITAL LETTER V WITH DOT BELOW\n                    \"Ỽ\",  // U+1EFC: LATIN CAPITAL LETTER MIDDLE-WELSH V\n                    \"Ⓥ\",  // U+24CB: CIRCLED LATIN CAPITAL LETTER V\n                    \"Ꝟ\",  // U+A75E: LATIN CAPITAL LETTER V WITH DIAGONAL STROKE\n                    \"Ꝩ\",  // U+A768: LATIN CAPITAL LETTER VEND\n                    \"Ｖ\", // U+FF36: FULLWIDTH LATIN CAPITAL LETTER V\n                ],\n                \"V\",\n            ),\n            (\n                &[\n                    \"ʋ\",  // U+028B: LATIN SMALL LETTER V WITH HOOK\n                    \"ʌ\",  // U+028C: LATIN SMALL LETTER TURNED V\n                    \"ᵥ\",  // U+1D65: LATIN SUBSCRIPT SMALL LETTER V\n                    \"ᶌ\",  // U+1D8C: LATIN SMALL LETTER V WITH PALATAL HOOK\n                    \"ṽ\",  // U+1E7D: LATIN SMALL LETTER V WITH TILDE\n                    \"ṿ\",  // U+1E7F: LATIN SMALL LETTER V WITH DOT BELOW\n                    \"ⓥ\",  // U+24E5: CIRCLED LATIN SMALL LETTER V\n                    \"ⱱ\",  // U+2C71: LATIN SMALL LETTER V WITH RIGHT HOOK\n                    \"ⱴ\",  // U+2C74: LATIN SMALL LETTER V WITH CURL\n                    \"ꝟ\",  // U+A75F: LATIN SMALL LETTER V WITH DIAGONAL STROKE\n                    \"ｖ\", // U+FF56: FULLWIDTH LATIN SMALL LETTER V\n                ],\n                \"v\",\n            ),\n            (\n                &[\n                    \"Ꝡ\", // U+A760: LATIN CAPITAL LETTER VY\n                ],\n                \"VY\",\n            ),\n            (\n                &[\n                    \"⒱\", // U+24B1: PARENTHESIZED LATIN SMALL LETTER V\n                ],\n                \"(v)\",\n            ),\n            (\n                &[\n                    \"ꝡ\", // U+A761: LATIN SMALL LETTER VY\n                ],\n                \"vy\",\n            ),\n            (\n                &[\n                    \"Ŵ\",  // U+0174: LATIN CAPITAL LETTER W WITH CIRCUMFLEX\n                    \"Ƿ\",  // U+01F7: LATIN CAPITAL LETTER WYNN\n                    \"ᴡ\",  // U+1D21: LATIN LETTER SMALL CAPITAL W\n                    \"Ẁ\",  // U+1E80: LATIN CAPITAL LETTER W WITH GRAVE\n                    \"Ẃ\",  // U+1E82: LATIN CAPITAL LETTER W WITH ACUTE\n                    \"Ẅ\",  // U+1E84: LATIN CAPITAL LETTER W WITH DIAERESIS\n                    \"Ẇ\",  // U+1E86: LATIN CAPITAL LETTER W WITH DOT ABOVE\n                    \"Ẉ\",  // U+1E88: LATIN CAPITAL LETTER W WITH DOT BELOW\n                    \"Ⓦ\",  // U+24CC: CIRCLED LATIN CAPITAL LETTER W\n                    \"Ⱳ\",  // U+2C72: LATIN CAPITAL LETTER W WITH HOOK\n                    \"Ｗ\", // U+FF37: FULLWIDTH LATIN CAPITAL LETTER W\n                ],\n                \"W\",\n            ),\n            (\n                &[\n                    \"ŵ\",  // U+0175: LATIN SMALL LETTER W WITH CIRCUMFLEX\n                    \"ƿ\",  // U+01BF: LATIN LETTER WYNN\n                    \"ʍ\",  // U+028D: LATIN SMALL LETTER TURNED W\n                    \"ẁ\",  // U+1E81: LATIN SMALL LETTER W WITH GRAVE\n                    \"ẃ\",  // U+1E83: LATIN SMALL LETTER W WITH ACUTE\n                    \"ẅ\",  // U+1E85: LATIN SMALL LETTER W WITH DIAERESIS\n                    \"ẇ\",  // U+1E87: LATIN SMALL LETTER W WITH DOT ABOVE\n                    \"ẉ\",  // U+1E89: LATIN SMALL LETTER W WITH DOT BELOW\n                    \"ẘ\",  // U+1E98: LATIN SMALL LETTER W WITH RING ABOVE\n                    \"ⓦ\",  // U+24E6: CIRCLED LATIN SMALL LETTER W\n                    \"ⱳ\",  // U+2C73: LATIN SMALL LETTER W WITH HOOK\n                    \"ｗ\", // U+FF57: FULLWIDTH LATIN SMALL LETTER W\n                ],\n                \"w\",\n            ),\n            (\n                &[\n                    \"⒲\", // U+24B2: PARENTHESIZED LATIN SMALL LETTER W\n                ],\n                \"(w)\",\n            ),\n            (\n                &[\n                    \"Ẋ\",  // U+1E8A: LATIN CAPITAL LETTER X WITH DOT ABOVE\n                    \"Ẍ\",  // U+1E8C: LATIN CAPITAL LETTER X WITH DIAERESIS\n                    \"Ⓧ\",  // U+24CD: CIRCLED LATIN CAPITAL LETTER X\n                    \"Ｘ\", // U+FF38: FULLWIDTH LATIN CAPITAL LETTER X\n                ],\n                \"X\",\n            ),\n            (\n                &[\n                    \"ᶍ\",  // U+1D8D: LATIN SMALL LETTER X WITH PALATAL HOOK\n                    \"ẋ\",  // U+1E8B: LATIN SMALL LETTER X WITH DOT ABOVE\n                    \"ẍ\",  // U+1E8D: LATIN SMALL LETTER X WITH DIAERESIS\n                    \"ₓ\",  // U+2093: LATIN SUBSCRIPT SMALL LETTER X\n                    \"ⓧ\",  // U+24E7: CIRCLED LATIN SMALL LETTER X\n                    \"ｘ\", // U+FF58: FULLWIDTH LATIN SMALL LETTER X\n                ],\n                \"x\",\n            ),\n            (\n                &[\n                    \"⒳\", // U+24B3: PARENTHESIZED LATIN SMALL LETTER X\n                ],\n                \"(x)\",\n            ),\n            (\n                &[\n                    \"Ý\",  // U+00DD: LATIN CAPITAL LETTER Y WITH ACUTE\n                    \"Ŷ\",  // U+0176: LATIN CAPITAL LETTER Y WITH CIRCUMFLEX\n                    \"Ÿ\",  // U+0178: LATIN CAPITAL LETTER Y WITH DIAERESIS\n                    \"Ƴ\",  // U+01B3: LATIN CAPITAL LETTER Y WITH HOOK\n                    \"Ȳ\",  // U+0232: LATIN CAPITAL LETTER Y WITH MACRON\n                    \"Ɏ\",  // U+024E: LATIN CAPITAL LETTER Y WITH STROKE\n                    \"ʏ\",  // U+028F: LATIN LETTER SMALL CAPITAL Y\n                    \"Ẏ\",  // U+1E8E: LATIN CAPITAL LETTER Y WITH DOT ABOVE\n                    \"Ỳ\",  // U+1EF2: LATIN CAPITAL LETTER Y WITH GRAVE\n                    \"Ỵ\",  // U+1EF4: LATIN CAPITAL LETTER Y WITH DOT BELOW\n                    \"Ỷ\",  // U+1EF6: LATIN CAPITAL LETTER Y WITH HOOK ABOVE\n                    \"Ỹ\",  // U+1EF8: LATIN CAPITAL LETTER Y WITH TILDE\n                    \"Ỿ\",  // U+1EFE: LATIN CAPITAL LETTER Y WITH LOOP\n                    \"Ⓨ\",  // U+24CE: CIRCLED LATIN CAPITAL LETTER Y\n                    \"Ｙ\", // U+FF39: FULLWIDTH LATIN CAPITAL LETTER Y\n                ],\n                \"Y\",\n            ),\n            (\n                &[\n                    \"ý\",  // U+00FD: LATIN SMALL LETTER Y WITH ACUTE\n                    \"ÿ\",  // U+00FF: LATIN SMALL LETTER Y WITH DIAERESIS\n                    \"ŷ\",  // U+0177: LATIN SMALL LETTER Y WITH CIRCUMFLEX\n                    \"ƴ\",  // U+01B4: LATIN SMALL LETTER Y WITH HOOK\n                    \"ȳ\",  // U+0233: LATIN SMALL LETTER Y WITH MACRON\n                    \"ɏ\",  // U+024F: LATIN SMALL LETTER Y WITH STROKE\n                    \"ʎ\",  // U+028E: LATIN SMALL LETTER TURNED Y\n                    \"ẏ\",  // U+1E8F: LATIN SMALL LETTER Y WITH DOT ABOVE\n                    \"ẙ\",  // U+1E99: LATIN SMALL LETTER Y WITH RING ABOVE\n                    \"ỳ\",  // U+1EF3: LATIN SMALL LETTER Y WITH GRAVE\n                    \"ỵ\",  // U+1EF5: LATIN SMALL LETTER Y WITH DOT BELOW\n                    \"ỷ\",  // U+1EF7: LATIN SMALL LETTER Y WITH HOOK ABOVE\n                    \"ỹ\",  // U+1EF9: LATIN SMALL LETTER Y WITH TILDE\n                    \"ỿ\",  // U+1EFF: LATIN SMALL LETTER Y WITH LOOP\n                    \"ⓨ\",  // U+24E8: CIRCLED LATIN SMALL LETTER Y\n                    \"ｙ\", // U+FF59: FULLWIDTH LATIN SMALL LETTER Y\n                ],\n                \"y\",\n            ),\n            (\n                &[\n                    \"⒴\", // U+24B4: PARENTHESIZED LATIN SMALL LETTER Y\n                ],\n                \"(y)\",\n            ),\n            (\n                &[\n                    \"Ź\",  // U+0179: LATIN CAPITAL LETTER Z WITH ACUTE\n                    \"Ż\",  // U+017B: LATIN CAPITAL LETTER Z WITH DOT ABOVE\n                    \"Ž\",  // U+017D: LATIN CAPITAL LETTER Z WITH CARON\n                    \"Ƶ\",  // U+01B5: LATIN CAPITAL LETTER Z WITH STROKE\n                    \"Ȝ\",  // U+021C: LATIN CAPITAL LETTER YOGH\n                    \"Ȥ\",  // U+0224: LATIN CAPITAL LETTER Z WITH HOOK\n                    \"ᴢ\",  // U+1D22: LATIN LETTER SMALL CAPITAL Z\n                    \"Ẑ\",  // U+1E90: LATIN CAPITAL LETTER Z WITH CIRCUMFLEX\n                    \"Ẓ\",  // U+1E92: LATIN CAPITAL LETTER Z WITH DOT BELOW\n                    \"Ẕ\",  // U+1E94: LATIN CAPITAL LETTER Z WITH LINE BELOW\n                    \"Ⓩ\",  // U+24CF: CIRCLED LATIN CAPITAL LETTER Z\n                    \"Ⱬ\",  // U+2C6B: LATIN CAPITAL LETTER Z WITH DESCENDER\n                    \"Ꝣ\",  // U+A762: LATIN CAPITAL LETTER VISIGOTHIC Z\n                    \"Ｚ\", // U+FF3A: FULLWIDTH LATIN CAPITAL LETTER Z\n                ],\n                \"Z\",\n            ),\n            (\n                &[\n                    \"ź\",  // U+017A: LATIN SMALL LETTER Z WITH ACUTE\n                    \"ż\",  // U+017C: LATIN SMALL LETTER Z WITH DOT ABOVE\n                    \"ž\",  // U+017E: LATIN SMALL LETTER Z WITH CARON\n                    \"ƶ\",  // U+01B6: LATIN SMALL LETTER Z WITH STROKE\n                    \"ȝ\",  // U+021D: LATIN SMALL LETTER YOGH\n                    \"ȥ\",  // U+0225: LATIN SMALL LETTER Z WITH HOOK\n                    \"ɀ\",  // U+0240: LATIN SMALL LETTER Z WITH SWASH TAIL\n                    \"ʐ\",  // U+0290: LATIN SMALL LETTER Z WITH RETROFLEX HOOK\n                    \"ʑ\",  // U+0291: LATIN SMALL LETTER Z WITH CURL\n                    \"ᵶ\",  // U+1D76: LATIN SMALL LETTER Z WITH MIDDLE TILDE\n                    \"ᶎ\",  // U+1D8E: LATIN SMALL LETTER Z WITH PALATAL HOOK\n                    \"ẑ\",  // U+1E91: LATIN SMALL LETTER Z WITH CIRCUMFLEX\n                    \"ẓ\",  // U+1E93: LATIN SMALL LETTER Z WITH DOT BELOW\n                    \"ẕ\",  // U+1E95: LATIN SMALL LETTER Z WITH LINE BELOW\n                    \"ⓩ\",  // U+24E9: CIRCLED LATIN SMALL LETTER Z\n                    \"ⱬ\",  // U+2C6C: LATIN SMALL LETTER Z WITH DESCENDER\n                    \"ꝣ\",  // U+A763: LATIN SMALL LETTER VISIGOTHIC Z\n                    \"ｚ\", // U+FF5A: FULLWIDTH LATIN SMALL LETTER Z\n                ],\n                \"z\",\n            ),\n            (\n                &[\n                    \"⒵\", // U+24B5: PARENTHESIZED LATIN SMALL LETTER Z\n                ],\n                \"(z)\",\n            ),\n            (\n                &[\n                    \"⁰\",  // U+2070: SUPERSCRIPT ZERO\n                    \"₀\",  // U+2080: SUBSCRIPT ZERO\n                    \"⓪\",  // U+24EA: CIRCLED DIGIT ZERO\n                    \"⓿\",  // U+24FF: NEGATIVE CIRCLED DIGIT ZERO\n                    \"０\", // U+FF10: FULLWIDTH DIGIT ZERO\n                ],\n                \"0\",\n            ),\n            (\n                &[\n                    \"¹\",  // U+00B9: SUPERSCRIPT ONE\n                    \"₁\",  // U+2081: SUBSCRIPT ONE\n                    \"①\",  // U+2460: CIRCLED DIGIT ONE\n                    \"⓵\",  // U+24F5: DOUBLE CIRCLED DIGIT ONE\n                    \"❶\",  // U+2776: DINGBAT NEGATIVE CIRCLED DIGIT ONE\n                    \"➀\",  // U+2780: DINGBAT CIRCLED SANS-SERIF DIGIT ONE\n                    \"➊\",  // U+278A: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT ONE\n                    \"１\", // U+FF11: FULLWIDTH DIGIT ONE\n                ],\n                \"1\",\n            ),\n            (\n                &[\n                    \"⒈\", // U+2488: DIGIT ONE FULL STOP\n                ],\n                \"1.\",\n            ),\n            (\n                &[\n                    \"⑴\", // U+2474: PARENTHESIZED DIGIT ONE\n                ],\n                \"(1)\",\n            ),\n            (\n                &[\n                    \"²\",  // U+00B2: SUPERSCRIPT TWO\n                    \"₂\",  // U+2082: SUBSCRIPT TWO\n                    \"②\",  // U+2461: CIRCLED DIGIT TWO\n                    \"⓶\",  // U+24F6: DOUBLE CIRCLED DIGIT TWO\n                    \"❷\",  // U+2777: DINGBAT NEGATIVE CIRCLED DIGIT TWO\n                    \"➁\",  // U+2781: DINGBAT CIRCLED SANS-SERIF DIGIT TWO\n                    \"➋\",  // U+278B: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT TWO\n                    \"２\", // U+FF12: FULLWIDTH DIGIT TWO\n                ],\n                \"2\",\n            ),\n            (\n                &[\n                    \"⒉\", // U+2489: DIGIT TWO FULL STOP\n                ],\n                \"2.\",\n            ),\n            (\n                &[\n                    \"⑵\", // U+2475: PARENTHESIZED DIGIT TWO\n                ],\n                \"(2)\",\n            ),\n            (\n                &[\n                    \"³\",  // U+00B3: SUPERSCRIPT THREE\n                    \"₃\",  // U+2083: SUBSCRIPT THREE\n                    \"③\",  // U+2462: CIRCLED DIGIT THREE\n                    \"⓷\",  // U+24F7: DOUBLE CIRCLED DIGIT THREE\n                    \"❸\",  // U+2778: DINGBAT NEGATIVE CIRCLED DIGIT THREE\n                    \"➂\",  // U+2782: DINGBAT CIRCLED SANS-SERIF DIGIT THREE\n                    \"➌\",  // U+278C: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT THREE\n                    \"３\", // U+FF13: FULLWIDTH DIGIT THREE\n                ],\n                \"3\",\n            ),\n            (\n                &[\n                    \"⒊\", // U+248A: DIGIT THREE FULL STOP\n                ],\n                \"3.\",\n            ),\n            (\n                &[\n                    \"⑶\", // U+2476: PARENTHESIZED DIGIT THREE\n                ],\n                \"(3)\",\n            ),\n            (\n                &[\n                    \"⁴\",  // U+2074: SUPERSCRIPT FOUR\n                    \"₄\",  // U+2084: SUBSCRIPT FOUR\n                    \"④\",  // U+2463: CIRCLED DIGIT FOUR\n                    \"⓸\",  // U+24F8: DOUBLE CIRCLED DIGIT FOUR\n                    \"❹\",  // U+2779: DINGBAT NEGATIVE CIRCLED DIGIT FOUR\n                    \"➃\",  // U+2783: DINGBAT CIRCLED SANS-SERIF DIGIT FOUR\n                    \"➍\",  // U+278D: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT FOUR\n                    \"４\", // U+FF14: FULLWIDTH DIGIT FOUR\n                ],\n                \"4\",\n            ),\n            (\n                &[\n                    \"⒋\", // U+248B: DIGIT FOUR FULL STOP\n                ],\n                \"4.\",\n            ),\n            (\n                &[\n                    \"⑷\", // U+2477: PARENTHESIZED DIGIT FOUR\n                ],\n                \"(4)\",\n            ),\n            (\n                &[\n                    \"⁵\",  // U+2075: SUPERSCRIPT FIVE\n                    \"₅\",  // U+2085: SUBSCRIPT FIVE\n                    \"⑤\",  // U+2464: CIRCLED DIGIT FIVE\n                    \"⓹\",  // U+24F9: DOUBLE CIRCLED DIGIT FIVE\n                    \"❺\",  // U+277A: DINGBAT NEGATIVE CIRCLED DIGIT FIVE\n                    \"➄\",  // U+2784: DINGBAT CIRCLED SANS-SERIF DIGIT FIVE\n                    \"➎\",  // U+278E: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT FIVE\n                    \"５\", // U+FF15: FULLWIDTH DIGIT FIVE\n                ],\n                \"5\",\n            ),\n            (\n                &[\n                    \"⒌\", // U+248C: DIGIT FIVE FULL STOP\n                ],\n                \"5.\",\n            ),\n            (\n                &[\n                    \"⑸\", // U+2478: PARENTHESIZED DIGIT FIVE\n                ],\n                \"(5)\",\n            ),\n            (\n                &[\n                    \"⁶\",  // U+2076: SUPERSCRIPT SIX\n                    \"₆\",  // U+2086: SUBSCRIPT SIX\n                    \"⑥\",  // U+2465: CIRCLED DIGIT SIX\n                    \"⓺\",  // U+24FA: DOUBLE CIRCLED DIGIT SIX\n                    \"❻\",  // U+277B: DINGBAT NEGATIVE CIRCLED DIGIT SIX\n                    \"➅\",  // U+2785: DINGBAT CIRCLED SANS-SERIF DIGIT SIX\n                    \"➏\",  // U+278F: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT SIX\n                    \"６\", // U+FF16: FULLWIDTH DIGIT SIX\n                ],\n                \"6\",\n            ),\n            (\n                &[\n                    \"⒍\", // U+248D: DIGIT SIX FULL STOP\n                ],\n                \"6.\",\n            ),\n            (\n                &[\n                    \"⑹\", // U+2479: PARENTHESIZED DIGIT SIX\n                ],\n                \"(6)\",\n            ),\n            (\n                &[\n                    \"⁷\",  // U+2077: SUPERSCRIPT SEVEN\n                    \"₇\",  // U+2087: SUBSCRIPT SEVEN\n                    \"⑦\",  // U+2466: CIRCLED DIGIT SEVEN\n                    \"⓻\",  // U+24FB: DOUBLE CIRCLED DIGIT SEVEN\n                    \"❼\",  // U+277C: DINGBAT NEGATIVE CIRCLED DIGIT SEVEN\n                    \"➆\",  // U+2786: DINGBAT CIRCLED SANS-SERIF DIGIT SEVEN\n                    \"➐\",  // U+2790: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT SEVEN\n                    \"７\", // U+FF17: FULLWIDTH DIGIT SEVEN\n                ],\n                \"7\",\n            ),\n            (\n                &[\n                    \"⒎\", // U+248E: DIGIT SEVEN FULL STOP\n                ],\n                \"7.\",\n            ),\n            (\n                &[\n                    \"⑺\", // U+247A: PARENTHESIZED DIGIT SEVEN\n                ],\n                \"(7)\",\n            ),\n            (\n                &[\n                    \"⁸\",  // U+2078: SUPERSCRIPT EIGHT\n                    \"₈\",  // U+2088: SUBSCRIPT EIGHT\n                    \"⑧\",  // U+2467: CIRCLED DIGIT EIGHT\n                    \"⓼\",  // U+24FC: DOUBLE CIRCLED DIGIT EIGHT\n                    \"❽\",  // U+277D: DINGBAT NEGATIVE CIRCLED DIGIT EIGHT\n                    \"➇\",  // U+2787: DINGBAT CIRCLED SANS-SERIF DIGIT EIGHT\n                    \"➑\",  // U+2791: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT EIGHT\n                    \"８\", // U+FF18: FULLWIDTH DIGIT EIGHT\n                ],\n                \"8\",\n            ),\n            (\n                &[\n                    \"⒏\", // U+248F: DIGIT EIGHT FULL STOP\n                ],\n                \"8.\",\n            ),\n            (\n                &[\n                    \"⑻\", // U+247B: PARENTHESIZED DIGIT EIGHT\n                ],\n                \"(8)\",\n            ),\n            (\n                &[\n                    \"⁹\",  // U+2079: SUPERSCRIPT NINE\n                    \"₉\",  // U+2089: SUBSCRIPT NINE\n                    \"⑨\",  // U+2468: CIRCLED DIGIT NINE\n                    \"⓽\",  // U+24FD: DOUBLE CIRCLED DIGIT NINE\n                    \"❾\",  // U+277E: DINGBAT NEGATIVE CIRCLED DIGIT NINE\n                    \"➈\",  // U+2788: DINGBAT CIRCLED SANS-SERIF DIGIT NINE\n                    \"➒\",  // U+2792: DINGBAT NEGATIVE CIRCLED SANS-SERIF DIGIT NINE\n                    \"９\", // U+FF19: FULLWIDTH DIGIT NINE\n                ],\n                \"9\",\n            ),\n            (\n                &[\n                    \"⒐\", // U+2490: DIGIT NINE FULL STOP\n                ],\n                \"9.\",\n            ),\n            (\n                &[\n                    \"⑼\", // U+247C: PARENTHESIZED DIGIT NINE\n                ],\n                \"(9)\",\n            ),\n            (\n                &[\n                    \"⑩\", // U+2469: CIRCLED NUMBER TEN\n                    \"⓾\", // U+24FE: DOUBLE CIRCLED NUMBER TEN\n                    \"❿\", // U+277F: DINGBAT NEGATIVE CIRCLED NUMBER TEN\n                    \"➉\", // U+2789: DINGBAT CIRCLED SANS-SERIF NUMBER TEN\n                    \"➓\", // U+2793: DINGBAT NEGATIVE CIRCLED SANS-SERIF NUMBER TEN\n                ],\n                \"10\",\n            ),\n            (\n                &[\n                    \"⒑\", // U+2491: NUMBER TEN FULL STOP\n                ],\n                \"10.\",\n            ),\n            (\n                &[\n                    \"⑽\", // U+247D: PARENTHESIZED NUMBER TEN\n                ],\n                \"(10)\",\n            ),\n            (\n                &[\n                    \"⑪\", // U+246A: CIRCLED NUMBER ELEVEN\n                    \"⓫\", // U+24EB: NEGATIVE CIRCLED NUMBER ELEVEN\n                ],\n                \"11\",\n            ),\n            (\n                &[\n                    \"⒒\", // U+2492: NUMBER ELEVEN FULL STOP\n                ],\n                \"11.\",\n            ),\n            (\n                &[\n                    \"⑾\", // U+247E: PARENTHESIZED NUMBER ELEVEN\n                ],\n                \"(11)\",\n            ),\n            (\n                &[\n                    \"⑫\", // U+246B: CIRCLED NUMBER TWELVE\n                    \"⓬\", // U+24EC: NEGATIVE CIRCLED NUMBER TWELVE\n                ],\n                \"12\",\n            ),\n            (\n                &[\n                    \"⒓\", // U+2493: NUMBER TWELVE FULL STOP\n                ],\n                \"12.\",\n            ),\n            (\n                &[\n                    \"⑿\", // U+247F: PARENTHESIZED NUMBER TWELVE\n                ],\n                \"(12)\",\n            ),\n            (\n                &[\n                    \"⑬\", // U+246C: CIRCLED NUMBER THIRTEEN\n                    \"⓭\", // U+24ED: NEGATIVE CIRCLED NUMBER THIRTEEN\n                ],\n                \"13\",\n            ),\n            (\n                &[\n                    \"⒔\", // U+2494: NUMBER THIRTEEN FULL STOP\n                ],\n                \"13.\",\n            ),\n            (\n                &[\n                    \"⒀\", // U+2480: PARENTHESIZED NUMBER THIRTEEN\n                ],\n                \"(13)\",\n            ),\n            (\n                &[\n                    \"⑭\", // U+246D: CIRCLED NUMBER FOURTEEN\n                    \"⓮\", // U+24EE: NEGATIVE CIRCLED NUMBER FOURTEEN\n                ],\n                \"14\",\n            ),\n            (\n                &[\n                    \"⒕\", // U+2495: NUMBER FOURTEEN FULL STOP\n                ],\n                \"14.\",\n            ),\n            (\n                &[\n                    \"⒁\", // U+2481: PARENTHESIZED NUMBER FOURTEEN\n                ],\n                \"(14)\",\n            ),\n            (\n                &[\n                    \"⑮\", // U+246E: CIRCLED NUMBER FIFTEEN\n                    \"⓯\", // U+24EF: NEGATIVE CIRCLED NUMBER FIFTEEN\n                ],\n                \"15\",\n            ),\n            (\n                &[\n                    \"⒖\", // U+2496: NUMBER FIFTEEN FULL STOP\n                ],\n                \"15.\",\n            ),\n            (\n                &[\n                    \"⒂\", // U+2482: PARENTHESIZED NUMBER FIFTEEN\n                ],\n                \"(15)\",\n            ),\n            (\n                &[\n                    \"⑯\", // U+246F: CIRCLED NUMBER SIXTEEN\n                    \"⓰\", // U+24F0: NEGATIVE CIRCLED NUMBER SIXTEEN\n                ],\n                \"16\",\n            ),\n            (\n                &[\n                    \"⒗\", // U+2497: NUMBER SIXTEEN FULL STOP\n                ],\n                \"16.\",\n            ),\n            (\n                &[\n                    \"⒃\", // U+2483: PARENTHESIZED NUMBER SIXTEEN\n                ],\n                \"(16)\",\n            ),\n            (\n                &[\n                    \"⑰\", // U+2470: CIRCLED NUMBER SEVENTEEN\n                    \"⓱\", // U+24F1: NEGATIVE CIRCLED NUMBER SEVENTEEN\n                ],\n                \"17\",\n            ),\n            (\n                &[\n                    \"⒘\", // U+2498: NUMBER SEVENTEEN FULL STOP\n                ],\n                \"17.\",\n            ),\n            (\n                &[\n                    \"⒄\", // U+2484: PARENTHESIZED NUMBER SEVENTEEN\n                ],\n                \"(17)\",\n            ),\n            (\n                &[\n                    \"⑱\", // U+2471: CIRCLED NUMBER EIGHTEEN\n                    \"⓲\", // U+24F2: NEGATIVE CIRCLED NUMBER EIGHTEEN\n                ],\n                \"18\",\n            ),\n            (\n                &[\n                    \"⒙\", // U+2499: NUMBER EIGHTEEN FULL STOP\n                ],\n                \"18.\",\n            ),\n            (\n                &[\n                    \"⒅\", // U+2485: PARENTHESIZED NUMBER EIGHTEEN\n                ],\n                \"(18)\",\n            ),\n            (\n                &[\n                    \"⑲\", // U+2472: CIRCLED NUMBER NINETEEN\n                    \"⓳\", // U+24F3: NEGATIVE CIRCLED NUMBER NINETEEN\n                ],\n                \"19\",\n            ),\n            (\n                &[\n                    \"⒚\", // U+249A: NUMBER NINETEEN FULL STOP\n                ],\n                \"19.\",\n            ),\n            (\n                &[\n                    \"⒆\", // U+2486: PARENTHESIZED NUMBER NINETEEN\n                ],\n                \"(19)\",\n            ),\n            (\n                &[\n                    \"⑳\", // U+2473: CIRCLED NUMBER TWENTY\n                    \"⓴\", // U+24F4: NEGATIVE CIRCLED NUMBER TWENTY\n                ],\n                \"20\",\n            ),\n            (\n                &[\n                    \"⒛\", // U+249B: NUMBER TWENTY FULL STOP\n                ],\n                \"20.\",\n            ),\n            (\n                &[\n                    \"⒇\", // U+2487: PARENTHESIZED NUMBER TWENTY\n                ],\n                \"(20)\",\n            ),\n            (\n                &[\n                    \"«\",  // U+00AB: LEFT-POINTING DOUBLE ANGLE QUOTATION MARK\n                    \"»\",  // U+00BB: RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK\n                    \"“\",  // U+201C: LEFT DOUBLE QUOTATION MARK\n                    \"”\",  // U+201D: RIGHT DOUBLE QUOTATION MARK\n                    \"„\",  // U+201E: DOUBLE LOW-9 QUOTATION MARK\n                    \"″\",  // U+2033: DOUBLE PRIME\n                    \"‶\",  // U+2036: REVERSED DOUBLE PRIME\n                    \"❝\",  // U+275D: HEAVY DOUBLE TURNED COMMA QUOTATION MARK ORNAMENT\n                    \"❞\",  // U+275E: HEAVY DOUBLE COMMA QUOTATION MARK ORNAMENT\n                    \"❮\",  // U+276E: HEAVY LEFT-POINTING ANGLE QUOTATION MARK ORNAMENT\n                    \"❯\",  // U+276F: HEAVY RIGHT-POINTING ANGLE QUOTATION MARK ORNAMENT\n                    \"＂\", // U+FF02: FULLWIDTH QUOTATION MARK\n                ],\n                \"\\\"\",\n            ),\n            (\n                &[\n                    \"‘\",  // U+2018: LEFT SINGLE QUOTATION MARK\n                    \"’\",  // U+2019: RIGHT SINGLE QUOTATION MARK\n                    \"‚\",  // U+201A: SINGLE LOW-9 QUOTATION MARK\n                    \"‛\",  // U+201B: SINGLE HIGH-REVERSED-9 QUOTATION MARK\n                    \"′\",  // U+2032: PRIME\n                    \"‵\",  // U+2035: REVERSED PRIME\n                    \"‹\",  // U+2039: SINGLE LEFT-POINTING ANGLE QUOTATION MARK\n                    \"›\",  // U+203A: SINGLE RIGHT-POINTING ANGLE QUOTATION MARK\n                    \"❛\",  // U+275B: HEAVY SINGLE TURNED COMMA QUOTATION MARK ORNAMENT\n                    \"❜\",  // U+275C: HEAVY SINGLE COMMA QUOTATION MARK ORNAMENT\n                    \"＇\", // U+FF07: FULLWIDTH APOSTROPHE\n                ],\n                \"'\",\n            ),\n            (\n                &[\n                    \"‐\",  // U+2010: HYPHEN\n                    \"‑\",  // U+2011: NON-BREAKING HYPHEN\n                    \"‒\",  // U+2012: FIGURE DASH\n                    \"–\",  // U+2013: EN DASH\n                    \"—\",  // U+2014: EM DASH\n                    \"⁻\",  // U+207B: SUPERSCRIPT MINUS\n                    \"₋\",  // U+208B: SUBSCRIPT MINUS\n                    \"－\", // U+FF0D: FULLWIDTH HYPHEN-MINUS\n                ],\n                \"-\",\n            ),\n            (\n                &[\n                    \"⁅\",  // U+2045: LEFT SQUARE BRACKET WITH QUILL\n                    \"❲\",  // U+2772: LIGHT LEFT TORTOISE SHELL BRACKET ORNAMENT\n                    \"［\", // U+FF3B: FULLWIDTH LEFT SQUARE BRACKET\n                ],\n                \"[\",\n            ),\n            (\n                &[\n                    \"⁆\",  // U+2046: RIGHT SQUARE BRACKET WITH QUILL\n                    \"❳\",  // U+2773: LIGHT RIGHT TORTOISE SHELL BRACKET ORNAMENT\n                    \"］\", // U+FF3D: FULLWIDTH RIGHT SQUARE BRACKET\n                ],\n                \"]\",\n            ),\n            (\n                &[\n                    \"⁽\",  // U+207D: SUPERSCRIPT LEFT PARENTHESIS\n                    \"₍\",  // U+208D: SUBSCRIPT LEFT PARENTHESIS\n                    \"❨\",  // U+2768: MEDIUM LEFT PARENTHESIS ORNAMENT\n                    \"❪\",  // U+276A: MEDIUM FLATTENED LEFT PARENTHESIS ORNAMENT\n                    \"（\", // U+FF08: FULLWIDTH LEFT PARENTHESIS\n                ],\n                \"(\",\n            ),\n            (\n                &[\n                    \"⸨\", // U+2E28: LEFT DOUBLE PARENTHESIS\n                ],\n                \"((\",\n            ),\n            (\n                &[\n                    \"⁾\",  // U+207E: SUPERSCRIPT RIGHT PARENTHESIS\n                    \"₎\",  // U+208E: SUBSCRIPT RIGHT PARENTHESIS\n                    \"❩\",  // U+2769: MEDIUM RIGHT PARENTHESIS ORNAMENT\n                    \"❫\",  // U+276B: MEDIUM FLATTENED RIGHT PARENTHESIS ORNAMENT\n                    \"）\", // U+FF09: FULLWIDTH RIGHT PARENTHESIS\n                ],\n                \")\",\n            ),\n            (\n                &[\n                    \"⸩\", // U+2E29: RIGHT DOUBLE PARENTHESIS\n                ],\n                \"))\",\n            ),\n            (\n                &[\n                    \"❬\",  // U+276C: MEDIUM LEFT-POINTING ANGLE BRACKET ORNAMENT\n                    \"❰\",  // U+2770: HEAVY LEFT-POINTING ANGLE BRACKET ORNAMENT\n                    \"＜\", // U+FF1C: FULLWIDTH LESS-THAN SIGN\n                ],\n                \"<\",\n            ),\n            (\n                &[\n                    \"❭\",  // U+276D: MEDIUM RIGHT-POINTING ANGLE BRACKET ORNAMENT\n                    \"❱\",  // U+2771: HEAVY RIGHT-POINTING ANGLE BRACKET ORNAMENT\n                    \"＞\", // U+FF1E: FULLWIDTH GREATER-THAN SIGN\n                ],\n                \">\",\n            ),\n            (\n                &[\n                    \"❴\",  // U+2774: MEDIUM LEFT CURLY BRACKET ORNAMENT\n                    \"｛\", // U+FF5B: FULLWIDTH LEFT CURLY BRACKET\n                ],\n                \"{\",\n            ),\n            (\n                &[\n                    \"❵\",  // U+2775: MEDIUM RIGHT CURLY BRACKET ORNAMENT\n                    \"｝\", // U+FF5D: FULLWIDTH RIGHT CURLY BRACKET\n                ],\n                \"}\",\n            ),\n            (\n                &[\n                    \"⁺\",  // U+207A: SUPERSCRIPT PLUS SIGN\n                    \"₊\",  // U+208A: SUBSCRIPT PLUS SIGN\n                    \"＋\", // U+FF0B: FULLWIDTH PLUS SIGN\n                ],\n                \"+\",\n            ),\n            (\n                &[\n                    \"⁼\",  // U+207C: SUPERSCRIPT EQUALS SIGN\n                    \"₌\",  // U+208C: SUBSCRIPT EQUALS SIGN\n                    \"＝\", // U+FF1D: FULLWIDTH EQUALS SIGN\n                ],\n                \"=\",\n            ),\n            (\n                &[\n                    \"！\", // U+FF01: FULLWIDTH EXCLAMATION MARK\n                ],\n                \"!\",\n            ),\n            (\n                &[\n                    \"‼\", // U+203C: DOUBLE EXCLAMATION MARK\n                ],\n                \"!!\",\n            ),\n            (\n                &[\n                    \"⁉\", // U+2049: EXCLAMATION QUESTION MARK\n                ],\n                \"!?\",\n            ),\n            (\n                &[\n                    \"＃\", // U+FF03: FULLWIDTH NUMBER SIGN\n                ],\n                \"#\",\n            ),\n            (\n                &[\n                    \"＄\", // U+FF04: FULLWIDTH DOLLAR SIGN\n                ],\n                \"$\",\n            ),\n            (\n                &[\n                    \"⁒\",  // U+2052: COMMERCIAL MINUS SIGN\n                    \"％\", // U+FF05: FULLWIDTH PERCENT SIGN\n                ],\n                \"%\",\n            ),\n            (\n                &[\n                    \"＆\", // U+FF06: FULLWIDTH AMPERSAND\n                ],\n                \"&\",\n            ),\n            (\n                &[\n                    \"⁎\",  // U+204E: LOW ASTERISK\n                    \"＊\", // U+FF0A: FULLWIDTH ASTERISK\n                ],\n                \"*\",\n            ),\n            (\n                &[\n                    \"，\", // U+FF0C: FULLWIDTH COMMA\n                ],\n                \",\",\n            ),\n            (\n                &[\n                    \"．\", // U+FF0E: FULLWIDTH FULL STOP\n                ],\n                \".\",\n            ),\n            (\n                &[\n                    \"⁄\",  // U+2044: FRACTION SLASH\n                    \"／\", // U+FF0F: FULLWIDTH SOLIDUS\n                ],\n                \"/\",\n            ),\n            (\n                &[\n                    \"：\", // U+FF1A: FULLWIDTH COLON\n                ],\n                \":\",\n            ),\n            (\n                &[\n                    \"⁏\",  // U+204F: REVERSED SEMICOLON\n                    \"；\", // U+FF1B: FULLWIDTH SEMICOLON\n                ],\n                \";\",\n            ),\n            (\n                &[\n                    \"？\", // U+FF1F: FULLWIDTH QUESTION MARK\n                ],\n                \"?\",\n            ),\n            (\n                &[\n                    \"⁇\", // U+2047: DOUBLE QUESTION MARK\n                ],\n                \"??\",\n            ),\n            (\n                &[\n                    \"⁈\", // U+2048: QUESTION EXCLAMATION MARK\n                ],\n                \"?!\",\n            ),\n            (\n                &[\n                    \"＠\", // U+FF20: FULLWIDTH COMMERCIAL AT\n                ],\n                \"@\",\n            ),\n            (\n                &[\n                    \"＼\", // U+FF3C: FULLWIDTH REVERSE SOLIDUS\n                ],\n                \"\\\\\",\n            ),\n            (\n                &[\n                    \"‸\",  // U+2038: CARET\n                    \"＾\", // U+FF3E: FULLWIDTH CIRCUMFLEX ACCENT\n                ],\n                \"^\",\n            ),\n            (\n                &[\n                    \"＿\", // U+FF3F: FULLWIDTH LOW LINE\n                ],\n                \"_\",\n            ),\n            (\n                &[\n                    \"⁓\",  // U+2053: SWUNG DASH\n                    \"～\", // U+FF5E: FULLWIDTH TILDE\n                ],\n                \"~\",\n            ),\n        ];\n\n        for (characters, folded) in foldings {\n            for &c in characters {\n                assert_eq!(\n                    folding_using_raw_tokenizer_helper(c),\n                    folded,\n                    \"testing that character \\\"{c}\\\" becomes \\\"{folded}\\\"\"\n                );\n            }\n        }\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/empty_tokenizer.rs",
    "content": "use crate::tokenizer::{Token, TokenStream, Tokenizer};\n\n#[derive(Clone)]\npub(crate) struct EmptyTokenizer;\n\nimpl Tokenizer for EmptyTokenizer {\n    type TokenStream<'a> = EmptyTokenStream;\n    fn token_stream(&mut self, _text: &str) -> EmptyTokenStream {\n        EmptyTokenStream::default()\n    }\n}\n\n#[derive(Default)]\npub struct EmptyTokenStream {\n    token: Token,\n}\n\nimpl TokenStream for EmptyTokenStream {\n    fn advance(&mut self) -> bool {\n        false\n    }\n\n    fn token(&self) -> &super::Token {\n        &self.token\n    }\n\n    fn token_mut(&mut self) -> &mut super::Token {\n        &mut self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::{TokenStream, Tokenizer};\n\n    #[test]\n    fn test_empty_tokenizer() {\n        let mut tokenizer = super::EmptyTokenizer;\n        let mut empty = tokenizer.token_stream(\"whatever string\");\n        assert!(!empty.advance());\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/facet_tokenizer.rs",
    "content": "use super::{Token, TokenStream, Tokenizer};\nuse crate::schema::FACET_SEP_BYTE;\n\n/// The `FacetTokenizer` process a `Facet` binary representation\n/// and emits a token for all of its parent.\n///\n/// For instance,  `/america/north_america/canada`\n/// will emit the three following tokens\n///     - `/america/north_america/canada`\n///     - `/america/north_america`\n///     - `/america`\n#[derive(Clone, Default)]\npub struct FacetTokenizer {\n    token: Token,\n}\n\n#[derive(Debug)]\nenum State {\n    RootFacetNotEmitted,\n    UpToPosition(usize), //< we already emitted facet prefix up to &text[..cursor]\n    Terminated,\n}\n\npub struct FacetTokenStream<'a> {\n    text: &'a str,\n    state: State,\n    token: &'a mut Token,\n}\n\nimpl Tokenizer for FacetTokenizer {\n    type TokenStream<'a> = FacetTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> FacetTokenStream<'a> {\n        self.token.reset();\n        self.token.position = 0;\n        FacetTokenStream {\n            text,\n            state: State::RootFacetNotEmitted, //< pos is the first char that has not been processed yet.\n            token: &mut self.token,\n        }\n    }\n}\n\nimpl TokenStream for FacetTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        match self.state {\n            State::RootFacetNotEmitted => {\n                self.state = if self.text.is_empty() {\n                    State::Terminated\n                } else {\n                    State::UpToPosition(0)\n                };\n                true\n            }\n            State::UpToPosition(cursor) => {\n                let bytes: &[u8] = self.text.as_bytes();\n                if let Some(next_sep_pos) = bytes[cursor + 1..]\n                    .iter()\n                    .cloned()\n                    .position(|b| b == FACET_SEP_BYTE)\n                    .map(|pos| cursor + 1 + pos)\n                {\n                    let facet_part = &self.text[cursor..next_sep_pos];\n                    self.token.text.push_str(facet_part);\n                    self.state = State::UpToPosition(next_sep_pos);\n                } else {\n                    let facet_part = &self.text[cursor..];\n                    self.token.text.push_str(facet_part);\n                    self.state = State::Terminated;\n                }\n                true\n            }\n            State::Terminated => false,\n        }\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::FacetTokenizer;\n    use crate::schema::Facet;\n    use crate::tokenizer::{Token, TokenStream, Tokenizer};\n\n    #[test]\n    fn test_facet_tokenizer() {\n        let facet = Facet::from_path(vec![\"top\", \"a\", \"b\"]);\n        let mut tokens = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                let facet = Facet::from_encoded(token.text.as_bytes().to_owned()).unwrap();\n                tokens.push(format!(\"{facet}\"));\n            };\n            FacetTokenizer::default()\n                .token_stream(facet.encoded_str())\n                .process(&mut add_token);\n        }\n        assert_eq!(tokens.len(), 4);\n        assert_eq!(tokens[0], \"/\");\n        assert_eq!(tokens[1], \"/top\");\n        assert_eq!(tokens[2], \"/top/a\");\n        assert_eq!(tokens[3], \"/top/a/b\");\n    }\n\n    #[test]\n    fn test_facet_tokenizer_root_facets() {\n        let facet = Facet::root();\n        let mut tokens = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                let facet = Facet::from_encoded(token.text.as_bytes().to_owned()).unwrap(); // ok test\n                tokens.push(format!(\"{facet}\"));\n            };\n            FacetTokenizer::default()\n                .token_stream(facet.encoded_str()) // ok test\n                .process(&mut add_token);\n        }\n        assert_eq!(tokens.len(), 1);\n        assert_eq!(tokens[0], \"/\");\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/lower_caser.rs",
    "content": "use std::mem;\n\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// Token filter that lowercase terms.\n#[derive(Clone)]\npub struct LowerCaser;\n\nimpl TokenFilter for LowerCaser {\n    type Tokenizer<T: Tokenizer> = LowerCaserFilter<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T> {\n        LowerCaserFilter {\n            tokenizer,\n            buffer: String::new(),\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct LowerCaserFilter<T> {\n    tokenizer: T,\n    buffer: String,\n}\n\nimpl<T: Tokenizer> Tokenizer for LowerCaserFilter<T> {\n    type TokenStream<'a> = LowerCaserTokenStream<'a, T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        self.buffer.clear();\n        LowerCaserTokenStream {\n            tail: self.tokenizer.token_stream(text),\n            buffer: &mut self.buffer,\n        }\n    }\n}\n\npub struct LowerCaserTokenStream<'a, T> {\n    buffer: &'a mut String,\n    tail: T,\n}\n\n// writes a lowercased version of text into output.\nfn to_lowercase_unicode(text: &str, output: &mut String) {\n    output.clear();\n    output.reserve(50);\n    for c in text.chars() {\n        // Contrary to the std, we do not take care of sigma special case.\n        // This will have an normalizationo effect, which is ok for search.\n        output.extend(c.to_lowercase());\n    }\n}\n\nimpl<T: TokenStream> TokenStream for LowerCaserTokenStream<'_, T> {\n    fn advance(&mut self) -> bool {\n        if !self.tail.advance() {\n            return false;\n        }\n        if self.token_mut().text.is_ascii() {\n            // fast track for ascii.\n            self.token_mut().text.make_ascii_lowercase();\n        } else {\n            to_lowercase_unicode(&self.tail.token().text, self.buffer);\n            mem::swap(&mut self.tail.token_mut().text, self.buffer);\n        }\n        true\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{LowerCaser, SimpleTokenizer, TextAnalyzer, Token};\n\n    #[test]\n    fn test_to_lower_case() {\n        let tokens = token_stream_helper(\"Tree\");\n        assert_eq!(tokens.len(), 1);\n        assert_token(&tokens[0], 0, \"tree\", 0, 4);\n\n        let tokens = token_stream_helper(\"Русский текст\");\n        assert_eq!(tokens.len(), 2);\n        assert_token(&tokens[0], 0, \"русский\", 0, 14);\n        assert_token(&tokens[1], 1, \"текст\", 15, 25);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut token_stream = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(LowerCaser)\n            .build();\n\n        let mut token_stream = token_stream.token_stream(text);\n        let mut tokens = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/mod.rs",
    "content": "//! Tokenizer are in charge of chopping text into a stream of tokens\n//! ready for indexing.\n//!\n//! You must define in your schema which tokenizer should be used for\n//! each of your fields :\n//!\n//! ```rust\n//! use tantivy::schema::*;\n//!\n//! let mut schema_builder = Schema::builder();\n//!\n//! let text_options = TextOptions::default()\n//!     .set_indexing_options(\n//!         TextFieldIndexing::default()\n//!             .set_tokenizer(\"en_stem\")\n//!             .set_index_option(IndexRecordOption::Basic)\n//!     )\n//!     .set_stored();\n//!\n//! let id_options = TextOptions::default()\n//!     .set_indexing_options(\n//!         TextFieldIndexing::default()\n//!             .set_tokenizer(\"raw_ids\")\n//!             .set_index_option(IndexRecordOption::WithFreqsAndPositions)\n//!     )\n//!     .set_stored();\n//!\n//! schema_builder.add_text_field(\"title\", text_options.clone());\n//! schema_builder.add_text_field(\"text\", text_options);\n//! schema_builder.add_text_field(\"uuid\", id_options);\n//!\n//! let schema = schema_builder.build();\n//! ```\n//!\n//! By default, `tantivy` offers the following tokenizers:\n//!\n//! ## `default`\n//!\n//! `default` is the tokenizer that will be used if you do not\n//! assign a specific tokenizer to your text field.\n//! It will chop your text on punctuation and whitespaces,\n//! removes tokens that are longer than 40 chars, and lowercase your text.\n//!\n//! ## `raw`\n//! Does not actual tokenizer your text. It keeps it entirely unprocessed.\n//! It can be useful to index uuids, or urls for instance.\n//!\n//! ## `en_stem`\n//!\n//! In addition to what `default` does, the `en_stem` tokenizer also\n//! apply stemming to your tokens. Stemming consists in trimming words to\n//! remove their inflection. This tokenizer is slower than the default one,\n//! but is recommended to improve recall.\n//!\n//! # Custom tokenizer Library\n//! Avoid using tantivy as dependency and prefer `tantivy-tokenizer-api` instead.\n//!\n//! # Custom tokenizers\n//!\n//! You can write your own tokenizer by implementing the [`Tokenizer`] trait\n//! or you can extend an existing [`Tokenizer`] by chaining it with several\n//! [`TokenFilter`]s.\n//!\n//! For instance, the `en_stem` is defined as follows.\n//!\n//! ```rust\n//! use tantivy::tokenizer::*;\n//!\n//! let en_stem = TextAnalyzer::builder(SimpleTokenizer::default())\n//!     .filter(RemoveLongFilter::limit(40))\n//!     .filter(LowerCaser)\n//!     .filter(Stemmer::new(Language::English))\n//!     .build();\n//! ```\n//!\n//! Once your tokenizer is defined, you need to\n//! register it with a name in your index's [`TokenizerManager`].\n//!\n//! ```rust\n//! # use tantivy::schema::Schema;\n//! # use tantivy::tokenizer::*;\n//! # use tantivy::Index;\n//! #\n//! let custom_en_tokenizer = SimpleTokenizer::default();\n//! # let schema = Schema::builder().build();\n//! let index = Index::create_in_ram(schema);\n//! index.tokenizers()\n//!      .register(\"custom_en\", custom_en_tokenizer);\n//! ```\n//!\n//! If you built your schema programmatically, a complete example\n//! could like this for instance.\n//!\n//! Note that tokens with a len greater or equal to\n//! [`MAX_TOKEN_LEN`].\n//!\n//! # Example\n//!\n//! ```rust\n//! use tantivy::schema::{Schema, IndexRecordOption, TextOptions, TextFieldIndexing};\n//! use tantivy::tokenizer::*;\n//! use tantivy::Index;\n//!\n//! let mut schema_builder = Schema::builder();\n//! let text_field_indexing = TextFieldIndexing::default()\n//!     .set_tokenizer(\"custom_en\")\n//!     .set_index_option(IndexRecordOption::WithFreqsAndPositions);\n//! let text_options = TextOptions::default()\n//!     .set_indexing_options(text_field_indexing)\n//!     .set_stored();\n//! schema_builder.add_text_field(\"title\", text_options);\n//! let schema = schema_builder.build();\n//! let index = Index::create_in_ram(schema);\n//!\n//! // We need to register our tokenizer :\n//! let custom_en_tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n//!     .filter(RemoveLongFilter::limit(40))\n//!     .filter(LowerCaser)\n//!     .build();\n//! index\n//!     .tokenizers()\n//!     .register(\"custom_en\", custom_en_tokenizer);\n//! ```\nmod alphanum_only;\nmod ascii_folding_filter;\nmod empty_tokenizer;\nmod facet_tokenizer;\nmod lower_caser;\nmod ngram_tokenizer;\nmod raw_tokenizer;\nmod regex_tokenizer;\nmod remove_long;\nmod simple_tokenizer;\nmod split_compound_words;\nmod stop_word_filter;\nmod tokenized_string;\nmod tokenizer;\nmod tokenizer_manager;\nmod whitespace_tokenizer;\n\n#[cfg(feature = \"stemmer\")]\nmod stemmer;\npub use tokenizer_api::{BoxTokenStream, Token, TokenFilter, TokenStream, Tokenizer};\n\npub use self::alphanum_only::AlphaNumOnlyFilter;\npub use self::ascii_folding_filter::AsciiFoldingFilter;\npub use self::facet_tokenizer::FacetTokenizer;\npub use self::lower_caser::LowerCaser;\npub use self::ngram_tokenizer::NgramTokenizer;\npub use self::raw_tokenizer::RawTokenizer;\npub use self::regex_tokenizer::RegexTokenizer;\npub use self::remove_long::RemoveLongFilter;\npub use self::simple_tokenizer::{SimpleTokenStream, SimpleTokenizer};\npub use self::split_compound_words::SplitCompoundWords;\n#[cfg(feature = \"stemmer\")]\npub use self::stemmer::{Language, Stemmer};\npub use self::stop_word_filter::StopWordFilter;\npub use self::tokenized_string::{PreTokenizedStream, PreTokenizedString};\npub use self::tokenizer::{TextAnalyzer, TextAnalyzerBuilder};\npub use self::tokenizer_manager::TokenizerManager;\npub use self::whitespace_tokenizer::WhitespaceTokenizer;\n\n/// Maximum authorized len (in bytes) for a token.\n///\n/// Tokenizers are in charge of not emitting tokens larger than this value.\n/// Currently, if a faulty tokenizer implementation emits tokens with a length larger than\n/// `2^16 - 1 - 5`, the token will simply be ignored downstream.\npub const MAX_TOKEN_LEN: usize = u16::MAX as usize - 5;\n\n#[cfg(test)]\npub(crate) mod tests {\n    use super::{Token, TokenizerManager};\n\n    /// This is a function that can be used in tests and doc tests\n    /// to assert a token's correctness.\n    pub fn assert_token(token: &Token, position: usize, text: &str, from: usize, to: usize) {\n        assert_eq!(\n            token.position, position,\n            \"expected position {position} but {token:?}\"\n        );\n        assert_eq!(token.text, text, \"expected text {text} but {token:?}\");\n        assert_eq!(\n            token.offset_from, from,\n            \"expected offset_from {from} but {token:?}\"\n        );\n        assert_eq!(token.offset_to, to, \"expected offset_to {to} but {token:?}\");\n    }\n\n    #[test]\n    fn test_raw_tokenizer2() {\n        let tokenizer_manager = TokenizerManager::default();\n        let mut en_tokenizer = tokenizer_manager.get(\"raw\").unwrap();\n        let mut tokens: Vec<Token> = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                tokens.push(token.clone());\n            };\n            en_tokenizer\n                .token_stream(\"Hello, happy tax payer!\")\n                .process(&mut add_token);\n        }\n        assert_eq!(tokens.len(), 1);\n        assert_token(&tokens[0], 0, \"Hello, happy tax payer!\", 0, 23);\n    }\n\n    #[test]\n    fn test_tokenizer_does_not_exist() {\n        let tokenizer_manager = TokenizerManager::default();\n        assert!(tokenizer_manager.get(\"en_doesnotexist\").is_none());\n    }\n\n    #[test]\n    fn test_tokenizer_empty() {\n        let tokenizer_manager = TokenizerManager::default();\n        let mut en_tokenizer = tokenizer_manager.get(\"default\").unwrap();\n        {\n            let mut tokens: Vec<Token> = vec![];\n            {\n                let mut add_token = |token: &Token| {\n                    tokens.push(token.clone());\n                };\n                en_tokenizer.token_stream(\" \").process(&mut add_token);\n            }\n            assert!(tokens.is_empty());\n        }\n        {\n            let mut tokens: Vec<Token> = vec![];\n            {\n                let mut add_token = |token: &Token| {\n                    tokens.push(token.clone());\n                };\n                en_tokenizer.token_stream(\" \").process(&mut add_token);\n            }\n            assert!(tokens.is_empty());\n        }\n    }\n\n    #[test]\n    fn test_whitespace_tokenizer() {\n        let tokenizer_manager = TokenizerManager::default();\n        let mut ws_tokenizer = tokenizer_manager.get(\"whitespace\").unwrap();\n        let mut tokens: Vec<Token> = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                tokens.push(token.clone());\n            };\n            ws_tokenizer\n                .token_stream(\"Hello, happy tax payer!\")\n                .process(&mut add_token);\n        }\n\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"Hello,\", 0, 6);\n        assert_token(&tokens[1], 1, \"happy\", 7, 12);\n        assert_token(&tokens[2], 2, \"tax\", 13, 16);\n        assert_token(&tokens[3], 3, \"payer!\", 17, 23);\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/ngram_tokenizer.rs",
    "content": "use super::{Token, TokenStream, Tokenizer};\nuse crate::TantivyError;\n\n/// Tokenize the text by splitting words into n-grams of the given size(s)\n///\n/// With this tokenizer, the `position` is always 0.\n/// Beware however, in presence of multiple value for the same field,\n/// the position will be `POSITION_GAP * index of value`.\n///\n/// Example 1: `hello` would be tokenized as (min_gram: 2, max_gram: 3, prefix_only: false)\n///\n/// | Term     | he  | hel | el  | ell | ll  | llo | lo |\n/// |----------|-----|-----|-----|-----|-----|-----|----|\n/// | Position | 0   | 0   | 0   | 0   | 0   | 0   | 0  |\n/// | Offsets  | 0,2 | 0,3 | 1,3 | 1,4 | 2,4 | 2,5 | 3,5|\n///\n/// Example 2: `hello` would be tokenized as (min_gram: 2, max_gram: 5, prefix_only: **true**)\n///\n/// | Term     | he  | hel | hell  | hello |\n/// |----------|-----|-----|-------|-------|\n/// | Position | 0   | 0   | 0     | 0     |\n/// | Offsets  | 0,2 | 0,3 | 0,4   | 0,5   |\n///\n/// Example 3: `hεllo` (non-ascii) would be tokenized as (min_gram: 2, max_gram: 5, prefix_only:\n/// **true**)\n///\n/// | Term     | hε  | hεl | hεll  | hεllo |\n/// |----------|-----|-----|-------|-------|\n/// | Position | 0   | 0   | 0     | 0     |\n/// | Offsets  | 0,3 | 0,4 | 0,5   | 0,6   |\n///\n/// # Example\n///\n/// ```rust\n/// use tantivy::tokenizer::*;\n///\n/// let mut tokenizer = NgramTokenizer::new(2, 3, false).unwrap();\n/// let mut stream = tokenizer.token_stream(\"hello\");\n/// {\n///     let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"he\");\n///     assert_eq!(token.offset_from, 0);\n///     assert_eq!(token.offset_to, 2);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"hel\");\n///     assert_eq!(token.offset_from, 0);\n///     assert_eq!(token.offset_to, 3);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"el\");\n///     assert_eq!(token.offset_from, 1);\n///     assert_eq!(token.offset_to, 3);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"ell\");\n///     assert_eq!(token.offset_from, 1);\n///     assert_eq!(token.offset_to, 4);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"ll\");\n///     assert_eq!(token.offset_from, 2);\n///     assert_eq!(token.offset_to, 4);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"llo\");\n///     assert_eq!(token.offset_from, 2);\n///     assert_eq!(token.offset_to, 5);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///   assert_eq!(token.text, \"lo\");\n///   assert_eq!(token.offset_from, 3);\n///   assert_eq!(token.offset_to, 5);\n/// }\n/// assert!(stream.next().is_none());\n/// ```\n#[derive(Clone, Debug)]\npub struct NgramTokenizer {\n    /// min size of the n-gram\n    min_gram: usize,\n    /// max size of the n-gram\n    max_gram: usize,\n    /// if true, will only parse the leading edge of the input\n    prefix_only: bool,\n    token: Token,\n}\n\nimpl NgramTokenizer {\n    /// Configures a new Ngram tokenizer\n    pub fn new(\n        min_gram: usize,\n        max_gram: usize,\n        prefix_only: bool,\n    ) -> crate::Result<NgramTokenizer> {\n        if min_gram == 0 {\n            return Err(TantivyError::InvalidArgument(\n                \"min_gram must be greater than 0\".to_string(),\n            ));\n        }\n        if min_gram > max_gram {\n            return Err(TantivyError::InvalidArgument(\n                \"min_gram must not be greater than max_gram\".to_string(),\n            ));\n        }\n        Ok(NgramTokenizer {\n            min_gram,\n            max_gram,\n            prefix_only,\n            token: Token::default(),\n        })\n    }\n\n    /// Create a `NGramTokenizer` which generates tokens for all inner ngrams.\n    ///\n    /// This is as opposed to only prefix ngrams    .\n    pub fn all_ngrams(min_gram: usize, max_gram: usize) -> crate::Result<NgramTokenizer> {\n        Self::new(min_gram, max_gram, false)\n    }\n\n    /// Create a `NGramTokenizer` which only generates tokens for the\n    /// prefix ngrams.\n    pub fn prefix_only(min_gram: usize, max_gram: usize) -> crate::Result<NgramTokenizer> {\n        Self::new(min_gram, max_gram, true)\n    }\n}\n\n/// TokenStream associate to the `NgramTokenizer`\npub struct NgramTokenStream<'a> {\n    /// parameters\n    ngram_charidx_iterator: StutteringIterator<CodepointFrontiers<'a>>,\n    /// true if the NgramTokenStream is in prefix mode.\n    prefix_only: bool,\n    /// input\n    text: &'a str,\n    /// output\n    token: &'a mut Token,\n}\n\nimpl Tokenizer for NgramTokenizer {\n    type TokenStream<'a> = NgramTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> NgramTokenStream<'a> {\n        self.token.reset();\n        NgramTokenStream {\n            ngram_charidx_iterator: StutteringIterator::new(\n                CodepointFrontiers::for_str(text),\n                self.min_gram,\n                self.max_gram,\n            ),\n            prefix_only: self.prefix_only,\n            text,\n            token: &mut self.token,\n        }\n    }\n}\n\nimpl TokenStream for NgramTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        if let Some((offset_from, offset_to)) = self.ngram_charidx_iterator.next() {\n            if self.prefix_only && offset_from > 0 {\n                return false;\n            }\n            self.token.position = 0;\n            self.token.offset_from = offset_from;\n            self.token.offset_to = offset_to;\n            self.token.text.clear();\n            self.token.text.push_str(&self.text[offset_from..offset_to]);\n            true\n        } else {\n            false\n        }\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n/// This iterator takes an underlying Iterator\n/// and emits all of the pairs `(a,b)` such that\n/// a and b are items emitted by the iterator at\n/// an interval between `min_gram` and `max_gram`.\n///\n/// The elements are emitted in the order of appearance\n/// of `a` first, `b` then.\n///\n/// See `test_stuttering_iterator` for an example of its\n/// output.\nstruct StutteringIterator<T> {\n    underlying: T,\n    min_gram: usize,\n    max_gram: usize,\n\n    memory: Vec<usize>,\n    cursor: usize,\n    gram_len: usize,\n}\n\nimpl<T> StutteringIterator<T>\nwhere T: Iterator<Item = usize>\n{\n    pub fn new(mut underlying: T, min_gram: usize, max_gram: usize) -> StutteringIterator<T> {\n        assert!(min_gram > 0);\n        let memory: Vec<usize> = (&mut underlying).take(max_gram + 1).collect();\n        if memory.len() <= min_gram {\n            // returns an empty iterator\n            StutteringIterator {\n                underlying,\n                min_gram: 1,\n                max_gram: 0,\n                memory,\n                cursor: 0,\n                gram_len: 0,\n            }\n        } else {\n            StutteringIterator {\n                underlying,\n                min_gram,\n                max_gram: memory.len() - 1,\n                memory,\n                cursor: 0,\n                gram_len: min_gram,\n            }\n        }\n    }\n}\n\nimpl<T> Iterator for StutteringIterator<T>\nwhere T: Iterator<Item = usize>\n{\n    type Item = (usize, usize);\n\n    fn next(&mut self) -> Option<(usize, usize)> {\n        if self.gram_len > self.max_gram {\n            // we have exhausted all options\n            // starting at `self.memory[self.cursor]`.\n            //\n            // Time to advance.\n            self.gram_len = self.min_gram;\n            if let Some(next_val) = self.underlying.next() {\n                self.memory[self.cursor] = next_val;\n            } else {\n                self.max_gram -= 1;\n            }\n            self.cursor += 1;\n            if self.cursor >= self.memory.len() {\n                self.cursor = 0;\n            }\n        }\n        if self.max_gram < self.min_gram {\n            return None;\n        }\n        let start = self.memory[self.cursor % self.memory.len()];\n        let stop = self.memory[(self.cursor + self.gram_len) % self.memory.len()];\n        self.gram_len += 1;\n        Some((start, stop))\n    }\n}\n\n/// Emits all of the offsets where a codepoint starts\n/// or a codepoint ends.\n///\n/// By convention, we emit `[0]` for the empty string.\nstruct CodepointFrontiers<'a> {\n    s: &'a str,\n    next_el: Option<usize>,\n}\n\nimpl<'a> CodepointFrontiers<'a> {\n    fn for_str(s: &'a str) -> Self {\n        CodepointFrontiers {\n            s,\n            next_el: Some(0),\n        }\n    }\n}\n\nimpl Iterator for CodepointFrontiers<'_> {\n    type Item = usize;\n\n    fn next(&mut self) -> Option<usize> {\n        self.next_el.inspect(|&offset| {\n            if self.s.is_empty() {\n                self.next_el = None;\n            } else {\n                let first_codepoint_width = utf8_codepoint_width(self.s.as_bytes()[0]);\n                self.s = &self.s[first_codepoint_width..];\n                self.next_el = Some(offset + first_codepoint_width);\n            }\n        })\n    }\n}\n\nconst CODEPOINT_UTF8_WIDTH: [u8; 16] = [1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 4];\n\n// Number of bytes to encode a codepoint in UTF-8 given\n// the first byte.\n//\n// To do that we count the number of higher significant bits set to `1`.\nfn utf8_codepoint_width(b: u8) -> usize {\n    let higher_4_bits = (b as usize) >> 4;\n    CODEPOINT_UTF8_WIDTH[higher_4_bits] as usize\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::{utf8_codepoint_width, CodepointFrontiers, NgramTokenizer, StutteringIterator};\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{Token, TokenStream, Tokenizer};\n\n    fn test_helper<T: TokenStream>(mut tokenizer: T) -> Vec<Token> {\n        let mut tokens: Vec<Token> = vec![];\n        tokenizer.process(&mut |token: &Token| tokens.push(token.clone()));\n        tokens\n    }\n\n    #[test]\n    fn test_utf8_codepoint_width() {\n        // 0xxx\n        for i in 0..128 {\n            assert_eq!(utf8_codepoint_width(i), 1);\n        }\n        // 110xx\n        for i in (128 | 64)..(128 | 64 | 32) {\n            assert_eq!(utf8_codepoint_width(i), 2);\n        }\n        // 1110xx\n        for i in (128 | 64 | 32)..(128 | 64 | 32 | 16) {\n            assert_eq!(utf8_codepoint_width(i), 3);\n        }\n        // 1111xx\n        for i in (128 | 64 | 32 | 16)..256 {\n            assert_eq!(utf8_codepoint_width(i as u8), 4);\n        }\n    }\n\n    #[test]\n    fn test_codepoint_frontiers() {\n        assert_eq!(CodepointFrontiers::for_str(\"\").collect::<Vec<_>>(), vec![0]);\n        assert_eq!(\n            CodepointFrontiers::for_str(\"abcd\").collect::<Vec<_>>(),\n            vec![0, 1, 2, 3, 4]\n        );\n        assert_eq!(\n            CodepointFrontiers::for_str(\"aあ\").collect::<Vec<_>>(),\n            vec![0, 1, 4]\n        );\n    }\n\n    #[test]\n    fn test_ngram_tokenizer_1_2_false() {\n        let tokens = test_helper(\n            NgramTokenizer::all_ngrams(1, 2)\n                .unwrap()\n                .token_stream(\"hello\"),\n        );\n        assert_eq!(tokens.len(), 9);\n        assert_token(&tokens[0], 0, \"h\", 0, 1);\n        assert_token(&tokens[1], 0, \"he\", 0, 2);\n        assert_token(&tokens[2], 0, \"e\", 1, 2);\n        assert_token(&tokens[3], 0, \"el\", 1, 3);\n        assert_token(&tokens[4], 0, \"l\", 2, 3);\n        assert_token(&tokens[5], 0, \"ll\", 2, 4);\n        assert_token(&tokens[6], 0, \"l\", 3, 4);\n        assert_token(&tokens[7], 0, \"lo\", 3, 5);\n        assert_token(&tokens[8], 0, \"o\", 4, 5);\n    }\n\n    #[test]\n    fn test_ngram_tokenizer_min_max_equal() {\n        let tokens = test_helper(\n            NgramTokenizer::all_ngrams(3, 3)\n                .unwrap()\n                .token_stream(\"hello\"),\n        );\n        assert_eq!(tokens.len(), 3);\n        assert_token(&tokens[0], 0, \"hel\", 0, 3);\n        assert_token(&tokens[1], 0, \"ell\", 1, 4);\n        assert_token(&tokens[2], 0, \"llo\", 2, 5);\n    }\n\n    #[test]\n    fn test_ngram_tokenizer_2_5_prefix() {\n        let tokens = test_helper(\n            NgramTokenizer::prefix_only(2, 5)\n                .unwrap()\n                .token_stream(\"frankenstein\"),\n        );\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"fr\", 0, 2);\n        assert_token(&tokens[1], 0, \"fra\", 0, 3);\n        assert_token(&tokens[2], 0, \"fran\", 0, 4);\n        assert_token(&tokens[3], 0, \"frank\", 0, 5);\n    }\n\n    #[test]\n    fn test_ngram_non_ascii_1_2() {\n        let tokens = test_helper(\n            NgramTokenizer::all_ngrams(1, 2)\n                .unwrap()\n                .token_stream(\"hεllo\"),\n        );\n        assert_eq!(tokens.len(), 9);\n        assert_token(&tokens[0], 0, \"h\", 0, 1);\n        assert_token(&tokens[1], 0, \"hε\", 0, 3);\n        assert_token(&tokens[2], 0, \"ε\", 1, 3);\n        assert_token(&tokens[3], 0, \"εl\", 1, 4);\n        assert_token(&tokens[4], 0, \"l\", 3, 4);\n        assert_token(&tokens[5], 0, \"ll\", 3, 5);\n        assert_token(&tokens[6], 0, \"l\", 4, 5);\n        assert_token(&tokens[7], 0, \"lo\", 4, 6);\n        assert_token(&tokens[8], 0, \"o\", 5, 6);\n    }\n\n    #[test]\n    fn test_ngram_non_ascii_2_5_prefix() {\n        let tokens = test_helper(\n            NgramTokenizer::prefix_only(2, 5)\n                .unwrap()\n                .token_stream(\"hεllo\"),\n        );\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"hε\", 0, 3);\n        assert_token(&tokens[1], 0, \"hεl\", 0, 4);\n        assert_token(&tokens[2], 0, \"hεll\", 0, 5);\n        assert_token(&tokens[3], 0, \"hεllo\", 0, 6);\n    }\n\n    #[test]\n    fn test_ngram_empty() {\n        let tokens = test_helper(NgramTokenizer::all_ngrams(1, 5).unwrap().token_stream(\"\"));\n        assert!(tokens.is_empty());\n        let tokens = test_helper(NgramTokenizer::all_ngrams(2, 5).unwrap().token_stream(\"\"));\n        assert!(tokens.is_empty());\n    }\n\n    #[test]\n    #[should_panic(expected = \"min_gram must be greater than 0\")]\n    fn test_ngram_min_max_interval_empty() {\n        test_helper(\n            NgramTokenizer::all_ngrams(0, 2)\n                .unwrap()\n                .token_stream(\"hellossss\"),\n        );\n    }\n\n    #[test]\n    #[should_panic(expected = \"min_gram must not be greater than max_gram\")]\n    fn test_invalid_interval_should_panic_if_smaller() {\n        NgramTokenizer::all_ngrams(2, 1).unwrap();\n    }\n\n    #[test]\n    fn test_stuttering_iterator_empty() {\n        let rg: Vec<usize> = vec![0];\n        let mut it = StutteringIterator::new(rg.into_iter(), 1, 2);\n        assert_eq!(it.next(), None);\n    }\n\n    #[test]\n    fn test_stuterring_iterator() {\n        let mut it = StutteringIterator::new(0..10, 1, 2);\n        assert_eq!(it.next(), Some((0, 1)));\n        assert_eq!(it.next(), Some((0, 2)));\n        assert_eq!(it.next(), Some((1, 2)));\n        assert_eq!(it.next(), Some((1, 3)));\n        assert_eq!(it.next(), Some((2, 3)));\n        assert_eq!(it.next(), Some((2, 4)));\n        assert_eq!(it.next(), Some((3, 4)));\n        assert_eq!(it.next(), Some((3, 5)));\n        assert_eq!(it.next(), Some((4, 5)));\n        assert_eq!(it.next(), Some((4, 6)));\n        assert_eq!(it.next(), Some((5, 6)));\n        assert_eq!(it.next(), Some((5, 7)));\n        assert_eq!(it.next(), Some((6, 7)));\n        assert_eq!(it.next(), Some((6, 8)));\n        assert_eq!(it.next(), Some((7, 8)));\n        assert_eq!(it.next(), Some((7, 9)));\n        assert_eq!(it.next(), Some((8, 9)));\n        assert_eq!(it.next(), None);\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/raw_tokenizer.rs",
    "content": "use super::{Token, TokenStream, Tokenizer};\n\n/// For each value of the field, emit a single unprocessed token.\n#[derive(Clone, Default)]\npub struct RawTokenizer {\n    token: Token,\n}\n\npub struct RawTokenStream<'a> {\n    token: &'a mut Token,\n    has_token: bool,\n}\n\nimpl Tokenizer for RawTokenizer {\n    type TokenStream<'a> = RawTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &str) -> RawTokenStream<'a> {\n        self.token.reset();\n        self.token.position = 0;\n        self.token.position_length = 1;\n        self.token.offset_from = 0;\n        self.token.offset_to = text.len();\n        self.token.text.clear();\n        self.token.text.push_str(text);\n        RawTokenStream {\n            token: &mut self.token,\n            has_token: true,\n        }\n    }\n}\n\nimpl TokenStream for RawTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        let result = self.has_token;\n        self.has_token = false;\n        result\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{RawTokenizer, TextAnalyzer, Token};\n\n    #[test]\n    fn test_raw_tokenizer() {\n        let tokens = token_stream_helper(\"Hello, happy tax payer!\");\n        assert_eq!(tokens.len(), 1);\n        assert_token(&tokens[0], 0, \"Hello, happy tax payer!\", 0, 23);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut a = TextAnalyzer::from(RawTokenizer::default());\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/regex_tokenizer.rs",
    "content": "use regex::Regex;\n\nuse super::{Token, TokenStream, Tokenizer};\nuse crate::TantivyError;\n\n/// Tokenize the text by using a regex pattern to split.\n///\n/// Each match of the regex emits a distinct token, empty tokens will not be emitted. Anchors such\n/// as `\\A` will match the text from the part where the last token was emitted or the beginning of\n/// the complete text if no token was emitted yet.\n///\n/// Example: `` 'aaa' bbb 'ccc' 'ddd' `` with the pattern `` '(?:\\w*)' `` will be tokenized as\n/// followed:\n///\n/// | Term     | aaa  | ccc    | ddd   |\n/// |----------|------|--------|-------|\n/// | Position | 1    | 2      | 3     |\n/// | Offsets  |0,5   | 10,15  | 16,21 |\n///\n///\n/// # Example\n///\n/// ```rust\n/// use tantivy::tokenizer::*;\n///\n/// let mut tokenizer = RegexTokenizer::new(r\"'(?:\\w*)'\").unwrap();\n/// let mut stream = tokenizer.token_stream(\"'aaa' bbb 'ccc' 'ddd'\");\n/// {\n///     let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"'aaa'\");\n///     assert_eq!(token.offset_from, 0);\n///     assert_eq!(token.offset_to, 5);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"'ccc'\");\n///     assert_eq!(token.offset_from, 10);\n///     assert_eq!(token.offset_to, 15);\n/// }\n/// {\n///   let token = stream.next().unwrap();\n///     assert_eq!(token.text, \"'ddd'\");\n///     assert_eq!(token.offset_from, 16);\n///     assert_eq!(token.offset_to, 21);\n/// }\n/// assert!(stream.next().is_none());\n/// ```\n\n#[derive(Clone)]\npub struct RegexTokenizer {\n    regex: Regex,\n    token: Token,\n}\n\nimpl RegexTokenizer {\n    /// Creates a new RegexTokenizer.\n    pub fn new(regex_pattern: &str) -> crate::Result<RegexTokenizer> {\n        Regex::new(regex_pattern)\n            .map_err(|_| TantivyError::InvalidArgument(regex_pattern.to_owned()))\n            .map(|regex| Self {\n                regex,\n                token: Token::default(),\n            })\n    }\n}\n\nimpl Tokenizer for RegexTokenizer {\n    type TokenStream<'a> = RegexTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> RegexTokenStream<'a> {\n        self.token.reset();\n        RegexTokenStream {\n            regex: self.regex.clone(),\n            text,\n            token: &mut self.token,\n            cursor: 0,\n        }\n    }\n}\n\npub struct RegexTokenStream<'a> {\n    regex: Regex,\n    text: &'a str,\n    token: &'a mut Token,\n    cursor: usize,\n}\n\nimpl TokenStream for RegexTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        let Some(regex_match) = self.regex.find(self.text) else {\n            return false;\n        };\n        if regex_match.as_str().is_empty() {\n            return false;\n        }\n        self.token.text.clear();\n        self.token.text.push_str(regex_match.as_str());\n\n        self.token.offset_from = self.cursor + regex_match.start();\n        self.cursor += regex_match.end();\n        self.token.offset_to = self.cursor;\n\n        self.token.position = self.token.position.wrapping_add(1);\n\n        self.text = &self.text[regex_match.end()..];\n        true\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::regex_tokenizer::RegexTokenizer;\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{TextAnalyzer, Token};\n\n    #[test]\n    fn test_regex_tokenizer() {\n        let tokens = token_stream_helper(\"'aaa' bbb 'ccc' 'ddd'\", r\"'(?:\\w*)'\");\n        assert_eq!(tokens.len(), 3);\n        assert_token(&tokens[0], 0, \"'aaa'\", 0, 5);\n        assert_token(&tokens[1], 1, \"'ccc'\", 10, 15);\n        assert_token(&tokens[2], 2, \"'ddd'\", 16, 21);\n    }\n\n    #[test]\n    fn test_regexp_tokenizer_no_match_on_input_data() {\n        let tokens = token_stream_helper(\"aaa\", r\"'(?:\\w*)'\");\n        assert_eq!(tokens.len(), 0);\n    }\n\n    #[test]\n    fn test_regexp_tokenizer_no_input_data() {\n        let tokens = token_stream_helper(\"\", r\"'(?:\\w*)'\");\n        assert_eq!(tokens.len(), 0);\n    }\n\n    #[test]\n    fn test_regexp_tokenizer_error_on_invalid_regex() {\n        let tokenizer = RegexTokenizer::new(r\"\\@(\");\n        assert_eq!(tokenizer.is_err(), true);\n        assert_eq!(\n            tokenizer.err().unwrap().to_string(),\n            \"An invalid argument was passed: '\\\\@('\"\n        );\n    }\n\n    fn token_stream_helper(text: &str, pattern: &str) -> Vec<Token> {\n        let r = RegexTokenizer::new(pattern).unwrap();\n        let mut a = TextAnalyzer::from(r);\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/remove_long.rs",
    "content": "//! # Example\n//! ```rust\n//! use tantivy::tokenizer::*;\n//!\n//! let mut tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n//!   .filter(RemoveLongFilter::limit(5))\n//!   .build();\n//!\n//! let mut stream = tokenizer.token_stream(\"toolong nice\");\n//! // because `toolong` is more than 5 characters, it is filtered\n//! // out of the token stream.\n//! assert_eq!(stream.next().unwrap().text, \"nice\");\n//! assert!(stream.next().is_none());\n//! ```\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// `RemoveLongFilter` removes tokens that are longer\n/// than a given number of bytes (in UTF-8 representation).\n///\n/// It is especially useful when indexing unconstrained content.\n/// e.g. Mail containing base-64 encoded pictures etc.\n#[derive(Clone)]\npub struct RemoveLongFilter {\n    length_limit: usize,\n}\n\nimpl RemoveLongFilter {\n    /// Creates a `RemoveLongFilter` given a limit in bytes of the UTF-8 representation.\n    pub fn limit(length_limit: usize) -> RemoveLongFilter {\n        RemoveLongFilter { length_limit }\n    }\n}\n\nimpl<T> RemoveLongFilterStream<T> {\n    fn predicate(&self, token: &Token) -> bool {\n        token.text.len() < self.token_length_limit\n    }\n}\n\nimpl TokenFilter for RemoveLongFilter {\n    type Tokenizer<T: Tokenizer> = RemoveLongFilterWrapper<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> RemoveLongFilterWrapper<T> {\n        RemoveLongFilterWrapper {\n            length_limit: self.length_limit,\n            inner: tokenizer,\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct RemoveLongFilterWrapper<T: Tokenizer> {\n    length_limit: usize,\n    inner: T,\n}\n\nimpl<T: Tokenizer> Tokenizer for RemoveLongFilterWrapper<T> {\n    type TokenStream<'a> = RemoveLongFilterStream<T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        RemoveLongFilterStream {\n            token_length_limit: self.length_limit,\n            tail: self.inner.token_stream(text),\n        }\n    }\n}\n\npub struct RemoveLongFilterStream<T> {\n    token_length_limit: usize,\n    tail: T,\n}\n\nimpl<T: TokenStream> TokenStream for RemoveLongFilterStream<T> {\n    fn advance(&mut self) -> bool {\n        while self.tail.advance() {\n            if self.predicate(self.tail.token()) {\n                return true;\n            }\n        }\n        false\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{RemoveLongFilter, SimpleTokenizer, TextAnalyzer, Token};\n\n    #[test]\n    fn test_remove_long() {\n        let tokens = token_stream_helper(\"hello tantivy, happy searching!\");\n        assert_eq!(tokens.len(), 2);\n        assert_token(&tokens[0], 0, \"hello\", 0, 5);\n        assert_token(&tokens[1], 2, \"happy\", 15, 20);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut a = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(RemoveLongFilter::limit(6))\n            .build();\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/simple_tokenizer.rs",
    "content": "use std::str::CharIndices;\n\nuse super::{Token, TokenStream, Tokenizer};\n\n/// Tokenize the text by splitting on whitespaces and punctuation.\n#[derive(Clone, Default)]\npub struct SimpleTokenizer {\n    token: Token,\n}\n\n/// TokenStream produced by the `SimpleTokenizer`.\npub struct SimpleTokenStream<'a> {\n    text: &'a str,\n    chars: CharIndices<'a>,\n    token: &'a mut Token,\n}\n\nimpl Tokenizer for SimpleTokenizer {\n    type TokenStream<'a> = SimpleTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> SimpleTokenStream<'a> {\n        self.token.reset();\n        SimpleTokenStream {\n            text,\n            chars: text.char_indices(),\n            token: &mut self.token,\n        }\n    }\n}\n\nimpl SimpleTokenStream<'_> {\n    // search for the end of the current token.\n    fn search_token_end(&mut self) -> usize {\n        (&mut self.chars)\n            .filter(|(_, c)| !c.is_alphanumeric())\n            .map(|(offset, _)| offset)\n            .next()\n            .unwrap_or(self.text.len())\n    }\n}\n\nimpl TokenStream for SimpleTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        self.token.text.clear();\n        self.token.position = self.token.position.wrapping_add(1);\n        while let Some((offset_from, c)) = self.chars.next() {\n            if c.is_alphanumeric() {\n                let offset_to = self.search_token_end();\n                self.token.offset_from = offset_from;\n                self.token.offset_to = offset_to;\n                self.token.text.push_str(&self.text[offset_from..offset_to]);\n                return true;\n            }\n        }\n        false\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{SimpleTokenizer, TextAnalyzer, Token};\n\n    #[test]\n    fn test_simple_tokenizer() {\n        let tokens = token_stream_helper(\"Hello, happy tax payer!\");\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"Hello\", 0, 5);\n        assert_token(&tokens[1], 1, \"happy\", 7, 12);\n        assert_token(&tokens[2], 2, \"tax\", 13, 16);\n        assert_token(&tokens[3], 3, \"payer\", 17, 22);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut a = TextAnalyzer::from(SimpleTokenizer::default());\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/split_compound_words.rs",
    "content": "use aho_corasick::{AhoCorasick, AhoCorasickBuilder, MatchKind};\n\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// A [`TokenFilter`] which splits compound words into their parts\n/// based on a given dictionary.\n///\n/// Words only will be split if they can be fully decomposed into\n/// consecutive matches into the given dictionary.\n///\n/// This is mostly useful to split [compound nouns][compound] common to many\n/// Germanic languages into their constituents.\n///\n/// # Example\n///\n/// The quality of the dictionary determines the quality of the splits,\n/// e.g. the missing stem \"back\" of \"backen\" implies that \"brotbackautomat\"\n/// is not split in the following example.\n///\n/// ```rust\n/// use tantivy::tokenizer::{SimpleTokenizer, SplitCompoundWords, TextAnalyzer};\n///\n/// let mut tokenizer =\n///        TextAnalyzer::builder(SimpleTokenizer::default())\n///        .filter(\n///            SplitCompoundWords::from_dictionary([\n///                 \"dampf\", \"schiff\", \"fahrt\", \"brot\", \"backen\", \"automat\",\n///            ])\n///            .unwrap()\n///        )\n///        .build();\n/// {\n///     let mut stream = tokenizer.token_stream(\"dampfschifffahrt\");\n///     assert_eq!(stream.next().unwrap().text, \"dampf\");\n///     assert_eq!(stream.next().unwrap().text, \"schiff\");\n///     assert_eq!(stream.next().unwrap().text, \"fahrt\");\n///     assert_eq!(stream.next(), None);\n/// }\n/// let mut stream = tokenizer.token_stream(\"brotbackautomat\");\n/// assert_eq!(stream.next().unwrap().text, \"brotbackautomat\");\n/// assert_eq!(stream.next(), None);\n/// ```\n///\n/// [compound]: https://en.wikipedia.org/wiki/Compound_(linguistics)\n#[derive(Clone)]\npub struct SplitCompoundWords {\n    dict: AhoCorasick,\n}\n\nimpl SplitCompoundWords {\n    /// Create a filter from a given dictionary.\n    ///\n    /// The dictionary will be used to construct an [`AhoCorasick`] automaton\n    /// with reasonable defaults. See [`from_automaton`][Self::from_automaton] if\n    /// more control over its construction is required.\n    pub fn from_dictionary<I, P>(dict: I) -> crate::Result<Self>\n    where\n        I: IntoIterator<Item = P>,\n        P: AsRef<[u8]>,\n    {\n        let dict = AhoCorasickBuilder::new()\n            .match_kind(MatchKind::LeftmostLongest)\n            .build(dict)\n            .map_err(|err| {\n                crate::TantivyError::InvalidArgument(format!(\n                    \"Failed to build Aho-Corasick automaton from dictionary: {err}\"\n                ))\n            })?;\n\n        Ok(Self::from_automaton(dict))\n    }\n\n    /// Create a filter from a given automaton.\n    ///\n    /// The automaton should use one of the leftmost-first match kinds\n    /// and it should not be anchored.\n    pub fn from_automaton(dict: AhoCorasick) -> Self {\n        Self { dict }\n    }\n}\n\nimpl TokenFilter for SplitCompoundWords {\n    type Tokenizer<T: Tokenizer> = SplitCompoundWordsFilter<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> SplitCompoundWordsFilter<T> {\n        SplitCompoundWordsFilter {\n            dict: self.dict,\n            inner: tokenizer,\n            cuts: Vec::new(),\n            parts: Vec::new(),\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct SplitCompoundWordsFilter<T> {\n    dict: AhoCorasick,\n    inner: T,\n    cuts: Vec<usize>,\n    parts: Vec<Token>,\n}\n\nimpl<T: Tokenizer> Tokenizer for SplitCompoundWordsFilter<T> {\n    type TokenStream<'a> = SplitCompoundWordsTokenStream<'a, T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        self.cuts.clear();\n        self.parts.clear();\n        SplitCompoundWordsTokenStream {\n            dict: self.dict.clone(),\n            tail: self.inner.token_stream(text),\n            cuts: &mut self.cuts,\n            parts: &mut self.parts,\n        }\n    }\n}\n\npub struct SplitCompoundWordsTokenStream<'a, T> {\n    dict: AhoCorasick,\n    tail: T,\n    cuts: &'a mut Vec<usize>,\n    parts: &'a mut Vec<Token>,\n}\n\nimpl<T: TokenStream> SplitCompoundWordsTokenStream<'_, T> {\n    // Will use `self.cuts` to fill `self.parts` if `self.tail.token()`\n    // can fully be split into consecutive matches against `self.dict`.\n    fn split(&mut self) {\n        let token = self.tail.token();\n        let mut text = token.text.as_str();\n\n        self.cuts.clear();\n        let mut pos = 0;\n\n        for match_ in self.dict.find_iter(text) {\n            if pos != match_.start() {\n                break;\n            }\n\n            self.cuts.push(pos);\n            pos = match_.end();\n        }\n\n        if pos == token.text.len() {\n            // Fill `self.parts` in reverse order,\n            // so that `self.parts.pop()` yields\n            // the tokens in their original order.\n            for pos in self.cuts.iter().rev() {\n                let (head, tail) = text.split_at(*pos);\n\n                text = head;\n                self.parts.push(Token {\n                    text: tail.to_owned(),\n                    ..*token\n                });\n            }\n        }\n    }\n}\n\nimpl<T: TokenStream> TokenStream for SplitCompoundWordsTokenStream<'_, T> {\n    fn advance(&mut self) -> bool {\n        self.parts.pop();\n\n        if !self.parts.is_empty() {\n            return true;\n        }\n\n        if !self.tail.advance() {\n            return false;\n        }\n\n        // Will yield either `self.parts.last()` or\n        // `self.tail.token()` if it could not be split.\n        self.split();\n        true\n    }\n\n    fn token(&self) -> &Token {\n        self.parts.last().unwrap_or_else(|| self.tail.token())\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.parts\n            .last_mut()\n            .unwrap_or_else(|| self.tail.token_mut())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::tokenizer::{SimpleTokenizer, TextAnalyzer};\n\n    #[test]\n    fn splitting_compound_words_works() {\n        let mut tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(SplitCompoundWords::from_dictionary([\"foo\", \"bar\"]).unwrap())\n            .build();\n\n        {\n            let mut stream = tokenizer.token_stream(\"\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foo bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobarbaz\");\n            assert_eq!(stream.next().unwrap().text, \"foobarbaz\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"baz foobar qux\");\n            assert_eq!(stream.next().unwrap().text, \"baz\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"qux\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobar foobar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobar foo bar foobar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobazbar foo bar foobar\");\n            assert_eq!(stream.next().unwrap().text, \"foobazbar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"foobar qux foobar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"qux\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next(), None);\n        }\n\n        {\n            let mut stream = tokenizer.token_stream(\"barfoo\");\n            assert_eq!(stream.next().unwrap().text, \"bar\");\n            assert_eq!(stream.next().unwrap().text, \"foo\");\n            assert_eq!(stream.next(), None);\n        }\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/stemmer.rs",
    "content": "use std::borrow::Cow;\nuse std::mem;\n\nuse rust_stemmers::Algorithm;\nuse serde::{Deserialize, Serialize};\n\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// Available stemmer languages.\n#[derive(Debug, Serialize, Deserialize, Eq, PartialEq, Copy, Clone, Hash)]\n#[allow(missing_docs)]\npub enum Language {\n    Arabic,\n    Danish,\n    Dutch,\n    English,\n    Finnish,\n    French,\n    German,\n    Greek,\n    Hungarian,\n    Italian,\n    Norwegian,\n    Portuguese,\n    Romanian,\n    Russian,\n    Spanish,\n    Swedish,\n    Tamil,\n    Turkish,\n}\n\nimpl Language {\n    fn algorithm(self) -> Algorithm {\n        use self::Language::*;\n        match self {\n            Arabic => Algorithm::Arabic,\n            Danish => Algorithm::Danish,\n            Dutch => Algorithm::Dutch,\n            English => Algorithm::English,\n            Finnish => Algorithm::Finnish,\n            French => Algorithm::French,\n            German => Algorithm::German,\n            Greek => Algorithm::Greek,\n            Hungarian => Algorithm::Hungarian,\n            Italian => Algorithm::Italian,\n            Norwegian => Algorithm::Norwegian,\n            Portuguese => Algorithm::Portuguese,\n            Romanian => Algorithm::Romanian,\n            Russian => Algorithm::Russian,\n            Spanish => Algorithm::Spanish,\n            Swedish => Algorithm::Swedish,\n            Tamil => Algorithm::Tamil,\n            Turkish => Algorithm::Turkish,\n        }\n    }\n}\n\n/// `Stemmer` token filter. Several languages are supported, see [`Language`] for the available\n/// languages.\n/// Tokens are expected to be lowercased beforehand.\n#[derive(Clone)]\npub struct Stemmer {\n    stemmer_algorithm: Algorithm,\n}\n\nimpl Stemmer {\n    /// Creates a new `Stemmer` [`TokenFilter`] for a given language algorithm.\n    pub fn new(language: Language) -> Stemmer {\n        Stemmer {\n            stemmer_algorithm: language.algorithm(),\n        }\n    }\n}\n\nimpl Default for Stemmer {\n    /// Creates a new `Stemmer` [`TokenFilter`] for [`Language::English`].\n    fn default() -> Self {\n        Stemmer::new(Language::English)\n    }\n}\n\nimpl TokenFilter for Stemmer {\n    type Tokenizer<T: Tokenizer> = StemmerFilter<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> StemmerFilter<T> {\n        StemmerFilter {\n            stemmer_algorithm: self.stemmer_algorithm,\n            inner: tokenizer,\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct StemmerFilter<T> {\n    stemmer_algorithm: Algorithm,\n    inner: T,\n}\n\nimpl<T: Tokenizer> Tokenizer for StemmerFilter<T> {\n    type TokenStream<'a> = StemmerTokenStream<T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        let stemmer = rust_stemmers::Stemmer::create(self.stemmer_algorithm);\n        StemmerTokenStream {\n            tail: self.inner.token_stream(text),\n            stemmer,\n            buffer: String::new(),\n        }\n    }\n}\n\npub struct StemmerTokenStream<T> {\n    tail: T,\n    stemmer: rust_stemmers::Stemmer,\n    buffer: String,\n}\n\nimpl<T: TokenStream> TokenStream for StemmerTokenStream<T> {\n    fn advance(&mut self) -> bool {\n        if !self.tail.advance() {\n            return false;\n        }\n        let token = self.tail.token_mut();\n        let stemmed_str = self.stemmer.stem(&token.text);\n        match stemmed_str {\n            Cow::Owned(stemmed_str) => token.text = stemmed_str,\n            Cow::Borrowed(stemmed_str) => {\n                self.buffer.clear();\n                self.buffer.push_str(stemmed_str);\n                mem::swap(&mut token.text, &mut self.buffer);\n            }\n        }\n        true\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use tokenizer_api::Token;\n\n    use super::*;\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{LowerCaser, SimpleTokenizer, TextAnalyzer, TokenizerManager};\n\n    #[test]\n    fn test_en_stem() {\n        let tokenizer_manager = TokenizerManager::default();\n        let mut en_tokenizer = tokenizer_manager.get(\"en_stem\").unwrap();\n        let mut tokens: Vec<Token> = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                tokens.push(token.clone());\n            };\n            en_tokenizer\n                .token_stream(\"Dogs are the bests!\")\n                .process(&mut add_token);\n        }\n\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"dog\", 0, 4);\n        assert_token(&tokens[1], 1, \"are\", 5, 8);\n        assert_token(&tokens[2], 2, \"the\", 9, 12);\n        assert_token(&tokens[3], 3, \"best\", 13, 18);\n    }\n\n    #[test]\n    fn test_non_en_stem() {\n        let tokenizer_manager = TokenizerManager::default();\n        tokenizer_manager.register(\n            \"el_stem\",\n            TextAnalyzer::builder(SimpleTokenizer::default())\n                .filter(LowerCaser)\n                .filter(Stemmer::new(Language::Greek))\n                .build(),\n        );\n        let mut el_tokenizer = tokenizer_manager.get(\"el_stem\").unwrap();\n        let mut tokens: Vec<Token> = vec![];\n        {\n            let mut add_token = |token: &Token| {\n                tokens.push(token.clone());\n            };\n            el_tokenizer\n                .token_stream(\"Καλημέρα, χαρούμενε φορολογούμενε!\")\n                .process(&mut add_token);\n        }\n\n        assert_eq!(tokens.len(), 3);\n        assert_token(&tokens[0], 0, \"καλημερ\", 0, 16);\n        assert_token(&tokens[1], 1, \"χαρουμεν\", 18, 36);\n        assert_token(&tokens[2], 2, \"φορολογουμεν\", 37, 63);\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/stop_word_filter/gen_stopwords.py",
    "content": "import requests\n\nLANGUAGES = [\n    \"danish\",\n    \"dutch\",\n    \"finnish\",\n    \"french\",\n    \"german\",\n    \"hungarian\",\n    \"italian\",\n    \"norwegian\",\n    \"portuguese\",\n    \"russian\",\n    \"spanish\",\n    \"swedish\",\n]\n\nwith requests.Session() as sess, open(\"stopwords.rs\", \"w\") as mod:\n    mod.write(\"/*\\n\")\n    mod.write(\n        \"These stop word lists are from the Snowball project (https://snowballstem.org/)\\nwhich carries the following copyright and license:\\n\\n\"\n    )\n\n    resp = sess.get(\n        \"https://raw.githubusercontent.com/snowballstem/snowball/master/COPYING\"\n    )\n    resp.raise_for_status()\n    mod.write(resp.text)\n    mod.write(\"*/\\n\\n\")\n\n    for lang in LANGUAGES:\n        resp = sess.get(f\"https://snowballstem.org/algorithms/{lang}/stop.txt\")\n        resp.raise_for_status()\n\n        mod.write(f\"pub const {lang.upper()}: &[&str] = &[\\n\")\n\n        for line in resp.text.splitlines():\n            line, _, _ = line.partition(\"|\")\n\n            for word in line.split():\n                mod.write(f'    \"{word}\",\\n')\n\n        mod.write(\"];\\n\\n\")\n"
  },
  {
    "path": "src/tokenizer/stop_word_filter/mod.rs",
    "content": "//! # Example\n//! ```rust\n//! use tantivy::tokenizer::*;\n//!\n//! let mut tokenizer = TextAnalyzer::builder(SimpleTokenizer::default())\n//!   .filter(StopWordFilter::remove(vec![\"the\".to_string(), \"is\".to_string()]))\n//!   .build();\n//!\n//! let mut stream = tokenizer.token_stream(\"the fox is crafty\");\n//! assert_eq!(stream.next().unwrap().text, \"fox\");\n//! assert_eq!(stream.next().unwrap().text, \"crafty\");\n//! assert!(stream.next().is_none());\n//! ```\n#[cfg(feature = \"stopwords\")]\n#[rustfmt::skip]\nmod stopwords;\n\nuse std::sync::Arc;\n\nuse rustc_hash::FxHashSet;\n\n#[cfg(feature = \"stopwords\")]\nuse super::Language;\nuse super::{Token, TokenFilter, TokenStream, Tokenizer};\n\n/// `TokenFilter` that removes stop words from a token stream\n#[derive(Clone)]\npub struct StopWordFilter {\n    words: Arc<FxHashSet<String>>,\n}\n\nimpl StopWordFilter {\n    /// Creates a new [`StopWordFilter`] for the given [`Language`]\n    ///\n    /// Returns `Some` if a list of stop words is available and `None` otherwise.\n    #[cfg(feature = \"stopwords\")]\n    pub fn new(language: Language) -> Option<Self> {\n        let words = match language {\n            Language::Danish => stopwords::DANISH,\n            Language::Dutch => stopwords::DUTCH,\n            Language::English => {\n                // This is the same list of words used by the Apache-licensed Lucene project,\n                // c.f. https://github.com/apache/lucene/blob/d5d6dc079395c47cd6d12dcce3bcfdd2c7d9dc63/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java#L46\n                &[\n                    \"a\", \"an\", \"and\", \"are\", \"as\", \"at\", \"be\", \"but\", \"by\", \"for\", \"if\", \"in\",\n                    \"into\", \"is\", \"it\", \"no\", \"not\", \"of\", \"on\", \"or\", \"such\", \"that\", \"the\",\n                    \"their\", \"then\", \"there\", \"these\", \"they\", \"this\", \"to\", \"was\", \"will\", \"with\",\n                ]\n            }\n            Language::Finnish => stopwords::FINNISH,\n            Language::French => stopwords::FRENCH,\n            Language::German => stopwords::GERMAN,\n            Language::Hungarian => stopwords::HUNGARIAN,\n            Language::Italian => stopwords::ITALIAN,\n            Language::Norwegian => stopwords::NORWEGIAN,\n            Language::Portuguese => stopwords::PORTUGUESE,\n            Language::Russian => stopwords::RUSSIAN,\n            Language::Spanish => stopwords::SPANISH,\n            Language::Swedish => stopwords::SWEDISH,\n            _ => return None,\n        };\n\n        Some(Self::remove(words.iter().map(|&word| word.to_owned())))\n    }\n\n    /// Creates a `StopWordFilter` given a list of words to remove\n    pub fn remove<W: IntoIterator<Item = String>>(words: W) -> StopWordFilter {\n        StopWordFilter {\n            words: Arc::new(words.into_iter().collect()),\n        }\n    }\n}\n\nimpl TokenFilter for StopWordFilter {\n    type Tokenizer<T: Tokenizer> = StopWordFilterWrapper<T>;\n\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> StopWordFilterWrapper<T> {\n        StopWordFilterWrapper {\n            words: self.words,\n            inner: tokenizer,\n        }\n    }\n}\n\n#[derive(Clone)]\npub struct StopWordFilterWrapper<T> {\n    words: Arc<FxHashSet<String>>,\n    inner: T,\n}\n\nimpl<T: Tokenizer> Tokenizer for StopWordFilterWrapper<T> {\n    type TokenStream<'a> = StopWordFilterStream<T::TokenStream<'a>>;\n\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        StopWordFilterStream {\n            words: self.words.clone(),\n            tail: self.inner.token_stream(text),\n        }\n    }\n}\n\npub struct StopWordFilterStream<T> {\n    words: Arc<FxHashSet<String>>,\n    tail: T,\n}\n\nimpl<T> StopWordFilterStream<T> {\n    fn predicate(&self, token: &Token) -> bool {\n        !self.words.contains(&token.text)\n    }\n}\n\nimpl<T: TokenStream> TokenStream for StopWordFilterStream<T> {\n    fn advance(&mut self) -> bool {\n        while self.tail.advance() {\n            if self.predicate(self.tail.token()) {\n                return true;\n            }\n        }\n        false\n    }\n\n    fn token(&self) -> &Token {\n        self.tail.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.tail.token_mut()\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{SimpleTokenizer, StopWordFilter, TextAnalyzer, Token};\n\n    #[test]\n    fn test_stop_word() {\n        let tokens = token_stream_helper(\"i am a cat. as yet i have no name.\");\n        assert_eq!(tokens.len(), 5);\n        assert_token(&tokens[0], 3, \"cat\", 7, 10);\n        assert_token(&tokens[1], 5, \"yet\", 15, 18);\n        assert_token(&tokens[2], 7, \"have\", 21, 25);\n        assert_token(&tokens[3], 8, \"no\", 26, 28);\n        assert_token(&tokens[4], 9, \"name\", 29, 33);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let stops = vec![\n            \"a\".to_string(),\n            \"as\".to_string(),\n            \"am\".to_string(),\n            \"i\".to_string(),\n        ];\n        let mut a = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(StopWordFilter::remove(stops))\n            .build();\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/stop_word_filter/stopwords.rs",
    "content": "/*\nThese stop word lists are from the Snowball project (https://snowballstem.org/)\nwhich carries the following copyright and license:\n\nCopyright (c) 2001, Dr Martin Porter\nCopyright (c) 2004,2005, Richard Boulton\nCopyright (c) 2013, Yoshiki Shibukawa\nCopyright (c) 2006,2007,2009,2010,2011,2014-2019, Olly Betts\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions\nare met:\n\n  1. Redistributions of source code must retain the above copyright notice,\n     this list of conditions and the following disclaimer.\n  2. Redistributions in binary form must reproduce the above copyright notice,\n     this list of conditions and the following disclaimer in the documentation\n     and/or other materials provided with the distribution.\n  3. Neither the name of the Snowball project nor the names of its contributors\n     may be used to endorse or promote products derived from this software\n     without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n*/\n\npub const DANISH: &[&str] = &[\n    \"og\",\n    \"i\",\n    \"jeg\",\n    \"det\",\n    \"at\",\n    \"en\",\n    \"den\",\n    \"til\",\n    \"er\",\n    \"som\",\n    \"på\",\n    \"de\",\n    \"med\",\n    \"han\",\n    \"af\",\n    \"for\",\n    \"ikke\",\n    \"der\",\n    \"var\",\n    \"mig\",\n    \"sig\",\n    \"men\",\n    \"et\",\n    \"har\",\n    \"om\",\n    \"vi\",\n    \"min\",\n    \"havde\",\n    \"ham\",\n    \"hun\",\n    \"nu\",\n    \"over\",\n    \"da\",\n    \"fra\",\n    \"du\",\n    \"ud\",\n    \"sin\",\n    \"dem\",\n    \"os\",\n    \"op\",\n    \"man\",\n    \"hans\",\n    \"hvor\",\n    \"eller\",\n    \"hvad\",\n    \"skal\",\n    \"selv\",\n    \"her\",\n    \"alle\",\n    \"vil\",\n    \"blev\",\n    \"kunne\",\n    \"ind\",\n    \"når\",\n    \"være\",\n    \"dog\",\n    \"noget\",\n    \"ville\",\n    \"jo\",\n    \"deres\",\n    \"efter\",\n    \"ned\",\n    \"skulle\",\n    \"denne\",\n    \"end\",\n    \"dette\",\n    \"mit\",\n    \"også\",\n    \"under\",\n    \"have\",\n    \"dig\",\n    \"anden\",\n    \"hende\",\n    \"mine\",\n    \"alt\",\n    \"meget\",\n    \"sit\",\n    \"sine\",\n    \"vor\",\n    \"mod\",\n    \"disse\",\n    \"hvis\",\n    \"din\",\n    \"nogle\",\n    \"hos\",\n    \"blive\",\n    \"mange\",\n    \"ad\",\n    \"bliver\",\n    \"hendes\",\n    \"været\",\n    \"thi\",\n    \"jer\",\n    \"sådan\",\n];\n\npub const DUTCH: &[&str] = &[\n    \"de\",\n    \"en\",\n    \"van\",\n    \"ik\",\n    \"te\",\n    \"dat\",\n    \"die\",\n    \"in\",\n    \"een\",\n    \"hij\",\n    \"het\",\n    \"niet\",\n    \"zijn\",\n    \"is\",\n    \"was\",\n    \"op\",\n    \"aan\",\n    \"met\",\n    \"als\",\n    \"voor\",\n    \"had\",\n    \"er\",\n    \"maar\",\n    \"om\",\n    \"hem\",\n    \"dan\",\n    \"zou\",\n    \"of\",\n    \"wat\",\n    \"mijn\",\n    \"men\",\n    \"dit\",\n    \"zo\",\n    \"door\",\n    \"over\",\n    \"ze\",\n    \"zich\",\n    \"bij\",\n    \"ook\",\n    \"tot\",\n    \"je\",\n    \"mij\",\n    \"uit\",\n    \"der\",\n    \"daar\",\n    \"haar\",\n    \"naar\",\n    \"heb\",\n    \"hoe\",\n    \"heeft\",\n    \"hebben\",\n    \"deze\",\n    \"u\",\n    \"want\",\n    \"nog\",\n    \"zal\",\n    \"me\",\n    \"zij\",\n    \"nu\",\n    \"ge\",\n    \"geen\",\n    \"omdat\",\n    \"iets\",\n    \"worden\",\n    \"toch\",\n    \"al\",\n    \"waren\",\n    \"veel\",\n    \"meer\",\n    \"doen\",\n    \"toen\",\n    \"moet\",\n    \"ben\",\n    \"zonder\",\n    \"kan\",\n    \"hun\",\n    \"dus\",\n    \"alles\",\n    \"onder\",\n    \"ja\",\n    \"eens\",\n    \"hier\",\n    \"wie\",\n    \"werd\",\n    \"altijd\",\n    \"doch\",\n    \"wordt\",\n    \"wezen\",\n    \"kunnen\",\n    \"ons\",\n    \"zelf\",\n    \"tegen\",\n    \"na\",\n    \"reeds\",\n    \"wil\",\n    \"kon\",\n    \"niets\",\n    \"uw\",\n    \"iemand\",\n    \"geweest\",\n    \"andere\",\n];\n\npub const FINNISH: &[&str] = &[\n    \"olla\",\n    \"olen\",\n    \"olet\",\n    \"on\",\n    \"olemme\",\n    \"olette\",\n    \"ovat\",\n    \"ole\",\n    \"oli\",\n    \"olisi\",\n    \"olisit\",\n    \"olisin\",\n    \"olisimme\",\n    \"olisitte\",\n    \"olisivat\",\n    \"olit\",\n    \"olin\",\n    \"olimme\",\n    \"olitte\",\n    \"olivat\",\n    \"ollut\",\n    \"olleet\",\n    \"en\",\n    \"et\",\n    \"ei\",\n    \"emme\",\n    \"ette\",\n    \"eivät\",\n    \"minä\",\n    \"minun\",\n    \"minut\",\n    \"minua\",\n    \"minussa\",\n    \"minusta\",\n    \"minuun\",\n    \"minulla\",\n    \"minulta\",\n    \"minulle\",\n    \"sinä\",\n    \"sinun\",\n    \"sinut\",\n    \"sinua\",\n    \"sinussa\",\n    \"sinusta\",\n    \"sinuun\",\n    \"sinulla\",\n    \"sinulta\",\n    \"sinulle\",\n    \"hän\",\n    \"hänen\",\n    \"hänet\",\n    \"häntä\",\n    \"hänessä\",\n    \"hänestä\",\n    \"häneen\",\n    \"hänellä\",\n    \"häneltä\",\n    \"hänelle\",\n    \"me\",\n    \"meidän\",\n    \"meidät\",\n    \"meitä\",\n    \"meissä\",\n    \"meistä\",\n    \"meihin\",\n    \"meillä\",\n    \"meiltä\",\n    \"meille\",\n    \"te\",\n    \"teidän\",\n    \"teidät\",\n    \"teitä\",\n    \"teissä\",\n    \"teistä\",\n    \"teihin\",\n    \"teillä\",\n    \"teiltä\",\n    \"teille\",\n    \"he\",\n    \"heidän\",\n    \"heidät\",\n    \"heitä\",\n    \"heissä\",\n    \"heistä\",\n    \"heihin\",\n    \"heillä\",\n    \"heiltä\",\n    \"heille\",\n    \"tämä\",\n    \"tämän\",\n    \"tätä\",\n    \"tässä\",\n    \"tästä\",\n    \"tähän\",\n    \"tällä\",\n    \"tältä\",\n    \"tälle\",\n    \"tänä\",\n    \"täksi\",\n    \"tuo\",\n    \"tuon\",\n    \"tuota\",\n    \"tuossa\",\n    \"tuosta\",\n    \"tuohon\",\n    \"tuolla\",\n    \"tuolta\",\n    \"tuolle\",\n    \"tuona\",\n    \"tuoksi\",\n    \"se\",\n    \"sen\",\n    \"sitä\",\n    \"siinä\",\n    \"siitä\",\n    \"siihen\",\n    \"sillä\",\n    \"siltä\",\n    \"sille\",\n    \"sinä\",\n    \"siksi\",\n    \"nämä\",\n    \"näiden\",\n    \"näitä\",\n    \"näissä\",\n    \"näistä\",\n    \"näihin\",\n    \"näillä\",\n    \"näiltä\",\n    \"näille\",\n    \"näinä\",\n    \"näiksi\",\n    \"nuo\",\n    \"noiden\",\n    \"noita\",\n    \"noissa\",\n    \"noista\",\n    \"noihin\",\n    \"noilla\",\n    \"noilta\",\n    \"noille\",\n    \"noina\",\n    \"noiksi\",\n    \"ne\",\n    \"niiden\",\n    \"niitä\",\n    \"niissä\",\n    \"niistä\",\n    \"niihin\",\n    \"niillä\",\n    \"niiltä\",\n    \"niille\",\n    \"niinä\",\n    \"niiksi\",\n    \"kuka\",\n    \"kenen\",\n    \"kenet\",\n    \"ketä\",\n    \"kenessä\",\n    \"kenestä\",\n    \"keneen\",\n    \"kenellä\",\n    \"keneltä\",\n    \"kenelle\",\n    \"kenenä\",\n    \"keneksi\",\n    \"ketkä\",\n    \"keiden\",\n    \"ketkä\",\n    \"keitä\",\n    \"keissä\",\n    \"keistä\",\n    \"keihin\",\n    \"keillä\",\n    \"keiltä\",\n    \"keille\",\n    \"keinä\",\n    \"keiksi\",\n    \"mikä\",\n    \"minkä\",\n    \"minkä\",\n    \"mitä\",\n    \"missä\",\n    \"mistä\",\n    \"mihin\",\n    \"millä\",\n    \"miltä\",\n    \"mille\",\n    \"minä\",\n    \"miksi\",\n    \"mitkä\",\n    \"joka\",\n    \"jonka\",\n    \"jota\",\n    \"jossa\",\n    \"josta\",\n    \"johon\",\n    \"jolla\",\n    \"jolta\",\n    \"jolle\",\n    \"jona\",\n    \"joksi\",\n    \"jotka\",\n    \"joiden\",\n    \"joita\",\n    \"joissa\",\n    \"joista\",\n    \"joihin\",\n    \"joilla\",\n    \"joilta\",\n    \"joille\",\n    \"joina\",\n    \"joiksi\",\n    \"että\",\n    \"ja\",\n    \"jos\",\n    \"koska\",\n    \"kuin\",\n    \"mutta\",\n    \"niin\",\n    \"sekä\",\n    \"sillä\",\n    \"tai\",\n    \"vaan\",\n    \"vai\",\n    \"vaikka\",\n    \"kanssa\",\n    \"mukaan\",\n    \"noin\",\n    \"poikki\",\n    \"yli\",\n    \"kun\",\n    \"nyt\",\n    \"itse\",\n];\n\npub const FRENCH: &[&str] = &[\n    \"au\",\n    \"aux\",\n    \"avec\",\n    \"ce\",\n    \"ces\",\n    \"dans\",\n    \"de\",\n    \"des\",\n    \"du\",\n    \"elle\",\n    \"en\",\n    \"et\",\n    \"eux\",\n    \"il\",\n    \"je\",\n    \"la\",\n    \"le\",\n    \"leur\",\n    \"lui\",\n    \"ma\",\n    \"mais\",\n    \"me\",\n    \"même\",\n    \"mes\",\n    \"moi\",\n    \"mon\",\n    \"ne\",\n    \"nos\",\n    \"notre\",\n    \"nous\",\n    \"on\",\n    \"ou\",\n    \"par\",\n    \"pas\",\n    \"pour\",\n    \"qu\",\n    \"que\",\n    \"qui\",\n    \"sa\",\n    \"se\",\n    \"ses\",\n    \"sur\",\n    \"ta\",\n    \"te\",\n    \"tes\",\n    \"toi\",\n    \"ton\",\n    \"tu\",\n    \"un\",\n    \"une\",\n    \"vos\",\n    \"votre\",\n    \"vous\",\n    \"c\",\n    \"d\",\n    \"j\",\n    \"l\",\n    \"à\",\n    \"m\",\n    \"n\",\n    \"s\",\n    \"t\",\n    \"y\",\n    \"étée\",\n    \"étées\",\n    \"étant\",\n    \"suis\",\n    \"es\",\n    \"êtes\",\n    \"sont\",\n    \"serai\",\n    \"seras\",\n    \"sera\",\n    \"serons\",\n    \"serez\",\n    \"seront\",\n    \"serais\",\n    \"serait\",\n    \"serions\",\n    \"seriez\",\n    \"seraient\",\n    \"étais\",\n    \"était\",\n    \"étions\",\n    \"étiez\",\n    \"étaient\",\n    \"fus\",\n    \"fut\",\n    \"fûmes\",\n    \"fûtes\",\n    \"furent\",\n    \"sois\",\n    \"soit\",\n    \"soyons\",\n    \"soyez\",\n    \"soient\",\n    \"fusse\",\n    \"fusses\",\n    \"fussions\",\n    \"fussiez\",\n    \"fussent\",\n    \"ayant\",\n    \"eu\",\n    \"eue\",\n    \"eues\",\n    \"eus\",\n    \"ai\",\n    \"avons\",\n    \"avez\",\n    \"ont\",\n    \"aurai\",\n    \"aurons\",\n    \"aurez\",\n    \"auront\",\n    \"aurais\",\n    \"aurait\",\n    \"aurions\",\n    \"auriez\",\n    \"auraient\",\n    \"avais\",\n    \"avait\",\n    \"aviez\",\n    \"avaient\",\n    \"eut\",\n    \"eûmes\",\n    \"eûtes\",\n    \"eurent\",\n    \"aie\",\n    \"aies\",\n    \"ait\",\n    \"ayons\",\n    \"ayez\",\n    \"aient\",\n    \"eusse\",\n    \"eusses\",\n    \"eût\",\n    \"eussions\",\n    \"eussiez\",\n    \"eussent\",\n    \"ceci\",\n    \"cela\",\n    \"celà\",\n    \"cet\",\n    \"cette\",\n    \"ici\",\n    \"ils\",\n    \"les\",\n    \"leurs\",\n    \"quel\",\n    \"quels\",\n    \"quelle\",\n    \"quelles\",\n    \"sans\",\n    \"soi\",\n];\n\npub const GERMAN: &[&str] = &[\n    \"aber\",\n    \"alle\",\n    \"allem\",\n    \"allen\",\n    \"aller\",\n    \"alles\",\n    \"als\",\n    \"also\",\n    \"am\",\n    \"an\",\n    \"ander\",\n    \"andere\",\n    \"anderem\",\n    \"anderen\",\n    \"anderer\",\n    \"anderes\",\n    \"anderm\",\n    \"andern\",\n    \"anderr\",\n    \"anders\",\n    \"auch\",\n    \"auf\",\n    \"aus\",\n    \"bei\",\n    \"bin\",\n    \"bis\",\n    \"bist\",\n    \"da\",\n    \"damit\",\n    \"dann\",\n    \"der\",\n    \"den\",\n    \"des\",\n    \"dem\",\n    \"die\",\n    \"das\",\n    \"daß\",\n    \"derselbe\",\n    \"derselben\",\n    \"denselben\",\n    \"desselben\",\n    \"demselben\",\n    \"dieselbe\",\n    \"dieselben\",\n    \"dasselbe\",\n    \"dazu\",\n    \"dein\",\n    \"deine\",\n    \"deinem\",\n    \"deinen\",\n    \"deiner\",\n    \"deines\",\n    \"denn\",\n    \"derer\",\n    \"dessen\",\n    \"dich\",\n    \"dir\",\n    \"du\",\n    \"dies\",\n    \"diese\",\n    \"diesem\",\n    \"diesen\",\n    \"dieser\",\n    \"dieses\",\n    \"doch\",\n    \"dort\",\n    \"durch\",\n    \"ein\",\n    \"eine\",\n    \"einem\",\n    \"einen\",\n    \"einer\",\n    \"eines\",\n    \"einig\",\n    \"einige\",\n    \"einigem\",\n    \"einigen\",\n    \"einiger\",\n    \"einiges\",\n    \"einmal\",\n    \"er\",\n    \"ihn\",\n    \"ihm\",\n    \"es\",\n    \"etwas\",\n    \"euer\",\n    \"eure\",\n    \"eurem\",\n    \"euren\",\n    \"eurer\",\n    \"eures\",\n    \"für\",\n    \"gegen\",\n    \"gewesen\",\n    \"hab\",\n    \"habe\",\n    \"haben\",\n    \"hat\",\n    \"hatte\",\n    \"hatten\",\n    \"hier\",\n    \"hin\",\n    \"hinter\",\n    \"ich\",\n    \"mich\",\n    \"mir\",\n    \"ihr\",\n    \"ihre\",\n    \"ihrem\",\n    \"ihren\",\n    \"ihrer\",\n    \"ihres\",\n    \"euch\",\n    \"im\",\n    \"in\",\n    \"indem\",\n    \"ins\",\n    \"ist\",\n    \"jede\",\n    \"jedem\",\n    \"jeden\",\n    \"jeder\",\n    \"jedes\",\n    \"jene\",\n    \"jenem\",\n    \"jenen\",\n    \"jener\",\n    \"jenes\",\n    \"jetzt\",\n    \"kann\",\n    \"kein\",\n    \"keine\",\n    \"keinem\",\n    \"keinen\",\n    \"keiner\",\n    \"keines\",\n    \"können\",\n    \"könnte\",\n    \"machen\",\n    \"man\",\n    \"manche\",\n    \"manchem\",\n    \"manchen\",\n    \"mancher\",\n    \"manches\",\n    \"mein\",\n    \"meine\",\n    \"meinem\",\n    \"meinen\",\n    \"meiner\",\n    \"meines\",\n    \"mit\",\n    \"muss\",\n    \"musste\",\n    \"nach\",\n    \"nicht\",\n    \"nichts\",\n    \"noch\",\n    \"nun\",\n    \"nur\",\n    \"ob\",\n    \"oder\",\n    \"ohne\",\n    \"sehr\",\n    \"sein\",\n    \"seine\",\n    \"seinem\",\n    \"seinen\",\n    \"seiner\",\n    \"seines\",\n    \"selbst\",\n    \"sich\",\n    \"sie\",\n    \"ihnen\",\n    \"sind\",\n    \"so\",\n    \"solche\",\n    \"solchem\",\n    \"solchen\",\n    \"solcher\",\n    \"solches\",\n    \"soll\",\n    \"sollte\",\n    \"sondern\",\n    \"sonst\",\n    \"über\",\n    \"um\",\n    \"und\",\n    \"uns\",\n    \"unse\",\n    \"unsem\",\n    \"unsen\",\n    \"unser\",\n    \"unses\",\n    \"unter\",\n    \"viel\",\n    \"vom\",\n    \"von\",\n    \"vor\",\n    \"während\",\n    \"war\",\n    \"waren\",\n    \"warst\",\n    \"was\",\n    \"weg\",\n    \"weil\",\n    \"weiter\",\n    \"welche\",\n    \"welchem\",\n    \"welchen\",\n    \"welcher\",\n    \"welches\",\n    \"wenn\",\n    \"werde\",\n    \"werden\",\n    \"wie\",\n    \"wieder\",\n    \"will\",\n    \"wir\",\n    \"wird\",\n    \"wirst\",\n    \"wo\",\n    \"wollen\",\n    \"wollte\",\n    \"würde\",\n    \"würden\",\n    \"zu\",\n    \"zum\",\n    \"zur\",\n    \"zwar\",\n    \"zwischen\",\n];\n\npub const HUNGARIAN: &[&str] = &[\n    \"a\",\n    \"ahogy\",\n    \"ahol\",\n    \"aki\",\n    \"akik\",\n    \"akkor\",\n    \"alatt\",\n    \"által\",\n    \"általában\",\n    \"amely\",\n    \"amelyek\",\n    \"amelyekben\",\n    \"amelyeket\",\n    \"amelyet\",\n    \"amelynek\",\n    \"ami\",\n    \"amit\",\n    \"amolyan\",\n    \"amíg\",\n    \"amikor\",\n    \"át\",\n    \"abban\",\n    \"ahhoz\",\n    \"annak\",\n    \"arra\",\n    \"arról\",\n    \"az\",\n    \"azok\",\n    \"azon\",\n    \"azt\",\n    \"azzal\",\n    \"azért\",\n    \"aztán\",\n    \"azután\",\n    \"azonban\",\n    \"bár\",\n    \"be\",\n    \"belül\",\n    \"benne\",\n    \"cikk\",\n    \"cikkek\",\n    \"cikkeket\",\n    \"csak\",\n    \"de\",\n    \"e\",\n    \"eddig\",\n    \"egész\",\n    \"egy\",\n    \"egyes\",\n    \"egyetlen\",\n    \"egyéb\",\n    \"egyik\",\n    \"egyre\",\n    \"ekkor\",\n    \"el\",\n    \"elég\",\n    \"ellen\",\n    \"elő\",\n    \"először\",\n    \"előtt\",\n    \"első\",\n    \"én\",\n    \"éppen\",\n    \"ebben\",\n    \"ehhez\",\n    \"emilyen\",\n    \"ennek\",\n    \"erre\",\n    \"ez\",\n    \"ezt\",\n    \"ezek\",\n    \"ezen\",\n    \"ezzel\",\n    \"ezért\",\n    \"és\",\n    \"fel\",\n    \"felé\",\n    \"hanem\",\n    \"hiszen\",\n    \"hogy\",\n    \"hogyan\",\n    \"igen\",\n    \"így\",\n    \"illetve\",\n    \"ill.\",\n    \"ill\",\n    \"ilyen\",\n    \"ilyenkor\",\n    \"ison\",\n    \"ismét\",\n    \"itt\",\n    \"jó\",\n    \"jól\",\n    \"jobban\",\n    \"kell\",\n    \"kellett\",\n    \"keresztül\",\n    \"keressünk\",\n    \"ki\",\n    \"kívül\",\n    \"között\",\n    \"közül\",\n    \"legalább\",\n    \"lehet\",\n    \"lehetett\",\n    \"legyen\",\n    \"lenne\",\n    \"lenni\",\n    \"lesz\",\n    \"lett\",\n    \"maga\",\n    \"magát\",\n    \"majd\",\n    \"majd\",\n    \"már\",\n    \"más\",\n    \"másik\",\n    \"meg\",\n    \"még\",\n    \"mellett\",\n    \"mert\",\n    \"mely\",\n    \"melyek\",\n    \"mi\",\n    \"mit\",\n    \"míg\",\n    \"miért\",\n    \"milyen\",\n    \"mikor\",\n    \"minden\",\n    \"mindent\",\n    \"mindenki\",\n    \"mindig\",\n    \"mint\",\n    \"mintha\",\n    \"mivel\",\n    \"most\",\n    \"nagy\",\n    \"nagyobb\",\n    \"nagyon\",\n    \"ne\",\n    \"néha\",\n    \"nekem\",\n    \"neki\",\n    \"nem\",\n    \"néhány\",\n    \"nélkül\",\n    \"nincs\",\n    \"olyan\",\n    \"ott\",\n    \"össze\",\n    \"ő\",\n    \"ők\",\n    \"őket\",\n    \"pedig\",\n    \"persze\",\n    \"rá\",\n    \"s\",\n    \"saját\",\n    \"sem\",\n    \"semmi\",\n    \"sok\",\n    \"sokat\",\n    \"sokkal\",\n    \"számára\",\n    \"szemben\",\n    \"szerint\",\n    \"szinte\",\n    \"talán\",\n    \"tehát\",\n    \"teljes\",\n    \"tovább\",\n    \"továbbá\",\n    \"több\",\n    \"úgy\",\n    \"ugyanis\",\n    \"új\",\n    \"újabb\",\n    \"újra\",\n    \"után\",\n    \"utána\",\n    \"utolsó\",\n    \"vagy\",\n    \"vagyis\",\n    \"valaki\",\n    \"valami\",\n    \"valamint\",\n    \"való\",\n    \"vagyok\",\n    \"van\",\n    \"vannak\",\n    \"volt\",\n    \"voltam\",\n    \"voltak\",\n    \"voltunk\",\n    \"vissza\",\n    \"vele\",\n    \"viszont\",\n    \"volna\",\n];\n\npub const ITALIAN: &[&str] = &[\n    \"ad\",\n    \"al\",\n    \"allo\",\n    \"ai\",\n    \"agli\",\n    \"all\",\n    \"agl\",\n    \"alla\",\n    \"alle\",\n    \"con\",\n    \"col\",\n    \"coi\",\n    \"da\",\n    \"dal\",\n    \"dallo\",\n    \"dai\",\n    \"dagli\",\n    \"dall\",\n    \"dagl\",\n    \"dalla\",\n    \"dalle\",\n    \"di\",\n    \"del\",\n    \"dello\",\n    \"dei\",\n    \"degli\",\n    \"dell\",\n    \"degl\",\n    \"della\",\n    \"delle\",\n    \"in\",\n    \"nel\",\n    \"nello\",\n    \"nei\",\n    \"negli\",\n    \"nell\",\n    \"negl\",\n    \"nella\",\n    \"nelle\",\n    \"su\",\n    \"sul\",\n    \"sullo\",\n    \"sui\",\n    \"sugli\",\n    \"sull\",\n    \"sugl\",\n    \"sulla\",\n    \"sulle\",\n    \"per\",\n    \"tra\",\n    \"contro\",\n    \"io\",\n    \"tu\",\n    \"lui\",\n    \"lei\",\n    \"noi\",\n    \"voi\",\n    \"loro\",\n    \"mio\",\n    \"mia\",\n    \"miei\",\n    \"mie\",\n    \"tuo\",\n    \"tua\",\n    \"tuoi\",\n    \"tue\",\n    \"suo\",\n    \"sua\",\n    \"suoi\",\n    \"sue\",\n    \"nostro\",\n    \"nostra\",\n    \"nostri\",\n    \"nostre\",\n    \"vostro\",\n    \"vostra\",\n    \"vostri\",\n    \"vostre\",\n    \"mi\",\n    \"ti\",\n    \"ci\",\n    \"vi\",\n    \"lo\",\n    \"la\",\n    \"li\",\n    \"le\",\n    \"gli\",\n    \"ne\",\n    \"il\",\n    \"un\",\n    \"uno\",\n    \"una\",\n    \"ma\",\n    \"ed\",\n    \"se\",\n    \"perché\",\n    \"anche\",\n    \"come\",\n    \"dov\",\n    \"dove\",\n    \"che\",\n    \"chi\",\n    \"cui\",\n    \"non\",\n    \"più\",\n    \"quale\",\n    \"quanto\",\n    \"quanti\",\n    \"quanta\",\n    \"quante\",\n    \"quello\",\n    \"quelli\",\n    \"quella\",\n    \"quelle\",\n    \"questo\",\n    \"questi\",\n    \"questa\",\n    \"queste\",\n    \"si\",\n    \"tutto\",\n    \"tutti\",\n    \"a\",\n    \"c\",\n    \"e\",\n    \"i\",\n    \"l\",\n    \"o\",\n    \"ho\",\n    \"hai\",\n    \"ha\",\n    \"abbiamo\",\n    \"avete\",\n    \"hanno\",\n    \"abbia\",\n    \"abbiate\",\n    \"abbiano\",\n    \"avrò\",\n    \"avrai\",\n    \"avrà\",\n    \"avremo\",\n    \"avrete\",\n    \"avranno\",\n    \"avrei\",\n    \"avresti\",\n    \"avrebbe\",\n    \"avremmo\",\n    \"avreste\",\n    \"avrebbero\",\n    \"avevo\",\n    \"avevi\",\n    \"aveva\",\n    \"avevamo\",\n    \"avevate\",\n    \"avevano\",\n    \"ebbi\",\n    \"avesti\",\n    \"ebbe\",\n    \"avemmo\",\n    \"aveste\",\n    \"ebbero\",\n    \"avessi\",\n    \"avesse\",\n    \"avessimo\",\n    \"avessero\",\n    \"avendo\",\n    \"avuto\",\n    \"avuta\",\n    \"avuti\",\n    \"avute\",\n    \"sono\",\n    \"sei\",\n    \"è\",\n    \"siamo\",\n    \"siete\",\n    \"sia\",\n    \"siate\",\n    \"siano\",\n    \"sarò\",\n    \"sarai\",\n    \"sarà\",\n    \"saremo\",\n    \"sarete\",\n    \"saranno\",\n    \"sarei\",\n    \"saresti\",\n    \"sarebbe\",\n    \"saremmo\",\n    \"sareste\",\n    \"sarebbero\",\n    \"ero\",\n    \"eri\",\n    \"era\",\n    \"eravamo\",\n    \"eravate\",\n    \"erano\",\n    \"fui\",\n    \"fosti\",\n    \"fu\",\n    \"fummo\",\n    \"foste\",\n    \"furono\",\n    \"fossi\",\n    \"fosse\",\n    \"fossimo\",\n    \"fossero\",\n    \"essendo\",\n    \"faccio\",\n    \"fai\",\n    \"facciamo\",\n    \"fanno\",\n    \"faccia\",\n    \"facciate\",\n    \"facciano\",\n    \"farò\",\n    \"farai\",\n    \"farà\",\n    \"faremo\",\n    \"farete\",\n    \"faranno\",\n    \"farei\",\n    \"faresti\",\n    \"farebbe\",\n    \"faremmo\",\n    \"fareste\",\n    \"farebbero\",\n    \"facevo\",\n    \"facevi\",\n    \"faceva\",\n    \"facevamo\",\n    \"facevate\",\n    \"facevano\",\n    \"feci\",\n    \"facesti\",\n    \"fece\",\n    \"facemmo\",\n    \"faceste\",\n    \"fecero\",\n    \"facessi\",\n    \"facesse\",\n    \"facessimo\",\n    \"facessero\",\n    \"facendo\",\n    \"sto\",\n    \"stai\",\n    \"sta\",\n    \"stiamo\",\n    \"stanno\",\n    \"stia\",\n    \"stiate\",\n    \"stiano\",\n    \"starò\",\n    \"starai\",\n    \"starà\",\n    \"staremo\",\n    \"starete\",\n    \"staranno\",\n    \"starei\",\n    \"staresti\",\n    \"starebbe\",\n    \"staremmo\",\n    \"stareste\",\n    \"starebbero\",\n    \"stavo\",\n    \"stavi\",\n    \"stava\",\n    \"stavamo\",\n    \"stavate\",\n    \"stavano\",\n    \"stetti\",\n    \"stesti\",\n    \"stette\",\n    \"stemmo\",\n    \"steste\",\n    \"stettero\",\n    \"stessi\",\n    \"stesse\",\n    \"stessimo\",\n    \"stessero\",\n    \"stando\",\n];\n\npub const NORWEGIAN: &[&str] = &[\n    \"og\",\n    \"i\",\n    \"jeg\",\n    \"det\",\n    \"at\",\n    \"en\",\n    \"et\",\n    \"den\",\n    \"til\",\n    \"er\",\n    \"som\",\n    \"på\",\n    \"de\",\n    \"med\",\n    \"han\",\n    \"av\",\n    \"ikke\",\n    \"ikkje\",\n    \"der\",\n    \"så\",\n    \"var\",\n    \"meg\",\n    \"seg\",\n    \"men\",\n    \"ett\",\n    \"har\",\n    \"om\",\n    \"vi\",\n    \"min\",\n    \"mitt\",\n    \"ha\",\n    \"hadde\",\n    \"hun\",\n    \"nå\",\n    \"over\",\n    \"da\",\n    \"ved\",\n    \"fra\",\n    \"du\",\n    \"ut\",\n    \"sin\",\n    \"dem\",\n    \"oss\",\n    \"opp\",\n    \"man\",\n    \"kan\",\n    \"hans\",\n    \"hvor\",\n    \"eller\",\n    \"hva\",\n    \"skal\",\n    \"selv\",\n    \"sjøl\",\n    \"her\",\n    \"alle\",\n    \"vil\",\n    \"bli\",\n    \"ble\",\n    \"blei\",\n    \"blitt\",\n    \"kunne\",\n    \"inn\",\n    \"når\",\n    \"være\",\n    \"kom\",\n    \"noen\",\n    \"noe\",\n    \"ville\",\n    \"dere\",\n    \"deres\",\n    \"kun\",\n    \"ja\",\n    \"etter\",\n    \"ned\",\n    \"skulle\",\n    \"denne\",\n    \"for\",\n    \"deg\",\n    \"si\",\n    \"sine\",\n    \"sitt\",\n    \"mot\",\n    \"å\",\n    \"meget\",\n    \"hvorfor\",\n    \"dette\",\n    \"disse\",\n    \"uten\",\n    \"hvordan\",\n    \"ingen\",\n    \"din\",\n    \"ditt\",\n    \"blir\",\n    \"samme\",\n    \"hvilken\",\n    \"hvilke\",\n    \"sånn\",\n    \"inni\",\n    \"mellom\",\n    \"vår\",\n    \"hver\",\n    \"hvem\",\n    \"vors\",\n    \"hvis\",\n    \"både\",\n    \"bare\",\n    \"enn\",\n    \"fordi\",\n    \"før\",\n    \"mange\",\n    \"også\",\n    \"slik\",\n    \"vært\",\n    \"båe\",\n    \"begge\",\n    \"siden\",\n    \"dykk\",\n    \"dykkar\",\n    \"dei\",\n    \"deira\",\n    \"deires\",\n    \"deim\",\n    \"di\",\n    \"då\",\n    \"eg\",\n    \"ein\",\n    \"eit\",\n    \"eitt\",\n    \"elles\",\n    \"honom\",\n    \"hjå\",\n    \"ho\",\n    \"hoe\",\n    \"henne\",\n    \"hennar\",\n    \"hennes\",\n    \"hoss\",\n    \"hossen\",\n    \"ingi\",\n    \"inkje\",\n    \"korleis\",\n    \"korso\",\n    \"kva\",\n    \"kvar\",\n    \"kvarhelst\",\n    \"kven\",\n    \"kvi\",\n    \"kvifor\",\n    \"me\",\n    \"medan\",\n    \"mi\",\n    \"mine\",\n    \"mykje\",\n    \"no\",\n    \"nokon\",\n    \"noka\",\n    \"nokor\",\n    \"noko\",\n    \"nokre\",\n    \"sia\",\n    \"sidan\",\n    \"so\",\n    \"somt\",\n    \"somme\",\n    \"um\",\n    \"upp\",\n    \"vere\",\n    \"vore\",\n    \"verte\",\n    \"vort\",\n    \"varte\",\n    \"vart\",\n];\n\npub const PORTUGUESE: &[&str] = &[\n    \"de\",\n    \"a\",\n    \"o\",\n    \"que\",\n    \"e\",\n    \"do\",\n    \"da\",\n    \"em\",\n    \"um\",\n    \"para\",\n    \"com\",\n    \"não\",\n    \"uma\",\n    \"os\",\n    \"no\",\n    \"se\",\n    \"na\",\n    \"por\",\n    \"mais\",\n    \"as\",\n    \"dos\",\n    \"como\",\n    \"mas\",\n    \"ao\",\n    \"ele\",\n    \"das\",\n    \"à\",\n    \"seu\",\n    \"sua\",\n    \"ou\",\n    \"quando\",\n    \"muito\",\n    \"nos\",\n    \"já\",\n    \"eu\",\n    \"também\",\n    \"só\",\n    \"pelo\",\n    \"pela\",\n    \"até\",\n    \"isso\",\n    \"ela\",\n    \"entre\",\n    \"depois\",\n    \"sem\",\n    \"mesmo\",\n    \"aos\",\n    \"seus\",\n    \"quem\",\n    \"nas\",\n    \"me\",\n    \"esse\",\n    \"eles\",\n    \"você\",\n    \"essa\",\n    \"num\",\n    \"nem\",\n    \"suas\",\n    \"meu\",\n    \"às\",\n    \"minha\",\n    \"numa\",\n    \"pelos\",\n    \"elas\",\n    \"qual\",\n    \"nós\",\n    \"lhe\",\n    \"deles\",\n    \"essas\",\n    \"esses\",\n    \"pelas\",\n    \"este\",\n    \"dele\",\n    \"tu\",\n    \"te\",\n    \"vocês\",\n    \"vos\",\n    \"lhes\",\n    \"meus\",\n    \"minhas\",\n    \"teu\",\n    \"tua\",\n    \"teus\",\n    \"tuas\",\n    \"nosso\",\n    \"nossa\",\n    \"nossos\",\n    \"nossas\",\n    \"dela\",\n    \"delas\",\n    \"esta\",\n    \"estes\",\n    \"estas\",\n    \"aquele\",\n    \"aquela\",\n    \"aqueles\",\n    \"aquelas\",\n    \"isto\",\n    \"aquilo\",\n    \"estou\",\n    \"está\",\n    \"estamos\",\n    \"estão\",\n    \"estive\",\n    \"esteve\",\n    \"estivemos\",\n    \"estiveram\",\n    \"estava\",\n    \"estávamos\",\n    \"estavam\",\n    \"estivera\",\n    \"estivéramos\",\n    \"esteja\",\n    \"estejamos\",\n    \"estejam\",\n    \"estivesse\",\n    \"estivéssemos\",\n    \"estivessem\",\n    \"estiver\",\n    \"estivermos\",\n    \"estiverem\",\n    \"hei\",\n    \"há\",\n    \"havemos\",\n    \"hão\",\n    \"houve\",\n    \"houvemos\",\n    \"houveram\",\n    \"houvera\",\n    \"houvéramos\",\n    \"haja\",\n    \"hajamos\",\n    \"hajam\",\n    \"houvesse\",\n    \"houvéssemos\",\n    \"houvessem\",\n    \"houver\",\n    \"houvermos\",\n    \"houverem\",\n    \"houverei\",\n    \"houverá\",\n    \"houveremos\",\n    \"houverão\",\n    \"houveria\",\n    \"houveríamos\",\n    \"houveriam\",\n    \"sou\",\n    \"somos\",\n    \"são\",\n    \"era\",\n    \"éramos\",\n    \"eram\",\n    \"fui\",\n    \"foi\",\n    \"fomos\",\n    \"foram\",\n    \"fora\",\n    \"fôramos\",\n    \"seja\",\n    \"sejamos\",\n    \"sejam\",\n    \"fosse\",\n    \"fôssemos\",\n    \"fossem\",\n    \"for\",\n    \"formos\",\n    \"forem\",\n    \"serei\",\n    \"será\",\n    \"seremos\",\n    \"serão\",\n    \"seria\",\n    \"seríamos\",\n    \"seriam\",\n    \"tenho\",\n    \"tem\",\n    \"temos\",\n    \"tém\",\n    \"tinha\",\n    \"tínhamos\",\n    \"tinham\",\n    \"tive\",\n    \"teve\",\n    \"tivemos\",\n    \"tiveram\",\n    \"tivera\",\n    \"tivéramos\",\n    \"tenha\",\n    \"tenhamos\",\n    \"tenham\",\n    \"tivesse\",\n    \"tivéssemos\",\n    \"tivessem\",\n    \"tiver\",\n    \"tivermos\",\n    \"tiverem\",\n    \"terei\",\n    \"terá\",\n    \"teremos\",\n    \"terão\",\n    \"teria\",\n    \"teríamos\",\n    \"teriam\",\n];\n\npub const RUSSIAN: &[&str] = &[\n    \"и\",\n    \"в\",\n    \"во\",\n    \"не\",\n    \"что\",\n    \"он\",\n    \"на\",\n    \"я\",\n    \"с\",\n    \"со\",\n    \"как\",\n    \"а\",\n    \"то\",\n    \"все\",\n    \"она\",\n    \"так\",\n    \"его\",\n    \"но\",\n    \"да\",\n    \"ты\",\n    \"к\",\n    \"у\",\n    \"же\",\n    \"вы\",\n    \"за\",\n    \"бы\",\n    \"по\",\n    \"только\",\n    \"ее\",\n    \"мне\",\n    \"было\",\n    \"вот\",\n    \"от\",\n    \"меня\",\n    \"еще\",\n    \"нет\",\n    \"о\",\n    \"из\",\n    \"ему\",\n    \"теперь\",\n    \"когда\",\n    \"даже\",\n    \"ну\",\n    \"вдруг\",\n    \"ли\",\n    \"если\",\n    \"уже\",\n    \"или\",\n    \"ни\",\n    \"быть\",\n    \"был\",\n    \"него\",\n    \"до\",\n    \"вас\",\n    \"нибудь\",\n    \"опять\",\n    \"уж\",\n    \"вам\",\n    \"сказал\",\n    \"ведь\",\n    \"там\",\n    \"потом\",\n    \"себя\",\n    \"ничего\",\n    \"ей\",\n    \"может\",\n    \"они\",\n    \"тут\",\n    \"где\",\n    \"есть\",\n    \"надо\",\n    \"ней\",\n    \"для\",\n    \"мы\",\n    \"тебя\",\n    \"их\",\n    \"чем\",\n    \"была\",\n    \"сам\",\n    \"чтоб\",\n    \"без\",\n    \"будто\",\n    \"человек\",\n    \"чего\",\n    \"раз\",\n    \"тоже\",\n    \"себе\",\n    \"под\",\n    \"жизнь\",\n    \"будет\",\n    \"ж\",\n    \"тогда\",\n    \"кто\",\n    \"этот\",\n    \"говорил\",\n    \"того\",\n    \"потому\",\n    \"этого\",\n    \"какой\",\n    \"совсем\",\n    \"ним\",\n    \"здесь\",\n    \"этом\",\n    \"один\",\n    \"почти\",\n    \"мой\",\n    \"тем\",\n    \"чтобы\",\n    \"нее\",\n    \"кажется\",\n    \"сейчас\",\n    \"были\",\n    \"куда\",\n    \"зачем\",\n    \"сказать\",\n    \"всех\",\n    \"никогда\",\n    \"сегодня\",\n    \"можно\",\n    \"при\",\n    \"наконец\",\n    \"два\",\n    \"об\",\n    \"другой\",\n    \"хоть\",\n    \"после\",\n    \"над\",\n    \"больше\",\n    \"тот\",\n    \"через\",\n    \"эти\",\n    \"нас\",\n    \"про\",\n    \"всего\",\n    \"них\",\n    \"какая\",\n    \"много\",\n    \"разве\",\n    \"сказала\",\n    \"три\",\n    \"эту\",\n    \"моя\",\n    \"впрочем\",\n    \"хорошо\",\n    \"свою\",\n    \"этой\",\n    \"перед\",\n    \"иногда\",\n    \"лучше\",\n    \"чуть\",\n    \"том\",\n    \"нельзя\",\n    \"такой\",\n    \"им\",\n    \"более\",\n    \"всегда\",\n    \"конечно\",\n    \"всю\",\n    \"между\",\n];\n\npub const SPANISH: &[&str] = &[\n    \"de\",\n    \"la\",\n    \"que\",\n    \"el\",\n    \"en\",\n    \"y\",\n    \"a\",\n    \"los\",\n    \"del\",\n    \"se\",\n    \"las\",\n    \"por\",\n    \"un\",\n    \"para\",\n    \"con\",\n    \"no\",\n    \"una\",\n    \"su\",\n    \"al\",\n    \"lo\",\n    \"como\",\n    \"más\",\n    \"pero\",\n    \"sus\",\n    \"le\",\n    \"ya\",\n    \"o\",\n    \"este\",\n    \"sí\",\n    \"porque\",\n    \"esta\",\n    \"entre\",\n    \"cuando\",\n    \"muy\",\n    \"sin\",\n    \"sobre\",\n    \"también\",\n    \"me\",\n    \"hasta\",\n    \"hay\",\n    \"donde\",\n    \"quien\",\n    \"desde\",\n    \"todo\",\n    \"nos\",\n    \"durante\",\n    \"todos\",\n    \"uno\",\n    \"les\",\n    \"ni\",\n    \"contra\",\n    \"otros\",\n    \"ese\",\n    \"eso\",\n    \"ante\",\n    \"ellos\",\n    \"e\",\n    \"esto\",\n    \"mí\",\n    \"antes\",\n    \"algunos\",\n    \"qué\",\n    \"unos\",\n    \"yo\",\n    \"otro\",\n    \"otras\",\n    \"otra\",\n    \"él\",\n    \"tanto\",\n    \"esa\",\n    \"estos\",\n    \"mucho\",\n    \"quienes\",\n    \"nada\",\n    \"muchos\",\n    \"cual\",\n    \"poco\",\n    \"ella\",\n    \"estar\",\n    \"estas\",\n    \"algunas\",\n    \"algo\",\n    \"nosotros\",\n    \"mi\",\n    \"mis\",\n    \"tú\",\n    \"te\",\n    \"ti\",\n    \"tu\",\n    \"tus\",\n    \"ellas\",\n    \"nosotras\",\n    \"vosotros\",\n    \"vosotras\",\n    \"os\",\n    \"mío\",\n    \"mía\",\n    \"míos\",\n    \"mías\",\n    \"tuyo\",\n    \"tuya\",\n    \"tuyos\",\n    \"tuyas\",\n    \"suyo\",\n    \"suya\",\n    \"suyos\",\n    \"suyas\",\n    \"nuestro\",\n    \"nuestra\",\n    \"nuestros\",\n    \"nuestras\",\n    \"vuestro\",\n    \"vuestra\",\n    \"vuestros\",\n    \"vuestras\",\n    \"esos\",\n    \"esas\",\n    \"estoy\",\n    \"estás\",\n    \"está\",\n    \"estamos\",\n    \"estáis\",\n    \"están\",\n    \"esté\",\n    \"estés\",\n    \"estemos\",\n    \"estéis\",\n    \"estén\",\n    \"estaré\",\n    \"estarás\",\n    \"estará\",\n    \"estaremos\",\n    \"estaréis\",\n    \"estarán\",\n    \"estaría\",\n    \"estarías\",\n    \"estaríamos\",\n    \"estaríais\",\n    \"estarían\",\n    \"estaba\",\n    \"estabas\",\n    \"estábamos\",\n    \"estabais\",\n    \"estaban\",\n    \"estuve\",\n    \"estuviste\",\n    \"estuvo\",\n    \"estuvimos\",\n    \"estuvisteis\",\n    \"estuvieron\",\n    \"estuviera\",\n    \"estuvieras\",\n    \"estuviéramos\",\n    \"estuvierais\",\n    \"estuvieran\",\n    \"estuviese\",\n    \"estuvieses\",\n    \"estuviésemos\",\n    \"estuvieseis\",\n    \"estuviesen\",\n    \"estando\",\n    \"estado\",\n    \"estada\",\n    \"estados\",\n    \"estadas\",\n    \"estad\",\n    \"he\",\n    \"has\",\n    \"ha\",\n    \"hemos\",\n    \"habéis\",\n    \"han\",\n    \"haya\",\n    \"hayas\",\n    \"hayamos\",\n    \"hayáis\",\n    \"hayan\",\n    \"habré\",\n    \"habrás\",\n    \"habrá\",\n    \"habremos\",\n    \"habréis\",\n    \"habrán\",\n    \"habría\",\n    \"habrías\",\n    \"habríamos\",\n    \"habríais\",\n    \"habrían\",\n    \"había\",\n    \"habías\",\n    \"habíamos\",\n    \"habíais\",\n    \"habían\",\n    \"hube\",\n    \"hubiste\",\n    \"hubo\",\n    \"hubimos\",\n    \"hubisteis\",\n    \"hubieron\",\n    \"hubiera\",\n    \"hubieras\",\n    \"hubiéramos\",\n    \"hubierais\",\n    \"hubieran\",\n    \"hubiese\",\n    \"hubieses\",\n    \"hubiésemos\",\n    \"hubieseis\",\n    \"hubiesen\",\n    \"habiendo\",\n    \"habido\",\n    \"habida\",\n    \"habidos\",\n    \"habidas\",\n    \"soy\",\n    \"eres\",\n    \"es\",\n    \"somos\",\n    \"sois\",\n    \"son\",\n    \"sea\",\n    \"seas\",\n    \"seamos\",\n    \"seáis\",\n    \"sean\",\n    \"seré\",\n    \"serás\",\n    \"será\",\n    \"seremos\",\n    \"seréis\",\n    \"serán\",\n    \"sería\",\n    \"serías\",\n    \"seríamos\",\n    \"seríais\",\n    \"serían\",\n    \"era\",\n    \"eras\",\n    \"éramos\",\n    \"erais\",\n    \"eran\",\n    \"fui\",\n    \"fuiste\",\n    \"fue\",\n    \"fuimos\",\n    \"fuisteis\",\n    \"fueron\",\n    \"fuera\",\n    \"fueras\",\n    \"fuéramos\",\n    \"fuerais\",\n    \"fueran\",\n    \"fuese\",\n    \"fueses\",\n    \"fuésemos\",\n    \"fueseis\",\n    \"fuesen\",\n    \"siendo\",\n    \"sido\",\n    \"tengo\",\n    \"tienes\",\n    \"tiene\",\n    \"tenemos\",\n    \"tenéis\",\n    \"tienen\",\n    \"tenga\",\n    \"tengas\",\n    \"tengamos\",\n    \"tengáis\",\n    \"tengan\",\n    \"tendré\",\n    \"tendrás\",\n    \"tendrá\",\n    \"tendremos\",\n    \"tendréis\",\n    \"tendrán\",\n    \"tendría\",\n    \"tendrías\",\n    \"tendríamos\",\n    \"tendríais\",\n    \"tendrían\",\n    \"tenía\",\n    \"tenías\",\n    \"teníamos\",\n    \"teníais\",\n    \"tenían\",\n    \"tuve\",\n    \"tuviste\",\n    \"tuvo\",\n    \"tuvimos\",\n    \"tuvisteis\",\n    \"tuvieron\",\n    \"tuviera\",\n    \"tuvieras\",\n    \"tuviéramos\",\n    \"tuvierais\",\n    \"tuvieran\",\n    \"tuviese\",\n    \"tuvieses\",\n    \"tuviésemos\",\n    \"tuvieseis\",\n    \"tuviesen\",\n    \"teniendo\",\n    \"tenido\",\n    \"tenida\",\n    \"tenidos\",\n    \"tenidas\",\n    \"tened\",\n];\n\npub const SWEDISH: &[&str] = &[\n    \"och\",\n    \"det\",\n    \"att\",\n    \"i\",\n    \"en\",\n    \"jag\",\n    \"hon\",\n    \"som\",\n    \"han\",\n    \"på\",\n    \"den\",\n    \"med\",\n    \"var\",\n    \"sig\",\n    \"för\",\n    \"så\",\n    \"till\",\n    \"är\",\n    \"men\",\n    \"ett\",\n    \"om\",\n    \"hade\",\n    \"de\",\n    \"av\",\n    \"icke\",\n    \"mig\",\n    \"du\",\n    \"henne\",\n    \"då\",\n    \"sin\",\n    \"nu\",\n    \"har\",\n    \"inte\",\n    \"hans\",\n    \"honom\",\n    \"skulle\",\n    \"hennes\",\n    \"där\",\n    \"min\",\n    \"man\",\n    \"ej\",\n    \"vid\",\n    \"kunde\",\n    \"något\",\n    \"från\",\n    \"ut\",\n    \"när\",\n    \"efter\",\n    \"upp\",\n    \"vi\",\n    \"dem\",\n    \"vara\",\n    \"vad\",\n    \"över\",\n    \"än\",\n    \"dig\",\n    \"kan\",\n    \"sina\",\n    \"här\",\n    \"ha\",\n    \"mot\",\n    \"alla\",\n    \"under\",\n    \"någon\",\n    \"eller\",\n    \"allt\",\n    \"mycket\",\n    \"sedan\",\n    \"ju\",\n    \"denna\",\n    \"själv\",\n    \"detta\",\n    \"åt\",\n    \"utan\",\n    \"varit\",\n    \"hur\",\n    \"ingen\",\n    \"mitt\",\n    \"ni\",\n    \"bli\",\n    \"blev\",\n    \"oss\",\n    \"din\",\n    \"dessa\",\n    \"några\",\n    \"deras\",\n    \"blir\",\n    \"mina\",\n    \"samma\",\n    \"vilken\",\n    \"er\",\n    \"sådan\",\n    \"vår\",\n    \"blivit\",\n    \"dess\",\n    \"inom\",\n    \"mellan\",\n    \"sådant\",\n    \"varför\",\n    \"varje\",\n    \"vilka\",\n    \"ditt\",\n    \"vem\",\n    \"vilket\",\n    \"sitt\",\n    \"sådana\",\n    \"vart\",\n    \"dina\",\n    \"vars\",\n    \"vårt\",\n    \"våra\",\n    \"ert\",\n    \"era\",\n    \"vilkas\",\n];\n\n"
  },
  {
    "path": "src/tokenizer/tokenized_string.rs",
    "content": "use std::cmp::Ordering;\nuse std::io;\nuse std::io::{Read, Write};\n\nuse common::*;\n\nuse crate::tokenizer::{Token, TokenStream};\n\n/// Struct representing pre-tokenized text\n#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, Eq, PartialEq)]\npub struct PreTokenizedString {\n    /// Original text\n    pub text: String,\n    /// Tokens derived from the text\n    pub tokens: Vec<Token>,\n}\n\nimpl Ord for PreTokenizedString {\n    fn cmp(&self, other: &Self) -> Ordering {\n        self.text.cmp(&other.text)\n    }\n}\n\nimpl PartialOrd for PreTokenizedString {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl BinarySerializable for PreTokenizedString {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        if let Ok(text) = serde_json::to_string(self) {\n            <String as BinarySerializable>::serialize(&text, writer)\n        } else {\n            Err(io::Error::other(\n                \"Failed to dump PreTokenizedString to json.\",\n            ))\n        }\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let json_text = <String as BinarySerializable>::deserialize(reader)?;\n\n        if let Ok(value) = serde_json::from_str(&json_text) {\n            Ok(value)\n        } else {\n            Err(io::Error::other(\n                \"Failed to parse string data as PreTokenizedString.\",\n            ))\n        }\n    }\n}\n\n/// [`TokenStream`] implementation which wraps [`PreTokenizedString`]\npub struct PreTokenizedStream {\n    tokenized_string: PreTokenizedString,\n    current_token: i64,\n}\n\nimpl From<PreTokenizedString> for PreTokenizedStream {\n    fn from(s: PreTokenizedString) -> PreTokenizedStream {\n        PreTokenizedStream {\n            tokenized_string: s,\n            current_token: -1,\n        }\n    }\n}\n\nimpl TokenStream for PreTokenizedStream {\n    fn advance(&mut self) -> bool {\n        self.current_token += 1;\n        self.current_token < self.tokenized_string.tokens.len() as i64\n    }\n\n    fn token(&self) -> &Token {\n        assert!(\n            self.current_token >= 0,\n            \"TokenStream not initialized. You should call advance() at least once.\"\n        );\n        &self.tokenized_string.tokens[self.current_token as usize]\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        assert!(\n            self.current_token >= 0,\n            \"TokenStream not initialized. You should call advance() at least once.\"\n        );\n        &mut self.tokenized_string.tokens[self.current_token as usize]\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n\n    #[test]\n    fn test_tokenized_stream() {\n        let tok_text = PreTokenizedString {\n            text: String::from(\"A a\"),\n            tokens: vec![\n                Token {\n                    offset_from: 0,\n                    offset_to: 1,\n                    position: 0,\n                    text: String::from(\"A\"),\n                    position_length: 1,\n                },\n                Token {\n                    offset_from: 2,\n                    offset_to: 3,\n                    position: 1,\n                    text: String::from(\"a\"),\n                    position_length: 1,\n                },\n            ],\n        };\n\n        let mut token_stream = PreTokenizedStream::from(tok_text.clone());\n\n        for expected_token in tok_text.tokens {\n            assert!(token_stream.advance());\n            assert_eq!(token_stream.token(), &expected_token);\n        }\n        assert!(!token_stream.advance());\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/tokenizer.rs",
    "content": "/// The tokenizer module contains all of the tools used to process\n/// text in `tantivy`.\nuse tokenizer_api::{BoxTokenStream, TokenFilter, Tokenizer};\n\nuse crate::tokenizer::empty_tokenizer::EmptyTokenizer;\n\n/// `TextAnalyzer` tokenizes an input text into tokens and modifies the resulting `TokenStream`.\n#[derive(Clone)]\npub struct TextAnalyzer {\n    tokenizer: Box<dyn BoxableTokenizer>,\n}\n\nimpl Tokenizer for Box<dyn BoxableTokenizer> {\n    type TokenStream<'a> = BoxTokenStream<'a>;\n\n    // Note: we want to call `box_token_stream` on the concrete `Tokenizer`\n    // implementation, not the `BoxableTokenizer` one as it will cause\n    // a recursive call (and a stack overflow).\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a> {\n        (**self).box_token_stream(text)\n    }\n}\n\nimpl Clone for Box<dyn BoxableTokenizer> {\n    // Note: we want to call `box_clone` on the concrete `Tokenizer`\n    // implementation in order to clone the concrete `Tokenizer`.\n    fn clone(&self) -> Self {\n        (**self).box_clone()\n    }\n}\n\n/// A boxable `Tokenizer`, with its `TokenStream` type erased.\npub trait BoxableTokenizer: 'static + Send + Sync {\n    /// Creates a boxed token stream for a given `str`.\n    fn box_token_stream<'a>(&'a mut self, text: &'a str) -> BoxTokenStream<'a>;\n    /// Clone this tokenizer.\n    fn box_clone(&self) -> Box<dyn BoxableTokenizer>;\n}\n\nimpl<T: Tokenizer> BoxableTokenizer for T {\n    fn box_token_stream<'a>(&'a mut self, text: &'a str) -> BoxTokenStream<'a> {\n        BoxTokenStream::new(self.token_stream(text))\n    }\n    fn box_clone(&self) -> Box<dyn BoxableTokenizer> {\n        Box::new(self.clone())\n    }\n}\n\nimpl Default for TextAnalyzer {\n    fn default() -> TextAnalyzer {\n        TextAnalyzer::from(EmptyTokenizer)\n    }\n}\n\nimpl<T: Tokenizer + Clone> From<T> for TextAnalyzer {\n    fn from(tokenizer: T) -> Self {\n        TextAnalyzer::builder(tokenizer).build()\n    }\n}\n\nimpl TextAnalyzer {\n    /// Create a new TextAnalyzerBuilder\n    pub fn builder<T: Tokenizer>(tokenizer: T) -> TextAnalyzerBuilder<T> {\n        TextAnalyzerBuilder { tokenizer }\n    }\n\n    /// Creates a token stream for a given `str`.\n    pub fn token_stream<'a>(&'a mut self, text: &'a str) -> BoxTokenStream<'a> {\n        self.tokenizer.token_stream(text)\n    }\n}\n\n/// Builder helper for [`TextAnalyzer`]\npub struct TextAnalyzerBuilder<T = Box<dyn BoxableTokenizer>> {\n    tokenizer: T,\n}\n\nimpl<T: Tokenizer> TextAnalyzerBuilder<T> {\n    /// Appends a token filter to the current builder.\n    ///\n    /// # Example\n    ///\n    /// ```rust\n    /// use tantivy::tokenizer::*;\n    ///\n    /// let en_stem = TextAnalyzer::builder(SimpleTokenizer::default())\n    ///     .filter(RemoveLongFilter::limit(40))\n    ///     .filter(LowerCaser)\n    ///     .filter(Stemmer::default())\n    ///     .build();\n    /// ```\n    pub fn filter<F: TokenFilter>(self, token_filter: F) -> TextAnalyzerBuilder<F::Tokenizer<T>> {\n        TextAnalyzerBuilder {\n            tokenizer: token_filter.transform(self.tokenizer),\n        }\n    }\n\n    /// Boxes the internal tokenizer. This is useful for adding dynamic filters.\n    /// Note: this will be less performant than the non boxed version.\n    pub fn dynamic(self) -> TextAnalyzerBuilder {\n        let boxed_tokenizer = Box::new(self.tokenizer);\n        TextAnalyzerBuilder {\n            tokenizer: boxed_tokenizer,\n        }\n    }\n\n    /// Appends a token filter to the current builder and returns a boxed version of the\n    /// tokenizer. This is useful when you want to build a `TextAnalyzer` dynamically.\n    /// Prefer using `TextAnalyzer::builder(tokenizer).filter(token_filter).build()` if\n    /// possible as it will be more performant and create less boxes.\n    pub fn filter_dynamic<F: TokenFilter>(self, token_filter: F) -> TextAnalyzerBuilder {\n        self.filter(token_filter).dynamic()\n    }\n\n    /// Finalize building the TextAnalyzer\n    pub fn build(self) -> TextAnalyzer {\n        TextAnalyzer {\n            tokenizer: Box::new(self.tokenizer),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::*;\n    use crate::tokenizer::{LowerCaser, RemoveLongFilter, SimpleTokenizer};\n\n    #[test]\n    fn test_text_analyzer_builder() {\n        let mut analyzer = TextAnalyzer::builder(SimpleTokenizer::default())\n            .filter(RemoveLongFilter::limit(40))\n            .filter(LowerCaser)\n            .build();\n        let mut stream = analyzer.token_stream(\"- first bullet point\");\n        assert_eq!(stream.next().unwrap().text, \"first\");\n        assert_eq!(stream.next().unwrap().text, \"bullet\");\n    }\n\n    #[test]\n    fn test_text_analyzer_with_filters_boxed() {\n        // This test shows how one can build a TextAnalyzer dynamically, by stacking a list\n        // of parametrizable token filters.\n        //\n        // The following enum is the thing that would be serializable.\n        // Note that token filters can have their own parameters, too, like the RemoveLongFilter\n        enum SerializableTokenFilterEnum {\n            LowerCaser(LowerCaser),\n            RemoveLongFilter(RemoveLongFilter),\n        }\n        // Note that everything below is dynamic.\n        let filters: Vec<SerializableTokenFilterEnum> = vec![\n            SerializableTokenFilterEnum::LowerCaser(LowerCaser),\n            SerializableTokenFilterEnum::RemoveLongFilter(RemoveLongFilter::limit(12)),\n        ];\n        let mut analyzer_builder: TextAnalyzerBuilder =\n            TextAnalyzer::builder(SimpleTokenizer::default())\n                .filter_dynamic(RemoveLongFilter::limit(40))\n                .filter_dynamic(LowerCaser);\n        for filter in filters {\n            analyzer_builder = match filter {\n                SerializableTokenFilterEnum::LowerCaser(lower_caser) => {\n                    analyzer_builder.filter_dynamic(lower_caser)\n                }\n                SerializableTokenFilterEnum::RemoveLongFilter(remove_long_filter) => {\n                    analyzer_builder.filter_dynamic(remove_long_filter)\n                }\n            }\n        }\n        let mut analyzer = analyzer_builder.build();\n        let mut stream = analyzer.token_stream(\"first bullet point\");\n        assert_eq!(stream.next().unwrap().text, \"first\");\n        assert_eq!(stream.next().unwrap().text, \"bullet\");\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/tokenizer_manager.rs",
    "content": "use std::collections::HashMap;\nuse std::sync::{Arc, RwLock};\n\nuse crate::tokenizer::tokenizer::TextAnalyzer;\nuse crate::tokenizer::{\n    LowerCaser, RawTokenizer, RemoveLongFilter, SimpleTokenizer, WhitespaceTokenizer,\n};\n\n/// The tokenizer manager serves as a store for\n/// all of the pre-configured tokenizer pipelines.\n///\n/// By default, it is populated with the following managers.\n///\n/// - `raw` : does not process nor tokenize the text.\n/// - `default` : Chops the text on according to whitespace and punctuation, removes tokens that are\n///   too long, and lowercases tokens.\n/// - `en_stem` : Like `default`, but also applies stemming on the resulting tokens. Stemming can\n///   improve the recall of your search engine.\n/// - `whitespace` : Splits the text on whitespaces.\n#[derive(Clone)]\npub struct TokenizerManager {\n    tokenizers: Arc<RwLock<HashMap<String, TextAnalyzer>>>,\n}\n\nimpl TokenizerManager {\n    /// Creates an empty tokenizer manager.\n    pub fn new() -> Self {\n        Self {\n            tokenizers: Arc::new(RwLock::new(HashMap::new())),\n        }\n    }\n\n    /// Registers a new tokenizer associated with a given name.\n    pub fn register<T>(&self, tokenizer_name: &str, tokenizer: T)\n    where TextAnalyzer: From<T> {\n        let boxed_tokenizer: TextAnalyzer = TextAnalyzer::from(tokenizer);\n        self.tokenizers\n            .write()\n            .expect(\"Acquiring the lock should never fail\")\n            .insert(tokenizer_name.to_string(), boxed_tokenizer);\n    }\n\n    /// Accessing a tokenizer given its name.\n    pub fn get(&self, tokenizer_name: &str) -> Option<TextAnalyzer> {\n        self.tokenizers\n            .read()\n            .expect(\"Acquiring the lock should never fail\")\n            .get(tokenizer_name)\n            .cloned()\n    }\n}\n\nimpl Default for TokenizerManager {\n    /// Creates an `TokenizerManager` prepopulated with\n    /// the default pre-configured tokenizers of `tantivy`.\n    fn default() -> TokenizerManager {\n        let manager = TokenizerManager::new();\n        manager.register(\"raw\", RawTokenizer::default());\n        manager.register(\n            \"default\",\n            TextAnalyzer::builder(SimpleTokenizer::default())\n                .filter(RemoveLongFilter::limit(40))\n                .filter(LowerCaser)\n                .build(),\n        );\n        #[cfg(feature = \"stemmer\")]\n        {\n            use crate::tokenizer::stemmer::{Language, Stemmer};\n            manager.register(\n                \"en_stem\",\n                TextAnalyzer::builder(SimpleTokenizer::default())\n                    .filter(RemoveLongFilter::limit(40))\n                    .filter(LowerCaser) // The stemmer does not lowercase\n                    .filter(Stemmer::new(Language::English))\n                    .build(),\n            );\n        }\n        manager.register(\"whitespace\", WhitespaceTokenizer::default());\n        manager\n    }\n}\n"
  },
  {
    "path": "src/tokenizer/whitespace_tokenizer.rs",
    "content": "use std::str::CharIndices;\n\nuse super::{Token, TokenStream, Tokenizer};\n\n/// Tokenize the text by splitting on whitespaces.\n#[derive(Clone, Default)]\npub struct WhitespaceTokenizer {\n    token: Token,\n}\n\npub struct WhitespaceTokenStream<'a> {\n    text: &'a str,\n    chars: CharIndices<'a>,\n    token: &'a mut Token,\n}\n\nimpl Tokenizer for WhitespaceTokenizer {\n    type TokenStream<'a> = WhitespaceTokenStream<'a>;\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> WhitespaceTokenStream<'a> {\n        self.token.reset();\n        WhitespaceTokenStream {\n            text,\n            chars: text.char_indices(),\n            token: &mut self.token,\n        }\n    }\n}\n\nimpl WhitespaceTokenStream<'_> {\n    // search for the end of the current token.\n    fn search_token_end(&mut self) -> usize {\n        (&mut self.chars)\n            .filter(|(_, c)| c.is_ascii_whitespace())\n            .map(|(offset, _)| offset)\n            .next()\n            .unwrap_or(self.text.len())\n    }\n}\n\nimpl TokenStream for WhitespaceTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        self.token.text.clear();\n        self.token.position = self.token.position.wrapping_add(1);\n        while let Some((offset_from, c)) = self.chars.next() {\n            if !c.is_ascii_whitespace() {\n                let offset_to = self.search_token_end();\n                self.token.offset_from = offset_from;\n                self.token.offset_to = offset_to;\n                self.token.text.push_str(&self.text[offset_from..offset_to]);\n                return true;\n            }\n        }\n        false\n    }\n\n    fn token(&self) -> &Token {\n        self.token\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.token\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use crate::tokenizer::tests::assert_token;\n    use crate::tokenizer::{TextAnalyzer, Token, WhitespaceTokenizer};\n\n    #[test]\n    fn test_whitespace_tokenizer() {\n        let tokens = token_stream_helper(\"Hello, happy tax payer!\");\n        assert_eq!(tokens.len(), 4);\n        assert_token(&tokens[0], 0, \"Hello,\", 0, 6);\n        assert_token(&tokens[1], 1, \"happy\", 7, 12);\n        assert_token(&tokens[2], 2, \"tax\", 13, 16);\n        assert_token(&tokens[3], 3, \"payer!\", 17, 23);\n    }\n\n    fn token_stream_helper(text: &str) -> Vec<Token> {\n        let mut a = TextAnalyzer::from(WhitespaceTokenizer::default());\n        let mut token_stream = a.token_stream(text);\n        let mut tokens: Vec<Token> = vec![];\n        let mut add_token = |token: &Token| {\n            tokens.push(token.clone());\n        };\n        token_stream.process(&mut add_token);\n        tokens\n    }\n}\n"
  },
  {
    "path": "sstable/Cargo.toml",
    "content": "[package]\nname = \"tantivy-sstable\"\nversion = \"0.6.0\"\nedition = \"2024\"\nlicense = \"MIT\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\nkeywords = [\"search\", \"information\", \"retrieval\", \"sstable\"]\ncategories = [\"database-implementations\", \"data-structures\", \"compression\"]\ndescription = \"sstables for tantivy\"\n\n[dependencies]\ncommon = {version= \"0.10\", path=\"../common\", package=\"tantivy-common\"}\nfutures-util = \"0.3.30\"\nitertools = \"0.14.0\"\ntantivy-bitpacker = { version= \"0.9\", path=\"../bitpacker\" }\ntantivy-fst = \"0.5\"\n# experimental gives us access to Decompressor::upper_bound\nzstd = { version = \"0.13\", optional = true, features = [\"experimental\"] }\n\n[features]\nzstd-compression = [\"zstd\"]\n\n[dev-dependencies]\nproptest = \"1\"\ncriterion = { version = \"0.5\", default-features = false }\nnames = \"0.14\"\nrand = \"0.9\"\n\n[[bench]]\nname = \"stream_bench\"\nharness = false\n\n[[bench]]\nname = \"ord_to_term\"\nharness = false\n\n"
  },
  {
    "path": "sstable/README.md",
    "content": "# SSTable\n\nThe `tantivy-sstable` crate is yet another sstable crate.\n\nIt has been designed to be used in `quickwit`:\n- as an alternative to the default tantivy fst dictionary.\n- as a way to store the column index for dynamic fast fields.\n\nThe benefit compared to the fst crate is locality.\nSearching a key in the fst crate requires downloading the entire dictionary.\n\nOnce the sstable index is downloaded, running a `get` in the sstable\ncrate only requires a single fetch.\n\nRight now, the block index and the default block size have been thought\nfor quickwit, and the performance of a get is very bad.\n\n# Sorted strings?\n\nSSTable stands for Sorted String Table.\nStrings have to be insert in sorted order.\n\nThat sorted order is used in different ways:\n- it makes gets and streaming ranges of keys\npossible.\n- it allows incremental encoding of the keys\n- the front compression is leveraged to optimize\nthe intersection with an automaton\n\n# On disk format\n\nOverview of the SSTable format. Unless noted otherwise, numbers are little-endian.\n\n### SSTable\n```\n+-------+-------+-----+--------+\n| Block | Block | ... | Footer |\n+-------+-------+-----+--------+\n|----( # of blocks)---|\n```\n- Block(`SSTBlock`): list of independent block, terminated by a single empty block.\n- Footer(`SSTFooter`)\n\n### SSTBlock\n```\n+----------+----------+--------+-------+-------+-----+\n| BlockLen | Compress | Values | Delta | Delta | ... |\n+----------+----------+--------+-------+-------+-----+\n                      |        |----( # of deltas)---|\n                      |------(maybe compressed)------|\n```\n- BlockLen(u32): length of the block, including the compress byte.\n- Compress(u8): indicate whether block is compressed. 0 if not compressed, 1 if compressed.\n- Values: an application defined format storing a sequence of value, capable of determining it own length\n- Delta\n\n### Delta\n```\n+---------+--------+\n| KeepAdd | Suffix |\n+---------+--------+\n```\n- KeepAdd\n- Suffix: KeepAdd.add bytes of key suffix\n\n### KeepAdd\nKeepAdd can be represented in two different representation, a very compact 1byte one which is enough for most usage, and a longer variable-len one when required\n\nWhen keep < 16 and add < 16\n```\n+-----+------+\n| Add | Keep |\n+-----+------+\n```\n- Add(u4): number of bytes to push\n- Keep(u4): number of bytes to pop\n\nOtherwise:\n```\n+------+------+-----+\n| 0x01 | Keep | Add |\n+------+------+-----+\n```\n- Add(VInt): number of bytes to push\n- Keep(VInt): number of bytes to pop\n\n\nNote: as the SSTable does not support redundant keys, there is no ambiguity between both representation. Add is always guaranteed to be non-zero, except for the very first key of an SSTable, where Keep is guaranteed to be zero.\n\n### SSTFooter\n```\n+-----+----------------+-------------+-------------+---------+---------+\n| Fst | BlockAddrStore | StoreOffset | IndexOffset | NumTerm | Version |\n+-----+----------------+-------------+-------------+---------+---------+\n```\n- Fst(Fst): finite state transducer mapping keys to a block number\n- BlockAddrStore(BlockAddrStore): store mapping a block number to its BlockAddr\n- StoreOffset(u64): Offset to start of the BlockAddrStore. If zero, see the SingleBlockSStable section\n- IndexOffset(u64): Offset to the start of the SSTFooter\n- NumTerm(u64): number of terms in the sstable\n- Version(u32): Currently equal to 3\n\n### Fst\n\nFst is in the format of tantivy\\_fst\n\n### BlockAddrStore\n\n+---------+-----------+-----------+-----+-----------+-----------+-----+\n| MetaLen | BlockMeta | BlockMeta | ... | BlockData | BlockData | ... |\n+---------+-----------+-----------+-----+-----------+-----------+-----+\n          |---------(N blocks)----------|---------(N blocks)----------|\n\n- MetaLen(u64): length of the BlockMeta section\n- BlockMeta(BlockAddrBlockMetadata): metadata to seek through BlockData\n- BlockData(CompactedBlockAddr): bitpacked per block metadata\n\n### BlockAddrBlockMetadata\n\n+--------+------------+--------------+------------+--------------+-------------------+-----------------+----------+\n| Offset | RangeStart | FirstOrdinal | RangeSlope | OrdinalSlope | FirstOrdinalNBits | RangeStartNBits | BlockLen |\n+--------+------------+--------------+------------+--------------+-------------------+-----------------+----------+\n\n- Offset(u64): offset of the corresponding BlockData in the datastream\n- RangeStart(u64): the start position of the first block\n- FirstOrdinal(u64): the first ordinal of the first block\n- RangeSlope(u32): slope predicted for start range evolution (see computation in BlockData)\n- OrdinalSlope(u64): slope predicted for first ordinal evolution (see computation in BlockData)\n- FirstOrdinalNBits(u8): number of bits per ordinal in datastream (see computation in BlockData)\n- RangeStartNBits(u8): number of bits per range start in datastream (see computation in BlockData)\n\n### BlockData\n\n+-----------------+-------------------+---------------+\n| RangeStartDelta | FirstOrdinalDelta | FinalRangeEnd |\n+-----------------+-------------------+---------------+\n|------(BlockLen repetitions)---------|\n\n- RangeStartDelta(var): RangeStartNBits *bits* of little endian number. See below for decoding\n- FirstOrdinalDelta(var): FirstOrdinalNBits *bits* of little endian number. See below for decoding\n- FinalRangeEnd(var): RangeStartNBits *bits* of integer. See below for decoding\n\nconverting a BlockData of index Index and a BlockAddrBlockMetadata to an actual block address is done as follow:\nrange\\_prediction := RangeStart + Index * RangeSlop;\nrange\\_derivation := RangeStartDelta - (1 << (RangeStartNBits-1));\nrange\\_start := range\\_prediction + range\\_derivation\n\nThe same computation can be done for ordinal.\n\nNote that `range_derivation` can take negative value. `RangeStartDelta` is just its translation to a positive range.\n\n\n## SingleBlockSStable\n\nThe format used for the index is meant to be compact, however it has a constant cost of around 70\nbytes, which isn't negligible for a table containing very few keys.\nTo limit the impact of that constant cost, single block sstable omit the Fst and BlockAddrStore from\ntheir index. Instead a block with first ordinal of 0, range start of 0 and range end of IndexOffset\nis implicitly used for every operations.\n"
  },
  {
    "path": "sstable/benches/ord_to_term.rs",
    "content": "use std::sync::Arc;\n\nuse common::OwnedBytes;\nuse common::file_slice::FileSlice;\nuse criterion::{Criterion, criterion_group, criterion_main};\nuse tantivy_sstable::{Dictionary, MonotonicU64SSTable};\n\nfn make_test_sstable(suffix: &str) -> FileSlice {\n    let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n\n    // 125 mio elements\n    for elem in 0..125_000_000 {\n        let key = format!(\"prefix.{elem:07X}{suffix}\").into_bytes();\n        builder.insert(&key, &elem).unwrap();\n    }\n\n    let table = builder.finish().unwrap();\n    let table = Arc::new(OwnedBytes::new(table));\n    common::file_slice::FileSlice::new(table.clone())\n}\n\npub fn criterion_benchmark(c: &mut Criterion) {\n    {\n        let slice = make_test_sstable(\".suffix\");\n        let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n        c.bench_function(\"ord_to_term_suffix\", |b| {\n            let mut res = Vec::new();\n            b.iter(|| {\n                assert!(dict.ord_to_term(100_000, &mut res).unwrap());\n                assert!(dict.ord_to_term(19_000_000, &mut res).unwrap());\n            })\n        });\n        c.bench_function(\"open_and_ord_to_term_suffix\", |b| {\n            let mut res = Vec::new();\n            b.iter(|| {\n                let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n                assert!(dict.ord_to_term(100_000, &mut res).unwrap());\n                assert!(dict.ord_to_term(19_000_000, &mut res).unwrap());\n            })\n        });\n        c.bench_function(\"term_ord_suffix\", |b| {\n            b.iter(|| {\n                assert_eq!(\n                    dict.term_ord(b\"prefix.00186A0.suffix\").unwrap().unwrap(),\n                    100_000\n                );\n                assert_eq!(\n                    dict.term_ord(b\"prefix.121EAC0.suffix\").unwrap().unwrap(),\n                    19_000_000\n                );\n            })\n        });\n        c.bench_function(\"open_and_term_ord_suffix\", |b| {\n            b.iter(|| {\n                let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n                assert_eq!(\n                    dict.term_ord(b\"prefix.00186A0.suffix\").unwrap().unwrap(),\n                    100_000\n                );\n                assert_eq!(\n                    dict.term_ord(b\"prefix.121EAC0.suffix\").unwrap().unwrap(),\n                    19_000_000\n                );\n            })\n        });\n    }\n    {\n        let slice = make_test_sstable(\"\");\n        let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n        c.bench_function(\"ord_to_term\", |b| {\n            let mut res = Vec::new();\n            b.iter(|| {\n                assert!(dict.ord_to_term(100_000, &mut res).unwrap());\n                assert!(dict.ord_to_term(19_000_000, &mut res).unwrap());\n            })\n        });\n        c.bench_function(\"open_and_ord_to_term\", |b| {\n            let mut res = Vec::new();\n            b.iter(|| {\n                let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n                assert!(dict.ord_to_term(100_000, &mut res).unwrap());\n                assert!(dict.ord_to_term(19_000_000, &mut res).unwrap());\n            })\n        });\n        c.bench_function(\"term_ord\", |b| {\n            b.iter(|| {\n                assert_eq!(dict.term_ord(b\"prefix.00186A0\").unwrap().unwrap(), 100_000);\n                assert_eq!(\n                    dict.term_ord(b\"prefix.121EAC0\").unwrap().unwrap(),\n                    19_000_000\n                );\n            })\n        });\n        c.bench_function(\"open_and_term_ord\", |b| {\n            b.iter(|| {\n                let dict = Dictionary::<MonotonicU64SSTable>::open(slice.clone()).unwrap();\n                assert_eq!(dict.term_ord(b\"prefix.00186A0\").unwrap().unwrap(), 100_000);\n                assert_eq!(\n                    dict.term_ord(b\"prefix.121EAC0\").unwrap().unwrap(),\n                    19_000_000\n                );\n            })\n        });\n    }\n}\n\ncriterion_group!(benches, criterion_benchmark);\ncriterion_main!(benches);\n"
  },
  {
    "path": "sstable/benches/stream_bench.rs",
    "content": "use std::collections::BTreeSet;\nuse std::io;\n\nuse common::file_slice::FileSlice;\nuse criterion::{Criterion, criterion_group, criterion_main};\nuse rand::rngs::StdRng;\nuse rand::{Rng, SeedableRng};\nuse tantivy_sstable::{Dictionary, MonotonicU64SSTable};\n\nconst CHARSET: &[u8] = b\"abcdefghij\";\n\nfn generate_key(rng: &mut impl Rng) -> String {\n    let len = rng.random_range(3..12);\n    std::iter::from_fn(|| {\n        let idx = rng.random_range(0..CHARSET.len());\n        Some(CHARSET[idx] as char)\n    })\n    .take(len)\n    .collect()\n}\n\nfn prepare_sstable() -> io::Result<Dictionary<MonotonicU64SSTable>> {\n    let mut rng = StdRng::from_seed([3u8; 32]);\n    let mut els = BTreeSet::new();\n    while els.len() < 100_000 {\n        els.insert(generate_key(&mut rng));\n    }\n    let mut dictionary_builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new())?;\n    for (ord, word) in els.iter().enumerate() {\n        dictionary_builder.insert(word, &(ord as u64))?;\n    }\n    let buffer = dictionary_builder.finish()?;\n    let dictionary = Dictionary::open(FileSlice::from(buffer))?;\n    Ok(dictionary)\n}\n\nfn stream_bench(\n    dictionary: &Dictionary<MonotonicU64SSTable>,\n    lower: &[u8],\n    upper: &[u8],\n    do_scan: bool,\n) -> usize {\n    let mut stream = dictionary\n        .range()\n        .ge(lower)\n        .lt(upper)\n        .into_stream()\n        .unwrap();\n    if !do_scan {\n        return 0;\n    }\n    let mut count = 0;\n    while stream.advance() {\n        count += 1;\n    }\n    count\n}\n\npub fn criterion_benchmark(c: &mut Criterion) {\n    let dict = prepare_sstable().unwrap();\n    c.bench_function(\"short_scan_init\", |b| {\n        b.iter(|| stream_bench(&dict, b\"fa\", b\"fana\", false))\n    });\n    c.bench_function(\"short_scan_init_and_scan\", |b| {\n        b.iter(|| {\n            assert_eq!(stream_bench(&dict, b\"fa\", b\"faz\", true), 971);\n        })\n    });\n    c.bench_function(\"full_scan_init_and_scan_full_with_bound\", |b| {\n        b.iter(|| {\n            assert_eq!(stream_bench(&dict, b\"\", b\"z\", true), 100_000);\n        })\n    });\n    c.bench_function(\"full_scan_init_and_scan_full_no_bounds\", |b| {\n        b.iter(|| {\n            let mut stream = dict.stream().unwrap();\n            let mut count = 0;\n            while stream.advance() {\n                count += 1;\n            }\n            count\n        })\n    });\n}\n\ncriterion_group!(benches, criterion_benchmark);\ncriterion_main!(benches);\n"
  },
  {
    "path": "sstable/src/block_match_automaton.rs",
    "content": "use tantivy_fst::Automaton;\n\n/// Returns whether a block can match an automaton based on its bounds.\n///\n/// start key is exclusive, and optional to account for the first block. end key is inclusive and\n/// mandatory.\npub(crate) fn can_block_match_automaton(\n    start_key_opt: Option<&[u8]>,\n    end_key: &[u8],\n    automaton: &impl Automaton,\n) -> bool {\n    let start_key = if let Some(start_key) = start_key_opt {\n        start_key\n    } else {\n        // if start_key_opt is None, we would allow an automaton matching the empty string to match\n        if automaton.is_match(&automaton.start()) {\n            return true;\n        }\n        &[]\n    };\n    can_block_match_automaton_with_start(start_key, end_key, automaton)\n}\n\n// similar to can_block_match_automaton, ignoring the edge case of the initial block\nfn can_block_match_automaton_with_start(\n    start_key: &[u8],\n    end_key: &[u8],\n    automaton: &impl Automaton,\n) -> bool {\n    // notation: in loops, we use `kb` to denotate a key byte (a byte taken from the start/end key),\n    // and `rb`, a range byte (usually all values higher than a `kb` when comparing with\n    // start_key, or all values lower than a `kb` when comparing with end_key)\n\n    if start_key >= end_key {\n        return false;\n    }\n\n    let common_prefix_len = crate::common_prefix_len(start_key, end_key);\n\n    let mut base_state = automaton.start();\n    for kb in &start_key[0..common_prefix_len] {\n        base_state = automaton.accept(&base_state, *kb);\n    }\n\n    // this is not required for correctness, but allows dodging more expensive checks\n    if !automaton.can_match(&base_state) {\n        return false;\n    }\n\n    // we have 3 distinct case:\n    // - keys are `abc` and `abcd` => we test for abc[\\0-d].*\n    // - keys are `abcd` and `abce` => we test for abc[d-e].*\n    // - keys are `abcd` and `abc` => contradiction with start_key < end_key.\n    //\n    // ideally for (abc, abcde] we could test for abc([\\0-c].*|d([\\0-d].*|e)?)\n    // but let's start simple (and correct), and tighten our bounds latter\n    //\n    // and for (abcde, abcfg] we could test for abc(d(e.+|[f-\\xff].*)|e.*|f([\\0-f].*|g)?)\n    // abc (\n    //  d(e.+|[f-\\xff].*) |\n    //  e.* |\n    //  f([\\0-f].*|g)?\n    // )\n    //\n    // these are all written as regex, but can be converted to operations we can do:\n    // - [x-y] is a for c in x..=y\n    // - .* is a can_match()\n    // - .+ is a for c in 0..=255 { accept(c).can_match() }\n    // - ? is a the thing before can_match(), or current state.is_match()\n    // - | means test both side\n\n    // we have two cases, either start_key is a prefix of end_key (e.g. (abc, abcjp]),\n    // or it is not (e.g. (abcdg, abcjp]). It is not possible however that end_key be a prefix of\n    // start_key (or that both are equal) because we already handled start_key >= end_key.\n    //\n    // if we are in the first case, we want to visit the following states:\n    // abc (\n    //   [\\0-i].* |\n    //   j (\n    //     [\\0-o].* |\n    //     p\n    //   )?\n    // )\n    // Everything after `abc` is handled by `match_range_end`\n    //\n    // if we are in the 2nd case, we want to visit the following states:\n    // abc (\n    //   d(g.+|[h-\\xff].*) | // this is handled by match_range_start\n    //\n    //   [e-i].* |           // this is handled here\n    //\n    //   j (                 // this is handled by match_range_end (but countrary to the other\n    //    [\\0-o].* |         // case, j is already consumed so to not check [\\0-i].* )\n    //    p\n    //   )?\n    // )\n\n    let Some(start_range) = start_key.get(common_prefix_len) else {\n        return match_range_end(&end_key[common_prefix_len..], &automaton, base_state);\n    };\n\n    let end_range = end_key[common_prefix_len];\n\n    // things starting with start_range were handled in match_range_start\n    // this starting with end_range are handled below.\n    // this can run for 0 iteration in cases such as (abc, abd]\n    for rb in (start_range + 1)..end_range {\n        let new_state = automaton.accept(&base_state, rb);\n        if automaton.can_match(&new_state) {\n            return true;\n        }\n    }\n\n    let state_for_start = automaton.accept(&base_state, *start_range);\n    if match_range_start(\n        &start_key[common_prefix_len + 1..],\n        &automaton,\n        state_for_start,\n    ) {\n        return true;\n    }\n\n    let state_for_end = automaton.accept(&base_state, end_range);\n    if automaton.is_match(&state_for_end) {\n        return true;\n    }\n    match_range_end(&end_key[common_prefix_len + 1..], &automaton, state_for_end)\n}\n\nfn match_range_start<S, A: Automaton<State = S>>(\n    start_key: &[u8],\n    automaton: &A,\n    mut state: S,\n) -> bool {\n    // case (abcdgj, abcpqr], `abcd` is already consumed, we need to handle:\n    // - [h-\\xff].*\n    // - g[k-\\xff].*\n    // - gj.+ == gf[\\0-\\xff].*\n\n    for kb in start_key {\n        // this is an optimisation, and is not needed for correctness\n        if !automaton.can_match(&state) {\n            return false;\n        }\n\n        // does the [h-\\xff].* part. we skip if kb==255 as [\\{0100}-\\xff] is an empty range, and\n        // this would overflow in our u8 world\n        if *kb < u8::MAX {\n            for rb in (kb + 1)..=u8::MAX {\n                let temp_state = automaton.accept(&state, rb);\n                if automaton.can_match(&temp_state) {\n                    return true;\n                }\n            }\n        }\n        // push g\n        state = automaton.accept(&state, *kb);\n    }\n\n    // this isn't required for correctness, but can save us from looping 256 below\n    if !automaton.can_match(&state) {\n        return false;\n    }\n\n    // does the final `.+`, which is the same as `[\\0-\\xff].*`\n    for rb in 0..=u8::MAX {\n        let temp_state = automaton.accept(&state, rb);\n        if automaton.can_match(&temp_state) {\n            return true;\n        }\n    }\n    false\n}\n\nfn match_range_end<S, A: Automaton<State = S>>(\n    end_key: &[u8],\n    automaton: &A,\n    mut state: S,\n) -> bool {\n    // for (abcdef, abcmps]. the prefix `abcm` has been consumed, `[d-l].*` was handled elsewhere,\n    // we just need to handle\n    // - [\\0-o].*\n    // - p\n    // - p[\\0-r].*\n    // - ps\n    for kb in end_key {\n        // this is an optimisation, and is not needed for correctness\n        if !automaton.can_match(&state) {\n            return false;\n        }\n\n        // does the `[\\0-o].*`\n        for rb in 0..*kb {\n            let temp_state = automaton.accept(&state, rb);\n            if automaton.can_match(&temp_state) {\n                return true;\n            }\n        }\n\n        // push p\n        state = automaton.accept(&state, *kb);\n        // verify the `p` case\n        if automaton.is_match(&state) {\n            return true;\n        }\n    }\n    false\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use proptest::prelude::*;\n    use tantivy_fst::Automaton;\n\n    use super::*;\n\n    pub(crate) struct EqBuffer(pub Vec<u8>);\n\n    impl Automaton for EqBuffer {\n        type State = Option<usize>;\n\n        fn start(&self) -> Self::State {\n            Some(0)\n        }\n\n        fn is_match(&self, state: &Self::State) -> bool {\n            *state == Some(self.0.len())\n        }\n\n        fn accept(&self, state: &Self::State, byte: u8) -> Self::State {\n            state\n                .filter(|pos| self.0.get(*pos) == Some(&byte))\n                .map(|pos| pos + 1)\n        }\n\n        fn can_match(&self, state: &Self::State) -> bool {\n            state.is_some()\n        }\n\n        fn will_always_match(&self, _state: &Self::State) -> bool {\n            false\n        }\n    }\n\n    fn gen_key_strategy() -> impl Strategy<Value = Vec<u8>> {\n        // we only generate bytes in [0, 1, 2, 254, 255] to reduce the search space without\n        // ignoring edge cases that might ocure with integer over/underflow\n        proptest::collection::vec(prop_oneof![0u8..=2, 254u8..=255], 0..5)\n    }\n\n    proptest! {\n        #![proptest_config(ProptestConfig {\n            cases: 10000, .. ProptestConfig::default()\n        })]\n\n        #[test]\n        fn test_proptest_automaton_match_block(start in gen_key_strategy(), end in gen_key_strategy(), key in gen_key_strategy()) {\n            let expected = start < key && end >= key;\n            let automaton = EqBuffer(key);\n\n            assert_eq!(can_block_match_automaton(Some(&start), &end, &automaton), expected);\n        }\n\n        #[test]\n        fn test_proptest_automaton_match_first_block(end in gen_key_strategy(), key in gen_key_strategy()) {\n            let expected = end >= key;\n            let automaton = EqBuffer(key);\n            assert_eq!(can_block_match_automaton(None, &end, &automaton), expected);\n        }\n    }\n}\n"
  },
  {
    "path": "sstable/src/block_reader.rs",
    "content": "use std::io::{self, Read};\nuse std::ops::Range;\n\nuse common::OwnedBytes;\n#[cfg(feature = \"zstd-compression\")]\nuse zstd::bulk::Decompressor;\n\npub struct BlockReader {\n    buffer: Vec<u8>,\n    reader: OwnedBytes,\n    next_readers: std::vec::IntoIter<OwnedBytes>,\n    offset: usize,\n}\n\nimpl BlockReader {\n    pub fn new(reader: OwnedBytes) -> BlockReader {\n        BlockReader {\n            buffer: Vec::new(),\n            reader,\n            next_readers: Vec::new().into_iter(),\n            offset: 0,\n        }\n    }\n\n    pub fn from_multiple_blocks(readers: Vec<OwnedBytes>) -> BlockReader {\n        let mut next_readers = readers.into_iter();\n        let reader = next_readers.next().unwrap_or_else(OwnedBytes::empty);\n        BlockReader {\n            buffer: Vec::new(),\n            reader,\n            next_readers,\n            offset: 0,\n        }\n    }\n\n    pub fn deserialize_u64(&mut self) -> u64 {\n        let (num_bytes, val) = super::vint::deserialize_read(self.buffer());\n        self.advance(num_bytes);\n        val\n    }\n\n    #[inline(always)]\n    pub fn buffer_from_to(&self, range: Range<usize>) -> &[u8] {\n        &self.buffer[range]\n    }\n\n    pub fn read_block(&mut self) -> io::Result<bool> {\n        self.offset = 0;\n        self.buffer.clear();\n\n        loop {\n            let block_len = match self.reader.len() {\n                0 => {\n                    // we are out of data for this block. Check if we have another block after\n                    match self.next_readers.next() {\n                        Some(new_reader) => {\n                            self.reader = new_reader;\n                            continue;\n                        }\n                        _ => {\n                            return Ok(false);\n                        }\n                    }\n                }\n                1..=3 => {\n                    return Err(io::Error::new(\n                        io::ErrorKind::UnexpectedEof,\n                        \"failed to read block_len\",\n                    ));\n                }\n                _ => self.reader.read_u32() as usize,\n            };\n            if block_len <= 1 {\n                return Ok(false);\n            }\n            let compress = self.reader.read_u8();\n            let block_len = block_len - 1;\n\n            if self.reader.len() < block_len {\n                return Err(io::Error::new(\n                    io::ErrorKind::UnexpectedEof,\n                    \"failed to read block content\",\n                ));\n            }\n            if compress == 1 {\n                #[cfg(feature = \"zstd-compression\")]\n                {\n                    let required_capacity =\n                        Decompressor::upper_bound(&self.reader[..block_len]).unwrap_or(1024 * 1024);\n                    self.buffer.reserve(required_capacity);\n                    Decompressor::new()?\n                        .decompress_to_buffer(&self.reader[..block_len], &mut self.buffer)?;\n\n                    self.reader.advance(block_len);\n                }\n\n                if cfg!(not(feature = \"zstd-compression\")) {\n                    return Err(io::Error::new(\n                        io::ErrorKind::Unsupported,\n                        \"zstd-compression feature is not enabled\",\n                    ));\n                }\n            } else {\n                self.buffer.resize(block_len, 0u8);\n                self.reader.read_exact(&mut self.buffer[..])?;\n            }\n\n            return Ok(true);\n        }\n    }\n\n    #[inline(always)]\n    pub fn offset(&self) -> usize {\n        self.offset\n    }\n\n    #[inline(always)]\n    pub fn advance(&mut self, num_bytes: usize) {\n        self.offset += num_bytes;\n    }\n\n    #[inline(always)]\n    pub fn buffer(&self) -> &[u8] {\n        &self.buffer[self.offset..]\n    }\n}\n\nimpl io::Read for BlockReader {\n    fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {\n        let len = self.buffer().read(buf)?;\n        self.advance(len);\n        Ok(len)\n    }\n\n    fn read_to_end(&mut self, buf: &mut Vec<u8>) -> io::Result<usize> {\n        let len = self.buffer.len();\n        buf.extend_from_slice(self.buffer());\n        self.advance(len);\n        Ok(len)\n    }\n\n    fn read_exact(&mut self, buf: &mut [u8]) -> io::Result<()> {\n        self.buffer().read_exact(buf)?;\n        self.advance(buf.len());\n        Ok(())\n    }\n}\n"
  },
  {
    "path": "sstable/src/delta.rs",
    "content": "use std::io::{self, BufWriter, Write};\nuse std::ops::Range;\n\nuse common::{CountingWriter, OwnedBytes};\n#[cfg(feature = \"zstd-compression\")]\nuse zstd::bulk::Compressor;\n\nuse super::value::ValueWriter;\nuse super::{BlockReader, value, vint};\n\nconst FOUR_BIT_LIMITS: usize = 1 << 4;\nconst VINT_MODE: u8 = 1u8;\nconst BLOCK_LEN: usize = 4_000;\n\npub struct DeltaWriter<W, TValueWriter>\nwhere W: io::Write\n{\n    block: Vec<u8>,\n    write: CountingWriter<BufWriter<W>>,\n    value_writer: TValueWriter,\n    // Only here to avoid allocations.\n    stateless_buffer: Vec<u8>,\n    block_len: usize,\n}\n\nimpl<W, TValueWriter> DeltaWriter<W, TValueWriter>\nwhere\n    W: io::Write,\n    TValueWriter: ValueWriter,\n{\n    pub fn new(wrt: W) -> Self {\n        DeltaWriter {\n            block: Vec::with_capacity(BLOCK_LEN * 2),\n            write: CountingWriter::wrap(BufWriter::new(wrt)),\n            value_writer: TValueWriter::default(),\n            stateless_buffer: Vec::new(),\n            block_len: BLOCK_LEN,\n        }\n    }\n\n    pub fn set_block_len(&mut self, block_len: usize) {\n        self.block_len = block_len\n    }\n\n    pub fn flush_block(&mut self) -> io::Result<Option<Range<usize>>> {\n        if self.block.is_empty() {\n            return Ok(None);\n        }\n        let start_offset = self.write.written_bytes() as usize;\n\n        let buffer: &mut Vec<u8> = &mut self.stateless_buffer;\n        self.value_writer.serialize_block(buffer);\n        self.value_writer.clear();\n\n        let block_len = buffer.len() + self.block.len();\n\n        if cfg!(feature = \"zstd-compression\") && block_len > 2048 {\n            #[cfg(feature = \"zstd-compression\")]\n            {\n                buffer.extend_from_slice(&self.block);\n                self.block.clear();\n\n                let max_len = zstd::zstd_safe::compress_bound(buffer.len());\n                self.block.reserve(max_len);\n                Compressor::new(3)?.compress_to_buffer(buffer, &mut self.block)?;\n\n                // verify compression had a positive impact\n                if self.block.len() < buffer.len() {\n                    self.write\n                        .write_all(&(self.block.len() as u32 + 1).to_le_bytes())?;\n                    self.write.write_all(&[1])?;\n                    self.write.write_all(&self.block[..])?;\n                } else {\n                    self.write\n                        .write_all(&(block_len as u32 + 1).to_le_bytes())?;\n                    self.write.write_all(&[0])?;\n                    self.write.write_all(&buffer[..])?;\n                }\n            }\n        } else {\n            self.write\n                .write_all(&(block_len as u32 + 1).to_le_bytes())?;\n            self.write.write_all(&[0])?;\n            self.write.write_all(&buffer[..])?;\n            self.write.write_all(&self.block[..])?;\n        }\n\n        let end_offset = self.write.written_bytes() as usize;\n        self.block.clear();\n        buffer.clear();\n        Ok(Some(start_offset..end_offset))\n    }\n\n    fn encode_keep_add(&mut self, keep_len: usize, add_len: usize) {\n        if keep_len < FOUR_BIT_LIMITS && add_len < FOUR_BIT_LIMITS {\n            let b = (keep_len | (add_len << 4)) as u8;\n            self.block.extend_from_slice(&[b])\n        } else {\n            let mut buf = [VINT_MODE; 20];\n            let mut len = 1 + vint::serialize(keep_len as u64, &mut buf[1..]);\n            len += vint::serialize(add_len as u64, &mut buf[len..]);\n            self.block.extend_from_slice(&buf[..len])\n        }\n    }\n\n    pub(crate) fn write_suffix(&mut self, common_prefix_len: usize, suffix: &[u8]) {\n        let keep_len = common_prefix_len;\n        let add_len = suffix.len();\n        self.encode_keep_add(keep_len, add_len);\n        self.block.extend_from_slice(suffix);\n    }\n\n    pub(crate) fn write_value(&mut self, value: &TValueWriter::Value) {\n        self.value_writer.write(value);\n    }\n\n    pub fn flush_block_if_required(&mut self) -> io::Result<Option<Range<usize>>> {\n        if self.block.len() > self.block_len {\n            return self.flush_block();\n        }\n        Ok(None)\n    }\n\n    pub fn finish(self) -> CountingWriter<BufWriter<W>> {\n        self.write\n    }\n}\n\npub struct DeltaReader<TValueReader> {\n    common_prefix_len: usize,\n    suffix_range: Range<usize>,\n    value_reader: TValueReader,\n    block_reader: BlockReader,\n    idx: usize,\n}\n\nimpl<TValueReader> DeltaReader<TValueReader>\nwhere TValueReader: value::ValueReader\n{\n    pub fn new(reader: OwnedBytes) -> Self {\n        DeltaReader {\n            idx: 0,\n            common_prefix_len: 0,\n            suffix_range: 0..0,\n            value_reader: TValueReader::default(),\n            block_reader: BlockReader::new(reader),\n        }\n    }\n\n    pub fn from_multiple_blocks(reader: Vec<OwnedBytes>) -> Self {\n        DeltaReader {\n            idx: 0,\n            common_prefix_len: 0,\n            suffix_range: 0..0,\n            value_reader: TValueReader::default(),\n            block_reader: BlockReader::from_multiple_blocks(reader),\n        }\n    }\n\n    pub fn empty() -> Self {\n        DeltaReader::new(OwnedBytes::empty())\n    }\n\n    fn deserialize_vint(&mut self) -> u64 {\n        self.block_reader.deserialize_u64()\n    }\n\n    fn read_keep_add(&mut self) -> Option<(usize, usize)> {\n        let b = {\n            let buf = &self.block_reader.buffer();\n            if buf.is_empty() {\n                return None;\n            }\n            buf[0]\n        };\n        self.block_reader.advance(1);\n        match b {\n            VINT_MODE => {\n                let keep = self.deserialize_vint() as usize;\n                let add = self.deserialize_vint() as usize;\n                Some((keep, add))\n            }\n            b => {\n                let keep = (b & 0b1111) as usize;\n                let add = (b >> 4) as usize;\n                Some((keep, add))\n            }\n        }\n    }\n\n    fn read_delta_key(&mut self) -> bool {\n        let Some((keep, add)) = self.read_keep_add() else {\n            return false;\n        };\n        self.common_prefix_len = keep;\n        let suffix_start = self.block_reader.offset();\n        self.suffix_range = suffix_start..(suffix_start + add);\n        self.block_reader.advance(add);\n        true\n    }\n\n    pub fn advance(&mut self) -> io::Result<bool> {\n        if self.block_reader.buffer().is_empty() {\n            if !self.block_reader.read_block()? {\n                return Ok(false);\n            }\n            let consumed_len = self.value_reader.load(self.block_reader.buffer())?;\n            self.block_reader.advance(consumed_len);\n            self.idx = 0;\n        } else {\n            self.idx += 1;\n        }\n        if !self.read_delta_key() {\n            return Ok(false);\n        }\n        Ok(true)\n    }\n\n    #[inline(always)]\n    pub fn common_prefix_len(&self) -> usize {\n        self.common_prefix_len\n    }\n\n    #[inline(always)]\n    pub fn suffix(&self) -> &[u8] {\n        self.block_reader.buffer_from_to(self.suffix_range.clone())\n    }\n\n    #[inline(always)]\n    pub fn value(&self) -> &TValueReader::Value {\n        self.value_reader.value(self.idx)\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::DeltaReader;\n    use crate::value::U64MonotonicValueReader;\n\n    #[test]\n    fn test_empty() {\n        let mut delta_reader: DeltaReader<U64MonotonicValueReader> = DeltaReader::empty();\n        assert!(!delta_reader.advance().unwrap());\n    }\n}\n"
  },
  {
    "path": "sstable/src/dictionary.rs",
    "content": "#![allow(clippy::needless_borrows_for_generic_args)]\n\nuse std::cmp::Ordering;\nuse std::io;\nuse std::marker::PhantomData;\nuse std::ops::{Bound, RangeBounds};\nuse std::sync::Arc;\n\nuse common::bounds::{TransformBound, transform_bound_inner_res};\nuse common::file_slice::FileSlice;\nuse common::{BinarySerializable, ByteCount, OwnedBytes};\nuse futures_util::{StreamExt, TryStreamExt, stream};\nuse itertools::Itertools;\nuse tantivy_fst::Automaton;\nuse tantivy_fst::automaton::AlwaysMatch;\n\nuse crate::sstable_index_v3::SSTableIndexV3Empty;\nuse crate::streamer::{Streamer, StreamerBuilder};\nuse crate::{\n    BlockAddr, DeltaReader, Reader, SSTable, SSTableIndex, SSTableIndexV3, TermOrdinal, VoidSSTable,\n};\n\n/// An SSTable is a sorted map that associates sorted `&[u8]` keys\n/// to any kind of typed values.\n///\n/// The SSTable is organized in blocks.\n/// In each block, keys and values are encoded separately.\n///\n/// The keys are encoded using incremental encoding.\n/// The values on the other hand, are encoded according to a value-specific\n/// codec defined in the TSSTable generic argument.\n///\n/// Finally, an index is joined to the Dictionary to make it possible,\n/// given a key to identify which block contains this key.\n///\n/// The codec was designed in such a way that the sstable\n/// reader is not aware of block, and yet can read any sequence of blocks,\n/// as long as the slice of bytes it is given starts and stops at\n/// block boundary.\n///\n/// (See also README.md)\n#[derive(Debug, Clone)]\npub struct Dictionary<TSSTable: SSTable = VoidSSTable> {\n    pub sstable_slice: FileSlice,\n    pub sstable_index: SSTableIndex,\n    num_bytes: ByteCount,\n    num_terms: u64,\n    phantom_data: PhantomData<TSSTable>,\n}\n\nimpl Dictionary<VoidSSTable> {\n    pub fn build_for_tests(terms: &[&str]) -> Dictionary {\n        let mut terms = terms.to_vec();\n        terms.sort();\n        let mut buffer = Vec::new();\n        let mut dictionary_writer = Self::builder(&mut buffer).unwrap();\n        for term in terms {\n            dictionary_writer.insert(term, &()).unwrap();\n        }\n        dictionary_writer.finish().unwrap();\n        Dictionary::from_bytes(OwnedBytes::new(buffer)).unwrap()\n    }\n}\n\n#[derive(Debug, Clone, PartialEq, Eq)]\npub enum TermOrdHit {\n    /// Exact term ord hit\n    Exact(TermOrdinal),\n    /// Next best term ordinal\n    Next(TermOrdinal),\n}\n\nimpl TermOrdHit {\n    fn into_exact(self) -> Option<TermOrdinal> {\n        match self {\n            TermOrdHit::Exact(ord) => Some(ord),\n            TermOrdHit::Next(_) => None,\n        }\n    }\n\n    fn map<F: FnOnce(TermOrdinal) -> TermOrdinal>(self, f: F) -> Self {\n        match self {\n            TermOrdHit::Exact(ord) => TermOrdHit::Exact(f(ord)),\n            TermOrdHit::Next(ord) => TermOrdHit::Next(f(ord)),\n        }\n    }\n}\n\nimpl<TSSTable: SSTable> Dictionary<TSSTable> {\n    pub fn builder<W: io::Write>(wrt: W) -> io::Result<crate::Writer<W, TSSTable::ValueWriter>> {\n        Ok(TSSTable::writer(wrt))\n    }\n\n    pub(crate) fn sstable_reader_block(\n        &self,\n        block_addr: BlockAddr,\n    ) -> io::Result<Reader<TSSTable::ValueReader>> {\n        let data = self.sstable_slice.read_bytes_slice(block_addr.byte_range)?;\n        Ok(TSSTable::reader(data))\n    }\n\n    pub(crate) async fn sstable_delta_reader_for_key_range_async(\n        &self,\n        key_range: impl RangeBounds<[u8]>,\n        limit: Option<u64>,\n        automaton: &impl Automaton,\n        merge_holes_under_bytes: usize,\n    ) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let match_all = automaton.will_always_match(&automaton.start());\n        if match_all {\n            let slice = self.file_slice_for_range(key_range, limit);\n            let data = slice.read_bytes_async().await?;\n            Ok(TSSTable::delta_reader(data))\n        } else {\n            let blocks = stream::iter(self.get_block_iterator_for_range_and_automaton(\n                key_range,\n                automaton,\n                merge_holes_under_bytes,\n            ));\n            let data = blocks\n                .map(|block_addr| {\n                    self.sstable_slice\n                        .read_bytes_slice_async(block_addr.byte_range)\n                })\n                .buffered(5)\n                .try_collect::<Vec<_>>()\n                .await?;\n            Ok(DeltaReader::from_multiple_blocks(data))\n        }\n    }\n\n    pub(crate) fn sstable_delta_reader_for_key_range(\n        &self,\n        key_range: impl RangeBounds<[u8]>,\n        limit: Option<u64>,\n        automaton: &impl Automaton,\n    ) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let match_all = automaton.will_always_match(&automaton.start());\n        if match_all {\n            let slice = self.file_slice_for_range(key_range, limit);\n            let data = slice.read_bytes()?;\n            Ok(TSSTable::delta_reader(data))\n        } else {\n            // if operations are sync, we assume latency is almost null, and there is no point in\n            // merging across holes\n            let blocks = self.get_block_iterator_for_range_and_automaton(key_range, automaton, 0);\n            let data = blocks\n                .map(|block_addr| self.sstable_slice.read_bytes_slice(block_addr.byte_range))\n                .collect::<Result<Vec<_>, _>>()?;\n            Ok(DeltaReader::from_multiple_blocks(data))\n        }\n    }\n\n    pub(crate) fn sstable_delta_reader_block(\n        &self,\n        block_addr: BlockAddr,\n    ) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let data = self.sstable_slice.read_bytes_slice(block_addr.byte_range)?;\n        Ok(TSSTable::delta_reader(data))\n    }\n\n    pub(crate) async fn sstable_delta_reader_block_async(\n        &self,\n        block_addr: BlockAddr,\n    ) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let data = self\n            .sstable_slice\n            .read_bytes_slice_async(block_addr.byte_range)\n            .await?;\n        Ok(TSSTable::delta_reader(data))\n    }\n\n    /// This function returns a file slice covering a set of sstable blocks\n    /// that include the key range passed in arguments. Optionally returns\n    /// only block for up to `limit` matching terms.\n    ///\n    /// It works by identifying\n    /// - `first_block`: the block containing the start boundary key\n    /// - `last_block`: the block containing the end boundary key.\n    ///\n    /// And then returning the range that spans over all blocks between.\n    /// and including first_block and last_block, aka:\n    /// `[first_block.start_offset .. last_block.end_offset)`\n    ///\n    /// Technically this function does not provide the tightest fit, as\n    /// for simplification, it treats the start bound of the `key_range`\n    /// as if it was inclusive, even if it is exclusive.\n    /// On the rare edge case where a user asks for `(start_key, end_key]`\n    /// and `start_key` happens to be the last key of a block, we return a\n    /// slice that is the first block was not necessary.\n    pub fn file_slice_for_range(\n        &self,\n        key_range: impl RangeBounds<[u8]>,\n        limit: Option<u64>,\n    ) -> FileSlice {\n        let first_block_id = match key_range.start_bound() {\n            Bound::Included(key) | Bound::Excluded(key) => {\n                let Some(first_block_id) = self.sstable_index.locate_with_key(key) else {\n                    return FileSlice::empty();\n                };\n                Some(first_block_id)\n            }\n            Bound::Unbounded => None,\n        };\n\n        let last_block_id = match key_range.end_bound() {\n            Bound::Included(key) | Bound::Excluded(key) => self.sstable_index.locate_with_key(key),\n            Bound::Unbounded => None,\n        };\n\n        let start_bound = if let Some(first_block_id) = first_block_id {\n            let Some(block_addr) = self.sstable_index.get_block(first_block_id) else {\n                return FileSlice::empty();\n            };\n            Bound::Included(block_addr.byte_range.start)\n        } else {\n            Bound::Unbounded\n        };\n\n        let last_block_id = if let Some(limit) = limit {\n            let second_block_id = first_block_id.map(|id| id + 1).unwrap_or(0);\n            if let Some(block_addr) = self.sstable_index.get_block(second_block_id) {\n                let ordinal_limit = block_addr.first_ordinal + limit;\n                let last_block_limit = self.sstable_index.locate_with_ord(ordinal_limit);\n                if let Some(last_block_id) = last_block_id {\n                    Some(last_block_id.min(last_block_limit))\n                } else {\n                    Some(last_block_limit)\n                }\n            } else {\n                last_block_id\n            }\n        } else {\n            last_block_id\n        };\n        let end_bound = last_block_id\n            .and_then(|block_id| self.sstable_index.get_block(block_id))\n            .map(|block_addr| Bound::Excluded(block_addr.byte_range.end))\n            .unwrap_or(Bound::Unbounded);\n\n        self.sstable_slice.slice((start_bound, end_bound))\n    }\n\n    fn get_block_iterator_for_range_and_automaton<'a>(\n        &'a self,\n        key_range: impl RangeBounds<[u8]>,\n        automaton: &'a impl Automaton,\n        merge_holes_under_bytes: usize,\n    ) -> impl Iterator<Item = BlockAddr> + 'a {\n        let lower_bound = match key_range.start_bound() {\n            Bound::Included(key) | Bound::Excluded(key) => {\n                self.sstable_index.locate_with_key(key).unwrap_or(u64::MAX)\n            }\n            Bound::Unbounded => 0,\n        };\n\n        let upper_bound = match key_range.end_bound() {\n            Bound::Included(key) | Bound::Excluded(key) => {\n                self.sstable_index.locate_with_key(key).unwrap_or(u64::MAX)\n            }\n            Bound::Unbounded => u64::MAX,\n        };\n        let block_range = lower_bound..=upper_bound;\n        self.sstable_index\n            .get_block_for_automaton(automaton)\n            .filter(move |(block_id, _)| block_range.contains(block_id))\n            .map(|(_, block_addr)| block_addr)\n            .coalesce(move |first, second| {\n                if first.byte_range.end + merge_holes_under_bytes >= second.byte_range.start {\n                    Ok(BlockAddr {\n                        first_ordinal: first.first_ordinal,\n                        byte_range: first.byte_range.start..second.byte_range.end,\n                    })\n                } else {\n                    Err((first, second))\n                }\n            })\n    }\n\n    /// Opens a `TermDictionary`.\n    pub fn open(term_dictionary_file: FileSlice) -> io::Result<Self> {\n        let num_bytes = term_dictionary_file.num_bytes();\n        let (main_slice, footer_len_slice) = term_dictionary_file.split_from_end(20);\n        let mut footer_len_bytes: OwnedBytes = footer_len_slice.read_bytes()?;\n        let index_offset = u64::deserialize(&mut footer_len_bytes)?;\n        let num_terms = u64::deserialize(&mut footer_len_bytes)?;\n        let version = u32::deserialize(&mut footer_len_bytes)?;\n        let (sstable_slice, index_slice) = main_slice.split(index_offset as usize);\n        let sstable_index_bytes = index_slice.read_bytes()?;\n\n        let sstable_index = match version {\n            2 => SSTableIndex::V2(\n                crate::sstable_index_v2::SSTableIndex::load(sstable_index_bytes).map_err(|_| {\n                    io::Error::new(io::ErrorKind::InvalidData, \"SSTable corruption\")\n                })?,\n            ),\n            3 => {\n                let (sstable_index_bytes, mut footerv3_len_bytes) = sstable_index_bytes.rsplit(8);\n                let store_offset = u64::deserialize(&mut footerv3_len_bytes)?;\n                if store_offset != 0 {\n                    SSTableIndex::V3(\n                        SSTableIndexV3::load(sstable_index_bytes, store_offset).map_err(|_| {\n                            io::Error::new(io::ErrorKind::InvalidData, \"SSTable corruption\")\n                        })?,\n                    )\n                } else {\n                    // if store_offset is zero, there is no index, so we build a pseudo-index\n                    // assuming a single block of sstable covering everything.\n                    SSTableIndex::V3Empty(SSTableIndexV3Empty::load(index_offset as usize))\n                }\n            }\n            _ => {\n                return Err(io::Error::other(format!(\n                    \"Unsupported sstable version, expected one of [2, 3], found {version}\"\n                )));\n            }\n        };\n\n        Ok(Dictionary {\n            sstable_slice,\n            sstable_index,\n            num_bytes,\n            num_terms,\n            phantom_data: PhantomData,\n        })\n    }\n\n    /// Creates a term dictionary from the supplied bytes.\n    pub fn from_bytes(owned_bytes: OwnedBytes) -> io::Result<Self> {\n        Dictionary::open(FileSlice::new(Arc::new(owned_bytes)))\n    }\n\n    /// Creates an empty term dictionary which contains no terms.\n    pub fn empty() -> Self {\n        let term_dictionary_data: Vec<u8> = Self::builder(Vec::<u8>::new())\n            .expect(\"Creating a TermDictionaryBuilder in a Vec<u8> should never fail\")\n            .finish()\n            .expect(\"Writing in a Vec<u8> should never fail\");\n        let empty_dict_file = FileSlice::from(term_dictionary_data);\n        Dictionary::open(empty_dict_file).unwrap()\n    }\n\n    /// Returns the number of terms in the dictionary.\n    /// Term ordinals range from 0 to `num_terms() - 1`.\n    pub fn num_terms(&self) -> usize {\n        self.num_terms as usize\n    }\n\n    /// Returns the total number of bytes used by the dictionary on disk.\n    pub fn num_bytes(&self) -> ByteCount {\n        self.num_bytes\n    }\n\n    /// Decode a DeltaReader up to key, returning the number of terms traversed\n    ///\n    /// If the key was not found, returns Ok(None).\n    /// After calling this function, it is possible to call `DeltaReader::value` to get the\n    /// associated value.\n    fn decode_up_to_key<K: AsRef<[u8]>>(\n        &self,\n        key: K,\n        sstable_delta_reader: &mut DeltaReader<TSSTable::ValueReader>,\n    ) -> io::Result<Option<TermOrdinal>> {\n        self.decode_up_to_or_next(key, sstable_delta_reader)\n            .map(|hit| hit.into_exact())\n    }\n    /// Decode a DeltaReader up to key, returning the number of terms traversed\n    ///\n    /// If the key was not found, it returns the next term id.\n    fn decode_up_to_or_next<K: AsRef<[u8]>>(\n        &self,\n        key: K,\n        sstable_delta_reader: &mut DeltaReader<TSSTable::ValueReader>,\n    ) -> io::Result<TermOrdHit> {\n        let mut term_ord = 0;\n        let key_bytes = key.as_ref();\n        let mut ok_bytes = 0;\n        while sstable_delta_reader.advance()? {\n            let prefix_len = sstable_delta_reader.common_prefix_len();\n            let suffix = sstable_delta_reader.suffix();\n\n            match prefix_len.cmp(&ok_bytes) {\n                Ordering::Less => return Ok(TermOrdHit::Next(term_ord)), /* popped bytes already matched => too far */\n                Ordering::Equal => (),\n                Ordering::Greater => {\n                    // the ok prefix is less than current entry prefix => continue to next elem\n                    term_ord += 1;\n                    continue;\n                }\n            }\n\n            // we have ok_bytes byte of common prefix, check if this key adds more\n            for (key_byte, suffix_byte) in key_bytes[ok_bytes..].iter().zip(suffix) {\n                match suffix_byte.cmp(key_byte) {\n                    Ordering::Less => break,          // byte too small\n                    Ordering::Equal => ok_bytes += 1, // new matching\n                    // byte\n                    Ordering::Greater => return Ok(TermOrdHit::Next(term_ord)), // too far\n                }\n            }\n\n            if ok_bytes == key_bytes.len() {\n                if prefix_len + suffix.len() == ok_bytes {\n                    return Ok(TermOrdHit::Exact(term_ord));\n                } else {\n                    // current key is a prefix of current element, not a match\n                    return Ok(TermOrdHit::Next(term_ord));\n                }\n            }\n\n            term_ord += 1;\n        }\n\n        Ok(TermOrdHit::Next(term_ord))\n    }\n\n    /// Returns the ordinal associated with a given term.\n    pub fn term_ord<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TermOrdinal>> {\n        let key_bytes = key.as_ref();\n\n        let Some(block_addr) = self.sstable_index.get_block_with_key(key_bytes) else {\n            return Ok(None);\n        };\n\n        let first_ordinal = block_addr.first_ordinal;\n        let mut sstable_delta_reader = self.sstable_delta_reader_block(block_addr)?;\n        self.decode_up_to_key(key_bytes, &mut sstable_delta_reader)\n            .map(|opt| opt.map(|ord| ord + first_ordinal))\n    }\n\n    /// Returns the ordinal associated with a given term or its closest next term_id\n    /// The closest next term_id may not exist.\n    pub fn term_ord_or_next<K: AsRef<[u8]>>(&self, key: K) -> io::Result<TermOrdHit> {\n        let key_bytes = key.as_ref();\n\n        let Some(block_addr) = self.sstable_index.get_block_with_key(key_bytes) else {\n            // TODO: Would be more consistent to return last_term id + 1\n            return Ok(TermOrdHit::Next(u64::MAX));\n        };\n\n        let first_ordinal = block_addr.first_ordinal;\n        let mut sstable_delta_reader = self.sstable_delta_reader_block(block_addr)?;\n        self.decode_up_to_or_next(key_bytes, &mut sstable_delta_reader)\n            .map(|opt| opt.map(|ord| ord + first_ordinal))\n    }\n\n    /// Converts strings into a Bound range.\n    /// This does handle several special cases if the term is not exactly in the dictionary.\n    /// e.g. [bbb, ddd]\n    /// lower_bound: Bound::Included(aaa) => Included(0) // \"Next\" term id\n    /// lower_bound: Bound::Excluded(aaa) => Included(0) // \"Next\" term id + Change the Bounds\n    /// lower_bound: Bound::Included(ccc) => Included(1) // \"Next\" term id\n    /// lower_bound: Bound::Excluded(ccc) => Included(1) // \"Next\" term id + Change the Bounds\n    /// lower_bound: Bound::Included(zzz) => Included(2) // \"Next\" term id\n    /// lower_bound: Bound::Excluded(zzz) => Included(2) // \"Next\" term id + Change the Bounds\n    /// For zzz we should have some post processing to return an empty query`\n    ///\n    /// upper_bound: Bound::Included(aaa) => Excluded(0) // \"Next\" term id + Change the bounds\n    /// upper_bound: Bound::Excluded(aaa) => Excluded(0) // \"Next\" term id\n    /// upper_bound: Bound::Included(ccc) => Excluded(1) // Next term id + Change the bounds\n    /// upper_bound: Bound::Excluded(ccc) => Excluded(1) // Next term id\n    /// upper_bound: Bound::Included(zzz) => Excluded(2) // Next term id + Change the bounds\n    /// upper_bound: Bound::Excluded(zzz) => Excluded(2) // Next term id\n    pub fn term_bounds_to_ord<K: AsRef<[u8]>>(\n        &self,\n        lower_bound: Bound<K>,\n        upper_bound: Bound<K>,\n    ) -> io::Result<(Bound<TermOrdinal>, Bound<TermOrdinal>)> {\n        let lower_bound = transform_bound_inner_res(&lower_bound, |start_bound_bytes| {\n            let ord = self.term_ord_or_next(start_bound_bytes)?;\n            match ord {\n                TermOrdHit::Exact(ord) => Ok(TransformBound::Existing(ord)),\n                TermOrdHit::Next(ord) => Ok(TransformBound::NewBound(Bound::Included(ord))), /* Change bounds to included */\n            }\n        })?;\n        let upper_bound = transform_bound_inner_res(&upper_bound, |end_bound_bytes| {\n            let ord = self.term_ord_or_next(end_bound_bytes)?;\n            match ord {\n                TermOrdHit::Exact(ord) => Ok(TransformBound::Existing(ord)),\n                TermOrdHit::Next(ord) => Ok(TransformBound::NewBound(Bound::Excluded(ord))), /* Change bounds to excluded */\n            }\n        })?;\n        Ok((lower_bound, upper_bound))\n    }\n\n    /// Returns the term associated with a given term ordinal.\n    ///\n    /// Term ordinals are defined as the position of the term in\n    /// the sorted list of terms.\n    ///\n    /// Returns true if and only if the term has been found.\n    ///\n    /// Regardless of whether the term is found or not,\n    /// the buffer may be modified.\n    pub fn ord_to_term(&self, ord: TermOrdinal, bytes: &mut Vec<u8>) -> io::Result<bool> {\n        // find block in which the term would be\n        let block_addr = self.sstable_index.get_block_with_ord(ord);\n        let first_ordinal = block_addr.first_ordinal;\n\n        // then search inside that block only\n        let mut sstable_delta_reader = self.sstable_delta_reader_block(block_addr)?;\n        for _ in first_ordinal..=ord {\n            if !sstable_delta_reader.advance()? {\n                return Ok(false);\n            }\n            bytes.truncate(sstable_delta_reader.common_prefix_len());\n            bytes.extend_from_slice(sstable_delta_reader.suffix());\n        }\n        Ok(true)\n    }\n\n    /// Returns the terms for a _sorted_ list of term ordinals.\n    ///\n    /// Returns true if and only if all terms have been found.\n    pub fn sorted_ords_to_term_cb<F: FnMut(&[u8]) -> io::Result<()>>(\n        &self,\n        mut ords: impl Iterator<Item = TermOrdinal>,\n        mut cb: F,\n    ) -> io::Result<bool> {\n        let Some(mut ord) = ords.next() else {\n            return Ok(true);\n        };\n\n        // Open the block for the first ordinal.\n        let mut bytes = Vec::new();\n        let mut current_block_addr = self.sstable_index.get_block_with_ord(ord);\n        let mut current_sstable_delta_reader =\n            self.sstable_delta_reader_block(current_block_addr.clone())?;\n        let mut current_block_ordinal = current_block_addr.first_ordinal;\n\n        loop {\n            // move to the ord inside the current block\n            while current_block_ordinal <= ord {\n                if !current_sstable_delta_reader.advance()? {\n                    return Ok(false);\n                }\n                bytes.truncate(current_sstable_delta_reader.common_prefix_len());\n                bytes.extend_from_slice(current_sstable_delta_reader.suffix());\n                current_block_ordinal += 1;\n            }\n            cb(&bytes)?;\n\n            // fetch the next ordinal\n            let Some(next_ord) = ords.next() else {\n                return Ok(true);\n            };\n\n            // advance forward if the new ord is different than the one we just processed\n            //\n            // this allows the input TermOrdinal iterator to contain duplicates, so long as it's\n            // still sorted\n            if next_ord < ord {\n                panic!(\"Ordinals were not sorted: received {next_ord} after {ord}\");\n            } else if next_ord > ord {\n                // check if block changed for new term_ord\n                let new_block_addr = self.sstable_index.get_block_with_ord(next_ord);\n                if new_block_addr != current_block_addr {\n                    current_block_addr = new_block_addr;\n                    current_block_ordinal = current_block_addr.first_ordinal;\n                    current_sstable_delta_reader =\n                        self.sstable_delta_reader_block(current_block_addr.clone())?;\n                    bytes.clear();\n                }\n                ord = next_ord;\n            } else {\n                // The next ord is equal to the previous ord: no need to seek or advance.\n            }\n        }\n    }\n\n    /// Returns the number of terms in the dictionary.\n    pub fn term_info_from_ord(&self, term_ord: TermOrdinal) -> io::Result<Option<TSSTable::Value>> {\n        // find block in which the term would be\n        let block_addr = self.sstable_index.get_block_with_ord(term_ord);\n        let first_ordinal = block_addr.first_ordinal;\n\n        // then search inside that block only\n        let mut sstable_reader = self.sstable_reader_block(block_addr)?;\n        for _ in first_ordinal..=term_ord {\n            if !sstable_reader.advance()? {\n                return Ok(None);\n            }\n        }\n        Ok(Some(sstable_reader.value().clone()))\n    }\n\n    /// Lookups the value corresponding to the key.\n    pub fn get<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TSSTable::Value>> {\n        if let Some(block_addr) = self.sstable_index.get_block_with_key(key.as_ref()) {\n            let sstable_reader = self.sstable_delta_reader_block(block_addr)?;\n            return self.do_get(key, sstable_reader);\n        }\n        Ok(None)\n    }\n\n    /// Lookups the value corresponding to the key.\n    pub async fn get_async<K: AsRef<[u8]>>(&self, key: K) -> io::Result<Option<TSSTable::Value>> {\n        if let Some(block_addr) = self.sstable_index.get_block_with_key(key.as_ref()) {\n            let sstable_reader = self.sstable_delta_reader_block_async(block_addr).await?;\n            return self.do_get(key, sstable_reader);\n        }\n        Ok(None)\n    }\n\n    fn do_get<K: AsRef<[u8]>>(\n        &self,\n        key: K,\n        mut reader: DeltaReader<TSSTable::ValueReader>,\n    ) -> io::Result<Option<TSSTable::Value>> {\n        if let Some(_ord) = self.decode_up_to_key(key, &mut reader)? {\n            Ok(Some(reader.value().clone()))\n        } else {\n            Ok(None)\n        }\n    }\n\n    /// Returns a range builder, to stream all of the terms\n    /// within an interval.\n    pub fn range(&self) -> StreamerBuilder<'_, TSSTable> {\n        StreamerBuilder::new(self, AlwaysMatch)\n    }\n\n    /// Returns a range builder filtered with a prefix.\n    pub fn prefix_range<K: AsRef<[u8]>>(&self, prefix: K) -> StreamerBuilder<'_, TSSTable> {\n        let lower_bound = prefix.as_ref();\n        let mut upper_bound = lower_bound.to_vec();\n        for idx in (0..upper_bound.len()).rev() {\n            if upper_bound[idx] == 255 {\n                upper_bound.pop();\n            } else {\n                upper_bound[idx] += 1;\n                break;\n            }\n        }\n        let mut builder = self.range().ge(lower_bound);\n        if !upper_bound.is_empty() {\n            builder = builder.lt(upper_bound);\n        }\n        builder\n    }\n\n    /// A stream of all the sorted terms.\n    pub fn stream(&self) -> io::Result<Streamer<'_, TSSTable>> {\n        self.range().into_stream()\n    }\n\n    /// Returns a search builder, to stream all of the terms\n    /// within the Automaton\n    pub fn search<'a, A: Automaton + 'a>(\n        &'a self,\n        automaton: A,\n    ) -> StreamerBuilder<'a, TSSTable, A>\n    where\n        A::State: Clone,\n    {\n        StreamerBuilder::<TSSTable, A>::new(self, automaton)\n    }\n\n    #[doc(hidden)]\n    pub async fn warm_up_dictionary(&self) -> io::Result<()> {\n        self.sstable_slice.read_bytes_async().await?;\n        Ok(())\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::ops::{Bound, Range};\n    use std::sync::{Arc, Mutex};\n\n    use common::OwnedBytes;\n\n    use super::Dictionary;\n    use crate::MonotonicU64SSTable;\n    use crate::dictionary::TermOrdHit;\n\n    #[derive(Debug)]\n    struct PermissionedHandle {\n        bytes: OwnedBytes,\n        allowed_range: Mutex<Range<usize>>,\n    }\n\n    impl PermissionedHandle {\n        fn new(bytes: Vec<u8>) -> Self {\n            let bytes = OwnedBytes::new(bytes);\n            PermissionedHandle {\n                allowed_range: Mutex::new(0..bytes.len()),\n                bytes,\n            }\n        }\n\n        fn restrict(&self, range: Range<usize>) {\n            *self.allowed_range.lock().unwrap() = range;\n        }\n    }\n\n    impl common::HasLen for PermissionedHandle {\n        fn len(&self) -> usize {\n            self.bytes.len()\n        }\n    }\n\n    impl common::file_slice::FileHandle for PermissionedHandle {\n        fn read_bytes(&self, range: Range<usize>) -> std::io::Result<OwnedBytes> {\n            let allowed_range = self.allowed_range.lock().unwrap();\n            if !allowed_range.contains(&range.start) || !allowed_range.contains(&(range.end - 1)) {\n                return Err(std::io::Error::other(format!(\n                    \"invalid range, allowed {allowed_range:?}, requested {range:?}\"\n                )));\n            }\n\n            Ok(self.bytes.slice(range))\n        }\n    }\n\n    fn make_test_sstable() -> (Dictionary<MonotonicU64SSTable>, Arc<PermissionedHandle>) {\n        let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n\n        // this makes 256k keys, enough to fill multiple blocks.\n        for elem in 0..0x3ffff {\n            let key = format!(\"{elem:05X}\").into_bytes();\n            builder.insert(&key, &elem).unwrap();\n        }\n\n        let table = builder.finish().unwrap();\n        let table = Arc::new(PermissionedHandle::new(table));\n        let slice = common::file_slice::FileSlice::new(table.clone());\n\n        let dictionary = Dictionary::<MonotonicU64SSTable>::open(slice).unwrap();\n\n        // if the last block is id 0, tests are meaningless\n        assert_ne!(dictionary.sstable_index.locate_with_ord(u64::MAX), 0);\n        assert_eq!(dictionary.num_terms(), 0x3ffff);\n        (dictionary, table)\n    }\n\n    #[test]\n    fn test_term_to_ord_or_next() {\n        let dict = {\n            let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n\n            builder.insert(b\"bbb\", &1).unwrap();\n            builder.insert(b\"ddd\", &2).unwrap();\n\n            let table = builder.finish().unwrap();\n            let table = Arc::new(PermissionedHandle::new(table));\n            let slice = common::file_slice::FileSlice::new(table.clone());\n\n            Dictionary::<MonotonicU64SSTable>::open(slice).unwrap()\n        };\n\n        assert_eq!(dict.term_ord_or_next(b\"aaa\").unwrap(), TermOrdHit::Next(0));\n        assert_eq!(dict.term_ord_or_next(b\"bbb\").unwrap(), TermOrdHit::Exact(0));\n        assert_eq!(dict.term_ord_or_next(b\"bb\").unwrap(), TermOrdHit::Next(0));\n        assert_eq!(dict.term_ord_or_next(b\"bbbb\").unwrap(), TermOrdHit::Next(1));\n        assert_eq!(dict.term_ord_or_next(b\"dd\").unwrap(), TermOrdHit::Next(1));\n        assert_eq!(dict.term_ord_or_next(b\"ddd\").unwrap(), TermOrdHit::Exact(1));\n        assert_eq!(dict.term_ord_or_next(b\"dddd\").unwrap(), TermOrdHit::Next(2));\n\n        // This is not u64::MAX because for very small sstables (only one block),\n        // we don't store an index, and the pseudo-index always reply that the\n        // answer lies in block number 0\n        assert_eq!(\n            dict.term_ord_or_next(b\"zzzzzzz\").unwrap(),\n            TermOrdHit::Next(2)\n        );\n    }\n    #[test]\n    fn test_term_to_ord_or_next_2() {\n        let dict = {\n            let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n\n            let mut term_ord = 0;\n            builder.insert(b\"bbb\", &term_ord).unwrap();\n\n            // Fill blocks in between\n            for elem in 0..50_000 {\n                term_ord += 1;\n                let key = format!(\"ccccc{elem:05X}\").into_bytes();\n                builder.insert(&key, &term_ord).unwrap();\n            }\n\n            term_ord += 1;\n            builder.insert(b\"eee\", &term_ord).unwrap();\n\n            let table = builder.finish().unwrap();\n            let table = Arc::new(PermissionedHandle::new(table));\n            let slice = common::file_slice::FileSlice::new(table.clone());\n\n            Dictionary::<MonotonicU64SSTable>::open(slice).unwrap()\n        };\n\n        assert_eq!(dict.term_ord(b\"bbb\").unwrap(), Some(0));\n        assert_eq!(dict.term_ord_or_next(b\"bbb\").unwrap(), TermOrdHit::Exact(0));\n        assert_eq!(dict.term_ord_or_next(b\"aaa\").unwrap(), TermOrdHit::Next(0));\n        assert_eq!(dict.term_ord_or_next(b\"bb\").unwrap(), TermOrdHit::Next(0));\n        assert_eq!(dict.term_ord_or_next(b\"bbbb\").unwrap(), TermOrdHit::Next(1));\n        assert_eq!(\n            dict.term_ord_or_next(b\"ee\").unwrap(),\n            TermOrdHit::Next(50001)\n        );\n        assert_eq!(\n            dict.term_ord_or_next(b\"eee\").unwrap(),\n            TermOrdHit::Exact(50001)\n        );\n        assert_eq!(\n            dict.term_ord_or_next(b\"eeee\").unwrap(),\n            TermOrdHit::Next(u64::MAX)\n        );\n\n        assert_eq!(\n            dict.term_ord_or_next(b\"zzzzzzz\").unwrap(),\n            TermOrdHit::Next(u64::MAX)\n        );\n    }\n\n    #[test]\n    fn test_term_bounds_to_ord() {\n        let dict = {\n            let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n\n            builder.insert(b\"bbb\", &1).unwrap();\n            builder.insert(b\"ddd\", &2).unwrap();\n\n            let table = builder.finish().unwrap();\n            let table = Arc::new(PermissionedHandle::new(table));\n            let slice = common::file_slice::FileSlice::new(table.clone());\n\n            Dictionary::<MonotonicU64SSTable>::open(slice).unwrap()\n        };\n\n        // Test cases for lower_bound\n        let test_lower_bound = |bound, expected| {\n            assert_eq!(\n                dict.term_bounds_to_ord::<&[u8]>(bound, Bound::Included(b\"ignored\"))\n                    .unwrap()\n                    .0,\n                expected\n            );\n        };\n\n        test_lower_bound(Bound::Included(b\"aaa\".as_slice()), Bound::Included(0));\n        test_lower_bound(Bound::Excluded(b\"aaa\".as_slice()), Bound::Included(0));\n\n        test_lower_bound(Bound::Included(b\"bbb\".as_slice()), Bound::Included(0));\n        test_lower_bound(Bound::Excluded(b\"bbb\".as_slice()), Bound::Excluded(0));\n\n        test_lower_bound(Bound::Included(b\"ccc\".as_slice()), Bound::Included(1));\n        test_lower_bound(Bound::Excluded(b\"ccc\".as_slice()), Bound::Included(1));\n\n        test_lower_bound(Bound::Included(b\"zzz\".as_slice()), Bound::Included(2));\n        test_lower_bound(Bound::Excluded(b\"zzz\".as_slice()), Bound::Included(2));\n\n        // Test cases for upper_bound\n        let test_upper_bound = |bound, expected| {\n            assert_eq!(\n                dict.term_bounds_to_ord::<&[u8]>(Bound::Included(b\"ignored\"), bound,)\n                    .unwrap()\n                    .1,\n                expected\n            );\n        };\n        test_upper_bound(Bound::Included(b\"ccc\".as_slice()), Bound::Excluded(1));\n        test_upper_bound(Bound::Excluded(b\"ccc\".as_slice()), Bound::Excluded(1));\n        test_upper_bound(Bound::Included(b\"zzz\".as_slice()), Bound::Excluded(2));\n        test_upper_bound(Bound::Excluded(b\"zzz\".as_slice()), Bound::Excluded(2));\n        test_upper_bound(Bound::Included(b\"ddd\".as_slice()), Bound::Included(1));\n        test_upper_bound(Bound::Excluded(b\"ddd\".as_slice()), Bound::Excluded(1));\n    }\n\n    #[test]\n    fn test_ord_term_conversion() {\n        let (dic, slice) = make_test_sstable();\n\n        let block = dic.sstable_index.get_block_with_ord(100_000);\n        slice.restrict(block.byte_range);\n\n        let mut res = Vec::new();\n\n        // middle of a block\n        assert!(dic.ord_to_term(100_000, &mut res).unwrap());\n        assert_eq!(res, format!(\"{:05X}\", 100_000).into_bytes());\n        assert_eq!(dic.term_info_from_ord(100_000).unwrap().unwrap(), 100_000);\n        assert_eq!(dic.get(&res).unwrap().unwrap(), 100_000);\n        assert_eq!(dic.term_ord(&res).unwrap().unwrap(), 100_000);\n\n        // start of a block\n        assert!(dic.ord_to_term(block.first_ordinal, &mut res).unwrap());\n        assert_eq!(res, format!(\"{:05X}\", block.first_ordinal).into_bytes());\n        assert_eq!(\n            dic.term_info_from_ord(block.first_ordinal)\n                .unwrap()\n                .unwrap(),\n            block.first_ordinal\n        );\n        assert_eq!(dic.get(&res).unwrap().unwrap(), block.first_ordinal);\n        assert_eq!(dic.term_ord(&res).unwrap().unwrap(), block.first_ordinal);\n\n        // end of a block\n        let ordinal = block.first_ordinal - 1;\n        let new_range = dic.sstable_index.get_block_with_ord(ordinal).byte_range;\n        slice.restrict(new_range);\n        assert!(dic.ord_to_term(ordinal, &mut res).unwrap());\n        assert_eq!(res, format!(\"{ordinal:05X}\").into_bytes());\n        assert_eq!(dic.term_info_from_ord(ordinal).unwrap().unwrap(), ordinal);\n        assert_eq!(dic.get(&res).unwrap().unwrap(), ordinal);\n        assert_eq!(dic.term_ord(&res).unwrap().unwrap(), ordinal);\n\n        // before first block\n        // 1st block must be loaded for key-related operations\n        let block = dic.sstable_index.get_block_with_ord(0);\n        slice.restrict(block.byte_range);\n\n        assert!(dic.get(b\"$$$\").unwrap().is_none());\n        assert!(dic.term_ord(b\"$$$\").unwrap().is_none());\n\n        // after last block\n        // last block must be loaded for ord related operations\n        let ordinal = 0x40000 + 10;\n        let new_range = dic.sstable_index.get_block_with_ord(ordinal).byte_range;\n        slice.restrict(new_range);\n        assert!(!dic.ord_to_term(ordinal, &mut res).unwrap());\n        assert!(dic.term_info_from_ord(ordinal).unwrap().is_none());\n\n        // last block isn't required to be loaded for key related operations\n        slice.restrict(0..0);\n        assert!(dic.get(b\"~~~\").unwrap().is_none());\n        assert!(dic.term_ord(b\"~~~\").unwrap().is_none());\n\n        slice.restrict(0..slice.bytes.len());\n        // between 1000F and 10010, test case where matched prefix > prefix kept\n        assert!(dic.term_ord(b\"1000G\").unwrap().is_none());\n        // shorter than 10000, tests prefix case\n        assert!(dic.term_ord(b\"1000\").unwrap().is_none());\n    }\n\n    #[test]\n    fn test_ords_term() {\n        let (dic, _slice) = make_test_sstable();\n\n        // Single term\n        let mut terms = Vec::new();\n        assert!(\n            dic.sorted_ords_to_term_cb(100_000..100_001, |term| {\n                terms.push(term.to_vec());\n                Ok(())\n            })\n            .unwrap()\n        );\n        assert_eq!(terms, vec![format!(\"{:05X}\", 100_000).into_bytes(),]);\n        // Single term\n        let mut terms = Vec::new();\n        assert!(\n            dic.sorted_ords_to_term_cb(100_001..100_002, |term| {\n                terms.push(term.to_vec());\n                Ok(())\n            })\n            .unwrap()\n        );\n        assert_eq!(terms, vec![format!(\"{:05X}\", 100_001).into_bytes(),]);\n        // both terms\n        let mut terms = Vec::new();\n        assert!(\n            dic.sorted_ords_to_term_cb(100_000..100_002, |term| {\n                terms.push(term.to_vec());\n                Ok(())\n            })\n            .unwrap()\n        );\n        assert_eq!(\n            terms,\n            vec![\n                format!(\"{:05X}\", 100_000).into_bytes(),\n                format!(\"{:05X}\", 100_001).into_bytes(),\n            ]\n        );\n        // Test cross block\n        let mut terms = Vec::new();\n        assert!(\n            dic.sorted_ords_to_term_cb(98653..=98655, |term| {\n                terms.push(term.to_vec());\n                Ok(())\n            })\n            .unwrap()\n        );\n        assert_eq!(\n            terms,\n            vec![\n                format!(\"{:05X}\", 98653).into_bytes(),\n                format!(\"{:05X}\", 98654).into_bytes(),\n                format!(\"{:05X}\", 98655).into_bytes(),\n            ]\n        );\n    }\n\n    #[test]\n    fn test_range() {\n        let (dic, slice) = make_test_sstable();\n\n        let start = dic\n            .sstable_index\n            .get_block_with_key(b\"10000\")\n            .unwrap()\n            .byte_range;\n        let end = dic\n            .sstable_index\n            .get_block_with_key(b\"18000\")\n            .unwrap()\n            .byte_range;\n        slice.restrict(start.start..end.end);\n\n        let mut stream = dic.range().ge(b\"10000\").lt(b\"18000\").into_stream().unwrap();\n\n        for i in 0x10000..0x18000 {\n            assert!(stream.advance());\n            assert_eq!(stream.term_ord(), i);\n            assert_eq!(stream.value(), &i);\n            assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n        }\n        assert!(!stream.advance());\n\n        // verify limiting the number of results reduce the size read\n        slice.restrict(start.start..(end.end - 1));\n\n        let mut stream = dic\n            .range()\n            .ge(b\"10000\")\n            .lt(b\"18000\")\n            .limit(0xfff)\n            .into_stream()\n            .unwrap();\n\n        for i in 0x10000..0x10fff {\n            assert!(stream.advance());\n            assert_eq!(stream.term_ord(), i);\n            assert_eq!(stream.value(), &i);\n            assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n        }\n        // there might be more successful elements after, though how many is undefined\n\n        slice.restrict(0..slice.bytes.len());\n\n        let mut stream = dic.stream().unwrap();\n        for i in 0..0x3ffff {\n            assert!(stream.advance());\n            assert_eq!(stream.term_ord(), i);\n            assert_eq!(stream.value(), &i);\n            assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n        }\n        assert!(!stream.advance());\n    }\n\n    #[test]\n    fn test_prefix() {\n        let (dic, _slice) = make_test_sstable();\n        {\n            let mut stream = dic.prefix_range(\"1\").into_stream().unwrap();\n            for i in 0x10000..0x20000 {\n                assert!(stream.advance());\n                assert_eq!(stream.term_ord(), i);\n                assert_eq!(stream.value(), &i);\n                assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n            }\n            assert!(!stream.advance());\n        }\n        {\n            let mut stream = dic.prefix_range(\"\").into_stream().unwrap();\n            for i in 0..0x3ffff {\n                assert!(stream.advance(), \"failed at {i:05X}\");\n                assert_eq!(stream.term_ord(), i);\n                assert_eq!(stream.value(), &i);\n                assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n            }\n            assert!(!stream.advance());\n        }\n        {\n            let mut stream = dic.prefix_range(\"0FF\").into_stream().unwrap();\n            for i in 0x0ff00..=0x0ffff {\n                assert!(stream.advance(), \"failed at {i:05X}\");\n                assert_eq!(stream.term_ord(), i);\n                assert_eq!(stream.value(), &i);\n                assert_eq!(stream.key(), format!(\"{i:05X}\").into_bytes());\n            }\n            assert!(!stream.advance());\n        }\n    }\n\n    #[test]\n    fn test_prefix_edge() {\n        let dict = {\n            let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n            builder.insert(&[0, 254], &0).unwrap();\n            builder.insert(&[0, 255], &1).unwrap();\n            builder.insert(&[0, 255, 12], &2).unwrap();\n            builder.insert(&[1], &2).unwrap();\n            builder.insert(&[1, 0], &2).unwrap();\n            let table = builder.finish().unwrap();\n            let table = Arc::new(PermissionedHandle::new(table));\n            let slice = common::file_slice::FileSlice::new(table.clone());\n            Dictionary::<MonotonicU64SSTable>::open(slice).unwrap()\n        };\n\n        let mut stream = dict.prefix_range(&[0, 255]).into_stream().unwrap();\n        assert!(stream.advance());\n        assert_eq!(stream.key(), &[0, 255]);\n        assert!(stream.advance());\n        assert_eq!(stream.key(), &[0, 255, 12]);\n        assert!(!stream.advance());\n    }\n}\n"
  },
  {
    "path": "sstable/src/lib.rs",
    "content": "//! `tantivy_sstable` is a crate that provides a sorted string table data structure.\n//!\n//! It is used in `tantivy` to store the term dictionary.\n//!\n//! A `sstable` is a map of sorted `&[u8]` keys to values.\n//! The keys are encoded using incremental encoding.\n//!\n//! Values and keys are compressed using zstd with the default feature flag `zstd-compression`.\n//!\n//! # Example\n//!\n//! Here is an example of how to create and search an `sstable`:\n//!\n//! ```rust\n//! use common::OwnedBytes;\n//! use tantivy_sstable::{Dictionary, MonotonicU64SSTable};\n//!\n//! // Create a new sstable in memory.\n//! let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n//! builder.insert(b\"apple\", &1).unwrap();\n//! builder.insert(b\"banana\", &2).unwrap();\n//! builder.insert(b\"orange\", &3).unwrap();\n//! let sstable_bytes = builder.finish().unwrap();\n//!\n//! // Open the sstable.\n//! let sstable =\n//!     Dictionary::<MonotonicU64SSTable>::from_bytes(OwnedBytes::new(sstable_bytes)).unwrap();\n//!\n//! // Search for a key.\n//! let value = sstable.get(b\"banana\").unwrap();\n//! assert_eq!(value, Some(2));\n//!\n//! // Search for a non-existent key.\n//! let value = sstable.get(b\"grape\").unwrap();\n//! assert_eq!(value, None);\n//! ```\n\nuse std::io::{self, Write};\nuse std::ops::Range;\n\nuse merge::ValueMerger;\n\nmod block_match_automaton;\nmod delta;\nmod dictionary;\npub mod merge;\nmod streamer;\npub mod value;\n\nmod sstable_index_v3;\npub use sstable_index_v3::{BlockAddr, SSTableIndex, SSTableIndexBuilder, SSTableIndexV3};\nmod sstable_index_v2;\npub(crate) mod vint;\npub use dictionary::{Dictionary, TermOrdHit};\npub use streamer::{Streamer, StreamerBuilder};\n\nmod block_reader;\nuse common::{BinarySerializable, OwnedBytes};\nuse value::{VecU32ValueReader, VecU32ValueWriter};\n\npub use self::block_reader::BlockReader;\npub use self::delta::{DeltaReader, DeltaWriter};\npub use self::merge::VoidMerge;\nuse self::value::{U64MonotonicValueReader, U64MonotonicValueWriter, ValueReader, ValueWriter};\nuse crate::value::{RangeValueReader, RangeValueWriter};\n\npub type TermOrdinal = u64;\n\nconst DEFAULT_KEY_CAPACITY: usize = 50;\nconst SSTABLE_VERSION: u32 = 3;\n\n/// Given two byte string returns the length of\n/// the longest common prefix.\nfn common_prefix_len(left: &[u8], right: &[u8]) -> usize {\n    left.iter()\n        .cloned()\n        .zip(right.iter().cloned())\n        .take_while(|(left, right)| left == right)\n        .count()\n}\n\n#[derive(Debug, Copy, Clone)]\npub struct SSTableDataCorruption;\n\n/// SSTable makes it possible to read and write\n/// sstables with typed values.\npub trait SSTable: Sized {\n    type Value: Clone;\n    type ValueReader: ValueReader<Value = Self::Value>;\n    type ValueWriter: ValueWriter<Value = Self::Value>;\n\n    fn delta_writer<W: io::Write>(write: W) -> DeltaWriter<W, Self::ValueWriter> {\n        DeltaWriter::new(write)\n    }\n\n    fn writer<W: io::Write>(wrt: W) -> Writer<W, Self::ValueWriter> {\n        Writer::new(wrt)\n    }\n\n    fn delta_reader(reader: OwnedBytes) -> DeltaReader<Self::ValueReader> {\n        DeltaReader::new(reader)\n    }\n\n    fn reader(reader: OwnedBytes) -> Reader<Self::ValueReader> {\n        Reader {\n            key: Vec::with_capacity(DEFAULT_KEY_CAPACITY),\n            delta_reader: Self::delta_reader(reader),\n        }\n    }\n\n    /// Returns an empty static reader.\n    fn create_empty_reader() -> Reader<Self::ValueReader> {\n        Self::reader(OwnedBytes::empty())\n    }\n\n    fn merge<W: io::Write, M: ValueMerger<Self::Value>>(\n        io_readers: Vec<OwnedBytes>,\n        w: W,\n        merger: M,\n    ) -> io::Result<()> {\n        let readers: Vec<_> = io_readers.into_iter().map(Self::reader).collect();\n        let writer = Self::writer(w);\n        merge::merge_sstable::<Self, _, _>(readers, writer, merger)\n    }\n}\n\npub struct VoidSSTable;\n\nimpl SSTable for VoidSSTable {\n    type Value = ();\n    type ValueReader = value::VoidValueReader;\n    type ValueWriter = value::VoidValueWriter;\n}\n\n/// SSTable associated keys to u64\n/// sorted in order.\n///\n/// In other words, two keys `k1` and `k2`\n/// such that `k1` <= `k2`, are required to observe\n/// `range_sstable[k1] <= range_sstable[k2]`.\npub struct MonotonicU64SSTable;\n\nimpl SSTable for MonotonicU64SSTable {\n    type Value = u64;\n\n    type ValueReader = U64MonotonicValueReader;\n\n    type ValueWriter = U64MonotonicValueWriter;\n}\n\n/// SSTable associating keys to ranges.\n/// The range are required to partition the\n/// space.\n///\n/// In other words, two consecutive keys `k1` and `k2`\n/// are required to observe\n/// `range_sstable[k1].end == range_sstable[k2].start`.\n///\n/// The first range is not required to start at `0`.\n#[derive(Clone, Copy, Debug)]\npub struct RangeSSTable;\n\nimpl SSTable for RangeSSTable {\n    type Value = Range<u64>;\n\n    type ValueReader = RangeValueReader;\n\n    type ValueWriter = RangeValueWriter;\n}\n\n/// SSTable associating keys to Vec<u32>.\npub struct VecU32ValueSSTable;\n\nimpl SSTable for VecU32ValueSSTable {\n    type Value = Vec<u32>;\n    type ValueReader = VecU32ValueReader;\n    type ValueWriter = VecU32ValueWriter;\n}\n\n/// SSTable reader.\npub struct Reader<TValueReader> {\n    key: Vec<u8>,\n    delta_reader: DeltaReader<TValueReader>,\n}\n\nimpl<TValueReader> Reader<TValueReader>\nwhere TValueReader: ValueReader\n{\n    pub fn advance(&mut self) -> io::Result<bool> {\n        if !self.delta_reader.advance()? {\n            return Ok(false);\n        }\n        let common_prefix_len = self.delta_reader.common_prefix_len();\n        let suffix = self.delta_reader.suffix();\n        let new_len = self.delta_reader.common_prefix_len() + suffix.len();\n        self.key.resize(new_len, 0u8);\n        self.key[common_prefix_len..].copy_from_slice(suffix);\n        Ok(true)\n    }\n\n    #[inline(always)]\n    pub fn key(&self) -> &[u8] {\n        &self.key\n    }\n\n    #[inline(always)]\n    pub fn value(&self) -> &TValueReader::Value {\n        self.delta_reader.value()\n    }\n}\n\nimpl<TValueReader> AsRef<[u8]> for Reader<TValueReader> {\n    #[inline(always)]\n    fn as_ref(&self) -> &[u8] {\n        &self.key\n    }\n}\n\npub struct Writer<W, TValueWriter>\nwhere W: io::Write\n{\n    previous_key: Vec<u8>,\n    index_builder: SSTableIndexBuilder,\n    delta_writer: DeltaWriter<W, TValueWriter>,\n    num_terms: u64,\n    first_ordinal_of_the_block: u64,\n}\n\nimpl<W, TValueWriter> Writer<W, TValueWriter>\nwhere\n    W: io::Write,\n    TValueWriter: value::ValueWriter,\n{\n    /// Use `Self::new`. This method only exists to match its\n    /// equivalent in fst.\n    /// TODO remove this function. (See Issue #1727)\n    #[doc(hidden)]\n    pub fn create(wrt: W) -> io::Result<Self> {\n        Ok(Self::new(wrt))\n    }\n\n    /// Creates a new `TermDictionaryBuilder`.\n    pub fn new(wrt: W) -> Self {\n        Writer {\n            previous_key: Vec::with_capacity(DEFAULT_KEY_CAPACITY),\n            num_terms: 0u64,\n            index_builder: SSTableIndexBuilder::default(),\n            delta_writer: DeltaWriter::new(wrt),\n            first_ordinal_of_the_block: 0u64,\n        }\n    }\n\n    /// Set the target block length.\n    ///\n    /// The delta part of a block will generally be slightly larger than the requested `block_len`,\n    /// however this does not account for the length of the Value part of the table.\n    pub fn set_block_len(&mut self, block_len: usize) {\n        self.delta_writer.set_block_len(block_len)\n    }\n\n    /// Returns the last inserted key.\n    /// If no key has been inserted yet, or the block was just\n    /// flushed, this function returns \"\".\n    #[inline(always)]\n    pub(crate) fn last_inserted_key(&self) -> &[u8] {\n        &self.previous_key[..]\n    }\n\n    /// Inserts a `(key, value)` pair in the term dictionary.\n    /// Keys have to be inserted in order.\n    ///\n    /// # Panics\n    ///\n    /// Will panics if keys are inserted in an invalid order.\n    #[inline]\n    pub fn insert<K: AsRef<[u8]>>(\n        &mut self,\n        key: K,\n        value: &TValueWriter::Value,\n    ) -> io::Result<()> {\n        self.insert_key(key.as_ref())?;\n        self.insert_value(value)?;\n        Ok(())\n    }\n\n    /// # Warning\n    ///\n    /// Horribly dangerous internal API. See `.insert(...)`.\n    #[doc(hidden)]\n    #[inline]\n    pub fn insert_key(&mut self, key: &[u8]) -> io::Result<()> {\n        // If this is the first key in the block, we use it to\n        // shorten the last term in the last block.\n        if self.first_ordinal_of_the_block == self.num_terms {\n            self.index_builder\n                .shorten_last_block_key_given_next_key(key);\n        }\n        let keep_len = common_prefix_len(&self.previous_key, key);\n        let add_len = key.len() - keep_len;\n        let increasing_keys = add_len > 0 && (self.previous_key.len() == keep_len)\n            || self.previous_key.is_empty()\n            || self.previous_key[keep_len] < key[keep_len];\n        assert!(\n            increasing_keys,\n            \"Keys should be increasing. ({:?} > {:?})\",\n            String::from_utf8_lossy(&self.previous_key),\n            String::from_utf8_lossy(key),\n        );\n        self.previous_key.resize(key.len(), 0u8);\n        self.previous_key[keep_len..].copy_from_slice(&key[keep_len..]);\n        self.delta_writer.write_suffix(keep_len, &key[keep_len..]);\n        Ok(())\n    }\n\n    /// # Warning\n    ///\n    /// Horribly dangerous internal API. See `.insert(...)`.\n    #[doc(hidden)]\n    #[inline]\n    pub fn insert_value(&mut self, value: &TValueWriter::Value) -> io::Result<()> {\n        self.delta_writer.write_value(value);\n        self.num_terms += 1u64;\n        self.flush_block_if_required()\n    }\n\n    pub fn flush_block_if_required(&mut self) -> io::Result<()> {\n        if let Some(byte_range) = self.delta_writer.flush_block_if_required()? {\n            self.index_builder.add_block(\n                &self.previous_key[..],\n                byte_range,\n                self.first_ordinal_of_the_block,\n            );\n            self.first_ordinal_of_the_block = self.num_terms;\n            self.previous_key.clear();\n        }\n        Ok(())\n    }\n\n    pub fn finish(mut self) -> io::Result<W> {\n        if let Some(byte_range) = self.delta_writer.flush_block()? {\n            self.index_builder.add_block(\n                &self.previous_key[..],\n                byte_range,\n                self.first_ordinal_of_the_block,\n            );\n            self.first_ordinal_of_the_block = self.num_terms;\n        }\n        let mut wrt = self.delta_writer.finish();\n        // add a final empty block as an end marker\n        wrt.write_all(&0u32.to_le_bytes())?;\n\n        let offset = wrt.written_bytes();\n\n        let fst_len: u64 = self.index_builder.serialize(&mut wrt)?;\n        wrt.write_all(&fst_len.to_le_bytes())?;\n        wrt.write_all(&offset.to_le_bytes())?;\n        wrt.write_all(&self.num_terms.to_le_bytes())?;\n\n        SSTABLE_VERSION.serialize(&mut wrt)?;\n\n        let wrt = wrt.finish();\n        Ok(wrt.into_inner()?)\n    }\n}\n\n#[cfg(test)]\nmod test {\n    use std::io;\n    use std::ops::Bound;\n\n    use common::OwnedBytes;\n\n    use super::{MonotonicU64SSTable, SSTable, VoidMerge, VoidSSTable, common_prefix_len};\n\n    fn aux_test_common_prefix_len(left: &str, right: &str, expect_len: usize) {\n        assert_eq!(\n            common_prefix_len(left.as_bytes(), right.as_bytes()),\n            expect_len\n        );\n        assert_eq!(\n            common_prefix_len(right.as_bytes(), left.as_bytes()),\n            expect_len\n        );\n    }\n\n    #[test]\n    fn test_common_prefix_len() {\n        aux_test_common_prefix_len(\"a\", \"ab\", 1);\n        aux_test_common_prefix_len(\"\", \"ab\", 0);\n        aux_test_common_prefix_len(\"ab\", \"abc\", 2);\n        aux_test_common_prefix_len(\"abde\", \"abce\", 2);\n    }\n\n    #[test]\n    fn test_long_key_diff() {\n        let long_key = (0..1_024).map(|x| (x % 255) as u8).collect::<Vec<_>>();\n        let long_key2 = (1..300).map(|x| (x % 255) as u8).collect::<Vec<_>>();\n        let mut buffer = vec![];\n        {\n            let mut sstable_writer = VoidSSTable::writer(&mut buffer);\n            assert!(sstable_writer.insert(&long_key[..], &()).is_ok());\n            assert!(sstable_writer.insert([0, 3, 4], &()).is_ok());\n            assert!(sstable_writer.insert(&long_key2[..], &()).is_ok());\n            assert!(sstable_writer.finish().is_ok());\n        }\n        let buffer = OwnedBytes::new(buffer);\n        let mut sstable_reader = VoidSSTable::reader(buffer);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &long_key[..]);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &[0, 3, 4]);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &long_key2[..]);\n        assert!(!sstable_reader.advance().unwrap());\n    }\n\n    #[test]\n    fn test_simple_sstable() {\n        let mut buffer = vec![];\n        {\n            let mut sstable_writer = VoidSSTable::writer(&mut buffer);\n            assert!(sstable_writer.insert([17u8], &()).is_ok());\n            assert!(sstable_writer.insert([17u8, 18u8, 19u8], &()).is_ok());\n            assert!(sstable_writer.insert([17u8, 20u8], &()).is_ok());\n            assert!(sstable_writer.finish().is_ok());\n        }\n        assert_eq!(\n            &buffer,\n            &[\n                // block\n                8, 0, 0, 0, // size of block\n                0, // compression\n                16, 17, 33, 18, 19, 17, 20, // data block\n                0, 0, 0, 0, // no more block\n                // index\n                0, 0, 0, 0, 0, 0, 0, 0, // fst length\n                16, 0, 0, 0, 0, 0, 0, 0, // index start offset\n                3, 0, 0, 0, 0, 0, 0, 0, // num term\n                3, 0, 0, 0, // version\n            ]\n        );\n        let buffer = OwnedBytes::new(buffer);\n        let mut sstable_reader = VoidSSTable::reader(buffer);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &[17u8]);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &[17u8, 18u8, 19u8]);\n        assert!(sstable_reader.advance().unwrap());\n        assert_eq!(sstable_reader.key(), &[17u8, 20u8]);\n        assert!(!sstable_reader.advance().unwrap());\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_simple_sstable_non_increasing_key() {\n        let mut buffer = vec![];\n        let mut sstable_writer = VoidSSTable::writer(&mut buffer);\n        assert!(sstable_writer.insert([17u8], &()).is_ok());\n        assert!(sstable_writer.insert([16u8], &()).is_ok());\n    }\n\n    #[test]\n    fn test_merge_abcd_abe() {\n        let mut buffer = Vec::new();\n        {\n            let mut writer = VoidSSTable::writer(&mut buffer);\n            writer.insert(b\"abcd\", &()).unwrap();\n            writer.insert(b\"abe\", &()).unwrap();\n            writer.finish().unwrap();\n        }\n        let buffer = OwnedBytes::new(buffer);\n        let mut output = Vec::new();\n        assert!(\n            VoidSSTable::merge(vec![buffer.clone(), buffer.clone()], &mut output, VoidMerge)\n                .is_ok()\n        );\n        assert_eq!(&output[..], &buffer[..]);\n    }\n\n    #[test]\n    fn test_sstable() {\n        let mut buffer = Vec::new();\n        {\n            let mut writer = VoidSSTable::writer(&mut buffer);\n            assert_eq!(writer.last_inserted_key(), b\"\");\n            writer.insert(b\"abcd\", &()).unwrap();\n            assert_eq!(writer.last_inserted_key(), b\"abcd\");\n            writer.insert(b\"abe\", &()).unwrap();\n            assert_eq!(writer.last_inserted_key(), b\"abe\");\n            writer.finish().unwrap();\n        }\n        let buffer = OwnedBytes::new(buffer);\n        let mut output = Vec::new();\n        assert!(\n            VoidSSTable::merge(vec![buffer.clone(), buffer.clone()], &mut output, VoidMerge)\n                .is_ok()\n        );\n        assert_eq!(&output[..], &buffer[..]);\n    }\n\n    #[test]\n    fn test_sstable_u64() -> io::Result<()> {\n        let mut buffer = Vec::new();\n        let mut writer = MonotonicU64SSTable::writer(&mut buffer);\n        writer.insert(b\"abcd\", &1u64)?;\n        writer.insert(b\"abe\", &4u64)?;\n        writer.insert(b\"gogo\", &4324234234234234u64)?;\n        writer.finish()?;\n        let buffer = OwnedBytes::new(buffer);\n        let mut reader = MonotonicU64SSTable::reader(buffer);\n        assert!(reader.advance()?);\n        assert_eq!(reader.key(), b\"abcd\");\n        assert_eq!(reader.value(), &1u64);\n        assert!(reader.advance()?);\n        assert_eq!(reader.key(), b\"abe\");\n        assert_eq!(reader.value(), &4u64);\n        assert!(reader.advance()?);\n        assert_eq!(reader.key(), b\"gogo\");\n        assert_eq!(reader.value(), &4324234234234234u64);\n        assert!(!reader.advance()?);\n        Ok(())\n    }\n\n    #[test]\n    fn test_sstable_empty() {\n        let mut sstable_range_empty = crate::RangeSSTable::create_empty_reader();\n        assert!(!sstable_range_empty.advance().unwrap());\n    }\n\n    use common::file_slice::FileSlice;\n    use proptest::prelude::*;\n\n    use crate::Dictionary;\n\n    fn bound_strategy() -> impl Strategy<Value = Bound<String>> {\n        prop_oneof![\n            Just(Bound::<String>::Unbounded),\n            \"[a-c]{0,5}\".prop_map(Bound::Included),\n            \"[a-c]{0,5}\".prop_map(Bound::Excluded),\n        ]\n    }\n\n    fn extract_key(bound: Bound<&String>) -> Option<&str> {\n        match bound.as_ref() {\n            Bound::Included(key) => Some(key.as_str()),\n            Bound::Excluded(key) => Some(key.as_str()),\n            Bound::Unbounded => None,\n        }\n    }\n\n    fn bounds_strategy() -> impl Strategy<Value = (Bound<String>, Bound<String>)> {\n        (bound_strategy(), bound_strategy()).prop_filter(\n            \"Lower bound <= Upper bound\",\n            |(left, right)| match (extract_key(left.as_ref()), extract_key(right.as_ref())) {\n                (None, _) => true,\n                (_, None) => true,\n                (left, right) => left < right,\n            },\n        )\n    }\n\n    proptest! {\n        #[test]\n        fn test_proptest_sstable_ranges(words in prop::collection::btree_set(\"[a-c]{0,6}\", 1..100),\n            (lower_bound, upper_bound) in bounds_strategy(),\n        ) {\n            let mut builder = Dictionary::<VoidSSTable>::builder(Vec::new()).unwrap();\n            builder.set_block_len(16);\n            for word in &words {\n                builder.insert(word.as_bytes(), &()).unwrap();\n            }\n            let buffer: Vec<u8> = builder.finish().unwrap();\n            let dictionary: Dictionary<VoidSSTable> = Dictionary::open(FileSlice::from(buffer)).unwrap();\n            let mut range_builder = dictionary.range();\n            range_builder = match lower_bound.as_ref() {\n                Bound::Included(key) => range_builder.ge(key.as_bytes()),\n                Bound::Excluded(key) => range_builder.gt(key.as_bytes()),\n                Bound::Unbounded => range_builder,\n            };\n            range_builder = match upper_bound.as_ref() {\n                Bound::Included(key) => range_builder.le(key.as_bytes()),\n                Bound::Excluded(key) => range_builder.lt(key.as_bytes()),\n                Bound::Unbounded => range_builder,\n            };\n            let mut stream = range_builder.into_stream().unwrap();\n            let mut btree_set_range = words.range((lower_bound, upper_bound));\n            while stream.advance() {\n                let val = btree_set_range.next().unwrap();\n                assert_eq!(val.as_bytes(), stream.key());\n            }\n            assert!(btree_set_range.next().is_none());\n        }\n    }\n}\n"
  },
  {
    "path": "sstable/src/merge/heap_merge.rs",
    "content": "use std::cmp::Ordering;\nuse std::collections::BinaryHeap;\nuse std::collections::binary_heap::PeekMut;\nuse std::io;\n\nuse super::{SingleValueMerger, ValueMerger};\nuse crate::{Reader, SSTable, Writer};\n\nstruct HeapItem<B: AsRef<[u8]>>(B);\n\nimpl<B: AsRef<[u8]>> Ord for HeapItem<B> {\n    fn cmp(&self, other: &Self) -> Ordering {\n        other.0.as_ref().cmp(self.0.as_ref())\n    }\n}\nimpl<B: AsRef<[u8]>> PartialOrd for HeapItem<B> {\n    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {\n        Some(self.cmp(other))\n    }\n}\n\nimpl<B: AsRef<[u8]>> Eq for HeapItem<B> {}\nimpl<B: AsRef<[u8]>> PartialEq for HeapItem<B> {\n    fn eq(&self, other: &Self) -> bool {\n        self.0.as_ref() == other.0.as_ref()\n    }\n}\n\npub fn merge_sstable<SST: SSTable, W: io::Write, M: ValueMerger<SST::Value>>(\n    readers: Vec<Reader<SST::ValueReader>>,\n    mut writer: Writer<W, SST::ValueWriter>,\n    mut merger: M,\n) -> io::Result<()> {\n    let mut heap: BinaryHeap<HeapItem<Reader<SST::ValueReader>>> =\n        BinaryHeap::with_capacity(readers.len());\n    for mut reader in readers {\n        if reader.advance()? {\n            heap.push(HeapItem(reader));\n        }\n    }\n    loop {\n        let len = heap.len();\n        let mut value_merger;\n        match heap.peek_mut() {\n            Some(mut head) => {\n                writer.insert_key(head.0.key()).unwrap();\n                value_merger = merger.new_value(head.0.value());\n                if !head.0.advance()? {\n                    PeekMut::pop(head);\n                }\n            }\n            _ => {\n                break;\n            }\n        }\n        for _ in 0..len - 1 {\n            if let Some(mut head) = heap.peek_mut()\n                && head.0.key() == writer.last_inserted_key()\n            {\n                value_merger.add(head.0.value());\n                if !head.0.advance()? {\n                    PeekMut::pop(head);\n                }\n                continue;\n            }\n            break;\n        }\n        let value = value_merger.finish();\n        writer.insert_value(&value)?;\n        writer.flush_block_if_required()?;\n    }\n    writer.finish()?;\n    Ok(())\n}\n"
  },
  {
    "path": "sstable/src/merge/mod.rs",
    "content": "mod heap_merge;\n\npub use self::heap_merge::merge_sstable;\n\npub trait SingleValueMerger<V> {\n    fn add(&mut self, v: &V);\n    fn finish(self) -> V;\n}\n\npub trait ValueMerger<V> {\n    type TSingleValueMerger: SingleValueMerger<V>;\n    fn new_value(&mut self, v: &V) -> Self::TSingleValueMerger;\n}\n\n#[derive(Default)]\npub struct KeepFirst;\n\npub struct FirstVal<V>(V);\n\nimpl<V: Clone> ValueMerger<V> for KeepFirst {\n    type TSingleValueMerger = FirstVal<V>;\n\n    fn new_value(&mut self, v: &V) -> FirstVal<V> {\n        FirstVal(v.clone())\n    }\n}\n\nimpl<V> SingleValueMerger<V> for FirstVal<V> {\n    fn add(&mut self, _: &V) {}\n\n    fn finish(self) -> V {\n        self.0\n    }\n}\n\npub struct VoidMerge;\nimpl ValueMerger<()> for VoidMerge {\n    type TSingleValueMerger = ();\n\n    fn new_value(&mut self, _: &()) {}\n}\n\npub struct U64Merge;\nimpl ValueMerger<u64> for U64Merge {\n    type TSingleValueMerger = u64;\n\n    fn new_value(&mut self, val: &u64) -> u64 {\n        *val\n    }\n}\n\nimpl SingleValueMerger<u64> for u64 {\n    fn add(&mut self, val: &u64) {\n        *self += *val;\n    }\n\n    fn finish(self) -> u64 {\n        self\n    }\n}\n\nimpl SingleValueMerger<()> for () {\n    fn add(&mut self, _: &()) {}\n\n    fn finish(self) {}\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::{BTreeMap, BTreeSet};\n    use std::str;\n\n    use common::OwnedBytes;\n\n    use super::super::{MonotonicU64SSTable, SSTable, VoidSSTable};\n    use super::{U64Merge, VoidMerge};\n\n    fn write_sstable(keys: &[&'static str]) -> OwnedBytes {\n        let mut buffer: Vec<u8> = vec![];\n        {\n            let mut sstable_writer = VoidSSTable::writer(&mut buffer);\n            for &key in keys {\n                assert!(sstable_writer.insert(key.as_bytes(), &()).is_ok());\n            }\n            assert!(sstable_writer.finish().is_ok());\n        }\n        OwnedBytes::new(buffer)\n    }\n\n    fn write_sstable_u64(keys: &[(&'static str, u64)]) -> OwnedBytes {\n        let mut buffer: Vec<u8> = vec![];\n        {\n            let mut sstable_writer = MonotonicU64SSTable::writer(&mut buffer);\n            for (key, val) in keys {\n                assert!(sstable_writer.insert(key.as_bytes(), val).is_ok());\n            }\n            assert!(sstable_writer.finish().is_ok());\n        }\n        OwnedBytes::new(buffer)\n    }\n\n    fn merge_test_aux(arrs: &[&[&'static str]]) {\n        let sstables = arrs.iter().cloned().map(write_sstable).collect::<Vec<_>>();\n        let mut merged = BTreeSet::new();\n        for &arr in arrs.iter() {\n            for &s in arr {\n                merged.insert(s.to_string());\n            }\n        }\n        let mut w = Vec::new();\n        assert!(VoidSSTable::merge(sstables, &mut w, VoidMerge).is_ok());\n        let w = OwnedBytes::new(w);\n        let mut reader = VoidSSTable::reader(w);\n        for k in merged {\n            assert!(reader.advance().unwrap());\n            assert_eq!(reader.key(), k.as_bytes());\n        }\n        assert!(!reader.advance().unwrap());\n    }\n\n    fn merge_test_u64_monotonic_aux(arrs: &[&[(&'static str, u64)]]) {\n        let sstables = arrs\n            .iter()\n            .cloned()\n            .map(write_sstable_u64)\n            .collect::<Vec<_>>();\n        let mut merged = BTreeMap::new();\n        for &arr in arrs.iter() {\n            for (key, val) in arr {\n                let entry = merged.entry(key.to_string()).or_insert(0u64);\n                *entry += val;\n            }\n        }\n        let mut w = Vec::new();\n        assert!(MonotonicU64SSTable::merge(sstables, &mut w, U64Merge).is_ok());\n        let w = OwnedBytes::new(w);\n        let mut reader = MonotonicU64SSTable::reader(w);\n        for (k, v) in merged {\n            assert!(reader.advance().unwrap());\n            assert_eq!(reader.key(), k.as_bytes());\n            assert_eq!(reader.value(), &v);\n        }\n        assert!(!reader.advance().unwrap());\n    }\n\n    #[test]\n    fn test_merge_simple_reproduce() {\n        let sstable_data = write_sstable(&[\"a\"]);\n        let mut reader = VoidSSTable::reader(sstable_data);\n        assert!(reader.advance().unwrap());\n        assert_eq!(reader.key(), b\"a\");\n        assert!(!reader.advance().unwrap());\n    }\n\n    #[test]\n    fn test_merge() {\n        merge_test_aux(&[]);\n        merge_test_aux(&[&[\"a\"]]);\n        merge_test_aux(&[&[\"a\", \"b\"], &[\"ab\"]]); // a, ab, b\n        merge_test_aux(&[&[\"a\", \"b\"], &[\"a\", \"b\"]]);\n        merge_test_aux(&[\n            &[\"happy\", \"hello\", \"payer\", \"tax\"],\n            &[\"habitat\", \"hello\", \"zoo\"],\n            &[],\n            &[\"a\"],\n        ]);\n        merge_test_aux(&[&[\"a\"]]);\n        merge_test_aux(&[&[\"a\", \"b\"], &[\"ab\"]]);\n        merge_test_aux(&[&[\"a\", \"b\"], &[\"a\", \"b\"]]);\n    }\n\n    #[test]\n    fn test_merge_u64() {\n        merge_test_u64_monotonic_aux(&[]);\n        merge_test_u64_monotonic_aux(&[&[(\"a\", 1u64)]]);\n        merge_test_u64_monotonic_aux(&[&[(\"a\", 1u64), (\"b\", 3u64)], &[(\"ab\", 2u64)]]); // a, ab, b\n        merge_test_u64_monotonic_aux(&[&[(\"a\", 1u64), (\"b\", 2u64)], &[(\"a\", 16u64), (\"b\", 23u64)]]);\n    }\n}\n"
  },
  {
    "path": "sstable/src/sstable_index_v2.rs",
    "content": "use common::OwnedBytes;\nuse tantivy_fst::Automaton;\n\nuse crate::block_match_automaton::can_block_match_automaton;\nuse crate::{BlockAddr, SSTable, SSTableDataCorruption, TermOrdinal};\n\n#[derive(Default, Debug, Clone)]\npub struct SSTableIndex {\n    pub(crate) blocks: Vec<BlockMeta>,\n}\n\nimpl SSTableIndex {\n    /// Load an index from its binary representation\n    pub fn load(data: OwnedBytes) -> Result<SSTableIndex, SSTableDataCorruption> {\n        let mut reader = IndexSSTable::reader(data);\n        let mut blocks = Vec::new();\n\n        while reader.advance().map_err(|_| SSTableDataCorruption)? {\n            blocks.push(BlockMeta {\n                last_key_or_greater: reader.key().to_vec(),\n                block_addr: reader.value().clone(),\n            });\n        }\n\n        Ok(SSTableIndex { blocks })\n    }\n\n    /// Get the [`BlockAddr`] of the requested block.\n    pub(crate) fn get_block(&self, block_id: usize) -> Option<BlockAddr> {\n        self.blocks\n            .get(block_id)\n            .map(|block_meta| block_meta.block_addr.clone())\n    }\n\n    /// Get the block id of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub(crate) fn locate_with_key(&self, key: &[u8]) -> Option<usize> {\n        let pos = self\n            .blocks\n            .binary_search_by_key(&key, |block| &block.last_key_or_greater);\n        match pos {\n            Ok(pos) => Some(pos),\n            Err(pos) => {\n                if pos < self.blocks.len() {\n                    Some(pos)\n                } else {\n                    // after end of last block: no block matches\n                    None\n                }\n            }\n        }\n    }\n\n    /// Get the [`BlockAddr`] of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub fn get_block_with_key(&self, key: &[u8]) -> Option<BlockAddr> {\n        self.locate_with_key(key).and_then(|id| self.get_block(id))\n    }\n\n    pub(crate) fn locate_with_ord(&self, ord: TermOrdinal) -> usize {\n        let pos = self\n            .blocks\n            .binary_search_by_key(&ord, |block| block.block_addr.first_ordinal);\n\n        match pos {\n            Ok(pos) => pos,\n            // Err(0) can't happen as the sstable starts with ordinal zero\n            Err(pos) => pos - 1,\n        }\n    }\n\n    /// Get the [`BlockAddr`] of the block containing the `ord`-th term.\n    pub(crate) fn get_block_with_ord(&self, ord: TermOrdinal) -> BlockAddr {\n        // locate_with_ord always returns an index within range\n        self.get_block(self.locate_with_ord(ord)).unwrap()\n    }\n\n    pub(crate) fn get_block_for_automaton<'a>(\n        &'a self,\n        automaton: &'a impl Automaton,\n    ) -> impl Iterator<Item = (u64, BlockAddr)> + 'a {\n        std::iter::once((None, &self.blocks[0]))\n            .chain(self.blocks.windows(2).map(|window| {\n                let [prev, curr] = window else {\n                    unreachable!();\n                };\n                (Some(&*prev.last_key_or_greater), curr)\n            }))\n            .enumerate()\n            .filter_map(move |(pos, (prev_key, current_block))| {\n                if can_block_match_automaton(\n                    prev_key,\n                    &current_block.last_key_or_greater,\n                    automaton,\n                ) {\n                    Some((pos as u64, current_block.block_addr.clone()))\n                } else {\n                    None\n                }\n            })\n    }\n}\n\n#[derive(Debug, Clone)]\npub(crate) struct BlockMeta {\n    /// Any byte string that is lexicographically greater or equal to\n    /// the last key in the block,\n    /// and yet strictly smaller than the first key in the next block.\n    pub last_key_or_greater: Vec<u8>,\n    pub block_addr: BlockAddr,\n}\n\n/// SSTable representing an index\n///\n/// `last_key_or_greater` is used as the key, the value contains the\n/// length and first ordinal of each block. The start offset is implicitly\n/// obtained from lengths.\nstruct IndexSSTable;\n\nimpl SSTable for IndexSSTable {\n    type Value = BlockAddr;\n\n    type ValueReader = crate::value::index::IndexValueReader;\n\n    type ValueWriter = crate::value::index::IndexValueWriter;\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::block_match_automaton::tests::EqBuffer;\n\n    #[test]\n    fn test_get_block_for_automaton() {\n        let sstable = SSTableIndex {\n            blocks: vec![\n                BlockMeta {\n                    last_key_or_greater: vec![0, 1, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 0,\n                        byte_range: 0..10,\n                    },\n                },\n                BlockMeta {\n                    last_key_or_greater: vec![0, 2, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 5,\n                        byte_range: 10..20,\n                    },\n                },\n                BlockMeta {\n                    last_key_or_greater: vec![0, 3, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 10,\n                        byte_range: 20..30,\n                    },\n                },\n            ],\n        };\n\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 1, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                0,\n                BlockAddr {\n                    first_ordinal: 0,\n                    byte_range: 0..10\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 2, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                1,\n                BlockAddr {\n                    first_ordinal: 5,\n                    byte_range: 10..20\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 3, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                2,\n                BlockAddr {\n                    first_ordinal: 10,\n                    byte_range: 20..30\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 4, 1]))\n            .collect::<Vec<_>>();\n        assert!(res.is_empty());\n\n        let complex_automaton = EqBuffer(vec![0, 1, 1]).union(EqBuffer(vec![0, 3, 1]));\n        let res = sstable\n            .get_block_for_automaton(&complex_automaton)\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![\n                (\n                    0,\n                    BlockAddr {\n                        first_ordinal: 0,\n                        byte_range: 0..10\n                    }\n                ),\n                (\n                    2,\n                    BlockAddr {\n                        first_ordinal: 10,\n                        byte_range: 20..30\n                    }\n                )\n            ]\n        );\n    }\n}\n"
  },
  {
    "path": "sstable/src/sstable_index_v3.rs",
    "content": "use std::io::{self, Read, Write};\nuse std::ops::Range;\nuse std::sync::Arc;\n\nuse common::{BinarySerializable, FixedSize, OwnedBytes};\nuse tantivy_bitpacker::{BitPacker, compute_num_bits};\nuse tantivy_fst::raw::Fst;\nuse tantivy_fst::{Automaton, IntoStreamer, Map, MapBuilder, Streamer};\n\nuse crate::block_match_automaton::can_block_match_automaton;\nuse crate::{SSTableDataCorruption, TermOrdinal, common_prefix_len};\n\n#[derive(Debug, Clone)]\npub enum SSTableIndex {\n    V2(crate::sstable_index_v2::SSTableIndex),\n    V3(SSTableIndexV3),\n    V3Empty(SSTableIndexV3Empty),\n}\n\nimpl SSTableIndex {\n    /// Get the [`BlockAddr`] of the requested block.\n    pub(crate) fn get_block(&self, block_id: u64) -> Option<BlockAddr> {\n        match self {\n            SSTableIndex::V2(v2_index) => v2_index.get_block(block_id as usize),\n            SSTableIndex::V3(v3_index) => v3_index.get_block(block_id),\n            SSTableIndex::V3Empty(v3_empty) => v3_empty.get_block(block_id),\n        }\n    }\n\n    /// Get the block id of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub(crate) fn locate_with_key(&self, key: &[u8]) -> Option<u64> {\n        match self {\n            SSTableIndex::V2(v2_index) => v2_index.locate_with_key(key).map(|i| i as u64),\n            SSTableIndex::V3(v3_index) => v3_index.locate_with_key(key),\n            SSTableIndex::V3Empty(v3_empty) => v3_empty.locate_with_key(key),\n        }\n    }\n\n    /// Get the [`BlockAddr`] of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub fn get_block_with_key(&self, key: &[u8]) -> Option<BlockAddr> {\n        match self {\n            SSTableIndex::V2(v2_index) => v2_index.get_block_with_key(key),\n            SSTableIndex::V3(v3_index) => v3_index.get_block_with_key(key),\n            SSTableIndex::V3Empty(v3_empty) => v3_empty.get_block_with_key(key),\n        }\n    }\n\n    pub(crate) fn locate_with_ord(&self, ord: TermOrdinal) -> u64 {\n        match self {\n            SSTableIndex::V2(v2_index) => v2_index.locate_with_ord(ord) as u64,\n            SSTableIndex::V3(v3_index) => v3_index.locate_with_ord(ord),\n            SSTableIndex::V3Empty(v3_empty) => v3_empty.locate_with_ord(ord),\n        }\n    }\n\n    /// Get the [`BlockAddr`] of the block containing the `ord`-th term.\n    pub(crate) fn get_block_with_ord(&self, ord: TermOrdinal) -> BlockAddr {\n        match self {\n            SSTableIndex::V2(v2_index) => v2_index.get_block_with_ord(ord),\n            SSTableIndex::V3(v3_index) => v3_index.get_block_with_ord(ord),\n            SSTableIndex::V3Empty(v3_empty) => v3_empty.get_block_with_ord(ord),\n        }\n    }\n\n    pub fn get_block_for_automaton<'a>(\n        &'a self,\n        automaton: &'a impl Automaton,\n    ) -> impl Iterator<Item = (u64, BlockAddr)> + 'a {\n        match self {\n            SSTableIndex::V2(v2_index) => {\n                BlockIter::V2(v2_index.get_block_for_automaton(automaton))\n            }\n            SSTableIndex::V3(v3_index) => {\n                BlockIter::V3(v3_index.get_block_for_automaton(automaton))\n            }\n            SSTableIndex::V3Empty(v3_empty) => {\n                BlockIter::V3Empty(std::iter::once((0, v3_empty.block_addr.clone())))\n            }\n        }\n    }\n}\n\nenum BlockIter<V2, V3, T> {\n    V2(V2),\n    V3(V3),\n    V3Empty(std::iter::Once<T>),\n}\n\nimpl<V2: Iterator<Item = T>, V3: Iterator<Item = T>, T> Iterator for BlockIter<V2, V3, T> {\n    type Item = T;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        match self {\n            BlockIter::V2(v2) => v2.next(),\n            BlockIter::V3(v3) => v3.next(),\n            BlockIter::V3Empty(once) => once.next(),\n        }\n    }\n}\n\n#[derive(Debug, Clone)]\npub struct SSTableIndexV3 {\n    fst_index: Arc<Map<OwnedBytes>>,\n    block_addr_store: BlockAddrStore,\n}\n\nimpl SSTableIndexV3 {\n    /// Load an index from its binary representation\n    pub fn load(\n        data: OwnedBytes,\n        fst_length: u64,\n    ) -> Result<SSTableIndexV3, SSTableDataCorruption> {\n        let (fst_slice, block_addr_store_slice) = data.split(fst_length as usize);\n        let fst_index = Fst::new(fst_slice)\n            .map_err(|_| SSTableDataCorruption)?\n            .into();\n        let block_addr_store =\n            BlockAddrStore::open(block_addr_store_slice).map_err(|_| SSTableDataCorruption)?;\n\n        Ok(SSTableIndexV3 {\n            fst_index: Arc::new(fst_index),\n            block_addr_store,\n        })\n    }\n\n    /// Get the [`BlockAddr`] of the requested block.\n    pub(crate) fn get_block(&self, block_id: u64) -> Option<BlockAddr> {\n        self.block_addr_store.get(block_id)\n    }\n\n    /// Get the block id of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub(crate) fn locate_with_key(&self, key: &[u8]) -> Option<u64> {\n        self.fst_index\n            .range()\n            .ge(key)\n            .into_stream()\n            .next()\n            .map(|(_key, id)| id)\n    }\n\n    /// Get the [`BlockAddr`] of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub fn get_block_with_key(&self, key: &[u8]) -> Option<BlockAddr> {\n        self.locate_with_key(key).and_then(|id| self.get_block(id))\n    }\n\n    pub(crate) fn locate_with_ord(&self, ord: TermOrdinal) -> u64 {\n        self.block_addr_store.binary_search_ord(ord).0\n    }\n\n    /// Get the [`BlockAddr`] of the block containing the `ord`-th term.\n    pub(crate) fn get_block_with_ord(&self, ord: TermOrdinal) -> BlockAddr {\n        self.block_addr_store.binary_search_ord(ord).1\n    }\n\n    pub(crate) fn get_block_for_automaton<'a>(\n        &'a self,\n        automaton: &'a impl Automaton,\n    ) -> impl Iterator<Item = (u64, BlockAddr)> + 'a {\n        // this is more complicated than other index formats: we don't have a ready made list of\n        // blocks, and instead need to stream-decode the sstable.\n\n        GetBlockForAutomaton {\n            streamer: self.fst_index.stream(),\n            block_addr_store: &self.block_addr_store,\n            prev_key: None,\n            automaton,\n        }\n    }\n}\n\n// TODO we iterate over the entire Map to find matching blocks,\n// we could manually iterate on the underlying Fst and skip whole branches if our Automaton says\n// cannot match. this isn't as bad as it sounds given the fst is a lot smaller than the rest of the\n// sstable.\n// To do that, we can't use tantivy_fst's Stream with an automaton, as we need to know 2 consecutive\n// fst keys to form a proper opinion on whether this is a match, which we want translate into a\n// single automaton\nstruct GetBlockForAutomaton<'a, A: Automaton> {\n    streamer: tantivy_fst::map::Stream<'a>,\n    block_addr_store: &'a BlockAddrStore,\n    prev_key: Option<Vec<u8>>,\n    automaton: &'a A,\n}\n\nimpl<A: Automaton> Iterator for GetBlockForAutomaton<'_, A> {\n    type Item = (u64, BlockAddr);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        while let Some((new_key, block_id)) = self.streamer.next() {\n            if let Some(prev_key) = self.prev_key.as_mut() {\n                if can_block_match_automaton(Some(prev_key), new_key, self.automaton) {\n                    prev_key.clear();\n                    prev_key.extend_from_slice(new_key);\n                    return Some((block_id, self.block_addr_store.get(block_id).unwrap()));\n                }\n                prev_key.clear();\n                prev_key.extend_from_slice(new_key);\n            } else {\n                self.prev_key = Some(new_key.to_owned());\n                if can_block_match_automaton(None, new_key, self.automaton) {\n                    return Some((block_id, self.block_addr_store.get(block_id).unwrap()));\n                }\n            }\n        }\n        None\n    }\n}\n\n#[derive(Debug, Clone)]\npub struct SSTableIndexV3Empty {\n    block_addr: BlockAddr,\n}\n\nimpl SSTableIndexV3Empty {\n    pub fn load(index_start_pos: usize) -> SSTableIndexV3Empty {\n        SSTableIndexV3Empty {\n            block_addr: BlockAddr {\n                first_ordinal: 0,\n                byte_range: 0..index_start_pos,\n            },\n        }\n    }\n\n    /// Get the [`BlockAddr`] of the requested block.\n    pub(crate) fn get_block(&self, _block_id: u64) -> Option<BlockAddr> {\n        Some(self.block_addr.clone())\n    }\n\n    /// Get the block id of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub(crate) fn locate_with_key(&self, _key: &[u8]) -> Option<u64> {\n        Some(0)\n    }\n\n    /// Get the [`BlockAddr`] of the block that would contain `key`.\n    ///\n    /// Returns None if `key` is lexicographically after the last key recorded.\n    pub fn get_block_with_key(&self, _key: &[u8]) -> Option<BlockAddr> {\n        Some(self.block_addr.clone())\n    }\n\n    pub(crate) fn locate_with_ord(&self, _ord: TermOrdinal) -> u64 {\n        0\n    }\n\n    /// Get the [`BlockAddr`] of the block containing the `ord`-th term.\n    pub(crate) fn get_block_with_ord(&self, _ord: TermOrdinal) -> BlockAddr {\n        self.block_addr.clone()\n    }\n}\n#[derive(Clone, Eq, PartialEq, Debug)]\npub struct BlockAddr {\n    pub first_ordinal: u64,\n    pub byte_range: Range<usize>,\n}\n\nimpl BlockAddr {\n    fn to_block_start(&self) -> BlockStartAddr {\n        BlockStartAddr {\n            first_ordinal: self.first_ordinal,\n            byte_range_start: self.byte_range.start,\n        }\n    }\n}\n\n#[derive(Debug, Clone, PartialEq, Eq)]\nstruct BlockStartAddr {\n    first_ordinal: u64,\n    byte_range_start: usize,\n}\n\nimpl BlockStartAddr {\n    fn to_block_addr(&self, byte_range_end: usize) -> BlockAddr {\n        BlockAddr {\n            first_ordinal: self.first_ordinal,\n            byte_range: self.byte_range_start..byte_range_end,\n        }\n    }\n}\n\n#[derive(Debug, Clone)]\npub(crate) struct BlockMeta {\n    /// Any byte string that is lexicographically greater or equal to\n    /// the last key in the block,\n    /// and yet strictly smaller than the first key in the next block.\n    pub last_key_or_greater: Vec<u8>,\n    pub block_addr: BlockAddr,\n}\n\nimpl BinarySerializable for BlockStartAddr {\n    fn serialize<W: Write + ?Sized>(&self, writer: &mut W) -> io::Result<()> {\n        let start = self.byte_range_start as u64;\n        start.serialize(writer)?;\n        self.first_ordinal.serialize(writer)\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let byte_range_start = u64::deserialize(reader)? as usize;\n        let first_ordinal = u64::deserialize(reader)?;\n        Ok(BlockStartAddr {\n            first_ordinal,\n            byte_range_start,\n        })\n    }\n\n    // Provided method\n    fn num_bytes(&self) -> u64 {\n        BlockStartAddr::SIZE_IN_BYTES as u64\n    }\n}\n\nimpl FixedSize for BlockStartAddr {\n    const SIZE_IN_BYTES: usize = 2 * u64::SIZE_IN_BYTES;\n}\n\n/// Given that left < right,\n/// mutates `left into a shorter byte string left'` that\n/// matches `left <= left' < right`.\nfn find_shorter_str_in_between(left: &mut Vec<u8>, right: &[u8]) {\n    assert!(&left[..] < right);\n    let common_len = common_prefix_len(left, right);\n    if left.len() == common_len {\n        return;\n    }\n    // It is possible to do one character shorter in some case,\n    // but it is not worth the extra complexity\n    for pos in (common_len + 1)..left.len() {\n        if left[pos] != u8::MAX {\n            left[pos] += 1;\n            left.truncate(pos + 1);\n            return;\n        }\n    }\n}\n\n#[derive(Default)]\npub struct SSTableIndexBuilder {\n    blocks: Vec<BlockMeta>,\n}\n\nimpl SSTableIndexBuilder {\n    /// In order to make the index as light as possible, we\n    /// try to find a shorter alternative to the last key of the last block\n    /// that is still smaller than the next key.\n    pub(crate) fn shorten_last_block_key_given_next_key(&mut self, next_key: &[u8]) {\n        if let Some(last_block) = self.blocks.last_mut() {\n            find_shorter_str_in_between(&mut last_block.last_key_or_greater, next_key);\n        }\n    }\n\n    pub fn add_block(&mut self, last_key: &[u8], byte_range: Range<usize>, first_ordinal: u64) {\n        self.blocks.push(BlockMeta {\n            last_key_or_greater: last_key.to_vec(),\n            block_addr: BlockAddr {\n                byte_range,\n                first_ordinal,\n            },\n        })\n    }\n\n    pub fn serialize<W: std::io::Write>(&self, wrt: W) -> io::Result<u64> {\n        if self.blocks.len() <= 1 {\n            return Ok(0);\n        }\n        let counting_writer = common::CountingWriter::wrap(wrt);\n        let mut map_builder = MapBuilder::new(counting_writer).map_err(fst_error_to_io_error)?;\n        for (i, block) in self.blocks.iter().enumerate() {\n            map_builder\n                .insert(&block.last_key_or_greater, i as u64)\n                .map_err(fst_error_to_io_error)?;\n        }\n        let counting_writer = map_builder.into_inner().map_err(fst_error_to_io_error)?;\n        let written_bytes = counting_writer.written_bytes();\n        let mut wrt = counting_writer.finish();\n\n        let mut block_store_writer = BlockAddrStoreWriter::new();\n        for block in &self.blocks {\n            block_store_writer.write_block_meta(block.block_addr.clone())?;\n        }\n        block_store_writer.serialize(&mut wrt)?;\n\n        Ok(written_bytes)\n    }\n}\n\nfn fst_error_to_io_error(error: tantivy_fst::Error) -> io::Error {\n    match error {\n        tantivy_fst::Error::Fst(fst_error) => io::Error::other(fst_error),\n        tantivy_fst::Error::Io(ioerror) => ioerror,\n    }\n}\n\nconst STORE_BLOCK_LEN: usize = 128;\n\n#[derive(Debug)]\nstruct BlockAddrBlockMetadata {\n    offset: u64,\n    ref_block_addr: BlockStartAddr,\n    range_start_slope: u32,\n    first_ordinal_slope: u32,\n    range_start_nbits: u8,\n    first_ordinal_nbits: u8,\n    block_len: u16,\n    // these fields are computed on deserialization, and not stored\n    range_shift: i64,\n    ordinal_shift: i64,\n}\n\nimpl BlockAddrBlockMetadata {\n    fn num_bits(&self) -> u8 {\n        self.first_ordinal_nbits + self.range_start_nbits\n    }\n\n    fn deserialize_block_addr(&self, data: &[u8], inner_offset: usize) -> Option<BlockAddr> {\n        if inner_offset == 0 {\n            let range_end = self.ref_block_addr.byte_range_start\n                + extract_bits(data, 0, self.range_start_nbits) as usize\n                + self.range_start_slope as usize\n                - self.range_shift as usize;\n            return Some(self.ref_block_addr.to_block_addr(range_end));\n        }\n        let inner_offset = inner_offset - 1;\n        if inner_offset >= self.block_len as usize {\n            return None;\n        }\n        let num_bits = self.num_bits() as usize;\n\n        let range_start_addr = num_bits * inner_offset;\n        let ordinal_addr = range_start_addr + self.range_start_nbits as usize;\n        let range_end_addr = range_start_addr + num_bits;\n\n        if (range_end_addr + self.range_start_nbits as usize).div_ceil(8) > data.len() {\n            return None;\n        }\n\n        let range_start = self.ref_block_addr.byte_range_start\n            + extract_bits(data, range_start_addr, self.range_start_nbits) as usize\n            + self.range_start_slope as usize * (inner_offset + 1)\n            - self.range_shift as usize;\n        let first_ordinal = self.ref_block_addr.first_ordinal\n            + extract_bits(data, ordinal_addr, self.first_ordinal_nbits)\n            + self.first_ordinal_slope as u64 * (inner_offset + 1) as u64\n            - self.ordinal_shift as u64;\n        let range_end = self.ref_block_addr.byte_range_start\n            + extract_bits(data, range_end_addr, self.range_start_nbits) as usize\n            + self.range_start_slope as usize * (inner_offset + 2)\n            - self.range_shift as usize;\n\n        Some(BlockAddr {\n            first_ordinal,\n            byte_range: range_start..range_end,\n        })\n    }\n\n    fn bisect_for_ord(&self, data: &[u8], target_ord: TermOrdinal) -> (u64, BlockAddr) {\n        let inner_target_ord = target_ord - self.ref_block_addr.first_ordinal;\n        let num_bits = self.num_bits() as usize;\n        let range_start_nbits = self.range_start_nbits as usize;\n        let get_ord = |index| {\n            extract_bits(\n                data,\n                num_bits * index as usize + range_start_nbits,\n                self.first_ordinal_nbits,\n            ) + self.first_ordinal_slope as u64 * (index + 1)\n                - self.ordinal_shift as u64\n        };\n\n        let inner_offset = match binary_search(self.block_len as u64, |index| {\n            get_ord(index).cmp(&inner_target_ord)\n        }) {\n            Ok(inner_offset) => inner_offset + 1,\n            Err(inner_offset) => inner_offset,\n        };\n        // we can unwrap because inner_offset <= self.block_len\n        (\n            inner_offset,\n            self.deserialize_block_addr(data, inner_offset as usize)\n                .unwrap(),\n        )\n    }\n}\n\n// TODO move this function to tantivy_common?\n#[inline(always)]\nfn extract_bits(data: &[u8], addr_bits: usize, num_bits: u8) -> u64 {\n    assert!(num_bits <= 56);\n    let addr_byte = addr_bits / 8;\n    let bit_shift = (addr_bits % 8) as u64;\n    let val_unshifted_unmasked: u64 = if data.len() >= addr_byte + 8 {\n        let b = data[addr_byte..addr_byte + 8].try_into().unwrap();\n        u64::from_le_bytes(b)\n    } else {\n        // the buffer is not large enough.\n        // Let's copy the few remaining bytes to a 8 byte buffer\n        // padded with 0s.\n        let mut buf = [0u8; 8];\n        let data_to_copy = &data[addr_byte..];\n        let nbytes = data_to_copy.len();\n        buf[..nbytes].copy_from_slice(data_to_copy);\n        u64::from_le_bytes(buf)\n    };\n    let val_shifted_unmasked = val_unshifted_unmasked >> bit_shift;\n    let mask = (1u64 << u64::from(num_bits)) - 1;\n    val_shifted_unmasked & mask\n}\n\nimpl BinarySerializable for BlockAddrBlockMetadata {\n    fn serialize<W: Write + ?Sized>(&self, write: &mut W) -> io::Result<()> {\n        self.offset.serialize(write)?;\n        self.ref_block_addr.serialize(write)?;\n        self.range_start_slope.serialize(write)?;\n        self.first_ordinal_slope.serialize(write)?;\n        write.write_all(&[self.first_ordinal_nbits, self.range_start_nbits])?;\n        self.block_len.serialize(write)?;\n        self.num_bits();\n        Ok(())\n    }\n\n    fn deserialize<R: Read>(reader: &mut R) -> io::Result<Self> {\n        let offset = u64::deserialize(reader)?;\n        let ref_block_addr = BlockStartAddr::deserialize(reader)?;\n        let range_start_slope = u32::deserialize(reader)?;\n        let first_ordinal_slope = u32::deserialize(reader)?;\n        let mut buffer = [0u8; 2];\n        reader.read_exact(&mut buffer)?;\n        let first_ordinal_nbits = buffer[0];\n        let range_start_nbits = buffer[1];\n        let block_len = u16::deserialize(reader)?;\n        Ok(BlockAddrBlockMetadata {\n            offset,\n            ref_block_addr,\n            range_start_slope,\n            first_ordinal_slope,\n            range_start_nbits,\n            first_ordinal_nbits,\n            block_len,\n            range_shift: 1 << (range_start_nbits - 1),\n            ordinal_shift: 1 << (first_ordinal_nbits - 1),\n        })\n    }\n}\n\nimpl FixedSize for BlockAddrBlockMetadata {\n    const SIZE_IN_BYTES: usize = u64::SIZE_IN_BYTES\n        + BlockStartAddr::SIZE_IN_BYTES\n        + 2 * u32::SIZE_IN_BYTES\n        + 2\n        + u16::SIZE_IN_BYTES;\n}\n\n#[derive(Debug, Clone)]\nstruct BlockAddrStore {\n    block_meta_bytes: OwnedBytes,\n    addr_bytes: OwnedBytes,\n}\n\nimpl BlockAddrStore {\n    fn open(term_info_store_file: OwnedBytes) -> io::Result<BlockAddrStore> {\n        let (mut len_slice, main_slice) = term_info_store_file.split(8);\n        let len = u64::deserialize(&mut len_slice)? as usize;\n        let (block_meta_bytes, addr_bytes) = main_slice.split(len);\n        Ok(BlockAddrStore {\n            block_meta_bytes,\n            addr_bytes,\n        })\n    }\n\n    fn get_block_meta(&self, store_block_id: usize) -> Option<BlockAddrBlockMetadata> {\n        let mut block_data: &[u8] = self\n            .block_meta_bytes\n            .get(store_block_id * BlockAddrBlockMetadata::SIZE_IN_BYTES..)?;\n        BlockAddrBlockMetadata::deserialize(&mut block_data).ok()\n    }\n\n    fn get(&self, block_id: u64) -> Option<BlockAddr> {\n        let store_block_id = (block_id as usize) / STORE_BLOCK_LEN;\n        let inner_offset = (block_id as usize) % STORE_BLOCK_LEN;\n        let block_addr_block_data = self.get_block_meta(store_block_id)?;\n        block_addr_block_data.deserialize_block_addr(\n            &self.addr_bytes[block_addr_block_data.offset as usize..],\n            inner_offset,\n        )\n    }\n\n    fn binary_search_ord(&self, ord: TermOrdinal) -> (u64, BlockAddr) {\n        let max_block =\n            (self.block_meta_bytes.len() / BlockAddrBlockMetadata::SIZE_IN_BYTES) as u64;\n        let get_first_ordinal = |block_id| {\n            // we can unwrap because block_id < max_block\n            self.get(block_id * STORE_BLOCK_LEN as u64)\n                .unwrap()\n                .first_ordinal\n        };\n        let store_block_id =\n            binary_search(max_block, |block_id| get_first_ordinal(block_id).cmp(&ord));\n        let store_block_id = match store_block_id {\n            Ok(store_block_id) => {\n                let block_id = store_block_id * STORE_BLOCK_LEN as u64;\n                // we can unwrap because store_block_id < max_block\n                return (block_id, self.get(block_id).unwrap());\n            }\n            Err(store_block_id) => store_block_id - 1,\n        };\n\n        // we can unwrap because store_block_id < max_block\n        let block_addr_block_data = self.get_block_meta(store_block_id as usize).unwrap();\n        let (inner_offset, block_addr) = block_addr_block_data.bisect_for_ord(\n            &self.addr_bytes[block_addr_block_data.offset as usize..],\n            ord,\n        );\n        (\n            store_block_id * STORE_BLOCK_LEN as u64 + inner_offset,\n            block_addr,\n        )\n    }\n}\n\nfn binary_search(max: u64, cmp_fn: impl Fn(u64) -> std::cmp::Ordering) -> Result<u64, u64> {\n    use std::cmp::Ordering::*;\n    let mut size = max;\n    let mut left = 0;\n    let mut right = size;\n    while left < right {\n        let mid = left + size / 2;\n\n        let cmp = cmp_fn(mid);\n\n        if cmp == Less {\n            left = mid + 1;\n        } else if cmp == Greater {\n            right = mid;\n        } else {\n            return Ok(mid);\n        }\n\n        size = right - left;\n    }\n    Err(left)\n}\n\nstruct BlockAddrStoreWriter {\n    buffer_block_metas: Vec<u8>,\n    buffer_addrs: Vec<u8>,\n    block_addrs: Vec<BlockAddr>,\n}\n\nimpl BlockAddrStoreWriter {\n    fn new() -> Self {\n        BlockAddrStoreWriter {\n            buffer_block_metas: Vec::new(),\n            buffer_addrs: Vec::new(),\n            block_addrs: Vec::with_capacity(STORE_BLOCK_LEN),\n        }\n    }\n\n    fn flush_block(&mut self) -> io::Result<()> {\n        if self.block_addrs.is_empty() {\n            return Ok(());\n        }\n        let ref_block_addr = self.block_addrs[0].clone();\n\n        for block_addr in &mut self.block_addrs {\n            block_addr.byte_range.start -= ref_block_addr.byte_range.start;\n            block_addr.first_ordinal -= ref_block_addr.first_ordinal;\n        }\n\n        // we are only called if block_addrs is not empty\n        let mut last_block_addr = self.block_addrs.last().unwrap().clone();\n        last_block_addr.byte_range.end -= ref_block_addr.byte_range.start;\n\n        // we skip(1), so we never give an index of 0 to find_best_slope\n        let (range_start_slope, range_start_nbits) = find_best_slope(\n            self.block_addrs\n                .iter()\n                .map(|block| block.byte_range.start as u64)\n                .chain(std::iter::once(last_block_addr.byte_range.end as u64))\n                .enumerate()\n                .skip(1),\n        );\n\n        // we skip(1), so we never give an index of 0 to find_best_slope\n        let (first_ordinal_slope, first_ordinal_nbits) = find_best_slope(\n            self.block_addrs\n                .iter()\n                .map(|block| block.first_ordinal)\n                .enumerate()\n                .skip(1),\n        );\n\n        let range_shift = 1 << (range_start_nbits - 1);\n        let ordinal_shift = 1 << (first_ordinal_nbits - 1);\n\n        let block_addr_block_meta = BlockAddrBlockMetadata {\n            offset: self.buffer_addrs.len() as u64,\n            ref_block_addr: ref_block_addr.to_block_start(),\n            range_start_slope,\n            first_ordinal_slope,\n            range_start_nbits,\n            first_ordinal_nbits,\n            block_len: self.block_addrs.len() as u16 - 1,\n            range_shift,\n            ordinal_shift,\n        };\n        block_addr_block_meta.serialize(&mut self.buffer_block_metas)?;\n\n        let mut bit_packer = BitPacker::new();\n\n        for (i, block_addr) in self.block_addrs.iter().enumerate().skip(1) {\n            let range_pred = (range_start_slope as usize * i) as i64;\n            bit_packer.write(\n                (block_addr.byte_range.start as i64 - range_pred + range_shift) as u64,\n                range_start_nbits,\n                &mut self.buffer_addrs,\n            )?;\n            let first_ordinal_pred = (first_ordinal_slope as u64 * i as u64) as i64;\n            bit_packer.write(\n                (block_addr.first_ordinal as i64 - first_ordinal_pred + ordinal_shift) as u64,\n                first_ordinal_nbits,\n                &mut self.buffer_addrs,\n            )?;\n        }\n\n        let range_pred = (range_start_slope as usize * self.block_addrs.len()) as i64;\n        bit_packer.write(\n            (last_block_addr.byte_range.end as i64 - range_pred + range_shift) as u64,\n            range_start_nbits,\n            &mut self.buffer_addrs,\n        )?;\n        bit_packer.flush(&mut self.buffer_addrs)?;\n\n        self.block_addrs.clear();\n        Ok(())\n    }\n\n    fn write_block_meta(&mut self, block_addr: BlockAddr) -> io::Result<()> {\n        self.block_addrs.push(block_addr);\n        if self.block_addrs.len() >= STORE_BLOCK_LEN {\n            self.flush_block()?;\n        }\n        Ok(())\n    }\n\n    fn serialize<W: std::io::Write>(&mut self, wrt: &mut W) -> io::Result<()> {\n        self.flush_block()?;\n        let len = self.buffer_block_metas.len() as u64;\n        len.serialize(wrt)?;\n        wrt.write_all(&self.buffer_block_metas)?;\n        wrt.write_all(&self.buffer_addrs)?;\n        Ok(())\n    }\n}\n\n/// Given an iterator over (index, value), returns the slope, and number of bits needed to\n/// represent the error to a prediction made by this slope.\n///\n/// The iterator may be empty, but all indexes in it must be non-zero.\nfn find_best_slope(elements: impl Iterator<Item = (usize, u64)> + Clone) -> (u32, u8) {\n    let slope_iterator = elements.clone();\n    let derivation_iterator = elements;\n\n    let mut min_slope_idx = 1;\n    let mut min_slope_val = 0;\n    let mut min_slope = u32::MAX;\n    let mut max_slope_idx = 1;\n    let mut max_slope_val = 0;\n    let mut max_slope = 0;\n    for (index, value) in slope_iterator {\n        let slope = (value / index as u64) as u32;\n        if slope <= min_slope {\n            min_slope = slope;\n            min_slope_idx = index;\n            min_slope_val = value;\n        }\n        if slope >= max_slope {\n            max_slope = slope;\n            max_slope_idx = index;\n            max_slope_val = value;\n        }\n    }\n\n    // above is an heuristic giving the \"highest\" and \"lowest\" point. It's imperfect in that in that\n    // a point that appear earlier might have a high slope derivation, but a smaller absolute\n    // derivation than a latter point.\n    // The actual best values can be obtained by using the symplex method, but the improvement is\n    // likely minimal, and computation is way more complex.\n    //\n    // Assuming these point are the furthest up and down, we find the slope that would cause the\n    // same positive derivation for the highest as negative derivation for the lowest.\n    // A is the optimal slope. B is the derivation to the guess\n    //\n    // 0 = min_slope_val - min_slope_idx * A - B\n    // 0 = max_slope_val - max_slope_idx * A + B\n    //\n    // 0 = min_slope_val + max_slope_val - (min_slope_idx + max_slope_idx) * A\n    // (min_slope_val + max_slope_val) / (min_slope_idx + max_slope_idx) = A\n    //\n    // we actually add some correcting factor to have proper rounding, not truncation.\n\n    let denominator = (min_slope_idx + max_slope_idx) as u64;\n    let final_slope = ((min_slope_val + max_slope_val + denominator / 2) / denominator) as u32;\n\n    // we don't solve for B because our choice of point is suboptimal, so it's actually a lower\n    // bound and we need to iterate to find the actual worst value.\n\n    let max_derivation: u64 = derivation_iterator\n        .map(|(index, value)| (value as i64 - final_slope as i64 * index as i64).unsigned_abs())\n        .max()\n        .unwrap_or(0);\n\n    (final_slope, compute_num_bits(max_derivation) + 1)\n}\n\n#[cfg(test)]\nmod tests {\n    use common::OwnedBytes;\n\n    use super::*;\n    use crate::SSTableDataCorruption;\n    use crate::block_match_automaton::tests::EqBuffer;\n\n    #[test]\n    fn test_sstable_index() {\n        let mut sstable_builder = SSTableIndexBuilder::default();\n        sstable_builder.add_block(b\"aaa\", 10..20, 0u64);\n        sstable_builder.add_block(b\"bbbbbbb\", 20..30, 5u64);\n        sstable_builder.add_block(b\"ccc\", 30..40, 10u64);\n        sstable_builder.add_block(b\"dddd\", 40..50, 15u64);\n        let mut buffer: Vec<u8> = Vec::new();\n        let fst_len = sstable_builder.serialize(&mut buffer).unwrap();\n        let buffer = OwnedBytes::new(buffer);\n        let sstable_index = SSTableIndexV3::load(buffer, fst_len).unwrap();\n        assert_eq!(\n            sstable_index.get_block_with_key(b\"bbbde\"),\n            Some(BlockAddr {\n                first_ordinal: 10u64,\n                byte_range: 30..40\n            })\n        );\n\n        assert_eq!(sstable_index.locate_with_key(b\"aa\").unwrap(), 0);\n        assert_eq!(sstable_index.locate_with_key(b\"aaa\").unwrap(), 0);\n        assert_eq!(sstable_index.locate_with_key(b\"aab\").unwrap(), 1);\n        assert_eq!(sstable_index.locate_with_key(b\"ccc\").unwrap(), 2);\n        assert!(sstable_index.locate_with_key(b\"e\").is_none());\n\n        assert_eq!(sstable_index.locate_with_ord(0), 0);\n        assert_eq!(sstable_index.locate_with_ord(1), 0);\n        assert_eq!(sstable_index.locate_with_ord(4), 0);\n        assert_eq!(sstable_index.locate_with_ord(5), 1);\n        assert_eq!(sstable_index.locate_with_ord(100), 3);\n    }\n\n    #[test]\n    fn test_sstable_with_corrupted_data() {\n        let mut sstable_builder = SSTableIndexBuilder::default();\n        sstable_builder.add_block(b\"aaa\", 10..20, 0u64);\n        sstable_builder.add_block(b\"bbbbbbb\", 20..30, 5u64);\n        sstable_builder.add_block(b\"ccc\", 30..40, 10u64);\n        sstable_builder.add_block(b\"dddd\", 40..50, 15u64);\n        let mut buffer: Vec<u8> = Vec::new();\n        let fst_len = sstable_builder.serialize(&mut buffer).unwrap();\n        buffer[2] = 9u8;\n        let buffer = OwnedBytes::new(buffer);\n        let data_corruption_err = SSTableIndexV3::load(buffer, fst_len).err().unwrap();\n        assert!(matches!(data_corruption_err, SSTableDataCorruption));\n    }\n\n    #[track_caller]\n    fn test_find_shorter_str_in_between_aux(left: &[u8], right: &[u8]) {\n        let mut left_buf = left.to_vec();\n        super::find_shorter_str_in_between(&mut left_buf, right);\n        assert!(left_buf.len() <= left.len());\n        assert!(left <= &left_buf);\n        assert!(&left_buf[..] < right);\n    }\n\n    #[test]\n    fn test_find_shorter_str_in_between() {\n        test_find_shorter_str_in_between_aux(b\"\", b\"hello\");\n        test_find_shorter_str_in_between_aux(b\"abc\", b\"abcd\");\n        test_find_shorter_str_in_between_aux(b\"abcd\", b\"abd\");\n        test_find_shorter_str_in_between_aux(&[0, 0, 0], &[1]);\n        test_find_shorter_str_in_between_aux(&[0, 0, 0], &[0, 0, 1]);\n        test_find_shorter_str_in_between_aux(&[0, 0, 255, 255, 255, 0u8], &[0, 1]);\n    }\n\n    use proptest::prelude::*;\n\n    proptest! {\n        #![proptest_config(ProptestConfig::with_cases(100))]\n        #[test]\n        fn test_proptest_find_shorter_str(left in any::<Vec<u8>>(), right in any::<Vec<u8>>()) {\n            if left < right {\n                test_find_shorter_str_in_between_aux(&left, &right);\n            }\n        }\n    }\n\n    #[test]\n    fn test_find_best_slop() {\n        assert_eq!(super::find_best_slope(std::iter::empty()), (0, 1));\n        assert_eq!(\n            super::find_best_slope(std::iter::once((1, 12345))),\n            (12345, 1)\n        );\n    }\n\n    #[test]\n    fn test_get_block_for_automaton() {\n        let sstable_index_builder = SSTableIndexBuilder {\n            blocks: vec![\n                BlockMeta {\n                    last_key_or_greater: vec![0, 1, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 0,\n                        byte_range: 0..10,\n                    },\n                },\n                BlockMeta {\n                    last_key_or_greater: vec![0, 2, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 5,\n                        byte_range: 10..20,\n                    },\n                },\n                BlockMeta {\n                    last_key_or_greater: vec![0, 3, 2],\n                    block_addr: BlockAddr {\n                        first_ordinal: 10,\n                        byte_range: 20..30,\n                    },\n                },\n            ],\n        };\n\n        let mut sstable_index_bytes = Vec::new();\n        let fst_len = sstable_index_builder\n            .serialize(&mut sstable_index_bytes)\n            .unwrap();\n\n        let sstable = SSTableIndexV3::load(OwnedBytes::new(sstable_index_bytes), fst_len).unwrap();\n\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 1, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                0,\n                BlockAddr {\n                    first_ordinal: 0,\n                    byte_range: 0..10\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 2, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                1,\n                BlockAddr {\n                    first_ordinal: 5,\n                    byte_range: 10..20\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 3, 1]))\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![(\n                2,\n                BlockAddr {\n                    first_ordinal: 10,\n                    byte_range: 20..30\n                }\n            )]\n        );\n        let res = sstable\n            .get_block_for_automaton(&EqBuffer(vec![0, 4, 1]))\n            .collect::<Vec<_>>();\n        assert!(res.is_empty());\n\n        let complex_automaton = EqBuffer(vec![0, 1, 1]).union(EqBuffer(vec![0, 3, 1]));\n        let res = sstable\n            .get_block_for_automaton(&complex_automaton)\n            .collect::<Vec<_>>();\n        assert_eq!(\n            res,\n            vec![\n                (\n                    0,\n                    BlockAddr {\n                        first_ordinal: 0,\n                        byte_range: 0..10\n                    }\n                ),\n                (\n                    2,\n                    BlockAddr {\n                        first_ordinal: 10,\n                        byte_range: 20..30\n                    }\n                )\n            ]\n        );\n    }\n}\n"
  },
  {
    "path": "sstable/src/streamer.rs",
    "content": "use std::io;\nuse std::ops::Bound;\n\nuse tantivy_fst::Automaton;\nuse tantivy_fst::automaton::AlwaysMatch;\n\nuse crate::dictionary::Dictionary;\nuse crate::{DeltaReader, SSTable, TermOrdinal};\n\n/// `StreamerBuilder` is a helper object used to define\n/// a range of terms that should be streamed.\npub struct StreamerBuilder<'a, TSSTable, A = AlwaysMatch>\nwhere\n    A: Automaton,\n    A::State: Clone,\n    TSSTable: SSTable,\n{\n    term_dict: &'a Dictionary<TSSTable>,\n    automaton: A,\n    lower: Bound<Vec<u8>>,\n    upper: Bound<Vec<u8>>,\n    limit: Option<u64>,\n}\n\nfn bound_as_byte_slice(bound: &Bound<Vec<u8>>) -> Bound<&[u8]> {\n    match bound.as_ref() {\n        Bound::Included(key) => Bound::Included(key.as_slice()),\n        Bound::Excluded(key) => Bound::Excluded(key.as_slice()),\n        Bound::Unbounded => Bound::Unbounded,\n    }\n}\n\nimpl<'a, TSSTable, A> StreamerBuilder<'a, TSSTable, A>\nwhere\n    A: Automaton,\n    A::State: Clone,\n    TSSTable: SSTable,\n{\n    pub(crate) fn new(term_dict: &'a Dictionary<TSSTable>, automaton: A) -> Self {\n        StreamerBuilder {\n            term_dict,\n            automaton,\n            lower: Bound::Unbounded,\n            upper: Bound::Unbounded,\n            limit: None,\n        }\n    }\n\n    /// Limit the range to terms greater or equal to the bound\n    pub fn ge<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.lower = Bound::Included(bound.as_ref().to_owned());\n        self\n    }\n\n    /// Limit the range to terms strictly greater than the bound\n    pub fn gt<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.lower = Bound::Excluded(bound.as_ref().to_owned());\n        self\n    }\n\n    /// Limit the range to terms lesser or equal to the bound\n    pub fn le<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.upper = Bound::Included(bound.as_ref().to_owned());\n        self\n    }\n\n    /// Limit the range to terms lesser or equal to the bound\n    pub fn lt<T: AsRef<[u8]>>(mut self, bound: T) -> Self {\n        self.upper = Bound::Excluded(bound.as_ref().to_owned());\n        self\n    }\n\n    /// Load no more data than what's required to to get `limit`\n    /// matching entries.\n    ///\n    /// The resulting [`Streamer`] can still return marginally\n    /// more than `limit` elements.\n    pub fn limit(mut self, limit: u64) -> Self {\n        self.limit = Some(limit);\n        self\n    }\n\n    fn delta_reader(&self) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let key_range = (\n            bound_as_byte_slice(&self.lower),\n            bound_as_byte_slice(&self.upper),\n        );\n        self.term_dict\n            .sstable_delta_reader_for_key_range(key_range, self.limit, &self.automaton)\n    }\n\n    async fn delta_reader_async(\n        &self,\n        merge_holes_under_bytes: usize,\n    ) -> io::Result<DeltaReader<TSSTable::ValueReader>> {\n        let key_range = (\n            bound_as_byte_slice(&self.lower),\n            bound_as_byte_slice(&self.upper),\n        );\n        self.term_dict\n            .sstable_delta_reader_for_key_range_async(\n                key_range,\n                self.limit,\n                &self.automaton,\n                merge_holes_under_bytes,\n            )\n            .await\n    }\n\n    fn into_stream_given_delta_reader(\n        self,\n        delta_reader: DeltaReader<<TSSTable as SSTable>::ValueReader>,\n    ) -> io::Result<Streamer<'a, TSSTable, A>> {\n        let start_state = self.automaton.start();\n        let start_key = bound_as_byte_slice(&self.lower);\n\n        let first_term = match start_key {\n            Bound::Included(key) | Bound::Excluded(key) => self\n                .term_dict\n                .sstable_index\n                .get_block_with_key(key)\n                .map(|block| block.first_ordinal)\n                .unwrap_or(0),\n            Bound::Unbounded => 0,\n        };\n\n        Ok(Streamer {\n            automaton: self.automaton,\n            states: vec![start_state],\n            delta_reader,\n            key: Vec::new(),\n            term_ord: first_term.checked_sub(1),\n            lower_bound: self.lower,\n            upper_bound: self.upper,\n            _lifetime: std::marker::PhantomData,\n        })\n    }\n\n    /// See `into_stream(..)`\n    pub async fn into_stream_async(self) -> io::Result<Streamer<'a, TSSTable, A>> {\n        self.into_stream_async_merging_holes(0).await\n    }\n\n    /// Same as `into_stream_async`, but tries to issue a single io operation when requesting\n    /// blocks that are not consecutive, but also less than `merge_holes_under_bytes` bytes apart.\n    pub async fn into_stream_async_merging_holes(\n        self,\n        merge_holes_under_bytes: usize,\n    ) -> io::Result<Streamer<'a, TSSTable, A>> {\n        let delta_reader = self.delta_reader_async(merge_holes_under_bytes).await?;\n        self.into_stream_given_delta_reader(delta_reader)\n    }\n\n    /// Creates the stream corresponding to the range\n    /// of terms defined using the `StreamerBuilder`.\n    pub fn into_stream(self) -> io::Result<Streamer<'a, TSSTable, A>> {\n        let delta_reader = self.delta_reader()?;\n        self.into_stream_given_delta_reader(delta_reader)\n    }\n}\n\n/// `Streamer` acts as a cursor over a range of terms of a segment.\n/// Terms are guaranteed to be sorted.\npub struct Streamer<'a, TSSTable, A = AlwaysMatch>\nwhere\n    A: Automaton,\n    A::State: Clone,\n    TSSTable: SSTable,\n{\n    automaton: A,\n    states: Vec<A::State>,\n    delta_reader: crate::DeltaReader<TSSTable::ValueReader>,\n    key: Vec<u8>,\n    term_ord: Option<TermOrdinal>,\n    lower_bound: Bound<Vec<u8>>,\n    upper_bound: Bound<Vec<u8>>,\n    // this field is used to please the type-interface of a dictionary in tantivy\n    _lifetime: std::marker::PhantomData<&'a ()>,\n}\n\nimpl<TSSTable> Streamer<'_, TSSTable, AlwaysMatch>\nwhere TSSTable: SSTable\n{\n    pub fn empty() -> Self {\n        Streamer {\n            automaton: AlwaysMatch,\n            states: Vec::new(),\n            delta_reader: DeltaReader::empty(),\n            key: Vec::new(),\n            term_ord: None,\n            lower_bound: Bound::Unbounded,\n            upper_bound: Bound::Unbounded,\n            _lifetime: std::marker::PhantomData,\n        }\n    }\n}\n\nimpl<TSSTable, A> Streamer<'_, TSSTable, A>\nwhere\n    A: Automaton,\n    A::State: Clone,\n    TSSTable: SSTable,\n{\n    /// Advance position the stream on the next item.\n    /// Before the first call to `.advance()`, the stream\n    /// is an uninitialized state.\n    pub fn advance(&mut self) -> bool {\n        while self.delta_reader.advance().unwrap() {\n            self.term_ord = Some(\n                self.term_ord\n                    .map(|term_ord| term_ord + 1u64)\n                    .unwrap_or(0u64),\n            );\n            let common_prefix_len = self.delta_reader.common_prefix_len();\n            self.states.truncate(common_prefix_len + 1);\n            self.key.truncate(common_prefix_len);\n            let mut state: A::State = self.states.last().unwrap().clone();\n            for &b in self.delta_reader.suffix() {\n                state = self.automaton.accept(&state, b);\n                self.states.push(state.clone());\n            }\n            self.key.extend_from_slice(self.delta_reader.suffix());\n            let match_lower_bound = match &self.lower_bound {\n                Bound::Unbounded => true,\n                Bound::Included(lower_bound_key) => lower_bound_key[..] <= self.key[..],\n                Bound::Excluded(lower_bound_key) => lower_bound_key[..] < self.key[..],\n            };\n            if !match_lower_bound {\n                continue;\n            }\n            // We match the lower key once. All subsequent keys will pass that bar.\n            self.lower_bound = Bound::Unbounded;\n            let match_upper_bound = match &self.upper_bound {\n                Bound::Unbounded => true,\n                Bound::Included(upper_bound_key) => upper_bound_key[..] >= self.key[..],\n                Bound::Excluded(upper_bound_key) => upper_bound_key[..] > self.key[..],\n            };\n            if !match_upper_bound {\n                return false;\n            }\n            if self.automaton.is_match(&state) {\n                return true;\n            }\n        }\n        false\n    }\n\n    /// Returns the `TermOrdinal` of the given term.\n    ///\n    /// May panic if the called as `.advance()` as never\n    /// been called before.\n    pub fn term_ord(&self) -> TermOrdinal {\n        self.term_ord.unwrap_or(0u64)\n    }\n\n    /// Accesses the current key.\n    ///\n    /// `.key()` should return the key that was returned\n    /// by the `.next()` method.\n    ///\n    /// If the end of the stream as been reached, and `.next()`\n    /// has been called and returned `None`, `.key()` remains\n    /// the value of the last key encountered.\n    ///\n    /// Before any call to `.next()`, `.key()` returns an empty array.\n    pub fn key(&self) -> &[u8] {\n        &self.key\n    }\n\n    /// Accesses the current value.\n    ///\n    /// Calling `.value()` after the end of the stream will return the\n    /// last `.value()` encountered.\n    ///\n    /// # Panics\n    ///\n    /// Calling `.value()` before the first call to `.advance()` returns\n    /// `V::default()`.\n    pub fn value(&self) -> &TSSTable::Value {\n        self.delta_reader.value()\n    }\n\n    /// Return the next `(key, value)` pair.\n    #[expect(clippy::should_implement_trait)]\n    pub fn next(&mut self) -> Option<(&[u8], &TSSTable::Value)> {\n        if self.advance() {\n            Some((self.key(), self.value()))\n        } else {\n            None\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use std::io;\n\n    use common::OwnedBytes;\n\n    use crate::{Dictionary, MonotonicU64SSTable};\n\n    fn create_test_dictionary() -> io::Result<Dictionary<MonotonicU64SSTable>> {\n        let mut dict_builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new())?;\n        dict_builder.insert(b\"abaisance\", &0)?;\n        dict_builder.insert(b\"abalation\", &1)?;\n        dict_builder.insert(b\"abalienate\", &2)?;\n        dict_builder.insert(b\"abandon\", &3)?;\n        let buffer = dict_builder.finish()?;\n        let owned_bytes = OwnedBytes::new(buffer);\n        Dictionary::from_bytes(owned_bytes)\n    }\n\n    #[test]\n    fn test_sstable_stream() -> io::Result<()> {\n        let dict = create_test_dictionary()?;\n        let mut streamer = dict.stream()?;\n        assert!(streamer.advance());\n        assert_eq!(streamer.key(), b\"abaisance\");\n        assert_eq!(streamer.value(), &0);\n        assert!(streamer.advance());\n        assert_eq!(streamer.key(), b\"abalation\");\n        assert_eq!(streamer.value(), &1);\n        assert!(streamer.advance());\n        assert_eq!(streamer.key(), b\"abalienate\");\n        assert_eq!(streamer.value(), &2);\n        assert!(streamer.advance());\n        assert_eq!(streamer.key(), b\"abandon\");\n        assert_eq!(streamer.value(), &3);\n        assert!(!streamer.advance());\n        Ok(())\n    }\n\n    #[test]\n    fn test_sstable_search() -> io::Result<()> {\n        let term_dict = create_test_dictionary()?;\n        let ptn = tantivy_fst::Regex::new(\"ab.*t.*\").unwrap();\n        let mut term_streamer = term_dict.search(ptn).into_stream()?;\n        assert!(term_streamer.advance());\n        assert_eq!(term_streamer.key(), b\"abalation\");\n        assert_eq!(term_streamer.value(), &1u64);\n        assert!(term_streamer.advance());\n        assert_eq!(term_streamer.key(), b\"abalienate\");\n        assert_eq!(term_streamer.value(), &2u64);\n        assert!(!term_streamer.advance());\n        Ok(())\n    }\n\n    // TODO add test for sparse search with a block of poison (starts with 0xffffffff) => such a\n    // block instantly causes an unexpected EOF error\n}\n"
  },
  {
    "path": "sstable/src/value/index.rs",
    "content": "use std::io;\n\nuse crate::value::{ValueReader, ValueWriter, deserialize_vint_u64};\nuse crate::{BlockAddr, vint};\n\n#[derive(Default)]\npub(crate) struct IndexValueReader {\n    vals: Vec<BlockAddr>,\n}\n\nimpl ValueReader for IndexValueReader {\n    type Value = BlockAddr;\n\n    #[inline(always)]\n    fn value(&self, idx: usize) -> &Self::Value {\n        &self.vals[idx]\n    }\n\n    fn load(&mut self, mut data: &[u8]) -> io::Result<usize> {\n        let original_num_bytes = data.len();\n        let num_vals = deserialize_vint_u64(&mut data) as usize;\n        self.vals.clear();\n        let mut first_ordinal = 0u64;\n        let mut prev_start = deserialize_vint_u64(&mut data) as usize;\n        for _ in 0..num_vals {\n            let len = deserialize_vint_u64(&mut data);\n            let delta_ordinal = deserialize_vint_u64(&mut data);\n\n            first_ordinal += delta_ordinal;\n            let end = prev_start + len as usize;\n            self.vals.push(BlockAddr {\n                byte_range: prev_start..end,\n                first_ordinal,\n            });\n            prev_start = end;\n        }\n        Ok(original_num_bytes - data.len())\n    }\n}\n\n#[derive(Default)]\npub(crate) struct IndexValueWriter {\n    vals: Vec<BlockAddr>,\n}\n\nimpl ValueWriter for IndexValueWriter {\n    type Value = BlockAddr;\n\n    fn write(&mut self, val: &Self::Value) {\n        self.vals.push(val.clone());\n    }\n\n    fn serialize_block(&self, output: &mut Vec<u8>) {\n        let mut prev_ord = 0u64;\n        vint::serialize_into_vec(self.vals.len() as u64, output);\n\n        let start_pos = if let Some(block_addr) = self.vals.first() {\n            block_addr.byte_range.start as u64\n        } else {\n            0\n        };\n        vint::serialize_into_vec(start_pos, output);\n\n        // TODO use array_windows when it gets stabilized\n        for elem in self.vals.windows(2) {\n            let [current, next] = elem else {\n                unreachable!(\"windows should always return exactly 2 elements\");\n            };\n            let len = next.byte_range.start - current.byte_range.start;\n            vint::serialize_into_vec(len as u64, output);\n            let delta = current.first_ordinal - prev_ord;\n            vint::serialize_into_vec(delta, output);\n            prev_ord = current.first_ordinal;\n        }\n        if let Some(last) = self.vals.last() {\n            let len = last.byte_range.end - last.byte_range.start;\n            vint::serialize_into_vec(len as u64, output);\n            let delta = last.first_ordinal - prev_ord;\n            vint::serialize_into_vec(delta, output);\n        }\n    }\n\n    fn clear(&mut self) {\n        self.vals.clear();\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_index_reader_writer() {\n        crate::value::tests::test_value_reader_writer::<_, IndexValueReader, IndexValueWriter>(&[]);\n        crate::value::tests::test_value_reader_writer::<_, IndexValueReader, IndexValueWriter>(&[\n            BlockAddr {\n                byte_range: 0..10,\n                first_ordinal: 0,\n            },\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, IndexValueReader, IndexValueWriter>(&[\n            BlockAddr {\n                byte_range: 0..10,\n                first_ordinal: 0,\n            },\n            BlockAddr {\n                byte_range: 10..20,\n                first_ordinal: 5,\n            },\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, IndexValueReader, IndexValueWriter>(&[\n            BlockAddr {\n                byte_range: 0..10,\n                first_ordinal: 0,\n            },\n            BlockAddr {\n                byte_range: 10..20,\n                first_ordinal: 5,\n            },\n            BlockAddr {\n                byte_range: 20..30,\n                first_ordinal: 10,\n            },\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, IndexValueReader, IndexValueWriter>(&[\n            BlockAddr {\n                byte_range: 5..10,\n                first_ordinal: 2,\n            },\n        ]);\n    }\n}\n"
  },
  {
    "path": "sstable/src/value/mod.rs",
    "content": "pub(crate) mod index;\nmod range;\nmod u64_monotonic;\nmod vec_u32;\nmod void;\n\nuse std::io;\n\npub use range::{RangeValueReader, RangeValueWriter};\npub use u64_monotonic::{U64MonotonicValueReader, U64MonotonicValueWriter};\npub use vec_u32::{VecU32ValueReader, VecU32ValueWriter};\npub use void::{VoidValueReader, VoidValueWriter};\n\n/// `ValueReader` is a trait describing the contract of something\n/// reading blocks of value, and offering random access within this values.\npub trait ValueReader: Default {\n    /// Type of the value being read.\n    type Value;\n\n    /// Access the value at index `idx`, in the last block that was read\n    /// via a call to `ValueReader::read`.\n    fn value(&self, idx: usize) -> &Self::Value;\n\n    /// Loads a block.\n    ///\n    /// Returns the number of bytes that were read.\n    fn load(&mut self, data: &[u8]) -> io::Result<usize>;\n}\n\n/// `ValueWriter` is a trait to make it possible to write blocks\n/// of value.\npub trait ValueWriter: Default {\n    /// Type of the value being written.\n    type Value;\n\n    /// Records a new value.\n    /// This method usually just accumulates data in a `Vec`,\n    /// only to be serialized on the call to `ValueWriter::serialize_block`.\n    fn write(&mut self, val: &Self::Value);\n\n    /// Serializes the accumulated values into the output buffer.\n    fn serialize_block(&self, output: &mut Vec<u8>);\n\n    /// Clears the `ValueWriter`. After a call to clear, the `ValueWriter`\n    /// should behave like a fresh `ValueWriter::default()`.\n    fn clear(&mut self);\n}\n\nfn deserialize_vint_u64(data: &mut &[u8]) -> u64 {\n    let (num_bytes, val) = super::vint::deserialize_read(data);\n    *data = &data[num_bytes..];\n    val\n}\n\n#[cfg(test)]\npub(crate) mod tests {\n    use std::fmt;\n\n    use super::{ValueReader, ValueWriter};\n\n    pub(crate) fn test_value_reader_writer<\n        V: Eq + fmt::Debug,\n        TReader: ValueReader<Value = V>,\n        TWriter: ValueWriter<Value = V>,\n    >(\n        value_block: &[V],\n    ) {\n        let mut buffer = Vec::new();\n        {\n            let mut writer = TWriter::default();\n            for value in value_block {\n                writer.write(value);\n            }\n            writer.serialize_block(&mut buffer);\n            writer.clear();\n        }\n        let data_len = buffer.len();\n        buffer.extend_from_slice(&b\"extradata\"[..]);\n        let mut reader = TReader::default();\n        assert_eq!(reader.load(&buffer[..]).unwrap(), data_len);\n        for (i, val) in value_block.iter().enumerate() {\n            assert_eq!(reader.value(i), val);\n        }\n    }\n}\n"
  },
  {
    "path": "sstable/src/value/range.rs",
    "content": "use std::io;\nuse std::ops::Range;\n\nuse crate::value::{ValueReader, ValueWriter, deserialize_vint_u64};\n\n/// See module comment.\n#[derive(Default)]\npub struct RangeValueReader {\n    vals: Vec<Range<u64>>,\n}\n\nimpl ValueReader for RangeValueReader {\n    type Value = Range<u64>;\n\n    #[inline(always)]\n    fn value(&self, idx: usize) -> &Range<u64> {\n        &self.vals[idx]\n    }\n\n    fn load(&mut self, mut data: &[u8]) -> io::Result<usize> {\n        self.vals.clear();\n        let original_num_bytes = data.len();\n        let len = deserialize_vint_u64(&mut data) as usize;\n        if len != 0 {\n            let mut prev_val = deserialize_vint_u64(&mut data);\n            for _ in 1..len {\n                let next_val = prev_val + deserialize_vint_u64(&mut data);\n                self.vals.push(prev_val..next_val);\n                prev_val = next_val;\n            }\n        }\n        Ok(original_num_bytes - data.len())\n    }\n}\n\n/// Range writer. The range are required to partition the\n/// space.\n///\n/// In other words, two consecutive keys `k1` and `k2`\n/// are required to observe\n/// `range_sstable[k1].end == range_sstable[k2].start`.\n///\n/// The writer will panic if the inserted value do not follow\n/// this property.\n///\n/// The first range is not required to start at `0`.\n#[derive(Default)]\npub struct RangeValueWriter {\n    vals: Vec<u64>,\n}\n\nimpl ValueWriter for RangeValueWriter {\n    type Value = Range<u64>;\n\n    fn write(&mut self, val: &Range<u64>) {\n        if let Some(previous_offset) = self.vals.last().copied() {\n            assert_eq!(previous_offset, val.start);\n            self.vals.push(val.end);\n        } else {\n            self.vals.push(val.start);\n            self.vals.push(val.end)\n        }\n    }\n\n    fn serialize_block(&self, writer: &mut Vec<u8>) {\n        let mut prev_val = 0u64;\n        crate::vint::serialize_into_vec(self.vals.len() as u64, writer);\n        for &val in &self.vals {\n            let delta = val - prev_val;\n            crate::vint::serialize_into_vec(delta, writer);\n            prev_val = val;\n        }\n    }\n\n    fn clear(&mut self) {\n        self.vals.clear();\n    }\n}\n\n#[cfg(test)]\n#[expect(clippy::single_range_in_vec_init)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_range_reader_writer() {\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[]);\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[\n            0..3,\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[\n            0..3,\n            3..10,\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[\n            0..0,\n            0..10,\n        ]);\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[\n            100..110,\n            110..121,\n            121..1250,\n        ]);\n    }\n\n    #[test]\n    #[should_panic]\n    fn test_range_reader_writer_panics() {\n        crate::value::tests::test_value_reader_writer::<_, RangeValueReader, RangeValueWriter>(&[\n            1..3,\n            4..10,\n        ]);\n    }\n}\n"
  },
  {
    "path": "sstable/src/value/u64_monotonic.rs",
    "content": "use std::io;\n\nuse crate::value::{ValueReader, ValueWriter, deserialize_vint_u64};\nuse crate::vint;\n\n#[derive(Default)]\npub struct U64MonotonicValueReader {\n    vals: Vec<u64>,\n}\n\nimpl ValueReader for U64MonotonicValueReader {\n    type Value = u64;\n\n    #[inline(always)]\n    fn value(&self, idx: usize) -> &Self::Value {\n        &self.vals[idx]\n    }\n\n    fn load(&mut self, mut data: &[u8]) -> io::Result<usize> {\n        let original_num_bytes = data.len();\n        let num_vals = deserialize_vint_u64(&mut data) as usize;\n        self.vals.clear();\n        let mut prev_val = 0u64;\n        for _ in 0..num_vals {\n            let delta = deserialize_vint_u64(&mut data);\n            let val = prev_val + delta;\n            self.vals.push(val);\n            prev_val = val;\n        }\n        Ok(original_num_bytes - data.len())\n    }\n}\n\n#[derive(Default)]\npub struct U64MonotonicValueWriter {\n    vals: Vec<u64>,\n}\n\nimpl ValueWriter for U64MonotonicValueWriter {\n    type Value = u64;\n\n    fn write(&mut self, val: &Self::Value) {\n        self.vals.push(*val);\n    }\n\n    fn serialize_block(&self, output: &mut Vec<u8>) {\n        let mut prev_val = 0u64;\n        vint::serialize_into_vec(self.vals.len() as u64, output);\n        for &val in &self.vals {\n            let delta = val - prev_val;\n            vint::serialize_into_vec(delta, output);\n            prev_val = val;\n        }\n    }\n\n    fn clear(&mut self) {\n        self.vals.clear();\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_u64_monotonic_reader_writer() {\n        crate::value::tests::test_value_reader_writer::<\n            _,\n            U64MonotonicValueReader,\n            U64MonotonicValueWriter,\n        >(&[]);\n        crate::value::tests::test_value_reader_writer::<\n            _,\n            U64MonotonicValueReader,\n            U64MonotonicValueWriter,\n        >(&[5]);\n        crate::value::tests::test_value_reader_writer::<\n            _,\n            U64MonotonicValueReader,\n            U64MonotonicValueWriter,\n        >(&[1u64, 30u64]);\n    }\n}\n"
  },
  {
    "path": "sstable/src/value/vec_u32.rs",
    "content": "use std::io;\n\nuse super::{ValueReader, ValueWriter};\n\n#[derive(Default)]\npub struct VecU32ValueReader {\n    vals: Vec<Vec<u32>>,\n}\n\nimpl ValueReader for VecU32ValueReader {\n    type Value = Vec<u32>;\n\n    #[inline(always)]\n    fn value(&self, idx: usize) -> &Self::Value {\n        &self.vals[idx]\n    }\n\n    fn load(&mut self, mut data: &[u8]) -> io::Result<usize> {\n        let original_num_bytes = data.len();\n        self.vals.clear();\n\n        // The first 4 bytes are the number of blocks\n        let num_blocks = u32::from_le_bytes(data[..4].try_into().unwrap()) as usize;\n        data = &data[4..];\n\n        for _ in 0..num_blocks {\n            // Each block starts with a 4-byte length\n            let segment_len = u32::from_le_bytes(data[..4].try_into().unwrap()) as usize;\n            data = &data[4..];\n\n            // Read the segment IDs for this block\n            let mut segment_ids = Vec::with_capacity(segment_len);\n            for _ in 0..segment_len {\n                let segment_id = u32::from_le_bytes(data[..4].try_into().unwrap());\n                segment_ids.push(segment_id);\n                data = &data[4..];\n            }\n            self.vals.push(segment_ids);\n        }\n\n        // Return the number of bytes consumed\n        Ok(original_num_bytes - data.len())\n    }\n}\n\n#[derive(Default)]\npub struct VecU32ValueWriter {\n    vals: Vec<Vec<u32>>,\n}\n\nimpl ValueWriter for VecU32ValueWriter {\n    type Value = Vec<u32>;\n\n    fn write(&mut self, val: &Self::Value) {\n        self.vals.push(val.to_vec());\n    }\n\n    fn serialize_block(&self, output: &mut Vec<u8>) {\n        let num_blocks = self.vals.len() as u32;\n        output.extend_from_slice(&num_blocks.to_le_bytes());\n        for vals in &self.vals {\n            let len = vals.len() as u32;\n            output.extend_from_slice(&len.to_le_bytes());\n            for &segment_id in vals.iter() {\n                output.extend_from_slice(&segment_id.to_le_bytes());\n            }\n        }\n    }\n\n    fn clear(&mut self) {\n        self.vals.clear();\n    }\n}\n"
  },
  {
    "path": "sstable/src/value/void.rs",
    "content": "use std::io;\n\nuse crate::value::{ValueReader, ValueWriter};\n\n#[derive(Default)]\npub struct VoidValueReader;\n\nimpl ValueReader for VoidValueReader {\n    type Value = ();\n\n    #[inline(always)]\n    fn value(&self, _idx: usize) -> &() {\n        &()\n    }\n\n    fn load(&mut self, _data: &[u8]) -> io::Result<usize> {\n        Ok(0)\n    }\n}\n\n#[derive(Default)]\npub struct VoidValueWriter;\n\nimpl ValueWriter for VoidValueWriter {\n    type Value = ();\n\n    fn write(&mut self, _val: &()) {}\n\n    fn serialize_block(&self, _output: &mut Vec<u8>) {}\n\n    fn clear(&mut self) {}\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn test_range_reader_writer() {\n        crate::value::tests::test_value_reader_writer::<_, VoidValueReader, VoidValueWriter>(&[]);\n        crate::value::tests::test_value_reader_writer::<_, VoidValueReader, VoidValueWriter>(&[()]);\n        crate::value::tests::test_value_reader_writer::<_, VoidValueReader, VoidValueWriter>(&[\n            (),\n            (),\n            (),\n        ]);\n    }\n}\n"
  },
  {
    "path": "sstable/src/vint.rs",
    "content": "const CONTINUE_BIT: u8 = 128u8;\n\npub fn serialize(mut val: u64, buffer: &mut [u8]) -> usize {\n    for (i, b) in buffer.iter_mut().enumerate() {\n        let next_byte: u8 = (val & 127u64) as u8;\n        val >>= 7;\n        if val == 0u64 {\n            *b = next_byte;\n            return i + 1;\n        } else {\n            *b = next_byte | CONTINUE_BIT;\n        }\n    }\n    10 //< actually unreachable\n}\n\npub fn serialize_into_vec(val: u64, buffer: &mut Vec<u8>) {\n    let mut buf = [0u8; 10];\n    let num_bytes = serialize(val, &mut buf[..]);\n    buffer.extend_from_slice(&buf[..num_bytes]);\n}\n\n// super slow but we don't care\npub fn deserialize_read(buf: &[u8]) -> (usize, u64) {\n    let mut result = 0u64;\n    let mut shift = 0u64;\n    let mut consumed = 0;\n\n    for &b in buf {\n        consumed += 1;\n        result |= u64::from(b % 128u8) << shift;\n        if b < CONTINUE_BIT {\n            break;\n        }\n        shift += 7;\n    }\n    (consumed, result)\n}\n\n#[cfg(test)]\nmod tests {\n    use super::{deserialize_read, serialize};\n\n    fn aux_test_int(val: u64, expect_len: usize) {\n        let mut buffer = [0u8; 14];\n        assert_eq!(serialize(val, &mut buffer[..]), expect_len);\n        assert_eq!(deserialize_read(&buffer), (expect_len, val));\n    }\n\n    #[test]\n    fn test_vint() {\n        aux_test_int(0u64, 1);\n        aux_test_int(17u64, 1);\n        aux_test_int(127u64, 1);\n        aux_test_int(128u64, 2);\n        aux_test_int(123423418u64, 4);\n        for i in 1..63 {\n            let power_of_two = 1u64 << i;\n            aux_test_int(power_of_two + 1, (i / 7) + 1);\n            aux_test_int(power_of_two, (i / 7) + 1);\n            aux_test_int(power_of_two - 1, ((i - 1) / 7) + 1);\n        }\n        aux_test_int(u64::MAX, 10);\n    }\n}\n"
  },
  {
    "path": "sstable/tests/sstable_test.rs",
    "content": "use common::OwnedBytes;\nuse tantivy_sstable::{Dictionary, MonotonicU64SSTable, VecU32ValueSSTable};\n\n#[test]\nfn test_create_and_search_sstable() {\n    // Create a new sstable in memory.\n    let mut builder = Dictionary::<MonotonicU64SSTable>::builder(Vec::new()).unwrap();\n    builder.insert(b\"apple\", &1).unwrap();\n    builder.insert(b\"banana\", &2).unwrap();\n    builder.insert(b\"orange\", &3).unwrap();\n    let sstable_bytes = builder.finish().unwrap();\n\n    // Open the sstable.\n    let sstable =\n        Dictionary::<MonotonicU64SSTable>::from_bytes(OwnedBytes::new(sstable_bytes)).unwrap();\n\n    // Search for a key.\n    let value = sstable.get(b\"banana\").unwrap();\n    assert_eq!(value, Some(2));\n\n    // Search for a non-existent key.\n    let value = sstable.get(b\"blub\").unwrap();\n    assert_eq!(value, None);\n}\n\n#[test]\nfn test_custom_value_sstable() {\n    // Create a new sstable with custom values.\n    let mut builder = Dictionary::<VecU32ValueSSTable>::builder(Vec::new()).unwrap();\n    builder.set_block_len(4096); // Ensure both values are in the same block\n    builder.insert(b\"first\", &vec![1, 2, 3]).unwrap();\n    builder.insert(b\"second\", &vec![4, 5]).unwrap();\n    let sstable_bytes = builder.finish().unwrap();\n\n    // Open the sstable.\n    let sstable =\n        Dictionary::<VecU32ValueSSTable>::from_bytes(OwnedBytes::new(sstable_bytes)).unwrap();\n\n    let mut stream = sstable.stream().unwrap();\n    assert!(stream.advance());\n    assert_eq!(stream.key(), b\"first\");\n    assert_eq!(stream.value(), &vec![1, 2, 3]);\n\n    assert!(stream.advance());\n    assert_eq!(stream.key(), b\"second\");\n    assert_eq!(stream.value(), &vec![4, 5]);\n\n    assert!(!stream.advance());\n}\n"
  },
  {
    "path": "stacker/Cargo.toml",
    "content": "[package]\nname = \"tantivy-stacker\"\nversion = \"0.6.0\"\nedition = \"2024\"\nlicense = \"MIT\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\ndescription = \"term hashmap used for indexing\"\n\n[dependencies]\nmurmurhash32 = \"0.3\"\ncommon = { version = \"0.10\", path = \"../common/\", package = \"tantivy-common\" }\nahash = { version = \"0.8.11\", default-features = false, optional = true }\n\n\n[[bench]]\nharness = false\nname = \"bench\"\npath = \"benches/bench.rs\"\n\n[[example]]\nname = \"hashmap\"\npath = \"example/hashmap.rs\"\n\n[dev-dependencies]\nrand = \"0.9\"\nzipf = \"7.0.0\"\nrustc-hash = \"2.1.0\"\nproptest = \"1.2.0\"\nbinggan = { version = \"0.14.0\" }\nrand_distr = \"0.5\"\n\n[features]\ncompare_hash_only = [\"ahash\"] # Compare hash only, not the key in the Hashmap\nunstable = [] # useful for benches.\n"
  },
  {
    "path": "stacker/Performance.md",
    "content": "\n# Notes\n\n- `extend_from_slice(&key)` calls memcpy, which is relatively slow, since most keys are relatively short. For now there's a specialized version toavoid memcpy calls.\n    Wild copy 16 bytes in a loop is faster, but would require a guard against overflow from the caller side. (We probably can do that). \n- Comparing two slices of unknown length calls memcmp. Same as above, we can do a specialized version.\n\nfastcmp and fastcpy both employ the same trick, to compare slices of odd length, e.g. 2 operations unconditional on 4 bytes, instead 3 operations with conditionals (1 4byte, 1 2byte, 1 1byte).\n[1, 2, 3, 4, 5, 6, 7]\n[1, 2, 3, 4]\n         [4, 5, 6, 7]\n\n- Since the hashmap writes the values on every key insert/update, the values like expull should be small. Therefore inlining of the values has been removed.\n- Currently the first call to Expull will get a capacity of 0. It would be beneficial if it could be initialized with some memory, so that the first call doesn't have to allocate. But that would mean we don't have `Default` impls.\n"
  },
  {
    "path": "stacker/benches/bench.rs",
    "content": "use binggan::plugins::PeakMemAllocPlugin;\nuse binggan::{BenchRunner, INSTRUMENTED_SYSTEM, PeakMemAlloc, black_box};\nuse rand::SeedableRng;\nuse rustc_hash::FxHashMap;\nuse tantivy_stacker::{ArenaHashMap, ExpUnrolledLinkedList, MemoryArena};\n\nconst ALICE: &str = include_str!(\"../../benches/alice.txt\");\n\n#[global_allocator]\npub static GLOBAL: &PeakMemAlloc<std::alloc::System> = &INSTRUMENTED_SYSTEM;\n\nfn bench_vint() {\n    let mut runner = BenchRunner::new();\n    // Set the peak mem allocator. This will enable peak memory reporting.\n    runner.add_plugin(PeakMemAllocPlugin::new(GLOBAL));\n\n    {\n        let input_bytes = ALICE.len();\n\n        let alice_terms_as_bytes: Vec<&[u8]> = ALICE\n            .split_ascii_whitespace()\n            .map(|el| el.as_bytes())\n            .collect();\n\n        let alice_terms_as_bytes_with_docid: Vec<(u32, &[u8])> = ALICE\n            .split_ascii_whitespace()\n            .map(|el| el.as_bytes())\n            .enumerate()\n            .map(|(docid, el)| (docid as u32, el))\n            .collect();\n\n        // Alice benchmark\n        let mut group = runner.new_group();\n        group.set_name(format!(\"alice (num terms: {})\", ALICE.len()));\n        group.set_input_size(input_bytes);\n        group.register_with_input(\"hashmap\", &alice_terms_as_bytes, move |data| {\n            black_box(create_hash_map(data.iter()));\n        });\n        group.register_with_input(\n            \"hasmap with postings\",\n            &alice_terms_as_bytes_with_docid,\n            move |data| {\n                black_box(create_hash_map_with_expull(data.iter().cloned()));\n            },\n        );\n        group.register_with_input(\n            \"fxhashmap ref postings\",\n            &alice_terms_as_bytes,\n            move |data| {\n                black_box(create_fx_hash_ref_map_with_expull(data.iter().cloned()));\n            },\n        );\n        group.register_with_input(\n            \"fxhasmap owned postings\",\n            &alice_terms_as_bytes,\n            move |data| {\n                black_box(create_fx_hash_owned_map_with_expull(data.iter().cloned()));\n            },\n        );\n        group.run();\n    }\n\n    {\n        for (num_numbers, num_numbers_label) in [\n            (100_000u64, \"100k\"),\n            (1_000_000, \"1mio\"),\n            (2_000_000, \"2mio\"),\n            (5_000_000, \"5mio\"),\n        ] {\n            // benchmark unique numbers\n            {\n                let numbers: Vec<[u8; 8]> = (0..num_numbers).map(|el| el.to_le_bytes()).collect();\n                let numbers_with_doc: Vec<_> = numbers\n                    .iter()\n                    .enumerate()\n                    .map(|(docid, el)| (docid as u32, el))\n                    .collect();\n\n                let input_bytes = numbers.len() * 8;\n                let mut group = runner.new_group();\n                group.set_name(format!(\"numbers unique {}\", num_numbers_label));\n                group.set_input_size(input_bytes);\n                group.register_with_input(\"only hashmap\", &numbers, move |data| {\n                    black_box(create_hash_map(data.iter()));\n                });\n                group.register_with_input(\"hasmap with postings\", &numbers_with_doc, move |data| {\n                    black_box(create_hash_map_with_expull(data.iter().cloned()));\n                });\n                group.run();\n            }\n            // benchmark zipfs distribution numbers\n            {\n                use rand::distr::Distribution;\n                use rand::rngs::StdRng;\n                let mut rng = StdRng::from_seed([3u8; 32]);\n                let zipf = rand_distr::Zipf::new(10_000.0f64, 1.03).unwrap();\n                let numbers: Vec<[u8; 8]> = (0..num_numbers)\n                    .map(|_| zipf.sample(&mut rng).to_le_bytes())\n                    .collect();\n                let numbers_with_doc: Vec<_> = numbers\n                    .iter()\n                    .enumerate()\n                    .map(|(docid, el)| (docid as u32, el))\n                    .collect();\n\n                let input_bytes = numbers.len() * 8;\n                let mut group = runner.new_group();\n                group.set_name(format!(\"zipfs numbers {}\", num_numbers_label));\n                group.set_input_size(input_bytes);\n                group.register_with_input(\"hashmap\", &numbers, move |data| {\n                    black_box(create_hash_map(data.iter()));\n                });\n                group.register_with_input(\"hasmap with postings\", &numbers_with_doc, move |data| {\n                    black_box(create_hash_map_with_expull(data.iter().cloned()));\n                });\n                group.run();\n            }\n        }\n    }\n}\n\nfn main() {\n    bench_vint();\n}\n\nconst HASHMAP_CAPACITY: usize = 1 << 15;\n\n/// Only records the doc ids\n#[derive(Clone, Default, Copy)]\npub struct DocIdRecorder {\n    stack: ExpUnrolledLinkedList,\n}\nimpl DocIdRecorder {\n    fn new_doc(&mut self, doc: u32, arena: &mut MemoryArena) {\n        self.stack.writer(arena).write_u32_vint(doc);\n    }\n}\n\nfn create_hash_map<T: AsRef<[u8]>>(terms: impl Iterator<Item = T>) -> ArenaHashMap {\n    let mut map = ArenaHashMap::with_capacity(HASHMAP_CAPACITY);\n    for term in terms {\n        map.mutate_or_create(term.as_ref(), |val| {\n            if let Some(mut val) = val {\n                val += 1;\n                val\n            } else {\n                1u64\n            }\n        });\n    }\n\n    map\n}\n\nfn create_hash_map_with_expull<T: AsRef<[u8]>>(\n    terms: impl Iterator<Item = (u32, T)>,\n) -> ArenaHashMap {\n    let mut memory_arena = MemoryArena::default();\n    let mut map = ArenaHashMap::with_capacity(HASHMAP_CAPACITY);\n    for (i, term) in terms {\n        map.mutate_or_create(term.as_ref(), |val: Option<DocIdRecorder>| {\n            if let Some(mut rec) = val {\n                rec.new_doc(i, &mut memory_arena);\n                rec\n            } else {\n                DocIdRecorder::default()\n            }\n        });\n    }\n\n    map\n}\n\nfn create_fx_hash_ref_map_with_expull(\n    terms: impl Iterator<Item = &'static [u8]>,\n) -> FxHashMap<&'static [u8], Vec<u32>> {\n    let terms = terms.enumerate();\n    let mut map = FxHashMap::with_capacity_and_hasher(HASHMAP_CAPACITY, Default::default());\n    for (i, term) in terms {\n        map.entry(term.as_ref())\n            .or_insert_with(Vec::new)\n            .push(i as u32);\n    }\n    map\n}\n\nfn create_fx_hash_owned_map_with_expull(\n    terms: impl Iterator<Item = &'static [u8]>,\n) -> FxHashMap<Vec<u8>, Vec<u32>> {\n    let terms = terms.enumerate();\n    let mut map = FxHashMap::with_capacity_and_hasher(HASHMAP_CAPACITY, Default::default());\n    for (i, term) in terms {\n        map.entry(term.as_ref().to_vec())\n            .or_insert_with(Vec::new)\n            .push(i as u32);\n    }\n    map\n}\n"
  },
  {
    "path": "stacker/example/hashmap.rs",
    "content": "use tantivy_stacker::ArenaHashMap;\n\nconst ALICE: &str = include_str!(\"../../benches/alice.txt\");\n\nfn main() {\n    create_hash_map((0..100_000_000).map(|el| el.to_string()));\n\n    for _ in 0..1000 {\n        create_hash_map(ALICE.split_whitespace());\n    }\n}\n\nfn create_hash_map<T: AsRef<str>>(terms: impl Iterator<Item = T>) -> ArenaHashMap {\n    let mut map = ArenaHashMap::with_capacity(4);\n    for term in terms {\n        map.mutate_or_create(term.as_ref().as_bytes(), |val| {\n            if let Some(mut val) = val {\n                val += 1;\n                val\n            } else {\n                1u64\n            }\n        });\n    }\n\n    map\n}\n"
  },
  {
    "path": "stacker/fuzz_test/Cargo.toml",
    "content": "[package]\nname = \"fuzz_test\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nahash = \"0.8.7\"\nrand = \"0.9\"\nrand_distr = \"0.5\"\ntantivy-stacker = { version = \"0.2.0\", path = \"..\" }\n\n[workspace]\n\n"
  },
  {
    "path": "stacker/fuzz_test/src/main.rs",
    "content": "use ahash::AHashMap;\nuse rand::{rngs::StdRng, Rng, SeedableRng};\nuse rand_distr::Exp;\nuse tantivy_stacker::ArenaHashMap;\n\nfn main() {\n    for _ in 0..1_000_000 {\n        let seed: u64 = rand::random();\n        test_with_seed(seed);\n    }\n}\n\nfn test_with_seed(seed: u64) {\n    let mut hash_map = AHashMap::new();\n    let mut arena_hashmap = ArenaHashMap::default();\n    let mut rng = StdRng::seed_from_u64(seed);\n    let key_count = rng.random_range(1_000..=1_000_000);\n    let exp = Exp::new(0.05).unwrap();\n\n    for _ in 0..key_count {\n        let key_length = rng.sample::<f32, _>(exp).min(u16::MAX as f32).max(1.0) as usize;\n\n        let key: Vec<u8> = (0..key_length).map(|_| rng.gen()).collect();\n\n        arena_hashmap.mutate_or_create(&key, |current_count| {\n            let count: u64 = current_count.unwrap_or(0);\n            count + 1\n        });\n        hash_map.entry(key).and_modify(|e| *e += 1).or_insert(1);\n    }\n\n    println!(\n        \"Seed: {} \\t {:.2}MB\",\n        seed,\n        arena_hashmap.memory_arena.len() as f32 / 1024.0 / 1024.0\n    );\n    // Check the contents of the ArenaHashMap\n    for (key, addr) in arena_hashmap.iter() {\n        let count: u64 = arena_hashmap.read(addr);\n        let count_expected = hash_map\n            .get(key)\n            .unwrap_or_else(|| panic!(\"NOT FOUND: Key: {:?}, Count: {}\", key, count));\n        assert_eq!(count, *count_expected);\n    }\n}\n"
  },
  {
    "path": "stacker/src/arena_hashmap.rs",
    "content": "use super::{Addr, MemoryArena};\nuse crate::shared_arena_hashmap::SharedArenaHashMap;\n\n/// Customized `HashMap` with `&[u8]` keys\n///\n/// Its main particularity is that rather than storing its\n/// keys in the heap, keys are stored in a memory arena\n/// inline with the values.\n///\n/// The quirky API has the benefit of avoiding\n/// the computation of the hash of the key twice,\n/// or copying the key as long as there is no insert.\n///\n/// ArenaHashMap is like SharedArenaHashMap but takes ownership\n/// of the memory arena. The memory arena stores the serialized\n/// keys and values.\npub struct ArenaHashMap {\n    shared_arena_hashmap: SharedArenaHashMap,\n    pub memory_arena: MemoryArena,\n}\n\nimpl Default for ArenaHashMap {\n    fn default() -> Self {\n        ArenaHashMap::with_capacity(4)\n    }\n}\n\nimpl ArenaHashMap {\n    pub fn with_capacity(table_size: usize) -> ArenaHashMap {\n        let memory_arena = MemoryArena::default();\n\n        ArenaHashMap {\n            shared_arena_hashmap: SharedArenaHashMap::with_capacity(table_size),\n            memory_arena,\n        }\n    }\n\n    #[inline]\n    pub fn read<Item: Copy + 'static>(&self, addr: Addr) -> Item {\n        self.memory_arena.read(addr)\n    }\n\n    #[inline]\n    pub fn mem_usage(&self) -> usize {\n        self.shared_arena_hashmap.mem_usage() + self.memory_arena.mem_usage()\n    }\n\n    #[inline]\n    pub fn is_empty(&self) -> bool {\n        self.shared_arena_hashmap.is_empty()\n    }\n\n    #[inline]\n    pub fn len(&self) -> usize {\n        self.shared_arena_hashmap.len()\n    }\n\n    #[inline]\n    pub fn iter(&self) -> impl Iterator<Item = (&[u8], Addr)> {\n        self.shared_arena_hashmap.iter(&self.memory_arena)\n    }\n\n    /// Get a value associated to a key.\n    #[inline]\n    pub fn get<V>(&self, key: &[u8]) -> Option<V>\n    where V: Copy + 'static {\n        self.shared_arena_hashmap.get(key, &self.memory_arena)\n    }\n\n    /// `update` create a new entry for a given key if it does not exist\n    /// or updates the existing entry.\n    ///\n    /// The actual logic for this update is define in the `updater`\n    /// argument.\n    ///\n    /// If the key is not present, `updater` will receive `None` and\n    /// will be in charge of returning a default value.\n    /// If the key already as an associated value, then it will be passed\n    /// `Some(previous_value)`.\n    #[inline]\n    pub fn mutate_or_create<V>(&mut self, key: &[u8], updater: impl FnMut(Option<V>) -> V)\n    where V: Copy + 'static {\n        self.shared_arena_hashmap\n            .mutate_or_create(key, &mut self.memory_arena, updater);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::HashMap;\n\n    use super::ArenaHashMap;\n\n    #[test]\n    fn test_hash_map() {\n        let mut hash_map: ArenaHashMap = ArenaHashMap::default();\n        hash_map.mutate_or_create(b\"abc\", |opt_val: Option<u32>| {\n            assert_eq!(opt_val, None);\n            3u32\n        });\n        hash_map.mutate_or_create(b\"abcd\", |opt_val: Option<u32>| {\n            assert_eq!(opt_val, None);\n            4u32\n        });\n        hash_map.mutate_or_create(b\"abc\", |opt_val: Option<u32>| {\n            assert_eq!(opt_val, Some(3u32));\n            5u32\n        });\n        let mut vanilla_hash_map = HashMap::new();\n        let iter_values = hash_map.iter();\n        for (key, addr) in iter_values {\n            let val: u32 = hash_map.memory_arena.read(addr);\n            vanilla_hash_map.insert(key.to_owned(), val);\n        }\n        assert_eq!(vanilla_hash_map.len(), 2);\n    }\n    #[test]\n    fn test_empty_hashmap() {\n        let hash_map: ArenaHashMap = ArenaHashMap::default();\n        assert_eq!(hash_map.get::<u32>(b\"abc\"), None);\n    }\n\n    #[test]\n    fn test_many_terms() {\n        let mut terms: Vec<String> = (0..20_000).map(|val| val.to_string()).collect();\n        let mut hash_map: ArenaHashMap = ArenaHashMap::default();\n        for term in terms.iter() {\n            hash_map.mutate_or_create(term.as_bytes(), |_opt_val: Option<u32>| 5u32);\n        }\n        let mut terms_back: Vec<String> = hash_map\n            .iter()\n            .map(|(bytes, _)| String::from_utf8(bytes.to_vec()).unwrap())\n            .collect();\n        terms_back.sort();\n        terms.sort();\n\n        for pos in 0..terms.len() {\n            assert_eq!(terms[pos], terms_back[pos]);\n        }\n    }\n}\n"
  },
  {
    "path": "stacker/src/expull.rs",
    "content": "use std::mem;\n\nuse common::serialize_vint_u32;\n\nuse crate::fastcpy::fast_short_slice_copy;\nuse crate::{Addr, MemoryArena};\n\nconst FIRST_BLOCK_NUM: u32 = 2;\n\n/// An exponential unrolled link.\n///\n/// The use case is as follows. Tantivy's indexer conceptually acts like a\n/// `HashMap<Term, Vec<u32>>`. As we come across a given term in document\n/// `D`, we lookup the term in the map and append the document id to its vector.\n///\n/// The vector is then only read when it is serialized.\n///\n/// The `ExpUnrolledLinkedList` offers a more efficient solution to this\n/// problem.\n///\n/// It combines the idea of the unrolled linked list and tries to address the\n/// problem of selecting an adequate block size using a strategy similar to\n/// that of the `Vec` amortized resize strategy.\n///\n/// Data is stored in a linked list of blocks. The first block has a size of `8`\n/// and each block has a length of twice that of the previous block up to\n/// `MAX_BLOCK_LEN = 1<<15`.\n///\n/// This strategy is a good trade off to handle numerous very rare terms\n/// and avoid wasting half of the memory for very frequent terms.\n#[derive(Debug, Clone, Copy)]\npub struct ExpUnrolledLinkedList {\n    // u16, since the max size of each block is (1<<next_cap_pow_2)\n    // Limited to 15, so we don't overflow remaining_cap.\n    remaining_cap: u16,\n    // Tracks the number of blocks allocated: block_num - FIRST_BLOCK_NUM\n    block_num: u32,\n    head: Addr,\n    tail: Addr,\n}\n\nimpl Default for ExpUnrolledLinkedList {\n    fn default() -> Self {\n        Self {\n            // 0 to trigger an initial allocation. Init with MemoryArena would be better.\n            remaining_cap: 0,\n            block_num: FIRST_BLOCK_NUM,\n            head: Addr::null_pointer(),\n            tail: Addr::null_pointer(),\n        }\n    }\n}\n\npub struct ExpUnrolledLinkedListWriter<'a> {\n    eull: &'a mut ExpUnrolledLinkedList,\n    arena: &'a mut MemoryArena,\n}\n\n#[inline]\nfn ensure_capacity<'a>(\n    eull: &'a mut ExpUnrolledLinkedList,\n    arena: &'a mut MemoryArena,\n    allocate: u32,\n) {\n    let new_block_addr: Addr = arena.allocate_space(allocate as usize + mem::size_of::<Addr>());\n    // Check first write\n    if eull.head.is_null() {\n        eull.head = new_block_addr;\n    } else {\n        arena.write_at(eull.tail, new_block_addr);\n    }\n\n    eull.tail = new_block_addr;\n    eull.remaining_cap = allocate as u16;\n}\n\nimpl ExpUnrolledLinkedListWriter<'_> {\n    #[inline]\n    pub fn write_u32_vint(&mut self, val: u32) {\n        let mut buf = [0u8; 8];\n        let data = serialize_vint_u32(val, &mut buf);\n        self.extend_from_slice(data);\n    }\n\n    #[inline]\n    pub fn extend_from_slice(&mut self, mut buf: &[u8]) {\n        while !buf.is_empty() {\n            let add_len: usize;\n            {\n                if self.eull.remaining_cap == 0 {\n                    // Double the next cap\n                    self.eull.increment_num_blocks();\n                    let block_size = get_block_size(self.eull.block_num);\n                    ensure_capacity(self.eull, self.arena, block_size as u32);\n                }\n\n                let output_buf = self\n                    .arena\n                    .slice_mut(self.eull.tail, self.eull.remaining_cap as usize);\n                add_len = buf.len().min(output_buf.len());\n                let output_buf = &mut output_buf[..add_len];\n                let buf = &buf[..add_len];\n\n                fast_short_slice_copy(buf, output_buf);\n            }\n            self.eull.remaining_cap -= add_len as u16;\n            self.eull.tail = self.eull.tail.offset(add_len as u32);\n            buf = &buf[add_len..];\n        }\n    }\n}\n\n// The block size is 2^block_num, but max 2^15 = 32KB\n// Initial size is 8 bytes (2^3), for the first block => block_num == 2\n// Block size caps at 32KB (2^15) regardless of how high block_num goes\n#[inline]\nfn get_block_size(block_num: u32) -> u16 {\n    // Cap at 15 to prevent block sizes > 32KB\n    // block_num can now be much larger than 15, but block size maxes out\n    let exp: u32 = block_num.min(15u32);\n    (1u32 << exp) as u16\n}\n\nimpl ExpUnrolledLinkedList {\n    #[inline(always)]\n    pub fn increment_num_blocks(&mut self) {\n        // Add overflow check as a safety measure\n        // With u32, we can handle up to ~4 billion blocks before overflow\n        // At 32KB per block (max size), that's 128 TB of data\n        self.block_num = self\n            .block_num\n            .checked_add(1)\n            .expect(\"ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks\");\n    }\n\n    #[inline]\n    pub fn writer<'a>(&'a mut self, arena: &'a mut MemoryArena) -> ExpUnrolledLinkedListWriter<'a> {\n        ExpUnrolledLinkedListWriter { eull: self, arena }\n    }\n\n    pub fn read_to_end(&self, arena: &MemoryArena, output: &mut Vec<u8>) {\n        let mut addr = self.head;\n        if addr.is_null() {\n            return;\n        }\n\n        // Calculate last block length with bounds checking to prevent underflow\n        let block_size = get_block_size(self.block_num) as usize;\n        let last_block_len = block_size.saturating_sub(self.remaining_cap as usize);\n\n        // Safety check: if remaining_cap > block_size, the metadata is corrupted\n        assert!(\n            self.remaining_cap as usize <= block_size,\n            \"ExpUnrolledLinkedList metadata corruption detected: remaining_cap ({}) > block_size \\\n             ({}). This indicates a serious bug, please report! (block_num={}, head={:?}, \\\n             tail={:?})\",\n            self.remaining_cap,\n            block_size,\n            self.block_num,\n            self.head,\n            self.tail\n        );\n\n        // Full Blocks (iterate through all blocks except the last one)\n        // Note: Blocks are numbered starting from FIRST_BLOCK_NUM+1 (=3) after first allocation\n        for block_num in FIRST_BLOCK_NUM + 1..self.block_num {\n            let cap = get_block_size(block_num) as usize;\n            let data = arena.slice(addr, cap);\n            output.extend_from_slice(data);\n            addr = arena.read(addr.offset(cap as u32));\n        }\n        // Last Block\n        let data = arena.slice(addr, last_block_len);\n        output.extend_from_slice(data);\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use common::{read_u32_vint, write_u32_vint};\n\n    use super::*;\n\n    #[test]\n    fn test_eull_empty() {\n        let arena = MemoryArena::default();\n        let stack = ExpUnrolledLinkedList::default();\n        {\n            let mut buffer = Vec::new();\n            stack.read_to_end(&arena, &mut buffer);\n            assert_eq!(&buffer[..], &[] as &[u8]);\n        }\n    }\n\n    #[test]\n    fn test_eull1() {\n        let mut arena = MemoryArena::default();\n        let mut stack = ExpUnrolledLinkedList::default();\n        stack.writer(&mut arena).extend_from_slice(&[1u8]);\n        stack.writer(&mut arena).extend_from_slice(&[2u8]);\n        stack.writer(&mut arena).extend_from_slice(&[3u8, 4u8]);\n        stack.writer(&mut arena).extend_from_slice(&[5u8]);\n        {\n            let mut buffer = Vec::new();\n            stack.read_to_end(&arena, &mut buffer);\n            assert_eq!(&buffer[..], &[1u8, 2u8, 3u8, 4u8, 5u8]);\n        }\n    }\n\n    #[test]\n    fn test_eull_vint1() {\n        let mut arena = MemoryArena::default();\n        let mut stack = ExpUnrolledLinkedList::default();\n        stack.writer(&mut arena).extend_from_slice(&[1u8]);\n        stack.writer(&mut arena).extend_from_slice(&[2u8]);\n        stack.writer(&mut arena).extend_from_slice(&[3u8, 4u8]);\n        stack.writer(&mut arena).extend_from_slice(&[5u8]);\n        {\n            let mut buffer = Vec::new();\n            stack.read_to_end(&arena, &mut buffer);\n            assert_eq!(&buffer[..], &[1u8, 2u8, 3u8, 4u8, 5u8]);\n        }\n    }\n\n    #[test]\n    fn test_eull_first_write_extends_cap() {\n        let mut arena = MemoryArena::default();\n        let mut stack = ExpUnrolledLinkedList::default();\n        stack\n            .writer(&mut arena)\n            .extend_from_slice(&[1u8, 2, 3, 4, 5, 6, 7, 8, 9]);\n        {\n            let mut buffer = Vec::new();\n            stack.read_to_end(&arena, &mut buffer);\n            assert_eq!(&buffer[..], &[1u8, 2, 3, 4, 5, 6, 7, 8, 9]);\n        }\n    }\n\n    #[test]\n    fn test_eull_long() {\n        let mut arena = MemoryArena::default();\n        let mut eull = ExpUnrolledLinkedList::default();\n        let data: Vec<u32> = (0..100).collect();\n        for &el in &data {\n            eull.writer(&mut arena).write_u32_vint(el);\n        }\n        let mut buffer = Vec::new();\n        eull.read_to_end(&arena, &mut buffer);\n        let mut result = vec![];\n        let mut remaining = &buffer[..];\n        while !remaining.is_empty() {\n            result.push(read_u32_vint(&mut remaining));\n        }\n        assert_eq!(&result[..], &data[..]);\n    }\n\n    #[test]\n    fn test_eull_limit() {\n        let mut eull = ExpUnrolledLinkedList::default();\n        for _ in 0..100 {\n            eull.increment_num_blocks();\n        }\n        assert_eq!(get_block_size(eull.block_num), 1 << 15);\n    }\n\n    #[test]\n    fn test_eull_interlaced() {\n        let mut arena = MemoryArena::default();\n        let mut stack = ExpUnrolledLinkedList::default();\n        let mut stack2 = ExpUnrolledLinkedList::default();\n\n        let mut vec1: Vec<u8> = vec![];\n        let mut vec2: Vec<u8> = vec![];\n\n        for i in 0..9 {\n            stack.writer(&mut arena).write_u32_vint(i);\n            assert!(write_u32_vint(i, &mut vec1).is_ok());\n            if i % 2 == 0 {\n                stack2.writer(&mut arena).write_u32_vint(i);\n                assert!(write_u32_vint(i, &mut vec2).is_ok());\n            }\n        }\n        let mut res1 = vec![];\n        let mut res2 = vec![];\n        stack.read_to_end(&arena, &mut res1);\n        stack2.read_to_end(&arena, &mut res2);\n        assert_eq!(&vec1[..], &res1[..]);\n        assert_eq!(&vec2[..], &res2[..]);\n    }\n\n    // Tests for u32 block_num fix (issue with large arrays)\n\n    #[test]\n    fn test_block_num_exceeds_u16_max() {\n        // Test that we can handle more than 65,535 blocks (old u16 limit)\n        let mut eull = ExpUnrolledLinkedList::default();\n\n        // Simulate allocating 70,000 blocks (exceeds u16::MAX of 65,535)\n        for _ in 0..70_000 {\n            eull.increment_num_blocks();\n        }\n\n        // Verify block_num is correct\n        assert_eq!(eull.block_num, FIRST_BLOCK_NUM + 70_000);\n\n        // Verify we can still get block size (should be capped at 32KB)\n        let block_size = get_block_size(eull.block_num);\n        assert_eq!(block_size, 1 << 15); // 32KB max\n    }\n\n    #[test]\n    #[allow(clippy::needless_range_loop)]\n    fn test_large_dataset_simulation() {\n        // Simulate the scenario: large arrays requiring many blocks\n        // We write enough data to require thousands of blocks\n        let mut arena = MemoryArena::default();\n        let mut eull = ExpUnrolledLinkedList::default();\n\n        // Write 100 MB of data (this will require ~3,200 blocks at 32KB each)\n        // This is enough to validate the system works with large datasets\n        // but not so much that the test is slow\n        let bytes_per_write = 10_000;\n        let num_writes = 10_000; // 10k * 10k = 100 MB\n\n        let data: Vec<u8> = (0..bytes_per_write).map(|i| (i % 256) as u8).collect();\n        for _ in 0..num_writes {\n            eull.writer(&mut arena).extend_from_slice(&data);\n        }\n\n        // Verify we allocated many blocks (should be in the thousands)\n        assert!(\n            eull.block_num > 1000,\n            \"block_num ({}) should be > 1000 for this much data\",\n            eull.block_num\n        );\n\n        // Verify we can read back correctly\n        let mut buffer = Vec::new();\n        eull.read_to_end(&arena, &mut buffer);\n        assert_eq!(buffer.len(), bytes_per_write * num_writes);\n\n        // Verify data integrity on a sample\n        for i in 0..bytes_per_write {\n            assert_eq!(buffer[i], (i % 256) as u8);\n        }\n    }\n\n    #[test]\n    fn test_get_block_size_with_large_block_num() {\n        // Test that get_block_size handles large u32 values correctly\n\n        // Small block numbers (under 15)\n        assert_eq!(get_block_size(2), 4); // 2^2 = 4\n        assert_eq!(get_block_size(3), 8); // 2^3 = 8\n        assert_eq!(get_block_size(10), 1024); // 2^10 = 1KB\n\n        // At the cap (15)\n        assert_eq!(get_block_size(15), 32768); // 2^15 = 32KB\n\n        // Beyond the cap (should stay at 32KB)\n        assert_eq!(get_block_size(16), 32768);\n        assert_eq!(get_block_size(100), 32768);\n        assert_eq!(get_block_size(65_536), 32768); // Old u16::MAX + 1\n        assert_eq!(get_block_size(100_000), 32768);\n        assert_eq!(get_block_size(1_000_000), 32768);\n    }\n\n    #[test]\n    fn test_increment_blocks_near_u16_boundary() {\n        // Test incrementing around the old u16::MAX boundary\n        let mut eull = ExpUnrolledLinkedList::default();\n\n        // Set to just before old limit\n        for _ in 0..65_533 {\n            eull.increment_num_blocks();\n        }\n        assert_eq!(eull.block_num, FIRST_BLOCK_NUM + 65_533);\n\n        // Cross the old u16::MAX boundary (this would have overflowed before)\n        eull.increment_num_blocks(); // 65,534\n        eull.increment_num_blocks(); // 65,535 (old max)\n        eull.increment_num_blocks(); // 65,536 (would overflow u16)\n        eull.increment_num_blocks(); // 65,537\n\n        // Verify we're past the old limit\n        assert_eq!(eull.block_num, FIRST_BLOCK_NUM + 65_537);\n    }\n\n    #[test]\n    fn test_write_and_read_with_many_blocks() {\n        // Test that write/read works correctly with many blocks\n        let mut arena = MemoryArena::default();\n        let mut eull = ExpUnrolledLinkedList::default();\n\n        // Write data that will span many blocks\n        let test_data: Vec<u8> = (0..50_000).map(|i| (i % 256) as u8).collect();\n        eull.writer(&mut arena).extend_from_slice(&test_data);\n\n        // Read it back\n        let mut buffer = Vec::new();\n        eull.read_to_end(&arena, &mut buffer);\n\n        // Verify data integrity\n        assert_eq!(buffer.len(), test_data.len());\n        assert_eq!(&buffer[..], &test_data[..]);\n    }\n\n    #[test]\n    fn test_multiple_eull_with_large_block_counts() {\n        // Test multiple ExpUnrolledLinkedLists with high block counts\n        // (simulates parallel columnar writes)\n        let mut arena = MemoryArena::default();\n        let mut eull1 = ExpUnrolledLinkedList::default();\n        let mut eull2 = ExpUnrolledLinkedList::default();\n\n        // Write different data to each\n        for i in 0..10_000u32 {\n            eull1.writer(&mut arena).write_u32_vint(i);\n            eull2.writer(&mut arena).write_u32_vint(i * 2);\n        }\n\n        // Read back and verify\n        let mut buf1 = Vec::new();\n        let mut buf2 = Vec::new();\n        eull1.read_to_end(&arena, &mut buf1);\n        eull2.read_to_end(&arena, &mut buf2);\n\n        // Deserialize and check\n        let mut cursor1 = &buf1[..];\n        let mut cursor2 = &buf2[..];\n        for i in 0..10_000u32 {\n            assert_eq!(read_u32_vint(&mut cursor1), i);\n            assert_eq!(read_u32_vint(&mut cursor2), i * 2);\n        }\n    }\n\n    #[test]\n    fn test_block_size_stays_capped() {\n        // Verify that even with massive block numbers, size stays at 32KB\n        let mut eull = ExpUnrolledLinkedList::default();\n\n        // Increment to a very large number\n        for _ in 0..200_000 {\n            eull.increment_num_blocks();\n        }\n\n        let block_size = get_block_size(eull.block_num);\n        assert_eq!(block_size, 32768, \"Block size should be capped at 32KB\");\n    }\n\n    #[test]\n    #[should_panic(expected = \"ExpUnrolledLinkedList block count overflow\")]\n    fn test_increment_overflow_protection() {\n        // Test that we panic gracefully if we somehow hit u32::MAX\n        // This is extremely unlikely in practice (would require 128TB of data)\n        let mut eull = ExpUnrolledLinkedList {\n            block_num: u32::MAX,\n            ..Default::default()\n        };\n\n        // This should panic with our custom error message\n        eull.increment_num_blocks();\n    }\n}\n\n#[cfg(all(test, feature = \"unstable\"))]\nmod bench {\n    use std::iter;\n\n    use test::Bencher;\n\n    use super::super::MemoryArena;\n    use super::ExpUnrolledLinkedList;\n\n    const NUM_STACK: usize = 10_000;\n    const STACK_SIZE: u32 = 1000;\n\n    #[bench]\n    fn bench_push_vec(bench: &mut Bencher) {\n        bench.iter(|| {\n            let mut vecs = Vec::with_capacity(100);\n            for _ in 0..NUM_STACK {\n                vecs.push(Vec::new());\n            }\n            for s in 0..NUM_STACK {\n                for i in 0u32..STACK_SIZE {\n                    let t = s * 392017 % NUM_STACK;\n                    vecs[t].push(i);\n                }\n            }\n        });\n    }\n\n    #[bench]\n    fn bench_push_stack(bench: &mut Bencher) {\n        bench.iter(|| {\n            let mut arena = MemoryArena::default();\n            let mut stacks: Vec<ExpUnrolledLinkedList> =\n                iter::repeat_with(ExpUnrolledLinkedList::default)\n                    .take(NUM_STACK)\n                    .collect();\n            for s in 0..NUM_STACK {\n                for i in 0u32..STACK_SIZE {\n                    let t = s * 392017 % NUM_STACK;\n                    stacks[t]\n                        .writer(&mut arena)\n                        .extend_from_slice(&i.to_ne_bytes());\n                }\n            }\n        });\n    }\n}\n"
  },
  {
    "path": "stacker/src/fastcmp.rs",
    "content": "/// fastcmp employs a trick to speed up the comparison of two slices of bytes.\n/// It's also possible to inline compared to the memcmp call.\n///\n/// E.g. Comparing equality of slice length 7 in two steps, by comparing two 4 byte slices\n/// unconditionally instead comparing the remaining 3 bytes if the first comparison was equal.\n/// [1, 2, 3, 4, 5, 6, 7]\n/// [1, 2, 3, 4]\n///          [4, 5, 6, 7]\n///\n/// This method uses the XMM register for bytes slices bigger than 16, else regular registers.\n#[inline]\npub fn fast_short_slice_compare(left: &[u8], right: &[u8]) -> bool {\n    let len = left.len();\n    if len != right.len() {\n        return false;\n    }\n\n    // This could be less equals, but to make the job a little bit easier for the branch predictor\n    // we put the length 8 into the bigger group (8-16 bytes), that compares two u64\n    // assuming that range 8-16 are more common than 4-7\n\n    // This weird branching is done on purpose to get the best assembly.\n    // if len< 4 {\n    // ..\n    // if len < 8\n    // will cause assembly inlined instead of jumps\n    if len < 8 {\n        if len >= 4 {\n            return double_check_trick::<4>(left, right);\n        } else {\n            return short_compare(left, right);\n        }\n    }\n\n    if len > 16 {\n        return fast_nbyte_slice_compare::<16>(left, right);\n    }\n\n    double_check_trick::<8>(left, right)\n}\n\n// Note: The straightforward left.chunks_exact(SIZE).zip(right.chunks_exact(SIZE)) produces slower\n// assembly\n#[inline]\npub fn fast_nbyte_slice_compare<const SIZE: usize>(left: &[u8], right: &[u8]) -> bool {\n    let last = left.len() - left.len() % SIZE;\n    let mut i = 0;\n    loop {\n        if unsafe { left.get_unchecked(i..i + SIZE) != right.get_unchecked(i..i + SIZE) } {\n            return false;\n        }\n        i += SIZE;\n        if i >= last {\n            break;\n        }\n    }\n    unsafe { left.get_unchecked(left.len() - SIZE..) == right.get_unchecked(right.len() - SIZE..) }\n}\n\n#[inline(always)]\nfn short_compare(left: &[u8], right: &[u8]) -> bool {\n    for (l, r) in left.iter().zip(right) {\n        if l != r {\n            return false;\n        }\n    }\n    true\n}\n\n#[inline(always)]\nfn double_check_trick<const SIZE: usize>(left: &[u8], right: &[u8]) -> bool {\n    left[0..SIZE] == right[0..SIZE] && left[left.len() - SIZE..] == right[right.len() - SIZE..]\n}\n\n#[cfg(test)]\nmod tests {\n    use proptest::prelude::*;\n\n    use super::*;\n\n    #[test]\n    fn test_slice_compare_bytes_len_8() {\n        let a = &[1, 2, 3, 4, 5, 6, 7, 8];\n        let b = &[1, 2, 3, 4, 5, 6, 7, 8];\n        let c = &[1, 2, 3, 4, 5, 6, 7, 7];\n\n        assert!(fast_short_slice_compare(a, b));\n        assert!(!fast_short_slice_compare(a, c));\n    }\n\n    #[test]\n    fn test_slice_compare_bytes_len_9() {\n        let a = &[1, 2, 3, 4, 5, 6, 7, 8, 9];\n        let b = &[1, 2, 3, 4, 5, 6, 7, 8, 9];\n        let c = &[0, 2, 3, 4, 5, 6, 7, 8, 9];\n\n        assert!(fast_short_slice_compare(a, b));\n        assert!(!fast_short_slice_compare(a, c));\n    }\n\n    #[test]\n    fn test_slice_compare_bytes_len_16() {\n        let a = &[1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8];\n        let b = &[1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8];\n        let c = &[1, 2, 3, 4, 5, 6, 7, 7, 1, 2, 3, 4, 5, 6, 7, 8];\n\n        assert!(fast_short_slice_compare(a, b));\n        assert!(!fast_short_slice_compare(a, c));\n    }\n\n    #[test]\n    fn test_slice_compare_bytes_short() {\n        let a = &[1, 2, 3, 4];\n        let b = &[1, 2, 3, 4];\n\n        assert!(fast_short_slice_compare(a, b));\n\n        let a = &[1, 2, 3];\n        let b = &[1, 2, 3];\n\n        assert!(fast_short_slice_compare(a, b));\n\n        let a = &[1, 2];\n        let b = &[1, 2];\n\n        assert!(fast_short_slice_compare(a, b));\n    }\n\n    proptest! {\n        #[test]\n        fn test_fast_short_slice_compare(left in prop::collection::vec(any::<u8>(), 0..100),\n                                          right in prop::collection::vec(any::<u8>(), 0..100)) {\n            let result = fast_short_slice_compare(&left, &right);\n            let expected = left == right;\n            prop_assert_eq!(result, expected, \"left: {:?}, right: {:?}\", left, right);\n        }\n\n        #[test]\n        fn test_fast_short_slice_compare_equal(left in prop::collection::vec(any::<u8>(), 0..100),\n                                          ) {\n            let result = fast_short_slice_compare(&left, &left);\n            let expected = left == left;\n            prop_assert_eq!(result, expected, \"left: {:?}, right: {:?}\", left, left);\n        }\n\n    }\n}\n"
  },
  {
    "path": "stacker/src/fastcpy.rs",
    "content": "/// Optimized copy for small sizes. All bounds checks are elided.\n/// Avoids call to memcpy\n/// Applies unbranched copy trick for sizes 8, 16, 32\n///\n/// src and dst must be num_bytes long.\n#[inline]\npub fn fast_short_slice_copy(src: &[u8], dst: &mut [u8]) {\n    #[inline(never)]\n    #[cold]\n    #[track_caller]\n    fn len_mismatch_fail(dst_len: usize, src_len: usize) -> ! {\n        panic!(\n            \"source slice length ({}) does not match destination slice length ({})\",\n            src_len, dst_len,\n        );\n    }\n\n    if src.len() != dst.len() {\n        len_mismatch_fail(src.len(), dst.len());\n    }\n    let len = src.len();\n\n    if src.is_empty() {\n        return;\n    }\n\n    if len < 4 {\n        short_copy(src, dst);\n        return;\n    }\n\n    if len < 8 {\n        double_copy_trick::<4>(src, dst);\n        return;\n    }\n\n    if len <= 16 {\n        double_copy_trick::<8>(src, dst);\n        return;\n    }\n\n    if len <= 32 {\n        double_copy_trick::<16>(src, dst);\n        return;\n    }\n\n    // The code will use the vmovdqu instruction to copy 32 bytes at a time.\n    #[cfg(target_feature = \"avx\")]\n    {\n        if len <= 64 {\n            double_copy_trick::<32>(src, dst);\n            return;\n        }\n    }\n\n    // For larger sizes we use the default, which calls memcpy\n    // memcpy does some virtual memory tricks to copy large chunks of memory.\n    //\n    // The theory should be that the checks above don't cost much relative to the copy call for\n    // larger copies.\n    // The bounds checks in `copy_from_slice` are elided.\n    dst.copy_from_slice(src);\n}\n\n#[inline(always)]\nfn short_copy(src: &[u8], dst: &mut [u8]) {\n    debug_assert_ne!(src.len(), 0);\n    debug_assert_eq!(src.len(), dst.len());\n    let len = src.len();\n\n    // length 1-3\n    dst[0] = src[0];\n    if len >= 2 {\n        double_copy_trick::<2>(src, dst);\n    }\n}\n\n#[inline(always)]\nfn double_copy_trick<const SIZE: usize>(src: &[u8], dst: &mut [u8]) {\n    debug_assert!(src.len() >= SIZE);\n    debug_assert!(dst.len() >= SIZE);\n    dst[0..SIZE].copy_from_slice(&src[0..SIZE]);\n    dst[src.len() - SIZE..].copy_from_slice(&src[src.len() - SIZE..]);\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    fn copy_test<const SIZE: usize>() {\n        let src: Vec<u8> = (0..SIZE as u8).collect();\n        let mut dst = [0u8; SIZE];\n        fast_short_slice_copy(&src, &mut dst);\n        assert_eq!(src, dst);\n    }\n\n    #[test]\n    fn copy_test_n() {\n        copy_test::<1>();\n        copy_test::<2>();\n        copy_test::<3>();\n        copy_test::<4>();\n        copy_test::<5>();\n        copy_test::<6>();\n        copy_test::<7>();\n        copy_test::<8>();\n        copy_test::<9>();\n        copy_test::<10>();\n        copy_test::<11>();\n        copy_test::<31>();\n        copy_test::<32>();\n        copy_test::<33>();\n        copy_test::<47>();\n        copy_test::<48>();\n        copy_test::<49>();\n    }\n}\n"
  },
  {
    "path": "stacker/src/lib.rs",
    "content": "#![cfg_attr(all(feature = \"unstable\", test), feature(test))]\n\n#[cfg(all(test, feature = \"unstable\"))]\nextern crate test;\n\nmod arena_hashmap;\nmod expull;\nmod fastcmp;\nmod fastcpy;\nmod memory_arena;\nmod shared_arena_hashmap;\n\npub use self::arena_hashmap::ArenaHashMap;\npub use self::expull::ExpUnrolledLinkedList;\npub use self::memory_arena::{Addr, MemoryArena};\npub use self::shared_arena_hashmap::{SharedArenaHashMap, compute_table_memory_size};\n\n/// When adding an element in a `ArenaHashMap`, we get a unique id associated to the given key.\npub type UnorderedId = u32;\n"
  },
  {
    "path": "stacker/src/memory_arena.rs",
    "content": "//! 32-bits Memory arena for types implementing `Copy`.\n//! This Memory arena has been implemented to fit the use of tantivy's indexer\n//! and has *twisted specifications*.\n//!\n//! - It works on stable rust.\n//! - One can get an accurate figure of the memory usage of the arena.\n//! - Allocation are very cheap.\n//! - Allocation happening consecutively are very likely to have great locality.\n//! - Addresses (`Addr`) are 32bits.\n//! - Dropping the whole `MemoryArena` is cheap.\n//!\n//! # Limitations\n//!\n//! - Your object shall not implement `Drop`.\n//! - `Addr` to the `Arena` are 32-bits. The maximum capacity of the arena is 4GB. *(Tantivy's\n//!   indexer uses one arena per indexing thread.)*\n//! - The arena only works for objects much smaller than  `1MB`. Allocating more than `1MB` at a\n//!   time will result in a panic, and allocating a lot of large object (> 500KB) will result in a\n//!   fragmentation.\n//! - Your objects are store in an unaligned fashion. For this reason, the API does not let you\n//!   access them as references.\n//!\n//! Instead, you store and access your data via `.write(...)` and `.read(...)`, which under the hood\n//! stores your object using `ptr::write_unaligned` and `ptr::read_unaligned`.\nuse std::{mem, ptr};\n\nconst NUM_BITS_PAGE_ADDR: usize = 20;\nconst PAGE_SIZE: usize = 1 << NUM_BITS_PAGE_ADDR; // pages are 1 MB large\n\n/// Represents a pointer into the `MemoryArena`\n/// .\n/// Pointer are 32-bits and are split into\n/// two parts.\n///\n/// The first 12 bits represent the id of a\n/// page of memory.\n///\n/// The last 20 bits are an address within this page of memory.\n#[derive(Copy, Clone, Debug)]\npub struct Addr(u32);\n\nimpl Addr {\n    /// Creates a null pointer.\n    #[inline]\n    pub fn null_pointer() -> Addr {\n        Addr(u32::MAX)\n    }\n\n    /// Returns the `Addr` object for `addr + offset`\n    #[inline]\n    pub fn offset(self, offset: u32) -> Addr {\n        Addr(self.0.wrapping_add(offset))\n    }\n\n    #[inline]\n    fn new(page_id: usize, local_addr: usize) -> Addr {\n        Addr(((page_id << NUM_BITS_PAGE_ADDR) | local_addr) as u32)\n    }\n\n    #[inline]\n    fn page_id(self) -> usize {\n        (self.0 as usize) >> NUM_BITS_PAGE_ADDR\n    }\n\n    #[inline]\n    fn page_local_addr(self) -> usize {\n        (self.0 as usize) & (PAGE_SIZE - 1)\n    }\n\n    /// Returns true if and only if the `Addr` is null.\n    #[inline]\n    pub fn is_null(self) -> bool {\n        self.0 == u32::MAX\n    }\n}\n\n#[inline(always)]\npub fn store<Item: Copy + 'static>(dest: &mut [u8], val: Item) {\n    debug_assert_eq!(dest.len(), std::mem::size_of::<Item>());\n    unsafe {\n        ptr::write_unaligned(dest.as_mut_ptr() as *mut Item, val);\n    }\n}\n\n#[inline]\npub fn load<Item: Copy + 'static>(data: &[u8]) -> Item {\n    debug_assert_eq!(data.len(), std::mem::size_of::<Item>());\n    unsafe { ptr::read_unaligned(data.as_ptr() as *const Item) }\n}\n\n/// The `MemoryArena`\npub struct MemoryArena {\n    pages: Vec<Page>,\n}\n\nimpl Default for MemoryArena {\n    fn default() -> MemoryArena {\n        let first_page = Page::new(0);\n        MemoryArena {\n            pages: vec![first_page],\n        }\n    }\n}\n\nimpl MemoryArena {\n    /// Returns an estimate in number of bytes\n    /// of resident memory consumed by the `MemoryArena`.\n    ///\n    /// Internally, it counts a number of `1MB` pages\n    /// and therefore delivers an upperbound.\n    pub fn mem_usage(&self) -> usize {\n        self.pages.len() * PAGE_SIZE\n    }\n\n    /// Returns the number of bytes allocated in the arena.\n    pub fn len(&self) -> usize {\n        self.pages.len().saturating_sub(1) * PAGE_SIZE + self.pages.last().unwrap().len\n    }\n\n    pub fn is_empty(&self) -> bool {\n        self.len() == 0\n    }\n\n    #[inline]\n    pub fn write_at<Item: Copy + 'static>(&mut self, addr: Addr, val: Item) {\n        let dest = self.slice_mut(addr, std::mem::size_of::<Item>());\n        store(dest, val);\n    }\n\n    /// Read an item in the memory arena at the given `address`.\n    ///\n    /// # Panics\n    ///\n    /// If the address is erroneous\n    #[inline]\n    pub fn read<Item: Copy + 'static>(&self, addr: Addr) -> Item {\n        load(self.slice(addr, mem::size_of::<Item>()))\n    }\n    #[inline]\n    fn get_page(&self, page_id: usize) -> &Page {\n        unsafe { self.pages.get_unchecked(page_id) }\n    }\n    #[inline]\n    fn get_page_mut(&mut self, page_id: usize) -> &mut Page {\n        unsafe { self.pages.get_unchecked_mut(page_id) }\n    }\n\n    #[inline]\n    pub fn slice(&self, addr: Addr, len: usize) -> &[u8] {\n        self.get_page(addr.page_id())\n            .slice(addr.page_local_addr(), len)\n    }\n\n    #[inline]\n    pub fn slice_from(&self, addr: Addr) -> &[u8] {\n        self.get_page(addr.page_id())\n            .slice_from(addr.page_local_addr())\n    }\n    #[inline]\n    pub fn slice_from_mut(&mut self, addr: Addr) -> &mut [u8] {\n        self.get_page_mut(addr.page_id())\n            .slice_from_mut(addr.page_local_addr())\n    }\n\n    #[inline]\n    pub fn slice_mut(&mut self, addr: Addr, len: usize) -> &mut [u8] {\n        self.get_page_mut(addr.page_id())\n            .slice_mut(addr.page_local_addr(), len)\n    }\n\n    /// Add a page and allocate len on it.\n    /// Return the address\n    fn add_page(&mut self, len: usize) -> Addr {\n        let new_page_id = self.pages.len();\n        let mut page = Page::new(new_page_id);\n        page.len = len;\n        self.pages.push(page);\n        Addr::new(new_page_id, 0)\n    }\n\n    /// Allocates `len` bytes and returns the allocated address.\n    #[inline]\n    pub fn allocate_space(&mut self, len: usize) -> Addr {\n        let page_id = self.pages.len() - 1;\n        if let Some(addr) = self.get_page_mut(page_id).allocate_space(len) {\n            return addr;\n        }\n        self.add_page(len)\n    }\n}\n\nstruct Page {\n    page_id: usize,\n    len: usize,\n    data: Box<[u8; PAGE_SIZE]>,\n}\n\nimpl Page {\n    fn new(page_id: usize) -> Page {\n        // We use 32-bits addresses.\n        // - 20 bits for the in-page addressing\n        // - 12 bits for the page id.\n        // This limits us to 2^12 - 1=4095 for the page id.\n        assert!(page_id < 4096);\n        Page {\n            page_id,\n            len: 0,\n            data: vec![0u8; PAGE_SIZE].into_boxed_slice().try_into().unwrap(),\n        }\n    }\n\n    #[inline]\n    fn is_available(&self, len: usize) -> bool {\n        len + self.len <= PAGE_SIZE\n    }\n\n    #[inline]\n    fn slice(&self, local_addr: usize, len: usize) -> &[u8] {\n        let data = &self.slice_from(local_addr);\n        unsafe { data.get_unchecked(..len) }\n    }\n\n    #[inline]\n    fn slice_from(&self, local_addr: usize) -> &[u8] {\n        &self.data[local_addr..]\n    }\n    #[inline]\n    fn slice_from_mut(&mut self, local_addr: usize) -> &mut [u8] {\n        &mut self.data[local_addr..]\n    }\n\n    #[inline]\n    fn slice_mut(&mut self, local_addr: usize, len: usize) -> &mut [u8] {\n        let data = &mut self.data[local_addr..];\n        unsafe { data.get_unchecked_mut(..len) }\n    }\n\n    #[inline]\n    fn allocate_space(&mut self, len: usize) -> Option<Addr> {\n        if self.is_available(len) {\n            let addr = Addr::new(self.page_id, self.len);\n            self.len += len;\n            Some(addr)\n        } else {\n            None\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use super::MemoryArena;\n    use crate::memory_arena::PAGE_SIZE;\n\n    #[test]\n    fn test_arena_allocate_slice() {\n        let mut arena = MemoryArena::default();\n        let a = b\"hello\";\n        let b = b\"happy tax payer\";\n\n        let addr_a = arena.allocate_space(a.len());\n        arena.slice_mut(addr_a, a.len()).copy_from_slice(a);\n\n        let addr_b = arena.allocate_space(b.len());\n        arena.slice_mut(addr_b, b.len()).copy_from_slice(b);\n\n        assert_eq!(arena.slice(addr_a, a.len()), a);\n        assert_eq!(arena.slice(addr_b, b.len()), b);\n    }\n\n    #[test]\n    fn test_arena_allocate_end_of_page() {\n        let mut arena = MemoryArena::default();\n\n        // A big block\n        let len_a = PAGE_SIZE - 2;\n        let addr_a = arena.allocate_space(len_a);\n        *arena.slice_mut(addr_a, len_a).last_mut().unwrap() = 1;\n\n        // Single bytes\n        let addr_b = arena.allocate_space(1);\n        arena.slice_mut(addr_b, 1)[0] = 2;\n\n        let addr_c = arena.allocate_space(1);\n        arena.slice_mut(addr_c, 1)[0] = 3;\n\n        let addr_d = arena.allocate_space(1);\n        arena.slice_mut(addr_d, 1)[0] = 4;\n\n        assert_eq!(arena.slice(addr_a, len_a)[len_a - 1], 1);\n        assert_eq!(arena.slice(addr_b, 1)[0], 2);\n        assert_eq!(arena.slice(addr_c, 1)[0], 3);\n        assert_eq!(arena.slice(addr_d, 1)[0], 4);\n    }\n\n    #[derive(Clone, Copy, Debug, Eq, PartialEq)]\n    struct MyTest {\n        pub a: usize,\n        pub b: u8,\n        pub c: u32,\n    }\n\n    #[test]\n    fn test_store_object() {\n        let mut arena = MemoryArena::default();\n        let a = MyTest {\n            a: 143,\n            b: 21,\n            c: 32,\n        };\n        let b = MyTest {\n            a: 113,\n            b: 221,\n            c: 12,\n        };\n\n        let num_bytes = std::mem::size_of::<MyTest>();\n        let addr_a = arena.allocate_space(num_bytes);\n        arena.write_at(addr_a, a);\n\n        let addr_b = arena.allocate_space(num_bytes);\n        arena.write_at(addr_b, b);\n\n        assert_eq!(arena.read::<MyTest>(addr_a), a);\n        assert_eq!(arena.read::<MyTest>(addr_b), b);\n    }\n}\n"
  },
  {
    "path": "stacker/src/shared_arena_hashmap.rs",
    "content": "use std::iter::{Cloned, Filter};\nuse std::mem;\n\nuse super::{Addr, MemoryArena};\nuse crate::fastcpy::fast_short_slice_copy;\nuse crate::memory_arena::store;\n\n/// Returns the actual memory size in bytes\n/// required to create a table with a given capacity.\n/// required to create a table of size\npub fn compute_table_memory_size(capacity: usize) -> usize {\n    capacity * mem::size_of::<KeyValue>()\n}\n\n#[cfg(not(feature = \"compare_hash_only\"))]\ntype HashType = u32;\n\n#[cfg(feature = \"compare_hash_only\")]\ntype HashType = u64;\n\n/// `KeyValue` is the item stored in the hash table.\n/// The key is actually a `BytesRef` object stored in an external memory arena.\n/// The `value_addr` also points to an address in the memory arena.\n#[derive(Copy, Clone)]\nstruct KeyValue {\n    key_value_addr: Addr,\n    hash: HashType,\n}\n\nimpl Default for KeyValue {\n    fn default() -> Self {\n        KeyValue {\n            key_value_addr: Addr::null_pointer(),\n            hash: 0,\n        }\n    }\n}\n\nimpl KeyValue {\n    #[inline]\n    fn is_empty(&self) -> bool {\n        self.key_value_addr.is_null()\n    }\n    #[inline]\n    fn is_not_empty_ref(&self) -> bool {\n        !self.key_value_addr.is_null()\n    }\n}\n\n/// Customized `HashMap` with `&[u8]` keys\n///\n/// Its main particularity is that rather than storing its\n/// keys in the heap, keys are stored in a memory arena\n/// inline with the values.\n///\n/// The quirky API has the benefit of avoiding\n/// the computation of the hash of the key twice,\n/// or copying the key as long as there is no insert.\n///\n/// SharedArenaHashMap is like ArenaHashMap but gets the memory arena\n/// passed as an argument to the methods.\n/// So one MemoryArena can be shared with multiple SharedArenaHashMap.\npub struct SharedArenaHashMap {\n    table: Vec<KeyValue>,\n    mask: usize,\n    len: usize,\n}\n\nstruct LinearProbing {\n    pos: usize,\n    mask: usize,\n}\n\nimpl LinearProbing {\n    #[inline]\n    fn compute(hash: HashType, mask: usize) -> LinearProbing {\n        LinearProbing {\n            pos: hash as usize,\n            mask,\n        }\n    }\n\n    #[inline]\n    fn next_probe(&mut self) -> usize {\n        // Not saving the masked version removes a dependency.\n        self.pos = self.pos.wrapping_add(1);\n        self.pos & self.mask\n    }\n}\n\ntype IterNonEmpty<'a> = Filter<Cloned<std::slice::Iter<'a, KeyValue>>, fn(&KeyValue) -> bool>;\n\npub struct Iter<'a> {\n    hashmap: &'a SharedArenaHashMap,\n    memory_arena: &'a MemoryArena,\n    inner: IterNonEmpty<'a>,\n}\n\nimpl<'a> Iterator for Iter<'a> {\n    type Item = (&'a [u8], Addr);\n\n    fn next(&mut self) -> Option<Self::Item> {\n        self.inner.next().map(move |kv| {\n            let (key, offset): (&'a [u8], Addr) = self\n                .hashmap\n                .get_key_value(kv.key_value_addr, self.memory_arena);\n            (key, offset)\n        })\n    }\n}\n\n/// Returns the greatest power of two lower or equal to `n`.\n/// Except if n == 0, in that case, return 1.\n///\n/// # Panics if n == 0\nfn compute_previous_power_of_two(n: usize) -> usize {\n    assert!(n > 0);\n    let msb = (63u32 - (n as u64).leading_zeros()) as u8;\n    1 << msb\n}\n\nimpl Default for SharedArenaHashMap {\n    fn default() -> Self {\n        SharedArenaHashMap::with_capacity(4)\n    }\n}\n\nimpl SharedArenaHashMap {\n    pub fn with_capacity(table_size: usize) -> SharedArenaHashMap {\n        let table_size_power_of_2 = compute_previous_power_of_two(table_size);\n        let table = vec![KeyValue::default(); table_size_power_of_2];\n\n        SharedArenaHashMap {\n            table,\n            mask: table_size_power_of_2 - 1,\n            len: 0,\n        }\n    }\n\n    #[inline]\n    #[cfg(not(feature = \"compare_hash_only\"))]\n    fn get_hash(&self, key: &[u8]) -> HashType {\n        murmurhash32::murmurhash2(key)\n    }\n\n    #[inline]\n    #[cfg(feature = \"compare_hash_only\")]\n    fn get_hash(&self, key: &[u8]) -> HashType {\n        /// Since we compare only the hash we need a high quality hash.\n        use std::hash::Hasher;\n        let mut hasher = ahash::AHasher::default();\n        hasher.write(key);\n        hasher.finish() as HashType\n    }\n\n    #[inline]\n    fn probe(&self, hash: HashType) -> LinearProbing {\n        LinearProbing::compute(hash, self.mask)\n    }\n\n    #[inline]\n    pub fn mem_usage(&self) -> usize {\n        self.table.len() * mem::size_of::<KeyValue>()\n    }\n\n    #[inline]\n    fn is_saturated(&self) -> bool {\n        self.table.len() <= self.len * 2\n    }\n\n    #[inline]\n    fn get_key_value<'a>(&'a self, addr: Addr, memory_arena: &'a MemoryArena) -> (&'a [u8], Addr) {\n        let data = memory_arena.slice_from(addr);\n        let key_bytes_len_bytes = unsafe { data.get_unchecked(..2) };\n        let key_bytes_len = u16::from_le_bytes(key_bytes_len_bytes.try_into().unwrap());\n        let key_bytes: &[u8] = unsafe { data.get_unchecked(2..2 + key_bytes_len as usize) };\n        (key_bytes, addr.offset(2 + key_bytes_len as u32))\n    }\n\n    #[inline]\n    #[cfg(not(feature = \"compare_hash_only\"))]\n    fn get_value_addr_if_key_match(\n        &self,\n        target_key: &[u8],\n        addr: Addr,\n        memory_arena: &MemoryArena,\n    ) -> Option<Addr> {\n        use crate::fastcmp::fast_short_slice_compare;\n\n        let (stored_key, value_addr) = self.get_key_value(addr, memory_arena);\n        if fast_short_slice_compare(stored_key, target_key) {\n            Some(value_addr)\n        } else {\n            None\n        }\n    }\n    #[inline]\n    #[cfg(feature = \"compare_hash_only\")]\n    fn get_value_addr_if_key_match(\n        &self,\n        _target_key: &[u8],\n        addr: Addr,\n        memory_arena: &MemoryArena,\n    ) -> Option<Addr> {\n        // For the compare_hash_only feature, it would make sense to store the keys at a different\n        // memory location. Here they will just pollute the cache.\n        let data = memory_arena.slice_from(addr);\n        let key_bytes_len_bytes = &data[..2];\n        let key_bytes_len = u16::from_le_bytes(key_bytes_len_bytes.try_into().unwrap());\n        let value_addr = addr.offset(2 + key_bytes_len as u32);\n\n        Some(value_addr)\n    }\n\n    #[inline]\n    fn set_bucket(&mut self, hash: HashType, key_value_addr: Addr, bucket: usize) {\n        self.len += 1;\n\n        self.table[bucket] = KeyValue {\n            key_value_addr,\n            hash,\n        };\n    }\n\n    #[inline]\n    pub fn is_empty(&self) -> bool {\n        self.len() == 0\n    }\n\n    #[inline]\n    pub fn len(&self) -> usize {\n        self.len\n    }\n\n    #[inline]\n    pub fn iter<'a>(&'a self, memory_arena: &'a MemoryArena) -> Iter<'a> {\n        Iter {\n            inner: self\n                .table\n                .iter()\n                .cloned()\n                .filter(KeyValue::is_not_empty_ref),\n            hashmap: self,\n            memory_arena,\n        }\n    }\n\n    fn resize(&mut self) {\n        let new_len = (self.table.len() * 2).max(1 << 3);\n        let mask = new_len - 1;\n        self.mask = mask;\n        let new_table = vec![KeyValue::default(); new_len];\n        let old_table = mem::replace(&mut self.table, new_table);\n        for key_value in old_table.into_iter().filter(KeyValue::is_not_empty_ref) {\n            let mut probe = LinearProbing::compute(key_value.hash, mask);\n            loop {\n                let bucket = probe.next_probe();\n                if self.table[bucket].is_empty() {\n                    self.table[bucket] = key_value;\n                    break;\n                }\n            }\n        }\n    }\n\n    /// Get a value associated to a key.\n    #[inline]\n    pub fn get<V>(&self, key: &[u8], memory_arena: &MemoryArena) -> Option<V>\n    where V: Copy + 'static {\n        let hash = self.get_hash(key);\n        let mut probe = self.probe(hash);\n        loop {\n            let bucket = probe.next_probe();\n            let kv: KeyValue = self.table[bucket];\n            if kv.is_empty() {\n                return None;\n            } else if kv.hash == hash\n                && let Some(val_addr) =\n                    self.get_value_addr_if_key_match(key, kv.key_value_addr, memory_arena)\n            {\n                let v = memory_arena.read(val_addr);\n                return Some(v);\n            }\n        }\n    }\n\n    /// `update` create a new entry for a given key if it does not exist\n    /// or updates the existing entry.\n    ///\n    /// The actual logic for this update is define in the `updater`\n    /// argument.\n    ///\n    /// If the key is not present, `updater` will receive `None` and\n    /// will be in charge of returning a default value.\n    /// If the key already as an associated value, then it will be passed\n    /// `Some(previous_value)`.\n    ///\n    /// The key will be truncated to u16::MAX bytes.\n    #[inline]\n    pub fn mutate_or_create<V>(\n        &mut self,\n        key: &[u8],\n        memory_arena: &mut MemoryArena,\n        mut updater: impl FnMut(Option<V>) -> V,\n    ) -> V\n    where\n        V: Copy + 'static,\n    {\n        if self.is_saturated() {\n            self.resize();\n        }\n        // Limit the key size to u16::MAX\n        let key = &key[..std::cmp::min(key.len(), u16::MAX as usize)];\n        let hash = self.get_hash(key);\n        let mut probe = self.probe(hash);\n        let mut bucket = probe.next_probe();\n        let mut kv: KeyValue = self.table[bucket];\n        loop {\n            if kv.is_empty() {\n                // The key does not exist yet.\n                let val = updater(None);\n                let num_bytes = std::mem::size_of::<u16>() + key.len() + std::mem::size_of::<V>();\n                let key_addr = memory_arena.allocate_space(num_bytes);\n                {\n                    let data = memory_arena.slice_mut(key_addr, num_bytes);\n                    let key_len_bytes: [u8; 2] = (key.len() as u16).to_le_bytes();\n                    data[..2].copy_from_slice(&key_len_bytes);\n                    let stop = 2 + key.len();\n                    fast_short_slice_copy(key, &mut data[2..stop]);\n                    store(&mut data[stop..], val);\n                }\n\n                self.set_bucket(hash, key_addr, bucket);\n                return val;\n            }\n            if kv.hash == hash\n                && let Some(val_addr) =\n                    self.get_value_addr_if_key_match(key, kv.key_value_addr, memory_arena)\n            {\n                let v = memory_arena.read(val_addr);\n                let new_v = updater(Some(v));\n                memory_arena.write_at(val_addr, new_v);\n                return new_v;\n            }\n            // This allows fetching the next bucket before the loop jmp\n            bucket = probe.next_probe();\n            kv = self.table[bucket];\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n\n    use std::collections::HashMap;\n\n    use super::{SharedArenaHashMap, compute_previous_power_of_two};\n    use crate::MemoryArena;\n\n    #[test]\n    fn test_hash_map() {\n        let mut memory_arena = MemoryArena::default();\n        let mut hash_map: SharedArenaHashMap = SharedArenaHashMap::default();\n        hash_map.mutate_or_create(b\"abc\", &mut memory_arena, |opt_val: Option<u32>| {\n            assert_eq!(opt_val, None);\n            3u32\n        });\n        hash_map.mutate_or_create(b\"abcd\", &mut memory_arena, |opt_val: Option<u32>| {\n            assert_eq!(opt_val, None);\n            4u32\n        });\n        hash_map.mutate_or_create(b\"abc\", &mut memory_arena, |opt_val: Option<u32>| {\n            assert_eq!(opt_val, Some(3u32));\n            5u32\n        });\n        let mut vanilla_hash_map = HashMap::new();\n        let iter_values = hash_map.iter(&memory_arena);\n        for (key, addr) in iter_values {\n            let val: u32 = memory_arena.read(addr);\n            vanilla_hash_map.insert(key.to_owned(), val);\n        }\n        assert_eq!(vanilla_hash_map.len(), 2);\n    }\n\n    #[test]\n    fn test_long_key_truncation() {\n        // Keys longer than u16::MAX are truncated.\n        let mut memory_arena = MemoryArena::default();\n        let mut hash_map: SharedArenaHashMap = SharedArenaHashMap::default();\n        let key1 = (0..u16::MAX as usize).map(|i| i as u8).collect::<Vec<_>>();\n        hash_map.mutate_or_create(&key1, &mut memory_arena, |opt_val: Option<u32>| {\n            assert_eq!(opt_val, None);\n            4u32\n        });\n        // Due to truncation, this key is the same as key1\n        let key2 = (0..u16::MAX as usize + 1)\n            .map(|i| i as u8)\n            .collect::<Vec<_>>();\n        hash_map.mutate_or_create(&key2, &mut memory_arena, |opt_val: Option<u32>| {\n            assert_eq!(opt_val, Some(4));\n            3u32\n        });\n        let mut vanilla_hash_map = HashMap::new();\n        let iter_values = hash_map.iter(&memory_arena);\n        for (key, addr) in iter_values {\n            let val: u32 = memory_arena.read(addr);\n            vanilla_hash_map.insert(key.to_owned(), val);\n            assert_eq!(key.len(), key1[..].len());\n            assert_eq!(key, &key1[..])\n        }\n        assert_eq!(vanilla_hash_map.len(), 1); // Both map to the same key\n    }\n\n    #[test]\n    fn test_empty_hashmap() {\n        let memory_arena = MemoryArena::default();\n        let hash_map: SharedArenaHashMap = SharedArenaHashMap::default();\n        assert_eq!(hash_map.get::<u32>(b\"abc\", &memory_arena), None);\n    }\n\n    #[test]\n    fn test_compute_previous_power_of_two() {\n        assert_eq!(compute_previous_power_of_two(8), 8);\n        assert_eq!(compute_previous_power_of_two(9), 8);\n        assert_eq!(compute_previous_power_of_two(7), 4);\n        assert_eq!(compute_previous_power_of_two(u64::MAX as usize), 1 << 63);\n    }\n\n    #[test]\n    fn test_many_terms() {\n        let mut memory_arena = MemoryArena::default();\n        let mut terms: Vec<String> = (0..20_000).map(|val| val.to_string()).collect();\n        let mut hash_map: SharedArenaHashMap = SharedArenaHashMap::default();\n        for term in terms.iter() {\n            hash_map.mutate_or_create(\n                term.as_bytes(),\n                &mut memory_arena,\n                |_opt_val: Option<u32>| 5u32,\n            );\n        }\n        let mut terms_back: Vec<String> = hash_map\n            .iter(&memory_arena)\n            .map(|(bytes, _)| String::from_utf8(bytes.to_vec()).unwrap())\n            .collect();\n        terms_back.sort();\n        terms.sort();\n\n        for pos in 0..terms.len() {\n            assert_eq!(terms[pos], terms_back[pos]);\n        }\n    }\n}\n"
  },
  {
    "path": "tests/compat_tests_data/index_v6/.managed.json",
    "content": "[\"00000000000000000000000000000000.store\",\"00000000000000000000000000000000.fast\",\"00000000000000000000000000000000.fieldnorm\",\"00000000000000000000000000000000.term\",\"00000000000000000000000000000000.idx\",\"meta.json\",\"00000000000000000000000000000000.pos\"]\n"
  },
  {
    "path": "tests/compat_tests_data/index_v6/meta.json",
    "content": "{\n  \"index_settings\": {\n    \"docstore_compression\": \"lz4\",\n    \"docstore_blocksize\": 16384\n  },\n  \"segments\": [\n    {\n      \"segment_id\": \"00000000-0000-0000-0000-000000000000\",\n      \"max_doc\": 1,\n      \"deletes\": null\n    }\n  ],\n  \"schema\": [\n    {\n      \"name\": \"label\",\n      \"type\": \"text\",\n      \"options\": {\n        \"indexing\": {\n          \"record\": \"position\",\n          \"fieldnorms\": true,\n          \"tokenizer\": \"default\"\n        },\n        \"stored\": true,\n        \"fast\": false\n      }\n    },\n    {\n      \"name\": \"date\",\n      \"type\": \"date\",\n      \"options\": {\n        \"indexed\": true,\n        \"fieldnorms\": true,\n        \"fast\": false,\n        \"stored\": true,\n        \"precision\": \"seconds\"\n      }\n    }\n  ],\n  \"opstamp\": 2\n}\n"
  },
  {
    "path": "tests/compat_tests_data/index_v7/.managed.json",
    "content": "[\"meta.json\",\"000002f0000000000000000000000000.fieldnorm\",\"000002f0000000000000000000000000.pos\",\"000002f0000000000000000000000000.store\",\"000002f0000000000000000000000000.term\",\"000002f0000000000000000000000000.fast\",\"000002f0000000000000000000000000.idx\"]\n"
  },
  {
    "path": "tests/compat_tests_data/index_v7/meta.json",
    "content": "{\n  \"index_settings\": {\n    \"docstore_compression\": \"lz4\",\n    \"docstore_blocksize\": 16384\n  },\n  \"segments\": [\n    {\n      \"segment_id\": \"000002f0-0000-0000-0000-000000000000\",\n      \"max_doc\": 1,\n      \"deletes\": null\n    }\n  ],\n  \"schema\": [\n    {\n      \"name\": \"label\",\n      \"type\": \"text\",\n      \"options\": {\n        \"indexing\": {\n          \"record\": \"position\",\n          \"fieldnorms\": true,\n          \"tokenizer\": \"default\"\n        },\n        \"stored\": true,\n        \"fast\": false\n      }\n    },\n    {\n      \"name\": \"date\",\n      \"type\": \"date\",\n      \"options\": {\n        \"indexed\": true,\n        \"fieldnorms\": true,\n        \"fast\": false,\n        \"stored\": true,\n        \"precision\": \"seconds\"\n      }\n    }\n  ],\n  \"opstamp\": 2\n}\n"
  },
  {
    "path": "tests/failpoints/mod.rs",
    "content": "use std::path::Path;\n\nuse tantivy::directory::{Directory, ManagedDirectory, RamDirectory, TerminatingWrite};\nuse tantivy::schema::{Schema, TEXT};\nuse tantivy::{doc, Index, IndexWriter, Term};\n\n#[test]\nfn test_failpoints_managed_directory_gc_if_delete_fails() {\n    let _scenario = fail::FailScenario::setup();\n\n    let test_path: &'static Path = Path::new(\"some_path_for_test\");\n\n    let ram_directory = Box::new(RamDirectory::create());\n    let mut managed_directory = ManagedDirectory::wrap(ram_directory).unwrap();\n    managed_directory\n        .open_write(test_path)\n        .unwrap()\n        .terminate()\n        .unwrap();\n    assert!(managed_directory.exists(test_path).unwrap());\n    // triggering gc and setting the delete operation to fail.\n    //\n    // We are checking that the gc operation is not removing the\n    // file from managed.json to ensure that the file will be removed\n    // in the next gc.\n    //\n    // The initial 1*off is there to allow for the removal of the\n    // lock file.\n    fail::cfg(\"RamDirectory::delete\", \"1*off->1*return\").unwrap();\n    assert!(managed_directory.garbage_collect(Default::default).is_ok());\n    assert!(managed_directory.exists(test_path).unwrap());\n\n    // running the gc a second time should remove the file.\n    assert!(managed_directory.garbage_collect(Default::default).is_ok());\n    assert!(\n        !managed_directory.exists(test_path).unwrap(),\n        \"The file should have been deleted\"\n    );\n}\n\n#[test]\nfn test_write_commit_fails() -> tantivy::Result<()> {\n    let _fail_scenario_guard = fail::FailScenario::setup();\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let index = Index::create_in_ram(schema_builder.build());\n\n    let mut index_writer: IndexWriter = index.writer_with_num_threads(1, 15_000_000)?;\n    for _ in 0..100 {\n        index_writer.add_document(doc!(text_field => \"a\"))?;\n    }\n    index_writer.commit()?;\n    fail::cfg(\"save_metas\", \"return(error_write_failed)\").unwrap();\n    for _ in 0..100 {\n        index_writer.add_document(doc!(text_field => \"b\"))?;\n    }\n    assert!(index_writer.commit().is_err());\n\n    let num_docs_containing = |s: &str| {\n        let term_a = Term::from_field_text(text_field, s);\n        index.reader()?.searcher().doc_freq(&term_a)\n    };\n    assert_eq!(num_docs_containing(\"a\")?, 100);\n    assert_eq!(num_docs_containing(\"b\")?, 0);\n    Ok(())\n}\n\n// Motivated by\n// - https://github.com/quickwit-oss/quickwit/issues/730\n// Details at\n// - https://github.com/quickwit-oss/tantivy/issues/1198\n#[test]\nfn test_fail_on_flush_segment() -> tantivy::Result<()> {\n    let _fail_scenario_guard = fail::FailScenario::setup();\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let index = Index::create_in_ram(schema_builder.build());\n    let index_writer: IndexWriter = index.writer_with_num_threads(1, 15_000_000)?;\n    fail::cfg(\"FieldSerializer::close_term\", \"return(simulatederror)\").unwrap();\n    for i in 0..100_000 {\n        if index_writer\n            .add_document(doc!(text_field => format!(\"hellohappytaxpayerlongtokenblabla{}\", i)))\n            .is_err()\n        {\n            return Ok(());\n        }\n    }\n    panic!(\"add_document should have returned an error\");\n}\n\n#[test]\nfn test_fail_on_flush_segment_but_one_worker_remains() -> tantivy::Result<()> {\n    let _fail_scenario_guard = fail::FailScenario::setup();\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let index = Index::create_in_ram(schema_builder.build());\n    let index_writer: IndexWriter = index.writer_with_num_threads(2, 30_000_000)?;\n    fail::cfg(\"FieldSerializer::close_term\", \"1*return(simulatederror)\").unwrap();\n    for i in 0..100_000 {\n        if index_writer\n            .add_document(doc!(text_field => format!(\"hellohappytaxpayerlongtokenblabla{}\", i)))\n            .is_err()\n        {\n            return Ok(());\n        }\n    }\n    panic!(\"add_document should have returned an error\");\n}\n\n#[test]\nfn test_fail_on_commit_segment() -> tantivy::Result<()> {\n    let _fail_scenario_guard = fail::FailScenario::setup();\n    let mut schema_builder = Schema::builder();\n    let text_field = schema_builder.add_text_field(\"text\", TEXT);\n    let index = Index::create_in_ram(schema_builder.build());\n    let mut index_writer: IndexWriter = index.writer_with_num_threads(1, 15_000_000)?;\n    fail::cfg(\"FieldSerializer::close_term\", \"return(simulatederror)\").unwrap();\n    for i in 0..10 {\n        index_writer\n            .add_document(doc!(text_field => format!(\"hellohappytaxpayerlongtokenblabla{}\", i)))?;\n    }\n    assert!(index_writer.commit().is_err());\n    Ok(())\n}\n"
  },
  {
    "path": "tokenizer-api/Cargo.toml",
    "content": "[package]\nname = \"tantivy-tokenizer-api\"\nversion = \"0.6.0\"\nlicense = \"MIT\"\nedition = \"2021\"\ndescription = \"Tokenizer API of tantivy\"\nhomepage = \"https://github.com/quickwit-oss/tantivy\"\nrepository = \"https://github.com/quickwit-oss/tantivy\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nserde = { version = \"1.0.152\", features = [\"derive\"] }\n"
  },
  {
    "path": "tokenizer-api/README.md",
    "content": "\n#Tokenizer-API\n\nAn API to interface a tokenizer with tantivy. \n\nThe API will be kept stable in order to not break support for existing tokenizers.\n"
  },
  {
    "path": "tokenizer-api/src/lib.rs",
    "content": "//! Tokenizer are in charge of chopping text into a stream of tokens\n//! ready for indexing. This is an separate crate from tantivy, so implementers don't need to update\n//! for each new tantivy version.\n//!\n//! To add support for a tokenizer, implement the [`Tokenizer`] trait.\n//! Checkout the [tantivy repo](https://github.com/quickwit-oss/tantivy/tree/main/src/tokenizer) for some examples.\n\nuse std::borrow::{Borrow, BorrowMut};\nuse std::ops::{Deref, DerefMut};\n\nuse serde::{Deserialize, Serialize};\n\n/// Token\n#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)]\npub struct Token {\n    /// Offset (byte index) of the first character of the token.\n    /// Offsets shall not be modified by token filters.\n    pub offset_from: usize,\n    /// Offset (byte index) of the last character of the token + 1.\n    /// The text that generated the token should be obtained by\n    /// &text[token.offset_from..token.offset_to]\n    pub offset_to: usize,\n    /// Position, expressed in number of tokens.\n    pub position: usize,\n    /// Actual text content of the token.\n    pub text: String,\n    /// Is the length expressed in term of number of original tokens.\n    pub position_length: usize,\n}\n\nimpl Default for Token {\n    fn default() -> Token {\n        Token {\n            offset_from: 0,\n            offset_to: 0,\n            position: usize::MAX,\n            text: String::new(),\n            position_length: 1,\n        }\n    }\n}\n\nimpl Token {\n    /// reset to default\n    pub fn reset(&mut self) {\n        self.offset_from = 0;\n        self.offset_to = 0;\n        self.position = usize::MAX;\n        self.text.clear();\n        self.position_length = 1;\n    }\n}\n\n/// `Tokenizer` are in charge of splitting text into a stream of token\n/// before indexing.\npub trait Tokenizer: 'static + Clone + Send + Sync {\n    /// The token stream returned by this Tokenizer.\n    type TokenStream<'a>: TokenStream;\n    /// Creates a token stream for a given `str`.\n    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>;\n}\n\n/// Simple wrapper of `Box<dyn TokenStream + 'a>`.\npub struct BoxTokenStream<'a>(Box<dyn TokenStream + 'a>);\n\nimpl TokenStream for BoxTokenStream<'_> {\n    fn advance(&mut self) -> bool {\n        self.0.advance()\n    }\n\n    fn token(&self) -> &Token {\n        self.0.token()\n    }\n\n    fn token_mut(&mut self) -> &mut Token {\n        self.0.token_mut()\n    }\n}\n\nimpl<'a> BoxTokenStream<'a> {\n    pub fn new<T: TokenStream + 'a>(token_stream: T) -> BoxTokenStream<'a> {\n        BoxTokenStream(Box::new(token_stream))\n    }\n}\n\nimpl<'a> Deref for BoxTokenStream<'a> {\n    type Target = dyn TokenStream + 'a;\n\n    fn deref(&self) -> &Self::Target {\n        &*self.0\n    }\n}\nimpl DerefMut for BoxTokenStream<'_> {\n    fn deref_mut(&mut self) -> &mut Self::Target {\n        &mut *self.0\n    }\n}\n\nimpl<'a> TokenStream for Box<dyn TokenStream + 'a> {\n    fn advance(&mut self) -> bool {\n        let token_stream: &mut dyn TokenStream = self.borrow_mut();\n        token_stream.advance()\n    }\n\n    fn token<'b>(&'b self) -> &'b Token {\n        let token_stream: &'b (dyn TokenStream + 'a) = self.borrow();\n        token_stream.token()\n    }\n\n    fn token_mut<'b>(&'b mut self) -> &'b mut Token {\n        let token_stream: &'b mut (dyn TokenStream + 'a) = self.borrow_mut();\n        token_stream.token_mut()\n    }\n}\n\n/// `TokenStream` is the result of the tokenization.\n///\n/// It consists consumable stream of `Token`s.\npub trait TokenStream {\n    /// Advance to the next token\n    ///\n    /// Returns false if there are no other tokens.\n    fn advance(&mut self) -> bool;\n\n    /// Returns a reference to the current token.\n    fn token(&self) -> &Token;\n\n    /// Returns a mutable reference to the current token.\n    fn token_mut(&mut self) -> &mut Token;\n\n    /// Helper to iterate over tokens. It\n    /// simply combines a call to `.advance()`\n    /// and `.token()`.\n    fn next(&mut self) -> Option<&Token> {\n        if self.advance() {\n            Some(self.token())\n        } else {\n            None\n        }\n    }\n\n    /// Helper function to consume the entire `TokenStream`\n    /// and push the tokens to a sink function.\n    fn process(&mut self, sink: &mut dyn FnMut(&Token)) {\n        while self.advance() {\n            sink(self.token());\n        }\n    }\n}\n\n/// Trait for the pluggable components of `Tokenizer`s.\npub trait TokenFilter: 'static + Send + Sync {\n    /// The Tokenizer type returned by this filter, typically parametrized by the underlying\n    /// Tokenizer.\n    type Tokenizer<T: Tokenizer>: Tokenizer;\n    /// Wraps a Tokenizer and returns a new one.\n    fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>;\n}\n\n#[cfg(test)]\nmod test {\n    use super::*;\n\n    #[test]\n    fn clone() {\n        let t1 = Token {\n            position: 1,\n            offset_from: 2,\n            offset_to: 3,\n            text: \"abc\".to_string(),\n            position_length: 1,\n        };\n        let t2 = t1.clone();\n\n        assert_eq!(t1.position, t2.position);\n        assert_eq!(t1.offset_from, t2.offset_from);\n        assert_eq!(t1.offset_to, t2.offset_to);\n        assert_eq!(t1.text, t2.text);\n    }\n}\n"
  }
]