[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: ''\nlabels: bug report\nassignees: ''\n\n---\n\n**First**\nSearch existing issues to see if it’s been reported and make sure you’re on the latest version.\n\n**Describe the bug**\nA clear and concise description of the bug.\n\n**To reproduce**\nUse this code to reproduce when possible:\n\n```ruby\nrequire \"bundler/inline\"\n\ngemfile do\n source \"https://rubygems.org\"\n\n gem \"activerecord\", require: \"active_record\"\n gem \"activejob\", require: \"active_job\"\n gem \"sqlite3\"\n gem \"searchkick\", git: \"https://github.com/ankane/searchkick.git\"\n # uncomment one\n # gem \"elasticsearch\"\n # gem \"opensearch-ruby\"\nend\n\nputs \"Searchkick version: #{Searchkick::VERSION}\"\nputs \"Server version: #{Searchkick.server_version}\"\n\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \":memory:\"\nActiveJob::Base.queue_adapter = :inline\n\nActiveRecord::Schema.define do\n create_table :products do |t|\n t.string :name\n end\nend\n\nclass Product < ActiveRecord::Base\n searchkick\nend\n\nProduct.reindex\nProduct.create!(name: \"Test\")\nProduct.search_index.refresh\np Product.search(\"test\", fields: [:name]).response\n```\n\n**Additional context**\nAdd any other context.\n"
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"content": "blank_issues_enabled: false\ncontact_links:\n - name: Help\n url: https://stackoverflow.com/questions/tagged/searchkick\n about: Ask and answer questions here\n"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"content": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: ''\nlabels: feature request\nassignees: ''\n\n---\n\n**First**\nSearch existing issues to see if it’s been discussed.\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of the problem.\n\n**Describe the solution you'd like**\nA clear and concise description of your idea.\n\n**Additional context**\nAdd any other context.\n"
},
{
"path": ".github/pull_request_template.md",
"content": "Thanks for contributing. You’re awesome! A few things to keep in mind:\n\n- Keep changes to a minimum\n- Follow the existing style\n- Add one or more tests if possible\n\nFinally, replace all this with a description of the changes.\n"
},
{
"path": ".github/workflows/build.yml",
"content": "name: build\non: [push, pull_request]\njobs:\n build:\n strategy:\n fail-fast: false\n matrix:\n include:\n - ruby: \"4.0\"\n gemfile: Gemfile\n elasticsearch: 9\n - ruby: 3.3\n gemfile: gemfiles/activerecord80.gemfile\n elasticsearch: 9.0.0\n - ruby: 3.2\n gemfile: gemfiles/activerecord72.gemfile\n elasticsearch: 8\n - ruby: 3.4\n gemfile: gemfiles/opensearch3.gemfile\n opensearch: 3\n - ruby: 3.3\n gemfile: gemfiles/opensearch2.gemfile\n opensearch: 2\n - ruby: 3.4\n gemfile: gemfiles/mongoid9.gemfile\n elasticsearch: 9\n mongodb: true\n - ruby: 3.2\n gemfile: gemfiles/mongoid8.gemfile\n # TODO fix plugin installation for earlier versions\n elasticsearch: 8.5.0\n mongodb: true\n runs-on: ubuntu-latest\n env:\n BUNDLE_GEMFILE: ${{ matrix.gemfile }}\n steps:\n - uses: actions/checkout@v6\n - uses: ruby/setup-ruby@v1\n with:\n ruby-version: ${{ matrix.ruby }}\n bundler-cache: true\n - run: bundle update\n\n - uses: actions/cache@v5\n if: ${{ matrix.elasticsearch }}\n with:\n path: ~/elasticsearch\n key: ${{ runner.os }}-elasticsearch-${{ matrix.elasticsearch }}\n - uses: ankane/setup-elasticsearch@v1\n if: ${{ matrix.elasticsearch }}\n with:\n elasticsearch-version: ${{ matrix.elasticsearch }}\n plugins: |\n analysis-kuromoji\n analysis-smartcn\n analysis-stempel\n analysis-ukrainian\n\n - uses: actions/cache@v5\n if: ${{ matrix.opensearch }}\n with:\n path: ~/opensearch\n key: ${{ runner.os }}-opensearch-${{ matrix.opensearch }}\n - uses: ankane/setup-opensearch@v1\n if: ${{ matrix.opensearch }}\n with:\n opensearch-version: ${{ matrix.opensearch }}\n plugins: |\n analysis-kuromoji\n analysis-smartcn\n analysis-stempel\n analysis-ukrainian\n\n - uses: ankane/setup-mongodb@v1\n if: ${{ matrix.mongodb }}\n\n - run: |\n sudo apt-get update\n sudo apt-get install redis-server\n sudo systemctl start redis-server\n - run: bundle exec rake test\n"
},
{
"path": ".gitignore",
"content": "*.gem\n*.rbc\n.bundle\n.config\n.yardoc\n*.lock\nInstalledFiles\n_yardoc\ncoverage\ndoc/\nlib/bundler/man\npkg\nrdoc\nspec/reports\ntest/tmp\ntest/version_tmp\ntmp\n*.log\n.DS_Store\n.ruby-*\n.idea/\n*.sqlite3\n"
},
{
"path": "CHANGELOG.md",
"content": "## 6.1.1 (unreleased)\n\n- Fixed smart aggs behavior with `_and`\n\n## 6.1.0 (2026-02-18)\n\n- Added `per` method\n- Fixed error with `aggs` method and non-hash arguments\n- Fixed smart aggs behavior when multiple `where` calls\n\n## 6.0.3 (2026-01-06)\n\n- Fixed `inspect` method for `Relation`\n\n## 6.0.2 (2025-10-24)\n\n- Fixed `as_json` method for `HashWrapper`\n\n## 6.0.1 (2025-10-24)\n\n- Fixed `to_json` method for `HashWrapper`\n\n## 6.0.0 (2025-10-19)\n\n- Added new query builder API (similar to Active Record)\n- Added `conversions_v2` option\n- Added `job_options` option\n- Added `parent_job` option\n- Added `opaque_id` option\n- Added `callback_options` option\n- Added `ignore_missing` option for partial reindex\n- Added support for `exists: false`\n- Added `quantization` to `knn` option for Elasticsearch\n- Changed async reindex to use ranges for numeric primary keys with Active Record\n- Fixed error with `case_sensitive` option and synonyms\n- Removed default quantization for `knn` option for Elasticsearch 8.14+\n- Removed `results` method (use `to_a` instead)\n- Removed `execute` option and method (no longer needed)\n- Removed `options` method (use individual methods instead)\n- Removed dependency on Hashie\n- Deprecated `conversions` option in favor of `conversions_v2`\n- Dropped support for Elasticsearch 7 and OpenSearch 1\n- Dropped support for Active Record < 7.2\n- Dropped support for Redis < 6.2\n\n## 5.5.2 (2025-05-20)\n\n- Fixed `scope` option for partial reindex\n\n## 5.5.1 (2025-04-24)\n\n- Added support for `elasticsearch` 9 gem\n\n## 5.5.0 (2025-04-03)\n\n- Added `m` and `ef_construction` to `knn` index option\n- Added `ef_search` to `knn` search option\n- Fixed exact cosine distance for OpenSearch 2.19+\n- Dropped support for Ruby < 3.2 and Active Record < 7.1\n- Dropped support for Mongoid < 8\n\n## 5.4.0 (2024-09-04)\n\n- Added `knn` option\n- Added `rrf` method\n- Added experimental support for scripting to `where` option\n- Added warning for `exists` with non-`true` values\n- Added warning for full reindex and `:queue` mode\n- Fixed `per_page` method when paginating beyond `max_result_window`\n- Dropped support for Ruby < 3.1\n\n## 5.3.1 (2023-11-28)\n\n- Fixed error with misspellings below and failed queries\n\n## 5.3.0 (2023-07-02)\n\n- Fixed error with `cutoff_frequency`\n- Dropped support for Ruby < 3 and Active Record < 6.1\n- Dropped support for Mongoid < 7\n\n## 5.2.4 (2023-05-11)\n\n- Fixed error with non-string routing and `:async` mode\n\n## 5.2.3 (2023-04-12)\n\n- Fixed error with missing records and multiple models\n\n## 5.2.2 (2023-04-01)\n\n- Fixed `total_docs` method\n- Fixed deprecation warning with Active Support 7.1\n\n## 5.2.1 (2023-02-21)\n\n- Added support for `redis-client` gem\n\n## 5.2.0 (2023-02-08)\n\n- Added model name to warning about missing records\n- Fixed unnecessary data loading when reindexing relations with `:async` and `:queue` modes\n\n## 5.1.2 (2023-01-29)\n\n- Fixed error with missing point in time\n\n## 5.1.1 (2022-12-05)\n\n- Added support for strings for `offset` and `per_page`\n\n## 5.1.0 (2022-10-12)\n\n- Added support for fractional search timeout\n- Fixed search timeout with `elasticsearch` 8+ and `opensearch-ruby` gems\n- Fixed search timeout not applying to `multi_search`\n\n## 5.0.5 (2022-10-09)\n\n- Added `model` method to `Searchkick::Relation`\n- Fixed deprecation warning with `redis` gem\n- Fixed `respond_to?` method on relation loading relation\n- Fixed `Relation loaded` error for non-mutating methods on relation\n\n## 5.0.4 (2022-06-16)\n\n- Added `max_result_window` option\n- Improved error message for unsupported versions of Elasticsearch\n\n## 5.0.3 (2022-03-13)\n\n- Fixed context for index name for inherited models\n\n## 5.0.2 (2022-03-03)\n\n- Fixed index name for inherited models\n\n## 5.0.1 (2022-02-27)\n\n- Prefer `mode: :async` over `async: true` for full reindex\n- Fixed instance method overriding with concerns\n\n## 5.0.0 (2022-02-21)\n\n- Searches now use lazy loading (similar to Active Record)\n- Added `unscope` option to better support working with default scopes\n- Added support for `:async` and `:queue` modes for `reindex` on relation\n- Added basic protection from unfiltered parameters to `where` option\n- Added `models` option to `similar` method\n- Changed async full reindex to fetch ids instead of using ranges for numeric primary keys with Active Record\n- Changed `searchkick_index_options` to return symbol keys (instead of mix of strings and symbols)\n- Changed non-anchored regular expressions to match expected results (previously warned)\n- Changed record reindex to return `true` to match model and relation reindex\n- Updated async reindex job to call `search_import` for nested associations\n- Fixed removing records when `should_index?` is `false` when `reindex` called on relation\n- Fixed issue with `merge_mappings` for fields that use `searchkick` options\n- Raise error when `search` called on relations\n- Raise `ArgumentError` (instead of warning) for invalid regular expression modifiers\n- Raise `ArgumentError` instead of `RuntimeError` for unknown operators\n- Removed mapping of `id` to `_id` with `order` option (not supported in Elasticsearch 8)\n- Removed `wordnet` option (no longer worked)\n- Removed dependency on `elasticsearch` gem (can use `elasticsearch` or `opensearch-ruby`)\n- Dropped support for Elasticsearch 6\n- Dropped support for Ruby < 2.6 and Active Record < 5.2\n- Dropped support for NoBrainer and Cequel\n- Dropped support for `faraday_middleware-aws-signers-v4` (use `faraday_middleware-aws-sigv4` instead)\n\n## 4.6.3 (2021-11-19)\n\n- Added support for reloadable synonyms for OpenSearch\n- Added experimental support for `opensearch-ruby` gem\n- Removed `elasticsearch-xpack` dependency for reloadable synonyms\n\n## 4.6.2 (2021-11-15)\n\n- Added support for beginless ranges to `where` option\n- Fixed `like` and `ilike` with `+` character\n- Fixed warning about accessing system indices when no model or index specified\n\n## 4.6.1 (2021-09-25)\n\n- Added `ilike` operator for Elasticsearch 7.10+\n- Fixed missing methods with `multi_search`\n\n## 4.6.0 (2021-08-22)\n\n- Added support for case-insensitive regular expressions with Elasticsearch 7.10+\n- Added support for `OPENSEARCH_URL`\n- Fixed error with `debug` option\n\n## 4.5.2 (2021-08-05)\n\n- Fixed error with reindex queue\n- Fixed error with `model_name` method with multiple models\n- Fixed error with `debug` option with elasticsearch-ruby 7.14\n\n## 4.5.1 (2021-08-03)\n\n- Improved performance of reindex queue\n\n## 4.5.0 (2021-06-07)\n\n- Added experimental support for OpenSearch\n- Added support for synonyms in Japanese\n\n## 4.4.4 (2021-03-12)\n\n- Fixed `too_long_frame_exception` with `scroll` method\n- Fixed multi-word emoji tokenization\n\n## 4.4.3 (2021-02-25)\n\n- Added support for Hunspell\n- Fixed warning about accessing system indices\n\n## 4.4.2 (2020-11-23)\n\n- Added `missing_records` method to results\n- Fixed issue with `like` and special characters\n\n## 4.4.1 (2020-06-24)\n\n- Added `stem_exclusion` and `stemmer_override` options\n- Added `with_score` method to search results\n- Improved error message for `reload_synonyms` with non-OSS version of Elasticsearch\n- Improved output for reindex rake task\n\n## 4.4.0 (2020-06-17)\n\n- Added support for reloadable, multi-word, search time synonyms\n- Fixed another deprecation warning in Ruby 2.7\n\n## 4.3.1 (2020-05-13)\n\n- Fixed error with `exclude` in certain cases for Elasticsearch 7.7\n\n## 4.3.0 (2020-02-19)\n\n- Fixed `like` queries with `\"` character\n- Better error when invalid parameters passed to `where`\n\n## 4.2.1 (2020-01-27)\n\n- Fixed deprecation warnings with Elasticsearch\n- Fixed deprecation warnings in Ruby 2.7\n\n## 4.2.0 (2019-12-18)\n\n- Added safety check for multiple `Model.reindex`\n- Added `deep_paging` option\n- Added request parameters to search notifications and curl representation\n- Removed curl from search notifications to prevent confusion\n\n## 4.1.1 (2019-11-19)\n\n- Added `chinese2` and `korean2` languages\n- Improved performance of async full reindex\n- Fixed `searchkick:reindex:all` rake task for Rails 6\n\n## 4.1.0 (2019-08-01)\n\n- Added `like` operator\n- Added `exists` operator\n- Added warnings for certain regular expressions\n- Fixed anchored regular expressions\n\n## 4.0.2 (2019-06-04)\n\n- Added block form of `scroll`\n- Added `clear_scroll` method\n- Fixed custom mappings\n\n## 4.0.1 (2019-05-30)\n\n- Added support for scroll API\n- Made type optional for custom mapping for Elasticsearch 6\n- Fixed error when suggestions empty\n- Fixed `models` option with inheritance\n\n## 4.0.0 (2019-04-11)\n\n- Added support for Elasticsearch 7\n- Added `models` option\n\nBreaking changes\n\n- Removed support for Elasticsearch 5\n- Removed support for multi-word synonyms (they no longer work with shingles)\n- Removed support for Active Record < 5\n\n## 3.1.3 (2019-04-11)\n\n- Added support for endless ranges\n- Added support for routing to `similar` method\n- Added `prefix` to `where`\n- Fixed error with elasticsearch-ruby 6.3\n- Fixed error with some language stemmers and Elasticsearch 6.5\n- Fixed issue with misspellings below and body block\n\n## 3.1.2 (2018-09-27)\n\n- Improved performance of indices boost\n- Fixed deletes with routing and `async` callbacks\n- Fixed deletes with routing and `queue` callbacks\n- Fixed deprecation warnings\n- Fixed field misspellings for older partial match format\n\n## 3.1.1 (2018-08-09)\n\n- Added per-field misspellings\n- Added `case_sensitive` option\n- Added `stem` option\n- Added `total_entries` option\n- Fixed `exclude` option with match all\n- Fixed `with_highlights` method\n\n## 3.1.0 (2018-05-12)\n\n- Added `:inline` as alias for `true` for `callbacks` and `mode` options\n- Friendlier error message for bad mapping with partial matches\n- Warn when records in search index do not exist in database\n- Easier merging for `merge_mapping`\n- Fixed `with_hit` and `with_highlights` when records in search index do not exist in database\n- Fixed error with highlights and match all\n\n## 3.0.3 (2018-04-22)\n\n- Added support for pagination with `body` option\n- Added `boost_by_recency` option\n- Fixed \"Model Search Data\" output for `debug` option\n- Fixed `reindex_status` error\n- Fixed error with optional operators in Ruby regexp\n- Fixed deprecation warnings for Elasticsearch 6.2+\n\n## 3.0.2 (2018-03-26)\n\n- Added support for Korean and Vietnamese\n- Fixed `Unsupported argument type: Symbol` for async partial reindex\n- Fixed infinite recursion with multi search and misspellings below\n- Do not raise an error when `id` is indexed\n\n## 3.0.1 (2018-03-14)\n\n- Added `scope` option for partial reindex\n- Added support for Japanese, Polish, and Ukrainian\n\n## 3.0.0 (2018-03-03)\n\n- Added support for Chinese\n- No longer requires fields to query for Elasticsearch 6\n- Results can be marshaled by default (unless using `highlight` option)\n\nBreaking changes\n\n- Removed support for Elasticsearch 2\n- Removed support for Active Record < 4.2 and Mongoid < 5\n- Types are no longer used\n- The `_all` field is disabled by default in Elasticsearch 5\n- Conversions are not stemmed by default\n- An `ArgumentError` is raised instead of a warning when options are incompatible with the `body` option\n- Removed `log` option from `boost_by`\n- Removed `Model.enable_search_callbacks`, `Model.disable_search_callbacks`, and `Model.search_callbacks?`\n- Removed `reindex_async` method, as `reindex` now defaults to callbacks mode specified on the model\n- Removed `async` option from `record.reindex`\n- Removed `search_hit` method - use `with_hit` instead\n- Removed `each_with_hit` - use `with_hit.each` instead\n- Removed `with_details` - use `with_highlights` instead\n- Bumped default `limit` to 10,000\n\n## 2.5.0 (2018-02-15)\n\n- Try requests 3 times before raising error\n- Better exception when trying to access results for failed multi-search query\n- More efficient aggregations with `where` clauses\n- Added support for `faraday_middleware-aws-sigv4`\n- Added `credentials` option to `aws_credentials`\n- Added `modifier` option to `boost_by`\n- Added `scope_results` option\n- Added `factor` option to `boost_by_distance`\n\n## 2.4.0 (2017-11-14)\n\n- Fixed `similar` for Elasticsearch 6\n- Added `inheritance` option\n- Added `_type` option\n- Fixed `Must specify fields to search` error when searching `*`\n\n## 2.3.2 (2017-09-08)\n\n- Added `_all` and `default_fields` options\n- Added global `index_prefix` option\n- Added `wait` option to async reindex\n- Added `model_includes` option\n- Added `missing` option for `boost_by`\n- Raise error for `reindex_status` when Redis not configured\n- Warn when incompatible options used with `body` option\n- Fixed bug where `routing` and `type` options were silently ignored with `body` option\n- Fixed `reindex(async: true)` for non-numeric primary keys in Postgres\n\n## 2.3.1 (2017-07-06)\n\n- Added support for `reindex(async: true)` for non-numeric primary keys\n- Added `conversions_term` option\n- Added support for passing fields to `suggest` option\n- Fixed `page_view_entries` for Kaminari\n\n## 2.3.0 (2017-05-06)\n\n- Fixed analyzer on dynamically mapped fields\n- Fixed error with `similar` method and `_all` field\n- Throw error when fields are needed\n- Added `queue_name` option\n- No longer require synonyms to be lowercase\n\n## 2.2.1 (2017-04-16)\n\n- Added `avg`, `cardinality`, `max`, `min`, and `sum` aggregations\n- Added `load: {dumpable: true}` option\n- Added `index_suffix` option\n- Accept string for `exclude` option\n\n## 2.2.0 (2017-03-19)\n\n- Fixed bug with text values longer than 256 characters and `_all` field - see [#850](https://github.com/ankane/searchkick/issues/850)\n- Fixed issue with `_all` field in `searchable`\n- Fixed `exclude` option with `word_start`\n\n## 2.1.1 (2017-01-17)\n\n- Fixed duplicate notifications\n- Added support for `connection_pool`\n- Added `exclude` option\n\n## 2.1.0 (2017-01-15)\n\n- Background reindexing and queues are officially supported\n- Log updates and deletes\n\n## 2.0.4 (2017-01-15)\n\n- Added support for queuing updates [experimental]\n- Added `refresh_interval` option to `reindex`\n- Prefer `search_index` over `searchkick_index`\n\n## 2.0.3 (2017-01-12)\n\n- Added `async` option to `reindex` [experimental]\n- Added `misspellings?` method to results\n\n## 2.0.2 (2017-01-08)\n\n- Added `retain` option to `reindex`\n- Added support for attributes in highlight tags\n- Fixed potentially silent errors in reindex job\n- Improved syntax for `boost_by_distance`\n\n## 2.0.1 (2016-12-30)\n\n- Added `search_hit` and `search_highlights` methods to models\n- Improved reindex performance\n\n## 2.0.0 (2016-12-28)\n\n- Added support for `reindex` on associations\n\nBreaking changes\n\n- Removed support for Elasticsearch 1 as it reaches [end of life](https://www.elastic.co/support/eol)\n- Removed facets, legacy options, and legacy methods\n- Invalid options now throw an `ArgumentError`\n- The `query` and `json` options have been removed in favor of `body`\n- The `include` option has been removed in favor of `includes`\n- The `personalize` option has been removed in favor of `boost_where`\n- The `partial` option has been removed in favor of `operator`\n- Renamed `select_v2` to `select` (legacy `select` no longer available)\n- The `_all` field is disabled if `searchable` option is used (for performance)\n- The `partial_reindex(:method_name)` method has been replaced with `reindex(:method_name)`\n- The `unsearchable` and `only_analyzed` options have been removed in favor of `searchable` and `filterable`\n- `load: false` no longer returns an array in Elasticsearch 2\n\n## 1.5.1 (2016-12-28)\n\n- Added `client_options`\n- Added `refresh` option to `reindex` method\n- Improved syntax for partial reindex\n\n## 1.5.0 (2016-12-23)\n\n- Added support for geo shape indexing and queries\n- Added `_and`, `_or`, `_not` to `where` option\n\n## 1.4.2 (2016-12-21)\n\n- Added support for directional synonyms\n- Easier AWS setup\n- Fixed `total_docs` method for ES 5+\n- Fixed exception on update errors\n\n## 1.4.1 (2016-12-11)\n\n- Added `partial_reindex` method\n- Added `debug` option to `search` method\n- Added `profile` option\n\n## 1.4.0 (2016-10-26)\n\n- Official support for Elasticsearch 5\n- Boost exact matches for partial matching\n- Added `searchkick_debug` method\n- Added `geo_polygon` filter\n\n## 1.3.6 (2016-10-08)\n\n- Fixed `Job adapter not found` error\n\n## 1.3.5 (2016-09-27)\n\n- Added support for Elasticsearch 5.0 beta\n- Added `request_params` option\n- Added `filterable` option\n\n## 1.3.4 (2016-08-23)\n\n- Added `resume` option to reindex\n- Added search timeout to payload\n\n## 1.3.3 (2016-08-02)\n\n- Fix for namespaced models (broken in 1.3.2)\n\n## 1.3.2 (2016-08-01)\n\n- Added `body_options` option\n- Added `date_histogram` aggregation\n- Added `indices_boost` option\n- Added support for multiple conversions\n\n## 1.3.1 (2016-07-10)\n\n- Fixed error with Ruby 2.0\n- Fixed error with indexing large fields\n\n## 1.3.0 (2016-05-04)\n\n- Added support for Elasticsearch 5.0 alpha\n- Added support for phrase matches\n- Added support for procs for `index_prefix` option\n\n## 1.2.1 (2016-02-15)\n\n- Added `multi_search` method\n- Added support for routing for Elasticsearch 2\n- Added support for `search_document_id` and `search_document_type` in models\n- Fixed error with instrumentation for searching multiple models\n- Fixed instrumentation for bulk updates\n\n## 1.2.0 (2016-02-03)\n\n- Fixed deprecation warnings with `alias_method_chain`\n- Added `analyzed_only` option for large text fields\n- Added `encoder` option to highlight\n- Fixed issue in `similar` method with `per_page` option\n- Added basic support for multiple models\n\n## 1.1.2 (2015-12-18)\n\n- Added bulk updates with `callbacks` method\n- Added `bulk_delete` method\n- Added `search_timeout` option\n- Fixed bug with new location format for `boost_by_distance`\n\n## 1.1.1 (2015-12-14)\n\n- Added support for `{lat: lat, lon: lon}` as preferred format for locations\n\n## 1.1.0 (2015-12-08)\n\n- Added `below` option to misspellings to improve performance\n- Fixed synonyms for `word_*` partial matches\n- Added `searchable` option\n- Added `similarity` option\n- Added `match` option\n- Added `word` option\n- Added highlighted fields to `load: false`\n\n## 1.0.3 (2015-11-27)\n\n- Added support for Elasticsearch 2.1\n\n## 1.0.2 (2015-11-15)\n\n- Throw `Searchkick::ImportError` for errors when importing records\n- Errors now inherit from `Searchkick::Error`\n- Added `order` option to aggregations\n- Added `mapping` method\n\n## 1.0.1 (2015-11-05)\n\n- Added aggregations method to get raw response\n- Use `execute: false` for lazy loading\n- Return nil when no aggs\n- Added emoji search\n\n## 1.0.0 (2015-10-30)\n\n- Added support for Elasticsearch 2.0\n- Added support for aggregations\n- Added ability to use misspellings for partial matches\n- Added `fragment_size` option for highlight\n- Added `took` method to results\n\nBreaking changes\n\n- Raise `Searchkick::DangerousOperation` error when calling reindex with scope\n- Enabled misspellings by default for partial matches\n- Enabled transpositions by default for misspellings\n\n## 0.9.1 (2015-08-31)\n\n- `and` now matches `&`\n- Added `transpositions` option to misspellings\n- Added `boost_mode` and `log` options to `boost_by`\n- Added `prefix_length` option to `misspellings`\n- Added ability to set env\n\n## 0.9.0 (2015-06-07)\n\n- Much better performance for where queries if no facets\n- Added basic support for regex\n- Added support for routing\n- Made `Searchkick.disable_callbacks` thread-safe\n\n## 0.8.7 (2015-02-14)\n\n- Fixed Mongoid import\n\n## 0.8.6 (2015-02-10)\n\n- Added support for NoBrainer\n- Added `stem_conversions: false` option\n- Added support for multiple `boost_where` values on the same field\n- Added support for array of values for `boost_where`\n- Fixed suggestions with partial match boost\n- Fixed redefining existing instance methods in models\n\n## 0.8.5 (2014-11-11)\n\n- Added support for Elasticsearch 1.4\n- Added `unsearchable` option\n- Added `select: true` option\n- Added `body` option\n\n## 0.8.4 (2014-11-05)\n\n- Added `boost_by_distance`\n- More flexible highlight options\n- Better `env` logic\n\n## 0.8.3 (2014-09-20)\n\n- Added support for Active Job\n- Added `timeout` setting\n- Fixed import with no records\n\n## 0.8.2 (2014-08-18)\n\n- Added `async` to `callbacks` option\n- Added `wordnet` option\n- Added `edit_distance` option to eventually replace `distance` option\n- Catch misspelling of `misspellings` option\n- Improved logging\n\n## 0.8.1 (2014-08-16)\n\n- Added `search_method_name` option\n- Fixed `order` for array of hashes\n- Added support for Mongoid 2\n\n## 0.8.0 (2014-07-12)\n\n- Added support for Elasticsearch 1.2\n\n## 0.7.9 (2014-06-30)\n\n- Added `tokens` method\n- Added `json` option\n- Added exact matches\n- Added `prev_page` for Kaminari pagination\n- Added `import` option to reindex\n\n## 0.7.8 (2014-06-22)\n\n- Added `boost_by` and `boost_where` options\n- Added ability to boost fields - `name^10`\n- Added `select` option for `load: false`\n\n## 0.7.7 (2014-06-10)\n\n- Added support for automatic failover\n- Fixed `operator` option (and default) for partial matches\n\n## 0.7.6 (2014-05-20)\n\n- Added `stats` option to facets\n- Added `padding` option\n\n## 0.7.5 (2014-05-13)\n\n- Do not throw errors when index becomes out of sync with database\n- Added custom exception types\n- Fixed `offset` and `offset_value`\n\n## 0.7.4 (2014-05-06)\n\n- Fixed reindex with inheritance\n\n## 0.7.3 (2014-04-30)\n\n- Fixed multi-index searches\n- Fixed suggestions for partial matches\n- Added `offset` and `length` for improved pagination\n\n## 0.7.2 (2014-04-24)\n\n- Added smart facets\n- Added more fields to `load: false` result\n- Fixed logging for multi-index searches\n- Added `first_page?` and `last_page?` for improved Kaminari support\n\n## 0.7.1 (2014-04-12)\n\n- Fixed huge issue w/ zero-downtime reindexing on 0.90\n\n## 0.7.0 (2014-04-10)\n\n- Added support for Elasticsearch 1.1\n- Dropped support for Elasticsearch below 0.90.4 (unfortunate side effect of above)\n\n## 0.6.3 (2014-04-08)\n\n- Removed patron since no support for Windows\n- Added error if `searchkick` is called multiple times\n\n## 0.6.2 (2014-04-05)\n\n- Added logging\n- Fixed index_name option\n- Added ability to use proc as the index name\n\n## 0.6.1 (2014-03-24)\n\n- Fixed huge issue w/ zero-downtime reindexing on 0.90 and elasticsearch-ruby 1.0\n- Restore load: false behavior\n- Restore total_entries method\n\n## 0.6.0 (2014-03-22)\n\n- Moved to elasticsearch-ruby\n- Added support for modifying the query and viewing the response\n- Added support for page_entries_info method\n\n## 0.5.3 (2014-02-24)\n\n- Fixed bug w/ word_* queries\n\n## 0.5.2 (2014-02-12)\n\n- Use after_commit hook for Active Record to prevent data inconsistencies\n\n## 0.5.1 (2014-02-12)\n\n- Replaced stop words with common terms query\n- Added language option\n- Fixed bug with empty array in where clause\n- Fixed bug with MongoDB integer _id\n- Fixed reindex bug when callbacks disabled\n\n## 0.5.0 (2014-01-20)\n\n- Better control over partial matches\n- Added merge_mappings option\n- Added batch_size option\n- Fixed bug with nil where clauses\n\n## 0.4.2 (2013-12-29)\n\n- Added `should_index?` method to control which records are indexed\n- Added ability to temporarily disable callbacks\n- Added custom mappings\n\n## 0.4.1 (2013-12-19)\n\n- Fixed issue w/ inheritance mapping\n\n## 0.4.0 (2013-12-11)\n\n- Added support for Mongoid 4\n- Added support for multiple locations\n\n## 0.3.5 (2013-12-08)\n\n- Added facet ranges\n- Added all operator\n\n## 0.3.4 (2013-11-22)\n\n- Added highlighting\n- Added :distance option to misspellings\n- Fixed issue w/ BigDecimal serialization\n\n## 0.3.3 (2013-11-04)\n\n- Better error messages\n- Added where: {field: nil} queries\n\n## 0.3.2 (2013-11-02)\n\n- Added support for single table inheritance\n- Removed Tire::Model::Search\n\n## 0.3.1 (2013-11-02)\n\n- Added index_prefix option\n- Fixed ES issue with incorrect facet counts\n- Added option to turn off special characters\n\n## 0.3.0 (2013-11-02)\n\n- Fixed reversed coordinates\n- Added bounded by a box queries\n- Expanded `or` queries\n\n## 0.2.8 (2013-09-30)\n\n- Added option to disable callbacks\n- Fixed bug with facets with Elasticsearch 0.90.5\n\n## 0.2.7 (2013-09-23)\n\n- Added limit to facet\n- Improved similar items\n\n## 0.2.6 (2013-09-10)\n\n- Added option to disable misspellings\n\n## 0.2.5 (2013-08-30)\n\n- Added geospartial searches\n- Create alias before importing document if no alias exists\n- Fixed exception when :per_page option is a string\n- Check `RAILS_ENV` if `RACK_ENV` is not set\n\n## 0.2.4 (2013-08-20)\n\n- Use `to_hash` instead of `as_json` for default `search_data` method\n- Works for Mongoid 1.3\n- Use one shard in test environment for consistent scores\n\n## 0.2.3 (2013-08-16)\n\n- Setup Travis\n- Clean old indices before reindex\n- Search for `*` returns all results\n- Fixed pagination\n- Added `similar` method\n\n## 0.2.2 (2013-08-11)\n\n- Clean old indices after reindex\n- More expansions for fuzzy queries\n\n## 0.2.1 (2013-08-11)\n\n- Added Rails logger\n- Only fetch ids when `load: true`\n\n## 0.2.0 (2013-08-10)\n\n- Added autocomplete\n- Added “Did you mean” suggestions\n- Added personalized searches\n\n## 0.1.4 (2013-08-03)\n\n- Bug fix\n\n## 0.1.3 (2013-08-03)\n\n- Changed edit distance to one for misspellings\n- Raise errors when indexing fails\n- Fixed pagination\n- Fixed :include option\n\n## 0.1.2 (2013-07-30)\n\n- Use conversions by default\n\n## 0.1.1 (2013-07-29)\n\n- Renamed `_source` to `search_data`\n- Renamed `searchkick_import` to `search_import`\n\n## 0.1.0 (2013-07-28)\n\n- Added `_source` method\n- Added `index_name` option\n\n## 0.0.2 (2013-07-17)\n\n- Added `conversions` option\n\n## 0.0.1 (2013-07-14)\n\n- First release\n"
},
{
"path": "Gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec\n\ngem \"rake\"\ngem \"minitest\"\ngem \"sqlite3\", platform: :ruby\ngem \"sqlite3-ffi\", platform: :jruby\ngem \"activerecord\", \"~> 8.1.0\"\ngem \"actionpack\", \"~> 8.1.0\"\ngem \"activejob\", \"~> 8.1.0\", require: \"active_job\"\ngem \"elasticsearch\", \"~> 9\"\ngem \"redis-client\"\ngem \"connection_pool\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\ngem \"parallel_tests\"\ngem \"typhoeus\", platform: :mri\ngem \"cgi\" # for elasticsearch\n"
},
{
"path": "LICENSE.txt",
"content": "Copyright (c) 2013-2026 Andrew Kane\n\nMIT License\n\nPermission is hereby granted, free of charge, to any person obtaining\na copy of this software and associated documentation files (the\n\"Software\"), to deal in the Software without restriction, including\nwithout limitation the rights to use, copy, modify, merge, publish,\ndistribute, sublicense, and/or sell copies of the Software, and to\npermit persons to whom the Software is furnished to do so, subject to\nthe following conditions:\n\nThe above copyright notice and this permission notice shall be\nincluded in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND,\nEXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF\nMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND\nNONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE\nLIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\nOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION\nWITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
},
{
"path": "README.md",
"content": "# Searchkick\n\n:rocket: Intelligent search made easy\n\n**Searchkick learns what your users are looking for.** As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.\n\nSearchkick handles:\n\n- stemming - `tomatoes` matches `tomato`\n- special characters - `jalapeno` matches `jalapeño`\n- extra whitespace - `dishwasher` matches `dish washer`\n- misspellings - `zuchini` matches `zucchini`\n- custom synonyms - `pop` matches `soda`\n\nPlus:\n\n- query like SQL - no need to learn a new query language\n- reindex without downtime\n- easily personalize results for each user\n- autocomplete\n- “Did you mean” suggestions\n- supports many languages\n- works with Active Record and Mongoid\n\nCheck out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions\n\n:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)\n\n[](https://github.com/ankane/searchkick/actions)\n\n## Contents\n\n- [Getting Started](#getting-started)\n- [Querying](#querying)\n- [Indexing](#indexing)\n- [Intelligent Search](#intelligent-search)\n- [Instant Search / Autocomplete](#instant-search--autocomplete)\n- [Aggregations](#aggregations)\n- [Testing](#testing)\n- [Deployment](#deployment)\n- [Performance](#performance)\n- [Advanced Search](#advanced)\n- [Reference](#reference)\n- [Contributing](#contributing)\n\nSearchkick 6.0 was recently released! See [how to upgrade](#upgrading)\n\n## Getting Started\n\nInstall [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:\n\n```sh\nbrew install opensearch\nbrew services start opensearch\n```\n\nAdd these lines to your application’s Gemfile:\n\n```ruby\ngem \"searchkick\"\n\ngem \"elasticsearch\" # select one\ngem \"opensearch-ruby\" # select one\n```\n\nThe latest version works with Elasticsearch 8 and 9 and OpenSearch 2 and 3. For Elasticsearch 7 and OpenSearch 1, use version 5.5.2 and [this readme](https://github.com/ankane/searchkick/blob/v5.5.2/README.md).\n\nAdd `searchkick` to models you want to search.\n\n```ruby\nclass Product < ApplicationRecord\n searchkick\nend\n```\n\nAdd data to the search index.\n\n```ruby\nProduct.reindex\n```\n\nAnd to query, use:\n\n```ruby\nproducts = Product.search(\"apples\")\nproducts.each do |product|\n puts product.name\nend\n```\n\nSearchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.\n\n## Querying\n\nQuery like SQL\n\n```ruby\nProduct.search(\"apples\").where(in_stock: true).limit(10).offset(50)\n```\n\nSearch specific fields\n\n```ruby\nfields(:name, :brand)\n```\n\nWhere\n\n```ruby\nwhere(store_id: 1, expires_at: Time.now..)\n```\n\n[These types of filters are supported](#filtering)\n\nOrder\n\n```ruby\norder(_score: :desc) # most relevant first - default\n```\n\n[All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html)\n\nLimit / offset\n\n```ruby\nlimit(20).offset(40)\n```\n\nSelect\n\n```ruby\nselect(:name)\n```\n\n[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering)\n\n### Results\n\nSearches return a `Searchkick::Relation` object. This responds like an array to most methods.\n\n```ruby\nresults = Product.search(\"milk\")\nresults.size\nresults.any?\nresults.each { |result| ... }\n```\n\nBy default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:\n\n```ruby\nProduct.search(\"apples\").load(false)\n```\n\nGet total results\n\n```ruby\nresults.total_count\n```\n\nGet the time the search took (in milliseconds)\n\n```ruby\nresults.took\n```\n\nGet the full response from the search server\n\n```ruby\nresults.response\n```\n\n**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.\n\n### Filtering\n\nEqual\n\n```ruby\nwhere(store_id: 1)\n```\n\nNot equal\n\n```ruby\nwhere.not(store_id: 2)\n```\n\nGreater than (`gt`), less than (`lt`), greater than or equal (`gte`), less than or equal (`lte`)\n\n```ruby\nwhere(expires_at: {gt: Time.now})\n```\n\nRange\n\n```ruby\nwhere(orders_count: 1..10)\n```\n\nIn\n\n```ruby\nwhere(aisle_id: [25, 30])\n```\n\nNot in\n\n```ruby\nwhere.not(aisle_id: [25, 30])\n```\n\nContains all\n\n```ruby\nwhere(user_ids: {all: [1, 3]})\n```\n\nLike\n\n```ruby\nwhere(category: {like: \"%frozen%\"})\n```\n\nCase-insensitive like\n\n```ruby\nwhere(category: {ilike: \"%frozen%\"})\n```\n\nRegular expression\n\n```ruby\nwhere(category: /frozen .+/)\n```\n\nPrefix\n\n```ruby\nwhere(category: {prefix: \"frozen\"})\n```\n\nExists\n\n```ruby\nwhere(store_id: {exists: true})\n```\n\nCombine filters with OR\n\n```ruby\nwhere(_or: [{in_stock: true}, {backordered: true}])\n```\n\n### Boosting\n\nBoost important fields\n\n```ruby\nfields(\"title^10\", \"description\")\n```\n\nBoost by the value of a field (field must be numeric)\n\n```ruby\nboost_by(:orders_count) # give popular documents a little boost\nboost_by(orders_count: {factor: 10}) # default factor is 1\n```\n\nBoost matching documents\n\n```ruby\nboost_where(user_id: 1)\nboost_where(user_id: {value: 1, factor: 100}) # default factor is 1000\nboost_where(user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}])\n```\n\nBoost by recency\n\n```ruby\nboost_by_recency(created_at: {scale: \"7d\", decay: 0.5})\n```\n\nYou can also boost by:\n\n- [Conversions](#intelligent-search)\n- [Distance](#boost-by-distance)\n\n### Get Everything\n\nUse a `*` for the query.\n\n```ruby\nProduct.search(\"*\")\n```\n\n### Pagination\n\nPlays nicely with kaminari and will_paginate.\n\n```ruby\n# controller\n@products = Product.search(\"milk\").page(params[:page]).per_page(20)\n```\n\nView with kaminari\n\n```erb\n<%= paginate @products %>\n```\n\nView with will_paginate\n\n```erb\n<%= will_paginate @products %>\n```\n\n### Partial Matches\n\nBy default, results must match all words in the query.\n\n```ruby\nProduct.search(\"fresh honey\") # fresh AND honey\n```\n\nTo change this, use:\n\n```ruby\nProduct.search(\"fresh honey\").operator(\"or\") # fresh OR honey\n```\n\nBy default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick word_start: [:name]\nend\n```\n\nAnd to search (after you reindex):\n\n```ruby\nProduct.search(\"back\").fields(:name).match(:word_start)\n```\n\nAvailable options are:\n\nOption | Matches | Example\n--- | --- | ---\n`:word` | entire word | `apple` matches `apple`\n`:word_start` | start of word | `app` matches `apple`\n`:word_middle` | any part of word | `ppl` matches `apple`\n`:word_end` | end of word | `ple` matches `apple`\n`:text_start` | start of text | `gre` matches `green apple`, `app` does not match\n`:text_middle` | any part of text | `een app` matches `green apple`\n`:text_end` | end of text | `ple` matches `green apple`, `een` does not match\n\nThe default is `:word`. The most matches will happen with `:word_middle`.\n\nTo specify different matching for different fields, use:\n\n```ruby\nProduct.search(query).fields({name: :word_start}, {brand: :word_middle})\n```\n\n### Exact Matches\n\nTo match a field exactly (case-sensitive), use:\n\n```ruby\nProduct.search(query).fields({name: :exact})\n```\n\n### Phrase Matches\n\nTo only match the exact order, use:\n\n```ruby\nProduct.search(\"fresh honey\").match(:phrase)\n```\n\n### Stemming and Language\n\nSearchkick stems words by default for better matching. `apple` and `apples` both stem to `appl`, so searches for either term will have the same matches.\n\nSearchkick defaults to English for stemming. To change this, use:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick language: \"german\"\nend\n```\n\nSee the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:\n\n- `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)\n- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html)\n- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html)\n- `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)\n- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html)\n- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html)\n- `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)\n- `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)\n\nYou can also use a Hunspell dictionary for stemming.\n\n```ruby\nclass Product < ApplicationRecord\n searchkick stemmer: {type: \"hunspell\", locale: \"en_US\"}\nend\n```\n\nDisable stemming with:\n\n```ruby\nclass Image < ApplicationRecord\n searchkick stem: false\nend\n```\n\nExclude certain words from stemming with:\n\n```ruby\nclass Image < ApplicationRecord\n searchkick stem_exclusion: [\"apples\"]\nend\n```\n\nOr change how words are stemmed:\n\n```ruby\nclass Image < ApplicationRecord\n searchkick stemmer_override: [\"apples => other\"]\nend\n```\n\n### Synonyms\n\n```ruby\nclass Product < ApplicationRecord\n searchkick search_synonyms: [[\"pop\", \"soda\"], [\"burger\", \"hamburger\"]]\nend\n```\n\nCall `Product.reindex` after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.\n\nFor directional synonyms, use:\n\n```ruby\nsearch_synonyms: [\"lightbulb => halogenlamp\"]\n```\n\n### Dynamic Synonyms\n\nThe above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex. We recommend placing synonyms in a file on the search server (in the `config` directory). This allows you to reload synonyms without reindexing.\n\n```txt\npop, soda\nburger, hamburger\n```\n\nThen use:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick search_synonyms: \"synonyms.txt\"\nend\n```\n\nAnd reload with:\n\n```ruby\nProduct.search_index.reload_synonyms\n```\n\n### Misspellings\n\nBy default, Searchkick handles misspelled queries by returning results with an [edit distance](https://en.wikipedia.org/wiki/Levenshtein_distance) of one.\n\nYou can change this with:\n\n```ruby\nProduct.search(\"zucini\").misspellings(edit_distance: 2) # zucchini\n```\n\nTo prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.\n\n```ruby\nProduct.search(\"zuchini\").misspellings(below: 5)\n```\n\nIf there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.\n\nTurn off misspellings with:\n\n```ruby\nProduct.search(\"zuchini\").misspellings(false) # no zucchini\n```\n\nSpecify which fields can include misspellings with:\n\n```ruby\nProduct.search(\"zucini\").fields(:name, :color).misspellings(fields: [:name])\n```\n\n> When doing this, you must also specify fields to search\n\n### Bad Matches\n\nIf a user searches `butter`, they may also get results for `peanut butter`. To prevent this, use:\n\n```ruby\nProduct.search(\"butter\").exclude(\"peanut butter\")\n```\n\nYou can map queries and terms to exclude with:\n\n```ruby\nexclude_queries = {\n \"butter\" => [\"peanut butter\"],\n \"cream\" => [\"ice cream\", \"whipped cream\"]\n}\n\nProduct.search(query).exclude(exclude_queries[query])\n```\n\nYou can demote results by boosting by a factor less than one:\n\n```ruby\nProduct.search(\"butter\").boost_where(category: {value: \"pantry\", factor: 0.5})\n```\n\n### Emoji\n\nSearch :ice_cream::cake: and get `ice cream cake`!\n\nAdd this line to your application’s Gemfile:\n\n```ruby\ngem \"gemoji-parser\"\n```\n\nAnd use:\n\n```ruby\nProduct.search(\"🍨🍰\").emoji\n```\n\n## Indexing\n\nControl what data is indexed with the `search_data` method. Call `Product.reindex` after changing this method.\n\n```ruby\nclass Product < ApplicationRecord\n belongs_to :department\n\n def search_data\n {\n name: name,\n department_name: department.name,\n on_sale: sale_price.present?\n }\n end\nend\n```\n\nSearchkick uses `find_in_batches` to import documents. To eager load associations, use the `search_import` scope.\n\n```ruby\nclass Product < ApplicationRecord\n scope :search_import, -> { includes(:department) }\nend\n```\n\nBy default, all records are indexed. To control which records are indexed, use the `should_index?` method.\n\n```ruby\nclass Product < ApplicationRecord\n def should_index?\n active # only index active records\n end\nend\n```\n\nIf a reindex is interrupted, you can resume it with:\n\n```ruby\nProduct.reindex(resume: true)\n```\n\nFor large data sets, try [parallel reindexing](#parallel-reindexing).\n\n### To Reindex, or Not to Reindex\n\n#### Reindex\n\n- when you install or upgrade searchkick\n- change the `search_data` method\n- change the `searchkick` method\n\n#### No need to reindex\n\n- app starts\n\n### Strategies\n\nThere are four strategies for keeping the index synced with your database.\n\n1. Inline (default)\n\n Anytime a record is inserted, updated, or deleted\n\n2. Asynchronous\n\n Use background jobs for better performance\n\n ```ruby\n class Product < ApplicationRecord\n searchkick callbacks: :async\n end\n ```\n\n Jobs are added to a queue named `searchkick`.\n\n3. Queuing\n\n Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See [how to set up](#queuing).\n\n4. Manual\n\n Turn off automatic syncing\n\n ```ruby\n class Product < ApplicationRecord\n searchkick callbacks: false\n end\n ```\n\n And reindex a record or relation manually.\n\n ```ruby\n product.reindex\n # or\n store.products.reindex(mode: :async)\n ```\n\nYou can also do bulk updates.\n\n```ruby\nSearchkick.callbacks(:bulk) do\n Product.find_each(&:update_fields)\nend\n```\n\nOr temporarily skip updates.\n\n```ruby\nSearchkick.callbacks(false) do\n Product.find_each(&:update_fields)\nend\n```\n\nOr override the model’s strategy.\n\n```ruby\nproduct.reindex(mode: :async) # :inline or :queue\n```\n\n### Associations\n\nData is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:\n\n```ruby\nclass Image < ApplicationRecord\n belongs_to :product\n\n after_commit :reindex_product\n\n def reindex_product\n product.reindex\n end\nend\n```\n\n### Default Scopes\n\nIf you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:\n\n```ruby\nclass Product < ApplicationRecord\n default_scope { where(deleted_at: nil) }\n\n def should_index?\n deleted_at.nil?\n end\nend\n```\n\nIf you want to index and search filtered records, set:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick unscope: true\nend\n```\n\n## Intelligent Search\n\nThe best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.\n\n```ruby\nProduct.search(\"apple\").track(user_id: current_user.id)\n```\n\n[See the docs](https://github.com/ankane/searchjoy) for how to install and use. Focus on top searches with a low conversion rate.\n\nSearchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.\n\nAdd conversion data with:\n\n```ruby\nclass Product < ApplicationRecord\n has_many :conversions, class_name: \"Searchjoy::Conversion\", as: :convertable\n has_many :searches, class_name: \"Searchjoy::Search\", through: :conversions\n\n searchkick conversions_v2: [:conversions] # name of field\n\n def search_data\n {\n name: name,\n conversions: searches.group(:query).distinct.count(:user_id)\n # {\"ice cream\" => 234, \"chocolate\" => 67, \"cream\" => 2}\n }\n end\nend\n```\n\nReindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set `conversions_v2(false)` in your search calls until the data is reindexed.\n\n### Performant Conversions\n\nA performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:\n\n```ruby\nadd_column :products, :search_conversions, :jsonb\n```\n\nFor MySQL, use `:json`, and for others, use `:text` with a [JSON serializer](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html).\n\nNext, update your model. Create a separate method for conversion data so you can use [partial reindexing](#partial-reindexing).\n\n```ruby\nclass Product < ApplicationRecord\n searchkick conversions_v2: [:conversions]\n\n def search_data\n {\n name: name,\n category: category\n }.merge(conversions_data)\n end\n\n def conversions_data\n {\n conversions: search_conversions || {}\n }\n end\nend\n```\n\nDeploy and reindex your data. For zero downtime deployment, temporarily set `conversions_v2(false)` in your search calls until the data is reindexed.\n\n```ruby\nProduct.reindex\n```\n\nThen, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:\n\n```ruby\nclass UpdateConversionsJob < ApplicationJob\n def perform(class_name, since: nil, update: true, reindex: true)\n model = Searchkick.load_model(class_name)\n\n # get records that have a recent conversion\n recently_converted_ids =\n Searchjoy::Conversion.where(convertable_type: class_name, created_at: since..)\n .order(:convertable_id).distinct.pluck(:convertable_id)\n\n # split into batches\n recently_converted_ids.in_groups_of(1000, false) do |ids|\n if update\n # fetch conversions\n conversions =\n Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)\n .joins(:search).where.not(searchjoy_searches: {user_id: nil})\n .group(:convertable_id, :query).distinct.count(:user_id)\n\n # group by record\n conversions_by_record = {}\n conversions.each do |(id, query), count|\n (conversions_by_record[id] ||= {})[query] = count\n end\n\n # update conversions column\n model.transaction do\n conversions_by_record.each do |id, conversions|\n model.where(id: id).update_all(search_conversions: conversions)\n end\n end\n end\n\n if reindex\n # reindex conversions data\n model.where(id: ids).reindex(:conversions_data, ignore_missing: true)\n end\n end\n end\nend\n```\n\nRun the job:\n\n```ruby\nUpdateConversionsJob.perform_now(\"Product\")\n```\n\nAnd set it up to run daily.\n\n```ruby\nUpdateConversionsJob.perform_later(\"Product\", since: 1.day.ago)\n```\n\n## Personalized Results\n\nOrder results differently for each user. For example, show a user’s previously purchased products before other results.\n\n```ruby\nclass Product < ApplicationRecord\n def search_data\n {\n name: name,\n orderer_ids: orders.pluck(:user_id) # boost this product for these users\n }\n end\nend\n```\n\nReindex and search with:\n\n```ruby\nProduct.search(\"milk\").boost_where(orderer_ids: current_user.id)\n```\n\n## Instant Search / Autocomplete\n\nAutocomplete predicts what a user will type, making the search experience faster and easier.\n\n\n\n**Note:** To autocomplete on search terms rather than results, check out [Autosuggest](https://github.com/ankane/autosuggest).\n\n**Note 2:** If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s *much* faster to load all records into JavaScript and autocomplete there (eliminates network requests).\n\nFirst, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing fast queries.\n\n```ruby\nclass Movie < ApplicationRecord\n searchkick word_start: [:title, :director]\nend\n```\n\nReindex and search with:\n\n```ruby\nMovie.search(\"jurassic pa\").fields(:title).match(:word_start)\n```\n\nUse a front-end library like [typeahead.js](https://twitter.github.io/typeahead.js/) to show the results.\n\n#### Here’s how to make it work with Rails\n\nFirst, add a route and controller action.\n\n```ruby\nclass MoviesController < ApplicationController\n def autocomplete\n render json: Movie.search(params[:query]).fields(\"title^5\", \"director\")\n .match(:word_start).limit(10).load(false).misspellings(below: 5).map(&:title)\n end\nend\n```\n\n**Note:** Use `load(false)` and `misspellings(below: n)` (or `misspellings(false)`) for best performance.\n\nThen add the search box and JavaScript code to a view.\n\n```html\n\n\n\n\n\n```\n\n## Suggestions\n\n\n\n```ruby\nclass Product < ApplicationRecord\n searchkick suggest: [:name] # fields to generate suggestions\nend\n```\n\nReindex and search with:\n\n```ruby\nproducts = Product.search(\"peantu butta\").suggest\nproducts.suggestions # [\"peanut butter\"]\n```\n\n## Aggregations\n\n[Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.\n\n\n\n```ruby\nproducts = Product.search(\"chuck taylor\").aggs(:product_type, :gender, :brand)\nproducts.aggs\n```\n\nBy default, `where` conditions apply to aggregations.\n\n```ruby\nProduct.search(\"wingtips\").where(color: \"brandy\").aggs(:size)\n# aggregations for brandy wingtips are returned\n```\n\nChange this with:\n\n```ruby\nProduct.search(\"wingtips\").where(color: \"brandy\").aggs(:size).smart_aggs(false)\n# aggregations for all wingtips are returned\n```\n\nSet `where` conditions for each aggregation separately with:\n\n```ruby\nProduct.search(\"wingtips\").aggs(size: {where: {color: \"brandy\"}})\n```\n\nLimit\n\n```ruby\nProduct.search(\"apples\").aggs(store_id: {limit: 10})\n```\n\nOrder\n\n```ruby\nProduct.search(\"wingtips\").aggs(color: {order: {\"_key\" => \"asc\"}}) # alphabetically\n```\n\n[All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)\n\nRanges\n\n```ruby\nprice_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]\nProduct.search(\"*\").aggs(price: {ranges: price_ranges})\n```\n\nMinimum document count\n\n```ruby\nProduct.search(\"apples\").aggs(store_id: {min_doc_count: 2})\n```\n\nScript support\n\n```ruby\nProduct.search(\"*\").aggs(color: {script: {source: \"'Color: ' + _value\"}})\n```\n\nDate histogram\n\n```ruby\nProduct.search(\"pear\").aggs(products_per_year: {date_histogram: {field: :created_at, interval: :year}})\n```\n\nFor other aggregation types, including sub-aggregations, use `body_options`:\n\n```ruby\nProduct.search(\"orange\").body_options(aggs: {price: {histogram: {field: :price, interval: 10}}})\n```\n\n## Highlight\n\nSpecify which fields to index with highlighting.\n\n```ruby\nclass Band < ApplicationRecord\n searchkick highlight: [:name]\nend\n```\n\nHighlight the search query in the results.\n\n```ruby\nbands = Band.search(\"cinema\").highlight\n```\n\nView the highlighted fields with:\n\n```ruby\nbands.with_highlights.each do |band, highlights|\n highlights[:name] # \"Two Door Cinema Club\"\nend\n```\n\nTo change the tag, use:\n\n```ruby\nBand.search(\"cinema\").highlight(tag: \"\")\n```\n\nTo highlight and search different fields, use:\n\n```ruby\nBand.search(\"cinema\").fields(:name).highlight(fields: [:description])\n```\n\nBy default, the entire field is highlighted. To get small snippets instead, use:\n\n```ruby\nbands = Band.search(\"cinema\").highlight(fragment_size: 20)\nbands.with_highlights(multiple: true).each do |band, highlights|\n highlights[:name].join(\" and \")\nend\n```\n\nAdditional options can be specified for each field:\n\n```ruby\nBand.search(\"cinema\").fields(:name).highlight(fields: {name: {fragment_size: 200}})\n```\n\nYou can find available highlight options in the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html) or [OpenSearch](https://opensearch.org/docs/latest/search-plugins/searching-data/highlight/) reference.\n\n## Similar Items\n\nFind similar items\n\n```ruby\nproduct = Product.first\nproduct.similar.fields(:name).where(size: \"12 oz\")\n```\n\n## Geospatial Searches\n\n```ruby\nclass Restaurant < ApplicationRecord\n searchkick locations: [:location]\n\n def search_data\n attributes.merge(location: {lat: latitude, lon: longitude})\n end\nend\n```\n\nReindex and search with:\n\n```ruby\nRestaurant.search(\"pizza\").where(location: {near: {lat: 37, lon: -114}, within: \"100mi\"}) # or 160km\n```\n\nBounded by a box\n\n```ruby\nRestaurant.search(\"sushi\").where(location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}})\n```\n\n**Note:** `top_right` and `bottom_left` also work\n\nBounded by a polygon\n\n```ruby\nRestaurant.search(\"dessert\").where(location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}})\n```\n\n### Boost By Distance\n\nBoost results by distance - closer results are boosted more\n\n```ruby\nRestaurant.search(\"noodles\").boost_by_distance(location: {origin: {lat: 37, lon: -122}})\n```\n\nAlso supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay)\n\n```ruby\nRestaurant.search(\"wings\").boost_by_distance(location: {origin: {lat: 37, lon: -122}, function: \"linear\", scale: \"30mi\", decay: 0.5})\n```\n\n### Geo Shapes\n\nYou can also index and search geo shapes.\n\n```ruby\nclass Restaurant < ApplicationRecord\n searchkick geo_shape: [:bounds]\n\n def search_data\n attributes.merge(\n bounds: {\n type: \"envelope\",\n coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}]\n }\n )\n end\nend\n```\n\nSee the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html) for details.\n\nFind shapes intersecting with the query shape\n\n```ruby\nRestaurant.search(\"soup\").where(bounds: {geo_shape: {type: \"polygon\", coordinates: [[{lat: 38, lon: -123}, ...]]}})\n```\n\nFalling entirely within the query shape\n\n```ruby\nRestaurant.search(\"salad\").where(bounds: {geo_shape: {type: \"circle\", relation: \"within\", coordinates: {lat: 38, lon: -123}, radius: \"1km\"}})\n```\n\nNot touching the query shape\n\n```ruby\nRestaurant.search(\"burger\").where(bounds: {geo_shape: {type: \"envelope\", relation: \"disjoint\", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}})\n```\n\n## Inheritance\n\nSearchkick supports single table inheritance.\n\n```ruby\nclass Dog < Animal\nend\n```\n\nIn your parent model, set:\n\n```ruby\nclass Animal < ApplicationRecord\n searchkick inheritance: true\nend\n```\n\nThe parent and child model can both reindex.\n\n```ruby\nAnimal.reindex\nDog.reindex # equivalent, all animals reindexed\n```\n\nAnd to search, use:\n\n```ruby\nAnimal.search(\"*\") # all animals\nDog.search(\"*\") # just dogs\nAnimal.search(\"*\").type(Cat, Dog) # just cats and dogs\n```\n\n**Notes:**\n\n1. The `suggest` option retrieves suggestions from the parent at the moment.\n\n ```ruby\n Dog.search(\"airbudd\").suggest # suggestions for all animals\n ```\n2. This relies on a `type` field that is automatically added to the indexed document. Be wary of defining your own `type` field in `search_data`, as it will take precedence.\n\n## Debugging Queries\n\nTo help with debugging queries, you can use:\n\n```ruby\nProduct.search(\"soap\").debug\n```\n\nThis prints useful info to `stdout`.\n\nSee how the search server scores your queries with:\n\n```ruby\nProduct.search(\"soap\").explain.response\n```\n\nSee how the search server tokenizes your queries with:\n\n```ruby\nProduct.search_index.tokens(\"Dish Washer Soap\", analyzer: \"searchkick_index\")\n# [\"dish\", \"dishwash\", \"washer\", \"washersoap\", \"soap\"]\n\nProduct.search_index.tokens(\"dishwasher soap\", analyzer: \"searchkick_search\")\n# [\"dishwashersoap\"] - no match\n\nProduct.search_index.tokens(\"dishwasher soap\", analyzer: \"searchkick_search2\")\n# [\"dishwash\", \"soap\"] - match!!\n```\n\nPartial matches\n\n```ruby\nProduct.search_index.tokens(\"San Diego\", analyzer: \"searchkick_word_start_index\")\n# [\"s\", \"sa\", \"san\", \"d\", \"di\", \"die\", \"dieg\", \"diego\"]\n\nProduct.search_index.tokens(\"dieg\", analyzer: \"searchkick_word_search\")\n# [\"dieg\"] - match!!\n```\n\nSee the [complete list of analyzers](lib/searchkick/index_options.rb#L36).\n\n## Testing\n\nAs you iterate on your search, it’s a good idea to add tests.\n\nFor performance, only enable Searchkick callbacks for the tests that need it.\n\n### Rails\n\nAdd to your `test/test_helper.rb`:\n\n```ruby\nmodule ActiveSupport\n class TestCase\n parallelize_setup do |worker|\n Searchkick.index_suffix = worker\n\n # reindex models for parallel tests\n Product.reindex\n end\n end\nend\n\n# reindex models for non-parallel tests\nProduct.reindex\n\n# and disable callbacks\nSearchkick.disable_callbacks\n```\n\nAnd use:\n\n```ruby\nclass ProductTest < ActiveSupport::TestCase\n setup do\n Searchkick.enable_callbacks\n end\n\n teardown do\n Searchkick.disable_callbacks\n end\n\n test \"search\" do\n Product.create!(name: \"Apple\")\n Product.search_index.refresh\n assert_equal [\"Apple\"], Product.search(\"apple\").map(&:name)\n end\nend\n```\n\n### Minitest\n\nAdd to your `test/test_helper.rb`:\n\n```ruby\n# reindex models\nProduct.reindex\n\n# and disable callbacks\nSearchkick.disable_callbacks\n```\n\nAnd use:\n\n```ruby\nclass ProductTest < Minitest::Test\n def setup\n Searchkick.enable_callbacks\n end\n\n def teardown\n Searchkick.disable_callbacks\n end\n\n def test_search\n Product.create!(name: \"Apple\")\n Product.search_index.refresh\n assert_equal [\"Apple\"], Product.search(\"apple\").map(&:name)\n end\nend\n```\n\n### RSpec\n\nAdd to your `spec/spec_helper.rb`:\n\n```ruby\nRSpec.configure do |config|\n config.before(:suite) do\n # reindex models\n Product.reindex\n\n # and disable callbacks\n Searchkick.disable_callbacks\n end\n\n config.around(:each, search: true) do |example|\n Searchkick.callbacks(nil) do\n example.run\n end\n end\nend\n```\n\nAnd use:\n\n```ruby\ndescribe Product, search: true do\n it \"searches\" do\n Product.create!(name: \"Apple\")\n Product.search_index.refresh\n assert_equal [\"Apple\"], Product.search(\"apple\").map(&:name)\n end\nend\n```\n\n### Factory Bot\n\nDefine a trait for each model:\n\n```ruby\nFactoryBot.define do\n factory :product do\n trait :reindex do\n after(:create) do |product, _|\n product.reindex(refresh: true)\n end\n end\n end\nend\n```\n\nAnd use:\n\n```ruby\nFactoryBot.create(:product, :reindex)\n```\n\n### GitHub Actions\n\nCheck out [setup-elasticsearch](https://github.com/ankane/setup-elasticsearch) for an easy way to install Elasticsearch:\n\n```yml\n - uses: ankane/setup-elasticsearch@v1\n```\n\nAnd [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy way to install OpenSearch:\n\n```yml\n - uses: ankane/setup-opensearch@v1\n```\n\n## Deployment\n\nFor the search server, Searchkick uses `ENV[\"ELASTICSEARCH_URL\"]` for Elasticsearch and `ENV[\"OPENSEARCH_URL\"]` for OpenSearch. This defaults to `http://localhost:9200`.\n\n- [Elastic Cloud](#elastic-cloud)\n- [Amazon OpenSearch Service](#amazon-opensearch-service)\n- [Heroku](#heroku)\n- [Self-Hosted and Other](#self-hosted-and-other)\n\n### Elastic Cloud\n\nCreate an initializer `config/initializers/elasticsearch.rb` with:\n\n```ruby\nENV[\"ELASTICSEARCH_URL\"] = \"https://user:password@host:port\"\n```\n\nThen deploy and reindex:\n\n```sh\nrake searchkick:reindex:all\n```\n\n### Amazon OpenSearch Service\n\nCreate an initializer `config/initializers/opensearch.rb` with:\n\n```ruby\nENV[\"OPENSEARCH_URL\"] = \"https://es-domain-1234.us-east-1.es.amazonaws.com:443\"\n```\n\nTo use signed requests, include in your Gemfile:\n\n```ruby\ngem \"faraday_middleware-aws-sigv4\"\n```\n\nand add to your initializer:\n\n```ruby\nSearchkick.aws_credentials = {\n access_key_id: ENV[\"AWS_ACCESS_KEY_ID\"],\n secret_access_key: ENV[\"AWS_SECRET_ACCESS_KEY\"],\n region: \"us-east-1\"\n}\n```\n\nThen deploy and reindex:\n\n```sh\nrake searchkick:reindex:all\n```\n\n### Heroku\n\nChoose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).\n\nFor Elasticsearch on Bonsai:\n\n```sh\nheroku addons:create bonsai\nheroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`\n```\n\nFor OpenSearch on Bonsai:\n\n```sh\nheroku addons:create bonsai --engine=opensearch\nheroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`\n```\n\nFor SearchBox:\n\n```sh\nheroku addons:create searchbox:starter\nheroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`\n```\n\nFor Elastic Cloud (previously Found):\n\n```sh\nheroku addons:create foundelasticsearch\nheroku addons:open foundelasticsearch\n```\n\nVisit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:\n\n```sh\nheroku config:get FOUNDELASTICSEARCH_URL\n```\n\nAnd add `elastic:password@` right after `https://` and add port `9243` at the end:\n\n```sh\nheroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws.found.io:9243\n```\n\nThen deploy and reindex:\n\n```sh\nheroku run rake searchkick:reindex:all\n```\n\n### Self-Hosted and Other\n\nCreate an initializer with:\n\n```ruby\nENV[\"ELASTICSEARCH_URL\"] = \"https://user:password@host:port\"\n# or\nENV[\"OPENSEARCH_URL\"] = \"https://user:password@host:port\"\n```\n\nThen deploy and reindex:\n\n```sh\nrake searchkick:reindex:all\n```\n\n### Data Protection\n\nWe recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to the search server.\n\nBonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.\n\n### Automatic Failover\n\nCreate an initializer with multiple hosts:\n\n```ruby\nENV[\"ELASTICSEARCH_URL\"] = \"https://user:password@host1,https://user:password@host2\"\n# or\nENV[\"OPENSEARCH_URL\"] = \"https://user:password@host1,https://user:password@host2\"\n```\n\n### Client Options\n\nCreate an initializer with:\n\n```ruby\nSearchkick.client_options[:reload_connections] = true\n```\n\nSee the docs for [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/advanced-config.html) or [Opensearch](https://rubydoc.info/gems/opensearch-transport#configuration) for a complete list of options.\n\n### Lograge\n\nAdd the following to `config/environments/production.rb`:\n\n```ruby\nconfig.lograge.custom_options = lambda do |event|\n options = {}\n options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0\n options\nend\n```\n\nSee [Production Rails](https://github.com/ankane/production_rails) for other good practices.\n\n## Performance\n\n### Persistent HTTP Connections\n\nSignificantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.\n\n```ruby\ngem \"typhoeus\"\n```\n\nTo reduce log noise, create an initializer with:\n\n```ruby\nEthon.logger = Logger.new(nil)\n```\n\n### Searchable Fields\n\nBy default, all string fields are searchable (can be used in `fields` option). Speed up indexing and reduce index size by only making some fields searchable.\n\n```ruby\nclass Product < ApplicationRecord\n searchkick searchable: [:name]\nend\n```\n\n### Filterable Fields\n\nBy default, all string fields are filterable (can be used in `where` option). Speed up indexing and reduce index size by only making some fields filterable.\n\n```ruby\nclass Product < ApplicationRecord\n searchkick filterable: [:brand]\nend\n```\n\n**Note:** Non-string fields are always filterable and should not be passed to this option.\n\n### Parallel Reindexing\n\nFor large data sets, you can use background jobs to parallelize reindexing.\n\n```ruby\nProduct.reindex(mode: :async)\n# {index_name: \"products_production_20250111210018065\"}\n```\n\nOnce the jobs complete, promote the new index with:\n\n```ruby\nProduct.search_index.promote(index_name)\n```\n\nYou can optionally track the status with Redis:\n\n```ruby\nSearchkick.redis = Redis.new\n```\n\nAnd use:\n\n```ruby\nSearchkick.reindex_status(index_name)\n```\n\nYou can also have Searchkick wait for reindexing to complete\n\n```ruby\nProduct.reindex(mode: :async, wait: true)\n```\n\nYou can use your background job framework to control concurrency. For Solid Queue, create an initializer with:\n\n```ruby\nmodule SearchkickBulkReindexConcurrency\n extend ActiveSupport::Concern\n\n included do\n limits_concurrency to: 3, key: \"\"\n end\nend\n\nRails.application.config.after_initialize do\n Searchkick::BulkReindexJob.include(SearchkickBulkReindexConcurrency)\nend\n```\n\nThis will allow only 3 jobs to run at once.\n\n### Refresh Interval\n\nYou can specify a longer refresh interval while reindexing to increase performance.\n\n```ruby\nProduct.reindex(mode: :async, refresh_interval: \"30s\")\n```\n\n**Note:** This only makes a noticeable difference with parallel reindexing.\n\nWhen promoting, have it restored to the value in your mapping (defaults to `1s`).\n\n```ruby\nProduct.search_index.promote(index_name, update_refresh_interval: true)\n```\n\n### Queuing\n\nPush ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).\n\n```ruby\nSearchkick.redis = ConnectionPool.new { Redis.new }\n```\n\nAnd ask your models to queue updates.\n\n```ruby\nclass Product < ApplicationRecord\n searchkick callbacks: :queue\nend\n```\n\nThen, set up a background job to run.\n\n```ruby\nSearchkick::ProcessQueueJob.perform_later(class_name: \"Product\")\n```\n\nYou can check the queue length with:\n\n```ruby\nProduct.search_index.reindex_queue.length\n```\n\nFor more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync).\n\n### Routing\n\nSearchkick supports [routing](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.\n\n```ruby\nclass Business < ApplicationRecord\n searchkick routing: true\n\n def search_routing\n city_id\n end\nend\n```\n\nReindex and search with:\n\n```ruby\nBusiness.search(\"ice cream\").routing(params[:city_id])\n```\n\n### Partial Reindexing\n\nReindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.\n\n```ruby\nclass Product < ApplicationRecord\n def search_data\n {\n name: name,\n category: category\n }.merge(prices_data)\n end\n\n def prices_data\n {\n price: price,\n sale_price: sale_price\n }\n end\nend\n```\n\nAnd use:\n\n```ruby\nProduct.reindex(:prices_data)\n```\n\nIgnore errors for missing documents with:\n\n```ruby\nProduct.reindex(:prices_data, ignore_missing: true)\n```\n\n## Advanced\n\nSearchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.\n\n### Advanced Mapping\n\nCreate a custom mapping:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick mappings: {\n properties: {\n name: {type: \"keyword\"}\n }\n }\nend\n```\n**Note:** If you use a custom mapping, you'll need to use [custom searching](#advanced-search) as well.\n\nTo keep the mappings and settings generated by Searchkick, use:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick merge_mappings: true, mappings: {...}\nend\n```\n\n### Advanced Search\n\nAnd use the `body` option to search:\n\n```ruby\nproducts = Product.search.body(query: {match: {name: \"milk\"}})\n```\n\nView the response with:\n\n```ruby\nproducts.response\n```\n\nTo modify the query generated by Searchkick, use:\n\n```ruby\nproducts = Product.search(\"milk\").body_options(min_score: 1)\n```\n\nor\n\n```ruby\nproducts =\n Product.search(\"apples\") do |body|\n body[:min_score] = 1\n end\n```\n\n### Client\n\nTo access the `Elasticsearch::Client` or `OpenSearch::Client` directly, use:\n\n```ruby\nSearchkick.client\n```\n\n## Multi Search\n\nTo batch search requests for performance, use:\n\n```ruby\nproducts = Product.search(\"snacks\")\ncoupons = Coupon.search(\"snacks\")\nSearchkick.multi_search([products, coupons])\n```\n\nThen use `products` and `coupons` as typical results.\n\n**Note:** Errors are not raised as with single requests. Use the `error` method on each query to check for errors.\n\n## Multiple Models\n\nSearch across multiple models with:\n\n```ruby\nSearchkick.search(\"milk\").models(Product, Category)\n```\n\nBoost specific models with:\n\n```ruby\nindices_boost(Category => 2, Product => 1)\n```\n\n## Multi-Tenancy\n\nCheck out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.\n\n## Scroll API\n\nSearchkick also supports the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results). Scrolling is not intended for real time user requests, but rather for processing large amounts of data.\n\n```ruby\nProduct.search(\"*\").scroll(\"1m\") do |batch|\n # process batch ...\nend\n```\n\nYou can also scroll batches manually.\n\n```ruby\nproducts = Product.search(\"*\").scroll(\"1m\")\nwhile products.any?\n # process batch ...\n\n products = products.scroll\nend\n\nproducts.clear_scroll\n```\n\n## Deep Paging\n\nBy default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick deep_paging: true\nend\n```\n\nIf you just need an accurate total count, you can instead use:\n\n```ruby\nProduct.search(\"pears\").body_options(track_total_hits: true)\n```\n\n## Nested Data\n\nTo query nested data, use dot notation.\n\n```ruby\nProduct.search(\"san\").fields(\"store.city\").where(\"store.zip_code\" => 12345)\n```\n\n## Nearest Neighbor Search\n\n*Available for Elasticsearch 8.6+ and OpenSearch 2.4+*\n\n```ruby\nclass Product < ApplicationRecord\n searchkick knn: {embedding: {dimensions: 3, distance: \"cosine\"}}\nend\n```\n\nAlso supports `euclidean` and `inner_product`\n\nReindex and search with:\n\n```ruby\nProduct.search.knn(field: :embedding, vector: [1, 2, 3]).limit(10)\n```\n\n### HNSW Options\n\nNearest neighbor search uses [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) for indexing.\n\nSpecify `m` and `ef_construction`\n\n```ruby\nclass Product < ApplicationRecord\n searchkick knn: {embedding: {dimensions: 3, distance: \"cosine\", m: 16, ef_construction: 100}}\nend\n```\n\nSpecify `ef_search`\n\n```ruby\nProduct.search.knn(field: :embedding, vector: [1, 2, 3], ef_search: 40).limit(10)\n```\n\n## Semantic Search\n\nFirst, add [nearest neighbor search](#nearest-neighbor-search) to your model\n\n```ruby\nclass Product < ApplicationRecord\n searchkick knn: {embedding: {dimensions: 768, distance: \"cosine\"}}\nend\n```\n\nGenerate an embedding for each record (you can use an external service or a library like [Informers](https://github.com/ankane/informers))\n\n```ruby\nembed = Informers.pipeline(\"embedding\", \"Snowflake/snowflake-arctic-embed-m-v1.5\")\nembed_options = {model_output: \"sentence_embedding\", pooling: \"none\"} # specific to embedding model\n\nProduct.find_each do |product|\n embedding = embed.(product.name, **embed_options)\n product.update!(embedding: embedding)\nend\n```\n\nFor search, generate an embedding for the query (the query prefix is specific to the [embedding model](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5))\n\n```ruby\nquery_prefix = \"Represent this sentence for searching relevant passages: \"\nquery_embedding = embed.(query_prefix + query, **embed_options)\n```\n\nAnd perform nearest neighbor search\n\n```ruby\nProduct.search.knn(field: :embedding, vector: query_embedding).limit(20)\n```\n\nSee a [full example](examples/semantic.rb)\n\n## Hybrid Search\n\nPerform keyword search and semantic search in parallel\n\n```ruby\nkeyword_search = Product.search(query).limit(20)\nsemantic_search = Product.search.knn(field: :embedding, vector: query_embedding).limit(20)\nSearchkick.multi_search([keyword_search, semantic_search])\n```\n\nTo combine the results, use Reciprocal Rank Fusion (RRF)\n\n```ruby\nSearchkick::Reranking.rrf(keyword_search, semantic_search).first(5)\n```\n\nOr a reranking model\n\n```ruby\nrerank = Informers.pipeline(\"reranking\", \"mixedbread-ai/mxbai-rerank-xsmall-v1\")\nresults = (keyword_search.to_a + semantic_search.to_a).uniq\nrerank.(query, results.map(&:name)).first(5).map { |v| results[v[:doc_id]] }\n```\n\nSee a [full example](examples/hybrid.rb)\n\n## Reference\n\nReindex one record\n\n```ruby\nproduct = Product.find(1)\nproduct.reindex\n```\n\nReindex multiple records\n\n```ruby\nProduct.where(store_id: 1).reindex\n```\n\nReindex associations\n\n```ruby\nstore.products.reindex\n```\n\nRemove old indices\n\n```ruby\nProduct.search_index.clean_indices\n```\n\nUse custom settings\n\n```ruby\nclass Product < ApplicationRecord\n searchkick settings: {number_of_shards: 3}\nend\n```\n\nUse a different index name\n\n```ruby\nclass Product < ApplicationRecord\n searchkick index_name: \"products_v2\"\nend\n```\n\nUse a dynamic index name\n\n```ruby\nclass Product < ApplicationRecord\n searchkick index_name: -> { \"#{name.tableize}-#{I18n.locale}\" }\nend\n```\n\nPrefix the index name\n\n```ruby\nclass Product < ApplicationRecord\n searchkick index_prefix: \"datakick\"\nend\n```\n\nFor all models\n\n```ruby\nSearchkick.index_prefix = \"datakick\"\n```\n\nUse a different term for boosting by conversions\n\n```ruby\nProduct.search(\"banana\").conversions_v2(term: \"organic banana\")\n```\n\nDefine multiple conversion fields\n\n```ruby\nclass Product < ApplicationRecord\n has_many :searches, class_name: \"Searchjoy::Search\"\n\n searchkick conversions_v2: [\"unique_conversions\", \"total_conversions\"]\n\n def search_data\n {\n name: name,\n unique_conversions: searches.group(:query).distinct.count(:user_id),\n total_conversions: searches.group(:query).count\n }\n end\nend\n```\n\nAnd specify which to use\n\n```ruby\nProduct.search(\"banana\") # boost by both fields (default)\nProduct.search(\"banana\").conversions_v2(\"total_conversions\") # only boost by total_conversions\nProduct.search(\"banana\").conversions_v2(false) # no conversion boosting\n```\n\nChange timeout\n\n```ruby\nSearchkick.timeout = 15 # defaults to 10\n```\n\nSet a lower timeout for searches\n\n```ruby\nSearchkick.search_timeout = 3\n```\n\nChange the search method name\n\n```ruby\nSearchkick.search_method_name = :lookup\n```\n\nChange the queue name\n\n```ruby\nSearchkick.queue_name = :search_reindex # defaults to :searchkick\n```\n\nChange the queue name or priority for a model\n\n```ruby\nclass Product < ApplicationRecord\n searchkick job_options: {queue: \"critical\", priority: 10}\nend\n```\n\nChange the queue name or priority for a specific call\n\n```ruby\nProduct.reindex(mode: :async, job_options: {queue: \"critical\", priority: 10})\n```\n\nChange the parent job\n\n```ruby\nSearchkick.parent_job = \"ApplicationJob\" # defaults to \"ActiveJob::Base\"\n```\n\nEager load associations\n\n```ruby\nProduct.search(\"milk\").includes(:brand, :stores)\n```\n\nEager load different associations by model\n\n```ruby\nSearchkick.search(\"*\").models(Product, Store).model_includes(Product => [:store], Store => [:product])\n```\n\nRun additional scopes on results\n\n```ruby\nProduct.search(\"milk\").scope_results(->(r) { r.with_attached_images })\n```\n\nSet opaque id for slow logs\n\n```ruby\nProduct.search(\"milk\").opaque_id(\"some-id\")\n# or\nSearchkick.multi_search(searches, opaque_id: \"some-id\")\n```\n\nSpecify default fields to search\n\n```ruby\nclass Product < ApplicationRecord\n searchkick default_fields: [:name]\nend\n```\n\nTurn off special characters\n\n```ruby\nclass Product < ApplicationRecord\n # A will not match Ä\n searchkick special_characters: false\nend\n```\n\nTurn on stemming for conversions\n\n```ruby\nclass Product < ApplicationRecord\n searchkick stem_conversions: true\nend\n```\n\nMake search case-sensitive\n\n```ruby\nclass Product < ApplicationRecord\n searchkick case_sensitive: true\nend\n```\n\n**Note:** If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.\n\nChange import batch size\n\n```ruby\nclass Product < ApplicationRecord\n searchkick batch_size: 200 # defaults to 1000\nend\n```\n\nCreate index without importing\n\n```ruby\nProduct.reindex(import: false)\n```\n\nUse a different id\n\n```ruby\nclass Product < ApplicationRecord\n def search_document_id\n custom_id\n end\nend\n```\n\nAdd [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`\n\n```ruby\nProduct.search(\"carrots\").request_params(search_type: \"dfs_query_then_fetch\")\n```\n\nSet options across all models\n\n```ruby\nSearchkick.model_options = {\n batch_size: 200\n}\n```\n\nReindex conditionally\n\n```ruby\nclass Product < ApplicationRecord\n searchkick callback_options: {if: :search_data_changed?}\n\n def search_data_changed?\n previous_changes.include?(\"name\")\n end\nend\n```\n\nReindex all models - Rails only\n\n```sh\nrake searchkick:reindex:all\n```\n\nTurn on misspellings after a certain number of characters\n\n```ruby\nProduct.search(\"api\").misspellings(prefix_length: 2) # api, apt, no ahi\n```\n\nBigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.\n\n```ruby\nclass Product < ApplicationRecord\n def search_data\n {\n units: units.to_s(\"F\")\n }\n end\nend\n```\n\n## Gotchas\n\n### Consistency\n\nElasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.\n\n```ruby\nproduct.save!\nProduct.search_index.refresh\n```\n\n### Inconsistent Scores\n\nDue to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick settings: {number_of_shards: 1}\nend\n```\n\nFor convenience, this is set by default in the test environment.\n\n## Upgrading\n\n### 6.0\n\nSearchkick 6 brings a new query builder API:\n\n```ruby\nProduct.search(\"apples\").where(in_stock: true).limit(10).offset(50)\n```\n\nAll existing options can be used as methods, or you can continue to use the existing API.\n\nThis release also significantly improves the performance of searches when using conversions. To upgrade conversions without downtime, add `conversions_v2` to your model and an additional field to `search_data`:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick conversions: [:conversions], conversions_v2: [:conversions_v2]\n\n def search_data\n conversions = searches.group(:query).distinct.count(:user_id)\n {\n conversions: conversions,\n conversions_v2: conversions\n }\n end\nend\n```\n\nReindex, then remove `conversions`:\n\n```ruby\nclass Product < ApplicationRecord\n searchkick conversions_v2: [:conversions_v2]\n\n def search_data\n {\n conversions_v2: searches.group(:query).distinct.count(:user_id)\n }\n end\nend\n```\n\nOther improvements include the option to ignore errors for missing documents with partial reindexing and more customization for background jobs. Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md) for the full list of changes.\n\n## History\n\nView the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md)\n\n## Thanks\n\nThanks to Karel Minarik for [Elasticsearch Ruby](https://github.com/elasticsearch/elasticsearch-ruby) and [Tire](https://github.com/karmi/retire), Jaroslav Kalistsuk for [zero downtime reindexing](https://gist.github.com/jarosan/3124884), and Alex Leschenko for [Elasticsearch autocomplete](https://github.com/leschenko/elasticsearch_autocomplete).\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/searchkick/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/searchkick/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development:\n\n```sh\ngit clone https://github.com/ankane/searchkick.git\ncd searchkick\nbundle install\nbundle exec rake test\n```\n\nFeel free to open an issue to get feedback on your idea before spending too much time on it.\n"
},
{
"path": "Rakefile",
"content": "require \"bundler/gem_tasks\"\nrequire \"rake/testtask\"\n\nRake::TestTask.new do |t|\n t.pattern = \"test/**/*_test.rb\"\nend\n\ntask default: :test\n\n# to test in parallel, uncomment and run:\n# rake parallel:test\n# require \"parallel_tests/tasks\"\n"
},
{
"path": "benchmark/Gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"../\"\n\ngem \"sqlite3\"\ngem \"pg\"\ngem \"activerecord\", \"~> 8.0.0\"\ngem \"activejob\"\ngem \"elasticsearch\"\n# gem \"opensearch-ruby\"\ngem \"redis\"\ngem \"sidekiq\"\n\n# performance\ngem \"typhoeus\"\ngem \"oj\"\ngem \"json\"\n\n# profiling\ngem \"ruby-prof\"\ngem \"allocation_stats\"\ngem \"get_process_mem\"\ngem \"memory_profiler\"\n# gem \"allocation_tracer\"\ngem \"benchmark-ips\"\n"
},
{
"path": "benchmark/index.rb",
"content": "require \"bundler/setup\"\nBundler.require(:default)\nrequire \"active_record\"\nrequire \"active_job\"\nrequire \"benchmark\"\nrequire \"active_support/notifications\"\n\nActiveSupport::Notifications.subscribe \"request.searchkick\" do |*args|\n event = ActiveSupport::Notifications::Event.new(*args)\n # puts \"Import: #{event.duration.round}ms\"\nend\n\n# ActiveJob::Base.queue_adapter = :sidekiq\n\nclass SearchSerializer\n def dump(object)\n JSON.generate(object)\n end\nend\n\n# Elasticsearch::API.settings[:serializer] = SearchSerializer.new\n# OpenSearch::API.settings[:serializer] = SearchSerializer.new\n\nSearchkick.redis = Redis.new\n\nActiveRecord.default_timezone = :utc\nActiveRecord::Base.time_zone_aware_attributes = true\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \"/tmp/searchkick\"\n# ActiveRecord::Base.establish_connection \"postgresql://localhost/searchkick_bench\"\n# ActiveRecord::Base.logger = Logger.new(STDOUT)\n\nActiveJob::Base.logger = nil\n\nclass Product < ActiveRecord::Base\n searchkick batch_size: 1000\n\n def search_data\n {\n name: name,\n color: color,\n store_id: store_id\n }\n end\nend\n\nif ENV[\"SETUP\"]\n total_docs = 100000\n\n ActiveRecord::Schema.define do\n create_table :products, force: :cascade do |t|\n t.string :name\n t.string :color\n t.integer :store_id\n end\n end\n\n records = []\n total_docs.times do |i|\n records << {\n name: \"Product #{i}\",\n color: [\"red\", \"blue\"].sample,\n store_id: rand(10)\n }\n end\n Product.insert_all(records)\n\n puts \"Imported\"\nend\n\nresult = nil\nreport = nil\nstats = nil\n\nProduct.searchkick_index.delete rescue nil\n\nGC.start\nGC.disable\nstart_mem = GetProcessMem.new.mb\n\ntime =\n Benchmark.realtime do\n # result = RubyProf::Profile.profile do\n # report = MemoryProfiler.report do\n # stats = AllocationStats.trace do\n reindex = Product.reindex #(async: true)\n # p reindex\n # end\n\n # 60.times do |i|\n # if reindex.is_a?(Hash)\n # docs = Searchkick::Index.new(reindex[:index_name]).total_docs\n # else\n # docs = Product.searchkick_index.total_docs\n # end\n # puts \"#{i}: #{docs}\"\n # if docs == total_docs\n # break\n # end\n # p Searchkick.reindex_status(reindex[:index_name]) if reindex.is_a?(Hash)\n # sleep(1)\n # # Product.searchkick_index.refresh\n # end\n end\n\nputs \"Time: #{time.round(1)}s\"\n\nif result\n printer = RubyProf::GraphPrinter.new(result)\n printer.print(STDOUT, min_percent: 5)\nend\n\nif report\n puts report.pretty_print\nend\n\nif stats\n puts result.allocations(alias_paths: true).group_by(:sourcefile, :class).to_text\nend\n"
},
{
"path": "benchmark/relation.rb",
"content": "require \"bundler/setup\"\nBundler.require(:default)\nrequire \"active_record\"\n\nclass Product < ActiveRecord::Base\n searchkick\nend\n\nProduct.all # initial Active Record allocations\n\nstats = AllocationStats.trace do\n Product.search(\"apples\").where(store_id: 1).where(in_stock: true).order(:name).limit(10).offset(50)\nend\nputs stats.allocations(alias_paths: true).to_text\n"
},
{
"path": "benchmark/search.rb",
"content": "require \"bundler/setup\"\nBundler.require(:default)\nrequire \"active_record\"\nrequire \"benchmark/ips\"\n\nActiveRecord.default_timezone = :utc\nActiveRecord::Base.time_zone_aware_attributes = true\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \"/tmp/searchkick\"\n\nclass Product < ActiveRecord::Base\n searchkick batch_size: 1000\n\n def search_data\n {\n name: name,\n color: color,\n store_id: store_id\n }\n end\nend\n\nif ENV[\"SETUP\"]\n total_docs = 1000000\n\n ActiveRecord::Schema.define do\n create_table :products, force: :cascade do |t|\n t.string :name\n t.string :color\n t.integer :store_id\n end\n end\n\n records = []\n total_docs.times do |i|\n records << {\n name: \"Product #{i}\",\n color: [\"red\", \"blue\"].sample,\n store_id: rand(10)\n }\n end\n Product.insert_all(records)\n\n puts \"Imported\"\n\n Product.reindex\n\n puts \"Reindexed\"\nend\n\nquery = Product.search(\"product\", fields: [:name], where: {color: \"red\", store_id: 5}, limit: 10000, load: false)\npp query.body.as_json\nputs\n\nBenchmark.ips do |x|\n x.report { query.dup.load }\nend\n"
},
{
"path": "examples/Gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"activerecord\"\ngem \"elasticsearch\"\ngem \"informers\"\ngem \"opensearch-ruby\"\ngem \"sqlite3\"\n"
},
{
"path": "examples/hybrid.rb",
"content": "require \"bundler/setup\"\nrequire \"active_record\"\nrequire \"elasticsearch\" # or \"opensearch-ruby\"\nrequire \"informers\"\nrequire \"searchkick\"\n\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \":memory:\"\nActiveRecord::Schema.verbose = false\nActiveRecord::Schema.define do\n create_table :products do |t|\n t.string :name\n t.json :embedding\n end\nend\n\nclass Product < ActiveRecord::Base\n searchkick knn: {embedding: {dimensions: 768, distance: \"cosine\"}}\nend\n\nProduct.reindex\n\nProduct.create!(name: \"Breakfast cereal\")\nProduct.create!(name: \"Ice cream\")\nProduct.create!(name: \"Eggs\")\n\nembed = Informers.pipeline(\"embedding\", \"Snowflake/snowflake-arctic-embed-m-v1.5\")\nembed_options = {model_output: \"sentence_embedding\", pooling: \"none\"} # specific to embedding model\n\nProduct.find_each do |product|\n embedding = embed.(product.name, **embed_options)\n product.update!(embedding: embedding)\nend\n\nProduct.search_index.refresh\n\nquery = \"breakfast\"\nkeyword_search = Product.search(query, limit: 20)\n\n# the query prefix is specific to the embedding model (https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5)\nquery_prefix = \"Represent this sentence for searching relevant passages: \"\nquery_embedding = embed.(query_prefix + query, **embed_options)\nsemantic_search = Product.search(knn: {field: :embedding, vector: query_embedding}, limit: 20)\n\nSearchkick.multi_search([keyword_search, semantic_search])\n\n# to combine the results, use Reciprocal Rank Fusion (RRF)\np Searchkick::Reranking.rrf(keyword_search, semantic_search).first(5).map { |v| v[:result].name }\n\n# or a reranking model\nrerank = Informers.pipeline(\"reranking\", \"mixedbread-ai/mxbai-rerank-xsmall-v1\")\nresults = (keyword_search.to_a + semantic_search.to_a).uniq\np rerank.(query, results.map(&:name)).first(5).map { |v| results[v[:doc_id]] }.map(&:name)\n"
},
{
"path": "examples/semantic.rb",
"content": "require \"bundler/setup\"\nrequire \"active_record\"\nrequire \"elasticsearch\" # or \"opensearch-ruby\"\nrequire \"informers\"\nrequire \"searchkick\"\n\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \":memory:\"\nActiveRecord::Schema.verbose = false\nActiveRecord::Schema.define do\n create_table :products do |t|\n t.string :name\n t.json :embedding\n end\nend\n\nclass Product < ActiveRecord::Base\n searchkick knn: {embedding: {dimensions: 768, distance: \"cosine\"}}\nend\n\nProduct.reindex\n\nProduct.create!(name: \"Cereal\")\nProduct.create!(name: \"Ice cream\")\nProduct.create!(name: \"Eggs\")\n\nembed = Informers.pipeline(\"embedding\", \"Snowflake/snowflake-arctic-embed-m-v1.5\")\nembed_options = {model_output: \"sentence_embedding\", pooling: \"none\"} # specific to embedding model\n\nProduct.find_each do |product|\n embedding = embed.(product.name, **embed_options)\n product.update!(embedding: embedding)\nend\n\nProduct.search_index.refresh\n\nquery = \"breakfast\"\n\n# the query prefix is specific to the embedding model (https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5)\nquery_prefix = \"Represent this sentence for searching relevant passages: \"\nquery_embedding = embed.(query_prefix + query, **embed_options)\npp Product.search(knn: {field: :embedding, vector: query_embedding}, limit: 20).map(&:name)\n"
},
{
"path": "gemfiles/activerecord72.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"sqlite3\"\ngem \"activerecord\", \"~> 7.2.0\"\ngem \"actionpack\", \"~> 7.2.0\"\ngem \"activejob\", \"~> 7.2.0\", require: \"active_job\"\ngem \"elasticsearch\", \"~> 8\"\ngem \"redis-client\"\ngem \"connection_pool\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\n"
},
{
"path": "gemfiles/activerecord80.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"sqlite3\"\ngem \"activerecord\", \"~> 8.0.0\"\ngem \"actionpack\", \"~> 8.0.0\"\ngem \"activejob\", \"~> 8.0.0\", require: \"active_job\"\ngem \"elasticsearch\", \"~> 9\"\ngem \"redis-client\"\ngem \"connection_pool\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\n"
},
{
"path": "gemfiles/mongoid8.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"mongoid\", \"~> 8\"\ngem \"activejob\", require: \"active_job\"\ngem \"redis\"\ngem \"elasticsearch\", \"~> 8\"\ngem \"actionpack\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\ngem \"ostruct\" # for mongoid\n"
},
{
"path": "gemfiles/mongoid9.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"mongoid\", \"~> 9\"\ngem \"activejob\", require: \"active_job\"\ngem \"redis\"\ngem \"elasticsearch\", \"~> 9\"\ngem \"actionpack\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\ngem \"ostruct\" # for mongoid\n"
},
{
"path": "gemfiles/opensearch2.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"sqlite3\"\ngem \"activerecord\", \"~> 7.2.0\"\ngem \"actionpack\", \"~> 7.2.0\"\ngem \"activejob\", \"~> 7.2.0\", require: \"active_job\"\ngem \"opensearch-ruby\", \"~> 2\"\ngem \"redis-client\"\ngem \"connection_pool\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\ngem \"parallel_tests\"\ngem \"typhoeus\"\n"
},
{
"path": "gemfiles/opensearch3.gemfile",
"content": "source \"https://rubygems.org\"\n\ngemspec path: \"..\"\n\ngem \"rake\"\ngem \"minitest\"\ngem \"sqlite3\"\ngem \"activerecord\", \"~> 8.0.0\"\ngem \"actionpack\", \"~> 8.0.0\"\ngem \"activejob\", \"~> 8.0.0\", require: \"active_job\"\ngem \"opensearch-ruby\", \"~> 3\"\ngem \"redis-client\"\ngem \"connection_pool\"\ngem \"kaminari\"\ngem \"gemoji-parser\"\ngem \"parallel_tests\"\ngem \"typhoeus\"\n"
},
{
"path": "lib/searchkick/bulk_reindex_job.rb",
"content": "module Searchkick\n class BulkReindexJob < Searchkick.parent_job.constantize\n queue_as { Searchkick.queue_name }\n\n def perform(class_name:, record_ids: nil, index_name: nil, method_name: nil, batch_id: nil, min_id: nil, max_id: nil, ignore_missing: nil)\n model = Searchkick.load_model(class_name)\n index = model.searchkick_index(name: index_name)\n\n record_ids ||= min_id..max_id\n\n relation = Searchkick.scope(model)\n relation = Searchkick.load_records(relation, record_ids)\n relation = relation.search_import if relation.respond_to?(:search_import)\n\n RecordIndexer.new(index).reindex(relation, mode: :inline, method_name: method_name, ignore_missing: ignore_missing, full: false)\n RelationIndexer.new(index).batch_completed(batch_id) if batch_id\n end\n end\nend\n"
},
{
"path": "lib/searchkick/controller_runtime.rb",
"content": "# based on https://gist.github.com/mnutt/566725\nmodule Searchkick\n module ControllerRuntime\n extend ActiveSupport::Concern\n\n protected\n\n attr_internal :searchkick_runtime\n\n def process_action(action, *args)\n # We also need to reset the runtime before each action\n # because of queries in middleware or in cases we are streaming\n # and it won't be cleaned up by the method below.\n Searchkick::LogSubscriber.reset_runtime\n super\n end\n\n def cleanup_view_runtime\n searchkick_rt_before_render = Searchkick::LogSubscriber.reset_runtime\n runtime = super\n searchkick_rt_after_render = Searchkick::LogSubscriber.reset_runtime\n self.searchkick_runtime = searchkick_rt_before_render + searchkick_rt_after_render\n runtime - searchkick_rt_after_render\n end\n\n def append_info_to_payload(payload)\n super\n payload[:searchkick_runtime] = (searchkick_runtime || 0) + Searchkick::LogSubscriber.reset_runtime\n end\n\n module ClassMethods\n def log_process_action(payload)\n messages = super\n runtime = payload[:searchkick_runtime]\n messages << (\"Searchkick: %.1fms\" % runtime.to_f) if runtime.to_f > 0\n messages\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/hash_wrapper.rb",
"content": "module Searchkick\n class HashWrapper\n def initialize(attributes)\n @attributes = attributes\n end\n\n def [](name)\n @attributes[name.to_s]\n end\n\n def to_h\n @attributes\n end\n\n def as_json(...)\n @attributes.as_json(...)\n end\n\n def to_json(...)\n @attributes.to_json(...)\n end\n\n def method_missing(name, ...)\n if @attributes.key?(name.to_s)\n self[name]\n else\n super\n end\n end\n\n def respond_to_missing?(name, ...)\n @attributes.key?(name.to_s) || super\n end\n\n def inspect\n attributes = @attributes.reject { |k, v| k[0] == \"_\" }.map { |k, v| \"#{k}: #{v.inspect}\" }\n attributes.unshift(attributes.pop) # move id to start\n \"#<#{self.class.name} #{attributes.join(\", \")}>\"\n end\n end\nend\n"
},
{
"path": "lib/searchkick/index.rb",
"content": "module Searchkick\n class Index\n attr_reader :name, :options\n\n def initialize(name, options = {})\n @name = name\n @options = options\n @klass_document_type = {} # cache\n end\n\n def index_options\n IndexOptions.new(self).index_options\n end\n\n def create(body = {})\n client.indices.create index: name, body: body\n end\n\n def delete\n if alias_exists?\n # can't call delete directly on aliases in ES 6\n indices = client.indices.get_alias(name: name).keys\n client.indices.delete index: indices\n else\n client.indices.delete index: name\n end\n end\n\n def exists?\n client.indices.exists index: name\n end\n\n def refresh\n client.indices.refresh index: name\n end\n\n def alias_exists?\n client.indices.exists_alias name: name\n end\n\n # call to_h for consistent results between elasticsearch gem 7 and 8\n # could do for all API calls, but just do for ones where return value is focus for now\n def mapping\n client.indices.get_mapping(index: name).to_h\n end\n\n # call to_h for consistent results between elasticsearch gem 7 and 8\n def settings\n client.indices.get_settings(index: name).to_h\n end\n\n def refresh_interval\n index_settings[\"refresh_interval\"]\n end\n\n def update_settings(settings)\n client.indices.put_settings index: name, body: settings\n end\n\n def tokens(text, options = {})\n client.indices.analyze(body: {text: text}.merge(options), index: name)[\"tokens\"].map { |t| t[\"token\"] }\n end\n\n def total_docs\n response =\n client.search(\n index: name,\n body: {\n query: {match_all: {}},\n size: 0,\n track_total_hits: true\n }\n )\n\n Results.new(nil, response).total_count\n end\n\n def promote(new_name, update_refresh_interval: false)\n if update_refresh_interval\n new_index = Index.new(new_name, @options)\n settings = options[:settings] || {}\n refresh_interval = (settings[:index] && settings[:index][:refresh_interval]) || \"1s\"\n new_index.update_settings(index: {refresh_interval: refresh_interval})\n end\n\n old_indices =\n begin\n client.indices.get_alias(name: name).keys\n rescue => e\n raise e unless Searchkick.not_found_error?(e)\n {}\n end\n actions = old_indices.map { |old_name| {remove: {index: old_name, alias: name}} } + [{add: {index: new_name, alias: name}}]\n client.indices.update_aliases body: {actions: actions}\n end\n alias_method :swap, :promote\n\n def retrieve(record)\n record_data = RecordData.new(self, record).record_data\n\n # remove underscore\n get_options = record_data.to_h { |k, v| [k.to_s.delete_prefix(\"_\").to_sym, v] }\n\n client.get(get_options)[\"_source\"]\n end\n\n def all_indices(unaliased: false)\n indices =\n begin\n if client.indices.respond_to?(:get_alias)\n client.indices.get_alias(index: \"#{name}*\")\n else\n client.indices.get_aliases\n end\n rescue => e\n raise e unless Searchkick.not_found_error?(e)\n {}\n end\n indices = indices.select { |_k, v| v.empty? || v[\"aliases\"].empty? } if unaliased\n indices.select { |k, _v| k =~ /\\A#{Regexp.escape(name)}_\\d{14,17}\\z/ }.keys\n end\n\n # remove old indices that start w/ index_name\n def clean_indices\n indices = all_indices(unaliased: true)\n indices.each do |index|\n Index.new(index).delete\n end\n indices\n end\n\n def store(record)\n notify(record, \"Store\") do\n queue_index([record])\n end\n end\n\n def remove(record)\n notify(record, \"Remove\") do\n queue_delete([record])\n end\n end\n\n def update_record(record, method_name)\n notify(record, \"Update\") do\n queue_update([record], method_name)\n end\n end\n\n def bulk_delete(records)\n return if records.empty?\n\n notify_bulk(records, \"Delete\") do\n queue_delete(records)\n end\n end\n\n def bulk_index(records)\n return if records.empty?\n\n notify_bulk(records, \"Import\") do\n queue_index(records)\n end\n end\n alias_method :import, :bulk_index\n\n def bulk_update(records, method_name, ignore_missing: nil)\n return if records.empty?\n\n notify_bulk(records, \"Update\") do\n queue_update(records, method_name, ignore_missing: ignore_missing)\n end\n end\n\n def search_id(record)\n RecordData.new(self, record).search_id\n end\n\n def document_type(record)\n RecordData.new(self, record).document_type\n end\n\n def similar_record(record, **options)\n options[:per_page] ||= 10\n options[:similar] = [RecordData.new(self, record).record_data]\n options[:models] ||= [record.class] unless options.key?(:model)\n\n Searchkick.search(\"*\", **options)\n end\n\n def reload_synonyms\n if Searchkick.opensearch?\n client.transport.perform_request \"POST\", \"_plugins/_refresh_search_analyzers/#{CGI.escape(name)}\"\n else\n begin\n client.transport.perform_request(\"GET\", \"#{CGI.escape(name)}/_reload_search_analyzers\")\n rescue => e\n raise Error, \"Requires non-OSS version of Elasticsearch\" if Searchkick.not_allowed_error?(e)\n raise e\n end\n end\n end\n\n # queue\n\n def reindex_queue\n ReindexQueue.new(name)\n end\n\n # reindex\n\n # note: this is designed to be used internally\n # so it does not check object matches index class\n def reindex(object, method_name: nil, ignore_missing: nil, full: false, **options)\n if @options[:job_options]\n options[:job_options] = (@options[:job_options] || {}).merge(options[:job_options] || {})\n end\n\n if object.is_a?(Array)\n # note: purposefully skip full\n return reindex_records(object, method_name: method_name, ignore_missing: ignore_missing, **options)\n end\n\n if !object.respond_to?(:searchkick_klass)\n raise Error, \"Cannot reindex object\"\n end\n\n scoped = Searchkick.relation?(object)\n # call searchkick_klass for inheritance\n relation = scoped ? object.all : Searchkick.scope(object.searchkick_klass).all\n\n refresh = options.fetch(:refresh, !scoped)\n options.delete(:refresh)\n\n if method_name || (scoped && !full)\n mode = options.delete(:mode) || :inline\n scope = options.delete(:scope)\n job_options = options.delete(:job_options)\n raise ArgumentError, \"unsupported keywords: #{options.keys.map(&:inspect).join(\", \")}\" if options.any?\n\n # import only\n import_scope(relation, method_name: method_name, mode: mode, scope: scope, ignore_missing: ignore_missing, job_options: job_options)\n self.refresh if refresh\n true\n else\n async = options.delete(:async)\n if async\n if async.is_a?(Hash) && async[:wait]\n Searchkick.warn \"async option is deprecated - use mode: :async, wait: true instead\"\n options[:wait] = true unless options.key?(:wait)\n else\n Searchkick.warn \"async option is deprecated - use mode: :async instead\"\n end\n options[:mode] ||= :async\n end\n\n full_reindex(relation, **options)\n end\n end\n\n def create_index(index_options: nil)\n index_options ||= self.index_options\n index = Index.new(\"#{name}_#{Time.now.strftime('%Y%m%d%H%M%S%L')}\", @options)\n index.create(index_options)\n index\n end\n\n def import_scope(relation, **options)\n relation_indexer.reindex(relation, **options)\n end\n\n def batches_left\n relation_indexer.batches_left\n end\n\n # private\n def klass_document_type(klass, ignore_type = false)\n @klass_document_type[[klass, ignore_type]] ||= begin\n if !ignore_type && klass.searchkick_klass.searchkick_options[:_type]\n type = klass.searchkick_klass.searchkick_options[:_type]\n type = type.call if type.respond_to?(:call)\n type\n else\n klass.model_name.to_s.underscore\n end\n end\n end\n\n # private\n def conversions_fields\n @conversions_fields ||= begin\n conversions = Array(options[:conversions])\n conversions.map(&:to_s) + conversions.map(&:to_sym)\n end\n end\n\n # private\n def conversions_v2_fields\n @conversions_v2_fields ||= Array(options[:conversions_v2]).map(&:to_s)\n end\n\n # private\n def suggest_fields\n @suggest_fields ||= Array(options[:suggest]).map(&:to_s)\n end\n\n # private\n def locations_fields\n @locations_fields ||= begin\n locations = Array(options[:locations])\n locations.map(&:to_s) + locations.map(&:to_sym)\n end\n end\n\n # private\n def uuid\n index_settings[\"uuid\"]\n end\n\n protected\n\n def client\n Searchkick.client\n end\n\n def queue_index(records)\n Searchkick.indexer.queue(records.map { |r| RecordData.new(self, r).index_data })\n end\n\n def queue_delete(records)\n Searchkick.indexer.queue(records.reject { |r| r.id.blank? }.map { |r| RecordData.new(self, r).delete_data })\n end\n\n def queue_update(records, method_name, ignore_missing:)\n items = records.map { |r| RecordData.new(self, r).update_data(method_name) }\n items.each { |i| i.instance_variable_set(:@ignore_missing, true) } if ignore_missing\n Searchkick.indexer.queue(items)\n end\n\n def relation_indexer\n @relation_indexer ||= RelationIndexer.new(self)\n end\n\n def index_settings\n settings.values.first[\"settings\"][\"index\"]\n end\n\n def import_before_promotion(index, relation, **import_options)\n index.import_scope(relation, **import_options)\n end\n\n def reindex_records(object, mode: nil, refresh: false, **options)\n mode ||= Searchkick.callbacks_value || @options[:callbacks] || :inline\n mode = :inline if mode == :bulk\n\n result = RecordIndexer.new(self).reindex(object, mode: mode, full: false, **options)\n self.refresh if refresh\n result\n end\n\n # https://gist.github.com/jarosan/3124884\n # https://www.elastic.co/blog/changing-mapping-with-zero-downtime/\n def full_reindex(relation, import: true, resume: false, retain: false, mode: nil, refresh_interval: nil, scope: nil, wait: nil, job_options: nil)\n raise ArgumentError, \"wait only available in :async mode\" if !wait.nil? && mode != :async\n raise ArgumentError, \"Full reindex does not support :queue mode - use :async mode instead\" if mode == :queue\n\n if resume\n index_name = all_indices.sort.last\n raise Error, \"No index to resume\" unless index_name\n index = Index.new(index_name, @options)\n else\n clean_indices unless retain\n\n index_options = relation.searchkick_index_options\n index_options.deep_merge!(settings: {index: {refresh_interval: refresh_interval}}) if refresh_interval\n index = create_index(index_options: index_options)\n end\n\n import_options = {\n mode: (mode || :inline),\n full: true,\n resume: resume,\n scope: scope,\n job_options: job_options\n }\n\n uuid = index.uuid\n\n # check if alias exists\n alias_exists = alias_exists?\n if alias_exists\n import_before_promotion(index, relation, **import_options) if import\n\n # get existing indices to remove\n unless mode == :async\n check_uuid(uuid, index.uuid)\n promote(index.name, update_refresh_interval: !refresh_interval.nil?)\n clean_indices unless retain\n end\n else\n delete if exists?\n promote(index.name, update_refresh_interval: !refresh_interval.nil?)\n\n # import after promotion\n index.import_scope(relation, **import_options) if import\n end\n\n if mode == :async\n if wait\n puts \"Created index: #{index.name}\"\n puts \"Jobs queued. Waiting...\"\n loop do\n sleep 3\n status = Searchkick.reindex_status(index.name)\n break if status[:completed]\n puts \"Batches left: #{status[:batches_left]}\"\n end\n # already promoted if alias didn't exist\n if alias_exists\n puts \"Jobs complete. Promoting...\"\n check_uuid(uuid, index.uuid)\n promote(index.name, update_refresh_interval: !refresh_interval.nil?)\n end\n clean_indices unless retain\n puts \"SUCCESS!\"\n end\n\n {index_name: index.name}\n else\n index.refresh\n true\n end\n rescue => e\n if Searchkick.transport_error?(e) && (e.message.include?(\"No handler for type [text]\") || e.message.include?(\"class java.util.ArrayList cannot be cast to class java.util.Map\"))\n raise UnsupportedVersionError\n end\n\n raise e\n end\n\n # safety check\n # still a chance for race condition since its called before promotion\n # ideal is for user to disable automatic index creation\n # https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation\n def check_uuid(old_uuid, new_uuid)\n if old_uuid != new_uuid\n raise Error, \"Safety check failed - only run one Model.reindex per model at a time\"\n end\n end\n\n def notify(record, name)\n if Searchkick.callbacks_value == :bulk\n yield\n else\n name = \"#{record.class.searchkick_klass.name} #{name}\" if record && record.class.searchkick_klass\n event = {\n name: name,\n id: search_id(record)\n }\n ActiveSupport::Notifications.instrument(\"request.searchkick\", event) do\n yield\n end\n end\n end\n\n def notify_bulk(records, name)\n if Searchkick.callbacks_value == :bulk\n yield\n else\n event = {\n name: \"#{records.first.class.searchkick_klass.name} #{name}\",\n count: records.size\n }\n ActiveSupport::Notifications.instrument(\"request.searchkick\", event) do\n yield\n end\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/index_cache.rb",
"content": "module Searchkick\n class IndexCache\n def initialize(max_size: 20)\n @data = {}\n @mutex = Mutex.new\n @max_size = max_size\n end\n\n # probably a better pattern for this\n # but keep it simple\n def fetch(name)\n # thread-safe in MRI without mutex\n # due to how context switching works\n @mutex.synchronize do\n if @data.key?(name)\n @data[name]\n else\n @data.clear if @data.size >= @max_size\n @data[name] = yield\n end\n end\n end\n\n def clear\n @mutex.synchronize do\n @data.clear\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/index_options.rb",
"content": "module Searchkick\n class IndexOptions\n attr_reader :options\n\n def initialize(index)\n @options = index.options\n end\n\n def index_options\n # mortal symbols are garbage collected in Ruby 2.2+\n custom_settings = (options[:settings] || {}).deep_symbolize_keys\n custom_mappings = (options[:mappings] || {}).deep_symbolize_keys\n\n if options[:mappings] && !options[:merge_mappings]\n settings = custom_settings\n mappings = custom_mappings\n else\n settings = generate_settings.deep_symbolize_keys.deep_merge(custom_settings)\n mappings = generate_mappings.deep_symbolize_keys.deep_merge(custom_mappings)\n end\n\n set_deep_paging(settings) if options[:deep_paging] || options[:max_result_window]\n\n {\n settings: settings,\n mappings: mappings\n }\n end\n\n def generate_settings\n language = options[:language]\n language = language.call if language.respond_to?(:call)\n\n settings = {\n analysis: {\n analyzer: {\n searchkick_keyword: {\n type: \"custom\",\n tokenizer: \"keyword\",\n filter: [\"lowercase\"] + (options[:stem_conversions] ? [\"searchkick_stemmer\"] : [])\n },\n default_analyzer => {\n type: \"custom\",\n # character filters -> tokenizer -> token filters\n # https://www.elastic.co/guide/en/elasticsearch/guide/current/analysis-intro.html\n char_filter: [\"ampersand\"],\n tokenizer: \"standard\",\n # synonym should come last, after stemming and shingle\n # shingle must come before searchkick_stemmer\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_index_shingle\", \"searchkick_stemmer\"]\n },\n searchkick_search: {\n type: \"custom\",\n char_filter: [\"ampersand\"],\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_search_shingle\", \"searchkick_stemmer\"]\n },\n searchkick_search2: {\n type: \"custom\",\n char_filter: [\"ampersand\"],\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_stemmer\"]\n },\n # https://github.com/leschenko/elasticsearch_autocomplete/blob/master/lib/elasticsearch_autocomplete/analyzers.rb\n searchkick_autocomplete_search: {\n type: \"custom\",\n tokenizer: \"keyword\",\n filter: [\"lowercase\", \"asciifolding\"]\n },\n searchkick_word_search: {\n type: \"custom\",\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\"]\n },\n searchkick_suggest_index: {\n type: \"custom\",\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_suggest_shingle\"]\n },\n searchkick_text_start_index: {\n type: \"custom\",\n tokenizer: \"keyword\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_edge_ngram\"]\n },\n searchkick_text_middle_index: {\n type: \"custom\",\n tokenizer: \"keyword\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_ngram\"]\n },\n searchkick_text_end_index: {\n type: \"custom\",\n tokenizer: \"keyword\",\n filter: [\"lowercase\", \"asciifolding\", \"reverse\", \"searchkick_edge_ngram\", \"reverse\"]\n },\n searchkick_word_start_index: {\n type: \"custom\",\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_edge_ngram\"]\n },\n searchkick_word_middle_index: {\n type: \"custom\",\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"searchkick_ngram\"]\n },\n searchkick_word_end_index: {\n type: \"custom\",\n tokenizer: \"standard\",\n filter: [\"lowercase\", \"asciifolding\", \"reverse\", \"searchkick_edge_ngram\", \"reverse\"]\n }\n },\n filter: {\n searchkick_index_shingle: {\n type: \"shingle\",\n token_separator: \"\"\n },\n # lucky find https://web.archiveorange.com/archive/v/AAfXfQ17f57FcRINsof7\n searchkick_search_shingle: {\n type: \"shingle\",\n token_separator: \"\",\n output_unigrams: false,\n output_unigrams_if_no_shingles: true\n },\n searchkick_suggest_shingle: {\n type: \"shingle\",\n max_shingle_size: 5\n },\n searchkick_edge_ngram: {\n type: \"edge_ngram\",\n min_gram: 1,\n max_gram: 50\n },\n searchkick_ngram: {\n type: \"ngram\",\n min_gram: 1,\n max_gram: 50\n },\n searchkick_stemmer: {\n # use stemmer if language is lowercase, snowball otherwise\n type: language == language.to_s.downcase ? \"stemmer\" : \"snowball\",\n language: language || \"English\"\n }\n },\n char_filter: {\n # https://www.elastic.co/guide/en/elasticsearch/guide/current/custom-analyzers.html\n # &_to_and\n ampersand: {\n type: \"mapping\",\n mappings: [\"&=> and \"]\n }\n }\n }\n }\n\n raise ArgumentError, \"Can't pass both language and stemmer\" if options[:stemmer] && language\n update_language(settings, language)\n update_stemming(settings)\n\n if Searchkick.env == \"test\"\n settings[:number_of_shards] = 1\n settings[:number_of_replicas] = 0\n end\n\n if options[:similarity]\n settings[:similarity] = {default: {type: options[:similarity]}}\n end\n\n settings[:index] = {\n max_ngram_diff: 49,\n max_shingle_diff: 4\n }\n\n if options[:knn]\n unless Searchkick.knn_support?\n if Searchkick.opensearch?\n raise Error, \"knn requires OpenSearch 2.4+\"\n else\n raise Error, \"knn requires Elasticsearch 8.6+\"\n end\n end\n\n if Searchkick.opensearch? && options[:knn].any? { |_, v| !v[:distance].nil? }\n # only enable if doing approximate search\n settings[:index][:knn] = true\n end\n end\n\n add_synonyms(settings)\n add_search_synonyms(settings)\n\n if options[:special_characters] == false\n settings[:analysis][:analyzer].each_value do |analyzer_settings|\n analyzer_settings[:filter].reject! { |f| f == \"asciifolding\" }\n end\n end\n\n if options[:case_sensitive]\n settings[:analysis][:analyzer].each do |_, analyzer|\n analyzer[:filter].delete(\"lowercase\")\n end\n end\n\n settings\n end\n\n def update_language(settings, language)\n case language\n when \"chinese\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: \"ik_smart\"\n },\n searchkick_search: {\n type: \"ik_smart\"\n },\n searchkick_search2: {\n type: \"ik_max_word\"\n }\n )\n when \"chinese2\", \"smartcn\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: \"smartcn\"\n },\n searchkick_search: {\n type: \"smartcn\"\n },\n searchkick_search2: {\n type: \"smartcn\"\n }\n )\n when \"japanese\", \"japanese2\"\n analyzer = {\n type: \"custom\",\n tokenizer: \"kuromoji_tokenizer\",\n filter: [\n \"kuromoji_baseform\",\n \"kuromoji_part_of_speech\",\n \"cjk_width\",\n \"ja_stop\",\n \"searchkick_stemmer\",\n \"lowercase\"\n ]\n }\n settings[:analysis][:analyzer].merge!(\n default_analyzer => analyzer.deep_dup,\n searchkick_search: analyzer.deep_dup,\n searchkick_search2: analyzer.deep_dup\n )\n settings[:analysis][:filter][:searchkick_stemmer] = {\n type: \"kuromoji_stemmer\"\n }\n when \"korean\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: \"openkoreantext-analyzer\"\n },\n searchkick_search: {\n type: \"openkoreantext-analyzer\"\n },\n searchkick_search2: {\n type: \"openkoreantext-analyzer\"\n }\n )\n when \"korean2\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: \"nori\"\n },\n searchkick_search: {\n type: \"nori\"\n },\n searchkick_search2: {\n type: \"nori\"\n }\n )\n when \"vietnamese\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: \"vi_analyzer\"\n },\n searchkick_search: {\n type: \"vi_analyzer\"\n },\n searchkick_search2: {\n type: \"vi_analyzer\"\n }\n )\n when \"polish\", \"ukrainian\"\n settings[:analysis][:analyzer].merge!(\n default_analyzer => {\n type: language\n },\n searchkick_search: {\n type: language\n },\n searchkick_search2: {\n type: language\n }\n )\n end\n end\n\n def update_stemming(settings)\n if options[:stemmer]\n stemmer = options[:stemmer]\n # could also support snowball and stemmer\n case stemmer[:type]\n when \"hunspell\"\n # supports all token filter options\n settings[:analysis][:filter][:searchkick_stemmer] = stemmer\n else\n raise ArgumentError, \"Unknown stemmer: #{stemmer[:type]}\"\n end\n end\n\n stem = options[:stem]\n\n # language analyzer used\n stem = false if settings[:analysis][:analyzer][default_analyzer][:type] != \"custom\"\n\n if stem == false\n settings[:analysis][:filter].delete(:searchkick_stemmer)\n settings[:analysis][:analyzer].each do |_, analyzer|\n analyzer[:filter].delete(\"searchkick_stemmer\") if analyzer[:filter]\n end\n end\n\n if options[:stemmer_override]\n stemmer_override = {\n type: \"stemmer_override\"\n }\n if options[:stemmer_override].is_a?(String)\n stemmer_override[:rules_path] = options[:stemmer_override]\n else\n stemmer_override[:rules] = options[:stemmer_override]\n end\n settings[:analysis][:filter][:searchkick_stemmer_override] = stemmer_override\n\n settings[:analysis][:analyzer].each do |_, analyzer|\n stemmer_index = analyzer[:filter].index(\"searchkick_stemmer\") if analyzer[:filter]\n analyzer[:filter].insert(stemmer_index, \"searchkick_stemmer_override\") if stemmer_index\n end\n end\n\n if options[:stem_exclusion]\n settings[:analysis][:filter][:searchkick_stem_exclusion] = {\n type: \"keyword_marker\",\n keywords: options[:stem_exclusion]\n }\n\n settings[:analysis][:analyzer].each do |_, analyzer|\n stemmer_index = analyzer[:filter].index(\"searchkick_stemmer\") if analyzer[:filter]\n analyzer[:filter].insert(stemmer_index, \"searchkick_stem_exclusion\") if stemmer_index\n end\n end\n end\n\n def generate_mappings\n mapping = {}\n\n keyword_mapping = {type: \"keyword\"}\n keyword_mapping[:ignore_above] = options[:ignore_above] || 30000\n\n # conversions\n Array(options[:conversions]).each do |conversions_field|\n mapping[conversions_field] = {\n type: \"nested\",\n properties: {\n query: {type: default_type, analyzer: \"searchkick_keyword\"},\n count: {type: \"integer\"}\n }\n }\n end\n\n Array(options[:conversions_v2]).each do |conversions_field|\n mapping[conversions_field] = {\n type: \"rank_features\"\n }\n end\n\n if (Array(options[:conversions_v2]).map(&:to_s) & Array(options[:conversions]).map(&:to_s)).any?\n raise ArgumentError, \"Must have separate conversions fields\"\n end\n\n mapping_options =\n [:suggest, :word, :text_start, :text_middle, :text_end, :word_start, :word_middle, :word_end, :highlight, :searchable, :filterable]\n .to_h { |type| [type, (options[type] || []).map(&:to_s)] }\n\n word = options[:word] != false && (!options[:match] || options[:match] == :word)\n\n mapping_options[:searchable].delete(\"_all\")\n\n analyzed_field_options = {type: default_type, index: true, analyzer: default_analyzer.to_s}\n\n mapping_options.values.flatten.uniq.each do |field|\n fields = {}\n\n if options.key?(:filterable) && !mapping_options[:filterable].include?(field)\n fields[field] = {type: default_type, index: false}\n else\n fields[field] = keyword_mapping\n end\n\n if !options[:searchable] || mapping_options[:searchable].include?(field)\n if word\n fields[:analyzed] = analyzed_field_options\n\n if mapping_options[:highlight].include?(field)\n fields[:analyzed][:term_vector] = \"with_positions_offsets\"\n end\n end\n\n mapping_options.except(:highlight, :searchable, :filterable, :word).each do |type, f|\n if options[:match] == type || f.include?(field)\n fields[type] = {type: default_type, index: true, analyzer: \"searchkick_#{type}_index\"}\n end\n end\n end\n\n mapping[field] = fields[field].merge(fields: fields.except(field))\n end\n\n (options[:locations] || []).map(&:to_s).each do |field|\n mapping[field] = {\n type: \"geo_point\"\n }\n end\n\n options[:geo_shape] = options[:geo_shape].product([{}]).to_h if options[:geo_shape].is_a?(Array)\n (options[:geo_shape] || {}).each do |field, shape_options|\n mapping[field] = shape_options.merge(type: \"geo_shape\")\n end\n\n (options[:knn] || []).each do |field, knn_options|\n distance = knn_options[:distance]\n quantization = knn_options[:quantization]\n\n if Searchkick.opensearch?\n if distance.nil?\n # avoid server crash if method not specified\n raise ArgumentError, \"Must specify a distance for OpenSearch\"\n end\n\n vector_options = {\n type: \"knn_vector\",\n dimension: knn_options[:dimensions]\n }\n\n if !distance.nil?\n space_type =\n case distance\n when \"cosine\"\n \"cosinesimil\"\n when \"euclidean\"\n \"l2\"\n when \"inner_product\"\n \"innerproduct\"\n else\n raise ArgumentError, \"Unknown distance: #{distance}\"\n end\n\n if !quantization.nil?\n raise ArgumentError, \"Quantization not supported yet for OpenSearch\"\n end\n\n vector_options[:method] = {\n name: \"hnsw\",\n space_type: space_type,\n engine: \"lucene\",\n parameters: knn_options.slice(:m, :ef_construction)\n }\n end\n\n mapping[field.to_s] = vector_options\n else\n vector_options = {\n type: \"dense_vector\",\n dims: knn_options[:dimensions],\n index: !distance.nil?\n }\n\n if !distance.nil?\n vector_options[:similarity] =\n case distance\n when \"cosine\"\n \"cosine\"\n when \"euclidean\"\n \"l2_norm\"\n when \"inner_product\"\n \"max_inner_product\"\n else\n raise ArgumentError, \"Unknown distance: #{distance}\"\n end\n\n type =\n case quantization\n when \"int8\", \"int4\", \"bbq\"\n \"#{quantization}_hnsw\"\n when nil\n \"hnsw\"\n else\n raise ArgumentError, \"Unknown quantization: #{quantization}\"\n end\n\n vector_index_options = knn_options.slice(:m, :ef_construction)\n vector_options[:index_options] = {type: type}.merge(vector_index_options)\n end\n\n mapping[field.to_s] = vector_options\n end\n end\n\n if options[:inheritance]\n mapping[:type] = keyword_mapping\n end\n\n routing = {}\n if options[:routing]\n routing = {required: true}\n unless options[:routing] == true\n routing[:path] = options[:routing].to_s\n end\n end\n\n dynamic_fields = {\n # analyzed field must be the default field for include_in_all\n # https://www.elastic.co/guide/reference/mapping/multi-field-type/\n # however, we can include the not_analyzed field in _all\n # and the _all index analyzer will take care of it\n \"{name}\" => keyword_mapping\n }\n\n if options.key?(:filterable)\n dynamic_fields[\"{name}\"] = {type: default_type, index: false}\n end\n\n unless options[:searchable]\n if options[:match] && options[:match] != :word\n dynamic_fields[options[:match]] = {type: default_type, index: true, analyzer: \"searchkick_#{options[:match]}_index\"}\n end\n\n if word\n dynamic_fields[:analyzed] = analyzed_field_options\n end\n end\n\n # https://www.elastic.co/guide/reference/mapping/multi-field-type/\n multi_field = dynamic_fields[\"{name}\"].merge(fields: dynamic_fields.except(\"{name}\"))\n\n mappings = {\n properties: mapping,\n _routing: routing,\n # https://gist.github.com/kimchy/2898285\n dynamic_templates: [\n {\n string_template: {\n match: \"*\",\n match_mapping_type: \"string\",\n mapping: multi_field\n }\n }\n ]\n }\n\n mappings\n end\n\n def add_synonyms(settings)\n synonyms = options[:synonyms] || []\n synonyms = synonyms.call if synonyms.respond_to?(:call)\n if synonyms.any?\n settings[:analysis][:filter][:searchkick_synonym] = {\n type: \"synonym\",\n # only remove a single space from synonyms so three-word synonyms will fail noisily instead of silently\n synonyms: synonyms.select { |s| s.size > 1 }.map { |s| s.is_a?(Array) ? s.map { |s2| s2.sub(/\\s+/, \"\") }.join(\",\") : s }.map(&:downcase)\n }\n # choosing a place for the synonym filter when stemming is not easy\n # https://groups.google.com/forum/#!topic/elasticsearch/p7qcQlgHdB8\n # TODO use a snowball stemmer on synonyms when creating the token filter\n\n # https://discuss.elastic.co/t/synonym-multi-words-search/10964\n # I find the following approach effective if you are doing multi-word synonyms (synonym phrases):\n # - Only apply the synonym expansion at index time\n # - Don't have the synonym filter applied search\n # - Use directional synonyms where appropriate. You want to make sure that you're not injecting terms that are too general.\n settings[:analysis][:analyzer][default_analyzer][:filter].insert(2, \"searchkick_synonym\")\n\n %w(word_start word_middle word_end).each do |type|\n settings[:analysis][:analyzer][\"searchkick_#{type}_index\".to_sym][:filter].insert(2, \"searchkick_synonym\")\n end\n end\n end\n\n def add_search_synonyms(settings)\n search_synonyms = options[:search_synonyms] || []\n search_synonyms = search_synonyms.call if search_synonyms.respond_to?(:call)\n if search_synonyms.is_a?(String) || search_synonyms.any?\n if search_synonyms.is_a?(String)\n synonym_graph = {\n type: \"synonym_graph\",\n synonyms_path: search_synonyms,\n updateable: true\n }\n else\n synonym_graph = {\n type: \"synonym_graph\",\n # TODO confirm this is correct\n synonyms: search_synonyms.select { |s| s.size > 1 }.map { |s| s.is_a?(Array) ? s.join(\",\") : s }.map(&:downcase)\n }\n end\n settings[:analysis][:filter][:searchkick_synonym_graph] = synonym_graph\n\n if [\"japanese\", \"japanese2\"].include?(options[:language])\n [:searchkick_search, :searchkick_search2].each do |analyzer|\n settings[:analysis][:analyzer][analyzer][:filter].insert(4, \"searchkick_synonym_graph\")\n end\n else\n [:searchkick_search2, :searchkick_word_search].each do |analyzer|\n unless settings[:analysis][:analyzer][analyzer].key?(:filter)\n raise Error, \"Search synonyms are not supported yet for language\"\n end\n\n settings[:analysis][:analyzer][analyzer][:filter].insert(2, \"searchkick_synonym_graph\")\n end\n end\n end\n end\n\n def set_deep_paging(settings)\n if !settings.dig(:index, :max_result_window) && !settings[:\"index.max_result_window\"]\n settings[:index] ||= {}\n settings[:index][:max_result_window] = options[:max_result_window] || 1_000_000_000\n end\n end\n\n def index_type\n @index_type ||= begin\n index_type = options[:_type]\n index_type = index_type.call if index_type.respond_to?(:call)\n index_type\n end\n end\n\n def default_type\n \"text\"\n end\n\n def default_analyzer\n :searchkick_index\n end\n end\nend\n"
},
{
"path": "lib/searchkick/indexer.rb",
"content": "# thread-local (technically fiber-local) indexer\n# used to aggregate bulk callbacks across models\nmodule Searchkick\n class Indexer\n attr_reader :queued_items\n\n def initialize\n @queued_items = []\n end\n\n def queue(items)\n @queued_items.concat(items)\n perform unless Searchkick.callbacks_value == :bulk\n end\n\n def perform\n items = @queued_items\n @queued_items = []\n\n return if items.empty?\n\n response = Searchkick.client.bulk(body: items)\n if response[\"errors\"]\n # note: delete does not set error when item not found\n first_with_error = response[\"items\"].map do |item|\n (item[\"index\"] || item[\"delete\"] || item[\"update\"])\n end.find.with_index { |item, i| item[\"error\"] && !ignore_missing?(items[i], item[\"error\"]) }\n if first_with_error\n raise ImportError, \"#{first_with_error[\"error\"]} on item with id '#{first_with_error[\"_id\"]}'\"\n end\n end\n\n # maybe return response in future\n nil\n end\n\n private\n\n def ignore_missing?(item, error)\n error[\"type\"] == \"document_missing_exception\" && item.instance_variable_defined?(:@ignore_missing)\n end\n end\nend\n"
},
{
"path": "lib/searchkick/log_subscriber.rb",
"content": "# based on https://gist.github.com/mnutt/566725\nmodule Searchkick\n class LogSubscriber < ActiveSupport::LogSubscriber\n def self.runtime=(value)\n Thread.current[:searchkick_runtime] = value\n end\n\n def self.runtime\n Thread.current[:searchkick_runtime] ||= 0\n end\n\n def self.reset_runtime\n rt = runtime\n self.runtime = 0\n rt\n end\n\n def search(event)\n self.class.runtime += event.duration\n return unless logger.debug?\n\n payload = event.payload\n name = \"#{payload[:name]} (#{event.duration.round(1)}ms)\"\n\n index = payload[:query][:index].is_a?(Array) ? payload[:query][:index].join(\",\") : payload[:query][:index]\n type = payload[:query][:type]\n request_params = payload[:query].except(:index, :type, :body, :opaque_id)\n\n params = []\n request_params.each do |k, v|\n params << \"#{CGI.escape(k.to_s)}=#{CGI.escape(v.to_s)}\"\n end\n\n debug \" #{color(name, YELLOW, bold: true)} #{index}#{type ? \"/#{type.join(',')}\" : ''}/_search#{params.any? ? '?' + params.join('&') : nil} #{payload[:query][:body].to_json}\"\n end\n\n def request(event)\n self.class.runtime += event.duration\n return unless logger.debug?\n\n payload = event.payload\n name = \"#{payload[:name]} (#{event.duration.round(1)}ms)\"\n\n debug \" #{color(name, YELLOW, bold: true)} #{payload.except(:name).to_json}\"\n end\n\n def multi_search(event)\n self.class.runtime += event.duration\n return unless logger.debug?\n\n payload = event.payload\n name = \"#{payload[:name]} (#{event.duration.round(1)}ms)\"\n\n debug \" #{color(name, YELLOW, bold: true)} _msearch #{payload[:body]}\"\n end\n end\nend\n"
},
{
"path": "lib/searchkick/middleware.rb",
"content": "require \"faraday\"\n\nmodule Searchkick\n class Middleware < Faraday::Middleware\n def call(env)\n path = env[:url].path.to_s\n if path.end_with?(\"/_search\")\n env[:request][:timeout] = Searchkick.search_timeout\n elsif path.end_with?(\"/_msearch\")\n # assume no concurrent searches for timeout for now\n searches = env[:request_body].count(\"\\n\") / 2\n # do not allow timeout to exceed Searchkick.timeout\n timeout = [Searchkick.search_timeout * searches, Searchkick.timeout].min\n env[:request][:timeout] = timeout\n end\n @app.call(env)\n end\n end\nend\n"
},
{
"path": "lib/searchkick/model.rb",
"content": "module Searchkick\n module Model\n def searchkick(**options)\n options = Searchkick.model_options.deep_merge(options)\n\n if options[:conversions]\n Searchkick.warn(\"The `conversions` option is deprecated in favor of `conversions_v2`, which provides much better search performance. Upgrade to `conversions_v2` or rename `conversions` to `conversions_v1`\")\n end\n\n if options.key?(:conversions_v1)\n options[:conversions] = options.delete(:conversions_v1)\n end\n\n unknown_keywords = options.keys - [:_all, :_type, :batch_size, :callbacks, :callback_options, :case_sensitive, :conversions, :conversions_v2, :deep_paging, :default_fields,\n :filterable, :geo_shape, :highlight, :ignore_above, :index_name, :index_prefix, :inheritance, :job_options, :knn, :language,\n :locations, :mappings, :match, :max_result_window, :merge_mappings, :routing, :searchable, :search_synonyms, :settings, :similarity,\n :special_characters, :stem, :stemmer, :stem_conversions, :stem_exclusion, :stemmer_override, :suggest, :synonyms, :text_end,\n :text_middle, :text_start, :unscope, :word, :word_end, :word_middle, :word_start]\n raise ArgumentError, \"unknown keywords: #{unknown_keywords.join(\", \")}\" if unknown_keywords.any?\n\n raise \"Only call searchkick once per model\" if respond_to?(:searchkick_index)\n\n Searchkick.models << self\n\n options[:_type] ||= -> { searchkick_index.klass_document_type(self, true) }\n options[:class_name] = model_name.name\n\n callbacks = options.key?(:callbacks) ? options[:callbacks] : :inline\n unless [:inline, true, false, :async, :queue].include?(callbacks)\n raise ArgumentError, \"Invalid value for callbacks\"\n end\n callback_options = (options[:callback_options] || {}).dup\n callback_options[:if] = [-> { Searchkick.callbacks?(default: callbacks) }, callback_options[:if]].compact.flatten(1)\n\n base = self\n\n mod = Module.new\n include(mod)\n mod.module_eval do\n def reindex(method_name = nil, mode: nil, refresh: false, ignore_missing: nil, job_options: nil)\n self.class.searchkick_index.reindex([self], method_name: method_name, mode: mode, refresh: refresh, ignore_missing: ignore_missing, job_options: job_options, single: true)\n end unless base.method_defined?(:reindex)\n\n def similar(**options)\n self.class.searchkick_index.similar_record(self, **options)\n end unless base.method_defined?(:similar)\n\n def search_data\n data = respond_to?(:to_hash) ? to_hash : serializable_hash\n data.delete(\"id\")\n data.delete(\"_id\")\n data.delete(\"_type\")\n data\n end unless base.method_defined?(:search_data)\n\n def should_index?\n true\n end unless base.method_defined?(:should_index?)\n end\n\n class_eval do\n cattr_reader :searchkick_options, :searchkick_klass, instance_reader: false\n\n class_variable_set :@@searchkick_options, options.dup\n class_variable_set :@@searchkick_klass, self\n class_variable_set :@@searchkick_index_cache, Searchkick::IndexCache.new\n\n class << self\n def searchkick_search(term = \"*\", **options, &block)\n if Searchkick.relation?(self)\n raise Searchkick::Error, \"search must be called on model, not relation\"\n end\n\n Searchkick.search(term, model: self, **options, &block)\n end\n alias_method Searchkick.search_method_name, :searchkick_search if Searchkick.search_method_name\n\n def searchkick_index(name: nil)\n index_name = name || searchkick_klass.searchkick_index_name\n index_name = index_name.call if index_name.respond_to?(:call)\n index_cache = class_variable_get(:@@searchkick_index_cache)\n index_cache.fetch(index_name) { Searchkick::Index.new(index_name, searchkick_options) }\n end\n alias_method :search_index, :searchkick_index unless method_defined?(:search_index)\n\n def searchkick_reindex(method_name = nil, **options)\n searchkick_index.reindex(self, method_name: method_name, **options)\n end\n alias_method :reindex, :searchkick_reindex unless method_defined?(:reindex)\n\n def searchkick_index_options\n searchkick_index.index_options\n end\n\n def searchkick_index_name\n @searchkick_index_name ||= begin\n options = class_variable_get(:@@searchkick_options)\n if options[:index_name]\n options[:index_name]\n elsif options[:index_prefix].respond_to?(:call)\n -> { [options[:index_prefix].call, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join(\"_\") }\n else\n [options.key?(:index_prefix) ? options[:index_prefix] : Searchkick.index_prefix, model_name.plural, Searchkick.env, Searchkick.index_suffix].compact.join(\"_\")\n end\n end\n end\n end\n\n # always add callbacks, even when callbacks is false\n # so Model.callbacks block can be used\n if respond_to?(:after_commit)\n after_commit :reindex, **callback_options\n elsif respond_to?(:after_save)\n after_save :reindex, **callback_options\n after_destroy :reindex, **callback_options\n end\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/multi_search.rb",
"content": "module Searchkick\n class MultiSearch\n attr_reader :queries\n\n def initialize(queries, opaque_id: nil)\n @queries = queries\n @opaque_id = opaque_id\n end\n\n def perform\n if queries.any?\n perform_search(queries)\n end\n end\n\n private\n\n def perform_search(search_queries, perform_retry: true)\n params = {\n body: search_queries.flat_map { |q| [q.params.except(:body), q.body] }\n }\n params[:opaque_id] = @opaque_id if @opaque_id\n responses = client.msearch(params)[\"responses\"]\n\n retry_queries = []\n search_queries.each_with_index do |query, i|\n if perform_retry && query.retry_misspellings?(responses[i])\n query.send(:prepare) # okay, since we don't want to expose this method outside Searchkick\n retry_queries << query\n else\n query.handle_response(responses[i])\n end\n end\n\n if retry_queries.any?\n perform_search(retry_queries, perform_retry: false)\n end\n\n search_queries\n end\n\n def client\n Searchkick.client\n end\n end\nend\n"
},
{
"path": "lib/searchkick/process_batch_job.rb",
"content": "module Searchkick\n class ProcessBatchJob < Searchkick.parent_job.constantize\n queue_as { Searchkick.queue_name }\n\n def perform(class_name:, record_ids:, index_name: nil)\n model = Searchkick.load_model(class_name)\n index = model.searchkick_index(name: index_name)\n\n items =\n record_ids.map do |r|\n parts = r.split(/(? e\n handle_error(e)\n end\n handle_response(response)\n end\n end\n\n def handle_response(response)\n opts = {\n page: @page,\n per_page: @per_page,\n padding: @padding,\n load: @load,\n includes: options[:includes],\n model_includes: options[:model_includes],\n json: !@json.nil?,\n match_suffix: @match_suffix,\n highlight: options[:highlight],\n highlighted_fields: @highlighted_fields || [],\n misspellings: @misspellings,\n term: term,\n scope_results: options[:scope_results],\n total_entries: options[:total_entries],\n index_mapping: @index_mapping,\n suggest: options[:suggest],\n scroll: options[:scroll],\n opaque_id: options[:opaque_id]\n }\n\n if options[:debug]\n server = Searchkick.opensearch? ? \"OpenSearch\" : \"Elasticsearch\"\n puts \"Searchkick #{Searchkick::VERSION}\"\n puts \"#{server} #{Searchkick.server_version}\"\n puts\n\n puts \"Model Options\"\n pp searchkick_options\n puts\n\n puts \"Search Options\"\n pp options\n puts\n\n if searchkick_index\n puts \"Record Data\"\n begin\n pp klass.limit(3).map { |r| RecordData.new(searchkick_index, r).index_data }\n rescue => e\n puts \"#{e.class.name}: #{e.message}\"\n end\n puts\n\n puts \"Mapping\"\n puts JSON.pretty_generate(searchkick_index.mapping)\n puts\n\n puts \"Settings\"\n puts JSON.pretty_generate(searchkick_index.settings)\n puts\n end\n\n puts \"Query\"\n puts JSON.pretty_generate(params[:body])\n puts\n\n puts \"Results\"\n puts JSON.pretty_generate(response.to_h)\n end\n\n # set execute for multi search\n @execute = Results.new(searchkick_klass, response, opts)\n end\n\n def retry_misspellings?(response)\n @misspellings_below && response[\"error\"].nil? && Results.new(searchkick_klass, response).total_count < @misspellings_below\n end\n\n private\n\n def handle_error(e)\n status_code = e.message[1..3].to_i\n if status_code == 404\n if e.message.include?(\"No search context found for id\")\n raise MissingIndexError, \"No search context found for id\"\n else\n raise MissingIndexError, \"Index missing - run #{reindex_command}\"\n end\n elsif status_code == 500 && (\n e.message.include?(\"IllegalArgumentException[minimumSimilarity >= 1]\") ||\n e.message.include?(\"No query registered for [multi_match]\") ||\n e.message.include?(\"[match] query does not support [cutoff_frequency]\") ||\n e.message.include?(\"No query registered for [function_score]\")\n )\n\n raise UnsupportedVersionError\n elsif status_code == 400\n if (\n e.message.include?(\"bool query does not support [filter]\") ||\n e.message.include?(\"[bool] filter does not support [filter]\")\n )\n\n raise UnsupportedVersionError\n elsif e.message.match?(/analyzer \\[searchkick_.+\\] not found/)\n raise InvalidQueryError, \"Bad mapping - run #{reindex_command}\"\n else\n raise InvalidQueryError, e.message\n end\n else\n raise e\n end\n end\n\n def reindex_command\n searchkick_klass ? \"#{searchkick_klass.name}.reindex\" : \"reindex\"\n end\n\n def execute_search\n name = searchkick_klass ? \"#{searchkick_klass.name} Search\" : \"Search\"\n event = {\n name: name,\n query: params\n }\n ActiveSupport::Notifications.instrument(\"search.searchkick\", event) do\n Searchkick.client.search(params)\n end\n end\n\n def prepare\n boost_fields, fields = set_fields\n\n operator = options[:operator] || \"and\"\n\n # pagination\n page = [options[:page].to_i, 1].max\n # maybe use index.max_result_window in the future\n default_limit = searchkick_options[:deep_paging] ? 1_000_000_000 : 10_000\n per_page = (options[:limit] || options[:per_page] || default_limit).to_i\n padding = [options[:padding].to_i, 0].max\n offset = (options[:offset] || (page - 1) * per_page + padding).to_i\n scroll = options[:scroll]\n opaque_id = options[:opaque_id]\n\n max_result_window = searchkick_options[:max_result_window]\n original_per_page = per_page\n if max_result_window\n offset = max_result_window if offset > max_result_window\n per_page = max_result_window - offset if offset + per_page > max_result_window\n end\n\n # model and eager loading\n load = options[:load].nil? ? true : options[:load]\n\n all = term == \"*\"\n\n @json = options[:body]\n if @json\n ignored_options = options.keys & [:aggs, :boost,\n :boost_by, :boost_by_distance, :boost_by_recency, :boost_where, :conversions, :conversions_term, :exclude, :explain,\n :fields, :highlight, :indices_boost, :match, :misspellings, :operator, :order,\n :profile, :select, :smart_aggs, :suggest, :where]\n raise ArgumentError, \"Options incompatible with body option: #{ignored_options.join(\", \")}\" if ignored_options.any?\n payload = @json\n else\n must_not = []\n should = []\n\n if options[:similar]\n like = options[:similar] == true ? term : options[:similar]\n query = {\n more_like_this: {\n like: like,\n min_doc_freq: 1,\n min_term_freq: 1,\n analyzer: \"searchkick_search2\"\n }\n }\n if fields.all? { |f| f.start_with?(\"*.\") }\n raise ArgumentError, \"Must specify fields to search\"\n end\n if fields != [\"_all\"]\n query[:more_like_this][:fields] = fields\n end\n elsif all && !options[:exclude]\n query = {\n match_all: {}\n }\n else\n queries = []\n\n misspellings =\n if options.key?(:misspellings)\n options[:misspellings]\n else\n true\n end\n\n if misspellings.is_a?(Hash) && misspellings[:below] && !@misspellings_below\n @misspellings_below = misspellings[:below].to_i\n misspellings = false\n end\n\n if misspellings != false\n edit_distance = (misspellings.is_a?(Hash) && (misspellings[:edit_distance] || misspellings[:distance])) || 1\n transpositions =\n if misspellings.is_a?(Hash) && misspellings.key?(:transpositions)\n {fuzzy_transpositions: misspellings[:transpositions]}\n else\n {fuzzy_transpositions: true}\n end\n prefix_length = (misspellings.is_a?(Hash) && misspellings[:prefix_length]) || 0\n default_max_expansions = @misspellings_below ? 20 : 3\n max_expansions = (misspellings.is_a?(Hash) && misspellings[:max_expansions]) || default_max_expansions\n misspellings_fields = misspellings.is_a?(Hash) && misspellings.key?(:fields) && misspellings[:fields].map(&:to_s)\n\n if misspellings_fields\n missing_fields = misspellings_fields - fields.map { |f| base_field(f) }\n if missing_fields.any?\n raise ArgumentError, \"All fields in per-field misspellings must also be specified in fields option\"\n end\n end\n\n @misspellings = true\n else\n @misspellings = false\n end\n\n fields.each do |field|\n queries_to_add = []\n qs = []\n\n factor = boost_fields[field] || 1\n shared_options = {\n query: term,\n boost: 10 * factor\n }\n\n match_type =\n if field.end_with?(\".phrase\")\n field =\n if field == \"_all.phrase\"\n \"_all\"\n else\n field.sub(/\\.phrase\\z/, \".analyzed\")\n end\n\n :match_phrase\n else\n :match\n end\n\n shared_options[:operator] = operator if match_type == :match\n\n exclude_analyzer = nil\n exclude_field = field\n\n field_misspellings = misspellings && (!misspellings_fields || misspellings_fields.include?(base_field(field)))\n\n if field == \"_all\" || field.end_with?(\".analyzed\")\n qs << shared_options.merge(analyzer: \"searchkick_search\")\n\n # searchkick_search and searchkick_search2 are the same for some languages\n unless %w(japanese japanese2 korean polish ukrainian vietnamese).include?(searchkick_options[:language])\n qs << shared_options.merge(analyzer: \"searchkick_search2\")\n end\n exclude_analyzer = \"searchkick_search2\"\n elsif field.end_with?(\".exact\")\n f = field.split(\".\")[0..-2].join(\".\")\n queries_to_add << {match: {f => shared_options.merge(analyzer: \"keyword\")}}\n exclude_field = f\n exclude_analyzer = \"keyword\"\n else\n analyzer = field.match?(/\\.word_(start|middle|end)\\z/) ? \"searchkick_word_search\" : \"searchkick_autocomplete_search\"\n qs << shared_options.merge(analyzer: analyzer)\n exclude_analyzer = analyzer\n end\n\n if field_misspellings != false && match_type == :match\n qs.concat(qs.map { |q| q.except(:cutoff_frequency).merge(fuzziness: edit_distance, prefix_length: prefix_length, max_expansions: max_expansions, boost: factor).merge(transpositions) })\n end\n\n if field.start_with?(\"*.\")\n q2 = qs.map { |q| {multi_match: q.merge(fields: [field], type: match_type == :match_phrase ? \"phrase\" : \"best_fields\")} }\n else\n q2 = qs.map { |q| {match_type => {field => q}} }\n end\n\n # boost exact matches more\n if field =~ /\\.word_(start|middle|end)\\z/ && searchkick_options[:word] != false\n queries_to_add << {\n bool: {\n must: {\n bool: {\n should: q2\n }\n },\n should: {match_type => {field.sub(/\\.word_(start|middle|end)\\z/, \".analyzed\") => qs.first}}\n }\n }\n else\n queries_to_add.concat(q2)\n end\n\n queries << queries_to_add\n\n if options[:exclude]\n must_not.concat(set_exclude(exclude_field, exclude_analyzer))\n end\n end\n\n # all + exclude option\n if all\n query = {\n match_all: {}\n }\n\n should = []\n else\n # higher score for matching more fields\n payload = {\n bool: {\n should: queries.map { |qs| {dis_max: {queries: qs}} }\n }\n }\n\n should.concat(set_conversions)\n should.concat(set_conversions_v2)\n end\n\n query = payload\n end\n\n payload = {}\n\n # type when inheritance\n where = ensure_permitted(options[:where] || {}).dup\n if searchkick_options[:inheritance] && (options[:type] || (klass != searchkick_klass && searchkick_index))\n where[:type] = [options[:type] || klass].flatten.map { |v| searchkick_index.klass_document_type(v, true) }\n end\n\n models = Array(options[:models])\n if models.any? { |m| m != m.searchkick_klass }\n index_type_or =\n models.map do |m|\n v = {_index: m.searchkick_index.name}\n v[:type] = m.searchkick_index.klass_document_type(m, true) if m != m.searchkick_klass\n v\n end\n\n where[:or] = Array(where[:or]) + [index_type_or]\n end\n\n # start everything as efficient filters\n # move to post_filters as aggs demand\n filters = where_filters(where)\n post_filters = []\n\n # aggregations\n set_aggregations(payload, filters, post_filters) if options[:aggs]\n\n # post filters\n set_post_filters(payload, post_filters) if post_filters.any?\n\n custom_filters = []\n multiply_filters = []\n\n set_boost_by(multiply_filters, custom_filters)\n set_boost_where(custom_filters)\n set_boost_by_distance(custom_filters) if options[:boost_by_distance]\n set_boost_by_recency(custom_filters) if options[:boost_by_recency]\n\n payload[:query] = build_query(query, filters, should, must_not, custom_filters, multiply_filters)\n\n payload[:explain] = options[:explain] if options[:explain]\n payload[:profile] = options[:profile] if options[:profile]\n\n # order\n set_order(payload) if options[:order]\n\n # indices_boost\n set_boost_by_indices(payload)\n\n # suggestions\n set_suggestions(payload, options[:suggest]) if options[:suggest]\n\n # highlight\n set_highlights(payload, fields) if options[:highlight]\n\n # timeout shortly after client times out\n payload[:timeout] ||= \"#{((Searchkick.search_timeout + 1) * 1000).round}ms\"\n\n # An empty array will cause only the _id and _type for each hit to be returned\n # https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html\n if options[:select]\n if options[:select] == []\n # intuitively [] makes sense to return no fields, but ES by default returns all fields\n payload[:_source] = false\n else\n payload[:_source] = options[:select]\n end\n elsif load\n payload[:_source] = false\n end\n end\n\n # knn\n set_knn(payload, options[:knn], per_page, offset) if options[:knn]\n\n # pagination\n pagination_options = options[:page] || options[:limit] || options[:per_page] || options[:offset] || options[:padding]\n if !options[:body] || pagination_options\n payload[:size] = per_page\n payload[:from] = offset if offset > 0\n end\n\n # type\n if !searchkick_options[:inheritance] && (options[:type] || (klass != searchkick_klass && searchkick_index))\n @type = [options[:type] || klass].flatten.map { |v| searchkick_index.klass_document_type(v) }\n end\n\n # routing\n @routing = options[:routing] if options[:routing]\n\n if track_total_hits?\n payload[:track_total_hits] = true\n end\n\n # merge more body options\n payload = payload.deep_merge(options[:body_options]) if options[:body_options]\n\n # run block\n options[:block].call(payload) if options[:block]\n\n # scroll optimization when iterating over all docs\n # https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html\n if options[:scroll] && payload[:query] == {match_all: {}}\n payload[:sort] ||= [\"_doc\"]\n end\n\n @body = payload\n @page = page\n @per_page = original_per_page\n @padding = padding\n @load = load\n @scroll = scroll\n @opaque_id = opaque_id\n end\n\n def set_fields\n boost_fields = {}\n fields = options[:fields] || searchkick_options[:default_fields] || searchkick_options[:searchable]\n all = searchkick_options.key?(:_all) ? searchkick_options[:_all] : false\n default_match = options[:match] || searchkick_options[:match] || :word\n fields =\n if fields\n fields.map do |value|\n k, v = value.is_a?(Hash) ? value.to_a.first : [value, default_match]\n k2, boost = k.to_s.split(\"^\", 2)\n field = \"#{k2}.#{v == :word ? 'analyzed' : v}\"\n boost_fields[field] = boost.to_f if boost\n field\n end\n elsif all && default_match == :word\n [\"_all\"]\n elsif all && default_match == :phrase\n [\"_all.phrase\"]\n elsif term != \"*\" && default_match == :exact\n raise ArgumentError, \"Must specify fields to search\"\n else\n [default_match == :word ? \"*.analyzed\" : \"*.#{default_match}\"]\n end\n [boost_fields, fields]\n end\n\n def build_query(query, filters, should, must_not, custom_filters, multiply_filters)\n if filters.any? || must_not.any? || should.any?\n bool = {}\n bool[:must] = query if query\n bool[:filter] = filters if filters.any? # where\n bool[:must_not] = must_not if must_not.any? # exclude\n bool[:should] = should if should.any? # conversions\n query = {bool: bool}\n end\n\n if custom_filters.any?\n query = {\n function_score: {\n functions: custom_filters,\n query: query,\n score_mode: \"sum\"\n }\n }\n end\n\n if multiply_filters.any?\n query = {\n function_score: {\n functions: multiply_filters,\n query: query,\n score_mode: \"multiply\"\n }\n }\n end\n\n query\n end\n\n def set_conversions\n conversions_fields = Array(options[:conversions] || searchkick_options[:conversions]).map(&:to_s)\n if conversions_fields.present? && options[:conversions] != false\n conversions_fields.map do |conversions_field|\n {\n nested: {\n path: conversions_field,\n score_mode: \"sum\",\n query: {\n function_score: {\n boost_mode: \"replace\",\n query: {\n match: {\n \"#{conversions_field}.query\" => options[:conversions_term] || term\n }\n },\n field_value_factor: {\n field: \"#{conversions_field}.count\"\n }\n }\n }\n }\n }\n end\n else\n []\n end\n end\n\n def set_conversions_v2\n conversions_v2 = options[:conversions_v2]\n return [] if conversions_v2.nil? && !searchkick_options[:conversions_v2]\n return [] if conversions_v2 == false\n\n # disable if searchkick_options[:conversions] to make it easy to upgrade without downtime\n return [] if conversions_v2.nil? && searchkick_options[:conversions]\n\n unless conversions_v2.is_a?(Hash)\n conversions_v2 = {field: conversions_v2}\n end\n\n conversions_fields =\n case conversions_v2[:field]\n when true, nil\n Array(searchkick_options[:conversions_v2]).map(&:to_s)\n else\n [conversions_v2[:field].to_s]\n end\n\n conversions_term = (conversions_v2[:term] || options[:conversions_term] || term).to_s\n unless searchkick_options[:case_sensitive]\n conversions_term = conversions_term.downcase\n end\n conversions_term = conversions_term.gsub(\".\", \"*\")\n\n conversions_fields.map do |conversions_field|\n {\n rank_feature: {\n field: \"#{conversions_field}.#{conversions_term}\",\n linear: {},\n boost: conversions_v2[:factor] || 1\n }\n }\n end\n end\n\n def set_exclude(field, analyzer)\n Array(options[:exclude]).map do |phrase|\n {\n multi_match: {\n fields: [field],\n query: phrase,\n analyzer: analyzer,\n type: \"phrase\"\n }\n }\n end\n end\n\n def set_boost_by_distance(custom_filters)\n boost_by_distance = options[:boost_by_distance] || {}\n\n # legacy format\n if boost_by_distance[:field]\n boost_by_distance = {boost_by_distance[:field] => boost_by_distance.except(:field)}\n end\n\n boost_by_distance.each do |field, attributes|\n attributes = {function: :gauss, scale: \"5mi\"}.merge(attributes)\n unless attributes[:origin]\n raise ArgumentError, \"boost_by_distance requires :origin\"\n end\n\n function_params = attributes.except(:factor, :function)\n function_params[:origin] = location_value(function_params[:origin])\n custom_filters << {\n weight: attributes[:factor] || 1,\n attributes[:function] => {\n field => function_params\n }\n }\n end\n end\n\n def set_boost_by_recency(custom_filters)\n options[:boost_by_recency].each do |field, attributes|\n attributes = {function: :gauss, origin: Time.now}.merge(attributes)\n\n custom_filters << {\n weight: attributes[:factor] || 1,\n attributes[:function] => {\n field => attributes.except(:factor, :function)\n }\n }\n end\n end\n\n def set_boost_by(multiply_filters, custom_filters)\n boost_by = options[:boost_by] || {}\n if boost_by.is_a?(Array)\n boost_by = boost_by.to_h { |f| [f, {factor: 1}] }\n elsif boost_by.is_a?(Hash)\n multiply_by, boost_by = boost_by.transform_values(&:dup).partition { |_, v| v.delete(:boost_mode) == \"multiply\" }.map(&:to_h)\n end\n boost_by[options[:boost]] = {factor: 1} if options[:boost]\n\n custom_filters.concat boost_filters(boost_by, modifier: \"ln2p\")\n multiply_filters.concat boost_filters(multiply_by || {})\n end\n\n def set_boost_where(custom_filters)\n boost_where = options[:boost_where] || {}\n boost_where.each do |field, value|\n if value.is_a?(Array) && value.first.is_a?(Hash)\n value.each do |value_factor|\n custom_filters << custom_filter(field, value_factor[:value], value_factor[:factor])\n end\n elsif value.is_a?(Hash)\n custom_filters << custom_filter(field, value[:value], value[:factor])\n else\n factor = 1000\n custom_filters << custom_filter(field, value, factor)\n end\n end\n end\n\n def set_boost_by_indices(payload)\n return unless options[:indices_boost]\n\n indices_boost = options[:indices_boost].map do |key, boost|\n index = key.respond_to?(:searchkick_index) ? key.searchkick_index.name : key\n {index => boost}\n end\n\n payload[:indices_boost] = indices_boost\n end\n\n def set_suggestions(payload, suggest)\n suggest_fields = nil\n\n if suggest.is_a?(Array)\n suggest_fields = suggest\n else\n suggest_fields = (searchkick_options[:suggest] || []).map(&:to_s)\n\n # intersection\n if options[:fields]\n suggest_fields &= options[:fields].map { |v| (v.is_a?(Hash) ? v.keys.first : v).to_s.split(\"^\", 2).first }\n end\n end\n\n if suggest_fields.any?\n payload[:suggest] = {text: term}\n suggest_fields.each do |field|\n payload[:suggest][field] = {\n phrase: {\n field: \"#{field}.suggest\"\n }\n }\n end\n else\n raise ArgumentError, \"Must pass fields to suggest option\"\n end\n end\n\n def set_highlights(payload, fields)\n payload[:highlight] = {\n fields: fields.to_h { |f| [f, {}] },\n fragment_size: 0\n }\n\n if options[:highlight].is_a?(Hash)\n if (tag = options[:highlight][:tag])\n payload[:highlight][:pre_tags] = [tag]\n payload[:highlight][:post_tags] = [tag.to_s.gsub(/\\A<(\\w+).+/, \"\")]\n end\n\n if (fragment_size = options[:highlight][:fragment_size])\n payload[:highlight][:fragment_size] = fragment_size\n end\n if (encoder = options[:highlight][:encoder])\n payload[:highlight][:encoder] = encoder\n end\n\n highlight_fields = options[:highlight][:fields]\n if highlight_fields\n payload[:highlight][:fields] = {}\n\n highlight_fields.each do |name, opts|\n payload[:highlight][:fields][\"#{name}.#{@match_suffix}\"] = opts || {}\n end\n end\n end\n\n @highlighted_fields = payload[:highlight][:fields].keys\n end\n\n def set_aggregations(payload, filters, post_filters)\n aggs = options[:aggs]\n payload[:aggs] = {}\n\n aggs = aggs.to_h { |f| [f, {}] } if aggs.is_a?(Array) # convert to more advanced syntax\n aggs.each do |field, agg_options|\n size = agg_options[:limit] ? agg_options[:limit] : 1_000\n shared_agg_options = agg_options.except(:limit, :field, :ranges, :date_ranges, :where)\n\n if agg_options[:ranges]\n payload[:aggs][field] = {\n range: {\n field: agg_options[:field] || field,\n ranges: agg_options[:ranges]\n }.merge(shared_agg_options)\n }\n elsif agg_options[:date_ranges]\n payload[:aggs][field] = {\n date_range: {\n field: agg_options[:field] || field,\n ranges: agg_options[:date_ranges]\n }.merge(shared_agg_options)\n }\n elsif (histogram = agg_options[:date_histogram])\n payload[:aggs][field] = {\n date_histogram: histogram\n }.merge(shared_agg_options)\n elsif (metric = @@metric_aggs.find { |k| agg_options.has_key?(k) })\n payload[:aggs][field] = {\n metric => {\n field: agg_options[metric][:field] || field\n }\n }.merge(shared_agg_options)\n else\n payload[:aggs][field] = {\n terms: {\n field: agg_options[:field] || field,\n size: size\n }.merge(shared_agg_options)\n }\n end\n\n agg_where = ensure_permitted(agg_options[:where] || {})\n if options[:smart_aggs] != false && options[:where]\n where = ensure_permitted(options[:where])\n where_without_field = where.reject { |k| k == field }\n # where_without_field = where_without_field(where, field.to_s)\n if where_without_field.any?\n if agg_where.any?\n agg_where = where.merge(agg_where)\n # agg_where = combine_agg_where(agg_where, where_without_field)\n else\n agg_where = where_without_field\n end\n end\n end\n agg_filters = where_filters(agg_where)\n\n # only do one level comparison for simplicity\n filters.select! do |filter|\n if agg_filters.include?(filter)\n true\n else\n post_filters << filter\n false\n end\n end\n\n if agg_filters.any?\n payload[:aggs][field] = {\n filter: {\n bool: {\n must: agg_filters\n }\n },\n aggs: {\n field => payload[:aggs][field]\n }\n }\n end\n end\n end\n\n def where_without_field(where, field)\n result = {}\n where.each do |f, v|\n case f\n when :_and\n r = v.map { |v2| where_without_field(v2, field) }.reject(&:empty?)\n result[f] = r unless r.empty?\n when :_or\n r = v.map { |v2| where_without_field(v2, field) }\n result[f] = r unless r.any?(&:empty?)\n when :or\n r = v.map { |v2| v2.map { |v3| where_without_field(v3, field) }.reject { |v2| v2.any?(&:empty?) } }\n result[f] = r unless r.empty?\n when :_not\n r = where_without_field(v, field)\n result[f] = r unless r.empty?\n when :_script\n result[f] = v\n else\n if f.to_s != field\n result[f] = v\n end\n end\n end\n result\n end\n\n def combine_agg_where(agg_where, where)\n result = agg_where.dup\n field_keys = result.except(:_and, :_or, :or, :_not, :_script).transform_keys(&:to_s)\n where.each do |f, v|\n case f\n when :_and, :_or, :or, :_not, :_script\n if result.key?(f)\n # combine with _and if needed\n result[:_and] ||= []\n result[:_and] += [{f => v}]\n else\n result[f] = v\n end\n else\n result[f] = v unless field_keys.include?(f.to_s)\n end\n end\n result\n end\n\n def set_knn(payload, knn, per_page, offset)\n if term != \"*\"\n raise ArgumentError, \"Use Searchkick.multi_search for hybrid search\"\n end\n\n field = knn[:field]\n field_options = searchkick_options.dig(:knn, field.to_sym) || searchkick_options.dig(:knn, field.to_s) || {}\n vector = knn[:vector]\n distance = knn[:distance] || field_options[:distance]\n exact = knn[:exact]\n exact = field_options[:distance].nil? || distance != field_options[:distance] if exact.nil?\n k = per_page + offset\n ef_search = knn[:ef_search]\n filter = payload.delete(:query)\n\n if distance.nil?\n raise ArgumentError, \"distance required\"\n elsif !exact && distance != field_options[:distance]\n raise ArgumentError, \"distance must match searchkick options for approximate search\"\n end\n\n if Searchkick.opensearch?\n if exact\n # https://opensearch.org/docs/latest/search-plugins/knn/knn-score-script/#spaces\n space_type =\n case distance\n when \"cosine\"\n \"cosinesimil\"\n when \"euclidean\"\n \"l2\"\n when \"taxicab\"\n \"l1\"\n when \"inner_product\"\n \"innerproduct\"\n when \"chebyshev\"\n \"linf\"\n else\n raise ArgumentError, \"Unknown distance: #{distance}\"\n end\n\n payload[:query] = {\n script_score: {\n query: {\n bool: {\n must: [filter, {exists: {field: field}}]\n }\n },\n script: {\n source: \"knn_score\",\n lang: \"knn\",\n params: {\n field: field,\n query_value: vector,\n space_type: space_type\n }\n },\n boost: distance == \"cosine\" && Searchkick.server_below?(\"2.19.0\") ? 0.5 : 1.0\n }\n }\n else\n if ef_search && Searchkick.server_below?(\"2.16.0\")\n raise Error, \"ef_search requires OpenSearch 2.16+\"\n end\n\n payload[:query] = {\n knn: {\n field.to_sym => {\n vector: vector,\n k: k,\n filter: filter\n }.merge(ef_search ? {method_parameters: {ef_search: ef_search}} : {})\n }\n }\n end\n else\n if exact\n # prevent incorrect distances/results with Elasticsearch 9.0.0-rc1\n if !Searchkick.server_below?(\"9.0.0\") && field_options[:distance] == \"cosine\" && distance != \"cosine\"\n raise ArgumentError, \"distance must match searchkick options\"\n end\n\n # https://github.com/elastic/elasticsearch/blob/main/docs/reference/vectors/vector-functions.asciidoc\n source =\n case distance\n when \"cosine\"\n \"(cosineSimilarity(params.query_vector, params.field) + 1.0) * 0.5\"\n when \"euclidean\"\n \"double l2 = l2norm(params.query_vector, params.field); 1 / (1 + l2 * l2)\"\n when \"taxicab\"\n \"1 / (1 + l1norm(params.query_vector, params.field))\"\n when \"inner_product\"\n \"double dot = dotProduct(params.query_vector, params.field); dot > 0 ? dot + 1 : 1 / (1 - dot)\"\n else\n raise ArgumentError, \"Unknown distance: #{distance}\"\n end\n\n payload[:query] = {\n script_score: {\n query: {\n bool: {\n must: [filter, {exists: {field: field}}]\n }\n },\n script: {\n source: source,\n params: {\n field: field,\n query_vector: vector\n }\n }\n }\n }\n else\n payload[:knn] = {\n field: field,\n query_vector: vector,\n k: k,\n filter: filter\n }.merge(ef_search ? {num_candidates: ef_search} : {})\n end\n end\n end\n\n def set_post_filters(payload, post_filters)\n payload[:post_filter] = {\n bool: {\n filter: post_filters\n }\n }\n end\n\n def set_order(payload)\n value = options[:order]\n payload[:sort] = value.is_a?(Enumerable) ? value : {value => :asc}\n end\n\n # provides *very* basic protection from unfiltered parameters\n # this is not meant to be comprehensive and may be expanded in the future\n def ensure_permitted(obj)\n obj.to_h\n end\n\n def where_filters(where)\n filters = []\n (where || {}).each do |field, value|\n field = :_id if field.to_s == \"id\"\n\n # update smart aggs when adding new symbol\n if field == :or\n value.each do |or_clause|\n filters << {bool: {should: or_clause.map { |or_statement| {bool: {filter: where_filters(or_statement)}} }}}\n end\n elsif field == :_or\n filters << {bool: {should: value.map { |or_statement| {bool: {filter: where_filters(or_statement)}} }}}\n elsif field == :_not\n filters << {bool: {must_not: where_filters(value)}}\n elsif field == :_and\n filters << {bool: {must: value.map { |or_statement| {bool: {filter: where_filters(or_statement)}} }}}\n elsif field == :_script\n unless value.is_a?(Script)\n raise TypeError, \"expected Searchkick::Script\"\n end\n\n filters << {script: {script: {source: value.source, lang: value.lang, params: value.params}}}\n else\n # expand ranges\n if value.is_a?(Range)\n value = expand_range(value)\n end\n\n value = {in: value} if value.is_a?(Array)\n\n if value.is_a?(Hash)\n value.each do |op, op_value|\n case op\n when :within, :bottom_right, :bottom_left\n # do nothing\n when :near\n filters << {\n geo_distance: {\n field => location_value(op_value),\n distance: value[:within] || \"50mi\"\n }\n }\n when :geo_polygon\n filters << {\n geo_polygon: {\n field => op_value\n }\n }\n when :geo_shape\n shape = op_value.except(:relation)\n shape[:coordinates] = coordinate_array(shape[:coordinates]) if shape[:coordinates]\n filters << {\n geo_shape: {\n field => {\n relation: op_value[:relation] || \"intersects\",\n shape: shape\n }\n }\n }\n when :top_left\n filters << {\n geo_bounding_box: {\n field => {\n top_left: location_value(op_value),\n bottom_right: location_value(value[:bottom_right])\n }\n }\n }\n when :top_right\n filters << {\n geo_bounding_box: {\n field => {\n top_right: location_value(op_value),\n bottom_left: location_value(value[:bottom_left])\n }\n }\n }\n when :like, :ilike\n # based on Postgres\n # https://www.postgresql.org/docs/current/functions-matching.html\n # % matches zero or more characters\n # _ matches one character\n # \\ is escape character\n # escape Lucene reserved characters\n # https://www.elastic.co/guide/en/elasticsearch/reference/current/regexp-syntax.html#regexp-optional-operators\n reserved = %w(\\\\ . ? + * | { } [ ] ( ) \")\n regex = op_value.dup\n reserved.each do |v|\n regex.gsub!(v, \"\\\\\\\\\" + v)\n end\n regex = regex.gsub(/(? {value: regex, flags: \"NONE\", case_insensitive: true}}}\n else\n filters << {regexp: {field => {value: regex, flags: \"NONE\"}}}\n end\n when :prefix\n filters << {prefix: {field => {value: op_value}}}\n when :regexp # support for regexp queries without using a regexp ruby object\n filters << {regexp: {field => {value: op_value}}}\n when :not, :_not # not equal\n filters << {bool: {must_not: term_filters(field, op_value)}}\n when :all\n op_value.each do |val|\n filters << term_filters(field, val)\n end\n when :in\n filters << term_filters(field, op_value)\n when :exists\n case op_value\n when true\n filters << {exists: {field: field}}\n when false\n filters << {bool: {must_not: {exists: {field: field}}}}\n else\n raise ArgumentError, \"Passing a value other than true or false to exists is not supported\"\n end\n else\n range_query =\n case op\n when :gt\n {gt: op_value}\n when :gte\n {gte: op_value}\n when :lt\n {lt: op_value}\n when :lte\n {lte: op_value}\n else\n raise ArgumentError, \"Unknown where operator: #{op.inspect}\"\n end\n # issue 132\n if (existing = filters.find { |f| f[:range] && f[:range][field] })\n existing[:range][field].merge!(range_query)\n else\n filters << {range: {field => range_query}}\n end\n end\n end\n else\n filters << term_filters(field, value)\n end\n end\n end\n filters\n end\n\n def term_filters(field, value)\n if value.is_a?(Array) # in query\n if value.any?(&:nil?)\n {bool: {should: [term_filters(field, nil), term_filters(field, value.compact)]}}\n else\n {terms: {field => value}}\n end\n elsif value.nil?\n {bool: {must_not: {exists: {field: field}}}}\n elsif value.is_a?(Regexp)\n source = value.source\n\n # TODO handle other regexp options\n\n # TODO handle other anchor characters, like ^, $, \\Z\n if source.start_with?(\"\\\\A\")\n source = source[2..-1]\n else\n source = \".*#{source}\"\n end\n\n if source.end_with?(\"\\\\z\")\n source = source[0..-3]\n else\n source = \"#{source}.*\"\n end\n\n {regexp: {field => {value: source, flags: \"NONE\", case_insensitive: value.casefold?}}}\n else\n # TODO add this for other values\n if value.as_json.is_a?(Enumerable)\n # query will fail, but this is better\n # same message as Active Record\n raise TypeError, \"can't cast #{value.class.name}\"\n end\n\n {term: {field => {value: value}}}\n end\n end\n\n def custom_filter(field, value, factor)\n {\n filter: where_filters(field => value),\n weight: factor\n }\n end\n\n def boost_filter(field, factor: 1, modifier: nil, missing: nil)\n script_score = {\n field_value_factor: {\n field: field,\n factor: factor.to_f,\n modifier: modifier\n }\n }\n\n if missing\n script_score[:field_value_factor][:missing] = missing.to_f\n else\n script_score[:filter] = {\n exists: {\n field: field\n }\n }\n end\n\n script_score\n end\n\n def boost_filters(boost_by, modifier: nil)\n boost_by.map do |field, value|\n boost_filter(field, modifier: modifier, **value)\n end\n end\n\n # Recursively descend through nesting of arrays until we reach either a lat/lon object or an array of numbers,\n # eventually returning the same structure with all values transformed to [lon, lat].\n #\n def coordinate_array(value)\n if value.is_a?(Hash)\n [value[:lon], value[:lat]]\n elsif value.is_a?(Array) and !value[0].is_a?(Numeric)\n value.map { |a| coordinate_array(a) }\n else\n value\n end\n end\n\n def location_value(value)\n if value.is_a?(Array)\n value.map(&:to_f).reverse\n else\n value\n end\n end\n\n def expand_range(range)\n expanded = {}\n expanded[:gte] = range.begin if range.begin\n\n if range.end && !(range.end.respond_to?(:infinite?) && range.end.infinite?)\n expanded[range.exclude_end? ? :lt : :lte] = range.end\n end\n\n expanded\n end\n\n def base_field(k)\n k.sub(/\\.(analyzed|word_start|word_middle|word_end|text_start|text_middle|text_end|exact)\\z/, \"\")\n end\n\n def track_total_hits?\n searchkick_options[:deep_paging] || body_options[:track_total_hits]\n end\n\n def body_options\n options[:body_options] || {}\n end\n end\nend\n"
},
{
"path": "lib/searchkick/railtie.rb",
"content": "module Searchkick\n class Railtie < Rails::Railtie\n rake_tasks do\n load \"tasks/searchkick.rake\"\n end\n end\nend\n"
},
{
"path": "lib/searchkick/record_data.rb",
"content": "module Searchkick\n class RecordData\n TYPE_KEYS = [\"type\", :type]\n\n attr_reader :index, :record\n\n def initialize(index, record)\n @index = index\n @record = record\n end\n\n def index_data\n data = record_data\n data[:data] = search_data\n {index: data}\n end\n\n def update_data(method_name)\n data = record_data\n data[:data] = {doc: search_data(method_name)}\n {update: data}\n end\n\n def delete_data\n {delete: record_data}\n end\n\n # custom id can be useful for load: false\n def search_id\n id = record.respond_to?(:search_document_id) ? record.search_document_id : record.id\n id.is_a?(Numeric) ? id : id.to_s\n end\n\n def document_type(ignore_type = false)\n index.klass_document_type(record.class, ignore_type)\n end\n\n def record_data\n data = {\n _index: index.name,\n _id: search_id\n }\n data[:routing] = record.search_routing if record.respond_to?(:search_routing)\n data\n end\n\n private\n\n def search_data(method_name = nil)\n partial_reindex = !method_name.nil?\n\n source = record.send(method_name || :search_data)\n\n # conversions\n index.conversions_fields.each do |conversions_field|\n if source[conversions_field]\n source[conversions_field] = source[conversions_field].map { |k, v| {query: k, count: v} }\n end\n end\n\n index.conversions_v2_fields.each do |conversions_field|\n key = source.key?(conversions_field) ? conversions_field : conversions_field.to_sym\n if !partial_reindex || source[key]\n if index.options[:case_sensitive]\n source[key] =\n (source[key] || {}).reduce(Hash.new(0)) do |memo, (k, v)|\n memo[k.to_s.gsub(\".\", \"*\")] += v\n memo\n end\n else\n source[key] =\n (source[key] || {}).reduce(Hash.new(0)) do |memo, (k, v)|\n memo[k.to_s.downcase.gsub(\".\", \"*\")] += v\n memo\n end\n end\n end\n end\n\n # hack to prevent generator field doesn't exist error\n if !partial_reindex\n index.suggest_fields.each do |field|\n if !source.key?(field) && !source.key?(field.to_sym)\n source[field] = nil\n end\n end\n end\n\n # locations\n index.locations_fields.each do |field|\n if source[field]\n if !source[field].is_a?(Hash) && (source[field].first.is_a?(Array) || source[field].first.is_a?(Hash))\n # multiple locations\n source[field] = source[field].map { |a| location_value(a) }\n else\n source[field] = location_value(source[field])\n end\n end\n end\n\n if index.options[:inheritance]\n if !TYPE_KEYS.any? { |tk| source.key?(tk) }\n source[:type] = document_type(true)\n end\n end\n\n cast_big_decimal(source)\n\n source\n end\n\n def location_value(value)\n if value.is_a?(Array)\n value.map(&:to_f).reverse\n elsif value.is_a?(Hash)\n {lat: value[:lat].to_f, lon: value[:lon].to_f}\n else\n value\n end\n end\n\n # change all BigDecimal values to floats due to\n # https://github.com/rails/rails/issues/6033\n # possible loss of precision :/\n def cast_big_decimal(obj)\n case obj\n when BigDecimal\n obj.to_f\n when Hash\n obj.each do |k, v|\n # performance\n if v.is_a?(BigDecimal)\n obj[k] = v.to_f\n elsif v.is_a?(Enumerable)\n obj[k] = cast_big_decimal(v)\n end\n end\n when Enumerable\n obj.map do |v|\n cast_big_decimal(v)\n end\n else\n obj\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/record_indexer.rb",
"content": "module Searchkick\n class RecordIndexer\n attr_reader :index\n\n def initialize(index)\n @index = index\n end\n\n def reindex(records, mode:, method_name:, ignore_missing:, full: false, single: false, job_options: nil)\n # prevents exists? check if records is a relation\n records = records.to_a\n return if records.empty?\n\n case mode\n when :async\n unless defined?(ActiveJob)\n raise Error, \"Active Job not found\"\n end\n\n job_options ||= {}\n\n # only add if set for backwards compatibility\n extra_options = {}\n if ignore_missing\n extra_options[:ignore_missing] = ignore_missing\n end\n\n # we could likely combine ReindexV2Job, BulkReindexJob, and ProcessBatchJob\n # but keep them separate for now\n if single\n record = records.first\n\n # always pass routing in case record is deleted\n # before the async job runs\n if record.respond_to?(:search_routing)\n routing = record.search_routing\n end\n\n Searchkick::ReindexV2Job.set(**job_options).perform_later(\n record.class.name,\n record.id.to_s,\n method_name ? method_name.to_s : nil,\n routing: routing,\n index_name: index.name,\n **extra_options\n )\n else\n Searchkick::BulkReindexJob.set(**job_options).perform_later(\n class_name: records.first.class.searchkick_options[:class_name],\n record_ids: records.map { |r| r.id.to_s },\n index_name: index.name,\n method_name: method_name ? method_name.to_s : nil,\n **extra_options\n )\n end\n when :queue\n if method_name\n raise Error, \"Partial reindex not supported with queue option\"\n end\n\n index.reindex_queue.push_records(records)\n when true, :inline\n index_records, other_records = records.partition { |r| index_record?(r) }\n import_inline(index_records, !full ? other_records : [], method_name: method_name, ignore_missing: ignore_missing, single: single)\n else\n raise ArgumentError, \"Invalid value for mode\"\n end\n\n # return true like model and relation reindex for now\n true\n end\n\n def reindex_items(klass, items, method_name:, ignore_missing:, single: false)\n routing = items.to_h { |r| [r[:id], r[:routing]] }\n record_ids = routing.keys\n\n relation = Searchkick.load_records(klass, record_ids)\n # call search_import even for single records for nested associations\n relation = relation.search_import if relation.respond_to?(:search_import)\n records = relation.select(&:should_index?)\n\n # determine which records to delete\n delete_ids = record_ids - records.map { |r| r.id.to_s }\n delete_records =\n delete_ids.map do |id|\n construct_record(klass, id, routing[id])\n end\n\n import_inline(records, delete_records, method_name: method_name, ignore_missing: ignore_missing, single: single)\n end\n\n private\n\n def index_record?(record)\n record.persisted? && !record.destroyed? && record.should_index?\n end\n\n # import in single request with retries\n def import_inline(index_records, delete_records, method_name:, ignore_missing:, single:)\n return if index_records.empty? && delete_records.empty?\n\n maybe_bulk(index_records, delete_records, method_name, single) do\n if index_records.any?\n if method_name\n index.bulk_update(index_records, method_name, ignore_missing: ignore_missing)\n else\n index.bulk_index(index_records)\n end\n end\n\n if delete_records.any?\n index.bulk_delete(delete_records)\n end\n end\n end\n\n def maybe_bulk(index_records, delete_records, method_name, single)\n if Searchkick.callbacks_value == :bulk\n yield\n else\n # set action and data\n action =\n if single && index_records.empty?\n \"Remove\"\n elsif method_name\n \"Update\"\n else\n single ? \"Store\" : \"Import\"\n end\n record = index_records.first || delete_records.first\n name = record.class.searchkick_klass.name\n message = lambda do |event|\n event[:name] = \"#{name} #{action}\"\n if single\n event[:id] = index.search_id(record)\n else\n event[:count] = index_records.size + delete_records.size\n end\n end\n\n with_retries do\n Searchkick.callbacks(:bulk, message: message) do\n yield\n end\n end\n end\n end\n\n def construct_record(klass, id, routing)\n record = klass.new\n record.id = id\n if routing\n record.define_singleton_method(:search_routing) do\n routing\n end\n end\n record\n end\n\n def with_retries\n retries = 0\n\n begin\n yield\n rescue Faraday::ClientError => e\n if retries < 1\n retries += 1\n retry\n end\n raise e\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/reindex_queue.rb",
"content": "module Searchkick\n class ReindexQueue\n attr_reader :name\n\n def initialize(name)\n @name = name\n\n raise Error, \"Searchkick.redis not set\" unless Searchkick.redis\n end\n\n # supports single and multiple ids\n def push(record_ids)\n Searchkick.with_redis { |r| r.call(\"LPUSH\", redis_key, record_ids) }\n end\n\n def push_records(records)\n record_ids =\n records.map do |record|\n # always pass routing in case record is deleted\n # before the queue job runs\n if record.respond_to?(:search_routing)\n routing = record.search_routing\n end\n\n # escape pipe with double pipe\n value = escape(record.id.to_s)\n value = \"#{value}|#{escape(routing)}\" if routing\n value\n end\n\n push(record_ids)\n end\n\n # TODO use reliable queuing\n def reserve(limit: 1000)\n Searchkick.with_redis { |r| r.call(\"RPOP\", redis_key, limit) }.to_a\n end\n\n def clear\n Searchkick.with_redis { |r| r.call(\"DEL\", redis_key) }\n end\n\n def length\n Searchkick.with_redis { |r| r.call(\"LLEN\", redis_key) }\n end\n\n private\n\n def redis_key\n \"searchkick:reindex_queue:#{name}\"\n end\n\n def escape(value)\n value.to_s.gsub(\"|\", \"||\")\n end\n end\nend\n"
},
{
"path": "lib/searchkick/reindex_v2_job.rb",
"content": "module Searchkick\n class ReindexV2Job < Searchkick.parent_job.constantize\n queue_as { Searchkick.queue_name }\n\n def perform(class_name, id, method_name = nil, routing: nil, index_name: nil, ignore_missing: nil)\n model = Searchkick.load_model(class_name, allow_child: true)\n index = model.searchkick_index(name: index_name)\n # use should_index? to decide whether to index (not default scope)\n # just like saving inline\n # could use Searchkick.scope() in future\n # but keep for now for backwards compatibility\n model = model.unscoped if model.respond_to?(:unscoped)\n items = [{id: id, routing: routing}]\n RecordIndexer.new(index).reindex_items(model, items, method_name: method_name, ignore_missing: ignore_missing, single: true)\n end\n end\nend\n"
},
{
"path": "lib/searchkick/relation.rb",
"content": "module Searchkick\n class Relation\n NO_DEFAULT_VALUE = Object.new\n\n # note: modifying body directly is not supported\n # and has no impact on query after being executed\n # TODO freeze body object?\n delegate :params, to: :query\n delegate_missing_to :private_execute\n\n attr_reader :model\n alias_method :klass, :model\n\n def initialize(model, term = \"*\", **options)\n @model = model\n @term = term\n @options = options\n\n # generate query to validate options\n query if options.any?\n end\n\n # same as Active Record\n def inspect\n entries = private_execute.first(11).map!(&:inspect)\n entries[10] = \"...\" if entries.size == 11\n \"#<#{self.class.name} [#{entries.join(', ')}]>\"\n end\n\n def aggs(*args, **kwargs)\n if args.empty? && kwargs.empty?\n private_execute.aggs\n else\n clone.aggs!(*args, **kwargs)\n end\n end\n\n def aggs!(*args, **kwargs)\n check_loaded\n aggs = {}\n args.flatten.each do |arg|\n if arg.is_a?(Hash)\n aggs.merge!(arg)\n else\n aggs[arg] = {}\n end\n end\n aggs.merge!(kwargs)\n merge_option(:aggs, aggs)\n self\n end\n\n def body(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n query.body\n else\n clone.body!(value)\n end\n end\n\n def body!(value)\n check_loaded\n @options[:body] = value\n self\n end\n\n def body_options(value)\n clone.body_options!(value)\n end\n\n def body_options!(value)\n check_loaded\n merge_option(:body_options, value)\n self\n end\n\n def boost(value)\n clone.boost!(value)\n end\n\n def boost!(value)\n check_loaded\n @options[:boost] = value\n self\n end\n\n def boost_by(value)\n clone.boost_by!(value)\n end\n\n def boost_by!(value)\n check_loaded\n if value.is_a?(Array)\n value = value.to_h { |f| [f, {factor: 1}] }\n elsif !value.is_a?(Hash)\n value = {value => {factor: 1}}\n end\n merge_option(:boost_by, value)\n self\n end\n\n def boost_by_distance(value)\n clone.boost_by_distance!(value)\n end\n\n def boost_by_distance!(value)\n check_loaded\n # legacy format\n value = {value[:field] => value.except(:field)} if value[:field]\n merge_option(:boost_by_distance, value)\n self\n end\n\n def boost_by_recency(value)\n clone.boost_by_recency!(value)\n end\n\n def boost_by_recency!(value)\n check_loaded\n merge_option(:boost_by_recency, value)\n self\n end\n\n def boost_where(value)\n clone.boost_where!(value)\n end\n\n def boost_where!(value)\n check_loaded\n # TODO merge duplicate fields\n merge_option(:boost_where, value)\n self\n end\n\n def conversions(value)\n clone.conversions!(value)\n end\n\n def conversions!(value)\n check_loaded\n @options[:conversions] = value\n self\n end\n\n def conversions_v1(value)\n clone.conversions_v1!(value)\n end\n\n def conversions_v1!(value)\n check_loaded\n @options[:conversions_v1] = value\n self\n end\n\n def conversions_v2(value)\n clone.conversions_v2!(value)\n end\n\n def conversions_v2!(value)\n check_loaded\n @options[:conversions_v2] = value\n self\n end\n\n def conversions_term(value)\n clone.conversions_term!(value)\n end\n\n def conversions_term!(value)\n check_loaded\n @options[:conversions_term] = value\n self\n end\n\n def debug(value = true)\n clone.debug!(value)\n end\n\n def debug!(value = true)\n check_loaded\n @options[:debug] = value\n self\n end\n\n def emoji(value = true)\n clone.emoji!(value)\n end\n\n def emoji!(value = true)\n check_loaded\n @options[:emoji] = value\n self\n end\n\n def exclude(*values)\n clone.exclude!(*values)\n end\n\n def exclude!(*values)\n check_loaded\n concat_option(:exclude, values.flatten)\n self\n end\n\n def explain(value = true)\n clone.explain!(value)\n end\n\n def explain!(value = true)\n check_loaded\n @options[:explain] = value\n self\n end\n\n def fields(*values)\n clone.fields!(*values)\n end\n\n def fields!(*values)\n check_loaded\n concat_option(:fields, values.flatten)\n self\n end\n\n def highlight(value)\n clone.highlight!(value)\n end\n\n def highlight!(value)\n check_loaded\n @options[:highlight] = value\n self\n end\n\n def includes(*values)\n clone.includes!(*values)\n end\n\n def includes!(*values)\n check_loaded\n concat_option(:includes, values.flatten)\n self\n end\n\n def index_name(*values)\n clone.index_name!(*values)\n end\n\n def index_name!(*values)\n check_loaded\n values = values.flatten\n if values.all? { |v| v.respond_to?(:searchkick_index) }\n models!(*values)\n else\n concat_option(:index_name, values)\n self\n end\n end\n\n def indices_boost(value)\n clone.indices_boost!(value)\n end\n\n def indices_boost!(value)\n check_loaded\n merge_option(:indices_boost, value)\n self\n end\n\n def knn(value)\n clone.knn!(value)\n end\n\n def knn!(value)\n check_loaded\n @options[:knn] = value\n self\n end\n\n def limit(value)\n clone.limit!(value)\n end\n\n def limit!(value)\n check_loaded\n @options[:limit] = value\n self\n end\n\n def load(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n private_execute\n self\n else\n clone.load!(value)\n end\n end\n\n def load!(value)\n check_loaded\n @options[:load] = value\n self\n end\n\n def match(value)\n clone.match!(value)\n end\n\n def match!(value)\n check_loaded\n @options[:match] = value\n self\n end\n\n def misspellings(value)\n clone.misspellings!(value)\n end\n\n def misspellings!(value)\n check_loaded\n @options[:misspellings] = value\n self\n end\n\n def models(*values)\n clone.models!(*values)\n end\n\n def models!(*values)\n check_loaded\n concat_option(:models, values.flatten)\n self\n end\n\n def model_includes(*values)\n clone.model_includes!(*values)\n end\n\n def model_includes!(*values)\n check_loaded\n concat_option(:model_includes, values.flatten)\n self\n end\n\n def offset(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n private_execute.offset\n else\n clone.offset!(value)\n end\n end\n\n def offset!(value)\n check_loaded\n @options[:offset] = value\n self\n end\n\n def opaque_id(value)\n clone.opaque_id!(value)\n end\n\n def opaque_id!(value)\n check_loaded\n @options[:opaque_id] = value\n self\n end\n\n def operator(value)\n clone.operator!(value)\n end\n\n def operator!(value)\n check_loaded\n @options[:operator] = value\n self\n end\n\n def order(*values)\n clone.order!(*values)\n end\n\n def order!(*values)\n check_loaded\n concat_option(:order, values.flatten)\n self\n end\n\n def padding(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n private_execute.padding\n else\n clone.padding!(value)\n end\n end\n\n def padding!(value)\n check_loaded\n @options[:padding] = value\n self\n end\n\n def page(value)\n clone.page!(value)\n end\n\n def page!(value)\n check_loaded\n @options[:page] = value\n self\n end\n\n def per_page(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n private_execute.per_page\n else\n clone.per_page!(value)\n end\n end\n\n def per(value)\n per_page(value)\n end\n\n def per_page!(value)\n check_loaded\n # TODO set limit?\n @options[:per_page] = value\n self\n end\n\n def profile(value = true)\n clone.profile!(value)\n end\n\n def profile!(value = true)\n check_loaded\n @options[:profile] = value\n self\n end\n\n def request_params(value)\n clone.request_params!(value)\n end\n\n def request_params!(value)\n check_loaded\n merge_option(:request_params, value)\n self\n end\n\n def routing(value)\n clone.routing!(value)\n end\n\n def routing!(value)\n check_loaded\n @options[:routing] = value\n self\n end\n\n def scope_results(value)\n clone.scope_results!(value)\n end\n\n def scope_results!(value)\n check_loaded\n @options[:scope_results] = value\n self\n end\n\n def scroll(value = NO_DEFAULT_VALUE, &block)\n if value == NO_DEFAULT_VALUE\n private_execute.scroll(&block)\n elsif block_given?\n clone.scroll!(value).scroll(&block)\n else\n clone.scroll!(value)\n end\n end\n\n def scroll!(value)\n check_loaded\n @options[:scroll] = value\n self\n end\n\n def select(*values, &block)\n if block_given?\n private_execute.select(*values, &block)\n else\n clone.select!(*values)\n end\n end\n\n def select!(*values)\n check_loaded\n concat_option(:select, values.flatten)\n self\n end\n\n def similar(value = true)\n clone.similar!(value)\n end\n\n def similar!(value = true)\n check_loaded\n @options[:similar] = value\n self\n end\n\n def smart_aggs(value)\n clone.smart_aggs!(value)\n end\n\n def smart_aggs!(value)\n check_loaded\n @options[:smart_aggs] = value\n self\n end\n\n def suggest(value = true)\n clone.suggest!(value)\n end\n\n def suggest!(value = true)\n check_loaded\n @options[:suggest] = value\n self\n end\n\n def total_entries(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n private_execute.total_entries\n else\n clone.total_entries!(value)\n end\n end\n\n def total_entries!(value)\n check_loaded\n @options[:total_entries] = value\n self\n end\n\n def track(value = true)\n clone.track!(value)\n end\n\n def track!(value = true)\n check_loaded\n @options[:track] = value\n self\n end\n\n def type(*values)\n clone.type!(*values)\n end\n\n def type!(*values)\n check_loaded\n concat_option(:type, values.flatten)\n self\n end\n\n def where(value = NO_DEFAULT_VALUE)\n if value == NO_DEFAULT_VALUE\n Where.new(self)\n else\n clone.where!(value)\n end\n end\n\n def where!(value)\n check_loaded\n value = ensure_permitted(value)\n if @options[:where]\n # keep simple when possible for smart aggs\n if !@options[:where].keys.intersect?(value.keys)\n merge_option(:where, value)\n elsif @options[:where][:_and].is_a?(Array)\n merge_option(:where, {_and: @options[:where][:_and] + [value]})\n else\n @options[:where] = {_and: [@options[:where], value]}\n end\n else\n @options[:where] = value\n end\n self\n end\n\n def first(value = NO_DEFAULT_VALUE)\n result =\n if loaded?\n private_execute\n else\n limit = value == NO_DEFAULT_VALUE ? 1 : value\n previous_limit = (@options[:limit] || @options[:per_page])&.to_i\n if previous_limit && previous_limit < limit\n limit = previous_limit\n end\n limit(limit).load\n end\n\n if value == NO_DEFAULT_VALUE\n result.first\n else\n result.first(value)\n end\n end\n\n def pluck(*keys)\n if !loaded? && @options[:load] == false\n select(*keys).send(:private_execute).pluck(*keys)\n else\n private_execute.pluck(*keys)\n end\n end\n\n def reorder(*values)\n clone.reorder!(*values)\n end\n\n def reorder!(*values)\n check_loaded\n @options[:order] = values\n self\n end\n\n def reselect(*values)\n clone.reselect!(*values)\n end\n\n def reselect!(*values)\n check_loaded\n @options[:select] = values\n self\n end\n\n def rewhere(value)\n clone.rewhere!(value)\n end\n\n def rewhere!(value)\n check_loaded\n @options[:where] = ensure_permitted(value)\n self\n end\n\n def only(*keys)\n Relation.new(@model, @term, **@options.slice(*keys))\n end\n\n def except(*keys)\n Relation.new(@model, @term, **@options.except(*keys))\n end\n\n def loaded?\n !@execute.nil?\n end\n\n undef_method :respond_to_missing?\n\n def respond_to_missing?(...)\n Results.new(nil, nil, nil).respond_to?(...) || super\n end\n\n # TODO uncomment in 7.0\n # def to_json(...)\n # private_execute.to_a.to_json(...)\n # end\n\n # TODO uncomment in 7.0\n # def as_json(...)\n # private_execute.to_a.as_json(...)\n # end\n\n def to_yaml\n private_execute.to_a.to_yaml\n end\n\n private\n\n def private_execute\n @execute ||= query.execute\n end\n\n def query\n @query ||= Query.new(@model, @term, **@options)\n end\n\n def check_loaded\n raise Error, \"Relation loaded\" if loaded?\n\n # reset query since options will change\n @query = nil\n end\n\n # provides *very* basic protection from unfiltered parameters\n # this is not meant to be comprehensive and may be expanded in the future\n def ensure_permitted(obj)\n obj.to_h\n end\n\n def initialize_copy(other)\n super\n # shallow dup and avoid updating values in-place\n @options = @options.dup\n @execute = nil\n end\n\n def concat_option(key, value)\n if @options[key]\n @options[key] += value\n else\n @options[key] = value.to_ary\n end\n end\n\n def merge_option(key, value)\n if @options[key]\n @options[key] = @options[key].merge(value)\n else\n @options[key] = value.to_hash\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/relation_indexer.rb",
"content": "module Searchkick\n class RelationIndexer\n attr_reader :index\n\n def initialize(index)\n @index = index\n end\n\n def reindex(relation, mode:, method_name: nil, ignore_missing: nil, full: false, resume: false, scope: nil, job_options: nil)\n # apply scopes\n if scope\n relation = relation.send(scope)\n elsif relation.respond_to?(:search_import)\n relation = relation.search_import\n end\n\n # remove unneeded loading for async and queue\n if mode == :async || mode == :queue\n if relation.respond_to?(:primary_key)\n relation = relation.except(:includes, :preload)\n unless mode == :queue && relation.klass.method_defined?(:search_routing)\n relation = relation.except(:select).select(relation.primary_key)\n end\n elsif relation.respond_to?(:only)\n unless mode == :queue && relation.klass.method_defined?(:search_routing)\n relation = relation.only(:_id)\n end\n end\n end\n\n if mode == :async && full\n return full_reindex_async(relation, job_options: job_options)\n end\n\n relation = resume_relation(relation) if resume\n\n reindex_options = {\n mode: mode,\n method_name: method_name,\n full: full,\n ignore_missing: ignore_missing,\n job_options: job_options\n }\n record_indexer = RecordIndexer.new(index)\n\n in_batches(relation) do |items|\n record_indexer.reindex(items, **reindex_options)\n end\n end\n\n def batches_left\n Searchkick.with_redis { |r| r.call(\"SCARD\", batches_key) }\n end\n\n def batch_completed(batch_id)\n Searchkick.with_redis { |r| r.call(\"SREM\", batches_key, [batch_id]) }\n end\n\n private\n\n def resume_relation(relation)\n if relation.respond_to?(:primary_key)\n # use total docs instead of max id since there's not a great way\n # to get the max _id without scripting since it's a string\n where = relation.arel_table[relation.primary_key].gt(index.total_docs)\n relation = relation.where(where)\n else\n raise Error, \"Resume not supported for Mongoid\"\n end\n end\n\n def in_batches(relation)\n if relation.respond_to?(:find_in_batches)\n klass = relation.klass\n # remove order to prevent possible warnings\n relation.except(:order).find_in_batches(batch_size: batch_size) do |batch|\n # prevent scope from affecting search_data as well as inline jobs\n # Active Record runs relation calls in scoping block\n # https://github.com/rails/rails/blob/main/activerecord/lib/active_record/relation/delegation.rb\n # note: we could probably just call klass.current_scope = nil\n # anywhere in reindex method (after initial all call),\n # but this is more cautious\n previous_scope = klass.current_scope(true)\n if previous_scope\n begin\n klass.current_scope = nil\n yield batch\n ensure\n klass.current_scope = previous_scope\n end\n else\n yield batch\n end\n end\n else\n klass = relation.klass\n each_batch(relation, batch_size: batch_size) do |batch|\n # prevent scope from affecting search_data as well as inline jobs\n # note: Model.with_scope doesn't always restore scope, so use custom logic\n previous_scope = Mongoid::Threaded.current_scope(klass)\n if previous_scope\n begin\n Mongoid::Threaded.set_current_scope(nil, klass)\n yield batch\n ensure\n Mongoid::Threaded.set_current_scope(previous_scope, klass)\n end\n else\n yield batch\n end\n end\n end\n end\n\n def each_batch(relation, batch_size:)\n # https://github.com/karmi/tire/blob/master/lib/tire/model/import.rb\n # use cursor for Mongoid\n items = []\n relation.all.each do |item|\n items << item\n if items.length == batch_size\n yield items\n items = []\n end\n end\n yield items if items.any?\n end\n\n def batch_size\n @batch_size ||= index.options[:batch_size] || 1000\n end\n\n def full_reindex_async(relation, job_options: nil)\n batch_id = 1\n class_name = relation.searchkick_options[:class_name]\n starting_id = false\n\n if relation.respond_to?(:primary_key)\n primary_key = relation.primary_key\n\n starting_id =\n begin\n relation.minimum(primary_key)\n rescue ActiveRecord::StatementInvalid\n false\n end\n end\n\n if starting_id.nil?\n # no records, do nothing\n elsif starting_id.is_a?(Numeric)\n max_id = relation.maximum(primary_key)\n batches_count = ((max_id - starting_id + 1) / batch_size.to_f).ceil\n\n batches_count.times do |i|\n min_id = starting_id + (i * batch_size)\n batch_job(class_name, batch_id, job_options, min_id: min_id, max_id: min_id + batch_size - 1)\n batch_id += 1\n end\n else\n in_batches(relation) do |items|\n batch_job(class_name, batch_id, job_options, record_ids: items.map(&:id).map { |v| v.instance_of?(Integer) ? v : v.to_s })\n batch_id += 1\n end\n end\n end\n\n def batch_job(class_name, batch_id, job_options, **options)\n job_options ||= {}\n # TODO expire Redis key\n Searchkick.with_redis { |r| r.call(\"SADD\", batches_key, [batch_id]) }\n Searchkick::BulkReindexJob.set(**job_options).perform_later(\n class_name: class_name,\n index_name: index.name,\n batch_id: batch_id,\n **options\n )\n end\n\n def batches_key\n \"searchkick:reindex:#{index.name}:batches\"\n end\n end\nend\n"
},
{
"path": "lib/searchkick/reranking.rb",
"content": "module Searchkick\n module Reranking\n def self.rrf(first_ranking, *rankings, k: 60)\n rankings.unshift(first_ranking)\n rankings.map!(&:to_ary)\n\n ranks = []\n results = []\n rankings.each do |ranking|\n ranks << ranking.map.with_index.to_h { |v, i| [v, i + 1] }\n results.concat(ranking)\n end\n\n results =\n results.uniq.map do |result|\n score =\n ranks.sum do |rank|\n r = rank[result]\n r ? 1.0 / (k + r) : 0.0\n end\n\n {result: result, score: score}\n end\n\n results.sort_by { |v| -v[:score] }\n end\n end\nend\n"
},
{
"path": "lib/searchkick/results.rb",
"content": "module Searchkick\n class Results\n include Enumerable\n extend Forwardable\n\n attr_reader :response\n\n def_delegators :results, :each, :any?, :empty?, :size, :length, :slice, :[], :to_ary\n\n def initialize(klass, response, options = {})\n @klass = klass\n @response = response\n @options = options\n end\n\n def with_hit\n return enum_for(:with_hit) unless block_given?\n\n build_hits.each do |result|\n yield result\n end\n end\n\n def missing_records\n @missing_records ||= with_hit_and_missing_records[1]\n end\n\n def suggestions\n if response[\"suggest\"]\n response[\"suggest\"].values.flat_map { |v| v.first[\"options\"] }.sort_by { |o| -o[\"score\"] }.map { |o| o[\"text\"] }.uniq\n elsif options[:suggest]\n []\n else\n raise \"Pass `suggest: true` to the search method for suggestions\"\n end\n end\n\n def aggregations\n response[\"aggregations\"]\n end\n\n def aggs\n @aggs ||= begin\n if aggregations\n aggregations.dup.each do |field, filtered_agg|\n buckets = filtered_agg[field]\n # move the buckets one level above into the field hash\n if buckets\n filtered_agg.delete(field)\n filtered_agg.merge!(buckets)\n end\n end\n end\n end\n end\n\n def took\n response[\"took\"]\n end\n\n def error\n response[\"error\"]\n end\n\n def model_name\n if klass.nil?\n ActiveModel::Name.new(self.class, nil, 'Result')\n else\n klass.model_name\n end\n end\n\n def entry_name(options = {})\n if options.empty?\n # backward compatibility\n model_name.human.downcase\n else\n default = options[:count] == 1 ? model_name.human : model_name.human.pluralize\n model_name.human(options.reverse_merge(default: default))\n end\n end\n\n def total_count\n if options[:total_entries]\n options[:total_entries]\n elsif response[\"hits\"][\"total\"].is_a?(Hash)\n response[\"hits\"][\"total\"][\"value\"]\n else\n response[\"hits\"][\"total\"]\n end\n end\n alias_method :total_entries, :total_count\n\n def current_page\n options[:page]\n end\n\n def per_page\n options[:per_page]\n end\n alias_method :limit_value, :per_page\n\n def padding\n options[:padding]\n end\n\n def total_pages\n (total_count / per_page.to_f).ceil\n end\n alias_method :num_pages, :total_pages\n\n def offset_value\n (current_page - 1) * per_page + padding\n end\n alias_method :offset, :offset_value\n\n def previous_page\n current_page > 1 ? (current_page - 1) : nil\n end\n alias_method :prev_page, :previous_page\n\n def next_page\n current_page < total_pages ? (current_page + 1) : nil\n end\n\n def first_page?\n previous_page.nil?\n end\n\n def last_page?\n next_page.nil?\n end\n\n def out_of_range?\n current_page > total_pages\n end\n\n def hits\n if error\n raise Error, \"Query error - use the error method to view it\"\n else\n @response[\"hits\"][\"hits\"]\n end\n end\n\n def highlights(multiple: false)\n hits.map do |hit|\n hit_highlights(hit, multiple: multiple)\n end\n end\n\n def with_highlights(multiple: false)\n return enum_for(:with_highlights, multiple: multiple) unless block_given?\n\n with_hit.each do |result, hit|\n yield result, hit_highlights(hit, multiple: multiple)\n end\n end\n\n def with_score\n return enum_for(:with_score) unless block_given?\n\n with_hit.each do |result, hit|\n yield result, hit[\"_score\"]\n end\n end\n\n def misspellings?\n @options[:misspellings]\n end\n\n def scroll_id\n @response[\"_scroll_id\"]\n end\n\n def scroll\n raise Error, \"Pass `scroll` option to the search method for scrolling\" unless scroll_id\n\n if block_given?\n records = self\n while records.any?\n yield records\n records = records.scroll\n end\n\n records.clear_scroll\n else\n begin\n # TODO Active Support notifications for this scroll call\n params = {\n scroll: options[:scroll],\n body: {scroll_id: scroll_id}\n }\n params[:opaque_id] = options[:opaque_id] if options[:opaque_id]\n Results.new(@klass, Searchkick.client.scroll(params), @options)\n rescue => e\n if Searchkick.not_found_error?(e) && e.message =~ /search_context_missing_exception/i\n raise Error, \"Scroll id has expired\"\n else\n raise e\n end\n end\n end\n end\n\n def clear_scroll\n begin\n # try to clear scroll\n # not required as scroll will expire\n # but there is a cost to open scrolls\n Searchkick.client.clear_scroll(scroll_id: scroll_id)\n rescue => e\n raise e unless Searchkick.transport_error?(e)\n end\n end\n\n private\n\n attr_reader :klass, :options\n\n def results\n @results ||= with_hit.map(&:first)\n end\n\n def with_hit_and_missing_records\n @with_hit_and_missing_records ||= begin\n missing_records = []\n\n if options[:load]\n grouped_hits = hits.group_by { |hit, _| hit[\"_index\"] }\n\n # determine models\n index_models = {}\n grouped_hits.each do |index, _|\n models =\n if @klass\n [@klass]\n else\n index_alias = index.split(\"_\")[0..-2].join(\"_\")\n Array((options[:index_mapping] || {})[index_alias])\n end\n raise Error, \"Unknown model for index: #{index}. Pass the `models` option to the search method.\" unless models.any?\n index_models[index] = models\n end\n\n # fetch results\n results = {}\n grouped_hits.each do |index, index_hits|\n results[index] = {}\n index_models[index].each do |model|\n results[index].merge!(results_query(model, index_hits).to_a.index_by { |r| r.id.to_s })\n end\n end\n\n # sort\n results =\n hits.map do |hit|\n result = results[hit[\"_index\"]][hit[\"_id\"].to_s]\n if result && !(options[:load].is_a?(Hash) && options[:load][:dumpable])\n if (hit[\"highlight\"] || options[:highlight]) && !result.respond_to?(:search_highlights)\n highlights = hit_highlights(hit)\n result.define_singleton_method(:search_highlights) do\n highlights\n end\n end\n end\n [result, hit]\n end.select do |result, hit|\n unless result\n models = index_models[hit[\"_index\"]]\n missing_records << {\n id: hit[\"_id\"],\n # may be multiple models for inheritance with child models\n # not ideal to return different types\n # but this situation shouldn't be common\n model: models.size == 1 ? models.first : models\n }\n end\n result\n end\n else\n results =\n hits.map do |hit|\n result =\n if hit[\"_source\"]\n hit.except(\"_source\").merge(hit[\"_source\"])\n elsif hit[\"fields\"]\n hit.except(\"fields\").merge(hit[\"fields\"])\n else\n hit\n end\n\n if hit[\"highlight\"] || options[:highlight]\n highlight = hit[\"highlight\"].to_a.to_h { |k, v| [base_field(k), v.first] }\n options[:highlighted_fields].map { |k| base_field(k) }.each do |k|\n result[\"highlighted_#{k}\"] ||= (highlight[k] || result[k])\n end\n end\n\n result[\"id\"] ||= result[\"_id\"] # needed for legacy reasons\n [HashWrapper.new(result), hit]\n end\n end\n\n [results, missing_records]\n end\n end\n\n def build_hits\n @build_hits ||= begin\n if missing_records.any?\n Searchkick.warn(\"Records in search index do not exist in database: #{missing_records.map { |v| \"#{Array(v[:model]).map(&:model_name).sort.join(\"/\")} #{v[:id]}\" }.join(\", \")}\")\n end\n with_hit_and_missing_records[0]\n end\n end\n\n def results_query(records, hits)\n records = Searchkick.scope(records)\n\n ids = hits.map { |hit| hit[\"_id\"] }\n if options[:includes] || options[:model_includes]\n included_relations = []\n combine_includes(included_relations, options[:includes])\n combine_includes(included_relations, options[:model_includes][records]) if options[:model_includes]\n\n records = records.includes(included_relations)\n end\n\n if options[:scope_results]\n records = options[:scope_results].call(records)\n end\n\n Searchkick.load_records(records, ids)\n end\n\n def combine_includes(result, inc)\n if inc\n if inc.is_a?(Array)\n result.concat(inc)\n else\n result << inc\n end\n end\n end\n\n def base_field(k)\n k.sub(/\\.(analyzed|word_start|word_middle|word_end|text_start|text_middle|text_end|exact)\\z/, \"\")\n end\n\n def hit_highlights(hit, multiple: false)\n if hit[\"highlight\"]\n hit[\"highlight\"].to_h { |k, v| [(options[:json] ? k : k.sub(/\\.#{@options[:match_suffix]}\\z/, \"\")).to_sym, multiple ? v : v.first] }\n else\n {}\n end\n end\n end\nend\n"
},
{
"path": "lib/searchkick/script.rb",
"content": "module Searchkick\n class Script\n attr_reader :source, :lang, :params\n\n def initialize(source, lang: \"painless\", params: {})\n @source = source\n @lang = lang\n @params = params\n end\n end\nend\n"
},
{
"path": "lib/searchkick/version.rb",
"content": "module Searchkick\n VERSION = \"6.1.0\"\nend\n"
},
{
"path": "lib/searchkick/where.rb",
"content": "module Searchkick\n class Where\n def initialize(relation)\n @relation = relation\n end\n\n def not(value)\n @relation.where(_not: value)\n end\n end\nend\n"
},
{
"path": "lib/searchkick.rb",
"content": "# dependencies\nrequire \"active_support\"\nrequire \"active_support/core_ext/hash/deep_merge\"\nrequire \"active_support/core_ext/module/attr_internal\"\nrequire \"active_support/core_ext/module/delegation\"\nrequire \"active_support/deprecation\"\nrequire \"active_support/log_subscriber\"\nrequire \"active_support/notifications\"\n\n# stdlib\nrequire \"forwardable\"\n\n# modules\nrequire_relative \"searchkick/controller_runtime\"\nrequire_relative \"searchkick/index\"\nrequire_relative \"searchkick/index_cache\"\nrequire_relative \"searchkick/index_options\"\nrequire_relative \"searchkick/indexer\"\nrequire_relative \"searchkick/hash_wrapper\"\nrequire_relative \"searchkick/log_subscriber\"\nrequire_relative \"searchkick/model\"\nrequire_relative \"searchkick/multi_search\"\nrequire_relative \"searchkick/query\"\nrequire_relative \"searchkick/reindex_queue\"\nrequire_relative \"searchkick/record_data\"\nrequire_relative \"searchkick/record_indexer\"\nrequire_relative \"searchkick/relation\"\nrequire_relative \"searchkick/relation_indexer\"\nrequire_relative \"searchkick/reranking\"\nrequire_relative \"searchkick/results\"\nrequire_relative \"searchkick/script\"\nrequire_relative \"searchkick/version\"\nrequire_relative \"searchkick/where\"\n\n# integrations\nrequire_relative \"searchkick/railtie\" if defined?(Rails)\n\nmodule Searchkick\n # requires faraday\n autoload :Middleware, \"searchkick/middleware\"\n\n # background jobs\n autoload :BulkReindexJob, \"searchkick/bulk_reindex_job\"\n autoload :ProcessBatchJob, \"searchkick/process_batch_job\"\n autoload :ProcessQueueJob, \"searchkick/process_queue_job\"\n autoload :ReindexV2Job, \"searchkick/reindex_v2_job\"\n\n # errors\n class Error < StandardError; end\n class MissingIndexError < Error; end\n class UnsupportedVersionError < Error\n def message\n \"This version of Searchkick requires Elasticsearch 8+ or OpenSearch 2+\"\n end\n end\n class InvalidQueryError < Error; end\n class DangerousOperation < Error; end\n class ImportError < Error; end\n\n class << self\n attr_accessor :search_method_name, :timeout, :models, :client_options, :redis, :index_prefix, :index_suffix, :queue_name, :model_options, :client_type, :parent_job\n attr_writer :client, :env, :search_timeout\n attr_reader :aws_credentials\n end\n self.search_method_name = :search\n self.timeout = 10\n self.models = []\n self.client_options = {}\n self.queue_name = :searchkick\n self.model_options = {}\n self.parent_job = \"ActiveJob::Base\"\n\n def self.client\n @client ||= begin\n client_type =\n if self.client_type\n self.client_type\n elsif defined?(OpenSearch::Client) && defined?(Elasticsearch::Client)\n raise Error, \"Multiple clients found - set Searchkick.client_type = :elasticsearch or :opensearch\"\n elsif defined?(OpenSearch::Client)\n :opensearch\n elsif defined?(Elasticsearch::Client)\n :elasticsearch\n else\n raise Error, \"No client found - install the `elasticsearch` or `opensearch-ruby` gem\"\n end\n\n if client_type == :opensearch\n OpenSearch::Client.new({\n url: ENV[\"OPENSEARCH_URL\"],\n transport_options: {request: {timeout: timeout}},\n retry_on_failure: 2\n }.deep_merge(client_options)) do |f|\n f.use Searchkick::Middleware\n f.request :aws_sigv4, signer_middleware_aws_params if aws_credentials\n end\n else\n raise Error, \"The `elasticsearch` gem must be 8+\" if Elasticsearch::VERSION.to_i < 8\n\n Elasticsearch::Client.new({\n url: ENV[\"ELASTICSEARCH_URL\"],\n transport_options: {request: {timeout: timeout}},\n retry_on_failure: 2\n }.deep_merge(client_options)) do |f|\n f.use Searchkick::Middleware\n f.request :aws_sigv4, signer_middleware_aws_params if aws_credentials\n end\n end\n end\n end\n\n def self.env\n @env ||= ENV[\"RAILS_ENV\"] || ENV[\"RACK_ENV\"] || \"development\"\n end\n\n def self.search_timeout\n (defined?(@search_timeout) && @search_timeout) || timeout\n end\n\n # private\n def self.server_info\n @server_info ||= client.info\n end\n\n def self.server_version\n @server_version ||= server_info[\"version\"][\"number\"]\n end\n\n def self.opensearch?\n unless defined?(@opensearch)\n @opensearch = server_info[\"version\"][\"distribution\"] == \"opensearch\"\n end\n @opensearch\n end\n\n def self.server_below?(version)\n Gem::Version.new(server_version.split(\"-\")[0]) < Gem::Version.new(version.split(\"-\")[0])\n end\n\n # private\n def self.knn_support?\n if opensearch?\n !server_below?(\"2.4.0\")\n else\n !server_below?(\"8.6.0\")\n end\n end\n\n def self.search(term = \"*\", model: nil, **options, &block)\n options = options.dup\n klass = model\n\n # convert index_name into models if possible\n # this should allow for easier upgrade\n if options[:index_name] && !options[:models] && Array(options[:index_name]).all? { |v| v.respond_to?(:searchkick_index) }\n options[:models] = options.delete(:index_name)\n end\n\n # make Searchkick.search(models: [Product]) and Product.search equivalent\n unless klass\n models = Array(options[:models])\n if models.size == 1\n klass = models.first\n options.delete(:models)\n end\n end\n\n if klass\n if (options[:models] && Array(options[:models]) != [klass]) || Array(options[:index_name]).any? { |v| v.respond_to?(:searchkick_index) && v != klass }\n raise ArgumentError, \"Use Searchkick.search to search multiple models\"\n end\n end\n\n options = options.merge(block: block) if block\n Relation.new(klass, term, **options)\n end\n\n def self.multi_search(queries, opaque_id: nil)\n return if queries.empty?\n\n queries = queries.map { |q| q.send(:query) }\n event = {\n name: \"Multi Search\",\n body: queries.flat_map { |q| [q.params.except(:body).to_json, q.body.to_json] }.map { |v| \"#{v}\\n\" }.join\n }\n ActiveSupport::Notifications.instrument(\"multi_search.searchkick\", event) do\n MultiSearch.new(queries, opaque_id: opaque_id).perform\n end\n end\n\n # script\n\n # experimental\n def self.script(source, **options)\n Script.new(source, **options)\n end\n\n # callbacks\n\n def self.enable_callbacks\n self.callbacks_value = nil\n end\n\n def self.disable_callbacks\n self.callbacks_value = false\n end\n\n def self.callbacks?(default: true)\n if callbacks_value.nil?\n default\n else\n callbacks_value != false\n end\n end\n\n # message is private\n def self.callbacks(value = nil, message: nil)\n if block_given?\n previous_value = callbacks_value\n begin\n self.callbacks_value = value\n result = yield\n if callbacks_value == :bulk && indexer.queued_items.any?\n event = {}\n if message\n message.call(event)\n else\n event[:name] = \"Bulk\"\n event[:count] = indexer.queued_items.size\n end\n ActiveSupport::Notifications.instrument(\"request.searchkick\", event) do\n indexer.perform\n end\n end\n result\n ensure\n self.callbacks_value = previous_value\n end\n else\n self.callbacks_value = value\n end\n end\n\n def self.aws_credentials=(creds)\n require \"faraday_middleware/aws_sigv4\"\n\n @aws_credentials = creds\n @client = nil # reset client\n end\n\n def self.reindex_status(index_name)\n raise Error, \"Redis not configured\" unless redis\n\n batches_left = Index.new(index_name).batches_left\n {\n completed: batches_left == 0,\n batches_left: batches_left\n }\n end\n\n def self.with_redis\n if redis\n if redis.respond_to?(:with)\n redis.with do |r|\n yield r\n end\n else\n yield redis\n end\n end\n end\n\n def self.warn(message)\n super(\"[searchkick] WARNING: #{message}\")\n end\n\n # private\n def self.load_records(relation, ids)\n relation =\n if relation.respond_to?(:primary_key)\n primary_key = relation.primary_key\n raise Error, \"Need primary key to load records\" if !primary_key\n\n relation.where(primary_key => ids)\n elsif relation.respond_to?(:queryable)\n relation.queryable.for_ids(ids)\n end\n\n raise Error, \"Not sure how to load records\" if !relation\n\n relation\n end\n\n # public (for reindexing conversions)\n def self.load_model(class_name, allow_child: false)\n model = class_name.safe_constantize\n raise Error, \"Could not find class: #{class_name}\" unless model\n if allow_child\n unless model.respond_to?(:searchkick_klass)\n raise Error, \"#{class_name} is not a searchkick model\"\n end\n else\n unless Searchkick.models.include?(model)\n raise Error, \"#{class_name} is not a searchkick model\"\n end\n end\n model\n end\n\n # private\n def self.indexer\n Thread.current[:searchkick_indexer] ||= Indexer.new\n end\n\n # private\n def self.callbacks_value\n Thread.current[:searchkick_callbacks_enabled]\n end\n\n # private\n def self.callbacks_value=(value)\n Thread.current[:searchkick_callbacks_enabled] = value\n end\n\n # private\n def self.signer_middleware_aws_params\n {service: \"es\", region: \"us-east-1\"}.merge(aws_credentials)\n end\n\n # private\n # methods are forwarded to base class\n # this check to see if scope exists on that class\n # it's a bit tricky, but this seems to work\n def self.relation?(klass)\n if klass.respond_to?(:current_scope)\n !klass.current_scope.nil?\n else\n klass.is_a?(Mongoid::Criteria) || !Mongoid::Threaded.current_scope(klass).nil?\n end\n end\n\n # private\n def self.scope(model)\n # safety check to make sure used properly in code\n raise Error, \"Cannot scope relation\" if relation?(model)\n\n if model.searchkick_options[:unscope]\n model.unscoped\n else\n model\n end\n end\n\n # private\n def self.not_found_error?(e)\n (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Errors::NotFound)) ||\n (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Errors::NotFound)) ||\n (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Errors::NotFound))\n end\n\n # private\n def self.transport_error?(e)\n (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Error)) ||\n (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Error)) ||\n (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Error))\n end\n\n # private\n def self.not_allowed_error?(e)\n (defined?(Elastic::Transport) && e.is_a?(Elastic::Transport::Transport::Errors::MethodNotAllowed)) ||\n (defined?(Elasticsearch::Transport) && e.is_a?(Elasticsearch::Transport::Transport::Errors::MethodNotAllowed)) ||\n (defined?(OpenSearch) && e.is_a?(OpenSearch::Transport::Transport::Errors::MethodNotAllowed))\n end\nend\n\nActiveSupport.on_load(:active_record) do\n extend Searchkick::Model\nend\n\nActiveSupport.on_load(:mongoid) do\n Mongoid::Document::ClassMethods.include Searchkick::Model\nend\n\nActiveSupport.on_load(:action_controller) do\n include Searchkick::ControllerRuntime\nend\n\nSearchkick::LogSubscriber.attach_to :searchkick\n"
},
{
"path": "lib/tasks/searchkick.rake",
"content": "namespace :searchkick do\n desc \"reindex a model (specify CLASS)\"\n task reindex: :environment do\n class_name = ENV[\"CLASS\"]\n abort \"USAGE: rake searchkick:reindex CLASS=Product\" unless class_name\n\n model =\n begin\n Searchkick.load_model(class_name)\n rescue Searchkick::Error => e\n abort e.message\n end\n\n puts \"Reindexing #{model.name}...\"\n model.reindex\n puts \"Reindex successful\"\n end\n\n namespace :reindex do\n desc \"reindex all models\"\n task all: :environment do\n # eager load models to populate Searchkick.models\n if Rails.respond_to?(:autoloaders) && Rails.autoloaders.zeitwerk_enabled?\n # fix for https://github.com/rails/rails/issues/37006\n Zeitwerk::Loader.eager_load_all\n else\n Rails.application.eager_load!\n end\n\n Searchkick.models.each do |model|\n puts \"Reindexing #{model.name}...\"\n model.reindex\n end\n puts \"Reindex complete\"\n end\n end\nend\n"
},
{
"path": "searchkick.gemspec",
"content": "require_relative \"lib/searchkick/version\"\n\nGem::Specification.new do |spec|\n spec.name = \"searchkick\"\n spec.version = Searchkick::VERSION\n spec.summary = \"Intelligent search made easy with Rails and Elasticsearch or OpenSearch\"\n spec.homepage = \"https://github.com/ankane/searchkick\"\n spec.license = \"MIT\"\n\n spec.author = \"Andrew Kane\"\n spec.email = \"andrew@ankane.org\"\n\n spec.files = Dir[\"*.{md,txt}\", \"{lib}/**/*\"]\n spec.require_path = \"lib\"\n\n spec.required_ruby_version = \">= 3.2\"\n\n spec.add_dependency \"activemodel\", \">= 7.2\"\nend\n"
},
{
"path": "test/aggs_test.rb",
"content": "require_relative \"test_helper\"\n\nclass AggsTest < Minitest::Test\n def setup\n super\n store [\n {name: \"Product Show\", latitude: 37.7833, longitude: 12.4167, store_id: 1, in_stock: true, color: \"blue\", price: 21, created_at: 2.days.ago},\n {name: \"Product Hide\", latitude: 29.4167, longitude: -98.5000, store_id: 2, in_stock: false, color: \"green\", price: 25, created_at: 2.days.from_now},\n {name: \"Product B\", latitude: 43.9333, longitude: -122.4667, store_id: 2, in_stock: false, color: \"red\", price: 5, created_at: Time.now},\n {name: \"Foo\", latitude: 43.9333, longitude: 12.4667, store_id: 3, in_stock: false, color: \"yellow\", price: 15, created_at: Time.now}\n ]\n end\n\n def test_single\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), Product.search(\"Product\").aggs(:store_id)\n end\n\n def test_multiple\n expected = {\"store_id\" => {1 => 1, 2 => 2}, \"color\" => {\"blue\" => 1, \"green\" => 1, \"red\" => 1}}\n assert_aggs expected, aggs: [:store_id, :color]\n assert_aggs expected, Product.search(\"Product\").aggs(:store_id, :color)\n assert_aggs expected, Product.search(\"Product\").aggs([:store_id, :color])\n end\n\n def test_multiple_where\n expected = {\"store_id\" => {1 => 1}, \"color\" => {\"blue\" => 1, \"green\" => 1, \"red\" => 1}}\n assert_aggs expected, aggs: {color: {}, store_id: {where: {in_stock: true}}}\n assert_aggs expected, Product.search(\"Product\").aggs(:color, store_id: {where: {in_stock: true}})\n end\n\n def test_none\n assert_nil Product.search(\"*\").aggs\n end\n\n def test_where\n assert_aggs ({\"store_id\" => {1 => 1}}), aggs: {store_id: {where: {in_stock: true}}}\n assert_aggs ({\"store_id\" => {1 => 1}}), Product.search(\"Product\").aggs(store_id: {where: {in_stock: true}})\n assert_aggs ({\"store_id\" => {1 => 1}}), Product.search(\"Product\").aggs({store_id: {where: {in_stock: true}}})\n assert_aggs ({\"store_id\" => {1 => 1}}), aggs: {store_id: {where: {_not: {in_stock: false}}}}\n end\n\n def test_field\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), aggs: {store_id: {}}\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), aggs: {store_id: {field: \"store_id\"}}\n assert_aggs ({\"store_id_new\" => {1 => 1, 2 => 2}}), aggs: {store_id_new: {field: \"store_id\"}}\n end\n\n def test_min_doc_count\n assert_aggs ({\"store_id\" => {2 => 2}}), aggs: {store_id: {min_doc_count: 2}}\n end\n\n def test_script\n expected = {\"color\" => {\"Color: blue\" => 1, \"Color: green\" => 1, \"Color: red\" => 1}}\n assert_aggs expected, aggs: {color: {script: {source: \"'Color: ' + _value\"}}}\n end\n\n def test_order\n agg = Product.search(\"Product\", aggs: {color: {order: {_key: \"desc\"}}}).aggs[\"color\"]\n assert_equal [\"red\", \"green\", \"blue\"], agg[\"buckets\"].map { |b| b[\"key\"] }\n end\n\n def test_limit\n agg = Product.search(\"Product\", aggs: {store_id: {limit: 1}}).aggs[\"store_id\"]\n assert_equal 1, agg[\"buckets\"].size\n # assert_equal 3, agg[\"doc_count\"]\n assert_equal(1, agg[\"sum_other_doc_count\"])\n end\n\n def test_ranges\n price_ranges = [{to: 10}, {from: 10, to: 20}, {from: 20}]\n agg = Product.search(\"Product\", aggs: {price: {ranges: price_ranges}}).aggs[\"price\"]\n assert_equal 3, agg[\"buckets\"].size\n assert_equal 10.0, agg[\"buckets\"][0][\"to\"]\n assert_equal 20.0, agg[\"buckets\"][2][\"from\"]\n assert_equal 1, agg[\"buckets\"][0][\"doc_count\"]\n assert_equal 0, agg[\"buckets\"][1][\"doc_count\"]\n assert_equal 2, agg[\"buckets\"][2][\"doc_count\"]\n end\n\n def test_date_ranges\n ranges = [{to: 1.day.ago}, {from: 1.day.ago, to: 1.day.from_now}, {from: 1.day.from_now}]\n agg = Product.search(\"Product\", aggs: {created_at: {date_ranges: ranges}}).aggs[\"created_at\"]\n assert_equal 1, agg[\"buckets\"][0][\"doc_count\"]\n assert_equal 1, agg[\"buckets\"][1][\"doc_count\"]\n assert_equal 1, agg[\"buckets\"][2][\"doc_count\"]\n end\n\n def test_group_by_date\n store [{name: \"Old Product\", created_at: 3.years.ago}]\n aggs = {products_per_year: {date_histogram: {field: :created_at, calendar_interval: :year}}}\n products = Product.search(\"Product\", where: {created_at: {lt: Time.now}}, aggs: aggs)\n assert_equal 4, products.aggs[\"products_per_year\"][\"buckets\"].size\n end\n\n def test_time_zone\n start_time = Time.at(1529366400)\n store [\n {name: \"Opera House Pass\", created_at: start_time},\n {name: \"London Eye Pass\", created_at: start_time + 16.hours},\n {name: \"London Tube Pass\", created_at: start_time + 16.hours}\n ]\n\n london_aggs = {products_per_day: {date_histogram: {field: :created_at, calendar_interval: :day, time_zone: \"+01:00\"}}}\n expected = [\n {\"key_as_string\" => \"2018-06-19T00:00:00.000+01:00\", \"key\" => 1529362800000, \"doc_count\" => 3}\n ]\n assert_equal expected, Product.search(\"Pass\", aggs: london_aggs).aggs[\"products_per_day\"][\"buckets\"]\n\n sydney_aggs = {products_per_day: {date_histogram: {field: :created_at, calendar_interval: :day, time_zone: \"+10:00\"}}}\n expected = [\n {\"key_as_string\" => \"2018-06-19T00:00:00.000+10:00\", \"key\" => 1529330400000, \"doc_count\" => 1},\n {\"key_as_string\" => \"2018-06-20T00:00:00.000+10:00\", \"key\" => 1529416800000, \"doc_count\" => 2}\n ]\n assert_equal expected, Product.search(\"Pass\", aggs: sydney_aggs).aggs[\"products_per_day\"][\"buckets\"]\n end\n\n def test_avg\n products = Product.search(\"*\", aggs: {avg_price: {avg: {field: :price}}})\n assert_equal 16.5, products.aggs[\"avg_price\"][\"value\"]\n end\n\n def test_cardinality\n products = Product.search(\"*\", aggs: {total_stores: {cardinality: {field: :store_id}}})\n assert_equal 3, products.aggs[\"total_stores\"][\"value\"]\n end\n\n def test_min_max\n products = Product.search(\"*\", aggs: {min_price: {min: {field: :price}}, max_price: {max: {field: :price}}})\n assert_equal 5, products.aggs[\"min_price\"][\"value\"]\n assert_equal 25, products.aggs[\"max_price\"][\"value\"]\n end\n\n def test_sum\n products = Product.search(\"*\", aggs: {sum_price: {sum: {field: :price}}})\n assert_equal 66, products.aggs[\"sum_price\"][\"value\"]\n end\n\n def test_body_options\n expected = {\"price\" => {0.0 => 1, 10.0 => 0, 20.0 => 2}}\n assert_aggs expected, body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}}\n end\n\n def test_smart_aggs\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {in_stock: true}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {in_stock: true}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_not: {in_stock: true}}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_not: {in_stock: true}}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {_and: [{in_stock: true}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_and: [{in_stock: true}]}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {_or: [{in_stock: true}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_or: [{in_stock: true}]}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {or: [[{in_stock: true}]]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {or: [[{in_stock: true}]]}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {_script: Searchkick.script(\"doc['in_stock'].value\")}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_script: Searchkick.script(\"doc['in_stock'].value\")}, aggs: [:store_id], smart_aggs: false\n end\n\n def test_smart_aggs_overlap\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: 2}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: 2}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {store_id: 2}, aggs: [\"store_id\"]\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {\"store_id\" => 2}, aggs: [:store_id]\n\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: {not: 2}}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: {not: 2}}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: {gt: 2}}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: {gt: 2}}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {1 => 1}}), where: {_not: {store_id: 2}}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_not: {store_id: 2}}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_and: [{store_id: 2}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_and: [{store_id: 2}]}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {}}), where: {_and: [{store_id: 2}, {in_stock: true}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_and: [{store_id: 2}, {in_stock: true}]}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_or: [{store_id: 2}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_or: [{store_id: 2}]}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_or: [{store_id: 2}, {in_stock: true}]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {_or: [{store_id: 2}, {in_stock: true}]}, aggs: [:store_id], smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {or: [[{store_id: 2}]]}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {or: [[{store_id: 2}]]}, aggs: [:store_id], smart_aggs: false\n\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 1}}), where: {store_id: 2, price: {gt: 5}}, aggs: [:store_id]\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), where: {store_id: 2, price: {gt: 5}}, aggs: [:store_id], smart_aggs: false\n end\n\n def test_smart_aggs_agg_where\n assert_aggs ({\"store_id\" => {2 => 1}}), where: {color: \"red\"}, aggs: {store_id: {where: {in_stock: false}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {color: \"red\"}, aggs: {store_id: {where: {in_stock: false}}}, smart_aggs: false\n\n assert_aggs ({\"store_id\" => {}}), where: {color: \"blue\"}, aggs: {store_id: {where: {in_stock: false}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {color: \"blue\"}, aggs: {store_id: {where: {in_stock: false}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_not: {color: \"red\"}}, aggs: {store_id: {where: {_not: {in_stock: true}}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_not: {color: \"red\"}}, aggs: {store_id: {where: {_not: {in_stock: true}}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_and: [{color: \"red\"}]}, aggs: {store_id: {where: {_and: [{in_stock: false}]}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_and: [{color: \"red\"}]}, aggs: {store_id: {where: {_and: [{in_stock: false}]}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_or: [{color: \"red\"}]}, aggs: {store_id: {where: {_or: [{in_stock: false}]}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_or: [{color: \"red\"}]}, aggs: {store_id: {where: {_or: [{in_stock: false}]}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {or: [[{color: \"red\"}]]}, aggs: {store_id: {where: {or: [[{in_stock: false}]]}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {or: [[{color: \"red\"}]]}, aggs: {store_id: {where: {or: [[{in_stock: false}]]}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_script: Searchkick.script(\"doc['color'].value == 'red'\")}, aggs: {store_id: {where: {_script: Searchkick.script(\"!doc['in_stock'].value\")}}}\n assert_aggs ({\"store_id\" => {2 => 2}}), where: {_script: Searchkick.script(\"doc['color'].value == 'red'\")}, aggs: {store_id: {where: {_script: Searchkick.script(\"!doc['in_stock'].value\")}}}, smart_aggs: false\n end\n\n # only basic conditions are overridden (the rest are additive)\n def test_smart_aggs_agg_where_overlap\n assert_aggs ({\"store_id\" => {}}), where: {color: \"red\"}, aggs: {store_id: {where: {in_stock: false, color: \"blue\"}}}\n assert_aggs ({\"store_id\" => {}}), where: {color: \"red\"}, aggs: {store_id: {where: {in_stock: false, color: \"blue\"}}}, smart_aggs: false\n\n assert_aggs ({\"store_id\" => {2 => 1}}), where: {color: \"blue\"}, aggs: {store_id: {where: {in_stock: false, color: \"red\"}}}\n assert_aggs ({\"store_id\" => {2 => 1}}), where: {color: \"blue\"}, aggs: {store_id: {where: {in_stock: false, color: \"red\"}}}, smart_aggs: false\n\n # TODO change\n assert_aggs ({\"store_id\" => {}}), where: {color: \"blue\"}, aggs: {store_id: {where: {in_stock: false, \"color\" => \"red\"}}}\n # TODO change\n assert_aggs ({\"store_id\" => {}}), where: {\"color\" => \"blue\"}, aggs: {store_id: {where: {in_stock: false, color: \"red\"}}}\n\n assert_aggs ({\"store_id\" => {}}), where: {_and: [{color: \"blue\"}]}, aggs: {store_id: {where: {in_stock: false, color: \"red\"}}}\n assert_aggs ({\"store_id\" => {2 => 1}}), where: {_and: [{color: \"blue\"}]}, aggs: {store_id: {where: {in_stock: false, color: \"red\"}}}, smart_aggs: false\n end\n\n def test_smart_aggs_relation\n # TODO change\n assert_aggs ({\"store_id\" => {1 => 1}}), Product.search(\"Product\").where.not(store_id: 2).aggs(:store_id)\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 2}}), Product.search(\"Product\").where.not(store_id: 2).aggs(:store_id).smart_aggs(false)\n\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 1}}), Product.search(\"Product\").where(store_id: 2).where(price: {gt: 5}).aggs(:store_id)\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 1}}), Product.search(\"Product\").where(store_id: 2, price: {gt: 5}).aggs(:store_id)\n assert_aggs ({\"store_id\" => {1 => 1, 2 => 1}}), Product.search(\"Product\").where(_and: [{price: {gt: 5}}]).where(store_id: 2).aggs(:store_id)\n assert_aggs ({\"store_id\" => {2 => 2}}), Product.search(\"Product\").where(color: \"red\").aggs(store_id: {where: {in_stock: false}}).smart_aggs(false)\n end\n\n protected\n\n def assert_aggs(expected, options)\n if options.is_a?(Searchkick::Relation)\n assert_equal expected, agg_buckets(options)\n else\n assert_equal expected, agg_buckets(Product.search(\"Product\", **options))\n assert_equal expected, agg_buckets(build_relation(Product, \"Product\", **options))\n end\n end\n\n def agg_buckets(relation)\n relation.aggs.to_h { |f, a| [f, a[\"buckets\"].to_h { |v| [v[\"key\"], v[\"doc_count\"]] }] }\n end\nend\n"
},
{
"path": "test/boost_test.rb",
"content": "require_relative \"test_helper\"\n\nclass BoostTest < Minitest::Test\n # global boost\n\n def test_boost\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", orders_count: 10},\n {name: \"Tomato C\", orders_count: 100}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], boost: \"orders_count\"\n end\n\n def test_boost_zero\n store [\n {name: \"Zero Boost\", orders_count: 0}\n ]\n assert_order \"zero\", [\"Zero Boost\"], boost: \"orders_count\"\n end\n\n # fields\n\n def test_fields\n store [\n {name: \"Red\", color: \"White\"},\n {name: \"White\", color: \"Red Red Red\"}\n ]\n assert_order \"red\", [\"Red\", \"White\"], fields: [\"name^10\", \"color\"]\n end\n\n def test_fields_decimal\n store [\n {name: \"Red\", color: \"White\"},\n {name: \"White\", color: \"Red Red Red\"}\n ]\n assert_order \"red\", [\"Red\", \"White\"], fields: [\"name^10.5\", \"color\"]\n end\n\n def test_fields_word_start\n store [\n {name: \"Red\", color: \"White\"},\n {name: \"White\", color: \"Red Red Red\"}\n ]\n assert_order \"red\", [\"Red\", \"White\"], fields: [{\"name^10\" => :word_start}, \"color\"]\n end\n\n # for issue #855\n def test_fields_apostrophes\n store_names [\"Valentine's Day Special\"]\n assert_search \"Valentines\", [\"Valentine's Day Special\"], fields: [\"name^5\"]\n assert_search \"Valentine's\", [\"Valentine's Day Special\"], fields: [\"name^5\"]\n assert_search \"Valentine\", [\"Valentine's Day Special\"], fields: [\"name^5\"]\n end\n\n def test_boost_by\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", orders_count: 10},\n {name: \"Tomato C\", orders_count: 100}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], boost_by: [:orders_count]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], boost_by: {orders_count: {factor: 10}}\n end\n\n def test_boost_by_missing\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", orders_count: 10}\n ]\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\"], boost_by: {orders_count: {missing: 100}}\n end\n\n def test_boost_by_boost_mode_multiply\n store [\n {name: \"Tomato A\", found_rate: 0.9},\n {name: \"Tomato B\"},\n {name: \"Tomato C\", found_rate: 0.5}\n ]\n assert_order \"tomato\", [\"Tomato B\", \"Tomato A\", \"Tomato C\"], boost_by: {found_rate: {boost_mode: \"multiply\"}}\n end\n\n def test_boost_where\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", user_ids: [1, 2]},\n {name: \"Tomato C\", user_ids: [3]}\n ]\n assert_first \"tomato\", \"Tomato B\", boost_where: {user_ids: 2}\n assert_first \"tomato\", \"Tomato B\", boost_where: {user_ids: 1..2}\n assert_first \"tomato\", \"Tomato B\", boost_where: {user_ids: [1, 4]}\n assert_first \"tomato\", \"Tomato B\", boost_where: {user_ids: {value: 2, factor: 10}}\n assert_first \"tomato\", \"Tomato B\", boost_where: {user_ids: {value: [1, 4], factor: 10}}\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], boost_where: {user_ids: [{value: 1, factor: 10}, {value: 3, factor: 20}]}\n end\n\n def test_boost_where_negative_boost\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", user_ids: [2]},\n {name: \"Tomato C\", user_ids: [2]}\n ]\n assert_first \"tomato\", \"Tomato A\", boost_where: {user_ids: {value: 2, factor: 0.5}}\n end\n\n def test_boost_by_recency\n store [\n {name: \"Article 1\", created_at: 2.days.ago},\n {name: \"Article 2\", created_at: 1.day.ago},\n {name: \"Article 3\", created_at: Time.now}\n ]\n assert_order \"article\", [\"Article 3\", \"Article 2\", \"Article 1\"], boost_by_recency: {created_at: {scale: \"7d\", decay: 0.5}}\n end\n\n def test_boost_by_recency_origin\n store [\n {name: \"Article 1\", created_at: 2.days.ago},\n {name: \"Article 2\", created_at: 1.day.ago},\n {name: \"Article 3\", created_at: Time.now}\n ]\n assert_order \"article\", [\"Article 1\", \"Article 2\", \"Article 3\"], boost_by_recency: {created_at: {origin: 2.days.ago, scale: \"7d\", decay: 0.5}}\n end\n\n def test_boost_by_distance\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_order \"san\", [\"San Francisco\", \"San Antonio\", \"San Marino\"], boost_by_distance: {field: :location, origin: [37, -122], scale: \"1000mi\"}\n end\n\n def test_boost_by_distance_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_order \"san\", [\"San Francisco\", \"San Antonio\", \"San Marino\"], boost_by_distance: {field: :location, origin: {lat: 37, lon: -122}, scale: \"1000mi\"}\n end\n\n def test_boost_by_distance_v2\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_order \"san\", [\"San Francisco\", \"San Antonio\", \"San Marino\"], boost_by_distance: {location: {origin: [37, -122], scale: \"1000mi\"}}\n end\n\n def test_boost_by_distance_v2_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_order \"san\", [\"San Francisco\", \"San Antonio\", \"San Marino\"], boost_by_distance: {location: {origin: {lat: 37, lon: -122}, scale: \"1000mi\"}}\n end\n\n def test_boost_by_distance_v2_factor\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167, found_rate: 0.1},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000, found_rate: 0.99},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667, found_rate: 0.2}\n ]\n assert_order \"san\", [\"San Antonio\", \"San Francisco\", \"San Marino\"], boost_by: {found_rate: {factor: 100}}, boost_by_distance: {location: {origin: [37, -122], scale: \"1000mi\"}}\n assert_order \"san\", [\"San Francisco\", \"San Antonio\", \"San Marino\"], boost_by: {found_rate: {factor: 100}}, boost_by_distance: {location: {origin: [37, -122], scale: \"1000mi\", factor: 100}}\n end\n\n def test_boost_by_indices\n setup_animal\n store_names [\"Rex\"], Animal\n store_names [\"Rexx\"], Product\n assert_order \"Rex\", [\"Rexx\", \"Rex\"], {models: [Animal, Product], indices_boost: {Animal => 1, Product => 200}, fields: [:name]}, Searchkick\n end\nend\n"
},
{
"path": "test/callbacks_test.rb",
"content": "require_relative \"test_helper\"\n\nclass CallbacksTest < Minitest::Test\n def test_false\n Searchkick.callbacks(false) do\n store_names [\"Product A\", \"Product B\"]\n end\n assert_search \"product\", []\n end\n\n def test_bulk\n Searchkick.callbacks(:bulk) do\n store_names [\"Product A\", \"Product B\"]\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_async\n assert_enqueued_jobs 2 do\n Searchkick.callbacks(:async) do\n store_names [\"Product A\", \"Product B\"]\n end\n end\n end\n\n def test_queue\n # TODO figure out which earlier test leaves records in index\n Product.reindex\n\n reindex_queue = Product.searchkick_index.reindex_queue\n reindex_queue.clear\n\n Searchkick.callbacks(:queue) do\n store_names [\"Product A\", \"Product B\"]\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [], load: false, conversions: false\n assert_equal 2, reindex_queue.length\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\"], load: false\n assert_equal 0, reindex_queue.length\n\n Searchkick.callbacks(:queue) do\n Product.where(name: \"Product B\").destroy_all\n Product.create!(name: \"Product C\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\"], load: false\n assert_equal 2, reindex_queue.length\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product C\"], load: false\n assert_equal 0, reindex_queue.length\n\n # ensure no error with empty queue\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n\n def test_record_async\n with_options({callbacks: :async}, Song) do\n assert_enqueued_jobs 1 do\n Song.create!(name: \"Product A\")\n end\n\n assert_enqueued_jobs 1 do\n Song.first.reindex\n end\n end\n end\n\n def test_relation_async\n with_options({callbacks: :async}, Song) do\n assert_enqueued_jobs 0 do\n Song.all.reindex\n end\n end\n end\n\n def test_disable_callbacks\n # make sure callbacks default to on\n assert Searchkick.callbacks?\n\n store_names [\"Product A\"]\n\n Searchkick.disable_callbacks\n assert !Searchkick.callbacks?\n\n store_names [\"Product B\"]\n assert_search \"product\", [\"Product A\"]\n\n Searchkick.enable_callbacks\n Product.reindex\n\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\nend\n"
},
{
"path": "test/conversions_test.rb",
"content": "require_relative \"test_helper\"\n\nclass ConversionsTest < Minitest::Test\n def setup\n super\n setup_speaker\n end\n\n def test_v1\n store [\n {name: \"Tomato A\", conversions: {\"tomato\" => 1}},\n {name: \"Tomato B\", conversions: {\"tomato\" => 2}},\n {name: \"Tomato C\", conversions: {\"tomato\" => 3}}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"]\n assert_order \"TOMATO\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"]\n assert_equal_scores \"tomato\", conversions_v1: false\n end\n\n def test_v1_case\n store [\n {name: \"Tomato A\", conversions: {\"tomato\" => 1, \"TOMATO\" => 1, \"tOmAtO\" => 1}},\n {name: \"Tomato B\", conversions: {\"tomato\" => 2}}\n ]\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\"]\n end\n\n def test_v1_case_sensitive\n with_options(case_sensitive: true) do\n store [\n {name: \"Tomato A\", conversions: {\"Tomato\" => 1, \"TOMATO\" => 1, \"tOmAtO\" => 1}},\n {name: \"Tomato B\", conversions: {\"Tomato\" => 2}}\n ]\n assert_order \"Tomato\", [\"Tomato B\", \"Tomato A\"]\n end\n ensure\n Product.reindex\n end\n\n def test_v1_term\n store [\n {name: \"Tomato A\", conversions: {\"tomato\" => 1, \"soup\" => 3}},\n {name: \"Tomato B\", conversions: {\"tomato\" => 2, \"soup\" => 2}},\n {name: \"Tomato C\", conversions: {\"tomato\" => 3, \"soup\" => 1}}\n ]\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\", \"Tomato C\"], conversions_term: \"soup\"\n end\n\n def test_v1_weight\n Product.reindex\n store [\n {name: \"Product Boost\", orders_count: 20},\n {name: \"Product Conversions\", conversions: {\"product\" => 10}}\n ]\n assert_order \"product\", [\"Product Conversions\", \"Product Boost\"], boost: \"orders_count\"\n end\n\n def test_v1_multiple_conversions\n store [\n {name: \"Speaker A\", conversions_a: {\"speaker\" => 1}, conversions_b: {\"speaker\" => 6}},\n {name: \"Speaker B\", conversions_a: {\"speaker\" => 2}, conversions_b: {\"speaker\" => 5}},\n {name: \"Speaker C\", conversions_a: {\"speaker\" => 3}, conversions_b: {\"speaker\" => 4}}\n ], Speaker\n\n assert_equal_scores \"speaker\", {conversions_v1: false}, Speaker\n assert_equal_scores \"speaker\", {}, Speaker\n assert_equal_scores \"speaker\", {conversions_v1: [\"conversions_a\", \"conversions_b\"]}, Speaker\n assert_equal_scores \"speaker\", {conversions_v1: [\"conversions_b\", \"conversions_a\"]}, Speaker\n assert_order \"speaker\", [\"Speaker C\", \"Speaker B\", \"Speaker A\"], {conversions_v1: \"conversions_a\"}, Speaker\n assert_order \"speaker\", [\"Speaker A\", \"Speaker B\", \"Speaker C\"], {conversions_v1: \"conversions_b\"}, Speaker\n end\n\n def test_v1_multiple_conversions_with_boost_term\n store [\n {name: \"Speaker A\", conversions_a: {\"speaker\" => 4, \"speaker_1\" => 1}},\n {name: \"Speaker B\", conversions_a: {\"speaker\" => 3, \"speaker_1\" => 2}},\n {name: \"Speaker C\", conversions_a: {\"speaker\" => 2, \"speaker_1\" => 3}},\n {name: \"Speaker D\", conversions_a: {\"speaker\" => 1, \"speaker_1\" => 4}}\n ], Speaker\n\n assert_order \"speaker\", [\"Speaker A\", \"Speaker B\", \"Speaker C\", \"Speaker D\"], {conversions_v1: \"conversions_a\"}, Speaker\n assert_order \"speaker\", [\"Speaker D\", \"Speaker C\", \"Speaker B\", \"Speaker A\"], {conversions_v1: \"conversions_a\", conversions_term: \"speaker_1\"}, Speaker\n end\n\n def test_v2\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"tomato\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato\" => 3}}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: true\n assert_order \"TOMATO\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: true\n assert_equal_scores \"tomato\", conversions_v2: false\n end\n\n def test_v2_case\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato\" => 1, \"TOMATO\" => 1, \"tOmAtO\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"tomato\" => 2}}\n ]\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\"], conversions_v2: true\n end\n\n def test_v2_case_sensitive\n with_options(case_sensitive: true) do\n store [\n {name: \"Tomato A\", conversions_v2: {\"Tomato\" => 1, \"TOMATO\" => 1, \"tOmAtO\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"Tomato\" => 2}}\n ]\n assert_order \"Tomato\", [\"Tomato B\", \"Tomato A\"], conversions_v2: true\n end\n ensure\n Product.reindex\n end\n\n def test_v2_term\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato\" => 1, \"soup\" => 3}},\n {name: \"Tomato B\", conversions_v2: {\"tomato\" => 2, \"soup\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato\" => 3, \"soup\" => 1}}\n ]\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\", \"Tomato C\"], conversions_v2: {term: \"soup\"}\n assert_order \"tomato\", [\"Tomato A\", \"Tomato B\", \"Tomato C\"], conversions_v2: true, conversions_term: \"soup\"\n end\n\n def test_v2_weight\n Product.reindex\n store [\n {name: \"Product Boost\", orders_count: 20},\n {name: \"Product Conversions\", conversions_v2: {\"product\" => 10}}\n ]\n assert_order \"product\", [\"Product Conversions\", \"Product Boost\"], conversions_v2: true, boost: \"orders_count\"\n end\n\n def test_v2_space\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato juice\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"tomato juice\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato juice\" => 3}}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: {term: \"tomato juice\"}\n end\n\n def test_v2_dot\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato.juice\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"tomato.juice\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato.juice\" => 3}}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: {term: \"tomato.juice\"}\n end\n\n def test_v2_unicode\n store [\n {name: \"Tomato A\", conversions_v2: {\"喰らう\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"喰らう\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"喰らう\" => 3}}\n ]\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: {term: \"喰らう\"}\n end\n\n def test_v2_score\n store [\n {name: \"Tomato A\", conversions: {\"tomato\" => 1}, conversions_v2: {\"tomato\" => 1}},\n {name: \"Tomato B\", conversions: {\"tomato\" => 2}, conversions_v2: {\"tomato\" => 2}},\n {name: \"Tomato C\", conversions: {\"tomato\" => 3}, conversions_v2: {\"tomato\" => 3}}\n ]\n scores = Product.search(\"tomato\", conversions_v2: false, load: false).map(&:_score)\n scores_v2 = Product.search(\"tomato\", conversions_v1: false, conversions_v2: true, load: false).map(&:_score)\n assert_equal scores, scores_v2\n end\n\n def test_v2_factor\n store [\n {name: \"Tomato A\", conversions: {\"tomato\" => 1}, conversions_v2: {\"tomato\" => 1}},\n {name: \"Tomato B\", conversions: {\"tomato\" => 2}, conversions_v2: {\"tomato\" => 2}},\n {name: \"Tomato C\", conversions: {\"tomato\" => 3}, conversions_v2: {\"tomato\" => 3}}\n ]\n scores = Product.search(\"tomato\", conversions_v1: false, conversions_v2: true, load: false).map(&:_score)\n scores2 = Product.search(\"tomato\", conversions_v1: false, conversions_v2: {factor: 3}, load: false).map(&:_score)\n diffs = scores.zip(scores2).map { |a, b| b - a }\n assert_in_delta 6, diffs[0]\n assert_in_delta 4, diffs[1]\n assert_in_delta 2, diffs[2]\n end\n\n def test_v2_no_tokenization\n store [\n {name: \"Tomato A\"},\n {name: \"Tomato B\", conversions_v2: {\"tomato juice\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato vine\" => 3}}\n ]\n assert_equal_scores \"tomato\", conversions_v2: true\n end\n\n def test_v2_max_conversions\n conversions = 66000.times.to_h { |i| [\"term#{i}\", 1] }\n store [{name: \"Tomato A\", conversions_v2: conversions}]\n\n conversions.merge!(1000.times.to_h { |i| [\"term#{conversions.size + i}\", 1] })\n assert_raises(Searchkick::ImportError) do\n store [{name: \"Tomato B\", conversions_v2: conversions}]\n end\n end\n\n def test_v2_max_length\n store [{name: \"Tomato A\", conversions_v2: {\"a\"*32766 => 1}}]\n\n assert_raises(Searchkick::ImportError) do\n store [{name: \"Tomato B\", conversions_v2: {\"a\"*32767 => 1}}]\n end\n end\n\n def test_v2_zero\n error = assert_raises(Searchkick::ImportError) do\n store [{name: \"Tomato A\", conversions_v2: {\"tomato\" => 0}}]\n end\n assert_match \"must be a positive normal float\", error.message\n end\n\n def test_v2_partial_reindex\n store [\n {name: \"Tomato A\", conversions_v2: {\"tomato\" => 1}},\n {name: \"Tomato B\", conversions_v2: {\"tomato\" => 2}},\n {name: \"Tomato C\", conversions_v2: {\"tomato\" => 3}}\n ]\n Product.reindex(:search_name, refresh: true)\n assert_order \"tomato\", [\"Tomato C\", \"Tomato B\", \"Tomato A\"], conversions_v2: true\n end\nend\n"
},
{
"path": "test/default_scope_test.rb",
"content": "require_relative \"test_helper\"\n\nclass DefaultScopeTest < Minitest::Test\n def setup\n setup_model(Band)\n end\n\n def test_reindex\n store [\n {name: \"Test\", active: true},\n {name: \"Test 2\", active: false}\n ], reindex: false\n\n Band.reindex\n assert_search \"*\", [\"Test\"], {load: false}\n end\n\n def test_search\n Band.reindex\n Band.search(\"*\") # test works\n\n error = assert_raises(Searchkick::Error) do\n Band.all.search(\"*\")\n end\n assert_equal \"search must be called on model, not relation\", error.message\n end\n\n def default_model\n Band\n end\nend\n"
},
{
"path": "test/exclude_test.rb",
"content": "require_relative \"test_helper\"\n\nclass ExcludeTest < Minitest::Test\n def test_butter\n store_names [\"Butter Tub\", \"Peanut Butter Tub\"]\n assert_search \"butter\", [\"Butter Tub\"], exclude: [\"peanut butter\"]\n end\n\n def test_butter_word_start\n store_names [\"Butter Tub\", \"Peanut Butter Tub\"]\n assert_search \"butter\", [\"Butter Tub\"], exclude: [\"peanut butter\"], match: :word_start\n end\n\n def test_butter_exact\n store_names [\"Butter Tub\", \"Peanut Butter Tub\"]\n assert_search \"butter\", [], exclude: [\"peanut butter\"], fields: [{name: :exact}]\n end\n\n def test_same_exact\n store_names [\"Butter Tub\", \"Peanut Butter Tub\"]\n assert_search \"Butter Tub\", [\"Butter Tub\"], exclude: [\"Peanut Butter Tub\"], fields: [{name: :exact}]\n end\n\n def test_egg_word_start\n store_names [\"eggs\", \"eggplant\"]\n assert_search \"egg\", [\"eggs\"], exclude: [\"eggplant\"], match: :word_start\n end\n\n def test_string\n store_names [\"Butter Tub\", \"Peanut Butter Tub\"]\n assert_search \"butter\", [\"Butter Tub\"], exclude: \"peanut butter\"\n end\n\n def test_match_all\n store_names [\"Butter\"]\n assert_search \"*\", [], exclude: \"butter\"\n end\n\n def test_match_all_fields\n store_names [\"Butter\"]\n assert_search \"*\", [], fields: [:name], exclude: \"butter\"\n assert_search \"*\", [\"Butter\"], fields: [:color], exclude: \"butter\"\n end\nend\n"
},
{
"path": "test/geo_shape_test.rb",
"content": "require_relative \"test_helper\"\n\nclass GeoShapeTest < Minitest::Test\n def setup\n setup_region\n store [\n {\n name: \"Region A\",\n text: \"The witch had a cat\",\n territory: {\n type: \"polygon\",\n coordinates: [[[30, 40], [35, 45], [40, 40], [40, 30], [30, 30], [30, 40]]]\n }\n },\n {\n name: \"Region B\",\n text: \"and a very tall hat\",\n territory: {\n type: \"polygon\",\n coordinates: [[[50, 60], [55, 65], [60, 60], [60, 50], [50, 50], [50, 60]]]\n }\n },\n {\n name: \"Region C\",\n text: \"and long ginger hair which she wore in a plait\",\n territory: {\n type: \"polygon\",\n coordinates: [[[10, 20], [15, 25], [20, 20], [20, 10], [10, 10], [10, 20]]]\n }\n }\n ]\n end\n\n def test_envelope\n assert_search \"*\", [\"Region A\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n coordinates: [[28, 42], [32, 38]]\n }\n }\n }\n }\n end\n\n def test_polygon\n assert_search \"*\", [\"Region A\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"polygon\",\n coordinates: [[[38, 42], [42, 42], [42, 38], [38, 38], [38, 42]]]\n }\n }\n }\n }\n end\n\n def test_multipolygon\n assert_search \"*\", [\"Region A\", \"Region B\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"multipolygon\",\n coordinates: [\n [[[38, 42], [42, 42], [42, 38], [38, 38], [38, 42]]],\n [[[58, 62], [62, 62], [62, 58], [58, 58], [58, 62]]]\n ]\n }\n }\n }\n }\n end\n\n def test_disjoint\n assert_search \"*\", [\"Region B\", \"Region C\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n relation: \"disjoint\",\n coordinates: [[28, 42], [32, 38]]\n }\n }\n }\n }\n end\n\n def test_within\n assert_search \"*\", [\"Region A\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n relation: \"within\",\n coordinates: [[20, 50], [50, 20]]\n }\n }\n }\n }\n end\n\n def test_search_match\n assert_search \"witch\", [\"Region A\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n coordinates: [[28, 42], [32, 38]]\n }\n }\n }\n }\n end\n\n def test_search_no_match\n assert_search \"ginger hair\", [], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n coordinates: [[28, 42], [32, 38]]\n }\n }\n }\n }\n end\n\n def test_latlon\n assert_search \"*\", [\"Region A\"], {\n where: {\n territory: {\n geo_shape: {\n type: \"envelope\",\n coordinates: [{lat: 42, lon: 28}, {lat: 38, lon: 32}]\n }\n }\n }\n }\n end\n\n def default_model\n Region\n end\nend\n"
},
{
"path": "test/highlight_test.rb",
"content": "require_relative \"test_helper\"\n\nclass HighlightTest < Minitest::Test\n def test_basic\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\", highlight: true).highlights.first[:name]\n end\n\n def test_with_highlights\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\", highlight: true).with_highlights.first.last[:name]\n end\n\n def test_tag\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\", highlight: {tag: \"\"}).highlights.first[:name]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\").highlight(tag: \"\").highlights.first[:name]\n end\n\n def test_tag_class\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\", highlight: {tag: \"\"}).highlights.first[:name]\n end\n\n def test_very_long\n store_names [(\"Two Door Cinema Club \" * 100).strip]\n assert_equal (\"Two Door Cinema Club \" * 100).strip, Product.search(\"cinema\", highlight: true).highlights.first[:name]\n end\n\n def test_multiple_fields\n store [{name: \"Two Door Cinema Club\", color: \"Cinema Orange\"}]\n highlights = Product.search(\"cinema\", fields: [:name, :color], highlight: true).highlights.first\n assert_equal \"Two Door Cinema Club\", highlights[:name]\n assert_equal \"Cinema Orange\", highlights[:color]\n end\n\n def test_fields\n store [{name: \"Two Door Cinema Club\", color: \"Cinema Orange\"}]\n highlights = Product.search(\"cinema\", fields: [:name, :color], highlight: {fields: [:name]}).highlights.first\n assert_equal \"Two Door Cinema Club\", highlights[:name]\n assert_nil highlights[:color]\n end\n\n def test_field_options\n store_names [\"Two Door Cinema Club are a Northern Irish indie rock band\"]\n fragment_size = ENV[\"MATCH\"] == \"word_start\" ? 26 : 21\n assert_equal \"Two Door Cinema Club are\", Product.search(\"cinema\", highlight: {fields: {name: {fragment_size: fragment_size}}}).highlights.first[:name]\n end\n\n def test_multiple_words\n store_names [\"Hello World Hello\"]\n assert_equal \"Hello World Hello\", Product.search(\"hello\", highlight: true).highlights.first[:name]\n end\n\n def test_encoder\n store_names [\"Hello\"]\n assert_equal \"<b>Hello</b>\", Product.search(\"hello\", highlight: {encoder: \"html\"}, misspellings: false).highlights.first[:name]\n end\n\n def test_word_middle\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"ine\", match: :word_middle, highlight: true).highlights.first[:name]\n end\n\n def test_body\n skip if ENV[\"MATCH\"] == \"word_start\"\n store_names [\"Two Door Cinema Club\"]\n body = {\n query: {\n match: {\n \"name.analyzed\" => \"cinema\"\n }\n },\n highlight: {\n pre_tags: [\"\"],\n post_tags: [\"\"],\n fields: {\n \"name.analyzed\" => {}\n }\n }\n }\n assert_equal \"Two Door Cinema Club\", Product.search(body: body).highlights.first[:\"name.analyzed\"]\n end\n\n def test_multiple_highlights\n store_names [\"Two Door Cinema Club Some Other Words And Much More Doors Cinema Club\"]\n highlights = Product.search(\"cinema\", highlight: {fragment_size: 20}).highlights(multiple: true).first[:name]\n assert highlights.is_a?(Array)\n assert_equal highlights.count, 2\n refute_equal highlights.first, highlights.last\n highlights.each do |highlight|\n assert highlight.include?(\"Cinema\")\n end\n end\n\n def test_search_highlights_method\n store_names [\"Two Door Cinema Club\"]\n assert_equal \"Two Door Cinema Club\", Product.search(\"cinema\", highlight: true).first.search_highlights[:name]\n end\n\n def test_match_all\n store_names [\"Two Door Cinema Club\"]\n assert_nil Product.search(\"*\", highlight: true).highlights.first[:name]\n end\n\n def test_match_all_load_false\n store_names [\"Two Door Cinema Club\"]\n assert_nil Product.search(\"*\", highlight: true, load: false).highlights.first[:name]\n end\n\n def test_match_all_search_highlights\n store_names [\"Two Door Cinema Club\"]\n assert_nil Product.search(\"*\", highlight: true).first.search_highlights[:name]\n end\nend\n"
},
{
"path": "test/hybrid_test.rb",
"content": "require_relative \"test_helper\"\n\nclass HybridTest < Minitest::Test\n def setup\n skip unless Searchkick.knn_support?\n super\n end\n\n def test_search\n error = assert_raises(ArgumentError) do\n Product.search(\"product\", knn: {field: :embedding, vector: [1, 2, 3]})\n end\n assert_equal \"Use Searchkick.multi_search for hybrid search\", error.message\n end\n\n def test_multi_search\n store [\n {name: \"The dog is barking\", embedding: [1, 2, 0]},\n {name: \"The cat is purring\", embedding: [1, 0, 0]},\n {name: \"The bear is growling\", embedding: [1, 2, 3]}\n ]\n\n keyword_search = Product.search(\"growling bear\")\n semantic_search = Product.search(knn: {field: :embedding, vector: [1, 2, 3]})\n Searchkick.multi_search([keyword_search, semantic_search])\n\n results = Searchkick::Reranking.rrf(keyword_search, semantic_search)\n expected = [\"The bear is growling\", \"The dog is barking\", \"The cat is purring\"]\n assert_equal expected, results.map { |v| v[:result].name }\n assert_in_delta 0.03279, results[0][:score]\n assert_in_delta 0.01612, results[1][:score]\n assert_in_delta 0.01587, results[2][:score]\n end\nend\n"
},
{
"path": "test/index_cache_test.rb",
"content": "require_relative \"test_helper\"\n\nclass IndexCacheTest < Minitest::Test\n def setup\n Product.class_variable_get(:@@searchkick_index_cache).clear\n end\n\n def test_default\n object_id = Product.searchkick_index.object_id\n 3.times do\n assert_equal object_id, Product.searchkick_index.object_id\n end\n end\n\n def test_max_size\n starting_ids = object_ids(20)\n assert_equal starting_ids, object_ids(20)\n Product.searchkick_index(name: \"other\")\n refute_equal starting_ids, object_ids(20)\n end\n\n def test_thread_safe\n object_ids = with_threads { object_ids(20) }\n assert_equal object_ids[0], object_ids[1]\n assert_equal object_ids[0], object_ids[2]\n end\n\n # object ids can differ since threads progress at different speeds\n # test to make sure doesn't crash\n def test_thread_safe_max_size\n with_threads { object_ids(1000) }\n end\n\n private\n\n def object_ids(count)\n count.times.map { |i| Product.searchkick_index(name: \"index#{i}\").object_id }\n end\n\n def with_threads\n previous = Thread.report_on_exception\n begin\n Thread.report_on_exception = true\n 3.times.map { Thread.new { yield } }.map(&:join).map(&:value)\n ensure\n Thread.report_on_exception = previous\n end\n end\nend\n"
},
{
"path": "test/index_options_test.rb",
"content": "require_relative \"test_helper\"\n\nclass IndexOptionsTest < Minitest::Test\n def setup\n Song.destroy_all\n end\n\n def test_case_sensitive\n with_options({case_sensitive: true}) do\n store_names [\"Test\", \"test\"]\n assert_search \"test\", [\"test\"], {misspellings: false}\n end\n end\n\n def test_no_stemming\n with_options({stem: false}) do\n store_names [\"milk\", \"milks\"]\n assert_search \"milks\", [\"milks\"], {misspellings: false}\n end\n end\n\n def test_no_stem_exclusion\n with_options({}) do\n store_names [\"animals\", \"anime\"]\n assert_search \"animals\", [\"animals\", \"anime\"], {misspellings: false}\n assert_search \"anime\", [\"animals\", \"anime\"], {misspellings: false}\n assert_equal [\"anim\"], Song.searchkick_index.tokens(\"anime\", analyzer: \"searchkick_index\")\n assert_equal [\"anim\"], Song.searchkick_index.tokens(\"anime\", analyzer: \"searchkick_search2\")\n end\n end\n\n def test_stem_exclusion\n with_options({stem_exclusion: [\"anime\"]}) do\n store_names [\"animals\", \"anime\"]\n assert_search \"animals\", [\"animals\"], {misspellings: false}\n assert_search \"anime\", [\"anime\"], {misspellings: false}\n assert_equal [\"anime\"], Song.searchkick_index.tokens(\"anime\", analyzer: \"searchkick_index\")\n assert_equal [\"anime\"], Song.searchkick_index.tokens(\"anime\", analyzer: \"searchkick_search2\")\n end\n end\n\n def test_no_stemmer_override\n with_options({}) do\n store_names [\"animals\", \"animations\"]\n assert_search \"animals\", [\"animals\", \"animations\"], {misspellings: false}\n assert_search \"animations\", [\"animals\", \"animations\"], {misspellings: false}\n assert_equal [\"anim\"], Song.searchkick_index.tokens(\"animations\", analyzer: \"searchkick_index\")\n assert_equal [\"anim\"], Song.searchkick_index.tokens(\"animations\", analyzer: \"searchkick_search2\")\n end\n end\n\n def test_stemmer_override\n with_options({stemmer_override: [\"animations => animat\"]}) do\n store_names [\"animals\", \"animations\"]\n assert_search \"animals\", [\"animals\"], {misspellings: false}\n assert_search \"animations\", [\"animations\"], {misspellings: false}\n assert_equal [\"animat\"], Song.searchkick_index.tokens(\"animations\", analyzer: \"searchkick_index\")\n assert_equal [\"animat\"], Song.searchkick_index.tokens(\"animations\", analyzer: \"searchkick_search2\")\n end\n end\n\n def test_special_characters\n with_options({special_characters: false}) do\n store_names [\"jalapeño\"]\n assert_search \"jalapeno\", [], {misspellings: false}\n end\n end\n\n def test_index_name\n with_options({index_name: \"songs_v2\"}) do\n assert_equal \"songs_v2\", Song.searchkick_index.name\n end\n end\n\n def test_index_name_callable\n with_options({index_name: -> { \"songs_v2\" }}) do\n assert_equal \"songs_v2\", Song.searchkick_index.name\n end\n end\n\n def test_index_prefix\n with_options({index_prefix: \"hello\"}) do\n assert_equal \"hello_songs_test\", Song.searchkick_index.name\n end\n end\n\n def test_index_prefix_callable\n with_options({index_prefix: -> { \"hello\" }}) do\n assert_equal \"hello_songs_test\", Song.searchkick_index.name\n end\n end\n\n def default_model\n Song\n end\nend\n"
},
{
"path": "test/index_test.rb",
"content": "require_relative \"test_helper\"\n\nclass IndexTest < Minitest::Test\n def setup\n super\n setup_region\n end\n\n def test_tokens\n assert_equal [\"dollar\", \"dollartre\", \"tree\"], Product.searchkick_index.tokens(\"Dollar Tree\", analyzer: \"searchkick_index\")\n end\n\n def test_tokens_analyzer\n assert_equal [\"dollar\", \"tree\"], Product.searchkick_index.tokens(\"Dollar Tree\", analyzer: \"searchkick_search2\")\n end\n\n def test_total_docs\n store_names [\"Product A\"]\n assert_equal 1, Product.searchkick_index.total_docs\n end\n\n def test_clean_indices\n suffix = Searchkick.index_suffix ? \"_#{Searchkick.index_suffix}\" : \"\"\n old_index = Searchkick::Index.new(\"products_test#{suffix}_20130801000000000\")\n different_index = Searchkick::Index.new(\"items_test#{suffix}_20130801000000000\")\n\n old_index.delete if old_index.exists?\n different_index.delete if different_index.exists?\n\n # create indexes\n old_index.create\n different_index.create\n\n Product.searchkick_index.clean_indices\n\n assert Product.searchkick_index.exists?\n assert different_index.exists?\n assert !old_index.exists?\n end\n\n def test_clean_indices_old_format\n suffix = Searchkick.index_suffix ? \"_#{Searchkick.index_suffix}\" : \"\"\n old_index = Searchkick::Index.new(\"products_test#{suffix}_20130801000000\")\n old_index.create\n\n Product.searchkick_index.clean_indices\n\n assert !old_index.exists?\n end\n\n def test_retain\n Product.reindex\n assert_equal 1, Product.searchkick_index.all_indices.size\n Product.reindex(retain: true)\n assert_equal 2, Product.searchkick_index.all_indices.size\n end\n\n def test_mappings\n store_names [\"Dollar Tree\"], Store\n assert_equal [\"Dollar Tree\"], Store.search(body: {query: {match: {name: \"dollar\"}}}).map(&:name)\n mapping = Store.searchkick_index.mapping\n assert_kind_of Hash, mapping\n assert_equal \"text\", mapping.values.first[\"mappings\"][\"properties\"][\"name\"][\"type\"]\n end\n\n def test_settings\n assert_kind_of Hash, Store.searchkick_index.settings\n end\n\n def test_remove_blank_id\n store_names [\"Product A\"]\n Product.searchkick_index.remove(Product.new)\n assert_search \"product\", [\"Product A\"]\n ensure\n Product.reindex\n end\n\n # keep simple for now, but maybe return client response in future\n def test_store_response\n product = Searchkick.callbacks(false) { Product.create!(name: \"Product A\") }\n assert_nil Product.searchkick_index.store(product)\n end\n\n # keep simple for now, but maybe return client response in future\n def test_bulk_index_response\n product = Searchkick.callbacks(false) { Product.create!(name: \"Product A\") }\n assert_nil Product.searchkick_index.bulk_index([product])\n end\n\n # TODO move\n\n def test_filterable\n store [{name: \"Product A\", alt_description: \"Hello\"}]\n error = assert_raises(Searchkick::InvalidQueryError) do\n assert_search \"*\", [], where: {alt_description: \"Hello\"}\n end\n assert_match \"Cannot search on field [alt_description] since it is not indexed\", error.message\n end\n\n def test_filterable_non_string\n store [{name: \"Product A\", store_id: 1}]\n assert_search \"*\", [\"Product A\"], where: {store_id: 1}\n end\n\n def test_large_value\n large_value = 1000.times.map { \"hello\" }.join(\" \")\n store [{name: \"Product A\", text: large_value}], Region\n assert_search \"product\", [\"Product A\"], {}, Region\n assert_search \"hello\", [\"Product A\"], {fields: [:name, :text]}, Region\n assert_search \"hello\", [\"Product A\"], {}, Region\n assert_search \"*\", [\"Product A\"], {where: {text: large_value}}, Region\n end\n\n def test_very_large_value\n # terms must be < 32 KB with Elasticsearch 8.10.3+\n # https://github.com/elastic/elasticsearch/pull/99818\n large_value = 5400.times.map { \"hello\" }.join(\" \")\n store [{name: \"Product A\", text: large_value}], Region\n assert_search \"product\", [\"Product A\"], {}, Region\n assert_search \"hello\", [\"Product A\"], {fields: [:name, :text]}, Region\n assert_search \"hello\", [\"Product A\"], {}, Region\n # keyword not indexed\n assert_search \"*\", [], {where: {text: large_value}}, Region\n end\n\n def test_bulk_import_raises_error\n valid_dog = Product.create(name: \"2016-01-02\")\n invalid_dog = Product.create(name: \"Ol' One-Leg\")\n mapping = {\n properties: {\n name: {type: \"date\"}\n }\n }\n index = Searchkick::Index.new \"dogs\", mappings: mapping, _type: \"dog\"\n index.delete if index.exists?\n index.create_index\n index.store valid_dog\n assert_raises(Searchkick::ImportError) do\n index.bulk_index [valid_dog, invalid_dog]\n end\n end\nend\n"
},
{
"path": "test/inheritance_test.rb",
"content": "require_relative \"test_helper\"\n\nclass InheritanceTest < Minitest::Test\n def setup\n super\n setup_animal\n end\n\n def test_child_reindex\n store_names [\"Max\"], Cat\n assert Dog.reindex\n assert_equal 1, Animal.search(\"*\").size\n end\n\n def test_child_index_name\n assert_equal \"animals_test#{ENV[\"TEST_ENV_NUMBER\"]}\", Dog.searchkick_index.name\n end\n\n def test_child_search\n store_names [\"Bear\"], Dog\n store_names [\"Bear\"], Cat\n assert_equal 1, Dog.search(\"bear\").size\n end\n\n def test_parent_search\n store_names [\"Bear\"], Dog\n store_names [\"Bear\"], Cat\n assert_equal 2, Animal.search(\"bear\").size\n end\n\n def test_force_one_type\n store_names [\"Green Bear\"], Dog\n store_names [\"Blue Bear\"], Cat\n assert_equal [\"Blue Bear\"], Animal.search(\"bear\", type: [Cat]).map(&:name)\n end\n\n def test_force_multiple_types\n store_names [\"Green Bear\"], Dog\n store_names [\"Blue Bear\"], Cat\n store_names [\"Red Bear\"], Animal\n assert_equal [\"Green Bear\", \"Blue Bear\"], Animal.search(\"bear\", type: [Dog, Cat]).map(&:name)\n end\n\n def test_child_autocomplete\n store_names [\"Max\"], Cat\n store_names [\"Mark\"], Dog\n assert_equal [\"Max\"], Cat.search(\"ma\", fields: [:name], match: :text_start).map(&:name)\n end\n\n def test_parent_autocomplete\n store_names [\"Max\"], Cat\n store_names [\"Bear\"], Dog\n assert_equal [\"Bear\"], Animal.search(\"bea\", fields: [:name], match: :text_start).map(&:name).sort\n end\n\n # def test_child_suggest\n # store_names [\"Shark\"], Cat\n # store_names [\"Sharp\"], Dog\n # assert_equal [\"shark\"], Cat.search(\"shar\", fields: [:name], suggest: true).suggestions\n # end\n\n def test_parent_suggest\n store_names [\"Shark\"], Cat\n store_names [\"Tiger\"], Dog\n assert_equal [\"tiger\"], Animal.search(\"tige\", fields: [:name], suggest: true).suggestions.sort\n end\n\n def test_reindex\n store_names [\"Bear A\"], Cat\n store_names [\"Bear B\"], Dog\n Animal.reindex\n assert_equal 2, Animal.search(\"bear\").size\n end\n\n def test_child_models_option\n store_names [\"Bear A\"], Cat\n store_names [\"Bear B\"], Dog\n Animal.reindex\n # note: the models option is less efficient than Animal.search(\"bear\", type: [Cat, Dog])\n # since it requires two database calls instead of one to Animal\n assert_equal 2, Searchkick.search(\"bear\", models: [Cat, Dog]).size\n end\n\n def test_missing_records\n store_names [\"Bear A\"], Cat\n store_names [\"Bear B\"], Dog\n Animal.reindex\n record = Animal.find_by(name: \"Bear A\")\n record.delete\n assert_output nil, /\\[searchkick\\] WARNING: Records in search index do not exist in database: Cat\\/Dog \\d+/ do\n result = Searchkick.search(\"bear\", models: [Cat, Dog])\n assert_equal [\"Bear B\"], result.map(&:name)\n assert_equal [record.id.to_s], result.missing_records.map { |v| v[:id] }\n assert_equal [[Cat, Dog]], result.missing_records.map { |v| v[:model].sort_by(&:model_name) }\n end\n assert_empty Product.search(\"bear\", load: false).missing_records\n ensure\n Animal.reindex\n end\n\n def test_inherited_and_non_inherited_models\n store_names [\"Bear A\"], Cat\n store_names [\"Bear B\"], Dog\n store_names [\"Bear C\"]\n Animal.reindex\n assert_equal 2, Searchkick.search(\"bear\", models: [Cat, Product]).size\n assert_equal 2, Searchkick.search(\"bear\", models: [Cat, Product]).hits.size\n assert_equal 2, Searchkick.search(\"bear\", models: [Cat, Product], per_page: 1).total_pages\n end\n\n # TODO move somewhere better\n\n def test_multiple_indices\n store_names [\"Product A\"]\n store_names [\"Product B\"], Animal\n assert_search \"product\", [\"Product A\", \"Product B\"], {models: [Product, Animal], conversions: false}, Searchkick\n assert_search \"product\", [\"Product A\", \"Product B\"], {index_name: [Product, Animal], conversions: false}, Searchkick\n end\n\n def test_index_name_model\n store_names [\"Product A\"]\n assert_equal [\"Product A\"], Searchkick.search(\"product\", index_name: [Product]).map(&:name)\n end\n\n def test_index_name_string\n store_names [\"Product A\"]\n error = assert_raises Searchkick::Error do\n Searchkick.search(\"product\", index_name: [Product.searchkick_index.name]).map(&:name)\n end\n assert_includes error.message, \"Unknown model\"\n end\n\n def test_similar\n store_names [\"Dog\", \"Other dog\"], Dog\n store_names [\"Not dog\"], Cat\n\n dog = Dog.find_by!(name: \"Dog\")\n assert_equal [\"Other dog\"], dog.similar(fields: [:name]).map(&:name)\n assert_equal [\"Not dog\", \"Other dog\"], dog.similar(fields: [:name], models: [Animal]).map(&:name).sort\n assert_equal [\"Not dog\"], dog.similar(fields: [:name], models: [Cat]).map(&:name).sort\n end\nend\n"
},
{
"path": "test/knn_test.rb",
"content": "require_relative \"test_helper\"\n\nclass KnnTest < Minitest::Test\n def setup\n skip unless Searchkick.knn_support?\n super\n\n # prevent null_pointer_exception with OpenSearch 3\n Product.reindex if Searchkick.opensearch? && !Searchkick.server_below?(\"3.0.0\")\n end\n\n def test_basic\n store [{name: \"A\", embedding: [1, 2, 3]}, {name: \"B\", embedding: [-1, -2, -3]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding, vector: [1, 2, 3]}\n\n scores = Product.search(knn: {field: :embedding, vector: [1, 2, 3]}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1, scores[0]\n assert_in_delta 0, scores[1]\n end\n\n def test_basic_exact\n store [{name: \"A\", embedding: [1, 2, 3]}, {name: \"B\", embedding: [-1, -2, -3]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding, vector: [1, 2, 3], exact: true}\n\n scores = Product.search(knn: {field: :embedding, vector: [1, 2, 3], exact: true}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1, scores[0]\n assert_in_delta 0, scores[1]\n end\n\n def test_where\n store [\n {name: \"A\", store_id: 1, embedding: [1, 2, 3]},\n {name: \"B\", store_id: 2, embedding: [1, 2, 3]},\n {name: \"C\", store_id: 1, embedding: [-1, -2, -3]},\n {name: \"D\", store_id: 1}\n ]\n assert_order \"*\", [\"A\", \"C\"], knn: {field: :embedding, vector: [1, 2, 3]}, where: {store_id: 1}\n end\n\n def test_where_exact\n store [\n {name: \"A\", store_id: 1, embedding: [1, 2, 3]},\n {name: \"B\", store_id: 2, embedding: [1, 2, 3]},\n {name: \"C\", store_id: 1, embedding: [-1, -2, -3]},\n {name: \"D\", store_id: 1}\n ]\n assert_order \"*\", [\"A\", \"C\"], knn: {field: :embedding, vector: [1, 2, 3], exact: true}, where: {store_id: 1}\n end\n\n def test_pagination\n store [\n {name: \"A\", embedding: [1, 2, 3]},\n {name: \"B\", embedding: [1, 2, 0]},\n {name: \"C\", embedding: [-1, -2, 0]},\n {name: \"D\", embedding: [-1, -2, -3]},\n {name: \"E\"}\n ]\n assert_order \"*\", [\"B\", \"C\"], knn: {field: :embedding, vector: [1, 2, 3]}, limit: 2, offset: 1\n end\n\n def test_pagination_exact\n store [\n {name: \"A\", embedding: [1, 2, 3]},\n {name: \"B\", embedding: [1, 2, 0]},\n {name: \"C\", embedding: [-1, -2, 0]},\n {name: \"D\", embedding: [-1, -2, -3]},\n {name: \"E\"}\n ]\n assert_order \"*\", [\"B\", \"C\"], knn: {field: :embedding, vector: [1, 2, 3], exact: true}, limit: 2, offset: 1\n end\n\n def test_euclidean\n store [{name: \"A\", embedding3: [1, 2, 3]}, {name: \"B\", embedding3: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding3, vector: [1, 2, 3]}\n\n scores = Product.search(knn: {field: :embedding3, vector: [1, 2, 3]}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1.0 / (1 + 0), scores[0]\n assert_in_delta 1.0 / (1 + 5**2), scores[1]\n end\n\n def test_euclidean_exact\n store [{name: \"A\", embedding2: [1, 2, 3]}, {name: \"B\", embedding2: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding2, vector: [1, 2, 3], distance: \"euclidean\"}\n\n scores = Product.search(knn: {field: :embedding2, vector: [1, 2, 3], distance: \"euclidean\"}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1.0 / (1 + 0), scores[0]\n assert_in_delta 1.0 / (1 + 5**2), scores[1]\n end\n\n def test_taxicab_exact\n store [{name: \"A\", embedding2: [1, 2, 3]}, {name: \"B\", embedding2: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding2, vector: [1, 2, 3], distance: \"taxicab\"}\n\n scores = Product.search(knn: {field: :embedding2, vector: [1, 2, 3], distance: \"taxicab\"}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1.0 / (1 + 0), scores[0]\n assert_in_delta 1.0 / (1 + 7), scores[1]\n end\n\n def test_chebyshev_exact\n skip unless Searchkick.opensearch?\n\n store [{name: \"A\", embedding: [1, 2, 3]}, {name: \"B\", embedding: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding, vector: [1, 2, 3], distance: \"chebyshev\"}\n\n scores = Product.search(knn: {field: :embedding, vector: [1, 2, 3], distance: \"chebyshev\"}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1.0 / (1 + 0), scores[0]\n assert_in_delta 1.0 / (1 + 4), scores[1]\n end\n\n def test_inner_product\n store [{name: \"A\", embedding2: [-1, -2, -3]}, {name: \"B\", embedding2: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"B\", \"A\"], knn: {field: :embedding2, vector: [1, 2, 3], distance: \"inner_product\"}\n\n scores = Product.search(knn: {field: :embedding2, vector: [1, 2, 3], distance: \"inner_product\"}).hits.map { |v| v[\"_score\"] }\n # d > 0: d + 1\n # else: 1 / (1 - d)\n assert_in_delta 1 + 32, scores[0], (!Searchkick.opensearch? ? 0.5 : 0.001)\n assert_in_delta 1.0 / (1 + 14), scores[1]\n end\n\n def test_inner_product_exact\n store [{name: \"A\", embedding3: [-1, -2, -3]}, {name: \"B\", embedding3: [1, 5, 7]}, {name: \"C\"}]\n assert_order \"*\", [\"B\", \"A\"], knn: {field: :embedding3, vector: [1, 2, 3], distance: \"inner_product\"}\n\n scores = Product.search(knn: {field: :embedding3, vector: [1, 2, 3], distance: \"inner_product\"}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1 + 32, scores[0]\n assert_in_delta 1.0 / (1 + 14), scores[1]\n end\n\n def test_unindexed\n skip if Searchkick.opensearch?\n\n store [{name: \"A\", embedding4: [1, 2, 3]}, {name: \"B\", embedding4: [-1, -2, -3]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding4, vector: [1, 2, 3], distance: \"cosine\"}\n\n scores = Product.search(knn: {field: :embedding4, vector: [1, 2, 3], distance: \"cosine\"}).hits.map { |v| v[\"_score\"] }\n assert_in_delta 1, scores[0]\n assert_in_delta 0, scores[1]\n\n error = assert_raises(ArgumentError) do\n Product.search(knn: {field: :embedding4, vector: [1, 2, 3]})\n end\n assert_match \"distance required\", error.message\n\n error = assert_raises(ArgumentError) do\n Product.search(knn: {field: :embedding4, vector: [1, 2, 3], exact: false})\n end\n assert_match \"distance required\", error.message\n\n error = assert_raises(ArgumentError) do\n Product.search(knn: {field: :embedding, vector: [1, 2, 3], distance: \"euclidean\", exact: false})\n end\n assert_equal \"distance must match searchkick options for approximate search\", error.message\n\n if !Searchkick.server_below?(\"9.0.0\")\n error = assert_raises(ArgumentError) do\n Product.search(knn: {field: :embedding, vector: [1, 2, 3], distance: \"euclidean\"})\n end\n assert_equal \"distance must match searchkick options\", error.message\n end\n end\n\n def test_explain\n store [{name: \"A\", embedding: [1, 2, 3], embedding2: [1, 2, 3], embedding3: [1, 2, 3], embedding4: [1, 2, 3]}]\n\n assert_approx true, :embedding, \"cosine\"\n\n if Searchkick.opensearch? || Searchkick.server_below?(\"9.0.0\")\n assert_approx false, :embedding, \"euclidean\"\n assert_approx false, :embedding, \"inner_product\"\n assert_approx false, :embedding, \"taxicab\"\n end\n\n if Searchkick.opensearch?\n assert_approx false, :embedding, \"chebyshev\"\n end\n\n assert_approx false, :embedding3, \"cosine\"\n assert_approx true, :embedding3, \"euclidean\"\n assert_approx false, :embedding3, \"inner_product\"\n\n unless Searchkick.opensearch?\n assert_approx false, :embedding4, \"cosine\"\n assert_approx false, :embedding4, \"euclidean\"\n assert_approx false, :embedding4, \"inner_product\"\n end\n\n assert_approx false, :embedding2, \"cosine\"\n assert_approx false, :embedding2, \"euclidean\"\n assert_approx true, :embedding2, \"inner_product\"\n\n assert_approx false, :embedding, \"cosine\", exact: true\n assert_approx true, :embedding, \"cosine\", exact: false\n\n error = assert_raises(ArgumentError) do\n assert_approx true, :embedding, \"euclidean\", exact: false\n end\n assert_equal \"distance must match searchkick options for approximate search\", error.message\n end\n\n def test_ef_search\n skip if Searchkick.opensearch? && Searchkick.server_below?(\"2.16.0\")\n\n store [{name: \"A\", embedding: [1, 2, 3]}, {name: \"B\", embedding: [-1, -2, -3]}, {name: \"C\"}]\n assert_order \"*\", [\"A\", \"B\"], knn: {field: :embedding, vector: [1, 2, 3], ef_search: 20}, limit: 10\n end\n\n private\n\n def assert_approx(approx, field, distance, **knn_options)\n response = Product.search(knn: {field: field, vector: [1, 2, 3], distance: distance, **knn_options}, explain: true).response.to_s\n if approx\n if Searchkick.opensearch?\n assert_match \"within top\", response\n else\n assert_match \"within top k documents\", response\n end\n else\n if Searchkick.opensearch?\n assert_match \"knn_score\", response\n else\n assert_match \"params.query_vector\", response\n end\n end\n end\nend\n"
},
{
"path": "test/language_test.rb",
"content": "require_relative \"test_helper\"\n\nclass LanguageTest < Minitest::Test\n def setup\n skip \"Requires plugin\" unless ci? || ENV[\"TEST_LANGUAGE\"]\n\n Song.destroy_all\n end\n\n def test_chinese\n skip if ci?\n\n # requires https://github.com/medcl/elasticsearch-analysis-ik\n with_options({language: \"chinese\"}) do\n store_names [\"中华人民共和国国歌\"]\n assert_language_search \"中华人民共和国\", [\"中华人民共和国国歌\"]\n assert_language_search \"国歌\", [\"中华人民共和国国歌\"]\n assert_language_search \"人\", []\n end\n end\n\n def test_chinese2\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-smartcn.html\n with_options({language: \"chinese2\"}) do\n store_names [\"中华人民共和国国歌\"]\n assert_language_search \"中华人民共和国\", [\"中华人民共和国国歌\"]\n # assert_language_search \"国歌\", [\"中华人民共和国国歌\"]\n assert_language_search \"人\", []\n end\n end\n\n def test_japanese\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html\n with_options({language: \"japanese\"}) do\n store_names [\"JR新宿駅の近くにビールを飲みに行こうか\"]\n assert_language_search \"飲む\", [\"JR新宿駅の近くにビールを飲みに行こうか\"]\n assert_language_search \"jr\", [\"JR新宿駅の近くにビールを飲みに行こうか\"]\n assert_language_search \"新\", []\n end\n end\n\n def test_japanese_search_synonyms\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-kuromoji.html\n with_options({language: \"japanese\", search_synonyms: [[\"飲む\", \"喰らう\"]]}) do\n store_names [\"JR新宿駅の近くにビールを飲みに行こうか\"]\n assert_language_search \"喰らう\", [\"JR新宿駅の近くにビールを飲みに行こうか\"]\n assert_language_search \"新\", []\n end\n end\n\n def test_korean\n skip if ci?\n\n # requires https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext\n with_options({language: \"korean\"}) do\n store_names [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"처리\", [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"한국어\", [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"를\", []\n end\n end\n\n def test_korean2\n skip if ci?\n\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-nori.html\n with_options({language: \"korean2\"}) do\n store_names [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"처리\", [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"한국어\", [\"한국어를 처리하는 예시입니닼ㅋㅋ\"]\n assert_language_search \"를\", []\n end\n end\n\n def test_polish\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-stempel.html\n with_options({language: \"polish\"}) do\n store_names [\"polski\"]\n assert_language_search \"polskimi\", [\"polski\"]\n end\n end\n\n def test_ukrainian\n # requires https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html\n with_options({language: \"ukrainian\"}) do\n store_names [\"ресторани\"]\n assert_language_search \"ресторан\", [\"ресторани\"]\n end\n end\n\n def test_vietnamese\n skip if ci?\n\n # requires https://github.com/duydo/elasticsearch-analysis-vietnamese\n with_options({language: \"vietnamese\"}) do\n store_names [\"công nghệ thông tin Việt Nam\"]\n assert_language_search \"công nghệ thông tin\", [\"công nghệ thông tin Việt Nam\"]\n assert_language_search \"công\", []\n end\n end\n\n def test_stemmer_hunspell\n skip if ci?\n\n with_options({stemmer: {type: \"hunspell\", locale: \"en_US\"}}) do\n store_names [\"the foxes jumping quickly\"]\n assert_language_search \"fox\", [\"the foxes jumping quickly\"]\n end\n end\n\n def test_stemmer_unknown_type\n error = assert_raises(ArgumentError) do\n with_options({stemmer: {type: \"bad\"}}) do\n end\n end\n assert_equal \"Unknown stemmer: bad\", error.message\n end\n\n def test_stemmer_language\n skip if ci?\n\n error = assert_raises(ArgumentError) do\n with_options({stemmer: {type: \"hunspell\", locale: \"en_US\"}, language: \"english\"}) do\n end\n end\n assert_equal \"Can't pass both language and stemmer\", error.message\n end\n\n def assert_language_search(term, expected)\n assert_search term, expected, {misspellings: false}\n end\n\n def default_model\n Song\n end\nend\n"
},
{
"path": "test/load_test.rb",
"content": "require_relative \"test_helper\"\n\nclass LoadTest < Minitest::Test\n def test_default\n store_names [\"Product A\"]\n product = Product.search(\"product\").first\n assert_kind_of Product, product\n if mongoid?\n assert_match \"# \"Product A\"}), product.as_json(only: [\"name\"])\n assert_equal ({\"name\" => \"Product A\"}), product.as_json(only: [:name])\n refute product.as_json(except: [\"name\"]).key?(\"name\")\n refute product.as_json(except: [:name]).key?(\"name\")\n assert_empty product.as_json(only: [\"missing\"])\n if mongoid?\n product.as_json(methods: [:missing])\n else\n assert_raises(NoMethodError) do\n product.as_json(methods: [:missing])\n end\n end\n end\n\n def test_false\n store_names [\"Product A\"]\n product = Product.search(\"product\", load: false).first\n assert_kind_of Searchkick::HashWrapper, product\n assert_match \"# \"Product A\"}), product.as_json(only: [\"name\"])\n # same behavior as Hashie::Mash\n assert_empty product.as_json(only: [:name])\n refute product.as_json(except: [\"name\"]).key?(\"name\")\n # same behavior as Hashie::Mash\n assert product.as_json(except: [:name]).key?(\"name\")\n assert_empty product.as_json(only: [\"missing\"])\n # same behavior as Hashie::Mash\n product.as_json(methods: [:missing])\n end\n\n def test_false_methods\n store_names [\"Product A\"]\n assert_equal \"Product A\", Product.search(\"product\", load: false).first.name\n end\n\n def test_false_with_includes\n store_names [\"Product A\"]\n assert_kind_of Searchkick::HashWrapper, Product.search(\"product\", load: false, includes: [:store]).first\n end\n\n def test_false_nested_object\n aisle = {\"id\" => 1, \"name\" => \"Frozen\"}\n store [{name: \"Product A\", aisle: aisle}]\n assert_equal aisle, Product.search(\"product\", load: false).first.aisle.to_hash\n end\nend\n"
},
{
"path": "test/log_subscriber_test.rb",
"content": "require_relative \"test_helper\"\n\nclass LogSubscriberTest < Minitest::Test\n def test_create\n output = capture_logs do\n Product.create!(name: \"Product A\")\n end\n assert_match \"Product Store\", output\n end\n\n def test_update\n product = Product.create!(name: \"Product A\")\n output = capture_logs do\n product.reindex(:search_name)\n end\n assert_match \"Product Update\", output\n end\n\n def test_destroy\n product = Product.create!(name: \"Product A\")\n output = capture_logs do\n product.destroy\n end\n assert_match \"Product Remove\", output\n end\n\n def test_bulk\n output = capture_logs do\n Searchkick.callbacks(:bulk) do\n Product.create!(name: \"Product A\")\n end\n end\n assert_match \"Bulk\", output\n refute_match \"Product Store\", output\n end\n\n def test_reindex\n create_products\n output = capture_logs do\n Product.reindex\n end\n assert_match \"Product Import\", output\n assert_match '\"count\":3', output\n end\n\n def test_reindex_relation\n products = create_products\n output = capture_logs do\n Product.where.not(id: products.last.id).reindex\n end\n assert_match \"Product Import\", output\n assert_match '\"count\":2', output\n end\n\n def test_search\n # prevent warnings\n Product.searchkick_index.refresh\n\n output = capture_logs do\n Product.search(\"product\").to_a\n end\n assert_match \"Product Search\", output\n end\n\n def test_multi_search\n # prevent warnings\n Product.searchkick_index.refresh\n\n output = capture_logs do\n Searchkick.multi_search([Product.search(\"product\")])\n end\n assert_match \"Multi Search\", output\n end\n\n private\n\n def create_products\n Searchkick.callbacks(false) do\n 3.times.map do\n Product.create!(name: \"Product A\")\n end\n end\n end\n\n def capture_logs\n previous_logger = ActiveSupport::LogSubscriber.logger\n io = StringIO.new\n begin\n ActiveSupport::LogSubscriber.logger = ActiveSupport::Logger.new(io)\n yield\n io.rewind\n output = io.read\n previous_logger.debug(output) if previous_logger\n puts output if ENV[\"LOG_SUBSCRIBER\"]\n output\n ensure\n ActiveSupport::LogSubscriber.logger = previous_logger\n end\n end\nend\n"
},
{
"path": "test/marshal_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MarshalTest < Minitest::Test\n def test_marshal\n store_names [\"Product A\"]\n assert Marshal.dump(Product.search(\"*\").to_a)\n end\n\n def test_marshal_highlights\n store_names [\"Product A\"]\n assert Marshal.dump(Product.search(\"product\", highlight: true, load: {dumpable: true}).to_a)\n end\nend\n"
},
{
"path": "test/match_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MatchTest < Minitest::Test\n # exact\n\n def test_match\n store_names [\"Whole Milk\", \"Fat Free Milk\", \"Milk\"]\n assert_search \"milk\", [\"Milk\", \"Whole Milk\", \"Fat Free Milk\"]\n end\n\n def test_case\n store_names [\"Whole Milk\", \"Fat Free Milk\", \"Milk\"]\n assert_search \"MILK\", [\"Milk\", \"Whole Milk\", \"Fat Free Milk\"]\n end\n\n def test_cheese_space_in_index\n store_names [\"Pepper Jack Cheese Skewers\"]\n assert_search \"pepperjack cheese skewers\", [\"Pepper Jack Cheese Skewers\"]\n end\n\n # def test_cheese_space_in_query\n # store_names [\"Pepperjack Cheese Skewers\"]\n # assert_search \"pepper jack cheese skewers\", [\"Pepperjack Cheese Skewers\"]\n # end\n\n def test_middle_token\n store_names [\"Dish Washer Amazing Organic Soap\"]\n assert_search \"dish soap\", [\"Dish Washer Amazing Organic Soap\"]\n end\n\n def test_middle_token_wine\n store_names [\"Beringer Wine Founders Estate Chardonnay\"]\n assert_search \"beringer chardonnay\", [\"Beringer Wine Founders Estate Chardonnay\"]\n end\n\n def test_percent\n store_names [\"1% Milk\", \"Whole Milk\"]\n assert_search \"1%\", [\"1% Milk\"]\n end\n\n # ascii\n\n def test_jalapenos\n store_names [\"Jalapeño\"]\n assert_search \"jalapeno\", [\"Jalapeño\"]\n end\n\n def test_swedish\n store_names [\"ÅÄÖ\"]\n assert_search \"aao\", [\"ÅÄÖ\"]\n end\n\n # stemming\n\n def test_stemming\n store_names [\"Whole Milk\", \"Fat Free Milk\", \"Milk\"]\n assert_search \"milks\", [\"Milk\", \"Whole Milk\", \"Fat Free Milk\"]\n assert_search \"milks\", [\"Milk\", \"Whole Milk\", \"Fat Free Milk\"], misspellings: false\n end\n\n def test_stemming_tokens\n assert_equal [\"milk\"], Product.searchkick_index.tokens(\"milks\", analyzer: \"searchkick_search\")\n assert_equal [\"milk\"], Product.searchkick_index.tokens(\"milks\", analyzer: \"searchkick_search2\")\n end\n\n # fuzzy\n\n def test_misspelling_sriracha\n store_names [\"Sriracha\"]\n assert_search \"siracha\", [\"Sriracha\"]\n end\n\n def test_misspelling_multiple\n store_names [\"Greek Yogurt\", \"Green Onions\"]\n assert_search \"greed\", [\"Greek Yogurt\", \"Green Onions\"]\n end\n\n def test_short_word\n store_names [\"Finn\"]\n assert_search \"fin\", [\"Finn\"]\n end\n\n def test_edit_distance_two\n store_names [\"Bingo\"]\n assert_search \"bin\", []\n assert_search \"bingooo\", []\n assert_search \"mango\", []\n end\n\n def test_edit_distance_one\n store_names [\"Bingo\"]\n assert_search \"bing\", [\"Bingo\"]\n assert_search \"bingoo\", [\"Bingo\"]\n assert_search \"ringo\", [\"Bingo\"]\n end\n\n def test_edit_distance_long_word\n store_names [\"thisisareallylongword\"]\n assert_search \"thisisareallylongwor\", [\"thisisareallylongword\"] # missing letter\n assert_search \"thisisareelylongword\", [] # edit distance = 2\n end\n\n def test_misspelling_tabasco\n store_names [\"Tabasco\"]\n assert_search \"tobasco\", [\"Tabasco\"]\n end\n\n def test_misspelling_zucchini\n store_names [\"Zucchini\"]\n assert_search \"zuchini\", [\"Zucchini\"]\n end\n\n def test_misspelling_ziploc\n store_names [\"Ziploc\"]\n assert_search \"zip lock\", [\"Ziploc\"]\n end\n\n def test_misspelling_zucchini_transposition\n store_names [\"zucchini\"]\n assert_search \"zuccihni\", [\"zucchini\"]\n\n # need to specify field\n # as transposition option isn't supported for multi_match queries\n # until Elasticsearch 6.1\n assert_search \"zuccihni\", [], misspellings: {transpositions: false}, fields: [:name]\n end\n\n def test_misspelling_lasagna\n store_names [\"lasagna\"]\n assert_search \"lasanga\", [\"lasagna\"], misspellings: {transpositions: true}\n assert_search \"lasgana\", [\"lasagna\"], misspellings: {transpositions: true}\n assert_search \"lasaang\", [], misspellings: {transpositions: true} # triple transposition, shouldn't work\n assert_search \"lsagana\", [], misspellings: {transpositions: true} # triple transposition, shouldn't work\n end\n\n def test_misspelling_lasagna_pasta\n store_names [\"lasagna pasta\"]\n assert_search \"lasanga\", [\"lasagna pasta\"], misspellings: {transpositions: true}\n assert_search \"lasanga pasta\", [\"lasagna pasta\"], misspellings: {transpositions: true}\n assert_search \"lasanga pasat\", [\"lasagna pasta\"], misspellings: {transpositions: true} # both words misspelled with a transposition should still work\n end\n\n def test_misspellings_word_start\n store_names [\"Sriracha\"]\n assert_search \"siracha\", [\"Sriracha\"], fields: [{name: :word_start}]\n end\n\n # spaces\n\n def test_spaces_in_field\n store_names [\"Red Bull\"]\n assert_search \"redbull\", [\"Red Bull\"], misspellings: false\n end\n\n def test_spaces_in_query\n store_names [\"Dishwasher\"]\n assert_search \"dish washer\", [\"Dishwasher\"], misspellings: false\n end\n\n def test_spaces_three_words\n store_names [\"Dish Washer Soap\", \"Dish Washer\"]\n assert_search \"dish washer soap\", [\"Dish Washer Soap\"]\n end\n\n def test_spaces_stemming\n store_names [\"Almond Milk\"]\n assert_search \"almondmilks\", [\"Almond Milk\"]\n end\n\n # other\n\n def test_all\n store_names [\"Product A\", \"Product B\"]\n assert_search \"*\", [\"Product A\", \"Product B\"]\n end\n\n def test_no_arguments\n store_names []\n assert_equal [], Product.search.to_a\n end\n\n def test_no_term\n store_names [\"Product A\"]\n assert_equal [\"Product A\"], Product.search(where: {name: \"Product A\"}).map(&:name)\n end\n\n def test_to_be_or_not_to_be\n store_names [\"to be or not to be\"]\n assert_search \"to be\", [\"to be or not to be\"]\n end\n\n def test_apostrophe\n store_names [\"Ben and Jerry's\"]\n assert_search \"ben and jerrys\", [\"Ben and Jerry's\"]\n end\n\n def test_apostrophe_search\n store_names [\"Ben and Jerrys\"]\n assert_search \"ben and jerry's\", [\"Ben and Jerrys\"]\n end\n\n def test_ampersand_index\n store_names [\"Ben & Jerry's\"]\n assert_search \"ben and jerrys\", [\"Ben & Jerry's\"]\n end\n\n def test_ampersand_search\n store_names [\"Ben and Jerry's\"]\n assert_search \"ben & jerrys\", [\"Ben and Jerry's\"]\n end\n\n def test_phrase\n store_names [\"Fresh Honey\", \"Honey Fresh\"]\n assert_search \"fresh honey\", [\"Fresh Honey\"], match: :phrase\n end\n\n def test_phrase_again\n store_names [\"Social entrepreneurs don't have it easy raising capital\"]\n assert_search \"social entrepreneurs don't have it easy raising capital\", [\"Social entrepreneurs don't have it easy raising capital\"], match: :phrase\n end\n\n def test_phrase_order\n store_names [\"Wheat Bread\", \"Whole Wheat Bread\"]\n assert_order \"wheat bread\", [\"Wheat Bread\", \"Whole Wheat Bread\"], match: :phrase, fields: [:name]\n end\n\n def test_dynamic_fields\n setup_speaker\n store_names [\"Red Bull\"], Speaker\n assert_search \"redbull\", [\"Red Bull\"], {fields: [:name]}, Speaker\n end\n\n def test_unsearchable\n skip\n store [\n {name: \"Unsearchable\", description: \"Almond\"}\n ]\n assert_search \"almond\", []\n end\n\n def test_unsearchable_where\n store [\n {name: \"Unsearchable\", description: \"Almond\"}\n ]\n assert_search \"*\", [\"Unsearchable\"], where: {description: \"Almond\"}\n end\n\n def test_emoji\n store_names [\"Banana\"]\n assert_search \"🍌\", [\"Banana\"], emoji: true\n end\n\n def test_emoji_multiple\n store_names [\"Ice Cream Cake\"]\n assert_search \"🍨🍰\", [\"Ice Cream Cake\"], emoji: true\n assert_search \"🍨🍰\", [\"Ice Cream Cake\"], emoji: true, misspellings: false\n end\n\n # operator\n\n def test_operator\n store_names [\"fresh\", \"honey\"]\n assert_search \"fresh honey\", [\"fresh\", \"honey\"], {operator: \"or\"}\n assert_search \"fresh honey\", [], {operator: \"and\"}\n assert_search \"fresh honey\", [\"fresh\", \"honey\"], {operator: :or}\n assert_search \"fresh honey\", [\"fresh\", \"honey\"], {operator: :or, body_options: {track_total_hits: true}}\n assert_search \"fresh honey\", [], {operator: :or, fields: [:name], match: :phrase, body_options: {track_total_hits: true}}\n end\n\n def test_operator_scoring\n store_names [\"Big Red Circle\", \"Big Green Circle\", \"Small Orange Circle\"]\n assert_order \"big red circle\", [\"Big Red Circle\", \"Big Green Circle\", \"Small Orange Circle\"], operator: \"or\"\n end\n\n # fields\n\n def test_fields_operator\n store [\n {name: \"red\", color: \"red\"},\n {name: \"blue\", color: \"blue\"},\n {name: \"cyan\", color: \"blue green\"},\n {name: \"magenta\", color: \"red blue\"},\n {name: \"green\", color: \"green\"}\n ]\n assert_search \"red blue\", [\"red\", \"blue\", \"cyan\", \"magenta\"], operator: \"or\", fields: [\"color\"]\n end\n\n def test_fields\n store [\n {name: \"red\", color: \"light blue\"},\n {name: \"blue\", color: \"red fish\"}\n ]\n assert_search \"blue\", [\"red\"], fields: [\"color\"]\n end\n\n def test_non_existent_field\n store_names [\"Milk\"]\n assert_search \"milk\", [], fields: [\"not_here\"]\n end\n\n def test_fields_both_match\n # have same score due to dismax\n store [\n {name: \"Blue A\", color: \"red\"},\n {name: \"Blue B\", color: \"light blue\"}\n ]\n assert_first \"blue\", \"Blue B\", fields: [:name, :color]\n end\nend\n"
},
{
"path": "test/misspellings_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MisspellingsTest < Minitest::Test\n def test_false\n store_names [\"abc\", \"abd\", \"aee\"]\n assert_search \"abc\", [\"abc\"], misspellings: false\n end\n\n def test_distance\n store_names [\"abbb\", \"aabb\"]\n assert_search \"aaaa\", [\"aabb\"], misspellings: {distance: 2}\n end\n\n def test_prefix_length\n store_names [\"ap\", \"api\", \"apt\", \"any\", \"nap\", \"ah\", \"ahi\"]\n assert_search \"ap\", [\"ap\", \"api\", \"apt\"], misspellings: {prefix_length: 2}\n assert_search \"api\", [\"ap\", \"api\", \"apt\"], misspellings: {prefix_length: 2}\n end\n\n def test_prefix_length_operator\n store_names [\"ap\", \"api\", \"apt\", \"any\", \"nap\", \"ah\", \"aha\"]\n assert_search \"ap ah\", [\"ap\", \"ah\", \"api\", \"apt\", \"aha\"], operator: \"or\", misspellings: {prefix_length: 2}\n assert_search \"api ahi\", [\"ap\", \"api\", \"apt\", \"ah\", \"aha\"], operator: \"or\", misspellings: {prefix_length: 2}\n end\n\n def test_fields_operator\n store [\n {name: \"red\", color: \"red\"},\n {name: \"blue\", color: \"blue\"},\n {name: \"cyan\", color: \"blue green\"},\n {name: \"magenta\", color: \"red blue\"},\n {name: \"green\", color: \"green\"}\n ]\n assert_search \"red blue\", [\"red\", \"blue\", \"cyan\", \"magenta\"], operator: \"or\", fields: [\"color\"], misspellings: false\n end\n\n def test_below_unmet\n store_names [\"abc\", \"abd\", \"aee\"]\n assert_search \"abc\", [\"abc\", \"abd\"], misspellings: {below: 2}\n end\n\n def test_below_unmet_result\n store_names [\"abc\", \"abd\", \"aee\"]\n assert Product.search(\"abc\", misspellings: {below: 2}).misspellings?\n end\n\n def test_below_met\n store_names [\"abc\", \"abd\", \"aee\"]\n assert_search \"abc\", [\"abc\"], misspellings: {below: 1}\n end\n\n def test_below_met_result\n store_names [\"abc\", \"abd\", \"aee\"]\n assert !Product.search(\"abc\", misspellings: {below: 1}).misspellings?\n end\n\n def test_field_correct_spelling_still_works\n store [{name: \"Sriracha\", color: \"blue\"}]\n assert_misspellings \"Sriracha\", [\"Sriracha\"], {fields: [:name, :color]}\n assert_misspellings \"blue\", [\"Sriracha\"], {fields: [:name, :color]}\n end\n\n def test_field_enabled\n store [{name: \"Sriracha\", color: \"blue\"}]\n assert_misspellings \"siracha\", [\"Sriracha\"], {fields: [:name]}\n assert_misspellings \"clue\", [\"Sriracha\"], {fields: [:color]}\n end\n\n def test_field_disabled\n store [{name: \"Sriracha\", color: \"blue\"}]\n assert_misspellings \"siracha\", [], {fields: [:color]}\n assert_misspellings \"clue\", [], {fields: [:name]}\n end\n\n def test_field_with_transpositions\n store [{name: \"Sriracha\", color: \"blue\"}]\n assert_misspellings \"lbue\", [], {transpositions: false, fields: [:color]}\n end\n\n def test_field_with_edit_distance\n store [{name: \"Sriracha\", color: \"blue\"}]\n assert_misspellings \"crue\", [\"Sriracha\"], {edit_distance: 2, fields: [:color]}\n end\n\n def test_field_multiple\n store [\n {name: \"Greek Yogurt\", color: \"white\"},\n {name: \"Green Onions\", color: \"yellow\"}\n ]\n assert_misspellings \"greed\", [\"Greek Yogurt\", \"Green Onions\"], {fields: [:name, :color]}\n assert_misspellings \"mellow\", [\"Green Onions\"], {fields: [:name, :color]}\n end\n\n def test_field_requires_explicit_search_fields\n store_names [\"Sriracha\"]\n assert_raises(ArgumentError) do\n assert_search \"siracha\", [\"Sriracha\"], {misspellings: {fields: [:name]}}\n end\n end\n\n def test_field_word_start\n store_names [\"Sriracha\"]\n assert_search \"siracha\", [\"Sriracha\"], fields: [{name: :word_middle}], misspellings: {fields: [:name]}\n end\n\n private\n\n def assert_misspellings(term, expected, misspellings = {}, model = default_model)\n options = {\n fields: [:name, :color],\n misspellings: misspellings\n }\n assert_search(term, expected, options, model)\n end\nend\n"
},
{
"path": "test/models/animal.rb",
"content": "class Animal\n searchkick \\\n inheritance: true,\n text_start: [:name],\n suggest: [:name]\nend\n"
},
{
"path": "test/models/artist.rb",
"content": "class Artist\n searchkick unscope: true\n\n def should_index?\n should_index\n end\nend\n"
},
{
"path": "test/models/band.rb",
"content": "class Band\n searchkick\nend\n"
},
{
"path": "test/models/product.rb",
"content": "class Product\n searchkick \\\n synonyms: [\n [\"clorox\", \"bleach\"],\n [\"burger\", \"hamburger\"],\n [\"bandaid\", \"bandages\"],\n [\"UPPERCASE\", \"lowercase\"],\n \"lightbulb => led,lightbulb\",\n \"lightbulb => halogenlamp\"\n ],\n suggest: [:name, :color],\n conversions_v1: [:conversions],\n conversions_v2: [:conversions_v2],\n locations: [:location, :multiple_locations],\n text_start: [:name],\n text_middle: [:name],\n text_end: [:name],\n word_start: [:name],\n word_middle: [:name],\n word_end: [:name],\n highlight: [:name],\n filterable: [:name, :color, :description],\n similarity: \"BM25\",\n match: ENV[\"MATCH\"] ? ENV[\"MATCH\"].to_sym : nil,\n knn: Searchkick.knn_support? ? {\n embedding: {dimensions: 3, distance: \"cosine\", m: 16, ef_construction: 100},\n embedding2: {dimensions: 3, distance: \"inner_product\"},\n embedding3: {dimensions: 3, distance: \"euclidean\"}\n }.merge(Searchkick.opensearch? ? {} : {embedding4: {dimensions: 3}}) : nil\n\n attr_accessor :conversions, :conversions_v2, :user_ids, :aisle, :details\n\n class << self\n attr_accessor :dynamic_data\n end\n\n def search_data\n return self.class.dynamic_data.call if self.class.dynamic_data\n\n serializable_hash.except(\"id\", \"_id\").merge(\n conversions: conversions,\n conversions_v2: conversions_v2,\n user_ids: user_ids,\n location: {lat: latitude, lon: longitude},\n multiple_locations: [{lat: latitude, lon: longitude}, {lat: 0, lon: 0}],\n aisle: aisle,\n details: details\n )\n end\n\n def should_index?\n name != \"DO NOT INDEX\"\n end\n\n def search_name\n {\n name: name\n }\n end\nend\n"
},
{
"path": "test/models/region.rb",
"content": "class Region\n searchkick \\\n geo_shape: [:territory]\n\n attr_accessor :territory\n\n def search_data\n {\n name: name,\n text: text,\n territory: territory\n }\n end\nend\n"
},
{
"path": "test/models/sku.rb",
"content": "class Sku\n searchkick callbacks: :async\nend\n"
},
{
"path": "test/models/song.rb",
"content": "class Song\n searchkick\n\n def search_routing\n name\n end\nend\n"
},
{
"path": "test/models/speaker.rb",
"content": "class Speaker\n searchkick \\\n conversions_v1: [\"conversions_a\", \"conversions_b\"],\n search_synonyms: [\n [\"clorox\", \"bleach\"],\n [\"burger\", \"hamburger\"],\n [\"bandaids\", \"bandages\"],\n [\"UPPERCASE\", \"lowercase\"],\n \"led => led,lightbulb\",\n \"halogen lamp => lightbulb\",\n [\"United States of America\", \"USA\"]\n ],\n word_start: [:name]\n\n attr_accessor :conversions_a, :conversions_b, :aisle\n\n def search_data\n serializable_hash.except(\"id\", \"_id\").merge(\n conversions_a: conversions_a,\n conversions_b: conversions_b,\n aisle: aisle\n )\n end\nend\n"
},
{
"path": "test/models/store.rb",
"content": "class Store\n mappings = {\n properties: {\n name: {type: \"text\"}\n }\n }\n\n searchkick \\\n routing: true,\n merge_mappings: true,\n mappings: mappings\n\n def search_document_id\n id\n end\n\n def search_routing\n name\n end\nend\n"
},
{
"path": "test/multi_indices_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MultiIndicesTest < Minitest::Test\n def setup\n super\n setup_speaker\n end\n\n def test_basic\n store_names [\"Product A\"]\n store_names [\"Product B\"], Speaker\n assert_search_multi \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_index_name\n store_names [\"Product A\"]\n assert_equal [\"Product A\"], Product.search(\"product\", index_name: Product.searchkick_index.name).map(&:name)\n assert_equal [\"Product A\"], Product.search(\"product\", index_name: Product).map(&:name)\n\n Speaker.searchkick_index.refresh\n assert_equal [], Product.search(\"product\", index_name: Speaker.searchkick_index.name, conversions: false).map(&:name)\n end\n\n def test_models_and_index_name\n store_names [\"Product A\"]\n store_names [\"Product B\"], Speaker\n assert_equal [\"Product A\"], Searchkick.search(\"product\", models: [Product, Store], index_name: Product.searchkick_index.name).map(&:name)\n error = assert_raises(Searchkick::Error) do\n Searchkick.search(\"product\", models: [Product, Store], index_name: Speaker.searchkick_index.name).map(&:name)\n end\n assert_includes error.message, \"Unknown model\"\n # legacy\n assert_equal [\"Product A\"], Searchkick.search(\"product\", index_name: [Product, Store]).map(&:name)\n end\n\n def test_model_with_another_model\n error = assert_raises(ArgumentError) do\n Product.search(models: [Store])\n end\n assert_includes error.message, \"Use Searchkick.search\"\n end\n\n def test_model_with_another_model_in_index_name\n error = assert_raises(ArgumentError) do\n # legacy protection\n Product.search(index_name: [Store, \"another\"])\n end\n assert_includes error.message, \"Use Searchkick.search\"\n end\n\n def test_no_models_or_index_name\n store_names [\"Product A\"]\n\n error = assert_raises(Searchkick::Error) do\n Searchkick.search(\"product\").to_a\n end\n assert_includes error.message, \"Unknown model\"\n end\n\n def test_no_models_or_index_name_load_false\n store_names [\"Product A\"]\n Searchkick.search(\"product\", load: false).to_a\n end\n\n private\n\n def assert_search_multi(term, expected, options = {})\n options[:models] = [Product, Speaker]\n options[:fields] = [:name]\n assert_search(term, expected, options, Searchkick)\n end\nend\n"
},
{
"path": "test/multi_search_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MultiSearchTest < Minitest::Test\n def test_basic\n store_names [\"Product A\"]\n store_names [\"Store A\"], Store\n products = Product.search(\"*\")\n stores = Store.search(\"*\")\n Searchkick.multi_search([products, stores])\n assert_equal [\"Product A\"], products.map(&:name)\n assert_equal [\"Store A\"], stores.map(&:name)\n end\n\n def test_methods\n result = Product.search(\"*\")\n query = Product.search(\"*\")\n assert_empty(result.methods - query.methods)\n end\n\n def test_error\n store_names [\"Product A\"]\n products = Product.search(\"*\")\n stores = Store.search(\"*\", order: [:bad_field])\n Searchkick.multi_search([products, stores])\n assert !products.error\n assert stores.error\n end\n\n def test_misspellings_below_unmet\n store_names [\"abc\", \"abd\", \"aee\"]\n products = Product.search(\"abc\", misspellings: {below: 5})\n Searchkick.multi_search([products])\n assert_equal [\"abc\", \"abd\"], products.map(&:name)\n end\n\n def test_misspellings_below_error\n products = Product.search(\"abc\", order: [:bad_field], misspellings: {below: 1})\n Searchkick.multi_search([products])\n assert products.error\n end\n\n def test_query_error\n products = Product.search(\"*\", order: {bad_field: :asc})\n Searchkick.multi_search([products])\n assert products.error\n error = assert_raises(Searchkick::Error) { products.to_a }\n assert_equal error.message, \"Query error - use the error method to view it\"\n end\nend\n"
},
{
"path": "test/multi_tenancy_test.rb",
"content": "require_relative \"test_helper\"\n\nclass MultiTenancyTest < Minitest::Test\n def setup\n skip unless defined?(Apartment)\n end\n\n def test_basic\n Apartment::Tenant.switch!(\"tenant1\")\n store_names [\"Product A\"]\n Apartment::Tenant.switch!(\"tenant2\")\n store_names [\"Product B\"]\n Apartment::Tenant.switch!(\"tenant1\")\n assert_search \"product\", [\"Product A\"], {load: false}\n Apartment::Tenant.switch!(\"tenant2\")\n assert_search \"product\", [\"Product B\"], {load: false}\n end\n\n def teardown\n Apartment::Tenant.reset if defined?(Apartment)\n end\n\n def default_model\n Tenant\n end\nend\n"
},
{
"path": "test/notifications_test.rb",
"content": "require_relative \"test_helper\"\n\nclass NotificationsTest < Minitest::Test\n def test_search\n Product.searchkick_index.refresh\n\n notifications = capture_notifications do\n Product.search(\"product\").to_a\n end\n\n assert_equal 1, notifications.size\n assert_equal \"search.searchkick\", notifications.last[:name]\n end\n\n private\n\n def capture_notifications\n notifications = []\n callback = lambda do |name, started, finished, unique_id, payload|\n notifications << {name: name, payload: payload}\n end\n ActiveSupport::Notifications.subscribed(callback, /searchkick/) do\n yield\n end\n notifications\n end\nend\n"
},
{
"path": "test/order_test.rb",
"content": "require_relative \"test_helper\"\n\nclass OrderTest < Minitest::Test\n def test_hash\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\"]\n assert_order \"product\", [\"Product D\", \"Product C\", \"Product B\", \"Product A\"], order: {name: :desc}\n assert_order_relation [\"Product D\", \"Product C\", \"Product B\", \"Product A\"], Product.search(\"product\").order(name: :desc)\n end\n\n def test_string\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\"]\n assert_order \"product\", [\"Product A\", \"Product B\", \"Product C\", \"Product D\"], order: \"name\"\n assert_order_relation [\"Product A\", \"Product B\", \"Product C\", \"Product D\"], Product.search(\"product\").order(\"name\")\n end\n\n def test_multiple\n store [\n {name: \"Product A\", color: \"blue\", store_id: 1},\n {name: \"Product B\", color: \"red\", store_id: 3},\n {name: \"Product C\", color: \"red\", store_id: 2}\n ]\n assert_order \"product\", [\"Product A\", \"Product B\", \"Product C\"], order: {color: :asc, store_id: :desc}\n assert_order_relation [\"Product A\", \"Product B\", \"Product C\"], Product.search(\"product\").order(color: :asc, store_id: :desc)\n assert_order_relation [\"Product A\", \"Product B\", \"Product C\"], Product.search(\"product\").order(:color, store_id: :desc)\n assert_order_relation [\"Product A\", \"Product B\", \"Product C\"], Product.search(\"product\").order(color: :asc).order(store_id: :desc)\n assert_order_relation [\"Product B\", \"Product C\", \"Product A\"], Product.search(\"product\").order(color: :asc).reorder(store_id: :desc)\n end\n\n def test_unmapped_type\n Product.searchkick_index.refresh\n assert_order \"product\", [], order: {not_mapped: {unmapped_type: \"long\"}}\n assert_order_relation [], Product.search(\"product\").order(not_mapped: {unmapped_type: \"long\"})\n end\n\n def test_array\n store [{name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167}]\n assert_order \"francisco\", [\"San Francisco\"], order: [{_geo_distance: {location: \"0,0\"}}]\n assert_order_relation [\"San Francisco\"], Product.search(\"francisco\").order([{_geo_distance: {location: \"0,0\"}}])\n end\n\n def test_script\n store_names [\"Red\", \"Green\", \"Blue\"]\n order = {_script: {type: \"number\", script: {source: \"doc['name'].value.length() * -1\"}}}\n assert_order \"*\", [\"Green\", \"Blue\", \"Red\"], order: order\n assert_order_relation [\"Green\", \"Blue\", \"Red\"], Product.search(\"*\").order(order)\n end\nend\n"
},
{
"path": "test/pagination_test.rb",
"content": "require_relative \"test_helper\"\n\nclass PaginationTest < Minitest::Test\n def test_limit\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\"]\n assert_order \"product\", [\"Product A\", \"Product B\"], order: {name: :asc}, limit: 2\n assert_order_relation [\"Product A\", \"Product B\"], Product.search(\"product\").order(name: :asc).limit(2)\n end\n\n def test_no_limit\n names = 20.times.map { |i| \"Product #{i}\" }\n store_names names\n assert_search \"product\", names\n end\n\n def test_offset\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\"]\n assert_order \"product\", [\"Product C\", \"Product D\"], order: {name: :asc}, offset: 2, limit: 100\n assert_order_relation [\"Product C\", \"Product D\"], Product.search(\"product\").order(name: :asc).offset(2).limit(100)\n end\n\n def test_pagination\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\", \"Product E\", \"Product F\"]\n products = Product.search(\"product\", order: {name: :asc}, page: 2, per_page: 2, padding: 1)\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n assert_equal \"product\", products.entry_name\n assert_equal 2, products.current_page\n assert_equal 1, products.padding\n assert_equal 2, products.per_page\n assert_equal 2, products.size\n assert_equal 2, products.length\n assert_equal 3, products.total_pages\n assert_equal 6, products.total_count\n assert_equal 6, products.total_entries\n assert_equal 2, products.limit_value\n assert_equal 3, products.offset_value\n assert_equal 3, products.offset\n assert_equal 3, products.next_page\n assert_equal 1, products.previous_page\n assert_equal 1, products.prev_page\n assert !products.first_page?\n assert !products.last_page?\n assert !products.empty?\n assert !products.out_of_range?\n assert products.any?\n end\n\n def test_relation\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\", \"Product E\", \"Product F\"]\n products = Product.search(\"product\", padding: 1).order(name: :asc).page(2).per_page(2)\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n assert_equal \"product\", products.entry_name\n assert_equal 2, products.current_page\n assert_equal 1, products.padding\n assert_equal 2, products.per_page\n assert_equal 2, products.size\n assert_equal 2, products.length\n assert_equal 3, products.total_pages\n assert_equal 6, products.total_count\n assert_equal 6, products.total_entries\n assert_equal 2, products.limit_value\n assert_equal 3, products.offset_value\n assert_equal 3, products.offset\n assert_equal 3, products.next_page\n assert_equal 1, products.previous_page\n assert_equal 1, products.prev_page\n assert !products.first_page?\n assert !products.last_page?\n assert !products.empty?\n assert !products.out_of_range?\n assert products.any?\n end\n\n def test_per\n store_names [\"Product A\", \"Product B\", \"Product C\"]\n assert_order_relation [\"Product A\", \"Product B\"], Product.search(\"product\").order(name: :asc).per(2)\n end\n\n def test_body\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\", \"Product E\", \"Product F\"]\n products = Product.search(\"product\", body: {query: {match_all: {}}, sort: [{name: \"asc\"}]}, page: 2, per_page: 2, padding: 1)\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n assert_equal \"product\", products.entry_name\n assert_equal 2, products.current_page\n assert_equal 1, products.padding\n assert_equal 2, products.per_page\n assert_equal 2, products.size\n assert_equal 2, products.length\n assert_equal 3, products.total_pages\n assert_equal 6, products.total_count\n assert_equal 6, products.total_entries\n assert_equal 2, products.limit_value\n assert_equal 3, products.offset_value\n assert_equal 3, products.offset\n assert_equal 3, products.next_page\n assert_equal 1, products.previous_page\n assert_equal 1, products.prev_page\n assert !products.first_page?\n assert !products.last_page?\n assert !products.empty?\n assert !products.out_of_range?\n assert products.any?\n end\n\n def test_nil_page\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\", \"Product E\"]\n products = Product.search(\"product\", order: {name: :asc}, page: nil, per_page: 2)\n assert_equal [\"Product A\", \"Product B\"], products.map(&:name)\n assert_equal 1, products.current_page\n assert products.first_page?\n end\n\n def test_strings\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\", \"Product E\", \"Product F\"]\n\n products = Product.search(\"product\", order: {name: :asc}, page: \"2\", per_page: \"2\", padding: \"1\")\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n\n products = Product.search(\"product\", order: {name: :asc}, limit: \"2\", offset: \"3\")\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n end\n\n def test_total_entries\n products = Product.search(\"product\", total_entries: 4)\n assert_equal 4, products.total_entries\n end\n\n def test_kaminari\n require \"action_view\"\n\n I18n.load_path = Dir[\"test/support/kaminari.yml\"]\n I18n.backend.load_translations\n\n view = ActionView::Base.new(ActionView::LookupContext.new([]), [], nil)\n\n store_names [\"Product A\"]\n assert_equal \"Displaying 1 product\", view.page_entries_info(Product.search(\"product\"))\n\n store_names [\"Product B\"]\n assert_equal \"Displaying all 2 products\", view.page_entries_info(Product.search(\"product\"))\n\n store_names [\"Product C\"]\n assert_equal \"Displaying products 1 - 2 of 3 in total\", view.page_entries_info(Product.search(\"product\").per_page(2))\n end\n\n def test_deep_paging\n with_options({deep_paging: true}, Song) do\n assert_empty Song.search(\"*\", offset: 10000, limit: 1).to_a\n end\n end\n\n def test_no_deep_paging\n Song.reindex\n error = assert_raises(Searchkick::InvalidQueryError) do\n Song.search(\"*\", offset: 10000, limit: 1).to_a\n end\n assert_match \"Result window is too large\", error.message\n end\n\n def test_max_result_window\n Song.delete_all\n with_options({max_result_window: 10000}, Song) do\n relation = Song.search(\"*\", offset: 10000, limit: 1)\n assert_empty relation.to_a\n assert_equal 1, relation.per_page\n assert_equal 0, relation.total_pages\n end\n end\n\n def test_search_after\n store_names [\"Product A\", \"Product B\", \"Product C\", \"Product D\"]\n # ensure different created_at\n store_names [\"Product B\"]\n\n options = {order: {name: :asc, created_at: :asc}, per_page: 2}\n\n products = Product.search(\"product\", **options)\n assert_equal [\"Product A\", \"Product B\"], products.map(&:name)\n\n search_after = products.hits.last[\"sort\"]\n products = Product.search(\"product\", body_options: {search_after: search_after}, **options)\n assert_equal [\"Product B\", \"Product C\"], products.map(&:name)\n\n search_after = products.hits.last[\"sort\"]\n products = Product.search(\"product\", body_options: {search_after: search_after}, **options)\n assert_equal [\"Product D\"], products.map(&:name)\n end\n\n def test_pit\n skip unless pit_supported?\n\n store_names [\"Product A\", \"Product B\", \"Product D\", \"Product E\", \"Product G\"]\n\n pit_id =\n if Searchkick.opensearch?\n path = \"#{CGI.escape(Product.searchkick_index.name)}/_search/point_in_time\"\n Searchkick.client.transport.perform_request(\"POST\", path, {keep_alive: \"5s\"}).body[\"pit_id\"]\n else\n Searchkick.client.open_point_in_time(index: Product.searchkick_index.name, keep_alive: \"5s\")[\"id\"]\n end\n\n store_names [\"Product C\", \"Product F\"]\n\n options = {\n order: {name: :asc},\n per_page: 2,\n body_options: {pit: {id: pit_id}},\n index_name: \"\"\n }\n\n products = Product.search(\"product\", **options)\n assert_equal [\"Product A\", \"Product B\"], products.map(&:name)\n\n products = Product.search(\"product\", page: 2, **options)\n assert_equal [\"Product D\", \"Product E\"], products.map(&:name)\n\n products = Product.search(\"product\", page: 3, **options)\n assert_equal [\"Product G\"], products.map(&:name)\n\n products = Product.search(\"product\", page: 4, **options)\n assert_empty products.map(&:name)\n\n if Searchkick.opensearch?\n Searchkick.client.transport.perform_request(\"DELETE\", \"_search/point_in_time\", {}, {pit_id: pit_id})\n else\n Searchkick.client.close_point_in_time(body: {id: pit_id})\n end\n\n error = assert_raises do\n Product.search(\"product\", **options).load\n end\n assert_match \"No search context found for id\", error.message\n end\n\n private\n\n def pit_supported?\n Searchkick.opensearch? ? !Searchkick.server_below?(\"2.4.0\") : true\n end\nend\n"
},
{
"path": "test/parameters_test.rb",
"content": "require_relative \"test_helper\"\n\nclass ParametersTest < Minitest::Test\n def setup\n require \"action_controller\"\n super\n end\n\n def test_options\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\", **params)\n end\n end\n\n def test_where\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\", where: params)\n end\n end\n\n def test_where_relation\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\").where(params)\n end\n end\n\n def test_rewhere_relation\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\").where(params)\n end\n end\n\n def test_where_permitted\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search \"product\", [\"Product A\"], where: params.permit(:store_id)\n end\n\n def test_where_permitted_relation\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search_relation [\"Product A\"], Product.search(\"product\").where(params.permit(:store_id))\n end\n\n def test_rewhere_permitted_relation\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search_relation [\"Product A\"], Product.search(\"product\").rewhere(params.permit(:store_id))\n end\n\n def test_where_value\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search \"product\", [\"Product A\"], where: {store_id: params[:store_id]}\n end\n\n def test_where_value_relation\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search_relation [\"Product A\"], Product.search(\"product\").where(store_id: params[:store_id])\n end\n\n def test_rewhere_value_relation\n store [{name: \"Product A\", store_id: 1}, {name: \"Product B\", store_id: 2}]\n params = ActionController::Parameters.new({store_id: 1})\n assert_search_relation [\"Product A\"], Product.search(\"product\").where(store_id: params[:store_id])\n end\n\n def test_where_hash\n params = ActionController::Parameters.new({store_id: {value: 10, boost: 2}})\n error = assert_raises(TypeError) do\n assert_search \"product\", [], where: {store_id: params[:store_id]}\n end\n assert_equal error.message, \"can't cast ActionController::Parameters\"\n end\n\n # TODO raise error without to_a\n def test_where_hash_relation\n params = ActionController::Parameters.new({store_id: {value: 10, boost: 2}})\n error = assert_raises(TypeError) do\n Product.search(\"product\").where(store_id: params[:store_id]).to_a\n end\n assert_equal error.message, \"can't cast ActionController::Parameters\"\n end\n\n # TODO raise error without to_a\n def test_rewhere_hash_relation\n params = ActionController::Parameters.new({store_id: {value: 10, boost: 2}})\n error = assert_raises(TypeError) do\n Product.search(\"product\").rewhere(store_id: params[:store_id]).to_a\n end\n assert_equal error.message, \"can't cast ActionController::Parameters\"\n end\n\n def test_aggs_where\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\", aggs: {size: {where: params}})\n end\n end\n\n def test_aggs_where_smart_aggs_false\n params = ActionController::Parameters.new({store_id: 1})\n assert_raises(ActionController::UnfilteredParameters) do\n Product.search(\"*\", aggs: {size: {where: params}}, smart_aggs: false)\n end\n end\nend\n"
},
{
"path": "test/partial_match_test.rb",
"content": "require_relative \"test_helper\"\n\nclass PartialMatchTest < Minitest::Test\n def test_autocomplete\n store_names [\"Hummus\"]\n assert_search \"hum\", [\"Hummus\"], match: :text_start\n end\n\n def test_autocomplete_two_words\n store_names [\"Organic Hummus\"]\n assert_search \"hum\", [], match: :text_start\n end\n\n def test_autocomplete_fields\n store_names [\"Hummus\"]\n assert_search \"hum\", [\"Hummus\"], match: :text_start, fields: [:name]\n end\n\n def test_text_start\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"where in the world is\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :text_start}]\n assert_search \"in the world\", [], fields: [{name: :text_start}]\n end\n\n def test_text_middle\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"where in the world is\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :text_middle}]\n assert_search \"n the wor\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :text_middle}]\n assert_search \"men san diego\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :text_middle}]\n assert_search \"world carmen\", [], fields: [{name: :text_middle}]\n end\n\n def test_text_end\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"men san diego\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :text_end}]\n assert_search \"carmen san\", [], fields: [{name: :text_end}]\n end\n\n def test_word_start\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"car san wor\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :word_start}]\n end\n\n def test_word_middle\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"orl\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :word_middle}]\n end\n\n def test_word_end\n store_names [\"Where in the World is Carmen San Diego\"]\n assert_search \"rld men ego\", [\"Where in the World is Carmen San Diego\"], fields: [{name: :word_end}]\n end\n\n def test_word_start_multiple_words\n store_names [\"Dark Grey\", \"Dark Blue\"]\n assert_search \"dark grey\", [\"Dark Grey\"], fields: [{name: :word_start}]\n end\n\n def test_word_start_exact\n store_names [\"Back Scratcher\", \"Backpack\"]\n assert_order \"back\", [\"Back Scratcher\", \"Backpack\"], fields: [{name: :word_start}]\n end\n\n def test_word_start_exact_martin\n store_names [\"Martina\", \"Martin\"]\n assert_order \"martin\", [\"Martin\", \"Martina\"], fields: [{name: :word_start}]\n end\n\n # TODO find a better place\n\n def test_exact\n store_names [\"hi@example.org\"]\n assert_search \"hi@example.org\", [\"hi@example.org\"], fields: [{name: :exact}]\n end\n\n def test_exact_case\n store_names [\"Hello\"]\n assert_search \"hello\", [], fields: [{name: :exact}]\n assert_search \"Hello\", [\"Hello\"], fields: [{name: :exact}]\n end\nend\n"
},
{
"path": "test/partial_reindex_test.rb",
"content": "require_relative \"test_helper\"\n\nclass PartialReindexTest < Minitest::Test\n def test_record_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Searchkick.callbacks(false) do\n product.update!(name: \"Bye\", color: \"Red\")\n end\n\n product.reindex(:search_name, refresh: true)\n\n # name updated, but not color\n assert_search \"bye\", [\"Bye\"], fields: [:name], load: false\n assert_search \"blue\", [\"Bye\"], fields: [:color], load: false\n end\n\n def test_record_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Searchkick.callbacks(false) do\n product.update!(name: \"Bye\", color: \"Red\")\n end\n\n perform_enqueued_jobs do\n product.reindex(:search_name, mode: :async)\n end\n Product.searchkick_index.refresh\n\n # name updated, but not color\n assert_search \"bye\", [\"Bye\"], fields: [:name], load: false\n assert_search \"blue\", [\"Bye\"], fields: [:color], load: false\n end\n\n def test_record_queue\n product = Product.create!(name: \"Hi\")\n error = assert_raises(Searchkick::Error) do\n product.reindex(:search_name, mode: :queue)\n end\n assert_equal \"Partial reindex not supported with queue option\", error.message\n end\n\n def test_record_missing_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n error = assert_raises(Searchkick::ImportError) do\n product.reindex(:search_name)\n end\n assert_match \"document missing\", error.message\n end\n\n def test_record_ignore_missing_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n product.reindex(:search_name, ignore_missing: true)\n Searchkick.callbacks(:bulk) do\n product.reindex(:search_name, ignore_missing: true)\n end\n end\n\n def test_record_missing_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n perform_enqueued_jobs do\n error = assert_raises(Searchkick::ImportError) do\n product.reindex(:search_name, mode: :async)\n end\n assert_match \"document missing\", error.message\n end\n end\n\n def test_record_ignore_missing_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n perform_enqueued_jobs do\n product.reindex(:search_name, mode: :async, ignore_missing: true)\n end\n end\n\n def test_relation_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Searchkick.callbacks(false) do\n product.update!(name: \"Bye\", color: \"Red\")\n end\n\n Product.reindex(:search_name)\n\n # name updated, but not color\n assert_search \"bye\", [\"Bye\"], fields: [:name], load: false\n assert_search \"blue\", [\"Bye\"], fields: [:color], load: false\n\n # scope\n Product.reindex(:search_name, scope: :all)\n end\n\n def test_relation_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Searchkick.callbacks(false) do\n product.update!(name: \"Bye\", color: \"Red\")\n end\n\n perform_enqueued_jobs do\n Product.reindex(:search_name, mode: :async)\n end\n\n # name updated, but not color\n assert_search \"bye\", [\"Bye\"], fields: [:name], load: false\n assert_search \"blue\", [\"Bye\"], fields: [:color], load: false\n end\n\n def test_relation_queue\n Product.create!(name: \"Hi\")\n error = assert_raises(Searchkick::Error) do\n Product.reindex(:search_name, mode: :queue)\n end\n assert_equal \"Partial reindex not supported with queue option\", error.message\n end\n\n def test_relation_missing_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n error = assert_raises(Searchkick::ImportError) do\n Product.reindex(:search_name)\n end\n assert_match \"document missing\", error.message\n end\n\n def test_relation_ignore_missing_inline\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n Product.where(id: product.id).reindex(:search_name, ignore_missing: true)\n end\n\n def test_relation_missing_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n perform_enqueued_jobs do\n error = assert_raises(Searchkick::ImportError) do\n Product.reindex(:search_name, mode: :async)\n end\n assert_match \"document missing\", error.message\n end\n end\n\n def test_relation_ignore_missing_async\n store [{name: \"Hi\", color: \"Blue\"}]\n\n product = Product.first\n Product.searchkick_index.remove(product)\n\n perform_enqueued_jobs do\n Product.where(id: product.id).reindex(:search_name, mode: :async, ignore_missing: true)\n end\n end\nend\n"
},
{
"path": "test/query_test.rb",
"content": "require_relative \"test_helper\"\n\nclass QueryTest < Minitest::Test\n def test_basic\n store_names [\"Milk\", \"Apple\"]\n query = Product.search(\"milk\", body: {query: {match_all: {}}})\n assert_equal [\"Apple\", \"Milk\"], query.map(&:name).sort\n end\n\n def test_with_uneffective_min_score\n store_names [\"Milk\", \"Milk2\"]\n assert_search \"milk\", [\"Milk\", \"Milk2\"], body_options: {min_score: 0.0001}\n end\n\n def test_default_timeout\n assert_equal \"6000ms\", Product.search(\"*\").body[:timeout]\n end\n\n def test_timeout_override\n assert_equal \"1s\", Product.search(\"*\", body_options: {timeout: \"1s\"}).body[:timeout]\n end\n\n def test_request_params\n assert_equal \"dfs_query_then_fetch\", Product.search(\"*\", request_params: {search_type: \"dfs_query_then_fetch\"}).params[:search_type]\n end\n\n def test_opaque_id\n store_names [\"Milk\"]\n set_search_slow_log(0)\n Product.search(\"*\", opaque_id: \"search\").load\n Product.search(\"*\").opaque_id(\"search_relation\").load\n Product.search(\"*\", scroll: \"5s\", opaque_id: \"scroll\").scroll { }\n Searchkick.multi_search([Product.search(\"*\")], opaque_id: \"multi_search\")\n ensure\n set_search_slow_log(-1)\n end\n\n def test_debug\n store_names [\"Milk\"]\n out, _ = capture_io do\n assert_search \"milk\", [\"Milk\"], debug: true\n end\n refute_includes out, \"Error\"\n end\n\n def test_big_decimal\n store [\n {name: \"Product\", latitude: 80.0}\n ]\n assert_search \"product\", [\"Product\"], where: {latitude: {gt: 79}}\n end\n\n # body_options\n\n def test_body_options_should_merge_into_body\n query = Product.search(\"*\", body_options: {min_score: 1.0})\n assert_equal 1.0, query.body[:min_score]\n end\n\n # nested\n\n def test_nested_search\n setup_speaker\n store [{name: \"Product A\", aisle: {\"id\" => 1, \"name\" => \"Frozen\"}}], Speaker\n assert_search \"frozen\", [\"Product A\"], {fields: [\"aisle.name\"]}, Speaker\n end\n\n # other tests\n\n def test_includes\n skip unless activerecord?\n\n store_names [\"Product A\"]\n assert Product.search(\"product\", includes: [:store]).first.association(:store).loaded?\n assert Product.search(\"product\").includes(:store).first.association(:store).loaded?\n end\n\n def test_model_includes\n skip unless activerecord?\n\n store_names [\"Product A\"]\n store_names [\"Store A\"], Store\n\n associations = {Product => [:store], Store => [:products]}\n result = Searchkick.search(\"*\", models: [Product, Store], model_includes: associations)\n\n assert_equal 2, result.length\n\n result.group_by(&:class).each_pair do |model, records|\n assert records.first.association(associations[model].first).loaded?\n end\n end\n\n def test_scope_results\n skip unless activerecord?\n\n store_names [\"Product A\", \"Product B\"]\n assert_warns \"Records in search index do not exist in database\" do\n assert_search \"product\", [\"Product A\"], scope_results: ->(r) { r.where(name: \"Product A\") }\n end\n end\n\n def test_scope_results_relation\n skip unless activerecord?\n\n store_names [\"Product A\", \"Product B\"]\n assert_warns \"Records in search index do not exist in database\" do\n assert_search_relation [\"Product A\"], Product.search(\"product\").scope_results(->(r) { r.where(name: \"Product A\") })\n end\n end\n\n private\n\n def set_search_slow_log(value)\n settings = {\n \"index.search.slowlog.threshold.query.warn\" => value\n }\n Product.searchkick_index.update_settings(settings)\n end\nend\n"
},
{
"path": "test/reindex_test.rb",
"content": "require_relative \"test_helper\"\n\nclass ReindexTest < Minitest::Test\n def test_record_inline\n store_names [\"Product A\", \"Product B\"], reindex: false\n\n product = Product.find_by!(name: \"Product A\")\n assert_equal true, product.reindex(refresh: true)\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_record_destroyed\n store_names [\"Product A\", \"Product B\"]\n\n product = Product.find_by!(name: \"Product A\")\n product.destroy\n Product.searchkick_index.refresh\n assert_equal true, product.reindex\n end\n\n def test_record_async\n store_names [\"Product A\", \"Product B\"], reindex: false\n\n product = Product.find_by!(name: \"Product A\")\n perform_enqueued_jobs do\n assert_equal true, product.reindex(mode: :async)\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_record_async_job_options\n product = Product.create!(name: \"Product A\")\n assert_enqueued_jobs(1, queue: \"test\") do\n assert_equal true, product.reindex(mode: :async, job_options: {queue: \"test\"})\n end\n end\n\n def test_record_queue\n reindex_queue = Product.searchkick_index.reindex_queue\n reindex_queue.clear\n\n store_names [\"Product A\", \"Product B\"], reindex: false\n\n product = Product.find_by!(name: \"Product A\")\n assert_equal true, product.reindex(mode: :queue)\n Product.searchkick_index.refresh\n assert_search \"product\", []\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_process_queue_job_options\n product = Product.create!(name: \"Product A\")\n product.reindex(mode: :queue)\n assert_enqueued_jobs(1, queue: \"test\") do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\", job_options: {queue: \"test\"})\n end\n end\n\n def test_record_index\n store_names [\"Product A\", \"Product B\"], reindex: false\n\n product = Product.find_by!(name: \"Product A\")\n assert_equal true, Product.searchkick_index.reindex([product], refresh: true)\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_relation_inline\n store_names [\"Product A\"]\n store_names [\"Product B\", \"Product C\"], reindex: false\n Product.where(name: \"Product B\").reindex(refresh: true)\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_relation_associations\n store_names [\"Product A\"]\n store = Store.create!(name: \"Test\")\n Product.create!(name: \"Product B\", store_id: store.id)\n assert_equal true, store.products.reindex(refresh: true)\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_relation_scoping\n store_names [\"Product A\", \"Product B\"]\n Product.dynamic_data = lambda do\n {\n name: \"Count #{Product.count}\"\n }\n end\n Product.where(name: \"Product A\").reindex(refresh: true)\n assert_search \"count\", [\"Count 2\"], load: false\n ensure\n Product.dynamic_data = nil\n end\n\n def test_relation_scoping_restored\n # TODO add test for Mongoid\n skip unless activerecord?\n\n assert_nil Product.current_scope\n Product.where(name: \"Product A\").scoping do\n scope = Product.current_scope\n refute_nil scope\n\n Product.all.reindex(refresh: true)\n\n # note: should be reset even if we don't do it\n assert_equal scope, Product.current_scope\n end\n assert_nil Product.current_scope\n end\n\n def test_relation_should_index\n store_names [\"Product A\", \"Product B\"]\n Searchkick.callbacks(false) do\n Product.find_by(name: \"Product B\").update!(name: \"DO NOT INDEX\")\n end\n assert_equal true, Product.where(name: \"DO NOT INDEX\").reindex\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_relation_async\n store_names [\"Product A\"]\n store_names [\"Product B\", \"Product C\"], reindex: false\n perform_enqueued_jobs do\n Product.where(name: \"Product B\").reindex(mode: :async)\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_relation_async_should_index\n store_names [\"Product A\", \"Product B\"]\n Searchkick.callbacks(false) do\n Product.find_by(name: \"Product B\").update!(name: \"DO NOT INDEX\")\n end\n perform_enqueued_jobs do\n assert_equal true, Product.where(name: \"DO NOT INDEX\").reindex(mode: :async)\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_relation_async_routing\n store_names [\"Store A\"], Store, reindex: false\n perform_enqueued_jobs do\n Store.where(name: \"Store A\").reindex(mode: :async)\n end\n Store.searchkick_index.refresh\n assert_search \"*\", [\"Store A\"], {routing: \"Store A\"}, Store\n end\n\n def test_relation_async_job_options\n store_names [\"Store A\"], Store, reindex: false\n assert_enqueued_jobs(1, queue: \"test\") do\n Store.where(name: \"Store A\").reindex(mode: :async, job_options: {queue: \"test\"})\n end\n end\n\n def test_relation_queue\n reindex_queue = Product.searchkick_index.reindex_queue\n reindex_queue.clear\n\n store_names [\"Product A\"]\n store_names [\"Product B\", \"Product C\"], reindex: false\n\n Product.where(name: \"Product B\").reindex(mode: :queue)\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_relation_queue_all\n reindex_queue = Product.searchkick_index.reindex_queue\n reindex_queue.clear\n\n store_names [\"Product A\"]\n store_names [\"Product B\", \"Product C\"], reindex: false\n\n Product.all.reindex(mode: :queue)\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\"]\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Product\")\n end\n Product.searchkick_index.refresh\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"]\n end\n\n def test_relation_queue_routing\n reindex_queue = Store.searchkick_index.reindex_queue\n reindex_queue.clear\n\n store_names [\"Store A\"], Store, reindex: false\n Store.where(name: \"Store A\").reindex(mode: :queue)\n Store.searchkick_index.refresh\n assert_search \"*\", [], {}, Store\n\n perform_enqueued_jobs do\n Searchkick::ProcessQueueJob.perform_now(class_name: \"Store\")\n end\n Store.searchkick_index.refresh\n assert_search \"*\", [\"Store A\"], {routing: \"Store A\"}, Store\n end\n\n def test_relation_index\n store_names [\"Product A\"]\n store_names [\"Product B\", \"Product C\"], reindex: false\n Product.searchkick_index.reindex(Product.where(name: \"Product B\"), refresh: true)\n assert_search \"product\", [\"Product A\", \"Product B\"]\n end\n\n def test_full_async\n store_names [\"Product A\"], reindex: false\n reindex = nil\n perform_enqueued_jobs do\n reindex = Product.reindex(mode: :async)\n assert_search \"product\", [], conversions: false\n end\n\n index = Searchkick::Index.new(reindex[:index_name])\n index.refresh\n assert_equal 1, index.total_docs\n\n reindex_status = Searchkick.reindex_status(reindex[:name])\n assert_equal true, reindex_status[:completed]\n assert_equal 0, reindex_status[:batches_left]\n\n Product.searchkick_index.promote(reindex[:index_name])\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_full_async_should_index\n store_names [\"Product A\", \"Product B\", \"DO NOT INDEX\"], reindex: false\n\n reindex = nil\n perform_enqueued_jobs do\n reindex = Product.reindex(mode: :async)\n end\n\n index = Searchkick::Index.new(reindex[:index_name])\n index.refresh\n assert_equal 2, index.total_docs\n index.delete\n end\n\n def test_full_async_wait\n store_names [\"Product A\"], reindex: false\n\n perform_enqueued_jobs do\n capture_io do\n Product.reindex(mode: :async, wait: true)\n end\n end\n\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_full_async_job_options\n store_names [\"Product A\"], reindex: false\n\n assert_enqueued_jobs(1, queue: \"test\") do\n Product.reindex(mode: :async, job_options: {queue: \"test\"})\n end\n end\n\n def test_full_async_non_integer_pk\n Sku.create(id: SecureRandom.hex, name: \"Test\")\n\n reindex = nil\n perform_enqueued_jobs do\n reindex = Sku.reindex(mode: :async)\n assert_search \"sku\", [], conversions: false\n end\n\n index = Searchkick::Index.new(reindex[:index_name])\n index.refresh\n assert_equal 1, index.total_docs\n index.delete\n ensure\n Sku.destroy_all\n end\n\n def test_full_queue\n error = assert_raises(ArgumentError) do\n Product.reindex(mode: :queue)\n end\n assert_equal \"Full reindex does not support :queue mode - use :async mode instead\", error.message\n end\n\n def test_full_refresh_interval\n reindex = Product.reindex(refresh_interval: \"30s\", mode: :async, import: false)\n index = Searchkick::Index.new(reindex[:index_name])\n assert_nil Product.searchkick_index.refresh_interval\n assert_equal \"30s\", index.refresh_interval\n\n Product.searchkick_index.promote(index.name, update_refresh_interval: true)\n assert_equal \"1s\", index.refresh_interval\n assert_equal \"1s\", Product.searchkick_index.refresh_interval\n end\n\n def test_full_resume\n Product.searchkick_index.clean_indices\n\n if mongoid?\n error = assert_raises(Searchkick::Error) do\n Product.reindex(resume: true)\n end\n assert_equal \"Resume not supported for Mongoid\", error.message\n else\n assert Product.reindex(resume: true)\n end\n end\n\n def test_full_refresh\n Product.reindex(refresh: true)\n end\n\n def test_full_partial_async\n store_names [\"Product A\"]\n Product.reindex(:search_name, mode: :async)\n assert_search \"product\", [\"Product A\"]\n end\n\n def test_wait_not_async\n error = assert_raises(ArgumentError) do\n Product.reindex(wait: false)\n end\n assert_equal \"wait only available in :async mode\", error.message\n end\n\n def test_object_index\n error = assert_raises(Searchkick::Error) do\n Product.searchkick_index.reindex(Object.new)\n end\n assert_equal \"Cannot reindex object\", error.message\n end\n\n def test_transaction\n skip unless activerecord?\n\n Product.transaction do\n store_names [\"Product A\"]\n raise ActiveRecord::Rollback\n end\n assert_search \"*\", []\n end\n\n def test_both_paths\n Product.searchkick_index.delete if Product.searchkick_index.exists?\n Product.reindex\n Product.reindex # run twice for both index paths\n end\nend\n"
},
{
"path": "test/reindex_v2_job_test.rb",
"content": "require_relative \"test_helper\"\n\nclass ReindexV2JobTest < Minitest::Test\n def test_create\n product = Searchkick.callbacks(false) { Product.create!(name: \"Boom\") }\n Product.searchkick_index.refresh\n assert_search \"*\", []\n Searchkick::ReindexV2Job.perform_now(\"Product\", product.id.to_s)\n Product.searchkick_index.refresh\n assert_search \"*\", [\"Boom\"]\n end\n\n def test_destroy\n product = Searchkick.callbacks(false) { Product.create!(name: \"Boom\") }\n Product.reindex\n assert_search \"*\", [\"Boom\"]\n Searchkick.callbacks(false) { product.destroy }\n Searchkick::ReindexV2Job.perform_now(\"Product\", product.id.to_s)\n Product.searchkick_index.refresh\n assert_search \"*\", []\n end\nend\n"
},
{
"path": "test/relation_test.rb",
"content": "require_relative \"test_helper\"\n\nclass RelationTest < Minitest::Test\n def test_loaded\n Product.searchkick_index.refresh\n products = Product.search(\"*\")\n refute products.loaded?\n assert_equal 0, products.count\n assert products.loaded?\n refute products.clone.loaded?\n refute products.dup.loaded?\n refute products.limit(2).loaded?\n error = assert_raises(Searchkick::Error) do\n products.limit!(2)\n end\n assert_equal \"Relation loaded\", error.message\n end\n\n def test_mutating\n store_names [\"Product A\", \"Product B\"]\n products = Product.search(\"*\").order(:name)\n products.limit!(1)\n assert_equal [\"Product A\"], products.map(&:name)\n end\n\n def test_non_mutating\n store_names [\"Product A\", \"Product B\"]\n products = Product.search(\"*\").order(:name)\n products.limit(1)\n assert_equal [\"Product A\", \"Product B\"], products.map(&:name)\n end\n\n def test_load\n products = Product.search(\"*\")\n refute products.loaded?\n assert products.load.loaded?\n assert products.load.load.loaded?\n end\n\n def test_clone\n products = Product.search(\"*\")\n assert_equal 10, products.limit(10).limit_value\n assert_equal 10000, products.limit_value\n end\n\n def test_only\n assert_equal 10, Product.search(\"*\").limit(10).only(:limit).limit_value\n end\n\n def test_except\n assert_equal 10000, Product.search(\"*\").limit(10).except(:limit).limit_value\n end\n\n def test_first\n store_names [\"Product A\", \"Product B\"]\n products = Product.search(\"product\")\n assert_kind_of Product, products.first\n assert_kind_of Array, products.first(1)\n assert_equal 1, products.limit(1).first(2).size\n end\n\n def test_first_loaded\n store_names [\"Product A\", \"Product B\"]\n products = Product.search(\"product\").load\n assert_kind_of Product, products.first\n end\n\n # TODO call pluck or select on Active Record query\n # currently uses pluck from Active Support enumerable\n def test_pluck\n store_names [\"Product A\", \"Product B\"]\n assert_equal [\"Product A\", \"Product B\"], Product.search(\"product\").pluck(:name).sort\n assert_equal [\"Product A\", \"Product B\"], Product.search(\"product\").load(false).pluck(:name).sort\n end\n\n def test_model\n assert_equal Product, Product.search(\"product\").model\n assert_nil Searchkick.search(\"product\").model\n end\n\n def test_klass\n assert_equal Product, Product.search(\"product\").klass\n assert_nil Searchkick.search(\"product\").klass\n end\n\n def test_respond_to\n relation = Product.search(\"product\")\n assert relation.respond_to?(:page)\n assert relation.respond_to?(:response)\n assert relation.respond_to?(:size)\n refute relation.respond_to?(:hello)\n refute relation.loaded?\n end\n\n def test_inspect\n store_names [\"Product A\"]\n assert_match \"# :word_start}]\n end\n\n def test_multiple_models\n skip # flaky test\n store_names [\"Great White Shark\", \"Hammerhead Shark\", \"Tiger Shark\"]\n assert_equal \"how big is a tiger shark\", Searchkick.search(\"How Big is a Tigre Shar\", suggest: [:name], fields: [:name]).suggestions.first\n end\n\n def test_multiple_models_no_fields\n store_names [\"Great White Shark\", \"Hammerhead Shark\", \"Tiger Shark\"]\n assert_raises(ArgumentError) { Searchkick.search(\"How Big is a Tigre Shar\", suggest: true) }\n end\n\n def test_star\n assert_equal [], Product.search(\"*\", suggest: true).suggestions\n end\n\n protected\n\n def assert_suggest(term, expected, options = {})\n result = Product.search(term, suggest: true, **options).suggestions.first\n if expected.nil?\n assert_nil result\n else\n assert_equal expected, result\n end\n end\n\n # any order\n def assert_suggest_all(term, expected, options = {})\n assert_equal expected.sort, Product.search(term, suggest: true, **options).suggestions.sort\n end\nend\n"
},
{
"path": "test/support/activerecord.rb",
"content": "require \"active_record\"\n\n# for debugging\nActiveRecord::Base.logger = $logger\n\n# rails does this in activerecord/lib/active_record/railtie.rb\nActiveRecord.default_timezone = :utc\nActiveRecord::Base.time_zone_aware_attributes = true\n\n# migrations\nActiveRecord::Base.establish_connection adapter: \"sqlite3\", database: \":memory:\"\n\nrequire_relative \"apartment\" if defined?(Apartment)\n\nActiveRecord::Migration.verbose = ENV[\"VERBOSE\"]\n\nActiveRecord::Schema.define do\n create_table :products do |t|\n t.string :name\n t.integer :store_id\n t.boolean :in_stock\n t.boolean :backordered\n t.integer :orders_count\n t.decimal :found_rate\n t.integer :price\n t.string :color\n t.decimal :latitude, precision: 10, scale: 7\n t.decimal :longitude, precision: 10, scale: 7\n t.text :description\n t.text :alt_description\n t.text :embedding\n t.text :embedding2\n t.text :embedding3\n t.text :embedding4\n t.timestamps null: true\n end\n\n create_table :stores do |t|\n t.string :name\n end\n\n create_table :regions do |t|\n t.string :name\n t.text :text\n end\n\n create_table :speakers do |t|\n t.string :name\n end\n\n create_table :animals do |t|\n t.string :name\n t.string :type\n end\n\n create_table :skus, id: :uuid do |t|\n t.string :name\n end\n\n create_table :songs do |t|\n t.string :name\n end\n\n create_table :bands do |t|\n t.string :name\n t.boolean :active\n end\n\n create_table :artists do |t|\n t.string :name\n t.boolean :active\n t.boolean :should_index\n end\nend\n\nclass Product < ActiveRecord::Base\n belongs_to :store\n\n serialize :embedding, coder: JSON\n serialize :embedding2, coder: JSON\n serialize :embedding3, coder: JSON\n serialize :embedding4, coder: JSON\nend\n\nclass Store < ActiveRecord::Base\n has_many :products\nend\n\nclass Region < ActiveRecord::Base\nend\n\nclass Speaker < ActiveRecord::Base\nend\n\nclass Animal < ActiveRecord::Base\nend\n\nclass Dog < Animal\nend\n\nclass Cat < Animal\nend\n\nclass Sku < ActiveRecord::Base\nend\n\nclass Song < ActiveRecord::Base\nend\n\nclass Band < ActiveRecord::Base\n default_scope { where(active: true).order(:name) }\nend\n\nclass Artist < ActiveRecord::Base\n default_scope { where(active: true).order(:name) }\nend\n"
},
{
"path": "test/support/apartment.rb",
"content": "module Rails\n def self.env\n ENV[\"RACK_ENV\"]\n end\nend\n\ntenants = [\"tenant1\", \"tenant2\"]\nApartment.configure do |config|\n config.tenant_names = tenants\n config.database_schema_file = false\n config.excluded_models = [\"Product\", \"Store\", \"Region\", \"Speaker\", \"Animal\", \"Dog\", \"Cat\", \"Sku\", \"Song\", \"Band\"]\nend\n\nclass Tenant < ActiveRecord::Base\n searchkick index_prefix: -> { Apartment::Tenant.current }\nend\n\ntenants.each do |tenant|\n begin\n Apartment::Tenant.create(tenant)\n rescue Apartment::TenantExists\n # do nothing\n end\n Apartment::Tenant.switch!(tenant)\n\n ActiveRecord::Schema.define do\n create_table :tenants, force: true do |t|\n t.string :name\n t.timestamps null: true\n end\n end\n\n Tenant.reindex\nend\n\nApartment::Tenant.reset\n"
},
{
"path": "test/support/helpers.rb",
"content": "class Minitest::Test\n include ActiveJob::TestHelper\n\n def setup\n [Product, Store].each do |model|\n setup_model(model)\n end\n end\n\n protected\n\n def setup_animal\n setup_model(Animal)\n end\n\n def setup_region\n setup_model(Region)\n end\n\n def setup_speaker\n setup_model(Speaker)\n end\n\n def setup_model(model)\n # reindex once\n ($setup_model ||= {})[model] ||= (model.reindex || true)\n\n # clear every time\n Searchkick.callbacks(:bulk) do\n model.destroy_all\n end\n end\n\n def store(documents, model = default_model, reindex: true)\n if reindex\n with_callbacks(:bulk) do\n with_transaction(model) do\n model.create!(documents.shuffle)\n end\n end\n model.searchkick_index.refresh\n else\n Searchkick.callbacks(false) do\n with_transaction(model) do\n model.create!(documents.shuffle)\n end\n end\n # prevent warnings\n model.searchkick_index.refresh\n end\n end\n\n def store_names(names, model = default_model, reindex: true)\n store names.map { |name| {name: name} }, model, reindex: reindex\n end\n\n # no order\n def assert_search(term, expected, options = {}, model = default_model)\n assert_equal expected.sort, model.search(term, **options).map(&:name).sort\n assert_equal expected.sort, build_relation(model, term, **options).map(&:name).sort\n end\n\n def assert_search_relation(expected, relation)\n assert_equal expected.sort, relation.map(&:name).sort\n end\n\n def assert_order(term, expected, options = {}, model = default_model)\n assert_equal expected, model.search(term, **options).map(&:name)\n assert_equal expected, build_relation(model, term, **options).map(&:name)\n end\n\n def assert_order_relation(expected, relation)\n assert_equal expected, relation.map(&:name)\n end\n\n def assert_equal_scores(term, options = {}, model = default_model)\n assert_equal 1, model.search(term, **options).hits.map { |a| a[\"_score\"] }.uniq.size\n end\n\n def assert_first(term, expected, options = {}, model = default_model)\n assert_equal expected, model.search(term, **options).map(&:name).first\n end\n\n def assert_warns(message)\n _, stderr = capture_io do\n yield\n end\n assert_match \"[searchkick] WARNING: #{message}\", stderr\n end\n\n def build_relation(model, term, **options)\n relation = model.search(term)\n options.each do |k, v|\n relation = relation.public_send(k, v)\n end\n relation\n end\n\n def with_options(options, model = default_model)\n previous_options = model.searchkick_options.dup\n begin\n model.instance_variable_set(:@searchkick_index_name, nil)\n model.searchkick_options.merge!(options)\n model.reindex\n yield\n ensure\n model.instance_variable_set(:@searchkick_index_name, nil)\n model.searchkick_options.clear\n model.searchkick_options.merge!(previous_options)\n end\n end\n\n def with_callbacks(value, &block)\n if Searchkick.callbacks?(default: nil).nil?\n Searchkick.callbacks(value, &block)\n else\n yield\n end\n end\n\n def with_transaction(model, &block)\n if model.respond_to?(:transaction) && !mongoid?\n model.transaction(&block)\n else\n yield\n end\n end\n\n def activerecord?\n defined?(ActiveRecord)\n end\n\n def mongoid?\n defined?(Mongoid)\n end\n\n def default_model\n Product\n end\n\n def ci?\n ENV[\"CI\"]\n end\n\n # for Active Job helpers\n def tagged_logger\n end\nend\n"
},
{
"path": "test/support/kaminari.yml",
"content": "en:\n views:\n pagination:\n first: \"« First\"\n last: \"Last »\"\n previous: \"‹ Prev\"\n next: \"Next ›\"\n truncate: \"…\"\n helpers:\n page_entries_info:\n entry:\n zero: \"entries\"\n one: \"entry\"\n other: \"entries\"\n one_page:\n display_entries:\n zero: \"No %{entry_name} found\"\n one: \"Displaying 1 %{entry_name}\"\n other: \"Displaying all %{count} %{entry_name}\"\n more_pages:\n display_entries: \"Displaying %{entry_name} %{first} - %{last} of %{total} in total\"\n"
},
{
"path": "test/support/mongoid.rb",
"content": "Mongoid.logger = $logger\nMongo::Logger.logger = $logger if defined?(Mongo::Logger)\n\nMongoid.configure do |config|\n config.connect_to \"searchkick_test\", server_selection_timeout: 1\nend\n\nclass Product\n include Mongoid::Document\n include Mongoid::Timestamps\n\n field :name\n field :store_id, type: Integer\n field :in_stock, type: Boolean\n field :backordered, type: Boolean\n field :orders_count, type: Integer\n field :found_rate, type: BigDecimal\n field :price, type: Integer\n field :color\n field :latitude, type: BigDecimal\n field :longitude, type: BigDecimal\n field :description\n field :alt_description\n field :embedding, type: Array\n field :embedding2, type: Array\n field :embedding3, type: Array\n field :embedding4, type: Array\nend\n\nclass Store\n include Mongoid::Document\n has_many :products\n\n field :name\nend\n\nclass Region\n include Mongoid::Document\n\n field :name\n field :text\nend\n\nclass Speaker\n include Mongoid::Document\n\n field :name\nend\n\nclass Animal\n include Mongoid::Document\n\n field :name\nend\n\nclass Dog < Animal\nend\n\nclass Cat < Animal\nend\n\nclass Sku\n include Mongoid::Document\n\n field :name\nend\n\nclass Song\n include Mongoid::Document\n\n field :name\nend\n\nclass Band\n include Mongoid::Document\n\n field :name\n field :active, type: Mongoid::Boolean\n\n default_scope -> { where(active: true).order(name: 1) }\nend\n\nclass Artist\n include Mongoid::Document\n\n field :name\n field :active, type: Mongoid::Boolean\n field :should_index, type: Mongoid::Boolean\n\n default_scope -> { where(active: true).order(name: 1) }\nend\n"
},
{
"path": "test/support/redis.rb",
"content": "options = {}\noptions[:logger] = $logger if !defined?(RedisClient)\n\nSearchkick.redis =\n if !defined?(Redis)\n RedisClient.config.new_pool\n elsif defined?(ConnectionPool)\n ConnectionPool.new { Redis.new(**options) }\n else\n Redis.new(**options)\n end\n\nmodule RedisInstrumentation\n def call(command, redis_config)\n $logger.info \"[redis] #{command.inspect}\"\n super\n end\n\n def call_pipelined(commands, redis_config)\n $logger.info \"[redis] #{commands.inspect}\"\n super\n end\nend\nRedisClient.register(RedisInstrumentation) if defined?(RedisClient)\n"
},
{
"path": "test/synonyms_test.rb",
"content": "require_relative \"test_helper\"\n\nclass SynonymsTest < Minitest::Test\n def test_bleach\n store_names [\"Clorox Bleach\", \"Kroger Bleach\"]\n assert_search \"clorox\", [\"Clorox Bleach\", \"Kroger Bleach\"]\n end\n\n def test_burger_buns\n store_names [\"Hamburger Buns\"]\n assert_search \"burger buns\", [\"Hamburger Buns\"]\n end\n\n def test_bandaids\n store_names [\"Band-Aid\", \"Kroger 12-Pack Bandages\"]\n assert_search \"bandaids\", [\"Band-Aid\", \"Kroger 12-Pack Bandages\"]\n end\n\n def test_reverse\n store_names [\"Hamburger\"]\n assert_search \"burger\", [\"Hamburger\"]\n end\n\n def test_stemmed\n store_names [\"Burger\"]\n assert_search \"hamburgers\", [\"Burger\"]\n end\n\n def test_word_start\n store_names [\"Clorox Bleach\", \"Kroger Bleach\"]\n assert_search \"clorox\", [\"Clorox Bleach\", \"Kroger Bleach\"], fields: [{name: :word_start}]\n end\n\n def test_directional\n store_names [\"Lightbulb\", \"Green Onions\", \"Led\"]\n assert_search \"led\", [\"Lightbulb\", \"Led\"]\n assert_search \"Lightbulb\", [\"Lightbulb\"]\n assert_search \"Halogen Lamp\", [\"Lightbulb\"]\n assert_search \"onions\", [\"Green Onions\"]\n end\n\n def test_case\n store_names [\"Uppercase\"]\n assert_search \"lowercase\", [\"Uppercase\"]\n end\nend\n"
},
{
"path": "test/test_helper.rb",
"content": "require \"bundler/setup\"\nBundler.require(:default)\nrequire \"minitest/autorun\"\nrequire \"active_support/notifications\"\n\nENV[\"RACK_ENV\"] = \"test\"\n\n# for reloadable synonyms\nif ENV[\"CI\"]\n ENV[\"ES_PATH\"] ||= File.join(ENV[\"HOME\"], Searchkick.opensearch? ? \"opensearch\" : \"elasticsearch\", Searchkick.server_version)\nend\n\n$logger = ActiveSupport::Logger.new(ENV[\"VERBOSE\"] ? STDOUT : nil)\n\nif ENV[\"LOG_TRANSPORT\"]\n transport_logger = ActiveSupport::Logger.new(STDOUT)\n if Searchkick.client.transport.respond_to?(:transport)\n Searchkick.client.transport.transport.logger = transport_logger\n else\n Searchkick.client.transport.logger = transport_logger\n end\nend\nSearchkick.search_timeout = 5\nSearchkick.index_suffix = ENV[\"TEST_ENV_NUMBER\"] # for parallel tests\n\nputs \"Running against #{Searchkick.opensearch? ? \"OpenSearch\" : \"Elasticsearch\"} #{Searchkick.server_version}\"\n\nI18n.config.enforce_available_locales = true\n\nActiveJob::Base.logger = $logger\nActiveJob::Base.queue_adapter = :test\n\nActiveSupport::LogSubscriber.logger = ActiveSupport::Logger.new(STDOUT) if ENV[\"VERBOSE\"]\n\nif defined?(Mongoid)\n require_relative \"support/mongoid\"\nelse\n require_relative \"support/activerecord\"\nend\n\nrequire_relative \"support/redis\"\n\n# models\nDir[\"#{__dir__}/models/*\"].each do |file|\n require file\nend\n\nrequire_relative \"support/helpers\"\n"
},
{
"path": "test/unscope_test.rb",
"content": "require_relative \"test_helper\"\n\nclass UnscopeTest < Minitest::Test\n def setup\n @@once ||= Artist.reindex\n\n Artist.unscoped.destroy_all\n end\n\n def test_reindex\n create_records\n\n Artist.reindex\n assert_search \"*\", [\"Test\", \"Test 2\"]\n assert_search \"*\", [\"Test\", \"Test 2\"], {load: false}\n end\n\n def test_relation_async\n create_records\n\n perform_enqueued_jobs do\n Artist.unscoped.reindex(mode: :async)\n end\n\n Artist.searchkick_index.refresh\n assert_search \"*\", [\"Test\", \"Test 2\"]\n end\n\n def create_records\n store [\n {name: \"Test\", active: true, should_index: true},\n {name: \"Test 2\", active: false, should_index: true},\n {name: \"Test 3\", active: false, should_index: false}\n ], reindex: false\n end\n\n def default_model\n Artist\n end\nend\n"
},
{
"path": "test/where_test.rb",
"content": "require_relative \"test_helper\"\n\nclass WhereTest < Minitest::Test\n def test_where\n now = Time.now\n store [\n {name: \"Product A\", store_id: 1, in_stock: true, backordered: true, created_at: now, orders_count: 4, user_ids: [1, 2, 3]},\n {name: \"Product B\", store_id: 2, in_stock: true, backordered: false, created_at: now - 1, orders_count: 3, user_ids: [1]},\n {name: \"Product C\", store_id: 3, in_stock: false, backordered: true, created_at: now - 2, orders_count: 2, user_ids: [1, 3]},\n {name: \"Product D\", store_id: 4, in_stock: false, backordered: false, created_at: now - 3, orders_count: 1}\n ]\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {in_stock: true}\n\n # arrays\n assert_search \"product\", [\"Product A\"], where: {user_ids: 2}\n assert_search \"product\", [\"Product A\", \"Product C\"], where: {user_ids: [2, 3]}\n\n # date\n assert_search \"product\", [\"Product A\"], where: {created_at: {gt: now - 1}}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {created_at: {gte: now - 1}}\n assert_search \"product\", [\"Product D\"], where: {created_at: {lt: now - 2}}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {created_at: {lte: now - 2}}\n\n # integer\n assert_search \"product\", [\"Product A\"], where: {store_id: {lt: 2}}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: {lte: 2}}\n assert_search \"product\", [\"Product D\"], where: {store_id: {gt: 3}}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {store_id: {gte: 3}}\n\n # range\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: 1..2}\n assert_search \"product\", [\"Product A\"], where: {store_id: 1...2}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: [1, 2]}\n assert_search \"product\", [\"Product B\", \"Product C\", \"Product D\"], where: {store_id: {not: 1}}\n assert_search \"product\", [\"Product B\", \"Product C\", \"Product D\"], where: {store_id: {_not: 1}}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {store_id: {not: [1, 2]}}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {store_id: {_not: [1, 2]}}\n assert_search \"product\", [\"Product A\"], where: {user_ids: {lte: 2, gte: 2}}\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\", \"Product D\"], where: {store_id: -Float::INFINITY..Float::INFINITY}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {store_id: 3..Float::INFINITY}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: -Float::INFINITY..2}\n assert_search \"product\", [\"Product C\", \"Product D\"], where: {store_id: 3..}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: ..2}\n assert_search \"product\", [\"Product A\", \"Product B\"], where: {store_id: ...3}\n\n # or\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {or: [[{in_stock: true}, {store_id: 3}]]}\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {or: [[{orders_count: [2, 4]}, {store_id: [1, 2]}]]}\n assert_search \"product\", [\"Product A\", \"Product D\"], where: {or: [[{orders_count: 1}, {created_at: {gte: now - 1}, backordered: true}]]}\n\n # _or\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {_or: [{in_stock: true}, {store_id: 3}]}\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {_or: [{orders_count: [2, 4]}, {store_id: [1, 2]}]}\n assert_search \"product\", [\"Product A\", \"Product D\"], where: {_or: [{orders_count: 1}, {created_at: {gte: now - 1}, backordered: true}]}\n\n # _and\n assert_search \"product\", [\"Product A\"], where: {_and: [{in_stock: true}, {backordered: true}]}\n\n # _not\n assert_search \"product\", [\"Product B\", \"Product C\"], where: {_not: {_or: [{orders_count: 1}, {created_at: {gte: now - 1}, backordered: true}]}}\n\n # all\n assert_search \"product\", [\"Product A\", \"Product C\"], where: {user_ids: {all: [1, 3]}}\n assert_search \"product\", [], where: {user_ids: {all: [1, 2, 3, 4]}}\n\n # any / nested terms\n assert_search \"product\", [\"Product B\", \"Product C\"], where: {user_ids: {not: [2], in: [1, 3]}}\n assert_search \"product\", [\"Product B\", \"Product C\"], where: {user_ids: {_not: [2], in: [1, 3]}}\n\n # not\n assert_search \"product\", [\"Product D\"], where: {user_ids: nil}\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {user_ids: {not: nil}}\n assert_search \"product\", [\"Product A\", \"Product B\", \"Product C\"], where: {user_ids: {_not: nil}}\n assert_search \"product\", [\"Product A\", \"Product C\", \"Product D\"], where: {user_ids: [3, nil]}\n assert_search \"product\", [\"Product B\"], where: {user_ids: {not: [3, nil]}}\n assert_search \"product\", [\"Product B\"], where: {user_ids: {_not: [3, nil]}}\n end\n\n def test_relation\n now = Time.now\n store [\n {name: \"Product A\", store_id: 1, in_stock: true, backordered: true, created_at: now, orders_count: 4, user_ids: [1, 2, 3]},\n {name: \"Product B\", store_id: 2, in_stock: true, backordered: false, created_at: now - 1, orders_count: 3, user_ids: [1]},\n {name: \"Product C\", store_id: 3, in_stock: false, backordered: true, created_at: now - 2, orders_count: 2, user_ids: [1, 3]},\n {name: \"Product D\", store_id: 4, in_stock: false, backordered: false, created_at: now - 3, orders_count: 1}\n ]\n assert_search_relation [\"Product A\", \"Product B\"], Product.search(\"product\").where(in_stock: true)\n\n # multiple where\n assert_search_relation [\"Product A\"], Product.search(\"product\").where(in_stock: true).where(backordered: true)\n assert_search_relation [\"Product A\"], Product.search(\"product\").where.not(store_id: 2).where.not(store_id: 3).where.not(store_id: 4)\n assert_search_relation [], Product.search(\"product\").where(in_stock: true).where(in_stock: false)\n assert_search_relation [], Product.search(\"product\").where(in_stock: true).where(\"in_stock\" => false)\n\n # rewhere\n assert_search_relation [\"Product A\", \"Product C\"], Product.search(\"product\").where(in_stock: true).rewhere(backordered: true)\n\n # not\n assert_search_relation [\"Product C\", \"Product D\"], Product.search(\"product\").where.not(in_stock: true)\n assert_search_relation [\"Product C\"], Product.search(\"product\").where.not(in_stock: true).where(backordered: true)\n assert_search_relation [\"Product A\", \"Product C\"], Product.search(\"product\").where.not(store_id: [2, 4])\n\n # compound\n assert_search_relation [\"Product B\", \"Product C\"], Product.search(\"product\").where(_or: [{in_stock: true}, {backordered: true}]).where(_or: [{store_id: 2}, {orders_count: 2}])\n end\n\n def test_string_operators\n error = assert_raises(ArgumentError) do\n assert_search \"product\", [], where: {store_id: {\"lt\" => 2}}\n end\n assert_includes error.message, \"Unknown where operator\"\n end\n\n def test_unknown_operator\n error = assert_raises(ArgumentError) do\n assert_search \"product\", [], where: {store_id: {contains: \"%2%\"}}\n end\n assert_includes error.message, \"Unknown where operator\"\n end\n\n def test_regexp\n store_names [\"Product A\"]\n assert_search \"*\", [\"Product A\"], where: {name: /\\APro.+\\z/}\n end\n\n def test_alternate_regexp\n store_names [\"Product A\", \"Item B\"]\n assert_search \"*\", [\"Product A\"], where: {name: {regexp: \"Pro.+\"}}\n end\n\n def test_special_regexp\n store_names [\"Product \", \"Item \"]\n assert_search \"*\", [\"Product \"], where: {name: /\\APro.+<.+\\z/}\n end\n\n def test_regexp_not_anchored\n store_names [\"abcde\"]\n assert_search \"*\", [\"abcde\"], where: {name: /abcd/}\n assert_search \"*\", [\"abcde\"], where: {name: /bcde/}\n assert_search \"*\", [\"abcde\"], where: {name: /abcde/}\n assert_search \"*\", [\"abcde\"], where: {name: /.*bcd.*/}\n end\n\n def test_regexp_anchored\n store_names [\"abcde\"]\n assert_search \"*\", [\"abcde\"], where: {name: /\\Aabcde\\z/}\n assert_search \"*\", [\"abcde\"], where: {name: /\\Aabc/}\n assert_search \"*\", [\"abcde\"], where: {name: /cde\\z/}\n assert_search \"*\", [], where: {name: /\\Abcd/}\n assert_search \"*\", [], where: {name: /bcd\\z/}\n end\n\n def test_regexp_case\n store_names [\"abcde\"]\n assert_search \"*\", [], where: {name: /\\AABCDE\\z/}\n assert_search \"*\", [\"abcde\"], where: {name: /\\AABCDE\\z/i}\n end\n\n def test_prefix\n store_names [\"Product A\", \"Product B\", \"Item C\"]\n assert_search \"*\", [\"Product A\", \"Product B\"], where: {name: {prefix: \"Pro\"}}\n end\n\n def test_exists\n store [\n {name: \"Product A\", user_ids: [1, 2]},\n {name: \"Product B\"}\n ]\n assert_search \"product\", [\"Product A\"], where: {user_ids: {exists: true}}\n assert_search \"product\", [\"Product B\"], where: {user_ids: {exists: false}}\n error = assert_raises(ArgumentError) do\n assert_search \"product\", [\"Product A\"], where: {user_ids: {exists: nil}}\n end\n assert_equal \"Passing a value other than true or false to exists is not supported\", error.message\n end\n\n def test_like\n store_names [\"Product ABC\", \"Product DEF\"]\n assert_search \"product\", [\"Product ABC\"], where: {name: {like: \"%ABC%\"}}\n assert_search \"product\", [\"Product ABC\"], where: {name: {like: \"%ABC\"}}\n assert_search \"product\", [], where: {name: {like: \"ABC\"}}\n assert_search \"product\", [], where: {name: {like: \"ABC%\"}}\n assert_search \"product\", [], where: {name: {like: \"ABC%\"}}\n assert_search \"product\", [\"Product ABC\"], where: {name: {like: \"Product_ABC\"}}\n end\n\n def test_like_escape\n store_names [\"Product 100%\", \"Product 1000\"]\n assert_search \"product\", [\"Product 100%\"], where: {name: {like: \"% 100\\\\%\"}}\n end\n\n def test_like_special_characters\n store_names [\n \"Product ABC\", \"Product.ABC\", \"Product?ABC\", \"Product+ABC\", \"Product*ABC\", \"Product|ABC\",\n \"Product{ABC}\", \"Product[ABC]\", \"Product(ABC)\", \"Product\\\"ABC\\\"\", \"Product\\\\ABC\"\n ]\n assert_search \"*\", [\"Product.ABC\"], where: {name: {like: \"Product.A%\"}}\n assert_search \"*\", [\"Product?ABC\"], where: {name: {like: \"Product?A%\"}}\n assert_search \"*\", [\"Product+ABC\"], where: {name: {like: \"Product+A%\"}}\n assert_search \"*\", [\"Product*ABC\"], where: {name: {like: \"Product*A%\"}}\n assert_search \"*\", [\"Product|ABC\"], where: {name: {like: \"Product|A%\"}}\n assert_search \"*\", [\"Product{ABC}\"], where: {name: {like: \"%{ABC}\"}}\n assert_search \"*\", [\"Product[ABC]\"], where: {name: {like: \"%[ABC]\"}}\n assert_search \"*\", [\"Product(ABC)\"], where: {name: {like: \"%(ABC)\"}}\n assert_search \"*\", [\"Product\\\"ABC\\\"\"], where: {name: {like: \"%\\\"ABC\\\"\"}}\n assert_search \"*\", [\"Product\\\\ABC\"], where: {name: {like: \"Product\\\\A%\"}}\n end\n\n def test_like_optional_operators\n store_names [\"Product A&B\", \"Product B\", \"Product <3\", \"Product @Home\"]\n assert_search \"product\", [\"Product A&B\"], where: {name: {like: \"%A&B\"}}\n assert_search \"product\", [\"Product <3\"], where: {name: {like: \"%<%\"}}\n assert_search \"product\", [\"Product @Home\"], where: {name: {like: \"%@Home%\"}}\n end\n\n def test_ilike\n store_names [\"Product ABC\", \"Product DEF\"]\n assert_search \"product\", [\"Product ABC\"], where: {name: {ilike: \"%abc%\"}}\n assert_search \"product\", [\"Product ABC\"], where: {name: {ilike: \"%abc\"}}\n assert_search \"product\", [], where: {name: {ilike: \"abc\"}}\n assert_search \"product\", [], where: {name: {ilike: \"abc%\"}}\n assert_search \"product\", [], where: {name: {ilike: \"abc%\"}}\n assert_search \"product\", [\"Product ABC\"], where: {name: {ilike: \"Product_abc\"}}\n end\n\n def test_ilike_escape\n store_names [\"Product 100%\", \"Product B\"]\n assert_search \"product\", [\"Product 100%\"], where: {name: {ilike: \"% 100\\\\%\"}}\n end\n\n def test_ilike_special_characters\n store_names [\"Product ABC\\\"\", \"Product B\"]\n assert_search \"product\", [\"Product ABC\\\"\"], where: {name: {ilike: \"%abc\\\"\"}}\n end\n\n def test_ilike_optional_operators\n store_names [\"Product A&B\", \"Product B\", \"Product <3\", \"Product @Home\"]\n assert_search \"product\", [\"Product A&B\"], where: {name: {ilike: \"%a&b\"}}\n assert_search \"product\", [\"Product <3\"], where: {name: {ilike: \"%<%\"}}\n assert_search \"product\", [\"Product @Home\"], where: {name: {ilike: \"%@home%\"}}\n end\n\n def test_script\n store [\n {name: \"Product A\", store_id: 1},\n {name: \"Product B\", store_id: 10}\n ]\n assert_search \"product\", [\"Product A\"], where: {_script: Searchkick.script(\"doc['store_id'].value < 10\")}\n assert_search \"product\", [\"Product A\"], where: {_script: Searchkick.script(\"doc['store_id'].value < 10\", lang: \"expression\")}\n assert_search \"product\", [\"Product A\"], where: {_script: Searchkick.script(\"doc['store_id'].value < params['value']\", params: {value: 10})}\n end\n\n def test_script_string\n error = assert_raises(TypeError) do\n assert_search \"product\", [\"Product A\"], where: {_script: \"doc['store_id'].value < 10\"}\n end\n assert_equal \"expected Searchkick::Script\", error.message\n end\n\n def test_string\n store [\n {name: \"Product A\", color: \"RED\"}\n ]\n assert_search \"product\", [\"Product A\"], where: {color: \"RED\"}\n end\n\n def test_nil\n store [\n {name: \"Product A\"},\n {name: \"Product B\", color: \"red\"}\n ]\n assert_search \"product\", [\"Product A\"], where: {color: nil}\n end\n\n def test_id\n store_names [\"Product A\"]\n product = Product.first\n assert_search \"product\", [\"Product A\"], where: {id: product.id.to_s}\n end\n\n def test_empty\n store_names [\"Product A\"]\n assert_search \"product\", [\"Product A\"], where: {}\n end\n\n def test_empty_array\n store_names [\"Product A\"]\n assert_search \"product\", [], where: {store_id: []}\n end\n\n # https://discuss.elastic.co/t/numeric-range-quey-or-filter-in-an-array-field-possible-or-not/14053\n # https://gist.github.com/jprante/7099463\n def test_range_array\n store [\n {name: \"Product A\", user_ids: [11, 23, 13, 16, 17, 23]},\n {name: \"Product B\", user_ids: [1, 2, 3, 4, 5, 6, 7, 8, 9]},\n {name: \"Product C\", user_ids: [101, 230, 150, 200]}\n ]\n assert_search \"product\", [\"Product A\"], where: {user_ids: {gt: 10, lt: 24}}\n end\n\n def test_range_array_again\n store [\n {name: \"Product A\", user_ids: [19, 32, 42]},\n {name: \"Product B\", user_ids: [13, 40, 52]}\n ]\n assert_search \"product\", [\"Product A\"], where: {user_ids: {gt: 26, lt: 36}}\n end\n\n def test_near\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {near: [37.5, -122.5]}}\n end\n\n def test_near_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {near: {lat: 37.5, lon: -122.5}}}\n end\n\n def test_near_within\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_search \"san\", [\"San Francisco\", \"San Antonio\"], where: {location: {near: [37, -122], within: \"2000mi\"}}\n end\n\n def test_near_within_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n assert_search \"san\", [\"San Francisco\", \"San Antonio\"], where: {location: {near: {lat: 37, lon: -122}, within: \"2000mi\"}}\n end\n\n def test_geo_polygon\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000},\n {name: \"San Marino\", latitude: 43.9333, longitude: 12.4667}\n ]\n polygon = [\n {lat: 42.185695, lon: -125.496146},\n {lat: 42.185695, lon: -94.125535},\n {lat: 27.122789, lon: -94.125535},\n {lat: 27.12278, lon: -125.496146}\n ]\n assert_search \"san\", [\"San Francisco\", \"San Antonio\"], where: {location: {geo_polygon: {points: polygon}}}\n\n polygon << polygon.first\n assert_search \"san\", [\"San Francisco\", \"San Antonio\"], where: {location: {geo_shape: {type: \"polygon\", coordinates: [polygon]}}}\n end\n\n def test_top_left_bottom_right\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {top_left: [38, -123], bottom_right: [37, -122]}}\n end\n\n def test_top_left_bottom_right_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}\n end\n\n def test_top_right_bottom_left\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {top_right: [38, -122], bottom_left: [37, -123]}}\n end\n\n def test_top_right_bottom_left_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {location: {top_right: {lat: 38, lon: -122}, bottom_left: {lat: 37, lon: -123}}}\n end\n\n def test_multiple_locations\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {multiple_locations: {near: [37.5, -122.5]}}\n end\n\n def test_multiple_locations_with_term_filter\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [], where: {multiple_locations: {near: [37.5, -122.5]}, name: \"San Antonio\"}\n assert_search \"san\", [\"San Francisco\"], where: {multiple_locations: {near: [37.5, -122.5]}, name: \"San Francisco\"}\n end\n\n def test_multiple_locations_hash\n store [\n {name: \"San Francisco\", latitude: 37.7833, longitude: -122.4167},\n {name: \"San Antonio\", latitude: 29.4167, longitude: -98.5000}\n ]\n assert_search \"san\", [\"San Francisco\"], where: {multiple_locations: {near: {lat: 37.5, lon: -122.5}}}\n end\n\n def test_nested\n store [\n {name: \"Product A\", details: {year: 2016}}\n ]\n assert_search \"product\", [\"Product A\"], where: {\"details.year\" => 2016}\n end\nend\n"
}
]