Showing preview only (295K chars total). Download the full file or copy to clipboard to get everything.
Repository: cloudant/mango
Branch: master
Commit: 3da3110a6ee1
Files: 43
Total size: 281.4 KB
Directory structure:
gitextract_yrrwmnq4/
├── .gitignore
├── LICENSE.txt
├── README.md
├── TODO.md
├── src/
│ ├── mango.app.src
│ ├── mango.hrl
│ ├── mango_crud.erl
│ ├── mango_cursor.erl
│ ├── mango_cursor.hrl
│ ├── mango_cursor_text.erl
│ ├── mango_cursor_view.erl
│ ├── mango_doc.erl
│ ├── mango_error.erl
│ ├── mango_fields.erl
│ ├── mango_httpd.erl
│ ├── mango_idx.erl
│ ├── mango_idx.hrl
│ ├── mango_idx_special.erl
│ ├── mango_idx_text.erl
│ ├── mango_idx_view.erl
│ ├── mango_json.erl
│ ├── mango_native_proc.erl
│ ├── mango_opts.erl
│ ├── mango_selector.erl
│ ├── mango_selector_text.erl
│ ├── mango_sort.erl
│ └── mango_util.erl
└── test/
├── 01-index-crud-test.py
├── 02-basic-find-test.py
├── 03-operator-test.py
├── 04-key-tests.py
├── 05-index-selection-test.py
├── 06-basic-text-test.py
├── 06-text-default-field-test.py
├── 07-text-custom-field-list-test.py
├── 08-text-limit-test.py
├── 09-text-sort-test.py
├── 10-disable-array-length-field-test.py
├── README.md
├── friend_docs.py
├── limit_docs.py
├── mango.py
└── user_docs.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
ebin/
test/*.pyc
venv/
.eunit
================================================
FILE: LICENSE.txt
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2014 IBM Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
Mango
=====
A MongoDB inspired query language interface for Apache CouchDB.
Motivation
----------
Mango provides a single HTTP API endpoint that accepts JSON bodies via HTTP POST. These bodies provide a set of instructions that will be handled with the results being returned to the client in the same order as they were specified. The general principle of this API is to be simple to implement on the client side while providing users a more natural conversion to Apache CouchDB than would otherwise exist using the standard RESTful HTTP interface that already exists.
Actions
-------
The general API exposes a set of actions that are similar to what MongoDB exposes (although not all of MongoDB's API is supported). These are meant to be loosely and obviously inspired by MongoDB but without too much attention to maintaining the exact behavior.
Each action is specified as a JSON object with a number of keys that affect the behavior. Each action object has at least one field named "action" which must
have a string value indicating the action to be performed. For each action there are zero or more fields that will affect behavior. Some of these fields are required and some are optional.
For convenience, the HTTP API will accept a JSON body that is either a single JSON object which specifies a single action or a JSON array that specifies a list of actions that will then be invoked serially. While multiple commands can be batched into a single HTTP request, there are no guarantees about atomicity or isolation for a batch of commands.
Activating Query on a cluster
--------------------------------------------
Query can be enabled by setting the following config:
```
rpc:multicall(config, set, ["native_query_servers", "query", "{mango_native_proc, start_link, []}"]).
```
HTTP API
========
This API adds a single URI endpoint to the existing CouchDB HTTP API. Creating databases, authentication, Map/Reduce views, etc are all still supported exactly as currently document. No existing behavior is changed.
The endpoint added is for the URL pattern `/dbname/_query` and has the following characteristics:
* The only HTTP method supported is `POST`.
* The request `Content-Type` must be `application/json`.
* The response status code will either be `200`, `4XX`, or `5XX`
* The response `Content-Type` will be `application/json`
* The response `Transfer-Encoding` will be `chunked`.
* The response is a single JSON object or array that matches to the single command or list of commands that exist in the request.
This is intended to be a significantly simpler use of HTTP than the current APIs. This is motivated by the fact that this entire API is aimed at customers who are not as savvy at HTTP or non-relational document stores. Once a customer is comfortable using this API we hope to expose any other "power features" through the existing HTTP API and its adherence to HTTP semantics.
Supported Actions
=================
This is a list of supported actions that Mango understands. For the time being it is limited to the four normal CRUD actions plus one meta action to create indices on the database.
insert
------
Insert a document or documents into the database.
Keys:
* action - "insert"
* docs - The JSON document to insert
* w (optional) (default: 2) - An integer > 0 for the write quorum size
If the provided document or documents do not contain an "\_id" field one will be added using an automatically generated UUID.
It is more performant to specify multiple documents in the "docs" field than it is to specify multiple independent insert actions. Each insert action is submitted as a single bulk update (ie, \_bulk\_docs in CouchDB terminology). This, however, does not make any guarantees on the isolation or atomicity of the bulk operation. It is merely a performance benefit.
find
----
Retrieve documents from the database.
Keys:
* action - "find"
* selector - JSON object following selector syntax, described below
* limit (optional) (default: 25) - integer >= 0, Limit the number of rows returned
* skip (optional) (default: 0) - integer >= 0, Skip the specified number of rows
* sort (optional) (default: []) - JSON array following sort syntax, described below
* fields (optional) (default: null) - JSON array following the field syntax, described below
* r (optional) (default: 1) - By default a find will return the document that was found when traversing the index. Optionally there can be a quorum read for each document using `r` as the read quorum. This is obviously less performant than using the document local to the index.
* conflicts (optional) (default: false) - boolean, whether or not to include information about any existing conflicts for the document.
The important thing to note about the find command is that it must execute over a generated index. If a selector is provided that cannot be satisfied using an existing index the list of basic indices that could be used will be returned.
For the most part, indices are generated in response to the "create\_index" action (described below) although there are two special indices that can be used as well. The "\_id" is automatically indexed and is similar to every other index. There is also a special "\_seq" index to retrieve documents in the order of their update sequence.
Its also quite possible to generate a query that can't be satisfied by any index. In this case an error will be returned stating that fact. Generally speaking the easiest way to stumble onto this is to attempt to OR two separate fields which would require a complete table scan. In the future I expect to support these more complicated queries using an extended indexing API (which deviates from the current MongoDB model a bit).
update
------
Update an existing document in the database
Keys:
* action - "update"
* selector - JSON object following selector syntax, described below
* update - JSON object following update syntax, described below
* upsert - (optional) (default: false) - boolean, Whether or not to create a new document if the selector does not match any documents in the database
* limit (optional) (default: 1) - integer > 0, How many documents returned from the selector should be modified. Currently has a maximum value of 100
* sort - (optional) (default: []) - JSON array following sort syntax, described below
* r (optional) (default: 1) - integer > 0, read quorum constant
* w (optional) (default: 2) - integer > 0, write quorum constant
Updates are fairly straightforward other than to mention that the selector (like find) must be satisifiable using an existing index.
On the update field, if the provided JSON object has one or more update operator (described below) then the operation is applied onto the existing document (if one exists) else the entire contents are replaced with exactly the value of the `update` field.
delete
------
Remove a document from the database.
Keys:
* action - "delete"
* selector - JSON object following selector syntax, described below
* force (optional) (default: false) - Delete all conflicted versions of the document as well
* limit - (optional) (default: 1) - integer > 0, How many documents to delete from the database. Currently has a maximum value of 100
* sort - (optional) (default: []) - JSON array following sort syntax, described below
* r (optional) (default: 1) - integer > 1, read quorum constant
* w (optional) (default: 2) - integer > 0, write quorum constant
Deletes behave quite similarly to update except they attempt to remove documents from the database. Its important to note that if a document has conflicts it may "appear" that delete's aren't having an effect. This is because the delete operation by default only removes a single revision. Specify `"force":true` if you would like to attempt to delete all live revisions.
If you wish to delete a specific revision of the document, you can specify it in the selector using the special "\_rev" field.
create\_index
-------------
Create an index on the database
Keys:
* action - "create\_index"
* index - JSON array following sort syntax, described below
* missing\_is\_null (optional) (default: false) - When indexing documents that do not contain a field required by the index they are usually ignored for the purpose of that index, using `missing\_is\_null` adds an entry to the index for the document with a value of `null`
* type (optional) (default: "json") - string, specifying the index type to create. Currently only "json" indexes are supported but in the future we will provide full-text indexes as well as Geo spatial indexes
* name (optional) - string, optionally specify a name for the index. If a name is not provided one will be automatically generated
* ddoc (optional) - Indexes can be grouped into design documents underneath the hood for efficiency. This is an advanced feature. Don't specify a design document here unless you know the consequences of index invalidation. By default each index is placed in its own separate design document for isolation.
Anytime an operation is required to locate a document in the database it is required that an index must exist that can be used to locate it. By default the only two indices that exist are for the document "\_id" and the special "\_seq" index.
Indices are created in the background. If you attempt to create an index on a large database and then immediately utilize it, the request may block for a considerable amount of time before the request completes.
Indices can specify multiple fields to index simultaneously. This is roughly analogous to a compound index in SQL with the corresponding tradeoffs. For instance, an index may contain the (ordered set of) fields "foo", "bar", and "baz". If a selector specifying "bar" is received, it can not be answered. Although if a selector specifying "foo" and "bar" is received, it can be answered more efficiently than if there were only an index on "foo" and "bar" independently.
NB: while the index allows the ability to specify sort directions these are currently not supported. The sort direction must currently be specified as "asc" in the JSON. [INTERNAL]: This will require that we patch the view engine as well as the cluster coordinators in Fabric to follow the specified sort orders. The concepts are straightforward but the implementation may need some thought to fit into the current shape of things.
list\_indexes
-------------
List the indexes that exist in a given database.
Keys:
* action - "list\_indexes"
delete\_index
-------------
Delete the specified index from the database.
Keys:
* action - "delete\_index"
* name - string, the index to delete
* design\_doc - string, the design doc id from which to delete the index. For auto-generated index names and design docs, you can retrieve this information from the `list\_indexes` action
Indexes require resources to maintain. If you find that an index is no longer necessary then it can be beneficial to remove it from the database.
describe\_selector
------------------
Shows debugging information for a given selector
Keys:
* action - "describe\_selector"
* selector - JSON object in selector syntax, described below
* extended (optional) (default: false) - Show information on what existing indexes could be used with this selector
This is a useful debugging utility that will show how a given selector is normalized before execution as well as information on what indexes could be used to satisfy it.
If `"extended": true` is included then the list of existing indices that could be used for this selector are also returned.
JSON Syntax Descriptions
========================
This API uses a few defined JSON structures for various operations. Here we'll describe each in detail.
Selector Syntax
---------------
The Mango query language is expressed as a JSON object describing documents of interest. Within this structure it is also possible to express conditional logic using specially named fields. This is inspired by and intended to maintain a fairly close parity to the existing MongoDB behavior.
As an example, the simplest selector for Mango might look something like such:
{"_id": "Paul"}
Which would match the document named "Paul" (if one exists). Extending this example using other fields might look like such:
{"_id": "Paul", "location": "Boston"}
This would match a document named "Paul" *AND* having a "location" value of "Boston". Seeing as though I'm sitting in my basement in Omaha, this is unlikely.
There are two special syntax elements for the object keys in a selector. The first is that the period (full stop, or simply `.`) character denotes subfields in a document. For instance, here are two equivalent examples:
{"location": {"city": "Omaha"}}
{"location.city": "Omaha"}
If the object's key contains the period it could be escaped with backslash, i.e.
{"location\\.city": "Omaha"}
Note that the double backslash here is necessary to encode an actual single backslash.
The second important syntax element is the use of a dollar sign (`$`) prefix to denote operators. For example:
{"age": {"$gt": 21}}
In this example, we have created the boolean expression `age > 21`.
There are two core types of operators in the selector syntax: combination operators and condition operators. In general, combination operators contain groups of condition operators. We'll describe the list of each below.
### Implicit Operators
For the most part every operator must be of the form `{"$operator": argument}`. Though there are two implicit operators for selectors.
First, any JSON object that is not the argument to a condition operator is an implicit `$and` operator on each field. For instance, these two examples are identical:
{"foo": "bar", "baz": true}
{"$and": [{"foo": {"$eq": "bar"}}, {"baz": {"$eq": true}}]}
And as shown, any field that contains a JSON value that has no operators in it is an equality condition. For instance, these are equivalent:
{"foo": "bar"}
{"foo": {"$eq": "bar"}}
And to be clear, these are also equivalent:
{"foo": {"bar": "baz"}}
{"foo": {"$eq": {"bar": "baz"}}}
Although, the previous example would actually be normalized internally to this:
{"foo.bar": {"$eq": "baz"}}
### Combination Operators
These operators are responsible for combining groups of condition operators. Most familiar are the standard boolean operators plus a few extra for working with JSON arrays.
Each of the combining operators take a single argument that is either a condition operator or an array of condition operators.
The list of combining characters:
* "$and" - array argument
* "$or" - array argument
* "$not" - single argument
* "$nor" - array argument
* "$all" - array argument (special operator for array values)
* "$elemMatch" - single argument (special operator for array values)
### Condition Operators
Condition operators are specified on a per field basis and apply to the value indexed for that field. For instance, the basic "$eq" operator matches when the indexed field is equal to its argument. There is currently support for the basic equality and inequality operators as well as a number of meta operators. Some of these operators will accept any JSON argument while some require a specific JSON formatted argument. Each is noted below.
The list of conditional arguments:
(In)equality operators
* "$lt" - any JSON
* "$lte" - any JSON
* "$eq" - any JSON
* "$ne" - any JSON
* "$gte" - any JSON
* "$gt" - any JSON
Object related operators
* "$exists" - boolean, check whether the field exists or not regardless of its value
* "$type" - string, check the document field's type
Array related operators
* "$in" - array of JSON values, the document field must exist in the list provided
* "$nin" - array of JSON values, the document field must not exist in the list provided
* "$size" - integer, special condition to match the length of an array field in a document. Non-array fields cannot match this condition.
Misc related operators
* "$mod" - [Divisor, Remainder], where Divisor and Remainder are both positive integers (ie, greater than 0). Matches documents where (field % Divisor == Remainder) is true. This is false for any non-integer field
* "$regex" - string, a regular expression pattern to match against the document field. Only matches when the field is a string value and matches the supplied matches
Update Syntax
-------------
Need to describe the syntax for update operators.
Sort Syntax
-----------
The sort syntax is a basic array of field name and direction pairs. It looks like such:
[{field1: dir1} | ...]
Where field1 can be any field (dotted notation is available for sub-document fields) and dir1 can be "asc" or "desc".
Note that it is highly recommended that you specify a single key per object in your sort ordering so that the order is not dependent on the combination of JSON libraries between your application and the internals of Mango's indexing engine.
Fields Syntax
-------------
When retrieving documents from the database you can specify that only a subset of the fields are returned. This allows you to limit your results strictly to the parts of the document that are interesting for the local application logic. The fields returned are specified as an array. Unlike MongoDB only the fields specified are included, there is no automatic inclusion of the "\_id" or other metadata fields when a field list is included.
A trivial example:
["foo", "bar", "baz"]
HTTP API
========
Short summary until the full documentation can be brought over.
POST /dbname/\_find
-------------------------
Issue a query.
Request body is a JSON object that has the selector and the various options like limit/skip etc. Or we could post the selector and put the other options into the query string. Though I'd probably prefer to have it all in the body for consistency.
Response is streamed out like a view.
POST /dbname/\_index
--------------------------
Request body contains the index definition.
Response body is empty and the result is returned as the status code (200 OK -> created, 3something for exists).
GET /dbname/\_index
-------------------------
Request body is empty.
Response body is all of the indexes that are available for use by find.
DELETE /dbname/\_index/ddocid/viewname
--------------------------------------------
Remove the specified index.
Request body is empty.
Response body is empty. The status code gives enough information.
================================================
FILE: TODO.md
================================================
* Patch the view engine to do alternative sorts. This will include both the lower level couch\_view* modules as well as the fabric coordinators.
* Patch the view engine so we can specify options when returning docs from cursors. We'll want this so that we can delete specific revisions from a document.
* Need to figure out how to do raw collation on some indices because at
least the _id index uses it forcefully.
* Add lots more to the update API. Mongo appears to be missing some pretty obvious easy functionality here. Things like managing values doing things like multiplying numbers, or common string mutations would be obvious examples. Also it could be interesting to add to the language so that you can do conditional updates based on other document attributes. Definitely not a V1 endeavor.
================================================
FILE: src/mango.app.src
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
{application, mango, [
{description, "MongoDB API compatibility layer for CouchDB"},
{vsn, git},
{registered, []},
{applications, [
kernel,
stdlib,
config,
twig,
fabric
]}
]}.
================================================
FILE: src/mango.hrl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-define(MANGO_ERROR(R), throw({mango_error, ?MODULE, R})).
================================================
FILE: src/mango_crud.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_crud).
-export([
insert/3,
find/5,
update/4,
delete/3,
explain/3
]).
-export([
collect_cb/2,
maybe_add_user_ctx/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
insert(Db, #doc{}=Doc, Opts) ->
insert(Db, [Doc], Opts);
insert(Db, {_}=Doc, Opts) ->
insert(Db, [Doc], Opts);
insert(Db, Docs, Opts0) when is_list(Docs) ->
Opts1 = maybe_add_user_ctx(Db, Opts0),
Opts2 = maybe_int_to_str(w, Opts1),
case fabric:update_docs(Db, Docs, Opts2) of
{ok, Results0} ->
{ok, lists:zipwith(fun result_to_json/2, Docs, Results0)};
{accepted, Results0} ->
{ok, lists:zipwith(fun result_to_json/2, Docs, Results0)};
{aborted, Errors} ->
{error, lists:map(fun result_to_json/1, Errors)}
end.
find(Db, Selector, Callback, UserAcc, Opts0) ->
Opts1 = maybe_add_user_ctx(Db, Opts0),
Opts2 = maybe_int_to_str(r, Opts1),
{ok, Cursor} = mango_cursor:create(Db, Selector, Opts2),
mango_cursor:execute(Cursor, Callback, UserAcc).
update(Db, Selector, Update, Options) ->
Upsert = proplists:get_value(upsert, Options),
case collect_docs(Db, Selector, Options) of
{ok, []} when Upsert ->
InitDoc = mango_doc:update_as_insert(Update),
case mango_doc:has_operators(InitDoc) of
true ->
?MANGO_ERROR(invalid_upsert_with_operators);
false ->
% Probably need to catch and rethrow errors from
% this function.
Doc = couch_doc:from_json_obj(InitDoc),
NewDoc = case Doc#doc.id of
<<"">> ->
Doc#doc{id=couch_uuids:new(), revs={0, []}};
_ ->
Doc
end,
insert(Db, NewDoc, Options)
end;
{ok, Docs} ->
NewDocs = lists:map(fun(Doc) ->
mango_doc:apply_update(Doc, Update)
end, Docs),
insert(Db, NewDocs, Options);
Else ->
Else
end.
delete(Db, Selector, Options) ->
case collect_docs(Db, Selector, Options) of
{ok, Docs} ->
NewDocs = lists:map(fun({Props}) ->
{[
{<<"_id">>, proplists:get_value(<<"_id">>, Props)},
{<<"_rev">>, proplists:get_value(<<"_rev">>, Props)},
{<<"_deleted">>, true}
]}
end, Docs),
insert(Db, NewDocs, Options);
Else ->
Else
end.
explain(Db, Selector, Opts0) ->
Opts1 = maybe_add_user_ctx(Db, Opts0),
Opts2 = maybe_int_to_str(r, Opts1),
{ok, Cursor} = mango_cursor:create(Db, Selector, Opts2),
mango_cursor:explain(Cursor).
maybe_add_user_ctx(Db, Opts) ->
case lists:keyfind(user_ctx, 1, Opts) of
{user_ctx, _} ->
Opts;
false ->
[{user_ctx, Db#db.user_ctx} | Opts]
end.
maybe_int_to_str(_Key, []) ->
[];
maybe_int_to_str(Key, [{Key, Val} | Rest]) when is_integer(Val) ->
[{Key, integer_to_list(Val)} | maybe_int_to_str(Key, Rest)];
maybe_int_to_str(Key, [KV | Rest]) ->
[KV | maybe_int_to_str(Key, Rest)].
result_to_json(#doc{id=Id}, Result) ->
result_to_json(Id, Result);
result_to_json({Props}, Result) ->
Id = couch_util:get_value(<<"_id">>, Props),
result_to_json(Id, Result);
result_to_json(DocId, {ok, NewRev}) ->
{[
{id, DocId},
{rev, couch_doc:rev_to_str(NewRev)}
]};
result_to_json(DocId, {accepted, NewRev}) ->
{[
{id, DocId},
{rev, couch_doc:rev_to_str(NewRev)},
{accepted, true}
]};
result_to_json(DocId, Error) ->
% chttpd:error_info/1 because this is coming from fabric
% and not internal mango operations.
{_Code, ErrorStr, Reason} = chttpd:error_info(Error),
{[
{id, DocId},
{error, ErrorStr},
{reason, Reason}
]}.
% This is for errors because for some reason we
% need a different return value for errors? Blargh.
result_to_json({{Id, Rev}, Error}) ->
{_Code, ErrorStr, Reason} = chttpd:error_info(Error),
{[
{id, Id},
{rev, couch_doc:rev_to_str(Rev)},
{error, ErrorStr},
{reason, Reason}
]}.
collect_docs(Db, Selector, Options) ->
Cb = fun ?MODULE:collect_cb/2,
case find(Db, Selector, Cb, [], Options) of
{ok, Docs} ->
{ok, lists:reverse(Docs)};
Else ->
Else
end.
collect_cb({row, Doc}, Acc) ->
{ok, [Doc | Acc]}.
================================================
FILE: src/mango_cursor.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_cursor).
-export([
create/3,
explain/1,
execute/3
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-include("mango_cursor.hrl").
-define(SUPERVISOR, mango_cursor_sup).
create(Db, Selector0, Opts) ->
Selector = mango_selector:normalize(Selector0),
ExistingIndexes = mango_idx:list(Db),
if ExistingIndexes /= [] -> ok; true ->
?MANGO_ERROR({no_usable_index, no_indexes_defined})
end,
FilteredIndexes = maybe_filter_indexes(ExistingIndexes, Opts),
if FilteredIndexes /= [] -> ok; true ->
?MANGO_ERROR({no_usable_index, no_index_matching_name})
end,
SortIndexes = mango_idx:for_sort(FilteredIndexes, Opts),
if SortIndexes /= [] -> ok; true ->
?MANGO_ERROR({no_usable_index, missing_sort_index})
end,
UsableFilter = fun(I) -> mango_idx:is_usable(I, Selector) end,
UsableIndexes = lists:filter(UsableFilter, SortIndexes),
if UsableIndexes /= [] -> ok; true ->
?MANGO_ERROR({no_usable_index, selector_unsupported})
end,
create_cursor(Db, UsableIndexes, Selector, Opts).
explain(#cursor{}=Cursor) ->
#cursor{
index = Idx,
selector = Selector,
opts = Opts0,
limit = Limit,
skip = Skip,
fields = Fields
} = Cursor,
Mod = mango_idx:cursor_mod(Idx),
Opts = lists:keydelete(user_ctx, 1, Opts0),
{[
{dbname, mango_idx:dbname(Idx)},
{index, mango_idx:to_json(Idx)},
{selector, Selector},
{opts, {Opts}},
{limit, Limit},
{skip, Skip},
{fields, Fields}
] ++ Mod:explain(Cursor)}.
execute(#cursor{index=Idx}=Cursor, UserFun, UserAcc) ->
Mod = mango_idx:cursor_mod(Idx),
Mod:execute(Cursor, UserFun, UserAcc).
maybe_filter_indexes(Indexes, Opts) ->
case lists:keyfind(use_index, 1, Opts) of
{use_index, []} ->
Indexes;
{use_index, [DesignId]} ->
filter_indexes(Indexes, DesignId);
{use_index, [DesignId, ViewName]} ->
filter_indexes(Indexes, DesignId, ViewName)
end.
filter_indexes(Indexes, DesignId0) ->
DesignId = case DesignId0 of
<<"_design/", _/binary>> ->
DesignId0;
Else ->
<<"_design/", Else/binary>>
end,
FiltFun = fun(I) -> mango_idx:ddoc(I) == DesignId end,
lists:filter(FiltFun, Indexes).
filter_indexes(Indexes0, DesignId, ViewName) ->
Indexes = filter_indexes(Indexes0, DesignId),
FiltFun = fun(I) -> mango_idx:name(I) == ViewName end,
lists:filter(FiltFun, Indexes).
create_cursor(Db, Indexes, Selector, Opts) ->
[{CursorMod, CursorModIndexes} | _] = group_indexes_by_type(Indexes),
CursorMod:create(Db, CursorModIndexes, Selector, Opts).
group_indexes_by_type(Indexes) ->
IdxDict = lists:foldl(fun(I, D) ->
dict:append(mango_idx:cursor_mod(I), I, D)
end, dict:new(), Indexes),
% The first cursor module that has indexes will be
% used to service this query. This is so that we
% don't suddenly switch indexes for existing client
% queries.
CursorModules = [
mango_cursor_view,
mango_cursor_text
],
lists:flatmap(fun(CMod) ->
case dict:find(CMod, IdxDict) of
{ok, CModIndexes} ->
[{CMod, CModIndexes}];
error ->
[]
end
end, CursorModules).
================================================
FILE: src/mango_cursor.hrl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-record(cursor, {
db,
index,
ranges,
selector,
opts,
limit = 10000000000,
skip = 0,
fields = undefined,
user_fun,
user_acc
}).
================================================
FILE: src/mango_cursor_text.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_cursor_text).
-export([
create/4,
explain/1,
execute/3
]).
-include_lib("couch/include/couch_db.hrl").
-include_lib("dreyfus/include/dreyfus.hrl").
-include("mango_cursor.hrl").
-include("mango.hrl").
-record(cacc, {
selector,
dbname,
ddocid,
idx_name,
query_args,
bookmark,
limit,
skip,
user_fun,
user_acc,
fields
}).
create(Db, Indexes, Selector, Opts0) ->
Index = case Indexes of
[Index0] ->
Index0;
_ ->
?MANGO_ERROR(multiple_text_indexes)
end,
Opts = unpack_bookmark(Db#db.name, Opts0),
DreyfusLimit = get_dreyfus_limit(),
Limit = erlang:min(DreyfusLimit, couch_util:get_value(limit, Opts, 50)),
Skip = couch_util:get_value(skip, Opts, 0),
Fields = couch_util:get_value(fields, Opts, all_fields),
{ok, #cursor{
db = Db,
index = Index,
ranges = null,
selector = Selector,
opts = Opts,
limit = Limit,
skip = Skip,
fields = Fields
}}.
explain(Cursor) ->
#cursor{
selector = Selector,
opts = Opts
} = Cursor,
[
{'query', mango_selector_text:convert(Selector)},
{sort, sort_query(Opts, Selector)}
].
execute(Cursor, UserFun, UserAcc) ->
#cursor{
db = Db,
index = Idx,
limit = Limit,
skip = Skip,
selector = Selector,
opts = Opts
} = Cursor,
QueryArgs = #index_query_args{
q = mango_selector_text:convert(Selector),
sort = sort_query(Opts, Selector),
raw_bookmark = true
},
CAcc = #cacc{
selector = Selector,
dbname = Db#db.name,
ddocid = ddocid(Idx),
idx_name = mango_idx:name(Idx),
bookmark = get_bookmark(Opts),
limit = Limit,
skip = Skip,
query_args = QueryArgs,
user_fun = UserFun,
user_acc = UserAcc,
fields = Cursor#cursor.fields
},
try
execute(CAcc)
catch
throw:{stop, FinalCAcc} ->
#cacc{
bookmark = FinalBM,
user_fun = UserFun,
user_acc = LastUserAcc
} = FinalCAcc,
JsonBM = dreyfus_bookmark:pack(FinalBM),
Arg = {add_key, bookmark, JsonBM},
{_Go, FinalUserAcc} = UserFun(Arg, LastUserAcc),
{ok, FinalUserAcc}
end.
execute(CAcc) ->
case search_docs(CAcc) of
{ok, Bookmark, []} ->
% If we don't have any results from the
% query it means the request has paged through
% all possible results and the request is over.
NewCAcc = CAcc#cacc{bookmark = Bookmark},
throw({stop, NewCAcc});
{ok, Bookmark, Hits} ->
NewCAcc = CAcc#cacc{bookmark = Bookmark},
HitDocs = get_json_docs(CAcc#cacc.dbname, Hits),
{ok, FinalCAcc} = handle_hits(NewCAcc, HitDocs),
execute(FinalCAcc)
end.
search_docs(CAcc) ->
#cacc{
dbname = DbName,
ddocid = DDocId,
idx_name = IdxName
} = CAcc,
QueryArgs = update_query_args(CAcc),
case dreyfus_fabric_search:go(DbName, DDocId, IdxName, QueryArgs) of
{ok, Bookmark, _, Hits, _, _} ->
{ok, Bookmark, Hits};
{error, Reason} ->
?MANGO_ERROR({text_search_error, {error, Reason}})
end.
handle_hits(CAcc, []) ->
{ok, CAcc};
handle_hits(CAcc0, [{Sort, Doc} | Rest]) ->
CAcc1 = handle_hit(CAcc0, Sort, Doc),
handle_hits(CAcc1, Rest).
handle_hit(CAcc0, Sort, Doc) ->
#cacc{
limit = Limit,
skip = Skip
} = CAcc0,
CAcc1 = update_bookmark(CAcc0, Sort),
case mango_selector:match(CAcc1#cacc.selector, Doc) of
true when Skip > 0 ->
CAcc1#cacc{skip = Skip - 1};
true when Limit == 0 ->
% We hit this case if the user spcified with a
% zero limit. Notice that in this case we need
% to return the bookmark from before this match
throw({stop, CAcc0});
true when Limit == 1 ->
NewCAcc = apply_user_fun(CAcc1, Doc),
throw({stop, NewCAcc});
true when Limit > 1 ->
NewCAcc = apply_user_fun(CAcc1, Doc),
NewCAcc#cacc{limit = Limit - 1};
false ->
CAcc1
end.
apply_user_fun(CAcc, Doc) ->
FinalDoc = mango_fields:extract(Doc, CAcc#cacc.fields),
#cacc{
user_fun = UserFun,
user_acc = UserAcc
} = CAcc,
case UserFun({row, FinalDoc}, UserAcc) of
{ok, NewUserAcc} ->
CAcc#cacc{user_acc = NewUserAcc};
{stop, NewUserAcc} ->
throw({stop, CAcc#cacc{user_acc = NewUserAcc}})
end.
%% Convert Query to Dreyfus sort specifications
%% Covert <<"Field">>, <<"desc">> to <<"-Field">>
%% and append to the dreyfus query
sort_query(Opts, Selector) ->
{sort, {Sort}} = lists:keyfind(sort, 1, Opts),
SortList = lists:map(fun(SortField) ->
{Dir, RawSortField} = case SortField of
{Field, <<"asc">>} -> {asc, Field};
{Field, <<"desc">>} -> {desc, Field};
Field when is_binary(Field) -> {asc, Field}
end,
SField = mango_selector_text:append_sort_type(RawSortField, Selector),
case Dir of
asc ->
SField;
desc ->
<<"-", SField/binary>>
end
end, Sort),
case SortList of
[] -> relevance;
_ -> SortList
end.
get_bookmark(Opts) ->
case lists:keyfind(bookmark, 1, Opts) of
{_, BM} when is_list(BM), BM /= [] ->
BM;
_ ->
nil
end.
update_bookmark(CAcc, Sortable) ->
BM = CAcc#cacc.bookmark,
QueryArgs = CAcc#cacc.query_args,
Sort = QueryArgs#index_query_args.sort,
NewBM = dreyfus_bookmark:update(Sort, BM, [Sortable]),
CAcc#cacc{bookmark = NewBM}.
pack_bookmark(Bookmark) ->
case dreyfus_bookmark:pack(Bookmark) of
null -> nil;
Enc -> Enc
end.
unpack_bookmark(DbName, Opts) ->
NewBM = case lists:keyfind(bookmark, 1, Opts) of
{_, nil} ->
[];
{_, Bin} ->
try
dreyfus_bookmark:unpack(DbName, Bin)
catch _:_ ->
?MANGO_ERROR({invalid_bookmark, Bin})
end
end,
lists:keystore(bookmark, 1, Opts, {bookmark, NewBM}).
ddocid(Idx) ->
case mango_idx:ddoc(Idx) of
<<"_design/", Rest/binary>> ->
Rest;
Else ->
Else
end.
update_query_args(CAcc) ->
#cacc{
bookmark = Bookmark,
query_args = QueryArgs
} = CAcc,
QueryArgs#index_query_args{
bookmark = pack_bookmark(Bookmark),
limit = get_limit(CAcc)
}.
get_limit(CAcc) ->
erlang:min(get_dreyfus_limit(), CAcc#cacc.limit + CAcc#cacc.skip).
get_dreyfus_limit() ->
list_to_integer(config:get("dreyfus", "max_limit", "200")).
get_json_docs(DbName, Hits) ->
Ids = lists:map(fun(#sortable{item = Item}) ->
couch_util:get_value(<<"_id">>, Item#hit.fields)
end, Hits),
{ok, IdDocs} = dreyfus_fabric:get_json_docs(DbName, Ids),
lists:map(fun(#sortable{item = Item} = Sort) ->
Id = couch_util:get_value(<<"_id">>, Item#hit.fields),
case lists:keyfind(Id, 1, IdDocs) of
{Id, {doc, Doc}} ->
{Sort, Doc};
false ->
{Sort, not_found}
end
end, Hits).
================================================
FILE: src/mango_cursor_view.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_cursor_view).
-export([
create/4,
explain/1,
execute/3
]).
-export([
handle_message/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango_cursor.hrl").
create(Db, Indexes, Selector, Opts) ->
FieldRanges = mango_idx_view:field_ranges(Selector),
Composited = composite_indexes(Indexes, FieldRanges),
{Index, IndexRanges} = choose_best_index(Db, Composited),
Limit = couch_util:get_value(limit, Opts, 10000000000),
Skip = couch_util:get_value(skip, Opts, 0),
Fields = couch_util:get_value(fields, Opts, all_fields),
{ok, #cursor{
db = Db,
index = Index,
ranges = IndexRanges,
selector = Selector,
opts = Opts,
limit = Limit,
skip = Skip,
fields = Fields
}}.
explain(Cursor) ->
#cursor{
index = Idx,
ranges = Ranges
} = Cursor,
case Ranges of
[empty] ->
[{range, empty}];
_ ->
[{range, {[
{start_key, mango_idx:start_key(Idx, Ranges)},
{end_key, mango_idx:end_key(Idx, Ranges)}
]}}]
end.
execute(#cursor{db = Db, index = Idx} = Cursor0, UserFun, UserAcc) ->
Cursor = Cursor0#cursor{
user_fun = UserFun,
user_acc = UserAcc
},
case Cursor#cursor.ranges of
[empty] ->
% empty indicates unsatisfiable ranges, so don't perform search
{ok, UserAcc};
_ ->
BaseArgs = #view_query_args{
view_type = red_map,
start_key = mango_idx:start_key(Idx, Cursor#cursor.ranges),
end_key = mango_idx:end_key(Idx, Cursor#cursor.ranges),
include_docs = true
},
Args = apply_opts(Cursor#cursor.opts, BaseArgs),
CB = fun ?MODULE:handle_message/2,
{ok, LastCursor} = case mango_idx:def(Idx) of
all_docs ->
fabric:all_docs(Db, CB, Cursor, Args);
_ ->
% Normal view
DDoc = ddocid(Idx),
Name = mango_idx:name(Idx),
fabric:query_view(Db, DDoc, Name, CB, Cursor, Args)
end,
{ok, LastCursor#cursor.user_acc}
end.
% Any of these indexes may be a composite index. For each
% index find the most specific set of fields for each
% index. Ie, if an index has columns a, b, c, d, then
% check FieldRanges for a, b, c, and d and return
% the longest prefix of columns found.
composite_indexes(Indexes, FieldRanges) ->
lists:foldl(fun(Idx, Acc) ->
Cols = mango_idx:columns(Idx),
Prefix = composite_prefix(Cols, FieldRanges),
[{Idx, Prefix} | Acc]
end, [], Indexes).
composite_prefix([], _) ->
[];
composite_prefix([Col | Rest], Ranges) ->
case lists:keyfind(Col, 1, Ranges) of
{Col, Range} ->
[Range | composite_prefix(Rest, Ranges)];
false ->
[]
end.
% Low and behold our query planner. Or something.
% So stupid, but we can fix this up later. First
% pass: Sort the IndexRanges by (num_columns, idx_name)
% and return the first element. Yes. Its going to
% be that dumb for now.
%
% In the future we can look into doing a cached parallel
% reduce view read on each index with the ranges to find
% the one that has the fewest number of rows or something.
choose_best_index(_DbName, IndexRanges) ->
Cmp = fun({A1, A2}, {B1, B2}) ->
case length(A2) - length(B2) of
N when N < 0 -> true;
N when N == 0 ->
% This is a really bad sort and will end
% up preferring indices based on the
% (dbname, ddocid, view_name) triple
A1 =< B1;
_ ->
false
end
end,
hd(lists:sort(Cmp, IndexRanges)).
handle_message({total_and_offset, _, _} = _TO, Cursor) ->
%twig:log(err, "TOTAL AND OFFSET: ~p", [_TO]),
{ok, Cursor};
handle_message({row, {Props}}, Cursor) ->
%twig:log(err, "ROW: ~p", [Props]),
case doc_member(Cursor#cursor.db, Props, Cursor#cursor.opts) of
{ok, Doc} ->
case mango_selector:match(Cursor#cursor.selector, Doc) of
true ->
FinalDoc = mango_fields:extract(Doc, Cursor#cursor.fields),
handle_doc(Cursor, FinalDoc);
false ->
{ok, Cursor}
end;
Error ->
twig:log(err, "~s :: Error loading doc: ~p", [?MODULE, Error]),
{ok, Cursor}
end;
handle_message(complete, Cursor) ->
%twig:log(err, "COMPLETE", []),
{ok, Cursor};
handle_message({error, Reason}, _Cursor) ->
%twig:log(err, "ERROR: ~p", [Reason]),
{error, Reason}.
handle_doc(#cursor{skip = S} = C, _) when S > 0 ->
{ok, C#cursor{skip = S - 1}};
handle_doc(#cursor{limit = L} = C, Doc) when L > 0 ->
UserFun = C#cursor.user_fun,
UserAcc = C#cursor.user_acc,
{Go, NewAcc} = UserFun({row, Doc}, UserAcc),
{Go, C#cursor{
user_acc = NewAcc,
limit = L - 1
}};
handle_doc(C, _Doc) ->
{stop, C}.
ddocid(Idx) ->
case mango_idx:ddoc(Idx) of
<<"_design/", Rest/binary>> ->
Rest;
Else ->
Else
end.
apply_opts([], Args) ->
Args;
apply_opts([{r, RStr} | Rest], Args) ->
IncludeDocs = case list_to_integer(RStr) of
1 ->
true;
R when R > 1 ->
% We don't load the doc in the view query because
% we have to do a quorum read in the coordinator
% so there's no point.
false
end,
NewArgs = Args#view_query_args{include_docs = IncludeDocs},
apply_opts(Rest, NewArgs);
apply_opts([{conflicts, true} | Rest], Args) ->
% I need to patch things so that views can specify
% parameters when loading the docs from disk
apply_opts(Rest, Args);
apply_opts([{conflicts, false} | Rest], Args) ->
% Ignored cause default
apply_opts(Rest, Args);
apply_opts([{sort, Sort} | Rest], Args) ->
% We only support single direction sorts
% so nothing fancy here.
case mango_sort:directions(Sort) of
[] ->
apply_opts(Rest, Args);
[<<"asc">> | _] ->
apply_opts(Rest, Args);
[<<"desc">> | _] ->
SK = Args#view_query_args.start_key,
SKDI = Args#view_query_args.start_docid,
EK = Args#view_query_args.end_key,
EKDI = Args#view_query_args.end_docid,
NewArgs = Args#view_query_args{
direction = rev,
start_key = EK,
start_docid = EKDI,
end_key = SK,
end_docid = SKDI
},
apply_opts(Rest, NewArgs)
end;
apply_opts([{_, _} | Rest], Args) ->
% Ignore unknown options
apply_opts(Rest, Args).
doc_member(Db, RowProps, Opts) ->
case couch_util:get_value(doc, RowProps) of
{DocProps} ->
{ok, {DocProps}};
undefined ->
Id = couch_util:get_value(id, RowProps),
case mango_util:defer(fabric, open_doc, [Db, Id, Opts]) of
{ok, #doc{}=Doc} ->
{ok, couch_doc:to_json_obj(Doc, [])};
Else ->
Else
end
end.
================================================
FILE: src/mango_doc.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_doc).
-export([
from_bson/1,
apply_update/2,
update_as_insert/1,
has_operators/1,
get_field/2,
get_field/3,
rem_field/2,
set_field/3
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
from_bson({Props}) ->
DocProps = case lists:keytake(<<"_id">>, 1, Props) of
{value, {<<"_id">>, DocId0}, RestProps} ->
DocId = case DocId0 of
{[{<<"$id">>, Id}]} ->
Id;
Else ->
Else
end,
[{<<"_id">>, DocId} | RestProps];
false ->
Props
end,
Doc = couch_doc:from_json_obj({DocProps}),
case Doc#doc.id of
<<"">> ->
Doc#doc{id=couch_uuids:new(), revs={0, []}};
_ ->
Doc
end.
apply_update(#doc{body={Props}}=Doc, Update) ->
NewProps = apply_update(Props, Update),
Doc#doc{body={NewProps}};
apply_update({Props}, {Update}) ->
Result = do_update({Props}, Update),
case has_operators(Result) of
true ->
?MANGO_ERROR(update_leaves_operators);
false ->
ok
end,
Result.
update_as_insert({Update}) ->
NewProps = do_update_to_insert(Update, {[]}),
apply_update(NewProps, {Update}).
has_operators(#doc{body=Body}) ->
has_operators(Body);
has_operators({Props}) when is_list(Props) ->
has_operators_obj(Props);
has_operators(Arr) when is_list(Arr) ->
has_operators_arr(Arr);
has_operators(Val) when is_atom(Val) ->
false;
has_operators(Val) when is_number(Val) ->
false;
has_operators(Val) when is_binary(Val) ->
false.
has_operators_obj([]) ->
false;
has_operators_obj([{K, V} | Rest]) ->
case K of
<<"$", _/binary>> ->
true;
_ ->
case has_operators(V) of
true ->
true;
false ->
has_operators_obj(Rest)
end
end.
has_operators_arr([]) ->
false;
has_operators_arr([V | Rest]) ->
case has_operators(V) of
true ->
true;
false ->
has_operators_arr(Rest)
end.
do_update(Props, []) ->
Props;
do_update(Props, [{Op, Value} | Rest]) ->
UpdateFun = update_operator_fun(Op),
NewProps = case UpdateFun of
undefined ->
lists:keystore(Op, 1, Props, {Op, Value});
Fun when is_function(Fun, 2) ->
case Value of
{ValueProps} ->
Fun(Props, ValueProps);
_ ->
?MANGO_ERROR({invalid_operand, Op, Value})
end
end,
do_update(NewProps, Rest).
update_operator_fun(<<"$", _/binary>> = Op) ->
OperatorFuns = [
% Object operators
{<<"$inc">>, fun do_update_inc/2},
{<<"$rename">>, fun do_update_rename/2},
{<<"$setOnInsert">>, fun do_update_set_on_insert/2},
{<<"$set">>, fun do_update_set/2},
{<<"$unset">>, fun do_update_unset/2},
% Array opereators
{<<"$addToSet">>, fun do_update_add_to_set/2},
{<<"$pop">>, fun do_update_pop/2},
{<<"$pullAll">>, fun do_update_pull_all/2},
{<<"$pull">>, fun do_update_pull/2},
{<<"$pushAll">>, fun do_update_push_all/2},
{<<"$push">>, fun do_update_push/2},
% Bitwise Operators
{<<"$bit">>, fun do_update_bitwise/2}
],
case lists:keyfind(Op, 1, OperatorFuns) of
{Op, Fun} ->
Fun;
false ->
?MANGO_ERROR({update_operator_not_supported, Op})
end;
update_operator_fun(_) ->
undefined.
do_update_inc(Props, []) ->
Props;
do_update_inc(Props, [{Field, Incr} | Rest]) ->
if is_number(Incr) -> ok; true ->
?MANGO_ERROR({invalid_increment, Incr})
end,
NewProps = case get_field(Props, Field, fun is_number/1) of
Value when is_number(Value) ->
set_field(Props, Field, Value + Incr);
not_found ->
set_field(Props, Field, Incr);
_ ->
Props
end,
do_update_inc(NewProps, Rest).
do_update_rename(Props, []) ->
Props;
do_update_rename(Props, [{OldField, NewField} | Rest]) ->
NewProps = case rem_field(Props, OldField) of
{RemProps, OldValue} ->
set_field(RemProps, NewField, OldValue);
_ ->
Props
end,
do_update_rename(NewProps, Rest).
do_update_set_on_insert(Props, _) ->
% This is only called during calls to apply_update/2
% which means this isn't an insert, so drop it on
% the floor.
Props.
do_update_set(Props, []) ->
Props;
do_update_set(Props, [{Field, Value} | Rest]) ->
NewProps = set_field(Props, Field, Value),
do_update_set(NewProps, Rest).
do_update_unset(Props, []) ->
Props;
do_update_unset(Props, [{Field, _} | Rest]) ->
NewProps = case rem_field(Props, Field) of
{RemProps, _} ->
RemProps;
_ ->
Props
end,
do_update_unset(NewProps, Rest).
do_update_add_to_set(Props, []) ->
Props;
do_update_add_to_set(Props, [{Field, NewValue} | Rest]) ->
ToAdd = case NewValue of
{[{<<"$each">>, NewValues}]} when is_list(NewValues) ->
NewValues;
{[{<<"$each">>, NewValue}]} ->
[NewValue];
Else ->
[Else]
end,
NewProps = case get_field(Props, Field) of
OldValues when is_list(OldValues) ->
FinalValues = lists:foldl(fun(V, Acc) ->
lists:append(Acc, [V])
end, OldValues, ToAdd),
set_field(Props, Field, FinalValues);
_ ->
Props
end,
do_update_add_to_set(NewProps, Rest).
do_update_pop(Props, []) ->
Props;
do_update_pop(Props, [{Field, Pos} | Rest]) ->
NewProps = case get_field(Props, Field) of
OldValues when is_list(OldValues) ->
NewValues = case Pos > 0 of
true ->
lists:sublist(OldValues, 1, length(OldValues) - 1);
false ->
lists:sublist(OldValues, 2, length(OldValues) - 1)
end,
set_field(Props, Field, NewValues);
_ ->
Props
end,
do_update_pop(NewProps, Rest).
do_update_pull_all(Props, []) ->
Props;
do_update_pull_all(Props, [{Field, Values} | Rest]) ->
ToRem = case is_list(Values) of
true -> Values;
false -> [Values]
end,
NewProps = case get_field(Props, Field) of
OldValues when is_list(OldValues) ->
NewValues = lists:foldl(fun(ValToRem, Acc) ->
% The logic in these filter functions is a bit
% subtle. The way to think of this is that we
% return true for all elements we want to keep.
FilterFun = case has_operators(ValToRem) of
true ->
fun(A) ->
Sel = mango_selector:normalize(ValToRem),
not mango_selector:match(A, Sel)
end;
false ->
fun(A) -> A /= ValToRem end
end,
lists:filter(FilterFun, Acc)
end, OldValues, ToRem),
set_field(Props, Field, NewValues);
_ ->
Props
end,
do_update_add_to_set(NewProps, Rest).
do_update_pull(Props, []) ->
Props;
do_update_pull(Props, [{Field, Value} | Rest]) ->
ToRem = case Value of
{[{<<"$each">>, Values}]} when is_list(Values) ->
Values;
{[{<<"$each">>, Value}]} ->
[Value];
Else ->
[Else]
end,
NewProps = do_update_pull_all(Props, [{Field, ToRem}]),
do_update_pull(NewProps, Rest).
do_update_push_all(_, []) ->
[];
do_update_push_all(Props, [{Field, Values} | Rest]) ->
ToAdd = case is_list(Values) of
true -> Values;
false -> [Values]
end,
NewProps = case get_field(Props, Field) of
OldValues when is_list(OldValues) ->
NewValues = OldValues ++ ToAdd,
set_field(Props, Field, NewValues);
_ ->
Props
end,
do_update_push_all(NewProps, Rest).
do_update_push(Props, []) ->
Props;
do_update_push(Props, [{Field, Value} | Rest]) ->
ToAdd = case Value of
{[{<<"$each">>, Values}]} when is_list(Values) ->
Values;
{[{<<"$each">>, Value}]} ->
[Value];
Else ->
[Else]
end,
NewProps = do_update_push_all(Props, [{Field, ToAdd}]),
do_update_push(NewProps, Rest).
do_update_bitwise(Props, []) ->
Props;
do_update_bitwise(Props, [{Field, Value} | Rest]) ->
DoOp = case Value of
{[{<<"and">>, Val}]} when is_integer(Val) ->
fun(V) -> V band Val end;
{[{<<"or">>, Val}]} when is_integer(Val) ->
fun(V) -> V bor Val end;
_ ->
fun(V) -> V end
end,
NewProps = case get_field(Props, Field, fun is_number/1) of
Value when is_number(Value) ->
NewValue = DoOp(Value),
set_field(Props, Field, NewValue);
_ ->
Props
end,
do_update_bitwise(NewProps, Rest).
do_update_to_insert([], Doc) ->
Doc;
do_update_to_insert([{<<"$setOnInsert">>, {FieldProps}}], Doc) ->
lists:foldl(fun({Field, Value}, DocAcc) ->
set_field(DocAcc, Field, Value)
end, Doc, FieldProps);
do_update_to_insert([{_, _} | Rest], Doc) ->
do_update_to_insert(Rest, Doc).
get_field(Props, Field) ->
get_field(Props, Field, no_validation).
get_field(Props, Field, Validator) when is_binary(Field) ->
{ok, Path} = mango_util:parse_field(Field),
get_field(Props, Path, Validator);
get_field(Props, [], no_validation) ->
Props;
get_field(Props, [], Validator) ->
case (catch Validator(Props)) of
true ->
Props;
_ ->
invalid_value
end;
get_field({Props}, [Name | Rest], Validator) ->
case lists:keyfind(Name, 1, Props) of
{Name, Value} ->
get_field(Value, Rest, Validator);
false ->
not_found
end;
get_field(Values, [Name | Rest], Validator) when is_list(Values) ->
% Name might be an integer index into an array
try
Pos = list_to_integer(binary_to_list(Name)),
case Pos >= 0 andalso Pos < length(Values) of
true ->
% +1 because Erlang uses 1 based list indices
Value = lists:nth(Pos + 1, Values),
get_field(Value, Rest, Validator);
false ->
bad_path
end
catch error:badarg ->
bad_path
end;
get_field(_, [_|_], _) ->
bad_path.
rem_field(Props, Field) when is_binary(Field) ->
{ok, Path} = mango_util:parse_field(Field),
rem_field(Props, Path);
rem_field({Props}, [Name]) ->
case lists:keytake(Name, 1, Props) of
{value, Value, NewProps} ->
{NewProps, Value};
false ->
not_found
end;
rem_field({Props}, [Name | Rest]) ->
case lists:keyfind(Name, 1, Props) of
{Name, Value} ->
case rem_field(Value, Rest) of
{NewValue, Ret} ->
NewObj = {lists:keystore(Name, 1, Props, {Name, NewValue})},
{NewObj, Ret};
Else ->
Else
end;
false ->
not_found
end;
rem_field(Values, [Name]) when is_list(Values) ->
% Name might be an integer index into an array
try
Pos = list_to_integer(binary_to_list(Name)),
case Pos >= 0 andalso Pos < length(Values) of
true ->
% +1 because Erlang uses 1 based list indices
rem_elem(Pos + 1, Values);
false ->
bad_path
end
catch error:badarg ->
bad_path
end;
rem_field(Values, [Name | Rest]) when is_list(Values) ->
% Name might be an integer index into an array
try
Pos = list_to_integer(binary_to_list(Name)),
case Pos >= 0 andalso Pos < length(Values) of
true ->
% +1 because Erlang uses 1 based list indices
Value = lists:nth(Pos + 1, Values),
case rem_field(Value, Rest) of
{NewValue, Ret} ->
{set_elem(Pos + 1, Values, NewValue), Ret};
Else ->
Else
end;
false ->
bad_path
end
catch error:badarg ->
bad_path
end;
rem_field(_, [_|_]) ->
bad_path.
set_field(Props, Field, Value) when is_binary(Field) ->
{ok, Path} = mango_util:parse_field(Field),
set_field(Props, Path, Value);
set_field({Props}, [Name], Value) ->
{lists:keystore(Name, 1, Props, {Name, Value})};
set_field({Props}, [Name | Rest], Value) ->
case lists:keyfind(Name, 1, Props) of
{Name, Elem} ->
Result = set_field(Elem, Rest, Value),
{lists:keystore(Name, 1, Props, {Name, Result})};
false ->
Nested = make_nested(Rest, Value),
{lists:keystore(Name, 1, Props, {Name, Nested})}
end;
set_field(Values, [Name], Value) when is_list(Values) ->
% Name might be an integer index into an array
try
Pos = list_to_integer(binary_to_list(Name)),
case Pos >= 0 andalso Pos < length(Values) of
true ->
% +1 because Erlang uses 1 based list indices
set_elem(Pos, Values, Value);
false ->
Values
end
catch error:badarg ->
Values
end;
set_field(Values, [Name | Rest], Value) when is_list(Values) ->
% Name might be an integer index into an array
try
Pos = list_to_integer(binary_to_list(Name)),
case Pos >= 0 andalso Pos < length(Values) of
true ->
% +1 because Erlang uses 1 based list indices
Elem = lists:nth(Pos + 1, Values),
Result = set_field(Elem, Rest, Value),
set_elem(Pos, Values, Result);
false ->
Values
end
catch error:badarg ->
Values
end;
set_field(Value, [_|_], _) ->
Value.
make_nested([], Value) ->
Value;
make_nested([Name | Rest], Value) ->
{[{Name, make_nested(Rest, Value)}]}.
rem_elem(1, [Value | Rest]) ->
{Rest, Value};
rem_elem(I, [Item | Rest]) when I > 1 ->
{Tail, Value} = rem_elem(I+1, Rest),
{[Item | Tail], Value}.
set_elem(1, [_ | Rest], Value) ->
[Value | Rest];
set_elem(I, [Item | Rest], Value) when I > 1 ->
[Item | set_elem(I-1, Rest, Value)].
================================================
FILE: src/mango_error.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_error).
-include_lib("couch/include/couch_db.hrl").
-export([
info/2
]).
info(mango_cursor, {no_usable_index, no_indexes_defined}) ->
{
400,
<<"no_usable_index">>,
<<"There are no indexes defined in this database.">>
};
info(mango_cursor, {no_usable_index, no_index_matching_name}) ->
{
400,
<<"no_usable_index">>,
<<"No index matches the index specified with \"use_index\"">>
};
info(mango_cursor, {no_usable_index, missing_sort_index}) ->
{
400,
<<"no_usable_index">>,
<<"No index exists for this sort, try indexing by the sort fields.">>
};
info(mango_cursor, {no_usable_index, selector_unsupported}) ->
{
400,
<<"no_usable_index">>,
<<"There is no index available for this selector.">>
};
info(mango_cursor_text, {invalid_bookmark, BadBookmark}) ->
{
400,
<<"invalid_bookmark">>,
fmt("Invalid boomkark value: ~s", [?JSON_ENCODE(BadBookmark)])
};
info(mango_cursor_text, multiple_text_indexes) ->
{
400,
<<"multiple_text_indexes">>,
<<"You must specify an index with the `use_index` parameter.">>
};
info(mango_cursor_text, {text_search_error, {error, {bad_request, Msg}}})
when is_binary(Msg) ->
{
400,
<<"text_search_error">>,
Msg
};
info(mango_cursor_text, {text_search_error, {error, Error}}) ->
{
400,
<<"text_search_error">>,
fmt("Error performing text search: ~p", [Error])
};
info(mango_fields, {invalid_fields_json, BadFields}) ->
{
400,
<<"invalid_fields">>,
fmt("Fields must be an array of strings, not: ~w", [BadFields])
};
info(mango_fields, {invalid_field_json, BadField}) ->
{
400,
<<"invalid_field">>,
fmt("Invalid JSON for field spec: ~w", [BadField])
};
info(mango_httpd, error_saving_ddoc) ->
{
500,
<<"error_saving_ddoc">>,
<<"Unknown error while saving the design document.">>
};
info(mango_httpd, {error_saving_ddoc, <<"conflict">>}) ->
{
500,
<<"error_saving_ddoc">>,
<<"Encountered a conflict while saving the design document.">>
};
info(mango_httpd, {error_saving_ddoc, Reason}) ->
{
500,
<<"error_saving_ddoc">>,
fmt("Unknown error while saving the design document: ~s", [Reason])
};
info(mango_idx, {invalid_index_type, BadType}) ->
{
400,
<<"invalid_index">>,
fmt("Invalid type for index: ~s", [BadType])
};
info(mango_idx, invalid_query_ddoc_language) ->
{
400,
<<"invalid_index">>,
<<"Invalid design document query language.">>
};
info(mango_idx, no_index_definition) ->
{
400,
<<"invalid_index">>,
<<"Index is missing its definition.">>
};
info(mango_idx_view, {invalid_index_json, BadIdx}) ->
{
400,
<<"invalid_index">>,
fmt("JSON indexes must be an object, not: ~w", [BadIdx])
};
info(mango_idx_view, {index_not_found, BadIdx}) ->
{
404,
<<"invalid_index">>,
fmt("JSON index ~s not found in this design doc.", [BadIdx])
};
info(mango_idx_text, {invalid_index_text, BadIdx}) ->
{
400,
<<"invalid_index">>,
fmt("Text indexes must be an object, not: ~w", [BadIdx])
};
info(mango_idx_text, {invalid_index_fields_definition, Def}) ->
{
400,
<<"invalid_index_fields_definition">>,
fmt("Text Index field definitions must be of the form
{\"name\": \"fieldname\", \"type\":
\"boolean,number, or string\"}. Def: ~p", [Def])
};
info(mango_idx_text, {index_not_found, BadIdx}) ->
{
404,
<<"index_not_found">>,
fmt("Text index ~s not found in this design doc.", [BadIdx])
};
info(mango_opts, {invalid_ejson, Val}) ->
{
400,
<<"invalid_ejson">>,
fmt("Invalid JSON value: ~w", [Val])
};
info(mango_opts, {invalid_key, Key}) ->
{
400,
<<"invalid_key">>,
fmt("Invalid key ~s for this request.", [Key])
};
info(mango_opts, {missing_required_key, Key}) ->
{
400,
<<"missing_required_key">>,
fmt("Missing required key: ~s", [Key])
};
info(mango_opts, {invalid_value, Name, Expect, Found}) ->
{
400,
<<"invalid_value">>,
fmt("Value for ~s is ~w, should be ~w", [Name, Found, Expect])
};
info(mango_opts, {invalid_value, Name, Value}) ->
{
400,
<<"invalid_value">>,
fmt("Invalid value for ~s: ~w", [Name, Value])
};
info(mango_opts, {invalid_string, Val}) ->
{
400,
<<"invalid_string">>,
fmt("Invalid string: ~w", [Val])
};
info(mango_opts, {invalid_boolean, Val}) ->
{
400,
<<"invalid_boolean">>,
fmt("Invalid boolean value: ~w", [Val])
};
info(mango_opts, {invalid_pos_integer, Val}) ->
{
400,
<<"invalid_pos_integer">>,
fmt("~w is not an integer greater than zero", [Val])
};
info(mango_opts, {invalid_non_neg_integer, Val}) ->
{
400,
<<"invalid_non_neg_integer">>,
fmt("~w is not an integer greater than or equal to zero", [Val])
};
info(mango_opts, {invalid_object, BadObj}) ->
{
400,
<<"invalid_object">>,
fmt("~w is not a JSON object", [BadObj])
};
info(mango_opts, {invalid_selector_json, BadSel}) ->
{
400,
<<"invalid_selector_json">>,
fmt("Selector must be a JSON object, not: ~w", [BadSel])
};
info(mango_opts, {invalid_index_name, BadName}) ->
{
400,
<<"invalid_index_name">>,
fmt("Invalid index name: ~w", [BadName])
};
info(mango_opts, {multiple_text_operator, {invalid_selector, BadSel}}) ->
{
400,
<<"multiple_text_selector">>,
fmt("Selector cannot contain more than one $text operator: ~w",
[BadSel])
};
info(mango_selector, {invalid_selector, missing_field_name}) ->
{
400,
<<"invalid_selector">>,
<<"One or more conditions is missing a field name.">>
};
info(mango_selector, {bad_arg, Op, Arg}) ->
{
400,
<<"bad_arg">>,
fmt("Bad argument for operator ~s: ~w", [Op, Arg])
};
info(mango_selector, {not_supported, Op}) ->
{
400,
<<"not_supported">>,
fmt("Unsupported operator: ~s", [Op])
};
info(mango_selector, {invalid_operator, Op}) ->
{
400,
<<"invalid_operator">>,
fmt("Invalid operator: ~s", [Op])
};
info(mango_selector, {bad_field, BadSel}) ->
{
400,
<<"bad_field">>,
fmt("Invalid field normalization on selector: ~w", [BadSel])
};
info(mango_selector_text, {invalid_operator, Op}) ->
{
400,
<<"invalid_operator">>,
fmt("Invalid text operator: ~s", [Op])
};
info(mango_selector_text, {text_sort_error, Field}) ->
S = binary_to_list(Field),
Msg = "Unspecified or ambiguous sort type. Try appending :number or"
" :string to the sort field. ~s",
{
400,
<<"text_sort_error">>,
fmt(Msg, [S])
};
info(mango_sort, {invalid_sort_json, BadSort}) ->
{
400,
<<"invalid_sort_json">>,
fmt("Sort must be an array of sort specs, not: ~w", [BadSort])
};
info(mango_sort, {invalid_sort_dir, BadSpec}) ->
{
400,
<<"invalid_sort_dir">>,
fmt("Invalid sort direction: ~w", BadSpec)
};
info(mango_sort, {invalid_sort_field, BadField}) ->
{
400,
<<"invalid_sort_field">>,
fmt("Invalid sort field: ~w", [BadField])
};
info(mango_sort, {unsupported, mixed_sort}) ->
{
400,
<<"unsupported_mixed_sort">>,
<<"Sorts currently only support a single direction for all fields.">>
};
info(mango_util, {error_loading_doc, DocId}) ->
{
500,
<<"internal_error">>,
fmt("Error loading doc: ~s", [DocId])
};
info(mango_util, error_loading_ddocs) ->
{
500,
<<"internal_error">>,
<<"Error loading design documents">>
};
info(mango_util, {invalid_ddoc_lang, Lang}) ->
{
400,
<<"invalid_ddoc_lang">>,
fmt("Existing design doc has an invalid language: ~w", [Lang])
};
info(Module, Reason) ->
{
500,
<<"unknown_error">>,
fmt("Unknown Error: ~s :: ~w", [Module, Reason])
}.
fmt(Format, Args) ->
iolist_to_binary(io_lib:format(Format, Args)).
================================================
FILE: src/mango_fields.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_fields).
-export([
new/1,
extract/2
]).
-include("mango.hrl").
new([]) ->
{ok, all_fields};
new(Fields) when is_list(Fields) ->
{ok, [field(F) || F <- Fields]};
new(Else) ->
?MANGO_ERROR({invalid_fields_json, Else}).
extract(Doc, undefined) ->
Doc;
extract(Doc, all_fields) ->
Doc;
extract(Doc, Fields) ->
lists:foldl(fun(F, NewDoc) ->
{ok, Path} = mango_util:parse_field(F),
case mango_doc:get_field(Doc, Path) of
not_found ->
NewDoc;
bad_path ->
NewDoc;
Value ->
mango_doc:set_field(NewDoc, Path, Value)
end
end, {[]}, Fields).
field(Val) when is_binary(Val) ->
Val;
field({Val}) when is_list(Val) ->
{Val};
field(Else) ->
?MANGO_ERROR({invalid_field_json, Else}).
================================================
FILE: src/mango_httpd.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_httpd).
-export([
handle_req/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
handle_req(#httpd{} = Req, Db0) ->
try
Db = set_user_ctx(Req, Db0),
handle_req_int(Req, Db)
catch
throw:{mango_error, Module, Reason} ->
%Stack = erlang:get_stacktrace(),
%twig:log(err, "Error: ~s :: ~w~n~p", [Module, Reason, Stack]),
{Code, ErrorStr, ReasonStr} = mango_error:info(Module, Reason),
Resp = {[
{<<"error">>, ErrorStr},
{<<"reason">>, ReasonStr}
]},
chttpd:send_json(Req, Code, Resp)
end.
handle_req_int(#httpd{path_parts=[_, <<"_index">> | _]} = Req, Db) ->
handle_index_req(Req, Db);
handle_req_int(#httpd{path_parts=[_, <<"_explain">> | _]} = Req, Db) ->
handle_explain_req(Req, Db);
handle_req_int(#httpd{path_parts=[_, <<"_find">> | _]} = Req, Db) ->
handle_find_req(Req, Db);
handle_req_int(_, _) ->
throw({not_found, missing}).
handle_index_req(#httpd{method='GET', path_parts=[_, _]}=Req, Db) ->
Idxs = lists:sort(mango_idx:list(Db)),
JsonIdxs = lists:map(fun mango_idx:to_json/1, Idxs),
chttpd:send_json(Req, {[{indexes, JsonIdxs}]});
handle_index_req(#httpd{method='POST', path_parts=[_, _]}=Req, Db) ->
{ok, Opts} = mango_opts:validate_idx_create(chttpd:json_body_obj(Req)),
{ok, Idx0} = mango_idx:new(Db, Opts),
{ok, Idx} = mango_idx:validate(Idx0),
Id = mango_idx:ddoc(Idx),
Name = mango_idx:name(Idx),
{ok, DDoc} = mango_util:load_ddoc(Db, mango_idx:ddoc(Idx)),
Status = case mango_idx:add(DDoc, Idx) of
{ok, DDoc} ->
<<"exists">>;
{ok, NewDDoc} ->
CreateOpts = get_idx_create_opts(Opts),
case mango_crud:insert(Db, NewDDoc, CreateOpts) of
{ok, [{RespProps}]} ->
case lists:keyfind(error, 1, RespProps) of
{error, Reason} ->
?MANGO_ERROR({error_saving_ddoc, Reason});
_ ->
<<"created">>
end;
_ ->
?MANGO_ERROR(error_saving_ddoc)
end
end,
chttpd:send_json(Req, {[{result, Status}, {id, Id}, {name, Name}]});
handle_index_req(#httpd{method='POST', path_parts=[_, <<"_index">>,
<<"_bulk_delete">>]}=Req, Db) ->
{ok, Opts} = mango_opts:validate_bulk_delete(chttpd:json_body_obj(Req)),
DDocIds = get_bulk_delete_ddocs_ids(Opts),
DelOpts = get_idx_create_opts(Opts),
{Success, Error} = mango_idx:bulk_delete(Db, DDocIds, DelOpts),
chttpd:send_json(Req, {[{<<"success">>, Success}, {<<"error">>, Error}]});
handle_index_req(#httpd{method='DELETE',
path_parts=[A, B, <<"_design">>, DDocId0, Type, Name]}=Req, Db) ->
PathParts = [A, B, <<"_design/", DDocId0/binary>>, Type, Name],
handle_index_req(Req#httpd{path_parts=PathParts}, Db);
handle_index_req(#httpd{method='DELETE',
path_parts=[_, _, DDocId0, Type, Name]}=Req, Db) ->
DDocId = case DDocId0 of
<<"_design/", _/binary>> -> DDocId0;
_ -> <<"_design/", DDocId0/binary>>
end,
Idxs = mango_idx:list(Db),
Filt = fun(Idx) ->
IsDDoc = mango_idx:ddoc(Idx) == DDocId,
IsType = mango_idx:type(Idx) == Type,
IsName = mango_idx:name(Idx) == Name,
IsDDoc andalso IsType andalso IsName
end,
case lists:filter(Filt, Idxs) of
[Idx] ->
{ok, DDoc} = mango_util:load_ddoc(Db, mango_idx:ddoc(Idx)),
{ok, NewDDoc} = mango_idx:remove(DDoc, Idx),
FinalDDoc = case NewDDoc#doc.body of
{[{<<"language">>, <<"query">>}]} ->
NewDDoc#doc{deleted = true, body = {[]}};
_ ->
NewDDoc
end,
DelOpts = get_idx_del_opts(Req),
case mango_crud:insert(Db, FinalDDoc, DelOpts) of
{ok, _} ->
chttpd:send_json(Req, {[{ok, true}]});
_ ->
?MANGO_ERROR(error_saving_ddoc)
end;
[] ->
throw({not_found, missing})
end;
handle_index_req(Req, _Db) ->
chttpd:send_method_not_allowed(Req, "GET,POST,DELETE").
handle_explain_req(#httpd{method='POST'}=Req, Db) ->
{ok, Opts0} = mango_opts:validate_find(chttpd:json_body_obj(Req)),
{value, {selector, Sel}, Opts} = lists:keytake(selector, 1, Opts0),
Resp = mango_crud:explain(Db, Sel, Opts),
chttpd:send_json(Req, Resp);
handle_explain_req(Req, _Db) ->
chttpd:send_method_not_allowed(Req, "POST").
handle_find_req(#httpd{method='POST'}=Req, Db) ->
{ok, Opts0} = mango_opts:validate_find(chttpd:json_body_obj(Req)),
{value, {selector, Sel}, Opts} = lists:keytake(selector, 1, Opts0),
{ok, Resp0} = start_find_resp(Req),
{ok, {Resp1, _, KVs}} = run_find(Resp0, Db, Sel, Opts),
end_find_resp(Resp1, KVs);
handle_find_req(Req, _Db) ->
chttpd:send_method_not_allowed(Req, "POST").
set_user_ctx(#httpd{user_ctx=Ctx}, Db) ->
Db#db{user_ctx=Ctx}.
get_idx_create_opts(Opts) ->
case lists:keyfind(w, 1, Opts) of
{w, N} when is_integer(N), N > 0 ->
[{w, integer_to_list(N)}];
_ ->
[{w, "2"}]
end.
get_bulk_delete_ddocs_ids(Opts) ->
case lists:keyfind(docids, 1, Opts) of
{docids, DDocs} when is_list(DDocs) ->
DDocs;
_ ->
[]
end.
get_idx_del_opts(Req) ->
try
WStr = chttpd:qs_value(Req, "w", "2"),
_ = list_to_integer(WStr),
[{w, WStr}]
catch _:_ ->
[{w, "2"}]
end.
start_find_resp(Req) ->
chttpd:start_delayed_json_response(Req, 200, [], "{\"docs\":[").
end_find_resp(Resp0, KVs) ->
FinalAcc = lists:foldl(fun({K, V}, Acc) ->
JK = ?JSON_ENCODE(K),
JV = ?JSON_ENCODE(V),
[JV, ": ", JK, ",\r\n" | Acc]
end, ["\r\n]"], KVs),
Chunk = lists:reverse(FinalAcc, ["}\r\n"]),
{ok, Resp1} = chttpd:send_delayed_chunk(Resp0, Chunk),
chttpd:end_delayed_json_response(Resp1).
run_find(Resp, Db, Sel, Opts) ->
mango_crud:find(Db, Sel, fun handle_doc/2, {Resp, "\r\n", []}, Opts).
handle_doc({add_key, Key, Value}, {Resp, Prepend, KVs}) ->
NewKVs = lists:keystore(Key, 1, KVs, {Key, Value}),
{ok, {Resp, Prepend, NewKVs}};
handle_doc({row, Doc}, {Resp0, Prepend, KVs}) ->
Chunk = [Prepend, ?JSON_ENCODE(Doc)],
{ok, Resp1} = chttpd:send_delayed_chunk(Resp0, Chunk),
{ok, {Resp1, ",\r\n", KVs}}.
================================================
FILE: src/mango_idx.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
% This module is for the "index object" as in, the data structure
% representing an index. Not to be confused with mango_index which
% contains APIs for managing indexes.
-module(mango_idx).
-export([
list/1,
recover/1,
for_sort/2,
new/2,
validate/1,
add/2,
remove/2,
bulk_delete/3,
from_ddoc/2,
special/1,
dbname/1,
ddoc/1,
name/1,
type/1,
def/1,
opts/1,
columns/1,
is_usable/2,
start_key/2,
end_key/2,
cursor_mod/1,
idx_mod/1,
to_json/1
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-include("mango_idx.hrl").
list(Db) ->
{ok, Indexes} = ddoc_cache:open(db_to_name(Db), ?MODULE),
Indexes.
recover(Db) ->
{ok, DDocs0} = mango_util:open_ddocs(Db),
Pred = fun({Props}) ->
case proplists:get_value(<<"language">>, Props) of
<<"query">> -> true;
_ -> false
end
end,
DDocs = lists:filter(Pred, DDocs0),
Special = special(Db),
{ok, Special ++ lists:flatmap(fun(Doc) ->
from_ddoc(Db, Doc)
end, DDocs)}.
for_sort(Indexes, Opts) ->
% If a sort was specified we have to find an index that
% can satisfy the request.
case lists:keyfind(sort, 1, Opts) of
{sort, {SProps}} when is_list(SProps) ->
for_sort_int(Indexes, {SProps});
_ ->
Indexes
end.
for_sort_int(Indexes, Sort) ->
Fields = mango_sort:fields(Sort),
FilterFun = fun(Idx) ->
Cols = mango_idx:columns(Idx),
case {mango_idx:type(Idx), Cols} of
{_, all_fields} ->
true;
{<<"text">>, _} ->
sets:is_subset(sets:from_list(Fields), sets:from_list(Cols));
{<<"json">>, _} ->
lists:prefix(Fields, Cols);
{<<"special">>, _} ->
lists:prefix(Fields, Cols)
end
end,
lists:filter(FilterFun, Indexes).
new(Db, Opts) ->
Def = get_idx_def(Opts),
Type = get_idx_type(Opts),
IdxName = get_idx_name(Def, Opts),
DDoc = get_idx_ddoc(Def, Opts),
{ok, #idx{
dbname = db_to_name(Db),
ddoc = DDoc,
name = IdxName,
type = Type,
def = Def,
opts = filter_opts(Opts)
}}.
validate(Idx) ->
Mod = idx_mod(Idx),
Mod:validate(Idx).
add(DDoc, Idx) ->
Mod = idx_mod(Idx),
{ok, NewDDoc} = Mod:add(DDoc, Idx),
% Round trip through JSON for normalization
Body = ?JSON_DECODE(?JSON_ENCODE(NewDDoc#doc.body)),
{ok, NewDDoc#doc{body = Body}}.
remove(DDoc, Idx) ->
Mod = idx_mod(Idx),
{ok, NewDDoc} = Mod:remove(DDoc, Idx),
% Round trip through JSON for normalization
Body = ?JSON_DECODE(?JSON_ENCODE(NewDDoc#doc.body)),
{ok, NewDDoc#doc{body = Body}}.
bulk_delete(Db, DDocIds, DelOpts0) ->
DelOpts = mango_crud:maybe_add_user_ctx(Db, DelOpts0),
{DeleteDocs, Errors} = lists:foldl(fun(DDocId0, {D, E}) ->
Id = {<<"id">>, DDocId0},
case get_bulk_delete_ddoc(Db, DDocId0) of
not_found ->
{D, [{[Id, {<<"error">>, <<"does not exist">>}]} | E]};
invalid_ddoc_lang ->
{D, [{[Id, {<<"error">>, <<"not a query doc">>}]} | E]};
error_loading_doc ->
{D, [{[Id, {<<"error">>, <<"loading doc">>}]} | E]};
DDoc ->
{[DDoc#doc{deleted = true, body = {[]}} | D], E }
end
end, {[], []}, DDocIds),
case fabric:update_docs(Db, DeleteDocs, DelOpts) of
{ok, Results} ->
bulk_delete_results(lists:zip(DeleteDocs, Results), Errors);
{accepted, Results} ->
bulk_delete_results(lists:zip(DeleteDocs, Results), Errors);
{aborted, Abort} ->
bulk_delete_results(lists:zip(DeleteDocs, Abort), Errors)
end.
bulk_delete_results(DeleteResults, LoadErrors) ->
{Success, Errors} = lists:foldl(fun({#doc{id=DDocId}, Result}, {S, E}) ->
Id = {<<"id">>, DDocId},
case Result of
{_, {_Pos, _}} ->
{[{[Id, {<<"ok">>, true}]} | S], E};
{{_Id, _Rev}, Error} ->
{_Code, ErrorStr, _Reason} = chttpd:error_info(Error),
{S, [{[Id, {<<"error">>, ErrorStr}]} | E]};
Error ->
{_Code, ErrorStr, _Reason} = chttpd:error_info(Error),
{S, [{[Id, {<<"error">>, ErrorStr}]} | E]}
end
end, {[], []}, DeleteResults),
{Success, Errors ++ LoadErrors}.
get_bulk_delete_ddoc(Db, Id0) ->
Id = case Id0 of
<<"_design/", _/binary>> -> Id0;
_ -> <<"_design/", Id0/binary>>
end,
try mango_util:open_doc(Db, Id) of
{ok, #doc{deleted = false} = Doc} ->
mango_util:check_lang(Doc),
Doc;
not_found ->
not_found
catch
{{mango_error, mango_util, {invalid_ddoc_lang, _}}} ->
invalid_ddoc_lang;
{{mango_error, mango_util, {error_loading_doc, _}}} ->
error_loading_doc
end.
from_ddoc(Db, {Props}) ->
DbName = db_to_name(Db),
DDoc = proplists:get_value(<<"_id">>, Props),
case proplists:get_value(<<"language">>, Props) of
<<"query">> -> ok;
_ ->
?MANGO_ERROR(invalid_query_ddoc_language)
end,
IdxMods = [mango_idx_view, mango_idx_text],
Idxs = lists:flatmap(fun(Mod) -> Mod:from_ddoc({Props}) end, IdxMods),
lists:map(fun(Idx) ->
Idx#idx{
dbname = DbName,
ddoc = DDoc
}
end, Idxs).
special(Db) ->
AllDocs = #idx{
dbname = db_to_name(Db),
name = <<"_all_docs">>,
type = <<"special">>,
def = all_docs,
opts = []
},
% Add one for _update_seq
[AllDocs].
dbname(#idx{dbname=DbName}) ->
DbName.
ddoc(#idx{ddoc=DDoc}) ->
DDoc.
name(#idx{name=Name}) ->
Name.
type(#idx{type=Type}) ->
Type.
def(#idx{def=Def}) ->
Def.
opts(#idx{opts=Opts}) ->
Opts.
to_json(#idx{}=Idx) ->
Mod = idx_mod(Idx),
Mod:to_json(Idx).
columns(#idx{}=Idx) ->
Mod = idx_mod(Idx),
Mod:columns(Idx).
is_usable(#idx{}=Idx, Selector) ->
Mod = idx_mod(Idx),
Mod:is_usable(Idx, Selector).
start_key(#idx{}=Idx, Ranges) ->
Mod = idx_mod(Idx),
Mod:start_key(Ranges).
end_key(#idx{}=Idx, Ranges) ->
Mod = idx_mod(Idx),
Mod:end_key(Ranges).
cursor_mod(#idx{type = <<"json">>}) ->
mango_cursor_view;
cursor_mod(#idx{def = all_docs, type= <<"special">>}) ->
mango_cursor_view;
cursor_mod(#idx{type = <<"text">>}) ->
mango_cursor_text.
idx_mod(#idx{type = <<"json">>}) ->
mango_idx_view;
idx_mod(#idx{type = <<"special">>}) ->
mango_idx_special;
idx_mod(#idx{type = <<"text">>}) ->
mango_idx_text.
db_to_name(#db{name=Name}) ->
Name;
db_to_name(Name) when is_binary(Name) ->
Name;
db_to_name(Name) when is_list(Name) ->
iolist_to_binary(Name).
get_idx_def(Opts) ->
case proplists:get_value(def, Opts) of
undefined ->
?MANGO_ERROR(no_index_definition);
Def ->
Def
end.
get_idx_type(Opts) ->
case proplists:get_value(type, Opts) of
<<"json">> -> <<"json">>;
<<"text">> -> <<"text">>;
%<<"geo">> -> <<"geo">>;
undefined -> <<"json">>;
BadType ->
?MANGO_ERROR({invalid_index_type, BadType})
end.
get_idx_ddoc(Idx, Opts) ->
case proplists:get_value(ddoc, Opts) of
<<"_design/", _Rest>> = Name ->
Name;
Name when is_binary(Name) ->
<<"_design/", Name/binary>>;
_ ->
Bin = gen_name(Idx, Opts),
<<"_design/", Bin/binary>>
end.
get_idx_name(Idx, Opts) ->
case proplists:get_value(name, Opts) of
Name when is_binary(Name) ->
Name;
_ ->
gen_name(Idx, Opts)
end.
gen_name(Idx, Opts0) ->
Opts = lists:usort(Opts0),
TermBin = term_to_binary({Idx, Opts}),
Sha = crypto:sha(TermBin),
mango_util:enc_hex(Sha).
filter_opts([]) ->
[];
filter_opts([{user_ctx, _} | Rest]) ->
filter_opts(Rest);
filter_opts([{ddoc, _} | Rest]) ->
filter_opts(Rest);
filter_opts([{name, _} | Rest]) ->
filter_opts(Rest);
filter_opts([{type, _} | Rest]) ->
filter_opts(Rest);
filter_opts([Opt | Rest]) ->
[Opt | filter_opts(Rest)].
================================================
FILE: src/mango_idx.hrl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-record(idx, {
dbname,
ddoc,
name,
type,
def,
opts
}).
================================================
FILE: src/mango_idx_special.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_idx_special).
-export([
validate/1,
add/2,
remove/2,
from_ddoc/1,
to_json/1,
columns/1,
is_usable/2,
start_key/1,
end_key/1
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango_idx.hrl").
validate(_) ->
erlang:exit(invalid_call).
add(_, _) ->
erlang:exit(invalid_call).
remove(_, _) ->
erlang:exit(invalid_call).
from_ddoc(_) ->
erlang:exit(invalid_call).
to_json(#idx{def=all_docs}) ->
{[
{ddoc, null},
{name, <<"_all_docs">>},
{type, <<"special">>},
{def, {[
{<<"fields">>, [{[
{<<"_id">>, <<"asc">>}
]}]}
]}}
]}.
columns(#idx{def=all_docs}) ->
[<<"_id">>].
is_usable(#idx{def=all_docs}, Selector) ->
Fields = mango_idx_view:indexable_fields(Selector),
lists:member(<<"_id">>, Fields).
start_key([{'$gt', Key, _, _}]) ->
case mango_json:special(Key) of
true ->
?MIN_STR;
false ->
Key
end;
start_key([{'$gte', Key, _, _}]) ->
false = mango_json:special(Key),
Key;
start_key([{'$eq', Key, '$eq', Key}]) ->
false = mango_json:special(Key),
Key.
end_key([{_, _, '$lt', Key}]) ->
case mango_json:special(Key) of
true ->
?MAX_STR;
false ->
Key
end;
end_key([{_, _, '$lte', Key}]) ->
false = mango_json:special(Key),
Key;
end_key([{'$eq', Key, '$eq', Key}]) ->
false = mango_json:special(Key),
Key.
================================================
FILE: src/mango_idx_text.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_idx_text).
-export([
validate/1,
validate_fields/1,
add/2,
remove/2,
from_ddoc/1,
to_json/1,
columns/1,
is_usable/2,
get_default_field_options/1
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-include("mango_idx.hrl").
validate(#idx{}=Idx) ->
{ok, Def} = do_validate(Idx#idx.def),
{ok, Idx#idx{def=Def}}.
add(#doc{body={Props0}}=DDoc, Idx) ->
Texts1 = case proplists:get_value(<<"indexes">>, Props0) of
{Texts0} -> Texts0;
_ -> []
end,
NewText = make_text(Idx),
Texts2 = lists:keystore(element(1, NewText), 1, Texts1, NewText),
Props1 = lists:keystore(<<"indexes">>, 1, Props0, {<<"indexes">>,
{Texts2}}),
{ok, DDoc#doc{body={Props1}}}.
remove(#doc{body={Props0}}=DDoc, Idx) ->
Texts1 = case proplists:get_value(<<"indexes">>, Props0) of
{Texts0} ->
Texts0;
_ ->
?MANGO_ERROR({index_not_found, Idx#idx.name})
end,
Texts2 = lists:keydelete(Idx#idx.name, 1, Texts1),
if Texts2 /= Texts1 -> ok; true ->
?MANGO_ERROR({index_not_found, Idx#idx.name})
end,
Props1 = case Texts2 of
[] ->
lists:keydelete(<<"indexes">>, 1, Props0);
_ ->
lists:keystore(<<"indexes">>, 1, Props0, {<<"indexes">>, {Texts2}})
end,
{ok, DDoc#doc{body={Props1}}}.
from_ddoc({Props}) ->
case lists:keyfind(<<"indexes">>, 1, Props) of
{<<"indexes">>, {Texts}} when is_list(Texts) ->
lists:flatmap(fun({Name, {VProps}}) ->
Def = proplists:get_value(<<"index">>, VProps),
I = #idx{
type = <<"text">>,
name = Name,
def = Def
},
% TODO: Validate the index definition
[I]
end, Texts);
_ ->
[]
end.
to_json(Idx) ->
{[
{ddoc, Idx#idx.ddoc},
{name, Idx#idx.name},
{type, Idx#idx.type},
{def, {def_to_json(Idx#idx.def)}}
]}.
columns(Idx) ->
{Props} = Idx#idx.def,
{<<"fields">>, Fields} = lists:keyfind(<<"fields">>, 1, Props),
case Fields of
<<"all_fields">> ->
all_fields;
_ ->
{DFProps} = couch_util:get_value(<<"default_field">>, Props, {[]}),
Enabled = couch_util:get_value(<<"enabled">>, DFProps, true),
Default = case Enabled of
true -> [<<"$default">>];
false -> []
end,
Default ++ lists:map(fun({FProps}) ->
{_, Name} = lists:keyfind(<<"name">>, 1, FProps),
{_, Type} = lists:keyfind(<<"type">>, 1, FProps),
iolist_to_binary([Name, ":", Type])
end, Fields)
end.
is_usable(Idx, Selector) ->
case columns(Idx) of
all_fields ->
true;
Cols ->
Fields = indexable_fields(Selector),
sets:is_subset(sets:from_list(Fields), sets:from_list(Cols))
end.
do_validate({Props}) ->
{ok, Opts} = mango_opts:validate(Props, opts()),
{ok, {Opts}};
do_validate(Else) ->
?MANGO_ERROR({invalid_index_text, Else}).
def_to_json({Props}) ->
def_to_json(Props);
def_to_json([]) ->
[];
def_to_json([{<<"fields">>, <<"all_fields">>} | Rest]) ->
[{<<"fields">>, []} | def_to_json(Rest)];
def_to_json([{fields, Fields} | Rest]) ->
[{<<"fields">>, fields_to_json(Fields)} | def_to_json(Rest)];
def_to_json([{<<"fields">>, Fields} | Rest]) ->
[{<<"fields">>, fields_to_json(Fields)} | def_to_json(Rest)];
def_to_json([{Key, Value} | Rest]) ->
[{Key, Value} | def_to_json(Rest)].
fields_to_json([]) ->
[];
fields_to_json([{[{<<"name">>, Name}, {<<"type">>, Type0}]} | Rest]) ->
Type = validate_field_type(Type0),
[{[{Name, Type}]} | fields_to_json(Rest)];
fields_to_json([{[{<<"type">>, Type0}, {<<"name">>, Name}]} | Rest]) ->
Type = validate_field_type(Type0),
[{[{Name, Type}]} | fields_to_json(Rest)].
validate_field_type(<<"string">>) ->
<<"string">>;
validate_field_type(<<"number">>) ->
<<"number">>;
validate_field_type(<<"boolean">>) ->
<<"boolean">>.
validate_fields(Fields) ->
try fields_to_json(Fields) of
_ ->
mango_fields:new(Fields)
catch error:function_clause ->
?MANGO_ERROR({invalid_index_fields_definition, Fields})
end.
opts() ->
[
{<<"default_analyzer">>, [
{tag, default_analyzer},
{optional, true},
{default, <<"keyword">>}
]},
{<<"default_field">>, [
{tag, default_field},
{optional, true},
{default, {[]}}
]},
{<<"selector">>, [
{tag, selector},
{optional, true},
{default, {[]}},
{validator, fun mango_opts:validate_selector/1}
]},
{<<"fields">>, [
{tag, fields},
{optional, true},
{default, []},
{validator, fun ?MODULE:validate_fields/1}
]},
{<<"index_array_lengths">>, [
{tag, index_array_lengths},
{optional, true},
{default, true},
{validator, fun mango_opts:is_boolean/1}
]}
].
make_text(Idx) ->
Text= {[
{<<"index">>, Idx#idx.def},
{<<"analyzer">>, construct_analyzer(Idx#idx.def)}
]},
{Idx#idx.name, Text}.
get_default_field_options(Props) ->
Default = couch_util:get_value(default_field, Props, {[]}),
case Default of
Bool when is_boolean(Bool) ->
{Bool, <<"standard">>};
{[]} ->
{true, <<"standard">>};
{Opts}->
Enabled = couch_util:get_value(<<"enabled">>, Opts, true),
Analyzer = couch_util:get_value(<<"analyzer">>, Opts,
<<"standard">>),
{Enabled, Analyzer}
end.
construct_analyzer({Props}) ->
DefaultAnalyzer = couch_util:get_value(default_analyzer, Props,
<<"keyword">>),
{DefaultField, DefaultFieldAnalyzer} = get_default_field_options(Props),
DefaultAnalyzerDef = case DefaultField of
true ->
[{<<"$default">>, DefaultFieldAnalyzer}];
_ ->
[]
end,
case DefaultAnalyzerDef of
[] ->
<<"keyword">>;
_ ->
{[
{<<"name">>, <<"perfield">>},
{<<"default">>, DefaultAnalyzer},
{<<"fields">>, {DefaultAnalyzerDef}}
]}
end.
indexable_fields(Selector) ->
TupleTree = mango_selector_text:convert([], Selector),
indexable_fields([], TupleTree).
indexable_fields(Fields, {op_and, Args}) when is_list(Args) ->
lists:foldl(fun(Arg, Fields0) -> indexable_fields(Fields0, Arg) end,
Fields, Args);
indexable_fields(Fields, {op_or, Args}) when is_list(Args) ->
lists:foldl(fun(Arg, Fields0) -> indexable_fields(Fields0, Arg) end,
Fields, Args);
indexable_fields(Fields, {op_not, {ExistsQuery, Arg}}) when is_tuple(Arg) ->
Fields0 = indexable_fields(Fields, ExistsQuery),
indexable_fields(Fields0, Arg);
indexable_fields(Fields, {op_insert, Arg}) when is_binary(Arg) ->
Fields;
indexable_fields(Fields, {op_field, {Name, _}}) ->
[iolist_to_binary(Name) | Fields];
%% In this particular case, the lucene index is doing a field_exists query
%% so it is looking at all sorts of combinations of field:* and field.*
%% We don't add the field because we cannot pre-determine what field will exist.
%% Hence we just return Fields and make it less restrictive.
indexable_fields(Fields, {op_fieldname, {_, _}}) ->
Fields;
%% Similar idea to op_fieldname but with fieldname:null
indexable_fields(Fields, {op_null, {_, _}}) ->
Fields;
indexable_fields(Fields, {op_default, _}) ->
[<<"$default">> | Fields].
================================================
FILE: src/mango_idx_view.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_idx_view).
-export([
validate/1,
add/2,
remove/2,
from_ddoc/1,
to_json/1,
is_usable/2,
columns/1,
start_key/1,
end_key/1,
indexable_fields/1,
field_ranges/1,
field_ranges/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-include("mango_idx.hrl").
validate(#idx{}=Idx) ->
{ok, Def} = do_validate(Idx#idx.def),
{ok, Idx#idx{def=Def}}.
add(#doc{body={Props0}}=DDoc, Idx) ->
Views1 = case proplists:get_value(<<"views">>, Props0) of
{Views0} -> Views0;
_ -> []
end,
NewView = make_view(Idx),
Views2 = lists:keystore(element(1, NewView), 1, Views1, NewView),
Props1 = lists:keystore(<<"views">>, 1, Props0, {<<"views">>, {Views2}}),
{ok, DDoc#doc{body={Props1}}}.
remove(#doc{body={Props0}}=DDoc, Idx) ->
Views1 = case proplists:get_value(<<"views">>, Props0) of
{Views0} ->
Views0;
_ ->
?MANGO_ERROR({index_not_found, Idx#idx.name})
end,
Views2 = lists:keydelete(Idx#idx.name, 1, Views1),
if Views2 /= Views1 -> ok; true ->
?MANGO_ERROR({index_not_found, Idx#idx.name})
end,
Props1 = case Views2 of
[] ->
lists:keydelete(<<"views">>, 1, Props0);
_ ->
lists:keystore(<<"views">>, 1, Props0, {<<"views">>, {Views2}})
end,
{ok, DDoc#doc{body={Props1}}}.
from_ddoc({Props}) ->
case lists:keyfind(<<"views">>, 1, Props) of
{<<"views">>, {Views}} when is_list(Views) ->
lists:flatmap(fun({Name, {VProps}}) ->
Def = proplists:get_value(<<"map">>, VProps),
{Opts0} = proplists:get_value(<<"options">>, VProps),
Opts = lists:keydelete(<<"sort">>, 1, Opts0),
I = #idx{
type = <<"json">>,
name = Name,
def = Def,
opts = Opts
},
% TODO: Validate the index definition
[I]
end, Views);
_ ->
[]
end.
to_json(Idx) ->
{[
{ddoc, Idx#idx.ddoc},
{name, Idx#idx.name},
{type, Idx#idx.type},
{def, {def_to_json(Idx#idx.def)}}
]}.
columns(Idx) ->
{Props} = Idx#idx.def,
{<<"fields">>, {Fields}} = lists:keyfind(<<"fields">>, 1, Props),
[Key || {Key, _} <- Fields].
is_usable(Idx, Selector) ->
% This index is usable if at least the first column is
% a member of the indexable fields of the selector.
Columns = columns(Idx),
Fields = indexable_fields(Selector),
lists:member(hd(Columns), Fields) and not is_text_search(Selector).
is_text_search({[]}) ->
false;
is_text_search({[{<<"$default">>, _}]}) ->
true;
is_text_search({[{_Field, Cond}]}) when is_list(Cond) ->
lists:foldl(fun(C, Exists) ->
Exists or is_text_search(C)
end, false, Cond);
is_text_search({[{_Field, Cond}]}) when is_tuple(Cond) ->
is_text_search(Cond);
is_text_search({[{_Field, _Cond}]}) ->
false;
%% we reached values, which should always be false
is_text_search(Val)
when is_number(Val); is_boolean(Val); is_binary(Val)->
false.
start_key([]) ->
[];
start_key([{'$gt', Key, _, _} | Rest]) ->
case mango_json:special(Key) of
true ->
[];
false ->
[Key | start_key(Rest)]
end;
start_key([{'$gte', Key, _, _} | Rest]) ->
false = mango_json:special(Key),
[Key | start_key(Rest)];
start_key([{'$eq', Key, '$eq', Key} | Rest]) ->
false = mango_json:special(Key),
[Key | start_key(Rest)].
end_key([]) ->
[{[]}];
end_key([{_, _, '$lt', Key} | Rest]) ->
case mango_json:special(Key) of
true ->
[{[]}];
false ->
[Key | end_key(Rest)]
end;
end_key([{_, _, '$lte', Key} | Rest]) ->
false = mango_json:special(Key),
[Key | end_key(Rest)];
end_key([{'$eq', Key, '$eq', Key} | Rest]) ->
false = mango_json:special(Key),
[Key | end_key(Rest)].
do_validate({Props}) ->
{ok, Opts} = mango_opts:validate(Props, opts()),
{ok, {Opts}};
do_validate(Else) ->
?MANGO_ERROR({invalid_index_json, Else}).
def_to_json({Props}) ->
def_to_json(Props);
def_to_json([]) ->
[];
def_to_json([{fields, Fields} | Rest]) ->
[{<<"fields">>, mango_sort:to_json(Fields)} | def_to_json(Rest)];
def_to_json([{<<"fields">>, Fields} | Rest]) ->
[{<<"fields">>, mango_sort:to_json(Fields)} | def_to_json(Rest)];
def_to_json([{Key, Value} | Rest]) ->
[{Key, Value} | def_to_json(Rest)].
opts() ->
[
{<<"fields">>, [
{tag, fields},
{validator, fun mango_opts:validate_sort/1}
]}
].
make_view(Idx) ->
View = {[
{<<"map">>, Idx#idx.def},
{<<"reduce">>, <<"_count">>},
{<<"options">>, {Idx#idx.opts}}
]},
{Idx#idx.name, View}.
% This function returns a list of indexes that
% can be used to restrict this query. This works by
% searching the selector looking for field names that
% can be "seen".
%
% Operators that can be seen through are '$and' and any of
% the logical comparisons ('$lt', '$eq', etc). Things like
% '$regex', '$in', '$nin', and '$or' can't be serviced by
% a single index scan so we disallow them. In the future
% we may become more clever and increase our ken such that
% we will be able to see through these with crafty indexes
% or new uses for existing indexes. For instance, I could
% see an '$or' between comparisons on the same field becoming
% the equivalent of a multi-query. But that's for another
% day.
% We can see through '$and' trivially
indexable_fields({[{<<"$and">>, Args}]}) ->
lists:usort(lists:flatten([indexable_fields(A) || A <- Args]));
% So far we can't see through any other operator
indexable_fields({[{<<"$", _/binary>>, _}]}) ->
[];
% If we have a field with a terminator that is locatable
% using an index then the field is a possible index
indexable_fields({[{Field, Cond}]}) ->
case indexable(Cond) of
true ->
[Field];
false ->
[]
end;
% An empty selector
indexable_fields({[]}) ->
[].
% Check if a condition is indexable. The logical
% comparisons are mostly straight forward. We
% currently don't understand '$in' which is
% theoretically supportable. '$nin' and '$ne'
% aren't currently supported because they require
% multiple index scans.
indexable({[{<<"$lt">>, _}]}) ->
true;
indexable({[{<<"$lte">>, _}]}) ->
true;
indexable({[{<<"$eq">>, _}]}) ->
true;
indexable({[{<<"$gt">>, _}]}) ->
true;
indexable({[{<<"$gte">>, _}]}) ->
true;
% All other operators are currently not indexable.
% This is also a subtle assertion that we don't
% call indexable/1 on a field name.
indexable({[{<<"$", _/binary>>, _}]}) ->
false.
% For each field, return {Field, Range}
field_ranges(Selector) ->
Fields = indexable_fields(Selector),
field_ranges(Selector, Fields).
field_ranges(Selector, Fields) ->
field_ranges(Selector, Fields, []).
field_ranges(_Selector, [], Acc) ->
lists:reverse(Acc);
field_ranges(Selector, [Field | Rest], Acc) ->
case range(Selector, Field) of
empty ->
[{Field, empty}];
Range ->
field_ranges(Selector, Rest, [{Field, Range} | Acc])
end.
% Find the complete range for a given index in this
% selector. This works by AND'ing logical comparisons
% together so that we can define the start and end
% keys for a given index.
%
% Selector must have been normalized before calling
% this function.
range(Selector, Index) ->
range(Selector, Index, '$gt', mango_json:min(), '$lt', mango_json:max()).
% Adjust Low and High based on values found for the
% givend Index in Selector.
range({[{<<"$and">>, Args}]}, Index, LCmp, Low, HCmp, High) ->
lists:foldl(fun
(Arg, {LC, L, HC, H}) ->
range(Arg, Index, LC, L, HC, H);
(_Arg, empty) ->
empty
end, {LCmp, Low, HCmp, High}, Args);
% We can currently only traverse '$and' operators
range({[{<<"$", _/binary>>}]}, _Index, LCmp, Low, HCmp, High) ->
{LCmp, Low, HCmp, High};
% If the field name matches the index see if we can narrow
% the acceptable range.
range({[{Index, Cond}]}, Index, LCmp, Low, HCmp, High) ->
range(Cond, LCmp, Low, HCmp, High);
% Else we have a field unrelated to this index so just
% return the current values.
range(_, _, LCmp, Low, HCmp, High) ->
{LCmp, Low, HCmp, High}.
% The comments below are a bit cryptic at first but they show
% where the Arg cand land in the current range.
%
% For instance, given:
%
% {$lt: N}
% Low = 1
% High = 5
%
% Depending on the value of N we can have one of five locations
% in regards to a given Low/High pair:
%
% min low mid high max
%
% That is:
% min = (N < Low)
% low = (N == Low)
% mid = (Low < N < High)
% high = (N == High)
% max = (High < N)
%
% If N < 1, (min) then the effective range is empty.
%
% If N == 1, (low) then we have to set the range to empty because
% N < 1 && N >= 1 is an empty set. If the operator had been '$lte'
% and LCmp was '$gte' or '$eq' then we could keep around the equality
% check on Arg by setting LCmp == HCmp = '$eq' and Low == High == Arg.
%
% If 1 < N < 5 (mid), then we set High to Arg and Arg has just
% narrowed our range. HCmp is set the the '$lt' operator that was
% part of the input.
%
% If N == 5 (high), We just set HCmp to '$lt' since its guaranteed
% to be equally or more restrictive than the current possible values
% of '$lt' or '$lte'.
%
% If N > 5 (max), nothing changes as our current range is already
% more narrow than the current condition.
%
% Obviously all of that logic gets tweaked for the other logical
% operators but its all straight forward once you figure out how
% we're basically just narrowing our logical ranges.
range({[{<<"$lt">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
empty;
low ->
empty;
mid ->
{LCmp, Low, '$lt', Arg};
high ->
{LCmp, Low, '$lt', Arg};
max ->
{LCmp, Low, HCmp, High}
end;
range({[{<<"$lte">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
empty;
low when LCmp == '$gte'; LCmp == '$eq' ->
{'$eq', Arg, '$eq', Arg};
low ->
empty;
mid ->
{LCmp, Low, '$lte', Arg};
high ->
{LCmp, Low, HCmp, High};
max ->
{LCmp, Low, HCmp, High}
end;
range({[{<<"$eq">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
empty;
low when LCmp == '$gte'; LCmp == '$eq' ->
{'$eq', Arg, '$eq', Arg};
low ->
empty;
mid ->
{'$eq', Arg, '$eq', Arg};
high when HCmp == '$lte'; HCmp == '$eq' ->
{'$eq', Arg, '$eq', Arg};
high ->
empty;
max ->
empty
end;
range({[{<<"$gte">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
{LCmp, Low, HCmp, High};
low ->
{LCmp, Low, HCmp, High};
mid ->
{'$gte', Arg, HCmp, High};
high when HCmp == '$lte'; HCmp == '$eq' ->
{'$eq', Arg, '$eq', Arg};
high ->
empty;
max ->
empty
end;
range({[{<<"$gt">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
{LCmp, Low, HCmp, High};
low ->
{'$gt', Arg, HCmp, High};
mid ->
{'$gt', Arg, HCmp, High};
high ->
empty;
max ->
empty
end;
% There's some other un-indexable restriction on the index
% that will be applied as a post-filter. Ignore it and
% carry on our merry way.
range({[{<<"$", _/binary>>, _}]}, LCmp, Low, HCmp, High) ->
{LCmp, Low, HCmp, High}.
% Returns the value min | low | mid | high | max depending
% on how Arg compares to Low and High.
range_pos(Low, Arg, High) ->
case mango_json:cmp(Arg, Low) of
N when N < 0 -> min;
N when N == 0 -> low;
_ ->
case mango_json:cmp(Arg, High) of
X when X < 0 ->
mid;
X when X == 0 ->
high;
_ ->
max
end
end.
================================================
FILE: src/mango_json.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_json).
-export([
min/0,
max/0,
cmp/2,
cmp_raw/2,
type/1,
special/1,
to_binary/1
]).
-define(MIN_VAL, mango_json_min).
-define(MAX_VAL, mango_json_max).
min() ->
?MIN_VAL.
max() ->
?MAX_VAL.
cmp(?MIN_VAL, ?MIN_VAL) ->
0;
cmp(?MIN_VAL, _) ->
-1;
cmp(_, ?MIN_VAL) ->
1;
cmp(?MAX_VAL, ?MAX_VAL) ->
0;
cmp(?MAX_VAL, _) ->
1;
cmp(_, ?MAX_VAL) ->
-1;
cmp(A, B) ->
couch_view:cmp_json(A, B).
cmp_raw(?MIN_VAL, ?MIN_VAL) ->
0;
cmp_raw(?MIN_VAL, _) ->
-1;
cmp_raw(_, ?MIN_VAL) ->
1;
cmp_raw(?MAX_VAL, ?MAX_VAL) ->
0;
cmp_raw(?MAX_VAL, _) ->
1;
cmp_raw(_, ?MAX_VAL) ->
-1;
cmp_raw(A, B) ->
case A < B of
true ->
-1;
false ->
case A > B of
true ->
1;
false ->
0
end
end.
type(null) ->
<<"null">>;
type(Bool) when is_boolean(Bool) ->
<<"boolean">>;
type(Num) when is_number(Num) ->
<<"number">>;
type(Str) when is_binary(Str) ->
<<"string">>;
type({Props}) when is_list(Props) ->
<<"object">>;
type(Vals) when is_list(Vals) ->
<<"array">>.
special(?MIN_VAL) ->
true;
special(?MAX_VAL) ->
true;
special(_) ->
false.
to_binary({Props}) ->
Pred = fun({Key, Value}) ->
{to_binary(Key), to_binary(Value)}
end,
{lists:map(Pred, Props)};
to_binary(Data) when is_list(Data) ->
[to_binary(D) || D <- Data];
to_binary(null) ->
null;
to_binary(true) ->
true;
to_binary(false) ->
false;
to_binary(Data) when is_atom(Data) ->
list_to_binary(atom_to_list(Data));
to_binary(Data) when is_number(Data) ->
Data;
to_binary(Data) when is_binary(Data) ->
Data.
================================================
FILE: src/mango_native_proc.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_native_proc).
-behavior(gen_server).
-export([
start_link/0,
set_timeout/2,
prompt/2
]).
-export([
init/1,
terminate/2,
handle_call/3,
handle_cast/2,
handle_info/2,
code_change/3
]).
-record(st, {
indexes = [],
timeout = 5000
}).
-record(tacc, {
index_array_lengths = true,
fields = all_fields,
path = []
}).
start_link() ->
gen_server:start_link(?MODULE, [], []).
set_timeout(Pid, TimeOut) when is_integer(TimeOut), TimeOut > 0 ->
gen_server:call(Pid, {set_timeout, TimeOut}).
prompt(Pid, Data) ->
gen_server:call(Pid, {prompt, Data}).
init(_) ->
{ok, #st{}}.
terminate(_Reason, _St) ->
ok.
handle_call({set_timeout, TimeOut}, _From, St) ->
{reply, ok, St#st{timeout=TimeOut}};
handle_call({prompt, [<<"reset">>]}, _From, St) ->
{reply, true, St#st{indexes=[]}};
handle_call({prompt, [<<"reset">>, _QueryConfig]}, _From, St) ->
{reply, true, St#st{indexes=[]}};
handle_call({prompt, [<<"add_fun">>, IndexInfo]}, _From, St) ->
Indexes = St#st.indexes ++ [IndexInfo],
NewSt = St#st{indexes = Indexes},
{reply, true, NewSt};
handle_call({prompt, [<<"map_doc">>, Doc]}, _From, St) ->
{reply, map_doc(St, mango_json:to_binary(Doc)), St};
handle_call({prompt, [<<"reduce">>, _, _]}, _From, St) ->
{reply, null, St};
handle_call({prompt, [<<"rereduce">>, _, _]}, _From, St) ->
{reply, null, St};
handle_call({prompt, [<<"index_doc">>, Doc]}, _From, St) ->
{reply, index_doc(St, mango_json:to_binary(Doc)), St};
handle_call(Msg, _From, St) ->
{stop, {invalid_call, Msg}, {invalid_call, Msg}, St}.
handle_cast(garbage_collect, St) ->
erlang:garbage_collect(),
{noreply, St};
handle_cast(Msg, St) ->
{stop, {invalid_cast, Msg}, St}.
handle_info(Msg, St) ->
{stop, {invalid_info, Msg}, St}.
code_change(_OldVsn, St, _Extra) ->
{ok, St}.
map_doc(#st{indexes=Indexes}, Doc) ->
lists:map(fun(Idx) -> get_index_entries(Idx, Doc) end, Indexes).
index_doc(#st{indexes=Indexes}, Doc) ->
lists:map(fun(Idx) -> get_text_entries(Idx, Doc) end, Indexes).
get_index_entries({IdxProps}, Doc) ->
{Fields} = couch_util:get_value(<<"fields">>, IdxProps),
Values = lists:map(fun({Field, _Dir}) ->
case mango_doc:get_field(Doc, Field) of
not_found -> not_found;
bad_path -> not_found;
Else -> Else
end
end, Fields),
case lists:member(not_found, Values) of
true ->
[];
false ->
[[Values, null]]
end.
get_text_entries({IdxProps}, Doc) ->
Selector = case couch_util:get_value(<<"selector">>, IdxProps) of
[] -> {[]};
Else -> Else
end,
case should_index(Selector, Doc) of
true ->
get_text_entries0(IdxProps, Doc);
false ->
[]
end.
get_text_entries0(IdxProps, Doc) ->
DefaultEnabled = get_default_enabled(IdxProps),
IndexArrayLengths = get_index_array_lengths(IdxProps),
FieldsList = get_text_field_list(IdxProps),
TAcc = #tacc{
index_array_lengths = IndexArrayLengths,
fields = FieldsList
},
Fields0 = get_text_field_values(Doc, TAcc),
Fields = if not DefaultEnabled -> Fields0; true ->
add_default_text_field(Fields0)
end,
FieldNames = get_field_names(Fields, []),
Converted = convert_text_fields(Fields),
FieldNames ++ Converted.
get_text_field_values({Props}, TAcc) when is_list(Props) ->
get_text_field_values_obj(Props, TAcc, []);
get_text_field_values(Values, TAcc) when is_list(Values) ->
IndexArrayLengths = TAcc#tacc.index_array_lengths,
NewPath = ["[]" | TAcc#tacc.path],
NewTAcc = TAcc#tacc{path = NewPath},
case IndexArrayLengths of
true ->
% We bypass make_text_field and directly call make_text_field_name
% because the length field name is not part of the path.
LengthFieldName = make_text_field_name(NewTAcc#tacc.path, <<"length">>),
LengthField = [{LengthFieldName, <<"length">>, length(Values)}],
get_text_field_values_arr(Values, NewTAcc, LengthField);
_ ->
get_text_field_values_arr(Values, NewTAcc, [])
end;
get_text_field_values(Bin, TAcc) when is_binary(Bin) ->
make_text_field(TAcc, <<"string">>, Bin);
get_text_field_values(Num, TAcc) when is_number(Num) ->
make_text_field(TAcc, <<"number">>, Num);
get_text_field_values(Bool, TAcc) when is_boolean(Bool) ->
make_text_field(TAcc, <<"boolean">>, Bool);
get_text_field_values(null, TAcc) ->
make_text_field(TAcc, <<"null">>, true).
get_text_field_values_obj([], _, FAcc) ->
FAcc;
get_text_field_values_obj([{Key, Val} | Rest], TAcc, FAcc) ->
NewPath = [Key | TAcc#tacc.path],
NewTAcc = TAcc#tacc{path = NewPath},
Fields = get_text_field_values(Val, NewTAcc),
get_text_field_values_obj(Rest, TAcc, Fields ++ FAcc).
get_text_field_values_arr([], _, FAcc) ->
FAcc;
get_text_field_values_arr([Value | Rest], TAcc, FAcc) ->
Fields = get_text_field_values(Value, TAcc),
get_text_field_values_arr(Rest, TAcc, Fields ++ FAcc).
get_default_enabled(Props) ->
case couch_util:get_value(<<"default_field">>, Props, {[]}) of
Bool when is_boolean(Bool) ->
Bool;
{[]} ->
true;
{Opts}->
couch_util:get_value(<<"enabled">>, Opts, true)
end.
get_index_array_lengths(Props) ->
couch_util:get_value(<<"index_array_lengths">>, Props, true).
add_default_text_field(Fields) ->
DefaultFields = add_default_text_field(Fields, []),
DefaultFields ++ Fields.
add_default_text_field([], Acc) ->
Acc;
add_default_text_field([{_Name, <<"string">>, Value} | Rest], Acc) ->
NewAcc = [{<<"$default">>, <<"string">>, Value} | Acc],
add_default_text_field(Rest, NewAcc);
add_default_text_field([_ | Rest], Acc) ->
add_default_text_field(Rest, Acc).
%% index of all field names
get_field_names([], FAcc) ->
FAcc;
get_field_names([{Name, _Type, _Value} | Rest], FAcc) ->
case lists:member([<<"$fieldnames">>, Name, []], FAcc) of
true ->
get_field_names(Rest, FAcc);
false ->
get_field_names(Rest, [[<<"$fieldnames">>, Name, []] | FAcc])
end.
convert_text_fields([]) ->
[];
convert_text_fields([{Name, _Type, Value} | Rest]) ->
[[Name, Value, []] | convert_text_fields(Rest)].
should_index(Selector, Doc) ->
% We should do this
NormSelector = mango_selector:normalize(Selector),
Matches = mango_selector:match(NormSelector, Doc),
IsDesign = case mango_doc:get_field(Doc, <<"_id">>) of
<<"_design/", _/binary>> -> true;
_ -> false
end,
Matches and not IsDesign.
get_text_field_list(IdxProps) ->
case couch_util:get_value(<<"fields">>, IdxProps) of
Fields when is_list(Fields) ->
RawList = lists:flatmap(fun get_text_field_info/1, Fields),
[mango_util:lucene_escape_user(Field) || Field <- RawList];
_ ->
all_fields
end.
get_text_field_info({Props}) ->
Name = couch_util:get_value(<<"name">>, Props),
Type0 = couch_util:get_value(<<"type">>, Props),
if not is_binary(Name) -> []; true ->
Type = get_text_field_type(Type0),
[iolist_to_binary([Name, ":", Type])]
end.
get_text_field_type(<<"number">>) ->
<<"number">>;
get_text_field_type(<<"boolean">>) ->
<<"boolean">>;
get_text_field_type(_) ->
<<"string">>.
make_text_field(TAcc, Type, Value) ->
FieldName = make_text_field_name(TAcc#tacc.path, Type),
Fields = TAcc#tacc.fields,
case Fields == all_fields orelse lists:member(FieldName, Fields) of
true ->
[{FieldName, Type, Value}];
false ->
[]
end.
make_text_field_name([P | Rest], Type) ->
Parts = lists:reverse(Rest, [iolist_to_binary([P, ":", Type])]),
Escaped = [mango_util:lucene_escape_field(N) || N <- Parts],
iolist_to_binary(mango_util:join(".", Escaped)).
================================================
FILE: src/mango_opts.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_opts).
-export([
validate_idx_create/1,
validate_find/1,
validate_bulk_delete/1
]).
-export([
validate/2,
is_string/1,
is_boolean/1,
is_pos_integer/1,
is_non_neg_integer/1,
is_object/1,
validate_idx_name/1,
validate_selector/1,
validate_use_index/1,
validate_bookmark/1,
validate_sort/1,
validate_fields/1
]).
-include("mango.hrl").
validate_idx_create({Props}) ->
Opts = [
{<<"index">>, [
{tag, def}
]},
{<<"type">>, [
{tag, type},
{optional, true},
{default, <<"json">>},
{validator, fun is_string/1}
]},
{<<"name">>, [
{tag, name},
{optional, true},
{default, auto_name},
{validator, fun validate_idx_name/1}
]},
{<<"ddoc">>, [
{tag, ddoc},
{optional, true},
{default, auto_name},
{validator, fun validate_idx_name/1}
]},
{<<"w">>, [
{tag, w},
{optional, true},
{default, 2},
{validator, fun is_pos_integer/1}
]}
],
validate(Props, Opts).
validate_find({Props}) ->
Opts = [
{<<"selector">>, [
{tag, selector},
{validator, fun validate_selector/1}
]},
{<<"use_index">>, [
{tag, use_index},
{optional, true},
{default, []},
{validator, fun validate_use_index/1}
]},
{<<"bookmark">>, [
{tag, bookmark},
{optional, true},
{default, <<>>},
{validator, fun validate_bookmark/1}
]},
{<<"limit">>, [
{tag, limit},
{optional, true},
{default, 10000000000},
{validator, fun is_non_neg_integer/1}
]},
{<<"skip">>, [
{tag, skip},
{optional, true},
{default, 0},
{validator, fun is_non_neg_integer/1}
]},
{<<"sort">>, [
{tag, sort},
{optional, true},
{default, []},
{validator, fun validate_sort/1}
]},
{<<"fields">>, [
{tag, fields},
{optional, true},
{default, []},
{validator, fun validate_fields/1}
]},
{<<"r">>, [
{tag, r},
{optional, true},
{default, 1},
{validator, fun mango_opts:is_pos_integer/1}
]},
{<<"conflicts">>, [
{tag, conflicts},
{optional, true},
{default, false},
{validator, fun mango_opts:is_boolean/1}
]}
],
validate(Props, Opts).
validate_bulk_delete({Props}) ->
Opts = [
{<<"docids">>, [
{tag, docids},
{validator, fun validate_bulk_docs/1}
]},
{<<"w">>, [
{tag, w},
{optional, true},
{default, 2},
{validator, fun is_pos_integer/1}
]}
],
validate(Props, Opts).
validate(Props, Opts) ->
case mango_util:assert_ejson({Props}) of
true ->
ok;
false ->
?MANGO_ERROR({invalid_ejson, {Props}})
end,
{Rest, Acc} = validate_opts(Opts, Props, []),
case Rest of
[] ->
ok;
[{BadKey, _} | _] ->
?MANGO_ERROR({invalid_key, BadKey})
end,
{ok, Acc}.
is_string(Val) when is_binary(Val) ->
{ok, Val};
is_string(Else) ->
?MANGO_ERROR({invalid_string, Else}).
is_boolean(true) ->
{ok, true};
is_boolean(false) ->
{ok, false};
is_boolean(Else) ->
?MANGO_ERROR({invalid_boolean, Else}).
is_pos_integer(V) when is_integer(V), V > 0 ->
{ok, V};
is_pos_integer(Else) ->
?MANGO_ERROR({invalid_pos_integer, Else}).
is_non_neg_integer(V) when is_integer(V), V >= 0 ->
{ok, V};
is_non_neg_integer(Else) ->
?MANGO_ERROR({invalid_non_neg_integer, Else}).
is_object({Props}) ->
true = mango_util:assert_ejson({Props}),
{ok, {Props}};
is_object(Else) ->
?MANGO_ERROR({invalid_object, Else}).
validate_idx_name(auto_name) ->
{ok, auto_name};
validate_idx_name(Else) ->
is_string(Else).
validate_selector({Props}) ->
Norm = mango_selector:normalize({Props}),
{ok, Norm};
validate_selector(Else) ->
?MANGO_ERROR({invalid_selector_json, Else}).
validate_bulk_docs(Docs) when is_list(Docs) ->
lists:foreach(fun ?MODULE:is_string/1, Docs),
{ok, Docs};
validate_bulk_docs(Else) ->
?MANGO_ERROR({invalid_bulk_docs, Else}).
validate_use_index(IndexName) when is_binary(IndexName) ->
case binary:split(IndexName, <<"/">>) of
[DesignId] ->
{ok, [DesignId]};
[<<"_design">>, DesignId] ->
{ok, [DesignId]};
[DesignId, ViewName] ->
{ok, [DesignId, ViewName]};
[<<"_design">>, DesignId, ViewName] ->
{ok, [DesignId, ViewName]};
_ ->
?MANGO_ERROR({invalid_index_name, IndexName})
end;
validate_use_index(null) ->
{ok, []};
validate_use_index([]) ->
{ok, []};
validate_use_index([DesignId]) when is_binary(DesignId) ->
{ok, [DesignId]};
validate_use_index([DesignId, ViewName])
when is_binary(DesignId), is_binary(ViewName) ->
{ok, [DesignId, ViewName]};
validate_use_index(Else) ->
?MANGO_ERROR({invalid_index_name, Else}).
validate_bookmark(null) ->
{ok, nil};
validate_bookmark(<<>>) ->
{ok, nil};
validate_bookmark(Bin) when is_binary(Bin) ->
{ok, Bin};
validate_bookmark(Else) ->
?MANGO_ERROR({invalid_bookmark, Else}).
validate_sort(Value) ->
mango_sort:new(Value).
validate_fields(Value) ->
mango_fields:new(Value).
validate_opts([], Props, Acc) ->
{Props, lists:reverse(Acc)};
validate_opts([{Name, Desc} | Rest], Props, Acc) ->
{tag, Tag} = lists:keyfind(tag, 1, Desc),
case lists:keytake(Name, 1, Props) of
{value, {Name, Prop}, RestProps} ->
NewAcc = [{Tag, validate_opt(Name, Desc, Prop)} | Acc],
validate_opts(Rest, RestProps, NewAcc);
false ->
NewAcc = [{Tag, validate_opt(Name, Desc, undefined)} | Acc],
validate_opts(Rest, Props, NewAcc)
end.
validate_opt(_Name, [], Value) ->
Value;
validate_opt(Name, Desc0, undefined) ->
case lists:keytake(optional, 1, Desc0) of
{value, {optional, true}, Desc1} ->
{value, {default, Value}, Desc2} = lists:keytake(default, 1, Desc1),
false = (Value == undefined),
validate_opt(Name, Desc2, Value);
_ ->
?MANGO_ERROR({missing_required_key, Name})
end;
validate_opt(Name, [{tag, _} | Rest], Value) ->
% Tags aren't really validated
validate_opt(Name, Rest, Value);
validate_opt(Name, [{optional, _} | Rest], Value) ->
% A value was specified for an optional value
validate_opt(Name, Rest, Value);
validate_opt(Name, [{default, _} | Rest], Value) ->
% A value was specified for an optional value
validate_opt(Name, Rest, Value);
validate_opt(Name, [{assert, Value} | Rest], Value) ->
validate_opt(Name, Rest, Value);
validate_opt(Name, [{assert, Expect} | _], Found) ->
?MANGO_ERROR({invalid_value, Name, Expect, Found});
validate_opt(Name, [{validator, Fun} | Rest], Value) ->
case Fun(Value) of
{ok, Validated} ->
validate_opt(Name, Rest, Validated);
false ->
?MANGO_ERROR({invalid_value, Name, Value})
end.
================================================
FILE: src/mango_selector.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_selector).
-export([
normalize/1,
match/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
% Validate and normalize each operator. This translates
% every selector operator into a consistent version that
% we can then rely on for all other selector functions.
% See the definition of each step below for more information
% on what each one does.
normalize({[]}) ->
{[]};
normalize(Selector) ->
Steps = [
fun norm_ops/1,
fun norm_fields/1,
fun norm_negations/1
],
{NProps} = lists:foldl(fun(Step, Sel) -> Step(Sel) end, Selector, Steps),
FieldNames = [Name || {Name, _} <- NProps],
case lists:member(<<>>, FieldNames) of
true ->
?MANGO_ERROR({invalid_selector, missing_field_name});
false ->
ok
end,
{NProps}.
% Match a selector against a #doc{} or EJSON value.
% This assumes that the Selector has been normalized.
% Returns true or false.
% An empty selector matches any value.
match({[]}, _) ->
true;
match(Selector, #doc{body=Body}) ->
match(Selector, Body, fun mango_json:cmp/2);
match(Selector, {Props}) ->
match(Selector, {Props}, fun mango_json:cmp/2).
% Convert each operator into a normalized version as well
% as convert an implict operators into their explicit
% versions.
norm_ops({[{<<"$and">>, Args}]}) when is_list(Args) ->
{[{<<"$and">>, [norm_ops(A) || A <- Args]}]};
norm_ops({[{<<"$and">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$and', Arg});
norm_ops({[{<<"$or">>, Args}]}) when is_list(Args) ->
{[{<<"$or">>, [norm_ops(A) || A <- Args]}]};
norm_ops({[{<<"$or">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$or', Arg});
norm_ops({[{<<"$not">>, {_}=Arg}]}) ->
{[{<<"$not">>, norm_ops(Arg)}]};
norm_ops({[{<<"$not">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$not', Arg});
norm_ops({[{<<"$nor">>, Args}]}) when is_list(Args) ->
{[{<<"$nor">>, [norm_ops(A) || A <- Args]}]};
norm_ops({[{<<"$nor">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$nor', Arg});
norm_ops({[{<<"$in">>, Args}]} = Cond) when is_list(Args) ->
Cond;
norm_ops({[{<<"$in">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$in', Arg});
norm_ops({[{<<"$nin">>, Args}]} = Cond) when is_list(Args) ->
Cond;
norm_ops({[{<<"$nin">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$nin', Arg});
norm_ops({[{<<"$exists">>, Arg}]} = Cond) when is_boolean(Arg) ->
Cond;
norm_ops({[{<<"$exists">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$exists', Arg});
norm_ops({[{<<"$type">>, Arg}]} = Cond) when is_binary(Arg) ->
Cond;
norm_ops({[{<<"$type">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$type', Arg});
norm_ops({[{<<"$mod">>, [D, R]}]} = Cond) when is_integer(D), is_integer(R) ->
Cond;
norm_ops({[{<<"$mod">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$mod', Arg});
norm_ops({[{<<"$regex">>, Regex}]} = Cond) when is_binary(Regex) ->
case re:compile(Regex) of
{ok, _} ->
Cond;
_ ->
?MANGO_ERROR({bad_arg, '$regex', Regex})
end;
norm_ops({[{<<"$all">>, Args}]}) when is_list(Args) ->
{[{<<"$all">>, Args}]};
norm_ops({[{<<"$all">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$all', Arg});
norm_ops({[{<<"$elemMatch">>, {_}=Arg}]}) ->
{[{<<"$elemMatch">>, norm_ops(Arg)}]};
norm_ops({[{<<"$elemMatch">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$elemMatch', Arg});
norm_ops({[{<<"$size">>, Arg}]}) when is_integer(Arg), Arg >= 0 ->
{[{<<"$size">>, Arg}]};
norm_ops({[{<<"$size">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$size', Arg});
norm_ops({[{<<"$text">>, Arg}]}) when is_binary(Arg); is_number(Arg);
is_boolean(Arg) ->
{[{<<"$default">>, {[{<<"$text">>, Arg}]}}]};
norm_ops({[{<<"$text">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$text', Arg});
% Not technically an operator but we pass it through here
% so that this function accepts its own output. This exists
% so that $text can have a field name value which simplifies
% logic elsewhere.
norm_ops({[{<<"$default">>, _}]} = Selector) ->
Selector;
% Terminals where we can't perform any validation
% on the value because any value is acceptable.
norm_ops({[{<<"$lt">>, _}]} = Cond) ->
Cond;
norm_ops({[{<<"$lte">>, _}]} = Cond) ->
Cond;
norm_ops({[{<<"$eq">>, _}]} = Cond) ->
Cond;
norm_ops({[{<<"$ne">>, _}]} = Cond) ->
Cond;
norm_ops({[{<<"$gte">>, _}]} = Cond) ->
Cond;
norm_ops({[{<<"$gt">>, _}]} = Cond) ->
Cond;
% Known but unsupported operators
norm_ops({[{<<"$where">>, _}]}) ->
?MANGO_ERROR({not_supported, '$where'});
norm_ops({[{<<"$geoWithin">>, _}]}) ->
?MANGO_ERROR({not_supported, '$geoWithin'});
norm_ops({[{<<"$geoIntersects">>, _}]}) ->
?MANGO_ERROR({not_supported, '$geoIntersects'});
norm_ops({[{<<"$near">>, _}]}) ->
?MANGO_ERROR({not_supported, '$near'});
norm_ops({[{<<"$nearSphere">>, _}]}) ->
?MANGO_ERROR({not_supported, '$nearSphere'});
% Unknown operator
norm_ops({[{<<"$", _/binary>>=Op, _}]}) ->
?MANGO_ERROR({invalid_operator, Op});
% A {Field: Cond} pair
norm_ops({[{Field, Cond}]}) ->
{[{Field, norm_ops(Cond)}]};
% An implicit $and
norm_ops({Props}) when length(Props) > 1 ->
{[{<<"$and">>, [norm_ops({[P]}) || P <- Props]}]};
% A bare value condition means equality
norm_ops(Value) ->
{[{<<"$eq">>, Value}]}.
% This takes a selector and normalizes all of the
% field names as far as possible. For instance:
%
% Unnormalized:
% {foo: {$and: [{$gt: 5}, {$lt: 10}]}}
%
% Normalized:
% {$and: [{foo: {$gt: 5}}, {foo: {$lt: 10}}]}
%
% And another example:
%
% Unnormalized:
% {foo: {bar: {$gt: 10}}}
%
% Normalized:
% {"foo.bar": {$gt: 10}}
%
% Its important to note that we can only normalize
% field names like this through boolean operators where
% we can gaurantee commutativity. We can't necessarily
% do the same through the '$elemMatch' operators but we
% can apply the same algorithm to its arguments.
norm_fields({[]}) ->
{[]};
norm_fields(Selector) ->
norm_fields(Selector, <<>>).
% Operators where we can push the field names further
% down the operator tree
norm_fields({[{<<"$and">>, Args}]}, Path) ->
{[{<<"$and">>, [norm_fields(A, Path) || A <- Args]}]};
norm_fields({[{<<"$or">>, Args}]}, Path) ->
{[{<<"$or">>, [norm_fields(A, Path) || A <- Args]}]};
norm_fields({[{<<"$not">>, Arg}]}, Path) ->
{[{<<"$not">>, norm_fields(Arg, Path)}]};
norm_fields({[{<<"$nor">>, Args}]}, Path) ->
{[{<<"$nor">>, [norm_fields(A, Path) || A <- Args]}]};
% Fields where we can normalize fields in the
% operator arguments independently.
norm_fields({[{<<"$elemMatch">>, Arg}]}, Path) ->
Cond = {[{<<"$elemMatch">>, norm_fields(Arg)}]},
{[{Path, Cond}]};
% The text operator operates against the internal
% $default field. This also asserts that the $default
% field is at the root as well as that it only has
% a $text operator applied.
norm_fields({[{<<"$default">>, {[{<<"$text">>, _Arg}]}}]}=Sel, <<>>) ->
Sel;
norm_fields({[{<<"$default">>, _}]} = Selector, _) ->
?MANGO_ERROR({bad_field, Selector});
% Any other operator is a terminal below which no
% field names should exist. Set the path to this
% terminal and return it.
norm_fields({[{<<"$", _/binary>>, _}]} = Cond, Path) ->
{[{Path, Cond}]};
% We've found a field name. Append it to the path
% and skip this node as we unroll the stack as
% the full path will be further down the branch.
norm_fields({[{Field, Cond}]}, <<>>) ->
% Don't include the '.' for the first element of
% the path.
norm_fields(Cond, Field);
norm_fields({[{Field, Cond}]}, Path) ->
norm_fields(Cond, <<Path/binary, ".", Field/binary>>);
% An empty selector
norm_fields({[]}, Path) ->
{Path, {[]}};
% Else we have an invalid selector
norm_fields(BadSelector, _) ->
?MANGO_ERROR({bad_field, BadSelector}).
% Take all the negation operators and move the logic
% as far down the branch as possible. This does things
% like:
%
% Unnormalized:
% {$not: {foo: {$gt: 10}}}
%
% Normalized:
% {foo: {$lte: 10}}
%
% And we also apply DeMorgan's laws
%
% Unnormalized:
% {$not: {$and: [{foo: {$gt: 10}}, {foo: {$lt: 5}}]}}
%
% Normalized:
% {$or: [{foo: {$lte: 10}}, {foo: {$gte: 5}}]}
%
% This logic is important because we can't "see" through
% a '$not' operator to be able to locate indices that may
% service a specific query. Though if we move the negations
% down to the terminals we may be able to negate specific
% operators which allows us to find usable indices.
% Operators that cause a negation
norm_negations({[{<<"$not">>, Arg}]}) ->
negate(Arg);
norm_negations({[{<<"$nor">>, Args}]}) ->
{[{<<"$and">>, [negate(A) || A <- Args]}]};
% Operators that we merely seek through as we look for
% negations.
norm_negations({[{<<"$and">>, Args}]}) ->
{[{<<"$and">>, [norm_negations(A) || A <- Args]}]};
norm_negations({[{<<"$or">>, Args}]}) ->
{[{<<"$or">>, [norm_negations(A) || A <- Args]}]};
norm_negations({[{<<"$elemMatch">>, Arg}]}) ->
{[{<<"$elemMatch">>, norm_negations(Arg)}]};
% All other conditions can't introduce negations anywhere
% further down the operator tree.
norm_negations(Cond) ->
Cond.
% Actually negate an expression. Make sure and read up
% on DeMorgan's laws if you're trying to read this, but
% in a nutshell:
%
% NOT(a AND b) == NOT(a) OR NOT(b)
% NOT(a OR b) == NOT(a) AND NOT(b)
%
% Also notice that if a negation hits another negation
% operator that we just nullify the combination. Its
% possible that below the nullification we have more
% negations so we have to recurse back to norm_negations/1.
% Negating negation, nullify but recurse to
% norm_negations/1
negate({[{<<"$not">>, Arg}]}) ->
norm_negations(Arg);
negate({[{<<"$nor">>, Args}]}) ->
{[{<<"$or">>, [norm_negations(A) || A <- Args]}]};
% DeMorgan Negations
negate({[{<<"$and">>, Args}]}) ->
{[{<<"$or">>, [negate(A) || A <- Args]}]};
negate({[{<<"$or">>, Args}]}) ->
{[{<<"$and">>, [negate(A) || A <- Args]}]};
negate({[{<<"$default">>, _}]} = Arg) ->
?MANGO_ERROR({bad_arg, '$not', Arg});
% Negating comparison operators is straight forward
negate({[{<<"$lt">>, Arg}]}) ->
{[{<<"$gte">>, Arg}]};
negate({[{<<"$lte">>, Arg}]}) ->
{[{<<"$gt">>, Arg}]};
negate({[{<<"$eq">>, Arg}]}) ->
{[{<<"$ne">>, Arg}]};
negate({[{<<"$ne">>, Arg}]}) ->
{[{<<"$eq">>, Arg}]};
negate({[{<<"$gte">>, Arg}]}) ->
{[{<<"$lt">>, Arg}]};
negate({[{<<"$gt">>, Arg}]}) ->
{[{<<"$lte">>, Arg}]};
negate({[{<<"$in">>, Args}]}) ->
{[{<<"$nin">>, Args}]};
negate({[{<<"$nin">>, Args}]}) ->
{[{<<"$in">>, Args}]};
% We can also trivially negate the exists operator
negate({[{<<"$exists">>, Arg}]}) ->
{[{<<"$exists">>, not Arg}]};
% Anything else we have to just terminate the
% negation by reinserting the negation operator
negate({[{<<"$", _/binary>>, _}]} = Cond) ->
{[{<<"$not">>, Cond}]};
% Finally, negating a field just means we negate its
% condition.
negate({[{Field, Cond}]}) ->
{[{Field, negate(Cond)}]}.
match({[{<<"$and">>, Args}]}, Value, Cmp) ->
Pred = fun(SubSel) -> match(SubSel, Value, Cmp) end,
lists:all(Pred, Args);
match({[{<<"$or">>, Args}]}, Value, Cmp) ->
Pred = fun(SubSel) -> match(SubSel, Value, Cmp) end,
lists:any(Pred, Args);
match({[{<<"$not">>, Arg}]}, Value, Cmp) ->
not match(Arg, Value, Cmp);
% All of the values in Args must exist in Values or
% Values == hd(Args) if Args is a single element list
% that contains a list.
match({[{<<"$all">>, Args}]}, Values, _Cmp) when is_list(Values) ->
Pred = fun(A) -> lists:member(A, Values) end,
HasArgs = lists:all(Pred, Args),
IsArgs = case Args of
[A] when is_list(A) ->
A == Values;
_ ->
false
end,
HasArgs orelse IsArgs;
match({[{<<"$all">>, _Args}]}, _Values, _Cmp) ->
false;
%% This is for $elemMatch and possibly $in because of our normalizer.
%% A selector such as {"field_name": {"$elemMatch": {"$gte": 80, "$lt": 85}}}
%% gets normalized to:
%% {[{<<"field_name">>,
%% {[{<<"$elemMatch">>,
%% {[{<<"$and">>, [
%% {[{<<>>,{[{<<"$gte">>,80}]}}]},
%% {[{<<>>,{[{<<"$lt">>,85}]}}]}
%% ]}]}
%% }]}
%% }]}.
%% So we filter out the <<>>.
match({[{<<>>, Arg}]}, Values, Cmp) ->
match(Arg, Values, Cmp);
% Matches when any element in values matches the
% sub-selector Arg.
match({[{<<"$elemMatch">>, Arg}]}, Values, Cmp) when is_list(Values) ->
try
lists:foreach(fun(V) ->
case match(Arg, V, Cmp) of
true -> throw(matched);
_ -> ok
end
end, Values),
false
catch
throw:matched ->
true;
_:_ ->
false
end;
match({[{<<"$elemMatch">>, _Arg}]}, _Value, _Cmp) ->
false;
% Our comparison operators are fairly straight forward
match({[{<<"$lt">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) < 0;
match({[{<<"$lte">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) =< 0;
match({[{<<"$eq">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) == 0;
match({[{<<"$ne">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) /= 0;
match({[{<<"$gte">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) >= 0;
match({[{<<"$gt">>, Arg}]}, Value, Cmp) ->
Cmp(Value, Arg) > 0;
match({[{<<"$in">>, Args}]}, Values, Cmp) when is_list(Values)->
Pred = fun(Arg) ->
lists:foldl(fun(Value,Match) ->
(Cmp(Value, Arg) == 0) or Match
end, false, Values)
end,
lists:any(Pred, Args);
match({[{<<"$in">>, Args}]}, Value, Cmp) ->
Pred = fun(Arg) -> Cmp(Value, Arg) == 0 end,
lists:any(Pred, Args);
match({[{<<"$nin">>, Args}]}, Value, Cmp) ->
Pred = fun(Arg) -> Cmp(Value, Arg) /= 0 end,
lists:all(Pred, Args);
% This logic is a bit subtle. Basically, if value is
% not undefined, then it exists.
match({[{<<"$exists">>, ShouldExist}]}, Value, _Cmp) ->
Exists = Value /= undefined,
ShouldExist andalso Exists;
match({[{<<"$type">>, Arg}]}, Value, _Cmp) when is_binary(Arg) ->
Arg == mango_json:type(Value);
match({[{<<"$mod">>, [D, R]}]}, Value, _Cmp) when is_integer(Value) ->
Value rem D == R;
match({[{<<"$mod">>, _}]}, _Value, _Cmp) ->
false;
match({[{<<"$regex">>, Regex}]}, Value, _Cmp) when is_binary(Value) ->
try
match == re:run(Value, Regex, [{capture, none}])
catch _:_ ->
false
end;
match({[{<<"$regex">>, _}]}, _Value, _Cmp) ->
false;
match({[{<<"$size">>, Arg}]}, Values, _Cmp) when is_list(Values) ->
length(Values) == Arg;
match({[{<<"$size">>, _}]}, _Value, _Cmp) ->
false;
% We don't have any choice but to believe that the text
% index returned valid matches
match({[{<<"$default">>, _}]}, _Value, _Cmp) ->
true;
% All other operators are internal assertion errors for
% matching because we either should've removed them during
% normalization or something else broke.
match({[{<<"$", _/binary>>=Op, _}]}, _, _) ->
?MANGO_ERROR({invalid_operator, Op});
% We need to traverse value to find field. The call to
% mango_doc:get_field/2 may return either not_found or
% bad_path in which case matching fails.
match({[{Field, Cond}]}, Value, Cmp) ->
case mango_doc:get_field(Value, Field) of
not_found when Cond == {[{<<"$exists">>, false}]} ->
true;
not_found ->
false;
bad_path ->
false;
SubValue when Field == <<"_id">> ->
match(Cond, SubValue, fun mango_json:cmp_raw/2);
SubValue ->
match(Cond, SubValue, Cmp)
end;
match({Props} = Sel, _Value, _Cmp) when length(Props) > 1 ->
erlang:error({unnormalized_selector, Sel}).
================================================
FILE: src/mango_selector_text.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_selector_text).
-export([
convert/1,
convert/2,
append_sort_type/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-define(PERIOD, "\\.").
convert(Object) ->
TupleTree = convert([], Object),
iolist_to_binary(to_query(TupleTree)).
convert(Path, {[{<<"$and">>, Args}]}) ->
Parts = [convert(Path, Arg) || Arg <- Args],
{op_and, Parts};
convert(Path, {[{<<"$or">>, Args}]}) ->
Parts = [convert(Path, Arg) || Arg <- Args],
{op_or, Parts};
convert(Path, {[{<<"$not">>, Arg}]}) ->
{op_not, {field_exists_query(Path), convert(Path, Arg)}};
convert(Path, {[{<<"$default">>, Arg}]}) ->
{op_field, {_, Query}} = convert(Path, Arg),
{op_default, Query};
% The $text operator specifies a Lucene syntax query
% so we just pull it in directly.
convert(Path, {[{<<"$text">>, Query}]}) when is_binary(Query) ->
{op_field, {make_field(Path, Query), value_str(Query)}};
% The MongoDB docs for $all are super confusing and read more
% like they screwed up the implementation of this operator
% and then just documented it as a feature.
%
% This implementation will match the behavior as closely as
% possible based on the available docs but we'll need to have
% the testing team validate how MongoDB handles edge conditions
convert(Path, {[{<<"$all">>, Args}]}) ->
case Args of
[Values] when is_list(Values) ->
% If Args is a single element array then we have to
% either match if Path is that array or if it contains
% the array as an element of an array (which isn't at all
% confusing). For Lucene to return us all possible matches
% that means we just need to search for each value in
% Path.[] and Path.[].[] and rely on our filtering to limit
% the results properly.
Fields1 = convert(Path, {[{<<"$eq">> , Values}]}),
Fields2 = convert([<<"[]">>| Path], {[{<<"$eq">> , Values}]}),
{op_or, [Fields1, Fields2]};
_ ->
% Otherwise the $all operator is equivalent to an $and
% operator so we treat it as such.
convert([<<"[]">> | Path], {[{<<"$and">>, Args}]})
end;
% The $elemMatch Lucene query is not an exact translation
% as we can't enforce that the matches are all for the same
% item in an array. We just rely on the final selector match
% to filter out anything that doesn't match. The only trick
% is that we have to add the `[]` path element since the docs
% say this has to match against an array.
convert(Path, {[{<<"$elemMatch">>, Arg}]}) ->
convert([<<"[]">> | Path], Arg);
% Our comparison operators are fairly straight forward
convert(Path, {[{<<"$lt">>, Arg}]}) when is_list(Arg); is_tuple(Arg);
Arg =:= null ->
field_exists_query(Path);
convert(Path, {[{<<"$lt">>, Arg}]}) ->
{op_field, {make_field(Path, Arg), range(lt, Arg)}};
convert(Path, {[{<<"$lte">>, Arg}]}) when is_list(Arg); is_tuple(Arg);
Arg =:= null->
field_exists_query(Path);
convert(Path, {[{<<"$lte">>, Arg}]}) ->
{op_field, {make_field(Path, Arg), range(lte, Arg)}};
%% This is for indexable_fields
convert(Path, {[{<<"$eq">>, Arg}]}) when Arg =:= null ->
{op_null, {make_field(Path, Arg), value_str(Arg)}};
convert(Path, {[{<<"$eq">>, Args}]}) when is_list(Args) ->
Path0 = [<<"[]">> | Path],
LPart = {op_field, {make_field(Path0, length), value_str(length(Args))}},
Parts0 = [convert(Path0, {[{<<"$eq">>, Arg}]}) || Arg <- Args],
Parts = [LPart | Parts0],
{op_and, Parts};
convert(Path, {[{<<"$eq">>, {_} = Arg}]}) ->
convert(Path, Arg);
convert(Path, {[{<<"$eq">>, Arg}]}) ->
{op_field, {make_field(Path, Arg), value_str(Arg)}};
convert(Path, {[{<<"$ne">>, Arg}]}) ->
{op_not, {field_exists_query(Path), convert(Path, {[{<<"$eq">>, Arg}]})}};
convert(Path, {[{<<"$gte">>, Arg}]}) when is_list(Arg); is_tuple(Arg);
Arg =:= null ->
field_exists_query(Path);
convert(Path, {[{<<"$gte">>, Arg}]}) ->
{op_field, {make_field(Path, Arg), range(gte, Arg)}};
convert(Path, {[{<<"$gt">>, Arg}]}) when is_list(Arg); is_tuple(Arg);
Arg =:= null->
field_exists_query(Path);
convert(Path, {[{<<"$gt">>, Arg}]}) ->
{op_field, {make_field(Path, Arg), range(gt, Arg)}};
convert(Path, {[{<<"$in">>, Args}]}) ->
{op_or, convert_in(Path, Args)};
convert(Path, {[{<<"$nin">>, Args}]}) ->
{op_not, {field_exists_query(Path), convert(Path, {[{<<"$in">>, Args}]})}};
convert(Path, {[{<<"$exists">>, ShouldExist}]}) ->
FieldExists = field_exists_query(Path),
case ShouldExist of
true -> FieldExists;
false -> {op_not, {FieldExists, false}}
end;
% We're not checking the actual type here, just looking for
% anything that has a possibility of matching by checking
% for the field name. We use the same logic for $exists on
% the actual query.
convert(Path, {[{<<"$type">>, _}]}) ->
field_exists_query(Path);
convert(Path, {[{<<"$mod">>, _}]}) ->
field_exists_query(Path, "number");
% The lucene regular expression engine does not use java's regex engine but
% instead a custom implementation. The syntax is therefore different, so we do
% would get different behavior than our view indexes. To be consistent, we will
% simply return docs for fields that exist and then run our match filter.
convert(Path, {[{<<"$regex">>, _}]}) ->
field_exists_query(Path, "string");
convert(Path, {[{<<"$size">>, Arg}]}) ->
{op_field, {make_field([<<"[]">> | Path], length), value_str(Arg)}};
% All other operators are internal assertion errors for
% matching because we either should've removed them during
% normalization or something else broke.
convert(_Path, {[{<<"$", _/binary>>=Op, _}]}) ->
?MANGO_ERROR({invalid_operator, Op});
% We've hit a field name specifier. Check if the field name is accessing
% arrays. Convert occurrences of element position references to .[]. Then we
% need to break the name into path parts and continue our conversion.
convert(Path, {[{Field0, Cond}]}) ->
{ok, PP0} = case Field0 of
<<>> ->
{ok, []};
_ ->
mango_util:parse_field(Field0)
end,
% Later on, we perform a lucene_escape_user call on the
% final Path, which calls parse_field again. Calling the function
% twice converts <<"a\\.b">> to [<<"a">>,<<"b">>]. This leads to
% an incorrect query since we need [<<"a.b">>]. Without breaking
% our escaping mechanism, we simply revert this first parse_field
% effect and replace instances of "." to "\\.".
MP = mango_util:cached_re(mango_period, ?PERIOD),
PP1 = [re:replace(P, MP, <<"\\\\.">>,
[global,{return,binary}]) || P <- PP0],
{PP2, HasInteger} = replace_array_indexes(PP1, [], false),
NewPath = PP2 ++ Path,
case HasInteger of
true ->
OldPath = lists:reverse(PP1, Path),
OldParts = convert(OldPath, Cond),
NewParts = convert(NewPath, Cond),
{op_or, [OldParts, NewParts]};
false ->
convert(NewPath, Cond)
end;
%% For $in
convert(Path, Val) when is_binary(Val); is_number(Val); is_boolean(Val) ->
{op_field, {make_field(Path, Val), value_str(Val)}};
% Anything else is a bad selector.
convert(_Path, {Props} = Sel) when length(Props) > 1 ->
erlang:error({unnormalized_selector, Sel}).
to_query({op_and, Args}) when is_list(Args) ->
QueryArgs = lists:map(fun to_query/1, Args),
["(", mango_util:join(<<" AND ">>, QueryArgs), ")"];
to_query({op_or, Args}) when is_list(Args) ->
["(", mango_util:join(" OR ", lists:map(fun to_query/1, Args)), ")"];
to_query({op_not, {ExistsQuery, Arg}}) when is_tuple(Arg) ->
["(", to_query(ExistsQuery), " AND NOT (", to_query(Arg), "))"];
%% For $exists:false
to_query({op_not, {ExistsQuery, false}}) ->
["($fieldnames:/.*/ ", " AND NOT (", to_query(ExistsQuery), "))"];
to_query({op_insert, Arg}) when is_binary(Arg) ->
["(", Arg, ")"];
%% We escape : and / for now for values and all lucene chars for fieldnames
%% This needs to be resolved.
to_query({op_field, {Name, Value}}) ->
NameBin = iolist_to_binary(Name),
["(", mango_util:lucene_escape_user(NameBin), ":", Value, ")"];
%% This is for indexable_fields
to_query({op_null, {Name, Value}}) ->
NameBin = iolist_to_binary(Name),
["(", mango_util:lucene_escape_user(NameBin), ":", Value, ")"];
to_query({op_fieldname, {Name, Wildcard}}) ->
NameBin = iolist_to_binary(Name),
["($fieldnames:", mango_util:lucene_escape_user(NameBin), Wildcard, ")"];
to_query({op_default, Value}) ->
["($default:", Value, ")"].
%% We match on fieldname and fieldname.[]
convert_in(Path, Args) ->
Path0 = [<<"[]">> | Path],
lists:map(fun(Arg) ->
case Arg of
{Object} ->
Parts = lists:map(fun (SubObject) ->
Fields1 = convert(Path, {[SubObject]}),
Fields2 = convert(Path0, {[SubObject]}),
{op_or, [Fields1, Fields2]}
end, Object),
{op_or, Parts};
SingleVal ->
Fields1 = {op_field, {make_field(Path, SingleVal),
value_str(SingleVal)}},
Fields2 = {op_field, {make_field(Path0, SingleVal),
value_str(SingleVal)}},
{op_or, [Fields1, Fields2]}
end
end, Args).
make_field(Path, length) ->
[path_str(Path), <<":length">>];
make_field(Path, Arg) ->
[path_str(Path), <<":">>, type_str(Arg)].
range(lt, Arg) ->
[<<"[-Infinity TO ">>, value_str(Arg), <<"}">>];
range(lte, Arg) ->
[<<"[-Infinity TO ">>, value_str(Arg), <<"]">>];
range(gte, Arg) ->
[<<"[">>, value_str(Arg), <<" TO Infinity]">>];
range(gt, Arg) ->
[<<"{">>, value_str(Arg), <<" TO Infinity]">>].
field_exists_query(Path) ->
% We specify two here for :* and .* so that we don't incorrectly
% match a path foo.name against foo.name_first (if were to just
% appened * isntead).
Parts = [
% We need to remove the period from the path list to indicate that it is
% a path separator. We escape the colon because it is not used as a
% separator and we escape colons in field names.
{op_fieldname, {[path_str(Path), ":"], "*"}},
{op_fieldname, {[path_str(Path)], ".*"}}
],
{op_or, Parts}.
field_exists_query(Path, Type) ->
{op_fieldname, {[path_str(Path), ":"], Type}}.
path_str(Path) ->
path_str(Path, []).
path_str([], Acc) ->
Acc;
path_str([Part], Acc) ->
% No reverse because Path is backwards
% during recursion of convert.
[Part | Acc];
path_str([Part | Rest], Acc) ->
case Part of
% do not append a period if Part is blank
<<>> ->
path_str(Rest, [Acc]);
_ ->
path_str(Rest, [<<".">>, Part | Acc])
end.
type_str(Value) when is_number(Value) ->
<<"number">>;
type_str(Value) when is_boolean(Value) ->
<<"boolean">>;
type_str(Value) when is_binary(Value) ->
<<"string">>;
type_str(null) ->
<<"null">>.
value_str(Value) when is_binary(Value) ->
case mango_util:is_number_string(Value) of
true ->
<<"\"", Value/binary, "\"">>;
false ->
mango_util:lucene_escape_query_value(Value)
end;
value_str(Value) when is_integer(Value) ->
list_to_binary(integer_to_list(Value));
value_str(Value) when is_float(Value) ->
list_to_binary(float_to_list(Value));
value_str(true) ->
<<"true">>;
value_str(false) ->
<<"false">>;
value_str(null) ->
<<"true">>.
append_sort_type(RawSortField, Selector) ->
EncodeField = mango_util:lucene_escape_user(RawSortField),
String = mango_util:has_suffix(EncodeField, <<"_3astring">>),
Number = mango_util:has_suffix(EncodeField, <<"_3anumber">>),
case {String, Number} of
{true, _} ->
<<EncodeField/binary, "<string>">>;
{_, true} ->
<<EncodeField/binary, "<number>">>;
_ ->
Type = get_sort_type(RawSortField, Selector),
<<EncodeField/binary, Type/binary>>
end.
get_sort_type(Field, Selector) ->
Types = get_sort_types(Field, Selector, []),
case lists:usort(Types) of
[str] -> <<"_3astring<string>">>;
[num] -> <<"_3anumber<number>">>;
_ -> ?MANGO_ERROR({text_sort_error, Field})
end.
get_sort_types(Field, {[{Field, {[{<<"$", _/binary>>, Cond}]}}]}, Acc)
when is_binary(Cond) ->
[str | Acc];
get_sort_types(Field, {[{Field, {[{<<"$", _/binary>>, Cond}]}}]}, Acc)
when is_number(Cond) ->
[num | Acc];
get_sort_types(Field, {[{_, Cond}]}, Acc) when is_list(Cond) ->
lists:foldl(fun(Arg, InnerAcc) ->
get_sort_types(Field, Arg, InnerAcc)
end, Acc, Cond);
get_sort_types(Field, {[{_, Cond}]}, Acc) when is_tuple(Cond)->
get_sort_types(Field, Cond, Acc);
get_sort_types(_Field, _, Acc) ->
Acc.
replace_array_indexes([], NewPartsAcc, HasIntAcc) ->
{NewPartsAcc, HasIntAcc};
replace_array_indexes([Part | Rest], NewPartsAcc, HasIntAcc) ->
{NewPart, HasInt} = try
_ = list_to_integer(binary_to_list(Part)),
{<<"[]">>, true}
catch _:_ ->
{Part, false}
end,
replace_array_indexes(Rest, [NewPart | NewPartsAcc],
HasInt or HasIntAcc).
================================================
FILE: src/mango_sort.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_sort).
-export([
new/1,
to_json/1,
fields/1,
directions/1
]).
-include("mango.hrl").
new(Fields) when is_list(Fields) ->
Sort = {[sort_field(Field) || Field <- Fields]},
validate(Sort),
{ok, Sort};
new(Else) ->
?MANGO_ERROR({invalid_sort_json, Else}).
to_json({Fields}) ->
to_json(Fields);
to_json([]) ->
[];
to_json([{Name, Dir} | Rest]) ->
[{[{Name, Dir}]} | to_json(Rest)].
fields({Props}) ->
[Name || {Name, _Dir} <- Props].
directions({Props}) ->
[Dir || {_Name, Dir} <- Props].
sort_field(Field) when is_binary(Field) ->
{Field, <<"asc">>};
sort_field({[{Name, <<"asc">>}]}) when is_binary(Name) ->
{Name, <<"asc">>};
sort_field({[{Name, <<"desc">>}]}) when is_binary(Name) ->
{Name, <<"desc">>};
sort_field({Name, BadDir}) when is_binary(Name) ->
?MANGO_ERROR({invalid_sort_dir, BadDir});
sort_field(Else) ->
?MANGO_ERROR({invalid_sort_field, Else}).
validate({Props}) ->
% Assert each field is in the same direction
% until we support mixed direction sorts.
Dirs = [D || {_, D} <- Props],
case lists:usort(Dirs) of
[] ->
ok;
[_] ->
ok;
_ ->
?MANGO_ERROR({unsupported, mixed_sort})
end.
================================================
FILE: src/mango_util.erl
================================================
% Licensed under the Apache License, Version 2.0 (the "License"); you may not
% use this file except in compliance with the License. You may obtain a copy of
% the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
% WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
% License for the specific language governing permissions and limitations under
% the License.
-module(mango_util).
-export([
open_doc/2,
open_ddocs/1,
load_ddoc/2,
defer/3,
do_defer/3,
assert_ejson/1,
to_lower/1,
enc_dbname/1,
dec_dbname/1,
enc_hex/1,
dec_hex/1,
lucene_escape_field/1,
lucene_escape_query_value/1,
lucene_escape_user/1,
is_number_string/1,
check_lang/1,
has_suffix/2,
join/2,
parse_field/1,
cached_re/2
]).
-include_lib("couch/include/couch_db.hrl").
-include("mango.hrl").
-define(DIGITS, "(\\p{N}+)").
-define(HEXDIGITS, "([0-9a-fA-F]+)").
-define(EXP, "[eE][+-]?" ++ ?DIGITS).
-define(NUMSTRING,
"[\\x00-\\x20]*" ++ "[+-]?(" ++ "NaN|"
++ "Infinity|" ++ "((("
++ ?DIGITS
++ "(\\.)?("
++ ?DIGITS
++ "?)("
++ ?EXP
++ ")?)|"
++ "(\\.("
++ ?DIGITS
++ ")("
++ ?EXP
++ ")?)|"
++ "(("
++ "(0[xX]"
++ ?HEXDIGITS
++ "(\\.)?)|"
++ "(0[xX]"
++ ?HEXDIGITS
++ "?(\\.)"
++ ?HEXDIGITS
++ ")"
++ ")[pP][+-]?" ++ ?DIGITS ++ "))" ++ "[fFdD]?))" ++ "[\\x00-\\x20]*").
open_doc(Db, DocId) ->
Opts = [deleted],
case mango_util:defer(fabric, open_doc, [Db, DocId, Opts]) of
{ok, Doc} ->
{ok, Doc};
{not_found, _} ->
not_found;
_ ->
?MANGO_ERROR({error_loading_doc, DocId})
end.
open_ddocs(Db) ->
case mango_util:defer(fabric, design_docs, [Db]) of
{ok, Docs} ->
{ok, Docs};
_ ->
?MANGO_ERROR(error_loading_ddocs)
end.
load_ddoc(Db, DDocId) ->
case mango_util:open_doc(Db, DDocId) of
{ok, Doc} ->
{ok, check_lang(Doc)};
not_found ->
Body = {[
{<<"language">>, <<"query">>}
]},
{ok, #doc{id = DDocId, body = Body}}
end.
defer(Mod, Fun, Args) ->
%twig:log(error, "MFA: ~p", [{Mod, Fun, Args}]),
{Pid, Ref} = erlang:spawn_monitor(?MODULE, do_defer, [Mod, Fun, Args]),
receive
{'DOWN', Ref, process, Pid, {mango_defer_ok, Value}} ->
Value;
{'DOWN', Ref, process, Pid, {mango_defer_throw, Value}} ->
erlang:throw(Value);
{'DOWN', Ref, process, Pid, {mango_defer_error, Value}} ->
erlang:error(Value);
{'DOWN', Ref, process, Pid, {mango_defer_exit, Value}} ->
erlang:exit(Value)
end.
do_defer(Mod, Fun, Args) ->
try erlang:apply(Mod, Fun, Args) of
Resp ->
erlang:exit({mango_defer_ok, Resp})
catch
throw:Error ->
Stack = erlang:get_stacktrace(),
twig:log(err, "Defered error: ~w~n ~p", [{throw, Error}, Stack]),
erlang:exit({mango_defer_throw, Error});
error:Error ->
Stack = erlang:get_stacktrace(),
twig:log(err, "Defered error: ~w~n ~p", [{error, Error}, Stack]),
erlang:exit({mango_defer_error, Error});
exit:Error ->
Stack = erlang:get_stacktrace(),
twig:log(err, "Defered error: ~w~n ~p", [{exit, Error}, Stack]),
erlang:exit({mango_defer_exit, Error})
end.
assert_ejson({Props}) ->
assert_ejson_obj(Props);
assert_ejson(Vals) when is_list(Vals) ->
assert_ejson_arr(Vals);
assert_ejson(null) ->
true;
assert_ejson(true) ->
true;
assert_ejson(false) ->
true;
assert_ejson(String) when is_binary(String) ->
true;
assert_ejson(Number) when is_number(Number) ->
true;
assert_ejson(_Else) ->
false.
assert_ejson_obj([]) ->
true;
assert_ejson_obj([{Key, Val} | Rest]) when is_binary(Key) ->
case assert_ejson(Val) of
true ->
assert_ejson_obj(Rest);
false ->
false
end;
assert_ejson_obj(_Else) ->
false.
assert_ejson_arr([]) ->
true;
assert_ejson_arr([Val | Rest]) ->
case assert_ejson(Val) of
true ->
assert_ejson_arr(Rest);
false ->
false
end.
check_lang(#doc{id = Id, deleted = true}) ->
Body = {[
{<<"language">>, <<"query">>}
]},
#doc{id = Id, body = Body};
check_lang(#doc{body = {Props}} = Doc) ->
case lists:keyfind(<<"language">>, 1, Props) of
{<<"language">>, <<"query">>} ->
Doc;
Else ->
?MANGO_ERROR({invalid_ddoc_lang, Else})
end.
to_lower(Key) when is_binary(Key) ->
KStr = binary_to_list(Key),
KLower = string:to_lower(KStr),
list_to_binary(KLower).
enc_dbname(<<>>) ->
<<>>;
enc_dbname(<<A:8/integer, Rest/binary>>) ->
Bytes = enc_db_byte(A),
Tail = enc_dbname(Rest),
<<Bytes/binary, Tail/binary>>.
enc_db_byte(N) when N >= $a, N =< $z -> <<N>>;
enc_db_byte(N) when N >= $0, N =< $9 -> <<N>>;
enc_db_byte(N) when N == $/; N == $_; N == $- -> <<N>>;
enc_db_byte(N) ->
H = enc_hex_byte(N div 16),
L = enc_hex_byte(N rem 16),
<<$$, H:8/integer, L:8/integer>>.
dec_dbname(<<>>) ->
<<>>;
dec_dbname(<<$$, _:8/integer>>) ->
throw(invalid_dbname_encoding);
dec_dbname(<<$$, H:8/integer, L:8/integer, Rest/binary>>) ->
Byte = (dec_hex_byte(H) bsl 4) bor dec_hex_byte(L),
Tail = dec_dbname(Rest),
<<Byte:8/integer, Tail/binary>>;
dec_dbname(<<N:8/integer, Rest/binary>>) ->
Tail = dec_dbname(Rest),
<<N:8/integer, Tail/binary>>.
enc_hex(<<>>) ->
<<>>;
enc_hex(<<V:8/integer, Rest/binary>>) ->
H = enc_hex_byte(V div 16),
L = enc_hex_byte(V rem 16),
Tail = enc_hex(Rest),
<<H:8/integer, L:8/integer, Tail/binary>>.
enc_hex_byte(N) when N >= 0, N < 10 -> $0 + N;
enc_hex_byte(N) when N >= 10, N < 16 -> $a + (N - 10);
enc_hex_byte(N) -> throw({invalid_hex_value, N}).
dec_hex(<<>>) ->
<<>>;
dec_hex(<<_:8/integer>>) ->
throw(invalid_hex_string);
dec_hex(<<H:8/integer, L:8/integer, Rest/binary>>) ->
Byte = (dec_hex_byte(H) bsl 4) bor dec_hex_byte(L),
Tail = dec_hex(Rest),
<<Byte:8/integer, Tail/binary>>.
dec_hex_byte(N) when N >= $0, N =< $9 -> (N - $0);
dec_hex_byte(N) when N >= $a, N =< $f -> (N - $a) + 10;
dec_hex_byte(N) when N >= $A, N =< $F -> (N - $A) + 10;
dec_hex_byte(N) -> throw({invalid_hex_character, N}).
lucene_escape_field(Bin) when is_binary(Bin) ->
Str = binary_to_list(Bin),
Enc = lucene_escape_field(Str),
iolist_to_binary(Enc);
lucene_escape_field([H | T]) when is_number(H), H >= 0, H =< 255 ->
if
H >= $a, $z >= H ->
[H | lucene_escape_field(T)];
H >= $A, $Z >= H ->
[H | lucene_escape_field(T)];
H >= $0, $9 >= H ->
[H | lucene_escape_field(T)];
true ->
Hi = enc_hex_byte(H div 16),
Lo = enc_hex_byte(H rem 16),
[$_, Hi, Lo | lucene_escape_field(T)]
end;
lucene_escape_field([]) ->
[].
lucene_escape_query_value(IoList) when is_list(IoList) ->
lucene_escape_query_value(iolist_to_binary(IoList));
lucene_escape_query_value(Bin) when is_binary(Bin) ->
IoList = lucene_escape_qv(Bin),
iolist_to_binary(IoList).
% This escapes the special Lucene query characters
% listed below as well as any whitespace.
%
% + - && || ! ( ) { } [ ] ^ ~ * ? : \ " /
%
lucene_escape_qv(<<>>) -> [];
lucene_escape_qv(<<"&&", Rest/binary>>) ->
["\\&&" | lucene_escape_qv(Rest)];
lucene_escape_qv(<<"||", Rest/binary>>) ->
["\\||" | lucene_escape_qv(Rest)];
lucene_escape_qv(<<C, Rest/binary>>) ->
NeedsEscape = "+-(){}[]!^~*?:/\\\" \t\r\n",
Out = case lists:member(C, NeedsEscape) of
true -> ["\\", C];
false -> [C]
end,
Out ++ lucene_escape_qv(Rest).
lucene_escape_user(Field) ->
{ok, Path} = parse_field(Field),
Escaped = [mango_util:lucene_escape_field(P) || P <- Path],
iolist_to_binary(join(".", Escaped)).
has_suffix(Bin, Suffix) when is_binary(Bin), is_binary(Suffix) ->
SBin = size(Bin),
SSuffix = size(Suffix),
if SBin < SSuffix -> false; true ->
PSize = SBin - SSuffix,
case Bin of
<<_:PSize/binary, Suffix/binary>> ->
true;
_ ->
false
end
end.
join(_Sep, [Item]) ->
[Item];
join(Sep, [Item | Rest]) ->
[Item, Sep | join(Sep, Rest)].
is_number_string(Value) when is_binary(Value) ->
is_number_string(binary_to_list(Value));
is_number_string(Value) when is_list(Value)->
MP = cached_re(mango_numstring_re, ?NUMSTRING),
case re:run(Value, MP) of
nomatch ->
false;
_ ->
true
end.
cached_re(Name, RE) ->
case mochiglobal:get(Name) of
undefined ->
{ok, MP} = re:compile(RE),
ok = mochiglobal:put(Name, MP),
MP;
MP ->
MP
end.
parse_field(Field) ->
case binary:match(Field, <<"\\">>, []) of
nomatch ->
% Fast path, no regex required
{ok, check_non_empty(Field, binary:split(Field, <<".">>, [global]))};
_ ->
parse_field_slow(Field)
end.
parse_field_slow(Field) ->
Path = lists:map(fun
(P) when P =:= <<>> ->
?MANGO_ERROR({invalid_field_name, Field});
(P) ->
re:replace(P, <<"\\\\">>, <<>>, [global, {return, binary}])
end, re:split(Field, <<"(?<!\\\\)\\.">>)),
{ok, Path}.
check_non_empty(Field, Parts) ->
case lists:member(<<>>, Parts) of
true ->
?MANGO_ERROR({invalid_field_name, Field});
false ->
Parts
end.
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
parse_field_test() ->
?assertEqual({ok, [<<"ab">>]}, parse_field(<<"ab">>)),
?assertEqual({ok, [<<"a">>, <<"b">>]}, parse_field(<<"a.b">>)),
?assertEqual({ok, [<<"a.b">>]}, parse_field(<<"a\\.b">>)),
?assertEqual({ok, [<<"a">>, <<"b">>, <<"c">>]}, parse_field(<<"a.b.c">>)),
?assertEqual({ok, [<<"a">>, <<"b.c">>]}, parse_field(<<"a.b\\.c">>)),
Exception = {mango_error, ?MODULE, {invalid_field_name, <<"a..b">>}},
?assertThrow(Exception, parse_field(<<"a..b">>)).
is_number_string_test() ->
?assert(is_number_string("0")),
?assert(is_number_string("1")),
?assert(is_number_string("1.0")),
?assert(is_number_string("1.0E10")),
?assert(is_number_string("0d")),
?assert(is_number_string("-1")),
?assert(is_number_string("-1.0")),
?assertNot(is_number_string("hello")),
?assertNot(is_number_string("")),
?assertMatch({match, _}, re:run("1.0", mochiglobal:get(mango_numstring_re))).
-endif.
================================================
FILE: test/01-index-crud-test.py
================================================
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy of
# the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.
import random
import mango
class IndexCrudTests(mango.DbPerClass):
def test_bad_fields(self):
bad_fields = [
None,
True,
False,
"bing",
2.0,
{"foo": "bar"},
[{"foo": 2}],
[{"foo": "asc", "bar": "desc"}],
[{"foo": "asc"}, {"bar": "desc"}]
]
for fields in bad_fields:
try:
self.db.create_index(fields)
except Exception, e:
assert e.response.status_code == 400
else:
raise AssertionError("bad create index")
def test_bad_types(self):
bad_types = [
None,
True,
False,
1.5,
"foo", # Future support
"geo", # Future support
{"foo": "bar"},
["baz", 3.0]
]
for bt in bad_types:
try:
self.db.create_index(["foo"], idx_type=bt)
except Exception, e:
assert e.response.status_code == 400, (bt, e.response.status_code)
else:
raise AssertionError("bad create index")
def test_bad_names(self):
bad_names = [
True,
False,
1.5,
{"foo": "bar"},
[None, False]
]
for bn in bad_names:
try:
self.db.create_index(["foo"], name=bn)
except Exception, e:
assert e.response.status_code == 400
else:
raise AssertionError("bad create index")
try:
self.db.create_index(["foo"], ddoc=bn)
except Exception, e:
assert e.response.status_code == 400
else:
raise AssertionError("bad create index")
def test_create_idx_01(self):
fields = ["foo", "bar"]
ret = self.db.create_index(fields, name="idx_01")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_01":
continue
assert idx["def"]["fields"] == [{"foo": "asc"}, {"bar": "asc"}]
return
raise AssertionError("index not created")
def test_create_idx_01_exists(self):
fields = ["foo", "bar"]
ret = self.db.create_index(fields, name="idx_01")
assert ret is False
def test_create_idx_02(self):
fields = ["baz", "foo"]
ret = self.db.create_index(fields, name="idx_02")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_02":
continue
assert idx["def"]["fields"] == [{"baz": "asc"}, {"foo": "asc"}]
return
raise AssertionError("index not created")
def test_read_idx_doc(self):
for idx in self.db.list_indexes():
if idx["type"] == "special":
continue
ddocid = idx["ddoc"]
doc = self.db.open_doc(ddocid)
assert doc["_id"] == ddocid
info = self.db.ddoc_info(ddocid)
assert info["name"] == ddocid
def test_delete_idx_escaped(self):
pre_indexes = self.db.list_indexes()
ret = self.db.create_index(["bing"], name="idx_del_1")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_del_1":
continue
assert idx["def"]["fields"] == [{"bing": "asc"}]
self.db.delete_index(idx["ddoc"].replace("/", "%2F"), idx["name"])
post_indexes = self.db.list_indexes()
assert pre_indexes == post_indexes
def test_delete_idx_unescaped(self):
pre_indexes = self.db.list_indexes()
ret = self.db.create_index(["bing"], name="idx_del_2")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_del_2":
continue
assert idx["def"]["fields"] == [{"bing": "asc"}]
self.db.delete_index(idx["ddoc"], idx["name"])
post_indexes = self.db.list_indexes()
assert pre_indexes == post_indexes
def test_delete_idx_no_design(self):
pre_indexes = self.db.list_indexes()
ret = self.db.create_index(["bing"], name="idx_del_3")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_del_3":
continue
assert idx["def"]["fields"] == [{"bing": "asc"}]
self.db.delete_index(idx["ddoc"].split("/")[-1], idx["name"])
post_indexes = self.db.list_indexes()
assert pre_indexes == post_indexes
def test_bulk_delete(self):
fields = ["field1"]
ret = self.db.create_index(fields, name="idx_01")
assert ret is True
fields = ["field2"]
ret = self.db.create_index(fields, name="idx_02")
assert ret is True
fields = ["field3"]
ret = self.db.create_index(fields, name="idx_03")
assert ret is True
docids = []
for idx in self.db.list_indexes():
if idx["ddoc"] is not None:
docids.append(idx["ddoc"])
docids.append("_design/this_is_not_an_index_name")
ret = self.db.bulk_delete(docids)
assert ret["error"][0]["id"] == "_design/this_is_not_an_index_name"
assert len(ret["success"]) == 3
for idx in self.db.list_indexes():
assert idx["type"] != "json"
assert idx["type"] != "text"
def test_recreate_index(self):
pre_indexes = self.db.list_indexes()
for i in range(5):
ret = self.db.create_index(["bing"], name="idx_recreate")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "idx_recreate":
continue
assert idx["def"]["fields"] == [{"bing": "asc"}]
self.db.delete_index(idx["ddoc"], idx["name"])
break
post_indexes = self.db.list_indexes()
assert pre_indexes == post_indexes
def test_delete_misisng(self):
# Missing design doc
try:
self.db.delete_index("this_is_not_a_design_doc_id", "foo")
except Exception, e:
assert e.response.status_code == 404
else:
raise AssertionError("bad index delete")
# Missing view name
indexes = self.db.list_indexes()
not_special = [idx for idx in indexes if idx["type"] != "special"]
idx = random.choice(not_special)
ddocid = idx["ddoc"].split("/")[-1]
try:
self.db.delete_index(ddocid, "this_is_not_an_index_name")
except Exception, e:
assert e.response.status_code == 404
else:
raise AssertionError("bad index delete")
# Bad view type
try:
self.db.delete_index(ddocid, idx["name"], idx_type="not_a_real_type")
except Exception, e:
assert e.response.status_code == 404
else:
raise AssertionError("bad index delete")
def test_create_text_idx(self):
fields = [
{"name":"stringidx", "type" : "string"},
{"name":"booleanidx", "type": "boolean"}
]
ret = self.db.create_text_index(fields=fields, name="text_idx_01")
assert ret is True
for idx in self.db.list_indexes():
if idx["name"] != "text_idx_01":
continue
print idx["def"]
assert idx["def"]["fields"] == [
{"stringidx": "string"},
{"booleanidx": "boolean"}
]
return
raise AssertionError("index not created")
def test_create_bad_text_idx(self):
bad_fields = [
True,
False,
"bing",
2.0,
["foo", "bar"],
[{"name": "foo2"}],
[{"name": "foo3", "type": "garbage"}],
[{"type": "number"}],
[{"name": "age", "type": "number"} , {"name": "bad"}],
[{"name": "age", "type": "number"} , "bla"]
]
for fields in bad_fields:
try:
self.db.create_text_index(fields=fields)
except Exception, e:
assert e.response.status_code == 400
else:
raise AssertionError("bad create text index")
================================================
FILE: test/02-basic-find-test.py
================================================
# -*- coding: latin-1 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy of
# the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.
import mango
class BasicFindTests(mango.UserDocsTests):
def test_bad_selector(self):
bad_selectors = [
None,
True,
False,
1.0,
"foobarbaz",
{"foo":{"$not_an_op": 2}},
{"$gt":2},
[None, "bing"]
]
for bs
gitextract_yrrwmnq4/
├── .gitignore
├── LICENSE.txt
├── README.md
├── TODO.md
├── src/
│ ├── mango.app.src
│ ├── mango.hrl
│ ├── mango_crud.erl
│ ├── mango_cursor.erl
│ ├── mango_cursor.hrl
│ ├── mango_cursor_text.erl
│ ├── mango_cursor_view.erl
│ ├── mango_doc.erl
│ ├── mango_error.erl
│ ├── mango_fields.erl
│ ├── mango_httpd.erl
│ ├── mango_idx.erl
│ ├── mango_idx.hrl
│ ├── mango_idx_special.erl
│ ├── mango_idx_text.erl
│ ├── mango_idx_view.erl
│ ├── mango_json.erl
│ ├── mango_native_proc.erl
│ ├── mango_opts.erl
│ ├── mango_selector.erl
│ ├── mango_selector_text.erl
│ ├── mango_sort.erl
│ └── mango_util.erl
└── test/
├── 01-index-crud-test.py
├── 02-basic-find-test.py
├── 03-operator-test.py
├── 04-key-tests.py
├── 05-index-selection-test.py
├── 06-basic-text-test.py
├── 06-text-default-field-test.py
├── 07-text-custom-field-list-test.py
├── 08-text-limit-test.py
├── 09-text-sort-test.py
├── 10-disable-array-length-field-test.py
├── README.md
├── friend_docs.py
├── limit_docs.py
├── mango.py
└── user_docs.py
SYMBOL INDEX (192 symbols across 15 files)
FILE: test/01-index-crud-test.py
class IndexCrudTests (line 18) | class IndexCrudTests(mango.DbPerClass):
method test_bad_fields (line 19) | def test_bad_fields(self):
method test_bad_types (line 39) | def test_bad_types(self):
method test_bad_names (line 58) | def test_bad_names(self):
method test_create_idx_01 (line 80) | def test_create_idx_01(self):
method test_create_idx_01_exists (line 91) | def test_create_idx_01_exists(self):
method test_create_idx_02 (line 96) | def test_create_idx_02(self):
method test_read_idx_doc (line 107) | def test_read_idx_doc(self):
method test_delete_idx_escaped (line 117) | def test_delete_idx_escaped(self):
method test_delete_idx_unescaped (line 129) | def test_delete_idx_unescaped(self):
method test_delete_idx_no_design (line 141) | def test_delete_idx_no_design(self):
method test_bulk_delete (line 153) | def test_bulk_delete(self):
method test_recreate_index (line 183) | def test_recreate_index(self):
method test_delete_misisng (line 197) | def test_delete_misisng(self):
method test_create_text_idx (line 226) | def test_create_text_idx(self):
method test_create_bad_text_idx (line 244) | def test_create_bad_text_idx(self):
FILE: test/02-basic-find-test.py
class BasicFindTests (line 18) | class BasicFindTests(mango.UserDocsTests):
method test_bad_selector (line 20) | def test_bad_selector(self):
method test_bad_limit (line 39) | def test_bad_limit(self):
method test_bad_skip (line 58) | def test_bad_skip(self):
method test_bad_sort (line 77) | def test_bad_sort(self):
method test_bad_fields (line 97) | def test_bad_fields(self):
method test_bad_r (line 117) | def test_bad_r(self):
method test_bad_conflicts (line 135) | def test_bad_conflicts(self):
method test_simple_find (line 151) | def test_simple_find(self):
method test_multi_cond_and (line 158) | def test_multi_cond_and(self):
method test_multi_cond_or (line 163) | def test_multi_cond_or(self):
method test_multi_col_idx (line 177) | def test_multi_col_idx(self):
method test_missing_not_indexed (line 188) | def test_missing_not_indexed(self):
method test_limit (line 203) | def test_limit(self):
method test_skip (line 210) | def test_skip(self):
method test_sort (line 217) | def test_sort(self):
method test_fields (line 226) | def test_fields(self):
method test_r (line 233) | def test_r(self):
method test_empty (line 238) | def test_empty(self):
method test_empty_subsel (line 246) | def test_empty_subsel(self):
method test_empty_subsel_match (line 253) | def test_empty_subsel_match(self):
method test_unsatisfiable_range (line 262) | def test_unsatisfiable_range(self):
FILE: test/03-operator-test.py
class OperatorTests (line 16) | class OperatorTests(mango.UserDocsTests):
method test_all (line 18) | def test_all(self):
method test_all_non_array (line 30) | def test_all_non_array(self):
method test_elem_match (line 37) | def test_elem_match(self):
method test_in_operator_array (line 66) | def test_in_operator_array(self):
method test_regex (line 75) | def test_regex(self):
method test_exists_false (line 84) | def test_exists_false(self):
FILE: test/04-key-tests.py
class KeyTests (line 56) | class KeyTests(mango.DbPerClass):
method setUpClass (line 58) | def setUpClass(klass):
method run_check (line 64) | def run_check(self, query, check, fields=None, indexes=None):
method test_dot_key (line 71) | def test_dot_key(self):
method test_peso_key (line 82) | def test_peso_key(self):
method test_unicode_in_fieldname (line 93) | def test_unicode_in_fieldname(self):
method test_unicode_in_selector_field (line 107) | def test_unicode_in_selector_field(self):
method test_internal_field_tests (line 114) | def test_internal_field_tests(self):
method test_escape_period (line 127) | def test_escape_period(self):
method test_object_period (line 139) | def test_object_period(self):
FILE: test/05-index-selection-test.py
class IndexSelectionTests (line 17) | class IndexSelectionTests(mango.UserDocsTests):
method setUpClass (line 19) | def setUpClass(klass):
method test_basic (line 23) | def test_basic(self):
method test_with_and (line 27) | def test_with_and(self):
method test_with_text (line 34) | def test_with_text(self):
method test_no_view_index (line 42) | def test_no_view_index(self):
method test_with_or (line 46) | def test_with_or(self):
method test_use_most_columns (line 55) | def test_use_most_columns(self):
class MultiTextIndexSelectionTests (line 73) | class MultiTextIndexSelectionTests(mango.UserDocsTests):
method setUpClass (line 75) | def setUpClass(klass):
method test_view_ok_with_multi_text (line 80) | def test_view_ok_with_multi_text(self):
method test_multi_text_index_is_error (line 84) | def test_multi_text_index_is_error(self):
method test_use_index_works (line 90) | def test_use_index_works(self):
FILE: test/06-basic-text-test.py
class BasicTextTests (line 20) | class BasicTextTests(mango.UserDocsTextTests):
method test_simple (line 21) | def test_simple(self):
method test_with_integer (line 26) | def test_with_integer(self):
method test_with_boolean (line 32) | def test_with_boolean(self):
method test_with_array (line 38) | def test_with_array(self):
method test_array_ref (line 44) | def test_array_ref(self):
method test_number_ref (line 57) | def test_number_ref(self):
method test_lt (line 66) | def test_lt(self):
method test_lte (line 84) | def test_lte(self):
method test_eq (line 97) | def test_eq(self):
method test_ne (line 113) | def test_ne(self):
method test_gt (line 124) | def test_gt(self):
method test_gte (line 137) | def test_gte(self):
method test_and (line 155) | def test_and(self):
method test_or (line 182) | def test_or(self):
method test_and_or (line 199) | def test_and_or(self):
method test_nor (line 233) | def test_nor(self):
method test_in_with_value (line 239) | def test_in_with_value(self):
method test_in_with_array (line 261) | def test_in_with_array(self):
method test_nin_with_value (line 276) | def test_nin_with_value(self):
method test_nin_with_array (line 299) | def test_nin_with_array(self):
method test_all (line 314) | def test_all(self):
method test_exists_field (line 332) | def test_exists_field(self):
method test_exists_array (line 343) | def test_exists_array(self):
method test_exists_object (line 354) | def test_exists_object(self):
method test_exists_object_member (line 365) | def test_exists_object_member(self):
method test_exists_and (line 375) | def test_exists_and(self):
method test_value_chars (line 397) | def test_value_chars(self):
method test_regex (line 402) | def test_regex(self):
method test_size (line 411) | def test_size(self):
class ElemMatchTests (line 419) | class ElemMatchTests(mango.FriendDocsTextTests):
method test_elem_match_non_object (line 420) | def test_elem_match_non_object(self):
method test_elem_match (line 438) | def test_elem_match(self):
class NumStringTests (line 549) | class NumStringTests(mango.DbPerClass):
method setUpClass (line 552) | def setUpClass(klass):
method isFinite (line 558) | def isFinite(num):
method test_floating_point_val (line 565) | def test_floating_point_val(self,f):
FILE: test/06-text-default-field-test.py
class NoDefaultFieldTest (line 17) | class NoDefaultFieldTest(mango.UserDocsTextTests):
method test_basic (line 21) | def test_basic(self):
method test_other_fields_exist (line 26) | def test_other_fields_exist(self):
class NoDefaultFieldWithAnalyzer (line 32) | class NoDefaultFieldWithAnalyzer(mango.UserDocsTextTests):
method test_basic (line 39) | def test_basic(self):
method test_other_fields_exist (line 43) | def test_other_fields_exist(self):
class DefaultFieldWithCustomAnalyzer (line 49) | class DefaultFieldWithCustomAnalyzer(mango.UserDocsTextTests):
method test_basic (line 56) | def test_basic(self):
method test_not_analyzed (line 61) | def test_not_analyzed(self):
FILE: test/07-text-custom-field-list-test.py
class CustomFieldsTest (line 17) | class CustomFieldsTest(mango.UserDocsTextTests):
method test_basic (line 33) | def test_basic(self):
method test_multi_field (line 38) | def test_multi_field(self):
method test_missing (line 46) | def test_missing(self):
method test_missing_type (line 49) | def test_missing_type(self):
method test_field_analyzer_is_keyword (line 57) | def test_field_analyzer_is_keyword(self):
method test_escaped_field (line 68) | def test_escaped_field(self):
method test_filtered_search_fields (line 79) | def test_filtered_search_fields(self):
FILE: test/08-text-limit-test.py
class LimitTests (line 16) | class LimitTests(mango.LimitDocsTextTests):
method test_limit_field (line 18) | def test_limit_field(self):
method test_limit_field2 (line 25) | def test_limit_field2(self):
method test_limit_field3 (line 32) | def test_limit_field3(self):
method test_limit_field4 (line 39) | def test_limit_field4(self):
method test_limit_field5 (line 44) | def test_limit_field5(self):
method test_limit_skip_field1 (line 52) | def test_limit_skip_field1(self):
method test_limit_skip_field2 (line 59) | def test_limit_skip_field2(self):
method test_limit_skip_field3 (line 64) | def test_limit_skip_field3(self):
method test_limit_skip_field4 (line 69) | def test_limit_skip_field4(self):
method test_limit_skip_field5 (line 74) | def test_limit_skip_field5(self):
method test_limit_skip_field6 (line 83) | def test_limit_skip_field6(self):
method test_limit_bookmark (line 93) | def test_limit_bookmark(self):
method run_bookmark_check (line 101) | def run_bookmark_check(self, size):
method run_bookmark_sort_check (line 117) | def run_bookmark_sort_check(self, size):
FILE: test/09-text-sort-test.py
class SortTests (line 16) | class SortTests(mango.UserDocsTextTests):
method test_number_sort (line 18) | def test_number_sort(self):
method test_number_sort_desc (line 24) | def test_number_sort_desc(self):
method test_string_sort (line 35) | def test_string_sort(self):
method test_notype_sort (line 41) | def test_notype_sort(self):
method test_array_sort (line 50) | def test_array_sort(self):
method test_multi_sort (line 56) | def test_multi_sort(self):
method test_guess_type_sort (line 63) | def test_guess_type_sort(self):
method test_guess_dup_type_sort (line 69) | def test_guess_dup_type_sort(self):
method test_ambiguous_type_sort (line 76) | def test_ambiguous_type_sort(self):
method test_guess_multi_sort (line 86) | def test_guess_multi_sort(self):
method test_guess_mix_sort (line 94) | def test_guess_mix_sort(self):
FILE: test/10-disable-array-length-field-test.py
class DisableIndexArrayLengthsTest (line 16) | class DisableIndexArrayLengthsTest(mango.UserDocsTextTests):
method setUpClass (line 19) | def setUpClass(klass):
method test_disable_index_array_length (line 26) | def test_disable_index_array_length(self):
method test_enable_index_array_length (line 32) | def test_enable_index_array_length(self):
FILE: test/friend_docs.py
function setup (line 44) | def setup(db, index_type="view"):
function add_text_indexes (line 53) | def add_text_indexes(db):
FILE: test/limit_docs.py
function setup (line 16) | def setup(db, index_type="view"):
function add_text_indexes (line 25) | def add_text_indexes(db):
FILE: test/mango.py
function random_db_name (line 25) | def random_db_name():
class Database (line 29) | class Database(object):
method __init__ (line 30) | def __init__(self, host, port, dbname, auth=None):
method url (line 40) | def url(self):
method path (line 43) | def path(self, parts):
method create (line 48) | def create(self, q=1, n=3):
method delete (line 54) | def delete(self):
method recreate (line 57) | def recreate(self):
method save_doc (line 63) | def save_doc(self, doc):
method save_docs (line 66) | def save_docs(self, docs, **kwargs):
method open_doc (line 74) | def open_doc(self, docid):
method ddoc_info (line 79) | def ddoc_info(self, ddocid):
method create_index (line 84) | def create_index(self, fields, idx_type="json", name=None, ddoc=None):
method create_text_index (line 103) | def create_text_index(self, analyzer=None, selector=None, idx_type="te...
method list_indexes (line 130) | def list_indexes(self):
method delete_index (line 135) | def delete_index(self, ddocid, name, idx_type="json"):
method bulk_delete (line 140) | def bulk_delete(self, docs):
method find (line 149) | def find(self, selector, limit=25, skip=0, sort=None, fields=None,
method find_one (line 178) | def find_one(self, *args, **kwargs):
class DbPerClass (line 188) | class DbPerClass(unittest.TestCase):
method setUpClass (line 191) | def setUpClass(klass):
method setUp (line 195) | def setUp(self):
class UserDocsTests (line 199) | class UserDocsTests(DbPerClass):
method setUpClass (line 202) | def setUpClass(klass):
class UserDocsTextTests (line 207) | class UserDocsTextTests(DbPerClass):
method setUpClass (line 213) | def setUpClass(klass):
class FriendDocsTextTests (line 223) | class FriendDocsTextTests(DbPerClass):
method setUpClass (line 226) | def setUpClass(klass):
class LimitDocsTextTests (line 230) | class LimitDocsTextTests(DbPerClass):
method setUpClass (line 233) | def setUpClass(klass):
class NumStringDocsTextTests (line 237) | class NumStringDocsTextTests(DbPerClass):
method setUpClass (line 240) | def setUpClass(klass):
FILE: test/user_docs.py
function setup (line 57) | def setup(db, index_type="view", **kwargs):
function add_view_indexes (line 66) | def add_view_indexes(db, kwargs):
function add_text_indexes (line 87) | def add_text_indexes(db, kwargs):
Condensed preview — 43 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (307K chars).
[
{
"path": ".gitignore",
"chars": 30,
"preview": "ebin/\ntest/*.pyc\nvenv/\n.eunit\n"
},
{
"path": "LICENSE.txt",
"chars": 11346,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 18525,
"preview": "Mango\n=====\n\nA MongoDB inspired query language interface for Apache CouchDB.\n\n\nMotivation\n----------\n\nMango provides a s"
},
{
"path": "TODO.md",
"chars": 804,
"preview": "\n* Patch the view engine to do alternative sorts. This will include both the lower level couch\\_view* modules as well as"
},
{
"path": "src/mango.app.src",
"chars": 777,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango.hrl",
"chars": 600,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_crud.erl",
"chars": 5207,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_cursor.erl",
"chars": 3960,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_cursor.hrl",
"chars": 707,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_cursor_text.erl",
"chars": 8095,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_cursor_view.erl",
"chars": 7928,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_doc.erl",
"chars": 15327,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_error.erl",
"chars": 9251,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_fields.erl",
"chars": 1387,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_httpd.erl",
"chars": 7146,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_idx.erl",
"chars": 8950,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_idx.hrl",
"chars": 620,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_idx_special.erl",
"chars": 2062,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_idx_text.erl",
"chars": 8471,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_idx_view.erl",
"chars": 13140,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_json.erl",
"chars": 2304,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_native_proc.erl",
"chars": 8633,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_opts.erl",
"chars": 8141,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_selector.erl",
"chars": 16252,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_selector_text.erl",
"chars": 13934,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_sort.erl",
"chars": 1816,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "src/mango_util.erl",
"chars": 11020,
"preview": "% Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n% use this file except in compliance with "
},
{
"path": "test/01-index-crud-test.py",
"chars": 9017,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/02-basic-find-test.py",
"chars": 7909,
"preview": "# -*- coding: latin-1 -*-\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file "
},
{
"path": "test/03-operator-test.py",
"chars": 2761,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/04-key-tests.py",
"chars": 4811,
"preview": "# -*- coding: latin-1 -*-\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file "
},
{
"path": "test/05-index-selection-test.py",
"chars": 3377,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/06-basic-text-test.py",
"chars": 18302,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/06-text-default-field-test.py",
"chars": 1943,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/07-text-custom-field-list-test.py",
"chars": 3331,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/08-text-limit-test.py",
"chars": 4782,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/09-text-sort-test.py",
"chars": 3649,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/10-disable-array-length-field-test.py",
"chars": 1530,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/README.md",
"chars": 207,
"preview": "Mango Tests\n===========\n\nTo run these, do this in the top level directory:\n\n $ virtualenv venv\n $ source venv/bin/"
},
{
"path": "test/friend_docs.py",
"chars": 13680,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/limit_docs.py",
"chars": 7113,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/mango.py",
"chars": 7117,
"preview": "# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file except in compliance with "
},
{
"path": "test/user_docs.py",
"chars": 12202,
"preview": "# -*- coding: utf-8 -*-\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n# use this file ex"
}
]
About this extraction
This page contains the full source code of the cloudant/mango GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 43 files (281.4 KB), approximately 77.5k tokens, and a symbol index with 192 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.